Data Acceleration
Configure local acceleration for datasets in Spice for faster queries
Datasets can be locally accelerated by the Spice runtime, pulling data from any Data Connector and storing it locally in a Data Accelerator for faster access. The data can be kept up-to-date in real-time or on a refresh schedule, ensuring users always have the latest data locally for querying.
Supported Data Accelerators
Dataset acceleration is enabled by setting the acceleration
configuration. Spice currently supports In-Memory Arrow, DuckDB, SQLite, PostgreSQL as accelerators. For engine specific configuration, see Data Accelerator Documentation
Example - Locally Accelerating taxi_trips with Arrow Accelerator
Refresh Modes
Spice supports three modes to refresh/update locally accelerated data from a connected data source. full
is the default mode. Refer to Data Refresh documentation for detailed refresh usage and configuration.
full
Replace/overwrite the entire dataset on each refresh
A table of users
append
Append/add data to the dataset on each refresh
Append-only, immutable datasets, such as time-series or log data
changes
Apply incremental changes
Customer order lifecycle table
Example - Accelerate with arrow accelerator under full refresh mode
Indexes
Database indexes are essential for optimizing query performance. Configure indexes for accelerators via indexes
field. For detailed configuration, refer to the index documentation.
Example - Configure indexes with SQLite Accelerator
Constraints
Constraints enforce data integrity in a database. Spice supports constraints on locally accelerated tables to ensure data quality and configure behavior for data updates that violate constraints.
Constraints are specified using column references in the Spicepod via the primary_key
field in the acceleration configuration. Additional unique constraints are specified via the indexes
field with the value unique
. Data that violates these constraints will result in a conflict. For constraints configuration details, visit Constraints Documentation.
Example - Configure primary key constraints with SQLite Accelerator
Last updated