Python SDK

The Python SDK spicepy queries Spice.ai from Python. It uses Apache Arrow Flight to stream results, returning Apache Arrow records that convert directly into pandas dataframes.

Requirements

• Python 3.10 or later

The following packages are installed automatically:

• pyarrow

• pandas

• certifi

• requests

Installation

Install from the GitHub repository, pinned to a release tag:

pip install git+https://github.com/spiceai/spicepy@v3.1.0

Do not run pip install spicepy. The spicepy name on PyPI belongs to an unrelated third-party project, not to this SDK. Install from the GitHub URL above.

To use parameterized queries, two additional packages are required:

pip install adbc-driver-flightsql adbc-driver-manager

Usage

Create a Client, then call query():

from spicepy import Client

client = Client(
    api_key='API_KEY',
    flight_url='grpc+tls://flight.spiceai.io',
)

data = client.query('SELECT trip_distance, total_amount FROM taxi_trips ORDER BY trip_distance DESC LIMIT 10;', timeout=5*60)
pd = data.read_pandas()

Client has the following arguments, all optional:

• api_key (string): App API key, used to authenticate with Spice.ai Cloud. Falls back to the SPICE_API_KEY environment variable.

• flight_url (string): Arrow Flight endpoint (default: grpc://localhost:50051). Use grpc+tls:// for TLS and grpc:// for plaintext.

• http_url (string): HTTP endpoint, used for dataset refreshes (default: https://data.spiceai.io).

• tls_root_cert (Path or string): Path to the TLS certificate to use for the secure connection (omit for automatic detection).

• user_agent (string): Overrides the reported user agent.

The SPICE_API_KEY environment variable authenticates HTTP requests only — it is not applied to Arrow Flight. Pass api_key to the constructor when querying Spice.ai Cloud.

Once a Client is obtained, queries can be made using the query() function, which returns a pyarrow.flight.FlightStreamReader. It has the following arguments:

• query (string, required): The SQL query.

• timeout (int, optional): The timeout in seconds.

If no timeout is specified, it will default to a 10 min timeout then cancel the query, and a TimeoutError exception will be raised.

Call read_pandas() to read the whole result into a dataframe, or read it incrementally — see Streaming.

Usage with local Spice runtime

Follow the quickstart guide to install and run spice locally. flight_url already defaults to the local runtime:

from spicepy import Client

client = Client(http_url='http://localhost:8090')
data = client.query('SELECT trip_distance, total_amount FROM taxi_trips ORDER BY trip_distance DESC LIMIT 10;', timeout=5*60)
pd = data.read_pandas()

flight_url defaults to the local runtime, but http_url defaults to Spice.ai Cloud. When working entirely locally, set http_url as above so dataset refreshes are sent to the local runtime.

Parameterized queries

query_with_params(sql, params) binds positional $1, $2 placeholders and returns a pyarrow.RecordBatchReader. It requires the two ADBC packages listed under Installation.

reader = client.query_with_params(
    'SELECT trip_distance, fare_amount FROM taxi_trips WHERE trip_distance > $1 LIMIT 10',
    [5.0],
)

for batch in reader:
    print(batch.to_pandas())

Parameter values may be plain Python values, whose Arrow type is inferred, or (value, pyarrow_type) tuples to set the type explicitly. Pass [] for a query with no parameters; None raises ValueError.

Refreshing a dataset

refresh_dataset(dataset, refresh_opts=None) triggers a refresh of an accelerated dataset over the HTTP endpoint. RefreshOpts accepts refresh_sql, refresh_mode, and refresh_jitter_max.

from spicepy import Client, RefreshOpts

client.refresh_dataset('taxi_trips', RefreshOpts(refresh_mode='full'))

Contributing

Contribute to or file an issue with the spicepy library at: https://github.com/spiceai/spicepy