Apache Arrow Flight API

SQL Query Apache Arrow Flight API

SQL query results can be served via a high-performance Apache Arrow Flight endpoint. Arrow Flight uses the gRPC protocol for efficient data transfer.

This setup enables high-speed access to your data in Python, Go, C++, C#, Rust, Java, and C# and makes it easy to use libraries like Pandas and NumPy.

SDKs

It's recommended to use the Spice.ai SDKs to connect and query the Arrow Flight endpoint. SDKs are available for Python, Node.js, Go, Rust, Java, and Dotnet.

In Python, query results can be easily converted to Pandas or NumPy formats.

You may also use Apache's pyarrow library directly.

Note on Apple Silicon Macs - How do I know if I have Apple Silicon?

The spicepy/pyarrow installation requires miniforge.

See the Python SDK page for installation steps.

Connecting to the Endpoint

• Endpoint URL: grpc+tls://flight.spiceai.io

• Basic Authentication:

  • Username can be set to an empty string

  • Password should be set to the API key of your app

Requirements

• Table names must be fully-qualified. For example spiceai.quickstart

Samples

Find code samples in Python in Arrow Flight Samples.

Troubleshooting

Mac/Windows Certificate issue

If you get this error:

Could not get default pem root certs

Install the Let's Encrypt root certificates.

Instructions for macOS

First download the roots.pem file from the Let's Encrypt server:

curl -Lo isrgrootx1.pem https://letsencrypt.org/certs/isrgrootx1.pem 

Before running your code/jupyter notebook the environment variable GRPC_DEFAULT_SSL_ROOTS_FILE_PATH must be set to the pem file path. If you are using command from a terminal this can be done from the folder containing isrgrootx1.pem with:

export GRPC_DEFAULT_SSL_ROOTS_FILE_PATH="$PWD/isrgrootx1.pem"

The export command will set this variable for this specific terminal and thus will need to be run every time you open a new terminal. Additionally you can add to your terminal profile.

Note that $PWD is a bash-specific variable that will be replaced by the current directory path. You can download the certificate file isrgrootx1.pem in a specific location and inform this path instead of $PWD.

Instructions for Windows

@powershell -NoProfile -ExecutionPolicy unrestricted -Command ^
    (new-object System.Net.WebClient).Downloadfile( ^
        'https://letsencrypt.org/certs/isrgrootx1.pem', 'isrgrootx1.pem')
set GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=%cd%\isrgrootx1.pem