Apache Arrow Flight API

Query web3 data with SQL via the Apache Arrow Flight API

SQL query results are now available as Apache Arrow data frames via a high-performance Apache Arrow Flight endpoint.

Arrow Flight is a data protocol built on the high-performance, open-source gRPC protocol.

This enables high-speed access to your data in Python, Go, C++, C#, and Rust, and makes it easy to use libraries like Pandas and NumPy.

We recommend using our SDKs to connect and query this endpoint. SDKs are available for Python, Node.js, and Go with more coming soon. In Python, the query results from the SDK can be easily converted to Pandas or NumPy format.

You may also use Apache's pyarrow library directly.

Note on Apple M1 Macs - How do I know if I have an M1?

The spicepy/pyarrow installation requires miniforge.

See the Python SDK page for installation steps.

Connecting to the endpoint

• Use the gRPC + TLS URL: grpc+tls://flight.spiceai.io

• For Firecache use the gRPC + TLS URL: grpc+tls://firecache.spiceai.io

  • For documentation on the Spice Firecache see Firecache

• Use basic authentication

  • Username can be set to an empty string

  • Password should be set to the API key of your app

Requirements

• Table names must be fully-qualified. For example eth.blocks

Samples

Find code samples in Python in Arrow Flight Samples.

Troubleshooting

Mac/Windows Certificate issue

If you get this error:

Could not get default pem root certs

Install the Let's Encrypt root certificates.

Instructions for macOS

First download the roots.pem file from the Let's Encrypt server:

curl -Lo isrgrootx1.pem https://letsencrypt.org/certs/isrgrootx1.pem 

Before running your code/jupyter notebook the environment variable GRPC_DEFAULT_SSL_ROOTS_FILE_PATH must be set to the pem file path. If you are using command from a terminal this can be done from the folder containing isrgrootx1.pem with:

export GRPC_DEFAULT_SSL_ROOTS_FILE_PATH="$PWD/isrgrootx1.pem"

The export command will set this variable for this specific terminal and thus will need to be run every time you open a new terminal. Additionally you can add to your terminal profile.

Note that $PWD is a bash-specific variable that will be replaced by the current directory path. You can download the certificate file isrgrootx1.pem in a specific location and inform this path instead of $PWD.

Instructions for Windows

@powershell -NoProfile -ExecutionPolicy unrestricted -Command ^
    (new-object System.Net.WebClient).Downloadfile( ^
        'https://letsencrypt.org/certs/isrgrootx1.pem', 'isrgrootx1.pem')
set GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=%cd%\isrgrootx1.pem