HTTPS

HTTP(s) Data Connector Documentation

The HTTP(s) Data Connector enables federated SQL query across supported file formats stored at an HTTP(s) endpoint.

datasets:
  - from: http://my-http-api.com/report.csv
    name: local_report
    params:
      http_password: ${env:MY_HTTP_PASS}

Configuration

from

The from field must contain a valid URI to the location of a supported file. For example, http://static_username@my-http-api/report.csv.

name

The dataset name. This will be used as the table name within Spice.

Example:

datasets:
  - from: http://[email protected]/report.csv
    name: cool_dataset
    params: ...

SELECT COUNT(*) FROM cool_dataset;

+----------+
| count(*) |
+----------+
| 6001215  |
+----------+

params

The connector supports Basic HTTP authentication via param values.

Parameter Name	Description
http_port	Optional. Port to create HTTP(s) connection over. Default: 80 and 443 for HTTP and HTTPS respectively.
http_username	Optional. Username to provide connection for HTTP basic authentication. Default: None.
http_password	Optional. Password to provide connection for HTTP basic authentication. Default: None. Use the secret replacement syntax to load the password from a secret store, e.g. ${secrets:my_http_pass}.
client_timeout	Optional. Specifies timeout for HTTP operations. Default value is 30s E.g. client_timeout: 60s

Examples

Basic example

datasets:
  - from: https://github.com/LAION-AI/audio-dataset/raw/7fd6ae3cfd7cde619f6bed817da7aa2202a5bc28/metadata/freesound/parquet/freesound_parquet.parquet
    name: laion_freesound

Using Basic Authentication

datasets:
  - from: http://[email protected]/report.csv
    name: local_report
    params:
      http_password: ${env:MY_HTTP_PASS}