BlogTwitterDiscordTelegramSignup/Login

HTTPS

HTTP(s) Data Connector Documentation

The HTTP(s) Data Connector enables federated SQL query across supported file formats stored at an HTTP(s) endpoint.

datasets:
  - from: http://my-http-api.com/report.csv
    name: local_report
    params:
      http_password: ${env:MY_HTTP_PASS}

Configuration

from

The from field must contain a valid URI to the location of a supported file. For example, http://static_username@my-http-api/report.csv.

name

The dataset name. This will be used as the table name within Spice.

Example:

datasets:
  - from: http://static_username@my-http-api.com/report.csv
    name: cool_dataset
    params: ...
SELECT COUNT(*) FROM cool_dataset;
+----------+
| count(*) |
+----------+
| 6001215  |
+----------+

params

The connector supports Basic HTTP authentication via param values.

Parameter Name
Description

http_port

Optional. Port to create HTTP(s) connection over. Default: 80 and 443 for HTTP and HTTPS respectively.

http_username

Optional. Username to provide connection for HTTP basic authentication. Default: None.

http_password

client_timeout

Optional. Specifies timeout for HTTP operations. Default value is 30s E.g. client_timeout: 60s

Examples

Basic example

datasets:
  - from: https://github.com/LAION-AI/audio-dataset/raw/7fd6ae3cfd7cde619f6bed817da7aa2202a5bc28/metadata/freesound/parquet/freesound_parquet.parquet
    name: laion_freesound

Using Basic Authentication

datasets:
  - from: http://static_username@my-http-api.com/report.csv
    name: local_report
    params:
      http_password: ${env:MY_HTTP_PASS}

Last updated

Was this helpful?