HTTPS
HTTP(s) Data Connector Documentation
The HTTP(s) Data Connector enables federated SQL query across supported file formats stored at an HTTP(s) endpoint.
datasets:
- from: http://my-http-api.com/report.csv
name: local_report
params:
http_password: ${env:MY_HTTP_PASS}
Configuration
from
from
The from
field must contain a valid URI to the location of a supported file. For example, http://static_username@my-http-api/report.csv
.
name
name
The dataset name. This will be used as the table name within Spice.
Example:
datasets:
- from: http://[email protected]/report.csv
name: cool_dataset
params: ...
SELECT COUNT(*) FROM cool_dataset;
+----------+
| count(*) |
+----------+
| 6001215 |
+----------+
params
params
The connector supports Basic HTTP authentication via param
values.
http_port
Optional. Port to create HTTP(s) connection over. Default: 80 and 443 for HTTP and HTTPS respectively.
http_username
Optional. Username to provide connection for HTTP basic authentication. Default: None.
http_password
Optional. Password to provide connection for HTTP basic authentication. Default: None. Use the secret replacement syntax to load the password from a secret store, e.g. ${secrets:my_http_pass}
.
client_timeout
Optional. Specifies timeout for HTTP operations. Default value is 30s
E.g. client_timeout: 60s
Examples
Basic example
datasets:
- from: https://github.com/LAION-AI/audio-dataset/raw/7fd6ae3cfd7cde619f6bed817da7aa2202a5bc28/metadata/freesound/parquet/freesound_parquet.parquet
name: laion_freesound
Using Basic Authentication
datasets:
- from: http://[email protected]/report.csv
name: local_report
params:
http_password: ${env:MY_HTTP_PASS}
Last updated
Was this helpful?