Delta Lake
Delta Lake Data Connector Documentation
Last updated
Was this helpful?
Delta Lake Data Connector Documentation
Last updated
Was this helpful?
Delta Lake data connector connector enables SQL queries from tables.
from
The from
field for the Delta Lake connector takes the form of delta_lake:path
where path
is any supported path, either local or to a cloud storage location. See the section below.
name
The dataset name. This will be used as the table name within Spice.
Example:
params
client_timeout
Optional. Specifies timeout for object store operations. Default value is 30s
. E.g. client_timeout: 60s
delta_lake_aws_region
Optional. The AWS region for the S3 object store. E.g. us-west-2
.
delta_lake_aws_access_key_id
The access key ID for the S3 object store.
delta_lake_aws_secret_access_key
The secret access key for the S3 object store.
delta_lake_aws_endpoint
Optional. The endpoint for the S3 object store. E.g. s3.us-west-2.amazonaws.com
.
delta_lake_azure_storage_account_name
The Azure Storage account name.
delta_lake_azure_storage_account_key
The Azure Storage master key for accessing the storage account.
delta_lake_azure_storage_client_id
The service principal client id for accessing the storage account.
delta_lake_azure_storage_client_secret
The service principal client secret for accessing the storage account.
delta_lake_azure_storage_sas_key
The shared access signature key for accessing the storage account.
delta_lake_azure_storage_endpoint
Optional. The endpoint for the Azure Blob storage account.
google_service_account
Filesystem path to the Google service account JSON key file.
The table below shows the Delta Lake data types supported, along with the type mapping to Apache Arrow types in Spice.
String
Utf8
Long
Int64
Integer
Int32
Short
Int16
Byte
Int8
Float
Float32
Double
Float64
Boolean
Boolean
Binary
Binary
Date
Date32
Timestamp
Timestamp(Microsecond, Some("UTC"))
TimestampNtz
Timestamp(Microsecond, None)
Decimal
Decimal128
Array
List
Struct
Struct
Map
Map
Delta Lake connector does not support reading Delta tables with the V2Checkpoint
feature enabled. To use the Delta Lake connector with such tables, drop the V2Checkpoint
feature by executing the following command:
Use the to reference a secret, e.g. ${secrets:aws_access_key_id}
.
For more details on dropping Delta table features, refer to the official documentation: