Databricks

This guide covers how to configure Databricks as a warehouse in Zeotap, including workspace setup, SQL warehouse configuration, authentication, and required permissions.

Prerequisites

• A Databricks workspace (AWS, Azure, or GCP)
• A SQL warehouse or all-purpose cluster running in the workspace
• A personal access token with read permissions
• Network access from Zeotap to your Databricks workspace

Connection Configuration

Required Fields

Host

The workspace URL is visible in your browser’s address bar when logged in to Databricks. Use just the hostname without https://:

# AWS
mycompany.cloud.databricks.com

# Azure
adb-1234567890123456.7.azuredatabricks.net

# GCP
mycompany.gcp.databricks.com

HTTP Path

The HTTP path identifies which compute resource to use for running queries. You can find it in the Databricks UI:

For SQL Warehouses (recommended):

1. Go to SQL Warehouses in the left sidebar
2. Click on your warehouse
3. Go to the Connection Details tab
4. Copy the HTTP Path value

The format is typically:

/sql/1.0/warehouses/<warehouse-id>

For All-Purpose Clusters:

1. Go to Compute in the left sidebar
2. Click on your cluster
3. Go to Advanced Options > JDBC/ODBC
4. Copy the HTTP Path value

The format is typically:

/sql/protocolv1/o/<org-id>/<cluster-id>

SQL Warehouses are recommended over all-purpose clusters for Zeotap because they start faster, scale automatically, and provide better cost isolation.

Catalog and Schema

If your Databricks workspace uses Unity Catalog, specify the catalog name (e.g., main, production). If you’re using the legacy Hive Metastore, set the catalog to hive_metastore.

-- Unity Catalog: catalog.schema.table
SELECT * FROM main.customer_data.users
 
-- Hive Metastore: schema.table (catalog is hive_metastore)
SELECT * FROM customer_data.users

Databricks is case-insensitive for schema and table names, so Customer_Data and customer_data reference the same schema.

Authentication

Personal Access Token

Zeotap uses Databricks personal access tokens (PATs) for authentication.

To create a PAT:

1. In the Databricks workspace, click your username in the top-right corner
2. Select Settings
3. Go to Developer > Access Tokens
4. Click Generate New Token
5. Set a description (e.g., “Zeotap CDP”) and expiration
6. Copy the token — it is only shown once

Token format: dapi1234567890abcdef1234567890abcdef

Token expiration: Set a reasonable expiration (e.g., 90 days) and establish a rotation process. When the token expires, update the source configuration in Zeotap and re-test the connection.

Service Principal (Alternative)

For production environments, consider using a Databricks service principal instead of a personal access token:

1. Create a service principal in Account Console > User Management
2. Generate an OAuth secret or PAT for the service principal
3. Grant the service principal access to the required catalog, schema, and tables

Required Permissions

Unity Catalog Permissions

-- Grant catalog access
GRANT USE CATALOG ON CATALOG main TO `zeotap-service-principal`;
 
-- Grant schema access
GRANT USE SCHEMA ON SCHEMA main.customer_data TO `zeotap-service-principal`;
 
-- Grant table read access
GRANT SELECT ON SCHEMA main.customer_data TO `zeotap-service-principal`;

Hive Metastore Permissions

-- Grant database access
GRANT USAGE ON DATABASE customer_data TO `cdp_user`;
 
-- Grant table read access
GRANT SELECT ON DATABASE customer_data TO `cdp_user`;

SQL Warehouse Access

The user or service principal must also have Can Use permission on the SQL warehouse:

1. Go to SQL Warehouses
2. Click on the warehouse
3. Go to the Permissions tab
4. Add the user/service principal with Can Use permission

Staged Loading (GCS)

When Zeotap loads or forwards data into Databricks, it stages the records as files in a Google Cloud Storage (GCS) bucket, which Databricks then reads through Unity Catalog using read_files(). For Databricks to read those staged files, Unity Catalog must govern the staging bucket.

Staged loading requires Unity Catalog — the legacy hive_metastore catalog cannot govern GCS access this way. Set the connection’s Catalog to a Unity Catalog catalog (not hive_metastore) when using staged loading.

Prerequisites

1. A Unity Catalog storage credential for GCS. A Databricks admin creates this in Unity Catalog. When created, Databricks generates a managed Google service-account email that looks like db-uc-credential-xxxxx@<region>.iam.gserviceaccount.com. Paste that email into the GCS Service Account field of the Zeotap Databricks connection. This is the service account Databricks impersonates to read GCS — it is not a key you download.

2. An external location for the staging bucket, bound to the storage credential above. Zeotap auto-creates this external location for each staging bucket at load time if the connecting principal (the token or service principal used in the connection) has the CREATE EXTERNAL LOCATION privilege on the metastore. If it does not, a Databricks admin must pre-create the external location manually under Catalog → External Data → External Locations, pointing it at the staging bucket URL (gs://<bucket>) and using the storage credential above.

3. Bucket read access for the credential’s service account. The storage credential’s service account needs read access on the staging bucket. Zeotap grants this automatically.

Data Types

Delta Lake Features

Databricks tables use Delta Lake format, which provides:

• Time travel — Query historical versions of tables in your model SQL
• Schema evolution — Tables can change schema over time; Zeotap detects column changes
• ACID transactions — Consistent reads even during concurrent writes

-- Time travel: query data as of a specific timestamp
SELECT * FROM customer_data.users TIMESTAMP AS OF '2025-01-01'
 
-- Time travel: query a specific version
SELECT * FROM customer_data.users VERSION AS OF 42

Example Configuration

curl -X POST https://composable.zeotap.com/api/v1/sources \
  -H "Authorization: Bearer $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Production Databricks",
    "type": "databricks",
    "config": {
      "host": "mycompany.cloud.databricks.com",
      "http_path": "/sql/1.0/warehouses/abc123def456",
      "token": "dapi1234567890abcdef",
      "catalog": "main",
      "schema": "customer_data"
    }
  }'

Network Configuration

If your Databricks workspace uses private networking (Private Link, VNet injection), ensure that Zeotap can reach the workspace’s public or private endpoint. Options include:

• Public endpoint with IP allowlisting — Add Zeotap IPs to the workspace’s IP access list
• Private connectivity — Contact Zeotap support for private link options

All Zeotap connections to your workspace originate from these static egress IPs:

To configure IP access lists in Databricks:

1. Go to Admin Console > Workspace Settings
2. Enable IP Access Lists
3. Add Zeotap’s egress IPs (34.76.7.172, 34.22.225.249) to the allowlist

These addresses are stable — Zeotap does not rotate them. If the list ever changes, this page is updated first.

Troubleshooting

Next Steps

• Create a model using your Databricks warehouse
• Use the SQL editor to write Delta Lake queries
• Set up a Reverse ETL sync to activate your data