Google Cloud Firestore

The Google Cloud Firestore loader imports documents from your Firestore collections into your warehouse. Zeotap reads each collection, captures the full document, and loads it into a table you can model, segment, and activate.

Prerequisites

• A Google Cloud project with a Firestore database in Native mode.
• A Google Cloud service account with read access to Firestore (the Cloud Datastore Viewer role, roles/datastore.viewer).
• A JSON key for that service account.
• A connected warehouse to load the data into.

Authentication

Zeotap authenticates with a service-account JSON key.

1. In the Google Cloud Console, go to IAM & Admin → Service Accounts and create (or select) a service account.
2. Grant it the Cloud Datastore Viewer role (roles/datastore.viewer) on the project that owns the Firestore database.
3. Open the service account, go to Keys → Add Key → Create new key, choose JSON, and download the key file.
4. In Zeotap, add a new Google Cloud Firestore loader and paste the full contents of the JSON key into the Service Account JSON field.

The key is stored encrypted and is only used to read documents from the configured project.

Configuration

Available Collections

Each Firestore collection becomes its own table. If you leave Collections blank, Zeotap discovers and loads every top-level collection in the database. To load only specific collections — or to include collections that are created later — list their IDs explicitly.

Because Firestore is schemaless (documents in the same collection can have different fields), every collection lands with the same stable set of columns:

The full document lives in the data column as JSON text. You extract individual fields downstream in a model using your warehouse’s JSON functions — for example, in BigQuery (JSON_VALUE reads directly from the JSON string; use PARSE_JSON(data) if you need a native JSON value):

SELECT
  id,
  JSON_VALUE(data, '$.name')       AS name,
  CAST(JSON_VALUE(data, '$.age') AS INT64) AS age,
  updated_time
FROM users

Firestore value types map into the JSON as you would expect: strings, numbers, and booleans become JSON scalars; timestamps become ISO-8601 strings; bytes become base64 strings; nested maps and arrays are preserved as nested JSON; document references become their path string; and geo points become a { "latitude": ..., "longitude": ... } object.

Sync Behavior

The loader supports two modes:

• Full refresh (default) — every run reads all documents in each selected collection. Use this when your collections are small, or when documents do not carry a reliable “last modified” field.
• Incremental — every run reads only documents that changed since the last run.

How incremental sync works

Firestore cannot filter or sort a query by the server-managed document update time. A query can only order by and filter on a field that is stored inside the document. Incremental sync therefore relies on a field your application maintains — for example, an updatedAt timestamp that you set on every write.

When you set Cursor Field (and its matching Cursor Field Type), each run:

1. Orders documents by the cursor field, ascending.
2. Reads only documents whose cursor value is greater than the highest value seen in the previous run.
3. Records the new high-watermark for the next run.

This means the cursor field must:

• Exist in every document in the collection. Firestore excludes documents that lack the ordering field, so any document missing the cursor field will never be synced. If some documents may lack the field, use full refresh instead.
• Increase strictly on every write. Each run resumes from after the highest cursor value it last saw (the boundary is exclusive). If two documents can share the same cursor value — common with coarse, manually-set fields such as a whole-second updatedAt, or with batch writes that stamp the same timestamp — a same-valued document that commits after a run completes will be skipped permanently. Prefer a server-set timestamp at sub-second (microsecond) resolution; avoid manually-assigned or low-resolution values. If the field can move backwards or stay the same on an update, use full refresh instead.
• Match the configured type. Choose Timestamp for an ISO/Firestore timestamp field, Number for a numeric version or sequence field, or String for a lexicographically increasing string.

The same cursor field is applied to every collection in the loader, so all selected collections should carry a field with that name.

Rate Limits

Firestore reads are billed per document and are subject to your project’s Firestore quotas. To keep syncs efficient and costs predictable:

• Prefer incremental sync for large collections so each run reads only changed documents.
• Schedule the loader at an interval that matches how fresh the data needs to be — more frequent runs read more documents.

Troubleshooting

Next Steps

• Create a Model over the loaded table.
• Build an Audience from the modeled data.