SAP Customer Data Cloud Loader

The SAP Customer Data Cloud loader pulls customer identity data out of SAP Customer Data Cloud (formerly Gigya) into your data warehouse using the accounts.search REST API, plus audit.search for API audit events.

It reads full accounts, lite/email accounts with their subscriptions, communication opt-in state, and the audit log. Profile, consent, and custom-schema containers land as JSON columns, so per-tenant schema customisation needs no configuration in Zeotap.

Password hashes are never imported. SELECT * in SAP Customer Data Cloud returns password.hash and hashSettings; Zeotap builds every query from an explicit field list instead, so hash material is excluded by construction and stripped again defensively before rows are written.

Prerequisites

• An SAP Customer Data Cloud site, and the data center it lives in (US1, EU1, AU1, EU2, CN1, or GLOBAL)
• The site API key from Console → Sites (e.g. 3_mKx…) — this is a public identifier, not a secret
• An application key and secret from Console → Admin → Applications, whose permission group includes Search on that site
• A connected Warehouse with write permissions on the target schema

Two things are worth confirming before you start:

• The application key must be allowed to open a cursor. SAP documents cursor paging (openCursor) as available only to server-to-server calls carrying an application key and secret. Without a cursor, a query returns at most 5,000 records in total — not per page — so a full extract is impossible. Zeotap’s connection test probes this explicitly.
• Partner secrets are not supported. SAP retired partner-secret authentication on 31 January 2023, and audit.search rejects it outright. Use an application key.

Authentication

Application Key & Secret (recommended)

1. In the SAP Customer Data Cloud Console, go to Admin → Applications and create (or open) an application
2. Copy the application key and generate/copy its secret
3. Make sure the application’s permission group grants Search on the site you are importing
4. In Zeotap, click Add Loader, select SAP Customer Data Cloud, and choose Application Key & Secret
5. Enter the application key as Application (User) Key and the secret as Application Secret

No request signing is needed with an application key — SAP explicitly waives the HMAC signature in this case. All requests are sent over HTTPS; SAP rejects a secret sent over plain HTTP with errorCode 400001.

OAuth 2.0 Client Credentials

Zeotap can instead exchange the same application key and secret for a bearer token:

POST https://oauth2.<data-center>/oauth2/token
grant_type=client_credentials
client_id=<application key>
client_secret=<application secret>
resource=https://api.gigya.com

Select OAuth 2.0 Client Credentials and supply the application key as Client ID and the secret as Client Secret. The token endpoint is derived from the data center you select — you do not enter a URL. Zeotap caches the token until one minute before it expires and re-mints it automatically when SAP answers 403025 Invalid token, retrying the affected page once.

One honest caveat: SAP’s accounts.search reference says openCursor is available only to calls carrying an application key and secret, while SAP’s own OAuth 2.0 documentation demonstrates a cursor scroll authenticated with a bearer token. The two statements conflict and we could not settle it against a live tenant. Zeotap’s connection test opens a real cursor, so if your tenant enforces the stricter reading, the test fails immediately with an explanation rather than silently truncating an import at 5,000 rows. If that happens, switch to Application Key & Secret.

The data center must match the site

There is no discovery API for a site’s data center — SAP’s own guidance is to hover the flag next to the site in the Console. A wrong selection does not produce a “wrong region” error. The site API key simply does not exist in that data center, and the call comes back as 403007 Permission denied, which reads exactly like a missing permission. If your credentials look correct and you still get a permission error, re-check the data center first.

Note that CN1 is cn1.sapcdm.cn, not a gigya.com host, and that CN1 and EU2 have separate Consoles — a site created in either is invisible from the global Console.

Configuration

Available Streams

Accounts is selected by default.

Why Email Accounts is a separate stream

SAP states plainly that a search from the Accounts object does not return subscriptions. Subscriptions live only under emailAccounts, so they are a separate stream rather than a column on Accounts. Enable both if you need profiles and subscription state together, and join on email — bearing in mind that one lite account can map to several full accounts sharing an email address, so email is not a unique key against accounts.UID.

Nested containers stay JSON

profile, data, preferences, identities, loginIDs, emails, groups, lastLoginLocation, customIdentifiers, rbaPolicy, addresses, and subscriptions are schema-driven and differ per tenant, so they land verbatim as JSON strings and are never flattened into columns. Query them with your warehouse’s JSON functions. This keeps the table shape stable when a customer adds a dataSchema field.

Fields deliberately excluded

The Audit Log stream is a different API

audit.search runs on the audit.<data-center> host, not accounts.<data-center>, and its rules differ: SELECT may only be *, the @timestamp predicate must carry both a lower and an upper bound (without valid timestamps SAP silently returns only the last 30 days), and no window may span more than 6 months. Zeotap sweeps a bounded window per run and abuts the next run’s window to the previous upper bound, so there is neither gap nor overlap. Two further notes: the @timestamp field is renamed event_timestamp in the warehouse because @ is not a portable column name, and calls made with application keys are not audited, so Zeotap’s own reads never appear in the data it imports.

Sync Modes

Full Refresh

Every enabled stream can run full refresh: Zeotap scrolls the entire result set with a cursor and appends every row.

Incremental

Accounts, Email Accounts, and Audit Log support incremental sync. Communications does not — see below.

Accounts and Email Accounts are keyed on lastUpdatedTimestamp, SAP’s epoch-milliseconds field, with the predicate

WHERE lastUpdatedTimestamp >= <stored watermark> ORDER BY lastUpdatedTimestamp

Three deliberate choices in that one line:

• The integer field, not the ISO string. SAP’s own optimisation guidance says date ranges evaluate far more efficiently against a timestamp field, integer literals have an unambiguous documented form (unquoted), and there is no documented literal format for comparing the ISO-8601 lastUpdated field in accounts.search. Filtering the string field would be guesswork.
• >=, not >. lastUpdatedTimestamp is millisecond-granular and SAP can stamp many accounts inside the same millisecond. > would silently drop every record tied on the boundary, permanently. >= re-reads the boundary millisecond instead, which costs a few duplicate rows in an append-only table — the right side of that trade.
• lastUpdatedTimestamp, not lastUpdated. SAP documents lastUpdatedTimestamp as also being bumped by login events, so it moves at least as often as lastUpdated. For a watermark that must not miss a change, the more frequently updated field is the safe one.

The cursor safety lag, and why it exists

SAP documents a delay between writing account data and that data becoming searchable: 1 second on average, up to 30 seconds. Without a guard, a record modified moments before a run finished would carry a timestamp below the new watermark while still being invisible to the query that set it — and the next run, filtering above the watermark, would never see it. That is a permanent, silent gap.

Zeotap therefore never advances a stored watermark past now minus the Cursor Safety Lag (default 60 seconds). The cost is that the final seconds of a window get re-read on the next run; the benefit is that no write can slip between two runs. Lower it only if you accept that risk, and do not set it below 30 seconds.

Communications is full refresh only

FROM communications exposes a single timestamp field, an ISO-8601 string. It has no epoch-milliseconds twin, and SAP documents no literal format for comparing it in a WHERE clause. Rather than ship a cursor the query language cannot reliably filter on, Zeotap declares the stream full-refresh only. Schedule it less frequently than Accounts if opt-in volumes are large.

Deleted accounts are invisible — plan for it

SAP states that deleted accounts do not appear in queries, and there is no tombstone, isDeleted flag, or deletion feed on accounts.search. A hard-deleted account simply stops being returned.

The consequence: incremental sync can never detect a deletion. A profile deleted in SAP Customer Data Cloud remains in your warehouse forever unless something removes it. The only mitigation is a periodic full refresh into a fresh table, then treating rows absent from the new snapshot as deleted — and even that detects absence rather than deletion, so it cannot distinguish a delete from a row that fell outside the query. If you are using this data to satisfy erasure requests, do not rely on the loader alone; drive deletions from SAP’s own consent and deletion tooling as well.

How It Works

1. Cursor open. Zeotap POSTs the query to https://accounts.<data-center>/accounts.search with openCursor=true. It is always a POST, never a GET: an explicit field list makes the query long enough to breach URL length limits.
2. Cursor continuation. Every subsequent request carries only the cursorId returned by the previous response — no query, no openCursor, no LIMIT. Passing query and cursorId together is an error in SAP Customer Data Cloud: the WHERE and ORDER BY are bound once, at open time, and the server holds the sorted result set for the life of the scroll.
3. Batches are emitted, never buffered. Each page is written to the warehouse as it arrives. This is required for memory reasons and because the cursor expires after 5 minutes of inactivity between hops — a large batch size plus a slow warehouse write can kill the scroll mid-way. That is why the default batch size is 500 rather than SAP’s 1000 ceiling.
4. Termination is guarded three ways. Zeotap stops when nextCursorId is absent or empty, or when it equals the cursor just sent, or when a page returns zero results. SAP’s reference documentation, SAP’s own code sample, and SAP KBA 3085264 disagree about which signal actually fires on the last page, so all three are honoured. A short page is explicitly not treated as the end: SAP states the batch size under a cursor is not guaranteed.
5. Errors are read from the body, not the status line. SAP returns HTTP 200 for application-level errors unless httpStatusCodes=true is sent. Zeotap sends that flag and checks errorCode on every response unconditionally, because the body remains authoritative. The deprecated statusCode/statusReason fields are ignored. Every error message includes SAP’s callId, which is the first thing SAP support will ask for.
6. Type fidelity. The six *Timestamp fields are 13-digit epoch millis. Responses are decoded with exact-number semantics so they land as true integers; decoded naively they would arrive as floating point and be written as 1.7e+12 into an integer column, failing at INSERT.
7. Up to four streams are read concurrently. One page at a time within each stream, but the loader pipeline runs several streams in parallel — so the searches-per-second budget below is shared across them. See Rate Limits.

Rate Limits

SAP’s Acceptable Use Policy tiers limits by contracted contact volume, and searches are by far the tightest bucket:

Exceeding a limit returns errorCode 403048 API Rate Limit Exceeded, which Zeotap classifies as retryable and backs off on.

This budget is shared with your production traffic. The search bucket is the same one your own applications draw on, and at the entry tier it can be as low as 5–10 calls per second. Zeotap reads one page at a time within each stream, but the pipeline runs up to four streams concurrently — so with several streams selected you can have up to four concurrent searches against that bucket. Select only the streams you need, and prefer scheduling this loader away from your peak hours. Do not work around this by running several loaders against the same tenant concurrently; a burst of searches can starve the quota your login flows depend on.

SAP does not publish the enforcement windows (burst versus sustained, per-data-center versus per-API-key) — that detail lives in login-walled Knowledge Base Articles 2702625, 2702339, and 3542818. Client-side and per-IP limits are likewise undocumented. Zeotap always sends server-to-server credentials so calls land in the documented server-side bucket rather than the unpublished client-side one.

Troubleshooting