SAP Commerce Cloud Loader

The SAP Commerce Cloud loader pulls your product catalog and category hierarchy out of SAP Commerce Cloud through the OCC v2 API (/occ/v2, the “Omni Commerce Connect” web services) and lands them in your data warehouse.

Read this first, because it sets expectations that no amount of configuration will change:

OCC v2 is a storefront API, not an extract API. It exists to serve a shopping front end — render a product page, run a search, show one signed-in shopper their own orders. That shape decides what this loader can and cannot bring you:

• What you get: products, the category tree, and (on B2B deployments) organization customers.
• What you do not get: orders and customers. There is no site-wide order list and no user list in OCC v2 — see Why there are no orders or customers.
• Full refresh only. No stream has an incremental cursor, because OCC v2 exposes nothing to filter one on — see Sync Modes.

If you need site-wide orders, customers, or genuine change-data capture out of SAP Commerce Cloud, OCC v2 is the wrong surface. The right one is the OData-based Integration API, which requires an Integration Object you define inside Commerce Cloud — a different integration, not a setting on this one.

Prerequisites

• An SAP Commerce Cloud deployment (Commerce Cloud in the Public Cloud, or an on-premise/private Commerce installation) with the commercewebservices extension deployed.
• Network access from Zeotap to your Commerce Cloud host. Commerce Cloud is customer-hosted, so there is no fixed vendor domain — you supply the host, and it must be reachable and TLS-terminated.
• An OAuth client configured for the client_credentials grant, bound to a user with read access to your catalog. See Authentication.
• The base site id you want to read (for example electronics-spa). Zeotap can list these for you once the host and OAuth client are set.
• For the Organization Customers stream only: a B2B deployment (the b2bcommerce extensions) and a user in b2badmingroup.
• A connected Warehouse with write permissions on the target schema.

Point the loader at an API node, not your storefront

Commerce Cloud publishes no rate limits for OCC, because you host it — the only limits are your own CPU, your Solr cluster and your database. A full catalog pull is a long sequence of FULL-mode reads, which is real load.

In Commerce Cloud in the Public Cloud, give Zeotap the api aspect endpoint rather than the storefront aspect. On a self-managed installation, point it at a node that is not serving shopper traffic. If you cannot separate them, raise Delay Between Pages and schedule the loader off-peak. Doing this well is the single biggest thing you control about this integration.

Authentication

The loader uses OAuth 2.0 client credentials. Zeotap exchanges your client id and secret for an access token at:

POST https://<your-host>/authorizationserver/oauth/token

Note that /authorizationserver is a sibling of /occ — it is not under /occ/v2, and it takes no base site in its path.

Creating the OAuth client

1. Log in to Backoffice as an administrator.
2. Go to System > OAuth > OAuth Clients.
3. Create a new client and set:
   • Client ID — any identifier you like, e.g. zeotap_client. This goes in the OAuth Client ID field.
   • Client Secret — a strong secret. This goes in the OAuth Client Secret field.
   • Authorities — ROLE_CLIENT.
   • Authorized grant types — must include client_credentials.
   • Scope — extended (the scope OCC declares for the client-credentials flow).
4. Bind the client to a service user that can read the catalog. client_credentials authenticates the client, not a shopper, so whatever that user can read is what the loader can read.
5. Paste the id and secret into Zeotap and click Test connection.

Token handling

client_credentials returns no refresh token, so there is nothing to refresh. Zeotap mints a token on demand and caches it until five minutes before it expires, then mints a new one. Token lifetime is read from the token response (expires_in) rather than assumed, because it is configurable per OAuth client in Backoffice — and Commerce Cloud has returned that field as both a bare number and a quoted string across releases, so both are accepted. If a token is revoked mid-run, the loader re-mints once and retries the page it was on.

Configuration

The fields parameter

Every OCC endpoint takes a fields selector, and its default is DEFAULT — a deliberately sparse projection. Ask for a product list without fields and Commerce Cloud returns objects with a handful of keys, no price, no stock, no classifications, and no error. The result is a warehouse table full of NULL columns and nothing to explain it.

So Zeotap sends fields on every single request, defaulting to FULL.

It applies only to GET /products/search. /catalogs, /basesites and /orgCustomers always use FULL, so narrowing the selector can never empty the categories or org_customers streams. You can narrow it if FULL is too heavy for your node — but a selector that omits pagination leaves the loader with no way to know where the result set ends, and the run fails with an explicit error rather than looping. Any custom selector must include pagination(DEFAULT):

products(code,name,price(FULL),stock(FULL)),pagination(DEFAULT)

Available Streams

Products and Categories are selected by default. Organization Customers is not — it is only offered once you turn on the B2B toggle, because on a B2C storefront the endpoint does not exist and every run would fail with a 404.

Products come from the search index, not the database

/products/search is the only endpoint in OCC v2 that lists products, and it is served by Solr, not by a direct catalog query. A product is therefore visible to this loader only if it is in the search index. In practice that means a product is invisible — with no error and no warning — when it is:

• not approved (only approved products are indexed by the standard indexer),
• not in the base site’s catalog version (typically Online, not Staged),
• excluded by your indexer’s configuration (a restricted SolrIndexedType, a visible=false flag, a custom index query),
• newly created and not yet indexed — the full/incremental Solr index jobs run on a cron, so there is always a lag,
• suppressed by a search restriction applied to the user your OAuth client is bound to.

If your row count is lower than your catalog count, this is almost always why. Compare it against the product count in Backoffice for the same catalog version, and check when your Solr index job last completed.

Prices are objects, not numbers

OCC returns money as a structured object, not a scalar:

{ "value": 115.87, "currencyIso": "EUR", "formattedValue": "€115.87", "priceType": "BUY" }

value is a genuine JSON number. Each price is therefore split into two typed scalar columns you can aggregate on directly, with the whole object kept alongside so nothing is lost:

The price you get is the price that user, in that currency, on that base site would see — OCC prices are resolved through the price rows and any customer-group discounts applicable to the user your OAuth client is bound to. If you need list prices independent of a user context, do not read them here.

Nested collections (categories, classifications, images, variant_options, base_options, future_stocks, volume_prices, potential_promotions) are stored as JSON text, and a raw column preserves the complete product as SAP returned it. On BigQuery and Databricks these are STRING columns and on Snowflake VARIANT; BigQuery’s JSON_VALUE, JSON_QUERY and PARSE_JSON all read a STRING column, so querying them is unaffected.

Categories are flattened, not nested

/catalogs returns every catalog with its versions, and each version carries a tree of categories with nested subcategories. Zeotap walks that tree depth-first and emits one row per node, carrying where it came from and where it sits:

• catalog_id / catalog_name / catalog_version_id / catalog_version_name — provenance. The same category id can legitimately appear in two catalog versions, so always filter on the version you care about.
• parent_id, depth, path — position. path is the slash-joined chain of ids from the root, so 1/1.1/1.1.1 is queryable with a simple LIKE.
• subcategory_count and subcategory_ids — the immediate children.

The endpoint is not paginated: one response can carry a whole catalog. Zeotap streams the flattened rows out in 500-row batches. The source tree itself is necessarily held whole while it is walked, because /catalogs is not paginated — there is no page boundary to stream on. A response beyond 256 MiB fails with an explicit error rather than exhausting the worker.

Why there are no orders or customers

This is the most common question about this loader, so here is the exact reason, verified against SAP’s published OCC API references for both the 1905 and 2105 releases.

Orders. OCC v2 has exactly three order-shaped read endpoints:

Customers. There is no GET /users at all. Every user endpoint is GET /{baseSiteId}/users/{userId}/… and requires a user id you already have.

So the chain breaks at enumeration, not at authorization: without a list endpoint there is no set of ids to walk, and no OAuth scope, role, or credential changes that. Shipping orders and customers streams here would mean shipping streams that can never return a row.

What to use instead. SAP’s answer for bulk order and customer extraction is the Integration API (/odata2webservices/…), where you author an Integration Object in Backoffice describing exactly the entities and fields you want, and read it over OData with real filtering. Because the shape of that API is defined by your Integration Object rather than by SAP, it needs a connector that discovers the schema at runtime — which is a separate integration, not an option on this page.

Sync Modes

Every stream is full refresh. This is a property of OCC v2 and not a limitation Zeotap could lift:

• Products carry no timestamp at all. The OCC Product model has no modifiedtime, no lastModified, no createdAt — nothing. There is no field to store as a high-water mark even if there were something to filter on.
• No product or category endpoint accepts a date parameter. /products/search takes query, sort, currentPage, pageSize and fields, and nothing else.
• sort cannot substitute for a cursor. OCC sort values are Solr sort codes (name-asc, price-desc, …) configured per deployment. There is no separate direction parameter and no timestamp sort code, so “sort newest-first and stop early” is not expressible.
• CategoryHierarchy.lastModified exists but is useless as a cursor. It is returned, and Zeotap lands it as the last_modified column so you can inspect it — but /catalogs accepts no filter and no sort, so it cannot narrow what is fetched.

Practical consequence: schedule these streams at a cadence that matches how often your catalog actually changes. A nightly full refresh of a catalog is normal and cheap relative to the storefront traffic the same node serves. Deletions are handled naturally — a full refresh replaces the table, so a product that has left the index simply is not in the new table.

How It Works

• Paging. OCC pages with currentPage (0-indexed) and pageSize. Zeotap drives the loop entirely off the response, never off a local counter — Commerce Cloud silently clamps pageSize to 100, so a loader that counted its own rows would stop early on any larger page size.
• Termination is over-determined on purpose. OCC returns two different pagination envelope shapes that serialize under the same pagination key with different field names (currentPage/totalResults in one, page/totalCount in the other). Only totalPages is common to both, so that is the primary bound — reinforced by stopping on any empty result array and by an absolute page ceiling. A missing pagination object is treated as a configuration error and reported, never as a reason to keep going.
• Streaming, not buffering. Each page is mapped to rows and handed off immediately; the category tree is flattened and emitted in sub-batches. The flattened rows are never all held at once; the single unpaginated /catalogs response is, by necessity.
• fields on every request. See The fields parameter.
• Deterministic ordering. A sort code is passed so page boundaries stay stable for the duration of a run. It is not a cursor and nothing is inferred from it.
• Retries. A 401 mid-run means the token expired: Zeotap mints a new one and retries the same page once. A 429, 502, 503 or 504 is retried in place on a short backoff (honouring Retry-After when present) so a momentarily busy node does not cost you a re-read of the whole catalog.
• Base site validation up front. Test connection checks your base site id against the live /occ/v2/basesites list. This exists because a wrong base site otherwise surfaces as a bare mid-run 404 with an empty body.

Rate Limits

SAP publishes no rate limits for OCC, and there is no vendor-side throttle to respect — Commerce Cloud runs on infrastructure you own or lease, so the ceiling is your own capacity: application CPU, Solr query throughput, and database load.

That makes throughput your decision, and the loader gives you the controls:

If your infrastructure or a CDN in front of it does throttle, Zeotap handles 429 and 503/504 by backing off and retrying the page rather than failing the run.

Troubleshooting

Next Steps

• Create a model to shape the raw catalog — flatten the categories JSON into a product-to-category bridge, or unpack classifications into product attributes
• Join products to categories on categories→code to build a browsable catalog dimension, filtering on catalog_version_id
• Use price_value and stock_level in audiences and traits — both are typed numeric columns, so no JSON parsing is needed