Rate-limited, retried and idempotent: integration fundamentals

Three properties, one design

These get treated as three items on a checklist, and they are not. They are a chain, and the order is not negotiable.

Start at the end and it is obvious why. Suppose you add retries to an integration that pushes orders into an ERP. The network drops a response after the ERP has already committed the write. Your client sees a timeout, retries, and now there are two orders. You have not made the integration more reliable. You have converted a rare visible failure into a frequent invisible one, which is strictly worse, because nobody gets paged for a duplicate until finance finds it three weeks later.

Now suppose you make the writes idempotent but do not think about the rate limit. Your retry loop hits a throttle, retries immediately, gets throttled again, and retries faster than the bucket refills. The integration is now generating the load that is stopping it from working, and it will stay in that state until somebody kills the process. Backoff is not politeness towards Shopify; it is the only way out of the hole.

So: idempotency, then retries, then rate-limit awareness. Each one is what makes the next one safe.

Does Shopify support idempotency keys? Yes, on some mutations, plus two other primitives

Idempotent means a write can be performed more than once without changing the outcome. The usual way to get there is a key: some value that identifies *this specific intended change*, so the second attempt can be recognized as the same change rather than a new one.

The `@idempotent` directive. Shopify's Admin GraphQL API now has a native key, but only on the mutations that say so. You declare a String! variable and apply the directive to the mutation field: inventorySetQuantities(input: $input) @idempotent(key: $idempotencyKey). The idempotency documentation says that "mutations that accept an @idempotent directive specify that they do in their descriptions", recommends a randomly generated UUID as the key, and says repeated requests with the same parameters are executed only once. On inventorySetQuantities the key was optional as of 2026-01 and required as of 2026-04. The changelog lists 17 mutations (refundCreate plus the inventory and location mutations such as inventoryAdjustQuantities, inventoryTransferCreate and locationActivate) and says that calling them without the directive "will result in an error at runtime", even though the schema does not mark it mandatory. Anything on API version 2026-04 or later that writes inventory or refunds needs this before it works at all.

The key only helps if a retry reuses it. Generate it once per intended change, persist it before the first send, and send the same value on every retry of that change; a new operation gets a new key. We have not verified how long Shopify retains a key or what it returns when the same key arrives with different input, and one developer-forum thread reports repeated adjustments succeeding with a fixed key. Test that behavior on a development store before you lean on it.

For the mutations the directive does not cover, Shopify builds the property into the mutation itself, which is better for the cases it covers. There are two documented mechanisms.

Upsert by your own identifier. The productSet mutation takes an identifier argument, documented as specifying "the identifier that will be used to lookup the resource". One of the accepted identifiers is customId, typed as a UniqueMetafieldValueInput and described as the "Custom ID of product to upsert". That is the hook: put your ERP or PIM's product key into a unique metafield, and every sync from then on is an upsert against a key you own rather than a create-or-update decision your code has to make correctly. Run the same sync twice and you get the same product, because the second run resolves to the record the first one created.

Be deliberate about what productSet does to omitted fields, because it is not uniform. For list-typed fields such as variants, collections and metafields, the documentation says it "creates new entries, updates existing entries, and deletes existing entries that aren't included in the mutation's input". For everything else, it "updates only the included fields. Any omitted fields will remain unchanged." A partial payload will therefore preserve a title it did not mention and delete a variant it did not mention. That is the correct behavior for a full sync from a system of record and a trap for a partial update.

Compare-and-set on inventory. inventorySetQuantities accepts a compareQuantity, and the documentation states that "the mutation will only update the quantity if the persisted quantity matches the compareQuantity value", returning an error if it does not. There is an ignoreCompareQuantity flag to opt out, and the documentation warns that opting out can compromise accuracy under concurrent requests.

That deserves a pause, because it does something subtler than deduplication. Inventory is the one field where several systems write at once: the warehouse, the storefront, a returns tool, a stock-take. Compare-and-set means a stale write loses rather than wins. The failure mode it prevents is not a duplicate; it is your nightly sync silently reinstating a quantity that a sale corrected four minutes ago. Use ignoreCompareQuantity only where you are the sole authority, and be realistic about how rarely that is true.

Inbound, the key is a header. For webhooks, Shopify's documentation on duplicate webhooks tells you to process webhooks with idempotent operations, and says "if your processing isn't idempotent, use the X-Shopify-Webhook-Id header to detect and skip duplicates". It also offers X-Shopify-Event-Id to correlate deliveries that came from the same merchant action. Store the id, check it before processing, and keep the table for long enough to cover the retry window. This is a dozen lines of code and it eliminates an entire category of support ticket.

For everything not covered by those (writing a metafield, or a mutation without the directive) you construct the key yourself: a deterministic identifier derived from the source event, recorded on your side before the call and checked after a failure. The rule is that the key must be fixed per intended change, not minted fresh on each attempt, or the retry will generate a new one and defeat the point.

Retries: the errors you must retry, and the ones you must not

Blanket retry logic is almost as bad as none, because it turns permanent failures into permanent load. The useful distinction is whether the same request, sent again unchanged, could plausibly succeed.

• Retry: connection failures, timeouts, and 5xx responses. The request may never have been processed, or may have been processed and lost on the way back, which is exactly the case idempotency exists for.
• Retry, with backoff: a 429. The request was well-formed and you were over the limit. This is the one where the backoff interval, not the retry itself, is the whole design.
• Never retry: a mutation that returned userErrors. These are errors in the input of a mutation, such as validation failures or business logic violations. The operation ran and the business logic rejected it. Sending it again produces the same rejection, at the cost of rate limit you needed for work that could succeed. Route these to a dead-letter queue a human reads.
• Never retry, and alert: authentication and authorization failures. An app whose token has been revoked or whose scopes have changed does not recover by trying harder, and a retry loop hides the fact that the integration has been dead since Tuesday.

How should you retry when Shopify throttles you?

Shopify meters the GraphQL Admin API by calculated query cost, and its rate-limit documentation lists restore rates of 100 points per second on standard plans, 200 on Advanced Shopify, 1,000 on Shopify Plus and 2,000 on Shopify for enterprise, with a documented ceiling of 1,000 points for any single query regardless of plan. The API tells you where you stand on every response: extensions.cost.throttleStatus carries maximumAvailable, currentlyAvailable and restoreRate, alongside requestedQueryCost and actualQueryCost, and the documentation notes the difference between the requested and actual cost is refunded.

That changes what good behavior looks like. You do not have to be throttled to find out you are close to it; every response already told you. An integration that reads currentlyAvailable and slows down before it hits zero never generates a 429 in the first place, which is a materially different system from one that treats 429 as a normal part of operation.

When you are throttled anyway, Shopify states that the recommended backoff time is one second. Add two things it does not give you. Exponential growth, so a sustained throttle does not become a one-second poll. And jitter, because if twenty workers are throttled by the same burst and all back off for exactly one second, they resume simultaneously and throttle each other again: a herd you built yourself.

The structural fix underneath all of this is one queue per shop with a concurrency limit. The bucket is shared across every job your app runs against that store, so the nightly catalog sync and the customer-facing stock lookup are drawing from the same budget whether or not your code knows it. A single queue is how the budget becomes visible to both.

When is a retry loop the wrong tool?

Some work does not belong in the request-by-request lane at all. If you are updating ten thousand products, no amount of backoff tuning makes that a good use of a leaky bucket.

Bulk operations exist for this. You stage a JSONL file with stagedUploadsCreate, hand it to bulkOperationRunMutation, and Shopify runs the mutation against each line asynchronously. The documentation states that the bulkOperationRunMutation request itself is not subject to the standard rate limits, and that from API version 2026-01 each app can run up to five bulk mutation operations per shop simultaneously. Earlier versions allowed only one of each type at a time per shop.

The property that makes bulk usable in practice is per-line error handling: "each mutation in your JSONL file is validated and executed independently", errors are reported at the line level in the output file, and the operation only fails outright (status: FAILED) on critical system errors. So a bad row does not abort the run, but it also does not retry itself. You download the results, filter the failures, and decide what to do with them. That is a job somebody has to write; bulk moves the work, it does not remove it.

And bulk is not idempotent on its own. Running the same file twice is only safe if the mutation inside it is, which is exactly why productSet with a customId is the right building block for a catalog sync, and why a bulk file of blind productCreate calls is a way to duplicate your catalog at speed.

The fourth property nobody budgets for

Idempotency, retries and rate-limit awareness get you a client that behaves. They do not get you a system that is correct, because they only cover the messages you received.

Shopify is unusually direct about this. The webhook documentation states that an app "shouldn't rely on receiving data from Shopify webhooks" because "webhook delivery isn't always guaranteed", and recommends periodic reconciliation jobs that use updated_at filters to fetch everything changed since the last run. See the webhook best practices.

It is equally direct about ordering: Shopify "doesn't guarantee ordering within a topic, or across different topics for the same resource", and gives the example of a products/update arriving before the products/create it depends on. The mitigation is to use the timestamps you are given (the X-Shopify-Triggered-At header, or updated_at in the payload) and discard an event older than the state you already hold. An integration that applies events in arrival order is not processing a stream. It is shuffling one.

Reconciliation is the least glamorous item on this list and the one that catches everything the other three miss. It is also, reliably, the first thing cut from a fixed-price integration quote, which is why so many integrations are discovered to be wrong by an accountant rather than by a monitor.

What we look at first on an integration we have inherited

Six questions. They are answerable in an afternoon and they usually locate the problem before anyone opens a log.

• What is the idempotency key for each write, and where is it stored? "We check whether it exists first" is not a key. It is a race condition with extra steps.
• Does the webhook endpoint dedup on X-Shopify-Webhook-Id, and does it discard events older than the state it holds?
• Is there a retry policy that distinguishes userErrors from 5xx, and where do the non-retryable failures go?
• Does anything read throttleStatus, or does the integration only discover the limit by hitting it?
• How many processes write to this store, and do they share a queue? If the answer is "three services and no", the throttling is not mysterious.
• Is there a reconciliation job, when did it last run, and what did it find? A reconciliation job that has never found a discrepancy is usually a reconciliation job that is not comparing anything.

The honest position

None of this is advanced. It is four well-understood properties, all of them documented, none requiring a framework or a platform. What makes them rare is that every one of them costs time on a project where the demo already works. The happy path is done in a fortnight, and idempotency, backoff, dead-letter handling and reconciliation are the fortnight after that, spent on failures nobody has seen yet.

That second fortnight is the integration. The first one is a prototype that happens to be in production. If you are choosing what to cut from a scope, cut a feature; the reliability work is what determines whether the features are true.

Questions this raises

Read next

• Custom Shopify apps and back ends
• Rate limits: designing an integration that does not throttle
• Webhooks, retries and the orders you never processed