The Black Friday runbook

Most documents called a Black Friday runbook are readiness checklists with the wrong title. They contain the code freeze dates, the app audit, the load test results and a project plan — all useful, none of it usable at the moment something breaks. During an incident nobody scrolls past four pages of October decisions to find the phone number they need.

Keep them separate. Readiness is the work: what to change, what to postpone, what to freeze, what to remove. The runbook is the operating manual for the four days, and it should be short enough that a person can hold the shape of it in their head. If it runs past a dozen pages, the scenarios have turned into essays.

The other thing it is not is a document about Shopify falling over. Checkout capacity is Shopify’s problem and they are better at it than you are. What fails at peak is what you added on top — the integrations, the apps, the discount logic, the export that runs every fifteen minutes and has never seen this volume. Write the runbook for those.

There is a structural reason peak incidents cluster where they do, and understanding it changes what you watch.

Shopify’s documentation is explicit that for the Storefront API, "Buyer traffic isn't rate-limited at all". Browsing and checkout are engineered to absorb the spike. The Admin API is a different system with a different contract: it uses a leaky bucket with a bucket size and restore rate set by the store’s plan, documented as 100 points per second on standard plans, 200 on Advanced, 1000 on Shopify Plus, and 2000 on enterprise. Short bursts are fine provided average cost stays under the restore rate.

So the shape of a peak failure is this: the storefront takes ten times the usual traffic without complaint, ten times the usual orders are created, and every system that reads or writes those orders through the Admin API is suddenly running at ten times its normal rate against a limit that has not moved. The export job that comfortably kept up in October starts falling behind at 9pm on Thursday, and nobody notices until the queue is four hours deep.

That is why the runbook’s most important dashboard is not page speed. It is the age of the oldest unprocessed order, and it belongs on the front page with a number beside it.

The single biggest source of delay during an incident is not diagnosis. It is someone at 2am wondering whether this is bad enough to wake a colleague. Take that decision away from them by writing it down in October, when everyone is calm and nobody is embarrassed.

Each threshold is a number, a consequence and a name. "If orders have not exported for thirty minutes, ring Priya — she does not mind." "If checkout completion drops more than X below the baseline on the front page for two consecutive fifteen-minute windows, disable the most recently added checkout extension and tell the channel." "If the fulfilment backlog passes a day’s capacity, stop the paid spend on the campaign driving it."

Set the numbers from your own baseline rather than from anything published. There is no industry threshold that knows how deep your fulfilment queue can safely get, and a borrowed number is worse than one you argued about internally for ten minutes.

Write the permission alongside it. An on-call engineer who has been told in advance that they may disable any app without asking will do it in ninety seconds. One who has not will spend forty minutes trying to reach a director.

"Revert the theme" is not a procedure. The procedure is: the name of the known-good theme version, where to find it, who has the permission to publish, roughly how long publishing takes, and what breaks in the meantime — a live cart, a running campaign pointing at a section that will disappear.

The same applies everywhere. For each app you might disable, note whether disabling removes its script from the theme immediately or leaves a block behind that must be removed by hand. For each integration, note whether switching to manual means turning something off, or turning something else on. For a discount, note whether deactivating it affects carts that already have it applied.

The test for every rollback entry is whether it can be executed by someone reading it for the first time. If a step assumes knowledge that lives in one person’s head, either write the knowledge down or accept that this rollback is only available when that person answers their phone.

A runbook that has never been used is a document about what somebody imagined would happen. The cheapest fix is a two-hour game day in October: pick three scenarios, do not warn the on-call person which, and run them against a staging store or in a read-only walkthrough against production.

The failures are always the same and always worth finding early. A phone number is wrong. A dashboard needs a login two of the four people do not have. The 3PL’s out-of-hours contact goes to a mailbox nobody reads until Monday. The revert step needs a permission the on-call engineer does not hold. None of these are interesting problems, and every one of them costs an hour during the actual event.

Rehearsing is also the only way the document gets shorter. Nobody deletes a paragraph from a runbook they have never had to read under pressure.

Most stores do not need a staffed war room. Six people watching dashboards for four days is expensive, tiring, and tends to produce interventions nobody asked for — somebody spots an anomaly at hour thirty and changes something, and now you have an incident that peak did not cause. One person on call with a good document, a clear escalation path and explicit permission to act beats a room of people waiting to be told what to do.

We would also talk you out of adding monitoring in November. New alerting with no baseline produces noise at exactly the moment noise is most expensive, and an on-call engineer who has learned to ignore an alarm is worse off than one who never had it. If the instrumentation was not in place by October, note the gap in the runbook and work around it.

The lasting version of this is not a document at all — it is having the same people looking after the store in March as in November, so the runbook is something they maintain rather than something they write once a year. That continuity is most of what a [support and maintenance arrangement](/services/support) is actually buying, and it is why the second peak is always calmer than the first.

Book thirty minutes on the Monday after, while it is still fresh, and capture three things: what actually happened, what the runbook got wrong, and what nobody had written down. Edit the document in that meeting rather than promising to later.

That is the only mechanism by which a runbook improves. A document reviewed once a year in October is rewritten from memory by people who have forgotten the details that mattered. One edited on the Monday is a record of what was true.

• Shopify support and maintenance
• Peak season readiness for a Shopify Plus store
• Analytics you can trust during peak