PLAYBOOK
The CRO testing playbook
How to find a problem worth testing, run the test fairly and keep what you learn.
THE SHORT ANSWER
Start from a problem in your data, write one hypothesis, check you have the traffic, fix the length, then decide and record. Without the traffic, make the change and measure it as a change.
IN SHORT
- A test answers a question about one change. It cannot rescue a page with no clear problem behind it, so step one is the problem, not the idea.
- Many stores do not have the traffic to test small changes. The honest alternative is to make the change, measure before and after, and say it is not a controlled test.
- Decide the metric and the length before you start, and do not stop early because the line looks good. Early stops are how false wins happen.
- Results do not always carry between the US, UK and Australia, because price display, tax and delivery expectations differ. Read each market on its own.
Step 1: find the problem, not the idea
Ideas are cheap. A drop-off in your own data is not.
- Map the path from landing page to order and note where the most visitors leave, on phone and desktop separately.
- Read support tickets and returns reasons for what confuses or surprises customers.
- Watch session recordings of the drop-off page, to see what people actually do.
- Split everything by market, because the problem in the US may not be the one in the UK.
- Pick the one problem that has the most visitors behind it.
Step 2: write a hypothesis
One sentence you can be proved wrong on.
- Write it as: because we saw X, changing Y should change Z for this group.
- Choose one primary metric, such as orders per session, before the test starts.
- Say what result would make you drop the change.
- Change one thing, or one tightly related group of things, so the result can be read.
Step 3: decide whether it can be tested
The most useful answer is sometimes no.
| Run a controlled testUSUALLY THIS | Make the change and watch it | |
|---|---|---|
| Right when | Enough traffic for the sample size you calculated, in the time you can wait | Traffic too small, or the change fixes something plainly broken |
| What you learn | A fair comparison of two versions at once | A before-and-after, which season and campaigns can distort |
| Say so | The length, the metric and the sample size | That it is not a controlled test |
Step 4: run it without peeking
The setup does the work. Then leave it alone.
- Fix the length before you start, covering whole weeks, and write it down.
- Do not change the pages, prices or campaigns in the test while it runs, or note it if you must.
- Check the versions render correctly on phones, and that the tracking fires on both.
- Do not stop early because one version is ahead.
- Avoid starting a test across a sale or a holiday unless that is what you want to learn about.
Step 5: decide and record
A test nobody remembers is a test you will run again.
- Ship, drop or retest, and write down which and why.
- Record the hypothesis, the length, the metric, the result and the market in one shared place.
- Note what you would do differently, such as a larger change or a different page.
- Check the winner against a second metric, such as returns or order value, before rolling it out.
- Feed the result into the next problem you pick in step one.
Read the guides
The blog posts behind each stage, for when you want the reasoning.
Where do you find the problem?
The product page audit: twenty things worth checking
A product page has one job and about twenty ways to quietly fail at it. This is the list we work through on a store audit.
Mobile conversion: the gap between your traffic and your revenue
Most of your sessions are phones and most of your revenue probably is not. Some of that gap is measurement, some of it is intent, and a stubborn part of it is your theme.
Attribution models: what to believe after the cookie
Every platform reports a different number and each is internally consistent. Pick one as the ledger, treat the rest as opinion, and test the things that decide budget.
What is worth testing first?
Cart drawer or cart page: the test nobody runs properly
Every case study you have read about this measured the wrong number on somebody else’s store. This is what the choice turns on.
Shop Pay, express checkout and the conversion you are leaving
Does Shop Pay express checkout raise conversion? Likely for returning shoppers, but measure your own funnel rather than trust published uplift figures.
Delivery promise: the date that changes conversion
Showing a delivery date helps because it removes a question, not because dates are persuasive. If your date is slow, showing it is still the right call, and it will cost you some orders.
Search and page content
The search box is a merchandising surface
Somebody curates your home page every week. Nobody has ever looked at what the search box returns for your best-selling product.
Image and video on product pages: the cost and the payoff
Video helps when it answers a question a photograph cannot. For most catalogs that is a small number of products, and the fourth photo would have done more.
Personalization without a personalization platform
Shopify already gives you segments, recommendations and the logged-in customer in Liquid. The wall is the anonymous first-time visitor, and most platforms are sold on the wrong side of it.
Questions about CRO testing
What is a CRO testing process for a Shopify store?
Find where visitors leave using your own analytics, write one hypothesis about why, check there is enough traffic to test it, run it for a fixed period without changes, then decide and record the result. The five stages above follow that order.
How much traffic do I need to run an A/B test?
It depends on your conversion rate and on the size of the change you want to detect, and small changes need far more traffic than large ones. A sample-size calculator takes those two inputs. If the answer is longer than you can wait, test a bigger change, or make the change and track it without calling it a test.
How long should a test run?
Long enough to cover your normal weekly cycle, and for as long as the sample-size calculation says. Pick the length before you start and keep to it. Stopping the day the line crosses is the commonest cause of a result that disappears later.
Which pages should I test first?
The ones where your own data shows the largest drop-off and that get the most visitors, often product pages and the cart or checkout. A page with high traffic and a clear drop gives you the most to learn and the most to gain.
Is a test result from one market true for the others?
Not necessarily. Tax-inclusive prices in the UK and Australia, shipping expectations and payment preferences differ from the US. Read results per market, and retest before assuming a win carries over.
How do I know a CRO project is worth the cost?
Put your own sessions, conversion rate, order value, margin and an assumed lift into the payback calculator. It returns the months to repay and the lift you would need to repay in twelve months. We do not publish a typical lift because it would not be yours.