Home / Journal / CRO experiments
How to run a CRO experiment without fooling yourself
Most conversion "wins" don't survive contact with reality. A team changes a button colour, sees a good week, and declares victory — then the number quietly drifts back. The problem isn't the idea. It's the method. Here is how we run experiments so the results actually mean something.
Start with a hypothesis, not a hunch
A proper experiment begins with a sentence you can be wrong about: "Because visitors hesitate at the pricing table, adding a plain-language guarantee will increase enquiries." Notice it names a cause, a change and an expected effect. If you can't write that sentence, you're not ready to test.
Decide how big a change matters
Before you launch, agree on the smallest improvement worth having. Chasing a 0.2% lift on low traffic is a good way to spend months learning nothing. Knowing your target keeps you honest about whether a test is even worth running.
Give it enough traffic and time
Two things kill CRO results: stopping too early and peeking too often. Let the test run its planned duration across full weeks so weekday and weekend behaviour both count. Resist the urge to call it the moment it looks good.
Read the result like a sceptic
- Did it reach the sample size you planned for?
- Is the difference big enough to matter, not just statistically visible?
- Would you bet your own money it will hold next month?
If the answer to any of those is shaky, treat the result as "interesting" rather than "proven" and run it again.
Bank the win, learn from the loss
When something works, roll it out and document why. When it doesn't, you've still learned something real about your customers — which is worth more than a lucky guess. Do this for a few cycles and the compounding effect starts to show up in the only chart that matters: revenue.