What A/B testing does and doesn't do
An A/B test splits your traffic between two versions — the original and a variant with one meaningful change — and measures which produces more conversions. Done right, it replaces "I think" with "we know."
Done wrong, it produces confident nonsense. The discipline is in valid setup and honest reading, not just in running a tool.
- Split live traffic between a control and one changed variant
- Change one meaningful thing so you know what caused the result
- Measure conversions, not clicks or vanity metrics
- Run until results are statistically trustworthy, not until they look good
- A losing test is still a win — it saves you from a bad change
What's worth testing
Not everything deserves a test. The highest-leverage elements — headline, offer, page structure, the call to action — can swing results enough to matter. Button colors rarely do.
We start with the changes big enough to move the number and skip the cosmetic tweaks that waste traffic.
- Headlines and the core promise
- The offer itself and how it's framed
- Page structure and what sits above the fold
- Call-to-action wording and placement
- Form length and which fields you ask for
- Long-form vs. short-form layout
Setup, sample size, and honesty
A test needs enough traffic and conversions to reach significance — otherwise you're reading noise. We calculate what's realistic before starting and tell you if your traffic can't support a clean test yet.
We also protect against the common mistakes: stopping early because the numbers look nice, testing five things at once, or ignoring that results need time to stabilize.
- Check up front whether your traffic can produce a valid result
- Run one clean variable at a time for interpretable results
- Let the test reach significance before calling it
- Guard against early-stopping and false positives
- Where traffic is too low, recommend direct improvements instead
More on landing pages & funnels
Frequently asked questions
How much traffic do I need to A/B test?
Enough that the difference you're testing can show up above random noise — which depends on your current conversion rate and the size of the change. Low-traffic pages can take a long time or never reach significance. We'll calculate it for your case and tell you honestly if testing isn't the right tool yet.
How long should a test run?
Until it reaches statistical significance and covers a full business cycle (usually at least a week or two, to avoid day-of-week skew) — not until it happens to look like a winner. Stopping early is the most common way to fool yourself, so we set the finish line before we start.
What should I test first?
The big levers: your headline, your offer, and your call to action. These move conversions far more than cosmetic changes like colors. We prioritize tests by potential impact so your traffic goes toward learning things that matter.