Product Management playbook/Module 6 · Evidence

Day 54 / 100 · Evidence · When not to A/B test

Not every question needs an experiment.

A/B tests are powerful but narrow. Sometimes another method gets you a better answer, faster and more safely.

Skip The A/B Test When…

There's too little traffic

You'd wait months for a trustworthy answer.

It's unfair or unsafe

Different prices or protections for similar customers.

The fix is obvious

A broken button doesn't need a test. Fix it.

Effects take months

Trust and brand don't show up in a two-week test.

Right question, right method.