Product Management playbook/Module 6 · Evidence

Day 57 / 100 · Evidence · Survivorship and selection bias

Who’s missing from your data?

The customers you can easily hear from are rarely typical. The ones who leave, or never start, are often the most important.

Survivorship bias

Only Studying Who Stayed

  • Churned customers don't answer your survey
  • Winning features get studied; failures are forgotten
  • “Our best customers love it” hides who left

Selection bias

Hearing From The Wrong Few

  • The loudest customers set the agenda
  • Beta volunteers are unusually keen
  • Survey responders differ from everyone else

Source: Brilliant ↗

Listen for the silence.