What is Holdout group?
A holdout group is a randomly chosen set of contacts who are deliberately not sent an email so their behaviour can be compared with those who were.
Attribution tells you what customers did after an email, but not what they would have done anyway. A holdout compares the two groups, so the gap shows the effect of the message itself.
The groups must be chosen randomly and be large enough to compare. Hold them out for long enough for purchasing to show up, and keep everything else the same for both.
Holdouts have a cost, because the held-out people miss the message. They are best used selectively, for example on a flow that you suspect mostly reaches people who would buy regardless.
A worked example.
A hypothetical candle brand holds back a small random slice of new subscribers from a replenishment reminder for a month. If the reminded group reorders noticeably more than the held-out group, the flow is adding value, and if not, the brand reconsiders it.
Common mistakes.
- Choosing the held-out group by hand
- Ending the test before purchases show up
- Changing the message partway through the test
Want this handled for your store? See our retention strategy service or book a free audit.
Guides that use this.
Find out what your list is actually worth.
We review your flows, list health, authentication records and last ninety days of performance, then send back what we found.
30 minutes, no obligation, and the findings are yours to keep either way.