How to A/B Test Cold Emails (Without Fooling Yourself)

Almost every cold email guide tells you to A/B test your subject lines. Almost none mention that at the volume most small senders have, a subject line test cannot produce a trustworthy answer. You will still get a number, and it will still look like a winner. It will just be noise wearing a suit.

This is the honest version: what is worth testing first, how many emails a result actually needs to mean something, how to spot a false win, and what to do instead when the math says you cannot run a real test. The arithmetic is shown in full so you can redo it with your own numbers.

Why cold email is a hard place to run experiments

A/B testing works well when the thing you measure happens often. Ecommerce checkout tests work because a few percent of thousands of visitors convert every day. Cold email is the opposite: the outcome you care about — a reply — happens to roughly 2% to 8% of the people you contact, and most small senders touch a few hundred people a month, not a few hundred thousand. Rare events plus small volume equals wide uncertainty. Everything else in this article follows from that one sentence.

The metric you should test on

Test on replies, and where you can, on meetings booked. Not opens.

Open tracking is broken. Privacy proxies fetch images before a human ever sees the message, security scanners click links to check them, and some clients block pixels entirely. The result is inflated in one direction by machine prefetching and deflated in another by blocked pixels, with no way for you to separate the two. A subject line "win" measured in opens is often just a variant that landed in more mailboxes with aggressive image proxies.

Replies are messier to count but they are real: a human read the message and typed something back. Meetings are better still, but rarer, which makes the sample size problem worse — so optimise on replies and sanity-check on meetings.

The arithmetic: how many emails do you need?

Here is the rule of thumb, and then where it comes from.

Sends needed per variant ≈ 16 × p × (1 − p) ÷ d²

where p is your average reply rate across both variants and d is the absolute improvement you want to detect. The 16 comes from the standard requirement of 95% confidence and 80% power: (1.96 + 0.84)² ≈ 7.85, doubled to about 16 because you are comparing two groups rather than one group against a fixed number. You do not have to trust the formula — you just have to run it on your own numbers.

Worked example 1: a realistic lift

Your current reply rate is 4% and you want to know whether a new variant gets you to 6% — a 50% relative improvement, a genuinely valuable change.

• Average of the two rates: p = 5%, so p × (1 − p) = 0.05 × 0.95 = 0.0475
• Difference to detect: d = 0.02, so d² = 0.0004
• 16 × 0.0475 ÷ 0.0004 = 0.76 ÷ 0.0004 = 1,900 sends per variant

That is 3,800 emails to answer one question about one variable. If you send 300 cold emails a month from a small office in Chicago, that test takes a year — by which point your list, your offer and the inbox providers have all changed underneath you.

Worked example 2: a modest lift

Now suppose the new version only moves you from 4% to 5%.

• p = 4.5%, so p × (1 − p) = 0.045 × 0.955 = 0.042975
• d = 0.01, so d² = 0.0001
• 16 × 0.042975 ÷ 0.0001 ≈ 6,900 sends per variant

Nearly 14,000 emails. Notice what happened: halving the effect size quadrupled the requirement, because sample size scales with the square of the difference. That is the most important fact in this article — it tells you exactly which tests a small sender can and cannot run.

Worked example 3: a big swing

Now instead of tweaking wording you change the entire offer — different promise, different reason to reply — and it takes you from 4% to 8%.

• p = 6%, so p × (1 − p) = 0.06 × 0.94 = 0.0564
• d = 0.04, so d² = 0.0016
• 16 × 0.0564 ÷ 0.0016 ≈ 560 sends per variant

Around 1,100 emails total — reachable inside a month or two. And if the change is dramatic enough to take you from 4% to 12%, the requirement drops to roughly 185 per variant, about 370 emails.

The conclusion is unavoidable: small senders can only detect big changes. At modest volume, testing "Quick question" against "Quick question, {{FirstName}}" wastes the only experimental capacity you have. Spend it on things that could plausibly double your reply rate.

Why 50 versus 50 proves nothing

The most common cold email "test" in the wild: 50 emails with version A, 50 with version B. A gets 3 replies, B gets 1. Version A wins by 3×, the founder rewrites the whole sequence, and the story gets told at a conference.

Look at the uncertainty. With 3 replies out of 50, the observed rate is 6%. The standard error is the square root of (0.06 × 0.94 ÷ 50), about 3.4 percentage points, and a 95% interval is roughly the observed rate plus or minus two standard errors: about 0% to 12.6%. For version B, 1 reply out of 50 is 2%, standard error about 2.0 points, interval roughly 0% to 5.9%.

Those ranges overlap across most of their width. The data is equally consistent with A being twice as good, with B being slightly better, and with the two being identical. Three replies versus one is exactly what chance produces when both versions have the same true rate — the same way four heads out of five flips does not prove the coin is bent.

The uncertainty at sizes that feel large

Two hundred sends per variant feels like a serious test. At a 4% reply rate that is 8 replies. The standard error is the square root of (0.04 × 0.96 ÷ 200), about 1.4 points, so the true rate behind those 8 replies sits somewhere around 1.3% to 6.7%. You cannot separate a mediocre variant from a good one inside a spread that wide.

A useful shortcut: uncertainty is driven by the number of replies, not the number of sends. Below roughly 25–30 replies per variant you are reading tea leaves; below 10 you are just reading. At a 4% reply rate, 30 replies means 750 sends per variant.

What to test first, in order

Given that you can only afford to detect big effects, test big things.

1. The list

Nothing in cold email moves reply rates like who you send to. The same message sent to a well-matched segment and a loosely-matched one can differ by a factor of three or more — an effect size a small sender can genuinely measure. Split by industry, company size, city, or whether the business has an obvious trigger (new location, hiring, recent funding). If a clean, tightly-targeted list is the bottleneck, that is a sourcing problem before it is a copywriting problem, and building one for a specific niche is what tools like JustLeadIt exist to shorten.

2. The offer

Not the wording — the offer itself. A free audit, a benchmark report, a 15-minute teardown and a paid pilot are different propositions, and they produce reply rates that differ by wide margins. Second-highest leverage, second most measurable.

3. The angle or opening premise

Why are you writing to this business specifically? A message anchored on something observable about them ("your booking page is down on mobile") plays in a different league from a generic capability pitch. Angle changes are big enough to detect.

4. The ask (CTA)

An interest-check ("worth a look?") versus a calendar link versus a specific proposed time produce genuinely different reply behaviour. Worth testing, but expect smaller effects than list or offer — often in the range that needs more volume than you have.

5. Subject lines and micro-copy

Last, and honestly optional at small volume. Subject line effects on replies are usually a point or two, which by the arithmetic above needs thousands of sends per variant. Write a plain subject that does not look like marketing, and move on.

How to run a test that is not garbage

One variable at a time

If you change the subject line, the opener and the CTA at once and replies go up, you have learned that this bundle beats that bundle — not which change did it, and nothing you can carry to the next campaign. At small volume that is sometimes an acceptable trade: a "champion versus challenger" test of two whole emails is a legitimate design. Just be honest about what it does and does not tell you.

Randomise within the same list and the same window

The classic ruined test: version A on Tuesday morning, version B on Thursday afternoon. Now you are testing day-of-week, time-of-day and copy at once and cannot separate them. Same for sending A to your best segment and B to the leftovers. Assign every contact to A or B at random inside the same list and the same sending window — alternate down the list if you have no tooling for it. It is the cheapest quality improvement available and most senders skip it.

Watch for deliverability drift

Deliverability is not a constant. A warming domain gets better inbox placement in week three than week one, and a provider can start throttling you mid-campaign. If A ran while your domain was healthy and B ran after a spam complaint spike, the copy had nothing to do with the outcome. Worse: if one variant contains a word or link pattern that trips a filter, it can land in spam for a whole class of recipients and look like weak copy when it was simply never read. Check placement per variant before believing the numbers.

Do not peek and stop

Checking daily and declaring a winner the moment one variant pulls ahead generates false wins reliably. Early in a test the ratio swings wildly — with 20 sends per side, one extra reply looks like a 100% lift. Stopping when the gap is largest is selecting for noise. Decide your sample size before you start, then look once at the end.

Count your comparisons

Test five variants at once, or read five metrics off one test, and you have given yourself five chances to find a "significant" result. At the usual 95% threshold, roughly one in twenty comparisons produces a false positive by pure chance. At five comparisons, a spurious winner is a routine outcome, not bad luck.

What small senders should do instead

Refusing to A/B test is not flying blind. It is switching to methods that work at low volume — and labelling their output honestly.

Sequential batch learning

Send one message to a batch of 100–150 contacts and record the reply rate. Change one big thing — the offer, the angle, the segment — and run the next batch. Keep a simple log of batch, change, replies.

You will not get statistical significance from this and you should not claim it. What you will get is a picture over ten or fifteen batches: some approaches consistently produce 1–2 replies per hundred and others consistently produce 6–8. Repeated directional signal across many batches is real information, even without a p-value. Just never act on a single batch.

Read the replies, not just the count

At 200 sends you have maybe eight replies — statistically useless, qualitatively gold. Are people confused about what you sell? Saying "wrong person"? Saying "we already have someone"? Each points at a different fix: clarity, targeting, timing. Negative replies are the most useful of all: "not relevant, we don't do X" tells you your list filter is wrong, which is the biggest lever you have. This is the highest information-per-email activity available to a small sender, and it requires no sample size at all.

Test lists and offers, not sentences

Because effect size is everything at low volume, spend your experimental budget on the two variables with the largest possible effects. A different vertical or a different offer can plausibly move you from 3% to 9%. A rewritten second sentence cannot.

Label your evidence honestly

Get comfortable saying "directionally, the audit offer looks better across four batches — worth continuing, not proven." That sentence is more useful to your future self than fake certainty. Teams that keep an honest log make better decisions in month six than teams that ran a dozen underpowered tests and believed each one.

A practical decision rule

Before setting up any test: estimate your current reply rate p from your last few hundred sends; decide the smallest improvement d that would actually change your behaviour; then run 16 × p × (1 − p) ÷ d². If the answer exceeds what you can send in six weeks, do not run the test — do sequential batch learning and say so. Six weeks is not arbitrary: beyond that, seasonality, list decay and deliverability drift contaminate the comparison badly enough that even a correctly-sized test stops being clean.

The short version

Cold email A/B testing is real, but it is a high-volume tool and most people using it do not have the volume. At a 4% baseline, detecting a move to 6% takes about 1,900 sends per variant; to 5%, about 6,900; to 8%, about 560. That is why small senders should test offers, lists and angles, and leave subject line optimisation to people sending tens of thousands of emails a month.

Measure replies, not opens. Randomise within one list and one window. Fix your sample size before you start and look once. And when the arithmetic says you cannot run a valid test — which will be most of the time — run sequential batches, read every reply carefully, and call your evidence what it is: directional. Honest uncertainty compounds into good judgement. Fake certainty compounds into a year of optimising the wrong sentence.