PAPERBOAT / EXPERIMENT DESIGN

Marketing Experiment Sample Planner

Find out whether the change you care about is large enough for your traffic to reveal.

Estimate observations per group for an equally allocated, two-group conversion-rate test. See the absolute difference, the relative change and the assumptions behind the number before you commit to the experiment.

By Robert Urban. Free and ungated.

“We will test it for a week” leaves out the important part.

A week might contain enough eligible observations to notice a large change and nowhere near enough to distinguish a small one. The calendar alone does not tell you which situation you are in.

Start with a binary outcome: each independent unit either converts or does not within a defined window. Decide what difference would matter to the business. Then ask how much evidence a planned test would need to detect that difference under its assumptions. This planner answers that narrow sizing question. It does not analyze completed results.

If the outcome definition is still unsettled, use the Campaign Measurement Plan Builder first. Counting requests, unique inquiries and qualified opportunities as though they were the same thing will not become more useful with a larger sample.

TWO GROUPS / ONE BINARY OUTCOME / EQUAL ALLOCATION

How much evidence would the test need?

Enter a baseline rate and a different comparison rate you want the design to detect. Both groups use the same outcome definition. Rates must be between 0.01% and 99.99%.

No plan calculated.

Entries stay in this page. No account, email address or campaign data connection is required. Download your plan before leaving.

The fictional 5% to 6% decision

Suppose an offer currently converts 5% of eligible independent visitors, and the team wants to plan around a change to 6%. That is a one-percentage-point difference and a 20% relative increase. Entering “20” as the comparison conversion rate would ask a very different question.

With a two-sided 5% significance level and 80% planned power, this approximation returns 8,158 observations in each group, or 16,316 in total. At 500 eligible observations a day across both groups, accumulating that sample would take about 33 days.

That is not an instruction to stop on day 33. Some conversions may need time to mature. Traffic may vary by day, promotions may overlap, and the experiment may need to cover a relevant business cycle. Write the sampling and stopping plan before the results start arriving.

Now change the comparison rate to 5.5%. The smaller difference is harder to distinguish from ordinary variation, so the required sample grows. The practical question is whether that smaller improvement would change your decision enough to justify the additional evidence.

The model under the number

The calculation uses a normal approximation for two independent proportions, with equal allocation. It uses pooled variance under equality and separate variances under the proposed alternative. The sample-size inversion omits the opposite, far tail and applies no continuity correction. The answer is rounded upward.

Per-group sample ≈ [z_1−α/2 √(2p̄(1−p̄)) + z_power √(p₀(1−p₀) + p₁(1−p₁))]² / (p₁−p₀)².

Here p₀ and p₁ are proportions, p̄ is their average, α is the significance level, and the z values are standard-normal quantiles. The method follows the assumptions documented for statsmodels’ two-proportion sample-size approximation and its two-proportion power calculation.

A 5% significance level controls the planned test’s false-positive probability under its null model; it is not a 95% probability that your preferred explanation is true. Planned power is the probability of rejection under the particular alternative used to size the test, assuming the model is appropriate.

The observations have to behave like the design

Choose the independent unit. Counting the same person’s sessions as unrelated observations can overstate the amount of evidence. Assignment by store, account or household may require a clustered design.
Keep the outcome window consistent. A new inquiry that has not had time to qualify is not automatically a failed conversion.
Plan the comparisons. This is one outcome and two groups. Several variants, multiple primary outcomes or repeated significance checks require an appropriate analysis plan.
Check sparse outcomes. The tool flags an expected success or failure count below ten in either group. Even above that threshold, a normal approximation does not certify the design.
Separate detectability from value. A measurable change can be too small to justify implementation. A commercially valuable change can be too subtle for the available traffic.

Random assignment, consistent measurement and a suitable analysis plan are work the calculator cannot do for you. If the experiment uses a different design, take the assumptions and the question to the person responsible for its statistical design.

Questions before the first assignment

Does this tool tell me whether my experiment won?

No. It plans sample size from assumed conversion rates. It does not accept observed test results or calculate a winner.

Why does a smaller improvement need a larger sample?

A smaller difference is harder to distinguish from variation. Holding the other assumptions fixed, reducing the difference increases the sample required by this approximation.

Can I use all website traffic as the daily input?

Use only eligible independent units entering this experiment, counted across both groups. The optional estimate divides the total required sample by that daily input; it does not include outcome delays or prescribe a stopping date.

Bring the decision with the data.

If your team needs help deciding what to measure, reconciling reports or connecting the evidence to a marketing decision, explore marketing analytics and reporting advisory. A clear question and a record of the assumptions are useful starting points.