Writing an A/B Test Planning Document

When designing an A/B Test, try using this One-Pager template.

It helps you plan and prepare more thoroughly and extract valuable lessons learned.

1

Experiment Preparation

Before starting the experiment, lay out the problem and the hypothesis as a single flow.

• Problem: Describe the problem users are running into in one sentence.

• Supporting data: Collect the qualitative and quantitative data that backs up the problem.

• Hypothesis: Describe what you will change and what result you expect.

• Duration: Set how long the experiment will run.

• User distribution timing: Define when users are distributed into the variations.

Example

• Problem: Product list views are high, but the product click-through rate is low.

• Hypothesis: Displaying the discount rate on the product list will increase the click-through rate.

• Duration: Approximately 2 weeks

• User distribution timing: When a user views the product list

2

Experiment Execution

Define the variations and how you will measure them.

• Changes: Describe the experience applied to Group A and Group B.

• Experiment target: Define which users are included in the experiment.

• Success Metric: Choose the key metric that determines whether the experiment succeeded.

• Supporting metrics (optional): Add metrics to monitor alongside the key metric.

• Guardrail Metrics (optional): Choose the metrics that must be maintained during the experiment.

• Experiment URL (optional): Add the URL where the experiment can be verified.

Example

• Changes: Group A keeps the current experience, and Group B gets the change applied.

• Experiment target: All users

• Success Metric: Product click-through rate — click_product / view_product_list

• Supporting metrics: Add-to-cart rate, purchase conversion rate

• Guardrail Metrics: Order cancellation rate, Retention

3

Post-Experiment

Interpret the results and connect them to your next hypothesis.

• Result summary: Record the winning group and how the key metric changed.

• Result interpretation (optional): Summarize whether the results support your hypothesis.

• Next hypothesis (optional): Write a follow-up hypothesis based on what you learned.

Example

• Result summary: Winner group: Group B

• Result interpretation: If the click-through rate increased for users who saw the discount rate, they are likely highly price-sensitive.

• Next hypothesis: If users are price-sensitive, should we offer sorting by discount rate?