For the complete documentation index, see llms.txt. This page is also available as Markdown.

Writing an A/B Test Planning Document

When designing an A/B Test, try using this One-Pager template.

It helps you plan and prepare more thoroughly and extract valuable lessons learned.

1

Experiment Preparation

Before starting the experiment, lay out the problem and the hypothesis as a single flow.

  • Problem: Describe the problem users are running into in one sentence.

  • Supporting data: Collect the qualitative and quantitative data that backs up the problem.

  • Hypothesis: Describe what you will change and what result you expect.

  • Duration: Set how long the experiment will run.

  • User distribution timing: Define when users are distributed into the variations.

Example

  • Problem: Product list views are high, but the product click-through rate is low.

  • Hypothesis: Displaying the discount rate on the product list will increase the click-through rate.

  • Duration: Approximately 2 weeks

  • User distribution timing: When a user views the product list

2

Experiment Execution

Define the variations and how you will measure them.

  • Changes: Describe the experience applied to Group A and Group B.

  • Experiment target: Define which users are included in the experiment.

  • Success Metric: Choose the key metric that determines whether the experiment succeeded.

  • Supporting metrics (optional): Add metrics to monitor alongside the key metric.

  • Guardrail Metrics (optional): Choose the metrics that must be maintained during the experiment.

  • Experiment URL (optional): Add the URL where the experiment can be verified.

Example

  • Changes: Group A keeps the current experience, and Group B gets the change applied.

  • Experiment target: All users

  • Success Metric: Product click-through rate — click_product / view_product_list

  • Supporting metrics: Add-to-cart rate, purchase conversion rate

  • Guardrail Metrics: Order cancellation rate, Retention

3

Post-Experiment

Interpret the results and connect them to your next hypothesis.

  • Result summary: Record the winning group and how the key metric changed.

  • Result interpretation (optional): Summarize whether the results support your hypothesis.

  • Next hypothesis (optional): Write a follow-up hypothesis based on what you learned.

Example

  • Result summary: Winner group: Group B

  • Result interpretation: If the click-through rate increased for users who saw the discount rate, they are likely highly price-sensitive.

  • Next hypothesis: If users are price-sensitive, should we offer sorting by discount rate?

Last updated