Create & Configure A/B Test

A/B Test Process

To run an A/B Test, you generally go through the following stages:

1. Prepare the A/B Test

2. Run the A/B Test

3. Conclude the A/B Test

This document walks you through each detailed step of the above process in order.

Prepare the A/B Test

Decide what data to collect.

1. Select the A/B Test subject

If any change could affect your existing users, it is worth considering as a candidate for an A/B Test. Typical examples include:

• Logic or algorithm changes

• UI/UX changes

• Marketing copy or banner changes

2. Create goals and hypotheses

To reach a conclusion after an A/B Test about which option — the existing version or the new version — is better, you need a criterion for judgment. This could be more user signups, more button clicks, reduced loading times, and many other possibilities. This criterion is called a goal.

If you find it difficult to define a goal, it helps to create a hypothesis. Think about why the new version would be better than the existing version, or what the reason for the change was, and then define a hypothesis.

Run the A/B Test

To start an A/B Test, you first need to create a new A/B Test.

Create an A/B Test, run it, and collect data.

1. Create an event

An event represents user behavior data such as click logs, content creation logs, and purchase logs. Events are used to calculate the goals that measure A/B Test results (e.g., click-through rate). Refer to the document below for how to create an event.

You do not need to create events first

You can also create them during the goal setup process in 3. Register goals.

Create Event

2. Create an A/B Test

When creating an A/B Test, you can configure the current version and the new versions you want to compare. For more details, refer to the document below.

Create a New A/B Test

3. Register goals

As described earlier, goals are the means to measure the performance of an A/B Test. You can register multiple goals to measure, and previously registered goals can be easily re-added. For more details, refer to the document below.

Set Metrics

4. Integrate Hackle platform

Platform integration is required to implement the A/B Test in code and measure goals. The following document lets you explore the features and supported languages of the SDK provided by Hackle, and check the integration guide for each language.

space

5. Start the A/B Test

You are one step away from starting the A/B Test: setting the traffic percentage. Hackle's Traffic Allocation feature lets you configure what percentage of total users should be exposed to the A/B Test.

Traffic Allocation

You are now ready to start the A/B Test. You can find out how to start and stop an A/B Test in the document below.

A/B Test Status

Run the test for at least one week

We recommend running the test for at least 1 week to prevent skewed results due to day-of-week effects.

Conclude the A/B Test

Analyze the collected data and draw conclusions about the results.

1. Interpret A/B Test results

The results of an A/B Test can be found in the Data Analytics tab on the A/B Test detail page. You can learn about the data available from the results and how to interpret them in the document below.

Data Analysis

If you need raw data

If you are on a paid plan (Pro plan or higher), you can export data for variation distribution and events. For more details on data export, refer to Data Export.

2. Conclude the test

Once you have the results, conclude the A/B Test. When you select a Winner group upon concluding the test, all users will see the Winner group's result.

After concluding the experiment, be sure to remove all code related to that A/B Test.