Create & Configure A/B Test
A/B Test Process
To run an A/B Test, you generally go through the following stages:
Prepare the A/B Test
Run the A/B Test
Conclude the A/B Test
This document walks you through each detailed step of the above process in order.
Prepare the A/B Test
Decide what data to collect.
1. Select the A/B Test subject
If any change could affect your existing users, it is worth considering as a candidate for an A/B Test. Typical examples include:
Logic or algorithm changes
UI/UX changes
Marketing copy or banner changes
2. Create goals and hypotheses
To reach a conclusion after an A/B Test about which option — the existing version or the new version — is better, you need a criterion for judgment. This could be more user signups, more button clicks, reduced loading times, and many other possibilities. This criterion is called a goal.
If you find it difficult to define a goal, it helps to create a hypothesis. Think about why the new version would be better than the existing version, or what the reason for the change was, and then define a hypothesis.
Run the A/B Test
To start an A/B Test, you first need to create a new A/B Test.
Create an A/B Test, run it, and collect data.
1. Create an event
An event represents user behavior data such as click logs, content creation logs, and purchase logs. Events are used to calculate the goals that measure A/B Test results (e.g., click-through rate). Refer to the document below for how to create an event.
You do not need to create events first
You can also create them during the goal setup process in 3. Register goals.
2. Create an A/B Test
When creating an A/B Test, you can configure the current version and the new versions you want to compare. For more details, refer to the document below.
Create a New A/B Test3. Register goals
As described earlier, goals are the means to measure the performance of an A/B Test. You can register multiple goals to measure, and previously registered goals can be easily re-added. For more details, refer to the document below.
Set Metrics4. Integrate Hackle platform
Platform integration is required to implement the A/B Test in code and measure goals. The following document lets you explore the features and supported languages of the SDK provided by Hackle, and check the integration guide for each language.
space5. Start the A/B Test
You are one step away from starting the A/B Test: setting the traffic percentage. Hackle's Traffic Allocation feature lets you configure what percentage of total users should be exposed to the A/B Test.
Traffic AllocationYou are now ready to start the A/B Test. You can find out how to start and stop an A/B Test in the document below.
A/B Test StatusRun the test for at least one week
We recommend running the test for at least 1 week to prevent skewed results due to day-of-week effects.
Conclude the A/B Test
Analyze the collected data and draw conclusions about the results.
1. Interpret A/B Test results
The results of an A/B Test can be found in the Data Analytics tab on the A/B Test detail page.
You can learn about the data available from the results and how to interpret them in the document below.
If you need raw data
If you are on a paid plan (Pro plan or higher), you can export data for variation distribution and events. For more details on data export, refer to Data Export.
2. Conclude the test
Once you have the results, conclude the A/B Test. When you select a Winner group upon concluding the test, all users will see the Winner group's result.
After concluding the experiment, be sure to remove all code related to that A/B Test.
Last updated