A/B Test Status

An A/B Test can have several different statuses. This document explains the available statuses and how to start, pause, stop, and archive an A/B Test from the Dashboard.

Types of A/B Test Status

The diagram below gives you an at-a-glance overview of A/B Test statuses by environment. The starting point is A/B Test creation — when you create a single A/B Test, it becomes available in both the Production Environment and the Development Environment.

A/B Test Status by Environment

Production Environment

• Provides five statuses: Draft - Running - Paused - Stopped - Archived.

• Status changes are possible.

• When changed to Stopped or Archived, the Development Environment status changes as well.

• Creating a new version of a test automatically changes the status to Draft. You cannot create a new version when the status is Draft or Archived.

Development Environment

• Provides five statuses: Draft - Running - Paused - Stopped - Archived.

• You can change the status between Draft -> Running <-> Paused, but Stopped and Archived follow the Production Environment status. In other words, when the Production Environment becomes Stopped, the Development Environment also becomes Stopped.

• Creating a new version of a test automatically changes the status to Draft. You cannot create a new version when the status is Draft or Archived.

Running tests in multiple environments

You can run an A/B Test simultaneously in both the Development Environment and Production Environment. It is also possible to run an A/B Test in only one of the two environments. However, keep in mind that stopping an A/B Test in the Production Environment will also stop it in the Development Environment.

Changing A/B Test Status

This section covers how to change the status of an A/B Test from the Dashboard. Changing to Stopped or Archived status is only available in the Production Environment.

Starting an A/B Test

Click the Start button in the upper right corner of the detail page of a Draft A/B Test. In the popup that appears, set the traffic allocation level and click the Start button again. Users will then be exposed to the A/B Test according to the traffic allocation level.

Pausing an A/B Test

Click the Pause button in the upper right corner of the detail page of a running A/B Test. All users of the paused A/B Test will see the state (screen, features, logic, etc.) corresponding to Group A.

If an issue occurs during an A/B Test or if a side effect from the A/B Test is suspected, you may consider switching to the paused state while investigating the cause. To resume, click the Start button again.

Stopping an A/B Test

Click the Stop Test button in the upper right corner of the detail page of a running or paused A/B Test. In the popup that appears, select which test group to use when stopping the A/B Test, then click the Stop Test button.

Once you select the Winner group when stopping the test, all subsequent Variation Distribution will be directed to the Winner group. For example, if you select Variation B as the Winner group, all users will thereafter encounter the logic corresponding to Variation B.

Remove A/B Test-related code after stopping

Leaving A/B Test-related code in place can cause service stability issues.

• Dependencies may become strong when writing code related to the subjects of the A/B Test.

• Unnecessary server traffic may be generated.

Therefore, always clean up the related code after stopping an A/B Test.

Creating a New Version of an A/B Test

When you want to restart an experiment while keeping the Experiment Key, you can use the new version feature. Click the Create New Version button in the upper right corner of the detail page of a Running/Paused/Stopped A/B Test. Hackle will preserve the current version's A/B Test results and create a new version. The newly created version is in Draft state and data starts being calculated from when it is changed back to Running.

1514

Archiving an A/B Test

A/B Tests that are no longer in use can be changed to Archived status. Click the Archive Test button in the upper right corner of the detail page of a Stopped A/B Test, and Hackle will preserve the results of that A/B Test.