A/B Test Settings

A/B Test Settings can be accessed from all environments.

However, settings related to A/B Test groups are only available in the Production Environment. Test Device registration is available in all environments.

Overview

In the Production Environment, this is the first tab when entering the A/B Test detail view. Here you can manage the test groups configured for the A/B Test, as well as manage Test Devices.

A/B Test Settings example screen

A/B Test Hypothesis

Enter or edit the hypothesis you want to validate through the A/B Test.

A/B Test Groups

Edit Groups

Group editing is only possible before the A/B Test starts.

You can create up to 10 groups (Group J).

Click the Edit button on the right to open a dialog for group editing. Here you can add or remove groups (Treatments) in the A/B Test. You can also change the description of a group.

1. Add: Click the Add New Group button at the bottom of the dialog to add a new group.

2. Delete: Click the trash icon to the right of the group you want to delete. Note that the trash icon only appears when there are 3 or more groups.

3. Apply changes: After completing group additions/deletions, click the Save button in the lower right corner of the dialog to apply the changes.

Excluding a Specific Group from a Running A/B Test

If the A/B Test is running and there are 3 or more groups in the test, you can exclude a specific group using the dialog that appears by clicking the Edit button on the right. Users assigned to the excluded group will subsequently see the settings (features, screens, logic, etc.) corresponding to Group A.

The exclusion feature is used when it is determined that testing for a specific group is unnecessary. For example, if a specific group's results are noticeably inferior compared to other groups, you can use the Exclude from Test feature to prevent users from being exposed to that group any further.

Test Device Registration

You can register user identifiers per test group to force specific users to be exposed to a particular test group. Those users will be classified as Override users during Variation Distribution.

Use Cases

During a running A/B Test in the Production Environment, this can be used to force-assign specific users to a specific test group. The most useful application is using it for QA (Quality Assurance) purposes before starting an A/B Test in the Production Environment.

Typical use cases include:

1. Verifying that the UI, features, logic, etc. subject to the A/B Test are correctly applied per test group

2. Validating that other features in the service unrelated to the A/B Test work without issues

Register a Test Device instead of starting the experiment if you only want to validate with specific users

If Traffic Allocation in the Production Environment is set to a value greater than 0%, even in the Development Environment, unregistered users may be assigned to a group other than Group A.

This is because the Production Environment's Traffic Allocation value also applies to the Development Environment. When a Traffic Allocation value exists, Hackle's Variation Distribution logic may expose unregistered users to a treatment group.

Therefore, to prevent users other than the registered ones from being exposed to treatment groups, do not start the experiment and force-distribute using Test Devices.

Configuration

Test Device registration information is environment-independent, meaning settings in the Development Environment and Production Environment are not shared. In other words, you must configure them separately per environment even for the same A/B Test. You can find the environment information in the top area of the Dashboard.

1. Enter User Identifier Values

Click the Register Test Device button to open the configuration dialog. You can directly register users you want for each group.

1. Direct input: Register a user using their "User Identifier value." If you are unsure what a User Identifier is, check User Identifier & Properties.

2. User group: Select and apply a user group registered under [User Group Management] in the left menu area.

Can't see the user group (Test Device) you created?

Check the User Identifier basis for both the A/B Test and the Test Device.

The A/B Test's identifier basis and the Test Device's identifier basis must match in order to register the user group for the A/B Test.

How to delete a user

• To delete a user already registered, click the x button to the right of the User Identifier value.

• To delete all registered users for a test group, click the x button to the right of the input field for that test group.

2. Save

Click the Save button in the lower right corner of the dialog to save the changes.

3. Verify

After completing the integration with the Hackle platform, run the A/B Test target using the identifier of each user saved to each group, and verify that the intended content appears for each test group.

Are the results not appearing as intended?

1. For the JavaScript SDK, settings are cached for user latency. Therefore, the assignment may not appear immediately. In this case, use Chrome's hard refresh feature to see the results right away. If this still does not work, open Developer Tools, right-click the refresh button, and click "Empty Cache and Hard Reload" to try again.

2. For mobile apps, you need to fully close and restart the app to see the results.