Mutually Exclusive Settings can be created and managed separately in the Production Environment and the Development Environment.
In general, even when multiple tests run simultaneously, each test is distributed independently. For example, if two tests, Test 1 and Test 2, run simultaneously, assignments such as (Test 1: A, Test 2: A) and (Test 1: A, Test 2: B) occur — since the distribution of individual experiments is distributed fairly to other experiments, there is no problem. (See Group 1 in the image below)
However, if Test 1 and Test 2 affect each other in terms of UX or system, or cannot run simultaneously, or if you want to rigorously measure the individual effect of a specific test, you need to allocate traffic exclusively for each test.
That is, this feature is recommended when you need to configure traffic between specific experiments to not overlap.
For example, if you run Test 1 (changing the search button color) and Test 3 (changing the search button position) simultaneously and want to check the click rate of the search button across both tests, it is difficult to measure the individual effect of each test.
In such cases, you can add A/B Tests to a Mutually Exclusive Group to accurately measure the impact of each test.
In the Dashboard's A/B Test menu, go to Mutually Exclusive Settings in the left tab, and click the + Configure Mutually Exclusive Settings button to create new mutually exclusive settings.
Update Mutually Exclusive Settings can only be used with SDK version 2.0.0 or higher. If you are using an older SDK version, you must update. Clicking this button takes you to a page with update instructions.
Mutually exclusive settings name: Required. A default name is provided. Enter the name of the mutually exclusive group you want to create.
Mutually exclusive settings description: Optional. A description helps other team members understand the mutually exclusive settings.
Mutually exclusive settings target: Refers to all users.
Click the Add Mutually Exclusive Group button to create up to 10 independent mutually exclusive groups.\
Group name: Required. Enter the name of the mutually exclusive group.
Group traffic setting: Traffic refers to the feature that configures how many users out of all users should be exposed to the A/B Test. You can set traffic within the available allocation. (Note: the minimum traffic for a mutually exclusive group is 10%.)
Add A/B Test: You can add the A/B Tests to run in a mutually exclusive manner. However, tests added to the same group are not mutually exclusive with each other.
You can add existing A/B Tests in Draft status to a mutually exclusive group.
Click the trash can icon to remove an added A/B Test at any time.
However, if a running A/B Test is removed from the mutually exclusive group, that test is exposed to all users. For a running experiment, users beyond the configured traffic will be exposed, which will affect the experiment results.
This shows multiple mutually exclusive groups added. The three groups in the image are mutually exclusive with each other, and users assigned to one group are no longer exposed to other tests. (e.g., a user in the Design Team's group is not exposed to the Marketing Team's [New Signup Exposure Timing Test].)
Click [Configure] to modify the group settings.
Click the icon to the right of [Configure] to delete the group. (Note: groups with A/B Tests added cannot be deleted.)
You can check the mutually exclusive settings information for that test in the upper area of the A/B Test detail screen. Click the settings name to view the details of the mutually exclusive group the test has been added to.
When an A/B Test is concluded, the configured Winner group is exposed only to some users as set in the mutually exclusive settings. By removing the A/B Test from the mutually exclusive group, you can expose the changes to all users.
After concluding the test, click the mutually exclusive settings name at the top to go directly to the detail screen. To expose the A/B Test results to all users, the test must be removed from the mutually exclusive group.
[Note]
Experiments that have already started (Running, Paused, etc. status) cannot be configured for mutually exclusive settings. You can configure mutually exclusive settings for a version-updated or Draft experiment.
Mutually exclusive settings can be applied regardless of experiment type (MAB, AB, URL).
Mutually Exclusive Group target refers to whether the user is within the specific experiment exposure target.
Example: Mutually Exclusive Group target O means a user included in group 1's experiment group called experiment 1. Mutually Exclusive Group target X means a user included in group 2's experiment group called experiment 1.
To expose the A/B Test to all users after conclusion, remove it from the mutually exclusive group.
By removing the A/B Test from the mutually exclusive group, you can apply the changes to all users.
Last updated