Create MAB Test

Create MAB Test

Step 1. Enter MAB Test Information

MAB Test Information

• Name: Required. Enter the name of the MAB Test you want to run.

• Description: Optional. A description helps other team members understand what test is being run.

• Hypothesis: Optional. Enter the hypothesis you want to validate through the test.

Step 2. Configure MAB Test Groups

This step involves registering descriptions and images for each individual test group that will be the subject of the MAB Test. If there are 3 or more test groups, click the + Add New Group button to add more. You can delete a specific group using the X button on the right.

Configure Test Groups

Step 3. Complete Test Creation

Once you have finished configuring the test groups, click the Create Test button in the lower right corner.

Step 4. Set Targeting (Optional)

You can configure targeting if you want only users with specific properties, users in a specific targeting group, or users included in a specific experiment or feature flag to participate in the MAB Test.

Set Targeting

Step 5-1. Set Metrics

Click the Set Metrics button located in the upper right corner of the Data Analysis tab on the test detail page.

Set Metrics

Step 5-2. Set Metrics

After clicking the Set Metrics button, a dialog like the one shown below appears, where you can register success metrics. Additionally, you are asked to set a "Calculation Condition," which specifies whether to include only events that occur within a certain period (1~24 hours) after exposure to the experiment.

📘 The success metric for a MAB Test must have a calculation type of "Count" and a denominator based on "All Users" or "Test Exposure".

Step 6-1. Start the Test

From the MAB Test in Draft state, click the Start button located in the upper right corner.

Start Test

Step 6-1. Start Test > Set Distribution Condition

Clicking the Start button displays the popup below.

In this screen, you can set the "Distribution Condition," which determines how many hours after the MAB Test starts before traffic distribution is automatically adjusted. The purpose of setting the "Distribution Condition" is to collect a minimum sample for automatic traffic distribution during the n hours after the MAB Test starts. If you are running a MAB Test in a low-traffic area, it is recommended to set this value higher.

Step 7-1. Monitoring > MAB Distribution Status

After the MAB Test starts, you can check the traffic distribution status by hour or by day on the test detail page. In the "Traffic Distribution" tab, you can see changes in the traffic distribution ratio per group. In the "Cumulative Distribution Status" tab, you can check the cumulative experiment participation by group from the MAB Test start time.

Step 7-2. Monitoring > Success Metric

After the MAB Test starts, you can check the success metric results in the Data Analysis tab. The values recorded in the upper table represent the most recently calculated results. The lower graph shows the metric result values by hour or by day, as well as the trend of the probability of being the best (Bayesian probability).