In-App Message A/B Test

You can create multiple message variants for an in-app message campaign, test each variant, and select the best-performing message for display.

Curious about real-world use cases for In-App Message A/B Tests? Read the In-App Message A/B Test Guide

Step 1. Writing A/B Test Messages

Click [+ A/B Test] in the message composition area to prepare the experiment and create multiple message variants.

• You can add message variants and delete them. Write a new message or copy the current variant to create multiple messages and run an A/B Test.

• You can add a Control Group. Click [Add Control Group] to run an A/B Test between users who have not seen the in-app message and those who have.

Step 2. A/B Test Settings

• The experiment measures the performance of each message based on a 'success metric', and maintains traffic distribution every 1/24/48 hours after the message is displayed, adjusting traffic distribution based on the performance of each message.

  • Click-through rate counts only the click-through rate of links entered in the in-app message settings. (The 'Close' button is not included in click-through rate aggregation.)

  • Traffic distribution uses a method called MAB. You can learn more from the MAB Test introduction document.

• Success Metric

  • Message click-through rate: The ratio of users who clicked a button or image in the in-app message relative to total exposures.

  • When selecting directly, only metrics with a calculation type of count and a denominator of test exposure can be registered.

    

Step 3. Running the A/B Test

• Start: Even when an experiment is configured, click the [Start] button to start the experiment.

• Running the experiment

  • Once the experiment is running, you can view the exposure count, conversion count, conversion rate, and probability of being the best performer for each group. ('Conversion' means a click.)

  • In the chart below, you can view detailed information such as the conversion rate trend, traffic distribution, and cumulative distribution for each group.

• Pausing and ending the experiment

  • After reviewing the results, if you want to display only the best-performing message, click [End Test] to select the best-performing message (Winner).

    • Clicking [End Test] only allows the best-performing message (e.g., Variant B) to continue being displayed.

  • Clicking [Pause] will pause the display of that message, and the experiment will also be paused simultaneously.

• End

  • After ending the experiment, click the [End] button to end the in-app message campaign.