Push Message A/B Test

If you want to find the most effective push message campaign creative, try using A/B testing.

You can compare and analyze which message drives higher user engagement under the same conditions to optimize your campaign.

Step 1. Write A/B Test Messages

In the message composition section, click [+ A/B Test] to prepare an experiment and create multiple messages.

You can write a new message or copy the currently written message to create multiple messages and run an A/B test. You can add up to 10 messages.

Success Metric

In an A/B test, the performance of each message is measured based on the 'Success Metric'. Push Message A/B Tests support message click-through rate by default.

• Message Click-Through Rate: The ratio of users who clicked the message to the total number of push messages delivered

• Custom Selection: Event metrics (e.g., click conversion rate per user, purchase conversion rate, average order value, etc.)

Step 2. Configure A/B Test Settings

The A/B test begins when the push message sending starts. Detailed A/B test settings can be continued in Send Settings.

Send settings button location

2-1. Traffic Settings

Set the traffic proportion of users to whom the A/B test message will be sent.

1. You can run an initial test on a portion of traffic, then send the best-performing message to the remaining users to optimize campaign results.

2. You can run a test on all traffic to check preferences for creative assets.

2-2. Test Schedule

• Start: The A/B test begins when the push message sending starts.

• End: The A/B test ends when the push message ends.

  • Manual End: You can manually end the test after reviewing the message sending results.

  • Scheduled End: You can pre-schedule the test end date.

Scheduled test end

When scheduling a test end, you can configure whether to send the winning message (Winner group) to the remaining traffic users after the test ends.

• Winner Group Selection Criteria

  • The group with superior success metrics is set as the Winner group and the test ends.

  • If configured to send messages to remaining users, the Winner group's message will be sent to those users.

  • For recurring messages, the Winner group's message will be sent to all users.

Step 3. Start A/B Test

The A/B test begins when the push message sending starts. Test results can be checked within up to 2 hours after message sending begins.

Active push message

❗️ Deactivating the campaign will end the ongoing A/B test.

After deactivating the campaign, you can edit the push message content. When the message is sent again, a new version of the A/B test begins.

Results from the previous version are saved separately, and you can view data by version.

The ongoing A/B test will end

View previous version data

Step 4. End A/B Test

The A/B test and push message campaign statuses can be set independently.

• After reviewing performance, if you want to send only the best-performing message, click [End Test] to select the winning message (Winner).

• For recurring messages, if the A/B test is ended while the campaign is still active, messages will continue to be sent.

1. End Test Click the End Test button directly to end the ongoing A/B test.

2. Send Messages If you ran an A/B test on a portion of traffic, you can send the winning message to the remaining users.

   • Schedule-based (Immediate Send, Scheduled Send): After sending to the configured traffic, the message will be sent to the remaining traffic.

   • Schedule-based (Recurring Send), Event-based, API-based: The Winner group's message will be sent to all users.

Click the [End] button in the upper right corner to end the push message campaign.