Retention
The Data Analysis tab is only accessible in the Production Environment.
About Retention
Hackle's A/B Test Retention feature allows you to view, via a retention chart, whether users exposed to the test return to the product — broken down by test group — while the A/B Test is running. In addition, you can check the improvement rate of the treatment (Group B) compared to the control (Group A) and the p-value, which can be used to make data-driven decisions based on test results.
Viewing the Retention Analysis Chart
Result data can be viewed within 1 hour of starting the A/B Test or creating a new retention, and data is updated daily thereafter. Data collection continues for up to 30 days after the A/B Test starts and stops when the A/B Test is terminated.
Time unit: You can view the configured retention data by day or by week.
Day0: The number and proportion of users who triggered the goal event on the same day (Day0) they triggered the start event.
Week0: The number and proportion of users who triggered the goal event in the same week (Week0) they triggered the start event.
Exclude duplicate users: Use this setting to choose how to handle a user who triggers the start event across multiple periods (days or weeks).
When excluded: Only counted for the period (day or week) in which the first start event occurred.
When not excluded: Counted for every period (day or week) in which the start event occurred.
User Count: The number of unique users in that test group who triggered the configured start event.
Retention: The retention per group and user count per the configured time unit. Retention per group is the proportion of users who triggered the start event that also triggered the goal event.
User Count: The number of users in that test group who triggered the goal event in each period.
Improvement Rate vs. Group A: The relative difference between the control (Group A) retention and the treatment (Group B) retention. The absolute value is shown in parentheses. For Day1, since 17 more users in the treatment (Group B) were retained compared to the control (Group A), retention increased by 98.4%. A + or - sign appears before the number depending on whether retention increased or decreased, and when the change is statistically significant, it is displayed in green or red respectively.
p-value: A value that expresses the reliability of A/B Test results. A lower p-value indicates higher reliability. When p-value is less than 0.05, the improvement rate vs. Group A is considered statistically significant. When p-value is 0.05 or greater, it is considered statistically insignificant.
You can identify data trends through a visualized graph. Hovering your mouse over the graph shows the retention per group, improvement rate vs. Group A, and p-value for that date.
Last updated