A/B Test FAQ
Frequently asked questions about designing, running, and interpreting A/B Tests.
Frequently asked questions about designing and running A/B Tests in Hackle.
How many users do I need to run a test?
There is no fixed minimum number of users. That said, 100 or more daily users is preferable.
Can I run a test without statistical knowledge?
Yes. Hackle presents the statistical significance of results visually.
You can check results for goals such as button click rate or purchase conversion rate. Statistical knowledge helps when you analyze raw data further.
What should I keep in mind when analyzing results?
Run the test for at least 1~2 weeks. Do not end a test early based on temporary results.
At least 1 to 2 weeks are needed to reduce distortion caused by day-of-week effects. Ending a test early can lead to distorted conclusions.
Why do the values differ between A/B Test and Data Analytics for the same metric?
Because the two features aggregate at different points in time.
An A/B Test calculates the denominator and numerator events that occur after the distribution point. Data Analytics calculates simply whether an event occurred.
Unlike an A/B Test, which reflects the order of events, Data Analytics does not consider it. Therefore, values can differ even for the same metric.
What problems occur if sample sizes are not even across groups?
The variability of a small group can be reflected excessively, which may distort the p-value. This can lower the reliability of the results.
What happens if I drop some test groups while the experiment is running?
Users in the dropped group are excluded from the experiment. Those users see the Group A screen. The experiment continues with the remaining groups.
What happens if I select a Winner group in an experiment where only part of the traffic is allocated?
All traffic is distributed to the Winner group. Users who do not meet the targeting conditions are assigned to Group A.
Is data generated on Test Devices reflected in the experiment results?
Generally it is not. However, it can be reflected if the user was already exposed before the Test Device was registered.
Why include non-experiment users instead of looking only at treatment groups?
Because when the change is actually rolled out, it applies to all users.
The current behavior level of non-experiment users has a large impact on the Site-wide result.
If the experiment participation rate is low, does the Site-wide effect also become smaller?
Yes. The lower the participation rate, the larger the share of existing metrics from non-experiment users.
As a result, the Site-wide effect is diluted.
Can I predict company-wide rollout from an experiment targeting only a specific OS or cohort?
Yes. Even an iOS experiment is calculated from the perspective of all users, including Android.
However, if behavior differs greatly by OS or by cohort, the prediction accuracy can vary.
Are there metrics that cannot be selected as Global Impact metrics?
Event metrics whose denominator is the total number of exposures are not supported.
Time metrics, funnel metrics, and retention metrics are also not supported.
My Global Impact metric results are not calculated.
Unlike other metrics, Global Impact metrics are calculated once per day.
I need the A/B Test raw data.
Data extraction is available on paid plans of Enterprise or higher.
You can extract variation distribution data and event data from the Dashboard.
Select a test
Go to the detail screen of the A/B Test whose raw data you need.
Request the data
Select ... to the right of the title → Extract Data.
Download the CSV file
Download the CSV file from the link sent to your registered email.
Last updated