Outlier
The outlier exclusion option in A/B Test results is only available in the Data Analysis tab on the A/B Test detail page.
About Outliers
In statistics, an outlier is a value that is abnormally high or low compared to other values in an observed data set. Outliers can severely distort the accuracy of analysis results performed using the observed data set and can potentially lead to incorrect conclusions. This is because the values we use in experiment goal analysis are mostly averages. As is well known, among the representative values in statistics (mean, median, mode), the mean is highly influenced by outliers.
For example, imagine an experiment at an online shopping mall to improve the total order amount per visitor. Typical visitors spend an average of 100,000 KRW per week at that shopping mall. If a small number of visitors place orders 100 times higher than the average (10,000,000 KRW)? If such extreme users are included in a specific group in the experiment goal analysis, it creates distortions in the comparison between A/B groups and can lead to incorrect conclusions from the experiment.
For this reason, Hackle provides an option to view experiment results with outliers removed in the Data Analysis tab when the denominator of the experiment goal is a count or value.
Outlier Detection and Handling Method
When the outlier exclusion option is activated, Hackle calculates the mean and standard deviation per goal registered in the experiment, converts the aggregated result (X) per individual user used in goal analysis to a z-score (mean = 0, standard deviation = 1), and identifies values where the result (z-score) is greater than 3 or less than -3 as outliers. These are then excluded and the goals are re-aggregated.
z-score = ( X - mean ) / standard deviation
When goals that can have outliers excluded (see table below) are registered, the outlier exclusion option can be activated as shown in the figure below. However, even for goals where outliers can be excluded, if there are no outliers in the source data, the results with the option activated may be the same as without it.
When All Exposed is selected as the denominator, the outlier exclusion option is applied to the following 2 goal types:
Value
User count
Calculates the average value generated from the selected event by users exposed to the A/B Test.
Average purchase amount per buyer
Count
User count
Calculates the average number of times the selected event was triggered by users exposed to the A/B Test.
Average button clicks per user, average purchases per user
When Specific Event is selected as the denominator, the outlier exclusion option is applied to the following 2 goal types:
Value
User count
Calculates the average value generated from the numerator event by users after triggering the denominator event.
Average purchase amount per buyer
Count
User count
Calculates the average number of times the numerator event was triggered by users after triggering the denominator event.
Average number of product clicks in search results by users who triggered a search event
Last updated