Global Impact
Global Impact metrics are not available for MAB Tests.
What is Global Impact?
Global Impact is a metric that measures the expected performance when A/B Test results are applied across the entire site. It predicts the actual business impact from a whole-service perspective, including users who did not participate in the experiment.
When should you use it?
When the experiment was conducted on a limited audience
Specific OS target: When applying an experiment targeting only iOS users to the entire user base including Android
Specific cohort: When expanding an experiment targeting only new users to all users
Regional test: When expanding an experiment targeting users in a specific region to the national/global level
Partial traffic: Predicting the effect of applying an experiment run on 10% traffic to 100%
When the experiment has network effects
Social features: Friend recommendations, feed algorithm changes
Marketplace: Seller-buyer interaction changes
Community: Post and comment policy changes
When the experiment is affected by or may affect internal resource constraints
Inventory management: Stock shortage due to discount policies
Server resources: System load from new features
Staffing: Increased customer service requests
What problem does it solve?
An A/B Test generally compares only the group of users who participated in the experiment. However, in actual business decision-making, you want to know the impact when a feature is deployed to the entire site. The Global Impact metric calculates the expected performance when a specific variant is applied site-wide, predicting the overall impact including not only experiment participants but also non-experiment users.
How is it measured?
Basic settings
Control: Users receiving the existing experience
Treatment: Users receiving the new experience
Site-wide users: All users not participating in the test
Metric calculation
Direct effect: Comparison of Control vs. Treatment metrics
Site-wide effect: Change in overall service metrics
Simple example
Experiment targeting iOS users
Total users: 10,000 (iOS 3,000, Android 7,000)
Experiment participants: 1,500 (50% of iOS users)
Group A (Control): 500 users, 45 converting users
Group B (Treatment): 500 users, 40 converting users
Group C (Treatment): 500 users, 60 converting users
Users not participating in experiment: 8,500 users, 850 converting users
Conversion rate calculation
Conversion rate for experiment participants
Group A: 9.0% (45 conversions / 500 users)
Group B: 8.0% (40 conversions / 500 users)
Group C: 12.0% (60 conversions / 500 users)
Conversion rate for non-experiment users: 10.0% (850 conversions / 8,500 users)
Site-wide conversion rate
Site-wide expected numerator = Non-experiment user count × Non-experiment conversion rate + Total experiment user count × Specific treatment conversion rate
Site-wide expected denominator = Total user count
Group A
Site-wide expected numerator = (8,500 × 0.10) + (1500 × 0.09) = 850 + 135 = 985
Site-wide denominator = 10,000
Site-wide conversion rate = 985 ÷ 10,000 = 9.85%
Group B
Site-wide expected numerator = (8,500 × 0.10) + (1500 × 0.08) = 850 + 120 = 970
Site-wide denominator = 10,000
Site-wide conversion rate = 970 ÷ 10,000 = 9.7%
Group C
Site-wide expected numerator = (8,500 × 0.10) + (1500 × 0.12) = 850 + 180 = 1,030
Site-wide denominator = 10,000
Site-wide conversion rate = 1,030 ÷ 10,000 = 10.3%
Interpretation: If the iOS experiment results are applied to the entire service (including Android), the overall conversion rate is expected to be 9.85% for Group A, 9.7% for Group B, and 10.3% for Group C.
How to use
In the Metric Management menu of the Dashboard, find the metric for which you want to understand Global Impact among the existing registered metrics, and turn on the Global Impact toggle.
Once the A/B Test is running, the Global Impact metric is calculated automatically.
Global Impact Metric 
Data Summary
FAQ
Q: Why include non-experiment users instead of just looking at the treatment group? A: Because when actually deploying site-wide, it applies to all users on the site. The current behavior of non-experiment users has a major impact on site-wide results.
Q: If the experiment participation rate is low, does the site-wide effect also become smaller? A: Yes. The lower the experiment participation rate, the greater the weight of non-experiment users' existing metrics, which dilutes the site-wide effect.
Q: Is it possible to predict site-wide application for experiments targeting only a specific OS or cohort? A: Yes. Even for an experiment targeting only iOS users, you can calculate the site-wide effect from the perspective of all users (including Android). However, if behavioral differences between OS types are large, prediction accuracy may vary.
Q: Some metrics cannot be selected as Global Impact metrics. A: Global Impact is not supported for event metrics where the denominator is total exposed user count (by count), as well as for time metrics, funnel metrics, and retention metrics.
Q: I added a Global Impact metric, but while other metrics show results, the Global Impact metric result is not being calculated. A: Unlike other metrics, Global Impact metrics are calculated once per day.
Tips
Establish prior hypotheses: Define expected site-wide impacts in advance
Phased rollout: Small-scale test → Check Global Impact → Gradual expansion
Multi-angle analysis: Comprehensive review of revenue, usability, and operating costs
Segment-by-segment validation: When running experiments on specific OS/cohort, review applicability to other segments
By using Global Impact metrics, you can accurately measure the true business impact of A/B Tests and make data-driven decisions.
Last updated