Global Impact

Global Impact metrics are not available for MAB Tests.

What is Global Impact?

Global Impact is a metric that measures the expected performance when A/B Test results are applied across the entire site. It predicts the actual business impact from a whole-service perspective, including users who did not participate in the experiment.

When should you use it?

When the experiment was conducted on a limited audience

• Specific OS target: When applying an experiment targeting only iOS users to the entire user base including Android

• Specific cohort: When expanding an experiment targeting only new users to all users

• Regional test: When expanding an experiment targeting users in a specific region to the national/global level

• Partial traffic: Predicting the effect of applying an experiment run on 10% traffic to 100%

When the experiment has network effects

• Social features: Friend recommendations, feed algorithm changes

• Marketplace: Seller-buyer interaction changes

• Community: Post and comment policy changes

When the experiment is affected by or may affect internal resource constraints

• Inventory management: Stock shortage due to discount policies

• Server resources: System load from new features

• Staffing: Increased customer service requests

What problem does it solve?

An A/B Test generally compares only the group of users who participated in the experiment. However, in actual business decision-making, you want to know the impact when a feature is deployed to the entire site. The Global Impact metric calculates the expected performance when a specific variant is applied site-wide, predicting the overall impact including not only experiment participants but also non-experiment users.

How is it measured?

1. Basic settings

   • Control: Users receiving the existing experience

   • Treatment: Users receiving the new experience

   • Site-wide users: All users not participating in the test

2. Metric calculation

   • Direct effect: Comparison of Control vs. Treatment metrics

   • Site-wide effect: Change in overall service metrics

Simple example

Experiment targeting iOS users

• Total users: 10,000 (iOS 3,000, Android 7,000)

• Experiment participants: 1,500 (50% of iOS users)

  • Group A (Control): 500 users, 45 converting users

  • Group B (Treatment): 500 users, 40 converting users

  • Group C (Treatment): 500 users, 60 converting users

• Users not participating in experiment: 8,500 users, 850 converting users

Conversion rate calculation

• Conversion rate for experiment participants

  • Group A: 9.0% (45 conversions / 500 users)

  • Group B: 8.0% (40 conversions / 500 users)

  • Group C: 12.0% (60 conversions / 500 users)

• Conversion rate for non-experiment users: 10.0% (850 conversions / 8,500 users)

• Site-wide conversion rate

  • Site-wide expected numerator = Non-experiment user count × Non-experiment conversion rate + Total experiment user count × Specific treatment conversion rate

  • Site-wide expected denominator = Total user count

    • Group A

      • Site-wide expected numerator = (8,500 × 0.10) + (1500 × 0.09) = 850 + 135 = 985

      • Site-wide denominator = 10,000

      • Site-wide conversion rate = 985 ÷ 10,000 = 9.85%

    • Group B

      • Site-wide expected numerator = (8,500 × 0.10) + (1500 × 0.08) = 850 + 120 = 970

      • Site-wide denominator = 10,000

      • Site-wide conversion rate = 970 ÷ 10,000 = 9.7%

    • Group C

      • Site-wide expected numerator = (8,500 × 0.10) + (1500 × 0.12) = 850 + 180 = 1,030

      • Site-wide denominator = 10,000

      • Site-wide conversion rate = 1,030 ÷ 10,000 = 10.3%

Interpretation: If the iOS experiment results are applied to the entire service (including Android), the overall conversion rate is expected to be 9.85% for Group A, 9.7% for Group B, and 10.3% for Group C.

How to use

1. In the Metric Management menu of the Dashboard, find the metric for which you want to understand Global Impact among the existing registered metrics, and turn on the Global Impact toggle.

1. Once the A/B Test is running, the Global Impact metric is calculated automatically.

   

   Global Impact Metric

   

   Data Summary

Tips

• Establish prior hypotheses: Define expected site-wide impacts in advance

• Phased rollout: Small-scale test → Check Global Impact → Gradual expansion

• Multi-angle analysis: Comprehensive review of revenue, usability, and operating costs

• Segment-by-segment validation: When running experiments on specific OS/cohort, review applicability to other segments

By using Global Impact metrics, you can accurately measure the true business impact of A/B Tests and make data-driven decisions.