# Global Impact {% hint style="info" %} Global Impact metrics are not available for MAB Tests. {% endhint %} ### What is Global Impact? Global Impact is a metric that measures the expected performance when A/B Test results are applied across the entire site.\ It predicts the actual business impact from a whole-service perspective, including users who did not participate in the experiment. ### When should you use it? #### When the experiment was conducted on a limited audience * Specific OS target: When applying an experiment targeting only iOS users to the entire user base including Android * Specific cohort: When expanding an experiment targeting only new users to all users * Regional test: When expanding an experiment targeting users in a specific region to the national/global level * Partial traffic: Predicting the effect of applying an experiment run on 10% traffic to 100% #### When the experiment has network effects * Social features: Friend recommendations, feed algorithm changes * Marketplace: Seller-buyer interaction changes * Community: Post and comment policy changes #### When the experiment is affected by or may affect internal resource constraints * Inventory management: Stock shortage due to discount policies * Server resources: System load from new features * Staffing: Increased customer service requests ### What problem does it solve? An A/B Test generally compares only the group of users who participated in the experiment. However, in actual business decision-making, you want to know the impact when a feature is deployed to the entire site. The Global Impact metric calculates the expected performance when a specific variant is applied site-wide, predicting the overall impact including not only experiment participants but also non-experiment users. ### How is it measured? 1. Basic settings * Control: Users receiving the existing experience * Treatment: Users receiving the new experience * Site-wide users: All users not participating in the test 2. Metric calculation * Direct effect: Comparison of Control vs. Treatment metrics * Site-wide effect: Change in overall service metrics #### Simple example **Experiment targeting iOS users** * Total users: 10,000 (iOS 3,000, Android 7,000) * Experiment participants: 1,500 (50% of iOS users) * Group A (Control): 500 users, 45 converting users * Group B (Treatment): 500 users, 40 converting users * Group C (Treatment): 500 users, 60 converting users * Users not participating in experiment: 8,500 users, 850 converting users **Conversion rate calculation** * Conversion rate for experiment participants * Group A: 9.0% (45 conversions / 500 users) * Group B: 8.0% (40 conversions / 500 users) * Group C: 12.0% (60 conversions / 500 users) * Conversion rate for non-experiment users: 10.0% (850 conversions / 8,500 users) * Site-wide conversion rate * Site-wide expected numerator = Non-experiment user count × Non-experiment conversion rate + Total experiment user count × Specific treatment conversion rate * Site-wide expected denominator = Total user count * Group A * Site-wide expected numerator = (8,500 × 0.10) + (1500 × 0.09) = 850 + 135 = 985 * Site-wide denominator = 10,000 * **Site-wide conversion rate = 985 ÷ 10,000 = 9.85%** * Group B * Site-wide expected numerator = (8,500 × 0.10) + (1500 × 0.08) = 850 + 120 = 970 * Site-wide denominator = 10,000 * **Site-wide conversion rate = 970 ÷ 10,000 = 9.7%** * Group C * Site-wide expected numerator = (8,500 × 0.10) + (1500 × 0.12) = 850 + 180 = 1,030 * Site-wide denominator = 10,000 * **Site-wide conversion rate = 1,030 ÷ 10,000 = 10.3%** Interpretation: If the iOS experiment results are applied to the entire service (including Android), the overall conversion rate is expected to be 9.85% for Group A, 9.7% for Group B, and 10.3% for Group C. ### How to use 1. In the Metric Management menu of the Dashboard, find the metric for which you want to understand Global Impact among the existing registered metrics, and turn on the Global Impact toggle. ![](/files/I8u2jHKHNnKd9VBd3LAz) 2. Once the A/B Test is running, the Global Impact metric is calculated automatically. ![Global Impact Metric](/files/JiDo9xf7cbAr0wyXHNfb) ![Data Summary](/files/IfxyASQ0ATfnchAF5Tx1) ### FAQ Q: Why include non-experiment users instead of just looking at the treatment group? A: Because when actually deploying site-wide, it applies to all users on the site. The current behavior of non-experiment users has a major impact on site-wide results. Q: If the experiment participation rate is low, does the site-wide effect also become smaller? A: Yes. The lower the experiment participation rate, the greater the weight of non-experiment users' existing metrics, which dilutes the site-wide effect. Q: Is it possible to predict site-wide application for experiments targeting only a specific OS or cohort? A: Yes. Even for an experiment targeting only iOS users, you can calculate the site-wide effect from the perspective of all users (including Android). However, if behavioral differences between OS types are large, prediction accuracy may vary. Q: Some metrics cannot be selected as Global Impact metrics. A: Global Impact is not supported for event metrics where the denominator is total exposed user count (by count), as well as for time metrics, funnel metrics, and retention metrics. Q: I added a Global Impact metric, but while other metrics show results, the Global Impact metric result is not being calculated. A: Unlike other metrics, Global Impact metrics are calculated once per day. ### Tips * Establish prior hypotheses: Define expected site-wide impacts in advance * Phased rollout: Small-scale test → Check Global Impact → Gradual expansion * Multi-angle analysis: Comprehensive review of revenue, usability, and operating costs * Segment-by-segment validation: When running experiments on specific OS/cohort, review applicability to other segments By using Global Impact metrics, you can accurately measure the true business impact of A/B Tests and make data-driven decisions. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.hackle.io/en/ab-test/data-analysis/global-impact.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.