# A/A Test In an A/A Test, the treatment and control groups receive the same product experience.\ While the goal of an A/B Test is to verify whether there are meaningful metric changes between the treatment and control groups, an A/A Test is conducted to validate that there is no difference between the two groups. ### Why run an A/A Test? An A/A Test is primarily conducted before launching a full-scale A/B Test or before introducing a new A/B Test platform. Through an A/A Test, organizations can evaluate the reliability of their A/B Test tool, and determine baseline conversion rates or minimum sample sizes to use in subsequent A/B Tests. Some Hackle customers also run A/A Tests during the onboarding process to verify that Hackle's A/B Test platform is trustworthy as an experimentation tool. ### What to watch out for when running an A/A Test When running an A/A Test, you may find results contrary to your expectations — that there is a significant difference between the treatment and control groups. The key point to keep in mind is that a significant result does not always mean there is a problem with the experiment. In hypothesis testing, statistical significance is a probability. Testing at 95% confidence means that, out of 20 experiments, 1 may produce a significantly different result by chance (Type I error).\ Therefore, if you can confirm that the result could reasonably occur within the bounds of statistical significance, it can be attributed to chance rather than a problem with the experiment. Running hundreds of thousands of A/A Tests to verify this yourself would be a waste of time.\ Instead, you can verify it without actually running the experiments by repeatedly redistributing users who participated in the A/A Test into 2 groups and calculating p-value or Bayesian probability distributions. ![](/files/UelatY3yyS3EcOKU3nEH) The experiment above is an actual A/A Test run at Hackle where results showed a significant difference between the treatment and control group data.\ Let's compare the first goal metric — purchase conversion rate — at 95% confidence, by repeatedly redistributing users who participated in this experiment into 2 groups. #### p-value Distribution In hypothesis testing, when the p-value is less than the significance level, we reject the null hypothesis (there is no difference between groups). Since the significance level is the probability of rejecting the null hypothesis and accepting the alternative hypothesis (there is a difference between groups) when the null hypothesis is true, the probability that the p-value is less than the significance level should equal the significance level. ![](/files/PCLNsv0f98Zrqu6UIaLS) When the null hypothesis is true, the test statistic T follows the distribution F(t). Letting P = F(T) as the p-value, for this relationship to hold regardless of the significance level, F(·) must be a uniform distribution. Here, F(·) is an invertible function. ![](/files/1IkPXPnUsJbPXDFbbb9a) A histogram was drawn from p-values obtained from 10,000 simulations using the A/A Test data introduced above.\ Even though the results showed significant differences between the treatment and control group data, the p-value shows a uniform distribution shape with values evenly distributed between 0 and 1. This means the experimental result could reasonably occur by chance within the bounds of statistical significance. #### Bayesian Probability Distribution ![](/files/8MDBhwjrmbLm99unvhWM) After estimating the posterior distributions of the treatment and control groups, we calculate the probability that the control's goal is higher than the treatment's. Drawing a histogram from the probabilities obtained by repeating the simulation 500 times shows a normal distribution shape centered around 0.5. If the probability that the control's goal is higher than the treatment's is close to 0.5, we can conclude that there is no difference between the two groups. If after running an A/A Test the p-value or Bayesian probability distribution shows a shape other than the distributions examined above, or if the conclusion that there is a significant difference between the treatment and control groups persists through multiple repetitions, there may be a problem with the experimental design or the data being aggregated. The root cause varies depending on the situation, but the most common causes are: * There are outliers in the data. → In Hackle, you can remove them using the [Outlier exclusion feature](/en/ab-test/data-analysis/outlier.md). * The experiment distribution trigger has an error, causing unintended users (who should not be included in the experiment) to be included in the exposure data. → In Hackle, you can check the [Real-time Exposure Status](/en/ab-test/management/realtime-status.md) to verify that distribution is working correctly per group. You can check whether distribution (exposure) data is being generated at a stage before or after the page where experiment distribution should occur, or whether the experiment is being distributed to users who are not the experiment target. * There is an error in equal distribution, resulting in a large difference in the number of users assigned to the treatment and control groups, or users with similar characteristics being assigned to the same group. → Hackle uses a bucketing method with hash functions to distribute users, so equal distribution is guaranteed if there is no problem with the experimental design. For a detailed explanation, refer to the [Variation Distribution](/en/ab-test/group-distribution.md) page. ### What to watch out for when running an A/A/B Test An A/A/B Test is used when you want to run an A/A Test and an A/B Test at the same time. An A/A/B Test divides traffic into three groups (A, A, or B), which reduces the number of users in each group and lowers the statistical power.\ Therefore, to ensure sufficient statistical power in an A/A/B Test, you need to increase traffic or run the experiment for a longer period. For this reason, Hackle recommends an A/A Test rather than an A/A/B Test. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.hackle.io/en/ab-test/data-analysis/aa-test.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.