Feature Flag Data Analysis
The 'Data Analysis' tab on the Feature Flag detail screen is where you can compare the differences between the group of users who received the existing feature and the group who received the new feature.
The primary purpose of this feature is to check whether users exposed to the new feature through the Feature Flag are experiencing any negative impact compared to users exposed to the existing feature. If there is an issue, it can guide you to turn the Feature Flag Off (disabled), and if there is no issue, to gradually allocate more traffic to the new feature.
The 'Data Analysis' tab screen is composed as shown in the image below.
Event Selection
Clicking the 'Set Events' button displays a modal like the one in the image below. In the modal, select the events that need data analysis and click 'Save.'
For selected events, conversion rate and average count metrics are calculated. Metric analysis results are provided for the user group exposed to the new feature (True, On) and the user group exposed to the existing feature (False, Off).
Data Analysis Tab Details
Version
By default, the analysis results in the 'Data Analysis' tab are provided per Feature Flag version. The version of a Feature Flag changes when a modification is made to the targeting conditions. (e.g., Version 1 → Version 2)
Cases where a Feature Flag targeting condition changes include:
Feature Flag status (On, Off) change
Feature Flag > User Targeting > Rule change (including traffic distribution ratio change)
Improvement Rate
For the metric selected on the graph, you can see how much the group of users exposed to the new feature (True, On) improved compared to the group exposed to the existing feature (False, Off).
Data from Version 1 through the latest version can be viewed at once in the graph. (See image below)
Select Box: You can select either conversion rate metric or average count metric.
Cumulative picker: Shows the improvement rate calculated based on data from the start point to the current calculation point per Feature Flag version.
Daily picker: Shows the improvement rate calculated based on daily data from the start point per Feature Flag version.
Metric Analysis Results
You can view the calculation results for all metrics calculated in the 'Data Analysis' tab in a table format.
While the 'Improvement Rate' graph area at the top could only show improvement rate trends, the metric analysis results table additionally provides p-value per version and a graph to track metric trends.
You can view metric analysis results per Feature Flag version by clicking the select box in the upper right of the table to select a version.
Clicking the 'Show Graph' button allows you to view the metric values for the True (On) and False (Off) groups, the improvement rate, and the trend of p-value per metric.
SDK versions that support the Feature Flag Data Analysis feature
Android
2.25.0+
iOS
2.23.0+
Java
2.18.0+
Node.js
11.21.0+
Python
3.6.0+
Javascript
11.21.0+
React
11.21.0+
Last updated