FAQ

1. Questions About Hackle Service

1. What kinds of companies use Hackle?

Companies of various sizes, including Yeogieottae, SKT, Oliveyoung, and Yogiyo, use Hackle.

More Details

Companies of all sizes and service types use Hackle for data-driven product development.

Companies actively using Hackle for data-driven product development include Yeogieottae, SKT, LG U+, Oliveyoung, Yogiyo, Carrot Insurance, Korea Credit Data, Sparta Coding Club, GameDuo, CatchTable, Kmong, and more.

Services using platforms such as Cafe24 or Imweb can also use Hackle.

For example, Imweb provides a feature to insert scripts, so you can insert the Hackle JavaScript SDK integration code into Imweb to enable data collection and use Hackle.

2. What features does Hackle offer?

A/B Testing, Feature Flags, Remote Config, Data Analytics, In-App Messages, and Push Message campaigns can all be used from a single Dashboard.

More Details

After completing the integration work with the Hackle platform using the provided SDK, you can conveniently use A/B Testing, Feature Flags, Remote Config, Data Analytics — essential for data-driven product development — as well as In-App Message and Push Message campaigns to boost user engagement, all from a single Dashboard.

3. Does the Hackle SDK integration key ever change?

The Hackle SDK integration key never changes.

4. How can I get help with integration or development-related inquiries?

Please first check the FAQ and Troubleshooting documents in the SDK category. If the issue is not resolved, use the contact information in the Dashboard.

More Details

Please check the FAQ and Troubleshooting documents to see if your issue is covered.

For questions that still remain unresolved, please use the contact information provided in the Hackle Dashboard.

5. Can I manage multiple products under one account?

Yes. Since a single email account can be used across multiple Workspaces, you can manage multiple products under one account.

6. Can I manage multiple domains (URLs) in one Workspace?

You must create one Workspace per URL or product.

More Details

We recommend creating one Workspace per URL or product.

Connecting multiple products to a single Workspace may cause metric contamination and errors when using CRM features.

7. Can services built with no-code tools use Hackle?

Yes. If the no-code tool has an HTML/script editing feature, you can insert the Hackle data collection script to use Hackle.

More Details

Since Hackle analyzes services by collecting data and events, even services built with no-code web page tools (e.g., Framer) can use Hackle — as long as the tool has an HTML/script editing feature where you can insert the Hackle data collection script, and URL auto-collection is used for event tracking.

8. How can I create a Workspace?

A Workspace is automatically created upon sign-up. To create additional Workspaces, sign up with a different email address or contact the Hackle team.

9. What is Hackle's default session duration? Can the session duration be changed?

The default session duration is 30 minutes, and it can be changed using the session timeout option during SDK initialization.

More Details

Sessions are automatically managed by Hackle's frontend SDK.

The default session duration is 30 minutes, and it is automatically extended if the user triggers an event within that window.

If you need to change the default session duration, you can set the session timeout option during SDK initialization.

10. Can data still be collected if ATT (App Tracking Transparency) consent is not obtained on iOS 14.5 or later?

Yes. Hackle is first-party data-based, so data can be collected regardless of whether ATT consent has been obtained.

11. How is billing usage/traffic measured?

Usage consists of Exposure events and Track (behavior) events. Draft and completed experiments are excluded from billing.

More Details

Hackle usage consists primarily of Exposure events (when A/B Tests, Feature Flags, or Remote Config are called for distribution) and general behavior events (Track).

Exposure events occur when an Experiment Key, Feature Key, or Remote Config Key is called and a valid value is returned. Calls made on experiments that are in draft or completed status are excluded from billing.

Behavior events include auto-collected events automatically collected by the Hackle SDK and custom events sent directly by customers.

To check your current usage, refer to the Check Usage documentation.

12. Are there charges for sample events? Is there a way to disable them?

Sample events are not actually sent via the SDK, so no costs are incurred.

2. Questions About A/B Testing

1. What types of A/B Tests can I run on the Hackle A/B Testing platform?

Server/client-based A/B Tests — such as screen changes, new feature releases, and algorithm changes — are all possible.

More Details

Hackle supports A/B Tests for screen changes in mobile apps and websites, as well as changes to server-side APIs. Server/client-based A/B Tests for screen changes, new feature releases, and algorithm changes are all possible.

2. What advantages does Hackle have over other A/B Testing platforms?

Support for a wide variety of SDKs, near real-time application, hourly result updates, flexible goal setting, and a cost-effective price are all advantages.

More Details

First, Hackle supports a wide variety of SDKs, enabling A/B Testing not only on websites but also in mobile apps and server environments. Hackle provides 10 types of SDKs including Android, iOS, JavaScript, and React.

Second, Hackle applies A/B Test changes to your service in near real-time. In contrast, Google Optimize may take several hours or more for test changes to be reflected in your service.

Third, Hackle updates A/B Test results on an hourly basis. Google Optimize calculates A/B Test results based on data aggregated up to the day before from GA, making it difficult to check up-to-the-minute test results quickly.

Fourth, Hackle allows you to freely create any Goals you want to measure. Regardless of your service's domain, you can create and measure any goal you want.

Finally, Hackle is one of the most cost-effective A/B Testing platforms available. Paid tools like Optimizely and Google Optimize 360 are known to cost at least tens of millions of Korean won per year.

3. How many users are needed to run an A/B Test?

There is no minimum user count, but having at least 100 daily active users is ideal.

4. Can I run A/B Tests without statistical knowledge?

Yes. Hackle visually provides statistical significance, so statistical knowledge is not required.

More Details

Yes, you can run A/B Tests without statistical knowledge. Hackle provides visual displays showing whether A/B Test results — such as click rates for specific buttons/banners or purchase conversion rates — are statistically significant.

Statistical knowledge may be helpful if you extract raw A/B Test data for additional analysis.

5. What should I keep in mind when analyzing A/B Test results?

Maintain a test duration of at least 1-2 weeks and watch out for result distortion caused by early termination.

More Details

A minimum test duration of one to two weeks is needed to prevent result distortion from day-of-week effects. Be cautious of ending a test prematurely if results look unusually good or bad over a short period.

Ending a test early carries the risk of treating distorted information as a conclusion. It is recommended to observe the test for at least the minimum set period (e.g., one week).

6. What is the difference between marketing A/B tests and product A/B tests?

Marketing A/B tests aim to maximize acquisition channel performance, while product A/B tests aim to optimize features/algorithms within the service.

More Details

A marketing A/B test is an experiment conducted from a marketing perspective to maximize the performance of landing pages, app push messages, and other user acquisition channels. Since it typically has one clear success goal — 'click rate' — techniques like MAB are also useful for marketing A/B tests.

A product A/B test involves experimenting within your website or mobile app to determine which screens, algorithms, or features are most effective for users in maximizing business goals.

7. The same metric shows different values in the A/B Test and in Data Analytics.

The two features aggregate at different points in time, which is why they may differ.

More Details

This can happen when the same metric shows different values in A/B Test and Data Analytics.

Fundamentally, an A/B Test distributes a variation to selected users at a specific point in time and then calculates occurrences of the "numerator event" among users who triggered the "denominator event" after that point. Since there is a causal relationship, if the denominator is 0, the value will naturally be 0.

As mentioned, an A/B Test calculates events based on the causal relationship relative to the distribution point, while Data Analytics only calculates the pure occurrence of the events you select — regardless of specific timing or the numerator/denominator sequence. Therefore it is correct for Data Analytics to show values even in cases where the A/B Test does not.

In short, the two features aggregate at different points in time, which is why they may differ.

8. User Explorer is not working in the Development Environment. Even after registering a Test Device, Override is not applied.

This is caused by Cookie management issues when using the vercel.app domain. We recommend using a custom domain.

More Details

When testing in the Development Environment, you may find that force-assignment to a different group does not work and only Group A appears in real-time exposure status.

In this case, please check whether you are using the vercel.app domain.

Hackle uses browser Cookies to identify users. When using the vercel.app domain, the Hackle SDK cannot properly manage cookies. (Reference link)

The recommended solution is to use a custom domain provided by Vercel.

9. What happens if I remove some Variation groups while an experiment is running?

Users in the removed group are excluded from the experiment and will see the Group A screen. The experiment continues with the remaining groups.

10. What problems occur if sample sizes are not evenly distributed between groups?

When sample sizes are uneven, the variability of the smaller group is overrepresented, which can distort the p-value.

More Details

In the t-test commonly used for A/B Tests, the p-value can be distorted. This is because the variability of the smaller group is reflected more heavily. Looking at the standard error formula used in a t-test:

t-test standard error formula

Here, S²A and S²B are the standard deviations of groups A and B respectively, and nA and nB are the sample sizes of each group. According to the formula, variability in larger groups is weighted less, while variability in smaller groups is weighted more. This affects the p-value calculation — the variability of the smaller group is overrepresented, or that of the larger group is underrepresented, distorting the p-value and ultimately reducing the reliability of the experiment results. This can lead to incorrect conclusions or decisions — effectively rendering the experiment worthless.

11. Is data from Test Devices reflected in experiment results?

Generally it is not, but data is included if the user was already exposed to the experiment before being registered as a Test Device.

12. If I set a Winner group in an experiment where only partial traffic was allocated, does all traffic go to the Winner group?

Yes, all traffic is distributed to the Winner group. Users who do not meet the Targeting conditions are assigned to Group A.

More Details

Yes, that is correct. All traffic is distributed to the Winner group. If the experiment has Targeting conditions set, all users who meet those conditions (e.g., all users, or those on iOS/Android app version 2.10.3 or higher) will be assigned to the Winner group. Users who do not meet the Targeting conditions will be assigned to Group A.

13. Are there charges for sample experiments? Is there a way to disable them?

Sample experiments do not generate actual distribution events, so no costs are incurred.

14. I want to run a landing page A/B Test. I was planning to use SSG for SEO, but if I use the Hackle React SDK, do I have to use client-side components only, or is there a way to use server-side components?

A/B Testing is difficult to implement with SSG. You must switch to SSR or use client-side components.

More Details

An A/B Test determines distribution when a user enters the page and then dynamically renders content (page or component) accordingly. This makes it difficult to implement Hackle's A/B Test with SSG, where content is determined at build time.

When using the Hackle React SDK, you have two options:

Switch to SSR (Server-Side Rendering) and use it that way, or maintain SSG while using a client-side component. If you must maintain SSG for SEO purposes, you will need to implement A/B Testing with client-side components. Currently, there is no way to use Hackle's A/B Test in a server-side component with SSG.

3. Questions About Data Analytics

1. What is the data aggregation criteria for funnel charts?

Funnel charts are aggregated based on Unique ID, not daily sums.

More Details

When you create a funnel chart and set different periods — 1 day, 3 days, 7 days — the 3-day value may differ from the 1-day value multiplied by 3. This is because funnel charts are aggregated based on Unique ID, not daily sums.

In other words, unique IDs that triggered the event over 3 days are counted, as shown in the example below:

• day 1 = 10 users

  • User 1, User 2, ... User 10 trigger events

• day 3 = 6 users

  • User 1 = triggered on day 1 => counted as 1

  • User 2 = triggered on day 1, day 2 => counted as 1

  • User 3 = triggered on day 3 => counted as 1

  • User 4 = triggered on day 1, day 2, day 3 => counted as 1

2. Can I change the name of a registered event?

Event names cannot be changed. Collect data under a new name and turn off the existing event.

3. What is the aggregation criteria for 'active users' in Data Analytics > Segment Items > Calculation method for a specific event?

'Active users' counts all Users who triggered Any Event.

More Details

Here, 'active users' counts all users who triggered Any Event.

For example, if you select the calculation method for a 'purchase completed' event as 'average count per active user,' it is calculated as: total number of occurrences of the specific event / total number of users based on Any Event.

4. The chart data for the same period looks different than it did yesterday!

Past events may be sent with a delay, and Hackle reflects data from up to 7 days ago.

More Details

Events may sometimes be sent with a delay after they originally occurred. This is especially common for app services, where events can be sent late due to various reasons such as the user's internet connection dropping or the app not being fully closed in the background. In Hackle, data from up to 7 days ago is treated as valid and reflected in Data Analytics results.

5. Why is the conversion rate different between the Funnel chart and the User Path chart?

The User Path chart only counts conversions within the same session, while the Funnel chart counts conversions within the period regardless of session.

More Details

The User Path chart only counts conversions within the same session, whereas the Funnel chart counts conversions within the selected period regardless of session. This means the Funnel chart's conversion rate may be higher depending on the period. Additionally, since the Funnel chart supports different aggregation types (simple funnel, closed funnel, sequential funnel), each type uses different criteria for counting conversions, which is why the two charts may show different conversion rates.

6. What is the measurement criteria for new visit events?

Based on Hackle ID, the first time an event is collected is counted as a new visit.

7. It's strange that the Day0 retention in the retention chart is lower than 100%.

If the start event and the metric event are different, it is normal for Day0 retention to be lower than 100%.

More Details

In the retention chart, you can set a start event and a metric event. If both events are the same, Day0 retention is measured at 100%. If the two events are different, it is normal for Day0 retention to be lower than 100%. [Example] • User count: number of users who triggered the start event on Day0 • Day0: users who triggered the start event on Day0 who also triggered the metric event on Day0 • Day1: users counted in Day0 who triggered the metric event on Day1

8. How is 'Total Users' in the retention chart calculated?

It is the ratio of unique users who actually triggered retention, out of the cohorts of unique users who could have triggered retention for that time unit.

More Details

The result displayed in the chart — the summary of Retention Analysis results — is the retention rate of 'Total Users' shown in the table below the chart. The retention of 'Total Users' is the ratio of unique users who actually achieved the retention for a given time unit (e.g., Week 2) out of the unique users in cohorts that could possibly achieve retention for that time unit (e.g., Week 2). For example, cohorts that entered within the last week are excluded from the denominator because they cannot possibly achieve Week 2 retention.

4. Questions About CRM Campaigns and Messages

1. I've set up an In-App Message, but the console shows [ARN] H Hackle: SDK not ready. and the In-App Message is not displayed.

An event was sent before the SDK was initialized. Change your code to send events inside the onReady callback.

More Details

This happens when an event is sent before the SDK is initialized — since Hackle retrieves In-App Message information from the server after SDK initialization. In this case, change your code as follows:

//as-is
hackleClient.track({ key: "이벤트 키" });

//to-be
hackleClient.onReady(() => {
	hackleClient.track({ key: "이벤트 키" });
});

2. Can In-App Messages be used to link to an external web page as an outlink in an app environment (e.g., a Google Form survey)?

By default, links open in the default browser. Using an in-app webview scheme allows them to open inside the app.

More Details

In an app environment, if you enter a web link URL in the click action or button, it will open in the phone's default browser (Safari, Chrome, etc.) by default. Generally, apps implement an internal scheme to display web views within the app, so by entering the URL according to that scheme, you can navigate to an in-app web view (e.g., myApp://webview?url=abcd).

3. How often are In-App Message and Push Message performance metrics (click rate, send rate) updated?

For In-App Messages you can check impressions vs. clicks, and for Push Messages you can check device-to-send and send-to-click. These metrics are updated every hour.

More Details

For ongoing In-App Messages and Push Messages, you can check campaign performance at the top of the detail page. For In-App Messages, impressions-to-clicks are easy to check; for Push Messages, device-to-send and send-to-click ratios are readily visible. You can also navigate to Data Analytics for more detailed performance analysis. These metrics are updated every hour.

4. Can In-App Message content be personalized?

Personalization at the time of display is possible by using In-App Message HTML.

5. How is the estimated send volume for Kakao Friendtalk/Alimtalk/Brand Messages calculated?

It is based on the number of users whose phone numbers have been collected in Hackle. Actual sending may differ due to issues such as not being a KakaoTalk Channel subscriber.

More Details

Hackle does not collect or evaluate information such as whether a user is a KakaoTalk Channel friend or whether Alimtalk has been blocked. The estimated send volume is calculated based on the number of users among the target audience from whom a phone number has been collected in Hackle. At the time of actual sending, some messages may not be delivered due to issues such as not being a KakaoTalk Channel friend, Alimtalk being blocked, or the user not using KakaoTalk. Actual cash deduction is based on successfully sent messages only.