Hackle vs VWO
Purpose
This document introduces the key differentiators you can experience with Hackle compared to VWO (Visual Web Optimizer) Web. Through this document, you will understand the following two points:
Features that Hackle provides in addition to VWO
The advantages Hackle users can gain from those features
Key Differences
Performance & Data
Service Speed Impact
SDK-based, with no impact on service speed
Loading may slow down as visual-editor-based tests increase
Configuration Reflection Speed
Changes are reflected in near real-time
Can take several hours
Results Update
At least once per hour
Up to 24-hour delay
SPA Support
Fully supported
May not work correctly in some cases
Data Accuracy
Based on the full dataset without sampling
Not specifically stated
Features & Flexibility
Metric Configuration
Freely configure numerator/denominator — AOV/ARPU/ARPPU, etc.
Limited
Segment Analysis
Based on platform, browser, app version, and custom properties
Limited
Targeting
Custom Targeting with multi-condition combinations
Basic Targeting conditions
SDK Support
Client + Server SDK (mobile/web/server)
Web browser-based only
User Identification
Identify by any criteria you want, such as Device ID
Session-based
Operations & Support
Technical Support
Always-on Slack Hotline
Not specifically stated
Experiment/Results Limits
Unlimited concurrent experiments and results
Limits apply
Environment Separation
Both Development and Production Environments provided
No environment separation
Detailed Description
1. Hackle A/B Testing does not slow down users' service experience.
Adopting Hackle A/B Testing will not slow down your platform's service speed. In contrast, as the number of A/B tests conducted using a Visual Editor or WYSIWYG editor increases, the loading time required to render the screen shown to users can become progressively slower. This occurs because the editor determines what screen the user should see.
2. Hackle A/B Testing reflects experiment setting changes in near real-time.
After integrating the Hackle SDK, updated configuration information from the Dashboard is periodically received and applied to the code.
Compared to VWO, where starting an A/B Test can take several hours, the Hackle SDK has a short cycle for updating configuration changes, allowing near real-time control over the experiment's progress.
3. Hackle A/B Testing updates result data frequently (at least once per hour).
Since VWO provides experiment results based on data aggregated up to the day before, there can be a time lag of up to 24 hours in some cases. In this situation, users find it difficult to make immediate judgments and responses about current conditions.
With Hackle, experiment results are updated at least once per hour, keeping the time lag within 1 hour.
4. Hackle A/B Testing supports SPA (Single Page Application).
VWO may fail to implement A/B tests correctly or fail to collect user data in services built with SPA (Single Page Application) — where only the parts of the current web page that need to change are updated, rather than loading a new page in the traditional browser sense. In contrast, Hackle has no issues running A/B tests on SPA-based services.
5. Hackle A/B Testing allows you to freely configure any metrics you want.
In Hackle, you can select any event as the numerator/denominator for a metric, and through various calculation types, you can measure metrics beyond just conversion rate — including Average Order Value (AOV), Average Revenue Per User (ARPU), and Average Revenue Per Paying User (ARPPU).
Additionally, a filter configuration feature is provided to measure metrics for specific user segments.
6. Hackle A/B Testing allows metrics to be analyzed at the segment level.
The results for the metrics you want to measure in an experiment can be analyzed by segment — based on platform (iOS, Android, Web, etc.), browser, app version, or internally managed property information (e.g., membership status, first purchase status, gender, age group, etc.).
This allows you to check whether an A/B Test metric is only affecting a specific segment.
7. Hackle A/B Testing provides more advanced Targeting features.
Hackle supports setting more diverse and customizable Targeting conditions than those provided by VWO, and allows multiple conditions to be set during Targeting, enabling you to run A/B Tests for the specific users you want.
8. Hackle provides both Client SDK and Server SDK.
VWO Web only supports web browser-based A/B Testing. In contrast, Hackle provides both client and server SDKs, supporting A/B Testing and Variation creation/implementation across mobile, desktop, server environments, and all programming languages used to build platforms.
9. Hackle A/B Testing can identify users by any criteria, not just sessions — such as Device ID.
As described in Hackle's User Identifier (User Identifier) document, accurately defining user identifiers is critically important for A/B Testing. In Hackle, customers can define user identifiers using any measurement criteria they want, overcoming the limitations of session-based user identification.
10. Hackle A/B Testing provides results based on the full dataset without sampling.
Hackle A/B Testing uses the complete dataset when calculating results, providing accurate calculation outputs.
11. Hackle provides constant technical support for customers via a Hotline.
Hackle supports a private channel per customer on Slack Messenger so you can comfortably ask questions about anything you've been curious about — including implementing A/B Test code, the SDK integration process, or running experiments.
12. Hackle A/B Testing is easy to run because it provides both Development and Production Environments for each experiment.
VWO does not separate Production and Development environments, which creates the possibility that developers might confuse them. When implementing A/B Tests in code, it is necessary to test in the Development Environment first, but without environment separation there is a high risk of accidentally modifying Production settings — which are exposed to actual users.
With Hackle, both Production and Development Environments are provided for each experiment, eliminating this problem.
Last updated