Data Integration - Data Import

This feature is only available on the Enterprise plan.

Data Import

The Data Import feature allows you to send event data stored in your AWS S3 or GCP GCS to Hackle. The Data Import feature imports data on a daily (Daily) basis.

Supported Cloud Storage

AWS	S3	Y
AWS	Redshift	Coming soon
GCP	GCS	Y
GCP	BigQuery	Coming soon

Requirements

The following tasks are required before importing data.

• Create a storage to store event data. (AWS S3, GCP GCS, etc.)

• Create a Key with access permissions for the storage where event data will be stored.

• Process the event data in the required format and store it by day. (e.g., 2023-01-01, 2023-01-02, etc.)

Create Key and Grant Permissions: GCP GCS

For GCP GCS, you can create a Key by referring to the GCP IAM > Creating and managing service account keys documentation.

The following permissions are required when creating a Key for GCS access.

storage.buckets.get
storage.objects.get
storage.objects.create
storage.objects.delete
storage.objects.list

Create Key and Grant Permissions: AWS S3

For AWS S3, you can create a Key and grant the necessary permissions by referring to the following documents.

1. Follow the AWS Docs: Creating an IAM User documentation to create an AWS IAM User.

2. Follow AWS Docs: Creating IAM Policies to create a Policy that includes the IAM Policy attached below. Then add the IAM Policy to the IAM Role created in the previous step.

3. Follow AWS Docs: Creating IAM Keys to create a Key.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
              "s3:GetObject",
              "s3:GetObjectVersion",
              "s3:DeleteObject",
              "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::<bucket>/<prefix>/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::<bucket>",
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "<prefix>/*",
                        "<prefix>/",
                        "<prefix>"
                    ]
                }
            }
        }
    ]
}

Data Import Format

Data Import currently supports Apache Parquet format. Below is the schema for the Parquet format data. Process and store data in the format described in the table below.

Column Category	Column Name	Column Type	Column Value (Example)	Description
Insert ID	insert_id	STRING	8fb8e088-9245-4fce-bb87-7e09d9917ed6	UUID value used to check for duplicate events.
Event Key	event_key	STRING	purchase	Event name.
Client Timestamp	ts	TIMESTAMP	2023-01-01 00:01:02.333 (UTC)	Timestamp in UTC (sub-millisecond values are truncated)
Metric Value	metric_value	DECIMAL(24, 6)	0.0	Used for value calculations in analytics and experiments. (Store 0.0 if not needed)
Identifiers	identifiers	Map<String, String>	{ "id": "8fb8e088-9245-4fce-bb87-7e09d9917ed6", "device_id": "89ABCDEF-01234567-89ABCDEF", "user_id": "49591", "session_id": "1659710029.4.1.1659710504.0" }	Map containing user identifiers (Optional) user_id: Logged-in user identifier (corresponds to userId when sent via Hackle SDK) (Required) id: Device identifier (corresponds to id when sent via Hackle SDK) (Required) device_id: Device identifier (corresponds to deviceId when sent via Hackle SDK) (Optional, stored when using GA) ga_session_id, ga_device_id Identifier key values are stored in lowercase.
Event Properties	event_properties	Map<String, String>	{ "product_id": "33537", "product_category": "LEISURE", "order_id": "291994100" }	Properties containing event information Property key values are stored in lowercase.
User Properties	user_properties	Map<String, String>	{ "grade": "GOLD", "date_signed": "2022-07-01", "date_recent": "2023-01-17" }	Properties containing user information Property key values are stored in lowercase.
Platform Properties	platform_properties	Map<String, String>	# Android example /{ "osname":"Android", "appversion": "6.9.0", "language":"ko", "osversion":"12", "devicevendor":"samsung", "versionname":"6.77.0-DEBUG", "platform":"Mobile", "devicemodel":"SM-S908N" /} iOS example { "osname":"iOS", "appversion": "6.9.3", "language":"ko-KR", "osversion":"16.0.2", "devicevendor":"Apple", "versionname":"6.77.0", "platform":"Mobile", "devicemodel":"iPhone14,2" /}	Properties containing platform information (Required) osname (Android, iOS) (Required) appversion Property key values are stored in lowercase.

Below is a summary of the data format described in the table above.

root
 |-- ts: timestamp (nullable = false)
 |-- event_key: string (nullable = false)
 |-- identifiers: string (nullable = false)
 |-- insert_id: string (nullable = false)
 |-- metric_value: decimal(24,6) (nullable = false)
 |-- user_properties: map<string, string> (nullable = false)
 |-- event_properties: map<string, string> (nullable = false)
 |-- platform_properties: map<string, string> (nullable = false)

Processing Data for Data Import

Process the data in the Parquet Format described above and store it by day within the bucket.

• When data processing is complete, create a 0-byte _SUCCESS (Signal) file.

• Data processing includes D-1 data. For example, when running Data Import for January 2nd, process and store January 1st data.

Below are examples of stored partitions and files.

# 2023-01-01 data
gcs://customer-data-hackle/test/prefix-custom/dt=2023-01-01/_SUCCESS
gcs://customer-data-hackle/test/prefix-custom/dt=2023-01-01/000000000000.parquet
gcs://customer-data-hackle/test/prefix-custom/dt=2023-01-01/000000000001.parquet

# 2023-01-02 data
gcs://customer-data-hackle/test/prefix-custom/dt=2023-01-02/_SUCCESS
gcs://customer-data-hackle/test/prefix-custom/dt=2023-01-02/000000000000.parquet
gcs://customer-data-hackle/test/prefix-custom/dt=2023-01-02/000000000001.parquet

Data Import Request

Please contact Hackle to request Data Import. The following information is required for Data Import.

• Key with granted access permissions

• AWS S3, GCS Bucket name and partition path within the bucket where data will be stored (example: gcs://customer-data-hackle/test/prefix-custom/dt=2023-01-01)

• Data storage deadline (example: stored by 13:00 KST)