# Data Integration - Data Import {% hint style="info" %} This feature is only available on the Enterprise plan. {% endhint %} ### Data Import The Data Import feature allows you to send event data stored in your AWS S3 or GCP GCS to Hackle. The Data Import feature imports data on a daily (Daily) basis. #### Supported Cloud Storage | | | | | --- | -------- | ----------- | | AWS | S3 | Y | | AWS | Redshift | Coming soon | | GCP | GCS | Y | | GCP | BigQuery | Coming soon | #### Requirements The following tasks are required before importing data. * [x] Create a storage to store event data. (AWS S3, GCP GCS, etc.) * [x] Create a Key with access permissions for the storage where event data will be stored. * [x] Process the event data in the required format and store it by day. (e.g., `2023-01-01`, `2023-01-02`, etc.) **Create Key and Grant Permissions: GCP GCS** For GCP GCS, you can create a Key by referring to the [GCP IAM > Creating and managing service account keys](https://cloud.google.com/iam/docs/creating-managing-service-account-keys?hl=ko) documentation. The following permissions are required when creating a Key for GCS access. ``` storage.buckets.get storage.objects.get storage.objects.create storage.objects.delete storage.objects.list ``` **Create Key and Grant Permissions: AWS S3** For AWS S3, you can create a Key and grant the necessary permissions by referring to the following documents. 1. Follow the [AWS Docs: Creating an IAM User](https://docs.aws.amazon.com/ko_kr/IAM/latest/UserGuide/id_users_create.html) documentation to create an AWS IAM User. 2. Follow [AWS Docs: Creating IAM Policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html#access_policies_create-start) to create a Policy that includes the IAM Policy attached below. Then add the IAM Policy to the IAM Role created in the previous step. 3. Follow [AWS Docs: Creating IAM Keys](https://docs.aws.amazon.com/ko_kr/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey) to create a Key. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion", "s3:DeleteObject", "s3:PutObject" ], "Resource": "arn:aws:s3::://*" }, { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation" ], "Resource": "arn:aws:s3:::", "Condition": { "StringLike": { "s3:prefix": [ "/*", "/", "" ] } } } ] } ``` #### Data Import Format Data Import currently supports [Apache Parquet](https://parquet.apache.org/) format. Below is the schema for the Parquet format data. Process and store data in the format described in the table below. | Column Category | Column Name | Column Type | Column Value (Example) | Description | | ------------------- | --------------------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Insert ID | `insert_id` | STRING | `8fb8e088-9245-4fce-bb87-7e09d9917ed6` | UUID value used to check for duplicate events. | | Event Key | `event_key` | STRING | `purchase` | Event name. | | Client Timestamp | `ts` | TIMESTAMP | `2023-01-01 00:01:02.333` (UTC) | Timestamp in UTC (sub-millisecond values are truncated) | | Metric Value | `metric_value` | DECIMAL(24, 6) | `0.0` | Used for value calculations in analytics and experiments. (Store `0.0` if not needed) | | Identifiers | `identifiers` | Map\ | `{ "id": "8fb8e088-9245-4fce-bb87-7e09d9917ed6", "device_id": "89ABCDEF-01234567-89ABCDEF", "user_id": "49591", "session_id": "1659710029.4.1.1659710504.0" }` |

Map containing user identifiers

  • (Optional) user\_id: Logged-in user identifier (corresponds to userId when sent via Hackle SDK)
  • (Required) id: Device identifier (corresponds to id when sent via Hackle SDK)
  • (Required) device\_id: Device identifier (corresponds to deviceId when sent via Hackle SDK)
  • (Optional, stored when using GA) ga\_session\_id, ga\_device\_id

Identifier key values are stored in lowercase.

| | Event Properties | `event_properties` | Map\ | `{ "product_id": "33537", "product_category": "LEISURE", "order_id": "291994100" }` |

Properties containing event information

Property key values are stored in lowercase.

| | User Properties | `user_properties` | Map\ | `{ "grade": "GOLD", "date_signed": "2022-07-01", "date_recent": "2023-01-17" }` |

Properties containing user information

Property key values are stored in lowercase.

| | Platform Properties | `platform_properties` | Map\ |

# Android example

/{ "osname":"Android", "appversion": "6.9.0", "language":"ko", "osversion":"12", "devicevendor":"samsung", "versionname":"6.77.0-DEBUG", "platform":"Mobile", "devicemodel":"SM-S908N" /}

iOS example

{ "osname":"iOS", "appversion": "6.9.3", "language":"ko-KR", "osversion":"16.0.2", "devicevendor":"Apple", "versionname":"6.77.0", "platform":"Mobile", "devicemodel":"iPhone14,2" /}

|

Properties containing platform information

  • (Required) osname (Android, iOS)
  • (Required) appversion

Property key values are stored in lowercase.

| Below is a summary of the data format described in the table above. ```json root |-- ts: timestamp (nullable = false) |-- event_key: string (nullable = false) |-- identifiers: string (nullable = false) |-- insert_id: string (nullable = false) |-- metric_value: decimal(24,6) (nullable = false) |-- user_properties: map (nullable = false) |-- event_properties: map (nullable = false) |-- platform_properties: map (nullable = false) ``` #### Processing Data for Data Import Process the data in the Parquet Format described above and store it by day within the bucket. * [x] When data processing is complete, create a 0-byte `_SUCCESS` (Signal) file. * [x] Data processing includes D-1 data. For example, when running Data Import for January 2nd, process and store January 1st data. Below are examples of stored partitions and files. ``` # 2023-01-01 data gcs://customer-data-hackle/test/prefix-custom/dt=2023-01-01/_SUCCESS gcs://customer-data-hackle/test/prefix-custom/dt=2023-01-01/000000000000.parquet gcs://customer-data-hackle/test/prefix-custom/dt=2023-01-01/000000000001.parquet # 2023-01-02 data gcs://customer-data-hackle/test/prefix-custom/dt=2023-01-02/_SUCCESS gcs://customer-data-hackle/test/prefix-custom/dt=2023-01-02/000000000000.parquet gcs://customer-data-hackle/test/prefix-custom/dt=2023-01-02/000000000001.parquet ``` #### Data Import Request Please contact Hackle to request Data Import. The following information is required for Data Import. * [x] Key with granted access permissions * [x] AWS S3, GCS Bucket name and partition path within the bucket where data will be stored (example: `gcs://customer-data-hackle/test/prefix-custom/dt=2023-01-01`) * [x] Data storage deadline (example: stored by 13:00 KST) --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.hackle.io/en/data-link/data-integration-data-import.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.