Data Integration - Data Export

This feature is only available on the Enterprise plan.

Data Export

The Data Export feature allows you to export raw data stored in Hackle to storage such as your AWS S3 or GCP GCS. The Data Export feature transmits data on a daily (Daily) basis.

Supported Cloud Storage

Cloud	Storage	Supported
AWS	S3	Y
AWS	Redshift	Coming soon
GCP	GCS	Y
GCP	BigQuery	Coming soon

Requirements

The following tasks are required before exporting data.

• Create a storage to receive raw data. (AWS S3, GCP GCS, etc.)

• Create a Key with the necessary permissions for the storage that will receive raw data.

Create Key and Grant Permissions: GCP GCS

For GCP GCS, you can create a Key by referring to the GCP IAM > Creating and managing service account keys documentation.

The following permissions are required when creating a Key for GCS access.

storage.buckets.get
storage.objects.get
storage.objects.create
storage.objects.delete
storage.objects.list

Create Key and Grant Permissions: AWS S3

For AWS S3, you can create a Key and grant the necessary permissions by referring to the following documents.

1. Follow the AWS Docs: Creating an IAM Role documentation to create an AWS IAM User.

2. Follow AWS Docs: Creating IAM Policies to create a Policy that includes the IAM Policy named iam_policy.json attached below. Then add the IAM Policy to the IAM Role created in the previous step.

3. Follow AWS Docs: Configuring IAM STS to add the IAM STS policy named iam_sts.json attached below to the IAM Role you created. At this point, enter the ARN value provided by Hackle. Also, pass the sts:ExternalId value you specify directly to Hackle.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
              "s3:GetObject",
              "s3:GetObjectVersion",
              "s3:DeleteObject",
              "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::<bucket>/<prefix>/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::<bucket>"
        }
    ]
}

{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Effect": "Allow",
     "Principal": {
       "AWS": "[Hackle-provided_ARN]"
     },
     "Action": "sts:AssumeRole",
     "Condition": {
       "StringEquals": {
         "sts:ExternalId": "[Customer-specified_random_string]"
       }
     }
   }
 ]
}

Data Export Sample

When raw data sent to Hackle is exported, it is delivered in Apache Parquet format. Below is the schema for the Parquet format data.

root
 |-- server_dt: date (nullable = false)
 |-- ts: timestamp (nullable = false)
 |-- environment: string (nullable = false)
 |-- event_key: string (nullable = false)
 |-- identifiers: string (nullable = false)
 |-- insert_id: string (nullable = false)
 |-- metric_value: decimal(24,6) (nullable = false)
 |-- hackle_properties: string (nullable = false)
 |-- user_properties: string (nullable = false)
 |-- event_properties: string (nullable = false)

Data Export Request

Please contact Hackle to request Data Export. The following information is required for Data Export.

• Key with granted access permissions

• AWS S3, GCS Bucket name and partition path within the bucket where data will be stored (example: gcs://customer-data-hackle/prefix-custom/ — the path below this is defined by Hackle)