# Datadog Integration

Datadog is a SaaS-based integrated monitoring and analytics platform for large-scale applications and infrastructure.

You can track real-time metrics for servers, containers, databases, and applications, and by integrating and analyzing these with Hackle data, you can monitor your service during the gradual rollout of Feature Flags.

### Datadog Integration

To integrate with Datadog, you need to enter the Datadog API Key information in the Hackle Dashboard.

#### Copying the Datadog API Key

Go to Datadog and find the API Key under Organization Settings > API Keys. The Organization Settings menu can be accessed by hovering over your email account in the lower left corner.

* API Keys menu URL: <https://app.datadoghq.com/organization-settings/api-keys>

![Organization Settings > API Keys](/files/ot7Fk9Gg7RhLyeTPfaWP)

#### Entering the Datadog API Key in the Hackle Dashboard

Enter the copied Datadog API Key in the Hackle Dashboard. Click the Settings icon in the upper right corner, then configure the Datadog integration under the External Integration menu.

On the Datadog integration screen shown in the image below, enter the API Key and click the 'Integrate' button to complete the configuration.

![Settings (located in the upper right GNB) > External Integration](/files/JZBVX9iAahle1a5iRUfB)

### Items Sent to Datadog

Once the Datadog integration is complete, Feature Flag exposure events that occur in the Production Environment will be sent. Other events can also be sent; if configuration is needed, please contact the Hackle team.

| Category              | Default                                                                   | Selectable Options                                          |
| --------------------- | ------------------------------------------------------------------------- | ----------------------------------------------------------- |
| Events                | Feature Flag events                                                       | Feature Flag, A/B Test, Remote Config, user behavior events |
| Configuration changes | Feature Flag rollout percentage (Rollout %), Feature Flag status (On/Off) | -                                                           |
| Environment           | Production Environment                                                    | Production Environment, Development Environment             |

### What You Can Monitor in Datadog

You can create a Dashboard using the Hackle Feature Flag events and other data sent to Datadog. You can build a comprehensive monitoring Dashboard by combining monitoring charts generated from Datadog-collected data with Feature Flag charts generated from Hackle data.

![](/files/Do9ymOSGsyocCGYlecdz)

| Widget Name in Screenshot Above                 | Description                                                                                                                                      |
| ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| Feature Flag Rollout configuration change count | The number of times a Feature Flag status (On/Off), rollout percentage (Rollout %), or targeting condition (Targeting Rule) change has occurred. |
| Feature Flag distribution event count           | The number of times a Feature Flag distribution event has occurred.                                                                              |
| Feature Flag Rollout configuration changes      | Shows the history of Feature Flag status, rollout percentage, and targeting condition changes made in the Hackle Dashboard.                      |

#### Monitoring Feature Flag Distribution Events in Datadog

{% hint style="info" %}
Go to Add Widgets > Graphs > Timeseries.
{% endhint %}

You can check where and how many Feature Flag calls occurred at a specific point in time to determine how many customers were impacted. By selecting `Metrics` > `hackle.feature_flag.count`, you can create a widget to view the number of Feature Flag distribution events. You can also use the following variables to monitor Feature Flag distribution events under specific conditions.

* key: Feature Flag key (e.g., 15)
* variation: Distribution result (true/false)
* decision\_reason: Distribution reason [Feature Flag Decision](/en/development-guide/java-kotlin/java-feature-flag.md)
* environment: Environment (production/development)

![Example Datadog Widget configuration for monitoring Feature Flag distribution event count](/files/u5MahxcZ5yN2LHuMN1Px)

#### Monitoring Feature Flag Rollout Configuration Changes in Datadog

{% hint style="info" %}
Go to Add Widgets > Graphs > Timeseries or List and Streams > List.
{% endhint %}

You can identify which Feature Flag had its Rollout conditions changed at a specific point in time, which helps you estimate causal relationships with elements you are monitoring in Datadog. By entering `Events` > `source:hackle`, an event is sent to Datadog every time an engineer changes a Feature Flag's status, rollout percentage, or targeting condition in the Hackle Dashboard. (e.g., changing Feature Flag rollout from 0% to 30%)

![Example Datadog Widget (Graph > Timeseries) configuration for monitoring Feature Flag Rollout configuration change count](/files/asuICQPmivigG4YkTHDE)

To see who changed a specific Feature Flag's Rollout conditions, when, and how, you can create a List-type Widget.

![Example Datadog Widget (List and Streams > List) configuration for viewing Feature Flag Rollout configuration change history](/files/Co3NJpzhn3YYHrV5WAdN)

By selecting `Reporting format size` as `Large (full event)`, you can view detailed information about Feature Flag Rollout configuration changes in the Widget as shown in the image below.

![](/files/IyV8KwD4pWqvKIuypxHV)


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.hackle.io/en/external-link/datadog-integration.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.