# Elastic

SureCloud connects to Elastic to retrieve index and data stream metadata, alerting rules, and security analytics configuration from your Elastic deployment. This allows SureCloud to verify that log ingestion is active, that security alerting rules are in place, and that audit log data is being retained for the required period — providing evidence for log management and security monitoring controls.

## Authentication and setup

Elastic supports API key authentication for programmatic access. SureCloud uses a read-only Elastic API key scoped to the indices and features it needs to monitor.

{% tabs %}
{% tab title="Elastic Cloud (ESS / ECE)" %}
{% stepper %}
{% step %}
**Create an API key in Kibana**

In your Elastic Cloud deployment, navigate to **Kibana → Stack Management → Security → API Keys**. Click **Create API key**. Give it a name such as `SureCloud CCM` and set an appropriate expiry. Under **Privileges**, restrict the key to read-only access on the relevant index patterns (see Required permissions below). Click **Create** and copy the encoded API key.
{% endstep %}

{% step %}
**Enter the API key in SureCloud**

Navigate to **SureCloud → Integrations → Elastic → Connect**. Enter your Elastic Cloud endpoint URL (e.g. `https://<deployment-id>.es.<region>.aws.elastic-cloud.com`) and paste the API key. Click **Save**.
{% endstep %}

{% step %}
**Test the connection**

Click **Test Connection**. SureCloud will call `GET /` to verify connectivity and confirm the API key has the required index privileges.
{% endstep %}
{% endstepper %}
{% endtab %}

{% tab title="Self-managed Elasticsearch" %}
{% stepper %}
{% step %}
**Create an API key via the Elasticsearch API**

On your self-managed Elasticsearch cluster, create a restricted API key using the Create API Key API:

```json
POST /_security/api_key
{
  "name": "surecloud-ccm",
  "role_descriptors": {
    "surecloud_reader": {
      "cluster": ["monitor"],
      "indices": [
        {
          "names": ["*"],
          "privileges": ["monitor", "read", "view_index_metadata"]
        }
      ]
    }
  }
}
```

Copy the `encoded` value from the response.
{% endstep %}

{% step %}
**Enter the API key in SureCloud**

Navigate to **SureCloud → Integrations → Elastic → Connect (Self-managed)**. Enter your Elasticsearch base URL and the encoded API key. Click **Save** and then **Test Connection**.
{% endstep %}
{% endstepper %}

{% hint style="warning" %}
Store the API key securely after copying it. SureCloud encrypts it at rest, but you should treat it as a credential. Rotate it before expiry and update it under **SureCloud → Integrations → Elastic → Edit Connection**.
{% endhint %}
{% endtab %}
{% endtabs %}

## Endpoints

| API Call                                | Use Case                                                                 |
| --------------------------------------- | ------------------------------------------------------------------------ |
| `GET /`                                 | Verify cluster connectivity and version during connection test           |
| `GET /_cat/indices?format=json`         | Enumerate indices to confirm log sources are ingesting data              |
| `GET /_data_stream`                     | List data streams (Elastic 7.9+) used for log and metrics storage        |
| `GET /_cat/indices/{index}?format=json` | Retrieve index-level metadata including document count and storage size  |
| `GET /_index_lifecycle/policy`          | Retrieve ILM policies to verify log retention configuration              |
| `GET /api/alerting/rules/_find`         | List alerting rules configured in Kibana to confirm monitoring is active |
| `GET /api/detection_engine/rules/_find` | List Security detection rules to confirm threat detection is configured  |
| `GET /_security/user`                   | Enumerate Elasticsearch users and their assigned roles                   |
| `GET /_security/role`                   | Retrieve role definitions to assess access controls on log data          |

## Pagination

`GET /api/alerting/rules/_find` and `GET /api/detection_engine/rules/_find` use page-based pagination. SureCloud sets `page` and `per_page` parameters and increments until the total returned results match the `total` field in the response.

```json
GET /api/alerting/rules/_find?page=2&per_page=100
```

`GET /_cat/indices` and `GET /_data_stream` return all results in a single response for most deployments. For very large clusters, SureCloud uses wildcard patterns to scope collection by log type.

## Required permissions

SureCloud requires a role (or equivalent API key privilege set) with the following Elasticsearch privileges:

```json
{
  "cluster": ["monitor"],
  "indices": [
    {
      "names": ["*"],
      "privileges": ["monitor", "read", "view_index_metadata"]
    }
  ],
  "applications": [
    {
      "application": "kibana-.kibana",
      "privileges": ["feature_alerting.read", "feature_siem.read"],
      "resources": ["*"]
    }
  ]
}
```

No write, delete, or cluster management privileges are required.

## Polling frequency

| Data Type                       | Interval       |
| ------------------------------- | -------------- |
| Index and data stream inventory | Every 24 hours |
| ILM retention policies          | Every 24 hours |
| Alerting and detection rules    | Every 24 hours |
| User and role configuration     | Every 24 hours |

## Troubleshooting

<details>

<summary>Connection test fails with 401 Unauthorized or "security_exception"</summary>

The API key is invalid, has expired, or does not have `monitor` cluster privilege. Verify the key is active in **Kibana → Stack Management → Security → API Keys**. If it has expired, create a new key with the required privileges and update it in **SureCloud → Integrations → Elastic → Edit Connection**.

</details>

<details>

<summary>Alerting rules or detection rules are missing from SureCloud</summary>

The Kibana alerting and detection engine APIs are separate from the Elasticsearch REST API. If SureCloud cannot reach these endpoints, confirm that the Kibana URL is entered correctly and that the API key has `feature_alerting.read` and `feature_siem.read` Kibana application privileges. For self-managed deployments, ensure Kibana is accessible from the SureCloud collector IP range.

</details>

<details>

<summary>Index data appears stale or document counts seem lower than expected</summary>

SureCloud reads index metadata, not raw log content. Document counts reflect what Elasticsearch reports at the time of polling. If ILM rollover policies are configured with short retention windows, older indices may have been deleted. Check your ILM policy definitions in Kibana under **Stack Management → Index Lifecycle Policies** to confirm retention periods align with your compliance requirements.

</details>

<details>

<summary>Self-managed cluster is unreachable</summary>

Confirm that the Elasticsearch HTTP endpoint (default port 9200) is accessible from the SureCloud collector network. Check firewall rules and ensure TLS certificates are valid if HTTPS is enforced. For private clusters, a network integration or tunnelling arrangement may be required — contact SureCloud Support for guidance.

</details>


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://surecloud.gitbook.io/surecloud-docs/integrations/ccm-and-evidence-collection-integrations/elastic.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.