# Splunk

Connecting Splunk gives SureCloud read access to your Splunk deployment's saved searches, alert configurations, index summaries, and audit log data. SureCloud verifies that security-relevant saved searches and alerts are active and scheduled, checks that critical log sources are being ingested into the expected indexes, and collects Splunk's internal audit log to confirm that administrative activity within the platform is captured. This provides continuous evidence that your SIEM is functioning as a reliable security event aggregation and alerting layer.

{% hint style="info" %}
SureCloud connects to Splunk using a Bearer Token (Splunk authentication token or service account session token). For Splunk Cloud Platform, token-based authentication is the supported method. For Splunk Enterprise on-premises, a service account token or username/password with token generation is used.
{% endhint %}

## Authentication and setup

{% tabs %}
{% tab title="Splunk Cloud Platform" %}
{% stepper %}
{% step %}

#### Create a service account user

In the Splunk Cloud Platform console, navigate to **Settings → Users and Authentication → Users → New User** and create a dedicated service account (e.g. `surecloud-ccm`).

Assign the following roles to the service account:

| Role         | Purpose                                   |
| ------------ | ----------------------------------------- |
| `user`       | Basic search and saved search read access |
| `can_delete` | Not required — do not assign              |
| `power`      | Not required — do not assign              |

A custom role with `list_inputs`, `list_storage_passwords`, `search`, and `get_metadata` capabilities is recommended over the built-in `power` role. See the Required Permissions section below.
{% endstep %}

{% step %}

#### Generate an authentication token

Log in to Splunk Web as the `surecloud-ccm` service account. Navigate to **Settings → Users and Authentication → Tokens → New Token**.

* **User**: `surecloud-ccm`
* **Expiry**: set an expiry date and record a reminder to rotate before this date
* **Audience**: `SureCloud CCM`

Click **Create** and copy the token value immediately.

{% hint style="warning" %}
Splunk authentication tokens expire on the date you set. If the token expires without being rotated in SureCloud, data collection will stop. Update the rotated token in **SureCloud → Integrations → Splunk → Edit Connection** before expiry.
{% endhint %}
{% endstep %}

{% step %}

#### Enter credentials in SureCloud

In SureCloud, navigate to **Integrations → Splunk → Connect** and provide:

* **Splunk base URL** (e.g. `https://yourinstance.splunkcloud.com:8089`)
* **Bearer Token**

Click **Test Connection**, then **Save**.
{% endstep %}
{% endstepper %}
{% endtab %}

{% tab title="Splunk Enterprise (on-premises)" %}
{% stepper %}
{% step %}

#### Create a service account and custom role

In Splunk Enterprise, navigate to **Settings → Users and Authentication → Roles → New Role** and create a role named `surecloud_readonly` with the following capabilities:

| Capability             | Purpose                                                           |
| ---------------------- | ----------------------------------------------------------------- |
| `search`               | Run searches against indexes                                      |
| `list_inputs`          | Read input (data source) configuration                            |
| `get_metadata`         | Read index metadata and summary statistics                        |
| `rest_apps_management` | Read saved search and alert configurations                        |
| `admin_all_objects`    | Read all objects across all apps (restrict to read-only contexts) |

Create a user (e.g. `surecloud-ccm`) and assign the `surecloud_readonly` role.
{% endstep %}

{% step %}

#### Generate an authentication token

In **Settings → Users and Authentication → Tokens**, generate a token for the `surecloud-ccm` account with an appropriate expiry.

Alternatively, SureCloud can use HTTP Basic Authentication (username and password) for Splunk Enterprise. Token-based authentication is preferred as it avoids storing credentials and supports rotation without changing the account password.
{% endstep %}

{% step %}

#### Confirm the Splunk REST API port is accessible

SureCloud communicates with Splunk via the REST API on port `8089` (default). Confirm that port 8089 is accessible from SureCloud's IP ranges on your network or firewall rules.

In SureCloud, navigate to **Integrations → Splunk → Connect** and provide:

* **Splunk base URL** (e.g. `https://splunk.yourcompany.com:8089`)
* **Bearer Token** (or username and password if token auth is not available)

Click **Test Connection**, then **Save**.
{% endstep %}
{% endstepper %}
{% endtab %}
{% endtabs %}

## Endpoints

| API Call                              | Use Case                                                                                             |
| ------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| `GET /services/saved/searches`        | Enumerate all saved searches and alerts; verify security-relevant searches are enabled and scheduled |
| `GET /services/saved/searches/{name}` | Read full saved search configuration including schedule, alert actions, and suppression settings     |
| `GET /services/data/indexes`          | List all indexes and their retention periods (frozen/cold/warm archive thresholds)                   |
| `GET /services/data/inputs/all`       | Enumerate configured data inputs; verify expected log sources are active                             |
| `GET /services/search/jobs`           | Read recent search job history (used to verify scheduled searches are executing)                     |
| `GET /services/admin/audit`           | Read Splunk internal audit log for administrative activity within the Splunk platform                |
| `GET /services/authentication/users`  | Enumerate Splunk users and their role assignments                                                    |
| `GET /services/authorization/roles`   | List all roles and their capabilities                                                                |
| `GET /services/apps/local`            | Enumerate installed Splunk apps; detect apps relevant to security content (ES, ITSI)                 |

## Pagination

The Splunk REST API uses offset-based pagination via `offset` and `count` query parameters. SureCloud requests pages of up to 100 results using `count=100` and increments the `offset` by 100 until the response `paging.total` count is reached.

```json
{
  "paging": {
    "total": 284,
    "perPage": 100,
    "offset": 100
  },
  "entry": [
    { "name": "Security - Failed Logins - Alert", "content": { "disabled": false, "cron_schedule": "*/15 * * * *", ... } }
  ]
}
```

## Required permissions

| Capability             | Purpose                                    |
| ---------------------- | ------------------------------------------ |
| `search`               | Execute read-only searches against indexes |
| `list_inputs`          | Read data input configuration              |
| `get_metadata`         | Read index metadata and summary            |
| `rest_apps_management` | Read saved search and alert configuration  |
| `list_settings`        | Read Splunk server configuration settings  |

For Splunk Cloud Platform, assigning the built-in `sc_admin` role is not recommended — use a custom role with only the capabilities listed above.

## Polling frequency

| Data type                                  | Collection interval |
| ------------------------------------------ | ------------------- |
| Saved search and alert configuration       | 24 hours            |
| Index configuration and retention settings | 24 hours            |
| Data input status                          | 24 hours            |
| Splunk internal audit log                  | 1 hour              |
| User and role inventory                    | 24 hours            |

## Troubleshooting

<details>

<summary>Test Connection fails with "Connection refused" or a timeout</summary>

The Splunk REST API port (default 8089) is not reachable from SureCloud. This is most commonly a network or firewall issue on Splunk Enterprise deployments.

1. Confirm port 8089 is open on the host running Splunk Enterprise and is reachable from SureCloud's outbound IP ranges.
2. For Splunk Cloud Platform, confirm the instance URL includes the port (e.g. `https://yourinstance.splunkcloud.com:8089`) and that Splunk Cloud management access is not restricted to specific IP allowlists that exclude SureCloud.
3. Test connectivity manually: `curl -k https://splunk.yourcompany.com:8089/services` should return a Splunk XML response.

</details>

<details>

<summary>Saved searches are appearing but alert configuration details are missing</summary>

Alert configuration properties (notification targets, suppression settings, trigger conditions) are returned in the full saved search detail endpoint. If these fields are missing, the service account may lack the `rest_apps_management` capability, which is required to read saved search content across all apps.

In Splunk, confirm the `surecloud_readonly` role has `rest_apps_management` granted, then trigger a manual refresh from **SureCloud → Integrations → Splunk → Edit Connection**.

</details>

<details>

<summary>Index retention periods are not appearing for some indexes</summary>

Index retention settings (`frozenTimePeriodInSecs`, `maxTotalDataSizeMB`) are returned in the `/services/data/indexes` response. If they are missing for specific indexes, those indexes may be virtual (summary indexes, acceleration data model indexes) or managed by a Splunk app that restricts their metadata via app-level access controls.

In Splunk, navigate to **Settings → Indexes** and confirm the indexes are listed with their retention configuration. App-scoped indexes may require the service account to be granted access within the relevant Splunk app.

</details>

<details>

<summary>Authentication token has expired and data collection has stopped</summary>

Splunk authentication tokens expire on the date set at creation. Once expired, all API calls will return a 401 Unauthorized response.

Generate a new token for the `surecloud-ccm` account in **Splunk → Settings → Users and Authentication → Tokens**, then update it in **SureCloud → Integrations → Splunk → Edit Connection → Bearer Token**. Set a calendar reminder for the new expiry date.

</details>

<a href="https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTprolog" class="button secondary">Splunk REST API reference</a> <a href="https://docs.splunk.com/Documentation/Splunk/latest/Security/UseAuthTokens" class="button secondary">Splunk authentication token documentation</a>


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://surecloud.gitbook.io/surecloud-docs/integrations/ccm-and-evidence-collection-integrations/splunk.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.