Skip to content

Debugging Splunk Integration

This guide provides troubleshooting steps for Splunk integration issues with the Traceable Platform.

Overview

When integrating Splunk with Traceable, you may encounter connection failures or event delivery issues. This guide helps diagnose and resolve common problems.

Splunk Running on HTTP (Not HTTPS)

If you see the following error:

Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: 
INVALID_ARGUMENT: URL configured in webhook is not https

Cause

By default, Traceable requires HTTPS for Splunk integration. If your Splunk instance is running on HTTP, the integration will fail.

Fix

Update the following Helm values to enable HTTP support:

Config Service:

traceableConfigServiceConfig:
  splunkIntegrationConfigServiceConfig:
    splunkHttpSupportEnabled: true

Integration Service:

splunkIntegrationServiceConfig:
  httpSupportEnabled: true

Apply the changes via TI UI:

  1. Navigate to Helm Releases → Config Service → Update user/values.yaml → Helm Upgrade
  2. Navigate to Helm Releases → Integration Service → Update user/values.yaml → Helm Upgrade

Data Channel is Missing Error

If you see the following error when testing the connection:

{"text":"Data channel is missing","code":10}

Cause

This error occurs when "Enable Indexer Acknowledgement" is enabled on the Splunk HEC token.

Troubleshooting Steps

1. Test connectivity from the Traceable server:

curl -kv "https://<splunk-hec-url>/services/collector/raw" \
  -H "Authorization: Splunk <your-hec-token>" \
  -d '{"event": "Test Event from Traceable", "sourcetype": "traceable"}'

Example:

curl -kv "https://http-inputs-mizuhous.splunkcloud.com/services/collector/raw" \
  -H "Authorization: Splunk 47121a87-ef52-420e-9c70-453cf50bcad6" \
  -d '{"event": "Test Event from Traceable", "sourcetype": "traceable"}'

If you receive the "Data channel is missing" error, proceed to the fix below.

Fix

  1. Login to Splunk
  2. Navigate to Settings → Data Inputs → HTTP Event Collector
  3. Click on the HEC token being used for Traceable integration
  4. Uncheck "Enable Indexer Acknowledgement"
  5. Save the changes

Reference: Splunk Community - Data channel is missing error

After implementing this fix, the test connection should succeed.

Connection Test Failures

Step 1: Verify Network Connectivity

Test connectivity from the Traceable platform to Splunk:

curl -kv "https://<splunk-hec-url>/services/collector/raw" \
  -H "Authorization: Splunk <your-hec-token>" \
  -d '{"event": "Test Event", "sourcetype": "traceable"}'

Expected successful response:

{"text":"Success","code":0}

Step 2: Verify HEC Token

Ensure the HEC token is:

  • Valid and not expired
  • Has the correct permissions
  • Is associated with the correct index

Step 3: Check TLS/SSL

If using HTTPS, verify:

  • The Splunk certificate is valid
  • The certificate chain is complete
  • The Traceable platform can verify the certificate
openssl s_client -connect <splunk-host>:443 -servername <splunk-host>

Step 4: Check Integration Service Logs

kubectl logs -n traceable -l app=integration-service --tail=100

Common Issues

Issue Error Solution
HTTP not supported URL configured in webhook is not https Enable httpSupportEnabled: true in config and integration services
Data channel missing {"text":"Data channel is missing","code":10} Disable "Enable Indexer Acknowledgement" in Splunk HEC settings
Connection timeout Connection timed out Check network connectivity and firewall rules
Invalid token {"text":"Invalid token","code":4} Verify HEC token is correct and active
Forbidden {"text":"Forbidden","code":403} Check token permissions and index access

Splunk HEC Response Codes

Code Text Description
0 Success Event received successfully
1 Token disabled HEC token is disabled
2 Token required No token provided
3 Invalid authorization Token format is invalid
4 Invalid token Token does not exist
5 No data No event data in request
6 Invalid data format Event data is malformed
7 Incorrect index Token cannot write to specified index
10 Data channel is missing Indexer acknowledgement enabled but no channel specified
11 Invalid data channel Invalid channel value
12 Event field required Event field is missing
13 Event field blank Event field is empty
14 ACK disabled Acknowledgement is disabled

Support

For Splunk integration assistance, contact: support@harness.io

Provide the following information:

  • Splunk HEC URL
  • Error messages from Traceable UI
  • Integration service logs
  • Curl test results from Traceable server