Debugging Syslog Integration

This guide provides troubleshooting steps for Syslog integration issues with the Traceable Platform.

Overview

When integrating Syslog with Traceable, you may encounter connection failures or notification delivery issues. This guide helps diagnose and resolve common problems.

---

Test Connection Failure

If you get a test connection failure error, follow these debugging steps:

Step 1: Validate Connectivity

Connect to the Syslog Server:

telnet <Syslog Host> <Syslog Port>

Trying <Syslog Host>...
Connected to <Syslog Host>.
Escape character is '^]'.

• If you got the successful message, then move ahead
• If failed, check with the Syslog admin if connectivity was opened for the Traceable platform or not

---

Step 2: Validate the Protocol

Try to curl over HTTP and HTTPS to the Syslog server.

A successful response from HTTPS:

curl -vk https://<Syslog Host>:<Syslog Port>
*   Trying <Syslog Host>:<Syslog Port>...
* Connected to <Syslog Host> (<Syslog Host>) port <Syslog Port>
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=Traceable; OU=Traceable; O=EngOps; L=San Mateo; ST=CA; C=US
*  start date: Apr 30 15:11:31 2021 GMT
*  expire date: Apr 30 15:11:31 2121 GMT
*  issuer: N=Traceable; OU=Traceable; O=EngOps; L=San Mateo; ST=CA; C=US
*  SSL certificate verify result: self-signed certificate in certificate chain (19), continuing anyway.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/1.x
> GET / HTTP/1.1
> Host: <Syslog Host>:<Syslog Port>
> User-Agent: curl/8.5.0
> Accept: */*
> 

A successful response from HTTP:

curl -vk http://<Syslog Host>:<Syslog Port>
*   Trying <Syslog Host>:<Syslog Port>...
* Connected to <Syslog Host> (<Syslog Host>) port <Syslog Port>
> GET / HTTP/1.1
> Host: <Syslog Host>:<Syslog Port>
> User-Agent: curl/8.5.0
> Accept: */*
> 

Important:

• If we get a response from an HTTP request, then it's a blocker as we do not support TCP/UDP-based Syslog for SaaS
• If we get a response from an HTTPS request, then move to the next step

---

Step 3: Use the Server CA Certificate

Download the certificate by running the below OpenSSL command:

openssl s_client -connect <Syslog Host>:<Syslog Port> 2>/dev/null </dev/null \
  |  sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'

Now, perform the test connection, providing the Server CA Certificate in the Traceable Platform integration settings.

---

Step 4: Provide the Token

Check with the Syslog team if you need a token to connect to the Syslog Server.

---

Step 5: Check Integration Service Logs

If all the above steps pass but the connection still fails, check the integration service logs for more insights:

kubectl logs -n traceable -l app=integration-service --tail=100

---

Failed to Validate Syslog (TimeoutException)

If you see the following error in the logs:

Stack trace:

2024-04-17 07:17:05.438 [grpc-default-executor-15] ERROR a.t.s.i.s.s.SyslogSender - Failed to validate syslog for tenant ID 7b5ddbea-f35f-4b23-80bb-d6fae36fb78d for server connection server_host: "10.164.0.4"
server_port: 514
log_format: SYSLOG_LOG_FORMAT_RFC_5424

java.util.concurrent.TimeoutException: null
 at java.base/java.util.concurrent.FutureTask.get(Unknown Source) ~[?:?]
 at com.google.common.util.concurrent.SimpleTimeLimiter.callWithTimeout(SimpleTimeLimiter.java:153) ~[guava-32.1.3-jre.jar:?]
 at ai.traceable.syslog.integration.service.sender.SyslogSender.lambda$validateConnection$3(SyslogSender.java:111) ~[syslog-integration-service-impl-0.4.86.jar:?]
 at io.micrometer.core.instrument.composite.CompositeTimer.recordCallable(CompositeTimer.java:129) ~[micrometer-core-1.10.2.jar:1.10.2]
 at ai.traceable.syslog.integration.service.sender.SyslogSender.validateConnection(SyslogSender.java:109) [syslog-integration-service-impl-0.4.86.jar:?]
 ...

Debugging Steps

1. Test connection from the pods:

Using curl:

curl -vk https://<Syslog Host>:<Port>

curl -vk http://<Syslog Host>:<Port>

Using wget:

wget https://<Syslog Host>:<Port>

wget http://<Syslog Host>:<Port>

If you see that we are not able to make a successful connection, then check with the network team to make sure that the connection can be made.

2. Configure HTTP instead of HTTPS:

Syslog integration makes the connection with HTTPS by default. If the connection works for HTTP but not HTTPS, you need to make a config change to use HTTP instead of HTTPS in the Integration service helm chart.

Fix: Update the Integration Service Helm values:

syslogIntegrationServiceConfig:
  ignoreSsl: true

Apply the change via TI UI: Helm Releases → Integration Service → Update user/values.yaml → Helm Upgrade

---

Notifications Not Received

If test connection succeeds but notifications are not being received:

Check Connectivity

1. Login to the platform
2. Perform the test connection again to validate that the configs are correct

Check Kafka Lag

1. Login to the Grafana dashboard for the Traceable Platform
2. Go to the Kafka exporter overview dashboard
3. Check the Lag metrics

Check Service Logs

Check the logs of the integration service and notification sender service:

# Integration service logs
kubectl logs -n traceable -l app=integration-service --tail=100

# Notification sender service logs
kubectl logs -n traceable -l app=notification-sender --tail=100

---

Common Issues

Issue	Cause	Solution
Test connection fails	Network connectivity blocked	Verify firewall rules allow traffic from Traceable to Syslog server
Test connection fails	Wrong protocol	Ensure TLS is used for SaaS; TCP/UDP not supported
Test connection fails	Certificate issues	Provide Server CA Certificate in integration settings
Notifications not received	Kafka lag	Check Grafana dashboard for consumer lag
Notifications not received	Service issues	Check integration-service and notification-sender logs

Refer SysLog Connection Matrix

---

Related Documentation

• Syslog Setup
• AlertManager Setup

---

Support

For Syslog integration assistance, contact: support@harness.io

Provide the following information:

• Syslog server host and port
• Protocol used (TCP/TLS)
• Test connection results
• Integration service logs
• Network connectivity test results