Traceable On-Prem Alerts Runbook¶

This runbook provides guidance for responding to alerts from the Traceable Platform monitoring system.

Overview¶

The Traceable Platform includes built-in monitoring with Prometheus and AlertManager. This runbook covers common alerts, their causes, and mitigation steps.

SEV1 Alerts (Critical)¶

KubePersistentVolumeFillingUp¶

Severity: CRITICAL

Alert Rule:

(kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes) < 0.03

Description: Persistent volume is running out of space (less than 3% available).

Impact: Services may fail to write data, causing data loss or service outages.

Mitigation Steps:

Identify the affected PVC:
- Navigate to TI → Namespaces → Browser → PVC
- Find the PVC mentioned in the alert
Check current utilization:

kubectl exec -it -n traceable <pod-name> -- df -h <mount-path>

Analyze disk usage in Grafana:
- Dashboard: "Kubernetes / Persistent Volumes"
- Dashboard: "Pinot Query Ingestion" (for Pinot volumes)
Increase PVC size:
- Navigate to TI → Namespace → Browser → PVC
- Click edit icon on the affected PVC
- Increase the storage value
- Save changes

Escalation: If mitigation fails, collect Grafana metrics, thread dumps, and logs for Traceable support.

OldGenGCOverhead¶

Severity: CRITICAL

Alert Rule:

delta(java_lang_g1_old_generation_collectiontime{pod=~"pinot-.*"}[1m]) > 15000
for: 5m

Description: High Old Generation GC overhead in Pinot components for more than 5 minutes.

Causes:

Complex regex queries
Large time range queries with filters/group by
Memory-intensive operations

Impact: Platform slowness, query timeouts, potential service instability.

Mitigation Steps:

Identify problematic queries:
- Check Pinot Broker and Server logs
- Look for long-running or complex queries
Restart affected pods:

kubectl rollout restart statefulset -n traceable pinot-broker
kubectl rollout restart statefulset -n traceable pinot-server

If issue persists:
- Increase Pinot memory limits
- Review and optimize problematic queries

Escalation: Collect Grafana metrics, thread dumps, and Pinot logs.

PinotServerHighCPUUsage¶

Severity: CRITICAL

Alert Rule:

sum by (namespace, pod) (container_cpu_usage) > sum by (namespace, pod) (kube_pod_container_resource_requests{container="pinot-server"})
for: 15m

Description: Pinot server CPU usage exceeds requested resources for 15+ minutes.

Causes:

Increased ingestion traffic
Lag in upstream services (e.g., trace-enricher)
Complex query processing

Mitigation Steps:

Check for upstream lag:
- Review "Kafka Stream Applications" Grafana dashboard

Increase resources:

Navigate to TI → Helm Releases → Search for Pinot
Update user/values.yaml:

hypertrace-pinot:
  server:
    replicaCount: 4
    resources:
      limits:
        cpu: 2
        memory: 9Gi
      requests:
        cpu: 1

Save and run Helm Upgrade

Scale horizontally if vertical scaling is insufficient.

PinotKafkaConsumerError / PinotKafkaConsumerLag¶

Severity: CRITICAL

Description: Pinot is experiencing Kafka consumer errors or significant lag.

Mitigation Steps:

Check Kafka connectivity:

kubectl exec -it -n traceable pinot-server-0 -- nc -zv kafka 9092

Review consumer lag in Grafana:
- Dashboard: "Kafka Stream Applications"
Restart Pinot consumers if needed:

kubectl rollout restart statefulset -n traceable pinot-server

Generate support bundle:

name: CollectLogs
namespaces:
  - match:
      filter: "traceable"
    logs:
      - pod:
          filter: "pinot-*"
      - pod:
          filter: "kafka*"

SEV2 Alerts (Warning)¶

PodOutOfMemory¶

Alert Rule:

kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} > 0

Description: Pod was terminated due to Out of Memory (OOM) condition.

Mitigation Steps:

Identify the pod:

kubectl get pods -n traceable | grep -i oom
kubectl describe pod <pod-name> -n traceable

Check memory usage:

kubectl top pod <pod-name> -n traceable

Increase memory limits:
- Navigate to TI → Helm Releases → Find the service
- Update memory limits in user/values.yaml
- Run Helm Upgrade
Monitor after changes:
- Verify pod stability
- Check memory usage trends in Grafana

KubePodCrashLooping¶

Description: Pod is repeatedly crashing and restarting.

Mitigation Steps:

Check pod status and events:

kubectl get events -n traceable | grep <pod-name>
kubectl describe pod <pod-name> -n traceable

Check logs:

kubectl logs <pod-name> -n traceable --previous

Common causes:
- OOM kills (exit code 137)
- Health check failures
- Configuration errors
- Dependency unavailable
Generate support bundle:

name: generic-pod-restart
namespaces:
  - match:
      filter: "traceable"
    logs:
      - pod:
          filter: "<pod-name>*"

PodRestarted¶

Alert Rule:

delta(kube_pod_container_status_restarts_total[6h]) > 5

Description: Pod has restarted more than 5 times in 6 hours.

Mitigation Steps:

Check pod events:

kubectl get events -n traceable | grep <pod-name>
kubectl describe pod <pod-name> -n traceable

Analyze restart reasons:
- Exit code 137: OOM kill
- Exit code 1: Application error
- Probe failures: Health check issues
Review logs from previous instance:

kubectl logs <pod-name> -n traceable --previous

Pinot Offline Segment Delay¶

Alert Names:

metricView_OFFLINE-pinot-offline-segment-delay
serviceCallView_OFFLINE-pinot-offline-segment-delay
domainEventView_OFFLINE-pinot-offline-segment-delay
(and other table variants)

Alert Rule:

max by (table) (pinot_controller_validation_offlinesegmentdelayhours_value) > 60

Description: Pinot offline table watermark is behind by more than 60 hours.

Impact:

Historical queries may return incomplete results
Must be resolved within 3 days to prevent data loss

Mitigation Steps:

Generate support bundle:

name: CollectLogs
namespaces:
  - match:
      filter: "traceable"
    logs:
      - pod:
          filter: "pinot-*"
      - pod:
          filter: "zookeeper*"

Contact Traceable support with the bundle.

connect-task-failure¶

Alert Rule:

sum by (connector) (kafka_connect_connect_worker_metrics_connector_failed_task_count) > 0

Description: Kafka Connect task has failed.

Mitigation Steps:

Check connector status:

curl -s http://kafka-connect.traceable.svc:8083/connectors | jq

Get detailed status:

curl -s http://kafka-connect.traceable.svc:8083/connectors/<connector-name>/status | jq

Restart failed tasks:

curl -X POST http://kafka-connect.traceable.svc:8083/connectors/<connector-name>/tasks/0/restart

Generate support bundle:

name: kafka-connect-bundle
namespaces:
  - match:
      filter: "traceable"
    logs:
      - pod:
          filter: "kafka-connect-*"

EntityCreationThresholdExceeded¶

Alert Names:

ApiEntityCreationThresholdExceeded
DomainEntityCreationThresholdExceeded
ServiceEntityCreationThresholdExceeded
BackendEntityCreationThresholdExceeded

Alert Rule:

max by (tenantId) (entities_count{entityType="DOMAIN"}) > 100000

Description: More than 100,000 entities created in a category.

Mitigation Steps:

Check entity counts:

kubectl exec -it -n traceable mongo-primary-0 -- mongo default_db --quiet --eval "
  print('DOMAIN: ' + db.raw_entities.count({entityType: 'DOMAIN'}));
  print('API: ' + db.raw_entities.count({entityType: 'API'}));
  print('SERVICE: ' + db.raw_entities.count({entityType: 'SERVICE'}));
"

Contact Traceable support for entity cleanup guidance.

HighUnsuccessfulResponseCodeIntegrationService¶

Alert Rule:

sum by (statusCode) (delta(grpc_server_processing_duration_seconds_count{app="integration-service",statusCode!="OK"}[1h])) > 10

Description: Integration service has more than 10 unsuccessful calls in the past hour.

Mitigation Steps:

Check integration-service pods:

kubectl get pods -n traceable -l app=integration-service
kubectl logs -n traceable -l app=integration-service --tail=100

Review integration configurations (WAF, Jira, SIEM, etc.)
Generate support bundle:

name: integration-service
namespaces:
  - match:
      filter: "traceable"
    logs:
      - pod:
          filter: "integration-service*"

How to Silence an Alert¶

Open AlertManager UI:
- Navigate to TI → Monitoring → AlertManager
- Or access directly: https://<platform-dns>/alertmanager
Select the alert(s) you want to silence
Click "Silence"
Set duration:
- 1h for 1 hour
- 1d for 1 day
- 1w for 1 week
Create the silence

The silence will automatically expire after the specified duration.

Generating Support Bundles¶

For escalation to Traceable support, generate a support bundle:

Navigate to TI → Tasks → Support Bundle
Use appropriate YAML configuration (examples provided in each alert section)
Download and share the generated bundle with support

Support¶

For alert assistance, contact: support@harness.io

Provide the following information:

Alert name and details
Support bundle
Grafana screenshots
Timeline of events

Traceable On-Prem Alerts Runbook¶

Overview¶

SEV1 Alerts (Critical)¶

KubePersistentVolumeFillingUp¶

OldGenGCOverhead¶

PinotServerHighCPUUsage¶

PinotKafkaConsumerError / PinotKafkaConsumerLag¶

SEV2 Alerts (Warning)¶

PodOutOfMemory¶

KubePodCrashLooping¶

PodRestarted¶

Pinot Offline Segment Delay¶

connect-task-failure¶

EntityCreationThresholdExceeded¶

HighUnsuccessfulResponseCodeIntegrationService¶

How to Silence an Alert¶

Generating Support Bundles¶

Related Documentation¶

Support¶