Setup AlertManager to Send Grafana Alerts

This guide explains how to configure Prometheus AlertManager to send Grafana alerts through various notification channels including Slack, email (SMTP), webhooks, and MS Teams.

Overview

AlertManager handles alerts sent by Prometheus and routes them to the appropriate notification channels. It supports grouping, deduplication, silencing, and routing of alerts to various receivers.

Prerequisites

Before configuring AlertManager, ensure you have:

• Traceable Platform installed with Prometheus Operator
• Access to modify platform configuration
• Notification channel credentials (Slack webhook, SMTP credentials, etc.)
• kubectl access to the monitoring namespace

---

Enable AlertManager

To enable AlertManager in your Traceable Platform deployment, add the following configuration:

prometheus:
  alertmanager:
    enabled: true

---

Integration Options

1. Slack Integration

Configure AlertManager to send alerts to Slack channels.

Basic Slack Configuration

prometheus:
  alertmanager:
    enabled: true
    config:
      global:
        slack_api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'

Complete Slack Configuration with Routing

prometheus:
  alertmanager:
    enabled: true
    config:
      global:
        slack_api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'
      receivers:
      - name: slack
        slack_configs:
        - channel: '#alerts'
          title: 'Traceable Platform Alert'
          text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'
      - name: "null"
      route:
        receiver: slack
        routes:
        - matchers:
          - alertname = Watchdog
          receiver: "null"
        - matchers:
          - alertname = InfoInhibitor
          receiver: "null"
        - receiver: slack

Getting Slack Webhook URL:

1. Go to your Slack workspace
2. Navigate to Apps → Incoming Webhooks
3. Create a new webhook for your channel
4. Copy the webhook URL

---

2. SMTP Integration (Email)

Configure AlertManager to send email notifications via SMTP.

SendGrid Example

prometheus:
  alertmanager:
    enabled: true
    config:
      global:
        slack_api_url: 'https://hooks.slack.com/'
        smtp_from: 'alerts@yourdomain.com'
        smtp_smarthost: 'smtp.sendgrid.net:587'
        smtp_auth_username: 'apikey'
        smtp_auth_password: '<your-sendgrid-api-key>'
        smtp_require_tls: true
      receivers:
      - name: email
        email_configs:
        - to: 'team@yourdomain.com,oncall@yourdomain.com'
      - name: "null"
      route:
        receiver: email
        routes:
        - matchers:
          - alertname = Watchdog
          receiver: "null"
        - matchers:
          - alertname = InfoInhibitor
          receiver: "null"
        - receiver: email

Generic SMTP Configuration

prometheus:
  alertmanager:
    enabled: true
    config:
      global:
        slack_api_url: 'https://hooks.slack.com/'
        smtp_from: 'alerts@yourdomain.com'
        smtp_smarthost: 'smtp.example.com:587'
        smtp_auth_username: 'your-username'
        smtp_auth_password: 'your-password'
        smtp_require_tls: true
      receivers:
      - name: email
        email_configs:
        - to: 'team@yourdomain.com'
          headers:
            Subject: 'Traceable Platform Alert: {{ .GroupLabels.alertname }}'
      route:
        receiver: email

Note: The slack_api_url field is required in all integrations due to how the Helm chart is deployed, even if you're not using Slack.

---

3. Webhook Integration

Configure AlertManager to send alerts to custom webhook endpoints.

Basic Webhook Configuration

prometheus:
  alertmanager:
    enabled: true
    config:
      global:
        slack_api_url: 'https://hooks.slack.com/'
        http_config:
          basic_auth:
            username: 'webhook-user'
            password: 'webhook-password'
      receivers:
      - name: webhook
        webhook_configs:
        - url: 'http://mywebhook.com/alerts'
      - name: "null"
      route:
        receiver: webhook
        routes:
        - matchers:
          - alertname = Watchdog
          receiver: "null"
        - matchers:
          - alertname = InfoInhibitor
          receiver: "null"
        - receiver: webhook

Webhook with Custom Headers

prometheus:
  alertmanager:
    enabled: true
    config:
      global:
        slack_api_url: 'https://hooks.slack.com/'
      receivers:
      - name: webhook
        webhook_configs:
        - url: 'https://api.example.com/alerts'
          http_config:
            bearer_token: 'your-bearer-token'
          send_resolved: true
      route:
        receiver: webhook

Note: The slack_api_url field is required in all integrations due to how the Helm chart is deployed.

---

4. Microsoft Teams Integration

AlertManager cannot directly format notifications for Microsoft Teams webhooks. You'll encounter this error if you try:

caller=dispatch.go:353 level=error component=dispatcher 
  msg="Notify for alerts failed" num_alerts=1 err="webhook/webhook[0]: 
    notify retry canceled due to unrecoverable error after 1 attempts: 
      unexpected status code 400: <Teams Webhook>: Summary or Text is required."

Solution: Use prometheus-msteams

To integrate with MS Teams, deploy the prometheus-msteams adapter:

1. Deploy prometheus-msteams Helm Chart:

Follow the guide: MS Teams Alerts for Kubernetes Cluster

1. Configure AlertManager to use the adapter:

prometheus:
  alertmanager:
    enabled: true
    config:
      global:
        slack_api_url: 'https://hooks.slack.com/'
      receivers:
      - name: msteams
        webhook_configs:
        - url: 'http://prometheus-msteams:2000/alertmanager'
      route:
        receiver: msteams

---

Advanced Routing

Send Only Specific Alerts

Configure AlertManager to route only specific alerts based on labels and matchers:

prometheus:
  alertmanager:
    enabled: true
    config:
      global:
        slack_api_url: 'https://hooks.slack.com/'
        resolve_timeout: 1m
      route:
        receiver: "null"
        routes:
          # Critical alerts - immediate notification
          - receiver: "email"
            matchers:
              - alertname=~"OldGenGCOverhead|PinotServerHighCPUUsage|PinotKafkaConsumerError|PinotKafkaConsumerLag|KubePersistentVolumeFillingUp"
              - severity=~"SEV1|critical"
            repeat_interval: 12h

          # Delayed alerts - wait 3 hours before notifying
          - receiver: "email"
            matchers:
              - alertname=~".*OFFLINE-pinot-offline-segment-delay|.*EntityCreationThresholdExceeded|.*NoModelsWritten"
            group_wait: 3h
            repeat_interval: 24h

          # Application alerts - wait 3 hours before notifying
          - receiver: "email"
            matchers:
              - alertname=~"NotificationChannelFailure|HighVulnerabilityStatusChanges|HighUnsuccessfulResponseCodeIntegrationService|connect-task-failure|DropInSpansFromTenant|PodOutOfMemory|KubePodCrashLooping|PodRestarted"
            group_wait: 3h
            repeat_interval: 24h

          # Allow alerts to continue to other receivers
          - receiver: "null"
            continue: true

      receivers:
        - name: "null"
        - name: "email"
          email_configs:
            - to: oncall@yourdomain.com
              from: alerts@yourdomain.com
              smarthost: smtp.example.com:587
              require_tls: true
              send_resolved: true

Routing Parameters:

• group_wait: How long to wait before sending the first notification for a group of alerts
• repeat_interval: How long to wait before resending a notification
• matchers: Label-based matching using regular expressions
• continue: Allow the alert to continue to the next route

Send Alerts to Multiple Receivers

To send the same alerts to different notification channels:

prometheus:
  alertmanager:
    enabled: true
    config:
      global:
        slack_api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'
      route:
        receiver: "default"
        routes:
          - receiver: "critical-alerts"
            matchers:
              - severity = critical
            continue: true  # Continue to next route
          - receiver: "all-alerts"

      receivers:
        - name: "default"
        - name: "critical-alerts"
          slack_configs:
          - channel: '#critical-alerts'
          email_configs:
          - to: 'oncall@yourdomain.com'
        - name: "all-alerts"
          slack_configs:
          - channel: '#all-alerts'

Reference: Send Notifications to Multiple Channels

---

Validation

Validate Configuration in Kubernetes

After applying the configuration, validate that AlertManager has the correct settings:

1. Check alertmanager.yaml Secret

kubectl get secret -n monitoring \
  alertmanager-prometheus-operator-alertmanager -o yaml \
  | yq '.data."alertmanager.yaml"' | base64 -d

2. Check Generated Configuration Secret

kubectl get secret -n monitoring \
  alertmanager-prometheus-operator-alertmanager-generated -o yaml \
  | yq '.data."alertmanager.yaml.gz"' | base64 -d | gunzip -

This secret contains the gzipped version of alertmanager.yaml.

3. Check Running Configuration in Pod

kubectl exec -t -n monitoring alertmanager-prometheus-operator-alertmanager-0 \
  -- cat /etc/alertmanager/config_out/alertmanager.env.yaml

Important: The output of all three commands should be identical.

Manual Configuration Update

If you need to manually update the configuration:

1. Extract and modify the configuration:

kubectl get secret -n monitoring \
  alertmanager-prometheus-operator-alertmanager-generated -o yaml \
  | yq '.data."alertmanager.yaml.gz"' | base64 -d | gunzip - > alertmanager.yaml

# Edit alertmanager.yaml
vim alertmanager.yaml

1. Compress and encode:

gzip alertmanager.yaml \
  && b=$(cat alertmanager.yaml.gz | base64 -w0) \
  && echo $b && gunzip alertmanager.yaml.gz

1. Update the secret with the new value and restart AlertManager pod:

kubectl delete pod -n monitoring alertmanager-prometheus-operator-alertmanager-0

---

Testing

Trigger a Test Alert

Use curl to send a test alert to AlertManager:

curl -w '\n' -i \
  --header 'Content-Type: application/json' \
  --data '[{
    "labels": {
      "alertname": "test_alert_traceable",
      "team": "traceable",
      "severity": "SEV1"
    },
    "endsAt": "2025-03-21T00:10:53-03:00"
  }]' \
  http://traceable-alertmanager.traceable.svc:9093/api/v2/alerts

Expected Response:

HTTP/1.1 200 OK
Content-Type: application/json

Verify Test Alert

1. Access AlertManager UI:

kubectl port-forward -n monitoring svc/prometheus-operator-alertmanager 9093:9093

Navigate to: http://localhost:9093

1. Check for the test alert in the AlertManager UI

2. Verify notification was sent to your configured channel (Slack, email, etc.)

Add More Labels to Test Alert

curl -w '\n' -i \
  --header 'Content-Type: application/json' \
  --data '[{
    "labels": {
      "alertname": "test_alert_traceable",
      "team": "traceable",
      "severity": "critical",
      "namespace": "traceable",
      "pod": "test-pod"
    },
    "annotations": {
      "summary": "This is a test alert",
      "description": "Testing AlertManager notification routing"
    },
    "endsAt": "2025-03-21T00:10:53-03:00"
  }]' \
  http://traceable-alertmanager.traceable.svc:9093/api/v2/alerts

---

Troubleshooting

Issue 1: Alerts Not Being Sent

Symptoms: AlertManager is running but notifications are not received.

Solutions:

1. Check AlertManager logs:

kubectl logs -n monitoring alertmanager-prometheus-operator-alertmanager-0

1. Verify configuration is loaded:

kubectl exec -t -n monitoring alertmanager-prometheus-operator-alertmanager-0 \
  -- cat /etc/alertmanager/config_out/alertmanager.env.yaml

1. Check for routing issues - ensure your alert labels match the matchers in your routes

2. Verify the receiver configuration is correct

Issue 2: SMTP Authentication Failed

Symptoms: Email notifications fail with authentication errors.

Solutions:

1. Verify SMTP credentials are correct
2. Check if SMTP server requires TLS:

smtp_require_tls: true

1. Test SMTP connection manually:

telnet smtp.example.com 587

1. For SendGrid, ensure you're using apikey as the username

Issue 3: Slack Webhook Not Working

Symptoms: Slack notifications are not received.

Solutions:

1. Verify the webhook URL is correct and active
2. Test the webhook manually:

curl -X POST -H 'Content-type: application/json' \
  --data '{"text":"Test message"}' \
  YOUR_WEBHOOK_URL

1. Check Slack app permissions
2. Ensure the channel exists and the app has access to it

Issue 4: Configuration Not Applied

Symptoms: Changes to AlertManager configuration are not reflected.

Solutions:

1. Verify the configuration was applied to the platform
2. Check if the secret was updated:

kubectl get secret -n monitoring \
  alertmanager-prometheus-operator-alertmanager -o yaml

1. Restart AlertManager pod:

kubectl delete pod -n monitoring alertmanager-prometheus-operator-alertmanager-0

1. Wait for the pod to restart and check logs

Issue 5: Duplicate Alerts

Symptoms: Receiving multiple notifications for the same alert.

Solutions:

1. Check repeat_interval setting - increase if needed
2. Verify group_by labels are configured correctly
3. Check if multiple routes are matching the same alert (use continue: false)

Issue 6: Alerts Not Resolving

Symptoms: Alerts remain active even after the issue is resolved.

Solutions:

1. Check resolve_timeout in global configuration
2. Verify Prometheus is sending resolved alerts
3. Ensure send_resolved: true is set in receiver configuration

---

Configuration Examples

Example 1: Production Setup with Multiple Channels

prometheus:
  alertmanager:
    enabled: true
    config:
      global:
        slack_api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'
        smtp_from: 'alerts@company.com'
        smtp_smarthost: 'smtp.sendgrid.net:587'
        smtp_auth_username: 'apikey'
        smtp_auth_password: '<sendgrid-api-key>'
        smtp_require_tls: true
        resolve_timeout: 5m

      route:
        receiver: "default"
        group_by: ['alertname', 'cluster', 'service']
        group_wait: 30s
        group_interval: 5m
        repeat_interval: 4h
        routes:
          # Critical alerts to oncall
          - receiver: "oncall"
            matchers:
              - severity = critical
            repeat_interval: 1h

          # Warning alerts to team channel
          - receiver: "team"
            matchers:
              - severity = warning

          # Info alerts - suppress
          - receiver: "null"
            matchers:
              - severity = info

      receivers:
        - name: "default"
          slack_configs:
          - channel: '#alerts-default'

        - name: "oncall"
          slack_configs:
          - channel: '#alerts-critical'
            title: 'CRITICAL: {{ .GroupLabels.alertname }}'
          email_configs:
          - to: 'oncall@company.com'
            send_resolved: true

        - name: "team"
          slack_configs:
          - channel: '#alerts-team'

        - name: "null"

Example 2: Airgap Environment with Internal SMTP

prometheus:
  alertmanager:
    enabled: true
    config:
      global:
        slack_api_url: 'https://hooks.slack.com/'
        smtp_from: 'alerts@internal.company.com'
        smtp_smarthost: 'smtp-internal.company.local:25'
        smtp_require_tls: false

      receivers:
      - name: email
        email_configs:
        - to: 'platform-team@company.com'
          headers:
            Subject: '[Traceable] {{ .GroupLabels.alertname }}'

      route:
        receiver: email
        routes:
        - matchers:
          - alertname = Watchdog
          receiver: "null"
        - matchers:
          - alertname = InfoInhibitor
          receiver: "null"

---

Best Practices

1. Test Configuration: Always test with a test alert before relying on production alerts
2. Use Matchers Wisely: Use specific matchers to avoid alert fatigue
3. Set Appropriate Intervals: Configure repeat_interval to avoid notification spam
4. Group Alerts: Use group_by to consolidate related alerts
5. Monitor AlertManager: Set up monitoring for AlertManager itself
6. Document Routes: Comment your routing configuration for maintainability
7. Use Inhibition Rules: Prevent redundant notifications with inhibition rules
8. Secure Credentials: Never commit credentials to version control
9. Regular Testing: Periodically test all notification channels
10. Version Control: Keep AlertManager configurations in version control

---

Security Considerations

1. Credential Management: Store credentials securely using Kubernetes secrets
2. TLS Verification: Enable TLS verification for production SMTP servers
3. Webhook Authentication: Use authentication for webhook endpoints
4. Access Control: Restrict access to AlertManager UI
5. Audit Logging: Enable audit logging for AlertManager configuration changes
6. Network Policies: Use network policies to restrict AlertManager traffic

---

Additional Resources

Official Documentation

• Prometheus AlertManager Configuration
• Receiver Integration Settings

Related Guides

• SMTP Setup - Configure SMTP for platform notifications
• Platform Monitoring - Kubernetes monitoring requirements

---

Support

For AlertManager configuration assistance or issues, contact: support@harness.io

Provide the following information when requesting support:

• AlertManager configuration (sanitized)
• AlertManager logs
• Test alert results
• Notification channel type (Slack, email, webhook, etc.)
• Error messages
• Platform version