Retain Custom Headers

This guide explains how to retain custom span headers in the Traceable Platform using span ingestion rules from the Traceable Operator (TI) GraphQL console.

Overview

Span ingestion rules control which span attributes are persisted or discarded when spans are written to backend stores such as Pinot. To retain custom headers, create request header rules with retentionAction: RETAIN.

Rules are ranked and evaluated in order. The highest-ranked rule is evaluated first, and the action from the first matching rule is applied. If no rule matches, the default action for the category is applied.

Default rules exist for each category. These default rules have the lowest rank and cannot be updated, deleted, or ranked.

---

Prerequisites

Before proceeding, ensure you have:

• Admin user access to the Traceable Platform
• Access to the Traceable Operator (TI)
• The DNS name for the deployment
• The custom request header names that must be retained

---

Access the GraphQL Console

Step 1: Login to the Platform

Open the Traceable Platform in a new browser tab:

http://<dns>

Login with an admin user.

Step 2: Login to the Traceable Operator

Open the Traceable Operator (TI):

https://<dns>/ti

Login with your TI credentials.

Step 3: Open GraphQL

In the TI, click GraphQL.

Step 4: Run the Query

Paste the required GraphQL query or mutation into the editor, then click the Run button.

---

Rule Template

Span ingestion rules use the following structure:

id
predicate {
  targetKeyPredicate {
    value
    operator
    anyOfValues
  }
}
retentionAction
expiration
scope {
  environmentIds
}

Field	Description
id	Rule ID. The ID is auto-generated when the rule is created.
predicate.targetKeyPredicate.operator	Key matching operator. Supported values are EQUALS, STARTS_WITH, ENDS_WITH, and CONTAINS.
predicate.targetKeyPredicate.value	Single header name to match.
predicate.targetKeyPredicate.anyOfValues	Multiple header names to match. A match occurs if the header matches any value in the list.
retentionAction	Action to apply when the rule matches. Use RETAIN to keep custom headers.
expiration	Optional expiration time for the rule. After this time, the rule is no longer applied.
scope.environmentIds	Optional list of environment IDs where the rule applies. If scope is not set, the rule applies to all environments.

Important: Use either value or anyOfValues in a single rule. Do not provide both.

---

Get Existing Request Header Rules

Use the following query to list existing request header rules:

{
  spanIngestionRules(
    input: { ingestionStage: INGESTION_STAGE_QUERY_STORE_PERSISTENCE }
  ) {
    requestHeaderRules {
      defaultRuleAction
      rules {
        id
        expiration
        predicate {
          targetKeyPredicate {
            value
            operator
            anyOfValues
          }
        }
        retentionAction
        scope {
          environmentIds
        }
      }
    }
  }
}

Expected response:

{
  "data": {
    "spanIngestionRules": {
      "requestHeaderRules": {
        "defaultRuleAction": "RETAIN",
        "rules": [
          {
            "id": "1c28a809-9de5-4d57-869f-156582145238",
            "expiration": null,
            "predicate": {
              "targetKeyPredicate": {
                "value": "my-header",
                "operator": "EQUALS",
                "anyOfValues": null
              }
            },
            "retentionAction": "RETAIN",
            "scope": null
          }
        ]
      }
    }
  }
}

---

Retain a Single Custom Header

Use this mutation to retain one custom request header:

mutation {
  createSpanIngestionRule(
    input: {
      ingestionStage: INGESTION_STAGE_QUERY_STORE_PERSISTENCE
      requestHeaderRuleData: {
        predicate: {
          targetKeyPredicate: { operator: EQUALS, value: "my-header" }
        }
        retentionAction: RETAIN
        expiration: "2026-12-31T23:59:59.000Z"
      }
    }
  ) {
    id
    expiration
    predicate {
      targetKeyPredicate {
        operator
        value
        anyOfValues
      }
    }
    retentionAction
    scope {
      environmentIds
    }
  }
}

Expected response:

{
  "data": {
    "createSpanIngestionRule": {
      "id": "8d9ab022-f6a8-47a6-b507-dbdde5f638e7",
      "expiration": "2026-12-31T23:59:59Z",
      "predicate": {
        "targetKeyPredicate": {
          "operator": "EQUALS",
          "value": "my-header",
          "anyOfValues": null
        }
      },
      "retentionAction": "RETAIN",
      "scope": null
    }
  }
}

---

Retain Multiple Custom Headers

Use anyOfValues to retain multiple custom request headers:

mutation {
  createSpanIngestionRule(
    input: {
      ingestionStage: INGESTION_STAGE_QUERY_STORE_PERSISTENCE
      requestHeaderRuleData: {
        predicate: {
          targetKeyPredicate: {
            operator: EQUALS
            anyOfValues: ["my-header-1", "my-header-2"]
          }
        }
        retentionAction: RETAIN
        scope: { environmentIds: ["<environment-id>"] }
      }
    }
  ) {
    id
    expiration
    predicate {
      targetKeyPredicate {
        operator
        value
        anyOfValues
      }
    }
    retentionAction
    scope {
      environmentIds
    }
  }
}

---

Update Retained Headers

Use the rule ID returned by the create or list operation:

mutation {
  updateSpanIngestionRule(
    input: {
      id: "<rule-id>"
      predicate: {
        targetKeyPredicate: {
          operator: EQUALS
          anyOfValues: ["my-header-1", "my-header-2", "my-header-3"]
        }
      }
      retentionAction: RETAIN
      scope: { environmentIds: ["<environment-id>"] }
    }
  ) {
    id
    expiration
    predicate {
      targetKeyPredicate {
        operator
        value
        anyOfValues
      }
    }
    retentionAction
    scope {
      environmentIds
    }
  }
}

---

Rank a Rule

Use this mutation to move a rule after another rule:

mutation {
  rankSpanIngestionRule(
    input: {
      id: "<rule-id>"
      precedingRuleId: "<preceding-rule-id>"
    }
  ) {
    success
  }
}

Expected response:

{
  "data": {
    "rankSpanIngestionRule": {
      "success": true
    }
  }
}

---

Delete a Rule

Use this mutation to delete a user-defined rule:

mutation {
  deleteSpanIngestionRule(
    input: { id: "<rule-id>" }
  ) {
    success
  }
}

Expected response:

{
  "data": {
    "deleteSpanIngestionRule": {
      "success": true
    }
  }
}

---

Validation

After creating, updating, ranking, or deleting a rule:

1. Run the Get Existing Request Header Rules query again.
2. Confirm the rule appears with the expected retentionAction, predicate, scope, and expiration.
3. Confirm rule order is correct if ranking was changed.
4. Verify the custom headers are retained in downstream span data.

---

Troubleshooting

GraphQL Mutation Fails

Symptoms: The GraphQL response contains an error and the rule is not created or updated.

Solutions:

1. Verify retentionAction is set to RETAIN.
2. Verify the operator is one of EQUALS, STARTS_WITH, ENDS_WITH, or CONTAINS.
3. Ensure only one of value or anyOfValues is set.
4. Confirm expiration uses a valid timestamp format, such as 2026-12-31T23:59:59.000Z.

Header Is Not Retained

Symptoms: The custom request header is not available in downstream span data.

Solutions:

1. Check whether a higher-ranked rule matches before this rule.
2. Confirm the rule is configured under requestHeaderRules.
3. Verify scope.environmentIds includes the relevant environment, or remove the scope to apply the rule globally.
4. Check whether the rule has expired.

Cannot Update or Delete a Rule

Symptoms: Update, delete, or rank operations fail for a rule.

Solutions:

1. Confirm the rule ID is correct.
2. Verify the rule is user-defined. Default rules cannot be updated, deleted, or ranked.
3. Run the list query again to confirm the rule still exists.

---

Security Considerations

• Avoid retaining sensitive headers unless there is a clear operational need.
• Do not retain authentication tokens, cookies, secrets, or personally identifiable information unless approved by your security team.
• Scope rules to specific environments when testing changes.
• Review retained request headers periodically.

---

Support

For assistance with retaining custom headers, contact: support@harness.io