Skip to content

Update TLS Certificates

This guide covers the procedure for updating TLS certificates on the Traceable Platform.

Overview

TLS certificates need to be updated when:

  • Certificates are expiring
  • Migrating from self-signed to CA-signed certificates
  • Domain name changes
  • Security policy requires certificate rotation

Prerequisites

Before updating certificates, ensure you have:

  • Access to the Traceable Operator (TI) UI
  • Certificate files in PEM format:
  • server.key - Private key
  • server.pem - Server certificate (with chain)
  • rootCA.pem - Root CA certificate

Pre-Checks

Validate Private Key Matches Certificate

Compare the modulus of the private key and certificate:

Extract modulus from private key:

openssl rsa -in server.key -noout -modulus | openssl md5

Extract modulus from certificate:

openssl x509 -in server.pem -noout -modulus | openssl md5

Result: Both commands should output the same MD5 hash.

Validate Certificate is Signed by CA

openssl verify -CAfile rootCA.pem server.pem

Expected output (valid):

server.pem: OK

Error output (self-signed):

error 18 at 0 depth lookup: self signed certificate

Verify Subject Alternative Name (SAN)

openssl x509 -in server.pem -text -noout | grep -A1 "Subject Alternative Name"

Expected output (with SAN):

X509v3 Subject Alternative Name:
    DNS:mydomain.com, DNS:www.mydomain.com, DNS:api.mydomain.com

Important: The certificate must have a SAN entry with the platform DNS name. CN (Common Name) is deprecated and the UI will be inaccessible without a valid SAN.

Verify Key is Not Encrypted

The private key must NOT have a passphrase:

cat server.key | head -n1

Valid output:

-----BEGIN PRIVATE KEY-----

or

-----BEGIN RSA PRIVATE KEY-----

Invalid output (encrypted):

-----BEGIN ENCRYPTED PRIVATE KEY-----

If the key is encrypted, remove the passphrase:

openssl rsa -in server.key -out server-decrypted.key

Certificate Requirements

Required Files

File Description Format
Private Key Server private key PEM
Server Certificate Signed certificate with chain PEM
Root CA Certificate Root CA certificate PEM

Server Certificate Chain Order

The server certificate file must contain certificates in this order:

  1. Server certificate (first)
  2. Intermediate certificates (if any)
  3. Root certificate (last)

Example structure:

-----BEGIN CERTIFICATE-----
[Server Certificate]
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
[Intermediate Certificate]
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
[Root CA Certificate]
-----END CERTIFICATE-----

Upload TLS Certificates

Backup Existing Certificates

Before uploading new certificates, download the existing ones:

  1. Navigate to TasksGenerateTlsCerts
  2. Click Download to save the current certificates

Upload New Certificates

  1. Navigate to TasksGenerateTlsCertsupload_install_tls_certs
  2. Upload the certificate files:
    • Private Key
    • Server Certificate (with chain)
    • Root CA Certificate
  3. Click Apply to create Kubernetes secrets

Restart Services

After uploading certificates, restart the following services for changes to take effect:

  • account-service
  • iam
  • license-metering-service
  • job-service
  • iam-v2

Via TI UI

  1. Navigate to TasksGenerateTlsCertsrestart_services
  2. Execute the task

Via kubectl

kubectl rollout restart deployment -n traceable account-service
kubectl rollout restart deployment -n traceable iam
kubectl rollout restart deployment -n traceable license-metering-service
kubectl rollout restart deployment -n traceable job-service
kubectl rollout restart deployment -n traceable iam-v2

Note: For platforms older than version 1.17.0, you must manually restart the iam-v2 service.


Update Platform Agent (TPA)

If using certificates signed by a private CA or self-signed CA, update the TPA to trust the certificate chain.

Update TRACEABLE_ROOT_CA_FILE

openssl s_client -connect $TRACEABLE_PLATFORM_ENDPOINT:443 -showcerts 2>/dev/null </dev/null | \
  sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' \
  >> $TRACEABLE_ROOT_CA_FILE

Replace:

  • TRACEABLE_PLATFORM_ENDPOINT - Your platform DNS
  • TRACEABLE_ROOT_CA_FILE - Path to CA file (found in /etc/traceable/agent/config.yaml as ca_cert_file)

Restart TPA

rm -rf /tmp/exporter_*
systemctl restart traceable

Caveats

Private CA or Self-Signed Certificates

When using certificates signed by a private CA or self-signed:

  1. Browser Warning: Users will see "Connection is Not Secure" warning. The connection is still encrypted via HTTPS, but the browser doesn't trust the certificate.

  2. Agent Trust: TPA and AST runners must be configured to trust the certificate chain (see Step 5).

Reference: DigiCert - Not Secure Warning


Verification

After updating certificates:

Check Certificate in Browser

  1. Navigate to https://<platform-dns>
  2. Click the lock icon in the browser
  3. View certificate details
  4. Verify:
    • Certificate is valid
    • SAN includes the platform DNS
    • Expiration date is correct

Check Certificate via CLI

echo | openssl s_client -connect <platform-dns>:443 2>/dev/null | openssl x509 -noout -dates -subject -issuer

Verify Services are Running

kubectl get pods -n traceable -l app=iam
kubectl get pods -n traceable -l app=iam-v2
kubectl get pods -n traceable -l app=account-service

Troubleshooting

Certificate Not Trusted by Browser

Symptoms: Browser shows "Not Secure" or certificate warning.

Causes:

  • Using self-signed or private CA certificate
  • Intermediate certificates missing from chain
  • Certificate expired

Solutions:

  1. Use a certificate from a trusted public CA
  2. Ensure all intermediate certificates are included
  3. Check certificate expiration date

Services Fail to Start After Certificate Update

Symptoms: Pods crash or fail to become ready.

Solutions:

  1. Check pod logs:
kubectl logs -n traceable -l app=iam
  1. Verify certificate format is correct (PEM)
  2. Verify private key matches certificate
  3. Restore backup certificates if needed

TPA Cannot Connect

Symptoms: Agent shows SSL/TLS errors.

Solutions:

  1. Update TRACEABLE_ROOT_CA_FILE with new certificate chain
  2. Restart TPA service
  3. Verify certificate chain is complete

SAN Mismatch Error

Symptoms: Browser shows hostname mismatch error.

Solutions:

  1. Verify certificate SAN includes the platform DNS
  2. Generate new certificate with correct SAN
  3. Check for typos in DNS name

Certificate Rotation Schedule

Recommended practices:

  • Monitor Expiration: Set up alerts for certificate expiration (30 days before)
  • Regular Rotation: Rotate certificates annually even if not expired
  • Document: Keep records of certificate changes and expiration dates

Generate Certificate Signing Request (CSR)

If you need to generate a new private key and CSR for your Certificate Authority:

Step 1: Generate New Private Key

Execute the generate_self_signed_certs task:

  1. Navigate to TasksGenerateTlsCertsgenerate_self_signed_certs
  2. Set overwrite to true
  3. Execute the task

Troubleshooting: If you see the error:

CA certificate and CA private key do not match

Remove all files under /app/data/clusters/traceable/data/self-signed-certs/<DNS>/ and re-run the task.

Step 2: Create Support Bundle for CSR

Create a new support bundle definition (if not present):

name: Generate-CSR
exec:
    - name: Generate-CSR
    command: |
      CERT_DIR={{.clusterDir}}/data/self-signed-certs/{{.tfModules.terraform.inputs.dns.value}}
      DNS={{.tfModules.terraform.inputs.dns.value}}
      echo "Generating CSR"
      openssl req -new -sha256 -out ${sbDir}/${DNS}.csr -key ${CERT_DIR}/${DNS}.key \
        -config ${CERT_DIR}/${DNS}.conf
      echo "Human-readable CSR"
      openssl req -in ${sbDir}/${DNS}.csr -noout -text
    - name: Copy-Cnf-Key
    command: |
      CERT_DIR={{.clusterDir}}/data/self-signed-certs/{{.tfModules.terraform.inputs.dns.value}}
      DNS={{.tfModules.terraform.inputs.dns.value}}
      echo "Copying Key file"
      cp ${CERT_DIR}/${DNS}.conf ${sbDir}/${DNS}.conf
    - name: Copy-Pvt-Key
    command: |
      CERT_DIR={{.clusterDir}}/data/self-signed-certs/{{.tfModules.terraform.inputs.dns.value}}
      DNS={{.tfModules.terraform.inputs.dns.value}}
      echo "Copying Key file"
      cp ${CERT_DIR}/${DNS}.key ${sbDir}/${DNS}.key

Step 3: Generate the Support Bundle

  1. Navigate to Support Bundle in TI UI
  2. Select the Generate-CSR bundle
  3. Execute the bundle

The Operator will generate a zip file containing:

  • <DNS>.key - Private key
  • <DNS>.csr - Certificate Signing Request
  • <DNS>.conf - SAN configuration file

Step 4: Submit CSR to Certificate Authority

Send the CSR file (<DNS>.csr) to your preferred Certificate Authority (CA). When ready, the CA will return:

  • Signed server certificate
  • Required root and intermediate certificates for the trust chain

Step 5: Update the Certificate

Once you receive the signed certificate from your CA, follow the Upload TLS Certificates section above to install it.



Support

For certificate update assistance, contact: support@harness.io

Provide the following information:

  • Certificate validation output (pre-checks)
  • Error messages from services
  • Browser certificate details
  • TPA configuration (if applicable)