Skip to content

Restart and Patching Guide

This guide covers procedures for restarting workloads and patching nodes in the Traceable Platform.

Overview

Regular maintenance activities such as OS patching, security updates, and workload restarts are essential for keeping your Traceable Platform healthy and secure. This guide provides step-by-step procedures for different deployment scenarios.

Prerequisites

  • Access to the Traceable Operator (TI) UI
  • kubectl access to the cluster
  • For multi-node: Additional nodes to handle workload during patching

Patching of Single Node / NFS

For single-node deployments or when patching the NFS server.

Bring Down the Cluster

  1. Login to the Traceable Operator (TI) Navigate to https://<platform-dns>/ti

  2. Access Shutdown Tasks Go to Tasks → ShutdownWorkloads

  3. Run the Workflow Execute the shutdown workflow.

Note: shutdown-traceable-operator is optional.

Bring the Cluster Up

After patching is complete, bring the cluster back up:

Step 1: Start PostgreSQL

  1. Login to https://<DNS>/timgr
  2. Click Workloads
  3. Click Filter Kubernetes Workloads and filter by postgres
  4. Select the postgres workload
  5. Click on the Scale button
  6. Click 1 to scale to 1 replica
  7. Click Confirm
  8. Click Close
  9. Wait for PostgreSQL to come up

Step 2: Start Traceable Installer

Perform the same scaling action for the Traceable Installer workload.

Step 3: Run Install Workflow

  1. Login to the Traceable Operator
  2. Click Install Workflow
  3. Click Save & Next
  4. Click Save & Next (Review)
  5. Click Start Install
  6. Wait for the installation to complete

Patching of Multi-Node

For multi-node deployments, patch nodes one at a time using drain and uncordon.

Assumption: You have additional nodes that can handle the workload when one node is down.

Procedure

  1. Login to the Traceable Operator (TI)

  2. Navigate to Nodes Go to the Nodes section

  3. Drain the Node Without making any changes, drain the node to be patched.

This will: - Mark the node as unschedulable - Evict all pods to other nodes

  1. Perform Patching Once the drain is completed, perform the patching activity on the node.

  2. Uncordon the Node After patching is complete, uncordon the node to allow pods to be scheduled again.

  3. Repeat Repeat the same process for each node until all nodes are patched.

kubectl Commands

If you prefer using kubectl directly:

Drain a node:

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

Uncordon a node:

kubectl uncordon <node-name>

Restart Workloads

To restart all workloads without patching:

  1. Login to the Traceable Operator (TI)

  2. Navigate to Install Workflow Go to Install Workflow

  3. Access Actions Click on Actions

  4. Select Restart Workloads Click on Restart Workloads

  5. Execute Restart Without making any changes, click on Restart Workload

Troubleshooting

Issue 1: Pods in Completed State

Symptoms: Pods that are part of deployments/statefulsets have status Completed due to exit code 0, causing service failures.

Solution:

Delete all pods in Completed state:

kubectl get pods -n traceable | grep Completed \
    | awk '{print $1}' \
    | xargs -I{} kubectl delete pod -n traceable {}

Note: Some job-related pods may also get deleted but this has no impact.

After deleting completed pods, run the restart workload job.

Issue 2: Restart Workloads Job Failed Due to Timeout

Symptoms: Usually schema-registry and keycloak restart fail due to timeouts.

Solution:

  1. Wait for the pods to come up after the job fails
  2. Uncheck all workloads up to and including the failed service
  3. Rerun the restart job

Example: If schema-registry failed, skip all workloads including schema-registry, then restart workloads.

Issue 3: Node Won't Drain

Symptoms: Drain command hangs or fails.

Possible Causes:

  • Pods with local storage that can't be evicted
  • PodDisruptionBudgets preventing eviction
  • Pods without controllers (standalone pods)

Solutions:

Force drain (use with caution):

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data --force

Check what's blocking:

kubectl get pods --all-namespaces -o wide | grep <node-name>

Issue 4: Pods Not Rescheduling After Uncordon

Symptoms: After uncordoning, pods don't move back to the node.

Explanation: Kubernetes doesn't automatically rebalance pods. Pods will only be scheduled on the uncordoned node when:

  • New pods are created
  • Existing pods are deleted and recreated
  • A restart workload is performed

Solution: Run the restart workloads procedure if immediate rebalancing is needed.

Best Practices

  1. Schedule Maintenance Windows: Plan patching during low-traffic periods
  2. Backup First: Always backup before major maintenance activities
  3. Test in Non-Production: Test patching procedures in non-production environments first
  4. Monitor During Patching: Watch pod status and logs during the process
  5. Document Changes: Keep records of patches applied and any issues encountered
  6. Verify After Patching: Run post-installation checks after bringing the cluster back up

Maintenance Checklist

Pre-Maintenance

  • Notify stakeholders of maintenance window
  • Backup critical data
  • Verify backup integrity
  • Document current state (pod counts, versions)

During Maintenance

  • Follow shutdown/drain procedures
  • Apply patches/updates
  • Verify node health after patching

Post-Maintenance

  • Bring cluster back up
  • Verify all pods are running
  • Run post-installation checks
  • Test platform functionality
  • Notify stakeholders of completion

Support

For maintenance assistance, contact: support@harness.io

Provide the following information:

  • Deployment type (single-node/multi-node)
  • Maintenance activity being performed
  • Error messages and logs
  • Pod status output