Skip to content

Rollback Procedures

This guide covers how to rollback the Traceable Platform to a previous version if issues occur during or after an upgrade.

Overview

As a part of the upgrade process, the installer automatically creates a backup of the current state. If a rollback is needed for any reason, the workflow from the previous state can be executed, bringing the environment back to the previous state.

When to Rollback

Consider rollback in the following scenarios:

  • Upgrade fails and cannot be recovered
  • Critical functionality is broken after upgrade
  • Performance degradation after upgrade
  • Data processing issues after upgrade
  • Instructed by Traceable support team

Rollback Process

1. Access Upgrade History

  1. Click on Upgrade Cluster in the sidebar menu
  2. Click on the Upgrade History button

This will display a list of all previous versions and their states.

2. Select Version to Rollback

  1. Choose the version you want to roll back to from the history
  2. Review the version details and timestamp
  3. Click the Rollback button

Important: This will only update the state of the cluster to the chosen version. The actual service rollout happens in the next steps.

3. Navigate to Install Workflow

After selecting the rollback version:

  1. Navigate to Install Workflow
  2. Continue the steps from the Global Variables section in the upgrade guide

4. Execute Rollback Workflow

Follow the same process as a regular upgrade:

  1. Review Global Variables - Verify variables match the previous state
  2. Review Global Helm Values - Ensure values are correct for the rollback version
  3. Select Services - Select all services that need to be rolled back
  4. Trigger Install - Click Start Install to begin the rollback

5. Monitor Rollback Progress

Monitor the rollback process the same way as an upgrade:

  • Watch the workflow execution
  • Monitor pod status in real-time
  • Check for any errors or failures

Rollback Verification

After the rollback completes:

1. Verify Pod Status

kubectl get pods -n <traceable-namespace>

All pods should be in Running state.

2. Check Service Versions

kubectl get pods -n <traceable-namespace> -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}{end}'

Verify all images match the rollback version.

3. Verify Platform Functionality

  1. Login to the Traceable UI
  2. Check if spans are flowing
  3. Verify dashboards are accessible
  4. Test critical functionality
  5. Confirm version in UI matches rollback target

4. Check Data Processing

  1. Verify spans from the last 15 minutes are visible
  2. Check that alerts are functioning
  3. Confirm API endpoints are responding correctly

Troubleshooting Rollback Issues

Rollback Workflow Fails

If the rollback workflow fails:

  1. Check pod logs:
kubectl logs <pod-name> -n <traceable-namespace>
  1. Check pod status:
kubectl describe pod <pod-name> -n <traceable-namespace>

  1. Restart failed workflow item:

    • Click Restart button next to the failed item

  2. Contact support:

    • If issues persist, contact Traceable support immediately
    • Provide workflow logs and pod status

Partial Rollback State

If the system is in a partial rollback state:

  1. Do not attempt another upgrade/rollback
  2. Document current state:
    • Which services rolled back successfully
    • Which services are still on new version
  3. Contact Traceable support with details

Data Inconsistency After Rollback

If you notice data inconsistencies:

  1. Check database connectivity
  2. Verify storage volumes are mounted correctly
  3. Review logs for data processing errors
  4. Contact Traceable support for data validation

Best Practices

Before Rollback

  • Document the reason for rollback
  • Take screenshots of errors or issues
  • Backup current logs and state
  • Notify stakeholders of rollback plan
  • Schedule maintenance window if possible

During Rollback

  • Monitor the rollback process continuously
  • Keep Traceable support informed
  • Document any errors or unexpected behavior
  • Do not interrupt the rollback process

After Rollback

  • Verify all functionality is restored
  • Document lessons learned
  • Work with Traceable support to identify root cause
  • Plan for retry with fixes if applicable

Prevention

To minimize the need for rollbacks:

  1. Always complete pre-upgrade checks
  2. Test upgrades in non-production environment first
  3. Schedule upgrades during maintenance windows
  4. Have Traceable support available during upgrade
  5. Keep backups of installer state
  6. Document custom configurations

Emergency Rollback

For critical production issues requiring immediate rollback:

  1. Contact Traceable support immediately
  2. Initiate rollback process
  3. Monitor closely
  4. Document all actions taken

Support

For rollback assistance or issues, contact: support@harness.io

Emergency Contact: Ensure you have emergency contact information for critical production issues.