Skip to content

Restore Your On-Prem Cluster

This guide covers the restore procedures for the Traceable Platform on-premises deployment.

Overview

The Restore Service allows you to restore platform components from backups created by the Backup Service:

  • MongoDB - Primary data store
  • PostgreSQL - Relational database
  • Zookeeper - Coordination service
  • Schema Registry - Schema management
  • Traceable Installer (TI) - Platform configuration

Prerequisites

  • Backups created using the Backup Service
  • Access to the Traceable Operator (TI) UI
  • Backup PVC mounted and accessible
  • Knowledge of which backup commit to restore

Restore Configuration

To restore from a backup, enable the appropriate restore job in the backup service configuration.

Basic Restore Configuration

restoreJobs:
  postgresql:
    enabled: true
    commit: "latest"
    clean: true

  installer:
    enabled: true
    commit: "latest"

  mongodb:
    enabled: true
    commit: "latest"
    clean: true

  schemaRegistry:
    enabled: true
    commit: "latest"

  zookeeper:
    enabled: true
    commit: "latest"
    resetPinotOffsets: false

Parameters

Parameter Description
enabled Set to true to enable restore for this service
commit Backup commit to restore (latest or specific commit ID)
resetPinotOffsets (Zookeeper only) Reset Pinot offsets during restore

Restore Procedures

Restore PostgreSQL

  1. Update Configuration:
restoreJobs:
  postgresql:
    enabled: true
    commit: "latest"
    clean: true

  1. Apply Configuration in the Traceable Operator

  2. Verify Restore:

kubectl logs -n traceable -l app=backup-service --tail=100 | grep postgresql

Restore MongoDB

  1. Update Configuration:
restoreJobs:
  mongodb:
    enabled: true
    commit: "latest"
    clean: true

  1. Apply Configuration in the Traceable Operator

  2. Verify Restore:

kubectl logs -n traceable -l app=backup-service --tail=100 | grep mongodb

Restore Zookeeper

  1. Update Configuration:
restoreJobs:
  zookeeper:
    enabled: true
    commit: "latest"
    resetPinotOffsets: true

  1. Apply Configuration in the Traceable Operator

  2. Verify Restore:

kubectl logs -n traceable -l app=backup-service --tail=100 | grep zookeeper

Note: Set resetPinotOffsets: true if you need to reset Pinot consumer offsets.

Restore Schema Registry

  1. Update Configuration:
restoreJobs:
  schemaRegistry:
    enabled: true
    commit: "latest"

  1. Apply Configuration in the Traceable Operator

  2. Verify Restore:

kubectl logs -n traceable -l app=backup-service --tail=100 | grep schema

Restore Traceable Installer

  1. Update Configuration:
restoreJobs:
  installer:
    enabled: true
    commit: "latest"

  1. Apply Configuration in the Traceable Operator

  2. Verify Restore:

kubectl logs -n traceable -l app=backup-service --tail=100 | grep installer

Finding Backup Commits

To list available backup commits:

# Access the backup pod
kubectl exec -it -n traceable <backup-pod> -- bash

# List snapshots for a service
restic -r /backup snapshots --tag mongodb
restic -r /backup snapshots --tag postgresql
restic -r /backup snapshots --tag zookeeper
restic -r /backup snapshots --tag schema-registry

Full Cluster Restore

For a complete cluster restore, follow this order:

  1. Restore Traceable Installer (configuration)
  2. Restore PostgreSQL (relational data)
  3. Restore MongoDB (primary data)
  4. Restore Zookeeper (coordination data)
  5. Restore Schema Registry (schema definitions)
  6. Restart all workloads

Full Restore Configuration

restoreJobs:
  installer:
    enabled: true
    commit: "latest"
  postgresql:
    enabled: true
    commit: "latest"
    clean: true
  mongodb:
    enabled: true
    commit: "latest"
    clean: true
  zookeeper:
    enabled: true
    commit: "latest"
    resetPinotOffsets: false
  schemaRegistry:
    enabled: true
    commit: "latest"

Post-Restore Steps

After restore completes:

  1. Verify Services:
kubectl get pods -n traceable

  1. Check Service Health:

    • Access the Platform UI
    • Verify data is present
    • Test critical functionality

  2. Restart Workloads (if needed):

    • Navigate to TI → Install Workflow → Actions → Restart Workloads

  3. Disable Restore Jobs: Set all restoreJobs.*.enabled to false to prevent accidental re-restore.

Troubleshooting

Restore Job Failing

Symptoms: Restore job shows Failed status.

Solutions:

  1. Check job logs:
kubectl logs -n traceable job/<restore-job-name>
  1. Verify backup exists for specified commit
  2. Check PVC is mounted correctly
  3. Verify restic password matches backup password

Data Not Restored

Symptoms: Service running but data missing.

Solutions:

  1. Verify correct commit was specified
  2. Check restore job completed successfully
  3. Restart the service after restore
  4. Check service logs for errors

Service Won't Start After Restore

Symptoms: Pods crash or fail to become ready.

Solutions:

  1. Check pod logs for errors
  2. Verify data integrity
  3. Check disk space
  4. Try restoring from a different backup commit

Best Practices

  1. Test Restores Regularly: Validate backup integrity by testing restores
  2. Document Commits: Keep records of important backup commits
  3. Staged Restore: Restore one service at a time in non-critical situations
  4. Backup Before Restore: Create a fresh backup before attempting restore
  5. Verify Data: Always verify restored data before resuming operations
  6. Disable After Restore: Disable restore jobs after successful restore

Support

For restore assistance, contact: support@harness.io

Provide the following information:

  • Platform version
  • Backup commit being restored
  • Restore job logs
  • Error messages
  • Services affected