Backup Your On-Prem Cluster¶

This guide covers the backup procedures for the Traceable Platform on-premises deployment.

Overview¶

The Backup Service creates backups of critical platform components:

MongoDB - Primary data store
PostgreSQL - Relational database
Zookeeper - Coordination service
Schema Registry - Schema management
Traceable Installer (TI) - Platform configuration

Availability: Backup Service is available starting from version 1.14.0.

Prerequisites¶

Traceable Platform version 1.14.0 or later
Access to the Traceable Operator (TI) UI
ReadWriteMany (RWX) storage class available
Sufficient storage capacity for backups

Enable Backup Service¶

The Backup Service is available as a task in the Traceable Operator.

Login to the Traceable Operator
Navigate to Tasks → Backup Service
Override the values as needed
Install the backup service

Create Backup PVC¶

Calculating Storage Size¶

Identify services to backup (MongoDB, PostgreSQL, etc.)
Check PVC sizes:

kubectl get pvc -n traceable | grep -e mongo -e postgres -e zookeeper -e schema-registry

Sum the PVC sizes (count only one PVC per service):
- Example: data-postgresql-0: 5Gi
- Example: mongo-persistent-storage-mongo-primary-0: 20Gi
- Total: 25 GB
Multiply by retention period:

25 GB × 3 days = 75 GB

Create a PersistentVolumeClaim for storing backups¶

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: platform-backup-pvc
  namespace: traceable
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 75Gi
  storageClassName: <RWX-storageclass>

Note: Replace <RWX-storageclass> with your ReadWriteMany storage class name.

Configuration¶

Common Configuration¶

clusterSuffix: <suffix-for-backup-filename>
imagePullSecrets:

- name: regcred

Restic Configuration¶

Restic is used for secure, encrypted backups:

restic:
  enabled: true
  password: <restic-password>
  repository: <backup-storage-path>

Parameters:

password: Password used to encrypt backup files
repository: Path where backups will be stored

PVC Configuration¶

extraVolumeMounts:

- mountPath: '<backup-storage-path>'
  name: platform-backup-vol

extraVolumes:

- name: platform-backup-vol
  persistentVolumeClaim:
    claimName: platform-backup-pvc

Important:

restic.repository must match extraVolumeMounts.mountPath
Path must be absolute (start with /)

Service-Specific Configuration¶

MongoDB Backup¶

mongodbBackup:
  enabled: true
  instance: secondary
  resticPrune: 
    keepBackupUntil: 3d
  schedule: "0 */10 * * *"

Parameters:

instance: Which MongoDB instance to backup
secondary - For multi-node setups (recommended)
primary - For single-node setups
keepBackupUntil: Retention period (e.g., 3d for 3 days)
schedule: Cron expression for backup frequency

PostgreSQL Backup¶

postgresqlBackup:
  enabled: true
  password: "{{ .tfi.postgresqlPostgresPassword.value }}"
  resticPrune:  
    keepBackupUntil: 3d
  schedule: "0 */9 * * *"

Parameters:

password: Password for the postgres user (uses template variable)

Schema Registry Backup¶

schemaRegistryBackup:
  enabled: true
  resticPrune:  
    keepBackupUntil: 3d
  schedule: "0 */11 * * *"

Zookeeper Backup¶

zookeeperBackup:
  enabled: true
  resticPrune:  
    keepBackupUntil: 3d
  schedule: "0 */12 * * *"

Traceable Installer Backup¶

installerBackup:
  enabled: true
  schedule: "0 */12 * * *"
  resticPrune:
    keepBackupUntil: 3d

Note: Available in version 0.1.26 onwards. Credentials for TI login are configured in installerBackup.username and installerBackup.password.

Complete Configuration Example¶

# ============================================
# COMMON CONFIGURATION
# ============================================
clusterSuffix: "primary"  # Identifier in backup filenames
imagePullSecrets:
    - name: regcred

# ============================================
# RESTIC CONFIGURATION (Backup Engine)
# ============================================
restic:
  enabled: true
  password: "<strong-restic-password>"  # IMPORTANT: Store securely, needed for restore
  repository: "/backups"                 # Must match extraVolumeMounts.mountPath

# ============================================
# VOLUME CONFIGURATION
# ============================================
extraVolumeMounts:
    - mountPath: "/backups"
    name: platform-backup-vol

extraVolumes:
    - name: platform-backup-vol
    persistentVolumeClaim:
      claimName: platform-backup-pvc

# ============================================
# SERVICE-SPECIFIC BACKUP SCHEDULES
# ============================================

# MongoDB Backup
mongodbBackup:
  enabled: true
  instance: secondary        # Use 'primary' for single-node MongoDB
  resticPrune:
    keepBackupUntil: 3d      # Retention period
  schedule: "0 */6 * * *"    # Every 6 hours

# PostgreSQL Backup
postgresqlBackup:
  enabled: true
  resticPrune:
    keepBackupUntil: 3d
  schedule: "0 */6 * * *"

# Schema Registry Backup
schemaRegistryBackup:
  enabled: true
  resticPrune:
    keepBackupUntil: 3d
  schedule: "0 */6 * * *"

# Zookeeper Backup (Kafka metadata + Pinot cluster state)
zookeeperBackup:
  enabled: true
  resticPrune:
    keepBackupUntil: 3d
  schedule: "0 */6 * * *"

# Traceable Installer Backup (TI config/secrets)
installerBackup:
  enabled: true
  resticPrune:
    keepBackupUntil: 3d
  schedule: "0 */12 * * *"

On-Demand Backup¶

To create an immediate backup outside the scheduled time:

# For Postgresql
kubectl create job backup-postgresql-ondemand-job --from=cronjob/backup-service-postgresql -n traceable
# For MongoDB
kubectl create job backup-mongodb-ondemand-job --from=cronjob/backup-service-mongodb -n traceable
# For Zookeeper
kubectl create job backup-zookeeper-ondemand-job --from=cronjob/backup-service-zookeeper -n traceable
# For Schema Registry
kubectl create job backup-schema-registry-ondemand-job --from=cronjob/backup-service-schema-registry -n traceable
# For Installer
kubectl create job backup-traceable-installer-ondemand-job --from=cronjob/backup-service-traceable-installer -n traceable

Validate that all the backup services are completed without any issues.

# For Postgresql
kubectl wait --for=condition=complete job/backup-postgresql-ondemand-job -n traceable --timeout=600s
# For MongoDB
kubectl wait --for=condition=complete job/backup-mongodb-ondemand-job -n traceable --timeout=600s
# For Zookeeper
kubectl wait --for=condition=complete job/backup-zookeeper-ondemand-job -n traceable --timeout=600s
# For Schema Registry
kubectl wait --for=condition=complete job/backup-schema-registry-ondemand-job -n traceable --timeout=600s
# For Installer
kubectl wait --for=condition=complete job/backup-traceable-installer-ondemand-job -n traceable --timeout=600s

Cron Schedule Reference¶

Use crontab.guru to generate cron expressions.

Common Schedules¶

Schedule	Cron Expression
Every 6 hours	`0 /6 * *`
Every 12 hours	`0 /12 * *`
Daily at midnight	`0 0 * * *`
Daily at 2 AM	`0 2 * * *`
Weekly on Sunday	`0 0 * * 0`

Verification¶

Check CronJobs¶

kubectl get cronjobs -n traceable | grep backup

Check Recent Jobs¶

kubectl get jobs -n traceable | grep backup

Check Backup Logs¶

kubectl logs -n traceable -l app=backup-service --tail=100

Verify Backup Files¶

kubectl exec -it -n traceable <backup-pod> -- ls -la /backup

Troubleshooting¶

Backup Job Failing¶

Symptoms: Backup jobs show Failed status.

Solutions:

Check job logs:

kubectl logs -n traceable job/<job-name>

Verify PVC is mounted correctly
Check storage space availability
Verify service connectivity (MongoDB, PostgreSQL, etc.)

Insufficient Storage¶

Symptoms: Backups fail with "no space left" errors.

Solutions:

Increase PVC size
Reduce retention period
Clean up old backups manually

Restic Repository Issues¶

Symptoms: Restic errors in backup logs.

Solutions:

Verify restic password is correct
Check repository path exists
Initialize repository if needed:

restic -r /backup init

Best Practices¶

Test Restores: Regularly test restore procedures
Monitor Backups: Set up alerts for backup failures
Offsite Storage: Consider copying backups to offsite storage
Encryption: Always use restic encryption (enabled by default)
Retention Policy: Balance storage costs with recovery needs
Documentation: Document backup schedules and retention policies
Stagger Schedules: Avoid running all backups at the same time

Support¶

For backup assistance, contact: support@harness.io

Provide the following information:

Platform version
Backup service configuration
Job logs
Error messages

Backup Your On-Prem Cluster¶

Overview¶

Prerequisites¶

Enable Backup Service¶

Create Backup PVC¶

Calculating Storage Size¶

Create a PersistentVolumeClaim for storing backups¶

Configuration¶

Common Configuration¶

Restic Configuration¶

PVC Configuration¶

Service-Specific Configuration¶

MongoDB Backup¶

PostgreSQL Backup¶

Schema Registry Backup¶

Zookeeper Backup¶

Traceable Installer Backup¶

Complete Configuration Example¶

On-Demand Backup¶

Cron Schedule Reference¶

Common Schedules¶

Verification¶

Check CronJobs¶

Check Recent Jobs¶

Check Backup Logs¶

Verify Backup Files¶

Troubleshooting¶

Backup Job Failing¶

Insufficient Storage¶

Restic Repository Issues¶

Best Practices¶

Related Documentation¶

Support¶