On-Premises Disaster Recovery Runbook¶

Table of Contents¶

Overview
Architecture
Prerequisites
Part 1: DR Setup (One-Time Configuration)
Part 2: Failover Procedure
Part 3: Failback Procedure
Caveats: Known Issues
Additional Considerations
Troubleshooting
Appendix

Overview¶

This document provides end-to-end instructions for setting up Disaster Recovery (DR) for an on-premises Traceable cluster.

Component	Purpose
Traceable Installer	Traceable Installer (TI) is the primary means of deploying and managing the Traceable platform.
Apache Pinot	Real-time OLAP datastore for analytics (stores ingested traffic data)
PostgreSQL	Relational database for platform metadata, configurations
MongoDB	Document store for API catalogs, policies, threat intelligence
Kafka	Event streaming backbone for data ingestion pipeline
Zookeeper	Coordination service for Kafka and Pinot cluster state
Schema Registry	Manages Kafka message schemas (Avro/Protobuf)

DR Strategy¶

Aspect	Approach
RTO (Recovery Time Objective)	~30-60 minutes (depending on data volume)
RPO (Recovery Point Objective)	Based on backup schedule + last NFS snapshot
Sync Method	NFS disk snapshots (customer-managed) + backup-service for databases
Failover Type	Manual (requires operator intervention)
Cluster Mode	Active-Passive (only one cluster can be active at a time)

Architecture¶

┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                     PRIMARY SITE                                                       │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  │
│  │  Traceable  │  │ Zookeeper/  │  │  Pinot      │  │  PostgreSQL │  │   MongoDB   │  │  NFS Model  │  │
│  │  Operator   │  │ Schema Reg/ │  │  Server/    │  │             │  │             │  │      /      │  │
│  │             │  │   Kafka     │  │  Controller │  │             │  │             │  │NFS DeepStore│  │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘  │
│         │                │                │                │                │                │         │
│         └────────────────┴────────────────┴────────────────┘────────────────┘────────────────┘         │
│                                           │                                                            │
│                                  ┌────────▼────────┐                                                   │
│                                  │   NFS Storage   │                                                   │
│                                  │  (Primary PVs)  │                                                   │
│                                  └────────┬────────┘                                                   │
└───────────────────────────────────────────┼────────────────────────────────────────────────────────────┘
                                            │
                                  ┌─────────▼─────────┐
                                  │  Disk Snapshot &  │
                                  │     Restore       │
                                  │ (Customer Managed)│
                                  └─────────┬─────────┘
                                            │
┌───────────────────────────────────────────┼─────────────────────────────────────────────────────────────┐
│                                       DR SITE                                                           │
│                                  ┌────────▼────────┐                                                    │
│                                  │   NFS Storage   │                                                    │
│                                  │    (DR PVs)     │                                                    │
│                                  └────────┬────────┘                                                    │
│         ┌────────────────┬────────────────┼────────────────┬─────────────────┬────────────────┐         │
│         │                │                │                │                 │                │         │
│  ┌──────▼──────┐  ┌──────▼──────┐  ┌──────▼──────┐  ┌───────▼──────┐  ┌──────▼──────┐  ┌──────▼──────┐  │
│  │  Traceable  │  │ Zookeeper/  │  │  Pinot      │  │  PostgreSQL  │  │   MongoDB   │  │   NFS Model │  │
│  │  Operator   │  │ Schema Reg/ │  │  Server/    │  │              │  │             │  │      /      │  │
│  │             │  │   Kafka     │  │  Controller │  │              │  │             │  │NFS DeepStore│  │
│  └─────────────┘  └─────────────┘  └─────────────┘  └──────────────┘  └─────────────┘  └─────────────┘  │
│                                    SECONDARY SITE                                                       │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Prerequisites¶

Infrastructure Requirements¶

Requirement	Details
Traceable Platform Version	≥ 1.14.0 (backup-service dependency)
Kubernetes Clusters	Primary + Secondary (identical specs recommended)
NFS Storage	RWX-capable storage class on both clusters
Network Connectivity	Secondary cluster must reach NFS for snapshot restore
DNS Management	Ability to update DNS records for failover

Pre-Configuration Checklist¶

Secondary Kubernetes cluster provisioned with same resource capacity
NFS storage provisioned on secondary site
Disk snapshot mechanism configured (vendor-specific: NetApp, Pure, etc.)
kubectl access to both clusters
Helm 3.x installed
Traceable Installer (TI) credentials available

Configure NFS Snapshot Schedule¶

Customer Action Required:

Configure your storage vendor's snapshot mechanism to match backup schedules:

Vendor	Documentation
NetApp	SnapMirror
Pure Storage	ActiveCluster
Dell EMC	RecoverPoint
Generic NFS	rsync-based replication with cron

Recommended Schedule: Ensure snapshots of key PVCs (controller, backup, deep-store, model-store) are taken regularly, aligned with backup-service schedules.

Identify PVCs to Backup¶

# List PVCs matching the patterns
kubectl get pvc -n traceable | grep -E 'controller|backup|deep-store|model-store'

Find NFS Paths for PVCs¶

To locate the underlying NFS paths for snapshotting:

# Get PV details for the PVCs
kubectl get pv -n traceable -o jsonpath='{range .items[?(@.spec.nfs)]}{.metadata.name}{"\t"}{.spec.nfs.server}{"\t"}{.spec.nfs.path}{"\n"}{end}' | grep -E 'controller|backup|deep-store|model-store'

Use the extracted server and path values to configure your storage vendor's snapshot tool (e.g., NetApp SnapMirror, Pure ActiveCluster).

Part 1: DR Setup (One-Time Configuration)¶

Step 1: Configure Backup Service on Primary¶

Follow steps in backup

Step 2: Export PV/PVC Definitions from Primary¶

Extract persistent volume configurations for critical services:

# Create directories for exports
mkdir -p traceable-pv traceable-pvc

# Export PV definitions (removes claimRef to allow rebinding)
kubectl get pv | grep -v 'NAME' \
  | grep -E 'controller|backup|deep-store|model-store' \
  | awk '{print $1}' \
  | xargs -I {} bash -c 'kubectl get pv {} -o yaml | sed "/claimRef:/,+6 d" > traceable-pv/{}.yaml'

# Export PVC definitions
kubectl get pvc -n traceable | grep -v 'NAME' \
  | grep -E 'controller|backup|deep-store|model-store' \
  | awk '{print $1}' \
  | xargs -I {} bash -c 'kubectl get pvc -n traceable {} -o yaml > traceable-pvc/{}.yaml'

If you are not taking backup of Traceable installer using backup service, then you need to sync the PVC of the Traceable Installer as well

Optional: Sync Pinot Server PVCs

In addition to the critical PVCs (controller, backup, deep-store, model-store), you can optionally sync Pinot server PVCs to the DR site. Pinot servers store local copies of segments for faster query performance, replicating data from the controller and deep-store.

Pros:

Faster Pinot server startup during failover, as servers load segments directly from local PVCs instead of copying from the controller, reducing recovery time.

Cons:

Increases data sync overhead, as Pinot server PVCs can be large (depending on dataset size), requiring more bandwidth and storage during snapshot replication from primary to DR.

How to Include: Update the export commands to include Pinot PVCs:

Add a new entry in the grep command to include the pinot-server

# Export PV definitions (add 'pinot' to the pattern)
kubectl get pv | grep -v 'NAME' \
  | grep -E 'installer|controller|backup|deep-store|model-store|pinot-server' \
  | awk '{print $1}' \
  | xargs -I {} bash -c 'kubectl get pv {} -o yaml | sed "/claimRef:/,+6 d" > traceable-pv/{}.yaml'

# Export PVC definitions
kubectl get pvc -n traceable | grep -v 'NAME' \
  | grep -E 'installer|controller|backup|deep-store|model-store|pinot-server' \
  | awk '{print $1}' \
  | xargs -I {} bash -c 'kubectl get pvc -n traceable {} -o yaml > traceable-pvc/{}.yaml'

Step 3: Transfer Exports to Secondary Site¶

These exported PV/PVC YAML files contain the definitions of persistent volumes and claims from the primary cluster. Transferring them to the secondary site ensures the DR cluster has identical storage configurations, enabling seamless data restoration during failover.

# SCP or rsync to secondary cluster's bastion/admin node
scp -r traceable-pv traceable-pvc user@dr-admin-node:/path/to/dr-setup/

Copy these files to the DR cluster's admin node (or a server with kubectl access to the DR cluster) to apply the PV/PVC resources in the next step.

If SCP access is unavailable, create single YAML files for each type (PVs and PVCs) and copy-paste their contents to the DR site.

# Create single PV YAML file
echo "" > traceable-pv/all-pvs.yaml
for file in traceable-pv/*.yaml; do
  cat "$file" >> traceable-pv/all-pvs.yaml
  echo "---" >> traceable-pv/all-pvs.yaml
done

# Create single PVC YAML file
echo "" > traceable-pvc/all-pvcs.yaml
for file in traceable-pvc/*.yaml; do
  cat "$file" >> traceable-pvc/all-pvcs.yaml
  echo "---" >> traceable-pvc/all-pvcs.yaml
done

Step 4: Create PV/PVC on Secondary Cluster¶

The secondary cluster requires the same PVs and PVCs as the primary for critical components (e.g., controller, backup, deep-store, model-store). However, the NFS server IP differs between sites, so update the PV definitions to point to the secondary's NFS server before applying. This creates pre-bound PVs/PVCs, allowing NFS snapshots to restore data directly into them during failover.

Important: Update NFS server details before applying.

# Set environment variables
export SOURCE_NFS="192.168.0.10"    # Primary NFS IP
export DEST_NFS="172.168.0.20"      # Secondary NFS IP

# Apply PVs with updated NFS server
cd traceable-pv
for pv in *.yaml; do
  sed "s/${SOURCE_NFS}/${DEST_NFS}/g" "$pv" | kubectl apply -f -
done

#Create traceable namespace
kubectl create ns traceable

# Apply PVCs
cd ../traceable-pvc
for pvc in *.yaml; do
  kubectl apply -f "$pvc"
done

Step 5: Install Traceable on Secondary (Standby Mode)¶

5.1 Create TI Values for DR¶

It should be same as Primary with the only difference between the NFS_SERVER

Sample:

# traceable-installer-dr-values.yaml
storageClassName: nfs-client
cluster:
  profile: xsmall              # Match primary cluster profile
  triggerInstall: false        # CRITICAL: Prevents auto-installation
  tfi:
    storageClass: nfs-client

traceable-nginx:
  enabled: true

nfs-provisioner:
  enabled: true
  nfs:
    server: "${NFS_SERVER}" #UPDATE
    path: "${NFS_PATH}" #UPDATE

If you are unsure about or have lost the override file, retrieve it using the following command on primary cluster:

helm get values traceable-installer -n traceable

5.2 Install TI on Secondary¶

./clustermgr install-ti \
  --dns=<platform-dns-name> \
  --registry-host="pkg.harness.io/ieq3j_9otlwbpfz-uryoda/docker-external" \
  --registry-prefix="" \
  --email admin@example.com \
  --password <ti-password> \
  --registry-user <registry-user> \
  --registry-password <registry-password> \
  --ti-values ./traceable-installer-dr-values.yaml

Make sure you use the same email and password credentials to avoid unforeseen issues

Step 6: Configure CoreDNS on Secondary¶

Update CoreDNS to resolve platform DNS internally:

# For Kurl
kubectl edit cm -n kube-system coredns

# For RKE
kubectl edit cm -n kube-system rke2-coredns-rke2-coredns

If coredns configmap is not there, run kubectl get cm -n kube-system to get the configmap name

Add the hosts block inside the Corefile:

hosts {
    172.168.0.11 platform.example.com
    172.168.0.12 platform.example.com
    172.168.0.13 platform.example.com
    fallthrough
}

Notes:

List all master/worker node IPs that run ingress
fallthrough is mandatory - without it, all other DNS resolution falls to the primary cluster.
Each entry must be on a separate line

# Restart CoreDNS to apply changes

# For kurl
kubectl rollout restart deployment coredns -n kube-system

For RKE2
kubectl rollout restart deployment rke2-coredns-rke2-coredns -n kube-system

Step 7: Access the DR TI¶

Option 1: Port forward¶

Port-forward the traceable-installer svc
kubectl port-forward -n traceable svc/traceable-installer 8000:8000 &
Access the TI using localhost:8000

You will not be able to access the platform

Option 2: Update local DNS entry¶

Open host file
For Linux/Mac: /etc/hosts
For Windows: C:\Windows\System32\drivers\etc\hosts
Add an entry of the Load Balancer IP
172.168.0.11 platform.example.com
Save and Quit
Login to the platform using the updated local DNS entry

Option 3: Use docker based browser¶

Start the firefox in docker

docker run -d \
  --name=firefox \
  -e PUID=1000 \
  -e PGID=1000 \
  -e TZ=Etc/UTC \
  -e FIREFOX_CLI=https://www.linuxserver.io/ `#optional` \
  -p 3000:3000 \
  -p 3001:3001 \
  --shm-size="2gb" \
  --add-host=platform.example.com:172.168.0.11 #`Update this` \
  --restart unless-stopped \
  lscr.io/linuxserver/firefox:latest

Access the firefox in your browser by accessing port localhost:3000
Using the browser, access the platform

Option 4: Install firefox in the Cluster¶

Using the below YAML, we deploy Firefox as a Kubernetes resource and use an Ingress to access it. This allows accessing the platform directly via the Firefox browser within the cluster.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: firefox
  labels:
    app: firefox
spec:
  replicas: 1
  selector:
    matchLabels:
      app: firefox
  template:
    metadata:
      labels:
        app: firefox
    spec:
      containers:
      - name: firefox
        image: lscr.io/linuxserver/firefox:latest
        ports:
        - containerPort: 3000
          name: http
          protocol: TCP
        - containerPort: 3001
          name: https
          protocol: TCP
        env:
        - name: PUID
          value: "1000"
        - name: PGID
          value: "1000"
        - name: TZ
          value: "Etc/UTC"
        - name: FIREFOX_CLI
          value: "https://www.linuxserver.io/"
        resources:
          limits:
            memory: "3Gi"
          requests:
            memory: "2Gi"
        volumeMounts:
        - name: dshm
          mountPath: /dev/shm
      volumes:
      - name: dshm
        emptyDir:
          medium: Memory
          sizeLimit: "2Gi"
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: firefox
  labels:
    app: firefox
spec:
  type: ClusterIP
  ports:
    - port: 3000
    targetPort: 3000
    protocol: TCP
    name: http
    - port: 3001
    targetPort: 3001
    protocol: TCP
    name: https
  selector:
    app: firefox
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: firefox
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
    - host: masternode-IP.sslip.io  # Update this to your desired hostname (masternode.sslip.io)
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: firefox
            port:
              number: 3000
  # Uncomment below for TLS/SSL
  # tls:
  # - hosts:
  #   - firefox.example.com
  #   secretName: firefox-tls

In your browser use https://masternode-IP.sslip.io to access the firefox

Part 2: Failover Procedure¶

Pre-Failover Checklist¶

Confirm primary site is unavailable or decision to failover is made
Verify latest NFS snapshot is available on DR site
Confirm backup-service backups are accessible
Notify stakeholders of planned failover
Document the timestamp of last known good state

Step 1: Scale Down Secondary Cluster¶

Login to TI
Click on Tasks
Select ShutdownWorkloads
Skip shutdown-traceable-operator and shutdown-monitoring
Click on Start Execution

Step 2: Restore NFS Snapshot on DR Site¶

Customer Action: Restore the latest NFS snapshot to DR NFS storage using your storage vendor's tools.

# Verify PVCs are bound after snapshot restore
kubectl get pvc -n traceable

Step 3: Prepare restore server¶

Copy the override file from the backup-service as this is common for both.

# ============================================
# COMMON CONFIGURATION
# ============================================
clusterSuffix: "primary"  # Identifier in backup filenames
imagePullSecrets:
    - name: regcred

# ============================================
# RESTIC CONFIGURATION (Backup Engine)
# ============================================
restic:
  enabled: true
  password: "<strong-restic-password>"  # IMPORTANT: Store securely, needed for restore
  repository: "/backups"                 # Must match extraVolumeMounts.mountPath

# ============================================
# VOLUME CONFIGURATION
# ============================================
extraVolumeMounts:
    - mountPath: "/backups"
    name: platform-backup-vol

extraVolumes:
    - name: platform-backup-vol
    persistentVolumeClaim:
      claimName: platform-backup-pvc

Step 4: Restore Traceable Installer (TI)¶

4.1 Configure Restore Values¶

Update restore-service/user/values.yaml:

restoreJobs:
  installer:
    enabled: true

4.2 Execute Restore¶

# 1. Perform Helm diff to validate changes
#    TI UI: Tasks | restore-service → HELM ACTIONS → Helm Diff

# 2. Verify job 'backup-service-installer-restore' will be created

# 3. Wait for the job to get completed
kubectl wait --for=condition=complete job/restore-service-traceable-installer-restore -n traceable --timeout=900s

# 3. Restart TI
kubectl rollout restart sts traceable-installer -n traceable

Refresh the page and relogin to the TI

Step 5: Restore PostgreSQL¶

Refer restore postgres

Monitor the restore status¶

# Helm upgrade backup-service, wait for job completion
kubectl wait --for=condition=complete job/restore-service-postgresql-restore -n traceable --timeout=900s

# Check logs for errors
kubectl logs job/restore-service-postgresql-restore -n traceable

Step 6: Restore MongoDB¶

Refer restore mongo

Monitor the restore status¶

kubectl wait --for=condition=complete job/restore-service-mongodb-restore -n traceable --timeout=1800s
kubectl logs job/backup-service-mongodb-restore -n traceable

Step 7: Restore Zookeeper¶

7.1 Install Zookeeper¶

# Via TI UI: Helm Releases → Zookeeper → HELM UPGRADE

7.2 Delete Stale Pinot Metadata¶

Critical: Pinot cluster state in Zookeeper references primary cluster. Must be cleared.

# Connect to Zookeeper pod
kubectl exec -it zookeeper-0 -n traceable -- /opt/zookeeper/bin/zkCli.sh

# Inside zkCli:
deleteall /pinot
quit

7.3 Configure and Execute Restore¶

Refer restore zookeeper

Monitor the restore status¶

kubectl wait --for=condition=complete job/restore-service-zookeeper-restore -n traceable --timeout=600s

Step 8: Restore Schema Registry¶

8.1 Install Kafka¶

# Via TI UI: Helm Releases → Kafka → HELM UPGRADE

8.2 Install Schema Registry¶

# Via TI UI: Helm Releases → Schema Registry → HELM UPGRADE

8.3 Restore the schema registry¶

Refer restore schema registry

Monitor the restore status¶

kubectl wait --for=condition=complete job/restore-service-schema-registry-restore -n traceable --timeout=600s

Step 9: Deploy Pinot (Staged Rollout)¶

Important: Pinot must be deployed in stages to prevent cluster state corruption.

9.1 Deploy Controller Only¶

kubectl scale pinot-controller --replicas=1 -n traceable

# Wait for controller to be ready
kubectl wait --for=condition=ready pod -l app=pinot,component=controller -n traceable --timeout=300s

9.2 Install Pinot Service¶

# Apply via TI UI: Helm Releases → Pinot → HELM UPGRADE

# Wait for all Pinot components
kubectl wait --for=condition=ready pod -l app=pinot,component=controller -n traceable --timeout=300s
kubectl wait --for=condition=ready pod -l app=pinot,component=server -n traceable --timeout=600s
kubectl wait --for=condition=ready pod -l app=pinot,component=broker -n traceable --timeout=300s

Step 10: Start Remaining Services¶

# Via TI UI: Install Workflow → Start Install

# Verify all pods are running
kubectl get pods -n traceable

Step 11: Update DNS¶

Update your DNS provider to point the platform FQDN to the DR cluster's ingress IP(s).

# Example: If using AWS Route53
aws route53 change-resource-record-sets --hosted-zone-id <zone-id> --change-batch '{
  "Changes": [{
    "Action": "UPSERT",
    "ResourceRecordSet": {
      "Name": "platform.example.com",
      "Type": "A",
      "TTL": 60,
      "ResourceRecords": [{"Value": "172.168.0.100"}]
    }
  }]
}'

Step 12: Validation¶

# 1. Test platform login
curl -k https://platform.example.com/

# 2. Verify data integrity
#    - Login to UI with existing credentials
#    - Check API catalog is present
#    - Verify historical traffic data in analytics
#    - Confirm threat detection rules are intact

Part 3: Failback Procedure¶

Failback restores operations to the primary site after it's recovered.

Pre-Failback Checklist¶

Primary site infrastructure is fully restored
NFS storage on primary is operational
Network connectivity verified
Maintenance window scheduled

Step 1: Sync Data from DR to Primary¶

NFS Snapshot Reverse Sync¶

Configure your storage vendor to replicate DR NFS to Primary NFS.

Backup-Service Export¶

# On DR cluster, trigger manual backups
kubectl create job --from=cronjob/backup-service-mongodb-backup manual-mongo-backup -n traceable
kubectl create job --from=cronjob/backup-service-postgresql-backup manual-postgres-backup -n traceable
kubectl create job --from=cronjob/backup-service-zookeeper-backup manual-zk-backup -n traceable
kubectl create job --from=cronjob/backup-service-schema-registry-backup manual-sr-backup -n traceable

# Wait for completion
kubectl wait --for=condition=complete job/manual-mongo-backup -n traceable --timeout=1800s

Step 2: Scale Down DR Cluster¶

Login to TI
Click on Tasks
Select ShutdownWorkloads
Skip shutdown-traceable-operator and shutdown-monitoring
Click on Start Execution

Step 3: Restore Primary Cluster¶

Follow the same restore procedure as Part 2: Failover Procedure, but targeting the primary cluster.

Step 4: Update DNS to Primary¶

Revert DNS records to point to primary cluster ingress.

Step 5: Reconfigure DR as Standby¶

Login to TI
Click on Tasks
Select ShutdownWorkloads
Skip shutdown-traceable-operator and shutdown-monitoring
Click on Start Execution

Re-enable backup sync from primary

Caveats: Known Issues¶

1. Kafka Consumer Offset Handling¶

Issue	Impact	Mitigation
Consumer offsets are stored in Zookeeper	After failover, consumers may reprocess or skip messages	Set `resetPinotOffsets: true` in Zookeeper restore; accept potential data duplication

2. Pinot Segment State¶

Issue	Impact	Mitigation
Pinot segments reference primary cluster paths	Segment loading fails on DR	Delete `/pinot` znode before restore; Pinot will rebuild from deep-store

3. Schema Registry ID Conflicts¶

Issue	Impact	Mitigation
Schema IDs are auto-incremented	New schemas on DR may conflict when failing back	Use schema registry backup/restore; avoid schema changes during DR

4. MongoDB Replica Set Configuration¶

Issue	Impact	Mitigation
Replica set members reference primary hostnames	MongoDB may fail to elect primary	Backup from `secondary` instance; verify replica set config post-restore

5. PostgreSQL Sequences¶

Issue	Impact	Mitigation
Auto-increment sequences may have gaps	Minor; no functional impact	Acceptable; sequences will continue from restored state

6. Incomplete Transactions¶

Issue	Impact	Mitigation
In-flight transactions at failure time are lost	Data loss for uncommitted transactions	RPO is limited by backup frequency + snapshot timing

7. Monitoring & Alerting¶

Issue	Impact	Mitigation
Prometheus/Grafana data is site-specific	Historical metrics lost	Do not sync monitoring PVCs; accept fresh metrics on DR

Additional Considerations¶

Cluster Usage Limitations¶

Single Active Cluster: Only one cluster (primary or secondary) can be active at any given time
No Active-Active Configuration: The system does not support active-active configuration
Secondary Cluster State: The secondary (DR) cluster is always down when not in use
Failover Process: Requires explicit manual failover to activate the secondary cluster

Stop the Backup service¶

To avoid corruption of backup of the databases, uninstall the backup service from the secondary cluster

# Login to TI -> Task -> Backup-Service
# Click on Helm Action -> Helm uninstall

Data Gaps¶

Cause: Pinot's retention policy (e.g., 7 days) purges old data, while periodic backups (every 6 hours) and disk snapshots capture point-in-time states, leading to gaps in pre-retention and post-backup data.

Impact: Results in "head" gaps (data older than retention window) and "tail" gaps (data newer than last backup), aligning with RPO.

Example: 7-day retention; last backup 1 day ago; restore on day 8 recovers data from days 2-7; days 1 and 8+ are lost.

Update Traceable Platform Agent and Runner¶

Depending on the DR implementation, update the DNS records for Traceable Platform Agents and Runners to point to the active cluster. Then, restart these services to ensure they resume reporting to the new active cluster.

Network & Security¶

Firewall Rules: Ensure DR cluster has identical ingress/egress rules
Network Policies: Replicate NetworkPolicy resources

Performance¶

Resource Parity: DR cluster should match primary's CPU/memory allocation
Storage IOPS: Ensure DR NFS has equivalent performance characteristics
Pinot Deep Store: Verify deep-store is accessible and has sufficient throughput

Operational¶

Runbook Testing: Perform DR drills quarterly
Backup Verification: Periodically test restore from backups
Documentation: Keep this runbook updated with environment-specific values
Communication Plan: Define escalation paths and notification procedures

Data Consistency¶

RPO Acceptance: Document accepted data loss window with stakeholders
Transaction Boundaries: Understand which operations are atomic
Eventual Consistency: Kafka-based systems may have lag; account for this

Kafka-Specific Considerations¶

Topic Replication: Topics are not replicated cross-cluster; data is in Pinot
Consumer Groups: Will be reset; consumers restart from configured offset
Producer Acknowledgments: In-flight produces at failure time are lost

Pinot-Specific Considerations¶

Segment Retention: Ensure deep-store retention aligns with DR requirements
Real-time vs Offline: Real-time segments since last snapshot are lost
Query Routing: Broker routing tables rebuild automatically

Troubleshooting¶

Backup Job Failures¶

# Check job status
kubectl get jobs -n traceable | grep backup

# View job logs
kubectl logs job/<job-name> -n traceable

# Common issues:
# - PVC not mounted: Check extraVolumes configuration
# - Restic password mismatch: Verify password in values.yaml
# - Insufficient storage: Expand backup PVC

Restore Job Failures¶

# Check restore job status
kubectl get jobs -n traceable | grep restore

# View logs
kubectl logs job/<restore-job-name> -n traceable

# Common issues:
# - Backup not found: Verify restic repository path
# - Permission denied: Check PVC mount permissions
# - Service not ready: Ensure target service is running

Pinot Segment Loading Issues¶

# Check Pinot controller logs
kubectl logs -l app=pinot,component=controller -n traceable

# Force segment reload
curl -X POST "http://pinot-controller:9000/segments/<table>/reload?forceDownload=true"

MongoDB Connection Issues¶

# Check replica set status
kubectl exec -it mongo-primary-0 -n traceable -- mongo --eval "rs.status()"

PostgreSQL Connection Issues¶

# Check PostgreSQL logs
kubectl logs -l app.kubernetes.io/name=postgresql -n traceable

# Test connection
kubectl exec -it postgresql-0 -n traceable -- psql -U postgres -c "SELECT 1"

DNS Resolution Issues¶

# Test from within cluster
kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup platform.example.com

# Check CoreDNS logs
kubectl logs -l k8s-app=kube-dns -n kube-system

Appendix¶

A. Backup Schedule Reference¶

Service	Default Schedule	Cron Expression
MongoDB	Every 6 hours	`0 /6 * *`
PostgreSQL	Every 6 hours	`0 /6 * *`
Schema Registry	Every 6 hours	`0 /6 * *`
Zookeeper	Every 6 hours	`0 /6 * *`
TI Config	Every 12 hours	`0 /12 * *`

B. Critical Secrets Reference¶

Secret	Location	Purpose
`postgresqlPostgresPassword`	TI Global Variables	PostgreSQL superuser
`restic.password`	backup-service values	Backup encryption
`regcred`	Kubernetes Secret	Container registry auth

C. Useful Commands¶

# Quick cluster health check
kubectl get pods -n traceable | grep -v Running | grep -v Completed

# Check all PVC status
kubectl get pvc -n traceable

# View recent events
kubectl get events -n traceable --sort-by='.lastTimestamp' | tail -20

# Force delete stuck pod
kubectl delete pod <pod-name> -n traceable --grace-period=0 --force

# Trigger manual backup
kubectl create job --from=cronjob/<cronjob-name> manual-backup-$(date +%s) -n traceable

On-Premises Disaster Recovery Runbook¶

Table of Contents¶

Overview¶

DR Strategy¶

Architecture¶

Prerequisites¶

Infrastructure Requirements¶

Pre-Configuration Checklist¶

Configure NFS Snapshot Schedule¶

Identify PVCs to Backup¶

Find NFS Paths for PVCs¶

Part 1: DR Setup (One-Time Configuration)¶

Step 1: Configure Backup Service on Primary¶

Step 2: Export PV/PVC Definitions from Primary¶

Step 3: Transfer Exports to Secondary Site¶

Step 4: Create PV/PVC on Secondary Cluster¶

Step 5: Install Traceable on Secondary (Standby Mode)¶

5.1 Create TI Values for DR¶

5.2 Install TI on Secondary¶

Step 6: Configure CoreDNS on Secondary¶

Step 7: Access the DR TI¶

Option 1: Port forward¶

Option 2: Update local DNS entry¶

Option 3: Use docker based browser¶

Option 4: Install firefox in the Cluster¶

Part 2: Failover Procedure¶

Pre-Failover Checklist¶

Step 1: Scale Down Secondary Cluster¶

Step 2: Restore NFS Snapshot on DR Site¶

Step 3: Prepare restore server¶

Step 4: Restore Traceable Installer (TI)¶

4.1 Configure Restore Values¶

4.2 Execute Restore¶

4.3 Login to the TI¶

Step 5: Restore PostgreSQL¶

Monitor the restore status¶

Step 6: Restore MongoDB¶

Monitor the restore status¶

Step 7: Restore Zookeeper¶

7.1 Install Zookeeper¶

7.2 Delete Stale Pinot Metadata¶

7.3 Configure and Execute Restore¶

Monitor the restore status¶

Step 8: Restore Schema Registry¶

8.1 Install Kafka¶

8.2 Install Schema Registry¶

8.3 Restore the schema registry¶

Monitor the restore status¶

Step 9: Deploy Pinot (Staged Rollout)¶

9.1 Deploy Controller Only¶

9.2 Install Pinot Service¶

Step 10: Start Remaining Services¶

Step 11: Update DNS¶

Step 12: Validation¶

Part 3: Failback Procedure¶

Pre-Failback Checklist¶

Step 1: Sync Data from DR to Primary¶

NFS Snapshot Reverse Sync¶

Backup-Service Export¶

Step 2: Scale Down DR Cluster¶

Step 3: Restore Primary Cluster¶

Step 4: Update DNS to Primary¶

Step 5: Reconfigure DR as Standby¶

Caveats: Known Issues¶

1. Kafka Consumer Offset Handling¶

2. Pinot Segment State¶

3. Schema Registry ID Conflicts¶

4. MongoDB Replica Set Configuration¶

5. PostgreSQL Sequences¶

6. Incomplete Transactions¶

7. Monitoring & Alerting¶

Additional Considerations¶

Cluster Usage Limitations¶

Stop the Backup service¶

Data Gaps¶

Update Traceable Platform Agent and Runner¶

Network & Security¶

Performance¶

Operational¶

Data Consistency¶