Remote-Pulse DR Drill Procedure

Status: Active
Owner: Ramon ([email protected])
Cadence: Quarterly (Q1, Q2, Q3, Q4)
Related: ADR-0008 Appendix F (DR procedure)

---

Purpose

Validate disaster recovery procedures for Remote-Pulse LXC 280 (rp-server). Measures RTO (Recovery Time Objective) and RPO (Recovery Point Objective) for documented failure scenarios.

Pre-requisites

Before executing drill:

• Access to Proxmox cluster (pmx-50 or pmx-51) via SSH
• Access to PBS (Proxmox Backup Server) at 192.168.0.186
• NAS mount /mnt/nas/backups/remote-pulse available
• SOPS age keys unlocked (for secrets)
• Healthchecks URL configured (optional, for verification)
• Telegram bot access for notifications (@Veraclawd_bot)
• Test LXC ID reserved: 283 (do NOT use production 280)

Quarterly Drill Schedule

Quarter	Target Month	Drill Window	Assigned
Q1 2026	January	Jan 15-31	Ramon
Q2 2026	April	Apr 15-30	Ramon
Q3 2026	July	Jul 15-31	Ramon
Q4 2026	October	Oct 15-31	Ramon

Next drill: Q3 2026 (July 15-31)

Drill Steps

Phase 1: Pre-Drill Verification (5 min)

1. Check production health:

   ssh pmx 'pct status 280'
   ssh pmx 'pct exec 280 -- systemctl status remote-pulse'
   

   Expected: status: running, systemd active (running)

2. Verify latest PBS snapshot exists:

   ssh pmx 'proxmox-backup-manager snapshot list --repository 192.168.0.186:main | grep "ct/280"'
   

   Expected: snapshot from last 24h

3. Verify latest pg_dump exists:

   ssh pmx 'pct exec 280 -- ls -lth /var/lib/rp/backups/ | head -5'
   

   Expected: dump file from today (03:00 cron)

4. Record pre-drill metrics:

5. Production uptime: pct exec 280 -- uptime
6. Agent count: pct exec 280 -- rp admin hosts | wc -l
7. Last heartbeat timestamps (via Grafana dashboard)

Phase 2: Simulated Disaster (15 min)

CAUTION: This intentionally stops production LXC 280. Schedule drill during low-usage window (e.g., weekend morning).

1. Announce drill:

   # Send Telegram notification
   curl -X POST "https://api.telegram.org/bot<TOKEN>/sendMessage" \
       -d chat_id=730947207 \
       -d text="🚨 DR Drill started: Remote-Pulse LXC 280 will be stopped for ~15min. Normal ops, no action needed."
   

2. Stop production LXC:

   ssh pmx 'pct stop 280'
   ssh pmx 'pct status 280'
   

   Expected: status: stopped

3. Record disaster timestamp:

   DISASTER_TS=$(date -Iseconds)
   echo "Disaster timestamp: $DISASTER_TS"
   

Phase 3: Recovery — Option R1 (PBS Restore) (10 min)

Preferred method: Fast restore from PBS snapshot.

1. List available PBS snapshots:

   ssh pmx 'proxmox-backup-manager snapshot list --repository 192.168.0.186:main | grep "ct/280" | tail -5'
   

2. Create test LXC 283 from latest snapshot:

   # Find latest backup ID
   BACKUP_ID=$(ssh pmx 'proxmox-backup-manager snapshot list --repository 192.168.0.186:main | grep "ct/280" | tail -1 | awk "{print \$1}"')
   
   # Restore to new LXC 283
   ssh pmx "pct restore 283 $BACKUP_ID --storage zfs-ha --hostname rp-server-test"
   

3. Start test LXC:

   ssh pmx 'pct start 283'
   ssh pmx 'pct exec 283 -- systemctl is-active remote-pulse'
   

   Expected: active

4. Verify Postgres + TimescaleDB:

   ssh pmx 'pct exec 283 -- sudo -u postgres psql -d remote_pulse -c "\dt"'
   ssh pmx 'pct exec 283 -- sudo -u postgres psql -d remote_pulse -c "SELECT * FROM hosts LIMIT 5;"'
   

   Expected: tables visible, data intact

5. Smoke test — heartbeat from external agent:

   # From a test agent host (not in production fleet)
   curl -k https://100.x.y.z:8080/v1/health
   

   Expected: {"status":"ok"}

6. Verify audit log preserved:

   ssh pmx 'pct exec 283 -- sudo -u postgres psql -d remote_pulse -c "SELECT COUNT(*) FROM commands;"'
   

   Expected: row count > 0 (if production had commands)

7. Verify SSH keys present:

   ssh pmx 'pct exec 283 -- ls -la /etc/rp/host_key'
   

   Expected: Ed25519 keypair exists

8. Verify TimescaleDB hypertables intact (if F3 deployed):

   ssh pmx 'pct exec 283 -- sudo -u postgres psql -d remote_pulse -c "\dx"'
   

   Expected: timescaledb extension listed

9. Record recovery timestamp:

   RECOVERY_TS=$(date -Iseconds)
   echo "Recovery timestamp: $RECOVERY_TS"
   

Phase 4: Recovery — Option R2 (pg_dump Restore) (30 min)

Alternative method: Rebuild from Ansible + restore pg_dump.

SKIP during drill if R1 succeeded. Document here for reference:

1. Provision new LXC 283 from Ansible:

   cd /path/to/homelab-infra/ansible
   ansible-playbook -i inventory.yml site.yml --tags remote_pulse_server --limit lxc-283-test
   

2. Restore latest pg_dump:

   LATEST_DUMP=$(ssh pmx 'ls -t /mnt/nas/backups/remote-pulse/*.dump | head -1')
   ssh pmx "pct push 283 $LATEST_DUMP /tmp/restore.dump"
   ssh pmx 'pct exec 283 -- sudo -u postgres pg_restore -d remote_pulse --clean /tmp/restore.dump'
   

3. Continue with verification steps 3-9 from R1

Phase 5: RTO/RPO Measurement (5 min)

1. Calculate RTO (Recovery Time Objective):

   # Time from disaster to service restored
   RTO_SECONDS=$(( $(date -d "$RECOVERY_TS" +%s) - $(date -d "$DISASTER_TS" +%s) ))
   RTO_MINUTES=$(( $RTO_SECONDS / 60 ))
   echo "RTO: $RTO_MINUTES minutes"
   

2. Calculate RPO (Recovery Point Objective):

   # Time from last backup to disaster
   LAST_BACKUP_TS=$(ssh pmx 'pct exec 283 -- stat -c %y /var/lib/rp/backups/remote-pulse-*.dump | sort | tail -1')
   RPO_SECONDS=$(( $(date -d "$DISASTER_TS" +%s) - $(date -d "$LAST_BACKUP_TS" +%s) ))
   RPO_MINUTES=$(( $RPO_SECONDS / 60 ))
   echo "RPO: $RPO_MINUTES minutes"
   

3. Expected targets:

4. RTO: < 10 minutes (R1 method)
5. RPO: < 60 minutes (daily backup at 03:00, disaster at any time = max 24h, typical < 1h)

Phase 6: Rollback to Production (5 min)

1. Stop test LXC 283:

   ssh pmx 'pct stop 283'
   ssh pmx 'pct destroy 283'
   

2. Restart production LXC 280:

   ssh pmx 'pct start 280'
   ssh pmx 'pct exec 280 -- systemctl status remote-pulse'
   

3. Verify fleet reconnects:

   ssh pmx 'pct exec 280 -- rp admin hosts'
   

   Expected: all agents show last_seen < 2 minutes

4. Announce drill complete:

   curl -X POST "https://api.telegram.org/bot<TOKEN>/sendMessage" \
       -d chat_id=730947207 \
       -d text="✅ DR Drill complete: Remote-Pulse restored. RTO: ${RTO_MINUTES}m, RPO: ${RPO_MINUTES}m. Production resumed."
   

Phase 7: Post-Drill Report (10 min)

1. Record results in rp-dr-drill-log.md:

2. Date, drill type (R1 or R2), RTO, RPO, outcome, notes

3. Update Grafana annotation:

   curl -X POST http://192.168.0.208:3030/api/annotations \
       -H "Authorization: Bearer <GRAFANA_TOKEN>" \
       -H "Content-Type: application/json" \
       -d '{
         "dashboardUID": "homelab-overview",
         "time": '$(date +%s000)',
         "tags": ["dr-drill", "remote-pulse"],
         "text": "DR Drill: RTO='$RTO_MINUTES'm, RPO='$RPO_MINUTES'm, method=R1"
       }'
   

4. If RTO/RPO exceeded targets: open issue in homelab-infra repo with root cause analysis

Post-Drill Report Template

Copy to rp-dr-drill-log.md after each drill:

### Drill YYYY-MM-DD

**Date:** YYYY-MM-DD HH:MM  
**Operator:** Ramon  
**Method:** R1 (PBS) | R2 (Ansible+pg_dump) | R3 (scratch)  
**RTO:** XX minutes  
**RPO:** XX minutes  
**Target RTO:** 10 minutes  
**Target RPO:** 60 minutes  
**Outcome:** ✅ Success | ⚠️ Partial | ❌ Failure  

**Notes:**
- [Any issues encountered]
- [Actions taken]
- [Follow-up items]

**Fleet status post-recovery:**
- Total agents: XX
- Reconnected within 5min: XX/XX
- Manual intervention required: [list if any]

**Improvements identified:**
- [Optimization 1]
- [Optimization 2]

Common Failures + Fixes

Failure: PBS snapshot not found

Symptom: proxmox-backup-manager snapshot list returns empty or old snapshot
Root cause: PBS job disabled or datastore full
Fix: 1. Check PBS job exists: pvesh get /cluster/backup | jq '.[] | select(.vmid == "280")' 2. Check datastore space: ssh 192.168.0.186 'df -h /mnt/datastore/main' 3. Manual trigger: vzdump 280 --storage pbs --mode snapshot

Failure: pg_restore fails with schema mismatch

Symptom: ERROR: relation "hosts" already exists
Root cause: Dump contains CREATE but DB already has schema
Fix: Use pg_restore --clean (already in script) or drop DB first:

pct exec 283 -- sudo -u postgres psql -c "DROP DATABASE remote_pulse;"
pct exec 283 -- sudo -u postgres psql -c "CREATE DATABASE remote_pulse;"

Failure: Agents don't reconnect after restore

Symptom: rp admin hosts shows all last_seen > 10 minutes
Root cause: Tailscale identity mismatch or server Ed25519 key rotated
Fix: 1. Check Tailscale status: pct exec 283 -- tailscale status 2. Re-join tailnet: pct exec 283 -- tailscale up --authkey=<NEW_KEY> 3. If key rotated, invoke /v1/agent/reauth endpoint on all agents (see ADR-0008 §F.1 step 3)

Failure: TimescaleDB extension not loaded

Symptom: psql -c "\dx" doesn't show timescaledb
Root cause: Extension needs manual CREATE EXTENSION after restore
Fix:

pct exec 283 -- sudo -u postgres psql -d remote_pulse -c "CREATE EXTENSION IF NOT EXISTS timescaledb;"

Failure: Test LXC 283 conflicts with production 280 (port collision)

Symptom: LXC 283 fails to start, port 8080 already bound
Root cause: Both LXCs use same host network config
Fix: Use different port for test:

pct set 283 -net0 name=eth0,bridge=vmbr0,ip=dhcp,firewall=1,tag=999

Or stop 280 during drill (already in procedure)

Automated Script Reference

Location: /Users/ramonkamibayashicarrera/homelab-infra/scripts/rp-dr-drill.sh

Usage:

# Dry-run (default, no changes)
./rp-dr-drill.sh --dry-run

# Execute drill
./rp-dr-drill.sh --execute

# Skip phases (e.g., if production already stopped)
./rp-dr-drill.sh --execute --skip-stop

Logs: /var/log/rp-dr-drill/<timestamp>.log

Related Documentation

• ADR-0008 Appendix F: DR procedure (disaster scenarios F1-F3)
• PBS restore drill: docs/runbooks/pbs-restore-drill.md
• Backup script: ansible/roles/remote_pulse_server/templates/pg_dump-backup.sh.j2
• Healthchecks dashboard: http://192.168.0.208:8073

---

Last updated: 2026-05-25
Next review: After Q2 2026 drill (April)