Disaster Recovery

What this covers

This page describes AuraBoot's disaster recovery strategy: what to back up, how to test that restores work, and the exact steps to execute when things go wrong. Two terms anchor everything here. RTO (Recovery Time Objective) is the maximum elapsed time from incident declaration to service restoration. RPO (Recovery Point Objective) is the maximum amount of data you can afford to lose, measured in time. Every decision in this document — backup frequency, replication topology, drill cadence — flows from the tier-specific RTO/RPO targets below.

RTO and RPO tiers

The HA cluster tier matches the platform's default production topology. The MTTR target for P0 incidents (full service unavailable) at that tier is < 15 minutes. Single-node deployments trade RTO/RPO for simplicity; that trade is only acceptable for development and staging environments.

The four assets to protect

PostgreSQL data is the authoritative store for all application state: tenants, users, metadata models, page schemas, command definitions, plugin configurations, and every dynamic business table (mt_*). It must be backed up with both logical dumps (pg_dump) for portability and physical base backups (pg_basebackup) for fast PITR. Nothing else replaces a missing or corrupted PostgreSQL backup.

Object storage (MinIO / S3-compatible) holds uploaded files, attachments, report exports, and crawler artifacts. The object store is write-once from the application's perspective — once a file is stored, the application only reads it back by key. Back it up with bucket replication or periodic sync to a separate storage endpoint. Do not depend on the application database to reconstruct which files exist; treat the object store as an independent durable asset.

Redis state holds distributed locks, session tokens, rate-limit buckets, and the L2 metadata cache. Redis data is intentionally ephemeral: sessions expire, locks release, caches warm from PostgreSQL. A cold restart with an empty Redis causes a latency spike during warmup (~1.5 s for a typical metadata set) but no data loss. The Sentinel or cluster topology covers Redis availability; there is no need to take Redis snapshots for DR purposes.

Secrets and configuration — environment variables, application-prod.yml overrides, TLS certificates, and encryption keys — must live in a secrets vault or equivalent versioned store, not only on the server's filesystem. A restore that recovers PostgreSQL but loses the encryption key for ab_cloud_config is not a successful restore. Treat secrets as a separate backup axis with its own rotation and recovery procedure.

PostgreSQL backup strategy

AuraBoot uses two complementary backup methods. Daily logical backups (pg_dump) are fast and portable. Weekly physical base backups (pg_basebackup) combined with continuous WAL archiving enable point-in-time recovery (PITR) to any second in the retention window.

WAL archiving — enable this first:

# postgresql.conf
wal_level = replica
archive_mode = on
archive_command = 'test ! -f /var/wal-archive/%f && cp %p /var/wal-archive/%f'
wal_keep_size = 1GB
checkpoint_completion_target = 0.9
checkpoint_timeout = 5min
max_wal_size = 1GB

Verify the archiver is healthy before relying on PITR:

psql -U auraboot -d aura_boot -c "SELECT * FROM pg_stat_archiver;"
# failed_count must be 0; last_archived_time must be recent

Weekly physical backup:

PGPASSWORD="$REPLICATION_PASSWORD" pg_basebackup \
    -h "$DB_HOST" \
    -U replicator \
    --pgdata=/var/backups/auraboot/weekly/basebackup-$(date +%Y%m%d) \
    --format=tar \
    --gzip \
    --compress=9 \
    --wal-method=stream \
    --checkpoint=fast \
    --progress

Daily logical backup (portable, verifiable):

pg_dump \
    -h "$DB_HOST" -U auraboot -d aura_boot \
    --format=custom --compress=9 \
    --file=/var/backups/auraboot/daily/aura_boot-$(date +%Y%m%d-%H%M%S).dump

# Integrity check — list contents without restoring
pg_restore --list /var/backups/auraboot/daily/aura_boot-*.dump > /dev/null

Retention policy:

PITR restore to a target timestamp:

# 1. Stop the application
systemctl stop auraboot && systemctl stop postgresql

# 2. Extract the most recent base backup before the target time
rm -rf /var/lib/postgresql/data/*
tar -xzf /var/backups/auraboot/weekly/basebackup-20260410/base.tar.gz \
    -C /var/lib/postgresql/data/
chown -R postgres:postgres /var/lib/postgresql/data
chmod 700 /var/lib/postgresql/data

# 3. Configure PITR (PostgreSQL 12+)
cat >> /var/lib/postgresql/data/postgresql.conf <<EOF
restore_command = 'cp /var/wal-archive/%f %p'
recovery_target_time = '2026-04-11 14:30:00'
recovery_target_action = 'promote'
EOF
touch /var/lib/postgresql/data/recovery.signal

# 4. Start and watch replay
systemctl start postgresql
tail -f /var/log/postgresql/postgresql-$(date +%Y-%m-%d).log

Object storage and file backup

Enable bucket replication between your primary object storage endpoint and a geographically separate secondary. At minimum, run a nightly sync:

# Generic S3-compatible sync (adapt to your tool of choice)
mc mirror --overwrite --remove primary/auraboot-files secondary/auraboot-files-backup

What you can safely reconstruct from PostgreSQL if the object store is lost: file metadata records in ab_file. What you cannot reconstruct: the actual file bytes. That asymmetry means losing the object store without a backup is permanent data loss regardless of how healthy PostgreSQL is.

Do not depend on local-disk MinIO for production durability. Local MinIO is appropriate for development and single-node staging only. For production, configure MinIO in distributed mode or point the platform at an external S3-compatible service with its own replication enabled.

Cross-region failover sequence

Execute these steps in order. Do not skip the replication lag check — promoting a standby that is minutes behind means accepting that RPO gap.

1. Detect and classify. Confirm the primary is unreachable: pg_isready -h <primary-ip> -p 5432 returns no response. Check pg_stat_replication on the standby to assess lag before promoting.

2. Check replication lag:

   psql -h <standby-ip> -U auraboot -d aura_boot -c \
     "SELECT EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp()))::INT AS lag_seconds;"

   Copy

3. Promote the standby:

   psql -h <standby-ip> -U auraboot -c "SELECT pg_promote();"
   # Verify: pg_is_in_recovery() must return false
   psql -h <standby-ip> -U auraboot -d aura_boot -c "SELECT pg_is_in_recovery();"

   Copy

4. Flip DNS or load-balancer upstream to point db.internal at the promoted standby's IP. If using a connection string env var directly, update DATASOURCE_URL on every app node.

5. Restart application workers and verify health:

   systemctl restart auraboot
   curl -s http://localhost:6443/api/health | python3 -m json.tool

   Copy

6. Validate data integrity (see Post-restore data validation below).

7. Notify and log. Record the actual failover time and replication lag at promotion. Both feed the post-incident RTO/RPO measurement.

Restore drill checklist

Run a full restore drill quarterly. An untested restore is not a backup. Each drill covers:

• PITR restore to a timestamp 2 hours before the drill window, on an isolated host
• Verify recovery.signal is consumed and PostgreSQL exits recovery mode
• Full logical restore from the most recent pg_dump into a aura_boot_drill_$(date +%Y%m%d) database
• Plugin reimport: import at least one production plugin package and confirm success: true
• Metadata cache warmup: start the application against the restored DB, confirm CacheWarmupRunner completes without errors
• Redis cold-start: flush Redis, restart the application, measure latency on the first 10 requests (warmup overhead should not exceed 3 s)
• Smoke commands: curl /api/health, fetch a tenant list, execute one dynamic list query
• Measure wall-clock time from "start restore" to "smoke commands pass" — this is your measured RTO; compare against your tier target
• Document results in the drill report template; file any gap as a follow-up action with an owner and deadline

Post-restore data validation

Run these checks before declaring the restore complete and before restarting production traffic.

-- Core row counts — compare against expected baseline
SELECT 'ab_user'        AS tbl, COUNT(*) AS rows FROM ab_user
UNION ALL
SELECT 'ab_tenant',              COUNT(*)         FROM ab_tenant
UNION ALL
SELECT 'ab_meta_model',          COUNT(*)         FROM ab_meta_model
UNION ALL
SELECT 'ab_page_schema',         COUNT(*)         FROM ab_page_schema
UNION ALL
SELECT 'ab_command_definition',  COUNT(*)         FROM ab_command_definition
ORDER BY tbl;

-- Dynamic business tables present (each plugin publishes mt_* tables)
SELECT COUNT(*) AS dynamic_table_count
FROM information_schema.tables
WHERE table_name LIKE 'mt_%' AND table_schema = 'public';

-- Audit log continuity — no gap > 2x the expected write interval
SELECT MAX(changed_at) AS last_audit_entry,
       NOW() - MAX(changed_at) AS staleness
FROM ab_data_change_log;

-- Active tenants can be resolved
SELECT name, status FROM ab_tenant WHERE status = 'active' LIMIT 5;

Key invariants to assert:

• ab_tenant has at least one active row.
• ab_meta_model count matches pre-restore count (plugin models are not re-created on restore; they are recovered from the dump).
• ab_data_change_log latest timestamp is within the expected RPO window of the restore point.
• No mt_* table referenced by an active plugin is missing from information_schema.tables.

If any count is zero where it should not be, stop and investigate before routing traffic. A partial restore is worse than a clean failure.

Common pitfalls

Next steps

• Backup and restore — detailed backup scripts, cron configuration, and restore runbooks
• Observability — metrics and alerts for backup success rate, replication lag, and WAL archive health
• Performance tuning — RTO is affected by how fast PostgreSQL replays WAL; tune before you need it
• Load balancing — DNS and upstream flip patterns during failover

Enterprise note — Managed cross-region failover with automatic DNS promotion, signed backup artifact provenance (SHA-256 manifest with tamper-evident audit trail), and live RTO/RPO SLA dashboards backed by real restore-drill telemetry are available in the Enterprise tier. OSS deployments have full access to every backup mechanism and runbook described above; the difference is the managed orchestration layer and the compliance-grade artifact signing, not the underlying PostgreSQL and WAL tooling.