site-migration-toolkit/docs/BackUP/postgresql_disaster_recovery.md

统一数据库备份与灾备方案 (PostgreSQL) | Unified DB Backup & DR Plan

🇨🇳 中文版在下方 | Chinese version below

---

English Version

This document is based on the actual deployment of the install.svc.plus production environment. It outlines the standard operating procedures for the global PostgreSQL database backup, restoration, and Disaster Recovery (DR).

1. Production Database Architecture Checklist

Based on the production environment reconnaissance, there are currently three independently running PostgreSQL instances. When performing unified backups, data must be exported from these three instances separately.

Instance Purpose	Runtime Mode	Listen Port/Address	Contained Business Databases
AI Core Services DB	Docker (postgresql-svc-plus)	127.0.0.1:5432	Account, Vault Storage, Artifact, LiteLLM, OpenClaw, QMD, RAG, Notification, Scheduler, Audit
Zitadel IAM DB	Docker (zitadel-db-1)	Container Internal Net	Zitadel IAM Data
Gitea Code DB	Native Systemd (PG 16)	127.0.0.1:5434	Gitea

2. Unified Backup Strategy

To achieve seamless disaster recovery, we recommend a strategy combining Logical Backup with Offsite Sync.

2.1 Global Backup Script

Create a unified backup script pg_backup.sh in the /opt/backup/ directory of the host (install.svc.plus):

#!/bin/bash
# ==========================================
# Unified PostgreSQL Backup Script
# ==========================================
BACKUP_DIR="/var/backups/postgresql"
DATE=$(date +"%Y%m%d_%H%M%S")
mkdir -p $BACKUP_DIR

echo "[INFO] Backing up AI Core Services databases (Port 15432)..."
DB_URL_BASE="postgres://svcplus_vps:<YOUR_PASSWORD>@127.0.0.1:15432"
for DB in account litellm openclaw qmd rag notification scheduler audit artifact vault_storage; do
    pg_dump "$DB_URL_BASE/$DB?sslmode=disable" | gzip > "$BACKUP_DIR/${DB}_$DATE.sql.gz"
done

echo "[INFO] Backing up Zitadel IAM DB (zitadel-db-1)..."
docker exec zitadel-db-1 pg_dumpall -U postgres | gzip > "$BACKUP_DIR/zitadel_$DATE.sql.gz"

echo "[INFO] Backing up Gitea Code DB (Native PG 16)..."
sudo -u postgres pg_dumpall -p 5434 | gzip > "$BACKUP_DIR/gitea_$DATE.sql.gz"

echo "[INFO] Cleaning up backups older than 7 days..."
find $BACKUP_DIR -type f -name "*.sql.gz" -mtime +7 -delete

echo "[INFO] Backup complete!"

Tip

It is highly recommended to push backup files to an offsite S3 storage (like MinIO) or sync via rclone to other cloud storage to prevent datacenter-level disasters.

2.2 Cron Schedule

Add an automatic backup job at 3:00 AM daily via crontab -e:

0 3 * * * /bin/bash /opt/backup/pg_backup.sh >> /var/log/pg_backup.log 2>&1

3. Disaster Recovery & Restore Operations

When data corruption or full-machine migration occurs, please strictly follow this restore sequence.

Warning

Before executing a restore, you must first stop the business services generating write traffic (like Gitea, LiteLLM, etc.), keeping ONLY the PostgreSQL processes running.

3.1 Restore AI Core Services DB (Port 15432)

Includes: Account, Vault Storage, Artifact, LiteLLM, OpenClaw, QMD, RAG, Notification, Scheduler, Audit

# Unzip backup and import directly via credential URI (Using account as an example)
gunzip -c /var/backups/postgresql/account_YYYYMMDD.sql.gz | psql "postgres://svcplus_vps:<YOUR_PASSWORD>@127.0.0.1:15432/account?sslmode=disable"

3.2 Restore Zitadel IAM DB (Docker)

# Unzip backup and import
gunzip -c /var/backups/postgresql/zitadel_YYYYMMDD.sql.gz | docker exec -i zitadel-db-1 psql -U postgres

3.3 Restore Gitea Code DB (Native)

Since Gitea is on the physical host, we need to switch to the postgres user:

# Unzip backup and import (Note port 5434)
gunzip -c /var/backups/postgresql/gitea_YYYYMMDD.sql.gz | sudo -u postgres psql -p 5434

4. High Availability & Advanced Recommendations

For the current single-node multi-instance architecture, logical backups (pg_dumpall) are sufficient for most scenarios, but the RPO (Recovery Point Objective) is approximately 24 hours (depending on backup frequency).

Important

Recommendations for upgrading DR tier:

1. Incremental Backups (WAL Archiving): For mission-critical databases like AI Core Services and Gitea, introduce pgBackRest or WAL-G to continuously archive physical WAL logs to S3 (MinIO) in real-time. This reduces the risk of data loss (RPO) to minutes.
2. Application-Level DR:
   • Vault Storage decryption keys cannot solely exist in the database. Ensure Vault's physical Root Token / Unseal Keys are safely stored offline.
   • For Gitea, in addition to backing up the database, you must periodically back up /var/lib/gitea/data (bare Git repositories and LFS artifacts) on the host. Using Gitea's built-in gitea dump command is highly recommended to achieve an integrated backup of both data and repositories.

---

中文版 (Chinese Version)

本文档基于 install.svc.plus 线上环境的实际部署情况，为你规划和整理了全局 PostgreSQL 数据库的备份、还原与灾备（DR）标准操作流程。

1. 线上环境数据库架构清单

根据线上环境探勘，当前系统存在 三个独立运行的 PostgreSQL 实例。在进行统一备份时，必须分别对这三个实例进行数据导出。

实例用途	运行模式	监听端口/地址	包含的业务系统库
AI 核心服务库	Docker (postgresql-svc-plus)	127.0.0.1:5432	Account, Vault Storage, Artifact, LiteLLM, OpenClaw, QMD, RAG, Notification, Scheduler, Audit
Zitadel 认证库	Docker (zitadel-db-1)	容器内部网	Zitadel IAM 数据
Gitea 代码库	Native Systemd (PG 16)	127.0.0.1:5434	Gitea

2. 统一备份策略 (Backup)

为了实现无缝灾备，建议采用 逻辑备份 (Logical Backup) 结合 定时异地转储 (Offsite Sync) 的策略。

2.1 编写全局备份脚本

在宿主机 (install.svc.plus) 的 /opt/backup/ 目录下创建统一备份脚本 pg_backup.sh：

#!/bin/bash
# ==========================================
# 统一 PostgreSQL 备份脚本
# ==========================================
BACKUP_DIR="/var/backups/postgresql"
DATE=$(date +"%Y%m%d_%H%M%S")
mkdir -p $BACKUP_DIR

echo "[INFO] 开始备份 AI 核心服务各业务库 (15432 端口)..."
DB_URL_BASE="postgres://svcplus_vps:<YOUR_PASSWORD>@127.0.0.1:15432"
for DB in account litellm openclaw qmd rag notification scheduler audit artifact vault_storage; do
    pg_dump "$DB_URL_BASE/$DB?sslmode=disable" | gzip > "$BACKUP_DIR/${DB}_$DATE.sql.gz"
done

echo "[INFO] 开始备份 Zitadel 认证库 (zitadel-db-1)..."
docker exec zitadel-db-1 pg_dumpall -U postgres | gzip > "$BACKUP_DIR/zitadel_$DATE.sql.gz"

echo "[INFO] 开始备份 Gitea 代码库 (Native PG 16)..."
sudo -u postgres pg_dumpall -p 5434 | gzip > "$BACKUP_DIR/gitea_$DATE.sql.gz"

echo "[INFO] 清理 7 天前的旧备份..."
find $BACKUP_DIR -type f -name "*.sql.gz" -mtime +7 -delete

echo "[INFO] 备份完成！"

Tip

建议将备份文件推送到异地 S3 (例如 MinIO) 或通过 rclone 同步到其他云存储，以防范机房级灾难。

2.2 配置定时任务 (Cron)

通过 crontab -e 添加每天凌晨 3 点自动备份：

0 3 * * * /bin/bash /opt/backup/pg_backup.sh >> /var/log/pg_backup.log 2>&1

3. 灾难恢复与还原 (Restore)

当发生数据损坏或整机迁移时，请严格按照以下顺序进行恢复。

Warning

在执行还原前，必须先停止产生写入流量的业务服务（如 Gitea, LiteLLM 等），仅保留 PostgreSQL 进程运行。

3.1 还原 AI 核心服务库 (15432 端口)

包含：Account, Vault Storage, Artifact, LiteLLM, OpenClaw, QMD, RAG, Notification, Scheduler, Audit

# 解压备份文件并使用凭证直接导入 (以 account 为例)
gunzip -c /var/backups/postgresql/account_YYYYMMDD.sql.gz | psql "postgres://svcplus_vps:<YOUR_PASSWORD>@127.0.0.1:15432/account?sslmode=disable"

3.2 还原 Zitadel 认证库 (Docker)

# 解压备份文件并导入
gunzip -c /var/backups/postgresql/zitadel_YYYYMMDD.sql.gz | docker exec -i zitadel-db-1 psql -U postgres

3.3 还原 Gitea 代码库 (Native)

由于 Gitea 在宿主机物理机上，我们需要切换到 postgres 用户执行：

# 解压备份文件并导入 (注意端口 5434)
gunzip -c /var/backups/postgresql/gitea_YYYYMMDD.sql.gz | sudo -u postgres psql -p 5434

4. 高可用与进阶建议 (Disaster Recovery)

对于目前的单机多实例架构，逻辑备份（pg_dumpall）足够应付大多数场景，但存在 RPO (恢复点目标) 约为 24 小时的问题（取决于备份频率）。

Important

提升容灾级别的建议：

1. 增量备份 (WAL Archiving): 针对业务极其核心的 AI 核心服务库和 Gitea，可引入 pgBackRest 或 WAL-G，将物理 WAL 日志实时归档至 S3（MinIO）。这样可以将数据丢失风险（RPO）降低到分钟级别。
2. 应用级灾备:
   • Vault Storage 的加解密密钥不能仅存在数据库中，请确保 Vault 的物理 Root Token/Unseal Keys 已被安全离线保存。
   • Gitea 除了备份数据库，还必须定期备份宿主机上的 /var/lib/gitea/data (Git 仓库裸数据和 LFS 制品)。可以借助 Gitea 自带的 gitea dump 命令来实现数据 + 仓库的一体化备份。