From 0a2362629886aace24e6ad8ef02c82dd25304f9b Mon Sep 17 00:00:00 2001 From: Haitao Pan Date: Wed, 1 Jul 2026 10:05:45 +0800 Subject: [PATCH] docs: restructure docs into ZH and EN directories and add unified index to README --- README.md | 16 ++ docs/BackUP/backup_dr_plan.md | 215 ------------------ docs/BackUP/postgresql_disaster_recovery.md | 201 ---------------- docs/EN/BackUP/backup_dr_plan.md | 12 + .../EN/BackUP/postgresql_disaster_recovery.md | 10 + .../PROD/live_systems_overview.md | 8 + .../PROD/live_systems_overview.md | 141 ------------ .../Site-Migration/implementation_plan.md | 0 .../BackUP/Site-Migration/walkthrough.md | 0 docs/ZH/BackUP/backup_dr_plan.md | 103 +++++++++ .../ZH/BackUP/postgresql_disaster_recovery.md | 96 ++++++++ .../PROD/live_systems_overview.md | 66 ++++++ 12 files changed, 311 insertions(+), 557 deletions(-) delete mode 100644 docs/BackUP/backup_dr_plan.md delete mode 100644 docs/BackUP/postgresql_disaster_recovery.md create mode 100644 docs/EN/BackUP/backup_dr_plan.md create mode 100644 docs/EN/BackUP/postgresql_disaster_recovery.md create mode 100644 docs/EN/Systems-Overview/PROD/live_systems_overview.md delete mode 100644 docs/Systems-Overview/PROD/live_systems_overview.md rename docs/{ => ZH}/BackUP/Site-Migration/implementation_plan.md (100%) rename docs/{ => ZH}/BackUP/Site-Migration/walkthrough.md (100%) create mode 100644 docs/ZH/BackUP/backup_dr_plan.md create mode 100644 docs/ZH/BackUP/postgresql_disaster_recovery.md create mode 100644 docs/ZH/Systems-Overview/PROD/live_systems_overview.md diff --git a/README.md b/README.md index ecd20f4..a7cf5be 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,13 @@ Welcome to the **Site Migration & Backup Toolkit**. This repository provides the orchestrations, runbooks, and automated playbooks for managing the disaster recovery lifecycle of the AI Workspace infrastructure. +## Documentation Index + +- [Live Systems Overview](docs/EN/Systems-Overview/PROD/live_systems_overview.md) +- [Full-Site Unified Backup & DR Plan](docs/EN/BackUP/backup_dr_plan.md) +- [Unified DB Backup & DR Plan (PostgreSQL)](docs/EN/BackUP/postgresql_disaster_recovery.md) + + ## Phase Roadmap This project is iteratively rolling out disaster recovery capabilities. Currently, we are heavily focused on **Phase 1**. @@ -116,6 +123,15 @@ EOF 欢迎使用 **Site Migration & Backup Toolkit**。本代码库提供了管理 AI Workspace 基础架构灾难恢复生命周期的编排、运维手册和自动化 Playbooks。 +## 文档索引 (Documentation Index) + +- [线上系统概览 (Live Systems Overview)](docs/ZH/Systems-Overview/PROD/live_systems_overview.md) +- [全站统一备份与灾备方案 (Full-Site Unified Backup & DR Plan)](docs/ZH/BackUP/backup_dr_plan.md) +- [统一数据库备份与灾备方案 (PostgreSQL)](docs/ZH/BackUP/postgresql_disaster_recovery.md) +- [站点迁移与数据同步实施总结 (Walkthrough)](docs/ZH/BackUP/Site-Migration/walkthrough.md) +- [全站单向数据流迁移与环境复制方案 (Implementation Plan)](docs/ZH/BackUP/Site-Migration/implementation_plan.md) + + ## 阶段演进路线图 (Phase Roadmap) 本项目正在迭代推出灾备能力。目前，我们正重点聚焦于 **Phase 1 (第一阶段)**。 diff --git a/docs/BackUP/backup_dr_plan.md b/docs/BackUP/backup_dr_plan.md deleted file mode 100644 index 66c1571..0000000 --- a/docs/BackUP/backup_dr_plan.md +++ /dev/null @@ -1,215 +0,0 @@ -# 全站统一备份与灾备方案 (AI Workspace) | Full-Site Unified Backup & DR Plan - -*🇨🇳 中文版在下方 | Chinese version below* - ---- - -# English Version - -This document is based on the actual deployment of the `install.svc.plus` (or `root@install.svc.plus`) production environment. It outlines the global Disaster Recovery (DR) standard operating procedures. In addition to the core PostgreSQL databases, this plan covers all core stateful components required for AI Workspace to run (including application attachments, private images, VPN configurations, etc.). - -## 1. Core Stateful Components & Directory Checklist - -When performing cross-datacenter/cross-zone full-machine migrations, the integrity of the following data must be ensured. - -### 1.1 Database Assets - -| Instance Purpose | Runtime Mode | Listen Port/Address | Contained Business Databases | -| :--- | :--- | :--- | :--- | -| AI Core Services DB | Docker (`postgresql-svc-plus`) | `127.0.0.1:15432` | Account, Vault Storage, Artifact, LiteLLM, OpenClaw, QMD, RAG, Notification, Scheduler, Audit | -| Zitadel IAM DB | Docker (`zitadel-db-1`) | Container Internal Net | Zitadel IAM Data | -| Gitea Code DB | Native Systemd (PG 16) | `127.0.0.1:5434` | Gitea Metadata | - -### 1.2 Critical Files & Stateful Volumes - -> [!IMPORTANT] -> The following directories are the "flesh and blood" of the restored system. If not migrated, even if the database is restored, fatal errors will occur (e.g., 404 images, failed model pulls, disconnected networking). - -* **Gitea Repository & Object Data:** `/var/lib/gitea/data` - Stores all bare Git repositories, LFS large files, user avatars, and attachments. -* **Private Image Registry (Harbor / Registry):** `/data/registry` or mounted `/data/harbor` - Private container images. Without this, the new machine will fail to find closed-source base images when running `docker-compose pull` upon restart. -* **OpenClaw (AI Gateway) Workdir:** `openclaw workdir` (Corresponds to the physical mount point during deployment) - Contains the AI Gateway's local cache files, vectorized temporary data, or loaded model assets. -* **Distributed Networking & VPN Config:** `/etc/wireguard` - Contains the local node's WireGuard Private Key and Peers routing table. If lost, the new host will be unable to rejoin the original Xworkmate VPN star network, breaking all cross-cloud microservice communication. -* **Object Storage (MinIO/S3):** `/data/minio` (if enabled) - The large file storage foundation for modern cloud-native components (like Wiki, RAG). -* **Environment Secret Files:** `.env` files under each component's deployment directory (e.g., `/opt/ai-workspace/`). - -> [!NOTE] -> **Exception List:** `/etc/caddy/data` or `acme.json` (Let's Encrypt certificates). -> **Strategy:** No need to sync during DR drills or actual migrations. Simply point the DNS A record to the new IP before bringing the new host online, then start Caddy to trigger automatic online renewal. (Note Let's Encrypt's rate limit of 5 duplicate certificates per week; staging certificates are recommended for drills). - -## 2. Unified Backup & Sync Strategy - -### 2.1 Logical Database Backup (PostgreSQL) - -It is recommended to create a unified backup script `pg_backup.sh` in the host's `/opt/backup/` directory and add it to Cron: - -```bash -#!/bin/bash -BACKUP_DIR="/var/backups/postgresql" -DATE=$(date +"%Y%m%d_%H%M%S") -mkdir -p $BACKUP_DIR - -echo "[INFO] Backing up AI Core Services DB (Port 15432)..." -DB_URL_BASE="postgres://svcplus_vps:@127.0.0.1:15432" -for DB in account litellm openclaw qmd rag notification scheduler audit artifact vault_storage; do - pg_dump "$DB_URL_BASE/$DB?sslmode=disable" | gzip > "$BACKUP_DIR/${DB}_$DATE.sql.gz" -done - -echo "[INFO] Backing up Zitadel IAM DB (zitadel-db-1)..." -docker exec zitadel-db-1 pg_dumpall -U postgres | gzip > "$BACKUP_DIR/zitadel_$DATE.sql.gz" - -echo "[INFO] Backing up Gitea Code DB (Native PG 16)..." -sudo -u postgres pg_dumpall -p 5434 | gzip > "$BACKUP_DIR/gitea_$DATE.sql.gz" - -# Clean up backups older than 7 days -find $BACKUP_DIR -type f -name "*.sql.gz" -mtime +7 -delete -``` - -### 2.2 File State Sync (Rsync Incremental) - -For file resources, we recommend pushing to the DR node using `rsync` via Ansible playbooks, or scheduling syncs to centralized object storage. Key `rsync` paths: - -```bash -rsync -avz --delete /var/lib/gitea/data/ backup-server:/backup/gitea/ -rsync -avz --delete /etc/wireguard/ backup-server:/backup/wireguard/ -rsync -avz --delete /data/registry/ backup-server:/backup/registry/ -``` - -## 3. Disaster Recovery & Restore Operations - -When a full machine crash or datacenter network outage requires activating the DR node, strictly restore in the following tiers: - -**Stage 1: Restore VPN & Network Tier** -Before installing any services, first restore the `/etc/wireguard` directory and start `wg-quick up wg0` to ensure the DR node re-acquires its original cluster internal IP. - -**Stage 2: Restore Object & File Persistence Tier** -Restore the backed-up Gitea, OpenClaw workdir, and image registry directories to their absolute paths on the host, and fix ownership permissions (e.g., `chown -R 1000:1000 /var/lib/gitea`). - -**Stage 3: Restore PostgreSQL Database Tier** -Ensure ONLY the database containers are started (keep application layer containers stopped). - -```bash -# 1. Restore AI Core DB -gunzip -c /var/backups/postgresql/account_YYYYMMDD.sql.gz | psql "postgres://...:15432/account" - -# 2. Restore Zitadel DB -gunzip -c /var/backups/postgresql/zitadel_YYYYMMDD.sql.gz | docker exec -i zitadel-db-1 psql -U postgres - -# 3. Restore Gitea DB -gunzip -c /var/backups/postgresql/gitea_YYYYMMDD.sql.gz | sudo -u postgres psql -p 5434 -``` - -**Stage 4: Start Ingress & Business Applications** -1. Confirm global DNS has been switched to the current DR node's public IP. -2. Start the Caddy service. The system will automatically initiate HTTP-01 challenges to apply for fresh certificates. -3. Bring everything fully online with `docker-compose up -d` to pull up all business line applications. - ---- - -# 中文版 (Chinese Version) - -本文档基于 `install.svc.plus` (或 `root@install.svc.plus`) 线上环境的实际部署情况，为你规划和整理了全局的灾备（DR）标准操作流程。 除了核心的 PostgreSQL 数据库外，本方案覆盖了 AI Workspace 运行所需的所有核心状态组件（含应用附件、私有镜像、VPN 配置等）。 - -## 1. 核心状态组件与目录清单 - -在进行跨机房/跨区整机迁移时，必须确保以下数据的完整迁移。 - -### 1.1 数据库资产 - -| 实例用途 | 运行模式 | 监听端口/地址 | 包含的业务系统库 | -| :--- | :--- | :--- | :--- | -| AI 核心服务库 | Docker (`postgresql-svc-plus`) | `127.0.0.1:15432` | Account, Vault Storage, Artifact, LiteLLM, OpenClaw, QMD, RAG, Notification, Scheduler, Audit | -| Zitadel 认证库 | Docker (`zitadel-db-1`) | 容器内部网 | Zitadel IAM 数据 | -| Gitea 代码库 | Native Systemd (PG 16) | `127.0.0.1:5434` | Gitea 元数据 | - -### 1.2 关键文件与持久化挂载卷 (Stateful Volumes) - -> [!IMPORTANT] -> 以下目录是系统恢复后的“血肉”，如果不迁移，即使数据库还原，也会出现图片 404、模型拉取失败、组网断开等致命故障。 - -* **Gitea 仓库与对象数据:** `/var/lib/gitea/data` - 存储所有的 Git 裸仓库、LFS 大文件、用户头像及附件。 -* **应用私有镜像库 (Harbor / Registry):** `/data/registry` 或挂载的 `/data/harbor` - 私有容器镜像。如果不迁移，新机在重启并执行 `docker-compose pull` 时将找不到未开源的基础环境镜像。 -* **OpenClaw (AI 网关) 工作目录:** `openclaw workdir` (对应部署时的物理挂载点) - 包含 AI Gateway 的本地缓存档、向量化临时数据或加载的模型资产。 -* **分布式组网与 VPN 配置:** `/etc/wireguard` - 包含本地节点的 WireGuard Private Key 与 Peers 路由表。若丢失此目录，新主机启动后将无法重新融入原有的 Xworkmate VPN 星型网络，所有跨云微服务通信将中断。 -* **对象存储 (MinIO/S3):** `/data/minio` (若开启) - 现代云原生组件（如 Wiki、RAG）的大文件存储底座。 -* **环境密钥文件:** 各组件部署目录下的 `.env` (如 `/opt/ai-workspace/`)。 - -> [!NOTE] -> **例外清单：** `/etc/caddy/data` 或 `acme.json`（Let's Encrypt 证书）。 -> **处理策略：** 灾备演练或真实迁移时无需同步。只需在新主机上线时，提前将 DNS A 记录解析至新 IP，再启动 Caddy 即可实现全自动在线续签。（需注意 Let's Encrypt 每周 5 次的重复签发限频，演练时建议使用 Staging 证书）。 - -## 2. 统一备份与同步策略 (Backup & Sync) - -### 2.1 数据库逻辑备份 (PostgreSQL) - -建议在宿主机的 `/opt/backup/` 目录下创建统一备份脚本 `pg_backup.sh` 并加入 Cron 定时任务： - -```bash -#!/bin/bash -BACKUP_DIR="/var/backups/postgresql" -DATE=$(date +"%Y%m%d_%H%M%S") -mkdir -p $BACKUP_DIR - -echo "[INFO] 备份 AI 核心服务库 (15432 端口)..." -DB_URL_BASE="postgres://svcplus_vps:@127.0.0.1:15432" -for DB in account litellm openclaw qmd rag notification scheduler audit artifact vault_storage; do - pg_dump "$DB_URL_BASE/$DB?sslmode=disable" | gzip > "$BACKUP_DIR/${DB}_$DATE.sql.gz" -done - -echo "[INFO] 备份 Zitadel 认证库 (zitadel-db-1)..." -docker exec zitadel-db-1 pg_dumpall -U postgres | gzip > "$BACKUP_DIR/zitadel_$DATE.sql.gz" - -echo "[INFO] 备份 Gitea 代码库 (Native PG 16)..." -sudo -u postgres pg_dumpall -p 5434 | gzip > "$BACKUP_DIR/gitea_$DATE.sql.gz" - -# 清理 7 天前的旧备份 -find $BACKUP_DIR -type f -name "*.sql.gz" -mtime +7 -delete -``` - -### 2.2 文件状态同步 (Rsync 增量同步) - -针对文件资源，建议使用 `rsync` 结合 Ansible 剧本向灾备节点推送，或定时同步到集中式对象存储。 关键的 `rsync` 路径示例： - -```bash -rsync -avz --delete /var/lib/gitea/data/ backup-server:/backup/gitea/ -rsync -avz --delete /etc/wireguard/ backup-server:/backup/wireguard/ -rsync -avz --delete /data/registry/ backup-server:/backup/registry/ -``` - -## 3. 灾难恢复与还原操作 (Restore) - -当发生整机崩溃或机房断网需要启用灾备节点时，严格按以下层级恢复： - -**阶段 1: 恢复 VPN 与网络层** -在安装任何服务前，首先将 `/etc/wireguard` 目录还原，并启动 `wg-quick up wg0`，确保灾备节点重新获取原集群内网 IP。 - -**阶段 2: 恢复对象与文件持久层** -将备份好的 Gitea、OpenClaw workdir、应用镜像目录等还原至宿主机的绝对路径中，并修复属主权限（如 `chown -R 1000:1000 /var/lib/gitea`）。 - -**阶段 3: 恢复 PostgreSQL 数据库层** -确保仅启动数据库容器（停掉应用层容器）。 - -```bash -# 1. 还原 AI 核心库 -gunzip -c /var/backups/postgresql/account_YYYYMMDD.sql.gz | psql "postgres://...:15432/account" - -# 2. 还原 Zitadel 库 -gunzip -c /var/backups/postgresql/zitadel_YYYYMMDD.sql.gz | docker exec -i zitadel-db-1 psql -U postgres - -# 3. 还原 Gitea 库 -gunzip -c /var/backups/postgresql/gitea_YYYYMMDD.sql.gz | sudo -u postgres psql -p 5434 -``` - -**阶段 4: 启动接入层与业务应用** -1. 确认全网 DNS 已切换至当前灾备节点公网 IP。 -2. 启动 Caddy 服务，系统会自动发起 HTTP-01 质询申请全新证书。 -3. 全面上线 `docker-compose up -d` 拉起各业务线应用。 diff --git a/docs/BackUP/postgresql_disaster_recovery.md b/docs/BackUP/postgresql_disaster_recovery.md deleted file mode 100644 index 16e991a..0000000 --- a/docs/BackUP/postgresql_disaster_recovery.md +++ /dev/null @@ -1,201 +0,0 @@ -# 统一数据库备份与灾备方案 (PostgreSQL) | Unified DB Backup & DR Plan - -*🇨🇳 中文版在下方 | Chinese version below* - ---- - -# English Version - -This document is based on the actual deployment of the `install.svc.plus` production environment. It outlines the standard operating procedures for the global PostgreSQL database backup, restoration, and Disaster Recovery (DR). - -## 1. Production Database Architecture Checklist - -Based on the production environment reconnaissance, there are currently **three independently running PostgreSQL instances**. When performing unified backups, data must be exported from these three instances separately. - -| Instance Purpose | Runtime Mode | Listen Port/Address | Contained Business Databases | -| :--- | :--- | :--- | :--- | -| **AI Core Services DB** | Docker (`postgresql-svc-plus`) | `127.0.0.1:5432` | Account, Vault Storage, Artifact, LiteLLM, OpenClaw, QMD, RAG, Notification, Scheduler, Audit | -| **Zitadel IAM DB** | Docker (`zitadel-db-1`) | Container Internal Net | Zitadel IAM Data | -| **Gitea Code DB** | Native Systemd (PG 16) | `127.0.0.1:5434` | Gitea | - -## 2. Unified Backup Strategy - -To achieve seamless disaster recovery, we recommend a strategy combining **Logical Backup** with **Offsite Sync**. - -### 2.1 Global Backup Script - -Create a unified backup script `pg_backup.sh` in the `/opt/backup/` directory of the host (`install.svc.plus`): - -```bash -#!/bin/bash -# ========================================== -# Unified PostgreSQL Backup Script -# ========================================== -BACKUP_DIR="/var/backups/postgresql" -DATE=$(date +"%Y%m%d_%H%M%S") -mkdir -p $BACKUP_DIR - -echo "[INFO] Backing up AI Core Services databases (Port 15432)..." -DB_URL_BASE="postgres://svcplus_vps:@127.0.0.1:15432" -for DB in account litellm openclaw qmd rag notification scheduler audit artifact vault_storage; do - pg_dump "$DB_URL_BASE/$DB?sslmode=disable" | gzip > "$BACKUP_DIR/${DB}_$DATE.sql.gz" -done - -echo "[INFO] Backing up Zitadel IAM DB (zitadel-db-1)..." -docker exec zitadel-db-1 pg_dumpall -U postgres | gzip > "$BACKUP_DIR/zitadel_$DATE.sql.gz" - -echo "[INFO] Backing up Gitea Code DB (Native PG 16)..." -sudo -u postgres pg_dumpall -p 5434 | gzip > "$BACKUP_DIR/gitea_$DATE.sql.gz" - -echo "[INFO] Cleaning up backups older than 7 days..." -find $BACKUP_DIR -type f -name "*.sql.gz" -mtime +7 -delete - -echo "[INFO] Backup complete!" -``` - -> [!TIP] -> It is highly recommended to push backup files to an offsite S3 storage (like MinIO) or sync via `rclone` to other cloud storage to prevent datacenter-level disasters. - -### 2.2 Cron Schedule - -Add an automatic backup job at 3:00 AM daily via `crontab -e`: -```cron -0 3 * * * /bin/bash /opt/backup/pg_backup.sh >> /var/log/pg_backup.log 2>&1 -``` - -## 3. Disaster Recovery & Restore Operations - -When data corruption or full-machine migration occurs, please strictly follow this restore sequence. - -> [!WARNING] -> Before executing a restore, you must first stop the business services generating write traffic (like Gitea, LiteLLM, etc.), keeping ONLY the PostgreSQL processes running. - -### 3.1 Restore AI Core Services DB (Port 15432) -Includes: `Account`, `Vault Storage`, `Artifact`, `LiteLLM`, `OpenClaw`, `QMD`, `RAG`, `Notification`, `Scheduler`, `Audit` -```bash -# Unzip backup and import directly via credential URI (Using account as an example) -gunzip -c /var/backups/postgresql/account_YYYYMMDD.sql.gz | psql "postgres://svcplus_vps:@127.0.0.1:15432/account?sslmode=disable" -``` - -### 3.2 Restore Zitadel IAM DB (Docker) -```bash -# Unzip backup and import -gunzip -c /var/backups/postgresql/zitadel_YYYYMMDD.sql.gz | docker exec -i zitadel-db-1 psql -U postgres -``` - -### 3.3 Restore Gitea Code DB (Native) -Since Gitea is on the physical host, we need to switch to the `postgres` user: -```bash -# Unzip backup and import (Note port 5434) -gunzip -c /var/backups/postgresql/gitea_YYYYMMDD.sql.gz | sudo -u postgres psql -p 5434 -``` - -## 4. High Availability & Advanced Recommendations - -For the current single-node multi-instance architecture, logical backups (`pg_dumpall`) are sufficient for most scenarios, but the RPO (Recovery Point Objective) is approximately 24 hours (depending on backup frequency). - -> [!IMPORTANT] -> **Recommendations for upgrading DR tier:** -> 1. **Incremental Backups (WAL Archiving)**: For mission-critical databases like AI Core Services and Gitea, introduce `pgBackRest` or `WAL-G` to continuously archive physical WAL logs to S3 (MinIO) in real-time. This reduces the risk of data loss (RPO) to minutes. -> 2. **Application-Level DR**: -> - Vault Storage decryption keys cannot solely exist in the database. Ensure Vault's physical Root Token / Unseal Keys are safely stored offline. -> - For Gitea, in addition to backing up the database, you must periodically back up `/var/lib/gitea/data` (bare Git repositories and LFS artifacts) on the host. Using Gitea's built-in `gitea dump` command is highly recommended to achieve an integrated backup of both data and repositories. - ---- - -# 中文版 (Chinese Version) - -本文档基于 `install.svc.plus` 线上环境的实际部署情况，为你规划和整理了全局 PostgreSQL 数据库的备份、还原与灾备（DR）标准操作流程。 - -## 1. 线上环境数据库架构清单 - -根据线上环境探勘，当前系统存在 **三个独立运行的 PostgreSQL 实例**。在进行统一备份时，必须分别对这三个实例进行数据导出。 - -| 实例用途 | 运行模式 | 监听端口/地址 | 包含的业务系统库 | -| :--- | :--- | :--- | :--- | -| **AI 核心服务库** | Docker (`postgresql-svc-plus`) | `127.0.0.1:5432` | Account, Vault Storage, Artifact, LiteLLM, OpenClaw, QMD, RAG, Notification, Scheduler, Audit | -| **Zitadel 认证库** | Docker (`zitadel-db-1`) | 容器内部网 | Zitadel IAM 数据 | -| **Gitea 代码库** | Native Systemd (PG 16) | `127.0.0.1:5434` | Gitea | - -## 2. 统一备份策略 (Backup) - -为了实现无缝灾备，建议采用 **逻辑备份 (Logical Backup)** 结合 **定时异地转储 (Offsite Sync)** 的策略。 - -### 2.1 编写全局备份脚本 - -在宿主机 (`install.svc.plus`) 的 `/opt/backup/` 目录下创建统一备份脚本 `pg_backup.sh`： - -```bash -#!/bin/bash -# ========================================== -# 统一 PostgreSQL 备份脚本 -# ========================================== -BACKUP_DIR="/var/backups/postgresql" -DATE=$(date +"%Y%m%d_%H%M%S") -mkdir -p $BACKUP_DIR - -echo "[INFO] 开始备份 AI 核心服务各业务库 (15432 端口)..." -DB_URL_BASE="postgres://svcplus_vps:@127.0.0.1:15432" -for DB in account litellm openclaw qmd rag notification scheduler audit artifact vault_storage; do - pg_dump "$DB_URL_BASE/$DB?sslmode=disable" | gzip > "$BACKUP_DIR/${DB}_$DATE.sql.gz" -done - -echo "[INFO] 开始备份 Zitadel 认证库 (zitadel-db-1)..." -docker exec zitadel-db-1 pg_dumpall -U postgres | gzip > "$BACKUP_DIR/zitadel_$DATE.sql.gz" - -echo "[INFO] 开始备份 Gitea 代码库 (Native PG 16)..." -sudo -u postgres pg_dumpall -p 5434 | gzip > "$BACKUP_DIR/gitea_$DATE.sql.gz" - -echo "[INFO] 清理 7 天前的旧备份..." -find $BACKUP_DIR -type f -name "*.sql.gz" -mtime +7 -delete - -echo "[INFO] 备份完成！" -``` - -> [!TIP] -> 建议将备份文件推送到异地 S3 (例如 MinIO) 或通过 `rclone` 同步到其他云存储，以防范机房级灾难。 - -### 2.2 配置定时任务 (Cron) - -通过 `crontab -e` 添加每天凌晨 3 点自动备份： -```cron -0 3 * * * /bin/bash /opt/backup/pg_backup.sh >> /var/log/pg_backup.log 2>&1 -``` - -## 3. 灾难恢复与还原 (Restore) - -当发生数据损坏或整机迁移时，请严格按照以下顺序进行恢复。 - -> [!WARNING] -> 在执行还原前，必须先停止产生写入流量的业务服务（如 Gitea, LiteLLM 等），仅保留 PostgreSQL 进程运行。 - -### 3.1 还原 AI 核心服务库 (15432 端口) -包含：`Account`, `Vault Storage`, `Artifact`, `LiteLLM`, `OpenClaw`, `QMD`, `RAG`, `Notification`, `Scheduler`, `Audit` -```bash -# 解压备份文件并使用凭证直接导入 (以 account 为例) -gunzip -c /var/backups/postgresql/account_YYYYMMDD.sql.gz | psql "postgres://svcplus_vps:@127.0.0.1:15432/account?sslmode=disable" -``` - -### 3.2 还原 Zitadel 认证库 (Docker) -```bash -# 解压备份文件并导入 -gunzip -c /var/backups/postgresql/zitadel_YYYYMMDD.sql.gz | docker exec -i zitadel-db-1 psql -U postgres -``` - -### 3.3 还原 Gitea 代码库 (Native) -由于 Gitea 在宿主机物理机上，我们需要切换到 postgres 用户执行： -```bash -# 解压备份文件并导入 (注意端口 5434) -gunzip -c /var/backups/postgresql/gitea_YYYYMMDD.sql.gz | sudo -u postgres psql -p 5434 -``` - -## 4. 高可用与进阶建议 (Disaster Recovery) - -对于目前的单机多实例架构，逻辑备份（`pg_dumpall`）足够应付大多数场景，但存在 RPO (恢复点目标) 约为 24 小时的问题（取决于备份频率）。 - -> [!IMPORTANT] -> **提升容灾级别的建议：** -> 1. **增量备份 (WAL Archiving)**: 针对业务极其核心的 AI 核心服务库和 Gitea，可引入 `pgBackRest` 或 `WAL-G`，将物理 WAL 日志实时归档至 S3（MinIO）。这样可以将数据丢失风险（RPO）降低到分钟级别。 -> 2. **应用级灾备**: -> - Vault Storage 的加解密密钥不能仅存在数据库中，请确保 Vault 的物理 Root Token/Unseal Keys 已被安全离线保存。 -> - Gitea 除了备份数据库，还必须定期备份宿主机上的 `/var/lib/gitea/data` (Git 仓库裸数据和 LFS 制品)。可以借助 Gitea 自带的 `gitea dump` 命令来实现数据 + 仓库的一体化备份。 diff --git a/docs/EN/BackUP/backup_dr_plan.md b/docs/EN/BackUP/backup_dr_plan.md new file mode 100644 index 0000000..edf9b14 --- /dev/null +++ b/docs/EN/BackUP/backup_dr_plan.md @@ -0,0 +1,12 @@ +# Full-Site Unified Backup & DR Plan + +This document is based on the actual deployment of the `install.svc.plus` (or `root@install.svc.plus`) production environment. It outlines the global Disaster Recovery (DR) standard operating procedures. In addition to the core PostgreSQL databases, this plan covers all core stateful components required for AI Workspace to run (including application attachments, private images, VPN configurations, etc.). + +## 1. Core Stateful Components & Directory Checklist + +When performing cross-datacenter/cross-zone full-machine migrations, the integrity of the following data must be ensured. + +### 1.1 Database Assets + +| Instance Purpose | Runtime Mode | Listen Port/Address | Contained Business Databases | +| : diff --git a/docs/EN/BackUP/postgresql_disaster_recovery.md b/docs/EN/BackUP/postgresql_disaster_recovery.md new file mode 100644 index 0000000..79b7085 --- /dev/null +++ b/docs/EN/BackUP/postgresql_disaster_recovery.md @@ -0,0 +1,10 @@ +# Unified DB Backup & DR Plan + +This document is based on the actual deployment of the `install.svc.plus` production environment. It outlines the standard operating procedures for the global PostgreSQL database backup, restoration, and Disaster Recovery (DR). + +## 1. Production Database Architecture Checklist + +Based on the production environment reconnaissance, there are currently **three independently running PostgreSQL instances**. When performing unified backups, data must be exported from these three instances separately. + +| Instance Purpose | Runtime Mode | Listen Port/Address | Contained Business Databases | +| : diff --git a/docs/EN/Systems-Overview/PROD/live_systems_overview.md b/docs/EN/Systems-Overview/PROD/live_systems_overview.md new file mode 100644 index 0000000..8df04e8 --- /dev/null +++ b/docs/EN/Systems-Overview/PROD/live_systems_overview.md @@ -0,0 +1,8 @@ +# Live Systems Overview (Caddy) + +By reverse-engineering the configurations under `/etc/caddy/conf.d/` on the `install.svc.plus` node, I have outlined all business systems currently running in the production environment, their corresponding domains, and internal port mappings. + +## 1. Core AI Infrastructure + +| Business System | Exposed Domain | Internal Upstream | Description | +| : diff --git a/docs/Systems-Overview/PROD/live_systems_overview.md b/docs/Systems-Overview/PROD/live_systems_overview.md deleted file mode 100644 index 5cd92ff..0000000 --- a/docs/Systems-Overview/PROD/live_systems_overview.md +++ /dev/null @@ -1,141 +0,0 @@ -# 线上环境全业务系统拓扑梳理 (Caddy) | Live Systems Overview (Caddy) - -*🇨🇳 中文版在下方 | Chinese version below* - ---- - -# English Version - -By reverse-engineering the configurations under `/etc/caddy/conf.d/` on the `install.svc.plus` node, I have outlined all business systems currently running in the production environment, their corresponding domains, and internal port mappings. - -## 1. Core AI Infrastructure - -| Business System | Exposed Domain | Internal Upstream | Description | -| :--- | :--- | :--- | :--- | -| API Gateway | `apigateway.svc.plus` | `127.0.0.1:9080` | Universal API Gateway | -| LiteLLM (API) | `api.svc.plus` | `127.0.0.1:4000` | Unified model routing & OpenAI/Anthropic path translation layer | -| LiteLLM (UI) | `litellm.svc.plus` | `127.0.0.1:4000` | LiteLLM Admin UI / Dashboard | -| OpenClaw (Bot) | `openclaw.svc.plus` | `127.0.0.1:18789` | OpenClaw chatbot backend service | -| RAG Server | `rag-server.svc.plus`
`rag-server-contabo-*.svc.plus` | `127.0.0.1:18084` | RAG (Retrieval-Augmented Generation) backend | -| XWorkmate Bridge | `xworkmate-bridge.svc.plus` | `127.0.0.1:8787` | Workflow bridge with Bearer Token strict authentication | - -## 2. DevOps & Tools - -| Business System | Exposed Domain | Internal Upstream | Description | -| :--- | :--- | :--- | :--- | -| Gitea | `gitea.svc.plus` | `localhost:3001` | Private Git code hosting platform | -| Vault | `vault.svc.plus` | `127.0.0.1:8200` | Unified credential and secret persistence service | -| ~~Code Server~~ | ~~`observability.svc.plus/code/`~~ | ~~`127.0.0.1:8443`~~ | ⚠️ **[To Be Deprecated]** Web-based VSCode development environment | -| ~~Jupyter Lab~~ | ~~`observability.svc.plus/jupyter/`~~ | ~~`127.0.0.1:8888`~~ | ⚠️ **[To Be Deprecated]** Data science Jupyter environment | - -## 3. IAM & Accounts - -| Business System | Exposed Domain | Internal Upstream | Description | -| :--- | :--- | :--- | :--- | -| Zitadel | `zitadel.svc.plus` | `127.0.0.1:19080/19081` | IAM Global SSO (API & UI) | -| Accounts | `accounts.svc.plus`
`accounts-contabo-*.svc.plus` | `127.0.0.1:18081` | Production account management service | -| ~~Accounts (Preview)~~ | ~~`accounts-preview.svc.plus`~~ | ~~`127.0.0.1:28081`~~ | ⚠️ **[To Be Deprecated]** Preview/Test account service (Will not be brought up on DR node) | - -## 4. Frontends - -| Business System | Exposed Domain | Internal Upstream | Description | -| :--- | :--- | :--- | :--- | -| Console | `console.svc.plus` | `127.0.0.1:3000` | AI Workspace main console dashboard | -| Docs | `docs.svc.plus`
`docs-contabo-*.svc.plus` | `127.0.0.1:18083` | System documentation (Reuses x-cloud-flow port) | -| Install Scripts | `install.svc.plus` | `302 Redir -> Github` | Short link distribution for curl-based one-click install scripts | -| Ebook | `ebook.svc.plus` | Static Files | Modern IT History ebook (`/opt/modern-it-history/current`) | - -## 5. Observability Stack - -`observability.svc.plus` acts as the global ingress for all monitoring components, routing by path: - -| Path Rule | Internal Upstream | Component / Purpose | -| :--- | :--- | :--- | -| `/grafana/*` | `127.0.0.1:3000` | Grafana visual dashboards (Also default root redirect) | -| `/ingest/metrics/*`
`/vmetrics/*` | `127.0.0.1:8428` | VictoriaMetrics (Prometheus metrics write & query) | -| `/ingest/logs/*`
`/vlogs/*` | `127.0.0.1:9428` | VictoriaLogs / Loki (Log write & query) | -| `/ingest/otlp/*`
`/vtraces/*` | `127.0.0.1:4318 / 10428` | OpenTelemetry distributed tracing | -| ~~`/insight/*`~~ | ~~`127.0.0.1:8082`~~ | ⚠️ **[To Be Deprecated]** Insight Workbench data analysis console | -| `/vmalert/*` | `127.0.0.1:8880` | VictoriaMetrics Alert engine | -| `/alertmgr/*` | `127.0.0.1:9059` | Alertmanager alert routing & dispatch | -| `/blackbox/*` | `127.0.0.1:9115` | Blackbox Exporter network probing | - -## 6. Other Underlying/Network Components - -| Business System | Exposed Domain | Internal Upstream | Description | -| :--- | :--- | :--- | :--- | -| ~~X-Cloud-Flow~~ | ~~`x-cloud-flow.svc.plus`~~ | ~~`127.0.0.1:18083/18087`~~ | ⚠️ **[To Be Deprecated]** Cloud traffic control or orchestration engine | -| ~~X-Ops-Agent~~ | ~~`x-ops-agent.svc.plus`~~ | ~~`127.0.0.1:18084/18086`~~ | ⚠️ **[To Be Deprecated]** Automated operations Agent | -| ~~X-Scope-Hub~~ | ~~`x-scope-hub.svc.plus`~~ | ~~`127.0.0.1:18085`~~ | ⚠️ **[To Be Deprecated]** Resource scoping hub | -| Hermes | `hermes.svc.plus` | `127.0.0.1:18180` | Message notification gateway | -| JP XHTTP / Xray | `jp-xhttp.svc.plus` | `/dev/shm/xray.sock` | Cross-border network proxy tunnel (gRPC) | -| PostgreSQL Tunnel | `postgresql-contabo...` | Static 200 Response | Network probe or specific proxy ingress for TLS handshakes | - ---- - -# 中文版 (Chinese Version) - -通过对 `install.svc.plus` 节点 `/etc/caddy/conf.d/` 下的配置进行逆向解析，我为你梳理出了线上环境中运行的所有业务系统、对应域名及内网端口映射。 - -## 1. 核心 AI 基础设施 (AI Infrastructure) - -| 业务系统 | 对外暴露域名 (Domain) | 内网代理目标 (Upstream) | 备注描述 | -| :--- | :--- | :--- | :--- | -| API Gateway | `apigateway.svc.plus` | `127.0.0.1:9080` | 通用 API 网关 | -| LiteLLM (API) | `api.svc.plus` | `127.0.0.1:4000` | 模型统一路由与 OpenAI/Anthropic 路径兼容转换层 | -| LiteLLM (UI) | `litellm.svc.plus` | `127.0.0.1:4000` | LiteLLM Admin UI / Dashboard | -| OpenClaw (Bot) | `openclaw.svc.plus` | `127.0.0.1:18789` | OpenClaw 聊天机器人后台服务 | -| RAG Server | `rag-server.svc.plus`
`rag-server-contabo-*.svc.plus` | `127.0.0.1:18084` | RAG (检索增强生成) 服务后端 | -| XWorkmate Bridge | `xworkmate-bridge.svc.plus` | `127.0.0.1:8787` | 带有 Bearer Token 强鉴权的工作流桥接器 | - -## 2. 开发者与协同工具 (DevOps & Tools) - -| 业务系统 | 对外暴露域名 (Domain) | 内网代理目标 (Upstream) | 备注描述 | -| :--- | :--- | :--- | :--- | -| Gitea | `gitea.svc.plus` | `localhost:3001` | 私有 Git 代码托管平台 | -| Vault | `vault.svc.plus` | `127.0.0.1:8200` | 统一凭证与密钥持久化服务 | -| ~~Code Server~~ | ~~`observability.svc.plus/code/`~~ | ~~`127.0.0.1:8443`~~ | ⚠️ **[计划下线]** 网页版 VSCode 开发环境 | -| ~~Jupyter Lab~~ | ~~`observability.svc.plus/jupyter/`~~ | ~~`127.0.0.1:8888`~~ | ⚠️ **[计划下线]** 数据科学 Jupyter 环境 | - -## 3. 身份认证与账号体系 (IAM & Accounts) - -| 业务系统 | 对外暴露域名 (Domain) | 内网代理目标 (Upstream) | 备注描述 | -| :--- | :--- | :--- | :--- | -| Zitadel | `zitadel.svc.plus` | `127.0.0.1:19080/19081` | IAM 全局单点登录 (API 与 UI) | -| Accounts | `accounts.svc.plus`
`accounts-contabo-*.svc.plus` | `127.0.0.1:18081` | 生产环境账户管理服务 | -| ~~Accounts (Preview)~~ | ~~`accounts-preview.svc.plus`~~ | ~~`127.0.0.1:28081`~~ | ⚠️ **[计划下线]** 预览 / 测试环境账户服务 (灾备节点不再拉起) | - -## 4. 控制台与前端应用 (Frontends) - -| 业务系统 | 对外暴露域名 (Domain) | 内网代理目标 (Upstream) | 备注描述 | -| :--- | :--- | :--- | :--- | -| Console | `console.svc.plus` | `127.0.0.1:3000` | AI Workspace 主控台面板 | -| Docs | `docs.svc.plus`
`docs-contabo-*.svc.plus` | `127.0.0.1:18083` | 系统帮助文档 (复用 x-cloud-flow 端口) | -| Install Scripts | `install.svc.plus` | `302 Redir -> Github` | 供 curl 拉取执行的一键安装脚本短链接分发 | -| Ebook | `ebook.svc.plus` | 静态文件 | Modern IT History 电子书 (`/opt/modern-it-history/current`) | - -## 5. 可观测性与日志监控 (Observability Stack) - -`observability.svc.plus` 为所有监控组件的全局入口，按 Path 转发到不同组件： - -| 路径规则 (Path) | 内网代理目标 (Upstream) | 对应组件 / 用途 | -| :--- | :--- | :--- | -| `/grafana/*` | `127.0.0.1:3000` | Grafana 可视化仪表盘 (也是根目录默认重定向) | -| `/ingest/metrics/*`
`/vmetrics/*` | `127.0.0.1:8428` | VictoriaMetrics (Prometheus 指标写入与查询) | -| `/ingest/logs/*`
`/vlogs/*` | `127.0.0.1:9428` | VictoriaLogs / Loki (日志写入与查询) | -| `/ingest/otlp/*`
`/vtraces/*` | `127.0.0.1:4318 / 10428` | OpenTelemetry 分布式链路追踪 | -| ~~`/insight/*`~~ | ~~`127.0.0.1:8082`~~ | ⚠️ **[计划下线]** Insight Workbench 数据分析台 | -| `/vmalert/*` | `127.0.0.1:8880` | VictoriaMetrics Alert 告警引擎 | -| `/alertmgr/*` | `127.0.0.1:9059` | Alertmanager 告警路由与发送 | -| `/blackbox/*` | `127.0.0.1:9115` | Blackbox Exporter 网络拨测 | - -## 6. 其他底层/网络组件 - -| 业务系统 | 对外暴露域名 (Domain) | 内网代理目标 (Upstream) | 备注描述 | -| :--- | :--- | :--- | :--- | -| ~~X-Cloud-Flow~~ | ~~`x-cloud-flow.svc.plus`~~ | ~~`127.0.0.1:18083/18087`~~ | ⚠️ **[计划下线]** 云上流控或编排引擎 | -| ~~X-Ops-Agent~~ | ~~`x-ops-agent.svc.plus`~~ | ~~`127.0.0.1:18084/18086`~~ | ⚠️ **[计划下线]** 自动化运维 Agent | -| ~~X-Scope-Hub~~ | ~~`x-scope-hub.svc.plus`~~ | ~~`127.0.0.1:18085`~~ | ⚠️ **[计划下线]** 资源范围枢纽 | -| Hermes | `hermes.svc.plus` | `127.0.0.1:18180` | 消息通知网关 (Notification) | -| JP XHTTP / Xray | `jp-xhttp.svc.plus` | `/dev/shm/xray.sock` | 跨境网络代理隧道 (gRPC) | -| PostgreSQL Tunnel | `postgresql-contabo...` | 静态 200 响应 | 仅作为 TLS 握手的网络探针或特定代理入口 | diff --git a/docs/BackUP/Site-Migration/implementation_plan.md b/docs/ZH/BackUP/Site-Migration/implementation_plan.md similarity index 100% rename from docs/BackUP/Site-Migration/implementation_plan.md rename to docs/ZH/BackUP/Site-Migration/implementation_plan.md diff --git a/docs/BackUP/Site-Migration/walkthrough.md b/docs/ZH/BackUP/Site-Migration/walkthrough.md similarity index 100% rename from docs/BackUP/Site-Migration/walkthrough.md rename to docs/ZH/BackUP/Site-Migration/walkthrough.md diff --git a/docs/ZH/BackUP/backup_dr_plan.md b/docs/ZH/BackUP/backup_dr_plan.md new file mode 100644 index 0000000..0072fa9 --- /dev/null +++ b/docs/ZH/BackUP/backup_dr_plan.md @@ -0,0 +1,103 @@ +# 全站统一备份与灾备方案 (AI Workspace) + +本文档基于 `install.svc.plus` (或 `root@install.svc.plus`) 线上环境的实际部署情况，为你规划和整理了全局的灾备（DR）标准操作流程。 除了核心的 PostgreSQL 数据库外，本方案覆盖了 AI Workspace 运行所需的所有核心状态组件（含应用附件、私有镜像、VPN 配置等）。 + +## 1. 核心状态组件与目录清单 + +在进行跨机房/跨区整机迁移时，必须确保以下数据的完整迁移。 + +### 1.1 数据库资产 + +| 实例用途 | 运行模式 | 监听端口/地址 | 包含的业务系统库 | +| :--- | :--- | :--- | :--- | +| AI 核心服务库 | Docker (`postgresql-svc-plus`) | `127.0.0.1:15432` | Account, Vault Storage, Artifact, LiteLLM, OpenClaw, QMD, RAG, Notification, Scheduler, Audit | +| Zitadel 认证库 | Docker (`zitadel-db-1`) | 容器内部网 | Zitadel IAM 数据 | +| Gitea 代码库 | Native Systemd (PG 16) | `127.0.0.1:5434` | Gitea 元数据 | + +### 1.2 关键文件与持久化挂载卷 (Stateful Volumes) + +> [!IMPORTANT] +> 以下目录是系统恢复后的“血肉”，如果不迁移，即使数据库还原，也会出现图片 404、模型拉取失败、组网断开等致命故障。 + +* **Gitea 仓库与对象数据:** `/var/lib/gitea/data` + 存储所有的 Git 裸仓库、LFS 大文件、用户头像及附件。 +* **应用私有镜像库 (Harbor / Registry):** `/data/registry` 或挂载的 `/data/harbor` + 私有容器镜像。如果不迁移，新机在重启并执行 `docker-compose pull` 时将找不到未开源的基础环境镜像。 +* **OpenClaw (AI 网关) 工作目录:** `openclaw workdir` (对应部署时的物理挂载点) + 包含 AI Gateway 的本地缓存档、向量化临时数据或加载的模型资产。 +* **分布式组网与 VPN 配置:** `/etc/wireguard` + 包含本地节点的 WireGuard Private Key 与 Peers 路由表。若丢失此目录，新主机启动后将无法重新融入原有的 Xworkmate VPN 星型网络，所有跨云微服务通信将中断。 +* **对象存储 (MinIO/S3):** `/data/minio` (若开启) + 现代云原生组件（如 Wiki、RAG）的大文件存储底座。 +* **环境密钥文件:** 各组件部署目录下的 `.env` (如 `/opt/ai-workspace/`)。 + +> [!NOTE] +> **例外清单：** `/etc/caddy/data` 或 `acme.json`（Let's Encrypt 证书）。 +> **处理策略：** 灾备演练或真实迁移时无需同步。只需在新主机上线时，提前将 DNS A 记录解析至新 IP，再启动 Caddy 即可实现全自动在线续签。（需注意 Let's Encrypt 每周 5 次的重复签发限频，演练时建议使用 Staging 证书）。 + +## 2. 统一备份与同步策略 (Backup & Sync) + +### 2.1 数据库逻辑备份 (PostgreSQL) + +建议在宿主机的 `/opt/backup/` 目录下创建统一备份脚本 `pg_backup.sh` 并加入 Cron 定时任务： + +```bash +#!/bin/bash +BACKUP_DIR="/var/backups/postgresql" +DATE=$(date +"%Y%m%d_%H%M%S") +mkdir -p $BACKUP_DIR + +echo "[INFO] 备份 AI 核心服务库 (15432 端口)..." +DB_URL_BASE="postgres://svcplus_vps:@127.0.0.1:15432" +for DB in account litellm openclaw qmd rag notification scheduler audit artifact vault_storage; do + pg_dump "$DB_URL_BASE/$DB?sslmode=disable" | gzip > "$BACKUP_DIR/${DB}_$DATE.sql.gz" +done + +echo "[INFO] 备份 Zitadel 认证库 (zitadel-db-1)..." +docker exec zitadel-db-1 pg_dumpall -U postgres | gzip > "$BACKUP_DIR/zitadel_$DATE.sql.gz" + +echo "[INFO] 备份 Gitea 代码库 (Native PG 16)..." +sudo -u postgres pg_dumpall -p 5434 | gzip > "$BACKUP_DIR/gitea_$DATE.sql.gz" + +# 清理 7 天前的旧备份 +find $BACKUP_DIR -type f -name "*.sql.gz" -mtime +7 -delete +``` + +### 2.2 文件状态同步 (Rsync 增量同步) + +针对文件资源，建议使用 `rsync` 结合 Ansible 剧本向灾备节点推送，或定时同步到集中式对象存储。 关键的 `rsync` 路径示例： + +```bash +rsync -avz --delete /var/lib/gitea/data/ backup-server:/backup/gitea/ +rsync -avz --delete /etc/wireguard/ backup-server:/backup/wireguard/ +rsync -avz --delete /data/registry/ backup-server:/backup/registry/ +``` + +## 3. 灾难恢复与还原操作 (Restore) + +当发生整机崩溃或机房断网需要启用灾备节点时，严格按以下层级恢复： + +**阶段 1: 恢复 VPN 与网络层** +在安装任何服务前，首先将 `/etc/wireguard` 目录还原，并启动 `wg-quick up wg0`，确保灾备节点重新获取原集群内网 IP。 + +**阶段 2: 恢复对象与文件持久层** +将备份好的 Gitea、OpenClaw workdir、应用镜像目录等还原至宿主机的绝对路径中，并修复属主权限（如 `chown -R 1000:1000 /var/lib/gitea`）。 + +**阶段 3: 恢复 PostgreSQL 数据库层** +确保仅启动数据库容器（停掉应用层容器）。 + +```bash +# 1. 还原 AI 核心库 +gunzip -c /var/backups/postgresql/account_YYYYMMDD.sql.gz | psql "postgres://...:15432/account" + +# 2. 还原 Zitadel 库 +gunzip -c /var/backups/postgresql/zitadel_YYYYMMDD.sql.gz | docker exec -i zitadel-db-1 psql -U postgres + +# 3. 还原 Gitea 库 +gunzip -c /var/backups/postgresql/gitea_YYYYMMDD.sql.gz | sudo -u postgres psql -p 5434 +``` + +**阶段 4: 启动接入层与业务应用** +1. 确认全网 DNS 已切换至当前灾备节点公网 IP。 +2. 启动 Caddy 服务，系统会自动发起 HTTP-01 质询申请全新证书。 +3. 全面上线 `docker-compose up -d` 拉起各业务线应用。 diff --git a/docs/ZH/BackUP/postgresql_disaster_recovery.md b/docs/ZH/BackUP/postgresql_disaster_recovery.md new file mode 100644 index 0000000..bb6ca9c --- /dev/null +++ b/docs/ZH/BackUP/postgresql_disaster_recovery.md @@ -0,0 +1,96 @@ +# 统一数据库备份与灾备方案 (PostgreSQL) + +本文档基于 `install.svc.plus` 线上环境的实际部署情况，为你规划和整理了全局 PostgreSQL 数据库的备份、还原与灾备（DR）标准操作流程。 + +## 1. 线上环境数据库架构清单 + +根据线上环境探勘，当前系统存在 **三个独立运行的 PostgreSQL 实例**。在进行统一备份时，必须分别对这三个实例进行数据导出。 + +| 实例用途 | 运行模式 | 监听端口/地址 | 包含的业务系统库 | +| :--- | :--- | :--- | :--- | +| **AI 核心服务库** | Docker (`postgresql-svc-plus`) | `127.0.0.1:5432` | Account, Vault Storage, Artifact, LiteLLM, OpenClaw, QMD, RAG, Notification, Scheduler, Audit | +| **Zitadel 认证库** | Docker (`zitadel-db-1`) | 容器内部网 | Zitadel IAM 数据 | +| **Gitea 代码库** | Native Systemd (PG 16) | `127.0.0.1:5434` | Gitea | + +## 2. 统一备份策略 (Backup) + +为了实现无缝灾备，建议采用 **逻辑备份 (Logical Backup)** 结合 **定时异地转储 (Offsite Sync)** 的策略。 + +### 2.1 编写全局备份脚本 + +在宿主机 (`install.svc.plus`) 的 `/opt/backup/` 目录下创建统一备份脚本 `pg_backup.sh`： + +```bash +#!/bin/bash +# ========================================== +# 统一 PostgreSQL 备份脚本 +# ========================================== +BACKUP_DIR="/var/backups/postgresql" +DATE=$(date +"%Y%m%d_%H%M%S") +mkdir -p $BACKUP_DIR + +echo "[INFO] 开始备份 AI 核心服务各业务库 (15432 端口)..." +DB_URL_BASE="postgres://svcplus_vps:@127.0.0.1:15432" +for DB in account litellm openclaw qmd rag notification scheduler audit artifact vault_storage; do + pg_dump "$DB_URL_BASE/$DB?sslmode=disable" | gzip > "$BACKUP_DIR/${DB}_$DATE.sql.gz" +done + +echo "[INFO] 开始备份 Zitadel 认证库 (zitadel-db-1)..." +docker exec zitadel-db-1 pg_dumpall -U postgres | gzip > "$BACKUP_DIR/zitadel_$DATE.sql.gz" + +echo "[INFO] 开始备份 Gitea 代码库 (Native PG 16)..." +sudo -u postgres pg_dumpall -p 5434 | gzip > "$BACKUP_DIR/gitea_$DATE.sql.gz" + +echo "[INFO] 清理 7 天前的旧备份..." +find $BACKUP_DIR -type f -name "*.sql.gz" -mtime +7 -delete + +echo "[INFO] 备份完成！" +``` + +> [!TIP] +> 建议将备份文件推送到异地 S3 (例如 MinIO) 或通过 `rclone` 同步到其他云存储，以防范机房级灾难。 + +### 2.2 配置定时任务 (Cron) + +通过 `crontab -e` 添加每天凌晨 3 点自动备份： +```cron +0 3 * * * /bin/bash /opt/backup/pg_backup.sh >> /var/log/pg_backup.log 2>&1 +``` + +## 3. 灾难恢复与还原 (Restore) + +当发生数据损坏或整机迁移时，请严格按照以下顺序进行恢复。 + +> [!WARNING] +> 在执行还原前，必须先停止产生写入流量的业务服务（如 Gitea, LiteLLM 等），仅保留 PostgreSQL 进程运行。 + +### 3.1 还原 AI 核心服务库 (15432 端口) +包含：`Account`, `Vault Storage`, `Artifact`, `LiteLLM`, `OpenClaw`, `QMD`, `RAG`, `Notification`, `Scheduler`, `Audit` +```bash +# 解压备份文件并使用凭证直接导入 (以 account 为例) +gunzip -c /var/backups/postgresql/account_YYYYMMDD.sql.gz | psql "postgres://svcplus_vps:@127.0.0.1:15432/account?sslmode=disable" +``` + +### 3.2 还原 Zitadel 认证库 (Docker) +```bash +# 解压备份文件并导入 +gunzip -c /var/backups/postgresql/zitadel_YYYYMMDD.sql.gz | docker exec -i zitadel-db-1 psql -U postgres +``` + +### 3.3 还原 Gitea 代码库 (Native) +由于 Gitea 在宿主机物理机上，我们需要切换到 postgres 用户执行： +```bash +# 解压备份文件并导入 (注意端口 5434) +gunzip -c /var/backups/postgresql/gitea_YYYYMMDD.sql.gz | sudo -u postgres psql -p 5434 +``` + +## 4. 高可用与进阶建议 (Disaster Recovery) + +对于目前的单机多实例架构，逻辑备份（`pg_dumpall`）足够应付大多数场景，但存在 RPO (恢复点目标) 约为 24 小时的问题（取决于备份频率）。 + +> [!IMPORTANT] +> **提升容灾级别的建议：** +> 1. **增量备份 (WAL Archiving)**: 针对业务极其核心的 AI 核心服务库和 Gitea，可引入 `pgBackRest` 或 `WAL-G`，将物理 WAL 日志实时归档至 S3（MinIO）。这样可以将数据丢失风险（RPO）降低到分钟级别。 +> 2. **应用级灾备**: +> - Vault Storage 的加解密密钥不能仅存在数据库中，请确保 Vault 的物理 Root Token/Unseal Keys 已被安全离线保存。 +> - Gitea 除了备份数据库，还必须定期备份宿主机上的 `/var/lib/gitea/data` (Git 仓库裸数据和 LFS 制品)。可以借助 Gitea 自带的 `gitea dump` 命令来实现数据 + 仓库的一体化备份。 diff --git a/docs/ZH/Systems-Overview/PROD/live_systems_overview.md b/docs/ZH/Systems-Overview/PROD/live_systems_overview.md new file mode 100644 index 0000000..a79de82 --- /dev/null +++ b/docs/ZH/Systems-Overview/PROD/live_systems_overview.md @@ -0,0 +1,66 @@ +# 线上环境全业务系统拓扑梳理 (Caddy) + +通过对 `install.svc.plus` 节点 `/etc/caddy/conf.d/` 下的配置进行逆向解析，我为你梳理出了线上环境中运行的所有业务系统、对应域名及内网端口映射。 + +## 1. 核心 AI 基础设施 (AI Infrastructure) + +| 业务系统 | 对外暴露域名 (Domain) | 内网代理目标 (Upstream) | 备注描述 | +| :--- | :--- | :--- | :--- | +| API Gateway | `apigateway.svc.plus` | `127.0.0.1:9080` | 通用 API 网关 | +| LiteLLM (API) | `api.svc.plus` | `127.0.0.1:4000` | 模型统一路由与 OpenAI/Anthropic 路径兼容转换层 | +| LiteLLM (UI) | `litellm.svc.plus` | `127.0.0.1:4000` | LiteLLM Admin UI / Dashboard | +| OpenClaw (Bot) | `openclaw.svc.plus` | `127.0.0.1:18789` | OpenClaw 聊天机器人后台服务 | +| RAG Server | `rag-server.svc.plus`
`rag-server-contabo-*.svc.plus` | `127.0.0.1:18084` | RAG (检索增强生成) 服务后端 | +| XWorkmate Bridge | `xworkmate-bridge.svc.plus` | `127.0.0.1:8787` | 带有 Bearer Token 强鉴权的工作流桥接器 | + +## 2. 开发者与协同工具 (DevOps & Tools) + +| 业务系统 | 对外暴露域名 (Domain) | 内网代理目标 (Upstream) | 备注描述 | +| :--- | :--- | :--- | :--- | +| Gitea | `gitea.svc.plus` | `localhost:3001` | 私有 Git 代码托管平台 | +| Vault | `vault.svc.plus` | `127.0.0.1:8200` | 统一凭证与密钥持久化服务 | +| ~~Code Server~~ | ~~`observability.svc.plus/code/`~~ | ~~`127.0.0.1:8443`~~ | ⚠️ **[计划下线]** 网页版 VSCode 开发环境 | +| ~~Jupyter Lab~~ | ~~`observability.svc.plus/jupyter/`~~ | ~~`127.0.0.1:8888`~~ | ⚠️ **[计划下线]** 数据科学 Jupyter 环境 | + +## 3. 身份认证与账号体系 (IAM & Accounts) + +| 业务系统 | 对外暴露域名 (Domain) | 内网代理目标 (Upstream) | 备注描述 | +| :--- | :--- | :--- | :--- | +| Zitadel | `zitadel.svc.plus` | `127.0.0.1:19080/19081` | IAM 全局单点登录 (API 与 UI) | +| Accounts | `accounts.svc.plus`
`accounts-contabo-*.svc.plus` | `127.0.0.1:18081` | 生产环境账户管理服务 | +| ~~Accounts (Preview)~~ | ~~`accounts-preview.svc.plus`~~ | ~~`127.0.0.1:28081`~~ | ⚠️ **[计划下线]** 预览 / 测试环境账户服务 (灾备节点不再拉起) | + +## 4. 控制台与前端应用 (Frontends) + +| 业务系统 | 对外暴露域名 (Domain) | 内网代理目标 (Upstream) | 备注描述 | +| :--- | :--- | :--- | :--- | +| Console | `console.svc.plus` | `127.0.0.1:3000` | AI Workspace 主控台面板 | +| Docs | `docs.svc.plus`
`docs-contabo-*.svc.plus` | `127.0.0.1:18083` | 系统帮助文档 (复用 x-cloud-flow 端口) | +| Install Scripts | `install.svc.plus` | `302 Redir -> Github` | 供 curl 拉取执行的一键安装脚本短链接分发 | +| Ebook | `ebook.svc.plus` | 静态文件 | Modern IT History 电子书 (`/opt/modern-it-history/current`) | + +## 5. 可观测性与日志监控 (Observability Stack) + +`observability.svc.plus` 为所有监控组件的全局入口，按 Path 转发到不同组件： + +| 路径规则 (Path) | 内网代理目标 (Upstream) | 对应组件 / 用途 | +| :--- | :--- | :--- | +| `/grafana/*` | `127.0.0.1:3000` | Grafana 可视化仪表盘 (也是根目录默认重定向) | +| `/ingest/metrics/*`
`/vmetrics/*` | `127.0.0.1:8428` | VictoriaMetrics (Prometheus 指标写入与查询) | +| `/ingest/logs/*`
`/vlogs/*` | `127.0.0.1:9428` | VictoriaLogs / Loki (日志写入与查询) | +| `/ingest/otlp/*`
`/vtraces/*` | `127.0.0.1:4318 / 10428` | OpenTelemetry 分布式链路追踪 | +| ~~`/insight/*`~~ | ~~`127.0.0.1:8082`~~ | ⚠️ **[计划下线]** Insight Workbench 数据分析台 | +| `/vmalert/*` | `127.0.0.1:8880` | VictoriaMetrics Alert 告警引擎 | +| `/alertmgr/*` | `127.0.0.1:9059` | Alertmanager 告警路由与发送 | +| `/blackbox/*` | `127.0.0.1:9115` | Blackbox Exporter 网络拨测 | + +## 6. 其他底层/网络组件 + +| 业务系统 | 对外暴露域名 (Domain) | 内网代理目标 (Upstream) | 备注描述 | +| :--- | :--- | :--- | :--- | +| ~~X-Cloud-Flow~~ | ~~`x-cloud-flow.svc.plus`~~ | ~~`127.0.0.1:18083/18087`~~ | ⚠️ **[计划下线]** 云上流控或编排引擎 | +| ~~X-Ops-Agent~~ | ~~`x-ops-agent.svc.plus`~~ | ~~`127.0.0.1:18084/18086`~~ | ⚠️ **[计划下线]** 自动化运维 Agent | +| ~~X-Scope-Hub~~ | ~~`x-scope-hub.svc.plus`~~ | ~~`127.0.0.1:18085`~~ | ⚠️ **[计划下线]** 资源范围枢纽 | +| Hermes | `hermes.svc.plus` | `127.0.0.1:18180` | 消息通知网关 (Notification) | +| JP XHTTP / Xray | `jp-xhttp.svc.plus` | `/dev/shm/xray.sock` | 跨境网络代理隧道 (gRPC) | +| PostgreSQL Tunnel | `postgresql-contabo...` | 静态 200 响应 | 仅作为 TLS 握手的网络探针或特定代理入口 |