From 3ea15f580605c6bca3dc30a1696dc942719aaadc Mon Sep 17 00:00:00 2001 From: Haitao Pan Date: Wed, 1 Jul 2026 08:57:08 +0800 Subject: [PATCH] docs: add bilingual backup and DR plan, update live systems overview --- README.md | 103 +++++++++ .../Site-Migration/implementation_plan.md | 73 ++++++ docs/BackUP/Site-Migration/walkthrough.md | 57 +++++ docs/BackUP/backup_dr_plan.md | 215 ++++++++++++++++++ docs/BackUP/postgresql_disaster_recovery.md | 201 ++++++++++++++++ .../PROD/live_systems_overview.md | 141 ++++++++++++ 6 files changed, 790 insertions(+) create mode 100644 README.md create mode 100644 docs/BackUP/Site-Migration/implementation_plan.md create mode 100644 docs/BackUP/Site-Migration/walkthrough.md create mode 100644 docs/BackUP/backup_dr_plan.md create mode 100644 docs/BackUP/postgresql_disaster_recovery.md create mode 100644 docs/Systems-Overview/PROD/live_systems_overview.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..8ef9ec9 --- /dev/null +++ b/README.md @@ -0,0 +1,103 @@ +# Site Migration & Backup Toolkit (site-recovery) + +*🇨🇳 中文版在下方 | Chinese version below* + +Welcome to the **Site Migration & Backup Toolkit**. This repository provides the orchestrations, runbooks, and automated playbooks for managing the disaster recovery lifecycle of the AI Workspace infrastructure. + +## Phase Roadmap + +This project is iteratively rolling out disaster recovery capabilities. Currently, we are heavily focused on **Phase 1**. + +### Phase 1: Migration / Cold Backup / Offline Restore (Current) + +Implement only the foundation: + +* Ansible automated migration skeleton +* File and configuration packaging +* PostgreSQL dump / restore skeleton +* Vault secret migration design placeholder +* DNS cutover pre-check placeholder +* GitHub Actions regression verification skeleton +* Manual recovery fallback scripts and runbook + +### Phase 2: Warm Standby / Scheduled Backup + +Only reserve documentation and interface placeholders for: + +* Scheduled backup +* Incremental sync +* Object storage archive +* Restore drill +* RPO / RTO report + +### Phase 3: Hot Backup / DTS / Replication + +Only reserve roadmap placeholders for: + +* PostgreSQL streaming replication +* Redis replication +* DTS / CDC +* Dual-write validation +* Failover plan + +### Phase 4: Multi-Active / DR Platform + +Only reserve roadmap placeholders for: + +* Multi-region deployment +* Traffic routing +* Data consistency strategy +* Active-Active / Active-Passive +* DR orchestration platform + +--- + +# 站点迁移与备份工具集 (Site Migration & Backup Toolkit) + +欢迎使用 **Site Migration & Backup Toolkit**。本代码库提供了管理 AI Workspace 基础架构灾难恢复生命周期的编排、运维手册和自动化 Playbooks。 + +## 阶段演进路线图 (Phase Roadmap) + +本项目正在迭代推出灾备能力。目前，我们正重点聚焦于 **Phase 1 (第一阶段)**。 + +### Phase 1: 迁移 / 冷备 / 离线恢复 (当前阶段) + +仅实现基础底座： + +* Ansible 自动化迁移骨架 +* 文件与配置打包 +* PostgreSQL 逻辑备份 / 还原骨架 +* Vault 凭证迁移设计占位符 +* DNS 流量切换前置检查占位符 +* GitHub Actions 回归验证骨架 +* 手动恢复降级脚本与运维手册 + +### Phase 2: 温备 / 定时备份 + +仅保留文档与接口占位符： + +* 定时计划备份 +* 增量同步 +* 对象存储归档 +* 还原演练 +* RPO / RTO 报告 + +### Phase 3: 热备 / DTS / 复制 + +仅保留路线图占位符： + +* PostgreSQL 流复制 +* Redis 数据复制 +* DTS / CDC (变更数据捕获) +* 双写验证 +* 故障转移 (Failover) 计划 + +### Phase 4: 多活 / 灾备管理平台 + +仅保留路线图占位符： + +* 多地域 (Multi-region) 部署 +* 流量智能路由 +* 数据一致性策略 +* 双活 (Active-Active) / 主备 (Active-Passive) +* DR 灾备编排平台 diff --git a/docs/BackUP/Site-Migration/implementation_plan.md b/docs/BackUP/Site-Migration/implementation_plan.md new file mode 100644 index 0000000..23aea21 --- /dev/null +++ b/docs/BackUP/Site-Migration/implementation_plan.md @@ -0,0 +1,73 @@ +# 全站单向数据流迁移与环境复制方案 (Implementation Plan) + +该方案旨在构建一套**声明式、单向数据流（Source -> Target）、域名支持参数化**的自动化环境迁移与恢复体系。基于你提供的需求，整体设计分为三个核心部分。 + +## Proposed Changes + +### 1. 声明式迁移配置文件模板 (Declarative YAML Spec) +在 `playbooks/roles/site_migration/defaults/main.yml` 中定义全局单向数据流配置： + +```yaml +# 迁移数据流方向声明 +migration_flow: + source: + host: "install.svc.plus" + domain_base: "svc.plus" + ssh_user: "root" + target: + host: "{{ target_env_ip }}" # IAC 流水线动态传入 + domain_base: "{{ target_domain }}" # IAC 流水线动态传入 (例如 contabo-xyz.svc.plus) + ssh_user: "root" + + # 定义需要迁移的数据模块 + modules: + - name: "ai_core_db" + type: "postgresql" + port: 15432 + databases: ["account", "litellm", "openclaw", "qmd", "rag", "notification", "scheduler", "audit", "artifact", "vault_storage"] + + - name: "zitadel_db" + type: "postgresql_docker" + container: "zitadel-db-1" + + - name: "gitea_data" + type: "gitea" + pg_port: 5434 + data_dirs: ["/var/lib/gitea/data"] + + - name: "caddy_configs" + type: "file_sync" + paths: ["/etc/caddy/conf.d/"] + # 核心特性：自动将 source 的域名（如 api.svc.plus）替换为 target 的域名（如 api.contabo.svc.plus） + domain_rewrite: true +``` + +### 2. 迁移与恢复 Ansible Role +#### [NEW] `playbooks/roles/site_migration/` +创建用于执行全站备份、传输与恢复的专用 Ansible 角色。 +- **数据提取 (Extract)**: 登录源环境，并行执行数据库 `pg_dump`，打包 Gitea 文件系统和 Caddy 配置。 +- **动态转换 (Transform)**: 利用 Ansible `replace` 或 `sed` 模块，在恢复前对 Caddy 配置和数据库 Dump 中的老域名 (`svc.plus`) 进行全局替换，实现域名参数化。 +- **装载恢复 (Load)**: 通过 `synchronize` (rsync) 单向推送到新环境，执行 `psql` 导入与 `systemctl/docker` 重启。 + +#### [NEW] `playbooks/migrate_site.yml` +作为入口 Playbook，加载上述 role，执行单向数据流迁移。 + +### 3. CI/CD IaC 流水线 +#### [NEW] `playbooks/.github/workflows/deploy-env-migration.yaml` +参考现有的 `deploy-ai-workspace-iac.yaml`，我们将在 `playbooks/.github/workflows/` 下新建一条流水线。该流水线的核心逻辑为： +1. **Provision (Terraform)**: 接收输入参数（如新大区、配置），调用底层 IaC 申请新机器并渲染出新的 IP 与域名。 +2. **Deploy (Ansible)**: 对新机器执行全新的基础环境构建 (run-on-host-bootstrap)。 +3. **Data Sync (Ansible Migration)**: 新环境 ready 后，以原生产环境为 source，新机器为 target，调用 `migrate_site.yml` 将数据和配置全量平滑克隆过去。 + +--- + +## User Review Required + +> [!IMPORTANT] +> 这是一个高危 / 核心架构级的操作，请审阅以下问题： + +1. **域名替换风险**: 直接在 SQL dump 和 Caddy 配置中查找替换 `domain_base`（如把 `svc.plus` 换成 `xxx.svc.plus`）是否满足你的预期？（注意某些数据库字段中可能包含硬编码的老域名）。 +2. **密钥库同步**: Vault Storage 的数据也会被原封不动克隆到新环境，这意味着新环境使用的解密密钥 (Unseal keys) 必须和原环境保持一致，这符合你的要求吗？ +3. **Pipeline 位置**: 你要求在 `playbooks/.github/workflows/` 下建立流水线。这是否意味着 playbooks 目录本身作为一个独立的 Git 仓库进行 Action 触发，还是随 `ai-workspace-infra` 整体触发？ + +如果设计方向无误，请批准此计划，我将开始为你生成 YAML 模板、Ansible Role 和 Pipeline 文件。 diff --git a/docs/BackUP/Site-Migration/walkthrough.md b/docs/BackUP/Site-Migration/walkthrough.md new file mode 100644 index 0000000..8f04bf6 --- /dev/null +++ b/docs/BackUP/Site-Migration/walkthrough.md @@ -0,0 +1,57 @@ +# 站点迁移与数据同步实施总结 (Walkthrough) + +我已经按照你确认的设计思路，完整实现了**全站单向数据流迁移与环境复制体系**。这一体系完全支持多环境部署，并且以声明式、参数化的方式自动改写域名（例如将 `svc.plus` 迁移至 `onwalk.net`）。 + +--- + +## 核心实现结构 + +### 1. 声明式配置模板 (YAML) +**文件**: [defaults/main.yml](file:///Users/shenlan/workspaces/ai-workspace-infra/playbooks/roles/site_migration/defaults/main.yml) + +在这里，我为你定义了基于数据流方向的全局变量 `migration_flow`。其中 `source.domain_base` 和 `destination.domain_base` 是完全参数化的，可以通过流水线按需覆盖。配置中清晰地拆分了 `ai_core_db`、`zitadel_db`、`gitea_data` 以及 `caddy_configs` 等四大模块，后续只需增删此处的 List 即可扩展迁移内容，无需改动底层逻辑代码。 + +### 2. 核心执行引擎 (Ansible Role) +**角色目录**: [roles/site_migration](file:///Users/shenlan/workspaces/ai-workspace-infra/playbooks/roles/site_migration/) +**入口剧本**: [migrate_site.yml](file:///Users/shenlan/workspaces/ai-workspace-infra/playbooks/migrate_site.yml) + +该角色分为两阶段执行： +- **Phase 1 (Extract)**: 登录 `migration_source` 机器，根据 YAML 配置执行并发的 `pg_dump`，并自动打包需要同步的文件路径。最终调用 Ansible 的拉取机制 (`fetch/synchronize pull`)，将所有制品安全缓存到执行机 (Runner)。 +- **Phase 2 (Load)**: 将制品通过推送机制 (`synchronize push`) 发送到 `migration_target` 机器上。执行阶段会自动拦截含有 `domain_rewrite: true` 的制品模块，利用流式替换 (`sed` 替换 SQL dumpfile、Caddy 配置文件内容以及重命名文件名），彻底将类似 `svc.plus` 冲刷为 `onwalk.net`。随后按需恢复并自动重启 Gitea, Caddy 及相关的 Docker 服务。 + +### 3. CI/CD IaC 协同流水线 +**流水线**: [.github/workflows/deploy-env-migration.yaml](file:///Users/shenlan/workspaces/ai-workspace-infra/playbooks/.github/workflows/deploy-env-migration.yaml) + +在 Github Actions 中，我为你搭建了一条包含了 `workflow_dispatch` 的新流水线。你可以直接在 UI 上填入 `source_domain_base`（如 `svc.plus`）和 `target_domain_base`（如 `onwalk.net`）。 +流水线完整编排了： +1. **Provision**: 调用底层 Terraform 申请新服务器。 +2. **Deploy Base**: 执行最初的 Bootstrap 环境搭建。 +3. **Data Migration**: 生成动态的 Ansible inventory，随后触发上述编写的 `migrate_site.yml` 完成旧环境数据与配置克隆。 + +### 4. 运行前置条件 (Vault OIDC 授权) + +由于我们新建了独立的 `site-recovery` 仓库，该仓库需要通过 GitHub Actions 的 OIDC 身份认证到 Vault 获取凭证。在首次运行流水线之前，你必须使用拥有管理员权限的凭据在你的终端（或 Vault 所在的 `vault.svc.plus` 主机）执行以下操作，以授权新的流水线拉取密钥： + +```bash +export VAULT_ADDR=https://vault.svc.plus +export VAULT_TOKEN="hvs.xxxxxxxxx" # 请替换为你拥有管理员权限的真实 Token + +vault write auth/jwt/role/github-actions-site-recovery - < [!TIP] +> 如果你在已经申请好并构建好的机器之间进行单纯的“灾备演练”或“环境数据同步”，你可以取消勾选流水线中的 `run_provision_and_deploy`，流水线将直接执行最核心的**数据克隆阶段**，速度会非常快。 diff --git a/docs/BackUP/backup_dr_plan.md b/docs/BackUP/backup_dr_plan.md new file mode 100644 index 0000000..66c1571 --- /dev/null +++ b/docs/BackUP/backup_dr_plan.md @@ -0,0 +1,215 @@ +# 全站统一备份与灾备方案 (AI Workspace) | Full-Site Unified Backup & DR Plan + +*🇨🇳 中文版在下方 | Chinese version below* + +--- + +# English Version + +This document is based on the actual deployment of the `install.svc.plus` (or `root@install.svc.plus`) production environment. It outlines the global Disaster Recovery (DR) standard operating procedures. In addition to the core PostgreSQL databases, this plan covers all core stateful components required for AI Workspace to run (including application attachments, private images, VPN configurations, etc.). + +## 1. Core Stateful Components & Directory Checklist + +When performing cross-datacenter/cross-zone full-machine migrations, the integrity of the following data must be ensured. + +### 1.1 Database Assets + +| Instance Purpose | Runtime Mode | Listen Port/Address | Contained Business Databases | +| :--- | :--- | :--- | :--- | +| AI Core Services DB | Docker (`postgresql-svc-plus`) | `127.0.0.1:15432` | Account, Vault Storage, Artifact, LiteLLM, OpenClaw, QMD, RAG, Notification, Scheduler, Audit | +| Zitadel IAM DB | Docker (`zitadel-db-1`) | Container Internal Net | Zitadel IAM Data | +| Gitea Code DB | Native Systemd (PG 16) | `127.0.0.1:5434` | Gitea Metadata | + +### 1.2 Critical Files & Stateful Volumes + +> [!IMPORTANT] +> The following directories are the "flesh and blood" of the restored system. If not migrated, even if the database is restored, fatal errors will occur (e.g., 404 images, failed model pulls, disconnected networking). + +* **Gitea Repository & Object Data:** `/var/lib/gitea/data` + Stores all bare Git repositories, LFS large files, user avatars, and attachments. +* **Private Image Registry (Harbor / Registry):** `/data/registry` or mounted `/data/harbor` + Private container images. Without this, the new machine will fail to find closed-source base images when running `docker-compose pull` upon restart. +* **OpenClaw (AI Gateway) Workdir:** `openclaw workdir` (Corresponds to the physical mount point during deployment) + Contains the AI Gateway's local cache files, vectorized temporary data, or loaded model assets. +* **Distributed Networking & VPN Config:** `/etc/wireguard` + Contains the local node's WireGuard Private Key and Peers routing table. If lost, the new host will be unable to rejoin the original Xworkmate VPN star network, breaking all cross-cloud microservice communication. +* **Object Storage (MinIO/S3):** `/data/minio` (if enabled) + The large file storage foundation for modern cloud-native components (like Wiki, RAG). +* **Environment Secret Files:** `.env` files under each component's deployment directory (e.g., `/opt/ai-workspace/`). + +> [!NOTE] +> **Exception List:** `/etc/caddy/data` or `acme.json` (Let's Encrypt certificates). +> **Strategy:** No need to sync during DR drills or actual migrations. Simply point the DNS A record to the new IP before bringing the new host online, then start Caddy to trigger automatic online renewal. (Note Let's Encrypt's rate limit of 5 duplicate certificates per week; staging certificates are recommended for drills). + +## 2. Unified Backup & Sync Strategy + +### 2.1 Logical Database Backup (PostgreSQL) + +It is recommended to create a unified backup script `pg_backup.sh` in the host's `/opt/backup/` directory and add it to Cron: + +```bash +#!/bin/bash +BACKUP_DIR="/var/backups/postgresql" +DATE=$(date +"%Y%m%d_%H%M%S") +mkdir -p $BACKUP_DIR + +echo "[INFO] Backing up AI Core Services DB (Port 15432)..." +DB_URL_BASE="postgres://svcplus_vps:@127.0.0.1:15432" +for DB in account litellm openclaw qmd rag notification scheduler audit artifact vault_storage; do + pg_dump "$DB_URL_BASE/$DB?sslmode=disable" | gzip > "$BACKUP_DIR/${DB}_$DATE.sql.gz" +done + +echo "[INFO] Backing up Zitadel IAM DB (zitadel-db-1)..." +docker exec zitadel-db-1 pg_dumpall -U postgres | gzip > "$BACKUP_DIR/zitadel_$DATE.sql.gz" + +echo "[INFO] Backing up Gitea Code DB (Native PG 16)..." +sudo -u postgres pg_dumpall -p 5434 | gzip > "$BACKUP_DIR/gitea_$DATE.sql.gz" + +# Clean up backups older than 7 days +find $BACKUP_DIR -type f -name "*.sql.gz" -mtime +7 -delete +``` + +### 2.2 File State Sync (Rsync Incremental) + +For file resources, we recommend pushing to the DR node using `rsync` via Ansible playbooks, or scheduling syncs to centralized object storage. Key `rsync` paths: + +```bash +rsync -avz --delete /var/lib/gitea/data/ backup-server:/backup/gitea/ +rsync -avz --delete /etc/wireguard/ backup-server:/backup/wireguard/ +rsync -avz --delete /data/registry/ backup-server:/backup/registry/ +``` + +## 3. Disaster Recovery & Restore Operations + +When a full machine crash or datacenter network outage requires activating the DR node, strictly restore in the following tiers: + +**Stage 1: Restore VPN & Network Tier** +Before installing any services, first restore the `/etc/wireguard` directory and start `wg-quick up wg0` to ensure the DR node re-acquires its original cluster internal IP. + +**Stage 2: Restore Object & File Persistence Tier** +Restore the backed-up Gitea, OpenClaw workdir, and image registry directories to their absolute paths on the host, and fix ownership permissions (e.g., `chown -R 1000:1000 /var/lib/gitea`). + +**Stage 3: Restore PostgreSQL Database Tier** +Ensure ONLY the database containers are started (keep application layer containers stopped). + +```bash +# 1. Restore AI Core DB +gunzip -c /var/backups/postgresql/account_YYYYMMDD.sql.gz | psql "postgres://...:15432/account" + +# 2. Restore Zitadel DB +gunzip -c /var/backups/postgresql/zitadel_YYYYMMDD.sql.gz | docker exec -i zitadel-db-1 psql -U postgres + +# 3. Restore Gitea DB +gunzip -c /var/backups/postgresql/gitea_YYYYMMDD.sql.gz | sudo -u postgres psql -p 5434 +``` + +**Stage 4: Start Ingress & Business Applications** +1. Confirm global DNS has been switched to the current DR node's public IP. +2. Start the Caddy service. The system will automatically initiate HTTP-01 challenges to apply for fresh certificates. +3. Bring everything fully online with `docker-compose up -d` to pull up all business line applications. + +--- + +# 中文版 (Chinese Version) + +本文档基于 `install.svc.plus` (或 `root@install.svc.plus`) 线上环境的实际部署情况，为你规划和整理了全局的灾备（DR）标准操作流程。 除了核心的 PostgreSQL 数据库外，本方案覆盖了 AI Workspace 运行所需的所有核心状态组件（含应用附件、私有镜像、VPN 配置等）。 + +## 1. 核心状态组件与目录清单 + +在进行跨机房/跨区整机迁移时，必须确保以下数据的完整迁移。 + +### 1.1 数据库资产 + +| 实例用途 | 运行模式 | 监听端口/地址 | 包含的业务系统库 | +| :--- | :--- | :--- | :--- | +| AI 核心服务库 | Docker (`postgresql-svc-plus`) | `127.0.0.1:15432` | Account, Vault Storage, Artifact, LiteLLM, OpenClaw, QMD, RAG, Notification, Scheduler, Audit | +| Zitadel 认证库 | Docker (`zitadel-db-1`) | 容器内部网 | Zitadel IAM 数据 | +| Gitea 代码库 | Native Systemd (PG 16) | `127.0.0.1:5434` | Gitea 元数据 | + +### 1.2 关键文件与持久化挂载卷 (Stateful Volumes) + +> [!IMPORTANT] +> 以下目录是系统恢复后的“血肉”，如果不迁移，即使数据库还原，也会出现图片 404、模型拉取失败、组网断开等致命故障。 + +* **Gitea 仓库与对象数据:** `/var/lib/gitea/data` + 存储所有的 Git 裸仓库、LFS 大文件、用户头像及附件。 +* **应用私有镜像库 (Harbor / Registry):** `/data/registry` 或挂载的 `/data/harbor` + 私有容器镜像。如果不迁移，新机在重启并执行 `docker-compose pull` 时将找不到未开源的基础环境镜像。 +* **OpenClaw (AI 网关) 工作目录:** `openclaw workdir` (对应部署时的物理挂载点) + 包含 AI Gateway 的本地缓存档、向量化临时数据或加载的模型资产。 +* **分布式组网与 VPN 配置:** `/etc/wireguard` + 包含本地节点的 WireGuard Private Key 与 Peers 路由表。若丢失此目录，新主机启动后将无法重新融入原有的 Xworkmate VPN 星型网络，所有跨云微服务通信将中断。 +* **对象存储 (MinIO/S3):** `/data/minio` (若开启) + 现代云原生组件（如 Wiki、RAG）的大文件存储底座。 +* **环境密钥文件:** 各组件部署目录下的 `.env` (如 `/opt/ai-workspace/`)。 + +> [!NOTE] +> **例外清单：** `/etc/caddy/data` 或 `acme.json`（Let's Encrypt 证书）。 +> **处理策略：** 灾备演练或真实迁移时无需同步。只需在新主机上线时，提前将 DNS A 记录解析至新 IP，再启动 Caddy 即可实现全自动在线续签。（需注意 Let's Encrypt 每周 5 次的重复签发限频，演练时建议使用 Staging 证书）。 + +## 2. 统一备份与同步策略 (Backup & Sync) + +### 2.1 数据库逻辑备份 (PostgreSQL) + +建议在宿主机的 `/opt/backup/` 目录下创建统一备份脚本 `pg_backup.sh` 并加入 Cron 定时任务： + +```bash +#!/bin/bash +BACKUP_DIR="/var/backups/postgresql" +DATE=$(date +"%Y%m%d_%H%M%S") +mkdir -p $BACKUP_DIR + +echo "[INFO] 备份 AI 核心服务库 (15432 端口)..." +DB_URL_BASE="postgres://svcplus_vps:@127.0.0.1:15432" +for DB in account litellm openclaw qmd rag notification scheduler audit artifact vault_storage; do + pg_dump "$DB_URL_BASE/$DB?sslmode=disable" | gzip > "$BACKUP_DIR/${DB}_$DATE.sql.gz" +done + +echo "[INFO] 备份 Zitadel 认证库 (zitadel-db-1)..." +docker exec zitadel-db-1 pg_dumpall -U postgres | gzip > "$BACKUP_DIR/zitadel_$DATE.sql.gz" + +echo "[INFO] 备份 Gitea 代码库 (Native PG 16)..." +sudo -u postgres pg_dumpall -p 5434 | gzip > "$BACKUP_DIR/gitea_$DATE.sql.gz" + +# 清理 7 天前的旧备份 +find $BACKUP_DIR -type f -name "*.sql.gz" -mtime +7 -delete +``` + +### 2.2 文件状态同步 (Rsync 增量同步) + +针对文件资源，建议使用 `rsync` 结合 Ansible 剧本向灾备节点推送，或定时同步到集中式对象存储。 关键的 `rsync` 路径示例： + +```bash +rsync -avz --delete /var/lib/gitea/data/ backup-server:/backup/gitea/ +rsync -avz --delete /etc/wireguard/ backup-server:/backup/wireguard/ +rsync -avz --delete /data/registry/ backup-server:/backup/registry/ +``` + +## 3. 灾难恢复与还原操作 (Restore) + +当发生整机崩溃或机房断网需要启用灾备节点时，严格按以下层级恢复： + +**阶段 1: 恢复 VPN 与网络层** +在安装任何服务前，首先将 `/etc/wireguard` 目录还原，并启动 `wg-quick up wg0`，确保灾备节点重新获取原集群内网 IP。 + +**阶段 2: 恢复对象与文件持久层** +将备份好的 Gitea、OpenClaw workdir、应用镜像目录等还原至宿主机的绝对路径中，并修复属主权限（如 `chown -R 1000:1000 /var/lib/gitea`）。 + +**阶段 3: 恢复 PostgreSQL 数据库层** +确保仅启动数据库容器（停掉应用层容器）。 + +```bash +# 1. 还原 AI 核心库 +gunzip -c /var/backups/postgresql/account_YYYYMMDD.sql.gz | psql "postgres://...:15432/account" + +# 2. 还原 Zitadel 库 +gunzip -c /var/backups/postgresql/zitadel_YYYYMMDD.sql.gz | docker exec -i zitadel-db-1 psql -U postgres + +# 3. 还原 Gitea 库 +gunzip -c /var/backups/postgresql/gitea_YYYYMMDD.sql.gz | sudo -u postgres psql -p 5434 +``` + +**阶段 4: 启动接入层与业务应用** +1. 确认全网 DNS 已切换至当前灾备节点公网 IP。 +2. 启动 Caddy 服务，系统会自动发起 HTTP-01 质询申请全新证书。 +3. 全面上线 `docker-compose up -d` 拉起各业务线应用。 diff --git a/docs/BackUP/postgresql_disaster_recovery.md b/docs/BackUP/postgresql_disaster_recovery.md new file mode 100644 index 0000000..16e991a --- /dev/null +++ b/docs/BackUP/postgresql_disaster_recovery.md @@ -0,0 +1,201 @@ +# 统一数据库备份与灾备方案 (PostgreSQL) | Unified DB Backup & DR Plan + +*🇨🇳 中文版在下方 | Chinese version below* + +--- + +# English Version + +This document is based on the actual deployment of the `install.svc.plus` production environment. It outlines the standard operating procedures for the global PostgreSQL database backup, restoration, and Disaster Recovery (DR). + +## 1. Production Database Architecture Checklist + +Based on the production environment reconnaissance, there are currently **three independently running PostgreSQL instances**. When performing unified backups, data must be exported from these three instances separately. + +| Instance Purpose | Runtime Mode | Listen Port/Address | Contained Business Databases | +| :--- | :--- | :--- | :--- | +| **AI Core Services DB** | Docker (`postgresql-svc-plus`) | `127.0.0.1:5432` | Account, Vault Storage, Artifact, LiteLLM, OpenClaw, QMD, RAG, Notification, Scheduler, Audit | +| **Zitadel IAM DB** | Docker (`zitadel-db-1`) | Container Internal Net | Zitadel IAM Data | +| **Gitea Code DB** | Native Systemd (PG 16) | `127.0.0.1:5434` | Gitea | + +## 2. Unified Backup Strategy + +To achieve seamless disaster recovery, we recommend a strategy combining **Logical Backup** with **Offsite Sync**. + +### 2.1 Global Backup Script + +Create a unified backup script `pg_backup.sh` in the `/opt/backup/` directory of the host (`install.svc.plus`): + +```bash +#!/bin/bash +# ========================================== +# Unified PostgreSQL Backup Script +# ========================================== +BACKUP_DIR="/var/backups/postgresql" +DATE=$(date +"%Y%m%d_%H%M%S") +mkdir -p $BACKUP_DIR + +echo "[INFO] Backing up AI Core Services databases (Port 15432)..." +DB_URL_BASE="postgres://svcplus_vps:@127.0.0.1:15432" +for DB in account litellm openclaw qmd rag notification scheduler audit artifact vault_storage; do + pg_dump "$DB_URL_BASE/$DB?sslmode=disable" | gzip > "$BACKUP_DIR/${DB}_$DATE.sql.gz" +done + +echo "[INFO] Backing up Zitadel IAM DB (zitadel-db-1)..." +docker exec zitadel-db-1 pg_dumpall -U postgres | gzip > "$BACKUP_DIR/zitadel_$DATE.sql.gz" + +echo "[INFO] Backing up Gitea Code DB (Native PG 16)..." +sudo -u postgres pg_dumpall -p 5434 | gzip > "$BACKUP_DIR/gitea_$DATE.sql.gz" + +echo "[INFO] Cleaning up backups older than 7 days..." +find $BACKUP_DIR -type f -name "*.sql.gz" -mtime +7 -delete + +echo "[INFO] Backup complete!" +``` + +> [!TIP] +> It is highly recommended to push backup files to an offsite S3 storage (like MinIO) or sync via `rclone` to other cloud storage to prevent datacenter-level disasters. + +### 2.2 Cron Schedule + +Add an automatic backup job at 3:00 AM daily via `crontab -e`: +```cron +0 3 * * * /bin/bash /opt/backup/pg_backup.sh >> /var/log/pg_backup.log 2>&1 +``` + +## 3. Disaster Recovery & Restore Operations + +When data corruption or full-machine migration occurs, please strictly follow this restore sequence. + +> [!WARNING] +> Before executing a restore, you must first stop the business services generating write traffic (like Gitea, LiteLLM, etc.), keeping ONLY the PostgreSQL processes running. + +### 3.1 Restore AI Core Services DB (Port 15432) +Includes: `Account`, `Vault Storage`, `Artifact`, `LiteLLM`, `OpenClaw`, `QMD`, `RAG`, `Notification`, `Scheduler`, `Audit` +```bash +# Unzip backup and import directly via credential URI (Using account as an example) +gunzip -c /var/backups/postgresql/account_YYYYMMDD.sql.gz | psql "postgres://svcplus_vps:@127.0.0.1:15432/account?sslmode=disable" +``` + +### 3.2 Restore Zitadel IAM DB (Docker) +```bash +# Unzip backup and import +gunzip -c /var/backups/postgresql/zitadel_YYYYMMDD.sql.gz | docker exec -i zitadel-db-1 psql -U postgres +``` + +### 3.3 Restore Gitea Code DB (Native) +Since Gitea is on the physical host, we need to switch to the `postgres` user: +```bash +# Unzip backup and import (Note port 5434) +gunzip -c /var/backups/postgresql/gitea_YYYYMMDD.sql.gz | sudo -u postgres psql -p 5434 +``` + +## 4. High Availability & Advanced Recommendations + +For the current single-node multi-instance architecture, logical backups (`pg_dumpall`) are sufficient for most scenarios, but the RPO (Recovery Point Objective) is approximately 24 hours (depending on backup frequency). + +> [!IMPORTANT] +> **Recommendations for upgrading DR tier:** +> 1. **Incremental Backups (WAL Archiving)**: For mission-critical databases like AI Core Services and Gitea, introduce `pgBackRest` or `WAL-G` to continuously archive physical WAL logs to S3 (MinIO) in real-time. This reduces the risk of data loss (RPO) to minutes. +> 2. **Application-Level DR**: +> - Vault Storage decryption keys cannot solely exist in the database. Ensure Vault's physical Root Token / Unseal Keys are safely stored offline. +> - For Gitea, in addition to backing up the database, you must periodically back up `/var/lib/gitea/data` (bare Git repositories and LFS artifacts) on the host. Using Gitea's built-in `gitea dump` command is highly recommended to achieve an integrated backup of both data and repositories. + +--- + +# 中文版 (Chinese Version) + +本文档基于 `install.svc.plus` 线上环境的实际部署情况，为你规划和整理了全局 PostgreSQL 数据库的备份、还原与灾备（DR）标准操作流程。 + +## 1. 线上环境数据库架构清单 + +根据线上环境探勘，当前系统存在 **三个独立运行的 PostgreSQL 实例**。在进行统一备份时，必须分别对这三个实例进行数据导出。 + +| 实例用途 | 运行模式 | 监听端口/地址 | 包含的业务系统库 | +| :--- | :--- | :--- | :--- | +| **AI 核心服务库** | Docker (`postgresql-svc-plus`) | `127.0.0.1:5432` | Account, Vault Storage, Artifact, LiteLLM, OpenClaw, QMD, RAG, Notification, Scheduler, Audit | +| **Zitadel 认证库** | Docker (`zitadel-db-1`) | 容器内部网 | Zitadel IAM 数据 | +| **Gitea 代码库** | Native Systemd (PG 16) | `127.0.0.1:5434` | Gitea | + +## 2. 统一备份策略 (Backup) + +为了实现无缝灾备，建议采用 **逻辑备份 (Logical Backup)** 结合 **定时异地转储 (Offsite Sync)** 的策略。 + +### 2.1 编写全局备份脚本 + +在宿主机 (`install.svc.plus`) 的 `/opt/backup/` 目录下创建统一备份脚本 `pg_backup.sh`： + +```bash +#!/bin/bash +# ========================================== +# 统一 PostgreSQL 备份脚本 +# ========================================== +BACKUP_DIR="/var/backups/postgresql" +DATE=$(date +"%Y%m%d_%H%M%S") +mkdir -p $BACKUP_DIR + +echo "[INFO] 开始备份 AI 核心服务各业务库 (15432 端口)..." +DB_URL_BASE="postgres://svcplus_vps:@127.0.0.1:15432" +for DB in account litellm openclaw qmd rag notification scheduler audit artifact vault_storage; do + pg_dump "$DB_URL_BASE/$DB?sslmode=disable" | gzip > "$BACKUP_DIR/${DB}_$DATE.sql.gz" +done + +echo "[INFO] 开始备份 Zitadel 认证库 (zitadel-db-1)..." +docker exec zitadel-db-1 pg_dumpall -U postgres | gzip > "$BACKUP_DIR/zitadel_$DATE.sql.gz" + +echo "[INFO] 开始备份 Gitea 代码库 (Native PG 16)..." +sudo -u postgres pg_dumpall -p 5434 | gzip > "$BACKUP_DIR/gitea_$DATE.sql.gz" + +echo "[INFO] 清理 7 天前的旧备份..." +find $BACKUP_DIR -type f -name "*.sql.gz" -mtime +7 -delete + +echo "[INFO] 备份完成！" +``` + +> [!TIP] +> 建议将备份文件推送到异地 S3 (例如 MinIO) 或通过 `rclone` 同步到其他云存储，以防范机房级灾难。 + +### 2.2 配置定时任务 (Cron) + +通过 `crontab -e` 添加每天凌晨 3 点自动备份： +```cron +0 3 * * * /bin/bash /opt/backup/pg_backup.sh >> /var/log/pg_backup.log 2>&1 +``` + +## 3. 灾难恢复与还原 (Restore) + +当发生数据损坏或整机迁移时，请严格按照以下顺序进行恢复。 + +> [!WARNING] +> 在执行还原前，必须先停止产生写入流量的业务服务（如 Gitea, LiteLLM 等），仅保留 PostgreSQL 进程运行。 + +### 3.1 还原 AI 核心服务库 (15432 端口) +包含：`Account`, `Vault Storage`, `Artifact`, `LiteLLM`, `OpenClaw`, `QMD`, `RAG`, `Notification`, `Scheduler`, `Audit` +```bash +# 解压备份文件并使用凭证直接导入 (以 account 为例) +gunzip -c /var/backups/postgresql/account_YYYYMMDD.sql.gz | psql "postgres://svcplus_vps:@127.0.0.1:15432/account?sslmode=disable" +``` + +### 3.2 还原 Zitadel 认证库 (Docker) +```bash +# 解压备份文件并导入 +gunzip -c /var/backups/postgresql/zitadel_YYYYMMDD.sql.gz | docker exec -i zitadel-db-1 psql -U postgres +``` + +### 3.3 还原 Gitea 代码库 (Native) +由于 Gitea 在宿主机物理机上，我们需要切换到 postgres 用户执行： +```bash +# 解压备份文件并导入 (注意端口 5434) +gunzip -c /var/backups/postgresql/gitea_YYYYMMDD.sql.gz | sudo -u postgres psql -p 5434 +``` + +## 4. 高可用与进阶建议 (Disaster Recovery) + +对于目前的单机多实例架构，逻辑备份（`pg_dumpall`）足够应付大多数场景，但存在 RPO (恢复点目标) 约为 24 小时的问题（取决于备份频率）。 + +> [!IMPORTANT] +> **提升容灾级别的建议：** +> 1. **增量备份 (WAL Archiving)**: 针对业务极其核心的 AI 核心服务库和 Gitea，可引入 `pgBackRest` 或 `WAL-G`，将物理 WAL 日志实时归档至 S3（MinIO）。这样可以将数据丢失风险（RPO）降低到分钟级别。 +> 2. **应用级灾备**: +> - Vault Storage 的加解密密钥不能仅存在数据库中，请确保 Vault 的物理 Root Token/Unseal Keys 已被安全离线保存。 +> - Gitea 除了备份数据库，还必须定期备份宿主机上的 `/var/lib/gitea/data` (Git 仓库裸数据和 LFS 制品)。可以借助 Gitea 自带的 `gitea dump` 命令来实现数据 + 仓库的一体化备份。 diff --git a/docs/Systems-Overview/PROD/live_systems_overview.md b/docs/Systems-Overview/PROD/live_systems_overview.md new file mode 100644 index 0000000..5cd92ff --- /dev/null +++ b/docs/Systems-Overview/PROD/live_systems_overview.md @@ -0,0 +1,141 @@ +# 线上环境全业务系统拓扑梳理 (Caddy) | Live Systems Overview (Caddy) + +*🇨🇳 中文版在下方 | Chinese version below* + +--- + +# English Version + +By reverse-engineering the configurations under `/etc/caddy/conf.d/` on the `install.svc.plus` node, I have outlined all business systems currently running in the production environment, their corresponding domains, and internal port mappings. + +## 1. Core AI Infrastructure + +| Business System | Exposed Domain | Internal Upstream | Description | +| :--- | :--- | :--- | :--- | +| API Gateway | `apigateway.svc.plus` | `127.0.0.1:9080` | Universal API Gateway | +| LiteLLM (API) | `api.svc.plus` | `127.0.0.1:4000` | Unified model routing & OpenAI/Anthropic path translation layer | +| LiteLLM (UI) | `litellm.svc.plus` | `127.0.0.1:4000` | LiteLLM Admin UI / Dashboard | +| OpenClaw (Bot) | `openclaw.svc.plus` | `127.0.0.1:18789` | OpenClaw chatbot backend service | +| RAG Server | `rag-server.svc.plus`
`rag-server-contabo-*.svc.plus` | `127.0.0.1:18084` | RAG (Retrieval-Augmented Generation) backend | +| XWorkmate Bridge | `xworkmate-bridge.svc.plus` | `127.0.0.1:8787` | Workflow bridge with Bearer Token strict authentication | + +## 2. DevOps & Tools + +| Business System | Exposed Domain | Internal Upstream | Description | +| :--- | :--- | :--- | :--- | +| Gitea | `gitea.svc.plus` | `localhost:3001` | Private Git code hosting platform | +| Vault | `vault.svc.plus` | `127.0.0.1:8200` | Unified credential and secret persistence service | +| ~~Code Server~~ | ~~`observability.svc.plus/code/`~~ | ~~`127.0.0.1:8443`~~ | ⚠️ **[To Be Deprecated]** Web-based VSCode development environment | +| ~~Jupyter Lab~~ | ~~`observability.svc.plus/jupyter/`~~ | ~~`127.0.0.1:8888`~~ | ⚠️ **[To Be Deprecated]** Data science Jupyter environment | + +## 3. IAM & Accounts + +| Business System | Exposed Domain | Internal Upstream | Description | +| :--- | :--- | :--- | :--- | +| Zitadel | `zitadel.svc.plus` | `127.0.0.1:19080/19081` | IAM Global SSO (API & UI) | +| Accounts | `accounts.svc.plus`
`accounts-contabo-*.svc.plus` | `127.0.0.1:18081` | Production account management service | +| ~~Accounts (Preview)~~ | ~~`accounts-preview.svc.plus`~~ | ~~`127.0.0.1:28081`~~ | ⚠️ **[To Be Deprecated]** Preview/Test account service (Will not be brought up on DR node) | + +## 4. Frontends + +| Business System | Exposed Domain | Internal Upstream | Description | +| :--- | :--- | :--- | :--- | +| Console | `console.svc.plus` | `127.0.0.1:3000` | AI Workspace main console dashboard | +| Docs | `docs.svc.plus`
`docs-contabo-*.svc.plus` | `127.0.0.1:18083` | System documentation (Reuses x-cloud-flow port) | +| Install Scripts | `install.svc.plus` | `302 Redir -> Github` | Short link distribution for curl-based one-click install scripts | +| Ebook | `ebook.svc.plus` | Static Files | Modern IT History ebook (`/opt/modern-it-history/current`) | + +## 5. Observability Stack + +`observability.svc.plus` acts as the global ingress for all monitoring components, routing by path: + +| Path Rule | Internal Upstream | Component / Purpose | +| :--- | :--- | :--- | +| `/grafana/*` | `127.0.0.1:3000` | Grafana visual dashboards (Also default root redirect) | +| `/ingest/metrics/*`
`/vmetrics/*` | `127.0.0.1:8428` | VictoriaMetrics (Prometheus metrics write & query) | +| `/ingest/logs/*`
`/vlogs/*` | `127.0.0.1:9428` | VictoriaLogs / Loki (Log write & query) | +| `/ingest/otlp/*`
`/vtraces/*` | `127.0.0.1:4318 / 10428` | OpenTelemetry distributed tracing | +| ~~`/insight/*`~~ | ~~`127.0.0.1:8082`~~ | ⚠️ **[To Be Deprecated]** Insight Workbench data analysis console | +| `/vmalert/*` | `127.0.0.1:8880` | VictoriaMetrics Alert engine | +| `/alertmgr/*` | `127.0.0.1:9059` | Alertmanager alert routing & dispatch | +| `/blackbox/*` | `127.0.0.1:9115` | Blackbox Exporter network probing | + +## 6. Other Underlying/Network Components + +| Business System | Exposed Domain | Internal Upstream | Description | +| :--- | :--- | :--- | :--- | +| ~~X-Cloud-Flow~~ | ~~`x-cloud-flow.svc.plus`~~ | ~~`127.0.0.1:18083/18087`~~ | ⚠️ **[To Be Deprecated]** Cloud traffic control or orchestration engine | +| ~~X-Ops-Agent~~ | ~~`x-ops-agent.svc.plus`~~ | ~~`127.0.0.1:18084/18086`~~ | ⚠️ **[To Be Deprecated]** Automated operations Agent | +| ~~X-Scope-Hub~~ | ~~`x-scope-hub.svc.plus`~~ | ~~`127.0.0.1:18085`~~ | ⚠️ **[To Be Deprecated]** Resource scoping hub | +| Hermes | `hermes.svc.plus` | `127.0.0.1:18180` | Message notification gateway | +| JP XHTTP / Xray | `jp-xhttp.svc.plus` | `/dev/shm/xray.sock` | Cross-border network proxy tunnel (gRPC) | +| PostgreSQL Tunnel | `postgresql-contabo...` | Static 200 Response | Network probe or specific proxy ingress for TLS handshakes | + +--- + +# 中文版 (Chinese Version) + +通过对 `install.svc.plus` 节点 `/etc/caddy/conf.d/` 下的配置进行逆向解析，我为你梳理出了线上环境中运行的所有业务系统、对应域名及内网端口映射。 + +## 1. 核心 AI 基础设施 (AI Infrastructure) + +| 业务系统 | 对外暴露域名 (Domain) | 内网代理目标 (Upstream) | 备注描述 | +| :--- | :--- | :--- | :--- | +| API Gateway | `apigateway.svc.plus` | `127.0.0.1:9080` | 通用 API 网关 | +| LiteLLM (API) | `api.svc.plus` | `127.0.0.1:4000` | 模型统一路由与 OpenAI/Anthropic 路径兼容转换层 | +| LiteLLM (UI) | `litellm.svc.plus` | `127.0.0.1:4000` | LiteLLM Admin UI / Dashboard | +| OpenClaw (Bot) | `openclaw.svc.plus` | `127.0.0.1:18789` | OpenClaw 聊天机器人后台服务 | +| RAG Server | `rag-server.svc.plus`
`rag-server-contabo-*.svc.plus` | `127.0.0.1:18084` | RAG (检索增强生成) 服务后端 | +| XWorkmate Bridge | `xworkmate-bridge.svc.plus` | `127.0.0.1:8787` | 带有 Bearer Token 强鉴权的工作流桥接器 | + +## 2. 开发者与协同工具 (DevOps & Tools) + +| 业务系统 | 对外暴露域名 (Domain) | 内网代理目标 (Upstream) | 备注描述 | +| :--- | :--- | :--- | :--- | +| Gitea | `gitea.svc.plus` | `localhost:3001` | 私有 Git 代码托管平台 | +| Vault | `vault.svc.plus` | `127.0.0.1:8200` | 统一凭证与密钥持久化服务 | +| ~~Code Server~~ | ~~`observability.svc.plus/code/`~~ | ~~`127.0.0.1:8443`~~ | ⚠️ **[计划下线]** 网页版 VSCode 开发环境 | +| ~~Jupyter Lab~~ | ~~`observability.svc.plus/jupyter/`~~ | ~~`127.0.0.1:8888`~~ | ⚠️ **[计划下线]** 数据科学 Jupyter 环境 | + +## 3. 身份认证与账号体系 (IAM & Accounts) + +| 业务系统 | 对外暴露域名 (Domain) | 内网代理目标 (Upstream) | 备注描述 | +| :--- | :--- | :--- | :--- | +| Zitadel | `zitadel.svc.plus` | `127.0.0.1:19080/19081` | IAM 全局单点登录 (API 与 UI) | +| Accounts | `accounts.svc.plus`
`accounts-contabo-*.svc.plus` | `127.0.0.1:18081` | 生产环境账户管理服务 | +| ~~Accounts (Preview)~~ | ~~`accounts-preview.svc.plus`~~ | ~~`127.0.0.1:28081`~~ | ⚠️ **[计划下线]** 预览 / 测试环境账户服务 (灾备节点不再拉起) | + +## 4. 控制台与前端应用 (Frontends) + +| 业务系统 | 对外暴露域名 (Domain) | 内网代理目标 (Upstream) | 备注描述 | +| :--- | :--- | :--- | :--- | +| Console | `console.svc.plus` | `127.0.0.1:3000` | AI Workspace 主控台面板 | +| Docs | `docs.svc.plus`
`docs-contabo-*.svc.plus` | `127.0.0.1:18083` | 系统帮助文档 (复用 x-cloud-flow 端口) | +| Install Scripts | `install.svc.plus` | `302 Redir -> Github` | 供 curl 拉取执行的一键安装脚本短链接分发 | +| Ebook | `ebook.svc.plus` | 静态文件 | Modern IT History 电子书 (`/opt/modern-it-history/current`) | + +## 5. 可观测性与日志监控 (Observability Stack) + +`observability.svc.plus` 为所有监控组件的全局入口，按 Path 转发到不同组件： + +| 路径规则 (Path) | 内网代理目标 (Upstream) | 对应组件 / 用途 | +| :--- | :--- | :--- | +| `/grafana/*` | `127.0.0.1:3000` | Grafana 可视化仪表盘 (也是根目录默认重定向) | +| `/ingest/metrics/*`
`/vmetrics/*` | `127.0.0.1:8428` | VictoriaMetrics (Prometheus 指标写入与查询) | +| `/ingest/logs/*`
`/vlogs/*` | `127.0.0.1:9428` | VictoriaLogs / Loki (日志写入与查询) | +| `/ingest/otlp/*`
`/vtraces/*` | `127.0.0.1:4318 / 10428` | OpenTelemetry 分布式链路追踪 | +| ~~`/insight/*`~~ | ~~`127.0.0.1:8082`~~ | ⚠️ **[计划下线]** Insight Workbench 数据分析台 | +| `/vmalert/*` | `127.0.0.1:8880` | VictoriaMetrics Alert 告警引擎 | +| `/alertmgr/*` | `127.0.0.1:9059` | Alertmanager 告警路由与发送 | +| `/blackbox/*` | `127.0.0.1:9115` | Blackbox Exporter 网络拨测 | + +## 6. 其他底层/网络组件 + +| 业务系统 | 对外暴露域名 (Domain) | 内网代理目标 (Upstream) | 备注描述 | +| :--- | :--- | :--- | :--- | +| ~~X-Cloud-Flow~~ | ~~`x-cloud-flow.svc.plus`~~ | ~~`127.0.0.1:18083/18087`~~ | ⚠️ **[计划下线]** 云上流控或编排引擎 | +| ~~X-Ops-Agent~~ | ~~`x-ops-agent.svc.plus`~~ | ~~`127.0.0.1:18084/18086`~~ | ⚠️ **[计划下线]** 自动化运维 Agent | +| ~~X-Scope-Hub~~ | ~~`x-scope-hub.svc.plus`~~ | ~~`127.0.0.1:18085`~~ | ⚠️ **[计划下线]** 资源范围枢纽 | +| Hermes | `hermes.svc.plus` | `127.0.0.1:18180` | 消息通知网关 (Notification) | +| JP XHTTP / Xray | `jp-xhttp.svc.plus` | `/dev/shm/xray.sock` | 跨境网络代理隧道 (gRPC) | +| PostgreSQL Tunnel | `postgresql-contabo...` | 静态 200 响应 | 仅作为 TLS 握手的网络探针或特定代理入口 |