From 3993f682bea5d8d415bc17f5154dc5bf0960685b Mon Sep 17 00:00:00 2001 From: Haitao Pan Date: Fri, 26 Jun 2026 23:18:04 +0800 Subject: [PATCH] docs(cases/06): definitive 4-layer chain incl. multi-session plugin + live verification MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rewrites the timeline (§1) and topology (§2) as the correct FOUR-layer chain App → bridge → openclaw-multi-session-plugins → OpenClaw gateway, and documents the plugin's multi-session/multi-thread role: session mapping (appThreadKey⇄openclawSessionKey), per-(session,run) artifactScope = tasks//, the strict sessionKey/runId/artifactScope triplet validation, and the expectedArtifactDirs workspace-root fallback scan. Live-verified against 127.0.0.1:8787 (plugin loaded, commit 2333c3e): - session.prepare returns a real mapping; chat.send returns runId; xworkmate.tasks.get is handled by the plugin but returns no_native_task_record with an empty task scope (chain reaches the plugin layer; the agent run produced no queryable task / no file — a layer-4 execution/landing issue). Adds §7 stability improvements grounded in this live run: - S0 install the plugin from a stable path (not /private/tmp) — the primary reliability fix. - S1 expectedArtifactDirs was [] → the plugin's workspace-root fallback is inert; bridge should always pass default dirs (reports/, artifacts/). - S2 no_native_task_record status ambiguity (running vs completed-without-artifact). - S3 sessionKey/runId/artifactScope triplet consistency (don't pre-prefix agent:main:). - S4 runtime observability across all four layers. Co-Authored-By: Claude Opus 4.8 --- ...6-gateway-turn-stability-and-robustness.md | 119 +++++++++++------- 1 file changed, 76 insertions(+), 43 deletions(-) diff --git a/docs/cases/06-gateway-turn-stability-and-robustness.md b/docs/cases/06-gateway-turn-stability-and-robustness.md index b970ca2c..bab4ca4e 100644 --- a/docs/cases/06-gateway-turn-stability-and-robustness.md +++ b/docs/cases/06-gateway-turn-stability-and-robustness.md @@ -17,71 +17,82 @@ --- -## 1. The full chain (one gateway turn) +## 1. The full chain (one gateway turn) — 四层，含 `openclaw-multi-session-plugins` -一次 gateway turn 的完整时序： +一次 gateway turn 跨**四层**：App → bridge(Go) → **openclaw-multi-session-plugins(TS 插件)** → OpenClaw gateway runtime。 +插件是关键中间层：它给「多会话/多线程」补上**逻辑隔离的 artifact scope** 与**会话映射 + 任务快照**，并以 `xworkmate.*` 网关方法暴露给 bridge。 ``` -App turn ──SSE POST──▶ Bridge http_handler ──▶ handleRequest ──▶ OpenClaw gateway submit - (executeTask) (text/event-stream) (阻塞 6min/60min) (WS 127.0.0.1:18789) - │ │ │ - │ keepalive 20s ◀────┘ ▼ - │ 返回 "running task handle"(runId + artifactScope) - ▼ │ - persist association ──▶ pollOpenClawTaskAssociationInternal (tasks.get 每 2s) - │ - isOpenClawRunningTaskHandle? ──yes──▶ persist + continue ──┐ - │ │ - no ──▶ applyGatewayChatResult (terminal) └─ ∞ (无 deadline) +App.executeTask ──SSE POST /acp/rpc──▶ Bridge.handleRequest + │ │ + │ ① xworkmate.session.prepare ───────▶ Gateway ──▶ [plugin] recordXWorkmateSessionMapping + │ (建 appThreadKey⇄openclawSessionKey 映射 + prepareXWorkmateArtifacts) + │ ◀── { artifactScope: tasks//, artifactDirectory, expectedArtifactDirs } + │ ② chat.send (gateway 原生) ────────▶ Gateway ──▶ 派发 agent run，返回 runId（~25ms，detached） + │ ◀── { runId, status: started } + ▼ + Bridge 记 sess.openClaw + DeadlineAt(budget by taskLoadClass)，返回 running 句柄 + │ + ▼ App 持久化 association，pollOpenClawTaskAssociationInternal 轮询： + ③ xworkmate.tasks.get(runId/openclawSessionKey/artifactScope) ─▶ Gateway ─▶ [plugin] getXWorkmateTaskSnapshot + ├─ resolveNativeTask(host task registry by runId) ──有──▶ 回 status(running/completed/failed) + 经 export 取 artifacts + └─ 无 native task ──▶ exportArtifactsForTaskLookup 兜底：扫 scope 目录 + expectedArtifactDirs（workspace 根 reports//artifacts/） + ├─ 有产物 ─▶ status=unknown, evidence=artifacts_present, 带 artifacts + └─ 无产物 ─▶ code=no_native_task_record / task_not_found + ④ 终态后取产物：xworkmate.artifacts.export / .list / .read（插件 exportXWorkmateArtifacts） + scopeRoot = workspaceRoot/tasks//；按 requiredArtifactExtensions(.md) 判 constraintSatisfied ``` -完成信号（completion signal）只活在**两个脆弱位置**： +**多会话/多线程隔离的核心（插件提供）**： +- `artifactScopeFor(sessionKey, runId)` = `tasks//`（冒号→下划线，如 `agent:main:draft:e2e` → `agent_main_draft_e2e`），每个 (会话,运行) 独立目录，互不串扰（`exportArtifacts.ts:126/164`）。 +- `recordXWorkmateSessionMapping` 把 `appThreadKey ⇄ openclawSessionKey` 持久成会话扩展（`taskState.ts`），让跨连接/重连仍能按 appThreadKey 找回 run。 +- `exportXWorkmateArtifacts` 严校 `requestedArtifactScope === artifactScopeFor(sessionKey,runId)`，不匹配抛 `artifactScope does not match sessionKey/runId`——**调用方必须传一致的 sessionKey/runId/artifactScope 三元组**（bridge 的 `taskGetParamsWithSessionScope` 负责从 session 记录补齐；外部手工探针易踩此坑）。 +- **workspace 根兜底扫描**：agent 常把产物写到 workspace 根的 `reports/`、`artifacts/` 而非 task scope；插件用 `expectedArtifactDirs` 回扫这些目录纳入产物（见 `openclaw-gateway-e2e-regression/ROOT_CAUSE_ANALYSIS.md` Fix 0）。 -1. **带内 SSE 结果帧**（result envelope / `[DONE]`）—— 依赖那条长连接活到任务结束； -2. **Gateway 对 `tasks.get` 的内存态应答** —— 依赖 bridge↔gateway 的 WS 连接仍持有该 run 的状态。 - -截图场景里这两条同时被打断，于是"任务跑完了，但客户端永远收不到终态"。 +**live 实测（2026-06-26，8787，插件已加载）**：① session.prepare 回真实 mapping ✓；② chat.send `runId=turn-…438450000` ✓（bridge 日志 `request_timing method=chat.send`）；③ tasks.get 经插件返回 `code=no_native_task_record`（mapping 在、但 gateway 无该 run 的 native task record）+ scope 目录空、无 `news.md`。即**链路打通到插件层**，但本轮 agent 未注册可查 task、也未落产物（属第 4 层 agent 执行/落盘问题，见 §4「残留」与 §7）。 关键代码锚点： | 环节 | 位置 | |---|---| -| App 发起 SSE turn | `xworkmate-app/lib/app/app_controller_desktop_thread_actions.dart:644` (`executeTask`) | -| Bridge SSE handler / keepalive 20s | `xworkmate-bridge/internal/acp/http_handler.go:198-283`、`:19` | -| Bridge 阻塞等待 gateway（6min 默认 / 60min 上限） | `xworkmate-bridge/internal/acp/orchestrator.go:31-33` | -| Bridge↔Gateway WebSocket（dial 18789） | `xworkmate-bridge/internal/gatewayruntime/runtime.go:372-376` | -| App 持久化 + 轮询 run | `app_controller_desktop_thread_actions.dart:680-693`、`:747` | -| running handle 判定 | `xworkmate-app/lib/runtime/go_task_service_client.dart:519` | -| 进度条 phase（仅看 pending） | `xworkmate-app/lib/widgets/assistant_task_progress_bar.dart:190` | +| App 发起 SSE turn / 轮询 | `xworkmate-app/.../app_controller_desktop_thread_actions.dart:644`、`:747` | +| Bridge session.prepare / tasks.get / cancel | `xworkmate-bridge/internal/acp/rpc_handler.go:96`、`:126`、`taskGetParamsWithSessionScope:177` | +| Bridge↔Gateway WebSocket（dial 18789） | `xworkmate-bridge/internal/gatewayruntime/runtime.go:372` | +| 插件注册 `xworkmate.*` 网关方法 | `openclaw-multi-session-plugins/index.ts:126/162/176/206/221`（session.prepare/tasks.get/artifacts.export/.list/.read）| +| 插件 artifact scope / 会话映射 / 任务快照 | `src/exportArtifacts.ts`、`src/taskState.ts:208 getXWorkmateTaskSnapshot` | --- -## 2. 三仓库端到端拓扑（实测自代码） +## 2. 端到端拓扑（实测自代码 + live 8787 验证） ``` -┌─ ai-workspace-lab/xworkmate-app (Flutter) -│ executeTask → SSE POST Accept: text/event-stream +┌─ ai-workspace-lab/xworkmate-app (Flutter) executeTask → SSE POST Accept: text/event-stream ▼ -┌─ ai-workspace-infra Caddy 入口 xworkmate-bridge.svc.plus -│ handle /acp* : flush_interval -1 ✓ read_timeout 30m write_timeout 30m keepalive 5m -│ handle /api* : (Caddy 默认超时, 无 flush_interval) ⚠ -│ handle / : (Caddy 默认) ⚠ +┌─ ai-workspace-infra Caddy 入口 xworkmate-bridge.svc.plus （本机直连 127.0.0.1:8787 不经 Caddy） +│ /acp* : flush_interval -1 ✓ read/write_timeout 70m(对齐 bridge 60min) keepalive 5m ← T1/T2 已修 ▼ -┌─ ai-workspace-lab/xworkmate-bridge (Go) -│ SSE handler keepalive 20s → handleRequest 阻塞 6min(默认)/60min(max) -│ gatewayruntime: WebSocket → 127.0.0.1:18789 (pending map 绑定在连接生命周期上) +┌─ ai-workspace-lab/xworkmate-bridge (Go) live commit 2333c3e +│ /acp/rpc handleRequest → session.prepare / chat.send / xworkmate.tasks.get / tasks.cancel +│ gatewayruntime: WebSocket → 127.0.0.1:18789；per-session 持久 run 仓(T7/T8/T9) +▼ (WS, gateway 原生 chat.send + 插件注册的 xworkmate.*) +┌─ ai-workspace-lab/openclaw-multi-session-plugins (TS 插件, enabled) ← 多会话/多线程 artifact 隔离层 +│ index.ts registerGatewayMethod: xworkmate.session.prepare / .tasks.get / .artifacts.export/.list/.read/.collect-and-snapshot +│ 会话映射(taskState) + artifactScope=tasks//(exportArtifacts) + workspace 根兜底扫描 +▼ (插件运行在网关进程内) +┌─ OpenClaw gateway runtime npm-global openclaw 2026.6.1 @ /opt/homebrew/lib/node_modules/openclaw +│ launchd ai.openclaw.gateway, WS 18789；host task registry(detached task) + agent 执行(deepseek-v4-flash) ▼ -┌─ ai-workspace-infra deploy_gateway_openclaw (roles/vhosts/gateway_openclaw) -│ OpenClaw gateway runtime, WS 18789 -▼ - OpenClaw 执行 → artifacts → tasks.get 回查 + workspace ~/.openclaw/workspace/tasks/// → 产物(.md 等) ``` -旁路（`ai-workspace-service`）：`accounts.svc.plus`（登录 / Token）、`console.svc.plus`（自带 openclaw assistant route）、`litellm` / `AI-Relay-Kit` / `codex-relay`（模型出口）、`qmd`（记忆）。 -它们不在本次卡死主链上，但 **Token 失效 / 出口异常会以同样的"连接中断"形态**表现出来，排查时需先用 `runId` 区分是主链断还是旁路断。 +**live 验证状态（2026-06-26 23:xx）**： +- ✅ 网关加载 `6 plugins … openclaw-multi-session-plugins`（重启后；详见 §4）。 +- ✅ `/api/ping` commit=2333c3e；session.prepare 回真实 mapping；chat.send 成功返回 runId。 +- ⚠️ `xworkmate.tasks.get` → `no_native_task_record`、scope 目录空：链路通到插件，但本轮 agent 未注册可查 task / 未落产物（第 4 层执行问题）。 -入口配置出处：`ai-workspace-infra/playbooks/roles/vhosts/xworkmate_bridge/templates/xworkmate-bridge-site.caddy.j2` -Gateway 部署出处：`ai-workspace-infra/playbooks/deploy_gateway_openclaw.yml` → `roles/vhosts/gateway_openclaw/` +旁路（`ai-workspace-service`）：`accounts.svc.plus`（登录/Token）、`console.svc.plus`（openclaw assistant route）、`litellm`/`AI-Relay-Kit`/`codex-relay`（模型出口）、`qmd`（记忆）——不在主链，但 Token/出口异常以「连接中断」形态出现，用 `runId` 区分。 + +出处：Caddy `…/xworkmate_bridge/templates/xworkmate-bridge-site.caddy.j2`；网关部署 `deploy_gateway_openclaw.yml`；插件 `~/.openclaw/extensions/openclaw-multi-session-plugins`（注册源当前指向临时 `/private/tmp/…`，见 §4 加固项）。 --- @@ -281,3 +292,25 @@ curl -sS -X POST http://127.0.0.1:8787/acp/rpc \ > 回归对照：本目录 `00-review-env-and-matrix.md` 第 2 节"通用验收标准"中"长任务执行期间状态流 / 取消 / 重试稳定""同一任务重复执行 3 次不卡死"，即本规划的回归出口。 > 产物交付链（artifact scope / workspace 路径）的独立缺陷与修复，见 `openclaw-gateway-e2e-regression/ROOT_CAUSE_ANALYSIS.md`。 + +--- + +## 7. 全链路稳定性改进（基于 2026-06-26 四层 live 验证） + +> 优先级按「直接决定一次任务能否产出」排序。S1/S2 是本轮 live 新发现。 + +- **S0 插件稳定安装（最高）** — 网关方法 `xworkmate.*` 全部依赖 `openclaw-multi-session-plugins`。当前注册源是临时路径 `/private/tmp/openclaw-multi-session-plugins/dist/index.js`，**重启/清 tmp 即丢插件**（本次故障正因网关启动早于该路径就位）。 + 改进：`openclaw plugins install ~/.openclaw/extensions/openclaw-multi-session-plugins`（或仓库稳定路径）落正式 install 记录；网关 supervisor 启动顺序保证「插件就位 → 启网关」。 + 验收：网关重启后启动日志稳定含 `… openclaw-multi-session-plugins`；`xworkmate.session.prepare` 不再 `unknown method`。 + +- **S1 `expectedArtifactDirs` 为空导致根目录兜底失效** — live 的 session mapping 为 `expectedArtifactDirs:[]`（contract 没派生出期望目录，`orchestrator.go:402/682`）。插件对「agent 把产物写到 workspace 根 `reports/`/`artifacts/` 而非 task scope」的兜底扫描**依赖 `expectedArtifactDirs`**；为空时兜底形同虚设 → 即便 agent 产出也收不到，表现「暂无文件」。 + 改进：bridge 在 `xworkmate.session.prepare` 必带一组缺省 `expectedArtifactDirs`（至少 `reports/`、`artifacts/`，并从 `requiredArtifactExtensions` / prompt 中的目标路径推导）；或插件在 `expectedArtifactDirs` 为空时回扫一组安全缺省目录。 + 验收：agent 写到 workspace 根的 `.md` 能被 `xworkmate.tasks.get/artifacts.export` 纳入产物。 + +- **S2 `no_native_task_record` 状态歧义** — `xworkmate.tasks.get` 的真值来自「gateway host task registry 有该 run 的 detached task」**或**「artifact 已存在」。live 中 chat.send 成功但 gateway 无 native task record（agent 可能以 inline chat 执行、未注册可查 task），且无产物 → 插件回 `no_native_task_record`，bridge 只能靠 T7 兜底续轮询到 deadline，**无法区分「还在跑」与「跑完没产物」**。 + 改进：①确认 gateway 侧 chat.send 是否应产出 detached task（agent 配置/ `tasks.*` 注册）；②插件/bridge 在 `no_native_task_record` 且超过最小执行时长时，下发更明确的 `running(no-record)` vs `completed(no-artifact)` 语义，配合 §5 T9 deadline 收口。 + 验收：agent 正常执行时 `tasks.get` 能返回真实 running→completed；异常时给确定终态而非无限 degraded。 + +- **S3 三元组一致性（已知约束）** — 插件严校 `sessionKey/runId/artifactScope` 三者一致（`exportArtifacts.ts:126`），且 bridge 的 openclawSessionKey 由 `agent:main:` + appThreadKey 组成。**调用方/探针不要预带 `agent:main:` 前缀**（否则双前缀 → `artifactScope does not match`）。bridge `taskGetParamsWithSessionScope` 已负责补齐；保持其为唯一可信来源，App/探针只传 `sessionId=draft:` + `runId`。 + +- **S4 运行态可观测** — 沿用 §5 T11/T12：bridge `/api/ping.commit`、网关 `N plugins` 列表、`openclaw plugins inspect` 三处纳入健康检查；`runId` 贯穿 App→bridge→插件→gateway 日志，便于定位断点落在四层中的哪一层。