ai-workspace-lab/xworkmate-app
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(cases/06): definitive 4-layer chain incl. multi-session plugin + live verification

Browse Source 
Rewrites the timeline (§1) and topology (§2) as the correct FOUR-layer chain
App → bridge → openclaw-multi-session-plugins → OpenClaw gateway, and documents the
plugin's multi-session/multi-thread role: session mapping (appThreadKey⇄openclawSessionKey),
per-(session,run) artifactScope = tasks/<sanitize(sessionKey)>/<runId>, the strict
sessionKey/runId/artifactScope triplet validation, and the expectedArtifactDirs
workspace-root fallback scan.

Live-verified against 127.0.0.1:8787 (plugin loaded, commit 2333c3e):
- session.prepare returns a real mapping; chat.send returns runId; xworkmate.tasks.get
  is handled by the plugin but returns no_native_task_record with an empty task scope
  (chain reaches the plugin layer; the agent run produced no queryable task / no file —
  a layer-4 execution/landing issue).

Adds §7 stability improvements grounded in this live run:
- S0 install the plugin from a stable path (not /private/tmp) — the primary reliability fix.
- S1 expectedArtifactDirs was [] → the plugin's workspace-root fallback is inert; bridge
  should always pass default dirs (reports/, artifacts/).
- S2 no_native_task_record status ambiguity (running vs completed-without-artifact).
- S3 sessionKey/runId/artifactScope triplet consistency (don't pre-prefix agent:main:).
- S4 runtime observability across all four layers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Haitao Pan 2026-06-26 23:18:04 +08:00
parent 0dd8a05650
commit 3993f682be
1 changed files with 76 additions and 43 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

119
docs/cases/06-gateway-turn-stability-and-robustness.md
View File

@ -17,71 +17,82 @@
---
## 1. The full chain (one gateway turn)
## 1. The full chain (one gateway turn) — 四层，含 `openclaw-multi-session-plugins`
一次 gateway turn 的完整时序：
一次 gateway turn 跨**四层**：App → bridge(Go) → **openclaw-multi-session-plugins(TS 插件)** → OpenClaw gateway runtime。
插件是关键中间层：它给「多会话/多线程」补上**逻辑隔离的 artifact scope** 与**会话映射 + 任务快照**，并以 `xworkmate.*` 网关方法暴露给 bridge。
```
App turn ──SSE POST──▶ Bridge http_handler ──▶ handleRequest ──▶ OpenClaw gateway submit
(executeTask) (text/event-stream) (阻塞 6min/60min) (WS 127.0.0.1:18789)
│ │ │
│ keepalive 20s ◀────┘ ▼
│ 返回 "running task handle"(runId + artifactScope)
▼ │
persist association ──▶ pollOpenClawTaskAssociationInternal (tasks.get 每 2s)
│
isOpenClawRunningTaskHandle? ──yes──▶ persist + continue ──┐
│ │
no ──▶ applyGatewayChatResult (terminal) └─ ∞ (无 deadline)
App.executeTask ──SSE POST /acp/rpc──▶ Bridge.handleRequest
│ │
│ ① xworkmate.session.prepare ───────▶ Gateway ──▶ [plugin] recordXWorkmateSessionMapping
│ (建 appThreadKey⇄openclawSessionKey 映射 + prepareXWorkmateArtifacts)
│ ◀── { artifactScope: tasks/<sani(sessionKey)>/<runId>, artifactDirectory, expectedArtifactDirs }
│ ② chat.send (gateway 原生) ────────▶ Gateway ──▶ 派发 agent run，返回 runId（~25ms，detached）
│ ◀── { runId, status: started }
▼
Bridge 记 sess.openClaw + DeadlineAt(budget by taskLoadClass)，返回 running 句柄
│
▼ App 持久化 association，pollOpenClawTaskAssociationInternal 轮询：
③ xworkmate.tasks.get(runId/openclawSessionKey/artifactScope) ─▶ Gateway ─▶ [plugin] getXWorkmateTaskSnapshot
├─ resolveNativeTask(host task registry by runId) ──有──▶ 回 status(running/completed/failed) + 经 export 取 artifacts
└─ 无 native task ──▶ exportArtifactsForTaskLookup 兜底：扫 scope 目录 + expectedArtifactDirs（workspace 根 reports//artifacts/）
├─ 有产物 ─▶ status=unknown, evidence=artifacts_present, 带 artifacts
└─ 无产物 ─▶ code=no_native_task_record / task_not_found
④ 终态后取产物：xworkmate.artifacts.export / .list / .read（插件 exportXWorkmateArtifacts）
scopeRoot = workspaceRoot/tasks/<sani(sessionKey)>/<runId>；按 requiredArtifactExtensions(.md) 判 constraintSatisfied
```
完成信号（completion signal）只活在**两个脆弱位置**：
**多会话/多线程隔离的核心（插件提供）**：
- `artifactScopeFor(sessionKey, runId)` = `tasks/<sanitize(sessionKey)>/<runId>`（冒号→下划线，如 `agent:main:draft:e2e` → `agent_main_draft_e2e`），每个 (会话,运行) 独立目录，互不串扰（`exportArtifacts.ts:126/164`）。
- `recordXWorkmateSessionMapping` 把 `appThreadKey ⇄ openclawSessionKey` 持久成会话扩展（`taskState.ts`），让跨连接/重连仍能按 appThreadKey 找回 run。
- `exportXWorkmateArtifacts` 严校 `requestedArtifactScope === artifactScopeFor(sessionKey,runId)`，不匹配抛 `artifactScope does not match sessionKey/runId`——**调用方必须传一致的 sessionKey/runId/artifactScope 三元组**（bridge 的 `taskGetParamsWithSessionScope` 负责从 session 记录补齐；外部手工探针易踩此坑）。
- **workspace 根兜底扫描**：agent 常把产物写到 workspace 根的 `reports/`、`artifacts/` 而非 task scope；插件用 `expectedArtifactDirs` 回扫这些目录纳入产物（见 `openclaw-gateway-e2e-regression/ROOT_CAUSE_ANALYSIS.md` Fix 0）。
1. **带内 SSE 结果帧**（result envelope / `[DONE]`）—— 依赖那条长连接活到任务结束；
2. **Gateway 对 `tasks.get` 的内存态应答** —— 依赖 bridge↔gateway 的 WS 连接仍持有该 run 的状态。
截图场景里这两条同时被打断，于是"任务跑完了，但客户端永远收不到终态"。
**live 实测（2026-06-26，8787，插件已加载）**：① session.prepare 回真实 mapping ✓；② chat.send `runId=turn-…438450000` ✓（bridge 日志 `request_timing method=chat.send`）；③ tasks.get 经插件返回 `code=no_native_task_record`（mapping 在、但 gateway 无该 run 的 native task record）+ scope 目录空、无 `news.md`。即**链路打通到插件层**，但本轮 agent 未注册可查 task、也未落产物（属第 4 层 agent 执行/落盘问题，见 §4「残留」与 §7）。
关键代码锚点：
| 环节 | 位置 |
|---|---|
| App 发起 SSE turn | `xworkmate-app/lib/app/app_controller_desktop_thread_actions.dart:644` (`executeTask`) |
| Bridge SSE handler / keepalive 20s | `xworkmate-bridge/internal/acp/http_handler.go:198-283`、`:19` |
| Bridge 阻塞等待 gateway（6min 默认 / 60min 上限） | `xworkmate-bridge/internal/acp/orchestrator.go:31-33` |
| Bridge↔Gateway WebSocket（dial 18789） | `xworkmate-bridge/internal/gatewayruntime/runtime.go:372-376` |
| App 持久化 + 轮询 run | `app_controller_desktop_thread_actions.dart:680-693`、`:747` |
| running handle 判定 | `xworkmate-app/lib/runtime/go_task_service_client.dart:519` |
| 进度条 phase（仅看 pending） | `xworkmate-app/lib/widgets/assistant_task_progress_bar.dart:190` |
| App 发起 SSE turn / 轮询 | `xworkmate-app/.../app_controller_desktop_thread_actions.dart:644`、`:747` |
| Bridge session.prepare / tasks.get / cancel | `xworkmate-bridge/internal/acp/rpc_handler.go:96`、`:126`、`taskGetParamsWithSessionScope:177` |
| Bridge↔Gateway WebSocket（dial 18789） | `xworkmate-bridge/internal/gatewayruntime/runtime.go:372` |
| 插件注册 `xworkmate.*` 网关方法 | `openclaw-multi-session-plugins/index.ts:126/162/176/206/221`（session.prepare/tasks.get/artifacts.export/.list/.read）|
| 插件 artifact scope / 会话映射 / 任务快照 | `src/exportArtifacts.ts`、`src/taskState.ts:208 getXWorkmateTaskSnapshot` |
---
## 2. 三仓库端到端拓扑（实测自代码）
## 2. 端到端拓扑（实测自代码 + live 8787 验证）
```
┌─ ai-workspace-lab/xworkmate-app (Flutter)
│ executeTask → SSE POST Accept: text/event-stream
┌─ ai-workspace-lab/xworkmate-app (Flutter) executeTask → SSE POST Accept: text/event-stream
▼
┌─ ai-workspace-infra Caddy 入口 xworkmate-bridge.svc.plus
│ handle /acp* : flush_interval -1 ✓ read_timeout 30m write_timeout 30m keepalive 5m
│ handle /api* : (Caddy 默认超时, 无 flush_interval) ⚠
│ handle / : (Caddy 默认) ⚠
┌─ ai-workspace-infra Caddy 入口 xworkmate-bridge.svc.plus （本机直连 127.0.0.1:8787 不经 Caddy）
│ /acp* : flush_interval -1 ✓ read/write_timeout 70m(对齐 bridge 60min) keepalive 5m ← T1/T2 已修
▼
┌─ ai-workspace-lab/xworkmate-bridge (Go)
│ SSE handler keepalive 20s → handleRequest 阻塞 6min(默认)/60min(max)
│ gatewayruntime: WebSocket → 127.0.0.1:18789 (pending map 绑定在连接生命周期上)
┌─ ai-workspace-lab/xworkmate-bridge (Go) live commit 2333c3e
│ /acp/rpc handleRequest → session.prepare / chat.send / xworkmate.tasks.get / tasks.cancel
│ gatewayruntime: WebSocket → 127.0.0.1:18789；per-session 持久 run 仓(T7/T8/T9)
▼ (WS, gateway 原生 chat.send + 插件注册的 xworkmate.*)
┌─ ai-workspace-lab/openclaw-multi-session-plugins (TS 插件, enabled) ← 多会话/多线程 artifact 隔离层
│ index.ts registerGatewayMethod: xworkmate.session.prepare / .tasks.get / .artifacts.export/.list/.read/.collect-and-snapshot
│ 会话映射(taskState) + artifactScope=tasks/<sani(sessionKey)>/<runId>(exportArtifacts) + workspace 根兜底扫描
▼ (插件运行在网关进程内)
┌─ OpenClaw gateway runtime npm-global openclaw 2026.6.1 @ /opt/homebrew/lib/node_modules/openclaw
│ launchd ai.openclaw.gateway, WS 18789；host task registry(detached task) + agent 执行(deepseek-v4-flash)
▼
┌─ ai-workspace-infra deploy_gateway_openclaw (roles/vhosts/gateway_openclaw)
│ OpenClaw gateway runtime, WS 18789
▼
OpenClaw 执行 → artifacts → tasks.get 回查
workspace ~/.openclaw/workspace/tasks/<sani(sessionKey)>/<runId>/ → 产物(.md 等)
```
旁路（`ai-workspace-service`）：`accounts.svc.plus`（登录 / Token）、`console.svc.plus`（自带 openclaw assistant route）、`litellm` / `AI-Relay-Kit` / `codex-relay`（模型出口）、`qmd`（记忆）。
它们不在本次卡死主链上，但 **Token 失效 / 出口异常会以同样的"连接中断"形态**表现出来，排查时需先用 `runId` 区分是主链断还是旁路断。
**live 验证状态（2026-06-26 23:xx）**：
- ✅ 网关加载 `6 plugins … openclaw-multi-session-plugins`（重启后；详见 §4）。
- ✅ `/api/ping` commit=2333c3e；session.prepare 回真实 mapping；chat.send 成功返回 runId。
- ⚠️ `xworkmate.tasks.get` → `no_native_task_record`、scope 目录空：链路通到插件，但本轮 agent 未注册可查 task / 未落产物（第 4 层执行问题）。
入口配置出处：`ai-workspace-infra/playbooks/roles/vhosts/xworkmate_bridge/templates/xworkmate-bridge-site.caddy.j2`
Gateway 部署出处：`ai-workspace-infra/playbooks/deploy_gateway_openclaw.yml` → `roles/vhosts/gateway_openclaw/`
旁路（`ai-workspace-service`）：`accounts.svc.plus`（登录/Token）、`console.svc.plus`（openclaw assistant route）、`litellm`/`AI-Relay-Kit`/`codex-relay`（模型出口）、`qmd`（记忆）——不在主链，但 Token/出口异常以「连接中断」形态出现，用 `runId` 区分。
出处：Caddy `…/xworkmate_bridge/templates/xworkmate-bridge-site.caddy.j2`；网关部署 `deploy_gateway_openclaw.yml`；插件 `~/.openclaw/extensions/openclaw-multi-session-plugins`（注册源当前指向临时 `/private/tmp/…`，见 §4 加固项）。
---
@ -281,3 +292,25 @@ curl -sS -X POST http://127.0.0.1:8787/acp/rpc \
> 回归对照：本目录 `00-review-env-and-matrix.md` 第 2 节"通用验收标准"中"长任务执行期间状态流 / 取消 / 重试稳定""同一任务重复执行 3 次不卡死"，即本规划的回归出口。
> 产物交付链（artifact scope / workspace 路径）的独立缺陷与修复，见 `openclaw-gateway-e2e-regression/ROOT_CAUSE_ANALYSIS.md`。
---
## 7. 全链路稳定性改进（基于 2026-06-26 四层 live 验证）
> 优先级按「直接决定一次任务能否产出」排序。S1/S2 是本轮 live 新发现。
- **S0 插件稳定安装（最高）** — 网关方法 `xworkmate.*` 全部依赖 `openclaw-multi-session-plugins`。当前注册源是临时路径 `/private/tmp/openclaw-multi-session-plugins/dist/index.js`，**重启/清 tmp 即丢插件**（本次故障正因网关启动早于该路径就位）。
改进：`openclaw plugins install ~/.openclaw/extensions/openclaw-multi-session-plugins`（或仓库稳定路径）落正式 install 记录；网关 supervisor 启动顺序保证「插件就位 → 启网关」。
验收：网关重启后启动日志稳定含 `… openclaw-multi-session-plugins`；`xworkmate.session.prepare` 不再 `unknown method`。
- **S1 `expectedArtifactDirs` 为空导致根目录兜底失效** — live 的 session mapping 为 `expectedArtifactDirs:[]`（contract 没派生出期望目录，`orchestrator.go:402/682`）。插件对「agent 把产物写到 workspace 根 `reports/`/`artifacts/` 而非 task scope」的兜底扫描**依赖 `expectedArtifactDirs`**；为空时兜底形同虚设 → 即便 agent 产出也收不到，表现「暂无文件」。
改进：bridge 在 `xworkmate.session.prepare` 必带一组缺省 `expectedArtifactDirs`（至少 `reports/`、`artifacts/`，并从 `requiredArtifactExtensions` / prompt 中的目标路径推导）；或插件在 `expectedArtifactDirs` 为空时回扫一组安全缺省目录。
验收：agent 写到 workspace 根的 `.md` 能被 `xworkmate.tasks.get/artifacts.export` 纳入产物。
- **S2 `no_native_task_record` 状态歧义** — `xworkmate.tasks.get` 的真值来自「gateway host task registry 有该 run 的 detached task」**或**「artifact 已存在」。live 中 chat.send 成功但 gateway 无 native task record（agent 可能以 inline chat 执行、未注册可查 task），且无产物 → 插件回 `no_native_task_record`，bridge 只能靠 T7 兜底续轮询到 deadline，**无法区分「还在跑」与「跑完没产物」**。
改进：①确认 gateway 侧 chat.send 是否应产出 detached task（agent 配置/ `tasks.*` 注册）；②插件/bridge 在 `no_native_task_record` 且超过最小执行时长时，下发更明确的 `running(no-record)` vs `completed(no-artifact)` 语义，配合 §5 T9 deadline 收口。
验收：agent 正常执行时 `tasks.get` 能返回真实 running→completed；异常时给确定终态而非无限 degraded。
- **S3 三元组一致性（已知约束）** — 插件严校 `sessionKey/runId/artifactScope` 三者一致（`exportArtifacts.ts:126`），且 bridge 的 openclawSessionKey 由 `agent:main:` + appThreadKey 组成。**调用方/探针不要预带 `agent:main:` 前缀**（否则双前缀 → `artifactScope does not match`）。bridge `taskGetParamsWithSessionScope` 已负责补齐；保持其为唯一可信来源，App/探针只传 `sessionId=draft:<id>` + `runId`。
- **S4 运行态可观测** — 沿用 §5 T11/T12：bridge `/api/ping.commit`、网关 `N plugins` 列表、`openclaw plugins inspect` 三处纳入健康检查；`runId` 贯穿 App→bridge→插件→gateway 日志，便于定位断点落在四层中的哪一层。