9 Commits

Author SHA1 Message Date
43ec2d66a3 feat(screencast): 自动激活窗口 / 多显示器 / 字幕独立时长
基于 SDK 新模块(前两个 commit)的 screencast 工具能力升级:

- runner.py:
  · 新增 activate_window: bool = True 参数,录屏前自动调
    activate_and_maximize_jiangchang_window,避免开头几秒录到 PowerShell/桌面
  · 新增 monitor_index: Optional[int] = None 参数,透传给 ScreenRecorder
  · 自带 SDK 路径兜底 _ensure_sdk_on_path,让 record_screencast.py 直接 python
    跑时(不走 pytest conftest)也能 import jiangchang_desktop_sdk.window_win32

- recorder.py:
  · ScreenRecorder.__init__ 加 monitor_index 参数
  · _capture_loop 用 idx 选 monitor,越界兜底到 0
  · None = 旧行为(monitors[0] 所有屏合并),1 = 主屏,2+ = 第 N 屏
  · 多显示器场景推荐显式传 monitor_index=1 避免录出半边空白

- subtitle.py:
  · SubtitleEngine.process_line 支持二元组(兼容旧脚本)和三元组(每条独立 duration)
  · 长等待场景可以让某条字幕停留更久(如 30s)覆盖中间无日志的"字幕真空"

所有改动向后兼容,全部新增参数有默认值。

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 11:14:11 +08:00
311441dab0 feat(sdk): 新增 e2e_helpers 共享模块,沉淀 desktop E2E 通用能力
把 disburse-payroll-icbc 实战出来的 6 个 desktop E2E 通用 helper 抽到
src/jiangchang_desktop_sdk/e2e_helpers.py,所有 skill 共用:

- drop_file_into_composer: DataTransfer 拖拽附件(绕开 SDK send_file
  在 v2.0.17+ 抛 NotImplementedError 的限制)
- wait_for_attachment_ready: 轮询附件 chip 出现
- wait_for_composer_enabled: 等输入框可见且非 disabled,避免 canSend=false 静默 no-op
- send_prompt_via_composer: fill + press_sequentially + Enter 发送
- wait_for_user_message: 60s 宽容找 user 节点(绕开 SDK ask() user_idx<0
  15s 死循环 bug,对脏 session 鲁棒)
- wait_for_agent_complete: 4 信号合流等 agent 完成
  (chat-page data-sending / window.__jc_sending / 无展开 execution-graph /
  末位 assistant has-body 且非 streaming + 文本稳定)

不抽业务断言(normalize_money / assert_zero_failures),那些是各 skill 自己的事。

所有 helper 只接受 Playwright Page 入口,不依赖 SDK client 实例。

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 11:13:47 +08:00
6e3f79dde9 feat(sdk): 抽出 window_win32 共享模块,client 激活方法薄壳化
把 _activate_and_maximize_windows 内嵌的 50+ 行 ctypes 实现抽到独立的 SDK 子模块
src/jiangchang_desktop_sdk/window_win32.py,暴露公开函数
activate_and_maximize_jiangchang_window,给 screencast 工具也复用(见后续 commit)。

client.py 的 _activate_and_maximize_windows 退化为 6 行薄壳,调用新模块。

设计:
- window_win32 是 SDK 子模块(放 src/jiangchang_desktop_sdk/ 下),不放 tools/,
  避免 SDK 反向依赖工具层
- ctypes EnumWindows + ShowWindow(SW_MAXIMIZE) + SetForegroundWindow 逻辑不变
- 跨平台行为不变:Windows 真执行,其他系统返回 False
- bring_to_front 调用方完全不感知变化(向后兼容)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 11:13:27 +08:00
ffb64b2cfc feat(screencast): 统一字幕样式 / 修字幕重叠 / 修子进程乱码
- composer.py:抽出 _JIANGCHANG_SUBTITLE_STYLE 常量,FontSize 28→14 /
  Outline 2→2(保持),1080p 屏幕约占 1.3% 高度,演示视频留足画面呼吸感
- subtitle.py:generate_srt 加 post-process 排队,相邻字幕互相让位
  (MIN_DURATION=1.0s / GAP=0.1s),任意时刻屏幕上最多 1 条字幕,
  避免开头几秒密集触发的字幕叠成两行
- runner.py:subprocess.Popen 注入 PYTHONIOENCODING=utf-8 + PYTHONUTF8=1,
  让 Windows 子进程 Python 强制 UTF-8 输出,runner 端 UTF-8 解码即对齐,
  修中文乱码

所有 skill 录屏受益,向后兼容(非 Windows 系统 UTF-8 注入无副作用)。

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:14:03 +08:00
c8d136fb79 feat(sdk): bring_to_front 用 Win32 API 激活并最大化主窗口
新增 _activate_and_maximize_windows 私有方法:用 ctypes 调 EnumWindows
找匠厂主窗口(标题含 匠厂/Jiangchang/ClawX/OpenClaw),
ShowWindow(SW_MAXIMIZE) + SetForegroundWindow 让窗口铺满桌面。

bring_to_front 改 OS 分支:
- Windows: 优先 ctypes 激活+最大化(dev / 安装版通吃,比 jiangchang://
  协议在 dev 模式下因 cwd 解析失败而报错可靠);失败时退回协议作为兜底
- macOS / Linux: 维持原协议激活路径(open / xdg-open)

适用场景:录屏 / 自动化测试需要 OS 级窗口前置。page.bring_to_front()
只切 Playwright tab 不影响 z-order,必须 OS 级 API 才能把 Electron
主窗口拉到桌面最前。

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:13:43 +08:00
34f9a2c141 docs: Stage 2 SDK data-jcid migration report
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-18 16:05:16 +08:00
4fc4cfb802 refactor(sdk): drop data-jcid, switch to semantic anchors + ClawX testids
- Introduce centralized SELECTOR constant table at top of client.py
- Replace all [data-jcid=...] selectors with:
  * Semantic anchors (data-role / data-message-id / data-streaming /
    data-sending / data-message-count) guarded by jiangchang main repo
  * ClawX upstream data-testids (chat-page / chat-composer-* / etc.)
- _latest_assistant_after: has_body now judged by inner_text non-empty
  (no longer depends on message-body sub-element)
- _gateway_state: degrade to window.__jc_gateway_state__ readback,
  returns 'unknown' until host exposes it
- new_task: rely solely on data-message-count==0 (welcome-screen no
  longer has a testid)
- wait_gateway_ready: degrade to chat-page selector wait
- send_file: raise NotImplementedError (Electron IPC hook missing)

Public API signatures unchanged. window.__jc_sending__ remains the
core sending signal.

Compatibility: jiangchang v2.0.17+ only.
Refs: jiangchang/docs/sdk-migration-discovery.md
      jiangchang/docs/JIANGCHANG_CUSTOM_ANCHORS.md
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-18 16:05:04 +08:00
089f00c3ff feat(tools): add screencast recording engine
Adds tools/screencast/ — a reusable screen-recording pipeline for
skill demo videos. Wraps a pytest desktop E2E run with:
  - ScreenRecorder: mss-based full-screen frame capture (background thread)
  - SubtitleEngine: stdout keyword → timestamped SRT generation
  - compose_video: FFmpeg filter_complex merge (video + subtitles + BGM)
  - run_screencast: main orchestrator wiring all three layers

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-17 20:38:47 +08:00
20294fd23a 新增依赖 2026-05-06 19:33:46 +08:00
12 changed files with 1086 additions and 124 deletions

View File

@@ -63,3 +63,57 @@ git push origin v0.1.0
```
仓库内还保留 `examples/``tools/``python-runtime/` 目录及既有 Skill / 前端相关的 GitHub Actions 工作流,与本 PyPI 包的发布彼此独立。
## v0.2.0 — data-jcid migration (internal refactor)
### TL;DR
本次内部重构:SDK 不再依赖匠厂主仓库的 `data-jcid` 属性。
所有公开 API(`ask` / `read` / `new_task` / `wait_gateway_ready` 等)
**签名与行为保持不变**,只有内部选择器实现切换到新的语义锚点 + ClawX testid。
### 改变了什么
- `client.py` 内所有 `[data-jcid="..."]` 选择器已移除
- 改用匠厂主仓库为 SDK 承诺的语义锚点:`data-role` / `data-message-id` /
`data-streaming`(在 ChatMessage 上)、`data-sending` / `data-message-count`
(在 chat-page 根上)
- `window.__jc_sending__` 仍是核心 sending 信号,保持不变
- 这些锚点由匠厂主仓库的以下机制守护:
- `docs/JIANGCHANG_CUSTOM_ANCHORS.md` 中央清单
- 源码内 `JIANGCHANG CUSTOM ANCHOR` 边界注释
- `harness/specs/jiangchang-custom-anchors.spec.ts` E2E 守护测试
- Gitea CI grep 检查 step
### 已知限制
- `send_file()` 暂未实现,抛 `NotImplementedError`
原因:附件上传走 Electron IPC,SDK 需要主仓库提供新 hook。
- `_gateway_state()` 永远返回 `'unknown'`
原因:gateway 状态不再暴露 DOM data-state;本方法已预留未来通过
`window.__jc_gateway_state__` 读取的实现路径。
这两个限制对常规 ask/read 工作流**无影响**,sending 完成判定依靠
`window.__jc_sending__` + `data-sending` + 文本稳定性合流即可。
### 升级影响
- 在 jiangchang v2.0.17(含)以上版本上跑:应正常工作
- 在 jiangchang v2.0.16(及更早含 data-jcid 的版本)上跑:不再兼容
如果需要在旧版本上跑,固定 SDK 版本到 v0.1.x
### 验证清单(集成方建议跑一次)
打开匠厂客户端 → 用 SDK 跑下面 3 步,确认基础流程通:
```python
from jiangchang_desktop_sdk import JiangchangDesktopClient
with JiangchangDesktopClient() as c:
c.ensure_app_running()
c.wait_gateway_ready()
c.new_task()
answer = c.ask("你好,请回复确认 SDK 可用")
assert answer, f"Empty answer: {answer!r}"
print("OK:", answer[:200])
```

View File

@@ -0,0 +1,131 @@
# Stage 2 — SDK data-jcid Migration Report
## 1. 执行摘要
- 起始 HEAD: `089f00c3ff6b75125d77f40b2e2dcb664a9ac5bf`
- 结束 HEAD: 本阶段未 commit,工作区状态见下
- 影响文件:
- `src/jiangchang_desktop_sdk/client.py`(主要改动,7 步合计)
- `README.md`(新增 v0.2.0 章节)
- `STAGE2_SDK_MIGRATION_REPORT.md`(本回执)
- 其他例外文件:无(`types.py` / `exceptions.py` 未改)
## 2. 7 步改动总览
| 步 | 内容 | 影响范围 |
|----|------|---------|
| 1 | SELECTOR 常量表 | client.py 顶部新增 36 行 |
| 2 | _get_*_locator x4 + 2 处 raise 文案 | client.py 局部 |
| 3 | read() | client.py 局部 |
| 4 | 9 个私有 helper | client.py 多处 |
| 5 | _wait_for_streaming_done | client.py 局部(docstring + 注释) |
| 6 | new_task / wait_gateway_ready / send_file | client.py 局部 |
| 7 | README + 扫尾 | README.md + 验证 |
**第 7 步附带**: `ask()` 兜底分支仍残留可执行 `[data-jcid="message"]` / `data-jcid-role`,已改为 `SEL_MESSAGE_ROW` + `data-role`(否则 7.1 可执行代码 grep 无法通过)。
## 3. 公开 API 兼容性证明
逐项列出公开 API 改造前后签名,确认一字不差:
| 方法 | 改前签名 | 改后签名 | 一致? |
|------|---------|---------|-------|
| ensure_app_running | (self, wait_timeout_s: float = 30.0) -> None | 同左 | ✓ |
| bring_to_front | (self) -> None | 同左 | ✓ |
| connect | (self, url: Optional[str] = None) -> None | 同左 | ✓ |
| launch_app | (self, options: Optional[LaunchOptions] = None) -> None | 同左 | ✓ |
| disconnect | (self) -> None | 同左 | ✓ |
| is_connected | (self) -> bool | 同左 | ✓ |
| get_page | (self) -> Page | 同左 | ✓ |
| new_task | (self) -> None | 同左 | ✓ |
| wait_gateway_ready | (self, timeout_ms: int = 30000) -> None | 同左 | ✓ |
| wait_for_response | (self, timeout: int = 120000) -> None | 同左 | ✓ |
| ask | (self, question: str, options: Optional[AskOptions] = None) -> str | 同左 | ✓ |
| send_file | (self, file_path: str, message: Optional[str] = None) -> None | 同左(行为变 NotImplementedError) | ✓ |
| read | (self) -> List[JiangchangMessage] | 同左 | ✓ |
| assert_contains | (self, expected: str, options: Optional[AssertOptions] = None) -> None | 同左 | ✓ |
| snapshot | (self, target_dir: str, tag: str = "snapshot") -> dict | 同左 | ✓ |
| __enter__ | (self) -> "JiangchangDesktopClient" | 同左 | ✓ |
| __exit__ | (self, exc_type, exc_val, exc_tb) -> None | 同左 | ✓ |
## 4. 临时限制说明
### send_file
- 状态:抛 NotImplementedError
- 原因:Electron IPC stage-paths,SDK 无 DOM 入口
- 恢复条件:匠厂主仓库暴露 `window.__jc_stage_paths__` 或带 testid 的隐藏 input
### _gateway_state
- 状态:永远返回 'unknown'
- 原因:HEAD 上 gateway 状态不暴露 DOM
- 行为影响:_wait_for_streaming_done 中 `if gw not in ("running", "unknown")`
实质 no-op,但保留以便未来生效
- 恢复条件:匠厂在 main / store 暴露 `window.__jc_gateway_state__`
## 5. 扫尾验证实际输出
### 5.1 `grep -n 'data-jcid' src/jiangchang_desktop_sdk/client.py`
```
src/jiangchang_desktop_sdk/client.py:10:3. 每次提问前新建任务(点击侧栏 `data-jcid="new-task-button"`),避免上下文污染。 (模块 docstring 历史描述)
src/jiangchang_desktop_sdk/client.py:14: 这些 DOM 节点 **不再带 `data-jcid="message"`**,而是 `data-testid="chat-execution-graph"` (模块 docstring 历史描述)
src/jiangchang_desktop_sdk/client.py:18: · **锚点**`ask()` 发送后等到最后一条 `[data-jcid-role="user"]` 节点(就是我们刚发的 (模块 docstring 历史描述)
src/jiangchang_desktop_sdk/client.py:19: 问题)在 DOM 里出现,记住它在 `[data-jcid="message"]` 列表里的 index作为切片起点 (模块 docstring 历史描述)
src/jiangchang_desktop_sdk/client.py:21: · `chat-root[data-jcid-sending] === 'false'` (模块 docstring 历史描述)
src/jiangchang_desktop_sdk/client.py:25: · 我们这条 user 之后存在 `[data-jcid-role="assistant"]` 且其内部有 (模块 docstring 历史描述)
src/jiangchang_desktop_sdk/client.py:26: `[data-jcid="message-body"]` (模块 docstring 历史描述)
src/jiangchang_desktop_sdk/client.py:29:5. read() 严格按 `data-jcid-role` 分类,没有该属性的 DOM 节点不再默认归为 assistant。 (模块 docstring 历史描述)
src/jiangchang_desktop_sdk/client.py:526: """返回 `[data-jcid="message"]` 节点列表中最后一条 role=user 的 index找不到返回 -1。""" (docstring 历史描述)
src/jiangchang_desktop_sdk/client.py:561: "streaming": bool, # data-jcid-streaming 属性 (docstring 历史描述)
src/jiangchang_desktop_sdk/client.py:610: """chat-root[data-jcid-sending] 的 bool 化;读不到返回 None。""" (docstring 历史描述)
src/jiangchang_desktop_sdk/client.py:796: "我们刚发出的 user 节点在 [data-jcid=\"message\"] 列表里的 index"。后续所有 (ask docstring 历史描述)
```
可执行代码(非 docstring / 非 `#` 注释):**0 命中**。
### 5.2 `grep -n '__jc_sending__' src/jiangchang_desktop_sdk/client.py`
```
src/jiangchang_desktop_sdk/client.py:22: · `window.__jc_sending__` !== true
src/jiangchang_desktop_sdk/client.py:633: """从 window.__jc_sending__ 读取当前 sending 状态;不可用返回 None。"""
src/jiangchang_desktop_sdk/client.py:636: val = page.evaluate("() => window.__jc_sending__")
src/jiangchang_desktop_sdk/client.py:673: C同时 window.__jc_sending__ 不是 true允许 None / False以容忍
```
### 5.3 import 可用性
```
import OK
```
## 6. 集成验证
- 是否在本地跑过 README 末尾的 3 步验证流程?
- [ ] 跑过 → 结果(pass/fail + 关键输出片段)
- [x] 未跑 → 原因:验收环境未启动匠厂客户端 / 本阶段仅完成代码与静态扫尾
- 是否运行了 platform-kit 自带的任何测试?
- [ ] 跑过 → 结果
- [x] 未跑 → 原因:本包无针对 desktop_sdk 的自动化集成测试;需实机 CDP
## 7. 未触及的事项
- `examples/``tools/screencast/`:未在本阶段验证;预期不直接依赖已移除的 data-jcid 选择器
- platform-kit stash 中的 `runtime_env.py` 改动:本阶段未碰,仍在 stash(`stash@{0}: wip: claw env var fallback...`)
- `D:\AI\jiangchang` 主仓库:未修改
- 模块级 / 部分 helper / `ask()` docstring 中 data-jcid 历史措辞:保留,见 §9
## 8. 自检
- [x] grep 'data-jcid' 在可执行代码中 → 0 命中
- [x] 公开 API 签名表逐项核对一致
- [x] send_file 抛 NotImplementedError 而非静默
- [x] _gateway_state 含 TODO 注释
- [x] README 新增 v0.2.0 章节,无格式错乱
- [x] 未触碰 D:\AI\jiangchang 主仓库任何文件
- [x] 未触碰 platform-kit 非 client.py / README.md / 本回执 的文件
- [ ] git status 显示工作区干净(除本阶段预期修改 + 已有 stash) → 见 §1,未 commit
## 9. 给验收方的备注
1. **docstring 历史 data-jcid**: 模块顶 docstring(L10L29)、`_last_user_node_index`(L526)、`_latest_assistant_after` 内联说明(L561)、`_chat_root_sending`(L610)、`ask()` docstring(L796) 仍描述旧 UI 模型;不影响运行。若需文档一致性,可单独开文档清理 PR。
2. **send_file**: 调用即 `NotImplementedError`,无连接检查(按规格整段替换方法体)。
3. **wait_gateway_ready**: 仅等待 `SEL_CHAT_PAGE` 可见,不保证 Gateway 进程已 ready。
4. **建议验收**: 在已安装 v2.0.17+ 匠厂 + Playwright 环境跑 README 末尾脚本;并确认 `send_file(...)` 按预期抛错。

View File

@@ -12,6 +12,7 @@ license = "MIT"
authors = [{ name = "client-jiangchang" }]
dependencies = [
"requests>=2.31.0",
"playwright>=1.42.0",
]
[project.urls]

View File

@@ -54,6 +54,42 @@ from .exceptions import (
)
from .types import AskOptions, AssertOptions, JiangchangMessage, LaunchOptions
# ============================================================================
# JIANGCHANG DOM SELECTORS
# ============================================================================
# 本节集中维护所有 Playwright 选择器,作为对匠厂前端的契约。
# 选择器分两类:
#
# (A) 主仓库语义锚点(由 docs/JIANGCHANG_CUSTOM_ANCHORS.md 守护,稳定)
# 这些是匠厂为 SDK 暴露的承诺契约,被 E2E 守护测试 + Gitea CI 双重保护。
#
# (B) 主仓库 ClawX 上游 data-testid(随上游同步可能漂移)
# 这些是 ClawX 上游或我们扩展的测试钩子,本身用于内部 E2E,
# SDK 借用但不保证不变。
#
# 修改前请确认 docs/sdk-migration-discovery.md 中对应元素仍存在。
# ============================================================================
# --- (A) 匠厂语义锚点 ---
SEL_CHAT_PAGE = '[data-testid="chat-page"]'
SEL_CHAT_PAGE_SENDING = '[data-testid="chat-page"][data-sending="true"]'
SEL_MESSAGE_ROW = '[data-role]' # ChatMessage 行容器
SEL_MESSAGE_ROW_USER = '[data-role="user"]'
SEL_MESSAGE_ROW_ASSISTANT = '[data-role="assistant"]'
SEL_MESSAGE_ROW_STREAMING = '[data-streaming="true"]'
# --- (B) ClawX/上游 data-testid ---
SEL_SIDEBAR_NEW_CHAT = '[data-testid="sidebar-new-chat"]'
SEL_CHAT_COMPOSER_INPUT = '[data-testid="chat-composer-input"]'
SEL_CHAT_COMPOSER_SEND = '[data-testid="chat-composer-send"]'
SEL_CHAT_MESSAGE_INDEXED = '[data-testid^="chat-message-"]' # chat-message-${idx}
SEL_EXECUTION_GRAPH = '[data-testid="chat-execution-graph"]'
SEL_EXECUTION_GRAPH_EXPANDED = '[data-testid="chat-execution-graph"][data-collapsed="false"]'
SEL_MAIN_LAYOUT = '[data-testid="main-layout"]'
# --- (C) 通用兜底(无 testid 时使用) ---
SEL_FALLBACK_TEXTAREA = 'textarea'
# ── 调试日志 ───────────────────────────────────────────────────────
_logger = logging.getLogger("jiangchang_desktop_sdk")
_handler_added = False
@@ -182,6 +218,15 @@ class JiangchangDesktopClient:
_logger.warning("[activate] 协议激活失败:%s", exc)
return False
def _activate_and_maximize_windows(self) -> bool:
"""Windows 专属:把匠厂主窗口拉到桌面最前并最大化。
实现已抽到 jiangchang_desktop_sdk.window_win32 模块SDK 子模块),
给 screencast 工具也复用。本方法保留为类内入口(兼容旧调用)。
"""
from .window_win32 import activate_and_maximize_jiangchang_window
return activate_and_maximize_jiangchang_window()
def _launch_via_executable(self, cdp_port: int) -> bool:
"""回退:直接 spawn 可执行文件,带 --remote-debugging-port。"""
exe = self._get_default_executable_path()
@@ -234,12 +279,38 @@ class JiangchangDesktopClient:
)
def bring_to_front(self) -> None:
"""把主窗口聚焦到最前。"""
"""把主窗口聚焦到桌面最前。
分两层:
1) page.bring_to_front() —— Playwright 把 page 切到当前 context 内的最前
tab。**不影响 OS 级窗口 z-order**。
2) OS 级窗口前置(录屏 / 自动化必需):
- Windows: 用 ctypes 调 Win32 EnumWindows 找匠厂主窗口 →
ShowWindow(SW_MAXIMIZE) + SetForegroundWindow让窗口铺满桌面
(详见 _activate_and_maximize_windows。失败时退回协议激活作为
兜底,但 dev 模式pnpm dev / electron .)下协议本身会因 cwd 解析
失败而无效,只有安装版协议兜底才可能成功。
- macOS / Linux: 走 jiangchang:// 协议open / xdg-open
非 Windows 系统下协议处理器没有 cwd 解析问题)。
"""
try:
page = self.get_page()
page.bring_to_front()
except Exception as exc:
_logger.debug("[bring_to_front] 失败:%s", exc)
_logger.debug("[bring_to_front] Playwright 切 page 失败:%s", exc)
if os.name == "nt":
if self._activate_and_maximize_windows():
return
try:
self._activate_via_protocol()
except Exception as exc:
_logger.debug("[bring_to_front] 协议激活失败:%s", exc)
else:
try:
self._activate_via_protocol()
except Exception as exc:
_logger.debug("[bring_to_front] 协议激活失败:%s", exc)
# ─── 连接 / 断开 ──────────────────────────────────────────────
@@ -375,51 +446,46 @@ class JiangchangDesktopClient:
def _get_chat_input_locator(self):
page = self.get_page()
locator = page.locator('[data-jcid="chat-input"]')
locator = page.locator(SEL_CHAT_COMPOSER_INPUT)
if locator.count() > 0:
return locator
locator = page.locator("textarea").last
locator = page.locator(SEL_FALLBACK_TEXTAREA).last
if locator.count() > 0:
return locator
raise ConnectionError("找不到聊天输入框textarea请确认匠厂已加 data-jcid=\"chat-input\"")
raise ConnectionError(
'找不到聊天输入框,请确认匠厂已加 data-testid="chat-composer-input" '
"或页面中存在 textarea"
)
def _get_send_button_locator(self):
page = self.get_page()
locator = page.locator('[data-jcid="send-button"]')
locator = page.locator(SEL_CHAT_COMPOSER_SEND)
if locator.count() > 0:
return locator
locator = page.locator('button[type="submit"]')
if locator.count() > 0:
return locator
raise ConnectionError("找不到发送按钮,请确认匠厂已加 data-jcid=\"send-button\"")
raise ConnectionError(
'找不到发送按钮,请确认匠厂已加 data-testid="chat-composer-send" '
'或 button[type="submit"]'
)
def _get_new_task_button_locator(self):
page = self.get_page()
locator = page.locator('[data-jcid="new-task-button"]')
locator = page.locator(SEL_SIDEBAR_NEW_CHAT)
if locator.count() > 0:
return locator
# 回退:旧版本 data-testid
locator = page.locator('[data-testid="sidebar-new-chat"]')
if locator.count() > 0:
return locator
# 最后兜底:文本"新建任务"
locator = page.get_by_text("新建任务", exact=False).first
return locator
def _get_message_list_locator(self):
page = self.get_page()
locator = page.locator('[data-jcid="message-list"]')
if locator.count() > 0:
return locator
locator = page.locator('[class*="space-y-3"]').first
if locator.count() > 0:
return locator
raise ConnectionError("找不到消息列表容器 data-jcid=\"message-list\"")
return page.locator(SEL_CHAT_PAGE)
# ─── 新建任务 ─────────────────────────────────────────────────
def new_task(self) -> None:
"""点击"新建任务"按钮,等待消息列表清空或欢迎屏出现"""
"""点击"新建任务"按钮,等待消息列表清空chat-page data-message-count=0"""
if not self._connected:
raise ConnectionError("未连接到应用")
page = self.get_page()
@@ -431,16 +497,12 @@ class JiangchangDesktopClient:
_logger.warning("[new_task] 点击失败:%s", exc)
return
# 等待 chat-root.message-count=0 或 welcome-screen 出现,最多 5s
deadline = time.time() + 5
while time.time() < deadline:
try:
count = page.locator('[data-jcid="chat-root"]').first.get_attribute("data-jcid-message-count")
count = page.locator(SEL_CHAT_PAGE).first.get_attribute("data-message-count")
if count == "0":
_logger.debug("[new_task] 已进入新任务message-count=0")
return
if page.locator('[data-jcid="welcome-screen"]').count() > 0:
_logger.debug("[new_task] 欢迎屏已出现")
_logger.debug("[new_task] 已进入新任务(data-message-count=0")
return
except Exception:
pass
@@ -450,25 +512,35 @@ class JiangchangDesktopClient:
# ─── 网关健康检查 ─────────────────────────────────────────────
def _gateway_state(self) -> str:
"""Read gateway state via Electron IPC bridge (jiangchang exposes no DOM data-state).
Strategy: try to read window.__jc_gateway_state__ if exposed;
otherwise return 'unknown' and let upper logic continue.
"""
# TODO: 等匠厂在 src/stores/gateway/* 或 main 进程暴露
# window.__jc_gateway_state__ 之后,本方法将自动开始返回真实状态。
# 当前永远返回 'unknown',行为退化为"假定 gateway 健康"。
try:
page = self.get_page()
el = page.locator('[data-jcid="gateway-status"]').first
if el.count() == 0:
return "unknown"
return el.get_attribute("data-state") or "unknown"
state = self.get_page().evaluate(
"() => window.__jc_gateway_state__"
)
if isinstance(state, str) and state:
return state
except Exception:
return "unknown"
pass
return 'unknown'
def wait_gateway_ready(self, timeout_ms: int = 30000) -> None:
"""等待匠厂前端就绪。
由于 v2.0.17 起 gateway 状态不再暴露 DOM data-state,本方法降级为
等待 chat-page 根元素可见(隐含 React + gateway store 已初始化)。
Gateway 实际就绪与否,由后续 ask() 内的 _gateway_state 检查兜底
(当前永远返回 'unknown')。
"""
page = self.get_page()
try:
page.wait_for_function(
"""() => {
const el = document.querySelector('[data-jcid="gateway-status"]');
return el && el.dataset.state === 'running';
}""",
timeout=timeout_ms,
)
page.wait_for_selector(SEL_CHAT_PAGE, timeout=timeout_ms)
except Exception as exc:
_logger.warning("[wait_gateway_ready] 超时/异常:%s(继续,但后续可能报 GatewayDown", exc)
@@ -481,7 +553,7 @@ class JiangchangDesktopClient:
def _message_node_count(self) -> int:
try:
return self.get_page().locator('[data-jcid="message"]').count()
return self.get_page().locator(SEL_MESSAGE_ROW).count()
except Exception:
return 0
@@ -489,10 +561,10 @@ class JiangchangDesktopClient:
"""返回 `[data-jcid="message"]` 节点列表中最后一条 role=user 的 index找不到返回 -1。"""
try:
page = self.get_page()
total = page.locator('[data-jcid="message"]').count()
total = page.locator(SEL_MESSAGE_ROW).count()
for i in range(total - 1, -1, -1):
node = page.locator('[data-jcid="message"]').nth(i)
role = node.get_attribute("data-jcid-role") or ""
node = page.locator(SEL_MESSAGE_ROW).nth(i)
role = node.get_attribute("data-role") or ""
if role.lower() == "user":
return i
return -1
@@ -507,7 +579,7 @@ class JiangchangDesktopClient:
return False
try:
page = self.get_page()
node = page.locator('[data-jcid="message"]').nth(user_idx)
node = page.locator(SEL_MESSAGE_ROW).nth(user_idx)
text = node.inner_text(timeout=2000) or ""
t = re.sub(r"\s+", "", text)
q = re.sub(r"\s+", "", question)
@@ -529,22 +601,19 @@ class JiangchangDesktopClient:
return None
try:
page = self.get_page()
nodes = page.locator('[data-jcid="message"]')
nodes = page.locator(SEL_MESSAGE_ROW)
total = nodes.count()
for i in range(total - 1, user_idx, -1):
node = nodes.nth(i)
role = (node.get_attribute("data-jcid-role") or "").lower()
role = (node.get_attribute("data-role") or "").lower()
if role != "assistant":
continue
body = node.locator('[data-jcid="message-body"]')
has_body = body.count() > 0
body_text = ""
if has_body:
try:
body_text = body.first.inner_text(timeout=2000) or ""
except Exception:
body_text = ""
streaming = (node.get_attribute("data-jcid-streaming") or "false").lower() == "true"
try:
body_text = node.inner_text(timeout=2000) or ""
except Exception:
body_text = ""
has_body = len(body_text.strip()) > 0
streaming = (node.get_attribute("data-streaming") or "false").lower() == "true"
return {
"index": i,
"has_body": has_body,
@@ -575,8 +644,8 @@ class JiangchangDesktopClient:
def _chat_root_sending(self) -> Optional[bool]:
"""chat-root[data-jcid-sending] 的 bool 化;读不到返回 None。"""
try:
val = self.get_page().locator('[data-jcid="chat-root"]').first.get_attribute(
"data-jcid-sending"
val = self.get_page().locator(SEL_CHAT_PAGE).first.get_attribute(
"data-sending"
)
if val is None:
return None
@@ -586,8 +655,8 @@ class JiangchangDesktopClient:
def _chat_root_message_count(self) -> Optional[int]:
try:
val = self.get_page().locator('[data-jcid="chat-root"]').first.get_attribute(
"data-jcid-message-count"
val = self.get_page().locator(SEL_CHAT_PAGE).first.get_attribute(
"data-message-count"
)
if val is None:
return None
@@ -626,16 +695,16 @@ class JiangchangDesktopClient:
"""
等待 agent 整轮(可能包含多次 thinking / tool_use 往返)完成。
核心判据 —— 只有 agent 最终答复才有 `message-body`(文本气泡)
核心判据 —— agent 最终答复在 data-role=assistant 行上有非空 inner_text
源码依据D:\\AI\\jiangchang/src/stores/chat/helpers.ts
- isToolOnlyMessage 显式允许 thinking 与 tool_use 共存仍视为中间轮
- 中间工具轮的 ChatMessage 无 hasText → 不渲染 MessageBubble →
也就不会有 `data-jcid="message-body"`
- 只有最终含 text 的助手消息才会渲染 message-body
- 中间工具轮可能无可见正文气泡 → data-role 行 inner_text 为空
- 只有最终含 text 的助手消息行 inner_text 非空has_body
因此可靠的完成条件为:
A最后一条 assistant message 存在 [data-jcid="message-body"]
B其文本连续 `stable_seconds` 秒无变化
Auser_msg_index 之后最后一条 assistant 的 has_body 为 True
(由 _latest_assistant_after 根据 inner_text 判定)
B其 body_text 连续 `stable_seconds` 秒无变化;
C同时 window.__jc_sending__ 不是 true允许 None / False以容忍
早期 React 未挂载注入钩子的情况)。
@@ -675,10 +744,10 @@ class JiangchangDesktopClient:
assistant = self._latest_assistant_after(user_msg_index)
# 完成判据需同时满足:
# (1) 三路 sending 信号全部静默chat-root / window / 未收起的 execution graph
# (2) user_msg_index 之后存在 assistant 消息
# (3) 该 assistant 存在 message-body
# (4) 该 assistant 的 data-jcid-streaming != 'true'
# (1) 三路 sending 信号全部静默chat-page data-sending / window / 未收起的 execution graph
# (2) user_msg_index 之后存在 assistant 消息data-role 行)
# (3) 该 assistant has_body 为 True_latest_assistant_afterinner_text 非空)
# (4) 该 assistant 的 data-streaming != 'true'
# (5) body 文本连续 stable_seconds 秒无变化
can_check_stable = (
not sending_any
@@ -740,7 +809,7 @@ class JiangchangDesktopClient:
raise JcTimeoutError(
f"等待 AI 回复超时({timeout_ms}ms。user_msg_index={user_msg_index} 最后阶段 phase={phase}"
"可能原因:模型思考时间过长 / 工具链循环未结束 / UI 未挂载 message-body / Gateway 卡住。"
"可能原因:模型思考时间过长 / 工具链循环未结束 / assistant 正文未出现data-role 行 inner_text 为空)/ Gateway 卡住。"
)
# ─── 核心交互ask / read / assert ────────────────────────────
@@ -844,70 +913,56 @@ class JiangchangDesktopClient:
poll_interval=opts.poll_interval,
)
# 6. 选答:只在 my_user_idx 之后的 assistant 中找带 message-body 的最后一条
# 6. 选答:只在 my_user_idx 之后的 assistant 中找 has_body 的最后一条
picked = self._latest_assistant_after(my_user_idx)
if picked is None or not picked["has_body"]:
_logger.warning(
"[ask] user_idx=%d 之后未找到带 message-body 的 assistantpicked=%s",
"[ask] user_idx=%d 之后未找到 has_body 的 assistantpicked=%s",
my_user_idx, picked,
)
# 兜底:取我们 user 之后的任意最后一条 assistant inner_text
try:
page = self.get_page()
nodes = page.locator('[data-jcid="message"]')
nodes = page.locator(SEL_MESSAGE_ROW)
total = nodes.count()
for i in range(total - 1, my_user_idx, -1):
node = nodes.nth(i)
if (node.get_attribute("data-jcid-role") or "").lower() == "assistant":
if (node.get_attribute("data-role") or "").lower() == "assistant":
return node.inner_text(timeout=2000) or ""
except Exception:
pass
return ""
_logger.debug(
"[ask] 选定 assistant_idx=%duser_idx=%d 之后第一条带 message-bodylen=%d",
"[ask] 选定 assistant_idx=%duser_idx=%d 之后 has_bodylen=%d",
picked["index"], my_user_idx, len(picked["body_text"]),
)
return picked["body_text"]
def send_file(self, file_path: str, message: Optional[str] = None) -> None:
if not self._connected:
raise ConnectionError("未连接到应用")
page = self.get_page()
"""[暂未实现] 发送文件附件。
attach_btn = page.locator('[data-jcid="attach-button"]')
if attach_btn.count() == 0:
attach_btn = page.locator("button").filter(
has_text=re.compile(r"附件|上传|paperclip|attach", re.I)
)
匠厂 v2.0.17(ClawX v0.4.3 合并后)的附件上传通道由原生 Electron
showOpenDialog + IPC stage-paths 实现,不再暴露 HTML <input type="file">。
file_input = page.locator('input[type="file"]')
if file_input.count() == 0 and attach_btn.count() > 0:
attach_btn.first.click()
time.sleep(0.5)
file_input = page.locator('input[type="file"]')
Playwright 无法直接触发 Electron 原生对话框。需要匠厂主仓库新增:
- window.__jc_stage_paths__(paths: string[]) 调用桥,或
- 一个隐藏 <input type="file"> 备用通道,带 data-testid
if file_input.count() == 0:
raise AssertError(
"找不到文件上传 input[type='file']。请确认匠厂已加 data-jcid=\"attach-button\""
)
在匠厂提供其中之一之前,本方法抛 NotImplementedError 而非静默成功。
abs_path = os.path.abspath(file_path)
if not os.path.exists(abs_path):
raise AppNotFoundError(f"上传文件不存在: {abs_path}")
file_input.set_input_files(abs_path)
time.sleep(1)
if message:
input_locator = self._get_chat_input_locator()
input_locator.click()
input_locator.press_sequentially(message, delay=25)
input_locator.press("Enter")
Tracking: 见 docs/sdk-migration-discovery.md §7
"""
raise NotImplementedError(
"send_file is temporarily unavailable on jiangchang v2.0.17+. "
"Reason: attachments now go through Electron IPC; SDK requires "
"a new DOM-or-window hook from the host application. "
"See docs/sdk-migration-discovery.md §7."
)
def read(self) -> List[JiangchangMessage]:
"""
严格读取:仅接受带 data-jcid="message" + data-jcid-role 节点。
严格读取:仅接受带 data-role 语义锚点的 ChatMessage 节点。
无 role 的节点返回 role='unknown'(不再默认为 assistant
"""
if not self._connected:
@@ -917,41 +972,35 @@ class JiangchangDesktopClient:
messages: List[JiangchangMessage] = []
try:
msg_list = self._get_message_list_locator()
msg_list.wait_for(state="visible", timeout=5000)
page.locator(SEL_CHAT_PAGE).wait_for(state="visible", timeout=5000)
except Exception as exc:
_logger.debug("[read] 消息列表不可见:%s", exc)
_logger.debug("[read] 聊天页不可见:%s", exc)
return messages
items = page.locator('[data-jcid="message"]')
items = page.locator(SEL_MESSAGE_ROW)
total = items.count()
_logger.debug("[read] data-jcid=\"message\" 节点数=%d", total)
_logger.debug("[read] data-role 消息行数=%d", total)
for i in range(total):
el = items.nth(i)
try:
role_attr = el.get_attribute("data-jcid-role")
streaming_attr = el.get_attribute("data-jcid-streaming") or "false"
# 优先从 message-body正文气泡取干净文本避免带入 Thinking / ToolCards
body_locator = el.locator('[data-jcid="message-body"]')
if body_locator.count() > 0:
content = body_locator.first.inner_text(timeout=2000)
else:
content = el.inner_text(timeout=2000)
role_attr = el.get_attribute("data-role")
streaming_attr = el.get_attribute("data-streaming") or "false"
content = el.inner_text(timeout=2000)
role = (role_attr or "unknown").lower()
tool_status = "streaming" if streaming_attr == "true" else None
messages.append(
JiangchangMessage(
id=el.get_attribute("data-jcid-message-id") or f"msg-{i}",
id=el.get_attribute("data-message-id") or f"msg-{i}",
role=role,
content=content,
timestamp=time.time(),
is_error=(el.get_attribute("data-jcid-error") == "true"),
tool_name=el.get_attribute("data-jcid-tool"),
is_error=False,
tool_name=None,
tool_status=tool_status,
)
)
if streaming_attr == "true":
messages[-1].tool_status = "streaming"
except Exception as exc:
_logger.debug("[read] 读取第%d条异常:%s", i, exc)
continue

View File

@@ -0,0 +1,261 @@
# -*- coding: utf-8 -*-
"""
桌面 E2E 通用 helper所有 skill 的 desktop E2E 测试共用。
为什么独立出来:
- disburse-payroll-icbc 实战验证SDK 的 ask() 在 user_idx<0 时有死循环 bug
且 send_file() 是 NotImplementedError。需要绕开它们自写完整测试流程
- 这些 helper拖拽附件、等 user 节点、多信号合流等待 agent 完成)是
**所有 skill 都通用的**,不应该每个 skill 都内嵌一份
设计原则:
- 所有 helper 只接受 Playwright Page 作为入口(不依赖 SDK client 实例)
- 时间常量默认值留出宽容user message 60s / agent complete 1200s
- 失败一律抛 TimeoutError 或 AssertionError由调用方决定怎么截屏/上报
依赖匠厂 DOM 锚点docs/JIANGCHANG_CUSTOM_ANCHORS.md 守护):
- [data-testid="chat-composer-input"]
- [data-testid="chat-page"][data-sending]
- [data-role="user"] / [data-role="assistant"]
- [data-testid="chat-execution-graph"][data-collapsed]
- window.__jc_sending__
"""
from __future__ import annotations
import base64
import re
import time
from pathlib import Path
from typing import Optional, Tuple
from playwright.sync_api import Page
__all__ = [
"drop_file_into_composer",
"wait_for_attachment_ready",
"wait_for_composer_enabled",
"send_prompt_via_composer",
"wait_for_user_message",
"wait_for_agent_complete",
]
def _normalize_ws(text: str) -> str:
return re.sub(r"\s+", "", text)
# ─── 附件上传(绕开 SDK send_file 未实现) ──────────────────────────
def drop_file_into_composer(page: Page, file_path: Path) -> None:
"""通过 DragEvent + DataTransfer 把文件拖入聊天输入框。
匠厂 ChatInput 的 onDrop 挂在外层 div 上,但 React 17+ 事件代理到 root
所以在 textarea 上派发 bubbles:true 的 drop 事件能正常触发 stageBufferFiles
→ /api/files/stage-buffer → 真实附件链路。
"""
payload_b64 = base64.b64encode(file_path.read_bytes()).decode("ascii")
page.evaluate(
"""
({ base64, fileName, mimeType }) => {
const binary = atob(base64);
const bytes = new Uint8Array(binary.length);
for (let i = 0; i < binary.length; i++) {
bytes[i] = binary.charCodeAt(i);
}
const file = new File([bytes], fileName, { type: mimeType });
const dt = new DataTransfer();
dt.items.add(file);
const textarea = document.querySelector('[data-testid="chat-composer-input"]');
if (!textarea) {
throw new Error('chat-composer-input not found');
}
const opts = { bubbles: true, cancelable: true, dataTransfer: dt };
textarea.dispatchEvent(new DragEvent('dragenter', opts));
textarea.dispatchEvent(new DragEvent('dragover', opts));
textarea.dispatchEvent(new DragEvent('drop', opts));
}
""",
{
"base64": payload_b64,
"fileName": file_path.name,
"mimeType": _guess_mime(file_path),
},
)
def _guess_mime(file_path: Path) -> str:
suffix = file_path.suffix.lower()
return {
".xlsx": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
".xls": "application/vnd.ms-excel",
".csv": "text/csv",
".pdf": "application/pdf",
".png": "image/png",
".jpg": "image/jpeg",
".jpeg": "image/jpeg",
".txt": "text/plain",
".json": "application/json",
}.get(suffix, "application/octet-stream")
def wait_for_attachment_ready(page: Page, file_name: str, timeout_s: float = 30.0) -> None:
"""stage-buffer 成功后附件 chip 会展示 fileName轮询直到可见。"""
deadline = time.time() + timeout_s
name_loc = page.get_by_text(file_name, exact=False)
while time.time() < deadline:
try:
n = name_loc.count()
for i in range(n):
if name_loc.nth(i).is_visible():
return
except Exception:
pass
time.sleep(0.5)
raise TimeoutError(
f"附件 {file_name!r}{timeout_s}s 内未变为 ready"
)
# ─── 发送 prompt绕开 SDK ask 的 15s 锁死) ────────────────────────
def wait_for_composer_enabled(page: Page, timeout_s: float = 15.0) -> None:
"""等输入框可见且非 disabled避免 canSend=false 静默 no-op"""
page.wait_for_selector(
'[data-testid="chat-composer-input"]',
state="visible",
timeout=int(timeout_s * 1000),
)
textarea = page.locator('[data-testid="chat-composer-input"]')
deadline = time.time() + timeout_s
while time.time() < deadline:
if not textarea.evaluate("(el) => el.disabled"):
return
time.sleep(0.3)
raise TimeoutError(
f"chat-composer-input 在 {timeout_s}s 内仍为 disabledgateway 可能未就绪)"
)
def send_prompt_via_composer(page: Page, prompt: str, *, delay_ms: int = 25) -> None:
"""拟人输入 + Enter 发送。不做任何等待,返回即调用结束。"""
textarea = page.locator('[data-testid="chat-composer-input"]')
textarea.click()
textarea.fill("")
textarea.press_sequentially(prompt, delay=delay_ms)
textarea.press("Enter")
def wait_for_user_message(page: Page, prompt: str, timeout_s: float = 60.0) -> int:
"""等我们这条 user 消息进入 DOM返回它在 [data-role] 列表里的 index。
绕开 SDK ask() 只给 15s 窗口的脆弱判据:用 prompt 文本反向匹配,
宽容到 60s。脏 sessionnew_task 后 gateway 载回历史)也能正确定位。
"""
prompt_norm = _normalize_ws(prompt)
deadline = time.time() + timeout_s
items = page.locator('[data-role]')
while time.time() < deadline:
total = items.count()
for i in range(total - 1, -1, -1):
node = items.nth(i)
role = (node.get_attribute("data-role") or "").lower()
if role != "user":
continue
text = node.inner_text(timeout=2000) or ""
text_norm = _normalize_ws(text)
if prompt_norm in text_norm or text_norm in prompt_norm:
return i
time.sleep(0.5)
raise TimeoutError(
f"{timeout_s}s 内未找到匹配 user 消息prompt 归一化前 80 字="
f"{prompt_norm[:80]!r})"
)
# ─── 等 agent 完成(多信号合流) ────────────────────────────────
def _chat_sending_false(page: Page) -> bool:
val = page.locator('[data-testid="chat-page"]').first.get_attribute("data-sending")
return val is not None and val.lower() == "false"
def _window_sending_not_true(page: Page) -> bool:
val = page.evaluate("() => window.__jc_sending__")
return val is not True
def _no_expanded_execution_graph(page: Page) -> bool:
return (
page.locator('[data-testid="chat-execution-graph"][data-collapsed="false"]').count()
== 0
)
def _latest_assistant_after(page: Page, user_idx: int) -> Optional[Tuple[int, str, bool]]:
items = page.locator('[data-role]')
total = items.count()
for i in range(total - 1, user_idx, -1):
node = items.nth(i)
role = (node.get_attribute("data-role") or "").lower()
if role != "assistant":
continue
body = node.inner_text(timeout=2000) or ""
streaming = (node.get_attribute("data-streaming") or "false").lower() == "true"
return i, body, streaming
return None
def wait_for_agent_complete(
page: Page,
user_idx: int,
*,
overall_timeout_s: float = 1200.0,
stable_seconds: float = 5.0,
poll_interval: float = 1.0,
) -> str:
"""多信号合流等本轮 agent 完成,返回最末 assistant 气泡文本。
完成条件(全部满足并稳定 stable_seconds 秒):
1) chat-page data-sending="false"
2) window.__jc_sending__ 不为 True
3) 不存在未折叠的 execution graph
4) user_idx 之后存在 assistant 节点,有非空 bodydata-streaming != "true"
5) 上面 assistant 节点的文本长度连续 stable_seconds 秒无变化
"""
deadline = time.time() + overall_timeout_s
last_body_len = -1
last_change_at = time.time()
while time.time() < deadline:
chat_ok = _chat_sending_false(page)
win_ok = _window_sending_not_true(page)
graph_ok = _no_expanded_execution_graph(page)
picked = _latest_assistant_after(page, user_idx)
body_text = ""
streaming = True
has_body = False
if picked is not None:
_, body_text, streaming = picked
has_body = len(body_text.strip()) > 0
all_ready = chat_ok and win_ok and graph_ok and has_body and not streaming
if not all_ready:
last_body_len = -1
last_change_at = time.time()
else:
cur_len = len(body_text)
if cur_len != last_body_len:
last_body_len = cur_len
last_change_at = time.time()
elif time.time() - last_change_at >= stable_seconds:
return body_text
time.sleep(poll_interval)
raise TimeoutError(
f"agent 在 {overall_timeout_s}s 内未完成user_idx={user_idx} "
f"last_body_len={last_body_len}"
)

View File

@@ -0,0 +1,85 @@
# -*- coding: utf-8 -*-
"""
Win32 窗口工具:跨 SDK / screencast 共享的"激活并最大化匠厂主窗口"实现。
为什么单独抽出来:
- SDK 的 bring_to_front 需要 OS 级窗口前置page.bring_to_front 只切 tab
- screencast 的 run_screencast 录屏前也要把匠厂浮到前面并铺满屏幕
- 两者用同一份 EnumWindows + ShowWindow(SW_MAXIMIZE) + SetForegroundWindow 实现,
避免代码重复维护
跨平台行为:
- Windows: 真正执行
- 其他系统: 直接返回 False调用方按需 fallback 到协议激活)
"""
from __future__ import annotations
import logging
import os
from typing import Optional, Tuple
_logger = logging.getLogger("jiangchang_desktop_sdk.window_win32")
_DEFAULT_KEYWORDS: Tuple[str, ...] = ("匠厂", "Jiangchang", "ClawX", "OpenClaw")
SW_MAXIMIZE = 3
def activate_and_maximize_jiangchang_window(
keywords: Optional[Tuple[str, ...]] = None,
) -> bool:
"""Windows: EnumWindows 找匠厂主窗口 → ShowWindow(SW_MAXIMIZE) + SetForegroundWindow。
Args:
keywords: 标题模糊匹配的关键词;不传则用默认
("匠厂", "Jiangchang", "ClawX", "OpenClaw")
Returns:
True: 找到窗口并执行了 ShowWindow + SetForegroundWindow
False: 非 Windows / ctypes 不可用 / 找不到匹配窗口 / 任何异常
"""
if os.name != "nt":
return False
try:
import ctypes
from ctypes import wintypes
except Exception as exc:
_logger.debug("ctypes 不可用:%s", exc)
return False
user32 = ctypes.windll.user32
kw = keywords or _DEFAULT_KEYWORDS
matching: list = []
@ctypes.WINFUNCTYPE(ctypes.c_bool, wintypes.HWND, wintypes.LPARAM)
def _enum_proc(hwnd, _lparam):
if not user32.IsWindowVisible(hwnd):
return True
length = user32.GetWindowTextLengthW(hwnd)
if length == 0:
return True
buf = ctypes.create_unicode_buffer(length + 1)
user32.GetWindowTextW(hwnd, buf, length + 1)
if any(k in buf.value for k in kw):
matching.append((hwnd, buf.value))
return True
try:
user32.EnumWindows(_enum_proc, 0)
except Exception as exc:
_logger.debug("EnumWindows 失败:%s", exc)
return False
if not matching:
_logger.debug("未找到匠厂主窗口keywords=%s", kw)
return False
hwnd, title = matching[0]
try:
user32.ShowWindow(hwnd, SW_MAXIMIZE)
user32.SetForegroundWindow(hwnd)
_logger.debug("已激活+最大化 hwnd=%s title=%s", hwnd, title)
return True
except Exception as exc:
_logger.debug("ShowWindow/SetForegroundWindow 失败:%s", exc)
return False

View File

@@ -0,0 +1,3 @@
from .runner import run_screencast
__all__ = ["run_screencast"]

View File

@@ -0,0 +1,80 @@
"""FFmpeg 合成:帧序列 + SRT 字幕 + 背景音乐 → MP4。"""
from __future__ import annotations
import random
import subprocess
import sys
from pathlib import Path
# 匠厂统一字幕样式 —— 所有 skill 录屏共用,做品牌一致性
# FontSize 141080p 屏幕约占 1.3% 高度,演示视频留足画面呼吸感
# Outline 214 号字配 Outline 3 会显胖2 已足够在浅色背景上保持可读
# 修改请评估对所有 skill 录屏的影响。
_JIANGCHANG_SUBTITLE_STYLE = (
"FontName=Arial,FontSize=14,"
"PrimaryColour=&H00FFFFFF,OutlineColour=&H00000000,"
"Outline=2,Shadow=1,Alignment=2"
)
def compose_video(
frames_dir: str,
fps: int,
subtitle_path: str,
music_dir: str,
output_path: str,
music_volume: float = 0.15,
) -> str:
frames_dir = Path(frames_dir)
subtitle_path = Path(subtitle_path)
output_path = Path(output_path)
output_path.parent.mkdir(parents=True, exist_ok=True)
# 随机选一首背景音乐
music_files = list(Path(music_dir).rglob("*.mp3"))
if not music_files:
raise FileNotFoundError(f"music_dir 下没有找到 MP3 文件: {music_dir}")
music_file = random.choice(music_files)
print(f"[screencast] 背景音乐: {music_file.name}")
# FFmpeg 字幕路径在 Windows 下需要转义冒号subtitles 滤镜内部语法)
srt_path_str = str(subtitle_path).replace("\\", "/").replace(":", "\\:")
subtitle_filter = (
f"subtitles='{srt_path_str}'"
f":force_style='{_JIANGCHANG_SUBTITLE_STYLE}'"
)
# 视频和音频都放进 filter_complex避免 -vf 与 -filter_complex 混用报错
filter_complex = (
f"[0:v]{subtitle_filter}[vout];"
f"[1:a]volume={music_volume},apad[aout]"
)
cmd = [
"ffmpeg", "-y",
"-framerate", str(fps),
"-i", str(frames_dir / "frame_%06d.png"),
"-i", str(music_file),
"-filter_complex", filter_complex,
"-map", "[vout]",
"-map", "[aout]",
"-c:v", "libx264",
"-preset", "fast",
"-crf", "23",
"-c:a", "aac",
"-b:a", "128k",
"-shortest",
"-pix_fmt", "yuv420p",
str(output_path),
]
print(f"[screencast] 运行 FFmpeg 合成...")
result = subprocess.run(cmd, capture_output=True, text=True, encoding="utf-8", errors="replace")
if result.returncode != 0:
print(result.stderr, file=sys.stderr)
raise RuntimeError(f"FFmpeg 合成失败,退出码 {result.returncode}")
print(f"[screencast] 输出: {output_path}")
return str(output_path)

View File

@@ -0,0 +1,63 @@
"""全屏截帧引擎,后台线程持续截图存为 PNG 帧序列。"""
from __future__ import annotations
import threading
import time
from pathlib import Path
from typing import Optional
import mss
import mss.tools
class ScreenRecorder:
def __init__(
self,
frames_dir: str,
fps: int = 10,
monitor_index: Optional[int] = None,
):
"""
Args:
monitor_index: mss 显示器索引None=monitors[0](所有屏合并区域,
旧行为1=主屏2+=第 N 屏。多显示器场景推荐显式传 1。
"""
self._frames_dir = Path(frames_dir)
self._fps = fps
self._monitor_index = monitor_index
self._stop_event = threading.Event()
self._thread: threading.Thread | None = None
self._frame_count = 0
def start(self) -> None:
self._frames_dir.mkdir(parents=True, exist_ok=True)
self._stop_event.clear()
self._frame_count = 0
self._thread = threading.Thread(target=self._capture_loop, daemon=True)
self._thread.start()
def stop(self) -> None:
self._stop_event.set()
if self._thread:
self._thread.join(timeout=10)
@property
def frame_count(self) -> int:
return self._frame_count
def _capture_loop(self) -> None:
interval = 1.0 / self._fps
with mss.mss() as sct:
idx = self._monitor_index if self._monitor_index is not None else 0
if idx < 0 or idx >= len(sct.monitors):
idx = 0 # 越界兜底
monitor = sct.monitors[idx]
while not self._stop_event.is_set():
t0 = time.perf_counter()
img = sct.grab(monitor)
path = self._frames_dir / f"frame_{self._frame_count:06d}.png"
mss.tools.to_png(img.rgb, img.size, output=str(path))
self._frame_count += 1
elapsed = time.perf_counter() - t0
sleep = max(0.0, interval - elapsed)
time.sleep(sleep)

View File

@@ -0,0 +1 @@
mss>=9.0.1

140
tools/screencast/runner.py Normal file
View File

@@ -0,0 +1,140 @@
"""主编排器:录屏 + 运行 pytest + 生成字幕 + FFmpeg 合成。"""
from __future__ import annotations
import os
import subprocess
import sys
import tempfile
import time
from datetime import datetime
from pathlib import Path
from typing import List, Optional, Tuple
def _ensure_sdk_on_path() -> None:
"""record_screencast.py 直接 python 跑时不走 conftestSDK 可能不在 sys.path。
自带兜底:从 tools/screencast/runner.py 上溯到 platform-kit/src。"""
import importlib.util
if importlib.util.find_spec("jiangchang_desktop_sdk") is not None:
return
# tools/screencast/runner.py → tools/screencast → tools → platform-kit
here = Path(__file__).resolve()
src = here.parent.parent.parent / "src"
if (src / "jiangchang_desktop_sdk").is_dir():
import sys as _sys
_sys.path.insert(0, str(src))
_ensure_sdk_on_path()
from jiangchang_desktop_sdk.window_win32 import activate_and_maximize_jiangchang_window # noqa: E402
from .composer import compose_video
from .recorder import ScreenRecorder
from .subtitle import SubtitleEngine
def run_screencast(
skill_slug: str,
subtitle_script: List[Tuple[str, str]],
pytest_args: List[str],
output_dir: str,
media_assets_root: Optional[str] = None,
music_subdir: str = "music",
fps: int = 10,
activate_window: bool = True,
monitor_index: Optional[int] = None,
) -> str:
"""
录制一次技能桌面 E2E 演示视频。
Args:
skill_slug: 技能唯一标识,如 "query-balance-icbc"
subtitle_script: [(关键词, 字幕文案), ...] 按顺序匹配 pytest stdout
pytest_args: 传递给 pytest 的参数列表
output_dir: 最终 MP4 输出目录
media_assets_root: media-assets 仓库本地路径;
不传时读 MEDIA_ASSETS_ROOT 环境变量,
再不存在则用默认值 D:\\OpenClaw\\client-commons\\media-assets
music_subdir: music_assets_root 下音乐子目录,默认 "music"
fps: 截帧帧率,默认 10
activate_window: 录屏开始前是否自动激活+最大化匠厂窗口(默认 True
Windows 走 ctypes非 Windows 走 jiangchang:// 协议)
monitor_index: 指定录哪个显示器None=所有显示器合并(旧行为),
1=主屏2+=第 N 屏(推荐多显示器场景显式传 1
Returns:
输出 MP4 的绝对路径
"""
if media_assets_root is None:
media_assets_root = os.environ.get(
"MEDIA_ASSETS_ROOT",
r"D:\OpenClaw\client-commons\media-assets",
)
music_dir = str(Path(media_assets_root) / music_subdir)
output_dir_path = Path(output_dir)
output_dir_path.mkdir(parents=True, exist_ok=True)
date_str = datetime.now().strftime("%Y%m%d_%H%M%S")
base_name = f"{skill_slug}_{date_str}"
output_mp4 = output_dir_path / f"{base_name}.mp4"
subtitle_file = output_dir_path / f"{base_name}.srt"
engine = SubtitleEngine(subtitle_script)
with tempfile.TemporaryDirectory(prefix="screencast_frames_") as frames_dir:
recorder = ScreenRecorder(frames_dir, fps=fps, monitor_index=monitor_index)
if activate_window:
ok = activate_and_maximize_jiangchang_window()
if ok:
print("[screencast] 已激活+最大化匠厂窗口")
time.sleep(2) # 等窗口浮起稳定
else:
print("[screencast] 未能激活匠厂窗口,继续录制(可能录到桌面边角)")
print(f"[screencast] 启动录制 → {output_mp4.name}")
recorder.start()
engine.set_start_time(time.time())
try:
# Windows 子进程 stdout 默认走系统 ANSI code page中文 Windows 是 cp936/GBK
# 这边用 encoding="utf-8" 解码会乱码。注入 PYTHONIOENCODING + PYTHONUTF8 让
# 子进程 Python 强制以 UTF-8 输出,两端编码对齐。非 Windows 默认就 UTF-8无副作用。
env = os.environ.copy()
env["PYTHONIOENCODING"] = "utf-8"
env["PYTHONUTF8"] = "1"
proc = subprocess.Popen(
[sys.executable, "-m", "pytest", *pytest_args],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
encoding="utf-8",
errors="replace",
env=env,
)
assert proc.stdout is not None
for line in proc.stdout:
sys.stdout.write(line)
sys.stdout.flush()
engine.process_line(line)
exit_code = proc.wait()
if exit_code != 0:
print(f"[screencast] pytest 退出码 {exit_code},录制仍继续合成")
finally:
recorder.stop()
print(f"[screencast] 录制停止,共 {recorder.frame_count}")
engine.generate_srt(str(subtitle_file))
print(f"[screencast] 字幕 → {subtitle_file.name}")
compose_video(
frames_dir=frames_dir,
fps=fps,
subtitle_path=str(subtitle_file),
music_dir=music_dir,
output_path=str(output_mp4),
)
return str(output_mp4)

View File

@@ -0,0 +1,94 @@
"""字幕引擎:根据 stdout 关键词打时间戳,生成 SRT 字幕文件。"""
from __future__ import annotations
import time
from dataclasses import dataclass
from pathlib import Path
from typing import List, Tuple
@dataclass
class _Entry:
start: float # 距录制开始的秒数
text: str
duration: float = 5.0 # 每条字幕默认显示 5 秒
class SubtitleEngine:
def __init__(
self,
script: List[Tuple],
default_duration: float = 5.0,
):
"""
script: [(关键词, 字幕文案) | (关键词, 字幕文案, 显示时长秒), ...]
- 二元组:用 default_duration向后兼容
- 三元组:本条字幕独立指定显示秒数(用于长等待场景延长字幕停留)
关键词大小写不敏感,每个关键词只触发一次。
"""
self._script = script
self._default_duration = default_duration
self._entries: List[_Entry] = []
self._matched: set[str] = set()
self._t0: float = 0.0
def set_start_time(self, t: float) -> None:
self._t0 = t
def process_line(self, line: str) -> None:
lower = line.lower()
for entry_tuple in self._script:
if len(entry_tuple) == 2:
keyword, text = entry_tuple
duration = self._default_duration
elif len(entry_tuple) == 3:
keyword, text, duration = entry_tuple
else:
continue
if keyword in self._matched:
continue
if keyword.lower() in lower:
elapsed = time.time() - self._t0
self._entries.append(_Entry(start=elapsed, text=text, duration=float(duration)))
self._matched.add(keyword)
def generate_srt(self, output_path: str) -> None:
"""生成 SRT 文件,自动处理字幕重叠。
字幕重叠修复策略(避免屏幕上同时显示 2+ 条叠成两行):
- 字幕按 start 时间升序排序后;
- 相邻字幕保留 GAP=0.1s 间隔;
- 每条字幕保证至少 MIN_DURATION=1.0s 显示时间(短到这个值仍读不完是字幕脚本设计问题);
- 若当前条让位给下一条后剩余时间 < MIN_DURATION则把下一条 start 往后推。
这样任意时刻屏幕上最多 1 条字幕。
"""
def _fmt(s: float) -> str:
h = int(s // 3600)
m = int((s % 3600) // 60)
sec = int(s % 60)
ms = int((s % 1) * 1000)
return f"{h:02d}:{m:02d}:{sec:02d},{ms:03d}"
# 防重叠 post-process
MIN_DURATION = 1.0
GAP = 0.1
entries = sorted(self._entries, key=lambda e: e.start)
for i in range(len(entries) - 1):
cur, nxt = entries[i], entries[i + 1]
max_end = nxt.start - GAP
new_dur = max_end - cur.start
if new_dur < MIN_DURATION:
# 太挤cur 至少撑 MIN_DURATION 秒,把 nxt 推迟
cur.duration = MIN_DURATION
nxt.start = cur.start + MIN_DURATION + GAP
else:
# 正常让位cur 显示到 (nxt.start - GAP)
cur.duration = min(cur.duration, new_dur)
# 最后一条不需要让位,保持原 duration
Path(output_path).parent.mkdir(parents=True, exist_ok=True)
with open(output_path, "w", encoding="utf-8") as f:
for i, e in enumerate(entries, 1):
f.write(f"{i}\n")
f.write(f"{_fmt(e.start)} --> {_fmt(e.start + e.duration)}\n")
f.write(f"{e.text}\n\n")