6 Commits

Author SHA1 Message Date
ffb64b2cfc feat(screencast): 统一字幕样式 / 修字幕重叠 / 修子进程乱码
- composer.py:抽出 _JIANGCHANG_SUBTITLE_STYLE 常量,FontSize 28→14 /
  Outline 2→2(保持),1080p 屏幕约占 1.3% 高度,演示视频留足画面呼吸感
- subtitle.py:generate_srt 加 post-process 排队,相邻字幕互相让位
  (MIN_DURATION=1.0s / GAP=0.1s),任意时刻屏幕上最多 1 条字幕,
  避免开头几秒密集触发的字幕叠成两行
- runner.py:subprocess.Popen 注入 PYTHONIOENCODING=utf-8 + PYTHONUTF8=1,
  让 Windows 子进程 Python 强制 UTF-8 输出,runner 端 UTF-8 解码即对齐,
  修中文乱码

所有 skill 录屏受益,向后兼容(非 Windows 系统 UTF-8 注入无副作用)。

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:14:03 +08:00
c8d136fb79 feat(sdk): bring_to_front 用 Win32 API 激活并最大化主窗口
新增 _activate_and_maximize_windows 私有方法:用 ctypes 调 EnumWindows
找匠厂主窗口(标题含 匠厂/Jiangchang/ClawX/OpenClaw),
ShowWindow(SW_MAXIMIZE) + SetForegroundWindow 让窗口铺满桌面。

bring_to_front 改 OS 分支:
- Windows: 优先 ctypes 激活+最大化(dev / 安装版通吃,比 jiangchang://
  协议在 dev 模式下因 cwd 解析失败而报错可靠);失败时退回协议作为兜底
- macOS / Linux: 维持原协议激活路径(open / xdg-open)

适用场景:录屏 / 自动化测试需要 OS 级窗口前置。page.bring_to_front()
只切 Playwright tab 不影响 z-order,必须 OS 级 API 才能把 Electron
主窗口拉到桌面最前。

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 10:13:43 +08:00
34f9a2c141 docs: Stage 2 SDK data-jcid migration report
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-18 16:05:16 +08:00
4fc4cfb802 refactor(sdk): drop data-jcid, switch to semantic anchors + ClawX testids
- Introduce centralized SELECTOR constant table at top of client.py
- Replace all [data-jcid=...] selectors with:
  * Semantic anchors (data-role / data-message-id / data-streaming /
    data-sending / data-message-count) guarded by jiangchang main repo
  * ClawX upstream data-testids (chat-page / chat-composer-* / etc.)
- _latest_assistant_after: has_body now judged by inner_text non-empty
  (no longer depends on message-body sub-element)
- _gateway_state: degrade to window.__jc_gateway_state__ readback,
  returns 'unknown' until host exposes it
- new_task: rely solely on data-message-count==0 (welcome-screen no
  longer has a testid)
- wait_gateway_ready: degrade to chat-page selector wait
- send_file: raise NotImplementedError (Electron IPC hook missing)

Public API signatures unchanged. window.__jc_sending__ remains the
core sending signal.

Compatibility: jiangchang v2.0.17+ only.
Refs: jiangchang/docs/sdk-migration-discovery.md
      jiangchang/docs/JIANGCHANG_CUSTOM_ANCHORS.md
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-18 16:05:04 +08:00
089f00c3ff feat(tools): add screencast recording engine
Adds tools/screencast/ — a reusable screen-recording pipeline for
skill demo videos. Wraps a pytest desktop E2E run with:
  - ScreenRecorder: mss-based full-screen frame capture (background thread)
  - SubtitleEngine: stdout keyword → timestamped SRT generation
  - compose_video: FFmpeg filter_complex merge (video + subtitles + BGM)
  - run_screencast: main orchestrator wiring all three layers

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-17 20:38:47 +08:00
20294fd23a 新增依赖 2026-05-06 19:33:46 +08:00
10 changed files with 730 additions and 124 deletions

View File

@@ -63,3 +63,57 @@ git push origin v0.1.0
``` ```
仓库内还保留 `examples/``tools/``python-runtime/` 目录及既有 Skill / 前端相关的 GitHub Actions 工作流,与本 PyPI 包的发布彼此独立。 仓库内还保留 `examples/``tools/``python-runtime/` 目录及既有 Skill / 前端相关的 GitHub Actions 工作流,与本 PyPI 包的发布彼此独立。
## v0.2.0 — data-jcid migration (internal refactor)
### TL;DR
本次内部重构:SDK 不再依赖匠厂主仓库的 `data-jcid` 属性。
所有公开 API(`ask` / `read` / `new_task` / `wait_gateway_ready` 等)
**签名与行为保持不变**,只有内部选择器实现切换到新的语义锚点 + ClawX testid。
### 改变了什么
- `client.py` 内所有 `[data-jcid="..."]` 选择器已移除
- 改用匠厂主仓库为 SDK 承诺的语义锚点:`data-role` / `data-message-id` /
`data-streaming`(在 ChatMessage 上)、`data-sending` / `data-message-count`
(在 chat-page 根上)
- `window.__jc_sending__` 仍是核心 sending 信号,保持不变
- 这些锚点由匠厂主仓库的以下机制守护:
- `docs/JIANGCHANG_CUSTOM_ANCHORS.md` 中央清单
- 源码内 `JIANGCHANG CUSTOM ANCHOR` 边界注释
- `harness/specs/jiangchang-custom-anchors.spec.ts` E2E 守护测试
- Gitea CI grep 检查 step
### 已知限制
- `send_file()` 暂未实现,抛 `NotImplementedError`
原因:附件上传走 Electron IPC,SDK 需要主仓库提供新 hook。
- `_gateway_state()` 永远返回 `'unknown'`
原因:gateway 状态不再暴露 DOM data-state;本方法已预留未来通过
`window.__jc_gateway_state__` 读取的实现路径。
这两个限制对常规 ask/read 工作流**无影响**,sending 完成判定依靠
`window.__jc_sending__` + `data-sending` + 文本稳定性合流即可。
### 升级影响
- 在 jiangchang v2.0.17(含)以上版本上跑:应正常工作
- 在 jiangchang v2.0.16(及更早含 data-jcid 的版本)上跑:不再兼容
如果需要在旧版本上跑,固定 SDK 版本到 v0.1.x
### 验证清单(集成方建议跑一次)
打开匠厂客户端 → 用 SDK 跑下面 3 步,确认基础流程通:
```python
from jiangchang_desktop_sdk import JiangchangDesktopClient
with JiangchangDesktopClient() as c:
c.ensure_app_running()
c.wait_gateway_ready()
c.new_task()
answer = c.ask("你好,请回复确认 SDK 可用")
assert answer, f"Empty answer: {answer!r}"
print("OK:", answer[:200])
```

View File

@@ -0,0 +1,131 @@
# Stage 2 — SDK data-jcid Migration Report
## 1. 执行摘要
- 起始 HEAD: `089f00c3ff6b75125d77f40b2e2dcb664a9ac5bf`
- 结束 HEAD: 本阶段未 commit,工作区状态见下
- 影响文件:
- `src/jiangchang_desktop_sdk/client.py`(主要改动,7 步合计)
- `README.md`(新增 v0.2.0 章节)
- `STAGE2_SDK_MIGRATION_REPORT.md`(本回执)
- 其他例外文件:无(`types.py` / `exceptions.py` 未改)
## 2. 7 步改动总览
| 步 | 内容 | 影响范围 |
|----|------|---------|
| 1 | SELECTOR 常量表 | client.py 顶部新增 36 行 |
| 2 | _get_*_locator x4 + 2 处 raise 文案 | client.py 局部 |
| 3 | read() | client.py 局部 |
| 4 | 9 个私有 helper | client.py 多处 |
| 5 | _wait_for_streaming_done | client.py 局部(docstring + 注释) |
| 6 | new_task / wait_gateway_ready / send_file | client.py 局部 |
| 7 | README + 扫尾 | README.md + 验证 |
**第 7 步附带**: `ask()` 兜底分支仍残留可执行 `[data-jcid="message"]` / `data-jcid-role`,已改为 `SEL_MESSAGE_ROW` + `data-role`(否则 7.1 可执行代码 grep 无法通过)。
## 3. 公开 API 兼容性证明
逐项列出公开 API 改造前后签名,确认一字不差:
| 方法 | 改前签名 | 改后签名 | 一致? |
|------|---------|---------|-------|
| ensure_app_running | (self, wait_timeout_s: float = 30.0) -> None | 同左 | ✓ |
| bring_to_front | (self) -> None | 同左 | ✓ |
| connect | (self, url: Optional[str] = None) -> None | 同左 | ✓ |
| launch_app | (self, options: Optional[LaunchOptions] = None) -> None | 同左 | ✓ |
| disconnect | (self) -> None | 同左 | ✓ |
| is_connected | (self) -> bool | 同左 | ✓ |
| get_page | (self) -> Page | 同左 | ✓ |
| new_task | (self) -> None | 同左 | ✓ |
| wait_gateway_ready | (self, timeout_ms: int = 30000) -> None | 同左 | ✓ |
| wait_for_response | (self, timeout: int = 120000) -> None | 同左 | ✓ |
| ask | (self, question: str, options: Optional[AskOptions] = None) -> str | 同左 | ✓ |
| send_file | (self, file_path: str, message: Optional[str] = None) -> None | 同左(行为变 NotImplementedError) | ✓ |
| read | (self) -> List[JiangchangMessage] | 同左 | ✓ |
| assert_contains | (self, expected: str, options: Optional[AssertOptions] = None) -> None | 同左 | ✓ |
| snapshot | (self, target_dir: str, tag: str = "snapshot") -> dict | 同左 | ✓ |
| __enter__ | (self) -> "JiangchangDesktopClient" | 同左 | ✓ |
| __exit__ | (self, exc_type, exc_val, exc_tb) -> None | 同左 | ✓ |
## 4. 临时限制说明
### send_file
- 状态:抛 NotImplementedError
- 原因:Electron IPC stage-paths,SDK 无 DOM 入口
- 恢复条件:匠厂主仓库暴露 `window.__jc_stage_paths__` 或带 testid 的隐藏 input
### _gateway_state
- 状态:永远返回 'unknown'
- 原因:HEAD 上 gateway 状态不暴露 DOM
- 行为影响:_wait_for_streaming_done 中 `if gw not in ("running", "unknown")`
实质 no-op,但保留以便未来生效
- 恢复条件:匠厂在 main / store 暴露 `window.__jc_gateway_state__`
## 5. 扫尾验证实际输出
### 5.1 `grep -n 'data-jcid' src/jiangchang_desktop_sdk/client.py`
```
src/jiangchang_desktop_sdk/client.py:10:3. 每次提问前新建任务(点击侧栏 `data-jcid="new-task-button"`),避免上下文污染。 (模块 docstring 历史描述)
src/jiangchang_desktop_sdk/client.py:14: 这些 DOM 节点 **不再带 `data-jcid="message"`**,而是 `data-testid="chat-execution-graph"` (模块 docstring 历史描述)
src/jiangchang_desktop_sdk/client.py:18: · **锚点**`ask()` 发送后等到最后一条 `[data-jcid-role="user"]` 节点(就是我们刚发的 (模块 docstring 历史描述)
src/jiangchang_desktop_sdk/client.py:19: 问题)在 DOM 里出现,记住它在 `[data-jcid="message"]` 列表里的 index作为切片起点 (模块 docstring 历史描述)
src/jiangchang_desktop_sdk/client.py:21: · `chat-root[data-jcid-sending] === 'false'` (模块 docstring 历史描述)
src/jiangchang_desktop_sdk/client.py:25: · 我们这条 user 之后存在 `[data-jcid-role="assistant"]` 且其内部有 (模块 docstring 历史描述)
src/jiangchang_desktop_sdk/client.py:26: `[data-jcid="message-body"]` (模块 docstring 历史描述)
src/jiangchang_desktop_sdk/client.py:29:5. read() 严格按 `data-jcid-role` 分类,没有该属性的 DOM 节点不再默认归为 assistant。 (模块 docstring 历史描述)
src/jiangchang_desktop_sdk/client.py:526: """返回 `[data-jcid="message"]` 节点列表中最后一条 role=user 的 index找不到返回 -1。""" (docstring 历史描述)
src/jiangchang_desktop_sdk/client.py:561: "streaming": bool, # data-jcid-streaming 属性 (docstring 历史描述)
src/jiangchang_desktop_sdk/client.py:610: """chat-root[data-jcid-sending] 的 bool 化;读不到返回 None。""" (docstring 历史描述)
src/jiangchang_desktop_sdk/client.py:796: "我们刚发出的 user 节点在 [data-jcid=\"message\"] 列表里的 index"。后续所有 (ask docstring 历史描述)
```
可执行代码(非 docstring / 非 `#` 注释):**0 命中**。
### 5.2 `grep -n '__jc_sending__' src/jiangchang_desktop_sdk/client.py`
```
src/jiangchang_desktop_sdk/client.py:22: · `window.__jc_sending__` !== true
src/jiangchang_desktop_sdk/client.py:633: """从 window.__jc_sending__ 读取当前 sending 状态;不可用返回 None。"""
src/jiangchang_desktop_sdk/client.py:636: val = page.evaluate("() => window.__jc_sending__")
src/jiangchang_desktop_sdk/client.py:673: C同时 window.__jc_sending__ 不是 true允许 None / False以容忍
```
### 5.3 import 可用性
```
import OK
```
## 6. 集成验证
- 是否在本地跑过 README 末尾的 3 步验证流程?
- [ ] 跑过 → 结果(pass/fail + 关键输出片段)
- [x] 未跑 → 原因:验收环境未启动匠厂客户端 / 本阶段仅完成代码与静态扫尾
- 是否运行了 platform-kit 自带的任何测试?
- [ ] 跑过 → 结果
- [x] 未跑 → 原因:本包无针对 desktop_sdk 的自动化集成测试;需实机 CDP
## 7. 未触及的事项
- `examples/``tools/screencast/`:未在本阶段验证;预期不直接依赖已移除的 data-jcid 选择器
- platform-kit stash 中的 `runtime_env.py` 改动:本阶段未碰,仍在 stash(`stash@{0}: wip: claw env var fallback...`)
- `D:\AI\jiangchang` 主仓库:未修改
- 模块级 / 部分 helper / `ask()` docstring 中 data-jcid 历史措辞:保留,见 §9
## 8. 自检
- [x] grep 'data-jcid' 在可执行代码中 → 0 命中
- [x] 公开 API 签名表逐项核对一致
- [x] send_file 抛 NotImplementedError 而非静默
- [x] _gateway_state 含 TODO 注释
- [x] README 新增 v0.2.0 章节,无格式错乱
- [x] 未触碰 D:\AI\jiangchang 主仓库任何文件
- [x] 未触碰 platform-kit 非 client.py / README.md / 本回执 的文件
- [ ] git status 显示工作区干净(除本阶段预期修改 + 已有 stash) → 见 §1,未 commit
## 9. 给验收方的备注
1. **docstring 历史 data-jcid**: 模块顶 docstring(L10L29)、`_last_user_node_index`(L526)、`_latest_assistant_after` 内联说明(L561)、`_chat_root_sending`(L610)、`ask()` docstring(L796) 仍描述旧 UI 模型;不影响运行。若需文档一致性,可单独开文档清理 PR。
2. **send_file**: 调用即 `NotImplementedError`,无连接检查(按规格整段替换方法体)。
3. **wait_gateway_ready**: 仅等待 `SEL_CHAT_PAGE` 可见,不保证 Gateway 进程已 ready。
4. **建议验收**: 在已安装 v2.0.17+ 匠厂 + Playwright 环境跑 README 末尾脚本;并确认 `send_file(...)` 按预期抛错。

View File

@@ -12,6 +12,7 @@ license = "MIT"
authors = [{ name = "client-jiangchang" }] authors = [{ name = "client-jiangchang" }]
dependencies = [ dependencies = [
"requests>=2.31.0", "requests>=2.31.0",
"playwright>=1.42.0",
] ]
[project.urls] [project.urls]

View File

@@ -54,6 +54,42 @@ from .exceptions import (
) )
from .types import AskOptions, AssertOptions, JiangchangMessage, LaunchOptions from .types import AskOptions, AssertOptions, JiangchangMessage, LaunchOptions
# ============================================================================
# JIANGCHANG DOM SELECTORS
# ============================================================================
# 本节集中维护所有 Playwright 选择器,作为对匠厂前端的契约。
# 选择器分两类:
#
# (A) 主仓库语义锚点(由 docs/JIANGCHANG_CUSTOM_ANCHORS.md 守护,稳定)
# 这些是匠厂为 SDK 暴露的承诺契约,被 E2E 守护测试 + Gitea CI 双重保护。
#
# (B) 主仓库 ClawX 上游 data-testid(随上游同步可能漂移)
# 这些是 ClawX 上游或我们扩展的测试钩子,本身用于内部 E2E,
# SDK 借用但不保证不变。
#
# 修改前请确认 docs/sdk-migration-discovery.md 中对应元素仍存在。
# ============================================================================
# --- (A) 匠厂语义锚点 ---
SEL_CHAT_PAGE = '[data-testid="chat-page"]'
SEL_CHAT_PAGE_SENDING = '[data-testid="chat-page"][data-sending="true"]'
SEL_MESSAGE_ROW = '[data-role]' # ChatMessage 行容器
SEL_MESSAGE_ROW_USER = '[data-role="user"]'
SEL_MESSAGE_ROW_ASSISTANT = '[data-role="assistant"]'
SEL_MESSAGE_ROW_STREAMING = '[data-streaming="true"]'
# --- (B) ClawX/上游 data-testid ---
SEL_SIDEBAR_NEW_CHAT = '[data-testid="sidebar-new-chat"]'
SEL_CHAT_COMPOSER_INPUT = '[data-testid="chat-composer-input"]'
SEL_CHAT_COMPOSER_SEND = '[data-testid="chat-composer-send"]'
SEL_CHAT_MESSAGE_INDEXED = '[data-testid^="chat-message-"]' # chat-message-${idx}
SEL_EXECUTION_GRAPH = '[data-testid="chat-execution-graph"]'
SEL_EXECUTION_GRAPH_EXPANDED = '[data-testid="chat-execution-graph"][data-collapsed="false"]'
SEL_MAIN_LAYOUT = '[data-testid="main-layout"]'
# --- (C) 通用兜底(无 testid 时使用) ---
SEL_FALLBACK_TEXTAREA = 'textarea'
# ── 调试日志 ─────────────────────────────────────────────────────── # ── 调试日志 ───────────────────────────────────────────────────────
_logger = logging.getLogger("jiangchang_desktop_sdk") _logger = logging.getLogger("jiangchang_desktop_sdk")
_handler_added = False _handler_added = False
@@ -182,6 +218,65 @@ class JiangchangDesktopClient:
_logger.warning("[activate] 协议激活失败:%s", exc) _logger.warning("[activate] 协议激活失败:%s", exc)
return False return False
def _activate_and_maximize_windows(self) -> bool:
"""Windows 专属:用 Win32 API 把主窗口拉到桌面最前并最大化。
EnumWindows 遍历所有顶层窗口,按标题模糊匹配(含"匠厂/Jiangchang/ClawX/OpenClaw"
找到后 ShowWindow(SW_MAXIMIZE) + SetForegroundWindow让录屏看到铺满桌面的匠厂
避免任务栏边角和其他应用露出来。
为什么改用 ctypes替代上一版 PowerShell AppActivate
1) AppActivate 只激活不最大化,依然能录到桌面边缘其他窗口;
2) ctypes 不 fork powershell启动更快~0ms vs ~300-800ms
3) 直接拿到 hwnd 后续可扩展(如监控窗口尺寸)。
"""
if os.name != "nt":
return False
try:
import ctypes
from ctypes import wintypes
except Exception as exc:
_logger.debug("[bring_to_front] ctypes 不可用:%s", exc)
return False
user32 = ctypes.windll.user32
SW_MAXIMIZE = 3
keywords = ("匠厂", "Jiangchang", "ClawX", "OpenClaw")
matching: list = []
@ctypes.WINFUNCTYPE(ctypes.c_bool, wintypes.HWND, wintypes.LPARAM)
def _enum_proc(hwnd, _lparam):
if not user32.IsWindowVisible(hwnd):
return True
length = user32.GetWindowTextLengthW(hwnd)
if length == 0:
return True
buf = ctypes.create_unicode_buffer(length + 1)
user32.GetWindowTextW(hwnd, buf, length + 1)
if any(kw in buf.value for kw in keywords):
matching.append((hwnd, buf.value))
return True
try:
user32.EnumWindows(_enum_proc, 0)
except Exception as exc:
_logger.debug("[bring_to_front] EnumWindows 失败:%s", exc)
return False
if not matching:
_logger.debug("[bring_to_front] 未找到匠厂主窗口")
return False
hwnd, title = matching[0]
try:
user32.ShowWindow(hwnd, SW_MAXIMIZE)
user32.SetForegroundWindow(hwnd)
_logger.debug("[bring_to_front] 已激活+最大化 hwnd=%s title=%s", hwnd, title)
return True
except Exception as exc:
_logger.debug("[bring_to_front] ShowWindow/SetForegroundWindow 失败:%s", exc)
return False
def _launch_via_executable(self, cdp_port: int) -> bool: def _launch_via_executable(self, cdp_port: int) -> bool:
"""回退:直接 spawn 可执行文件,带 --remote-debugging-port。""" """回退:直接 spawn 可执行文件,带 --remote-debugging-port。"""
exe = self._get_default_executable_path() exe = self._get_default_executable_path()
@@ -234,12 +329,38 @@ class JiangchangDesktopClient:
) )
def bring_to_front(self) -> None: def bring_to_front(self) -> None:
"""把主窗口聚焦到最前。""" """把主窗口聚焦到桌面最前。
分两层:
1) page.bring_to_front() —— Playwright 把 page 切到当前 context 内的最前
tab。**不影响 OS 级窗口 z-order**。
2) OS 级窗口前置(录屏 / 自动化必需):
- Windows: 用 ctypes 调 Win32 EnumWindows 找匠厂主窗口 →
ShowWindow(SW_MAXIMIZE) + SetForegroundWindow让窗口铺满桌面
(详见 _activate_and_maximize_windows。失败时退回协议激活作为
兜底,但 dev 模式pnpm dev / electron .)下协议本身会因 cwd 解析
失败而无效,只有安装版协议兜底才可能成功。
- macOS / Linux: 走 jiangchang:// 协议open / xdg-open
非 Windows 系统下协议处理器没有 cwd 解析问题)。
"""
try: try:
page = self.get_page() page = self.get_page()
page.bring_to_front() page.bring_to_front()
except Exception as exc: except Exception as exc:
_logger.debug("[bring_to_front] 失败:%s", exc) _logger.debug("[bring_to_front] Playwright 切 page 失败:%s", exc)
if os.name == "nt":
if self._activate_and_maximize_windows():
return
try:
self._activate_via_protocol()
except Exception as exc:
_logger.debug("[bring_to_front] 协议激活失败:%s", exc)
else:
try:
self._activate_via_protocol()
except Exception as exc:
_logger.debug("[bring_to_front] 协议激活失败:%s", exc)
# ─── 连接 / 断开 ────────────────────────────────────────────── # ─── 连接 / 断开 ──────────────────────────────────────────────
@@ -375,51 +496,46 @@ class JiangchangDesktopClient:
def _get_chat_input_locator(self): def _get_chat_input_locator(self):
page = self.get_page() page = self.get_page()
locator = page.locator('[data-jcid="chat-input"]') locator = page.locator(SEL_CHAT_COMPOSER_INPUT)
if locator.count() > 0: if locator.count() > 0:
return locator return locator
locator = page.locator("textarea").last locator = page.locator(SEL_FALLBACK_TEXTAREA).last
if locator.count() > 0: if locator.count() > 0:
return locator return locator
raise ConnectionError("找不到聊天输入框textarea请确认匠厂已加 data-jcid=\"chat-input\"") raise ConnectionError(
'找不到聊天输入框,请确认匠厂已加 data-testid="chat-composer-input" '
"或页面中存在 textarea"
)
def _get_send_button_locator(self): def _get_send_button_locator(self):
page = self.get_page() page = self.get_page()
locator = page.locator('[data-jcid="send-button"]') locator = page.locator(SEL_CHAT_COMPOSER_SEND)
if locator.count() > 0: if locator.count() > 0:
return locator return locator
locator = page.locator('button[type="submit"]') locator = page.locator('button[type="submit"]')
if locator.count() > 0: if locator.count() > 0:
return locator return locator
raise ConnectionError("找不到发送按钮,请确认匠厂已加 data-jcid=\"send-button\"") raise ConnectionError(
'找不到发送按钮,请确认匠厂已加 data-testid="chat-composer-send" '
'或 button[type="submit"]'
)
def _get_new_task_button_locator(self): def _get_new_task_button_locator(self):
page = self.get_page() page = self.get_page()
locator = page.locator('[data-jcid="new-task-button"]') locator = page.locator(SEL_SIDEBAR_NEW_CHAT)
if locator.count() > 0: if locator.count() > 0:
return locator return locator
# 回退:旧版本 data-testid
locator = page.locator('[data-testid="sidebar-new-chat"]')
if locator.count() > 0:
return locator
# 最后兜底:文本"新建任务"
locator = page.get_by_text("新建任务", exact=False).first locator = page.get_by_text("新建任务", exact=False).first
return locator return locator
def _get_message_list_locator(self): def _get_message_list_locator(self):
page = self.get_page() page = self.get_page()
locator = page.locator('[data-jcid="message-list"]') return page.locator(SEL_CHAT_PAGE)
if locator.count() > 0:
return locator
locator = page.locator('[class*="space-y-3"]').first
if locator.count() > 0:
return locator
raise ConnectionError("找不到消息列表容器 data-jcid=\"message-list\"")
# ─── 新建任务 ───────────────────────────────────────────────── # ─── 新建任务 ─────────────────────────────────────────────────
def new_task(self) -> None: def new_task(self) -> None:
"""点击"新建任务"按钮,等待消息列表清空或欢迎屏出现""" """点击"新建任务"按钮,等待消息列表清空chat-page data-message-count=0"""
if not self._connected: if not self._connected:
raise ConnectionError("未连接到应用") raise ConnectionError("未连接到应用")
page = self.get_page() page = self.get_page()
@@ -431,16 +547,12 @@ class JiangchangDesktopClient:
_logger.warning("[new_task] 点击失败:%s", exc) _logger.warning("[new_task] 点击失败:%s", exc)
return return
# 等待 chat-root.message-count=0 或 welcome-screen 出现,最多 5s
deadline = time.time() + 5 deadline = time.time() + 5
while time.time() < deadline: while time.time() < deadline:
try: try:
count = page.locator('[data-jcid="chat-root"]').first.get_attribute("data-jcid-message-count") count = page.locator(SEL_CHAT_PAGE).first.get_attribute("data-message-count")
if count == "0": if count == "0":
_logger.debug("[new_task] 已进入新任务message-count=0") _logger.debug("[new_task] 已进入新任务(data-message-count=0")
return
if page.locator('[data-jcid="welcome-screen"]').count() > 0:
_logger.debug("[new_task] 欢迎屏已出现")
return return
except Exception: except Exception:
pass pass
@@ -450,25 +562,35 @@ class JiangchangDesktopClient:
# ─── 网关健康检查 ───────────────────────────────────────────── # ─── 网关健康检查 ─────────────────────────────────────────────
def _gateway_state(self) -> str: def _gateway_state(self) -> str:
"""Read gateway state via Electron IPC bridge (jiangchang exposes no DOM data-state).
Strategy: try to read window.__jc_gateway_state__ if exposed;
otherwise return 'unknown' and let upper logic continue.
"""
# TODO: 等匠厂在 src/stores/gateway/* 或 main 进程暴露
# window.__jc_gateway_state__ 之后,本方法将自动开始返回真实状态。
# 当前永远返回 'unknown',行为退化为"假定 gateway 健康"。
try: try:
page = self.get_page() state = self.get_page().evaluate(
el = page.locator('[data-jcid="gateway-status"]').first "() => window.__jc_gateway_state__"
if el.count() == 0: )
return "unknown" if isinstance(state, str) and state:
return el.get_attribute("data-state") or "unknown" return state
except Exception: except Exception:
return "unknown" pass
return 'unknown'
def wait_gateway_ready(self, timeout_ms: int = 30000) -> None: def wait_gateway_ready(self, timeout_ms: int = 30000) -> None:
"""等待匠厂前端就绪。
由于 v2.0.17 起 gateway 状态不再暴露 DOM data-state,本方法降级为
等待 chat-page 根元素可见(隐含 React + gateway store 已初始化)。
Gateway 实际就绪与否,由后续 ask() 内的 _gateway_state 检查兜底
(当前永远返回 'unknown')。
"""
page = self.get_page() page = self.get_page()
try: try:
page.wait_for_function( page.wait_for_selector(SEL_CHAT_PAGE, timeout=timeout_ms)
"""() => {
const el = document.querySelector('[data-jcid="gateway-status"]');
return el && el.dataset.state === 'running';
}""",
timeout=timeout_ms,
)
except Exception as exc: except Exception as exc:
_logger.warning("[wait_gateway_ready] 超时/异常:%s(继续,但后续可能报 GatewayDown", exc) _logger.warning("[wait_gateway_ready] 超时/异常:%s(继续,但后续可能报 GatewayDown", exc)
@@ -481,7 +603,7 @@ class JiangchangDesktopClient:
def _message_node_count(self) -> int: def _message_node_count(self) -> int:
try: try:
return self.get_page().locator('[data-jcid="message"]').count() return self.get_page().locator(SEL_MESSAGE_ROW).count()
except Exception: except Exception:
return 0 return 0
@@ -489,10 +611,10 @@ class JiangchangDesktopClient:
"""返回 `[data-jcid="message"]` 节点列表中最后一条 role=user 的 index找不到返回 -1。""" """返回 `[data-jcid="message"]` 节点列表中最后一条 role=user 的 index找不到返回 -1。"""
try: try:
page = self.get_page() page = self.get_page()
total = page.locator('[data-jcid="message"]').count() total = page.locator(SEL_MESSAGE_ROW).count()
for i in range(total - 1, -1, -1): for i in range(total - 1, -1, -1):
node = page.locator('[data-jcid="message"]').nth(i) node = page.locator(SEL_MESSAGE_ROW).nth(i)
role = node.get_attribute("data-jcid-role") or "" role = node.get_attribute("data-role") or ""
if role.lower() == "user": if role.lower() == "user":
return i return i
return -1 return -1
@@ -507,7 +629,7 @@ class JiangchangDesktopClient:
return False return False
try: try:
page = self.get_page() page = self.get_page()
node = page.locator('[data-jcid="message"]').nth(user_idx) node = page.locator(SEL_MESSAGE_ROW).nth(user_idx)
text = node.inner_text(timeout=2000) or "" text = node.inner_text(timeout=2000) or ""
t = re.sub(r"\s+", "", text) t = re.sub(r"\s+", "", text)
q = re.sub(r"\s+", "", question) q = re.sub(r"\s+", "", question)
@@ -529,22 +651,19 @@ class JiangchangDesktopClient:
return None return None
try: try:
page = self.get_page() page = self.get_page()
nodes = page.locator('[data-jcid="message"]') nodes = page.locator(SEL_MESSAGE_ROW)
total = nodes.count() total = nodes.count()
for i in range(total - 1, user_idx, -1): for i in range(total - 1, user_idx, -1):
node = nodes.nth(i) node = nodes.nth(i)
role = (node.get_attribute("data-jcid-role") or "").lower() role = (node.get_attribute("data-role") or "").lower()
if role != "assistant": if role != "assistant":
continue continue
body = node.locator('[data-jcid="message-body"]') try:
has_body = body.count() > 0 body_text = node.inner_text(timeout=2000) or ""
body_text = "" except Exception:
if has_body: body_text = ""
try: has_body = len(body_text.strip()) > 0
body_text = body.first.inner_text(timeout=2000) or "" streaming = (node.get_attribute("data-streaming") or "false").lower() == "true"
except Exception:
body_text = ""
streaming = (node.get_attribute("data-jcid-streaming") or "false").lower() == "true"
return { return {
"index": i, "index": i,
"has_body": has_body, "has_body": has_body,
@@ -575,8 +694,8 @@ class JiangchangDesktopClient:
def _chat_root_sending(self) -> Optional[bool]: def _chat_root_sending(self) -> Optional[bool]:
"""chat-root[data-jcid-sending] 的 bool 化;读不到返回 None。""" """chat-root[data-jcid-sending] 的 bool 化;读不到返回 None。"""
try: try:
val = self.get_page().locator('[data-jcid="chat-root"]').first.get_attribute( val = self.get_page().locator(SEL_CHAT_PAGE).first.get_attribute(
"data-jcid-sending" "data-sending"
) )
if val is None: if val is None:
return None return None
@@ -586,8 +705,8 @@ class JiangchangDesktopClient:
def _chat_root_message_count(self) -> Optional[int]: def _chat_root_message_count(self) -> Optional[int]:
try: try:
val = self.get_page().locator('[data-jcid="chat-root"]').first.get_attribute( val = self.get_page().locator(SEL_CHAT_PAGE).first.get_attribute(
"data-jcid-message-count" "data-message-count"
) )
if val is None: if val is None:
return None return None
@@ -626,16 +745,16 @@ class JiangchangDesktopClient:
""" """
等待 agent 整轮(可能包含多次 thinking / tool_use 往返)完成。 等待 agent 整轮(可能包含多次 thinking / tool_use 往返)完成。
核心判据 —— 只有 agent 最终答复才有 `message-body`(文本气泡) 核心判据 —— agent 最终答复在 data-role=assistant 行上有非空 inner_text
源码依据D:\\AI\\jiangchang/src/stores/chat/helpers.ts 源码依据D:\\AI\\jiangchang/src/stores/chat/helpers.ts
- isToolOnlyMessage 显式允许 thinking 与 tool_use 共存仍视为中间轮 - isToolOnlyMessage 显式允许 thinking 与 tool_use 共存仍视为中间轮
- 中间工具轮的 ChatMessage 无 hasText → 不渲染 MessageBubble → - 中间工具轮可能无可见正文气泡 → data-role 行 inner_text 为空
也就不会有 `data-jcid="message-body"` - 只有最终含 text 的助手消息行 inner_text 非空has_body
- 只有最终含 text 的助手消息才会渲染 message-body
因此可靠的完成条件为: 因此可靠的完成条件为:
A最后一条 assistant message 存在 [data-jcid="message-body"] Auser_msg_index 之后最后一条 assistant 的 has_body 为 True
B其文本连续 `stable_seconds` 秒无变化 (由 _latest_assistant_after 根据 inner_text 判定)
B其 body_text 连续 `stable_seconds` 秒无变化;
C同时 window.__jc_sending__ 不是 true允许 None / False以容忍 C同时 window.__jc_sending__ 不是 true允许 None / False以容忍
早期 React 未挂载注入钩子的情况)。 早期 React 未挂载注入钩子的情况)。
@@ -675,10 +794,10 @@ class JiangchangDesktopClient:
assistant = self._latest_assistant_after(user_msg_index) assistant = self._latest_assistant_after(user_msg_index)
# 完成判据需同时满足: # 完成判据需同时满足:
# (1) 三路 sending 信号全部静默chat-root / window / 未收起的 execution graph # (1) 三路 sending 信号全部静默chat-page data-sending / window / 未收起的 execution graph
# (2) user_msg_index 之后存在 assistant 消息 # (2) user_msg_index 之后存在 assistant 消息data-role 行)
# (3) 该 assistant 存在 message-body # (3) 该 assistant has_body 为 True_latest_assistant_afterinner_text 非空)
# (4) 该 assistant 的 data-jcid-streaming != 'true' # (4) 该 assistant 的 data-streaming != 'true'
# (5) body 文本连续 stable_seconds 秒无变化 # (5) body 文本连续 stable_seconds 秒无变化
can_check_stable = ( can_check_stable = (
not sending_any not sending_any
@@ -740,7 +859,7 @@ class JiangchangDesktopClient:
raise JcTimeoutError( raise JcTimeoutError(
f"等待 AI 回复超时({timeout_ms}ms。user_msg_index={user_msg_index} 最后阶段 phase={phase}" f"等待 AI 回复超时({timeout_ms}ms。user_msg_index={user_msg_index} 最后阶段 phase={phase}"
"可能原因:模型思考时间过长 / 工具链循环未结束 / UI 未挂载 message-body / Gateway 卡住。" "可能原因:模型思考时间过长 / 工具链循环未结束 / assistant 正文未出现data-role 行 inner_text 为空)/ Gateway 卡住。"
) )
# ─── 核心交互ask / read / assert ──────────────────────────── # ─── 核心交互ask / read / assert ────────────────────────────
@@ -844,70 +963,56 @@ class JiangchangDesktopClient:
poll_interval=opts.poll_interval, poll_interval=opts.poll_interval,
) )
# 6. 选答:只在 my_user_idx 之后的 assistant 中找带 message-body 的最后一条 # 6. 选答:只在 my_user_idx 之后的 assistant 中找 has_body 的最后一条
picked = self._latest_assistant_after(my_user_idx) picked = self._latest_assistant_after(my_user_idx)
if picked is None or not picked["has_body"]: if picked is None or not picked["has_body"]:
_logger.warning( _logger.warning(
"[ask] user_idx=%d 之后未找到带 message-body 的 assistantpicked=%s", "[ask] user_idx=%d 之后未找到 has_body 的 assistantpicked=%s",
my_user_idx, picked, my_user_idx, picked,
) )
# 兜底:取我们 user 之后的任意最后一条 assistant inner_text # 兜底:取我们 user 之后的任意最后一条 assistant inner_text
try: try:
page = self.get_page() page = self.get_page()
nodes = page.locator('[data-jcid="message"]') nodes = page.locator(SEL_MESSAGE_ROW)
total = nodes.count() total = nodes.count()
for i in range(total - 1, my_user_idx, -1): for i in range(total - 1, my_user_idx, -1):
node = nodes.nth(i) node = nodes.nth(i)
if (node.get_attribute("data-jcid-role") or "").lower() == "assistant": if (node.get_attribute("data-role") or "").lower() == "assistant":
return node.inner_text(timeout=2000) or "" return node.inner_text(timeout=2000) or ""
except Exception: except Exception:
pass pass
return "" return ""
_logger.debug( _logger.debug(
"[ask] 选定 assistant_idx=%duser_idx=%d 之后第一条带 message-bodylen=%d", "[ask] 选定 assistant_idx=%duser_idx=%d 之后 has_bodylen=%d",
picked["index"], my_user_idx, len(picked["body_text"]), picked["index"], my_user_idx, len(picked["body_text"]),
) )
return picked["body_text"] return picked["body_text"]
def send_file(self, file_path: str, message: Optional[str] = None) -> None: def send_file(self, file_path: str, message: Optional[str] = None) -> None:
if not self._connected: """[暂未实现] 发送文件附件。
raise ConnectionError("未连接到应用")
page = self.get_page()
attach_btn = page.locator('[data-jcid="attach-button"]') 匠厂 v2.0.17(ClawX v0.4.3 合并后)的附件上传通道由原生 Electron
if attach_btn.count() == 0: showOpenDialog + IPC stage-paths 实现,不再暴露 HTML <input type="file">。
attach_btn = page.locator("button").filter(
has_text=re.compile(r"附件|上传|paperclip|attach", re.I)
)
file_input = page.locator('input[type="file"]') Playwright 无法直接触发 Electron 原生对话框。需要匠厂主仓库新增:
if file_input.count() == 0 and attach_btn.count() > 0: - window.__jc_stage_paths__(paths: string[]) 调用桥,或
attach_btn.first.click() - 一个隐藏 <input type="file"> 备用通道,带 data-testid
time.sleep(0.5)
file_input = page.locator('input[type="file"]')
if file_input.count() == 0: 在匠厂提供其中之一之前,本方法抛 NotImplementedError 而非静默成功。
raise AssertError(
"找不到文件上传 input[type='file']。请确认匠厂已加 data-jcid=\"attach-button\""
)
abs_path = os.path.abspath(file_path) Tracking: 见 docs/sdk-migration-discovery.md §7
if not os.path.exists(abs_path): """
raise AppNotFoundError(f"上传文件不存在: {abs_path}") raise NotImplementedError(
"send_file is temporarily unavailable on jiangchang v2.0.17+. "
file_input.set_input_files(abs_path) "Reason: attachments now go through Electron IPC; SDK requires "
time.sleep(1) "a new DOM-or-window hook from the host application. "
"See docs/sdk-migration-discovery.md §7."
if message: )
input_locator = self._get_chat_input_locator()
input_locator.click()
input_locator.press_sequentially(message, delay=25)
input_locator.press("Enter")
def read(self) -> List[JiangchangMessage]: def read(self) -> List[JiangchangMessage]:
""" """
严格读取:仅接受带 data-jcid="message" + data-jcid-role 节点。 严格读取:仅接受带 data-role 语义锚点的 ChatMessage 节点。
无 role 的节点返回 role='unknown'(不再默认为 assistant 无 role 的节点返回 role='unknown'(不再默认为 assistant
""" """
if not self._connected: if not self._connected:
@@ -917,41 +1022,35 @@ class JiangchangDesktopClient:
messages: List[JiangchangMessage] = [] messages: List[JiangchangMessage] = []
try: try:
msg_list = self._get_message_list_locator() page.locator(SEL_CHAT_PAGE).wait_for(state="visible", timeout=5000)
msg_list.wait_for(state="visible", timeout=5000)
except Exception as exc: except Exception as exc:
_logger.debug("[read] 消息列表不可见:%s", exc) _logger.debug("[read] 聊天页不可见:%s", exc)
return messages return messages
items = page.locator('[data-jcid="message"]') items = page.locator(SEL_MESSAGE_ROW)
total = items.count() total = items.count()
_logger.debug("[read] data-jcid=\"message\" 节点数=%d", total) _logger.debug("[read] data-role 消息行数=%d", total)
for i in range(total): for i in range(total):
el = items.nth(i) el = items.nth(i)
try: try:
role_attr = el.get_attribute("data-jcid-role") role_attr = el.get_attribute("data-role")
streaming_attr = el.get_attribute("data-jcid-streaming") or "false" streaming_attr = el.get_attribute("data-streaming") or "false"
# 优先从 message-body正文气泡取干净文本避免带入 Thinking / ToolCards content = el.inner_text(timeout=2000)
body_locator = el.locator('[data-jcid="message-body"]')
if body_locator.count() > 0:
content = body_locator.first.inner_text(timeout=2000)
else:
content = el.inner_text(timeout=2000)
role = (role_attr or "unknown").lower() role = (role_attr or "unknown").lower()
tool_status = "streaming" if streaming_attr == "true" else None
messages.append( messages.append(
JiangchangMessage( JiangchangMessage(
id=el.get_attribute("data-jcid-message-id") or f"msg-{i}", id=el.get_attribute("data-message-id") or f"msg-{i}",
role=role, role=role,
content=content, content=content,
timestamp=time.time(), timestamp=time.time(),
is_error=(el.get_attribute("data-jcid-error") == "true"), is_error=False,
tool_name=el.get_attribute("data-jcid-tool"), tool_name=None,
tool_status=tool_status,
) )
) )
if streaming_attr == "true":
messages[-1].tool_status = "streaming"
except Exception as exc: except Exception as exc:
_logger.debug("[read] 读取第%d条异常:%s", i, exc) _logger.debug("[read] 读取第%d条异常:%s", i, exc)
continue continue

View File

@@ -0,0 +1,3 @@
from .runner import run_screencast
__all__ = ["run_screencast"]

View File

@@ -0,0 +1,80 @@
"""FFmpeg 合成:帧序列 + SRT 字幕 + 背景音乐 → MP4。"""
from __future__ import annotations
import random
import subprocess
import sys
from pathlib import Path
# 匠厂统一字幕样式 —— 所有 skill 录屏共用,做品牌一致性
# FontSize 141080p 屏幕约占 1.3% 高度,演示视频留足画面呼吸感
# Outline 214 号字配 Outline 3 会显胖2 已足够在浅色背景上保持可读
# 修改请评估对所有 skill 录屏的影响。
_JIANGCHANG_SUBTITLE_STYLE = (
"FontName=Arial,FontSize=14,"
"PrimaryColour=&H00FFFFFF,OutlineColour=&H00000000,"
"Outline=2,Shadow=1,Alignment=2"
)
def compose_video(
frames_dir: str,
fps: int,
subtitle_path: str,
music_dir: str,
output_path: str,
music_volume: float = 0.15,
) -> str:
frames_dir = Path(frames_dir)
subtitle_path = Path(subtitle_path)
output_path = Path(output_path)
output_path.parent.mkdir(parents=True, exist_ok=True)
# 随机选一首背景音乐
music_files = list(Path(music_dir).rglob("*.mp3"))
if not music_files:
raise FileNotFoundError(f"music_dir 下没有找到 MP3 文件: {music_dir}")
music_file = random.choice(music_files)
print(f"[screencast] 背景音乐: {music_file.name}")
# FFmpeg 字幕路径在 Windows 下需要转义冒号subtitles 滤镜内部语法)
srt_path_str = str(subtitle_path).replace("\\", "/").replace(":", "\\:")
subtitle_filter = (
f"subtitles='{srt_path_str}'"
f":force_style='{_JIANGCHANG_SUBTITLE_STYLE}'"
)
# 视频和音频都放进 filter_complex避免 -vf 与 -filter_complex 混用报错
filter_complex = (
f"[0:v]{subtitle_filter}[vout];"
f"[1:a]volume={music_volume},apad[aout]"
)
cmd = [
"ffmpeg", "-y",
"-framerate", str(fps),
"-i", str(frames_dir / "frame_%06d.png"),
"-i", str(music_file),
"-filter_complex", filter_complex,
"-map", "[vout]",
"-map", "[aout]",
"-c:v", "libx264",
"-preset", "fast",
"-crf", "23",
"-c:a", "aac",
"-b:a", "128k",
"-shortest",
"-pix_fmt", "yuv420p",
str(output_path),
]
print(f"[screencast] 运行 FFmpeg 合成...")
result = subprocess.run(cmd, capture_output=True, text=True, encoding="utf-8", errors="replace")
if result.returncode != 0:
print(result.stderr, file=sys.stderr)
raise RuntimeError(f"FFmpeg 合成失败,退出码 {result.returncode}")
print(f"[screencast] 输出: {output_path}")
return str(output_path)

View File

@@ -0,0 +1,48 @@
"""全屏截帧引擎,后台线程持续截图存为 PNG 帧序列。"""
from __future__ import annotations
import threading
import time
from pathlib import Path
import mss
import mss.tools
class ScreenRecorder:
def __init__(self, frames_dir: str, fps: int = 10):
self._frames_dir = Path(frames_dir)
self._fps = fps
self._stop_event = threading.Event()
self._thread: threading.Thread | None = None
self._frame_count = 0
def start(self) -> None:
self._frames_dir.mkdir(parents=True, exist_ok=True)
self._stop_event.clear()
self._frame_count = 0
self._thread = threading.Thread(target=self._capture_loop, daemon=True)
self._thread.start()
def stop(self) -> None:
self._stop_event.set()
if self._thread:
self._thread.join(timeout=10)
@property
def frame_count(self) -> int:
return self._frame_count
def _capture_loop(self) -> None:
interval = 1.0 / self._fps
with mss.mss() as sct:
monitor = sct.monitors[0] # 全屏monitor[0] 是所有显示器合并区域)
while not self._stop_event.is_set():
t0 = time.perf_counter()
img = sct.grab(monitor)
path = self._frames_dir / f"frame_{self._frame_count:06d}.png"
mss.tools.to_png(img.rgb, img.size, output=str(path))
self._frame_count += 1
elapsed = time.perf_counter() - t0
sleep = max(0.0, interval - elapsed)
time.sleep(sleep)

View File

@@ -0,0 +1 @@
mss>=9.0.1

108
tools/screencast/runner.py Normal file
View File

@@ -0,0 +1,108 @@
"""主编排器:录屏 + 运行 pytest + 生成字幕 + FFmpeg 合成。"""
from __future__ import annotations
import os
import subprocess
import sys
import tempfile
import time
from datetime import datetime
from pathlib import Path
from typing import List, Optional, Tuple
from .composer import compose_video
from .recorder import ScreenRecorder
from .subtitle import SubtitleEngine
def run_screencast(
skill_slug: str,
subtitle_script: List[Tuple[str, str]],
pytest_args: List[str],
output_dir: str,
media_assets_root: Optional[str] = None,
music_subdir: str = "music",
fps: int = 10,
) -> str:
"""
录制一次技能桌面 E2E 演示视频。
Args:
skill_slug: 技能唯一标识,如 "query-balance-icbc"
subtitle_script: [(关键词, 字幕文案), ...] 按顺序匹配 pytest stdout
pytest_args: 传递给 pytest 的参数列表
output_dir: 最终 MP4 输出目录
media_assets_root: media-assets 仓库本地路径;
不传时读 MEDIA_ASSETS_ROOT 环境变量,
再不存在则用默认值 D:\\OpenClaw\\client-commons\\media-assets
music_subdir: music_assets_root 下音乐子目录,默认 "music"
fps: 截帧帧率,默认 10
Returns:
输出 MP4 的绝对路径
"""
if media_assets_root is None:
media_assets_root = os.environ.get(
"MEDIA_ASSETS_ROOT",
r"D:\OpenClaw\client-commons\media-assets",
)
music_dir = str(Path(media_assets_root) / music_subdir)
output_dir_path = Path(output_dir)
output_dir_path.mkdir(parents=True, exist_ok=True)
date_str = datetime.now().strftime("%Y%m%d_%H%M%S")
base_name = f"{skill_slug}_{date_str}"
output_mp4 = output_dir_path / f"{base_name}.mp4"
subtitle_file = output_dir_path / f"{base_name}.srt"
engine = SubtitleEngine(subtitle_script)
with tempfile.TemporaryDirectory(prefix="screencast_frames_") as frames_dir:
recorder = ScreenRecorder(frames_dir, fps=fps)
print(f"[screencast] 启动录制 → {output_mp4.name}")
recorder.start()
engine.set_start_time(time.time())
try:
# Windows 子进程 stdout 默认走系统 ANSI code page中文 Windows 是 cp936/GBK
# 这边用 encoding="utf-8" 解码会乱码。注入 PYTHONIOENCODING + PYTHONUTF8 让
# 子进程 Python 强制以 UTF-8 输出,两端编码对齐。非 Windows 默认就 UTF-8无副作用。
env = os.environ.copy()
env["PYTHONIOENCODING"] = "utf-8"
env["PYTHONUTF8"] = "1"
proc = subprocess.Popen(
[sys.executable, "-m", "pytest", *pytest_args],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
encoding="utf-8",
errors="replace",
env=env,
)
assert proc.stdout is not None
for line in proc.stdout:
sys.stdout.write(line)
sys.stdout.flush()
engine.process_line(line)
exit_code = proc.wait()
if exit_code != 0:
print(f"[screencast] pytest 退出码 {exit_code},录制仍继续合成")
finally:
recorder.stop()
print(f"[screencast] 录制停止,共 {recorder.frame_count}")
engine.generate_srt(str(subtitle_file))
print(f"[screencast] 字幕 → {subtitle_file.name}")
compose_video(
frames_dir=frames_dir,
fps=fps,
subtitle_path=str(subtitle_file),
music_dir=music_dir,
output_path=str(output_mp4),
)
return str(output_mp4)

View File

@@ -0,0 +1,81 @@
"""字幕引擎:根据 stdout 关键词打时间戳,生成 SRT 字幕文件。"""
from __future__ import annotations
import time
from dataclasses import dataclass
from pathlib import Path
from typing import List, Tuple
@dataclass
class _Entry:
start: float # 距录制开始的秒数
text: str
duration: float = 5.0 # 每条字幕默认显示 5 秒
class SubtitleEngine:
def __init__(self, script: List[Tuple[str, str]], default_duration: float = 5.0):
"""
script: [(关键词, 字幕文案), ...]
关键词大小写不敏感,每个关键词只触发一次。
"""
self._script = script
self._default_duration = default_duration
self._entries: List[_Entry] = []
self._matched: set[str] = set()
self._t0: float = 0.0
def set_start_time(self, t: float) -> None:
self._t0 = t
def process_line(self, line: str) -> None:
lower = line.lower()
for keyword, text in self._script:
if keyword in self._matched:
continue
if keyword.lower() in lower:
elapsed = time.time() - self._t0
self._entries.append(_Entry(start=elapsed, text=text, duration=self._default_duration))
self._matched.add(keyword)
def generate_srt(self, output_path: str) -> None:
"""生成 SRT 文件,自动处理字幕重叠。
字幕重叠修复策略(避免屏幕上同时显示 2+ 条叠成两行):
- 字幕按 start 时间升序排序后;
- 相邻字幕保留 GAP=0.1s 间隔;
- 每条字幕保证至少 MIN_DURATION=1.0s 显示时间(短到这个值仍读不完是字幕脚本设计问题);
- 若当前条让位给下一条后剩余时间 < MIN_DURATION则把下一条 start 往后推。
这样任意时刻屏幕上最多 1 条字幕。
"""
def _fmt(s: float) -> str:
h = int(s // 3600)
m = int((s % 3600) // 60)
sec = int(s % 60)
ms = int((s % 1) * 1000)
return f"{h:02d}:{m:02d}:{sec:02d},{ms:03d}"
# 防重叠 post-process
MIN_DURATION = 1.0
GAP = 0.1
entries = sorted(self._entries, key=lambda e: e.start)
for i in range(len(entries) - 1):
cur, nxt = entries[i], entries[i + 1]
max_end = nxt.start - GAP
new_dur = max_end - cur.start
if new_dur < MIN_DURATION:
# 太挤cur 至少撑 MIN_DURATION 秒,把 nxt 推迟
cur.duration = MIN_DURATION
nxt.start = cur.start + MIN_DURATION + GAP
else:
# 正常让位cur 显示到 (nxt.start - GAP)
cur.duration = min(cur.duration, new_dur)
# 最后一条不需要让位,保持原 duration
Path(output_path).parent.mkdir(parents=True, exist_ok=True)
with open(output_path, "w", encoding="utf-8") as f:
for i, e in enumerate(entries, 1):
f.write(f"{i}\n")
f.write(f"{_fmt(e.start)} --> {_fmt(e.start + e.duration)}\n")
f.write(f"{e.text}\n\n")