chore: remove tools/ui_inspector; point docs to jiangchang-ui-recorder.

Product lives at https://git.jc2009.com/client-jiangchang/jiangchang-ui-recorder; prior commit keeps an archive snapshot.

Co-authored-by: Cursor <cursoragent@cursor.com>

development/RPA_ELEMENT_LOCATING.md

@@ -43,6 +43,8 @@
 
 未完成第 2～6 步时，代码只能保留明确的 `TODO: NEED_REAL_UI_INSPECTION`，不得写“看起来可能正确”的 selector。
 
+开发期推荐使用独立仓 [jiangchang-ui-recorder](https://git.jc2009.com/client-jiangchang/jiangchang-ui-recorder)（本机常见路径 `D:\OpenClaw\client-jiangchang\jiangchang-ui-recorder`，`python run.py`）：以 **Raw Event Journal → Action → Element Catalog → 异步 Observation** 录制，优先把 `action_flow.json` 与 `element_catalog.json` 交给 AI 编程工具；原始事件可重新归并，tree/index 作为诊断证据。默认不再让完整页面 dump 阻塞动作采集。本模板仓内的 `tools/ui_inspector` 已移除，勿再从 skill-template 启动。
+
 ## 3. 元素清单
 
 每个关键输入、按钮、列表、弹窗、成功状态和错误状态都必须登记。推荐 YAML：

tools/README.md

@@ -1,13 +1,14 @@
 # skill-template 工具
 
-## 桌面 UIA 拾取（开发期）
+## UI 拾取（开发期）
 
-```powershell
+`tools/ui_inspector` 已迁出本模板仓。请使用独立产品仓：
-cd D:\OpenClaw\client-commons\skill-template
-python tools/ui_inspector/run.py --scene demo
-```
 
-说明见 [`ui_inspector/README.md`](ui_inspector/README.md)。企微暖树依赖本机 `jiangchang-platform-kit>=1.2.5`（与技能同调 `ensure_ui_ready`）。
+- Gitea：https://git.jc2009.com/client-jiangchang/jiangchang-ui-recorder
+- 本机常见路径：`D:\OpenClaw\client-jiangchang\jiangchang-ui-recorder`
+- 启动：`python run.py`
+
+模板仓历史中仍保留删除前的归档提交（可按需从 Git 恢复旧副本）；日常开发与运行以独立仓为准。
 
 ## 业务仓也可以直接 clone（来源 A）
 

tools/ui_inspector/.gitignore

@@ -1,7 +0,0 @@
-# 仅本工具目录使用；不修改 skill-template 根 .gitignore
-_captures/
-_browser_profile/
-user_prefs.json
-__pycache__/
-*.pyc
-.pytest_cache/

tools/ui_inspector/README.md

@@ -1,189 +0,0 @@
-# OpenClaw UI Inspector（UI Automation Recorder v0.16.0）
-
-默认窗口采用窄长控制台布局：停靠到鼠标所在显示器右侧，顶部预留目标应用标题栏安全区；也可在界面中一键切换到左侧。
-
-> **定位**：以人工操作为主线的开发期录制器，产出可交给 AI 编程工具的 Action Flow + Element Catalog。
-> **不是**：每 2 秒无条件 dump 页面的流水线（该行为仅保留为 Legacy 模式）。
-
-## 核心模型
-
-```text
-人工操作
-  → Raw Event Journal（先落事实，不阻塞）
-  → Action Reducer（去重、输入事务、pointerdown/click 合并）
-  → Target Resolver（距离 + 面积 + 语义）
-  → ActionRecord + Element Catalog
-  → 异步合并式 Observation / State / Screenshot
-```
-
-| 概念 | 作用 |
-|------|------|
-| **Action** | 流程主线（click / fill / press_key / checkpoint …） |
-| **State** | 动作前后上下文与证据（tree/index/screenshot） |
-| **Element Catalog** | 开发期与运行期共用的元素资产 |
-| **Locator** | 结构化属性 + 编译后的 expression；须验证匹配数 |
-| **原始 dump** | 诊断证据；不再把每次 heartbeat 登记为流程步骤 |
-| **Raw Event Journal** | 可重算的原始事实；Reducer 出错时无需重走人工流程 |
-
-## 启动
-
-```powershell
-cd D:\OpenClaw\client-commons\skill-template
-python tools/ui_inspector/run.py --scene demo
-```
-
-依赖：
-
-```powershell
-pip install -r tools/ui_inspector/requirements.txt
-pip install "jiangchang-platform-kit>=1.2.11"
-```
-
-## 录制流程
-
-主界面固定使用自动动作录制，只保留 **开始 / 暂停（继续）/ 结束**。Recorder 立即记录 Action，完整证据由独立后台队列异步合并采集。Inspector 自身窗口上的点击和输入不会写入目标 Action Flow。
-
-“运行状态与拾取结果”会对浏览器和桌面统一采用单步骤覆盖式显示；新动作替换上一动作，同一动作的异步证据返回后在原位置更新定位器、唯一性或失败原因。迟到的旧动作证据不会覆盖更新的步骤，完整历史始终保存在 `action_flow.json`、`actions/` 与 `element_catalog.json` 中。
-
-引导、手动检查点和 Legacy 间隔 dump 仅作为底层兼容与开发诊断能力保留，不进入普通用户流程；旧偏好会自动迁移回自动录制。
-
-## 运行时稳定性
-
-- Tk 主线程只处理界面；导航、Dump、截图、动作落盘均在后台执行。
-- 后台结果通过 UI 消息队列回传，后台线程不读取 Tk Variable 或 Widget。
-- 原始事件先追加到 `journal/raw_events.jsonl`，再进入语义 Action FIFO。
-- Action 热路径不执行完整页面 dump；慢速证据在独立队列按页面合并采集。
-- 异步证据标记为 `coalesced_eventual`，不会伪装成事件时刻的精确 after state。
-- Playwright 命令均有超时；导航失败不会被伪装成“已就绪”。
-- 浏览器使用 context 级事件桥并保留轮询回退，整页跳转前的 click 不随旧 document 丢失。
-
-## 拾取通道
-
-- **浏览器**：Inspector 自管 Playwright；起始 URL 可选；Browser Recorder（捕获阶段监听，不 preventDefault）
-  - 动作后的 Dump / 截图绑定事件来源 `page_id`；手动检查点才按当前可见标签页回退，不再固定 `pages[0]`
-  - 列表页新开详情标签后，切到详情再 dump/录制即可；关闭详情回到列表同理
-  - Recorder 会在所有已打开标签上安装监听并汇总事件
-  - 输入框、`contenteditable`、SVG/图标点击会沿 `composedPath` 提升到真实可操作目标；原始命中节点仍保留作诊断证据
-  - 语义定位不足时生成完整 CSS 路径，并在事件来源页面实时验证唯一性后才进入 Locator Catalog
-  - 链接保留 `href`，可编辑区生成稳定祖先作用域选择器；事件时刻的唯一性计数随快照保存，避免跳转后误判
-  - `press_key` 保留原始事件、Page/Document 身份和目标元素引用，不再依赖前一步推断焦点
-  - 页面跳转采用短暂因果重排，确保触发跳转的 click 先于 navigation；新标签同时保留 opener `page_id`
-  - 同时监听 `pointerdown` 与 `click`：正常 click 会去重；页面提前卸载时 pointerdown 作为保底 Action
-  - 通用目标评分限制祖先距离与面积，不会把小图标提升成整页容器
-  - 目标解析联合考虑语义、几何、事件时刻唯一性和定位器稳定性；SVG/path 仅作命中证据，优先选择唯一的可点击包装节点
-  - 输入事务与 Element Catalog 共用稳定元素身份，焦点/校验/悬停 class 变化不会拆分一次连续输入
-  - 输入事务使用浏览器源时间并保存全部 `raw_event_ids`，支持从原始日志审计和重算
-  - 完整 Dump / Screenshot 由后台 Evidence Coordinator 合并执行；手动检查点仍保留稳定采集
-- **桌面**：UIA；Windows Recorder 使用进程外 `WH_MOUSE_LL` / `WH_KEYBOARD_LL` 事件监听，点击后由独立线程执行 `ControlFromPoint`。Hook 不注入目标进程；`GetAsyncKeyState` 仅作为 Hook 无法安装时的降级路径。
-  - 输入使用焦点控件 Value/Text 变化聚合为一次 `fill`，不保存中文输入法的拼音按键；混合 DPI 应用会在键盘阶段重新探测正确坐标空间。
-  - 已注册的桌面适配器会在宣布“录制就绪”前完成必要的 Accessibility warm；企业微信首次启动可能多等十几秒，但不会再让首个输入事务与后台暖树竞争。
-  - 输入法候选/组合窗口只作为临时系统 UI 忽略，不会替换业务输入框；编辑器内的 Backspace/Delete 合并进最终 `fill`。
-  - `Enter`、`Tab` 等功能键优先继承刚才的文本编辑器目标，不再退化为整个根窗口。
-  - 鼠标按下时会枚举光标下可见顶层窗口，处理 owner window 吞掉浮窗命中的应用；后置树再把搜索/列表大容器提升到坐标命中的具体行。
-  - 超过 5 秒的延迟证据不能用动态消息摘要/预览改写旧点击；坐标不可靠且只有一个可见结果行时，才使用唯一结果行作为保守回退。
-  - 原始键盘活动只写入不含内容的事务标记；若发生输入但没有生成 `fill/type`，质量检查会明确告警。
-- **手机**：未实现，不得宣称已支持
-
-## 输出目录（双写兼容）
-
-```text
-_captures/<ts>_<flow>/
-  flow.json              # v1 schema；可选 v2 指针字段
-  SUMMARY.md
-  action_flow.json       # v2
-  element_catalog.json   # v2
-  pages.json             # Page / Document 记录
-  journal/raw_events.jsonl
-  diagnostics/metrics.json
-  diagnostics/validation.json
-  diagnostics/replay_report.json  # 静态、非破坏性的回放就绪预检
-  steps/NN_xxx/          # v1：meta/tree/index/locatable/screen/SUMMARY
-  states/<id>/           # v2：证据
-  observations/<id>/     # v2：每次真实采集证据（不去重）
-  actions/<id>.json
-  elements/<id>.json
-```
-
-旧读取方只认 `flow.json` + `steps/` 仍可工作。`flow.json` 中的 `v2` 字段可忽略。
-
-### Schema 名
-
-- `openclaw.ui_inspector.state.v1`
-- `openclaw.ui_inspector.observation.v1`
-- `openclaw.ui_inspector.action.v1`
-- `openclaw.ui_inspector.element.v1`
-- `openclaw.ui_inspector.element_catalog.v1`
-- `openclaw.ui_inspector.action_flow.v1`
-- `openclaw.ui_inspector.flow.v1`（原有）
-
-## 交给 AI 编程工具
-
-优先提供：
-
-1. `action_flow.json` — 动作序列、目标和证据关联；异步证据不冒充精确 after state
-2. `element_catalog.json` — logical_name、结构化 locator 候选、唯一性结果
-3. 需要排障时再附对应 `states/<id>/` 或旧 `steps/` 下的 tree/locatable/screen
-
-不要只丢一堆无动作语义的定时 dump。
-
-`diagnostics.capture_validator` 会报告缺失的生产级定位器、目标元素引用、原始事件关联、根窗口按键目标、列表大容器点击及丢失的输入事务，避免“动作录到了”被误判为“可以稳定生成代码”。
-
-## 当前能力边界
-
-| 能力 | 状态 |
-|------|------|
-| click | 已实现（browser + windows） |
-| fill/type 聚合 | 已实现 |
-| press_key（Enter/Tab/Esc 等） | 已实现 |
-| checkpoint | 已实现 |
-| navigation | 部分（URL 变化观察） |
-| 其余动作类型 | Schema 预留，`capability=schema_only` |
-| 手机录制 | **未实现** |
-| 截图像素级脱敏 | 未实现（敏感页在 meta 标记 `may_contain_sensitive_data`） |
-
-## 敏感信息
-
-- 密码框 / token / cookie / 手机号 / 邮箱等会脱敏，不写入 action_flow / catalog 明文。
-- 截图无法可靠脱敏时标记风险；可用配置关闭敏感页截图保存。
-- **禁止**在测试或产物中使用真实客户数据。
-
-## WeCom Adapter
-
-企业微信专用规则在 `adapters/wecom.py`：
-
-- `WeWorkWindow` → profile `wecom`
-- `SearchResultWindow2` → profile `wecom_search`
-- Warm **cache key = profile + hwnd**，主窗 warm 不会假装搜索浮层已暖
-- Recorder **捕获点击目标前不执行全量 warm**；warm 属于 dump/checkpoint/live 验证路径
-
-## Legacy 间隔模式
-
-Legacy 能力仅供底层兼容与开发排障，不在标准界面暴露。
-默认动作驱动模式下，heartbeat 只做输入聚合刷新，**相同 fingerprint 不新增 State，不产生 Action**。
-
-## 冒烟与单测
-
-```powershell
-python tools/ui_inspector/smoke_check.py
-python -m pytest tools/ui_inspector/tests -q
-```
-
-## 目录结构
-
-```text
-ui_inspector/
-  core/                  # 三端共用模型、管线、策略
-  surfaces/              # browser / windows / future mobile
-  evidence/              # 异步证据协调
-  locators/              # Locator Catalog 与稳定性
-  storage/               # 新旧采集格式读写
-  diagnostics/           # 指标与通用质量检查
-  compatibility/         # v1/legacy 兼容入口
-  ui/                    # 新 UI facade；app.py 暂保留为 Tk 兼容入口
-  adapters/              # 可选产品增强，不参与核心正确性
-  tests/                 # unit / contract / integration / e2e
-```
-
-`models/`、`recorder/`、`drivers/` 等旧路径暂时保留为兼容实现；新入口已从 `surfaces/`、`core/` 和 `storage/` 访问，后续可按模块逐步内收而不破坏现有技能。
-
-暖树：`ensure_ui_ready`（kit≥1.2.6）；整页 dump：kit≥1.2.11。

tools/ui_inspector/__init__.py

@@ -1,3 +0,0 @@
-"""UI Inspector — development-only multi-surface capture (windows UIA + browser Playwright)."""
-
-__version__ = "0.16.0"

tools/ui_inspector/__main__.py

@@ -1,12 +0,0 @@
-"""python -m tools.ui_inspector  (from skill-template root, if tools is a package)
-
-Prefer:  python tools/ui_inspector/run.py
-"""
-
-from __future__ import annotations
-
-import runpy
-from pathlib import Path
-
-if __name__ == "__main__":
-    runpy.run_path(str(Path(__file__).resolve().parent / "app.py"), run_name="__main__")

tools/ui_inspector/adapters/__init__.py

@@ -1,7 +0,0 @@
-"""App adapters registry (WeCom-specific rules live here, not in generic layers)."""
-
-from .base import AppAdapter
-from .registry import get_adapter, register_adapter
-from . import wecom as _wecom  # noqa: F401 — register on import
-
-__all__ = ["AppAdapter", "get_adapter", "register_adapter"]

tools/ui_inspector/adapters/base.py

@@ -1,17 +0,0 @@
-"""Adapter base protocol."""
-
-from __future__ import annotations
-
-from typing import Any, Optional, Protocol
-
-
-class AppAdapter(Protocol):
-    name: str
-
-    def matches(self, *, class_name: str = "", title: str = "", process_name: str = "") -> bool: ...
-
-    def wake_profile(self, *, class_name: str = "", title: str = "") -> Optional[str]: ...
-
-    def cache_key(self, *, profile: str, hwnd: int = 0) -> str: ...
-
-    def propose_postconditions(self, changes: list[dict[str, Any]]) -> list[dict[str, Any]]: ...

tools/ui_inspector/adapters/registry.py

@@ -1,31 +0,0 @@
-"""Adapter registry."""
-
-from __future__ import annotations
-
-from typing import Any, Optional
-
-_REGISTRY: list[Any] = []
-
-
-def register_adapter(adapter: Any) -> Any:
-    _REGISTRY.append(adapter)
-    return adapter
-
-
-def get_adapter(
-    *,
-    class_name: str = "",
-    title: str = "",
-    process_name: str = "",
-) -> Optional[Any]:
-    for ad in _REGISTRY:
-        try:
-            if ad.matches(class_name=class_name, title=title, process_name=process_name):
-                return ad
-        except Exception:
-            continue
-    return None
-
-
-def all_adapters() -> list[Any]:
-    return list(_REGISTRY)

tools/ui_inspector/adapters/wecom.py

@@ -1,103 +0,0 @@
-"""WeCom (企业微信) adapter — profile / window / postcondition helpers."""
-
-from __future__ import annotations
-
-from typing import Any, Optional
-
-from .registry import register_adapter
-
-PROFILE_WECOM = "wecom"
-PROFILE_WECOM_SEARCH = "wecom_search"
-
-_THRESHOLDS = {
-    PROFILE_WECOM: 80,
-    PROFILE_WECOM_SEARCH: 20,
-}
-
-
-class WeComAdapter:
-    name = "wecom"
-
-    def matches(
-        self, *, class_name: str = "", title: str = "", process_name: str = ""
-    ) -> bool:
-        return self.wake_profile(class_name=class_name, title=title) is not None or (
-            "WXWork" in (process_name or "")
-        )
-
-    def wake_profile(self, *, class_name: str = "", title: str = "") -> Optional[str]:
-        cls = (class_name or "").strip()
-        title = title or ""
-        if cls == "SearchResultWindow2":
-            return PROFILE_WECOM_SEARCH
-        if cls == "WeWorkWindow":
-            return PROFILE_WECOM
-        if "WeWork" in cls and (
-            "企业微信" in title or "WeCom" in title or "wecom" in title.lower()
-        ):
-            return PROFILE_WECOM
-        return None
-
-    def cache_key(self, *, profile: str, hwnd: int = 0) -> str:
-        """Separate main window vs SearchResultWindow2 warm caches."""
-        return f"{profile}:hwnd:{int(hwnd or 0)}"
-
-    def warm_threshold(self, profile: str) -> int:
-        return int(_THRESHOLDS.get(profile, 80))
-
-    def is_search_popup(self, *, class_name: str = "") -> bool:
-        return (class_name or "") == "SearchResultWindow2"
-
-    def is_main_window(self, *, class_name: str = "") -> bool:
-        return (class_name or "") == "WeWorkWindow"
-
-    def known_ids(self) -> dict[str, str]:
-        return {
-            "searchedit": "search_box",
-            "groupbuddyname": "contact_row_name",
-            "search_name": "search_result_name",
-            "send_rich_edit": "compose_box",
-            "panel_title_label": "panel_title",
-        }
-
-    def propose_postconditions(self, changes: list[dict[str, Any]]) -> list[dict[str, Any]]:
-        props: list[dict[str, Any]] = []
-        for ch in changes or []:
-            kind = ch.get("kind")
-            detail = ch.get("detail") or {}
-            if kind == "popup_disappeared" and "SearchResult" in str(
-                detail.get("class_name") or ""
-            ):
-                props.append(
-                    {
-                        "description": "SearchResultWindow2 disappeared",
-                        "kind": "window_absent",
-                        "status": "proposed",
-                        "detail": detail,
-                    }
-                )
-            if kind == "element_appeared":
-                name = str(detail.get("name") or detail.get("automation_id") or "")
-                if name in {"send_rich_edit", "main_chattext", "panel_title_label"}:
-                    props.append(
-                        {
-                            "description": f"WeCom element {name} appeared",
-                            "kind": "element_present",
-                            "status": "proposed",
-                            "detail": detail,
-                        }
-                    )
-            if kind == "title_changed":
-                props.append(
-                    {
-                        "description": f"Chat title became {detail.get('to')!r}",
-                        "kind": "title_equals",
-                        "status": "proposed",
-                        "detail": detail,
-                    }
-                )
-        return props
-
-
-wecom_adapter = WeComAdapter()
-register_adapter(wecom_adapter)

tools/ui_inspector/artifacts.py

@@ -1,735 +0,0 @@
-"""Flow-oriented capture artifacts: one flow document, many steps."""
-
-from __future__ import annotations
-
-import json
-import re
-from datetime import datetime
-from pathlib import Path
-from typing import Any, Optional
-
-try:
-    from . import __version__
-except ImportError:
-    try:
-        import importlib.util
-        from pathlib import Path
-
-        _spec = importlib.util.spec_from_file_location(
-            "ui_inspector_pkg", Path(__file__).resolve().parent / "__init__.py"
-        )
-        _mod = importlib.util.module_from_spec(_spec)  # type: ignore[arg-type]
-        assert _spec and _spec.loader
-        _spec.loader.exec_module(_mod)
-        __version__ = getattr(_mod, "__version__", "0.16.0")
-    except Exception:
-        __version__ = "0.16.0"
-
-
-def _local_ts() -> str:
-    """Local wall-clock stamp for capture folder names: YYYYMMDDHHMMSS."""
-    return datetime.now().astimezone().strftime("%Y%m%d%H%M%S")
-
-
-def safe_slug(text: str, *, fallback: str = "session") -> str:
-    text = (text or "").strip()
-    if not text:
-        return fallback
-    text = re.sub(r"\s+", "-", text)
-    text = re.sub(r"[^\w\u4e00-\u9fff\-]+", "", text, flags=re.UNICODE)
-    text = text.strip("-_") or fallback
-    return text[:80]
-
-
-def _clean_ui_text(text: str) -> str:
-    """Strip WeCom-style rich markup and collapse whitespace."""
-    s = (text or "").strip()
-    if not s:
-        return ""
-    s = re.sub(r"<[^>]+>", "", s)
-    s = re.sub(r"\s+", " ", s).strip()
-    return s
-
-
-# Name/AutomationId keys that usually mark "what page am I on"
-_PAGE_HINT_KEYS: dict[str, int] = {
-    "panel_title_label": 100,
-    "title": 95,
-    "ww_btn_text": 70,
-    "main_chattext": 45,
-}
-
-_LAYOUT_NOISE = {
-    "bg",
-    "navigation_bar",
-    "outer_layout",
-    "content_container",
-    "title_layout",
-    "closebtn",
-    "VerticalLayoutUI",
-    "HorizontalLayoutUI",
-    "wbtn_container",
-    "search_layout",
-    "search_layout_border",
-    "inner_search_layout",
-    "ControlUI",
-    "LabelUI",
-    "item",
-    "externals",
-    "status_layout",
-    "search_button_region",
-    "search_icon",
-    "ww_btn_icon",
-    "wbadge_label",
-    "lasttime",
-    "groupbuddydescription",
-    "bottom_text",
-    "chattext",
-    "pane",
-    "Pane",
-}
-
-_CHROME_TEXT = {
-    "关闭",
-    "最小化",
-    "最大化",
-    "还原",
-    "取消",
-    "确定",
-    "关闭窗口",
-}
-
-# Left-rail / chrome values that appear on every WeCom page — not page identity
-_GLOBAL_NAV_TEXT = {
-    "消息",
-    "邮件",
-    "文档",
-    "日程",
-    "待办",
-    "会议",
-    "智能文档",
-    "通讯录",
-    "微盘",
-    "高级功能",
-    "工作台",
-    "收藏",
-    "智能总结",
-    "回到最新位置",
-}
-
-
-def _row_key(r: dict[str, Any]) -> str:
-    return _clean_ui_text(str(r.get("name") or r.get("automation_id") or ""))
-
-
-def _looks_like_search_dialog(
-    window_title: str, window_class: str, rows: list[dict[str, Any]]
-) -> bool:
-    t = window_title or ""
-    c = window_class or ""
-    if "Search" in c or "搜索" in t or "添加客户" in t or "添加联系人" in t:
-        return True
-    for r in rows:
-        if _row_key(r) == "title":
-            v = _clean_ui_text(str(r.get("value") or ""))
-            if "添加" in v or "搜索" in v:
-                return True
-    return False
-
-
-def _short_entity(text: str, *, limit: int = 16) -> str:
-    """Clean contact/chat title: strip markup leftovers and @微信 suffix."""
-    s = _clean_ui_text(text)
-    if not s:
-        return ""
-    s = re.sub(r"@微信\s*$", "", s)
-    s = re.sub(r"@WeCom\s*$", "", s, flags=re.I)
-    s = s.strip(" -_|")
-    if len(s) > limit:
-        s = s[:limit].rstrip("-_")
-    return s
-
-
-def _raw_value(r: dict[str, Any]) -> str:
-    return str(r.get("value") or "")
-
-
-def _is_selected_buddy(raw_value: str) -> bool:
-    # WeCom marks the active conversation with semi-transparent white color markup.
-    return "#99ffffff" in (raw_value or "").lower()
-
-
-def derive_step_label(
-    *,
-    window_title: str = "",
-    window_class: str = "",
-    locatable: Optional[list[dict[str, Any]]] = None,
-    max_hints: int = 2,
-) -> str:
-    """
-    Build a human-readable step label from window + page-specific UIA hints.
-
-    Prefers: search box text, page title, selected chat, compose draft, action buttons.
-    Avoids bare「会话列表」when a concrete chat/search/input signal exists.
-    """
-    rows = list(locatable or [])
-    base = _clean_ui_text(window_title)
-    if not base:
-        for r in rows:
-            if (r.get("control_type") or "") == "WindowControl":
-                base = _clean_ui_text(str(r.get("name") or ""))
-                if base:
-                    break
-    if not base and window_class:
-        base = window_class
-
-    has_panel_title = any(
-        _row_key(r) == "panel_title_label" and _clean_ui_text(str(r.get("value") or ""))
-        for r in rows
-    )
-    has_groupbuddy = any(_row_key(r) == "groupbuddyname" for r in rows)
-    has_search_result = any(_row_key(r) == "search_result" for r in rows)
-    has_searchedit = any(_row_key(r) in {"searchedit", "search_edit"} for r in rows)
-    search_dialog = _looks_like_search_dialog(window_title, window_class, rows)
-
-    scored: list[tuple[int, str]] = []
-    searchedit_value = ""
-    selected_buddy = ""
-    first_buddy = ""
-
-    for r in rows:
-        key = _row_key(r)
-        if not key or key in _LAYOUT_NOISE:
-            continue
-        raw = _raw_value(r)
-        value = _clean_ui_text(raw)
-        name_disp = _clean_ui_text(str(r.get("name") or ""))
-        ctype = str(r.get("control_type") or "")
-
-        if key in {"searchedit", "search_edit"} and value:
-            searchedit_value = _short_entity(value, limit=20)
-            # Main-window search AND add-friend dialog both matter.
-            scored.append((98, f"搜索{searchedit_value}"))
-            continue
-
-        if key == "send_rich_edit" and value:
-            draft = _short_entity(value, limit=12)
-            if draft:
-                scored.append((86, f"输入{draft}"))
-            continue
-
-        if key == "groupbuddyname" and value:
-            entity = _short_entity(value, limit=14)
-            if not entity:
-                continue
-            if _is_selected_buddy(raw) and not selected_buddy:
-                selected_buddy = entity
-                scored.append((92, entity))
-            elif not first_buddy and not r.get("is_offscreen"):
-                first_buddy = entity
-            continue
-
-        pri = _PAGE_HINT_KEYS.get(key)
-        if pri is not None and value:
-            if value in _CHROME_TEXT or value in _GLOBAL_NAV_TEXT:
-                continue
-            if key == "title" and value == base:
-                continue
-            if key == "main_chattext" and (has_panel_title or has_groupbuddy):
-                continue
-            entity = _short_entity(value, limit=16)
-            if not entity:
-                continue
-            scored.append((pri, entity))
-            continue
-
-        if ctype in {
-            "ButtonControl",
-            "EditControl",
-            "TabItemControl",
-            "TextControl",
-            "HyperlinkControl",
-            "MenuItemControl",
-            "CheckBoxControl",
-            "RadioButtonControl",
-            "ComboBoxControl",
-            "ListItemControl",
-        }:
-            label = _short_entity(value or name_disp, limit=16)
-            if (
-                not label
-                or label in _CHROME_TEXT
-                or label in _LAYOUT_NOISE
-                or label in _GLOBAL_NAV_TEXT
-            ):
-                continue
-            scored.append((40, label))
-
-    # Conversation context when no selected buddy / search text yet
-    if (
-        has_groupbuddy
-        and not selected_buddy
-        and first_buddy
-        and not has_panel_title
-        and not searchedit_value
-    ):
-        scored.append((78, first_buddy))
-
-    strong = any(p >= 90 for p, _ in scored)
-    if search_dialog and has_searchedit and not searchedit_value and not strong:
-        scored.append((75, "搜索框"))
-    elif search_dialog and has_search_result and searchedit_value and not strong:
-        scored.append((50, "搜索结果"))
-    elif (
-        has_groupbuddy
-        and not has_panel_title
-        and not strong
-        and not first_buddy
-        and not selected_buddy
-    ):
-        scored.append((60, "会话列表"))
-
-    scored.sort(key=lambda x: (-x[0], x[1]))
-    hints: list[str] = []
-    seen: set[str] = set()
-    for _pri, text in scored:
-        if not text or text == base or text in seen:
-            continue
-        if text in _CHROME_TEXT or text in _GLOBAL_NAV_TEXT:
-            continue
-        compact = (
-            text[2:] if text.startswith(("搜索", "输入")) and len(text) > 2 else text
-        )
-        if compact and any(
-            compact == s or s.endswith(compact) or compact in s for s in seen
-        ):
-            continue
-        seen.add(text)
-        if compact:
-            seen.add(compact)
-        hints.append(text)
-        if len(hints) >= max_hints:
-            break
-
-    parts = [p for p in [base, *hints] if p]
-    if not parts:
-        parts = ["step"]
-    return "_".join(parts)
-
-
-def _label_slug_key(label: str) -> str:
-    return safe_slug(label, fallback="step")
-
-
-def _unique_step_dirname(steps_root: Path, idx: int, label: str) -> str:
-    """NN_label, with -2/-3 suffix if the folder already exists."""
-    slug = _label_slug_key(label)
-    base = f"{idx:02d}_{slug}"
-    if not (steps_root / base).exists():
-        return base
-    n = 2
-    while (steps_root / f"{base}-{n}").exists():
-        n += 1
-    return f"{base}-{n}"
-
-
-def _existing_label_slugs(flow: dict[str, Any]) -> set[str]:
-    out: set[str] = set()
-    for s in flow.get("steps") or []:
-        lab = s.get("label") or ""
-        name = str(s.get("name") or "")
-        if lab:
-            out.add(_label_slug_key(str(lab)))
-        m = re.match(r"^\d{2}_(.+?)(?:-\d+)?$", name)
-        if m:
-            out.add(m.group(1))
-    return out
-
-
-def resolve_step_label(
-    flow: dict[str, Any],
-    *,
-    window_title: str = "",
-    window_class: str = "",
-    locatable: Optional[list[dict[str, Any]]] = None,
-) -> str:
-    """Derive label; if it collides with earlier steps, add more hints."""
-    used = _existing_label_slugs(flow)
-    label = derive_step_label(
-        window_title=window_title,
-        window_class=window_class,
-        locatable=locatable,
-        max_hints=2,
-    )
-    if _label_slug_key(label) not in used:
-        return label
-    label3 = derive_step_label(
-        window_title=window_title,
-        window_class=window_class,
-        locatable=locatable,
-        max_hints=3,
-    )
-    if _label_slug_key(label3) not in used:
-        return label3
-    # Last resort: keep richest label; folder uniquifier adds -2
-    return label3
-
-
-def default_captures_root(tool_dir: Optional[Path] = None) -> Path:
-    base = tool_dir or Path(__file__).resolve().parent
-    return base / "_captures"
-
-
-def write_json(path: Path, data: Any) -> None:
-    path.parent.mkdir(parents=True, exist_ok=True)
-    path.write_text(
-        json.dumps(data, ensure_ascii=False, indent=2) + "\n",
-        encoding="utf-8",
-    )
-
-
-def write_jsonl(path: Path, rows: list[dict[str, Any]]) -> None:
-    path.parent.mkdir(parents=True, exist_ok=True)
-    with path.open("w", encoding="utf-8") as fh:
-        for row in rows:
-            fh.write(json.dumps(row, ensure_ascii=False) + "\n")
-
-
-def read_json(path: Path) -> dict[str, Any]:
-    return json.loads(path.read_text(encoding="utf-8"))
-
-
-def build_meta(
-    *,
-    window: dict[str, Any],
-    scene: str,
-    skill_root: Optional[str] = None,
-    extra: Optional[dict[str, Any]] = None,
-    surface: str = "windows",
-    driver: str = "uiautomation",
-    url: str = "",
-) -> dict[str, Any]:
-    now = datetime.now().astimezone()
-    meta = {
-        "schema": "openclaw.ui_inspector.capture.v1",
-        "surface": surface or "windows",
-        "captured_at": now.isoformat(),
-        "scene": scene,
-        "skill_root": skill_root,
-        "app": {
-            "title": window.get("title"),
-            "class_name": window.get("class_name"),
-            "process_name": window.get("process_name"),
-            "pid": window.get("pid"),
-            "hwnd": window.get("hwnd"),
-            "url": url or window.get("url") or "",
-        },
-        "environment": {
-            "os": "windows",
-            "tz": now.tzname(),
-        },
-        "tool": {
-            "name": "openclaw-ui-inspector",
-            "version": __version__,
-            "driver": driver or "uiautomation",
-        },
-    }
-    if extra:
-        meta["extra"] = extra
-    return meta
-
-
-def noteworthy_from_index(index: list[dict[str, Any]], *, limit: int = 80) -> list[dict[str, Any]]:
-    rows = [
-        r
-        for r in index
-        if (r.get("name") or r.get("value") or r.get("automation_id"))
-    ]
-    rows.sort(
-        key=lambda r: (
-            0 if r.get("value") else 1,
-            0 if r.get("name") else 1,
-            int(r.get("depth") or 0) if isinstance(r.get("depth"), int) and r.get("depth") >= 0 else 99,
-        )
-    )
-    return rows[:limit]
-
-
-def capture_window_png(hwnd: int, dest: Path) -> bool:
-    try:
-        from PIL import ImageGrab
-    except ImportError:
-        return False
-    try:
-        import ctypes
-        from ctypes import wintypes
-
-        user32 = ctypes.windll.user32
-        rect = wintypes.RECT()
-        if not user32.GetWindowRect(int(hwnd), ctypes.byref(rect)):
-            return False
-        bbox = (int(rect.left), int(rect.top), int(rect.right), int(rect.bottom))
-        if bbox[2] <= bbox[0] or bbox[3] <= bbox[1]:
-            return False
-        img = ImageGrab.grab(bbox=bbox, all_screens=True)
-        dest.parent.mkdir(parents=True, exist_ok=True)
-        img.save(str(dest))
-        return True
-    except Exception:
-        try:
-            img = ImageGrab.grab(all_screens=True)
-            dest.parent.mkdir(parents=True, exist_ok=True)
-            img.save(str(dest))
-            return True
-        except Exception:
-            return False
-
-
-# --- Flow document API -------------------------------------------------
-
-def create_flow(
-    captures_root: Path,
-    *,
-    flow_name: str = "desktop",
-    recording_mode: str = "auto",
-) -> Path:
-    """开始流程：新建一份流程文档目录（v1 steps + 可选 v2 资产目录）。"""
-    captures_root.mkdir(parents=True, exist_ok=True)
-    slug = safe_slug(flow_name, fallback="flow")
-    flow_dir = captures_root / f"{_local_ts()}_{slug}"
-    flow_dir.mkdir(parents=True, exist_ok=False)
-    (flow_dir / "steps").mkdir(parents=True, exist_ok=True)
-    # v2 dual-write directories (empty until recorder writes)
-    for sub in ("states", "observations", "actions", "elements", "journal", "diagnostics"):
-        (flow_dir / sub).mkdir(parents=True, exist_ok=True)
-    now = datetime.now().astimezone().isoformat()
-    flow = {
-        "schema": "openclaw.ui_inspector.flow.v1",
-        "flow_name": flow_name,
-        "status": "open",
-        "created_at": now,
-        "updated_at": now,
-        "ended_at": None,
-        "step_count": 0,
-        "steps": [],
-        "tool": {
-            "name": "openclaw-ui-inspector",
-            "version": __version__,
-        },
-        # Optional v2 pointers — old readers ignore unknown fields.
-        "v2": {
-            "recording_mode": recording_mode or "auto",
-            "action_flow": "action_flow.json",
-            "element_catalog": "element_catalog.json",
-            "states_dir": "states/",
-            "observations_dir": "observations/",
-            "actions_dir": "actions/",
-            "elements_dir": "elements/",
-            "raw_event_journal": "journal/raw_events.jsonl",
-            "pages": "pages.json",
-            "diagnostics_dir": "diagnostics/",
-            "validation": "diagnostics/validation.json",
-            "replay_report": "diagnostics/replay_report.json",
-        },
-    }
-    write_json(flow_dir / "flow.json", flow)
-    write_flow_summary(flow_dir, flow)
-    return flow_dir
-
-
-def load_flow(flow_dir: Path) -> dict[str, Any]:
-    return read_json(flow_dir / "flow.json")
-
-
-def save_flow(flow_dir: Path, flow: dict[str, Any]) -> None:
-    flow["updated_at"] = datetime.now().astimezone().isoformat()
-    write_json(flow_dir / "flow.json", flow)
-    write_flow_summary(flow_dir, flow)
-
-
-def next_step_index(flow: dict[str, Any]) -> int:
-    return int(flow.get("step_count") or 0) + 1
-
-
-def add_step_dir(
-    flow_dir: Path,
-    flow: dict[str, Any],
-    *,
-    window_title: str = "",
-    window_class: str = "",
-    locatable: Optional[list[dict[str, Any]]] = None,
-    label: Optional[str] = None,
-) -> tuple[Path, int, str]:
-    """Allocate steps/NN_<page-fingerprint>/ under an open flow."""
-    if flow.get("status") != "open":
-        raise RuntimeError("当前流程已结束，请先「开始流程」。")
-    idx = next_step_index(flow)
-    if not label:
-        label = resolve_step_label(
-            flow,
-            window_title=window_title,
-            window_class=window_class,
-            locatable=locatable,
-        )
-    steps_root = flow_dir / "steps"
-    steps_root.mkdir(parents=True, exist_ok=True)
-    step_name = _unique_step_dirname(steps_root, idx, label)
-    step_dir = steps_root / step_name
-    step_dir.mkdir(parents=True, exist_ok=False)
-    return step_dir, idx, step_name
-
-
-def register_step(
-    flow_dir: Path,
-    flow: dict[str, Any],
-    *,
-    step_index: int,
-    step_name: str,
-    window: dict[str, Any],
-    stats: dict[str, Any],
-    label: Optional[str] = None,
-    surface: str = "windows",
-    url: str = "",
-) -> dict[str, Any]:
-    entry = {
-        "index": step_index,
-        "name": step_name,
-        "label": label or step_name,
-        "captured_at": datetime.now().astimezone().isoformat(),
-        "surface": surface or "windows",
-        "url": url or window.get("url") or "",
-        "window_title": window.get("title"),
-        "window_class": window.get("class_name"),
-        "pid": window.get("pid"),
-        "hwnd": window.get("hwnd"),
-        "findall": stats.get("findall_count"),
-        "locatable": stats.get("locatable_count"),
-        "kept": stats.get("nodes_kept"),
-        "uia_sparse": stats.get("uia_sparse"),
-        "path": f"steps/{step_name}",
-    }
-    steps = list(flow.get("steps") or [])
-    steps.append(entry)
-    flow["steps"] = steps
-    flow["step_count"] = len(steps)
-    save_flow(flow_dir, flow)
-    return entry
-
-
-def end_flow(flow_dir: Path, flow: dict[str, Any]) -> dict[str, Any]:
-    if flow.get("status") == "ended":
-        return flow
-    flow["status"] = "ended"
-    flow["ended_at"] = datetime.now().astimezone().isoformat()
-    save_flow(flow_dir, flow)
-    return flow
-
-
-def write_flow_summary(flow_dir: Path, flow: dict[str, Any]) -> None:
-    lines = [
-        f"# 流程：{flow.get('flow_name')}",
-        "",
-        f"- status: `{flow.get('status')}`",
-        f"- created_at: `{flow.get('created_at')}`",
-        f"- updated_at: `{flow.get('updated_at')}`",
-        f"- ended_at: `{flow.get('ended_at')}`",
-        f"- step_count: `{flow.get('step_count')}`",
-        "",
-        "## 步骤",
-        "",
-    ]
-    steps = flow.get("steps") or []
-    if not steps:
-        lines.append("（尚无拾取步骤。点「拾取当前页」追加。）")
-        lines.append("")
-    else:
-        for s in steps:
-            lines.append(f"### {s.get('index')}. `{s.get('name')}`")
-            lines.append("")
-            if s.get("label"):
-                lines.append(f"- label: `{s.get('label')}`")
-            lines.append(f"- surface: `{s.get('surface') or 'windows'}`")
-            if s.get("url"):
-                lines.append(f"- url: `{s.get('url')}`")
-            lines.append(
-                f"- window: `{s.get('window_title')}` / `{s.get('window_class')}`"
-            )
-            lines.append(
-                f"- stats: findall=`{s.get('findall')}` locatable=`{s.get('locatable')}` "
-                f"kept=`{s.get('kept')}` sparse=`{s.get('uia_sparse')}`"
-            )
-            lines.append(f"- path: `{s.get('path')}/`")
-            lines.append("")
-    lines.extend(
-        [
-            "## 说明",
-            "",
-            "- 一个流程对应本目录一份文档（`flow.json` + `SUMMARY.md` + `steps/`）。",
-            "- 每拾取一页，在 `steps/` 下新增一步；给 AI 时优先读本 `SUMMARY.md` 与各步 `locatable.json`。",
-            "- `surface=windows` 为桌面 UIA；`surface=browser` 为 Playwright 整页 dump。",
-            "- Name ≠ 界面中文时请看 Value；浏览器步优先看 `primary_locator` / `playwright_candidates`。",
-            "- v2（若存在）：`action_flow.json` + `element_catalog.json` + `states/` + `actions/` + `elements/`。",
-            "- 给 AI 编程工具时优先提供 `action_flow.json` 与 `element_catalog.json`，诊断再用 steps/ 原始树。",
-            "",
-        ]
-    )
-    (flow_dir / "SUMMARY.md").write_text("\n".join(lines), encoding="utf-8")
-
-
-def write_step_summary(
-    step_dir: Path,
-    *,
-    meta: dict[str, Any],
-    stats: Optional[dict[str, Any]] = None,
-    noteworthy: Optional[list[dict[str, Any]]] = None,
-) -> None:
-    app = meta.get("app") or {}
-    tool = meta.get("tool") or {}
-    lines = [
-        "# Step Summary",
-        "",
-        f"- captured_at: `{meta.get('captured_at')}`",
-        f"- surface: `{meta.get('surface')}` driver: `{tool.get('driver')}`",
-        f"- window: `{app.get('title')}` / `{app.get('class_name')}` pid=`{app.get('pid')}`",
-        "",
-    ]
-    if app.get("url"):
-        lines.append(f"- url: `{app.get('url')}`")
-        lines.append("")
-    if stats:
-        lines.extend(
-            [
-                "## Stats",
-                "",
-                f"- kept: `{stats.get('nodes_kept')}`",
-                f"- findall: `{stats.get('findall_count')}`",
-                f"- locatable: `{stats.get('locatable_count')}`",
-                f"- uia_sparse: `{stats.get('uia_sparse')}`",
-                "",
-            ]
-        )
-    if noteworthy:
-        lines.extend(["## Notable", ""])
-        for item in noteworthy[:40]:
-            primary = (item.get("primary_locator") or {}).get("expr") or ""
-            extra = f" locator=`{primary}`" if primary else ""
-            lines.append(
-                f"- `{item.get('control_type')}` Name=`{item.get('name')}` "
-                f"Value=`{item.get('value') or item.get('text')}`{extra}"
-            )
-        lines.append("")
-    lines.extend(
-        [
-            "## Files",
-            "",
-            "- `meta.json` / `tree.json` / `index.jsonl` / `locatable.json` / `screen.png`",
-            "",
-        ]
-    )
-    (step_dir / "SUMMARY.md").write_text("\n".join(lines), encoding="utf-8")
-
-
-
-# Back-compat aliases used by older smoke scripts
-def create_session_dir(captures_root: Path, *, scene: str = "desktop") -> Path:
-    return create_flow(captures_root, flow_name=scene)

tools/ui_inspector/compatibility/__init__.py

@@ -1,2 +0,0 @@
-"""Backward-compatible readers and entry points."""
-

tools/ui_inspector/compatibility/v1_reader.py

@@ -1,6 +0,0 @@
-"""Legacy flow.json reader."""
-
-from artifacts import load_flow
-
-__all__ = ["load_flow"]
-

tools/ui_inspector/compatibility/v1_writer.py

@@ -1,6 +0,0 @@
-"""Legacy steps/ writer exports."""
-
-from artifacts import add_step_dir, create_flow, save_flow
-
-__all__ = ["add_step_dir", "create_flow", "save_flow"]
-

tools/ui_inspector/core/__init__.py

@@ -1,2 +0,0 @@
-"""Surface-independent UI Inspector domain and recording pipeline."""
-

tools/ui_inspector/core/models/__init__.py

@@ -1,16 +0,0 @@
-"""Canonical model exports.
-
-The compatibility ``models`` package remains importable while callers migrate
-to this namespace.
-"""
-
-from models.action import ActionFlow, ActionRecord
-from models.element import ElementCatalog, ElementRef
-from models.locator import LocatorCandidate
-from models.observation import UIObservation
-from models.state import UIState
-
-__all__ = [
-    "ActionFlow", "ActionRecord", "ElementCatalog", "ElementRef",
-    "LocatorCandidate", "UIObservation", "UIState",
-]

tools/ui_inspector/core/pipeline/__init__.py

@@ -1,6 +0,0 @@
-"""Shared event, action, and evidence pipeline."""
-
-from .event_journal import EventJournal
-
-__all__ = ["EventJournal"]
-

tools/ui_inspector/core/pipeline/event_journal.py

@@ -1,49 +0,0 @@
-"""Append-only raw interaction journal.
-
-Raw events are facts.  They are persisted before semantic reduction so a
-future reducer can rebuild Action Flow without asking the user to repeat the
-workflow.
-"""
-
-from __future__ import annotations
-
-import json
-import threading
-from dataclasses import asdict, is_dataclass
-from datetime import datetime
-from pathlib import Path
-from typing import Any
-
-from redaction import sanitize_artifact
-
-
-class EventJournal:
-    def __init__(self, flow_dir: Path) -> None:
-        self.path = Path(flow_dir) / "journal" / "raw_events.jsonl"
-        self.path.parent.mkdir(parents=True, exist_ok=True)
-        self._lock = threading.Lock()
-        self._sequence = 0
-
-    def append(self, event: Any, *, channel: str = "surface") -> int:
-        if is_dataclass(event) and not isinstance(event, type):
-            payload = asdict(event)
-        elif isinstance(event, dict):
-            payload = dict(event)
-        else:
-            payload = dict(getattr(event, "__dict__", {}) or {})
-        with self._lock:
-            self._sequence += 1
-            record = {
-                "journal_sequence": self._sequence,
-                "received_at": datetime.now().astimezone().isoformat(),
-                "channel": channel,
-                "event": sanitize_artifact(payload),
-            }
-            line = json.dumps(
-                record, ensure_ascii=False, sort_keys=True, separators=(",", ":")
-            )
-            with self.path.open("a", encoding="utf-8", newline="\n") as stream:
-                stream.write(line + "\n")
-                stream.flush()
-            return self._sequence
-

tools/ui_inspector/core/policies/__init__.py

@@ -1,7 +0,0 @@
-"""Surface-independent decision policies."""
-
-from .element_identity import element_identity_key
-from .target_scoring import resolve_browser_target
-
-__all__ = ["element_identity_key", "resolve_browser_target"]
-

tools/ui_inspector/core/policies/element_identity.py

@@ -1,62 +0,0 @@
-"""Stable element identity independent of transient value/focus classes."""
-
-from __future__ import annotations
-
-import hashlib
-import json
-import re
-from typing import Any
-from urllib.parse import urlsplit
-
-
-def _origin(url: str) -> str:
-    try:
-        parts = urlsplit(url or "")
-        return f"{parts.scheme}://{parts.netloc}" if parts.netloc else ""
-    except Exception:
-        return ""
-
-
-def _structural_css(css: str) -> str:
-    # Classes frequently encode focus/hover/build hashes.  Keep tag depth and
-    # stable IDs; that distinguishes nested anonymous nodes without freezing a
-    # generated class list into identity.
-    parts = []
-    for raw in (css or "").split(">"):
-        part = raw.strip()
-        tag_match = re.match(r"^[A-Za-z][\w-]*", part)
-        id_match = re.search(r"#[A-Za-z0-9_-]+", part)
-        tag = tag_match.group(0).lower() if tag_match else "*"
-        parts.append(tag + (id_match.group(0) if id_match else ""))
-    return ">".join(parts)
-
-
-def element_identity_key(
-    element: dict[str, Any], *, surface: str, page_url: str = "", frame_name: str = ""
-) -> str:
-    el = dict(element or {})
-    role = str(el.get("control_type") or el.get("role") or el.get("tag") or "")
-    anchor = ""
-    for key in (
-        "test_id", "automation_id", "id", "name_attr", "href",
-        "placeholder", "aria_label", "label",
-    ):
-        value = str(el.get(key) or "").strip()
-        if value:
-            anchor = f"{key}:{value}"
-            break
-    if not anchor and not el.get("contenteditable"):
-        name = str(el.get("semantic_name") or el.get("name") or "").strip()
-        if name and len(name) <= 96:
-            anchor = f"name:{name}"
-    if not anchor:
-        anchor = f"structure:{_structural_css(str(el.get('css') or ''))}"
-    payload = {
-        "surface": surface or "",
-        "origin": _origin(page_url) if surface == "browser" else "",
-        "frame": frame_name or "",
-        "role": role.lower(),
-        "anchor": anchor,
-    }
-    raw = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
-    return hashlib.sha256(raw.encode("utf-8")).hexdigest()

tools/ui_inspector/core/policies/target_scoring.py

@@ -1,145 +0,0 @@
-"""Distance- and geometry-aware target selection.
-
-The resolver never relies on a product selector.  It ranks the composed path
-using native semantics, explicit accessibility metadata, distance from the hit
-node, and geometry.  Huge ancestors are evidence containers, not controls.
-"""
-
-from __future__ import annotations
-
-import re
-from typing import Any, Iterable
-
-_NATIVE_TAGS = {"button", "a", "input", "textarea", "select", "summary", "label"}
-_INTERACTIVE_ROLES = {
-    "button", "link", "textbox", "combobox", "checkbox", "radio",
-    "menuitem", "option", "tab", "switch", "slider", "spinbutton",
-}
-_ACTION_CLASS_TOKENS = {
-    "button", "btn", "submit", "send", "publish", "post", "play",
-    "pause", "action", "click", "toggle", "menu", "option", "tab",
-}
-_GRAPHIC_LEAF_TAGS = {"path", "use", "g", "svg"}
-
-
-def _tokens(value: str) -> set[str]:
-    value = re.sub(r"([a-z0-9])([A-Z])", r"\1 \2", value or "")
-    return {x.lower() for x in re.split(r"[^A-Za-z0-9]+", value) if x}
-
-
-def _area(rect: dict[str, Any]) -> float:
-    try:
-        return max(0.0, float(rect.get("width") or 0)) * max(
-            0.0, float(rect.get("height") or 0)
-        )
-    except (TypeError, ValueError):
-        return 0.0
-
-
-def _contains(
-    rect: dict[str, Any], x: int | None, y: int | None, *, slop: float = 12.0
-) -> bool:
-    if x is None or y is None or not rect:
-        return True
-    try:
-        return (
-            float(rect.get("left")) - slop <= x <= float(rect.get("right")) + slop
-            and float(rect.get("top")) - slop <= y <= float(rect.get("bottom")) + slop
-        )
-    except (TypeError, ValueError):
-        return True
-
-
-def resolve_browser_target(
-    hit_target: dict[str, Any],
-    candidates: Iterable[dict[str, Any]],
-    *,
-    x: int | None = None,
-    y: int | None = None,
-) -> tuple[dict[str, Any], list[dict[str, Any]]]:
-    """Return the best actionable node and an auditable scored candidate list."""
-    chain = [dict(item or {}) for item in candidates if isinstance(item, dict)]
-    if not chain:
-        chain = [dict(hit_target or {})]
-    hit = dict(hit_target or chain[0] or {})
-    hit_area = max(1.0, _area(hit.get("rect") or {}))
-    scored: list[dict[str, Any]] = []
-
-    for depth, item in enumerate(chain):
-        tag = str(item.get("tag") or "").lower()
-        role = str(item.get("role") or "").lower()
-        explicit_role = str(item.get("explicit_role") or "").lower()
-        rect = dict(item.get("rect") or {})
-        area = _area(rect)
-        score = 24.0 if depth == 0 else 0.0
-        reasons: list[str] = [f"depth={depth}"]
-
-        if tag in _NATIVE_TAGS:
-            score = max(score, 100.0)
-            reasons.append("native_control")
-        if item.get("contenteditable_attr"):
-            score = max(score, 99.0)
-            reasons.append("editable_root")
-        elif item.get("contenteditable"):
-            score = max(score, 82.0)
-            reasons.append("editable_descendant")
-        if explicit_role in _INTERACTIVE_ROLES:
-            score = max(score, 97.0)
-            reasons.append("explicit_interactive_role")
-        elif role in _INTERACTIVE_ROLES:
-            score = max(score, 88.0)
-            reasons.append("computed_interactive_role")
-        if item.get("test_id") or item.get("aria_label"):
-            score = max(score, 91.0)
-            reasons.append("stable_semantic_attribute")
-        if item.get("tabindex") not in (None, ""):
-            score = max(score, 76.0)
-            reasons.append("keyboard_focusable")
-        if _tokens(str(item.get("class_name") or "")) & _ACTION_CLASS_TOKENS:
-            score = max(score, 82.0)
-            reasons.append("action_class_token")
-        if str(item.get("cursor") or "").lower() == "pointer":
-            score = max(score, 68.0)
-            reasons.append("pointer_cursor")
-
-        # Target selection and locator quality are one decision.  Event-time
-        # counts are captured before a click can navigate or mutate the DOM.
-        counts = [
-            item.get("stable_css_count"),
-            item.get("href_count"),
-            item.get("css_count"),
-        ]
-        known_counts = [int(value) for value in counts if value is not None]
-        if 1 in known_counts:
-            score += 22.0
-            reasons.append("event_time_unique_locator")
-        elif known_counts and min(known_counts) > 1:
-            score -= 18.0
-            reasons.append("event_time_ambiguous_locator")
-
-        if tag in _GRAPHIC_LEAF_TAGS:
-            score -= 28.0
-            reasons.append("graphic_leaf")
-        elif (
-            str(item.get("cursor") or "").lower() == "pointer"
-            and 20.0 <= float(rect.get("width") or 0.0) <= 96.0
-            and 20.0 <= float(rect.get("height") or 0.0) <= 96.0
-        ):
-            score += 14.0
-            reasons.append("compact_clickable_wrapper")
-
-        score -= depth * 3.5
-        if not _contains(rect, x, y):
-            score -= 100.0
-            reasons.append("outside_pointer")
-        if area and area > max(hit_area * 25.0, 200_000.0):
-            score -= 70.0
-            reasons.append("oversized_ancestor")
-        elif area and area > max(hit_area * 8.0, 60_000.0):
-            score -= 25.0
-            reasons.append("large_ancestor")
-
-        scored.append({"depth": depth, "score": round(score, 3), "reasons": reasons, "target": item})
-
-    winner = max(scored, key=lambda row: (row["score"], -row["depth"]))
-    return dict(winner["target"]), scored

tools/ui_inspector/diagnostics/__init__.py

@@ -1,6 +0,0 @@
-"""Capture validation and timeline diagnostics."""
-
-from .capture_validator import validate_capture
-from .replay_preflight import build_replay_preflight, write_replay_preflight
-
-__all__ = ["validate_capture", "build_replay_preflight", "write_replay_preflight"]

tools/ui_inspector/diagnostics/capture_validator.py

@@ -1,134 +0,0 @@
-"""Product-neutral capture quality checks."""
-
-from __future__ import annotations
-
-import json
-from pathlib import Path
-from typing import Any
-
-
-def _read(path: Path) -> dict[str, Any]:
-    return json.loads(path.read_text(encoding="utf-8"))
-
-
-def validate_capture(flow_dir: Path) -> dict[str, Any]:
-    root = Path(flow_dir)
-    issues: list[dict[str, Any]] = []
-    actions = sorted(
-        (_read(path) for path in (root / "actions").glob("*.json")),
-        key=lambda item: int(item.get("sequence") or 0),
-    )
-    expected = list(range(1, len(actions) + 1))
-    actual = [int(item.get("sequence") or 0) for item in actions]
-    if actual != expected:
-        issues.append({"severity": "error", "code": "ACTION_SEQUENCE_GAP", "actual": actual})
-    elements = {
-        item.get("id"): item
-        for item in (_read(path) for path in (root / "elements").glob("*.json"))
-        if item.get("id")
-    }
-    interactive_types = {"click", "fill", "type", "press_key", "select", "toggle"}
-    previous_action: dict[str, Any] | None = None
-    for action in actions:
-        target = dict(action.get("target_snapshot") or {})
-        rect = dict(target.get("rect") or target.get("bounds") or {})
-        area = float(rect.get("width") or 0) * float(rect.get("height") or 0)
-        if action.get("action_type") in {"click", "fill", "select", "toggle"} and not target:
-            issues.append({"severity": "error", "code": "ACTION_TARGET_MISSING", "action_id": action.get("id")})
-        if action.get("action_type") == "click" and area > 1_000_000:
-            issues.append({"severity": "error", "code": "CLICK_TARGET_OVERSIZED", "action_id": action.get("id"), "area": area})
-        target_name = str(target.get("name") or target.get("automation_id") or "").lower()
-        if (
-            action.get("surface") == "windows"
-            and action.get("action_type") == "click"
-            and str(target.get("control_type") or "") in {"TreeControl", "PaneControl", "GroupControl"}
-            and any(token in target_name for token in ("search_result", "listview", "list_view", "watermark", "container", "layout"))
-        ):
-            issues.append({"severity": "warning", "code": "CLICK_TARGET_GENERIC_CONTAINER", "action_id": action.get("id"), "target_name": target_name})
-        if (
-            action.get("surface") == "windows"
-            and action.get("action_type") == "press_key"
-            and (str(target.get("control_type") or "") == "WindowControl" or area > 1_000_000)
-        ):
-            issues.append({"severity": "warning", "code": "KEY_TARGET_ROOT_WINDOW", "action_id": action.get("id")})
-        if action.get("action_type") == "click" and str(target.get("tag") or "").lower() in {"path", "use", "g", "svg"}:
-            issues.append({"severity": "warning", "code": "GRAPHIC_LEAF_TARGET", "action_id": action.get("id"), "tag": target.get("tag")})
-        evidence = dict(action.get("evidence") or {})
-        if not action.get("after_observation_id") and not evidence.get("eventual_observation_id"):
-            issues.append({"severity": "warning", "code": "ACTION_EVIDENCE_MISSING", "action_id": action.get("id")})
-        if evidence.get("locator_unique") is False:
-            issues.append({"severity": "warning", "code": "LOCATOR_NOT_UNIQUE", "action_id": action.get("id")})
-        if action.get("surface") == "browser" and action.get("action_type") in interactive_types:
-            element_id = action.get("target_element_id")
-            if target and not element_id:
-                issues.append({"severity": "warning", "code": "TARGET_ELEMENT_REFERENCE_MISSING", "action_id": action.get("id")})
-            element = elements.get(element_id) if element_id else None
-            primary = dict((element or {}).get("primary_locator") or {})
-            if element_id and (not primary or not primary.get("production_eligible")):
-                issues.append({"severity": "warning", "code": "ACTION_PRODUCTION_LOCATOR_MISSING", "action_id": action.get("id"), "element_id": element_id})
-            elif primary and float(primary.get("stability_score") or 0.0) < 0.55:
-                issues.append({"severity": "warning", "code": "LOW_STABILITY_LOCATOR", "action_id": action.get("id"), "element_id": element_id, "strategy": primary.get("strategy"), "stability_score": primary.get("stability_score")})
-            raw_meta = dict((action.get("raw_event") or {}).get("meta") or {})
-            if not raw_meta.get("event_id"):
-                issues.append({"severity": "warning", "code": "ACTION_RAW_EVENT_LINK_MISSING", "action_id": action.get("id")})
-        if action.get("surface") == "windows" and action.get("action_type") in interactive_types:
-            element_id = action.get("target_element_id")
-            if target and not element_id:
-                issues.append({"severity": "warning", "code": "TARGET_ELEMENT_REFERENCE_MISSING", "action_id": action.get("id")})
-            element = elements.get(element_id) if element_id else None
-            primary = dict((element or {}).get("primary_locator") or {})
-            if element_id and not primary:
-                issues.append({"severity": "warning", "code": "ACTION_PRODUCTION_LOCATOR_MISSING", "action_id": action.get("id"), "element_id": element_id})
-        if previous_action and action.get("action_type") == "fill" and previous_action.get("action_type") == "fill":
-            same_target = bool(action.get("target_element_id")) and action.get("target_element_id") == previous_action.get("target_element_id")
-            old_text = str((previous_action.get("input") or {}).get("text") or "")
-            new_text = str((action.get("input") or {}).get("text") or "")
-            old_ms = ((previous_action.get("raw_event") or {}).get("meta") or {}).get("source_ts_ms")
-            new_ms = ((action.get("raw_event") or {}).get("meta") or {}).get("source_ts_ms")
-            close = False
-            try:
-                close = 0 <= float(new_ms) - float(old_ms) <= 2000
-            except (TypeError, ValueError):
-                pass
-            if same_target and old_text and new_text.startswith(old_text) and new_text != old_text and close:
-                issues.append({"severity": "warning", "code": "DUPLICATE_FILL_TRANSACTION", "action_id": action.get("id"), "previous_action_id": previous_action.get("id"), "element_id": action.get("target_element_id")})
-        previous_action = action
-    journal = root / "journal" / "raw_events.jsonl"
-    if not journal.exists():
-        issues.append({"severity": "error", "code": "RAW_EVENT_JOURNAL_MISSING"})
-    if not actions:
-        issues.append({"severity": "error", "code": "ZERO_ACTIONS"})
-    typing_transactions = 0
-    if journal.exists():
-        for line in journal.read_text(encoding="utf-8").splitlines():
-            try:
-                payload = json.loads(line)
-            except (TypeError, ValueError, json.JSONDecodeError):
-                continue
-            event = dict(payload.get("event") or payload)
-            if event.get("kind") == "typing_activity":
-                typing_transactions += 1
-    fill_count = sum(
-        1 for action in actions if action.get("action_type") in {"fill", "type"}
-    )
-    if typing_transactions > fill_count:
-        issues.append(
-            {
-                "severity": "warning",
-                "code": "TEXT_INPUT_TRANSACTION_MISSING",
-                "typing_transactions": typing_transactions,
-                "fill_actions": fill_count,
-            }
-        )
-    screenshot_count = len(list((root / "observations").glob("*/screen.png")))
-    return {
-        "schema": "openclaw.ui_inspector.validation.v1",
-        "flow_dir": str(root),
-        "action_count": len(actions),
-        "element_count": len(elements),
-        "observation_count": len(list((root / "observations").glob("*/meta.json"))),
-        "screenshot_count": screenshot_count,
-        "error_count": sum(1 for issue in issues if issue["severity"] == "error"),
-        "warning_count": sum(1 for issue in issues if issue["severity"] == "warning"),
-        "issues": issues,
-    }

tools/ui_inspector/diagnostics/replay_preflight.py

@@ -1,133 +0,0 @@
-"""Static, non-mutating replay readiness report for a finished capture."""
-
-from __future__ import annotations
-
-import json
-from pathlib import Path
-from typing import Any
-
-
-_INTERACTIVE = {"click", "fill", "type", "press_key", "select", "toggle"}
-
-
-def _read(path: Path) -> dict[str, Any]:
-    return json.loads(path.read_text(encoding="utf-8"))
-
-
-def build_replay_preflight(flow_dir: Path) -> dict[str, Any]:
-    """Assess whether recorded actions contain executable, verified locators.
-
-    This deliberately does not click or navigate.  `event_time_unique` means
-    the locator count was measured in the event-origin document during capture;
-    it is not represented as a cross-session replay result.
-    """
-    root = Path(flow_dir)
-    actions = sorted(
-        (_read(path) for path in (root / "actions").glob("*.json")),
-        key=lambda item: int(item.get("sequence") or 0),
-    )
-    elements = {
-        item.get("id"): item
-        for item in (_read(path) for path in (root / "elements").glob("*.json"))
-        if item.get("id")
-    }
-    rows: list[dict[str, Any]] = []
-    for action in actions:
-        action_type = str(action.get("action_type") or "")
-        if action_type not in _INTERACTIVE:
-            rows.append({
-                "action_id": action.get("id"),
-                "sequence": action.get("sequence"),
-                "action_type": action_type,
-                "result": "context_only",
-                "reason": "non_interactive_or_navigation",
-            })
-            continue
-        element_id = action.get("target_element_id")
-        element = elements.get(element_id) or {}
-        primary = dict(element.get("primary_locator") or {})
-        eligible = bool(primary.get("production_eligible"))
-        unique = primary.get("unique") is True
-        score = float(primary.get("stability_score") or 0.0)
-        result = "pass" if eligible and unique and score >= 0.55 else "warning"
-        if not eligible or not unique:
-            result = "fail"
-        target = dict(action.get("target_snapshot") or {})
-        bounds = dict(target.get("bounds") or target.get("rect") or {})
-        area = float(bounds.get("width") or 0) * float(bounds.get("height") or 0)
-        oversized_click = action_type == "click" and area > 1_000_000
-        root_window_target = (
-            action_type == "click"
-            and str(target.get("control_type") or "") == "WindowControl"
-        )
-        if oversized_click or root_window_target:
-            result = "fail"
-        validation = str(primary.get("validation") or "none")
-        verification = (
-            "event_time_unique" if validation == "live" and unique
-            else "snapshot_unique" if validation in {"snapshot_only", "live_tree"} and unique
-            else "not_validated"
-        )
-        raw_meta = dict((action.get("raw_event") or {}).get("meta") or {})
-        rows.append({
-            "action_id": action.get("id"),
-            "sequence": action.get("sequence"),
-            "action_type": action_type,
-            "result": result,
-            "element_id": element_id,
-            "logical_name": element.get("logical_name") or "",
-            "locator": primary.get("expression") or "",
-            "strategy": primary.get("strategy") or "",
-            "stability_score": score,
-            "match_count": primary.get("actual_count"),
-            "verification": verification,
-            "page_id": raw_meta.get("page_id") or "",
-            "document_id": raw_meta.get("document_id") or "",
-            "raw_event_ids": list(action.get("raw_event_ids") or []),
-            "reason": (
-                "oversized_or_root_window_click_target"
-                if oversized_click or root_window_target else ""
-            ),
-        })
-    counts = {
-        name: sum(1 for row in rows if row.get("result") == name)
-        for name in ("pass", "warning", "fail", "context_only")
-    }
-    from diagnostics.capture_validator import validate_capture
-
-    quality = validate_capture(root)
-    blocking_codes = {
-        "TEXT_INPUT_TRANSACTION_MISSING",
-        "CLICK_TARGET_GENERIC_CONTAINER",
-        "KEY_TARGET_ROOT_WINDOW",
-        "CLICK_TARGET_OVERSIZED",
-        "ACTION_TARGET_MISSING",
-        "ZERO_ACTIONS",
-    }
-    quality_blockers = [
-        issue
-        for issue in quality.get("issues") or []
-        if issue.get("severity") == "error" or issue.get("code") in blocking_codes
-    ]
-    counts["quality_blockers"] = len(quality_blockers)
-    return {
-        "schema": "openclaw.ui_inspector.replay_preflight.v1",
-        "mode": "static_non_mutating",
-        "note": "事件时刻唯一性不等同于跨会话真实回放；本报告不会点击、输入或导航。",
-        "flow_dir": str(root),
-        "ready": not quality_blockers and counts.get("fail", 0) == 0,
-        "summary": counts,
-        "quality_gate": {
-            "passed": not quality_blockers,
-            "issues": quality_blockers,
-        },
-        "actions": rows,
-    }
-
-
-def write_replay_preflight(flow_dir: Path) -> dict[str, Any]:
-    report = build_replay_preflight(flow_dir)
-    path = Path(flow_dir) / "diagnostics" / "replay_report.json"
-    path.parent.mkdir(parents=True, exist_ok=True)
-    path.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
-    return report

tools/ui_inspector/drivers/__init__.py

@@ -1,29 +0,0 @@
-"""Surface drivers for UI Inspector (windows / browser / mobile stub)."""
-
-from __future__ import annotations
-
-from .base import DumpResult, SurfaceDriver
-from .browser_pw import BrowserPlaywrightDriver
-from .mobile import MobileDriver
-from .windows_uia import WindowsUiaDriver
-import target as _target
-
-
-def get_driver(surface: str) -> SurfaceDriver:
-    if surface == _target.SURFACE_WINDOWS:
-        return WindowsUiaDriver()
-    if surface == _target.SURFACE_BROWSER:
-        return BrowserPlaywrightDriver()
-    if surface in {_target.SURFACE_ANDROID, _target.SURFACE_IOS}:
-        return MobileDriver(surface=surface)
-    raise RuntimeError(f"ERROR:SURFACE_UNSUPPORTED unknown surface={surface!r}")
-
-
-__all__ = [
-    "DumpResult",
-    "SurfaceDriver",
-    "WindowsUiaDriver",
-    "BrowserPlaywrightDriver",
-    "MobileDriver",
-    "get_driver",
-]

tools/ui_inspector/drivers/base.py

@@ -1,38 +0,0 @@
-"""Driver protocol: dump one step's UI tree for a surface."""
-
-from __future__ import annotations
-
-from dataclasses import dataclass, field
-from typing import Any, Optional, Protocol
-
-
-@dataclass
-class DumpResult:
-    surface: str
-    driver: str
-    window: dict[str, Any]
-    tree: dict[str, Any]
-    index: list[dict[str, Any]]
-    locatable: list[dict[str, Any]]
-    stats: dict[str, Any]
-    selection: dict[str, Any] = field(default_factory=dict)
-    url: str = ""
-    page_title: str = ""
-    locatable_text: str = ""
-    tree_text: str = ""
-    screenshot_bytes: Optional[bytes] = None  # if set, artifacts write screen.png from bytes
-    use_hwnd_screenshot: bool = True  # windows: ImageGrab by hwnd
-
-
-class SurfaceDriver(Protocol):
-    def dump(
-        self,
-        *,
-        info: Any,
-        selection_meta: dict[str, Any],
-        max_depth: int = 16,
-        max_nodes: int = 8000,
-        warmup_mode: str = "full",
-        merge_same_pid_if_sparse: bool = True,
-        warmed_pids: Optional[set[int]] = None,
-    ) -> DumpResult: ...

tools/ui_inspector/drivers/browser_pw.py

@@ -1,90 +0,0 @@
-"""Browser whole-page dump via Inspector Playwright session + kit page_dump."""
-
-from __future__ import annotations
-
-from typing import Any, Optional
-
-import kit_browser
-import target
-from .base import DumpResult
-
-
-class BrowserPlaywrightDriver:
-    surface = target.SURFACE_BROWSER
-    driver_name = "playwright"
-
-    def dump(
-        self,
-        *,
-        info: Any,
-        selection_meta: dict[str, Any],
-        max_depth: int = 16,
-        max_nodes: int = 8000,
-        warmup_mode: str = "full",
-        merge_same_pid_if_sparse: bool = True,
-        warmed_pids: Optional[set[int]] = None,
-        page_id: str = "",
-        document_id: str = "",
-        frame_id: str = "",
-    ) -> DumpResult:
-        del max_depth, warmup_mode, merge_same_pid_if_sparse, warmed_pids  # unused
-        sess = kit_browser.ensure_session()
-        dump = kit_browser.dump_current_page(
-            max_nodes=max_nodes, locatable_limit=300, page_id=page_id
-        )
-        try:
-            png = kit_browser.screenshot_png(page_id=page_id)
-        except Exception:
-            png = None
-
-        from jiangchang_skill_core.rpa.page_dump import (
-            format_locatable_text,
-            format_tree_text,
-        )
-
-        url = dump.get("url") or ""
-        page_title = dump.get("title") or ""
-        window = info.to_dict() if hasattr(info, "to_dict") else dict(info or {})
-        # Prefer live page identity for labeling / meta
-        window = {
-            **window,
-            "title": page_title or window.get("title") or "",
-            "class_name": window.get("class_name") or "PlaywrightPage",
-            "process_name": window.get("process_name") or "chrome.exe",
-            "url": url,
-        }
-        stats = dict(dump.get("stats") or {})
-        active_page = dict(stats.get("active_page") or {})
-        stats["browser_session"] = {
-            "profile_dir": sess.get("profile_dir"),
-            "reused": sess.get("reused"),
-            "page_count": active_page.get("page_count") or sess.get("page_count"),
-            "resolve": active_page.get("resolve") or "visible",
-            "note": (
-                "dump 来自 Inspector 自管 Playwright 的当前可见标签页"
-                "（document.visibilityState），不是系统 Chrome 的 UIA，"
-                "也不是固定的 pages[0]。多标签时请切到目标页再拾取。"
-            ),
-        }
-        sel = dict(selection_meta or {})
-        sel["surface"] = self.surface
-        sel["browser_url"] = url
-        sel["page_id"] = page_id or active_page.get("page_id") or ""
-        sel["document_id"] = document_id
-        sel["frame_id"] = frame_id
-        return DumpResult(
-            surface=self.surface,
-            driver=self.driver_name,
-            window=window,
-            tree=dump.get("tree") or {},
-            index=list(dump.get("index") or []),
-            locatable=list(dump.get("locatable") or []),
-            stats=stats,
-            selection=sel,
-            url=url,
-            page_title=page_title,
-            locatable_text=format_locatable_text(dump.get("locatable") or []),
-            tree_text=format_tree_text(dump.get("tree") or {}),
-            screenshot_bytes=png,
-            use_hwnd_screenshot=False,
-        )

tools/ui_inspector/drivers/mobile.py

@@ -1,30 +0,0 @@
-"""Mobile surface stub — reserved for future android/ios dump."""
-
-from __future__ import annotations
-
-from typing import Any, Optional
-
-import target
-from .base import DumpResult
-
-
-class MobileDriver:
-    def __init__(self, *, surface: str = target.SURFACE_ANDROID) -> None:
-        self.surface = surface
-        self.driver_name = "mobile_stub"
-
-    def dump(
-        self,
-        *,
-        info: Any,
-        selection_meta: dict[str, Any],
-        max_depth: int = 16,
-        max_nodes: int = 8000,
-        warmup_mode: str = "full",
-        merge_same_pid_if_sparse: bool = True,
-        warmed_pids: Optional[set[int]] = None,
-    ) -> DumpResult:
-        raise RuntimeError(
-            f"ERROR:SURFACE_UNSUPPORTED 手机端拾取尚未实现（surface={self.surface}）。"
-            "请先用桌面或浏览器目标。"
-        )

tools/ui_inspector/drivers/windows_uia.py

@@ -1,58 +0,0 @@
-"""Windows desktop dump via existing desktop_uia."""
-
-from __future__ import annotations
-
-from typing import Any, Optional
-
-import desktop_uia as uia
-import target
-from .base import DumpResult
-
-
-class WindowsUiaDriver:
-    surface = target.SURFACE_WINDOWS
-    driver_name = "uiautomation"
-
-    def dump(
-        self,
-        *,
-        info: Any,
-        selection_meta: dict[str, Any],
-        max_depth: int = 16,
-        max_nodes: int = 8000,
-        warmup_mode: str = "full",
-        merge_same_pid_if_sparse: bool = True,
-        warmed_pids: Optional[set[object]] = None,
-    ) -> DumpResult:
-        warmed = warmed_pids if warmed_pids is not None else set()
-        mode = warmup_mode
-        warm_key = (int(info.pid), str(info.class_name or ""))
-        if (warm_key in warmed or int(info.pid) in warmed) and mode == "full":
-            mode = "light"
-        merge_sparse = merge_same_pid_if_sparse
-        if (info.class_name or "") == "SearchResultWindow2":
-            merge_sparse = False
-        tree, index, stats, locatable = uia.dump_window_tree(
-            info.hwnd,
-            max_depth=max_depth,
-            max_nodes=max_nodes,
-            only_visible=False,
-            warmup_mode=mode,
-            merge_same_pid_if_sparse=merge_sparse,
-        )
-        return DumpResult(
-            surface=self.surface,
-            driver=self.driver_name,
-            window=info.to_dict(),
-            tree=tree,
-            index=index,
-            locatable=locatable,
-            stats=stats,
-            selection=dict(selection_meta or {}),
-            url="",
-            page_title="",
-            locatable_text=uia.format_locatable_text(locatable),
-            tree_text=uia.format_tree_text(tree),
-            screenshot_bytes=None,
-            use_hwnd_screenshot=True,
-        )

tools/ui_inspector/evidence/__init__.py

@@ -1,6 +0,0 @@
-"""Asynchronous evidence capture."""
-
-from .coordinator import EvidenceCoordinator
-
-__all__ = ["EvidenceCoordinator"]
-

tools/ui_inspector/evidence/coordinator.py

@@ -1,255 +0,0 @@
-"""Coalesced background full-page evidence capture.
-
-Semantic actions are committed immediately.  Slow dumps are supplementary
-evidence and are explicitly labelled eventual instead of being misrepresented
-as an event-time after-state.
-"""
-
-from __future__ import annotations
-
-import queue
-import threading
-import time
-from dataclasses import dataclass
-from typing import Any, Callable, Optional
-
-
-@dataclass
-class EvidenceRequest:
-    action_id: str
-    context: dict[str, Any]
-    requested_monotonic: float
-
-
-class EvidenceCoordinator:
-    def __init__(
-        self,
-        manager: Any,
-        dump_fn: Callable[..., Any],
-        *,
-        coalesce_sec: float = 0.25,
-        on_dump: Optional[Callable[[Any], None]] = None,
-    ) -> None:
-        self.manager = manager
-        self.dump_fn = dump_fn
-        self.coalesce_sec = coalesce_sec
-        self.on_dump = on_dump
-        self._queue: queue.Queue[EvidenceRequest | None] = queue.Queue(maxsize=512)
-        self._urgent_queue: queue.Queue[EvidenceRequest | None] = queue.Queue(maxsize=128)
-        self._thread: Optional[threading.Thread] = None
-        self._urgent_thread: Optional[threading.Thread] = None
-        self._stop = threading.Event()
-        self.dropped = 0
-        self.capture_count = 0
-        self.failure_count = 0
-
-    def start(self) -> None:
-        if self._thread and self._thread.is_alive():
-            return
-        self._stop.clear()
-        self._thread = threading.Thread(target=self._run, name="ui-inspector-evidence", daemon=True)
-        self._thread.start()
-        if str(getattr(self.manager.action_flow, "surface", "")) == "windows":
-            self._urgent_thread = threading.Thread(
-                target=self._run_urgent,
-                name="ui-inspector-evidence-transient",
-                daemon=True,
-            )
-            self._urgent_thread.start()
-
-    def schedule(self, action: Any) -> None:
-        raw = dict(getattr(action, "raw_event", {}) or {})
-        meta = dict(raw.get("meta") or {})
-        context = {
-            "page_id": meta.get("page_id") or "",
-            "document_id": meta.get("document_id") or "",
-            "frame_id": meta.get("frame_id") or "",
-            "source_ts_ms": meta.get("source_ts_ms"),
-            "url": raw.get("url") or "",
-        }
-        scope = dict((raw.get("target") or {}).get("window_scope") or {})
-        context.update(
-            {
-                "target_hwnd": int(scope.get("hwnd") or 0),
-                "target_pid": int(scope.get("pid") or 0),
-                "target_class_name": str(scope.get("class_name") or ""),
-                "target_title": str(scope.get("title") or ""),
-            }
-        )
-        out_queue = (
-            self._urgent_queue
-            if context["target_class_name"] == "SearchResultWindow2"
-            else self._queue
-        )
-        try:
-            out_queue.put_nowait(EvidenceRequest(action.id, context, time.monotonic()))
-        except queue.Full:
-            self.dropped += 1
-
-    def prime(self) -> None:
-        """Start one background enrichment without creating a semantic action."""
-        try:
-            self._queue.put_nowait(EvidenceRequest("", {}, time.monotonic()))
-        except queue.Full:
-            self.dropped += 1
-
-    def _dump(self, context: dict[str, Any]) -> Any:
-        kwargs = {
-            k: context.get(k)
-            for k in (
-                "page_id", "document_id", "frame_id", "target_hwnd",
-                "target_pid", "target_class_name", "target_title",
-            )
-            if context.get(k)
-        }
-        if kwargs:
-            try:
-                return self.dump_fn(**kwargs)
-            except TypeError as exc:
-                if "unexpected keyword" not in str(exc) and "positional" not in str(exc):
-                    raise
-        return self.dump_fn()
-
-    def _run(self) -> None:
-        while not self._stop.is_set():
-            try:
-                first = self._queue.get(timeout=0.2)
-            except queue.Empty:
-                continue
-            if first is None:
-                self._queue.task_done()
-                self._stop.set()
-                break
-            batch = [first]
-            stop_after_batch = False
-            deadline = time.monotonic() + self.coalesce_sec
-            while time.monotonic() < deadline:
-                try:
-                    item = self._queue.get(timeout=max(0.01, deadline - time.monotonic()))
-                except queue.Empty:
-                    break
-                if item is None:
-                    self._queue.task_done()
-                    stop_after_batch = True
-                    break
-                batch.append(item)
-
-            groups: dict[str, list[EvidenceRequest]] = {}
-            for item in batch:
-                key = str(item.context.get("page_id") or "active")
-                groups.setdefault(key, []).append(item)
-            for items in groups.values():
-                latest = items[-1]
-                try:
-                    dump = self._dump(latest.context)
-                    if self.on_dump is not None:
-                        self.on_dump(dump)
-                    self.capture_count += 1
-                    # A desktop provider dump can take seconds. Actions that
-                    # arrived while it was running are still earlier than this
-                    # eventual observation, so fold them into the same result
-                    # instead of immediately launching another identical dump.
-                    if str(getattr(self.manager.action_flow, "surface", "")) == "windows":
-                        while True:
-                            try:
-                                late = self._queue.get_nowait()
-                            except queue.Empty:
-                                break
-                            if late is None:
-                                self._queue.task_done()
-                                stop_after_batch = True
-                                break
-                            batch.append(late)
-                            items.append(late)
-                    if not self._stop.is_set():
-                        self.manager.attach_eventual_evidence(
-                            [item.action_id for item in items], dump=dump,
-                            requested_monotonic=min(item.requested_monotonic for item in items),
-                        )
-                except Exception as exc:
-                    self.failure_count += 1
-                    if not self._stop.is_set():
-                        self.manager.note_evidence_failure([item.action_id for item in items], str(exc))
-            for _item in batch:
-                self._queue.task_done()
-            if stop_after_batch:
-                self._stop.set()
-
-    def _run_urgent(self) -> None:
-        """Capture transient native popups independently of a slow main dump."""
-        while not self._stop.is_set():
-            try:
-                first = self._urgent_queue.get(timeout=0.1)
-            except queue.Empty:
-                continue
-            if first is None:
-                self._urgent_queue.task_done()
-                break
-            batch = [first]
-            # A very short coalesce keeps double-clicks together without waiting
-            # long enough for a search-result popup to disappear.
-            deadline = time.monotonic() + 0.04
-            while time.monotonic() < deadline:
-                try:
-                    item = self._urgent_queue.get_nowait()
-                except queue.Empty:
-                    time.sleep(0.005)
-                    continue
-                if item is None:
-                    self._urgent_queue.task_done()
-                    break
-                if item.context.get("target_hwnd") == first.context.get("target_hwnd"):
-                    batch.append(item)
-                else:
-                    # Preserve a different transient window for the next pass.
-                    try:
-                        self._urgent_queue.put_nowait(item)
-                    except queue.Full:
-                        self.dropped += 1
-                    self._urgent_queue.task_done()
-                    break
-            try:
-                dump = self._dump(batch[-1].context)
-                if self.on_dump is not None:
-                    self.on_dump(dump)
-                self.capture_count += 1
-                if not self._stop.is_set():
-                    self.manager.attach_eventual_evidence(
-                        [item.action_id for item in batch],
-                        dump=dump,
-                        requested_monotonic=min(item.requested_monotonic for item in batch),
-                    )
-            except Exception as exc:
-                self.failure_count += 1
-                if not self._stop.is_set():
-                    self.manager.note_evidence_failure(
-                        [item.action_id for item in batch], str(exc)
-                    )
-            finally:
-                for _item in batch:
-                    self._urgent_queue.task_done()
-
-    def stop(self, *, timeout: float = 35.0) -> None:
-        if not self._thread:
-            return
-        # Drain the transient-window lane first.  A WeCom main-window dump can
-        # be stuck in an uninterruptible provider call for tens of seconds; it
-        # must not cancel the short-lived popup evidence when the user ends the
-        # flow immediately after selecting a result.
-        if self._urgent_thread:
-            try:
-                self._urgent_queue.put(None, timeout=1.0)
-            except queue.Full:
-                pass
-            self._urgent_thread.join(timeout=min(max(timeout, 3.0), 8.0))
-        try:
-            self._queue.put(None, timeout=2.0)
-        except queue.Full:
-            pass
-        self._thread.join(timeout=timeout)
-        if self._thread.is_alive():
-            # A provider dump may be uninterruptible. Prevent a late result
-            # from mutating a manager which is already being finalized.
-            self._stop.set()
-        self._thread = None
-        self._urgent_thread = None

tools/ui_inspector/flow_session.py

@@ -1,408 +0,0 @@
-"""Flow session glue: v2 ActionRecordingManager + surface recorders + v1 dual-write."""
-
-from __future__ import annotations
-
-import queue
-import threading
-import uuid
-from pathlib import Path
-from typing import Any, Callable, Optional
-
-import artifacts as art
-from core.pipeline.event_journal import EventJournal
-from evidence.coordinator import EvidenceCoordinator
-from recorder.base import RawUserEvent
-from recorder.manager import ActionRecordingManager
-
-
-class FlowSession:
-    def __init__(
-        self,
-        flow_dir: Path,
-        *,
-        flow_name: str,
-        surface: str,
-        recording_mode: str = "auto",
-        dump_fn: Optional[Callable[[], Any]] = None,
-        write_v1_step: Optional[Callable[..., Any]] = None,
-        exclude_hwnds: Optional[Callable[[], set[int]]] = None,
-        on_action_update: Optional[Callable[[Any, str], None]] = None,
-        stability_timeout_sec: float = 3.0,
-    ) -> None:
-        self.flow_dir = Path(flow_dir)
-        self.surface = surface
-        self.recording_mode = recording_mode
-        self._dump_fn = dump_fn
-        self.exclude_hwnds = exclude_hwnds or (lambda: set())
-        self.manager = ActionRecordingManager(
-            self.flow_dir,
-            flow_name=flow_name,
-            flow_id=self.flow_dir.name,
-            surface=surface,
-            recording_mode=recording_mode,
-            dump_fn=dump_fn,
-            write_v1_step=write_v1_step,
-            stability_timeout_sec=stability_timeout_sec,
-            defer_action_evidence=(
-                surface in {"browser", "windows"}
-                and recording_mode != "legacy_interval"
-            ),
-            on_action_update=on_action_update,
-        )
-        self.journal = EventJournal(self.flow_dir)
-        self._evidence = (
-            EvidenceCoordinator(self.manager, dump_fn, on_dump=self._on_evidence_dump)
-            if surface in {"browser", "windows"}
-            and dump_fn is not None
-            and recording_mode != "legacy_interval"
-            else None
-        )
-        if self._evidence is not None:
-            self.manager.on_action = self._evidence.schedule
-        self._recorder = None
-        self._recorder_health: dict[str, Any] = {}
-        self._event_queue: queue.Queue[tuple[str, Any]] = queue.Queue(maxsize=512)
-        self._event_thread: Optional[threading.Thread] = None
-        self._event_stop = threading.Event()
-        self._stats_lock = threading.Lock()
-        self._stats_cache: dict[str, Any] = {
-            "recorder_status": "idle",
-            "surface_recorder": "off",
-            "state_count": 0,
-            "observation_count": 0,
-            "action_count": 0,
-            "element_count": 0,
-            "queued_events": 0,
-            "dropped_events": 0,
-        }
-        self._dropped_events = 0
-
-    def start(self, *, entry_dump: Optional[Any] = None) -> None:
-        if self.recording_mode == "legacy_interval":
-            self.manager.start(entry_dump=entry_dump)
-            self._refresh_stats()
-            return
-        # Start observation before entry persistence. Events are journaled and
-        # buffered until manager.start() completes, eliminating startup loss.
-        self._start_surface_recorder()
-        if self.surface == "windows" and entry_dump is not None:
-            self._on_evidence_dump(entry_dump)
-        if self._evidence is not None:
-            self._evidence.start()
-        try:
-            self.manager.start(entry_dump=entry_dump)
-        except Exception:
-            if self._recorder:
-                self._recorder.stop()
-                self._recorder = None
-            if self._evidence is not None:
-                self._evidence.stop(timeout=3.0)
-            raise
-        self._start_event_consumer()
-        if self.surface == "windows" and self._evidence is not None:
-            # Warm/enrich UIA immediately, but on the evidence worker. Raw
-            # recording and Action reduction remain fully independent.
-            self._evidence.prime()
-        self._refresh_stats()
-
-    def _submit(self, kind: str, payload: Any = None) -> bool:
-        try:
-            if kind == "stop":
-                self._event_queue.put((kind, payload), timeout=2.0)
-            else:
-                self._event_queue.put_nowait((kind, payload))
-            return True
-        except queue.Full:
-            self._dropped_events += 1
-            self.manager.last_warning = (
-                f"recorder queue full; dropped={self._dropped_events}"
-            )
-            self._refresh_stats()
-            return False
-
-    def _start_event_consumer(self) -> None:
-        if self._event_thread and self._event_thread.is_alive():
-            return
-        self._event_stop.clear()
-
-        def consume() -> None:
-            while not self._event_stop.is_set():
-                try:
-                    kind, payload = self._event_queue.get(timeout=0.2)
-                except queue.Empty:
-                    continue
-                try:
-                    if kind == "stop":
-                        break
-                    if kind == "event":
-                        self.manager.handle_raw_event(payload)
-                    elif kind == "heartbeat":
-                        self.manager.heartbeat()
-                    elif kind == "pause":
-                        self.manager.pause()
-                    elif kind == "resume":
-                        self.manager.resume()
-                except Exception as exc:
-                    self.manager.last_warning = str(exc)
-                finally:
-                    self._event_queue.task_done()
-                    self._refresh_stats()
-            self._event_stop.set()
-
-        self._event_thread = threading.Thread(
-            target=consume,
-            name="ui-inspector-action-consumer",
-            daemon=True,
-        )
-        self._event_thread.start()
-
-    def _start_surface_recorder(self) -> None:
-        def on_event(ev: RawUserEvent) -> None:
-            # Preserve the observed order and bound memory/thread growth.
-            if self.surface == "windows" and not (ev.meta or {}).get("event_id"):
-                ev.meta["event_id"] = f"windows:{uuid.uuid4().hex}"
-            self.journal.append(ev, channel="semantic")
-            self._submit("event", ev)
-
-        def on_raw_event(payload: dict[str, Any]) -> None:
-            self.journal.append(payload, channel="browser_dom")
-
-        if self.surface == "browser":
-            from surfaces.browser import BrowserRecorder
-
-            self._recorder = BrowserRecorder(
-                on_event=on_event, on_raw_event=on_raw_event
-            )
-        else:
-            from surfaces.windows import WindowsRecorder
-
-            self._recorder = WindowsRecorder(
-                on_event=on_event,
-                exclude_hwnds=self.exclude_hwnds,
-            )
-            self._seed_windows_element_atlas()
-        self._recorder.start()
-
-    def _on_evidence_dump(self, dump: Any) -> None:
-        recorder = self._recorder
-        if self.surface != "windows" or recorder is None or not hasattr(recorder, "learn_elements"):
-            return
-        try:
-            recorder.learn_elements(
-                list(getattr(dump, "locatable", None) or []),
-                window=dict(getattr(dump, "window", None) or {}),
-            )
-        except Exception:
-            pass
-
-    def _seed_windows_element_atlas(self) -> None:
-        """Reuse recent captures as the event-time element repository.
-
-        Values are discarded by ``learn_elements``; only stable names and
-        geometry are retained. This lets a new recording identify custom
-        editors before their slow accessibility provider completes a new dump.
-        """
-        recorder = self._recorder
-        if recorder is None or not hasattr(recorder, "learn_elements"):
-            return
-        try:
-            siblings = sorted(
-                (
-                    path for path in self.flow_dir.parent.iterdir()
-                    if path.is_dir() and path != self.flow_dir
-                ),
-                key=lambda path: path.stat().st_mtime,
-                reverse=True,
-            )[:8]
-        except Exception:
-            return
-        learned_classes: set[str] = set()
-        for sibling in siblings:
-            observations = sibling / "observations"
-            if not observations.is_dir():
-                continue
-            try:
-                metas = sorted(
-                    observations.glob("*/meta.json"),
-                    key=lambda path: path.stat().st_mtime,
-                    reverse=True,
-                )
-            except Exception:
-                continue
-            for meta_path in metas:
-                try:
-                    import json
-
-                    meta = json.loads(meta_path.read_text(encoding="utf-8"))
-                    if str(meta.get("surface") or "") != "windows":
-                        continue
-                    selection = dict(meta.get("selection") or {})
-                    class_name = str(
-                        selection.get("chosen_class")
-                        or selection.get("event_class")
-                        or ""
-                    )
-                    if not class_name or class_name in learned_classes:
-                        continue
-                    locatable_path = meta_path.parent / "locatable.json"
-                    rows = json.loads(locatable_path.read_text(encoding="utf-8"))
-                    recorder.learn_elements(
-                        rows,
-                        window={
-                            "class_name": class_name,
-                            "hwnd": int(selection.get("chosen_hwnd") or 0),
-                            "pid": int(selection.get("chosen_pid") or 0),
-                            "title": str(selection.get("event_title") or ""),
-                        },
-                    )
-                    learned_classes.add(class_name)
-                except Exception:
-                    continue
-            if "WeWorkWindow" in learned_classes and "SearchResultWindow2" in learned_classes:
-                break
-
-    def pause(self) -> None:
-        if self._recorder:
-            self._recorder.pause()
-        self._submit("pause")
-
-    def resume(self) -> None:
-        if self._recorder:
-            self._recorder.resume()
-        self._submit("resume")
-
-    def stop(self) -> None:
-        recorder_stop_error: Optional[BaseException] = None
-        if self._recorder:
-            try:
-                self._recorder.stop()
-            except Exception as exc:
-                recorder_stop_error = exc
-            if hasattr(self._recorder, "health"):
-                try:
-                    self._recorder_health = self._recorder.health()
-                except Exception as exc:
-                    self._recorder_health = {"last_error": str(exc)}
-            self._recorder = None
-        if self._event_thread and self._event_thread.is_alive():
-            self._submit("stop")
-            self._event_thread.join(timeout=35.0)
-            if self._event_thread.is_alive():
-                self.manager.last_warning = "event consumer did not stop within 35s"
-                raise RuntimeError(self.manager.last_warning)
-            else:
-                self._event_thread = None
-        self._event_stop.set()
-        if self._evidence is not None:
-            self._evidence.stop(timeout=3.0)
-        self.manager.stop()
-        # Point flow.json v2 fields (already present from create_flow).
-        try:
-            flow = art.load_flow(self.flow_dir)
-            v2 = dict(flow.get("v2") or {})
-            v2["action_count"] = len(self.manager.action_flow.actions)
-            v2["state_count"] = len(self.manager.action_flow.states)
-            v2["observation_count"] = len(
-                self.manager.snapshots.observation_ids()
-            )
-            v2["element_count"] = len(self.manager.catalog.catalog.elements)
-            v2["raw_event_journal"] = "journal/raw_events.jsonl"
-            v2["pages"] = "pages.json"
-            v2["diagnostics_dir"] = "diagnostics/"
-            v2["validation"] = "diagnostics/validation.json"
-            v2["replay_report"] = "diagnostics/replay_report.json"
-            flow["v2"] = v2
-            art.save_flow(self.flow_dir, flow)
-        except Exception:
-            pass
-        self._write_diagnostics()
-        self._refresh_stats()
-        if recorder_stop_error is not None:
-            self.manager.last_warning = str(recorder_stop_error)
-            raise RuntimeError(str(recorder_stop_error)) from recorder_stop_error
-
-    def checkpoint(self) -> Any:
-        return self.manager.record_checkpoint()
-
-    def heartbeat(self) -> None:
-        self._submit("heartbeat")
-
-    def stats(self) -> dict[str, Any]:
-        with self._stats_lock:
-            return dict(self._stats_cache)
-
-    def _refresh_stats(self) -> None:
-        st = self.manager.stats()
-        st["surface_recorder"] = (
-            self._recorder.status if self._recorder else "off"
-        )
-        st["queued_events"] = self._event_queue.qsize()
-        st["dropped_events"] = self._dropped_events
-        if self._recorder is not None and hasattr(self._recorder, "health"):
-            try:
-                st["input_backend"] = self._recorder.health()
-            except Exception as exc:
-                st["input_backend"] = {"last_error": str(exc)}
-        elif self._recorder_health:
-            st["input_backend"] = dict(self._recorder_health)
-        if self._evidence is not None:
-            st["evidence_capture_count"] = self._evidence.capture_count
-            st["evidence_dropped"] = self._evidence.dropped
-            st["evidence_queued"] = self._evidence._queue.qsize()
-            st["transient_evidence_queued"] = self._evidence._urgent_queue.qsize()
-            st["evidence_failures"] = self._evidence.failure_count
-        with self._stats_lock:
-            self._stats_cache = st
-
-    def _write_diagnostics(self) -> None:
-        pages: dict[str, dict[str, Any]] = {}
-        lags: list[int] = []
-        for action in self.manager._actions_by_id.values():
-            raw = dict(action.raw_event or {})
-            meta = dict(raw.get("meta") or {})
-            page_id = str(meta.get("page_id") or "")
-            if page_id:
-                page = pages.setdefault(page_id, {"page_id": page_id, "urls": [], "document_ids": []})
-                url = str(raw.get("url") or (action.input or {}).get("url") or "")
-                document_id = str(meta.get("document_id") or "")
-                if url and url not in page["urls"]:
-                    page["urls"].append(url)
-                if document_id and document_id not in page["document_ids"]:
-                    page["document_ids"].append(document_id)
-            lag = (action.evidence or {}).get("evidence_lag_ms")
-            if isinstance(lag, int):
-                lags.append(lag)
-        diagnostics = self.flow_dir / "diagnostics"
-        diagnostics.mkdir(parents=True, exist_ok=True)
-        try:
-            art.write_json(self.flow_dir / "pages.json", {
-                "schema": "openclaw.ui_inspector.pages.v1",
-                "pages": list(pages.values()),
-            })
-            recorder_health = dict(self._recorder_health)
-            if self._recorder is not None and hasattr(self._recorder, "health"):
-                try:
-                    recorder_health = self._recorder.health()
-                except Exception as exc:
-                    recorder_health = {"last_error": str(exc)}
-            art.write_json(diagnostics / "metrics.json", {
-                "schema": "openclaw.ui_inspector.metrics.v1",
-                "raw_event_count": self.journal._sequence,
-                "action_count": len(self.manager.action_flow.actions),
-                "state_count": len(self.manager.action_flow.states),
-                "observation_count": len(self.manager.snapshots.observation_ids()),
-                "element_count": len(self.manager.catalog.catalog.elements),
-                "event_queue_dropped": self._dropped_events,
-                "evidence_queue_dropped": self._evidence.dropped if self._evidence else 0,
-                "evidence_capture_count": self._evidence.capture_count if self._evidence else 0,
-                "evidence_failure_count": self._evidence.failure_count if self._evidence else 0,
-                "evidence_lag_ms_max": max(lags) if lags else None,
-                "evidence_lag_ms_average": round(sum(lags) / len(lags), 2) if lags else None,
-                "input_backend": recorder_health,
-            })
-            from diagnostics import validate_capture, write_replay_preflight
-
-            art.write_json(diagnostics / "validation.json", validate_capture(self.flow_dir))
-            write_replay_preflight(self.flow_dir)
-        except Exception as exc:
-            self.manager.last_warning = f"diagnostics write failed: {exc}"

tools/ui_inspector/kit_browser.py

@@ -1,729 +0,0 @@
-"""Inspector-owned Playwright browser session (dev-time page dump).
-
-All Playwright sync API calls run on one dedicated worker thread
-(Playwright sync is not cross-thread safe).
-
-Dump / screenshot / recorder target the **currently visible** page
-(document.visibilityState === 'visible'), not a sticky pages[0].
-This keeps list ↔ detail tab switching aligned with what the user sees.
-
-Uses jiangchang_skill_core.rpa stealth launch parts + dump_page_sync.
-Profile lives under ui_inspector/_browser_profile (not account-manager).
-"""
-
-from __future__ import annotations
-
-import atexit
-import queue
-import threading
-import uuid
-from pathlib import Path
-from typing import Any, Callable, Optional
-
-_HERE = Path(__file__).resolve().parent
-_DEFAULT_PROFILE = _HERE / "_browser_profile"
-_DEFAULT_START_URL = "about:blank"
-
-_cmd_q: queue.Queue = queue.Queue()
-_worker: Optional[threading.Thread] = None
-_worker_lock = threading.Lock()
-
-# Owned exclusively by worker thread:
-_pw = None
-_context = None
-_page = None
-_started = False
-_bridge_installed = False
-_bridge_sink: Optional[Callable[[dict[str, Any]], None]] = None
-_bridge_lock = threading.Lock()
-_BRIDGE_NAME = "__openclawUiInspectorEmit"
-_page_ids: dict[int, str] = {}
-_page_refs: dict[int, Any] = {}
-_pages_by_id: dict[str, Any] = {}
-_frame_ids: dict[int, str] = {}
-_frame_refs: dict[int, Any] = {}
-
-
-def _page_id_for(page: Any) -> str:
-    if not _page_is_open(page):
-        return ""
-    key = id(page)
-    page_id = _page_ids.get(key) if _page_refs.get(key) is page else None
-    if not page_id:
-        page_id = f"page_{uuid.uuid4().hex[:12]}"
-        _page_ids[key] = page_id
-        _page_refs[key] = page
-    _pages_by_id[page_id] = page
-    return page_id
-
-
-def _frame_id_for(frame: Any, *, page_id: str = "") -> str:
-    if frame is None:
-        return ""
-    key = id(frame)
-    frame_id = _frame_ids.get(key) if _frame_refs.get(key) is frame else None
-    if not frame_id:
-        prefix = page_id or "page"
-        frame_id = f"{prefix}:frame_{uuid.uuid4().hex[:10]}"
-        _frame_ids[key] = frame_id
-        _frame_refs[key] = frame
-    return frame_id
-
-
-def _page_for_id(page_id: str) -> Any:
-    page = _pages_by_id.get(str(page_id or ""))
-    if _page_is_open(page):
-        return page
-    if page_id:
-        _pages_by_id.pop(str(page_id), None)
-    return None
-
-
-def kit_page_dump_available() -> bool:
-    try:
-        from jiangchang_skill_core.rpa.page_dump import dump_page_sync  # noqa: F401
-
-        return True
-    except ImportError:
-        return False
-
-
-def profile_dir() -> Path:
-    return _DEFAULT_PROFILE
-
-
-def _ensure_worker() -> None:
-    global _worker
-    with _worker_lock:
-        if _worker is not None and _worker.is_alive():
-            return
-
-        def loop() -> None:
-            while True:
-                item = _cmd_q.get()
-                if item is None:
-                    break
-                fn, args, kwargs, out_q = item
-                try:
-                    out_q.put(("ok", fn(*args, **kwargs)))
-                except BaseException as exc:
-                    out_q.put(("err", exc))
-
-        _worker = threading.Thread(target=loop, name="ui-inspector-pw", daemon=True)
-        _worker.start()
-        atexit.register(shutdown)
-
-
-def _call_on_worker(
-    fn: Callable[..., Any],
-    *args: Any,
-    timeout_sec: float = 30.0,
-    **kwargs: Any,
-) -> Any:
-    _ensure_worker()
-    out_q: queue.Queue = queue.Queue()
-    _cmd_q.put((fn, args, kwargs, out_q))
-    try:
-        status, payload = out_q.get(timeout=max(0.1, float(timeout_sec)))
-    except queue.Empty as exc:
-        raise RuntimeError(
-            f"ERROR:BROWSER_WORKER_TIMEOUT Playwright 命令超过 {timeout_sec:g} 秒未返回。"
-        ) from exc
-    if status == "err":
-        raise payload
-    return payload
-
-
-def _teardown_unlocked() -> None:
-    global _pw, _context, _page, _started, _bridge_installed, _bridge_sink
-    try:
-        if _context is not None:
-            _context.close()
-    except Exception:
-        pass
-    try:
-        if _pw is not None:
-            _pw.stop()
-    except Exception:
-        pass
-    _pw = None
-    _context = None
-    _page = None
-    _started = False
-    _bridge_installed = False
-    with _bridge_lock:
-        _bridge_sink = None
-    _page_ids.clear()
-    _page_refs.clear()
-    _pages_by_id.clear()
-    _frame_ids.clear()
-    _frame_refs.clear()
-
-
-def _goto_safe(page: Any, start_url: str) -> dict[str, Any]:
-    url = (start_url or "").strip()
-    if not url or url == "about:blank":
-        return {"ok": True, "requested_url": url, "actual_url": page.url or ""}
-    try:
-        page.goto(url, wait_until="domcontentloaded", timeout=60_000)
-    except Exception as exc:
-        actual = ""
-        try:
-            actual = str(page.url or "")
-        except Exception:
-            pass
-        # Some sites keep loading after domcontentloaded timeout.  A real HTTP
-        # page is usable, but the warning must remain visible to callers.
-        if actual.startswith(("http://", "https://")):
-            return {
-                "ok": True,
-                "requested_url": url,
-                "actual_url": actual,
-                "warning": str(exc),
-            }
-        raise RuntimeError(
-            f"ERROR:NAVIGATION_FAILED requested={url!r} actual={actual!r}: {exc}"
-        ) from exc
-    actual = str(page.url or "")
-    if not actual or actual == "about:blank" or actual.startswith("about:"):
-        raise RuntimeError(
-            f"ERROR:NAVIGATION_FAILED requested={url!r} actual={actual!r}"
-        )
-    return {"ok": True, "requested_url": url, "actual_url": actual}
-
-
-def _page_is_open(page: Any) -> bool:
-    try:
-        return page is not None and not page.is_closed()
-    except Exception:
-        return False
-
-
-def _open_pages(context: Any) -> list[Any]:
-    if context is None:
-        return []
-    out: list[Any] = []
-    try:
-        pages = list(context.pages or [])
-    except Exception:
-        return []
-    for p in pages:
-        if _page_is_open(p):
-            _page_id_for(p)
-            out.append(p)
-    return out
-
-
-def _page_visibility(page: Any) -> str:
-    """Return document.visibilityState or '' on failure."""
-    if not _page_is_open(page):
-        return ""
-    try:
-        state = page.evaluate("() => document.visibilityState")
-        return str(state or "").strip().lower()
-    except Exception:
-        return ""
-
-
-def resolve_active_page(
-    context: Any,
-    *,
-    preferred: Any = None,
-) -> tuple[Any, dict[str, Any]]:
-    """Pick the page the user is looking at (visible tab).
-
-    Returns (page, meta). Prefer visibilityState===visible; fall back to
-    preferred if still open, else first open page.
-    """
-    pages = _open_pages(context)
-    meta: dict[str, Any] = {
-        "resolve": "none",
-        "page_count": len(pages),
-        "visible_count": 0,
-    }
-    if not pages:
-        raise RuntimeError("ERROR:NO_OPEN_PAGE Playwright 会话中没有可用标签页。")
-
-    visible: list[Any] = []
-    for p in pages:
-        if _page_visibility(p) == "visible":
-            visible.append(p)
-    meta["visible_count"] = len(visible)
-
-    if len(visible) == 1:
-        meta["resolve"] = "visibility"
-        return visible[0], meta
-    if len(visible) > 1:
-        # Multiple windows each with a visible tab — prefer preferred if among them
-        if preferred is not None and preferred in visible:
-            meta["resolve"] = "visibility_preferred"
-            return preferred, meta
-        meta["resolve"] = "visibility_first"
-        return visible[0], meta
-
-    if preferred is not None and _page_is_open(preferred) and preferred in pages:
-        meta["resolve"] = "preferred_fallback"
-        return preferred, meta
-
-    meta["resolve"] = "first_open"
-    return pages[0], meta
-
-
-def _sync_active_page(*, update_global: bool = True) -> tuple[Any, dict[str, Any]]:
-    """Resolve visible page; optionally assign to module ``_page``."""
-    global _page
-    if _context is None:
-        raise RuntimeError("ERROR:NO_BROWSER_SESSION Playwright 会话未启动。")
-    page, meta = resolve_active_page(_context, preferred=_page)
-    if update_global:
-        _page = page
-    try:
-        meta["url"] = page.url or ""
-        meta["title"] = page.title() or ""
-    except Exception:
-        meta["url"] = ""
-        meta["title"] = ""
-    meta["page_id"] = _page_id_for(page)
-    return page, meta
-
-
-def _resolve_capture_page(page_id: str = "") -> tuple[Any, dict[str, Any]]:
-    global _page
-    requested = str(page_id or "")
-    if requested:
-        page = _page_for_id(requested)
-        if page is not None:
-            _page = page
-            meta = {
-                "resolve": "event_page_id",
-                "page_id": requested,
-                "page_count": len(_open_pages(_context)),
-                "visible_count": None,
-            }
-            try:
-                meta["url"] = page.url or ""
-                meta["title"] = page.title() or ""
-            except Exception:
-                meta["url"] = ""
-                meta["title"] = ""
-            return page, meta
-        raise RuntimeError(f"ERROR:EVENT_PAGE_CLOSED page_id={requested}")
-    return _sync_active_page(update_global=True)
-
-
-def _ensure_session_worker(
-    *,
-    start_url: str = _DEFAULT_START_URL,
-    force_goto: bool = False,
-) -> dict[str, Any]:
-    global _pw, _context, _page, _started
-    want = (start_url or "").strip() or _DEFAULT_START_URL
-
-    if _started and _context is not None:
-        try:
-            navigation = None
-            pages = _open_pages(_context)
-            if not pages:
-                _teardown_unlocked()
-            else:
-                # Start-flow navigation: prefer first open tab (stable home),
-                # not whatever detail tab happens to be visible.
-                if force_goto and want and want != "about:blank":
-                    home = pages[0]
-                    cur = ""
-                    try:
-                        cur = home.url or ""
-                    except Exception:
-                        cur = ""
-                    if cur.rstrip("/") != want.rstrip("/"):
-                        navigation = _goto_safe(home, want)
-                    _page = home
-                else:
-                    _page, _meta = _sync_active_page(update_global=True)
-                    if want and want != "about:blank" and (
-                        (_page.url or "") in {"", "about:blank"}
-                        or str(_page.url or "").startswith("about:")
-                    ):
-                        navigation = _goto_safe(_page, want)
-                        _page, _meta = _sync_active_page(update_global=True)
-                try:
-                    live_url = _page.url or ""
-                except Exception:
-                    live_url = ""
-                return {
-                    "ok": True,
-                    "reused": True,
-                    "url": live_url,
-                    "profile_dir": str(profile_dir()),
-                    "page_count": len(_open_pages(_context)),
-                    "navigation": navigation,
-                }
-        except Exception:
-            _teardown_unlocked()
-
-    if not kit_page_dump_available():
-        raise RuntimeError(
-            "ERROR:KIT_PAGE_DUMP_MISSING 需要 jiangchang-platform-kit>=1.2.11 "
-            "(jiangchang_skill_core.rpa.page_dump)。请升级 kit 后重试。"
-        )
-
-    try:
-        from playwright.sync_api import sync_playwright
-    except ImportError as exc:
-        raise RuntimeError(
-            "ERROR:MISSING_PLAYWRIGHT 未安装 playwright。"
-            "请在与技能相同的 Python 环境中安装后重试。"
-        ) from exc
-
-    from jiangchang_skill_core.runtime_env import find_chrome_executable
-    from jiangchang_skill_core.rpa.stealth import (
-        STEALTH_INIT_SCRIPT,
-        persistent_context_launch_parts,
-        stealth_enabled,
-    )
-
-    chrome = find_chrome_executable()
-    if not chrome:
-        raise RuntimeError(
-            "ERROR:MISSING_BROWSER 未检测到 Chrome 或 Edge，请安装后重试。"
-        )
-
-    prof = profile_dir()
-    prof.mkdir(parents=True, exist_ok=True)
-    args, ignore = persistent_context_launch_parts(extra_args=["--start-maximized"])
-    launch_kwargs: dict[str, Any] = dict(
-        user_data_dir=str(prof),
-        headless=False,
-        executable_path=chrome,
-        locale="zh-CN",
-        no_viewport=True,
-        args=args,
-    )
-    if ignore is not None:
-        launch_kwargs["ignore_default_args"] = ignore
-
-    _pw = sync_playwright().start()
-    try:
-        _context = _pw.chromium.launch_persistent_context(**launch_kwargs)
-        if stealth_enabled():
-            _context.add_init_script(STEALTH_INIT_SCRIPT)
-        if _context.pages:
-            _page = _context.pages[0]
-        else:
-            _page = _context.new_page()
-        navigation = None
-        if want and want != "about:blank":
-            navigation = _goto_safe(_page, want)
-        elif want and (
-            _page.url in {"", "about:blank"} or str(_page.url).startswith("about:")
-        ):
-            navigation = _goto_safe(_page, want)
-        _started = True
-        return {
-            "ok": True,
-            "reused": False,
-            "url": _page.url,
-            "profile_dir": str(prof),
-            "page_count": len(_open_pages(_context)),
-            "navigation": navigation,
-        }
-    except Exception:
-        _teardown_unlocked()
-        raise
-
-
-def _dump_worker(*, max_nodes: int, locatable_limit: int, page_id: str = "") -> dict[str, Any]:
-    from jiangchang_skill_core.rpa.page_dump import dump_page_sync
-
-    _ensure_session_worker()
-    page, meta = _resolve_capture_page(page_id)
-    result = dump_page_sync(page, max_nodes=max_nodes, locatable_limit=locatable_limit)
-    stats = dict(result.get("stats") or {})
-    stats["active_page"] = meta
-    result["stats"] = stats
-    return result
-
-
-def _screenshot_worker(*, page_id: str = "") -> bytes:
-    _ensure_session_worker()
-    page, _meta = _resolve_capture_page(page_id)
-    return page.screenshot(type="png", full_page=False)
-
-
-def _session_info_worker() -> dict[str, Any]:
-    if not _started or _context is None:
-        return {"active": False}
-    try:
-        page, meta = _sync_active_page(update_global=True)
-        return {
-            "active": True,
-            "url": page.url,
-            "title": page.title(),
-            "profile_dir": str(profile_dir()),
-            "page_count": meta.get("page_count"),
-            "resolve": meta.get("resolve"),
-            "page_id": meta.get("page_id"),
-        }
-    except Exception as exc:
-        return {"active": False, "error": str(exc)}
-
-
-def _dispatch_bridge_event(source: Any, payload: Any) -> None:
-    if not isinstance(payload, dict):
-        return
-    item = dict(payload)
-    try:
-        page = source.get("page") if isinstance(source, dict) else None
-        frame = source.get("frame") if isinstance(source, dict) else None
-        page_id = _page_id_for(page)
-        item["page_id"] = page_id
-        item["frame_id"] = _frame_id_for(frame, page_id=page_id)
-    except Exception:
-        pass
-    with _bridge_lock:
-        sink = _bridge_sink
-    if sink is not None:
-        try:
-            sink(item)
-        except Exception:
-            pass
-
-
-def _install_event_bridge_worker(
-    *, init_script: str, sink: Callable[[dict[str, Any]], None]
-) -> dict[str, Any]:
-    global _bridge_installed, _bridge_sink
-    _ensure_session_worker()
-    assert _context is not None
-    with _bridge_lock:
-        _bridge_sink = sink
-    if not _bridge_installed:
-        _context.expose_binding(_BRIDGE_NAME, _dispatch_bridge_event)
-        _bridge_installed = True
-    _context.add_init_script(init_script)
-    installed = 0
-    for page in _open_pages(_context):
-        try:
-            page.evaluate(init_script)
-            installed += 1
-        except Exception:
-            pass
-    return {"ok": True, "binding": _BRIDGE_NAME, "installed_pages": installed}
-
-
-def install_event_bridge(
-    init_script: str, sink: Callable[[dict[str, Any]], None]
-) -> dict[str, Any]:
-    """Install a context-level recorder bridge before user actions begin."""
-    return _call_on_worker(
-        _install_event_bridge_worker,
-        init_script=init_script,
-        sink=sink,
-        timeout_sec=15.0,
-    )
-
-
-def clear_event_bridge() -> None:
-    global _bridge_sink
-    with _bridge_lock:
-        _bridge_sink = None
-
-
-def ensure_session(
-    *,
-    start_url: str = _DEFAULT_START_URL,
-    force_goto: bool = False,
-) -> dict[str, Any]:
-    """Start or reuse headed persistent context; return status dict.
-
-    force_goto=True: navigate to start_url even when session is reused
-    (used when「开始流程」带起始 URL).
-    """
-    return _call_on_worker(
-        _ensure_session_worker,
-        start_url=start_url,
-        force_goto=force_goto,
-        timeout_sec=75.0,
-    )
-
-
-def session_info() -> dict[str, Any]:
-    return _call_on_worker(_session_info_worker, timeout_sec=10.0)
-
-
-def dump_current_page(
-    *, max_nodes: int = 8000, locatable_limit: int = 300, page_id: str = ""
-) -> dict[str, Any]:
-    """Dump an event-origin page, or resolve the active page as fallback."""
-    return _call_on_worker(
-        _dump_worker,
-        max_nodes=max_nodes,
-        locatable_limit=locatable_limit,
-        page_id=page_id,
-        timeout_sec=30.0,
-    )
-
-
-def screenshot_png(*, page_id: str = "") -> bytes:
-    return _call_on_worker(_screenshot_worker, page_id=page_id, timeout_sec=20.0)
-
-
-def page_id_for(page: Any) -> str:
-    """Return the stable id for a Page while already on the browser worker."""
-    return _page_id_for(page)
-
-
-def _validate_element_worker(
-    *, page_id: str, element: dict[str, Any], frame: Optional[dict[str, Any]] = None
-) -> dict[str, int]:
-    page, _meta = _resolve_capture_page(page_id)
-    scope: Any = page
-    frame_meta = dict(frame or {})
-    frame_url = str(frame_meta.get("url") or "")
-    frame_name = str(frame_meta.get("name") or "")
-    if frame_url or frame_name:
-        try:
-            for candidate in page.frames:
-                if frame_name and candidate.name == frame_name:
-                    scope = candidate
-                    break
-                if frame_url and candidate.url == frame_url:
-                    scope = candidate
-                    break
-        except Exception:
-            pass
-
-    el = dict(element or {})
-    role = str(el.get("role") or "").strip()
-    name = str(el.get("name") or "").strip()
-    test_id = str(el.get("test_id") or "").strip()
-    label = str(el.get("label") or "").strip()
-    placeholder = str(el.get("placeholder") or "").strip()
-    name_attr = str(el.get("name_attr") or "").strip()
-    href = str(el.get("href") or "").strip()
-    text_value = str(el.get("text") or "").strip()
-    css_id = str(el.get("id") or "").strip()
-    css_path = str(el.get("css") or el.get("css_path") or "").strip()
-    stable_css = str(el.get("stable_css") or "").strip()
-    semantic_role = str(el.get("semantic_role") or "").strip()
-    semantic_name = str(el.get("semantic_name") or "").strip()
-    counts: dict[str, int] = {}
-
-    def count(strategy: str, build: Callable[[], Any]) -> None:
-        try:
-            counts[strategy] = int(build().count())
-        except Exception:
-            pass
-
-    def count_actionable(strategy: str, build: Callable[[], Any]) -> None:
-        try:
-            locator = build()
-            total = min(int(locator.count()), 50)
-            actionable = 0
-            for index in range(total):
-                item = locator.nth(index)
-                if item.is_visible() and item.is_enabled():
-                    actionable += 1
-            counts[strategy] = actionable
-        except Exception:
-            pass
-
-    if role and name:
-        # Playwright accessible names normalize whitespace (SVG/icon buttons are
-        # a common example), while DOM innerText may not.  Non-exact still
-        # yields a trustworthy uniqueness count and mirrors getByRole defaults.
-        count("role", lambda: scope.get_by_role(role, name=name, exact=False))
-    elif role and bool(el.get("contenteditable")):
-        count("role", lambda: scope.get_by_role(role))
-    if semantic_role and semantic_name:
-        count_actionable(
-            "semantic_role",
-            lambda: scope.get_by_role(
-                semantic_role, name=semantic_name, exact=False
-            ),
-        )
-    if test_id:
-        count("test_id", lambda: scope.get_by_test_id(test_id))
-    if label:
-        count("label", lambda: scope.get_by_label(label, exact=True))
-    if placeholder:
-        count("placeholder", lambda: scope.get_by_placeholder(placeholder, exact=True))
-    if name_attr:
-        escaped_name = name_attr.replace("\\", "\\\\").replace('"', '\\"')
-        count("name_attr", lambda: scope.locator(f'[name="{escaped_name}"]'))
-    if href:
-        escaped_href = href.replace("\\", "\\\\").replace('"', '\\"')
-        count("href", lambda: scope.locator(f'a[href="{escaped_href}"]'))
-    if text_value and len(text_value) <= 80:
-        count("text", lambda: scope.get_by_text(text_value, exact=True))
-    if css_id and not css_id[0].isdigit():
-        escaped = css_id.replace("\\", "\\\\").replace('"', '\\"')
-        count("css_id", lambda: scope.locator(f'[id="{escaped}"]'))
-    if css_path:
-        count("css", lambda: scope.locator(css_path))
-    if stable_css:
-        count("stable_css", lambda: scope.locator(stable_css))
-    return counts
-
-
-def validate_element_on_page(
-    *, page_id: str, element: dict[str, Any], frame: Optional[dict[str, Any]] = None
-) -> dict[str, int]:
-    """Count locator candidates against the exact live event-origin page."""
-    if not page_id:
-        return {}
-    return _call_on_worker(
-        _validate_element_worker,
-        page_id=page_id,
-        element=element,
-        frame=frame,
-        timeout_sec=10.0,
-    )
-
-
-def call_on_page(fn: Callable[[Any], Any]) -> Any:
-    """Run ``fn(page)`` on the **visible** Playwright page (worker thread)."""
-
-    def _run() -> Any:
-        _ensure_session_worker()
-        page, _meta = _sync_active_page(update_global=True)
-        return fn(page)
-
-    return _call_on_worker(_run, timeout_sec=30.0)
-
-
-def call_on_each_open_page(fn: Callable[[Any], Any]) -> list[Any]:
-    """Run ``fn(page)`` for every open tab; returns list of per-page results."""
-
-    def _run() -> list[Any]:
-        _ensure_session_worker()
-        assert _context is not None
-        pages = _open_pages(_context)
-        # Keep _page synced to visible for dump consistency
-        try:
-            _sync_active_page(update_global=True)
-        except Exception:
-            pass
-        results: list[Any] = []
-        for p in pages:
-            try:
-                results.append(fn(p))
-            except Exception as exc:
-                results.append({"error": str(exc)})
-        return results
-
-    return _call_on_worker(_run, timeout_sec=20.0)
-
-
-def shutdown() -> None:
-    def _stop() -> None:
-        _teardown_unlocked()
-
-    try:
-        _call_on_worker(_stop, timeout_sec=5.0)
-    except Exception:
-        pass
-    try:
-        _cmd_q.put(None)
-    except Exception:
-        pass

tools/ui_inspector/kit_wake.py

@@ -1,284 +0,0 @@
-"""调用 jiangchang_desktop_wake（只调不复制注入逻辑）。
-
-- 主窗 WeWorkWindow → ``ensure_ui_ready("wecom")``
-- 搜索浮层 SearchResultWindow2 → ``ensure_ui_ready("wecom_search")``
-未安装 kit 时降级为跳过，不阻断采集。
-
-Warm cache key = profile + hwnd（经 WeCom Adapter），避免主窗 warm
-状态错误代表 SearchResultWindow2 已暖。
-"""
-
-from __future__ import annotations
-
-import time
-from typing import Any, Optional
-
-# A successful provider probe remains valid for the lifetime of a normal
-# recording flow.  The old 45 second TTL made long desktop flows re-run an
-# expensive WeCom wake in the middle of recording and produced minute-late
-# evidence.
-_CACHE_TTL_SEC = 600.0
-
-# cache_key -> cache (profile:hwnd:N)
-_caches: dict[str, dict[str, Any]] = {}
-
-_PROFILE_WECOM = "wecom"
-_PROFILE_WECOM_SEARCH = "wecom_search"
-
-_THRESHOLDS = {
-    _PROFILE_WECOM: 80,
-    _PROFILE_WECOM_SEARCH: 20,
-}
-
-
-def _wecom_adapter():
-    try:
-        from adapters.wecom import wecom_adapter
-
-        return wecom_adapter
-    except Exception:
-        try:
-            from .adapters.wecom import wecom_adapter  # type: ignore
-
-            return wecom_adapter
-        except Exception:
-            return None
-
-
-def resolve_wake_profile(*, class_name: str = "", title: str = "") -> Optional[str]:
-    """返回应调用的 kit profile id；非企微相关则 None。"""
-    ad = _wecom_adapter()
-    if ad is not None:
-        return ad.wake_profile(class_name=class_name, title=title)
-    # Compatibility fallback if adapter import fails
-    cls = (class_name or "").strip()
-    title = title or ""
-    if cls == "SearchResultWindow2":
-        return _PROFILE_WECOM_SEARCH
-    if cls == "WeWorkWindow":
-        return _PROFILE_WECOM
-    if "WeWork" in cls and ("企业微信" in title or "WeCom" in title or "wecom" in title.lower()):
-        return _PROFILE_WECOM
-    return None
-
-
-def looks_like_wecom(*, class_name: str = "", title: str = "") -> bool:
-    return resolve_wake_profile(class_name=class_name, title=title) is not None
-
-
-def kit_available() -> bool:
-    try:
-        from jiangchang_desktop_wake import ensure_ui_ready  # noqa: F401
-
-        return True
-    except ImportError:
-        return False
-
-
-def _cache_key(profile: str, hwnd: Optional[int]) -> str:
-    ad = _wecom_adapter()
-    if ad is not None:
-        return ad.cache_key(profile=profile, hwnd=int(hwnd or 0))
-    return f"{profile}:hwnd:{int(hwnd or 0)}"
-
-
-def auto_ensure_if_wecom(
-    *,
-    class_name: str = "",
-    title: str = "",
-    hwnd: Optional[int] = None,
-    force: bool = False,
-    findall_hint: Optional[int] = None,
-) -> dict[str, Any]:
-    """若目标像企微主窗/搜索浮层则静默 ensure；否则跳过。"""
-    profile = resolve_wake_profile(class_name=class_name, title=title)
-    if profile is None:
-        return {
-            "applicable": False,
-            "skipped": True,
-            "reason": "not_wecom",
-            "available": kit_available(),
-        }
-
-    ad = _wecom_adapter()
-    warm_min = int(ad.warm_threshold(profile) if ad else _THRESHOLDS.get(profile, 80))
-    key = _cache_key(profile, hwnd)
-    cache = _caches.setdefault(
-        key, {"t": 0.0, "warm": False, "hwnd": int(hwnd or 0), "findall": 0, "profile": profile}
-    )
-    now = time.monotonic()
-    hint = int(findall_hint) if findall_hint is not None else -1
-    # Require same hwnd in cache — main window warm must not cover search popup.
-    cached_hwnd = int(cache.get("hwnd") or 0)
-    same_window = (not hwnd) or (cached_hwnd == int(hwnd))
-    if (
-        not force
-        and same_window
-        and cache.get("warm")
-        and (now - float(cache.get("t") or 0.0)) < _CACHE_TTL_SEC
-    ):
-        return {
-            "applicable": True,
-            "available": True,
-            "skipped": True,
-            "reason": "cache_warm",
-            "ok": True,
-            "warm": True,
-            "wake_applied": False,
-            "findall": int(cache.get("findall") or 0),
-            "hwnd": int(cache.get("hwnd") or 0),
-            "profile": profile,
-            "cache_key": key,
-            "observed_findall_hint": hint,
-        }
-
-    # A direct probe is authoritative. Do not run an invasive wake merely
-    # because this process did not create the cache entry itself.
-    if not force and hint >= warm_min:
-        cache.update(
-            {"t": now, "warm": True, "hwnd": int(hwnd or 0), "findall": hint}
-        )
-        return {
-            "applicable": True,
-            "available": kit_available(),
-            "skipped": True,
-            "reason": "probe_warm",
-            "ok": True,
-            "warm": True,
-            "wake_applied": False,
-            "findall": hint,
-            "hwnd": int(hwnd or 0),
-            "profile": profile,
-            "cache_key": key,
-        }
-
-    try:
-        from jiangchang_desktop_wake import ensure_ui_ready
-        from jiangchang_desktop_wake.errors import DesktopWakeError
-    except ImportError:
-        return {
-            "applicable": True,
-            "available": False,
-            "skipped": True,
-            "reason": "kit_not_installed",
-            "ok": False,
-            "profile": profile,
-            "hint": "请安装 jiangchang-platform-kit>=1.2.6（含 wecom_search）后再采企微",
-        }
-
-    t0 = time.perf_counter()
-    try:
-        result = ensure_ui_ready(
-            profile,
-            hwnd=int(hwnd) if hwnd else None,
-            raise_on_cold=True,
-        )
-    except DesktopWakeError as exc:
-        raw = (getattr(exc, "code", None) or "").strip()
-        code = raw[6:] if raw.startswith("ERROR:") else (raw or "WAKE_FAILED")
-        return {
-            "applicable": True,
-            "available": True,
-            "skipped": False,
-            "ok": False,
-            "warm": False,
-            "wake_applied": False,
-            "error_code": code,
-            "message": str(getattr(exc, "message", None) or exc),
-            "elapsed_ms": int(round((time.perf_counter() - t0) * 1000)),
-            "profile": profile,
-        }
-    except Exception as exc:
-        return {
-            "applicable": True,
-            "available": True,
-            "skipped": False,
-            "ok": False,
-            "warm": False,
-            "error_code": "ENSURE_EXCEPTION",
-            "message": f"{type(exc).__name__}: {exc}",
-            "elapsed_ms": int(round((time.perf_counter() - t0) * 1000)),
-            "profile": profile,
-        }
-
-    after = result.probe_after
-    findall = int(after.findall_count) if after is not None else 0
-    report = {
-        "applicable": True,
-        "available": True,
-        "skipped": False,
-        "ok": bool(result.warm),
-        "warm": bool(result.warm),
-        "wake_applied": bool(result.wake_applied),
-        "findall": findall,
-        "hwnd": int(result.hwnd) if result.hwnd else 0,
-        "elapsed_ms": int(round((time.perf_counter() - t0) * 1000)),
-        "profile": profile,
-        "markers_found": list(after.markers_found) if after is not None else [],
-    }
-    report["cache_key"] = key
-    if result.warm:
-        # Persist under result hwnd so main≠search caches stay separate.
-        final_key = _cache_key(profile, report["hwnd"] or hwnd)
-        final_cache = _caches.setdefault(
-            final_key,
-            {"t": 0.0, "warm": False, "hwnd": int(report["hwnd"] or 0), "findall": 0, "profile": profile},
-        )
-        final_cache["t"] = now
-        final_cache["warm"] = True
-        final_cache["hwnd"] = int(report["hwnd"] or hwnd or 0)
-        final_cache["findall"] = findall
-        report["cache_key"] = final_key
-    else:
-        cache["warm"] = False
-    return report
-
-
-def note_provider_warm(
-    *, class_name: str = "", title: str = "", hwnd: int = 0, findall: int = 0
-) -> bool:
-    """Pin a proven-warm provider so recorder evidence never re-wakes it."""
-    profile = resolve_wake_profile(class_name=class_name, title=title)
-    adapter = _wecom_adapter()
-    threshold = int(
-        adapter.warm_threshold(profile)
-        if adapter and profile
-        else _THRESHOLDS.get(profile or "", 80)
-    )
-    count = int(findall or 0)
-    if not profile or count < threshold:
-        return False
-    key = _cache_key(profile, hwnd)
-    _caches[key] = {
-        "t": time.monotonic(),
-        "warm": True,
-        "hwnd": int(hwnd or 0),
-        "findall": count,
-        "profile": profile,
-    }
-    return True
-
-
-def format_kit_wake_line(report: Optional[dict[str, Any]]) -> str:
-    if not report:
-        return "kit_wake: (none)"
-    if not report.get("applicable"):
-        return "kit_wake: n/a (非企微)"
-    if report.get("reason") == "kit_not_installed":
-        return "kit_wake: 未安装 platform-kit（企微可能冷树）"
-    prof = report.get("profile") or "?"
-    if report.get("skipped") and report.get("reason") == "cache_warm":
-        return (
-            f"kit_wake[{prof}]: cache warm=True findall={report.get('findall')} "
-            f"hwnd={report.get('hwnd')}"
-        )
-    if report.get("ok"):
-        return (
-            f"kit_wake[{prof}]: warm={report.get('warm')} "
-            f"wake_applied={report.get('wake_applied')} "
-            f"findall={report.get('findall')} elapsed_ms={report.get('elapsed_ms')}"
-        )
-    return (
-        f"kit_wake[{prof}]: FAILED code={report.get('error_code')} "
-        f"msg={report.get('message')}"
-    )

tools/ui_inspector/locators/__init__.py

@@ -1,13 +0,0 @@
-"""Locator compilers + validators + element catalog."""
-
-from .browser import compile_browser_locators
-from .catalog import ElementCatalogService
-from .validator import validate_locator_against_rows
-from .windows import compile_windows_locators
-
-__all__ = [
-    "compile_browser_locators",
-    "compile_windows_locators",
-    "validate_locator_against_rows",
-    "ElementCatalogService",
-]

tools/ui_inspector/locators/base.py

@@ -1,74 +0,0 @@
-"""Shared locator compiler helpers."""
-
-from __future__ import annotations
-
-from typing import Any, Optional
-
-from models.locator import LocatorCandidate
-
-
-def rank_and_select_primary(
-    candidates: list[LocatorCandidate],
-) -> tuple[Optional[LocatorCandidate], list[LocatorCandidate]]:
-    """
-    Pick production primary only if unique and not coordinate-based.
-    Fallback list excludes primary.
-    """
-    ranked = sorted(
-        candidates,
-        key=lambda c: (
-            0 if c.production_eligible else 1,
-            0 if c.unique else 1,
-            0 if c.uses_coordinates else 0,
-            1 if c.uses_coordinates else 0,
-            1 if c.uses_index else 0,
-            -float(c.stability_score or 0.0),
-        ),
-    )
-    primary: Optional[LocatorCandidate] = None
-    for c in ranked:
-        if c.production_eligible and not c.uses_coordinates:
-            primary = c
-            break
-    fallbacks = [c for c in ranked if c is not primary]
-    return primary, fallbacks
-
-
-def count_matches(
-    rows: list[dict[str, Any]],
-    *,
-    properties: dict[str, Any],
-    scope: Optional[dict[str, Any]] = None,
-) -> int:
-    """Count rows matching structured properties (snapshot / live tree)."""
-    props = {k: v for k, v in (properties or {}).items() if v not in (None, "")}
-    if not props:
-        return 0
-    scope = scope or {}
-    n = 0
-    for r in rows or []:
-        if not isinstance(r, dict):
-            continue
-        if scope.get("window_class"):
-            # optional soft filter if row carries window info
-            rc = r.get("window_class") or r.get("window_class_name")
-            if rc and rc != scope.get("window_class"):
-                continue
-        ok = True
-        for k, v in props.items():
-            if k == "control_type":
-                rv = r.get("control_type") or r.get("role") or ""
-            elif k == "role":
-                rv = r.get("role") or r.get("control_type") or ""
-            elif k == "text":
-                rv = r.get("text") or r.get("value") or r.get("name") or ""
-            elif k == "test_id":
-                rv = r.get("test_id") or r.get("automation_id") or ""
-            else:
-                rv = r.get(k)
-            if str(rv or "") != str(v):
-                ok = False
-                break
-        if ok:
-            n += 1
-    return n

tools/ui_inspector/locators/browser.py

@@ -1,190 +0,0 @@
-"""Browser locator compiler (role / test id / label / placeholder / text / css id)."""
-
-from __future__ import annotations
-
-from typing import Any, Optional
-
-from models.locator import LocatorCandidate
-
-from .base import count_matches, rank_and_select_primary
-
-
-def _expr(strategy: str, props: dict[str, Any]) -> str:
-    parts = [f"{k}={v!r}" for k, v in props.items() if v not in (None, "")]
-    return f"{strategy}:" + ",".join(parts)
-
-
-def compile_browser_locators(
-    element: dict[str, Any],
-    *,
-    rows: Optional[list[dict[str, Any]]] = None,
-    frame: Optional[dict[str, Any]] = None,
-) -> list[LocatorCandidate]:
-    """
-    Generate ordered Playwright-oriented locator candidates.
-
-    Priority roughly: role+name → test id → label → placeholder → text → css id → scoped.
-    Coordinates are never emitted as production candidates.
-    """
-    el = element or {}
-    rows = list(rows or [])
-    scope = {"frame_url": (frame or {}).get("url") or "", "frame_name": (frame or {}).get("name") or ""}
-    cands: list[LocatorCandidate] = []
-
-    role = str(el.get("role") or el.get("control_type") or "").strip()
-    name = str(el.get("name") or "").strip()
-    text = str(el.get("text") or el.get("value") or "").strip()
-    is_editable = bool(el.get("contenteditable"))
-    if is_editable:
-        # Draft.js/contenteditable often exposes current user text as its
-        # accessible name.  That value is payload, never element identity.
-        current_value = str(el.get("value") or "").strip()
-        if name == current_value or name == text:
-            name = ""
-        text = ""
-    test_id = str(el.get("test_id") or el.get("data_testid") or el.get("automation_id") or "").strip()
-    label = str(el.get("label") or "").strip()
-    placeholder = str(el.get("placeholder") or "").strip()
-    name_attr = str(el.get("name_attr") or "").strip()
-    href = str(el.get("href") or "").strip()
-    css_id = str(el.get("id") or el.get("css_id") or "").strip()
-    css_path = str(el.get("css") or el.get("css_path") or "").strip()
-    stable_css = str(el.get("stable_css") or "").strip()
-    semantic_role = str(el.get("semantic_role") or "").strip()
-    semantic_name = str(el.get("semantic_name") or "").strip()
-    # Prefer existing primary_locator from page_dump if present
-    existing = el.get("primary_locator") or {}
-    if isinstance(existing, dict) and existing.get("expr"):
-        props = dict(existing.get("properties") or {})
-        if not props and existing.get("strategy"):
-            props = {"expr": existing.get("expr")}
-        c = LocatorCandidate(
-            strategy=str(existing.get("strategy") or "primary"),
-            scope=scope,
-            properties=props or {"expression": existing.get("expr")},
-            expression=str(existing.get("expr") or ""),
-            stability_score=0.85,
-            reasons=["from_page_dump_primary_locator"],
-        )
-        if rows and props:
-            c.mark_counts(count_matches(rows, properties=props, scope=scope))
-            c.validation = "snapshot_only"
-        elif rows and existing.get("expr"):
-            # Cannot count without structured props — leave unique unknown
-            c.validation = "snapshot_only"
-        cands.append(c)
-
-    for pw in el.get("playwright_candidates") or []:
-        if not isinstance(pw, dict):
-            continue
-        props = dict(pw.get("properties") or {})
-        expr = str(pw.get("expr") or pw.get("expression") or "")
-        strat = str(pw.get("strategy") or "playwright")
-        c = LocatorCandidate(
-            strategy=strat,
-            scope=scope,
-            properties=props or {"expression": expr},
-            expression=expr,
-            stability_score=float(pw.get("score") or 0.7),
-            uses_index=bool(pw.get("uses_index")),
-            uses_coordinates=bool(pw.get("uses_coordinates")),
-            reasons=["from_playwright_candidates"],
-        )
-        if rows and props:
-            c.mark_counts(count_matches(rows, properties=props, scope=scope))
-            c.validation = "snapshot_only"
-        cands.append(c)
-
-    def add(strategy: str, props: dict[str, Any], score: float, reason: str) -> None:
-        if not props or not any(str(v).strip() for v in props.values()):
-            return
-        c = LocatorCandidate(
-            strategy=strategy,
-            scope=dict(scope),
-            properties=dict(props),
-            expression=_expr(strategy, props),
-            stability_score=score,
-            reasons=[reason],
-        )
-        if rows:
-            c.mark_counts(count_matches(rows, properties=props, scope=scope))
-            c.validation = "snapshot_only"
-        cands.append(c)
-
-    if role and name:
-        add("role", {"role": role, "name": name}, 0.95, "role_name")
-    elif role and is_editable:
-        add("role", {"role": role}, 0.78, "editable_role")
-    if semantic_role and semantic_name:
-        add(
-            "semantic_role",
-            {"role": semantic_role, "name": semantic_name},
-            0.86,
-            "semantic_target_inferred_from_form_scope",
-        )
-    if test_id:
-        add("test_id", {"test_id": test_id}, 0.93, "test_id")
-    if label:
-        add("label", {"label": label}, 0.88, "label")
-    if placeholder:
-        add("placeholder", {"placeholder": placeholder}, 0.84, "placeholder")
-    if name_attr:
-        add("name_attr", {"name_attr": name_attr}, 0.89, "html_name_attribute")
-    if href and role.lower() == "link":
-        add("href", {"href": href}, 0.91, "link_destination")
-    if text and len(text) <= 80:
-        add("text", {"text": text}, 0.75, "visible_text")
-    if css_id and not css_id[0].isdigit():
-        add("css_id", {"id": css_id}, 0.8, "css_id")
-    if stable_css:
-        c = LocatorCandidate(
-            strategy="stable_css",
-            scope=dict(scope),
-            properties={"stable_css": stable_css},
-            expression=f"css:{stable_css}",
-            stability_score=0.78 if ("#" in stable_css or "data-testid" in stable_css) else 0.62,
-            reasons=["stable_attribute_selector", "live_unique_required"],
-        )
-        c.production_eligible = False
-        cands.append(c)
-    if css_path:
-        # Full/scoped CSS is a last-resort executable locator.  It only becomes
-        # production eligible after live validation proves it is unique.
-        c = LocatorCandidate(
-            strategy="css",
-            scope=dict(scope),
-            properties={"css": css_path},
-            expression=f"css:{css_path}",
-            stability_score=0.48 if "#" in css_path else 0.32,
-            reasons=["event_dom_path", "live_unique_required"],
-        )
-        c.production_eligible = False
-        cands.append(c)
-
-    # Never add coordinate locator as candidate for production
-    if el.get("bounds") or el.get("rect"):
-        c = LocatorCandidate(
-            strategy="bounding_rect",
-            scope=dict(scope),
-            properties={"bounds": el.get("bounds") or el.get("rect")},
-            expression="coordinates:(evidence_only)",
-            stability_score=0.05,
-            uses_coordinates=True,
-            production_eligible=False,
-            reasons=["coordinates_evidence_only"],
-        )
-        c.unique = False
-        c.ambiguous = True
-        cands.append(c)
-
-    # Re-validate production_eligible after counts
-    for c in cands:
-        if c.actual_count is not None:
-            c.mark_counts(int(c.actual_count), expected=int(c.expected_count or 1))
-        elif c.uses_coordinates:
-            c.production_eligible = False
-
-    primary, _ = rank_and_select_primary(cands)
-    # Ensure ambiguous never becomes primary via side effect — rank handles it.
-    del primary
-    return cands

tools/ui_inspector/locators/catalog.py

@@ -1,294 +0,0 @@
-"""Element Catalog: merge observations, locator history, parameterization."""
-
-from __future__ import annotations
-
-import re
-from pathlib import Path
-from typing import Any, Optional
-
-from models.element import ElementCatalog, ElementRef, element_fingerprint
-from models.locator import LocatorCandidate
-from models.serialize import to_dict, write_model_json
-from redaction import redact_element_fields
-from core.policies.element_identity import element_identity_key
-
-from .base import rank_and_select_primary
-from .browser import compile_browser_locators
-from .validator import validate_locator_against_rows
-from .windows import compile_windows_locators
-
-_DYNAMIC_VALUE_HINTS = re.compile(
-    r"(联系人|搜索|search|contact|user|chat|title|name)",
-    re.I,
-)
-
-
-def suggest_logical_name(element: dict[str, Any]) -> str:
-    aid = str(element.get("automation_id") or "").strip()
-    name = str(element.get("name") or "").strip()
-    role = str(element.get("control_type") or element.get("role") or "el").strip()
-    text = str(element.get("text") or "").strip()
-    semantic_name = str(element.get("semantic_name") or "").strip()
-    if element.get("contenteditable"):
-        base = aid or str(element.get("placeholder") or element.get("label") or "") or role
-    else:
-        base = aid or semantic_name or name or (text if 0 < len(text) <= 48 else "")
-        if not base:
-            for ancestor in element.get("parent_chain") or []:
-                ancestor_id = str((ancestor or {}).get("id") or "").strip()
-                ancestor_text = str((ancestor or {}).get("text") or "").strip()
-                if ancestor_id:
-                    base = f"{ancestor_id}_{role}"
-                    break
-                if ancestor_text and len(ancestor_text) <= 32:
-                    base = f"{ancestor_text}_{role}"
-                    break
-        base = base or role
-    base = re.sub(r"[^\w\u4e00-\u9fff]+", "_", base, flags=re.UNICODE).strip("_")
-    base = base[:48] or "element"
-    return base.lower() if base.isascii() else base
-
-
-def _maybe_parameterize(value: str, *, field: str) -> tuple[str, dict[str, str]]:
-    """Replace likely dynamic business values with ${param} placeholders."""
-    v = (value or "").strip()
-    if not v or len(v) > 64:
-        return v, {}
-    if field in {"value", "text", "name"} and _DYNAMIC_VALUE_HINTS.search(field):
-        return "${value}", {"value": v}
-    # Contact-like short Chinese/ASCII names in value when name is generic
-    if field == "value" and 1 < len(v) <= 32 and not v.startswith("${"):
-        # Only parameterize when looks like a person/search term (letters/CJK, not chrome)
-        if re.fullmatch(r"[\w\u4e00-\u9fff \-_.@]+", v, flags=re.UNICODE):
-            chrome = {"关闭", "最小化", "最大化", "取消", "确定", "发送", "搜索"}
-            if v not in chrome:
-                return "${contact_name}", {"contact_name": v}
-    return v, {}
-
-
-class ElementCatalogService:
-    def __init__(self, flow_dir: Path, *, flow_id: str = "") -> None:
-        self.flow_dir = Path(flow_dir)
-        self.elements_dir = self.flow_dir / "elements"
-        self.elements_dir.mkdir(parents=True, exist_ok=True)
-        self.catalog_path = self.flow_dir / "element_catalog.json"
-        self.catalog = ElementCatalog(flow_id=flow_id)
-        self._by_fp: dict[str, ElementRef] = {}
-        self._by_identity: dict[str, ElementRef] = {}
-        self._by_name: dict[str, ElementRef] = {}
-
-    def load(self) -> None:
-        if not self.catalog_path.exists():
-            return
-        import json
-        from models.serialize import from_dict
-
-        data = json.loads(self.catalog_path.read_text(encoding="utf-8"))
-        self.catalog.flow_id = data.get("flow_id") or self.catalog.flow_id
-        for item in data.get("elements") or []:
-            # Reconstruct LocatorCandidate lists lightly
-            locs = []
-            for lc in item.get("locator_candidates") or []:
-                locs.append(from_dict(LocatorCandidate, lc))
-            primary = None
-            if item.get("primary_locator"):
-                primary = from_dict(LocatorCandidate, item["primary_locator"])
-            fallbacks = [
-                from_dict(LocatorCandidate, x) for x in (item.get("fallback_locators") or [])
-            ]
-            el = ElementRef(
-                id=item.get("id") or "",
-                logical_name=item.get("logical_name") or "",
-                surface=item.get("surface") or "windows",
-                backend=item.get("backend") or "",
-                app=item.get("app") or {},
-                window_scope=item.get("window_scope") or {},
-                element_scope=item.get("element_scope") or {},
-                role=item.get("role") or "",
-                control_type=item.get("control_type") or "",
-                name=item.get("name") or "",
-                value=item.get("value") or "",
-                automation_id=item.get("automation_id") or "",
-                class_name=item.get("class_name") or "",
-                framework_id=item.get("framework_id") or "",
-                states=item.get("states") or {},
-                bounds=item.get("bounds") or {},
-                actions=list(item.get("actions") or []),
-                parent_chain=list(item.get("parent_chain") or []),
-                raw=item.get("raw") or {},
-                fingerprint=item.get("fingerprint") or "",
-                sensitive=bool(item.get("sensitive")),
-                locator_candidates=locs,
-                primary_locator=primary,
-                fallback_locators=fallbacks,
-                expected_count=int(item.get("expected_count") or 1),
-                verified_count=item.get("verified_count"),
-                verification_level=item.get("verification_level") or "none",
-                stability_score=float(item.get("stability_score") or 0.0),
-                verified_at=item.get("verified_at"),
-                first_seen=item.get("first_seen") or "",
-                last_seen=item.get("last_seen") or "",
-                observation_count=int(item.get("observation_count") or 1),
-                locator_history=list(item.get("locator_history") or []),
-                parameterized=dict(item.get("parameterized") or {}),
-            )
-            if el.fingerprint:
-                self._by_fp[el.fingerprint] = el
-            identity = str((el.raw or {}).get("_identity_key") or "")
-            if not identity:
-                identity = element_identity_key(
-                    el.raw or to_dict(el),
-                    surface=el.surface,
-                    page_url=str((el.element_scope or {}).get("page_url") or ""),
-                    frame_name=str((el.element_scope or {}).get("frame_name") or ""),
-                )
-            self._by_identity[identity] = el
-            if el.logical_name:
-                self._by_name[el.logical_name] = el
-            self.catalog.elements.append(el)
-
-    def save(self) -> None:
-        from datetime import datetime
-
-        self.catalog.updated_at = datetime.now().astimezone().isoformat()
-        write_model_json(self.catalog_path, self.catalog)
-        for el in self.catalog.elements:
-            write_model_json(self.elements_dir / f"{el.id}.json", el)
-
-    def upsert_from_snapshot(
-        self,
-        element: dict[str, Any],
-        *,
-        surface: str,
-        backend: str = "",
-        window_scope: Optional[dict[str, Any]] = None,
-        rows: Optional[list[dict[str, Any]]] = None,
-        logical_name: Optional[str] = None,
-        frame: Optional[dict[str, Any]] = None,
-        live_counts: Optional[dict[str, int]] = None,
-    ) -> ElementRef:
-        cleaned, sensitive = redact_element_fields(dict(element or {}))
-        win = dict(window_scope or {})
-        value = str(cleaned.get("value") or "")
-        is_editable = bool(cleaned.get("contenteditable"))
-        param_value, params = _maybe_parameterize(value, field="value")
-        if params:
-            cleaned["value"] = param_value
-
-        fp = element_fingerprint(
-            surface=surface,
-            role=str(cleaned.get("control_type") or cleaned.get("role") or ""),
-            name=str(cleaned.get("name") or ""),
-            automation_id=str(cleaned.get("automation_id") or ""),
-            class_name=str(cleaned.get("class_name") or ""),
-            value=str(cleaned.get("value") or ""),
-            window_scope=win,
-            dom_fingerprint=str(cleaned.get("fingerprint") or cleaned.get("css") or ""),
-            page_scope=str((frame or {}).get("url") or win.get("url") or ""),
-            contenteditable=is_editable,
-        )
-
-        identity = element_identity_key(
-            cleaned,
-            surface=surface,
-            page_url=str((frame or {}).get("url") or win.get("url") or ""),
-            frame_name=str((frame or {}).get("name") or ""),
-        )
-        cleaned["_identity_key"] = identity
-
-        existing = self._by_identity.get(identity) or self._by_fp.get(fp)
-        if existing is None and logical_name and logical_name in self._by_name:
-            existing = self._by_name[logical_name]
-
-        if surface == "browser":
-            cands = compile_browser_locators(cleaned, rows=rows, frame=frame)
-        else:
-            cands = compile_windows_locators(cleaned, rows=rows, window_scope=win)
-
-        for c in cands:
-            if rows is not None and not c.uses_coordinates:
-                validate_locator_against_rows(c, rows, level="snapshot_only")
-            if live_counts is not None and c.strategy in live_counts:
-                c.mark_counts(int(live_counts[c.strategy]))
-                c.validation = "live"
-
-        primary, fallbacks = rank_and_select_primary(cands)
-        # Ambiguous / coordinates never primary
-        if primary is not None and (primary.ambiguous or not primary.production_eligible):
-            primary = None
-
-        lname = logical_name or (existing.logical_name if existing else "") or suggest_logical_name(
-            cleaned
-        )
-
-        if existing:
-            # Merge observation; keep locator history
-            if primary is not None:
-                old = existing.primary_locator
-                if old is not None and to_dict(old) != to_dict(primary):
-                    existing.locator_history.append(
-                        {"replaced_at": existing.last_seen, "locator": to_dict(old)}
-                    )
-                existing.primary_locator = primary
-            existing.locator_candidates = cands
-            existing.fallback_locators = fallbacks
-            existing.touch()
-            existing.raw = cleaned
-            existing.bounds = dict(cleaned.get("bounds") or cleaned.get("rect") or existing.bounds or {})
-            if params:
-                existing.parameterized.update(params)
-            existing.sensitive = existing.sensitive or sensitive
-            existing.verification_level = "live" if live_counts else "snapshot_only"
-            self._by_fp[fp] = existing
-            self._by_identity[identity] = existing
-            if primary and primary.actual_count is not None:
-                existing.verified_count = primary.actual_count
-            self.save()
-            return existing
-
-        el = ElementRef(
-            logical_name=lname,
-            surface=surface,
-            backend=backend,
-            window_scope=win,
-            element_scope={
-                "page_url": str((frame or {}).get("url") or win.get("url") or ""),
-                "frame_name": str((frame or {}).get("name") or ""),
-            },
-            role=str(cleaned.get("role") or ""),
-            control_type=str(cleaned.get("control_type") or ""),
-            name=str(cleaned.get("name") or ""),
-            value=str(cleaned.get("value") or ""),
-            automation_id=str(cleaned.get("automation_id") or ""),
-            class_name=str(cleaned.get("class_name") or ""),
-            framework_id=str(cleaned.get("framework_id") or ""),
-            states={
-                "is_enabled": cleaned.get("is_enabled"),
-                "is_offscreen": cleaned.get("is_offscreen"),
-            },
-            bounds=dict(cleaned.get("bounds") or cleaned.get("rect") or {}),
-            parent_chain=list(cleaned.get("parent_chain") or []),
-            raw=cleaned,
-            fingerprint=fp,
-            sensitive=sensitive,
-            locator_candidates=cands,
-            primary_locator=primary,
-            fallback_locators=fallbacks,
-            verification_level="live" if live_counts else "snapshot_only",
-            verified_count=(primary.actual_count if primary else None),
-            parameterized=params,
-            stability_score=float(primary.stability_score) if primary else 0.0,
-        )
-        # Ensure unique logical_name
-        base = el.logical_name
-        i = 2
-        while el.logical_name in self._by_name:
-            el.logical_name = f"{base}_{i}"
-            i += 1
-
-        self.catalog.elements.append(el)
-        self._by_fp[fp] = el
-        self._by_identity[identity] = el
-        self._by_name[el.logical_name] = el
-        self.save()
-        return el

tools/ui_inspector/locators/validator.py

@@ -1,50 +0,0 @@
-"""Locator uniqueness validation against snapshot / live tree rows."""
-
-from __future__ import annotations
-
-from typing import Any, Optional
-
-from models.locator import LocatorCandidate
-
-from .base import count_matches
-
-
-def validate_locator_against_rows(
-    candidate: LocatorCandidate,
-    rows: list[dict[str, Any]],
-    *,
-    level: str = "snapshot_only",
-    expected_count: int = 1,
-) -> LocatorCandidate:
-    """
-    Validate structured locator properties against a row list.
-
-    Levels supported in this phase: snapshot_only | live_tree
-    """
-    actual = count_matches(
-        rows,
-        properties=candidate.properties,
-        scope=candidate.scope,
-    )
-    candidate.mark_counts(actual, expected=expected_count)
-    candidate.validation = level
-    if not candidate.unique:
-        candidate.reasons = list(candidate.reasons or [])
-        if "ambiguous_match_count" not in candidate.reasons:
-            candidate.reasons.append("ambiguous_match_count")
-        candidate.production_eligible = False
-    if candidate.uses_coordinates:
-        candidate.production_eligible = False
-        if "coordinates_never_primary" not in (candidate.reasons or []):
-            candidate.reasons = list(candidate.reasons or []) + ["coordinates_never_primary"]
-    return candidate
-
-
-def choose_production_primary(
-    candidates: list[LocatorCandidate],
-) -> Optional[LocatorCandidate]:
-    """Return first production-eligible unique non-coordinate locator, else None."""
-    for c in candidates:
-        if c.production_eligible and c.unique and not c.uses_coordinates:
-            return c
-    return None

tools/ui_inspector/locators/windows.py

@@ -1,180 +0,0 @@
-"""Windows UIA locator compiler (Name ≠ Value; no coordinate primary)."""
-
-from __future__ import annotations
-
-from typing import Any, Optional
-
-from models.locator import LocatorCandidate
-
-from .base import count_matches, rank_and_select_primary
-
-
-def _expr(strategy: str, props: dict[str, Any]) -> str:
-    parts = [f"{k}={v!r}" for k, v in props.items() if v not in (None, "")]
-    return f"{strategy}:" + ",".join(parts)
-
-
-def compile_windows_locators(
-    element: dict[str, Any],
-    *,
-    rows: Optional[list[dict[str, Any]]] = None,
-    window_scope: Optional[dict[str, Any]] = None,
-) -> list[LocatorCandidate]:
-    """
-    Priority:
-    1. Window scope + AutomationId + ControlType
-    2. Window scope + stable Name + ControlType
-    3. Window scope + Name + Value + ControlType
-    4. Business parent + child attrs
-    5. Visible Value + ControlType
-    6. Relative ancestor scope
-    7. Index with business reason (low score)
-    8. BoundingRectangle evidence only
-    """
-    el = element or {}
-    rows = list(rows or [])
-    win = dict(window_scope or el.get("window_scope") or {})
-    scope = {
-        "window_class": win.get("class_name") or win.get("window_class") or "",
-        "window_title": win.get("title") or win.get("window_title") or "",
-        "process_name": win.get("process_name") or "",
-    }
-    cands: list[LocatorCandidate] = []
-
-    aid = str(el.get("automation_id") or "").strip()
-    name = str(el.get("name") or "").strip()
-    value = str(el.get("value") or "").strip()
-    ctype = str(el.get("control_type") or el.get("role") or "").strip()
-    class_name = str(el.get("class_name") or "").strip()
-    parent_chain = list(el.get("parent_chain") or [])
-
-    def add(
-        strategy: str,
-        props: dict[str, Any],
-        score: float,
-        reason: str,
-        *,
-        uses_index: bool = False,
-        uses_coordinates: bool = False,
-    ) -> None:
-        if not props or not any(str(v).strip() for v in props.values() if not isinstance(v, (dict, list))):
-            return
-        c = LocatorCandidate(
-            strategy=strategy,
-            scope=dict(scope),
-            properties=dict(props),
-            expression=_expr(strategy, props),
-            stability_score=score,
-            uses_index=uses_index,
-            uses_coordinates=uses_coordinates,
-            reasons=[reason],
-        )
-        if rows:
-            c.mark_counts(count_matches(rows, properties=props, scope=scope))
-            c.validation = "snapshot_only"
-        elif uses_coordinates:
-            c.production_eligible = False
-            c.unique = False
-            c.ambiguous = True
-        cands.append(c)
-
-    if aid and ctype:
-        add(
-            "window_automation_id_type",
-            {"automation_id": aid, "control_type": ctype},
-            0.96,
-            "automation_id_control_type",
-        )
-    elif aid:
-        add("window_automation_id", {"automation_id": aid}, 0.9, "automation_id")
-
-    if name and ctype and name.lower() not in {"item", "pane", "unknown"}:
-        add(
-            "window_name_type",
-            {"name": name, "control_type": ctype},
-            0.88,
-            "stable_name_control_type",
-        )
-
-    if name and value and ctype:
-        add(
-            "window_name_value_type",
-            {"name": name, "value": value, "control_type": ctype},
-            0.86,
-            "name_value_control_type",
-        )
-
-    # Business parent + child
-    if parent_chain:
-        parent = parent_chain[0] if isinstance(parent_chain[0], dict) else {}
-        parent_name = str(parent.get("name") or parent.get("automation_id") or "").strip()
-        if parent_name and (aid or name or value):
-            props: dict[str, Any] = {"parent_name": parent_name}
-            if aid:
-                props["automation_id"] = aid
-            if name:
-                props["name"] = name
-            if value:
-                props["value"] = value
-            if ctype:
-                props["control_type"] = ctype
-            add("parent_scoped", props, 0.82, "business_parent_scope")
-
-    if value and ctype and len(value) <= 64:
-        add(
-            "value_type",
-            {"value": value, "control_type": ctype},
-            0.78,
-            "visible_value_control_type",
-        )
-
-    if class_name and ctype:
-        add(
-            "class_type",
-            {"class_name": class_name, "control_type": ctype},
-            0.55,
-            "class_name_control_type",
-        )
-
-    # Index only with explicit business reason — low score, not auto primary
-    idx = el.get("index")
-    if idx is not None and (aid or name):
-        props = {"index": int(idx)}
-        if aid:
-            props["automation_id"] = aid
-        if name:
-            props["name"] = name
-        if ctype:
-            props["control_type"] = ctype
-        add(
-            "indexed",
-            props,
-            0.35,
-            "index_requires_business_justification",
-            uses_index=True,
-        )
-
-    bounds = el.get("bounds") or el.get("rect") or el.get("bounding_rectangle")
-    if bounds:
-        c = LocatorCandidate(
-            strategy="bounding_rectangle",
-            scope=dict(scope),
-            properties={"bounds": bounds},
-            expression="coordinates:(evidence_only)",
-            stability_score=0.05,
-            uses_coordinates=True,
-            production_eligible=False,
-            unique=False,
-            ambiguous=True,
-            reasons=["coordinates_never_primary"],
-        )
-        cands.append(c)
-
-    for c in cands:
-        if c.actual_count is not None:
-            c.mark_counts(int(c.actual_count), expected=int(c.expected_count or 1))
-        if c.uses_coordinates:
-            c.production_eligible = False
-
-    rank_and_select_primary(cands)
-    return cands

tools/ui_inspector/models/__init__.py

@@ -1,22 +0,0 @@
-"""UI Inspector v2 data models (Action / State / Element / Flow)."""
-
-from .action import ACTION_TYPES, ActionFlow, ActionRecord
-from .element import ElementRef
-from .locator import LocatorCandidate
-from .observation import UIObservation
-from .serialize import from_dict, to_dict, write_model_json
-from .state import UIState, compute_state_fingerprint
-
-__all__ = [
-    "ACTION_TYPES",
-    "ActionFlow",
-    "ActionRecord",
-    "ElementRef",
-    "LocatorCandidate",
-    "UIState",
-    "UIObservation",
-    "compute_state_fingerprint",
-    "from_dict",
-    "to_dict",
-    "write_model_json",
-]

tools/ui_inspector/models/action.py

@@ -1,119 +0,0 @@
-"""ActionRecord + ActionFlow models."""
-
-from __future__ import annotations
-
-import uuid
-from dataclasses import dataclass, field
-from datetime import datetime
-from typing import Any, Optional
-
-SCHEMA_ACTION = "openclaw.ui_inspector.action.v1"
-SCHEMA_ACTION_FLOW = "openclaw.ui_inspector.action_flow.v1"
-
-# Full vocabulary — capability flags live elsewhere for unimplemented types.
-ACTION_TYPES: tuple[str, ...] = (
-    "click",
-    "double_click",
-    "right_click",
-    "fill",
-    "type",
-    "clear",
-    "press_key",
-    "hotkey",
-    "select",
-    "toggle",
-    "hover",
-    "scroll",
-    "drag_drop",
-    "focus",
-    "window_switch",
-    "navigation",
-    "checkpoint",
-    "wait",
-    "verify",
-    "unknown",
-)
-
-# Capability status for this Inspector release.
-ACTION_CAPABILITY: dict[str, str] = {
-    "click": "implemented",
-    "double_click": "schema_only",
-    "right_click": "schema_only",
-    "fill": "implemented",
-    "type": "implemented",  # aggregated into fill/type
-    "clear": "schema_only",
-    "press_key": "implemented",
-    "hotkey": "schema_only",
-    "select": "schema_only",
-    "toggle": "schema_only",
-    "hover": "schema_only",
-    "scroll": "schema_only",
-    "drag_drop": "schema_only",
-    "focus": "schema_only",
-    "window_switch": "schema_only",
-    "navigation": "partial",
-    "checkpoint": "implemented",
-    "wait": "schema_only",
-    "verify": "schema_only",
-    "unknown": "fallback",
-}
-
-
-def _now_iso() -> str:
-    return datetime.now().astimezone().isoformat()
-
-
-def _new_id(prefix: str = "act") -> str:
-    return f"{prefix}_{uuid.uuid4().hex[:12]}"
-
-
-@dataclass
-class ActionRecord:
-    schema: str = SCHEMA_ACTION
-    id: str = field(default_factory=lambda: _new_id("act"))
-    sequence: int = 0
-    action_type: str = "unknown"
-    started_at: str = field(default_factory=_now_iso)
-    ended_at: Optional[str] = None
-    surface: str = "windows"
-    target_element_id: Optional[str] = None
-    target_snapshot: dict[str, Any] = field(default_factory=dict)
-    input: dict[str, Any] = field(default_factory=dict)
-    sensitive: bool = False
-    before_state_id: Optional[str] = None
-    after_state_id: Optional[str] = None
-    before_observation_id: Optional[str] = None
-    after_observation_id: Optional[str] = None
-    observed_changes: list[dict[str, Any]] = field(default_factory=list)
-    proposed_postconditions: list[dict[str, Any]] = field(default_factory=list)
-    status: str = "ok"  # ok | partial | failed | skipped
-    confidence: float = 0.0
-    evidence: dict[str, Any] = field(default_factory=dict)
-    raw_event: dict[str, Any] = field(default_factory=dict)
-    raw_event_ids: list[str] = field(default_factory=list)
-    capability: str = ""
-
-    def __post_init__(self) -> None:
-        if self.action_type not in ACTION_TYPES:
-            self.action_type = "unknown"
-        if not self.capability:
-            self.capability = ACTION_CAPABILITY.get(self.action_type, "schema_only")
-
-
-@dataclass
-class ActionFlow:
-    schema: str = SCHEMA_ACTION_FLOW
-    flow_id: str = ""
-    flow_name: str = ""
-    created_at: str = field(default_factory=_now_iso)
-    ended_at: Optional[str] = None
-    states: list[str] = field(default_factory=list)  # state ids in order seen
-    observations: list[str] = field(default_factory=list)  # every concrete capture
-    actions: list[str] = field(default_factory=list)  # action ids in sequence
-    elements: list[str] = field(default_factory=list)  # element ids
-    entry_state_id: Optional[str] = None
-    final_state_id: Optional[str] = None
-    metadata: dict[str, Any] = field(default_factory=dict)
-    recording_mode: str = "auto"  # auto | guided | checkpoint | legacy_interval
-    surface: str = ""
-    status: str = "open"

tools/ui_inspector/models/element.py

@@ -1,126 +0,0 @@
-"""ElementRef model for Element Catalog."""
-
-from __future__ import annotations
-
-import hashlib
-import json
-import uuid
-from dataclasses import dataclass, field
-from datetime import datetime
-from typing import Any, Optional
-
-from .locator import LocatorCandidate
-
-SCHEMA_ELEMENT = "openclaw.ui_inspector.element.v1"
-SCHEMA_CATALOG = "openclaw.ui_inspector.element_catalog.v1"
-
-
-def _now_iso() -> str:
-    return datetime.now().astimezone().isoformat()
-
-
-def _new_id(prefix: str = "el") -> str:
-    return f"{prefix}_{uuid.uuid4().hex[:12]}"
-
-
-def element_fingerprint(
-    *,
-    surface: str,
-    role: str = "",
-    name: str = "",
-    automation_id: str = "",
-    class_name: str = "",
-    value: str = "",
-    window_scope: Optional[dict[str, Any]] = None,
-    dom_fingerprint: str = "",
-    page_scope: str = "",
-    contenteditable: bool = False,
-) -> str:
-    payload = {
-        "surface": surface or "",
-        "role": role or "",
-        "name": "" if contenteditable else (name or ""),
-        "automation_id": automation_id or "",
-        "class_name": class_name or "",
-        # Value is identity only when short/stable-looking (not long free text).
-        "value": "" if contenteditable else ((value or "") if len(value or "") <= 64 else ""),
-        "dom_fingerprint": dom_fingerprint or "",
-        "page_scope": page_scope or "",
-        "window_scope": {
-            "class_name": (window_scope or {}).get("class_name") or "",
-            "title": (window_scope or {}).get("title") or "",
-        },
-    }
-    raw = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
-    return hashlib.sha256(raw.encode("utf-8")).hexdigest()
-
-
-@dataclass
-class ElementRef:
-    schema: str = SCHEMA_ELEMENT
-    id: str = field(default_factory=lambda: _new_id("el"))
-    logical_name: str = ""
-    surface: str = "windows"
-    backend: str = ""
-    app: dict[str, Any] = field(default_factory=dict)
-    window_scope: dict[str, Any] = field(default_factory=dict)
-    element_scope: dict[str, Any] = field(default_factory=dict)
-    role: str = ""
-    control_type: str = ""
-    name: str = ""
-    value: str = ""
-    automation_id: str = ""
-    class_name: str = ""
-    framework_id: str = ""
-    states: dict[str, Any] = field(default_factory=dict)
-    bounds: dict[str, Any] = field(default_factory=dict)
-    actions: list[str] = field(default_factory=list)
-    parent_chain: list[dict[str, Any]] = field(default_factory=list)
-    raw: dict[str, Any] = field(default_factory=dict)
-    fingerprint: str = ""
-    sensitive: bool = False
-    locator_candidates: list[LocatorCandidate] = field(default_factory=list)
-    primary_locator: Optional[LocatorCandidate] = None
-    fallback_locators: list[LocatorCandidate] = field(default_factory=list)
-    expected_count: int = 1
-    verified_count: Optional[int] = None
-    verification_level: str = "none"
-    stability_score: float = 0.0
-    verified_at: Optional[str] = None
-    first_seen: str = field(default_factory=_now_iso)
-    last_seen: str = field(default_factory=_now_iso)
-    observation_count: int = 1
-    locator_history: list[dict[str, Any]] = field(default_factory=list)
-    parameterized: dict[str, str] = field(default_factory=dict)
-
-    def ensure_fingerprint(self) -> str:
-        if self.fingerprint:
-            return self.fingerprint
-        self.fingerprint = element_fingerprint(
-            surface=self.surface,
-            role=self.role or self.control_type,
-            name=self.name,
-            automation_id=self.automation_id,
-            class_name=self.class_name,
-            value=self.value,
-            window_scope=self.window_scope,
-            dom_fingerprint=str(self.raw.get("fingerprint") or self.raw.get("css") or ""),
-            page_scope=str(self.element_scope.get("page_url") or ""),
-            contenteditable=bool(self.raw.get("contenteditable")),
-        )
-        return self.fingerprint
-
-    def touch(self) -> None:
-        self.last_seen = _now_iso()
-        self.observation_count = int(self.observation_count or 0) + 1
-
-
-@dataclass
-class ElementCatalog:
-    schema: str = SCHEMA_CATALOG
-    flow_id: str = ""
-    updated_at: str = field(default_factory=_now_iso)
-    elements: list[ElementRef] = field(default_factory=list)
-
-    def by_logical_name(self) -> dict[str, ElementRef]:
-        return {e.logical_name: e for e in self.elements if e.logical_name}

tools/ui_inspector/models/flow.py

@@ -1,11 +0,0 @@
-"""Flow-level helpers (re-export ActionFlow for models.flow import path)."""
-
-from .action import ActionFlow, ActionRecord
-from .element import ElementCatalog, ElementRef
-from .state import UIState
-from .observation import UIObservation
-
-__all__ = [
-    "ActionFlow", "ActionRecord", "ElementCatalog", "ElementRef", "UIState",
-    "UIObservation",
-]

tools/ui_inspector/models/locator.py

@@ -1,40 +0,0 @@
-"""Structured locator candidate model."""
-
-from __future__ import annotations
-
-from dataclasses import dataclass, field
-from typing import Any, Optional
-
-
-SCHEMA_LOCATOR = "openclaw.ui_inspector.locator_candidate.v1"
-
-
-@dataclass
-class LocatorCandidate:
-    schema: str = SCHEMA_LOCATOR
-    strategy: str = ""
-    scope: dict[str, Any] = field(default_factory=dict)
-    properties: dict[str, Any] = field(default_factory=dict)
-    expression: str = ""
-    expected_count: int = 1
-    actual_count: Optional[int] = None
-    unique: Optional[bool] = None
-    stability_score: float = 0.0
-    uses_index: bool = False
-    uses_coordinates: bool = False
-    validation: str = "none"  # snapshot_only | live_tree | cross_state | cross_session | none
-    reasons: list[str] = field(default_factory=list)
-    ambiguous: bool = False
-    production_eligible: bool = False
-
-    def mark_counts(self, actual: int, *, expected: int = 1) -> None:
-        self.expected_count = int(expected)
-        self.actual_count = int(actual)
-        self.unique = self.actual_count == self.expected_count
-        self.ambiguous = not bool(self.unique)
-        # Coordinates / index never become production primary automatically.
-        self.production_eligible = bool(
-            self.unique
-            and not self.uses_coordinates
-            and not (self.uses_index and self.stability_score < 0.7)
-        )

tools/ui_inspector/models/observation.py

@@ -1,47 +0,0 @@
-"""Per-capture evidence model.
-
-UIState represents a deduplicated semantic state.  UIObservation represents
-one concrete capture and is therefore never deduplicated.
-"""
-
-from __future__ import annotations
-
-import uuid
-from dataclasses import dataclass, field
-from datetime import datetime
-from typing import Any, Optional
-
-SCHEMA_OBSERVATION = "openclaw.ui_inspector.observation.v1"
-
-
-def _now_iso() -> str:
-    return datetime.now().astimezone().isoformat()
-
-
-def _new_id() -> str:
-    return f"obs_{uuid.uuid4().hex[:12]}"
-
-
-@dataclass
-class UIObservation:
-    schema: str = SCHEMA_OBSERVATION
-    id: str = field(default_factory=_new_id)
-    captured_at: str = field(default_factory=_now_iso)
-    state_id: str = ""
-    previous_observation_id: Optional[str] = None
-    change_reason: str = "capture"
-    surface: str = "windows"
-    driver: str = ""
-    page_id: str = ""
-    document_id: str = ""
-    frame_id: str = ""
-    url: str = ""
-    page_title: str = ""
-    tree_path: str = ""
-    index_path: str = ""
-    locatable_path: str = ""
-    screenshot_path: str = ""
-    stats: dict[str, Any] = field(default_factory=dict)
-    selection: dict[str, Any] = field(default_factory=dict)
-    may_contain_sensitive_data: bool = False
-

tools/ui_inspector/models/serialize.py

@@ -1,64 +0,0 @@
-"""Stable JSON serialization helpers for v2 models."""
-
-from __future__ import annotations
-
-import json
-from dataclasses import asdict, is_dataclass
-from datetime import datetime
-from pathlib import Path
-from typing import Any, TypeVar
-
-T = TypeVar("T")
-
-
-def _normalize(obj: Any) -> Any:
-    if obj is None or isinstance(obj, (str, int, float, bool)):
-        return obj
-    if isinstance(obj, datetime):
-        return obj.isoformat()
-    if isinstance(obj, Path):
-        return str(obj).replace("\\", "/")
-    if is_dataclass(obj) and not isinstance(obj, type):
-        return {k: _normalize(v) for k, v in asdict(obj).items()}
-    if isinstance(obj, dict):
-        return {str(k): _normalize(v) for k, v in obj.items()}
-    if isinstance(obj, (list, tuple)):
-        return [_normalize(x) for x in obj]
-    return str(obj)
-
-
-def to_dict(obj: Any) -> dict[str, Any]:
-    data = _normalize(obj)
-    if not isinstance(data, dict):
-        raise TypeError(f"expected dataclass/dict, got {type(obj)!r}")
-    return data
-
-
-def from_dict(cls: type[T], data: dict[str, Any]) -> T:
-    """Best-effort dataclass reconstruction (ignores unknown keys)."""
-    if not is_dataclass(cls):
-        raise TypeError(f"{cls!r} is not a dataclass")
-    import dataclasses
-
-    fields = {f.name for f in dataclasses.fields(cls)}
-    kwargs = {k: v for k, v in (data or {}).items() if k in fields}
-    return cls(**kwargs)  # type: ignore[call-arg]
-
-
-def dumps_json(obj: Any, *, indent: int = 2) -> str:
-    return json.dumps(
-        to_dict(obj) if is_dataclass(obj) and not isinstance(obj, type) else _normalize(obj),
-        ensure_ascii=False,
-        indent=indent,
-        sort_keys=False,
-    ) + "\n"
-
-
-def write_model_json(path: Path, obj: Any) -> None:
-    path = Path(path)
-    path.parent.mkdir(parents=True, exist_ok=True)
-    path.write_text(dumps_json(obj), encoding="utf-8")
-
-
-def read_json(path: Path) -> Any:
-    return json.loads(Path(path).read_text(encoding="utf-8"))

tools/ui_inspector/models/state.py

@@ -1,175 +0,0 @@
-"""UIState model + stable business fingerprint."""
-
-from __future__ import annotations
-
-import hashlib
-import json
-import uuid
-from dataclasses import dataclass, field
-from datetime import datetime, timezone
-from typing import Any, Optional
-
-SCHEMA_STATE = "openclaw.ui_inspector.state.v1"
-
-# Keys ignored when hashing business identity.
-_FINGERPRINT_EXCLUDE_TOP = {
-    "captured_at",
-    "id",
-    "previous_state_id",
-    "tree_path",
-    "index_path",
-    "locatable_path",
-    "screenshot_path",
-    "duplicate_of",
-    "observation_count",
-    "change_reason",
-}
-
-_SESSION_KEYS = {
-    "pid",
-    "hwnd",
-    "process_id",
-    "thread_id",
-    "elapsed_ms",
-    "dump_ms",
-    "warmup_ms",
-    "timings",
-    "screenshot",
-    "screen_path",
-}
-
-
-def _now_iso() -> str:
-    return datetime.now().astimezone().isoformat()
-
-
-def _new_id(prefix: str = "st") -> str:
-    return f"{prefix}_{uuid.uuid4().hex[:12]}"
-
-
-def _strip_session(obj: Any) -> Any:
-    if isinstance(obj, dict):
-        out: dict[str, Any] = {}
-        for k, v in obj.items():
-            lk = str(k).lower()
-            if lk in _SESSION_KEYS or lk.endswith("_ms") or lk.endswith("_path"):
-                continue
-            if lk in {"left", "top", "right", "bottom", "x", "y", "width", "height"}:
-                # Absolute coordinates are not primary identity.
-                continue
-            if lk == "rect" or lk == "bounds" or lk == "bounding_rectangle":
-                continue
-            out[str(k)] = _strip_session(v)
-        return out
-    if isinstance(obj, list):
-        return [_strip_session(x) for x in obj]
-    return obj
-
-
-def _locatable_signature(rows: list[dict[str, Any]], *, limit: int = 120) -> list[dict[str, Any]]:
-    """Compact visible-business signature for fingerprinting."""
-    sig: list[dict[str, Any]] = []
-    for r in rows[:limit]:
-        if not isinstance(r, dict):
-            continue
-        entry = {
-            "control_type": r.get("control_type") or r.get("role") or "",
-            "name": r.get("name") or "",
-            "value": r.get("value") or r.get("text") or "",
-            "automation_id": r.get("automation_id") or r.get("test_id") or "",
-            "class_name": r.get("class_name") or "",
-        }
-        # Drop empty noise rows
-        if not any(str(v).strip() for v in entry.values()):
-            continue
-        sig.append(entry)
-    # Stable order
-    sig.sort(
-        key=lambda e: (
-            str(e.get("control_type") or ""),
-            str(e.get("automation_id") or ""),
-            str(e.get("name") or ""),
-            str(e.get("value") or ""),
-        )
-    )
-    return sig
-
-
-def compute_state_fingerprint(
-    *,
-    surface: str,
-    driver: str = "",
-    url: str = "",
-    page_title: str = "",
-    window: Optional[dict[str, Any]] = None,
-    locatable: Optional[list[dict[str, Any]]] = None,
-    app: Optional[dict[str, Any]] = None,
-    extra_business: Optional[dict[str, Any]] = None,
-) -> str:
-    """
-    SHA-256 of stable business identity.
-
-    Excludes timestamps, pid/hwnd, screenshot paths, absolute coords, timings.
-    """
-    win = dict(window or {})
-    app_d = dict(app or {})
-    payload = {
-        "surface": surface or "",
-        "driver": driver or "",
-        "url": url or win.get("url") or app_d.get("url") or "",
-        "page_title": page_title or "",
-        "window": {
-            "title": win.get("title") or app_d.get("title") or "",
-            "class_name": win.get("class_name") or app_d.get("class_name") or "",
-            "process_name": win.get("process_name") or app_d.get("process_name") or "",
-        },
-        "locatable": _locatable_signature(list(locatable or [])),
-        "extra": _strip_session(extra_business or {}),
-    }
-    raw = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
-    return hashlib.sha256(raw.encode("utf-8")).hexdigest()
-
-
-@dataclass
-class UIState:
-    schema: str = SCHEMA_STATE
-    id: str = field(default_factory=lambda: _new_id("st"))
-    surface: str = "windows"
-    driver: str = ""
-    captured_at: str = field(default_factory=_now_iso)
-    fingerprint: str = ""
-    previous_state_id: Optional[str] = None
-    change_reason: str = ""
-    app: dict[str, Any] = field(default_factory=dict)
-    window: dict[str, Any] = field(default_factory=dict)
-    url: str = ""
-    page_title: str = ""
-    tree_path: str = ""
-    index_path: str = ""
-    locatable_path: str = ""
-    screenshot_path: str = ""
-    stats: dict[str, Any] = field(default_factory=dict)
-    selection: dict[str, Any] = field(default_factory=dict)
-    duplicate_of: Optional[str] = None
-    observation_count: int = 1
-    may_contain_sensitive_data: bool = False
-    # In-memory locatable used for fingerprint/diff (not always persisted as field).
-    locatable: list[dict[str, Any]] = field(default_factory=list, repr=False)
-
-    def ensure_fingerprint(self) -> str:
-        if self.fingerprint:
-            return self.fingerprint
-        self.fingerprint = compute_state_fingerprint(
-            surface=self.surface,
-            driver=self.driver,
-            url=self.url,
-            page_title=self.page_title,
-            window=self.window,
-            locatable=self.locatable,
-            app=self.app,
-        )
-        return self.fingerprint
-
-    def bump_observation(self) -> None:
-        self.observation_count = int(self.observation_count or 0) + 1
-        self.captured_at = _now_iso()

tools/ui_inspector/recorder/__init__.py

@@ -1,13 +0,0 @@
-"""Recorder package: observe user actions outside target processes."""
-
-from .base import RawUserEvent, RecorderBase, RecorderState
-from .input_aggregator import InputAggregator
-from .manager import ActionRecordingManager
-
-__all__ = [
-    "RawUserEvent",
-    "RecorderBase",
-    "RecorderState",
-    "InputAggregator",
-    "ActionRecordingManager",
-]

tools/ui_inspector/recorder/base.py

@@ -1,79 +0,0 @@
-"""Recorder abstractions."""
-
-from __future__ import annotations
-
-import threading
-from abc import ABC, abstractmethod
-from dataclasses import dataclass, field
-from datetime import datetime
-from typing import Any, Callable, Optional
-
-
-def _now_iso() -> str:
-    return datetime.now().astimezone().isoformat()
-
-
-@dataclass
-class RawUserEvent:
-    kind: str  # mouse_down | mouse_up | key | input | navigation | focus | checkpoint
-    surface: str
-    timestamp: str = field(default_factory=_now_iso)
-    button: str = ""
-    key: str = ""
-    text: str = ""
-    x: Optional[int] = None
-    y: Optional[int] = None
-    target: dict[str, Any] = field(default_factory=dict)
-    frame: dict[str, Any] = field(default_factory=dict)
-    url: str = ""
-    meta: dict[str, Any] = field(default_factory=dict)
-    from_inspector: bool = False
-
-
-class RecorderState:
-    IDLE = "idle"
-    RECORDING = "recording"
-    PAUSED = "paused"
-    STOPPED = "stopped"
-
-
-class RecorderBase(ABC):
-    """Process-external observer. No injection / hooks into target."""
-
-    def __init__(
-        self,
-        *,
-        on_event: Optional[Callable[[RawUserEvent], None]] = None,
-        exclude_hwnds: Optional[Callable[[], set[int]]] = None,
-    ) -> None:
-        self.on_event = on_event
-        self.exclude_hwnds = exclude_hwnds or (lambda: set())
-        self.status = RecorderState.IDLE
-        self._thread: Optional[threading.Thread] = None
-        self._stop = threading.Event()
-
-    def emit(self, event: RawUserEvent) -> None:
-        if self.status != RecorderState.RECORDING:
-            return
-        if event.from_inspector:
-            return
-        if self.on_event:
-            self.on_event(event)
-
-    @abstractmethod
-    def start(self) -> None: ...
-
-    @abstractmethod
-    def stop(self) -> None: ...
-
-    def pause(self) -> None:
-        if self.status == RecorderState.RECORDING:
-            self.status = RecorderState.PAUSED
-
-    def resume(self) -> None:
-        if self.status == RecorderState.PAUSED:
-            self.status = RecorderState.RECORDING
-
-    @property
-    def capability(self) -> dict[str, str]:
-        return {}

tools/ui_inspector/recorder/browser.py

@@ -1,740 +0,0 @@
-"""Browser recorder for Inspector-owned Playwright sessions.
-
-Injects capture-phase listeners that do NOT preventDefault / stopPropagation
-and do NOT mutate business DOM. Re-attaches after navigation.
-"""
-
-from __future__ import annotations
-
-import json
-import queue
-import threading
-import time
-from typing import Any, Callable, Optional
-
-from .base import RawUserEvent, RecorderBase, RecorderState
-
-_LISTEN_SCRIPT = r"""
-(() => {
-  if (window.__openclawUiInspectorRecorderInstalled) return;
-  window.__openclawUiInspectorRecorderInstalled = true;
-  window.__openclawUiInspectorEvents = window.__openclawUiInspectorEvents || [];
-  window.__openclawUiInspectorDocId = window.__openclawUiInspectorDocId ||
-    (Date.now().toString(36) + "-" + Math.random().toString(36).slice(2));
-  window.__openclawUiInspectorSeq = window.__openclawUiInspectorSeq || 0;
-
-  function cssPath(el) {
-    if (!el || !el.tagName) return "";
-    const esc = (value) => {
-      try { return CSS.escape(String(value)); } catch (e) { return String(value).replace(/[^a-zA-Z0-9_-]/g, "\\$&"); }
-    };
-    const parts = [];
-    let cur = el;
-    for (let i = 0; cur && i < 12; i++) {
-      let part = cur.tagName.toLowerCase();
-      if (cur.id) { part += "#" + esc(cur.id); parts.unshift(part); break; }
-      const cls = (cur.className && typeof cur.className === "string")
-        ? cur.className.trim().split(/\s+/).slice(0, 3).map(esc).join(".")
-        : "";
-      if (cls) part += "." + cls;
-      parts.unshift(part);
-      cur = cur.parentElement;
-    }
-    return parts.join(" > ");
-  }
-
-  function stableCss(el) {
-    if (!el || !el.getAttribute) return "";
-    const quote = (value) => JSON.stringify(String(value));
-    const ownId = el.id || "";
-    if (ownId) return "#" + CSS.escape(ownId);
-    const testId = el.getAttribute("data-testid") || "";
-    if (testId) return `[data-testid=${quote(testId)}]`;
-    const tag = (el.tagName || "").toLowerCase();
-    let leaf = "";
-    const href = el.getAttribute("href") || "";
-    if (tag === "a" && href) leaf = `a[href=${quote(href)}]`;
-    else if (el.getAttribute("contenteditable") != null) leaf = `${tag || "*"}[contenteditable]`;
-    else if (el.getAttribute("name")) leaf = `${tag || "*"}[name=${quote(el.getAttribute("name"))}]`;
-    else if (el.getAttribute("aria-label")) leaf = `${tag || "*"}[aria-label=${quote(el.getAttribute("aria-label"))}]`;
-    else if (el.getAttribute("placeholder")) leaf = `${tag || "*"}[placeholder=${quote(el.getAttribute("placeholder"))}]`;
-    const stableClasses = (node) => {
-      if (!node || typeof node.className !== "string") return [];
-      return node.className.trim().split(/\s+/).filter((token) => {
-        if (!token || token.length < 4 || token.length > 64) return false;
-        const lower = token.toLowerCase();
-        const semantic = /(button|btn|input|editor|control|action|submit|send|publish|post|play|pause|menu|option|tab|dialog|form|toolbar|container)/.test(lower);
-        const structured = /[-_]/.test(token) && !/^[a-z0-9_-]{6,14}$/i.test(token);
-        return semantic || structured;
-      }).slice(0, 2);
-    };
-    if (!leaf) {
-      const classes = stableClasses(el);
-      leaf = tag + classes.map(cls => "." + CSS.escape(cls)).join("");
-    }
-    let cur = el.parentElement;
-    const relative = [leaf];
-    for (let i = 0; cur && i < 10; i++, cur = cur.parentElement) {
-      if (cur.id) return `#${CSS.escape(cur.id)} > ${relative.reverse().join(" > ")}`;
-      const ancestorTestId = cur.getAttribute?.("data-testid") || "";
-      if (ancestorTestId) return `[data-testid=${quote(ancestorTestId)}] > ${relative.reverse().join(" > ")}`;
-      const curTag = (cur.tagName || "").toLowerCase();
-      const classes = stableClasses(cur);
-      relative.push(curTag + classes.map(cls => "." + CSS.escape(cls)).join(""));
-    }
-    return "";
-  }
-
-  function editableValue(el) {
-    if (!el) return "";
-    if (el.value != null) return String(el.value);
-    if (el.isContentEditable || el.getAttribute?.("contenteditable") != null) {
-      return String(el.innerText != null ? el.innerText : (el.textContent || ""));
-    }
-    return "";
-  }
-
-  function isEditable(el) {
-    if (!el || !el.getAttribute) return false;
-    const tag = (el.tagName || "").toLowerCase();
-    return tag === "input" || tag === "textarea" || tag === "select" ||
-      !!el.isContentEditable || el.getAttribute("contenteditable") != null;
-  }
-
-  function editableRoot(el) {
-    let cur = el;
-    for (let i = 0; cur && i < 12; i++) {
-      if (cur.getAttribute && cur.getAttribute("contenteditable") != null) return cur;
-      const tag = (cur.tagName || "").toLowerCase();
-      if (tag === "input" || tag === "textarea" || tag === "select") return cur;
-      cur = cur.parentElement;
-    }
-    return null;
-  }
-
-  function semanticRole(el) {
-    if (!el || !el.getAttribute) return "";
-    const explicit = el.getAttribute("role");
-    if (explicit) return explicit;
-    const tag = (el.tagName || "").toLowerCase();
-    if (el.isContentEditable) return "textbox";
-    if (tag === "a") return "link";
-    if (tag === "textarea") return "textbox";
-    if (tag === "select") return "combobox";
-    if (tag === "input") {
-      const type = (el.type || "text").toLowerCase();
-      if (type === "checkbox") return "checkbox";
-      if (type === "radio") return "radio";
-      if (["button", "submit", "reset"].includes(type)) return "button";
-      return "textbox";
-    }
-    return tag;
-  }
-
-  function actionableFromEvent(ev) {
-    const path = (ev && typeof ev.composedPath === "function") ? ev.composedPath() : [];
-    const nodes = path.length ? path : [ev && ev.target];
-    let best = ev && ev.target;
-    let bestScore = -1;
-    for (const el of nodes) {
-      if (!el || !el.matches) continue;
-      const tag = (el.tagName || "").toLowerCase();
-      const role = (el.getAttribute("role") || "").toLowerCase();
-      const cls = typeof el.className === "string" ? el.className.toLowerCase() : "";
-      let cursor = "";
-      try { cursor = getComputedStyle(el).cursor || ""; } catch (e) {}
-      let score = 0;
-      if (el.matches("button,a,input,textarea,select,summary,label")) score = 100;
-      if (el.isContentEditable || el.getAttribute("contenteditable") != null) score = 98;
-      if (["button","link","textbox","combobox","checkbox","radio","menuitem","option","tab"].includes(role)) score = Math.max(score, 96);
-      else if (role) score = Math.max(score, 52);
-      if (el.getAttribute("data-testid") || el.getAttribute("aria-label")) score = Math.max(score, 90);
-      if (el.getAttribute("tabindex") != null) score = Math.max(score, 74);
-      if (cursor === "pointer") score = Math.max(score, 68);
-      // Many production apps use camelCase classes such as sendControl or
-      // playButton.  Prefer that actionable wrapper over its inherited-cursor
-      // SVG/path child while still retaining hit_target for diagnostics.
-      if (tag.includes("-") || /(button|btn|submit|send|play|control|action|click)/i.test(cls)) score = Math.max(score, 72);
-      if (score > bestScore) { best = el; bestScore = score; }
-    }
-    return best;
-  }
-
-  function snap(el) {
-    if (!el || !el.getAttribute) return {};
-    const role = semanticRole(el);
-    let cursor = "";
-    let rect = {};
-    try {
-      cursor = getComputedStyle(el).cursor || "";
-      const r = el.getBoundingClientRect();
-      rect = {left:r.left, top:r.top, right:r.right, bottom:r.bottom, width:r.width, height:r.height};
-    } catch (e) {}
-    const tag = el.tagName ? el.tagName.toLowerCase() : "";
-    const css = cssPath(el);
-    const stable = stableCss(el);
-    let cssCount = null;
-    let stableCssCount = null;
-    try { cssCount = css ? document.querySelectorAll(css).length : null; } catch (e) {}
-    try { stableCssCount = stable ? document.querySelectorAll(stable).length : null; } catch (e) {}
-    const href = el.getAttribute("href") || "";
-    let hrefCount = null;
-    try {
-      hrefCount = href ? Array.from(document.querySelectorAll("a[href]")).filter(
-        node => node.getAttribute("href") === href
-      ).length : null;
-    } catch (e) {}
-    const roleNames = ["button","link","textbox","combobox","checkbox","radio","menuitem","option","tab"];
-    return {
-      role: role,
-      explicit_role: el.getAttribute("role") || "",
-      aria_label: el.getAttribute("aria-label") || "",
-      name: el.getAttribute("aria-label") || el.name || el.id ||
-        ((el.matches?.("button,a,[role],label,summary") || roleNames.includes(role) || cursor === "pointer" || tag.includes("-"))
-          ? (el.innerText || el.textContent || "").trim().slice(0, 120) : ""),
-      text: (el.innerText || el.textContent || "").trim().slice(0, 120),
-      value: editableValue(el).slice(0, 500),
-      test_id: el.getAttribute("data-testid") || "",
-      name_attr: el.getAttribute("name") || "",
-      href: href,
-      href_count: hrefCount,
-      title: el.getAttribute("title") || "",
-      alt: el.getAttribute("alt") || "",
-      id: el.id || "",
-      placeholder: el.getAttribute("placeholder") || el.getAttribute("aria-placeholder") || "",
-      label: "",
-      input_type: (el.type || "").toLowerCase(),
-      tag: tag,
-      class_name: (typeof el.className === "string" ? el.className : "").slice(0, 300),
-      cursor: cursor,
-      rect: rect,
-      disabled: !!el.disabled || el.getAttribute("aria-disabled") === "true",
-      contenteditable: !!el.isContentEditable || el.getAttribute("contenteditable") != null,
-      contenteditable_attr: el.getAttribute("contenteditable") != null,
-      tabindex: el.getAttribute("tabindex"),
-      css: css,
-      css_count: cssCount,
-      stable_css: stable,
-      stable_css_count: stableCssCount,
-      fingerprint: [
-        el.tagName?.toLowerCase() || "", el.id || "",
-        el.getAttribute("data-testid") || "", el.getAttribute("name") || "",
-        css
-      ].join("|"),
-    };
-  }
-
-  function ancestry(el) {
-    const out = [];
-    let cur = el;
-    for (let i = 0; cur && i < 12; i++) {
-      out.push(snap(cur));
-      cur = cur.parentElement;
-    }
-    return out;
-  }
-
-  function push(kind, payload) {
-    try {
-      const item = Object.assign({
-        event_id: window.__openclawUiInspectorDocId + ":" + (++window.__openclawUiInspectorSeq),
-        document_id: window.__openclawUiInspectorDocId,
-        sequence: window.__openclawUiInspectorSeq,
-        kind: kind,
-        ts: Date.now(),
-        url: location.href,
-        title: document.title,
-        frame_url: location.href,
-        frame_name: window.name || "",
-      }, payload || {});
-      window.__openclawUiInspectorEvents.push(item);
-      if (typeof window.__openclawUiInspectorEmit === "function") {
-        try { window.__openclawUiInspectorEmit(item); } catch (e) {}
-      }
-      if (window.__openclawUiInspectorEvents.length > 200) {
-        window.__openclawUiInspectorEvents.splice(0, 50);
-      }
-    } catch (e) {}
-  }
-
-  document.addEventListener("click", (ev) => {
-    const hit = ev.target;
-    const action = actionableFromEvent(ev) || hit;
-    push("click", {
-      target: snap(action),
-      hit_target: snap(hit),
-      action_target: snap(action),
-      ancestry: ancestry(hit),
-      candidate_chain: ancestry(hit),
-      button: ev.button,
-      x: ev.clientX,
-      y: ev.clientY
-    });
-  }, true);
-
-  document.addEventListener("pointerdown", (ev) => {
-    const hit = ev.target;
-    push("pointerdown", {
-      target: snap(hit),
-      hit_target: snap(hit),
-      candidate_chain: ancestry(hit),
-      button: ev.button,
-      pointer_id: ev.pointerId,
-      pointer_type: ev.pointerType || "",
-      x: ev.clientX,
-      y: ev.clientY
-    });
-  }, true);
-
-  document.addEventListener("input", (ev) => {
-    if (ev.isComposing) return;
-    const t = editableRoot(ev.target) || actionableFromEvent(ev) || ev.target;
-    push("input", {
-      target: snap(t), text: editableValue(t), input_event: ev.inputType || "input",
-      candidate_chain: ancestry(ev.target), is_trusted: !!ev.isTrusted
-    });
-  }, true);
-
-  document.addEventListener("change", (ev) => {
-    const t = editableRoot(ev.target) || actionableFromEvent(ev) || ev.target;
-    push("input", { target: snap(t), text: editableValue(t), input_event: "change",
-      candidate_chain: ancestry(ev.target), is_trusted: !!ev.isTrusted });
-  }, true);
-
-  document.addEventListener("compositionend", (ev) => {
-    const t = editableRoot(ev.target) || actionableFromEvent(ev) || ev.target;
-    push("input", {
-      target: snap(t), text: editableValue(t), input_event: "compositionend",
-      candidate_chain: ancestry(ev.target), is_trusted: !!ev.isTrusted
-    });
-  }, true);
-
-  document.addEventListener("paste", (ev) => {
-    const t = editableRoot(ev.target) || actionableFromEvent(ev) || ev.target;
-    setTimeout(() => push("input", {
-      target: snap(t), text: editableValue(t), input_event: "paste",
-      candidate_chain: ancestry(ev.target), is_trusted: !!ev.isTrusted
-    }), 0);
-  }, true);
-
-  document.addEventListener("keydown", (ev) => {
-    const special = ["Enter","Escape","Tab","Backspace","Delete"];
-    const editing = isEditable(ev.target);
-    const mutatingEditKey = editing && (ev.key === "Backspace" || ev.key === "Delete");
-    if (!mutatingEditKey && (special.indexOf(ev.key) >= 0 || (ev.key && ev.key.length > 1))) {
-      const t = actionableFromEvent(ev) || ev.target;
-      push("key", { key: ev.key, target: snap(t) });
-    }
-  }, true);
-})();
-"""
-
-
-class BrowserRecorder(RecorderBase):
-    capability_map = {
-        "click": "implemented",
-        "fill": "implemented",
-        "press_key": "implemented",
-        "navigation": "partial",
-    }
-
-    def __init__(
-        self,
-        *,
-        on_event: Optional[Callable[[RawUserEvent], None]] = None,
-        on_raw_event: Optional[Callable[[dict[str, Any]], None]] = None,
-        poll_sec: float = 0.25,
-    ) -> None:
-        super().__init__(on_event=on_event)
-        self.on_raw_event = on_raw_event
-        self.poll_sec = poll_sec
-        self._last_url = ""
-        self._last_urls_by_page: dict[str, str] = {}
-        self._installed_pages: set[int] = set()
-        self._bound_events: queue.Queue[dict[str, Any]] = queue.Queue(maxsize=512)
-        self._seen_event_ids: set[str] = set()
-        self._seen_event_order: list[str] = []
-        self._known_page_ids: set[str] = set()
-        self._pending_navigations: dict[str, dict[str, Any]] = {}
-        self._pending_pointerdowns: dict[str, dict[str, Any]] = {}
-        self._navigation_reorder_sec = 0.45
-        self._pointer_fallback_sec = 0.35
-
-    @property
-    def capability(self) -> dict[str, str]:
-        return dict(self.capability_map)
-
-    def start(self) -> None:
-        if self._thread and self._thread.is_alive():
-            self.status = RecorderState.RECORDING
-            return
-        self._stop.clear()
-        self.status = RecorderState.RECORDING
-        try:
-            import kit_browser
-
-            kit_browser.install_event_bridge(_LISTEN_SCRIPT, self._receive_bound_event)
-            initial_ids = kit_browser.call_on_each_open_page(
-                lambda page: kit_browser.page_id_for(page)
-            )
-            self._known_page_ids.update(
-                str(page_id) for page_id in initial_ids if page_id
-            )
-        except Exception:
-            # Polling remains as a compatibility fallback.
-            pass
-        self._thread = threading.Thread(
-            target=self._loop, name="ui-inspector-browser-recorder", daemon=True
-        )
-        self._thread.start()
-
-    def stop(self) -> None:
-        self._stop.set()
-        if self._thread and self._thread.is_alive():
-            self._thread.join(timeout=2.0)
-        self._thread = None
-        # Disable the bridge before draining its local queue so an event that
-        # arrived just before Stop is not silently lost, while no new events
-        # can race with the drain.
-        try:
-            import kit_browser
-
-            kit_browser.clear_event_bridge()
-        except Exception:
-            pass
-        if self.status == RecorderState.RECORDING:
-            try:
-                self._drain_bound_events()
-                self._flush_pending_pointerdowns(force=True)
-                self._flush_pending_navigations(force=True)
-            except Exception:
-                pass
-        self.status = RecorderState.STOPPED
-
-    def _receive_bound_event(self, event: dict[str, Any]) -> None:
-        try:
-            self._bound_events.put_nowait(dict(event or {}))
-        except queue.Full:
-            try:
-                self._bound_events.get_nowait()
-                self._bound_events.put_nowait(dict(event or {}))
-            except (queue.Empty, queue.Full):
-                pass
-
-    def _loop(self) -> None:
-        while not self._stop.is_set():
-            if self.status == RecorderState.PAUSED:
-                time.sleep(0.1)
-                continue
-            if self.status != RecorderState.RECORDING:
-                time.sleep(0.05)
-                continue
-            try:
-                self._tick()
-            except Exception:
-                pass
-            time.sleep(self.poll_sec)
-
-    def _tick(self) -> None:
-        import kit_browser
-
-        self._drain_bound_events()
-
-        # Install + drain on every open tab (detail may be a second tab).
-        payloads = kit_browser.call_on_each_open_page(self._install_and_drain)
-        if payloads:
-            for payload in payloads:
-                if not isinstance(payload, dict):
-                    continue
-                page_id = str(payload.get("page_id") or "")
-                url = str(payload.get("url") or "")
-                previous_url = self._last_urls_by_page.get(page_id, "") if page_id else ""
-                if previous_url and url and url != previous_url:
-                    self._schedule_navigation(
-                        page_id=page_id, from_url=previous_url, to_url=url
-                    )
-                elif (
-                    page_id
-                    and page_id not in self._known_page_ids
-                    and self._known_page_ids
-                    and url
-                    and not url.startswith("about:")
-                ):
-                    opener_id = str(payload.get("opener_page_id") or "")
-                    self._schedule_navigation(
-                        page_id=page_id,
-                        from_url=self._last_urls_by_page.get(opener_id, ""),
-                        to_url=url,
-                        opener_page_id=opener_id,
-                        new_page=True,
-                    )
-                if page_id and url:
-                    self._last_urls_by_page[page_id] = url
-                    self._known_page_ids.add(page_id)
-                for ev in payload.get("events") or []:
-                    if isinstance(ev, dict) and not ev.get("page_id"):
-                        ev["page_id"] = payload.get("page_id") or ""
-                    self._emit_dom_event(ev)
-
-        self._flush_pending_pointerdowns()
-
-        # Navigation is intentionally emitted after both bridge and polling
-        # queues have been drained.  This preserves click -> navigation order
-        # even when the context bridge is unavailable and polling is used.
-        self._flush_pending_navigations()
-
-    def _drain_bound_events(self) -> None:
-        while True:
-            try:
-                self._emit_dom_event(self._bound_events.get_nowait())
-            except queue.Empty:
-                break
-
-    def _schedule_navigation(
-        self,
-        *,
-        page_id: str,
-        from_url: str,
-        to_url: str,
-        opener_page_id: str = "",
-        new_page: bool = False,
-    ) -> None:
-        self._pending_navigations[page_id] = {
-            "due": time.monotonic() + self._navigation_reorder_sec,
-            "from": from_url,
-            "to": to_url,
-            "page_id": page_id,
-            "opener_page_id": opener_page_id,
-            "new_page": new_page,
-        }
-
-    def _flush_pending_navigations(self, *, force: bool = False) -> None:
-        now = time.monotonic()
-        ready = [
-            page_id
-            for page_id, item in self._pending_navigations.items()
-            if force or float(item.get("due") or 0.0) <= now
-        ]
-        for page_id in ready:
-            self._flush_pending_pointerdowns(force=True, page_id=page_id)
-            item = self._pending_navigations.pop(page_id, None) or {}
-            self.emit(
-                RawUserEvent(
-                    kind="navigation",
-                    surface="browser",
-                    url=str(item.get("to") or ""),
-                    meta={
-                        "from": item.get("from") or "",
-                        "to": item.get("to") or "",
-                        "page_id": page_id,
-                        "opener_page_id": item.get("opener_page_id") or "",
-                        "new_page": bool(item.get("new_page")),
-                    },
-                )
-            )
-
-    def _install_and_drain(self, page: Any) -> dict[str, Any]:
-        """Runs on Playwright worker thread (once per open tab)."""
-        try:
-            page.evaluate(_LISTEN_SCRIPT)
-        except Exception:
-            pass
-        # Also try frames
-        try:
-            for frame in page.frames:
-                try:
-                    frame.evaluate(_LISTEN_SCRIPT)
-                except Exception:
-                    pass
-        except Exception:
-            pass
-        events = []
-        try:
-            events = page.evaluate(
-                """() => {
-                  const e = window.__openclawUiInspectorEvents || [];
-                  window.__openclawUiInspectorEvents = [];
-                  return e;
-                }"""
-            ) or []
-        except Exception:
-            events = []
-        url = ""
-        try:
-            url = page.url or ""
-        except Exception:
-            url = ""
-        page_id = ""
-        opener_page_id = ""
-        try:
-            import kit_browser
-
-            page_id = kit_browser.page_id_for(page)
-            try:
-                opener = page.opener()
-                if opener is not None:
-                    opener_page_id = kit_browser.page_id_for(opener)
-            except Exception:
-                pass
-        except Exception:
-            pass
-        return {
-            "url": url,
-            "page_id": page_id,
-            "opener_page_id": opener_page_id,
-            "events": events,
-        }
-
-    def _emit_dom_event(self, ev: dict[str, Any]) -> None:
-        event_id = str(ev.get("event_id") or "")
-        if event_id:
-            if event_id in self._seen_event_ids:
-                return
-            self._seen_event_ids.add(event_id)
-            self._seen_event_order.append(event_id)
-            if len(self._seen_event_order) > 2000:
-                old = self._seen_event_order.pop(0)
-                self._seen_event_ids.discard(old)
-        if self.on_raw_event:
-            try:
-                self.on_raw_event(dict(ev))
-            except Exception:
-                pass
-        kind = str(ev.get("kind") or "")
-        hit_target = dict(ev.get("hit_target") or ev.get("target") or {})
-        candidate_chain = list(ev.get("candidate_chain") or ev.get("ancestry") or [])
-        if not candidate_chain:
-            candidate_chain = [hit_target]
-            action_hint = dict(ev.get("action_target") or {})
-            if action_hint and action_hint != hit_target:
-                candidate_chain.append(action_hint)
-        try:
-            from core.policies.target_scoring import resolve_browser_target
-
-            target, target_scores = resolve_browser_target(
-                hit_target,
-                candidate_chain,
-                x=int(ev.get("x")) if ev.get("x") is not None else None,
-                y=int(ev.get("y")) if ev.get("y") is not None else None,
-            )
-        except Exception:
-            target = dict(ev.get("action_target") or ev.get("target") or hit_target)
-            target_scores = []
-        if candidate_chain:
-            target["parent_chain"] = candidate_chain
-        frame = {
-            "url": ev.get("frame_url") or "",
-            "name": ev.get("frame_name") or "",
-            "id": ev.get("frame_id") or "",
-        }
-        if kind in {"pointerdown", "click"}:
-            raw = RawUserEvent(
-                    kind="click",
-                    surface="browser",
-                    target=target,
-                    frame=frame,
-                    url=str(ev.get("url") or ""),
-                    x=int(ev.get("x")) if ev.get("x") is not None else None,
-                    y=int(ev.get("y")) if ev.get("y") is not None else None,
-                    meta={
-                        "button": ev.get("button"),
-                        "event_id": event_id,
-                        "document_id": ev.get("document_id"),
-                        "page_id": ev.get("page_id"),
-                        "frame_id": ev.get("frame_id"),
-                        "sequence": ev.get("sequence"),
-                        "source_ts_ms": ev.get("ts"),
-                        "hit_target": ev.get("hit_target") or {},
-                        "action_target": ev.get("action_target") or target,
-                        "target_scores": target_scores,
-                        "captured_from": kind,
-                    },
-                )
-            if kind == "pointerdown":
-                key = self._pointer_key(ev)
-                self._pending_pointerdowns[key] = {
-                    "due": time.monotonic() + self._pointer_fallback_sec,
-                    "event": raw,
-                    "page_id": str(ev.get("page_id") or ""),
-                }
-                return
-            self._cancel_pointerdown_for_click(ev)
-            self.emit(raw)
-        elif kind == "input":
-            self.emit(
-                RawUserEvent(
-                    kind="input",
-                    surface="browser",
-                    text=str(ev.get("text") or target.get("value") or ""),
-                    target=target,
-                    frame=frame,
-                    url=str(ev.get("url") or ""),
-                    meta={
-                        "input_type": target.get("input_type") or "",
-                        "event_id": event_id,
-                        "document_id": ev.get("document_id"),
-                        "page_id": ev.get("page_id"),
-                        "frame_id": ev.get("frame_id"),
-                        "sequence": ev.get("sequence"),
-                        "source_ts_ms": ev.get("ts"),
-                        "input_event": ev.get("input_event") or "input",
-                        "is_trusted": bool(ev.get("is_trusted")),
-                    },
-                )
-            )
-        elif kind == "key":
-            self.emit(
-                RawUserEvent(
-                    kind="key",
-                    surface="browser",
-                    key=str(ev.get("key") or ""),
-                    target=target,
-                    frame=frame,
-                    url=str(ev.get("url") or ""),
-                    meta={
-                        "event_id": event_id,
-                        "document_id": ev.get("document_id"),
-                        "page_id": ev.get("page_id"),
-                        "frame_id": ev.get("frame_id"),
-                        "sequence": ev.get("sequence"),
-                        "source_ts_ms": ev.get("ts"),
-                    },
-                )
-            )
-
-    @staticmethod
-    def _pointer_key(ev: dict[str, Any]) -> str:
-        return "|".join(
-            str(ev.get(key) or "")
-            for key in ("page_id", "document_id", "pointer_id", "button")
-        )
-
-    def _cancel_pointerdown_for_click(self, ev: dict[str, Any]) -> None:
-        exact = self._pointer_key(ev)
-        if exact in self._pending_pointerdowns:
-            self._pending_pointerdowns.pop(exact, None)
-            return
-        page_id = str(ev.get("page_id") or "")
-        button = str(ev.get("button") or "")
-        for key, item in list(self._pending_pointerdowns.items()):
-            event = item.get("event")
-            meta = getattr(event, "meta", {}) or {}
-            if str(meta.get("page_id") or "") == page_id and str(meta.get("button") or "") == button:
-                self._pending_pointerdowns.pop(key, None)
-                return
-
-    def _flush_pending_pointerdowns(
-        self, *, force: bool = False, page_id: str = ""
-    ) -> None:
-        now = time.monotonic()
-        ready = [
-            key for key, item in self._pending_pointerdowns.items()
-            if (not page_id or item.get("page_id") == page_id)
-            and (force or float(item.get("due") or 0.0) <= now)
-        ]
-        for key in ready:
-            item = self._pending_pointerdowns.pop(key, None) or {}
-            event = item.get("event")
-            if event is not None:
-                event.meta["pointerdown_fallback"] = True
-                self.emit(event)

tools/ui_inspector/recorder/input_aggregator.py

@@ -1,234 +0,0 @@
-"""Aggregate keystrokes / value changes into fill/type + press_key."""
-
-from __future__ import annotations
-
-import time
-from dataclasses import dataclass, field
-from typing import Any, Optional
-
-from redaction import redact_input_payload
-
-# Keys recorded as independent press_key actions.
-_FUNCTION_KEYS = {
-    "Enter",
-    "Escape",
-    "Tab",
-    "Backspace",
-    "Delete",
-    "ArrowUp",
-    "ArrowDown",
-    "ArrowLeft",
-    "ArrowRight",
-    "Home",
-    "End",
-    "PageUp",
-    "PageDown",
-    "F1",
-    "F2",
-    "F3",
-    "F4",
-    "F5",
-    "F6",
-    "F7",
-    "F8",
-    "F9",
-    "F10",
-    "F11",
-    "F12",
-}
-
-
-@dataclass
-class AggregatedInput:
-    action_type: str  # fill | type | press_key
-    text: str = ""
-    key: str = ""
-    target: dict[str, Any] = field(default_factory=dict)
-    sensitive: bool = False
-    input_payload: dict[str, Any] = field(default_factory=dict)
-    started_at: float = 0.0
-    ended_at: float = 0.0
-    capture_context: dict[str, Any] = field(default_factory=dict)
-    raw_event: dict[str, Any] = field(default_factory=dict)
-    raw_event_ids: list[str] = field(default_factory=list)
-
-
-class InputAggregator:
-    """
-    Do not emit one Action per key.
-    Observe focus element value changes; flush after idle gap or function key.
-    """
-
-    def __init__(self, *, idle_sec: float = 0.8) -> None:
-        self.idle_sec = idle_sec
-        self._buf_text = ""
-        self._target: dict[str, Any] = {}
-        self._started: float = 0.0
-        self._last: float = 0.0
-        self._input_type = ""
-        self._field_name = ""
-        self._capture_context: dict[str, Any] = {}
-        self._raw_event: dict[str, Any] = {}
-        self._raw_event_ids: list[str] = []
-        self._target_identity = ""
-
-    @staticmethod
-    def _stable_target_identity(
-        target: dict[str, Any], capture_context: dict[str, Any]
-    ) -> str:
-        from core.policies.element_identity import element_identity_key
-
-        return element_identity_key(
-            target or {},
-            surface=str(capture_context.get("surface") or "windows"),
-            page_url=str(capture_context.get("url") or ""),
-            frame_name=str(capture_context.get("frame_name") or ""),
-        )
-
-    @staticmethod
-    def _event_id(raw_event: Optional[dict[str, Any]]) -> str:
-        return str(((raw_event or {}).get("meta") or {}).get("event_id") or "")
-
-    def on_key(
-        self,
-        *,
-        key: str,
-        text: str = "",
-        target: Optional[dict[str, Any]] = None,
-        input_type: str = "",
-        field_name: str = "",
-        capture_context: Optional[dict[str, Any]] = None,
-        raw_event: Optional[dict[str, Any]] = None,
-        value_event: bool = False,
-        now: Optional[float] = None,
-    ) -> list[AggregatedInput]:
-        now = time.monotonic() if now is None else now
-        out: list[AggregatedInput] = []
-        k = key or ""
-        if k in _FUNCTION_KEYS or (len(k) > 1 and k.startswith("F") and k[1:].isdigit()):
-            out.extend(self.flush(now=now))
-            payload, sens = redact_input_payload(
-                {"key": k},
-                field_name=field_name or self._field_name,
-                input_type=input_type or self._input_type,
-            )
-            out.append(
-                AggregatedInput(
-                    action_type="press_key",
-                    key=k,
-                    target=dict(target or self._target or {}),
-                    sensitive=sens,
-                    input_payload=payload,
-                    started_at=now,
-                    ended_at=now,
-                    capture_context=dict(capture_context or self._capture_context or {}),
-                    raw_event=dict(raw_event or {}),
-                    raw_event_ids=(
-                        [self._event_id(raw_event)] if self._event_id(raw_event) else []
-                    ),
-                )
-            )
-            return out
-
-        # Printable / value change
-        if target:
-            # Focus changed → flush previous
-            next_context = dict(capture_context or self._capture_context or {})
-            next_identity = self._stable_target_identity(target, next_context)
-            if self._target_identity and next_identity != self._target_identity:
-                out.extend(self.flush(now=now))
-            self._target = dict(target)
-            self._target_identity = next_identity
-        if input_type:
-            self._input_type = input_type
-        if field_name:
-            self._field_name = field_name
-        if capture_context:
-            self._capture_context = dict(capture_context)
-        if raw_event:
-            self._raw_event = dict(raw_event)
-            event_id = self._event_id(raw_event)
-            if event_id and event_id not in self._raw_event_ids:
-                self._raw_event_ids.append(event_id)
-
-        chunk = text if text else (k if len(k) == 1 else "")
-        if not chunk and not text and not value_event:
-            return out
-
-        if not self._started:
-            self._started = now
-        # If observer sends full value, replace buffer; else append char
-        if value_event:
-            self._buf_text = text
-        elif text and len(text) >= len(self._buf_text):
-            self._buf_text = text
-        else:
-            self._buf_text += chunk
-        self._last = now
-        return out
-
-    def on_value_change(
-        self,
-        *,
-        value: str,
-        target: Optional[dict[str, Any]] = None,
-        input_type: str = "",
-        field_name: str = "",
-        capture_context: Optional[dict[str, Any]] = None,
-        raw_event: Optional[dict[str, Any]] = None,
-        now: Optional[float] = None,
-    ) -> list[AggregatedInput]:
-        return self.on_key(
-            key="",
-            text=value,
-            target=target,
-            input_type=input_type,
-            field_name=field_name,
-            capture_context=capture_context,
-            raw_event=raw_event,
-            value_event=True,
-            now=now,
-        )
-
-    def flush(self, *, now: Optional[float] = None) -> list[AggregatedInput]:
-        now = (
-            (time.time() if self._last > 100_000_000 else time.monotonic())
-            if now is None else now
-        )
-        if not self._buf_text and not self._started:
-            return []
-        payload, sens = redact_input_payload(
-            {"text": self._buf_text, "name": self._field_name},
-            field_name=self._field_name,
-            input_type=self._input_type,
-        )
-        item = AggregatedInput(
-            action_type="fill",
-            text=str(payload.get("text") or ""),
-            target=dict(self._target or {}),
-            sensitive=sens,
-            input_payload=payload,
-            started_at=self._started or now,
-            ended_at=now,
-            capture_context=dict(self._capture_context or {}),
-            raw_event=dict(getattr(self, "_raw_event", {}) or {}),
-            raw_event_ids=list(self._raw_event_ids),
-        )
-        self._buf_text = ""
-        self._started = 0.0
-        self._last = 0.0
-        self._input_type = ""
-        self._capture_context = {}
-        self._raw_event = {}
-        self._raw_event_ids = []
-        # keep target for next session
-        return [item]
-
-    def maybe_flush_idle(self, *, now: Optional[float] = None) -> list[AggregatedInput]:
-        now = (
-            (time.time() if self._last > 100_000_000 else time.monotonic())
-            if now is None else now
-        )
-        if self._last and (now - self._last) >= self.idle_sec and self._started:
-            return self.flush(now=now)
-        return []

tools/ui_inspector/redaction.py

@@ -1,206 +0,0 @@
-"""Sensitive-data redaction for Inspector artifacts."""
-
-from __future__ import annotations
-
-import re
-from typing import Any, Optional
-
-# Field / attribute name hints (case-insensitive).
-_SENSITIVE_NAME_RE = re.compile(
-    r"(password|passwd|pwd|token|secret|cookie|authorization|auth|otp|"
-    r"captcha|verify.?code|verification.?code|sms.?code|pin)",
-    re.I,
-)
-
-_PHONE_RE = re.compile(r"(?<!\d)(?:\+?86[-\s]?)?1[3-9]\d{9}(?!\d)")
-_EMAIL_RE = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
-_TOKENISH_RE = re.compile(r"\b(?:Bearer\s+)?[A-Za-z0-9_\-]{24,}\b")
-
-REDACTED = "[REDACTED]"
-
-
-def is_sensitive_field_name(name: str) -> bool:
-    return bool(_SENSITIVE_NAME_RE.search(name or ""))
-
-
-def looks_sensitive_value(value: str) -> bool:
-    s = value or ""
-    if not s:
-        return False
-    if _PHONE_RE.search(s) or _EMAIL_RE.search(s) or _TOKENISH_RE.fullmatch(s.strip()):
-        return True
-    if "password" in s.lower() or "secret" in s.lower():
-        return True
-    return False
-
-
-def sanitize_artifact(value: Any, *, sensitive_hint: bool = False) -> Any:
-    """Recursively sanitize event/evidence data before it is persisted.
-
-    ``redact_input_payload`` protects the normalized Action input, but recorder
-    events also contain copies of the same value in target snapshots, ancestry
-    and raw_event.  All persisted copies must pass through one boundary.
-    """
-    if isinstance(value, dict):
-        input_type = str(value.get("input_type") or value.get("type") or "").lower()
-        field_name = str(
-            value.get("name")
-            or value.get("field")
-            or value.get("automation_id")
-            or ""
-        )
-        local_sensitive = bool(
-            sensitive_hint
-            or input_type == "password"
-            or is_sensitive_field_name(field_name)
-        )
-        out: dict[str, Any] = {}
-        for key, item in value.items():
-            key_sensitive = local_sensitive or is_sensitive_field_name(str(key))
-            if key in {
-                "name", "automation_id", "class_name", "control_type",
-                "framework_id", "path_hint", "logical_name",
-            } and isinstance(item, str) and not key_sensitive:
-                # Machine identifiers can be long token-like strings (for
-                # example conversation_list_container) without being secrets.
-                # Still remove explicit personal data embedded in a label.
-                out[key] = _EMAIL_RE.sub(REDACTED, _PHONE_RE.sub(REDACTED, item))
-                continue
-            if key in {"value", "text", "input", "typed", "content", "plaintext"}:
-                if item in (None, ""):
-                    out[key] = item
-                elif key_sensitive or looks_sensitive_value(str(item)):
-                    out[key] = REDACTED
-                else:
-                    out[key] = redact_text(str(item))
-            else:
-                out[key] = sanitize_artifact(item, sensitive_hint=key_sensitive)
-        if local_sensitive:
-            out["sensitive"] = True
-        return out
-    if isinstance(value, list):
-        return [sanitize_artifact(item, sensitive_hint=sensitive_hint) for item in value]
-    if isinstance(value, tuple):
-        return [sanitize_artifact(item, sensitive_hint=sensitive_hint) for item in value]
-    if isinstance(value, str):
-        if sensitive_hint or looks_sensitive_value(value):
-            return REDACTED
-        return redact_text(value)
-    return value
-
-
-def redact_text(value: str, *, force: bool = False) -> str:
-    if value is None:
-        return ""
-    s = str(value)
-    if force:
-        return REDACTED
-    s = _PHONE_RE.sub(REDACTED, s)
-    s = _EMAIL_RE.sub(REDACTED, s)
-    return s
-
-
-def redact_input_payload(
-    payload: dict[str, Any],
-    *,
-    field_name: str = "",
-    input_type: str = "",
-) -> tuple[dict[str, Any], bool]:
-    """
-    Return (redacted_copy, was_sensitive).
-    Never leave raw password/token in action_flow / catalog.
-    """
-    sensitive = False
-    out = dict(payload or {})
-    itype = (input_type or out.get("input_type") or "").lower()
-    fname = field_name or str(out.get("name") or out.get("field") or "")
-
-    if itype == "password" or is_sensitive_field_name(fname):
-        sensitive = True
-        for key in ("text", "value", "input", "typed", "content"):
-            if key in out and out[key] not in (None, ""):
-                out[key] = REDACTED
-        out["redacted"] = True
-        return out, True
-
-    for key in ("text", "value", "input", "typed", "content"):
-        if key not in out:
-            continue
-        raw = out.get(key)
-        if raw is None:
-            continue
-        s = str(raw)
-        if looks_sensitive_value(s) or is_sensitive_field_name(str(out.get("name") or "")):
-            sensitive = True
-            force = ("@" in s) or bool(_PHONE_RE.search(s)) or bool(_TOKENISH_RE.search(s))
-            out[key] = redact_text(s, force=force)
-            if _TOKENISH_RE.fullmatch(s.strip()) or looks_sensitive_value(s):
-                out[key] = REDACTED
-    if sensitive:
-        out["redacted"] = True
-    return out, sensitive
-
-
-def redact_element_fields(el: dict[str, Any]) -> tuple[dict[str, Any], bool]:
-    out = dict(el or {})
-    sensitive = False
-    name = str(out.get("name") or out.get("automation_id") or "")
-    role = str(out.get("control_type") or out.get("role") or "")
-    input_type = str(out.get("input_type") or out.get("type") or "")
-    if input_type == "password" or is_sensitive_field_name(name):
-        sensitive = True
-        if out.get("value"):
-            out["value"] = REDACTED
-        if out.get("text"):
-            out["text"] = REDACTED
-        out["sensitive"] = True
-        return out, True
-    for key in ("value", "text", "name"):
-        if key not in out or out[key] in (None, ""):
-            continue
-        s = str(out[key])
-        # Stable element names/automation ids are locator structure.  Long
-        # machine identifiers (OperationAndSendMessageRegion, generated DOM
-        # ids, etc.) must not be destroyed merely because they look token-like.
-        # Personal data embedded in a visible name is still scrubbed.
-        if key == "name":
-            cleaned = _EMAIL_RE.sub(REDACTED, _PHONE_RE.sub(REDACTED, s))
-            out[key] = cleaned
-            sensitive = sensitive or cleaned != s
-        elif looks_sensitive_value(s):
-            sensitive = True
-            out[key] = REDACTED
-    if sensitive:
-        out["sensitive"] = True
-    # Edit controls with password-ish class
-    if "password" in role.lower() or "password" in str(out.get("class_name") or "").lower():
-        sensitive = True
-        out["value"] = REDACTED
-        out["sensitive"] = True
-    return out, sensitive
-
-
-def redact_locatable_rows(rows: list[dict[str, Any]]) -> tuple[list[dict[str, Any]], bool]:
-    any_sens = False
-    out: list[dict[str, Any]] = []
-    for r in rows or []:
-        cleaned, sens = redact_element_fields(r if isinstance(r, dict) else {})
-        any_sens = any_sens or sens
-        out.append(cleaned)
-    return out, any_sens
-
-
-def screenshot_sensitivity_meta(
-    *,
-    may_contain_sensitive: bool,
-    save_sensitive_screenshots: bool = False,
-) -> dict[str, Any]:
-    return {
-        "may_contain_sensitive_data": bool(may_contain_sensitive),
-        "save_sensitive_screenshots": bool(save_sensitive_screenshots),
-        "note": (
-            "截图暂时无法可靠脱敏；敏感页请关闭截图或人工审查。"
-            if may_contain_sensitive
-            else ""
-        ),
-    }

tools/ui_inspector/requirements.txt

@@ -1,4 +0,0 @@
-uiautomation>=2.0.18
-# 浏览器整页 dump（与技能共用宿主环境；本地开发可 editable 安装）
-jiangchang-platform-kit>=1.2.11
-# playwright 由宿主 / kit 环境提供，勿在技能目录 playwright install

tools/ui_inspector/run.py

@@ -1,10 +0,0 @@
-#!/usr/bin/env python3
-"""Entry: python tools/ui_inspector/run.py [--out DIR] [--scene NAME]"""
-
-from __future__ import annotations
-
-import runpy
-from pathlib import Path
-
-if __name__ == "__main__":
-    runpy.run_path(str(Path(__file__).resolve().parent / "app.py"), run_name="__main__")

tools/ui_inspector/skip_policy.py

@@ -1,89 +0,0 @@
-"""Decide whether a dump is blank/empty-shell and should not become a flow step.
-
-Only applied to browser surface. Does NOT skip same-URL scrolls
-(content may change without URL change).
-"""
-
-from __future__ import annotations
-
-from typing import Any, Optional
-
-import target
-
-# Locatable below this counts as "no useful candidates"
-_MIN_LOCATABLE = 3
-# Tree/index tiny enough to treat as empty shell
-_MAX_EMPTY_NODES = 2
-
-
-def _norm_url(url: str) -> str:
-    return (url or "").strip()
-
-
-def is_blankish_url(url: str) -> bool:
-    u = _norm_url(url).lower()
-    if not u:
-        return True
-    if u == "about:blank" or u.startswith("about:blank?"):
-        return True
-    if u in {"about:srcdoc", "about:newtab"}:
-        return True
-    if u.startswith("chrome://newtab") or u.startswith("chrome://new-tab-page"):
-        return True
-    if u.startswith("edge://newtab") or u.startswith("edge://new-tab-page"):
-        return True
-    return False
-
-
-def should_skip_empty_dump(
-    *,
-    surface: str,
-    url: str = "",
-    locatable_count: Optional[int] = None,
-    nodes_kept: Optional[int] = None,
-    sparse: bool = False,
-    locatable: Optional[list[dict[str, Any]]] = None,
-    stats: Optional[dict[str, Any]] = None,
-) -> Optional[str]:
-    """Return skip reason code, or None if the dump should be kept.
-
-    Safe rules (browser only):
-    - blank/newtab URL **and** almost no locatable → skip
-    - tiny tree + sparse/blankish + almost no locatable → skip
-
-    A page with about:blank URL but rich locatable (nav race) is **kept**.
-    """
-    if surface != target.SURFACE_BROWSER:
-        return None
-
-    st = stats or {}
-    if locatable_count is None:
-        if locatable is not None:
-            locatable_count = len(locatable)
-        else:
-            locatable_count = int(st.get("locatable_count") or 0)
-    if nodes_kept is None:
-        nodes_kept = int(st.get("nodes_kept") or st.get("findall_count") or 0)
-    if not sparse:
-        sparse = bool(st.get("uia_sparse"))
-
-    loc_n = int(locatable_count or 0)
-    kept = int(nodes_kept or 0)
-    blankish = is_blankish_url(url)
-
-    if blankish and loc_n < _MIN_LOCATABLE:
-        return "blank_or_newtab_empty"
-
-    if loc_n < _MIN_LOCATABLE and kept <= _MAX_EMPTY_NODES and (sparse or blankish):
-        return "empty_shell"
-
-    return None
-
-
-def skip_reason_text(reason: str, *, url: str = "") -> str:
-    u = _norm_url(url) or "(empty)"
-    if reason == "blank_or_newtab_empty":
-        return f"跳过空页：url={u!r} 且可定位候选过少（blank/新标签页）。"
-    if reason == "empty_shell":
-        return f"跳过空壳页：url={u!r} 节点过少且 sparse。"
-    return f"跳过本拍：{reason} url={u!r}"

tools/ui_inspector/smoke_check.py

@@ -1,383 +0,0 @@
-"""Smoke test for flow document API + surface routing + kit helpers (v0.10)."""
-
-from __future__ import annotations
-
-import sys
-import tempfile
-from pathlib import Path
-
-_HERE = Path(__file__).resolve().parent
-if str(_HERE) not in sys.path:
-    sys.path.insert(0, str(_HERE))
-
-import artifacts as art
-import desktop_uia as uia
-import kit_browser
-import kit_wake
-import skip_policy
-import target
-import user_prefs
-from surfaces.registry import get_driver
-from drivers.mobile import MobileDriver
-from jiangchang_skill_core.rpa.page_dump import build_page_dump_result
-
-
-def _assert_selection_api() -> None:
-    assert callable(uia.window_info_under_cursor)
-    assert callable(uia.find_wecom_search_result_window)
-    assert callable(uia.resolve_capture_target)
-    try:
-        _info, _cands, meta = uia.resolve_capture_target(
-            under_cursor=False,
-            prefer_wecom_search_popup=False,
-        )
-        assert "strategy" in meta
-    except RuntimeError as exc:
-        assert "未找到可拾取" in str(exc)
-    print("OK selection api")
-
-
-def _assert_kit_wake_detect() -> None:
-    assert kit_wake.looks_like_wecom(class_name="WeWorkWindow", title="企业微信")
-    assert kit_wake.resolve_wake_profile(class_name="SearchResultWindow2") == "wecom_search"
-    assert not kit_wake.looks_like_wecom(class_name="Chrome_WidgetWin_1", title="Google")
-    r = kit_wake.auto_ensure_if_wecom(class_name="NotWeCom", title="x")
-    assert r.get("skipped") and r.get("reason") == "not_wecom"
-    print("OK kit_wake detect")
-
-
-def _assert_surface_routing() -> None:
-    assert (
-        target.resolve_surface(process_name="chrome.exe", class_name="Chrome_WidgetWin_1")
-        == target.SURFACE_BROWSER
-    )
-    assert (
-        target.resolve_surface(process_name="msedge.exe", class_name="Chrome_WidgetWin_1")
-        == target.SURFACE_BROWSER
-    )
-    assert (
-        target.resolve_surface(process_name="WXWork.exe", class_name="WeWorkWindow")
-        == target.SURFACE_WINDOWS
-    )
-    assert get_driver(target.SURFACE_WINDOWS).driver_name == "uiautomation"
-    assert get_driver(target.SURFACE_BROWSER).driver_name == "playwright"
-    mobile = MobileDriver(surface=target.SURFACE_ANDROID)
-    try:
-        mobile.dump(info=None, selection_meta={})
-        raise AssertionError("mobile dump should raise")
-    except RuntimeError as exc:
-        assert "SURFACE_UNSUPPORTED" in str(exc)
-    print("OK surface routing")
-
-
-def _assert_user_prefs() -> None:
-    with tempfile.TemporaryDirectory(prefix="ui_prefs_") as tmp:
-        path = Path(tmp) / "user_prefs.json"
-        # monkeypatch prefs path
-        old = user_prefs._PREFS_PATH
-        try:
-            user_prefs._PREFS_PATH = path
-            user_prefs.save_prefs(mode="windows", start_url="https://example.com")
-            loaded = user_prefs.load_prefs()
-            assert loaded["mode"] == "windows"
-            assert loaded["start_url"] == "https://example.com"
-            user_prefs.save_prefs(mode="browser", start_url="")
-            loaded_empty = user_prefs.load_prefs()
-            assert loaded_empty["start_url"] == ""
-            user_prefs.save_prefs(mode="mobile", start_url="https://a.test")
-            loaded2 = user_prefs.load_prefs()
-            assert loaded2["mode"] == target.SURFACE_ANDROID
-        finally:
-            user_prefs._PREFS_PATH = old
-    print("OK user_prefs")
-
-
-def _assert_skip_policy() -> None:
-    # blank + empty locatable → skip
-    assert (
-        skip_policy.should_skip_empty_dump(
-            surface="browser",
-            url="about:blank",
-            locatable_count=0,
-            nodes_kept=1,
-            sparse=True,
-        )
-        == "blank_or_newtab_empty"
-    )
-    # blank URL but rich content (nav race like step05) → keep
-    assert (
-        skip_policy.should_skip_empty_dump(
-            surface="browser",
-            url="about:blank",
-            locatable_count=74,
-            nodes_kept=97,
-            sparse=False,
-        )
-        is None
-    )
-    # real page → keep
-    assert (
-        skip_policy.should_skip_empty_dump(
-            surface="browser",
-            url="https://www.douyin.com/jingxuan",
-            locatable_count=150,
-            nodes_kept=160,
-            sparse=False,
-        )
-        is None
-    )
-    # windows never skipped by this policy
-    assert (
-        skip_policy.should_skip_empty_dump(
-            surface="windows",
-            url="",
-            locatable_count=0,
-            nodes_kept=1,
-            sparse=True,
-        )
-        is None
-    )
-    # same URL with content is NOT skipped (scroll-safe)
-    assert (
-        skip_policy.should_skip_empty_dump(
-            surface="browser",
-            url="https://www.douyin.com/jingxuan",
-            locatable_count=20,
-            nodes_kept=30,
-            sparse=False,
-        )
-        is None
-    )
-    print("OK skip_policy")
-
-
-def _assert_active_page_resolve() -> None:
-    """Unit-level: visible tab wins over sticky pages[0]."""
-
-    class _FakePage:
-        def __init__(self, url: str, visible: bool, *, closed: bool = False) -> None:
-            self.url = url
-            self._visible = visible
-            self._closed = closed
-
-        def is_closed(self) -> bool:
-            return self._closed
-
-        def evaluate(self, _script: str) -> str:
-            return "visible" if self._visible else "hidden"
-
-        def title(self) -> str:
-            return self.url
-
-    class _FakeCtx:
-        def __init__(self, pages: list) -> None:
-            self.pages = pages
-
-    search = _FakePage("https://www.douyin.com/search/foo", visible=False)
-    detail = _FakePage("https://www.douyin.com/video/123", visible=True)
-    ctx = _FakeCtx([search, detail])
-
-    page, meta = kit_browser.resolve_active_page(ctx, preferred=search)
-    assert page is detail, (page.url, meta)
-    assert meta["resolve"] == "visibility"
-    assert meta["page_count"] == 2
-    assert meta["visible_count"] == 1
-
-    # No visible → preferred if still open
-    both_hidden = _FakeCtx(
-        [
-            _FakePage("https://a.test/1", visible=False),
-            _FakePage("https://a.test/2", visible=False),
-        ]
-    )
-    preferred = both_hidden.pages[0]
-    page2, meta2 = kit_browser.resolve_active_page(both_hidden, preferred=preferred)
-    assert page2 is preferred
-    assert meta2["resolve"] == "preferred_fallback"
-
-    # Closed preferred → first open
-    closed_pref = _FakePage("https://gone", visible=False, closed=True)
-    open_only = _FakePage("https://alive", visible=False)
-    page3, meta3 = kit_browser.resolve_active_page(
-        _FakeCtx([closed_pref, open_only]), preferred=closed_pref
-    )
-    assert page3 is open_only
-    assert meta3["resolve"] == "first_open"
-    print("OK active page resolve")
-
-
-def _assert_page_dump_kit() -> None:
-    if not kit_browser.kit_page_dump_available():
-        raise RuntimeError(
-            "jiangchang_skill_core.rpa.page_dump 不可用；"
-            "请 pip install -e jiangchang-platform-kit>=1.2.11"
-        )
-    result = build_page_dump_result(
-        url="https://example.com",
-        title="Example",
-        a11y_snapshot={
-            "role": "RootWebArea",
-            "name": "Example",
-            "children": [{"role": "button", "name": "OK", "children": []}],
-        },
-        dom_nodes=[
-            {
-                "tag": "button",
-                "role": "button",
-                "id": "",
-                "name_attr": "",
-                "testid": "",
-                "aria_label": "OK",
-                "placeholder": "",
-                "title": "",
-                "alt": "",
-                "href": "",
-                "type": "submit",
-                "text": "OK",
-                "disabled": False,
-                "rect": {},
-            }
-        ],
-    )
-    assert result["stats"]["locatable_count"] >= 1
-    assert result["locatable"][0].get("primary_locator")
-    print("OK page_dump kit builder")
-
-
-def _assert_label() -> None:
-    msg_page = [
-        {"control_type": "TreeItemControl", "name": "groupbuddyname", "value": "某群"},
-        {"control_type": "TreeItemControl", "name": "send_rich_edit", "value": "你好"},
-        {"control_type": "TreeItemControl", "name": "con_op", "value": "消息"},
-    ]
-    searching = [
-        {"control_type": "TreeItemControl", "name": "groupbuddyname", "value": "徐书建"},
-        {"control_type": "TreeControl", "name": "searchedit", "value": "陈德炼"},
-        {"control_type": "TreeItemControl", "name": "send_rich_edit", "value": "陈德炼"},
-    ]
-    selected = [
-        {
-            "control_type": "TreeItemControl",
-            "name": "groupbuddyname",
-            "value": "陈德炼<c #99ffffff>@微信</c>",
-        },
-        {"control_type": "TreeItemControl", "name": "send_rich_edit", "value": "您好"},
-    ]
-    contacts = [
-        {"control_type": "TreeItemControl", "name": "panel_title_label", "value": "RPA之家"},
-        {"control_type": "TreeItemControl", "name": "main_chattext", "value": "新的客户"},
-    ]
-    new_customer = [
-        {"control_type": "TreeItemControl", "name": "panel_title_label", "value": "新的客户"},
-        {"control_type": "TreeItemControl", "name": "ww_btn_text", "value": "添加"},
-    ]
-    add_empty = [
-        {"control_type": "TreeItemControl", "name": "title", "value": "添加客户"},
-        {"control_type": "TreeControl", "name": "searchedit", "value": ""},
-        {"control_type": "WindowControl", "name": "添加客户", "value": ""},
-    ]
-    add_result = [
-        {"control_type": "TreeItemControl", "name": "title", "value": "添加客户"},
-        {"control_type": "TreeControl", "name": "searchedit", "value": "13316463772"},
-        {"control_type": "TreeItemControl", "name": "main_chattext", "value": "Nancy"},
-        {"control_type": "TreeItemControl", "name": "ww_btn_text", "value": "添加为"},
-        {"control_type": "TreeControl", "name": "search_result", "value": ""},
-    ]
-
-    a = art.derive_step_label(window_title="企业微信", locatable=msg_page)
-    b = art.derive_step_label(window_title="企业微信", locatable=searching)
-    c = art.derive_step_label(window_title="企业微信", locatable=selected)
-    d = art.derive_step_label(window_title="企业微信", locatable=contacts)
-    e = art.derive_step_label(window_title="企业微信", locatable=new_customer)
-    f = art.derive_step_label(window_title="添加客户", locatable=add_empty)
-    g = art.derive_step_label(window_title="添加客户", locatable=add_result)
-
-    assert "输入你好" in a or "某群" in a, a
-    assert "搜索陈德炼" in b, b
-    assert "陈德炼" in c and "输入您好" in c, c
-    assert "RPA之家" in d, d
-    assert "新的客户" in e and "添加" in e, e
-    assert f.startswith("添加客户"), f
-    assert "13316463772" in g or "搜索13316463772" in g, g
-    assert "会话列表" not in a and "会话列表" not in b and "会话列表" not in c
-    print(f"OK labels:\n  {a}\n  {b}\n  {c}\n  {d}\n  {e}\n  {f}\n  {g}")
-
-
-def main() -> int:
-    _assert_selection_api()
-    _assert_kit_wake_detect()
-    _assert_surface_routing()
-    _assert_user_prefs()
-    _assert_skip_policy()
-    _assert_active_page_resolve()
-    _assert_page_dump_kit()
-    _assert_label()
-    with tempfile.TemporaryDirectory(prefix="ui_flow_smoke_") as tmp:
-        root = Path(tmp)
-        flow_dir = art.create_flow(root, flow_name="demo-flow")
-        flow = art.load_flow(flow_dir)
-
-        loc = [
-            {
-                "control_type": "TreeItemControl",
-                "name": "groupbuddyname",
-                "value": "陈德炼<c #99ffffff>@微信</c>",
-            },
-            {"control_type": "TreeItemControl", "name": "send_rich_edit", "value": "您好"},
-        ]
-        label = art.resolve_step_label(
-            flow, window_title="企业微信", window_class="WeWorkWindow", locatable=loc
-        )
-        step_dir, idx, name = art.add_step_dir(
-            flow_dir,
-            flow,
-            window_title="企业微信",
-            window_class="WeWorkWindow",
-            locatable=loc,
-            label=label,
-        )
-        assert idx == 1
-        assert "陈德炼" in name, name
-        art.write_json(
-            step_dir / "meta.json",
-            art.build_meta(
-                window={"title": "企业微信", "class_name": "WeWorkWindow", "pid": 1, "hwnd": 2},
-                scene="demo",
-                surface="windows",
-                driver="uiautomation",
-            ),
-        )
-        art.register_step(
-            flow_dir,
-            flow,
-            step_index=idx,
-            step_name=name,
-            window={"title": "企业微信", "class_name": "WeWorkWindow", "pid": 1, "hwnd": 2},
-            stats={"findall_count": 10, "locatable_count": 5, "nodes_kept": 3, "uia_sparse": False},
-            label=label,
-            surface="windows",
-        )
-        # browser-shaped step registration
-        blow = art.load_flow(flow_dir)
-        art.register_step(
-            flow_dir,
-            blow,
-            step_index=2,
-            step_name="02_browser-demo",
-            window={"title": "精选", "class_name": "PlaywrightPage", "pid": 3, "hwnd": 4},
-            stats={"findall_count": 20, "locatable_count": 12, "nodes_kept": 20, "uia_sparse": False},
-            label="浏览器精选",
-            surface="browser",
-            url="https://www.douyin.com/jingxuan",
-        )
-        flow = art.end_flow(flow_dir, art.load_flow(flow_dir))
-        assert flow["status"] == "ended"
-        summary = (flow_dir / "SUMMARY.md").read_text(encoding="utf-8")
-        assert "surface: `browser`" in summary or "surface: `windows`" in summary
-        print(f"OK flow={flow_dir.name} step={name}")
-    print("OK ui_inspector flow smoke passed")
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())

tools/ui_inspector/snapshot_service.py

@@ -1,400 +0,0 @@
-"""Snapshot service: wrap SurfaceDriver.dump into deduped UIState assets."""
-
-from __future__ import annotations
-
-import time
-from dataclasses import asdict, is_dataclass
-from pathlib import Path
-from typing import Any, Optional
-
-from models.observation import UIObservation
-from models.serialize import to_dict, write_model_json
-from models.state import UIState, compute_state_fingerprint
-from redaction import (
-    redact_locatable_rows,
-    sanitize_artifact,
-    screenshot_sensitivity_meta,
-)
-
-
-def _dump_to_parts(dump: Any) -> dict[str, Any]:
-    if dump is None:
-        return {}
-    if isinstance(dump, dict):
-        return dict(dump)
-    # DumpResult dataclass or duck-typed dump objects (tests / drivers).
-    if is_dataclass(dump) and not isinstance(dump, type) or hasattr(dump, "surface"):
-        return {
-            "surface": getattr(dump, "surface", "") or "",
-            "driver": getattr(dump, "driver", "") or "",
-            "window": dict(getattr(dump, "window", None) or {}),
-            "tree": getattr(dump, "tree", None) or {},
-            "index": list(getattr(dump, "index", None) or []),
-            "locatable": list(getattr(dump, "locatable", None) or []),
-            "stats": dict(getattr(dump, "stats", None) or {}),
-            "selection": dict(getattr(dump, "selection", None) or {}),
-            "url": getattr(dump, "url", "") or "",
-            "page_title": getattr(dump, "page_title", "") or "",
-            "screenshot_bytes": getattr(dump, "screenshot_bytes", None),
-            "use_hwnd_screenshot": bool(getattr(dump, "use_hwnd_screenshot", False)),
-            "locatable_text": getattr(dump, "locatable_text", "") or "",
-            "tree_text": getattr(dump, "tree_text", "") or "",
-        }
-    return {}
-
-
-class SnapshotService:
-    """
-    Creates UIState records with fingerprint dedupe.
-
-    Reuses SurfaceDriver.dump(); does not reimplement browser/desktop dump.
-    """
-
-    def __init__(
-        self,
-        flow_dir: Path,
-        *,
-        save_sensitive_screenshots: bool = False,
-        persist_evidence: bool = True,
-    ) -> None:
-        self.flow_dir = Path(flow_dir)
-        self.states_dir = self.flow_dir / "states"
-        self.states_dir.mkdir(parents=True, exist_ok=True)
-        self.observations_dir = self.flow_dir / "observations"
-        self.observations_dir.mkdir(parents=True, exist_ok=True)
-        self.save_sensitive_screenshots = save_sensitive_screenshots
-        self.persist_evidence = persist_evidence
-        self._by_fp: dict[str, UIState] = {}
-        self._order: list[str] = []
-        self.latest: Optional[UIState] = None
-        self.latest_observation: Optional[UIObservation] = None
-        self._observation_order: list[str] = []
-
-    def load_existing(self) -> None:
-        """Index existing state JSON files (best-effort)."""
-        if not self.states_dir.exists():
-            return
-        for path in sorted(self.states_dir.glob("*.json")):
-            try:
-                import json
-
-                data = json.loads(path.read_text(encoding="utf-8"))
-                st = UIState(
-                    id=data.get("id") or path.stem,
-                    surface=data.get("surface") or "windows",
-                    driver=data.get("driver") or "",
-                    captured_at=data.get("captured_at") or "",
-                    fingerprint=data.get("fingerprint") or "",
-                    previous_state_id=data.get("previous_state_id"),
-                    change_reason=data.get("change_reason") or "",
-                    app=data.get("app") or {},
-                    window=data.get("window") or {},
-                    url=data.get("url") or "",
-                    page_title=data.get("page_title") or "",
-                    tree_path=data.get("tree_path") or "",
-                    index_path=data.get("index_path") or "",
-                    locatable_path=data.get("locatable_path") or "",
-                    screenshot_path=data.get("screenshot_path") or "",
-                    stats=data.get("stats") or {},
-                    selection=data.get("selection") or {},
-                    duplicate_of=data.get("duplicate_of"),
-                    observation_count=int(data.get("observation_count") or 1),
-                    may_contain_sensitive_data=bool(
-                        data.get("may_contain_sensitive_data")
-                    ),
-                )
-                if st.fingerprint:
-                    self._by_fp[st.fingerprint] = st
-                    if st.id not in self._order:
-                        self._order.append(st.id)
-                    self.latest = st
-            except Exception:
-                continue
-        for path in sorted(self.observations_dir.glob("*/meta.json")):
-            try:
-                import json
-
-                data = json.loads(path.read_text(encoding="utf-8"))
-                obs = UIObservation(
-                    **{k: v for k, v in data.items() if k in UIObservation.__dataclass_fields__}
-                )
-                self._observation_order.append(obs.id)
-                self.latest_observation = obs
-            except Exception:
-                continue
-
-    def capture_from_dump(
-        self,
-        dump: Any,
-        *,
-        change_reason: str = "",
-        force_persist: bool = False,
-        previous_state_id: Optional[str] = None,
-        redact: bool = True,
-    ) -> UIState:
-        parts = _dump_to_parts(dump)
-        locatable = list(parts.get("locatable") or [])
-        sensitive = False
-        if redact:
-            locatable, sensitive = redact_locatable_rows(locatable)
-
-        window = dict(parts.get("window") or {})
-        url = parts.get("url") or ""
-        page_title = parts.get("page_title") or ""
-        surface = parts.get("surface") or "windows"
-        driver = parts.get("driver") or ""
-        fp = compute_state_fingerprint(
-            surface=surface,
-            driver=driver,
-            url=url,
-            page_title=page_title,
-            window=window,
-            locatable=locatable,
-            app={
-                "title": window.get("title"),
-                "class_name": window.get("class_name"),
-                "process_name": window.get("process_name"),
-            },
-        )
-
-        existing = self._by_fp.get(fp)
-        if existing is not None:
-            existing.bump_observation()
-            existing.duplicate_of = None
-            # Update meta path only — do not rewrite full tree evidence.
-            write_model_json(self.states_dir / f"{existing.id}.json", existing)
-            self.latest = existing
-            self._persist_observation(
-                existing,
-                parts,
-                locatable=locatable,
-                sensitive=sensitive,
-                change_reason=change_reason or "capture",
-            )
-            return existing
-
-        prev = previous_state_id
-        if prev is None and self.latest is not None:
-            prev = self.latest.id
-
-        state = UIState(
-            surface=surface,
-            driver=driver,
-            fingerprint=fp,
-            previous_state_id=prev,
-            change_reason=change_reason or "capture",
-            app={
-                "title": window.get("title"),
-                "class_name": window.get("class_name"),
-                "process_name": window.get("process_name"),
-                "pid": window.get("pid"),
-                "hwnd": window.get("hwnd"),
-            },
-            window=window,
-            url=url,
-            page_title=page_title,
-            stats=dict(parts.get("stats") or {}),
-            selection=dict(parts.get("selection") or {}),
-            locatable=locatable,
-            may_contain_sensitive_data=sensitive,
-        )
-        state.ensure_fingerprint()
-
-        should_write_evidence = force_persist or bool(change_reason) or self.persist_evidence
-        if should_write_evidence:
-            self._persist_evidence(state, parts, sensitive=sensitive)
-
-        write_model_json(self.states_dir / f"{state.id}.json", state)
-        self._by_fp[fp] = state
-        self._order.append(state.id)
-        self.latest = state
-        self._persist_observation(
-            state,
-            parts,
-            locatable=locatable,
-            sensitive=sensitive,
-            change_reason=change_reason or "capture",
-        )
-        return state
-
-    def _persist_observation(
-        self,
-        state: UIState,
-        parts: dict[str, Any],
-        *,
-        locatable: list[dict[str, Any]],
-        sensitive: bool,
-        change_reason: str,
-    ) -> UIObservation:
-        stats = dict(parts.get("stats") or {})
-        selection = dict(parts.get("selection") or {})
-        active_page = dict(stats.get("active_page") or {})
-        browser_session = dict(stats.get("browser_session") or {})
-        obs = UIObservation(
-            state_id=state.id,
-            previous_observation_id=(
-                self.latest_observation.id if self.latest_observation else None
-            ),
-            change_reason=change_reason,
-            surface=state.surface,
-            driver=state.driver,
-            page_id=str(
-                active_page.get("page_id")
-                or browser_session.get("page_id")
-                or selection.get("page_id")
-                or ""
-            ),
-            document_id=str(selection.get("document_id") or ""),
-            frame_id=str(selection.get("frame_id") or ""),
-            url=state.url,
-            page_title=state.page_title,
-            stats=stats,
-            selection=selection,
-            may_contain_sensitive_data=sensitive,
-        )
-        evidence_dir = self.observations_dir / obs.id
-        evidence_dir.mkdir(parents=True, exist_ok=True)
-        import artifacts as art
-
-        art.write_json(
-            evidence_dir / "tree.json",
-            sanitize_artifact(parts.get("tree") or {}),
-        )
-        clean_index = sanitize_artifact(list(parts.get("index") or []))
-        art.write_jsonl(evidence_dir / "index.jsonl", list(clean_index or []))
-        art.write_json(evidence_dir / "locatable.json", list(locatable or []))
-        obs.tree_path = f"observations/{obs.id}/tree.json"
-        obs.index_path = f"observations/{obs.id}/index.jsonl"
-        obs.locatable_path = f"observations/{obs.id}/locatable.json"
-
-        sensitivity_meta = screenshot_sensitivity_meta(
-            may_contain_sensitive=sensitive,
-            save_sensitive_screenshots=self.save_sensitive_screenshots,
-        )
-        art.write_json(evidence_dir / "sensitivity.json", sensitivity_meta)
-        if (not sensitive) or self.save_sensitive_screenshots:
-            png = parts.get("screenshot_bytes")
-            if png:
-                try:
-                    (evidence_dir / "screen.png").write_bytes(png)
-                    obs.screenshot_path = f"observations/{obs.id}/screen.png"
-                except Exception:
-                    pass
-            elif parts.get("use_hwnd_screenshot"):
-                hwnd = int((parts.get("window") or {}).get("hwnd") or 0)
-                if hwnd and art.capture_window_png(hwnd, evidence_dir / "screen.png"):
-                    obs.screenshot_path = f"observations/{obs.id}/screen.png"
-
-        write_model_json(evidence_dir / "meta.json", obs)
-        self.latest_observation = obs
-        self._observation_order.append(obs.id)
-        return obs
-
-    def _persist_evidence(
-        self,
-        state: UIState,
-        parts: dict[str, Any],
-        *,
-        sensitive: bool,
-    ) -> None:
-        evidence_dir = self.states_dir / state.id
-        evidence_dir.mkdir(parents=True, exist_ok=True)
-        import artifacts as art
-
-        art.write_json(
-            evidence_dir / "tree.json",
-            sanitize_artifact(parts.get("tree") or {}),
-        )
-        clean_index = sanitize_artifact(list(parts.get("index") or []))
-        art.write_jsonl(evidence_dir / "index.jsonl", list(clean_index or []))
-        art.write_json(evidence_dir / "locatable.json", list(state.locatable or []))
-        state.tree_path = f"states/{state.id}/tree.json"
-        state.index_path = f"states/{state.id}/index.jsonl"
-        state.locatable_path = f"states/{state.id}/locatable.json"
-
-        sens_meta = screenshot_sensitivity_meta(
-            may_contain_sensitive=sensitive,
-            save_sensitive_screenshots=self.save_sensitive_screenshots,
-        )
-        art.write_json(evidence_dir / "sensitivity.json", sens_meta)
-
-        save_png = (not sensitive) or self.save_sensitive_screenshots
-        if save_png:
-            png = parts.get("screenshot_bytes")
-            if png:
-                try:
-                    (evidence_dir / "screen.png").write_bytes(png)
-                    state.screenshot_path = f"states/{state.id}/screen.png"
-                except Exception:
-                    pass
-            elif parts.get("use_hwnd_screenshot"):
-                hwnd = int((parts.get("window") or {}).get("hwnd") or 0)
-                if hwnd:
-                    ok = art.capture_window_png(hwnd, evidence_dir / "screen.png")
-                    if ok:
-                        state.screenshot_path = f"states/{state.id}/screen.png"
-        elif sensitive:
-            state.may_contain_sensitive_data = True
-
-    def wait_until_stable(
-        self,
-        dump_fn,
-        *,
-        timeout_sec: float = 3.0,
-        poll_sec: float = 0.25,
-        change_reason: str = "after_action",
-    ) -> UIState:
-        """
-        Poll dump_fn until two consecutive fingerprints match or timeout.
-        dump_fn() -> DumpResult / dump-like object
-        """
-        deadline = time.monotonic() + max(0.2, float(timeout_sec))
-        last_fp: Optional[str] = None
-        last_dump = None
-        while True:
-            last_dump = dump_fn()
-            parts = _dump_to_parts(last_dump)
-            locatable, _ = redact_locatable_rows(list(parts.get("locatable") or []))
-            fp = compute_state_fingerprint(
-                surface=parts.get("surface") or "windows",
-                driver=parts.get("driver") or "",
-                url=parts.get("url") or "",
-                page_title=parts.get("page_title") or "",
-                window=parts.get("window") or {},
-                locatable=locatable,
-            )
-            if last_fp is not None and fp == last_fp:
-                break
-            last_fp = fp
-            if time.monotonic() >= deadline:
-                break
-            time.sleep(max(0.05, float(poll_sec)))
-        return self.capture_from_dump(
-            last_dump,
-            change_reason=change_reason,
-            force_persist=True,
-        )
-
-    def state_ids(self) -> list[str]:
-        return list(self._order)
-
-    def observation_ids(self) -> list[str]:
-        return list(self._observation_order)
-
-    def get(self, state_id: str) -> Optional[UIState]:
-        for st in self._by_fp.values():
-            if st.id == state_id:
-                return st
-        path = self.states_dir / f"{state_id}.json"
-        if path.exists():
-            import json
-            from models.state import UIState as S
-
-            data = json.loads(path.read_text(encoding="utf-8"))
-            return S(**{k: v for k, v in data.items() if k in S.__dataclass_fields__})
-        return None
-
-    def to_summary_dict(self, state: UIState) -> dict[str, Any]:
-        d = to_dict(state)
-        d["locatable"] = list(state.locatable or [])
-        return d

tools/ui_inspector/state_diff.py

@@ -1,275 +0,0 @@
-"""State Diff + proposed postconditions."""
-
-from __future__ import annotations
-
-from dataclasses import dataclass, field
-from typing import Any, Optional
-
-
-@dataclass
-class StateChange:
-    kind: str
-    detail: dict[str, Any] = field(default_factory=dict)
-    confidence: float = 0.7
-
-
-@dataclass
-class ProposedPostcondition:
-    description: str
-    kind: str
-    status: str = "proposed"  # proposed | user_confirmed | runtime_verified
-    detail: dict[str, Any] = field(default_factory=dict)
-
-
-def _row_key(r: dict[str, Any]) -> str:
-    return "|".join(
-        [
-            str(r.get("control_type") or r.get("role") or ""),
-            str(r.get("automation_id") or ""),
-            str(r.get("name") or ""),
-            str(r.get("class_name") or ""),
-        ]
-    )
-
-
-def _row_value(r: dict[str, Any]) -> str:
-    return str(r.get("value") or r.get("text") or "")
-
-
-def _index_rows(rows: Optional[list[dict[str, Any]]]) -> dict[str, dict[str, Any]]:
-    out: dict[str, dict[str, Any]] = {}
-    for r in rows or []:
-        if not isinstance(r, dict):
-            continue
-        k = _row_key(r)
-        if k.strip("|"):
-            out[k] = r
-    return out
-
-
-def diff_states(
-    before: Optional[dict[str, Any]],
-    after: Optional[dict[str, Any]],
-) -> list[StateChange]:
-    """
-    Diff two state-like dicts (UIState.to_dict or lightweight summaries).
-
-    Expected keys: url, page_title, window, locatable, stats, selection
-    """
-    b = before or {}
-    a = after or {}
-    changes: list[StateChange] = []
-
-    b_url = str(b.get("url") or "")
-    a_url = str(a.get("url") or "")
-    if b_url != a_url:
-        changes.append(
-            StateChange(
-                kind="url_changed",
-                detail={"from": b_url, "to": a_url},
-            )
-        )
-
-    b_title = str(b.get("page_title") or (b.get("window") or {}).get("title") or "")
-    a_title = str(a.get("page_title") or (a.get("window") or {}).get("title") or "")
-    if b_title != a_title:
-        changes.append(
-            StateChange(
-                kind="title_changed",
-                detail={"from": b_title, "to": a_title},
-            )
-        )
-
-    b_cls = str((b.get("window") or {}).get("class_name") or "")
-    a_cls = str((a.get("window") or {}).get("class_name") or "")
-    b_hwnd = (b.get("window") or {}).get("hwnd")
-    a_hwnd = (a.get("window") or {}).get("hwnd")
-    if b_cls and a_cls and b_cls != a_cls:
-        changes.append(
-            StateChange(
-                kind="window_changed",
-                detail={"from_class": b_cls, "to_class": a_cls},
-            )
-        )
-    if b_hwnd and not a_hwnd:
-        changes.append(StateChange(kind="window_disappeared", detail={"class_name": b_cls}))
-    if a_hwnd and not b_hwnd:
-        changes.append(StateChange(kind="window_appeared", detail={"class_name": a_cls}))
-
-    # Popup heuristics by class name
-    for label, cls in (("before", b_cls), ("after", a_cls)):
-        del label
-    if "SearchResult" in a_cls and "SearchResult" not in b_cls:
-        changes.append(
-            StateChange(kind="popup_appeared", detail={"class_name": a_cls})
-        )
-    if "SearchResult" in b_cls and "SearchResult" not in a_cls:
-        changes.append(
-            StateChange(kind="popup_disappeared", detail={"class_name": b_cls})
-        )
-
-    b_rows = _index_rows(b.get("locatable") or [])
-    a_rows = _index_rows(a.get("locatable") or [])
-    b_keys = set(b_rows)
-    a_keys = set(a_rows)
-
-    for k in sorted(a_keys - b_keys)[:40]:
-        r = a_rows[k]
-        changes.append(
-            StateChange(
-                kind="element_appeared",
-                detail={
-                    "key": k,
-                    "name": r.get("name"),
-                    "value": _row_value(r),
-                    "control_type": r.get("control_type") or r.get("role"),
-                    "automation_id": r.get("automation_id"),
-                },
-            )
-        )
-    for k in sorted(b_keys - a_keys)[:40]:
-        r = b_rows[k]
-        changes.append(
-            StateChange(
-                kind="element_disappeared",
-                detail={
-                    "key": k,
-                    "name": r.get("name"),
-                    "value": _row_value(r),
-                    "control_type": r.get("control_type") or r.get("role"),
-                    "automation_id": r.get("automation_id"),
-                },
-            )
-        )
-    for k in sorted(b_keys & a_keys)[:80]:
-        bv = _row_value(b_rows[k])
-        av = _row_value(a_rows[k])
-        if bv != av:
-            changes.append(
-                StateChange(
-                    kind="element_value_changed",
-                    detail={
-                        "key": k,
-                        "name": a_rows[k].get("name"),
-                        "from": bv,
-                        "to": av,
-                        "automation_id": a_rows[k].get("automation_id"),
-                    },
-                )
-            )
-
-    b_focus = (b.get("selection") or {}).get("focused") or (b.get("selection") or {}).get(
-        "focus"
-    )
-    a_focus = (a.get("selection") or {}).get("focused") or (a.get("selection") or {}).get(
-        "focus"
-    )
-    if b_focus != a_focus and (b_focus or a_focus):
-        changes.append(
-            StateChange(
-                kind="focus_changed",
-                detail={"from": b_focus, "to": a_focus},
-            )
-        )
-
-    b_loc = int((b.get("stats") or {}).get("locatable_count") or len(b.get("locatable") or []))
-    a_loc = int((a.get("stats") or {}).get("locatable_count") or len(a.get("locatable") or []))
-    if b_loc != a_loc:
-        changes.append(
-            StateChange(
-                kind="locatable_count_changed",
-                detail={"from": b_loc, "to": a_loc},
-            )
-        )
-
-    return changes
-
-
-def propose_postconditions(changes: list[StateChange]) -> list[ProposedPostcondition]:
-    """Generate suggested postconditions (always status=proposed)."""
-    props: list[ProposedPostcondition] = []
-    for ch in changes:
-        kind = ch.kind
-        d = ch.detail or {}
-        if kind == "url_changed":
-            props.append(
-                ProposedPostcondition(
-                    description=f"URL became {d.get('to')!r}",
-                    kind="url_equals",
-                    detail=d,
-                )
-            )
-        elif kind == "popup_disappeared":
-            props.append(
-                ProposedPostcondition(
-                    description=f"Window/popup {d.get('class_name')!r} disappeared",
-                    kind="window_absent",
-                    detail=d,
-                )
-            )
-        elif kind == "popup_appeared":
-            props.append(
-                ProposedPostcondition(
-                    description=f"Window/popup {d.get('class_name')!r} appeared",
-                    kind="window_present",
-                    detail=d,
-                )
-            )
-        elif kind == "title_changed":
-            props.append(
-                ProposedPostcondition(
-                    description=f"Title became {d.get('to')!r}",
-                    kind="title_equals",
-                    detail=d,
-                )
-            )
-        elif kind == "element_appeared":
-            name = d.get("name") or d.get("automation_id") or d.get("value")
-            props.append(
-                ProposedPostcondition(
-                    description=f"Element {name!r} appeared",
-                    kind="element_present",
-                    detail=d,
-                )
-            )
-        elif kind == "element_disappeared":
-            name = d.get("name") or d.get("automation_id") or d.get("value")
-            props.append(
-                ProposedPostcondition(
-                    description=f"Element {name!r} disappeared",
-                    kind="element_absent",
-                    detail=d,
-                )
-            )
-        elif kind == "element_value_changed":
-            props.append(
-                ProposedPostcondition(
-                    description=(
-                        f"Element {d.get('name') or d.get('automation_id')!r} "
-                        f"value became {d.get('to')!r}"
-                    ),
-                    kind="element_value_equals",
-                    detail=d,
-                )
-            )
-    return props
-
-
-def changes_to_dicts(changes: list[StateChange]) -> list[dict[str, Any]]:
-    return [
-        {"kind": c.kind, "detail": c.detail, "confidence": c.confidence} for c in changes
-    ]
-
-
-def postconditions_to_dicts(
-    props: list[ProposedPostcondition],
-) -> list[dict[str, Any]]:
-    return [
-        {
-            "description": p.description,
-            "kind": p.kind,
-            "status": p.status,
-            "detail": p.detail,
-        }
-        for p in props
-    ]

tools/ui_inspector/storage/__init__.py

@@ -1,6 +0,0 @@
-"""Capture persistence API."""
-
-from core.pipeline.event_journal import EventJournal
-
-__all__ = ["EventJournal"]
-

tools/ui_inspector/storage/capture_reader.py

@@ -1,20 +0,0 @@
-"""Stable capture reader used by diagnostics and downstream AI tooling."""
-
-from __future__ import annotations
-
-import json
-from pathlib import Path
-from typing import Any
-
-
-def read_json(path: Path) -> dict[str, Any]:
-    return json.loads(Path(path).read_text(encoding="utf-8"))
-
-
-def read_action_flow(flow_dir: Path) -> dict[str, Any]:
-    return read_json(Path(flow_dir) / "action_flow.json")
-
-
-def read_element_catalog(flow_dir: Path) -> dict[str, Any]:
-    return read_json(Path(flow_dir) / "element_catalog.json")
-

tools/ui_inspector/storage/capture_writer.py

@@ -1,6 +0,0 @@
-"""Compatibility export for the current atomic artifact writer."""
-
-from artifacts import create_flow, load_flow, save_flow, write_json
-
-__all__ = ["create_flow", "load_flow", "save_flow", "write_json"]
-

tools/ui_inspector/surfaces/__init__.py

@@ -1,2 +0,0 @@
-"""Automation surface implementations."""
-

tools/ui_inspector/surfaces/base.py

@@ -1,19 +0,0 @@
-"""Surface contracts shared by browser, Windows, and future mobile drivers."""
-
-from __future__ import annotations
-
-from typing import Any, Protocol
-
-
-class SurfaceRecorder(Protocol):
-    status: str
-    def start(self) -> None: ...
-    def stop(self) -> None: ...
-    def pause(self) -> None: ...
-    def resume(self) -> None: ...
-
-
-class SurfaceDumpDriver(Protocol):
-    surface: str
-    def dump(self, **kwargs: Any) -> Any: ...
-

tools/ui_inspector/surfaces/browser/__init__.py

@@ -1,6 +0,0 @@
-"""Browser surface public API."""
-
-from .recorder import BrowserRecorder
-from .dump_driver import BrowserPlaywrightDriver
-
-__all__ = ["BrowserRecorder", "BrowserPlaywrightDriver"]

tools/ui_inspector/surfaces/browser/dump_driver.py

@@ -1,6 +0,0 @@
-"""Browser full-dump driver export."""
-
-from drivers.browser_pw import BrowserPlaywrightDriver
-
-__all__ = ["BrowserPlaywrightDriver"]
-

tools/ui_inspector/surfaces/browser/locator_compiler.py

@@ -1,6 +0,0 @@
-"""Browser locator compiler export."""
-
-from locators.browser import compile_browser_locators
-
-__all__ = ["compile_browser_locators"]
-

tools/ui_inspector/surfaces/browser/locator_validator.py

@@ -1,6 +0,0 @@
-"""Browser live locator validator export."""
-
-from kit_browser import validate_element_on_page
-
-__all__ = ["validate_element_on_page"]
-

tools/ui_inspector/surfaces/browser/page_registry.py

@@ -1,6 +0,0 @@
-"""Stable browser page/frame identity facade."""
-
-from kit_browser import page_id_for
-
-__all__ = ["page_id_for"]
-

tools/ui_inspector/surfaces/browser/recorder.py

@@ -1,6 +0,0 @@
-"""Browser recorder compatibility export during staged migration."""
-
-from recorder.browser import BrowserRecorder
-
-__all__ = ["BrowserRecorder"]
-

tools/ui_inspector/surfaces/browser/runtime.py

@@ -1,23 +0,0 @@
-"""Inspector-owned Playwright runtime facade."""
-
-from kit_browser import (
-    call_on_each_open_page,
-    call_on_page,
-    clear_event_bridge,
-    dump_current_page,
-    ensure_session,
-    install_event_bridge,
-    page_id_for,
-    screenshot_png,
-    session_info,
-    shutdown,
-    validate_element_on_page,
-)
-
-__all__ = [
-    "call_on_each_open_page", "call_on_page", "clear_event_bridge",
-    "dump_current_page", "ensure_session", "install_event_bridge",
-    "page_id_for", "screenshot_png", "session_info", "shutdown",
-    "validate_element_on_page",
-]
-

tools/ui_inspector/surfaces/mobile/__init__.py

@@ -1,6 +0,0 @@
-"""Future mobile surface public API."""
-
-from drivers.mobile import MobileDriver
-
-__all__ = ["MobileDriver"]
-

tools/ui_inspector/surfaces/registry.py

@@ -1,22 +0,0 @@
-"""Resolve a dump driver through the canonical Surface namespace."""
-
-from __future__ import annotations
-
-import target
-from .browser.dump_driver import BrowserPlaywrightDriver
-from .mobile import MobileDriver
-from .windows.dump_driver import WindowsUiaDriver
-
-
-def get_driver(surface: str):
-    if surface == target.SURFACE_BROWSER:
-        return BrowserPlaywrightDriver()
-    if surface == target.SURFACE_WINDOWS:
-        return WindowsUiaDriver()
-    if surface in {target.SURFACE_ANDROID, target.SURFACE_IOS}:
-        return MobileDriver(surface=surface)
-    raise RuntimeError(f"ERROR:SURFACE_UNSUPPORTED unknown surface={surface!r}")
-
-
-__all__ = ["get_driver"]
-

tools/ui_inspector/surfaces/windows/__init__.py

@@ -1,6 +0,0 @@
-"""Windows surface public API."""
-
-from .recorder import WindowsRecorder
-from .dump_driver import WindowsUiaDriver
-
-__all__ = ["WindowsRecorder", "WindowsUiaDriver"]

tools/ui_inspector/surfaces/windows/dump_driver.py

@@ -1,4 +0,0 @@
-from drivers.windows_uia import WindowsUiaDriver
-
-__all__ = ["WindowsUiaDriver"]
-

tools/ui_inspector/surfaces/windows/locator_compiler.py

@@ -1,4 +0,0 @@
-from locators.windows import compile_windows_locators
-
-__all__ = ["compile_windows_locators"]
-

tools/ui_inspector/surfaces/windows/recorder.py

@@ -1,4 +0,0 @@
-from recorder.windows import WindowsRecorder
-
-__all__ = ["WindowsRecorder"]
-

tools/ui_inspector/target.py

@@ -1,82 +0,0 @@
-"""Resolve capture surface from a Win32 window (windows | browser | mobile stub)."""
-
-from __future__ import annotations
-
-from typing import Any, Optional
-
-# Logical surfaces (align with RPA_ELEMENT_LOCATING.md)
-SURFACE_WINDOWS = "windows"
-SURFACE_BROWSER = "browser"
-SURFACE_ANDROID = "android"
-SURFACE_IOS = "ios"
-
-_BROWSER_PROCESS_NAMES = {
-    "chrome.exe",
-    "msedge.exe",
-    "chromium.exe",
-    "brave.exe",
-    "opera.exe",
-    "vivaldi.exe",
-}
-
-_BROWSER_CLASS_HINTS = {
-    "Chrome_WidgetWin_1",
-    "Chrome_WidgetWin_0",
-    "Chrome_WidgetWin_2",
-}
-
-
-def normalize_process_name(name: str) -> str:
-    return (name or "").strip().lower()
-
-
-def is_browser_window(
-    *,
-    process_name: str = "",
-    class_name: str = "",
-    title: str = "",
-) -> bool:
-    proc = normalize_process_name(process_name)
-    if proc in _BROWSER_PROCESS_NAMES:
-        return True
-    cls = (class_name or "").strip()
-    if cls in _BROWSER_CLASS_HINTS:
-        return True
-    # Edge/Chrome sometimes expose empty process briefly; title heuristics last
-    t = title or ""
-    if " - Google Chrome" in t or " - Microsoft Edge" in t or t.endswith(" - Edge"):
-        return True
-    if t.endswith(" - Chrome"):
-        return True
-    return False
-
-
-def resolve_surface(
-    *,
-    process_name: str = "",
-    class_name: str = "",
-    title: str = "",
-) -> str:
-    """Return surface id for this top-level window."""
-    if is_browser_window(
-        process_name=process_name, class_name=class_name, title=title
-    ):
-        return SURFACE_BROWSER
-    return SURFACE_WINDOWS
-
-
-def surface_from_window_info(info: Any) -> str:
-    return resolve_surface(
-        process_name=getattr(info, "process_name", "") or "",
-        class_name=getattr(info, "class_name", "") or "",
-        title=getattr(info, "title", "") or "",
-    )
-
-
-def surface_label(surface: str) -> str:
-    return {
-        SURFACE_WINDOWS: "桌面 UIA",
-        SURFACE_BROWSER: "浏览器 Playwright",
-        SURFACE_ANDROID: "Android（未实现）",
-        SURFACE_IOS: "iOS（未实现）",
-    }.get(surface, surface)

tools/ui_inspector/tests/contract/__init__.py

@@ -1 +0,0 @@
-

tools/ui_inspector/tests/contract/test_capture_contract.py

@@ -1,207 +0,0 @@
-from pathlib import Path
-
-from flow_session import FlowSession
-from recorder.base import RawUserEvent
-from diagnostics.capture_validator import validate_capture
-from diagnostics.replay_preflight import build_replay_preflight
-
-
-def test_session_always_creates_raw_event_journal(tmp_path: Path) -> None:
-    session = FlowSession(tmp_path, flow_name="contract", surface="browser", dump_fn=None)
-    session.manager.start()
-    session.manager.status = "recording"
-    event = RawUserEvent(kind="click", surface="browser", target={"role": "button", "name": "Go"})
-    session.journal.append(event, channel="semantic")
-    assert (tmp_path / "journal" / "raw_events.jsonl").exists()
-
-
-def test_validator_flags_oversized_click_target(tmp_path: Path) -> None:
-    import json
-    actions = tmp_path / "actions"
-    actions.mkdir()
-    (tmp_path / "elements").mkdir()
-    (tmp_path / "observations").mkdir()
-    (tmp_path / "journal").mkdir()
-    (tmp_path / "journal" / "raw_events.jsonl").write_text("{}\n", encoding="utf-8")
-    (actions / "a.json").write_text(
-        json.dumps({
-            "id": "a", "sequence": 1, "action_type": "click",
-            "target_snapshot": {"rect": {"width": 1200, "height": 5000}},
-            "evidence": {"eventual_observation_id": "obs"},
-        }),
-        encoding="utf-8",
-    )
-    result = validate_capture(tmp_path)
-    assert any(issue["code"] == "CLICK_TARGET_OVERSIZED" for issue in result["issues"])
-
-
-def test_validator_flags_missing_desktop_input_and_generic_targets(tmp_path: Path) -> None:
-    import json
-
-    (tmp_path / "actions").mkdir()
-    (tmp_path / "elements").mkdir()
-    (tmp_path / "observations").mkdir()
-    (tmp_path / "journal").mkdir()
-    journal = {
-        "event": {
-            "kind": "typing_activity", "surface": "windows",
-            "target": {"name": "searchedit", "control_type": "TreeControl"},
-        }
-    }
-    (tmp_path / "journal" / "raw_events.jsonl").write_text(
-        json.dumps(journal) + "\n", encoding="utf-8"
-    )
-    rows = [
-        {
-            "id": "a1", "sequence": 1, "surface": "windows", "action_type": "click",
-            "target_snapshot": {
-                "name": "search_result", "control_type": "TreeControl",
-                "bounds": {"width": 600, "height": 500},
-            },
-            "evidence": {"eventual_observation_id": "obs"},
-        },
-        {
-            "id": "a2", "sequence": 2, "surface": "windows", "action_type": "press_key",
-            "target_snapshot": {
-                "name": "WeCom", "control_type": "WindowControl",
-                "bounds": {"width": 1440, "height": 900},
-            },
-            "input": {"key": "Enter"},
-            "evidence": {"eventual_observation_id": "obs"},
-        },
-    ]
-    for row in rows:
-        (tmp_path / "actions" / f"{row['id']}.json").write_text(
-            json.dumps(row), encoding="utf-8"
-        )
-    codes = {issue["code"] for issue in validate_capture(tmp_path)["issues"]}
-    assert "CLICK_TARGET_GENERIC_CONTAINER" in codes
-    assert "KEY_TARGET_ROOT_WINDOW" in codes
-    assert "TEXT_INPUT_TRANSACTION_MISSING" in codes
-    replay = build_replay_preflight(tmp_path)
-    assert replay["ready"] is False
-    assert replay["quality_gate"]["passed"] is False
-    assert replay["summary"]["quality_blockers"] >= 1
-
-
-def test_validator_flags_missing_production_locator_and_raw_link(tmp_path: Path) -> None:
-    import json
-
-    actions = tmp_path / "actions"
-    elements = tmp_path / "elements"
-    actions.mkdir()
-    elements.mkdir()
-    (tmp_path / "observations").mkdir()
-    (tmp_path / "journal").mkdir()
-    (tmp_path / "journal" / "raw_events.jsonl").write_text("{}\n", encoding="utf-8")
-    (elements / "el.json").write_text(
-        json.dumps({"id": "el", "primary_locator": None}), encoding="utf-8"
-    )
-    (actions / "a.json").write_text(
-        json.dumps({
-            "id": "a", "sequence": 1, "surface": "browser",
-            "action_type": "click", "target_element_id": "el",
-            "target_snapshot": {"role": "link", "rect": {"width": 20, "height": 20}},
-            "raw_event": {}, "evidence": {"eventual_observation_id": "obs"},
-        }),
-        encoding="utf-8",
-    )
-    result = validate_capture(tmp_path)
-    codes = {issue["code"] for issue in result["issues"]}
-    assert "ACTION_PRODUCTION_LOCATOR_MISSING" in codes
-    assert "ACTION_RAW_EVENT_LINK_MISSING" in codes
-
-
-def test_validator_flags_graphic_leaf_low_stability_and_split_fill(tmp_path: Path) -> None:
-    import json
-
-    actions = tmp_path / "actions"
-    elements = tmp_path / "elements"
-    actions.mkdir()
-    elements.mkdir()
-    (tmp_path / "observations").mkdir()
-    (tmp_path / "journal").mkdir()
-    (tmp_path / "journal" / "raw_events.jsonl").write_text("{}\n", encoding="utf-8")
-    locator = {
-        "strategy": "css", "expression": "css:.hash", "actual_count": 1,
-        "unique": True, "production_eligible": True, "stability_score": 0.32,
-        "validation": "live",
-    }
-    (elements / "el.json").write_text(
-        json.dumps({"id": "el", "primary_locator": locator}), encoding="utf-8"
-    )
-    rows = [
-        {
-            "id": "a1", "sequence": 1, "surface": "browser", "action_type": "fill",
-            "target_element_id": "el", "target_snapshot": {"tag": "input"},
-            "input": {"text": "a"},
-            "raw_event": {"meta": {"event_id": "doc:1", "source_ts_ms": 1000}},
-            "evidence": {"eventual_observation_id": "obs", "locator_unique": True},
-        },
-        {
-            "id": "a2", "sequence": 2, "surface": "browser", "action_type": "fill",
-            "target_element_id": "el", "target_snapshot": {"tag": "input"},
-            "input": {"text": "ai"},
-            "raw_event": {"meta": {"event_id": "doc:2", "source_ts_ms": 1100}},
-            "evidence": {"eventual_observation_id": "obs", "locator_unique": True},
-        },
-        {
-            "id": "a3", "sequence": 3, "surface": "browser", "action_type": "click",
-            "target_element_id": "el", "target_snapshot": {"tag": "path"},
-            "raw_event": {"meta": {"event_id": "doc:3"}},
-            "evidence": {"eventual_observation_id": "obs", "locator_unique": True},
-        },
-    ]
-    for row in rows:
-        (actions / f"{row['id']}.json").write_text(json.dumps(row), encoding="utf-8")
-    codes = {issue["code"] for issue in validate_capture(tmp_path)["issues"]}
-    assert "DUPLICATE_FILL_TRANSACTION" in codes
-    assert "GRAPHIC_LEAF_TARGET" in codes
-    assert "LOW_STABILITY_LOCATOR" in codes
-
-
-def test_replay_preflight_is_honest_about_event_time_validation(tmp_path: Path) -> None:
-    import json
-
-    (tmp_path / "actions").mkdir()
-    (tmp_path / "elements").mkdir()
-    (tmp_path / "actions" / "a.json").write_text(json.dumps({
-        "id": "a", "sequence": 1, "action_type": "click",
-        "target_element_id": "el", "raw_event_ids": ["doc:1"],
-        "raw_event": {"meta": {"page_id": "page", "document_id": "doc"}},
-    }), encoding="utf-8")
-    (tmp_path / "elements" / "el.json").write_text(json.dumps({
-        "id": "el", "logical_name": "go", "primary_locator": {
-            "strategy": "role", "expression": "role:button,Go",
-            "actual_count": 1, "unique": True, "production_eligible": True,
-            "stability_score": 0.95, "validation": "live",
-        },
-    }), encoding="utf-8")
-    report = build_replay_preflight(tmp_path)
-    assert report["mode"] == "static_non_mutating"
-    assert report["summary"]["pass"] == 1
-    assert report["actions"][0]["verification"] == "event_time_unique"
-
-
-def test_replay_preflight_rejects_unique_root_window_click(tmp_path: Path) -> None:
-    import json
-
-    (tmp_path / "actions").mkdir()
-    (tmp_path / "elements").mkdir()
-    (tmp_path / "actions" / "a.json").write_text(json.dumps({
-        "id": "a", "sequence": 1, "surface": "windows", "action_type": "click",
-        "target_element_id": "root", "target_snapshot": {
-            "control_type": "WindowControl", "name": "WeCom",
-            "bounds": {"width": 1440, "height": 900},
-        },
-    }), encoding="utf-8")
-    (tmp_path / "elements" / "root.json").write_text(json.dumps({
-        "id": "root", "logical_name": "WeCom", "primary_locator": {
-            "strategy": "window_name_type", "expression": "window:WeCom",
-            "actual_count": 1, "unique": True, "production_eligible": True,
-            "stability_score": 0.9, "validation": "snapshot_only",
-        },
-    }), encoding="utf-8")
-    report = build_replay_preflight(tmp_path)
-    assert report["summary"]["fail"] == 1
-    assert report["actions"][0]["reason"] == "oversized_or_root_window_click_target"

tools/ui_inspector/tests/test_locators_catalog.py

@@ -1,215 +0,0 @@
-"""Locator compiler / catalog / uniqueness tests."""
-
-from __future__ import annotations
-
-import sys
-import tempfile
-import unittest
-from pathlib import Path
-
-_HERE = Path(__file__).resolve().parents[1]
-if str(_HERE) not in sys.path:
-    sys.path.insert(0, str(_HERE))
-
-from locators.browser import compile_browser_locators
-from locators.catalog import ElementCatalogService
-from locators.validator import choose_production_primary, validate_locator_against_rows
-from locators.windows import compile_windows_locators
-from models.locator import LocatorCandidate
-
-
-class TestBrowserLocators(unittest.TestCase):
-    def test_ordering_prefers_role_and_testid(self) -> None:
-        el = {
-            "role": "button",
-            "name": "Submit",
-            "test_id": "submit-btn",
-            "text": "Submit",
-            "id": "x1",
-        }
-        rows = [
-            {"role": "button", "name": "Submit", "test_id": "submit-btn", "text": "Submit", "id": "x1"},
-            {"role": "button", "name": "Cancel", "test_id": "cancel-btn", "text": "Cancel"},
-        ]
-        cands = compile_browser_locators(el, rows=rows)
-        strategies = [c.strategy for c in cands]
-        self.assertIn("role", strategies)
-        self.assertIn("test_id", strategies)
-        primary = choose_production_primary(cands)
-        self.assertIsNotNone(primary)
-        assert primary is not None
-        self.assertFalse(primary.uses_coordinates)
-
-    def test_contenteditable_ignores_dynamic_value_and_uses_role_css(self) -> None:
-        cands = compile_browser_locators(
-            {
-                "role": "combobox",
-                "name": "user typed comment",
-                "text": "user typed comment",
-                "value": "user typed comment",
-                "contenteditable": True,
-                "css": "#comment-input [contenteditable=true]",
-            }
-        )
-        role = next(c for c in cands if c.strategy == "role")
-        css = next(c for c in cands if c.strategy == "css")
-        self.assertEqual(role.properties, {"role": "combobox"})
-        self.assertNotIn("user typed comment", role.expression)
-        self.assertFalse(css.production_eligible)
-
-    def test_live_unique_css_becomes_executable_fallback(self) -> None:
-        with tempfile.TemporaryDirectory() as tmp:
-            service = ElementCatalogService(Path(tmp), flow_id="f")
-            el = service.upsert_from_snapshot(
-                {
-                    "role": "path",
-                    "tag": "path",
-                    "css": "#comment-input .submit svg path",
-                    "fingerprint": "path|submit",
-                },
-                surface="browser",
-                frame={"url": "https://example.test/video"},
-                live_counts={"css": 1},
-            )
-            self.assertIsNotNone(el.primary_locator)
-            assert el.primary_locator is not None
-            self.assertEqual(el.primary_locator.strategy, "css")
-            self.assertTrue(el.primary_locator.unique)
-
-    def test_link_href_beats_ambiguous_card_css(self) -> None:
-        with tempfile.TemporaryDirectory() as tmp:
-            service = ElementCatalogService(Path(tmp), flow_id="f")
-            el = service.upsert_from_snapshot(
-                {
-                    "role": "link",
-                    "tag": "a",
-                    "name": "A result card with dynamic text",
-                    "href": "/detail/42",
-                    "href_count": 1,
-                    "css": "#results > li.card > a.card-link",
-                    "css_count": 24,
-                    "fingerprint": "a|card",
-                },
-                surface="browser",
-                frame={"url": "https://example.test/search"},
-                live_counts={"href": 1, "css": 24},
-            )
-            self.assertIsNotNone(el.primary_locator)
-            assert el.primary_locator is not None
-            self.assertEqual(el.primary_locator.strategy, "href")
-            self.assertEqual(el.primary_locator.actual_count, 1)
-            self.assertTrue(el.primary_locator.production_eligible)
-
-    def test_scoped_contenteditable_stable_css_can_be_primary(self) -> None:
-        with tempfile.TemporaryDirectory() as tmp:
-            service = ElementCatalogService(Path(tmp), flow_id="f")
-            el = service.upsert_from_snapshot(
-                {
-                    "role": "combobox",
-                    "tag": "div",
-                    "contenteditable": True,
-                    "stable_css": "#comment-form div[contenteditable]",
-                    "stable_css_count": 1,
-                    "css": "#comment-form > div.hash-a > div.hash-b",
-                    "css_count": 1,
-                    "fingerprint": "div|editor",
-                },
-                surface="browser",
-                frame={"url": "https://example.test/detail"},
-                live_counts={"stable_css": 1, "css": 1},
-            )
-            self.assertIsNotNone(el.primary_locator)
-            assert el.primary_locator is not None
-            self.assertEqual(el.primary_locator.strategy, "stable_css")
-            self.assertGreater(el.primary_locator.stability_score, 0.7)
-
-
-class TestWindowsLocators(unittest.TestCase):
-    def test_ordering_and_name_value_split(self) -> None:
-        el = {
-            "automation_id": "searchedit",
-            "name": "searchedit",
-            "value": "张三",
-            "control_type": "EditControl",
-            "parent_chain": [{"name": "search_layout"}],
-        }
-        rows = [
-            {
-                "automation_id": "searchedit",
-                "name": "searchedit",
-                "value": "张三",
-                "control_type": "EditControl",
-            }
-        ]
-        cands = compile_windows_locators(
-            el, rows=rows, window_scope={"class_name": "WeWorkWindow", "title": "企业微信"}
-        )
-        strategies = [c.strategy for c in cands]
-        self.assertTrue(strategies[0].startswith("window_automation_id"))
-        self.assertIn("window_name_value_type", strategies)
-        primary = choose_production_primary(cands)
-        self.assertIsNotNone(primary)
-        assert primary is not None
-        self.assertEqual(primary.actual_count, 1)
-
-    def test_coordinates_never_primary(self) -> None:
-        el = {
-            "name": "x",
-            "control_type": "ButtonControl",
-            "bounds": {"left": 1, "top": 2, "right": 3, "bottom": 4},
-        }
-        # Two matches → ambiguous for name+type; coords present
-        rows = [
-            {"name": "x", "control_type": "ButtonControl"},
-            {"name": "x", "control_type": "ButtonControl"},
-        ]
-        cands = compile_windows_locators(el, rows=rows)
-        primary = choose_production_primary(cands)
-        self.assertIsNone(primary)
-        self.assertTrue(any(c.uses_coordinates for c in cands))
-        for c in cands:
-            if c.uses_coordinates:
-                self.assertFalse(c.production_eligible)
-
-
-class TestUniqueness(unittest.TestCase):
-    def test_ambiguous_not_primary(self) -> None:
-        c = LocatorCandidate(
-            strategy="name",
-            properties={"name": "item", "control_type": "ButtonControl"},
-            stability_score=0.8,
-        )
-        rows = [
-            {"name": "item", "control_type": "ButtonControl"},
-            {"name": "item", "control_type": "ButtonControl"},
-        ]
-        validate_locator_against_rows(c, rows)
-        self.assertTrue(c.ambiguous)
-        self.assertFalse(c.production_eligible)
-        self.assertIsNone(choose_production_primary([c]))
-
-
-class TestCatalogMerge(unittest.TestCase):
-    def test_merge_observations(self) -> None:
-        with tempfile.TemporaryDirectory() as tmp:
-            svc = ElementCatalogService(Path(tmp), flow_id="f1")
-            el = {
-                "automation_id": "ok",
-                "name": "ok",
-                "control_type": "ButtonControl",
-                "value": "",
-            }
-            rows = [el]
-            a = svc.upsert_from_snapshot(
-                el, surface="windows", window_scope={"class_name": "X"}, rows=rows
-            )
-            b = svc.upsert_from_snapshot(
-                el, surface="windows", window_scope={"class_name": "X"}, rows=rows
-            )
-            self.assertEqual(a.id, b.id)
-            self.assertEqual(b.observation_count, 2)
-            self.assertTrue(svc.catalog_path.exists())
-
-
-if __name__ == "__main__":
-    unittest.main()

tools/ui_inspector/tests/test_models_fingerprint_diff.py

@@ -1,295 +0,0 @@
-"""Phase 2 unit tests: models, fingerprint, dedupe, state diff, redaction."""
-
-from __future__ import annotations
-
-import sys
-import tempfile
-import unittest
-from pathlib import Path
-
-_HERE = Path(__file__).resolve().parents[1]
-if str(_HERE) not in sys.path:
-    sys.path.insert(0, str(_HERE))
-
-from models.action import ActionFlow, ActionRecord
-from models.element import ElementRef, element_fingerprint
-from models.locator import LocatorCandidate
-from models.serialize import dumps_json, to_dict
-from models.state import UIState, compute_state_fingerprint
-from core.policies.element_identity import element_identity_key
-from redaction import redact_input_payload, redact_text
-from snapshot_service import SnapshotService
-from state_diff import diff_states, propose_postconditions
-
-
-class TestSerialization(unittest.TestCase):
-    def test_state_serialize(self) -> None:
-        st = UIState(
-            surface="browser",
-            driver="playwright",
-            url="https://example.com",
-            page_title="Example",
-            locatable=[{"control_type": "button", "name": "Go", "value": ""}],
-        )
-        st.ensure_fingerprint()
-        raw = dumps_json(st)
-        self.assertIn("openclaw.ui_inspector.state.v1", raw)
-        self.assertIn("https://example.com", raw)
-        self.assertNotIn("\\u", raw)  # ensure_ascii=False for CJK; ASCII ok either way
-        d = to_dict(st)
-        self.assertEqual(d["schema"], "openclaw.ui_inspector.state.v1")
-        self.assertTrue(d["fingerprint"])
-
-    def test_action_serialize(self) -> None:
-        act = ActionRecord(
-            action_type="click",
-            surface="windows",
-            before_state_id="st_a",
-            after_state_id="st_b",
-            sequence=1,
-        )
-        d = to_dict(act)
-        self.assertEqual(d["schema"], "openclaw.ui_inspector.action.v1")
-        self.assertEqual(d["capability"], "implemented")
-        flow = ActionFlow(flow_id="f1", flow_name="demo", actions=[act.id])
-        self.assertEqual(to_dict(flow)["schema"], "openclaw.ui_inspector.action_flow.v1")
-
-    def test_element_locator_serialize(self) -> None:
-        loc = LocatorCandidate(
-            strategy="automation_id",
-            properties={"automation_id": "ok", "control_type": "Button"},
-            expression="AutomationId=ok",
-            stability_score=0.9,
-        )
-        loc.mark_counts(1)
-        el = ElementRef(
-            logical_name="ok_button",
-            surface="windows",
-            automation_id="ok",
-            control_type="ButtonControl",
-            locator_candidates=[loc],
-            primary_locator=loc,
-        )
-        el.ensure_fingerprint()
-        d = to_dict(el)
-        self.assertEqual(d["schema"], "openclaw.ui_inspector.element.v1")
-        self.assertTrue(d["primary_locator"]["production_eligible"])
-
-
-class TestFingerprint(unittest.TestCase):
-    def test_stable_fingerprint(self) -> None:
-        kwargs = dict(
-            surface="browser",
-            url="https://a.test/x",
-            page_title="T",
-            window={"title": "T", "class_name": "Chrome", "process_name": "chrome.exe"},
-            locatable=[
-                {"control_type": "textbox", "name": "q", "value": "hello", "automation_id": "q"}
-            ],
-        )
-        a = compute_state_fingerprint(**kwargs)
-        b = compute_state_fingerprint(**kwargs)
-        self.assertEqual(a, b)
-
-    def test_session_fields_ignored(self) -> None:
-        base_loc = [{"control_type": "button", "name": "A", "value": "", "automation_id": "a"}]
-        fp1 = compute_state_fingerprint(
-            surface="windows",
-            window={"title": "App", "class_name": "X", "pid": 111, "hwnd": 222},
-            locatable=base_loc,
-        )
-        fp2 = compute_state_fingerprint(
-            surface="windows",
-            window={"title": "App", "class_name": "X", "pid": 999, "hwnd": 888},
-            locatable=base_loc,
-        )
-        self.assertEqual(fp1, fp2)
-
-    def test_url_changes_fingerprint(self) -> None:
-        loc = [{"control_type": "button", "name": "A", "value": "", "automation_id": "a"}]
-        a = compute_state_fingerprint(surface="browser", url="https://a.test/1", locatable=loc)
-        b = compute_state_fingerprint(surface="browser", url="https://a.test/2", locatable=loc)
-        self.assertNotEqual(a, b)
-
-    def test_anonymous_dom_elements_do_not_merge_across_paths(self) -> None:
-        common = dict(surface="browser", role="span", name="", value="")
-        a = element_fingerprint(**common, dom_fingerprint="span|#video-tab")
-        b = element_fingerprint(**common, dom_fingerprint="span|#comment-input")
-        self.assertNotEqual(a, b)
-
-    def test_contenteditable_value_is_not_identity(self) -> None:
-        common = dict(
-            surface="browser",
-            role="combobox",
-            contenteditable=True,
-            dom_fingerprint="div|#comment-input [contenteditable]",
-        )
-        self.assertEqual(
-            element_fingerprint(**common, name="first", value="first"),
-            element_fingerprint(**common, name="second", value="second"),
-        )
-
-    def test_link_href_is_identity_when_visible_text_changes(self) -> None:
-        common = {
-            "role": "link",
-            "tag": "a",
-            "href": "/detail/42",
-            "css": "#results > li.card > a",
-        }
-        self.assertEqual(
-            element_identity_key(
-                {**common, "name": "Result with 10 likes"},
-                surface="browser",
-                page_url="https://example.test/search?page=1",
-            ),
-            element_identity_key(
-                {**common, "name": "Result with 11 likes"},
-                surface="browser",
-                page_url="https://example.test/search?page=2",
-            ),
-        )
-
-    def test_value_change_detected(self) -> None:
-        a = compute_state_fingerprint(
-            surface="windows",
-            locatable=[{"control_type": "Edit", "name": "box", "value": "1", "automation_id": "e"}],
-        )
-        b = compute_state_fingerprint(
-            surface="windows",
-            locatable=[{"control_type": "Edit", "name": "box", "value": "2", "automation_id": "e"}],
-        )
-        self.assertNotEqual(a, b)
-
-
-class TestDedupe(unittest.TestCase):
-    def test_same_state_deduped(self) -> None:
-        with tempfile.TemporaryDirectory() as tmp:
-            svc = SnapshotService(Path(tmp), persist_evidence=False)
-
-            class Dump:
-                surface = "browser"
-                driver = "playwright"
-                window = {"title": "T", "class_name": "P"}
-                tree = {}
-                index = []
-                locatable = [
-                    {"control_type": "button", "name": "Go", "value": "", "automation_id": "go"}
-                ]
-                stats = {"locatable_count": 1, "nodes_kept": 1}
-                selection = {}
-                url = "https://example.com"
-                page_title = "T"
-                screenshot_bytes = None
-                use_hwnd_screenshot = False
-
-            s1 = svc.capture_from_dump(Dump(), change_reason="entry")
-            s2 = svc.capture_from_dump(Dump(), change_reason="heartbeat")
-            self.assertEqual(s1.id, s2.id)
-            self.assertEqual(s2.observation_count, 2)
-            self.assertEqual(len(svc.state_ids()), 1)
-
-
-class TestStateDiff(unittest.TestCase):
-    def test_diff_and_postconditions(self) -> None:
-        before = {
-            "url": "https://a.test/1",
-            "page_title": "One",
-            "window": {"class_name": "WeWorkWindow", "title": "企业微信", "hwnd": 1},
-            "locatable": [
-                {
-                    "control_type": "Edit",
-                    "name": "searchedit",
-                    "value": "q",
-                    "automation_id": "searchedit",
-                }
-            ],
-            "stats": {"locatable_count": 1},
-        }
-        after = {
-            "url": "https://a.test/1",
-            "page_title": "Chat",
-            "window": {"class_name": "WeWorkWindow", "title": "企业微信", "hwnd": 1},
-            "locatable": [
-                {
-                    "control_type": "Edit",
-                    "name": "send_rich_edit",
-                    "value": "",
-                    "automation_id": "send_rich_edit",
-                }
-            ],
-            "stats": {"locatable_count": 1},
-        }
-        changes = diff_states(before, after)
-        kinds = {c.kind for c in changes}
-        self.assertIn("title_changed", kinds)
-        self.assertIn("element_appeared", kinds)
-        self.assertIn("element_disappeared", kinds)
-        props = propose_postconditions(changes)
-        self.assertTrue(all(p.status == "proposed" for p in props))
-        self.assertTrue(any("appeared" in p.description.lower() or "出现" in p.description or "appeared" in p.description for p in props) or props)
-
-
-class TestRedaction(unittest.TestCase):
-    def test_password_redacted(self) -> None:
-        out, sens = redact_input_payload(
-            {"text": "secret123", "name": "password"},
-            field_name="password",
-            input_type="password",
-        )
-        self.assertTrue(sens)
-        self.assertEqual(out["text"], "[REDACTED]")
-
-    def test_phone_redacted(self) -> None:
-        self.assertIn("[REDACTED]", redact_text("call 13800138000 please"))
-
-    def test_raw_tree_and_index_are_sanitized_before_write(self) -> None:
-        secret = "Bearer_abcdefghijklmnopqrstuvwxyz123456"
-
-        class SensitiveDump:
-            surface = "browser"
-            driver = "playwright"
-            window = {"title": "Login"}
-            tree = {
-                "role": "textbox",
-                "name": "password",
-                "input_type": "password",
-                "value": secret,
-            }
-            index = [dict(tree)]
-            locatable = [dict(tree)]
-            stats = {}
-            selection = {}
-            url = "https://example.com/login"
-            page_title = "Login"
-            screenshot_bytes = None
-            use_hwnd_screenshot = False
-
-        with tempfile.TemporaryDirectory() as tmp:
-            root = Path(tmp)
-            svc = SnapshotService(root)
-            svc.capture_from_dump(
-                SensitiveDump(), change_reason="sensitive", force_persist=True
-            )
-            persisted = "\n".join(
-                p.read_text(encoding="utf-8", errors="ignore")
-                for p in root.rglob("*")
-                if p.is_file()
-            )
-            self.assertNotIn(secret, persisted)
-
-
-class TestActionBeforeAfter(unittest.TestCase):
-    def test_click_links_states(self) -> None:
-        act = ActionRecord(
-            action_type="click",
-            before_state_id="st_before",
-            after_state_id="st_after",
-            target_snapshot={"name": "btn"},
-        )
-        self.assertEqual(act.before_state_id, "st_before")
-        self.assertEqual(act.after_state_id, "st_after")
-        self.assertEqual(act.capability, "implemented")
-
-
-if __name__ == "__main__":
-    unittest.main()

tools/ui_inspector/tests/unit/__init__.py

@@ -1 +0,0 @@
-

tools/ui_inspector/tests/unit/core/__init__.py

@@ -1 +0,0 @@
-

tools/ui_inspector/tests/unit/core/test_event_journal_identity.py

@@ -1,30 +0,0 @@
-import json
-from pathlib import Path
-
-from core.pipeline.event_journal import EventJournal
-from core.policies.element_identity import element_identity_key
-
-
-def test_journal_is_append_only(tmp_path: Path) -> None:
-    journal = EventJournal(tmp_path)
-    assert journal.append({"kind": "pointerdown"}) == 1
-    assert journal.append({"kind": "click"}) == 2
-    rows = [json.loads(line) for line in journal.path.read_text(encoding="utf-8").splitlines()]
-    assert [row["journal_sequence"] for row in rows] == [1, 2]
-    assert [row["event"]["kind"] for row in rows] == ["pointerdown", "click"]
-
-
-def test_focus_classes_and_query_strings_do_not_split_identity() -> None:
-    first = {
-        "role": "textbox", "tag": "input", "placeholder": "Search",
-        "css": "#header input.search",
-    }
-    focused = {
-        **first,
-        "class_name": "search focused hash-123",
-        "css": "#header input.search.focused.hash-123",
-    }
-    one = element_identity_key(first, surface="browser", page_url="https://example.com/home?a=1")
-    two = element_identity_key(focused, surface="browser", page_url="https://example.com/results?q=x")
-    assert one == two
-

tools/ui_inspector/tests/unit/core/test_target_scoring.py

@@ -1,70 +0,0 @@
-from core.policies.target_scoring import resolve_browser_target
-
-
-def test_huge_control_named_ancestor_never_beats_comment_hit() -> None:
-    hit = {
-        "tag": "path", "role": "path", "cursor": "pointer",
-        "rect": {"left": 1047, "top": 545, "right": 1070, "bottom": 568, "width": 23, "height": 23},
-        "css": "#comment-input-container .send > svg > path",
-    }
-    huge = {
-        "tag": "div", "role": "div", "class_name": "playerControlHeight",
-        "rect": {"left": 160, "top": -244, "right": 1440, "bottom": 6698, "width": 1280, "height": 6942},
-        "css": "#root .playerControlHeight",
-    }
-    target, scored = resolve_browser_target(hit, [hit, huge], x=1056, y=558)
-    assert target["tag"] == "path"
-    assert any("oversized_ancestor" in row["reasons"] for row in scored)
-
-
-def test_nearest_semantic_wrapper_beats_svg_path() -> None:
-    hit = {
-        "tag": "path", "role": "path", "cursor": "pointer",
-        "rect": {"left": 10, "top": 10, "right": 30, "bottom": 30, "width": 20, "height": 20},
-    }
-    wrapper = {
-        "tag": "span", "role": "span", "cursor": "pointer", "class_name": "sendControl",
-        "rect": {"left": 5, "top": 5, "right": 35, "bottom": 35, "width": 30, "height": 30},
-    }
-    target, _ = resolve_browser_target(hit, [hit, wrapper], x=20, y=20)
-    assert target["class_name"] == "sendControl"
-
-
-def test_contenteditable_root_beats_transient_inner_block() -> None:
-    inner = {
-        "tag": "div", "role": "textbox", "contenteditable": True,
-        "contenteditable_attr": False,
-        "rect": {"left": 10, "top": 10, "right": 300, "bottom": 35, "width": 290, "height": 25},
-    }
-    root = {
-        "tag": "div", "role": "combobox", "explicit_role": "combobox",
-        "contenteditable": True, "contenteditable_attr": True,
-        "rect": {"left": 10, "top": 10, "right": 300, "bottom": 35, "width": 290, "height": 25},
-    }
-    target, _ = resolve_browser_target(inner, [inner, root], x=20, y=20)
-    assert target["contenteditable_attr"] is True
-
-
-def test_unique_clickable_wrapper_beats_ambiguous_graphic_leaf() -> None:
-    rect_path = {"left": 1047, "top": 345, "right": 1070, "bottom": 368, "width": 23, "height": 23}
-    rect_wrapper = {"left": 1041, "top": 338, "right": 1077, "bottom": 374, "width": 36, "height": 36}
-    path = {
-        "tag": "path", "role": "path", "cursor": "pointer",
-        "rect": rect_path, "css_count": 2,
-    }
-    svg = {
-        "tag": "svg", "role": "svg", "cursor": "pointer",
-        "rect": rect_wrapper, "css_count": 1,
-    }
-    wrapper = {
-        "tag": "span", "role": "span", "cursor": "pointer",
-        "rect": rect_wrapper, "css_count": 1,
-        "stable_css": "#comment-form > div.controls > span",
-        "stable_css_count": 1,
-    }
-    target, scored = resolve_browser_target(
-        path, [path, svg, wrapper], x=1055, y=352
-    )
-    assert target["tag"] == "span"
-    assert any("graphic_leaf" in row["reasons"] for row in scored)
-    assert any("event_time_unique_locator" in row["reasons"] for row in scored)

tools/ui_inspector/tests/unit/test_app_layout.py

@@ -1,120 +0,0 @@
-import json
-import sys
-from pathlib import Path
-
-_HERE = Path(__file__).resolve().parents[2]
-if str(_HERE) not in sys.path:
-    sys.path.insert(0, str(_HERE))
-
-from app import (
-    _DEFAULT_FLOW_NAME_SLUGS,
-    _default_window_geometry,
-    _flow_name_for_mode,
-    _format_geometry,
-    _format_live_action,
-    _merge_current_live_action,
-)
-import artifacts as art
-import target
-import user_prefs
-
-
-def test_default_geometry_is_narrow_tall_and_leaves_title_bar_clear() -> None:
-    width, height, x, y = _default_window_geometry((0, 0, 1920, 1040))
-
-    assert (width, height) == (576, 951)
-    assert (x, y) == (1328, 73)
-    assert y >= 64
-    assert x + width == 1920 - 16
-
-
-def test_default_geometry_supports_small_and_left_side_monitors() -> None:
-    width, height, x, y = _default_window_geometry(
-        (-1366, 0, 0, 728), side="left"
-    )
-
-    assert (width, height) == (560, 648)
-    assert (x, y) == (-1350, 64)
-    assert _format_geometry(width, height, x, y) == "560x648-1350+64"
-
-
-def test_flow_name_defaults_follow_pick_mode() -> None:
-    assert _flow_name_for_mode(target.SURFACE_BROWSER) == "browser"
-    assert _flow_name_for_mode(target.SURFACE_WINDOWS) == "desktop"
-    assert _flow_name_for_mode(target.SURFACE_ANDROID) == "mobile"
-    assert _DEFAULT_FLOW_NAME_SLUGS == frozenset({"browser", "desktop", "mobile"})
-
-
-def test_local_ts_is_compact_yyyymmddhhmmss() -> None:
-    stamp = art._local_ts()
-    assert len(stamp) == 14
-    assert stamp.isdigit()
-    assert "T" not in stamp
-    assert "+" not in stamp
-
-
-def test_create_flow_dir_uses_compact_timestamp(tmp_path: Path) -> None:
-    flow_dir = art.create_flow(tmp_path, flow_name="browser")
-    name = flow_dir.name
-    assert name.endswith("_browser")
-    stamp = name[: -len("_browser")]
-    assert len(stamp) == 14
-    assert stamp.isdigit()
-
-
-def test_hidden_legacy_recording_preference_migrates_to_auto(
-    tmp_path: Path, monkeypatch
-) -> None:
-    prefs_path = tmp_path / "user_prefs.json"
-    prefs_path.write_text(
-        json.dumps(
-            {
-                "mode": "browser",
-                "start_url": "https://example.com",
-                "recording_mode": "legacy_interval",
-            }
-        ),
-        encoding="utf-8",
-    )
-    monkeypatch.setattr(user_prefs, "_PREFS_PATH", prefs_path)
-
-    assert user_prefs.load_prefs()["recording_mode"] == "auto"
-
-
-def test_live_action_formatter_is_surface_neutral_and_two_phase() -> None:
-    text = _format_live_action(
-        {
-            "sequence": 3,
-            "action_type": "click",
-            "phase": "captured",
-            "surface": "browser",
-            "target_name": "提交",
-            "target_type": "button",
-        }
-    )
-    assert "#3 点击" in text
-    assert "browser" in text
-    assert "证据采集中" in text
-    assert "等待异步证据" in text
-
-
-def test_current_live_action_overwrites_without_allowing_late_evidence_to_regress() -> None:
-    first = {"id": "action-1", "sequence": 1, "phase": "captured"}
-    first_evidence = {"id": "action-1", "sequence": 1, "phase": "evidence"}
-    second = {"id": "action-2", "sequence": 2, "phase": "captured"}
-
-    current = _merge_current_live_action(None, first)
-    current = _merge_current_live_action(current, first_evidence)
-    assert current == first_evidence
-
-    current = _merge_current_live_action(current, second)
-    assert current == second
-
-    current = _merge_current_live_action(current, first_evidence)
-    assert current == second
-
-
-def test_current_live_action_ignores_updates_without_an_action_id() -> None:
-    current = {"id": "action-2", "sequence": 2, "phase": "captured"}
-
-    assert _merge_current_live_action(current, {"sequence": 3}) is current

tools/ui_inspector/ui/__init__.py

@@ -1,2 +0,0 @@
-"""User-interface package; legacy app.py remains the compatibility entry."""
-

tools/ui_inspector/ui/controller.py

@@ -1,10 +0,0 @@
-"""Application-facing session facade.
-
-The Tk view remains in app.py during the compatibility migration; new callers
-should depend on this facade instead of constructing recorder internals.
-"""
-
-from flow_session import FlowSession
-
-__all__ = ["FlowSession"]
-

tools/ui_inspector/user_prefs.py

@@ -1,88 +0,0 @@
-"""Persist UI Inspector prefs (pick mode + start URL + recording mode) across runs."""
-
-from __future__ import annotations
-
-import json
-from pathlib import Path
-from typing import Any
-
-import target
-
-_HERE = Path(__file__).resolve().parent
-_PREFS_PATH = _HERE / "user_prefs.json"
-
-DEFAULT_MODE = target.SURFACE_BROWSER
-# 起始 URL：默认空；仅记住用户主动填写过的值
-DEFAULT_START_URL = ""
-DEFAULT_RECORDING_MODE = "auto"
-
-_VALID_MODES = {
-    target.SURFACE_BROWSER,
-    target.SURFACE_WINDOWS,
-    target.SURFACE_ANDROID,  # UI label「手机」
-}
-
-_VALID_RECORDING = {"auto", "guided", "checkpoint", "legacy_interval"}
-
-
-def prefs_path() -> Path:
-    return _PREFS_PATH
-
-
-def load_prefs() -> dict[str, Any]:
-    raw: dict[str, Any] = {}
-    try:
-        if _PREFS_PATH.is_file():
-            data = json.loads(_PREFS_PATH.read_text(encoding="utf-8"))
-            if isinstance(data, dict):
-                raw = data
-    except Exception:
-        raw = {}
-    mode = str(raw.get("mode") or DEFAULT_MODE).strip().lower()
-    if mode in {"mobile", "phone", "ios"}:
-        mode = target.SURFACE_ANDROID
-    if mode not in _VALID_MODES:
-        mode = DEFAULT_MODE
-    # 允许空字符串；未写过 start_url 键时也视为空
-    if "start_url" in raw:
-        start_url = str(raw.get("start_url") or "").strip()
-    else:
-        start_url = DEFAULT_START_URL
-    # v0.14.3 standardizes the public UI on automatic action recording.
-    # Older values remain accepted by lower-level capture APIs, but must not
-    # silently put the simplified UI into a hidden legacy mode.
-    recording_mode = DEFAULT_RECORDING_MODE
-    return {
-        "mode": mode,
-        "start_url": start_url,
-        "recording_mode": recording_mode,
-    }
-
-
-def save_prefs(
-    *,
-    mode: str,
-    start_url: str,
-    recording_mode: str = DEFAULT_RECORDING_MODE,
-) -> None:
-    mode_n = (mode or DEFAULT_MODE).strip().lower()
-    if mode_n in {"mobile", "phone", "ios"}:
-        mode_n = target.SURFACE_ANDROID
-    if mode_n not in _VALID_MODES:
-        mode_n = DEFAULT_MODE
-    url = (start_url or "").strip()
-    rec = (recording_mode or DEFAULT_RECORDING_MODE).strip().lower()
-    if rec not in _VALID_RECORDING:
-        rec = DEFAULT_RECORDING_MODE
-    payload = {
-        "mode": mode_n,
-        "start_url": url,
-        "recording_mode": rec,
-    }
-    try:
-        _PREFS_PATH.write_text(
-            json.dumps(payload, ensure_ascii=False, indent=2) + "\n",
-            encoding="utf-8",
-        )
-    except Exception:
-        pass