chore: archive tools/ui_inspector snapshot before migration removal.

Keep the last in-template copy on main so history can restore it; product continues in jiangchang-ui-recorder.

Co-authored-by: Cursor <cursoragent@cursor.com>

tools/ui_inspector/.gitignore

@@ -1,5 +1,7 @@
 # 仅本工具目录使用；不修改 skill-template 根 .gitignore
 _captures/
+_browser_profile/
+user_prefs.json
 __pycache__/
 *.pyc
 .pytest_cache/

tools/ui_inspector/README.md

@@ -1,14 +1,34 @@
-# OpenClaw UI Inspector（桌面试验版 v0.8.1）
+# OpenClaw UI Inspector（UI Automation Recorder v0.16.0）
 
-> **有焦点的窗口 = 采集目标**（与大小无关）。  
-> 企微暖树：**只调用** `jiangchang_desktop_wake.ensure_ui_ready`（与技能同路径），不复制注入逻辑。  
-> 主窗 → profile `wecom`；搜索浮层 `SearchResultWindow2` → profile `wecom_search`（需 kit≥1.2.6）。
+默认窗口采用窄长控制台布局：停靠到鼠标所在显示器右侧，顶部预留目标应用标题栏安全区；也可在界面中一键切换到左侧。
+
+> **定位**：以人工操作为主线的开发期录制器，产出可交给 AI 编程工具的 Action Flow + Element Catalog。
+> **不是**：每 2 秒无条件 dump 页面的流水线（该行为仅保留为 Legacy 模式）。
+
+## 核心模型
+
+```text
+人工操作
+  → Raw Event Journal（先落事实，不阻塞）
+  → Action Reducer（去重、输入事务、pointerdown/click 合并）
+  → Target Resolver（距离 + 面积 + 语义）
+  → ActionRecord + Element Catalog
+  → 异步合并式 Observation / State / Screenshot
+```
+
+| 概念 | 作用 |
+|------|------|
+| **Action** | 流程主线（click / fill / press_key / checkpoint …） |
+| **State** | 动作前后上下文与证据（tree/index/screenshot） |
+| **Element Catalog** | 开发期与运行期共用的元素资产 |
+| **Locator** | 结构化属性 + 编译后的 expression；须验证匹配数 |
+| **原始 dump** | 诊断证据；不再把每次 heartbeat 登记为流程步骤 |
+| **Raw Event Journal** | 可重算的原始事实；Reducer 出错时无需重走人工流程 |
 
 ## 启动
 
 ```powershell
 cd D:\OpenClaw\client-commons\skill-template
-# 建议使用已安装 jiangchang-platform-kit>=1.2.5 的 Python（与宿主共享 venv 一致）
 python tools/ui_inspector/run.py --scene demo
 ```
 
@@ -16,34 +36,154 @@ python tools/ui_inspector/run.py --scene demo
 
 ```powershell
 pip install -r tools/ui_inspector/requirements.txt
-# 企微自动暖树（强烈建议）
-pip install "jiangchang-platform-kit>=1.2.5"   # 或指向你们 Gitea/本地 index
+pip install "jiangchang-platform-kit>=1.2.11"
 ```
 
-## 规则
+## 录制流程
 
-1. 焦点在谁就采谁（排除 Inspector 自身 hwnd）  
-2. **自动拾取默认勾选**；点「开始流程」后先 **5 秒倒数**，请在此期间点击目标窗  
-3. Inspector **保持置顶**便于看进度；置顶 ≠ 焦点，不因此改采本面板  
-4. **不强制最大化**目标窗，保持操作者当时的大小/位置（仅最小化时会还原）  
-5. 不会因为窗口小 / FindAll 少而改采别的软件  
-6. **企微自动暖树（无感）**：识别到 `WeWorkWindow` 时，在准备/拾取前静默调用  
-   `ensure_ui_ready("wecom")`；已暖则快速返回。结果写入每步 `meta.json` → `dump_stats.kit_wake`
+主界面固定使用自动动作录制，只保留 **开始 / 暂停（继续）/ 结束**。Recorder 立即记录 Action，完整证据由独立后台队列异步合并采集。Inspector 自身窗口上的点击和输入不会写入目标 Action Flow。
 
-## 空壳说明
+“运行状态与拾取结果”会对浏览器和桌面统一采用单步骤覆盖式显示；新动作替换上一动作，同一动作的异步证据返回后在原位置更新定位器、唯一性或失败原因。迟到的旧动作证据不会覆盖更新的步骤，完整历史始终保存在 `action_flow.json`、`actions/` 与 `element_catalog.json` 中。
 
-- **已装 kit**：冷企微应被 n2-wake 暖到 `findall` 较大、`provider_cold=false`。若仍冷，看状态栏 `kit_wake: FAILED …`。  
-- **未装 kit**：行为与旧版相同，可能 `findall≈2`、`provider_cold=true`；状态栏会提示未安装 platform-kit。
+引导、手动检查点和 Legacy 间隔 dump 仅作为底层兼容与开发诊断能力保留，不进入普通用户流程；旧偏好会自动迁移回自动录制。
 
-## 验收建议（企微）
+## 运行时稳定性
 
-1. 打开并登录桌面企业微信  
-2. 启动 Inspector →「开始流程」→ 倒数内点到企微主窗  
-3. 看详情区 / 第一步 `meta.json`：`kit_wake.warm=true`，`findall` 通常 ≫ 80  
-4. 再搜联系人、开会话，确认 `groupbuddyname` 的 Value 能采到  
+- Tk 主线程只处理界面；导航、Dump、截图、动作落盘均在后台执行。
+- 后台结果通过 UI 消息队列回传，后台线程不读取 Tk Variable 或 Widget。
+- 原始事件先追加到 `journal/raw_events.jsonl`，再进入语义 Action FIFO。
+- Action 热路径不执行完整页面 dump；慢速证据在独立队列按页面合并采集。
+- 异步证据标记为 `coalesced_eventual`，不会伪装成事件时刻的精确 after state。
+- Playwright 命令均有超时；导航失败不会被伪装成“已就绪”。
+- 浏览器使用 context 级事件桥并保留轮询回退，整页跳转前的 click 不随旧 document 丢失。
 
-## 冒烟（不弹 GUI）
+## 拾取通道
+
+- **浏览器**：Inspector 自管 Playwright；起始 URL 可选；Browser Recorder（捕获阶段监听，不 preventDefault）
+  - 动作后的 Dump / 截图绑定事件来源 `page_id`；手动检查点才按当前可见标签页回退，不再固定 `pages[0]`
+  - 列表页新开详情标签后，切到详情再 dump/录制即可；关闭详情回到列表同理
+  - Recorder 会在所有已打开标签上安装监听并汇总事件
+  - 输入框、`contenteditable`、SVG/图标点击会沿 `composedPath` 提升到真实可操作目标；原始命中节点仍保留作诊断证据
+  - 语义定位不足时生成完整 CSS 路径，并在事件来源页面实时验证唯一性后才进入 Locator Catalog
+  - 链接保留 `href`，可编辑区生成稳定祖先作用域选择器；事件时刻的唯一性计数随快照保存，避免跳转后误判
+  - `press_key` 保留原始事件、Page/Document 身份和目标元素引用，不再依赖前一步推断焦点
+  - 页面跳转采用短暂因果重排，确保触发跳转的 click 先于 navigation；新标签同时保留 opener `page_id`
+  - 同时监听 `pointerdown` 与 `click`：正常 click 会去重；页面提前卸载时 pointerdown 作为保底 Action
+  - 通用目标评分限制祖先距离与面积，不会把小图标提升成整页容器
+  - 目标解析联合考虑语义、几何、事件时刻唯一性和定位器稳定性；SVG/path 仅作命中证据，优先选择唯一的可点击包装节点
+  - 输入事务与 Element Catalog 共用稳定元素身份，焦点/校验/悬停 class 变化不会拆分一次连续输入
+  - 输入事务使用浏览器源时间并保存全部 `raw_event_ids`，支持从原始日志审计和重算
+  - 完整 Dump / Screenshot 由后台 Evidence Coordinator 合并执行；手动检查点仍保留稳定采集
+- **桌面**：UIA；Windows Recorder 使用进程外 `WH_MOUSE_LL` / `WH_KEYBOARD_LL` 事件监听，点击后由独立线程执行 `ControlFromPoint`。Hook 不注入目标进程；`GetAsyncKeyState` 仅作为 Hook 无法安装时的降级路径。
+  - 输入使用焦点控件 Value/Text 变化聚合为一次 `fill`，不保存中文输入法的拼音按键；混合 DPI 应用会在键盘阶段重新探测正确坐标空间。
+  - 已注册的桌面适配器会在宣布“录制就绪”前完成必要的 Accessibility warm；企业微信首次启动可能多等十几秒，但不会再让首个输入事务与后台暖树竞争。
+  - 输入法候选/组合窗口只作为临时系统 UI 忽略，不会替换业务输入框；编辑器内的 Backspace/Delete 合并进最终 `fill`。
+  - `Enter`、`Tab` 等功能键优先继承刚才的文本编辑器目标，不再退化为整个根窗口。
+  - 鼠标按下时会枚举光标下可见顶层窗口，处理 owner window 吞掉浮窗命中的应用；后置树再把搜索/列表大容器提升到坐标命中的具体行。
+  - 超过 5 秒的延迟证据不能用动态消息摘要/预览改写旧点击；坐标不可靠且只有一个可见结果行时，才使用唯一结果行作为保守回退。
+  - 原始键盘活动只写入不含内容的事务标记；若发生输入但没有生成 `fill/type`，质量检查会明确告警。
+- **手机**：未实现，不得宣称已支持
+
+## 输出目录（双写兼容）
+
+```text
+_captures/<ts>_<flow>/
+  flow.json              # v1 schema；可选 v2 指针字段
+  SUMMARY.md
+  action_flow.json       # v2
+  element_catalog.json   # v2
+  pages.json             # Page / Document 记录
+  journal/raw_events.jsonl
+  diagnostics/metrics.json
+  diagnostics/validation.json
+  diagnostics/replay_report.json  # 静态、非破坏性的回放就绪预检
+  steps/NN_xxx/          # v1：meta/tree/index/locatable/screen/SUMMARY
+  states/<id>/           # v2：证据
+  observations/<id>/     # v2：每次真实采集证据（不去重）
+  actions/<id>.json
+  elements/<id>.json
+```
+
+旧读取方只认 `flow.json` + `steps/` 仍可工作。`flow.json` 中的 `v2` 字段可忽略。
+
+### Schema 名
+
+- `openclaw.ui_inspector.state.v1`
+- `openclaw.ui_inspector.observation.v1`
+- `openclaw.ui_inspector.action.v1`
+- `openclaw.ui_inspector.element.v1`
+- `openclaw.ui_inspector.element_catalog.v1`
+- `openclaw.ui_inspector.action_flow.v1`
+- `openclaw.ui_inspector.flow.v1`（原有）
+
+## 交给 AI 编程工具
+
+优先提供：
+
+1. `action_flow.json` — 动作序列、目标和证据关联；异步证据不冒充精确 after state
+2. `element_catalog.json` — logical_name、结构化 locator 候选、唯一性结果
+3. 需要排障时再附对应 `states/<id>/` 或旧 `steps/` 下的 tree/locatable/screen
+
+不要只丢一堆无动作语义的定时 dump。
+
+`diagnostics.capture_validator` 会报告缺失的生产级定位器、目标元素引用、原始事件关联、根窗口按键目标、列表大容器点击及丢失的输入事务，避免“动作录到了”被误判为“可以稳定生成代码”。
+
+## 当前能力边界
+
+| 能力 | 状态 |
+|------|------|
+| click | 已实现（browser + windows） |
+| fill/type 聚合 | 已实现 |
+| press_key（Enter/Tab/Esc 等） | 已实现 |
+| checkpoint | 已实现 |
+| navigation | 部分（URL 变化观察） |
+| 其余动作类型 | Schema 预留，`capability=schema_only` |
+| 手机录制 | **未实现** |
+| 截图像素级脱敏 | 未实现（敏感页在 meta 标记 `may_contain_sensitive_data`） |
+
+## 敏感信息
+
+- 密码框 / token / cookie / 手机号 / 邮箱等会脱敏，不写入 action_flow / catalog 明文。
+- 截图无法可靠脱敏时标记风险；可用配置关闭敏感页截图保存。
+- **禁止**在测试或产物中使用真实客户数据。
+
+## WeCom Adapter
+
+企业微信专用规则在 `adapters/wecom.py`：
+
+- `WeWorkWindow` → profile `wecom`
+- `SearchResultWindow2` → profile `wecom_search`
+- Warm **cache key = profile + hwnd**，主窗 warm 不会假装搜索浮层已暖
+- Recorder **捕获点击目标前不执行全量 warm**；warm 属于 dump/checkpoint/live 验证路径
+
+## Legacy 间隔模式
+
+Legacy 能力仅供底层兼容与开发排障，不在标准界面暴露。
+默认动作驱动模式下，heartbeat 只做输入聚合刷新，**相同 fingerprint 不新增 State，不产生 Action**。
+
+## 冒烟与单测
 
 ```powershell
 python tools/ui_inspector/smoke_check.py
+python -m pytest tools/ui_inspector/tests -q
 ```
+
+## 目录结构
+
+```text
+ui_inspector/
+  core/                  # 三端共用模型、管线、策略
+  surfaces/              # browser / windows / future mobile
+  evidence/              # 异步证据协调
+  locators/              # Locator Catalog 与稳定性
+  storage/               # 新旧采集格式读写
+  diagnostics/           # 指标与通用质量检查
+  compatibility/         # v1/legacy 兼容入口
+  ui/                    # 新 UI facade；app.py 暂保留为 Tk 兼容入口
+  adapters/              # 可选产品增强，不参与核心正确性
+  tests/                 # unit / contract / integration / e2e
+```
+
+`models/`、`recorder/`、`drivers/` 等旧路径暂时保留为兼容实现；新入口已从 `surfaces/`、`core/` 和 `storage/` 访问，后续可按模块逐步内收而不破坏现有技能。
+
+暖树：`ensure_ui_ready`（kit≥1.2.6）；整页 dump：kit≥1.2.11。

tools/ui_inspector/__init__.py

@@ -1,3 +1,3 @@
-"""Desktop UI Inspector — development-only UIA capture tool (additive, disposable)."""
+"""UI Inspector — development-only multi-surface capture (windows UIA + browser Playwright)."""
 
-__version__ = "0.8.1"
+__version__ = "0.16.0"

tools/ui_inspector/adapters/__init__.py

@@ -0,0 +1,7 @@
+"""App adapters registry (WeCom-specific rules live here, not in generic layers)."""
+
+from .base import AppAdapter
+from .registry import get_adapter, register_adapter
+from . import wecom as _wecom  # noqa: F401 — register on import
+
+__all__ = ["AppAdapter", "get_adapter", "register_adapter"]

tools/ui_inspector/adapters/base.py

@@ -0,0 +1,17 @@
+"""Adapter base protocol."""
+
+from __future__ import annotations
+
+from typing import Any, Optional, Protocol
+
+
+class AppAdapter(Protocol):
+    name: str
+
+    def matches(self, *, class_name: str = "", title: str = "", process_name: str = "") -> bool: ...
+
+    def wake_profile(self, *, class_name: str = "", title: str = "") -> Optional[str]: ...
+
+    def cache_key(self, *, profile: str, hwnd: int = 0) -> str: ...
+
+    def propose_postconditions(self, changes: list[dict[str, Any]]) -> list[dict[str, Any]]: ...

tools/ui_inspector/adapters/registry.py

@@ -0,0 +1,31 @@
+"""Adapter registry."""
+
+from __future__ import annotations
+
+from typing import Any, Optional
+
+_REGISTRY: list[Any] = []
+
+
+def register_adapter(adapter: Any) -> Any:
+    _REGISTRY.append(adapter)
+    return adapter
+
+
+def get_adapter(
+    *,
+    class_name: str = "",
+    title: str = "",
+    process_name: str = "",
+) -> Optional[Any]:
+    for ad in _REGISTRY:
+        try:
+            if ad.matches(class_name=class_name, title=title, process_name=process_name):
+                return ad
+        except Exception:
+            continue
+    return None
+
+
+def all_adapters() -> list[Any]:
+    return list(_REGISTRY)

tools/ui_inspector/adapters/wecom.py

@@ -0,0 +1,103 @@
+"""WeCom (企业微信) adapter — profile / window / postcondition helpers."""
+
+from __future__ import annotations
+
+from typing import Any, Optional
+
+from .registry import register_adapter
+
+PROFILE_WECOM = "wecom"
+PROFILE_WECOM_SEARCH = "wecom_search"
+
+_THRESHOLDS = {
+    PROFILE_WECOM: 80,
+    PROFILE_WECOM_SEARCH: 20,
+}
+
+
+class WeComAdapter:
+    name = "wecom"
+
+    def matches(
+        self, *, class_name: str = "", title: str = "", process_name: str = ""
+    ) -> bool:
+        return self.wake_profile(class_name=class_name, title=title) is not None or (
+            "WXWork" in (process_name or "")
+        )
+
+    def wake_profile(self, *, class_name: str = "", title: str = "") -> Optional[str]:
+        cls = (class_name or "").strip()
+        title = title or ""
+        if cls == "SearchResultWindow2":
+            return PROFILE_WECOM_SEARCH
+        if cls == "WeWorkWindow":
+            return PROFILE_WECOM
+        if "WeWork" in cls and (
+            "企业微信" in title or "WeCom" in title or "wecom" in title.lower()
+        ):
+            return PROFILE_WECOM
+        return None
+
+    def cache_key(self, *, profile: str, hwnd: int = 0) -> str:
+        """Separate main window vs SearchResultWindow2 warm caches."""
+        return f"{profile}:hwnd:{int(hwnd or 0)}"
+
+    def warm_threshold(self, profile: str) -> int:
+        return int(_THRESHOLDS.get(profile, 80))
+
+    def is_search_popup(self, *, class_name: str = "") -> bool:
+        return (class_name or "") == "SearchResultWindow2"
+
+    def is_main_window(self, *, class_name: str = "") -> bool:
+        return (class_name or "") == "WeWorkWindow"
+
+    def known_ids(self) -> dict[str, str]:
+        return {
+            "searchedit": "search_box",
+            "groupbuddyname": "contact_row_name",
+            "search_name": "search_result_name",
+            "send_rich_edit": "compose_box",
+            "panel_title_label": "panel_title",
+        }
+
+    def propose_postconditions(self, changes: list[dict[str, Any]]) -> list[dict[str, Any]]:
+        props: list[dict[str, Any]] = []
+        for ch in changes or []:
+            kind = ch.get("kind")
+            detail = ch.get("detail") or {}
+            if kind == "popup_disappeared" and "SearchResult" in str(
+                detail.get("class_name") or ""
+            ):
+                props.append(
+                    {
+                        "description": "SearchResultWindow2 disappeared",
+                        "kind": "window_absent",
+                        "status": "proposed",
+                        "detail": detail,
+                    }
+                )
+            if kind == "element_appeared":
+                name = str(detail.get("name") or detail.get("automation_id") or "")
+                if name in {"send_rich_edit", "main_chattext", "panel_title_label"}:
+                    props.append(
+                        {
+                            "description": f"WeCom element {name} appeared",
+                            "kind": "element_present",
+                            "status": "proposed",
+                            "detail": detail,
+                        }
+                    )
+            if kind == "title_changed":
+                props.append(
+                    {
+                        "description": f"Chat title became {detail.get('to')!r}",
+                        "kind": "title_equals",
+                        "status": "proposed",
+                        "detail": detail,
+                    }
+                )
+        return props
+
+
+wecom_adapter = WeComAdapter()
+register_adapter(wecom_adapter)

tools/ui_inspector/artifacts.py

@@ -21,14 +21,14 @@ except ImportError:
         _mod = importlib.util.module_from_spec(_spec)  # type: ignore[arg-type]
         assert _spec and _spec.loader
         _spec.loader.exec_module(_mod)
-        __version__ = getattr(_mod, "__version__", "0.8.0")
+        __version__ = getattr(_mod, "__version__", "0.16.0")
     except Exception:
-        __version__ = "0.8.0"
+        __version__ = "0.16.0"
 
 
 def _local_ts() -> str:
-    now = datetime.now().astimezone()
-    return now.strftime("%Y%m%dT%H%M%S%z")
+    """Local wall-clock stamp for capture folder names: YYYYMMDDHHMMSS."""
+    return datetime.now().astimezone().strftime("%Y%m%d%H%M%S")
 
 
 def safe_slug(text: str, *, fallback: str = "session") -> str:
@@ -405,11 +405,14 @@ def build_meta(
     scene: str,
     skill_root: Optional[str] = None,
     extra: Optional[dict[str, Any]] = None,
+    surface: str = "windows",
+    driver: str = "uiautomation",
+    url: str = "",
 ) -> dict[str, Any]:
     now = datetime.now().astimezone()
     meta = {
         "schema": "openclaw.ui_inspector.capture.v1",
-        "surface": "windows",
+        "surface": surface or "windows",
         "captured_at": now.isoformat(),
         "scene": scene,
         "skill_root": skill_root,
@@ -419,6 +422,7 @@ def build_meta(
             "process_name": window.get("process_name"),
             "pid": window.get("pid"),
             "hwnd": window.get("hwnd"),
+            "url": url or window.get("url") or "",
         },
         "environment": {
             "os": "windows",
@@ -427,7 +431,7 @@ def build_meta(
         "tool": {
             "name": "openclaw-ui-inspector",
             "version": __version__,
-            "driver": "uiautomation",
+            "driver": driver or "uiautomation",
         },
     }
     if extra:
@@ -486,14 +490,18 @@ def capture_window_png(hwnd: int, dest: Path) -> bool:
 def create_flow(
     captures_root: Path,
     *,
-    flow_name: str = "desktop-flow",
+    flow_name: str = "desktop",
+    recording_mode: str = "auto",
 ) -> Path:
-    """开始流程：新建一份流程文档目录。"""
+    """开始流程：新建一份流程文档目录（v1 steps + 可选 v2 资产目录）。"""
     captures_root.mkdir(parents=True, exist_ok=True)
     slug = safe_slug(flow_name, fallback="flow")
     flow_dir = captures_root / f"{_local_ts()}_{slug}"
     flow_dir.mkdir(parents=True, exist_ok=False)
     (flow_dir / "steps").mkdir(parents=True, exist_ok=True)
+    # v2 dual-write directories (empty until recorder writes)
+    for sub in ("states", "observations", "actions", "elements", "journal", "diagnostics"):
+        (flow_dir / sub).mkdir(parents=True, exist_ok=True)
     now = datetime.now().astimezone().isoformat()
     flow = {
         "schema": "openclaw.ui_inspector.flow.v1",
@@ -508,6 +516,21 @@ def create_flow(
             "name": "openclaw-ui-inspector",
             "version": __version__,
         },
+        # Optional v2 pointers — old readers ignore unknown fields.
+        "v2": {
+            "recording_mode": recording_mode or "auto",
+            "action_flow": "action_flow.json",
+            "element_catalog": "element_catalog.json",
+            "states_dir": "states/",
+            "observations_dir": "observations/",
+            "actions_dir": "actions/",
+            "elements_dir": "elements/",
+            "raw_event_journal": "journal/raw_events.jsonl",
+            "pages": "pages.json",
+            "diagnostics_dir": "diagnostics/",
+            "validation": "diagnostics/validation.json",
+            "replay_report": "diagnostics/replay_report.json",
+        },
     }
     write_json(flow_dir / "flow.json", flow)
     write_flow_summary(flow_dir, flow)
@@ -565,12 +588,16 @@ def register_step(
     window: dict[str, Any],
     stats: dict[str, Any],
     label: Optional[str] = None,
+    surface: str = "windows",
+    url: str = "",
 ) -> dict[str, Any]:
     entry = {
         "index": step_index,
         "name": step_name,
         "label": label or step_name,
         "captured_at": datetime.now().astimezone().isoformat(),
+        "surface": surface or "windows",
+        "url": url or window.get("url") or "",
         "window_title": window.get("title"),
         "window_class": window.get("class_name"),
         "pid": window.get("pid"),
@@ -621,6 +648,9 @@ def write_flow_summary(flow_dir: Path, flow: dict[str, Any]) -> None:
             lines.append("")
             if s.get("label"):
                 lines.append(f"- label: `{s.get('label')}`")
+            lines.append(f"- surface: `{s.get('surface') or 'windows'}`")
+            if s.get("url"):
+                lines.append(f"- url: `{s.get('url')}`")
             lines.append(
                 f"- window: `{s.get('window_title')}` / `{s.get('window_class')}`"
             )
@@ -636,7 +666,10 @@ def write_flow_summary(flow_dir: Path, flow: dict[str, Any]) -> None:
             "",
             "- 一个流程对应本目录一份文档（`flow.json` + `SUMMARY.md` + `steps/`）。",
             "- 每拾取一页，在 `steps/` 下新增一步；给 AI 时优先读本 `SUMMARY.md` 与各步 `locatable.json`。",
-            "- Name ≠ 界面中文时请看 Value。",
+            "- `surface=windows` 为桌面 UIA；`surface=browser` 为 Playwright 整页 dump。",
+            "- Name ≠ 界面中文时请看 Value；浏览器步优先看 `primary_locator` / `playwright_candidates`。",
+            "- v2（若存在）：`action_flow.json` + `element_catalog.json` + `states/` + `actions/` + `elements/`。",
+            "- 给 AI 编程工具时优先提供 `action_flow.json` 与 `element_catalog.json`，诊断再用 steps/ 原始树。",
             "",
         ]
     )
@@ -651,13 +684,18 @@ def write_step_summary(
     noteworthy: Optional[list[dict[str, Any]]] = None,
 ) -> None:
     app = meta.get("app") or {}
+    tool = meta.get("tool") or {}
     lines = [
         "# Step Summary",
         "",
         f"- captured_at: `{meta.get('captured_at')}`",
+        f"- surface: `{meta.get('surface')}` driver: `{tool.get('driver')}`",
         f"- window: `{app.get('title')}` / `{app.get('class_name')}` pid=`{app.get('pid')}`",
         "",
     ]
+    if app.get("url"):
+        lines.append(f"- url: `{app.get('url')}`")
+        lines.append("")
     if stats:
         lines.extend(
             [
@@ -673,9 +711,11 @@ def write_step_summary(
     if noteworthy:
         lines.extend(["## Notable", ""])
         for item in noteworthy[:40]:
+            primary = (item.get("primary_locator") or {}).get("expr") or ""
+            extra = f" locator=`{primary}`" if primary else ""
             lines.append(
                 f"- `{item.get('control_type')}` Name=`{item.get('name')}` "
-                f"Value=`{item.get('value')}`"
+                f"Value=`{item.get('value') or item.get('text')}`{extra}"
             )
         lines.append("")
     lines.extend(
@@ -689,6 +729,7 @@ def write_step_summary(
     (step_dir / "SUMMARY.md").write_text("\n".join(lines), encoding="utf-8")
 
 
+
 # Back-compat aliases used by older smoke scripts
 def create_session_dir(captures_root: Path, *, scene: str = "desktop") -> Path:
     return create_flow(captures_root, flow_name=scene)

tools/ui_inspector/compatibility/__init__.py

@@ -0,0 +1,2 @@
+"""Backward-compatible readers and entry points."""
+

tools/ui_inspector/compatibility/v1_reader.py

@@ -0,0 +1,6 @@
+"""Legacy flow.json reader."""
+
+from artifacts import load_flow
+
+__all__ = ["load_flow"]
+

tools/ui_inspector/compatibility/v1_writer.py

@@ -0,0 +1,6 @@
+"""Legacy steps/ writer exports."""
+
+from artifacts import add_step_dir, create_flow, save_flow
+
+__all__ = ["add_step_dir", "create_flow", "save_flow"]
+

tools/ui_inspector/core/__init__.py

@@ -0,0 +1,2 @@
+"""Surface-independent UI Inspector domain and recording pipeline."""
+

tools/ui_inspector/core/models/__init__.py

@@ -0,0 +1,16 @@
+"""Canonical model exports.
+
+The compatibility ``models`` package remains importable while callers migrate
+to this namespace.
+"""
+
+from models.action import ActionFlow, ActionRecord
+from models.element import ElementCatalog, ElementRef
+from models.locator import LocatorCandidate
+from models.observation import UIObservation
+from models.state import UIState
+
+__all__ = [
+    "ActionFlow", "ActionRecord", "ElementCatalog", "ElementRef",
+    "LocatorCandidate", "UIObservation", "UIState",
+]

tools/ui_inspector/core/pipeline/__init__.py

@@ -0,0 +1,6 @@
+"""Shared event, action, and evidence pipeline."""
+
+from .event_journal import EventJournal
+
+__all__ = ["EventJournal"]
+

tools/ui_inspector/core/pipeline/event_journal.py

@@ -0,0 +1,49 @@
+"""Append-only raw interaction journal.
+
+Raw events are facts.  They are persisted before semantic reduction so a
+future reducer can rebuild Action Flow without asking the user to repeat the
+workflow.
+"""
+
+from __future__ import annotations
+
+import json
+import threading
+from dataclasses import asdict, is_dataclass
+from datetime import datetime
+from pathlib import Path
+from typing import Any
+
+from redaction import sanitize_artifact
+
+
+class EventJournal:
+    def __init__(self, flow_dir: Path) -> None:
+        self.path = Path(flow_dir) / "journal" / "raw_events.jsonl"
+        self.path.parent.mkdir(parents=True, exist_ok=True)
+        self._lock = threading.Lock()
+        self._sequence = 0
+
+    def append(self, event: Any, *, channel: str = "surface") -> int:
+        if is_dataclass(event) and not isinstance(event, type):
+            payload = asdict(event)
+        elif isinstance(event, dict):
+            payload = dict(event)
+        else:
+            payload = dict(getattr(event, "__dict__", {}) or {})
+        with self._lock:
+            self._sequence += 1
+            record = {
+                "journal_sequence": self._sequence,
+                "received_at": datetime.now().astimezone().isoformat(),
+                "channel": channel,
+                "event": sanitize_artifact(payload),
+            }
+            line = json.dumps(
+                record, ensure_ascii=False, sort_keys=True, separators=(",", ":")
+            )
+            with self.path.open("a", encoding="utf-8", newline="\n") as stream:
+                stream.write(line + "\n")
+                stream.flush()
+            return self._sequence
+

tools/ui_inspector/core/policies/__init__.py

@@ -0,0 +1,7 @@
+"""Surface-independent decision policies."""
+
+from .element_identity import element_identity_key
+from .target_scoring import resolve_browser_target
+
+__all__ = ["element_identity_key", "resolve_browser_target"]
+

tools/ui_inspector/core/policies/element_identity.py

@@ -0,0 +1,62 @@
+"""Stable element identity independent of transient value/focus classes."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import re
+from typing import Any
+from urllib.parse import urlsplit
+
+
+def _origin(url: str) -> str:
+    try:
+        parts = urlsplit(url or "")
+        return f"{parts.scheme}://{parts.netloc}" if parts.netloc else ""
+    except Exception:
+        return ""
+
+
+def _structural_css(css: str) -> str:
+    # Classes frequently encode focus/hover/build hashes.  Keep tag depth and
+    # stable IDs; that distinguishes nested anonymous nodes without freezing a
+    # generated class list into identity.
+    parts = []
+    for raw in (css or "").split(">"):
+        part = raw.strip()
+        tag_match = re.match(r"^[A-Za-z][\w-]*", part)
+        id_match = re.search(r"#[A-Za-z0-9_-]+", part)
+        tag = tag_match.group(0).lower() if tag_match else "*"
+        parts.append(tag + (id_match.group(0) if id_match else ""))
+    return ">".join(parts)
+
+
+def element_identity_key(
+    element: dict[str, Any], *, surface: str, page_url: str = "", frame_name: str = ""
+) -> str:
+    el = dict(element or {})
+    role = str(el.get("control_type") or el.get("role") or el.get("tag") or "")
+    anchor = ""
+    for key in (
+        "test_id", "automation_id", "id", "name_attr", "href",
+        "placeholder", "aria_label", "label",
+    ):
+        value = str(el.get(key) or "").strip()
+        if value:
+            anchor = f"{key}:{value}"
+            break
+    if not anchor and not el.get("contenteditable"):
+        name = str(el.get("semantic_name") or el.get("name") or "").strip()
+        if name and len(name) <= 96:
+            anchor = f"name:{name}"
+    if not anchor:
+        anchor = f"structure:{_structural_css(str(el.get('css') or ''))}"
+    payload = {
+        "surface": surface or "",
+        "origin": _origin(page_url) if surface == "browser" else "",
+        "frame": frame_name or "",
+        "role": role.lower(),
+        "anchor": anchor,
+    }
+    raw = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
+    return hashlib.sha256(raw.encode("utf-8")).hexdigest()

tools/ui_inspector/core/policies/target_scoring.py

@@ -0,0 +1,145 @@
+"""Distance- and geometry-aware target selection.
+
+The resolver never relies on a product selector.  It ranks the composed path
+using native semantics, explicit accessibility metadata, distance from the hit
+node, and geometry.  Huge ancestors are evidence containers, not controls.
+"""
+
+from __future__ import annotations
+
+import re
+from typing import Any, Iterable
+
+_NATIVE_TAGS = {"button", "a", "input", "textarea", "select", "summary", "label"}
+_INTERACTIVE_ROLES = {
+    "button", "link", "textbox", "combobox", "checkbox", "radio",
+    "menuitem", "option", "tab", "switch", "slider", "spinbutton",
+}
+_ACTION_CLASS_TOKENS = {
+    "button", "btn", "submit", "send", "publish", "post", "play",
+    "pause", "action", "click", "toggle", "menu", "option", "tab",
+}
+_GRAPHIC_LEAF_TAGS = {"path", "use", "g", "svg"}
+
+
+def _tokens(value: str) -> set[str]:
+    value = re.sub(r"([a-z0-9])([A-Z])", r"\1 \2", value or "")
+    return {x.lower() for x in re.split(r"[^A-Za-z0-9]+", value) if x}
+
+
+def _area(rect: dict[str, Any]) -> float:
+    try:
+        return max(0.0, float(rect.get("width") or 0)) * max(
+            0.0, float(rect.get("height") or 0)
+        )
+    except (TypeError, ValueError):
+        return 0.0
+
+
+def _contains(
+    rect: dict[str, Any], x: int | None, y: int | None, *, slop: float = 12.0
+) -> bool:
+    if x is None or y is None or not rect:
+        return True
+    try:
+        return (
+            float(rect.get("left")) - slop <= x <= float(rect.get("right")) + slop
+            and float(rect.get("top")) - slop <= y <= float(rect.get("bottom")) + slop
+        )
+    except (TypeError, ValueError):
+        return True
+
+
+def resolve_browser_target(
+    hit_target: dict[str, Any],
+    candidates: Iterable[dict[str, Any]],
+    *,
+    x: int | None = None,
+    y: int | None = None,
+) -> tuple[dict[str, Any], list[dict[str, Any]]]:
+    """Return the best actionable node and an auditable scored candidate list."""
+    chain = [dict(item or {}) for item in candidates if isinstance(item, dict)]
+    if not chain:
+        chain = [dict(hit_target or {})]
+    hit = dict(hit_target or chain[0] or {})
+    hit_area = max(1.0, _area(hit.get("rect") or {}))
+    scored: list[dict[str, Any]] = []
+
+    for depth, item in enumerate(chain):
+        tag = str(item.get("tag") or "").lower()
+        role = str(item.get("role") or "").lower()
+        explicit_role = str(item.get("explicit_role") or "").lower()
+        rect = dict(item.get("rect") or {})
+        area = _area(rect)
+        score = 24.0 if depth == 0 else 0.0
+        reasons: list[str] = [f"depth={depth}"]
+
+        if tag in _NATIVE_TAGS:
+            score = max(score, 100.0)
+            reasons.append("native_control")
+        if item.get("contenteditable_attr"):
+            score = max(score, 99.0)
+            reasons.append("editable_root")
+        elif item.get("contenteditable"):
+            score = max(score, 82.0)
+            reasons.append("editable_descendant")
+        if explicit_role in _INTERACTIVE_ROLES:
+            score = max(score, 97.0)
+            reasons.append("explicit_interactive_role")
+        elif role in _INTERACTIVE_ROLES:
+            score = max(score, 88.0)
+            reasons.append("computed_interactive_role")
+        if item.get("test_id") or item.get("aria_label"):
+            score = max(score, 91.0)
+            reasons.append("stable_semantic_attribute")
+        if item.get("tabindex") not in (None, ""):
+            score = max(score, 76.0)
+            reasons.append("keyboard_focusable")
+        if _tokens(str(item.get("class_name") or "")) & _ACTION_CLASS_TOKENS:
+            score = max(score, 82.0)
+            reasons.append("action_class_token")
+        if str(item.get("cursor") or "").lower() == "pointer":
+            score = max(score, 68.0)
+            reasons.append("pointer_cursor")
+
+        # Target selection and locator quality are one decision.  Event-time
+        # counts are captured before a click can navigate or mutate the DOM.
+        counts = [
+            item.get("stable_css_count"),
+            item.get("href_count"),
+            item.get("css_count"),
+        ]
+        known_counts = [int(value) for value in counts if value is not None]
+        if 1 in known_counts:
+            score += 22.0
+            reasons.append("event_time_unique_locator")
+        elif known_counts and min(known_counts) > 1:
+            score -= 18.0
+            reasons.append("event_time_ambiguous_locator")
+
+        if tag in _GRAPHIC_LEAF_TAGS:
+            score -= 28.0
+            reasons.append("graphic_leaf")
+        elif (
+            str(item.get("cursor") or "").lower() == "pointer"
+            and 20.0 <= float(rect.get("width") or 0.0) <= 96.0
+            and 20.0 <= float(rect.get("height") or 0.0) <= 96.0
+        ):
+            score += 14.0
+            reasons.append("compact_clickable_wrapper")
+
+        score -= depth * 3.5
+        if not _contains(rect, x, y):
+            score -= 100.0
+            reasons.append("outside_pointer")
+        if area and area > max(hit_area * 25.0, 200_000.0):
+            score -= 70.0
+            reasons.append("oversized_ancestor")
+        elif area and area > max(hit_area * 8.0, 60_000.0):
+            score -= 25.0
+            reasons.append("large_ancestor")
+
+        scored.append({"depth": depth, "score": round(score, 3), "reasons": reasons, "target": item})
+
+    winner = max(scored, key=lambda row: (row["score"], -row["depth"]))
+    return dict(winner["target"]), scored

tools/ui_inspector/desktop_uia.py

@@ -508,6 +508,70 @@ def prefer_last_active_popup(info: WindowInfo) -> WindowInfo:
         return info
 
 
+def window_info_under_cursor(
+    *,
+    exclude_hwnds: Optional[set[int]] = None,
+) -> Optional[WindowInfo]:
+    """光标下的顶层窗（不 Activate、不点击）。光标在 Inspector 上则返回 None。"""
+    exclude = {int(h) for h in (exclude_hwnds or set()) if h}
+    exclude |= {get_toplevel_hwnd(h) for h in list(exclude)}
+    pt = wintypes.POINT()
+    if not user32.GetCursorPos(ctypes.byref(pt)):
+        return None
+    hwnd = int(user32.WindowFromPoint(pt) or 0)
+    if not hwnd:
+        return None
+    top = get_toplevel_hwnd(hwnd)
+    if not top or top in exclude:
+        return None
+    if not user32.IsWindowVisible(top) or user32.IsIconic(top) or _is_cloaked(top):
+        return None
+    try:
+        info = window_info_from_hwnd(top)
+    except Exception:
+        return None
+    if is_inspector_window(info):
+        return None
+    if "UI Inspector" in (info.title or "") or "OpenClaw UI Inspector" in (info.title or ""):
+        return None
+    return info
+
+
+def find_wecom_search_result_window(
+    *,
+    pid: Optional[int] = None,
+    exclude_hwnds: Optional[set[int]] = None,
+) -> Optional[WindowInfo]:
+    """同进程可见的 SearchResultWindow2（搜索联系人浮层），不要求前台焦点。"""
+    exclude = {int(h) for h in (exclude_hwnds or set()) if h}
+    exclude |= {get_toplevel_hwnd(h) for h in list(exclude)}
+    best: Optional[WindowInfo] = None
+    best_area = -1
+    for hwnd in iter_toplevel_hwnds_zorder():
+        if hwnd in exclude:
+            continue
+        if not user32.IsWindowVisible(hwnd) or user32.IsIconic(hwnd) or _is_cloaked(hwnd):
+            continue
+        cls_buf = ctypes.create_unicode_buffer(256)
+        user32.GetClassNameW(hwnd, cls_buf, 256)
+        if (cls_buf.value or "") != "SearchResultWindow2":
+            continue
+        try:
+            info = window_info_from_hwnd(hwnd)
+        except Exception:
+            continue
+        if pid is not None and int(info.pid) != int(pid):
+            continue
+        rect = _window_rect(hwnd) or {}
+        area = int(rect.get("width") or 0) * int(rect.get("height") or 0)
+        if area < 40 * 40:
+            continue
+        if area > best_area:
+            best_area = area
+            best = info
+    return best
+
+
 def foreground_toplevel_hwnd(*, exclude_hwnds: Optional[set[int]] = None) -> Optional[int]:
     """True foreground top-level hwnd (GetForegroundWindow), if capturable."""
     exclude = {int(h) for h in (exclude_hwnds or set()) if h}
@@ -530,16 +594,38 @@ def find_capture_target(
     *,
     exclude_hwnds: Optional[set[int]] = None,
     prefer_pid: Optional[int] = None,
+    under_cursor: bool = False,
+    prefer_wecom_search_popup: bool = True,
 ) -> tuple[WindowInfo, list[WindowInfo]]:
     """
-    Pick the frontmost capturable window.
+    Pick a capturable top-level window.
 
-    Priority:
-      1) GetForegroundWindow (true frontmost), if capturable
+    Default (under_cursor=False) keeps legacy behaviour:
+      1) GetForegroundWindow, if capturable
       2) else Z-order top capturable candidate
-    Optional prefer_pid only filters when sticky (same front app pid);
-    never jumps to another app just because it is larger.
+      3) prefer_last_active_popup (same pid)
+
+    Optional:
+      - under_cursor: 光标下顶层窗优先（不点击、不抢焦点）；失败则回退前台逻辑
+      - prefer_wecom_search_popup: 若候选为企微主窗且存在 SearchResultWindow2，改采浮层
     """
+    info, cands, _meta = resolve_capture_target(
+        exclude_hwnds=exclude_hwnds,
+        prefer_pid=prefer_pid,
+        under_cursor=under_cursor,
+        prefer_wecom_search_popup=prefer_wecom_search_popup,
+    )
+    return info, cands
+
+
+def resolve_capture_target(
+    *,
+    exclude_hwnds: Optional[set[int]] = None,
+    prefer_pid: Optional[int] = None,
+    under_cursor: bool = False,
+    prefer_wecom_search_popup: bool = True,
+) -> tuple[WindowInfo, list[WindowInfo], dict[str, Any]]:
+    """与 find_capture_target 相同，额外返回 selection meta（strategy 链）。"""
     candidates = list_capturable_candidates(exclude_hwnds=exclude_hwnds)
     if not candidates:
         raise RuntimeError(
@@ -548,21 +634,31 @@ def find_capture_target(
 
     by_hwnd = {int(c.hwnd): c for c in candidates}
     chosen: Optional[WindowInfo] = None
+    strategy_parts: list[str] = []
 
-    fg = foreground_toplevel_hwnd(exclude_hwnds=exclude_hwnds)
-    if fg and fg in by_hwnd:
-        # Frontmost always wins — never replace with a larger/sticky other app.
-        chosen = by_hwnd[fg]
-    elif fg:
-        # Foreground exists but filtered from candidates — still try to describe it.
-        try:
-            info = window_info_from_hwnd(fg)
-            if not is_inspector_window(info) and not (
-                is_system_chrome_window(info) and not (info.title or "").strip()
-            ):
-                chosen = info
-        except Exception:
-            chosen = None
+    if under_cursor:
+        cur = window_info_under_cursor(exclude_hwnds=exclude_hwnds)
+        if cur is not None:
+            chosen = cur
+            strategy_parts.append("under_cursor")
+
+    if chosen is None:
+        fg = foreground_toplevel_hwnd(exclude_hwnds=exclude_hwnds)
+        if fg and fg in by_hwnd:
+            # Frontmost always wins — never replace with a larger/sticky other app.
+            chosen = by_hwnd[fg]
+            strategy_parts.append("foreground")
+        elif fg:
+            # Foreground exists but filtered from candidates — still try to describe it.
+            try:
+                info = window_info_from_hwnd(fg)
+                if not is_inspector_window(info) and not (
+                    is_system_chrome_window(info) and not (info.title or "").strip()
+                ):
+                    chosen = info
+                    strategy_parts.append("foreground_raw")
+            except Exception:
+                chosen = None
 
     if chosen is None:
         # No usable foreground: Z-order front. prefer_pid only as soft filter here.
@@ -570,15 +666,35 @@ def find_capture_target(
             for c in candidates:
                 if int(c.pid) == int(prefer_pid):
                     chosen = c
+                    strategy_parts.append("sticky_pid")
                     break
         if chosen is None:
             chosen = candidates[0]
+            strategy_parts.append("zorder")
 
     # Owned popup of the *same* front window only (e.g. search panel), never another app
     popup = prefer_last_active_popup(chosen)
-    if int(popup.pid) == int(chosen.pid):
+    if int(popup.pid) == int(chosen.pid) and int(popup.hwnd) != int(chosen.hwnd):
         chosen = popup
-    return chosen, candidates
+        strategy_parts.append("last_active_popup")
+
+    # 企微：主窗为前台时，若搜索浮层仍打开，优先采浮层（无需点击进会话）
+    if prefer_wecom_search_popup and (chosen.class_name or "") == "WeWorkWindow":
+        search_win = find_wecom_search_result_window(
+            pid=int(chosen.pid), exclude_hwnds=exclude_hwnds
+        )
+        if search_win is not None:
+            chosen = search_win
+            strategy_parts.append("wecom_search_popup")
+
+    meta = {
+        "strategy": "+".join(strategy_parts) if strategy_parts else "unknown",
+        "under_cursor": bool(under_cursor),
+        "prefer_wecom_search_popup": bool(prefer_wecom_search_popup),
+        "chosen_class": chosen.class_name,
+        "chosen_hwnd": int(chosen.hwnd),
+    }
+    return chosen, candidates, meta
 
 
 def wait_frontmost_target(
@@ -632,8 +748,6 @@ def prepare_frontmost_for_capture(
 
     企微：先静默调用 kit ``ensure_ui_ready("wecom")``（与技能同路径），再轻量 poke。
     """
-    import kit_wake
-
     info, cands, wait_meta = wait_frontmost_target(
         exclude_hwnds=exclude_hwnds, timeout_sec=timeout_sec
     )
@@ -642,28 +756,29 @@ def prepare_frontmost_for_capture(
         # Keep operator size/position; only restore if minimized.
         act = activate_main_hwnd(info.hwnd, maximize=False)
 
+    # Product adapters may need to make their accessibility provider ready
+    # before the operator starts typing.  Running this on the app's existing
+    # startup worker keeps Tk responsive and prevents the first input
+    # transaction from racing a 10-20 second background warmup.
+    import kit_wake
+
     kit_report = kit_wake.auto_ensure_if_wecom(
         class_name=info.class_name or "",
         title=info.title or "",
         hwnd=info.hwnd,
-        force=True,
     )
+    findall = kit_report.get("findall") if kit_report.get("ok") else None
+    provider_cold = bool(kit_report.get("applicable") and not kit_report.get("ok"))
 
-    warm = warmup_uia_provider(
-        info.hwnd, interact_window=False, allow_foreground_nudge=False
-    )
-    n = int(warm.get("findall_after") or findall_count_for_hwnd(info.hwnd) or 0)
-    if kit_report.get("ok") and int(kit_report.get("findall") or 0) > n:
-        n = int(kit_report["findall"])
     return {
         "ready": True,  # frontmost resolved => ready to capture that window
         "window": info.to_dict(),
         "candidates": len(cands),
-        "findall": n,
-        "uia_sparse": n < 8,
-        "provider_cold": is_provider_cold(info.hwnd, findall=n),
+        "findall": findall,
+        "uia_sparse": bool(provider_cold),
+        "provider_cold": provider_cold,
         "activate": act,
-        "warmup": warm,
+        "warmup": {"mode": "startup_adapter" if kit_report.get("applicable") else "deferred"},
         "kit_wake": kit_report,
         "wait": wait_meta,
         "strategy": "foreground_only",
@@ -933,6 +1048,7 @@ def dump_window_tree(
     warmup_mode:
       - full: MSAA/UIA poke on target; restore only if minimized
       - light: ElementFromHandle poke only (no ShowWindow)
+      - probe: light poke + FindAll, but never invoke a blocking product wake
       - off: no provider warmup
     Child / same-PID merges never call ShowWindow (light poke at most).
     """
@@ -942,7 +1058,7 @@ def dump_window_tree(
     mode = (warmup_mode or "full").lower()
     if not warmup:
         mode = "off"
-    if mode not in {"full", "light", "off"}:
+    if mode not in {"full", "light", "probe", "off"}:
         mode = "full"
 
     kit_report: dict[str, Any] = {"applicable": False, "skipped": True, "reason": "not_checked"}
@@ -950,7 +1066,11 @@ def dump_window_tree(
         winfo_for_kit = window_info_from_hwnd(hwnd)
     except Exception:
         winfo_for_kit = None
-    if winfo_for_kit is not None:
+    # 搜索浮层稀疏时勿合并主窗会话列表，否则验收会误判「已有数据」
+    allow_same_pid_merge = bool(merge_same_pid_if_sparse)
+    if winfo_for_kit is not None and (winfo_for_kit.class_name or "") == "SearchResultWindow2":
+        allow_same_pid_merge = False
+    if winfo_for_kit is not None and mode not in {"off", "probe"}:
         hint = findall_count_for_hwnd(hwnd)
         kit_report = kit_wake.auto_ensure_if_wecom(
             class_name=winfo_for_kit.class_name or "",
@@ -977,7 +1097,7 @@ def dump_window_tree(
             warmup_report = warmup_uia_provider(
                 hwnd, interact_window=True, allow_foreground_nudge=True
             )
-    elif mode == "light":
+    elif mode in {"light", "probe"}:
         warmup_report = poke_uia_provider(hwnd)
 
     def _run_dump() -> tuple[dict[str, Any], list[dict[str, Any]], dict[str, Any]]:
@@ -1035,7 +1155,7 @@ def dump_window_tree(
                 t = dict(t)
                 t["children"] = children_nodes
 
-        if use_findall:
+        if use_findall and mode != "off":
             found = collect_findall_flat(root_ctrl, max_nodes=max_nodes)
             st["findall_count"] = len(found)
             existing = {r.get("path_hint") for r in idx}
@@ -1047,6 +1167,13 @@ def dump_window_tree(
         return t, idx, st
 
     tree, index, stats = _run_dump()
+    if winfo_for_kit is not None:
+        kit_wake.note_provider_warm(
+            class_name=winfo_for_kit.class_name or "",
+            title=winfo_for_kit.title or "",
+            hwnd=hwnd,
+            findall=int(stats.get("findall_count") or 0),
+        )
     stats["warmup"] = warmup_report
     stats["warmup_mode"] = mode
     stats["kit_wake"] = kit_report
@@ -1061,7 +1188,7 @@ def dump_window_tree(
     )
 
     # 仍冷且目标为企微：强制再 ensure 一次（绕过短时 cache）后重 dump
-    if stats["provider_cold"] and winfo_for_kit is not None and kit_wake.looks_like_wecom(
+    if mode not in {"off", "probe"} and stats["provider_cold"] and winfo_for_kit is not None and kit_wake.looks_like_wecom(
         class_name=winfo_for_kit.class_name or "",
         title=winfo_for_kit.title or "",
     ):
@@ -1117,7 +1244,9 @@ def dump_window_tree(
 
     # Sparse fallback: light poke + FindAll on a few large same-pid top-levels.
     # Includes WeWorkWindow when current target is a thin popup.
-    if merge_same_pid_if_sparse and stats["uia_sparse"]:
+    # Never merge into SearchResultWindow2 (see allow_same_pid_merge).
+    stats["merge_same_pid_allowed"] = allow_same_pid_merge
+    if allow_same_pid_merge and stats["uia_sparse"]:
         try:
             info = window_info_from_hwnd(hwnd)
             merged_n = 0
@@ -1329,15 +1458,20 @@ def format_tree_text(tree: dict[str, Any], *, max_lines: int = 2000) -> str:
     return "\n".join(lines)
 
 
-# Back-compat alias used by older app versions
-def resolve_capture_target(
+# Back-compat: older callers expected a single WindowInfo (foreground only).
+# Do NOT name this resolve_capture_target — that name is the full 3-tuple API above.
+def resolve_frontmost_window(
     *,
     self_hwnd: int = 0,
     poll_sec: float = 0.15,
     wait_sec: float = 2.0,
 ) -> WindowInfo:
     _ = poll_sec, wait_sec
-    info, _cands = find_capture_target(exclude_hwnds={int(self_hwnd)} if self_hwnd else None)
+    info, _cands = find_capture_target(
+        exclude_hwnds={int(self_hwnd)} if self_hwnd else None,
+        under_cursor=False,
+        prefer_wecom_search_popup=False,
+    )
     return info
 
 

tools/ui_inspector/diagnostics/__init__.py

@@ -0,0 +1,6 @@
+"""Capture validation and timeline diagnostics."""
+
+from .capture_validator import validate_capture
+from .replay_preflight import build_replay_preflight, write_replay_preflight
+
+__all__ = ["validate_capture", "build_replay_preflight", "write_replay_preflight"]

tools/ui_inspector/diagnostics/capture_validator.py

@@ -0,0 +1,134 @@
+"""Product-neutral capture quality checks."""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+
+def _read(path: Path) -> dict[str, Any]:
+    return json.loads(path.read_text(encoding="utf-8"))
+
+
+def validate_capture(flow_dir: Path) -> dict[str, Any]:
+    root = Path(flow_dir)
+    issues: list[dict[str, Any]] = []
+    actions = sorted(
+        (_read(path) for path in (root / "actions").glob("*.json")),
+        key=lambda item: int(item.get("sequence") or 0),
+    )
+    expected = list(range(1, len(actions) + 1))
+    actual = [int(item.get("sequence") or 0) for item in actions]
+    if actual != expected:
+        issues.append({"severity": "error", "code": "ACTION_SEQUENCE_GAP", "actual": actual})
+    elements = {
+        item.get("id"): item
+        for item in (_read(path) for path in (root / "elements").glob("*.json"))
+        if item.get("id")
+    }
+    interactive_types = {"click", "fill", "type", "press_key", "select", "toggle"}
+    previous_action: dict[str, Any] | None = None
+    for action in actions:
+        target = dict(action.get("target_snapshot") or {})
+        rect = dict(target.get("rect") or target.get("bounds") or {})
+        area = float(rect.get("width") or 0) * float(rect.get("height") or 0)
+        if action.get("action_type") in {"click", "fill", "select", "toggle"} and not target:
+            issues.append({"severity": "error", "code": "ACTION_TARGET_MISSING", "action_id": action.get("id")})
+        if action.get("action_type") == "click" and area > 1_000_000:
+            issues.append({"severity": "error", "code": "CLICK_TARGET_OVERSIZED", "action_id": action.get("id"), "area": area})
+        target_name = str(target.get("name") or target.get("automation_id") or "").lower()
+        if (
+            action.get("surface") == "windows"
+            and action.get("action_type") == "click"
+            and str(target.get("control_type") or "") in {"TreeControl", "PaneControl", "GroupControl"}
+            and any(token in target_name for token in ("search_result", "listview", "list_view", "watermark", "container", "layout"))
+        ):
+            issues.append({"severity": "warning", "code": "CLICK_TARGET_GENERIC_CONTAINER", "action_id": action.get("id"), "target_name": target_name})
+        if (
+            action.get("surface") == "windows"
+            and action.get("action_type") == "press_key"
+            and (str(target.get("control_type") or "") == "WindowControl" or area > 1_000_000)
+        ):
+            issues.append({"severity": "warning", "code": "KEY_TARGET_ROOT_WINDOW", "action_id": action.get("id")})
+        if action.get("action_type") == "click" and str(target.get("tag") or "").lower() in {"path", "use", "g", "svg"}:
+            issues.append({"severity": "warning", "code": "GRAPHIC_LEAF_TARGET", "action_id": action.get("id"), "tag": target.get("tag")})
+        evidence = dict(action.get("evidence") or {})
+        if not action.get("after_observation_id") and not evidence.get("eventual_observation_id"):
+            issues.append({"severity": "warning", "code": "ACTION_EVIDENCE_MISSING", "action_id": action.get("id")})
+        if evidence.get("locator_unique") is False:
+            issues.append({"severity": "warning", "code": "LOCATOR_NOT_UNIQUE", "action_id": action.get("id")})
+        if action.get("surface") == "browser" and action.get("action_type") in interactive_types:
+            element_id = action.get("target_element_id")
+            if target and not element_id:
+                issues.append({"severity": "warning", "code": "TARGET_ELEMENT_REFERENCE_MISSING", "action_id": action.get("id")})
+            element = elements.get(element_id) if element_id else None
+            primary = dict((element or {}).get("primary_locator") or {})
+            if element_id and (not primary or not primary.get("production_eligible")):
+                issues.append({"severity": "warning", "code": "ACTION_PRODUCTION_LOCATOR_MISSING", "action_id": action.get("id"), "element_id": element_id})
+            elif primary and float(primary.get("stability_score") or 0.0) < 0.55:
+                issues.append({"severity": "warning", "code": "LOW_STABILITY_LOCATOR", "action_id": action.get("id"), "element_id": element_id, "strategy": primary.get("strategy"), "stability_score": primary.get("stability_score")})
+            raw_meta = dict((action.get("raw_event") or {}).get("meta") or {})
+            if not raw_meta.get("event_id"):
+                issues.append({"severity": "warning", "code": "ACTION_RAW_EVENT_LINK_MISSING", "action_id": action.get("id")})
+        if action.get("surface") == "windows" and action.get("action_type") in interactive_types:
+            element_id = action.get("target_element_id")
+            if target and not element_id:
+                issues.append({"severity": "warning", "code": "TARGET_ELEMENT_REFERENCE_MISSING", "action_id": action.get("id")})
+            element = elements.get(element_id) if element_id else None
+            primary = dict((element or {}).get("primary_locator") or {})
+            if element_id and not primary:
+                issues.append({"severity": "warning", "code": "ACTION_PRODUCTION_LOCATOR_MISSING", "action_id": action.get("id"), "element_id": element_id})
+        if previous_action and action.get("action_type") == "fill" and previous_action.get("action_type") == "fill":
+            same_target = bool(action.get("target_element_id")) and action.get("target_element_id") == previous_action.get("target_element_id")
+            old_text = str((previous_action.get("input") or {}).get("text") or "")
+            new_text = str((action.get("input") or {}).get("text") or "")
+            old_ms = ((previous_action.get("raw_event") or {}).get("meta") or {}).get("source_ts_ms")
+            new_ms = ((action.get("raw_event") or {}).get("meta") or {}).get("source_ts_ms")
+            close = False
+            try:
+                close = 0 <= float(new_ms) - float(old_ms) <= 2000
+            except (TypeError, ValueError):
+                pass
+            if same_target and old_text and new_text.startswith(old_text) and new_text != old_text and close:
+                issues.append({"severity": "warning", "code": "DUPLICATE_FILL_TRANSACTION", "action_id": action.get("id"), "previous_action_id": previous_action.get("id"), "element_id": action.get("target_element_id")})
+        previous_action = action
+    journal = root / "journal" / "raw_events.jsonl"
+    if not journal.exists():
+        issues.append({"severity": "error", "code": "RAW_EVENT_JOURNAL_MISSING"})
+    if not actions:
+        issues.append({"severity": "error", "code": "ZERO_ACTIONS"})
+    typing_transactions = 0
+    if journal.exists():
+        for line in journal.read_text(encoding="utf-8").splitlines():
+            try:
+                payload = json.loads(line)
+            except (TypeError, ValueError, json.JSONDecodeError):
+                continue
+            event = dict(payload.get("event") or payload)
+            if event.get("kind") == "typing_activity":
+                typing_transactions += 1
+    fill_count = sum(
+        1 for action in actions if action.get("action_type") in {"fill", "type"}
+    )
+    if typing_transactions > fill_count:
+        issues.append(
+            {
+                "severity": "warning",
+                "code": "TEXT_INPUT_TRANSACTION_MISSING",
+                "typing_transactions": typing_transactions,
+                "fill_actions": fill_count,
+            }
+        )
+    screenshot_count = len(list((root / "observations").glob("*/screen.png")))
+    return {
+        "schema": "openclaw.ui_inspector.validation.v1",
+        "flow_dir": str(root),
+        "action_count": len(actions),
+        "element_count": len(elements),
+        "observation_count": len(list((root / "observations").glob("*/meta.json"))),
+        "screenshot_count": screenshot_count,
+        "error_count": sum(1 for issue in issues if issue["severity"] == "error"),
+        "warning_count": sum(1 for issue in issues if issue["severity"] == "warning"),
+        "issues": issues,
+    }

tools/ui_inspector/diagnostics/replay_preflight.py

@@ -0,0 +1,133 @@
+"""Static, non-mutating replay readiness report for a finished capture."""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+
+_INTERACTIVE = {"click", "fill", "type", "press_key", "select", "toggle"}
+
+
+def _read(path: Path) -> dict[str, Any]:
+    return json.loads(path.read_text(encoding="utf-8"))
+
+
+def build_replay_preflight(flow_dir: Path) -> dict[str, Any]:
+    """Assess whether recorded actions contain executable, verified locators.
+
+    This deliberately does not click or navigate.  `event_time_unique` means
+    the locator count was measured in the event-origin document during capture;
+    it is not represented as a cross-session replay result.
+    """
+    root = Path(flow_dir)
+    actions = sorted(
+        (_read(path) for path in (root / "actions").glob("*.json")),
+        key=lambda item: int(item.get("sequence") or 0),
+    )
+    elements = {
+        item.get("id"): item
+        for item in (_read(path) for path in (root / "elements").glob("*.json"))
+        if item.get("id")
+    }
+    rows: list[dict[str, Any]] = []
+    for action in actions:
+        action_type = str(action.get("action_type") or "")
+        if action_type not in _INTERACTIVE:
+            rows.append({
+                "action_id": action.get("id"),
+                "sequence": action.get("sequence"),
+                "action_type": action_type,
+                "result": "context_only",
+                "reason": "non_interactive_or_navigation",
+            })
+            continue
+        element_id = action.get("target_element_id")
+        element = elements.get(element_id) or {}
+        primary = dict(element.get("primary_locator") or {})
+        eligible = bool(primary.get("production_eligible"))
+        unique = primary.get("unique") is True
+        score = float(primary.get("stability_score") or 0.0)
+        result = "pass" if eligible and unique and score >= 0.55 else "warning"
+        if not eligible or not unique:
+            result = "fail"
+        target = dict(action.get("target_snapshot") or {})
+        bounds = dict(target.get("bounds") or target.get("rect") or {})
+        area = float(bounds.get("width") or 0) * float(bounds.get("height") or 0)
+        oversized_click = action_type == "click" and area > 1_000_000
+        root_window_target = (
+            action_type == "click"
+            and str(target.get("control_type") or "") == "WindowControl"
+        )
+        if oversized_click or root_window_target:
+            result = "fail"
+        validation = str(primary.get("validation") or "none")
+        verification = (
+            "event_time_unique" if validation == "live" and unique
+            else "snapshot_unique" if validation in {"snapshot_only", "live_tree"} and unique
+            else "not_validated"
+        )
+        raw_meta = dict((action.get("raw_event") or {}).get("meta") or {})
+        rows.append({
+            "action_id": action.get("id"),
+            "sequence": action.get("sequence"),
+            "action_type": action_type,
+            "result": result,
+            "element_id": element_id,
+            "logical_name": element.get("logical_name") or "",
+            "locator": primary.get("expression") or "",
+            "strategy": primary.get("strategy") or "",
+            "stability_score": score,
+            "match_count": primary.get("actual_count"),
+            "verification": verification,
+            "page_id": raw_meta.get("page_id") or "",
+            "document_id": raw_meta.get("document_id") or "",
+            "raw_event_ids": list(action.get("raw_event_ids") or []),
+            "reason": (
+                "oversized_or_root_window_click_target"
+                if oversized_click or root_window_target else ""
+            ),
+        })
+    counts = {
+        name: sum(1 for row in rows if row.get("result") == name)
+        for name in ("pass", "warning", "fail", "context_only")
+    }
+    from diagnostics.capture_validator import validate_capture
+
+    quality = validate_capture(root)
+    blocking_codes = {
+        "TEXT_INPUT_TRANSACTION_MISSING",
+        "CLICK_TARGET_GENERIC_CONTAINER",
+        "KEY_TARGET_ROOT_WINDOW",
+        "CLICK_TARGET_OVERSIZED",
+        "ACTION_TARGET_MISSING",
+        "ZERO_ACTIONS",
+    }
+    quality_blockers = [
+        issue
+        for issue in quality.get("issues") or []
+        if issue.get("severity") == "error" or issue.get("code") in blocking_codes
+    ]
+    counts["quality_blockers"] = len(quality_blockers)
+    return {
+        "schema": "openclaw.ui_inspector.replay_preflight.v1",
+        "mode": "static_non_mutating",
+        "note": "事件时刻唯一性不等同于跨会话真实回放；本报告不会点击、输入或导航。",
+        "flow_dir": str(root),
+        "ready": not quality_blockers and counts.get("fail", 0) == 0,
+        "summary": counts,
+        "quality_gate": {
+            "passed": not quality_blockers,
+            "issues": quality_blockers,
+        },
+        "actions": rows,
+    }
+
+
+def write_replay_preflight(flow_dir: Path) -> dict[str, Any]:
+    report = build_replay_preflight(flow_dir)
+    path = Path(flow_dir) / "diagnostics" / "replay_report.json"
+    path.parent.mkdir(parents=True, exist_ok=True)
+    path.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+    return report

tools/ui_inspector/drivers/__init__.py

@@ -0,0 +1,29 @@
+"""Surface drivers for UI Inspector (windows / browser / mobile stub)."""
+
+from __future__ import annotations
+
+from .base import DumpResult, SurfaceDriver
+from .browser_pw import BrowserPlaywrightDriver
+from .mobile import MobileDriver
+from .windows_uia import WindowsUiaDriver
+import target as _target
+
+
+def get_driver(surface: str) -> SurfaceDriver:
+    if surface == _target.SURFACE_WINDOWS:
+        return WindowsUiaDriver()
+    if surface == _target.SURFACE_BROWSER:
+        return BrowserPlaywrightDriver()
+    if surface in {_target.SURFACE_ANDROID, _target.SURFACE_IOS}:
+        return MobileDriver(surface=surface)
+    raise RuntimeError(f"ERROR:SURFACE_UNSUPPORTED unknown surface={surface!r}")
+
+
+__all__ = [
+    "DumpResult",
+    "SurfaceDriver",
+    "WindowsUiaDriver",
+    "BrowserPlaywrightDriver",
+    "MobileDriver",
+    "get_driver",
+]

tools/ui_inspector/drivers/base.py

@@ -0,0 +1,38 @@
+"""Driver protocol: dump one step's UI tree for a surface."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any, Optional, Protocol
+
+
+@dataclass
+class DumpResult:
+    surface: str
+    driver: str
+    window: dict[str, Any]
+    tree: dict[str, Any]
+    index: list[dict[str, Any]]
+    locatable: list[dict[str, Any]]
+    stats: dict[str, Any]
+    selection: dict[str, Any] = field(default_factory=dict)
+    url: str = ""
+    page_title: str = ""
+    locatable_text: str = ""
+    tree_text: str = ""
+    screenshot_bytes: Optional[bytes] = None  # if set, artifacts write screen.png from bytes
+    use_hwnd_screenshot: bool = True  # windows: ImageGrab by hwnd
+
+
+class SurfaceDriver(Protocol):
+    def dump(
+        self,
+        *,
+        info: Any,
+        selection_meta: dict[str, Any],
+        max_depth: int = 16,
+        max_nodes: int = 8000,
+        warmup_mode: str = "full",
+        merge_same_pid_if_sparse: bool = True,
+        warmed_pids: Optional[set[int]] = None,
+    ) -> DumpResult: ...

tools/ui_inspector/drivers/browser_pw.py

@@ -0,0 +1,90 @@
+"""Browser whole-page dump via Inspector Playwright session + kit page_dump."""
+
+from __future__ import annotations
+
+from typing import Any, Optional
+
+import kit_browser
+import target
+from .base import DumpResult
+
+
+class BrowserPlaywrightDriver:
+    surface = target.SURFACE_BROWSER
+    driver_name = "playwright"
+
+    def dump(
+        self,
+        *,
+        info: Any,
+        selection_meta: dict[str, Any],
+        max_depth: int = 16,
+        max_nodes: int = 8000,
+        warmup_mode: str = "full",
+        merge_same_pid_if_sparse: bool = True,
+        warmed_pids: Optional[set[int]] = None,
+        page_id: str = "",
+        document_id: str = "",
+        frame_id: str = "",
+    ) -> DumpResult:
+        del max_depth, warmup_mode, merge_same_pid_if_sparse, warmed_pids  # unused
+        sess = kit_browser.ensure_session()
+        dump = kit_browser.dump_current_page(
+            max_nodes=max_nodes, locatable_limit=300, page_id=page_id
+        )
+        try:
+            png = kit_browser.screenshot_png(page_id=page_id)
+        except Exception:
+            png = None
+
+        from jiangchang_skill_core.rpa.page_dump import (
+            format_locatable_text,
+            format_tree_text,
+        )
+
+        url = dump.get("url") or ""
+        page_title = dump.get("title") or ""
+        window = info.to_dict() if hasattr(info, "to_dict") else dict(info or {})
+        # Prefer live page identity for labeling / meta
+        window = {
+            **window,
+            "title": page_title or window.get("title") or "",
+            "class_name": window.get("class_name") or "PlaywrightPage",
+            "process_name": window.get("process_name") or "chrome.exe",
+            "url": url,
+        }
+        stats = dict(dump.get("stats") or {})
+        active_page = dict(stats.get("active_page") or {})
+        stats["browser_session"] = {
+            "profile_dir": sess.get("profile_dir"),
+            "reused": sess.get("reused"),
+            "page_count": active_page.get("page_count") or sess.get("page_count"),
+            "resolve": active_page.get("resolve") or "visible",
+            "note": (
+                "dump 来自 Inspector 自管 Playwright 的当前可见标签页"
+                "（document.visibilityState），不是系统 Chrome 的 UIA，"
+                "也不是固定的 pages[0]。多标签时请切到目标页再拾取。"
+            ),
+        }
+        sel = dict(selection_meta or {})
+        sel["surface"] = self.surface
+        sel["browser_url"] = url
+        sel["page_id"] = page_id or active_page.get("page_id") or ""
+        sel["document_id"] = document_id
+        sel["frame_id"] = frame_id
+        return DumpResult(
+            surface=self.surface,
+            driver=self.driver_name,
+            window=window,
+            tree=dump.get("tree") or {},
+            index=list(dump.get("index") or []),
+            locatable=list(dump.get("locatable") or []),
+            stats=stats,
+            selection=sel,
+            url=url,
+            page_title=page_title,
+            locatable_text=format_locatable_text(dump.get("locatable") or []),
+            tree_text=format_tree_text(dump.get("tree") or {}),
+            screenshot_bytes=png,
+            use_hwnd_screenshot=False,
+        )

tools/ui_inspector/drivers/mobile.py

@@ -0,0 +1,30 @@
+"""Mobile surface stub — reserved for future android/ios dump."""
+
+from __future__ import annotations
+
+from typing import Any, Optional
+
+import target
+from .base import DumpResult
+
+
+class MobileDriver:
+    def __init__(self, *, surface: str = target.SURFACE_ANDROID) -> None:
+        self.surface = surface
+        self.driver_name = "mobile_stub"
+
+    def dump(
+        self,
+        *,
+        info: Any,
+        selection_meta: dict[str, Any],
+        max_depth: int = 16,
+        max_nodes: int = 8000,
+        warmup_mode: str = "full",
+        merge_same_pid_if_sparse: bool = True,
+        warmed_pids: Optional[set[int]] = None,
+    ) -> DumpResult:
+        raise RuntimeError(
+            f"ERROR:SURFACE_UNSUPPORTED 手机端拾取尚未实现（surface={self.surface}）。"
+            "请先用桌面或浏览器目标。"
+        )

tools/ui_inspector/drivers/windows_uia.py

@@ -0,0 +1,58 @@
+"""Windows desktop dump via existing desktop_uia."""
+
+from __future__ import annotations
+
+from typing import Any, Optional
+
+import desktop_uia as uia
+import target
+from .base import DumpResult
+
+
+class WindowsUiaDriver:
+    surface = target.SURFACE_WINDOWS
+    driver_name = "uiautomation"
+
+    def dump(
+        self,
+        *,
+        info: Any,
+        selection_meta: dict[str, Any],
+        max_depth: int = 16,
+        max_nodes: int = 8000,
+        warmup_mode: str = "full",
+        merge_same_pid_if_sparse: bool = True,
+        warmed_pids: Optional[set[object]] = None,
+    ) -> DumpResult:
+        warmed = warmed_pids if warmed_pids is not None else set()
+        mode = warmup_mode
+        warm_key = (int(info.pid), str(info.class_name or ""))
+        if (warm_key in warmed or int(info.pid) in warmed) and mode == "full":
+            mode = "light"
+        merge_sparse = merge_same_pid_if_sparse
+        if (info.class_name or "") == "SearchResultWindow2":
+            merge_sparse = False
+        tree, index, stats, locatable = uia.dump_window_tree(
+            info.hwnd,
+            max_depth=max_depth,
+            max_nodes=max_nodes,
+            only_visible=False,
+            warmup_mode=mode,
+            merge_same_pid_if_sparse=merge_sparse,
+        )
+        return DumpResult(
+            surface=self.surface,
+            driver=self.driver_name,
+            window=info.to_dict(),
+            tree=tree,
+            index=index,
+            locatable=locatable,
+            stats=stats,
+            selection=dict(selection_meta or {}),
+            url="",
+            page_title="",
+            locatable_text=uia.format_locatable_text(locatable),
+            tree_text=uia.format_tree_text(tree),
+            screenshot_bytes=None,
+            use_hwnd_screenshot=True,
+        )

tools/ui_inspector/evidence/__init__.py

@@ -0,0 +1,6 @@
+"""Asynchronous evidence capture."""
+
+from .coordinator import EvidenceCoordinator
+
+__all__ = ["EvidenceCoordinator"]
+

tools/ui_inspector/evidence/coordinator.py

@@ -0,0 +1,255 @@
+"""Coalesced background full-page evidence capture.
+
+Semantic actions are committed immediately.  Slow dumps are supplementary
+evidence and are explicitly labelled eventual instead of being misrepresented
+as an event-time after-state.
+"""
+
+from __future__ import annotations
+
+import queue
+import threading
+import time
+from dataclasses import dataclass
+from typing import Any, Callable, Optional
+
+
+@dataclass
+class EvidenceRequest:
+    action_id: str
+    context: dict[str, Any]
+    requested_monotonic: float
+
+
+class EvidenceCoordinator:
+    def __init__(
+        self,
+        manager: Any,
+        dump_fn: Callable[..., Any],
+        *,
+        coalesce_sec: float = 0.25,
+        on_dump: Optional[Callable[[Any], None]] = None,
+    ) -> None:
+        self.manager = manager
+        self.dump_fn = dump_fn
+        self.coalesce_sec = coalesce_sec
+        self.on_dump = on_dump
+        self._queue: queue.Queue[EvidenceRequest | None] = queue.Queue(maxsize=512)
+        self._urgent_queue: queue.Queue[EvidenceRequest | None] = queue.Queue(maxsize=128)
+        self._thread: Optional[threading.Thread] = None
+        self._urgent_thread: Optional[threading.Thread] = None
+        self._stop = threading.Event()
+        self.dropped = 0
+        self.capture_count = 0
+        self.failure_count = 0
+
+    def start(self) -> None:
+        if self._thread and self._thread.is_alive():
+            return
+        self._stop.clear()
+        self._thread = threading.Thread(target=self._run, name="ui-inspector-evidence", daemon=True)
+        self._thread.start()
+        if str(getattr(self.manager.action_flow, "surface", "")) == "windows":
+            self._urgent_thread = threading.Thread(
+                target=self._run_urgent,
+                name="ui-inspector-evidence-transient",
+                daemon=True,
+            )
+            self._urgent_thread.start()
+
+    def schedule(self, action: Any) -> None:
+        raw = dict(getattr(action, "raw_event", {}) or {})
+        meta = dict(raw.get("meta") or {})
+        context = {
+            "page_id": meta.get("page_id") or "",
+            "document_id": meta.get("document_id") or "",
+            "frame_id": meta.get("frame_id") or "",
+            "source_ts_ms": meta.get("source_ts_ms"),
+            "url": raw.get("url") or "",
+        }
+        scope = dict((raw.get("target") or {}).get("window_scope") or {})
+        context.update(
+            {
+                "target_hwnd": int(scope.get("hwnd") or 0),
+                "target_pid": int(scope.get("pid") or 0),
+                "target_class_name": str(scope.get("class_name") or ""),
+                "target_title": str(scope.get("title") or ""),
+            }
+        )
+        out_queue = (
+            self._urgent_queue
+            if context["target_class_name"] == "SearchResultWindow2"
+            else self._queue
+        )
+        try:
+            out_queue.put_nowait(EvidenceRequest(action.id, context, time.monotonic()))
+        except queue.Full:
+            self.dropped += 1
+
+    def prime(self) -> None:
+        """Start one background enrichment without creating a semantic action."""
+        try:
+            self._queue.put_nowait(EvidenceRequest("", {}, time.monotonic()))
+        except queue.Full:
+            self.dropped += 1
+
+    def _dump(self, context: dict[str, Any]) -> Any:
+        kwargs = {
+            k: context.get(k)
+            for k in (
+                "page_id", "document_id", "frame_id", "target_hwnd",
+                "target_pid", "target_class_name", "target_title",
+            )
+            if context.get(k)
+        }
+        if kwargs:
+            try:
+                return self.dump_fn(**kwargs)
+            except TypeError as exc:
+                if "unexpected keyword" not in str(exc) and "positional" not in str(exc):
+                    raise
+        return self.dump_fn()
+
+    def _run(self) -> None:
+        while not self._stop.is_set():
+            try:
+                first = self._queue.get(timeout=0.2)
+            except queue.Empty:
+                continue
+            if first is None:
+                self._queue.task_done()
+                self._stop.set()
+                break
+            batch = [first]
+            stop_after_batch = False
+            deadline = time.monotonic() + self.coalesce_sec
+            while time.monotonic() < deadline:
+                try:
+                    item = self._queue.get(timeout=max(0.01, deadline - time.monotonic()))
+                except queue.Empty:
+                    break
+                if item is None:
+                    self._queue.task_done()
+                    stop_after_batch = True
+                    break
+                batch.append(item)
+
+            groups: dict[str, list[EvidenceRequest]] = {}
+            for item in batch:
+                key = str(item.context.get("page_id") or "active")
+                groups.setdefault(key, []).append(item)
+            for items in groups.values():
+                latest = items[-1]
+                try:
+                    dump = self._dump(latest.context)
+                    if self.on_dump is not None:
+                        self.on_dump(dump)
+                    self.capture_count += 1
+                    # A desktop provider dump can take seconds. Actions that
+                    # arrived while it was running are still earlier than this
+                    # eventual observation, so fold them into the same result
+                    # instead of immediately launching another identical dump.
+                    if str(getattr(self.manager.action_flow, "surface", "")) == "windows":
+                        while True:
+                            try:
+                                late = self._queue.get_nowait()
+                            except queue.Empty:
+                                break
+                            if late is None:
+                                self._queue.task_done()
+                                stop_after_batch = True
+                                break
+                            batch.append(late)
+                            items.append(late)
+                    if not self._stop.is_set():
+                        self.manager.attach_eventual_evidence(
+                            [item.action_id for item in items], dump=dump,
+                            requested_monotonic=min(item.requested_monotonic for item in items),
+                        )
+                except Exception as exc:
+                    self.failure_count += 1
+                    if not self._stop.is_set():
+                        self.manager.note_evidence_failure([item.action_id for item in items], str(exc))
+            for _item in batch:
+                self._queue.task_done()
+            if stop_after_batch:
+                self._stop.set()
+
+    def _run_urgent(self) -> None:
+        """Capture transient native popups independently of a slow main dump."""
+        while not self._stop.is_set():
+            try:
+                first = self._urgent_queue.get(timeout=0.1)
+            except queue.Empty:
+                continue
+            if first is None:
+                self._urgent_queue.task_done()
+                break
+            batch = [first]
+            # A very short coalesce keeps double-clicks together without waiting
+            # long enough for a search-result popup to disappear.
+            deadline = time.monotonic() + 0.04
+            while time.monotonic() < deadline:
+                try:
+                    item = self._urgent_queue.get_nowait()
+                except queue.Empty:
+                    time.sleep(0.005)
+                    continue
+                if item is None:
+                    self._urgent_queue.task_done()
+                    break
+                if item.context.get("target_hwnd") == first.context.get("target_hwnd"):
+                    batch.append(item)
+                else:
+                    # Preserve a different transient window for the next pass.
+                    try:
+                        self._urgent_queue.put_nowait(item)
+                    except queue.Full:
+                        self.dropped += 1
+                    self._urgent_queue.task_done()
+                    break
+            try:
+                dump = self._dump(batch[-1].context)
+                if self.on_dump is not None:
+                    self.on_dump(dump)
+                self.capture_count += 1
+                if not self._stop.is_set():
+                    self.manager.attach_eventual_evidence(
+                        [item.action_id for item in batch],
+                        dump=dump,
+                        requested_monotonic=min(item.requested_monotonic for item in batch),
+                    )
+            except Exception as exc:
+                self.failure_count += 1
+                if not self._stop.is_set():
+                    self.manager.note_evidence_failure(
+                        [item.action_id for item in batch], str(exc)
+                    )
+            finally:
+                for _item in batch:
+                    self._urgent_queue.task_done()
+
+    def stop(self, *, timeout: float = 35.0) -> None:
+        if not self._thread:
+            return
+        # Drain the transient-window lane first.  A WeCom main-window dump can
+        # be stuck in an uninterruptible provider call for tens of seconds; it
+        # must not cancel the short-lived popup evidence when the user ends the
+        # flow immediately after selecting a result.
+        if self._urgent_thread:
+            try:
+                self._urgent_queue.put(None, timeout=1.0)
+            except queue.Full:
+                pass
+            self._urgent_thread.join(timeout=min(max(timeout, 3.0), 8.0))
+        try:
+            self._queue.put(None, timeout=2.0)
+        except queue.Full:
+            pass
+        self._thread.join(timeout=timeout)
+        if self._thread.is_alive():
+            # A provider dump may be uninterruptible. Prevent a late result
+            # from mutating a manager which is already being finalized.
+            self._stop.set()
+        self._thread = None
+        self._urgent_thread = None

tools/ui_inspector/flow_session.py

@@ -0,0 +1,408 @@
+"""Flow session glue: v2 ActionRecordingManager + surface recorders + v1 dual-write."""
+
+from __future__ import annotations
+
+import queue
+import threading
+import uuid
+from pathlib import Path
+from typing import Any, Callable, Optional
+
+import artifacts as art
+from core.pipeline.event_journal import EventJournal
+from evidence.coordinator import EvidenceCoordinator
+from recorder.base import RawUserEvent
+from recorder.manager import ActionRecordingManager
+
+
+class FlowSession:
+    def __init__(
+        self,
+        flow_dir: Path,
+        *,
+        flow_name: str,
+        surface: str,
+        recording_mode: str = "auto",
+        dump_fn: Optional[Callable[[], Any]] = None,
+        write_v1_step: Optional[Callable[..., Any]] = None,
+        exclude_hwnds: Optional[Callable[[], set[int]]] = None,
+        on_action_update: Optional[Callable[[Any, str], None]] = None,
+        stability_timeout_sec: float = 3.0,
+    ) -> None:
+        self.flow_dir = Path(flow_dir)
+        self.surface = surface
+        self.recording_mode = recording_mode
+        self._dump_fn = dump_fn
+        self.exclude_hwnds = exclude_hwnds or (lambda: set())
+        self.manager = ActionRecordingManager(
+            self.flow_dir,
+            flow_name=flow_name,
+            flow_id=self.flow_dir.name,
+            surface=surface,
+            recording_mode=recording_mode,
+            dump_fn=dump_fn,
+            write_v1_step=write_v1_step,
+            stability_timeout_sec=stability_timeout_sec,
+            defer_action_evidence=(
+                surface in {"browser", "windows"}
+                and recording_mode != "legacy_interval"
+            ),
+            on_action_update=on_action_update,
+        )
+        self.journal = EventJournal(self.flow_dir)
+        self._evidence = (
+            EvidenceCoordinator(self.manager, dump_fn, on_dump=self._on_evidence_dump)
+            if surface in {"browser", "windows"}
+            and dump_fn is not None
+            and recording_mode != "legacy_interval"
+            else None
+        )
+        if self._evidence is not None:
+            self.manager.on_action = self._evidence.schedule
+        self._recorder = None
+        self._recorder_health: dict[str, Any] = {}
+        self._event_queue: queue.Queue[tuple[str, Any]] = queue.Queue(maxsize=512)
+        self._event_thread: Optional[threading.Thread] = None
+        self._event_stop = threading.Event()
+        self._stats_lock = threading.Lock()
+        self._stats_cache: dict[str, Any] = {
+            "recorder_status": "idle",
+            "surface_recorder": "off",
+            "state_count": 0,
+            "observation_count": 0,
+            "action_count": 0,
+            "element_count": 0,
+            "queued_events": 0,
+            "dropped_events": 0,
+        }
+        self._dropped_events = 0
+
+    def start(self, *, entry_dump: Optional[Any] = None) -> None:
+        if self.recording_mode == "legacy_interval":
+            self.manager.start(entry_dump=entry_dump)
+            self._refresh_stats()
+            return
+        # Start observation before entry persistence. Events are journaled and
+        # buffered until manager.start() completes, eliminating startup loss.
+        self._start_surface_recorder()
+        if self.surface == "windows" and entry_dump is not None:
+            self._on_evidence_dump(entry_dump)
+        if self._evidence is not None:
+            self._evidence.start()
+        try:
+            self.manager.start(entry_dump=entry_dump)
+        except Exception:
+            if self._recorder:
+                self._recorder.stop()
+                self._recorder = None
+            if self._evidence is not None:
+                self._evidence.stop(timeout=3.0)
+            raise
+        self._start_event_consumer()
+        if self.surface == "windows" and self._evidence is not None:
+            # Warm/enrich UIA immediately, but on the evidence worker. Raw
+            # recording and Action reduction remain fully independent.
+            self._evidence.prime()
+        self._refresh_stats()
+
+    def _submit(self, kind: str, payload: Any = None) -> bool:
+        try:
+            if kind == "stop":
+                self._event_queue.put((kind, payload), timeout=2.0)
+            else:
+                self._event_queue.put_nowait((kind, payload))
+            return True
+        except queue.Full:
+            self._dropped_events += 1
+            self.manager.last_warning = (
+                f"recorder queue full; dropped={self._dropped_events}"
+            )
+            self._refresh_stats()
+            return False
+
+    def _start_event_consumer(self) -> None:
+        if self._event_thread and self._event_thread.is_alive():
+            return
+        self._event_stop.clear()
+
+        def consume() -> None:
+            while not self._event_stop.is_set():
+                try:
+                    kind, payload = self._event_queue.get(timeout=0.2)
+                except queue.Empty:
+                    continue
+                try:
+                    if kind == "stop":
+                        break
+                    if kind == "event":
+                        self.manager.handle_raw_event(payload)
+                    elif kind == "heartbeat":
+                        self.manager.heartbeat()
+                    elif kind == "pause":
+                        self.manager.pause()
+                    elif kind == "resume":
+                        self.manager.resume()
+                except Exception as exc:
+                    self.manager.last_warning = str(exc)
+                finally:
+                    self._event_queue.task_done()
+                    self._refresh_stats()
+            self._event_stop.set()
+
+        self._event_thread = threading.Thread(
+            target=consume,
+            name="ui-inspector-action-consumer",
+            daemon=True,
+        )
+        self._event_thread.start()
+
+    def _start_surface_recorder(self) -> None:
+        def on_event(ev: RawUserEvent) -> None:
+            # Preserve the observed order and bound memory/thread growth.
+            if self.surface == "windows" and not (ev.meta or {}).get("event_id"):
+                ev.meta["event_id"] = f"windows:{uuid.uuid4().hex}"
+            self.journal.append(ev, channel="semantic")
+            self._submit("event", ev)
+
+        def on_raw_event(payload: dict[str, Any]) -> None:
+            self.journal.append(payload, channel="browser_dom")
+
+        if self.surface == "browser":
+            from surfaces.browser import BrowserRecorder
+
+            self._recorder = BrowserRecorder(
+                on_event=on_event, on_raw_event=on_raw_event
+            )
+        else:
+            from surfaces.windows import WindowsRecorder
+
+            self._recorder = WindowsRecorder(
+                on_event=on_event,
+                exclude_hwnds=self.exclude_hwnds,
+            )
+            self._seed_windows_element_atlas()
+        self._recorder.start()
+
+    def _on_evidence_dump(self, dump: Any) -> None:
+        recorder = self._recorder
+        if self.surface != "windows" or recorder is None or not hasattr(recorder, "learn_elements"):
+            return
+        try:
+            recorder.learn_elements(
+                list(getattr(dump, "locatable", None) or []),
+                window=dict(getattr(dump, "window", None) or {}),
+            )
+        except Exception:
+            pass
+
+    def _seed_windows_element_atlas(self) -> None:
+        """Reuse recent captures as the event-time element repository.
+
+        Values are discarded by ``learn_elements``; only stable names and
+        geometry are retained. This lets a new recording identify custom
+        editors before their slow accessibility provider completes a new dump.
+        """
+        recorder = self._recorder
+        if recorder is None or not hasattr(recorder, "learn_elements"):
+            return
+        try:
+            siblings = sorted(
+                (
+                    path for path in self.flow_dir.parent.iterdir()
+                    if path.is_dir() and path != self.flow_dir
+                ),
+                key=lambda path: path.stat().st_mtime,
+                reverse=True,
+            )[:8]
+        except Exception:
+            return
+        learned_classes: set[str] = set()
+        for sibling in siblings:
+            observations = sibling / "observations"
+            if not observations.is_dir():
+                continue
+            try:
+                metas = sorted(
+                    observations.glob("*/meta.json"),
+                    key=lambda path: path.stat().st_mtime,
+                    reverse=True,
+                )
+            except Exception:
+                continue
+            for meta_path in metas:
+                try:
+                    import json
+
+                    meta = json.loads(meta_path.read_text(encoding="utf-8"))
+                    if str(meta.get("surface") or "") != "windows":
+                        continue
+                    selection = dict(meta.get("selection") or {})
+                    class_name = str(
+                        selection.get("chosen_class")
+                        or selection.get("event_class")
+                        or ""
+                    )
+                    if not class_name or class_name in learned_classes:
+                        continue
+                    locatable_path = meta_path.parent / "locatable.json"
+                    rows = json.loads(locatable_path.read_text(encoding="utf-8"))
+                    recorder.learn_elements(
+                        rows,
+                        window={
+                            "class_name": class_name,
+                            "hwnd": int(selection.get("chosen_hwnd") or 0),
+                            "pid": int(selection.get("chosen_pid") or 0),
+                            "title": str(selection.get("event_title") or ""),
+                        },
+                    )
+                    learned_classes.add(class_name)
+                except Exception:
+                    continue
+            if "WeWorkWindow" in learned_classes and "SearchResultWindow2" in learned_classes:
+                break
+
+    def pause(self) -> None:
+        if self._recorder:
+            self._recorder.pause()
+        self._submit("pause")
+
+    def resume(self) -> None:
+        if self._recorder:
+            self._recorder.resume()
+        self._submit("resume")
+
+    def stop(self) -> None:
+        recorder_stop_error: Optional[BaseException] = None
+        if self._recorder:
+            try:
+                self._recorder.stop()
+            except Exception as exc:
+                recorder_stop_error = exc
+            if hasattr(self._recorder, "health"):
+                try:
+                    self._recorder_health = self._recorder.health()
+                except Exception as exc:
+                    self._recorder_health = {"last_error": str(exc)}
+            self._recorder = None
+        if self._event_thread and self._event_thread.is_alive():
+            self._submit("stop")
+            self._event_thread.join(timeout=35.0)
+            if self._event_thread.is_alive():
+                self.manager.last_warning = "event consumer did not stop within 35s"
+                raise RuntimeError(self.manager.last_warning)
+            else:
+                self._event_thread = None
+        self._event_stop.set()
+        if self._evidence is not None:
+            self._evidence.stop(timeout=3.0)
+        self.manager.stop()
+        # Point flow.json v2 fields (already present from create_flow).
+        try:
+            flow = art.load_flow(self.flow_dir)
+            v2 = dict(flow.get("v2") or {})
+            v2["action_count"] = len(self.manager.action_flow.actions)
+            v2["state_count"] = len(self.manager.action_flow.states)
+            v2["observation_count"] = len(
+                self.manager.snapshots.observation_ids()
+            )
+            v2["element_count"] = len(self.manager.catalog.catalog.elements)
+            v2["raw_event_journal"] = "journal/raw_events.jsonl"
+            v2["pages"] = "pages.json"
+            v2["diagnostics_dir"] = "diagnostics/"
+            v2["validation"] = "diagnostics/validation.json"
+            v2["replay_report"] = "diagnostics/replay_report.json"
+            flow["v2"] = v2
+            art.save_flow(self.flow_dir, flow)
+        except Exception:
+            pass
+        self._write_diagnostics()
+        self._refresh_stats()
+        if recorder_stop_error is not None:
+            self.manager.last_warning = str(recorder_stop_error)
+            raise RuntimeError(str(recorder_stop_error)) from recorder_stop_error
+
+    def checkpoint(self) -> Any:
+        return self.manager.record_checkpoint()
+
+    def heartbeat(self) -> None:
+        self._submit("heartbeat")
+
+    def stats(self) -> dict[str, Any]:
+        with self._stats_lock:
+            return dict(self._stats_cache)
+
+    def _refresh_stats(self) -> None:
+        st = self.manager.stats()
+        st["surface_recorder"] = (
+            self._recorder.status if self._recorder else "off"
+        )
+        st["queued_events"] = self._event_queue.qsize()
+        st["dropped_events"] = self._dropped_events
+        if self._recorder is not None and hasattr(self._recorder, "health"):
+            try:
+                st["input_backend"] = self._recorder.health()
+            except Exception as exc:
+                st["input_backend"] = {"last_error": str(exc)}
+        elif self._recorder_health:
+            st["input_backend"] = dict(self._recorder_health)
+        if self._evidence is not None:
+            st["evidence_capture_count"] = self._evidence.capture_count
+            st["evidence_dropped"] = self._evidence.dropped
+            st["evidence_queued"] = self._evidence._queue.qsize()
+            st["transient_evidence_queued"] = self._evidence._urgent_queue.qsize()
+            st["evidence_failures"] = self._evidence.failure_count
+        with self._stats_lock:
+            self._stats_cache = st
+
+    def _write_diagnostics(self) -> None:
+        pages: dict[str, dict[str, Any]] = {}
+        lags: list[int] = []
+        for action in self.manager._actions_by_id.values():
+            raw = dict(action.raw_event or {})
+            meta = dict(raw.get("meta") or {})
+            page_id = str(meta.get("page_id") or "")
+            if page_id:
+                page = pages.setdefault(page_id, {"page_id": page_id, "urls": [], "document_ids": []})
+                url = str(raw.get("url") or (action.input or {}).get("url") or "")
+                document_id = str(meta.get("document_id") or "")
+                if url and url not in page["urls"]:
+                    page["urls"].append(url)
+                if document_id and document_id not in page["document_ids"]:
+                    page["document_ids"].append(document_id)
+            lag = (action.evidence or {}).get("evidence_lag_ms")
+            if isinstance(lag, int):
+                lags.append(lag)
+        diagnostics = self.flow_dir / "diagnostics"
+        diagnostics.mkdir(parents=True, exist_ok=True)
+        try:
+            art.write_json(self.flow_dir / "pages.json", {
+                "schema": "openclaw.ui_inspector.pages.v1",
+                "pages": list(pages.values()),
+            })
+            recorder_health = dict(self._recorder_health)
+            if self._recorder is not None and hasattr(self._recorder, "health"):
+                try:
+                    recorder_health = self._recorder.health()
+                except Exception as exc:
+                    recorder_health = {"last_error": str(exc)}
+            art.write_json(diagnostics / "metrics.json", {
+                "schema": "openclaw.ui_inspector.metrics.v1",
+                "raw_event_count": self.journal._sequence,
+                "action_count": len(self.manager.action_flow.actions),
+                "state_count": len(self.manager.action_flow.states),
+                "observation_count": len(self.manager.snapshots.observation_ids()),
+                "element_count": len(self.manager.catalog.catalog.elements),
+                "event_queue_dropped": self._dropped_events,
+                "evidence_queue_dropped": self._evidence.dropped if self._evidence else 0,
+                "evidence_capture_count": self._evidence.capture_count if self._evidence else 0,
+                "evidence_failure_count": self._evidence.failure_count if self._evidence else 0,
+                "evidence_lag_ms_max": max(lags) if lags else None,
+                "evidence_lag_ms_average": round(sum(lags) / len(lags), 2) if lags else None,
+                "input_backend": recorder_health,
+            })
+            from diagnostics import validate_capture, write_replay_preflight
+
+            art.write_json(diagnostics / "validation.json", validate_capture(self.flow_dir))
+            write_replay_preflight(self.flow_dir)
+        except Exception as exc:
+            self.manager.last_warning = f"diagnostics write failed: {exc}"

tools/ui_inspector/kit_browser.py

@@ -0,0 +1,729 @@
+"""Inspector-owned Playwright browser session (dev-time page dump).
+
+All Playwright sync API calls run on one dedicated worker thread
+(Playwright sync is not cross-thread safe).
+
+Dump / screenshot / recorder target the **currently visible** page
+(document.visibilityState === 'visible'), not a sticky pages[0].
+This keeps list ↔ detail tab switching aligned with what the user sees.
+
+Uses jiangchang_skill_core.rpa stealth launch parts + dump_page_sync.
+Profile lives under ui_inspector/_browser_profile (not account-manager).
+"""
+
+from __future__ import annotations
+
+import atexit
+import queue
+import threading
+import uuid
+from pathlib import Path
+from typing import Any, Callable, Optional
+
+_HERE = Path(__file__).resolve().parent
+_DEFAULT_PROFILE = _HERE / "_browser_profile"
+_DEFAULT_START_URL = "about:blank"
+
+_cmd_q: queue.Queue = queue.Queue()
+_worker: Optional[threading.Thread] = None
+_worker_lock = threading.Lock()
+
+# Owned exclusively by worker thread:
+_pw = None
+_context = None
+_page = None
+_started = False
+_bridge_installed = False
+_bridge_sink: Optional[Callable[[dict[str, Any]], None]] = None
+_bridge_lock = threading.Lock()
+_BRIDGE_NAME = "__openclawUiInspectorEmit"
+_page_ids: dict[int, str] = {}
+_page_refs: dict[int, Any] = {}
+_pages_by_id: dict[str, Any] = {}
+_frame_ids: dict[int, str] = {}
+_frame_refs: dict[int, Any] = {}
+
+
+def _page_id_for(page: Any) -> str:
+    if not _page_is_open(page):
+        return ""
+    key = id(page)
+    page_id = _page_ids.get(key) if _page_refs.get(key) is page else None
+    if not page_id:
+        page_id = f"page_{uuid.uuid4().hex[:12]}"
+        _page_ids[key] = page_id
+        _page_refs[key] = page
+    _pages_by_id[page_id] = page
+    return page_id
+
+
+def _frame_id_for(frame: Any, *, page_id: str = "") -> str:
+    if frame is None:
+        return ""
+    key = id(frame)
+    frame_id = _frame_ids.get(key) if _frame_refs.get(key) is frame else None
+    if not frame_id:
+        prefix = page_id or "page"
+        frame_id = f"{prefix}:frame_{uuid.uuid4().hex[:10]}"
+        _frame_ids[key] = frame_id
+        _frame_refs[key] = frame
+    return frame_id
+
+
+def _page_for_id(page_id: str) -> Any:
+    page = _pages_by_id.get(str(page_id or ""))
+    if _page_is_open(page):
+        return page
+    if page_id:
+        _pages_by_id.pop(str(page_id), None)
+    return None
+
+
+def kit_page_dump_available() -> bool:
+    try:
+        from jiangchang_skill_core.rpa.page_dump import dump_page_sync  # noqa: F401
+
+        return True
+    except ImportError:
+        return False
+
+
+def profile_dir() -> Path:
+    return _DEFAULT_PROFILE
+
+
+def _ensure_worker() -> None:
+    global _worker
+    with _worker_lock:
+        if _worker is not None and _worker.is_alive():
+            return
+
+        def loop() -> None:
+            while True:
+                item = _cmd_q.get()
+                if item is None:
+                    break
+                fn, args, kwargs, out_q = item
+                try:
+                    out_q.put(("ok", fn(*args, **kwargs)))
+                except BaseException as exc:
+                    out_q.put(("err", exc))
+
+        _worker = threading.Thread(target=loop, name="ui-inspector-pw", daemon=True)
+        _worker.start()
+        atexit.register(shutdown)
+
+
+def _call_on_worker(
+    fn: Callable[..., Any],
+    *args: Any,
+    timeout_sec: float = 30.0,
+    **kwargs: Any,
+) -> Any:
+    _ensure_worker()
+    out_q: queue.Queue = queue.Queue()
+    _cmd_q.put((fn, args, kwargs, out_q))
+    try:
+        status, payload = out_q.get(timeout=max(0.1, float(timeout_sec)))
+    except queue.Empty as exc:
+        raise RuntimeError(
+            f"ERROR:BROWSER_WORKER_TIMEOUT Playwright 命令超过 {timeout_sec:g} 秒未返回。"
+        ) from exc
+    if status == "err":
+        raise payload
+    return payload
+
+
+def _teardown_unlocked() -> None:
+    global _pw, _context, _page, _started, _bridge_installed, _bridge_sink
+    try:
+        if _context is not None:
+            _context.close()
+    except Exception:
+        pass
+    try:
+        if _pw is not None:
+            _pw.stop()
+    except Exception:
+        pass
+    _pw = None
+    _context = None
+    _page = None
+    _started = False
+    _bridge_installed = False
+    with _bridge_lock:
+        _bridge_sink = None
+    _page_ids.clear()
+    _page_refs.clear()
+    _pages_by_id.clear()
+    _frame_ids.clear()
+    _frame_refs.clear()
+
+
+def _goto_safe(page: Any, start_url: str) -> dict[str, Any]:
+    url = (start_url or "").strip()
+    if not url or url == "about:blank":
+        return {"ok": True, "requested_url": url, "actual_url": page.url or ""}
+    try:
+        page.goto(url, wait_until="domcontentloaded", timeout=60_000)
+    except Exception as exc:
+        actual = ""
+        try:
+            actual = str(page.url or "")
+        except Exception:
+            pass
+        # Some sites keep loading after domcontentloaded timeout.  A real HTTP
+        # page is usable, but the warning must remain visible to callers.
+        if actual.startswith(("http://", "https://")):
+            return {
+                "ok": True,
+                "requested_url": url,
+                "actual_url": actual,
+                "warning": str(exc),
+            }
+        raise RuntimeError(
+            f"ERROR:NAVIGATION_FAILED requested={url!r} actual={actual!r}: {exc}"
+        ) from exc
+    actual = str(page.url or "")
+    if not actual or actual == "about:blank" or actual.startswith("about:"):
+        raise RuntimeError(
+            f"ERROR:NAVIGATION_FAILED requested={url!r} actual={actual!r}"
+        )
+    return {"ok": True, "requested_url": url, "actual_url": actual}
+
+
+def _page_is_open(page: Any) -> bool:
+    try:
+        return page is not None and not page.is_closed()
+    except Exception:
+        return False
+
+
+def _open_pages(context: Any) -> list[Any]:
+    if context is None:
+        return []
+    out: list[Any] = []
+    try:
+        pages = list(context.pages or [])
+    except Exception:
+        return []
+    for p in pages:
+        if _page_is_open(p):
+            _page_id_for(p)
+            out.append(p)
+    return out
+
+
+def _page_visibility(page: Any) -> str:
+    """Return document.visibilityState or '' on failure."""
+    if not _page_is_open(page):
+        return ""
+    try:
+        state = page.evaluate("() => document.visibilityState")
+        return str(state or "").strip().lower()
+    except Exception:
+        return ""
+
+
+def resolve_active_page(
+    context: Any,
+    *,
+    preferred: Any = None,
+) -> tuple[Any, dict[str, Any]]:
+    """Pick the page the user is looking at (visible tab).
+
+    Returns (page, meta). Prefer visibilityState===visible; fall back to
+    preferred if still open, else first open page.
+    """
+    pages = _open_pages(context)
+    meta: dict[str, Any] = {
+        "resolve": "none",
+        "page_count": len(pages),
+        "visible_count": 0,
+    }
+    if not pages:
+        raise RuntimeError("ERROR:NO_OPEN_PAGE Playwright 会话中没有可用标签页。")
+
+    visible: list[Any] = []
+    for p in pages:
+        if _page_visibility(p) == "visible":
+            visible.append(p)
+    meta["visible_count"] = len(visible)
+
+    if len(visible) == 1:
+        meta["resolve"] = "visibility"
+        return visible[0], meta
+    if len(visible) > 1:
+        # Multiple windows each with a visible tab — prefer preferred if among them
+        if preferred is not None and preferred in visible:
+            meta["resolve"] = "visibility_preferred"
+            return preferred, meta
+        meta["resolve"] = "visibility_first"
+        return visible[0], meta
+
+    if preferred is not None and _page_is_open(preferred) and preferred in pages:
+        meta["resolve"] = "preferred_fallback"
+        return preferred, meta
+
+    meta["resolve"] = "first_open"
+    return pages[0], meta
+
+
+def _sync_active_page(*, update_global: bool = True) -> tuple[Any, dict[str, Any]]:
+    """Resolve visible page; optionally assign to module ``_page``."""
+    global _page
+    if _context is None:
+        raise RuntimeError("ERROR:NO_BROWSER_SESSION Playwright 会话未启动。")
+    page, meta = resolve_active_page(_context, preferred=_page)
+    if update_global:
+        _page = page
+    try:
+        meta["url"] = page.url or ""
+        meta["title"] = page.title() or ""
+    except Exception:
+        meta["url"] = ""
+        meta["title"] = ""
+    meta["page_id"] = _page_id_for(page)
+    return page, meta
+
+
+def _resolve_capture_page(page_id: str = "") -> tuple[Any, dict[str, Any]]:
+    global _page
+    requested = str(page_id or "")
+    if requested:
+        page = _page_for_id(requested)
+        if page is not None:
+            _page = page
+            meta = {
+                "resolve": "event_page_id",
+                "page_id": requested,
+                "page_count": len(_open_pages(_context)),
+                "visible_count": None,
+            }
+            try:
+                meta["url"] = page.url or ""
+                meta["title"] = page.title() or ""
+            except Exception:
+                meta["url"] = ""
+                meta["title"] = ""
+            return page, meta
+        raise RuntimeError(f"ERROR:EVENT_PAGE_CLOSED page_id={requested}")
+    return _sync_active_page(update_global=True)
+
+
+def _ensure_session_worker(
+    *,
+    start_url: str = _DEFAULT_START_URL,
+    force_goto: bool = False,
+) -> dict[str, Any]:
+    global _pw, _context, _page, _started
+    want = (start_url or "").strip() or _DEFAULT_START_URL
+
+    if _started and _context is not None:
+        try:
+            navigation = None
+            pages = _open_pages(_context)
+            if not pages:
+                _teardown_unlocked()
+            else:
+                # Start-flow navigation: prefer first open tab (stable home),
+                # not whatever detail tab happens to be visible.
+                if force_goto and want and want != "about:blank":
+                    home = pages[0]
+                    cur = ""
+                    try:
+                        cur = home.url or ""
+                    except Exception:
+                        cur = ""
+                    if cur.rstrip("/") != want.rstrip("/"):
+                        navigation = _goto_safe(home, want)
+                    _page = home
+                else:
+                    _page, _meta = _sync_active_page(update_global=True)
+                    if want and want != "about:blank" and (
+                        (_page.url or "") in {"", "about:blank"}
+                        or str(_page.url or "").startswith("about:")
+                    ):
+                        navigation = _goto_safe(_page, want)
+                        _page, _meta = _sync_active_page(update_global=True)
+                try:
+                    live_url = _page.url or ""
+                except Exception:
+                    live_url = ""
+                return {
+                    "ok": True,
+                    "reused": True,
+                    "url": live_url,
+                    "profile_dir": str(profile_dir()),
+                    "page_count": len(_open_pages(_context)),
+                    "navigation": navigation,
+                }
+        except Exception:
+            _teardown_unlocked()
+
+    if not kit_page_dump_available():
+        raise RuntimeError(
+            "ERROR:KIT_PAGE_DUMP_MISSING 需要 jiangchang-platform-kit>=1.2.11 "
+            "(jiangchang_skill_core.rpa.page_dump)。请升级 kit 后重试。"
+        )
+
+    try:
+        from playwright.sync_api import sync_playwright
+    except ImportError as exc:
+        raise RuntimeError(
+            "ERROR:MISSING_PLAYWRIGHT 未安装 playwright。"
+            "请在与技能相同的 Python 环境中安装后重试。"
+        ) from exc
+
+    from jiangchang_skill_core.runtime_env import find_chrome_executable
+    from jiangchang_skill_core.rpa.stealth import (
+        STEALTH_INIT_SCRIPT,
+        persistent_context_launch_parts,
+        stealth_enabled,
+    )
+
+    chrome = find_chrome_executable()
+    if not chrome:
+        raise RuntimeError(
+            "ERROR:MISSING_BROWSER 未检测到 Chrome 或 Edge，请安装后重试。"
+        )
+
+    prof = profile_dir()
+    prof.mkdir(parents=True, exist_ok=True)
+    args, ignore = persistent_context_launch_parts(extra_args=["--start-maximized"])
+    launch_kwargs: dict[str, Any] = dict(
+        user_data_dir=str(prof),
+        headless=False,
+        executable_path=chrome,
+        locale="zh-CN",
+        no_viewport=True,
+        args=args,
+    )
+    if ignore is not None:
+        launch_kwargs["ignore_default_args"] = ignore
+
+    _pw = sync_playwright().start()
+    try:
+        _context = _pw.chromium.launch_persistent_context(**launch_kwargs)
+        if stealth_enabled():
+            _context.add_init_script(STEALTH_INIT_SCRIPT)
+        if _context.pages:
+            _page = _context.pages[0]
+        else:
+            _page = _context.new_page()
+        navigation = None
+        if want and want != "about:blank":
+            navigation = _goto_safe(_page, want)
+        elif want and (
+            _page.url in {"", "about:blank"} or str(_page.url).startswith("about:")
+        ):
+            navigation = _goto_safe(_page, want)
+        _started = True
+        return {
+            "ok": True,
+            "reused": False,
+            "url": _page.url,
+            "profile_dir": str(prof),
+            "page_count": len(_open_pages(_context)),
+            "navigation": navigation,
+        }
+    except Exception:
+        _teardown_unlocked()
+        raise
+
+
+def _dump_worker(*, max_nodes: int, locatable_limit: int, page_id: str = "") -> dict[str, Any]:
+    from jiangchang_skill_core.rpa.page_dump import dump_page_sync
+
+    _ensure_session_worker()
+    page, meta = _resolve_capture_page(page_id)
+    result = dump_page_sync(page, max_nodes=max_nodes, locatable_limit=locatable_limit)
+    stats = dict(result.get("stats") or {})
+    stats["active_page"] = meta
+    result["stats"] = stats
+    return result
+
+
+def _screenshot_worker(*, page_id: str = "") -> bytes:
+    _ensure_session_worker()
+    page, _meta = _resolve_capture_page(page_id)
+    return page.screenshot(type="png", full_page=False)
+
+
+def _session_info_worker() -> dict[str, Any]:
+    if not _started or _context is None:
+        return {"active": False}
+    try:
+        page, meta = _sync_active_page(update_global=True)
+        return {
+            "active": True,
+            "url": page.url,
+            "title": page.title(),
+            "profile_dir": str(profile_dir()),
+            "page_count": meta.get("page_count"),
+            "resolve": meta.get("resolve"),
+            "page_id": meta.get("page_id"),
+        }
+    except Exception as exc:
+        return {"active": False, "error": str(exc)}
+
+
+def _dispatch_bridge_event(source: Any, payload: Any) -> None:
+    if not isinstance(payload, dict):
+        return
+    item = dict(payload)
+    try:
+        page = source.get("page") if isinstance(source, dict) else None
+        frame = source.get("frame") if isinstance(source, dict) else None
+        page_id = _page_id_for(page)
+        item["page_id"] = page_id
+        item["frame_id"] = _frame_id_for(frame, page_id=page_id)
+    except Exception:
+        pass
+    with _bridge_lock:
+        sink = _bridge_sink
+    if sink is not None:
+        try:
+            sink(item)
+        except Exception:
+            pass
+
+
+def _install_event_bridge_worker(
+    *, init_script: str, sink: Callable[[dict[str, Any]], None]
+) -> dict[str, Any]:
+    global _bridge_installed, _bridge_sink
+    _ensure_session_worker()
+    assert _context is not None
+    with _bridge_lock:
+        _bridge_sink = sink
+    if not _bridge_installed:
+        _context.expose_binding(_BRIDGE_NAME, _dispatch_bridge_event)
+        _bridge_installed = True
+    _context.add_init_script(init_script)
+    installed = 0
+    for page in _open_pages(_context):
+        try:
+            page.evaluate(init_script)
+            installed += 1
+        except Exception:
+            pass
+    return {"ok": True, "binding": _BRIDGE_NAME, "installed_pages": installed}
+
+
+def install_event_bridge(
+    init_script: str, sink: Callable[[dict[str, Any]], None]
+) -> dict[str, Any]:
+    """Install a context-level recorder bridge before user actions begin."""
+    return _call_on_worker(
+        _install_event_bridge_worker,
+        init_script=init_script,
+        sink=sink,
+        timeout_sec=15.0,
+    )
+
+
+def clear_event_bridge() -> None:
+    global _bridge_sink
+    with _bridge_lock:
+        _bridge_sink = None
+
+
+def ensure_session(
+    *,
+    start_url: str = _DEFAULT_START_URL,
+    force_goto: bool = False,
+) -> dict[str, Any]:
+    """Start or reuse headed persistent context; return status dict.
+
+    force_goto=True: navigate to start_url even when session is reused
+    (used when「开始流程」带起始 URL).
+    """
+    return _call_on_worker(
+        _ensure_session_worker,
+        start_url=start_url,
+        force_goto=force_goto,
+        timeout_sec=75.0,
+    )
+
+
+def session_info() -> dict[str, Any]:
+    return _call_on_worker(_session_info_worker, timeout_sec=10.0)
+
+
+def dump_current_page(
+    *, max_nodes: int = 8000, locatable_limit: int = 300, page_id: str = ""
+) -> dict[str, Any]:
+    """Dump an event-origin page, or resolve the active page as fallback."""
+    return _call_on_worker(
+        _dump_worker,
+        max_nodes=max_nodes,
+        locatable_limit=locatable_limit,
+        page_id=page_id,
+        timeout_sec=30.0,
+    )
+
+
+def screenshot_png(*, page_id: str = "") -> bytes:
+    return _call_on_worker(_screenshot_worker, page_id=page_id, timeout_sec=20.0)
+
+
+def page_id_for(page: Any) -> str:
+    """Return the stable id for a Page while already on the browser worker."""
+    return _page_id_for(page)
+
+
+def _validate_element_worker(
+    *, page_id: str, element: dict[str, Any], frame: Optional[dict[str, Any]] = None
+) -> dict[str, int]:
+    page, _meta = _resolve_capture_page(page_id)
+    scope: Any = page
+    frame_meta = dict(frame or {})
+    frame_url = str(frame_meta.get("url") or "")
+    frame_name = str(frame_meta.get("name") or "")
+    if frame_url or frame_name:
+        try:
+            for candidate in page.frames:
+                if frame_name and candidate.name == frame_name:
+                    scope = candidate
+                    break
+                if frame_url and candidate.url == frame_url:
+                    scope = candidate
+                    break
+        except Exception:
+            pass
+
+    el = dict(element or {})
+    role = str(el.get("role") or "").strip()
+    name = str(el.get("name") or "").strip()
+    test_id = str(el.get("test_id") or "").strip()
+    label = str(el.get("label") or "").strip()
+    placeholder = str(el.get("placeholder") or "").strip()
+    name_attr = str(el.get("name_attr") or "").strip()
+    href = str(el.get("href") or "").strip()
+    text_value = str(el.get("text") or "").strip()
+    css_id = str(el.get("id") or "").strip()
+    css_path = str(el.get("css") or el.get("css_path") or "").strip()
+    stable_css = str(el.get("stable_css") or "").strip()
+    semantic_role = str(el.get("semantic_role") or "").strip()
+    semantic_name = str(el.get("semantic_name") or "").strip()
+    counts: dict[str, int] = {}
+
+    def count(strategy: str, build: Callable[[], Any]) -> None:
+        try:
+            counts[strategy] = int(build().count())
+        except Exception:
+            pass
+
+    def count_actionable(strategy: str, build: Callable[[], Any]) -> None:
+        try:
+            locator = build()
+            total = min(int(locator.count()), 50)
+            actionable = 0
+            for index in range(total):
+                item = locator.nth(index)
+                if item.is_visible() and item.is_enabled():
+                    actionable += 1
+            counts[strategy] = actionable
+        except Exception:
+            pass
+
+    if role and name:
+        # Playwright accessible names normalize whitespace (SVG/icon buttons are
+        # a common example), while DOM innerText may not.  Non-exact still
+        # yields a trustworthy uniqueness count and mirrors getByRole defaults.
+        count("role", lambda: scope.get_by_role(role, name=name, exact=False))
+    elif role and bool(el.get("contenteditable")):
+        count("role", lambda: scope.get_by_role(role))
+    if semantic_role and semantic_name:
+        count_actionable(
+            "semantic_role",
+            lambda: scope.get_by_role(
+                semantic_role, name=semantic_name, exact=False
+            ),
+        )
+    if test_id:
+        count("test_id", lambda: scope.get_by_test_id(test_id))
+    if label:
+        count("label", lambda: scope.get_by_label(label, exact=True))
+    if placeholder:
+        count("placeholder", lambda: scope.get_by_placeholder(placeholder, exact=True))
+    if name_attr:
+        escaped_name = name_attr.replace("\\", "\\\\").replace('"', '\\"')
+        count("name_attr", lambda: scope.locator(f'[name="{escaped_name}"]'))
+    if href:
+        escaped_href = href.replace("\\", "\\\\").replace('"', '\\"')
+        count("href", lambda: scope.locator(f'a[href="{escaped_href}"]'))
+    if text_value and len(text_value) <= 80:
+        count("text", lambda: scope.get_by_text(text_value, exact=True))
+    if css_id and not css_id[0].isdigit():
+        escaped = css_id.replace("\\", "\\\\").replace('"', '\\"')
+        count("css_id", lambda: scope.locator(f'[id="{escaped}"]'))
+    if css_path:
+        count("css", lambda: scope.locator(css_path))
+    if stable_css:
+        count("stable_css", lambda: scope.locator(stable_css))
+    return counts
+
+
+def validate_element_on_page(
+    *, page_id: str, element: dict[str, Any], frame: Optional[dict[str, Any]] = None
+) -> dict[str, int]:
+    """Count locator candidates against the exact live event-origin page."""
+    if not page_id:
+        return {}
+    return _call_on_worker(
+        _validate_element_worker,
+        page_id=page_id,
+        element=element,
+        frame=frame,
+        timeout_sec=10.0,
+    )
+
+
+def call_on_page(fn: Callable[[Any], Any]) -> Any:
+    """Run ``fn(page)`` on the **visible** Playwright page (worker thread)."""
+
+    def _run() -> Any:
+        _ensure_session_worker()
+        page, _meta = _sync_active_page(update_global=True)
+        return fn(page)
+
+    return _call_on_worker(_run, timeout_sec=30.0)
+
+
+def call_on_each_open_page(fn: Callable[[Any], Any]) -> list[Any]:
+    """Run ``fn(page)`` for every open tab; returns list of per-page results."""
+
+    def _run() -> list[Any]:
+        _ensure_session_worker()
+        assert _context is not None
+        pages = _open_pages(_context)
+        # Keep _page synced to visible for dump consistency
+        try:
+            _sync_active_page(update_global=True)
+        except Exception:
+            pass
+        results: list[Any] = []
+        for p in pages:
+            try:
+                results.append(fn(p))
+            except Exception as exc:
+                results.append({"error": str(exc)})
+        return results
+
+    return _call_on_worker(_run, timeout_sec=20.0)
+
+
+def shutdown() -> None:
+    def _stop() -> None:
+        _teardown_unlocked()
+
+    try:
+        _call_on_worker(_stop, timeout_sec=5.0)
+    except Exception:
+        pass
+    try:
+        _cmd_q.put(None)
+    except Exception:
+        pass

tools/ui_inspector/kit_wake.py

@@ -3,6 +3,9 @@
 - 主窗 WeWorkWindow → ``ensure_ui_ready("wecom")``
 - 搜索浮层 SearchResultWindow2 → ``ensure_ui_ready("wecom_search")``
 未安装 kit 时降级为跳过，不阻断采集。
+
+Warm cache key = profile + hwnd（经 WeCom Adapter），避免主窗 warm
+状态错误代表 SearchResultWindow2 已暖。
 """
 
 from __future__ import annotations
@@ -10,9 +13,13 @@ from __future__ import annotations
 import time
 from typing import Any, Optional
 
-_CACHE_TTL_SEC = 45.0
+# A successful provider probe remains valid for the lifetime of a normal
+# recording flow.  The old 45 second TTL made long desktop flows re-run an
+# expensive WeCom wake in the middle of recording and produced minute-late
+# evidence.
+_CACHE_TTL_SEC = 600.0
 
-# profile -> cache
+# cache_key -> cache (profile:hwnd:N)
 _caches: dict[str, dict[str, Any]] = {}
 
 _PROFILE_WECOM = "wecom"
@@ -24,8 +31,26 @@ _THRESHOLDS = {
 }
 
 
+def _wecom_adapter():
+    try:
+        from adapters.wecom import wecom_adapter
+
+        return wecom_adapter
+    except Exception:
+        try:
+            from .adapters.wecom import wecom_adapter  # type: ignore
+
+            return wecom_adapter
+        except Exception:
+            return None
+
+
 def resolve_wake_profile(*, class_name: str = "", title: str = "") -> Optional[str]:
     """返回应调用的 kit profile id；非企微相关则 None。"""
+    ad = _wecom_adapter()
+    if ad is not None:
+        return ad.wake_profile(class_name=class_name, title=title)
+    # Compatibility fallback if adapter import fails
     cls = (class_name or "").strip()
     title = title or ""
     if cls == "SearchResultWindow2":
@@ -50,6 +75,13 @@ def kit_available() -> bool:
         return False
 
 
+def _cache_key(profile: str, hwnd: Optional[int]) -> str:
+    ad = _wecom_adapter()
+    if ad is not None:
+        return ad.cache_key(profile=profile, hwnd=int(hwnd or 0))
+    return f"{profile}:hwnd:{int(hwnd or 0)}"
+
+
 def auto_ensure_if_wecom(
     *,
     class_name: str = "",
@@ -68,17 +100,22 @@ def auto_ensure_if_wecom(
             "available": kit_available(),
         }
 
-    warm_min = int(_THRESHOLDS.get(profile, 80))
+    ad = _wecom_adapter()
+    warm_min = int(ad.warm_threshold(profile) if ad else _THRESHOLDS.get(profile, 80))
+    key = _cache_key(profile, hwnd)
     cache = _caches.setdefault(
-        profile, {"t": 0.0, "warm": False, "hwnd": 0, "findall": 0}
+        key, {"t": 0.0, "warm": False, "hwnd": int(hwnd or 0), "findall": 0, "profile": profile}
     )
     now = time.monotonic()
     hint = int(findall_hint) if findall_hint is not None else -1
+    # Require same hwnd in cache — main window warm must not cover search popup.
+    cached_hwnd = int(cache.get("hwnd") or 0)
+    same_window = (not hwnd) or (cached_hwnd == int(hwnd))
     if (
         not force
+        and same_window
         and cache.get("warm")
         and (now - float(cache.get("t") or 0.0)) < _CACHE_TTL_SEC
-        and (hint < 0 or hint >= warm_min)
     ):
         return {
             "applicable": True,
@@ -91,6 +128,28 @@ def auto_ensure_if_wecom(
             "findall": int(cache.get("findall") or 0),
             "hwnd": int(cache.get("hwnd") or 0),
             "profile": profile,
+            "cache_key": key,
+            "observed_findall_hint": hint,
+        }
+
+    # A direct probe is authoritative. Do not run an invasive wake merely
+    # because this process did not create the cache entry itself.
+    if not force and hint >= warm_min:
+        cache.update(
+            {"t": now, "warm": True, "hwnd": int(hwnd or 0), "findall": hint}
+        )
+        return {
+            "applicable": True,
+            "available": kit_available(),
+            "skipped": True,
+            "reason": "probe_warm",
+            "ok": True,
+            "warm": True,
+            "wake_applied": False,
+            "findall": hint,
+            "hwnd": int(hwnd or 0),
+            "profile": profile,
+            "cache_key": key,
         }
 
     try:
@@ -157,16 +216,49 @@ def auto_ensure_if_wecom(
         "profile": profile,
         "markers_found": list(after.markers_found) if after is not None else [],
     }
+    report["cache_key"] = key
     if result.warm:
-        cache["t"] = now
-        cache["warm"] = True
-        cache["hwnd"] = report["hwnd"]
-        cache["findall"] = findall
+        # Persist under result hwnd so main≠search caches stay separate.
+        final_key = _cache_key(profile, report["hwnd"] or hwnd)
+        final_cache = _caches.setdefault(
+            final_key,
+            {"t": 0.0, "warm": False, "hwnd": int(report["hwnd"] or 0), "findall": 0, "profile": profile},
+        )
+        final_cache["t"] = now
+        final_cache["warm"] = True
+        final_cache["hwnd"] = int(report["hwnd"] or hwnd or 0)
+        final_cache["findall"] = findall
+        report["cache_key"] = final_key
     else:
         cache["warm"] = False
     return report
 
 
+def note_provider_warm(
+    *, class_name: str = "", title: str = "", hwnd: int = 0, findall: int = 0
+) -> bool:
+    """Pin a proven-warm provider so recorder evidence never re-wakes it."""
+    profile = resolve_wake_profile(class_name=class_name, title=title)
+    adapter = _wecom_adapter()
+    threshold = int(
+        adapter.warm_threshold(profile)
+        if adapter and profile
+        else _THRESHOLDS.get(profile or "", 80)
+    )
+    count = int(findall or 0)
+    if not profile or count < threshold:
+        return False
+    key = _cache_key(profile, hwnd)
+    _caches[key] = {
+        "t": time.monotonic(),
+        "warm": True,
+        "hwnd": int(hwnd or 0),
+        "findall": count,
+        "profile": profile,
+    }
+    return True
+
+
 def format_kit_wake_line(report: Optional[dict[str, Any]]) -> str:
     if not report:
         return "kit_wake: (none)"

tools/ui_inspector/locators/__init__.py

@@ -0,0 +1,13 @@
+"""Locator compilers + validators + element catalog."""
+
+from .browser import compile_browser_locators
+from .catalog import ElementCatalogService
+from .validator import validate_locator_against_rows
+from .windows import compile_windows_locators
+
+__all__ = [
+    "compile_browser_locators",
+    "compile_windows_locators",
+    "validate_locator_against_rows",
+    "ElementCatalogService",
+]

tools/ui_inspector/locators/base.py

@@ -0,0 +1,74 @@
+"""Shared locator compiler helpers."""
+
+from __future__ import annotations
+
+from typing import Any, Optional
+
+from models.locator import LocatorCandidate
+
+
+def rank_and_select_primary(
+    candidates: list[LocatorCandidate],
+) -> tuple[Optional[LocatorCandidate], list[LocatorCandidate]]:
+    """
+    Pick production primary only if unique and not coordinate-based.
+    Fallback list excludes primary.
+    """
+    ranked = sorted(
+        candidates,
+        key=lambda c: (
+            0 if c.production_eligible else 1,
+            0 if c.unique else 1,
+            0 if c.uses_coordinates else 0,
+            1 if c.uses_coordinates else 0,
+            1 if c.uses_index else 0,
+            -float(c.stability_score or 0.0),
+        ),
+    )
+    primary: Optional[LocatorCandidate] = None
+    for c in ranked:
+        if c.production_eligible and not c.uses_coordinates:
+            primary = c
+            break
+    fallbacks = [c for c in ranked if c is not primary]
+    return primary, fallbacks
+
+
+def count_matches(
+    rows: list[dict[str, Any]],
+    *,
+    properties: dict[str, Any],
+    scope: Optional[dict[str, Any]] = None,
+) -> int:
+    """Count rows matching structured properties (snapshot / live tree)."""
+    props = {k: v for k, v in (properties or {}).items() if v not in (None, "")}
+    if not props:
+        return 0
+    scope = scope or {}
+    n = 0
+    for r in rows or []:
+        if not isinstance(r, dict):
+            continue
+        if scope.get("window_class"):
+            # optional soft filter if row carries window info
+            rc = r.get("window_class") or r.get("window_class_name")
+            if rc and rc != scope.get("window_class"):
+                continue
+        ok = True
+        for k, v in props.items():
+            if k == "control_type":
+                rv = r.get("control_type") or r.get("role") or ""
+            elif k == "role":
+                rv = r.get("role") or r.get("control_type") or ""
+            elif k == "text":
+                rv = r.get("text") or r.get("value") or r.get("name") or ""
+            elif k == "test_id":
+                rv = r.get("test_id") or r.get("automation_id") or ""
+            else:
+                rv = r.get(k)
+            if str(rv or "") != str(v):
+                ok = False
+                break
+        if ok:
+            n += 1
+    return n

tools/ui_inspector/locators/browser.py

@@ -0,0 +1,190 @@
+"""Browser locator compiler (role / test id / label / placeholder / text / css id)."""
+
+from __future__ import annotations
+
+from typing import Any, Optional
+
+from models.locator import LocatorCandidate
+
+from .base import count_matches, rank_and_select_primary
+
+
+def _expr(strategy: str, props: dict[str, Any]) -> str:
+    parts = [f"{k}={v!r}" for k, v in props.items() if v not in (None, "")]
+    return f"{strategy}:" + ",".join(parts)
+
+
+def compile_browser_locators(
+    element: dict[str, Any],
+    *,
+    rows: Optional[list[dict[str, Any]]] = None,
+    frame: Optional[dict[str, Any]] = None,
+) -> list[LocatorCandidate]:
+    """
+    Generate ordered Playwright-oriented locator candidates.
+
+    Priority roughly: role+name → test id → label → placeholder → text → css id → scoped.
+    Coordinates are never emitted as production candidates.
+    """
+    el = element or {}
+    rows = list(rows or [])
+    scope = {"frame_url": (frame or {}).get("url") or "", "frame_name": (frame or {}).get("name") or ""}
+    cands: list[LocatorCandidate] = []
+
+    role = str(el.get("role") or el.get("control_type") or "").strip()
+    name = str(el.get("name") or "").strip()
+    text = str(el.get("text") or el.get("value") or "").strip()
+    is_editable = bool(el.get("contenteditable"))
+    if is_editable:
+        # Draft.js/contenteditable often exposes current user text as its
+        # accessible name.  That value is payload, never element identity.
+        current_value = str(el.get("value") or "").strip()
+        if name == current_value or name == text:
+            name = ""
+        text = ""
+    test_id = str(el.get("test_id") or el.get("data_testid") or el.get("automation_id") or "").strip()
+    label = str(el.get("label") or "").strip()
+    placeholder = str(el.get("placeholder") or "").strip()
+    name_attr = str(el.get("name_attr") or "").strip()
+    href = str(el.get("href") or "").strip()
+    css_id = str(el.get("id") or el.get("css_id") or "").strip()
+    css_path = str(el.get("css") or el.get("css_path") or "").strip()
+    stable_css = str(el.get("stable_css") or "").strip()
+    semantic_role = str(el.get("semantic_role") or "").strip()
+    semantic_name = str(el.get("semantic_name") or "").strip()
+    # Prefer existing primary_locator from page_dump if present
+    existing = el.get("primary_locator") or {}
+    if isinstance(existing, dict) and existing.get("expr"):
+        props = dict(existing.get("properties") or {})
+        if not props and existing.get("strategy"):
+            props = {"expr": existing.get("expr")}
+        c = LocatorCandidate(
+            strategy=str(existing.get("strategy") or "primary"),
+            scope=scope,
+            properties=props or {"expression": existing.get("expr")},
+            expression=str(existing.get("expr") or ""),
+            stability_score=0.85,
+            reasons=["from_page_dump_primary_locator"],
+        )
+        if rows and props:
+            c.mark_counts(count_matches(rows, properties=props, scope=scope))
+            c.validation = "snapshot_only"
+        elif rows and existing.get("expr"):
+            # Cannot count without structured props — leave unique unknown
+            c.validation = "snapshot_only"
+        cands.append(c)
+
+    for pw in el.get("playwright_candidates") or []:
+        if not isinstance(pw, dict):
+            continue
+        props = dict(pw.get("properties") or {})
+        expr = str(pw.get("expr") or pw.get("expression") or "")
+        strat = str(pw.get("strategy") or "playwright")
+        c = LocatorCandidate(
+            strategy=strat,
+            scope=scope,
+            properties=props or {"expression": expr},
+            expression=expr,
+            stability_score=float(pw.get("score") or 0.7),
+            uses_index=bool(pw.get("uses_index")),
+            uses_coordinates=bool(pw.get("uses_coordinates")),
+            reasons=["from_playwright_candidates"],
+        )
+        if rows and props:
+            c.mark_counts(count_matches(rows, properties=props, scope=scope))
+            c.validation = "snapshot_only"
+        cands.append(c)
+
+    def add(strategy: str, props: dict[str, Any], score: float, reason: str) -> None:
+        if not props or not any(str(v).strip() for v in props.values()):
+            return
+        c = LocatorCandidate(
+            strategy=strategy,
+            scope=dict(scope),
+            properties=dict(props),
+            expression=_expr(strategy, props),
+            stability_score=score,
+            reasons=[reason],
+        )
+        if rows:
+            c.mark_counts(count_matches(rows, properties=props, scope=scope))
+            c.validation = "snapshot_only"
+        cands.append(c)
+
+    if role and name:
+        add("role", {"role": role, "name": name}, 0.95, "role_name")
+    elif role and is_editable:
+        add("role", {"role": role}, 0.78, "editable_role")
+    if semantic_role and semantic_name:
+        add(
+            "semantic_role",
+            {"role": semantic_role, "name": semantic_name},
+            0.86,
+            "semantic_target_inferred_from_form_scope",
+        )
+    if test_id:
+        add("test_id", {"test_id": test_id}, 0.93, "test_id")
+    if label:
+        add("label", {"label": label}, 0.88, "label")
+    if placeholder:
+        add("placeholder", {"placeholder": placeholder}, 0.84, "placeholder")
+    if name_attr:
+        add("name_attr", {"name_attr": name_attr}, 0.89, "html_name_attribute")
+    if href and role.lower() == "link":
+        add("href", {"href": href}, 0.91, "link_destination")
+    if text and len(text) <= 80:
+        add("text", {"text": text}, 0.75, "visible_text")
+    if css_id and not css_id[0].isdigit():
+        add("css_id", {"id": css_id}, 0.8, "css_id")
+    if stable_css:
+        c = LocatorCandidate(
+            strategy="stable_css",
+            scope=dict(scope),
+            properties={"stable_css": stable_css},
+            expression=f"css:{stable_css}",
+            stability_score=0.78 if ("#" in stable_css or "data-testid" in stable_css) else 0.62,
+            reasons=["stable_attribute_selector", "live_unique_required"],
+        )
+        c.production_eligible = False
+        cands.append(c)
+    if css_path:
+        # Full/scoped CSS is a last-resort executable locator.  It only becomes
+        # production eligible after live validation proves it is unique.
+        c = LocatorCandidate(
+            strategy="css",
+            scope=dict(scope),
+            properties={"css": css_path},
+            expression=f"css:{css_path}",
+            stability_score=0.48 if "#" in css_path else 0.32,
+            reasons=["event_dom_path", "live_unique_required"],
+        )
+        c.production_eligible = False
+        cands.append(c)
+
+    # Never add coordinate locator as candidate for production
+    if el.get("bounds") or el.get("rect"):
+        c = LocatorCandidate(
+            strategy="bounding_rect",
+            scope=dict(scope),
+            properties={"bounds": el.get("bounds") or el.get("rect")},
+            expression="coordinates:(evidence_only)",
+            stability_score=0.05,
+            uses_coordinates=True,
+            production_eligible=False,
+            reasons=["coordinates_evidence_only"],
+        )
+        c.unique = False
+        c.ambiguous = True
+        cands.append(c)
+
+    # Re-validate production_eligible after counts
+    for c in cands:
+        if c.actual_count is not None:
+            c.mark_counts(int(c.actual_count), expected=int(c.expected_count or 1))
+        elif c.uses_coordinates:
+            c.production_eligible = False
+
+    primary, _ = rank_and_select_primary(cands)
+    # Ensure ambiguous never becomes primary via side effect — rank handles it.
+    del primary
+    return cands

tools/ui_inspector/locators/catalog.py

@@ -0,0 +1,294 @@
+"""Element Catalog: merge observations, locator history, parameterization."""
+
+from __future__ import annotations
+
+import re
+from pathlib import Path
+from typing import Any, Optional
+
+from models.element import ElementCatalog, ElementRef, element_fingerprint
+from models.locator import LocatorCandidate
+from models.serialize import to_dict, write_model_json
+from redaction import redact_element_fields
+from core.policies.element_identity import element_identity_key
+
+from .base import rank_and_select_primary
+from .browser import compile_browser_locators
+from .validator import validate_locator_against_rows
+from .windows import compile_windows_locators
+
+_DYNAMIC_VALUE_HINTS = re.compile(
+    r"(联系人|搜索|search|contact|user|chat|title|name)",
+    re.I,
+)
+
+
+def suggest_logical_name(element: dict[str, Any]) -> str:
+    aid = str(element.get("automation_id") or "").strip()
+    name = str(element.get("name") or "").strip()
+    role = str(element.get("control_type") or element.get("role") or "el").strip()
+    text = str(element.get("text") or "").strip()
+    semantic_name = str(element.get("semantic_name") or "").strip()
+    if element.get("contenteditable"):
+        base = aid or str(element.get("placeholder") or element.get("label") or "") or role
+    else:
+        base = aid or semantic_name or name or (text if 0 < len(text) <= 48 else "")
+        if not base:
+            for ancestor in element.get("parent_chain") or []:
+                ancestor_id = str((ancestor or {}).get("id") or "").strip()
+                ancestor_text = str((ancestor or {}).get("text") or "").strip()
+                if ancestor_id:
+                    base = f"{ancestor_id}_{role}"
+                    break
+                if ancestor_text and len(ancestor_text) <= 32:
+                    base = f"{ancestor_text}_{role}"
+                    break
+        base = base or role
+    base = re.sub(r"[^\w\u4e00-\u9fff]+", "_", base, flags=re.UNICODE).strip("_")
+    base = base[:48] or "element"
+    return base.lower() if base.isascii() else base
+
+
+def _maybe_parameterize(value: str, *, field: str) -> tuple[str, dict[str, str]]:
+    """Replace likely dynamic business values with ${param} placeholders."""
+    v = (value or "").strip()
+    if not v or len(v) > 64:
+        return v, {}
+    if field in {"value", "text", "name"} and _DYNAMIC_VALUE_HINTS.search(field):
+        return "${value}", {"value": v}
+    # Contact-like short Chinese/ASCII names in value when name is generic
+    if field == "value" and 1 < len(v) <= 32 and not v.startswith("${"):
+        # Only parameterize when looks like a person/search term (letters/CJK, not chrome)
+        if re.fullmatch(r"[\w\u4e00-\u9fff \-_.@]+", v, flags=re.UNICODE):
+            chrome = {"关闭", "最小化", "最大化", "取消", "确定", "发送", "搜索"}
+            if v not in chrome:
+                return "${contact_name}", {"contact_name": v}
+    return v, {}
+
+
+class ElementCatalogService:
+    def __init__(self, flow_dir: Path, *, flow_id: str = "") -> None:
+        self.flow_dir = Path(flow_dir)
+        self.elements_dir = self.flow_dir / "elements"
+        self.elements_dir.mkdir(parents=True, exist_ok=True)
+        self.catalog_path = self.flow_dir / "element_catalog.json"
+        self.catalog = ElementCatalog(flow_id=flow_id)
+        self._by_fp: dict[str, ElementRef] = {}
+        self._by_identity: dict[str, ElementRef] = {}
+        self._by_name: dict[str, ElementRef] = {}
+
+    def load(self) -> None:
+        if not self.catalog_path.exists():
+            return
+        import json
+        from models.serialize import from_dict
+
+        data = json.loads(self.catalog_path.read_text(encoding="utf-8"))
+        self.catalog.flow_id = data.get("flow_id") or self.catalog.flow_id
+        for item in data.get("elements") or []:
+            # Reconstruct LocatorCandidate lists lightly
+            locs = []
+            for lc in item.get("locator_candidates") or []:
+                locs.append(from_dict(LocatorCandidate, lc))
+            primary = None
+            if item.get("primary_locator"):
+                primary = from_dict(LocatorCandidate, item["primary_locator"])
+            fallbacks = [
+                from_dict(LocatorCandidate, x) for x in (item.get("fallback_locators") or [])
+            ]
+            el = ElementRef(
+                id=item.get("id") or "",
+                logical_name=item.get("logical_name") or "",
+                surface=item.get("surface") or "windows",
+                backend=item.get("backend") or "",
+                app=item.get("app") or {},
+                window_scope=item.get("window_scope") or {},
+                element_scope=item.get("element_scope") or {},
+                role=item.get("role") or "",
+                control_type=item.get("control_type") or "",
+                name=item.get("name") or "",
+                value=item.get("value") or "",
+                automation_id=item.get("automation_id") or "",
+                class_name=item.get("class_name") or "",
+                framework_id=item.get("framework_id") or "",
+                states=item.get("states") or {},
+                bounds=item.get("bounds") or {},
+                actions=list(item.get("actions") or []),
+                parent_chain=list(item.get("parent_chain") or []),
+                raw=item.get("raw") or {},
+                fingerprint=item.get("fingerprint") or "",
+                sensitive=bool(item.get("sensitive")),
+                locator_candidates=locs,
+                primary_locator=primary,
+                fallback_locators=fallbacks,
+                expected_count=int(item.get("expected_count") or 1),
+                verified_count=item.get("verified_count"),
+                verification_level=item.get("verification_level") or "none",
+                stability_score=float(item.get("stability_score") or 0.0),
+                verified_at=item.get("verified_at"),
+                first_seen=item.get("first_seen") or "",
+                last_seen=item.get("last_seen") or "",
+                observation_count=int(item.get("observation_count") or 1),
+                locator_history=list(item.get("locator_history") or []),
+                parameterized=dict(item.get("parameterized") or {}),
+            )
+            if el.fingerprint:
+                self._by_fp[el.fingerprint] = el
+            identity = str((el.raw or {}).get("_identity_key") or "")
+            if not identity:
+                identity = element_identity_key(
+                    el.raw or to_dict(el),
+                    surface=el.surface,
+                    page_url=str((el.element_scope or {}).get("page_url") or ""),
+                    frame_name=str((el.element_scope or {}).get("frame_name") or ""),
+                )
+            self._by_identity[identity] = el
+            if el.logical_name:
+                self._by_name[el.logical_name] = el
+            self.catalog.elements.append(el)
+
+    def save(self) -> None:
+        from datetime import datetime
+
+        self.catalog.updated_at = datetime.now().astimezone().isoformat()
+        write_model_json(self.catalog_path, self.catalog)
+        for el in self.catalog.elements:
+            write_model_json(self.elements_dir / f"{el.id}.json", el)
+
+    def upsert_from_snapshot(
+        self,
+        element: dict[str, Any],
+        *,
+        surface: str,
+        backend: str = "",
+        window_scope: Optional[dict[str, Any]] = None,
+        rows: Optional[list[dict[str, Any]]] = None,
+        logical_name: Optional[str] = None,
+        frame: Optional[dict[str, Any]] = None,
+        live_counts: Optional[dict[str, int]] = None,
+    ) -> ElementRef:
+        cleaned, sensitive = redact_element_fields(dict(element or {}))
+        win = dict(window_scope or {})
+        value = str(cleaned.get("value") or "")
+        is_editable = bool(cleaned.get("contenteditable"))
+        param_value, params = _maybe_parameterize(value, field="value")
+        if params:
+            cleaned["value"] = param_value
+
+        fp = element_fingerprint(
+            surface=surface,
+            role=str(cleaned.get("control_type") or cleaned.get("role") or ""),
+            name=str(cleaned.get("name") or ""),
+            automation_id=str(cleaned.get("automation_id") or ""),
+            class_name=str(cleaned.get("class_name") or ""),
+            value=str(cleaned.get("value") or ""),
+            window_scope=win,
+            dom_fingerprint=str(cleaned.get("fingerprint") or cleaned.get("css") or ""),
+            page_scope=str((frame or {}).get("url") or win.get("url") or ""),
+            contenteditable=is_editable,
+        )
+
+        identity = element_identity_key(
+            cleaned,
+            surface=surface,
+            page_url=str((frame or {}).get("url") or win.get("url") or ""),
+            frame_name=str((frame or {}).get("name") or ""),
+        )
+        cleaned["_identity_key"] = identity
+
+        existing = self._by_identity.get(identity) or self._by_fp.get(fp)
+        if existing is None and logical_name and logical_name in self._by_name:
+            existing = self._by_name[logical_name]
+
+        if surface == "browser":
+            cands = compile_browser_locators(cleaned, rows=rows, frame=frame)
+        else:
+            cands = compile_windows_locators(cleaned, rows=rows, window_scope=win)
+
+        for c in cands:
+            if rows is not None and not c.uses_coordinates:
+                validate_locator_against_rows(c, rows, level="snapshot_only")
+            if live_counts is not None and c.strategy in live_counts:
+                c.mark_counts(int(live_counts[c.strategy]))
+                c.validation = "live"
+
+        primary, fallbacks = rank_and_select_primary(cands)
+        # Ambiguous / coordinates never primary
+        if primary is not None and (primary.ambiguous or not primary.production_eligible):
+            primary = None
+
+        lname = logical_name or (existing.logical_name if existing else "") or suggest_logical_name(
+            cleaned
+        )
+
+        if existing:
+            # Merge observation; keep locator history
+            if primary is not None:
+                old = existing.primary_locator
+                if old is not None and to_dict(old) != to_dict(primary):
+                    existing.locator_history.append(
+                        {"replaced_at": existing.last_seen, "locator": to_dict(old)}
+                    )
+                existing.primary_locator = primary
+            existing.locator_candidates = cands
+            existing.fallback_locators = fallbacks
+            existing.touch()
+            existing.raw = cleaned
+            existing.bounds = dict(cleaned.get("bounds") or cleaned.get("rect") or existing.bounds or {})
+            if params:
+                existing.parameterized.update(params)
+            existing.sensitive = existing.sensitive or sensitive
+            existing.verification_level = "live" if live_counts else "snapshot_only"
+            self._by_fp[fp] = existing
+            self._by_identity[identity] = existing
+            if primary and primary.actual_count is not None:
+                existing.verified_count = primary.actual_count
+            self.save()
+            return existing
+
+        el = ElementRef(
+            logical_name=lname,
+            surface=surface,
+            backend=backend,
+            window_scope=win,
+            element_scope={
+                "page_url": str((frame or {}).get("url") or win.get("url") or ""),
+                "frame_name": str((frame or {}).get("name") or ""),
+            },
+            role=str(cleaned.get("role") or ""),
+            control_type=str(cleaned.get("control_type") or ""),
+            name=str(cleaned.get("name") or ""),
+            value=str(cleaned.get("value") or ""),
+            automation_id=str(cleaned.get("automation_id") or ""),
+            class_name=str(cleaned.get("class_name") or ""),
+            framework_id=str(cleaned.get("framework_id") or ""),
+            states={
+                "is_enabled": cleaned.get("is_enabled"),
+                "is_offscreen": cleaned.get("is_offscreen"),
+            },
+            bounds=dict(cleaned.get("bounds") or cleaned.get("rect") or {}),
+            parent_chain=list(cleaned.get("parent_chain") or []),
+            raw=cleaned,
+            fingerprint=fp,
+            sensitive=sensitive,
+            locator_candidates=cands,
+            primary_locator=primary,
+            fallback_locators=fallbacks,
+            verification_level="live" if live_counts else "snapshot_only",
+            verified_count=(primary.actual_count if primary else None),
+            parameterized=params,
+            stability_score=float(primary.stability_score) if primary else 0.0,
+        )
+        # Ensure unique logical_name
+        base = el.logical_name
+        i = 2
+        while el.logical_name in self._by_name:
+            el.logical_name = f"{base}_{i}"
+            i += 1
+
+        self.catalog.elements.append(el)
+        self._by_fp[fp] = el
+        self._by_identity[identity] = el
+        self._by_name[el.logical_name] = el
+        self.save()
+        return el

tools/ui_inspector/locators/validator.py

@@ -0,0 +1,50 @@
+"""Locator uniqueness validation against snapshot / live tree rows."""
+
+from __future__ import annotations
+
+from typing import Any, Optional
+
+from models.locator import LocatorCandidate
+
+from .base import count_matches
+
+
+def validate_locator_against_rows(
+    candidate: LocatorCandidate,
+    rows: list[dict[str, Any]],
+    *,
+    level: str = "snapshot_only",
+    expected_count: int = 1,
+) -> LocatorCandidate:
+    """
+    Validate structured locator properties against a row list.
+
+    Levels supported in this phase: snapshot_only | live_tree
+    """
+    actual = count_matches(
+        rows,
+        properties=candidate.properties,
+        scope=candidate.scope,
+    )
+    candidate.mark_counts(actual, expected=expected_count)
+    candidate.validation = level
+    if not candidate.unique:
+        candidate.reasons = list(candidate.reasons or [])
+        if "ambiguous_match_count" not in candidate.reasons:
+            candidate.reasons.append("ambiguous_match_count")
+        candidate.production_eligible = False
+    if candidate.uses_coordinates:
+        candidate.production_eligible = False
+        if "coordinates_never_primary" not in (candidate.reasons or []):
+            candidate.reasons = list(candidate.reasons or []) + ["coordinates_never_primary"]
+    return candidate
+
+
+def choose_production_primary(
+    candidates: list[LocatorCandidate],
+) -> Optional[LocatorCandidate]:
+    """Return first production-eligible unique non-coordinate locator, else None."""
+    for c in candidates:
+        if c.production_eligible and c.unique and not c.uses_coordinates:
+            return c
+    return None

tools/ui_inspector/locators/windows.py

@@ -0,0 +1,180 @@
+"""Windows UIA locator compiler (Name ≠ Value; no coordinate primary)."""
+
+from __future__ import annotations
+
+from typing import Any, Optional
+
+from models.locator import LocatorCandidate
+
+from .base import count_matches, rank_and_select_primary
+
+
+def _expr(strategy: str, props: dict[str, Any]) -> str:
+    parts = [f"{k}={v!r}" for k, v in props.items() if v not in (None, "")]
+    return f"{strategy}:" + ",".join(parts)
+
+
+def compile_windows_locators(
+    element: dict[str, Any],
+    *,
+    rows: Optional[list[dict[str, Any]]] = None,
+    window_scope: Optional[dict[str, Any]] = None,
+) -> list[LocatorCandidate]:
+    """
+    Priority:
+    1. Window scope + AutomationId + ControlType
+    2. Window scope + stable Name + ControlType
+    3. Window scope + Name + Value + ControlType
+    4. Business parent + child attrs
+    5. Visible Value + ControlType
+    6. Relative ancestor scope
+    7. Index with business reason (low score)
+    8. BoundingRectangle evidence only
+    """
+    el = element or {}
+    rows = list(rows or [])
+    win = dict(window_scope or el.get("window_scope") or {})
+    scope = {
+        "window_class": win.get("class_name") or win.get("window_class") or "",
+        "window_title": win.get("title") or win.get("window_title") or "",
+        "process_name": win.get("process_name") or "",
+    }
+    cands: list[LocatorCandidate] = []
+
+    aid = str(el.get("automation_id") or "").strip()
+    name = str(el.get("name") or "").strip()
+    value = str(el.get("value") or "").strip()
+    ctype = str(el.get("control_type") or el.get("role") or "").strip()
+    class_name = str(el.get("class_name") or "").strip()
+    parent_chain = list(el.get("parent_chain") or [])
+
+    def add(
+        strategy: str,
+        props: dict[str, Any],
+        score: float,
+        reason: str,
+        *,
+        uses_index: bool = False,
+        uses_coordinates: bool = False,
+    ) -> None:
+        if not props or not any(str(v).strip() for v in props.values() if not isinstance(v, (dict, list))):
+            return
+        c = LocatorCandidate(
+            strategy=strategy,
+            scope=dict(scope),
+            properties=dict(props),
+            expression=_expr(strategy, props),
+            stability_score=score,
+            uses_index=uses_index,
+            uses_coordinates=uses_coordinates,
+            reasons=[reason],
+        )
+        if rows:
+            c.mark_counts(count_matches(rows, properties=props, scope=scope))
+            c.validation = "snapshot_only"
+        elif uses_coordinates:
+            c.production_eligible = False
+            c.unique = False
+            c.ambiguous = True
+        cands.append(c)
+
+    if aid and ctype:
+        add(
+            "window_automation_id_type",
+            {"automation_id": aid, "control_type": ctype},
+            0.96,
+            "automation_id_control_type",
+        )
+    elif aid:
+        add("window_automation_id", {"automation_id": aid}, 0.9, "automation_id")
+
+    if name and ctype and name.lower() not in {"item", "pane", "unknown"}:
+        add(
+            "window_name_type",
+            {"name": name, "control_type": ctype},
+            0.88,
+            "stable_name_control_type",
+        )
+
+    if name and value and ctype:
+        add(
+            "window_name_value_type",
+            {"name": name, "value": value, "control_type": ctype},
+            0.86,
+            "name_value_control_type",
+        )
+
+    # Business parent + child
+    if parent_chain:
+        parent = parent_chain[0] if isinstance(parent_chain[0], dict) else {}
+        parent_name = str(parent.get("name") or parent.get("automation_id") or "").strip()
+        if parent_name and (aid or name or value):
+            props: dict[str, Any] = {"parent_name": parent_name}
+            if aid:
+                props["automation_id"] = aid
+            if name:
+                props["name"] = name
+            if value:
+                props["value"] = value
+            if ctype:
+                props["control_type"] = ctype
+            add("parent_scoped", props, 0.82, "business_parent_scope")
+
+    if value and ctype and len(value) <= 64:
+        add(
+            "value_type",
+            {"value": value, "control_type": ctype},
+            0.78,
+            "visible_value_control_type",
+        )
+
+    if class_name and ctype:
+        add(
+            "class_type",
+            {"class_name": class_name, "control_type": ctype},
+            0.55,
+            "class_name_control_type",
+        )
+
+    # Index only with explicit business reason — low score, not auto primary
+    idx = el.get("index")
+    if idx is not None and (aid or name):
+        props = {"index": int(idx)}
+        if aid:
+            props["automation_id"] = aid
+        if name:
+            props["name"] = name
+        if ctype:
+            props["control_type"] = ctype
+        add(
+            "indexed",
+            props,
+            0.35,
+            "index_requires_business_justification",
+            uses_index=True,
+        )
+
+    bounds = el.get("bounds") or el.get("rect") or el.get("bounding_rectangle")
+    if bounds:
+        c = LocatorCandidate(
+            strategy="bounding_rectangle",
+            scope=dict(scope),
+            properties={"bounds": bounds},
+            expression="coordinates:(evidence_only)",
+            stability_score=0.05,
+            uses_coordinates=True,
+            production_eligible=False,
+            unique=False,
+            ambiguous=True,
+            reasons=["coordinates_never_primary"],
+        )
+        cands.append(c)
+
+    for c in cands:
+        if c.actual_count is not None:
+            c.mark_counts(int(c.actual_count), expected=int(c.expected_count or 1))
+        if c.uses_coordinates:
+            c.production_eligible = False
+
+    rank_and_select_primary(cands)
+    return cands

tools/ui_inspector/models/__init__.py

@@ -0,0 +1,22 @@
+"""UI Inspector v2 data models (Action / State / Element / Flow)."""
+
+from .action import ACTION_TYPES, ActionFlow, ActionRecord
+from .element import ElementRef
+from .locator import LocatorCandidate
+from .observation import UIObservation
+from .serialize import from_dict, to_dict, write_model_json
+from .state import UIState, compute_state_fingerprint
+
+__all__ = [
+    "ACTION_TYPES",
+    "ActionFlow",
+    "ActionRecord",
+    "ElementRef",
+    "LocatorCandidate",
+    "UIState",
+    "UIObservation",
+    "compute_state_fingerprint",
+    "from_dict",
+    "to_dict",
+    "write_model_json",
+]

tools/ui_inspector/models/action.py

@@ -0,0 +1,119 @@
+"""ActionRecord + ActionFlow models."""
+
+from __future__ import annotations
+
+import uuid
+from dataclasses import dataclass, field
+from datetime import datetime
+from typing import Any, Optional
+
+SCHEMA_ACTION = "openclaw.ui_inspector.action.v1"
+SCHEMA_ACTION_FLOW = "openclaw.ui_inspector.action_flow.v1"
+
+# Full vocabulary — capability flags live elsewhere for unimplemented types.
+ACTION_TYPES: tuple[str, ...] = (
+    "click",
+    "double_click",
+    "right_click",
+    "fill",
+    "type",
+    "clear",
+    "press_key",
+    "hotkey",
+    "select",
+    "toggle",
+    "hover",
+    "scroll",
+    "drag_drop",
+    "focus",
+    "window_switch",
+    "navigation",
+    "checkpoint",
+    "wait",
+    "verify",
+    "unknown",
+)
+
+# Capability status for this Inspector release.
+ACTION_CAPABILITY: dict[str, str] = {
+    "click": "implemented",
+    "double_click": "schema_only",
+    "right_click": "schema_only",
+    "fill": "implemented",
+    "type": "implemented",  # aggregated into fill/type
+    "clear": "schema_only",
+    "press_key": "implemented",
+    "hotkey": "schema_only",
+    "select": "schema_only",
+    "toggle": "schema_only",
+    "hover": "schema_only",
+    "scroll": "schema_only",
+    "drag_drop": "schema_only",
+    "focus": "schema_only",
+    "window_switch": "schema_only",
+    "navigation": "partial",
+    "checkpoint": "implemented",
+    "wait": "schema_only",
+    "verify": "schema_only",
+    "unknown": "fallback",
+}
+
+
+def _now_iso() -> str:
+    return datetime.now().astimezone().isoformat()
+
+
+def _new_id(prefix: str = "act") -> str:
+    return f"{prefix}_{uuid.uuid4().hex[:12]}"
+
+
+@dataclass
+class ActionRecord:
+    schema: str = SCHEMA_ACTION
+    id: str = field(default_factory=lambda: _new_id("act"))
+    sequence: int = 0
+    action_type: str = "unknown"
+    started_at: str = field(default_factory=_now_iso)
+    ended_at: Optional[str] = None
+    surface: str = "windows"
+    target_element_id: Optional[str] = None
+    target_snapshot: dict[str, Any] = field(default_factory=dict)
+    input: dict[str, Any] = field(default_factory=dict)
+    sensitive: bool = False
+    before_state_id: Optional[str] = None
+    after_state_id: Optional[str] = None
+    before_observation_id: Optional[str] = None
+    after_observation_id: Optional[str] = None
+    observed_changes: list[dict[str, Any]] = field(default_factory=list)
+    proposed_postconditions: list[dict[str, Any]] = field(default_factory=list)
+    status: str = "ok"  # ok | partial | failed | skipped
+    confidence: float = 0.0
+    evidence: dict[str, Any] = field(default_factory=dict)
+    raw_event: dict[str, Any] = field(default_factory=dict)
+    raw_event_ids: list[str] = field(default_factory=list)
+    capability: str = ""
+
+    def __post_init__(self) -> None:
+        if self.action_type not in ACTION_TYPES:
+            self.action_type = "unknown"
+        if not self.capability:
+            self.capability = ACTION_CAPABILITY.get(self.action_type, "schema_only")
+
+
+@dataclass
+class ActionFlow:
+    schema: str = SCHEMA_ACTION_FLOW
+    flow_id: str = ""
+    flow_name: str = ""
+    created_at: str = field(default_factory=_now_iso)
+    ended_at: Optional[str] = None
+    states: list[str] = field(default_factory=list)  # state ids in order seen
+    observations: list[str] = field(default_factory=list)  # every concrete capture
+    actions: list[str] = field(default_factory=list)  # action ids in sequence
+    elements: list[str] = field(default_factory=list)  # element ids
+    entry_state_id: Optional[str] = None
+    final_state_id: Optional[str] = None
+    metadata: dict[str, Any] = field(default_factory=dict)
+    recording_mode: str = "auto"  # auto | guided | checkpoint | legacy_interval
+    surface: str = ""
+    status: str = "open"

tools/ui_inspector/models/element.py

@@ -0,0 +1,126 @@
+"""ElementRef model for Element Catalog."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import uuid
+from dataclasses import dataclass, field
+from datetime import datetime
+from typing import Any, Optional
+
+from .locator import LocatorCandidate
+
+SCHEMA_ELEMENT = "openclaw.ui_inspector.element.v1"
+SCHEMA_CATALOG = "openclaw.ui_inspector.element_catalog.v1"
+
+
+def _now_iso() -> str:
+    return datetime.now().astimezone().isoformat()
+
+
+def _new_id(prefix: str = "el") -> str:
+    return f"{prefix}_{uuid.uuid4().hex[:12]}"
+
+
+def element_fingerprint(
+    *,
+    surface: str,
+    role: str = "",
+    name: str = "",
+    automation_id: str = "",
+    class_name: str = "",
+    value: str = "",
+    window_scope: Optional[dict[str, Any]] = None,
+    dom_fingerprint: str = "",
+    page_scope: str = "",
+    contenteditable: bool = False,
+) -> str:
+    payload = {
+        "surface": surface or "",
+        "role": role or "",
+        "name": "" if contenteditable else (name or ""),
+        "automation_id": automation_id or "",
+        "class_name": class_name or "",
+        # Value is identity only when short/stable-looking (not long free text).
+        "value": "" if contenteditable else ((value or "") if len(value or "") <= 64 else ""),
+        "dom_fingerprint": dom_fingerprint or "",
+        "page_scope": page_scope or "",
+        "window_scope": {
+            "class_name": (window_scope or {}).get("class_name") or "",
+            "title": (window_scope or {}).get("title") or "",
+        },
+    }
+    raw = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
+    return hashlib.sha256(raw.encode("utf-8")).hexdigest()
+
+
+@dataclass
+class ElementRef:
+    schema: str = SCHEMA_ELEMENT
+    id: str = field(default_factory=lambda: _new_id("el"))
+    logical_name: str = ""
+    surface: str = "windows"
+    backend: str = ""
+    app: dict[str, Any] = field(default_factory=dict)
+    window_scope: dict[str, Any] = field(default_factory=dict)
+    element_scope: dict[str, Any] = field(default_factory=dict)
+    role: str = ""
+    control_type: str = ""
+    name: str = ""
+    value: str = ""
+    automation_id: str = ""
+    class_name: str = ""
+    framework_id: str = ""
+    states: dict[str, Any] = field(default_factory=dict)
+    bounds: dict[str, Any] = field(default_factory=dict)
+    actions: list[str] = field(default_factory=list)
+    parent_chain: list[dict[str, Any]] = field(default_factory=list)
+    raw: dict[str, Any] = field(default_factory=dict)
+    fingerprint: str = ""
+    sensitive: bool = False
+    locator_candidates: list[LocatorCandidate] = field(default_factory=list)
+    primary_locator: Optional[LocatorCandidate] = None
+    fallback_locators: list[LocatorCandidate] = field(default_factory=list)
+    expected_count: int = 1
+    verified_count: Optional[int] = None
+    verification_level: str = "none"
+    stability_score: float = 0.0
+    verified_at: Optional[str] = None
+    first_seen: str = field(default_factory=_now_iso)
+    last_seen: str = field(default_factory=_now_iso)
+    observation_count: int = 1
+    locator_history: list[dict[str, Any]] = field(default_factory=list)
+    parameterized: dict[str, str] = field(default_factory=dict)
+
+    def ensure_fingerprint(self) -> str:
+        if self.fingerprint:
+            return self.fingerprint
+        self.fingerprint = element_fingerprint(
+            surface=self.surface,
+            role=self.role or self.control_type,
+            name=self.name,
+            automation_id=self.automation_id,
+            class_name=self.class_name,
+            value=self.value,
+            window_scope=self.window_scope,
+            dom_fingerprint=str(self.raw.get("fingerprint") or self.raw.get("css") or ""),
+            page_scope=str(self.element_scope.get("page_url") or ""),
+            contenteditable=bool(self.raw.get("contenteditable")),
+        )
+        return self.fingerprint
+
+    def touch(self) -> None:
+        self.last_seen = _now_iso()
+        self.observation_count = int(self.observation_count or 0) + 1
+
+
+@dataclass
+class ElementCatalog:
+    schema: str = SCHEMA_CATALOG
+    flow_id: str = ""
+    updated_at: str = field(default_factory=_now_iso)
+    elements: list[ElementRef] = field(default_factory=list)
+
+    def by_logical_name(self) -> dict[str, ElementRef]:
+        return {e.logical_name: e for e in self.elements if e.logical_name}

tools/ui_inspector/models/flow.py

@@ -0,0 +1,11 @@
+"""Flow-level helpers (re-export ActionFlow for models.flow import path)."""
+
+from .action import ActionFlow, ActionRecord
+from .element import ElementCatalog, ElementRef
+from .state import UIState
+from .observation import UIObservation
+
+__all__ = [
+    "ActionFlow", "ActionRecord", "ElementCatalog", "ElementRef", "UIState",
+    "UIObservation",
+]

tools/ui_inspector/models/locator.py

@@ -0,0 +1,40 @@
+"""Structured locator candidate model."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any, Optional
+
+
+SCHEMA_LOCATOR = "openclaw.ui_inspector.locator_candidate.v1"
+
+
+@dataclass
+class LocatorCandidate:
+    schema: str = SCHEMA_LOCATOR
+    strategy: str = ""
+    scope: dict[str, Any] = field(default_factory=dict)
+    properties: dict[str, Any] = field(default_factory=dict)
+    expression: str = ""
+    expected_count: int = 1
+    actual_count: Optional[int] = None
+    unique: Optional[bool] = None
+    stability_score: float = 0.0
+    uses_index: bool = False
+    uses_coordinates: bool = False
+    validation: str = "none"  # snapshot_only | live_tree | cross_state | cross_session | none
+    reasons: list[str] = field(default_factory=list)
+    ambiguous: bool = False
+    production_eligible: bool = False
+
+    def mark_counts(self, actual: int, *, expected: int = 1) -> None:
+        self.expected_count = int(expected)
+        self.actual_count = int(actual)
+        self.unique = self.actual_count == self.expected_count
+        self.ambiguous = not bool(self.unique)
+        # Coordinates / index never become production primary automatically.
+        self.production_eligible = bool(
+            self.unique
+            and not self.uses_coordinates
+            and not (self.uses_index and self.stability_score < 0.7)
+        )

tools/ui_inspector/models/observation.py

@@ -0,0 +1,47 @@
+"""Per-capture evidence model.
+
+UIState represents a deduplicated semantic state.  UIObservation represents
+one concrete capture and is therefore never deduplicated.
+"""
+
+from __future__ import annotations
+
+import uuid
+from dataclasses import dataclass, field
+from datetime import datetime
+from typing import Any, Optional
+
+SCHEMA_OBSERVATION = "openclaw.ui_inspector.observation.v1"
+
+
+def _now_iso() -> str:
+    return datetime.now().astimezone().isoformat()
+
+
+def _new_id() -> str:
+    return f"obs_{uuid.uuid4().hex[:12]}"
+
+
+@dataclass
+class UIObservation:
+    schema: str = SCHEMA_OBSERVATION
+    id: str = field(default_factory=_new_id)
+    captured_at: str = field(default_factory=_now_iso)
+    state_id: str = ""
+    previous_observation_id: Optional[str] = None
+    change_reason: str = "capture"
+    surface: str = "windows"
+    driver: str = ""
+    page_id: str = ""
+    document_id: str = ""
+    frame_id: str = ""
+    url: str = ""
+    page_title: str = ""
+    tree_path: str = ""
+    index_path: str = ""
+    locatable_path: str = ""
+    screenshot_path: str = ""
+    stats: dict[str, Any] = field(default_factory=dict)
+    selection: dict[str, Any] = field(default_factory=dict)
+    may_contain_sensitive_data: bool = False
+

tools/ui_inspector/models/serialize.py

@@ -0,0 +1,64 @@
+"""Stable JSON serialization helpers for v2 models."""
+
+from __future__ import annotations
+
+import json
+from dataclasses import asdict, is_dataclass
+from datetime import datetime
+from pathlib import Path
+from typing import Any, TypeVar
+
+T = TypeVar("T")
+
+
+def _normalize(obj: Any) -> Any:
+    if obj is None or isinstance(obj, (str, int, float, bool)):
+        return obj
+    if isinstance(obj, datetime):
+        return obj.isoformat()
+    if isinstance(obj, Path):
+        return str(obj).replace("\\", "/")
+    if is_dataclass(obj) and not isinstance(obj, type):
+        return {k: _normalize(v) for k, v in asdict(obj).items()}
+    if isinstance(obj, dict):
+        return {str(k): _normalize(v) for k, v in obj.items()}
+    if isinstance(obj, (list, tuple)):
+        return [_normalize(x) for x in obj]
+    return str(obj)
+
+
+def to_dict(obj: Any) -> dict[str, Any]:
+    data = _normalize(obj)
+    if not isinstance(data, dict):
+        raise TypeError(f"expected dataclass/dict, got {type(obj)!r}")
+    return data
+
+
+def from_dict(cls: type[T], data: dict[str, Any]) -> T:
+    """Best-effort dataclass reconstruction (ignores unknown keys)."""
+    if not is_dataclass(cls):
+        raise TypeError(f"{cls!r} is not a dataclass")
+    import dataclasses
+
+    fields = {f.name for f in dataclasses.fields(cls)}
+    kwargs = {k: v for k, v in (data or {}).items() if k in fields}
+    return cls(**kwargs)  # type: ignore[call-arg]
+
+
+def dumps_json(obj: Any, *, indent: int = 2) -> str:
+    return json.dumps(
+        to_dict(obj) if is_dataclass(obj) and not isinstance(obj, type) else _normalize(obj),
+        ensure_ascii=False,
+        indent=indent,
+        sort_keys=False,
+    ) + "\n"
+
+
+def write_model_json(path: Path, obj: Any) -> None:
+    path = Path(path)
+    path.parent.mkdir(parents=True, exist_ok=True)
+    path.write_text(dumps_json(obj), encoding="utf-8")
+
+
+def read_json(path: Path) -> Any:
+    return json.loads(Path(path).read_text(encoding="utf-8"))

tools/ui_inspector/models/state.py

@@ -0,0 +1,175 @@
+"""UIState model + stable business fingerprint."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import uuid
+from dataclasses import dataclass, field
+from datetime import datetime, timezone
+from typing import Any, Optional
+
+SCHEMA_STATE = "openclaw.ui_inspector.state.v1"
+
+# Keys ignored when hashing business identity.
+_FINGERPRINT_EXCLUDE_TOP = {
+    "captured_at",
+    "id",
+    "previous_state_id",
+    "tree_path",
+    "index_path",
+    "locatable_path",
+    "screenshot_path",
+    "duplicate_of",
+    "observation_count",
+    "change_reason",
+}
+
+_SESSION_KEYS = {
+    "pid",
+    "hwnd",
+    "process_id",
+    "thread_id",
+    "elapsed_ms",
+    "dump_ms",
+    "warmup_ms",
+    "timings",
+    "screenshot",
+    "screen_path",
+}
+
+
+def _now_iso() -> str:
+    return datetime.now().astimezone().isoformat()
+
+
+def _new_id(prefix: str = "st") -> str:
+    return f"{prefix}_{uuid.uuid4().hex[:12]}"
+
+
+def _strip_session(obj: Any) -> Any:
+    if isinstance(obj, dict):
+        out: dict[str, Any] = {}
+        for k, v in obj.items():
+            lk = str(k).lower()
+            if lk in _SESSION_KEYS or lk.endswith("_ms") or lk.endswith("_path"):
+                continue
+            if lk in {"left", "top", "right", "bottom", "x", "y", "width", "height"}:
+                # Absolute coordinates are not primary identity.
+                continue
+            if lk == "rect" or lk == "bounds" or lk == "bounding_rectangle":
+                continue
+            out[str(k)] = _strip_session(v)
+        return out
+    if isinstance(obj, list):
+        return [_strip_session(x) for x in obj]
+    return obj
+
+
+def _locatable_signature(rows: list[dict[str, Any]], *, limit: int = 120) -> list[dict[str, Any]]:
+    """Compact visible-business signature for fingerprinting."""
+    sig: list[dict[str, Any]] = []
+    for r in rows[:limit]:
+        if not isinstance(r, dict):
+            continue
+        entry = {
+            "control_type": r.get("control_type") or r.get("role") or "",
+            "name": r.get("name") or "",
+            "value": r.get("value") or r.get("text") or "",
+            "automation_id": r.get("automation_id") or r.get("test_id") or "",
+            "class_name": r.get("class_name") or "",
+        }
+        # Drop empty noise rows
+        if not any(str(v).strip() for v in entry.values()):
+            continue
+        sig.append(entry)
+    # Stable order
+    sig.sort(
+        key=lambda e: (
+            str(e.get("control_type") or ""),
+            str(e.get("automation_id") or ""),
+            str(e.get("name") or ""),
+            str(e.get("value") or ""),
+        )
+    )
+    return sig
+
+
+def compute_state_fingerprint(
+    *,
+    surface: str,
+    driver: str = "",
+    url: str = "",
+    page_title: str = "",
+    window: Optional[dict[str, Any]] = None,
+    locatable: Optional[list[dict[str, Any]]] = None,
+    app: Optional[dict[str, Any]] = None,
+    extra_business: Optional[dict[str, Any]] = None,
+) -> str:
+    """
+    SHA-256 of stable business identity.
+
+    Excludes timestamps, pid/hwnd, screenshot paths, absolute coords, timings.
+    """
+    win = dict(window or {})
+    app_d = dict(app or {})
+    payload = {
+        "surface": surface or "",
+        "driver": driver or "",
+        "url": url or win.get("url") or app_d.get("url") or "",
+        "page_title": page_title or "",
+        "window": {
+            "title": win.get("title") or app_d.get("title") or "",
+            "class_name": win.get("class_name") or app_d.get("class_name") or "",
+            "process_name": win.get("process_name") or app_d.get("process_name") or "",
+        },
+        "locatable": _locatable_signature(list(locatable or [])),
+        "extra": _strip_session(extra_business or {}),
+    }
+    raw = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
+    return hashlib.sha256(raw.encode("utf-8")).hexdigest()
+
+
+@dataclass
+class UIState:
+    schema: str = SCHEMA_STATE
+    id: str = field(default_factory=lambda: _new_id("st"))
+    surface: str = "windows"
+    driver: str = ""
+    captured_at: str = field(default_factory=_now_iso)
+    fingerprint: str = ""
+    previous_state_id: Optional[str] = None
+    change_reason: str = ""
+    app: dict[str, Any] = field(default_factory=dict)
+    window: dict[str, Any] = field(default_factory=dict)
+    url: str = ""
+    page_title: str = ""
+    tree_path: str = ""
+    index_path: str = ""
+    locatable_path: str = ""
+    screenshot_path: str = ""
+    stats: dict[str, Any] = field(default_factory=dict)
+    selection: dict[str, Any] = field(default_factory=dict)
+    duplicate_of: Optional[str] = None
+    observation_count: int = 1
+    may_contain_sensitive_data: bool = False
+    # In-memory locatable used for fingerprint/diff (not always persisted as field).
+    locatable: list[dict[str, Any]] = field(default_factory=list, repr=False)
+
+    def ensure_fingerprint(self) -> str:
+        if self.fingerprint:
+            return self.fingerprint
+        self.fingerprint = compute_state_fingerprint(
+            surface=self.surface,
+            driver=self.driver,
+            url=self.url,
+            page_title=self.page_title,
+            window=self.window,
+            locatable=self.locatable,
+            app=self.app,
+        )
+        return self.fingerprint
+
+    def bump_observation(self) -> None:
+        self.observation_count = int(self.observation_count or 0) + 1
+        self.captured_at = _now_iso()

tools/ui_inspector/recorder/__init__.py

@@ -0,0 +1,13 @@
+"""Recorder package: observe user actions outside target processes."""
+
+from .base import RawUserEvent, RecorderBase, RecorderState
+from .input_aggregator import InputAggregator
+from .manager import ActionRecordingManager
+
+__all__ = [
+    "RawUserEvent",
+    "RecorderBase",
+    "RecorderState",
+    "InputAggregator",
+    "ActionRecordingManager",
+]

tools/ui_inspector/recorder/base.py

@@ -0,0 +1,79 @@
+"""Recorder abstractions."""
+
+from __future__ import annotations
+
+import threading
+from abc import ABC, abstractmethod
+from dataclasses import dataclass, field
+from datetime import datetime
+from typing import Any, Callable, Optional
+
+
+def _now_iso() -> str:
+    return datetime.now().astimezone().isoformat()
+
+
+@dataclass
+class RawUserEvent:
+    kind: str  # mouse_down | mouse_up | key | input | navigation | focus | checkpoint
+    surface: str
+    timestamp: str = field(default_factory=_now_iso)
+    button: str = ""
+    key: str = ""
+    text: str = ""
+    x: Optional[int] = None
+    y: Optional[int] = None
+    target: dict[str, Any] = field(default_factory=dict)
+    frame: dict[str, Any] = field(default_factory=dict)
+    url: str = ""
+    meta: dict[str, Any] = field(default_factory=dict)
+    from_inspector: bool = False
+
+
+class RecorderState:
+    IDLE = "idle"
+    RECORDING = "recording"
+    PAUSED = "paused"
+    STOPPED = "stopped"
+
+
+class RecorderBase(ABC):
+    """Process-external observer. No injection / hooks into target."""
+
+    def __init__(
+        self,
+        *,
+        on_event: Optional[Callable[[RawUserEvent], None]] = None,
+        exclude_hwnds: Optional[Callable[[], set[int]]] = None,
+    ) -> None:
+        self.on_event = on_event
+        self.exclude_hwnds = exclude_hwnds or (lambda: set())
+        self.status = RecorderState.IDLE
+        self._thread: Optional[threading.Thread] = None
+        self._stop = threading.Event()
+
+    def emit(self, event: RawUserEvent) -> None:
+        if self.status != RecorderState.RECORDING:
+            return
+        if event.from_inspector:
+            return
+        if self.on_event:
+            self.on_event(event)
+
+    @abstractmethod
+    def start(self) -> None: ...
+
+    @abstractmethod
+    def stop(self) -> None: ...
+
+    def pause(self) -> None:
+        if self.status == RecorderState.RECORDING:
+            self.status = RecorderState.PAUSED
+
+    def resume(self) -> None:
+        if self.status == RecorderState.PAUSED:
+            self.status = RecorderState.RECORDING
+
+    @property
+    def capability(self) -> dict[str, str]:
+        return {}

tools/ui_inspector/recorder/browser.py

@@ -0,0 +1,740 @@
+"""Browser recorder for Inspector-owned Playwright sessions.
+
+Injects capture-phase listeners that do NOT preventDefault / stopPropagation
+and do NOT mutate business DOM. Re-attaches after navigation.
+"""
+
+from __future__ import annotations
+
+import json
+import queue
+import threading
+import time
+from typing import Any, Callable, Optional
+
+from .base import RawUserEvent, RecorderBase, RecorderState
+
+_LISTEN_SCRIPT = r"""
+(() => {
+  if (window.__openclawUiInspectorRecorderInstalled) return;
+  window.__openclawUiInspectorRecorderInstalled = true;
+  window.__openclawUiInspectorEvents = window.__openclawUiInspectorEvents || [];
+  window.__openclawUiInspectorDocId = window.__openclawUiInspectorDocId ||
+    (Date.now().toString(36) + "-" + Math.random().toString(36).slice(2));
+  window.__openclawUiInspectorSeq = window.__openclawUiInspectorSeq || 0;
+
+  function cssPath(el) {
+    if (!el || !el.tagName) return "";
+    const esc = (value) => {
+      try { return CSS.escape(String(value)); } catch (e) { return String(value).replace(/[^a-zA-Z0-9_-]/g, "\\$&"); }
+    };
+    const parts = [];
+    let cur = el;
+    for (let i = 0; cur && i < 12; i++) {
+      let part = cur.tagName.toLowerCase();
+      if (cur.id) { part += "#" + esc(cur.id); parts.unshift(part); break; }
+      const cls = (cur.className && typeof cur.className === "string")
+        ? cur.className.trim().split(/\s+/).slice(0, 3).map(esc).join(".")
+        : "";
+      if (cls) part += "." + cls;
+      parts.unshift(part);
+      cur = cur.parentElement;
+    }
+    return parts.join(" > ");
+  }
+
+  function stableCss(el) {
+    if (!el || !el.getAttribute) return "";
+    const quote = (value) => JSON.stringify(String(value));
+    const ownId = el.id || "";
+    if (ownId) return "#" + CSS.escape(ownId);
+    const testId = el.getAttribute("data-testid") || "";
+    if (testId) return `[data-testid=${quote(testId)}]`;
+    const tag = (el.tagName || "").toLowerCase();
+    let leaf = "";
+    const href = el.getAttribute("href") || "";
+    if (tag === "a" && href) leaf = `a[href=${quote(href)}]`;
+    else if (el.getAttribute("contenteditable") != null) leaf = `${tag || "*"}[contenteditable]`;
+    else if (el.getAttribute("name")) leaf = `${tag || "*"}[name=${quote(el.getAttribute("name"))}]`;
+    else if (el.getAttribute("aria-label")) leaf = `${tag || "*"}[aria-label=${quote(el.getAttribute("aria-label"))}]`;
+    else if (el.getAttribute("placeholder")) leaf = `${tag || "*"}[placeholder=${quote(el.getAttribute("placeholder"))}]`;
+    const stableClasses = (node) => {
+      if (!node || typeof node.className !== "string") return [];
+      return node.className.trim().split(/\s+/).filter((token) => {
+        if (!token || token.length < 4 || token.length > 64) return false;
+        const lower = token.toLowerCase();
+        const semantic = /(button|btn|input|editor|control|action|submit|send|publish|post|play|pause|menu|option|tab|dialog|form|toolbar|container)/.test(lower);
+        const structured = /[-_]/.test(token) && !/^[a-z0-9_-]{6,14}$/i.test(token);
+        return semantic || structured;
+      }).slice(0, 2);
+    };
+    if (!leaf) {
+      const classes = stableClasses(el);
+      leaf = tag + classes.map(cls => "." + CSS.escape(cls)).join("");
+    }
+    let cur = el.parentElement;
+    const relative = [leaf];
+    for (let i = 0; cur && i < 10; i++, cur = cur.parentElement) {
+      if (cur.id) return `#${CSS.escape(cur.id)} > ${relative.reverse().join(" > ")}`;
+      const ancestorTestId = cur.getAttribute?.("data-testid") || "";
+      if (ancestorTestId) return `[data-testid=${quote(ancestorTestId)}] > ${relative.reverse().join(" > ")}`;
+      const curTag = (cur.tagName || "").toLowerCase();
+      const classes = stableClasses(cur);
+      relative.push(curTag + classes.map(cls => "." + CSS.escape(cls)).join(""));
+    }
+    return "";
+  }
+
+  function editableValue(el) {
+    if (!el) return "";
+    if (el.value != null) return String(el.value);
+    if (el.isContentEditable || el.getAttribute?.("contenteditable") != null) {
+      return String(el.innerText != null ? el.innerText : (el.textContent || ""));
+    }
+    return "";
+  }
+
+  function isEditable(el) {
+    if (!el || !el.getAttribute) return false;
+    const tag = (el.tagName || "").toLowerCase();
+    return tag === "input" || tag === "textarea" || tag === "select" ||
+      !!el.isContentEditable || el.getAttribute("contenteditable") != null;
+  }
+
+  function editableRoot(el) {
+    let cur = el;
+    for (let i = 0; cur && i < 12; i++) {
+      if (cur.getAttribute && cur.getAttribute("contenteditable") != null) return cur;
+      const tag = (cur.tagName || "").toLowerCase();
+      if (tag === "input" || tag === "textarea" || tag === "select") return cur;
+      cur = cur.parentElement;
+    }
+    return null;
+  }
+
+  function semanticRole(el) {
+    if (!el || !el.getAttribute) return "";
+    const explicit = el.getAttribute("role");
+    if (explicit) return explicit;
+    const tag = (el.tagName || "").toLowerCase();
+    if (el.isContentEditable) return "textbox";
+    if (tag === "a") return "link";
+    if (tag === "textarea") return "textbox";
+    if (tag === "select") return "combobox";
+    if (tag === "input") {
+      const type = (el.type || "text").toLowerCase();
+      if (type === "checkbox") return "checkbox";
+      if (type === "radio") return "radio";
+      if (["button", "submit", "reset"].includes(type)) return "button";
+      return "textbox";
+    }
+    return tag;
+  }
+
+  function actionableFromEvent(ev) {
+    const path = (ev && typeof ev.composedPath === "function") ? ev.composedPath() : [];
+    const nodes = path.length ? path : [ev && ev.target];
+    let best = ev && ev.target;
+    let bestScore = -1;
+    for (const el of nodes) {
+      if (!el || !el.matches) continue;
+      const tag = (el.tagName || "").toLowerCase();
+      const role = (el.getAttribute("role") || "").toLowerCase();
+      const cls = typeof el.className === "string" ? el.className.toLowerCase() : "";
+      let cursor = "";
+      try { cursor = getComputedStyle(el).cursor || ""; } catch (e) {}
+      let score = 0;
+      if (el.matches("button,a,input,textarea,select,summary,label")) score = 100;
+      if (el.isContentEditable || el.getAttribute("contenteditable") != null) score = 98;
+      if (["button","link","textbox","combobox","checkbox","radio","menuitem","option","tab"].includes(role)) score = Math.max(score, 96);
+      else if (role) score = Math.max(score, 52);
+      if (el.getAttribute("data-testid") || el.getAttribute("aria-label")) score = Math.max(score, 90);
+      if (el.getAttribute("tabindex") != null) score = Math.max(score, 74);
+      if (cursor === "pointer") score = Math.max(score, 68);
+      // Many production apps use camelCase classes such as sendControl or
+      // playButton.  Prefer that actionable wrapper over its inherited-cursor
+      // SVG/path child while still retaining hit_target for diagnostics.
+      if (tag.includes("-") || /(button|btn|submit|send|play|control|action|click)/i.test(cls)) score = Math.max(score, 72);
+      if (score > bestScore) { best = el; bestScore = score; }
+    }
+    return best;
+  }
+
+  function snap(el) {
+    if (!el || !el.getAttribute) return {};
+    const role = semanticRole(el);
+    let cursor = "";
+    let rect = {};
+    try {
+      cursor = getComputedStyle(el).cursor || "";
+      const r = el.getBoundingClientRect();
+      rect = {left:r.left, top:r.top, right:r.right, bottom:r.bottom, width:r.width, height:r.height};
+    } catch (e) {}
+    const tag = el.tagName ? el.tagName.toLowerCase() : "";
+    const css = cssPath(el);
+    const stable = stableCss(el);
+    let cssCount = null;
+    let stableCssCount = null;
+    try { cssCount = css ? document.querySelectorAll(css).length : null; } catch (e) {}
+    try { stableCssCount = stable ? document.querySelectorAll(stable).length : null; } catch (e) {}
+    const href = el.getAttribute("href") || "";
+    let hrefCount = null;
+    try {
+      hrefCount = href ? Array.from(document.querySelectorAll("a[href]")).filter(
+        node => node.getAttribute("href") === href
+      ).length : null;
+    } catch (e) {}
+    const roleNames = ["button","link","textbox","combobox","checkbox","radio","menuitem","option","tab"];
+    return {
+      role: role,
+      explicit_role: el.getAttribute("role") || "",
+      aria_label: el.getAttribute("aria-label") || "",
+      name: el.getAttribute("aria-label") || el.name || el.id ||
+        ((el.matches?.("button,a,[role],label,summary") || roleNames.includes(role) || cursor === "pointer" || tag.includes("-"))
+          ? (el.innerText || el.textContent || "").trim().slice(0, 120) : ""),
+      text: (el.innerText || el.textContent || "").trim().slice(0, 120),
+      value: editableValue(el).slice(0, 500),
+      test_id: el.getAttribute("data-testid") || "",
+      name_attr: el.getAttribute("name") || "",
+      href: href,
+      href_count: hrefCount,
+      title: el.getAttribute("title") || "",
+      alt: el.getAttribute("alt") || "",
+      id: el.id || "",
+      placeholder: el.getAttribute("placeholder") || el.getAttribute("aria-placeholder") || "",
+      label: "",
+      input_type: (el.type || "").toLowerCase(),
+      tag: tag,
+      class_name: (typeof el.className === "string" ? el.className : "").slice(0, 300),
+      cursor: cursor,
+      rect: rect,
+      disabled: !!el.disabled || el.getAttribute("aria-disabled") === "true",
+      contenteditable: !!el.isContentEditable || el.getAttribute("contenteditable") != null,
+      contenteditable_attr: el.getAttribute("contenteditable") != null,
+      tabindex: el.getAttribute("tabindex"),
+      css: css,
+      css_count: cssCount,
+      stable_css: stable,
+      stable_css_count: stableCssCount,
+      fingerprint: [
+        el.tagName?.toLowerCase() || "", el.id || "",
+        el.getAttribute("data-testid") || "", el.getAttribute("name") || "",
+        css
+      ].join("|"),
+    };
+  }
+
+  function ancestry(el) {
+    const out = [];
+    let cur = el;
+    for (let i = 0; cur && i < 12; i++) {
+      out.push(snap(cur));
+      cur = cur.parentElement;
+    }
+    return out;
+  }
+
+  function push(kind, payload) {
+    try {
+      const item = Object.assign({
+        event_id: window.__openclawUiInspectorDocId + ":" + (++window.__openclawUiInspectorSeq),
+        document_id: window.__openclawUiInspectorDocId,
+        sequence: window.__openclawUiInspectorSeq,
+        kind: kind,
+        ts: Date.now(),
+        url: location.href,
+        title: document.title,
+        frame_url: location.href,
+        frame_name: window.name || "",
+      }, payload || {});
+      window.__openclawUiInspectorEvents.push(item);
+      if (typeof window.__openclawUiInspectorEmit === "function") {
+        try { window.__openclawUiInspectorEmit(item); } catch (e) {}
+      }
+      if (window.__openclawUiInspectorEvents.length > 200) {
+        window.__openclawUiInspectorEvents.splice(0, 50);
+      }
+    } catch (e) {}
+  }
+
+  document.addEventListener("click", (ev) => {
+    const hit = ev.target;
+    const action = actionableFromEvent(ev) || hit;
+    push("click", {
+      target: snap(action),
+      hit_target: snap(hit),
+      action_target: snap(action),
+      ancestry: ancestry(hit),
+      candidate_chain: ancestry(hit),
+      button: ev.button,
+      x: ev.clientX,
+      y: ev.clientY
+    });
+  }, true);
+
+  document.addEventListener("pointerdown", (ev) => {
+    const hit = ev.target;
+    push("pointerdown", {
+      target: snap(hit),
+      hit_target: snap(hit),
+      candidate_chain: ancestry(hit),
+      button: ev.button,
+      pointer_id: ev.pointerId,
+      pointer_type: ev.pointerType || "",
+      x: ev.clientX,
+      y: ev.clientY
+    });
+  }, true);
+
+  document.addEventListener("input", (ev) => {
+    if (ev.isComposing) return;
+    const t = editableRoot(ev.target) || actionableFromEvent(ev) || ev.target;
+    push("input", {
+      target: snap(t), text: editableValue(t), input_event: ev.inputType || "input",
+      candidate_chain: ancestry(ev.target), is_trusted: !!ev.isTrusted
+    });
+  }, true);
+
+  document.addEventListener("change", (ev) => {
+    const t = editableRoot(ev.target) || actionableFromEvent(ev) || ev.target;
+    push("input", { target: snap(t), text: editableValue(t), input_event: "change",
+      candidate_chain: ancestry(ev.target), is_trusted: !!ev.isTrusted });
+  }, true);
+
+  document.addEventListener("compositionend", (ev) => {
+    const t = editableRoot(ev.target) || actionableFromEvent(ev) || ev.target;
+    push("input", {
+      target: snap(t), text: editableValue(t), input_event: "compositionend",
+      candidate_chain: ancestry(ev.target), is_trusted: !!ev.isTrusted
+    });
+  }, true);
+
+  document.addEventListener("paste", (ev) => {
+    const t = editableRoot(ev.target) || actionableFromEvent(ev) || ev.target;
+    setTimeout(() => push("input", {
+      target: snap(t), text: editableValue(t), input_event: "paste",
+      candidate_chain: ancestry(ev.target), is_trusted: !!ev.isTrusted
+    }), 0);
+  }, true);
+
+  document.addEventListener("keydown", (ev) => {
+    const special = ["Enter","Escape","Tab","Backspace","Delete"];
+    const editing = isEditable(ev.target);
+    const mutatingEditKey = editing && (ev.key === "Backspace" || ev.key === "Delete");
+    if (!mutatingEditKey && (special.indexOf(ev.key) >= 0 || (ev.key && ev.key.length > 1))) {
+      const t = actionableFromEvent(ev) || ev.target;
+      push("key", { key: ev.key, target: snap(t) });
+    }
+  }, true);
+})();
+"""
+
+
+class BrowserRecorder(RecorderBase):
+    capability_map = {
+        "click": "implemented",
+        "fill": "implemented",
+        "press_key": "implemented",
+        "navigation": "partial",
+    }
+
+    def __init__(
+        self,
+        *,
+        on_event: Optional[Callable[[RawUserEvent], None]] = None,
+        on_raw_event: Optional[Callable[[dict[str, Any]], None]] = None,
+        poll_sec: float = 0.25,
+    ) -> None:
+        super().__init__(on_event=on_event)
+        self.on_raw_event = on_raw_event
+        self.poll_sec = poll_sec
+        self._last_url = ""
+        self._last_urls_by_page: dict[str, str] = {}
+        self._installed_pages: set[int] = set()
+        self._bound_events: queue.Queue[dict[str, Any]] = queue.Queue(maxsize=512)
+        self._seen_event_ids: set[str] = set()
+        self._seen_event_order: list[str] = []
+        self._known_page_ids: set[str] = set()
+        self._pending_navigations: dict[str, dict[str, Any]] = {}
+        self._pending_pointerdowns: dict[str, dict[str, Any]] = {}
+        self._navigation_reorder_sec = 0.45
+        self._pointer_fallback_sec = 0.35
+
+    @property
+    def capability(self) -> dict[str, str]:
+        return dict(self.capability_map)
+
+    def start(self) -> None:
+        if self._thread and self._thread.is_alive():
+            self.status = RecorderState.RECORDING
+            return
+        self._stop.clear()
+        self.status = RecorderState.RECORDING
+        try:
+            import kit_browser
+
+            kit_browser.install_event_bridge(_LISTEN_SCRIPT, self._receive_bound_event)
+            initial_ids = kit_browser.call_on_each_open_page(
+                lambda page: kit_browser.page_id_for(page)
+            )
+            self._known_page_ids.update(
+                str(page_id) for page_id in initial_ids if page_id
+            )
+        except Exception:
+            # Polling remains as a compatibility fallback.
+            pass
+        self._thread = threading.Thread(
+            target=self._loop, name="ui-inspector-browser-recorder", daemon=True
+        )
+        self._thread.start()
+
+    def stop(self) -> None:
+        self._stop.set()
+        if self._thread and self._thread.is_alive():
+            self._thread.join(timeout=2.0)
+        self._thread = None
+        # Disable the bridge before draining its local queue so an event that
+        # arrived just before Stop is not silently lost, while no new events
+        # can race with the drain.
+        try:
+            import kit_browser
+
+            kit_browser.clear_event_bridge()
+        except Exception:
+            pass
+        if self.status == RecorderState.RECORDING:
+            try:
+                self._drain_bound_events()
+                self._flush_pending_pointerdowns(force=True)
+                self._flush_pending_navigations(force=True)
+            except Exception:
+                pass
+        self.status = RecorderState.STOPPED
+
+    def _receive_bound_event(self, event: dict[str, Any]) -> None:
+        try:
+            self._bound_events.put_nowait(dict(event or {}))
+        except queue.Full:
+            try:
+                self._bound_events.get_nowait()
+                self._bound_events.put_nowait(dict(event or {}))
+            except (queue.Empty, queue.Full):
+                pass
+
+    def _loop(self) -> None:
+        while not self._stop.is_set():
+            if self.status == RecorderState.PAUSED:
+                time.sleep(0.1)
+                continue
+            if self.status != RecorderState.RECORDING:
+                time.sleep(0.05)
+                continue
+            try:
+                self._tick()
+            except Exception:
+                pass
+            time.sleep(self.poll_sec)
+
+    def _tick(self) -> None:
+        import kit_browser
+
+        self._drain_bound_events()
+
+        # Install + drain on every open tab (detail may be a second tab).
+        payloads = kit_browser.call_on_each_open_page(self._install_and_drain)
+        if payloads:
+            for payload in payloads:
+                if not isinstance(payload, dict):
+                    continue
+                page_id = str(payload.get("page_id") or "")
+                url = str(payload.get("url") or "")
+                previous_url = self._last_urls_by_page.get(page_id, "") if page_id else ""
+                if previous_url and url and url != previous_url:
+                    self._schedule_navigation(
+                        page_id=page_id, from_url=previous_url, to_url=url
+                    )
+                elif (
+                    page_id
+                    and page_id not in self._known_page_ids
+                    and self._known_page_ids
+                    and url
+                    and not url.startswith("about:")
+                ):
+                    opener_id = str(payload.get("opener_page_id") or "")
+                    self._schedule_navigation(
+                        page_id=page_id,
+                        from_url=self._last_urls_by_page.get(opener_id, ""),
+                        to_url=url,
+                        opener_page_id=opener_id,
+                        new_page=True,
+                    )
+                if page_id and url:
+                    self._last_urls_by_page[page_id] = url
+                    self._known_page_ids.add(page_id)
+                for ev in payload.get("events") or []:
+                    if isinstance(ev, dict) and not ev.get("page_id"):
+                        ev["page_id"] = payload.get("page_id") or ""
+                    self._emit_dom_event(ev)
+
+        self._flush_pending_pointerdowns()
+
+        # Navigation is intentionally emitted after both bridge and polling
+        # queues have been drained.  This preserves click -> navigation order
+        # even when the context bridge is unavailable and polling is used.
+        self._flush_pending_navigations()
+
+    def _drain_bound_events(self) -> None:
+        while True:
+            try:
+                self._emit_dom_event(self._bound_events.get_nowait())
+            except queue.Empty:
+                break
+
+    def _schedule_navigation(
+        self,
+        *,
+        page_id: str,
+        from_url: str,
+        to_url: str,
+        opener_page_id: str = "",
+        new_page: bool = False,
+    ) -> None:
+        self._pending_navigations[page_id] = {
+            "due": time.monotonic() + self._navigation_reorder_sec,
+            "from": from_url,
+            "to": to_url,
+            "page_id": page_id,
+            "opener_page_id": opener_page_id,
+            "new_page": new_page,
+        }
+
+    def _flush_pending_navigations(self, *, force: bool = False) -> None:
+        now = time.monotonic()
+        ready = [
+            page_id
+            for page_id, item in self._pending_navigations.items()
+            if force or float(item.get("due") or 0.0) <= now
+        ]
+        for page_id in ready:
+            self._flush_pending_pointerdowns(force=True, page_id=page_id)
+            item = self._pending_navigations.pop(page_id, None) or {}
+            self.emit(
+                RawUserEvent(
+                    kind="navigation",
+                    surface="browser",
+                    url=str(item.get("to") or ""),
+                    meta={
+                        "from": item.get("from") or "",
+                        "to": item.get("to") or "",
+                        "page_id": page_id,
+                        "opener_page_id": item.get("opener_page_id") or "",
+                        "new_page": bool(item.get("new_page")),
+                    },
+                )
+            )
+
+    def _install_and_drain(self, page: Any) -> dict[str, Any]:
+        """Runs on Playwright worker thread (once per open tab)."""
+        try:
+            page.evaluate(_LISTEN_SCRIPT)
+        except Exception:
+            pass
+        # Also try frames
+        try:
+            for frame in page.frames:
+                try:
+                    frame.evaluate(_LISTEN_SCRIPT)
+                except Exception:
+                    pass
+        except Exception:
+            pass
+        events = []
+        try:
+            events = page.evaluate(
+                """() => {
+                  const e = window.__openclawUiInspectorEvents || [];
+                  window.__openclawUiInspectorEvents = [];
+                  return e;
+                }"""
+            ) or []
+        except Exception:
+            events = []
+        url = ""
+        try:
+            url = page.url or ""
+        except Exception:
+            url = ""
+        page_id = ""
+        opener_page_id = ""
+        try:
+            import kit_browser
+
+            page_id = kit_browser.page_id_for(page)
+            try:
+                opener = page.opener()
+                if opener is not None:
+                    opener_page_id = kit_browser.page_id_for(opener)
+            except Exception:
+                pass
+        except Exception:
+            pass
+        return {
+            "url": url,
+            "page_id": page_id,
+            "opener_page_id": opener_page_id,
+            "events": events,
+        }
+
+    def _emit_dom_event(self, ev: dict[str, Any]) -> None:
+        event_id = str(ev.get("event_id") or "")
+        if event_id:
+            if event_id in self._seen_event_ids:
+                return
+            self._seen_event_ids.add(event_id)
+            self._seen_event_order.append(event_id)
+            if len(self._seen_event_order) > 2000:
+                old = self._seen_event_order.pop(0)
+                self._seen_event_ids.discard(old)
+        if self.on_raw_event:
+            try:
+                self.on_raw_event(dict(ev))
+            except Exception:
+                pass
+        kind = str(ev.get("kind") or "")
+        hit_target = dict(ev.get("hit_target") or ev.get("target") or {})
+        candidate_chain = list(ev.get("candidate_chain") or ev.get("ancestry") or [])
+        if not candidate_chain:
+            candidate_chain = [hit_target]
+            action_hint = dict(ev.get("action_target") or {})
+            if action_hint and action_hint != hit_target:
+                candidate_chain.append(action_hint)
+        try:
+            from core.policies.target_scoring import resolve_browser_target
+
+            target, target_scores = resolve_browser_target(
+                hit_target,
+                candidate_chain,
+                x=int(ev.get("x")) if ev.get("x") is not None else None,
+                y=int(ev.get("y")) if ev.get("y") is not None else None,
+            )
+        except Exception:
+            target = dict(ev.get("action_target") or ev.get("target") or hit_target)
+            target_scores = []
+        if candidate_chain:
+            target["parent_chain"] = candidate_chain
+        frame = {
+            "url": ev.get("frame_url") or "",
+            "name": ev.get("frame_name") or "",
+            "id": ev.get("frame_id") or "",
+        }
+        if kind in {"pointerdown", "click"}:
+            raw = RawUserEvent(
+                    kind="click",
+                    surface="browser",
+                    target=target,
+                    frame=frame,
+                    url=str(ev.get("url") or ""),
+                    x=int(ev.get("x")) if ev.get("x") is not None else None,
+                    y=int(ev.get("y")) if ev.get("y") is not None else None,
+                    meta={
+                        "button": ev.get("button"),
+                        "event_id": event_id,
+                        "document_id": ev.get("document_id"),
+                        "page_id": ev.get("page_id"),
+                        "frame_id": ev.get("frame_id"),
+                        "sequence": ev.get("sequence"),
+                        "source_ts_ms": ev.get("ts"),
+                        "hit_target": ev.get("hit_target") or {},
+                        "action_target": ev.get("action_target") or target,
+                        "target_scores": target_scores,
+                        "captured_from": kind,
+                    },
+                )
+            if kind == "pointerdown":
+                key = self._pointer_key(ev)
+                self._pending_pointerdowns[key] = {
+                    "due": time.monotonic() + self._pointer_fallback_sec,
+                    "event": raw,
+                    "page_id": str(ev.get("page_id") or ""),
+                }
+                return
+            self._cancel_pointerdown_for_click(ev)
+            self.emit(raw)
+        elif kind == "input":
+            self.emit(
+                RawUserEvent(
+                    kind="input",
+                    surface="browser",
+                    text=str(ev.get("text") or target.get("value") or ""),
+                    target=target,
+                    frame=frame,
+                    url=str(ev.get("url") or ""),
+                    meta={
+                        "input_type": target.get("input_type") or "",
+                        "event_id": event_id,
+                        "document_id": ev.get("document_id"),
+                        "page_id": ev.get("page_id"),
+                        "frame_id": ev.get("frame_id"),
+                        "sequence": ev.get("sequence"),
+                        "source_ts_ms": ev.get("ts"),
+                        "input_event": ev.get("input_event") or "input",
+                        "is_trusted": bool(ev.get("is_trusted")),
+                    },
+                )
+            )
+        elif kind == "key":
+            self.emit(
+                RawUserEvent(
+                    kind="key",
+                    surface="browser",
+                    key=str(ev.get("key") or ""),
+                    target=target,
+                    frame=frame,
+                    url=str(ev.get("url") or ""),
+                    meta={
+                        "event_id": event_id,
+                        "document_id": ev.get("document_id"),
+                        "page_id": ev.get("page_id"),
+                        "frame_id": ev.get("frame_id"),
+                        "sequence": ev.get("sequence"),
+                        "source_ts_ms": ev.get("ts"),
+                    },
+                )
+            )
+
+    @staticmethod
+    def _pointer_key(ev: dict[str, Any]) -> str:
+        return "|".join(
+            str(ev.get(key) or "")
+            for key in ("page_id", "document_id", "pointer_id", "button")
+        )
+
+    def _cancel_pointerdown_for_click(self, ev: dict[str, Any]) -> None:
+        exact = self._pointer_key(ev)
+        if exact in self._pending_pointerdowns:
+            self._pending_pointerdowns.pop(exact, None)
+            return
+        page_id = str(ev.get("page_id") or "")
+        button = str(ev.get("button") or "")
+        for key, item in list(self._pending_pointerdowns.items()):
+            event = item.get("event")
+            meta = getattr(event, "meta", {}) or {}
+            if str(meta.get("page_id") or "") == page_id and str(meta.get("button") or "") == button:
+                self._pending_pointerdowns.pop(key, None)
+                return
+
+    def _flush_pending_pointerdowns(
+        self, *, force: bool = False, page_id: str = ""
+    ) -> None:
+        now = time.monotonic()
+        ready = [
+            key for key, item in self._pending_pointerdowns.items()
+            if (not page_id or item.get("page_id") == page_id)
+            and (force or float(item.get("due") or 0.0) <= now)
+        ]
+        for key in ready:
+            item = self._pending_pointerdowns.pop(key, None) or {}
+            event = item.get("event")
+            if event is not None:
+                event.meta["pointerdown_fallback"] = True
+                self.emit(event)

tools/ui_inspector/recorder/input_aggregator.py

@@ -0,0 +1,234 @@
+"""Aggregate keystrokes / value changes into fill/type + press_key."""
+
+from __future__ import annotations
+
+import time
+from dataclasses import dataclass, field
+from typing import Any, Optional
+
+from redaction import redact_input_payload
+
+# Keys recorded as independent press_key actions.
+_FUNCTION_KEYS = {
+    "Enter",
+    "Escape",
+    "Tab",
+    "Backspace",
+    "Delete",
+    "ArrowUp",
+    "ArrowDown",
+    "ArrowLeft",
+    "ArrowRight",
+    "Home",
+    "End",
+    "PageUp",
+    "PageDown",
+    "F1",
+    "F2",
+    "F3",
+    "F4",
+    "F5",
+    "F6",
+    "F7",
+    "F8",
+    "F9",
+    "F10",
+    "F11",
+    "F12",
+}
+
+
+@dataclass
+class AggregatedInput:
+    action_type: str  # fill | type | press_key
+    text: str = ""
+    key: str = ""
+    target: dict[str, Any] = field(default_factory=dict)
+    sensitive: bool = False
+    input_payload: dict[str, Any] = field(default_factory=dict)
+    started_at: float = 0.0
+    ended_at: float = 0.0
+    capture_context: dict[str, Any] = field(default_factory=dict)
+    raw_event: dict[str, Any] = field(default_factory=dict)
+    raw_event_ids: list[str] = field(default_factory=list)
+
+
+class InputAggregator:
+    """
+    Do not emit one Action per key.
+    Observe focus element value changes; flush after idle gap or function key.
+    """
+
+    def __init__(self, *, idle_sec: float = 0.8) -> None:
+        self.idle_sec = idle_sec
+        self._buf_text = ""
+        self._target: dict[str, Any] = {}
+        self._started: float = 0.0
+        self._last: float = 0.0
+        self._input_type = ""
+        self._field_name = ""
+        self._capture_context: dict[str, Any] = {}
+        self._raw_event: dict[str, Any] = {}
+        self._raw_event_ids: list[str] = []
+        self._target_identity = ""
+
+    @staticmethod
+    def _stable_target_identity(
+        target: dict[str, Any], capture_context: dict[str, Any]
+    ) -> str:
+        from core.policies.element_identity import element_identity_key
+
+        return element_identity_key(
+            target or {},
+            surface=str(capture_context.get("surface") or "windows"),
+            page_url=str(capture_context.get("url") or ""),
+            frame_name=str(capture_context.get("frame_name") or ""),
+        )
+
+    @staticmethod
+    def _event_id(raw_event: Optional[dict[str, Any]]) -> str:
+        return str(((raw_event or {}).get("meta") or {}).get("event_id") or "")
+
+    def on_key(
+        self,
+        *,
+        key: str,
+        text: str = "",
+        target: Optional[dict[str, Any]] = None,
+        input_type: str = "",
+        field_name: str = "",
+        capture_context: Optional[dict[str, Any]] = None,
+        raw_event: Optional[dict[str, Any]] = None,
+        value_event: bool = False,
+        now: Optional[float] = None,
+    ) -> list[AggregatedInput]:
+        now = time.monotonic() if now is None else now
+        out: list[AggregatedInput] = []
+        k = key or ""
+        if k in _FUNCTION_KEYS or (len(k) > 1 and k.startswith("F") and k[1:].isdigit()):
+            out.extend(self.flush(now=now))
+            payload, sens = redact_input_payload(
+                {"key": k},
+                field_name=field_name or self._field_name,
+                input_type=input_type or self._input_type,
+            )
+            out.append(
+                AggregatedInput(
+                    action_type="press_key",
+                    key=k,
+                    target=dict(target or self._target or {}),
+                    sensitive=sens,
+                    input_payload=payload,
+                    started_at=now,
+                    ended_at=now,
+                    capture_context=dict(capture_context or self._capture_context or {}),
+                    raw_event=dict(raw_event or {}),
+                    raw_event_ids=(
+                        [self._event_id(raw_event)] if self._event_id(raw_event) else []
+                    ),
+                )
+            )
+            return out
+
+        # Printable / value change
+        if target:
+            # Focus changed → flush previous
+            next_context = dict(capture_context or self._capture_context or {})
+            next_identity = self._stable_target_identity(target, next_context)
+            if self._target_identity and next_identity != self._target_identity:
+                out.extend(self.flush(now=now))
+            self._target = dict(target)
+            self._target_identity = next_identity
+        if input_type:
+            self._input_type = input_type
+        if field_name:
+            self._field_name = field_name
+        if capture_context:
+            self._capture_context = dict(capture_context)
+        if raw_event:
+            self._raw_event = dict(raw_event)
+            event_id = self._event_id(raw_event)
+            if event_id and event_id not in self._raw_event_ids:
+                self._raw_event_ids.append(event_id)
+
+        chunk = text if text else (k if len(k) == 1 else "")
+        if not chunk and not text and not value_event:
+            return out
+
+        if not self._started:
+            self._started = now
+        # If observer sends full value, replace buffer; else append char
+        if value_event:
+            self._buf_text = text
+        elif text and len(text) >= len(self._buf_text):
+            self._buf_text = text
+        else:
+            self._buf_text += chunk
+        self._last = now
+        return out
+
+    def on_value_change(
+        self,
+        *,
+        value: str,
+        target: Optional[dict[str, Any]] = None,
+        input_type: str = "",
+        field_name: str = "",
+        capture_context: Optional[dict[str, Any]] = None,
+        raw_event: Optional[dict[str, Any]] = None,
+        now: Optional[float] = None,
+    ) -> list[AggregatedInput]:
+        return self.on_key(
+            key="",
+            text=value,
+            target=target,
+            input_type=input_type,
+            field_name=field_name,
+            capture_context=capture_context,
+            raw_event=raw_event,
+            value_event=True,
+            now=now,
+        )
+
+    def flush(self, *, now: Optional[float] = None) -> list[AggregatedInput]:
+        now = (
+            (time.time() if self._last > 100_000_000 else time.monotonic())
+            if now is None else now
+        )
+        if not self._buf_text and not self._started:
+            return []
+        payload, sens = redact_input_payload(
+            {"text": self._buf_text, "name": self._field_name},
+            field_name=self._field_name,
+            input_type=self._input_type,
+        )
+        item = AggregatedInput(
+            action_type="fill",
+            text=str(payload.get("text") or ""),
+            target=dict(self._target or {}),
+            sensitive=sens,
+            input_payload=payload,
+            started_at=self._started or now,
+            ended_at=now,
+            capture_context=dict(self._capture_context or {}),
+            raw_event=dict(getattr(self, "_raw_event", {}) or {}),
+            raw_event_ids=list(self._raw_event_ids),
+        )
+        self._buf_text = ""
+        self._started = 0.0
+        self._last = 0.0
+        self._input_type = ""
+        self._capture_context = {}
+        self._raw_event = {}
+        self._raw_event_ids = []
+        # keep target for next session
+        return [item]
+
+    def maybe_flush_idle(self, *, now: Optional[float] = None) -> list[AggregatedInput]:
+        now = (
+            (time.time() if self._last > 100_000_000 else time.monotonic())
+            if now is None else now
+        )
+        if self._last and (now - self._last) >= self.idle_sec and self._started:
+            return self.flush(now=now)
+        return []

tools/ui_inspector/redaction.py

@@ -0,0 +1,206 @@
+"""Sensitive-data redaction for Inspector artifacts."""
+
+from __future__ import annotations
+
+import re
+from typing import Any, Optional
+
+# Field / attribute name hints (case-insensitive).
+_SENSITIVE_NAME_RE = re.compile(
+    r"(password|passwd|pwd|token|secret|cookie|authorization|auth|otp|"
+    r"captcha|verify.?code|verification.?code|sms.?code|pin)",
+    re.I,
+)
+
+_PHONE_RE = re.compile(r"(?<!\d)(?:\+?86[-\s]?)?1[3-9]\d{9}(?!\d)")
+_EMAIL_RE = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
+_TOKENISH_RE = re.compile(r"\b(?:Bearer\s+)?[A-Za-z0-9_\-]{24,}\b")
+
+REDACTED = "[REDACTED]"
+
+
+def is_sensitive_field_name(name: str) -> bool:
+    return bool(_SENSITIVE_NAME_RE.search(name or ""))
+
+
+def looks_sensitive_value(value: str) -> bool:
+    s = value or ""
+    if not s:
+        return False
+    if _PHONE_RE.search(s) or _EMAIL_RE.search(s) or _TOKENISH_RE.fullmatch(s.strip()):
+        return True
+    if "password" in s.lower() or "secret" in s.lower():
+        return True
+    return False
+
+
+def sanitize_artifact(value: Any, *, sensitive_hint: bool = False) -> Any:
+    """Recursively sanitize event/evidence data before it is persisted.
+
+    ``redact_input_payload`` protects the normalized Action input, but recorder
+    events also contain copies of the same value in target snapshots, ancestry
+    and raw_event.  All persisted copies must pass through one boundary.
+    """
+    if isinstance(value, dict):
+        input_type = str(value.get("input_type") or value.get("type") or "").lower()
+        field_name = str(
+            value.get("name")
+            or value.get("field")
+            or value.get("automation_id")
+            or ""
+        )
+        local_sensitive = bool(
+            sensitive_hint
+            or input_type == "password"
+            or is_sensitive_field_name(field_name)
+        )
+        out: dict[str, Any] = {}
+        for key, item in value.items():
+            key_sensitive = local_sensitive or is_sensitive_field_name(str(key))
+            if key in {
+                "name", "automation_id", "class_name", "control_type",
+                "framework_id", "path_hint", "logical_name",
+            } and isinstance(item, str) and not key_sensitive:
+                # Machine identifiers can be long token-like strings (for
+                # example conversation_list_container) without being secrets.
+                # Still remove explicit personal data embedded in a label.
+                out[key] = _EMAIL_RE.sub(REDACTED, _PHONE_RE.sub(REDACTED, item))
+                continue
+            if key in {"value", "text", "input", "typed", "content", "plaintext"}:
+                if item in (None, ""):
+                    out[key] = item
+                elif key_sensitive or looks_sensitive_value(str(item)):
+                    out[key] = REDACTED
+                else:
+                    out[key] = redact_text(str(item))
+            else:
+                out[key] = sanitize_artifact(item, sensitive_hint=key_sensitive)
+        if local_sensitive:
+            out["sensitive"] = True
+        return out
+    if isinstance(value, list):
+        return [sanitize_artifact(item, sensitive_hint=sensitive_hint) for item in value]
+    if isinstance(value, tuple):
+        return [sanitize_artifact(item, sensitive_hint=sensitive_hint) for item in value]
+    if isinstance(value, str):
+        if sensitive_hint or looks_sensitive_value(value):
+            return REDACTED
+        return redact_text(value)
+    return value
+
+
+def redact_text(value: str, *, force: bool = False) -> str:
+    if value is None:
+        return ""
+    s = str(value)
+    if force:
+        return REDACTED
+    s = _PHONE_RE.sub(REDACTED, s)
+    s = _EMAIL_RE.sub(REDACTED, s)
+    return s
+
+
+def redact_input_payload(
+    payload: dict[str, Any],
+    *,
+    field_name: str = "",
+    input_type: str = "",
+) -> tuple[dict[str, Any], bool]:
+    """
+    Return (redacted_copy, was_sensitive).
+    Never leave raw password/token in action_flow / catalog.
+    """
+    sensitive = False
+    out = dict(payload or {})
+    itype = (input_type or out.get("input_type") or "").lower()
+    fname = field_name or str(out.get("name") or out.get("field") or "")
+
+    if itype == "password" or is_sensitive_field_name(fname):
+        sensitive = True
+        for key in ("text", "value", "input", "typed", "content"):
+            if key in out and out[key] not in (None, ""):
+                out[key] = REDACTED
+        out["redacted"] = True
+        return out, True
+
+    for key in ("text", "value", "input", "typed", "content"):
+        if key not in out:
+            continue
+        raw = out.get(key)
+        if raw is None:
+            continue
+        s = str(raw)
+        if looks_sensitive_value(s) or is_sensitive_field_name(str(out.get("name") or "")):
+            sensitive = True
+            force = ("@" in s) or bool(_PHONE_RE.search(s)) or bool(_TOKENISH_RE.search(s))
+            out[key] = redact_text(s, force=force)
+            if _TOKENISH_RE.fullmatch(s.strip()) or looks_sensitive_value(s):
+                out[key] = REDACTED
+    if sensitive:
+        out["redacted"] = True
+    return out, sensitive
+
+
+def redact_element_fields(el: dict[str, Any]) -> tuple[dict[str, Any], bool]:
+    out = dict(el or {})
+    sensitive = False
+    name = str(out.get("name") or out.get("automation_id") or "")
+    role = str(out.get("control_type") or out.get("role") or "")
+    input_type = str(out.get("input_type") or out.get("type") or "")
+    if input_type == "password" or is_sensitive_field_name(name):
+        sensitive = True
+        if out.get("value"):
+            out["value"] = REDACTED
+        if out.get("text"):
+            out["text"] = REDACTED
+        out["sensitive"] = True
+        return out, True
+    for key in ("value", "text", "name"):
+        if key not in out or out[key] in (None, ""):
+            continue
+        s = str(out[key])
+        # Stable element names/automation ids are locator structure.  Long
+        # machine identifiers (OperationAndSendMessageRegion, generated DOM
+        # ids, etc.) must not be destroyed merely because they look token-like.
+        # Personal data embedded in a visible name is still scrubbed.
+        if key == "name":
+            cleaned = _EMAIL_RE.sub(REDACTED, _PHONE_RE.sub(REDACTED, s))
+            out[key] = cleaned
+            sensitive = sensitive or cleaned != s
+        elif looks_sensitive_value(s):
+            sensitive = True
+            out[key] = REDACTED
+    if sensitive:
+        out["sensitive"] = True
+    # Edit controls with password-ish class
+    if "password" in role.lower() or "password" in str(out.get("class_name") or "").lower():
+        sensitive = True
+        out["value"] = REDACTED
+        out["sensitive"] = True
+    return out, sensitive
+
+
+def redact_locatable_rows(rows: list[dict[str, Any]]) -> tuple[list[dict[str, Any]], bool]:
+    any_sens = False
+    out: list[dict[str, Any]] = []
+    for r in rows or []:
+        cleaned, sens = redact_element_fields(r if isinstance(r, dict) else {})
+        any_sens = any_sens or sens
+        out.append(cleaned)
+    return out, any_sens
+
+
+def screenshot_sensitivity_meta(
+    *,
+    may_contain_sensitive: bool,
+    save_sensitive_screenshots: bool = False,
+) -> dict[str, Any]:
+    return {
+        "may_contain_sensitive_data": bool(may_contain_sensitive),
+        "save_sensitive_screenshots": bool(save_sensitive_screenshots),
+        "note": (
+            "截图暂时无法可靠脱敏；敏感页请关闭截图或人工审查。"
+            if may_contain_sensitive
+            else ""
+        ),
+    }

tools/ui_inspector/requirements.txt

@@ -1,5 +1,4 @@
 uiautomation>=2.0.18
-# 可选：导出 screen.png 需要 Pillow；没有也能 dump JSON
-Pillow>=10.0.0
-# 企微自动暖树：与技能同款 ensure_ui_ready（请用宿主共享 venv 或本机安装）
-# jiangchang-platform-kit>=1.2.5
+# 浏览器整页 dump（与技能共用宿主环境；本地开发可 editable 安装）
+jiangchang-platform-kit>=1.2.11
+# playwright 由宿主 / kit 环境提供，勿在技能目录 playwright install

tools/ui_inspector/skip_policy.py

@@ -0,0 +1,89 @@
+"""Decide whether a dump is blank/empty-shell and should not become a flow step.
+
+Only applied to browser surface. Does NOT skip same-URL scrolls
+(content may change without URL change).
+"""
+
+from __future__ import annotations
+
+from typing import Any, Optional
+
+import target
+
+# Locatable below this counts as "no useful candidates"
+_MIN_LOCATABLE = 3
+# Tree/index tiny enough to treat as empty shell
+_MAX_EMPTY_NODES = 2
+
+
+def _norm_url(url: str) -> str:
+    return (url or "").strip()
+
+
+def is_blankish_url(url: str) -> bool:
+    u = _norm_url(url).lower()
+    if not u:
+        return True
+    if u == "about:blank" or u.startswith("about:blank?"):
+        return True
+    if u in {"about:srcdoc", "about:newtab"}:
+        return True
+    if u.startswith("chrome://newtab") or u.startswith("chrome://new-tab-page"):
+        return True
+    if u.startswith("edge://newtab") or u.startswith("edge://new-tab-page"):
+        return True
+    return False
+
+
+def should_skip_empty_dump(
+    *,
+    surface: str,
+    url: str = "",
+    locatable_count: Optional[int] = None,
+    nodes_kept: Optional[int] = None,
+    sparse: bool = False,
+    locatable: Optional[list[dict[str, Any]]] = None,
+    stats: Optional[dict[str, Any]] = None,
+) -> Optional[str]:
+    """Return skip reason code, or None if the dump should be kept.
+
+    Safe rules (browser only):
+    - blank/newtab URL **and** almost no locatable → skip
+    - tiny tree + sparse/blankish + almost no locatable → skip
+
+    A page with about:blank URL but rich locatable (nav race) is **kept**.
+    """
+    if surface != target.SURFACE_BROWSER:
+        return None
+
+    st = stats or {}
+    if locatable_count is None:
+        if locatable is not None:
+            locatable_count = len(locatable)
+        else:
+            locatable_count = int(st.get("locatable_count") or 0)
+    if nodes_kept is None:
+        nodes_kept = int(st.get("nodes_kept") or st.get("findall_count") or 0)
+    if not sparse:
+        sparse = bool(st.get("uia_sparse"))
+
+    loc_n = int(locatable_count or 0)
+    kept = int(nodes_kept or 0)
+    blankish = is_blankish_url(url)
+
+    if blankish and loc_n < _MIN_LOCATABLE:
+        return "blank_or_newtab_empty"
+
+    if loc_n < _MIN_LOCATABLE and kept <= _MAX_EMPTY_NODES and (sparse or blankish):
+        return "empty_shell"
+
+    return None
+
+
+def skip_reason_text(reason: str, *, url: str = "") -> str:
+    u = _norm_url(url) or "(empty)"
+    if reason == "blank_or_newtab_empty":
+        return f"跳过空页：url={u!r} 且可定位候选过少（blank/新标签页）。"
+    if reason == "empty_shell":
+        return f"跳过空壳页：url={u!r} 节点过少且 sparse。"
+    return f"跳过本拍：{reason} url={u!r}"

tools/ui_inspector/smoke_check.py

@@ -1,4 +1,4 @@
-"""Smoke test for flow document API + step labeling + kit_wake helpers (v0.8.0)."""
+"""Smoke test for flow document API + surface routing + kit helpers (v0.10)."""
 
 from __future__ import annotations
 
@@ -11,7 +11,30 @@ if str(_HERE) not in sys.path:
     sys.path.insert(0, str(_HERE))
 
 import artifacts as art
+import desktop_uia as uia
+import kit_browser
 import kit_wake
+import skip_policy
+import target
+import user_prefs
+from surfaces.registry import get_driver
+from drivers.mobile import MobileDriver
+from jiangchang_skill_core.rpa.page_dump import build_page_dump_result
+
+
+def _assert_selection_api() -> None:
+    assert callable(uia.window_info_under_cursor)
+    assert callable(uia.find_wecom_search_result_window)
+    assert callable(uia.resolve_capture_target)
+    try:
+        _info, _cands, meta = uia.resolve_capture_target(
+            under_cursor=False,
+            prefer_wecom_search_popup=False,
+        )
+        assert "strategy" in meta
+    except RuntimeError as exc:
+        assert "未找到可拾取" in str(exc)
+    print("OK selection api")
 
 
 def _assert_kit_wake_detect() -> None:
@@ -23,6 +46,204 @@ def _assert_kit_wake_detect() -> None:
     print("OK kit_wake detect")
 
 
+def _assert_surface_routing() -> None:
+    assert (
+        target.resolve_surface(process_name="chrome.exe", class_name="Chrome_WidgetWin_1")
+        == target.SURFACE_BROWSER
+    )
+    assert (
+        target.resolve_surface(process_name="msedge.exe", class_name="Chrome_WidgetWin_1")
+        == target.SURFACE_BROWSER
+    )
+    assert (
+        target.resolve_surface(process_name="WXWork.exe", class_name="WeWorkWindow")
+        == target.SURFACE_WINDOWS
+    )
+    assert get_driver(target.SURFACE_WINDOWS).driver_name == "uiautomation"
+    assert get_driver(target.SURFACE_BROWSER).driver_name == "playwright"
+    mobile = MobileDriver(surface=target.SURFACE_ANDROID)
+    try:
+        mobile.dump(info=None, selection_meta={})
+        raise AssertionError("mobile dump should raise")
+    except RuntimeError as exc:
+        assert "SURFACE_UNSUPPORTED" in str(exc)
+    print("OK surface routing")
+
+
+def _assert_user_prefs() -> None:
+    with tempfile.TemporaryDirectory(prefix="ui_prefs_") as tmp:
+        path = Path(tmp) / "user_prefs.json"
+        # monkeypatch prefs path
+        old = user_prefs._PREFS_PATH
+        try:
+            user_prefs._PREFS_PATH = path
+            user_prefs.save_prefs(mode="windows", start_url="https://example.com")
+            loaded = user_prefs.load_prefs()
+            assert loaded["mode"] == "windows"
+            assert loaded["start_url"] == "https://example.com"
+            user_prefs.save_prefs(mode="browser", start_url="")
+            loaded_empty = user_prefs.load_prefs()
+            assert loaded_empty["start_url"] == ""
+            user_prefs.save_prefs(mode="mobile", start_url="https://a.test")
+            loaded2 = user_prefs.load_prefs()
+            assert loaded2["mode"] == target.SURFACE_ANDROID
+        finally:
+            user_prefs._PREFS_PATH = old
+    print("OK user_prefs")
+
+
+def _assert_skip_policy() -> None:
+    # blank + empty locatable → skip
+    assert (
+        skip_policy.should_skip_empty_dump(
+            surface="browser",
+            url="about:blank",
+            locatable_count=0,
+            nodes_kept=1,
+            sparse=True,
+        )
+        == "blank_or_newtab_empty"
+    )
+    # blank URL but rich content (nav race like step05) → keep
+    assert (
+        skip_policy.should_skip_empty_dump(
+            surface="browser",
+            url="about:blank",
+            locatable_count=74,
+            nodes_kept=97,
+            sparse=False,
+        )
+        is None
+    )
+    # real page → keep
+    assert (
+        skip_policy.should_skip_empty_dump(
+            surface="browser",
+            url="https://www.douyin.com/jingxuan",
+            locatable_count=150,
+            nodes_kept=160,
+            sparse=False,
+        )
+        is None
+    )
+    # windows never skipped by this policy
+    assert (
+        skip_policy.should_skip_empty_dump(
+            surface="windows",
+            url="",
+            locatable_count=0,
+            nodes_kept=1,
+            sparse=True,
+        )
+        is None
+    )
+    # same URL with content is NOT skipped (scroll-safe)
+    assert (
+        skip_policy.should_skip_empty_dump(
+            surface="browser",
+            url="https://www.douyin.com/jingxuan",
+            locatable_count=20,
+            nodes_kept=30,
+            sparse=False,
+        )
+        is None
+    )
+    print("OK skip_policy")
+
+
+def _assert_active_page_resolve() -> None:
+    """Unit-level: visible tab wins over sticky pages[0]."""
+
+    class _FakePage:
+        def __init__(self, url: str, visible: bool, *, closed: bool = False) -> None:
+            self.url = url
+            self._visible = visible
+            self._closed = closed
+
+        def is_closed(self) -> bool:
+            return self._closed
+
+        def evaluate(self, _script: str) -> str:
+            return "visible" if self._visible else "hidden"
+
+        def title(self) -> str:
+            return self.url
+
+    class _FakeCtx:
+        def __init__(self, pages: list) -> None:
+            self.pages = pages
+
+    search = _FakePage("https://www.douyin.com/search/foo", visible=False)
+    detail = _FakePage("https://www.douyin.com/video/123", visible=True)
+    ctx = _FakeCtx([search, detail])
+
+    page, meta = kit_browser.resolve_active_page(ctx, preferred=search)
+    assert page is detail, (page.url, meta)
+    assert meta["resolve"] == "visibility"
+    assert meta["page_count"] == 2
+    assert meta["visible_count"] == 1
+
+    # No visible → preferred if still open
+    both_hidden = _FakeCtx(
+        [
+            _FakePage("https://a.test/1", visible=False),
+            _FakePage("https://a.test/2", visible=False),
+        ]
+    )
+    preferred = both_hidden.pages[0]
+    page2, meta2 = kit_browser.resolve_active_page(both_hidden, preferred=preferred)
+    assert page2 is preferred
+    assert meta2["resolve"] == "preferred_fallback"
+
+    # Closed preferred → first open
+    closed_pref = _FakePage("https://gone", visible=False, closed=True)
+    open_only = _FakePage("https://alive", visible=False)
+    page3, meta3 = kit_browser.resolve_active_page(
+        _FakeCtx([closed_pref, open_only]), preferred=closed_pref
+    )
+    assert page3 is open_only
+    assert meta3["resolve"] == "first_open"
+    print("OK active page resolve")
+
+
+def _assert_page_dump_kit() -> None:
+    if not kit_browser.kit_page_dump_available():
+        raise RuntimeError(
+            "jiangchang_skill_core.rpa.page_dump 不可用；"
+            "请 pip install -e jiangchang-platform-kit>=1.2.11"
+        )
+    result = build_page_dump_result(
+        url="https://example.com",
+        title="Example",
+        a11y_snapshot={
+            "role": "RootWebArea",
+            "name": "Example",
+            "children": [{"role": "button", "name": "OK", "children": []}],
+        },
+        dom_nodes=[
+            {
+                "tag": "button",
+                "role": "button",
+                "id": "",
+                "name_attr": "",
+                "testid": "",
+                "aria_label": "OK",
+                "placeholder": "",
+                "title": "",
+                "alt": "",
+                "href": "",
+                "type": "submit",
+                "text": "OK",
+                "disabled": False,
+                "rect": {},
+            }
+        ],
+    )
+    assert result["stats"]["locatable_count"] >= 1
+    assert result["locatable"][0].get("primary_locator")
+    print("OK page_dump kit builder")
+
+
 def _assert_label() -> None:
     msg_page = [
         {"control_type": "TreeItemControl", "name": "groupbuddyname", "value": "某群"},
@@ -83,7 +304,13 @@ def _assert_label() -> None:
 
 
 def main() -> int:
+    _assert_selection_api()
     _assert_kit_wake_detect()
+    _assert_surface_routing()
+    _assert_user_prefs()
+    _assert_skip_policy()
+    _assert_active_page_resolve()
+    _assert_page_dump_kit()
     _assert_label()
     with tempfile.TemporaryDirectory(prefix="ui_flow_smoke_") as tmp:
         root = Path(tmp)
@@ -111,7 +338,15 @@ def main() -> int:
         )
         assert idx == 1
         assert "陈德炼" in name, name
-        art.write_json(step_dir / "meta.json", {"ok": True})
+        art.write_json(
+            step_dir / "meta.json",
+            art.build_meta(
+                window={"title": "企业微信", "class_name": "WeWorkWindow", "pid": 1, "hwnd": 2},
+                scene="demo",
+                surface="windows",
+                driver="uiautomation",
+            ),
+        )
         art.register_step(
             flow_dir,
             flow,
@@ -120,9 +355,25 @@ def main() -> int:
             window={"title": "企业微信", "class_name": "WeWorkWindow", "pid": 1, "hwnd": 2},
             stats={"findall_count": 10, "locatable_count": 5, "nodes_kept": 3, "uia_sparse": False},
             label=label,
+            surface="windows",
+        )
+        # browser-shaped step registration
+        blow = art.load_flow(flow_dir)
+        art.register_step(
+            flow_dir,
+            blow,
+            step_index=2,
+            step_name="02_browser-demo",
+            window={"title": "精选", "class_name": "PlaywrightPage", "pid": 3, "hwnd": 4},
+            stats={"findall_count": 20, "locatable_count": 12, "nodes_kept": 20, "uia_sparse": False},
+            label="浏览器精选",
+            surface="browser",
+            url="https://www.douyin.com/jingxuan",
         )
         flow = art.end_flow(flow_dir, art.load_flow(flow_dir))
         assert flow["status"] == "ended"
+        summary = (flow_dir / "SUMMARY.md").read_text(encoding="utf-8")
+        assert "surface: `browser`" in summary or "surface: `windows`" in summary
         print(f"OK flow={flow_dir.name} step={name}")
     print("OK ui_inspector flow smoke passed")
     return 0

tools/ui_inspector/snapshot_service.py

@@ -0,0 +1,400 @@
+"""Snapshot service: wrap SurfaceDriver.dump into deduped UIState assets."""
+
+from __future__ import annotations
+
+import time
+from dataclasses import asdict, is_dataclass
+from pathlib import Path
+from typing import Any, Optional
+
+from models.observation import UIObservation
+from models.serialize import to_dict, write_model_json
+from models.state import UIState, compute_state_fingerprint
+from redaction import (
+    redact_locatable_rows,
+    sanitize_artifact,
+    screenshot_sensitivity_meta,
+)
+
+
+def _dump_to_parts(dump: Any) -> dict[str, Any]:
+    if dump is None:
+        return {}
+    if isinstance(dump, dict):
+        return dict(dump)
+    # DumpResult dataclass or duck-typed dump objects (tests / drivers).
+    if is_dataclass(dump) and not isinstance(dump, type) or hasattr(dump, "surface"):
+        return {
+            "surface": getattr(dump, "surface", "") or "",
+            "driver": getattr(dump, "driver", "") or "",
+            "window": dict(getattr(dump, "window", None) or {}),
+            "tree": getattr(dump, "tree", None) or {},
+            "index": list(getattr(dump, "index", None) or []),
+            "locatable": list(getattr(dump, "locatable", None) or []),
+            "stats": dict(getattr(dump, "stats", None) or {}),
+            "selection": dict(getattr(dump, "selection", None) or {}),
+            "url": getattr(dump, "url", "") or "",
+            "page_title": getattr(dump, "page_title", "") or "",
+            "screenshot_bytes": getattr(dump, "screenshot_bytes", None),
+            "use_hwnd_screenshot": bool(getattr(dump, "use_hwnd_screenshot", False)),
+            "locatable_text": getattr(dump, "locatable_text", "") or "",
+            "tree_text": getattr(dump, "tree_text", "") or "",
+        }
+    return {}
+
+
+class SnapshotService:
+    """
+    Creates UIState records with fingerprint dedupe.
+
+    Reuses SurfaceDriver.dump(); does not reimplement browser/desktop dump.
+    """
+
+    def __init__(
+        self,
+        flow_dir: Path,
+        *,
+        save_sensitive_screenshots: bool = False,
+        persist_evidence: bool = True,
+    ) -> None:
+        self.flow_dir = Path(flow_dir)
+        self.states_dir = self.flow_dir / "states"
+        self.states_dir.mkdir(parents=True, exist_ok=True)
+        self.observations_dir = self.flow_dir / "observations"
+        self.observations_dir.mkdir(parents=True, exist_ok=True)
+        self.save_sensitive_screenshots = save_sensitive_screenshots
+        self.persist_evidence = persist_evidence
+        self._by_fp: dict[str, UIState] = {}
+        self._order: list[str] = []
+        self.latest: Optional[UIState] = None
+        self.latest_observation: Optional[UIObservation] = None
+        self._observation_order: list[str] = []
+
+    def load_existing(self) -> None:
+        """Index existing state JSON files (best-effort)."""
+        if not self.states_dir.exists():
+            return
+        for path in sorted(self.states_dir.glob("*.json")):
+            try:
+                import json
+
+                data = json.loads(path.read_text(encoding="utf-8"))
+                st = UIState(
+                    id=data.get("id") or path.stem,
+                    surface=data.get("surface") or "windows",
+                    driver=data.get("driver") or "",
+                    captured_at=data.get("captured_at") or "",
+                    fingerprint=data.get("fingerprint") or "",
+                    previous_state_id=data.get("previous_state_id"),
+                    change_reason=data.get("change_reason") or "",
+                    app=data.get("app") or {},
+                    window=data.get("window") or {},
+                    url=data.get("url") or "",
+                    page_title=data.get("page_title") or "",
+                    tree_path=data.get("tree_path") or "",
+                    index_path=data.get("index_path") or "",
+                    locatable_path=data.get("locatable_path") or "",
+                    screenshot_path=data.get("screenshot_path") or "",
+                    stats=data.get("stats") or {},
+                    selection=data.get("selection") or {},
+                    duplicate_of=data.get("duplicate_of"),
+                    observation_count=int(data.get("observation_count") or 1),
+                    may_contain_sensitive_data=bool(
+                        data.get("may_contain_sensitive_data")
+                    ),
+                )
+                if st.fingerprint:
+                    self._by_fp[st.fingerprint] = st
+                    if st.id not in self._order:
+                        self._order.append(st.id)
+                    self.latest = st
+            except Exception:
+                continue
+        for path in sorted(self.observations_dir.glob("*/meta.json")):
+            try:
+                import json
+
+                data = json.loads(path.read_text(encoding="utf-8"))
+                obs = UIObservation(
+                    **{k: v for k, v in data.items() if k in UIObservation.__dataclass_fields__}
+                )
+                self._observation_order.append(obs.id)
+                self.latest_observation = obs
+            except Exception:
+                continue
+
+    def capture_from_dump(
+        self,
+        dump: Any,
+        *,
+        change_reason: str = "",
+        force_persist: bool = False,
+        previous_state_id: Optional[str] = None,
+        redact: bool = True,
+    ) -> UIState:
+        parts = _dump_to_parts(dump)
+        locatable = list(parts.get("locatable") or [])
+        sensitive = False
+        if redact:
+            locatable, sensitive = redact_locatable_rows(locatable)
+
+        window = dict(parts.get("window") or {})
+        url = parts.get("url") or ""
+        page_title = parts.get("page_title") or ""
+        surface = parts.get("surface") or "windows"
+        driver = parts.get("driver") or ""
+        fp = compute_state_fingerprint(
+            surface=surface,
+            driver=driver,
+            url=url,
+            page_title=page_title,
+            window=window,
+            locatable=locatable,
+            app={
+                "title": window.get("title"),
+                "class_name": window.get("class_name"),
+                "process_name": window.get("process_name"),
+            },
+        )
+
+        existing = self._by_fp.get(fp)
+        if existing is not None:
+            existing.bump_observation()
+            existing.duplicate_of = None
+            # Update meta path only — do not rewrite full tree evidence.
+            write_model_json(self.states_dir / f"{existing.id}.json", existing)
+            self.latest = existing
+            self._persist_observation(
+                existing,
+                parts,
+                locatable=locatable,
+                sensitive=sensitive,
+                change_reason=change_reason or "capture",
+            )
+            return existing
+
+        prev = previous_state_id
+        if prev is None and self.latest is not None:
+            prev = self.latest.id
+
+        state = UIState(
+            surface=surface,
+            driver=driver,
+            fingerprint=fp,
+            previous_state_id=prev,
+            change_reason=change_reason or "capture",
+            app={
+                "title": window.get("title"),
+                "class_name": window.get("class_name"),
+                "process_name": window.get("process_name"),
+                "pid": window.get("pid"),
+                "hwnd": window.get("hwnd"),
+            },
+            window=window,
+            url=url,
+            page_title=page_title,
+            stats=dict(parts.get("stats") or {}),
+            selection=dict(parts.get("selection") or {}),
+            locatable=locatable,
+            may_contain_sensitive_data=sensitive,
+        )
+        state.ensure_fingerprint()
+
+        should_write_evidence = force_persist or bool(change_reason) or self.persist_evidence
+        if should_write_evidence:
+            self._persist_evidence(state, parts, sensitive=sensitive)
+
+        write_model_json(self.states_dir / f"{state.id}.json", state)
+        self._by_fp[fp] = state
+        self._order.append(state.id)
+        self.latest = state
+        self._persist_observation(
+            state,
+            parts,
+            locatable=locatable,
+            sensitive=sensitive,
+            change_reason=change_reason or "capture",
+        )
+        return state
+
+    def _persist_observation(
+        self,
+        state: UIState,
+        parts: dict[str, Any],
+        *,
+        locatable: list[dict[str, Any]],
+        sensitive: bool,
+        change_reason: str,
+    ) -> UIObservation:
+        stats = dict(parts.get("stats") or {})
+        selection = dict(parts.get("selection") or {})
+        active_page = dict(stats.get("active_page") or {})
+        browser_session = dict(stats.get("browser_session") or {})
+        obs = UIObservation(
+            state_id=state.id,
+            previous_observation_id=(
+                self.latest_observation.id if self.latest_observation else None
+            ),
+            change_reason=change_reason,
+            surface=state.surface,
+            driver=state.driver,
+            page_id=str(
+                active_page.get("page_id")
+                or browser_session.get("page_id")
+                or selection.get("page_id")
+                or ""
+            ),
+            document_id=str(selection.get("document_id") or ""),
+            frame_id=str(selection.get("frame_id") or ""),
+            url=state.url,
+            page_title=state.page_title,
+            stats=stats,
+            selection=selection,
+            may_contain_sensitive_data=sensitive,
+        )
+        evidence_dir = self.observations_dir / obs.id
+        evidence_dir.mkdir(parents=True, exist_ok=True)
+        import artifacts as art
+
+        art.write_json(
+            evidence_dir / "tree.json",
+            sanitize_artifact(parts.get("tree") or {}),
+        )
+        clean_index = sanitize_artifact(list(parts.get("index") or []))
+        art.write_jsonl(evidence_dir / "index.jsonl", list(clean_index or []))
+        art.write_json(evidence_dir / "locatable.json", list(locatable or []))
+        obs.tree_path = f"observations/{obs.id}/tree.json"
+        obs.index_path = f"observations/{obs.id}/index.jsonl"
+        obs.locatable_path = f"observations/{obs.id}/locatable.json"
+
+        sensitivity_meta = screenshot_sensitivity_meta(
+            may_contain_sensitive=sensitive,
+            save_sensitive_screenshots=self.save_sensitive_screenshots,
+        )
+        art.write_json(evidence_dir / "sensitivity.json", sensitivity_meta)
+        if (not sensitive) or self.save_sensitive_screenshots:
+            png = parts.get("screenshot_bytes")
+            if png:
+                try:
+                    (evidence_dir / "screen.png").write_bytes(png)
+                    obs.screenshot_path = f"observations/{obs.id}/screen.png"
+                except Exception:
+                    pass
+            elif parts.get("use_hwnd_screenshot"):
+                hwnd = int((parts.get("window") or {}).get("hwnd") or 0)
+                if hwnd and art.capture_window_png(hwnd, evidence_dir / "screen.png"):
+                    obs.screenshot_path = f"observations/{obs.id}/screen.png"
+
+        write_model_json(evidence_dir / "meta.json", obs)
+        self.latest_observation = obs
+        self._observation_order.append(obs.id)
+        return obs
+
+    def _persist_evidence(
+        self,
+        state: UIState,
+        parts: dict[str, Any],
+        *,
+        sensitive: bool,
+    ) -> None:
+        evidence_dir = self.states_dir / state.id
+        evidence_dir.mkdir(parents=True, exist_ok=True)
+        import artifacts as art
+
+        art.write_json(
+            evidence_dir / "tree.json",
+            sanitize_artifact(parts.get("tree") or {}),
+        )
+        clean_index = sanitize_artifact(list(parts.get("index") or []))
+        art.write_jsonl(evidence_dir / "index.jsonl", list(clean_index or []))
+        art.write_json(evidence_dir / "locatable.json", list(state.locatable or []))
+        state.tree_path = f"states/{state.id}/tree.json"
+        state.index_path = f"states/{state.id}/index.jsonl"
+        state.locatable_path = f"states/{state.id}/locatable.json"
+
+        sens_meta = screenshot_sensitivity_meta(
+            may_contain_sensitive=sensitive,
+            save_sensitive_screenshots=self.save_sensitive_screenshots,
+        )
+        art.write_json(evidence_dir / "sensitivity.json", sens_meta)
+
+        save_png = (not sensitive) or self.save_sensitive_screenshots
+        if save_png:
+            png = parts.get("screenshot_bytes")
+            if png:
+                try:
+                    (evidence_dir / "screen.png").write_bytes(png)
+                    state.screenshot_path = f"states/{state.id}/screen.png"
+                except Exception:
+                    pass
+            elif parts.get("use_hwnd_screenshot"):
+                hwnd = int((parts.get("window") or {}).get("hwnd") or 0)
+                if hwnd:
+                    ok = art.capture_window_png(hwnd, evidence_dir / "screen.png")
+                    if ok:
+                        state.screenshot_path = f"states/{state.id}/screen.png"
+        elif sensitive:
+            state.may_contain_sensitive_data = True
+
+    def wait_until_stable(
+        self,
+        dump_fn,
+        *,
+        timeout_sec: float = 3.0,
+        poll_sec: float = 0.25,
+        change_reason: str = "after_action",
+    ) -> UIState:
+        """
+        Poll dump_fn until two consecutive fingerprints match or timeout.
+        dump_fn() -> DumpResult / dump-like object
+        """
+        deadline = time.monotonic() + max(0.2, float(timeout_sec))
+        last_fp: Optional[str] = None
+        last_dump = None
+        while True:
+            last_dump = dump_fn()
+            parts = _dump_to_parts(last_dump)
+            locatable, _ = redact_locatable_rows(list(parts.get("locatable") or []))
+            fp = compute_state_fingerprint(
+                surface=parts.get("surface") or "windows",
+                driver=parts.get("driver") or "",
+                url=parts.get("url") or "",
+                page_title=parts.get("page_title") or "",
+                window=parts.get("window") or {},
+                locatable=locatable,
+            )
+            if last_fp is not None and fp == last_fp:
+                break
+            last_fp = fp
+            if time.monotonic() >= deadline:
+                break
+            time.sleep(max(0.05, float(poll_sec)))
+        return self.capture_from_dump(
+            last_dump,
+            change_reason=change_reason,
+            force_persist=True,
+        )
+
+    def state_ids(self) -> list[str]:
+        return list(self._order)
+
+    def observation_ids(self) -> list[str]:
+        return list(self._observation_order)
+
+    def get(self, state_id: str) -> Optional[UIState]:
+        for st in self._by_fp.values():
+            if st.id == state_id:
+                return st
+        path = self.states_dir / f"{state_id}.json"
+        if path.exists():
+            import json
+            from models.state import UIState as S
+
+            data = json.loads(path.read_text(encoding="utf-8"))
+            return S(**{k: v for k, v in data.items() if k in S.__dataclass_fields__})
+        return None
+
+    def to_summary_dict(self, state: UIState) -> dict[str, Any]:
+        d = to_dict(state)
+        d["locatable"] = list(state.locatable or [])
+        return d

tools/ui_inspector/state_diff.py

@@ -0,0 +1,275 @@
+"""State Diff + proposed postconditions."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any, Optional
+
+
+@dataclass
+class StateChange:
+    kind: str
+    detail: dict[str, Any] = field(default_factory=dict)
+    confidence: float = 0.7
+
+
+@dataclass
+class ProposedPostcondition:
+    description: str
+    kind: str
+    status: str = "proposed"  # proposed | user_confirmed | runtime_verified
+    detail: dict[str, Any] = field(default_factory=dict)
+
+
+def _row_key(r: dict[str, Any]) -> str:
+    return "|".join(
+        [
+            str(r.get("control_type") or r.get("role") or ""),
+            str(r.get("automation_id") or ""),
+            str(r.get("name") or ""),
+            str(r.get("class_name") or ""),
+        ]
+    )
+
+
+def _row_value(r: dict[str, Any]) -> str:
+    return str(r.get("value") or r.get("text") or "")
+
+
+def _index_rows(rows: Optional[list[dict[str, Any]]]) -> dict[str, dict[str, Any]]:
+    out: dict[str, dict[str, Any]] = {}
+    for r in rows or []:
+        if not isinstance(r, dict):
+            continue
+        k = _row_key(r)
+        if k.strip("|"):
+            out[k] = r
+    return out
+
+
+def diff_states(
+    before: Optional[dict[str, Any]],
+    after: Optional[dict[str, Any]],
+) -> list[StateChange]:
+    """
+    Diff two state-like dicts (UIState.to_dict or lightweight summaries).
+
+    Expected keys: url, page_title, window, locatable, stats, selection
+    """
+    b = before or {}
+    a = after or {}
+    changes: list[StateChange] = []
+
+    b_url = str(b.get("url") or "")
+    a_url = str(a.get("url") or "")
+    if b_url != a_url:
+        changes.append(
+            StateChange(
+                kind="url_changed",
+                detail={"from": b_url, "to": a_url},
+            )
+        )
+
+    b_title = str(b.get("page_title") or (b.get("window") or {}).get("title") or "")
+    a_title = str(a.get("page_title") or (a.get("window") or {}).get("title") or "")
+    if b_title != a_title:
+        changes.append(
+            StateChange(
+                kind="title_changed",
+                detail={"from": b_title, "to": a_title},
+            )
+        )
+
+    b_cls = str((b.get("window") or {}).get("class_name") or "")
+    a_cls = str((a.get("window") or {}).get("class_name") or "")
+    b_hwnd = (b.get("window") or {}).get("hwnd")
+    a_hwnd = (a.get("window") or {}).get("hwnd")
+    if b_cls and a_cls and b_cls != a_cls:
+        changes.append(
+            StateChange(
+                kind="window_changed",
+                detail={"from_class": b_cls, "to_class": a_cls},
+            )
+        )
+    if b_hwnd and not a_hwnd:
+        changes.append(StateChange(kind="window_disappeared", detail={"class_name": b_cls}))
+    if a_hwnd and not b_hwnd:
+        changes.append(StateChange(kind="window_appeared", detail={"class_name": a_cls}))
+
+    # Popup heuristics by class name
+    for label, cls in (("before", b_cls), ("after", a_cls)):
+        del label
+    if "SearchResult" in a_cls and "SearchResult" not in b_cls:
+        changes.append(
+            StateChange(kind="popup_appeared", detail={"class_name": a_cls})
+        )
+    if "SearchResult" in b_cls and "SearchResult" not in a_cls:
+        changes.append(
+            StateChange(kind="popup_disappeared", detail={"class_name": b_cls})
+        )
+
+    b_rows = _index_rows(b.get("locatable") or [])
+    a_rows = _index_rows(a.get("locatable") or [])
+    b_keys = set(b_rows)
+    a_keys = set(a_rows)
+
+    for k in sorted(a_keys - b_keys)[:40]:
+        r = a_rows[k]
+        changes.append(
+            StateChange(
+                kind="element_appeared",
+                detail={
+                    "key": k,
+                    "name": r.get("name"),
+                    "value": _row_value(r),
+                    "control_type": r.get("control_type") or r.get("role"),
+                    "automation_id": r.get("automation_id"),
+                },
+            )
+        )
+    for k in sorted(b_keys - a_keys)[:40]:
+        r = b_rows[k]
+        changes.append(
+            StateChange(
+                kind="element_disappeared",
+                detail={
+                    "key": k,
+                    "name": r.get("name"),
+                    "value": _row_value(r),
+                    "control_type": r.get("control_type") or r.get("role"),
+                    "automation_id": r.get("automation_id"),
+                },
+            )
+        )
+    for k in sorted(b_keys & a_keys)[:80]:
+        bv = _row_value(b_rows[k])
+        av = _row_value(a_rows[k])
+        if bv != av:
+            changes.append(
+                StateChange(
+                    kind="element_value_changed",
+                    detail={
+                        "key": k,
+                        "name": a_rows[k].get("name"),
+                        "from": bv,
+                        "to": av,
+                        "automation_id": a_rows[k].get("automation_id"),
+                    },
+                )
+            )
+
+    b_focus = (b.get("selection") or {}).get("focused") or (b.get("selection") or {}).get(
+        "focus"
+    )
+    a_focus = (a.get("selection") or {}).get("focused") or (a.get("selection") or {}).get(
+        "focus"
+    )
+    if b_focus != a_focus and (b_focus or a_focus):
+        changes.append(
+            StateChange(
+                kind="focus_changed",
+                detail={"from": b_focus, "to": a_focus},
+            )
+        )
+
+    b_loc = int((b.get("stats") or {}).get("locatable_count") or len(b.get("locatable") or []))
+    a_loc = int((a.get("stats") or {}).get("locatable_count") or len(a.get("locatable") or []))
+    if b_loc != a_loc:
+        changes.append(
+            StateChange(
+                kind="locatable_count_changed",
+                detail={"from": b_loc, "to": a_loc},
+            )
+        )
+
+    return changes
+
+
+def propose_postconditions(changes: list[StateChange]) -> list[ProposedPostcondition]:
+    """Generate suggested postconditions (always status=proposed)."""
+    props: list[ProposedPostcondition] = []
+    for ch in changes:
+        kind = ch.kind
+        d = ch.detail or {}
+        if kind == "url_changed":
+            props.append(
+                ProposedPostcondition(
+                    description=f"URL became {d.get('to')!r}",
+                    kind="url_equals",
+                    detail=d,
+                )
+            )
+        elif kind == "popup_disappeared":
+            props.append(
+                ProposedPostcondition(
+                    description=f"Window/popup {d.get('class_name')!r} disappeared",
+                    kind="window_absent",
+                    detail=d,
+                )
+            )
+        elif kind == "popup_appeared":
+            props.append(
+                ProposedPostcondition(
+                    description=f"Window/popup {d.get('class_name')!r} appeared",
+                    kind="window_present",
+                    detail=d,
+                )
+            )
+        elif kind == "title_changed":
+            props.append(
+                ProposedPostcondition(
+                    description=f"Title became {d.get('to')!r}",
+                    kind="title_equals",
+                    detail=d,
+                )
+            )
+        elif kind == "element_appeared":
+            name = d.get("name") or d.get("automation_id") or d.get("value")
+            props.append(
+                ProposedPostcondition(
+                    description=f"Element {name!r} appeared",
+                    kind="element_present",
+                    detail=d,
+                )
+            )
+        elif kind == "element_disappeared":
+            name = d.get("name") or d.get("automation_id") or d.get("value")
+            props.append(
+                ProposedPostcondition(
+                    description=f"Element {name!r} disappeared",
+                    kind="element_absent",
+                    detail=d,
+                )
+            )
+        elif kind == "element_value_changed":
+            props.append(
+                ProposedPostcondition(
+                    description=(
+                        f"Element {d.get('name') or d.get('automation_id')!r} "
+                        f"value became {d.get('to')!r}"
+                    ),
+                    kind="element_value_equals",
+                    detail=d,
+                )
+            )
+    return props
+
+
+def changes_to_dicts(changes: list[StateChange]) -> list[dict[str, Any]]:
+    return [
+        {"kind": c.kind, "detail": c.detail, "confidence": c.confidence} for c in changes
+    ]
+
+
+def postconditions_to_dicts(
+    props: list[ProposedPostcondition],
+) -> list[dict[str, Any]]:
+    return [
+        {
+            "description": p.description,
+            "kind": p.kind,
+            "status": p.status,
+            "detail": p.detail,
+        }
+        for p in props
+    ]

tools/ui_inspector/storage/__init__.py

@@ -0,0 +1,6 @@
+"""Capture persistence API."""
+
+from core.pipeline.event_journal import EventJournal
+
+__all__ = ["EventJournal"]
+

tools/ui_inspector/storage/capture_reader.py

@@ -0,0 +1,20 @@
+"""Stable capture reader used by diagnostics and downstream AI tooling."""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+
+def read_json(path: Path) -> dict[str, Any]:
+    return json.loads(Path(path).read_text(encoding="utf-8"))
+
+
+def read_action_flow(flow_dir: Path) -> dict[str, Any]:
+    return read_json(Path(flow_dir) / "action_flow.json")
+
+
+def read_element_catalog(flow_dir: Path) -> dict[str, Any]:
+    return read_json(Path(flow_dir) / "element_catalog.json")
+

tools/ui_inspector/storage/capture_writer.py

@@ -0,0 +1,6 @@
+"""Compatibility export for the current atomic artifact writer."""
+
+from artifacts import create_flow, load_flow, save_flow, write_json
+
+__all__ = ["create_flow", "load_flow", "save_flow", "write_json"]
+

tools/ui_inspector/surfaces/__init__.py

@@ -0,0 +1,2 @@
+"""Automation surface implementations."""
+

tools/ui_inspector/surfaces/base.py

@@ -0,0 +1,19 @@
+"""Surface contracts shared by browser, Windows, and future mobile drivers."""
+
+from __future__ import annotations
+
+from typing import Any, Protocol
+
+
+class SurfaceRecorder(Protocol):
+    status: str
+    def start(self) -> None: ...
+    def stop(self) -> None: ...
+    def pause(self) -> None: ...
+    def resume(self) -> None: ...
+
+
+class SurfaceDumpDriver(Protocol):
+    surface: str
+    def dump(self, **kwargs: Any) -> Any: ...
+

tools/ui_inspector/surfaces/browser/__init__.py

@@ -0,0 +1,6 @@
+"""Browser surface public API."""
+
+from .recorder import BrowserRecorder
+from .dump_driver import BrowserPlaywrightDriver
+
+__all__ = ["BrowserRecorder", "BrowserPlaywrightDriver"]

tools/ui_inspector/surfaces/browser/dump_driver.py

@@ -0,0 +1,6 @@
+"""Browser full-dump driver export."""
+
+from drivers.browser_pw import BrowserPlaywrightDriver
+
+__all__ = ["BrowserPlaywrightDriver"]
+

tools/ui_inspector/surfaces/browser/locator_compiler.py

@@ -0,0 +1,6 @@
+"""Browser locator compiler export."""
+
+from locators.browser import compile_browser_locators
+
+__all__ = ["compile_browser_locators"]
+

tools/ui_inspector/surfaces/browser/locator_validator.py

@@ -0,0 +1,6 @@
+"""Browser live locator validator export."""
+
+from kit_browser import validate_element_on_page
+
+__all__ = ["validate_element_on_page"]
+

tools/ui_inspector/surfaces/browser/page_registry.py

@@ -0,0 +1,6 @@
+"""Stable browser page/frame identity facade."""
+
+from kit_browser import page_id_for
+
+__all__ = ["page_id_for"]
+

tools/ui_inspector/surfaces/browser/recorder.py

@@ -0,0 +1,6 @@
+"""Browser recorder compatibility export during staged migration."""
+
+from recorder.browser import BrowserRecorder
+
+__all__ = ["BrowserRecorder"]
+

tools/ui_inspector/surfaces/browser/runtime.py

@@ -0,0 +1,23 @@
+"""Inspector-owned Playwright runtime facade."""
+
+from kit_browser import (
+    call_on_each_open_page,
+    call_on_page,
+    clear_event_bridge,
+    dump_current_page,
+    ensure_session,
+    install_event_bridge,
+    page_id_for,
+    screenshot_png,
+    session_info,
+    shutdown,
+    validate_element_on_page,
+)
+
+__all__ = [
+    "call_on_each_open_page", "call_on_page", "clear_event_bridge",
+    "dump_current_page", "ensure_session", "install_event_bridge",
+    "page_id_for", "screenshot_png", "session_info", "shutdown",
+    "validate_element_on_page",
+]
+

tools/ui_inspector/surfaces/mobile/__init__.py

@@ -0,0 +1,6 @@
+"""Future mobile surface public API."""
+
+from drivers.mobile import MobileDriver
+
+__all__ = ["MobileDriver"]
+

tools/ui_inspector/surfaces/registry.py

@@ -0,0 +1,22 @@
+"""Resolve a dump driver through the canonical Surface namespace."""
+
+from __future__ import annotations
+
+import target
+from .browser.dump_driver import BrowserPlaywrightDriver
+from .mobile import MobileDriver
+from .windows.dump_driver import WindowsUiaDriver
+
+
+def get_driver(surface: str):
+    if surface == target.SURFACE_BROWSER:
+        return BrowserPlaywrightDriver()
+    if surface == target.SURFACE_WINDOWS:
+        return WindowsUiaDriver()
+    if surface in {target.SURFACE_ANDROID, target.SURFACE_IOS}:
+        return MobileDriver(surface=surface)
+    raise RuntimeError(f"ERROR:SURFACE_UNSUPPORTED unknown surface={surface!r}")
+
+
+__all__ = ["get_driver"]
+

tools/ui_inspector/surfaces/windows/__init__.py

@@ -0,0 +1,6 @@
+"""Windows surface public API."""
+
+from .recorder import WindowsRecorder
+from .dump_driver import WindowsUiaDriver
+
+__all__ = ["WindowsRecorder", "WindowsUiaDriver"]

tools/ui_inspector/surfaces/windows/dump_driver.py

@@ -0,0 +1,4 @@
+from drivers.windows_uia import WindowsUiaDriver
+
+__all__ = ["WindowsUiaDriver"]
+

tools/ui_inspector/surfaces/windows/locator_compiler.py

@@ -0,0 +1,4 @@
+from locators.windows import compile_windows_locators
+
+__all__ = ["compile_windows_locators"]
+

tools/ui_inspector/surfaces/windows/recorder.py

@@ -0,0 +1,4 @@
+from recorder.windows import WindowsRecorder
+
+__all__ = ["WindowsRecorder"]
+

tools/ui_inspector/target.py

@@ -0,0 +1,82 @@
+"""Resolve capture surface from a Win32 window (windows | browser | mobile stub)."""
+
+from __future__ import annotations
+
+from typing import Any, Optional
+
+# Logical surfaces (align with RPA_ELEMENT_LOCATING.md)
+SURFACE_WINDOWS = "windows"
+SURFACE_BROWSER = "browser"
+SURFACE_ANDROID = "android"
+SURFACE_IOS = "ios"
+
+_BROWSER_PROCESS_NAMES = {
+    "chrome.exe",
+    "msedge.exe",
+    "chromium.exe",
+    "brave.exe",
+    "opera.exe",
+    "vivaldi.exe",
+}
+
+_BROWSER_CLASS_HINTS = {
+    "Chrome_WidgetWin_1",
+    "Chrome_WidgetWin_0",
+    "Chrome_WidgetWin_2",
+}
+
+
+def normalize_process_name(name: str) -> str:
+    return (name or "").strip().lower()
+
+
+def is_browser_window(
+    *,
+    process_name: str = "",
+    class_name: str = "",
+    title: str = "",
+) -> bool:
+    proc = normalize_process_name(process_name)
+    if proc in _BROWSER_PROCESS_NAMES:
+        return True
+    cls = (class_name or "").strip()
+    if cls in _BROWSER_CLASS_HINTS:
+        return True
+    # Edge/Chrome sometimes expose empty process briefly; title heuristics last
+    t = title or ""
+    if " - Google Chrome" in t or " - Microsoft Edge" in t or t.endswith(" - Edge"):
+        return True
+    if t.endswith(" - Chrome"):
+        return True
+    return False
+
+
+def resolve_surface(
+    *,
+    process_name: str = "",
+    class_name: str = "",
+    title: str = "",
+) -> str:
+    """Return surface id for this top-level window."""
+    if is_browser_window(
+        process_name=process_name, class_name=class_name, title=title
+    ):
+        return SURFACE_BROWSER
+    return SURFACE_WINDOWS
+
+
+def surface_from_window_info(info: Any) -> str:
+    return resolve_surface(
+        process_name=getattr(info, "process_name", "") or "",
+        class_name=getattr(info, "class_name", "") or "",
+        title=getattr(info, "title", "") or "",
+    )
+
+
+def surface_label(surface: str) -> str:
+    return {
+        SURFACE_WINDOWS: "桌面 UIA",
+        SURFACE_BROWSER: "浏览器 Playwright",
+        SURFACE_ANDROID: "Android（未实现）",
+        SURFACE_IOS: "iOS（未实现）",
+    }.get(surface, surface)

tools/ui_inspector/tests/contract/__init__.py

@@ -0,0 +1 @@
+

tools/ui_inspector/tests/contract/test_capture_contract.py

@@ -0,0 +1,207 @@
+from pathlib import Path
+
+from flow_session import FlowSession
+from recorder.base import RawUserEvent
+from diagnostics.capture_validator import validate_capture
+from diagnostics.replay_preflight import build_replay_preflight
+
+
+def test_session_always_creates_raw_event_journal(tmp_path: Path) -> None:
+    session = FlowSession(tmp_path, flow_name="contract", surface="browser", dump_fn=None)
+    session.manager.start()
+    session.manager.status = "recording"
+    event = RawUserEvent(kind="click", surface="browser", target={"role": "button", "name": "Go"})
+    session.journal.append(event, channel="semantic")
+    assert (tmp_path / "journal" / "raw_events.jsonl").exists()
+
+
+def test_validator_flags_oversized_click_target(tmp_path: Path) -> None:
+    import json
+    actions = tmp_path / "actions"
+    actions.mkdir()
+    (tmp_path / "elements").mkdir()
+    (tmp_path / "observations").mkdir()
+    (tmp_path / "journal").mkdir()
+    (tmp_path / "journal" / "raw_events.jsonl").write_text("{}\n", encoding="utf-8")
+    (actions / "a.json").write_text(
+        json.dumps({
+            "id": "a", "sequence": 1, "action_type": "click",
+            "target_snapshot": {"rect": {"width": 1200, "height": 5000}},
+            "evidence": {"eventual_observation_id": "obs"},
+        }),
+        encoding="utf-8",
+    )
+    result = validate_capture(tmp_path)
+    assert any(issue["code"] == "CLICK_TARGET_OVERSIZED" for issue in result["issues"])
+
+
+def test_validator_flags_missing_desktop_input_and_generic_targets(tmp_path: Path) -> None:
+    import json
+
+    (tmp_path / "actions").mkdir()
+    (tmp_path / "elements").mkdir()
+    (tmp_path / "observations").mkdir()
+    (tmp_path / "journal").mkdir()
+    journal = {
+        "event": {
+            "kind": "typing_activity", "surface": "windows",
+            "target": {"name": "searchedit", "control_type": "TreeControl"},
+        }
+    }
+    (tmp_path / "journal" / "raw_events.jsonl").write_text(
+        json.dumps(journal) + "\n", encoding="utf-8"
+    )
+    rows = [
+        {
+            "id": "a1", "sequence": 1, "surface": "windows", "action_type": "click",
+            "target_snapshot": {
+                "name": "search_result", "control_type": "TreeControl",
+                "bounds": {"width": 600, "height": 500},
+            },
+            "evidence": {"eventual_observation_id": "obs"},
+        },
+        {
+            "id": "a2", "sequence": 2, "surface": "windows", "action_type": "press_key",
+            "target_snapshot": {
+                "name": "WeCom", "control_type": "WindowControl",
+                "bounds": {"width": 1440, "height": 900},
+            },
+            "input": {"key": "Enter"},
+            "evidence": {"eventual_observation_id": "obs"},
+        },
+    ]
+    for row in rows:
+        (tmp_path / "actions" / f"{row['id']}.json").write_text(
+            json.dumps(row), encoding="utf-8"
+        )
+    codes = {issue["code"] for issue in validate_capture(tmp_path)["issues"]}
+    assert "CLICK_TARGET_GENERIC_CONTAINER" in codes
+    assert "KEY_TARGET_ROOT_WINDOW" in codes
+    assert "TEXT_INPUT_TRANSACTION_MISSING" in codes
+    replay = build_replay_preflight(tmp_path)
+    assert replay["ready"] is False
+    assert replay["quality_gate"]["passed"] is False
+    assert replay["summary"]["quality_blockers"] >= 1
+
+
+def test_validator_flags_missing_production_locator_and_raw_link(tmp_path: Path) -> None:
+    import json
+
+    actions = tmp_path / "actions"
+    elements = tmp_path / "elements"
+    actions.mkdir()
+    elements.mkdir()
+    (tmp_path / "observations").mkdir()
+    (tmp_path / "journal").mkdir()
+    (tmp_path / "journal" / "raw_events.jsonl").write_text("{}\n", encoding="utf-8")
+    (elements / "el.json").write_text(
+        json.dumps({"id": "el", "primary_locator": None}), encoding="utf-8"
+    )
+    (actions / "a.json").write_text(
+        json.dumps({
+            "id": "a", "sequence": 1, "surface": "browser",
+            "action_type": "click", "target_element_id": "el",
+            "target_snapshot": {"role": "link", "rect": {"width": 20, "height": 20}},
+            "raw_event": {}, "evidence": {"eventual_observation_id": "obs"},
+        }),
+        encoding="utf-8",
+    )
+    result = validate_capture(tmp_path)
+    codes = {issue["code"] for issue in result["issues"]}
+    assert "ACTION_PRODUCTION_LOCATOR_MISSING" in codes
+    assert "ACTION_RAW_EVENT_LINK_MISSING" in codes
+
+
+def test_validator_flags_graphic_leaf_low_stability_and_split_fill(tmp_path: Path) -> None:
+    import json
+
+    actions = tmp_path / "actions"
+    elements = tmp_path / "elements"
+    actions.mkdir()
+    elements.mkdir()
+    (tmp_path / "observations").mkdir()
+    (tmp_path / "journal").mkdir()
+    (tmp_path / "journal" / "raw_events.jsonl").write_text("{}\n", encoding="utf-8")
+    locator = {
+        "strategy": "css", "expression": "css:.hash", "actual_count": 1,
+        "unique": True, "production_eligible": True, "stability_score": 0.32,
+        "validation": "live",
+    }
+    (elements / "el.json").write_text(
+        json.dumps({"id": "el", "primary_locator": locator}), encoding="utf-8"
+    )
+    rows = [
+        {
+            "id": "a1", "sequence": 1, "surface": "browser", "action_type": "fill",
+            "target_element_id": "el", "target_snapshot": {"tag": "input"},
+            "input": {"text": "a"},
+            "raw_event": {"meta": {"event_id": "doc:1", "source_ts_ms": 1000}},
+            "evidence": {"eventual_observation_id": "obs", "locator_unique": True},
+        },
+        {
+            "id": "a2", "sequence": 2, "surface": "browser", "action_type": "fill",
+            "target_element_id": "el", "target_snapshot": {"tag": "input"},
+            "input": {"text": "ai"},
+            "raw_event": {"meta": {"event_id": "doc:2", "source_ts_ms": 1100}},
+            "evidence": {"eventual_observation_id": "obs", "locator_unique": True},
+        },
+        {
+            "id": "a3", "sequence": 3, "surface": "browser", "action_type": "click",
+            "target_element_id": "el", "target_snapshot": {"tag": "path"},
+            "raw_event": {"meta": {"event_id": "doc:3"}},
+            "evidence": {"eventual_observation_id": "obs", "locator_unique": True},
+        },
+    ]
+    for row in rows:
+        (actions / f"{row['id']}.json").write_text(json.dumps(row), encoding="utf-8")
+    codes = {issue["code"] for issue in validate_capture(tmp_path)["issues"]}
+    assert "DUPLICATE_FILL_TRANSACTION" in codes
+    assert "GRAPHIC_LEAF_TARGET" in codes
+    assert "LOW_STABILITY_LOCATOR" in codes
+
+
+def test_replay_preflight_is_honest_about_event_time_validation(tmp_path: Path) -> None:
+    import json
+
+    (tmp_path / "actions").mkdir()
+    (tmp_path / "elements").mkdir()
+    (tmp_path / "actions" / "a.json").write_text(json.dumps({
+        "id": "a", "sequence": 1, "action_type": "click",
+        "target_element_id": "el", "raw_event_ids": ["doc:1"],
+        "raw_event": {"meta": {"page_id": "page", "document_id": "doc"}},
+    }), encoding="utf-8")
+    (tmp_path / "elements" / "el.json").write_text(json.dumps({
+        "id": "el", "logical_name": "go", "primary_locator": {
+            "strategy": "role", "expression": "role:button,Go",
+            "actual_count": 1, "unique": True, "production_eligible": True,
+            "stability_score": 0.95, "validation": "live",
+        },
+    }), encoding="utf-8")
+    report = build_replay_preflight(tmp_path)
+    assert report["mode"] == "static_non_mutating"
+    assert report["summary"]["pass"] == 1
+    assert report["actions"][0]["verification"] == "event_time_unique"
+
+
+def test_replay_preflight_rejects_unique_root_window_click(tmp_path: Path) -> None:
+    import json
+
+    (tmp_path / "actions").mkdir()
+    (tmp_path / "elements").mkdir()
+    (tmp_path / "actions" / "a.json").write_text(json.dumps({
+        "id": "a", "sequence": 1, "surface": "windows", "action_type": "click",
+        "target_element_id": "root", "target_snapshot": {
+            "control_type": "WindowControl", "name": "WeCom",
+            "bounds": {"width": 1440, "height": 900},
+        },
+    }), encoding="utf-8")
+    (tmp_path / "elements" / "root.json").write_text(json.dumps({
+        "id": "root", "logical_name": "WeCom", "primary_locator": {
+            "strategy": "window_name_type", "expression": "window:WeCom",
+            "actual_count": 1, "unique": True, "production_eligible": True,
+            "stability_score": 0.9, "validation": "snapshot_only",
+        },
+    }), encoding="utf-8")
+    report = build_replay_preflight(tmp_path)
+    assert report["summary"]["fail"] == 1
+    assert report["actions"][0]["reason"] == "oversized_or_root_window_click_target"

tools/ui_inspector/tests/test_locators_catalog.py

@@ -0,0 +1,215 @@
+"""Locator compiler / catalog / uniqueness tests."""
+
+from __future__ import annotations
+
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+_HERE = Path(__file__).resolve().parents[1]
+if str(_HERE) not in sys.path:
+    sys.path.insert(0, str(_HERE))
+
+from locators.browser import compile_browser_locators
+from locators.catalog import ElementCatalogService
+from locators.validator import choose_production_primary, validate_locator_against_rows
+from locators.windows import compile_windows_locators
+from models.locator import LocatorCandidate
+
+
+class TestBrowserLocators(unittest.TestCase):
+    def test_ordering_prefers_role_and_testid(self) -> None:
+        el = {
+            "role": "button",
+            "name": "Submit",
+            "test_id": "submit-btn",
+            "text": "Submit",
+            "id": "x1",
+        }
+        rows = [
+            {"role": "button", "name": "Submit", "test_id": "submit-btn", "text": "Submit", "id": "x1"},
+            {"role": "button", "name": "Cancel", "test_id": "cancel-btn", "text": "Cancel"},
+        ]
+        cands = compile_browser_locators(el, rows=rows)
+        strategies = [c.strategy for c in cands]
+        self.assertIn("role", strategies)
+        self.assertIn("test_id", strategies)
+        primary = choose_production_primary(cands)
+        self.assertIsNotNone(primary)
+        assert primary is not None
+        self.assertFalse(primary.uses_coordinates)
+
+    def test_contenteditable_ignores_dynamic_value_and_uses_role_css(self) -> None:
+        cands = compile_browser_locators(
+            {
+                "role": "combobox",
+                "name": "user typed comment",
+                "text": "user typed comment",
+                "value": "user typed comment",
+                "contenteditable": True,
+                "css": "#comment-input [contenteditable=true]",
+            }
+        )
+        role = next(c for c in cands if c.strategy == "role")
+        css = next(c for c in cands if c.strategy == "css")
+        self.assertEqual(role.properties, {"role": "combobox"})
+        self.assertNotIn("user typed comment", role.expression)
+        self.assertFalse(css.production_eligible)
+
+    def test_live_unique_css_becomes_executable_fallback(self) -> None:
+        with tempfile.TemporaryDirectory() as tmp:
+            service = ElementCatalogService(Path(tmp), flow_id="f")
+            el = service.upsert_from_snapshot(
+                {
+                    "role": "path",
+                    "tag": "path",
+                    "css": "#comment-input .submit svg path",
+                    "fingerprint": "path|submit",
+                },
+                surface="browser",
+                frame={"url": "https://example.test/video"},
+                live_counts={"css": 1},
+            )
+            self.assertIsNotNone(el.primary_locator)
+            assert el.primary_locator is not None
+            self.assertEqual(el.primary_locator.strategy, "css")
+            self.assertTrue(el.primary_locator.unique)
+
+    def test_link_href_beats_ambiguous_card_css(self) -> None:
+        with tempfile.TemporaryDirectory() as tmp:
+            service = ElementCatalogService(Path(tmp), flow_id="f")
+            el = service.upsert_from_snapshot(
+                {
+                    "role": "link",
+                    "tag": "a",
+                    "name": "A result card with dynamic text",
+                    "href": "/detail/42",
+                    "href_count": 1,
+                    "css": "#results > li.card > a.card-link",
+                    "css_count": 24,
+                    "fingerprint": "a|card",
+                },
+                surface="browser",
+                frame={"url": "https://example.test/search"},
+                live_counts={"href": 1, "css": 24},
+            )
+            self.assertIsNotNone(el.primary_locator)
+            assert el.primary_locator is not None
+            self.assertEqual(el.primary_locator.strategy, "href")
+            self.assertEqual(el.primary_locator.actual_count, 1)
+            self.assertTrue(el.primary_locator.production_eligible)
+
+    def test_scoped_contenteditable_stable_css_can_be_primary(self) -> None:
+        with tempfile.TemporaryDirectory() as tmp:
+            service = ElementCatalogService(Path(tmp), flow_id="f")
+            el = service.upsert_from_snapshot(
+                {
+                    "role": "combobox",
+                    "tag": "div",
+                    "contenteditable": True,
+                    "stable_css": "#comment-form div[contenteditable]",
+                    "stable_css_count": 1,
+                    "css": "#comment-form > div.hash-a > div.hash-b",
+                    "css_count": 1,
+                    "fingerprint": "div|editor",
+                },
+                surface="browser",
+                frame={"url": "https://example.test/detail"},
+                live_counts={"stable_css": 1, "css": 1},
+            )
+            self.assertIsNotNone(el.primary_locator)
+            assert el.primary_locator is not None
+            self.assertEqual(el.primary_locator.strategy, "stable_css")
+            self.assertGreater(el.primary_locator.stability_score, 0.7)
+
+
+class TestWindowsLocators(unittest.TestCase):
+    def test_ordering_and_name_value_split(self) -> None:
+        el = {
+            "automation_id": "searchedit",
+            "name": "searchedit",
+            "value": "张三",
+            "control_type": "EditControl",
+            "parent_chain": [{"name": "search_layout"}],
+        }
+        rows = [
+            {
+                "automation_id": "searchedit",
+                "name": "searchedit",
+                "value": "张三",
+                "control_type": "EditControl",
+            }
+        ]
+        cands = compile_windows_locators(
+            el, rows=rows, window_scope={"class_name": "WeWorkWindow", "title": "企业微信"}
+        )
+        strategies = [c.strategy for c in cands]
+        self.assertTrue(strategies[0].startswith("window_automation_id"))
+        self.assertIn("window_name_value_type", strategies)
+        primary = choose_production_primary(cands)
+        self.assertIsNotNone(primary)
+        assert primary is not None
+        self.assertEqual(primary.actual_count, 1)
+
+    def test_coordinates_never_primary(self) -> None:
+        el = {
+            "name": "x",
+            "control_type": "ButtonControl",
+            "bounds": {"left": 1, "top": 2, "right": 3, "bottom": 4},
+        }
+        # Two matches → ambiguous for name+type; coords present
+        rows = [
+            {"name": "x", "control_type": "ButtonControl"},
+            {"name": "x", "control_type": "ButtonControl"},
+        ]
+        cands = compile_windows_locators(el, rows=rows)
+        primary = choose_production_primary(cands)
+        self.assertIsNone(primary)
+        self.assertTrue(any(c.uses_coordinates for c in cands))
+        for c in cands:
+            if c.uses_coordinates:
+                self.assertFalse(c.production_eligible)
+
+
+class TestUniqueness(unittest.TestCase):
+    def test_ambiguous_not_primary(self) -> None:
+        c = LocatorCandidate(
+            strategy="name",
+            properties={"name": "item", "control_type": "ButtonControl"},
+            stability_score=0.8,
+        )
+        rows = [
+            {"name": "item", "control_type": "ButtonControl"},
+            {"name": "item", "control_type": "ButtonControl"},
+        ]
+        validate_locator_against_rows(c, rows)
+        self.assertTrue(c.ambiguous)
+        self.assertFalse(c.production_eligible)
+        self.assertIsNone(choose_production_primary([c]))
+
+
+class TestCatalogMerge(unittest.TestCase):
+    def test_merge_observations(self) -> None:
+        with tempfile.TemporaryDirectory() as tmp:
+            svc = ElementCatalogService(Path(tmp), flow_id="f1")
+            el = {
+                "automation_id": "ok",
+                "name": "ok",
+                "control_type": "ButtonControl",
+                "value": "",
+            }
+            rows = [el]
+            a = svc.upsert_from_snapshot(
+                el, surface="windows", window_scope={"class_name": "X"}, rows=rows
+            )
+            b = svc.upsert_from_snapshot(
+                el, surface="windows", window_scope={"class_name": "X"}, rows=rows
+            )
+            self.assertEqual(a.id, b.id)
+            self.assertEqual(b.observation_count, 2)
+            self.assertTrue(svc.catalog_path.exists())
+
+
+if __name__ == "__main__":
+    unittest.main()

tools/ui_inspector/tests/test_models_fingerprint_diff.py

@@ -0,0 +1,295 @@
+"""Phase 2 unit tests: models, fingerprint, dedupe, state diff, redaction."""
+
+from __future__ import annotations
+
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+_HERE = Path(__file__).resolve().parents[1]
+if str(_HERE) not in sys.path:
+    sys.path.insert(0, str(_HERE))
+
+from models.action import ActionFlow, ActionRecord
+from models.element import ElementRef, element_fingerprint
+from models.locator import LocatorCandidate
+from models.serialize import dumps_json, to_dict
+from models.state import UIState, compute_state_fingerprint
+from core.policies.element_identity import element_identity_key
+from redaction import redact_input_payload, redact_text
+from snapshot_service import SnapshotService
+from state_diff import diff_states, propose_postconditions
+
+
+class TestSerialization(unittest.TestCase):
+    def test_state_serialize(self) -> None:
+        st = UIState(
+            surface="browser",
+            driver="playwright",
+            url="https://example.com",
+            page_title="Example",
+            locatable=[{"control_type": "button", "name": "Go", "value": ""}],
+        )
+        st.ensure_fingerprint()
+        raw = dumps_json(st)
+        self.assertIn("openclaw.ui_inspector.state.v1", raw)
+        self.assertIn("https://example.com", raw)
+        self.assertNotIn("\\u", raw)  # ensure_ascii=False for CJK; ASCII ok either way
+        d = to_dict(st)
+        self.assertEqual(d["schema"], "openclaw.ui_inspector.state.v1")
+        self.assertTrue(d["fingerprint"])
+
+    def test_action_serialize(self) -> None:
+        act = ActionRecord(
+            action_type="click",
+            surface="windows",
+            before_state_id="st_a",
+            after_state_id="st_b",
+            sequence=1,
+        )
+        d = to_dict(act)
+        self.assertEqual(d["schema"], "openclaw.ui_inspector.action.v1")
+        self.assertEqual(d["capability"], "implemented")
+        flow = ActionFlow(flow_id="f1", flow_name="demo", actions=[act.id])
+        self.assertEqual(to_dict(flow)["schema"], "openclaw.ui_inspector.action_flow.v1")
+
+    def test_element_locator_serialize(self) -> None:
+        loc = LocatorCandidate(
+            strategy="automation_id",
+            properties={"automation_id": "ok", "control_type": "Button"},
+            expression="AutomationId=ok",
+            stability_score=0.9,
+        )
+        loc.mark_counts(1)
+        el = ElementRef(
+            logical_name="ok_button",
+            surface="windows",
+            automation_id="ok",
+            control_type="ButtonControl",
+            locator_candidates=[loc],
+            primary_locator=loc,
+        )
+        el.ensure_fingerprint()
+        d = to_dict(el)
+        self.assertEqual(d["schema"], "openclaw.ui_inspector.element.v1")
+        self.assertTrue(d["primary_locator"]["production_eligible"])
+
+
+class TestFingerprint(unittest.TestCase):
+    def test_stable_fingerprint(self) -> None:
+        kwargs = dict(
+            surface="browser",
+            url="https://a.test/x",
+            page_title="T",
+            window={"title": "T", "class_name": "Chrome", "process_name": "chrome.exe"},
+            locatable=[
+                {"control_type": "textbox", "name": "q", "value": "hello", "automation_id": "q"}
+            ],
+        )
+        a = compute_state_fingerprint(**kwargs)
+        b = compute_state_fingerprint(**kwargs)
+        self.assertEqual(a, b)
+
+    def test_session_fields_ignored(self) -> None:
+        base_loc = [{"control_type": "button", "name": "A", "value": "", "automation_id": "a"}]
+        fp1 = compute_state_fingerprint(
+            surface="windows",
+            window={"title": "App", "class_name": "X", "pid": 111, "hwnd": 222},
+            locatable=base_loc,
+        )
+        fp2 = compute_state_fingerprint(
+            surface="windows",
+            window={"title": "App", "class_name": "X", "pid": 999, "hwnd": 888},
+            locatable=base_loc,
+        )
+        self.assertEqual(fp1, fp2)
+
+    def test_url_changes_fingerprint(self) -> None:
+        loc = [{"control_type": "button", "name": "A", "value": "", "automation_id": "a"}]
+        a = compute_state_fingerprint(surface="browser", url="https://a.test/1", locatable=loc)
+        b = compute_state_fingerprint(surface="browser", url="https://a.test/2", locatable=loc)
+        self.assertNotEqual(a, b)
+
+    def test_anonymous_dom_elements_do_not_merge_across_paths(self) -> None:
+        common = dict(surface="browser", role="span", name="", value="")
+        a = element_fingerprint(**common, dom_fingerprint="span|#video-tab")
+        b = element_fingerprint(**common, dom_fingerprint="span|#comment-input")
+        self.assertNotEqual(a, b)
+
+    def test_contenteditable_value_is_not_identity(self) -> None:
+        common = dict(
+            surface="browser",
+            role="combobox",
+            contenteditable=True,
+            dom_fingerprint="div|#comment-input [contenteditable]",
+        )
+        self.assertEqual(
+            element_fingerprint(**common, name="first", value="first"),
+            element_fingerprint(**common, name="second", value="second"),
+        )
+
+    def test_link_href_is_identity_when_visible_text_changes(self) -> None:
+        common = {
+            "role": "link",
+            "tag": "a",
+            "href": "/detail/42",
+            "css": "#results > li.card > a",
+        }
+        self.assertEqual(
+            element_identity_key(
+                {**common, "name": "Result with 10 likes"},
+                surface="browser",
+                page_url="https://example.test/search?page=1",
+            ),
+            element_identity_key(
+                {**common, "name": "Result with 11 likes"},
+                surface="browser",
+                page_url="https://example.test/search?page=2",
+            ),
+        )
+
+    def test_value_change_detected(self) -> None:
+        a = compute_state_fingerprint(
+            surface="windows",
+            locatable=[{"control_type": "Edit", "name": "box", "value": "1", "automation_id": "e"}],
+        )
+        b = compute_state_fingerprint(
+            surface="windows",
+            locatable=[{"control_type": "Edit", "name": "box", "value": "2", "automation_id": "e"}],
+        )
+        self.assertNotEqual(a, b)
+
+
+class TestDedupe(unittest.TestCase):
+    def test_same_state_deduped(self) -> None:
+        with tempfile.TemporaryDirectory() as tmp:
+            svc = SnapshotService(Path(tmp), persist_evidence=False)
+
+            class Dump:
+                surface = "browser"
+                driver = "playwright"
+                window = {"title": "T", "class_name": "P"}
+                tree = {}
+                index = []
+                locatable = [
+                    {"control_type": "button", "name": "Go", "value": "", "automation_id": "go"}
+                ]
+                stats = {"locatable_count": 1, "nodes_kept": 1}
+                selection = {}
+                url = "https://example.com"
+                page_title = "T"
+                screenshot_bytes = None
+                use_hwnd_screenshot = False
+
+            s1 = svc.capture_from_dump(Dump(), change_reason="entry")
+            s2 = svc.capture_from_dump(Dump(), change_reason="heartbeat")
+            self.assertEqual(s1.id, s2.id)
+            self.assertEqual(s2.observation_count, 2)
+            self.assertEqual(len(svc.state_ids()), 1)
+
+
+class TestStateDiff(unittest.TestCase):
+    def test_diff_and_postconditions(self) -> None:
+        before = {
+            "url": "https://a.test/1",
+            "page_title": "One",
+            "window": {"class_name": "WeWorkWindow", "title": "企业微信", "hwnd": 1},
+            "locatable": [
+                {
+                    "control_type": "Edit",
+                    "name": "searchedit",
+                    "value": "q",
+                    "automation_id": "searchedit",
+                }
+            ],
+            "stats": {"locatable_count": 1},
+        }
+        after = {
+            "url": "https://a.test/1",
+            "page_title": "Chat",
+            "window": {"class_name": "WeWorkWindow", "title": "企业微信", "hwnd": 1},
+            "locatable": [
+                {
+                    "control_type": "Edit",
+                    "name": "send_rich_edit",
+                    "value": "",
+                    "automation_id": "send_rich_edit",
+                }
+            ],
+            "stats": {"locatable_count": 1},
+        }
+        changes = diff_states(before, after)
+        kinds = {c.kind for c in changes}
+        self.assertIn("title_changed", kinds)
+        self.assertIn("element_appeared", kinds)
+        self.assertIn("element_disappeared", kinds)
+        props = propose_postconditions(changes)
+        self.assertTrue(all(p.status == "proposed" for p in props))
+        self.assertTrue(any("appeared" in p.description.lower() or "出现" in p.description or "appeared" in p.description for p in props) or props)
+
+
+class TestRedaction(unittest.TestCase):
+    def test_password_redacted(self) -> None:
+        out, sens = redact_input_payload(
+            {"text": "secret123", "name": "password"},
+            field_name="password",
+            input_type="password",
+        )
+        self.assertTrue(sens)
+        self.assertEqual(out["text"], "[REDACTED]")
+
+    def test_phone_redacted(self) -> None:
+        self.assertIn("[REDACTED]", redact_text("call 13800138000 please"))
+
+    def test_raw_tree_and_index_are_sanitized_before_write(self) -> None:
+        secret = "Bearer_abcdefghijklmnopqrstuvwxyz123456"
+
+        class SensitiveDump:
+            surface = "browser"
+            driver = "playwright"
+            window = {"title": "Login"}
+            tree = {
+                "role": "textbox",
+                "name": "password",
+                "input_type": "password",
+                "value": secret,
+            }
+            index = [dict(tree)]
+            locatable = [dict(tree)]
+            stats = {}
+            selection = {}
+            url = "https://example.com/login"
+            page_title = "Login"
+            screenshot_bytes = None
+            use_hwnd_screenshot = False
+
+        with tempfile.TemporaryDirectory() as tmp:
+            root = Path(tmp)
+            svc = SnapshotService(root)
+            svc.capture_from_dump(
+                SensitiveDump(), change_reason="sensitive", force_persist=True
+            )
+            persisted = "\n".join(
+                p.read_text(encoding="utf-8", errors="ignore")
+                for p in root.rglob("*")
+                if p.is_file()
+            )
+            self.assertNotIn(secret, persisted)
+
+
+class TestActionBeforeAfter(unittest.TestCase):
+    def test_click_links_states(self) -> None:
+        act = ActionRecord(
+            action_type="click",
+            before_state_id="st_before",
+            after_state_id="st_after",
+            target_snapshot={"name": "btn"},
+        )
+        self.assertEqual(act.before_state_id, "st_before")
+        self.assertEqual(act.after_state_id, "st_after")
+        self.assertEqual(act.capability, "implemented")
+
+
+if __name__ == "__main__":
+    unittest.main()

tools/ui_inspector/tests/unit/__init__.py

@@ -0,0 +1 @@
+

tools/ui_inspector/tests/unit/core/__init__.py

@@ -0,0 +1 @@
+

tools/ui_inspector/tests/unit/core/test_event_journal_identity.py

@@ -0,0 +1,30 @@
+import json
+from pathlib import Path
+
+from core.pipeline.event_journal import EventJournal
+from core.policies.element_identity import element_identity_key
+
+
+def test_journal_is_append_only(tmp_path: Path) -> None:
+    journal = EventJournal(tmp_path)
+    assert journal.append({"kind": "pointerdown"}) == 1
+    assert journal.append({"kind": "click"}) == 2
+    rows = [json.loads(line) for line in journal.path.read_text(encoding="utf-8").splitlines()]
+    assert [row["journal_sequence"] for row in rows] == [1, 2]
+    assert [row["event"]["kind"] for row in rows] == ["pointerdown", "click"]
+
+
+def test_focus_classes_and_query_strings_do_not_split_identity() -> None:
+    first = {
+        "role": "textbox", "tag": "input", "placeholder": "Search",
+        "css": "#header input.search",
+    }
+    focused = {
+        **first,
+        "class_name": "search focused hash-123",
+        "css": "#header input.search.focused.hash-123",
+    }
+    one = element_identity_key(first, surface="browser", page_url="https://example.com/home?a=1")
+    two = element_identity_key(focused, surface="browser", page_url="https://example.com/results?q=x")
+    assert one == two
+

tools/ui_inspector/tests/unit/core/test_target_scoring.py

@@ -0,0 +1,70 @@
+from core.policies.target_scoring import resolve_browser_target
+
+
+def test_huge_control_named_ancestor_never_beats_comment_hit() -> None:
+    hit = {
+        "tag": "path", "role": "path", "cursor": "pointer",
+        "rect": {"left": 1047, "top": 545, "right": 1070, "bottom": 568, "width": 23, "height": 23},
+        "css": "#comment-input-container .send > svg > path",
+    }
+    huge = {
+        "tag": "div", "role": "div", "class_name": "playerControlHeight",
+        "rect": {"left": 160, "top": -244, "right": 1440, "bottom": 6698, "width": 1280, "height": 6942},
+        "css": "#root .playerControlHeight",
+    }
+    target, scored = resolve_browser_target(hit, [hit, huge], x=1056, y=558)
+    assert target["tag"] == "path"
+    assert any("oversized_ancestor" in row["reasons"] for row in scored)
+
+
+def test_nearest_semantic_wrapper_beats_svg_path() -> None:
+    hit = {
+        "tag": "path", "role": "path", "cursor": "pointer",
+        "rect": {"left": 10, "top": 10, "right": 30, "bottom": 30, "width": 20, "height": 20},
+    }
+    wrapper = {
+        "tag": "span", "role": "span", "cursor": "pointer", "class_name": "sendControl",
+        "rect": {"left": 5, "top": 5, "right": 35, "bottom": 35, "width": 30, "height": 30},
+    }
+    target, _ = resolve_browser_target(hit, [hit, wrapper], x=20, y=20)
+    assert target["class_name"] == "sendControl"
+
+
+def test_contenteditable_root_beats_transient_inner_block() -> None:
+    inner = {
+        "tag": "div", "role": "textbox", "contenteditable": True,
+        "contenteditable_attr": False,
+        "rect": {"left": 10, "top": 10, "right": 300, "bottom": 35, "width": 290, "height": 25},
+    }
+    root = {
+        "tag": "div", "role": "combobox", "explicit_role": "combobox",
+        "contenteditable": True, "contenteditable_attr": True,
+        "rect": {"left": 10, "top": 10, "right": 300, "bottom": 35, "width": 290, "height": 25},
+    }
+    target, _ = resolve_browser_target(inner, [inner, root], x=20, y=20)
+    assert target["contenteditable_attr"] is True
+
+
+def test_unique_clickable_wrapper_beats_ambiguous_graphic_leaf() -> None:
+    rect_path = {"left": 1047, "top": 345, "right": 1070, "bottom": 368, "width": 23, "height": 23}
+    rect_wrapper = {"left": 1041, "top": 338, "right": 1077, "bottom": 374, "width": 36, "height": 36}
+    path = {
+        "tag": "path", "role": "path", "cursor": "pointer",
+        "rect": rect_path, "css_count": 2,
+    }
+    svg = {
+        "tag": "svg", "role": "svg", "cursor": "pointer",
+        "rect": rect_wrapper, "css_count": 1,
+    }
+    wrapper = {
+        "tag": "span", "role": "span", "cursor": "pointer",
+        "rect": rect_wrapper, "css_count": 1,
+        "stable_css": "#comment-form > div.controls > span",
+        "stable_css_count": 1,
+    }
+    target, scored = resolve_browser_target(
+        path, [path, svg, wrapper], x=1055, y=352
+    )
+    assert target["tag"] == "span"
+    assert any("graphic_leaf" in row["reasons"] for row in scored)
+    assert any("event_time_unique_locator" in row["reasons"] for row in scored)

tools/ui_inspector/tests/unit/test_app_layout.py

@@ -0,0 +1,120 @@
+import json
+import sys
+from pathlib import Path
+
+_HERE = Path(__file__).resolve().parents[2]
+if str(_HERE) not in sys.path:
+    sys.path.insert(0, str(_HERE))
+
+from app import (
+    _DEFAULT_FLOW_NAME_SLUGS,
+    _default_window_geometry,
+    _flow_name_for_mode,
+    _format_geometry,
+    _format_live_action,
+    _merge_current_live_action,
+)
+import artifacts as art
+import target
+import user_prefs
+
+
+def test_default_geometry_is_narrow_tall_and_leaves_title_bar_clear() -> None:
+    width, height, x, y = _default_window_geometry((0, 0, 1920, 1040))
+
+    assert (width, height) == (576, 951)
+    assert (x, y) == (1328, 73)
+    assert y >= 64
+    assert x + width == 1920 - 16
+
+
+def test_default_geometry_supports_small_and_left_side_monitors() -> None:
+    width, height, x, y = _default_window_geometry(
+        (-1366, 0, 0, 728), side="left"
+    )
+
+    assert (width, height) == (560, 648)
+    assert (x, y) == (-1350, 64)
+    assert _format_geometry(width, height, x, y) == "560x648-1350+64"
+
+
+def test_flow_name_defaults_follow_pick_mode() -> None:
+    assert _flow_name_for_mode(target.SURFACE_BROWSER) == "browser"
+    assert _flow_name_for_mode(target.SURFACE_WINDOWS) == "desktop"
+    assert _flow_name_for_mode(target.SURFACE_ANDROID) == "mobile"
+    assert _DEFAULT_FLOW_NAME_SLUGS == frozenset({"browser", "desktop", "mobile"})
+
+
+def test_local_ts_is_compact_yyyymmddhhmmss() -> None:
+    stamp = art._local_ts()
+    assert len(stamp) == 14
+    assert stamp.isdigit()
+    assert "T" not in stamp
+    assert "+" not in stamp
+
+
+def test_create_flow_dir_uses_compact_timestamp(tmp_path: Path) -> None:
+    flow_dir = art.create_flow(tmp_path, flow_name="browser")
+    name = flow_dir.name
+    assert name.endswith("_browser")
+    stamp = name[: -len("_browser")]
+    assert len(stamp) == 14
+    assert stamp.isdigit()
+
+
+def test_hidden_legacy_recording_preference_migrates_to_auto(
+    tmp_path: Path, monkeypatch
+) -> None:
+    prefs_path = tmp_path / "user_prefs.json"
+    prefs_path.write_text(
+        json.dumps(
+            {
+                "mode": "browser",
+                "start_url": "https://example.com",
+                "recording_mode": "legacy_interval",
+            }
+        ),
+        encoding="utf-8",
+    )
+    monkeypatch.setattr(user_prefs, "_PREFS_PATH", prefs_path)
+
+    assert user_prefs.load_prefs()["recording_mode"] == "auto"
+
+
+def test_live_action_formatter_is_surface_neutral_and_two_phase() -> None:
+    text = _format_live_action(
+        {
+            "sequence": 3,
+            "action_type": "click",
+            "phase": "captured",
+            "surface": "browser",
+            "target_name": "提交",
+            "target_type": "button",
+        }
+    )
+    assert "#3 点击" in text
+    assert "browser" in text
+    assert "证据采集中" in text
+    assert "等待异步证据" in text
+
+
+def test_current_live_action_overwrites_without_allowing_late_evidence_to_regress() -> None:
+    first = {"id": "action-1", "sequence": 1, "phase": "captured"}
+    first_evidence = {"id": "action-1", "sequence": 1, "phase": "evidence"}
+    second = {"id": "action-2", "sequence": 2, "phase": "captured"}
+
+    current = _merge_current_live_action(None, first)
+    current = _merge_current_live_action(current, first_evidence)
+    assert current == first_evidence
+
+    current = _merge_current_live_action(current, second)
+    assert current == second
+
+    current = _merge_current_live_action(current, first_evidence)
+    assert current == second
+
+
+def test_current_live_action_ignores_updates_without_an_action_id() -> None:
+    current = {"id": "action-2", "sequence": 2, "phase": "captured"}
+
+    assert _merge_current_live_action(current, {"sequence": 3}) is current

tools/ui_inspector/ui/__init__.py

@@ -0,0 +1,2 @@
+"""User-interface package; legacy app.py remains the compatibility entry."""
+

tools/ui_inspector/ui/controller.py

@@ -0,0 +1,10 @@
+"""Application-facing session facade.
+
+The Tk view remains in app.py during the compatibility migration; new callers
+should depend on this facade instead of constructing recorder internals.
+"""
+
+from flow_session import FlowSession
+
+__all__ = ["FlowSession"]
+

tools/ui_inspector/user_prefs.py

@@ -0,0 +1,88 @@
+"""Persist UI Inspector prefs (pick mode + start URL + recording mode) across runs."""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+import target
+
+_HERE = Path(__file__).resolve().parent
+_PREFS_PATH = _HERE / "user_prefs.json"
+
+DEFAULT_MODE = target.SURFACE_BROWSER
+# 起始 URL：默认空；仅记住用户主动填写过的值
+DEFAULT_START_URL = ""
+DEFAULT_RECORDING_MODE = "auto"
+
+_VALID_MODES = {
+    target.SURFACE_BROWSER,
+    target.SURFACE_WINDOWS,
+    target.SURFACE_ANDROID,  # UI label「手机」
+}
+
+_VALID_RECORDING = {"auto", "guided", "checkpoint", "legacy_interval"}
+
+
+def prefs_path() -> Path:
+    return _PREFS_PATH
+
+
+def load_prefs() -> dict[str, Any]:
+    raw: dict[str, Any] = {}
+    try:
+        if _PREFS_PATH.is_file():
+            data = json.loads(_PREFS_PATH.read_text(encoding="utf-8"))
+            if isinstance(data, dict):
+                raw = data
+    except Exception:
+        raw = {}
+    mode = str(raw.get("mode") or DEFAULT_MODE).strip().lower()
+    if mode in {"mobile", "phone", "ios"}:
+        mode = target.SURFACE_ANDROID
+    if mode not in _VALID_MODES:
+        mode = DEFAULT_MODE
+    # 允许空字符串；未写过 start_url 键时也视为空
+    if "start_url" in raw:
+        start_url = str(raw.get("start_url") or "").strip()
+    else:
+        start_url = DEFAULT_START_URL
+    # v0.14.3 standardizes the public UI on automatic action recording.
+    # Older values remain accepted by lower-level capture APIs, but must not
+    # silently put the simplified UI into a hidden legacy mode.
+    recording_mode = DEFAULT_RECORDING_MODE
+    return {
+        "mode": mode,
+        "start_url": start_url,
+        "recording_mode": recording_mode,
+    }
+
+
+def save_prefs(
+    *,
+    mode: str,
+    start_url: str,
+    recording_mode: str = DEFAULT_RECORDING_MODE,
+) -> None:
+    mode_n = (mode or DEFAULT_MODE).strip().lower()
+    if mode_n in {"mobile", "phone", "ios"}:
+        mode_n = target.SURFACE_ANDROID
+    if mode_n not in _VALID_MODES:
+        mode_n = DEFAULT_MODE
+    url = (start_url or "").strip()
+    rec = (recording_mode or DEFAULT_RECORDING_MODE).strip().lower()
+    if rec not in _VALID_RECORDING:
+        rec = DEFAULT_RECORDING_MODE
+    payload = {
+        "mode": mode_n,
+        "start_url": url,
+        "recording_mode": rec,
+    }
+    try:
+        _PREFS_PATH.write_text(
+            json.dumps(payload, ensure_ascii=False, indent=2) + "\n",
+            encoding="utf-8",
+        )
+    except Exception:
+        pass