feat: add three-surface RPA locator standards and examples

examples/real_desktop_rpa/README.md

@@ -0,0 +1,135 @@
+# Real Windows Desktop RPA Example
+
+这是从已跑通业务技能 `D:\OpenClaw\client-marketing\add-friend-wecom` 提炼的 Windows UIA 参考案例。它用于教技术人员和 AI 编程工具如何从真实拾取结果得到稳定定位器，并完成窗口、控件、动作、等待和诊断闭环。
+
+## 状态
+
+- **技术路线已真实验证**：Windows UIA + Python `uiautomation`
+- **来源业务已真实跑通**：企业微信添加客户
+- **本目录默认安全模式**：只提供 probe、dry-run、assisted 范式，不默认发送真实邀请
+- **不是完整 skill 仓库**：按 Copy map 选择性复制
+
+## 先读
+
+1. [`../../development/RPA_ELEMENT_LOCATING.md`](../../development/RPA_ELEMENT_LOCATING.md)
+2. [`../../development/RPA_DESKTOP.md`](../../development/RPA_DESKTOP.md)
+3. 本 README 的“原始拾取到生产定位器”
+
+## 成功案例验证了什么
+
+```text
+枚举目标顶层窗口
+→ 按进程/窗口类/标题确认作用域
+→ 从 HWND 获取 UIA root
+→ 在父容器中定位元素
+→ 激活窗口并真实点击/输入
+→ 等待子窗口或业务结果
+→ 处理独立/嵌套弹窗
+→ probe-ui 诊断
+```
+
+来源案例还提供 `dry-run / assisted / auto` 三档、频率控制和业务记忆。本 example 只提炼 UIA 元素工程；账号、风控和数据库仍由具体业务技能实现。
+
+## 目录结构
+
+```text
+real_desktop_rpa/
+  README.md
+  assets/
+    element-map.sample.yaml
+    selector-capture.sample.xml
+  scripts/
+    service/
+      desktop_session.py
+      uia_locator.py
+      wecom_reference_flow.py
+    util/
+      selectors.py
+  tests/
+    test_selector_contract.py
+```
+
+## 原始拾取到生产定位器
+
+来源案例的影刀 `selectorsV2` 导出通常包含：
+
+```text
+Window(class=SearchExternalsWnd,title=添加客户)
+  → 多层 Layout/index
+  → search_layout
+  → searchedit
+```
+
+正确提炼方式：
+
+1. 保留稳定顶层窗口：进程、Window Class、标题。
+2. 保留最小业务父容器：如 `search_layout`。
+3. 使用叶子元素稳定属性：如 `Name=searchedit`。
+4. 删除只反映当前布局的深层 `index`。
+5. 验证父容器内候选数量为 `1`。
+6. 重启应用后再次验证。
+
+不要把完整 XML 路径或 `item[0]` 原样变成生产定位器。
+
+## Element map
+
+本目录的 [`assets/element-map.sample.yaml`](assets/element-map.sample.yaml) 只展示来源案例的元素登记方法。复制后必须重新拾取目标软件并全部替换。
+
+关键规则：
+
+- 窗口不唯一时失败，不取 `hwnds[0]`。
+- 控件不唯一时失败，不取第一个匹配。
+- `item`、`status_layout`、`wbtn_container` 等通用 Name 必须放进稳定父容器。
+- 行顺序不是身份；用业务文案或稳定属性选择结果行。
+- 只有主定位成功、窗口前台且矩形有效时，才允许从 UIA BoundingRectangle 降级坐标点击。
+
+## 三种实施模式
+
+| 模式 | 行为 | 用途 |
+|---|---|---|
+| `probe` | 只检查依赖、窗口和关键控件 | 安装、自检、版本变更排障 |
+| `dry-run` | 导航并填写，提交前停止 | 开发和验收默认 |
+| `assisted` | 填写后等待用户人工确认 | 不可逆动作或现场联调 |
+
+业务技能若增加 `auto`，必须实现明确确认策略、频率限制、业务后置状态和 `RESULT_UNCONFIRMED`。
+
+## Copy map
+
+| 来源 | 目标 | 说明 |
+|---|---|---|
+| `scripts/service/desktop_session.py` | `scripts/service/desktop_session.py` | 顶层窗口枚举、唯一性、激活 |
+| `scripts/service/uia_locator.py` | `scripts/service/uia_locator.py` | 作用域内唯一定位、点击、输入 |
+| `scripts/util/selectors.py` | `scripts/util/<platform>_selectors.py` | element spec；必须全部替换 |
+| `scripts/service/wecom_reference_flow.py` | `scripts/service/<business>_flow.py` | 只复制流程分层，不复制企微 selector |
+| `assets/element-map.sample.yaml` | `assets/<platform>-element-map.yaml` | 真实拾取后填写 |
+| `tests/test_selector_contract.py` | `tests/test_selector_contract.py` | 定位器契约测试思路 |
+
+## 必须补齐的业务能力
+
+- 目标应用安装/版本 doctor
+- 真实业务元素清单
+- 失败截图、UIA树和顶层窗口快照
+- Activity/RpaVideoSession 步骤
+- 不可逆动作确认
+- 业务成功/失败/结果不明确状态
+- 频率限制、幂等和断点续跑
+- 真实集成测试（默认不进 CI）
+
+## 禁止照抄
+
+- `WXWork`、`WeWorkWindow`、`SearchExternalsWnd`
+- `org_op_new`、`searchedit`、`wbtn_container`
+- 企业微信列表第 0/1 行含义
+- 来源案例的本机路径、账号和业务数据
+- 任何未经目标软件真实拾取验证的 fallback
+
+## 验收
+
+- `probe` 能明确报告依赖、窗口和关键元素
+- 每个关键元素在目标作用域内唯一
+- 关闭重开软件后仍能定位
+- 多窗口时不会误选
+- UI变化时停止并保存截图/UIA树
+- dry-run 不执行不可逆提交
+- assisted 清楚提示用户确认
+- 提交后仅在业务后置状态成立时报告成功

examples/real_desktop_rpa/assets/element-map.sample.yaml

@@ -0,0 +1,64 @@
+source_case: add-friend-wecom
+status: verified_source_redacted_example
+surface: windows
+driver: uiautomation
+elements:
+  - logical_name: main_window
+    purpose: 企业微信主窗口
+    scope:
+      process_name: WXWork
+    primary_locator:
+      window_class: WeWorkWindow
+      title: 企业微信
+    expected_count: 1
+    postcondition: UIA root 可读取
+    replace_in_new_skill: true
+
+  - logical_name: contacts_entry
+    purpose: 打开通讯录
+    scope:
+      logical_parent: main_window
+    primary_locator:
+      name: org_op_new
+      control_type: TreeItem
+    expected_count: 1
+    postcondition: 通讯录业务面板出现
+    replace_in_new_skill: true
+
+  - logical_name: add_customer_window
+    purpose: 添加客户窗口
+    scope:
+      process_name: WXWork
+    primary_locator:
+      window_class: SearchExternalsWnd
+      title: 添加客户
+    expected_count: 1
+    postcondition: search_input 可定位
+    replace_in_new_skill: true
+
+  - logical_name: search_input
+    purpose: 输入待搜索账号
+    scope:
+      logical_parent: add_customer_window
+      ancestor_name: search_layout
+    primary_locator:
+      name: searchedit
+    expected_count: 1
+    ready_condition:
+      visible: true
+      enabled: true
+    postcondition: 结果列表或用户不存在弹窗出现
+    replace_in_new_skill: true
+
+  - logical_name: verify_message_input
+    purpose: 填写添加验证语
+    scope:
+      window_class: InputReasonWnd
+      title_any:
+        - 输入认证信息
+        - 发送添加邀请
+    primary_locator:
+      name: name_edit
+    expected_count: 1
+    postcondition: 输入值可读取或发送按钮可用
+    replace_in_new_skill: true

examples/real_desktop_rpa/assets/selector-capture.sample.xml

@@ -0,0 +1,12 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!--
+  脱敏教学摘录：展示“拾取器长路径”而非可直接复制的生产 selector。
+  生产代码应保留稳定窗口、业务父容器和叶子属性，删除无业务含义的深层 index。
+-->
+<selector name="search_input" source="inspector-export">
+  <window process="WXWork" class="SearchExternalsWnd" title="添加客户" />
+  <uia name="content_container" index="1" />
+  <uia name="search_layout" index="0" />
+  <uia name="inner_search_layout" index="1" />
+  <uia name="searchedit" index="1" />
+</selector>

examples/real_desktop_rpa/scripts/service/desktop_session.py

@@ -0,0 +1,111 @@
+"""Windows 顶层窗口作用域：枚举、唯一性校验和激活。"""
+
+from __future__ import annotations
+
+import ctypes
+import ntpath
+from ctypes import wintypes
+from dataclasses import dataclass
+from typing import Iterable
+
+from util.selectors import WindowSpec
+
+
+class DesktopRpaError(RuntimeError):
+    def __init__(self, code: str, message: str) -> None:
+        super().__init__(f"ERROR:{code} {message}")
+        self.code = code
+        self.message = message
+
+
+@dataclass(frozen=True)
+class WindowCandidate:
+    hwnd: int
+    pid: int
+    process_name: str
+    class_name: str
+    title: str
+    visible: bool
+
+
+user32 = ctypes.windll.user32
+kernel32 = ctypes.windll.kernel32
+SW_RESTORE = 9
+PROCESS_QUERY_LIMITED_INFORMATION = 0x1000
+
+
+def _process_name(pid: int) -> str:
+    handle = kernel32.OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, False, pid)
+    if not handle:
+        return ""
+    try:
+        size = wintypes.DWORD(32768)
+        buffer = ctypes.create_unicode_buffer(size.value)
+        if not kernel32.QueryFullProcessImageNameW(
+            handle,
+            0,
+            buffer,
+            ctypes.byref(size),
+        ):
+            return ""
+        return ntpath.splitext(ntpath.basename(buffer.value))[0]
+    finally:
+        kernel32.CloseHandle(handle)
+
+
+def enumerate_top_windows() -> list[WindowCandidate]:
+    found: list[WindowCandidate] = []
+
+    @ctypes.WINFUNCTYPE(ctypes.c_bool, wintypes.HWND, wintypes.LPARAM)
+    def callback(hwnd, _lparam):
+        pid = wintypes.DWORD()
+        user32.GetWindowThreadProcessId(hwnd, ctypes.byref(pid))
+        cls = ctypes.create_unicode_buffer(256)
+        user32.GetClassNameW(hwnd, cls, 256)
+        length = user32.GetWindowTextLengthW(hwnd)
+        title = ctypes.create_unicode_buffer(length + 1)
+        user32.GetWindowTextW(hwnd, title, length + 1)
+        found.append(
+            WindowCandidate(
+                hwnd=int(hwnd),
+                pid=int(pid.value),
+                process_name=_process_name(int(pid.value)),
+                class_name=cls.value,
+                title=title.value,
+                visible=bool(user32.IsWindowVisible(hwnd)),
+            )
+        )
+        return True
+
+    user32.EnumWindows(callback, 0)
+    return found
+
+
+def resolve_unique_window(
+    spec: WindowSpec,
+    *,
+    candidates: Iterable[WindowCandidate] | None = None,
+) -> WindowCandidate:
+    pool = list(candidates) if candidates is not None else enumerate_top_windows()
+    matched = [
+        item
+        for item in pool
+        if item.visible
+        and item.process_name.casefold() == spec.process_name.casefold()
+        and item.class_name == spec.class_name
+        and (spec.title is None or item.title == spec.title)
+    ]
+    if not matched:
+        raise DesktopRpaError("WINDOW_NOT_FOUND", f"{spec.logical_name} 未找到")
+    if len(matched) != 1:
+        raise DesktopRpaError(
+            "ELEMENT_AMBIGUOUS",
+            f"{spec.logical_name} 匹配 {len(matched)} 个窗口，请按 PID/账号补充作用域",
+        )
+    return matched[0]
+
+
+def activate_window(window: WindowCandidate) -> None:
+    user32.ShowWindow(window.hwnd, SW_RESTORE)
+    if not user32.SetForegroundWindow(window.hwnd):
+        raise DesktopRpaError("WINDOW_ACTIVATE_FAILED", f"无法激活 hwnd={window.hwnd}")

examples/real_desktop_rpa/scripts/service/uia_locator.py

@@ -0,0 +1,96 @@
+"""作用域内 UIA 唯一定位、真实点击和逐字输入。"""
+
+from __future__ import annotations
+
+import random
+from typing import Any, Iterator
+
+from service.desktop_session import DesktopRpaError, WindowCandidate, activate_window
+from util.selectors import ElementSpec
+
+
+def require_uiautomation():
+    try:
+        import uiautomation as auto
+    except ImportError as exc:
+        raise DesktopRpaError("UIA_UNAVAILABLE", "缺少 uiautomation 依赖") from exc
+    return auto
+
+
+def root_from_window(window: WindowCandidate):
+    auto = require_uiautomation()
+    return auto.ControlFromHandle(window.hwnd)
+
+
+def _walk(root: Any, *, depth: int = 0, max_depth: int = 25) -> Iterator[Any]:
+    if depth > max_depth:
+        return
+    yield root
+    if depth == max_depth:
+        return
+    try:
+        children = root.GetChildren()
+    except Exception:
+        children = []
+    for child in children:
+        yield from _walk(child, depth=depth + 1, max_depth=max_depth)
+
+
+def _matches(control: Any, spec: ElementSpec) -> bool:
+    if spec.name is not None and (getattr(control, "Name", None) or "") != spec.name:
+        return False
+    if spec.automation_id is not None and (
+        getattr(control, "AutomationId", None) or ""
+    ) != spec.automation_id:
+        return False
+    if spec.control_type is not None and (
+        (getattr(control, "ControlTypeName", None) or "").removesuffix("Control")
+        != spec.control_type.removesuffix("Control")
+    ):
+        return False
+    if spec.class_name is not None and (
+        getattr(control, "ClassName", None) or ""
+    ) != spec.class_name:
+        return False
+    return True
+
+
+def locate_unique(root: Any, spec: ElementSpec, *, max_depth: int = 25):
+    search_root = root
+    if spec.ancestor_name:
+        ancestor_spec = ElementSpec(
+            logical_name=f"{spec.logical_name}.ancestor",
+            name=spec.ancestor_name,
+        )
+        search_root = locate_unique(root, ancestor_spec, max_depth=max_depth)
+
+    matched = [control for control in _walk(search_root, max_depth=max_depth) if _matches(control, spec)]
+    if not matched:
+        raise DesktopRpaError("ELEMENT_NOT_FOUND", f"{spec.logical_name} 未找到")
+    if len(matched) != spec.expected_count:
+        raise DesktopRpaError(
+            "ELEMENT_AMBIGUOUS",
+            f"{spec.logical_name} 期望 {spec.expected_count} 个，实际 {len(matched)} 个",
+        )
+    return matched[0]
+
+
+def click_control(control: Any, *, window: WindowCandidate) -> None:
+    activate_window(window)
+    rect = control.BoundingRectangle
+    if rect.width() <= 0 or rect.height() <= 0 or rect.left < -10000:
+        raise DesktopRpaError("ELEMENT_NOT_READY", "目标元素不可见或边界矩形无效")
+    control.Click(
+        x=random.randint(-2, 2),
+        y=random.randint(-2, 2),
+        simulateMove=True,
+    )
+
+
+def type_text(control: Any, text: str, *, window: WindowCandidate) -> None:
+    auto = require_uiautomation()
+    click_control(control, window=window)
+    auto.SendKeys("{Ctrl}a", waitTime=0.05)
+    auto.SendKeys("{Delete}", waitTime=0.05)
+    for char in text:
+        auto.SendKeys(char, waitTime=random.uniform(0.04, 0.12))

examples/real_desktop_rpa/scripts/service/wecom_reference_flow.py

@@ -0,0 +1,37 @@
+"""企业微信成功案例的安全提炼：probe 和 dry-run，不默认发送邀请。"""
+
+from __future__ import annotations
+
+from dataclasses import asdict
+from typing import Any
+
+from service.desktop_session import resolve_unique_window
+from service.uia_locator import locate_unique, root_from_window, type_text
+from util.selectors import ADD_CUSTOMER_WINDOW, CONTACTS_ENTRY, MAIN_WINDOW, SEARCH_INPUT
+
+
+def probe() -> dict[str, Any]:
+    main = resolve_unique_window(MAIN_WINDOW)
+    root = root_from_window(main)
+    contacts = locate_unique(root, CONTACTS_ENTRY)
+    return {
+        "success": True,
+        "main_window": asdict(main),
+        "contacts_entry": {
+            "name": getattr(contacts, "Name", ""),
+            "control_type": getattr(contacts, "ControlTypeName", ""),
+        },
+    }
+
+
+def fill_search_dry_run(target: str) -> dict[str, Any]:
+    """要求操作者已打开“添加客户”窗口；只填写搜索框，不提交。"""
+    search_window = resolve_unique_window(ADD_CUSTOMER_WINDOW)
+    root = root_from_window(search_window)
+    search_input = locate_unique(root, SEARCH_INPUT)
+    type_text(search_input, target, window=search_window)
+    return {
+        "success": True,
+        "mode": "dry-run",
+        "message": "已填写搜索条件，未发送邀请",
+    }

examples/real_desktop_rpa/scripts/util/selectors.py

@@ -0,0 +1,44 @@
+"""脱敏 UIA element specs；复制到新技能后必须用真实拾取结果替换。"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Optional
+
+
+@dataclass(frozen=True)
+class WindowSpec:
+    logical_name: str
+    process_name: str
+    class_name: str
+    title: Optional[str] = None
+
+
+@dataclass(frozen=True)
+class ElementSpec:
+    logical_name: str
+    name: Optional[str] = None
+    automation_id: Optional[str] = None
+    control_type: Optional[str] = None
+    class_name: Optional[str] = None
+    ancestor_name: Optional[str] = None
+    expected_count: int = 1
+
+
+MAIN_WINDOW = WindowSpec("main_window", "WXWork", "WeWorkWindow", "企业微信")
+ADD_CUSTOMER_WINDOW = WindowSpec(
+    "add_customer_window",
+    "WXWork",
+    "SearchExternalsWnd",
+    "添加客户",
+)
+CONTACTS_ENTRY = ElementSpec(
+    "contacts_entry",
+    name="org_op_new",
+    control_type="TreeItem",
+)
+SEARCH_INPUT = ElementSpec(
+    "search_input",
+    name="searchedit",
+    ancestor_name="search_layout",
+)

examples/real_desktop_rpa/tests/test_selector_contract.py

@@ -0,0 +1,44 @@
+from __future__ import annotations
+
+import sys
+import unittest
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT / "scripts"))
+
+from service.desktop_session import (  # noqa: E402
+    DesktopRpaError,
+    WindowCandidate,
+    resolve_unique_window,
+)
+from util.selectors import MAIN_WINDOW  # noqa: E402
+
+
+class SelectorContractTest(unittest.TestCase):
+    def test_unique_window_is_required(self) -> None:
+        candidate = WindowCandidate(1, 10, "WXWork", "WeWorkWindow", "企业微信", True)
+        self.assertEqual(
+            resolve_unique_window(MAIN_WINDOW, candidates=[candidate]),
+            candidate,
+        )
+
+    def test_duplicate_windows_fail_instead_of_taking_first(self) -> None:
+        candidates = [
+            WindowCandidate(1, 10, "WXWork", "WeWorkWindow", "企业微信", True),
+            WindowCandidate(2, 11, "WXWork", "WeWorkWindow", "企业微信", True),
+        ]
+        with self.assertRaises(DesktopRpaError) as ctx:
+            resolve_unique_window(MAIN_WINDOW, candidates=candidates)
+        self.assertEqual(ctx.exception.code, "ELEMENT_AMBIGUOUS")
+
+    def test_hidden_window_is_not_a_candidate(self) -> None:
+        hidden = WindowCandidate(1, 10, "WXWork", "WeWorkWindow", "企业微信", False)
+        with self.assertRaises(DesktopRpaError) as ctx:
+            resolve_unique_window(MAIN_WINDOW, candidates=[hidden])
+        self.assertEqual(ctx.exception.code, "WINDOW_NOT_FOUND")
+
+
+if __name__ == "__main__":
+    unittest.main()