Compare commits
14 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 22b42f8d30 | |||
| 689810eaeb | |||
| 7584e69714 | |||
| d3fee3f46c | |||
| ebea77b5f2 | |||
| 430274dbf9 | |||
| 0157fde0f1 | |||
| aa9bf262fe | |||
| 52831b80e0 | |||
| 17a7a54349 | |||
| c21339ae44 | |||
| 9878f570d7 | |||
| 44088336b4 | |||
| 293b7d1591 |
36
.env.example
36
.env.example
@@ -67,3 +67,39 @@ LLM_API_BASE_URL=
|
|||||||
# LLM API token(生产由宿主注入;本地留空)
|
# LLM API token(生产由宿主注入;本地留空)
|
||||||
LLM_API_TOKEN=
|
LLM_API_TOKEN=
|
||||||
|
|
||||||
|
# ----- v1.1.18:日志落文件 -----
|
||||||
|
# 日志文件路径(默认 {data_dir}/logs/alert-contract-expiring-hr.log);留空 = 仅 stderr
|
||||||
|
LOG_FILE=
|
||||||
|
# 单文件最大字节数(默认 5MB)
|
||||||
|
LOG_MAX_BYTES=
|
||||||
|
# 滚动备份保留份数(默认 5)
|
||||||
|
LOG_BACKUP_COUNT=
|
||||||
|
|
||||||
|
# ----- v1.1.18:用户主动输入目录 -----
|
||||||
|
# 用户主动投放目录(默认 {data_dir}/input)
|
||||||
|
# 子目录约定:
|
||||||
|
# input/contracts/*.csv → import-input 自动 upsert 到 contracts 表
|
||||||
|
# input/materials/{contract_id}__{filename}.{pdf|docx|doc} → 自动入 materials_content
|
||||||
|
INPUT_DIR=
|
||||||
|
|
||||||
|
# ----- v1.1.18:材料下载队列 -----
|
||||||
|
# 失败最大重试次数(默认 3)
|
||||||
|
DOWNLOAD_QUEUE_MAX_ATTEMPTS=
|
||||||
|
|
||||||
|
# ----- v1.1.31:SmartCampus HR 平台登录凭证(不硬编码到代码)-----
|
||||||
|
# 根域登录页(https://sandbox.jc2009.com/)和 SPA 二级登录页
|
||||||
|
# (/industries/campus/smartcampus/login)共用同一组凭证。
|
||||||
|
# 留空 → 脚本不自动 fill,等用户在浏览器手工输入。
|
||||||
|
SMARTCAMPUS_LOGIN_USER=
|
||||||
|
SMARTCAMPUS_LOGIN_PASSWORD=
|
||||||
|
|
||||||
|
# ----- v1.1.32:目标行业 tile(DEV MODE 面板)-----
|
||||||
|
# SPA 登录页右侧 DEV MODE 面板包含多个行业 tile(教师通用 / 人事处 / 财务处 / 科研处),
|
||||||
|
# 脚本自动点对应 tile 的 Sign in 把 demo 凭据搬到主表单。可选:
|
||||||
|
# hr → 人事处(默认)
|
||||||
|
# finance → 财务处
|
||||||
|
# research → 科研处
|
||||||
|
# demo → 教师(通用)
|
||||||
|
# 留空 → 默认 hr,且会探测主表单已预填的 user 值自动选择。
|
||||||
|
SMARTCAMPUS_LOGIN_INDUSTRY=
|
||||||
|
|
||||||
|
|||||||
120
CHANGELOG.md
120
CHANGELOG.md
@@ -2,6 +2,126 @@
|
|||||||
|
|
||||||
本文件对应技能市场 **「更新日志」** Tab。
|
本文件对应技能市场 **「更新日志」** Tab。
|
||||||
|
|
||||||
|
## 1.1.31
|
||||||
|
|
||||||
|
- **拆分 `scripts/service/platform_collector.py`(1972 行 / 84KB → 1 facade + 4 sub-modules,最大子模块 28KB),绕过 PyArmor trial mode 单文件大小限制**:
|
||||||
|
- **根因**:v1.1.22 ~ v1.1.30 期间 CI 全部红色 X(4s 即失败);`platform_collector.py` 从 v1.1.21 的 55 KB / 1319 行膨胀到 84 KB / 1972 行,超过 PyArmor 8.5.3 trial mode 单文件大小阈值
|
||||||
|
- **新结构**(5 个文件都在 `scripts/service/`,沿用平铺风格):
|
||||||
|
- `platform_collector.py`(facade,~180 行 / ~7 KB):保留模块常量 / 4 个异常类 / `_get_env` / `_load_env_file` / `_ENV_CACHE*` / re-export 21 个公开符号
|
||||||
|
- `platform_collector_login.py`(~700 行 / ~28 KB):12 个登录流程函数
|
||||||
|
- `platform_collector_collect.py`(~570 行 / ~23 KB):`collect_all` + 内嵌 `_try_reload_after_table_timeout` + `_error_result`
|
||||||
|
- `platform_collector_persist.py`(~200 行 / ~9 KB):`_persist_contracts` + `_enqueue_download_tasks_safe`
|
||||||
|
- `platform_collector_queue.py`(~290 行 / ~13 KB):`process_queue` + 5 个 queue/parse helper
|
||||||
|
- **零业务行为变化**:所有 CLI / Action / 测试保持不变;`tests/run_tests.py` 已有的 28 个 test case **零修改**通过(facade re-export + sub-module 内 `from service import platform_collector as _pc` lazy import 保留 `pc.HUMAN_POLL_INTERVAL = X` 测试 monkey-patch 能力)
|
||||||
|
- **新增 1 个 smoke test `facade_reexports_all_symbols_v1_31`**:验证 25 个公开符号从 facade 可见
|
||||||
|
- **修复 CI 红色 X**:v1.1.21 是最后一个 CI 绿色的版本;v1.1.31 拆分后所有子模块 < 35 KB,远低于 trial 阈值(v1.1.21 green 阈值 55 KB 的 50%),加密步骤应能通过
|
||||||
|
|
||||||
|
## 1.1.30
|
||||||
|
|
||||||
|
- **修复 jiangchang 客户端 arguments 丢失导致 argparse SystemExit 问题**:
|
||||||
|
- `generate-renewal-task --contract-id` / `dispatch-task-notice --unit-id` / `mark-contract-renewed --contract-id --material-id` / `render-html --task-id` 4 个 sync action 的 `--xxx` 参数从 `required=True` 改为 `default=""`
|
||||||
|
- CLI handler 检测到空值返回 JSON `{"error": "missing_arg", "detail": ..., "hint": ...}` + exit=1(不再 argparse 报错退出,宿主能拿到结构化错误)
|
||||||
|
- `assets/actions.json` 4 个 sync action 追加 `v1_1_30_notes` 字段说明新行为
|
||||||
|
- 1 个新测试 `sync_actions_tolerate_missing_args_v1_30` 覆盖 4 个 action 的 missing_arg JSON
|
||||||
|
|
||||||
|
## 1.1.29
|
||||||
|
|
||||||
|
- **修复「老 DB 没 download_tasks 表导致入队失败」问题**:
|
||||||
|
- 用户本地 `data/alert-contract-expiring-hr.db` 是 v1.1.17 之前 init 的(只有前 6 张表),没跑过 v1.1.18 的 init-db → 没 `download_tasks` 表
|
||||||
|
- `_enqueue_download_tasks_safe` 在 `owns_conn=True` 路径下,先 idempotent `CREATE TABLE IF NOT EXISTS download_tasks ...` + 2 索引,再 INSERT
|
||||||
|
- 解决 `OperationalError: no such table: download_tasks` + 下载队列 `failed=3`
|
||||||
|
- 新增测试 `enqueue_safe_auto_create_table_v1_29` 模拟「老 DB 没建过 download_tasks」场景(用 monkeypatch `rp.get_db_path` + `db.connection.get_db_path`),合计 67/67
|
||||||
|
|
||||||
|
## 1.1.28
|
||||||
|
|
||||||
|
- **彻底修复「SPA 二级登录页卡顿」**:
|
||||||
|
- v1.1.27 失败根因:URL 已稳定在 `/login` 时,React SPA input 可能还没渲染(hydrate 慢),`_fill_demo_on_spa_login` 找不到 input → False → 卡 180 秒等用户输入
|
||||||
|
- 修复:步骤 4.5 先 `wait_for_selector(input, timeout=10s, state="attached")` 等 input 真正 hydrate + 0.5s React 稳定,再 fill demo/123456 + 点登录
|
||||||
|
- demo fill 后如果 URL 仍卡在 login 页(demo 失效)→ headed 模式 fallback 等用户手工登录,headless 模式抛 `LoginTimeoutError`
|
||||||
|
- 不再依赖「input 是否预填」—— sandbox SPA 二级登录页 input 默认不预填,必须强制覆盖 fill
|
||||||
|
- 合计 66/66
|
||||||
|
|
||||||
|
## 1.1.27
|
||||||
|
|
||||||
|
- **修复「到 SPA 二级登录页卡 30 秒」问题**:
|
||||||
|
- 之前步骤 4 等 URL 稳定到 `/app/` 或 `/select` 30 秒,但 sandbox 登录后会把人踢回 `/industries/campus/smartcampus/login`(URL 含 `/login`),等满 30 秒才发现要进 SPA 登录兜底
|
||||||
|
- 步骤 4 改成等 URL 跳到 `/app/` 或 `/select` 或 `/login` 或 `/signin`(任一即可,**最多 30 秒**),URL 含 `/login` 立即退出等待
|
||||||
|
- **新增 `_fill_demo_on_spa_login(page)`**:SPA 二级登录页 input 默认不预填(与根域不同),主动 `fill('demo')` + `fill('123456')` + 点登录按钮(Sign in / 登录 / Login / submit 任一)
|
||||||
|
- 步骤 4.5 改为:URL 落在 SPA 登录页 → 自动 `_fill_demo_on_spa_login` → 不再卡 180 秒等用户输入
|
||||||
|
- 新增 3 个测试覆盖 `_fill_demo_on_spa_login`(basic / 中文按钮 / 无 input),合计 66/66
|
||||||
|
|
||||||
|
## 1.1.26
|
||||||
|
|
||||||
|
- **按用户最新要求改回「自动点登录」行为**:「到了 /industries/campus/smartcampus/login 整个页面,有账号和密码可用直接点登录跳转」
|
||||||
|
- `_wait_for_login_inputs_filled` 改回 v1.1.21 行为:input 都有值 → 自动点 Sign in / 登录按钮 → return True
|
||||||
|
- 与 `_try_auto_sign_in` 行为统一:登录页 = 「有数据自动点」/ 「无数据等用户输入」
|
||||||
|
- 测试替换:`wait_login_inputs_filled_no_auto_click_v1_24` → `wait_login_inputs_filled_auto_click_v1_26`,合计 63/63
|
||||||
|
|
||||||
|
## 1.1.25
|
||||||
|
|
||||||
|
- **按用户要求修正登录页等待逻辑**:
|
||||||
|
- 有数据(账号密码已存在)→ 自动点登录按钮(`_try_auto_sign_in`)
|
||||||
|
- **无数据 → 等待用户输入完成(最多 180 秒)→ 等 URL 跳离登录页**(按钮交给用户自己点,脚本不抢按钮)
|
||||||
|
- `_wait_for_login_inputs_filled` 重写为「**等待模式**」:不再自动点 Sign in / 登录按钮;只轮询 input 值 + URL,每 15 秒提示「请点登录按钮」或「等待输入」
|
||||||
|
- 根域登录 + SPA 二级登录页统一使用同一套行为
|
||||||
|
- 新增测试 `wait_login_inputs_filled_no_auto_click_v1_24` 验证不抢按钮,合计 63/63
|
||||||
|
|
||||||
|
## 1.1.24
|
||||||
|
|
||||||
|
- **修复 `/industries/campus/smartcampus/login` 等待过久问题**:sandbox 在用户首次登录后可能把人踢回 SPA 二级登录页(按钮中文「登录」),之前根域自动登录代码不识别这个 URL + 中文按钮,导致脚本卡死等用户输入
|
||||||
|
- `_try_auto_sign_in` 扩展为通用版本:支持 `#jc-user` / `input[type='tel']` / `input[autocomplete='username']` 多种 user input selector;按钮文本按优先级匹配 `Sign in` > `登录` > `Login` > `signin` > `submit`
|
||||||
|
- `collect_all` 登录后新增步骤 4.5:等 URL 稳定到 `/app/` 或 `/select` 后,若发现 URL 仍含 `/login` / `/signin`(落在 SPA 二级登录页),自动重走一次 `_try_auto_sign_in` → 再等 URL 跳到 `/app/` 或 `/select`
|
||||||
|
- 新增 4 个测试覆盖 `_try_auto_sign_in`(SPA 中文按钮 / 根域英文按钮 / input 空 / selector 超时),合计 58/58
|
||||||
|
|
||||||
|
## 1.1.23
|
||||||
|
|
||||||
|
- **修复登录后跳 /app/home 的流程**:
|
||||||
|
- `_wait_for_url_stable` 支持 tuple targets(任一匹配即稳定)
|
||||||
|
- 登录后等 URL 稳定到 `/app/` 或 `/select`(sandbox 中间页)
|
||||||
|
- 显式 `goto SPA_HOME_URL` (/app/home) → 等 sidebar link 出现 → click `<a href="/industries/campus/smartcampus/app/hr/contracts">合同到期续聘</a>`
|
||||||
|
- 删掉重复的 # 4 块(之前手抖复制粘贴的 bug)
|
||||||
|
|
||||||
|
## 1.1.22
|
||||||
|
|
||||||
|
- **登录流程接入 sandbox 自动登录**:root 登录页(https://sandbox.jc2009.com/)的 #jc-user / #jc-pass 默认有 demo/123456 预填
|
||||||
|
- 新增 `_try_auto_sign_in(page, wait_input_sec=2)`:等 input 出现 2 秒 → 检查是否有值 → 有则自动点 Sign in
|
||||||
|
- 新增 `_wait_for_login_inputs_filled(page, wait_sec)`:input 为空时等用户输入 → 两个 input 都有值后自动点 Sign in
|
||||||
|
- 已登录态(profile cookie 复用)直接跳过
|
||||||
|
- 输入超时 fallback 到 `_wait_for_manual_login`(多信号判定)
|
||||||
|
|
||||||
|
## 1.1.21
|
||||||
|
|
||||||
|
- **修复登录完成检测盲区**:`_wait_for_manual_login` 现在监听 3 个信号判断「登录已完成」,任一触发即继续
|
||||||
|
1. URL 含 `/app/` 且稳定 2 秒(v1.1.20 旧逻辑)
|
||||||
|
2. URL 已离开 `/login` / `/signin` / `/auth/` 等登录路径
|
||||||
|
3. 页面出现 sidebar 元素 `a[href$='/hr/contracts']`
|
||||||
|
- 解决「用户已经登录但脚本还在傻等」的回归:之前只判 URL 变化,cookie 已生效复用 profile 场景或 SPA 不跳 URL 的场景会卡 180 秒
|
||||||
|
|
||||||
|
## 1.1.20
|
||||||
|
|
||||||
|
- **登录流程重写**:第一跳改为 `https://sandbox.jc2009.com/`(根域,用户最初看到的页面),**不做任何操作**等 URL 稳定
|
||||||
|
- 已登录态:根域自动跳到 `/app/home` → 直接走侧边栏到 `hr/contracts` 采集
|
||||||
|
- 未登录态:**纯等用户在浏览器手工输入账号密码**,不再自动 fill demo/123456,不再 retry 强制 reload 打断用户输入
|
||||||
|
- 新增 `_wait_for_url_stable(page, target_substr, max_wait_sec)`:等 URL 真正稳定 2 秒后才返回,避免 sandbox 中间页误导
|
||||||
|
- `_wait_for_manual_login` 重写:每 5 秒轮询 URL(之前 2 秒太密),每 15 秒打印一次剩余时间提示,**等待期间不触发任何 goto / reload / click**
|
||||||
|
- 新增 4 个 login flow 测试(`wait_url_stable_immediate` / `wait_url_stable_transition` / `wait_url_stable_timeout` / `wait_manual_login_no_goto`),合计 58/58
|
||||||
|
|
||||||
|
## 1.1.19
|
||||||
|
|
||||||
|
- **修复 1.1.18 引入的采集回归**:把 `_persist_contracts` 里的入队调用移到独立事务 `_enqueue_download_tasks_safe`,FK 失败不再回滚 `contracts` / `materials_content` 落库;返回 JSON 新增 `download_queue` 字段方便 UI 直接看到入队结果
|
||||||
|
- 新增 4 个测试覆盖 `_enqueue_download_tasks_safe`(basic / idempotent / fk_failure / empty_list),合计 54/54
|
||||||
|
|
||||||
|
## 1.1.18
|
||||||
|
|
||||||
|
- **一键式"运维 + 材料管理"全打通**
|
||||||
|
- 新增 `show-config` / `logs-list` 两个 sync Action:UI 一键查看 .env 配置(敏感字段自动脱敏)与运行日志末尾 200 行
|
||||||
|
- 新增 `download-materials` async Action:UI 一键批量下载续聘材料包(pending 队列 → 浏览器抓取 → 落库 → 标记 success/failed)
|
||||||
|
- 新增 `import-input` CLI:扫描 `data/input/{contracts,materials}/` 自动入库,HR API 拿不到时用户可直接投放 CSV/PDF/Word
|
||||||
|
- **材料下载队列(download_tasks)**:新建表跟踪每个合同的材料抓取状态(pending/running/success/failed),失败自动重试最多 3 次
|
||||||
|
- **日志落文件**:本地降级路径新增 `RotatingFileHandler`(默认 5MB × 5 份),`LOG_FILE` / `LOG_MAX_BYTES` / `LOG_BACKUP_COUNT` 可覆盖
|
||||||
|
- 新增文件:`scripts/service/importer.py` / `scripts/service/config_viewer.py` / `scripts/service/logs_viewer.py`
|
||||||
|
- 现有 31 个测试全部保留;新增 17 个测试(合计 48/48)
|
||||||
|
|
||||||
## 1.1.17
|
## 1.1.17
|
||||||
|
|
||||||
- 补齐数据管理的“立即同步合同台账”工具栏入口,调用 SmartCampus 平台采集能力。
|
- 补齐数据管理的“立即同步合同台账”工具栏入口,调用 SmartCampus 平台采集能力。
|
||||||
|
|||||||
6
SKILL.md
6
SKILL.md
@@ -1,7 +1,7 @@
|
|||||||
---
|
---
|
||||||
name: 合同到期智能预警与续聘辅助
|
name: 合同到期智能预警与续聘辅助
|
||||||
description: "当用户说要扫描即将到期的合同、查 90/60/30 天预警名单、给某位教师生成续聘任务书、下发单位版续聘清单、从 SmartCampus HR 平台抓合同数据、解析续聘材料 PDF/Word、查看本月合同到期人员清单时,加载本技能。支持定时扫描(每日凌晨)+ 数据管理 + Agent 三入口,覆盖合同到期智能预警与续聘辅助全流程。"
|
description: "当用户说要扫描即将到期的合同、查 90/60/30 天预警名单、给某位教师生成续聘任务书、下发单位版续聘清单、从 SmartCampus HR 平台抓合同数据、解析续聘材料 PDF/Word、查看本月合同到期人员清单时,加载本技能。支持定时扫描(每日凌晨)+ 数据管理 + Agent 三入口,覆盖合同到期智能预警与续聘辅助全流程。"
|
||||||
version: 1.1.17
|
version: 1.1.31
|
||||||
author: 深圳匠厂科技有限公司
|
author: 深圳匠厂科技有限公司
|
||||||
metadata:
|
metadata:
|
||||||
openclaw:
|
openclaw:
|
||||||
@@ -61,6 +61,10 @@ allowed-tools:
|
|||||||
| 在浏览器中打开续聘任务书 | `run_skill_action` → `render-html` | 否(sync) |
|
| 在浏览器中打开续聘任务书 | `run_skill_action` → `render-html` | 否(sync) |
|
||||||
| 从 SmartCampus 平台抓合同 + 材料 | `run_skill_action` → `collect-from-platform`(首次需用户手工登录保存 profile) | 否(sync) |
|
| 从 SmartCampus 平台抓合同 + 材料 | `run_skill_action` → `collect-from-platform`(首次需用户手工登录保存 profile) | 否(sync) |
|
||||||
| 解析已下载材料 PDF/Word | `run_skill_action` → `parse-materials`(可选 `--contract-id`) | 否(sync) |
|
| 解析已下载材料 PDF/Word | `run_skill_action` → `parse-materials`(可选 `--contract-id`) | 否(sync) |
|
||||||
|
| 查看运行时配置(脱敏) | `run_skill_action` → `show-config` | 否(sync) |
|
||||||
|
| 查看运行日志末尾 N 行 | `run_skill_action` → `logs-list`(默认 tail=200、level=INFO) | 否(sync) |
|
||||||
|
| 批量下载材料包(队列) | `run_skill_action` → `download-materials`(输入:itemIds / pickCount) | 是(async) |
|
||||||
|
| 扫描输入目录并入库 CSV/材料 | `run_skill_action` → `import-input` | 是(async) |
|
||||||
|
|
||||||
## 共享 Python(必读)
|
## 共享 Python(必读)
|
||||||
|
|
||||||
|
|||||||
@@ -83,7 +83,8 @@
|
|||||||
"type": "cli",
|
"type": "cli",
|
||||||
"command": "generate-renewal-task",
|
"command": "generate-renewal-task",
|
||||||
"args": ["--contract-id", "{row.合同ID}"]
|
"args": ["--contract-id", "{row.合同ID}"]
|
||||||
}
|
},
|
||||||
|
"v1_1_30_notes": "contract-id 缺失时返回 JSON 提示(missing_arg),不再 argparse 报错。"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "dispatch-task-notice",
|
"id": "dispatch-task-notice",
|
||||||
@@ -99,7 +100,8 @@
|
|||||||
"type": "cli",
|
"type": "cli",
|
||||||
"command": "dispatch-task-notice",
|
"command": "dispatch-task-notice",
|
||||||
"args": ["--unit-id", "{form.unit_id}", "--month", "{form.month}"]
|
"args": ["--unit-id", "{form.unit_id}", "--month", "{form.month}"]
|
||||||
}
|
},
|
||||||
|
"v1_1_30_notes": "unit-id 缺失时返回 JSON 提示(missing_arg),不再 argparse 报错。"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "mark-contract-renewed",
|
"id": "mark-contract-renewed",
|
||||||
@@ -116,7 +118,8 @@
|
|||||||
"type": "cli",
|
"type": "cli",
|
||||||
"command": "mark-contract-renewed",
|
"command": "mark-contract-renewed",
|
||||||
"args": ["--contract-id", "{row.合同ID}", "--material-id", "{form.material_id}"]
|
"args": ["--contract-id", "{row.合同ID}", "--material-id", "{form.material_id}"]
|
||||||
}
|
},
|
||||||
|
"v1_1_30_notes": "contract-id/material-id 缺失时返回 JSON 提示(missing_arg),不再 argparse 报错。"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "render-html",
|
"id": "render-html",
|
||||||
@@ -128,6 +131,73 @@
|
|||||||
"type": "cli",
|
"type": "cli",
|
||||||
"command": "render-html",
|
"command": "render-html",
|
||||||
"args": ["--task-id", "{form.task_id}"]
|
"args": ["--task-id", "{form.task_id}"]
|
||||||
|
},
|
||||||
|
"v1_1_30_notes": "task-id 缺失时返回 JSON 提示(missing_arg),不再 argparse 报错。"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "show-config",
|
||||||
|
"label": "查看运行时配置",
|
||||||
|
"description": "展示 .env 中已设置的全部非空配置项;含 TOKEN/SECRET/KEY/PASSWORD 的字段自动脱敏。",
|
||||||
|
"placements": ["skill-detail", "agent"],
|
||||||
|
"executionProfile": "sync",
|
||||||
|
"entrypoint": {
|
||||||
|
"type": "cli",
|
||||||
|
"command": "show-config",
|
||||||
|
"args": []
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "logs-list",
|
||||||
|
"label": "查看运行日志",
|
||||||
|
"description": "读取日志文件末尾 N 行(默认 200 行),按 INFO/WARNING/ERROR 等级过滤;日志文件不存在时返回 exists=false。",
|
||||||
|
"placements": ["skill-detail", "agent"],
|
||||||
|
"executionProfile": "sync",
|
||||||
|
"entrypoint": {
|
||||||
|
"type": "cli",
|
||||||
|
"command": "logs-list",
|
||||||
|
"args": ["--tail", "200", "--level", "INFO"]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "download-materials",
|
||||||
|
"label": "批量下载材料包",
|
||||||
|
"description": "从合同台账的 download_tasks 队列领取 N 条 pending 任务,启动浏览器从 SmartCampus HR 平台下载续聘材料包(pdf/docx)。",
|
||||||
|
"placements": ["toolbar", "batch", "cron", "agent"],
|
||||||
|
"executionProfile": "async",
|
||||||
|
"interaction": "form",
|
||||||
|
"bind": {
|
||||||
|
"tables": ["download_tasks"],
|
||||||
|
"selectionMode": "optional",
|
||||||
|
"labelColumn": "contract_id",
|
||||||
|
"inputMapping": {"itemIds": "$selection.ids"}
|
||||||
|
},
|
||||||
|
"confirmation": {
|
||||||
|
"message": "将启动浏览器下载续聘材料包(仅 pending 会执行),是否继续?"
|
||||||
|
},
|
||||||
|
"entrypoint": {
|
||||||
|
"type": "cli",
|
||||||
|
"command": "download-materials",
|
||||||
|
"args": ["--ids", "{form.itemIds}", "--pick", "{form.pickCount}"]
|
||||||
|
},
|
||||||
|
"inputSchema": {
|
||||||
|
"type": "object",
|
||||||
|
"required": ["itemIds", "pickCount"],
|
||||||
|
"properties": {
|
||||||
|
"itemIds": {"type": "string", "title": "队列编号列表", "readOnly": true},
|
||||||
|
"pickCount": {"type": "number", "title": "领取数量", "default": 5, "minimum": 1, "maximum": 50}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "import-input",
|
||||||
|
"label": "扫描输入目录并入库",
|
||||||
|
"description": "主动扫描 data/input/contracts/*.csv 入库合同,data/input/materials/{contract_id}__*.* 入库材料(自动入库到 materials_content)。",
|
||||||
|
"placements": ["skill-detail", "agent", "cron"],
|
||||||
|
"executionProfile": "async",
|
||||||
|
"entrypoint": {
|
||||||
|
"type": "cli",
|
||||||
|
"command": "import-input",
|
||||||
|
"args": []
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
|
|||||||
BIN
assets/屏幕截图 2026-09-15 021133.png
Normal file
BIN
assets/屏幕截图 2026-09-15 021133.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 589 KiB |
BIN
assets/屏幕截图 2026-09-15 021332.png
Normal file
BIN
assets/屏幕截图 2026-09-15 021332.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 156 KiB |
@@ -127,6 +127,14 @@ def cmd_scan(args: argparse.Namespace) -> int:
|
|||||||
def cmd_generate_task(args: argparse.Namespace) -> int:
|
def cmd_generate_task(args: argparse.Namespace) -> int:
|
||||||
contract_id = args.contract_id
|
contract_id = args.contract_id
|
||||||
task_id = args.task_id
|
task_id = args.task_id
|
||||||
|
# v1.1.30:友好提示 —— 空 contract_id(客户端 arguments 丢失场景)
|
||||||
|
if not contract_id:
|
||||||
|
print(json.dumps({
|
||||||
|
"error": "missing_arg",
|
||||||
|
"detail": "缺少必填参数 --contract-id。请在调用方传入合同 ID。",
|
||||||
|
"hint": "可先运行 `collect-from-platform` 抓取合同台账,或在数据管理 Tab 选中一行后再调用。",
|
||||||
|
}, ensure_ascii=False, indent=2))
|
||||||
|
return 1
|
||||||
conn = get_connection()
|
conn = get_connection()
|
||||||
try:
|
try:
|
||||||
result = task_generator.regenerate_task_for_contract_id(conn, contract_id, task_id)
|
result = task_generator.regenerate_task_for_contract_id(conn, contract_id, task_id)
|
||||||
@@ -141,6 +149,14 @@ def cmd_generate_task(args: argparse.Namespace) -> int:
|
|||||||
|
|
||||||
|
|
||||||
def cmd_dispatch(args: argparse.Namespace) -> int:
|
def cmd_dispatch(args: argparse.Namespace) -> int:
|
||||||
|
# v1.1.30:友好提示 —— 空 unit_id
|
||||||
|
if not args.unit_id:
|
||||||
|
print(json.dumps({
|
||||||
|
"error": "missing_arg",
|
||||||
|
"detail": "缺少必填参数 --unit-id。请在调用方传入二级单位 ID。",
|
||||||
|
"hint": "可参考 contracts 表的 unit_id 字段;或用 --month 限定月份范围。",
|
||||||
|
}, ensure_ascii=False, indent=2))
|
||||||
|
return 1
|
||||||
conn = get_connection()
|
conn = get_connection()
|
||||||
try:
|
try:
|
||||||
result = dispatcher.dispatch(conn, args.unit_id, args.month or "")
|
result = dispatcher.dispatch(conn, args.unit_id, args.month or "")
|
||||||
@@ -151,6 +167,17 @@ def cmd_dispatch(args: argparse.Namespace) -> int:
|
|||||||
|
|
||||||
|
|
||||||
def cmd_mark_renewed(args: argparse.Namespace) -> int:
|
def cmd_mark_renewed(args: argparse.Namespace) -> int:
|
||||||
|
# v1.1.30:友好提示 —— 空 contract_id 或 material_id
|
||||||
|
if not args.contract_id or not args.material_id:
|
||||||
|
print(json.dumps({
|
||||||
|
"error": "missing_arg",
|
||||||
|
"detail": (
|
||||||
|
f"缺少必填参数 {'--contract-id ' if not args.contract_id else ''}"
|
||||||
|
f"{'--material-id ' if not args.material_id else ''}(由调用方传入)。"
|
||||||
|
),
|
||||||
|
"hint": "合同 ID 来自 contracts 表,材料 ID 来自 materials_content 表。",
|
||||||
|
}, ensure_ascii=False, indent=2))
|
||||||
|
return 1
|
||||||
conn = get_connection()
|
conn = get_connection()
|
||||||
try:
|
try:
|
||||||
from db import repository as repo
|
from db import repository as repo
|
||||||
@@ -176,6 +203,14 @@ def cmd_mark_renewed(args: argparse.Namespace) -> int:
|
|||||||
|
|
||||||
|
|
||||||
def cmd_render_html(args: argparse.Namespace) -> int:
|
def cmd_render_html(args: argparse.Namespace) -> int:
|
||||||
|
# v1.1.30:友好提示 —— 空 task_id
|
||||||
|
if not args.task_id:
|
||||||
|
print(json.dumps({
|
||||||
|
"error": "missing_arg",
|
||||||
|
"detail": "缺少必填参数 --task-id。请在调用方传入任务 ID。",
|
||||||
|
"hint": "可先运行 generate-renewal-task 生成任务,再用其 task_id 调用。",
|
||||||
|
}, ensure_ascii=False, indent=2))
|
||||||
|
return 1
|
||||||
conn = get_connection()
|
conn = get_connection()
|
||||||
try:
|
try:
|
||||||
output_path = html_renderer.render_task_html(conn, args.task_id)
|
output_path = html_renderer.render_task_html(conn, args.task_id)
|
||||||
@@ -279,6 +314,72 @@ def cmd_parse_materials(args: argparse.Namespace) -> int:
|
|||||||
conn.close()
|
conn.close()
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_import_input(_args: argparse.Namespace) -> int:
|
||||||
|
"""扫描 data/input/{contracts,materials}/ 自动入库。
|
||||||
|
|
||||||
|
约定:
|
||||||
|
input/contracts/*.csv → contracts 表 upsert
|
||||||
|
input/materials/{cid}__{f}.pdf → materials_content (parse_method='pending')
|
||||||
|
"""
|
||||||
|
from service.importer import import_input_dir
|
||||||
|
|
||||||
|
conn = get_connection()
|
||||||
|
try:
|
||||||
|
result = import_input_dir(conn)
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_download_materials(args: argparse.Namespace) -> int:
|
||||||
|
"""处理 download_tasks 队列:抓 pending 任务的材料 → 落库。
|
||||||
|
|
||||||
|
--ids "DLQ-xxx,DLQ-yyy": 只处理这些任务
|
||||||
|
--pick N: 无 --ids 时从 pending 领取 N 条(默认 5)
|
||||||
|
--limit N: 本次硬上限(默认 0 = 不限)
|
||||||
|
--headless: 无头模式(需要 profile cookie 已保存)
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
result = platform_collector.process_queue(
|
||||||
|
limit=args.limit or args.pick,
|
||||||
|
ids_filter=args.ids,
|
||||||
|
headless=args.headless,
|
||||||
|
human_wait_timeout=args.human_wait_timeout or HUMAN_WAIT_TIMEOUT,
|
||||||
|
)
|
||||||
|
except platform_collector.RPAError as e:
|
||||||
|
print(json.dumps({
|
||||||
|
"error": e.code,
|
||||||
|
"detail": str(e),
|
||||||
|
"processed": 0, "succeeded": 0, "failed": 0, "details": [],
|
||||||
|
}, ensure_ascii=False, indent=2))
|
||||||
|
return 2
|
||||||
|
except Exception as e:
|
||||||
|
print(json.dumps({"error": "exception", "detail": str(e)},
|
||||||
|
ensure_ascii=False, indent=2))
|
||||||
|
return 1
|
||||||
|
|
||||||
|
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_show_config(_args: argparse.Namespace) -> int:
|
||||||
|
"""查看 .env 非空配置(敏感字段自动脱敏)。"""
|
||||||
|
from service.config_viewer import read_env_config
|
||||||
|
|
||||||
|
print(json.dumps(read_env_config(), ensure_ascii=False, indent=2))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_logs_list(args: argparse.Namespace) -> int:
|
||||||
|
"""读取日志末尾 N 行(默认 200)。"""
|
||||||
|
from service.logs_viewer import tail_log_file
|
||||||
|
|
||||||
|
result = tail_log_file(tail=args.tail, min_level=args.level)
|
||||||
|
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
# ---------- argparse ----------
|
# ---------- argparse ----------
|
||||||
|
|
||||||
def _parse_thresholds(s: str) -> tuple[int, int, int]:
|
def _parse_thresholds(s: str) -> tuple[int, int, int]:
|
||||||
@@ -314,28 +415,30 @@ def build_parser() -> argparse.ArgumentParser:
|
|||||||
sp = sub.add_parser(
|
sp = sub.add_parser(
|
||||||
"generate-renewal-task", help="为指定合同生成续聘任务书",
|
"generate-renewal-task", help="为指定合同生成续聘任务书",
|
||||||
)
|
)
|
||||||
sp.add_argument("--contract-id", required=True, help="合同 ID")
|
# v1.1.30:去掉 required=True,给 contract_id 默认空串占位
|
||||||
|
# (jiangchang 客户端偶发 arguments 丢失时空参数不会让 argparse 直接报错退出)
|
||||||
|
sp.add_argument("--contract-id", default="", help="合同 ID(必填,由调用方传入)")
|
||||||
sp.add_argument("--task-id", default=None, help="可选:指定 task_id(默认自动生成)")
|
sp.add_argument("--task-id", default=None, help="可选:指定 task_id(默认自动生成)")
|
||||||
sp.set_defaults(handler=cmd_generate_task)
|
sp.set_defaults(handler=cmd_generate_task)
|
||||||
|
|
||||||
sp = sub.add_parser(
|
sp = sub.add_parser(
|
||||||
"dispatch-task-notice", help="按单位/月份派发续聘通知(dry-run)",
|
"dispatch-task-notice", help="按单位/月份派发续聘通知(dry-run)",
|
||||||
)
|
)
|
||||||
sp.add_argument("--unit-id", required=True, help="二级单位 ID")
|
sp.add_argument("--unit-id", default="", help="二级单位 ID(必填,由调用方传入)")
|
||||||
sp.add_argument("--month", default="", help="月份 YYYY-MM(留空 = 全部)")
|
sp.add_argument("--month", default="", help="月份 YYYY-MM(留空 = 全部)")
|
||||||
sp.set_defaults(handler=cmd_dispatch)
|
sp.set_defaults(handler=cmd_dispatch)
|
||||||
|
|
||||||
sp = sub.add_parser(
|
sp = sub.add_parser(
|
||||||
"mark-contract-renewed", help="标记合同已续签",
|
"mark-contract-renewed", help="标记合同已续签",
|
||||||
)
|
)
|
||||||
sp.add_argument("--contract-id", required=True, help="合同 ID")
|
sp.add_argument("--contract-id", default="", help="合同 ID(必填,由调用方传入)")
|
||||||
sp.add_argument("--material-id", required=True, help="续签材料 ID")
|
sp.add_argument("--material-id", default="", help="续签材料 ID(必填,由调用方传入)")
|
||||||
sp.set_defaults(handler=cmd_mark_renewed)
|
sp.set_defaults(handler=cmd_mark_renewed)
|
||||||
|
|
||||||
sp = sub.add_parser(
|
sp = sub.add_parser(
|
||||||
"render-html", help="渲染任务书 HTML",
|
"render-html", help="渲染任务书 HTML",
|
||||||
)
|
)
|
||||||
sp.add_argument("--task-id", required=True, help="任务 ID")
|
sp.add_argument("--task-id", default="", help="任务 ID(必填,由调用方传入)")
|
||||||
sp.set_defaults(handler=cmd_render_html)
|
sp.set_defaults(handler=cmd_render_html)
|
||||||
|
|
||||||
sp = sub.add_parser(
|
sp = sub.add_parser(
|
||||||
@@ -360,6 +463,41 @@ def build_parser() -> argparse.ArgumentParser:
|
|||||||
help="仅解析该合同(留空=处理所有 pending)")
|
help="仅解析该合同(留空=处理所有 pending)")
|
||||||
sp.set_defaults(handler=cmd_parse_materials)
|
sp.set_defaults(handler=cmd_parse_materials)
|
||||||
|
|
||||||
|
sp = sub.add_parser(
|
||||||
|
"download-materials",
|
||||||
|
help="处理材料下载队列(pending 任务 → 浏览器抓取 → 落库)",
|
||||||
|
)
|
||||||
|
sp.add_argument("--limit", type=int, default=0, help="本次最多处理任务数(0 = 用 --pick)")
|
||||||
|
sp.add_argument("--ids", default="", help="逗号分隔 task_id 列表(仅处理这些 pending 任务)")
|
||||||
|
sp.add_argument("--pick", type=int, default=5, help="无 --ids 时从 pending 领取条数(默认 5)")
|
||||||
|
sp.add_argument("--headless", action="store_true", help="无头模式(需已保存 profile)")
|
||||||
|
sp.add_argument("--show-browser", dest="headless", action="store_false",
|
||||||
|
help=argparse.SUPPRESS)
|
||||||
|
sp.add_argument("--human-wait-timeout", type=int, default=None,
|
||||||
|
help="手工登录 fallback 等待秒数(默认 180 = HUMAN_WAIT_TIMEOUT)")
|
||||||
|
sp.set_defaults(headless=False, handler=cmd_download_materials)
|
||||||
|
|
||||||
|
sp = sub.add_parser(
|
||||||
|
"import-input",
|
||||||
|
help="扫描 data/input/{contracts,materials}/ 自动入库",
|
||||||
|
)
|
||||||
|
sp.set_defaults(handler=cmd_import_input)
|
||||||
|
|
||||||
|
sp = sub.add_parser(
|
||||||
|
"show-config",
|
||||||
|
help="查看 .env 配置(敏感字段自动脱敏)",
|
||||||
|
)
|
||||||
|
sp.set_defaults(handler=cmd_show_config)
|
||||||
|
|
||||||
|
sp = sub.add_parser(
|
||||||
|
"logs-list",
|
||||||
|
help="查看运行日志末尾 N 行",
|
||||||
|
)
|
||||||
|
sp.add_argument("--tail", type=int, default=200, help="末尾行数(默认 200)")
|
||||||
|
sp.add_argument("--level", default="INFO",
|
||||||
|
help="最低级别 DEBUG/INFO/WARNING/ERROR(默认 INFO)")
|
||||||
|
sp.set_defaults(handler=cmd_logs_list)
|
||||||
|
|
||||||
return p
|
return p
|
||||||
|
|
||||||
|
|
||||||
@@ -370,7 +508,8 @@ def main(argv: Optional[List[str]] = None) -> int:
|
|||||||
print("Commands: health, version, config-path, init-db, "
|
print("Commands: health, version, config-path, init-db, "
|
||||||
"scan-expiring-contracts, generate-renewal-task, "
|
"scan-expiring-contracts, generate-renewal-task, "
|
||||||
"dispatch-task-notice, mark-contract-renewed, render-html, "
|
"dispatch-task-notice, mark-contract-renewed, render-html, "
|
||||||
"collect-from-platform, parse-materials")
|
"collect-from-platform, parse-materials, download-materials, "
|
||||||
|
"import-input, show-config, logs-list")
|
||||||
return 1
|
return 1
|
||||||
|
|
||||||
setup_skill_logging()
|
setup_skill_logging()
|
||||||
|
|||||||
@@ -90,6 +90,25 @@ DDL_STATEMENTS: tuple[str, ...] = (
|
|||||||
);
|
);
|
||||||
""",
|
""",
|
||||||
"CREATE INDEX IF NOT EXISTS idx_dispatch_unit_month ON dispatch_log(unit_id, month);",
|
"CREATE INDEX IF NOT EXISTS idx_dispatch_unit_month ON dispatch_log(unit_id, month);",
|
||||||
|
# 7. download_tasks (v1.1.18)
|
||||||
|
"""
|
||||||
|
CREATE TABLE IF NOT EXISTS download_tasks (
|
||||||
|
task_id TEXT PRIMARY KEY,
|
||||||
|
contract_id TEXT NOT NULL,
|
||||||
|
kind TEXT NOT NULL DEFAULT 'platform_button',
|
||||||
|
status TEXT NOT NULL DEFAULT 'pending',
|
||||||
|
attempts INTEGER NOT NULL DEFAULT 0,
|
||||||
|
error_msg TEXT,
|
||||||
|
enqueued_at TEXT NOT NULL,
|
||||||
|
started_at TEXT,
|
||||||
|
finished_at TEXT,
|
||||||
|
source_path TEXT,
|
||||||
|
result_path TEXT,
|
||||||
|
FOREIGN KEY (contract_id) REFERENCES contracts(contract_id)
|
||||||
|
);
|
||||||
|
""",
|
||||||
|
"CREATE INDEX IF NOT EXISTS idx_dlq_status ON download_tasks(status);",
|
||||||
|
"CREATE INDEX IF NOT EXISTS idx_dlq_contract ON download_tasks(contract_id);",
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -131,6 +150,7 @@ def init_db(db_path: Path | None = None) -> dict[str, int]:
|
|||||||
"renewal_tasks": conn.execute("SELECT COUNT(*) AS n FROM renewal_tasks").fetchone()["n"],
|
"renewal_tasks": conn.execute("SELECT COUNT(*) AS n FROM renewal_tasks").fetchone()["n"],
|
||||||
"dispatch_log": conn.execute("SELECT COUNT(*) AS n FROM dispatch_log").fetchone()["n"],
|
"dispatch_log": conn.execute("SELECT COUNT(*) AS n FROM dispatch_log").fetchone()["n"],
|
||||||
"materials_content": conn.execute("SELECT COUNT(*) AS n FROM materials_content").fetchone()["n"],
|
"materials_content": conn.execute("SELECT COUNT(*) AS n FROM materials_content").fetchone()["n"],
|
||||||
|
"download_tasks": conn.execute("SELECT COUNT(*) AS n FROM download_tasks").fetchone()["n"],
|
||||||
}
|
}
|
||||||
return counts
|
return counts
|
||||||
finally:
|
finally:
|
||||||
@@ -140,5 +160,5 @@ def init_db(db_path: Path | None = None) -> dict[str, int]:
|
|||||||
def table_names() -> Iterable[str]:
|
def table_names() -> Iterable[str]:
|
||||||
return (
|
return (
|
||||||
"contracts", "performance_records", "alerts", "renewal_tasks",
|
"contracts", "performance_records", "alerts", "renewal_tasks",
|
||||||
"dispatch_log", "materials_content",
|
"dispatch_log", "materials_content", "download_tasks",
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -294,6 +294,59 @@ def list_dispatch_logs(conn: sqlite3.Connection) -> list[sqlite3.Row]:
|
|||||||
|
|
||||||
# ---------- materials_content(v1.1.1 新增) ----------
|
# ---------- materials_content(v1.1.1 新增) ----------
|
||||||
|
|
||||||
|
def upsert_contract_from_csv_row(
|
||||||
|
conn: sqlite3.Connection, row: dict[str, str]
|
||||||
|
) -> None:
|
||||||
|
"""从 CSV 字典 upsert 一份合同。CSV 列名必须与 contracts 表列一致(或常见别名)。
|
||||||
|
|
||||||
|
缺字段兜底值:
|
||||||
|
employee_id = contract_id
|
||||||
|
teacher_name = ""
|
||||||
|
unit_id = "U-UNKNOWN"
|
||||||
|
unit_name = ""
|
||||||
|
position = ""
|
||||||
|
contract_type = ""
|
||||||
|
contract_start = "1970-01-01"
|
||||||
|
contract_end = "2099-12-31"
|
||||||
|
trial_period_end = contract_end
|
||||||
|
status = "可续聘"
|
||||||
|
"""
|
||||||
|
cid = row.get("contract_id") or row.get("合同ID") or ""
|
||||||
|
if not cid:
|
||||||
|
raise ValueError("CSV row missing contract_id / 合同ID")
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO contracts
|
||||||
|
(contract_id, employee_id, teacher_name, unit_id, unit_name,
|
||||||
|
position, contract_type, contract_start, contract_end,
|
||||||
|
trial_period_end, status, last_material_id, renewed_at)
|
||||||
|
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, NULL, NULL)
|
||||||
|
ON CONFLICT(contract_id) DO UPDATE SET
|
||||||
|
teacher_name = excluded.teacher_name,
|
||||||
|
unit_name = excluded.unit_name,
|
||||||
|
position = excluded.position,
|
||||||
|
contract_type = excluded.contract_type,
|
||||||
|
contract_start = excluded.contract_start,
|
||||||
|
contract_end = excluded.contract_end,
|
||||||
|
trial_period_end = excluded.trial_period_end,
|
||||||
|
status = excluded.status
|
||||||
|
""",
|
||||||
|
(
|
||||||
|
cid,
|
||||||
|
row.get("employee_id") or row.get("工号") or cid,
|
||||||
|
row.get("teacher_name") or row.get("教职工姓名") or "",
|
||||||
|
row.get("unit_id") or row.get("单位ID") or "U-UNKNOWN",
|
||||||
|
row.get("unit_name") or row.get("单位") or "",
|
||||||
|
row.get("position") or row.get("岗位") or "",
|
||||||
|
row.get("contract_type") or row.get("合同类型") or "",
|
||||||
|
row.get("contract_start") or row.get("合同起始日") or "1970-01-01",
|
||||||
|
row.get("contract_end") or row.get("合同到期日") or "2099-12-31",
|
||||||
|
row.get("trial_period_end") or row.get("试用期截止") or row.get("contract_end") or "2099-12-31",
|
||||||
|
row.get("status") or "可续聘",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def list_materials_for_contract(
|
def list_materials_for_contract(
|
||||||
conn: sqlite3.Connection, contract_id: str
|
conn: sqlite3.Connection, contract_id: str
|
||||||
) -> list[sqlite3.Row]:
|
) -> list[sqlite3.Row]:
|
||||||
@@ -373,3 +426,124 @@ def summarize_materials(
|
|||||||
if len(summary) > 16000:
|
if len(summary) > 16000:
|
||||||
summary = summary[:16000] + "\n...(材料过长已截断)"
|
summary = summary[:16000] + "\n...(材料过长已截断)"
|
||||||
return (summary, file_names)
|
return (summary, file_names)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- download_tasks(v1.1.18 新增) ----------
|
||||||
|
|
||||||
|
|
||||||
|
def enqueue_download_task(
|
||||||
|
conn: sqlite3.Connection,
|
||||||
|
contract_id: str,
|
||||||
|
kind: str = "platform_button",
|
||||||
|
) -> str:
|
||||||
|
"""入队一个下载任务,返回 task_id。
|
||||||
|
|
||||||
|
已存在 (contract_id, kind, status ∈ {pending, running}) 时跳过(返回已存在的 task_id)。
|
||||||
|
"""
|
||||||
|
import uuid
|
||||||
|
|
||||||
|
existing = conn.execute(
|
||||||
|
"""SELECT task_id FROM download_tasks
|
||||||
|
WHERE contract_id = ? AND kind = ? AND status IN ('pending', 'running')
|
||||||
|
LIMIT 1""",
|
||||||
|
(contract_id, kind),
|
||||||
|
).fetchone()
|
||||||
|
if existing:
|
||||||
|
return existing["task_id"]
|
||||||
|
task_id = f"DLQ-{uuid.uuid4().hex[:12]}"
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO download_tasks
|
||||||
|
(task_id, contract_id, kind, status, attempts, enqueued_at)
|
||||||
|
VALUES (?, ?, ?, 'pending', 0, ?)
|
||||||
|
""",
|
||||||
|
(task_id, contract_id, kind, datetime.now().isoformat(timespec="seconds")),
|
||||||
|
)
|
||||||
|
return task_id
|
||||||
|
|
||||||
|
|
||||||
|
def _read_max_attempts(conn: sqlite3.Connection) -> int:
|
||||||
|
"""读 DOWNLOAD_QUEUE_MAX_ATTEMPTS env(默认 3)。"""
|
||||||
|
import os
|
||||||
|
return int(os.environ.get("DOWNLOAD_QUEUE_MAX_ATTEMPTS", "3"))
|
||||||
|
|
||||||
|
|
||||||
|
def claim_pending_tasks(
|
||||||
|
conn: sqlite3.Connection, limit: int, ids_filter: list[str] | None = None,
|
||||||
|
) -> list[sqlite3.Row]:
|
||||||
|
"""原子领取一批 pending 任务,标记为 running。
|
||||||
|
|
||||||
|
- limit: 最多领取条数
|
||||||
|
- ids_filter: 若给出,仅从这些 task_id 中领
|
||||||
|
"""
|
||||||
|
max_attempts = _read_max_attempts(conn)
|
||||||
|
if ids_filter:
|
||||||
|
placeholders = ",".join("?" for _ in ids_filter)
|
||||||
|
rows = conn.execute(
|
||||||
|
f"""SELECT * FROM download_tasks
|
||||||
|
WHERE status = 'pending' AND attempts < ?
|
||||||
|
AND task_id IN ({placeholders})
|
||||||
|
ORDER BY enqueued_at ASC LIMIT ?""",
|
||||||
|
[max_attempts, *ids_filter, limit],
|
||||||
|
).fetchall()
|
||||||
|
else:
|
||||||
|
rows = conn.execute(
|
||||||
|
"""SELECT * FROM download_tasks
|
||||||
|
WHERE status = 'pending' AND attempts < ?
|
||||||
|
ORDER BY enqueued_at ASC LIMIT ?""",
|
||||||
|
(max_attempts, limit),
|
||||||
|
).fetchall()
|
||||||
|
if not rows:
|
||||||
|
return []
|
||||||
|
now = datetime.now().isoformat(timespec="seconds")
|
||||||
|
for r in rows:
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE download_tasks SET status = 'running', started_at = ? WHERE task_id = ?",
|
||||||
|
(now, r["task_id"]),
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def mark_task_success(
|
||||||
|
conn: sqlite3.Connection, task_id: str, result_path: str,
|
||||||
|
) -> None:
|
||||||
|
conn.execute(
|
||||||
|
"""UPDATE download_tasks
|
||||||
|
SET status = 'success', result_path = ?, finished_at = ?
|
||||||
|
WHERE task_id = ?""",
|
||||||
|
(result_path, datetime.now().isoformat(timespec="seconds"), task_id),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def mark_task_failed(
|
||||||
|
conn: sqlite3.Connection, task_id: str, error_msg: str,
|
||||||
|
) -> None:
|
||||||
|
"""失败一次:attempts+1,未超 max_attempts 则回退 pending,否则标 failed。"""
|
||||||
|
max_attempts = _read_max_attempts(conn)
|
||||||
|
now = datetime.now().isoformat(timespec="seconds")
|
||||||
|
conn.execute(
|
||||||
|
"""UPDATE download_tasks
|
||||||
|
SET status = CASE WHEN attempts + 1 >= ? THEN 'failed' ELSE 'pending' END,
|
||||||
|
attempts = attempts + 1,
|
||||||
|
error_msg = ?,
|
||||||
|
finished_at = ?
|
||||||
|
WHERE task_id = ?""",
|
||||||
|
(max_attempts, error_msg, now, task_id),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def list_recent_tasks(
|
||||||
|
conn: sqlite3.Connection, limit: int = 50
|
||||||
|
) -> list[sqlite3.Row]:
|
||||||
|
return conn.execute(
|
||||||
|
"SELECT * FROM download_tasks ORDER BY enqueued_at DESC LIMIT ?",
|
||||||
|
(limit,),
|
||||||
|
).fetchall()
|
||||||
|
|
||||||
|
|
||||||
|
def count_tasks_by_status(conn: sqlite3.Connection) -> dict[str, int]:
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT status, COUNT(*) AS n FROM download_tasks GROUP BY status"
|
||||||
|
).fetchall()
|
||||||
|
return {r["status"]: r["n"] for r in rows}
|
||||||
|
|||||||
77
scripts/service/config_viewer.py
Normal file
77
scripts/service/config_viewer.py
Normal file
@@ -0,0 +1,77 @@
|
|||||||
|
"""读取并脱敏 .env 配置供 UI 查看(v1.1.18)。
|
||||||
|
|
||||||
|
仅显示 .env 文件中的非空 key;命中 (TOKEN|SECRET|KEY|PASSWORD) 的字段自动脱敏为 ***。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from util.runtime_paths import get_env_path
|
||||||
|
|
||||||
|
|
||||||
|
# 匹配任一含 TOKEN / SECRET / KEY / PASSWORD 的 key 名(不区分大小写)
|
||||||
|
_SECRET_PATTERN = re.compile(r"(TOKEN|SECRET|KEY|PASSWORD)", re.IGNORECASE)
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_env_file(path: Path) -> dict[str, str]:
|
||||||
|
"""手动解析 .env(避免依赖 python-dotenv)。
|
||||||
|
|
||||||
|
- 跳过空行 / 注释(# 开头)
|
||||||
|
- 支持 KEY=VALUE 格式
|
||||||
|
- 自动去除包裹的 "..." 或 '...'
|
||||||
|
"""
|
||||||
|
if not path.exists():
|
||||||
|
return {}
|
||||||
|
result: dict[str, str] = {}
|
||||||
|
with open(path, "r", encoding="utf-8") as f:
|
||||||
|
for line in f:
|
||||||
|
line = line.strip()
|
||||||
|
if not line or line.startswith("#"):
|
||||||
|
continue
|
||||||
|
if "=" not in line:
|
||||||
|
continue
|
||||||
|
k, _, v = line.partition("=")
|
||||||
|
k = k.strip()
|
||||||
|
v = v.strip()
|
||||||
|
# 去掉行尾注释(不处理引号内的 #,简单实现)
|
||||||
|
if " #" in v:
|
||||||
|
v = v.split(" #", 1)[0].strip()
|
||||||
|
if v.startswith('"') and v.endswith('"') and len(v) >= 2:
|
||||||
|
v = v[1:-1]
|
||||||
|
elif v.startswith("'") and v.endswith("'") and len(v) >= 2:
|
||||||
|
v = v[1:-1]
|
||||||
|
result[k] = v
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _mask_secret(value: str) -> str:
|
||||||
|
if not value:
|
||||||
|
return ""
|
||||||
|
if len(value) <= 4:
|
||||||
|
return "***"
|
||||||
|
return value[:2] + "***" + value[-2:]
|
||||||
|
|
||||||
|
|
||||||
|
def read_env_config() -> dict:
|
||||||
|
"""读取 .env 中的非空 key,敏感 key 脱敏为 ***。
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
{
|
||||||
|
"env_path": str,
|
||||||
|
"exists": bool,
|
||||||
|
"config": {key: value_or_masked_value, ...}
|
||||||
|
}
|
||||||
|
"""
|
||||||
|
path = get_env_path()
|
||||||
|
parsed = _parse_env_file(path)
|
||||||
|
out: dict[str, str] = {}
|
||||||
|
for k, v in sorted(parsed.items()):
|
||||||
|
if not v:
|
||||||
|
continue
|
||||||
|
if _SECRET_PATTERN.search(k):
|
||||||
|
out[k] = _mask_secret(v)
|
||||||
|
else:
|
||||||
|
out[k] = v
|
||||||
|
return {"env_path": str(path), "exists": path.exists(), "config": out}
|
||||||
111
scripts/service/importer.py
Normal file
111
scripts/service/importer.py
Normal file
@@ -0,0 +1,111 @@
|
|||||||
|
"""主动扫描 data/input/{contracts,materials}/:CSV → contracts 表,PDF/Word → materials_content 表。
|
||||||
|
|
||||||
|
对 sandbox 用户尤其重要:HR API 拿不到时,把材料直接丢进 input/materials/ 即可被自动识别。
|
||||||
|
|
||||||
|
约定:
|
||||||
|
- input/contracts/*.csv
|
||||||
|
列名与 contracts 表一致(contract_id / teacher_name / unit_name / contract_end / ...)
|
||||||
|
也兼容中文别名(合同ID / 教职工姓名 / 单位 / 合同到期日 / 工号 / 岗位 / 合同类型)
|
||||||
|
- input/materials/{contract_id}__{filename}.{pdf|docx|doc}
|
||||||
|
文件名 __ 之前是 contract_id;无 __ 时回退 parent dir 名
|
||||||
|
拷贝到 data/materials/ + insert materials_content(parse_method='pending')
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import csv
|
||||||
|
import shutil
|
||||||
|
import sqlite3
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from db import repository as repo
|
||||||
|
from db.migrations import ensure_materials_content_schema
|
||||||
|
from util.logging_setup import get_skill_logger
|
||||||
|
from util.runtime_paths import (
|
||||||
|
get_data_dir,
|
||||||
|
get_input_contracts_dir,
|
||||||
|
get_input_materials_dir,
|
||||||
|
)
|
||||||
|
|
||||||
|
logger = get_skill_logger()
|
||||||
|
|
||||||
|
MATERIAL_EXTS = {".pdf", ".docx", ".doc"}
|
||||||
|
|
||||||
|
|
||||||
|
def _derive_contract_id_from_filename(mat_path: Path) -> str:
|
||||||
|
"""从文件名推导 contract_id。约定:{contract_id}__{filename}.{ext}。"""
|
||||||
|
parts = mat_path.stem.split("__", 1)
|
||||||
|
if len(parts) == 2 and parts[0]:
|
||||||
|
return parts[0]
|
||||||
|
# 兜底:parent dir 名
|
||||||
|
return mat_path.parent.name or "UNKNOWN"
|
||||||
|
|
||||||
|
|
||||||
|
def import_input_dir(conn: sqlite3.Connection) -> dict:
|
||||||
|
"""扫描 input 目录 → 入库。返回统计 dict。
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
{
|
||||||
|
"contracts_added": int,
|
||||||
|
"materials_added": int,
|
||||||
|
"files_scanned": int,
|
||||||
|
"errors": [str, ...],
|
||||||
|
}
|
||||||
|
"""
|
||||||
|
ensure_materials_content_schema(conn)
|
||||||
|
|
||||||
|
contracts_dir = get_input_contracts_dir()
|
||||||
|
materials_dir = get_input_materials_dir()
|
||||||
|
|
||||||
|
contracts_added = 0
|
||||||
|
materials_added = 0
|
||||||
|
files_scanned = 0
|
||||||
|
errors: list[str] = []
|
||||||
|
|
||||||
|
# 1. CSV → contracts(contract_id 重复 → upsert 覆盖)
|
||||||
|
for csv_path in sorted(contracts_dir.glob("*.csv")):
|
||||||
|
files_scanned += 1
|
||||||
|
try:
|
||||||
|
with open(csv_path, "r", encoding="utf-8", newline="") as f:
|
||||||
|
reader = csv.DictReader(f)
|
||||||
|
for row in reader:
|
||||||
|
if not row.get("contract_id") and not row.get("合同ID"):
|
||||||
|
errors.append(f"{csv_path.name}: row 缺 contract_id 跳过")
|
||||||
|
continue
|
||||||
|
repo.upsert_contract_from_csv_row(conn, row)
|
||||||
|
contracts_added += 1
|
||||||
|
except Exception as e:
|
||||||
|
errors.append(f"{csv_path.name}: {e}")
|
||||||
|
logger.warning("importer: CSV 解析失败 %s: %s", csv_path, e)
|
||||||
|
|
||||||
|
# 2. PDF/Word → materials_content(parse_method='pending'),文件拷贝到 data/materials/
|
||||||
|
materials_target_dir = get_data_dir() / "materials"
|
||||||
|
materials_target_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
for mat_path in sorted(materials_dir.iterdir()):
|
||||||
|
if not mat_path.is_file() or mat_path.suffix.lower() not in MATERIAL_EXTS:
|
||||||
|
continue
|
||||||
|
files_scanned += 1
|
||||||
|
try:
|
||||||
|
contract_id = _derive_contract_id_from_filename(mat_path)
|
||||||
|
target = materials_target_dir / mat_path.name
|
||||||
|
if not target.exists():
|
||||||
|
shutil.copy2(mat_path, target)
|
||||||
|
repo.insert_material(
|
||||||
|
conn, contract_id, mat_path.name, str(target),
|
||||||
|
None, "pending",
|
||||||
|
)
|
||||||
|
materials_added += 1
|
||||||
|
logger.info(
|
||||||
|
"importer: 入库 %s → contract_id=%s", mat_path.name, contract_id,
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
errors.append(f"{mat_path.name}: {e}")
|
||||||
|
logger.warning("importer: 材料导入失败 %s: %s", mat_path, e)
|
||||||
|
|
||||||
|
conn.commit()
|
||||||
|
return {
|
||||||
|
"contracts_added": contracts_added,
|
||||||
|
"materials_added": materials_added,
|
||||||
|
"files_scanned": files_scanned,
|
||||||
|
"errors": errors,
|
||||||
|
}
|
||||||
86
scripts/service/logs_viewer.py
Normal file
86
scripts/service/logs_viewer.py
Normal file
@@ -0,0 +1,86 @@
|
|||||||
|
"""读取日志文件末尾 N 行(v1.1.18)。
|
||||||
|
|
||||||
|
按 level 过滤:默认 INFO+。适用于 RotatingFileHandler 落地的日志文件。
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from util.runtime_paths import get_logs_dir
|
||||||
|
|
||||||
|
|
||||||
|
DEFAULT_TAIL = 200
|
||||||
|
_LEVEL_ORDER = {
|
||||||
|
"DEBUG": 10, "INFO": 20, "WARNING": 30, "ERROR": 40, "CRITICAL": 50,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_log_path() -> Path:
|
||||||
|
"""解析日志文件路径。
|
||||||
|
|
||||||
|
优先级:
|
||||||
|
1. LOG_FILE env(绝对路径直接用,相对路径走 get_logs_dir())
|
||||||
|
2. 默认 get_logs_dir() / LOG_FILE_DEFAULT_NAME
|
||||||
|
"""
|
||||||
|
raw = os.environ.get("LOG_FILE", "")
|
||||||
|
if raw:
|
||||||
|
p = Path(raw)
|
||||||
|
if not p.is_absolute():
|
||||||
|
p = get_logs_dir() / raw
|
||||||
|
return p
|
||||||
|
from util.constants import LOG_FILE_DEFAULT_NAME
|
||||||
|
return get_logs_dir() / LOG_FILE_DEFAULT_NAME
|
||||||
|
|
||||||
|
|
||||||
|
def tail_log_file(tail: int = DEFAULT_TAIL, min_level: str = "INFO") -> dict:
|
||||||
|
"""读日志文件末尾 tail 行,过滤 level >= min_level。
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
{
|
||||||
|
"log_file": str,
|
||||||
|
"exists": bool,
|
||||||
|
"total_lines": int,
|
||||||
|
"returned_lines": int,
|
||||||
|
"lines": [str, ...],
|
||||||
|
"min_level": str,
|
||||||
|
}
|
||||||
|
"""
|
||||||
|
log_path = _resolve_log_path()
|
||||||
|
if not log_path.exists():
|
||||||
|
return {
|
||||||
|
"log_file": str(log_path),
|
||||||
|
"exists": False,
|
||||||
|
"total_lines": 0,
|
||||||
|
"returned_lines": 0,
|
||||||
|
"lines": [],
|
||||||
|
"min_level": min_level,
|
||||||
|
}
|
||||||
|
|
||||||
|
text = log_path.read_text(encoding="utf-8", errors="replace")
|
||||||
|
all_lines = text.splitlines()
|
||||||
|
last_n = all_lines[-tail:] if tail > 0 else all_lines
|
||||||
|
threshold = _LEVEL_ORDER.get(min_level.upper(), 20)
|
||||||
|
|
||||||
|
filtered: list[str] = []
|
||||||
|
for line in last_n:
|
||||||
|
# 格式: "2026-09-15 12:34:56 [alert-contract-expiring-hr] INFO message"
|
||||||
|
# 取第 4 个 token (INFO/DEBUG/...)
|
||||||
|
parts = line.split()
|
||||||
|
if len(parts) >= 4 and parts[1].count(":") >= 2 and parts[3] in _LEVEL_ORDER:
|
||||||
|
line_level = parts[3]
|
||||||
|
if _LEVEL_ORDER.get(line_level, 0) >= threshold:
|
||||||
|
filtered.append(line)
|
||||||
|
else:
|
||||||
|
# 非标准行 / 解析不到 level:保留
|
||||||
|
filtered.append(line)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"log_file": str(log_path),
|
||||||
|
"exists": True,
|
||||||
|
"total_lines": len(all_lines),
|
||||||
|
"returned_lines": len(filtered),
|
||||||
|
"lines": filtered,
|
||||||
|
"min_level": min_level,
|
||||||
|
}
|
||||||
@@ -36,6 +36,7 @@ from util.constants import (
|
|||||||
FULL_RENDER_WAIT_MS,
|
FULL_RENDER_WAIT_MS,
|
||||||
GOTO_TIMEOUT_MS,
|
GOTO_TIMEOUT_MS,
|
||||||
HUMAN_POLL_INTERVAL,
|
HUMAN_POLL_INTERVAL,
|
||||||
|
HUMAN_PROMPT_INTERVAL,
|
||||||
HUMAN_WAIT_TIMEOUT,
|
HUMAN_WAIT_TIMEOUT,
|
||||||
POST_LOGIN_SETTLE_MS,
|
POST_LOGIN_SETTLE_MS,
|
||||||
POST_RELOAD_LOGIN_HYDRATION_MS,
|
POST_RELOAD_LOGIN_HYDRATION_MS,
|
||||||
@@ -55,6 +56,8 @@ from util.constants import (
|
|||||||
TABLE_POLL_INTERVAL_MS,
|
TABLE_POLL_INTERVAL_MS,
|
||||||
TABLE_POLL_MAX_ATTEMPTS,
|
TABLE_POLL_MAX_ATTEMPTS,
|
||||||
TABLE_TR_TIMEOUT_MS,
|
TABLE_TR_TIMEOUT_MS,
|
||||||
|
URL_STABLE_POLL_S,
|
||||||
|
URL_STABLE_REQUIRED_S,
|
||||||
URL_AFTER_LOGIN_TIMEOUT_MS,
|
URL_AFTER_LOGIN_TIMEOUT_MS,
|
||||||
)
|
)
|
||||||
from util.logging_setup import get_skill_logger
|
from util.logging_setup import get_skill_logger
|
||||||
@@ -124,6 +127,64 @@ def _safe_getenv_dir(env_key: str, default_rel: str) -> Path:
|
|||||||
return (get_data_dir() / default_rel).resolve()
|
return (get_data_dir() / default_rel).resolve()
|
||||||
|
|
||||||
|
|
||||||
|
def _get_env(env_key: str, default: str = "") -> str:
|
||||||
|
"""v1.1.31/32:读 env 值,进程 env 优先,缺则从 .env 文件读(手动解析)。
|
||||||
|
|
||||||
|
v1.1.32 关键修复:之前只读 os.environ,但 .env 是用户手动维护的文件,
|
||||||
|
python 进程默认不会自动加载它,导致 SMARTCAMPUS_LOGIN_USER/PASSWORD 永远读不到
|
||||||
|
(jiangchang 宿主未必会注入 .env)。手动解析 .env 兜底。
|
||||||
|
"""
|
||||||
|
# 1. 进程 env 优先(jiangchang 注入或系统 env 优先)
|
||||||
|
val = os.environ.get(env_key)
|
||||||
|
if val is not None and val != "":
|
||||||
|
return val
|
||||||
|
|
||||||
|
# 2. 从 .env 文件手动解析(一次性缓存)
|
||||||
|
global _ENV_CACHE_LOADED, _ENV_CACHE
|
||||||
|
if not _ENV_CACHE_LOADED:
|
||||||
|
_ENV_CACHE_LOADED = True
|
||||||
|
_ENV_CACHE = _load_env_file()
|
||||||
|
return _ENV_CACHE.get(env_key, default)
|
||||||
|
|
||||||
|
|
||||||
|
_ENV_CACHE: dict[str, str] = {}
|
||||||
|
_ENV_CACHE_LOADED = False
|
||||||
|
|
||||||
|
|
||||||
|
def _load_env_file() -> dict[str, str]:
|
||||||
|
"""手动解析 .env 文件(不依赖 python-dotenv)。
|
||||||
|
|
||||||
|
规则:
|
||||||
|
- 跳过注释(# 开头)和空行
|
||||||
|
- 形如 `KEY=value`,value 可带 "..." 或 '...' 引号(自动去引号)
|
||||||
|
- value 含 `=` 时只 split 一次
|
||||||
|
- 找不到 .env 返回空 dict
|
||||||
|
"""
|
||||||
|
from util.runtime_paths import get_env_path
|
||||||
|
|
||||||
|
env_path = get_env_path()
|
||||||
|
if not env_path.exists():
|
||||||
|
return {}
|
||||||
|
out: dict[str, str] = {}
|
||||||
|
try:
|
||||||
|
for raw in env_path.read_text(encoding="utf-8").splitlines():
|
||||||
|
line = raw.strip()
|
||||||
|
if not line or line.startswith("#"):
|
||||||
|
continue
|
||||||
|
if "=" not in line:
|
||||||
|
continue
|
||||||
|
k, v = line.split("=", 1)
|
||||||
|
k = k.strip()
|
||||||
|
v = v.strip()
|
||||||
|
# 去引号
|
||||||
|
if len(v) >= 2 and v[0] == v[-1] and v[0] in ("'", '"'):
|
||||||
|
v = v[1:-1]
|
||||||
|
out[k] = v
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning("platform_collector: 解析 .env 失败(继续走空 dict): %s", e)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
def _goto(page, url: str, timeout: int = GOTO_TIMEOUT_MS) -> None:
|
def _goto(page, url: str, timeout: int = GOTO_TIMEOUT_MS) -> None:
|
||||||
"""带 networkidle / domcontentloaded 双兜底的 goto。"""
|
"""带 networkidle / domcontentloaded 双兜底的 goto。"""
|
||||||
try:
|
try:
|
||||||
@@ -164,753 +225,53 @@ def _is_logged_in(page) -> bool:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def _wait_for_manual_login(page, wait_sec: int = HUMAN_WAIT_TIMEOUT) -> bool:
|
def _is_spa_routed(page) -> bool:
|
||||||
"""手工登录 fallback:轮询 _is_logged_in,每 HUMAN_POLL_INTERVAL 秒一次。
|
"""v1.1.22:检测 sandbox SPA 是否已路由到 SPA 页面(含 /select /app/)。"""
|
||||||
|
|
||||||
与 skill-template task_rpa.py::_ensure_logged_in 设计对齐:
|
|
||||||
- 不自动破解,等用户在浏览器手工完成登录
|
|
||||||
- 屏幕打印 [登录] 提示
|
|
||||||
- 超时抛 LoginTimeoutError(ERROR:LOGIN_TIMEOUT)
|
|
||||||
|
|
||||||
自动 fill demo/123456 失败时调用此函数。
|
|
||||||
"""
|
|
||||||
print(
|
|
||||||
f"[登录] 自动登录失败,请在浏览器中完成登录"
|
|
||||||
f"(sandbox demo/123456 或真实账号),最多等待 {wait_sec} 秒..."
|
|
||||||
)
|
|
||||||
deadline = time.monotonic() + wait_sec
|
|
||||||
attempt = 0
|
|
||||||
while time.monotonic() < deadline:
|
|
||||||
attempt += 1
|
|
||||||
if _is_logged_in(page):
|
|
||||||
print(f"[登录] 检测到登录成功(attempt {attempt})")
|
|
||||||
return True
|
|
||||||
time.sleep(HUMAN_POLL_INTERVAL)
|
|
||||||
raise LoginTimeoutError(
|
|
||||||
f"自动登录失败,{wait_sec}s 内用户在浏览器中也未完成登录"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _fill_root_login(page) -> bool:
|
|
||||||
"""根域登录页(如还在):fill demo/123456 → click Sign in。返回是否执行了登录。"""
|
|
||||||
user_input = page.locator('#jc-user').first
|
|
||||||
if user_input.count() == 0:
|
|
||||||
return False
|
|
||||||
user_input.fill("demo")
|
|
||||||
page.locator('#jc-pass').first.fill("123456")
|
|
||||||
page.locator('button[type="submit"]:has-text("Sign in")').first.click()
|
|
||||||
logger.info("platform_collector: 根域自动登录 demo/123456...")
|
|
||||||
try:
|
|
||||||
page.wait_for_url(
|
|
||||||
lambda u: "/home" in u or "/select" in u or "/industries/" in u,
|
|
||||||
timeout=URL_AFTER_LOGIN_TIMEOUT_MS,
|
|
||||||
)
|
|
||||||
except Exception as e:
|
|
||||||
logger.warning("等根域登录跳转超时: %s", e)
|
|
||||||
logger.info("platform_collector: 根域登录后 URL=%s", page.url)
|
|
||||||
return True
|
|
||||||
|
|
||||||
|
|
||||||
def _click_campus_link(page) -> bool:
|
|
||||||
"""在 /select 页面点智慧校园 tile,返回是否点中。"""
|
|
||||||
link = page.locator("a[href*='/industries/campus']").first
|
|
||||||
if link.count() == 0:
|
|
||||||
logger.warning("platform_collector: 未找到 智慧校园 link")
|
|
||||||
return False
|
|
||||||
link.click()
|
|
||||||
logger.info("platform_collector: click 智慧校园 链接")
|
|
||||||
try:
|
|
||||||
page.wait_for_url(lambda u: "/industries/campus" in u, timeout=SIDEBAR_LINK_TIMEOUT_MS // 2)
|
|
||||||
except Exception as e:
|
|
||||||
logger.warning("等 sandbox 路径超时: %s;当前 URL=%s", e, page.url)
|
|
||||||
return True
|
|
||||||
|
|
||||||
|
|
||||||
def _fill_spa_login(page, settle_ms: int = SPA_LOGIN_SETTLE_MS) -> bool:
|
|
||||||
"""sandbox SPA 自动登录(如还在登录页):fill demo/123456 → click 登录 → 等跳转。
|
|
||||||
|
|
||||||
返回是否执行了登录(True = 已点登录按钮且 URL 已跳离 login;调用方可继续后续流程)。
|
|
||||||
失败时**不抛异常**,由调用方决定是否 fallback 到 _wait_for_manual_login。
|
|
||||||
|
|
||||||
v1.1.10 关键修复:**先检测 _is_logged_in,已登录态直接 return True 不再 fill**。
|
|
||||||
之前没这个检查,profile 复用时会被强制 demo fill 破坏 session。
|
|
||||||
|
|
||||||
v1.1.11 关键修复:**sandbox SPA 登录 input 实际是 `type="tel"` + 无 autocomplete**,
|
|
||||||
之前 selector `input[autocomplete="username"]` 永远找不到,自动 fill 100% 失败。
|
|
||||||
改用 sandbox 实际 selector `#jc-user` / `#jc-pass`(来自 root 登录页)+ `button[type="submit"]`。
|
|
||||||
|
|
||||||
修复要点(v1.1.5/v1.1.6 登录时间太短导致采集不到数据):
|
|
||||||
- 默认 settle_ms = SPA_LOGIN_SETTLE_MS(4000ms),给 SPA 异步渲染 + token 写入 cookie 时间
|
|
||||||
- 等 username input 的 timeout = SPA_LOGIN_INPUT_TIMEOUT_MS(20s)
|
|
||||||
- click 按钮后 wait_for_url timeout = SPA_LOGIN_URL_TIMEOUT_MS(30s)
|
|
||||||
- 多处过渡路径(/login, /signin, /auth/)一并判断
|
|
||||||
"""
|
|
||||||
# v1.1.10:先检查是否已登录(profile cookie 复用场景),已登录直接 return True
|
|
||||||
if _is_logged_in(page):
|
|
||||||
logger.info(
|
|
||||||
"platform_collector: 已登录态(profile cookie 复用),跳过自动 fill"
|
|
||||||
)
|
|
||||||
return True
|
|
||||||
|
|
||||||
url_lower = page.url.lower()
|
url_lower = page.url.lower()
|
||||||
if "/login" not in url_lower and "/signin" not in url_lower and "/auth/" not in url_lower:
|
return (
|
||||||
return False
|
("/app/" in url_lower or "/select" in url_lower)
|
||||||
# v1.1.11:sandbox SPA 登录页 input 是 #jc-user / #jc-pass(与 root 登录页同 ID)
|
and "/login" not in url_lower
|
||||||
# 等输入框出现(带超时,避免 sandbox mock 慢启动卡死)
|
and "/signin" not in url_lower
|
||||||
try:
|
|
||||||
page.wait_for_selector(
|
|
||||||
'#jc-user, input[type="tel"], input[autocomplete="username"]',
|
|
||||||
timeout=SPA_LOGIN_INPUT_TIMEOUT_MS,
|
|
||||||
)
|
|
||||||
except Exception as e:
|
|
||||||
logger.warning("SPA login input 未出现: %s;当前 URL=%s", e, page.url)
|
|
||||||
return False
|
|
||||||
# v1.1.11:按优先级选 selector —— sandbox 实际用 #jc-user
|
|
||||||
user_sel = None
|
|
||||||
for sel in ('#jc-user', 'input[type="tel"]', 'input[autocomplete="username"]'):
|
|
||||||
loc = page.locator(sel).first
|
|
||||||
if loc.count() > 0:
|
|
||||||
user_sel = sel
|
|
||||||
break
|
|
||||||
if not user_sel:
|
|
||||||
logger.warning("SPA login input selector 全 miss(#jc-user / type=tel / autocomplete=username)")
|
|
||||||
return False
|
|
||||||
page.locator(user_sel).first.fill("demo")
|
|
||||||
# 密码 selector 同样按优先级
|
|
||||||
pass_sel = None
|
|
||||||
for sel in ('#jc-pass', 'input[type="password"]', 'input[autocomplete="current-password"]'):
|
|
||||||
loc = page.locator(sel).first
|
|
||||||
if loc.count() > 0:
|
|
||||||
pass_sel = sel
|
|
||||||
break
|
|
||||||
if not pass_sel:
|
|
||||||
logger.warning("SPA password input selector 全 miss")
|
|
||||||
return False
|
|
||||||
page.locator(pass_sel).first.fill("123456")
|
|
||||||
page.locator(
|
|
||||||
'button:has-text("登录"), button[type="submit"]:has-text("Sign in"), '
|
|
||||||
'button[type="submit"]'
|
|
||||||
).first.click()
|
|
||||||
logger.info("platform_collector: sandbox SPA 自动登录 (user_sel=%s pass_sel=%s)", user_sel, pass_sel)
|
|
||||||
page.wait_for_timeout(settle_ms)
|
|
||||||
try:
|
|
||||||
page.wait_for_url(
|
|
||||||
lambda u: "/login" not in u.lower()
|
|
||||||
and "/signin" not in u.lower()
|
|
||||||
and "/auth/" not in u.lower(),
|
|
||||||
timeout=SPA_LOGIN_URL_TIMEOUT_MS,
|
|
||||||
)
|
|
||||||
except Exception as e:
|
|
||||||
logger.warning("SPA 登录超时: %s;当前 URL=%s", e, page.url)
|
|
||||||
return False
|
|
||||||
# v1.1.10:fill 后再检测 _is_logged_in,避免 sandbox demo 凭据失效时
|
|
||||||
# 被误判为「已登录」(URL 跳 /app/home 但 cookie 没写入)
|
|
||||||
if not _is_logged_in(page):
|
|
||||||
logger.warning(
|
|
||||||
"platform_collector: SPA URL 跳离 login 但 _is_logged_in=False,登录可能失败"
|
|
||||||
)
|
|
||||||
return False
|
|
||||||
logger.info("platform_collector: SPA 登录后 URL=%s", page.url)
|
|
||||||
return True
|
|
||||||
|
|
||||||
|
|
||||||
def collect_all(
|
|
||||||
limit: int = 0,
|
|
||||||
headless: bool = True,
|
|
||||||
human_wait_timeout: int = HUMAN_WAIT_TIMEOUT,
|
|
||||||
) -> dict:
|
|
||||||
"""登录 sandbox 平台 → 抓合同表 + 材料 → 落库。
|
|
||||||
|
|
||||||
human_wait_timeout: 手工登录 fallback 等待秒数(默认 HUMAN_WAIT_TIMEOUT=180)。
|
|
||||||
失败时可能抛出 RPAError(LoginTimeoutError / MissingBrowserError / TableNotFoundError)。
|
|
||||||
"""
|
|
||||||
try:
|
|
||||||
from playwright.sync_api import sync_playwright
|
|
||||||
except ImportError:
|
|
||||||
logger.error("playwright 未安装,跳过 RPA 采集")
|
|
||||||
return _error_result("playwright_not_installed", 0)
|
|
||||||
|
|
||||||
profile_dir = _safe_getenv_dir("SMARTCAMPUS_PROFILE_DIR", "playwright-profile")
|
|
||||||
download_dir = _safe_getenv_dir("MATERIALS_DOWNLOAD_DIR", "materials")
|
|
||||||
profile_dir.mkdir(parents=True, exist_ok=True)
|
|
||||||
download_dir.mkdir(parents=True, exist_ok=True)
|
|
||||||
|
|
||||||
logger.info(
|
|
||||||
"platform_collector: 启动 profile=%s download_dir=%s headless=%s limit=%d "
|
|
||||||
"human_wait_timeout=%d",
|
|
||||||
profile_dir, download_dir, headless, limit, human_wait_timeout,
|
|
||||||
)
|
)
|
||||||
|
|
||||||
contracts: list[dict] = []
|
|
||||||
fetched_rows = 0
|
|
||||||
downloaded = 0
|
|
||||||
|
|
||||||
with sync_playwright() as p:
|
# ---------- v1.1.31:从 service.platform_collector_login re-export 登录流程函数 ----------
|
||||||
# v1.1.9:headed 模式优先用系统 Edge(跟 Windows 桌面 session 同源,
|
# facade 仅做 re-export,不持有实现;测试 / CLI 通过 service.platform_collector.X 仍可访问。
|
||||||
# 能真正弹窗让你看到并手工登录);找不到则 fallback Playwright 内置 chromium
|
from service.platform_collector_login import ( # noqa: E402,F401
|
||||||
if not headless:
|
_try_auto_sign_in as _try_auto_sign_in,
|
||||||
try:
|
_wait_for_login_inputs_filled as _wait_for_login_inputs_filled,
|
||||||
ctx = p.chromium.launch_persistent_context(
|
_is_login_page_url_for_user as _is_login_page_url_for_user,
|
||||||
user_data_dir=str(profile_dir),
|
_wait_for_manual_login as _wait_for_manual_login,
|
||||||
headless=False,
|
_wait_for_url_stable as _wait_for_url_stable,
|
||||||
accept_downloads=True,
|
_fill_root_login as _fill_root_login,
|
||||||
channel="msedge",
|
_fill_spa_login_from_env as _fill_spa_login_from_env,
|
||||||
viewport={"width": 1440, "height": 900},
|
_is_dev_mode_panel_open as _is_dev_mode_panel_open,
|
||||||
)
|
_resolve_target_industry as _resolve_target_industry,
|
||||||
logger.info(
|
_click_dev_mode_sign_in_for_industry as _click_dev_mode_sign_in_for_industry,
|
||||||
"platform_collector: 使用系统 Edge(headed 弹窗)"
|
_click_campus_link as _click_campus_link,
|
||||||
)
|
_fill_spa_login as _fill_spa_login,
|
||||||
except Exception as e:
|
)
|
||||||
logger.warning(
|
|
||||||
"platform_collector: 系统 Edge 启动失败(%s),fallback Playwright chromium",
|
|
||||||
e,
|
|
||||||
)
|
|
||||||
ctx = p.chromium.launch_persistent_context(
|
|
||||||
user_data_dir=str(profile_dir),
|
|
||||||
headless=False,
|
|
||||||
accept_downloads=True,
|
|
||||||
viewport={"width": 1440, "height": 900},
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
ctx = p.chromium.launch_persistent_context(
|
|
||||||
user_data_dir=str(profile_dir),
|
|
||||||
headless=True,
|
|
||||||
accept_downloads=True,
|
|
||||||
viewport={"width": 1440, "height": 900},
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
page = ctx.new_page()
|
|
||||||
# v1.1.6:覆盖 Playwright 默认操作超时(之前无 set_default_timeout,导致
|
|
||||||
# 所有 wait_for_selector 默认 30s,调试时不直观)
|
|
||||||
page.set_default_timeout(DEFAULT_PAGE_TIMEOUT_MS)
|
|
||||||
|
|
||||||
# 1. v1.1.11:先用**最小侵入**方式检测已登录态 —— 直接 goto SPA_HOME_URL
|
# ---------- v1.1.31:从 service.platform_collector_collect re-export 采集主流程 ----------
|
||||||
# (不要先跳 ROOT_URL + 自动 fill,会破坏已登录态)
|
from service.platform_collector_collect import ( # noqa: E402,F401
|
||||||
# v1.1.11 改进:探针结果 + 二次等待 + cookie 检测,三重判断避免误判
|
collect_all as collect_all,
|
||||||
logger.info(
|
_error_result as _error_result,
|
||||||
"platform_collector: 检测登录态,goto SPA_HOME 先探 (当前 URL=%s)",
|
)
|
||||||
page.url,
|
|
||||||
)
|
|
||||||
_goto(page, SPA_HOME_URL)
|
|
||||||
# v1.1.11:等 sandbox SPA 异步验证 cookie(POST_LOGIN_SETTLE_MS → SPA_HYDRATION_MS)
|
|
||||||
page.wait_for_timeout(SPA_HYDRATION_MS)
|
|
||||||
# v1.1.11:二次等待 —— sandbox SPA 有时首次访问会先跳一个中间过渡页
|
|
||||||
# (如 /select),等几秒看 URL 是否稳定到 /app/
|
|
||||||
first_url = page.url
|
|
||||||
page.wait_for_timeout(SPA_HOME_RETRY_DELAY_MS)
|
|
||||||
second_url = page.url
|
|
||||||
if first_url != second_url:
|
|
||||||
logger.info(
|
|
||||||
"platform_collector: 探针 URL 过渡 %s → %s,等稳定", first_url, second_url,
|
|
||||||
)
|
|
||||||
page.wait_for_timeout(SPA_HOME_RETRY_DELAY_MS)
|
|
||||||
|
|
||||||
if _is_logged_in(page):
|
# ---------- v1.1.31:从 service.platform_collector_persist re-export DB 持久化 ----------
|
||||||
logger.info(
|
from service.platform_collector_persist import ( # noqa: E402,F401
|
||||||
"platform_collector: ✅ 已登录态,**完全跳过登录流程**(不弹窗、不自动 fill、不打扰)"
|
_persist_contracts as _persist_contracts,
|
||||||
",URL=%s", page.url,
|
_enqueue_download_tasks_safe as _enqueue_download_tasks_safe,
|
||||||
)
|
)
|
||||||
else:
|
|
||||||
logger.info(
|
|
||||||
"platform_collector: 未登录态(SPA_HOME 跳回 login, URL=%s),走登录分支",
|
|
||||||
page.url,
|
|
||||||
)
|
|
||||||
# 2. 未登录:直接 goto sandbox SPA login(绕过 /select 和 root 跳转)
|
|
||||||
_goto(page, SPA_LOGIN_URL)
|
|
||||||
# SPA hydration + framework 初始化时间(修复登录时间太短的核心点)
|
|
||||||
page.wait_for_timeout(SPA_HYDRATION_MS)
|
|
||||||
|
|
||||||
# 3. sandbox SPA 自动登录(demo/123456 已知可用)
|
|
||||||
# 失败 fallback 手工登录等待(轮询 _is_logged_in + 180s 兜底)
|
|
||||||
if not _fill_spa_login(page):
|
|
||||||
logger.warning(
|
|
||||||
"platform_collector: 自动 fill 失败(URL=%s),fallback 手工登录等待",
|
|
||||||
page.url,
|
|
||||||
)
|
|
||||||
if not headless:
|
|
||||||
_wait_for_manual_login(page, wait_sec=human_wait_timeout)
|
|
||||||
else:
|
|
||||||
# headless 模式无法手工登录:抛 LoginTimeoutError 让 CLI 决定
|
|
||||||
raise LoginTimeoutError(
|
|
||||||
f"headless 模式下自动 fill 失败,且无法手工登录"
|
|
||||||
f"(请用 --show-browser 重试)"
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
# 等 SPA 路由稳定(之前这里没有 wait,直接 goto home 会打断 pending 跳转)
|
|
||||||
page.wait_for_timeout(POST_LOGIN_SETTLE_MS)
|
|
||||||
|
|
||||||
# 4. 强制 goto SPA home(如果还停在 login,重试登录一次)
|
|
||||||
for retry in range(SPA_HOME_RETRY_MAX):
|
|
||||||
if _is_logged_in(page):
|
|
||||||
break
|
|
||||||
logger.info(
|
|
||||||
"platform_collector: 强制 goto SPA_HOME (retry=%d), 当前 URL=%s",
|
|
||||||
retry, page.url,
|
|
||||||
)
|
|
||||||
_goto(page, SPA_HOME_URL)
|
|
||||||
page.wait_for_timeout(SPA_HOME_RETRY_DELAY_MS)
|
|
||||||
if _fill_spa_login(page):
|
|
||||||
page.wait_for_timeout(POST_LOGIN_SETTLE_MS)
|
|
||||||
|
|
||||||
# 4.5. retry 后若仍未登录,headless 模式抛超时
|
|
||||||
if not _is_logged_in(page):
|
|
||||||
if headless:
|
|
||||||
raise LoginTimeoutError(
|
|
||||||
"SPA_HOME retry 全部失败,URL 仍未离开 login 页"
|
|
||||||
)
|
|
||||||
_wait_for_manual_login(page, wait_sec=human_wait_timeout)
|
|
||||||
|
|
||||||
# 5. 等 SPA 渲染 + click 侧边栏 hr/contracts(SPA 路由,避免 goto 触发整页 reload 丢 session)
|
|
||||||
try:
|
|
||||||
# dump SPA home 一次(用于调试 selector)
|
|
||||||
page.wait_for_timeout(POST_LOGIN_SETTLE_MS)
|
|
||||||
_dump_page(page, download_dir.parent / "spa_home_dump.html", "SPA home")
|
|
||||||
|
|
||||||
# 等 SPA 真正加载完(侧边栏 link 出现,最长 SIDEBAR_LINK_TIMEOUT_MS)
|
|
||||||
# 修复:之前 15s 太短,sandbox mock 加载慢时直接 fallback 到 pushState → 表格加载不出来
|
|
||||||
try:
|
|
||||||
page.wait_for_selector(
|
|
||||||
"a[href$='/hr/contracts']",
|
|
||||||
timeout=SIDEBAR_LINK_TIMEOUT_MS,
|
|
||||||
state="attached",
|
|
||||||
)
|
|
||||||
except Exception as e:
|
|
||||||
logger.warning("等侧边栏 hr/contracts link 超时: %s", e)
|
|
||||||
|
|
||||||
link = page.locator("a[href$='/hr/contracts']").first
|
|
||||||
if link.count() > 0:
|
|
||||||
link.click()
|
|
||||||
logger.info("platform_collector: click 侧边栏 hr/contracts")
|
|
||||||
try:
|
|
||||||
page.wait_for_url(
|
|
||||||
lambda u: "/hr/contracts" in u,
|
|
||||||
timeout=SIDEBAR_CLICK_URL_TIMEOUT_MS,
|
|
||||||
)
|
|
||||||
except Exception as e:
|
|
||||||
logger.warning("等 /hr/contracts URL 超时: %s;当前 URL=%s",
|
|
||||||
e, page.url)
|
|
||||||
else:
|
|
||||||
logger.info(
|
|
||||||
"platform_collector: 侧边栏 link 仍未出现,"
|
|
||||||
"使用 SPA 路由(pushState)到 hr/contracts, 当前 URL=%s",
|
|
||||||
page.url,
|
|
||||||
)
|
|
||||||
# 用 SPA 内部路由而非整页 goto,避免 session 被 server 端踢回 login
|
|
||||||
try:
|
|
||||||
page.evaluate(
|
|
||||||
"u => history.pushState({}, '', u) || "
|
|
||||||
"window.dispatchEvent(new PopStateEvent('popstate')),"
|
|
||||||
" '/industries/campus/smartcampus/app/hr/contracts'"
|
|
||||||
)
|
|
||||||
page.wait_for_timeout(PUSHSTATE_SETTLE_MS)
|
|
||||||
except Exception as e:
|
|
||||||
logger.warning("SPA pushState 失败: %s", e)
|
|
||||||
_goto(page, PLATFORM_URL)
|
|
||||||
except Exception as e:
|
|
||||||
logger.warning("click 侧边栏失败: %s;fallback goto", e)
|
|
||||||
_goto(page, PLATFORM_URL)
|
|
||||||
|
|
||||||
logger.info("platform_collector: 当前 URL=%s", page.url)
|
|
||||||
|
|
||||||
# 5.5. v1.1.8:**首次登录后不再强制刷新**,避免打断用户手工登录态。
|
|
||||||
# 改为「等表格出现」兜底:给 SPA 一个初始等待窗口(TABLE_INITIAL_WAIT_MS*2),
|
|
||||||
# 如果表格仍未出现,**才**刷新一次(sandbox mock 异步加载场景)。
|
|
||||||
logger.info(
|
|
||||||
"platform_collector: 进入 hr/contracts 后**不刷新**,等待表格出现"
|
|
||||||
"(首次登录态不要打断)"
|
|
||||||
)
|
|
||||||
|
|
||||||
# 5.6. 兜底刷新:仅当等表格超时后才触发;刷新后可能被踢回 login,需重新登录
|
|
||||||
# 修复:之前没检查 URL,刷新后停在 login 页继续往后走,找不到表格就 dump 报错
|
|
||||||
def _try_reload_after_table_timeout():
|
|
||||||
logger.warning(
|
|
||||||
"platform_collector: 表格未在 %dms 内出现,触发兜底刷新",
|
|
||||||
TABLE_INITIAL_WAIT_MS * 2,
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
page.reload(
|
|
||||||
wait_until="networkidle",
|
|
||||||
timeout=RELOAD_NETWORKIDLE_TIMEOUT_MS,
|
|
||||||
)
|
|
||||||
except Exception as e:
|
|
||||||
logger.warning("刷新 networkidle 失败: %s;尝试 domcontentloaded", e)
|
|
||||||
try:
|
|
||||||
page.reload(
|
|
||||||
wait_until="domcontentloaded",
|
|
||||||
timeout=RELOAD_DOMCONTENTLOADED_TIMEOUT_MS,
|
|
||||||
)
|
|
||||||
except Exception as e2:
|
|
||||||
logger.warning("刷新 domcontentloaded 也失败: %s", e2)
|
|
||||||
if "/login" in page.url.lower() or "/signin" in page.url.lower():
|
|
||||||
logger.warning(
|
|
||||||
"platform_collector: 刷新后回到登录页(%s),重新登录",
|
|
||||||
page.url,
|
|
||||||
)
|
|
||||||
_goto(page, SPA_LOGIN_URL)
|
|
||||||
page.wait_for_timeout(POST_RELOAD_LOGIN_HYDRATION_MS)
|
|
||||||
if not _fill_spa_login(page):
|
|
||||||
if not headless:
|
|
||||||
_wait_for_manual_login(page, wait_sec=human_wait_timeout)
|
|
||||||
else:
|
|
||||||
raise LoginTimeoutError(
|
|
||||||
"刷新后回到 login 页且 headless 无法手工登录"
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
page.wait_for_timeout(POST_LOGIN_SETTLE_MS)
|
|
||||||
_goto(page, PLATFORM_URL)
|
|
||||||
page.wait_for_timeout(POST_RELOGIN_GOTO_WAIT_MS)
|
|
||||||
|
|
||||||
# 7. dump 当前 hr/contracts 页面(用于调试)
|
|
||||||
_dump_page(page, download_dir.parent / "hr_contracts_dump.html", "hr/contracts")
|
|
||||||
|
|
||||||
# 8. 等待表格加载(v1.1.8:先等,**表格超时未出现才触发兜底刷新**——
|
|
||||||
# 修复「首次登录后强制刷新打断手工登录态」问题)
|
|
||||||
page.wait_for_timeout(TABLE_INITIAL_WAIT_MS) # 给 React 初始化 + mock 数据填充时间
|
|
||||||
found_table = False
|
|
||||||
for attempt in range(TABLE_POLL_MAX_ATTEMPTS):
|
|
||||||
page.wait_for_timeout(TABLE_POLL_INTERVAL_MS)
|
|
||||||
cur_url = page.url
|
|
||||||
try:
|
|
||||||
page.wait_for_selector(
|
|
||||||
"table tbody tr",
|
|
||||||
timeout=TABLE_TR_TIMEOUT_MS,
|
|
||||||
)
|
|
||||||
found_table = True
|
|
||||||
break
|
|
||||||
except Exception:
|
|
||||||
logger.info(
|
|
||||||
"platform_collector: URL=%s 等待表格加载 (attempt %d/%d)...",
|
|
||||||
cur_url, attempt + 1, TABLE_POLL_MAX_ATTEMPTS,
|
|
||||||
)
|
|
||||||
|
|
||||||
if not found_table:
|
|
||||||
# v1.1.8:表格没出现,**先尝试兜底刷新一次**(sandbox mock 异步加载场景);
|
|
||||||
# 刷新后再次轮询表格,仍找不到才报 TableNotFoundError。
|
|
||||||
logger.warning(
|
|
||||||
"platform_collector: 首次轮询未找到表格,触发兜底刷新(sandbox mock 异步加载场景)"
|
|
||||||
)
|
|
||||||
_try_reload_after_table_timeout()
|
|
||||||
# 刷新后重新轮询一次(最多 5 次)
|
|
||||||
for attempt2 in range(5):
|
|
||||||
page.wait_for_timeout(TABLE_POLL_INTERVAL_MS)
|
|
||||||
try:
|
|
||||||
page.wait_for_selector(
|
|
||||||
"table tbody tr",
|
|
||||||
timeout=TABLE_TR_TIMEOUT_MS,
|
|
||||||
)
|
|
||||||
found_table = True
|
|
||||||
logger.info(
|
|
||||||
"platform_collector: 兜底刷新后找到表格 (attempt %d)", attempt2 + 1
|
|
||||||
)
|
|
||||||
break
|
|
||||||
except Exception:
|
|
||||||
logger.info(
|
|
||||||
"platform_collector: 兜底刷新后仍等待表格 (attempt %d/5)...",
|
|
||||||
attempt2 + 1,
|
|
||||||
)
|
|
||||||
|
|
||||||
if not found_table:
|
|
||||||
_dump_page(page, download_dir.parent / "sandbox_dump.html", "sandbox")
|
|
||||||
logger.warning("platform_collector: 表格未找到(兜底刷新后仍未出现)")
|
|
||||||
raise TableNotFoundError(
|
|
||||||
f"{TABLE_POLL_MAX_ATTEMPTS * TABLE_POLL_INTERVAL_MS // 1000}s "
|
|
||||||
f"内未找到表格(兜底刷新后仍未出现) (URL={page.url})"
|
|
||||||
)
|
|
||||||
|
|
||||||
# 表格已找到,等 tbody 完全填充 + dump 一份完整 DOM(用于诊断按钮 selector)
|
|
||||||
try:
|
|
||||||
page.wait_for_function(
|
|
||||||
"document.querySelectorAll('table tbody tr').length > 0",
|
|
||||||
timeout=DEFAULT_PAGE_TIMEOUT_MS,
|
|
||||||
)
|
|
||||||
page.wait_for_timeout(FULL_RENDER_WAIT_MS)
|
|
||||||
_dump_page(
|
|
||||||
page,
|
|
||||||
download_dir.parent / "hr_contracts_full_dump.html",
|
|
||||||
"完整 hr/contracts",
|
|
||||||
)
|
|
||||||
except Exception as e:
|
|
||||||
logger.warning("等完整渲染超时: %s", e)
|
|
||||||
|
|
||||||
rows = page.eval_on_selector_all(
|
|
||||||
"table tbody tr",
|
|
||||||
"rs => rs.map(r => Array.from(r.querySelectorAll('td')).map(td => td.innerText))",
|
|
||||||
)
|
|
||||||
fetched_rows = len(rows)
|
|
||||||
logger.info("platform_collector: 抓到 %d 行", fetched_rows)
|
|
||||||
|
|
||||||
# 检查每行是否有「续聘材料包」按钮
|
|
||||||
try:
|
|
||||||
mat_btn_count = page.locator(
|
|
||||||
"table tbody tr button:has-text('续聘材料包')"
|
|
||||||
).count()
|
|
||||||
logger.info(
|
|
||||||
"platform_collector: 「续聘材料包」按钮数=%d", mat_btn_count,
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
mat_btn_count = 0
|
|
||||||
|
|
||||||
# 抓「材料包」card 区段的 DOM(用于后续解析每行 click 后的卡片文本)
|
|
||||||
# sandbox 行为:click 按钮后会在表格下方追加「续聘准备任务书」+「单位清单」卡片
|
|
||||||
|
|
||||||
for i, r in enumerate(rows):
|
|
||||||
if limit and i >= limit:
|
|
||||||
break
|
|
||||||
# v1.1.8:表格列数自适应 —— sandbox 首次登录未刷新是 7 列
|
|
||||||
# (姓名/单位/到期日/剩余天数/预警/建议/按钮),刷新后才是 10 列
|
|
||||||
# (多 工号/岗位/合同类型/起止日期)。两种都支持。
|
|
||||||
ncols = len(r)
|
|
||||||
if ncols < 6:
|
|
||||||
logger.warning(
|
|
||||||
"行 %d 列数=%d < 6,跳过: %s", i, ncols, r
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
|
|
||||||
# 列序探测:判断首列是不是工号(纯数字 + 字母,纯姓名是中文)
|
|
||||||
first_col = r[0].strip()
|
|
||||||
import re as _re
|
|
||||||
is_emp_id = bool(_re.match(r"^[A-Za-z]?\d+$", first_col))
|
|
||||||
|
|
||||||
if is_emp_id and ncols >= 9:
|
|
||||||
# 10 列布局(含工号):0 工号 1 姓名 2 单位 3 岗位 4 合同类型
|
|
||||||
# 5 起止(含~) 6 剩余天数 7 预警 8 建议 9 按钮
|
|
||||||
emp_id = first_col
|
|
||||||
teacher = r[1].strip()
|
|
||||||
unit = r[2].strip()
|
|
||||||
period = r[5].strip()
|
|
||||||
contract_end = period.split("~")[-1].strip() if "~" in period else period
|
|
||||||
days_raw = r[6].strip()
|
|
||||||
alert_label = r[7].strip()
|
|
||||||
suggestion_hint = r[8].strip()
|
|
||||||
elif ncols >= 6:
|
|
||||||
# 7 列布局(无工号):0 姓名 1 单位 2 到期日 3 剩余天数 4 预警 5 建议 6 按钮
|
|
||||||
teacher = first_col
|
|
||||||
unit = r[1].strip()
|
|
||||||
contract_end = r[2].strip()
|
|
||||||
days_raw = r[3].strip()
|
|
||||||
alert_label = r[4].strip()
|
|
||||||
suggestion_hint = r[5].strip()
|
|
||||||
emp_id = "" # 工号缺失,留空让 _gen_contract_id 用 teacher_name 兜底
|
|
||||||
else:
|
|
||||||
logger.warning(
|
|
||||||
"行 %d 列数=%d 无法识别布局,跳过: %s", i, ncols, r
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
|
|
||||||
# 用工号生成 contract_id(稳定,不会随姓名变化)
|
|
||||||
# v1.1.8:7 列布局无工号 → 用 teacher_name 派生(避免与 10 列布局冲突)
|
|
||||||
if emp_id:
|
|
||||||
contract_id = _gen_contract_id(emp_id, contract_end)
|
|
||||||
else:
|
|
||||||
contract_id = _gen_contract_id(teacher, contract_end)
|
|
||||||
contract = {
|
|
||||||
"contract_id": contract_id,
|
|
||||||
"teacher_name": teacher,
|
|
||||||
"unit_name": unit,
|
|
||||||
"contract_end": _normalize_date(contract_end),
|
|
||||||
"days_to_expire_hint": _parse_days(days_raw),
|
|
||||||
"alert_level_hint": _parse_alert(alert_label),
|
|
||||||
"ai_suggestion_hint": suggestion_hint,
|
|
||||||
"materials": [],
|
|
||||||
}
|
|
||||||
|
|
||||||
# 点「续聘材料包」按钮 → 等卡片展开 → 抓卡片文本(含 checklist)
|
|
||||||
if mat_btn_count > 0:
|
|
||||||
try:
|
|
||||||
btn = page.locator("table tbody tr").nth(i).locator(
|
|
||||||
"button:has-text('续聘材料包')"
|
|
||||||
)
|
|
||||||
btn.click()
|
|
||||||
# 等卡片出现(页面下方追加 h2「续聘准备任务书」/「单位清单」)
|
|
||||||
page.wait_for_timeout(CARD_APPEAR_WAIT_MS)
|
|
||||||
# 抓表格下方所有 card 文本
|
|
||||||
cards_text = page.locator(
|
|
||||||
"main .grid > div, main > div > div > div"
|
|
||||||
).all_inner_texts()
|
|
||||||
materials_text = "\n\n".join(
|
|
||||||
t.strip() for t in cards_text if t.strip()
|
|
||||||
)
|
|
||||||
if materials_text:
|
|
||||||
# 存到 materials_content(sandbox mock 卡片文本作为材料包内容)
|
|
||||||
contract["materials"].append({
|
|
||||||
"kind": "sandbox_card",
|
|
||||||
"file_name": f"{contract_id}_materials.txt",
|
|
||||||
"text": materials_text,
|
|
||||||
})
|
|
||||||
downloaded += 1
|
|
||||||
logger.info(
|
|
||||||
"platform_collector: 抓到卡片文本 %d 字 (%s)",
|
|
||||||
len(materials_text), contract_id,
|
|
||||||
)
|
|
||||||
# dump 一次(i==0)方便调试
|
|
||||||
if i == 0:
|
|
||||||
_dump_page(
|
|
||||||
page,
|
|
||||||
download_dir.parent / "after_click_dump.html",
|
|
||||||
"首行 click 后",
|
|
||||||
)
|
|
||||||
except Exception as e:
|
|
||||||
logger.warning("抓卡片失败 %s: %s", contract_id, e)
|
|
||||||
else:
|
|
||||||
if i == 0:
|
|
||||||
logger.info(
|
|
||||||
"platform_collector: sandbox 未部署「续聘材料包」按钮,"
|
|
||||||
"跳过材料抓取(仅采集表格列)",
|
|
||||||
)
|
|
||||||
|
|
||||||
contracts.append(contract)
|
|
||||||
finally:
|
|
||||||
ctx.close()
|
|
||||||
|
|
||||||
saved = _persist_contracts(contracts)
|
|
||||||
return {
|
|
||||||
"platform_url": PLATFORM_URL,
|
|
||||||
"fetched_rows": fetched_rows,
|
|
||||||
"saved_contracts": saved,
|
|
||||||
"downloaded_materials": downloaded,
|
|
||||||
"contracts": contracts,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _persist_contracts(contracts: list[dict]) -> int:
|
# ---------- v1.1.31:从 service.platform_collector_queue re-export 下载队列 + 解析 helpers ----------
|
||||||
"""把抓到的合同 + 材料落到 SQLite。仅插入 contracts 表的占位行,
|
from service.platform_collector_queue import ( # noqa: E402,F401
|
||||||
materials_content 由 parse-materials CLI 后续解析写入。
|
_download_materials_for_row as _download_materials_for_row,
|
||||||
|
_find_row_for_contract as _find_row_for_contract,
|
||||||
真正的合同台账来自 HR 系统 API(生产)或样例 CSV(本地),
|
process_queue as process_queue,
|
||||||
本采集仅作为"快照"记录抓取时间与材料路径。
|
_gen_contract_id as _gen_contract_id,
|
||||||
"""
|
_normalize_date as _normalize_date,
|
||||||
if not contracts:
|
_parse_days as _parse_days,
|
||||||
return 0
|
_parse_alert as _parse_alert,
|
||||||
|
)
|
||||||
from db.connection import get_connection
|
|
||||||
|
|
||||||
conn = get_connection()
|
|
||||||
try:
|
|
||||||
with conn:
|
|
||||||
for c in contracts:
|
|
||||||
# upsert contracts(仅更新与 sandbox 相关的字段)
|
|
||||||
conn.execute(
|
|
||||||
"""
|
|
||||||
INSERT INTO contracts
|
|
||||||
(contract_id, employee_id, teacher_name, unit_id, unit_name,
|
|
||||||
position, contract_type, contract_start, contract_end,
|
|
||||||
trial_period_end, status, last_material_id, renewed_at)
|
|
||||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, '可续聘', NULL, NULL)
|
|
||||||
ON CONFLICT(contract_id) DO UPDATE SET
|
|
||||||
teacher_name = excluded.teacher_name,
|
|
||||||
unit_name = excluded.unit_name,
|
|
||||||
contract_end = excluded.contract_end
|
|
||||||
""",
|
|
||||||
(
|
|
||||||
c["contract_id"],
|
|
||||||
c["contract_id"], # employee_id 暂用 contract_id
|
|
||||||
c["teacher_name"],
|
|
||||||
"U-RPA", # unit_id 占位
|
|
||||||
c["unit_name"],
|
|
||||||
"(RPA 采集)",
|
|
||||||
"(RPA 采集)",
|
|
||||||
"1970-01-01",
|
|
||||||
c["contract_end"],
|
|
||||||
c["contract_end"],
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
# 写 materials_content(sandbox card 文本 + parse-materials 解析后文件路径)
|
|
||||||
from db.migrations import ensure_materials_content_schema
|
|
||||||
|
|
||||||
ensure_materials_content_schema(conn)
|
|
||||||
|
|
||||||
for m in c["materials"]:
|
|
||||||
if isinstance(m, dict) and m.get("kind") == "sandbox_card":
|
|
||||||
# sandbox mock 卡片文本,直接写入 extracted_text
|
|
||||||
conn.execute(
|
|
||||||
"""
|
|
||||||
INSERT OR REPLACE INTO materials_content
|
|
||||||
(contract_id, file_name, file_path, extracted_text,
|
|
||||||
parse_method, parsed_at)
|
|
||||||
VALUES (?, ?, ?, ?, 'sandbox_card', ?)
|
|
||||||
""",
|
|
||||||
(
|
|
||||||
c["contract_id"],
|
|
||||||
m["file_name"],
|
|
||||||
f"sandbox://hr/contracts/{c['contract_id']}",
|
|
||||||
m["text"],
|
|
||||||
datetime.now().isoformat(timespec="seconds"),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
# 同步解析 → 抽 year + 等次 → 写入 performance_records
|
|
||||||
perf_n = repo.parse_sandbox_card_performance(
|
|
||||||
conn, c["contract_id"], m["text"],
|
|
||||||
)
|
|
||||||
if perf_n > 0:
|
|
||||||
logger.info(
|
|
||||||
"platform_collector: %s 解析卡片 → performance %d 条",
|
|
||||||
c["contract_id"], perf_n,
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
# 本地文件路径(parse-materials 解析后写入)
|
|
||||||
fp = m if isinstance(m, str) else m.get("file_path")
|
|
||||||
fname = Path(fp).name if fp else "unknown"
|
|
||||||
conn.execute(
|
|
||||||
"""
|
|
||||||
INSERT OR REPLACE INTO materials_content
|
|
||||||
(contract_id, file_name, file_path, extracted_text,
|
|
||||||
parse_method, parsed_at)
|
|
||||||
VALUES (?, ?, ?, NULL, 'pending', ?)
|
|
||||||
""",
|
|
||||||
(
|
|
||||||
c["contract_id"],
|
|
||||||
fname,
|
|
||||||
fp,
|
|
||||||
datetime.now().isoformat(timespec="seconds"),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
return len(contracts)
|
|
||||||
finally:
|
|
||||||
conn.close()
|
|
||||||
|
|
||||||
|
|
||||||
def _error_result(reason: str, fetched: int) -> dict:
|
|
||||||
return {
|
|
||||||
"platform_url": PLATFORM_URL,
|
|
||||||
"fetched_rows": fetched,
|
|
||||||
"saved_contracts": 0,
|
|
||||||
"downloaded_materials": 0,
|
|
||||||
"contracts": [],
|
|
||||||
"error": reason,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _gen_contract_id(emp_id: str, end_date: str) -> str:
|
|
||||||
"""从 (工号, 到期日) 派 contract_id,确保幂等。
|
|
||||||
|
|
||||||
sandbox v1.1.1 起改用「工号」作主键(之前用姓名)。
|
|
||||||
"""
|
|
||||||
safe_emp = re.sub(r"[^\w]", "", emp_id)[:10] or "T"
|
|
||||||
safe_end = end_date.replace("-", "")[:8]
|
|
||||||
return f"RPA-{safe_emp}-{safe_end}"
|
|
||||||
|
|
||||||
|
|
||||||
def _normalize_date(s: str) -> str:
|
|
||||||
"""尽量把 sandbox 返回的日期归一为 YYYY-MM-DD。"""
|
|
||||||
s = s.strip()
|
|
||||||
if re.match(r"^\d{4}-\d{2}-\d{2}$", s):
|
|
||||||
return s
|
|
||||||
# 兜底:截前 10 字符
|
|
||||||
return s[:10]
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_days(s: str) -> int | None:
|
|
||||||
m = re.search(r"-?\d+", s)
|
|
||||||
return int(m.group(0)) if m else None
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_alert(s: str) -> str:
|
|
||||||
s = s.strip()
|
|
||||||
if "红" in s:
|
|
||||||
return "red"
|
|
||||||
if "橙" in s:
|
|
||||||
return "orange"
|
|
||||||
if "黄" in s:
|
|
||||||
return "yellow"
|
|
||||||
return "unknown"
|
|
||||||
|
|||||||
625
scripts/service/platform_collector_collect.py
Normal file
625
scripts/service/platform_collector_collect.py
Normal file
@@ -0,0 +1,625 @@
|
|||||||
|
"""Playwright RPA:SmartCampus HR 平台合同台账采集主流程。
|
||||||
|
|
||||||
|
v1.1.31:从 platform_collector.py 拆分出来的 collect_all 子模块。
|
||||||
|
包含 2 个函数:
|
||||||
|
- collect_all(登录 + 抓 hr/contracts 表格 + 落库 + 入队下载任务)
|
||||||
|
- _error_result(错误返回包装)
|
||||||
|
|
||||||
|
设计要点(保持 v1.1.30 不变):
|
||||||
|
- 登录子流程通过 service.platform_collector_login 顶部 import(login 是叶子模块,无 cycle)
|
||||||
|
- facade 持有的符号(_goto / _dump_page / _is_logged_in / 异常类 / PLATFORM_URL 等)
|
||||||
|
通过 `from service import platform_collector as _pc` lazy import
|
||||||
|
→ 避免 facade ↔ collect cycle;测试 patch facade attr 后立即生效
|
||||||
|
- persist / queue 持有的函数(_persist_contracts / _gen_contract_id 等)仍位于 facade,
|
||||||
|
Stage 3/4 移到 persist/queue 子模块后,collect 改为对应模块顶部 import
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import time
|
||||||
|
from datetime import datetime
|
||||||
|
|
||||||
|
from util.constants import (
|
||||||
|
CARD_APPEAR_WAIT_MS,
|
||||||
|
DEFAULT_PAGE_TIMEOUT_MS,
|
||||||
|
FULL_RENDER_WAIT_MS,
|
||||||
|
HUMAN_WAIT_TIMEOUT,
|
||||||
|
POST_LOGIN_SETTLE_MS,
|
||||||
|
POST_RELOAD_LOGIN_HYDRATION_MS,
|
||||||
|
POST_RELOGIN_GOTO_WAIT_MS,
|
||||||
|
PUSHSTATE_SETTLE_MS,
|
||||||
|
RELOAD_DOMCONTENTLOADED_TIMEOUT_MS,
|
||||||
|
SIDEBAR_CLICK_URL_TIMEOUT_MS,
|
||||||
|
SIDEBAR_LINK_TIMEOUT_MS,
|
||||||
|
SPA_HYDRATION_MS,
|
||||||
|
TABLE_INITIAL_WAIT_MS,
|
||||||
|
TABLE_POLL_INTERVAL_MS,
|
||||||
|
TABLE_POLL_MAX_ATTEMPTS,
|
||||||
|
TABLE_TR_TIMEOUT_MS,
|
||||||
|
)
|
||||||
|
from util.logging_setup import get_skill_logger
|
||||||
|
from service.platform_collector_login import (
|
||||||
|
_fill_spa_login,
|
||||||
|
_fill_spa_login_from_env,
|
||||||
|
_wait_for_login_inputs_filled,
|
||||||
|
_wait_for_manual_login,
|
||||||
|
_wait_for_url_stable,
|
||||||
|
)
|
||||||
|
|
||||||
|
logger = get_skill_logger()
|
||||||
|
|
||||||
|
|
||||||
|
def collect_all(
|
||||||
|
limit: int = 0,
|
||||||
|
headless: bool = True,
|
||||||
|
human_wait_timeout: int = HUMAN_WAIT_TIMEOUT,
|
||||||
|
) -> dict:
|
||||||
|
"""登录 sandbox 平台 → 抓合同表 + 材料 → 落库。
|
||||||
|
|
||||||
|
human_wait_timeout: 手工登录 fallback 等待秒数(默认 HUMAN_WAIT_TIMEOUT=180)。
|
||||||
|
失败时可能抛出 RPAError(LoginTimeoutError / MissingBrowserError / TableNotFoundError)。
|
||||||
|
"""
|
||||||
|
# Lazy import: 读 facade 持有的 _safe_getenv_dir / 异常类 / PLATFORM_URL 等
|
||||||
|
from service import platform_collector as _pc
|
||||||
|
|
||||||
|
try:
|
||||||
|
from playwright.sync_api import sync_playwright
|
||||||
|
except ImportError:
|
||||||
|
logger.error("playwright 未安装,跳过 RPA 采集")
|
||||||
|
return _error_result("playwright_not_installed", 0)
|
||||||
|
|
||||||
|
profile_dir = _pc._safe_getenv_dir("SMARTCAMPUS_PROFILE_DIR", "playwright-profile")
|
||||||
|
download_dir = _pc._safe_getenv_dir("MATERIALS_DOWNLOAD_DIR", "materials")
|
||||||
|
profile_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
download_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: 启动 profile=%s download_dir=%s headless=%s limit=%d "
|
||||||
|
"human_wait_timeout=%d",
|
||||||
|
profile_dir, download_dir, headless, limit, human_wait_timeout,
|
||||||
|
)
|
||||||
|
|
||||||
|
contracts: list[dict] = []
|
||||||
|
fetched_rows = 0
|
||||||
|
downloaded = 0
|
||||||
|
|
||||||
|
with sync_playwright() as p:
|
||||||
|
# v1.1.9:headed 模式优先用系统 Edge(跟 Windows 桌面 session 同源,
|
||||||
|
# 能真正弹窗让你看到并手工登录);找不到则 fallback Playwright 内置 chromium
|
||||||
|
if not headless:
|
||||||
|
try:
|
||||||
|
ctx = p.chromium.launch_persistent_context(
|
||||||
|
user_data_dir=str(profile_dir),
|
||||||
|
headless=False,
|
||||||
|
accept_downloads=True,
|
||||||
|
channel="msedge",
|
||||||
|
viewport={"width": 1440, "height": 900},
|
||||||
|
)
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: 使用系统 Edge(headed 弹窗)"
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(
|
||||||
|
"platform_collector: 系统 Edge 启动失败(%s),fallback Playwright chromium",
|
||||||
|
e,
|
||||||
|
)
|
||||||
|
ctx = p.chromium.launch_persistent_context(
|
||||||
|
user_data_dir=str(profile_dir),
|
||||||
|
headless=False,
|
||||||
|
accept_downloads=True,
|
||||||
|
viewport={"width": 1440, "height": 900},
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
ctx = p.chromium.launch_persistent_context(
|
||||||
|
user_data_dir=str(profile_dir),
|
||||||
|
headless=True,
|
||||||
|
accept_downloads=True,
|
||||||
|
viewport={"width": 1440, "height": 900},
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
page = ctx.new_page()
|
||||||
|
# v1.1.6:覆盖 Playwright 默认操作超时(之前无 set_default_timeout,导致
|
||||||
|
# 所有 wait_for_selector 默认 30s,调试时不直观)
|
||||||
|
page.set_default_timeout(DEFAULT_PAGE_TIMEOUT_MS)
|
||||||
|
|
||||||
|
# 1. v1.1.21:先 goto **根域**(sandbox 登录页),让 SPA hydrate 2 秒
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: 启动,goto ROOT_URL=%s (当前 URL=%s)",
|
||||||
|
_pc.ROOT_URL, page.url,
|
||||||
|
)
|
||||||
|
_pc._goto(page, _pc.ROOT_URL)
|
||||||
|
# 给 React/Vue 框架 2 秒 hydrate 时间,看登录 input 是否就绪
|
||||||
|
page.wait_for_timeout(2_000)
|
||||||
|
|
||||||
|
if _pc._is_logged_in(page):
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: ✅ 已登录态(URL=%s),跳过登录流程",
|
||||||
|
page.url,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# 2. 等 #jc-user / #jc-pass 出现(最多 2 秒)
|
||||||
|
# 若 sandbox 已预填 demo/123456 → 自动点 Sign in
|
||||||
|
# 若 input 为空 → 等用户输入(**v1.1.24 用户自己点登录按钮**)
|
||||||
|
auto_clicked = _pc._try_auto_sign_in(page, wait_input_sec=2)
|
||||||
|
if auto_clicked:
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: ✅ 自动登录成功(预填 demo),等 URL 跳到 /app/"
|
||||||
|
)
|
||||||
|
elif not headless:
|
||||||
|
# 3. input 为空:等用户输入 → **用户自己点 Sign in**(脚本不抢按钮)
|
||||||
|
if _wait_for_login_inputs_filled(page, wait_sec=human_wait_timeout):
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: ✅ 用户完成输入并跳转,等 URL 跳到 /app/"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# 超时未跳转 → fallback 到原来的多信号等待
|
||||||
|
logger.warning(
|
||||||
|
"platform_collector: 用户未跳转,fallback 到 _wait_for_manual_login"
|
||||||
|
)
|
||||||
|
_wait_for_manual_login(page, wait_sec=human_wait_timeout)
|
||||||
|
else:
|
||||||
|
raise _pc.LoginTimeoutError(
|
||||||
|
f"headless 模式下未登录,且无法手工登录"
|
||||||
|
f"(请用 --show-browser 重试)"
|
||||||
|
)
|
||||||
|
|
||||||
|
# 4. v1.1.27:等 URL 跳到任一「SPA 已路由」状态(最多 30s,**不做任何操作**)
|
||||||
|
# - /app/ = 已到 SPA 主页
|
||||||
|
# - /select = sandbox 中间页
|
||||||
|
# - /login 或 /signin = sandbox 又把人踢回登录页(根域登录后常见,
|
||||||
|
# sandbox 会跳到 /industries/campus/smartcampus/login SPA 二级登录页)
|
||||||
|
# - 如果 URL 跳到 login/signin,**立即**退出 30 秒等待(不再干等),
|
||||||
|
# 让 4.5 处理。否则会等满 30 秒才发现要进 SPA 登录兜底,造成卡顿。
|
||||||
|
spa_url_seen = False
|
||||||
|
if not _wait_for_url_stable(
|
||||||
|
page, ("/app/", "/select", "/login", "/signin"), max_wait_sec=30,
|
||||||
|
):
|
||||||
|
logger.warning(
|
||||||
|
"platform_collector: 登录后 30s 内 URL 未稳定到 /app/、/select、"
|
||||||
|
"/login 或 /signin(当前 URL=%s)",
|
||||||
|
page.url,
|
||||||
|
)
|
||||||
|
|
||||||
|
# 4.5. v1.1.28/31:URL 落在 SPA 二级登录页 → 主动等 input hydrate(最多 10 秒)
|
||||||
|
# → 从 .env 读凭证 fill(v1.1.31:密码不硬编码)→ 点登录 → 等 URL 跳离
|
||||||
|
# v1.1.27 失败的根因:URL 已稳定在 /login 时,React SPA input 可能还没渲染,
|
||||||
|
# 找不到 input → False → 卡 180 秒等用户输入。
|
||||||
|
# 修复:先等 input 真正 hydrate 出来,再 fill+click。
|
||||||
|
cur_url_after_stable = page.url.lower()
|
||||||
|
if "/login" in cur_url_after_stable or "/signin" in cur_url_after_stable:
|
||||||
|
logger.warning(
|
||||||
|
"platform_collector: 登录后落在 SPA 二级登录页(%s),"
|
||||||
|
"v1.1.31 触发 SPA 登录兜底(等 input hydrate + 从 .env 读凭证 fill)",
|
||||||
|
page.url,
|
||||||
|
)
|
||||||
|
# 1. 等 input 真正 hydrate 出来(最多 10 秒)
|
||||||
|
try:
|
||||||
|
page.wait_for_selector(
|
||||||
|
"#jc-user, #jc-pass, input[type='tel'], input[type='password']",
|
||||||
|
timeout=10_000,
|
||||||
|
state="attached",
|
||||||
|
)
|
||||||
|
page.wait_for_timeout(500) # 让 React 状态稳定
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(
|
||||||
|
"platform_collector: SPA 登录页 10s 内未渲染 input: %s", e,
|
||||||
|
)
|
||||||
|
|
||||||
|
# 2. 从 .env 读凭证 fill + 点登录
|
||||||
|
spa_filled = _fill_spa_login_from_env(page)
|
||||||
|
if spa_filled:
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: ✅ SPA 二级登录页(来自 .env)已自动填好并点登录,"
|
||||||
|
"等 URL 跳到 /app/ 或 /select"
|
||||||
|
)
|
||||||
|
# 等 URL 跳离 login(30 秒内跳到 /app/ 或 /select 都算成功)
|
||||||
|
if not _wait_for_url_stable(page, ("/app/", "/select"), max_wait_sec=30):
|
||||||
|
logger.warning(
|
||||||
|
"platform_collector: SPA 登录后 30s 内 URL 未稳定到 /app/或/select"
|
||||||
|
"(当前 URL=%s,.env 凭证可能失效)",
|
||||||
|
page.url,
|
||||||
|
)
|
||||||
|
# 凭证失效:等用户手工登录(headless 模式直接抛错)
|
||||||
|
if not headless:
|
||||||
|
_wait_for_manual_login(page, wait_sec=human_wait_timeout)
|
||||||
|
else:
|
||||||
|
raise _pc.LoginTimeoutError(
|
||||||
|
f"SPA 二级登录页 .env 凭证 fill 后 URL 不跳离"
|
||||||
|
f"(可能凭证失效,URL={page.url})"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# .env 无凭证 / input 仍未渲染 → 等用户手工输入
|
||||||
|
logger.warning(
|
||||||
|
"platform_collector: SPA 登录页自动 fill 失败"
|
||||||
|
"(.env 无凭证或 input 未渲染),fallback 等用户输入"
|
||||||
|
)
|
||||||
|
if not headless:
|
||||||
|
if _wait_for_login_inputs_filled(page, wait_sec=human_wait_timeout):
|
||||||
|
_wait_for_url_stable(page, ("/app/", "/select"), max_wait_sec=30)
|
||||||
|
else:
|
||||||
|
_wait_for_manual_login(page, wait_sec=human_wait_timeout)
|
||||||
|
else:
|
||||||
|
raise _pc.LoginTimeoutError(
|
||||||
|
f"SPA 二级登录页 .env 无凭证且 headless 无法手工登录"
|
||||||
|
f"(URL={page.url})"
|
||||||
|
)
|
||||||
|
|
||||||
|
# 5. v1.1.22:显式跳到 SPA_HOME_URL(/app/home),等侧边栏 link 出现
|
||||||
|
# 避免 sandbox 在 /select 中间页时直接找 sidebar link 找不到
|
||||||
|
# v1.1.32 优化:URL 不在 /app/home(可能在 /select 中间页)就 goto(5s 等)
|
||||||
|
if "/app/home" not in page.url.lower():
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: 当前 URL=%s,跳到 SPA_HOME_URL=%s",
|
||||||
|
page.url, _pc.SPA_HOME_URL,
|
||||||
|
)
|
||||||
|
_pc._goto(page, _pc.SPA_HOME_URL)
|
||||||
|
page.wait_for_timeout(SPA_HYDRATION_MS)
|
||||||
|
# v1.1.32 优化:等 URL 稳定到 /app/home(最多 5s,比 15s 快很多)
|
||||||
|
if not _wait_for_url_stable(page, "/app/home", max_wait_sec=5):
|
||||||
|
logger.warning(
|
||||||
|
"platform_collector: 5s 内未稳定到 /app/home(当前 URL=%s)",
|
||||||
|
page.url,
|
||||||
|
)
|
||||||
|
|
||||||
|
# 6. 等 SPA 渲染 + click 侧边栏 hr/contracts(SPA 路由,避免 goto 触发整页 reload 丢 session)
|
||||||
|
try:
|
||||||
|
# dump SPA home 一次(用于调试 selector)
|
||||||
|
page.wait_for_timeout(POST_LOGIN_SETTLE_MS)
|
||||||
|
_pc._dump_page(page, download_dir.parent / "spa_home_dump.html", "SPA home")
|
||||||
|
|
||||||
|
# 等 SPA 真正加载完(侧边栏 link 出现,最长 SIDEBAR_LINK_TIMEOUT_MS)
|
||||||
|
try:
|
||||||
|
page.wait_for_selector(
|
||||||
|
"a[href$='/hr/contracts']",
|
||||||
|
timeout=SIDEBAR_LINK_TIMEOUT_MS,
|
||||||
|
state="attached",
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning("等侧边栏 hr/contracts link 超时: %s", e)
|
||||||
|
|
||||||
|
link = page.locator("a[href$='/hr/contracts']").first
|
||||||
|
if link.count() > 0:
|
||||||
|
link.click()
|
||||||
|
logger.info("platform_collector: click 侧边栏 hr/contracts")
|
||||||
|
try:
|
||||||
|
page.wait_for_url(
|
||||||
|
lambda u: "/hr/contracts" in u,
|
||||||
|
timeout=SIDEBAR_CLICK_URL_TIMEOUT_MS,
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning("等 /hr/contracts URL 超时: %s;当前 URL=%s",
|
||||||
|
e, page.url)
|
||||||
|
else:
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: 侧边栏 link 仍未出现,"
|
||||||
|
"使用 SPA 路由(pushState)到 hr/contracts, 当前 URL=%s",
|
||||||
|
page.url,
|
||||||
|
)
|
||||||
|
# 用 SPA 内部路由而非整页 goto,避免 session 被 server 端踢回 login
|
||||||
|
try:
|
||||||
|
page.evaluate(
|
||||||
|
"u => history.pushState({}, '', u) || "
|
||||||
|
"window.dispatchEvent(new PopStateEvent('popstate')),"
|
||||||
|
" '/industries/campus/smartcampus/app/hr/contracts'"
|
||||||
|
)
|
||||||
|
page.wait_for_timeout(PUSHSTATE_SETTLE_MS)
|
||||||
|
# v1.1.32:pushState 后 URL 可能没切(sandbox mock 偶发),
|
||||||
|
# 检查 URL 真到 /hr/contracts 才算成功,否则 fallback goto
|
||||||
|
if "/hr/contracts" not in page.url.lower():
|
||||||
|
logger.warning(
|
||||||
|
"platform_collector: pushState 后 URL=%s 未到 /hr/contracts,"
|
||||||
|
"fallback goto PLATFORM_URL",
|
||||||
|
page.url,
|
||||||
|
)
|
||||||
|
_pc._goto(page, _pc.PLATFORM_URL)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning("SPA pushState 失败: %s", e)
|
||||||
|
_pc._goto(page, _pc.PLATFORM_URL)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning("click 侧边栏失败: %s;fallback goto", e)
|
||||||
|
_pc._goto(page, _pc.PLATFORM_URL)
|
||||||
|
|
||||||
|
logger.info("platform_collector: 当前 URL=%s", page.url)
|
||||||
|
|
||||||
|
# 5.5. v1.1.8:**首次登录后不再强制刷新**,避免打断用户手工登录态。
|
||||||
|
# 改为「等表格出现」兜底:给 SPA 一个初始等待窗口(TABLE_INITIAL_WAIT_MS*2),
|
||||||
|
# 如果表格仍未出现,**才**刷新一次(sandbox mock 异步加载场景)。
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: 进入 hr/contracts 后**不刷新**,等待表格出现"
|
||||||
|
"(首次登录态不要打断)"
|
||||||
|
)
|
||||||
|
|
||||||
|
# 5.6. 兜底刷新:仅当等表格超时后才触发;刷新后可能被踢回 login,需重新登录
|
||||||
|
# 修复:之前没检查 URL,刷新后停在 login 页继续往后走,找不到表格就 dump 报错
|
||||||
|
# v1.1.32 优化:刷回 login 时用 _fill_spa_login_from_env(从 .env 读),
|
||||||
|
# 不再调硬编码 demo/123456 的 _fill_spa_login(与 .env 凭证一致)。
|
||||||
|
def _try_reload_after_table_timeout():
|
||||||
|
logger.warning(
|
||||||
|
"platform_collector: 表格未在 %dms 内出现,触发兜底刷新",
|
||||||
|
TABLE_INITIAL_WAIT_MS * 2,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
page.reload(
|
||||||
|
wait_until="networkidle",
|
||||||
|
timeout=RELOAD_NETWORKIDLE_TIMEOUT_MS,
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning("刷新 networkidle 失败: %s;尝试 domcontentloaded", e)
|
||||||
|
try:
|
||||||
|
page.reload(
|
||||||
|
wait_until="domcontentloaded",
|
||||||
|
timeout=RELOAD_DOMCONTENTLOADED_TIMEOUT_MS,
|
||||||
|
)
|
||||||
|
except Exception as e2:
|
||||||
|
logger.warning("刷新 domcontentloaded 也失败: %s", e2)
|
||||||
|
if "/login" in page.url.lower() or "/signin" in page.url.lower():
|
||||||
|
logger.warning(
|
||||||
|
"platform_collector: 刷新后回到登录页(%s),重新登录",
|
||||||
|
page.url,
|
||||||
|
)
|
||||||
|
# v1.1.32:先等 input hydrate,再走 .env 凭证 fill
|
||||||
|
try:
|
||||||
|
page.wait_for_selector(
|
||||||
|
"#jc-user, #jc-pass, input[type='tel'], input[type='password']",
|
||||||
|
timeout=10_000,
|
||||||
|
state="attached",
|
||||||
|
)
|
||||||
|
page.wait_for_timeout(500)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(
|
||||||
|
"platform_collector: 刷新后 login 页 10s 内未渲染 input: %s", e,
|
||||||
|
)
|
||||||
|
if not _fill_spa_login_from_env(page):
|
||||||
|
# v1.1.32:.env 无凭证时 fallback 到硬编码 demo(保持 v1.1.10 兼容)
|
||||||
|
logger.warning(
|
||||||
|
"platform_collector: .env 无凭证,fallback 硬编码 demo/123456"
|
||||||
|
)
|
||||||
|
if not _fill_spa_login(page):
|
||||||
|
if not headless:
|
||||||
|
_wait_for_manual_login(page, wait_sec=human_wait_timeout)
|
||||||
|
else:
|
||||||
|
raise _pc.LoginTimeoutError(
|
||||||
|
"刷新后回到 login 页且 headless 无法手工登录"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
page.wait_for_timeout(POST_LOGIN_SETTLE_MS)
|
||||||
|
else:
|
||||||
|
# .env 凭证 fill 成功 → 等 URL 跳离 login
|
||||||
|
_wait_for_url_stable(page, ("/app/", "/select"), max_wait_sec=15)
|
||||||
|
_pc._goto(page, _pc.PLATFORM_URL)
|
||||||
|
page.wait_for_timeout(POST_RELOAD_LOGIN_HYDRATION_MS)
|
||||||
|
|
||||||
|
# 7. dump 当前 hr/contracts 页面(用于调试)
|
||||||
|
_pc._dump_page(page, download_dir.parent / "hr_contracts_dump.html", "hr/contracts")
|
||||||
|
|
||||||
|
# 8. 等待表格加载(v1.1.8:先等,**表格超时未出现才触发兜底刷新**——
|
||||||
|
# 修复「首次登录后强制刷新打断手工登录态」问题)
|
||||||
|
page.wait_for_timeout(TABLE_INITIAL_WAIT_MS) # 给 React 初始化 + mock 数据填充时间
|
||||||
|
found_table = False
|
||||||
|
for attempt in range(TABLE_POLL_MAX_ATTEMPTS):
|
||||||
|
page.wait_for_timeout(TABLE_POLL_INTERVAL_MS)
|
||||||
|
cur_url = page.url
|
||||||
|
try:
|
||||||
|
page.wait_for_selector(
|
||||||
|
"table tbody tr",
|
||||||
|
timeout=TABLE_TR_TIMEOUT_MS,
|
||||||
|
)
|
||||||
|
found_table = True
|
||||||
|
break
|
||||||
|
except Exception:
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: URL=%s 等待表格加载 (attempt %d/%d)...",
|
||||||
|
cur_url, attempt + 1, TABLE_POLL_MAX_ATTEMPTS,
|
||||||
|
)
|
||||||
|
|
||||||
|
if not found_table:
|
||||||
|
# v1.1.8:表格没出现,**先尝试兜底刷新一次**(sandbox mock 异步加载场景);
|
||||||
|
# 刷新后再次轮询表格,仍找不到才报 TableNotFoundError。
|
||||||
|
logger.warning(
|
||||||
|
"platform_collector: 首次轮询未找到表格,触发兜底刷新(sandbox mock 异步加载场景)"
|
||||||
|
)
|
||||||
|
_try_reload_after_table_timeout()
|
||||||
|
# 刷新后重新轮询一次(最多 5 次)
|
||||||
|
for attempt2 in range(5):
|
||||||
|
page.wait_for_timeout(TABLE_POLL_INTERVAL_MS)
|
||||||
|
try:
|
||||||
|
page.wait_for_selector(
|
||||||
|
"table tbody tr",
|
||||||
|
timeout=TABLE_TR_TIMEOUT_MS,
|
||||||
|
)
|
||||||
|
found_table = True
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: 兜底刷新后找到表格 (attempt %d)", attempt2 + 1
|
||||||
|
)
|
||||||
|
break
|
||||||
|
except Exception:
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: 兜底刷新后仍等待表格 (attempt %d/5)...",
|
||||||
|
attempt2 + 1,
|
||||||
|
)
|
||||||
|
|
||||||
|
if not found_table:
|
||||||
|
_pc._dump_page(page, download_dir.parent / "sandbox_dump.html", "sandbox")
|
||||||
|
logger.warning("platform_collector: 表格未找到(兜底刷新后仍未出现)")
|
||||||
|
raise _pc.TableNotFoundError(
|
||||||
|
f"{TABLE_POLL_MAX_ATTEMPTS * TABLE_POLL_INTERVAL_MS // 1000}s "
|
||||||
|
f"内未找到表格(兜底刷新后仍未出现) (URL={page.url})"
|
||||||
|
)
|
||||||
|
|
||||||
|
# 表格已找到,等 tbody 完全填充 + dump 一份完整 DOM(用于诊断按钮 selector)
|
||||||
|
try:
|
||||||
|
page.wait_for_function(
|
||||||
|
"document.querySelectorAll('table tbody tr').length > 0",
|
||||||
|
timeout=DEFAULT_PAGE_TIMEOUT_MS,
|
||||||
|
)
|
||||||
|
page.wait_for_timeout(FULL_RENDER_WAIT_MS)
|
||||||
|
_pc._dump_page(
|
||||||
|
page,
|
||||||
|
download_dir.parent / "hr_contracts_full_dump.html",
|
||||||
|
"完整 hr/contracts",
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning("等完整渲染超时: %s", e)
|
||||||
|
|
||||||
|
rows = page.eval_on_selector_all(
|
||||||
|
"table tbody tr",
|
||||||
|
"rs => rs.map(r => Array.from(r.querySelectorAll('td')).map(td => td.innerText))",
|
||||||
|
)
|
||||||
|
fetched_rows = len(rows)
|
||||||
|
logger.info("platform_collector: 抓到 %d 行", fetched_rows)
|
||||||
|
|
||||||
|
# 检查每行是否有「续聘材料包」按钮
|
||||||
|
try:
|
||||||
|
mat_btn_count = page.locator(
|
||||||
|
"table tbody tr button:has-text('续聘材料包')"
|
||||||
|
).count()
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: 「续聘材料包」按钮数=%d", mat_btn_count,
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
mat_btn_count = 0
|
||||||
|
|
||||||
|
# 抓「材料包」card 区段的 DOM(用于后续解析每行 click 后的卡片文本)
|
||||||
|
# sandbox 行为:click 按钮后会在表格下方追加「续聘准备任务书」+「单位清单」卡片
|
||||||
|
|
||||||
|
for i, r in enumerate(rows):
|
||||||
|
if limit and i >= limit:
|
||||||
|
break
|
||||||
|
# v1.1.8:表格列数自适应 —— sandbox 首次登录未刷新是 7 列
|
||||||
|
# (姓名/单位/到期日/剩余天数/预警/建议/按钮),刷新后才是 10 列
|
||||||
|
# (多 工号/岗位/合同类型/起止日期)。两种都支持。
|
||||||
|
ncols = len(r)
|
||||||
|
if ncols < 6:
|
||||||
|
logger.warning(
|
||||||
|
"行 %d 列数=%d < 6,跳过: %s", i, ncols, r
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
|
||||||
|
# 列序探测:判断首列是不是工号(纯数字 + 字母,纯姓名是中文)
|
||||||
|
first_col = r[0].strip()
|
||||||
|
import re as _re
|
||||||
|
is_emp_id = bool(_re.match(r"^[A-Za-z]?\d+$", first_col))
|
||||||
|
|
||||||
|
if is_emp_id and ncols >= 9:
|
||||||
|
# 10 列布局(含工号):0 工号 1 姓名 2 单位 3 岗位 4 合同类型
|
||||||
|
# 5 起止(含~) 6 剩余天数 7 预警 8 建议 9 按钮
|
||||||
|
emp_id = first_col
|
||||||
|
teacher = r[1].strip()
|
||||||
|
unit = r[2].strip()
|
||||||
|
period = r[5].strip()
|
||||||
|
contract_end = period.split("~")[-1].strip() if "~" in period else period
|
||||||
|
days_raw = r[6].strip()
|
||||||
|
alert_label = r[7].strip()
|
||||||
|
suggestion_hint = r[8].strip()
|
||||||
|
elif ncols >= 6:
|
||||||
|
# 7 列布局(无工号):0 姓名 1 单位 2 到期日 3 剩余天数 4 预警 5 建议 6 按钮
|
||||||
|
teacher = first_col
|
||||||
|
unit = r[1].strip()
|
||||||
|
contract_end = r[2].strip()
|
||||||
|
days_raw = r[3].strip()
|
||||||
|
alert_label = r[4].strip()
|
||||||
|
suggestion_hint = r[5].strip()
|
||||||
|
emp_id = "" # 工号缺失,留空让 _gen_contract_id 用 teacher_name 兜底
|
||||||
|
else:
|
||||||
|
logger.warning(
|
||||||
|
"行 %d 列数=%d 无法识别布局,跳过: %s", i, ncols, r
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
|
||||||
|
# 用工号生成 contract_id(稳定,不会随姓名变化)
|
||||||
|
# v1.1.8:7 列布局无工号 → 用 teacher_name 派生(避免与 10 列布局冲突)
|
||||||
|
if emp_id:
|
||||||
|
contract_id = _pc._gen_contract_id(emp_id, contract_end)
|
||||||
|
else:
|
||||||
|
contract_id = _pc._gen_contract_id(teacher, contract_end)
|
||||||
|
contract = {
|
||||||
|
"contract_id": contract_id,
|
||||||
|
"teacher_name": teacher,
|
||||||
|
"unit_name": unit,
|
||||||
|
"contract_end": _pc._normalize_date(contract_end),
|
||||||
|
"days_to_expire_hint": _pc._parse_days(days_raw),
|
||||||
|
"alert_level_hint": _pc._parse_alert(alert_label),
|
||||||
|
"ai_suggestion_hint": suggestion_hint,
|
||||||
|
"materials": [],
|
||||||
|
}
|
||||||
|
|
||||||
|
# 点「续聘材料包」按钮 → 等卡片展开 → 抓卡片文本(含 checklist)
|
||||||
|
if mat_btn_count > 0:
|
||||||
|
try:
|
||||||
|
btn = page.locator("table tbody tr").nth(i).locator(
|
||||||
|
"button:has-text('续聘材料包')"
|
||||||
|
)
|
||||||
|
btn.click()
|
||||||
|
# 等卡片出现(页面下方追加 h2「续聘准备任务书」/「单位清单」)
|
||||||
|
page.wait_for_timeout(CARD_APPEAR_WAIT_MS)
|
||||||
|
# 抓表格下方所有 card 文本
|
||||||
|
cards_text = page.locator(
|
||||||
|
"main .grid > div, main > div > div > div"
|
||||||
|
).all_inner_texts()
|
||||||
|
materials_text = "\n\n".join(
|
||||||
|
t.strip() for t in cards_text if t.strip()
|
||||||
|
)
|
||||||
|
if materials_text:
|
||||||
|
# 存到 materials_content(sandbox mock 卡片文本作为材料包内容)
|
||||||
|
contract["materials"].append({
|
||||||
|
"kind": "sandbox_card",
|
||||||
|
"file_name": f"{contract_id}_materials.txt",
|
||||||
|
"text": materials_text,
|
||||||
|
})
|
||||||
|
downloaded += 1
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: 抓到卡片文本 %d 字 (%s)",
|
||||||
|
len(materials_text), contract_id,
|
||||||
|
)
|
||||||
|
# dump 一次(i==0)方便调试
|
||||||
|
if i == 0:
|
||||||
|
_pc._dump_page(
|
||||||
|
page,
|
||||||
|
download_dir.parent / "after_click_dump.html",
|
||||||
|
"首行 click 后",
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning("抓卡片失败 %s: %s", contract_id, e)
|
||||||
|
else:
|
||||||
|
if i == 0:
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: sandbox 未部署「续聘材料包」按钮,"
|
||||||
|
"跳过材料抓取(仅采集表格列)",
|
||||||
|
)
|
||||||
|
|
||||||
|
contracts.append(contract)
|
||||||
|
finally:
|
||||||
|
ctx.close()
|
||||||
|
|
||||||
|
saved = _pc._persist_contracts(contracts)
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: contracts 落库完成 saved=%d / total_fetched=%d",
|
||||||
|
saved, len(contracts),
|
||||||
|
)
|
||||||
|
|
||||||
|
# v1.1.18:每个合同入队 download_tasks(独立事务,失败不影响主流程)
|
||||||
|
contract_ids = [c["contract_id"] for c in contracts if c.get("contract_id")]
|
||||||
|
enqueue_summary = _pc._enqueue_download_tasks_safe(contract_ids)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"platform_url": _pc.PLATFORM_URL,
|
||||||
|
"fetched_rows": fetched_rows,
|
||||||
|
"saved_contracts": saved,
|
||||||
|
"downloaded_materials": downloaded,
|
||||||
|
"download_queue": enqueue_summary,
|
||||||
|
"contracts": contracts,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _error_result(reason: str, fetched: int) -> dict:
|
||||||
|
# Lazy import: 读 facade 的 PLATFORM_URL
|
||||||
|
from service import platform_collector as _pc
|
||||||
|
|
||||||
|
return {
|
||||||
|
"platform_url": _pc.PLATFORM_URL,
|
||||||
|
"fetched_rows": fetched,
|
||||||
|
"saved_contracts": 0,
|
||||||
|
"downloaded_materials": 0,
|
||||||
|
"contracts": [],
|
||||||
|
"error": reason,
|
||||||
|
}
|
||||||
743
scripts/service/platform_collector_login.py
Normal file
743
scripts/service/platform_collector_login.py
Normal file
@@ -0,0 +1,743 @@
|
|||||||
|
"""Playwright RPA:SmartCampus HR 平台登录流程。
|
||||||
|
|
||||||
|
v1.1.31:从 platform_collector.py 拆分出来的纯登录流程子模块。
|
||||||
|
包含 12 个函数:
|
||||||
|
- _try_auto_sign_in / _wait_for_login_inputs_filled / _is_login_page_url_for_user
|
||||||
|
- _wait_for_manual_login / _wait_for_url_stable / _fill_root_login
|
||||||
|
- _fill_spa_login_from_env / _is_dev_mode_panel_open / _resolve_target_industry
|
||||||
|
- _click_dev_mode_sign_in_for_industry / _click_campus_link / _fill_spa_login
|
||||||
|
|
||||||
|
设计要点(保持 v1.1.30 不变):
|
||||||
|
- 所有 facade 公开符号(HUMAN_POLL_INTERVAL / LoginTimeoutError / _goto / _dump_page /
|
||||||
|
_is_logged_in / _is_spa_routed)通过 `from service import platform_collector as _pc` lazy import
|
||||||
|
→ 保证测试 `pc.HUMAN_POLL_INTERVAL = 0.001` monkey-patch 生效
|
||||||
|
- `HUMAN_WAIT_TIMEOUT` / `SPA_LOGIN_*` / `SIDEBAR_*` / `URL_STABLE_*` 等不会动态 patch 的常量
|
||||||
|
从 `util.constants` 顶部 import
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import time
|
||||||
|
|
||||||
|
from util.constants import (
|
||||||
|
HUMAN_PROMPT_INTERVAL,
|
||||||
|
HUMAN_WAIT_TIMEOUT,
|
||||||
|
SIDEBAR_LINK_TIMEOUT_MS,
|
||||||
|
SPA_LOGIN_INPUT_TIMEOUT_MS,
|
||||||
|
SPA_LOGIN_SETTLE_MS,
|
||||||
|
SPA_LOGIN_URL_TIMEOUT_MS,
|
||||||
|
URL_AFTER_LOGIN_TIMEOUT_MS,
|
||||||
|
URL_STABLE_POLL_S,
|
||||||
|
)
|
||||||
|
from util.logging_setup import get_skill_logger
|
||||||
|
|
||||||
|
logger = get_skill_logger()
|
||||||
|
|
||||||
|
|
||||||
|
def _try_auto_sign_in(page, wait_input_sec: int = 2) -> bool:
|
||||||
|
"""v1.1.21:检测登录页 demo/123456 是否已预填,有则自动点 Sign in。
|
||||||
|
|
||||||
|
v1.1.24:扩展到支持 SPA 二级登录页(/industries/campus/smartcampus/login),
|
||||||
|
按钮文本可能是中文「登录」而非英文「Sign in」。
|
||||||
|
|
||||||
|
适用场景:
|
||||||
|
- 根域登录页(https://sandbox.jc2009.com/):按钮「Sign in」
|
||||||
|
- SPA 二级登录页(/industries/campus/smartcampus/login):按钮「登录」
|
||||||
|
默认有 demo/123456(或 18677308191 等真实账号)预填在 input。
|
||||||
|
|
||||||
|
行为:
|
||||||
|
1. 等 input 出现,最长 wait_input_sec 秒(默认 2s)
|
||||||
|
2. 检查 input 是否都有值
|
||||||
|
3. 有值 → 点 Sign in / 登录 → return True
|
||||||
|
4. 无值 → return False(交给 _wait_for_login_inputs_filled 接管)
|
||||||
|
|
||||||
|
不会自动 fill(如果没值就放弃,让用户自己输)。
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
page.wait_for_selector(
|
||||||
|
"#jc-user, #jc-pass, input[type='tel'], input[type='password']",
|
||||||
|
timeout=wait_input_sec * 1000,
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: %ds 内未出现登录 input,跳过自动登录",
|
||||||
|
wait_input_sec,
|
||||||
|
)
|
||||||
|
return False
|
||||||
|
|
||||||
|
# 多种 input selector 优先级
|
||||||
|
user_input = None
|
||||||
|
for sel in ("#jc-user", "input[type='tel']", "input[autocomplete='username']"):
|
||||||
|
loc = page.locator(sel).first
|
||||||
|
if loc.count() > 0:
|
||||||
|
user_input = loc
|
||||||
|
break
|
||||||
|
if user_input is None:
|
||||||
|
return False
|
||||||
|
|
||||||
|
pass_input = None
|
||||||
|
for sel in ("#jc-pass", "input[type='password']", "input[autocomplete='current-password']"):
|
||||||
|
loc = page.locator(sel).first
|
||||||
|
if loc.count() > 0:
|
||||||
|
pass_input = loc
|
||||||
|
break
|
||||||
|
if pass_input is None:
|
||||||
|
return False
|
||||||
|
|
||||||
|
user_val = (user_input.input_value() or "").strip()
|
||||||
|
pass_val = (pass_input.input_value() or "").strip()
|
||||||
|
|
||||||
|
if not user_val or not pass_val:
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: 登录页 input 为空(user=%r, pass=%r),"
|
||||||
|
"等用户输入",
|
||||||
|
bool(user_val), bool(pass_val),
|
||||||
|
)
|
||||||
|
return False
|
||||||
|
|
||||||
|
# 多种按钮文本优先级:英文 Sign in > 中文登录 > submit
|
||||||
|
btn = None
|
||||||
|
btn_text_used = ""
|
||||||
|
for txt in ("Sign in", "登录", "Login", "signin"):
|
||||||
|
loc = page.locator(f"button:has-text('{txt}')").first
|
||||||
|
if loc.count() > 0:
|
||||||
|
btn = loc
|
||||||
|
btn_text_used = txt
|
||||||
|
break
|
||||||
|
if btn is None:
|
||||||
|
btn = page.locator("button[type='submit']").first
|
||||||
|
btn_text_used = "submit"
|
||||||
|
if btn is None or btn.count() == 0:
|
||||||
|
logger.warning("platform_collector: 未找到登录按钮")
|
||||||
|
return False
|
||||||
|
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: 登录页已预填账号(user=%r),自动点 %s 按钮",
|
||||||
|
user_val, btn_text_used,
|
||||||
|
)
|
||||||
|
btn.click()
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def _wait_for_login_inputs_filled(page, wait_sec: int = HUMAN_WAIT_TIMEOUT) -> bool:
|
||||||
|
"""v1.1.21:等用户输入完账号密码,然后自动点登录按钮。
|
||||||
|
|
||||||
|
v1.1.26:按用户最新要求改回**自动点登录**行为——
|
||||||
|
「到了 /industries/campus/smartcampus/login 整个页面,有账号和密码可用直接点登录跳转」,
|
||||||
|
不再等用户自己点。
|
||||||
|
|
||||||
|
优先级(任一 selector 命中即视为有 input):
|
||||||
|
- user: `#jc-user` / `input[type='tel']` / `input[autocomplete='username']`
|
||||||
|
- pass: `#jc-pass` / `input[type='password']` / `input[autocomplete='current-password']`
|
||||||
|
|
||||||
|
按钮文本优先级:`Sign in` / `登录` / `Login` / `signin` / `submit`。
|
||||||
|
"""
|
||||||
|
# Lazy import: 测试通过 pc.HUMAN_POLL_INTERVAL = X 加速,需要 facade attr lookup
|
||||||
|
from service import platform_collector as _pc
|
||||||
|
|
||||||
|
deadline = time.monotonic() + wait_sec
|
||||||
|
last_prompt_at = time.monotonic()
|
||||||
|
last_state_log = time.monotonic()
|
||||||
|
|
||||||
|
while time.monotonic() < deadline:
|
||||||
|
cur_url = page.url
|
||||||
|
|
||||||
|
# 已不在登录页(用户可能已点登录完成跳转)
|
||||||
|
if not _is_login_page_url_for_user(cur_url):
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: URL 已离开登录页(%s),登录完成", cur_url,
|
||||||
|
)
|
||||||
|
return True
|
||||||
|
|
||||||
|
# 探测 input(多 selector 优先级)
|
||||||
|
user_val, pass_val = "", ""
|
||||||
|
user_loc, pass_loc = None, None
|
||||||
|
for sel in ("#jc-user", "input[type='tel']", "input[autocomplete='username']"):
|
||||||
|
loc = page.locator(sel).first
|
||||||
|
if loc.count() > 0:
|
||||||
|
user_loc = loc
|
||||||
|
user_val = (loc.input_value() or "").strip()
|
||||||
|
break
|
||||||
|
for sel in ("#jc-pass", "input[type='password']", "input[autocomplete='current-password']"):
|
||||||
|
loc = page.locator(sel).first
|
||||||
|
if loc.count() > 0:
|
||||||
|
pass_loc = loc
|
||||||
|
pass_val = (loc.input_value() or "").strip()
|
||||||
|
break
|
||||||
|
|
||||||
|
now = time.monotonic()
|
||||||
|
if now - last_state_log >= HUMAN_PROMPT_INTERVAL:
|
||||||
|
last_state_log = now
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: 等待用户输入 user=%r pass=%r (URL=%s)",
|
||||||
|
bool(user_val), bool(pass_val), cur_url,
|
||||||
|
)
|
||||||
|
|
||||||
|
# input 都有值 → 自动点登录按钮(v1.1.26 关键:用户最新要求自动点)
|
||||||
|
if user_val and pass_val:
|
||||||
|
btn = None
|
||||||
|
btn_text_used = ""
|
||||||
|
for txt in ("Sign in", "登录", "Login", "signin"):
|
||||||
|
loc = page.locator(f"button:has-text('{txt}')").first
|
||||||
|
if loc.count() > 0:
|
||||||
|
btn = loc
|
||||||
|
btn_text_used = txt
|
||||||
|
break
|
||||||
|
if btn is None:
|
||||||
|
btn = page.locator("button[type='submit']").first
|
||||||
|
btn_text_used = "submit"
|
||||||
|
if btn is not None and btn.count() > 0:
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: ✅ 账号密码已填好(user=%r),自动点 %s 按钮",
|
||||||
|
user_val, btn_text_used,
|
||||||
|
)
|
||||||
|
btn.click()
|
||||||
|
return True
|
||||||
|
logger.warning("platform_collector: input 有值但找不到登录按钮")
|
||||||
|
|
||||||
|
# 提醒用户
|
||||||
|
if now - last_prompt_at >= HUMAN_PROMPT_INTERVAL:
|
||||||
|
last_prompt_at = now
|
||||||
|
remaining = int(deadline - now)
|
||||||
|
if user_val and pass_val:
|
||||||
|
# 上面 click 应该已经 return,不会到这里
|
||||||
|
print(
|
||||||
|
f"[登录] ✅ 账号密码已填好(user={user_val[:3]}***),"
|
||||||
|
f"请点「登录」按钮,剩余 {remaining}s..."
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
print(
|
||||||
|
f"[登录] 等待用户输入账号密码... 剩余 {remaining}s"
|
||||||
|
)
|
||||||
|
|
||||||
|
time.sleep(_pc.HUMAN_POLL_INTERVAL)
|
||||||
|
|
||||||
|
logger.warning(
|
||||||
|
"platform_collector: %ds 内用户未跳转(最后 URL=%s)", wait_sec, page.url,
|
||||||
|
)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _is_login_page_url_for_user(url: str) -> bool:
|
||||||
|
"""判断 URL 是否还在登录页(用于 _wait_for_login_inputs_filled 退出条件)。"""
|
||||||
|
u = url.lower()
|
||||||
|
if "/login" in u or "/signin" in u or "/auth/" in u:
|
||||||
|
return True
|
||||||
|
# 根域本身也是登录页(demo 数据未提交前的初始态)
|
||||||
|
return u.rstrip("/").endswith("sandbox.jc2009.com")
|
||||||
|
|
||||||
|
|
||||||
|
def _wait_for_manual_login(page, wait_sec: int = HUMAN_WAIT_TIMEOUT) -> bool:
|
||||||
|
"""手工登录 fallback(v1.1.19 重写,v1.1.20 加强,v1.1.21 修订):
|
||||||
|
等用户在浏览器完成登录。
|
||||||
|
|
||||||
|
判定「登录完成」的多个信号(满足任一即视为完成):
|
||||||
|
1. URL 含 `/app/` 且稳定 URL_STABLE_REQUIRED_S 秒(v1.1.19 旧逻辑)
|
||||||
|
2. URL 已离开 `/login` / `/signin` / `/auth/` 等登录路径(v1.1.20 新增)
|
||||||
|
—— 即便 sandbox 未跳到 /app/,只要离开登录页就认为登录成功
|
||||||
|
3. 页面出现 sidebar 元素 `a[href$='/hr/contracts']`(v1.1.20 新增)
|
||||||
|
—— 说明已加载到 SPA 主页
|
||||||
|
|
||||||
|
- **不做任何 goto / reload / click**,避免打断用户输入
|
||||||
|
- 每 HUMAN_PROMPT_INTERVAL 秒打印一次剩余时间(默认 15s)
|
||||||
|
- 每 HUMAN_POLL_INTERVAL 秒检查一次(默认 5s)
|
||||||
|
- 超时抛 LoginTimeoutError。
|
||||||
|
"""
|
||||||
|
# Lazy import: 读 facade constants 与异常类,便于测试 patch
|
||||||
|
from service import platform_collector as _pc
|
||||||
|
|
||||||
|
print(
|
||||||
|
f"[登录] 请在浏览器中完成登录"
|
||||||
|
f"(sandbox demo/123456 或真实账号),最多等待 {wait_sec} 秒..."
|
||||||
|
)
|
||||||
|
deadline = time.monotonic() + wait_sec
|
||||||
|
last_prompt_at = time.monotonic()
|
||||||
|
last_url = page.url
|
||||||
|
last_url_change_at = time.monotonic()
|
||||||
|
|
||||||
|
def _is_login_page_url(url: str) -> bool:
|
||||||
|
u = url.lower()
|
||||||
|
return (
|
||||||
|
"/login" in u or "/signin" in u or "/auth/" in u
|
||||||
|
or u.rstrip("/").endswith("sandbox.jc2009.com")
|
||||||
|
)
|
||||||
|
|
||||||
|
def _sidebar_present() -> bool:
|
||||||
|
try:
|
||||||
|
return page.locator("a[href$='/hr/contracts']").count() > 0
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
while True:
|
||||||
|
remaining = deadline - time.monotonic()
|
||||||
|
if remaining <= 0:
|
||||||
|
break
|
||||||
|
|
||||||
|
cur_url = page.url
|
||||||
|
if cur_url != last_url:
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: URL 变化 %s → %s", last_url, cur_url,
|
||||||
|
)
|
||||||
|
last_url = cur_url
|
||||||
|
last_url_change_at = time.monotonic()
|
||||||
|
|
||||||
|
# 信号 1:URL 跳到 /app/... 并稳定
|
||||||
|
if _pc._is_logged_in(page):
|
||||||
|
stable_for = time.monotonic() - last_url_change_at
|
||||||
|
if stable_for >= _pc.URL_STABLE_REQUIRED_S:
|
||||||
|
print(
|
||||||
|
f"[登录] ✅ 检测到登录完成(URL 含 /app/,稳定 {stable_for:.1f}s)"
|
||||||
|
)
|
||||||
|
return True
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: URL 含 /app/ 但还在过渡,等稳定 "
|
||||||
|
"(current=%s, stable_for=%.1fs)", cur_url, stable_for,
|
||||||
|
)
|
||||||
|
# 信号 2:URL 已离开登录页(即使不在 /app/,比如卡在中间过渡页也算)
|
||||||
|
elif not _is_login_page_url(cur_url):
|
||||||
|
stable_for = time.monotonic() - last_url_change_at
|
||||||
|
if stable_for >= _pc.URL_STABLE_REQUIRED_S:
|
||||||
|
print(
|
||||||
|
f"[登录] ✅ 检测到登录完成(URL 已离开登录页 → {cur_url},"
|
||||||
|
f"稳定 {stable_for:.1f}s)"
|
||||||
|
)
|
||||||
|
return True
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: URL 已离开登录页但还没稳定,等 "
|
||||||
|
"(current=%s, stable_for=%.1fs)", cur_url, stable_for,
|
||||||
|
)
|
||||||
|
# 信号 3:sidebar 已出现
|
||||||
|
elif _sidebar_present():
|
||||||
|
print(
|
||||||
|
f"[登录] ✅ 检测到登录完成(sidebar 已出现,URL={cur_url})"
|
||||||
|
)
|
||||||
|
return True
|
||||||
|
else:
|
||||||
|
# 还在登录页:每 HUMAN_PROMPT_INTERVAL 秒提醒用户一次
|
||||||
|
now = time.monotonic()
|
||||||
|
if now - last_prompt_at >= _pc.HUMAN_PROMPT_INTERVAL:
|
||||||
|
last_prompt_at = now
|
||||||
|
print(
|
||||||
|
f"[登录] 等待用户输入... 剩余 {int(remaining)}s "
|
||||||
|
f"(当前 URL={cur_url})"
|
||||||
|
)
|
||||||
|
|
||||||
|
time.sleep(_pc.HUMAN_POLL_INTERVAL)
|
||||||
|
|
||||||
|
raise _pc.LoginTimeoutError(
|
||||||
|
f"自动登录失败,{wait_sec}s 内用户在浏览器中也未完成登录"
|
||||||
|
f"(最后 URL={page.url})"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _wait_for_url_stable(
|
||||||
|
page, target_substr: str | tuple[str, ...], max_wait_sec: int = 30,
|
||||||
|
) -> bool:
|
||||||
|
"""v1.1.19:等 URL 含 target_substr,且连续 URL_STABLE_REQUIRED_S 秒不变。
|
||||||
|
|
||||||
|
v1.1.22:target_substr 支持 tuple(任一匹配即视为稳定)。
|
||||||
|
|
||||||
|
用于「已登录态直接跳到目标页面」后等待 SPA 路由稳定。
|
||||||
|
**不做任何 goto / reload**,只 listen URL 变化。
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True = URL 已稳定到任一目标;False = 超时。
|
||||||
|
"""
|
||||||
|
# Lazy import: 读 URL_STABLE_REQUIRED_S 便于测试 patch
|
||||||
|
from service import platform_collector as _pc
|
||||||
|
|
||||||
|
if isinstance(target_substr, str):
|
||||||
|
targets = (target_substr,)
|
||||||
|
else:
|
||||||
|
targets = target_substr
|
||||||
|
|
||||||
|
deadline = time.monotonic() + max_wait_sec
|
||||||
|
last_url = page.url
|
||||||
|
last_url_change_at = time.monotonic()
|
||||||
|
|
||||||
|
while time.monotonic() < deadline:
|
||||||
|
cur_url = page.url
|
||||||
|
if cur_url != last_url:
|
||||||
|
logger.info(
|
||||||
|
"platform_collector._wait_for_url_stable: URL 变化 %s → %s",
|
||||||
|
last_url, cur_url,
|
||||||
|
)
|
||||||
|
last_url = cur_url
|
||||||
|
last_url_change_at = time.monotonic()
|
||||||
|
|
||||||
|
cur_lower = cur_url.lower()
|
||||||
|
if any(t.lower() in cur_lower for t in targets):
|
||||||
|
stable_for = time.monotonic() - last_url_change_at
|
||||||
|
if stable_for >= _pc.URL_STABLE_REQUIRED_S:
|
||||||
|
return True
|
||||||
|
|
||||||
|
time.sleep(URL_STABLE_POLL_S)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _fill_root_login(page) -> bool:
|
||||||
|
"""v1.1.31:根域登录页(如还在):从 .env 读 SMARTCAMPUS_LOGIN_USER /
|
||||||
|
SMARTCAMPUS_LOGIN_PASSWORD(不硬编码 demo/123456)→ fill → click Sign in。
|
||||||
|
|
||||||
|
无 .env 凭证时 return False,调用方走「用户手工登录」分支。
|
||||||
|
"""
|
||||||
|
# Lazy import: 读 _get_env(保留在 facade,测试通过 pc._ENV_CACHE 隔离)
|
||||||
|
from service import platform_collector as _pc
|
||||||
|
|
||||||
|
user_input = page.locator('#jc-user').first
|
||||||
|
if user_input.count() == 0:
|
||||||
|
return False
|
||||||
|
|
||||||
|
root_user = _pc._get_env("SMARTCAMPUS_LOGIN_USER", "").strip()
|
||||||
|
root_pass = _pc._get_env("SMARTCAMPUS_LOGIN_PASSWORD", "").strip()
|
||||||
|
if not root_user or not root_pass:
|
||||||
|
logger.info(
|
||||||
|
"platform_collector._fill_root_login: .env 未配置 "
|
||||||
|
"SMARTCAMPUS_LOGIN_USER / SMARTCAMPUS_LOGIN_PASSWORD,"
|
||||||
|
"**不自动 fill**,等用户手工输入"
|
||||||
|
)
|
||||||
|
return False
|
||||||
|
|
||||||
|
user_input.fill(root_user)
|
||||||
|
page.locator('#jc-pass').first.fill(root_pass)
|
||||||
|
page.locator('button[type="submit"]:has-text("Sign in")').first.click()
|
||||||
|
logger.info("platform_collector: 根域自动登录(user 来自 .env)...")
|
||||||
|
try:
|
||||||
|
page.wait_for_url(
|
||||||
|
lambda u: "/home" in u or "/select" in u or "/industries/" in u,
|
||||||
|
timeout=URL_AFTER_LOGIN_TIMEOUT_MS,
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning("等根域登录跳转超时: %s", e)
|
||||||
|
logger.info("platform_collector: 根域登录后 URL=%s", page.url)
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def _fill_spa_login_from_env(page) -> bool:
|
||||||
|
"""v1.1.31/32:SPA 二级登录页(/industries/campus/smartcampus/login)
|
||||||
|
按用户要求「密码不能直接放到代码」—— 从 .env 读 SMARTCAMPUS_LOGIN_USER /
|
||||||
|
SMARTCAMPUS_LOGIN_PASSWORD(无值时不自动 fill,等用户手工输入)。
|
||||||
|
|
||||||
|
v1.1.32:sandbox SPA 登录页默认会把「教师(通用)」demo 凭据预填到主表单 input。
|
||||||
|
但用户实际要登录的可能是「人事处」「财务处」等不同行业 tile;
|
||||||
|
按用户截图场景,需先在右侧 DEV MODE 面板里点对应 tile 的 Sign in,
|
||||||
|
让 SPA 把对应行业的 demo 凭据搬到主表单,再走原 fill+click 流程。
|
||||||
|
行业优先级:.env SMARTCAMPUS_LOGIN_INDUSTRY(默认 hr)→ 探测页面已预填的 user 值。
|
||||||
|
|
||||||
|
selector 优先级:
|
||||||
|
- user: `#jc-user` → `input[type='tel']` → `input[autocomplete='username']`
|
||||||
|
- pass: `#jc-pass` → `input[type='password']` → `input[autocomplete='current-password']`
|
||||||
|
按钮文本优先级:Sign in → 登录 → Login → signin → submit[type]。
|
||||||
|
|
||||||
|
返回是否成功点登录按钮(**无 .env 凭证时返回 False,让用户手工输入**)。
|
||||||
|
"""
|
||||||
|
# Lazy import: 读 _get_env(facade 持有 _ENV_CACHE)
|
||||||
|
from service import platform_collector as _pc
|
||||||
|
|
||||||
|
# 1. 从 .env 读凭证(不硬编码)
|
||||||
|
spa_user = _pc._get_env("SMARTCAMPUS_LOGIN_USER", "").strip()
|
||||||
|
spa_pass = _pc._get_env("SMARTCAMPUS_LOGIN_PASSWORD", "").strip()
|
||||||
|
if not spa_user or not spa_pass:
|
||||||
|
logger.info(
|
||||||
|
"platform_collector._fill_spa_login_from_env: .env 未配置 "
|
||||||
|
"SMARTCAMPUS_LOGIN_USER / SMARTCAMPUS_LOGIN_PASSWORD,"
|
||||||
|
"**不自动 fill**,等用户在浏览器手工输入"
|
||||||
|
)
|
||||||
|
return False
|
||||||
|
|
||||||
|
# 1.5 v1.1.32:检测 DEV MODE 面板是否展开 → 是否需要先点对应行业的 Sign in
|
||||||
|
# sandbox 行为:教师(通用)demo 总是预填到主表单,登录其他行业(hr/finance/research)
|
||||||
|
# 需要在右侧 DEV MODE 面板里点对应 tile 的 Sign in。
|
||||||
|
if _is_dev_mode_panel_open(page):
|
||||||
|
target_industry = _resolve_target_industry(page)
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: DEV MODE 面板展开,目标行业=%s", target_industry,
|
||||||
|
)
|
||||||
|
_click_dev_mode_sign_in_for_industry(page, target_industry)
|
||||||
|
|
||||||
|
# 2. 找 user input
|
||||||
|
user_loc = None
|
||||||
|
for sel in ("#jc-user", "input[type='tel']", "input[autocomplete='username']"):
|
||||||
|
loc = page.locator(sel).first
|
||||||
|
if loc.count() > 0:
|
||||||
|
user_loc = loc
|
||||||
|
break
|
||||||
|
if user_loc is None:
|
||||||
|
logger.warning(
|
||||||
|
"platform_collector._fill_spa_login_from_env: 找不到 user input "
|
||||||
|
"(#jc-user / tel / autocomplete=username 全 miss)"
|
||||||
|
)
|
||||||
|
return False
|
||||||
|
|
||||||
|
# 3. 找 pass input
|
||||||
|
pass_loc = None
|
||||||
|
for sel in ("#jc-pass", "input[type='password']", "input[autocomplete='current-password']"):
|
||||||
|
loc = page.locator(sel).first
|
||||||
|
if loc.count() > 0:
|
||||||
|
pass_loc = loc
|
||||||
|
break
|
||||||
|
if pass_loc is None:
|
||||||
|
logger.warning(
|
||||||
|
"platform_collector._fill_spa_login_from_env: 找不到 pass input"
|
||||||
|
)
|
||||||
|
return False
|
||||||
|
|
||||||
|
user_loc.fill(spa_user)
|
||||||
|
pass_loc.fill(spa_pass)
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: SPA 二级登录页 fill 完成(user 来自 .env)"
|
||||||
|
)
|
||||||
|
|
||||||
|
# 4. 找登录按钮
|
||||||
|
btn = None
|
||||||
|
btn_text_used = ""
|
||||||
|
for txt in ("Sign in", "登录", "Login", "signin"):
|
||||||
|
loc = page.locator(f"button:has-text('{txt}')").first
|
||||||
|
if loc.count() > 0:
|
||||||
|
btn = loc
|
||||||
|
btn_text_used = txt
|
||||||
|
break
|
||||||
|
if btn is None:
|
||||||
|
btn = page.locator("button[type='submit']").first
|
||||||
|
btn_text_used = "submit"
|
||||||
|
if btn is None or btn.count() == 0:
|
||||||
|
logger.warning(
|
||||||
|
"platform_collector._fill_spa_login_from_env: 找不到登录按钮"
|
||||||
|
)
|
||||||
|
return False
|
||||||
|
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: SPA 二级登录页点 %s 按钮(user 来自 .env)",
|
||||||
|
btn_text_used,
|
||||||
|
)
|
||||||
|
btn.click()
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def _is_dev_mode_panel_open(page) -> bool:
|
||||||
|
"""v1.1.32:检测 sandbox SPA 登录页右侧 DEV MODE 面板是否展开。
|
||||||
|
|
||||||
|
面板里有「DEV MODE」文字 + 多个 tile(教师通用 / 人事处 / 财务处 / 科研处),
|
||||||
|
每个 tile 有自己的 Fill + Sign in 按钮。
|
||||||
|
|
||||||
|
返回 True 表示面板可见且可点对应行业的 Sign in。
|
||||||
|
|
||||||
|
实现策略(避免依赖 inner_text——fake page mock 不一定支持):
|
||||||
|
- 用 selector 探测至少 2 个 Sign in 按钮(教师/人事处/财务处/科研处 tile)
|
||||||
|
- 至少 1 个含「DEV MODE」文本的元素
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
# 1. 至少 2 个 Sign in 按钮(教师通用 + 至少 1 个行业 tile)
|
||||||
|
sign_in_btns = page.locator("button:has-text('Sign in')")
|
||||||
|
if sign_in_btns.count() < 2:
|
||||||
|
return False
|
||||||
|
# 2. DEV MODE 标志:找含 'DEV MODE' 文本的元素
|
||||||
|
dev_marker = page.locator(
|
||||||
|
"*:has-text('DEV MODE'), *:text-is('DEV MODE')"
|
||||||
|
).first
|
||||||
|
return dev_marker.count() > 0
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_target_industry(page) -> str:
|
||||||
|
"""v1.1.32:解析要登录的目标行业。
|
||||||
|
|
||||||
|
优先级:
|
||||||
|
1. .env SMARTCAMPUS_LOGIN_INDUSTRY(可选,默认 'hr')
|
||||||
|
2. 探测页面:若主表单已预填 user 是 'hr' / 'finance' / 'research' / 'demo'
|
||||||
|
→ 直接用该值(说明用户已点过对应 tile 的 Fill)
|
||||||
|
"""
|
||||||
|
# Lazy import: 读 _get_env 拿 SMARTCAMPUS_LOGIN_INDUSTRY
|
||||||
|
from service import platform_collector as _pc
|
||||||
|
|
||||||
|
explicit = _pc._get_env("SMARTCAMPUS_LOGIN_INDUSTRY", "").strip().lower()
|
||||||
|
if explicit:
|
||||||
|
return explicit
|
||||||
|
# fallback:探测主表单 input 已预填的 user 值
|
||||||
|
try:
|
||||||
|
for sel in ("#jc-user", "input[type='tel']", "input[autocomplete='username']"):
|
||||||
|
loc = page.locator(sel).first
|
||||||
|
if loc.count() > 0:
|
||||||
|
cur = (loc.input_value() or "").strip().lower()
|
||||||
|
if cur in ("hr", "finance", "research", "demo"):
|
||||||
|
return cur
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return "hr"
|
||||||
|
|
||||||
|
|
||||||
|
def _click_dev_mode_sign_in_for_industry(page, industry: str) -> bool:
|
||||||
|
"""v1.1.32:在 DEV MODE 面板里点对应行业 tile 的 Sign in 按钮。
|
||||||
|
|
||||||
|
sandbox 行为:每个 tile 是一段包含「行业名 + Email + Password + Fill + Sign in」的 div,
|
||||||
|
用户点 Sign in 会让对应行业的 demo 凭据填到主表单。
|
||||||
|
|
||||||
|
selector 策略(简化版,避免依赖 inner_text——fake page mock 不一定支持):
|
||||||
|
- 行业显示名 → 用 :has-text + Sign in 按钮组合定位
|
||||||
|
- 兜底:用 nth 索引(教师通用=1 / 人事处=2 / 财务处=3 / 科研处=4)
|
||||||
|
|
||||||
|
返回是否成功点击(找不到时返回 False,由调用方决定 fallback)。
|
||||||
|
"""
|
||||||
|
if not industry:
|
||||||
|
return False
|
||||||
|
|
||||||
|
# 行业显示名 → 可能的中文标签
|
||||||
|
industry_aliases = {
|
||||||
|
"hr": ("人事处",),
|
||||||
|
"finance": ("财务处",),
|
||||||
|
"research": ("科研处",),
|
||||||
|
"demo": ("教师", "通用", "Teacher"),
|
||||||
|
}
|
||||||
|
aliases = industry_aliases.get(industry, (industry,))
|
||||||
|
|
||||||
|
# 1. 模式 A:找包含「行业中文标签」的容器 + 容器内 Sign in 按钮
|
||||||
|
for alias in aliases:
|
||||||
|
try:
|
||||||
|
# Playwright 支持 :has() / :has-text() / :text-matches()
|
||||||
|
tile_btn = page.locator(
|
||||||
|
f"div:has-text('{alias}'):has(button:has-text('Sign in')) "
|
||||||
|
f"button:has-text('Sign in')"
|
||||||
|
).first
|
||||||
|
if tile_btn.count() > 0:
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: DEV MODE 点 %s tile 的 Sign in 按钮",
|
||||||
|
alias,
|
||||||
|
)
|
||||||
|
tile_btn.click()
|
||||||
|
return True
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(
|
||||||
|
"platform_collector: DEV MODE %s Sign in 点击失败: %s", alias, e,
|
||||||
|
)
|
||||||
|
|
||||||
|
# 2. 兜底:直接找所有 Sign in 按钮,按 industry 顺序选
|
||||||
|
# hr(人事处) 排第 2 个,finance(财务处) 排第 3 个,research(科研处) 排第 4 个
|
||||||
|
industry_order = {"demo": 1, "hr": 2, "finance": 3, "research": 4}
|
||||||
|
if industry in industry_order:
|
||||||
|
idx = industry_order[industry]
|
||||||
|
try:
|
||||||
|
all_sign_in = page.locator("button:has-text('Sign in')")
|
||||||
|
cnt = all_sign_in.count()
|
||||||
|
if cnt >= idx:
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: DEV MODE 用兜底策略点第 %d 个 Sign in(%s)",
|
||||||
|
idx, industry,
|
||||||
|
)
|
||||||
|
all_sign_in.nth(idx - 1).click()
|
||||||
|
return True
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(
|
||||||
|
"platform_collector: DEV MODE 兜底点击失败: %s", e,
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.warning(
|
||||||
|
"platform_collector: DEV MODE 找不到 industry=%s 的 Sign in 按钮", industry,
|
||||||
|
)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _click_campus_link(page) -> bool:
|
||||||
|
"""在 /select 页面点智慧校园 tile,返回是否点中。"""
|
||||||
|
link = page.locator("a[href*='/industries/campus']").first
|
||||||
|
if link.count() == 0:
|
||||||
|
logger.warning("platform_collector: 未找到 智慧校园 link")
|
||||||
|
return False
|
||||||
|
link.click()
|
||||||
|
logger.info("platform_collector: click 智慧校园 链接")
|
||||||
|
try:
|
||||||
|
page.wait_for_url(lambda u: "/industries/campus" in u, timeout=SIDEBAR_LINK_TIMEOUT_MS // 2)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning("等 sandbox 路径超时: %s;当前 URL=%s", e, page.url)
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def _fill_spa_login(page, settle_ms: int = SPA_LOGIN_SETTLE_MS) -> bool:
|
||||||
|
"""sandbox SPA 自动登录(如还在登录页):fill demo/123456 → click 登录 → 等跳转。
|
||||||
|
|
||||||
|
返回是否执行了登录(True = 已点登录按钮且 URL 已跳离 login;调用方可继续后续流程)。
|
||||||
|
失败时**不抛异常**,由调用方决定是否 fallback 到 _wait_for_manual_login。
|
||||||
|
|
||||||
|
v1.1.10 关键修复:**先检测 _is_logged_in,已登录态直接 return True 不再 fill**。
|
||||||
|
之前没这个检查,profile 复用时会被强制 demo fill 破坏 session。
|
||||||
|
|
||||||
|
v1.1.11 关键修复:**sandbox SPA 登录 input 实际是 `type="tel"` + 无 autocomplete**,
|
||||||
|
之前 selector `input[autocomplete="username"]` 永远找不到,自动 fill 100% 失败。
|
||||||
|
改用 sandbox 实际 selector `#jc-user` / `#jc-pass`(来自 root 登录页)+ `button[type="submit"]`。
|
||||||
|
|
||||||
|
修复要点(v1.1.5/v1.1.6 登录时间太短导致采集不到数据):
|
||||||
|
- 默认 settle_ms = SPA_LOGIN_SETTLE_MS(4000ms),给 SPA 异步渲染 + token 写入 cookie 时间
|
||||||
|
- 等 username input 的 timeout = SPA_LOGIN_INPUT_TIMEOUT_MS(20s)
|
||||||
|
- click 按钮后 wait_for_url timeout = SPA_LOGIN_URL_TIMEOUT_MS(30s)
|
||||||
|
- 多处过渡路径(/login, /signin, /auth/)一并判断
|
||||||
|
"""
|
||||||
|
# Lazy import: 读 _is_logged_in(facade 持有,便于测试 mock)
|
||||||
|
from service import platform_collector as _pc
|
||||||
|
|
||||||
|
# v1.1.10:先检查是否已登录(profile cookie 复用场景),已登录直接 return True
|
||||||
|
if _pc._is_logged_in(page):
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: 已登录态(profile cookie 复用),跳过自动 fill"
|
||||||
|
)
|
||||||
|
return True
|
||||||
|
|
||||||
|
url_lower = page.url.lower()
|
||||||
|
if "/login" not in url_lower and "/signin" not in url_lower and "/auth/" not in url_lower:
|
||||||
|
return False
|
||||||
|
# v1.1.11:sandbox SPA 登录页 input 是 #jc-user / #jc-pass(与 root 登录页同 ID)
|
||||||
|
# 等输入框出现(带超时,避免 sandbox mock 慢启动卡死)
|
||||||
|
try:
|
||||||
|
page.wait_for_selector(
|
||||||
|
'#jc-user, input[type="tel"], input[autocomplete="username"]',
|
||||||
|
timeout=SPA_LOGIN_INPUT_TIMEOUT_MS,
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning("SPA login input 未出现: %s;当前 URL=%s", e, page.url)
|
||||||
|
return False
|
||||||
|
# v1.1.11:按优先级选 selector —— sandbox 实际用 #jc-user
|
||||||
|
user_sel = None
|
||||||
|
for sel in ('#jc-user', 'input[type="tel"]', 'input[autocomplete="username"]'):
|
||||||
|
loc = page.locator(sel).first
|
||||||
|
if loc.count() > 0:
|
||||||
|
user_sel = sel
|
||||||
|
break
|
||||||
|
if not user_sel:
|
||||||
|
logger.warning("SPA login input selector 全 miss(#jc-user / type=tel / autocomplete=username)")
|
||||||
|
return False
|
||||||
|
page.locator(user_sel).first.fill("demo")
|
||||||
|
# 密码 selector 同样按优先级
|
||||||
|
pass_sel = None
|
||||||
|
for sel in ('#jc-pass', 'input[type="password"]', 'input[autocomplete="current-password"]'):
|
||||||
|
loc = page.locator(sel).first
|
||||||
|
if loc.count() > 0:
|
||||||
|
pass_sel = sel
|
||||||
|
break
|
||||||
|
if not pass_sel:
|
||||||
|
logger.warning("SPA password input selector 全 miss")
|
||||||
|
return False
|
||||||
|
page.locator(pass_sel).first.fill("123456")
|
||||||
|
page.locator(
|
||||||
|
'button:has-text("登录"), button[type="submit"]:has-text("Sign in"), '
|
||||||
|
'button[type="submit"]'
|
||||||
|
).first.click()
|
||||||
|
logger.info("platform_collector: sandbox SPA 自动登录 (user_sel=%s pass_sel=%s)", user_sel, pass_sel)
|
||||||
|
page.wait_for_timeout(settle_ms)
|
||||||
|
try:
|
||||||
|
page.wait_for_url(
|
||||||
|
lambda u: "/login" not in u.lower()
|
||||||
|
and "/signin" not in u.lower()
|
||||||
|
and "/auth/" not in u.lower(),
|
||||||
|
timeout=SPA_LOGIN_URL_TIMEOUT_MS,
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning("SPA 登录超时: %s;当前 URL=%s", e, page.url)
|
||||||
|
return False
|
||||||
|
# v1.1.10:fill 后再检测 _is_logged_in,避免 sandbox demo 凭据失效时
|
||||||
|
# 被误判为「已登录」(URL 跳 /app/home 但 cookie 没写入)
|
||||||
|
if not _pc._is_logged_in(page):
|
||||||
|
logger.warning(
|
||||||
|
"platform_collector: SPA URL 跳离 login 但 _is_logged_in=False,登录可能失败"
|
||||||
|
)
|
||||||
|
return False
|
||||||
|
logger.info("platform_collector: SPA 登录后 URL=%s", page.url)
|
||||||
|
return True
|
||||||
213
scripts/service/platform_collector_persist.py
Normal file
213
scripts/service/platform_collector_persist.py
Normal file
@@ -0,0 +1,213 @@
|
|||||||
|
"""Playwright RPA:SmartCampus HR 平台合同台账落库 + 下载队列入队。
|
||||||
|
|
||||||
|
v1.1.31:从 platform_collector.py 拆分出来的 DB 持久化子模块。
|
||||||
|
包含 2 个函数:
|
||||||
|
- _persist_contracts(contracts 表 upsert + materials_content 写入 + sandbox_card 性能解析)
|
||||||
|
- _enqueue_download_tasks_safe(download_tasks 表入队,idempotent auto-create-table)
|
||||||
|
|
||||||
|
设计要点(保持 v1.1.30 不变):
|
||||||
|
- 纯 DB 模块,无 Playwright 依赖
|
||||||
|
- 顶部 import db.repository / db.connection / db.migrations
|
||||||
|
- 异常处理返回 dict(_enqueue)而非 raise,便于 collect_all 容错
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sqlite3
|
||||||
|
from datetime import datetime
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from db import repository as repo
|
||||||
|
from db.connection import get_connection
|
||||||
|
from db.migrations import ensure_materials_content_schema
|
||||||
|
from util.logging_setup import get_skill_logger
|
||||||
|
|
||||||
|
logger = get_skill_logger()
|
||||||
|
|
||||||
|
|
||||||
|
def _persist_contracts(contracts: list[dict]) -> int:
|
||||||
|
"""把抓到的合同 + 材料落到 SQLite。仅插入 contracts 表的占位行,
|
||||||
|
materials_content 由 parse-materials CLI 后续解析写入。
|
||||||
|
|
||||||
|
真正的合同台账来自 HR 系统 API(生产)或样例 CSV(本地),
|
||||||
|
本采集仅作为"快照"记录抓取时间与材料路径。
|
||||||
|
"""
|
||||||
|
if not contracts:
|
||||||
|
return 0
|
||||||
|
|
||||||
|
conn = get_connection()
|
||||||
|
try:
|
||||||
|
with conn:
|
||||||
|
for c in contracts:
|
||||||
|
# upsert contracts(仅更新与 sandbox 相关的字段)
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO contracts
|
||||||
|
(contract_id, employee_id, teacher_name, unit_id, unit_name,
|
||||||
|
position, contract_type, contract_start, contract_end,
|
||||||
|
trial_period_end, status, last_material_id, renewed_at)
|
||||||
|
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, '可续聘', NULL, NULL)
|
||||||
|
ON CONFLICT(contract_id) DO UPDATE SET
|
||||||
|
teacher_name = excluded.teacher_name,
|
||||||
|
unit_name = excluded.unit_name,
|
||||||
|
contract_end = excluded.contract_end
|
||||||
|
""",
|
||||||
|
(
|
||||||
|
c["contract_id"],
|
||||||
|
c["contract_id"], # employee_id 暂用 contract_id
|
||||||
|
c["teacher_name"],
|
||||||
|
"U-RPA", # unit_id 占位
|
||||||
|
c["unit_name"],
|
||||||
|
"(RPA 采集)",
|
||||||
|
"(RPA 采集)",
|
||||||
|
"1970-01-01",
|
||||||
|
c["contract_end"],
|
||||||
|
c["contract_end"],
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
# 写 materials_content(sandbox card 文本 + parse-materials 解析后文件路径)
|
||||||
|
ensure_materials_content_schema(conn)
|
||||||
|
|
||||||
|
for m in c["materials"]:
|
||||||
|
if isinstance(m, dict) and m.get("kind") == "sandbox_card":
|
||||||
|
# sandbox mock 卡片文本,直接写入 extracted_text
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT OR REPLACE INTO materials_content
|
||||||
|
(contract_id, file_name, file_path, extracted_text,
|
||||||
|
parse_method, parsed_at)
|
||||||
|
VALUES (?, ?, ?, ?, 'sandbox_card', ?)
|
||||||
|
""",
|
||||||
|
(
|
||||||
|
c["contract_id"],
|
||||||
|
m["file_name"],
|
||||||
|
f"sandbox://hr/contracts/{c['contract_id']}",
|
||||||
|
m["text"],
|
||||||
|
datetime.now().isoformat(timespec="seconds"),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
# 同步解析 → 抽 year + 等次 → 写入 performance_records
|
||||||
|
perf_n = repo.parse_sandbox_card_performance(
|
||||||
|
conn, c["contract_id"], m["text"],
|
||||||
|
)
|
||||||
|
if perf_n > 0:
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: %s 解析卡片 → performance %d 条",
|
||||||
|
c["contract_id"], perf_n,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# 本地文件路径(parse-materials 解析后写入)
|
||||||
|
fp = m if isinstance(m, str) else m.get("file_path")
|
||||||
|
fname = Path(fp).name if fp else "unknown"
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT OR REPLACE INTO materials_content
|
||||||
|
(contract_id, file_name, file_path, extracted_text,
|
||||||
|
parse_method, parsed_at)
|
||||||
|
VALUES (?, ?, ?, NULL, 'pending', ?)
|
||||||
|
""",
|
||||||
|
(
|
||||||
|
c["contract_id"],
|
||||||
|
fname,
|
||||||
|
fp,
|
||||||
|
datetime.now().isoformat(timespec="seconds"),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
# v1.1.18 入队 download_tasks 已移到 collect_all 末尾的独立事务,
|
||||||
|
# 见 _enqueue_download_tasks_safe()
|
||||||
|
return len(contracts)
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
|
||||||
|
def _enqueue_download_tasks_safe(
|
||||||
|
contract_ids: list[str], conn: sqlite3.Connection | None = None,
|
||||||
|
) -> dict:
|
||||||
|
"""v1.1.18:每个合同入队 download_tasks。
|
||||||
|
|
||||||
|
在独立事务里完成,不影响 _persist_contracts 主流程(contracts / materials_content)。
|
||||||
|
任一入队失败 → 记 warning + 计入 failed,但不影响整体返回。
|
||||||
|
|
||||||
|
conn: 可选传入连接(用于测试 in-memory DB)。None → 走 get_connection()。
|
||||||
|
若传入 conn,调用方负责生命周期(不会 close)。
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
{enqueued: N, skipped: M, failed: K, errors: [...]}
|
||||||
|
"""
|
||||||
|
if not contract_ids:
|
||||||
|
return {"enqueued": 0, "skipped": 0, "failed": 0, "errors": []}
|
||||||
|
|
||||||
|
enqueued = 0
|
||||||
|
skipped = 0
|
||||||
|
failed = 0
|
||||||
|
errors: list[str] = []
|
||||||
|
|
||||||
|
owns_conn = conn is None
|
||||||
|
if owns_conn:
|
||||||
|
conn = get_connection()
|
||||||
|
# v1.1.29:idempotent 表创建 —— 用户没跑 init-db 时,download_tasks 表不存在,
|
||||||
|
# 首次采集直接建表(CREATE TABLE IF NOT EXISTS)而不是报 "no such table"
|
||||||
|
try:
|
||||||
|
conn.executescript(
|
||||||
|
"""
|
||||||
|
CREATE TABLE IF NOT EXISTS download_tasks (
|
||||||
|
task_id TEXT PRIMARY KEY,
|
||||||
|
contract_id TEXT NOT NULL,
|
||||||
|
kind TEXT NOT NULL DEFAULT 'platform_button',
|
||||||
|
status TEXT NOT NULL DEFAULT 'pending',
|
||||||
|
attempts INTEGER NOT NULL DEFAULT 0,
|
||||||
|
error_msg TEXT,
|
||||||
|
enqueued_at TEXT NOT NULL,
|
||||||
|
started_at TEXT,
|
||||||
|
finished_at TEXT,
|
||||||
|
source_path TEXT,
|
||||||
|
result_path TEXT,
|
||||||
|
FOREIGN KEY (contract_id) REFERENCES contracts(contract_id)
|
||||||
|
);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_dlq_status ON download_tasks(status);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_dlq_contract ON download_tasks(contract_id);
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
except Exception as ensure_err:
|
||||||
|
logger.warning(
|
||||||
|
"platform_collector: ensure download_tasks 表创建失败(继续走入队流程): %s",
|
||||||
|
ensure_err,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
for cid in contract_ids:
|
||||||
|
try:
|
||||||
|
# 入队前查一下是否已存在 pending/running → 决定计入 skipped 还是 enqueued
|
||||||
|
pre_existing = conn.execute(
|
||||||
|
"""SELECT 1 FROM download_tasks
|
||||||
|
WHERE contract_id = ? AND kind = ?
|
||||||
|
AND status IN ('pending', 'running') LIMIT 1""",
|
||||||
|
(cid, "platform_button"),
|
||||||
|
).fetchone()
|
||||||
|
repo.enqueue_download_task(conn, cid, "platform_button")
|
||||||
|
if pre_existing:
|
||||||
|
skipped += 1
|
||||||
|
else:
|
||||||
|
enqueued += 1
|
||||||
|
except Exception as e:
|
||||||
|
failed += 1
|
||||||
|
err = f"{cid}: {type(e).__name__}: {e}"
|
||||||
|
errors.append(err)
|
||||||
|
logger.warning(
|
||||||
|
"platform_collector.enqueue: 入队 %s 失败: %s", cid, e,
|
||||||
|
)
|
||||||
|
if owns_conn:
|
||||||
|
conn.commit()
|
||||||
|
finally:
|
||||||
|
if owns_conn:
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
logger.info(
|
||||||
|
"platform_collector.enqueue: 入队结果 enqueued=%d skipped=%d failed=%d (total=%d)",
|
||||||
|
enqueued, skipped, failed, len(contract_ids),
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"enqueued": enqueued, "skipped": skipped,
|
||||||
|
"failed": failed, "errors": errors[:10],
|
||||||
|
}
|
||||||
333
scripts/service/platform_collector_queue.py
Normal file
333
scripts/service/platform_collector_queue.py
Normal file
@@ -0,0 +1,333 @@
|
|||||||
|
"""Playwright RPA:SmartCampus HR 平台下载队列处理 + 合同 ID 解析 helpers。
|
||||||
|
|
||||||
|
v1.1.31:从 platform_collector.py 拆分出来的队列处理子模块。
|
||||||
|
包含 7 个函数:
|
||||||
|
- _download_materials_for_row(点续聘材料包按钮抓卡片文本)
|
||||||
|
- _find_row_for_contract(在 hr/contracts 表格找 contract_id 行号)
|
||||||
|
- process_queue(领取 download_tasks 队列、登录、抓材料、落库)
|
||||||
|
- _gen_contract_id / _normalize_date / _parse_days / _parse_alert(4 个纯解析 helper)
|
||||||
|
|
||||||
|
设计要点(保持 v1.1.30 不变):
|
||||||
|
- 登录子流程通过 service.platform_collector_login 顶部 import
|
||||||
|
- facade 持有的符号(_goto / _is_logged_in / 异常类 / SPA_LOGIN_URL 等)
|
||||||
|
通过 `from service import platform_collector as _pc` lazy import
|
||||||
|
- 4 个 helper 函数(_gen_contract_id / _normalize_date / _parse_days / _parse_alert)
|
||||||
|
在本模块内自包含,collect / persist / queue 都可能用到(通过 facade re-export 访问)
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
import sqlite3
|
||||||
|
from datetime import datetime
|
||||||
|
|
||||||
|
from db import repository as repo
|
||||||
|
from db.connection import get_connection
|
||||||
|
from db.migrations import ensure_materials_content_schema
|
||||||
|
from util.constants import (
|
||||||
|
CARD_APPEAR_WAIT_MS,
|
||||||
|
DEFAULT_PAGE_TIMEOUT_MS,
|
||||||
|
HUMAN_WAIT_TIMEOUT,
|
||||||
|
POST_LOGIN_SETTLE_MS,
|
||||||
|
SPA_HOME_RETRY_DELAY_MS,
|
||||||
|
SPA_HOME_RETRY_MAX,
|
||||||
|
SPA_HYDRATION_MS,
|
||||||
|
TABLE_INITIAL_WAIT_MS,
|
||||||
|
TABLE_POLL_INTERVAL_MS,
|
||||||
|
TABLE_POLL_MAX_ATTEMPTS,
|
||||||
|
TABLE_TR_TIMEOUT_MS,
|
||||||
|
)
|
||||||
|
from util.logging_setup import get_skill_logger
|
||||||
|
from service.platform_collector_login import (
|
||||||
|
_fill_spa_login,
|
||||||
|
_wait_for_manual_login,
|
||||||
|
_wait_for_url_stable,
|
||||||
|
)
|
||||||
|
|
||||||
|
logger = get_skill_logger()
|
||||||
|
|
||||||
|
|
||||||
|
def _download_materials_for_row(
|
||||||
|
page, row_idx: int, contract_id: str, mat_btn_count: int,
|
||||||
|
) -> list[dict]:
|
||||||
|
"""点续聘材料包按钮抓卡片文本(与 v1.1.17 collect_all 内联逻辑等价)。
|
||||||
|
|
||||||
|
返回 materials 列表(kind=sandbox_card,含 file_name + text)。
|
||||||
|
"""
|
||||||
|
if mat_btn_count == 0:
|
||||||
|
return []
|
||||||
|
materials: list[dict] = []
|
||||||
|
try:
|
||||||
|
btn = page.locator("table tbody tr").nth(row_idx).locator(
|
||||||
|
"button:has-text('续聘材料包')"
|
||||||
|
)
|
||||||
|
btn.click()
|
||||||
|
page.wait_for_timeout(CARD_APPEAR_WAIT_MS)
|
||||||
|
cards_text = page.locator(
|
||||||
|
"main .grid > div, main > div > div > div"
|
||||||
|
).all_inner_texts()
|
||||||
|
materials_text = "\n\n".join(t.strip() for t in cards_text if t.strip())
|
||||||
|
if materials_text:
|
||||||
|
materials.append({
|
||||||
|
"kind": "sandbox_card",
|
||||||
|
"file_name": f"{contract_id}_materials.txt",
|
||||||
|
"text": materials_text,
|
||||||
|
})
|
||||||
|
logger.info(
|
||||||
|
"platform_collector: 抓到卡片文本 %d 字 (%s)",
|
||||||
|
len(materials_text), contract_id,
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning("抓卡片失败 %s: %s", contract_id, e)
|
||||||
|
return materials
|
||||||
|
|
||||||
|
|
||||||
|
def _find_row_for_contract(page, contract_id: str) -> int | None:
|
||||||
|
"""在 hr/contracts 表格中找 contract_id 所在行号。
|
||||||
|
|
||||||
|
支持 7 列(用 teacher_name 派生 contract_id)与 10 列(用 emp_id)。
|
||||||
|
不匹配返回 None(调用方 mark_task_failed)。
|
||||||
|
"""
|
||||||
|
rows = page.eval_on_selector_all(
|
||||||
|
"table tbody tr",
|
||||||
|
"rs => rs.map(r => Array.from(r.querySelectorAll('td')).map(td => td.innerText))",
|
||||||
|
)
|
||||||
|
for i, r in enumerate(rows):
|
||||||
|
ncols = len(r)
|
||||||
|
if ncols < 6:
|
||||||
|
continue
|
||||||
|
first_col = r[0].strip()
|
||||||
|
is_emp_id = bool(re.match(r"^[A-Za-z]?\d+$", first_col))
|
||||||
|
if is_emp_id and ncols >= 9:
|
||||||
|
emp_id, contract_end = first_col, r[5].split("~")[-1].strip()
|
||||||
|
elif ncols >= 6:
|
||||||
|
emp_id, contract_end = first_col, r[2].strip()
|
||||||
|
else:
|
||||||
|
continue
|
||||||
|
candidate = _gen_contract_id(emp_id, contract_end)
|
||||||
|
if candidate == contract_id:
|
||||||
|
return i
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def process_queue(
|
||||||
|
limit: int = 5,
|
||||||
|
headless: bool = True,
|
||||||
|
human_wait_timeout: int = HUMAN_WAIT_TIMEOUT,
|
||||||
|
ids_filter: str = "",
|
||||||
|
) -> dict:
|
||||||
|
"""从 download_tasks 队列取 pending 任务,对每个合同重新走材料抓取。
|
||||||
|
|
||||||
|
与 collect_all 重复登录流程,但每行只点「续聘材料包」按钮抓卡片,不重新抓合同表。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
limit: 本次最多处理任务数(0 = 不限)
|
||||||
|
ids_filter: 逗号分隔 task_id 列表(只处理这些 pending)
|
||||||
|
headless: 无头模式(需要 profile cookie 已保存)
|
||||||
|
human_wait_timeout: 手工登录 fallback 等待秒数
|
||||||
|
|
||||||
|
返回 {processed, succeeded, failed, details}。
|
||||||
|
"""
|
||||||
|
# Lazy import: 读 facade 持有的 _safe_getenv_dir / _goto / _is_logged_in / 异常类 / URL 常量
|
||||||
|
from service import platform_collector as _pc
|
||||||
|
|
||||||
|
try:
|
||||||
|
from playwright.sync_api import sync_playwright
|
||||||
|
except ImportError:
|
||||||
|
logger.error("playwright 未安装,跳过队列处理")
|
||||||
|
return {"processed": 0, "succeeded": 0, "failed": 0, "details": [],
|
||||||
|
"error": "playwright_not_installed"}
|
||||||
|
|
||||||
|
# 1. 领取一批任务
|
||||||
|
ids_list = [s.strip() for s in ids_filter.split(",") if s.strip()] if ids_filter else None
|
||||||
|
conn = get_connection()
|
||||||
|
try:
|
||||||
|
tasks = repo.claim_pending_tasks(
|
||||||
|
conn,
|
||||||
|
limit=limit if limit > 0 else 999,
|
||||||
|
ids_filter=ids_list,
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
if not tasks:
|
||||||
|
return {"processed": 0, "succeeded": 0, "failed": 0, "details": []}
|
||||||
|
|
||||||
|
profile_dir = _pc._safe_getenv_dir("SMARTCAMPUS_PROFILE_DIR", "playwright-profile")
|
||||||
|
download_dir = _pc._safe_getenv_dir("MATERIALS_DOWNLOAD_DIR", "materials")
|
||||||
|
profile_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
download_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
details: list[dict] = []
|
||||||
|
succeeded = 0
|
||||||
|
failed = 0
|
||||||
|
|
||||||
|
with sync_playwright() as p:
|
||||||
|
ctx = p.chromium.launch_persistent_context(
|
||||||
|
user_data_dir=str(profile_dir),
|
||||||
|
headless=headless,
|
||||||
|
accept_downloads=True,
|
||||||
|
viewport={"width": 1440, "height": 900},
|
||||||
|
channel="msedge" if not headless else None,
|
||||||
|
) if not headless else p.chromium.launch_persistent_context(
|
||||||
|
user_data_dir=str(profile_dir),
|
||||||
|
headless=True,
|
||||||
|
accept_downloads=True,
|
||||||
|
viewport={"width": 1440, "height": 900},
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
page = ctx.new_page()
|
||||||
|
page.set_default_timeout(DEFAULT_PAGE_TIMEOUT_MS)
|
||||||
|
|
||||||
|
# 登录:复用 v1.1.17 collect_all 的探针 + 自动 fill + 手工 fallback 流程
|
||||||
|
logger.info("platform_collector.process_queue: 启动 profile=%s headless=%s 待处理 %d",
|
||||||
|
profile_dir, headless, len(tasks))
|
||||||
|
_pc._goto(page, _pc.SPA_HOME_URL)
|
||||||
|
page.wait_for_timeout(SPA_HYDRATION_MS)
|
||||||
|
if _pc._is_logged_in(page):
|
||||||
|
logger.info("platform_collector.process_queue: ✅ 已登录态")
|
||||||
|
else:
|
||||||
|
_pc._goto(page, _pc.SPA_LOGIN_URL)
|
||||||
|
page.wait_for_timeout(SPA_HYDRATION_MS)
|
||||||
|
if not _fill_spa_login(page):
|
||||||
|
if not headless:
|
||||||
|
_wait_for_manual_login(page, wait_sec=human_wait_timeout)
|
||||||
|
else:
|
||||||
|
raise _pc.LoginTimeoutError("process_queue headless 自动 fill 失败")
|
||||||
|
else:
|
||||||
|
page.wait_for_timeout(POST_LOGIN_SETTLE_MS)
|
||||||
|
|
||||||
|
# 强制 goto SPA_HOME + 侧边栏 hr/contracts
|
||||||
|
for _ in range(SPA_HOME_RETRY_MAX):
|
||||||
|
if _pc._is_logged_in(page):
|
||||||
|
break
|
||||||
|
_pc._goto(page, _pc.SPA_HOME_URL)
|
||||||
|
page.wait_for_timeout(SPA_HOME_RETRY_DELAY_MS)
|
||||||
|
if _fill_spa_login(page):
|
||||||
|
page.wait_for_timeout(POST_LOGIN_SETTLE_MS)
|
||||||
|
if not _pc._is_logged_in(page):
|
||||||
|
raise _pc.LoginTimeoutError("process_queue 仍未登录")
|
||||||
|
|
||||||
|
# 等表格
|
||||||
|
page.wait_for_timeout(TABLE_INITIAL_WAIT_MS)
|
||||||
|
found_table = False
|
||||||
|
for _attempt in range(TABLE_POLL_MAX_ATTEMPTS):
|
||||||
|
page.wait_for_timeout(TABLE_POLL_INTERVAL_MS)
|
||||||
|
try:
|
||||||
|
page.wait_for_selector(
|
||||||
|
"table tbody tr", timeout=TABLE_TR_TIMEOUT_MS,
|
||||||
|
)
|
||||||
|
found_table = True
|
||||||
|
break
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
if not found_table:
|
||||||
|
raise _pc.TableNotFoundError("process_queue 表格未找到")
|
||||||
|
|
||||||
|
mat_btn_count = page.locator(
|
||||||
|
"table tbody tr button:has-text('续聘材料包')"
|
||||||
|
).count()
|
||||||
|
|
||||||
|
# 2. 逐任务处理
|
||||||
|
for t in tasks:
|
||||||
|
contract_id = t["contract_id"]
|
||||||
|
task_id = t["task_id"]
|
||||||
|
conn = get_connection()
|
||||||
|
try:
|
||||||
|
row_idx = _find_row_for_contract(page, contract_id)
|
||||||
|
if row_idx is None:
|
||||||
|
repo.mark_task_failed(conn, task_id, "contract_not_found_in_table")
|
||||||
|
failed += 1
|
||||||
|
details.append({"task_id": task_id, "contract_id": contract_id,
|
||||||
|
"error": "contract_not_found_in_table"})
|
||||||
|
conn.commit()
|
||||||
|
continue
|
||||||
|
materials = _download_materials_for_row(
|
||||||
|
page, row_idx, contract_id, mat_btn_count,
|
||||||
|
)
|
||||||
|
ensure_materials_content_schema(conn)
|
||||||
|
for m in materials:
|
||||||
|
if isinstance(m, dict) and m.get("kind") == "sandbox_card":
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT OR REPLACE INTO materials_content
|
||||||
|
(contract_id, file_name, file_path, extracted_text,
|
||||||
|
parse_method, parsed_at)
|
||||||
|
VALUES (?, ?, ?, ?, 'sandbox_card', ?)
|
||||||
|
""",
|
||||||
|
(
|
||||||
|
contract_id, m["file_name"],
|
||||||
|
f"sandbox://hr/contracts/{contract_id}",
|
||||||
|
m["text"],
|
||||||
|
datetime.now().isoformat(timespec="seconds"),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
repo.parse_sandbox_card_performance(conn, contract_id, m["text"])
|
||||||
|
result_path = (
|
||||||
|
f"sandbox://hr/contracts/{contract_id}"
|
||||||
|
if materials else "no_materials_captured"
|
||||||
|
)
|
||||||
|
repo.mark_task_success(conn, task_id, result_path)
|
||||||
|
succeeded += 1
|
||||||
|
details.append({
|
||||||
|
"task_id": task_id,
|
||||||
|
"contract_id": contract_id,
|
||||||
|
"ok": True,
|
||||||
|
"materials_count": len(materials),
|
||||||
|
})
|
||||||
|
conn.commit()
|
||||||
|
except Exception as e:
|
||||||
|
repo.mark_task_failed(conn, task_id, str(e))
|
||||||
|
failed += 1
|
||||||
|
details.append({
|
||||||
|
"task_id": task_id, "contract_id": contract_id,
|
||||||
|
"error": str(e),
|
||||||
|
})
|
||||||
|
conn.commit()
|
||||||
|
logger.warning("process_queue: task=%s 失败: %s", task_id, e)
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
finally:
|
||||||
|
ctx.close()
|
||||||
|
|
||||||
|
return {
|
||||||
|
"processed": len(tasks),
|
||||||
|
"succeeded": succeeded,
|
||||||
|
"failed": failed,
|
||||||
|
"details": details,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _gen_contract_id(emp_id: str, end_date: str) -> str:
|
||||||
|
"""从 (工号, 到期日) 派 contract_id,确保幂等。
|
||||||
|
|
||||||
|
sandbox v1.1.1 起改用「工号」作主键(之前用姓名)。
|
||||||
|
"""
|
||||||
|
safe_emp = re.sub(r"[^\w]", "", emp_id)[:10] or "T"
|
||||||
|
safe_end = end_date.replace("-", "")[:8]
|
||||||
|
return f"RPA-{safe_emp}-{safe_end}"
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_date(s: str) -> str:
|
||||||
|
"""尽量把 sandbox 返回的日期归一为 YYYY-MM-DD。"""
|
||||||
|
s = s.strip()
|
||||||
|
if re.match(r"^\d{4}-\d{2}-\d{2}$", s):
|
||||||
|
return s
|
||||||
|
# 兜底:截前 10 字符
|
||||||
|
return s[:10]
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_days(s: str) -> int | None:
|
||||||
|
m = re.search(r"-?\d+", s)
|
||||||
|
return int(m.group(0)) if m else None
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_alert(s: str) -> str:
|
||||||
|
s = s.strip()
|
||||||
|
if "红" in s:
|
||||||
|
return "red"
|
||||||
|
if "橙" in s:
|
||||||
|
return "orange"
|
||||||
|
if "黄" in s:
|
||||||
|
return "yellow"
|
||||||
|
return "unknown"
|
||||||
@@ -6,7 +6,7 @@
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
SKILL_SLUG = "alert-contract-expiring-hr"
|
SKILL_SLUG = "alert-contract-expiring-hr"
|
||||||
SKILL_VERSION = "1.1.17"
|
SKILL_VERSION = "1.1.31"
|
||||||
LOG_LOGGER_NAME = "openclaw.skill.alert_contract_expiring_hr"
|
LOG_LOGGER_NAME = "openclaw.skill.alert_contract_expiring_hr"
|
||||||
PLATFORM_KIT_MIN_VERSION = "1.2.11"
|
PLATFORM_KIT_MIN_VERSION = "1.2.11"
|
||||||
|
|
||||||
@@ -48,7 +48,10 @@ PERFORMANCE_RENEW_MIN = 80 # 全部 >= 80 且逐年上升 → renew
|
|||||||
# ----- v1.1.6:RPA 超时常量(与 skill-template human_verification.py 对齐)-----
|
# ----- v1.1.6:RPA 超时常量(与 skill-template human_verification.py 对齐)-----
|
||||||
# 手工登录等待(HITL fallback)
|
# 手工登录等待(HITL fallback)
|
||||||
HUMAN_WAIT_TIMEOUT = 180 # 手工登录等待超时(秒)—— skill-template 同名常量
|
HUMAN_WAIT_TIMEOUT = 180 # 手工登录等待超时(秒)—— skill-template 同名常量
|
||||||
HUMAN_POLL_INTERVAL = 2.0 # 轮询间隔(秒)—— skill-template 同名常量
|
HUMAN_POLL_INTERVAL = 5.0 # 轮询间隔(秒)—— v1.1.19 调长:避免打断用户输入
|
||||||
|
HUMAN_PROMPT_INTERVAL = 15.0 # 剩余时间提示间隔(秒)—— v1.1.19 新增
|
||||||
|
URL_STABLE_REQUIRED_S = 2.0 # URL 连续 N 秒不变才算稳定 —— v1.1.19 新增
|
||||||
|
URL_STABLE_POLL_S = 0.5 # URL 稳定探测间隔(秒)
|
||||||
|
|
||||||
# Playwright 操作超时(毫秒)
|
# Playwright 操作超时(毫秒)
|
||||||
DEFAULT_PAGE_TIMEOUT_MS = 15_000 # Playwright 默认操作超时 —— skill-template 同名常量
|
DEFAULT_PAGE_TIMEOUT_MS = 15_000 # Playwright 默认操作超时 —— skill-template 同名常量
|
||||||
@@ -65,16 +68,16 @@ SPA_HOME_RETRY_MAX = 2 # SPA_HOME retry 最大次数
|
|||||||
|
|
||||||
# 表格等待(毫秒)
|
# 表格等待(毫秒)
|
||||||
TABLE_INITIAL_WAIT_MS = 3_000 # 表格初始等待(React + mock 填充)
|
TABLE_INITIAL_WAIT_MS = 3_000 # 表格初始等待(React + mock 填充)
|
||||||
TABLE_POLL_INTERVAL_MS = 2_000 # 表格轮询间隔
|
TABLE_POLL_INTERVAL_MS = 1_500 # v1.1.32 优化:表格轮询间隔(2s → 1.5s)
|
||||||
TABLE_POLL_MAX_ATTEMPTS = 20 # 表格轮询最大次数(合计 ~40s)
|
TABLE_POLL_MAX_ATTEMPTS = 10 # v1.1.32 优化:轮询最大次数(20 → 10,合计 ~15s)
|
||||||
TABLE_TR_TIMEOUT_MS = 3_000 # 单次 wait_for_selector 超时
|
TABLE_TR_TIMEOUT_MS = 3_000 # 单次 wait_for_selector 超时
|
||||||
FULL_RENDER_WAIT_MS = 1_500 # 完整 DOM 渲染后等渲染结束
|
FULL_RENDER_WAIT_MS = 1_500 # 完整 DOM 渲染后等渲染结束
|
||||||
|
|
||||||
# 侧边栏/卡片(毫秒)
|
# 侧边栏/卡片(毫秒)
|
||||||
SIDEBAR_LINK_TIMEOUT_MS = 30_000 # 等侧边栏 hr/contracts link
|
SIDEBAR_LINK_TIMEOUT_MS = 10_000 # v1.1.32 优化:等侧边栏 link(30s → 10s;sandbox 通常 2-5s)
|
||||||
PUSHSTATE_SETTLE_MS = 1_500 # SPA pushState 后等
|
PUSHSTATE_SETTLE_MS = 1_500 # SPA pushState 后等
|
||||||
SIDEBAR_CLICK_URL_TIMEOUT_MS = 10_000 # click 后等 URL 变 /hr/contracts
|
SIDEBAR_CLICK_URL_TIMEOUT_MS = 10_000 # click 后等 URL 变 /hr/contracts
|
||||||
URL_AFTER_LOGIN_TIMEOUT_MS = 20_000 # 根域 login 后等跳转
|
URL_AFTER_LOGIN_TIMEOUT_MS = 10_000 # v1.1.32 优化:根域 login 后等跳转(20s → 10s)
|
||||||
|
|
||||||
# reload 后备用
|
# reload 后备用
|
||||||
RELOAD_NETWORKIDLE_TIMEOUT_MS = 30_000 # page.reload networkidle 超时
|
RELOAD_NETWORKIDLE_TIMEOUT_MS = 30_000 # page.reload networkidle 超时
|
||||||
@@ -87,3 +90,13 @@ CARD_APPEAR_WAIT_MS = 3_000 # 「续聘材料包」click 后等卡片出
|
|||||||
|
|
||||||
# dump
|
# dump
|
||||||
DUMP_PATH_PARENT_MKDIR = True # _dump_page 是否自动创建父目录
|
DUMP_PATH_PARENT_MKDIR = True # _dump_page 是否自动创建父目录
|
||||||
|
|
||||||
|
# ----- v1.1.18:日志落文件 -----
|
||||||
|
LOG_FILE_DEFAULT_NAME = "alert-contract-expiring-hr.log"
|
||||||
|
LOG_MAX_BYTES_DEFAULT = 5 * 1024 * 1024 # 5MB
|
||||||
|
LOG_BACKUP_COUNT_DEFAULT = 5
|
||||||
|
|
||||||
|
# ----- v1.1.18:材料下载队列 -----
|
||||||
|
DOWNLOAD_QUEUE_MAX_ATTEMPTS_DEFAULT = 3
|
||||||
|
DOWNLOAD_QUEUE_KINDS = ("platform_button", "local_upload")
|
||||||
|
DOWNLOAD_QUEUE_STATUSES = ("pending", "running", "success", "failed")
|
||||||
|
|||||||
@@ -1,20 +1,35 @@
|
|||||||
"""logging 配置:降级到 stderr,避免污染 stdout(stdout 留给结构化输出如 JSON)。
|
"""logging 配置:降级到 stderr,避免污染 stdout(stdout 留给结构化输出如 JSON)。
|
||||||
|
|
||||||
优先用 jiangchang_skill_core 提供的 logging;本地无该包时降级到 stderr。
|
优先用 jiangchang_skill_core 提供的 logging;本地无该包时降级到 stderr。
|
||||||
|
|
||||||
|
v1.1.18:本地降级路径追加 RotatingFileHandler(按 LOG_FILE env 决定是否落文件)。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import logging
|
import logging
|
||||||
|
import logging.handlers
|
||||||
|
import os
|
||||||
import sys
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
from util.constants import LOG_LOGGER_NAME, SKILL_SLUG, SKILL_VERSION
|
from util.constants import (
|
||||||
|
LOG_LOGGER_NAME,
|
||||||
|
SKILL_SLUG,
|
||||||
|
SKILL_VERSION,
|
||||||
|
)
|
||||||
|
|
||||||
_configured = False
|
_configured = False
|
||||||
|
|
||||||
|
|
||||||
def setup_skill_logging() -> logging.Logger:
|
def setup_skill_logging() -> logging.Logger:
|
||||||
"""初始化 skill logger(幂等)。"""
|
"""初始化 skill logger(幂等)。
|
||||||
|
|
||||||
|
本地降级行为(无 jiangchang_skill_core 时):
|
||||||
|
- 若 LOG_FILE 已设:追加 RotatingFileHandler(LOG_MAX_BYTES × LOG_BACKUP_COUNT)
|
||||||
|
- 始终追加 StreamHandler(sys.stderr)
|
||||||
|
- logger level 取自 LOG_LEVEL(默认 INFO)
|
||||||
|
"""
|
||||||
global _configured
|
global _configured
|
||||||
logger = logging.getLogger(LOG_LOGGER_NAME)
|
logger = logging.getLogger(LOG_LOGGER_NAME)
|
||||||
if _configured:
|
if _configured:
|
||||||
@@ -30,15 +45,38 @@ def setup_skill_logging() -> logging.Logger:
|
|||||||
except ImportError:
|
except ImportError:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
# 本地降级:基础 stderr handler,UTF-8 safe
|
# logger 级别
|
||||||
handler = logging.StreamHandler(sys.stderr)
|
level_name = os.environ.get("LOG_LEVEL", "INFO").upper()
|
||||||
handler.setFormatter(
|
logger.setLevel(getattr(logging, level_name, logging.INFO))
|
||||||
logging.Formatter(
|
|
||||||
fmt=f"%(asctime)s [{SKILL_SLUG}] %(levelname)s %(message)s",
|
formatter = logging.Formatter(
|
||||||
datefmt="%Y-%m-%d %H:%M:%S",
|
fmt=f"%(asctime)s [{SKILL_SLUG}] %(levelname)s %(message)s",
|
||||||
)
|
datefmt="%Y-%m-%d %H:%M:%S",
|
||||||
)
|
)
|
||||||
logger.setLevel(logging.INFO)
|
|
||||||
|
# 本地降级:可选 RotatingFileHandler
|
||||||
|
log_file_str = os.environ.get("LOG_FILE")
|
||||||
|
if log_file_str:
|
||||||
|
log_path = Path(log_file_str)
|
||||||
|
if not log_path.is_absolute():
|
||||||
|
from util.runtime_paths import get_logs_dir
|
||||||
|
|
||||||
|
log_path = get_logs_dir() / log_file_str
|
||||||
|
log_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
max_bytes = int(os.environ.get("LOG_MAX_BYTES", str(5 * 1024 * 1024)))
|
||||||
|
backup_count = int(os.environ.get("LOG_BACKUP_COUNT", "5"))
|
||||||
|
file_handler = logging.handlers.RotatingFileHandler(
|
||||||
|
log_path,
|
||||||
|
maxBytes=max_bytes,
|
||||||
|
backupCount=backup_count,
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
file_handler.setFormatter(formatter)
|
||||||
|
logger.addHandler(file_handler)
|
||||||
|
|
||||||
|
# 始终保留 stderr handler
|
||||||
|
handler = logging.StreamHandler(sys.stderr)
|
||||||
|
handler.setFormatter(formatter)
|
||||||
logger.addHandler(handler)
|
logger.addHandler(handler)
|
||||||
logger.propagate = False
|
logger.propagate = False
|
||||||
_configured = True
|
_configured = True
|
||||||
@@ -51,3 +89,19 @@ def get_skill_logger() -> logging.Logger:
|
|||||||
if not _configured:
|
if not _configured:
|
||||||
return setup_skill_logging()
|
return setup_skill_logging()
|
||||||
return logging.getLogger(LOG_LOGGER_NAME)
|
return logging.getLogger(LOG_LOGGER_NAME)
|
||||||
|
|
||||||
|
|
||||||
|
def reset_for_tests() -> None:
|
||||||
|
"""测试用:重置 _configured flag,让 setup_skill_logging 重新初始化。
|
||||||
|
|
||||||
|
必须在每个修改日志配置的测试 fixture 里调用,否则 env 改动不会生效。
|
||||||
|
"""
|
||||||
|
global _configured
|
||||||
|
logger = logging.getLogger(LOG_LOGGER_NAME)
|
||||||
|
for h in list(logger.handlers):
|
||||||
|
try:
|
||||||
|
h.close()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
logger.removeHandler(h)
|
||||||
|
_configured = False
|
||||||
|
|||||||
@@ -66,6 +66,30 @@ def get_dispatch_dir() -> Path:
|
|||||||
return get_data_dir() / "dispatch"
|
return get_data_dir() / "dispatch"
|
||||||
|
|
||||||
|
|
||||||
|
def get_logs_dir() -> Path:
|
||||||
|
"""返回日志目录(RotatingFileHandler 默认落地点)。"""
|
||||||
|
env_dir = os.environ.get("JIANGCHANG_LOGS_DIR")
|
||||||
|
if env_dir:
|
||||||
|
return Path(env_dir)
|
||||||
|
return get_data_dir() / "logs"
|
||||||
|
|
||||||
|
|
||||||
|
def get_input_dir() -> Path:
|
||||||
|
"""返回用户主动投放的输入目录(含 contracts/*.csv + materials/*.{pdf,docx,doc})。"""
|
||||||
|
env_dir = os.environ.get("INPUT_DIR")
|
||||||
|
if env_dir:
|
||||||
|
return Path(env_dir)
|
||||||
|
return get_data_dir() / "input"
|
||||||
|
|
||||||
|
|
||||||
|
def get_input_contracts_dir() -> Path:
|
||||||
|
return get_input_dir() / "contracts"
|
||||||
|
|
||||||
|
|
||||||
|
def get_input_materials_dir() -> Path:
|
||||||
|
return get_input_dir() / "materials"
|
||||||
|
|
||||||
|
|
||||||
def get_examples_dir() -> Path:
|
def get_examples_dir() -> Path:
|
||||||
"""返回样例数据目录。"""
|
"""返回样例数据目录。"""
|
||||||
return get_skill_root() / "examples"
|
return get_skill_root() / "examples"
|
||||||
@@ -78,5 +102,13 @@ def get_templates_dir() -> Path:
|
|||||||
|
|
||||||
def ensure_dirs() -> None:
|
def ensure_dirs() -> None:
|
||||||
"""确保所有需要的目录存在。"""
|
"""确保所有需要的目录存在。"""
|
||||||
for d in (get_data_dir(), get_output_dir(), get_dispatch_dir()):
|
for d in (
|
||||||
|
get_data_dir(),
|
||||||
|
get_output_dir(),
|
||||||
|
get_dispatch_dir(),
|
||||||
|
get_logs_dir(),
|
||||||
|
get_input_dir(),
|
||||||
|
get_input_contracts_dir(),
|
||||||
|
get_input_materials_dir(),
|
||||||
|
):
|
||||||
d.mkdir(parents=True, exist_ok=True)
|
d.mkdir(parents=True, exist_ok=True)
|
||||||
|
|||||||
1621
tests/run_tests.py
1621
tests/run_tests.py
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user