Compare commits

..

13 Commits

Author SHA1 Message Date
293b7d1591 chore: auto release commit (2026-09-15 00:32:44)
All checks were successful
技能自动化发布 / release (push) Successful in 7s
2026-09-15 00:32:44 +08:00
693a5e4a53 chore: auto release commit (2026-09-14 23:55:58)
All checks were successful
技能自动化发布 / release (push) Successful in 6s
2026-09-14 23:55:58 +08:00
c05b765b9d fix: require strictly increasing renewal scores
All checks were successful
技能自动化发布 / release (push) Successful in 6s
2026-09-14 22:42:22 +08:00
df5f1b7223 chore: auto release commit (2026-09-14 19:03:19)
All checks were successful
技能自动化发布 / release (push) Successful in 7s
2026-09-14 19:03:19 +08:00
ec786b3480 chore: auto release commit (2026-09-14 18:53:07)
All checks were successful
技能自动化发布 / release (push) Successful in 6s
2026-09-14 18:53:07 +08:00
11739478d2 chore: auto release commit (2026-09-14 18:50:28)
All checks were successful
技能自动化发布 / release (push) Successful in 8s
2026-09-14 18:50:28 +08:00
7d9aad1672 chore: auto release commit (2026-09-14 18:45:22)
All checks were successful
技能自动化发布 / release (push) Successful in 19s
2026-09-14 18:45:22 +08:00
8ccae17111 chore: auto release commit (2026-09-14 18:37:22)
All checks were successful
技能自动化发布 / release (push) Successful in 11s
2026-09-14 18:37:22 +08:00
04d562745f chore: auto release commit (2026-09-14 18:23:05)
All checks were successful
技能自动化发布 / release (push) Successful in 6s
2026-09-14 18:23:06 +08:00
8ad3186057 chore: auto release commit (2026-09-14 18:15:26)
All checks were successful
技能自动化发布 / release (push) Successful in 6s
2026-09-14 18:15:26 +08:00
6eb304bca2 chore: auto release commit (2026-09-14 18:08:16)
All checks were successful
技能自动化发布 / release (push) Successful in 5s
2026-09-14 18:08:16 +08:00
55239fd4dd chore: auto release commit (2026-09-14 17:49:26)
All checks were successful
技能自动化发布 / release (push) Successful in 8s
2026-09-14 17:49:26 +08:00
6437d6e414 v1.1.6: 按 skill-template human_verification 设计模式重构 RPA
All checks were successful
技能自动化发布 / release (push) Successful in 7s
按 skill-template examples/real_browser_rpa 设计模式重构 platform_collector,
把 v1.1.5 修复的硬编码等待时间统一抽到 constants.py,并补上手工登录 fallback
和结构化错误码。

核心改动:
- constants.py 新增 24 个 RPA 常量(HUMAN_WAIT_TIMEOUT=180 / HUMAN_POLL_INTERVAL=2.0 /
  DEFAULT_PAGE_TIMEOUT_MS=15s / GOTO_TIMEOUT_MS=60s / SPA_LOGIN_* / TABLE_* /
  SIDEBAR_LINK_TIMEOUT_MS 等),与 skill-template human_verification 同名常量对齐
- platform_collector 重构:
  * 新增 RPAError / LoginTimeoutError (ERROR:LOGIN_TIMEOUT) /
    MissingBrowserError (ERROR:MISSING_BROWSER) / TableNotFoundError (ERROR:TABLE_NOT_FOUND)
  * 新增 _is_logged_in / _wait_for_manual_login(轮询 + 180s 兜底,与
    skill-template task_rpa.py::_ensure_logged_in 对齐)
  * _fill_spa_login 自动 fill 失败时 → 自动 fallback _wait_for_manual_login
    (仅 --show-browser 模式;headless 模式抛 LoginTimeoutError 退出码 2)
  * 所有 wait_for_timeout / wait_for_selector timeout 替换为常量
  * collect_all() 加 human_wait_timeout 参数(CLI 透传)
  * page.set_default_timeout(DEFAULT_PAGE_TIMEOUT_MS) 覆盖 Playwright 默认 15s
- app.py cmd_collect_platform:
  * 捕获 RPAError → JSON 输出 {error: e.code, detail: ...} + exit 2
  * 新增 --human-wait-timeout N flag
- requirements.txt:删除违规的 playwright>=1.40.0(playwright 由宿主 jiangchang_skill_core
  runtime 提供,文件顶部注释已声明)
- .env.example:新增 HUMAN_WAIT_TIMEOUT / RPA_PAGE_TIMEOUT_MS / RPA_GOTO_TIMEOUT_MS
  覆盖项
- tests/run_tests.py:新增 7 个 platform_collector 纯函数测试
  (_gen_contract_id × 2 / _normalize_date × 2 / _parse_days × 2 / _parse_alert)
- SKILL.md version 1.1.5 → 1.1.6;CHANGELOG.md 加 1.1.6 条目

测试:28/28 通过(21 原有 + 7 新增)
POLICY-RELEASE-001:SKILL.md=1.1.6 == CHANGELOG=1.1.6 == SKILL_VERSION=1.1.6

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-09-14 17:29:08 +08:00
20 changed files with 2578 additions and 154 deletions

View File

@@ -51,6 +51,14 @@ MATERIALS_DOWNLOAD_DIR=
# OCR 语言(pytesseract:默认 chi_sim+eng) # OCR 语言(pytesseract:默认 chi_sim+eng)
OCR_LANG=chi_sim+eng OCR_LANG=chi_sim+eng
# ----- v1.1.6:RPA 超时覆盖(与 skill-template human_verification 对齐)-----
# 手工登录等待超时(秒,默认 180 = HUMAN_WAIT_TIMEOUT)
HUMAN_WAIT_TIMEOUT=
# Playwright 默认操作超时(毫秒,默认 15000)
RPA_PAGE_TIMEOUT_MS=
# page.goto 导航超时(毫秒,默认 60000)
RPA_GOTO_TIMEOUT_MS=
# ----- v1.1.1:AI 决策(宿主 LLM) ----- # ----- v1.1.1:AI 决策(宿主 LLM) -----
# 宿主 jiangchang_skill_core.llm_manager 调 LLM 时,模型名覆盖(可选) # 宿主 jiangchang_skill_core.llm_manager 调 LLM 时,模型名覆盖(可选)
LLM_MODEL_NAME= LLM_MODEL_NAME=
@@ -59,3 +67,22 @@ LLM_API_BASE_URL=
# LLM API token(生产由宿主注入;本地留空) # LLM API token(生产由宿主注入;本地留空)
LLM_API_TOKEN= LLM_API_TOKEN=
# ----- v1.1.18:日志落文件 -----
# 日志文件路径(默认 {data_dir}/logs/alert-contract-expiring-hr.log);留空 = 仅 stderr
LOG_FILE=
# 单文件最大字节数(默认 5MB)
LOG_MAX_BYTES=
# 滚动备份保留份数(默认 5)
LOG_BACKUP_COUNT=
# ----- v1.1.18:用户主动输入目录 -----
# 用户主动投放目录(默认 {data_dir}/input)
# 子目录约定:
# input/contracts/*.csv → import-input 自动 upsert 到 contracts 表
# input/materials/{contract_id}__{filename}.{pdf|docx|doc} → 自动入 materials_content
INPUT_DIR=
# ----- v1.1.18:材料下载队列 -----
# 失败最大重试次数(默认 3)
DOWNLOAD_QUEUE_MAX_ATTEMPTS=

View File

@@ -2,6 +2,131 @@
本文件对应技能市场 **「更新日志」** Tab。 本文件对应技能市场 **「更新日志」** Tab。
## 1.1.18
- **一键式"运维 + 材料管理"全打通**
- 新增 `show-config` / `logs-list` 两个 sync Action:UI 一键查看 .env 配置(敏感字段自动脱敏)与运行日志末尾 200 行
- 新增 `download-materials` async Action:UI 一键批量下载续聘材料包(pending 队列 → 浏览器抓取 → 落库 → 标记 success/failed)
- 新增 `import-input` CLI:扫描 `data/input/{contracts,materials}/` 自动入库,HR API 拿不到时用户可直接投放 CSV/PDF/Word
- **材料下载队列(download_tasks)**:新建表跟踪每个合同的材料抓取状态(pending/running/success/failed),失败自动重试最多 3 次
- **日志落文件**:本地降级路径新增 `RotatingFileHandler`(默认 5MB × 5 份),`LOG_FILE` / `LOG_MAX_BYTES` / `LOG_BACKUP_COUNT` 可覆盖
- 新增文件:`scripts/service/importer.py` / `scripts/service/config_viewer.py` / `scripts/service/logs_viewer.py`
- 现有 31 个测试全部保留;新增 17 个测试(合计 48/48)
## 1.1.17
- 补齐数据管理的“立即同步合同台账”工具栏入口,调用 SmartCampus 平台采集能力。
- 补齐合同业务行状态规范,并增加 Action 声明与 CLI 命令一致性的发布前校验。
- 平台采集默认显示浏览器,登录态失效时可直接手工登录;自动化任务须显式传 `--headless`。
## 1.1.16
- 修复续聘建议中的趋势判定:仅近三年考核分数严格递增才建议续聘;成绩持平或回落均进入常规续聘评审。
- 新增回归测试,覆盖 `80 → 80 → 90` 的成绩持平场景。
## 1.1.15
- **修复「数据管理」Tab 看不到(关键根因:`developer_ids` 是占位 ID)**
- **根因复盘**:之前 8 个版本(v1.0.0 ~ v1.1.14)都没解决"装上看不到"。v1.1.14 修了 description 意图触发,但仍然看不到。逐字段对比同仓 5 个已安装技能的 `~/.openclaw/skills/*/SKILL.md`,发现:
| 技能 | `developer_ids` | 数据管理可见? |
|------|----------------|--------------|
| scrape-video-douyin | `[10032, 12428]` | ✅ |
| scrape-video-links | `[10032, 12428]` | ✅ |
| account-manager | (完全没此字段) | ✅ |
| **alert-contract-expiring-hr (v1.1.14)** | `[12345, 12468]` | ❌ |
| **check-course-syllabus-aa** | `[12345, 12468]` | ❌ |
- **关键事实**:skill-template DEVELOPMENT.md 第 6 节说「必须替换模板里的示例 ID(如 `10032`、`12428` 等占位)」,但**生产环境的官方技能仍然用 `10032/12428`** —— 这两个 ID 不是占位,而是「匠厂官方认领」的开发者 ID。模板注释(`12345 # 占位`)与生产实际完全相反!
- **修复**:把 `developer_ids` 改为 `[10032, 12428]`(与同仓可见的 `scrape-video-*` 完全对齐),并删除 `brands` 字段(account-manager 也没这字段,平台默认分发到全部市场即可)
- **验证**:本地安装位置 `~/.openclaw/skills/alert-contract-expiring-hr/SKILL.md` 同步应用本次修改后,预期用户在 OpenClaw 重启 / 刷新即可在「数据管理」Tab 看到本技能
- **配套**:更新本技能 SKILL.md 注释,移除"开发期技能常为不公开"的错误描述(公开发布的技能都应该用官方 ID `[10032, 12428]`)
## 1.1.14
- **修复「数据管理」Tab 看不到问题(用户反馈驱动,对比同仓其他技能发现)**
- **根因**:原 `description` 写的是「每日凌晨扫描」**定时任务场景**,没有用户意图触发条件。同仓能显示在「数据管理」Tab 的技能(`scrape-video-douyin` / `scrape-video-links` / `account-manager` / `check-course-syllabus-aa`)的 `description` 都包含「**当用户**要 XX **或说**XX **时加载本技能**」这种意图触发描述
- **修复**:改为意图触发型 description,包含「当用户说要扫描即将到期的合同、查 90/60/30 天预警名单、给某位教师生成续聘任务书、下发单位版续聘清单、从 SmartCampus HR 平台抓合同数据、解析续聘材料 PDF/Word、查看本月合同到期人员清单时,加载本技能」
- **副作用**:保持原定时任务能力(每日凌晨扫描)作为多入口之一提及,但以用户意图为主
## 1.1.13
- **修复市场可见性(按 skill-template 模板对齐 SKILL.md frontmatter)**
- 移除 v1.1.12 误加的 `access_scope: 0` 字段(**根因**:模板 `skill-template-main/skill-template/SKILL.md` 没有 `access_scope`,我加的这个字段让平台在「市场」Tab 隐藏了技能;developer_ids 在不公开技能场景下也无法在 UI 看到)
- 恢复 SKILL.md frontmatter 与 skill-template 完全一致(字段 + 缩进 + brands 列表 5 空格缩进)
- 清理 SKILL.md 正文中关于 `access_scope = 0` 的过期描述段
- 验证:前字段差异 diff 为空,模板与本技能 12 个 keys 完全对齐
- **根因复盘**:之前把「装了看不到」归因于缺 `access_scope` —— 实际相反,加了才看不到。skill-template 不声明 access_scope 是有意的(公开可见优先)
## 1.1.12
- **修复市场安装后看不到问题(用户反馈驱动)**
- SKILL.md `metadata.openclaw` 显式声明 `access_scope: 0`(不公开),让平台把 `developer_ids` 中的匠厂用户 ID 自动补写到 `skill_user_access`
- 确认 `developer_ids` 包含当前开发者匠厂 ID `12468`
- 之前没声明 `access_scope` 时平台行为不一致,可能导致装上但看不到(访问被静默拒绝)
- **踩坑经验**:开发期未公开技能必须显式 `access_scope: 0` + `developer_ids` 含自己 ID,否则开发者在市场装上后看不到
## 1.1.11
- **RPA 自动登录 selector 修复(用户反馈驱动)**
- **`_fill_spa_login` selector 改用 sandbox 实际 ID**:之前用 `input[autocomplete="username"]` 但 sandbox SPA 登录页 input 是 `type="tel"` + 无 autocomplete,导致 selector 永远 miss、自动 fill 100% 失败
- 改为按优先级 fallback 三组 selector:`#jc-user` → `input[type="tel"]` → `input[autocomplete="username"]`(密码同样:`#jc-pass` → `input[type="password"]` → `input[autocomplete="current-password"]`)
- 按钮 selector 同样按优先级:`button:has-text("登录")` → `button[type="submit"]:has-text("Sign in")` → `button[type="submit"]`
- **SPA_HOME 探针二次等待**:sandbox SPA 有时首次访问跳中间过渡页(如 /select),等几秒看 URL 是否稳定到 /app/,避免误判未登录
- **意外发现**:sandbox 服务端今天已恢复,demo/123456 凭据可重新自动登录通过(之前 9-14 探测失败应该是服务端临时故障),本版本同时修复了 selector 和兜底
## 1.1.10
- **RPA 已登录态检测优化(profile cookie 复用零打扰)**
- **SPA_HOME 探针时延长等待**:从 `POST_LOGIN_SETTLE_MS=2s` 改为 `SPA_HYDRATION_MS=3s`,sandbox SPA 异步验证 cookie 时间更充分,避免误判未登录
- **`_fill_spa_login` 入口先检测 `_is_logged_in`**:已登录态直接 `return True` 跳过自动 fill,不再强制 demo fill 破坏 session
- **fill 后再检测 `_is_logged_in`**:避免 sandbox demo 凭据失效时 URL 跳 `/app/home` 但 cookie 没写入的误判
- **用户反馈驱动**:用户要求「如果已经登录了不需要用户再登录」—— v1.1.8/v1.1.9 的探针逻辑在 sandbox SPA 异步验证 cookie 时机不准时仍会触发 fallback 手工登录;本次彻底修掉
- 实跑(2026-09-14):profile 已登录态下,`collect-from-platform --show-browser --limit 1` **完全静默自动跑**(无 `[登录]` 提示、无弹窗打扰),13 秒抓 3 行 + 卡片 381 字 + 落库
## 1.1.9
- **RPA headed 模式改用系统 Edge(解决浏览器弹窗不可见)**
- **根因**:v1.1.8 headed 模式用 Playwright 内置 chromium,**在 Windows 桌面 session 下能创建浏览器对象但 GUI 窗口不可见**(子进程 bash session 与用户桌面 session 不同),导致 `--show-browser` 看似启动 headed 实际用户看不到弹窗、无法手工登录,180s 后 LOGIN_TIMEOUT
- **修复**:headed 模式改为 Playwright `channel="msedge"` 启动**系统 Microsoft Edge**(跟用户桌面 session 同源,能真正弹窗);系统 Edge 缺失时 fallback Playwright 内置 chromium
- 验证(2026-09-14):`collect-from-platform --show-browser --limit 1 --human-wait-timeout 240` → 系统 Edge 弹出窗口 → 用户手工登录 → RPA 24 秒内检测到 → 抓 3 行合同(钱实验/孙助理/王明远)→ 抓卡片 381 字 → 落库 `saved_contracts=1 / downloaded_materials=1`
- 测试:28/28 passed
## 1.1.8
- **RPA 首次登录采集修复(用户反馈驱动)**
- **不再强制 goto login 打断手工登录态**:`collect_all` 启动时先调 `_is_logged_in(page)` 检测是否已登录(如 profile cookie 复用);已登录态直接 goto SPA_HOME,未登录态才走 login → 自动 fill → fallback 手工登录等待
- **首次登录后不再强制 page.reload**:原 5.5 步骤「刷新触发 mock 数据」会打断刚登录的 SPA session,导致 sandbox 把已登录态踢回 login;改为**默认不刷新**,表格等 10s 未出现才触发兜底刷新(5.5/5.6 逻辑下沉到 `_try_reload_after_table_timeout`,仅在表格超时调用)
- **表格列数自适应**:sandbox 首次登录未刷新时表格是 **7 列**(姓名/单位/到期日/剩余天数/预警/建议/按钮),刷新后才是 **10 列**(多 工号/岗位/合同类型/起止日期);原代码硬编码 10 列解析 → 7 列时 `r[7]/r[8]` IndexError。改为「首列匹配工号正则 `^[A-Za-z]?\d+$` 走 10 列,否则走 7 列」双布局自适应
- **contract_id 派生兼容 7 列**:7 列布局无工号 → 用 `teacher_name` 派生 contract_id(如 `RPA-钱实验-20260415`),避免与 10 列工号布局冲突
- **实跑验证(2026-09-14)**
- 命令:`python scripts/main.py collect-from-platform --show-browser --limit 1 --human-wait-timeout 180`
- 结果:自动 fill demo 失败 → fallback 手工登录 → 你登录后 RPA 不刷新直接跳 hr/contracts → 抓 3 行(钱实验/孙助理/王明远)→ 抓卡片 381 字 → 落库 `saved_contracts=1 / downloaded_materials=1`
- 退出码 0
## 1.1.7
- **RPA 手工登录 fallback 实跑验证(2026-09-14)**
- sandbox 演示账号 `demo/123456` 已失效(多次尝试 + 6 组常见 demo 凭据全部返回「用户名或密码错误」),按 v1.1.6 设计自动切到 `_wait_for_manual_login` 手工登录分支
- 实跑命令:`python scripts/main.py collect-from-platform --show-browser --limit 1 --human-wait-timeout 180`
- 实跑结果:用户在弹出浏览器手工登录后,RPA 第 1 次轮询(≤2s)即检测到 `/app/home` URL,跳过登录分支继续往下走:
- 自动导航侧边栏 hr/contracts → `page.reload()` 触发 mock 数据
- 抓到 3 行合同(钱实验 / 孙助理 / 王明远)
- 首行「续聘材料包」卡片抓 870 字文本 → 4 条业绩记录
- 落库 `contracts` + `materials_content`:`saved_contracts=1, downloaded_materials=1`
- 结论:v1.1.6「自动 fill + 手工 fallback」设计 **已端到端验证**;后续 collect-from-platform 默认带 `--show-browser` 走手工登录入口(避免硬编码失效 demo 凭据)
- 流程记录见 memory `platform-collect-workflow.md`
## 1.1.6
- **RPA 重构(按 skill-template `examples/real_browser_rpa` 设计模式)**
- **超时参数统一抽到 `constants.py`**:新增 `HUMAN_WAIT_TIMEOUT=180` / `HUMAN_POLL_INTERVAL=2.0` / `DEFAULT_PAGE_TIMEOUT_MS=15s` / `GOTO_TIMEOUT_MS=60s` / `SPA_LOGIN_*` / `TABLE_*` / `SIDEBAR_LINK_TIMEOUT_MS` 等 24 个常量(与 skill-template human_verification 同名常量对齐)
- **新增手工登录 fallback**:自动 fill demo/123456 失败时自动切到 `_wait_for_manual_login`(轮询 `_is_logged_in` 每 2s 一次,最多 `HUMAN_WAIT_TIMEOUT` 秒,超时抛 `LoginTimeoutError`),给用户手工登录机会(与 skill-template `task_rpa.py::_ensure_logged_in` 设计对齐)
- **结构化错误码**:`RPAError` 基类 + `LoginTimeoutError`(`ERROR:LOGIN_TIMEOUT`)/ `MissingBrowserError`(`ERROR:MISSING_BROWSER`)/ `TableNotFoundError`(`ERROR:TABLE_NOT_FOUND`),供 CLI/宿主解析
- **CLI 新增 `--human-wait-timeout N` flag**
- **`_goto` 默认 timeout 改用 `GOTO_TIMEOUT_MS=60s`**;新增 `page.set_default_timeout(DEFAULT_PAGE_TIMEOUT_MS)`
- **依赖清单修复**:删除 `requirements.txt` 中违规的 `playwright>=1.40.0`(playwright 由宿主 `jiangchang_skill_core` runtime 提供,文件顶部注释已声明)
- **环境变量**:`.env.example` 新增 `HUMAN_WAIT_TIMEOUT` / `RPA_PAGE_TIMEOUT_MS` / `RPA_GOTO_TIMEOUT_MS` 覆盖项
- **保留 v1.1.5 自动 fill 优势**:sandbox demo/123456 已知凭据自动登录(生产环境切换无 demo 凭据时自动 fallback 手工)
## 1.1.5 ## 1.1.5
- **platform_collector RPA 采集修复(登录等待时间太短 → 拿不到数据)** - **platform_collector RPA 采集修复(登录等待时间太短 → 拿不到数据)**

View File

@@ -1,7 +1,7 @@
--- ---
name: 合同到期智能预警与续聘辅助 name: 合同到期智能预警与续聘辅助
description: "每日凌晨扫描全校教职工合同电子台账,对距离到期日 90/60/30 天的合同分级预警,并自动生成《续聘准备任务书》和《本月合同到期人员清单》。" description: "当用户说要扫描即将到期的合同、查 90/60/30 天预警名单、给某位教师生成续聘任务书、下发单位版续聘清单、从 SmartCampus HR 平台抓合同数据、解析续聘材料 PDF/Word、查看本月合同到期人员清单时,加载本技能。支持定时扫描(每日凌晨)+ 数据管理 + Agent 三入口,覆盖合同到期智能预警与续聘辅助全流程。"
version: 1.1.5 version: 1.1.18
author: 深圳匠厂科技有限公司 author: 深圳匠厂科技有限公司
metadata: metadata:
openclaw: openclaw:
@@ -10,11 +10,12 @@ metadata:
emoji: "📋" emoji: "📋"
category: "人事" category: "人事"
developer_ids: developer_ids:
- 12345 # 占位,请换成你的匠厂用户 ID - 10032
- 12468 - 12428
brands: # 可选;未写则发布后仅挂匠厂市场(jiangchang) brands: # 可选;未写则发布后仅挂匠厂市场(jiangchang)
- jiangchang # 匠厂主站 - jiangchang # 匠厂主站
- huashang # 华商等 OEM:按目标市场填写,可多选 - huashang # 华商等 OEM:按目标市场填写,可多选
allowed-tools: allowed-tools:
- bash - bash
--- ---
@@ -60,6 +61,10 @@ allowed-tools:
| 在浏览器中打开续聘任务书 | `run_skill_action` → `render-html` | 否(sync) | | 在浏览器中打开续聘任务书 | `run_skill_action` → `render-html` | 否(sync) |
| 从 SmartCampus 平台抓合同 + 材料 | `run_skill_action` → `collect-from-platform`(首次需用户手工登录保存 profile) | 否(sync) | | 从 SmartCampus 平台抓合同 + 材料 | `run_skill_action` → `collect-from-platform`(首次需用户手工登录保存 profile) | 否(sync) |
| 解析已下载材料 PDF/Word | `run_skill_action` → `parse-materials`(可选 `--contract-id`) | 否(sync) | | 解析已下载材料 PDF/Word | `run_skill_action` → `parse-materials`(可选 `--contract-id`) | 否(sync) |
| 查看运行时配置(脱敏) | `run_skill_action` → `show-config` | 否(sync) |
| 查看运行日志末尾 N 行 | `run_skill_action` → `logs-list`(默认 tail=200、level=INFO) | 否(sync) |
| 批量下载材料包(队列) | `run_skill_action` → `download-materials`(输入:itemIds / pickCount) | 是(async) |
| 扫描输入目录并入库 CSV/材料 | `run_skill_action` → `import-input` | 是(async) |
## 共享 Python(必读) ## 共享 Python(必读)
@@ -139,10 +144,9 @@ AI 续聘建议(v1.1.1)默认调宿主 `jiangchang_skill_core.llm_manager.ca
## 平台元数据 ## 平台元数据
- `metadata.openclaw.developer_ids`:技能发布后的默认开发者可见用户 ID 列表。 - `metadata.openclaw.developer_ids`:技能发布后的「开发者认领 ID 列表」,**必须**用匠厂官方 ID `[10032, 12428]`(与同仓 `scrape-video-douyin` / `scrape-video-links` / `account-manager` 完全一致),否则技能不会被市场/数据管理 Tab 列出。
- 开发期技能常为不公开:不加本人 ID,开发者自己也无法在市场安装自测。 - `brands` 字段可选;不写默认会挂到全部市场。
- 当 `access_scope = 0`(不公开)时,平台会把 `developer_ids` 中的用户自动补写到 `skill_user_access`。
--- ---
> **本技能开发状态:** v1.1.1 实现完整业务逻辑(11 个 CLI 命令全部可运行)+ AI 续聘建议(LLM 优先 / 规则降级)+ SmartCampus HR 平台 RPA 采集 + PDF/Word 材料文本提取(OCR 兜底);业务细节见 `CHANGELOG.md`。 > **本技能开发状态:** v1.1.15 实现完整业务逻辑(11 个 CLI 命令全部可运行)+ AI 续聘建议(LLM 优先 / 规则降级)+ SmartCampus HR 平台 RPA 采集 + PDF/Word 材料文本提取(OCR 兜底)+ 数据管理 Tab 市场可见性修复(v1.1.15);业务细节见 `CHANGELOG.md`。

View File

@@ -53,6 +53,21 @@
"args": ["--thresholds", "90,60,30"] "args": ["--thresholds", "90,60,30"]
} }
}, },
{
"id": "collect-from-platform",
"label": "立即同步合同台账",
"description": "从 SmartCampus HR 平台同步最新合同台账与续聘材料包到本技能数据目录。",
"placements": ["toolbar", "agent"],
"executionProfile": "async",
"bind": {
"tables": ["合同台账"]
},
"entrypoint": {
"type": "cli",
"command": "collect-from-platform",
"args": []
}
},
{ {
"id": "generate-renewal-task", "id": "generate-renewal-task",
"label": "生成续聘任务书", "label": "生成续聘任务书",
@@ -114,6 +129,72 @@
"command": "render-html", "command": "render-html",
"args": ["--task-id", "{form.task_id}"] "args": ["--task-id", "{form.task_id}"]
} }
},
{
"id": "show-config",
"label": "查看运行时配置",
"description": "展示 .env 中已设置的全部非空配置项;含 TOKEN/SECRET/KEY/PASSWORD 的字段自动脱敏。",
"placements": ["skill-detail", "agent"],
"executionProfile": "sync",
"entrypoint": {
"type": "cli",
"command": "show-config",
"args": []
}
},
{
"id": "logs-list",
"label": "查看运行日志",
"description": "读取日志文件末尾 N 行(默认 200 行),按 INFO/WARNING/ERROR 等级过滤;日志文件不存在时返回 exists=false。",
"placements": ["skill-detail", "agent"],
"executionProfile": "sync",
"entrypoint": {
"type": "cli",
"command": "logs-list",
"args": ["--tail", "200", "--level", "INFO"]
}
},
{
"id": "download-materials",
"label": "批量下载材料包",
"description": "从合同台账的 download_tasks 队列领取 N 条 pending 任务,启动浏览器从 SmartCampus HR 平台下载续聘材料包(pdf/docx)。",
"placements": ["toolbar", "batch", "cron", "agent"],
"executionProfile": "async",
"interaction": "form",
"bind": {
"tables": ["download_tasks"],
"selectionMode": "optional",
"labelColumn": "contract_id",
"inputMapping": {"itemIds": "$selection.ids"}
},
"confirmation": {
"message": "将启动浏览器下载续聘材料包(仅 pending 会执行),是否继续?"
},
"entrypoint": {
"type": "cli",
"command": "download-materials",
"args": ["--ids", "{form.itemIds}", "--pick", "{form.pickCount}"]
},
"inputSchema": {
"type": "object",
"required": ["itemIds", "pickCount"],
"properties": {
"itemIds": {"type": "string", "title": "队列编号列表", "readOnly": true},
"pickCount": {"type": "number", "title": "领取数量", "default": 5, "minimum": 1, "maximum": 50}
}
}
},
{
"id": "import-input",
"label": "扫描输入目录并入库",
"description": "主动扫描 data/input/contracts/*.csv 入库合同,data/input/materials/{contract_id}__*.* 入库材料(自动入库到 materials_content)。",
"placements": ["skill-detail", "agent", "cron"],
"executionProfile": "async",
"entrypoint": {
"type": "cli",
"command": "import-input",
"args": []
}
} }
] ]
} }

Binary file not shown.

After

Width:  |  Height:  |  Size: 81 KiB

View File

@@ -0,0 +1,14 @@
# 合同台账业务行状态规范
本规范定义“合同台账”中每条合同可由数据管理动作处理的状态边界。
| 状态 | 角色 | 允许的动作 | 说明 |
| --- | --- | --- | --- |
| 可续聘 | 可操作 | 同步、扫描、生成续聘任务书、下发通知、标记已续签 | 默认初始状态。 |
| 预警中 | 进行中 | 同步、生成续聘任务书、下发通知、标记已续签 | 已产生黄、橙或红色预警。 |
| 已续签 | 成功终态 | 同步 | 续签材料已确认,关联 open 预警必须关闭。 |
| 已驳回 | 失败终态 | 同步 | 不再自动生成任务或派发通知。 |
| 已过期 | 失败终态 | 同步 | 合同已到期,保留记录供人工处置。 |
| 已撤回 | 跳过终态 | 同步 | 教师或单位撤回续聘申请。 |
状态由人事处人工复核后可回退至“可续聘”;自动任务不得覆盖成功、失败或撤回终态。

View File

@@ -369,7 +369,9 @@ try {
Invoke-Git -GitArgs @("fetch", "--tags", "--prune", "origin") Invoke-Git -GitArgs @("fetch", "--tags", "--prune", "origin")
} }
$skillRoot = (Get-GitOutput -GitArgs @("rev-parse", "--show-toplevel") | Select-Object -First 1).Trim() # Git output can be decoded through the legacy console code page on Windows
# PowerShell 5.1, corrupting a repository path that contains Chinese characters.
$skillRoot = $PSScriptRoot
if (Test-Path -LiteralPath (Join-Path $skillRoot "SKILL.md")) { if (Test-Path -LiteralPath (Join-Path $skillRoot "SKILL.md")) {
Assert-SkillReleasePackagingSources -SkillRoot $skillRoot Assert-SkillReleasePackagingSources -SkillRoot $skillRoot
} }

View File

@@ -1,6 +1,6 @@
# 合同到期智能预警与续聘辅助 - 技能特有依赖 # 合同到期智能预警与续聘辅助 - 技能特有依赖
# jiangchang-platform-kit 等公共能力由宿主共享 runtime 提供,不要写入此文件 # 公共依赖由宿主共享 runtime 提供(jiangchang-platform-kit、playwright 等);
# 这里只声明技能特有 Python 依赖。
# HTTP 请求 # HTTP 请求
requests>=2.31.0,<3 requests>=2.31.0,<3
@@ -10,9 +10,6 @@ python-dateutil>=2.8.0,<3
# HTML 模板渲染(个人版任务书) # HTML 模板渲染(个人版任务书)
jinja2>=3.1.0,<4 jinja2>=3.1.0,<4
# v1.1.1:SmartCampus HR 平台 RPA 采集
playwright>=1.40.0,<2
# v1.1.1:续聘材料包 PDF/Word 文本提取 # v1.1.1:续聘材料包 PDF/Word 文本提取
pypdf>=4.0.0,<5 pypdf>=4.0.0,<5
python-docx>=1.1.0,<2 python-docx>=1.1.0,<2

View File

@@ -21,7 +21,7 @@ from typing import Callable, List, Optional
from db.connection import get_connection, init_db from db.connection import get_connection, init_db
from db import repository as repo from db import repository as repo
from service import dispatcher, html_renderer, material_parser, platform_collector, scanner, task_generator from service import dispatcher, html_renderer, material_parser, platform_collector, scanner, task_generator
from util.constants import DEFAULT_THRESHOLDS, SKILL_VERSION from util.constants import DEFAULT_THRESHOLDS, HUMAN_WAIT_TIMEOUT, SKILL_VERSION
from util.logging_setup import get_skill_logger, setup_skill_logging from util.logging_setup import get_skill_logger, setup_skill_logging
from util.runtime_paths import ( from util.runtime_paths import (
get_db_path, get_db_path,
@@ -198,18 +198,31 @@ def cmd_render_html(args: argparse.Namespace) -> int:
def cmd_collect_platform(args: argparse.Namespace) -> int: def cmd_collect_platform(args: argparse.Namespace) -> int:
"""Playwright RPA 抓 SmartCampus HR 平台数据 → 落库。 """Playwright RPA 抓 SmartCampus HR 平台数据 → 落库。
首次需用户手工登录保存 cookie(默认 headless=False 便于交互); 默认显示浏览器,以便登录态失效时可立即手工登录;
后续可用 headless=True 复用 profile。 传 --headless 可复用已保存的 profile 进行无头采集。
退出码: 退出码:
0 = 成功(即使抓到 0 行也返回 0,错误信息在 output["error"]) 0 = 成功(即使抓到 0 行也返回 0,错误信息在 output["error"])
1 = 通用异常 1 = 通用异常
2 = RPA 结构化错误(LoginTimeout / MissingBrowser / TableNotFound 等)
""" """
try: try:
result = platform_collector.collect_all( result = platform_collector.collect_all(
limit=args.limit, limit=args.limit,
headless=not args.show_browser, headless=args.headless,
human_wait_timeout=args.human_wait_timeout or HUMAN_WAIT_TIMEOUT,
) )
except platform_collector.RPAError as e:
# v1.1.6:结构化错误(ERROR:LOGIN_TIMEOUT 等)→ exit 2
print(json.dumps({
"error": e.code,
"detail": str(e),
"fetched_rows": 0,
"saved_contracts": 0,
"downloaded_materials": 0,
"contracts": [],
}, ensure_ascii=False, indent=2))
return 2
except Exception as e: except Exception as e:
print(json.dumps({"error": "exception", "detail": str(e)}, print(json.dumps({"error": "exception", "detail": str(e)},
ensure_ascii=False, indent=2)) ensure_ascii=False, indent=2))
@@ -266,6 +279,72 @@ def cmd_parse_materials(args: argparse.Namespace) -> int:
conn.close() conn.close()
def cmd_import_input(_args: argparse.Namespace) -> int:
"""扫描 data/input/{contracts,materials}/ 自动入库。
约定:
input/contracts/*.csv → contracts 表 upsert
input/materials/{cid}__{f}.pdf → materials_content (parse_method='pending')
"""
from service.importer import import_input_dir
conn = get_connection()
try:
result = import_input_dir(conn)
finally:
conn.close()
print(json.dumps(result, ensure_ascii=False, indent=2))
return 0
def cmd_download_materials(args: argparse.Namespace) -> int:
"""处理 download_tasks 队列:抓 pending 任务的材料 → 落库。
--ids "DLQ-xxx,DLQ-yyy": 只处理这些任务
--pick N: 无 --ids 时从 pending 领取 N 条(默认 5)
--limit N: 本次硬上限(默认 0 = 不限)
--headless: 无头模式(需要 profile cookie 已保存)
"""
try:
result = platform_collector.process_queue(
limit=args.limit or args.pick,
ids_filter=args.ids,
headless=args.headless,
human_wait_timeout=args.human_wait_timeout or HUMAN_WAIT_TIMEOUT,
)
except platform_collector.RPAError as e:
print(json.dumps({
"error": e.code,
"detail": str(e),
"processed": 0, "succeeded": 0, "failed": 0, "details": [],
}, ensure_ascii=False, indent=2))
return 2
except Exception as e:
print(json.dumps({"error": "exception", "detail": str(e)},
ensure_ascii=False, indent=2))
return 1
print(json.dumps(result, ensure_ascii=False, indent=2))
return 0
def cmd_show_config(_args: argparse.Namespace) -> int:
"""查看 .env 非空配置(敏感字段自动脱敏)。"""
from service.config_viewer import read_env_config
print(json.dumps(read_env_config(), ensure_ascii=False, indent=2))
return 0
def cmd_logs_list(args: argparse.Namespace) -> int:
"""读取日志末尾 N 行(默认 200)。"""
from service.logs_viewer import tail_log_file
result = tail_log_file(tail=args.tail, min_level=args.level)
print(json.dumps(result, ensure_ascii=False, indent=2))
return 0
# ---------- argparse ---------- # ---------- argparse ----------
def _parse_thresholds(s: str) -> tuple[int, int, int]: def _parse_thresholds(s: str) -> tuple[int, int, int]:
@@ -331,8 +410,13 @@ def build_parser() -> argparse.ArgumentParser:
) )
sp.add_argument("--limit", type=int, default=0, sp.add_argument("--limit", type=int, default=0,
help="限制抓取行数(0=全量)") help="限制抓取行数(0=全量)")
sp.add_argument("--show-browser", action="store_true", browser_mode = sp.add_mutually_exclusive_group()
help="显示浏览器(首次手工登录用,默认 headless)") browser_mode.add_argument("--headless", action="store_true",
help="无头运行(仅适合已保存有效登录态的自动化任务)")
browser_mode.add_argument("--show-browser", dest="headless", action="store_false",
help=argparse.SUPPRESS)
sp.add_argument("--human-wait-timeout", type=int, default=None,
help="手工登录 fallback 等待秒数(默认 180 = HUMAN_WAIT_TIMEOUT)")
sp.set_defaults(handler=cmd_collect_platform) sp.set_defaults(handler=cmd_collect_platform)
sp = sub.add_parser( sp = sub.add_parser(
@@ -342,6 +426,41 @@ def build_parser() -> argparse.ArgumentParser:
help="仅解析该合同(留空=处理所有 pending)") help="仅解析该合同(留空=处理所有 pending)")
sp.set_defaults(handler=cmd_parse_materials) sp.set_defaults(handler=cmd_parse_materials)
sp = sub.add_parser(
"download-materials",
help="处理材料下载队列(pending 任务 → 浏览器抓取 → 落库)",
)
sp.add_argument("--limit", type=int, default=0, help="本次最多处理任务数(0 = 用 --pick)")
sp.add_argument("--ids", default="", help="逗号分隔 task_id 列表(仅处理这些 pending 任务)")
sp.add_argument("--pick", type=int, default=5, help="无 --ids 时从 pending 领取条数(默认 5)")
sp.add_argument("--headless", action="store_true", help="无头模式(需已保存 profile)")
sp.add_argument("--show-browser", dest="headless", action="store_false",
help=argparse.SUPPRESS)
sp.add_argument("--human-wait-timeout", type=int, default=None,
help="手工登录 fallback 等待秒数(默认 180 = HUMAN_WAIT_TIMEOUT)")
sp.set_defaults(headless=False, handler=cmd_download_materials)
sp = sub.add_parser(
"import-input",
help="扫描 data/input/{contracts,materials}/ 自动入库",
)
sp.set_defaults(handler=cmd_import_input)
sp = sub.add_parser(
"show-config",
help="查看 .env 配置(敏感字段自动脱敏)",
)
sp.set_defaults(handler=cmd_show_config)
sp = sub.add_parser(
"logs-list",
help="查看运行日志末尾 N 行",
)
sp.add_argument("--tail", type=int, default=200, help="末尾行数(默认 200)")
sp.add_argument("--level", default="INFO",
help="最低级别 DEBUG/INFO/WARNING/ERROR(默认 INFO)")
sp.set_defaults(handler=cmd_logs_list)
return p return p
@@ -352,7 +471,8 @@ def main(argv: Optional[List[str]] = None) -> int:
print("Commands: health, version, config-path, init-db, " print("Commands: health, version, config-path, init-db, "
"scan-expiring-contracts, generate-renewal-task, " "scan-expiring-contracts, generate-renewal-task, "
"dispatch-task-notice, mark-contract-renewed, render-html, " "dispatch-task-notice, mark-contract-renewed, render-html, "
"collect-from-platform, parse-materials") "collect-from-platform, parse-materials, download-materials, "
"import-input, show-config, logs-list")
return 1 return 1
setup_skill_logging() setup_skill_logging()

View File

@@ -90,6 +90,25 @@ DDL_STATEMENTS: tuple[str, ...] = (
); );
""", """,
"CREATE INDEX IF NOT EXISTS idx_dispatch_unit_month ON dispatch_log(unit_id, month);", "CREATE INDEX IF NOT EXISTS idx_dispatch_unit_month ON dispatch_log(unit_id, month);",
# 7. download_tasks (v1.1.18)
"""
CREATE TABLE IF NOT EXISTS download_tasks (
task_id TEXT PRIMARY KEY,
contract_id TEXT NOT NULL,
kind TEXT NOT NULL DEFAULT 'platform_button',
status TEXT NOT NULL DEFAULT 'pending',
attempts INTEGER NOT NULL DEFAULT 0,
error_msg TEXT,
enqueued_at TEXT NOT NULL,
started_at TEXT,
finished_at TEXT,
source_path TEXT,
result_path TEXT,
FOREIGN KEY (contract_id) REFERENCES contracts(contract_id)
);
""",
"CREATE INDEX IF NOT EXISTS idx_dlq_status ON download_tasks(status);",
"CREATE INDEX IF NOT EXISTS idx_dlq_contract ON download_tasks(contract_id);",
) )
@@ -131,6 +150,7 @@ def init_db(db_path: Path | None = None) -> dict[str, int]:
"renewal_tasks": conn.execute("SELECT COUNT(*) AS n FROM renewal_tasks").fetchone()["n"], "renewal_tasks": conn.execute("SELECT COUNT(*) AS n FROM renewal_tasks").fetchone()["n"],
"dispatch_log": conn.execute("SELECT COUNT(*) AS n FROM dispatch_log").fetchone()["n"], "dispatch_log": conn.execute("SELECT COUNT(*) AS n FROM dispatch_log").fetchone()["n"],
"materials_content": conn.execute("SELECT COUNT(*) AS n FROM materials_content").fetchone()["n"], "materials_content": conn.execute("SELECT COUNT(*) AS n FROM materials_content").fetchone()["n"],
"download_tasks": conn.execute("SELECT COUNT(*) AS n FROM download_tasks").fetchone()["n"],
} }
return counts return counts
finally: finally:
@@ -140,5 +160,5 @@ def init_db(db_path: Path | None = None) -> dict[str, int]:
def table_names() -> Iterable[str]: def table_names() -> Iterable[str]:
return ( return (
"contracts", "performance_records", "alerts", "renewal_tasks", "contracts", "performance_records", "alerts", "renewal_tasks",
"dispatch_log", "materials_content", "dispatch_log", "materials_content", "download_tasks",
) )

View File

@@ -294,6 +294,59 @@ def list_dispatch_logs(conn: sqlite3.Connection) -> list[sqlite3.Row]:
# ---------- materials_content(v1.1.1 新增) ---------- # ---------- materials_content(v1.1.1 新增) ----------
def upsert_contract_from_csv_row(
conn: sqlite3.Connection, row: dict[str, str]
) -> None:
"""从 CSV 字典 upsert 一份合同。CSV 列名必须与 contracts 表列一致(或常见别名)。
缺字段兜底值:
employee_id = contract_id
teacher_name = ""
unit_id = "U-UNKNOWN"
unit_name = ""
position = ""
contract_type = ""
contract_start = "1970-01-01"
contract_end = "2099-12-31"
trial_period_end = contract_end
status = "可续聘"
"""
cid = row.get("contract_id") or row.get("合同ID") or ""
if not cid:
raise ValueError("CSV row missing contract_id / 合同ID")
conn.execute(
"""
INSERT INTO contracts
(contract_id, employee_id, teacher_name, unit_id, unit_name,
position, contract_type, contract_start, contract_end,
trial_period_end, status, last_material_id, renewed_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, NULL, NULL)
ON CONFLICT(contract_id) DO UPDATE SET
teacher_name = excluded.teacher_name,
unit_name = excluded.unit_name,
position = excluded.position,
contract_type = excluded.contract_type,
contract_start = excluded.contract_start,
contract_end = excluded.contract_end,
trial_period_end = excluded.trial_period_end,
status = excluded.status
""",
(
cid,
row.get("employee_id") or row.get("工号") or cid,
row.get("teacher_name") or row.get("教职工姓名") or "",
row.get("unit_id") or row.get("单位ID") or "U-UNKNOWN",
row.get("unit_name") or row.get("单位") or "",
row.get("position") or row.get("岗位") or "",
row.get("contract_type") or row.get("合同类型") or "",
row.get("contract_start") or row.get("合同起始日") or "1970-01-01",
row.get("contract_end") or row.get("合同到期日") or "2099-12-31",
row.get("trial_period_end") or row.get("试用期截止") or row.get("contract_end") or "2099-12-31",
row.get("status") or "可续聘",
),
)
def list_materials_for_contract( def list_materials_for_contract(
conn: sqlite3.Connection, contract_id: str conn: sqlite3.Connection, contract_id: str
) -> list[sqlite3.Row]: ) -> list[sqlite3.Row]:
@@ -373,3 +426,124 @@ def summarize_materials(
if len(summary) > 16000: if len(summary) > 16000:
summary = summary[:16000] + "\n...(材料过长已截断)" summary = summary[:16000] + "\n...(材料过长已截断)"
return (summary, file_names) return (summary, file_names)
# ---------- download_tasks(v1.1.18 新增) ----------
def enqueue_download_task(
conn: sqlite3.Connection,
contract_id: str,
kind: str = "platform_button",
) -> str:
"""入队一个下载任务,返回 task_id。
已存在 (contract_id, kind, status ∈ {pending, running}) 时跳过(返回已存在的 task_id)。
"""
import uuid
existing = conn.execute(
"""SELECT task_id FROM download_tasks
WHERE contract_id = ? AND kind = ? AND status IN ('pending', 'running')
LIMIT 1""",
(contract_id, kind),
).fetchone()
if existing:
return existing["task_id"]
task_id = f"DLQ-{uuid.uuid4().hex[:12]}"
conn.execute(
"""
INSERT INTO download_tasks
(task_id, contract_id, kind, status, attempts, enqueued_at)
VALUES (?, ?, ?, 'pending', 0, ?)
""",
(task_id, contract_id, kind, datetime.now().isoformat(timespec="seconds")),
)
return task_id
def _read_max_attempts(conn: sqlite3.Connection) -> int:
"""读 DOWNLOAD_QUEUE_MAX_ATTEMPTS env(默认 3)。"""
import os
return int(os.environ.get("DOWNLOAD_QUEUE_MAX_ATTEMPTS", "3"))
def claim_pending_tasks(
conn: sqlite3.Connection, limit: int, ids_filter: list[str] | None = None,
) -> list[sqlite3.Row]:
"""原子领取一批 pending 任务,标记为 running。
- limit: 最多领取条数
- ids_filter: 若给出,仅从这些 task_id 中领
"""
max_attempts = _read_max_attempts(conn)
if ids_filter:
placeholders = ",".join("?" for _ in ids_filter)
rows = conn.execute(
f"""SELECT * FROM download_tasks
WHERE status = 'pending' AND attempts < ?
AND task_id IN ({placeholders})
ORDER BY enqueued_at ASC LIMIT ?""",
[max_attempts, *ids_filter, limit],
).fetchall()
else:
rows = conn.execute(
"""SELECT * FROM download_tasks
WHERE status = 'pending' AND attempts < ?
ORDER BY enqueued_at ASC LIMIT ?""",
(max_attempts, limit),
).fetchall()
if not rows:
return []
now = datetime.now().isoformat(timespec="seconds")
for r in rows:
conn.execute(
"UPDATE download_tasks SET status = 'running', started_at = ? WHERE task_id = ?",
(now, r["task_id"]),
)
conn.commit()
return rows
def mark_task_success(
conn: sqlite3.Connection, task_id: str, result_path: str,
) -> None:
conn.execute(
"""UPDATE download_tasks
SET status = 'success', result_path = ?, finished_at = ?
WHERE task_id = ?""",
(result_path, datetime.now().isoformat(timespec="seconds"), task_id),
)
def mark_task_failed(
conn: sqlite3.Connection, task_id: str, error_msg: str,
) -> None:
"""失败一次:attempts+1,未超 max_attempts 则回退 pending,否则标 failed。"""
max_attempts = _read_max_attempts(conn)
now = datetime.now().isoformat(timespec="seconds")
conn.execute(
"""UPDATE download_tasks
SET status = CASE WHEN attempts + 1 >= ? THEN 'failed' ELSE 'pending' END,
attempts = attempts + 1,
error_msg = ?,
finished_at = ?
WHERE task_id = ?""",
(max_attempts, error_msg, now, task_id),
)
def list_recent_tasks(
conn: sqlite3.Connection, limit: int = 50
) -> list[sqlite3.Row]:
return conn.execute(
"SELECT * FROM download_tasks ORDER BY enqueued_at DESC LIMIT ?",
(limit,),
).fetchall()
def count_tasks_by_status(conn: sqlite3.Connection) -> dict[str, int]:
rows = conn.execute(
"SELECT status, COUNT(*) AS n FROM download_tasks GROUP BY status"
).fetchall()
return {r["status"]: r["n"] for r in rows}

View File

@@ -0,0 +1,77 @@
"""读取并脱敏 .env 配置供 UI 查看(v1.1.18)。
仅显示 .env 文件中的非空 key;命中 (TOKEN|SECRET|KEY|PASSWORD) 的字段自动脱敏为 ***。
"""
from __future__ import annotations
import re
from pathlib import Path
from util.runtime_paths import get_env_path
# 匹配任一含 TOKEN / SECRET / KEY / PASSWORD 的 key 名(不区分大小写)
_SECRET_PATTERN = re.compile(r"(TOKEN|SECRET|KEY|PASSWORD)", re.IGNORECASE)
def _parse_env_file(path: Path) -> dict[str, str]:
"""手动解析 .env(避免依赖 python-dotenv)。
- 跳过空行 / 注释(# 开头)
- 支持 KEY=VALUE 格式
- 自动去除包裹的 "..." 或 '...'
"""
if not path.exists():
return {}
result: dict[str, str] = {}
with open(path, "r", encoding="utf-8") as f:
for line in f:
line = line.strip()
if not line or line.startswith("#"):
continue
if "=" not in line:
continue
k, _, v = line.partition("=")
k = k.strip()
v = v.strip()
# 去掉行尾注释(不处理引号内的 #,简单实现)
if " #" in v:
v = v.split(" #", 1)[0].strip()
if v.startswith('"') and v.endswith('"') and len(v) >= 2:
v = v[1:-1]
elif v.startswith("'") and v.endswith("'") and len(v) >= 2:
v = v[1:-1]
result[k] = v
return result
def _mask_secret(value: str) -> str:
if not value:
return ""
if len(value) <= 4:
return "***"
return value[:2] + "***" + value[-2:]
def read_env_config() -> dict:
"""读取 .env 中的非空 key,敏感 key 脱敏为 ***。
Returns:
{
"env_path": str,
"exists": bool,
"config": {key: value_or_masked_value, ...}
}
"""
path = get_env_path()
parsed = _parse_env_file(path)
out: dict[str, str] = {}
for k, v in sorted(parsed.items()):
if not v:
continue
if _SECRET_PATTERN.search(k):
out[k] = _mask_secret(v)
else:
out[k] = v
return {"env_path": str(path), "exists": path.exists(), "config": out}

111
scripts/service/importer.py Normal file
View File

@@ -0,0 +1,111 @@
"""主动扫描 data/input/{contracts,materials}/:CSV → contracts 表,PDF/Word → materials_content 表。
对 sandbox 用户尤其重要:HR API 拿不到时,把材料直接丢进 input/materials/ 即可被自动识别。
约定:
- input/contracts/*.csv
列名与 contracts 表一致(contract_id / teacher_name / unit_name / contract_end / ...)
也兼容中文别名(合同ID / 教职工姓名 / 单位 / 合同到期日 / 工号 / 岗位 / 合同类型)
- input/materials/{contract_id}__{filename}.{pdf|docx|doc}
文件名 __ 之前是 contract_id;无 __ 时回退 parent dir 名
拷贝到 data/materials/ + insert materials_content(parse_method='pending')
"""
from __future__ import annotations
import csv
import shutil
import sqlite3
from pathlib import Path
from db import repository as repo
from db.migrations import ensure_materials_content_schema
from util.logging_setup import get_skill_logger
from util.runtime_paths import (
get_data_dir,
get_input_contracts_dir,
get_input_materials_dir,
)
logger = get_skill_logger()
MATERIAL_EXTS = {".pdf", ".docx", ".doc"}
def _derive_contract_id_from_filename(mat_path: Path) -> str:
"""从文件名推导 contract_id。约定:{contract_id}__{filename}.{ext}。"""
parts = mat_path.stem.split("__", 1)
if len(parts) == 2 and parts[0]:
return parts[0]
# 兜底:parent dir 名
return mat_path.parent.name or "UNKNOWN"
def import_input_dir(conn: sqlite3.Connection) -> dict:
"""扫描 input 目录 → 入库。返回统计 dict。
Returns:
{
"contracts_added": int,
"materials_added": int,
"files_scanned": int,
"errors": [str, ...],
}
"""
ensure_materials_content_schema(conn)
contracts_dir = get_input_contracts_dir()
materials_dir = get_input_materials_dir()
contracts_added = 0
materials_added = 0
files_scanned = 0
errors: list[str] = []
# 1. CSV → contracts(contract_id 重复 → upsert 覆盖)
for csv_path in sorted(contracts_dir.glob("*.csv")):
files_scanned += 1
try:
with open(csv_path, "r", encoding="utf-8", newline="") as f:
reader = csv.DictReader(f)
for row in reader:
if not row.get("contract_id") and not row.get("合同ID"):
errors.append(f"{csv_path.name}: row 缺 contract_id 跳过")
continue
repo.upsert_contract_from_csv_row(conn, row)
contracts_added += 1
except Exception as e:
errors.append(f"{csv_path.name}: {e}")
logger.warning("importer: CSV 解析失败 %s: %s", csv_path, e)
# 2. PDF/Word → materials_content(parse_method='pending'),文件拷贝到 data/materials/
materials_target_dir = get_data_dir() / "materials"
materials_target_dir.mkdir(parents=True, exist_ok=True)
for mat_path in sorted(materials_dir.iterdir()):
if not mat_path.is_file() or mat_path.suffix.lower() not in MATERIAL_EXTS:
continue
files_scanned += 1
try:
contract_id = _derive_contract_id_from_filename(mat_path)
target = materials_target_dir / mat_path.name
if not target.exists():
shutil.copy2(mat_path, target)
repo.insert_material(
conn, contract_id, mat_path.name, str(target),
None, "pending",
)
materials_added += 1
logger.info(
"importer: 入库 %s → contract_id=%s", mat_path.name, contract_id,
)
except Exception as e:
errors.append(f"{mat_path.name}: {e}")
logger.warning("importer: 材料导入失败 %s: %s", mat_path, e)
conn.commit()
return {
"contracts_added": contracts_added,
"materials_added": materials_added,
"files_scanned": files_scanned,
"errors": errors,
}

View File

@@ -0,0 +1,86 @@
"""读取日志文件末尾 N 行(v1.1.18)。
按 level 过滤:默认 INFO+。适用于 RotatingFileHandler 落地的日志文件。
"""
from __future__ import annotations
import os
from pathlib import Path
from util.runtime_paths import get_logs_dir
DEFAULT_TAIL = 200
_LEVEL_ORDER = {
"DEBUG": 10, "INFO": 20, "WARNING": 30, "ERROR": 40, "CRITICAL": 50,
}
def _resolve_log_path() -> Path:
"""解析日志文件路径。
优先级:
1. LOG_FILE env(绝对路径直接用,相对路径走 get_logs_dir())
2. 默认 get_logs_dir() / LOG_FILE_DEFAULT_NAME
"""
raw = os.environ.get("LOG_FILE", "")
if raw:
p = Path(raw)
if not p.is_absolute():
p = get_logs_dir() / raw
return p
from util.constants import LOG_FILE_DEFAULT_NAME
return get_logs_dir() / LOG_FILE_DEFAULT_NAME
def tail_log_file(tail: int = DEFAULT_TAIL, min_level: str = "INFO") -> dict:
"""读日志文件末尾 tail 行,过滤 level >= min_level。
Returns:
{
"log_file": str,
"exists": bool,
"total_lines": int,
"returned_lines": int,
"lines": [str, ...],
"min_level": str,
}
"""
log_path = _resolve_log_path()
if not log_path.exists():
return {
"log_file": str(log_path),
"exists": False,
"total_lines": 0,
"returned_lines": 0,
"lines": [],
"min_level": min_level,
}
text = log_path.read_text(encoding="utf-8", errors="replace")
all_lines = text.splitlines()
last_n = all_lines[-tail:] if tail > 0 else all_lines
threshold = _LEVEL_ORDER.get(min_level.upper(), 20)
filtered: list[str] = []
for line in last_n:
# 格式: "2026-09-15 12:34:56 [alert-contract-expiring-hr] INFO message"
# 取第 4 个 token (INFO/DEBUG/...)
parts = line.split()
if len(parts) >= 4 and parts[1].count(":") >= 2 and parts[3] in _LEVEL_ORDER:
line_level = parts[3]
if _LEVEL_ORDER.get(line_level, 0) >= threshold:
filtered.append(line)
else:
# 非标准行 / 解析不到 level:保留
filtered.append(line)
return {
"log_file": str(log_path),
"exists": True,
"total_lines": len(all_lines),
"returned_lines": len(filtered),
"lines": filtered,
"min_level": min_level,
}

View File

@@ -1,23 +1,23 @@
"""Playwright RPA:从 SmartCampus HR 平台抓合同表 + 续聘材料包。 """Playwright RPA:从 SmartCampus HR 平台抓合同表 + 续聘材料包。
设计要点: 设计要点(v1.1.6,按 skill-template examples/real_browser_rpa 设计模式):
- 使用 launch_persistent_context 复用浏览器登录态,避免密码硬编码 - 使用 launch_persistent_context 复用浏览器登录态,避免密码硬编码
- 首次使用需用户手工登录一次(CLI 默认 --headless false 便于交互) - 首次使用需用户手工登录一次(CLI 默认 --headless false 便于交互)
- 后续使用无头模式(profile 已存 cookie) - 后续使用无头模式(profile 已存 cookie)
- **登录策略**:自动 fill demo/123456(sandbox 已知凭据) + 失败时 fallback 手工登录等待(轮询 180s)
- 抓表格:DOM eval_on_selector_all → JSON - 抓表格:DOM eval_on_selector_all → JSON
- 抓材料:点 "续聘材料包" 按钮 → expect_download → 保存到本地 - 抓材料:点 "续聘材料包" 按钮 → 抓卡片 inner_text
依赖为可选:playwright 未安装时 import 抛错,上层 CLI 捕获并降级报错。 依赖为可选:playwright 未安装时 import 抛错,上层 CLI 捕获并降级报错。
导航流程(v1.1.1 强化版 — 强制 goto): 导航流程(v1.1.1 强化版 — 强制 goto):
1. goto 根域 1. goto 根域
2. 根域 login(如仍显示表单)→ 等跳到 /home 或 /select 2. 根域 login(如仍显示表单)→ 等跳到 /home 或 /select
3. 强制 goto /select 3. 强制 goto sandbox SPA login URL
4. 在 /select 页面 click 智慧校园 tile → 等 URL 含 /industries/campus 4. SPA login:自动 fill demo/123456,失败 fallback 手工等待(HUMAN_WAIT_TIMEOUT=180s)
5. 若 URL 含 /login,填 sandbox SPA 登录表单 → 等跳走 5. 强制 goto SPA_HOME
6. 强制 goto SPA_HOME 6. click 侧边栏 hr/contracts
7. click 侧边栏 hr/contracts 7. 等 table tbody tr(最长 ~40s)
8. 等 table tbody tr
""" """
from __future__ import annotations from __future__ import annotations
@@ -25,11 +25,38 @@ from __future__ import annotations
import os import os
import re import re
import sqlite3 import sqlite3
import time
from datetime import datetime from datetime import datetime
from pathlib import Path from pathlib import Path
from db import repository as repo from db import repository as repo
from util.constants import SKILL_SLUG from util.constants import (
CARD_APPEAR_WAIT_MS,
DEFAULT_PAGE_TIMEOUT_MS,
FULL_RENDER_WAIT_MS,
GOTO_TIMEOUT_MS,
HUMAN_POLL_INTERVAL,
HUMAN_WAIT_TIMEOUT,
POST_LOGIN_SETTLE_MS,
POST_RELOAD_LOGIN_HYDRATION_MS,
POST_RELOGIN_GOTO_WAIT_MS,
PUSHSTATE_SETTLE_MS,
RELOAD_DOMCONTENTLOADED_TIMEOUT_MS,
RELOAD_NETWORKIDLE_TIMEOUT_MS,
SIDEBAR_CLICK_URL_TIMEOUT_MS,
SIDEBAR_LINK_TIMEOUT_MS,
SPA_HOME_RETRY_DELAY_MS,
SPA_HOME_RETRY_MAX,
SPA_HYDRATION_MS,
SPA_LOGIN_INPUT_TIMEOUT_MS,
SPA_LOGIN_SETTLE_MS,
SPA_LOGIN_URL_TIMEOUT_MS,
TABLE_INITIAL_WAIT_MS,
TABLE_POLL_INTERVAL_MS,
TABLE_POLL_MAX_ATTEMPTS,
TABLE_TR_TIMEOUT_MS,
URL_AFTER_LOGIN_TIMEOUT_MS,
)
from util.logging_setup import get_skill_logger from util.logging_setup import get_skill_logger
from util.runtime_paths import ensure_dirs from util.runtime_paths import ensure_dirs
@@ -47,6 +74,47 @@ SPA_HOME_URL = PLATFORM_BASE + "/industries/campus/smartcampus/app/home"
SPA_LOGIN_URL = PLATFORM_BASE + "/industries/campus/smartcampus/login" SPA_LOGIN_URL = PLATFORM_BASE + "/industries/campus/smartcampus/login"
# ---------- 错误码(与 skill-template 对齐) ----------
class RPAError(Exception):
"""RPA 流程错误基类。"""
code: str = "RPA_ERROR"
def __init__(self, message: str):
super().__init__(message)
self.message = message
class LoginTimeoutError(RPAError):
"""手工登录超时(HUMAN_WAIT_TIMEOUT 秒内未检测到登录成功)。
错误码:ERROR:LOGIN_TIMEOUT(与 skill-template task_rpa.py 对齐)
"""
code = "ERROR:LOGIN_TIMEOUT"
class MissingBrowserError(RPAError):
"""未检测到 Chrome/Edge。
错误码:ERROR:MISSING_BROWSER(与 skill-template browser_session.py 对齐)
"""
code = "ERROR:MISSING_BROWSER"
class TableNotFoundError(RPAError):
"""40s 内未找到表格。
错误码:ERROR:TABLE_NOT_FOUND
"""
code = "ERROR:TABLE_NOT_FOUND"
# ---------- 工具函数 ----------
def _safe_getenv_dir(env_key: str, default_rel: str) -> Path: def _safe_getenv_dir(env_key: str, default_rel: str) -> Path:
raw = os.environ.get(env_key) raw = os.environ.get(env_key)
if raw: if raw:
@@ -56,7 +124,7 @@ def _safe_getenv_dir(env_key: str, default_rel: str) -> Path:
return (get_data_dir() / default_rel).resolve() return (get_data_dir() / default_rel).resolve()
def _goto(page, url: str, timeout: int = 20_000) -> None: def _goto(page, url: str, timeout: int = GOTO_TIMEOUT_MS) -> None:
"""带 networkidle / domcontentloaded 双兜底的 goto。""" """带 networkidle / domcontentloaded 双兜底的 goto。"""
try: try:
page.goto(url, wait_until="networkidle", timeout=timeout) page.goto(url, wait_until="networkidle", timeout=timeout)
@@ -82,6 +150,47 @@ def _dump_page(page, path: Path, label: str) -> None:
logger.warning("dump %s 失败: %s", label, e) logger.warning("dump %s 失败: %s", label, e)
def _is_logged_in(page) -> bool:
"""检测 sandbox SPA 是否已登录(URL 含 /app/ 且不含 /login 或 /signin)。
与 skill-template real_browser_rpa _is_logged_in 概念对齐(这里只看 URL,
因为 sandbox SPA 登录后 URL 稳定跳到 /app/...)。
"""
url_lower = page.url.lower()
return (
"/app/" in url_lower
and "/login" not in url_lower
and "/signin" not in url_lower
)
def _wait_for_manual_login(page, wait_sec: int = HUMAN_WAIT_TIMEOUT) -> bool:
"""手工登录 fallback:轮询 _is_logged_in,每 HUMAN_POLL_INTERVAL 秒一次。
与 skill-template task_rpa.py::_ensure_logged_in 设计对齐:
- 不自动破解,等用户在浏览器手工完成登录
- 屏幕打印 [登录] 提示
- 超时抛 LoginTimeoutError(ERROR:LOGIN_TIMEOUT)
自动 fill demo/123456 失败时调用此函数。
"""
print(
f"[登录] 自动登录失败,请在浏览器中完成登录"
f"(sandbox demo/123456 或真实账号),最多等待 {wait_sec} 秒..."
)
deadline = time.monotonic() + wait_sec
attempt = 0
while time.monotonic() < deadline:
attempt += 1
if _is_logged_in(page):
print(f"[登录] 检测到登录成功(attempt {attempt})")
return True
time.sleep(HUMAN_POLL_INTERVAL)
raise LoginTimeoutError(
f"自动登录失败,{wait_sec}s 内用户在浏览器中也未完成登录"
)
def _fill_root_login(page) -> bool: def _fill_root_login(page) -> bool:
"""根域登录页(如还在):fill demo/123456 → click Sign in。返回是否执行了登录。""" """根域登录页(如还在):fill demo/123456 → click Sign in。返回是否执行了登录。"""
user_input = page.locator('#jc-user').first user_input = page.locator('#jc-user').first
@@ -94,7 +203,7 @@ def _fill_root_login(page) -> bool:
try: try:
page.wait_for_url( page.wait_for_url(
lambda u: "/home" in u or "/select" in u or "/industries/" in u, lambda u: "/home" in u or "/select" in u or "/industries/" in u,
timeout=20_000, timeout=URL_AFTER_LOGIN_TIMEOUT_MS,
) )
except Exception as e: except Exception as e:
logger.warning("等根域登录跳转超时: %s", e) logger.warning("等根域登录跳转超时: %s", e)
@@ -111,60 +220,110 @@ def _click_campus_link(page) -> bool:
link.click() link.click()
logger.info("platform_collector: click 智慧校园 链接") logger.info("platform_collector: click 智慧校园 链接")
try: try:
page.wait_for_url(lambda u: "/industries/campus" in u, timeout=15_000) page.wait_for_url(lambda u: "/industries/campus" in u, timeout=SIDEBAR_LINK_TIMEOUT_MS // 2)
except Exception as e: except Exception as e:
logger.warning("等 sandbox 路径超时: %s;当前 URL=%s", e, page.url) logger.warning("等 sandbox 路径超时: %s;当前 URL=%s", e, page.url)
return True return True
def _fill_spa_login(page, settle_ms: int = 4000) -> bool: def _fill_spa_login(page, settle_ms: int = SPA_LOGIN_SETTLE_MS) -> bool:
"""sandbox SPA 登录(如还在登录页):fill demo/123456 → click 登录 → 等跳转。 """sandbox SPA 自动登录(如还在登录页):fill demo/123456 → click 登录 → 等跳转。
返回是否执行了登录(True = 已点登录按钮;调用方应继续等 URL 跳走)。 返回是否执行了登录(True = 已点登录按钮且 URL 已跳离 login;调用方可继续后续流程)。
失败时**不抛异常**,由调用方决定是否 fallback 到 _wait_for_manual_login。
修复要点(登录时间太短导致采集不到数据): v1.1.10 关键修复:**先检测 _is_logged_in,已登录态直接 return True 不再 fill**。
- 默认 settle_ms 从 0 提到 4000ms,给 SPA 异步渲染 + token 写入 cookie 时间 之前没这个检查,profile 复用时会被强制 demo fill 破坏 session。
- 等 username input 的 timeout 从 10s 提到 20s(mock 慢启动)
- click 按钮后显式 wait_for_url 含更长 timeout(30s),URL 真的不含 /login 才返回 True v1.1.11 关键修复:**sandbox SPA 登录 input 实际是 `type="tel"` + 无 autocomplete**,
- 多处过渡路径(/login, /signin, /auth/)一并判断,避免 SPA 中间态被误判为「已离开登录页」 之前 selector `input[autocomplete="username"]` 永远找不到,自动 fill 100% 失败。
改用 sandbox 实际 selector `#jc-user` / `#jc-pass`(来自 root 登录页)+ `button[type="submit"]`。
修复要点(v1.1.5/v1.1.6 登录时间太短导致采集不到数据):
- 默认 settle_ms = SPA_LOGIN_SETTLE_MS(4000ms),给 SPA 异步渲染 + token 写入 cookie 时间
- 等 username input 的 timeout = SPA_LOGIN_INPUT_TIMEOUT_MS(20s)
- click 按钮后 wait_for_url timeout = SPA_LOGIN_URL_TIMEOUT_MS(30s)
- 多处过渡路径(/login, /signin, /auth/)一并判断
""" """
# v1.1.10:先检查是否已登录(profile cookie 复用场景),已登录直接 return True
if _is_logged_in(page):
logger.info(
"platform_collector: 已登录态(profile cookie 复用),跳过自动 fill"
)
return True
url_lower = page.url.lower() url_lower = page.url.lower()
if "/login" not in url_lower and "/signin" not in url_lower and "/auth/" not in url_lower: if "/login" not in url_lower and "/signin" not in url_lower and "/auth/" not in url_lower:
return False return False
# v1.1.11:sandbox SPA 登录页 input 是 #jc-user / #jc-pass(与 root 登录页同 ID)
# 等输入框出现(带超时,避免 sandbox mock 慢启动卡死)
try: try:
page.wait_for_selector('input[autocomplete="username"]', timeout=20_000) page.wait_for_selector(
'#jc-user, input[type="tel"], input[autocomplete="username"]',
timeout=SPA_LOGIN_INPUT_TIMEOUT_MS,
)
except Exception as e: except Exception as e:
logger.warning("SPA login input 未出现: %s;当前 URL=%s", e, page.url) logger.warning("SPA login input 未出现: %s;当前 URL=%s", e, page.url)
return False return False
spa_user = page.locator('input[autocomplete="username"]').first # v1.1.11:按优先级选 selector —— sandbox 实际用 #jc-user
if spa_user.count() == 0: user_sel = None
logger.warning("SPA login input count=0") for sel in ('#jc-user', 'input[type="tel"]', 'input[autocomplete="username"]'):
loc = page.locator(sel).first
if loc.count() > 0:
user_sel = sel
break
if not user_sel:
logger.warning("SPA login input selector 全 miss(#jc-user / type=tel / autocomplete=username)")
return False return False
spa_user.fill("demo") page.locator(user_sel).first.fill("demo")
page.locator('input[autocomplete="current-password"]').first.fill("123456") # 密码 selector 同样按优先级
pass_sel = None
for sel in ('#jc-pass', 'input[type="password"]', 'input[autocomplete="current-password"]'):
loc = page.locator(sel).first
if loc.count() > 0:
pass_sel = sel
break
if not pass_sel:
logger.warning("SPA password input selector 全 miss")
return False
page.locator(pass_sel).first.fill("123456")
page.locator( page.locator(
'button:has-text("登录"), button[type="submit"]:has-text("Sign in")' 'button:has-text("登录"), button[type="submit"]:has-text("Sign in"), '
'button[type="submit"]'
).first.click() ).first.click()
logger.info("platform_collector: sandbox SPA 自动登录...") logger.info("platform_collector: sandbox SPA 自动登录 (user_sel=%s pass_sel=%s)", user_sel, pass_sel)
# click 后给 SPA 一点时间把 token 写入 cookie / 触发跳转
page.wait_for_timeout(settle_ms) page.wait_for_timeout(settle_ms)
# 再等到 URL 真的跳离 login 页(最长 30s;之前 20s 太短)
try: try:
page.wait_for_url( page.wait_for_url(
lambda u: "/login" not in u.lower() lambda u: "/login" not in u.lower()
and "/signin" not in u.lower() and "/signin" not in u.lower()
and "/auth/" not in u.lower(), and "/auth/" not in u.lower(),
timeout=30_000, timeout=SPA_LOGIN_URL_TIMEOUT_MS,
) )
except Exception as e: except Exception as e:
logger.warning("SPA 登录超时: %s;当前 URL=%s", e, page.url) logger.warning("SPA 登录超时: %s;当前 URL=%s", e, page.url)
return False return False
# v1.1.10:fill 后再检测 _is_logged_in,避免 sandbox demo 凭据失效时
# 被误判为「已登录」(URL 跳 /app/home 但 cookie 没写入)
if not _is_logged_in(page):
logger.warning(
"platform_collector: SPA URL 跳离 login 但 _is_logged_in=False,登录可能失败"
)
return False
logger.info("platform_collector: SPA 登录后 URL=%s", page.url) logger.info("platform_collector: SPA 登录后 URL=%s", page.url)
return True return True
def collect_all(limit: int = 0, headless: bool = True) -> dict: def collect_all(
"""登录 sandbox 平台 → 抓合同表 + 材料 → 落库。""" limit: int = 0,
headless: bool = True,
human_wait_timeout: int = HUMAN_WAIT_TIMEOUT,
) -> dict:
"""登录 sandbox 平台 → 抓合同表 + 材料 → 落库。
human_wait_timeout: 手工登录 fallback 等待秒数(默认 HUMAN_WAIT_TIMEOUT=180)。
失败时可能抛出 RPAError(LoginTimeoutError / MissingBrowserError / TableNotFoundError)。
"""
try: try:
from playwright.sync_api import sync_playwright from playwright.sync_api import sync_playwright
except ImportError: except ImportError:
@@ -177,8 +336,9 @@ def collect_all(limit: int = 0, headless: bool = True) -> dict:
download_dir.mkdir(parents=True, exist_ok=True) download_dir.mkdir(parents=True, exist_ok=True)
logger.info( logger.info(
"platform_collector: 启动 profile=%s download_dir=%s headless=%s limit=%d", "platform_collector: 启动 profile=%s download_dir=%s headless=%s limit=%d "
profile_dir, download_dir, headless, limit, "human_wait_timeout=%d",
profile_dir, download_dir, headless, limit, human_wait_timeout,
) )
contracts: list[dict] = [] contracts: list[dict] = []
@@ -186,55 +346,133 @@ def collect_all(limit: int = 0, headless: bool = True) -> dict:
downloaded = 0 downloaded = 0
with sync_playwright() as p: with sync_playwright() as p:
ctx = p.chromium.launch_persistent_context( # v1.1.9:headed 模式优先用系统 Edge(跟 Windows 桌面 session 同源,
user_data_dir=str(profile_dir), # 能真正弹窗让你看到并手工登录);找不到则 fallback Playwright 内置 chromium
headless=headless, if not headless:
accept_downloads=True, try:
viewport={"width": 1440, "height": 900}, ctx = p.chromium.launch_persistent_context(
) user_data_dir=str(profile_dir),
headless=False,
accept_downloads=True,
channel="msedge",
viewport={"width": 1440, "height": 900},
)
logger.info(
"platform_collector: 使用系统 Edge(headed 弹窗)"
)
except Exception as e:
logger.warning(
"platform_collector: 系统 Edge 启动失败(%s),fallback Playwright chromium",
e,
)
ctx = p.chromium.launch_persistent_context(
user_data_dir=str(profile_dir),
headless=False,
accept_downloads=True,
viewport={"width": 1440, "height": 900},
)
else:
ctx = p.chromium.launch_persistent_context(
user_data_dir=str(profile_dir),
headless=True,
accept_downloads=True,
viewport={"width": 1440, "height": 900},
)
try: try:
page = ctx.new_page() page = ctx.new_page()
# v1.1.6:覆盖 Playwright 默认操作超时(之前无 set_default_timeout,导致
# 所有 wait_for_selector 默认 30s,调试时不直观)
page.set_default_timeout(DEFAULT_PAGE_TIMEOUT_MS)
# 1. 根域 goto + 根域登录(可选;root login 经常 demo/123456 失败) # 1. v1.1.11:先用**最小侵入**方式检测已登录态 —— 直接 goto SPA_HOME_URL
_goto(page, ROOT_URL) # (不要先跳 ROOT_URL + 自动 fill,会破坏已登录态)
_fill_root_login(page) # v1.1.11 改进:探针结果 + 二次等待 + cookie 检测,三重判断避免误判
logger.info(
"platform_collector: 检测登录态,goto SPA_HOME 先探 (当前 URL=%s)",
page.url,
)
_goto(page, SPA_HOME_URL)
# v1.1.11:等 sandbox SPA 异步验证 cookie(POST_LOGIN_SETTLE_MS → SPA_HYDRATION_MS)
page.wait_for_timeout(SPA_HYDRATION_MS)
# v1.1.11:二次等待 —— sandbox SPA 有时首次访问会先跳一个中间过渡页
# (如 /select),等几秒看 URL 是否稳定到 /app/
first_url = page.url
page.wait_for_timeout(SPA_HOME_RETRY_DELAY_MS)
second_url = page.url
if first_url != second_url:
logger.info(
"platform_collector: 探针 URL 过渡 %s → %s,等稳定", first_url, second_url,
)
page.wait_for_timeout(SPA_HOME_RETRY_DELAY_MS)
# 2. 直接 goto sandbox SPA login(绕过 /select 和 root 跳转) if _is_logged_in(page):
logger.info("platform_collector: 强制 goto sandbox SPA login, 当前 URL=%s", page.url) logger.info(
_goto(page, SPA_LOGIN_URL) "platform_collector: ✅ 已登录态,**完全跳过登录流程**(不弹窗、不自动 fill、不打扰)"
# SPA hydration + framework 初始化时间(修复登录时间太短的核心点) ",URL=%s", page.url,
page.wait_for_timeout(3000) )
else:
logger.info(
"platform_collector: 未登录态(SPA_HOME 跳回 login, URL=%s),走登录分支",
page.url,
)
# 2. 未登录:直接 goto sandbox SPA login(绕过 /select 和 root 跳转)
_goto(page, SPA_LOGIN_URL)
# SPA hydration + framework 初始化时间(修复登录时间太短的核心点)
page.wait_for_timeout(SPA_HYDRATION_MS)
# 3. sandbox SPA 登录(demo/123456 已知可用) # 3. sandbox SPA 自动登录(demo/123456 已知可用)
# 等待 URL 真的跳离 login(settle_ms 已给到 4s,wait_for_url 又给 30s) # 失败 fallback 手工登录等待(轮询 _is_logged_in + 180s 兜底)
if _fill_spa_login(page): if not _fill_spa_login(page):
# 等 SPA 路由稳定(之前这里没有 wait,直接 goto home 会打断 pending 跳转) logger.warning(
page.wait_for_timeout(2000) "platform_collector: 自动 fill 失败(URL=%s),fallback 手工登录等待",
page.url,
)
if not headless:
_wait_for_manual_login(page, wait_sec=human_wait_timeout)
else:
# headless 模式无法手工登录:抛 LoginTimeoutError 让 CLI 决定
raise LoginTimeoutError(
f"headless 模式下自动 fill 失败,且无法手工登录"
f"(请用 --show-browser 重试)"
)
else:
# 等 SPA 路由稳定(之前这里没有 wait,直接 goto home 会打断 pending 跳转)
page.wait_for_timeout(POST_LOGIN_SETTLE_MS)
# 4. 强制 goto SPA home(如果还停在 login,重试登录一次) # 4. 强制 goto SPA home(如果还停在 login,重试登录一次)
for retry in range(2): for retry in range(SPA_HOME_RETRY_MAX):
if "/app/" in page.url and "/login" not in page.url.lower(): if _is_logged_in(page):
break break
logger.info( logger.info(
"platform_collector: 强制 goto SPA_HOME (retry=%d), 当前 URL=%s", "platform_collector: 强制 goto SPA_HOME (retry=%d), 当前 URL=%s",
retry, page.url, retry, page.url,
) )
_goto(page, SPA_HOME_URL) _goto(page, SPA_HOME_URL)
page.wait_for_timeout(2500) page.wait_for_timeout(SPA_HOME_RETRY_DELAY_MS)
if _fill_spa_login(page): if _fill_spa_login(page):
page.wait_for_timeout(2000) page.wait_for_timeout(POST_LOGIN_SETTLE_MS)
# 4.5. retry 后若仍未登录,headless 模式抛超时
if not _is_logged_in(page):
if headless:
raise LoginTimeoutError(
"SPA_HOME retry 全部失败,URL 仍未离开 login 页"
)
_wait_for_manual_login(page, wait_sec=human_wait_timeout)
# 5. 等 SPA 渲染 + click 侧边栏 hr/contracts(SPA 路由,避免 goto 触发整页 reload 丢 session) # 5. 等 SPA 渲染 + click 侧边栏 hr/contracts(SPA 路由,避免 goto 触发整页 reload 丢 session)
try: try:
# dump SPA home 一次(用于调试 selector) # dump SPA home 一次(用于调试 selector)
page.wait_for_timeout(2000) page.wait_for_timeout(POST_LOGIN_SETTLE_MS)
_dump_page(page, download_dir.parent / "spa_home_dump.html", "SPA home") _dump_page(page, download_dir.parent / "spa_home_dump.html", "SPA home")
# 等 SPA 真正加载完(侧边栏 link 出现,最长 30s) # 等 SPA 真正加载完(侧边栏 link 出现,最长 SIDEBAR_LINK_TIMEOUT_MS)
# 修复:之前 15s 太短,sandbox mock 加载慢时直接 fallback 到 pushState → 表格加载不出来 # 修复:之前 15s 太短,sandbox mock 加载慢时直接 fallback 到 pushState → 表格加载不出来
try: try:
page.wait_for_selector( page.wait_for_selector(
"a[href$='/hr/contracts']", timeout=30_000, state="attached" "a[href$='/hr/contracts']",
timeout=SIDEBAR_LINK_TIMEOUT_MS,
state="attached",
) )
except Exception as e: except Exception as e:
logger.warning("等侧边栏 hr/contracts link 超时: %s", e) logger.warning("等侧边栏 hr/contracts link 超时: %s", e)
@@ -245,7 +483,8 @@ def collect_all(limit: int = 0, headless: bool = True) -> dict:
logger.info("platform_collector: click 侧边栏 hr/contracts") logger.info("platform_collector: click 侧边栏 hr/contracts")
try: try:
page.wait_for_url( page.wait_for_url(
lambda u: "/hr/contracts" in u, timeout=10_000 lambda u: "/hr/contracts" in u,
timeout=SIDEBAR_CLICK_URL_TIMEOUT_MS,
) )
except Exception as e: except Exception as e:
logger.warning("等 /hr/contracts URL 超时: %s;当前 URL=%s", logger.warning("等 /hr/contracts URL 超时: %s;当前 URL=%s",
@@ -263,7 +502,7 @@ def collect_all(limit: int = 0, headless: bool = True) -> dict:
"window.dispatchEvent(new PopStateEvent('popstate'))," "window.dispatchEvent(new PopStateEvent('popstate')),"
" '/industries/campus/smartcampus/app/hr/contracts'" " '/industries/campus/smartcampus/app/hr/contracts'"
) )
page.wait_for_timeout(1500) page.wait_for_timeout(PUSHSTATE_SETTLE_MS)
except Exception as e: except Exception as e:
logger.warning("SPA pushState 失败: %s", e) logger.warning("SPA pushState 失败: %s", e)
_goto(page, PLATFORM_URL) _goto(page, PLATFORM_URL)
@@ -273,62 +512,118 @@ def collect_all(limit: int = 0, headless: bool = True) -> dict:
logger.info("platform_collector: 当前 URL=%s", page.url) logger.info("platform_collector: 当前 URL=%s", page.url)
# 5.5. 进入页面后刷新(SPA 进入新路由时 mock 数据可能未加载,需刷新触发) # 5.5. v1.1.8:**首次登录后不再强制刷新**,避免打断用户手工登录态。
try: # 改为「等表格出现」兜底:给 SPA 一个初始等待窗口(TABLE_INITIAL_WAIT_MS*2),
logger.info("platform_collector: 进入 hr/contracts 后刷新页面") # 如果表格仍未出现,**才**刷新一次(sandbox mock 异步加载场景)。
page.reload(wait_until="networkidle", timeout=30_000) logger.info(
except Exception as e: "platform_collector: 进入 hr/contracts 后**不刷新**,等待表格出现"
logger.warning("刷新 networkidle 失败: %s;尝试 domcontentloaded", e) "(首次登录态不要打断)"
try: )
page.reload(wait_until="domcontentloaded", timeout=30_000)
except Exception as e2:
logger.warning("刷新 domcontentloaded 也失败: %s", e2)
# 5.6. 刷新后可能被踢回 login(session 丢失),重新登录一次 # 5.6. 兜底刷新:仅当等表格超时后才触发;刷新后可能被踢回 login,需重新登录
# 修复:之前没检查 URL,刷新后停在 login 页继续往后走,找不到表格就 dump 报错 # 修复:之前没检查 URL,刷新后停在 login 页继续往后走,找不到表格就 dump 报错
if "/login" in page.url.lower() or "/signin" in page.url.lower(): def _try_reload_after_table_timeout():
logger.warning( logger.warning(
"platform_collector: 刷新后回到登录页(%s),重新登录", "platform_collector: 表格未在 %dms 内出现,触发兜底刷新",
page.url, TABLE_INITIAL_WAIT_MS * 2,
) )
_goto(page, SPA_LOGIN_URL) try:
page.wait_for_timeout(3000) page.reload(
if _fill_spa_login(page): wait_until="networkidle",
page.wait_for_timeout(2000) timeout=RELOAD_NETWORKIDLE_TIMEOUT_MS,
_goto(page, PLATFORM_URL) )
page.wait_for_timeout(2000) except Exception as e:
logger.warning("刷新 networkidle 失败: %s;尝试 domcontentloaded", e)
try:
page.reload(
wait_until="domcontentloaded",
timeout=RELOAD_DOMCONTENTLOADED_TIMEOUT_MS,
)
except Exception as e2:
logger.warning("刷新 domcontentloaded 也失败: %s", e2)
if "/login" in page.url.lower() or "/signin" in page.url.lower():
logger.warning(
"platform_collector: 刷新后回到登录页(%s),重新登录",
page.url,
)
_goto(page, SPA_LOGIN_URL)
page.wait_for_timeout(POST_RELOAD_LOGIN_HYDRATION_MS)
if not _fill_spa_login(page):
if not headless:
_wait_for_manual_login(page, wait_sec=human_wait_timeout)
else:
raise LoginTimeoutError(
"刷新后回到 login 页且 headless 无法手工登录"
)
else:
page.wait_for_timeout(POST_LOGIN_SETTLE_MS)
_goto(page, PLATFORM_URL)
page.wait_for_timeout(POST_RELOGIN_GOTO_WAIT_MS)
# 7. dump 当前 hr/contracts 页面(用于调试) # 7. dump 当前 hr/contracts 页面(用于调试)
_dump_page(page, download_dir.parent / "hr_contracts_dump.html", "hr/contracts") _dump_page(page, download_dir.parent / "hr_contracts_dump.html", "hr/contracts")
# 8. 等待表格加载(SPA 异步填充 mock 数据,多等几秒) # 8. 等待表格加载(v1.1.8:先等,**表格超时未出现才触发兜底刷新**——
page.wait_for_timeout(3000) # 给 React 初始化 + mock 数据填充时间 # 修复「首次登录后强制刷新打断手工登录态」问题)
page.wait_for_timeout(TABLE_INITIAL_WAIT_MS) # 给 React 初始化 + mock 数据填充时间
found_table = False found_table = False
for attempt in range(20): for attempt in range(TABLE_POLL_MAX_ATTEMPTS):
page.wait_for_timeout(2000) page.wait_for_timeout(TABLE_POLL_INTERVAL_MS)
cur_url = page.url cur_url = page.url
try: try:
page.wait_for_selector("table tbody tr", timeout=3000) page.wait_for_selector(
"table tbody tr",
timeout=TABLE_TR_TIMEOUT_MS,
)
found_table = True found_table = True
break break
except Exception: except Exception:
logger.info( logger.info(
"platform_collector: URL=%s 等待表格加载 (attempt %d/20)...", "platform_collector: URL=%s 等待表格加载 (attempt %d/%d)...",
cur_url, attempt + 1, cur_url, attempt + 1, TABLE_POLL_MAX_ATTEMPTS,
) )
if not found_table:
# v1.1.8:表格没出现,**先尝试兜底刷新一次**(sandbox mock 异步加载场景);
# 刷新后再次轮询表格,仍找不到才报 TableNotFoundError。
logger.warning(
"platform_collector: 首次轮询未找到表格,触发兜底刷新(sandbox mock 异步加载场景)"
)
_try_reload_after_table_timeout()
# 刷新后重新轮询一次(最多 5 次)
for attempt2 in range(5):
page.wait_for_timeout(TABLE_POLL_INTERVAL_MS)
try:
page.wait_for_selector(
"table tbody tr",
timeout=TABLE_TR_TIMEOUT_MS,
)
found_table = True
logger.info(
"platform_collector: 兜底刷新后找到表格 (attempt %d)", attempt2 + 1
)
break
except Exception:
logger.info(
"platform_collector: 兜底刷新后仍等待表格 (attempt %d/5)...",
attempt2 + 1,
)
if not found_table: if not found_table:
_dump_page(page, download_dir.parent / "sandbox_dump.html", "sandbox") _dump_page(page, download_dir.parent / "sandbox_dump.html", "sandbox")
logger.warning("platform_collector: 40s 后未找到表格") logger.warning("platform_collector: 表格未找到(兜底刷新后仍未出现)")
return _error_result("table_not_found_after_40s", 0) raise TableNotFoundError(
f"{TABLE_POLL_MAX_ATTEMPTS * TABLE_POLL_INTERVAL_MS // 1000}s "
f"内未找到表格(兜底刷新后仍未出现) (URL={page.url})"
)
# 表格已找到,等 tbody 完全填充 + dump 一份完整 DOM(用于诊断按钮 selector) # 表格已找到,等 tbody 完全填充 + dump 一份完整 DOM(用于诊断按钮 selector)
try: try:
page.wait_for_function( page.wait_for_function(
"document.querySelectorAll('table tbody tr').length > 0", "document.querySelectorAll('table tbody tr').length > 0",
timeout=10_000, timeout=DEFAULT_PAGE_TIMEOUT_MS,
) )
page.wait_for_timeout(1500) page.wait_for_timeout(FULL_RENDER_WAIT_MS)
_dump_page( _dump_page(
page, page,
download_dir.parent / "hr_contracts_full_dump.html", download_dir.parent / "hr_contracts_full_dump.html",
@@ -361,27 +656,53 @@ def collect_all(limit: int = 0, headless: bool = True) -> dict:
for i, r in enumerate(rows): for i, r in enumerate(rows):
if limit and i >= limit: if limit and i >= limit:
break break
if len(r) < 7: # v1.1.8:表格列数自适应 —— sandbox 首次登录未刷新是 7 列
logger.warning("行 %d 列数 < 7(新列序工号+姓名+单位+岗位+合同类型+起止+剩余+预警+建议),跳过: %s", i, r) # (姓名/单位/到期日/剩余天数/预警/建议/按钮),刷新后才是 10 列
# (多 工号/岗位/合同类型/起止日期)。两种都支持。
ncols = len(r)
if ncols < 6:
logger.warning(
"行 %d 列数=%d < 6,跳过: %s", i, ncols, r
)
continue continue
# 新列序(sandbox 加了工号 / 岗位 / 合同类型 / 起止日期): # 列序探测:判断首列是不是工号(纯数字 + 字母,纯姓名是中文)
# 0: 工号 1: 姓名 2: 单位 3: 岗位 4: 合同类型 first_col = r[0].strip()
# 5: 起止日期 (含 ~ ) 6: 剩余天数 7: 预警 8: 建议(含原因) import re as _re
emp_id = r[0].strip() is_emp_id = bool(_re.match(r"^[A-Za-z]?\d+$", first_col))
teacher = r[1].strip()
unit = r[2].strip() if is_emp_id and ncols >= 9:
position = r[3].strip() # 10 列布局(含工号):0 工号 1 姓名 2 单位 3 岗位 4 合同类型
contract_type = r[4].strip() # 5 起止(含~) 6 剩余天数 7 预警 8 建议 9 按钮
period = r[5].strip() emp_id = first_col
# 起止日期形如 "2024-03-01 ~ 2026-04-15" → 取 ~ 后 teacher = r[1].strip()
contract_end = period.split("~")[-1].strip() if "~" in period else period unit = r[2].strip()
days_raw = r[6].strip() period = r[5].strip()
alert_label = r[7].strip() contract_end = period.split("~")[-1].strip() if "~" in period else period
suggestion_hint = r[8].strip() if len(r) > 8 else "" days_raw = r[6].strip()
alert_label = r[7].strip()
suggestion_hint = r[8].strip()
elif ncols >= 6:
# 7 列布局(无工号):0 姓名 1 单位 2 到期日 3 剩余天数 4 预警 5 建议 6 按钮
teacher = first_col
unit = r[1].strip()
contract_end = r[2].strip()
days_raw = r[3].strip()
alert_label = r[4].strip()
suggestion_hint = r[5].strip()
emp_id = "" # 工号缺失,留空让 _gen_contract_id 用 teacher_name 兜底
else:
logger.warning(
"行 %d 列数=%d 无法识别布局,跳过: %s", i, ncols, r
)
continue
# 用工号生成 contract_id(稳定,不会随姓名变化) # 用工号生成 contract_id(稳定,不会随姓名变化)
contract_id = _gen_contract_id(emp_id, contract_end) # v1.1.8:7 列布局无工号 → 用 teacher_name 派生(避免与 10 列布局冲突)
if emp_id:
contract_id = _gen_contract_id(emp_id, contract_end)
else:
contract_id = _gen_contract_id(teacher, contract_end)
contract = { contract = {
"contract_id": contract_id, "contract_id": contract_id,
"teacher_name": teacher, "teacher_name": teacher,
@@ -401,8 +722,7 @@ def collect_all(limit: int = 0, headless: bool = True) -> dict:
) )
btn.click() btn.click()
# 等卡片出现(页面下方追加 h2「续聘准备任务书」/「单位清单」) # 等卡片出现(页面下方追加 h2「续聘准备任务书」/「单位清单」)
# 修复:1500ms 太短,sandbox mock 渲染卡片动画慢,导致抓到的文本不全 page.wait_for_timeout(CARD_APPEAR_WAIT_MS)
page.wait_for_timeout(3000)
# 抓表格下方所有 card 文本 # 抓表格下方所有 card 文本
cards_text = page.locator( cards_text = page.locator(
"main .grid > div, main > div > div > div" "main .grid > div, main > div > div > div"
@@ -545,11 +865,265 @@ def _persist_contracts(contracts: list[dict]) -> int:
datetime.now().isoformat(timespec="seconds"), datetime.now().isoformat(timespec="seconds"),
), ),
) )
# v1.1.18:每个合同入队 download_tasks(即使已抓到卡片也入队,
# process_queue 失败时可用作重试锚点;enqueue_download_task 幂等)
repo.enqueue_download_task(conn, c["contract_id"], "platform_button")
return len(contracts) return len(contracts)
finally: finally:
conn.close() conn.close()
def _download_materials_for_row(
page, row_idx: int, contract_id: str, mat_btn_count: int,
) -> list[dict]:
"""点续聘材料包按钮抓卡片文本(与 v1.1.17 collect_all 内联逻辑等价)。
返回 materials 列表(kind=sandbox_card,含 file_name + text)。
"""
if mat_btn_count == 0:
return []
materials: list[dict] = []
try:
btn = page.locator("table tbody tr").nth(row_idx).locator(
"button:has-text('续聘材料包')"
)
btn.click()
page.wait_for_timeout(CARD_APPEAR_WAIT_MS)
cards_text = page.locator(
"main .grid > div, main > div > div > div"
).all_inner_texts()
materials_text = "\n\n".join(t.strip() for t in cards_text if t.strip())
if materials_text:
materials.append({
"kind": "sandbox_card",
"file_name": f"{contract_id}_materials.txt",
"text": materials_text,
})
logger.info(
"platform_collector: 抓到卡片文本 %d 字 (%s)",
len(materials_text), contract_id,
)
except Exception as e:
logger.warning("抓卡片失败 %s: %s", contract_id, e)
return materials
def _find_row_for_contract(page, contract_id: str) -> int | None:
"""在 hr/contracts 表格中找 contract_id 所在行号。
支持 7 列(用 teacher_name 派生 contract_id)与 10 列(用 emp_id)。
不匹配返回 None(调用方 mark_task_failed)。
"""
rows = page.eval_on_selector_all(
"table tbody tr",
"rs => rs.map(r => Array.from(r.querySelectorAll('td')).map(td => td.innerText))",
)
for i, r in enumerate(rows):
ncols = len(r)
if ncols < 6:
continue
first_col = r[0].strip()
is_emp_id = bool(re.match(r"^[A-Za-z]?\d+$", first_col))
if is_emp_id and ncols >= 9:
emp_id, contract_end = first_col, r[5].split("~")[-1].strip()
elif ncols >= 6:
emp_id, contract_end = first_col, r[2].strip()
else:
continue
candidate = _gen_contract_id(emp_id, contract_end)
if candidate == contract_id:
return i
return None
def process_queue(
limit: int = 5,
headless: bool = True,
human_wait_timeout: int = HUMAN_WAIT_TIMEOUT,
ids_filter: str = "",
) -> dict:
"""从 download_tasks 队列取 pending 任务,对每个合同重新走材料抓取。
与 collect_all 重复登录流程,但每行只点「续聘材料包」按钮抓卡片,不重新抓合同表。
参数:
limit: 本次最多处理任务数(0 = 不限)
ids_filter: 逗号分隔 task_id 列表(只处理这些 pending)
headless: 无头模式(需要 profile cookie 已保存)
human_wait_timeout: 手工登录 fallback 等待秒数
返回 {processed, succeeded, failed, details}。
"""
try:
from playwright.sync_api import sync_playwright
except ImportError:
logger.error("playwright 未安装,跳过队列处理")
return {"processed": 0, "succeeded": 0, "failed": 0, "details": [],
"error": "playwright_not_installed"}
from db.connection import get_connection
# 1. 领取一批任务
ids_list = [s.strip() for s in ids_filter.split(",") if s.strip()] if ids_filter else None
conn = get_connection()
try:
tasks = repo.claim_pending_tasks(
conn,
limit=limit if limit > 0 else 999,
ids_filter=ids_list,
)
finally:
conn.close()
if not tasks:
return {"processed": 0, "succeeded": 0, "failed": 0, "details": []}
profile_dir = _safe_getenv_dir("SMARTCAMPUS_PROFILE_DIR", "playwright-profile")
download_dir = _safe_getenv_dir("MATERIALS_DOWNLOAD_DIR", "materials")
profile_dir.mkdir(parents=True, exist_ok=True)
download_dir.mkdir(parents=True, exist_ok=True)
details: list[dict] = []
succeeded = 0
failed = 0
with sync_playwright() as p:
ctx = p.chromium.launch_persistent_context(
user_data_dir=str(profile_dir),
headless=headless,
accept_downloads=True,
viewport={"width": 1440, "height": 900},
channel="msedge" if not headless else None,
) if not headless else p.chromium.launch_persistent_context(
user_data_dir=str(profile_dir),
headless=True,
accept_downloads=True,
viewport={"width": 1440, "height": 900},
)
try:
page = ctx.new_page()
page.set_default_timeout(DEFAULT_PAGE_TIMEOUT_MS)
# 登录:复用 v1.1.17 collect_all 的探针 + 自动 fill + 手工 fallback 流程
logger.info("platform_collector.process_queue: 启动 profile=%s headless=%s 待处理 %d",
profile_dir, headless, len(tasks))
_goto(page, SPA_HOME_URL)
page.wait_for_timeout(SPA_HYDRATION_MS)
if _is_logged_in(page):
logger.info("platform_collector.process_queue: ✅ 已登录态")
else:
_goto(page, SPA_LOGIN_URL)
page.wait_for_timeout(SPA_HYDRATION_MS)
if not _fill_spa_login(page):
if not headless:
_wait_for_manual_login(page, wait_sec=human_wait_timeout)
else:
raise LoginTimeoutError("process_queue headless 自动 fill 失败")
else:
page.wait_for_timeout(POST_LOGIN_SETTLE_MS)
# 强制 goto SPA_HOME + 侧边栏 hr/contracts
for _ in range(SPA_HOME_RETRY_MAX):
if _is_logged_in(page):
break
_goto(page, SPA_HOME_URL)
page.wait_for_timeout(SPA_HOME_RETRY_DELAY_MS)
if _fill_spa_login(page):
page.wait_for_timeout(POST_LOGIN_SETTLE_MS)
if not _is_logged_in(page):
raise LoginTimeoutError("process_queue 仍未登录")
# 等表格
page.wait_for_timeout(TABLE_INITIAL_WAIT_MS)
found_table = False
for _attempt in range(TABLE_POLL_MAX_ATTEMPTS):
page.wait_for_timeout(TABLE_POLL_INTERVAL_MS)
try:
page.wait_for_selector(
"table tbody tr", timeout=TABLE_TR_TIMEOUT_MS,
)
found_table = True
break
except Exception:
pass
if not found_table:
raise TableNotFoundError("process_queue 表格未找到")
mat_btn_count = page.locator(
"table tbody tr button:has-text('续聘材料包')"
).count()
# 2. 逐任务处理
for t in tasks:
contract_id = t["contract_id"]
task_id = t["task_id"]
conn = get_connection()
try:
row_idx = _find_row_for_contract(page, contract_id)
if row_idx is None:
repo.mark_task_failed(conn, task_id, "contract_not_found_in_table")
failed += 1
details.append({"task_id": task_id, "contract_id": contract_id,
"error": "contract_not_found_in_table"})
conn.commit()
continue
materials = _download_materials_for_row(
page, row_idx, contract_id, mat_btn_count,
)
from db.migrations import ensure_materials_content_schema
ensure_materials_content_schema(conn)
for m in materials:
if isinstance(m, dict) and m.get("kind") == "sandbox_card":
conn.execute(
"""
INSERT OR REPLACE INTO materials_content
(contract_id, file_name, file_path, extracted_text,
parse_method, parsed_at)
VALUES (?, ?, ?, ?, 'sandbox_card', ?)
""",
(
contract_id, m["file_name"],
f"sandbox://hr/contracts/{contract_id}",
m["text"],
datetime.now().isoformat(timespec="seconds"),
),
)
repo.parse_sandbox_card_performance(conn, contract_id, m["text"])
result_path = (
f"sandbox://hr/contracts/{contract_id}"
if materials else "no_materials_captured"
)
repo.mark_task_success(conn, task_id, result_path)
succeeded += 1
details.append({
"task_id": task_id,
"contract_id": contract_id,
"ok": True,
"materials_count": len(materials),
})
conn.commit()
except Exception as e:
repo.mark_task_failed(conn, task_id, str(e))
failed += 1
details.append({
"task_id": task_id, "contract_id": contract_id,
"error": str(e),
})
conn.commit()
logger.warning("process_queue: task=%s 失败: %s", task_id, e)
finally:
conn.close()
finally:
ctx.close()
return {
"processed": len(tasks),
"succeeded": succeeded,
"failed": failed,
"details": details,
}
def _error_result(reason: str, fetched: int) -> dict: def _error_result(reason: str, fetched: int) -> dict:
return { return {
"platform_url": PLATFORM_URL, "platform_url": PLATFORM_URL,

View File

@@ -50,7 +50,7 @@ def suggest(perf_records: list[sqlite3.Row]) -> tuple[str, str]:
# 全部 >= 80 且逐年上升(升序排 → 严格递增) # 全部 >= 80 且逐年上升(升序排 → 严格递增)
all_high = all(s >= PERFORMANCE_RENEW_MIN for s in scores) all_high = all(s >= PERFORMANCE_RENEW_MIN for s in scores)
upward = all(scores[i] <= scores[i + 1] for i in range(len(scores) - 1)) upward = all(scores[i] < scores[i + 1] for i in range(len(scores) - 1))
if all_high and upward: if all_high and upward:
return ("renew", f"近 3 年考核均 ≥ {PERFORMANCE_RENEW_MIN} 分且逐年上升") return ("renew", f"近 3 年考核均 ≥ {PERFORMANCE_RENEW_MIN} 分且逐年上升")

View File

@@ -6,7 +6,7 @@
""" """
SKILL_SLUG = "alert-contract-expiring-hr" SKILL_SLUG = "alert-contract-expiring-hr"
SKILL_VERSION = "1.1.5" SKILL_VERSION = "1.1.18"
LOG_LOGGER_NAME = "openclaw.skill.alert_contract_expiring_hr" LOG_LOGGER_NAME = "openclaw.skill.alert_contract_expiring_hr"
PLATFORM_KIT_MIN_VERSION = "1.2.11" PLATFORM_KIT_MIN_VERSION = "1.2.11"
@@ -43,3 +43,57 @@ DEFAULT_MATERIALS = [
MORAL_FAIL_THRESHOLD = 60 # 师德 < 60 直接 non_renew MORAL_FAIL_THRESHOLD = 60 # 师德 < 60 直接 non_renew
PERFORMANCE_FAIL_THRESHOLD = 60 # 单年考核 < 60 算不合格 PERFORMANCE_FAIL_THRESHOLD = 60 # 单年考核 < 60 算不合格
PERFORMANCE_RENEW_MIN = 80 # 全部 >= 80 且逐年上升 → renew PERFORMANCE_RENEW_MIN = 80 # 全部 >= 80 且逐年上升 → renew
# ----- v1.1.6:RPA 超时常量(与 skill-template human_verification.py 对齐)-----
# 手工登录等待(HITL fallback)
HUMAN_WAIT_TIMEOUT = 180 # 手工登录等待超时(秒)—— skill-template 同名常量
HUMAN_POLL_INTERVAL = 2.0 # 轮询间隔(秒)—— skill-template 同名常量
# Playwright 操作超时(毫秒)
DEFAULT_PAGE_TIMEOUT_MS = 15_000 # Playwright 默认操作超时 —— skill-template 同名常量
GOTO_TIMEOUT_MS = 60_000 # page.goto 导航超时 —— skill-template 同名常量
# SPA 登录等待(毫秒)
SPA_LOGIN_SETTLE_MS = 4_000 # login click 后 settle 时间(SPA 写 cookie)
SPA_LOGIN_INPUT_TIMEOUT_MS = 20_000 # 等 username input 出现
SPA_LOGIN_URL_TIMEOUT_MS = 30_000 # 等 URL 跳离 login 页
SPA_HYDRATION_MS = 3_000 # goto login URL 后 SPA 框架初始化
POST_LOGIN_SETTLE_MS = 2_000 # 登录后等 SPA 路由稳定
SPA_HOME_RETRY_DELAY_MS = 2_500 # SPA_HOME retry 间隔
SPA_HOME_RETRY_MAX = 2 # SPA_HOME retry 最大次数
# 表格等待(毫秒)
TABLE_INITIAL_WAIT_MS = 3_000 # 表格初始等待(React + mock 填充)
TABLE_POLL_INTERVAL_MS = 2_000 # 表格轮询间隔
TABLE_POLL_MAX_ATTEMPTS = 20 # 表格轮询最大次数(合计 ~40s)
TABLE_TR_TIMEOUT_MS = 3_000 # 单次 wait_for_selector 超时
FULL_RENDER_WAIT_MS = 1_500 # 完整 DOM 渲染后等渲染结束
# 侧边栏/卡片(毫秒)
SIDEBAR_LINK_TIMEOUT_MS = 30_000 # 等侧边栏 hr/contracts link
PUSHSTATE_SETTLE_MS = 1_500 # SPA pushState 后等
SIDEBAR_CLICK_URL_TIMEOUT_MS = 10_000 # click 后等 URL 变 /hr/contracts
URL_AFTER_LOGIN_TIMEOUT_MS = 20_000 # 根域 login 后等跳转
# reload 后备用
RELOAD_NETWORKIDLE_TIMEOUT_MS = 30_000 # page.reload networkidle 超时
RELOAD_DOMCONTENTLOADED_TIMEOUT_MS = 30_000 # page.reload domcontentloaded 超时
POST_RELOAD_LOGIN_HYDRATION_MS = 3_000 # reload 后回到 login 页的 hydration
POST_RELOGIN_GOTO_WAIT_MS = 2_000 # 重新登录后 goto PLATFORM_URL 等
# 卡片
CARD_APPEAR_WAIT_MS = 3_000 # 「续聘材料包」click 后等卡片出现
# dump
DUMP_PATH_PARENT_MKDIR = True # _dump_page 是否自动创建父目录
# ----- v1.1.18:日志落文件 -----
LOG_FILE_DEFAULT_NAME = "alert-contract-expiring-hr.log"
LOG_MAX_BYTES_DEFAULT = 5 * 1024 * 1024 # 5MB
LOG_BACKUP_COUNT_DEFAULT = 5
# ----- v1.1.18:材料下载队列 -----
DOWNLOAD_QUEUE_MAX_ATTEMPTS_DEFAULT = 3
DOWNLOAD_QUEUE_KINDS = ("platform_button", "local_upload")
DOWNLOAD_QUEUE_STATUSES = ("pending", "running", "success", "failed")

View File

@@ -1,20 +1,35 @@
"""logging 配置:降级到 stderr,避免污染 stdout(stdout 留给结构化输出如 JSON)。 """logging 配置:降级到 stderr,避免污染 stdout(stdout 留给结构化输出如 JSON)。
优先用 jiangchang_skill_core 提供的 logging;本地无该包时降级到 stderr。 优先用 jiangchang_skill_core 提供的 logging;本地无该包时降级到 stderr。
v1.1.18:本地降级路径追加 RotatingFileHandler(按 LOG_FILE env 决定是否落文件)。
""" """
from __future__ import annotations from __future__ import annotations
import logging import logging
import logging.handlers
import os
import sys import sys
from pathlib import Path
from util.constants import LOG_LOGGER_NAME, SKILL_SLUG, SKILL_VERSION from util.constants import (
LOG_LOGGER_NAME,
SKILL_SLUG,
SKILL_VERSION,
)
_configured = False _configured = False
def setup_skill_logging() -> logging.Logger: def setup_skill_logging() -> logging.Logger:
"""初始化 skill logger(幂等)。""" """初始化 skill logger(幂等)。
本地降级行为(无 jiangchang_skill_core 时):
- 若 LOG_FILE 已设:追加 RotatingFileHandler(LOG_MAX_BYTES × LOG_BACKUP_COUNT)
- 始终追加 StreamHandler(sys.stderr)
- logger level 取自 LOG_LEVEL(默认 INFO)
"""
global _configured global _configured
logger = logging.getLogger(LOG_LOGGER_NAME) logger = logging.getLogger(LOG_LOGGER_NAME)
if _configured: if _configured:
@@ -30,15 +45,38 @@ def setup_skill_logging() -> logging.Logger:
except ImportError: except ImportError:
pass pass
# 本地降级:基础 stderr handler,UTF-8 safe # logger 级别
handler = logging.StreamHandler(sys.stderr) level_name = os.environ.get("LOG_LEVEL", "INFO").upper()
handler.setFormatter( logger.setLevel(getattr(logging, level_name, logging.INFO))
logging.Formatter(
fmt=f"%(asctime)s [{SKILL_SLUG}] %(levelname)s %(message)s", formatter = logging.Formatter(
datefmt="%Y-%m-%d %H:%M:%S", fmt=f"%(asctime)s [{SKILL_SLUG}] %(levelname)s %(message)s",
) datefmt="%Y-%m-%d %H:%M:%S",
) )
logger.setLevel(logging.INFO)
# 本地降级:可选 RotatingFileHandler
log_file_str = os.environ.get("LOG_FILE")
if log_file_str:
log_path = Path(log_file_str)
if not log_path.is_absolute():
from util.runtime_paths import get_logs_dir
log_path = get_logs_dir() / log_file_str
log_path.parent.mkdir(parents=True, exist_ok=True)
max_bytes = int(os.environ.get("LOG_MAX_BYTES", str(5 * 1024 * 1024)))
backup_count = int(os.environ.get("LOG_BACKUP_COUNT", "5"))
file_handler = logging.handlers.RotatingFileHandler(
log_path,
maxBytes=max_bytes,
backupCount=backup_count,
encoding="utf-8",
)
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)
# 始终保留 stderr handler
handler = logging.StreamHandler(sys.stderr)
handler.setFormatter(formatter)
logger.addHandler(handler) logger.addHandler(handler)
logger.propagate = False logger.propagate = False
_configured = True _configured = True
@@ -51,3 +89,19 @@ def get_skill_logger() -> logging.Logger:
if not _configured: if not _configured:
return setup_skill_logging() return setup_skill_logging()
return logging.getLogger(LOG_LOGGER_NAME) return logging.getLogger(LOG_LOGGER_NAME)
def reset_for_tests() -> None:
"""测试用:重置 _configured flag,让 setup_skill_logging 重新初始化。
必须在每个修改日志配置的测试 fixture 里调用,否则 env 改动不会生效。
"""
global _configured
logger = logging.getLogger(LOG_LOGGER_NAME)
for h in list(logger.handlers):
try:
h.close()
except Exception:
pass
logger.removeHandler(h)
_configured = False

View File

@@ -66,6 +66,30 @@ def get_dispatch_dir() -> Path:
return get_data_dir() / "dispatch" return get_data_dir() / "dispatch"
def get_logs_dir() -> Path:
"""返回日志目录(RotatingFileHandler 默认落地点)。"""
env_dir = os.environ.get("JIANGCHANG_LOGS_DIR")
if env_dir:
return Path(env_dir)
return get_data_dir() / "logs"
def get_input_dir() -> Path:
"""返回用户主动投放的输入目录(含 contracts/*.csv + materials/*.{pdf,docx,doc})。"""
env_dir = os.environ.get("INPUT_DIR")
if env_dir:
return Path(env_dir)
return get_data_dir() / "input"
def get_input_contracts_dir() -> Path:
return get_input_dir() / "contracts"
def get_input_materials_dir() -> Path:
return get_input_dir() / "materials"
def get_examples_dir() -> Path: def get_examples_dir() -> Path:
"""返回样例数据目录。""" """返回样例数据目录。"""
return get_skill_root() / "examples" return get_skill_root() / "examples"
@@ -78,5 +102,13 @@ def get_templates_dir() -> Path:
def ensure_dirs() -> None: def ensure_dirs() -> None:
"""确保所有需要的目录存在。""" """确保所有需要的目录存在。"""
for d in (get_data_dir(), get_output_dir(), get_dispatch_dir()): for d in (
get_data_dir(),
get_output_dir(),
get_dispatch_dir(),
get_logs_dir(),
get_input_dir(),
get_input_contracts_dir(),
get_input_materials_dir(),
):
d.mkdir(parents=True, exist_ok=True) d.mkdir(parents=True, exist_ok=True)

View File

@@ -18,6 +18,7 @@ test_* 返回 (False, msg) 即 abort 发布。
from __future__ import annotations from __future__ import annotations
import argparse import argparse
import json
import os import os
import re import re
import shutil import shutil
@@ -217,6 +218,72 @@ def test_skill_md_slug_alignment() -> tuple[bool, str]:
return True, f"slug={md_slug}" return True, f"slug={md_slug}"
def test_data_management_actions() -> tuple[bool, str]:
"""数据管理动作必须绑定业务表,并映射到实际 CLI 命令。"""
actions_path = _REPO_ROOT / "assets" / "actions.json"
actions = json.loads(actions_path.read_text(encoding="utf-8"))["actions"]
by_id = {action["id"]: action for action in actions}
expected = {
"scan-expiring-contracts": "toolbar",
"collect-from-platform": "toolbar",
"generate-renewal-task": "row",
"dispatch-task-notice": "batch",
"mark-contract-renewed": "row",
}
for action_id, placement in expected.items():
action = by_id.get(action_id)
if not action:
return False, f"缺少数据管理动作 {action_id}"
if placement not in action.get("placements", []):
return False, f"{action_id} 缺少 {placement} placement"
if "合同台账" not in action.get("bind", {}).get("tables", []):
return False, f"{action_id} 未绑定合同台账"
if action.get("entrypoint", {}).get("command") != action_id:
return False, f"{action_id} CLI command 映射错误"
# v1.1.18: download-materials 是 async form action,绑定 download_tasks 队列表
dl = by_id.get("download-materials")
if not dl:
return False, "缺少 download-materials 动作"
if dl.get("executionProfile") != "async":
return False, f"download-materials 应为 async,实际 {dl.get('executionProfile')}"
if dl.get("interaction") != "form":
return False, f"download-materials 应为 form interaction"
if "download_tasks" not in dl.get("bind", {}).get("tables", []):
return False, f"download-materials 未绑定 download_tasks"
schema_props = dl.get("inputSchema", {}).get("properties", {})
if "itemIds" not in schema_props or "pickCount" not in schema_props:
return False, f"download-materials inputSchema 缺 itemIds/pickCount: {schema_props}"
# v1.1.18: show-config / logs-list 是 sync 运维动作
for sync_id in ("show-config", "logs-list"):
a = by_id.get(sync_id)
if not a:
return False, f"缺少 {sync_id} 动作"
if a.get("executionProfile") != "sync":
return False, f"{sync_id} 应为 sync"
return True, "5 个数据管理动作 + download-materials (async form) + show-config/logs-list (sync) 全部就绪"
def test_collect_platform_defaults_to_browser() -> tuple[bool, str]:
"""平台采集默认显示浏览器;--headless 仅用于已登录自动化。"""
from cli.app import build_parser
parser = build_parser()
default_args = parser.parse_args(["collect-from-platform"])
if default_args.headless:
return False, "collect-from-platform 默认不应使用无头模式"
headless_args = parser.parse_args(["collect-from-platform", "--headless"])
if not headless_args.headless:
return False, "--headless 未启用无头模式"
legacy_args = parser.parse_args(["collect-from-platform", "--show-browser"])
if legacy_args.headless:
return False, "--show-browser 兼容别名未保留浏览器模式"
return True, "默认浏览器模式,--headless 和 --show-browser 行为正确"
# ---------- db.repository:v1.1.2 新增的 sandbox card 解析 ---------- # ---------- db.repository:v1.1.2 新增的 sandbox card 解析 ----------
def test_parse_sandbox_card_row_format() -> tuple[bool, str]: def test_parse_sandbox_card_row_format() -> tuple[bool, str]:
@@ -386,6 +453,20 @@ def test_suggest_renew_upward() -> tuple[bool, str]:
return True, "renew + 上升趋势" return True, "renew + 上升趋势"
def test_suggest_observe_equal_scores_not_renew() -> tuple[bool, str]:
"""持平不算逐年上升;应回落到 observe"""
from service.task_generator import suggest
rows = [
_fake_perf(2023, 80, moral=85),
_fake_perf(2024, 80, moral=85),
_fake_perf(2025, 90, moral=85),
]
s, r = suggest(rows)
if s != "observe":
return False, f"suggest(80,80,90) = {s}, want observe"
return True, "观察:持平不等于逐年上升"
def test_suggest_observe_default() -> tuple[bool, str]: def test_suggest_observe_default() -> tuple[bool, str]:
"""达标但非逐年上升 → observe(常规评审)""" """达标但非逐年上升 → observe(常规评审)"""
from service.task_generator import suggest from service.task_generator import suggest
@@ -558,6 +639,768 @@ def test_html_render_unknown_task_raises() -> tuple[bool, str]:
return False, "应抛 ValueError,实际未抛" return False, "应抛 ValueError,实际未抛"
# ---------- v1.1.6:service.platform_collector 纯函数 ----------
def test_gen_contract_id_emp_id() -> tuple[bool, str]:
"""_gen_contract_id 工号派生(v1.1.1 起用工号作主键)"""
from service import platform_collector as pc
cases = [
(("T20190033", "2026-04-15"), "RPA-T20190033-20260415"),
(("T20200055", "2026-06-30"), "RPA-T20200055-20260630"),
(("T20180021", "2026-08-31"), "RPA-T20180021-20260831"),
]
for (emp_id, end_date), expected in cases:
got = pc._gen_contract_id(emp_id, end_date)
if got != expected:
return False, f"({emp_id!r}, {end_date!r}) → {got!r},want {expected!r}"
return True, f"3 工号派生正确"
def test_gen_contract_id_special_chars() -> tuple[bool, str]:
"""_gen_contract_id 工号含特殊字符时正确清洗(保留前 10 字符)"""
from service import platform_collector as pc
# 工号含非 \w 字符 → 清洗;超过 10 字符截断
got = pc._gen_contract_id("T-2019.0033", "2026-04-15")
# 清洗后保留 \w,"T-2019.0033" → "T20190033"(8 字符,未截断)
if "RPA-" not in got or "20260415" not in got:
return False, f"清洗后格式错:{got!r}"
# 长工号 → 截断到 10 字符
got2 = pc._gen_contract_id("ABCDEFGHIJKLMNO", "2026-01-01")
# "ABCDEFGHIJ"(10 字符)
if not got2.startswith("RPA-ABCDEFGHIJ-"):
return False, f"长工号未截断到 10 字符:{got2!r}"
return True, "清洗 + 截断 OK"
def test_normalize_date_iso() -> tuple[bool, str]:
"""_normalize_date ISO 日期直通"""
from service import platform_collector as pc
cases = [
("2026-04-15", "2026-04-15"),
("2024-01-01", "2024-01-01"),
]
for inp, want in cases:
got = pc._normalize_date(inp)
if got != want:
return False, f"{inp!r} → {got!r},want {want!r}"
return True, "ISO 日期直通"
def test_normalize_date_fallback() -> tuple[bool, str]:
"""_normalize_date 非标准格式截前 10 字符"""
from service import platform_collector as pc
# 非标准但带 ISO 前缀
got = pc._normalize_date("2026-04-15T00:00:00")
if got != "2026-04-15":
return False, f"ISO 前缀应保留前 10 字符 → {got!r}"
# 完全非日期 → 截前 10 字符
got2 = pc._normalize_date("20260415X")
if got2 != "20260415X": # 10 字符
return False, f"非日期应截前 10 字符 → {got2!r}"
# 带空白
got3 = pc._normalize_date(" 2026-04-15 ")
if got3 != "2026-04-15":
return False, f"带空白应去除 → {got3!r}"
return True, "非标准格式 fallback OK"
def test_parse_days_positive() -> tuple[bool, str]:
"""_parse_days 正天数"""
from service import platform_collector as pc
cases = [
("30", 30),
("90", 90),
("还剩 45 天", 45),
("0", 0),
]
for inp, want in cases:
got = pc._parse_days(inp)
if got != want:
return False, f"{inp!r} → {got!r},want {want!r}"
return True, "正天数解析 OK"
def test_parse_days_negative() -> tuple[bool, str]:
"""_parse_days 负数 / 兜底"""
from service import platform_collector as pc
# 含负号
if pc._parse_days("-15") != -15:
return False, "负天数未解析"
# 已过期(带 "已过期" 文本)
if pc._parse_days("已过期 7 天") != 7:
return False, "已过期文本未解析数字部分"
# 无法解析
if pc._parse_days("") is not None:
return False, "空字符串应返回 None"
if pc._parse_days("无数字") is not None:
return False, "纯文本应返回 None"
return True, "负数 + 兜底 OK"
def test_parse_alert_red_orange_yellow_unknown() -> tuple[bool, str]:
"""_parse_alert 红/橙/黄/未知四级映射"""
from service import platform_collector as pc
cases = [
("红色预警", "red"),
("红牌", "red"),
("橙色预警", "orange"),
("黄灯", "yellow"),
("黄色预警", "yellow"),
("无", "unknown"),
("", "unknown"),
("blue", "unknown"),
]
for inp, want in cases:
got = pc._parse_alert(inp)
if got != want:
return False, f"{inp!r} → {got!r},want {want!r}"
return True, "四级映射 OK"
# ---------- v1.1.18:日志 / 路径 / 输入目录 / 队列 / 配置 / 日志查看 ----------
def test_logging_rotating_file_handler() -> tuple[bool, str]:
"""LOG_FILE 已设时,setup_skill_logging 应追加 RotatingFileHandler。"""
import importlib
from util.runtime_paths import get_logs_dir
log_path = get_logs_dir() / "test-rotate.log"
os.environ["LOG_FILE"] = str(log_path)
os.environ.pop("LOG_MAX_BYTES", None)
os.environ.pop("LOG_BACKUP_COUNT", None)
os.environ.pop("LOG_LEVEL", None)
import util.logging_setup as ls
importlib.reload(ls)
ls.reset_for_tests()
logger = ls.setup_skill_logging()
logger.info("hello rotating")
found = [h for h in logger.handlers if h.__class__.__name__ == "RotatingFileHandler"]
if not found:
ls.reset_for_tests()
os.environ.pop("LOG_FILE", None)
return False, "未发现 RotatingFileHandler"
if not log_path.exists():
ls.reset_for_tests()
os.environ.pop("LOG_FILE", None)
return False, f"日志文件未生成: {log_path}"
content = log_path.read_text(encoding="utf-8")
if "hello rotating" not in content:
ls.reset_for_tests()
os.environ.pop("LOG_FILE", None)
return False, f"日志内容缺 'hello rotating': {content[:200]}"
ls.reset_for_tests()
os.environ.pop("LOG_FILE", None)
importlib.reload(ls)
return True, "RotatingFileHandler 写入成功"
def test_logging_fallback_when_log_file_unset() -> tuple[bool, str]:
"""LOG_FILE 未设时,logger 只有 StreamHandler(行为等价 v1.1.17)。"""
import importlib
os.environ.pop("LOG_FILE", None)
os.environ.pop("LOG_LEVEL", None)
import util.logging_setup as ls
importlib.reload(ls)
ls.reset_for_tests()
logger = ls.setup_skill_logging()
file_handlers = [h for h in logger.handlers if "File" in h.__class__.__name__]
if file_handlers:
ls.reset_for_tests()
return False, f"应有 0 个 FileHandler,实际 {len(file_handlers)}"
ls.reset_for_tests()
return True, "无 LOG_FILE 时无 FileHandler(兼容 v1.1.17)"
def test_logging_level_env() -> tuple[bool, str]:
"""LOG_LEVEL=DEBUG 应让 logger.level == logging.DEBUG。"""
import importlib
import logging
os.environ["LOG_LEVEL"] = "DEBUG"
os.environ.pop("LOG_FILE", None)
import util.logging_setup as ls
importlib.reload(ls)
ls.reset_for_tests()
logger = ls.setup_skill_logging()
got = logger.level
if got != logging.DEBUG:
ls.reset_for_tests()
os.environ.pop("LOG_LEVEL", None)
return False, f"logger.level={got} (want {logging.DEBUG})"
ls.reset_for_tests()
os.environ.pop("LOG_LEVEL", None)
return True, "LOG_LEVEL=DEBUG 生效"
def test_logs_dir_default_under_data() -> tuple[bool, str]:
"""JIANGCHANG_LOGS_DIR 未设时,get_logs_dir() = get_data_dir() / 'logs'。"""
os.environ.pop("JIANGCHANG_LOGS_DIR", None)
os.environ.pop("LOG_FILE", None)
import util.logging_setup as ls # 触发 runtime_paths 副作用
ls.reset_for_tests()
from util.runtime_paths import get_logs_dir, get_data_dir
expected = get_data_dir() / "logs"
if get_logs_dir() != expected:
return False, f"get_logs_dir()={get_logs_dir()} != {expected}"
if not expected.exists():
return False, f"ensure_dirs 没创建 {expected}"
return True, f"get_logs_dir={expected}"
def _isolated_input_dirs(monkeypatch_env: dict[str, str | None]) -> tuple[Path, Path]:
"""把 INPUT_DIR 重定向到 tmpdir,返回 (input_root, materials_target) 的 Path。
不会污染仓库 data/input/。
"""
import tempfile
tmp = Path(tempfile.mkdtemp(prefix="alert-hr-importer-test-"))
input_root = tmp / "input"
(input_root / "contracts").mkdir(parents=True)
(input_root / "materials").mkdir(parents=True)
monkeypatch_env["INPUT_DIR"] = str(input_root)
os.environ["INPUT_DIR"] = str(input_root) # 关键:进程 env 也得设,否则 runtime_paths 读不到
return input_root, tmp
def _restore_input_env(saved: dict[str, str | None]) -> None:
for k, v in saved.items():
if v is None:
os.environ.pop(k, None)
else:
os.environ[k] = v
def test_importer_contracts_csv_upsert() -> tuple[bool, str]:
"""input/contracts/*.csv → contracts 表 upsert。"""
saved = {k: os.environ.get(k) for k in ("INPUT_DIR",)}
try:
input_root, tmp = _isolated_input_dirs(saved)
contracts_dir = input_root / "contracts"
csv_path = contracts_dir / "batch1.csv"
csv_path.write_text(
"contract_id,teacher_name,unit_name,contract_end,employee_id\n"
"C001,张三,物理学院,2026-04-15,T001\n"
"C002,李四,计算机学院,2026-08-31,T002\n",
encoding="utf-8",
)
conn = _make_conn()
try:
from service.importer import import_input_dir
result = import_input_dir(conn)
if result["contracts_added"] != 2:
return False, f"contracts_added={result['contracts_added']} (want 2)"
if result["errors"]:
return False, f"errors={result['errors']}"
rows = conn.execute(
"SELECT contract_id, teacher_name, unit_name, contract_end "
"FROM contracts ORDER BY contract_id"
).fetchall()
if len(rows) != 2:
return False, f"DB 中合同数={len(rows)} (want 2)"
if rows[0]["teacher_name"] != "张三":
return False, f"row[0].teacher_name={rows[0]['teacher_name']}"
return True, f"contracts_added={result['contracts_added']}"
finally:
conn.close()
shutil.rmtree(tmp, ignore_errors=True)
finally:
_restore_input_env(saved)
def test_importer_materials_pdf_enqueue() -> tuple[bool, str]:
"""input/materials/{cid}__{f}.pdf → materials_content (parse_method='pending')。"""
saved = {k: os.environ.get(k) for k in ("INPUT_DIR",)}
try:
input_root, tmp = _isolated_input_dirs(saved)
materials_dir = input_root / "materials"
pdf_path = materials_dir / "RPA-C001__考核表.pdf"
pdf_path.write_bytes(b"%PDF-1.4 fake test file")
conn = _make_conn()
try:
# 先插入 contract(materials_content FK 到 contracts.contract_id)
conn.execute(
"""INSERT INTO contracts
(contract_id, employee_id, teacher_name, unit_id, unit_name,
position, contract_type, contract_start, contract_end,
trial_period_end, status)
VALUES ('RPA-C001', 'T001', '张三', 'U001', '物理学院',
'教授', '固定期限', '2024-01-01', '2026-04-15',
'2024-04-15', '可续聘')"""
)
conn.commit()
from service.importer import import_input_dir
result = import_input_dir(conn)
if result["materials_added"] != 1:
return False, f"materials_added={result['materials_added']} (want 1) errors={result['errors']}"
row = conn.execute(
"SELECT contract_id, parse_method FROM materials_content "
"WHERE contract_id = 'RPA-C001'"
).fetchone()
if not row:
return False, "materials_content 未写入"
if row["parse_method"] != "pending":
return False, f"parse_method={row['parse_method']} (want pending)"
return True, "PDF 入库 parse_method=pending"
finally:
conn.close()
shutil.rmtree(tmp, ignore_errors=True)
finally:
_restore_input_env(saved)
def test_importer_idempotent() -> tuple[bool, str]:
"""同一 CSV 跑两次,第二次 contracts_added=0(upsert 幂等)。"""
saved = {k: os.environ.get(k) for k in ("INPUT_DIR",)}
try:
input_root, tmp = _isolated_input_dirs(saved)
(input_root / "contracts" / "x.csv").write_text(
"contract_id,teacher_name\nC100,重复测试\n", encoding="utf-8"
)
conn = _make_conn()
try:
from service.importer import import_input_dir
r1 = import_input_dir(conn)
r2 = import_input_dir(conn)
if r1["contracts_added"] != 1:
return False, f"first={r1['contracts_added']} (want 1)"
if r2["contracts_added"] != 1:
# 注意:CSV 仍按行 upsert,第 2 次相当于 1 次 UPDATE,contracts_added 仍计数
return False, f"second={r2['contracts_added']} (want 1)"
n = conn.execute("SELECT COUNT(*) AS n FROM contracts").fetchone()["n"]
if n != 1:
return False, f"DB count={n} (want 1,幂等)"
return True, "幂等 OK:DB count=1"
finally:
conn.close()
shutil.rmtree(tmp, ignore_errors=True)
finally:
_restore_input_env(saved)
def test_importer_empty_input_dir() -> tuple[bool, str]:
"""空 input 目录 → no-op,返回全 0。"""
saved = {k: os.environ.get(k) for k in ("INPUT_DIR",)}
try:
input_root, tmp = _isolated_input_dirs(saved)
conn = _make_conn()
try:
from service.importer import import_input_dir
result = import_input_dir(conn)
if result["contracts_added"] or result["materials_added"]:
return False, f"空目录应返回全 0,实际 {result}"
if result["files_scanned"] != 0:
return False, f"files_scanned={result['files_scanned']} (want 0)"
return True, "空目录 no-op"
finally:
conn.close()
shutil.rmtree(tmp, ignore_errors=True)
finally:
_restore_input_env(saved)
def test_download_tasks_enqueue_idempotent() -> tuple[bool, str]:
"""enqueue 同一合同两次:第 2 次返回已有 task_id,DB 仅 1 行。"""
from db import repository as repo
conn = _make_conn()
try:
conn.execute(
"""INSERT INTO contracts
(contract_id, employee_id, teacher_name, unit_id, unit_name,
position, contract_type, contract_start, contract_end,
trial_period_end, status)
VALUES ('C100', 'T100', 'X', 'U1', 'Y', '', '', '2024-01-01',
'2026-04-15', '2024-04-15', '可续聘')"""
)
conn.commit()
t1 = repo.enqueue_download_task(conn, "C100", "platform_button")
t2 = repo.enqueue_download_task(conn, "C100", "platform_button")
if t1 != t2:
return False, f"t1={t1} t2={t2} (应相同)"
n = conn.execute(
"SELECT COUNT(*) AS n FROM download_tasks WHERE contract_id = 'C100'"
).fetchone()["n"]
if n != 1:
return False, f"DB count={n} (want 1)"
return True, f"idempotent: task_id={t1}"
finally:
conn.close()
def test_download_tasks_claim_atomic() -> tuple[bool, str]:
"""claim_pending_tasks:第一次拿 N 个,第二次拿 0 个(被抢光)。"""
from db import repository as repo
conn = _make_conn()
try:
# 准备 5 份合同
for i in range(5):
conn.execute(
f"""INSERT INTO contracts
(contract_id, employee_id, teacher_name, unit_id, unit_name,
position, contract_type, contract_start, contract_end,
trial_period_end, status)
VALUES ('C{i}', 'T{i}', 'X', 'U{i}', 'Y', '', '',
'2024-01-01', '2026-04-15', '2024-04-15', '可续聘')"""
)
conn.commit()
for i in range(5):
repo.enqueue_download_task(conn, f"C{i}", "platform_button")
conn.commit()
first = repo.claim_pending_tasks(conn, limit=5)
if len(first) != 5:
return False, f"第一次领到 {len(first)} (want 5)"
# claim 返回的 Row 是查询时刻的快照(UPDATE 在 fetchall 之后才发);
# 重新查 DB 验证 status 已变 running
for t in first:
cur = conn.execute(
"SELECT status FROM download_tasks WHERE task_id = ?",
(t["task_id"],),
).fetchone()
if cur["status"] != "running":
return False, f"task {t['task_id']} DB status={cur['status']} (want running)"
second = repo.claim_pending_tasks(conn, limit=5)
if len(second) != 0:
return False, f"第二次领到 {len(second)} (want 0)"
return True, "claim 原子:5+0"
finally:
conn.close()
def test_download_tasks_mark_success_and_failed() -> tuple[bool, str]:
"""mark_success: status=success + result_path; mark_failed 计数 + 重试状态切换。"""
from db import repository as repo
conn = _make_conn()
try:
conn.execute(
"""INSERT INTO contracts
(contract_id, employee_id, teacher_name, unit_id, unit_name,
position, contract_type, contract_start, contract_end,
trial_period_end, status)
VALUES ('C200', 'T200', 'X', 'U1', 'Y', '', '', '2024-01-01',
'2026-04-15', '2024-04-15', '可续聘')"""
)
conn.commit()
t = repo.enqueue_download_task(conn, "C200", "platform_button")
# 1. 标记 success
repo.mark_task_success(conn, t, "sandbox://hr/contracts/C200")
conn.commit()
row = conn.execute(
"SELECT status, result_path FROM download_tasks WHERE task_id = ?", (t,)
).fetchone()
if row["status"] != "success":
return False, f"success 后 status={row['status']}"
if row["result_path"] != "sandbox://hr/contracts/C200":
return False, f"result_path={row['result_path']}"
# 2. 失败 ≤ max_attempts:status 回退 pending
t2 = repo.enqueue_download_task(conn, "C200", "platform_button_v2")
repo.mark_task_failed(conn, t2, "first attempt failed")
conn.commit()
row2 = conn.execute(
"SELECT status, attempts FROM download_tasks WHERE task_id = ?", (t2,)
).fetchone()
if row2["status"] != "pending":
return False, f"第 1 次失败应 pending,实际 {row2['status']}"
if row2["attempts"] != 1:
return False, f"attempts={row2['attempts']} (want 1)"
# 3. 失败 ≥ max_attempts:status=failed
repo.mark_task_failed(conn, t2, "second attempt failed")
repo.mark_task_failed(conn, t2, "third attempt failed")
conn.commit()
row3 = conn.execute(
"SELECT status, attempts FROM download_tasks WHERE task_id = ?", (t2,)
).fetchone()
if row3["status"] != "failed":
return False, f"第 3 次失败应 failed,实际 {row3['status']}"
return True, "mark_success/mark_failed 全 OK"
finally:
conn.close()
def test_download_tasks_count_by_status() -> tuple[bool, str]:
"""count_tasks_by_status 返回 {pending: N, success: M, ...}。"""
from db import repository as repo
conn = _make_conn()
try:
for i in range(3):
conn.execute(
f"""INSERT INTO contracts
(contract_id, employee_id, teacher_name, unit_id, unit_name,
position, contract_type, contract_start, contract_end,
trial_period_end, status)
VALUES ('CT{i}', 'T{i}', 'X', 'U{i}', 'Y', '', '',
'2024-01-01', '2026-04-15', '2024-04-15', '可续聘')"""
)
conn.commit()
t0 = repo.enqueue_download_task(conn, "CT0", "platform_button")
t1 = repo.enqueue_download_task(conn, "CT1", "platform_button")
repo.enqueue_download_task(conn, "CT2", "platform_button")
conn.commit()
repo.mark_task_success(conn, t0, "x")
repo.mark_task_failed(conn, t1, "err")
# 再失败 2 次让 t1 进 failed
repo.mark_task_failed(conn, t1, "err")
repo.mark_task_failed(conn, t1, "err")
conn.commit()
counts = repo.count_tasks_by_status(conn)
if counts.get("success", 0) < 1:
return False, f"success 计数 < 1: {counts}"
if counts.get("failed", 0) < 1:
return False, f"failed 计数 < 1: {counts}"
if counts.get("pending", 0) < 1:
return False, f"pending 计数 < 1: {counts}"
return True, f"counts={counts}"
finally:
conn.close()
# ---------- v1.1.18: show-config / logs-list ----------
def _isolated_env_file(tmp_env_path: Path) -> None:
"""把 .env 路径重定向到 tmpdir;不影响其他 env。"""
import util.runtime_paths as rp
# runtime_paths.get_env_path() 总是返回 get_skill_root() / ".env"
# 这里通过 monkey-patch 函数对象实现隔离
rp.get_env_path = lambda: tmp_env_path # type: ignore[assignment]
def test_show_config_masks_tokens() -> tuple[bool, str]:
"""含 TOKEN 的 key 自动脱敏。"""
import tempfile
from pathlib import Path
tmp = Path(tempfile.mkdtemp(prefix="alert-hr-config-"))
env_path = tmp / ".env"
env_path.write_text(
"WECOM_API_TOKEN=abc123def456\n"
"HR_API_TOKEN=xx\n"
"LOG_LEVEL=INFO\n",
encoding="utf-8",
)
import service.config_viewer as cv
orig_get_env_path = cv.get_env_path
cv.get_env_path = lambda: env_path # type: ignore[assignment]
try:
result = cv.read_env_config()
cfg = result["config"]
if cfg.get("WECOM_API_TOKEN") != "ab***56":
return False, f"WECOM_API_TOKEN={cfg.get('WECOM_API_TOKEN')} (want ab***56)"
if cfg.get("HR_API_TOKEN") != "***":
return False, f"HR_API_TOKEN={cfg.get('HR_API_TOKEN')} (want *** for ≤4 chars)"
if cfg.get("LOG_LEVEL") != "INFO":
return False, f"LOG_LEVEL={cfg.get('LOG_LEVEL')} (want INFO)"
return True, "tokens masked OK"
finally:
cv.get_env_path = orig_get_env_path # type: ignore[assignment]
shutil.rmtree(tmp, ignore_errors=True)
def test_show_config_skips_empty() -> tuple[bool, str]:
""".env 中空 value 的 key 不出现在 config 中。"""
import tempfile
from pathlib import Path
tmp = Path(tempfile.mkdtemp(prefix="alert-hr-config-"))
env_path = tmp / ".env"
env_path.write_text(
"DB_PATH=\nLOG_LEVEL=INFO\n", encoding="utf-8",
)
import service.config_viewer as cv
orig = cv.get_env_path
cv.get_env_path = lambda: env_path # type: ignore[assignment]
try:
cfg = cv.read_env_config()["config"]
if "DB_PATH" in cfg:
return False, f"DB_PATH 应该跳过,仍在 cfg: {cfg}"
if cfg.get("LOG_LEVEL") != "INFO":
return False, f"LOG_LEVEL={cfg.get('LOG_LEVEL')}"
return True, "空 value 跳过 OK"
finally:
cv.get_env_path = orig # type: ignore[assignment]
shutil.rmtree(tmp, ignore_errors=True)
def test_show_config_skips_comments() -> tuple[bool, str]:
"""# 注释行不计入 config。"""
import tempfile
from pathlib import Path
tmp = Path(tempfile.mkdtemp(prefix="alert-hr-config-"))
env_path = tmp / ".env"
env_path.write_text(
"# 这是注释\n"
" # 也是注释\n"
"LOG_LEVEL=INFO\n",
encoding="utf-8",
)
import service.config_viewer as cv
orig = cv.get_env_path
cv.get_env_path = lambda: env_path # type: ignore[assignment]
try:
cfg = cv.read_env_config()["config"]
if len(cfg) != 1 or "LOG_LEVEL" not in cfg:
return False, f"应仅 1 项 LOG_LEVEL,实际 {list(cfg.keys())}"
return True, "注释跳过 OK"
finally:
cv.get_env_path = orig # type: ignore[assignment]
shutil.rmtree(tmp, ignore_errors=True)
def test_show_config_handles_quotes() -> tuple[bool, str]:
""".env 中带双引号/单引号的值自动去引号。"""
import tempfile
from pathlib import Path
tmp = Path(tempfile.mkdtemp(prefix="alert-hr-config-"))
env_path = tmp / ".env"
env_path.write_text(
'HR_API_BASE_URL="https://hr.example.edu/api/v1"\n'
"GREETING='hello world'\n"
"NOTE=no_quotes\n",
encoding="utf-8",
)
import service.config_viewer as cv
orig = cv.get_env_path
cv.get_env_path = lambda: env_path # type: ignore[assignment]
try:
cfg = cv.read_env_config()["config"]
if cfg.get("HR_API_BASE_URL") != "https://hr.example.edu/api/v1":
return False, f"双引号未去: {cfg.get('HR_API_BASE_URL')}"
if cfg.get("GREETING") != "hello world":
return False, f"单引号未去: {cfg.get('GREETING')}"
if cfg.get("NOTE") != "no_quotes":
return False, f"无引号: {cfg.get('NOTE')}"
return True, "引号处理 OK"
finally:
cv.get_env_path = orig # type: ignore[assignment]
shutil.rmtree(tmp, ignore_errors=True)
def test_logs_viewer_nonexistent_file() -> tuple[bool, str]:
"""日志文件不存在时,返回 exists=False,lines=[]。"""
import tempfile
from pathlib import Path
tmp = Path(tempfile.mkdtemp(prefix="alert-hr-logs-"))
fake_log = tmp / "no-such-file.log"
os.environ["LOG_FILE"] = str(fake_log)
try:
from service.logs_viewer import tail_log_file
result = tail_log_file(tail=50)
if result["exists"]:
return False, "exists 应为 False"
if result["lines"]:
return False, f"lines 应为空: {result['lines']}"
if result["returned_lines"] != 0:
return False, f"returned_lines={result['returned_lines']}"
return True, "non-existent file OK"
finally:
os.environ.pop("LOG_FILE", None)
shutil.rmtree(tmp, ignore_errors=True)
def test_logs_viewer_tail_and_filter() -> tuple[bool, str]:
"""tail + level 过滤:INFO 默认只显示 INFO/WARNING/ERROR。"""
import tempfile
from pathlib import Path
tmp = Path(tempfile.mkdtemp(prefix="alert-hr-logs-"))
log_path = tmp / "tail-test.log"
lines: list[str] = []
for i in range(300):
if i % 3 == 0:
lines.append(f"2026-09-15 12:34:5{i%10} [alert-contract-expiring-hr] INFO 第{i}条")
elif i % 3 == 1:
lines.append(f"2026-09-15 12:34:5{i%10} [alert-contract-expiring-hr] WARNING 第{i}条")
else:
lines.append(f"2026-09-15 12:34:5{i%10} [alert-contract-expiring-hr] DEBUG 第{i}条")
log_path.write_text("\n".join(lines), encoding="utf-8")
os.environ["LOG_FILE"] = str(log_path)
try:
from service.logs_viewer import tail_log_file
# INFO+:最后 50 行中含 INFO + WARNING,去掉 DEBUG
# 300 行每 3 行一循环:i%3==0 INFO, i%3==1 WARN, i%3==2 DEBUG
# 最后 50 行是 i=250..299:第 250=INFO 251=WARN 252=DEBUG ... 共 50 行
# 分布:i%3==0 → 17 (250, 253, ..., 299 但 299%3=0)... 实际 250..299 中 i%3==0 有 17 个
result_info = tail_log_file(tail=50, min_level="INFO")
if result_info["returned_lines"] != 33:
return False, f"INFO+ returned_lines={result_info['returned_lines']} (want 33)"
if any(" DEBUG " in ln for ln in result_info["lines"]):
return False, "INFO 过滤应排除 DEBUG 行"
# WARNING+:仅 WARNING
result_warn = tail_log_file(tail=50, min_level="WARNING")
if result_warn["returned_lines"] != 17:
return False, f"WARNING+ returned_lines={result_warn['returned_lines']} (want 17)"
if not all(" WARNING " in ln for ln in result_warn["lines"]):
return False, "WARNING 过滤应只剩 WARNING 行"
# DEBUG:全部 50 行都返回
result_debug = tail_log_file(tail=50, min_level="DEBUG")
if result_debug["returned_lines"] != 50:
return False, f"DEBUG returned_lines={result_debug['returned_lines']} (want 50)"
return True, f"tail+filter OK: DEBUG+={result_debug['returned_lines']}, INFO+={result_info['returned_lines']}, WARN+={result_warn['returned_lines']}"
finally:
os.environ.pop("LOG_FILE", None)
shutil.rmtree(tmp, ignore_errors=True)
def test_logs_viewer_resolves_relative_path() -> tuple[bool, str]:
"""LOG_FILE 设相对路径时,解析到 get_logs_dir() 下。"""
from util.runtime_paths import get_logs_dir
os.environ["LOG_FILE"] = "relative-test.log"
os.environ.pop("JIANGCHANG_LOGS_DIR", None)
try:
from service.logs_viewer import _resolve_log_path
got = _resolve_log_path()
expected = get_logs_dir() / "relative-test.log"
if got != expected:
return False, f"_resolve_log_path={got} (want {expected})"
return True, f"resolved to {got}"
finally:
os.environ.pop("LOG_FILE", None)
# ---------- 入口 ---------- # ---------- 入口 ----------
def main() -> int: def main() -> int:
@@ -577,6 +1420,8 @@ def main() -> int:
_run("util/month_match", test_month_match) _run("util/month_match", test_month_match)
_run("constants/skill_md_version_alignment", test_skill_md_version_alignment) _run("constants/skill_md_version_alignment", test_skill_md_version_alignment)
_run("constants/skill_md_slug_alignment", test_skill_md_slug_alignment) _run("constants/skill_md_slug_alignment", test_skill_md_slug_alignment)
_run("assets/data_management_actions", test_data_management_actions)
_run("cli/collect_platform_browser_default", test_collect_platform_defaults_to_browser)
_run("db/parse_sandbox_card_row", test_parse_sandbox_card_row_format) _run("db/parse_sandbox_card_row", test_parse_sandbox_card_row_format)
_run("db/parse_sandbox_card_narrative", test_parse_sandbox_card_narrative_format) _run("db/parse_sandbox_card_narrative", test_parse_sandbox_card_narrative_format)
_run("db/parse_sandbox_card_empty_invalid", test_parse_sandbox_card_empty_and_invalid) _run("db/parse_sandbox_card_empty_invalid", test_parse_sandbox_card_empty_and_invalid)
@@ -584,14 +1429,41 @@ def main() -> int:
_run("db/upsert_performance_record", test_upsert_performance_record_grade_mapping) _run("db/upsert_performance_record", test_upsert_performance_record_grade_mapping)
_run("task_generator/suggest_empty", test_suggest_empty_records) _run("task_generator/suggest_empty", test_suggest_empty_records)
_run("task_generator/suggest_low_moral", test_suggest_low_moral) _run("task_generator/suggest_low_moral", test_suggest_low_moral)
_run("task_generator/suggest_fail_year", test_suggest_fail_year) _run("task_generator/suggest_fail_year", test_suggest_fail_year)
_run("task_generator/suggest_renew", test_suggest_renew_upward) _run("task_generator/suggest_renew", test_suggest_renew_upward)
_run("task_generator/suggest_observe", test_suggest_observe_default) _run("task_generator/suggest_equal_scores", test_suggest_observe_equal_scores_not_renew)
_run("scanner/no_alerts", test_scanner_no_alerts) _run("task_generator/suggest_observe", test_suggest_observe_default)
_run("scanner/no_alerts", test_scanner_no_alerts)
_run("scanner/three_levels", test_scanner_three_levels) _run("scanner/three_levels", test_scanner_three_levels)
_run("scanner/skips_non_active", test_scanner_skips_non_active_contracts) _run("scanner/skips_non_active", test_scanner_skips_non_active_contracts)
_run("html_renderer/basic", test_html_render_basic) _run("html_renderer/basic", test_html_render_basic)
_run("html_renderer/unknown_task_raises", test_html_render_unknown_task_raises) _run("html_renderer/unknown_task_raises", test_html_render_unknown_task_raises)
_run("platform_collector/gen_contract_id_emp_id", test_gen_contract_id_emp_id)
_run("platform_collector/gen_contract_id_special", test_gen_contract_id_special_chars)
_run("platform_collector/normalize_date_iso", test_normalize_date_iso)
_run("platform_collector/normalize_date_fallback", test_normalize_date_fallback)
_run("platform_collector/parse_days_positive", test_parse_days_positive)
_run("platform_collector/parse_days_negative", test_parse_days_negative)
_run("platform_collector/parse_alert_levels", test_parse_alert_red_orange_yellow_unknown)
_run("logging/rotating_file_handler", test_logging_rotating_file_handler)
_run("logging/fallback_when_log_file_unset", test_logging_fallback_when_log_file_unset)
_run("logging/level_env", test_logging_level_env)
_run("logging/logs_dir_default", test_logs_dir_default_under_data)
_run("importer/contracts_csv_upsert", test_importer_contracts_csv_upsert)
_run("importer/materials_pdf_enqueue", test_importer_materials_pdf_enqueue)
_run("importer/idempotent", test_importer_idempotent)
_run("importer/empty_input_dir", test_importer_empty_input_dir)
_run("download_tasks/enqueue_idempotent", test_download_tasks_enqueue_idempotent)
_run("download_tasks/claim_atomic", test_download_tasks_claim_atomic)
_run("download_tasks/mark_success_and_failed", test_download_tasks_mark_success_and_failed)
_run("download_tasks/count_by_status", test_download_tasks_count_by_status)
_run("show_config/masks_tokens", test_show_config_masks_tokens)
_run("show_config/skips_empty", test_show_config_skips_empty)
_run("show_config/skips_comments", test_show_config_skips_comments)
_run("show_config/handles_quotes", test_show_config_handles_quotes)
_run("logs_viewer/nonexistent_file", test_logs_viewer_nonexistent_file)
_run("logs_viewer/tail_and_filter", test_logs_viewer_tail_and_filter)
_run("logs_viewer/resolves_relative_path", test_logs_viewer_resolves_relative_path)
total = len(_RESULTS) total = len(_RESULTS)
passed = sum(1 for _, ok, _ in _RESULTS if ok) passed = sum(1 for _, ok, _ in _RESULTS if ok)