fix(pilot): debounce process resource alarms - #140
Conversation
Split CPU and memory process alarms so they no longer collide in aggregation. Debounce normalized CPU and soft memory thresholds, add hard and critical memory tiers, and cool down repeated process resource alerts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reviewer self-checkNo blocking findings in the submitted diff. Checked scope: 1 commit ( Verification confirmed:
|
ralf0131
left a comment
ralf0131
left a comment
There was a problem hiding this comment.
Summary
Improves process resource alarm reliability by splitting the generic PROCESS_RESOURCE_ALARM into specific PROCESS_CPU_ALARM and PROCESS_MEMORY_ALARM types. CPU threshold now normalizes by core count (cpuPercent / cpuCoreCount) and requires 3 consecutive high samples before alarming — correctly preventing false positives on multi-core machines. Memory uses a tiered approach: soft (512MB, 3 consecutive samples), hard (1024MB, immediate), critical (2048MB, immediate, level 3). A 1-hour per-type cooldown (lastProcessAlarmAt map) prevents alarm spam. Tests are comprehensive — covering debounce behavior, tiered levels, CPU normalization, combined CPU+memory windows, and cooldown timing.
No issues found — well-designed debounce logic with thorough test coverage.
Automated review by github-manager-bot
|
📋 Ready to merge — approved 76h ago, CI passing, no merge conflicts. This PR has been approved and all checks pass, but hasn't been merged yet. Pinging maintainers for follow-up. Automated notification by github-manager-bot |
| } | ||
|
|
||
| const cpuCoreCount = Math.max(os.cpus().length, 1); | ||
| const normalizedCpuPercent = Math.round((cpuPercent / cpuCoreCount) * 100) / 100; |
There was a problem hiding this comment.
[High] CPU 归一化后阈值对单线程 Node 主进程可能不可达
metrics.cpu 由 calcCpuPercent 计算,为 per-core 口径(100% = 占满 1 核,可 >100)。这里除以 os.cpus().length 归一化为整机百分比后再与 80% 比较。
影响: loongsuite-pilot 以单线程事件循环为主,典型 CPU 失控约占满 1 核 = raw ~100%,归一化后在 8 核机上仅 12.5%,永远达不到 80%;要触发需 raw > 80×核数(8 核需 >640%)。结果 PROCESS_CPU_ALARM 对主进程在多核机器上静默失效。
建议: 二选一并明确文档:(a) 维持 per-core 口径不除以核数,以"占满 ~0.8 核"为线,更贴合单线程 Node;或 (b) 确要整机口径则将阈值按核数放缩/独立配置。请确认当前阈值不可达是否为预期。
| this.cpuHighSamples++; | ||
| if (this.cpuHighSamples < CPU_ALARM_CONSECUTIVE_SAMPLES) return; | ||
|
|
||
| this.recordProcessAlarm( |
There was a problem hiding this comment.
[Medium] PROCESS_RESOURCE_ALARM 类型迁移需同步下游告警规则
拆分后 PROCESS_RESOURCE_ALARM 不再产出,改为 PROCESS_CPU_ALARM / PROCESS_MEMORY_ALARM(旧枚举仅为历史数据兼容)。
影响: 下游若有按 alarm_type == 'PROCESS_RESOURCE_ALARM' 建立的 SLS 告警规则/看板,上线后将静默停止命中,形成"告警看似正常实则未覆盖"的盲区。
建议: 上线前同步下游告警规则/看板到新类型,并在 PR 描述或运维变更单显式列出需更新项,release note 标注类型迁移。
🔍 Code Review Summary
Lifecycle Verdict
总体结论方向正确、无绝对阻断项。拆分 CPU/Memory 告警类型从根源解决了 合入前需确认两点:
Highlights(正向实践)
评审报告详见: |
3 participants
背景
PROCESS_RESOURCE_ALARM当前对 CPU/Memory 采用单次超阈即告警，并且 CPU 与 Memory 共用同一个告警类型，容易造成瞬时尖峰噪声、内存重复告警以及同窗口聚合覆盖。核心改动
PROCESS_CPU_ALARM与PROCESS_MEMORY_ALARM，保留原PROCESS_RESOURCE_ALARM类型兼容历史数据。修改文件
src/metrics/alarm-manager.tssrc/metrics/metrics-writer.tstests/unit/metrics/metrics-writer.test.ts验证结果
npx vitest run tests/unit/metrics/，62/62 PASSnpm run typecheck，PASS/Users/lukechen/.codex/sessions/2026/07/15/rollout-2026-07-15T10-49-06-019f63ad-9019-71f1-9022-074d5cead2aa.jsonl，Pilot trace/Users/lukechen/.loongsuite-pilot/logs/otlp-debug/loongsuite-pilot-pr-gate-codex-2026-07-15.jsonlvalidate-trace: 1 trace / 16 spans，ENTRY=1、AGENT=1、STEP=5、LLM=5、TOOL=4，PASS=33，WARN=170，ERROR=0PROCESS_CPU_ALARM1 条，PROCESS_MEMORY_ALARM1 条，均在loongsuite_alarm/cn-shanghai查询命中备注
开源 PR 分支从 GitHub
main创建并重放本次 3 文件补丁，避免把内部仓库历史带入开源 PR。