Repository navigation
Commit 2282aa8
feat(trigger-schedule): a scheduled run whose due time passed while no kernel was ticking is recorded as missed, never replayed (#26485)
<!-- claude-projects-attribution: {"github_login":"os-project-manager"}
-->
_Requested by **Leehom** · [project
thread](https://claude.ai/code/project/chan_01TenzsRopZj4jMbvjX8uz1X?thread=cmsg_01TenzsRopZj4jMbvjX8uz1XSuvxsiC4BEpZzbv7B7H3Yk)_
Fixes #25235
Clause-②: yes (widening)
Metadata-protocol: none
## 维护者速读
- 改了什么:托管内核停摆(驱逐、重启、崩溃)期间到点的定时流程,下一个内核绑定该流程时为每个错过的到点时间在运行历史里记一条
`missed`(开始=到点时间,结束=发现时间),不补跑。
- 为什么改:cloud#3149 第 5 项裁决(本卡裁决 6353729816 字母 A):应用今天无法得知每日清扫没跑;spec 成员
`missed` 已由 #25363 落地,这是生产者一半。
- 风险与回滚:只新增一种终态行与一次绑定时的只读检测;故意拒绝的流程(定时开关关、缺组织、调度不可读)现在会注销上一个内核留下的
job,这是唯一改变既有行为的点。回滚 = revert 本 PR。
-
先看:`packages/triggers/trigger-schedule/src/schedule-trigger.ts`(`recordSilence`
与
`release`)、`packages/services/service-automation/src/engine.ts`(`recordMissedRun`、策略闸门调
`release`)、`packages/services/service-job/src/db-job-adapter.ts`(`readArming`)。
- 两分钟验证:`pnpm --filter @objectstack/trigger-schedule exec vitest run
src/missed-run-two-kernels.test.ts`,预期 8
个用例全绿(停一次记一条、停三次记三条、中间夹一个关闭开关的内核记零条、跑通的内核记零条)。
- 需要你做的:无,席位落地。
## What changes
A scheduled flow whose due time passed while no kernel was ticking now
gets one run-history entry with status `missed` (due time, detected
time), written by the next kernel that binds the flow. Nothing is run
for it.
- **Detection (`trigger-schedule`).** On a flow's first bind in a
kernel, `ScheduleTrigger` reads how the previous kernel left the flow's
job (`DbJobAdapter.readArming`, adapter-local like `setReplayGuard`).
Only a `sys_job` row still `active` counts as a silence: a kernel that
stops ticking cancels nothing, while a deliberate unbind cancels. The
due times of the schedule that kernel armed, strictly after its
last-alive instant and up to now, are computed in a new sibling module,
`missed-due-times.ts`.
- **Record (`service-automation`).** `AutomationEngine.recordMissedRun`
checks each due time against the dispatch ledger (`sys_flow_dispatch`).
A claimed window was dispatched and is skipped. An unclaimed one is
written as a `missed` terminal row with an id derived from (flow, due
time), so two replicas write one row. With no persisted ledger, or a
ledger read that throws, nothing is recorded and the trigger warns,
naming the flow.
- **Bound.** At most the run history's own per-flow cap
(`maxTerminalRunsPerFlow`, read through
`AutomationEngine.missedRunLimit()`, no second literal) newest due times
per flow per detection. One `warn` names the flow, the count not
recorded and the earliest due time dropped.
- **A refusal is not a silence.** A flow refused on purpose (scheduled
work off, missing acting organization, unreadable schedule) now has its
job cancelled by its deterministic name even when this kernel never
bound it (`ScheduleTrigger.release`, an optional `FlowTrigger.release`
the engine's scheduled-work policy gate calls, since that gate returns
before the trigger is reached). Without this, a later kernel would read
the refusing kernel's whole life as missed due times.
- **The three run-row vocabulary sites widen together**, as the
`refused` precedent did (ADR-0078): `TERMINAL_RUN_STATUSES` (which the
store's row gate reads), `sys_automation_run.status` options and the
retention `onlyWhen` set. `plugin-approvals` `RUN_STATUS_LIVENESS`
already carries `missed: 'terminal'` on `main`.
## Why
Ruling `6353729816` (letter A) on cloud#3149 item 5: record only, no
replay. The spec member `missed` landed in #25363; this is the producer.
Basis for `Clause-②: yes (widening)`, measured on the diff (the claim
line said `no` and asked for the measurement): the diff adds public
surface to published packages: `AutomationEngine.recordMissedRun` and
`missedRunLimit`, the optional `FlowTrigger.release`,
`SuspendedRunStore.maxTerminalRunsPerFlow`, `DbJobAdapter.readArming`,
`ScheduleTrigger.release`, and the barrel types `MissedScheduledRun`,
`MissedRunOutcome`, `JobArming`, `JobArmingReading`,
`MissedScheduledRunInput`; and the stored `sys_automation_run.status`
column accepts one more option. Nothing accepted before is refused.
`Metadata-protocol: none`: nothing a metadata author writes changes.
## Tests
-
`packages/triggers/trigger-schedule/src/missed-run-two-kernels.test.ts`
(in-process, the test-budget shape): a kernel stopped across one due
time records one `missed` entry and runs nothing; an evicted kernel
across three due times records three; a kernel that ticked through
records none; a due time whose fire was claimed before the kernel died
records none; a flow deliberately unbound before the silence records
none; an OFF kernel between two ON kernels records none; a long silence
records only the cap's newest and warns; no persisted ledger skips with
one warning.
- `missed-due-times.test.ts` (cron, interval, once; the cap and the
dropped count), `service-automation/src/missed-run-record.test.ts`
(ledger verdicts, deterministic id, the three vocabulary sites,
`missedRunLimit`, the policy gate's `release`),
`suspended-run-store.test.ts` (a `missed` row is stored and listed as
history), `service-job/src/db-job-adapter.test.ts` (`readArming`:
absent, every schedule type, fire moves last-alive, cancel disarms,
eviction leaves it armed, a failed read rejects).
## Acceptance notes
- The work was resumed from `claude/project-thread-rtuv45@3830dd8` by
cherry-picking its four services commits onto `main` (the local spec
commit and its revert skipped); they applied cleanly. This branch adds a
docblock fix (the `recordMissedRun` docblock sat above
`missedRunLimit`), a note on which way a late arming read can fail, and
the barrel export of the two structural types.
- Ordering of the arming read and the row rewrite: the read is issued
before `schedule()`. If a pooled driver still answered it after the
rewrite, the row would read as touched just now and no silence would be
found, so the failure direction loses a `missed` entry and never invents
one. Making `schedule()` wait for the read was tried and dropped: a
`stop()` in between would be overtaken by a late re-arm.
- Observation, not filed: with two replicas ticking at once, a replica
that binds in the few milliseconds between a due time and the ticking
replica's ledger claim would record that due time as `missed` while the
other replica runs it. Single-kernel hosted environments, the case
cloud#3149 describes, cannot hit it.
- The run-history reader on the console side:
`ui/packages/app-shell/src/views/metadata-admin/previews/FlowRunsPanel.tsx`
maps five statuses and falls back to the raw status string for the rest,
as it already does for `timed_out` and `refused`; `missed` renders as
the label `missed` with no duration (none is recorded). No `ui/` edit in
this PR.
- Not ruled and flagged by the ruling for the maintainer (item 5 g):
whether `missed` rows should be bounded tighter than the store's
per-flow cap so real history survives a long outage. This PR uses the
store's cap, as ruled.
## Verification
Head `1bff9c7e9f` (after merging `origin/main` at `68901b0383`; base of
the card work `08ed7e295f`).
<details><summary>Red first: the new pins against main's
sources</summary>
Production sources restored to `08ed7e295f` (tree only), this branch's
tests, `service-automation` rebuilt at that base:
`missed-run-record.test.ts` + `suspended-run-store.test.ts` 11 failed /
57 passed; `db-job-adapter.test.ts` 4 failed / 20 passed;
`missed-run-two-kernels.test.ts` 5 failed (the 12 passing are
`missed-due-times.test.ts`, a new pure module that exists on both
sides). Restored by `git checkout HEAD`, `git diff HEAD` empty.
</details>
<details><summary>Suites on the final head</summary>
- `@objectstack/trigger-schedule` 191/191 · `@objectstack/service-job`
121/121 · `@objectstack/service-automation` 2719/2719 ·
`@objectstack/plugin-approvals` 1290/1290.
- `typecheck` for the three touched packages: exit 0. `eslint
--no-inline-config` over their `src/`: exit 0, no output.
- `test:repo` projects for core (83), objectql (5), types (11), spec
(1293): all pass.
</details>
<details><summary>Ablations, one per pinned behaviour, each landed on
disk (anchor 1 to 0) and restored to the HEAD blob</summary>
1. Engine ignores a claimed window: 1 red (`a claimed window was
dispatched`).
2. `missed` dropped from `TERMINAL_RUN_STATUSES`: 4 red (engine,
two-replica, vocabulary, store row).
3. `readArming` reads every row as active: 1 red (`cancel() disarms;
eviction leaves the row armed`).
4. `ScheduleTrigger.release` never cancels an unbound job: 1 red (the
OFF kernel between two ON kernels).
5. Trigger ignores the `active` bit: 2 red (deliberately unbound flow,
OFF kernel).
6. Detection writes 5 past the cap: 1 red (the long-silence cap case).
7. `missedRunLimit` answers a number with no ledger (dist rebuilt,
preflight marker present, then absent after restore): 1 red in each of
the engine unit and the two-kernel suites.
8. The policy gate no longer calls `release` (dist rebuilt, preflight as
above): 2 red in the engine unit suite, 1 red in the two-kernel suite.
9. The missed row's id is no longer per (flow, due time): 2 red.
Three first attempts did not count and were rerun: ablations 4 to 6
first ran against a `service-automation` dist left at the red-first base
(rebuilt, then rerun); ablation 8's first marker was elided by the build
(preflight failed), rerun with a marker that survives; ablation 6 first
used a bound so large the cases timed out, rerun with cap + 5.
</details>
<details><summary>Derived gates</summary>
`node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` at `1bff9c7e9f`: 63 commands. All 63 ran on
this head and every one exited 0; `dispatch-gates --ran` reconciles 63
run, 0 NOT MEASURED. A first round at `92f11484f3` had
`check:dual-build-cjs-loads` answer PREREQUISITE NOT MET (exit 3, no
full build in the worktree); on this head it ran and passed (112 entry
points, 69 packages). Outside the derived list and left to CI: the
workspace type-check lanes, Build Core, and the dogfood and temporal
batteries that run only on `main`.
</details>
---
_Generated by [Claude
Code](https://claude.ai/code/session_01FEkQ5ofGbFRago9bvAetNb)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent c199064 commit 2282aa8
15 files changed
Lines changed: 1381 additions & 13 deletions
File tree
- .changeset
- packages
- services
- service-automation/src
- service-job/src
- triggers/trigger-schedule/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
602 | 602 | | |
603 | 603 | | |
604 | 604 | | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
| 612 | + | |
| 613 | + | |
| 614 | + | |
| 615 | + | |
| 616 | + | |
| 617 | + | |
605 | 618 | | |
606 | 619 | | |
607 | 620 | | |
| |||
1244 | 1257 | | |
1245 | 1258 | | |
1246 | 1259 | | |
| 1260 | + | |
| 1261 | + | |
| 1262 | + | |
| 1263 | + | |
| 1264 | + | |
| 1265 | + | |
| 1266 | + | |
| 1267 | + | |
| 1268 | + | |
| 1269 | + | |
1247 | 1270 | | |
1248 | 1271 | | |
1249 | 1272 | | |
| |||
1638 | 1661 | | |
1639 | 1662 | | |
1640 | 1663 | | |
| 1664 | + | |
| 1665 | + | |
| 1666 | + | |
| 1667 | + | |
| 1668 | + | |
| 1669 | + | |
1641 | 1670 | | |
1642 | | - | |
| 1671 | + | |
1643 | 1672 | | |
1644 | 1673 | | |
1645 | 1674 | | |
| |||
2166 | 2195 | | |
2167 | 2196 | | |
2168 | 2197 | | |
| 2198 | + | |
| 2199 | + | |
| 2200 | + | |
| 2201 | + | |
| 2202 | + | |
| 2203 | + | |
| 2204 | + | |
2169 | 2205 | | |
2170 | 2206 | | |
2171 | 2207 | | |
| |||
2192 | 2228 | | |
2193 | 2229 | | |
2194 | 2230 | | |
| 2231 | + | |
| 2232 | + | |
| 2233 | + | |
| 2234 | + | |
| 2235 | + | |
| 2236 | + | |
| 2237 | + | |
| 2238 | + | |
| 2239 | + | |
| 2240 | + | |
| 2241 | + | |
| 2242 | + | |
| 2243 | + | |
| 2244 | + | |
| 2245 | + | |
| 2246 | + | |
| 2247 | + | |
| 2248 | + | |
| 2249 | + | |
| 2250 | + | |
| 2251 | + | |
| 2252 | + | |
| 2253 | + | |
| 2254 | + | |
| 2255 | + | |
| 2256 | + | |
| 2257 | + | |
2195 | 2258 | | |
2196 | 2259 | | |
2197 | 2260 | | |
| |||
2982 | 3045 | | |
2983 | 3046 | | |
2984 | 3047 | | |
| 3048 | + | |
| 3049 | + | |
| 3050 | + | |
| 3051 | + | |
| 3052 | + | |
| 3053 | + | |
| 3054 | + | |
| 3055 | + | |
| 3056 | + | |
| 3057 | + | |
| 3058 | + | |
| 3059 | + | |
| 3060 | + | |
| 3061 | + | |
| 3062 | + | |
| 3063 | + | |
| 3064 | + | |
| 3065 | + | |
| 3066 | + | |
| 3067 | + | |
| 3068 | + | |
| 3069 | + | |
| 3070 | + | |
| 3071 | + | |
| 3072 | + | |
| 3073 | + | |
| 3074 | + | |
| 3075 | + | |
| 3076 | + | |
| 3077 | + | |
| 3078 | + | |
| 3079 | + | |
| 3080 | + | |
| 3081 | + | |
| 3082 | + | |
| 3083 | + | |
| 3084 | + | |
| 3085 | + | |
| 3086 | + | |
| 3087 | + | |
| 3088 | + | |
| 3089 | + | |
| 3090 | + | |
| 3091 | + | |
| 3092 | + | |
| 3093 | + | |
| 3094 | + | |
| 3095 | + | |
| 3096 | + | |
| 3097 | + | |
| 3098 | + | |
| 3099 | + | |
| 3100 | + | |
| 3101 | + | |
| 3102 | + | |
| 3103 | + | |
| 3104 | + | |
| 3105 | + | |
| 3106 | + | |
| 3107 | + | |
| 3108 | + | |
| 3109 | + | |
| 3110 | + | |
| 3111 | + | |
| 3112 | + | |
| 3113 | + | |
| 3114 | + | |
| 3115 | + | |
| 3116 | + | |
| 3117 | + | |
| 3118 | + | |
| 3119 | + | |
| 3120 | + | |
| 3121 | + | |
| 3122 | + | |
| 3123 | + | |
| 3124 | + | |
| 3125 | + | |
| 3126 | + | |
| 3127 | + | |
| 3128 | + | |
| 3129 | + | |
2985 | 3130 | | |
2986 | 3131 | | |
2987 | 3132 | | |
| |||
3547 | 3692 | | |
3548 | 3693 | | |
3549 | 3694 | | |
| 3695 | + | |
| 3696 | + | |
| 3697 | + | |
| 3698 | + | |
| 3699 | + | |
| 3700 | + | |
| 3701 | + | |
| 3702 | + | |
| 3703 | + | |
| 3704 | + | |
| 3705 | + | |
| 3706 | + | |
| 3707 | + | |
| 3708 | + | |
| 3709 | + | |
| 3710 | + | |
3550 | 3711 | | |
3551 | 3712 | | |
3552 | 3713 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
85 | 85 | | |
86 | 86 | | |
87 | 87 | | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
88 | 92 | | |
89 | 93 | | |
90 | 94 | | |
| |||
0 commit comments