docs: v2.1 fix — cron catch-up leaves the keeper; faults are incidents
Some checks failed
ClawSweeper Dispatch / dispatch (push) Has been cancelled
CodeQL / Security High (actions) (push) Has been cancelled
CodeQL / Security High (channel-runtime-boundary) (push) Has been cancelled
CodeQL / Security High (core-auth-secrets) (push) Has been cancelled
CodeQL / Security High (mcp-process-tool-boundary) (push) Has been cancelled
CodeQL / Security High (network-ssrf-boundary) (push) Has been cancelled
CodeQL / Security High (plugin-trust-boundary) (push) Has been cancelled
CodeQL / Security High (process-exec-boundary) (push) Has been cancelled
Docs / docs (push) Has been cancelled
OpenClaw Stable Main Closeout / Resolve stable release closeout inputs (push) Has been cancelled
OpenClaw Stable Main Closeout / Verify stable main closeout (push) Has been cancelled
Workflow Sanity / no-tabs (push) Has been cancelled
Workflow Sanity / actionlint (push) Has been cancelled
Workflow Sanity / generated-doc-baselines (push) Has been cancelled
Some checks failed
ClawSweeper Dispatch / dispatch (push) Has been cancelled
CodeQL / Security High (actions) (push) Has been cancelled
CodeQL / Security High (channel-runtime-boundary) (push) Has been cancelled
CodeQL / Security High (core-auth-secrets) (push) Has been cancelled
CodeQL / Security High (mcp-process-tool-boundary) (push) Has been cancelled
CodeQL / Security High (network-ssrf-boundary) (push) Has been cancelled
CodeQL / Security High (plugin-trust-boundary) (push) Has been cancelled
CodeQL / Security High (process-exec-boundary) (push) Has been cancelled
Docs / docs (push) Has been cancelled
OpenClaw Stable Main Closeout / Resolve stable release closeout inputs (push) Has been cancelled
OpenClaw Stable Main Closeout / Verify stable main closeout (push) Has been cancelled
Workflow Sanity / no-tabs (push) Has been cancelled
Workflow Sanity / actionlint (push) Has been cancelled
Workflow Sanity / generated-doc-baselines (push) Has been cancelled
alvis: a missed cron is an operational fault, not a task-lifecycle event, and re-firing it would make the keeper a trigger. Removed cron catch-up from fabric-keeper duties entirely. - Keeper is now strictly janitorial: neither assigns NOR triggers work (lease sweeps, deadlines, dead-letter, digests, metrics only). - Schedules that must not be missed are monitored in Zabbix like any other infra fault. - NEW §6d: faults are incidents, handled PER PROBLEM CLASS by a future ops-agent (watches Zabbix, applies known mitigation or files a task with the incident as context) — never a blanket catch-up policy, since a missed briefing / backup / consolidation want different responses. - Decision log entry 22. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014t8Qg9gi7H7HtT8MncoXAB
This commit is contained in:
@@ -24,8 +24,8 @@ rule. This document is the reference; the Kanboard A2A tasks implement it.
|
||||
- **Context ref** — a pointer (Hindsight bank id, git ref, KB task id, file
|
||||
path) passed *instead of* pasted content.
|
||||
- **fabric-keeper** — the janitor daemon owning time semantics (§6b): lease
|
||||
sweeps, deadlines, cron catch-up, dead-letter, inbox digests. It never
|
||||
assigns work.
|
||||
sweeps, deadlines, dead-letter, inbox digests. It never assigns work and
|
||||
never triggers work.
|
||||
|
||||
## 1. Why
|
||||
|
||||
@@ -298,14 +298,39 @@ four concerns, each with its own owner:
|
||||
| Triggering | when do tasks appear? | OpenClaw cron, agents' delayed self-submissions, humans |
|
||||
| Dispatch | which task runs next? | each queue's worker (claim by priority under its a(t)) |
|
||||
| Admission | may it run now? | LiteLLM budgets/rate + the budget governor |
|
||||
| **Time semantics** | deadlines, missed crons, stuck tasks? | **the fabric-keeper** |
|
||||
| **Time semantics** | expired leases, deadlines, stuck tasks? | **the fabric-keeper** |
|
||||
| Failure/anomaly response | something went wrong — now what? | Zabbix → the ops-agent (§6d) |
|
||||
|
||||
The **fabric-keeper** is one tiny always-on daemon that never assigns work: it
|
||||
sweeps expired leases back to queued, enforces deadlines (escalating to
|
||||
inboxes), applies per-schedule **cron catch-up policy** (missed window →
|
||||
run-once | skip, configured per schedule), moves poison tasks to dead-letter,
|
||||
emits the human daily digest, and exports queue depths to Zabbix. Deadline and
|
||||
catch-up semantics live here and nowhere else.
|
||||
The **fabric-keeper** is one tiny always-on daemon that neither assigns nor
|
||||
triggers work. It sweeps expired leases back to queued, enforces task deadlines
|
||||
(escalating to the responsible inbox), moves poison tasks to dead-letter, emits
|
||||
the human daily digest, and exports queue depths/ages to Zabbix.
|
||||
|
||||
Its defining property is being **off the critical path**: kill it and work still
|
||||
flows (workers keep pulling and running) — only hygiene degrades. That is what
|
||||
separates a janitor from a scheduler, which in a push system would stop
|
||||
everything.
|
||||
|
||||
**Deliberately NOT a keeper duty: missed crons.** A cron window missed because
|
||||
the host was down is not a task-lifecycle event — it's an *operational fault*,
|
||||
and re-firing it would make the keeper a trigger. Instead: schedules that must
|
||||
not be missed are **monitored in Zabbix** (a missed run raises a warning like
|
||||
any other infra fault), and the response is handled per-incident, not by a
|
||||
blanket catch-up policy (§6d).
|
||||
|
||||
### 6d. Faults are incidents, not keeper chores
|
||||
|
||||
Anything that "went wrong" — a missed critical cron, a service down, a queue
|
||||
backing up, a stale backup — surfaces as a **Zabbix problem**. Zabbix is the
|
||||
single place operational faults are detected.
|
||||
|
||||
Response is **per-incident, not global**: each Zabbix problem class gets its own
|
||||
mitigation path, expressed as a task. Later this is automated by a dedicated
|
||||
**ops-agent** — a worker that watches Zabbix problems and, per problem class,
|
||||
either applies a known mitigation or files a task (to the right agent, or to a
|
||||
human inbox) with the incident as context. One blanket "catch-up policy" would
|
||||
be exactly the wrong abstraction: a missed briefing, a missed backup and a
|
||||
missed consolidation want completely different responses.
|
||||
|
||||
### 6c. Kanboard is tier-0 now
|
||||
|
||||
@@ -371,8 +396,8 @@ Added in v2.1 (hardening review, same day):
|
||||
15. Done column = unverified completion; closed task = verified; the producer
|
||||
never closes its own task.
|
||||
16. Scheduler = decomposed (cron/self-submission triggers; workers dispatch;
|
||||
LiteLLM+governor admit); the **fabric-keeper** janitor owns time semantics;
|
||||
cron catch-up policy is per-schedule config.
|
||||
LiteLLM+governor admit); the **fabric-keeper** janitor owns time semantics
|
||||
(leases, deadlines, dead-letter, digests) and neither assigns nor triggers.
|
||||
17. Personas/Cards live in git, deployed — never edited live.
|
||||
18. Backbone/routing changes gated by golden-set shadow eval (Langfuse).
|
||||
19. Fabric hygiene: creation quotas, ancestry depth cap, cycle detection,
|
||||
@@ -381,3 +406,6 @@ Added in v2.1 (hardening review, same day):
|
||||
reranker); pre-load checks.
|
||||
21. Transport: sync = direct A2A RPC (journaled); async = KB polling; polling
|
||||
never on a sync path.
|
||||
22. **Missed crons are not a keeper duty**: schedules that must not be missed
|
||||
are monitored in **Zabbix**; faults are incidents handled **per problem
|
||||
class** by a future **ops-agent** (§6d), never by a blanket catch-up policy.
|
||||
|
||||
Reference in New Issue
Block a user