docs: v2.1 fix — cron catch-up leaves the keeper; faults are incidents
Some checks failed
ClawSweeper Dispatch / dispatch (push) Has been cancelled
CodeQL / Security High (actions) (push) Has been cancelled
CodeQL / Security High (channel-runtime-boundary) (push) Has been cancelled
CodeQL / Security High (core-auth-secrets) (push) Has been cancelled
CodeQL / Security High (mcp-process-tool-boundary) (push) Has been cancelled
CodeQL / Security High (network-ssrf-boundary) (push) Has been cancelled
CodeQL / Security High (plugin-trust-boundary) (push) Has been cancelled
CodeQL / Security High (process-exec-boundary) (push) Has been cancelled
Docs / docs (push) Has been cancelled
OpenClaw Stable Main Closeout / Resolve stable release closeout inputs (push) Has been cancelled
OpenClaw Stable Main Closeout / Verify stable main closeout (push) Has been cancelled
Workflow Sanity / no-tabs (push) Has been cancelled
Workflow Sanity / actionlint (push) Has been cancelled
Workflow Sanity / generated-doc-baselines (push) Has been cancelled

alvis: a missed cron is an operational fault, not a task-lifecycle event,
and re-firing it would make the keeper a trigger. Removed cron catch-up
from fabric-keeper duties entirely.

- Keeper is now strictly janitorial: neither assigns NOR triggers work
  (lease sweeps, deadlines, dead-letter, digests, metrics only).
- Schedules that must not be missed are monitored in Zabbix like any
  other infra fault.
- NEW §6d: faults are incidents, handled PER PROBLEM CLASS by a future
  ops-agent (watches Zabbix, applies known mitigation or files a task
  with the incident as context) — never a blanket catch-up policy, since
  a missed briefing / backup / consolidation want different responses.
- Decision log entry 22.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014t8Qg9gi7H7HtT8MncoXAB
This commit is contained in:
2026-07-21 08:03:27 +00:00
parent 714a9ca785
commit 1db3dd1b34

View File

@@ -24,8 +24,8 @@ rule. This document is the reference; the Kanboard A2A tasks implement it.
- **Context ref** — a pointer (Hindsight bank id, git ref, KB task id, file
path) passed *instead of* pasted content.
- **fabric-keeper** — the janitor daemon owning time semantics (§6b): lease
sweeps, deadlines, cron catch-up, dead-letter, inbox digests. It never
assigns work.
sweeps, deadlines, dead-letter, inbox digests. It never assigns work and
never triggers work.
## 1. Why
@@ -298,14 +298,39 @@ four concerns, each with its own owner:
| Triggering | when do tasks appear? | OpenClaw cron, agents' delayed self-submissions, humans |
| Dispatch | which task runs next? | each queue's worker (claim by priority under its a(t)) |
| Admission | may it run now? | LiteLLM budgets/rate + the budget governor |
| **Time semantics** | deadlines, missed crons, stuck tasks? | **the fabric-keeper** |
| **Time semantics** | expired leases, deadlines, stuck tasks? | **the fabric-keeper** |
| Failure/anomaly response | something went wrong — now what? | Zabbix → the ops-agent (§6d) |
The **fabric-keeper** is one tiny always-on daemon that never assigns work: it
sweeps expired leases back to queued, enforces deadlines (escalating to
inboxes), applies per-schedule **cron catch-up policy** (missed window →
run-once | skip, configured per schedule), moves poison tasks to dead-letter,
emits the human daily digest, and exports queue depths to Zabbix. Deadline and
catch-up semantics live here and nowhere else.
The **fabric-keeper** is one tiny always-on daemon that neither assigns nor
triggers work. It sweeps expired leases back to queued, enforces task deadlines
(escalating to the responsible inbox), moves poison tasks to dead-letter, emits
the human daily digest, and exports queue depths/ages to Zabbix.
Its defining property is being **off the critical path**: kill it and work still
flows (workers keep pulling and running) — only hygiene degrades. That is what
separates a janitor from a scheduler, which in a push system would stop
everything.
**Deliberately NOT a keeper duty: missed crons.** A cron window missed because
the host was down is not a task-lifecycle event — it's an *operational fault*,
and re-firing it would make the keeper a trigger. Instead: schedules that must
not be missed are **monitored in Zabbix** (a missed run raises a warning like
any other infra fault), and the response is handled per-incident, not by a
blanket catch-up policy (§6d).
### 6d. Faults are incidents, not keeper chores
Anything that "went wrong" — a missed critical cron, a service down, a queue
backing up, a stale backup — surfaces as a **Zabbix problem**. Zabbix is the
single place operational faults are detected.
Response is **per-incident, not global**: each Zabbix problem class gets its own
mitigation path, expressed as a task. Later this is automated by a dedicated
**ops-agent** — a worker that watches Zabbix problems and, per problem class,
either applies a known mitigation or files a task (to the right agent, or to a
human inbox) with the incident as context. One blanket "catch-up policy" would
be exactly the wrong abstraction: a missed briefing, a missed backup and a
missed consolidation want completely different responses.
### 6c. Kanboard is tier-0 now
@@ -371,8 +396,8 @@ Added in v2.1 (hardening review, same day):
15. Done column = unverified completion; closed task = verified; the producer
never closes its own task.
16. Scheduler = decomposed (cron/self-submission triggers; workers dispatch;
LiteLLM+governor admit); the **fabric-keeper** janitor owns time semantics;
cron catch-up policy is per-schedule config.
LiteLLM+governor admit); the **fabric-keeper** janitor owns time semantics
(leases, deadlines, dead-letter, digests) and neither assigns nor triggers.
17. Personas/Cards live in git, deployed — never edited live.
18. Backbone/routing changes gated by golden-set shadow eval (Langfuse).
19. Fabric hygiene: creation quotas, ancestry depth cap, cycle detection,
@@ -381,3 +406,6 @@ Added in v2.1 (hardening review, same day):
reranker); pre-load checks.
21. Transport: sync = direct A2A RPC (journaled); async = KB polling; polling
never on a sync path.
22. **Missed crons are not a keeper duty**: schedules that must not be missed
are monitored in **Zabbix**; faults are incidents handled **per problem
class** by a future **ops-agent** (§6d), never by a blanket catch-up policy.