Update Backups

Backups.md

@@ -1,6 +1,7 @@
 # Backups
 
-Daily backups run from **root's crontab** (`sudo crontab -e`).
+Daily backups run from **root's crontab** (`sudo crontab -e`), except Kanboard's
+(see below), which runs from **alvis's own crontab** (`crontab -e`, no sudo).
 
 ## Gitea
 
@@ -53,6 +54,14 @@ Destination: `/mnt/backups/vaultwarden/<timestamp>/`
 Retention: last 5 backups
 Zabbix: trapper item `vaultwarden.backup.ts` (id 70368) — trigger if no backup in 4 days
 
+> **Known bug (found 2026-07-21, kb#158):** the script pushes a formatted date
+> *string* (`date '+%Y-%m-%d %H:%M'`) into `vaultwarden.backup.ts`, which is a
+> numeric (unsigned) item — every push is silently rejected, so the item has
+> **never recorded a real value** (`lastclock: 0`) and its "overdue" trigger has
+> been permanently in the problem state. Not fixed as part of kb#158 (out of
+> scope — flagged here for a follow-up task). Kanboard's own backup below pushes
+> a proper unix timestamp and was verified to actually populate the item.
+
 ## Seafile
 
 Script: `seafile/backup.sh` — mysqldump of 3 DBs + rsync of seafile data. Runs every 3 days.
@@ -80,6 +89,84 @@ Script: `users-backup.sh` — rsyncs alvis and liza home dirs from `/mnt/misc` t
 
 Zabbix: trapper item `users.backup.ts` (id 70379) — trigger (id 32869) if no backup in 4 days
 
+## Kanboard (kb#158, A2A-26 — tier-0 hardening)
+
+Kanboard was promoted to **tier-0** (fabric backbone, DESIGN-a2a-agents.md v2.1
+§6c) on 2026-07-21: backups on par with the vault, plus live service/API
+monitoring (see Zabbix Monitoring below).
+
+Script: `/home/alvis/agap_git/kanboard/backup.sh` — the `kanboard` container has
+no `sqlite3` CLI and no PHP `sqlite3` extension (only `pdo_sqlite`), so instead of
+a CLI dump it runs SQLite's own `VACUUM INTO` via PDO inside the container — an
+atomic, consistent online snapshot safe against concurrent writers (the same
+safety property `vaultwarden backup` gives that service). Also tars up the
+`kanboard_plugins` volume (`PLUGIN_INSTALLER=true` means plugins can be installed
+at runtime, not just baked into the image). Runs every 3 days.
+
+```
+0 3 */3 * * /home/alvis/agap_git/kanboard/backup.sh >> /home/alvis/agap_git/kanboard/backup.log 2>&1
+```
+
+**Runs from alvis's own crontab, not root's** — unlike the other services above.
+`/mnt/backups/kanboard/` is `chown`'d to `alvis` (one-time bootstrap: alvis is in
+the `docker` group, which is root-equivalent on this host, so a throwaway
+`alpine` container did the `mkdir`+`chown` without needing an interactive sudo
+password) so the rest of the pipeline — the dump itself, the Zabbix push, the
+crontab — needs no root at all.
+
+Destination: `/mnt/backups/kanboard/<timestamp>/` (`db.sqlite` + `plugins.tar.gz`)
+Retention: last 5 backups
+Zabbix: trapper item `kanboard.backup.ts` (id 70605, real unix timestamp) —
+trigger `Kanboard backup overdue (>4 days)` (id 32958)
+
+### Kanboard service + JSON-RPC health
+
+Script: `/home/alvis/agap_git/kanboard/healthcheck.sh`, every 2 minutes from
+alvis's crontab:
+
+```
+*/2 * * * * /home/alvis/agap_git/kanboard/healthcheck.sh >> /home/alvis/agap_git/kanboard/healthcheck.log 2>&1
+```
+
+Pushes two Zabbix trapper items every run (a "down" value is real signal, not a
+gap to wait out via `nodata()`):
+
+| Item key | id | Meaning | Trigger |
+|----------|----|---------|---------|
+| `kanboard.service.up` | 70606 | 1 if the `kanboard` container's own Docker healthcheck reports `healthy` | `Kanboard service is down` (id 32959): `last()=0` or `nodata(10m)` |
+| `kanboard.jsonrpc.up` | 70607 | 1 if an authenticated `getVersion` JSON-RPC call round-trips | `Kanboard JSON-RPC API is down` (id 32960): `last()=0` or `nodata(10m)` |
+
+Verified live (kb#158): pushed a manual `service.up=0`, watched the trigger fire
+(`problem.get` returned the event within seconds), then restored the real value
+and watched it clear.
+
+### Degraded mode (fabric ops fail soft when Kanboard is down)
+
+Verified/hardened as part of kb#158, tested by pointing clients at an
+unreachable `KANBOARD_URL` rather than stopping the live container:
+
+- **`kanboard-mcp`** (Adolf's and Claude's MCP tool server) already failed soft
+  — every `kanboard_*` tool handler wraps its call in try/catch and returns an
+  MCP `isError: true` result on failure; the server process itself never
+  crashes. Verified by invoking `kanboard.js`'s functions directly against an
+  unreachable URL — both calls returned clean tool errors, process stayed
+  alive. So when Kanboard is down, an Adolf turn that touches a `kanboard_*`
+  tool just sees that one tool call fail; the rest of the turn (and Matrix chat
+  in general) is unaffected.
+- **`kb_worker.py`'s daemon loop had a real bug**, fixed in kb#158: `kb-claim`'s
+  `rpc()` calls `sys.exit()` on an unreachable/erroring Kanboard, and the
+  daemon's main loop didn't catch that — an unreachable Kanboard **killed the
+  whole worker process** instead of parking it (reproduced live: pointed a
+  worker at an unreachable `KANBOARD_URL`, it died on its first loop
+  iteration). Fixed by wrapping the loop body in `except (SystemExit,
+  Exception)`, logging `parked: kanboard unreachable...` and sleeping
+  `park_interval` before retrying — same treatment as an `a(t)=0` probe.
+  Re-verified live after the fix: the same unreachable-URL run now parks and
+  keeps looping instead of dying.
+- A task claimed but not yet reported when Kanboard drops keeps its lease;
+  the lease expires and the fabric-keeper (#154) returns it to `Ready` —
+  nothing is silently lost, just delayed.
+
 ## Zabbix Monitoring
 
 Backups monitored via Zabbix on host `AgapHost`.
@@ -90,8 +177,11 @@ Backups monitored via Zabbix on host `AgapHost`.
 | `gitea.backup.age` | Seconds since last dump zip | High if > 25h |
 | `dbs.backup.age` | Seconds since last rsync (`.last_sync` mtime) | High if > 25h |
 | `immich.backup.age` | Seconds since last immich backup (`.last_sync` mtime) | High if > 25h |
-| `vaultwarden.backup.ts` | Timestamp of last backup (trapper) | Warning if > 4 days |
+| `vaultwarden.backup.ts` | Timestamp of last backup (trapper) | Warning if > 4 days (⚠ never actually populated — see bug note above) |
 | `seafile.backup.ts` | Timestamp of last backup (trapper) | Warning if > 4 days |
 | `users.backup.ts` | Timestamp of last backup (trapper) | Warning if > 4 days |
+| `kanboard.backup.ts` | Timestamp of last backup (trapper, real unixtime) | High if no backup in 4 days |
+| `kanboard.service.up` | Container health, pushed every 2 min | High if down or stale >10 min |
+| `kanboard.jsonrpc.up` | JSON-RPC API health, pushed every 2 min | High if down or stale >10 min |
 
 Agent config: `/etc/zabbix/zabbix_agent2.d/gitea_backup.conf`