From a792bcf78a3d8d01cb06f2250c53836d6340342b Mon Sep 17 00:00:00 2001 From: agap-mcp Date: Tue, 21 Jul 2026 14:01:10 +0000 Subject: [PATCH] Update Backups --- Backups.md | 94 ++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 92 insertions(+), 2 deletions(-) diff --git a/Backups.md b/Backups.md index 065b711..ba347a7 100644 --- a/Backups.md +++ b/Backups.md @@ -1,6 +1,7 @@ # Backups -Daily backups run from **root's crontab** (`sudo crontab -e`). +Daily backups run from **root's crontab** (`sudo crontab -e`), except Kanboard's +(see below), which runs from **alvis's own crontab** (`crontab -e`, no sudo). ## Gitea @@ -53,6 +54,14 @@ Destination: `/mnt/backups/vaultwarden//` Retention: last 5 backups Zabbix: trapper item `vaultwarden.backup.ts` (id 70368) — trigger if no backup in 4 days +> **Known bug (found 2026-07-21, kb#158):** the script pushes a formatted date +> *string* (`date '+%Y-%m-%d %H:%M'`) into `vaultwarden.backup.ts`, which is a +> numeric (unsigned) item — every push is silently rejected, so the item has +> **never recorded a real value** (`lastclock: 0`) and its "overdue" trigger has +> been permanently in the problem state. Not fixed as part of kb#158 (out of +> scope — flagged here for a follow-up task). Kanboard's own backup below pushes +> a proper unix timestamp and was verified to actually populate the item. + ## Seafile Script: `seafile/backup.sh` — mysqldump of 3 DBs + rsync of seafile data. Runs every 3 days. @@ -80,6 +89,84 @@ Script: `users-backup.sh` — rsyncs alvis and liza home dirs from `/mnt/misc` t Zabbix: trapper item `users.backup.ts` (id 70379) — trigger (id 32869) if no backup in 4 days +## Kanboard (kb#158, A2A-26 — tier-0 hardening) + +Kanboard was promoted to **tier-0** (fabric backbone, DESIGN-a2a-agents.md v2.1 +§6c) on 2026-07-21: backups on par with the vault, plus live service/API +monitoring (see Zabbix Monitoring below). + +Script: `/home/alvis/agap_git/kanboard/backup.sh` — the `kanboard` container has +no `sqlite3` CLI and no PHP `sqlite3` extension (only `pdo_sqlite`), so instead of +a CLI dump it runs SQLite's own `VACUUM INTO` via PDO inside the container — an +atomic, consistent online snapshot safe against concurrent writers (the same +safety property `vaultwarden backup` gives that service). Also tars up the +`kanboard_plugins` volume (`PLUGIN_INSTALLER=true` means plugins can be installed +at runtime, not just baked into the image). Runs every 3 days. + +``` +0 3 */3 * * /home/alvis/agap_git/kanboard/backup.sh >> /home/alvis/agap_git/kanboard/backup.log 2>&1 +``` + +**Runs from alvis's own crontab, not root's** — unlike the other services above. +`/mnt/backups/kanboard/` is `chown`'d to `alvis` (one-time bootstrap: alvis is in +the `docker` group, which is root-equivalent on this host, so a throwaway +`alpine` container did the `mkdir`+`chown` without needing an interactive sudo +password) so the rest of the pipeline — the dump itself, the Zabbix push, the +crontab — needs no root at all. + +Destination: `/mnt/backups/kanboard//` (`db.sqlite` + `plugins.tar.gz`) +Retention: last 5 backups +Zabbix: trapper item `kanboard.backup.ts` (id 70605, real unix timestamp) — +trigger `Kanboard backup overdue (>4 days)` (id 32958) + +### Kanboard service + JSON-RPC health + +Script: `/home/alvis/agap_git/kanboard/healthcheck.sh`, every 2 minutes from +alvis's crontab: + +``` +*/2 * * * * /home/alvis/agap_git/kanboard/healthcheck.sh >> /home/alvis/agap_git/kanboard/healthcheck.log 2>&1 +``` + +Pushes two Zabbix trapper items every run (a "down" value is real signal, not a +gap to wait out via `nodata()`): + +| Item key | id | Meaning | Trigger | +|----------|----|---------|---------| +| `kanboard.service.up` | 70606 | 1 if the `kanboard` container's own Docker healthcheck reports `healthy` | `Kanboard service is down` (id 32959): `last()=0` or `nodata(10m)` | +| `kanboard.jsonrpc.up` | 70607 | 1 if an authenticated `getVersion` JSON-RPC call round-trips | `Kanboard JSON-RPC API is down` (id 32960): `last()=0` or `nodata(10m)` | + +Verified live (kb#158): pushed a manual `service.up=0`, watched the trigger fire +(`problem.get` returned the event within seconds), then restored the real value +and watched it clear. + +### Degraded mode (fabric ops fail soft when Kanboard is down) + +Verified/hardened as part of kb#158, tested by pointing clients at an +unreachable `KANBOARD_URL` rather than stopping the live container: + +- **`kanboard-mcp`** (Adolf's and Claude's MCP tool server) already failed soft + — every `kanboard_*` tool handler wraps its call in try/catch and returns an + MCP `isError: true` result on failure; the server process itself never + crashes. Verified by invoking `kanboard.js`'s functions directly against an + unreachable URL — both calls returned clean tool errors, process stayed + alive. So when Kanboard is down, an Adolf turn that touches a `kanboard_*` + tool just sees that one tool call fail; the rest of the turn (and Matrix chat + in general) is unaffected. +- **`kb_worker.py`'s daemon loop had a real bug**, fixed in kb#158: `kb-claim`'s + `rpc()` calls `sys.exit()` on an unreachable/erroring Kanboard, and the + daemon's main loop didn't catch that — an unreachable Kanboard **killed the + whole worker process** instead of parking it (reproduced live: pointed a + worker at an unreachable `KANBOARD_URL`, it died on its first loop + iteration). Fixed by wrapping the loop body in `except (SystemExit, + Exception)`, logging `parked: kanboard unreachable...` and sleeping + `park_interval` before retrying — same treatment as an `a(t)=0` probe. + Re-verified live after the fix: the same unreachable-URL run now parks and + keeps looping instead of dying. +- A task claimed but not yet reported when Kanboard drops keeps its lease; + the lease expires and the fabric-keeper (#154) returns it to `Ready` — + nothing is silently lost, just delayed. + ## Zabbix Monitoring Backups monitored via Zabbix on host `AgapHost`. @@ -90,8 +177,11 @@ Backups monitored via Zabbix on host `AgapHost`. | `gitea.backup.age` | Seconds since last dump zip | High if > 25h | | `dbs.backup.age` | Seconds since last rsync (`.last_sync` mtime) | High if > 25h | | `immich.backup.age` | Seconds since last immich backup (`.last_sync` mtime) | High if > 25h | -| `vaultwarden.backup.ts` | Timestamp of last backup (trapper) | Warning if > 4 days | +| `vaultwarden.backup.ts` | Timestamp of last backup (trapper) | Warning if > 4 days (⚠ never actually populated — see bug note above) | | `seafile.backup.ts` | Timestamp of last backup (trapper) | Warning if > 4 days | | `users.backup.ts` | Timestamp of last backup (trapper) | Warning if > 4 days | +| `kanboard.backup.ts` | Timestamp of last backup (trapper, real unixtime) | High if no backup in 4 days | +| `kanboard.service.up` | Container health, pushed every 2 min | High if down or stale >10 min | +| `kanboard.jsonrpc.up` | JSON-RPC API health, pushed every 2 min | High if down or stale >10 min | Agent config: `/etc/zabbix/zabbix_agent2.d/gitea_backup.conf`