Update Backups
94
Backups.md
94
Backups.md
@@ -1,6 +1,7 @@
|
||||
# Backups
|
||||
|
||||
Daily backups run from **root's crontab** (`sudo crontab -e`).
|
||||
Daily backups run from **root's crontab** (`sudo crontab -e`), except Kanboard's
|
||||
(see below), which runs from **alvis's own crontab** (`crontab -e`, no sudo).
|
||||
|
||||
## Gitea
|
||||
|
||||
@@ -53,6 +54,14 @@ Destination: `/mnt/backups/vaultwarden/<timestamp>/`
|
||||
Retention: last 5 backups
|
||||
Zabbix: trapper item `vaultwarden.backup.ts` (id 70368) — trigger if no backup in 4 days
|
||||
|
||||
> **Known bug (found 2026-07-21, kb#158):** the script pushes a formatted date
|
||||
> *string* (`date '+%Y-%m-%d %H:%M'`) into `vaultwarden.backup.ts`, which is a
|
||||
> numeric (unsigned) item — every push is silently rejected, so the item has
|
||||
> **never recorded a real value** (`lastclock: 0`) and its "overdue" trigger has
|
||||
> been permanently in the problem state. Not fixed as part of kb#158 (out of
|
||||
> scope — flagged here for a follow-up task). Kanboard's own backup below pushes
|
||||
> a proper unix timestamp and was verified to actually populate the item.
|
||||
|
||||
## Seafile
|
||||
|
||||
Script: `seafile/backup.sh` — mysqldump of 3 DBs + rsync of seafile data. Runs every 3 days.
|
||||
@@ -80,6 +89,84 @@ Script: `users-backup.sh` — rsyncs alvis and liza home dirs from `/mnt/misc` t
|
||||
|
||||
Zabbix: trapper item `users.backup.ts` (id 70379) — trigger (id 32869) if no backup in 4 days
|
||||
|
||||
## Kanboard (kb#158, A2A-26 — tier-0 hardening)
|
||||
|
||||
Kanboard was promoted to **tier-0** (fabric backbone, DESIGN-a2a-agents.md v2.1
|
||||
§6c) on 2026-07-21: backups on par with the vault, plus live service/API
|
||||
monitoring (see Zabbix Monitoring below).
|
||||
|
||||
Script: `/home/alvis/agap_git/kanboard/backup.sh` — the `kanboard` container has
|
||||
no `sqlite3` CLI and no PHP `sqlite3` extension (only `pdo_sqlite`), so instead of
|
||||
a CLI dump it runs SQLite's own `VACUUM INTO` via PDO inside the container — an
|
||||
atomic, consistent online snapshot safe against concurrent writers (the same
|
||||
safety property `vaultwarden backup` gives that service). Also tars up the
|
||||
`kanboard_plugins` volume (`PLUGIN_INSTALLER=true` means plugins can be installed
|
||||
at runtime, not just baked into the image). Runs every 3 days.
|
||||
|
||||
```
|
||||
0 3 */3 * * /home/alvis/agap_git/kanboard/backup.sh >> /home/alvis/agap_git/kanboard/backup.log 2>&1
|
||||
```
|
||||
|
||||
**Runs from alvis's own crontab, not root's** — unlike the other services above.
|
||||
`/mnt/backups/kanboard/` is `chown`'d to `alvis` (one-time bootstrap: alvis is in
|
||||
the `docker` group, which is root-equivalent on this host, so a throwaway
|
||||
`alpine` container did the `mkdir`+`chown` without needing an interactive sudo
|
||||
password) so the rest of the pipeline — the dump itself, the Zabbix push, the
|
||||
crontab — needs no root at all.
|
||||
|
||||
Destination: `/mnt/backups/kanboard/<timestamp>/` (`db.sqlite` + `plugins.tar.gz`)
|
||||
Retention: last 5 backups
|
||||
Zabbix: trapper item `kanboard.backup.ts` (id 70605, real unix timestamp) —
|
||||
trigger `Kanboard backup overdue (>4 days)` (id 32958)
|
||||
|
||||
### Kanboard service + JSON-RPC health
|
||||
|
||||
Script: `/home/alvis/agap_git/kanboard/healthcheck.sh`, every 2 minutes from
|
||||
alvis's crontab:
|
||||
|
||||
```
|
||||
*/2 * * * * /home/alvis/agap_git/kanboard/healthcheck.sh >> /home/alvis/agap_git/kanboard/healthcheck.log 2>&1
|
||||
```
|
||||
|
||||
Pushes two Zabbix trapper items every run (a "down" value is real signal, not a
|
||||
gap to wait out via `nodata()`):
|
||||
|
||||
| Item key | id | Meaning | Trigger |
|
||||
|----------|----|---------|---------|
|
||||
| `kanboard.service.up` | 70606 | 1 if the `kanboard` container's own Docker healthcheck reports `healthy` | `Kanboard service is down` (id 32959): `last()=0` or `nodata(10m)` |
|
||||
| `kanboard.jsonrpc.up` | 70607 | 1 if an authenticated `getVersion` JSON-RPC call round-trips | `Kanboard JSON-RPC API is down` (id 32960): `last()=0` or `nodata(10m)` |
|
||||
|
||||
Verified live (kb#158): pushed a manual `service.up=0`, watched the trigger fire
|
||||
(`problem.get` returned the event within seconds), then restored the real value
|
||||
and watched it clear.
|
||||
|
||||
### Degraded mode (fabric ops fail soft when Kanboard is down)
|
||||
|
||||
Verified/hardened as part of kb#158, tested by pointing clients at an
|
||||
unreachable `KANBOARD_URL` rather than stopping the live container:
|
||||
|
||||
- **`kanboard-mcp`** (Adolf's and Claude's MCP tool server) already failed soft
|
||||
— every `kanboard_*` tool handler wraps its call in try/catch and returns an
|
||||
MCP `isError: true` result on failure; the server process itself never
|
||||
crashes. Verified by invoking `kanboard.js`'s functions directly against an
|
||||
unreachable URL — both calls returned clean tool errors, process stayed
|
||||
alive. So when Kanboard is down, an Adolf turn that touches a `kanboard_*`
|
||||
tool just sees that one tool call fail; the rest of the turn (and Matrix chat
|
||||
in general) is unaffected.
|
||||
- **`kb_worker.py`'s daemon loop had a real bug**, fixed in kb#158: `kb-claim`'s
|
||||
`rpc()` calls `sys.exit()` on an unreachable/erroring Kanboard, and the
|
||||
daemon's main loop didn't catch that — an unreachable Kanboard **killed the
|
||||
whole worker process** instead of parking it (reproduced live: pointed a
|
||||
worker at an unreachable `KANBOARD_URL`, it died on its first loop
|
||||
iteration). Fixed by wrapping the loop body in `except (SystemExit,
|
||||
Exception)`, logging `parked: kanboard unreachable...` and sleeping
|
||||
`park_interval` before retrying — same treatment as an `a(t)=0` probe.
|
||||
Re-verified live after the fix: the same unreachable-URL run now parks and
|
||||
keeps looping instead of dying.
|
||||
- A task claimed but not yet reported when Kanboard drops keeps its lease;
|
||||
the lease expires and the fabric-keeper (#154) returns it to `Ready` —
|
||||
nothing is silently lost, just delayed.
|
||||
|
||||
## Zabbix Monitoring
|
||||
|
||||
Backups monitored via Zabbix on host `AgapHost`.
|
||||
@@ -90,8 +177,11 @@ Backups monitored via Zabbix on host `AgapHost`.
|
||||
| `gitea.backup.age` | Seconds since last dump zip | High if > 25h |
|
||||
| `dbs.backup.age` | Seconds since last rsync (`.last_sync` mtime) | High if > 25h |
|
||||
| `immich.backup.age` | Seconds since last immich backup (`.last_sync` mtime) | High if > 25h |
|
||||
| `vaultwarden.backup.ts` | Timestamp of last backup (trapper) | Warning if > 4 days |
|
||||
| `vaultwarden.backup.ts` | Timestamp of last backup (trapper) | Warning if > 4 days (⚠ never actually populated — see bug note above) |
|
||||
| `seafile.backup.ts` | Timestamp of last backup (trapper) | Warning if > 4 days |
|
||||
| `users.backup.ts` | Timestamp of last backup (trapper) | Warning if > 4 days |
|
||||
| `kanboard.backup.ts` | Timestamp of last backup (trapper, real unixtime) | High if no backup in 4 days |
|
||||
| `kanboard.service.up` | Container health, pushed every 2 min | High if down or stale >10 min |
|
||||
| `kanboard.jsonrpc.up` | JSON-RPC API health, pushed every 2 min | High if down or stale >10 min |
|
||||
|
||||
Agent config: `/etc/zabbix/zabbix_agent2.d/gitea_backup.conf`
|
||||
|
||||
Reference in New Issue
Block a user