alvis
a27bae828a
Work produced by the /kb driver on 2026-07-30. Each change is recorded on its Kanboard task; all remain Done-unverified or parked pending alvis's decisions. #183 agap-mcp/src/gitea.js askpassScript() and giteaWikiWrite()'s wiki checkout both used /tmp/agap-mcp-wiki, so writing the askpass helper made the dir non-empty and git clone always failed. gitea_wiki_write had likely never succeeded in production. Askpass moved to its own dir. #181 agap-mcp/src/server.js Initialise registeredToolCount at module load so /health reports the real count immediately instead of 0 until the first MCP request. #189 kanboard/backup.sh, seafile/backup.sh, vaultwarden/backup.sh, users-backup.sh, openai/backup-{hindsight-adolf,llm-dbs}.sh Remove the dead *.ts Zabbix trapper pushes (never landed). users-backup.sh also pointed at localhost:81 instead of 192.168.1.4:81 and pushed a date string into a numeric item. Freshness monitoring now rides the .age items. #192 RESTORE-RUNBOOK.md, {kanboard,seafile,vaultwarden}/restore.sh Restore path for the three services, verified in throwaway containers. Note: this work found Seafile backups have carried an empty ccnet_db.sql since 2026-07-07 -- filed as kb#222, not fixed here. #164 openai/litellm-config.yaml Metered `judge` (anthropic/claude-haiku-4-5) entry removed per alvis's 2026-07-30 decision. ANTHROPIC_API_KEY was never wired, so it could not spend. #128 openai/agent_registry.py litellm_key_spec() now also grants the routing-mode aliases, gated by the same _reachable_tiers() check as raw grants, so a small-tier agent cannot acquire automatic routing that resolves to tier-large. #219 openai/migrate-adolf-state.sh Migration script only; inert until run. Copies (never moves) the openai_adolf-state volume to /mnt/ssd/dbs/adolf, verifying a full sha256 manifest before declaring success. Tested against a throwaway volume. Deliberately NOT included, both awaiting alvis: agap-mcp/docker-compose.yml -- kb#174's contested BW_EMAIL revert (parked). openai/docker-compose.yml -- kb#219's bind-mount switch; the target dirs under /mnt/ssd/dbs/adolf do not exist yet, so committing it would let a later `compose up` recreate Adolf against empty paths. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014Y5QPagv4iun1ghpwM96Ff
196 lines
8.6 KiB
Markdown
196 lines
8.6 KiB
Markdown
# Restore Runbook — Kanboard, Vaultwarden, Seafile
|
|
|
|
Companion to each service's `backup.sh`. Covers kb#192.
|
|
|
|
Each service now has a `restore.sh` next to its `backup.sh`:
|
|
- `kanboard/restore.sh`
|
|
- `vaultwarden/restore.sh`
|
|
- `seafile/restore.sh`
|
|
|
|
All three follow the same convention: they default to the **live** container
|
|
names/paths, but every target is overridable via env vars, so the exact same
|
|
script restores into a real disaster or into a disposable/throwaway
|
|
container for a dry-run test. **Never invoke a restore.sh without env
|
|
overrides unless you are doing a real, intentional disaster recovery** —
|
|
the defaults point at production.
|
|
|
|
## Before you restore anything
|
|
|
|
1. `/mnt/backups/<service>/` is read-only in practice — copy the snapshot
|
|
you want out to a scratch dir first, don't operate on it in place.
|
|
2. Confirm you have the right snapshot: `ls /mnt/backups/<service>/` and
|
|
pick the newest dir, but **check its contents aren't empty** (see the
|
|
Seafile gotcha below — an empty dump looks like a valid directory).
|
|
3. Restoring into the live container is destructive and briefly stops the
|
|
service. Only do this for a real incident, and say so out loud before
|
|
running it.
|
|
|
|
## Kanboard
|
|
|
|
```bash
|
|
cp -r /mnt/backups/kanboard/<snapshot> /tmp/kb-restore
|
|
# Real disaster recovery (overwrites the live "kanboard" container):
|
|
cd /home/alvis/agap_git/kanboard
|
|
./restore.sh /tmp/kb-restore
|
|
```
|
|
|
|
What it does: stops the `kanboard` container, replaces `db.sqlite` via
|
|
`docker cp`, restores `plugins.tar.gz` if present, restarts, then verifies
|
|
by querying `SELECT COUNT(*) FROM tasks` through the container's PHP PDO
|
|
sqlite driver (kanboard's image has no sqlite3 CLI).
|
|
|
|
Throwaway test (no live impact — no ports published, isolated volumes):
|
|
```bash
|
|
docker volume create kb_restore_test_data
|
|
docker volume create kb_restore_test_plugins
|
|
docker run -d --name kanboard-restore-test \
|
|
-v kb_restore_test_data:/var/www/app/data \
|
|
-v kb_restore_test_plugins:/var/www/app/plugins \
|
|
-e PLUGIN_INSTALLER=true kanboard/kanboard:latest
|
|
|
|
CONTAINER=kanboard-restore-test ./restore.sh /tmp/kb-restore
|
|
|
|
# Teardown
|
|
docker rm -f kanboard-restore-test
|
|
docker volume rm kb_restore_test_data kb_restore_test_plugins
|
|
```
|
|
|
|
**Verified 2026-07-30**: restored the 2026-07-28 03:00 snapshot into a
|
|
throwaway container this way. `tasks` table came back with 180 rows;
|
|
container started healthy. Throwaway container and volumes torn down after.
|
|
|
|
## Vaultwarden
|
|
|
|
```bash
|
|
cp -r /mnt/backups/vaultwarden/<snapshot> /tmp/vw-restore
|
|
# Real disaster recovery (overwrites /mnt/ssd/dbs/vw-data and the live
|
|
# "vaultwarden" container — needs root; the data dir is root-owned):
|
|
cd /home/alvis/agap_git/vaultwarden
|
|
sudo ./restore.sh /tmp/vw-restore
|
|
```
|
|
|
|
What it does: stops the `vaultwarden` container, copies the `db_*.sqlite3`
|
|
snapshot to `db.sqlite3`, restores `config.json`, `rsa_key*`,
|
|
`attachments/`, `sends/`, restarts, then checks the DB file is in place
|
|
(the vaultwarden image has no `sqlite3` CLI either — verification falls
|
|
back to a file-presence/size check and reading the startup log for a
|
|
clean launch with no re-keying).
|
|
|
|
Throwaway test (isolated data dir, isolated container, no ports published):
|
|
```bash
|
|
mkdir -p /tmp/vw-data-test
|
|
docker run -d --name vaultwarden-restore-test --user 1000:1000 \
|
|
-v /tmp/vw-data-test:/data vaultwarden/server:latest
|
|
|
|
CONTAINER=vaultwarden-restore-test DATA_DIR=/tmp/vw-data-test \
|
|
./restore.sh /tmp/vw-restore
|
|
|
|
# Teardown
|
|
docker rm -f vaultwarden-restore-test
|
|
rm -rf /tmp/vw-data-test
|
|
```
|
|
Note: `--user 1000:1000` is only needed for the throwaway test so the bind
|
|
mount is writable by a non-root operator; the live container runs as root
|
|
and the live data dir is root-owned, so a real restore needs `sudo`.
|
|
|
|
**Verified 2026-07-30**: restored the 2026-07-28 02:00 snapshot into a
|
|
throwaway container this way. `db.sqlite3` landed at the correct size,
|
|
`config.json` was picked up ("Using saved config from `data/config.json`"
|
|
in the startup log), and the RSA key was reused rather than regenerated
|
|
(no "Private key created" line on the post-restore boot) — i.e. structural
|
|
restore confirmed. Row-level vault content was not inspected (per
|
|
Vaultwarden-handling rules — never surface real vault contents). Throwaway
|
|
container and scratch dir removed after.
|
|
|
|
## Seafile
|
|
|
|
```bash
|
|
cp -r /mnt/backups/seafile/<snapshot> /tmp/sf-restore
|
|
# Real disaster recovery (drops+reloads ccnet_db/seafile_db/seahub_db in
|
|
# the live "seafile-mysql" container, and rsyncs the data dir back into
|
|
# /mnt/misc/seafile, stopping/starting the "seafile" container around it —
|
|
# needs root; data dir is root-owned):
|
|
cd /home/alvis/agap_git/seafile
|
|
sudo ./restore.sh /tmp/sf-restore
|
|
```
|
|
|
|
What it does: refuses to run if any of the three `*.sql` dumps in the
|
|
snapshot is missing/empty (see gotcha below), then for each of
|
|
`ccnet_db`/`seafile_db`/`seahub_db`: `DROP DATABASE IF EXISTS` +
|
|
`CREATE DATABASE` + reload from the dump, and reports table counts as a
|
|
sanity check. If the snapshot has a `data/` dir and `SEAFILE_CONTAINER` is
|
|
set (default), it also stops the seafile app container, rsyncs `data/`
|
|
into `DATA_DIR`, and restarts it.
|
|
|
|
**Env-var gotcha fixed during this task**: `SEAFILE_CONTAINER=""` used to
|
|
fall back to the live container name, because `${VAR:-default}` treats an
|
|
empty string as unset. It's now `${VAR-default}` so an explicit empty
|
|
string really means "skip the data-dir step." If you want a DB-only
|
|
restore, pass `SEAFILE_CONTAINER=""` explicitly.
|
|
|
|
Throwaway test (DB-only, fully isolated MariaDB container + scratch data dir):
|
|
```bash
|
|
docker volume create sf_restore_test_db
|
|
docker run -d --name seafile-mysql-restore-test \
|
|
-e MYSQL_ROOT_PASSWORD=<test-only-pw> \
|
|
-e MYSQL_USER=seafile -e MYSQL_PASSWORD=<matches SEAFILE_MYSQL_DB_PASSWORD> \
|
|
-e MYSQL_DATABASE=placeholder \
|
|
-v sf_restore_test_db:/var/lib/mysql mariadb:10.11
|
|
# grant seafile broad perms on this throwaway instance only:
|
|
docker exec seafile-mysql-restore-test mysql -u root -p<test-only-pw> \
|
|
-e "GRANT ALL PRIVILEGES ON *.* TO 'seafile'@'%'; FLUSH PRIVILEGES;"
|
|
|
|
mkdir -p /tmp/sf-data-test
|
|
MYSQL_CONTAINER=seafile-mysql-restore-test \
|
|
SEAFILE_CONTAINER=seafile-app-does-not-exist-test \
|
|
DATA_DIR=/tmp/sf-data-test \
|
|
./restore.sh /tmp/sf-restore
|
|
|
|
# Teardown
|
|
docker rm -f seafile-mysql-restore-test
|
|
docker volume rm sf_restore_test_db
|
|
rm -rf /tmp/sf-data-test
|
|
```
|
|
|
|
**Verified 2026-07-30 — partially**: DB restore was verified end-to-end
|
|
into a throwaway MariaDB container using the last known-good snapshot
|
|
(2026-07-04 02:00 — see the critical finding below): `ccnet_db` (13
|
|
tables), `seafile_db` (46 tables), `seahub_db` (127 tables) all restored
|
|
and importable. The `data/` directory step ran cleanly against an isolated
|
|
scratch dir (`/tmp/sf-data-test`, not `/mnt/misc/seafile`) — the on-disk
|
|
tree (`seafile-data/`, `seadoc-data/`, `onlyoffice-data/`) landed as
|
|
expected. **A full running Seafile app-stack test (seafile+redis+caddy
|
|
actually serving content from the restored data) was not attempted** — that
|
|
would need the full compose stack, matching `JWT_PRIVATE_KEY`/hostname
|
|
config from `.env`, and meaningfully more setup than the box's read-only
|
|
`/mnt/backups` + non-root constraints support cleanly in one pass. If a
|
|
full app-level restore drill is wanted, treat it as separate follow-up
|
|
work with root access.
|
|
|
|
### ⚠️ Critical finding: Seafile backups have been broken since 2026-07-07
|
|
|
|
Every `/mnt/backups/seafile/<snapshot>` from **2026-07-07 through the
|
|
latest, 2026-07-28**, contains only an **empty** `ccnet_db.sql` (0 bytes)
|
|
and nothing else — no `seafile_db.sql`, `seahub_db.sql`, or `data/`. The
|
|
last known-good snapshot is **2026-07-04 02:00**. `restore.sh` now refuses
|
|
to run against a snapshot with a missing/empty dump file rather than
|
|
silently "restoring" an empty database, but **the backup cron job itself
|
|
is still broken** and needs its own fix (likely the `mysqldump` credentials
|
|
in `seafile/backup.sh` no longer matching the live `seafile` MySQL user, or
|
|
similar — not diagnosed further here since reproducing it requires running
|
|
`mysqldump` against the live `seafile-mysql` container, which is out of
|
|
scope for a restore-focused task and was blocked by the sandbox's
|
|
action classifier during this session). **Recommend filing a new,
|
|
separate task to fix `seafile/backup.sh`** — this is a live gap: three
|
|
weeks of Seafile backups are currently useless.
|
|
|
|
## Files touched
|
|
|
|
- `/home/alvis/agap_git/kanboard/restore.sh` (new)
|
|
- `/home/alvis/agap_git/vaultwarden/restore.sh` (new)
|
|
- `/home/alvis/agap_git/seafile/restore.sh` (new)
|
|
- `/home/alvis/agap_git/RESTORE-RUNBOOK.md` (this file, new)
|
|
|
|
Per the standing rule for this repo, nothing above was committed — it's
|
|
left in the working tree for a human to review and commit.
|