AgapHost/RESTORE-RUNBOOK.md

Restore Runbook — Kanboard, Vaultwarden, Seafile

Companion to each service's backup.sh. Covers kb#192.

Each service now has a restore.sh next to its backup.sh:

• kanboard/restore.sh
• vaultwarden/restore.sh
• seafile/restore.sh

All three follow the same convention: they default to the live container names/paths, but every target is overridable via env vars, so the exact same script restores into a real disaster or into a disposable/throwaway container for a dry-run test. Never invoke a restore.sh without env overrides unless you are doing a real, intentional disaster recovery — the defaults point at production.

Before you restore anything

1. /mnt/backups/<service>/ is read-only in practice — copy the snapshot you want out to a scratch dir first, don't operate on it in place.
2. Confirm you have the right snapshot: ls /mnt/backups/<service>/ and pick the newest dir, but check its contents aren't empty (see the Seafile gotcha below — an empty dump looks like a valid directory).
3. Restoring into the live container is destructive and briefly stops the service. Only do this for a real incident, and say so out loud before running it.

Kanboard

cp -r /mnt/backups/kanboard/<snapshot>  /tmp/kb-restore
# Real disaster recovery (overwrites the live "kanboard" container):
cd /home/alvis/agap_git/kanboard
./restore.sh /tmp/kb-restore

What it does: stops the kanboard container, replaces db.sqlite via docker cp, restores plugins.tar.gz if present, restarts, then verifies by querying SELECT COUNT(*) FROM tasks through the container's PHP PDO sqlite driver (kanboard's image has no sqlite3 CLI).

Throwaway test (no live impact — no ports published, isolated volumes):

docker volume create kb_restore_test_data
docker volume create kb_restore_test_plugins
docker run -d --name kanboard-restore-test \
  -v kb_restore_test_data:/var/www/app/data \
  -v kb_restore_test_plugins:/var/www/app/plugins \
  -e PLUGIN_INSTALLER=true kanboard/kanboard:latest

CONTAINER=kanboard-restore-test ./restore.sh /tmp/kb-restore

# Teardown
docker rm -f kanboard-restore-test
docker volume rm kb_restore_test_data kb_restore_test_plugins

Verified 2026-07-30: restored the 2026-07-28 03:00 snapshot into a throwaway container this way. tasks table came back with 180 rows; container started healthy. Throwaway container and volumes torn down after.

Vaultwarden

cp -r /mnt/backups/vaultwarden/<snapshot>  /tmp/vw-restore
# Real disaster recovery (overwrites /mnt/ssd/dbs/vw-data and the live
# "vaultwarden" container — needs root; the data dir is root-owned):
cd /home/alvis/agap_git/vaultwarden
sudo ./restore.sh /tmp/vw-restore

What it does: stops the vaultwarden container, copies the db_*.sqlite3 snapshot to db.sqlite3, restores config.json, rsa_key*, attachments/, sends/, restarts, then checks the DB file is in place (the vaultwarden image has no sqlite3 CLI either — verification falls back to a file-presence/size check and reading the startup log for a clean launch with no re-keying).

Throwaway test (isolated data dir, isolated container, no ports published):

mkdir -p /tmp/vw-data-test
docker run -d --name vaultwarden-restore-test --user 1000:1000 \
  -v /tmp/vw-data-test:/data vaultwarden/server:latest

CONTAINER=vaultwarden-restore-test DATA_DIR=/tmp/vw-data-test \
  ./restore.sh /tmp/vw-restore

# Teardown
docker rm -f vaultwarden-restore-test
rm -rf /tmp/vw-data-test

Note: --user 1000:1000 is only needed for the throwaway test so the bind mount is writable by a non-root operator; the live container runs as root and the live data dir is root-owned, so a real restore needs sudo.

Verified 2026-07-30: restored the 2026-07-28 02:00 snapshot into a throwaway container this way. db.sqlite3 landed at the correct size, config.json was picked up ("Using saved config from data/config.json" in the startup log), and the RSA key was reused rather than regenerated (no "Private key created" line on the post-restore boot) — i.e. structural restore confirmed. Row-level vault content was not inspected (per Vaultwarden-handling rules — never surface real vault contents). Throwaway container and scratch dir removed after.

Seafile

cp -r /mnt/backups/seafile/<snapshot>  /tmp/sf-restore
# Real disaster recovery (drops+reloads ccnet_db/seafile_db/seahub_db in
# the live "seafile-mysql" container, and rsyncs the data dir back into
# /mnt/misc/seafile, stopping/starting the "seafile" container around it —
# needs root; data dir is root-owned):
cd /home/alvis/agap_git/seafile
sudo ./restore.sh /tmp/sf-restore

What it does: refuses to run if any of the three *.sql dumps in the snapshot is missing/empty (see gotcha below), then for each of ccnet_db/seafile_db/seahub_db: DROP DATABASE IF EXISTS + CREATE DATABASE + reload from the dump, and reports table counts as a sanity check. If the snapshot has a data/ dir and SEAFILE_CONTAINER is set (default), it also stops the seafile app container, rsyncs data/ into DATA_DIR, and restarts it.

Env-var gotcha fixed during this task: SEAFILE_CONTAINER="" used to fall back to the live container name, because ${VAR:-default} treats an empty string as unset. It's now ${VAR-default} so an explicit empty string really means "skip the data-dir step." If you want a DB-only restore, pass SEAFILE_CONTAINER="" explicitly.

Throwaway test (DB-only, fully isolated MariaDB container + scratch data dir):

docker volume create sf_restore_test_db
docker run -d --name seafile-mysql-restore-test \
  -e MYSQL_ROOT_PASSWORD=<test-only-pw> \
  -e MYSQL_USER=seafile -e MYSQL_PASSWORD=<matches SEAFILE_MYSQL_DB_PASSWORD> \
  -e MYSQL_DATABASE=placeholder \
  -v sf_restore_test_db:/var/lib/mysql mariadb:10.11
# grant seafile broad perms on this throwaway instance only:
docker exec seafile-mysql-restore-test mysql -u root -p<test-only-pw> \
  -e "GRANT ALL PRIVILEGES ON *.* TO 'seafile'@'%'; FLUSH PRIVILEGES;"

mkdir -p /tmp/sf-data-test
MYSQL_CONTAINER=seafile-mysql-restore-test \
  SEAFILE_CONTAINER=seafile-app-does-not-exist-test \
  DATA_DIR=/tmp/sf-data-test \
  ./restore.sh /tmp/sf-restore

# Teardown
docker rm -f seafile-mysql-restore-test
docker volume rm sf_restore_test_db
rm -rf /tmp/sf-data-test

Verified 2026-07-30 — partially: DB restore was verified end-to-end into a throwaway MariaDB container using the last known-good snapshot (2026-07-04 02:00 — see the critical finding below): ccnet_db (13 tables), seafile_db (46 tables), seahub_db (127 tables) all restored and importable. The data/ directory step ran cleanly against an isolated scratch dir (/tmp/sf-data-test, not /mnt/misc/seafile) — the on-disk tree (seafile-data/, seadoc-data/, onlyoffice-data/) landed as expected. A full running Seafile app-stack test (seafile+redis+caddy actually serving content from the restored data) was not attempted — that would need the full compose stack, matching JWT_PRIVATE_KEY/hostname config from .env, and meaningfully more setup than the box's read-only /mnt/backups + non-root constraints support cleanly in one pass. If a full app-level restore drill is wanted, treat it as separate follow-up work with root access.

⚠️ Critical finding: Seafile backups have been broken since 2026-07-07

Every /mnt/backups/seafile/<snapshot> from 2026-07-07 through the latest, 2026-07-28, contains only an empty ccnet_db.sql (0 bytes) and nothing else — no seafile_db.sql, seahub_db.sql, or data/. The last known-good snapshot is 2026-07-04 02:00. restore.sh now refuses to run against a snapshot with a missing/empty dump file rather than silently "restoring" an empty database, but the backup cron job itself is still broken and needs its own fix (likely the mysqldump credentials in seafile/backup.sh no longer matching the live seafile MySQL user, or similar — not diagnosed further here since reproducing it requires running mysqldump against the live seafile-mysql container, which is out of scope for a restore-focused task and was blocked by the sandbox's action classifier during this session). Recommend filing a new, separate task to fix seafile/backup.sh — this is a live gap: three weeks of Seafile backups are currently useless.

Files touched

• /home/alvis/agap_git/kanboard/restore.sh (new)
• /home/alvis/agap_git/vaultwarden/restore.sh (new)
• /home/alvis/agap_git/seafile/restore.sh (new)
• /home/alvis/agap_git/RESTORE-RUNBOOK.md (this file, new)

Per the standing rule for this repo, nothing above was committed — it's left in the working tree for a human to review and commit.