diff --git a/RESTORE-RUNBOOK.md b/RESTORE-RUNBOOK.md new file mode 100644 index 0000000..5af9f74 --- /dev/null +++ b/RESTORE-RUNBOOK.md @@ -0,0 +1,195 @@ +# Restore Runbook — Kanboard, Vaultwarden, Seafile + +Companion to each service's `backup.sh`. Covers kb#192. + +Each service now has a `restore.sh` next to its `backup.sh`: +- `kanboard/restore.sh` +- `vaultwarden/restore.sh` +- `seafile/restore.sh` + +All three follow the same convention: they default to the **live** container +names/paths, but every target is overridable via env vars, so the exact same +script restores into a real disaster or into a disposable/throwaway +container for a dry-run test. **Never invoke a restore.sh without env +overrides unless you are doing a real, intentional disaster recovery** — +the defaults point at production. + +## Before you restore anything + +1. `/mnt/backups//` is read-only in practice — copy the snapshot + you want out to a scratch dir first, don't operate on it in place. +2. Confirm you have the right snapshot: `ls /mnt/backups//` and + pick the newest dir, but **check its contents aren't empty** (see the + Seafile gotcha below — an empty dump looks like a valid directory). +3. Restoring into the live container is destructive and briefly stops the + service. Only do this for a real incident, and say so out loud before + running it. + +## Kanboard + +```bash +cp -r /mnt/backups/kanboard/ /tmp/kb-restore +# Real disaster recovery (overwrites the live "kanboard" container): +cd /home/alvis/agap_git/kanboard +./restore.sh /tmp/kb-restore +``` + +What it does: stops the `kanboard` container, replaces `db.sqlite` via +`docker cp`, restores `plugins.tar.gz` if present, restarts, then verifies +by querying `SELECT COUNT(*) FROM tasks` through the container's PHP PDO +sqlite driver (kanboard's image has no sqlite3 CLI). + +Throwaway test (no live impact — no ports published, isolated volumes): +```bash +docker volume create kb_restore_test_data +docker volume create kb_restore_test_plugins +docker run -d --name kanboard-restore-test \ + -v kb_restore_test_data:/var/www/app/data \ + -v kb_restore_test_plugins:/var/www/app/plugins \ + -e PLUGIN_INSTALLER=true kanboard/kanboard:latest + +CONTAINER=kanboard-restore-test ./restore.sh /tmp/kb-restore + +# Teardown +docker rm -f kanboard-restore-test +docker volume rm kb_restore_test_data kb_restore_test_plugins +``` + +**Verified 2026-07-30**: restored the 2026-07-28 03:00 snapshot into a +throwaway container this way. `tasks` table came back with 180 rows; +container started healthy. Throwaway container and volumes torn down after. + +## Vaultwarden + +```bash +cp -r /mnt/backups/vaultwarden/ /tmp/vw-restore +# Real disaster recovery (overwrites /mnt/ssd/dbs/vw-data and the live +# "vaultwarden" container — needs root; the data dir is root-owned): +cd /home/alvis/agap_git/vaultwarden +sudo ./restore.sh /tmp/vw-restore +``` + +What it does: stops the `vaultwarden` container, copies the `db_*.sqlite3` +snapshot to `db.sqlite3`, restores `config.json`, `rsa_key*`, +`attachments/`, `sends/`, restarts, then checks the DB file is in place +(the vaultwarden image has no `sqlite3` CLI either — verification falls +back to a file-presence/size check and reading the startup log for a +clean launch with no re-keying). + +Throwaway test (isolated data dir, isolated container, no ports published): +```bash +mkdir -p /tmp/vw-data-test +docker run -d --name vaultwarden-restore-test --user 1000:1000 \ + -v /tmp/vw-data-test:/data vaultwarden/server:latest + +CONTAINER=vaultwarden-restore-test DATA_DIR=/tmp/vw-data-test \ + ./restore.sh /tmp/vw-restore + +# Teardown +docker rm -f vaultwarden-restore-test +rm -rf /tmp/vw-data-test +``` +Note: `--user 1000:1000` is only needed for the throwaway test so the bind +mount is writable by a non-root operator; the live container runs as root +and the live data dir is root-owned, so a real restore needs `sudo`. + +**Verified 2026-07-30**: restored the 2026-07-28 02:00 snapshot into a +throwaway container this way. `db.sqlite3` landed at the correct size, +`config.json` was picked up ("Using saved config from `data/config.json`" +in the startup log), and the RSA key was reused rather than regenerated +(no "Private key created" line on the post-restore boot) — i.e. structural +restore confirmed. Row-level vault content was not inspected (per +Vaultwarden-handling rules — never surface real vault contents). Throwaway +container and scratch dir removed after. + +## Seafile + +```bash +cp -r /mnt/backups/seafile/ /tmp/sf-restore +# Real disaster recovery (drops+reloads ccnet_db/seafile_db/seahub_db in +# the live "seafile-mysql" container, and rsyncs the data dir back into +# /mnt/misc/seafile, stopping/starting the "seafile" container around it — +# needs root; data dir is root-owned): +cd /home/alvis/agap_git/seafile +sudo ./restore.sh /tmp/sf-restore +``` + +What it does: refuses to run if any of the three `*.sql` dumps in the +snapshot is missing/empty (see gotcha below), then for each of +`ccnet_db`/`seafile_db`/`seahub_db`: `DROP DATABASE IF EXISTS` + +`CREATE DATABASE` + reload from the dump, and reports table counts as a +sanity check. If the snapshot has a `data/` dir and `SEAFILE_CONTAINER` is +set (default), it also stops the seafile app container, rsyncs `data/` +into `DATA_DIR`, and restarts it. + +**Env-var gotcha fixed during this task**: `SEAFILE_CONTAINER=""` used to +fall back to the live container name, because `${VAR:-default}` treats an +empty string as unset. It's now `${VAR-default}` so an explicit empty +string really means "skip the data-dir step." If you want a DB-only +restore, pass `SEAFILE_CONTAINER=""` explicitly. + +Throwaway test (DB-only, fully isolated MariaDB container + scratch data dir): +```bash +docker volume create sf_restore_test_db +docker run -d --name seafile-mysql-restore-test \ + -e MYSQL_ROOT_PASSWORD= \ + -e MYSQL_USER=seafile -e MYSQL_PASSWORD= \ + -e MYSQL_DATABASE=placeholder \ + -v sf_restore_test_db:/var/lib/mysql mariadb:10.11 +# grant seafile broad perms on this throwaway instance only: +docker exec seafile-mysql-restore-test mysql -u root -p \ + -e "GRANT ALL PRIVILEGES ON *.* TO 'seafile'@'%'; FLUSH PRIVILEGES;" + +mkdir -p /tmp/sf-data-test +MYSQL_CONTAINER=seafile-mysql-restore-test \ + SEAFILE_CONTAINER=seafile-app-does-not-exist-test \ + DATA_DIR=/tmp/sf-data-test \ + ./restore.sh /tmp/sf-restore + +# Teardown +docker rm -f seafile-mysql-restore-test +docker volume rm sf_restore_test_db +rm -rf /tmp/sf-data-test +``` + +**Verified 2026-07-30 — partially**: DB restore was verified end-to-end +into a throwaway MariaDB container using the last known-good snapshot +(2026-07-04 02:00 — see the critical finding below): `ccnet_db` (13 +tables), `seafile_db` (46 tables), `seahub_db` (127 tables) all restored +and importable. The `data/` directory step ran cleanly against an isolated +scratch dir (`/tmp/sf-data-test`, not `/mnt/misc/seafile`) — the on-disk +tree (`seafile-data/`, `seadoc-data/`, `onlyoffice-data/`) landed as +expected. **A full running Seafile app-stack test (seafile+redis+caddy +actually serving content from the restored data) was not attempted** — that +would need the full compose stack, matching `JWT_PRIVATE_KEY`/hostname +config from `.env`, and meaningfully more setup than the box's read-only +`/mnt/backups` + non-root constraints support cleanly in one pass. If a +full app-level restore drill is wanted, treat it as separate follow-up +work with root access. + +### ⚠️ Critical finding: Seafile backups have been broken since 2026-07-07 + +Every `/mnt/backups/seafile/` from **2026-07-07 through the +latest, 2026-07-28**, contains only an **empty** `ccnet_db.sql` (0 bytes) +and nothing else — no `seafile_db.sql`, `seahub_db.sql`, or `data/`. The +last known-good snapshot is **2026-07-04 02:00**. `restore.sh` now refuses +to run against a snapshot with a missing/empty dump file rather than +silently "restoring" an empty database, but **the backup cron job itself +is still broken** and needs its own fix (likely the `mysqldump` credentials +in `seafile/backup.sh` no longer matching the live `seafile` MySQL user, or +similar — not diagnosed further here since reproducing it requires running +`mysqldump` against the live `seafile-mysql` container, which is out of +scope for a restore-focused task and was blocked by the sandbox's +action classifier during this session). **Recommend filing a new, +separate task to fix `seafile/backup.sh`** — this is a live gap: three +weeks of Seafile backups are currently useless. + +## Files touched + +- `/home/alvis/agap_git/kanboard/restore.sh` (new) +- `/home/alvis/agap_git/vaultwarden/restore.sh` (new) +- `/home/alvis/agap_git/seafile/restore.sh` (new) +- `/home/alvis/agap_git/RESTORE-RUNBOOK.md` (this file, new) + +Per the standing rule for this repo, nothing above was committed — it's +left in the working tree for a human to review and commit. diff --git a/agap-mcp/src/gitea.js b/agap-mcp/src/gitea.js index f021678..977d211 100644 --- a/agap-mcp/src/gitea.js +++ b/agap-mcp/src/gitea.js @@ -9,7 +9,10 @@ import { join } from 'path'; let _askpassPath = null; function askpassScript() { if (_askpassPath) return _askpassPath; - const dir = join(tmpdir(), 'agap-mcp-wiki'); + // Deliberately separate from the wiki checkout dir (agap-mcp-wiki) below — + // sharing a dir made it non-empty before `git clone` ran, so clone always + // failed with "destination path already exists" on any fresh checkout. + const dir = join(tmpdir(), 'agap-mcp-wiki-askpass'); mkdirSync(dir, { recursive: true }); const scriptPath = join(dir, 'git-askpass.sh'); writeFileSync(scriptPath, '#!/bin/sh\nprintf %s "$GITEA_ASKPASS_TOKEN"\n', { mode: 0o700 }); diff --git a/agap-mcp/src/server.js b/agap-mcp/src/server.js index 6375e14..6bac074 100644 --- a/agap-mcp/src/server.js +++ b/agap-mcp/src/server.js @@ -493,6 +493,9 @@ if (isMainModule) { console.error(`agap-mcp refusing to start: ${e.message}`); process.exit(1); } + // kb#181: compute tool count from actual server.tool registrations at startup, + // before listening, so /health has the accurate count from the beginning + createServer(); // temporary server instance just to count tools; throws away the server init() .then(() => { app.listen(PORT, () => console.log(`agap-mcp listening on :${PORT}`)); diff --git a/kanboard/backup.sh b/kanboard/backup.sh index 8b0fa5e..3116d0a 100755 --- a/kanboard/backup.sh +++ b/kanboard/backup.sh @@ -1,7 +1,7 @@ #!/bin/bash # Kanboard backup — tier-0 hardening (kb#158, A2A-26, DESIGN-a2a-agents.md v2.1 §6c). # Mirrors the vaultwarden backup.sh pattern (same repo, ~/agap_git/vaultwarden/backup.sh): -# scheduled dump -> /mnt/backups, retention of last 5, Zabbix freshness trapper. +# scheduled dump -> /mnt/backups, retention of last 5. Backup-freshness monitored via .age items. # # Runs every 3 days via alvis's user crontab (NOT root crontab like vaultwarden's -- # /mnt/backups/kanboard was bootstrapped chown'd to alvis specifically so this backup, @@ -17,9 +17,6 @@ set -euo pipefail BACKUP_DIR="/mnt/backups/kanboard" -ZABBIX_TOKEN_FILE="/home/alvis/.zabbix_token" -ZABBIX_URL="http://192.168.1.4:81/api_jsonrpc.php" -ZABBIX_ITEM_ID="70605" # kanboard.backup.ts on host AgapHost (10776) DATE=$(date '+%Y%m%d-%H%M') DEST="$BACKUP_DIR/$DATE" @@ -48,22 +45,9 @@ docker run --rm --user 1000:1000 -v kanboard_plugins:/plugins:ro -v "$DEST":/des echo "$(date): Backup complete: $DEST" ls -la "$DEST/" - -# Notify Zabbix (trapper item kanboard.backup.ts, unixtime) -- pushes a real epoch -# timestamp, unlike vaultwarden.backup.ts which (kb#158 finding) pushes a formatted -# date STRING into a numeric item and has therefore never recorded a valid value. -if [[ -f "$ZABBIX_TOKEN_FILE" ]]; then - ZABBIX_TOKEN=$(cat "$ZABBIX_TOKEN_FILE") - NOW_EPOCH=$(date '+%s') - env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \ - curl -s -X POST "$ZABBIX_URL" \ - -H "Content-Type: application/json" \ - -H "Authorization: Bearer $ZABBIX_TOKEN" \ - -d "{\"jsonrpc\":\"2.0\",\"method\":\"history.push\",\"id\":1,\"params\":{\"itemid\":\"$ZABBIX_ITEM_ID\",\"value\":$NOW_EPOCH}}" > /dev/null \ - && echo "Zabbix notified (kanboard.backup.ts=$NOW_EPOCH)." -else - echo "WARNING: $ZABBIX_TOKEN_FILE not found -- skipped Zabbix freshness push." >&2 -fi +# Backup-freshness monitoring is now done via .age items (calculated fields showing +# age of the backup). The .ts (timestamp) trappers were unreliable (history.push not +# landing); removed in kb#189 in favor of .age overdue triggers. # Rotate: keep last 5 backups ls -1dt "$BACKUP_DIR"/[0-9]*-[0-9]* 2>/dev/null | tail -n +6 | xargs -r rm -rf diff --git a/kanboard/restore.sh b/kanboard/restore.sh new file mode 100755 index 0000000..01655af --- /dev/null +++ b/kanboard/restore.sh @@ -0,0 +1,82 @@ +#!/bin/bash +# Kanboard restore — companion to backup.sh (kb#192). +# +# Restores a snapshot produced by backup.sh (db.sqlite + optional plugins.tar.gz) +# into a running Kanboard container. Defaults to the live "kanboard" container/ +# volumes, but every target is overridable via env vars so the exact same script +# can be pointed at a disposable/throwaway container for a dry-run restore test +# (see kb#192 runbook for the recommended throwaway-container recipe). +# +# Usage: +# ./restore.sh /mnt/backups/kanboard/ +# +# Env overrides (defaults = live service): +# CONTAINER=kanboard # target container name +# DATA_PATH=/var/www/app/data # data dir inside the container +# PLUGINS_PATH=/var/www/app/plugins # plugins dir inside the container +# +# WARNING: this overwrites the target container's live database. Never run +# against the "kanboard" container name unless you intend a real disaster +# recovery — for testing, point CONTAINER at a throwaway container instead. + +set -euo pipefail + +CONTAINER="${CONTAINER:-kanboard}" +DATA_PATH="${DATA_PATH:-/var/www/app/data}" +PLUGINS_PATH="${PLUGINS_PATH:-/var/www/app/plugins}" + +if [ $# -lt 1 ]; then + echo "Usage: $0 " >&2 + echo " e.g. $0 /mnt/backups/kanboard/20260728-0300" >&2 + exit 1 +fi + +SRC="$(realpath "$1")" +DB_FILE="$SRC/db.sqlite" +PLUGINS_FILE="$SRC/plugins.tar.gz" + +if [ ! -f "$DB_FILE" ]; then + echo "Error: $DB_FILE not found" >&2 + exit 1 +fi + +if ! docker inspect "$CONTAINER" > /dev/null 2>&1; then + echo "Error: container '$CONTAINER' does not exist" >&2 + exit 1 +fi + +echo "Restoring into container '$CONTAINER' from $SRC" + +# Stop the app so the sqlite file isn't being written to concurrently. +docker stop "$CONTAINER" > /dev/null + +# Replace the database file. +docker cp "$DB_FILE" "$CONTAINER:$DATA_PATH/db.sqlite" + +# Restore plugins, if present in the snapshot. +if [ -f "$PLUGINS_FILE" ]; then + docker cp "$PLUGINS_FILE" "$CONTAINER:/tmp/plugins.tar.gz" + docker start "$CONTAINER" > /dev/null + # Extract inside the container so ownership matches what the app expects. + docker exec "$CONTAINER" sh -c "rm -rf '$PLUGINS_PATH'/* && tar -xzf /tmp/plugins.tar.gz -C '$PLUGINS_PATH' && rm -f /tmp/plugins.tar.gz" +else + echo "Note: no plugins.tar.gz in snapshot, skipping plugin restore" + docker start "$CONTAINER" > /dev/null +fi + +echo "Waiting for Kanboard to come up..." +for i in $(seq 1 30); do + if docker exec "$CONTAINER" php -r 'exit(file_exists("'"$DATA_PATH"'/db.sqlite") ? 0 : 1);' > /dev/null 2>&1; then + break + fi + sleep 1 +done + +echo "Verifying restored database..." +docker exec "$CONTAINER" php -r ' +$db = new PDO("sqlite:'"$DATA_PATH"'/db.sqlite"); +$count = $db->query("SELECT COUNT(*) FROM tasks")->fetchColumn(); +echo "tasks table row count: $count\n"; +' + +echo "Restore complete: $CONTAINER" diff --git a/openai/agent_registry.py b/openai/agent_registry.py index db9f1a5..d71944a 100755 --- a/openai/agent_registry.py +++ b/openai/agent_registry.py @@ -210,6 +210,33 @@ def litellm_key_spec(registry, agent_id, model_registry=None): if name not in models: models.append(name) + # kb#128 gap (flagged 2026-07-26, closed 2026-07-30): the raw litellm_ + # model_names above (e.g. "ollama/gemma3:4b") are the BACKING deployments + # for openai/litellm-config.yaml's alias model_names -- tier-small/ + # tier-large (alvis's "tier" routing mode) and auto_router/ + # complexity_router (alvis's "automatic" routing mode). Without granting + # the aliases too, a provisioned key could reach a model directly but not + # by tier or through the router, so "all three routing modes exercisable" + # (kb#128 acceptance) wasn't actually true per-agent. Gate exactly like + # the raw grants above -- reachable tiers, not a separate allow-list -- + # so an agent's routing-mode access never exceeds its direct-model access: + # - "small" reachable -> tier-small (mirrors the always-granted small + # pool; every agent with a backbone gets at least this). + # - "large" reachable -> tier-large, PLUS auto_router/complexity_router. + # Both routers' pools include tier-large in their upper bands (COMPLEX/ + # REASONING, or the semantic "complex reasoning" route), so granting + # them to a small-only (sandboxed) agent would let automatic routing + # escalate it past its trust class -- exactly the asymmetry + # _reachable_tiers()/kb#147 exists to prevent. A small-only agent gets + # neither router: it can still call tier-small directly. + reachable = _reachable_tiers(a.get("preferred_tier")) + if "small" in reachable and "tier-small" not in models: + models.append("tier-small") + if "large" in reachable: + for alias in ("tier-large", "auto_router", "complexity_router"): + if alias not in models: + models.append(alias) + classes = registry.get("trust_classes", {}) cls = classes.get(a["trust_class"], {}) return { diff --git a/openai/backup-hindsight-adolf.sh b/openai/backup-hindsight-adolf.sh index 69d7ca3..804f97c 100755 --- a/openai/backup-hindsight-adolf.sh +++ b/openai/backup-hindsight-adolf.sh @@ -2,8 +2,8 @@ # Backup script for hindsight (Adolf's long-term memory bank) and the # openai_adolf-state Docker volume (Matrix E2EE identity/sessions + config). # Mirrors the seafile/vaultwarden/openai-llm-dbs backup.sh pattern (same repo): -# dump/tar via `docker exec`, gzip, retention of last 5, Zabbix freshness -# trapper item per target. +# dump/tar via `docker exec`, gzip, retention of last 5. Backup-freshness +# monitored via .age items. # # hindsight is an embedded Postgres (pg0) instance living at # /mnt/ssd/dbs/hindsight on the host, bind-mounted into the `hindsight` @@ -36,30 +36,14 @@ set -euo pipefail BACKUP_DIR="/mnt/backups/hindsight-adolf" -ZABBIX_TOKEN_FILE="/root/.zabbix_token" -ZABBIX_URL="http://192.168.1.4:81/api_jsonrpc.php" DATE=$(date '+%Y%m%d-%H%M') DEST="$BACKUP_DIR/$DATE" mkdir -p "$DEST" - -notify_zabbix() { - local itemid="$1" label="$2" - if [[ -f "$ZABBIX_TOKEN_FILE" ]]; then - local token now_epoch - token=$(cat "$ZABBIX_TOKEN_FILE") - now_epoch=$(date '+%s') - env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \ - curl -s -X POST "$ZABBIX_URL" \ - -H "Content-Type: application/json" \ - -H "Authorization: Bearer $token" \ - -d "{\"jsonrpc\":\"2.0\",\"method\":\"history.push\",\"id\":1,\"params\":{\"itemid\":\"$itemid\",\"value\":$now_epoch}}" > /dev/null \ - && echo "Zabbix notified ($label=$now_epoch)." - else - echo "WARNING: $ZABBIX_TOKEN_FILE not found -- skipped Zabbix freshness push for $label." >&2 - fi -} +# Backup-freshness monitoring is now done via .age items (calculated fields showing +# age of the backup). The .ts (timestamp) trappers were unreliable (history.push not +# landing); removed in kb#189 in favor of .age overdue triggers. # --- hindsight (Postgres logical dump, live/read-only) --- echo "Dumping hindsight..." @@ -67,13 +51,11 @@ docker exec -e PGPASSWORD=hindsight hindsight \ /home/hindsight/.pg0/installation/18.1.0/bin/pg_dump -U hindsight -h 127.0.0.1 -p 5432 hindsight \ | gzip > "$DEST/hindsight.sql.gz" echo "Dumped: hindsight -> $DEST/hindsight.sql.gz" -notify_zabbix "70639" "hindsight.backup.ts" # --- adolf-state (tar the volume from inside the adolf container) --- echo "Archiving adolf-state..." docker exec adolf tar czf - -C /home/node/.openclaw . > "$DEST/adolf-state.tar.gz" echo "Archived: adolf-state -> $DEST/adolf-state.tar.gz" -notify_zabbix "70640" "adolf-state.backup.ts" echo "$(date): Backup complete: $DEST" ls -la "$DEST/" diff --git a/openai/backup-llm-dbs.sh b/openai/backup-llm-dbs.sh index debb736..9a624fa 100755 --- a/openai/backup-llm-dbs.sh +++ b/openai/backup-llm-dbs.sh @@ -2,9 +2,9 @@ # Backup script for litellm-db and langfuse-db (openai stack postgres containers). # litellm-db holds provisioned virtual keys + spend; langfuse-db holds all traces. # Mirrors the seafile/vaultwarden backup.sh pattern (same repo): dump via -# `docker exec pg_dump`, gzip, retention of last 5, Zabbix freshness -# trapper item per DB. Uses pg_dump (safe against a live/running DB, no downtime -# needed — unlike gitea's stop-the-world dump). +# `docker exec pg_dump`, gzip, retention of last 5. Uses pg_dump (safe +# against a live/running DB, no downtime needed — unlike gitea's stop-the-world dump). +# Backup-freshness monitored via .age items. # # Run every 3 days via root crontab (same schedule as vaultwarden/seafile), e.g.: # 0 3 */3 * * /home/alvis/agap_git/openai/backup-llm-dbs.sh >> /mnt/backups/openai-llm-dbs/backup.log 2>&1 @@ -21,42 +21,24 @@ set -euo pipefail BACKUP_DIR="/mnt/backups/openai-llm-dbs" -ZABBIX_TOKEN_FILE="/root/.zabbix_token" -ZABBIX_URL="http://192.168.1.4:81/api_jsonrpc.php" DATE=$(date '+%Y%m%d-%H%M') DEST="$BACKUP_DIR/$DATE" mkdir -p "$DEST" - -notify_zabbix() { - local itemid="$1" label="$2" - if [[ -f "$ZABBIX_TOKEN_FILE" ]]; then - local token now_epoch - token=$(cat "$ZABBIX_TOKEN_FILE") - now_epoch=$(date '+%s') - env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \ - curl -s -X POST "$ZABBIX_URL" \ - -H "Content-Type: application/json" \ - -H "Authorization: Bearer $token" \ - -d "{\"jsonrpc\":\"2.0\",\"method\":\"history.push\",\"id\":1,\"params\":{\"itemid\":\"$itemid\",\"value\":$now_epoch}}" > /dev/null \ - && echo "Zabbix notified ($label=$now_epoch)." - else - echo "WARNING: $ZABBIX_TOKEN_FILE not found -- skipped Zabbix freshness push for $label." >&2 - fi -} +# Backup-freshness monitoring is now done via .age items (calculated fields showing +# age of the backup). The .ts (timestamp) trappers were unreliable (history.push not +# landing); removed in kb#189 in favor of .age overdue triggers. # --- litellm-db --- echo "Dumping litellm-db..." docker exec litellm-db pg_dump -U litellm litellm | gzip > "$DEST/litellm-db.sql.gz" echo "Dumped: litellm-db -> $DEST/litellm-db.sql.gz" -notify_zabbix "70637" "litellm.db.backup.ts" # --- langfuse-db --- echo "Dumping langfuse-db..." docker exec langfuse-db pg_dump -U langfuse langfuse | gzip > "$DEST/langfuse-db.sql.gz" echo "Dumped: langfuse-db -> $DEST/langfuse-db.sql.gz" -notify_zabbix "70638" "langfuse.db.backup.ts" echo "$(date): Backup complete: $DEST" ls -la "$DEST/" diff --git a/openai/litellm-config.yaml b/openai/litellm-config.yaml index 2d75854..4935877 100644 --- a/openai/litellm-config.yaml +++ b/openai/litellm-config.yaml @@ -21,10 +21,11 @@ model_list: model: ollama/bge-m3 api_base: http://host.docker.internal:11436 - - model_name: judge - litellm_params: - model: anthropic/claude-haiku-4-5-20251001 - api_key: os.environ/ANTHROPIC_API_KEY + # kb#164: the `judge` alias (anthropic/claude-haiku-4-5, metered) was removed + # 2026-07-30 by alvis's decision. No ANTHROPIC_API_KEY was ever set in this + # container or .env, so it could not spend; it was kept only as a latent + # paid-fallback footgun. Per design §3a (no metered API by default), do not + # re-add a metered deployment without an explicit opt-in decision. # Kimi Code CLI agent (own container, own Moonshot/Kimi subscription via `kimi login`) - model_name: kimi-agent diff --git a/openai/migrate-adolf-state.sh b/openai/migrate-adolf-state.sh new file mode 100755 index 0000000..67ec48c --- /dev/null +++ b/openai/migrate-adolf-state.sh @@ -0,0 +1,151 @@ +#!/bin/bash +# Migration script for kb#219 — move Adolf's runtime state off the named +# Docker volume (openai_adolf-state) onto a host bind mount at +# /mnt/ssd/dbs/adolf/state, matching the convention every other Agap +# service already follows (hindsight, litellm, qdrant, langfuse, ...). +# +# SAFETY MODEL: +# - COPY ONLY. Never touches or deletes the source volume. The volume +# stays intact and usable as a rollback source until a human explicitly +# removes it (see rollback section in the compose-diff writeup / +# kb#219 report), long after this script has run and the container has +# been soak-tested on the new mount. +# - Dry-run by default. Pass --apply to actually copy. +# - Idempotent. Safe to re-run; re-copying onto an already-populated +# destination just refreshes it (cp -a overwrite-in-place). It will +# NOT delete files at the destination that were removed from the +# source between runs -- if that matters, wipe the dest dir yourself +# before re-running. +# - Verifies file counts + a sha256 manifest diff between source and +# destination before declaring success. Non-zero exit if they disagree. +# - Uses only `docker run` (alvis is in the `docker` group -- no `sudo` +# needed for container operations) to read the volume; never reads +# /var/lib/docker/volumes directly (root-only, 0700). +# - Does NOT create /mnt/ssd/dbs/adolf itself. That directory tree is +# root-owned (/mnt/ssd/dbs is 0755 root:root, same as every other +# service dir under it) and must be created + chowned by a human with +# sudo first -- see the paste-ready root block in the kb#219 report. +# This script aborts early with a clear message if the destination +# parent doesn't exist or isn't writable. +# +# USAGE: +# ./migrate-adolf-state.sh # dry run (default), prints plan +# ./migrate-adolf-state.sh --apply # actually copies + verifies +# ./migrate-adolf-state.sh --apply --dest /path/to/scratch --volume some-test-volume +# # point at a throwaway volume/dest for a trial run +# +# This script is NOT executed against live state as part of kb#219 prep. +# It has been dry-run tested and trial-run tested against a throwaway +# volume with a handful of files (see kb#219 report for the transcript). + +set -euo pipefail + +SRC_VOLUME="openai_adolf-state" +DEST_DIR="/mnt/ssd/dbs/adolf/state" +APPLY=0 + +while [ $# -gt 0 ]; do + case "$1" in + --apply) APPLY=1; shift ;; + --dest) DEST_DIR="$2"; shift 2 ;; + --volume) SRC_VOLUME="$2"; shift 2 ;; + -h|--help) + grep '^#' "$0" | sed 's/^#//' + exit 0 + ;; + *) + echo "Unknown argument: $1" >&2 + exit 2 + ;; + esac +done + +echo "== kb#219 adolf-state migration ==" +echo "Source volume : $SRC_VOLUME" +echo "Dest dir : $DEST_DIR" +echo "Mode : $([ "$APPLY" -eq 1 ] && echo APPLY || echo DRY-RUN)" +echo + +# --- 0. sanity: source volume exists --- +if ! docker volume inspect "$SRC_VOLUME" >/dev/null 2>&1; then + echo "ERROR: source volume '$SRC_VOLUME' does not exist." >&2 + exit 1 +fi + +# --- 1. sanity: destination parent exists and is writable --- +DEST_PARENT="$(dirname "$DEST_DIR")" +if [ ! -d "$DEST_PARENT" ]; then + cat >&2 <&2 + exit 1 +fi + +mkdir -p "$DEST_DIR" + +# --- 2. source manifest (counts + sha256, computed inside a container) --- +echo "-- Computing source manifest (read-only mount of $SRC_VOLUME) --" +SRC_COUNT=$(docker run --rm -v "$SRC_VOLUME":/from:ro alpine sh -c "find /from -type f | wc -l") +echo "Source file count: $SRC_COUNT" + +if [ "$APPLY" -eq 0 ]; then + echo + echo "[DRY RUN] Would copy $SRC_COUNT files from volume '$SRC_VOLUME' into $DEST_DIR," + echo "[DRY RUN] then verify file count + sha256 manifest match." + echo "[DRY RUN] Re-run with --apply to actually copy." + exit 0 +fi + +# --- 3. copy (tar stream preserves ownership/perms across the boundary) --- +echo "-- Copying (tar stream, preserves perms/ownership) --" +docker run --rm \ + -v "$SRC_VOLUME":/from:ro \ + -v "$DEST_DIR":/to \ + alpine sh -c "cd /from && tar cf - . | (cd /to && tar xf -)" + +# --- 4. verify: file count --- +DEST_COUNT=$(docker run --rm -v "$DEST_DIR":/to:ro alpine sh -c "find /to -type f | wc -l") +echo "Dest file count: $DEST_COUNT" +if [ "$SRC_COUNT" != "$DEST_COUNT" ]; then + echo "ERROR: file count mismatch (source=$SRC_COUNT dest=$DEST_COUNT). NOT declaring success." >&2 + exit 1 +fi + +# --- 5. verify: sha256 manifest diff --- +echo "-- Verifying sha256 manifests match --" +SRC_MANIFEST=$(mktemp) +DEST_MANIFEST=$(mktemp) +trap 'rm -f "$SRC_MANIFEST" "$DEST_MANIFEST"' EXIT + +docker run --rm -v "$SRC_VOLUME":/from:ro alpine sh -c \ + "cd /from && find . -type f -exec sha256sum {} \; | sort -k2" > "$SRC_MANIFEST" +docker run --rm -v "$DEST_DIR":/to:ro alpine sh -c \ + "cd /to && find . -type f -exec sha256sum {} \; | sort -k2" > "$DEST_MANIFEST" + +if diff -u "$SRC_MANIFEST" "$DEST_MANIFEST" > /tmp/adolf-state-migration.diff; then + echo "OK: manifests match byte-for-byte ($SRC_COUNT files)." +else + echo "ERROR: manifest mismatch, see /tmp/adolf-state-migration.diff" >&2 + cat /tmp/adolf-state-migration.diff >&2 + exit 1 +fi + +echo +echo "== Migration copy verified OK ==" +echo "Source volume '$SRC_VOLUME' left untouched (not deleted, not modified)." +echo "Next steps (NOT done by this script -- human-supervised, see kb#219 report):" +echo " 1. Apply the docker-compose.yml bind-mount diff for the 'adolf' service." +echo " 2. docker compose -f openai/docker-compose.yml config -q # validate" +echo " 3. docker compose -f openai/docker-compose.yml up -d adolf # recreates container on new mount" +echo " 4. Verify: docker inspect adolf shows /mnt/ssd/dbs/adolf/state, not the volume;" +echo " Matrix session survives (no re-login), memory/config/persona intact." +echo " 5. Only after a soak period: docker volume rm $SRC_VOLUME" diff --git a/seafile/backup.sh b/seafile/backup.sh index b2d0e07..3616771 100755 --- a/seafile/backup.sh +++ b/seafile/backup.sh @@ -2,7 +2,7 @@ # Seafile backup script. # Backs up MySQL databases and seafile data directory. # Runs every 3 days via root crontab. Keeps last 5 backups. -# Notifies Zabbix (item seafile.backup.ts, id 70369 on AgapHost) after success. +# Backup-freshness monitored via .age items. set -euo pipefail @@ -29,18 +29,9 @@ rsync -a --delete \ echo "$(date): Backup complete: $DEST" ls "$DEST/" - -# Notify Zabbix -if [[ -f /root/.zabbix_token ]]; then - ZABBIX_TOKEN=$(cat /root/.zabbix_token) - NOW_EPOCH=$(date '+%s') - env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \ - curl -s -X POST http://192.168.1.4:81/api_jsonrpc.php \ - -H "Content-Type: application/json" \ - -H "Authorization: Bearer $ZABBIX_TOKEN" \ - -d "{\"jsonrpc\":\"2.0\",\"method\":\"history.push\",\"id\":1,\"params\":{\"itemid\":\"70369\",\"value\":$NOW_EPOCH}}" > /dev/null \ - && echo "Zabbix notified (seafile.backup.ts=$NOW_EPOCH)." -fi +# Backup-freshness monitoring is now done via .age items (calculated fields showing +# age of the backup). The .ts (timestamp) trappers were unreliable (history.push not +# landing); removed in kb#189 in favor of .age overdue triggers. # Rotate: keep last 5 backups ls -1dt "$BACKUP_DIR"/[0-9]*-[0-9]* 2>/dev/null | tail -n +6 | xargs -r rm -rf diff --git a/seafile/restore.sh b/seafile/restore.sh new file mode 100755 index 0000000..e08b3e1 --- /dev/null +++ b/seafile/restore.sh @@ -0,0 +1,100 @@ +#!/bin/bash +# Seafile restore — companion to backup.sh (kb#192). +# +# Restores a snapshot produced by backup.sh (ccnet_db.sql, seafile_db.sql, +# seahub_db.sql, and a data/ directory tree) into a running MariaDB +# container plus a data directory. Defaults to the live containers/paths, +# but every target is overridable via env vars so the same script can be +# pointed at throwaway containers + a scratch data dir for a dry-run +# restore test. +# +# Usage: +# ./restore.sh /mnt/backups/seafile/ +# +# Env overrides (defaults = live service): +# MYSQL_CONTAINER=seafile-mysql +# SEAFILE_CONTAINER=seafile # stopped/started around the data-dir copy; set to "" to skip +# DATA_DIR=/mnt/misc/seafile # host path the seafile-net containers bind-mount +# MYSQL_USER=seafile +# MYSQL_PASSWORD= +# +# WARNING: this drops and reloads the target's live ccnet/seafile/seahub +# databases and overwrites its data directory. Never run against the live +# "seafile-mysql"/"seafile" containers or /mnt/misc/seafile unless you +# intend a real disaster recovery — for testing, point the *_CONTAINER +# and DATA_DIR vars at throwaway equivalents instead. +# +# NOTE: restoring only the three databases (no data/ directory, e.g. +# MYSQL_CONTAINER set but SEAFILE_CONTAINER="" and no data/ present in the +# snapshot) is a valid partial restore for verifying DB integrity — +# the script skips the data-dir step automatically if snapshot has no data/. + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +[ -f "$SCRIPT_DIR/.env" ] && source "$SCRIPT_DIR/.env" + +MYSQL_CONTAINER="${MYSQL_CONTAINER:-seafile-mysql}" +# NOTE: use ${VAR-default} (no colon) for SEAFILE_CONTAINER so that an +# explicit empty string (SEAFILE_CONTAINER="") disables the data-dir step, +# as documented above — ${VAR:-default} would treat "" as unset and fall +# back to the live container name, which is not what an operator asking +# to skip that step intends. +SEAFILE_CONTAINER="${SEAFILE_CONTAINER-seafile}" +DATA_DIR="${DATA_DIR:-/mnt/misc/seafile}" +MYSQL_USER="${MYSQL_USER:-seafile}" +MYSQL_PASSWORD="${MYSQL_PASSWORD:-${SEAFILE_MYSQL_DB_PASSWORD:-FWsYYeZa15ro6x}}" + +if [ $# -lt 1 ]; then + echo "Usage: $0 " >&2 + echo " e.g. $0 /mnt/backups/seafile/20260728-0200" >&2 + exit 1 +fi + +SRC="$(realpath "$1")" + +for DB in ccnet_db seafile_db seahub_db; do + if [ ! -s "$SRC/${DB}.sql" ]; then + echo "Error: $SRC/${DB}.sql is missing or empty — refusing to restore from a broken snapshot" >&2 + exit 1 + fi +done + +if ! docker inspect "$MYSQL_CONTAINER" > /dev/null 2>&1; then + echo "Error: mysql container '$MYSQL_CONTAINER' does not exist" >&2 + exit 1 +fi + +echo "Restoring databases into '$MYSQL_CONTAINER' from $SRC" + +for DB in ccnet_db seafile_db seahub_db; do + echo "Restoring $DB..." + docker exec -i "$MYSQL_CONTAINER" mysql -u "$MYSQL_USER" -p"$MYSQL_PASSWORD" \ + -e "DROP DATABASE IF EXISTS \`$DB\`; CREATE DATABASE \`$DB\` CHARACTER SET utf8mb4;" + docker exec -i "$MYSQL_CONTAINER" mysql -u "$MYSQL_USER" -p"$MYSQL_PASSWORD" "$DB" < "$SRC/${DB}.sql" + echo "Restored: $DB" +done + +echo "Verifying restored databases..." +for DB in ccnet_db seafile_db seahub_db; do + TABLES=$(docker exec "$MYSQL_CONTAINER" mysql -u "$MYSQL_USER" -p"$MYSQL_PASSWORD" -N -e "SHOW TABLES FROM \`$DB\`;" | wc -l) + echo "$DB: $TABLES tables" +done + +if [ -d "$SRC/data" ] && [ -n "$SEAFILE_CONTAINER" ]; then + echo "Restoring data directory into $DATA_DIR..." + if docker inspect "$SEAFILE_CONTAINER" > /dev/null 2>&1; then + docker stop "$SEAFILE_CONTAINER" > /dev/null + fi + rsync -a --delete \ + --exclude='seafile-mysql/' \ + --exclude='seafile-caddy/' \ + "$SRC/data/" "$DATA_DIR/" + if docker inspect "$SEAFILE_CONTAINER" > /dev/null 2>&1; then + docker start "$SEAFILE_CONTAINER" > /dev/null + fi +else + echo "Note: no data/ dir in snapshot or SEAFILE_CONTAINER unset — skipping data-dir restore (DB-only restore)" +fi + +echo "Restore complete." diff --git a/users-backup.sh b/users-backup.sh index 86ba977..b2c44e9 100755 --- a/users-backup.sh +++ b/users-backup.sh @@ -1,7 +1,6 @@ #!/bin/bash # Backup /mnt/misc/alvis and /mnt/misc/liza to /mnt/backups/users/ -# Runs every 3 days via root crontab. -# Notifies Zabbix (item users.backup.ts, id 70379 on AgapHost) after success. +# Runs every 3 days via root crontab. Backup-freshness monitored via .age items. set -euo pipefail @@ -13,13 +12,6 @@ rsync -a --delete /mnt/misc/alvis/ "$DEST/alvis/" rsync -a --delete /mnt/misc/liza/ "$DEST/liza/" echo "$(date): Backup complete." - -# Notify Zabbix (token stored in /root/.zabbix_token) -if [[ -f /root/.zabbix_token ]]; then - ZABBIX_TOKEN=$(cat /root/.zabbix_token) - curl -s -X POST http://localhost:81/api_jsonrpc.php \ - -H "Content-Type: application/json" \ - -H "Authorization: Bearer $ZABBIX_TOKEN" \ - -d "{\"jsonrpc\":\"2.0\",\"method\":\"history.push\",\"id\":1,\"params\":{\"itemid\":\"70379\",\"value\":\"$(date '+%Y-%m-%d %H:%M')\"}}" > /dev/null \ - && echo "Zabbix notified." -fi +# Backup-freshness monitoring is now done via .age items (calculated fields showing +# age of the backup). The .ts (timestamp) trappers were unreliable (history.push not +# landing); removed in kb#189 in favor of .age overdue triggers. diff --git a/vaultwarden/backup.sh b/vaultwarden/backup.sh index faa2c97..c9817f9 100755 --- a/vaultwarden/backup.sh +++ b/vaultwarden/backup.sh @@ -1,7 +1,7 @@ #!/bin/bash # Vaultwarden backup — uses built-in container backup command (safe with live DB). # Runs every 3 days via root crontab. Keeps last 5 backups. -# Notifies Zabbix (item vaultwarden.backup.ts, id 70368 on AgapHost) after success. +# Backup-freshness monitored via .age items. set -euo pipefail @@ -26,18 +26,9 @@ cp "$DATA_DIR"/rsa_key* "$DEST/" echo "$(date): Backup complete: $DEST" ls "$DEST/" - -# Notify Zabbix (token stored in /root/.zabbix_token) -if [[ -f /root/.zabbix_token ]]; then - ZABBIX_TOKEN=$(cat /root/.zabbix_token) - NOW_EPOCH=$(date '+%s') - env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \ - curl -s -X POST http://192.168.1.4:81/api_jsonrpc.php \ - -H "Content-Type: application/json" \ - -H "Authorization: Bearer $ZABBIX_TOKEN" \ - -d "{\"jsonrpc\":\"2.0\",\"method\":\"history.push\",\"id\":1,\"params\":{\"itemid\":\"70368\",\"value\":$NOW_EPOCH}}" > /dev/null \ - && echo "Zabbix notified (vaultwarden.backup.ts=$NOW_EPOCH)." -fi +# Backup-freshness monitoring is now done via .age items (calculated fields showing +# age of the backup). The .ts (timestamp) trappers were unreliable (history.push not +# landing); removed in kb#189 in favor of .age overdue triggers. # Rotate: keep last 5 backups ls -1dt "$BACKUP_DIR"/[0-9]*-[0-9]* 2>/dev/null | tail -n +6 | xargs -r rm -rf diff --git a/vaultwarden/restore.sh b/vaultwarden/restore.sh new file mode 100755 index 0000000..bd9aca1 --- /dev/null +++ b/vaultwarden/restore.sh @@ -0,0 +1,72 @@ +#!/bin/bash +# Vaultwarden restore — companion to backup.sh (kb#192). +# +# Restores a snapshot produced by backup.sh (db_*.sqlite3, config.json, +# rsa_key*, attachments/, sends/) into a Vaultwarden data directory. +# Defaults to the live container/data dir, but every target is overridable +# via env vars so the same script can be pointed at a throwaway +# container + scratch data dir for a dry-run restore test. +# +# Usage: +# ./restore.sh /mnt/backups/vaultwarden/ +# +# Env overrides (defaults = live service): +# CONTAINER=vaultwarden +# DATA_DIR=/mnt/ssd/dbs/vw-data +# +# WARNING: this overwrites the target's live vault database. Never run +# against the "vaultwarden" container / /mnt/ssd/dbs/vw-data unless you +# intend a real disaster recovery — for testing, point CONTAINER/DATA_DIR +# at a throwaway container and a scratch directory instead. + +set -euo pipefail + +CONTAINER="${CONTAINER:-vaultwarden}" +DATA_DIR="${DATA_DIR:-/mnt/ssd/dbs/vw-data}" + +if [ $# -lt 1 ]; then + echo "Usage: $0 " >&2 + echo " e.g. $0 /mnt/backups/vaultwarden/20260728-0200" >&2 + exit 1 +fi + +SRC="$(realpath "$1")" +DB_FILE="$(find "$SRC" -maxdepth 1 -name 'db_*.sqlite3' | head -n1)" + +if [ -z "$DB_FILE" ] || [ ! -f "$DB_FILE" ]; then + echo "Error: no db_*.sqlite3 file found in $SRC" >&2 + exit 1 +fi + +echo "Restoring into '$CONTAINER' (data dir: $DATA_DIR) from $SRC" + +docker stop "$CONTAINER" > /dev/null + +mkdir -p "$DATA_DIR" +cp "$DB_FILE" "$DATA_DIR/db.sqlite3" +[ -f "$SRC/config.json" ] && cp "$SRC/config.json" "$DATA_DIR/" +[ -f "$SRC/rsa_key.pem" ] && cp "$SRC"/rsa_key* "$DATA_DIR/" 2>/dev/null || true +[ -d "$SRC/attachments" ] && rsync -a --delete "$SRC/attachments/" "$DATA_DIR/attachments/" +[ -d "$SRC/sends" ] && rsync -a --delete "$SRC/sends/" "$DATA_DIR/sends/" + +docker start "$CONTAINER" > /dev/null + +echo "Waiting for Vaultwarden to come up..." +for i in $(seq 1 30); do + if docker exec "$CONTAINER" test -f /data/db.sqlite3 > /dev/null 2>&1; then + break + fi + sleep 1 +done + +echo "Verifying restored database..." +docker exec "$CONTAINER" sh -c ' +if command -v sqlite3 >/dev/null 2>&1; then + echo "users row count: $(sqlite3 /data/db.sqlite3 "SELECT COUNT(*) FROM users;")" +else + echo "(sqlite3 CLI not present in image; file size check only)" + ls -la /data/db.sqlite3 +fi +' + +echo "Restore complete: $CONTAINER"