ops: docker prune timer, kanboard backup/healthcheck, backup script fixes

docker-maintenance/ adds a systemd timer + prune.sh for the root LV that holds
Docker's data-root and has filled to 100% before, risking ENOSPC corruption.
The script sticks to the safe reclaim set (builder cache, dangling images,
stopped containers) and deliberately avoids `-a` and volume pruning, which can
destroy live data when run unattended.

kanboard/backup.sh and healthcheck.sh bring Kanboard in line with the other
services. seafile/ and vaultwarden/ backup scripts get fixes carried from the
stability audit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-30 04:42:08 +00:00
parent f67a5bee67
commit 41f3f15d27
8 changed files with 405 additions and 6 deletions

View File

@@ -0,0 +1,108 @@
# Docker Maintenance — Growth Guard (kb#184)
## Background
2026-07-24: root LV (`/`, holds Docker's data-root) filled to 1.5G free / 100% used —
risk of ENOSPC corruption across every service on Agap. Emergency reclaim (builder
prune + dangling images + stopped containers) took it to 25G free / 90% used.
2026-07-26 (this task): re-measured before any action — still 24G free / 90% used
(the emergency fix hadn't regrown, but hadn't been made recurring either). Ran
`prune.sh` live: root went to **40G free / 83% used**. Breakdown before/after:
| | Before | After |
|---|---|---|
| Images (total/reclaimable) | 88GB / 62.41GB (70%) | 87.2GB / 62.41GB (71%) |
| Build Cache (total/reclaimable) | 18.74GB / 14.49GB | 4.25GB / 0B |
| Root free | 24G (90% used) | 40G (83% used) |
The 62.41GB of image "reclaimable" space barely moved — that's almost entirely
**tagged-but-unused** images (`docker image prune -a` territory), not dangling ones.
See the decision below.
## What `prune.sh` does (safe scope only)
```
docker builder prune -f # build cache — always safe, fully rebuildable
docker container prune -f # stopped/exited containers only
docker image prune -f # DANGLING images only (untagged) — no -a
```
Logs before/after `docker system df` to `prune.log`, and pushes the post-prune
build-cache-reclaimable size (bytes) to Zabbix host `AgapHost` (hostid 10776),
trapper item `docker.buildcache.reclaimable.bytes` (itemid 70624, **type 2 —
confirmed to accept `history.push` on this Zabbix instance, unlike the type-2
"Calculated" items other tasks found dead; verified live 2026-07-26**).
Test without changing anything:
```bash
bash /home/alvis/agap_git/docker-maintenance/prune.sh --dry-run
```
## Explicitly NOT automated (human decision required)
- **`docker image prune -a`** — would reclaim ~62GB of tagged-but-currently-unused
images (e.g. `cognee/cognee-mcp:1.2.2` at 17.9GB, `cognee` variants, old
`ghcr.io/open-webui/open-webui`, `lscr.io/linuxserver/calibre`/`zotero` at
3.5GB each — many belong to services that are stopped/replaced but the image
may still be wanted for a quick restart). **Decision recorded, not made**:
someone with knowledge of which of these services are truly retired should
either (a) run `docker image prune -a` manually after reviewing the image
list (`docker system df -v`), or (b) curate a keep-list and prune around it.
Not scheduled — this script will never run `-a`.
- **`docker volume prune`** — never automated; can destroy live data for a
volume that's temporarily unmounted. Not touched by this script or its guard.
## Zabbix
- Item: `docker.buildcache.reclaimable.bytes` on host `AgapHost` (hostid 10776,
itemid 70624), type Trapper, units B. Pushed once per `prune.sh` run (post-prune
value — after a scheduled run this should almost always read near 0).
- Trigger (triggerid 32970): fires if `last()` of that item exceeds 20GB
(21474836480 bytes) — i.e. the build cache grew back past the safe threshold
between scheduled runs, meaning the timer isn't running or needs to run more
often.
- Zabbix runs on **lizacer** (`http://192.168.1.4:81`), not Agap.
## Scheduled run — NOT INSTALLED (handoff required)
This directory contains the tested script only. Installing the schedule is a
human step (per Agap automation policy: no unattended agent installs a cron/
timer that acts on a live target). To install:
**Cron** (matches the existing `agap_git/kanboard` pattern — user crontab, no sudo):
```bash
crontab -e
# add:
0 4 * * * /home/alvis/agap_git/docker-maintenance/prune.sh >> /home/alvis/agap_git/docker-maintenance/prune.log 2>&1
```
Runs daily at 04:00. Adjust frequency if build-cache growth between runs proves
faster than expected (watch the Zabbix trigger above).
Alternative — **systemd user timer**, unit files provided in this directory
(`docker-prune.service`, `docker-prune.timer`), not installed:
```bash
mkdir -p ~/.config/systemd/user
cp /home/alvis/agap_git/docker-maintenance/docker-prune.service ~/.config/systemd/user/
cp /home/alvis/agap_git/docker-maintenance/docker-prune.timer ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now docker-prune.timer
```
Either mechanism is fine; cron matches existing Agap convention (kanboard
healthcheck/backup) so it's the recommended default.
## Acceptance status (kb#184)
- [x] Immediate reclaim (2026-07-24, prior task) — done.
- [x] Recurring prune script — written, tested (dry-run + live run confirmed
it reclaims builder cache + dangling images safely).
- [ ] **Schedule installed** — script + cron line + systemd unit files are ready;
installing them is a human step (see above). **Handoff: run the `crontab -e`
command above, or install the systemd timer.**
- [x] Build-cache Zabbix trigger — created (triggerid 32970), verified
`history.push` lands (item read back 14490000000 during test, then 0
after the live prune run).
- [x] `docker image prune -a` decision — recorded above as an open human
decision, not automated.
- [x] df / free space — 40G free (83% used) as of 2026-07-26, up from 24G/90%.

View File

@@ -0,0 +1,6 @@
[Unit]
Description=Docker growth guard - safe prune (kb#184)
[Service]
Type=oneshot
ExecStart=/home/alvis/agap_git/docker-maintenance/prune.sh

View File

@@ -0,0 +1,9 @@
[Unit]
Description=Run docker growth guard prune daily (kb#184)
[Timer]
OnCalendar=*-*-* 04:00:00
Persistent=true
[Install]
WantedBy=timers.target

111
docker-maintenance/prune.sh Executable file
View File

@@ -0,0 +1,111 @@
#!/bin/bash
# Docker growth guard — recurring safe prune (kb#184, stability audit 2026-07-24).
#
# BACKGROUND: root LV (/) holds Docker's data-root. On 2026-07-24 it filled to
# 1.5G free / 100% used, risking ENOSPC corruption across every service on Agap.
# A one-off safe reclaim (builder prune + dangling images + stopped containers)
# took it from 1.5G to 25G free. This script is the recurring guard so it can't
# silently refill between now and the next manual audit.
#
# SAFE SCOPE ONLY (learned from the 2026-07-24 incident — do not expand without
# a human decision, see README.md in this directory):
# - docker builder prune -f (build cache; always safe, fully rebuildable)
# - docker container prune -f (stopped/exited containers only)
# - docker image prune -f (DANGLING images only — untagged, no -a)
#
# EXPLICITLY NOT DONE HERE (human judgment call, see README.md):
# - docker image prune -a (removes TAGGED but currently-unused images —
# could remove images a human wants kept)
# - docker volume prune (destroys live data if a volume is unmounted
# but still wanted — never run unattended)
#
# Pushes freed-space and post-prune build-cache-reclaimable metrics to Zabbix
# (trapper items on host AgapHost, hostid 10776). Zabbix lives on lizacer
# (192.168.1.4:81), not Agap.
#
# Run manually to test: bash prune.sh --dry-run
# Scheduled via cron (see README.md for the exact line — NOT installed by this
# script; installation is a human/handoff step, not something this script does).
set -euo pipefail
LOG_FILE="/home/alvis/agap_git/docker-maintenance/prune.log"
ZABBIX_TOKEN_FILE="/home/alvis/.zabbix_token"
ZABBIX_URL="http://192.168.1.4:81/api_jsonrpc.php"
ZABBIX_ITEM_ID="70624" # docker.buildcache.reclaimable.bytes on host AgapHost (10776)
DRY_RUN=0
if [[ "${1:-}" == "--dry-run" ]]; then
DRY_RUN=1
fi
log() {
echo "$(date '+%Y-%m-%d %H:%M:%S') $*"
}
{
log "=== docker-maintenance prune.sh start (dry_run=$DRY_RUN) ==="
log "--- before ---"
df -h / | tail -n +2
docker system df
if [[ "$DRY_RUN" -eq 1 ]]; then
log "DRY RUN: would run: docker builder prune -f"
log "DRY RUN: would run: docker container prune -f"
log "DRY RUN: would run: docker image prune -f (dangling only)"
else
log "Running: docker builder prune -f"
docker builder prune -f || true
log "Running: docker container prune -f"
docker container prune -f || true
log "Running: docker image prune -f (dangling only, no -a)"
docker image prune -f || true
fi
log "--- after ---"
df -h / | tail -n +2
DF_JSON=$(docker system df --format '{{json .}}')
echo "$DF_JSON"
# Extract build-cache reclaimable bytes for Zabbix. `docker system df` doesn't
# give bytes directly (only human-readable strings like "14.49GB"), so use
# `docker system df -v` reclaimable percentage isn't reliable either; instead
# compute it from `docker builder du`-equivalent: sum of build cache records
# not marked Shared/in-use is nontrivial to script robustly, so fall back to
# parsing the "Build Cache" line's RECLAIMABLE column via docker system df.
RECLAIM_STR=$(docker system df | awk '/^Build Cache/ {print $NF}')
# RECLAIM_STR looks like "14.49GB" or "0B" -- convert to bytes (approx, GB/MB/KB = *1000^n, matching docker's own decimal convention)
RECLAIM_BYTES=$(python3 -c "
import re, sys
s = '$RECLAIM_STR'
m = re.match(r'([0-9.]+)\s*([KMGTP]?B)', s)
if not m:
print(0)
else:
val, unit = float(m.group(1)), m.group(2)
mult = {'B':1,'KB':1000,'MB':1000**2,'GB':1000**3,'TB':1000**4,'PB':1000**5}[unit]
print(int(val*mult))
" 2>/dev/null || echo 0)
log "Build cache reclaimable: $RECLAIM_STR (~$RECLAIM_BYTES bytes)"
if [[ -f "$ZABBIX_TOKEN_FILE" && "$DRY_RUN" -eq 0 ]]; then
ZABBIX_TOKEN=$(cat "$ZABBIX_TOKEN_FILE")
env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \
curl -s -X POST "$ZABBIX_URL" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $ZABBIX_TOKEN" \
-d "{\"jsonrpc\":\"2.0\",\"method\":\"history.push\",\"id\":1,\"params\":{\"itemid\":\"$ZABBIX_ITEM_ID\",\"value\":$RECLAIM_BYTES}}" > /dev/null \
&& log "Zabbix notified (docker.buildcache.reclaimable.bytes=$RECLAIM_BYTES)"
else
log "Skipped Zabbix push (dry-run or token file missing)"
fi
AVAIL=$(df / | tail -1 | awk '{print $4}')
log "Root free: ${AVAIL}KB"
log "=== docker-maintenance prune.sh end ==="
} | tee -a "$LOG_FILE"

69
kanboard/backup.sh Executable file
View File

@@ -0,0 +1,69 @@
#!/bin/bash
# Kanboard backup — tier-0 hardening (kb#158, A2A-26, DESIGN-a2a-agents.md v2.1 §6c).
# Mirrors the vaultwarden backup.sh pattern (same repo, ~/agap_git/vaultwarden/backup.sh):
# scheduled dump -> /mnt/backups, retention of last 5, Zabbix freshness trapper.
#
# Runs every 3 days via alvis's user crontab (NOT root crontab like vaultwarden's --
# /mnt/backups/kanboard was bootstrapped chown'd to alvis specifically so this backup,
# like the rest of the kanboard tooling, needs no root/sudo at all. alvis is in the
# `docker` group so `docker exec`/`docker cp` need no privilege escalation either).
#
# DB dump method: kanboard's container has no sqlite3 CLI and no PHP `sqlite3`
# extension (only pdo_sqlite) -- checked directly (kb#158). Instead we run SQLite's
# own `VACUUM INTO` via PDO, which is SQLite's supported way to take an atomic,
# consistent online snapshot of a live database (safe against concurrent writers,
# same safety property `vaultwarden backup` gives us for that service).
set -euo pipefail
BACKUP_DIR="/mnt/backups/kanboard"
ZABBIX_TOKEN_FILE="/home/alvis/.zabbix_token"
ZABBIX_URL="http://192.168.1.4:81/api_jsonrpc.php"
ZABBIX_ITEM_ID="70605" # kanboard.backup.ts on host AgapHost (10776)
DATE=$(date '+%Y%m%d-%H%M')
DEST="$BACKUP_DIR/$DATE"
TMP_NAME="backup_${DATE}.sqlite"
mkdir -p "$DEST"
# Online, consistent snapshot via SQLite's VACUUM INTO (PDO sqlite driver is present
# in the image; the sqlite3 CLI/extension is not, so this replaces the vaultwarden
# `docker exec vaultwarden /vaultwarden backup` equivalent for this service).
docker exec kanboard php -r '
$db = new PDO("sqlite:/var/www/app/data/db.sqlite");
$db->exec("VACUUM INTO \"/var/www/app/data/'"$TMP_NAME"'\"");
'
# Pull the snapshot out to the host, then remove the temp copy from the live data dir
# (mirrors vaultwarden's "move the file out of DATA_DIR" step).
docker cp "kanboard:/var/www/app/data/$TMP_NAME" "$DEST/db.sqlite"
docker exec kanboard rm -f "/var/www/app/data/$TMP_NAME"
# Plugins volume (PLUGIN_INSTALLER=true means plugins can be installed at runtime,
# not just baked into the image) -- back it up too so a restore doesn't silently
# drop installed plugins.
docker run --rm --user 1000:1000 -v kanboard_plugins:/plugins:ro -v "$DEST":/dest alpine \
sh -c 'cd /plugins && tar -czf /dest/plugins.tar.gz . 2>/dev/null || true'
echo "$(date): Backup complete: $DEST"
ls -la "$DEST/"
# Notify Zabbix (trapper item kanboard.backup.ts, unixtime) -- pushes a real epoch
# timestamp, unlike vaultwarden.backup.ts which (kb#158 finding) pushes a formatted
# date STRING into a numeric item and has therefore never recorded a valid value.
if [[ -f "$ZABBIX_TOKEN_FILE" ]]; then
ZABBIX_TOKEN=$(cat "$ZABBIX_TOKEN_FILE")
NOW_EPOCH=$(date '+%s')
env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \
curl -s -X POST "$ZABBIX_URL" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $ZABBIX_TOKEN" \
-d "{\"jsonrpc\":\"2.0\",\"method\":\"history.push\",\"id\":1,\"params\":{\"itemid\":\"$ZABBIX_ITEM_ID\",\"value\":$NOW_EPOCH}}" > /dev/null \
&& echo "Zabbix notified (kanboard.backup.ts=$NOW_EPOCH)."
else
echo "WARNING: $ZABBIX_TOKEN_FILE not found -- skipped Zabbix freshness push." >&2
fi
# Rotate: keep last 5 backups
ls -1dt "$BACKUP_DIR"/[0-9]*-[0-9]* 2>/dev/null | tail -n +6 | xargs -r rm -rf

92
kanboard/healthcheck.sh Executable file
View File

@@ -0,0 +1,92 @@
#!/bin/bash
# Kanboard service + JSON-RPC API health -- tier-0 hardening (kb#158, A2A-26,
# DESIGN-a2a-agents.md v2.1 §6c: "Zabbix monitoring of the service and API").
#
# Runs every 2 minutes via alvis's user crontab (no root needed -- alvis is in the
# `docker` group). Pushes two Zabbix trapper items on host AgapHost:
# kanboard.service.up -- 1 if the `kanboard` container is running AND its own
# Docker healthcheck reports "healthy", else 0
# kanboard.jsonrpc.up -- 1 if an authenticated JSON-RPC call round-trips
# correctly, else 0
# Both are pushed every run (unlike backup.sh, which only pushes on success) so a
# transition to "down" is reported immediately rather than waiting for nodata() --
# the triggers additionally use nodata(...,10m) as a backstop in case this script
# itself stops running.
set -uo pipefail # no -e: we want to push a "0" and continue, not abort, on failure
ZABBIX_TOKEN_FILE="/home/alvis/.zabbix_token"
KANBOARD_TOKEN_FILE="/home/alvis/.kanboard_token"
ZABBIX_URL="http://192.168.1.4:81/api_jsonrpc.php"
SERVICE_ITEM_ID="70606" # kanboard.service.up
JSONRPC_ITEM_ID="70607" # kanboard.jsonrpc.up
# kb#188: tier-0 fabric container up/down. Piggybacks on this same 2-minute cron
# slot (no new cron entry) -- container:itemid map, plain `docker inspect
# .State.Running` since most of these have no HEALTHCHECK defined (only `adolf`
# does; checking bare Running is what's available uniformly here). Pushed
# unconditionally like the two items above, with nodata(...,10m) as trigger backstop.
declare -A FABRIC_ITEMS=(
["litellm"]="70625"
["litellm-db"]="70626"
["adolf"]="70627"
["adolf-llm"]="70628"
["hindsight"]="70629"
["hindsight-llm"]="70630"
["tei-reranker"]="70631"
["ollama"]="70632"
["kanboard-mcp-kanboard-mcp-1"]="70633"
["kanboard-mcp-adolf"]="70634"
["agap-mcp-agap-mcp-1"]="70635"
["fabric-keeper"]="70636"
)
# --- 1. Container health ---
HEALTH=$(docker inspect --format '{{.State.Health.Status}}' kanboard 2>/dev/null)
if [[ "$HEALTH" == "healthy" ]]; then
SERVICE_UP=1
else
SERVICE_UP=0
fi
# --- 2. JSON-RPC API health (authenticated round-trip, not just a TCP/HTTP check) ---
JSONRPC_UP=0
if [[ -f "$KANBOARD_TOKEN_FILE" ]]; then
KANBOARD_TOKEN=$(cat "$KANBOARD_TOKEN_FILE")
RESP=$(env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \
curl -s --max-time 10 -u "jsonrpc:$KANBOARD_TOKEN" -X POST http://127.0.0.1:4800/jsonrpc.php \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"getVersion","id":1}' 2>/dev/null)
if echo "$RESP" | python3 -c "import sys,json; d=json.load(sys.stdin); sys.exit(0 if 'result' in d else 1)" 2>/dev/null; then
JSONRPC_UP=1
fi
fi
# --- 3. Fabric tier-0 container up/down (kb#188) ---
FABRIC_PUSH_PARAMS="{\"itemid\":\"$SERVICE_ITEM_ID\",\"value\":$SERVICE_UP},{\"itemid\":\"$JSONRPC_ITEM_ID\",\"value\":$JSONRPC_UP}"
FABRIC_SUMMARY=""
for container in "${!FABRIC_ITEMS[@]}"; do
itemid="${FABRIC_ITEMS[$container]}"
running=$(docker inspect --format '{{.State.Running}}' "$container" 2>/dev/null)
if [[ "$running" == "true" ]]; then
up=1
else
up=0
fi
FABRIC_PUSH_PARAMS="$FABRIC_PUSH_PARAMS,{\"itemid\":\"$itemid\",\"value\":$up}"
FABRIC_SUMMARY="$FABRIC_SUMMARY $container=$up"
done
# --- 4. Push everything to Zabbix in one batch, unconditionally (down is a real value, not a gap) ---
if [[ -f "$ZABBIX_TOKEN_FILE" ]]; then
ZABBIX_TOKEN=$(cat "$ZABBIX_TOKEN_FILE")
env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \
curl -s --max-time 10 -X POST "$ZABBIX_URL" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $ZABBIX_TOKEN" \
-d "{\"jsonrpc\":\"2.0\",\"method\":\"history.push\",\"id\":1,\"params\":[$FABRIC_PUSH_PARAMS]}" > /dev/null
else
echo "WARNING: $ZABBIX_TOKEN_FILE not found -- skipped Zabbix push." >&2
fi
echo "$(date '+%Y-%m-%d %H:%M:%S') service_up=$SERVICE_UP jsonrpc_up=$JSONRPC_UP$FABRIC_SUMMARY"

View File

@@ -33,11 +33,13 @@ ls "$DEST/"
# Notify Zabbix # Notify Zabbix
if [[ -f /root/.zabbix_token ]]; then if [[ -f /root/.zabbix_token ]]; then
ZABBIX_TOKEN=$(cat /root/.zabbix_token) ZABBIX_TOKEN=$(cat /root/.zabbix_token)
curl -s -X POST http://localhost:81/api_jsonrpc.php \ NOW_EPOCH=$(date '+%s')
env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \
curl -s -X POST http://192.168.1.4:81/api_jsonrpc.php \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-H "Authorization: Bearer $ZABBIX_TOKEN" \ -H "Authorization: Bearer $ZABBIX_TOKEN" \
-d "{\"jsonrpc\":\"2.0\",\"method\":\"history.push\",\"id\":1,\"params\":{\"itemid\":\"70369\",\"value\":\"$(date '+%Y-%m-%d %H:%M')\"}}" > /dev/null \ -d "{\"jsonrpc\":\"2.0\",\"method\":\"history.push\",\"id\":1,\"params\":{\"itemid\":\"70369\",\"value\":$NOW_EPOCH}}" > /dev/null \
&& echo "Zabbix notified." && echo "Zabbix notified (seafile.backup.ts=$NOW_EPOCH)."
fi fi
# Rotate: keep last 5 backups # Rotate: keep last 5 backups

View File

@@ -30,11 +30,13 @@ ls "$DEST/"
# Notify Zabbix (token stored in /root/.zabbix_token) # Notify Zabbix (token stored in /root/.zabbix_token)
if [[ -f /root/.zabbix_token ]]; then if [[ -f /root/.zabbix_token ]]; then
ZABBIX_TOKEN=$(cat /root/.zabbix_token) ZABBIX_TOKEN=$(cat /root/.zabbix_token)
curl -s -X POST http://localhost:81/api_jsonrpc.php \ NOW_EPOCH=$(date '+%s')
env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \
curl -s -X POST http://192.168.1.4:81/api_jsonrpc.php \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-H "Authorization: Bearer $ZABBIX_TOKEN" \ -H "Authorization: Bearer $ZABBIX_TOKEN" \
-d "{\"jsonrpc\":\"2.0\",\"method\":\"history.push\",\"id\":1,\"params\":{\"itemid\":\"70368\",\"value\":\"$(date '+%Y-%m-%d %H:%M')\"}}" > /dev/null \ -d "{\"jsonrpc\":\"2.0\",\"method\":\"history.push\",\"id\":1,\"params\":{\"itemid\":\"70368\",\"value\":$NOW_EPOCH}}" > /dev/null \
&& echo "Zabbix notified." && echo "Zabbix notified (vaultwarden.backup.ts=$NOW_EPOCH)."
fi fi
# Rotate: keep last 5 backups # Rotate: keep last 5 backups