ops: docker prune timer, kanboard backup/healthcheck, backup script fixes

docker-maintenance/ adds a systemd timer + prune.sh for the root LV that holds Docker's data-root and has filled to 100% before, risking ENOSPC corruption. The script sticks to the safe reclaim set (builder cache, dangling images, stopped containers) and deliberately avoids `-a` and volume pruning, which can destroy live data when run unattended. kanboard/backup.sh and healthcheck.sh bring Kanboard in line with the other services. seafile/ and vaultwarden/ backup scripts get fixes carried from the stability audit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 04:42:08 +00:00
parent f67a5bee67
commit 41f3f15d27
8 changed files with 405 additions and 6 deletions
--- a/docker-maintenance/README.md
+++ b/docker-maintenance/README.md
@@ -0,0 +1,108 @@
+# Docker Maintenance — Growth Guard (kb#184)
+
+## Background
+
+2026-07-24: root LV (`/`, holds Docker's data-root) filled to 1.5G free / 100% used —
+risk of ENOSPC corruption across every service on Agap. Emergency reclaim (builder
+prune + dangling images + stopped containers) took it to 25G free / 90% used.
+
+2026-07-26 (this task): re-measured before any action — still 24G free / 90% used
+(the emergency fix hadn't regrown, but hadn't been made recurring either). Ran
+`prune.sh` live: root went to **40G free / 83% used**. Breakdown before/after:
+
+| | Before | After |
+|---|---|---|
+| Images (total/reclaimable) | 88GB / 62.41GB (70%) | 87.2GB / 62.41GB (71%) |
+| Build Cache (total/reclaimable) | 18.74GB / 14.49GB | 4.25GB / 0B |
+| Root free | 24G (90% used) | 40G (83% used) |
+
+The 62.41GB of image "reclaimable" space barely moved — that's almost entirely
+**tagged-but-unused** images (`docker image prune -a` territory), not dangling ones.
+See the decision below.
+
+## What `prune.sh` does (safe scope only)
+
+```
+docker builder prune -f      # build cache — always safe, fully rebuildable
+docker container prune -f    # stopped/exited containers only
+docker image prune -f        # DANGLING images only (untagged) — no -a
+```
+
+Logs before/after `docker system df` to `prune.log`, and pushes the post-prune
+build-cache-reclaimable size (bytes) to Zabbix host `AgapHost` (hostid 10776),
+trapper item `docker.buildcache.reclaimable.bytes` (itemid 70624, **type 2 —
+confirmed to accept `history.push` on this Zabbix instance, unlike the type-2
+"Calculated" items other tasks found dead; verified live 2026-07-26**).
+
+Test without changing anything:
+```bash
+bash /home/alvis/agap_git/docker-maintenance/prune.sh --dry-run
+```
+
+## Explicitly NOT automated (human decision required)
+
+- **`docker image prune -a`** — would reclaim ~62GB of tagged-but-currently-unused
+  images (e.g. `cognee/cognee-mcp:1.2.2` at 17.9GB, `cognee` variants, old
+  `ghcr.io/open-webui/open-webui`, `lscr.io/linuxserver/calibre`/`zotero` at
+  3.5GB each — many belong to services that are stopped/replaced but the image
+  may still be wanted for a quick restart). **Decision recorded, not made**:
+  someone with knowledge of which of these services are truly retired should
+  either (a) run `docker image prune -a` manually after reviewing the image
+  list (`docker system df -v`), or (b) curate a keep-list and prune around it.
+  Not scheduled — this script will never run `-a`.
+- **`docker volume prune`** — never automated; can destroy live data for a
+  volume that's temporarily unmounted. Not touched by this script or its guard.
+
+## Zabbix
+
+- Item: `docker.buildcache.reclaimable.bytes` on host `AgapHost` (hostid 10776,
+  itemid 70624), type Trapper, units B. Pushed once per `prune.sh` run (post-prune
+  value — after a scheduled run this should almost always read near 0).
+- Trigger (triggerid 32970): fires if `last()` of that item exceeds 20GB
+  (21474836480 bytes) — i.e. the build cache grew back past the safe threshold
+  between scheduled runs, meaning the timer isn't running or needs to run more
+  often.
+- Zabbix runs on **lizacer** (`http://192.168.1.4:81`), not Agap.
+
+## Scheduled run — NOT INSTALLED (handoff required)
+
+This directory contains the tested script only. Installing the schedule is a
+human step (per Agap automation policy: no unattended agent installs a cron/
+timer that acts on a live target). To install:
+
+**Cron** (matches the existing `agap_git/kanboard` pattern — user crontab, no sudo):
+```bash
+crontab -e
+# add:
+0 4 * * * /home/alvis/agap_git/docker-maintenance/prune.sh >> /home/alvis/agap_git/docker-maintenance/prune.log 2>&1
+```
+Runs daily at 04:00. Adjust frequency if build-cache growth between runs proves
+faster than expected (watch the Zabbix trigger above).
+
+Alternative — **systemd user timer**, unit files provided in this directory
+(`docker-prune.service`, `docker-prune.timer`), not installed:
+```bash
+mkdir -p ~/.config/systemd/user
+cp /home/alvis/agap_git/docker-maintenance/docker-prune.service ~/.config/systemd/user/
+cp /home/alvis/agap_git/docker-maintenance/docker-prune.timer ~/.config/systemd/user/
+systemctl --user daemon-reload
+systemctl --user enable --now docker-prune.timer
+```
+
+Either mechanism is fine; cron matches existing Agap convention (kanboard
+healthcheck/backup) so it's the recommended default.
+
+## Acceptance status (kb#184)
+
+- [x] Immediate reclaim (2026-07-24, prior task) — done.
+- [x] Recurring prune script — written, tested (dry-run + live run confirmed
+      it reclaims builder cache + dangling images safely).
+- [ ] **Schedule installed** — script + cron line + systemd unit files are ready;
+      installing them is a human step (see above). **Handoff: run the `crontab -e`
+      command above, or install the systemd timer.**
+- [x] Build-cache Zabbix trigger — created (triggerid 32970), verified
+      `history.push` lands (item read back 14490000000 during test, then 0
+      after the live prune run).
+- [x] `docker image prune -a` decision — recorded above as an open human
+      decision, not automated.
+- [x] df / free space — 40G free (83% used) as of 2026-07-26, up from 24G/90%.
--- a/docker-maintenance/docker-prune.service
+++ b/docker-maintenance/docker-prune.service
@@ -0,0 +1,6 @@
+[Unit]
+Description=Docker growth guard - safe prune (kb#184)
+
+[Service]
+Type=oneshot
+ExecStart=/home/alvis/agap_git/docker-maintenance/prune.sh
--- a/docker-maintenance/docker-prune.timer
+++ b/docker-maintenance/docker-prune.timer
@@ -0,0 +1,9 @@
+[Unit]
+Description=Run docker growth guard prune daily (kb#184)
+
+[Timer]
+OnCalendar=*-*-* 04:00:00
+Persistent=true
+
+[Install]
+WantedBy=timers.target
--- a/docker-maintenance/prune.sh
+++ b/docker-maintenance/prune.sh
@@ -0,0 +1,111 @@
+#!/bin/bash
+# Docker growth guard — recurring safe prune (kb#184, stability audit 2026-07-24).
+#
+# BACKGROUND: root LV (/) holds Docker's data-root. On 2026-07-24 it filled to
+# 1.5G free / 100% used, risking ENOSPC corruption across every service on Agap.
+# A one-off safe reclaim (builder prune + dangling images + stopped containers)
+# took it from 1.5G to 25G free. This script is the recurring guard so it can't
+# silently refill between now and the next manual audit.
+#
+# SAFE SCOPE ONLY (learned from the 2026-07-24 incident — do not expand without
+# a human decision, see README.md in this directory):
+#   - docker builder prune -f          (build cache; always safe, fully rebuildable)
+#   - docker container prune -f        (stopped/exited containers only)
+#   - docker image prune -f            (DANGLING images only — untagged, no -a)
+#
+# EXPLICITLY NOT DONE HERE (human judgment call, see README.md):
+#   - docker image prune -a            (removes TAGGED but currently-unused images —
+#                                        could remove images a human wants kept)
+#   - docker volume prune              (destroys live data if a volume is unmounted
+#                                        but still wanted — never run unattended)
+#
+# Pushes freed-space and post-prune build-cache-reclaimable metrics to Zabbix
+# (trapper items on host AgapHost, hostid 10776). Zabbix lives on lizacer
+# (192.168.1.4:81), not Agap.
+#
+# Run manually to test: bash prune.sh --dry-run
+# Scheduled via cron (see README.md for the exact line — NOT installed by this
+# script; installation is a human/handoff step, not something this script does).
+
+set -euo pipefail
+
+LOG_FILE="/home/alvis/agap_git/docker-maintenance/prune.log"
+ZABBIX_TOKEN_FILE="/home/alvis/.zabbix_token"
+ZABBIX_URL="http://192.168.1.4:81/api_jsonrpc.php"
+ZABBIX_ITEM_ID="70624"   # docker.buildcache.reclaimable.bytes on host AgapHost (10776)
+
+DRY_RUN=0
+if [[ "${1:-}" == "--dry-run" ]]; then
+  DRY_RUN=1
+fi
+
+log() {
+  echo "$(date '+%Y-%m-%d %H:%M:%S') $*"
+}
+
+{
+  log "=== docker-maintenance prune.sh start (dry_run=$DRY_RUN) ==="
+
+  log "--- before ---"
+  df -h / | tail -n +2
+  docker system df
+
+  if [[ "$DRY_RUN" -eq 1 ]]; then
+    log "DRY RUN: would run: docker builder prune -f"
+    log "DRY RUN: would run: docker container prune -f"
+    log "DRY RUN: would run: docker image prune -f (dangling only)"
+  else
+    log "Running: docker builder prune -f"
+    docker builder prune -f || true
+
+    log "Running: docker container prune -f"
+    docker container prune -f || true
+
+    log "Running: docker image prune -f (dangling only, no -a)"
+    docker image prune -f || true
+  fi
+
+  log "--- after ---"
+  df -h / | tail -n +2
+  DF_JSON=$(docker system df --format '{{json .}}')
+  echo "$DF_JSON"
+
+  # Extract build-cache reclaimable bytes for Zabbix. `docker system df` doesn't
+  # give bytes directly (only human-readable strings like "14.49GB"), so use
+  # `docker system df -v` reclaimable percentage isn't reliable either; instead
+  # compute it from `docker builder du`-equivalent: sum of build cache records
+  # not marked Shared/in-use is nontrivial to script robustly, so fall back to
+  # parsing the "Build Cache" line's RECLAIMABLE column via docker system df.
+  RECLAIM_STR=$(docker system df | awk '/^Build Cache/ {print $NF}')
+  # RECLAIM_STR looks like "14.49GB" or "0B" -- convert to bytes (approx, GB/MB/KB = *1000^n, matching docker's own decimal convention)
+  RECLAIM_BYTES=$(python3 -c "
+import re, sys
+s = '$RECLAIM_STR'
+m = re.match(r'([0-9.]+)\s*([KMGTP]?B)', s)
+if not m:
+    print(0)
+else:
+    val, unit = float(m.group(1)), m.group(2)
+    mult = {'B':1,'KB':1000,'MB':1000**2,'GB':1000**3,'TB':1000**4,'PB':1000**5}[unit]
+    print(int(val*mult))
+" 2>/dev/null || echo 0)
+
+  log "Build cache reclaimable: $RECLAIM_STR (~$RECLAIM_BYTES bytes)"
+
+  if [[ -f "$ZABBIX_TOKEN_FILE" && "$DRY_RUN" -eq 0 ]]; then
+    ZABBIX_TOKEN=$(cat "$ZABBIX_TOKEN_FILE")
+    env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \
+      curl -s -X POST "$ZABBIX_URL" \
+      -H "Content-Type: application/json" \
+      -H "Authorization: Bearer $ZABBIX_TOKEN" \
+      -d "{\"jsonrpc\":\"2.0\",\"method\":\"history.push\",\"id\":1,\"params\":{\"itemid\":\"$ZABBIX_ITEM_ID\",\"value\":$RECLAIM_BYTES}}" > /dev/null \
+      && log "Zabbix notified (docker.buildcache.reclaimable.bytes=$RECLAIM_BYTES)"
+  else
+    log "Skipped Zabbix push (dry-run or token file missing)"
+  fi
+
+  AVAIL=$(df / | tail -1 | awk '{print $4}')
+  log "Root free: ${AVAIL}KB"
+
+  log "=== docker-maintenance prune.sh end ==="
+} | tee -a "$LOG_FILE"