alvis
41f3f15d27
docker-maintenance/ adds a systemd timer + prune.sh for the root LV that holds Docker's data-root and has filled to 100% before, risking ENOSPC corruption. The script sticks to the safe reclaim set (builder cache, dangling images, stopped containers) and deliberately avoids `-a` and volume pruning, which can destroy live data when run unattended. kanboard/backup.sh and healthcheck.sh bring Kanboard in line with the other services. seafile/ and vaultwarden/ backup scripts get fixes carried from the stability audit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
109 lines
5.1 KiB
Markdown
109 lines
5.1 KiB
Markdown
# Docker Maintenance — Growth Guard (kb#184)
|
|
|
|
## Background
|
|
|
|
2026-07-24: root LV (`/`, holds Docker's data-root) filled to 1.5G free / 100% used —
|
|
risk of ENOSPC corruption across every service on Agap. Emergency reclaim (builder
|
|
prune + dangling images + stopped containers) took it to 25G free / 90% used.
|
|
|
|
2026-07-26 (this task): re-measured before any action — still 24G free / 90% used
|
|
(the emergency fix hadn't regrown, but hadn't been made recurring either). Ran
|
|
`prune.sh` live: root went to **40G free / 83% used**. Breakdown before/after:
|
|
|
|
| | Before | After |
|
|
|---|---|---|
|
|
| Images (total/reclaimable) | 88GB / 62.41GB (70%) | 87.2GB / 62.41GB (71%) |
|
|
| Build Cache (total/reclaimable) | 18.74GB / 14.49GB | 4.25GB / 0B |
|
|
| Root free | 24G (90% used) | 40G (83% used) |
|
|
|
|
The 62.41GB of image "reclaimable" space barely moved — that's almost entirely
|
|
**tagged-but-unused** images (`docker image prune -a` territory), not dangling ones.
|
|
See the decision below.
|
|
|
|
## What `prune.sh` does (safe scope only)
|
|
|
|
```
|
|
docker builder prune -f # build cache — always safe, fully rebuildable
|
|
docker container prune -f # stopped/exited containers only
|
|
docker image prune -f # DANGLING images only (untagged) — no -a
|
|
```
|
|
|
|
Logs before/after `docker system df` to `prune.log`, and pushes the post-prune
|
|
build-cache-reclaimable size (bytes) to Zabbix host `AgapHost` (hostid 10776),
|
|
trapper item `docker.buildcache.reclaimable.bytes` (itemid 70624, **type 2 —
|
|
confirmed to accept `history.push` on this Zabbix instance, unlike the type-2
|
|
"Calculated" items other tasks found dead; verified live 2026-07-26**).
|
|
|
|
Test without changing anything:
|
|
```bash
|
|
bash /home/alvis/agap_git/docker-maintenance/prune.sh --dry-run
|
|
```
|
|
|
|
## Explicitly NOT automated (human decision required)
|
|
|
|
- **`docker image prune -a`** — would reclaim ~62GB of tagged-but-currently-unused
|
|
images (e.g. `cognee/cognee-mcp:1.2.2` at 17.9GB, `cognee` variants, old
|
|
`ghcr.io/open-webui/open-webui`, `lscr.io/linuxserver/calibre`/`zotero` at
|
|
3.5GB each — many belong to services that are stopped/replaced but the image
|
|
may still be wanted for a quick restart). **Decision recorded, not made**:
|
|
someone with knowledge of which of these services are truly retired should
|
|
either (a) run `docker image prune -a` manually after reviewing the image
|
|
list (`docker system df -v`), or (b) curate a keep-list and prune around it.
|
|
Not scheduled — this script will never run `-a`.
|
|
- **`docker volume prune`** — never automated; can destroy live data for a
|
|
volume that's temporarily unmounted. Not touched by this script or its guard.
|
|
|
|
## Zabbix
|
|
|
|
- Item: `docker.buildcache.reclaimable.bytes` on host `AgapHost` (hostid 10776,
|
|
itemid 70624), type Trapper, units B. Pushed once per `prune.sh` run (post-prune
|
|
value — after a scheduled run this should almost always read near 0).
|
|
- Trigger (triggerid 32970): fires if `last()` of that item exceeds 20GB
|
|
(21474836480 bytes) — i.e. the build cache grew back past the safe threshold
|
|
between scheduled runs, meaning the timer isn't running or needs to run more
|
|
often.
|
|
- Zabbix runs on **lizacer** (`http://192.168.1.4:81`), not Agap.
|
|
|
|
## Scheduled run — NOT INSTALLED (handoff required)
|
|
|
|
This directory contains the tested script only. Installing the schedule is a
|
|
human step (per Agap automation policy: no unattended agent installs a cron/
|
|
timer that acts on a live target). To install:
|
|
|
|
**Cron** (matches the existing `agap_git/kanboard` pattern — user crontab, no sudo):
|
|
```bash
|
|
crontab -e
|
|
# add:
|
|
0 4 * * * /home/alvis/agap_git/docker-maintenance/prune.sh >> /home/alvis/agap_git/docker-maintenance/prune.log 2>&1
|
|
```
|
|
Runs daily at 04:00. Adjust frequency if build-cache growth between runs proves
|
|
faster than expected (watch the Zabbix trigger above).
|
|
|
|
Alternative — **systemd user timer**, unit files provided in this directory
|
|
(`docker-prune.service`, `docker-prune.timer`), not installed:
|
|
```bash
|
|
mkdir -p ~/.config/systemd/user
|
|
cp /home/alvis/agap_git/docker-maintenance/docker-prune.service ~/.config/systemd/user/
|
|
cp /home/alvis/agap_git/docker-maintenance/docker-prune.timer ~/.config/systemd/user/
|
|
systemctl --user daemon-reload
|
|
systemctl --user enable --now docker-prune.timer
|
|
```
|
|
|
|
Either mechanism is fine; cron matches existing Agap convention (kanboard
|
|
healthcheck/backup) so it's the recommended default.
|
|
|
|
## Acceptance status (kb#184)
|
|
|
|
- [x] Immediate reclaim (2026-07-24, prior task) — done.
|
|
- [x] Recurring prune script — written, tested (dry-run + live run confirmed
|
|
it reclaims builder cache + dangling images safely).
|
|
- [ ] **Schedule installed** — script + cron line + systemd unit files are ready;
|
|
installing them is a human step (see above). **Handoff: run the `crontab -e`
|
|
command above, or install the systemd timer.**
|
|
- [x] Build-cache Zabbix trigger — created (triggerid 32970), verified
|
|
`history.push` lands (item read back 14490000000 during test, then 0
|
|
after the live prune run).
|
|
- [x] `docker image prune -a` decision — recorded above as an open human
|
|
decision, not automated.
|
|
- [x] df / free space — 40G free (83% used) as of 2026-07-26, up from 24G/90%.
|