ops: docker prune timer, kanboard backup/healthcheck, backup script fixes
docker-maintenance/ adds a systemd timer + prune.sh for the root LV that holds Docker's data-root and has filled to 100% before, risking ENOSPC corruption. The script sticks to the safe reclaim set (builder cache, dangling images, stopped containers) and deliberately avoids `-a` and volume pruning, which can destroy live data when run unattended. kanboard/backup.sh and healthcheck.sh bring Kanboard in line with the other services. seafile/ and vaultwarden/ backup scripts get fixes carried from the stability audit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
108
docker-maintenance/README.md
Normal file
108
docker-maintenance/README.md
Normal file
@@ -0,0 +1,108 @@
|
|||||||
|
# Docker Maintenance — Growth Guard (kb#184)
|
||||||
|
|
||||||
|
## Background
|
||||||
|
|
||||||
|
2026-07-24: root LV (`/`, holds Docker's data-root) filled to 1.5G free / 100% used —
|
||||||
|
risk of ENOSPC corruption across every service on Agap. Emergency reclaim (builder
|
||||||
|
prune + dangling images + stopped containers) took it to 25G free / 90% used.
|
||||||
|
|
||||||
|
2026-07-26 (this task): re-measured before any action — still 24G free / 90% used
|
||||||
|
(the emergency fix hadn't regrown, but hadn't been made recurring either). Ran
|
||||||
|
`prune.sh` live: root went to **40G free / 83% used**. Breakdown before/after:
|
||||||
|
|
||||||
|
| | Before | After |
|
||||||
|
|---|---|---|
|
||||||
|
| Images (total/reclaimable) | 88GB / 62.41GB (70%) | 87.2GB / 62.41GB (71%) |
|
||||||
|
| Build Cache (total/reclaimable) | 18.74GB / 14.49GB | 4.25GB / 0B |
|
||||||
|
| Root free | 24G (90% used) | 40G (83% used) |
|
||||||
|
|
||||||
|
The 62.41GB of image "reclaimable" space barely moved — that's almost entirely
|
||||||
|
**tagged-but-unused** images (`docker image prune -a` territory), not dangling ones.
|
||||||
|
See the decision below.
|
||||||
|
|
||||||
|
## What `prune.sh` does (safe scope only)
|
||||||
|
|
||||||
|
```
|
||||||
|
docker builder prune -f # build cache — always safe, fully rebuildable
|
||||||
|
docker container prune -f # stopped/exited containers only
|
||||||
|
docker image prune -f # DANGLING images only (untagged) — no -a
|
||||||
|
```
|
||||||
|
|
||||||
|
Logs before/after `docker system df` to `prune.log`, and pushes the post-prune
|
||||||
|
build-cache-reclaimable size (bytes) to Zabbix host `AgapHost` (hostid 10776),
|
||||||
|
trapper item `docker.buildcache.reclaimable.bytes` (itemid 70624, **type 2 —
|
||||||
|
confirmed to accept `history.push` on this Zabbix instance, unlike the type-2
|
||||||
|
"Calculated" items other tasks found dead; verified live 2026-07-26**).
|
||||||
|
|
||||||
|
Test without changing anything:
|
||||||
|
```bash
|
||||||
|
bash /home/alvis/agap_git/docker-maintenance/prune.sh --dry-run
|
||||||
|
```
|
||||||
|
|
||||||
|
## Explicitly NOT automated (human decision required)
|
||||||
|
|
||||||
|
- **`docker image prune -a`** — would reclaim ~62GB of tagged-but-currently-unused
|
||||||
|
images (e.g. `cognee/cognee-mcp:1.2.2` at 17.9GB, `cognee` variants, old
|
||||||
|
`ghcr.io/open-webui/open-webui`, `lscr.io/linuxserver/calibre`/`zotero` at
|
||||||
|
3.5GB each — many belong to services that are stopped/replaced but the image
|
||||||
|
may still be wanted for a quick restart). **Decision recorded, not made**:
|
||||||
|
someone with knowledge of which of these services are truly retired should
|
||||||
|
either (a) run `docker image prune -a` manually after reviewing the image
|
||||||
|
list (`docker system df -v`), or (b) curate a keep-list and prune around it.
|
||||||
|
Not scheduled — this script will never run `-a`.
|
||||||
|
- **`docker volume prune`** — never automated; can destroy live data for a
|
||||||
|
volume that's temporarily unmounted. Not touched by this script or its guard.
|
||||||
|
|
||||||
|
## Zabbix
|
||||||
|
|
||||||
|
- Item: `docker.buildcache.reclaimable.bytes` on host `AgapHost` (hostid 10776,
|
||||||
|
itemid 70624), type Trapper, units B. Pushed once per `prune.sh` run (post-prune
|
||||||
|
value — after a scheduled run this should almost always read near 0).
|
||||||
|
- Trigger (triggerid 32970): fires if `last()` of that item exceeds 20GB
|
||||||
|
(21474836480 bytes) — i.e. the build cache grew back past the safe threshold
|
||||||
|
between scheduled runs, meaning the timer isn't running or needs to run more
|
||||||
|
often.
|
||||||
|
- Zabbix runs on **lizacer** (`http://192.168.1.4:81`), not Agap.
|
||||||
|
|
||||||
|
## Scheduled run — NOT INSTALLED (handoff required)
|
||||||
|
|
||||||
|
This directory contains the tested script only. Installing the schedule is a
|
||||||
|
human step (per Agap automation policy: no unattended agent installs a cron/
|
||||||
|
timer that acts on a live target). To install:
|
||||||
|
|
||||||
|
**Cron** (matches the existing `agap_git/kanboard` pattern — user crontab, no sudo):
|
||||||
|
```bash
|
||||||
|
crontab -e
|
||||||
|
# add:
|
||||||
|
0 4 * * * /home/alvis/agap_git/docker-maintenance/prune.sh >> /home/alvis/agap_git/docker-maintenance/prune.log 2>&1
|
||||||
|
```
|
||||||
|
Runs daily at 04:00. Adjust frequency if build-cache growth between runs proves
|
||||||
|
faster than expected (watch the Zabbix trigger above).
|
||||||
|
|
||||||
|
Alternative — **systemd user timer**, unit files provided in this directory
|
||||||
|
(`docker-prune.service`, `docker-prune.timer`), not installed:
|
||||||
|
```bash
|
||||||
|
mkdir -p ~/.config/systemd/user
|
||||||
|
cp /home/alvis/agap_git/docker-maintenance/docker-prune.service ~/.config/systemd/user/
|
||||||
|
cp /home/alvis/agap_git/docker-maintenance/docker-prune.timer ~/.config/systemd/user/
|
||||||
|
systemctl --user daemon-reload
|
||||||
|
systemctl --user enable --now docker-prune.timer
|
||||||
|
```
|
||||||
|
|
||||||
|
Either mechanism is fine; cron matches existing Agap convention (kanboard
|
||||||
|
healthcheck/backup) so it's the recommended default.
|
||||||
|
|
||||||
|
## Acceptance status (kb#184)
|
||||||
|
|
||||||
|
- [x] Immediate reclaim (2026-07-24, prior task) — done.
|
||||||
|
- [x] Recurring prune script — written, tested (dry-run + live run confirmed
|
||||||
|
it reclaims builder cache + dangling images safely).
|
||||||
|
- [ ] **Schedule installed** — script + cron line + systemd unit files are ready;
|
||||||
|
installing them is a human step (see above). **Handoff: run the `crontab -e`
|
||||||
|
command above, or install the systemd timer.**
|
||||||
|
- [x] Build-cache Zabbix trigger — created (triggerid 32970), verified
|
||||||
|
`history.push` lands (item read back 14490000000 during test, then 0
|
||||||
|
after the live prune run).
|
||||||
|
- [x] `docker image prune -a` decision — recorded above as an open human
|
||||||
|
decision, not automated.
|
||||||
|
- [x] df / free space — 40G free (83% used) as of 2026-07-26, up from 24G/90%.
|
||||||
6
docker-maintenance/docker-prune.service
Normal file
6
docker-maintenance/docker-prune.service
Normal file
@@ -0,0 +1,6 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Docker growth guard - safe prune (kb#184)
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
ExecStart=/home/alvis/agap_git/docker-maintenance/prune.sh
|
||||||
9
docker-maintenance/docker-prune.timer
Normal file
9
docker-maintenance/docker-prune.timer
Normal file
@@ -0,0 +1,9 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Run docker growth guard prune daily (kb#184)
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnCalendar=*-*-* 04:00:00
|
||||||
|
Persistent=true
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
111
docker-maintenance/prune.sh
Executable file
111
docker-maintenance/prune.sh
Executable file
@@ -0,0 +1,111 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Docker growth guard — recurring safe prune (kb#184, stability audit 2026-07-24).
|
||||||
|
#
|
||||||
|
# BACKGROUND: root LV (/) holds Docker's data-root. On 2026-07-24 it filled to
|
||||||
|
# 1.5G free / 100% used, risking ENOSPC corruption across every service on Agap.
|
||||||
|
# A one-off safe reclaim (builder prune + dangling images + stopped containers)
|
||||||
|
# took it from 1.5G to 25G free. This script is the recurring guard so it can't
|
||||||
|
# silently refill between now and the next manual audit.
|
||||||
|
#
|
||||||
|
# SAFE SCOPE ONLY (learned from the 2026-07-24 incident — do not expand without
|
||||||
|
# a human decision, see README.md in this directory):
|
||||||
|
# - docker builder prune -f (build cache; always safe, fully rebuildable)
|
||||||
|
# - docker container prune -f (stopped/exited containers only)
|
||||||
|
# - docker image prune -f (DANGLING images only — untagged, no -a)
|
||||||
|
#
|
||||||
|
# EXPLICITLY NOT DONE HERE (human judgment call, see README.md):
|
||||||
|
# - docker image prune -a (removes TAGGED but currently-unused images —
|
||||||
|
# could remove images a human wants kept)
|
||||||
|
# - docker volume prune (destroys live data if a volume is unmounted
|
||||||
|
# but still wanted — never run unattended)
|
||||||
|
#
|
||||||
|
# Pushes freed-space and post-prune build-cache-reclaimable metrics to Zabbix
|
||||||
|
# (trapper items on host AgapHost, hostid 10776). Zabbix lives on lizacer
|
||||||
|
# (192.168.1.4:81), not Agap.
|
||||||
|
#
|
||||||
|
# Run manually to test: bash prune.sh --dry-run
|
||||||
|
# Scheduled via cron (see README.md for the exact line — NOT installed by this
|
||||||
|
# script; installation is a human/handoff step, not something this script does).
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
LOG_FILE="/home/alvis/agap_git/docker-maintenance/prune.log"
|
||||||
|
ZABBIX_TOKEN_FILE="/home/alvis/.zabbix_token"
|
||||||
|
ZABBIX_URL="http://192.168.1.4:81/api_jsonrpc.php"
|
||||||
|
ZABBIX_ITEM_ID="70624" # docker.buildcache.reclaimable.bytes on host AgapHost (10776)
|
||||||
|
|
||||||
|
DRY_RUN=0
|
||||||
|
if [[ "${1:-}" == "--dry-run" ]]; then
|
||||||
|
DRY_RUN=1
|
||||||
|
fi
|
||||||
|
|
||||||
|
log() {
|
||||||
|
echo "$(date '+%Y-%m-%d %H:%M:%S') $*"
|
||||||
|
}
|
||||||
|
|
||||||
|
{
|
||||||
|
log "=== docker-maintenance prune.sh start (dry_run=$DRY_RUN) ==="
|
||||||
|
|
||||||
|
log "--- before ---"
|
||||||
|
df -h / | tail -n +2
|
||||||
|
docker system df
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" -eq 1 ]]; then
|
||||||
|
log "DRY RUN: would run: docker builder prune -f"
|
||||||
|
log "DRY RUN: would run: docker container prune -f"
|
||||||
|
log "DRY RUN: would run: docker image prune -f (dangling only)"
|
||||||
|
else
|
||||||
|
log "Running: docker builder prune -f"
|
||||||
|
docker builder prune -f || true
|
||||||
|
|
||||||
|
log "Running: docker container prune -f"
|
||||||
|
docker container prune -f || true
|
||||||
|
|
||||||
|
log "Running: docker image prune -f (dangling only, no -a)"
|
||||||
|
docker image prune -f || true
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "--- after ---"
|
||||||
|
df -h / | tail -n +2
|
||||||
|
DF_JSON=$(docker system df --format '{{json .}}')
|
||||||
|
echo "$DF_JSON"
|
||||||
|
|
||||||
|
# Extract build-cache reclaimable bytes for Zabbix. `docker system df` doesn't
|
||||||
|
# give bytes directly (only human-readable strings like "14.49GB"), so use
|
||||||
|
# `docker system df -v` reclaimable percentage isn't reliable either; instead
|
||||||
|
# compute it from `docker builder du`-equivalent: sum of build cache records
|
||||||
|
# not marked Shared/in-use is nontrivial to script robustly, so fall back to
|
||||||
|
# parsing the "Build Cache" line's RECLAIMABLE column via docker system df.
|
||||||
|
RECLAIM_STR=$(docker system df | awk '/^Build Cache/ {print $NF}')
|
||||||
|
# RECLAIM_STR looks like "14.49GB" or "0B" -- convert to bytes (approx, GB/MB/KB = *1000^n, matching docker's own decimal convention)
|
||||||
|
RECLAIM_BYTES=$(python3 -c "
|
||||||
|
import re, sys
|
||||||
|
s = '$RECLAIM_STR'
|
||||||
|
m = re.match(r'([0-9.]+)\s*([KMGTP]?B)', s)
|
||||||
|
if not m:
|
||||||
|
print(0)
|
||||||
|
else:
|
||||||
|
val, unit = float(m.group(1)), m.group(2)
|
||||||
|
mult = {'B':1,'KB':1000,'MB':1000**2,'GB':1000**3,'TB':1000**4,'PB':1000**5}[unit]
|
||||||
|
print(int(val*mult))
|
||||||
|
" 2>/dev/null || echo 0)
|
||||||
|
|
||||||
|
log "Build cache reclaimable: $RECLAIM_STR (~$RECLAIM_BYTES bytes)"
|
||||||
|
|
||||||
|
if [[ -f "$ZABBIX_TOKEN_FILE" && "$DRY_RUN" -eq 0 ]]; then
|
||||||
|
ZABBIX_TOKEN=$(cat "$ZABBIX_TOKEN_FILE")
|
||||||
|
env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \
|
||||||
|
curl -s -X POST "$ZABBIX_URL" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-H "Authorization: Bearer $ZABBIX_TOKEN" \
|
||||||
|
-d "{\"jsonrpc\":\"2.0\",\"method\":\"history.push\",\"id\":1,\"params\":{\"itemid\":\"$ZABBIX_ITEM_ID\",\"value\":$RECLAIM_BYTES}}" > /dev/null \
|
||||||
|
&& log "Zabbix notified (docker.buildcache.reclaimable.bytes=$RECLAIM_BYTES)"
|
||||||
|
else
|
||||||
|
log "Skipped Zabbix push (dry-run or token file missing)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
AVAIL=$(df / | tail -1 | awk '{print $4}')
|
||||||
|
log "Root free: ${AVAIL}KB"
|
||||||
|
|
||||||
|
log "=== docker-maintenance prune.sh end ==="
|
||||||
|
} | tee -a "$LOG_FILE"
|
||||||
69
kanboard/backup.sh
Executable file
69
kanboard/backup.sh
Executable file
@@ -0,0 +1,69 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Kanboard backup — tier-0 hardening (kb#158, A2A-26, DESIGN-a2a-agents.md v2.1 §6c).
|
||||||
|
# Mirrors the vaultwarden backup.sh pattern (same repo, ~/agap_git/vaultwarden/backup.sh):
|
||||||
|
# scheduled dump -> /mnt/backups, retention of last 5, Zabbix freshness trapper.
|
||||||
|
#
|
||||||
|
# Runs every 3 days via alvis's user crontab (NOT root crontab like vaultwarden's --
|
||||||
|
# /mnt/backups/kanboard was bootstrapped chown'd to alvis specifically so this backup,
|
||||||
|
# like the rest of the kanboard tooling, needs no root/sudo at all. alvis is in the
|
||||||
|
# `docker` group so `docker exec`/`docker cp` need no privilege escalation either).
|
||||||
|
#
|
||||||
|
# DB dump method: kanboard's container has no sqlite3 CLI and no PHP `sqlite3`
|
||||||
|
# extension (only pdo_sqlite) -- checked directly (kb#158). Instead we run SQLite's
|
||||||
|
# own `VACUUM INTO` via PDO, which is SQLite's supported way to take an atomic,
|
||||||
|
# consistent online snapshot of a live database (safe against concurrent writers,
|
||||||
|
# same safety property `vaultwarden backup` gives us for that service).
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
BACKUP_DIR="/mnt/backups/kanboard"
|
||||||
|
ZABBIX_TOKEN_FILE="/home/alvis/.zabbix_token"
|
||||||
|
ZABBIX_URL="http://192.168.1.4:81/api_jsonrpc.php"
|
||||||
|
ZABBIX_ITEM_ID="70605" # kanboard.backup.ts on host AgapHost (10776)
|
||||||
|
|
||||||
|
DATE=$(date '+%Y%m%d-%H%M')
|
||||||
|
DEST="$BACKUP_DIR/$DATE"
|
||||||
|
TMP_NAME="backup_${DATE}.sqlite"
|
||||||
|
|
||||||
|
mkdir -p "$DEST"
|
||||||
|
|
||||||
|
# Online, consistent snapshot via SQLite's VACUUM INTO (PDO sqlite driver is present
|
||||||
|
# in the image; the sqlite3 CLI/extension is not, so this replaces the vaultwarden
|
||||||
|
# `docker exec vaultwarden /vaultwarden backup` equivalent for this service).
|
||||||
|
docker exec kanboard php -r '
|
||||||
|
$db = new PDO("sqlite:/var/www/app/data/db.sqlite");
|
||||||
|
$db->exec("VACUUM INTO \"/var/www/app/data/'"$TMP_NAME"'\"");
|
||||||
|
'
|
||||||
|
|
||||||
|
# Pull the snapshot out to the host, then remove the temp copy from the live data dir
|
||||||
|
# (mirrors vaultwarden's "move the file out of DATA_DIR" step).
|
||||||
|
docker cp "kanboard:/var/www/app/data/$TMP_NAME" "$DEST/db.sqlite"
|
||||||
|
docker exec kanboard rm -f "/var/www/app/data/$TMP_NAME"
|
||||||
|
|
||||||
|
# Plugins volume (PLUGIN_INSTALLER=true means plugins can be installed at runtime,
|
||||||
|
# not just baked into the image) -- back it up too so a restore doesn't silently
|
||||||
|
# drop installed plugins.
|
||||||
|
docker run --rm --user 1000:1000 -v kanboard_plugins:/plugins:ro -v "$DEST":/dest alpine \
|
||||||
|
sh -c 'cd /plugins && tar -czf /dest/plugins.tar.gz . 2>/dev/null || true'
|
||||||
|
|
||||||
|
echo "$(date): Backup complete: $DEST"
|
||||||
|
ls -la "$DEST/"
|
||||||
|
|
||||||
|
# Notify Zabbix (trapper item kanboard.backup.ts, unixtime) -- pushes a real epoch
|
||||||
|
# timestamp, unlike vaultwarden.backup.ts which (kb#158 finding) pushes a formatted
|
||||||
|
# date STRING into a numeric item and has therefore never recorded a valid value.
|
||||||
|
if [[ -f "$ZABBIX_TOKEN_FILE" ]]; then
|
||||||
|
ZABBIX_TOKEN=$(cat "$ZABBIX_TOKEN_FILE")
|
||||||
|
NOW_EPOCH=$(date '+%s')
|
||||||
|
env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \
|
||||||
|
curl -s -X POST "$ZABBIX_URL" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-H "Authorization: Bearer $ZABBIX_TOKEN" \
|
||||||
|
-d "{\"jsonrpc\":\"2.0\",\"method\":\"history.push\",\"id\":1,\"params\":{\"itemid\":\"$ZABBIX_ITEM_ID\",\"value\":$NOW_EPOCH}}" > /dev/null \
|
||||||
|
&& echo "Zabbix notified (kanboard.backup.ts=$NOW_EPOCH)."
|
||||||
|
else
|
||||||
|
echo "WARNING: $ZABBIX_TOKEN_FILE not found -- skipped Zabbix freshness push." >&2
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Rotate: keep last 5 backups
|
||||||
|
ls -1dt "$BACKUP_DIR"/[0-9]*-[0-9]* 2>/dev/null | tail -n +6 | xargs -r rm -rf
|
||||||
92
kanboard/healthcheck.sh
Executable file
92
kanboard/healthcheck.sh
Executable file
@@ -0,0 +1,92 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Kanboard service + JSON-RPC API health -- tier-0 hardening (kb#158, A2A-26,
|
||||||
|
# DESIGN-a2a-agents.md v2.1 §6c: "Zabbix monitoring of the service and API").
|
||||||
|
#
|
||||||
|
# Runs every 2 minutes via alvis's user crontab (no root needed -- alvis is in the
|
||||||
|
# `docker` group). Pushes two Zabbix trapper items on host AgapHost:
|
||||||
|
# kanboard.service.up -- 1 if the `kanboard` container is running AND its own
|
||||||
|
# Docker healthcheck reports "healthy", else 0
|
||||||
|
# kanboard.jsonrpc.up -- 1 if an authenticated JSON-RPC call round-trips
|
||||||
|
# correctly, else 0
|
||||||
|
# Both are pushed every run (unlike backup.sh, which only pushes on success) so a
|
||||||
|
# transition to "down" is reported immediately rather than waiting for nodata() --
|
||||||
|
# the triggers additionally use nodata(...,10m) as a backstop in case this script
|
||||||
|
# itself stops running.
|
||||||
|
|
||||||
|
set -uo pipefail # no -e: we want to push a "0" and continue, not abort, on failure
|
||||||
|
|
||||||
|
ZABBIX_TOKEN_FILE="/home/alvis/.zabbix_token"
|
||||||
|
KANBOARD_TOKEN_FILE="/home/alvis/.kanboard_token"
|
||||||
|
ZABBIX_URL="http://192.168.1.4:81/api_jsonrpc.php"
|
||||||
|
SERVICE_ITEM_ID="70606" # kanboard.service.up
|
||||||
|
JSONRPC_ITEM_ID="70607" # kanboard.jsonrpc.up
|
||||||
|
|
||||||
|
# kb#188: tier-0 fabric container up/down. Piggybacks on this same 2-minute cron
|
||||||
|
# slot (no new cron entry) -- container:itemid map, plain `docker inspect
|
||||||
|
# .State.Running` since most of these have no HEALTHCHECK defined (only `adolf`
|
||||||
|
# does; checking bare Running is what's available uniformly here). Pushed
|
||||||
|
# unconditionally like the two items above, with nodata(...,10m) as trigger backstop.
|
||||||
|
declare -A FABRIC_ITEMS=(
|
||||||
|
["litellm"]="70625"
|
||||||
|
["litellm-db"]="70626"
|
||||||
|
["adolf"]="70627"
|
||||||
|
["adolf-llm"]="70628"
|
||||||
|
["hindsight"]="70629"
|
||||||
|
["hindsight-llm"]="70630"
|
||||||
|
["tei-reranker"]="70631"
|
||||||
|
["ollama"]="70632"
|
||||||
|
["kanboard-mcp-kanboard-mcp-1"]="70633"
|
||||||
|
["kanboard-mcp-adolf"]="70634"
|
||||||
|
["agap-mcp-agap-mcp-1"]="70635"
|
||||||
|
["fabric-keeper"]="70636"
|
||||||
|
)
|
||||||
|
|
||||||
|
# --- 1. Container health ---
|
||||||
|
HEALTH=$(docker inspect --format '{{.State.Health.Status}}' kanboard 2>/dev/null)
|
||||||
|
if [[ "$HEALTH" == "healthy" ]]; then
|
||||||
|
SERVICE_UP=1
|
||||||
|
else
|
||||||
|
SERVICE_UP=0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# --- 2. JSON-RPC API health (authenticated round-trip, not just a TCP/HTTP check) ---
|
||||||
|
JSONRPC_UP=0
|
||||||
|
if [[ -f "$KANBOARD_TOKEN_FILE" ]]; then
|
||||||
|
KANBOARD_TOKEN=$(cat "$KANBOARD_TOKEN_FILE")
|
||||||
|
RESP=$(env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \
|
||||||
|
curl -s --max-time 10 -u "jsonrpc:$KANBOARD_TOKEN" -X POST http://127.0.0.1:4800/jsonrpc.php \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"jsonrpc":"2.0","method":"getVersion","id":1}' 2>/dev/null)
|
||||||
|
if echo "$RESP" | python3 -c "import sys,json; d=json.load(sys.stdin); sys.exit(0 if 'result' in d else 1)" 2>/dev/null; then
|
||||||
|
JSONRPC_UP=1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# --- 3. Fabric tier-0 container up/down (kb#188) ---
|
||||||
|
FABRIC_PUSH_PARAMS="{\"itemid\":\"$SERVICE_ITEM_ID\",\"value\":$SERVICE_UP},{\"itemid\":\"$JSONRPC_ITEM_ID\",\"value\":$JSONRPC_UP}"
|
||||||
|
FABRIC_SUMMARY=""
|
||||||
|
for container in "${!FABRIC_ITEMS[@]}"; do
|
||||||
|
itemid="${FABRIC_ITEMS[$container]}"
|
||||||
|
running=$(docker inspect --format '{{.State.Running}}' "$container" 2>/dev/null)
|
||||||
|
if [[ "$running" == "true" ]]; then
|
||||||
|
up=1
|
||||||
|
else
|
||||||
|
up=0
|
||||||
|
fi
|
||||||
|
FABRIC_PUSH_PARAMS="$FABRIC_PUSH_PARAMS,{\"itemid\":\"$itemid\",\"value\":$up}"
|
||||||
|
FABRIC_SUMMARY="$FABRIC_SUMMARY $container=$up"
|
||||||
|
done
|
||||||
|
|
||||||
|
# --- 4. Push everything to Zabbix in one batch, unconditionally (down is a real value, not a gap) ---
|
||||||
|
if [[ -f "$ZABBIX_TOKEN_FILE" ]]; then
|
||||||
|
ZABBIX_TOKEN=$(cat "$ZABBIX_TOKEN_FILE")
|
||||||
|
env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \
|
||||||
|
curl -s --max-time 10 -X POST "$ZABBIX_URL" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-H "Authorization: Bearer $ZABBIX_TOKEN" \
|
||||||
|
-d "{\"jsonrpc\":\"2.0\",\"method\":\"history.push\",\"id\":1,\"params\":[$FABRIC_PUSH_PARAMS]}" > /dev/null
|
||||||
|
else
|
||||||
|
echo "WARNING: $ZABBIX_TOKEN_FILE not found -- skipped Zabbix push." >&2
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "$(date '+%Y-%m-%d %H:%M:%S') service_up=$SERVICE_UP jsonrpc_up=$JSONRPC_UP$FABRIC_SUMMARY"
|
||||||
@@ -33,11 +33,13 @@ ls "$DEST/"
|
|||||||
# Notify Zabbix
|
# Notify Zabbix
|
||||||
if [[ -f /root/.zabbix_token ]]; then
|
if [[ -f /root/.zabbix_token ]]; then
|
||||||
ZABBIX_TOKEN=$(cat /root/.zabbix_token)
|
ZABBIX_TOKEN=$(cat /root/.zabbix_token)
|
||||||
curl -s -X POST http://localhost:81/api_jsonrpc.php \
|
NOW_EPOCH=$(date '+%s')
|
||||||
|
env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \
|
||||||
|
curl -s -X POST http://192.168.1.4:81/api_jsonrpc.php \
|
||||||
-H "Content-Type: application/json" \
|
-H "Content-Type: application/json" \
|
||||||
-H "Authorization: Bearer $ZABBIX_TOKEN" \
|
-H "Authorization: Bearer $ZABBIX_TOKEN" \
|
||||||
-d "{\"jsonrpc\":\"2.0\",\"method\":\"history.push\",\"id\":1,\"params\":{\"itemid\":\"70369\",\"value\":\"$(date '+%Y-%m-%d %H:%M')\"}}" > /dev/null \
|
-d "{\"jsonrpc\":\"2.0\",\"method\":\"history.push\",\"id\":1,\"params\":{\"itemid\":\"70369\",\"value\":$NOW_EPOCH}}" > /dev/null \
|
||||||
&& echo "Zabbix notified."
|
&& echo "Zabbix notified (seafile.backup.ts=$NOW_EPOCH)."
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# Rotate: keep last 5 backups
|
# Rotate: keep last 5 backups
|
||||||
|
|||||||
@@ -30,11 +30,13 @@ ls "$DEST/"
|
|||||||
# Notify Zabbix (token stored in /root/.zabbix_token)
|
# Notify Zabbix (token stored in /root/.zabbix_token)
|
||||||
if [[ -f /root/.zabbix_token ]]; then
|
if [[ -f /root/.zabbix_token ]]; then
|
||||||
ZABBIX_TOKEN=$(cat /root/.zabbix_token)
|
ZABBIX_TOKEN=$(cat /root/.zabbix_token)
|
||||||
curl -s -X POST http://localhost:81/api_jsonrpc.php \
|
NOW_EPOCH=$(date '+%s')
|
||||||
|
env -u HTTPS_PROXY -u HTTP_PROXY -u ALL_PROXY -u https_proxy -u http_proxy -u all_proxy \
|
||||||
|
curl -s -X POST http://192.168.1.4:81/api_jsonrpc.php \
|
||||||
-H "Content-Type: application/json" \
|
-H "Content-Type: application/json" \
|
||||||
-H "Authorization: Bearer $ZABBIX_TOKEN" \
|
-H "Authorization: Bearer $ZABBIX_TOKEN" \
|
||||||
-d "{\"jsonrpc\":\"2.0\",\"method\":\"history.push\",\"id\":1,\"params\":{\"itemid\":\"70368\",\"value\":\"$(date '+%Y-%m-%d %H:%M')\"}}" > /dev/null \
|
-d "{\"jsonrpc\":\"2.0\",\"method\":\"history.push\",\"id\":1,\"params\":{\"itemid\":\"70368\",\"value\":$NOW_EPOCH}}" > /dev/null \
|
||||||
&& echo "Zabbix notified."
|
&& echo "Zabbix notified (vaultwarden.backup.ts=$NOW_EPOCH)."
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# Rotate: keep last 5 backups
|
# Rotate: keep last 5 backups
|
||||||
|
|||||||
Reference in New Issue
Block a user