From e5438ace7984cf0b677ffea465b0da6a81e3277b Mon Sep 17 00:00:00 2001 From: alvis Date: Thu, 30 Jul 2026 04:43:06 +0000 Subject: [PATCH] docs: refresh CLAUDE.md and README, add swap exhaustion analysis CLAUDE.md and README.md catch up with where services actually run -- notably that Zabbix and Home Assistant live on lizacer, not Agap -- and with the current service list. SWAP_EXHAUSTION_ANALYSIS_20260726.md records the 2026-07-26 swap exhaustion investigation. Co-Authored-By: Claude Opus 5 --- CLAUDE.md | 28 +++-- README.md | 120 ++++++++++++++------ SWAP_EXHAUSTION_ANALYSIS_20260726.md | 161 +++++++++++++++++++++++++++ 3 files changed, 265 insertions(+), 44 deletions(-) create mode 100644 SWAP_EXHAUSTION_ANALYSIS_20260726.md diff --git a/CLAUDE.md b/CLAUDE.md index 2a82c83..34c7ffb 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,15 +4,25 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co ## Overview -This repository manages Docker Compose configurations for the **Agap** self-hosted home server. It is not a software project — it is infrastructure-as-config for several independent services. +This repository is the **server CONFIGURATION** repo for **Agap**, the self-hosted +home server. It is not a software project — it holds `docker-compose` + host-level +config for every service running on Agap. Convention: Dockerfiles/application source +live in the service's own Gitea repo; `agap_git` keeps only the compose + config +that runs it. See [README.md](./README.md) for the full service list, the +self-developed-services table, and a known gap (matrixbot/moodtracker/marketplace-mcp/oO +currently have no compose here — see README for details). ## Services +Selected services with notes below; see [README.md](./README.md) for the complete list. + | Directory | Service | Port | Notes | |-----------|---------|------|-------| | `immich-app/` | Immich (photo management) | 2283 | Main compose via root `docker-compose.yml` | | `gitea/` | Gitea (git hosting) + Postgres | 3000, 222 | Standalone compose | -| `openai/` | Open WebUI + Ollama (AI chat) | 3125 | Requires NVIDIA GPU | +| `openai/` | Adolf (OpenClaw gateway) + LiteLLM + Hindsight + Qdrant + Langfuse | see `openai/docker-compose.yml` | Requires NVIDIA GPU | +| `openwebui/` | Open WebUI (AI chat) | 3125 | Standalone compose | +| `ollama/` | Ollama (local LLM runtime) | 11436 | Requires NVIDIA GPU | | `vaultwarden/` | Vaultwarden (password manager) | 8041 | Backup script in `vaultwarden/backup.sh` | | `kanboard/` | Kanboard (kanban board) | 4800 | Tasks assignable to the `claude` bot user — see `kanboard/CLAUDE.md` | @@ -89,7 +99,7 @@ When changes are made to infrastructure (services, config, setup), update the re | Home | Index — links to all pages | | Network | Netplan bridge setup, Caddy reverse proxy | | Storage | LVM setup and commands | -| Home-Assistant | KVM-based Home Assistant setup | +| Home-Assistant | Home Assistant (runs on lizacer, `192.168.1.4`, Docker) | | 3X-UI | VPN proxy panel | | Gitea | Git hosting Docker service | | Vaultwarden | Password manager, CLI setup, backup | @@ -201,27 +211,29 @@ Home Assistant automations push alerts to Zabbix via `history.push` API (Zabbix ## Zabbix API -**Instance**: `http://localhost:81` (local), `https://zb.alogins.net` (external) -**Endpoint**: `http://localhost:81/api_jsonrpc.php` +**Zabbix does not run on Agap — it lives on lizacer (`192.168.1.4`).** Config for the stack moved to the `lizacer` Gitea repo (kb#81). + +**Instance**: `http://192.168.1.4:81` (local), `https://zb.alogins.net` (external, Caddy on Agap → `192.168.1.4:81`) +**Endpoint**: `http://192.168.1.4:81/api_jsonrpc.php` **Token**: Read from `$ZABBIX_TOKEN` environment variable — never hardcode it **Auth header**: `Authorization: Bearer ` ### Common Requests ```bash # Check API version -curl -s -X POST http://localhost:81/api_jsonrpc.php \ +curl -s -X POST http://192.168.1.4:81/api_jsonrpc.php \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $ZABBIX_TOKEN" \ -d '{"jsonrpc":"2.0","method":"apiinfo.version","params":{},"id":1}' # Get all hosts -curl -s -X POST http://localhost:81/api_jsonrpc.php \ +curl -s -X POST http://192.168.1.4:81/api_jsonrpc.php \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $ZABBIX_TOKEN" \ -d '{"jsonrpc":"2.0","method":"host.get","params":{"output":"extend"},"id":1}' # Get problems/issues -curl -s -X POST http://localhost:81/api_jsonrpc.php \ +curl -s -X POST http://192.168.1.4:81/api_jsonrpc.php \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $ZABBIX_TOKEN" \ -d '{"jsonrpc":"2.0","method":"problem.get","params":{"output":"extend"},"id":1}' diff --git a/README.md b/README.md index d2703ee..3e291fa 100644 --- a/README.md +++ b/README.md @@ -1,60 +1,108 @@ -# Agap Home Server +# agap_git — Agap Server Configuration -Docker Compose configurations for the Agap self-hosted home server infrastructure. +`agap_git` is the **configuration repo** for Agap, the self-hosted home server. It holds: -## Services +- `docker-compose` files for services running on Agap +- host-level config: Caddy (`Caddyfile`), backup/install scripts, `.env` files -- **Immich** (`immich-app/`) — Photo management and backup (port 2283) -- **Gitea** (`gitea/`) — Self-hosted Git server with web UI (port 3000, SSH 222) -- **Open WebUI** (`openai/`) — AI chat interface with Ollama, GPU-accelerated (port 3125) +**Convention:** a Dockerfile, application source, or anything you'd `build:` from an +image belongs in the *service's own Gitea repo* — not here. `agap_git` keeps the +compose file that runs the resulting image, plus whatever config the compose needs. +Third-party services (pulling a public image) live here in full, since there's no +source to split out. + +## Self-developed services + +Services with source written in-house are meant to be dual: source lives in their +own Gitea repo, and the compose that runs them is described here. + +| Service | Own repo | Compose in agap_git | +|---|---|---| +| matrixbot (Matrix bot, Adolf channel adapter) | [`alvis/matrixbot`](http://localhost:3000/alvis/matrixbot) | **not present** — runs from `~/matrixbot/docker-compose.yml` in its own repo | +| moodtracker | [`alvis/moodtracker`](http://localhost:3000/alvis/moodtracker) | **not present** — runs from `~/moodtracker/docker-compose.yml` | +| marketplace-mcp | [`alvis/marketplace-mcp`](http://localhost:3000/alvis/marketplace-mcp) | **not present** — runs from `~/marketplace-mcp/docker-compose.yml` | +| oO | [`alvis/oO`](http://localhost:3000/alvis/oO) | **not present** — runs from `oO/infra/docker/docker-compose.yml`; oO was already a fully separate project before this convention existed | +| mood (mood-archive) | none yet | fully vendored here (`mood/`, source + compose) — Kanboard **#209** will extract it to its own repo | + +**Known gap:** the convention calls for compose to be described in `agap_git` *and* +source to live in its own repo. For matrixbot / moodtracker / marketplace-mcp / oO, +extraction (kb#78) moved the compose file along with the source into each new repo, +so `agap_git` currently has zero footprint for these four — no compose, no config. +Not fixed in this doc-only pass; flagged for a follow-up decision: either copy each +compose back into `agap_git`, or formally amend the convention to let self-developed +services keep compose in their own repo. + +## Third-party services (compose lives here in full) + +| Directory | Service | Port | +|---|---|---| +| `immich-app/` | Immich — photo management | 2283 | +| `gitea/` | Gitea — git hosting + Postgres | 3000, 222 | +| `openai/` | Adolf (OpenClaw gateway) + LiteLLM + Hindsight + Qdrant + Langfuse + Whisper/TTS stack | see `openai/docker-compose.yml` (adolf-llm on 8010) | +| `adolf/` | Adolf config only (`openclaw.json`, design docs) — the `adolf` compose service lives in `openai/docker-compose.yml`; the OpenClaw fork source is a separate tree at `~/adolf` | — | +| `vaultwarden/` | Password manager | 8041 | +| `kanboard/` | Kanban board, task orchestration for the `claude` bot | 4800 | +| `seafile/` | File sync, OnlyOffice, WebDAV (multiple compose files) | 8078 (server), 8077 (Caddy) | +| `matrix/` | Synapse homeserver + coturn + LiveKit (not the `matrixbot` bot — see above) | — | +| `overleaf/` | LaTeX editor (ShareLaTeX/Overleaf CE) | — | +| `linkwarden/` | Bookmark manager | 3012 | +| `vikunja/` | Task manager | 3457 | +| `radicale/` | CalDAV/CardDAV server | 5232 | +| `ntfy/` | Push notifications | 8840 | +| `syncthing/` | File sync | 8384 | +| `qbittorrent/` | Torrent client | 8085 | +| `searxng/` | Metasearch engine | 11437 | +| `ollama/` | Local LLM runtime | 11436 | +| `openwebui/` | AI chat UI | 3125 | +| `anki/` | Anki sync server | 8180 | +| `family/` | MediaWiki (family wiki) | 8099 | +| `freshrss/` | RSS reader | 8091 | +| `iperf3/` | Network throughput testing | 8095 | +| `otter/` | OtterWiki | 8083 | +| `agap-mcp/` | MCP tool server for Claude (Node/Express) — vendored source, no separate repo | 3100 | +| `personal-sensing/` | SQLite store + schema for HA/Health Connect data (no compose; library used by an ETL, see kb#207) | — | + +## Removed / moved out + +- `windows/` — removed (kb#82): no VM, no container, dead config. +- `zabbix/`, `haos/` — moved to [`alvis/lizacer`](http://localhost:3000/alvis/lizacer) (kb#81). Zabbix runs on the **lizacer** server (`192.168.1.4:81`), not Agap. + +## Top-level files + +- `Caddyfile` — reverse proxy / TLS routing for all services +- `nvidia-docker-install.sh`, `install-cuda.sh` — GPU host setup +- `users-backup.sh` — host user backup ## Quick Start -### Start Immich (main service) +Each service is standalone; from its directory: ```bash docker compose up -d +docker compose restart +docker compose logs -f +docker compose pull ``` -### Start Gitea (from gitea/ directory) - -```bash -cd gitea -docker compose up -d -``` - -### Start Open WebUI (from openai/ directory) - -```bash -cd openai -docker compose up -d -``` - -## Configuration - -Environment variables are in the root `.env` file for Immich: -- `UPLOAD_LOCATION` — where photo originals are stored -- `THUMB_LOCATION` — thumbnail cache directory -- `ENCODED_VIDEO_LOCATION` — transcoded video cache -- `DB_DATA_LOCATION` — Postgres database directory -- `DB_PASSWORD` — Postgres password +The root `docker-compose.yml` is an alias that includes `immich-app/docker-compose.yml`. ## Storage -Media is stored on: - `/mnt/media/upload` — Immich originals -- `/mnt/ssd1/media/` — Immich thumbnails, encoded video, and Postgres database +- `/mnt/ssd1/media/` — Immich thumbnails, encoded video, Postgres - `/mnt/misc/gitea` — Gitea repositories and data ## GPU Support -For GPU acceleration (Open WebUI/Ollama, Immich ML): +For GPU acceleration (Ollama/Open WebUI, Immich ML): -1. Install NVIDIA Docker runtime: `sudo ./nvidia-docker-install.sh` -2. Install CUDA toolkit: `./install-cuda.sh` +1. `sudo ./nvidia-docker-install.sh` — Docker + NVIDIA Container Toolkit +2. `./install-cuda.sh` — CUDA toolkit ## Documentation -See [CLAUDE.md](./CLAUDE.md) for detailed developer instructions and Gitea wiki integration guidelines. +See [CLAUDE.md](./CLAUDE.md) for developer instructions, credentials handling, and +Gitea/Zabbix/Home Assistant API integration details. -See the [Gitea wiki](http://localhost:3000/alvis/AgapHost/wiki) for infrastructure documentation (storage, network, services setup). +See the [Gitea wiki](http://localhost:3000/alvis/AgapHost/wiki) for infrastructure +documentation (storage, network, per-service setup). diff --git a/SWAP_EXHAUSTION_ANALYSIS_20260726.md b/SWAP_EXHAUSTION_ANALYSIS_20260726.md new file mode 100644 index 0000000..f65818e --- /dev/null +++ b/SWAP_EXHAUSTION_ANALYSIS_20260726.md @@ -0,0 +1,161 @@ +# Swap Exhaustion Analysis — 2026-07-26 + +**Status:** CRITICAL — Swap 4.0Gi/4.0Gi exhausted (8.0Ki free) +**Alert Status:** Zabbix "High swap space usage" FIRING on AgapHost since 2026-07-26 04:42 +**Memory Pressure:** 12Gi/15Gi RAM used (458Mi free, 3.5Gi available with cache) + +## Current Measurements (2026-07-26 11:46 UTC+3) + +``` +RAM: 12Gi/15Gi (80% used, 458Mi free, 3.5Gi cache) +Swap: 4.0Gi/4.0Gi (100% EXHAUSTED, 8.0Ki free) +``` + +## Top Swap Consumers + +### 1. Claude Code Processes (Host) — 265 MB swap total + +These are interactive development sessions running on the host, not containers: + +| PID | Process | Swap | RSS | Description | +|-----|---------|------|-----|-------------| +| 4036111 | claude 2.1.220 main | 71.7 MB | 266 MB | Active main session (opus model) | +| 4036089 | claude bg-pty-host | 62.6 MB | 42.7 MB | Background PTY host | +| 4034268 | claude bg-pty-host | 63.9 MB | 38.6 MB | Background PTY host | +| 4034247 | claude | 42.6 MB | 102 MB | Claude process | +| 4034281 | claude bg-spare | 29.8 MB | 79.7 MB | Spare background process | + +**Finding:** Multiple interactive Claude Code sessions are consuming ~500 MB combined RSS and paging ~265 MB to swap due to RAM pressure. + +### 2. Docker Containers (Top 4 by Memory) + +| Container | Image | Memory | Swap | Status | +|-----------|-------|--------|------|--------| +| hindsight | ghcr.io/vectorize-io/hindsight | 834.3 MiB | 11.7 MB | Memory-intensive but stable | +| sharelatex | sharelatex/sharelatex:6.1.2 | 612.8 MiB | <1 MB | Large footprint | +| tei-reranker | openai-tei-reranker | 363.9 MiB | ~1 MB | Minimal swap | +| adolf (matrixbot) | adolf:local | 478.2 MiB | 73.6 MB | Modest swap usage | + +### 3. Other Notable Processes + +- `hindsight-api` (PID 996): 11.7 MB swap, 717 MB RSS +- `qbittorrent-nox`: 7.4 MB swap, 53.8 MB RSS +- `syncthing` (2 instances): 2.2 MB swap, 82 MB RSS combined +- `postgres`: <1 MB swap per process + +## Root Cause Analysis + +**Primary driver:** Multiple interactive Claude Code sessions consuming ~500 MB combined memory, with 265 MB swapped out due to low available RAM. + +**Secondary pressure:** Hindsight (834 MiB) and ShareLatex (612 MiB) are large but mostly RSS; they don't cause the swap explosion directly, but contribute to overall memory pressure that forces smaller processes into swap. + +**System state:** With only 458 Mi RAM free and cache being reclaimed, any process trying to allocate memory gets swapped, including the interactive Claude sessions. + +## Mitigation Options (Staged) + +### Stage 1: Kill Idle Claude Sessions (IMMEDIATE, ZERO RISK) + +**Action:** Terminate idle/background Claude Code sessions, keep only essential active session(s). + +**Impact:** Frees ~200–300 MB swap (5–7% relief), swap would drop to ~3.7 Gi. + +**Risk:** None — these are human-driven interactive sessions, not persistent services. + +**Commands:** +```bash +# Kill all background Claude processes except the main session +pkill -f "claude.*bg-pty-host" +pkill -f "claude.*bg-spare" +# Or selectively: kill 4034268 4036089 4034281 +``` + +**Expected result:** Immediate swap relief; Zabbix alert will clear once usage drops below 80%. + +--- + +### Stage 2: Evaluate ShareLatex (SHORT TERM, IF NEEDED) + +**Action:** If ShareLatex is not actively used, remove it. + +**Impact:** Frees ~612 MB RAM; would bring total free RAM to ~1 Gi. + +**Risk:** Low if ShareLatex is idle; medium if it's required. + +**Commands:** +```bash +docker compose stop sharelatex +docker compose rm sharelatex +``` + +--- + +### Stage 3: Add Memory Limits to Containers (MEDIUM TERM, REQUIRES RESTART) + +**Action:** Add explicit memory limits to docker-compose.yml for hindsight and other memory-heavy services. + +**Example for hindsight:** +```yaml +services: + hindsight: + image: ghcr.io/vectorize-io/hindsight:latest + deploy: + resources: + limits: + memory: 512M # or 768M depending on headroom needed +``` + +**Impact:** Prevents services from consuming unbounded memory; predictable resource allocation. + +**Risk:** Medium — requires container restart; if limit is too low, hindsight may OOM. + +**Recommendation:** Test at 768M first; monitor for OOM events. Hindsight's memory is cache-heavy (RSS 834 MiB suggests it might stabilize lower). + +--- + +### Stage 4: Increase Swap (TEMPORARY, LOW PRIORITY) + +**Action:** Add more swap (6–8 Gi). + +**Impact:** Symptom relief only; doesn't address root cause (working set is larger than available RAM). + +**Risk:** Low operational risk, but performance would degrade under paging load. + +**Not recommended** as primary fix; use only if Stages 1–3 are insufficient. + +--- + +## Zabbix Alert Status + +**Trigger:** `Linux: High swap space usage` on AgapHost +**Value:** 1 (FIRING) +**Last change:** 2026-07-26 04:42:29 UTC+3 +**Condition:** Swap usage > 80% + +The alert will **automatically clear** once swap drops below the threshold (typically when used ≤ 3.2 Gi / 4.0 Gi). + +## Long-Term Recommendations + +1. **Docker Compose Memory Limits:** Add `deploy.resources.limits.memory` to all services in docker-compose.yml. This prevents runaway memory consumption and makes the system predictable. + +2. **Monitor Claude Code Sessions:** Interactive development tools are memory-intensive. Consider: + - Limiting the number of concurrent sessions + - Restarting sessions periodically if they grow unbounded + - Monitoring growth patterns + +3. **Hindsight Configuration:** Check if hindsight can reduce its cache footprint via environment variables or config (e.g., max memory, cache size limits). + +4. **Periodic Audits:** Add task to quarterly review top memory consumers and adjust limits as needed. + +--- + +## Acceptance Criteria Status + +| Criterion | Status | Notes | +|-----------|--------|-------| +| Swap free >1 GB sustained | ❌ Pending mitigation | Currently 8 Ki free; Stage 1 would bring to ~700 MiB | +| Zabbix 'High swap' clears | ❌ Pending mitigation | Alert will clear once swap < 80% (~3.2 Gi) | +| Note on dominant consumer + mitigation | ✅ Complete | This document | + +--- + +**Next step:** Execute Stage 1 (kill idle Claude sessions) for immediate relief, then monitor Zabbix alert status.