alvis/AgapHost
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: refresh CLAUDE.md and README, add swap exhaustion analysis

Browse Source 
CLAUDE.md and README.md catch up with where services actually run -- notably
that Zabbix and Home Assistant live on lizacer, not Agap -- and with the
current service list.

SWAP_EXHAUSTION_ANALYSIS_20260726.md records the 2026-07-26 swap exhaustion
investigation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
alvis
2026-07-30 04:43:06 +00:00
parent d37801806d
commit e5438ace79
3 changed files with 265 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
CLAUDE.md
View File

@@ -4,15 +4,25 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
## Overview
This repository manages Docker Compose configurations for the **Agap** self-hosted home server. It is not a software project — it is infrastructure-as-config for several independent services.
This repository is the **server CONFIGURATION** repo for **Agap**, the self-hosted
home server. It is not a software project — it holds `docker-compose` + host-level
config for every service running on Agap. Convention: Dockerfiles/application source
live in the service's own Gitea repo; `agap_git` keeps only the compose + config
that runs it. See [README.md](./README.md) for the full service list, the
self-developed-services table, and a known gap (matrixbot/moodtracker/marketplace-mcp/oO
currently have no compose here — see README for details).
## Services
Selected services with notes below; see [README.md](./README.md) for the complete list.
| Directory | Service | Port | Notes |
|-----------|---------|------|-------|
| `immich-app/` | Immich (photo management) | 2283 | Main compose via root `docker-compose.yml` |
| `gitea/` | Gitea (git hosting) + Postgres | 3000, 222 | Standalone compose |
| `openai/` | Open WebUI + Ollama (AI chat) | 3125 | Requires NVIDIA GPU |
| `openai/` | Adolf (OpenClaw gateway) + LiteLLM + Hindsight + Qdrant + Langfuse | see `openai/docker-compose.yml` | Requires NVIDIA GPU |
| `openwebui/` | Open WebUI (AI chat) | 3125 | Standalone compose |
| `ollama/` | Ollama (local LLM runtime) | 11436 | Requires NVIDIA GPU |
| `vaultwarden/` | Vaultwarden (password manager) | 8041 | Backup script in `vaultwarden/backup.sh` |
| `kanboard/` | Kanboard (kanban board) | 4800 | Tasks assignable to the `claude` bot user — see `kanboard/CLAUDE.md` |
@@ -89,7 +99,7 @@ When changes are made to infrastructure (services, config, setup), update the re
| Home | Index — links to all pages |
| Network | Netplan bridge setup, Caddy reverse proxy |
| Storage | LVM setup and commands |
| Home-Assistant | KVM-based Home Assistant setup |
| Home-Assistant | Home Assistant (runs on lizacer, `192.168.1.4`, Docker) |
| 3X-UI | VPN proxy panel |
| Gitea | Git hosting Docker service |
| Vaultwarden | Password manager, CLI setup, backup |
@@ -201,27 +211,29 @@ Home Assistant automations push alerts to Zabbix via `history.push` API (Zabbix
## Zabbix API
**Instance**: `http://localhost:81` (local), `https://zb.alogins.net` (external)
**Endpoint**: `http://localhost:81/api_jsonrpc.php`
**Zabbix does not run on Agap — it lives on lizacer (`192.168.1.4`).** Config for the stack moved to the `lizacer` Gitea repo (kb#81).
**Instance**: `http://192.168.1.4:81` (local), `https://zb.alogins.net` (external, Caddy on Agap → `192.168.1.4:81`)
**Endpoint**: `http://192.168.1.4:81/api_jsonrpc.php`
**Token**: Read from `$ZABBIX_TOKEN` environment variable — never hardcode it
**Auth header**: `Authorization: Bearer <token>`
### Common Requests
```bash
# Check API version
curl -s -X POST http://localhost:81/api_jsonrpc.php \
curl -s -X POST http://192.168.1.4:81/api_jsonrpc.php \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $ZABBIX_TOKEN" \
-d '{"jsonrpc":"2.0","method":"apiinfo.version","params":{},"id":1}'
# Get all hosts
curl -s -X POST http://localhost:81/api_jsonrpc.php \
curl -s -X POST http://192.168.1.4:81/api_jsonrpc.php \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $ZABBIX_TOKEN" \
-d '{"jsonrpc":"2.0","method":"host.get","params":{"output":"extend"},"id":1}'
# Get problems/issues
curl -s -X POST http://localhost:81/api_jsonrpc.php \
curl -s -X POST http://192.168.1.4:81/api_jsonrpc.php \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $ZABBIX_TOKEN" \
-d '{"jsonrpc":"2.0","method":"problem.get","params":{"output":"extend"},"id":1}'

120
README.md
View File

@@ -1,60 +1,108 @@
# Agap Home Server
# agap_git — Agap Server Configuration
Docker Compose configurations for the Agap self-hosted home server infrastructure.
`agap_git` is the **configuration repo** for Agap, the self-hosted home server. It holds:
## Services
- `docker-compose` files for services running on Agap
- host-level config: Caddy (`Caddyfile`), backup/install scripts, `.env` files
- **Immich** (`immich-app/`) — Photo management and backup (port 2283)
- **Gitea** (`gitea/`) — Self-hosted Git server with web UI (port 3000, SSH 222)
- **Open WebUI** (`openai/`) — AI chat interface with Ollama, GPU-accelerated (port 3125)
**Convention:** a Dockerfile, application source, or anything you'd `build:` from an
image belongs in the *service's own Gitea repo* — not here. `agap_git` keeps the
compose file that runs the resulting image, plus whatever config the compose needs.
Third-party services (pulling a public image) live here in full, since there's no
source to split out.
## Self-developed services
Services with source written in-house are meant to be dual: source lives in their
own Gitea repo, and the compose that runs them is described here.
| Service | Own repo | Compose in agap_git |
|---|---|---|
| matrixbot (Matrix bot, Adolf channel adapter) | [`alvis/matrixbot`](http://localhost:3000/alvis/matrixbot) | **not present** — runs from `~/matrixbot/docker-compose.yml` in its own repo |
| moodtracker | [`alvis/moodtracker`](http://localhost:3000/alvis/moodtracker) | **not present** — runs from `~/moodtracker/docker-compose.yml` |
| marketplace-mcp | [`alvis/marketplace-mcp`](http://localhost:3000/alvis/marketplace-mcp) | **not present** — runs from `~/marketplace-mcp/docker-compose.yml` |
| oO | [`alvis/oO`](http://localhost:3000/alvis/oO) | **not present** — runs from `oO/infra/docker/docker-compose.yml`; oO was already a fully separate project before this convention existed |
| mood (mood-archive) | none yet | fully vendored here (`mood/`, source + compose) — Kanboard **#209** will extract it to its own repo |
**Known gap:** the convention calls for compose to be described in `agap_git` *and*
source to live in its own repo. For matrixbot / moodtracker / marketplace-mcp / oO,
extraction (kb#78) moved the compose file along with the source into each new repo,
so `agap_git` currently has zero footprint for these four — no compose, no config.
Not fixed in this doc-only pass; flagged for a follow-up decision: either copy each
compose back into `agap_git`, or formally amend the convention to let self-developed
services keep compose in their own repo.
## Third-party services (compose lives here in full)
| Directory | Service | Port |
|---|---|---|
| `immich-app/` | Immich — photo management | 2283 |
| `gitea/` | Gitea — git hosting + Postgres | 3000, 222 |
| `openai/` | Adolf (OpenClaw gateway) + LiteLLM + Hindsight + Qdrant + Langfuse + Whisper/TTS stack | see `openai/docker-compose.yml` (adolf-llm on 8010) |
| `adolf/` | Adolf config only (`openclaw.json`, design docs) — the `adolf` compose service lives in `openai/docker-compose.yml`; the OpenClaw fork source is a separate tree at `~/adolf` | — |
| `vaultwarden/` | Password manager | 8041 |
| `kanboard/` | Kanban board, task orchestration for the `claude` bot | 4800 |
| `seafile/` | File sync, OnlyOffice, WebDAV (multiple compose files) | 8078 (server), 8077 (Caddy) |
| `matrix/` | Synapse homeserver + coturn + LiveKit (not the `matrixbot` bot — see above) | — |
| `overleaf/` | LaTeX editor (ShareLaTeX/Overleaf CE) | — |
| `linkwarden/` | Bookmark manager | 3012 |
| `vikunja/` | Task manager | 3457 |
| `radicale/` | CalDAV/CardDAV server | 5232 |
| `ntfy/` | Push notifications | 8840 |
| `syncthing/` | File sync | 8384 |
| `qbittorrent/` | Torrent client | 8085 |
| `searxng/` | Metasearch engine | 11437 |
| `ollama/` | Local LLM runtime | 11436 |
| `openwebui/` | AI chat UI | 3125 |
| `anki/` | Anki sync server | 8180 |
| `family/` | MediaWiki (family wiki) | 8099 |
| `freshrss/` | RSS reader | 8091 |
| `iperf3/` | Network throughput testing | 8095 |
| `otter/` | OtterWiki | 8083 |
| `agap-mcp/` | MCP tool server for Claude (Node/Express) — vendored source, no separate repo | 3100 |
| `personal-sensing/` | SQLite store + schema for HA/Health Connect data (no compose; library used by an ETL, see kb#207) | — |
## Removed / moved out
- `windows/` — removed (kb#82): no VM, no container, dead config.
- `zabbix/`, `haos/` — moved to [`alvis/lizacer`](http://localhost:3000/alvis/lizacer) (kb#81). Zabbix runs on the **lizacer** server (`192.168.1.4:81`), not Agap.
## Top-level files
- `Caddyfile` — reverse proxy / TLS routing for all services
- `nvidia-docker-install.sh`, `install-cuda.sh` — GPU host setup
- `users-backup.sh` — host user backup
## Quick Start
### Start Immich (main service)
Each service is standalone; from its directory:
```bash
docker compose up -d
docker compose restart
docker compose logs -f
docker compose pull
```
### Start Gitea (from gitea/ directory)
```bash
cd gitea
docker compose up -d
```
### Start Open WebUI (from openai/ directory)
```bash
cd openai
docker compose up -d
```
## Configuration
Environment variables are in the root `.env` file for Immich:
- `UPLOAD_LOCATION` — where photo originals are stored
- `THUMB_LOCATION` — thumbnail cache directory
- `ENCODED_VIDEO_LOCATION` — transcoded video cache
- `DB_DATA_LOCATION` — Postgres database directory
- `DB_PASSWORD` — Postgres password
The root `docker-compose.yml` is an alias that includes `immich-app/docker-compose.yml`.
## Storage
Media is stored on:
- `/mnt/media/upload` — Immich originals
- `/mnt/ssd1/media/` — Immich thumbnails, encoded video, and Postgres database
- `/mnt/ssd1/media/` — Immich thumbnails, encoded video, Postgres
- `/mnt/misc/gitea` — Gitea repositories and data
## GPU Support
For GPU acceleration (Open WebUI/Ollama, Immich ML):
For GPU acceleration (Ollama/Open WebUI, Immich ML):
1. Install NVIDIA Docker runtime: `sudo ./nvidia-docker-install.sh`
2. Install CUDA toolkit: `./install-cuda.sh`
1. `sudo ./nvidia-docker-install.sh` — Docker + NVIDIA Container Toolkit
2. `./install-cuda.sh` — CUDA toolkit
## Documentation
See [CLAUDE.md](./CLAUDE.md) for detailed developer instructions and Gitea wiki integration guidelines.
See [CLAUDE.md](./CLAUDE.md) for developer instructions, credentials handling, and
Gitea/Zabbix/Home Assistant API integration details.
See the [Gitea wiki](http://localhost:3000/alvis/AgapHost/wiki) for infrastructure documentation (storage, network, services setup).
See the [Gitea wiki](http://localhost:3000/alvis/AgapHost/wiki) for infrastructure
documentation (storage, network, per-service setup).

161
SWAP_EXHAUSTION_ANALYSIS_20260726.md Normal file
View File

@@ -0,0 +1,161 @@
# Swap Exhaustion Analysis — 2026-07-26
**Status:** CRITICAL — Swap 4.0Gi/4.0Gi exhausted (8.0Ki free)
**Alert Status:** Zabbix "High swap space usage" FIRING on AgapHost since 2026-07-26 04:42
**Memory Pressure:** 12Gi/15Gi RAM used (458Mi free, 3.5Gi available with cache)
## Current Measurements (2026-07-26 11:46 UTC+3)
```
RAM: 12Gi/15Gi (80% used, 458Mi free, 3.5Gi cache)
Swap: 4.0Gi/4.0Gi (100% EXHAUSTED, 8.0Ki free)
```
## Top Swap Consumers
### 1. Claude Code Processes (Host) — 265 MB swap total
These are interactive development sessions running on the host, not containers:
| PID | Process | Swap | RSS | Description |
|-----|---------|------|-----|-------------|
| 4036111 | claude 2.1.220 main | 71.7 MB | 266 MB | Active main session (opus model) |
| 4036089 | claude bg-pty-host | 62.6 MB | 42.7 MB | Background PTY host |
| 4034268 | claude bg-pty-host | 63.9 MB | 38.6 MB | Background PTY host |
| 4034247 | claude | 42.6 MB | 102 MB | Claude process |
| 4034281 | claude bg-spare | 29.8 MB | 79.7 MB | Spare background process |
**Finding:** Multiple interactive Claude Code sessions are consuming ~500 MB combined RSS and paging ~265 MB to swap due to RAM pressure.
### 2. Docker Containers (Top 4 by Memory)
| Container | Image | Memory | Swap | Status |
|-----------|-------|--------|------|--------|
| hindsight | ghcr.io/vectorize-io/hindsight | 834.3 MiB | 11.7 MB | Memory-intensive but stable |
| sharelatex | sharelatex/sharelatex:6.1.2 | 612.8 MiB | <1 MB | Large footprint |
| tei-reranker | openai-tei-reranker | 363.9 MiB | ~1 MB | Minimal swap |
| adolf (matrixbot) | adolf:local | 478.2 MiB | 73.6 MB | Modest swap usage |
### 3. Other Notable Processes
- `hindsight-api` (PID 996): 11.7 MB swap, 717 MB RSS
- `qbittorrent-nox`: 7.4 MB swap, 53.8 MB RSS
- `syncthing` (2 instances): 2.2 MB swap, 82 MB RSS combined
- `postgres`: <1 MB swap per process
## Root Cause Analysis
**Primary driver:** Multiple interactive Claude Code sessions consuming ~500 MB combined memory, with 265 MB swapped out due to low available RAM.
**Secondary pressure:** Hindsight (834 MiB) and ShareLatex (612 MiB) are large but mostly RSS; they don't cause the swap explosion directly, but contribute to overall memory pressure that forces smaller processes into swap.
**System state:** With only 458 Mi RAM free and cache being reclaimed, any process trying to allocate memory gets swapped, including the interactive Claude sessions.
## Mitigation Options (Staged)
### Stage 1: Kill Idle Claude Sessions (IMMEDIATE, ZERO RISK)
**Action:** Terminate idle/background Claude Code sessions, keep only essential active session(s).
**Impact:** Frees ~200–300 MB swap (5–7% relief), swap would drop to ~3.7 Gi.
**Risk:** None — these are human-driven interactive sessions, not persistent services.
**Commands:**
```bash
# Kill all background Claude processes except the main session
pkill -f "claude.*bg-pty-host"
pkill -f "claude.*bg-spare"
# Or selectively: kill 4034268 4036089 4034281
```
**Expected result:** Immediate swap relief; Zabbix alert will clear once usage drops below 80%.
---
### Stage 2: Evaluate ShareLatex (SHORT TERM, IF NEEDED)
**Action:** If ShareLatex is not actively used, remove it.
**Impact:** Frees ~612 MB RAM; would bring total free RAM to ~1 Gi.
**Risk:** Low if ShareLatex is idle; medium if it's required.
**Commands:**
```bash
docker compose stop sharelatex
docker compose rm sharelatex
```
---
### Stage 3: Add Memory Limits to Containers (MEDIUM TERM, REQUIRES RESTART)
**Action:** Add explicit memory limits to docker-compose.yml for hindsight and other memory-heavy services.
**Example for hindsight:**
```yaml
services:
hindsight:
image: ghcr.io/vectorize-io/hindsight:latest
deploy:
resources:
limits:
memory: 512M # or 768M depending on headroom needed
```
**Impact:** Prevents services from consuming unbounded memory; predictable resource allocation.
**Risk:** Medium — requires container restart; if limit is too low, hindsight may OOM.
**Recommendation:** Test at 768M first; monitor for OOM events. Hindsight's memory is cache-heavy (RSS 834 MiB suggests it might stabilize lower).
---
### Stage 4: Increase Swap (TEMPORARY, LOW PRIORITY)
**Action:** Add more swap (6–8 Gi).
**Impact:** Symptom relief only; doesn't address root cause (working set is larger than available RAM).
**Risk:** Low operational risk, but performance would degrade under paging load.
**Not recommended** as primary fix; use only if Stages 1–3 are insufficient.
---
## Zabbix Alert Status
**Trigger:** `Linux: High swap space usage` on AgapHost
**Value:** 1 (FIRING)
**Last change:** 2026-07-26 04:42:29 UTC+3
**Condition:** Swap usage > 80%
The alert will **automatically clear** once swap drops below the threshold (typically when used ≤ 3.2 Gi / 4.0 Gi).
## Long-Term Recommendations
1. **Docker Compose Memory Limits:** Add `deploy.resources.limits.memory` to all services in docker-compose.yml. This prevents runaway memory consumption and makes the system predictable.
2. **Monitor Claude Code Sessions:** Interactive development tools are memory-intensive. Consider:
- Limiting the number of concurrent sessions
- Restarting sessions periodically if they grow unbounded
- Monitoring growth patterns
3. **Hindsight Configuration:** Check if hindsight can reduce its cache footprint via environment variables or config (e.g., max memory, cache size limits).
4. **Periodic Audits:** Add task to quarterly review top memory consumers and adjust limits as needed.
---
## Acceptance Criteria Status
| Criterion | Status | Notes |
|-----------|--------|-------|
| Swap free >1 GB sustained | ❌ Pending mitigation | Currently 8 Ki free; Stage 1 would bring to ~700 MiB |
| Zabbix 'High swap' clears | ❌ Pending mitigation | Alert will clear once swap < 80% (~3.2 Gi) |
| Note on dominant consumer + mitigation | ✅ Complete | This document |
---
**Next step:** Execute Stage 1 (kill idle Claude sessions) for immediate relief, then monitor Zabbix alert status.