AgapHost/SWAP_EXHAUSTION_ANALYSIS_20260726.md

Swap Exhaustion Analysis — 2026-07-26

Status: CRITICAL — Swap 4.0Gi/4.0Gi exhausted (8.0Ki free)
Alert Status: Zabbix "High swap space usage" FIRING on AgapHost since 2026-07-26 04:42
Memory Pressure: 12Gi/15Gi RAM used (458Mi free, 3.5Gi available with cache)

Current Measurements (2026-07-26 11:46 UTC+3)

RAM:  12Gi/15Gi (80% used, 458Mi free, 3.5Gi cache)
Swap: 4.0Gi/4.0Gi (100% EXHAUSTED, 8.0Ki free)

Top Swap Consumers

1. Claude Code Processes (Host) — 265 MB swap total

These are interactive development sessions running on the host, not containers:

PID	Process	Swap	RSS	Description
4036111	claude 2.1.220 main	71.7 MB	266 MB	Active main session (opus model)
4036089	claude bg-pty-host	62.6 MB	42.7 MB	Background PTY host
4034268	claude bg-pty-host	63.9 MB	38.6 MB	Background PTY host
4034247	claude	42.6 MB	102 MB	Claude process
4034281	claude bg-spare	29.8 MB	79.7 MB	Spare background process

Finding: Multiple interactive Claude Code sessions are consuming ~500 MB combined RSS and paging ~265 MB to swap due to RAM pressure.

2. Docker Containers (Top 4 by Memory)

Container	Image	Memory	Swap	Status
hindsight	ghcr.io/vectorize-io/hindsight	834.3 MiB	11.7 MB	Memory-intensive but stable
sharelatex	sharelatex/sharelatex:6.1.2	612.8 MiB	<1 MB	Large footprint
tei-reranker	openai-tei-reranker	363.9 MiB	~1 MB	Minimal swap
adolf (matrixbot)	adolf:local	478.2 MiB	73.6 MB	Modest swap usage

3. Other Notable Processes

• hindsight-api (PID 996): 11.7 MB swap, 717 MB RSS
• qbittorrent-nox: 7.4 MB swap, 53.8 MB RSS
• syncthing (2 instances): 2.2 MB swap, 82 MB RSS combined
• postgres: <1 MB swap per process

Root Cause Analysis

Primary driver: Multiple interactive Claude Code sessions consuming ~500 MB combined memory, with 265 MB swapped out due to low available RAM.

Secondary pressure: Hindsight (834 MiB) and ShareLatex (612 MiB) are large but mostly RSS; they don't cause the swap explosion directly, but contribute to overall memory pressure that forces smaller processes into swap.

System state: With only 458 Mi RAM free and cache being reclaimed, any process trying to allocate memory gets swapped, including the interactive Claude sessions.

Mitigation Options (Staged)

Stage 1: Kill Idle Claude Sessions (IMMEDIATE, ZERO RISK)

Action: Terminate idle/background Claude Code sessions, keep only essential active session(s).

Impact: Frees ~200–300 MB swap (5–7% relief), swap would drop to ~3.7 Gi.

Risk: None — these are human-driven interactive sessions, not persistent services.

Commands:

# Kill all background Claude processes except the main session
pkill -f "claude.*bg-pty-host"
pkill -f "claude.*bg-spare"
# Or selectively: kill 4034268 4036089 4034281

Expected result: Immediate swap relief; Zabbix alert will clear once usage drops below 80%.

---

Stage 2: Evaluate ShareLatex (SHORT TERM, IF NEEDED)

Action: If ShareLatex is not actively used, remove it.

Impact: Frees ~612 MB RAM; would bring total free RAM to ~1 Gi.

Risk: Low if ShareLatex is idle; medium if it's required.

Commands:

docker compose stop sharelatex
docker compose rm sharelatex

---

Stage 3: Add Memory Limits to Containers (MEDIUM TERM, REQUIRES RESTART)

Action: Add explicit memory limits to docker-compose.yml for hindsight and other memory-heavy services.

Example for hindsight:

services:
  hindsight:
    image: ghcr.io/vectorize-io/hindsight:latest
    deploy:
      resources:
        limits:
          memory: 512M  # or 768M depending on headroom needed

Impact: Prevents services from consuming unbounded memory; predictable resource allocation.

Risk: Medium — requires container restart; if limit is too low, hindsight may OOM.

Recommendation: Test at 768M first; monitor for OOM events. Hindsight's memory is cache-heavy (RSS 834 MiB suggests it might stabilize lower).

---

Stage 4: Increase Swap (TEMPORARY, LOW PRIORITY)

Action: Add more swap (6–8 Gi).

Impact: Symptom relief only; doesn't address root cause (working set is larger than available RAM).

Risk: Low operational risk, but performance would degrade under paging load.

Not recommended as primary fix; use only if Stages 1–3 are insufficient.

---

Zabbix Alert Status

Trigger: Linux: High swap space usage on AgapHost
Value: 1 (FIRING)
Last change: 2026-07-26 04:42:29 UTC+3
Condition: Swap usage > 80%

The alert will automatically clear once swap drops below the threshold (typically when used ≤ 3.2 Gi / 4.0 Gi).

Long-Term Recommendations

1. Docker Compose Memory Limits: Add deploy.resources.limits.memory to all services in docker-compose.yml. This prevents runaway memory consumption and makes the system predictable.

2. Monitor Claude Code Sessions: Interactive development tools are memory-intensive. Consider:

   • Limiting the number of concurrent sessions
   • Restarting sessions periodically if they grow unbounded
   • Monitoring growth patterns

3. Hindsight Configuration: Check if hindsight can reduce its cache footprint via environment variables or config (e.g., max memory, cache size limits).

4. Periodic Audits: Add task to quarterly review top memory consumers and adjust limits as needed.

---

Acceptance Criteria Status

Criterion	Status	Notes
Swap free >1 GB sustained	❌ Pending mitigation	Currently 8 Ki free; Stage 1 would bring to ~700 MiB
Zabbix 'High swap' clears	❌ Pending mitigation	Alert will clear once swap < 80% (~3.2 Gi)
Note on dominant consumer + mitigation	✅ Complete	This document

---

Next step: Execute Stage 1 (kill idle Claude sessions) for immediate relief, then monitor Zabbix alert status.