alvis
e5438ace79
CLAUDE.md and README.md catch up with where services actually run -- notably that Zabbix and Home Assistant live on lizacer, not Agap -- and with the current service list. SWAP_EXHAUSTION_ANALYSIS_20260726.md records the 2026-07-26 swap exhaustion investigation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
162 lines
6.0 KiB
Markdown
162 lines
6.0 KiB
Markdown
# Swap Exhaustion Analysis — 2026-07-26
|
||
|
||
**Status:** CRITICAL — Swap 4.0Gi/4.0Gi exhausted (8.0Ki free)
|
||
**Alert Status:** Zabbix "High swap space usage" FIRING on AgapHost since 2026-07-26 04:42
|
||
**Memory Pressure:** 12Gi/15Gi RAM used (458Mi free, 3.5Gi available with cache)
|
||
|
||
## Current Measurements (2026-07-26 11:46 UTC+3)
|
||
|
||
```
|
||
RAM: 12Gi/15Gi (80% used, 458Mi free, 3.5Gi cache)
|
||
Swap: 4.0Gi/4.0Gi (100% EXHAUSTED, 8.0Ki free)
|
||
```
|
||
|
||
## Top Swap Consumers
|
||
|
||
### 1. Claude Code Processes (Host) — 265 MB swap total
|
||
|
||
These are interactive development sessions running on the host, not containers:
|
||
|
||
| PID | Process | Swap | RSS | Description |
|
||
|-----|---------|------|-----|-------------|
|
||
| 4036111 | claude 2.1.220 main | 71.7 MB | 266 MB | Active main session (opus model) |
|
||
| 4036089 | claude bg-pty-host | 62.6 MB | 42.7 MB | Background PTY host |
|
||
| 4034268 | claude bg-pty-host | 63.9 MB | 38.6 MB | Background PTY host |
|
||
| 4034247 | claude | 42.6 MB | 102 MB | Claude process |
|
||
| 4034281 | claude bg-spare | 29.8 MB | 79.7 MB | Spare background process |
|
||
|
||
**Finding:** Multiple interactive Claude Code sessions are consuming ~500 MB combined RSS and paging ~265 MB to swap due to RAM pressure.
|
||
|
||
### 2. Docker Containers (Top 4 by Memory)
|
||
|
||
| Container | Image | Memory | Swap | Status |
|
||
|-----------|-------|--------|------|--------|
|
||
| hindsight | ghcr.io/vectorize-io/hindsight | 834.3 MiB | 11.7 MB | Memory-intensive but stable |
|
||
| sharelatex | sharelatex/sharelatex:6.1.2 | 612.8 MiB | <1 MB | Large footprint |
|
||
| tei-reranker | openai-tei-reranker | 363.9 MiB | ~1 MB | Minimal swap |
|
||
| adolf (matrixbot) | adolf:local | 478.2 MiB | 73.6 MB | Modest swap usage |
|
||
|
||
### 3. Other Notable Processes
|
||
|
||
- `hindsight-api` (PID 996): 11.7 MB swap, 717 MB RSS
|
||
- `qbittorrent-nox`: 7.4 MB swap, 53.8 MB RSS
|
||
- `syncthing` (2 instances): 2.2 MB swap, 82 MB RSS combined
|
||
- `postgres`: <1 MB swap per process
|
||
|
||
## Root Cause Analysis
|
||
|
||
**Primary driver:** Multiple interactive Claude Code sessions consuming ~500 MB combined memory, with 265 MB swapped out due to low available RAM.
|
||
|
||
**Secondary pressure:** Hindsight (834 MiB) and ShareLatex (612 MiB) are large but mostly RSS; they don't cause the swap explosion directly, but contribute to overall memory pressure that forces smaller processes into swap.
|
||
|
||
**System state:** With only 458 Mi RAM free and cache being reclaimed, any process trying to allocate memory gets swapped, including the interactive Claude sessions.
|
||
|
||
## Mitigation Options (Staged)
|
||
|
||
### Stage 1: Kill Idle Claude Sessions (IMMEDIATE, ZERO RISK)
|
||
|
||
**Action:** Terminate idle/background Claude Code sessions, keep only essential active session(s).
|
||
|
||
**Impact:** Frees ~200–300 MB swap (5–7% relief), swap would drop to ~3.7 Gi.
|
||
|
||
**Risk:** None — these are human-driven interactive sessions, not persistent services.
|
||
|
||
**Commands:**
|
||
```bash
|
||
# Kill all background Claude processes except the main session
|
||
pkill -f "claude.*bg-pty-host"
|
||
pkill -f "claude.*bg-spare"
|
||
# Or selectively: kill 4034268 4036089 4034281
|
||
```
|
||
|
||
**Expected result:** Immediate swap relief; Zabbix alert will clear once usage drops below 80%.
|
||
|
||
---
|
||
|
||
### Stage 2: Evaluate ShareLatex (SHORT TERM, IF NEEDED)
|
||
|
||
**Action:** If ShareLatex is not actively used, remove it.
|
||
|
||
**Impact:** Frees ~612 MB RAM; would bring total free RAM to ~1 Gi.
|
||
|
||
**Risk:** Low if ShareLatex is idle; medium if it's required.
|
||
|
||
**Commands:**
|
||
```bash
|
||
docker compose stop sharelatex
|
||
docker compose rm sharelatex
|
||
```
|
||
|
||
---
|
||
|
||
### Stage 3: Add Memory Limits to Containers (MEDIUM TERM, REQUIRES RESTART)
|
||
|
||
**Action:** Add explicit memory limits to docker-compose.yml for hindsight and other memory-heavy services.
|
||
|
||
**Example for hindsight:**
|
||
```yaml
|
||
services:
|
||
hindsight:
|
||
image: ghcr.io/vectorize-io/hindsight:latest
|
||
deploy:
|
||
resources:
|
||
limits:
|
||
memory: 512M # or 768M depending on headroom needed
|
||
```
|
||
|
||
**Impact:** Prevents services from consuming unbounded memory; predictable resource allocation.
|
||
|
||
**Risk:** Medium — requires container restart; if limit is too low, hindsight may OOM.
|
||
|
||
**Recommendation:** Test at 768M first; monitor for OOM events. Hindsight's memory is cache-heavy (RSS 834 MiB suggests it might stabilize lower).
|
||
|
||
---
|
||
|
||
### Stage 4: Increase Swap (TEMPORARY, LOW PRIORITY)
|
||
|
||
**Action:** Add more swap (6–8 Gi).
|
||
|
||
**Impact:** Symptom relief only; doesn't address root cause (working set is larger than available RAM).
|
||
|
||
**Risk:** Low operational risk, but performance would degrade under paging load.
|
||
|
||
**Not recommended** as primary fix; use only if Stages 1–3 are insufficient.
|
||
|
||
---
|
||
|
||
## Zabbix Alert Status
|
||
|
||
**Trigger:** `Linux: High swap space usage` on AgapHost
|
||
**Value:** 1 (FIRING)
|
||
**Last change:** 2026-07-26 04:42:29 UTC+3
|
||
**Condition:** Swap usage > 80%
|
||
|
||
The alert will **automatically clear** once swap drops below the threshold (typically when used ≤ 3.2 Gi / 4.0 Gi).
|
||
|
||
## Long-Term Recommendations
|
||
|
||
1. **Docker Compose Memory Limits:** Add `deploy.resources.limits.memory` to all services in docker-compose.yml. This prevents runaway memory consumption and makes the system predictable.
|
||
|
||
2. **Monitor Claude Code Sessions:** Interactive development tools are memory-intensive. Consider:
|
||
- Limiting the number of concurrent sessions
|
||
- Restarting sessions periodically if they grow unbounded
|
||
- Monitoring growth patterns
|
||
|
||
3. **Hindsight Configuration:** Check if hindsight can reduce its cache footprint via environment variables or config (e.g., max memory, cache size limits).
|
||
|
||
4. **Periodic Audits:** Add task to quarterly review top memory consumers and adjust limits as needed.
|
||
|
||
---
|
||
|
||
## Acceptance Criteria Status
|
||
|
||
| Criterion | Status | Notes |
|
||
|-----------|--------|-------|
|
||
| Swap free >1 GB sustained | ❌ Pending mitigation | Currently 8 Ki free; Stage 1 would bring to ~700 MiB |
|
||
| Zabbix 'High swap' clears | ❌ Pending mitigation | Alert will clear once swap < 80% (~3.2 Gi) |
|
||
| Note on dominant consumer + mitigation | ✅ Complete | This document |
|
||
|
||
---
|
||
|
||
**Next step:** Execute Stage 1 (kill idle Claude sessions) for immediate relief, then monitor Zabbix alert status.
|