# Swap Exhaustion Analysis — 2026-07-26

**Status:** CRITICAL — Swap 4.0Gi/4.0Gi exhausted (8.0Ki free)  
**Alert Status:** Zabbix "High swap space usage" FIRING on AgapHost since 2026-07-26 04:42  
**Memory Pressure:** 12Gi/15Gi RAM used (458Mi free, 3.5Gi available with cache)

## Current Measurements (2026-07-26 11:46 UTC+3)

```
RAM:  12Gi/15Gi (80% used, 458Mi free, 3.5Gi cache)
Swap: 4.0Gi/4.0Gi (100% EXHAUSTED, 8.0Ki free)
```

## Top Swap Consumers

### 1. Claude Code Processes (Host) — 265 MB swap total

These are interactive development sessions running on the host, not containers:

| PID | Process | Swap | RSS | Description |
|-----|---------|------|-----|-------------|
| 4036111 | claude 2.1.220 main | 71.7 MB | 266 MB | Active main session (opus model) |
| 4036089 | claude bg-pty-host | 62.6 MB | 42.7 MB | Background PTY host |
| 4034268 | claude bg-pty-host | 63.9 MB | 38.6 MB | Background PTY host |
| 4034247 | claude | 42.6 MB | 102 MB | Claude process |
| 4034281 | claude bg-spare | 29.8 MB | 79.7 MB | Spare background process |

**Finding:** Multiple interactive Claude Code sessions are consuming ~500 MB combined RSS and paging ~265 MB to swap due to RAM pressure.

### 2. Docker Containers (Top 4 by Memory)

| Container | Image | Memory | Swap | Status |
|-----------|-------|--------|------|--------|
| hindsight | ghcr.io/vectorize-io/hindsight | 834.3 MiB | 11.7 MB | Memory-intensive but stable |
| sharelatex | sharelatex/sharelatex:6.1.2 | 612.8 MiB | <1 MB | Large footprint |
| tei-reranker | openai-tei-reranker | 363.9 MiB | ~1 MB | Minimal swap |
| adolf (matrixbot) | adolf:local | 478.2 MiB | 73.6 MB | Modest swap usage |

### 3. Other Notable Processes

- `hindsight-api` (PID 996): 11.7 MB swap, 717 MB RSS
- `qbittorrent-nox`: 7.4 MB swap, 53.8 MB RSS
- `syncthing` (2 instances): 2.2 MB swap, 82 MB RSS combined
- `postgres`: <1 MB swap per process

## Root Cause Analysis

**Primary driver:** Multiple interactive Claude Code sessions consuming ~500 MB combined memory, with 265 MB swapped out due to low available RAM.

**Secondary pressure:** Hindsight (834 MiB) and ShareLatex (612 MiB) are large but mostly RSS; they don't cause the swap explosion directly, but contribute to overall memory pressure that forces smaller processes into swap.

**System state:** With only 458 Mi RAM free and cache being reclaimed, any process trying to allocate memory gets swapped, including the interactive Claude sessions.

## Mitigation Options (Staged)

### Stage 1: Kill Idle Claude Sessions (IMMEDIATE, ZERO RISK)

**Action:** Terminate idle/background Claude Code sessions, keep only essential active session(s).

**Impact:** Frees ~200–300 MB swap (5–7% relief), swap would drop to ~3.7 Gi.

**Risk:** None — these are human-driven interactive sessions, not persistent services.

**Commands:**
```bash
# Kill all background Claude processes except the main session
pkill -f "claude.*bg-pty-host"
pkill -f "claude.*bg-spare"
# Or selectively: kill 4034268 4036089 4034281
```

**Expected result:** Immediate swap relief; Zabbix alert will clear once usage drops below 80%.

---

### Stage 2: Evaluate ShareLatex (SHORT TERM, IF NEEDED)

**Action:** If ShareLatex is not actively used, remove it.

**Impact:** Frees ~612 MB RAM; would bring total free RAM to ~1 Gi.

**Risk:** Low if ShareLatex is idle; medium if it's required.

**Commands:**
```bash
docker compose stop sharelatex
docker compose rm sharelatex
```

---

### Stage 3: Add Memory Limits to Containers (MEDIUM TERM, REQUIRES RESTART)

**Action:** Add explicit memory limits to docker-compose.yml for hindsight and other memory-heavy services.

**Example for hindsight:**
```yaml
services:
  hindsight:
    image: ghcr.io/vectorize-io/hindsight:latest
    deploy:
      resources:
        limits:
          memory: 512M  # or 768M depending on headroom needed
```

**Impact:** Prevents services from consuming unbounded memory; predictable resource allocation.

**Risk:** Medium — requires container restart; if limit is too low, hindsight may OOM.

**Recommendation:** Test at 768M first; monitor for OOM events. Hindsight's memory is cache-heavy (RSS 834 MiB suggests it might stabilize lower).

---

### Stage 4: Increase Swap (TEMPORARY, LOW PRIORITY)

**Action:** Add more swap (6–8 Gi).

**Impact:** Symptom relief only; doesn't address root cause (working set is larger than available RAM).

**Risk:** Low operational risk, but performance would degrade under paging load.

**Not recommended** as primary fix; use only if Stages 1–3 are insufficient.

---

## Zabbix Alert Status

**Trigger:** `Linux: High swap space usage` on AgapHost  
**Value:** 1 (FIRING)  
**Last change:** 2026-07-26 04:42:29 UTC+3  
**Condition:** Swap usage > 80%

The alert will **automatically clear** once swap drops below the threshold (typically when used ≤ 3.2 Gi / 4.0 Gi).

## Long-Term Recommendations

1. **Docker Compose Memory Limits:** Add `deploy.resources.limits.memory` to all services in docker-compose.yml. This prevents runaway memory consumption and makes the system predictable.

2. **Monitor Claude Code Sessions:** Interactive development tools are memory-intensive. Consider:
   - Limiting the number of concurrent sessions
   - Restarting sessions periodically if they grow unbounded
   - Monitoring growth patterns

3. **Hindsight Configuration:** Check if hindsight can reduce its cache footprint via environment variables or config (e.g., max memory, cache size limits).

4. **Periodic Audits:** Add task to quarterly review top memory consumers and adjust limits as needed.

---

## Acceptance Criteria Status

| Criterion | Status | Notes |
|-----------|--------|-------|
| Swap free >1 GB sustained | ❌ Pending mitigation | Currently 8 Ki free; Stage 1 would bring to ~700 MiB |
| Zabbix 'High swap' clears | ❌ Pending mitigation | Alert will clear once swap < 80% (~3.2 Gi) |
| Note on dominant consumer + mitigation | ✅ Complete | This document |

---

**Next step:** Execute Stage 1 (kill idle Claude sessions) for immediate relief, then monitor Zabbix alert status.