Hindsight: route consolidation/reflect to local gemma3:4b (kb#88) + rationale

Per-stage LLM routing so Hindsight's high-volume background jobs stop burning
the rate-limited Kimi subscription:
- CONSOLIDATION + REFLECT -> ollama/gemma3:4b via LiteLLM :4000 (local GPU, free)
- RETAIN (fact extraction) stays on Kimi (hindsight-llm) for quality
Fixes the Kimi 5h window maxing at 100% from ~1100 background calls/3h (dropped
to ~38%). Documents the reasoning in HINDSIGHT-MIGRATION.md section 10:
frequent/mechanical/background stages -> cheap local model; user-facing,
quality-critical, low-volume -> Kimi.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014t8Qg9gi7H7HtT8MncoXAB

adolf/HINDSIGHT-MIGRATION.md

@@ -78,8 +78,8 @@ Matrix ⇄ OpenClaw ("Adolf" gateway)
                         hindsight  (ONE container)
                           :8888 REST + /mcp   ·   :9999 UI
                           built-in Postgres (pg0)
-                          LLM  → LiteLLM :4000 (or Ollama)   [decision, §6]
-                          embeddings → Ollama / built-in     [decision, §6]
+                          LLM  → per-stage routing           [resolved, §10]
+                          embeddings → ollama bge-m3 (GPU)   [resolved, §10]
 ```
 
 Reused infra: **LiteLLM `:4000`** and/or **Ollama** for Hindsight's model calls.
@@ -217,3 +217,37 @@ The Kimi/OpenClaw/Matrix substrate is untouched: `adolf` gateway, `adolf-llm`
 (:8010) provider, the SSE-heartbeat/idle-watchdog fix (kb#71), the
 `openclaw-tools` bridge, `kanboard`/`marketplace` MCP servers, Matrix allow-list
 and E2EE. Only the memory backend and its two integration surfaces change.
+
+## 10. LLM provider strategy — per-stage routing (kb#84, kb#88)
+
+Hindsight calls an LLM in several distinct stages, and they have very different
+cost/quality profiles. Hindsight supports a **separate provider per stage**
+(`HINDSIGHT_API_<STAGE>_LLM_{PROVIDER,BASE_URL,MODEL,API_KEY}`, falling back to
+the global `HINDSIGHT_API_LLM_*` when unset), so each is routed to the model that
+fits it. Config lives in `openai/docker-compose.yml`, `hindsight` service.
+
+| Stage | Model | Where | Why this model |
+|-------|-------|-------|----------------|
+| **RETAIN / extraction** | Kimi — `hindsight-llm:8012` | flat-rate subscription | Pulls atomic facts + entities out of raw **Russian** conversation. Quality-critical and user-visible (bad extraction ⇒ bad memory), and **low volume** — roughly one pass per turn. Worth the strong model. |
+| **CONSOLIDATION** | `ollama/gemma3:4b` via LiteLLM `:4000` | local GPU (free) | Merges / dedups / reconciles stored memories. **Very high volume** — a background reconcile loop + per-retain triggers: ~900 calls / 3 h at ~190 memories. Mostly mechanical; a small model is good enough. |
+| **REFLECT / mental-models** | `ollama/gemma3:4b` via LiteLLM `:4000` | local GPU (free) | Periodic synthesis over the bank and the `reflect` tool. Background, frequent (~170 calls / 3 h), never on the reply path. |
+| Embeddings | `bge-m3` (ollama, GPU) | local | multilingual, GPU-served — see §3 / kb#84. |
+| Reranker | `jina-reranker-v2-base-multilingual` | local (CPU) | multilingual; kb#84. |
+
+**Why the split matters — the kb#88 incident.** Originally (H1b) the *entire*
+Hindsight LLM was pointed at Kimi via `HINDSIGHT_API_LLM_PROVIDER` alone. The
+background consolidation + mental-model jobs then hammered Kimi ~1100 calls / 3 h
+and **maxed the Kimi 5-hour rate window (100 %)** with *no* chat traffic at all —
+because Kimi is a **rate-limited flat subscription**, not a per-token API, and
+these jobs run continuously regardless of user activity. Moving the
+high-volume / low-stakes stages to a free local GPU model dropped Kimi 5 h usage
+100 % → ~38 % while keeping fact extraction on the good model.
+
+**Rule of thumb:** *frequent, background, mechanical* stages (consolidation,
+reflect, mental-models) → **cheap local model**; *user-facing, quality-critical,
+low-volume* work (extraction, and the assistant's own replies) → **Kimi**.
+
+Levers if the local box gets loaded or quality is off:
+`HINDSIGHT_API_CONSOLIDATION_RECONCILE_INTERVAL_SECONDS` (raise to run less
+often), `_MAX_MEMORIES_PER_ROUND` (lower). If `gemma3:4b`'s Russian consolidation
+quality is too weak, `qwen3.5:9b` (same ollama box, `:11436`) is the next step up.

openai/docker-compose.yml

@@ -310,6 +310,19 @@ services:
       # hindsight-llm ignores the key entirely (Kimi CLI wrapper, no real
       # OpenAI auth) — dummy value, non-empty so the client constructs.
       - HINDSIGHT_API_LLM_API_KEY=sk-hindsight-llm-local
+      # Per-stage LLM routing (kb#88): the heavy BACKGROUND stages —
+      # consolidation (~930 calls/3h) + reflect/mental-models (~177/3h) —
+      # were burning the Kimi 5h window (hit 100%). Route them to a cheap
+      # LOCAL model via LiteLLM/ollama (GPU, free); RETAIN extraction inherits
+      # the main HINDSIGHT_API_LLM_* above (Kimi) to keep fact quality.
+      - HINDSIGHT_API_CONSOLIDATION_LLM_PROVIDER=openai
+      - HINDSIGHT_API_CONSOLIDATION_LLM_BASE_URL=http://litellm:4000/v1
+      - HINDSIGHT_API_CONSOLIDATION_LLM_MODEL=ollama/gemma3:4b
+      - HINDSIGHT_API_CONSOLIDATION_LLM_API_KEY=sk-fjQC1BxAiGFSMs
+      - HINDSIGHT_API_REFLECT_LLM_PROVIDER=openai
+      - HINDSIGHT_API_REFLECT_LLM_BASE_URL=http://litellm:4000/v1
+      - HINDSIGHT_API_REFLECT_LLM_MODEL=ollama/gemma3:4b
+      - HINDSIGHT_API_REFLECT_LLM_API_KEY=sk-fjQC1BxAiGFSMs
       - HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai
       - HINDSIGHT_API_EMBEDDINGS_OPENAI_BASE_URL=http://host.docker.internal:11436/v1
       - HINDSIGHT_API_EMBEDDINGS_OPENAI_MODEL=bge-m3