From d9668928c033be5f864544ab7ae64f4243b1537e Mon Sep 17 00:00:00 2001 From: alvis Date: Thu, 16 Jul 2026 20:30:35 +0000 Subject: [PATCH] Hindsight: route consolidation/reflect to local gemma3:4b (kb#88) + rationale Per-stage LLM routing so Hindsight's high-volume background jobs stop burning the rate-limited Kimi subscription: - CONSOLIDATION + REFLECT -> ollama/gemma3:4b via LiteLLM :4000 (local GPU, free) - RETAIN (fact extraction) stays on Kimi (hindsight-llm) for quality Fixes the Kimi 5h window maxing at 100% from ~1100 background calls/3h (dropped to ~38%). Documents the reasoning in HINDSIGHT-MIGRATION.md section 10: frequent/mechanical/background stages -> cheap local model; user-facing, quality-critical, low-volume -> Kimi. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_014t8Qg9gi7H7HtT8MncoXAB --- adolf/HINDSIGHT-MIGRATION.md | 38 ++++++++++++++++++++++++++++++++++-- openai/docker-compose.yml | 13 ++++++++++++ 2 files changed, 49 insertions(+), 2 deletions(-) diff --git a/adolf/HINDSIGHT-MIGRATION.md b/adolf/HINDSIGHT-MIGRATION.md index fe702de..544a938 100644 --- a/adolf/HINDSIGHT-MIGRATION.md +++ b/adolf/HINDSIGHT-MIGRATION.md @@ -78,8 +78,8 @@ Matrix ⇄ OpenClaw ("Adolf" gateway) hindsight (ONE container) :8888 REST + /mcp · :9999 UI built-in Postgres (pg0) - LLM → LiteLLM :4000 (or Ollama) [decision, §6] - embeddings → Ollama / built-in [decision, §6] + LLM → per-stage routing [resolved, §10] + embeddings → ollama bge-m3 (GPU) [resolved, §10] ``` Reused infra: **LiteLLM `:4000`** and/or **Ollama** for Hindsight's model calls. @@ -217,3 +217,37 @@ The Kimi/OpenClaw/Matrix substrate is untouched: `adolf` gateway, `adolf-llm` (:8010) provider, the SSE-heartbeat/idle-watchdog fix (kb#71), the `openclaw-tools` bridge, `kanboard`/`marketplace` MCP servers, Matrix allow-list and E2EE. Only the memory backend and its two integration surfaces change. + +## 10. LLM provider strategy — per-stage routing (kb#84, kb#88) + +Hindsight calls an LLM in several distinct stages, and they have very different +cost/quality profiles. Hindsight supports a **separate provider per stage** +(`HINDSIGHT_API__LLM_{PROVIDER,BASE_URL,MODEL,API_KEY}`, falling back to +the global `HINDSIGHT_API_LLM_*` when unset), so each is routed to the model that +fits it. Config lives in `openai/docker-compose.yml`, `hindsight` service. + +| Stage | Model | Where | Why this model | +|-------|-------|-------|----------------| +| **RETAIN / extraction** | Kimi — `hindsight-llm:8012` | flat-rate subscription | Pulls atomic facts + entities out of raw **Russian** conversation. Quality-critical and user-visible (bad extraction ⇒ bad memory), and **low volume** — roughly one pass per turn. Worth the strong model. | +| **CONSOLIDATION** | `ollama/gemma3:4b` via LiteLLM `:4000` | local GPU (free) | Merges / dedups / reconciles stored memories. **Very high volume** — a background reconcile loop + per-retain triggers: ~900 calls / 3 h at ~190 memories. Mostly mechanical; a small model is good enough. | +| **REFLECT / mental-models** | `ollama/gemma3:4b` via LiteLLM `:4000` | local GPU (free) | Periodic synthesis over the bank and the `reflect` tool. Background, frequent (~170 calls / 3 h), never on the reply path. | +| Embeddings | `bge-m3` (ollama, GPU) | local | multilingual, GPU-served — see §3 / kb#84. | +| Reranker | `jina-reranker-v2-base-multilingual` | local (CPU) | multilingual; kb#84. | + +**Why the split matters — the kb#88 incident.** Originally (H1b) the *entire* +Hindsight LLM was pointed at Kimi via `HINDSIGHT_API_LLM_PROVIDER` alone. The +background consolidation + mental-model jobs then hammered Kimi ~1100 calls / 3 h +and **maxed the Kimi 5-hour rate window (100 %)** with *no* chat traffic at all — +because Kimi is a **rate-limited flat subscription**, not a per-token API, and +these jobs run continuously regardless of user activity. Moving the +high-volume / low-stakes stages to a free local GPU model dropped Kimi 5 h usage +100 % → ~38 % while keeping fact extraction on the good model. + +**Rule of thumb:** *frequent, background, mechanical* stages (consolidation, +reflect, mental-models) → **cheap local model**; *user-facing, quality-critical, +low-volume* work (extraction, and the assistant's own replies) → **Kimi**. + +Levers if the local box gets loaded or quality is off: +`HINDSIGHT_API_CONSOLIDATION_RECONCILE_INTERVAL_SECONDS` (raise to run less +often), `_MAX_MEMORIES_PER_ROUND` (lower). If `gemma3:4b`'s Russian consolidation +quality is too weak, `qwen3.5:9b` (same ollama box, `:11436`) is the next step up. diff --git a/openai/docker-compose.yml b/openai/docker-compose.yml index 9b44fe7..7a3e03e 100644 --- a/openai/docker-compose.yml +++ b/openai/docker-compose.yml @@ -310,6 +310,19 @@ services: # hindsight-llm ignores the key entirely (Kimi CLI wrapper, no real # OpenAI auth) — dummy value, non-empty so the client constructs. - HINDSIGHT_API_LLM_API_KEY=sk-hindsight-llm-local + # Per-stage LLM routing (kb#88): the heavy BACKGROUND stages — + # consolidation (~930 calls/3h) + reflect/mental-models (~177/3h) — + # were burning the Kimi 5h window (hit 100%). Route them to a cheap + # LOCAL model via LiteLLM/ollama (GPU, free); RETAIN extraction inherits + # the main HINDSIGHT_API_LLM_* above (Kimi) to keep fact quality. + - HINDSIGHT_API_CONSOLIDATION_LLM_PROVIDER=openai + - HINDSIGHT_API_CONSOLIDATION_LLM_BASE_URL=http://litellm:4000/v1 + - HINDSIGHT_API_CONSOLIDATION_LLM_MODEL=ollama/gemma3:4b + - HINDSIGHT_API_CONSOLIDATION_LLM_API_KEY=sk-fjQC1BxAiGFSMs + - HINDSIGHT_API_REFLECT_LLM_PROVIDER=openai + - HINDSIGHT_API_REFLECT_LLM_BASE_URL=http://litellm:4000/v1 + - HINDSIGHT_API_REFLECT_LLM_MODEL=ollama/gemma3:4b + - HINDSIGHT_API_REFLECT_LLM_API_KEY=sk-fjQC1BxAiGFSMs - HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai - HINDSIGHT_API_EMBEDDINGS_OPENAI_BASE_URL=http://host.docker.internal:11436/v1 - HINDSIGHT_API_EMBEDDINGS_OPENAI_MODEL=bge-m3