hindsight: route ALL LLM stages to local gemma3:4b
Retain/extraction moved off the Kimi wrapper (hindsight-llm:8012) onto ollama/gemma3:4b via LiteLLM, joining consolidation + reflect which were already local (kb#88). Kimi is no longer in the Hindsight path at all, so the memory backend consumes zero quota and Adolf's 5h window is left entirely for conversation. Accepted tradeoff: kb#84/kb#88 kept retain on Kimi for fact quality, since gemma3:4b's Russian fluency was never verified and this bank is largely Russian. A dry-run extraction on Russian input after the change returned clean, parseable output but captured only one of two facts in the sample, so extraction is coherent but lossier. Revert this first if facts degrade. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -304,17 +304,21 @@ services:
|
||||
container_name: hindsight
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
# ALL stages on the local model (alvis, 2026-07-26): retain/extraction
|
||||
# moved OFF Kimi (hindsight-llm:8012) onto ollama/gemma3:4b via LiteLLM,
|
||||
# joining consolidation + reflect which were already local. Kimi is no
|
||||
# longer in the Hindsight path at all, so the memory backend costs zero
|
||||
# quota and Adolf's 5h window is left entirely for conversation.
|
||||
#
|
||||
# ⚠️ Accepted tradeoff: the kb#88/kb#84 rationale for keeping retain on
|
||||
# Kimi was fact QUALITY — gemma3:4b's Russian fluency was never verified
|
||||
# (it was originally picked only to dodge qwen3:8b's <think>-token bug),
|
||||
# and this bank's content is largely Russian. Watch extraction quality on
|
||||
# the next retains; if facts degrade, this is the first thing to revert.
|
||||
- HINDSIGHT_API_LLM_PROVIDER=openai
|
||||
- HINDSIGHT_API_LLM_BASE_URL=http://hindsight-llm:8012/v1
|
||||
- HINDSIGHT_API_LLM_MODEL=openai/hindsight-llm
|
||||
# hindsight-llm ignores the key entirely (Kimi CLI wrapper, no real
|
||||
# OpenAI auth) — dummy value, non-empty so the client constructs.
|
||||
- HINDSIGHT_API_LLM_API_KEY=sk-hindsight-llm-local
|
||||
# Per-stage LLM routing (kb#88): the heavy BACKGROUND stages —
|
||||
# consolidation (~930 calls/3h) + reflect/mental-models (~177/3h) —
|
||||
# were burning the Kimi 5h window (hit 100%). Route them to a cheap
|
||||
# LOCAL model via LiteLLM/ollama (GPU, free); RETAIN extraction inherits
|
||||
# the main HINDSIGHT_API_LLM_* above (Kimi) to keep fact quality.
|
||||
- HINDSIGHT_API_LLM_BASE_URL=http://litellm:4000/v1
|
||||
- HINDSIGHT_API_LLM_MODEL=ollama/gemma3:4b
|
||||
- HINDSIGHT_API_LLM_API_KEY=sk-fjQC1BxAiGFSMs
|
||||
- HINDSIGHT_API_CONSOLIDATION_LLM_PROVIDER=openai
|
||||
- HINDSIGHT_API_CONSOLIDATION_LLM_BASE_URL=http://litellm:4000/v1
|
||||
- HINDSIGHT_API_CONSOLIDATION_LLM_MODEL=ollama/gemma3:4b
|
||||
|
||||
Reference in New Issue
Block a user