alvis/AgapHost
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

Hindsight: route consolidation/reflect to local gemma3:4b (kb#88) + rationale

Browse Source 
Per-stage LLM routing so Hindsight's high-volume background jobs stop burning
the rate-limited Kimi subscription:
- CONSOLIDATION + REFLECT -> ollama/gemma3:4b via LiteLLM :4000 (local GPU, free)
- RETAIN (fact extraction) stays on Kimi (hindsight-llm) for quality
Fixes the Kimi 5h window maxing at 100% from ~1100 background calls/3h (dropped
to ~38%). Documents the reasoning in HINDSIGHT-MIGRATION.md section 10:
frequent/mechanical/background stages -> cheap local model; user-facing,
quality-critical, low-volume -> Kimi.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014t8Qg9gi7H7HtT8MncoXAB
This commit is contained in:
alvis
2026-07-16 20:30:35 +00:00
parent ebb7fb8af3
commit d9668928c0
2 changed files with 49 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
openai/docker-compose.yml
View File

@@ -310,6 +310,19 @@ services:
# hindsight-llm ignores the key entirely (Kimi CLI wrapper, no real
# OpenAI auth) — dummy value, non-empty so the client constructs.
- HINDSIGHT_API_LLM_API_KEY=sk-hindsight-llm-local
# Per-stage LLM routing (kb#88): the heavy BACKGROUND stages —
# consolidation (~930 calls/3h) + reflect/mental-models (~177/3h) —
# were burning the Kimi 5h window (hit 100%). Route them to a cheap
# LOCAL model via LiteLLM/ollama (GPU, free); RETAIN extraction inherits
# the main HINDSIGHT_API_LLM_* above (Kimi) to keep fact quality.
- HINDSIGHT_API_CONSOLIDATION_LLM_PROVIDER=openai
- HINDSIGHT_API_CONSOLIDATION_LLM_BASE_URL=http://litellm:4000/v1
- HINDSIGHT_API_CONSOLIDATION_LLM_MODEL=ollama/gemma3:4b
- HINDSIGHT_API_CONSOLIDATION_LLM_API_KEY=sk-fjQC1BxAiGFSMs
- HINDSIGHT_API_REFLECT_LLM_PROVIDER=openai
- HINDSIGHT_API_REFLECT_LLM_BASE_URL=http://litellm:4000/v1
- HINDSIGHT_API_REFLECT_LLM_MODEL=ollama/gemma3:4b
- HINDSIGHT_API_REFLECT_LLM_API_KEY=sk-fjQC1BxAiGFSMs
- HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai
- HINDSIGHT_API_EMBEDDINGS_OPENAI_BASE_URL=http://host.docker.internal:11436/v1
- HINDSIGHT_API_EMBEDDINGS_OPENAI_MODEL=bge-m3