ai: migrate LLM backbone from Kimi CLI to Codex CLI
Retires the Moonshot/Kimi subscription in favour of the already-paid ChatGPT plan. Both CLI wrappers now run `codex exec`; the kimi-agent container is gone. adolf-llm + hindsight-llm: - runKimi -> runCodex (`codex exec --json --skip-git-repo-check`), resume via `codex exec resume <thread_id>`. - MCP moves from a per-session .mcp.json (a workaround for Kimi having no --mcp-config-file flag) to a $CODEX_HOME/config.toml generated once at startup from shared-mcp.json. Field translation is load-bearing: bearerTokenEnvVar -> bearer_token_env_var, enabledTools -> enabled_tools. - approval_policy="never" + sandbox_mode required, or unattended turns block on an approval prompt nobody can answer. kimi-agent removed. It was the ONLY large-tier deployment behind LiteLLM, so deleting it outright would have silently degraded every large-tier request to the local 4B model via the existing fallbacks. tier-large, the auto_router complex-reasoning route and their fallbacks now point at the codex-backed adolf-llm wrapper (model_name: codex-agent). Three environment blockers fixed along the way: - OpenAI geo-blocks this host (403 unsupported_country_region_territory). Both containers now egress via the host xray proxy, with NO_PROXY keeping MCP and *.alogins.net traffic off the tunnel. - node:22-slim ships no system CA store; the Rust codex binary validates TLS against it, so every HTTPS call failed with a generic transport error while Node's own fetch worked. ca-certificates added to both images. - `codex exec resume` rejects -C/--cd (plain `codex exec` accepts it), which broke follow-up turns while first turns succeeded. Known regression: Kimi's managed-usage API has no Codex equivalent, so the /usage route returns 501 and there is no quota probe for the codex model. The two quota plugins degrade quietly to no output. Also: stop tracking cognee.env (live LLM + JWT secrets) and gitignore it. The secrets remain in earlier history and should be rotated. Verified live: plain turn, SSE streaming, session resume, MCP tool call, bearer-token MCP call, and completions through both LiteLLM routes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014Y5QPagv4iun1ghpwM96Ff
This commit is contained in:
248
ai/litellm-config.yaml
Normal file
248
ai/litellm-config.yaml
Normal file
@@ -0,0 +1,248 @@
|
||||
model_list:
|
||||
# ── oO aliases (used by ml/serving; see oO/CLAUDE.md AI stack) ──────────
|
||||
- model_name: tip-generator
|
||||
litellm_params:
|
||||
model: ollama/qwen2.5:1.5b
|
||||
api_base: http://host.docker.internal:11434
|
||||
|
||||
- model_name: embedder
|
||||
litellm_params:
|
||||
model: ollama/nomic-embed-text
|
||||
api_base: http://host.docker.internal:11434
|
||||
|
||||
# kb#164: the ACTUAL embedder in use (§3a routing classifier for Auto
|
||||
# Router v2) is bge-m3 on 11436, not the `embedder` alias above (which
|
||||
# still points at nomic-embed-text on 11434 -- that alias is legacy/
|
||||
# unused by the current stack, left as-is per kb#164 scope: add bge-m3,
|
||||
# don't touch the mismatch beyond noting it). model-registry.yaml's
|
||||
# `bge-m3` entry's litellm_model_name now matches this model_name.
|
||||
- model_name: bge-m3
|
||||
litellm_params:
|
||||
model: ollama/bge-m3
|
||||
api_base: http://host.docker.internal:11436
|
||||
|
||||
# kb#164: the `judge` alias (anthropic/claude-haiku-4-5, metered) was removed
|
||||
# 2026-07-30 by alvis's decision. No ANTHROPIC_API_KEY was ever set in this
|
||||
# container or .env, so it could not spend; it was kept only as a latent
|
||||
# paid-fallback footgun. Per design §3a (no metered API by default), do not
|
||||
# re-add a metered deployment without an explicit opt-in decision.
|
||||
|
||||
# Codex CLI agent. Replaces the retired `kimi-agent` container (2026-08-01,
|
||||
# Kimi purge): that was the ONLY large-tier deployment behind LiteLLM, so
|
||||
# deleting it outright would have silently degraded every large-tier request
|
||||
# to the local 4B model via the fallbacks below. Repointed at the codex-backed
|
||||
# adolf-llm wrapper (:8010, OpenAI-compatible, model id "adolf") instead of
|
||||
# standing up a third CLI container with its own login.
|
||||
- model_name: codex-agent
|
||||
litellm_params:
|
||||
model: openai/adolf
|
||||
api_base: http://adolf-llm:8010/v1
|
||||
api_key: dummy
|
||||
|
||||
# ── raw model exposure ─────────────────────────────────────────────────
|
||||
- model_name: ollama/qwen3.5:4b
|
||||
litellm_params:
|
||||
model: ollama/qwen3.5:4b
|
||||
api_base: http://host.docker.internal:11436
|
||||
|
||||
- model_name: ollama/qwen3:8b
|
||||
litellm_params:
|
||||
model: ollama/qwen3:8b
|
||||
api_base: http://host.docker.internal:11436
|
||||
|
||||
- model_name: ollama/qwen2.5:1.5b
|
||||
litellm_params:
|
||||
model: ollama/qwen2.5:1.5b
|
||||
api_base: http://host.docker.internal:11436
|
||||
|
||||
- model_name: ollama/qwen2.5:0.5b
|
||||
litellm_params:
|
||||
model: ollama/qwen2.5:0.5b
|
||||
api_base: http://host.docker.internal:11436
|
||||
|
||||
- model_name: ollama/gemma3:4b
|
||||
litellm_params:
|
||||
model: ollama/gemma3:4b
|
||||
api_base: http://host.docker.internal:11436
|
||||
|
||||
- model_name: ollama/gemma3:1b
|
||||
litellm_params:
|
||||
model: ollama/gemma3:1b
|
||||
api_base: http://host.docker.internal:11435
|
||||
|
||||
- model_name: ollama/nomic-embed-text
|
||||
litellm_params:
|
||||
model: ollama/nomic-embed-text
|
||||
api_base: http://host.docker.internal:11435
|
||||
|
||||
# OpenRouter free-tier models
|
||||
- model_name: meta-llama/llama-3.3-70b-instruct:free
|
||||
litellm_params:
|
||||
model: openrouter/meta-llama/llama-3.3-70b-instruct:free
|
||||
api_key: os.environ/OPENROUTER_API_KEY
|
||||
|
||||
- model_name: meta-llama/llama-3.2-3b-instruct:free
|
||||
litellm_params:
|
||||
model: openrouter/meta-llama/llama-3.2-3b-instruct:free
|
||||
api_key: os.environ/OPENROUTER_API_KEY
|
||||
|
||||
- model_name: deepseek/deepseek-r1:free
|
||||
litellm_params:
|
||||
model: openrouter/deepseek/deepseek-r1:free
|
||||
api_key: os.environ/OPENROUTER_API_KEY
|
||||
|
||||
- model_name: qwen/qwen3-4b:free
|
||||
litellm_params:
|
||||
model: openrouter/qwen/qwen3-4b:free
|
||||
api_key: os.environ/OPENROUTER_API_KEY
|
||||
|
||||
- model_name: qwen/qwen3-coder:free
|
||||
litellm_params:
|
||||
model: openrouter/qwen/qwen3-coder:free
|
||||
api_key: os.environ/OPENROUTER_API_KEY
|
||||
|
||||
- model_name: google/gemma-3-27b-it:free
|
||||
litellm_params:
|
||||
model: openrouter/google/gemma-3-27b-it:free
|
||||
api_key: os.environ/OPENROUTER_API_KEY
|
||||
|
||||
- model_name: google/gemma-3-12b-it:free
|
||||
litellm_params:
|
||||
model: openrouter/google/gemma-3-12b-it:free
|
||||
api_key: os.environ/OPENROUTER_API_KEY
|
||||
|
||||
- model_name: mistralai/mistral-small-3.1-24b-instruct:free
|
||||
litellm_params:
|
||||
model: openrouter/mistralai/mistral-small-3.1-24b-instruct:free
|
||||
api_key: os.environ/OPENROUTER_API_KEY
|
||||
|
||||
- model_name: nvidia/nemotron-3-super-120b-a12b:free
|
||||
litellm_params:
|
||||
model: openrouter/nvidia/nemotron-3-super-120b-a12b:free
|
||||
api_key: os.environ/OPENROUTER_API_KEY
|
||||
|
||||
- model_name: openai/gpt-oss-120b:free
|
||||
litellm_params:
|
||||
model: openrouter/openai/gpt-oss-120b:free
|
||||
api_key: os.environ/OPENROUTER_API_KEY
|
||||
|
||||
- model_name: minimax/minimax-m2.5:free
|
||||
litellm_params:
|
||||
model: openrouter/minimax/minimax-m2.5:free
|
||||
api_key: os.environ/OPENROUTER_API_KEY
|
||||
|
||||
- model_name: nousresearch/hermes-3-llama-3.1-405b:free
|
||||
litellm_params:
|
||||
model: openrouter/nousresearch/hermes-3-llama-3.1-405b:free
|
||||
api_key: os.environ/OPENROUTER_API_KEY
|
||||
|
||||
# ── kb#128 (A2A-16): tier pools — alvis's "tier" routing mode ───────────
|
||||
# target = constraint-set ("any large model"), not a specific backbone.
|
||||
# Two litellm_params entries sharing one model_name = a LiteLLM deployment
|
||||
# group; the router load-balances/fails-over across them. tier-large lists
|
||||
# codex-agent FIRST so it's preferred, with local-small as the in-group
|
||||
# failover partner -- this is also what the fallbacks: block below promotes
|
||||
# to an explicit, auditable quota-429-degrades-to-local path (design §2
|
||||
# theorem 2: quota-gated a(t)=0 -> park/degrade, never fail).
|
||||
# tier-small mirrors model-registry.yaml's routing.tiers.small = [local-small].
|
||||
- model_name: tier-small
|
||||
litellm_params:
|
||||
model: ollama/gemma3:4b
|
||||
api_base: http://host.docker.internal:11436
|
||||
|
||||
- model_name: tier-large
|
||||
litellm_params:
|
||||
model: openai/adolf
|
||||
api_base: http://adolf-llm:8010/v1
|
||||
api_key: dummy
|
||||
|
||||
# ── kb#128: Auto Router v2 -- embedding-based classification on the LOCAL
|
||||
# bge-m3 (design §3a/§3b: no classifier LLM, no API spend). Human-readable
|
||||
# source of truth for these routes: openai/auto-router-routes.json (keep
|
||||
# both in sync by hand -- see that file's _note for why).
|
||||
#
|
||||
# auto_router_config is INLINE JSON, not auto_router_config_path. This is
|
||||
# the open Auto Router v2 embedding bug the task brief warned about,
|
||||
# verified hands-on 2026-07-26 against litellm:main-latest: the _path
|
||||
# loader (AutoRouter -> SemanticRouter.from_json) unconditionally builds a
|
||||
# throwaway semantic_router encoder from scratch and demands a real
|
||||
# provider API key even for a local model name like "bge-m3" --
|
||||
# ValueError: "Expected API key via `api_key` parameter or
|
||||
# `{TYPE}_API_KEY` environment variable." The inline-string loader never
|
||||
# touches that code path (it just reads the `routes` key), and was
|
||||
# confirmed end-to-end: real `litellm.embedding(model=ollama/bge-m3)`
|
||||
# calls, zero metered spend, "hi there" -> ollama/gemma3:4b, a refactor/
|
||||
# dependency-injection prompt -> kimi-agent.
|
||||
#
|
||||
# default_model is the free local tier -- an unmatched/low-confidence
|
||||
# request degrades to free compute, never to a paid model.
|
||||
- model_name: auto_router
|
||||
litellm_params:
|
||||
model: auto_router/semantic-v1
|
||||
auto_router_default_model: ollama/gemma3:4b
|
||||
auto_router_embedding_model: bge-m3
|
||||
auto_router_config: >
|
||||
{"routes": [
|
||||
{"name": "ollama/gemma3:4b", "description": "Simple, short, low-stakes requests -- greetings, quick factual lookups, formatting, one-line questions.",
|
||||
"utterances": ["hi", "hello", "what time is it", "what's the weather", "thanks", "what does this word mean", "summarize this in one sentence", "give me a quick yes or no", "format this as a list", "what is 2 plus 2"],
|
||||
"score_threshold": 0.5},
|
||||
{"name": "codex-agent", "description": "Complex reasoning, multi-step planning, coding, or anything needing tool use and deep context.",
|
||||
"utterances": ["write a function that parses this log file and extracts errors", "refactor this class to use dependency injection", "think through the tradeoffs of these two architectures step by step", "debug why this docker container keeps crashing", "plan out the migration from cognee to hindsight across five tasks", "analyze this design document and find inconsistencies", "write a SQL query that joins these three tables and aggregates by month", "review this pull request for security issues"],
|
||||
"score_threshold": 0.5}
|
||||
]}
|
||||
|
||||
# ── kb#128: heuristic keyword/length fallback classifier ────────────────
|
||||
# Auto Router v2 (2026-07-14) has an open embedding-related bug report
|
||||
# (task #128 brief) -- LiteLLM's built-in ComplexityRouter is exactly the
|
||||
# "keyword/length heuristic" fallback the brief calls for: pure regex/
|
||||
# token-count scoring, <1ms, ZERO external calls (verified hands-on by
|
||||
# reading router_strategy/complexity_router/complexity_router.py in the
|
||||
# running litellm:main-latest image, 2026-07-26). Tiers are overridden
|
||||
# here -- the package DEFAULT tiers point at gpt-4o/gpt-4o-mini/claude-
|
||||
# sonnet (metered!), which would silently violate §3a if left as-is; every
|
||||
# tier below maps only to already-governed non-metered deployments.
|
||||
- model_name: complexity_router
|
||||
litellm_params:
|
||||
model: auto_router/complexity_router
|
||||
complexity_router_default_model: ollama/gemma3:4b
|
||||
complexity_router_config:
|
||||
tiers:
|
||||
SIMPLE: ollama/gemma3:4b
|
||||
MEDIUM: ollama/gemma3:4b
|
||||
COMPLEX: tier-large
|
||||
REASONING: tier-large
|
||||
|
||||
litellm_settings:
|
||||
success_callback: ["langfuse"]
|
||||
failure_callback: ["langfuse"]
|
||||
drop_params: true
|
||||
# kb#148 (A2A-16): per-agent attribution + KB-task granularity in Langfuse.
|
||||
# `user_api_key_alias` is populated automatically by LiteLLM from the
|
||||
# calling virtual key (kb#128 provisioned one per agent with key_alias set
|
||||
# to the agent id -- adolf/claude-coder/torgash/researcher), so every
|
||||
# trace is tagged with its agent for free as soon as callers use their
|
||||
# per-agent key. `agent`, `task-id` and `queue` are NOT auto-populated --
|
||||
# callers must pass them explicitly as
|
||||
# `extra_body={"metadata": {"agent": "...", "task-id": "...", "queue": "..."}}`
|
||||
# (OpenAI-SDK-style) or the LiteLLM-native `metadata` field on the request;
|
||||
# LiteLLM copies matching keys straight onto the Langfuse trace as tags.
|
||||
# Wiring individual callers (adolf-llm, kimi-agent wrapper, thin workers)
|
||||
# to actually send that metadata is separate follow-up work, out of this
|
||||
# task's declared scope (docker-compose.yml + litellm-config.yaml only) --
|
||||
# flagged in the kb#148 report as adjacent work.
|
||||
langfuse_default_tags: ["agent", "task-id", "queue", "user_api_key_alias"]
|
||||
fallbacks:
|
||||
- deepseek/deepseek-r1:free: ["ollama/qwen3.5:4b"]
|
||||
# kb#128 acceptance: "a forced 429 degrades cleanly". codex-agent is the
|
||||
# only large deployment routed through LiteLLM today (the `codex`
|
||||
# model-registry id is also called directly via the adolf-llm/
|
||||
# hindsight-llm wrappers, outside LiteLLM by design -- see model-
|
||||
# registry.yaml's codex entry). Both the raw deployment and the tier-large
|
||||
# pool degrade to the free local-small model on 429/quota-exhaustion
|
||||
# rather than failing the caller.
|
||||
- codex-agent: ["ollama/gemma3:4b"]
|
||||
- tier-large: ["tier-small"]
|
||||
# auto_router's embedding path is the one with the open bug report
|
||||
# (design §3a) -- if it errors, fail over to the zero-API-call heuristic
|
||||
# classifier rather than the caller seeing an error.
|
||||
- auto_router: ["complexity_router"]
|
||||
Reference in New Issue
Block a user