ai: migrate LLM backbone from Kimi CLI to Codex CLI

Retires the Moonshot/Kimi subscription in favour of the already-paid ChatGPT
plan. Both CLI wrappers now run `codex exec`; the kimi-agent container is gone.

adolf-llm + hindsight-llm:
- runKimi -> runCodex (`codex exec --json --skip-git-repo-check`), resume via
  `codex exec resume <thread_id>`.
- MCP moves from a per-session .mcp.json (a workaround for Kimi having no
  --mcp-config-file flag) to a $CODEX_HOME/config.toml generated once at
  startup from shared-mcp.json. Field translation is load-bearing:
  bearerTokenEnvVar -> bearer_token_env_var, enabledTools -> enabled_tools.
- approval_policy="never" + sandbox_mode required, or unattended turns block
  on an approval prompt nobody can answer.

kimi-agent removed. It was the ONLY large-tier deployment behind LiteLLM, so
deleting it outright would have silently degraded every large-tier request to
the local 4B model via the existing fallbacks. tier-large, the auto_router
complex-reasoning route and their fallbacks now point at the codex-backed
adolf-llm wrapper (model_name: codex-agent).

Three environment blockers fixed along the way:
- OpenAI geo-blocks this host (403 unsupported_country_region_territory).
  Both containers now egress via the host xray proxy, with NO_PROXY keeping
  MCP and *.alogins.net traffic off the tunnel.
- node:22-slim ships no system CA store; the Rust codex binary validates TLS
  against it, so every HTTPS call failed with a generic transport error while
  Node's own fetch worked. ca-certificates added to both images.
- `codex exec resume` rejects -C/--cd (plain `codex exec` accepts it), which
  broke follow-up turns while first turns succeeded.

Known regression: Kimi's managed-usage API has no Codex equivalent, so the
/usage route returns 501 and there is no quota probe for the codex model.
The two quota plugins degrade quietly to no output.

Also: stop tracking cognee.env (live LLM + JWT secrets) and gitignore it.
The secrets remain in earlier history and should be rotated.

Verified live: plain turn, SSE streaming, session resume, MCP tool call,
bearer-token MCP call, and completions through both LiteLLM routes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Y5QPagv4iun1ghpwM96Ff
This commit is contained in:
2026-08-01 06:13:27 +00:00
parent a27bae828a
commit 9094d71e2f
66 changed files with 653 additions and 851 deletions

View File

@@ -0,0 +1,13 @@
# CUDA torch base with Pascal (sm_61) support — cu118 wheels include sm_61,
# so the GTX 1070 works (unlike the stock TEI GPU image, which needs sm_75+).
FROM pytorch/pytorch:2.3.1-cuda11.8-cudnn8-runtime
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY server.py .
ENV HF_HOME=/root/.cache/huggingface
EXPOSE 80
CMD ["uvicorn", "server:app", "--host", "0.0.0.0", "--port", "80"]

View File

@@ -0,0 +1,8 @@
# torch/cuda come from the pytorch base image. Pin transformers to a version
# known-compatible with jina-reranker-v2's custom modeling code.
transformers==4.44.2
einops>=0.7
sentencepiece>=0.1.99
protobuf>=3.20
fastapi>=0.110
uvicorn[standard]>=0.29

85
ai/tei-reranker/server.py Normal file
View File

@@ -0,0 +1,85 @@
"""Minimal TEI-compatible cross-encoder rerank server (GPU).
Why this exists: HuggingFace's official Text-Embeddings-Inference GPU images
require CUDA compute capability >= 7.5 (Turing+). This box has a GTX 1070
(Pascal, 6.1), so the stock TEI image won't run. Plain CUDA torch DOES support
Pascal (that's why ollama works here), so we serve the same
`jina-reranker-v2-base-multilingual` cross-encoder via torch and expose only the
two endpoints Hindsight's `tei` reranker provider calls:
GET /info -> JSON (init/health probe)
POST /rerank -> {"query": str, "texts": [str], ...}
-> bare list [{"index": i, "score": f}, ...] sorted desc
See hindsight_api/engine/cross_encoder.py::RemoteTEICrossEncoder for the client.
"""
import os
import torch
from fastapi import FastAPI
from pydantic import BaseModel
from transformers import AutoModelForSequenceClassification
MODEL_ID = os.environ.get("RERANKER_MODEL", "jinaai/jina-reranker-v2-base-multilingual")
DEVICE = os.environ.get("RERANKER_DEVICE", "cuda")
MAX_LENGTH = int(os.environ.get("RERANKER_MAX_LENGTH", "1024"))
# fp16 on GPU halves the ~1.1GB fp32 footprint; Pascal supports fp16 storage.
DTYPE = torch.float16 if DEVICE == "cuda" else torch.float32
app = FastAPI(title="tei-reranker")
_model = None
def _load():
global _model
if _model is not None:
return
m = AutoModelForSequenceClassification.from_pretrained(
MODEL_ID, torch_dtype=DTYPE, trust_remote_code=True
)
m.to(DEVICE)
m.eval()
_model = m
@app.on_event("startup")
def startup():
_load()
class RerankRequest(BaseModel):
query: str
texts: list[str]
return_text: bool = False
truncate: bool | None = None
raw_scores: bool | None = None
@app.get("/info")
def info():
# Hindsight only needs a 200 JSON here to consider the server initialized.
return {
"model_id": MODEL_ID,
"model_dtype": str(DTYPE).replace("torch.", ""),
"model_type": {"reranker": {}},
"max_input_length": MAX_LENGTH,
"device": DEVICE,
}
@app.get("/health")
def health():
return {"status": "ok" if _model is not None else "loading"}
@app.post("/rerank")
def rerank(req: RerankRequest):
if not req.texts:
return []
pairs = [[req.query, t] for t in req.texts]
with torch.no_grad():
# jina-reranker-v2 exposes compute_score (batches + moves to device).
scores = _model.compute_score(pairs, max_length=MAX_LENGTH)
if not isinstance(scores, list):
scores = [scores]
results = [{"index": i, "score": float(s)} for i, s in enumerate(scores)]
results.sort(key=lambda r: r["score"], reverse=True)
return results