From df2071d5ee172e6f2673b79b6bb619119b5027a6 Mon Sep 17 00:00:00 2001 From: alvis Date: Tue, 21 Jul 2026 09:02:24 +0000 Subject: [PATCH] =?UTF-8?q?docs:=20v2.1=20=E2=80=94=20no=20metered=20API?= =?UTF-8?q?=20by=20default;=20embedding-based=20routing?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit alvis: the workflow is scoped to Claude Code + Kimi wrapper + local GPU embedder + small local model. We do not pay per-token for API usage. - NEW §3a cost model: the binding constraints are QUOTA and VRAM, not money. The "cheap tier" is the LOCAL small model, not a cheap paid one; paid deployments are opt-in fallback per virtual key and never route implicitly. Claude Code is a flat-subscription runtime, not a metered API. - Routing classification is EMBEDDING-based on the already-resident local bge-m3 (LiteLLM Auto Router / semantic-router takes a configurable embedding_model) — no classifier LLM, no API spend. Keep a heuristic fallback: Auto Router v2 is a week old with an open embedding bug report. - Budget governor reframed: arbitrates quota and GPU, not spend. - GPU residency section renumbered 3b. - Decision log 23-25 (incl. Hindsight caller-supplied cognition mode as an optional extra path, default API/queue path unchanged). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_014t8Qg9gi7H7HtT8MncoXAB --- DESIGN-a2a-agents.md | 50 ++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 48 insertions(+), 2 deletions(-) diff --git a/DESIGN-a2a-agents.md b/DESIGN-a2a-agents.md index 48cdf1c6..2e4729c4 100644 --- a/DESIGN-a2a-agents.md +++ b/DESIGN-a2a-agents.md @@ -103,6 +103,10 @@ task's acceptance criteria. Lifecycle: … → done (unverified) → closed (Kimi): a(t)=0 when the window is spent — the queue **parks**, nothing fails, drains on reset. cost-gated: a(t)=0 past budget. on-demand (remote llama): a(t)=0 until woken. Four lifecycles, one function. + + **Note on "cost":** in this lab the binding constraint is **quota and VRAM, + not money** — see §3a. The cost-gated shape exists for the optional paid + fallback only; it is not the normal case. 3. **Hindsight reflect is just a submit** — `{intent: reflect, refs: bank+query, constraints: tier≥large}`, async. Same for consolidation (low priority). 4. **Backbone swap is a constraint edit.** Persona agents name constraints, not @@ -141,8 +145,12 @@ fabric free of micro-churn by construction. │ specific backbone → pinned model_name │ │ “tier” routing → tier pool │ │ automatic router → auto_router/complexity_router │ +│ Classification is EMBEDDING-based on the local bge-m3 (semantic- │ +│ router): no classifier LLM, no API spend (§3a). │ +│ Deployments: kimi (wrapper) | local small model | local embedder; │ +│ metered/paid = opt-in fallback only, never implicit. │ │ The fabric owns everything LiteLLM cannot: ASYNC queueing, parking │ -│ across quota windows, leases, cross-agent budget arbitration. │ +│ across quota windows, leases, cross-agent quota/GPU arbitration. │ └─────────────────────────────────────────────────────────────────────┘ ┌─ Context stores ────────────────────────────────────────────────────┐ │ Hindsight banks (per-agent memory) · gitea (code, docs, this file) │ @@ -150,7 +158,34 @@ fabric free of micro-churn by construction. └─────────────────────────────────────────────────────────────────────┘ ``` -### GPU residency — a local model's a(t) is not 1 +### 3a. Cost model — no metered API by default + +**Hard scope constraint (alvis):** the workflow is **Claude Code + the Kimi +wrapper + a local GPU embedder + a small weak local model**. We do **not** pay +per-token for API usage. Consequences that shape the design: + +| Resource | Nature | Constraint | +|---|---|---| +| **Claude Code** | flat subscription; a *runtime*, not a metered API | the `claude-coder` agent's capacity | +| **Kimi** (via adolf-llm / hindsight-llm wrappers) | flat subscription, windowed | ~60 msgs/5h, ~300/wk — quota, not money | +| **local small model** (ollama) | free | GPU/VRAM contention (§3b) | +| **local embedder** (bge-m3) | free, already resident | never-evict | +| paid API (Haiku/Flash/…) | metered | **optional fallback only — disabled by default, explicitly opt-in** | + +So the "cheap tier" is the **local small model**, not a cheap paid model. The +budget governor (§5) therefore arbitrates **quota and GPU**, not spend. Any +paid deployment in the LiteLLM config must be explicitly enabled per agent via +its virtual key; nothing routes to a metered model implicitly. + +**Routing classification is embedding-based, not model-based.** LiteLLM's Auto +Router (built on `semantic-router`) takes a configurable `embedding_model` + +`match_threshold`, so complexity/semantic classification runs on the **local +bge-m3** we already keep resident — no classifier LLM call, no API spend, ~zero +marginal cost. (Auto Router v2 is new as of 2026-07-14 and has an open +embedding-related bug report: verify hands-on and keep a heuristic +keyword/length fallback for the classifier.) + +### 3b. GPU residency — a local model's a(t) is not 1 Local "free" models contend for VRAM with interactive components (measured on the 8 GB GTX 1070: bge-m3 + gemma3:4b + tei-reranker ≈ 6.2 GB; loading anything @@ -409,3 +444,14 @@ Added in v2.1 (hardening review, same day): 22. **Missed crons are not a keeper duty**: schedules that must not be missed are monitored in **Zabbix**; faults are incidents handled **per problem class** by a future **ops-agent** (§6d), never by a blanket catch-up policy. +23. **No metered API by default** (§3a): scope = Claude Code + Kimi wrapper + + local GPU embedder + small local model. The cheap tier is the *local* + model, not a paid one; paid deployments are opt-in fallback per virtual + key. The governor arbitrates **quota and GPU, not money**. +24. **Routing classification is embedding-based** on the local bge-m3 via + LiteLLM Auto Router (`semantic-router`): no classifier LLM call, no API + spend. Keep a heuristic fallback — Auto Router v2 is new (2026-07-14) with + an open embedding bug report. +25. Hindsight gains an **optional caller-supplied cognition mode** (#131) so it + can run inside a Claude Code session with zero extra API calls; the + API/queue path stays the default. Same verb, different `target`.