docs: add Agap agent platform design (A2A, per-model queues, agents)

Overall design for moving from a single tool-heavy Adolf to a multi-agent
platform: agents as personas (prompt+memory+tools) that can swap backbones,
models as the scarce queued resource (per-model queues with always-on /
quota-gated / cost-gated / on-demand lifecycles), and A2A patterns
(agent cards, task lifecycle, context-by-reference) to move work between them.

Covers the required cases: Hindsight reflect as an async queued task to the
large model, the Claude Code loop as just another queue-consuming agent, why
queues are keyed by model rather than agent, and how the lab grows to remote
llama / extra GPUs / research agents.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014t8Qg9gi7H7HtT8MncoXAB

DESIGN-a2a-agents.md

@@ -0,0 +1,170 @@
+# DESIGN — Agap Agent Platform (A2A, model queues, agents)
+
+Status: **draft for review** · Owner: alvis · Drafted 2026-07-21
+
+This is the overall design for turning the Agap homelab from "one Adolf carrying every
+tool + a few background LLM calls" into a **multi-agent platform**: agents as
+personas, models as queued compute, and A2A as the way work moves between them.
+
+---
+
+## 1. Why
+
+Current pain, all observed on the live stack:
+
+- **Duplicated LLM spend.** Every Adolf turn costs two Kimi calls: the reply
+  (~32.8K tokens in) and a *separate* background Hindsight retain/extraction
+  (~22.8K). Only ~425 tokens of that is the actual conversation.
+- **Tool bloat.** One Adolf carries ~84 MCP tool schemas (~12K tokens) + ~26
+  built-in Kimi tools (~10K) on **every** turn, whether relevant or not.
+- **Quota cliffs.** Kimi is a flat, window-limited subscription (~60 messages per
+  5h, ~300/week measured). When the window is spent, Adolf goes dark. There is no
+  graceful degradation and no way to park work until the window resets.
+- **Background work is hardcoded to a model.** Hindsight's reflect/consolidate
+  call a fixed LLM directly. There is no scheduling, no priority, no quota
+  awareness, no way to say "do this on the big model when it's free".
+- **No room to grow.** The ambition is autonomous research agents, remote llama
+  nodes, more GPUs, more agents. None of that fits a single hardcoded assistant.
+
+## 2. Core concepts (and the distinctions that matter)
+
+The central insight: **an agent is not a queue, and a model is not an agent.**
+
+### Task
+The unit of work. Durable, addressable, and **context-by-reference**: a task
+carries *pointers* (memory bank id, git ref, board task id, file path), never
+pasted context. Fields: id, intent, required capability/tier, target model queue,
+priority, status, context refs, result ref, submitter, deadline.
+
+### Model (backbone) — the scarce resource
+A concrete LLM endpoint reached through the LiteLLM gateway. Examples today:
+`kimi` (flat quota), `claude-haiku` (paid, already wired), local ollama
+(`qwen3.5:4b`, `qwen3:8b`, `gemma3:4b`), later a remote llama box or a second GPU.
+
+**Each model has its own queue and its own worker**, because the model is what is
+actually scarce (quota, VRAM, cost, rate limit).
+
+### Agent — the persona
+An agent is a **combination of personality + system prompt + memory + tool scope**
+(e.g. Adolf the proactive auditor; Torgash the marketplace analyst; a research
+agent; the Claude coding loop). An agent is a *configuration*, not a runtime
+resource. Critically:
+
+> **An agent may change its backbone LLM.** Adolf on Kimi today, on a local model
+> tomorrow, on Claude for a hard task. Therefore **queues are keyed by model, not
+> by agent.** An agent *submits into* and *consumes from* model queues.
+
+### Queue — per model, async, with a lifecycle
+Queues are asynchronous by design and differ in how they drain:
+
+| Lifecycle | Behaviour | Example |
+|---|---|---|
+| **always-on** | worker drains continuously in the background | local ollama models |
+| **quota-gated** | drains until the window is exhausted, then parks and resumes on reset | Kimi |
+| **cost-gated** | drains under a budget ceiling; stops/falls back when spent | paid Haiku/Flash |
+| **on-demand** | node is woken/attached when work exists | future remote llama / extra GPU |
+
+A task parked on a quota-gated queue is not lost — it waits for the window, or is
+re-routed if it is urgent and another queue can satisfy the required capability.
+
+## 3. Architecture
+
+Four planes. Keeping them separate is the whole point.
+
+```
+┌─ Coordination plane ──────────────────────────────────────────┐
+│  Task registry + lifecycle (Kanboard as blackboard today)      │
+│  context-by-reference; claim/status; audit trail               │
+└───────────────────────────────────────────────────────────────┘
+┌─ Agent plane ─────────────────────────────────────────────────┐
+│  Agent registry: persona + system prompt + memory bank +       │
+│  tool scope + preferred capability tier                        │
+│  (Adolf, Torgash, research-agent, claude-coder, …)             │
+└───────────────────────────────────────────────────────────────┘
+┌─ Scheduling plane ────────────────────────────────────────────┐
+│  Per-MODEL queues + workers; lifecycle policy (always-on /     │
+│  quota-gated / cost-gated / on-demand); priority; claiming     │
+└───────────────────────────────────────────────────────────────┘
+┌─ Model plane ─────────────────────────────────────────────────┐
+│  LiteLLM gateway: kimi | claude-haiku | local ollama | remote  │
+│  routing, fallback on 429/quota, per-agent virtual keys+budget │
+└───────────────────────────────────────────────────────────────┘
+```
+
+**Shared context stores** (what task references point at): Hindsight (memory
+banks), git/gitea (code + docs), Kanboard (task context), files.
+
+### A2A on top
+A2A gives the vocabulary we otherwise have to invent: **agent cards**
+(capability advertisement), **task lifecycle states**, structured task
+submission/tracking, and — most importantly — the **context-by-reference**
+pattern (send a `contextId`, let the worker read the shared store). We adopt the
+*patterns* first; the wire protocol can follow once more than one runtime needs
+to interoperate.
+
+## 4. Worked examples (the required minimal set)
+
+**(1) Hindsight `reflect` becomes an A2A task.** Reflect is async by design.
+Instead of Hindsight calling a fixed LLM inline, it **submits a task** — intent
+`reflect`, context ref = bank + query, required tier = *large* — onto the
+large-model queue. A worker runs it when that model has capacity/quota; the
+result is written back to the bank. Same for consolidation. This removes the
+hardcoded background call and makes memory work schedulable, priced, and
+quota-aware. (See also the "in-loop extraction" option, which is the cheaper
+counterpart for the *retain* path.)
+
+**(2) The Claude Code CLI loop is just an agent.** `claude-coder` = an agent
+whose persona is "implementer", whose backbone is a Claude model, and whose
+consumption rule is *pull complex/coding tasks*. It is a **special case of a
+queue consumer**, not a privileged component. This is why it already works:
+Adolf files tasks, the Claude loop pulls them. We are formalising what exists.
+
+**(3) Model queues ≠ agent queues.** Adolf may run on Kimi now and something else
+later; Torgash may be cheap-tier normally and escalate to a large model for a
+tricky comparison. So a task is queued against **the capability/model it needs**,
+and the agent identity travels *with the task* (persona + memory refs), not with
+the queue.
+
+**(4) Queues drain differently.** The local queue works all night; the Kimi queue
+stops at 100% of the 5h window and resumes after reset; a paid queue stops at its
+budget. Submitters therefore must state urgency, and the router must be able to
+re-route or park.
+
+## 5. Growing the lab
+
+- **More GPUs / remote llama** → new model entries + their own queues and
+  workers; `on-demand` lifecycle for nodes that are not always up. Nothing else
+  changes.
+- **More agents** (research, finance, home) → new agent registry entries with
+  scoped tools + their own memory banks. They inherit queues and A2A for free.
+- **Autonomous research agents** → long-running, low-priority tasks on always-on
+  local queues, escalating to the large model only for synthesis. This is exactly
+  what per-model queues + priorities make affordable.
+
+## 6. Migration (phased, smallest useful step first)
+
+1. **Registries + schemas** — model registry (endpoint, capability, lifecycle,
+   quota), agent registry (persona/prompt/memory/tools), task schema.
+2. **One queue + one worker** — always-on local model, end-to-end.
+3. **Quota-aware worker** — Kimi: park on exhaustion, resume on reset.
+4. **A2A submission/tracking** with context-by-reference.
+5. **Cut over the examples** — Hindsight reflect → queue; Claude loop → declared
+   agent/consumer; Adolf → declared agent with scoped tools.
+6. **Scale** — remote/extra models, more agents.
+
+## 7. Open questions
+
+- Is Kanboard the queue itself, or does it stay the *human-facing* board while
+  workers use a dedicated queue store (and the two are synced)?
+- Where does the routing decision live — submitter picks the tier, or a central
+  policy re-routes based on live quota/budget?
+- How much A2A do we actually implement (patterns only vs the real protocol)?
+- Claim/lease semantics: what happens to a task whose worker dies mid-run?
+- Does an agent's memory bank follow it across backbones (yes, by design) — and
+  what does that mean for extraction quality when the backbone is weak?
+
+## 8. Related
+
+- Kanboard epic: architecture + LiteLLM gateway + multi-agent framework.
+- Hindsight in-loop extraction (the cheap counterpart to queued reflect).
+- Per-agent tool scoping (why Adolf stops carrying every tool).

Dockerfile

@@ -191,13 +191,21 @@ RUN --mount=type=cache,id=openclaw-bookworm-apt-cache,target=/var/cache/apt,shar
     --mount=type=cache,id=openclaw-bookworm-apt-lists,target=/var/lib/apt,sharing=locked \
     apt-get update && \
     DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
-      ca-certificates curl git hostname lsof openssl procps python3 tini && \
+      ca-certificates chromium curl git hostname lsof openssl procps python3 tini && \
     update-ca-certificates
+# chromium (Debian's real package, not snap) added for the OpenClaw `browser`
+# plugin — it resolves a system Chrome/Chromium to drive via CDP, and the
+# node:*-slim base ships none. Its shared libs (libnss3, libgbm1, ...) come in
+# as chromium's Depends. Adolf fork addition (kb#64: authenticated web access).
 
 RUN chown node:node /app
 
 COPY --from=runtime-assets --chown=node:node /app/dist ./dist
 COPY --from=runtime-assets --chown=node:node /app/node_modules ./node_modules
+# node_modules/@openclaw/ai is a workspace symlink -> ../../packages/ai; the
+# built lib must ship so the gateway's `import "@openclaw/ai"` resolves at
+# runtime (otherwise: "Cannot find package '@openclaw/ai'"). Adolf fork fix.
+COPY --from=runtime-assets --chown=node:node /app/packages/ai ./packages/ai
 COPY --from=runtime-assets --chown=node:node /app/package.json .
 COPY --from=runtime-assets --chown=node:node /app/pnpm-workspace.yaml .
 COPY --from=runtime-assets --chown=node:node /app/patches ./patches

docs/SPIKE-FINDINGS.md

@@ -0,0 +1,184 @@
+# Adolf — P0 De-Risking Spike Findings
+
+**Date:** 2026-07-05 · **Status:** Resolved — see verdicts below · **Scope:** ARCHITECTURE.md §4 gates 1-5
+
+Method: live inspection of the running `kimi-agent` container/image (Dockerfile, `server.js`,
+`docker exec` into it), throwaway containers built from the same image with a **copy** of its
+`/root/.kimi-code` volume (created, tested, and fully destroyed — the live `kimi-agent` service was
+never stopped, restarted, or modified), decompiled/grepped `@moonshot-ai/kimi-code` bundle source,
+and direct reads of the vendored OpenClaw source already in this repo (`src/auto-reply/reply/*`,
+`docs/concepts/model-providers.md`) plus upstream web docs/search.
+
+---
+
+## Gate 1 — Kimi CLI package, home dir, MCP path, flags, auth persistence
+
+**Verdict: architecture assumption was backwards. No change needed — current setup is already correct.**
+
+- The running `kimi-agent` container uses `@moonshot-ai/kimi-code@0.22.3` (binary `kimi`), home dir
+  `/root/.kimi-code`. This **is** the current upstream package — `MoonshotAI/kimi-code` is the
+  active/maintained CLI; `MoonshotAI/kimi-cli` (home `~/.kimi/`) is the **legacy** predecessor being
+  phased out. Confirmed both externally (web search: "Kimi CLI is evolving into Kimi Code CLI... the
+  kimi-cli project will be gradually wound down") and internally — the shipped binary's own `kimi
+  migrate` command copies data **from** `~/.kimi/` (legacy) **into** `~/.kimi-code/` (current), and its
+  bundle contains the string `"Old data kept at ~/.kimi/ — kimi-cli still works."` So `~/.kimi/mcp.json`
+  is the *old* path, not the upstream target.
+- **MCP config**: there is no `kimi mcp` subcommand and no `--mcp-config-file` flag in this version
+  (`kimi --help` confirmed; grepped the bundle for the flag string — absent). MCP servers are
+  configured via a **project-root `.mcp.json`** (Claude-Code-compatible schema, auto-discovered by
+  walking up from cwd) or the interactive `/mcp-config` TUI command. Verified empirically: placing
+  `.mcp.json` in a headless run's working directory does not error and `kimi doctor` validates config
+  files cleanly from that directory. **Action for `adolf-llm`/`cognee-llm`: drop `shared-mcp.json` as
+  `.mcp.json` into each session's working directory** rather than relying on a CLI flag that doesn't exist.
+- **Resume flags**: `-r`/`--resume` are **hidden, undocumented aliases for `-S`/`--session`** (confirmed
+  by the CLI's own `session.resume_hint` meta line: `"command":"kimi -r session_..."`, and empirically
+  by resuming a session with `-r <id>` and getting correct context recall). `-p`, `--output-format
+  stream-json` both exist and work exactly as `kimi-agent/server.js` already uses them.
+- **Auth persistence in a fresh container/volume — confirmed YES.** Copied the live `kimi-agent`'s
+  `/root/.kimi-code` into a brand-new named volume, mounted it on a throwaway container (different
+  container, same image), and made a real API call (`kimi -p "..." --output-format stream-json`) —
+  it authenticated and returned a real completion, and session resume (`-r <id>`) worked correctly
+  across separate `docker exec` invocations. **Subscription auth is fully portable via the CLI-home
+  volume** — `adolf-llm`/`cognee-llm` can each get their own volume seeded from (or sharing) the same
+  OAuth credential files.
+- Cleanup: throwaway container, throwaway volume, and the host-side credential copy under `/tmp` were
+  all deleted after testing. The live `kimi-agent` container was untouched throughout (never
+  stopped/restarted/execed with anything destructive) and is still `Up` on its original uptime.
+
+---
+
+## Gate 2 — OpenClaw → provider session identity
+
+**Verdict: no header/`user`-field wiring for custom providers — but OpenClaw gives us something
+better: a structured, parseable session-identity JSON block embedded directly in every request's
+prompt content. Use it as the primary session key; history-hash is not needed as a fallback for the
+common case.**
+
+Read directly from the vendored OpenClaw source in this repo
+(`src/auto-reply/reply/inbound-meta.ts`) plus `docs/concepts/model-providers.md`:
+
+- **Confirmed no hidden attribution headers or dynamic `user` field for custom `openai-completions`
+  baseUrl providers.** `docs/concepts/model-providers.md:680`: *"Proxy-style OpenAI-compatible routes
+  also skip native OpenAI-only request shaping: ... and no hidden OpenClaw attribution headers."*
+  Attribution headers (`originator`, `version`, `User-Agent`) are attached **only** on verified native
+  hosts (`api.openai.com`, `chatgpt.com/backend-api`) — `adolf-llm` will not receive them.
+- **The actual mechanism**: `buildInboundMetaSystemPrompt()` injects a stable
+  `schema: "openclaw.inbound_meta.v2"` JSON block (`account_id`, `channel`, `provider`, `surface`,
+  `chat_type`) into the **system prompt** of every turn — deliberately excluding anything that
+  changes per-turn, to preserve provider-side prompt-cache prefix stability.
+- **The stable per-conversation key we need — `chat_id`** — is emitted by
+  `buildInboundUserContextPrefix()` into the **user-role** message content instead, specifically as a
+  `"Conversation info (untrusted metadata):"` JSON code block containing
+  `chat_id: ctx.OriginatingTo` (plus `message_id`, `sender`, `timestamp`, etc.). `OriginatingTo` is a
+  channel-agnostic routing id (e.g. `"whatsapp:+1555..."`, `"telegram:chat-1"`; for us it will be the
+  Matrix room/peer id) and is confirmed present for direct-message channels that aren't `webchat`
+  (Matrix qualifies) via `shouldIncludeConversationInfo = !isDirect || (channel && channel !== "webchat")`.
+  This is populated by the core reply pipeline (`get-reply-run.*`) for **every** provider, native or
+  custom — it's provider-agnostic.
+- **Action for `adolf-llm`**: on each incoming request, scan the latest user message content for the
+  `"Conversation info (untrusted metadata):"` JSON block, parse `chat_id`, and use it directly as the
+  session-map key → 1:1 Kimi session resume. This is strictly more robust than a history hash (it's
+  stable even across edited/truncated history) and needs no OpenClaw-side config changes. Keep the
+  history-hash approach only as a defensive fallback if the block is ever absent (e.g. webchat surface,
+  or a future OpenClaw version relocates the field — grep on the label string, don't assume a fixed
+  line offset).
+- Escape hatch confirmed but not needed here: `agents.defaults.models["provider/model"].params.extra_body`
+  can merge static extra JSON into the outbound body for vendor-specific fields, but it's a config-time
+  value, not a per-request dynamic session id, so it doesn't help with mapping.
+
+---
+
+## Gate 3 — Headless image input via file-path reference
+
+**Verdict: confirmed working end-to-end.** No CLI flag is needed — the agent has a built-in
+`ReadMediaFile` tool it invokes autonomously.
+
+Empirical test in a throwaway authed container: prompted `kimi -p "Look at ./test2.png and tell me its
+exact pixel dimensions and dominant color."` (a real 64×64 solid-color PNG placed in the cwd). Observed
+in the `stream-json` output:
+1. `{"role":"assistant","tool_calls":[{"function":{"name":"ReadMediaFile","arguments":"{\"path\":\"./test2.png\"}"}}]}`
+2. Tool result: reads the file, reports `Original dimensions: 64x64 pixels`, and returns an
+   `image_url` content part with a base64 `data:image/png;base64,...` payload wired into the model context.
+3. Final assistant answer correctly identified the exact color used (`rgb(200, 30, 30)`).
+
+(A first attempt with a degenerate 1×1 pixel PNG failed at the provider with `400 ... failed to decode
+image: invalid or unsupported image format` — that's the upstream vision API rejecting a
+near-empty test fixture, not a CLI/tool-path limitation; the 64×64 real PNG round-tripped cleanly.)
+
+**Action for `adolf-llm`**: persist inbound Matrix images to the session's working directory and
+reference them by relative path in the prompt text (e.g. "See attached image: `./img_3.jpg`") — the CLI
+will autonomously call `ReadMediaFile` on it. No special flag or placeholder syntax
+(`[image #N ...]`, which is TUI-paste-only) is required for headless mode.
+
+---
+
+## Gate 4 — Graph store: Kuzu vs Neo4j for Cognee
+
+**Verdict: Kuzu (embedded). Use it; do not stand up Neo4j.**
+
+- **Kuzu is Cognee's own default graph backend** (`GRAPH_DATABASE_PROVIDER=kuzu`), file-based, runs
+  embedded in-process with no network setup, no extra container, no auth surface, and stores its data
+  as plain files (fits directly under `/mnt/ssd/dbs/cognee/`, consistent with the rest of Adolf's
+  storage layout).
+- Cognee's own guidance: Kuzu is recommended for local/single-user use; it uses file-based locking and
+  is explicitly **not** meant for multi-agent/concurrent-process access — that's when Cognee's docs
+  point to Neo4j instead.
+- Adolf is explicitly single-user, single-agent, home-server scale (§1: "auto + tool" memory, one
+  Matrix bot). Neo4j would add: a whole extra JVM-based service (~0.5-1GB+ RAM), its own
+  backup/upgrade/security surface, and a network-exposed port — for zero benefit at this scale, since
+  we don't need Cypher-browser visualization or concurrent multi-writer access.
+- **Recommendation**: Kuzu embedded, no separate container. Revisit only if/when a second concurrent
+  agent needs to write to the same graph, or ad-hoc Cypher-browser graph exploration becomes a real
+  requirement — neither applies to the current design.
+
+---
+
+## Gate 5 — `cognee-llm` suitability: agentic Kimi CLI vs LiteLLM fallback for batch cognify
+
+**Verdict: default cognee's LLM backend to LiteLLM (fallback plan in ARCHITECTURE.md §3.3); keep
+`cognee-llm`/Kimi CLI as an optional low-volume path only. JSON reliability is fine; per-call latency
+and subscription/concurrency risk are the real blockers for batch use.**
+
+Empirical test in a throwaway authed container (same one used for gate 1, cleaned up after):
+
+| Call | Wall time | JSON cleanliness |
+|---|---|---|
+| Trivial prompt (`{"ok":true}`) — process floor | **~5.1 s** | clean |
+| Structured entity/relationship extraction, inline text (no tool call) | **~21.9 s** | clean, exact schema match |
+| Same extraction via file reference (adds a `Read` tool round-trip) | **~24.0 s** | clean, exact schema match |
+
+Findings:
+- **JSON reliability is good** — in both extraction runs the final assistant `content` was strict,
+  schema-conformant JSON with no prose/markdown fences, when explicitly instructed. This de-risks the
+  "won't emit clean JSON" half of the concern.
+- **Latency is the real problem.** ~5 s is the CLI's fixed per-invocation floor (Node process cold
+  start, config/credential load, provider round-trip) even for a near-empty completion; a realistic
+  extraction call runs ~20-25 s. Cognify issues one such call per chunk/entity-extraction step — for a
+  batch of even a few dozen chunks, serialized wall time reaches many minutes, and true parallelism is
+  unverified: the Kimi subscription is a single-seat, interactive-coding-oriented plan, and hammering it
+  with concurrent batch CLI invocations risks rate-limiting or provider-side throttling that this spike
+  did not (and should not) test at scale — that's a live-account risk, not something to probe casually.
+- **Every call is also agentic** (tool-call round trips are possible/likely even for "just extract
+  JSON" prompts, as seen in the file-reference test), adding non-determinism to latency and requiring
+  the wrapper to reliably strip tool-call/meta lines from `stream-json` output before parsing the final
+  JSON payload — extra parsing complexity for zero benefit in a task that doesn't need agentic tool use.
+- **Recommendation**: point Cognee's `LLM_API_BASE` at a LiteLLM-routed model (`judge`/local qwen, per
+  ARCHITECTURE.md §3.3's own stated fallback) as the default for cognify. This is materially faster (no
+  process-spawn floor, no agent loop, no tool-call risk) and avoids risking the shared subscription
+  under batch load. Keep the `cognee-llm` Kimi-CLI wrapper buildable/available for low-volume or
+  experimental use, but do not make it the default path.
+
+---
+
+## Summary table
+
+| Gate | Verdict |
+|---|---|
+| 1. Kimi CLI package/home/MCP/auth | `@moonshot-ai/kimi-code` + `/root/.kimi-code` **is already current upstream** (kimi-cli/`~/.kimi/` is legacy). No `--mcp-config-file`/`kimi mcp`; use project-root `.mcp.json` instead. `-r`/`-p`/`--output-format stream-json` all confirmed. Subscription auth **persists** across a fresh container given a copied CLI-home volume. |
+| 2. OpenClaw session identity | No header/`user`-field wiring for custom providers. Use the `chat_id` (`ctx.OriginatingTo`) embedded in every turn's user-content "Conversation info" JSON block as the session key — parse it from prompt content, not headers. |
+| 3. Headless image input | Confirmed working: CLI autonomously calls a built-in `ReadMediaFile` tool on a referenced file path in `-p` mode; no flag/placeholder syntax needed. |
+| 4. Graph store | **Kuzu (embedded)** — Cognee's own default, matches single-user/no-extra-service goal. Neo4j not justified at this scale. |
+| 5. cognee-llm suitability | JSON output is clean, but ~5-24 s per call plus unverified batch-concurrency limits on a single-seat subscription make it unsuitable as the default. **Default to LiteLLM fallback** for cognify; keep Kimi-CLI path optional. |
+
+All gates are resolved with either direct empirical verification (1, 3, 5) or authoritative source
+reading (2, plus corroborating docs for 4). No gate required an unresolved forbidden action.