diff --git a/docs/SPIKE-FINDINGS.md b/docs/SPIKE-FINDINGS.md new file mode 100644 index 00000000..440fc26f --- /dev/null +++ b/docs/SPIKE-FINDINGS.md @@ -0,0 +1,184 @@ +# Adolf — P0 De-Risking Spike Findings + +**Date:** 2026-07-05 · **Status:** Resolved — see verdicts below · **Scope:** ARCHITECTURE.md §4 gates 1-5 + +Method: live inspection of the running `kimi-agent` container/image (Dockerfile, `server.js`, +`docker exec` into it), throwaway containers built from the same image with a **copy** of its +`/root/.kimi-code` volume (created, tested, and fully destroyed — the live `kimi-agent` service was +never stopped, restarted, or modified), decompiled/grepped `@moonshot-ai/kimi-code` bundle source, +and direct reads of the vendored OpenClaw source already in this repo (`src/auto-reply/reply/*`, +`docs/concepts/model-providers.md`) plus upstream web docs/search. + +--- + +## Gate 1 — Kimi CLI package, home dir, MCP path, flags, auth persistence + +**Verdict: architecture assumption was backwards. No change needed — current setup is already correct.** + +- The running `kimi-agent` container uses `@moonshot-ai/kimi-code@0.22.3` (binary `kimi`), home dir + `/root/.kimi-code`. This **is** the current upstream package — `MoonshotAI/kimi-code` is the + active/maintained CLI; `MoonshotAI/kimi-cli` (home `~/.kimi/`) is the **legacy** predecessor being + phased out. Confirmed both externally (web search: "Kimi CLI is evolving into Kimi Code CLI... the + kimi-cli project will be gradually wound down") and internally — the shipped binary's own `kimi + migrate` command copies data **from** `~/.kimi/` (legacy) **into** `~/.kimi-code/` (current), and its + bundle contains the string `"Old data kept at ~/.kimi/ — kimi-cli still works."` So `~/.kimi/mcp.json` + is the *old* path, not the upstream target. +- **MCP config**: there is no `kimi mcp` subcommand and no `--mcp-config-file` flag in this version + (`kimi --help` confirmed; grepped the bundle for the flag string — absent). MCP servers are + configured via a **project-root `.mcp.json`** (Claude-Code-compatible schema, auto-discovered by + walking up from cwd) or the interactive `/mcp-config` TUI command. Verified empirically: placing + `.mcp.json` in a headless run's working directory does not error and `kimi doctor` validates config + files cleanly from that directory. **Action for `adolf-llm`/`cognee-llm`: drop `shared-mcp.json` as + `.mcp.json` into each session's working directory** rather than relying on a CLI flag that doesn't exist. +- **Resume flags**: `-r`/`--resume` are **hidden, undocumented aliases for `-S`/`--session`** (confirmed + by the CLI's own `session.resume_hint` meta line: `"command":"kimi -r session_..."`, and empirically + by resuming a session with `-r ` and getting correct context recall). `-p`, `--output-format + stream-json` both exist and work exactly as `kimi-agent/server.js` already uses them. +- **Auth persistence in a fresh container/volume — confirmed YES.** Copied the live `kimi-agent`'s + `/root/.kimi-code` into a brand-new named volume, mounted it on a throwaway container (different + container, same image), and made a real API call (`kimi -p "..." --output-format stream-json`) — + it authenticated and returned a real completion, and session resume (`-r `) worked correctly + across separate `docker exec` invocations. **Subscription auth is fully portable via the CLI-home + volume** — `adolf-llm`/`cognee-llm` can each get their own volume seeded from (or sharing) the same + OAuth credential files. +- Cleanup: throwaway container, throwaway volume, and the host-side credential copy under `/tmp` were + all deleted after testing. The live `kimi-agent` container was untouched throughout (never + stopped/restarted/execed with anything destructive) and is still `Up` on its original uptime. + +--- + +## Gate 2 — OpenClaw → provider session identity + +**Verdict: no header/`user`-field wiring for custom providers — but OpenClaw gives us something +better: a structured, parseable session-identity JSON block embedded directly in every request's +prompt content. Use it as the primary session key; history-hash is not needed as a fallback for the +common case.** + +Read directly from the vendored OpenClaw source in this repo +(`src/auto-reply/reply/inbound-meta.ts`) plus `docs/concepts/model-providers.md`: + +- **Confirmed no hidden attribution headers or dynamic `user` field for custom `openai-completions` + baseUrl providers.** `docs/concepts/model-providers.md:680`: *"Proxy-style OpenAI-compatible routes + also skip native OpenAI-only request shaping: ... and no hidden OpenClaw attribution headers."* + Attribution headers (`originator`, `version`, `User-Agent`) are attached **only** on verified native + hosts (`api.openai.com`, `chatgpt.com/backend-api`) — `adolf-llm` will not receive them. +- **The actual mechanism**: `buildInboundMetaSystemPrompt()` injects a stable + `schema: "openclaw.inbound_meta.v2"` JSON block (`account_id`, `channel`, `provider`, `surface`, + `chat_type`) into the **system prompt** of every turn — deliberately excluding anything that + changes per-turn, to preserve provider-side prompt-cache prefix stability. +- **The stable per-conversation key we need — `chat_id`** — is emitted by + `buildInboundUserContextPrefix()` into the **user-role** message content instead, specifically as a + `"Conversation info (untrusted metadata):"` JSON code block containing + `chat_id: ctx.OriginatingTo` (plus `message_id`, `sender`, `timestamp`, etc.). `OriginatingTo` is a + channel-agnostic routing id (e.g. `"whatsapp:+1555..."`, `"telegram:chat-1"`; for us it will be the + Matrix room/peer id) and is confirmed present for direct-message channels that aren't `webchat` + (Matrix qualifies) via `shouldIncludeConversationInfo = !isDirect || (channel && channel !== "webchat")`. + This is populated by the core reply pipeline (`get-reply-run.*`) for **every** provider, native or + custom — it's provider-agnostic. +- **Action for `adolf-llm`**: on each incoming request, scan the latest user message content for the + `"Conversation info (untrusted metadata):"` JSON block, parse `chat_id`, and use it directly as the + session-map key → 1:1 Kimi session resume. This is strictly more robust than a history hash (it's + stable even across edited/truncated history) and needs no OpenClaw-side config changes. Keep the + history-hash approach only as a defensive fallback if the block is ever absent (e.g. webchat surface, + or a future OpenClaw version relocates the field — grep on the label string, don't assume a fixed + line offset). +- Escape hatch confirmed but not needed here: `agents.defaults.models["provider/model"].params.extra_body` + can merge static extra JSON into the outbound body for vendor-specific fields, but it's a config-time + value, not a per-request dynamic session id, so it doesn't help with mapping. + +--- + +## Gate 3 — Headless image input via file-path reference + +**Verdict: confirmed working end-to-end.** No CLI flag is needed — the agent has a built-in +`ReadMediaFile` tool it invokes autonomously. + +Empirical test in a throwaway authed container: prompted `kimi -p "Look at ./test2.png and tell me its +exact pixel dimensions and dominant color."` (a real 64×64 solid-color PNG placed in the cwd). Observed +in the `stream-json` output: +1. `{"role":"assistant","tool_calls":[{"function":{"name":"ReadMediaFile","arguments":"{\"path\":\"./test2.png\"}"}}]}` +2. Tool result: reads the file, reports `Original dimensions: 64x64 pixels`, and returns an + `image_url` content part with a base64 `data:image/png;base64,...` payload wired into the model context. +3. Final assistant answer correctly identified the exact color used (`rgb(200, 30, 30)`). + +(A first attempt with a degenerate 1×1 pixel PNG failed at the provider with `400 ... failed to decode +image: invalid or unsupported image format` — that's the upstream vision API rejecting a +near-empty test fixture, not a CLI/tool-path limitation; the 64×64 real PNG round-tripped cleanly.) + +**Action for `adolf-llm`**: persist inbound Matrix images to the session's working directory and +reference them by relative path in the prompt text (e.g. "See attached image: `./img_3.jpg`") — the CLI +will autonomously call `ReadMediaFile` on it. No special flag or placeholder syntax +(`[image #N ...]`, which is TUI-paste-only) is required for headless mode. + +--- + +## Gate 4 — Graph store: Kuzu vs Neo4j for Cognee + +**Verdict: Kuzu (embedded). Use it; do not stand up Neo4j.** + +- **Kuzu is Cognee's own default graph backend** (`GRAPH_DATABASE_PROVIDER=kuzu`), file-based, runs + embedded in-process with no network setup, no extra container, no auth surface, and stores its data + as plain files (fits directly under `/mnt/ssd/dbs/cognee/`, consistent with the rest of Adolf's + storage layout). +- Cognee's own guidance: Kuzu is recommended for local/single-user use; it uses file-based locking and + is explicitly **not** meant for multi-agent/concurrent-process access — that's when Cognee's docs + point to Neo4j instead. +- Adolf is explicitly single-user, single-agent, home-server scale (§1: "auto + tool" memory, one + Matrix bot). Neo4j would add: a whole extra JVM-based service (~0.5-1GB+ RAM), its own + backup/upgrade/security surface, and a network-exposed port — for zero benefit at this scale, since + we don't need Cypher-browser visualization or concurrent multi-writer access. +- **Recommendation**: Kuzu embedded, no separate container. Revisit only if/when a second concurrent + agent needs to write to the same graph, or ad-hoc Cypher-browser graph exploration becomes a real + requirement — neither applies to the current design. + +--- + +## Gate 5 — `cognee-llm` suitability: agentic Kimi CLI vs LiteLLM fallback for batch cognify + +**Verdict: default cognee's LLM backend to LiteLLM (fallback plan in ARCHITECTURE.md §3.3); keep +`cognee-llm`/Kimi CLI as an optional low-volume path only. JSON reliability is fine; per-call latency +and subscription/concurrency risk are the real blockers for batch use.** + +Empirical test in a throwaway authed container (same one used for gate 1, cleaned up after): + +| Call | Wall time | JSON cleanliness | +|---|---|---| +| Trivial prompt (`{"ok":true}`) — process floor | **~5.1 s** | clean | +| Structured entity/relationship extraction, inline text (no tool call) | **~21.9 s** | clean, exact schema match | +| Same extraction via file reference (adds a `Read` tool round-trip) | **~24.0 s** | clean, exact schema match | + +Findings: +- **JSON reliability is good** — in both extraction runs the final assistant `content` was strict, + schema-conformant JSON with no prose/markdown fences, when explicitly instructed. This de-risks the + "won't emit clean JSON" half of the concern. +- **Latency is the real problem.** ~5 s is the CLI's fixed per-invocation floor (Node process cold + start, config/credential load, provider round-trip) even for a near-empty completion; a realistic + extraction call runs ~20-25 s. Cognify issues one such call per chunk/entity-extraction step — for a + batch of even a few dozen chunks, serialized wall time reaches many minutes, and true parallelism is + unverified: the Kimi subscription is a single-seat, interactive-coding-oriented plan, and hammering it + with concurrent batch CLI invocations risks rate-limiting or provider-side throttling that this spike + did not (and should not) test at scale — that's a live-account risk, not something to probe casually. +- **Every call is also agentic** (tool-call round trips are possible/likely even for "just extract + JSON" prompts, as seen in the file-reference test), adding non-determinism to latency and requiring + the wrapper to reliably strip tool-call/meta lines from `stream-json` output before parsing the final + JSON payload — extra parsing complexity for zero benefit in a task that doesn't need agentic tool use. +- **Recommendation**: point Cognee's `LLM_API_BASE` at a LiteLLM-routed model (`judge`/local qwen, per + ARCHITECTURE.md §3.3's own stated fallback) as the default for cognify. This is materially faster (no + process-spawn floor, no agent loop, no tool-call risk) and avoids risking the shared subscription + under batch load. Keep the `cognee-llm` Kimi-CLI wrapper buildable/available for low-volume or + experimental use, but do not make it the default path. + +--- + +## Summary table + +| Gate | Verdict | +|---|---| +| 1. Kimi CLI package/home/MCP/auth | `@moonshot-ai/kimi-code` + `/root/.kimi-code` **is already current upstream** (kimi-cli/`~/.kimi/` is legacy). No `--mcp-config-file`/`kimi mcp`; use project-root `.mcp.json` instead. `-r`/`-p`/`--output-format stream-json` all confirmed. Subscription auth **persists** across a fresh container given a copied CLI-home volume. | +| 2. OpenClaw session identity | No header/`user`-field wiring for custom providers. Use the `chat_id` (`ctx.OriginatingTo`) embedded in every turn's user-content "Conversation info" JSON block as the session key — parse it from prompt content, not headers. | +| 3. Headless image input | Confirmed working: CLI autonomously calls a built-in `ReadMediaFile` tool on a referenced file path in `-p` mode; no flag/placeholder syntax needed. | +| 4. Graph store | **Kuzu (embedded)** — Cognee's own default, matches single-user/no-extra-service goal. Neo4j not justified at this scale. | +| 5. cognee-llm suitability | JSON output is clean, but ~5-24 s per call plus unverified batch-concurrency limits on a single-seat subscription make it unsuitable as the default. **Default to LiteLLM fallback** for cognify; keep Kimi-CLI path optional. | + +All gates are resolved with either direct empirical verification (1, 3, 5) or authoritative source +reading (2, plus corroborating docs for 4). No gate required an unresolved forbidden action.