# Adolf memory migration — Cognee → Hindsight **Status:** Planned · **Date:** 2026-07-13 · **Owner:** alvis This is the authoritative design + plan for replacing Adolf's long-term memory subsystem (**Cognee**) with **Hindsight** (Vectorize, MIT, self-hosted). It supersedes the Cognee-specific parts of `docs/ARCHITECTURE.md` and gates 4–5 of `docs/SPIKE-FINDINGS.md` in the OpenClaw fork (`/home/alvis/adolf`). Memory stays integrated into Adolf **exactly the two ways Cognee was** — as a **tool** (MCP) and as **forced hooks** (an OpenClaw memory plugin) — so no behaviour the user sees is lost; only the backend changes. > Scope note: this document + the kanboard **Ready** tasks (H1–H5) are the > migration. No live service, compose file, `openclaw.json`, or plugin code has > been changed yet — those edits are the H-tasks. --- ## 1. Why migrate Cognee works, but its Agap deployment carries three structural costs, all documented in the (now-retired) kanboard cognee tasks: - **Three bespoke services** to keep the memory stack alive: `cognee` (FastAPI + embedded Kuzu graph + Qdrant vectors), `cognee-mcp` (a patched MCP→HTTP proxy, local build overlay from kb#70), and `cognee-llm` (a stateless Kimi-CLI wrapper that exists *only* to give Cognee an LLM on the flat subscription). - **The cognify pipeline is fragile and expensive.** "Cognify" (turning raw turns into a graph) is an LLM step. On the Kimi CLI it runs ~5–24 s per call and drains the single-seat subscription quota (SPIKE gate 5 recommended LiteLLM instead). The plugin's async "cognify sweep" also silently stalled twice (kb#69) because of OpenClaw plugin-lifecycle edge cases, so freshly told facts weren't retrievable cross-session until the sweep was re-armed. - **Scoping is best-effort.** Under `ENABLE_BACKEND_ACCESS_CONTROL=False` all datasets share one graph/vector backend, so per-chat isolation leaks (kb#59). ## 2. What Hindsight gives us - **One container.** `ghcr.io/vectorize-io/hindsight:latest` — REST API on **:8888**, web UI on **:9999**, built-in PostgreSQL (`pg0`, persisted under `/home/hindsight/.pg0`). It owns its own vector + graph + temporal representation internally (the "four-network" model), so **no external Qdrant or Kuzu** is needed for memory. - **Built-in MCP server** mounted at `/mcp` on the same port — ~30 tools including `retain` / `recall` / `reflect`. This **removes the need for a separate `cognee-mcp` proxy container entirely**. - **Retain learns on its own.** `retain` runs Hindsight's extraction/reflection pipeline internally (optionally `async`), so there is **no separate "cognify sweep" to arm, throttle, or watch** — the whole class of kb#69 bugs disappears. - **Memory banks** are first-class isolation units, scoped by URL path (`/v1/{tenant}/banks/{bank_id}/…`), so per-chat / per-user scoping is real, not best-effort. - **Bring-your-own LLM/embeddings** — OpenAI-compatible, Anthropic, Ollama, LMStudio, etc. We point it at the infra we already run (LiteLLM `:4000` and/or Ollama), so we can **delete `cognee-llm`** rather than port it. Net: **3 services → 1**, plus we drop the cognify-sweep machinery. ## 3. Target architecture ``` Matrix ⇄ OpenClaw ("Adolf" gateway) │ provider adolf-llm (Kimi wrapper, :8010) — unchanged ▼ adolf-llm ── kimi CLI (agent) │ ┌─────────┴───────────────── memory is a boundary concern ─────────────┐ │ │ │ (a) FORCED HOOKS — hindsight-memory OpenClaw plugin │ │ before_prompt_build → recall → inject as prependContext │ │ agent_end → retain (async:true) the turn │ │ │ │ (b) TOOL — Hindsight built-in MCP in openclaw.json mcp.servers │ │ http://hindsight:8888/mcp/adolf/ → retain/recall/reflect/… │ └───────────────────────────────┬───────────────────────────────────────┘ ▼ hindsight (ONE container) :8888 REST + /mcp · :9999 UI built-in Postgres (pg0) LLM → per-stage routing [resolved, §10] embeddings → ollama bge-m3 (GPU) [resolved, §10] ``` Reused infra: **LiteLLM `:4000`** and/or **Ollama** for Hindsight's model calls. **Retired:** `cognee`, `cognee-mcp`, `cognee-llm`, Qdrant-for-cognee, Kuzu, the `cognee-openclaw-plugin`, and all `openclaw.json` cognee references. ### 3.1 Surface (a) — forced hooks (the `hindsight-memory` plugin) A new OpenClaw memory plugin replacing `cognee-openclaw-plugin`, modelled on the same Honcho touchpoints Cognee used, so the plugin shape is familiar: | Cognee plugin (`cognee-memory`) | Hindsight plugin (`hindsight-memory`) | |-----------------------------------------------------|---------------------------------------------------------------| | `before_prompt_build` → LLM-free graph recall inject | `before_prompt_build` → `recall` → inject `prependContext` | | `agent_end` → raw `/add` (no inline cognify) | `agent_end` → `retain` (`async:true`) the user+assistant turn | | throttled **cognify sweep** (dirty-tracker, timers) | **removed** — retain does extraction/learning internally | | `cognee_recall` registered tool | `hindsight_recall` (+ optional `hindsight_reflect`) tool | - Activated via `plugins.entries.hindsight-memory` in `openclaw.json` with the same hook grants Cognee needed: `hooks.allowConversationAccess: true` and `allowPromptInjection: true` (external plugins must opt in). - **Recall stays off the hot LLM path.** Hindsight `recall` is retrieval (semantic + BM25 + graph + temporal) with evidence scoring — no generative synthesis — so it's the direct analogue of Cognee's LLM-free `onlyContext` recall. The LLM-backed synthesis path is `reflect`, exposed as a deliberate tool, not run per-turn. - **No freshness dial / sweep.** Retain with `async:true` returns fast and lets Hindsight extract/consolidate in the background; there is no plugin-owned timer to stall. - Same "untrusted metadata" framing on the injected block; same cleaning of OpenClaw's `Conversation info (untrusted metadata):` and the memory block out of stored/queried text. ### 3.2 Surface (b) — tool (built-in MCP) Replace the `cognee` entry in `openclaw.json` `mcp.servers` with: ```jsonc mcp: { servers: { hindsight: { type: "http", url: "http://hindsight:8888/mcp/adolf/", // bank-scoped by URL path }, // openclaw-tools, kanboard, marketplace — unchanged }, } ``` The single bank in the path (`adolf`, or a per-chat bank) selects isolation; the built-in MCP then exposes `retain`, `recall`, `reflect`, plus mental-model / directive / memory-browse tools. This replaces cognee-mcp's `remember` / `recall` / `forget` — and gives deliberate delete via the memory-management tools instead of the hand-patched `forget(data_id)` from kb#70. ## 4. REST / MCP API mapping Base path: `http://hindsight:8888/v1/default` (tenant `default`). Bank id is a **path** parameter. | Operation | Cognee (old) | Hindsight (new) | |------------------|------------------------------------------------|-----------------------------------------------------------------------| | store a turn | `POST /api/v1/add` (+ later `/cognify`) | `POST /banks/{bank}/memories` body `{items:[{content,context,tags,timestamp}], async:true}` | | recall (no LLM) | `POST /api/v1/search` `GRAPH_COMPLETION` `onlyContext:true` | `POST /banks/{bank}/memories/recall` body `{query, budget, max_tokens, tags}` | | deep answer (LLM)| cognee-mcp `recall` (GRAPH_COMPLETION) | `POST /banks/{bank}/reflect` body `{query, budget, max_tokens, response_schema?}` | | delete an entry | patched `forget(dataset, data_id)` (kb#70) | memory-management endpoints / MCP (`delete`, `clear_memories`) | | list / inspect | dataset status polling | `GET /banks/{bank}/memories/list`, `GET /banks` | | create bank | dataset created implicitly on add | `PUT /banks/{bank}` | Built-in MCP tools live at `http://hindsight:8888/mcp/{bank}/` (HTTP transport; bank via URL path, `X-Bank-Id` header, or `HINDSIGHT_MCP_BANK_ID` default). > Exact request-body field names and any auth headers must be confirmed against > the running instance's OpenAPI (`http://localhost:8888/docs`) and the Hindsight > configuration docs during H1 — treat the bodies above as the shape, not gospel. ## 5. Bank scoping Mirror Cognee's per-conversation `chat_` dataset with a per-conversation **bank**: derive `bank_id` from OpenClaw's `chat_id` (the `Conversation info (untrusted metadata):` block; see SPIKE gate 2), sanitized to `chat_`. A single shared `adolf` bank is the simpler alternative if cross-chat recall is actually wanted — decide in H3. Banks are hard isolation in Hindsight, so per-chat is now safe (unlike Cognee's leaky datasets). ## 6. Open decisions (resolve in H1) 1. **LLM backend for retain/reflect.** SPIKE gate 5 already concluded the extraction workload should *not* sit on the Kimi CLI (latency + single-seat quota). Recommendation: point Hindsight's LLM at **LiteLLM `:4000`** (or a local **Ollama** model for zero marginal cost). This is why `cognee-llm` is deleted, not ported. Confirm Hindsight's provider env-var names on the image. 2. **Embeddings.** Prefer the local **Ollama** embedder already available (`nomic-embed` / `bge-m3` at `host.docker.internal:11436`) or Hindsight's built-in, to keep embeddings off any paid path. 3. **Storage path.** Persist `pg0` under `/mnt/ssd/dbs/hindsight/` to match the Agap storage layout (replaces `/mnt/ssd/dbs/cognee/`). 4. **UI exposure.** Whether to reverse-proxy the `:9999` UI (Caddy) or keep it internal-only. 5. **Auth.** Open by default; enable the tenant API-key extension (`HINDSIGHT_API_TENANT_API_KEY`, `Authorization: Bearer`) if the service is reachable beyond the compose network. ## 7. Data migration Cognee's Kuzu graph is **not** portable into Hindsight's store. The memory corpus is low-value conversational history, so **start Hindsight empty** rather than building an exporter. Optionally replay a handful of durable facts by calling `retain` once at cutover. The two throwaway datasets left in Cognee (`chat_verify`, `chat_webchat`) are discarded with the stack. ## 8. Migration phases (kanboard **Ready**, project *Adolf*) - **H1 · Deploy Hindsight service** — add the `hindsight` container to `openai/docker-compose.yml` (image, ports 8888/9999, `pg0` volume, LLM + embedding provider env → LiteLLM/Ollama), bring it up, confirm `/docs` + a round-trip `retain`→`recall`. Resolves §6 decisions. - **H2 · Wire built-in MCP as an Adolf tool** — swap `mcp.servers.cognee` → `mcp.servers.hindsight` (`/mcp/{bank}/`) in `openclaw.json`; verify Adolf can call `retain`/`recall`/`reflect` as tools. - **H3 · `hindsight-memory` OpenClaw plugin (forced hooks)** — build the plugin replacing `cognee-openclaw-plugin`: `before_prompt_build`→recall inject, `agent_end`→`retain(async)`, `hindsight_recall`/`hindsight_reflect` tools, bank scoping; activate in `openclaw.json`. Delete the cognify-sweep machinery. - **H4 · Decommission Cognee** — remove `cognee`, `cognee-mcp`, `cognee-llm` services + volumes, the `cognee-openclaw-plugin`, and all `openclaw.json`/ `shared-mcp.json` cognee references. Free `/mnt/ssd/dbs/cognee`. - **H5 · End-to-end verification** — state a fact → fresh session → recalled via injected memory (no LLM on the recall path); measure recall latency; confirm per-bank isolation; confirm no sweep/timer exists to stall. ## 9. What is unchanged The Kimi/OpenClaw/Matrix substrate is untouched: `adolf` gateway, `adolf-llm` (:8010) provider, the SSE-heartbeat/idle-watchdog fix (kb#71), the `openclaw-tools` bridge, `kanboard`/`marketplace` MCP servers, Matrix allow-list and E2EE. Only the memory backend and its two integration surfaces change. ## 10. LLM provider strategy — per-stage routing (kb#84, kb#88) Hindsight calls an LLM in several distinct stages, and they have very different cost/quality profiles. Hindsight supports a **separate provider per stage** (`HINDSIGHT_API__LLM_{PROVIDER,BASE_URL,MODEL,API_KEY}`, falling back to the global `HINDSIGHT_API_LLM_*` when unset), so each is routed to the model that fits it. Config lives in `openai/docker-compose.yml`, `hindsight` service. | Stage | Model | Where | Why this model | |-------|-------|-------|----------------| | **RETAIN / extraction** | Kimi — `hindsight-llm:8012` | flat-rate subscription | Pulls atomic facts + entities out of raw **Russian** conversation. Quality-critical and user-visible (bad extraction ⇒ bad memory), and **low volume** — roughly one pass per turn. Worth the strong model. | | **CONSOLIDATION** | `ollama/gemma3:4b` via LiteLLM `:4000` | local GPU (free) | Merges / dedups / reconciles stored memories. **Very high volume** — a background reconcile loop + per-retain triggers: ~900 calls / 3 h at ~190 memories. Mostly mechanical; a small model is good enough. | | **REFLECT / mental-models** | `ollama/gemma3:4b` via LiteLLM `:4000` | local GPU (free) | Periodic synthesis over the bank and the `reflect` tool. Background, frequent (~170 calls / 3 h), never on the reply path. | | Embeddings | `bge-m3` (ollama, GPU) | local | multilingual, GPU-served — see §3 / kb#84. | | Reranker | `jina-reranker-v2-base-multilingual` | local (CPU) | multilingual; kb#84. | **Why the split matters — the kb#88 incident.** Originally (H1b) the *entire* Hindsight LLM was pointed at Kimi via `HINDSIGHT_API_LLM_PROVIDER` alone. The background consolidation + mental-model jobs then hammered Kimi ~1100 calls / 3 h and **maxed the Kimi 5-hour rate window (100 %)** with *no* chat traffic at all — because Kimi is a **rate-limited flat subscription**, not a per-token API, and these jobs run continuously regardless of user activity. Moving the high-volume / low-stakes stages to a free local GPU model dropped Kimi 5 h usage 100 % → ~38 % while keeping fact extraction on the good model. **Rule of thumb:** *frequent, background, mechanical* stages (consolidation, reflect, mental-models) → **cheap local model**; *user-facing, quality-critical, low-volume* work (extraction, and the assistant's own replies) → **Kimi**. Levers if the local box gets loaded or quality is off: `HINDSIGHT_API_CONSOLIDATION_RECONCILE_INTERVAL_SECONDS` (raise to run less often), `_MAX_MEMORIES_PER_ROUND` (lower). If `gemma3:4b`'s Russian consolidation quality is too weak, `qwen3.5:9b` (same ollama box, `:11436`) is the next step up.