AgapHost/adolf/HINDSIGHT-MIGRATION.md

Adolf memory migration — Cognee → Hindsight

Status: Planned · Date: 2026-07-13 · Owner: alvis

This is the authoritative design + plan for replacing Adolf's long-term memory subsystem (Cognee) with Hindsight (Vectorize, MIT, self-hosted). It supersedes the Cognee-specific parts of docs/ARCHITECTURE.md and gates 4–5 of docs/SPIKE-FINDINGS.md in the OpenClaw fork (/home/alvis/adolf).

Memory stays integrated into Adolf exactly the two ways Cognee was — as a tool (MCP) and as forced hooks (an OpenClaw memory plugin) — so no behaviour the user sees is lost; only the backend changes.

Scope note: this document + the kanboard Ready tasks (H1–H5) are the migration. No live service, compose file, openclaw.json, or plugin code has been changed yet — those edits are the H-tasks.

---

1. Why migrate

Cognee works, but its Agap deployment carries three structural costs, all documented in the (now-retired) kanboard cognee tasks:

• Three bespoke services to keep the memory stack alive: cognee (FastAPI + embedded Kuzu graph + Qdrant vectors), cognee-mcp (a patched MCP→HTTP proxy, local build overlay from kb#70), and cognee-llm (a stateless Kimi-CLI wrapper that exists only to give Cognee an LLM on the flat subscription).
• The cognify pipeline is fragile and expensive. "Cognify" (turning raw turns into a graph) is an LLM step. On the Kimi CLI it runs ~5–24 s per call and drains the single-seat subscription quota (SPIKE gate 5 recommended LiteLLM instead). The plugin's async "cognify sweep" also silently stalled twice (kb#69) because of OpenClaw plugin-lifecycle edge cases, so freshly told facts weren't retrievable cross-session until the sweep was re-armed.
• Scoping is best-effort. Under ENABLE_BACKEND_ACCESS_CONTROL=False all datasets share one graph/vector backend, so per-chat isolation leaks (kb#59).

2. What Hindsight gives us

• One container. ghcr.io/vectorize-io/hindsight:latest — REST API on :8888, web UI on :9999, built-in PostgreSQL (pg0, persisted under /home/hindsight/.pg0). It owns its own vector + graph + temporal representation internally (the "four-network" model), so no external Qdrant or Kuzu is needed for memory.
• Built-in MCP server mounted at /mcp on the same port — ~30 tools including retain / recall / reflect. This removes the need for a separate cognee-mcp proxy container entirely.
• Retain learns on its own. retain runs Hindsight's extraction/reflection pipeline internally (optionally async), so there is no separate "cognify sweep" to arm, throttle, or watch — the whole class of kb#69 bugs disappears.
• Memory banks are first-class isolation units, scoped by URL path (/v1/{tenant}/banks/{bank_id}/…), so per-chat / per-user scoping is real, not best-effort.
• Bring-your-own LLM/embeddings — OpenAI-compatible, Anthropic, Ollama, LMStudio, etc. We point it at the infra we already run (LiteLLM :4000 and/or Ollama), so we can delete cognee-llm rather than port it.

Net: 3 services → 1, plus we drop the cognify-sweep machinery.

3. Target architecture

Matrix ⇄ OpenClaw ("Adolf" gateway)
             │  provider adolf-llm (Kimi wrapper, :8010)  — unchanged
             ▼
        adolf-llm ── kimi CLI (agent)
             │
   ┌─────────┴───────────────── memory is a boundary concern ─────────────┐
   │                                                                       │
   │  (a) FORCED HOOKS — hindsight-memory OpenClaw plugin                  │
   │      before_prompt_build → recall  → inject as prependContext        │
   │      agent_end            → retain (async:true) the turn             │
   │                                                                       │
   │  (b) TOOL — Hindsight built-in MCP in openclaw.json mcp.servers       │
   │      http://hindsight:8888/mcp/adolf/  → retain/recall/reflect/…      │
   └───────────────────────────────┬───────────────────────────────────────┘
                                    ▼
                        hindsight  (ONE container)
                          :8888 REST + /mcp   ·   :9999 UI
                          built-in Postgres (pg0)
                          LLM  → per-stage routing           [resolved, §10]
                          embeddings → ollama bge-m3 (GPU)   [resolved, §10]

Reused infra: LiteLLM :4000 and/or Ollama for Hindsight's model calls. Retired: cognee, cognee-mcp, cognee-llm, Qdrant-for-cognee, Kuzu, the cognee-openclaw-plugin, and all openclaw.json cognee references.

3.1 Surface (a) — forced hooks (the hindsight-memory plugin)

A new OpenClaw memory plugin replacing cognee-openclaw-plugin, modelled on the same Honcho touchpoints Cognee used, so the plugin shape is familiar:

Cognee plugin (cognee-memory)	Hindsight plugin (hindsight-memory)
before_prompt_build → LLM-free graph recall inject	before_prompt_build → recall → inject prependContext
agent_end → raw /add (no inline cognify)	agent_end → retain (async:true) the user+assistant turn
throttled cognify sweep (dirty-tracker, timers)	removed — retain does extraction/learning internally
cognee_recall registered tool	hindsight_recall (+ optional hindsight_reflect) tool

• Activated via plugins.entries.hindsight-memory in openclaw.json with the same hook grants Cognee needed: hooks.allowConversationAccess: true and allowPromptInjection: true (external plugins must opt in).
• Recall stays off the hot LLM path. Hindsight recall is retrieval (semantic + BM25 + graph + temporal) with evidence scoring — no generative synthesis — so it's the direct analogue of Cognee's LLM-free onlyContext recall. The LLM-backed synthesis path is reflect, exposed as a deliberate tool, not run per-turn.
• No freshness dial / sweep. Retain with async:true returns fast and lets Hindsight extract/consolidate in the background; there is no plugin-owned timer to stall.
• Same "untrusted metadata" framing on the injected block; same cleaning of OpenClaw's Conversation info (untrusted metadata): and the memory block out of stored/queried text.

3.2 Surface (b) — tool (built-in MCP)

Replace the cognee entry in openclaw.json mcp.servers with:

mcp: {
  servers: {
    hindsight: {
      type: "http",
      url: "http://hindsight:8888/mcp/adolf/",   // bank-scoped by URL path
    },
    // openclaw-tools, kanboard, marketplace — unchanged
  },
}

The single bank in the path (adolf, or a per-chat bank) selects isolation; the built-in MCP then exposes retain, recall, reflect, plus mental-model / directive / memory-browse tools. This replaces cognee-mcp's remember / recall / forget — and gives deliberate delete via the memory-management tools instead of the hand-patched forget(data_id) from kb#70.

4. REST / MCP API mapping

Base path: http://hindsight:8888/v1/default (tenant default). Bank id is a path parameter.

Operation	Cognee (old)	Hindsight (new)
store a turn	POST /api/v1/add (+ later /cognify)	POST /banks/{bank}/memories body {items:[{content,context,tags,timestamp}], async:true}
recall (no LLM)	POST /api/v1/search GRAPH_COMPLETION onlyContext:true	POST /banks/{bank}/memories/recall body {query, budget, max_tokens, tags}
deep answer (LLM)	cognee-mcp recall (GRAPH_COMPLETION)	POST /banks/{bank}/reflect body {query, budget, max_tokens, response_schema?}
delete an entry	patched forget(dataset, data_id) (kb#70)	memory-management endpoints / MCP (delete, clear_memories)
list / inspect	dataset status polling	GET /banks/{bank}/memories/list, GET /banks
create bank	dataset created implicitly on add	PUT /banks/{bank}

Built-in MCP tools live at http://hindsight:8888/mcp/{bank}/ (HTTP transport; bank via URL path, X-Bank-Id header, or HINDSIGHT_MCP_BANK_ID default).

Exact request-body field names and any auth headers must be confirmed against the running instance's OpenAPI (http://localhost:8888/docs) and the Hindsight configuration docs during H1 — treat the bodies above as the shape, not gospel.

5. Bank scoping

Mirror Cognee's per-conversation chat_<chatId> dataset with a per-conversation bank: derive bank_id from OpenClaw's chat_id (the Conversation info (untrusted metadata): block; see SPIKE gate 2), sanitized to chat_<slug>. A single shared adolf bank is the simpler alternative if cross-chat recall is actually wanted — decide in H3. Banks are hard isolation in Hindsight, so per-chat is now safe (unlike Cognee's leaky datasets).

6. Open decisions (resolve in H1)

1. LLM backend for retain/reflect. SPIKE gate 5 already concluded the extraction workload should not sit on the Kimi CLI (latency + single-seat quota). Recommendation: point Hindsight's LLM at LiteLLM :4000 (or a local Ollama model for zero marginal cost). This is why cognee-llm is deleted, not ported. Confirm Hindsight's provider env-var names on the image.
2. Embeddings. Prefer the local Ollama embedder already available (nomic-embed / bge-m3 at host.docker.internal:11436) or Hindsight's built-in, to keep embeddings off any paid path.
3. Storage path. Persist pg0 under /mnt/ssd/dbs/hindsight/ to match the Agap storage layout (replaces /mnt/ssd/dbs/cognee/).
4. UI exposure. Whether to reverse-proxy the :9999 UI (Caddy) or keep it internal-only.
5. Auth. Open by default; enable the tenant API-key extension (HINDSIGHT_API_TENANT_API_KEY, Authorization: Bearer) if the service is reachable beyond the compose network.

7. Data migration

Cognee's Kuzu graph is not portable into Hindsight's store. The memory corpus is low-value conversational history, so start Hindsight empty rather than building an exporter. Optionally replay a handful of durable facts by calling retain once at cutover. The two throwaway datasets left in Cognee (chat_verify, chat_webchat) are discarded with the stack.

8. Migration phases (kanboard Ready, project Adolf)

• H1 · Deploy Hindsight service — add the hindsight container to openai/docker-compose.yml (image, ports 8888/9999, pg0 volume, LLM + embedding provider env → LiteLLM/Ollama), bring it up, confirm /docs + a round-trip retain→recall. Resolves §6 decisions.
• H2 · Wire built-in MCP as an Adolf tool — swap mcp.servers.cognee → mcp.servers.hindsight (/mcp/{bank}/) in openclaw.json; verify Adolf can call retain/recall/reflect as tools.
• H3 · hindsight-memory OpenClaw plugin (forced hooks) — build the plugin replacing cognee-openclaw-plugin: before_prompt_build→recall inject, agent_end→retain(async), hindsight_recall/hindsight_reflect tools, bank scoping; activate in openclaw.json. Delete the cognify-sweep machinery.
• H4 · Decommission Cognee — remove cognee, cognee-mcp, cognee-llm services + volumes, the cognee-openclaw-plugin, and all openclaw.json/ shared-mcp.json cognee references. Free /mnt/ssd/dbs/cognee.
• H5 · End-to-end verification — state a fact → fresh session → recalled via injected memory (no LLM on the recall path); measure recall latency; confirm per-bank isolation; confirm no sweep/timer exists to stall.

9. What is unchanged

The Kimi/OpenClaw/Matrix substrate is untouched: adolf gateway, adolf-llm (:8010) provider, the SSE-heartbeat/idle-watchdog fix (kb#71), the openclaw-tools bridge, kanboard/marketplace MCP servers, Matrix allow-list and E2EE. Only the memory backend and its two integration surfaces change.

10. LLM provider strategy — per-stage routing (kb#84, kb#88)

Hindsight calls an LLM in several distinct stages, and they have very different cost/quality profiles. Hindsight supports a separate provider per stage (HINDSIGHT_API_<STAGE>_LLM_{PROVIDER,BASE_URL,MODEL,API_KEY}, falling back to the global HINDSIGHT_API_LLM_* when unset), so each is routed to the model that fits it. Config lives in openai/docker-compose.yml, hindsight service.

Stage	Model	Where	Why this model
RETAIN / extraction	Kimi — hindsight-llm:8012	flat-rate subscription	Pulls atomic facts + entities out of raw Russian conversation. Quality-critical and user-visible (bad extraction ⇒ bad memory), and low volume — roughly one pass per turn. Worth the strong model.
CONSOLIDATION	ollama/gemma3:4b via LiteLLM :4000	local GPU (free)	Merges / dedups / reconciles stored memories. Very high volume — a background reconcile loop + per-retain triggers: ~900 calls / 3 h at ~190 memories. Mostly mechanical; a small model is good enough.
REFLECT / mental-models	ollama/gemma3:4b via LiteLLM :4000	local GPU (free)	Periodic synthesis over the bank and the reflect tool. Background, frequent (~170 calls / 3 h), never on the reply path.
Embeddings	bge-m3 (ollama, GPU)	local	multilingual, GPU-served — see §3 / kb#84.
Reranker	jina-reranker-v2-base-multilingual	local (CPU)	multilingual; kb#84.

Why the split matters — the kb#88 incident. Originally (H1b) the entire Hindsight LLM was pointed at Kimi via HINDSIGHT_API_LLM_PROVIDER alone. The background consolidation + mental-model jobs then hammered Kimi ~1100 calls / 3 h and maxed the Kimi 5-hour rate window (100 %) with no chat traffic at all — because Kimi is a rate-limited flat subscription, not a per-token API, and these jobs run continuously regardless of user activity. Moving the high-volume / low-stakes stages to a free local GPU model dropped Kimi 5 h usage 100 % → ~38 % while keeping fact extraction on the good model.

Rule of thumb: frequent, background, mechanical stages (consolidation, reflect, mental-models) → cheap local model; user-facing, quality-critical, low-volume work (extraction, and the assistant's own replies) → Kimi.

Levers if the local box gets loaded or quality is off: HINDSIGHT_API_CONSOLIDATION_RECONCILE_INTERVAL_SECONDS (raise to run less often), _MAX_MEMORIES_PER_ROUND (lower). If gemma3:4b's Russian consolidation quality is too weak, qwen3.5:9b (same ollama box, :11436) is the next step up.