AgapHost/adolf/HINDSIGHT-MIGRATION.md

# Adolf memory migration — Cognee → Hindsight

**Status:** Planned · **Date:** 2026-07-13 · **Owner:** alvis

This is the authoritative design + plan for replacing Adolf's long-term memory
subsystem (**Cognee**) with **Hindsight** (Vectorize, MIT, self-hosted). It
supersedes the Cognee-specific parts of `docs/ARCHITECTURE.md` and gates 4–5 of
`docs/SPIKE-FINDINGS.md` in the OpenClaw fork (`/home/alvis/adolf`).

Memory stays integrated into Adolf **exactly the two ways Cognee was** — as a
**tool** (MCP) and as **forced hooks** (an OpenClaw memory plugin) — so no
behaviour the user sees is lost; only the backend changes.

> Scope note: this document + the kanboard **Ready** tasks (H1–H5) are the
> migration. No live service, compose file, `openclaw.json`, or plugin code has
> been changed yet — those edits are the H-tasks.

---

## 1. Why migrate

Cognee works, but its Agap deployment carries three structural costs, all
documented in the (now-retired) kanboard cognee tasks:

- **Three bespoke services** to keep the memory stack alive: `cognee` (FastAPI +
  embedded Kuzu graph + Qdrant vectors), `cognee-mcp` (a patched MCP→HTTP proxy,
  local build overlay from kb#70), and `cognee-llm` (a stateless Kimi-CLI wrapper
  that exists *only* to give Cognee an LLM on the flat subscription).
- **The cognify pipeline is fragile and expensive.** "Cognify" (turning raw
  turns into a graph) is an LLM step. On the Kimi CLI it runs ~5–24 s per call
  and drains the single-seat subscription quota (SPIKE gate 5 recommended
  LiteLLM instead). The plugin's async "cognify sweep" also silently stalled
  twice (kb#69) because of OpenClaw plugin-lifecycle edge cases, so freshly told
  facts weren't retrievable cross-session until the sweep was re-armed.
- **Scoping is best-effort.** Under `ENABLE_BACKEND_ACCESS_CONTROL=False` all
  datasets share one graph/vector backend, so per-chat isolation leaks (kb#59).

## 2. What Hindsight gives us

- **One container.** `ghcr.io/vectorize-io/hindsight:latest` — REST API on
  **:8888**, web UI on **:9999**, built-in PostgreSQL (`pg0`, persisted under
  `/home/hindsight/.pg0`). It owns its own vector + graph + temporal
  representation internally (the "four-network" model), so **no external Qdrant
  or Kuzu** is needed for memory.
- **Built-in MCP server** mounted at `/mcp` on the same port — ~30 tools
  including `retain` / `recall` / `reflect`. This **removes the need for a
  separate `cognee-mcp` proxy container entirely**.
- **Retain learns on its own.** `retain` runs Hindsight's extraction/reflection
  pipeline internally (optionally `async`), so there is **no separate "cognify
  sweep" to arm, throttle, or watch** — the whole class of kb#69 bugs disappears.
- **Memory banks** are first-class isolation units, scoped by URL path
  (`/v1/{tenant}/banks/{bank_id}/…`), so per-chat / per-user scoping is real, not
  best-effort.
- **Bring-your-own LLM/embeddings** — OpenAI-compatible, Anthropic, Ollama,
  LMStudio, etc. We point it at the infra we already run (LiteLLM `:4000` and/or
  Ollama), so we can **delete `cognee-llm`** rather than port it.

Net: **3 services → 1**, plus we drop the cognify-sweep machinery.

## 3. Target architecture

```
Matrix ⇄ OpenClaw ("Adolf" gateway)
             │  provider adolf-llm (Kimi wrapper, :8010)  — unchanged
             ▼
        adolf-llm ── kimi CLI (agent)
             │
   ┌─────────┴───────────────── memory is a boundary concern ─────────────┐
   │                                                                       │
   │  (a) FORCED HOOKS — hindsight-memory OpenClaw plugin                  │
   │      before_prompt_build → recall  → inject as prependContext        │
   │      agent_end            → retain (async:true) the turn             │
   │                                                                       │
   │  (b) TOOL — Hindsight built-in MCP in openclaw.json mcp.servers       │
   │      http://hindsight:8888/mcp/adolf/  → retain/recall/reflect/…      │
   └───────────────────────────────┬───────────────────────────────────────┘
                                    ▼
                        hindsight  (ONE container)
                          :8888 REST + /mcp   ·   :9999 UI
                          built-in Postgres (pg0)
                          LLM  → per-stage routing           [resolved, §10]
                          embeddings → ollama bge-m3 (GPU)   [resolved, §10]
```

Reused infra: **LiteLLM `:4000`** and/or **Ollama** for Hindsight's model calls.
**Retired:** `cognee`, `cognee-mcp`, `cognee-llm`, Qdrant-for-cognee, Kuzu, the
`cognee-openclaw-plugin`, and all `openclaw.json` cognee references.

### 3.1 Surface (a) — forced hooks (the `hindsight-memory` plugin)

A new OpenClaw memory plugin replacing `cognee-openclaw-plugin`, modelled on the
same Honcho touchpoints Cognee used, so the plugin shape is familiar:

| Cognee plugin (`cognee-memory`)                     | Hindsight plugin (`hindsight-memory`)                         |
|-----------------------------------------------------|---------------------------------------------------------------|
| `before_prompt_build` → LLM-free graph recall inject | `before_prompt_build` → `recall` → inject `prependContext`    |
| `agent_end` → raw `/add` (no inline cognify)         | `agent_end` → `retain` (`async:true`) the user+assistant turn |
| throttled **cognify sweep** (dirty-tracker, timers)  | **removed** — retain does extraction/learning internally      |
| `cognee_recall` registered tool                      | `hindsight_recall` (+ optional `hindsight_reflect`) tool      |

- Activated via `plugins.entries.hindsight-memory` in `openclaw.json` with the
  same hook grants Cognee needed: `hooks.allowConversationAccess: true` and
  `allowPromptInjection: true` (external plugins must opt in).
- **Recall stays off the hot LLM path.** Hindsight `recall` is retrieval
  (semantic + BM25 + graph + temporal) with evidence scoring — no generative
  synthesis — so it's the direct analogue of Cognee's LLM-free `onlyContext`
  recall. The LLM-backed synthesis path is `reflect`, exposed as a deliberate
  tool, not run per-turn.
- **No freshness dial / sweep.** Retain with `async:true` returns fast and lets
  Hindsight extract/consolidate in the background; there is no plugin-owned timer
  to stall.
- Same "untrusted metadata" framing on the injected block; same cleaning of
  OpenClaw's `Conversation info (untrusted metadata):` and the memory block out
  of stored/queried text.

### 3.2 Surface (b) — tool (built-in MCP)

Replace the `cognee` entry in `openclaw.json` `mcp.servers` with:

```jsonc
mcp: {
  servers: {
    hindsight: {
      type: "http",
      url: "http://hindsight:8888/mcp/adolf/",   // bank-scoped by URL path
    },
    // openclaw-tools, kanboard, marketplace — unchanged
  },
}
```

The single bank in the path (`adolf`, or a per-chat bank) selects isolation; the
built-in MCP then exposes `retain`, `recall`, `reflect`, plus mental-model /
directive / memory-browse tools. This replaces cognee-mcp's `remember` / `recall`
/ `forget` — and gives deliberate delete via the memory-management tools instead
of the hand-patched `forget(data_id)` from kb#70.

## 4. REST / MCP API mapping

Base path: `http://hindsight:8888/v1/default` (tenant `default`). Bank id is a
**path** parameter.

| Operation        | Cognee (old)                                   | Hindsight (new)                                                        |
|------------------|------------------------------------------------|-----------------------------------------------------------------------|
| store a turn     | `POST /api/v1/add` (+ later `/cognify`)        | `POST /banks/{bank}/memories`  body `{items:[{content,context,tags,timestamp}], async:true}` |
| recall (no LLM)  | `POST /api/v1/search` `GRAPH_COMPLETION` `onlyContext:true` | `POST /banks/{bank}/memories/recall`  body `{query, budget, max_tokens, tags}` |
| deep answer (LLM)| cognee-mcp `recall` (GRAPH_COMPLETION)         | `POST /banks/{bank}/reflect`  body `{query, budget, max_tokens, response_schema?}` |
| delete an entry  | patched `forget(dataset, data_id)` (kb#70)     | memory-management endpoints / MCP (`delete`, `clear_memories`)        |
| list / inspect   | dataset status polling                         | `GET /banks/{bank}/memories/list`, `GET /banks`                       |
| create bank      | dataset created implicitly on add              | `PUT /banks/{bank}`                                                    |

Built-in MCP tools live at `http://hindsight:8888/mcp/{bank}/` (HTTP transport;
bank via URL path, `X-Bank-Id` header, or `HINDSIGHT_MCP_BANK_ID` default).

> Exact request-body field names and any auth headers must be confirmed against
> the running instance's OpenAPI (`http://localhost:8888/docs`) and the Hindsight
> configuration docs during H1 — treat the bodies above as the shape, not gospel.

## 5. Bank scoping

Mirror Cognee's per-conversation `chat_<chatId>` dataset with a per-conversation
**bank**: derive `bank_id` from OpenClaw's `chat_id` (the
`Conversation info (untrusted metadata):` block; see SPIKE gate 2), sanitized to
`chat_<slug>`. A single shared `adolf` bank is the simpler alternative if
cross-chat recall is actually wanted — decide in H3. Banks are hard isolation in
Hindsight, so per-chat is now safe (unlike Cognee's leaky datasets).

## 6. Open decisions (resolve in H1)

1. **LLM backend for retain/reflect.** SPIKE gate 5 already concluded the
   extraction workload should *not* sit on the Kimi CLI (latency + single-seat
   quota). Recommendation: point Hindsight's LLM at **LiteLLM `:4000`** (or a
   local **Ollama** model for zero marginal cost). This is why `cognee-llm` is
   deleted, not ported. Confirm Hindsight's provider env-var names on the image.
2. **Embeddings.** Prefer the local **Ollama** embedder already available
   (`nomic-embed` / `bge-m3` at `host.docker.internal:11436`) or Hindsight's
   built-in, to keep embeddings off any paid path.
3. **Storage path.** Persist `pg0` under `/mnt/ssd/dbs/hindsight/` to match the
   Agap storage layout (replaces `/mnt/ssd/dbs/cognee/`).
4. **UI exposure.** Whether to reverse-proxy the `:9999` UI (Caddy) or keep it
   internal-only.
5. **Auth.** Open by default; enable the tenant API-key extension
   (`HINDSIGHT_API_TENANT_API_KEY`, `Authorization: Bearer`) if the service is
   reachable beyond the compose network.

## 7. Data migration

Cognee's Kuzu graph is **not** portable into Hindsight's store. The memory corpus
is low-value conversational history, so **start Hindsight empty** rather than
building an exporter. Optionally replay a handful of durable facts by calling
`retain` once at cutover. The two throwaway datasets left in Cognee
(`chat_verify`, `chat_webchat`) are discarded with the stack.

## 8. Migration phases (kanboard **Ready**, project *Adolf*)

- **H1 · Deploy Hindsight service** — add the `hindsight` container to
  `openai/docker-compose.yml` (image, ports 8888/9999, `pg0` volume, LLM +
  embedding provider env → LiteLLM/Ollama), bring it up, confirm `/docs` + a
  round-trip `retain`→`recall`. Resolves §6 decisions.
- **H2 · Wire built-in MCP as an Adolf tool** — swap `mcp.servers.cognee` →
  `mcp.servers.hindsight` (`/mcp/{bank}/`) in `openclaw.json`; verify Adolf can
  call `retain`/`recall`/`reflect` as tools.
- **H3 · `hindsight-memory` OpenClaw plugin (forced hooks)** — build the plugin
  replacing `cognee-openclaw-plugin`: `before_prompt_build`→recall inject,
  `agent_end`→`retain(async)`, `hindsight_recall`/`hindsight_reflect` tools, bank
  scoping; activate in `openclaw.json`. Delete the cognify-sweep machinery.
- **H4 · Decommission Cognee** — remove `cognee`, `cognee-mcp`, `cognee-llm`
  services + volumes, the `cognee-openclaw-plugin`, and all `openclaw.json`/
  `shared-mcp.json` cognee references. Free `/mnt/ssd/dbs/cognee`.
- **H5 · End-to-end verification** — state a fact → fresh session → recalled via
  injected memory (no LLM on the recall path); measure recall latency; confirm
  per-bank isolation; confirm no sweep/timer exists to stall.

## 9. What is unchanged

The Kimi/OpenClaw/Matrix substrate is untouched: `adolf` gateway, `adolf-llm`
(:8010) provider, the SSE-heartbeat/idle-watchdog fix (kb#71), the
`openclaw-tools` bridge, `kanboard`/`marketplace` MCP servers, Matrix allow-list
and E2EE. Only the memory backend and its two integration surfaces change.

## 10. LLM provider strategy — per-stage routing (kb#84, kb#88)

Hindsight calls an LLM in several distinct stages, and they have very different
cost/quality profiles. Hindsight supports a **separate provider per stage**
(`HINDSIGHT_API_<STAGE>_LLM_{PROVIDER,BASE_URL,MODEL,API_KEY}`, falling back to
the global `HINDSIGHT_API_LLM_*` when unset), so each is routed to the model that
fits it. Config lives in `openai/docker-compose.yml`, `hindsight` service.

| Stage | Model | Where | Why this model |
|-------|-------|-------|----------------|
| **RETAIN / extraction** | Kimi — `hindsight-llm:8012` | flat-rate subscription | Pulls atomic facts + entities out of raw **Russian** conversation. Quality-critical and user-visible (bad extraction ⇒ bad memory), and **low volume** — roughly one pass per turn. Worth the strong model. |
| **CONSOLIDATION** | `ollama/gemma3:4b` via LiteLLM `:4000` | local GPU (free) | Merges / dedups / reconciles stored memories. **Very high volume** — a background reconcile loop + per-retain triggers: ~900 calls / 3 h at ~190 memories. Mostly mechanical; a small model is good enough. |
| **REFLECT / mental-models** | `ollama/gemma3:4b` via LiteLLM `:4000` | local GPU (free) | Periodic synthesis over the bank and the `reflect` tool. Background, frequent (~170 calls / 3 h), never on the reply path. |
| Embeddings | `bge-m3` (ollama, GPU) | local | multilingual, GPU-served — see §3 / kb#84. |
| Reranker | `jina-reranker-v2-base-multilingual` | local (CPU) | multilingual; kb#84. |

**Why the split matters — the kb#88 incident.** Originally (H1b) the *entire*
Hindsight LLM was pointed at Kimi via `HINDSIGHT_API_LLM_PROVIDER` alone. The
background consolidation + mental-model jobs then hammered Kimi ~1100 calls / 3 h
and **maxed the Kimi 5-hour rate window (100 %)** with *no* chat traffic at all —
because Kimi is a **rate-limited flat subscription**, not a per-token API, and
these jobs run continuously regardless of user activity. Moving the
high-volume / low-stakes stages to a free local GPU model dropped Kimi 5 h usage
100 % → ~38 % while keeping fact extraction on the good model.

**Rule of thumb:** *frequent, background, mechanical* stages (consolidation,
reflect, mental-models) → **cheap local model**; *user-facing, quality-critical,
low-volume* work (extraction, and the assistant's own replies) → **Kimi**.

Levers if the local box gets loaded or quality is off:
`HINDSIGHT_API_CONSOLIDATION_RECONCILE_INTERVAL_SECONDS` (raise to run less
often), `_MAX_MEMORIES_PER_ROUND` (lower). If `gemma3:4b`'s Russian consolidation
quality is too weak, `qwen3.5:9b` (same ollama box, `:11436`) is the next step up.