Vendor OpenClaw source as Adolf fork baseline

Adolf is a fork/vendored clone of github.com/openclaw/openclaw (v2026.6.11),
free to diverge. Tree copied sans upstream .git; upstream remote added for
future syncs. Node pinned to 24 (.nvmrc); engines already require >=22.19.
Preserves docs/ARCHITECTURE.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LeqyaxJF2nbRXJtae2kNB2

.agents/skills/openclaw-test-heap-leaks/SKILL.md

@@ -0,0 +1,109 @@
+---
+name: openclaw-test-heap-leaks
+description: Investigate OpenClaw pnpm test memory growth, Vitest OOMs, RSS spikes, and heap snapshot deltas.
+---
+
+# OpenClaw Test Heap Leaks
+
+Use this skill for test-memory investigations. Do not guess from RSS alone when heap snapshots are available. Treat snapshot-name deltas as triage evidence, not proof, until retainers or dominators support the call.
+
+For **runtime fixes** (e.g., closure leaks in long-running services like the gateway), see [Validating runtime fixes](#validating-runtime-fixes-not-test-memory) below — that uses a dedicated harness, not the test-parallel snapshot machinery.
+
+## Workflow
+
+1. Reproduce the failing shape first.
+   - Match the real entrypoint if possible. For Linux CI-style unit failures, start with:
+   - `pnpm canvas:a2ui:bundle && OPENCLAW_TEST_MEMORY_TRACE=1 OPENCLAW_TEST_HEAPSNAPSHOT_INTERVAL_MS=60000 OPENCLAW_TEST_HEAPSNAPSHOT_DIR=.tmp/heapsnap OPENCLAW_TEST_WORKERS=2 OPENCLAW_TEST_MAX_OLD_SPACE_SIZE_MB=6144 pnpm test`
+   - Keep `OPENCLAW_TEST_MEMORY_TRACE=1` enabled so the wrapper prints per-file RSS summaries alongside the snapshots.
+   - If the report is about a specific shard or worker budget, preserve that shape.
+   - Before you analyze snapshots, identify the real lane names from `[test-parallel] start ...` lines or `pnpm test --plan`. Do not assume a single `unit-fast` lane; local plans often split into `unit-fast-batch-*`.
+
+2. Wait for repeated snapshots before concluding anything.
+   - Take at least two intervals from the same lane.
+   - Compare snapshots from the same PID inside the real lane directory such as `.tmp/heapsnap/unit-fast-batch-2/`.
+   - Use `.agents/skills/openclaw-test-heap-leaks/scripts/heapsnapshot-delta.mjs` to compare either two files directly or the earliest/latest pair per PID in one lane directory.
+   - If the helper suggests transformed-module retention, confirm the top entries in DevTools retainers/dominators before calling it solved.
+
+3. Classify the growth before choosing a fix.
+   - If growth is dominated by Vite/Vitest transformed source strings, `Module`, `system / Context`, bytecode, descriptor arrays, or property maps, treat it as likely retained module graph growth in long-lived workers.
+   - If growth is dominated by app objects, caches, buffers, server handles, timers, mock state, sqlite state, or similar runtime objects, treat it as a likely cleanup or lifecycle leak.
+   - If the names are ambiguous, stop short of a confident label and inspect retainers/dominators in DevTools for the top deltas.
+
+4. Fix the right layer.
+   - For likely retained transformed-module growth in shared workers:
+   - Prefer timing and hotspot-driven scheduling fixes first. Check whether the file is already represented in `test/fixtures/test-timings.unit.json` and whether `scripts/test-update-memory-hotspots.mjs` should refresh the measured hotspot manifest before hand-editing behavior overrides.
+   - Move hotspot files out of the real shared lane by updating `test/fixtures/test-parallel.behavior.json` only when timing-driven peeling is insufficient.
+   - Prefer `singletonIsolated` for files that are safe alone but inflate shared worker heaps.
+   - If the file should already have been peeled out by timings but is absent from `test/fixtures/test-timings.unit.json`, call that out explicitly. Missing timings are a scheduling blind spot.
+   - For real leaks:
+   - Patch the implicated test or runtime cleanup path.
+   - Look for missing `afterEach`/`afterAll`, module-reset gaps, retained global state, unreleased DB handles, or listeners/timers that survive the file.
+
+5. Verify with the most direct proof.
+   - Re-run the targeted lane or file with heap snapshots enabled if the suite still finishes in reasonable time.
+   - If snapshot overhead pushes tests over Vitest timeouts, fall back to the same lane without snapshots and confirm the RSS trend or OOM is reduced.
+   - For wrapper-only changes, at minimum verify the expected lanes start and the snapshot files are written.
+
+## Heuristics
+
+- Do not call everything a leak. In this repo, large `unit-fast` or `unit-fast-batch-*` growth can be a worker-lifetime problem rather than an application object leak.
+- `scripts/test-parallel.mjs` and `scripts/test-parallel-memory.mjs` are the primary control points for wrapper diagnostics.
+- The lane names printed by `[test-parallel] start ...` and `[test-parallel][mem] summary ...` tell you where to focus.
+- When one or two files account for most of the delta and they are missing from timings, reducing impact by isolating them is usually the first pragmatic fix.
+- When the same retained object families grow across multiple intervals in the same worker PID, trust the snapshots over intuition, then confirm ambiguous calls with retainer evidence.
+
+## Snapshot Comparison
+
+- Direct comparison:
+  - `node .agents/skills/openclaw-test-heap-leaks/scripts/heapsnapshot-delta.mjs before.heapsnapshot after.heapsnapshot`
+- Auto-select earliest/latest snapshots per PID within one lane:
+  - `node .agents/skills/openclaw-test-heap-leaks/scripts/heapsnapshot-delta.mjs --lane-dir .tmp/heapsnap/unit-fast-batch-2`
+- Useful flags:
+  - `--top 40`
+  - `--min-kb 32`
+  - `--pid 16133`
+
+Read the top positive deltas first. Large positive growth in module-transform artifacts suggests lane isolation; large positive growth in runtime objects suggests a real leak. If the names alone do not settle it, open the same snapshot pair in DevTools and inspect retainers/dominators for the top rows before declaring root cause.
+
+## Validating runtime fixes (not test-memory)
+
+The workflow above is for diagnosing Vitest worker memory growth. For
+validating that a runtime/closure fix actually releases captured state, use the
+dedicated harness:
+
+- `pnpm leak:embedded-run` — runs `scripts/embedded-run-abort-leak.ts`. Loops N
+  aborted runs in a function-shaped scope mimicking `runEmbeddedAttempt`,
+  writes heap snapshots, and reports a PASS/FAIL verdict on retention growth
+  using `FinalizationRegistry` for tracked-instance counting plus RSS delta.
+
+Modes:
+
+- `closure-extracted` (default) — production fix shape (helper at module scope).
+- `closure-inline` — pre-fix shape (closure inside the runner scope). Use as a
+  sensitivity check: if it passes you've broken the harness, not fixed a bug.
+- `synthetic-leak` — deliberately retains via a module-level bucket. Use to
+  confirm the harness can detect leaks before trusting a PASS on a real fix.
+
+Snapshots land in `.tmp/embedded-run-abort-leak/`. Diff with the same script
+as above:
+
+```
+node .agents/skills/openclaw-test-heap-leaks/scripts/heapsnapshot-delta.mjs \
+  .tmp/embedded-run-abort-leak/baseline-*.heapsnapshot \
+  .tmp/embedded-run-abort-leak/batch-N-*.heapsnapshot --top 30
+```
+
+When fixing a different runtime leak, add a new harness alongside this one
+rather than retrofitting it. The fixture function should mimic the lexical
+scope of the function where the leak lives, not be a generic abort-loop.
+
+## Output Expectations
+
+When using this skill, report:
+
+- The exact reproduce command.
+- Which lane and PID were compared.
+- The dominant retained object families from the snapshot delta.
+- Whether the issue is a likely real leak or likely shared-worker retained module growth, plus whether retainers/dominators confirmed it.
+- The concrete fix or impact-reduction patch.
+- What you verified, and what snapshot overhead prevented you from verifying.

.agents/skills/openclaw-test-heap-leaks/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Test Heap Leaks"
+  short_description: "Investigate test OOMs with heap snapshots"
+  default_prompt: "Use $openclaw-test-heap-leaks to investigate test memory growth with heap snapshots and reduce its impact."

.agents/skills/openclaw-test-heap-leaks/scripts/heapsnapshot-delta.mjs

@@ -0,0 +1,556 @@
+#!/usr/bin/env node
+/**
+ * Heap snapshot diff utility for OpenClaw test memory leak investigations.
+ */
+
+import fs from "node:fs";
+import path from "node:path";
+
+function printUsage() {
+  console.error(
+    "Usage: node heapsnapshot-delta.mjs <before.heapsnapshot> <after.heapsnapshot> [--top N] [--min-kb N]",
+  );
+  console.error(
+    "   or: node heapsnapshot-delta.mjs --lane-dir <dir> [--pid PID] [--top N] [--min-kb N]",
+  );
+}
+
+function fail(message) {
+  console.error(message);
+  process.exit(1);
+}
+
+function parseArgs(argv) {
+  const options = {
+    top: 30,
+    minKb: 64,
+    laneDir: null,
+    pid: null,
+    files: [],
+  };
+
+  for (let index = 0; index < argv.length; index += 1) {
+    const arg = argv[index];
+    if (arg === "--top") {
+      options.top = Number.parseInt(argv[index + 1] ?? "", 10);
+      index += 1;
+      continue;
+    }
+    if (arg === "--min-kb") {
+      options.minKb = Number.parseInt(argv[index + 1] ?? "", 10);
+      index += 1;
+      continue;
+    }
+    if (arg === "--lane-dir") {
+      options.laneDir = argv[index + 1] ?? null;
+      index += 1;
+      continue;
+    }
+    if (arg === "--pid") {
+      options.pid = Number.parseInt(argv[index + 1] ?? "", 10);
+      index += 1;
+      continue;
+    }
+    options.files.push(arg);
+  }
+
+  if (!Number.isFinite(options.top) || options.top <= 0) {
+    fail("--top must be a positive integer");
+  }
+  if (!Number.isFinite(options.minKb) || options.minKb < 0) {
+    fail("--min-kb must be a non-negative integer");
+  }
+  if (options.pid !== null && (!Number.isInteger(options.pid) || options.pid <= 0)) {
+    fail("--pid must be a positive integer");
+  }
+
+  return options;
+}
+
+class JsonStreamScanner {
+  constructor(filePath) {
+    this.stream = fs.createReadStream(filePath, {
+      encoding: "utf8",
+      highWaterMark: 1024 * 1024,
+    });
+    this.iterator = this.stream[Symbol.asyncIterator]();
+    this.buffer = "";
+    this.offset = 0;
+    this.done = false;
+  }
+
+  compactBuffer() {
+    if (this.offset > 65536) {
+      this.buffer = this.buffer.slice(this.offset);
+      this.offset = 0;
+    }
+  }
+
+  async ensureAvailable(count = 1) {
+    while (!this.done && this.buffer.length - this.offset < count) {
+      const next = await this.iterator.next();
+      if (next.done) {
+        this.done = true;
+        break;
+      }
+      this.buffer += next.value;
+    }
+  }
+
+  async peek() {
+    await this.ensureAvailable(1);
+    return this.buffer[this.offset] ?? null;
+  }
+
+  async next() {
+    await this.ensureAvailable(1);
+    if (this.offset >= this.buffer.length) {
+      return null;
+    }
+    const char = this.buffer[this.offset];
+    this.offset += 1;
+    this.compactBuffer();
+    return char;
+  }
+
+  async skipWhitespace() {
+    while (true) {
+      const char = await this.peek();
+      if (char === null || !/\s/u.test(char)) {
+        return;
+      }
+      await this.next();
+    }
+  }
+
+  async expectChar(expected) {
+    const char = await this.next();
+    if (char !== expected) {
+      fail(`Expected ${expected} but found ${char ?? "<eof>"}`);
+    }
+  }
+
+  async find(sequence) {
+    let matched = 0;
+    while (true) {
+      const char = await this.next();
+      if (char === null) {
+        fail(`Could not find ${sequence}`);
+      }
+      if (char === sequence[matched]) {
+        matched += 1;
+        if (matched === sequence.length) {
+          return;
+        }
+        continue;
+      }
+      matched = char === sequence[0] ? 1 : 0;
+      if (matched === sequence.length) {
+        return;
+      }
+    }
+  }
+
+  async readBalancedObject() {
+    const start = await this.next();
+    if (start !== "{") {
+      fail(`Expected { but found ${start ?? "<eof>"}`);
+    }
+    let text = "{";
+    let depth = 1;
+    let inString = false;
+    let escaped = false;
+    while (depth > 0) {
+      const char = await this.next();
+      if (char === null) {
+        fail("Unexpected EOF while reading JSON object");
+      }
+      text += char;
+      if (inString) {
+        if (escaped) {
+          escaped = false;
+        } else if (char === "\\") {
+          escaped = true;
+        } else if (char === '"') {
+          inString = false;
+        }
+        continue;
+      }
+      if (char === '"') {
+        inString = true;
+      } else if (char === "{") {
+        depth += 1;
+      } else if (char === "}") {
+        depth -= 1;
+      }
+    }
+    return text;
+  }
+
+  async parseNumberArray(onValue) {
+    await this.skipWhitespace();
+    await this.expectChar("[");
+    await this.skipWhitespace();
+    if ((await this.peek()) === "]") {
+      await this.next();
+      return;
+    }
+
+    let token = "";
+    let index = 0;
+    const flush = () => {
+      if (token.length === 0) {
+        fail("Unexpected empty number token");
+      }
+      const value = Number.parseInt(token, 10);
+      if (!Number.isFinite(value)) {
+        fail(`Invalid numeric token: ${token}`);
+      }
+      onValue(value, index);
+      index += 1;
+      token = "";
+    };
+
+    while (true) {
+      const char = await this.next();
+      if (char === null) {
+        fail("Unexpected EOF while reading number array");
+      }
+      if (char === "]") {
+        flush();
+        return;
+      }
+      if (char === ",") {
+        flush();
+        continue;
+      }
+      if (/\s/u.test(char)) {
+        continue;
+      }
+      token += char;
+    }
+  }
+
+  async readJsonString() {
+    await this.expectChar('"');
+    let value = "";
+    while (true) {
+      const char = await this.next();
+      if (char === null) {
+        fail("Unexpected EOF while reading JSON string");
+      }
+      if (char === '"') {
+        return value;
+      }
+      if (char !== "\\") {
+        value += char;
+        continue;
+      }
+      const escaped = await this.next();
+      if (escaped === null) {
+        fail("Unexpected EOF while reading JSON string escape");
+      }
+      if (escaped === "u") {
+        let hex = "";
+        for (let index = 0; index < 4; index += 1) {
+          const hexChar = await this.next();
+          if (hexChar === null) {
+            fail("Unexpected EOF while reading JSON unicode escape");
+          }
+          hex += hexChar;
+        }
+        value += String.fromCharCode(Number.parseInt(hex, 16));
+        continue;
+      }
+      value +=
+        escaped === "b"
+          ? "\b"
+          : escaped === "f"
+            ? "\f"
+            : escaped === "n"
+              ? "\n"
+              : escaped === "r"
+                ? "\r"
+                : escaped === "t"
+                  ? "\t"
+                  : escaped;
+    }
+  }
+
+  async parseStringArray(onValue) {
+    await this.skipWhitespace();
+    await this.expectChar("[");
+    await this.skipWhitespace();
+    if ((await this.peek()) === "]") {
+      await this.next();
+      return;
+    }
+
+    let index = 0;
+    while (true) {
+      const value = await this.readJsonString();
+      onValue(value, index);
+      index += 1;
+      await this.skipWhitespace();
+      const separator = await this.next();
+      if (separator === "]") {
+        return;
+      }
+      if (separator !== ",") {
+        fail(`Expected , or ] but found ${separator ?? "<eof>"}`);
+      }
+      await this.skipWhitespace();
+    }
+  }
+}
+
+function parseHeapFilename(filePath) {
+  const base = path.basename(filePath);
+  const match = base.match(
+    /^Heap\.(?<stamp>\d{8}\.\d{6})\.(?<pid>\d+)\.0\.(?<seq>\d+)\.heapsnapshot$/u,
+  );
+  if (!match?.groups) {
+    return null;
+  }
+  return {
+    filePath,
+    pid: Number.parseInt(match.groups.pid, 10),
+    stamp: match.groups.stamp,
+    sequence: Number.parseInt(match.groups.seq, 10),
+  };
+}
+
+function resolvePair(options) {
+  if (options.laneDir) {
+    const entries = fs
+      .readdirSync(options.laneDir)
+      .map((name) => parseHeapFilename(path.join(options.laneDir, name)))
+      .filter((entry) => entry !== null)
+      .filter((entry) => options.pid === null || entry.pid === options.pid)
+      .toSorted((left, right) => {
+        if (left.pid !== right.pid) {
+          return left.pid - right.pid;
+        }
+        if (left.stamp !== right.stamp) {
+          return left.stamp.localeCompare(right.stamp);
+        }
+        return left.sequence - right.sequence;
+      });
+
+    if (entries.length === 0) {
+      fail(`No matching heap snapshots found in ${options.laneDir}`);
+    }
+
+    const groups = new Map();
+    for (const entry of entries) {
+      const group = groups.get(entry.pid) ?? [];
+      group.push(entry);
+      groups.set(entry.pid, group);
+    }
+
+    const candidates = Array.from(groups.values())
+      .map((group) => ({
+        pid: group[0].pid,
+        before: group[0],
+        after: group.at(-1),
+        count: group.length,
+      }))
+      .filter((entry) => entry.count >= 2);
+
+    if (candidates.length === 0) {
+      fail(`Need at least two snapshots for one PID in ${options.laneDir}`);
+    }
+
+    const chosen =
+      options.pid !== null
+        ? (candidates.find((entry) => entry.pid === options.pid) ?? null)
+        : candidates.toSorted((left, right) => right.count - left.count || left.pid - right.pid)[0];
+
+    if (!chosen) {
+      fail(`No PID with at least two snapshots matched in ${options.laneDir}`);
+    }
+
+    return {
+      before: chosen.before.filePath,
+      after: chosen.after.filePath,
+      pid: chosen.pid,
+      snapshotCount: chosen.count,
+    };
+  }
+
+  if (options.files.length !== 2) {
+    printUsage();
+    process.exit(1);
+  }
+
+  return {
+    before: options.files[0],
+    after: options.files[1],
+    pid: null,
+    snapshotCount: 2,
+  };
+}
+
+async function parseSnapshotMeta(scanner) {
+  await scanner.find('"snapshot":');
+  await scanner.skipWhitespace();
+  const metaObjectText = await scanner.readBalancedObject();
+  const parsed = JSON.parse(metaObjectText);
+  return parsed?.meta ?? null;
+}
+
+async function buildSummary(filePath) {
+  const scanner = new JsonStreamScanner(filePath);
+  const meta = await parseSnapshotMeta(scanner);
+  if (!meta) {
+    fail(`Invalid heap snapshot: ${filePath}`);
+  }
+
+  const nodeFieldCount = meta.node_fields.length;
+  const typeNames = meta.node_types[0];
+  const typeIndex = meta.node_fields.indexOf("type");
+  const nameIndex = meta.node_fields.indexOf("name");
+  const selfSizeIndex = meta.node_fields.indexOf("self_size");
+  if (typeIndex === -1 || nameIndex === -1 || selfSizeIndex === -1) {
+    fail(`Unsupported heap snapshot schema: ${filePath}`);
+  }
+
+  const summaryByIndex = new Map();
+  let nodeCount = 0;
+  let currentTypeId = 0;
+  let currentNameId = 0;
+  let currentSelfSize = 0;
+  await scanner.find('"nodes":');
+  await scanner.parseNumberArray((value, index) => {
+    const fieldIndex = index % nodeFieldCount;
+    if (fieldIndex === typeIndex) {
+      currentTypeId = value;
+      return;
+    }
+    if (fieldIndex === nameIndex) {
+      currentNameId = value;
+      return;
+    }
+    if (fieldIndex === selfSizeIndex) {
+      currentSelfSize = value;
+    }
+    if (fieldIndex !== nodeFieldCount - 1) {
+      return;
+    }
+    const key = `${currentTypeId}\t${currentNameId}`;
+    const current = summaryByIndex.get(key) ?? {
+      typeId: currentTypeId,
+      nameId: currentNameId,
+      selfSize: 0,
+      count: 0,
+    };
+    current.selfSize += currentSelfSize;
+    current.count += 1;
+    summaryByIndex.set(key, current);
+    nodeCount += 1;
+  });
+
+  const requiredNameIds = new Set(
+    Array.from(summaryByIndex.values(), (entry) => entry.nameId).filter((value) => value >= 0),
+  );
+  const nameStrings = new Map();
+  await scanner.find('"strings":');
+  await scanner.parseStringArray((value, index) => {
+    if (requiredNameIds.has(index)) {
+      nameStrings.set(index, value);
+    }
+  });
+
+  const summary = new Map();
+  for (const entry of summaryByIndex.values()) {
+    const key = `${typeNames[entry.typeId] ?? "unknown"}\t${nameStrings.get(entry.nameId) ?? ""}`;
+    summary.set(key, {
+      type: typeNames[entry.typeId] ?? "unknown",
+      name: nameStrings.get(entry.nameId) ?? "",
+      selfSize: entry.selfSize,
+      count: entry.count,
+    });
+  }
+
+  return {
+    nodeCount,
+    summary,
+  };
+}
+
+function formatBytes(bytes) {
+  if (Math.abs(bytes) >= 1024 ** 2) {
+    return `${(bytes / 1024 ** 2).toFixed(2)} MiB`;
+  }
+  if (Math.abs(bytes) >= 1024) {
+    return `${(bytes / 1024).toFixed(1)} KiB`;
+  }
+  return `${bytes} B`;
+}
+
+function formatDelta(bytes) {
+  return `${bytes >= 0 ? "+" : "-"}${formatBytes(Math.abs(bytes))}`;
+}
+
+function truncate(text, maxLength) {
+  return text.length <= maxLength ? text : `${text.slice(0, maxLength - 1)}…`;
+}
+
+async function main() {
+  const options = parseArgs(process.argv.slice(2));
+  const pair = resolvePair(options);
+  const before = await buildSummary(pair.before);
+  const after = await buildSummary(pair.after);
+  const minBytes = options.minKb * 1024;
+
+  const rows = [];
+  for (const [key, next] of after.summary) {
+    const previous = before.summary.get(key) ?? { selfSize: 0, count: 0 };
+    const sizeDelta = next.selfSize - previous.selfSize;
+    const countDelta = next.count - previous.count;
+    if (sizeDelta < minBytes) {
+      continue;
+    }
+    rows.push({
+      type: next.type,
+      name: next.name,
+      sizeDelta,
+      countDelta,
+      afterSize: next.selfSize,
+      afterCount: next.count,
+    });
+  }
+
+  rows.sort(
+    (left, right) => right.sizeDelta - left.sizeDelta || right.countDelta - left.countDelta,
+  );
+
+  console.log(`before: ${pair.before}`);
+  console.log(`after:  ${pair.after}`);
+  if (pair.pid !== null) {
+    console.log(`pid:    ${pair.pid} (${pair.snapshotCount} snapshots found)`);
+  }
+  console.log(
+    `nodes:   ${before.nodeCount} -> ${after.nodeCount} (${after.nodeCount - before.nodeCount >= 0 ? "+" : ""}${after.nodeCount - before.nodeCount})`,
+  );
+  console.log(`filter:  top=${options.top} min=${options.minKb} KiB`);
+  console.log("");
+
+  if (rows.length === 0) {
+    console.log("No entries exceeded the minimum delta.");
+    return;
+  }
+
+  for (const row of rows.slice(0, options.top)) {
+    console.log(
+      [
+        formatDelta(row.sizeDelta).padStart(11),
+        `count ${row.countDelta >= 0 ? "+" : ""}${row.countDelta}`.padStart(10),
+        row.type.padEnd(16),
+        truncate(row.name || "(empty)", 96),
+      ].join("  "),
+    );
+  }
+}
+
+await main();