adolf/.agents/skills/crabbox/SKILL.md

---
name: crabbox
description: Use the Crabbox wrapper for OpenClaw remote validation across Linux, macOS, Windows, and WSL2, including delegated Blacksmith Testbox proof. Report the actual provider and id.
---

# Crabbox

OpenClaw agent sessions use the Crabbox wrapper by default for tests and
computationally intensive work: builds, typechecks, lint fan-out, broad gates,
CI-parity checks, secrets, hosted services, Docker/E2E/package lanes, warmed
reusable boxes, sync timing, logs/results, cache inspection, and lease cleanup.

Crabbox is the transport/orchestration surface. The actual backend can be:

- brokered AWS Crabbox: direct provider, `provider=aws`, lease ids like
  `cbx_...`, `syncDelegated=false`
- Blacksmith Testbox through Crabbox: delegated provider,
  `provider=blacksmith-testbox`, ids like `tbx_...`, `syncDelegated=true`

Blacksmith Testbox through the Crabbox wrapper is the default OpenClaw agent
backend for trusted maintainer code and heavy `pnpm` gates. The configured
Blacksmith workflow hydrates provider and agent credentials, so never sync or
run untrusted contributor/fork code there. Use secretless fork CI or
sanitized direct AWS Crabbox for untrusted source. Do not describe
Blacksmith runs as "AWS Crabbox"; report them as Testbox-through-Crabbox with
the `tbx_...` id and Actions run.

Pass `--provider aws` when the task specifically needs direct AWS Crabbox
behavior, persistent direct-provider leases, `--fresh-pr`, `--full-resync`,
environment forwarding, capture/download support, or provider comparison. Use
`--provider blacksmith-testbox` for the default OpenClaw agent path.

## First Checks

- Run from the repo root. Crabbox sync mirrors the current checkout.
- Check the wrapper and providers before remote work:

```sh
command -v crabbox
../crabbox/bin/crabbox --version
pnpm crabbox:run -- --help | sed -n '1,120p'
../crabbox/bin/crabbox desktop launch --help
../crabbox/bin/crabbox webvnc --help
```

- OpenClaw scripts prefer `../crabbox/bin/crabbox` when present. The user PATH
  shim can be stale.
- Check `.crabbox.yaml` for the provider default. Omitting `--provider`
  means Blacksmith Testbox through Crabbox for normal Linux paths; the wrapper
  selects Azure for unqualified Windows/WSL2 runs when the local Crabbox
  binary advertises Azure. Pass `--provider aws` for direct brokered AWS runs.
- The brokered AWS image is a Linux developer image in `eu-west-1`; the repo
  config pins hot `eu-west-1a/b/c` placement so Fast Snapshot Restore can apply.
  If warmup drifts well past the minute-scale path, verify image promotion,
  region/AZ placement, and FSR state before blaming OpenClaw.
- For trusted OpenClaw agent tests and computationally intensive work, use the
  repo wrapper with `--provider blacksmith-testbox` or the repo Testbox helpers.
- Treat contributor/fork source as untrusted unless a maintainer explicitly
  approves credentialed execution after review. Run untrusted source only in
  secretless fork CI or sanitized direct AWS Crabbox. For every untrusted AWS
  run, launch an installed trusted Crabbox binary from a clean trusted `main`
  checkout and fetch the remote PR with `--fresh-pr`; never execute the
  untrusted checkout's wrapper or config locally. Set
  `CRABBOX_ENV_ALLOW=CI` to replace the repo's `OPENCLAW_*`/`NODE_OPTIONS`
  allowlist, pass `--provider aws --no-hydrate`, and use a fresh temporary
  remote `HOME` on a newly warmed lease dedicated to that untrusted source.
  Unset `CRABBOX_AWS_INSTANCE_PROFILE` and fail closed unless resolved
  `aws.instanceProfile` is empty. Before install/test, use trusted absolute-path
  tools to require an IMDSv2 token, prove the IAM credentials endpoint returns
  404, and verify remote `git rev-parse HEAD` equals the full reviewed PR head
  SHA. Bind the lease to that SHA; stop and rewarm when the head changes. Do not
  inherit Tailscale: unset every `CRABBOX_TAILSCALE*` override, force
  `--network public --tailscale=false`, clear exit-node/LAN flags, and require
  `crabbox inspect` to report public networking with no Tailscale state before
  uploading any script. Execute PR code only through trusted
  `scripts/crabbox-untrusted-bootstrap.sh`, uploaded from clean `main` alongside
  `--fresh-pr`; it installs pinned Node/pnpm and rejects a changed PR
  `packageManager` pin before install. Never reuse a trusted or previously
  hydrated lease. If the broker cannot provide
  the no-role proof or no remote PR exists, use secretless fork CI. Never use
  `hydrate-github` or a credential-hydrated Testbox workflow for untrusted code.
- Cold Testbox acquisition and hydration often take about a minute. At the
  start of any task likely to change code or need tests/heavy proof, immediately
  start, after confirming the source is trusted,
  `node scripts/crabbox-wrapper.mjs warmup --provider blacksmith-testbox --keep --timing-json`
  in a background command session while inspecting and editing. Poll later,
  reuse the returned `tbx_...` with
  `--provider blacksmith-testbox --id <tbx_id>`, and stop it before handoff.
  For untrusted source, switch to a clean trusted `main` checkout and pre-warm
  with the installed binary after the empty-instance-profile check below.
  Do not warm for read-only, docs-only, or clearly trivial work that will not
  run tests or heavy commands.
- Run untrusted source only with the sanitized form below. The explicit
  allowlist prevents locally exported `OPENCLAW_*` credentials from crossing
  the SSH boundary; `--no-hydrate` and temporary `HOME` prevent auth-profile
  reuse:

```sh
env -u CRABBOX_AWS_INSTANCE_PROFILE \
  crabbox config show --json | \
  jq -e '.aws.instanceProfile == ""' >/dev/null
env -u CRABBOX_AWS_INSTANCE_PROFILE \
  -u CRABBOX_TAILSCALE \
  -u CRABBOX_TAILSCALE_AUTH_KEY \
  -u CRABBOX_TAILSCALE_AUTH_KEY_ENV \
  -u CRABBOX_TAILSCALE_EXIT_NODE \
  -u CRABBOX_TAILSCALE_EXIT_NODE_ALLOW_LAN_ACCESS \
  -u CRABBOX_TAILSCALE_HOSTNAME_TEMPLATE \
  -u CRABBOX_TAILSCALE_TAGS \
  crabbox warmup \
  --provider aws \
  --network public \
  --tailscale=false \
  --tailscale-exit-node= \
  --tailscale-exit-node-allow-lan-access=false \
  --keep \
  --timing-json
crabbox inspect --provider aws --id <cbx_id> --json | \
  jq -e '.network == "public" and .tailscale == null' >/dev/null
env -u CRABBOX_AWS_INSTANCE_PROFILE \
  CRABBOX_ENV_ALLOW=CI \
  crabbox run \
  --provider aws \
  --id <cbx_id> \
  --fresh-pr <owner/repo#number> \
  --no-hydrate \
  --timing-json \
  --script scripts/crabbox-untrusted-bootstrap.sh -- \
  <expected_head_sha> /usr/local/bin/pnpm test <path>
# After all proof:
env -u CRABBOX_AWS_INSTANCE_PROFILE \
  crabbox stop --provider aws <cbx_id>
```

- Always report the actual provider and id. `cbx_...` means AWS Crabbox;
  `tbx_...` means Blacksmith Testbox through Crabbox. If the output only says
  `blacksmith testbox list`, use `blacksmith testbox list --all` before
  concluding no box exists.
- If a warm direct-provider lease smells stale, retry with `--full-resync`
  (alias `--fresh-sync`) before replacing the lease. This resets the remote
  workdir, skips the fingerprint fast path, reseeds Git when possible, and
  uploads the checkout from scratch.
- For live/provider bugs, use the configured secret workflow before downgrading
  to mocks. Copy only the exact needed key into the remote process environment
  for that one command. Do not print it, do not sync it as a repo file, and do
  not leave it in remote shell history or logs. If no secret-safe injection path
  is available, say true live provider auth is blocked instead of silently using
  a fake key.
- Agent-run tests, including targeted edit-loop tests, default to a pre-warmed
  remote box selected by source trust. Local test execution requires an
  explicit user request or a reported remote-provider blocker.
- Do not treat inherited shell env as operator intent. In particular,
  `OPENCLAW_LOCAL_CHECK_MODE=throttled` from the local shell is not permission
  to move broad `pnpm check:changed`, `pnpm test:changed`, full `pnpm test`, or
  lint/typecheck fan-out onto the laptop.
- Only use `OPENCLAW_LOCAL_CHECK_MODE=throttled|full` when the user explicitly
  asks for local proof in the current task. If Testbox is queued or capacity is
  constrained, report the blocker; do not silently move heavy work onto the
  laptop.

## macOS And Windows Targets

Use these only when the task needs an existing non-Linux host. OpenClaw broad
Linux validation uses the repo Crabbox config unless a provider is explicitly
requested.

Native brokered Windows is available for Windows-specific proof. Prefer Azure
for Windows/WSL2 when the subscription has quota or credits and the local
Crabbox binary advertises Azure. Keep broad Linux gates on Linux/Testbox unless
the bug is Windows-specific, and only force AWS when the operator asks for the
older AWS developer image/cache path or Azure is unavailable:

```sh
pnpm crabbox:warmup -- \
  --target windows \
  --windows-mode wsl2 \
  --timing-json
```

The hydrate workflow assumes Docker should already be baked into Linux images
and only installs it as a fallback. Do not add per-run Docker installs to proof
commands unless the image probe shows Docker is actually missing.

When the user explicitly asks for brokered macOS runners, use Crabbox AWS
macOS only after confirming the deployed coordinator supports EC2 Mac host
lifecycle/image routes and the operator has AWS EC2 Mac Dedicated Host quota
and IAM. Prefer `CRABBOX_HOST_ID` for a known Crabbox-managed Dedicated Host,
or run the no-spend preflight first:

```sh
crabbox admin hosts quota --provider aws --target macos --region eu-west-1 --type mac2.metal --json
crabbox admin hosts allocate --provider aws --target macos --region eu-west-1 --type mac2.metal --dry-run --json
CRABBOX_MACOS_TYPES=all scripts/macos-host-region-preflight.sh
```

Do not silently substitute AWS macOS for normal OpenClaw Linux proof. Report
paid-host blockers as quota, IAM, coordinator deployment, or host availability
instead of falling back to local macOS.

Crabbox supports static SSH targets:

```sh
../crabbox/bin/crabbox run --provider ssh --target macos --static-host mac-studio.local -- xcodebuild test
../crabbox/bin/crabbox run --provider ssh --target windows --windows-mode normal --static-host win-dev.local -- pwsh -NoProfile -Command "dotnet test"
../crabbox/bin/crabbox run --provider ssh --target windows --windows-mode wsl2 --static-host win-dev.local -- pnpm test
```

- `target=macos` and `target=windows --windows-mode wsl2` use the POSIX SSH,
  bash, Git, rsync, and tar contract.
- Native Windows uses OpenSSH, PowerShell, Git, and tar; sync is manifest tar
  archive transfer into `static.workRoot`. Direct native Windows runs support
  `--script*`, `--env-from-profile`, `--preflight`, and PowerShell `--shell`.
- `crabbox actions hydrate/register` are Linux-only today; use plain
  `crabbox run` loops for static macOS and Windows hosts.
- Live proof needs a reachable, operator-managed SSH host. Without one, verify
  with `../crabbox/bin/crabbox run --help`, config/flag tests, and the Crabbox
  Go test suite.

## Direct Brokered AWS Backend

Use this when the task needs direct AWS Crabbox semantics rather than the
prepared Blacksmith Testbox CI environment.

Changed gate:

```sh
pnpm crabbox:run -- \
  --provider aws \
  --idle-timeout 90m \
  --ttl 240m \
  --timing-json \
  --shell -- \
  "pnpm test:changed"
```

Full suite:

```sh
pnpm crabbox:run -- \
  --provider aws \
  --idle-timeout 90m \
  --ttl 240m \
  --timing-json \
  --shell -- \
  "pnpm verify"
```

Use `pnpm verify` when you need check plus full Vitest proof. It emits
`CRABBOX_PHASE:check` and `CRABBOX_PHASE:test`, making Crabbox summaries show
which stage failed. Use plain `pnpm test` only when check proof is already
covered or intentionally skipped.

Focused rerun:

```sh
pnpm crabbox:run -- \
  --provider aws \
  --idle-timeout 90m \
  --ttl 240m \
  --timing-json \
  --shell -- \
  "pnpm test <path-or-filter>"
```

Read the JSON summary. Useful fields:

- `provider`: `aws`
- `leaseId`: `cbx_...`
- `syncDelegated`: `false`
- `commandPhases`: populated when the command prints `CRABBOX_PHASE:<name>`
- `commandMs` / `totalMs`
- `exitCode`

Crabbox should stop one-shot AWS leases automatically after the run. Verify
cleanup when a run fails, is interrupted, or the command output is unclear:

```sh
../crabbox/bin/crabbox list --provider aws
```

## Blacksmith Testbox Through Crabbox

Use this for OpenClaw maintainer broad/heavy `pnpm` gates when the prepared CI
environment is the right proof surface:

```sh
node scripts/crabbox-wrapper.mjs run \
  --provider blacksmith-testbox \
  --blacksmith-org openclaw \
  --blacksmith-workflow .github/workflows/ci-check-testbox.yml \
  --blacksmith-job check \
  --blacksmith-ref main \
  --idle-timeout 90m \
  --ttl 240m \
  --timing-json \
  -- \
  corepack pnpm check:changed
```

Read the JSON summary and the Testbox line. Useful fields:

- `provider`: `blacksmith-testbox`
- `leaseId`: `tbx_...`
- `syncDelegated`: `true`
- `syncPhases`: delegated/skipped because Blacksmith owns checkout/sync
- Actions run URL/id from the Testbox output
- `exitCode`

Use provider-backed cache volumes only for rebuildable caches, not secrets or
checkout state. On Blacksmith, Crabbox forwards them as sticky disks:

```sh
node scripts/crabbox-wrapper.mjs run \
  --provider blacksmith-testbox \
  --cache-volume pnpm-store=openclaw-node24-pnpm-lock:/tmp/openclaw-pnpm-store \
  --timing-json \
  -- \
  corepack pnpm check:changed
```

The selected provider must advertise cache-volume support. If not, omit
`--cache-volume` and rely on kept-lease caches.

`blacksmith testbox list` may hide hydrating or ready boxes. Use:

```sh
blacksmith testbox list --all
blacksmith testbox status <tbx_id>
```

## Observability Flags

Use these on debugging runs before inventing ad hoc logging:

- `--preflight`: prints run context, workspace mode, SSH target, remote user/cwd,
  and target-specific tool probes. Defaults cover `git`, `tar`, `node`, `npm`,
  `corepack`, `pnpm`, `yarn`, `bun`, `docker`, plus POSIX
  `sudo`/`apt`/`bubblewrap` and native Windows
  `powershell`/`execution_policy`/`longpaths`/`temp`/`pwsh`. Add
  `--preflight-tools node,bun,docker`, `CRABBOX_PREFLIGHT_TOOLS`, or repo
  `run.preflightTools` to replace the list. `default` expands built-ins; `none`
  prints only the workspace summary. Preflight is diagnostic only; install
  toolchains through Actions hydration, images, devcontainer/Nix/mise/asdf, or
  the run script. On `blacksmith-testbox`, this prints a delegated-unsupported
  note because the workflow owns setup.
- `CRABBOX_ENV_ALLOW=NAME,...`: forwards only listed local env vars for direct
  providers and prints `set len=N secret=true` style summaries. On
  `blacksmith-testbox`, env forwarding is unsupported; put secrets in the
  Testbox workflow instead.
- `--env-from-profile <file>` plus `--allow-env NAME`: loads simple
  `export NAME=value` / `NAME=value` lines from a local profile without
  executing it, then forwards only allowlisted names. `--allow-env` is
  repeatable and comma-separated. Profile values override ambient allowlisted
  env values for that run. Direct POSIX, WSL2, and native Windows runs are
  supported; delegated providers are not. Crabbox probes the uploaded profile
  remotely and prints redacted presence/length metadata before the command.
- `--env-helper <name>`: with `--env-from-profile` on POSIX SSH targets,
  persists `.crabbox/env/<name>` and `.crabbox/env/<name>.env` so follow-up
  commands on the same lease can run through `./.crabbox/env/<name> <command>`.
  Use only on leases you control; the profile stays until cleanup, lease reset,
  or `--full-resync`.
- `--script <file>` / `--script-stdin`: upload a local script into
  `.crabbox/scripts/` and execute it on the remote box. Shebang scripts execute
  directly on POSIX; scripts without a shebang run through `bash`. Native
  Windows uploads run through Windows PowerShell, and Crabbox appends `.ps1`
  when needed. Arguments after `--` become script args.
- `--fresh-pr owner/repo#123|URL|number`: skip dirty local sync and create a
  fresh remote checkout of the GitHub PR. Bare numbers use the current repo's
  GitHub origin. Add `--apply-local-patch` only when the current local
  `git diff --binary HEAD` should be applied on top of that PR checkout.
- `--full-resync` / `--fresh-sync`: reset a stale direct-provider workdir
  before syncing. Use after sync fingerprints look wrong, SSH times out before
  sync, or rsync watchdog output suggests it. It is redundant with
  `--fresh-pr`, incompatible with `--no-sync`, and unsupported by delegated
  providers.
- `--capture-stdout <path>` / `--capture-stderr <path>`: write remote streams to
  local files and keep binary/noisy output out of retained logs. Parent
  directories must already exist. These are direct-provider only.
- `--capture-on-fail`: on non-zero direct-provider exits, downloads
  `.crabbox/captures/*.tar.gz` with `test-results`, `playwright-report`,
  `coverage`, JUnit XML, and nearby logs. Treat as secret-bearing until reviewed.
- `--keep-on-failure`: leave a failed one-shot lease alive for live debugging
  until idle/TTL expiry. Useful on direct providers and delegated one-shots.
- `--timing-json`: final machine-readable timing. Add
  `echo CRABBOX_PHASE:install`, `CRABBOX_PHASE:test`, etc. in long shell
  commands; direct providers and Blacksmith Testbox both report them as
  `commandPhases`.

Live-provider debug template for direct AWS/Hetzner leases:

```sh
mkdir -p .crabbox/logs
pnpm crabbox:run -- --provider aws \
  --preflight \
  --allow-env OPENAI_API_KEY,OPENAI_BASE_URL \
  --timing-json \
  --capture-stdout .crabbox/logs/live-provider.stdout.log \
  --capture-stderr .crabbox/logs/live-provider.stderr.log \
  --capture-on-fail \
  --shell -- \
  "echo CRABBOX_PHASE:install; pnpm install --frozen-lockfile; echo CRABBOX_PHASE:test; pnpm test:live"
```

Do not pass `--capture-*`, `--download`, `--checksum`, `--force-sync-large`, or
`--sync-only` to delegated providers. Also do not pass `--script*`,
`--fresh-pr`, `--full-resync`, or `--env-helper` there. Crabbox rejects these
because the provider owns sync or command transport. `--keep-on-failure` is OK
for delegated one-shots when you need to inspect a failed lease.

## Efficient Bug E2E Verification

Use the smallest Crabbox lane that proves the reported user path, not just the
touched code. Aim for one after-fix E2E proof before commenting, closing, or
opening a PR for a user-visible bug.

When the user says "test in Crabbox", do not simply copy tests to the remote
box and run them there. Crabbox is for remote real-scenario proof: copy or
install OpenClaw as the user would, run the same setup/update/CLI/Gateway/API
call that failed, and capture behavior from that entrypoint. For regressions or
bug reports, prove the broken state first when feasible, then run the same
scenario after the fix.

Pick the lane by symptom:

- Docker/setup/install bug: build a package tarball and run the matching
  `scripts/e2e/*-docker.sh` or package script. This proves npm packaging,
  install paths, runtime deps, config writes, and container behavior.
- Provider/model/auth bug: prefer true live E2E. Use the configured secret
  workflow, then inject the single needed key into Crabbox if needed. Scrub
  unrelated provider env vars in the child command so interactive defaults do
  not drift to another provider. If only a dummy key is used, label the proof
  narrowly, e.g. "UI/install path only; live provider auth not exercised."
- Channel delivery bug: use the channel Docker/live lane when available; include
  setup, config, gateway start, send/receive or agent-turn proof, and redacted
  logs.
- Gateway/session/tool bug: prefer an end-to-end CLI or Gateway RPC command that
  creates real state and inspects the resulting files/API output.
- Pure parser/config bug: targeted tests may be enough, but still run a
  Crabbox command when OS, package, Docker, secrets, or service lifecycle could
  change behavior.

Efficient flow:

1. Reproduce or prove the pre-fix symptom from the real user-facing entrypoint
   when feasible. If the issue cannot be reproduced, capture the exact command
   and observed behavior instead.
2. Patch locally and run narrow tests on the pre-warmed remote box.
3. Run one Crabbox E2E command that starts from the user-facing entrypoint:
   package install, Docker setup, onboarding, channel add, gateway start, or
   agent turn as appropriate.
4. Record proof as: Testbox id, command, environment shape, redacted secret
   source, and copied success/failure output.
5. If the issue says "cannot reproduce", ask for the missing config/log fields
   that would distinguish the tested path from the reporter's path.

Keep it efficient:

- Reuse existing E2E scripts and helper assertions before writing ad hoc shell.
- Use `--script <file>` or `--script-stdin` for multi-line E2E commands instead
  of quote-heavy `--shell` strings on direct SSH providers.
- Use `--fresh-pr <pr>` when validating an upstream PR in isolation from the
  local dirty tree. Add `--apply-local-patch` only when testing a local fixup on
  top of that PR.
- Use `--full-resync` before replacing a warmed direct-provider lease when the
  remote workdir or sync fingerprint appears stale.
- For agent code tasks, reuse the pre-warmed remote box across focused tests
  and heavy proof. Use a one-shot only when a single late proof is genuinely
  the task's only remote command.
- Prefer `OPENCLAW_CURRENT_PACKAGE_TGZ` with Docker/package lanes when testing a
  candidate tarball; prefer the repo's package helper instead of direct source
  execution when the bug might be packaging/install related.
- Keep secrets redacted. It is fine to report key presence, source, and length;
  never print secret values.
- Include `--timing-json` on broad or flaky runs when command duration or sync
  behavior matters.

Before/after PR proof on delegated Testbox:

- For PRs that should prove "broken before, fixed after", compare base and PR
  on the same Testbox when practical. Fetch both refs, create detached temp
  worktrees under `/tmp`, install in each, then run the same harness twice.
- Do not checkout base/PR refs in the synced repo root. Delegated Testbox sync
  may leave the root dirty with local files; `git checkout` can abort or mix
  proof state.
- Temp harness files under `/tmp` do not resolve repo packages by default. Put
  the harness inside the worktree, or in ESM use
  `createRequire(path.join(process.cwd(), "package.json"))` before requiring
  workspace deps such as `@lydell/node-pty`.
- For full-screen TUI/CLI bugs, a PTY harness is stronger than helper-only
  assertions. Use a real PTY, wait for visible lifecycle markers, send input,
  then send control keys and assert process exit/stuck behavior.
- When validating a rebased local branch before push, remember delegated sync
  usually validates synced file content on a detached dirty checkout, not a
  remote commit object. Record the local head SHA, changed files, Testbox id,
  and final success markers; after pushing, ensure the pushed SHA has the same
  file content.
- If GitHub CI is still queued but the exact changed content passed Testbox
  `pnpm check:changed`, `pnpm check:test-types`, and the real E2E proof, it is
  reasonable to merge once required checks allow it. Note any still-running
  unrelated shards in the proof comment instead of waiting forever.

Interactive CLI/onboarding:

- For full-screen or prompt-heavy CLI flows, run the target command inside tmux
  on the Crabbox and drive it with `tmux send-keys`; capture proof with
  `tmux capture-pane`, redacted through `sed`.
- Prefer deterministic arrow navigation over search typing for Clack-style
  searchable selects. Raw `send-keys -l openai` may not trigger filtering in a
  tmux pane; inspect option order locally or on-box and send exact Down/Enter
  sequences.
- Isolate mutable state with `OPENCLAW_STATE_DIR=$(mktemp -d)`. Plugin npm
  installs live under that state dir (`npm/node_modules/...`), not under
  `OPENCLAW_CONFIG_DIR`. Verify downloads by checking the state dir, package
  lock, and installed package metadata.
- To test automatic setup installs against local package artifacts, use
  `OPENCLAW_ALLOW_PLUGIN_INSTALL_OVERRIDES=1` plus
  `OPENCLAW_PLUGIN_INSTALL_OVERRIDES='{"plugin-id":"npm-pack:/tmp/plugin.tgz"}'`.
  Pack with `npm pack`, set an isolated `OPENCLAW_STATE_DIR`, and verify the
  package under `npm/node_modules`. Overrides are test-only and must not be
  treated as official/trusted-source installs.
- For OpenAI/Codex onboarding proof, the useful markers are the UI line
  `Installed Codex plugin`, `npm/node_modules/@openclaw/codex`, and the
  package-lock entry showing the bundled `@openai/codex` dependency. A dummy
  OpenAI-shaped key can prove only UI/install behavior; it is not live auth.

## Reuse And Keepalive

Agent code tasks should pre-warm and reuse one remote box selected by source
trust for focused tests and heavy proof. One-shot runs remain appropriate for a
single late proof when early warmup was not warranted.

Reuse the lease, not stale source. Each command must sync the current checkout;
use `--no-sync` only to rerun an unchanged, already-synced tree intentionally.
Untrusted reuse still requires `CRABBOX_ENV_ALLOW=CI`,
`--no-hydrate`, and a fresh temporary remote `HOME` on every command. Reuse
only a fresh lease dedicated to the same untrusted source; never a trusted or
previously hydrated lease. Launch from the clean trusted `main` checkout and
use `--fresh-pr` plus the same reviewed-SHA check on every run. Keep
`CRABBOX_AWS_INSTANCE_PROFILE` unset for warmup, run, and cleanup. The lease is
valid only for that reviewed SHA; stop and rewarm after any head change.

If Crabbox returns a reusable id or you intentionally keep a lease:

```sh
node scripts/crabbox-wrapper.mjs run --provider <blacksmith-testbox-or-aws> --id <id-or-slug> --timing-json --shell -- "corepack pnpm test <path>"
```

Stop boxes you created before handoff:

```sh
pnpm crabbox:stop -- <id-or-slug>
blacksmith testbox stop --id <tbx_id>
```

## Interactive Desktop And WebVNC

Prefer WebVNC for human inspection because the browser portal can preload the
lease VNC password and avoids a native VNC client's copy/paste/password dance.
Use native `crabbox vnc` only when WebVNC is unavailable, the browser portal is
broken, or the user explicitly wants a local VNC client.

Common desktop flow:

```sh
../crabbox/bin/crabbox warmup --provider hetzner --desktop --browser --class standard --idle-timeout 60m --ttl 240m
../crabbox/bin/crabbox desktop launch --provider hetzner --id <cbx_id-or-slug> --browser --url https://example.com --webvnc --open --take-control
```

Useful WebVNC commands:

```sh
../crabbox/bin/crabbox webvnc --provider hetzner --id <cbx_id-or-slug> --open --take-control
../crabbox/bin/crabbox webvnc daemon start --provider hetzner --id <cbx_id-or-slug> --open --take-control
../crabbox/bin/crabbox webvnc daemon status --provider hetzner --id <cbx_id-or-slug>
../crabbox/bin/crabbox webvnc daemon stop --provider hetzner --id <cbx_id-or-slug>
../crabbox/bin/crabbox webvnc status --provider hetzner --id <cbx_id-or-slug>
../crabbox/bin/crabbox webvnc reset --provider hetzner --id <cbx_id-or-slug> --open --take-control
../crabbox/bin/crabbox desktop doctor --provider hetzner --id <cbx_id-or-slug>
../crabbox/bin/crabbox desktop click --provider hetzner --id <cbx_id-or-slug> --x 640 --y 420
../crabbox/bin/crabbox desktop paste --provider hetzner --id <cbx_id-or-slug> --text "user@example.com"
../crabbox/bin/crabbox desktop key --provider hetzner --id <cbx_id-or-slug> ctrl+l
../crabbox/bin/crabbox artifacts collect --id <cbx_id-or-slug> --all --output artifacts/<slug>
../crabbox/bin/crabbox artifacts publish --dir artifacts/<slug> --pr <number>
```

`desktop launch --webvnc --open` is usually the nicest one-shot: it starts the
browser/app inside the visible session, bridges the lease into the authenticated
WebVNC portal, and opens the portal. Keep browsers windowed for human QA; use
`--fullscreen` only for capture/video workflows.
For human handoff, include `--take-control` so the opened portal viewer gets
keyboard/mouse control automatically instead of landing as an observer.

Human handoff preflight:

- Do not assume a visible desktop or launched browser means the repo CLI/app is
  installed, built, or on the interactive terminal's `PATH`.
- Before handing WebVNC to a human tester, prove the expected command from the
  same kept lease and from a neutral directory such as `~`.
- If the handoff needs repo-local code, sync/build/link it explicitly on that
  lease. Source-tree CLIs often need build output before a symlink works.
- Prefer a real `command -v <expected-command> && <expected-command> --version`
  check over a repo-root-only `pnpm ...` command.

Generic handoff repair pattern:

```sh
../crabbox/bin/crabbox run --id <cbx_id-or-slug> --full-resync --shell -- \
  "set -euo pipefail
   pnpm install --frozen-lockfile
   pnpm build
   sudo ln -sf \"\$PWD/<cli-entry>\" /usr/local/bin/<expected-command>
   cd ~
   command -v <expected-command>
   <expected-command> --version"
```

## If Crabbox Fails

Keep the fallback narrow. First decide whether the failure is Crabbox itself,
the brokered AWS lease, Blacksmith/Testbox, repo hydration, sync, or the test
command.

Fast checks:

```sh
command -v crabbox
../crabbox/bin/crabbox --version
pnpm crabbox:run -- --help | sed -n '1,140p'
../crabbox/bin/crabbox doctor
command -v blacksmith
blacksmith --version
blacksmith testbox list
```

Common Crabbox-only failures:

- Provider missing or old CLI: use `../crabbox/bin/crabbox` from the sibling
  repo, or update/install Crabbox before retrying.
- Bad local config: inspect `.crabbox.yaml`, `crabbox config show`, and
  `crabbox whoami`; normal OpenClaw agent proof should use Blacksmith Testbox.
  Direct AWS is an explicit fallback and must use brokered auth, not raw keys.
- Slug/claim confusion: use the raw `cbx_...` / `tbx_...` id, or run one-shot
  without `--id`.
- Sync/timing bug: add `--debug --timing-json`; capture the final JSON and the
  printed Actions URL. Large sync warnings now include top source directories
  by file count and a hint to update `.crabboxignore` / `sync.exclude`; inspect
  those before reaching for `--force-sync-large`. Quiet rsync watchdogs and SSH
  timeouts now print `next_action=` hints; follow them, usually `--full-resync`
  first and a fresh lease second.
- Cleanup uncertainty: run `crabbox list --provider aws`; for explicit
  Blacksmith runs, use `blacksmith testbox list` and stop only boxes you
  created.
- Testbox queued/capacity pressure: do not retry Blacksmith repeatedly. Rerun
  once with `--provider aws` when direct AWS still proves the requested
  surface, or report the Blacksmith blocker if Testbox itself is required.

If brokered AWS cannot dispatch, sync, attach, or stop, retry once with
`--debug` and `--timing-json`:

```sh
pnpm crabbox:run -- --provider aws --debug --timing-json -- \
  pnpm test:changed
```

Full suite:

```sh
pnpm crabbox:run -- --provider aws --debug --timing-json -- \
  pnpm test
```

Auth fallback, only when `blacksmith` says auth is missing:

```sh
blacksmith auth login --non-interactive --organization openclaw
```

Raw Blacksmith footguns:

- Run from repo root. The CLI syncs the current directory.
- Save the returned `tbx_...` id in the session.
- Reuse that id for focused reruns; stop it before handoff.
- Raw commit SHAs are not reliable `warmup --ref` refs; use a branch or tag.
- Treat `blacksmith testbox list` as cleanup diagnostics, not a shared reusable
  queue.

Use Blacksmith Testbox through Crabbox by default for OpenClaw agent tests and
heavy work. If Blacksmith is down or quota-limited, do not keep probing it;
switch to direct AWS only when that backend proves the same surface, and note
the delegated-provider outage.

## Blacksmith Backend Notes

Crabbox Blacksmith backend delegates setup to:

- org: `openclaw`
- workflow: `.github/workflows/ci-check-testbox.yml`
- job: `check`
- ref: `main` unless testing a branch/tag intentionally

The hydration workflow owns checkout, Node/pnpm setup, dependency install,
secrets, ready marker, and keepalive. Crabbox owns dispatch, sync, SSH command
execution, timing, logs/results, cleanup, and cache-volume requests. Blacksmith
implements cache volumes as sticky disks.

Minimal Blacksmith-backed Crabbox run, from repo root:

```sh
pnpm crabbox:run -- --provider blacksmith-testbox --timing-json -- \
  corepack pnpm test:changed
```

Use direct Blacksmith only when Crabbox is the broken layer and you are
isolating a Crabbox bug. Prefer direct `blacksmith testbox list` for cleanup
diagnostics, not as a reusable work queue.

Important Blacksmith footguns:

- Always run from repo root. The CLI syncs the current directory.
- Raw commit SHAs are not reliable `warmup --ref` refs; use a branch or tag.
- If auth is missing and browser auth is acceptable:

```sh
blacksmith auth login --non-interactive --organization openclaw
```

## Brokered AWS Fallback

Use direct AWS when Testbox is unavailable, when the task needs direct-provider
semantics, or when an explicit backend comparison is required. The repo
`.crabbox.yaml` defaults to Blacksmith Testbox, so pass `--provider aws`.

```sh
pnpm crabbox:warmup -- --provider aws --class beast --market on-demand --idle-timeout 90m
pnpm crabbox:hydrate -- --provider aws --id <cbx_id-or-slug>
pnpm crabbox:run -- --provider aws --id <cbx_id-or-slug> --timing-json --shell -- "pnpm test:changed"
pnpm crabbox:stop -- --provider aws <cbx_id-or-slug>
```

Install/auth for owned Crabbox if needed:

```sh
brew install openclaw/tap/crabbox
crabbox login --url https://crabbox.openclaw.ai --provider aws
```

New users should self-resolve broker auth before anyone asks for AWS keys:

```sh
crabbox config show
crabbox doctor
crabbox whoami
```

- If broker auth is missing, run `crabbox login --url https://crabbox.openclaw.ai --provider aws`.
- If the CLI asks for `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, or AWS
  profile setup during normal OpenClaw validation, assume the agent selected
  the wrong path. Use brokered `crabbox login` or an existing brokered lease
  before asking the user for cloud credentials.
- Ask for AWS keys only for explicit direct-provider/account administration,
  not for normal brokered OpenClaw proof.
- Trusted automation may still use
  `printf '%s' "$CRABBOX_COORDINATOR_TOKEN" | crabbox login --url https://crabbox.openclaw.ai --provider aws --token-stdin`.

macOS config lives at:

```text
~/Library/Application Support/crabbox/config.yaml
```

It should include `broker.url`, `broker.token`, and usually `provider: aws`
for OpenClaw lanes. Let that config drive normal validation.

### Interactive Desktop / WebVNC

For human desktop demos, prefer `webvnc` over native `vnc` and keep the remote
desktop visible/windowed. Do not fullscreen the remote browser or hide the XFCE
panel/window chrome unless the explicit goal is video/capture output. After
launch, verify a screenshot shows the desktop panel plus browser title bar. If
Chrome is fullscreen, toggle it back with:

```sh
crabbox run --id <lease> --shell -- 'DISPLAY=:99 xdotool search --onlyvisible --class google-chrome windowactivate key F11'
```

## Diagnostics

```sh
crabbox status --id <id-or-slug> --wait
crabbox inspect --id <id-or-slug> --json
crabbox sync-plan
crabbox history --limit 20
crabbox history --lease <id-or-slug>
crabbox attach <run_id>
crabbox events <run_id> --json
crabbox logs <run_id>
crabbox results <run_id>
crabbox cache stats --id <id-or-slug>
crabbox cache volumes
crabbox ssh --id <id-or-slug>
blacksmith testbox list
```

Use `--debug` on `run` when measuring sync timing.
Use `--timing-json` on warmup, hydrate, and run when comparing backends.
Use `--market spot|on-demand` only on AWS warmup/one-shot runs.

## Failure Triage

- Crabbox cannot find provider: verify `../crabbox/bin/crabbox --help` lists
  the provider selected by `.crabbox.yaml`; update Crabbox before falling back.
- Hydration stuck or failed: open the printed GitHub Actions run URL and inspect
  the hydration step.
- Sync failed: rerun with `--debug`; check changed-file count and whether the
  checkout is dirty.
- Command failed: rerun only the failing shard/file first. Do not rerun a full
  suite until the focused failure is understood.
- Cleanup uncertain: `crabbox list --provider aws`; for explicit Blacksmith
  runs, use `blacksmith testbox list` and stop owned `tbx_...` leases you
  created.
- Crabbox broken but Blacksmith works: use the direct Blacksmith fallback above,
  then file/fix the Crabbox issue.

## Boundary

Do not add OpenClaw-specific setup to Crabbox itself. Put repo setup in the
hydration workflow and keep Crabbox generic around lease, sync, command
execution, logs/results, timing, and cleanup.