Vendor OpenClaw source as Adolf fork baseline
Some checks failed
ClawSweeper Dispatch / dispatch (push) Has been cancelled
CodeQL / Security High (actions) (push) Has been cancelled
CodeQL / Security High (channel-runtime-boundary) (push) Has been cancelled
CodeQL / Security High (core-auth-secrets) (push) Has been cancelled
CodeQL / Security High (mcp-process-tool-boundary) (push) Has been cancelled
CodeQL / Security High (network-ssrf-boundary) (push) Has been cancelled
CodeQL / Security High (plugin-trust-boundary) (push) Has been cancelled
CodeQL / Security High (process-exec-boundary) (push) Has been cancelled
Docs Sync Publish Repo / sync-publish-repo (push) Has been cancelled
Docs / docs (push) Has been cancelled
OpenClaw Stable Main Closeout / Resolve stable release closeout inputs (push) Has been cancelled
OpenClaw Stable Main Closeout / Verify stable main closeout (push) Has been cancelled
Workflow Sanity / no-tabs (push) Has been cancelled
Workflow Sanity / actionlint (push) Has been cancelled
Workflow Sanity / generated-doc-baselines (push) Has been cancelled
CI / runner-admission (push) Has been cancelled
CI / preflight (push) Has been cancelled
CI / security-fast (push) Has been cancelled
CI / pnpm-store-warmup (push) Has been cancelled
CI / build-artifacts (push) Has been cancelled
CI / native-i18n (push) Has been cancelled
CI / ${{ matrix.check_name }} (push) Has been cancelled
CI / ${{ matrix.checkName }} (push) Has been cancelled
CI / checks-node-compat-node22 (push) Has been cancelled
CI / check-bundled-channel-config-metadata (push) Has been cancelled
CI / check-dependencies (push) Has been cancelled
CI / check-guards (push) Has been cancelled
CI / check-lint (push) Has been cancelled
CI / check-prod-types (push) Has been cancelled
CI / check-shrinkwrap (push) Has been cancelled
CI / check-test-types (push) Has been cancelled
CI / check-additional-boundaries-a (push) Has been cancelled
CI / check-additional-boundaries-bcd (push) Has been cancelled
CI / check-additional-extension-bundled (push) Has been cancelled
CI / check-additional-extension-channels (push) Has been cancelled
CI / check-additional-extension-package-boundary (push) Has been cancelled
CI / check-additional-runtime-topology-architecture (push) Has been cancelled
CI / check-session-accessor-boundary (push) Has been cancelled
CI / check-session-transcript-reader-boundary (push) Has been cancelled
CI / check-docs (push) Has been cancelled
CI / skills-python (push) Has been cancelled
CI / macos-swift (push) Has been cancelled
CI / ios-build (push) Has been cancelled
CI / ci-timings-summary (push) Has been cancelled
Native App Locale Refresh / Refresh native fa (push) Has been cancelled
Native App Locale Refresh / Refresh native fr (push) Has been cancelled
Native App Locale Refresh / Refresh native hi (push) Has been cancelled
Native App Locale Refresh / Refresh native id (push) Has been cancelled
Native App Locale Refresh / Refresh native it (push) Has been cancelled
Native App Locale Refresh / Refresh native ja-JP (push) Has been cancelled
Control UI Locale Refresh / plan (push) Has been cancelled
Control UI Locale Refresh / Refresh ${{ matrix.locale }} (push) Has been cancelled
Control UI Locale Refresh / Commit control UI locale refresh (push) Has been cancelled
Live Media Runner Image / Build live media runner image (push) Has been cancelled
Native App Locale Refresh / Refresh native ar (push) Has been cancelled
Native App Locale Refresh / Refresh native de (push) Has been cancelled
Native App Locale Refresh / Refresh native es (push) Has been cancelled
Native App Locale Refresh / Refresh native ko (push) Has been cancelled
Native App Locale Refresh / Refresh native nl (push) Has been cancelled
Native App Locale Refresh / Refresh native pl (push) Has been cancelled
Native App Locale Refresh / Refresh native pt-BR (push) Has been cancelled
Native App Locale Refresh / Refresh native ru (push) Has been cancelled
Native App Locale Refresh / Refresh native sv (push) Has been cancelled
Native App Locale Refresh / Refresh native th (push) Has been cancelled
Native App Locale Refresh / Refresh native tr (push) Has been cancelled
Native App Locale Refresh / Refresh native uk (push) Has been cancelled
Native App Locale Refresh / Refresh native vi (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-CN (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-TW (push) Has been cancelled
Native App Locale Refresh / Commit native locale refresh (push) Has been cancelled
Plugin Init Scaffold Validation / Validate provider scaffold (push) Has been cancelled
Plugin NPM Release / preview_plugins_npm (push) Has been cancelled
Plugin NPM Release / Validate release publish approval (push) Has been cancelled
Plugin NPM Release / preview_plugin_pack (push) Has been cancelled
Plugin NPM Release / publish_plugins_npm (push) Has been cancelled
Sandbox Common Smoke / sandbox-common-smoke (push) Has been cancelled
Website Installer Sync / static (push) Has been cancelled
Website Installer Sync / linux-docker (push) Has been cancelled
Website Installer Sync / macos-installer (push) Has been cancelled
Website Installer Sync / windows-installer (push) Has been cancelled
Website Installer Sync / sync-website (push) Has been cancelled
Some checks failed
ClawSweeper Dispatch / dispatch (push) Has been cancelled
CodeQL / Security High (actions) (push) Has been cancelled
CodeQL / Security High (channel-runtime-boundary) (push) Has been cancelled
CodeQL / Security High (core-auth-secrets) (push) Has been cancelled
CodeQL / Security High (mcp-process-tool-boundary) (push) Has been cancelled
CodeQL / Security High (network-ssrf-boundary) (push) Has been cancelled
CodeQL / Security High (plugin-trust-boundary) (push) Has been cancelled
CodeQL / Security High (process-exec-boundary) (push) Has been cancelled
Docs Sync Publish Repo / sync-publish-repo (push) Has been cancelled
Docs / docs (push) Has been cancelled
OpenClaw Stable Main Closeout / Resolve stable release closeout inputs (push) Has been cancelled
OpenClaw Stable Main Closeout / Verify stable main closeout (push) Has been cancelled
Workflow Sanity / no-tabs (push) Has been cancelled
Workflow Sanity / actionlint (push) Has been cancelled
Workflow Sanity / generated-doc-baselines (push) Has been cancelled
CI / runner-admission (push) Has been cancelled
CI / preflight (push) Has been cancelled
CI / security-fast (push) Has been cancelled
CI / pnpm-store-warmup (push) Has been cancelled
CI / build-artifacts (push) Has been cancelled
CI / native-i18n (push) Has been cancelled
CI / ${{ matrix.check_name }} (push) Has been cancelled
CI / ${{ matrix.checkName }} (push) Has been cancelled
CI / checks-node-compat-node22 (push) Has been cancelled
CI / check-bundled-channel-config-metadata (push) Has been cancelled
CI / check-dependencies (push) Has been cancelled
CI / check-guards (push) Has been cancelled
CI / check-lint (push) Has been cancelled
CI / check-prod-types (push) Has been cancelled
CI / check-shrinkwrap (push) Has been cancelled
CI / check-test-types (push) Has been cancelled
CI / check-additional-boundaries-a (push) Has been cancelled
CI / check-additional-boundaries-bcd (push) Has been cancelled
CI / check-additional-extension-bundled (push) Has been cancelled
CI / check-additional-extension-channels (push) Has been cancelled
CI / check-additional-extension-package-boundary (push) Has been cancelled
CI / check-additional-runtime-topology-architecture (push) Has been cancelled
CI / check-session-accessor-boundary (push) Has been cancelled
CI / check-session-transcript-reader-boundary (push) Has been cancelled
CI / check-docs (push) Has been cancelled
CI / skills-python (push) Has been cancelled
CI / macos-swift (push) Has been cancelled
CI / ios-build (push) Has been cancelled
CI / ci-timings-summary (push) Has been cancelled
Native App Locale Refresh / Refresh native fa (push) Has been cancelled
Native App Locale Refresh / Refresh native fr (push) Has been cancelled
Native App Locale Refresh / Refresh native hi (push) Has been cancelled
Native App Locale Refresh / Refresh native id (push) Has been cancelled
Native App Locale Refresh / Refresh native it (push) Has been cancelled
Native App Locale Refresh / Refresh native ja-JP (push) Has been cancelled
Control UI Locale Refresh / plan (push) Has been cancelled
Control UI Locale Refresh / Refresh ${{ matrix.locale }} (push) Has been cancelled
Control UI Locale Refresh / Commit control UI locale refresh (push) Has been cancelled
Live Media Runner Image / Build live media runner image (push) Has been cancelled
Native App Locale Refresh / Refresh native ar (push) Has been cancelled
Native App Locale Refresh / Refresh native de (push) Has been cancelled
Native App Locale Refresh / Refresh native es (push) Has been cancelled
Native App Locale Refresh / Refresh native ko (push) Has been cancelled
Native App Locale Refresh / Refresh native nl (push) Has been cancelled
Native App Locale Refresh / Refresh native pl (push) Has been cancelled
Native App Locale Refresh / Refresh native pt-BR (push) Has been cancelled
Native App Locale Refresh / Refresh native ru (push) Has been cancelled
Native App Locale Refresh / Refresh native sv (push) Has been cancelled
Native App Locale Refresh / Refresh native th (push) Has been cancelled
Native App Locale Refresh / Refresh native tr (push) Has been cancelled
Native App Locale Refresh / Refresh native uk (push) Has been cancelled
Native App Locale Refresh / Refresh native vi (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-CN (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-TW (push) Has been cancelled
Native App Locale Refresh / Commit native locale refresh (push) Has been cancelled
Plugin Init Scaffold Validation / Validate provider scaffold (push) Has been cancelled
Plugin NPM Release / preview_plugins_npm (push) Has been cancelled
Plugin NPM Release / Validate release publish approval (push) Has been cancelled
Plugin NPM Release / preview_plugin_pack (push) Has been cancelled
Plugin NPM Release / publish_plugins_npm (push) Has been cancelled
Sandbox Common Smoke / sandbox-common-smoke (push) Has been cancelled
Website Installer Sync / static (push) Has been cancelled
Website Installer Sync / linux-docker (push) Has been cancelled
Website Installer Sync / macos-installer (push) Has been cancelled
Website Installer Sync / windows-installer (push) Has been cancelled
Website Installer Sync / sync-website (push) Has been cancelled
Adolf is a fork/vendored clone of github.com/openclaw/openclaw (v2026.6.11), free to diverge. Tree copied sans upstream .git; upstream remote added for future syncs. Node pinned to 24 (.nvmrc); engines already require >=22.19. Preserves docs/ARCHITECTURE.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LeqyaxJF2nbRXJtae2kNB2
This commit is contained in:
409
docs/concepts/mantis.md
Normal file
409
docs/concepts/mantis.md
Normal file
@@ -0,0 +1,409 @@
|
||||
---
|
||||
summary: "Mantis is the visual end-to-end verification system for reproducing OpenClaw bugs on live transports, capturing before and after evidence, and attaching artifacts to PRs."
|
||||
title: "Mantis"
|
||||
read_when:
|
||||
- Building or running live visual QA for OpenClaw bugs
|
||||
- Adding before and after verification for a pull request
|
||||
- Adding Discord, Slack, WhatsApp, or other live transport scenarios
|
||||
- Debugging QA runs that need screenshots, browser automation, or VNC access
|
||||
---
|
||||
|
||||
Mantis reruns a bug scenario against a known-bad baseline ref and a candidate
|
||||
ref on a real transport, then publishes a before/after comparison as CI
|
||||
artifacts and a PR comment. Discord shipped first: real bot auth, real guild
|
||||
channels, reactions, threads, and a browser witness a human can check. Slack
|
||||
and Telegram lanes exist too; WhatsApp and Matrix are unimplemented.
|
||||
|
||||
## Ownership
|
||||
|
||||
- OpenClaw (`extensions/qa-lab/src/mantis/*`): scenario runtime, `pnpm openclaw qa mantis <command>` CLI, evidence schema.
|
||||
- QA Lab (`extensions/qa-lab/src/live-transports/*`): live transport harness, driver/SUT bots, report/evidence writers.
|
||||
- Crabbox (`openclaw/crabbox`): warmed Linux machines, leases, VNC, `crabbox media preview`.
|
||||
- GitHub Actions (`.github/workflows/mantis-*.yml`): remote entrypoints, artifact retention.
|
||||
- ClawSweeper: parses maintainer PR commands, dispatches workflows, posts the final PR comment.
|
||||
|
||||
## CLI commands
|
||||
|
||||
All commands are `pnpm openclaw qa mantis <command>`, defined in
|
||||
`extensions/qa-lab/src/mantis/cli.ts`. Requires `OPENCLAW_ENABLE_PRIVATE_QA_CLI=1`
|
||||
at build/run time (bundled workflows set `OPENCLAW_BUILD_PRIVATE_QA=1` and
|
||||
`OPENCLAW_ENABLE_PRIVATE_QA_CLI=1` before building).
|
||||
|
||||
| Command | Purpose |
|
||||
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `discord-smoke` | Verify the Mantis Discord bot can see the guild/channel, post, and react. |
|
||||
| `run` | Run a before/after scenario against baseline and candidate refs (Discord only). |
|
||||
| `desktop-browser-smoke` | Lease/reuse a Crabbox desktop, open a visible browser, capture screenshot + video. |
|
||||
| `slack-desktop-smoke` | Lease/reuse a Crabbox desktop, run Slack QA inside it, open Slack Web, capture evidence. |
|
||||
| `telegram-desktop-builder` | Lease/reuse a Crabbox desktop, install Telegram Desktop, optionally configure an OpenClaw gateway. |
|
||||
| `visual-task` / `visual-driver` | Generic Crabbox desktop capture with optional image-understanding assertions; `visual-driver` is the driver half launched under `crabbox record --while`. |
|
||||
|
||||
Every command accepts `--repo-root <path>` and `--output-dir <path>`; Crabbox
|
||||
commands also accept `--crabbox-bin`, `--provider`, `--machine-class`/`--class`,
|
||||
`--lease-id`, `--idle-timeout`, `--ttl`, and `--keep-lease`. Local CLI defaults
|
||||
for provider/class are `hetzner`/`beast` unless noted otherwise; CI workflows
|
||||
usually override both.
|
||||
|
||||
### `discord-smoke`
|
||||
|
||||
```bash
|
||||
pnpm openclaw qa mantis discord-smoke \
|
||||
--output-dir .artifacts/qa-e2e/mantis/discord-smoke
|
||||
```
|
||||
|
||||
Calls the Discord REST API (`https://discord.com/api/v10`) to fetch the bot
|
||||
user, the guild, the guild's channels, and the target channel, asserts the
|
||||
channel belongs to the guild, then (unless `--skip-post`) posts a message and
|
||||
adds a `👀` reaction. Writes `mantis-discord-smoke-summary.json` and
|
||||
`mantis-discord-smoke-report.md`.
|
||||
|
||||
Token resolution order: `--token-file` value, then `OPENCLAW_QA_DISCORD_MANTIS_BOT_TOKEN`
|
||||
(override with `--token-env`), then a file named by `OPENCLAW_QA_DISCORD_MANTIS_BOT_TOKEN_FILE`
|
||||
(override with `--token-file-env`). Guild/channel ids come from
|
||||
`OPENCLAW_QA_DISCORD_GUILD_ID` / `OPENCLAW_QA_DISCORD_CHANNEL_ID` (override with
|
||||
`--guild-id` / `--channel-id`) and must be 17-20 digit Discord snowflakes. Set
|
||||
`OPENCLAW_QA_REDACT_PUBLIC_METADATA=1` to replace bot/guild/channel/message ids
|
||||
and names with `<redacted>` in the published summary and report.
|
||||
|
||||
### `run`
|
||||
|
||||
```bash
|
||||
pnpm openclaw qa mantis run \
|
||||
--transport discord \
|
||||
--scenario discord-status-reactions-tool-only \
|
||||
--baseline origin/main \
|
||||
--candidate HEAD \
|
||||
--output-dir .artifacts/qa-e2e/mantis/local-discord-status-reactions
|
||||
```
|
||||
|
||||
`--transport` currently only accepts `discord`. `--scenario` is one of two
|
||||
built-in ids, each with its own default baseline ref and expected before/after
|
||||
labels (`extensions/qa-lab/src/mantis/run.runtime.ts`):
|
||||
|
||||
| Scenario | Default baseline | Baseline expects | Candidate expects |
|
||||
| ------------------------------------------ | ------------------------------------------ | ---------------------------------------- | ---------------------------- |
|
||||
| `discord-status-reactions-tool-only` | `0bf06e953fdda290799fc9fb9244a8f67fdae593` | `queued-only` | `queued -> thinking -> done` |
|
||||
| `discord-thread-reply-filepath-attachment` | `81349cdc2a9d5143fd0991ed858b739e7d96e05c` | thread reply omits `filePath` attachment | thread reply includes it |
|
||||
|
||||
`--candidate` defaults to `HEAD`. Other flags: `--credential-source`
|
||||
(default `convex`), `--credential-role` (default `ci`), `--provider-mode`
|
||||
(default `live-frontier`), `--fast` (default on), `--skip-install`, `--skip-build`.
|
||||
|
||||
The runner creates detached `git worktree` checkouts for baseline and
|
||||
candidate under `<output-dir>/worktrees/`, runs `pnpm install`/`pnpm build` in
|
||||
each (unless skipped), then runs
|
||||
`pnpm openclaw qa discord --scenario <id> --model openai/gpt-5.4 --alt-model openai/gpt-5.4 --allow-failures`
|
||||
against each worktree. Each lane writes `discord-qa-reaction-timelines.json`
|
||||
plus a `<scenario-id>-timeline.html`/`.png` pair; the runner copies this
|
||||
evidence back under `baseline/`/`candidate/`, writes `comparison.json`,
|
||||
`mantis-report.md`, and `mantis-evidence.json` in the output directory, and
|
||||
exits nonzero if the comparison did not pass (baseline `fail` and candidate
|
||||
`pass`).
|
||||
|
||||
The second Discord scenario (`discord-thread-reply-filepath-attachment`) posts
|
||||
a parent message with the driver bot, creates a real thread, calls the SUT's
|
||||
`message.thread-reply` action with a repo-local `filePath`, then polls the
|
||||
thread for the reply and the attachment filename. It expects an attachment
|
||||
named `mantis-thread-report.md`.
|
||||
|
||||
### `desktop-browser-smoke`
|
||||
|
||||
```bash
|
||||
pnpm openclaw qa mantis desktop-browser-smoke \
|
||||
--output-dir .artifacts/qa-e2e/mantis/desktop-browser
|
||||
```
|
||||
|
||||
Leases or reuses a Crabbox desktop, launches a browser inside the VNC session
|
||||
pointed at `--browser-url` (default `https://openclaw.ai`) or a rendered
|
||||
`--html-file`, waits, screenshots with `scrot`, optionally records an MP4 with
|
||||
`ffmpeg`, and rsyncs `desktop-browser-smoke.png` / `.mp4` / `remote-metadata.json`
|
||||
back to `--output-dir`.
|
||||
|
||||
Flags:
|
||||
|
||||
- `--lease-id <cbx_...>` reuses a warmed desktop instead of creating one.
|
||||
- `--browser-profile-dir <remote-path>` reuses a remote Chrome user-data-dir so a persistent desktop stays logged in between runs (used for a long-lived Discord Web viewer profile).
|
||||
- `--browser-profile-archive-env <name>` restores a base64 `.tgz` Chrome profile archive from that env var before launch (default `OPENCLAW_MANTIS_BROWSER_PROFILE_TGZ_B64`); used for logged-in witnesses like Discord Web.
|
||||
- `--video-duration <seconds>` controls MP4 capture length (default 10s).
|
||||
- `--keep-lease` (or `OPENCLAW_MANTIS_KEEP_VM=1`) keeps a lease this run created open for VNC inspection; failed runs that created a lease also keep it by default.
|
||||
|
||||
For Discord Web evidence, Mantis uses a dedicated viewer account, not a bot
|
||||
token. The Discord REST oracle (via `qa discord`) remains authoritative; when
|
||||
`OPENCLAW_QA_DISCORD_CAPTURE_UI_METADATA=1` is set, the scenario also writes a
|
||||
Discord Web URL artifact, and `OPENCLAW_QA_DISCORD_KEEP_THREADS=1` leaves the
|
||||
thread open long enough for the browser to open it.
|
||||
|
||||
The GitHub workflow prefers a persistent viewer profile via
|
||||
`MANTIS_DISCORD_VIEWER_CHROME_PROFILE_DIR` (full profile archives can outgrow
|
||||
GitHub's secret size limit); for small/bootstrap profiles it can restore a
|
||||
base64 `.tgz` from `MANTIS_DISCORD_VIEWER_CHROME_PROFILE_TGZ_B64` instead. With
|
||||
neither source configured, the workflow still publishes the deterministic
|
||||
baseline/candidate screenshots and logs that the logged-in witness was
|
||||
skipped.
|
||||
|
||||
### `slack-desktop-smoke`
|
||||
|
||||
```bash
|
||||
pnpm openclaw qa mantis slack-desktop-smoke \
|
||||
--output-dir .artifacts/qa-e2e/mantis/slack-desktop \
|
||||
--gateway-setup \
|
||||
--scenario slack-canary \
|
||||
--keep-lease
|
||||
```
|
||||
|
||||
Leases or reuses a Crabbox desktop, syncs the checkout into the VM, runs
|
||||
`pnpm openclaw qa slack` inside it, opens Slack Web in the VNC browser,
|
||||
captures the desktop, and copies both the Slack QA artifacts (`slack-qa/`) and
|
||||
the VNC screenshot/video back locally. This is the only Mantis shape where the
|
||||
SUT gateway and the browser both run inside the same VM.
|
||||
|
||||
With `--gateway-setup`, the command creates a persistent disposable OpenClaw
|
||||
home at `$HOME/.openclaw-mantis/slack-openclaw` in the VM, patches Slack
|
||||
Socket Mode config for the target channel, starts
|
||||
`openclaw gateway run --dev --allow-unconfigured --port 38973`, and leaves
|
||||
Chrome running in the VNC session; omitting `--gateway-setup` runs the normal
|
||||
bot-to-bot Slack QA lane instead.
|
||||
|
||||
Required env for `--credential-source env` (local default is `env`; role
|
||||
default is `maintainer`):
|
||||
|
||||
- `OPENCLAW_QA_SLACK_CHANNEL_ID`
|
||||
- `OPENCLAW_QA_SLACK_DRIVER_BOT_TOKEN`
|
||||
- `OPENCLAW_QA_SLACK_SUT_BOT_TOKEN`
|
||||
- `OPENCLAW_QA_SLACK_SUT_APP_TOKEN`
|
||||
- `OPENCLAW_LIVE_OPENAI_KEY` for the remote model lane (if only `OPENAI_API_KEY`
|
||||
is set locally, Mantis copies it to `OPENCLAW_LIVE_OPENAI_KEY` before
|
||||
invoking Crabbox)
|
||||
|
||||
With `--credential-source convex`, Mantis leases the Slack SUT credential from
|
||||
the shared pool before creating the VM and forwards channel id, app token, and
|
||||
bot token into the VM as `OPENCLAW_MANTIS_SLACK_*` env vars, so GitHub
|
||||
workflows only need the Convex broker secret, not raw Slack tokens.
|
||||
|
||||
Other flags: `--slack-url <url>` opens a specific URL (otherwise Mantis derives
|
||||
`https://app.slack.com/client/<team>/<channel>` from `auth.test`);
|
||||
`--slack-channel-id <id>` sets the gateway allowlist channel;
|
||||
`OPENCLAW_MANTIS_SLACK_BROWSER_PROFILE_DIR` controls the persistent Chrome
|
||||
profile inside the VM (default `$HOME/.config/openclaw-mantis/slack-chrome-profile`);
|
||||
`--approval-checkpoints` runs the native Slack approval scenarios
|
||||
(`slack-approval-exec-native`, `slack-approval-plugin-native`) and renders
|
||||
pending/resolved checkpoint screenshots instead of gateway setup (mutually
|
||||
exclusive with `--gateway-setup`); `--hydrate-mode source|prehydrated`,
|
||||
`--provider-mode`, `--model`, `--alt-model`, and `--fast` pass through to the
|
||||
Slack live lane.
|
||||
|
||||
Approval checkpoint screenshots are rendered from the Slack API message the
|
||||
scenario observed, not the live Slack UI; `slack-desktop-smoke.png` is only
|
||||
proof of Slack Web itself when the lease's browser profile was already logged
|
||||
in.
|
||||
|
||||
### `telegram-desktop-builder`
|
||||
|
||||
```bash
|
||||
pnpm openclaw qa mantis telegram-desktop-builder \
|
||||
--credential-source convex \
|
||||
--credential-role maintainer \
|
||||
--keep-lease
|
||||
```
|
||||
|
||||
Leases or reuses a Crabbox desktop, installs native Linux Telegram Desktop,
|
||||
optionally restores a user-session archive, configures OpenClaw with the
|
||||
leased Telegram SUT bot token, starts
|
||||
`openclaw gateway run --dev --allow-unconfigured --port 38974`, posts a
|
||||
driver-bot readiness message to the leased private group, then captures a
|
||||
screenshot and MP4. A bot token only configures OpenClaw; it never logs
|
||||
Telegram Desktop in. The desktop viewer is a separate Telegram user session
|
||||
restored from `--telegram-profile-archive-env <name>` or logged in manually
|
||||
through VNC and kept alive with `--keep-lease`.
|
||||
|
||||
Flags: `--lease-id <cbx_...>` reruns against a VM already logged in to
|
||||
Telegram Desktop; `--telegram-profile-archive-env <name>` restores a base64
|
||||
`.tgz` profile archive before launch; `--telegram-profile-dir <remote-path>`
|
||||
sets the remote profile directory (default `$HOME/.local/share/TelegramDesktop`);
|
||||
`--no-gateway-setup` installs and opens Telegram Desktop only;
|
||||
`--credential-source`/`--credential-role` default to `convex`/`maintainer`.
|
||||
|
||||
## Evidence manifest
|
||||
|
||||
Every scenario that publishes to a PR writes `mantis-evidence.json` next to
|
||||
its report:
|
||||
|
||||
```json
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"id": "discord-status-reactions",
|
||||
"title": "Mantis Discord Status Reactions QA",
|
||||
"summary": "Human-readable top summary for the PR comment.",
|
||||
"scenario": "discord-status-reactions-tool-only",
|
||||
"comparison": {
|
||||
"baseline": { "sha": "...", "status": "fail", "expected": "queued-only" },
|
||||
"candidate": { "sha": "...", "status": "pass", "expected": "queued -> thinking -> done" },
|
||||
"pass": true
|
||||
},
|
||||
"artifacts": [
|
||||
{
|
||||
"kind": "timeline",
|
||||
"lane": "baseline",
|
||||
"label": "Baseline queued-only",
|
||||
"path": "baseline/timeline.png",
|
||||
"targetPath": "baseline.png",
|
||||
"alt": "Baseline Discord timeline",
|
||||
"width": 420
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Artifact `path` is relative to the manifest's directory; `targetPath` is
|
||||
relative to the configured R2/S3 artifact prefix. `scripts/mantis/publish-pr-evidence.mjs`
|
||||
rejects path traversal and skips entries with `"required": false` when the
|
||||
file is missing.
|
||||
|
||||
Artifact kinds: `timeline` (deterministic before/after screenshot),
|
||||
`desktopScreenshot` (VNC/browser screenshot), `motionPreview` (inline animated
|
||||
GIF from the recording), `motionClip` (motion-trimmed MP4), `fullVideo` (full
|
||||
recording), `metadata` (JSON/log sidecar), `report` (Markdown report).
|
||||
|
||||
A run's on-disk artifact layout:
|
||||
|
||||
```text
|
||||
.artifacts/qa-e2e/mantis/<run-id>/
|
||||
mantis-report.md
|
||||
mantis-evidence.json
|
||||
baseline/
|
||||
candidate/
|
||||
comparison.json
|
||||
```
|
||||
|
||||
Screenshots are evidence, not secrets, but still need redaction discipline:
|
||||
private channel names, usernames, or message content may appear. Set
|
||||
`OPENCLAW_QA_REDACT_PUBLIC_METADATA=1` for public artifact uploads; it is
|
||||
enabled by default in the Discord/Slack/Telegram GitHub workflows.
|
||||
|
||||
## GitHub automation
|
||||
|
||||
`scripts/mantis/publish-pr-evidence.mjs` is the reusable publisher. Workflows
|
||||
call it with the manifest, target PR, artifact target root, comment marker,
|
||||
artifact URL, run URL, and request source. It uploads declared artifacts to
|
||||
the Mantis R2 bucket, builds a summary-first PR comment with inline
|
||||
images/previews and linked videos, then updates the existing marker comment or
|
||||
creates a new one. Required env:
|
||||
|
||||
- `MANTIS_ARTIFACT_R2_ACCESS_KEY_ID`
|
||||
- `MANTIS_ARTIFACT_R2_SECRET_ACCESS_KEY`
|
||||
- `MANTIS_ARTIFACT_R2_BUCKET` (workflows set `openclaw-crabbox-artifacts`)
|
||||
- `MANTIS_ARTIFACT_R2_ENDPOINT`
|
||||
- `MANTIS_ARTIFACT_R2_REGION` (workflows set `auto`)
|
||||
- `MANTIS_ARTIFACT_R2_PUBLIC_BASE_URL` (workflows set `https://artifacts.openclaw.ai`)
|
||||
|
||||
Comments post through the Mantis GitHub App (`MANTIS_GITHUB_APP_ID` /
|
||||
`MANTIS_GITHUB_APP_PRIVATE_KEY`), not `github-actions[bot]`, using a hidden
|
||||
marker comment as the upsert key.
|
||||
|
||||
| Workflow | Trigger | What it does |
|
||||
| --------------------------------- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `Mantis Discord Smoke` | manual dispatch | Runs `discord-smoke` against a chosen ref. |
|
||||
| `Mantis Discord Status Reactions` | PR comment or manual dispatch | Builds separate baseline/candidate worktrees, runs `discord-status-reactions-tool-only` on each, renders each lane's timeline in a Crabbox desktop browser, generates motion-trimmed GIF/MP4 previews with `crabbox media preview`, uploads artifacts, posts inline PR evidence. |
|
||||
| `Mantis Scenario` | manual dispatch | Generic dispatcher: takes `scenario_id` (`discord-status-reactions-tool-only`, `discord-thread-reply-filepath-attachment`, `slack-desktop-smoke`, `telegram-live`, `telegram-desktop-proof`), `baseline_ref`, `candidate_ref`, `pr_number`, and forwards to the matching scenario workflow. |
|
||||
| `Mantis Slack Desktop Smoke` | manual dispatch | Leases a Crabbox Linux desktop (defaults to `aws`, choice of `hetzner`), runs `slack-desktop-smoke --gateway-setup` against the candidate, records the desktop, generates a motion preview, uploads artifacts, posts PR evidence when a PR number is given. |
|
||||
| `Mantis Telegram Live` | PR comment or manual dispatch | Runs the bot-API Telegram live QA lane (`openclaw qa telegram`), writes `mantis-evidence.json` from the QA summary, renders redacted evidence HTML through a Crabbox desktop browser, generates a motion GIF, posts PR evidence. Telegram Web login is not required for this lane. |
|
||||
| `Mantis Telegram Desktop Proof` | maintainer PR label (`mantis: telegram-visible-proof`) plus PR comment, or manual dispatch | Agentic native Telegram Desktop before/after proof. Hands the PR, baseline/candidate refs, and maintainer instructions to Codex, which runs the real-user Crabbox Telegram Desktop proof lane for both refs and posts a 2-column PR evidence table. |
|
||||
|
||||
`Mantis Discord Status Reactions` and `Mantis Telegram Live` both accept
|
||||
`baseline_ref`/`candidate_ref` (or `baseline=`/`candidate=` in a PR comment)
|
||||
and validate that the resolved SHA is either an ancestor of `origin/main`, a
|
||||
release tag (`v*`), or the head of an open PR before running with
|
||||
secret-bearing credentials.
|
||||
|
||||
Comment triggers, from a PR with write/maintain/admin access:
|
||||
|
||||
```text
|
||||
@openclaw-mantis discord status reactions
|
||||
@openclaw-mantis discord status reactions baseline=origin/main candidate=HEAD
|
||||
@openclaw-mantis telegram
|
||||
@openclaw-mantis telegram scenario=telegram-status-command
|
||||
@openclaw-mantis telegram scenarios=telegram-status-command,telegram-mentioned-message-reply
|
||||
```
|
||||
|
||||
Telegram comment triggers default to the PR head SHA as candidate and
|
||||
`telegram-status-command` as scenario; they accept `provider=aws|hetzner` and
|
||||
`lease=<cbx_...>` to target a specific Crabbox provider or a pre-warmed
|
||||
desktop. `Mantis Telegram Desktop Proof` only responds to a PR comment when
|
||||
the PR already carries the `mantis: telegram-visible-proof` label.
|
||||
|
||||
ClawSweeper can also dispatch a scenario directly:
|
||||
|
||||
```text
|
||||
@clawsweeper mantis discord discord-status-reactions-tool-only
|
||||
```
|
||||
|
||||
## Machines and secrets
|
||||
|
||||
Local CLI Crabbox defaults are `--provider hetzner --class beast`; override
|
||||
with `--provider`, `--class`/`--machine-class`, or
|
||||
`OPENCLAW_MANTIS_CRABBOX_PROVIDER` / `OPENCLAW_MANTIS_CRABBOX_CLASS`. GitHub
|
||||
workflows commonly override both (for example `--class standard`, and the
|
||||
Slack workflow's `aws`/`hetzner` provider choice input). If a provider is too
|
||||
slow or unavailable, add it behind the same Crabbox interface rather than
|
||||
hardcoding a fallback.
|
||||
|
||||
VM baseline: Linux with a desktop-capable Chrome/Chromium, CDP access, VNC/
|
||||
noVNC, Node 22+ and pnpm, an OpenClaw checkout, and outbound access to the
|
||||
target transport, GitHub, model providers, and the credential broker.
|
||||
|
||||
Secret names used across the Mantis workflows:
|
||||
|
||||
- `OPENCLAW_QA_DISCORD_MANTIS_BOT_TOKEN`
|
||||
- `OPENCLAW_QA_DISCORD_DRIVER_BOT_TOKEN`
|
||||
- `OPENCLAW_QA_DISCORD_SUT_BOT_TOKEN`
|
||||
- `OPENCLAW_QA_DISCORD_GUILD_ID`
|
||||
- `OPENCLAW_QA_DISCORD_CHANNEL_ID`
|
||||
- `OPENCLAW_QA_REDACT_PUBLIC_METADATA=1` for public artifact uploads
|
||||
- `OPENCLAW_QA_CONVEX_SITE_URL`, `OPENCLAW_QA_CONVEX_SECRET_CI`
|
||||
- `CRABBOX_COORDINATOR` / `CRABBOX_COORDINATOR_TOKEN` (workflows also accept
|
||||
`OPENCLAW_QA_MANTIS_CRABBOX_COORDINATOR` / `_TOKEN` as a fallback and map
|
||||
them onto the plain names before invoking Crabbox)
|
||||
- `MANTIS_GITHUB_APP_ID`, `MANTIS_GITHUB_APP_PRIVATE_KEY`
|
||||
|
||||
The Mantis runner must never print Discord/Slack/Telegram bot tokens,
|
||||
provider API keys, browser cookies, auth profile contents, VNC passwords, or
|
||||
raw credential payloads. If a token leaks into an issue, PR, chat, or log,
|
||||
rotate it after the replacement secret is stored.
|
||||
|
||||
## Run outcomes
|
||||
|
||||
A scenario fails in one of two distinguishable ways, and the report separates
|
||||
them so a flaky environment does not read as a product regression:
|
||||
|
||||
- **Bug reproduced**: baseline failed the way the scenario expects.
|
||||
- **Harness failure**: environment setup, credentials, transport API, browser,
|
||||
or provider failed before the oracle was meaningful.
|
||||
|
||||
## Adding a scenario
|
||||
|
||||
Scenarios are TypeScript-defined per transport (see
|
||||
`MANTIS_SCENARIO_CONFIGS` in `extensions/qa-lab/src/mantis/run.runtime.ts` for
|
||||
the Discord before/after shape), not a standalone declarative file format.
|
||||
Each scenario needs: id and title, transport, required credentials, baseline
|
||||
ref policy, candidate ref policy, OpenClaw config patch, setup/stimulus steps,
|
||||
expected baseline and candidate oracle, visual capture targets, timeout
|
||||
budget, and cleanup steps.
|
||||
|
||||
Prefer small, typed oracles over vision checks: Discord reaction state or
|
||||
message references, Slack thread `ts`/reaction API state, email message ids
|
||||
and headers. Use browser screenshots when UI is the only reliable observable,
|
||||
and keep vision checks additive to a platform-API oracle where one exists.
|
||||
|
||||
After Discord, Slack, and Telegram, the same runner shape extends to WhatsApp
|
||||
(QR login, re-identification, delivery, media, reactions) and Matrix
|
||||
(encrypted rooms, thread/reply relations, restart resume); neither is
|
||||
implemented yet.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Which Discord bot should be the driver vs. the SUT when the existing Mantis
|
||||
bot is reused?
|
||||
- How long should GitHub retain Mantis artifacts for PRs?
|
||||
- When should ClawSweeper automatically recommend a Mantis scenario instead of
|
||||
waiting for a maintainer command?
|
||||
- Should screenshots be redacted or cropped before upload for public PRs?
|
||||
Reference in New Issue
Block a user