alvis
bedb527145
Some checks failed
ClawSweeper Dispatch / dispatch (push) Has been cancelled
CodeQL / Security High (actions) (push) Has been cancelled
CodeQL / Security High (channel-runtime-boundary) (push) Has been cancelled
CodeQL / Security High (core-auth-secrets) (push) Has been cancelled
CodeQL / Security High (mcp-process-tool-boundary) (push) Has been cancelled
CodeQL / Security High (network-ssrf-boundary) (push) Has been cancelled
CodeQL / Security High (plugin-trust-boundary) (push) Has been cancelled
CodeQL / Security High (process-exec-boundary) (push) Has been cancelled
Docs Sync Publish Repo / sync-publish-repo (push) Has been cancelled
Docs / docs (push) Has been cancelled
OpenClaw Stable Main Closeout / Resolve stable release closeout inputs (push) Has been cancelled
OpenClaw Stable Main Closeout / Verify stable main closeout (push) Has been cancelled
Workflow Sanity / no-tabs (push) Has been cancelled
Workflow Sanity / actionlint (push) Has been cancelled
Workflow Sanity / generated-doc-baselines (push) Has been cancelled
CI / runner-admission (push) Has been cancelled
CI / preflight (push) Has been cancelled
CI / security-fast (push) Has been cancelled
CI / pnpm-store-warmup (push) Has been cancelled
CI / build-artifacts (push) Has been cancelled
CI / native-i18n (push) Has been cancelled
CI / ${{ matrix.check_name }} (push) Has been cancelled
CI / ${{ matrix.checkName }} (push) Has been cancelled
CI / checks-node-compat-node22 (push) Has been cancelled
CI / check-bundled-channel-config-metadata (push) Has been cancelled
CI / check-dependencies (push) Has been cancelled
CI / check-guards (push) Has been cancelled
CI / check-lint (push) Has been cancelled
CI / check-prod-types (push) Has been cancelled
CI / check-shrinkwrap (push) Has been cancelled
CI / check-test-types (push) Has been cancelled
CI / check-additional-boundaries-a (push) Has been cancelled
CI / check-additional-boundaries-bcd (push) Has been cancelled
CI / check-additional-extension-bundled (push) Has been cancelled
CI / check-additional-extension-channels (push) Has been cancelled
CI / check-additional-extension-package-boundary (push) Has been cancelled
CI / check-additional-runtime-topology-architecture (push) Has been cancelled
CI / check-session-accessor-boundary (push) Has been cancelled
CI / check-session-transcript-reader-boundary (push) Has been cancelled
CI / check-docs (push) Has been cancelled
CI / skills-python (push) Has been cancelled
CI / macos-swift (push) Has been cancelled
CI / ios-build (push) Has been cancelled
CI / ci-timings-summary (push) Has been cancelled
Native App Locale Refresh / Refresh native fa (push) Has been cancelled
Native App Locale Refresh / Refresh native fr (push) Has been cancelled
Native App Locale Refresh / Refresh native hi (push) Has been cancelled
Native App Locale Refresh / Refresh native id (push) Has been cancelled
Native App Locale Refresh / Refresh native it (push) Has been cancelled
Native App Locale Refresh / Refresh native ja-JP (push) Has been cancelled
Control UI Locale Refresh / plan (push) Has been cancelled
Control UI Locale Refresh / Refresh ${{ matrix.locale }} (push) Has been cancelled
Control UI Locale Refresh / Commit control UI locale refresh (push) Has been cancelled
Live Media Runner Image / Build live media runner image (push) Has been cancelled
Native App Locale Refresh / Refresh native ar (push) Has been cancelled
Native App Locale Refresh / Refresh native de (push) Has been cancelled
Native App Locale Refresh / Refresh native es (push) Has been cancelled
Native App Locale Refresh / Refresh native ko (push) Has been cancelled
Native App Locale Refresh / Refresh native nl (push) Has been cancelled
Native App Locale Refresh / Refresh native pl (push) Has been cancelled
Native App Locale Refresh / Refresh native pt-BR (push) Has been cancelled
Native App Locale Refresh / Refresh native ru (push) Has been cancelled
Native App Locale Refresh / Refresh native sv (push) Has been cancelled
Native App Locale Refresh / Refresh native th (push) Has been cancelled
Native App Locale Refresh / Refresh native tr (push) Has been cancelled
Native App Locale Refresh / Refresh native uk (push) Has been cancelled
Native App Locale Refresh / Refresh native vi (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-CN (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-TW (push) Has been cancelled
Native App Locale Refresh / Commit native locale refresh (push) Has been cancelled
Plugin Init Scaffold Validation / Validate provider scaffold (push) Has been cancelled
Plugin NPM Release / preview_plugins_npm (push) Has been cancelled
Plugin NPM Release / Validate release publish approval (push) Has been cancelled
Plugin NPM Release / preview_plugin_pack (push) Has been cancelled
Plugin NPM Release / publish_plugins_npm (push) Has been cancelled
Sandbox Common Smoke / sandbox-common-smoke (push) Has been cancelled
Website Installer Sync / static (push) Has been cancelled
Website Installer Sync / linux-docker (push) Has been cancelled
Website Installer Sync / macos-installer (push) Has been cancelled
Website Installer Sync / windows-installer (push) Has been cancelled
Website Installer Sync / sync-website (push) Has been cancelled
Adolf is a fork/vendored clone of github.com/openclaw/openclaw (v2026.6.11), free to diverge. Tree copied sans upstream .git; upstream remote added for future syncs. Node pinned to 24 (.nvmrc); engines already require >=22.19. Preserves docs/ARCHITECTURE.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LeqyaxJF2nbRXJtae2kNB2
73 lines
4.1 KiB
Markdown
73 lines
4.1 KiB
Markdown
---
|
|
summary: "Local qa-channel scenarios for privacy-preserving personal assistant workflow checks."
|
|
read_when:
|
|
- Running local personal agent reliability checks
|
|
- Extending the repo-backed QA scenario catalog
|
|
- Verifying reminder, reply, memory, redaction, safe tool followthrough, task status, share-safe diagnostics, proof-backed completion claims, and failure recovery
|
|
title: "Personal agent benchmark pack"
|
|
---
|
|
|
|
The Personal Agent Benchmark Pack is a small repo-backed QA scenario pack for
|
|
local personal assistant workflows. It is not a generic model benchmark and
|
|
needs no new runner: it reuses the private QA stack ([QA overview](/concepts/qa-e2e-automation)),
|
|
the synthetic [QA channel](/channels/qa-channel), and the existing
|
|
`qa/scenarios` YAML catalog.
|
|
|
|
## Scenarios
|
|
|
|
Ten scenarios, defined in `qa/scenarios/personal/*.yaml`:
|
|
|
|
| Scenario id | Checks |
|
|
| ------------------------------------------ | -------------------------------------------------------------------------------------------- |
|
|
| `personal-reminder-roundtrip` | Fake personal reminders through local cron delivery |
|
|
| `personal-channel-thread-reply` | Fake DM and thread reply routing through `qa-channel` |
|
|
| `personal-memory-preference-recall` | Fake preference recall from the temporary QA workspace memory files |
|
|
| `personal-redaction-no-secret-leak` | Fake secret no-echo checks |
|
|
| `personal-tool-safety-followthrough` | Safe read-backed tool followthrough after a short approval-style turn |
|
|
| `personal-approval-denial-stop` | Approval denial stop behavior for a sensitive local read request |
|
|
| `personal-task-followthrough-status` | Proof-backed task status reporting that keeps pending, blocked, and done separate |
|
|
| `personal-share-safe-diagnostics-artifact` | Share-safe diagnostics artifacts that keep useful status while omitting raw personal content |
|
|
| `personal-no-fake-progress` | Proof-backed completion claims that avoid fake progress before local evidence exists |
|
|
| `personal-failure-recovery` | Failure recovery that reports partial status and keeps retry boundaries clear |
|
|
|
|
The machine-readable pack metadata (id list, title, description) lives in
|
|
`extensions/qa-lab/src/scenario-packs.ts` as `QA_PERSONAL_AGENT_SCENARIO_IDS`.
|
|
Run the pack with `--pack personal-agent`:
|
|
|
|
```bash
|
|
OPENCLAW_ENABLE_PRIVATE_QA_CLI=1 pnpm openclaw qa suite \
|
|
--provider-mode mock-openai \
|
|
--pack personal-agent \
|
|
--concurrency 1
|
|
```
|
|
|
|
`--pack` is additive with repeated `--scenario` flags. Explicit scenarios run
|
|
first, then the pack scenarios run in `QA_PERSONAL_AGENT_SCENARIO_IDS` order
|
|
with duplicates removed.
|
|
|
|
The pack targets `qa-channel` with `mock-openai` or another local QA provider
|
|
lane. Do not point it at live chat services or real personal accounts.
|
|
|
|
## Privacy Model
|
|
|
|
Scenarios use only fake users, fake preferences, fake secrets, and the
|
|
temporary QA gateway workspace created by the suite. They must not read or
|
|
write real OpenClaw user memory, sessions, credentials, launch agents, global
|
|
configs, or live gateway state.
|
|
|
|
Artifacts stay under the existing QA suite artifact directory and are treated
|
|
like test output. Redaction checks use fake markers so failures are safe to
|
|
inspect and file in issues.
|
|
|
|
## Extending the pack
|
|
|
|
Add new `.yaml` cases under `qa/scenarios/personal/`, then add the scenario id
|
|
to `QA_PERSONAL_AGENT_SCENARIO_IDS`. Keep each case small, local, deterministic
|
|
in `mock-openai`, and focused on one personal assistant behavior.
|
|
|
|
Good follow-up candidates: redacted trajectory export checks, local-only
|
|
plugin workflow checks.
|
|
|
|
Avoid adding a new runner, plugin, dependency, live transport, or model judge
|
|
until the scenario catalog has enough stable cases to justify that surface.
|