adolf/qa/new-scenarios-2026-04.md

QA Scenario Expansion - Round 2

Ten repo-grounded candidate scenarios to add after the current seed suite.

1. On-demand memory tools in channel context

• Goal: verify the agent uses memory_search plus memory_get instead of bluffing when a channel message asks about prior notes.
• Flow:
  • Seed MEMORY.md or memory/*.md with a fact not present in the current transcript.
  • Ask in a channel thread for that fact.
  • Verify tool usage and final answer accuracy.
• Pass:
  • memory_search runs first.
  • memory_get narrows to the right lines.
  • Final answer cites the remembered fact correctly without cross-session leakage.
• Docs: docs/concepts/memory.md, docs/concepts/memory-search.md
• Code: extensions/memory-core/src/tools.ts, extensions/memory-core/src/prompt-section.ts

2. Memory failure fallback

• Goal: verify memory failure is graceful when embeddings/search are unavailable.
• Flow:
  • Disable or break the embedding-backed memory path.
  • Ask for prior-note recall.
  • Verify the agent surfaces uncertainty and next action instead of hallucinating.
• Pass:
  • Tool failure does not crash the run.
  • Agent says it checked and could not confirm.
  • Report includes the remediation hint.
• Docs: docs/concepts/memory.md, docs/help/faq.md
• Code: extensions/memory-core/src/tools.shared.ts, extensions/memory-core/src/tools.citations.test.ts

3. Model switch with tool continuity

• Goal: verify model switching preserves session context and tool availability, not just plain text continuity.
• Flow:
  • Start on one model.
  • Switch to another configured model.
  • Ask for a tool-using follow-up such as file read or memory lookup.
• Pass:
  • Switch is reflected in runtime state.
  • Tool call still succeeds after the switch.
  • Final answer keeps prior context.
• Docs: docs/help/testing.md, docs/concepts/model-failover.md
• Code: extensions/qa-lab/src/suite.ts, docs/web/webchat.md

4. MCP-backed recall via QMD/mcporter

• Goal: verify an MCP-backed tool path works end to end, not just core tools.
• Flow:
  • Enable memory.qmd.mcporter.
  • Ask for recall that should route through the QMD MCP bridge.
  • Verify response and captured MCP execution path.
• Pass:
  • MCP-backed search path is used.
  • Returned snippet matches the right note.
  • Failure mode is explicit if the daemon/tool is missing.
• Docs: docs/gateway/secrets.md, docs/concepts/memory-qmd.md
• Code: extensions/memory-core/src/memory/qmd-manager.ts, extensions/memory-core/src/memory/qmd-manager.test.ts

5. Skill visibility and invocation

• Goal: verify the agent sees a workspace/project skill and actually uses it.
• Flow:
  • Add a simple workspace or .agents skill.
  • Confirm skill visibility through runtime inventory.
  • Ask for a task that should trigger the skill.
• Pass:
  • Skill appears in skills.status.
  • Agent invocation reflects the installed skill instructions.
  • Per-agent allowlist behavior is respected.
• Docs: docs/tools/skills.md, docs/gateway/protocol.md, docs/gateway/configuration.md
• Code: .agents/skills/openclaw-qa-testing/SKILL.md, docs/gateway/protocol.md

6. Skill install and hot availability

• Goal: verify a newly installed skill becomes usable without a broken intermediate state.
• Flow:
  • Install a ClawHub or gateway-managed skill.
  • Re-check skill inventory.
  • Ask the agent to perform the skill-backed task.
• Pass:
  • Install succeeds.
  • skills.status or skills.bins reflects the new skill.
  • Agent can use the skill immediately or after the expected reload path.
• Docs: docs/tools/skills.md, docs/cli/skills.md, docs/gateway/protocol.md
• Code: docs/gateway/protocol.md, docs/tools/skills.md

7. Native image generation

• Goal: verify image_generate appears only when configured and returns a real attachment/artifact.
• Flow:
  • Configure agents.defaults.imageGenerationModel.primary.
  • Ask for a simple generated image.
  • Verify generated media is returned in the reply path.
• Pass:
  • image_generate is in the effective tool set.
  • Generation succeeds with the configured provider/model.
  • Output is attached and the agent summarizes what it created.
• Docs: docs/tools/image-generation.md, docs/providers/openai.md
• Code: src/agents/openclaw-tools.image-generation.test.ts, src/image-generation/runtime.ts

8. Config patch skill disable

• Goal: verify config.patch can disable a workspace skill and the restarted gateway exposes the disabled state cleanly.
• Flow:
  • Add a workspace skill and verify it is eligible.
  • Use config.patch to disable that skill.
  • Wait for the gateway restart and read skills.status again.
• Pass:
  • Patch succeeds.
  • Gateway restarts cleanly.
  • The skill flips from eligible to disabled.
• Docs: docs/gateway/configuration.md, docs/gateway/protocol.md
• Code: docs/gateway/configuration.md, docs/web/control-ui.md

9. Restart-required config apply with wake-up

• Goal: verify a restart-required config change restarts cleanly and wakes the session back up.
• Flow:
  • Use config.apply or update.run on a restart-required surface.
  • Provide sessionKey so the operator gets the post-restart ping.
  • Resume the task after restart.
• Pass:
  • Restart happens once.
  • Session wake-up ping arrives.
  • Agent continues in the same logical workflow after restart.
• Docs: docs/gateway/configuration.md, docs/web/control-ui.md
• Code: docs/gateway/configuration.md, docs/gateway/protocol.md

10. Runtime inventory drift check

• Goal: verify the reported tool and skill inventory matches what the agent can really use after config/plugin changes.
• Flow:
  • Read tools.effective and skills.status.
  • Ask the agent to use one enabled thing and one disabled thing.
  • Compare actual behavior vs reported inventory.
• Pass:
  • Enabled item is callable.
  • Disabled item is absent or blocked for the right reason.
  • Inventory and runtime behavior stay in sync.
• Docs: docs/gateway/protocol.md, docs/web/webchat.md
• Code: docs/gateway/protocol.md, docs/web/control-ui.md

Best next additions to the executable suite

If we only promote three right away:

1. On-demand memory tools in channel context
2. Native image generation
3. Config patch skill disable