Vendor OpenClaw source as Adolf fork baseline
Some checks failed
ClawSweeper Dispatch / dispatch (push) Has been cancelled
CodeQL / Security High (actions) (push) Has been cancelled
CodeQL / Security High (channel-runtime-boundary) (push) Has been cancelled
CodeQL / Security High (core-auth-secrets) (push) Has been cancelled
CodeQL / Security High (mcp-process-tool-boundary) (push) Has been cancelled
CodeQL / Security High (network-ssrf-boundary) (push) Has been cancelled
CodeQL / Security High (plugin-trust-boundary) (push) Has been cancelled
CodeQL / Security High (process-exec-boundary) (push) Has been cancelled
Docs Sync Publish Repo / sync-publish-repo (push) Has been cancelled
Docs / docs (push) Has been cancelled
OpenClaw Stable Main Closeout / Resolve stable release closeout inputs (push) Has been cancelled
OpenClaw Stable Main Closeout / Verify stable main closeout (push) Has been cancelled
Workflow Sanity / no-tabs (push) Has been cancelled
Workflow Sanity / actionlint (push) Has been cancelled
Workflow Sanity / generated-doc-baselines (push) Has been cancelled
CI / runner-admission (push) Has been cancelled
CI / preflight (push) Has been cancelled
CI / security-fast (push) Has been cancelled
CI / pnpm-store-warmup (push) Has been cancelled
CI / build-artifacts (push) Has been cancelled
CI / native-i18n (push) Has been cancelled
CI / ${{ matrix.check_name }} (push) Has been cancelled
CI / ${{ matrix.checkName }} (push) Has been cancelled
CI / checks-node-compat-node22 (push) Has been cancelled
CI / check-bundled-channel-config-metadata (push) Has been cancelled
CI / check-dependencies (push) Has been cancelled
CI / check-guards (push) Has been cancelled
CI / check-lint (push) Has been cancelled
CI / check-prod-types (push) Has been cancelled
CI / check-shrinkwrap (push) Has been cancelled
CI / check-test-types (push) Has been cancelled
CI / check-additional-boundaries-a (push) Has been cancelled
CI / check-additional-boundaries-bcd (push) Has been cancelled
CI / check-additional-extension-bundled (push) Has been cancelled
CI / check-additional-extension-channels (push) Has been cancelled
CI / check-additional-extension-package-boundary (push) Has been cancelled
CI / check-additional-runtime-topology-architecture (push) Has been cancelled
CI / check-session-accessor-boundary (push) Has been cancelled
CI / check-session-transcript-reader-boundary (push) Has been cancelled
CI / check-docs (push) Has been cancelled
CI / skills-python (push) Has been cancelled
CI / macos-swift (push) Has been cancelled
CI / ios-build (push) Has been cancelled
CI / ci-timings-summary (push) Has been cancelled
Native App Locale Refresh / Refresh native fa (push) Has been cancelled
Native App Locale Refresh / Refresh native fr (push) Has been cancelled
Native App Locale Refresh / Refresh native hi (push) Has been cancelled
Native App Locale Refresh / Refresh native id (push) Has been cancelled
Native App Locale Refresh / Refresh native it (push) Has been cancelled
Native App Locale Refresh / Refresh native ja-JP (push) Has been cancelled
Control UI Locale Refresh / plan (push) Has been cancelled
Control UI Locale Refresh / Refresh ${{ matrix.locale }} (push) Has been cancelled
Control UI Locale Refresh / Commit control UI locale refresh (push) Has been cancelled
Live Media Runner Image / Build live media runner image (push) Has been cancelled
Native App Locale Refresh / Refresh native ar (push) Has been cancelled
Native App Locale Refresh / Refresh native de (push) Has been cancelled
Native App Locale Refresh / Refresh native es (push) Has been cancelled
Native App Locale Refresh / Refresh native ko (push) Has been cancelled
Native App Locale Refresh / Refresh native nl (push) Has been cancelled
Native App Locale Refresh / Refresh native pl (push) Has been cancelled
Native App Locale Refresh / Refresh native pt-BR (push) Has been cancelled
Native App Locale Refresh / Refresh native ru (push) Has been cancelled
Native App Locale Refresh / Refresh native sv (push) Has been cancelled
Native App Locale Refresh / Refresh native th (push) Has been cancelled
Native App Locale Refresh / Refresh native tr (push) Has been cancelled
Native App Locale Refresh / Refresh native uk (push) Has been cancelled
Native App Locale Refresh / Refresh native vi (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-CN (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-TW (push) Has been cancelled
Native App Locale Refresh / Commit native locale refresh (push) Has been cancelled
Plugin Init Scaffold Validation / Validate provider scaffold (push) Has been cancelled
Plugin NPM Release / preview_plugins_npm (push) Has been cancelled
Plugin NPM Release / Validate release publish approval (push) Has been cancelled
Plugin NPM Release / preview_plugin_pack (push) Has been cancelled
Plugin NPM Release / publish_plugins_npm (push) Has been cancelled
Sandbox Common Smoke / sandbox-common-smoke (push) Has been cancelled
Website Installer Sync / static (push) Has been cancelled
Website Installer Sync / linux-docker (push) Has been cancelled
Website Installer Sync / macos-installer (push) Has been cancelled
Website Installer Sync / windows-installer (push) Has been cancelled
Website Installer Sync / sync-website (push) Has been cancelled
Some checks failed
ClawSweeper Dispatch / dispatch (push) Has been cancelled
CodeQL / Security High (actions) (push) Has been cancelled
CodeQL / Security High (channel-runtime-boundary) (push) Has been cancelled
CodeQL / Security High (core-auth-secrets) (push) Has been cancelled
CodeQL / Security High (mcp-process-tool-boundary) (push) Has been cancelled
CodeQL / Security High (network-ssrf-boundary) (push) Has been cancelled
CodeQL / Security High (plugin-trust-boundary) (push) Has been cancelled
CodeQL / Security High (process-exec-boundary) (push) Has been cancelled
Docs Sync Publish Repo / sync-publish-repo (push) Has been cancelled
Docs / docs (push) Has been cancelled
OpenClaw Stable Main Closeout / Resolve stable release closeout inputs (push) Has been cancelled
OpenClaw Stable Main Closeout / Verify stable main closeout (push) Has been cancelled
Workflow Sanity / no-tabs (push) Has been cancelled
Workflow Sanity / actionlint (push) Has been cancelled
Workflow Sanity / generated-doc-baselines (push) Has been cancelled
CI / runner-admission (push) Has been cancelled
CI / preflight (push) Has been cancelled
CI / security-fast (push) Has been cancelled
CI / pnpm-store-warmup (push) Has been cancelled
CI / build-artifacts (push) Has been cancelled
CI / native-i18n (push) Has been cancelled
CI / ${{ matrix.check_name }} (push) Has been cancelled
CI / ${{ matrix.checkName }} (push) Has been cancelled
CI / checks-node-compat-node22 (push) Has been cancelled
CI / check-bundled-channel-config-metadata (push) Has been cancelled
CI / check-dependencies (push) Has been cancelled
CI / check-guards (push) Has been cancelled
CI / check-lint (push) Has been cancelled
CI / check-prod-types (push) Has been cancelled
CI / check-shrinkwrap (push) Has been cancelled
CI / check-test-types (push) Has been cancelled
CI / check-additional-boundaries-a (push) Has been cancelled
CI / check-additional-boundaries-bcd (push) Has been cancelled
CI / check-additional-extension-bundled (push) Has been cancelled
CI / check-additional-extension-channels (push) Has been cancelled
CI / check-additional-extension-package-boundary (push) Has been cancelled
CI / check-additional-runtime-topology-architecture (push) Has been cancelled
CI / check-session-accessor-boundary (push) Has been cancelled
CI / check-session-transcript-reader-boundary (push) Has been cancelled
CI / check-docs (push) Has been cancelled
CI / skills-python (push) Has been cancelled
CI / macos-swift (push) Has been cancelled
CI / ios-build (push) Has been cancelled
CI / ci-timings-summary (push) Has been cancelled
Native App Locale Refresh / Refresh native fa (push) Has been cancelled
Native App Locale Refresh / Refresh native fr (push) Has been cancelled
Native App Locale Refresh / Refresh native hi (push) Has been cancelled
Native App Locale Refresh / Refresh native id (push) Has been cancelled
Native App Locale Refresh / Refresh native it (push) Has been cancelled
Native App Locale Refresh / Refresh native ja-JP (push) Has been cancelled
Control UI Locale Refresh / plan (push) Has been cancelled
Control UI Locale Refresh / Refresh ${{ matrix.locale }} (push) Has been cancelled
Control UI Locale Refresh / Commit control UI locale refresh (push) Has been cancelled
Live Media Runner Image / Build live media runner image (push) Has been cancelled
Native App Locale Refresh / Refresh native ar (push) Has been cancelled
Native App Locale Refresh / Refresh native de (push) Has been cancelled
Native App Locale Refresh / Refresh native es (push) Has been cancelled
Native App Locale Refresh / Refresh native ko (push) Has been cancelled
Native App Locale Refresh / Refresh native nl (push) Has been cancelled
Native App Locale Refresh / Refresh native pl (push) Has been cancelled
Native App Locale Refresh / Refresh native pt-BR (push) Has been cancelled
Native App Locale Refresh / Refresh native ru (push) Has been cancelled
Native App Locale Refresh / Refresh native sv (push) Has been cancelled
Native App Locale Refresh / Refresh native th (push) Has been cancelled
Native App Locale Refresh / Refresh native tr (push) Has been cancelled
Native App Locale Refresh / Refresh native uk (push) Has been cancelled
Native App Locale Refresh / Refresh native vi (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-CN (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-TW (push) Has been cancelled
Native App Locale Refresh / Commit native locale refresh (push) Has been cancelled
Plugin Init Scaffold Validation / Validate provider scaffold (push) Has been cancelled
Plugin NPM Release / preview_plugins_npm (push) Has been cancelled
Plugin NPM Release / Validate release publish approval (push) Has been cancelled
Plugin NPM Release / preview_plugin_pack (push) Has been cancelled
Plugin NPM Release / publish_plugins_npm (push) Has been cancelled
Sandbox Common Smoke / sandbox-common-smoke (push) Has been cancelled
Website Installer Sync / static (push) Has been cancelled
Website Installer Sync / linux-docker (push) Has been cancelled
Website Installer Sync / macos-installer (push) Has been cancelled
Website Installer Sync / windows-installer (push) Has been cancelled
Website Installer Sync / sync-website (push) Has been cancelled
Adolf is a fork/vendored clone of github.com/openclaw/openclaw (v2026.6.11), free to diverge. Tree copied sans upstream .git; upstream remote added for future syncs. Node pinned to 24 (.nvmrc); engines already require >=22.19. Preserves docs/ARCHITECTURE.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LeqyaxJF2nbRXJtae2kNB2
This commit is contained in:
95
qa/scenarios/channels/channel-chat-baseline.yaml
Normal file
95
qa/scenarios/channels/channel-chat-baseline.yaml
Normal file
@@ -0,0 +1,95 @@
|
||||
title: Channel baseline conversation
|
||||
|
||||
scenario:
|
||||
id: channel-chat-baseline
|
||||
surface: channel
|
||||
coverage:
|
||||
primary:
|
||||
- channels.group-messages
|
||||
secondary:
|
||||
- channels.qa-channel
|
||||
objective: Verify the QA agent can respond correctly in a shared channel and respect mention-driven group semantics.
|
||||
successCriteria:
|
||||
- Agent replies in the shared channel transcript.
|
||||
- Agent visible reply contains the scenario marker.
|
||||
- Agent keeps the conversation scoped to the channel.
|
||||
- Agent respects mention-driven group routing semantics.
|
||||
docsRefs:
|
||||
- docs/channels/group-messages.md
|
||||
- docs/channels/qa-channel.md
|
||||
codeRefs:
|
||||
- extensions/qa-channel/src/inbound.ts
|
||||
- extensions/qa-lab/src/bus-state.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Verify the QA agent can respond correctly in a shared channel and respect mention-driven group semantics.
|
||||
config:
|
||||
expectedMarker: QA-CHANNEL-BASELINE-OK
|
||||
mentionPrompt: "@openclaw qa channel baseline marker check. Reply exactly: QA-CHANNEL-BASELINE-OK"
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: ignores unmentioned channel chatter
|
||||
actions:
|
||||
- call: waitForGatewayHealthy
|
||||
args:
|
||||
- ref: env
|
||||
- 60000
|
||||
- call: waitForQaChannelReady
|
||||
args:
|
||||
- ref: env
|
||||
- 60000
|
||||
- call: reset
|
||||
- call: sleep
|
||||
args:
|
||||
- 1200
|
||||
- set: outboundStartIndex
|
||||
value:
|
||||
expr: "state.getSnapshot().messages.filter((message) => message.direction === 'outbound').length"
|
||||
- sendInbound:
|
||||
conversation:
|
||||
id: qa-room
|
||||
kind: channel
|
||||
title: QA Room
|
||||
senderId: alice
|
||||
senderName: Alice
|
||||
text: hello team, no bot ping here
|
||||
- waitForNoOutbound:
|
||||
quietMs: 1200
|
||||
sinceIndex:
|
||||
ref: outboundStartIndex
|
||||
- name: replies when mentioned in channel
|
||||
actions:
|
||||
- call: waitForGatewayHealthy
|
||||
args:
|
||||
- ref: env
|
||||
- 60000
|
||||
- call: waitForQaChannelReady
|
||||
args:
|
||||
- ref: env
|
||||
- 60000
|
||||
- sendInbound:
|
||||
conversation:
|
||||
id: qa-room
|
||||
kind: channel
|
||||
title: QA Room
|
||||
senderId: alice
|
||||
senderName: Alice
|
||||
text:
|
||||
expr: config.mentionPrompt
|
||||
- call: waitForOutboundMessage
|
||||
saveAs: message
|
||||
args:
|
||||
- ref: state
|
||||
- lambda:
|
||||
params: [candidate]
|
||||
expr: "candidate.direction === 'outbound' && candidate.conversation.id === 'qa-room' && candidate.conversation.kind === 'channel' && !candidate.threadId && String(candidate.text ?? '').includes(config.expectedMarker)"
|
||||
- expr: liveTurnTimeoutMs(env, 180000)
|
||||
- set: matchingOutbound
|
||||
value:
|
||||
expr: "state.getSnapshot().messages.filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-room' && candidate.conversation.kind === 'channel' && String(candidate.text ?? '').includes(config.expectedMarker))"
|
||||
- assert:
|
||||
expr: matchingOutbound.length === 1
|
||||
message:
|
||||
expr: "`expected exactly one channel baseline marker reply, saw ${matchingOutbound.length}; transcript=${formatTransportTranscript(state, { conversationId: 'qa-room' })}`"
|
||||
detailsExpr: message.text
|
||||
81
qa/scenarios/channels/channel-message-flows.yaml
Normal file
81
qa/scenarios/channels/channel-message-flows.yaml
Normal file
@@ -0,0 +1,81 @@
|
||||
title: Channel streaming message flow
|
||||
|
||||
scenario:
|
||||
id: channel-message-flows
|
||||
surface: channels
|
||||
coverage:
|
||||
primary:
|
||||
- channels.streaming
|
||||
secondary:
|
||||
- runtime.delivery
|
||||
objective: Verify streaming channel replies produce visible previews that resolve to one final answer.
|
||||
gatewayConfigPatch:
|
||||
channels:
|
||||
telegram:
|
||||
streaming:
|
||||
mode: partial
|
||||
successCriteria:
|
||||
- The selected transport exposes at least one preview event before final delivery.
|
||||
- The final answer replaces or follows the preview without losing the requested text.
|
||||
docsRefs:
|
||||
- docs/channels/qa-channel.md
|
||||
- docs/channels/telegram.md
|
||||
- docs/concepts/qa-e2e-automation.md
|
||||
codeRefs:
|
||||
- extensions/qa-channel/src/inbound.ts
|
||||
- extensions/qa-lab/src/crabline-transport.ts
|
||||
- extensions/qa-lab/src/providers/mock-openai/server.ts
|
||||
- extensions/qa-lab/src/qa-transport.ts
|
||||
- extensions/telegram/src/draft-stream.ts
|
||||
execution:
|
||||
kind: flow
|
||||
channel: telegram
|
||||
summary: Stream a deterministic answer through QA Channel or Crabline Telegram and assert its preview lifecycle.
|
||||
config:
|
||||
requiredProviderMode: mock-openai
|
||||
conversationId: "-1001234567890"
|
||||
senderId: "100001"
|
||||
finalMarker: QA-CHANNEL-STREAMING-PREVIEW-FINAL-OK-1234567890
|
||||
prompt: "Final-only marker streaming QA check. Reply exactly: QA-CHANNEL-STREAMING-PREVIEW-FINAL-OK-1234567890"
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: streams a preview into one final reply
|
||||
actions:
|
||||
- assert:
|
||||
expr: env.providerMode === config.requiredProviderMode
|
||||
message: this deterministic streaming proof requires mock-openai
|
||||
- call: waitForGatewayHealthy
|
||||
args:
|
||||
- ref: env
|
||||
- 60000
|
||||
- call: waitForTransportReady
|
||||
args:
|
||||
- ref: env
|
||||
- 60000
|
||||
- resetTransport: true
|
||||
- sendInbound:
|
||||
conversation:
|
||||
id:
|
||||
ref: config.conversationId
|
||||
kind: group
|
||||
senderId:
|
||||
ref: config.senderId
|
||||
senderName: QA Streaming Operator
|
||||
text:
|
||||
ref: config.prompt
|
||||
- waitForOutboundSequence:
|
||||
conversationId:
|
||||
ref: config.conversationId
|
||||
finalTextIncludes:
|
||||
ref: config.finalMarker
|
||||
finalSettleMs: 500
|
||||
minimumPreviewEvents: 1
|
||||
timeoutMs:
|
||||
expr: liveTurnTimeoutMs(env, 45000)
|
||||
saveAs: sequence
|
||||
- assert:
|
||||
expr: sequence.events.length >= 2
|
||||
message:
|
||||
expr: "`expected a preview followed by the final marker; events=${JSON.stringify(sequence.events)}`"
|
||||
detailsExpr: "`${sequence.events.map((event) => event.kind).join(' -> ')}: ${sequence.final.text}`"
|
||||
59
qa/scenarios/channels/dm-chat-baseline.yaml
Normal file
59
qa/scenarios/channels/dm-chat-baseline.yaml
Normal file
@@ -0,0 +1,59 @@
|
||||
title: DM baseline conversation
|
||||
|
||||
scenario:
|
||||
id: dm-chat-baseline
|
||||
surface: dm
|
||||
coverage:
|
||||
primary:
|
||||
- channels.dm
|
||||
secondary:
|
||||
- channels.qa-channel
|
||||
objective: Verify the QA agent can chat coherently in a DM, explain the QA setup, and stay in character.
|
||||
successCriteria:
|
||||
- Agent replies in DM without channel routing mistakes.
|
||||
- Agent visible reply contains the scenario marker.
|
||||
- Agent explains the QA lab and message bus correctly.
|
||||
- Agent keeps the dev C-3PO personality.
|
||||
docsRefs:
|
||||
- docs/channels/qa-channel.md
|
||||
- docs/help/testing.md
|
||||
codeRefs:
|
||||
- extensions/qa-channel/src/gateway.ts
|
||||
- extensions/qa-lab/src/lab-server.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Verify the QA agent can chat coherently in a DM, explain the QA setup, and stay in character.
|
||||
config:
|
||||
expectedMarker: QA-DM-BASELINE-OK
|
||||
prompt: "DM baseline marker check. Include exact marker: `QA-DM-BASELINE-OK` and briefly identify the QA lab message bus."
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: replies coherently in DM
|
||||
actions:
|
||||
- resetTransport: true
|
||||
- sendInbound:
|
||||
conversation:
|
||||
id: alice
|
||||
kind: direct
|
||||
senderId: alice
|
||||
senderName: Alice
|
||||
text:
|
||||
ref: config.prompt
|
||||
- waitForOutbound:
|
||||
conversation:
|
||||
id: alice
|
||||
kind: direct
|
||||
textIncludes:
|
||||
ref: config.expectedMarker
|
||||
timeoutMs:
|
||||
expr: liveTurnTimeoutMs(env, 45000)
|
||||
saveAs: outbound
|
||||
- set: matchingOutbound
|
||||
value:
|
||||
expr: "state.getSnapshot().messages.filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'alice' && candidate.conversation.kind === 'direct' && String(candidate.text ?? '').includes(config.expectedMarker))"
|
||||
- assert:
|
||||
expr: matchingOutbound.length === 1
|
||||
message:
|
||||
expr: "`expected exactly one DM baseline marker reply, saw ${matchingOutbound.length}; transcript=${formatTransportTranscript(state, { conversationId: 'alice' })}`"
|
||||
detailsExpr: outbound.text
|
||||
@@ -0,0 +1,95 @@
|
||||
title: Group fallback when message tool is unavailable
|
||||
|
||||
scenario:
|
||||
id: group-message-tool-unavailable-fallback
|
||||
surface: channel
|
||||
coverage:
|
||||
primary:
|
||||
- channels.group-visible-replies
|
||||
secondary:
|
||||
- channels.qa-channel
|
||||
- tools.message
|
||||
objective: Reproduce the group-visible-reply bug class where message_tool mode selected tool-only delivery even though group tool policy removed the message tool.
|
||||
gatewayConfigPatch:
|
||||
messages:
|
||||
groupChat:
|
||||
visibleReplies: message_tool
|
||||
channels:
|
||||
qa-channel:
|
||||
groups:
|
||||
qa-fallback-room:
|
||||
tools:
|
||||
allow:
|
||||
- read
|
||||
successCriteria:
|
||||
- The group policy removes the message tool for this room.
|
||||
- The mock provider returns a normal final answer with the marker.
|
||||
- OpenClaw falls back to automatic delivery and posts the marker to the same group.
|
||||
docsRefs:
|
||||
- docs/channels/groups.md
|
||||
- docs/channels/qa-channel.md
|
||||
codeRefs:
|
||||
- src/auto-reply/reply/dispatch-from-config.ts
|
||||
- extensions/qa-channel/src/inbound.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Verify message_tool visible replies degrade to automatic delivery when the active group policy removes message.
|
||||
config:
|
||||
requiredChannelDriver: qa-channel
|
||||
conversationId: qa-fallback-room
|
||||
promptSnippet: qa group message unavailable fallback check
|
||||
prompt: "@openclaw qa group message unavailable fallback check. exact marker: `QA-GROUP-FALLBACK-OK`"
|
||||
expectedMarker: QA-GROUP-FALLBACK-OK
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: falls back to final-answer delivery when message is not available
|
||||
actions:
|
||||
- call: waitForGatewayHealthy
|
||||
args:
|
||||
- ref: env
|
||||
- 60000
|
||||
- call: waitForQaChannelReady
|
||||
args:
|
||||
- ref: env
|
||||
- 60000
|
||||
- call: reset
|
||||
- set: requestCountBefore
|
||||
value:
|
||||
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).length : 0"
|
||||
- sendInbound:
|
||||
conversation:
|
||||
id:
|
||||
expr: config.conversationId
|
||||
kind: group
|
||||
title: QA Fallback Room
|
||||
senderId: alice
|
||||
senderName: Alice
|
||||
text:
|
||||
expr: config.prompt
|
||||
- call: waitForOutboundMessage
|
||||
saveAs: outbound
|
||||
args:
|
||||
- ref: state
|
||||
- lambda:
|
||||
params: [candidate]
|
||||
expr: "candidate.conversation.id === config.conversationId && candidate.conversation.kind === 'group' && !candidate.threadId && candidate.text.includes(config.expectedMarker)"
|
||||
- expr: liveTurnTimeoutMs(env, 180000)
|
||||
- set: matchingOutbound
|
||||
value:
|
||||
expr: "state.getSnapshot().messages.filter((message) => message.direction === 'outbound' && message.conversation.id === config.conversationId && message.conversation.kind === 'group' && String(message.text ?? '').includes(config.expectedMarker))"
|
||||
- assert:
|
||||
expr: matchingOutbound.length === 1
|
||||
message:
|
||||
expr: "`expected exactly one fallback group reply, saw ${matchingOutbound.length}`"
|
||||
- set: scenarioRequests
|
||||
value:
|
||||
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).slice(requestCountBefore).filter((request) => String(request.allInputText ?? '').includes(config.promptSnippet)) : []"
|
||||
- assert:
|
||||
expr: "!env.mock || scenarioRequests.length > 0"
|
||||
message: expected mock request evidence for fallback scenario
|
||||
- assert:
|
||||
expr: "!env.mock || scenarioRequests.every((request) => request.plannedToolName !== 'message')"
|
||||
message:
|
||||
expr: "`message tool should not be planned when group policy removes it, saw ${JSON.stringify(scenarioRequests.map((request) => request.plannedToolName ?? null))}`"
|
||||
detailsExpr: "`${outbound.conversation.kind}:${outbound.conversation.id}:${outbound.text}`"
|
||||
92
qa/scenarios/channels/group-visible-reply-tool.yaml
Normal file
92
qa/scenarios/channels/group-visible-reply-tool.yaml
Normal file
@@ -0,0 +1,92 @@
|
||||
title: Group visible reply via message tool
|
||||
|
||||
scenario:
|
||||
id: group-visible-reply-tool
|
||||
surface: channel
|
||||
coverage:
|
||||
primary:
|
||||
- channels.group-visible-replies
|
||||
secondary:
|
||||
- channels.qa-channel
|
||||
- tools.message
|
||||
objective: Verify a group-sourced QA channel turn replies visibly through message(action=send) in the same room.
|
||||
gatewayConfigPatch:
|
||||
messages:
|
||||
groupChat:
|
||||
visibleReplies: message_tool
|
||||
successCriteria:
|
||||
- Agent receives a synthetic shared-room turn.
|
||||
- Mock provider calls the shared message tool instead of relying on final-answer delivery.
|
||||
- The visible reply lands once in the same group transcript.
|
||||
docsRefs:
|
||||
- docs/channels/groups.md
|
||||
- docs/channels/qa-channel.md
|
||||
codeRefs:
|
||||
- extensions/qa-channel/src/inbound.ts
|
||||
- extensions/qa-channel/src/outbound.ts
|
||||
- src/auto-reply/reply/dispatch-from-config.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Send a mentioned group message and verify visible output uses the message tool in the source group.
|
||||
config:
|
||||
conversationId: qa-visible-tool-room
|
||||
promptSnippet: qa group visible reply tool check
|
||||
prompt: "@openclaw qa group visible reply tool check. Use the visible room reply path. exact marker: `QA-GROUP-TOOL-OK`"
|
||||
expectedMarker: QA-GROUP-TOOL-OK
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: posts visible room output through message tool
|
||||
actions:
|
||||
- call: waitForGatewayHealthy
|
||||
args:
|
||||
- ref: env
|
||||
- 60000
|
||||
- call: waitForQaChannelReady
|
||||
args:
|
||||
- ref: env
|
||||
- 60000
|
||||
- call: reset
|
||||
- set: requestCountBefore
|
||||
value:
|
||||
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).length : 0"
|
||||
- sendInbound:
|
||||
conversation:
|
||||
id:
|
||||
expr: config.conversationId
|
||||
kind: group
|
||||
title: QA Visible Tool Room
|
||||
senderId: alice
|
||||
senderName: Alice
|
||||
text:
|
||||
expr: config.prompt
|
||||
- call: waitForCondition
|
||||
args:
|
||||
- lambda:
|
||||
async: true
|
||||
params: []
|
||||
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).slice(requestCountBefore).find((request) => String(request.allInputText ?? '').includes(config.promptSnippet)) : true"
|
||||
- expr: liveTurnTimeoutMs(env, 180000)
|
||||
- set: scenarioRequests
|
||||
value:
|
||||
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).slice(requestCountBefore).filter((request) => String(request.allInputText ?? '').includes(config.promptSnippet)) : []"
|
||||
- assert:
|
||||
expr: "!env.mock || scenarioRequests.some((request) => request.plannedToolName === 'message' && request.plannedToolArgs?.action === 'send' && request.plannedToolArgs?.message === config.expectedMarker)"
|
||||
message:
|
||||
expr: "`expected message(action=send) with marker, saw ${JSON.stringify(scenarioRequests.map((request) => ({ plannedToolName: request.plannedToolName ?? null, plannedToolArgs: request.plannedToolArgs ?? null, toolOutput: request.toolOutput ?? '', tools: Array.isArray(request.body?.tools) ? request.body.tools.map((tool) => tool?.name ?? tool?.function?.name ?? tool?.type ?? null).filter(Boolean).slice(0, 25) : [] })))} `"
|
||||
- call: waitForOutboundMessage
|
||||
saveAs: outbound
|
||||
args:
|
||||
- ref: state
|
||||
- lambda:
|
||||
params: [candidate]
|
||||
expr: "candidate.conversation.id === config.conversationId && candidate.conversation.kind === 'group' && !candidate.threadId && candidate.text.includes(config.expectedMarker)"
|
||||
- expr: liveTurnTimeoutMs(env, 180000)
|
||||
- set: matchingOutbound
|
||||
value:
|
||||
expr: "state.getSnapshot().messages.filter((message) => message.direction === 'outbound' && message.conversation.id === config.conversationId && message.conversation.kind === 'group' && String(message.text ?? '').includes(config.expectedMarker))"
|
||||
- assert:
|
||||
expr: matchingOutbound.length === 1
|
||||
message:
|
||||
expr: "`expected exactly one visible group reply, saw ${matchingOutbound.length}`"
|
||||
detailsExpr: "`${outbound.conversation.kind}:${outbound.conversation.id}:${outbound.text}`"
|
||||
91
qa/scenarios/channels/message-tool-stranded-final-reply.yaml
Normal file
91
qa/scenarios/channels/message-tool-stranded-final-reply.yaml
Normal file
@@ -0,0 +1,91 @@
|
||||
title: Message-tool-only private final reply warning
|
||||
|
||||
scenario:
|
||||
id: message-tool-stranded-final-reply
|
||||
surface: channel
|
||||
coverage:
|
||||
primary:
|
||||
- channels.direct-visible-replies
|
||||
secondary:
|
||||
- channels.qa-channel
|
||||
- tools.message
|
||||
objective: Reproduce #85714 — under messages.visibleReplies=message_tool a long private final reply that never calls the message tool is kept private (no outbound), and the gateway emits the private-final WARN.
|
||||
gatewayConfigPatch:
|
||||
messages:
|
||||
visibleReplies: message_tool
|
||||
successCriteria:
|
||||
- The mock provider returns a long normal final answer and does not plan the message tool.
|
||||
- Under message_tool_only delivery the reply is kept private, so the direct conversation receives no outbound message.
|
||||
- The gateway logs the private-final WARN from source-reply/private-final.
|
||||
docsRefs:
|
||||
- docs/channels/qa-channel.md
|
||||
codeRefs:
|
||||
- src/auto-reply/reply/agent-runner.ts
|
||||
- src/auto-reply/reply/private-message-tool-final.ts
|
||||
- src/auto-reply/reply/dispatch-from-config.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Send a direct message_tool_only turn whose model reply omits the message tool, and verify a substantive private final warns without outbound delivery.
|
||||
config:
|
||||
requiredProviderMode: mock-openai
|
||||
conversationId: qa-stranded-dm
|
||||
promptSnippet: qa private final reply warning check
|
||||
prompt: "qa private final reply warning check. Reply to me directly in two complete sentences with `QA-STRANDED-85714` in the first sentence and a short explanation in the second sentence. Do NOT call any tool. Do NOT use the message tool."
|
||||
expectedMarker: QA-STRANDED-85714
|
||||
privateFinalLogNeedle: "source-reply/private-final"
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: warns for substantive private final text when the model omits the message tool
|
||||
actions:
|
||||
- assert:
|
||||
expr: "env.providerMode === config.requiredProviderMode"
|
||||
message: this seeded scenario is mock-openai only
|
||||
- call: waitForGatewayHealthy
|
||||
args:
|
||||
- ref: env
|
||||
- 60000
|
||||
- call: waitForQaChannelReady
|
||||
args:
|
||||
- ref: env
|
||||
- 60000
|
||||
- call: reset
|
||||
- set: logCursor
|
||||
value:
|
||||
expr: markGatewayLogCursor()
|
||||
- set: requestCountBefore
|
||||
value:
|
||||
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).length : 0"
|
||||
- sendInbound:
|
||||
conversation:
|
||||
id:
|
||||
expr: config.conversationId
|
||||
kind: direct
|
||||
senderId: alice
|
||||
senderName: Alice
|
||||
text:
|
||||
expr: config.prompt
|
||||
- waitForNoOutbound:
|
||||
quietMs:
|
||||
expr: liveTurnTimeoutMs(env, 30000)
|
||||
- set: scenarioRequests
|
||||
value:
|
||||
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).slice(requestCountBefore).filter((request) => String(request.allInputText ?? '').includes(config.promptSnippet)) : []"
|
||||
- assert:
|
||||
expr: "!env.mock || scenarioRequests.length > 0"
|
||||
message: expected mock request evidence that the turn actually ran
|
||||
- assert:
|
||||
expr: "!env.mock || scenarioRequests.every((request) => request.plannedToolName !== 'message')"
|
||||
message:
|
||||
expr: "`model should not have planned the message tool, saw ${JSON.stringify(scenarioRequests.map((request) => request.plannedToolName ?? null))}`"
|
||||
- set: privateFinalLog
|
||||
value:
|
||||
expr: "String(readGatewayLogs() ?? '').slice(logCursor)"
|
||||
- set: privateFinalLine
|
||||
value:
|
||||
expr: "(privateFinalLog.split('\\n').find((line) => line.includes(config.privateFinalLogNeedle)) ?? '').trim()"
|
||||
- assert:
|
||||
expr: "privateFinalLog.includes(config.privateFinalLogNeedle)"
|
||||
message:
|
||||
expr: "`expected the gateway to log ${config.privateFinalLogNeedle} after a substantive private message_tool_only reply, but it was absent`"
|
||||
detailsExpr: "`no-outbound private final; WARN logged=${privateFinalLog.includes(config.privateFinalLogNeedle)}; mock requests=${scenarioRequests.length}; gateway log: ${privateFinalLine}`"
|
||||
120
qa/scenarios/channels/native-command-session-target.yaml
Normal file
120
qa/scenarios/channels/native-command-session-target.yaml
Normal file
@@ -0,0 +1,120 @@
|
||||
title: Native command active session target evidence
|
||||
|
||||
scenario:
|
||||
id: native-command-session-target
|
||||
surface: channel-framework
|
||||
category: channel-framework.channel-actions-commands-and-approvals
|
||||
coverage:
|
||||
primary:
|
||||
- channels.native-command-session-target
|
||||
secondary:
|
||||
- channels.native-commands
|
||||
objective: Verify a channel-native `/stop` command aborts the active routed conversation session instead of its separate slash-command session.
|
||||
successCriteria:
|
||||
- A real delayed agent turn is active on the routed channel conversation session.
|
||||
- The selected transport sends a provider-native command that targets the routed conversation session.
|
||||
- Native `/stop` aborts the active turn, returns the abort acknowledgement, and unblocks the next turn.
|
||||
docsRefs:
|
||||
- docs/channels/qa-channel.md
|
||||
- docs/channels/telegram.md
|
||||
- docs/help/testing.md
|
||||
codeRefs:
|
||||
- extensions/qa-channel/src/inbound.ts
|
||||
- extensions/qa-lab/src/crabline-transport.ts
|
||||
- extensions/telegram/src/bot-native-commands.ts
|
||||
- src/channels/native-command-session-targets.ts
|
||||
- src/auto-reply/reply/abort.ts
|
||||
execution:
|
||||
kind: flow
|
||||
channel: telegram
|
||||
suiteIsolation: isolated
|
||||
isolationReason: Waits for the one active routed session before interrupting it with a provider-native command.
|
||||
summary: Start a real delayed channel turn, abort it through native `/stop`, then prove the conversation is unblocked.
|
||||
config:
|
||||
requiredProviderMode: mock-openai
|
||||
conversationId: native-stop-target
|
||||
senderId: qa-native-operator
|
||||
delayedPrompt: "Subagent recovery worker native command target proof. Wait until stopped."
|
||||
abortReplyNeedle: Agent was aborted
|
||||
recoveryMarker: QA-NATIVE-STOP-RECOVERY-OK
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: native stop targets the active conversation session
|
||||
actions:
|
||||
- assert:
|
||||
expr: "env.providerMode === config.requiredProviderMode"
|
||||
message: this deterministic active-run proof requires mock-openai
|
||||
- call: waitForGatewayHealthy
|
||||
args:
|
||||
- ref: env
|
||||
- 60000
|
||||
- call: waitForTransportReady
|
||||
args:
|
||||
- ref: env
|
||||
- 60000
|
||||
- resetTransport: true
|
||||
- sendInbound:
|
||||
conversation:
|
||||
id:
|
||||
expr: config.conversationId
|
||||
kind: direct
|
||||
senderId:
|
||||
expr: config.senderId
|
||||
senderName: QA Native Operator
|
||||
text:
|
||||
expr: config.delayedPrompt
|
||||
- call: waitForCondition
|
||||
saveAs: activeSession
|
||||
args:
|
||||
- lambda:
|
||||
async: true
|
||||
expr: "env.gateway.call('sessions.list', {}).then((result) => result.sessions?.find((session) => session.hasActiveRun === true))"
|
||||
- expr: liveTurnTimeoutMs(env, 15000)
|
||||
- 100
|
||||
- set: startIndex
|
||||
value:
|
||||
expr: "state.getSnapshot().messages.filter((message) => message.direction === 'outbound').length"
|
||||
- sendNativeCommand:
|
||||
command: stop
|
||||
conversation:
|
||||
id:
|
||||
expr: config.conversationId
|
||||
kind: direct
|
||||
senderId:
|
||||
expr: config.senderId
|
||||
senderName: QA Native Operator
|
||||
- waitForOutbound:
|
||||
conversation:
|
||||
id:
|
||||
expr: config.conversationId
|
||||
kind: direct
|
||||
sinceIndex:
|
||||
ref: startIndex
|
||||
textIncludes:
|
||||
expr: config.abortReplyNeedle
|
||||
timeoutMs: 15000
|
||||
saveAs: abortReply
|
||||
- sendInbound:
|
||||
conversation:
|
||||
id:
|
||||
expr: config.conversationId
|
||||
kind: direct
|
||||
senderId:
|
||||
expr: config.senderId
|
||||
senderName: QA Native Operator
|
||||
text:
|
||||
expr: "`Reply exactly: ${config.recoveryMarker}`"
|
||||
- waitForOutbound:
|
||||
conversation:
|
||||
id:
|
||||
expr: config.conversationId
|
||||
kind: direct
|
||||
sinceIndex:
|
||||
ref: startIndex
|
||||
textIncludes:
|
||||
expr: config.recoveryMarker
|
||||
timeoutMs:
|
||||
expr: liveTurnTimeoutMs(env, 60000)
|
||||
saveAs: recoveryReply
|
||||
detailsExpr: "`native command reply=${abortReply.text}; recovery reply=${recoveryReply.text}`"
|
||||
121
qa/scenarios/channels/qa-channel-reconnect-dedupe.yaml
Normal file
121
qa/scenarios/channels/qa-channel-reconnect-dedupe.yaml
Normal file
@@ -0,0 +1,121 @@
|
||||
title: QA channel reconnect dedupe
|
||||
|
||||
scenario:
|
||||
id: qa-channel-reconnect-dedupe
|
||||
surface: channel
|
||||
coverage:
|
||||
primary:
|
||||
- channels.reconnect
|
||||
secondary:
|
||||
- channels.dedup
|
||||
- runtime.delivery
|
||||
objective: Verify qa-channel readiness polling keeps prior delivery stable and does not replay the last outbound message.
|
||||
successCriteria:
|
||||
- Agent replies once before a reconnect-style readiness cycle.
|
||||
- qa-channel reports ready again without replaying prior outbound delivery.
|
||||
- Follow-up delivery produces one new reply without duplicating the first reply.
|
||||
docsRefs:
|
||||
- docs/channels/qa-channel.md
|
||||
- docs/gateway/configuration.md
|
||||
codeRefs:
|
||||
- extensions/qa-lab/src/qa-channel-transport.ts
|
||||
- extensions/qa-lab/src/bus-state.ts
|
||||
- extensions/qa-lab/src/suite-runtime-gateway.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Verify qa-channel readiness recovery does not duplicate old outbound delivery.
|
||||
config:
|
||||
requiredChannelDriver: qa-channel
|
||||
firstPrompt: "@openclaw Reconnect dedupe setup marker. Reply exactly: RECONNECT-FIRST-OK"
|
||||
secondPrompt: "@openclaw Reconnect dedupe follow-up marker. Reply exactly: RECONNECT-SECOND-OK"
|
||||
firstMarker: RECONNECT-FIRST-OK
|
||||
secondMarker: RECONNECT-SECOND-OK
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: reconnects without replaying prior outbound
|
||||
actions:
|
||||
- call: waitForGatewayHealthy
|
||||
args:
|
||||
- ref: env
|
||||
- 60000
|
||||
- call: waitForQaChannelReady
|
||||
args:
|
||||
- ref: env
|
||||
- 60000
|
||||
- call: reset
|
||||
- set: sessionKey
|
||||
value:
|
||||
expr: "`agent:qa:channel-reconnect:${randomUUID().slice(0, 8)}`"
|
||||
- call: runAgentPrompt
|
||||
args:
|
||||
- ref: env
|
||||
- sessionKey:
|
||||
ref: sessionKey
|
||||
to: channel:qa-room
|
||||
message:
|
||||
expr: config.firstPrompt
|
||||
timeoutMs:
|
||||
expr: liveTurnTimeoutMs(env, 45000)
|
||||
- call: waitForOutboundMessage
|
||||
saveAs: firstOutbound
|
||||
args:
|
||||
- ref: state
|
||||
- lambda:
|
||||
params: [candidate]
|
||||
expr: "candidate.conversation.id === 'qa-room' && candidate.direction === 'outbound' && String(candidate.text ?? '').includes(config.firstMarker)"
|
||||
- expr: liveTurnTimeoutMs(env, 60000)
|
||||
- set: beforeRestartCursor
|
||||
value:
|
||||
expr: state.getSnapshot().messages.length
|
||||
- call: sleep
|
||||
args:
|
||||
- 1000
|
||||
- call: waitForQaChannelReady
|
||||
args:
|
||||
- ref: env
|
||||
- 60000
|
||||
- set: firstMatchesBeforeFollowup
|
||||
value:
|
||||
expr: "state.getSnapshot().messages.filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-room')"
|
||||
- assert:
|
||||
expr: "firstMatchesBeforeFollowup.length === 1 && String(firstMatchesBeforeFollowup[0]?.text ?? '').includes(config.firstMarker)"
|
||||
message:
|
||||
expr: "`readiness cycle should preserve exactly one marked first reply, saw ${firstMatchesBeforeFollowup.length}; transcript=${formatTransportTranscript(state, { conversationId: 'qa-room' })}`"
|
||||
- call: runAgentPrompt
|
||||
args:
|
||||
- ref: env
|
||||
- sessionKey:
|
||||
ref: sessionKey
|
||||
to: channel:qa-room
|
||||
message:
|
||||
expr: config.secondPrompt
|
||||
timeoutMs:
|
||||
expr: liveTurnTimeoutMs(env, 45000)
|
||||
- call: waitForOutboundMessage
|
||||
saveAs: secondOutbound
|
||||
args:
|
||||
- ref: state
|
||||
- lambda:
|
||||
params: [candidate]
|
||||
expr: "candidate.conversation.id === 'qa-room' && candidate.direction === 'outbound' && String(candidate.text ?? '').includes(config.secondMarker)"
|
||||
- expr: liveTurnTimeoutMs(env, 60000)
|
||||
- sinceIndex:
|
||||
ref: beforeRestartCursor
|
||||
- set: snapshot
|
||||
value:
|
||||
expr: state.getSnapshot()
|
||||
- set: firstMatches
|
||||
value:
|
||||
expr: "snapshot.messages.slice(0, beforeRestartCursor).filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-room' && String(candidate.text ?? '').includes(config.firstMarker))"
|
||||
- set: secondMatches
|
||||
value:
|
||||
expr: "snapshot.messages.slice(beforeRestartCursor).filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-room' && String(candidate.text ?? '').includes(config.secondMarker))"
|
||||
- set: postRestartOutbounds
|
||||
value:
|
||||
expr: "snapshot.messages.slice(beforeRestartCursor).filter((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-room')"
|
||||
- assert:
|
||||
expr: "firstMatches.length === 1 && secondMatches.length === 1 && postRestartOutbounds.length === 1 && !postRestartOutbounds.some((candidate) => String(candidate.text ?? '').includes(config.firstMarker))"
|
||||
message:
|
||||
expr: "`expected one marked pre-restart reply and exactly one marked post-restart reply without replaying the first marker; first=${firstMatches.length} second=${secondMatches.length} post=${postRestartOutbounds.length}; transcript=${formatTransportTranscript(state, { conversationId: 'qa-room' })}`"
|
||||
detailsExpr: "`before=${firstOutbound.text}\\nafter=${secondOutbound.text}`"
|
||||
79
qa/scenarios/channels/reaction-edit-delete.yaml
Normal file
79
qa/scenarios/channels/reaction-edit-delete.yaml
Normal file
@@ -0,0 +1,79 @@
|
||||
title: Reaction, edit, delete lifecycle
|
||||
|
||||
scenario:
|
||||
id: reaction-edit-delete
|
||||
surface: message-actions
|
||||
coverage:
|
||||
primary:
|
||||
- channels.message-actions
|
||||
secondary:
|
||||
- channels.qa-channel
|
||||
objective: Verify the agent can use channel-owned message actions and that the QA transcript reflects them.
|
||||
successCriteria:
|
||||
- Agent adds at least one reaction.
|
||||
- Agent edits or replaces a message when asked.
|
||||
- Transcript shows the action lifecycle correctly.
|
||||
docsRefs:
|
||||
- docs/channels/qa-channel.md
|
||||
codeRefs:
|
||||
- extensions/qa-channel/src/channel-actions.ts
|
||||
- extensions/qa-lab/src/self-check-scenario.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Verify the agent can use channel-owned message actions and that the QA transcript reflects them.
|
||||
config:
|
||||
requiredChannelDriver: qa-channel
|
||||
target: "channel:qa-room"
|
||||
seedText: "seed message"
|
||||
editedText: "seed message (edited)"
|
||||
reactionEmoji: "white_check_mark"
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: records reaction, edit, and delete actions
|
||||
actions:
|
||||
- call: reset
|
||||
- call: state.addOutboundMessage
|
||||
saveAs: seed
|
||||
args:
|
||||
- to:
|
||||
expr: config.target
|
||||
text:
|
||||
expr: config.seedText
|
||||
- call: handleQaAction
|
||||
args:
|
||||
- env:
|
||||
ref: env
|
||||
action: react
|
||||
args:
|
||||
messageId:
|
||||
expr: seed.id
|
||||
emoji:
|
||||
expr: config.reactionEmoji
|
||||
- call: handleQaAction
|
||||
args:
|
||||
- env:
|
||||
ref: env
|
||||
action: edit
|
||||
args:
|
||||
messageId:
|
||||
expr: seed.id
|
||||
text:
|
||||
expr: config.editedText
|
||||
- call: handleQaAction
|
||||
args:
|
||||
- env:
|
||||
ref: env
|
||||
action: delete
|
||||
args:
|
||||
messageId:
|
||||
expr: seed.id
|
||||
- call: state.readMessage
|
||||
saveAs: message
|
||||
args:
|
||||
- messageId:
|
||||
expr: seed.id
|
||||
- assert:
|
||||
expr: "message.reactions.length > 0 && message.deleted && message.text.includes('(edited)')"
|
||||
message: message lifecycle did not persist
|
||||
detailsExpr: message.text
|
||||
77
qa/scenarios/channels/thread-follow-up.yaml
Normal file
77
qa/scenarios/channels/thread-follow-up.yaml
Normal file
@@ -0,0 +1,77 @@
|
||||
title: Threaded follow-up
|
||||
|
||||
scenario:
|
||||
id: thread-follow-up
|
||||
surface: thread
|
||||
coverage:
|
||||
primary:
|
||||
- channels.threads
|
||||
- channels.thread-parent-child-placement
|
||||
secondary:
|
||||
- channels.qa-channel
|
||||
objective: Verify the agent can keep follow-up work inside a thread and not leak context into the root channel.
|
||||
successCriteria:
|
||||
- Agent creates or uses a thread for deeper work.
|
||||
- Follow-up messages stay attached to the thread.
|
||||
- Thread report references the correct prior context.
|
||||
docsRefs:
|
||||
- docs/channels/qa-channel.md
|
||||
- docs/channels/group-messages.md
|
||||
codeRefs:
|
||||
- extensions/qa-channel/src/protocol.ts
|
||||
- extensions/qa-lab/src/bus-state.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Verify the agent can keep follow-up work inside a thread and not leak context into the root channel.
|
||||
config:
|
||||
requiredChannelDriver: qa-channel
|
||||
prompt: "@openclaw reply in one short sentence inside this thread only. Do not use ACP or any external runtime. Confirm you stayed in-thread."
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: keeps follow-up inside the thread
|
||||
actions:
|
||||
- call: reset
|
||||
- call: handleQaAction
|
||||
saveAs: threadPayload
|
||||
args:
|
||||
- env:
|
||||
ref: env
|
||||
action: thread-create
|
||||
args:
|
||||
channelId: qa-room
|
||||
title: QA deep dive
|
||||
- set: threadId
|
||||
value:
|
||||
expr: "threadPayload?.thread?.id"
|
||||
- assert:
|
||||
expr: "Boolean(threadId)"
|
||||
message: missing thread id
|
||||
- sendInbound:
|
||||
conversation:
|
||||
id: qa-room
|
||||
kind: channel
|
||||
title: QA Room
|
||||
senderId: alice
|
||||
senderName: Alice
|
||||
text:
|
||||
expr: config.prompt
|
||||
threadId:
|
||||
ref: threadId
|
||||
threadTitle: QA deep dive
|
||||
- call: waitForOutboundMessage
|
||||
saveAs: outbound
|
||||
args:
|
||||
- ref: state
|
||||
- lambda:
|
||||
params: [candidate]
|
||||
expr: "candidate.conversation.id === 'qa-room' && candidate.threadId === threadId"
|
||||
- expr: "env.providerMode === 'mock-openai' ? 15000 : 45000"
|
||||
- assert:
|
||||
expr: "!state.getSnapshot().messages.some((candidate) => candidate.direction === 'outbound' && candidate.conversation.id === 'qa-room' && !candidate.threadId)"
|
||||
message: thread reply leaked into root channel
|
||||
- assert:
|
||||
expr: "!['acp backend','acpx','not configured'].some((needle) => normalizeLowercaseStringOrEmpty(outbound.text).includes(needle))"
|
||||
message:
|
||||
expr: "`thread reply fell back to ACP error: ${outbound.text}`"
|
||||
detailsExpr: outbound.text
|
||||
106
qa/scenarios/channels/webchat-direct-reply-routing.yaml
Normal file
106
qa/scenarios/channels/webchat-direct-reply-routing.yaml
Normal file
@@ -0,0 +1,106 @@
|
||||
title: WebChat direct reply routing
|
||||
|
||||
scenario:
|
||||
id: webchat-direct-reply-routing
|
||||
surface: qa-channel
|
||||
runtimeParityTier: live-only
|
||||
coverage:
|
||||
primary:
|
||||
- runtime.direct-reply-routing
|
||||
secondary:
|
||||
- tools.message
|
||||
- channels.webchat
|
||||
objective: Verify a current-chat reply is delivered as assistant text, not by calling `message(action=send)` and ending with `Sent.`.
|
||||
gatewayConfigPatch:
|
||||
session:
|
||||
dmScope: per-channel-peer
|
||||
successCriteria:
|
||||
- The visible outbound reply contains the requested marker exactly once.
|
||||
- The session transcript does not include a `message(action=send)` call followed by final assistant text `Sent.`.
|
||||
docsRefs:
|
||||
- docs/concepts/qa-e2e-automation.md
|
||||
- qa/scenarios/index.yaml
|
||||
codeRefs:
|
||||
- extensions/qa-lab/src/suite-runtime-agent-session.ts
|
||||
- extensions/qa-lab/src/gateway-log-sentinel.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Run a direct current-chat reply and inspect the actual transcript for self-message routing.
|
||||
config:
|
||||
conversationId: qa-operator
|
||||
expectedMarker: WEBCHAT-DIRECT-REPLY-OK
|
||||
promptSnippet: Reply exactly
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: replies directly instead of sending a self-message
|
||||
actions:
|
||||
- call: waitForGatewayHealthy
|
||||
args:
|
||||
- ref: env
|
||||
- 60000
|
||||
- call: waitForQaChannelReady
|
||||
args:
|
||||
- ref: env
|
||||
- 60000
|
||||
- call: reset
|
||||
- set: conversationId
|
||||
value:
|
||||
expr: config.conversationId
|
||||
- set: delivery
|
||||
value:
|
||||
expr: "transport.buildAgentDelivery({ target: `dm:${conversationId}` })"
|
||||
- set: sessionKey
|
||||
value:
|
||||
expr: "buildAgentSessionKey({ agentId: 'qa', channel: delivery.channel, accountId: transport.accountId, peer: { kind: 'direct', id: delivery.replyTo }, dmScope: env.cfg.session?.dmScope, identityLinks: env.cfg.session?.identityLinks })"
|
||||
- set: startIndex
|
||||
value:
|
||||
expr: state.getSnapshot().messages.length
|
||||
- set: requestCountBefore
|
||||
value:
|
||||
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).length : 0"
|
||||
- sendInbound:
|
||||
conversation:
|
||||
id:
|
||||
ref: conversationId
|
||||
kind: direct
|
||||
senderId:
|
||||
ref: conversationId
|
||||
senderName: WebChat QA
|
||||
text:
|
||||
expr: "`Reply exactly \\`${config.expectedMarker}\\` in this current chat. Do not call the message tool.`"
|
||||
- try:
|
||||
actions:
|
||||
- call: waitForCondition
|
||||
saveAs: scenarioRequest
|
||||
args:
|
||||
- lambda:
|
||||
async: true
|
||||
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).slice(requestCountBefore).find((request) => String(request.allInputText ?? '').includes(config.promptSnippet)) : true"
|
||||
- expr: liveTurnTimeoutMs(env, 60000)
|
||||
- 500
|
||||
- call: waitForOutboundMessage
|
||||
saveAs: outbound
|
||||
args:
|
||||
- ref: state
|
||||
- lambda:
|
||||
params: [candidate]
|
||||
expr: "candidate.conversation.id === conversationId && normalizeLowercaseStringOrEmpty(candidate.text).includes(normalizeLowercaseStringOrEmpty(config.expectedMarker))"
|
||||
- expr: liveTurnTimeoutMs(env, 60000)
|
||||
- sinceIndex:
|
||||
ref: startIndex
|
||||
catchAs: directReplyError
|
||||
catch:
|
||||
- set: directReplyDebugRequests
|
||||
value:
|
||||
expr: "env.mock ? (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).slice(requestCountBefore).map((request) => ({ plannedToolName: request.plannedToolName ?? null, plannedToolArgs: request.plannedToolArgs ?? null, allInputText: String(request.allInputText ?? '').slice(0, 400), finalText: String(request.finalText ?? '').slice(0, 200), toolOutput: request.toolOutput ? String(request.toolOutput).slice(0, 200) : null })) : []"
|
||||
- throw:
|
||||
expr: "`direct reply marker missing: ${directReplyError?.message ?? directReplyError}; transcript=${formatTransportTranscript(state, { conversationId })}; requests=${JSON.stringify(directReplyDebugRequests)}`"
|
||||
- set: transcriptSummary
|
||||
value:
|
||||
expr: "await readSessionTranscriptSummary(env, sessionKey)"
|
||||
- assert:
|
||||
expr: "!transcriptSummary.hasDirectReplySelfMessage"
|
||||
message:
|
||||
expr: "`assistant self-sent direct reply through message(action=send); finalText=${transcriptSummary.finalText}`"
|
||||
detailsExpr: outbound.text
|
||||
Reference in New Issue
Block a user