Vendor OpenClaw source as Adolf fork baseline
Some checks failed
ClawSweeper Dispatch / dispatch (push) Has been cancelled
CodeQL / Security High (actions) (push) Has been cancelled
CodeQL / Security High (channel-runtime-boundary) (push) Has been cancelled
CodeQL / Security High (core-auth-secrets) (push) Has been cancelled
CodeQL / Security High (mcp-process-tool-boundary) (push) Has been cancelled
CodeQL / Security High (network-ssrf-boundary) (push) Has been cancelled
CodeQL / Security High (plugin-trust-boundary) (push) Has been cancelled
CodeQL / Security High (process-exec-boundary) (push) Has been cancelled
Docs Sync Publish Repo / sync-publish-repo (push) Has been cancelled
Docs / docs (push) Has been cancelled
OpenClaw Stable Main Closeout / Resolve stable release closeout inputs (push) Has been cancelled
OpenClaw Stable Main Closeout / Verify stable main closeout (push) Has been cancelled
Workflow Sanity / no-tabs (push) Has been cancelled
Workflow Sanity / actionlint (push) Has been cancelled
Workflow Sanity / generated-doc-baselines (push) Has been cancelled
CI / runner-admission (push) Has been cancelled
CI / preflight (push) Has been cancelled
CI / security-fast (push) Has been cancelled
CI / pnpm-store-warmup (push) Has been cancelled
CI / build-artifacts (push) Has been cancelled
CI / native-i18n (push) Has been cancelled
CI / ${{ matrix.check_name }} (push) Has been cancelled
CI / ${{ matrix.checkName }} (push) Has been cancelled
CI / checks-node-compat-node22 (push) Has been cancelled
CI / check-bundled-channel-config-metadata (push) Has been cancelled
CI / check-dependencies (push) Has been cancelled
CI / check-guards (push) Has been cancelled
CI / check-lint (push) Has been cancelled
CI / check-prod-types (push) Has been cancelled
CI / check-shrinkwrap (push) Has been cancelled
CI / check-test-types (push) Has been cancelled
CI / check-additional-boundaries-a (push) Has been cancelled
CI / check-additional-boundaries-bcd (push) Has been cancelled
CI / check-additional-extension-bundled (push) Has been cancelled
CI / check-additional-extension-channels (push) Has been cancelled
CI / check-additional-extension-package-boundary (push) Has been cancelled
CI / check-additional-runtime-topology-architecture (push) Has been cancelled
CI / check-session-accessor-boundary (push) Has been cancelled
CI / check-session-transcript-reader-boundary (push) Has been cancelled
CI / check-docs (push) Has been cancelled
CI / skills-python (push) Has been cancelled
CI / macos-swift (push) Has been cancelled
CI / ios-build (push) Has been cancelled
CI / ci-timings-summary (push) Has been cancelled
Native App Locale Refresh / Refresh native fa (push) Has been cancelled
Native App Locale Refresh / Refresh native fr (push) Has been cancelled
Native App Locale Refresh / Refresh native hi (push) Has been cancelled
Native App Locale Refresh / Refresh native id (push) Has been cancelled
Native App Locale Refresh / Refresh native it (push) Has been cancelled
Native App Locale Refresh / Refresh native ja-JP (push) Has been cancelled
Control UI Locale Refresh / plan (push) Has been cancelled
Control UI Locale Refresh / Refresh ${{ matrix.locale }} (push) Has been cancelled
Control UI Locale Refresh / Commit control UI locale refresh (push) Has been cancelled
Live Media Runner Image / Build live media runner image (push) Has been cancelled
Native App Locale Refresh / Refresh native ar (push) Has been cancelled
Native App Locale Refresh / Refresh native de (push) Has been cancelled
Native App Locale Refresh / Refresh native es (push) Has been cancelled
Native App Locale Refresh / Refresh native ko (push) Has been cancelled
Native App Locale Refresh / Refresh native nl (push) Has been cancelled
Native App Locale Refresh / Refresh native pl (push) Has been cancelled
Native App Locale Refresh / Refresh native pt-BR (push) Has been cancelled
Native App Locale Refresh / Refresh native ru (push) Has been cancelled
Native App Locale Refresh / Refresh native sv (push) Has been cancelled
Native App Locale Refresh / Refresh native th (push) Has been cancelled
Native App Locale Refresh / Refresh native tr (push) Has been cancelled
Native App Locale Refresh / Refresh native uk (push) Has been cancelled
Native App Locale Refresh / Refresh native vi (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-CN (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-TW (push) Has been cancelled
Native App Locale Refresh / Commit native locale refresh (push) Has been cancelled
Plugin Init Scaffold Validation / Validate provider scaffold (push) Has been cancelled
Plugin NPM Release / preview_plugins_npm (push) Has been cancelled
Plugin NPM Release / Validate release publish approval (push) Has been cancelled
Plugin NPM Release / preview_plugin_pack (push) Has been cancelled
Plugin NPM Release / publish_plugins_npm (push) Has been cancelled
Sandbox Common Smoke / sandbox-common-smoke (push) Has been cancelled
Website Installer Sync / static (push) Has been cancelled
Website Installer Sync / linux-docker (push) Has been cancelled
Website Installer Sync / macos-installer (push) Has been cancelled
Website Installer Sync / windows-installer (push) Has been cancelled
Website Installer Sync / sync-website (push) Has been cancelled
Some checks failed
ClawSweeper Dispatch / dispatch (push) Has been cancelled
CodeQL / Security High (actions) (push) Has been cancelled
CodeQL / Security High (channel-runtime-boundary) (push) Has been cancelled
CodeQL / Security High (core-auth-secrets) (push) Has been cancelled
CodeQL / Security High (mcp-process-tool-boundary) (push) Has been cancelled
CodeQL / Security High (network-ssrf-boundary) (push) Has been cancelled
CodeQL / Security High (plugin-trust-boundary) (push) Has been cancelled
CodeQL / Security High (process-exec-boundary) (push) Has been cancelled
Docs Sync Publish Repo / sync-publish-repo (push) Has been cancelled
Docs / docs (push) Has been cancelled
OpenClaw Stable Main Closeout / Resolve stable release closeout inputs (push) Has been cancelled
OpenClaw Stable Main Closeout / Verify stable main closeout (push) Has been cancelled
Workflow Sanity / no-tabs (push) Has been cancelled
Workflow Sanity / actionlint (push) Has been cancelled
Workflow Sanity / generated-doc-baselines (push) Has been cancelled
CI / runner-admission (push) Has been cancelled
CI / preflight (push) Has been cancelled
CI / security-fast (push) Has been cancelled
CI / pnpm-store-warmup (push) Has been cancelled
CI / build-artifacts (push) Has been cancelled
CI / native-i18n (push) Has been cancelled
CI / ${{ matrix.check_name }} (push) Has been cancelled
CI / ${{ matrix.checkName }} (push) Has been cancelled
CI / checks-node-compat-node22 (push) Has been cancelled
CI / check-bundled-channel-config-metadata (push) Has been cancelled
CI / check-dependencies (push) Has been cancelled
CI / check-guards (push) Has been cancelled
CI / check-lint (push) Has been cancelled
CI / check-prod-types (push) Has been cancelled
CI / check-shrinkwrap (push) Has been cancelled
CI / check-test-types (push) Has been cancelled
CI / check-additional-boundaries-a (push) Has been cancelled
CI / check-additional-boundaries-bcd (push) Has been cancelled
CI / check-additional-extension-bundled (push) Has been cancelled
CI / check-additional-extension-channels (push) Has been cancelled
CI / check-additional-extension-package-boundary (push) Has been cancelled
CI / check-additional-runtime-topology-architecture (push) Has been cancelled
CI / check-session-accessor-boundary (push) Has been cancelled
CI / check-session-transcript-reader-boundary (push) Has been cancelled
CI / check-docs (push) Has been cancelled
CI / skills-python (push) Has been cancelled
CI / macos-swift (push) Has been cancelled
CI / ios-build (push) Has been cancelled
CI / ci-timings-summary (push) Has been cancelled
Native App Locale Refresh / Refresh native fa (push) Has been cancelled
Native App Locale Refresh / Refresh native fr (push) Has been cancelled
Native App Locale Refresh / Refresh native hi (push) Has been cancelled
Native App Locale Refresh / Refresh native id (push) Has been cancelled
Native App Locale Refresh / Refresh native it (push) Has been cancelled
Native App Locale Refresh / Refresh native ja-JP (push) Has been cancelled
Control UI Locale Refresh / plan (push) Has been cancelled
Control UI Locale Refresh / Refresh ${{ matrix.locale }} (push) Has been cancelled
Control UI Locale Refresh / Commit control UI locale refresh (push) Has been cancelled
Live Media Runner Image / Build live media runner image (push) Has been cancelled
Native App Locale Refresh / Refresh native ar (push) Has been cancelled
Native App Locale Refresh / Refresh native de (push) Has been cancelled
Native App Locale Refresh / Refresh native es (push) Has been cancelled
Native App Locale Refresh / Refresh native ko (push) Has been cancelled
Native App Locale Refresh / Refresh native nl (push) Has been cancelled
Native App Locale Refresh / Refresh native pl (push) Has been cancelled
Native App Locale Refresh / Refresh native pt-BR (push) Has been cancelled
Native App Locale Refresh / Refresh native ru (push) Has been cancelled
Native App Locale Refresh / Refresh native sv (push) Has been cancelled
Native App Locale Refresh / Refresh native th (push) Has been cancelled
Native App Locale Refresh / Refresh native tr (push) Has been cancelled
Native App Locale Refresh / Refresh native uk (push) Has been cancelled
Native App Locale Refresh / Refresh native vi (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-CN (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-TW (push) Has been cancelled
Native App Locale Refresh / Commit native locale refresh (push) Has been cancelled
Plugin Init Scaffold Validation / Validate provider scaffold (push) Has been cancelled
Plugin NPM Release / preview_plugins_npm (push) Has been cancelled
Plugin NPM Release / Validate release publish approval (push) Has been cancelled
Plugin NPM Release / preview_plugin_pack (push) Has been cancelled
Plugin NPM Release / publish_plugins_npm (push) Has been cancelled
Sandbox Common Smoke / sandbox-common-smoke (push) Has been cancelled
Website Installer Sync / static (push) Has been cancelled
Website Installer Sync / linux-docker (push) Has been cancelled
Website Installer Sync / macos-installer (push) Has been cancelled
Website Installer Sync / windows-installer (push) Has been cancelled
Website Installer Sync / sync-website (push) Has been cancelled
Adolf is a fork/vendored clone of github.com/openclaw/openclaw (v2026.6.11), free to diverge. Tree copied sans upstream .git; upstream remote added for future syncs. Node pinned to 24 (.nvmrc); engines already require >=22.19. Preserves docs/ARCHITECTURE.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LeqyaxJF2nbRXJtae2kNB2
This commit is contained in:
51
qa/scenarios/runtime/tools/apply-patch.yaml
Normal file
51
qa/scenarios/runtime/tools/apply-patch.yaml
Normal file
@@ -0,0 +1,51 @@
|
||||
title: "Runtime tool fixture — apply-patch"
|
||||
|
||||
scenario:
|
||||
id: runtime-tool-apply-patch
|
||||
surface: runtime-tools
|
||||
runtimeParityTier: standard
|
||||
coverage:
|
||||
primary:
|
||||
- tools.apply-patch
|
||||
objective: Verify apply_patch behavior is tracked across OpenClaw and Codex while Codex owns patching natively.
|
||||
successCriteria:
|
||||
- OpenClaw may expose OpenClaw apply_patch while Codex app-server mode may omit duplicate OpenClaw dynamic apply_patch.
|
||||
- Mock provider apply_patch plans are reported as fixture intent, not as actual runtime tool calls.
|
||||
- The row stays report-only until fault injection uses valid patch-shaped inputs.
|
||||
docsRefs:
|
||||
- qa/scenarios/index.yaml
|
||||
codeRefs:
|
||||
- src/agents/apply-patch.ts
|
||||
- extensions/qa-lab/src/runtime-tool-fixture.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Exercise the apply_patch runtime tool family.
|
||||
config:
|
||||
toolName: apply_patch
|
||||
toolCoverage:
|
||||
family: apply-patch
|
||||
actualTool: apply_patch
|
||||
bucket: codex-native-workspace
|
||||
expectedLayer: codex-native-workspace
|
||||
required: true
|
||||
tracking: "#80320"
|
||||
codexDefaultImpact: P4
|
||||
qaImpact: P2
|
||||
action: fix fixture fault injection
|
||||
reason: Codex app-server intentionally owns apply_patch natively; this fixture still needs valid patch-shaped fault injection before it can prove product behavior.
|
||||
knownHarnessGap:
|
||||
issue: "#80320"
|
||||
reason: Codex-native apply_patch is intentionally not an OpenClaw dynamic tool; QA fault injection still uses synthetic failure-path inputs.
|
||||
promptSnippet: "target=apply_patch"
|
||||
failurePromptSnippet: "failure target=apply_patch"
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: exercises apply_patch happy and failure paths
|
||||
actions:
|
||||
- call: runRuntimeToolFixture
|
||||
saveAs: result
|
||||
args:
|
||||
- ref: env
|
||||
- ref: config
|
||||
detailsExpr: result
|
||||
52
qa/scenarios/runtime/tools/bash.yaml
Normal file
52
qa/scenarios/runtime/tools/bash.yaml
Normal file
@@ -0,0 +1,52 @@
|
||||
title: "Runtime tool fixture — bash"
|
||||
|
||||
scenario:
|
||||
id: runtime-tool-bash
|
||||
surface: runtime-tools
|
||||
runtimeParityTier: standard
|
||||
coverage:
|
||||
primary:
|
||||
- tools.bash
|
||||
objective: Verify shell command behavior is tracked across OpenClaw and Codex while Codex owns exec/process natively.
|
||||
successCriteria:
|
||||
- OpenClaw may expose OpenClaw exec while Codex app-server mode may omit duplicate OpenClaw dynamic exec/process.
|
||||
- Mock provider exec plans are reported as fixture intent, not as actual runtime tool calls.
|
||||
- The row stays report-only until the fixture validates native Codex command behavior directly.
|
||||
docsRefs:
|
||||
- qa/scenarios/index.yaml
|
||||
codeRefs:
|
||||
- src/agents/agent-tools.ts
|
||||
- src/agents/bash-tools.schemas.ts
|
||||
- extensions/qa-lab/src/runtime-tool-fixture.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Exercise the bash family through the OpenClaw exec tool.
|
||||
config:
|
||||
toolName: exec
|
||||
toolCoverage:
|
||||
family: bash
|
||||
actualTool: exec
|
||||
bucket: codex-native-workspace
|
||||
expectedLayer: codex-native-workspace
|
||||
required: true
|
||||
tracking: "#80319"
|
||||
codexDefaultImpact: P4
|
||||
qaImpact: P1
|
||||
action: split native command behavior from OpenClaw dynamic tool parity
|
||||
reason: Codex app-server intentionally owns command execution natively; the fixture must not require OpenClaw dynamic exec exposure.
|
||||
knownHarnessGap:
|
||||
issue: "#80319"
|
||||
reason: QA tool-defaults currently needs native command behavior coverage instead of OpenClaw dynamic exec exposure.
|
||||
promptSnippet: "target=exec"
|
||||
failurePromptSnippet: "failure target=exec"
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: exercises bash happy and failure paths
|
||||
actions:
|
||||
- call: runRuntimeToolFixture
|
||||
saveAs: result
|
||||
args:
|
||||
- ref: env
|
||||
- ref: config
|
||||
detailsExpr: result
|
||||
51
qa/scenarios/runtime/tools/edit.yaml
Normal file
51
qa/scenarios/runtime/tools/edit.yaml
Normal file
@@ -0,0 +1,51 @@
|
||||
title: "Runtime tool fixture — edit"
|
||||
|
||||
scenario:
|
||||
id: runtime-tool-edit
|
||||
surface: runtime-tools
|
||||
runtimeParityTier: standard
|
||||
coverage:
|
||||
primary:
|
||||
- tools.edit
|
||||
objective: Verify targeted edit behavior is tracked across OpenClaw and Codex while Codex owns edit natively.
|
||||
successCriteria:
|
||||
- OpenClaw may expose OpenClaw edit while Codex app-server mode may omit duplicate OpenClaw dynamic edit.
|
||||
- Mock provider edit plans are reported as fixture intent, not as actual runtime tool calls.
|
||||
- The row stays report-only until the fixture validates native Codex edit behavior directly.
|
||||
docsRefs:
|
||||
- qa/scenarios/index.yaml
|
||||
codeRefs:
|
||||
- src/agents/agent-tools.ts
|
||||
- extensions/qa-lab/src/runtime-tool-fixture.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Exercise the edit runtime tool family.
|
||||
config:
|
||||
toolName: edit
|
||||
toolCoverage:
|
||||
family: edit
|
||||
actualTool: edit
|
||||
bucket: codex-native-workspace
|
||||
expectedLayer: codex-native-workspace
|
||||
required: true
|
||||
tracking: "#80319"
|
||||
codexDefaultImpact: P4
|
||||
qaImpact: P1
|
||||
action: split native edit behavior from OpenClaw dynamic tool parity
|
||||
reason: Codex app-server intentionally owns edit natively; the fixture must not require OpenClaw dynamic edit exposure.
|
||||
knownHarnessGap:
|
||||
issue: "#80319"
|
||||
reason: QA tool-defaults currently needs native edit behavior coverage instead of OpenClaw dynamic edit exposure.
|
||||
promptSnippet: "target=edit"
|
||||
failurePromptSnippet: "failure target=edit"
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: exercises edit happy and failure paths
|
||||
actions:
|
||||
- call: runRuntimeToolFixture
|
||||
saveAs: result
|
||||
args:
|
||||
- ref: env
|
||||
- ref: config
|
||||
detailsExpr: result
|
||||
51
qa/scenarios/runtime/tools/exec.yaml
Normal file
51
qa/scenarios/runtime/tools/exec.yaml
Normal file
@@ -0,0 +1,51 @@
|
||||
title: "Runtime tool fixture — exec"
|
||||
|
||||
scenario:
|
||||
id: runtime-tool-exec
|
||||
surface: runtime-tools
|
||||
runtimeParityTier: standard
|
||||
coverage:
|
||||
primary:
|
||||
- tools.exec
|
||||
objective: Verify command execution behavior is tracked across OpenClaw and Codex while Codex owns exec/process natively.
|
||||
successCriteria:
|
||||
- OpenClaw may expose OpenClaw exec while Codex app-server mode may omit duplicate OpenClaw dynamic exec/process.
|
||||
- Mock provider exec plans are reported as fixture intent, not as actual runtime tool calls.
|
||||
- The row stays report-only until the fixture validates native Codex command behavior directly.
|
||||
docsRefs:
|
||||
- qa/scenarios/index.yaml
|
||||
codeRefs:
|
||||
- src/agents/bash-tools.schemas.ts
|
||||
- extensions/qa-lab/src/runtime-tool-fixture.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Exercise the exec runtime tool family.
|
||||
config:
|
||||
toolName: exec
|
||||
toolCoverage:
|
||||
family: exec
|
||||
actualTool: exec
|
||||
bucket: codex-native-workspace
|
||||
expectedLayer: codex-native-workspace
|
||||
required: true
|
||||
tracking: "#80319"
|
||||
codexDefaultImpact: P4
|
||||
qaImpact: P1
|
||||
action: split native command behavior from OpenClaw dynamic tool parity
|
||||
reason: Codex app-server intentionally owns command execution natively; the fixture must not require OpenClaw dynamic exec exposure.
|
||||
knownHarnessGap:
|
||||
issue: "#80319"
|
||||
reason: QA tool-defaults currently needs native command behavior coverage instead of OpenClaw dynamic exec exposure.
|
||||
promptSnippet: "target=exec"
|
||||
failurePromptSnippet: "failure target=exec"
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: exercises exec happy and failure paths
|
||||
actions:
|
||||
- call: runRuntimeToolFixture
|
||||
saveAs: result
|
||||
args:
|
||||
- ref: env
|
||||
- ref: config
|
||||
detailsExpr: result
|
||||
51
qa/scenarios/runtime/tools/fs-list.yaml
Normal file
51
qa/scenarios/runtime/tools/fs-list.yaml
Normal file
@@ -0,0 +1,51 @@
|
||||
title: "Runtime tool fixture — fs.list"
|
||||
|
||||
scenario:
|
||||
id: runtime-tool-fs-list
|
||||
surface: runtime-tools
|
||||
runtimeParityTier: standard
|
||||
coverage:
|
||||
primary:
|
||||
- tools.fs.list
|
||||
objective: Verify directory inspection behavior is tracked through read while Codex owns file inspection natively.
|
||||
successCriteria:
|
||||
- OpenClaw may expose OpenClaw read while Codex app-server mode may omit duplicate OpenClaw dynamic read.
|
||||
- Mock provider read plans are reported as fixture intent, not as actual runtime tool calls.
|
||||
- The row stays report-only until directory fault injection proves native Codex read behavior directly.
|
||||
docsRefs:
|
||||
- qa/scenarios/index.yaml
|
||||
codeRefs:
|
||||
- src/agents/agent-tools.read.ts
|
||||
- extensions/qa-lab/src/runtime-tool-fixture.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Exercise fs.list coverage through the current read tool surface.
|
||||
config:
|
||||
toolName: read
|
||||
toolCoverage:
|
||||
family: fs.list
|
||||
actualTool: read
|
||||
bucket: codex-native-workspace
|
||||
expectedLayer: codex-native-workspace
|
||||
required: true
|
||||
tracking: "#80312"
|
||||
codexDefaultImpact: P4
|
||||
qaImpact: P2
|
||||
action: model native read/list behavior separately from provider-plan capture
|
||||
reason: Codex app-server intentionally owns read natively; current OpenClaw coding surface has no separate list tool.
|
||||
knownHarnessGap:
|
||||
issue: "#80312"
|
||||
reason: QA mock failure-path capture currently reports provider-plan args, not proven Codex native read/list behavior.
|
||||
promptSnippet: "target=read"
|
||||
failurePromptSnippet: "failure target=read"
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: exercises fs.list happy and failure paths
|
||||
actions:
|
||||
- call: runRuntimeToolFixture
|
||||
saveAs: result
|
||||
args:
|
||||
- ref: env
|
||||
- ref: config
|
||||
detailsExpr: result
|
||||
51
qa/scenarios/runtime/tools/fs-read.yaml
Normal file
51
qa/scenarios/runtime/tools/fs-read.yaml
Normal file
@@ -0,0 +1,51 @@
|
||||
title: "Runtime tool fixture — fs.read"
|
||||
|
||||
scenario:
|
||||
id: runtime-tool-fs-read
|
||||
surface: runtime-tools
|
||||
runtimeParityTier: standard
|
||||
coverage:
|
||||
primary:
|
||||
- tools.fs.read
|
||||
objective: Verify file read behavior is tracked across OpenClaw and Codex while Codex owns read natively.
|
||||
successCriteria:
|
||||
- OpenClaw may expose OpenClaw read while Codex app-server mode may omit duplicate OpenClaw dynamic read.
|
||||
- Mock provider read plans are reported as fixture intent, not as actual runtime tool calls.
|
||||
- The row stays report-only until failure-path injection proves native Codex read behavior directly.
|
||||
docsRefs:
|
||||
- qa/scenarios/index.yaml
|
||||
codeRefs:
|
||||
- src/agents/agent-tools.read.ts
|
||||
- extensions/qa-lab/src/runtime-tool-fixture.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Exercise the read runtime tool family.
|
||||
config:
|
||||
toolName: read
|
||||
toolCoverage:
|
||||
family: fs.read
|
||||
actualTool: read
|
||||
bucket: codex-native-workspace
|
||||
expectedLayer: codex-native-workspace
|
||||
required: true
|
||||
tracking: "#80312"
|
||||
codexDefaultImpact: P4
|
||||
qaImpact: P2
|
||||
action: model native read behavior separately from provider-plan capture
|
||||
reason: Codex app-server intentionally owns read natively; QA mock failure-path capture currently reports provider-plan args, not proven Codex native read behavior.
|
||||
knownHarnessGap:
|
||||
issue: "#80312"
|
||||
reason: QA mock failure-path capture currently reports provider-plan args, not proven Codex native read behavior.
|
||||
promptSnippet: "target=read"
|
||||
failurePromptSnippet: "failure target=read"
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: exercises fs.read happy and failure paths
|
||||
actions:
|
||||
- call: runRuntimeToolFixture
|
||||
saveAs: result
|
||||
args:
|
||||
- ref: env
|
||||
- ref: config
|
||||
detailsExpr: result
|
||||
51
qa/scenarios/runtime/tools/fs-write.yaml
Normal file
51
qa/scenarios/runtime/tools/fs-write.yaml
Normal file
@@ -0,0 +1,51 @@
|
||||
title: "Runtime tool fixture — fs.write"
|
||||
|
||||
scenario:
|
||||
id: runtime-tool-fs-write
|
||||
surface: runtime-tools
|
||||
runtimeParityTier: standard
|
||||
coverage:
|
||||
primary:
|
||||
- tools.fs.write
|
||||
objective: Verify file write behavior is tracked across OpenClaw and Codex while Codex owns write natively.
|
||||
successCriteria:
|
||||
- OpenClaw may expose OpenClaw write while Codex app-server mode may omit duplicate OpenClaw dynamic write.
|
||||
- Mock provider write plans are reported as fixture intent, not as actual runtime tool calls.
|
||||
- The row stays report-only until the fixture validates native Codex write behavior directly.
|
||||
docsRefs:
|
||||
- qa/scenarios/index.yaml
|
||||
codeRefs:
|
||||
- src/agents/agent-tools.workspace-paths.test.ts
|
||||
- extensions/qa-lab/src/runtime-tool-fixture.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Exercise the write runtime tool family.
|
||||
config:
|
||||
toolName: write
|
||||
toolCoverage:
|
||||
family: fs.write
|
||||
actualTool: write
|
||||
bucket: codex-native-workspace
|
||||
expectedLayer: codex-native-workspace
|
||||
required: true
|
||||
tracking: "#80319"
|
||||
codexDefaultImpact: P4
|
||||
qaImpact: P1
|
||||
action: split native write behavior from OpenClaw dynamic tool parity
|
||||
reason: Codex app-server intentionally owns write natively; the fixture must not require OpenClaw dynamic write exposure.
|
||||
knownHarnessGap:
|
||||
issue: "#80319"
|
||||
reason: QA tool-defaults currently needs native write behavior coverage instead of OpenClaw dynamic write exposure.
|
||||
promptSnippet: "target=write"
|
||||
failurePromptSnippet: "failure target=write"
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: exercises fs.write happy and failure paths
|
||||
actions:
|
||||
- call: runRuntimeToolFixture
|
||||
saveAs: result
|
||||
args:
|
||||
- ref: env
|
||||
- ref: config
|
||||
detailsExpr: result
|
||||
51
qa/scenarios/runtime/tools/grep.yaml
Normal file
51
qa/scenarios/runtime/tools/grep.yaml
Normal file
@@ -0,0 +1,51 @@
|
||||
title: "Runtime tool fixture — grep"
|
||||
|
||||
scenario:
|
||||
id: runtime-tool-grep
|
||||
surface: runtime-tools
|
||||
runtimeParityTier: standard
|
||||
coverage:
|
||||
primary:
|
||||
- tools.grep
|
||||
objective: Verify grep-style search behavior is tracked through command execution while Codex owns exec/process natively.
|
||||
successCriteria:
|
||||
- OpenClaw may expose OpenClaw exec while Codex app-server mode may omit duplicate OpenClaw dynamic exec/process.
|
||||
- Mock provider exec plans are reported as fixture intent, not as actual runtime tool calls.
|
||||
- The row stays report-only until the fixture validates native Codex search/command behavior directly.
|
||||
docsRefs:
|
||||
- qa/scenarios/index.yaml
|
||||
codeRefs:
|
||||
- src/agents/agent-tools.ts
|
||||
- extensions/qa-lab/src/runtime-tool-fixture.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Exercise grep coverage through the current exec tool surface.
|
||||
config:
|
||||
toolName: exec
|
||||
toolCoverage:
|
||||
family: grep
|
||||
actualTool: exec
|
||||
bucket: codex-native-workspace
|
||||
expectedLayer: codex-native-workspace
|
||||
required: true
|
||||
tracking: "#80319"
|
||||
codexDefaultImpact: P4
|
||||
qaImpact: P1
|
||||
action: split native search/command behavior from OpenClaw dynamic tool parity
|
||||
reason: Codex app-server intentionally owns command execution natively; current OpenClaw coding surface routes grep-style searches through exec.
|
||||
knownHarnessGap:
|
||||
issue: "#80319"
|
||||
reason: QA tool-defaults currently needs native search/command behavior coverage instead of OpenClaw dynamic exec exposure.
|
||||
promptSnippet: "target=exec"
|
||||
failurePromptSnippet: "failure target=exec"
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: exercises grep happy and failure paths
|
||||
actions:
|
||||
- call: runRuntimeToolFixture
|
||||
saveAs: result
|
||||
args:
|
||||
- ref: env
|
||||
- ref: config
|
||||
detailsExpr: result
|
||||
56
qa/scenarios/runtime/tools/image-generate.yaml
Normal file
56
qa/scenarios/runtime/tools/image-generate.yaml
Normal file
@@ -0,0 +1,56 @@
|
||||
title: "Runtime tool fixture — image_generate"
|
||||
|
||||
scenario:
|
||||
id: runtime-tool-image-generate
|
||||
surface: runtime-tools
|
||||
runtimeParityTier: optional
|
||||
coverage:
|
||||
primary:
|
||||
- tools.image-generate
|
||||
objective: Verify image_generate preserves arguments and result shape across OpenClaw and Codex.
|
||||
successCriteria:
|
||||
- Scenario is mock-openai only because live image generation uses async task lifecycle proof outside this direct failure-path fixture.
|
||||
- Effective tools expose image_generate after QA image-generation config is applied.
|
||||
- The mock provider plans exactly one happy-path image_generate call.
|
||||
- The mock provider plans one denied-input failure-path image_generate call.
|
||||
- Runtime parity coverage records async image start/result drift outside the standard direct-loading gate.
|
||||
docsRefs:
|
||||
- docs/tools/image-generation.md
|
||||
codeRefs:
|
||||
- src/agents/tools/image-generate-tool.ts
|
||||
- extensions/qa-lab/src/runtime-tool-fixture.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Exercise the image_generate runtime tool family.
|
||||
config:
|
||||
requiredProviderMode: mock-openai
|
||||
toolName: image_generate
|
||||
ensureImageGeneration: true
|
||||
toolCoverage:
|
||||
family: image_generate
|
||||
actualTool: image_generate
|
||||
bucket: openclaw-dynamic-integration
|
||||
expectedLayer: openclaw-dynamic
|
||||
capabilityLayer: openclaw-dynamic-direct
|
||||
required: false
|
||||
codexDefaultImpact: P4
|
||||
qaImpact: P1
|
||||
action: optional runtime parity gate with async image completion coverage
|
||||
reason: image_generate is an OpenClaw integration tool whose happy path yields for async completion, so standard direct call/result parity would compare different lifecycle phases.
|
||||
promptSnippet: "target=image_generate"
|
||||
failurePromptSnippet: "failure target=image_generate"
|
||||
happyPathOutputRequired: false
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: exercises image_generate happy and failure paths
|
||||
actions:
|
||||
- assert:
|
||||
expr: "env.providerMode === config.requiredProviderMode"
|
||||
message: this seeded scenario is mock-openai only
|
||||
- call: runRuntimeToolFixture
|
||||
saveAs: result
|
||||
args:
|
||||
- ref: env
|
||||
- ref: config
|
||||
detailsExpr: result
|
||||
51
qa/scenarios/runtime/tools/memory-add.yaml
Normal file
51
qa/scenarios/runtime/tools/memory-add.yaml
Normal file
@@ -0,0 +1,51 @@
|
||||
title: "Runtime tool fixture — memory.add"
|
||||
|
||||
scenario:
|
||||
id: runtime-tool-memory-add
|
||||
surface: runtime-tools
|
||||
runtimeParityTier: optional
|
||||
coverage:
|
||||
primary:
|
||||
- tools.memory.add
|
||||
objective: Track memory.add parity once a first-class memory add tool is exposed.
|
||||
successCriteria:
|
||||
- If memory_add is present, the fixture exercises happy and failure paths.
|
||||
- If memory_add is absent, the fixture records the known-broken tracking marker.
|
||||
docsRefs:
|
||||
- qa/scenarios/index.yaml
|
||||
codeRefs:
|
||||
- extensions/memory-lancedb/index.ts
|
||||
- extensions/qa-lab/src/runtime-tool-fixture.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Track memory.add runtime parity coverage.
|
||||
config:
|
||||
toolName: memory_add
|
||||
expectedAvailable: false
|
||||
toolCoverage:
|
||||
family: memory.add
|
||||
actualTool: memory_add
|
||||
bucket: optional-profile-or-plugin
|
||||
expectedLayer: profile-or-plugin
|
||||
required: false
|
||||
tracking: "#80173"
|
||||
codexDefaultImpact: P4
|
||||
qaImpact: P3
|
||||
action: keep optional until memory_add exists in the configured default surface
|
||||
reason: The phase matrix includes memory.add, but the current plugin surface exposes recall/search contracts instead.
|
||||
knownBroken:
|
||||
issue: "#80173"
|
||||
reason: memory_add is not exposed by the current default tool surface.
|
||||
promptSnippet: "target=memory_add"
|
||||
failurePromptSnippet: "failure target=memory_add"
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: exercises or records memory.add coverage
|
||||
actions:
|
||||
- call: runRuntimeToolFixture
|
||||
saveAs: result
|
||||
args:
|
||||
- ref: env
|
||||
- ref: config
|
||||
detailsExpr: result
|
||||
51
qa/scenarios/runtime/tools/memory-recall.yaml
Normal file
51
qa/scenarios/runtime/tools/memory-recall.yaml
Normal file
@@ -0,0 +1,51 @@
|
||||
title: "Runtime tool fixture — memory.recall"
|
||||
|
||||
scenario:
|
||||
id: runtime-tool-memory-recall
|
||||
surface: runtime-tools
|
||||
runtimeParityTier: optional
|
||||
coverage:
|
||||
primary:
|
||||
- tools.memory.recall
|
||||
objective: Verify memory_recall parity when the memory plugin exposes the tool.
|
||||
successCriteria:
|
||||
- If memory_recall is present, the fixture exercises happy and failure paths.
|
||||
- If memory_recall is absent, the fixture records the known-broken tracking marker.
|
||||
docsRefs:
|
||||
- qa/scenarios/index.yaml
|
||||
codeRefs:
|
||||
- extensions/memory-lancedb/index.ts
|
||||
- extensions/qa-lab/src/runtime-tool-fixture.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Exercise or track the memory_recall runtime tool family.
|
||||
config:
|
||||
toolName: memory_recall
|
||||
expectedAvailable: false
|
||||
toolCoverage:
|
||||
family: memory.recall
|
||||
actualTool: memory_recall
|
||||
bucket: optional-profile-or-plugin
|
||||
expectedLayer: profile-or-plugin
|
||||
required: false
|
||||
tracking: "#80173"
|
||||
codexDefaultImpact: P4
|
||||
qaImpact: P3
|
||||
action: keep optional unless memory plugin coverage is explicitly enabled
|
||||
reason: memory_recall is plugin-dependent and absent from some mock QA runs.
|
||||
knownBroken:
|
||||
issue: "#80173"
|
||||
reason: memory_recall is plugin-dependent and absent from the default mock QA run.
|
||||
promptSnippet: "target=memory_recall"
|
||||
failurePromptSnippet: "failure target=memory_recall"
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: exercises or records memory_recall coverage
|
||||
actions:
|
||||
- call: runRuntimeToolFixture
|
||||
saveAs: result
|
||||
args:
|
||||
- ref: env
|
||||
- ref: config
|
||||
detailsExpr: result
|
||||
49
qa/scenarios/runtime/tools/message-tool.yaml
Normal file
49
qa/scenarios/runtime/tools/message-tool.yaml
Normal file
@@ -0,0 +1,49 @@
|
||||
title: "Runtime tool fixture — direct message tool"
|
||||
|
||||
scenario:
|
||||
id: runtime-tool-message-tool
|
||||
surface: runtime-tools
|
||||
runtimeParityTier: optional
|
||||
coverage:
|
||||
primary:
|
||||
- tools.message-tool
|
||||
objective: Report whether a direct message tool is present. The coding-profile session surface normally uses sessions_send instead.
|
||||
successCriteria:
|
||||
- The fixture is report-only when the coding profile does not expose a direct message tool.
|
||||
- If a direct message tool is exposed, the mock provider plans exactly one happy-path message call.
|
||||
- If a direct message tool is exposed, the mock provider plans one denied-input failure-path message call.
|
||||
docsRefs:
|
||||
- qa/scenarios/index.yaml
|
||||
codeRefs:
|
||||
- src/agents/embedded-agent-messaging.ts
|
||||
- src/agents/tools/sessions-send-tool.ts
|
||||
- extensions/qa-lab/src/runtime-tool-fixture.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Inventory the direct message runtime tool family.
|
||||
config:
|
||||
toolName: message
|
||||
expectedAvailable: false
|
||||
toolCoverage:
|
||||
family: message-tool
|
||||
actualTool: message
|
||||
bucket: optional-profile-or-plugin
|
||||
expectedLayer: profile-or-plugin
|
||||
required: false
|
||||
codexDefaultImpact: P4
|
||||
qaImpact: P4
|
||||
action: keep report-only in coding profile
|
||||
reason: Direct message is not part of the coding-profile default surface; session messaging uses sessions_send.
|
||||
promptSnippet: "target=message"
|
||||
failurePromptSnippet: "failure target=message"
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: exercises message happy and failure paths
|
||||
actions:
|
||||
- call: runRuntimeToolFixture
|
||||
saveAs: result
|
||||
args:
|
||||
- ref: env
|
||||
- ref: config
|
||||
detailsExpr: result
|
||||
49
qa/scenarios/runtime/tools/session-status.yaml
Normal file
49
qa/scenarios/runtime/tools/session-status.yaml
Normal file
@@ -0,0 +1,49 @@
|
||||
title: "Runtime tool fixture — session_status"
|
||||
|
||||
scenario:
|
||||
id: runtime-tool-session-status
|
||||
surface: runtime-tools
|
||||
runtimeParityTier: standard
|
||||
coverage:
|
||||
primary:
|
||||
- tools.session-status
|
||||
objective: Verify session_status preserves arguments and result shape across OpenClaw and Codex.
|
||||
successCriteria:
|
||||
- Effective tools expose session_status.
|
||||
- The mock provider plans exactly one happy-path session_status call.
|
||||
- The mock provider plans one denied-input failure-path session_status call.
|
||||
- Runtime parity coverage hard-fails call/result drift in the standard direct-loading gate.
|
||||
docsRefs:
|
||||
- qa/scenarios/index.yaml
|
||||
codeRefs:
|
||||
- src/agents/tools/session-status-tool.ts
|
||||
- extensions/qa-lab/src/runtime-tool-fixture.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Exercise the session_status runtime tool family.
|
||||
config:
|
||||
toolName: session_status
|
||||
toolCoverage:
|
||||
family: session_status
|
||||
actualTool: session_status
|
||||
bucket: openclaw-dynamic-integration
|
||||
expectedLayer: openclaw-dynamic
|
||||
capabilityLayer: openclaw-dynamic-direct
|
||||
required: true
|
||||
codexDefaultImpact: P4
|
||||
qaImpact: P1
|
||||
action: hard gate in the standard direct-loading tier
|
||||
reason: session_status is an OpenClaw integration tool and must stay visible and callable under OpenClaw and Codex direct runtime parity.
|
||||
promptSnippet: "target=session_status"
|
||||
failurePromptSnippet: "failure target=session_status"
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: exercises session_status happy and failure paths
|
||||
actions:
|
||||
- call: runRuntimeToolFixture
|
||||
saveAs: result
|
||||
args:
|
||||
- ref: env
|
||||
- ref: config
|
||||
detailsExpr: result
|
||||
51
qa/scenarios/runtime/tools/sessions-spawn.yaml
Normal file
51
qa/scenarios/runtime/tools/sessions-spawn.yaml
Normal file
@@ -0,0 +1,51 @@
|
||||
title: "Runtime tool fixture — sessions_spawn"
|
||||
|
||||
scenario:
|
||||
id: runtime-tool-sessions-spawn
|
||||
surface: runtime-tools
|
||||
runtimeParityTier: standard
|
||||
coverage:
|
||||
primary:
|
||||
- tools.sessions-spawn
|
||||
objective: Verify sessions_spawn preserves arguments and result shape across OpenClaw and Codex.
|
||||
successCriteria:
|
||||
- Effective tools expose sessions_spawn.
|
||||
- The mock provider plans exactly one happy-path sessions_spawn call.
|
||||
- The mock provider plans one denied-input failure-path sessions_spawn call.
|
||||
- Runtime parity coverage hard-fails call/result drift in the standard direct-loading gate.
|
||||
docsRefs:
|
||||
- qa/scenarios/index.yaml
|
||||
codeRefs:
|
||||
- src/agents/tools/sessions-spawn-tool.ts
|
||||
- extensions/qa-lab/src/runtime-tool-fixture.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Exercise the sessions_spawn runtime tool family.
|
||||
config:
|
||||
toolName: sessions_spawn
|
||||
toolCoverage:
|
||||
family: sessions_spawn
|
||||
actualTool: sessions_spawn
|
||||
bucket: openclaw-dynamic-integration
|
||||
expectedLayer: openclaw-dynamic
|
||||
capabilityLayer: openclaw-dynamic-direct
|
||||
required: true
|
||||
codexDefaultImpact: P4
|
||||
qaImpact: P1
|
||||
action: hard gate in the standard direct-loading tier
|
||||
reason: sessions_spawn is an OpenClaw integration tool and must stay visible and callable under OpenClaw and Codex direct runtime parity.
|
||||
knownHarnessGap:
|
||||
reason: Live failure-path injection for sessions_spawn can be refused safely in prose before the model calls the tool; happy-path visibility and execution still run in this fixture.
|
||||
promptSnippet: "target=sessions_spawn"
|
||||
failurePromptSnippet: "failure target=sessions_spawn"
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: exercises sessions_spawn happy and failure paths
|
||||
actions:
|
||||
- call: runRuntimeToolFixture
|
||||
saveAs: result
|
||||
args:
|
||||
- ref: env
|
||||
- ref: config
|
||||
detailsExpr: result
|
||||
51
qa/scenarios/runtime/tools/skill-invocation.yaml
Normal file
51
qa/scenarios/runtime/tools/skill-invocation.yaml
Normal file
@@ -0,0 +1,51 @@
|
||||
title: "Runtime tool fixture — skill invocation"
|
||||
|
||||
scenario:
|
||||
id: runtime-tool-skill-invocation
|
||||
surface: runtime-tools
|
||||
runtimeParityTier: optional
|
||||
coverage:
|
||||
primary:
|
||||
- tools.skill-invocation
|
||||
objective: Track skill invocation parity once skill tools are represented as first-class runtime tools.
|
||||
successCriteria:
|
||||
- If skill_invoke is present, the fixture exercises happy and failure paths.
|
||||
- If skill_invoke is absent, the fixture records the known-broken tracking marker.
|
||||
docsRefs:
|
||||
- docs/tools/skills.md
|
||||
codeRefs:
|
||||
- src/agents/skills-clawhub.ts
|
||||
- extensions/qa-lab/src/runtime-tool-fixture.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Track first-class skill invocation runtime parity coverage.
|
||||
config:
|
||||
toolName: skill_invoke
|
||||
expectedAvailable: false
|
||||
toolCoverage:
|
||||
family: skill-invocation
|
||||
actualTool: skill_invoke
|
||||
bucket: optional-profile-or-plugin
|
||||
expectedLayer: profile-or-plugin
|
||||
required: false
|
||||
tracking: "#80173"
|
||||
codexDefaultImpact: P4
|
||||
qaImpact: P3
|
||||
action: keep optional until stable skill_invoke tool semantics exist
|
||||
reason: Skills are currently prompt/inventory-driven in QA, not exposed as a stable skill_invoke tool.
|
||||
knownBroken:
|
||||
issue: "#80173"
|
||||
reason: skill_invoke is not exposed by the current default tool surface.
|
||||
promptSnippet: "target=skill_invoke"
|
||||
failurePromptSnippet: "failure target=skill_invoke"
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: exercises or records skill invocation coverage
|
||||
actions:
|
||||
- call: runRuntimeToolFixture
|
||||
saveAs: result
|
||||
args:
|
||||
- ref: env
|
||||
- ref: config
|
||||
detailsExpr: result
|
||||
50
qa/scenarios/runtime/tools/tavily-extract.yaml
Normal file
50
qa/scenarios/runtime/tools/tavily-extract.yaml
Normal file
@@ -0,0 +1,50 @@
|
||||
title: "Runtime tool fixture — tavily_extract"
|
||||
|
||||
scenario:
|
||||
id: runtime-tool-tavily-extract
|
||||
surface: runtime-tools
|
||||
runtimeParityTier: optional
|
||||
coverage:
|
||||
primary:
|
||||
- tools.tavily-extract
|
||||
objective: Track tavily_extract parity once the tool is present in the runtime surface.
|
||||
successCriteria:
|
||||
- If tavily_extract is present, the fixture exercises happy and failure paths.
|
||||
- If tavily_extract is absent, the fixture records the known-broken tracking marker.
|
||||
docsRefs:
|
||||
- qa/scenarios/index.yaml
|
||||
codeRefs:
|
||||
- extensions/qa-lab/src/runtime-tool-fixture.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Track tavily_extract runtime parity coverage.
|
||||
config:
|
||||
toolName: tavily_extract
|
||||
expectedAvailable: false
|
||||
toolCoverage:
|
||||
family: tavily_extract
|
||||
actualTool: tavily_extract
|
||||
bucket: optional-profile-or-plugin
|
||||
expectedLayer: profile-or-plugin
|
||||
required: false
|
||||
tracking: "#80173"
|
||||
codexDefaultImpact: P4
|
||||
qaImpact: P3
|
||||
action: keep optional unless Tavily integration is explicitly enabled
|
||||
reason: Tavily tools are listed in the phase matrix but are not exposed by the current default tool surface.
|
||||
knownBroken:
|
||||
issue: "#80173"
|
||||
reason: tavily_extract is not exposed by the current default tool surface.
|
||||
promptSnippet: "target=tavily_extract"
|
||||
failurePromptSnippet: "failure target=tavily_extract"
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: exercises or records tavily_extract coverage
|
||||
actions:
|
||||
- call: runRuntimeToolFixture
|
||||
saveAs: result
|
||||
args:
|
||||
- ref: env
|
||||
- ref: config
|
||||
detailsExpr: result
|
||||
50
qa/scenarios/runtime/tools/tavily-search.yaml
Normal file
50
qa/scenarios/runtime/tools/tavily-search.yaml
Normal file
@@ -0,0 +1,50 @@
|
||||
title: "Runtime tool fixture — tavily_search"
|
||||
|
||||
scenario:
|
||||
id: runtime-tool-tavily-search
|
||||
surface: runtime-tools
|
||||
runtimeParityTier: optional
|
||||
coverage:
|
||||
primary:
|
||||
- tools.tavily-search
|
||||
objective: Track tavily_search parity once the tool is present in the runtime surface.
|
||||
successCriteria:
|
||||
- If tavily_search is present, the fixture exercises happy and failure paths.
|
||||
- If tavily_search is absent, the fixture records the known-broken tracking marker.
|
||||
docsRefs:
|
||||
- qa/scenarios/index.yaml
|
||||
codeRefs:
|
||||
- extensions/qa-lab/src/runtime-tool-fixture.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Track tavily_search runtime parity coverage.
|
||||
config:
|
||||
toolName: tavily_search
|
||||
expectedAvailable: false
|
||||
toolCoverage:
|
||||
family: tavily_search
|
||||
actualTool: tavily_search
|
||||
bucket: optional-profile-or-plugin
|
||||
expectedLayer: profile-or-plugin
|
||||
required: false
|
||||
tracking: "#80173"
|
||||
codexDefaultImpact: P4
|
||||
qaImpact: P3
|
||||
action: keep optional unless Tavily integration is explicitly enabled
|
||||
reason: Tavily tools are listed in the phase matrix but are not exposed by the current default tool surface.
|
||||
knownBroken:
|
||||
issue: "#80173"
|
||||
reason: tavily_search is not exposed by the current default tool surface.
|
||||
promptSnippet: "target=tavily_search"
|
||||
failurePromptSnippet: "failure target=tavily_search"
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: exercises or records tavily_search coverage
|
||||
actions:
|
||||
- call: runRuntimeToolFixture
|
||||
saveAs: result
|
||||
args:
|
||||
- ref: env
|
||||
- ref: config
|
||||
detailsExpr: result
|
||||
51
qa/scenarios/runtime/tools/tts.yaml
Normal file
51
qa/scenarios/runtime/tools/tts.yaml
Normal file
@@ -0,0 +1,51 @@
|
||||
title: "Runtime tool fixture — tts"
|
||||
|
||||
scenario:
|
||||
id: runtime-tool-tts
|
||||
surface: runtime-tools
|
||||
runtimeParityTier: optional
|
||||
coverage:
|
||||
primary:
|
||||
- tools.tts
|
||||
objective: Verify tts preserves arguments and result shape when the tool is present.
|
||||
successCriteria:
|
||||
- If tts is present, the fixture exercises happy and failure paths.
|
||||
- If tts is absent, the fixture records the known-broken tracking marker.
|
||||
docsRefs:
|
||||
- qa/scenarios/index.yaml
|
||||
codeRefs:
|
||||
- src/agents/tools/tts-tool.ts
|
||||
- extensions/qa-lab/src/runtime-tool-fixture.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Exercise or track the tts runtime tool family.
|
||||
config:
|
||||
toolName: tts
|
||||
expectedAvailable: false
|
||||
toolCoverage:
|
||||
family: tts
|
||||
actualTool: tts
|
||||
bucket: optional-profile-or-plugin
|
||||
expectedLayer: profile-or-plugin
|
||||
required: false
|
||||
tracking: "#80173"
|
||||
codexDefaultImpact: P4
|
||||
qaImpact: P3
|
||||
action: keep optional unless TTS is explicitly configured
|
||||
reason: TTS is configuration-dependent and is not exposed by every mock QA run.
|
||||
knownBroken:
|
||||
issue: "#80173"
|
||||
reason: tts is configuration-dependent and absent from the default mock QA run.
|
||||
promptSnippet: "target=tts"
|
||||
failurePromptSnippet: "failure target=tts"
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: exercises or records tts coverage
|
||||
actions:
|
||||
- call: runRuntimeToolFixture
|
||||
saveAs: result
|
||||
args:
|
||||
- ref: env
|
||||
- ref: config
|
||||
detailsExpr: result
|
||||
49
qa/scenarios/runtime/tools/web-fetch.yaml
Normal file
49
qa/scenarios/runtime/tools/web-fetch.yaml
Normal file
@@ -0,0 +1,49 @@
|
||||
title: "Runtime tool fixture — web_fetch"
|
||||
|
||||
scenario:
|
||||
id: runtime-tool-web-fetch
|
||||
surface: runtime-tools
|
||||
runtimeParityTier: standard
|
||||
coverage:
|
||||
primary:
|
||||
- tools.web-fetch
|
||||
objective: Verify web_fetch preserves arguments and result shape across OpenClaw and Codex.
|
||||
successCriteria:
|
||||
- Effective tools expose web_fetch.
|
||||
- The mock provider plans exactly one happy-path web_fetch call.
|
||||
- The mock provider plans one denied-input failure-path web_fetch call.
|
||||
- Runtime parity coverage hard-fails call/result drift in the standard direct-loading gate.
|
||||
docsRefs:
|
||||
- qa/scenarios/index.yaml
|
||||
codeRefs:
|
||||
- src/agents/tools/web-fetch.ts
|
||||
- extensions/qa-lab/src/runtime-tool-fixture.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Exercise the web_fetch runtime tool family.
|
||||
config:
|
||||
toolName: web_fetch
|
||||
toolCoverage:
|
||||
family: web_fetch
|
||||
actualTool: web_fetch
|
||||
bucket: openclaw-dynamic-integration
|
||||
expectedLayer: openclaw-dynamic
|
||||
capabilityLayer: openclaw-dynamic-direct
|
||||
required: true
|
||||
codexDefaultImpact: P4
|
||||
qaImpact: P1
|
||||
action: hard gate in the standard direct-loading tier
|
||||
reason: web_fetch is an OpenClaw integration tool and must stay visible and callable under OpenClaw and Codex direct runtime parity.
|
||||
promptSnippet: "target=web_fetch"
|
||||
failurePromptSnippet: "failure target=web_fetch"
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: exercises web_fetch happy and failure paths
|
||||
actions:
|
||||
- call: runRuntimeToolFixture
|
||||
saveAs: result
|
||||
args:
|
||||
- ref: env
|
||||
- ref: config
|
||||
detailsExpr: result
|
||||
59
qa/scenarios/runtime/tools/web-search.yaml
Normal file
59
qa/scenarios/runtime/tools/web-search.yaml
Normal file
@@ -0,0 +1,59 @@
|
||||
title: "Runtime tool fixture — web_search"
|
||||
|
||||
scenario:
|
||||
id: runtime-tool-web-search
|
||||
surface: runtime-tools
|
||||
runtimeParityTier: standard
|
||||
coverage:
|
||||
primary:
|
||||
- tools.web-search
|
||||
objective: Verify web_search preserves arguments and result shape across OpenClaw and Codex.
|
||||
successCriteria:
|
||||
- Effective tools expose web_search.
|
||||
- The mock provider plans exactly one happy-path web_search call.
|
||||
- The mock provider plans one denied-input failure-path web_search call.
|
||||
- Runtime parity coverage hard-fails call/result drift in the standard direct-loading gate.
|
||||
plugins:
|
||||
- qa-lab
|
||||
gatewayConfigPatch:
|
||||
tools:
|
||||
web:
|
||||
search:
|
||||
enabled: true
|
||||
provider: qa-lab-search
|
||||
docsRefs:
|
||||
- qa/scenarios/index.yaml
|
||||
codeRefs:
|
||||
- src/agents/tools/web-search.ts
|
||||
- extensions/qa-lab/src/qa-web-search-provider.ts
|
||||
- extensions/qa-lab/src/runtime-tool-fixture.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Exercise the web_search runtime tool family.
|
||||
config:
|
||||
toolName: web_search
|
||||
toolCoverage:
|
||||
family: web_search
|
||||
actualTool: web_search
|
||||
bucket: openclaw-dynamic-integration
|
||||
expectedLayer: openclaw-dynamic
|
||||
capabilityLayer: openclaw-dynamic-direct
|
||||
required: true
|
||||
codexDefaultImpact: P4
|
||||
qaImpact: P1
|
||||
action: hard gate in the standard direct-loading tier
|
||||
reason: web_search is an OpenClaw integration tool and must stay visible and callable under OpenClaw and Codex direct runtime parity.
|
||||
promptSnippet: "target=web_search"
|
||||
failurePrompt: "tool search qa failure target=web_search. Call web_search exactly once with query OPENCLAW_QA_WEB_SEARCH_DENIED_INPUT and then summarize the failure."
|
||||
failurePromptSnippet: "failure target=web_search"
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: exercises web_search happy and failure paths
|
||||
actions:
|
||||
- call: runRuntimeToolFixture
|
||||
saveAs: result
|
||||
args:
|
||||
- ref: env
|
||||
- ref: config
|
||||
detailsExpr: result
|
||||
Reference in New Issue
Block a user