Vendor OpenClaw source as Adolf fork baseline
Some checks failed
ClawSweeper Dispatch / dispatch (push) Has been cancelled
CodeQL / Security High (actions) (push) Has been cancelled
CodeQL / Security High (channel-runtime-boundary) (push) Has been cancelled
CodeQL / Security High (core-auth-secrets) (push) Has been cancelled
CodeQL / Security High (mcp-process-tool-boundary) (push) Has been cancelled
CodeQL / Security High (network-ssrf-boundary) (push) Has been cancelled
CodeQL / Security High (plugin-trust-boundary) (push) Has been cancelled
CodeQL / Security High (process-exec-boundary) (push) Has been cancelled
Docs Sync Publish Repo / sync-publish-repo (push) Has been cancelled
Docs / docs (push) Has been cancelled
OpenClaw Stable Main Closeout / Resolve stable release closeout inputs (push) Has been cancelled
OpenClaw Stable Main Closeout / Verify stable main closeout (push) Has been cancelled
Workflow Sanity / no-tabs (push) Has been cancelled
Workflow Sanity / actionlint (push) Has been cancelled
Workflow Sanity / generated-doc-baselines (push) Has been cancelled
CI / runner-admission (push) Has been cancelled
CI / preflight (push) Has been cancelled
CI / security-fast (push) Has been cancelled
CI / pnpm-store-warmup (push) Has been cancelled
CI / build-artifacts (push) Has been cancelled
CI / native-i18n (push) Has been cancelled
CI / ${{ matrix.check_name }} (push) Has been cancelled
CI / ${{ matrix.checkName }} (push) Has been cancelled
CI / checks-node-compat-node22 (push) Has been cancelled
CI / check-bundled-channel-config-metadata (push) Has been cancelled
CI / check-dependencies (push) Has been cancelled
CI / check-guards (push) Has been cancelled
CI / check-lint (push) Has been cancelled
CI / check-prod-types (push) Has been cancelled
CI / check-shrinkwrap (push) Has been cancelled
CI / check-test-types (push) Has been cancelled
CI / check-additional-boundaries-a (push) Has been cancelled
CI / check-additional-boundaries-bcd (push) Has been cancelled
CI / check-additional-extension-bundled (push) Has been cancelled
CI / check-additional-extension-channels (push) Has been cancelled
CI / check-additional-extension-package-boundary (push) Has been cancelled
CI / check-additional-runtime-topology-architecture (push) Has been cancelled
CI / check-session-accessor-boundary (push) Has been cancelled
CI / check-session-transcript-reader-boundary (push) Has been cancelled
CI / check-docs (push) Has been cancelled
CI / skills-python (push) Has been cancelled
CI / macos-swift (push) Has been cancelled
CI / ios-build (push) Has been cancelled
CI / ci-timings-summary (push) Has been cancelled
Native App Locale Refresh / Refresh native fa (push) Has been cancelled
Native App Locale Refresh / Refresh native fr (push) Has been cancelled
Native App Locale Refresh / Refresh native hi (push) Has been cancelled
Native App Locale Refresh / Refresh native id (push) Has been cancelled
Native App Locale Refresh / Refresh native it (push) Has been cancelled
Native App Locale Refresh / Refresh native ja-JP (push) Has been cancelled
Control UI Locale Refresh / plan (push) Has been cancelled
Control UI Locale Refresh / Refresh ${{ matrix.locale }} (push) Has been cancelled
Control UI Locale Refresh / Commit control UI locale refresh (push) Has been cancelled
Live Media Runner Image / Build live media runner image (push) Has been cancelled
Native App Locale Refresh / Refresh native ar (push) Has been cancelled
Native App Locale Refresh / Refresh native de (push) Has been cancelled
Native App Locale Refresh / Refresh native es (push) Has been cancelled
Native App Locale Refresh / Refresh native ko (push) Has been cancelled
Native App Locale Refresh / Refresh native nl (push) Has been cancelled
Native App Locale Refresh / Refresh native pl (push) Has been cancelled
Native App Locale Refresh / Refresh native pt-BR (push) Has been cancelled
Native App Locale Refresh / Refresh native ru (push) Has been cancelled
Native App Locale Refresh / Refresh native sv (push) Has been cancelled
Native App Locale Refresh / Refresh native th (push) Has been cancelled
Native App Locale Refresh / Refresh native tr (push) Has been cancelled
Native App Locale Refresh / Refresh native uk (push) Has been cancelled
Native App Locale Refresh / Refresh native vi (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-CN (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-TW (push) Has been cancelled
Native App Locale Refresh / Commit native locale refresh (push) Has been cancelled
Plugin Init Scaffold Validation / Validate provider scaffold (push) Has been cancelled
Plugin NPM Release / preview_plugins_npm (push) Has been cancelled
Plugin NPM Release / Validate release publish approval (push) Has been cancelled
Plugin NPM Release / preview_plugin_pack (push) Has been cancelled
Plugin NPM Release / publish_plugins_npm (push) Has been cancelled
Sandbox Common Smoke / sandbox-common-smoke (push) Has been cancelled
Website Installer Sync / static (push) Has been cancelled
Website Installer Sync / linux-docker (push) Has been cancelled
Website Installer Sync / macos-installer (push) Has been cancelled
Website Installer Sync / windows-installer (push) Has been cancelled
Website Installer Sync / sync-website (push) Has been cancelled
Some checks failed
ClawSweeper Dispatch / dispatch (push) Has been cancelled
CodeQL / Security High (actions) (push) Has been cancelled
CodeQL / Security High (channel-runtime-boundary) (push) Has been cancelled
CodeQL / Security High (core-auth-secrets) (push) Has been cancelled
CodeQL / Security High (mcp-process-tool-boundary) (push) Has been cancelled
CodeQL / Security High (network-ssrf-boundary) (push) Has been cancelled
CodeQL / Security High (plugin-trust-boundary) (push) Has been cancelled
CodeQL / Security High (process-exec-boundary) (push) Has been cancelled
Docs Sync Publish Repo / sync-publish-repo (push) Has been cancelled
Docs / docs (push) Has been cancelled
OpenClaw Stable Main Closeout / Resolve stable release closeout inputs (push) Has been cancelled
OpenClaw Stable Main Closeout / Verify stable main closeout (push) Has been cancelled
Workflow Sanity / no-tabs (push) Has been cancelled
Workflow Sanity / actionlint (push) Has been cancelled
Workflow Sanity / generated-doc-baselines (push) Has been cancelled
CI / runner-admission (push) Has been cancelled
CI / preflight (push) Has been cancelled
CI / security-fast (push) Has been cancelled
CI / pnpm-store-warmup (push) Has been cancelled
CI / build-artifacts (push) Has been cancelled
CI / native-i18n (push) Has been cancelled
CI / ${{ matrix.check_name }} (push) Has been cancelled
CI / ${{ matrix.checkName }} (push) Has been cancelled
CI / checks-node-compat-node22 (push) Has been cancelled
CI / check-bundled-channel-config-metadata (push) Has been cancelled
CI / check-dependencies (push) Has been cancelled
CI / check-guards (push) Has been cancelled
CI / check-lint (push) Has been cancelled
CI / check-prod-types (push) Has been cancelled
CI / check-shrinkwrap (push) Has been cancelled
CI / check-test-types (push) Has been cancelled
CI / check-additional-boundaries-a (push) Has been cancelled
CI / check-additional-boundaries-bcd (push) Has been cancelled
CI / check-additional-extension-bundled (push) Has been cancelled
CI / check-additional-extension-channels (push) Has been cancelled
CI / check-additional-extension-package-boundary (push) Has been cancelled
CI / check-additional-runtime-topology-architecture (push) Has been cancelled
CI / check-session-accessor-boundary (push) Has been cancelled
CI / check-session-transcript-reader-boundary (push) Has been cancelled
CI / check-docs (push) Has been cancelled
CI / skills-python (push) Has been cancelled
CI / macos-swift (push) Has been cancelled
CI / ios-build (push) Has been cancelled
CI / ci-timings-summary (push) Has been cancelled
Native App Locale Refresh / Refresh native fa (push) Has been cancelled
Native App Locale Refresh / Refresh native fr (push) Has been cancelled
Native App Locale Refresh / Refresh native hi (push) Has been cancelled
Native App Locale Refresh / Refresh native id (push) Has been cancelled
Native App Locale Refresh / Refresh native it (push) Has been cancelled
Native App Locale Refresh / Refresh native ja-JP (push) Has been cancelled
Control UI Locale Refresh / plan (push) Has been cancelled
Control UI Locale Refresh / Refresh ${{ matrix.locale }} (push) Has been cancelled
Control UI Locale Refresh / Commit control UI locale refresh (push) Has been cancelled
Live Media Runner Image / Build live media runner image (push) Has been cancelled
Native App Locale Refresh / Refresh native ar (push) Has been cancelled
Native App Locale Refresh / Refresh native de (push) Has been cancelled
Native App Locale Refresh / Refresh native es (push) Has been cancelled
Native App Locale Refresh / Refresh native ko (push) Has been cancelled
Native App Locale Refresh / Refresh native nl (push) Has been cancelled
Native App Locale Refresh / Refresh native pl (push) Has been cancelled
Native App Locale Refresh / Refresh native pt-BR (push) Has been cancelled
Native App Locale Refresh / Refresh native ru (push) Has been cancelled
Native App Locale Refresh / Refresh native sv (push) Has been cancelled
Native App Locale Refresh / Refresh native th (push) Has been cancelled
Native App Locale Refresh / Refresh native tr (push) Has been cancelled
Native App Locale Refresh / Refresh native uk (push) Has been cancelled
Native App Locale Refresh / Refresh native vi (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-CN (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-TW (push) Has been cancelled
Native App Locale Refresh / Commit native locale refresh (push) Has been cancelled
Plugin Init Scaffold Validation / Validate provider scaffold (push) Has been cancelled
Plugin NPM Release / preview_plugins_npm (push) Has been cancelled
Plugin NPM Release / Validate release publish approval (push) Has been cancelled
Plugin NPM Release / preview_plugin_pack (push) Has been cancelled
Plugin NPM Release / publish_plugins_npm (push) Has been cancelled
Sandbox Common Smoke / sandbox-common-smoke (push) Has been cancelled
Website Installer Sync / static (push) Has been cancelled
Website Installer Sync / linux-docker (push) Has been cancelled
Website Installer Sync / macos-installer (push) Has been cancelled
Website Installer Sync / windows-installer (push) Has been cancelled
Website Installer Sync / sync-website (push) Has been cancelled
Adolf is a fork/vendored clone of github.com/openclaw/openclaw (v2026.6.11), free to diverge. Tree copied sans upstream .git; upstream remote added for future syncs. Node pinned to 24 (.nvmrc); engines already require >=22.19. Preserves docs/ARCHITECTURE.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LeqyaxJF2nbRXJtae2kNB2
This commit is contained in:
259
qa/scenarios/models/claude-cli-provider-capabilities.yaml
Normal file
259
qa/scenarios/models/claude-cli-provider-capabilities.yaml
Normal file
@@ -0,0 +1,259 @@
|
||||
title: Claude CLI provider capabilities API key
|
||||
|
||||
scenario:
|
||||
id: claude-cli-provider-capabilities
|
||||
surface: model-provider
|
||||
coverage:
|
||||
primary:
|
||||
- models.provider-capabilities
|
||||
secondary:
|
||||
- models.claude-cli
|
||||
objective: Verify the Claude CLI model-provider lane can use the Anthropic API key path to talk, read an attached image, use bundled MCP tools, and apply workspace skills.
|
||||
successCriteria:
|
||||
- A live-frontier run fails fast unless the selected primary provider is claude-cli.
|
||||
- The Claude CLI backend preserves ANTHROPIC_API_KEY for this run instead of using native subscription auth.
|
||||
- The agent replies through the Claude CLI provider in a direct chat turn.
|
||||
- The agent describes an attached image through the Claude CLI image path.
|
||||
- The agent can reach memory via the bundled MCP/tool bridge.
|
||||
- The agent sees and follows a workspace skill.
|
||||
docsRefs:
|
||||
- docs/gateway/cli-backends.md
|
||||
- docs/tools/skills.md
|
||||
- docs/cli/mcp.md
|
||||
- docs/tools/index.md
|
||||
codeRefs:
|
||||
- extensions/anthropic/cli-backend.ts
|
||||
- src/agents/cli-backends.ts
|
||||
- src/mcp/plugin-tools-serve.ts
|
||||
- extensions/qa-lab/src/suite.ts
|
||||
execution:
|
||||
kind: flow
|
||||
summary: Run with `pnpm openclaw qa suite --provider-mode live-frontier --cli-auth-mode api-key --model claude-cli/claude-sonnet-4-6 --alt-model claude-cli/claude-sonnet-4-6 --scenario claude-cli-provider-capabilities`.
|
||||
config:
|
||||
authMode: api-key
|
||||
requiredProviderMode: live-frontier
|
||||
requiredProvider: claude-cli
|
||||
chatPrompt: "Claude CLI provider marker check. Reply exactly: CLAUDE-CLI-CHAT-OK"
|
||||
chatExpected: CLAUDE-CLI-CHAT-OK
|
||||
imagePrompt: "Image understanding check: describe the top and bottom colors in the attached image in one short sentence."
|
||||
imageColorGroups:
|
||||
- [red, scarlet, crimson]
|
||||
- [blue, azure, teal, cyan, aqua]
|
||||
memoryFact: "Hidden Claude CLI MCP fact: the provider bridge codename is ORBIT-9."
|
||||
memoryQuery: "provider bridge codename ORBIT-9"
|
||||
memoryExpected: ORBIT-9
|
||||
memoryPrompt: "Memory tools check: use the available memory search MCP/tool bridge to find the hidden provider bridge codename stored only in memory. Reply with the codename."
|
||||
memoryPromptSnippet: "Memory tools check"
|
||||
skillName: qa-claude-cli-skill
|
||||
skillExpected: VISIBLE-SKILL-OK
|
||||
skillBody: |-
|
||||
---
|
||||
name: qa-claude-cli-skill
|
||||
description: Claude CLI QA skill marker
|
||||
---
|
||||
When the user asks for the Claude CLI skill marker exactly, or explicitly asks you to use qa-claude-cli-skill, reply with exactly: VISIBLE-SKILL-OK
|
||||
skillPrompt: "Use qa-claude-cli-skill now. Reply exactly with the visible skill marker and nothing else."
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: confirms the selected live provider and Claude CLI auth mode
|
||||
actions:
|
||||
- set: selected
|
||||
value:
|
||||
expr: splitModelRef(env.primaryModel)
|
||||
- set: preserveEnv
|
||||
value:
|
||||
expr: "String(env.gateway.runtimeEnv.OPENCLAW_LIVE_CLI_BACKEND_PRESERVE_ENV ?? '')"
|
||||
- assert:
|
||||
expr: "env.providerMode !== 'live-frontier' || selected?.provider === config.requiredProvider"
|
||||
message:
|
||||
expr: "`expected live primary provider ${config.requiredProvider}, got ${env.primaryModel}`"
|
||||
- assert:
|
||||
expr: "env.providerMode !== 'live-frontier' || env.gateway.runtimeEnv.OPENCLAW_LIVE_CLI_BACKEND_AUTH_MODE === config.authMode"
|
||||
message:
|
||||
expr: "`expected Claude CLI auth mode ${config.authMode}, got ${env.gateway.runtimeEnv.OPENCLAW_LIVE_CLI_BACKEND_AUTH_MODE ?? 'unset'}`"
|
||||
- assert:
|
||||
expr: "env.providerMode !== 'live-frontier' || preserveEnv.includes('ANTHROPIC_API_KEY')"
|
||||
message:
|
||||
expr: "`expected ANTHROPIC_API_KEY to be preserved for Claude CLI API-key QA mode, got ${preserveEnv}`"
|
||||
detailsExpr: "env.providerMode === 'live-frontier' ? `provider=${selected?.provider} model=${selected?.model} auth=${env.gateway.runtimeEnv.OPENCLAW_LIVE_CLI_BACKEND_AUTH_MODE} preserve=${preserveEnv}` : `mock-compatible provider=${selected?.provider}`"
|
||||
- name: talks through the selected provider
|
||||
actions:
|
||||
- call: reset
|
||||
- set: selected
|
||||
value:
|
||||
expr: splitModelRef(env.primaryModel)
|
||||
- call: runAgentPrompt
|
||||
args:
|
||||
- ref: env
|
||||
- sessionKey:
|
||||
expr: "`agent:qa:claude-cli-${config.authMode}-chat`"
|
||||
message:
|
||||
expr: config.chatPrompt
|
||||
provider:
|
||||
expr: selected?.provider
|
||||
model:
|
||||
expr: selected?.model
|
||||
timeoutMs:
|
||||
expr: resolveQaLiveTurnTimeoutMs(env, 45000, env.primaryModel)
|
||||
- call: waitForOutboundMessage
|
||||
saveAs: chatOutbound
|
||||
args:
|
||||
- ref: state
|
||||
- lambda:
|
||||
params: [candidate]
|
||||
expr: "candidate.conversation.id === 'qa-operator'"
|
||||
- expr: resolveQaLiveTurnTimeoutMs(env, 20000, env.primaryModel)
|
||||
- assert:
|
||||
expr: "chatOutbound.text.includes(config.chatExpected)"
|
||||
message:
|
||||
expr: "`chat marker missing: ${chatOutbound.text}`"
|
||||
detailsExpr: chatOutbound.text
|
||||
- name: describes an attached image through the selected provider
|
||||
actions:
|
||||
- call: reset
|
||||
- set: selected
|
||||
value:
|
||||
expr: splitModelRef(env.primaryModel)
|
||||
- call: runAgentPrompt
|
||||
args:
|
||||
- ref: env
|
||||
- sessionKey:
|
||||
expr: "`agent:qa:claude-cli-${config.authMode}-image`"
|
||||
message:
|
||||
expr: config.imagePrompt
|
||||
provider:
|
||||
expr: selected?.provider
|
||||
model:
|
||||
expr: selected?.model
|
||||
attachments:
|
||||
- mimeType: image/png
|
||||
fileName: claude-cli-red-top-blue-bottom.png
|
||||
content:
|
||||
expr: imageUnderstandingValidPngBase64
|
||||
timeoutMs:
|
||||
expr: resolveQaLiveTurnTimeoutMs(env, 60000, env.primaryModel)
|
||||
- call: waitForOutboundMessage
|
||||
saveAs: imageOutbound
|
||||
args:
|
||||
- ref: state
|
||||
- lambda:
|
||||
params: [candidate]
|
||||
expr: "candidate.conversation.id === 'qa-operator'"
|
||||
- expr: resolveQaLiveTurnTimeoutMs(env, 30000, env.primaryModel)
|
||||
- assert:
|
||||
expr: "config.imageColorGroups.every((group) => group.some((color) => normalizeLowercaseStringOrEmpty(imageOutbound.text).includes(color)))"
|
||||
message:
|
||||
expr: "`missing expected image colors: ${imageOutbound.text}`"
|
||||
- assert:
|
||||
expr: "!env.mock || (((await fetchJson(`${env.mock.baseUrl}/debug/requests`)).find((request) => String(request.prompt ?? '').includes('Image understanding check'))?.imageInputCount ?? 0) >= 1)"
|
||||
message: expected image input to reach mock provider
|
||||
detailsExpr: imageOutbound.text
|
||||
- name: reaches memory through the MCP/tool bridge
|
||||
actions:
|
||||
- call: reset
|
||||
- call: fs.writeFile
|
||||
args:
|
||||
- expr: "path.join(env.gateway.workspaceDir, 'MEMORY.md')"
|
||||
- expr: "`${config.memoryFact}\\n`"
|
||||
- utf8
|
||||
- call: forceMemoryIndex
|
||||
args:
|
||||
- env:
|
||||
ref: env
|
||||
query:
|
||||
expr: config.memoryQuery
|
||||
expectedNeedle:
|
||||
expr: config.memoryExpected
|
||||
- call: createSession
|
||||
saveAs: mcpSessionKey
|
||||
args:
|
||||
- ref: env
|
||||
- expr: "`Claude CLI MCP bridge ${config.authMode}`"
|
||||
- call: readEffectiveTools
|
||||
saveAs: mcpTools
|
||||
args:
|
||||
- ref: env
|
||||
- ref: mcpSessionKey
|
||||
- assert:
|
||||
expr: "mcpTools.has('memory_search')"
|
||||
message: memory_search missing from effective tools before MCP bridge check
|
||||
- set: selected
|
||||
value:
|
||||
expr: splitModelRef(env.primaryModel)
|
||||
- call: runAgentPrompt
|
||||
args:
|
||||
- ref: env
|
||||
- sessionKey:
|
||||
ref: mcpSessionKey
|
||||
message:
|
||||
expr: config.memoryPrompt
|
||||
provider:
|
||||
expr: selected?.provider
|
||||
model:
|
||||
expr: selected?.model
|
||||
timeoutMs:
|
||||
expr: resolveQaLiveTurnTimeoutMs(env, 90000, env.primaryModel)
|
||||
- call: waitForOutboundMessage
|
||||
saveAs: mcpOutbound
|
||||
args:
|
||||
- ref: state
|
||||
- lambda:
|
||||
params: [candidate]
|
||||
expr: "candidate.conversation.id === 'qa-operator'"
|
||||
- expr: resolveQaLiveTurnTimeoutMs(env, 45000, env.primaryModel)
|
||||
- assert:
|
||||
expr: "mcpOutbound.text.includes(config.memoryExpected)"
|
||||
message:
|
||||
expr: "`MCP memory result missing ${config.memoryExpected}: ${mcpOutbound.text}`"
|
||||
- assert:
|
||||
expr: "!env.mock || (await fetchJson(`${env.mock.baseUrl}/debug/requests`)).filter((request) => String(request.allInputText ?? '').includes(config.memoryPromptSnippet)).some((request) => request.plannedToolName === 'memory_search')"
|
||||
message: expected mock model to plan memory_search for MCP bridge prompt
|
||||
detailsExpr: mcpOutbound.text
|
||||
- name: applies a workspace skill through the selected provider
|
||||
actions:
|
||||
- call: reset
|
||||
- call: writeWorkspaceSkill
|
||||
args:
|
||||
- env:
|
||||
ref: env
|
||||
name:
|
||||
expr: config.skillName
|
||||
body:
|
||||
expr: config.skillBody
|
||||
- call: waitForCondition
|
||||
args:
|
||||
- lambda:
|
||||
async: true
|
||||
expr: "((await readSkillStatus(env)).find((skill) => skill.name === config.skillName)?.eligible ? true : undefined)"
|
||||
- 15000
|
||||
- 200
|
||||
- set: selected
|
||||
value:
|
||||
expr: splitModelRef(env.primaryModel)
|
||||
- call: runAgentPrompt
|
||||
args:
|
||||
- ref: env
|
||||
- sessionKey:
|
||||
expr: "`agent:qa:claude-cli-${config.authMode}-skill`"
|
||||
message:
|
||||
expr: config.skillPrompt
|
||||
provider:
|
||||
expr: selected?.provider
|
||||
model:
|
||||
expr: selected?.model
|
||||
timeoutMs:
|
||||
expr: resolveQaLiveTurnTimeoutMs(env, 60000, env.primaryModel)
|
||||
- call: waitForOutboundMessage
|
||||
saveAs: skillOutbound
|
||||
args:
|
||||
- ref: state
|
||||
- lambda:
|
||||
params: [candidate]
|
||||
expr: "candidate.conversation.id === 'qa-operator'"
|
||||
- expr: resolveQaLiveTurnTimeoutMs(env, 30000, env.primaryModel)
|
||||
- assert:
|
||||
expr: "skillOutbound.text.includes(config.skillExpected)"
|
||||
message:
|
||||
expr: "`skill marker missing: ${skillOutbound.text}`"
|
||||
detailsExpr: skillOutbound.text
|
||||
Reference in New Issue
Block a user