Vendor OpenClaw source as Adolf fork baseline
Some checks failed
ClawSweeper Dispatch / dispatch (push) Has been cancelled
CodeQL / Security High (actions) (push) Has been cancelled
CodeQL / Security High (channel-runtime-boundary) (push) Has been cancelled
CodeQL / Security High (core-auth-secrets) (push) Has been cancelled
CodeQL / Security High (mcp-process-tool-boundary) (push) Has been cancelled
CodeQL / Security High (network-ssrf-boundary) (push) Has been cancelled
CodeQL / Security High (plugin-trust-boundary) (push) Has been cancelled
CodeQL / Security High (process-exec-boundary) (push) Has been cancelled
Docs Sync Publish Repo / sync-publish-repo (push) Has been cancelled
Docs / docs (push) Has been cancelled
OpenClaw Stable Main Closeout / Resolve stable release closeout inputs (push) Has been cancelled
OpenClaw Stable Main Closeout / Verify stable main closeout (push) Has been cancelled
Workflow Sanity / no-tabs (push) Has been cancelled
Workflow Sanity / actionlint (push) Has been cancelled
Workflow Sanity / generated-doc-baselines (push) Has been cancelled
CI / runner-admission (push) Has been cancelled
CI / preflight (push) Has been cancelled
CI / security-fast (push) Has been cancelled
CI / pnpm-store-warmup (push) Has been cancelled
CI / build-artifacts (push) Has been cancelled
CI / native-i18n (push) Has been cancelled
CI / ${{ matrix.check_name }} (push) Has been cancelled
CI / ${{ matrix.checkName }} (push) Has been cancelled
CI / checks-node-compat-node22 (push) Has been cancelled
CI / check-bundled-channel-config-metadata (push) Has been cancelled
CI / check-dependencies (push) Has been cancelled
CI / check-guards (push) Has been cancelled
CI / check-lint (push) Has been cancelled
CI / check-prod-types (push) Has been cancelled
CI / check-shrinkwrap (push) Has been cancelled
CI / check-test-types (push) Has been cancelled
CI / check-additional-boundaries-a (push) Has been cancelled
CI / check-additional-boundaries-bcd (push) Has been cancelled
CI / check-additional-extension-bundled (push) Has been cancelled
CI / check-additional-extension-channels (push) Has been cancelled
CI / check-additional-extension-package-boundary (push) Has been cancelled
CI / check-additional-runtime-topology-architecture (push) Has been cancelled
CI / check-session-accessor-boundary (push) Has been cancelled
CI / check-session-transcript-reader-boundary (push) Has been cancelled
CI / check-docs (push) Has been cancelled
CI / skills-python (push) Has been cancelled
CI / macos-swift (push) Has been cancelled
CI / ios-build (push) Has been cancelled
CI / ci-timings-summary (push) Has been cancelled
Native App Locale Refresh / Refresh native fa (push) Has been cancelled
Native App Locale Refresh / Refresh native fr (push) Has been cancelled
Native App Locale Refresh / Refresh native hi (push) Has been cancelled
Native App Locale Refresh / Refresh native id (push) Has been cancelled
Native App Locale Refresh / Refresh native it (push) Has been cancelled
Native App Locale Refresh / Refresh native ja-JP (push) Has been cancelled
Control UI Locale Refresh / plan (push) Has been cancelled
Control UI Locale Refresh / Refresh ${{ matrix.locale }} (push) Has been cancelled
Control UI Locale Refresh / Commit control UI locale refresh (push) Has been cancelled
Live Media Runner Image / Build live media runner image (push) Has been cancelled
Native App Locale Refresh / Refresh native ar (push) Has been cancelled
Native App Locale Refresh / Refresh native de (push) Has been cancelled
Native App Locale Refresh / Refresh native es (push) Has been cancelled
Native App Locale Refresh / Refresh native ko (push) Has been cancelled
Native App Locale Refresh / Refresh native nl (push) Has been cancelled
Native App Locale Refresh / Refresh native pl (push) Has been cancelled
Native App Locale Refresh / Refresh native pt-BR (push) Has been cancelled
Native App Locale Refresh / Refresh native ru (push) Has been cancelled
Native App Locale Refresh / Refresh native sv (push) Has been cancelled
Native App Locale Refresh / Refresh native th (push) Has been cancelled
Native App Locale Refresh / Refresh native tr (push) Has been cancelled
Native App Locale Refresh / Refresh native uk (push) Has been cancelled
Native App Locale Refresh / Refresh native vi (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-CN (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-TW (push) Has been cancelled
Native App Locale Refresh / Commit native locale refresh (push) Has been cancelled
Plugin Init Scaffold Validation / Validate provider scaffold (push) Has been cancelled
Plugin NPM Release / preview_plugins_npm (push) Has been cancelled
Plugin NPM Release / Validate release publish approval (push) Has been cancelled
Plugin NPM Release / preview_plugin_pack (push) Has been cancelled
Plugin NPM Release / publish_plugins_npm (push) Has been cancelled
Sandbox Common Smoke / sandbox-common-smoke (push) Has been cancelled
Website Installer Sync / static (push) Has been cancelled
Website Installer Sync / linux-docker (push) Has been cancelled
Website Installer Sync / macos-installer (push) Has been cancelled
Website Installer Sync / windows-installer (push) Has been cancelled
Website Installer Sync / sync-website (push) Has been cancelled
Some checks failed
ClawSweeper Dispatch / dispatch (push) Has been cancelled
CodeQL / Security High (actions) (push) Has been cancelled
CodeQL / Security High (channel-runtime-boundary) (push) Has been cancelled
CodeQL / Security High (core-auth-secrets) (push) Has been cancelled
CodeQL / Security High (mcp-process-tool-boundary) (push) Has been cancelled
CodeQL / Security High (network-ssrf-boundary) (push) Has been cancelled
CodeQL / Security High (plugin-trust-boundary) (push) Has been cancelled
CodeQL / Security High (process-exec-boundary) (push) Has been cancelled
Docs Sync Publish Repo / sync-publish-repo (push) Has been cancelled
Docs / docs (push) Has been cancelled
OpenClaw Stable Main Closeout / Resolve stable release closeout inputs (push) Has been cancelled
OpenClaw Stable Main Closeout / Verify stable main closeout (push) Has been cancelled
Workflow Sanity / no-tabs (push) Has been cancelled
Workflow Sanity / actionlint (push) Has been cancelled
Workflow Sanity / generated-doc-baselines (push) Has been cancelled
CI / runner-admission (push) Has been cancelled
CI / preflight (push) Has been cancelled
CI / security-fast (push) Has been cancelled
CI / pnpm-store-warmup (push) Has been cancelled
CI / build-artifacts (push) Has been cancelled
CI / native-i18n (push) Has been cancelled
CI / ${{ matrix.check_name }} (push) Has been cancelled
CI / ${{ matrix.checkName }} (push) Has been cancelled
CI / checks-node-compat-node22 (push) Has been cancelled
CI / check-bundled-channel-config-metadata (push) Has been cancelled
CI / check-dependencies (push) Has been cancelled
CI / check-guards (push) Has been cancelled
CI / check-lint (push) Has been cancelled
CI / check-prod-types (push) Has been cancelled
CI / check-shrinkwrap (push) Has been cancelled
CI / check-test-types (push) Has been cancelled
CI / check-additional-boundaries-a (push) Has been cancelled
CI / check-additional-boundaries-bcd (push) Has been cancelled
CI / check-additional-extension-bundled (push) Has been cancelled
CI / check-additional-extension-channels (push) Has been cancelled
CI / check-additional-extension-package-boundary (push) Has been cancelled
CI / check-additional-runtime-topology-architecture (push) Has been cancelled
CI / check-session-accessor-boundary (push) Has been cancelled
CI / check-session-transcript-reader-boundary (push) Has been cancelled
CI / check-docs (push) Has been cancelled
CI / skills-python (push) Has been cancelled
CI / macos-swift (push) Has been cancelled
CI / ios-build (push) Has been cancelled
CI / ci-timings-summary (push) Has been cancelled
Native App Locale Refresh / Refresh native fa (push) Has been cancelled
Native App Locale Refresh / Refresh native fr (push) Has been cancelled
Native App Locale Refresh / Refresh native hi (push) Has been cancelled
Native App Locale Refresh / Refresh native id (push) Has been cancelled
Native App Locale Refresh / Refresh native it (push) Has been cancelled
Native App Locale Refresh / Refresh native ja-JP (push) Has been cancelled
Control UI Locale Refresh / plan (push) Has been cancelled
Control UI Locale Refresh / Refresh ${{ matrix.locale }} (push) Has been cancelled
Control UI Locale Refresh / Commit control UI locale refresh (push) Has been cancelled
Live Media Runner Image / Build live media runner image (push) Has been cancelled
Native App Locale Refresh / Refresh native ar (push) Has been cancelled
Native App Locale Refresh / Refresh native de (push) Has been cancelled
Native App Locale Refresh / Refresh native es (push) Has been cancelled
Native App Locale Refresh / Refresh native ko (push) Has been cancelled
Native App Locale Refresh / Refresh native nl (push) Has been cancelled
Native App Locale Refresh / Refresh native pl (push) Has been cancelled
Native App Locale Refresh / Refresh native pt-BR (push) Has been cancelled
Native App Locale Refresh / Refresh native ru (push) Has been cancelled
Native App Locale Refresh / Refresh native sv (push) Has been cancelled
Native App Locale Refresh / Refresh native th (push) Has been cancelled
Native App Locale Refresh / Refresh native tr (push) Has been cancelled
Native App Locale Refresh / Refresh native uk (push) Has been cancelled
Native App Locale Refresh / Refresh native vi (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-CN (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-TW (push) Has been cancelled
Native App Locale Refresh / Commit native locale refresh (push) Has been cancelled
Plugin Init Scaffold Validation / Validate provider scaffold (push) Has been cancelled
Plugin NPM Release / preview_plugins_npm (push) Has been cancelled
Plugin NPM Release / Validate release publish approval (push) Has been cancelled
Plugin NPM Release / preview_plugin_pack (push) Has been cancelled
Plugin NPM Release / publish_plugins_npm (push) Has been cancelled
Sandbox Common Smoke / sandbox-common-smoke (push) Has been cancelled
Website Installer Sync / static (push) Has been cancelled
Website Installer Sync / linux-docker (push) Has been cancelled
Website Installer Sync / macos-installer (push) Has been cancelled
Website Installer Sync / windows-installer (push) Has been cancelled
Website Installer Sync / sync-website (push) Has been cancelled
Adolf is a fork/vendored clone of github.com/openclaw/openclaw (v2026.6.11), free to diverge. Tree copied sans upstream .git; upstream remote added for future syncs. Node pinned to 24 (.nvmrc); engines already require >=22.19. Preserves docs/ARCHITECTURE.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LeqyaxJF2nbRXJtae2kNB2
This commit is contained in:
301
extensions/google/api.test.ts
Normal file
301
extensions/google/api.test.ts
Normal file
@@ -0,0 +1,301 @@
|
||||
// Google tests cover api plugin behavior.
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
isGoogleGenerativeAiApi,
|
||||
isGoogleVertexBaseUrl,
|
||||
isGoogleVertexHostname,
|
||||
normalizeGoogleApiBaseUrl,
|
||||
normalizeGoogleGenerativeAiBaseUrl,
|
||||
normalizeGoogleProviderConfig,
|
||||
parseGeminiAuth,
|
||||
resolveGoogleGenerativeAiHttpRequestConfig,
|
||||
resolveGoogleGenerativeAiApiOrigin,
|
||||
resolveGoogleGenerativeAiTransport,
|
||||
shouldNormalizeGoogleGenerativeAiProviderConfig,
|
||||
} from "./api.js";
|
||||
|
||||
describe("google generative ai helpers", () => {
|
||||
it("detects the Google Generative AI transport id", () => {
|
||||
expect(isGoogleGenerativeAiApi("google-generative-ai")).toBe(true);
|
||||
expect(isGoogleGenerativeAiApi("google-gemini-cli")).toBe(false);
|
||||
expect(isGoogleGenerativeAiApi(undefined)).toBe(false);
|
||||
});
|
||||
|
||||
it("normalizes only explicit Google Generative AI baseUrls", () => {
|
||||
expect(normalizeGoogleGenerativeAiBaseUrl("https://generativelanguage.googleapis.com")).toBe(
|
||||
"https://generativelanguage.googleapis.com/v1beta",
|
||||
);
|
||||
expect(normalizeGoogleGenerativeAiBaseUrl("https://proxy.example.com/google/v1beta")).toBe(
|
||||
"https://proxy.example.com/google/v1beta",
|
||||
);
|
||||
expect(normalizeGoogleGenerativeAiBaseUrl("https://aiplatform.googleapis.com")).toBe(
|
||||
"https://aiplatform.googleapis.com",
|
||||
);
|
||||
expect(normalizeGoogleGenerativeAiBaseUrl("proxy/generativelanguage.googleapis.com")).toBe(
|
||||
"proxy/generativelanguage.googleapis.com",
|
||||
);
|
||||
expect(normalizeGoogleGenerativeAiBaseUrl("generativelanguage.googleapis.com")).toBe(
|
||||
"generativelanguage.googleapis.com",
|
||||
);
|
||||
expect(normalizeGoogleGenerativeAiBaseUrl("https://xgenerativelanguage.googleapis.com")).toBe(
|
||||
"https://xgenerativelanguage.googleapis.com",
|
||||
);
|
||||
expect(normalizeGoogleGenerativeAiBaseUrl()).toBeUndefined();
|
||||
});
|
||||
|
||||
it("keeps /openai on generic Google base URL normalization and strips it only for native Gemini callers", () => {
|
||||
expect(
|
||||
normalizeGoogleApiBaseUrl("https://generativelanguage.googleapis.com/v1beta/openai"),
|
||||
).toBe("https://generativelanguage.googleapis.com/v1beta/openai");
|
||||
expect(
|
||||
normalizeGoogleGenerativeAiBaseUrl("https://generativelanguage.googleapis.com/v1beta/openai"),
|
||||
).toBe("https://generativelanguage.googleapis.com/v1beta");
|
||||
expect(
|
||||
normalizeGoogleGenerativeAiBaseUrl(
|
||||
"https://generativelanguage.googleapis.com/v1alpha/openai/",
|
||||
),
|
||||
).toBe("https://generativelanguage.googleapis.com/v1alpha");
|
||||
});
|
||||
|
||||
it("normalizes Google provider configs by provider key, provider api, or model api", () => {
|
||||
expect(
|
||||
shouldNormalizeGoogleGenerativeAiProviderConfig("google", {
|
||||
models: [{ api: "openai-completions" }],
|
||||
}),
|
||||
).toBe(true);
|
||||
expect(
|
||||
shouldNormalizeGoogleGenerativeAiProviderConfig("custom", {
|
||||
api: "google-generative-ai",
|
||||
models: [{ api: "openai-completions" }],
|
||||
}),
|
||||
).toBe(true);
|
||||
expect(
|
||||
shouldNormalizeGoogleGenerativeAiProviderConfig("custom", {
|
||||
models: [{ api: "google-generative-ai" }],
|
||||
}),
|
||||
).toBe(true);
|
||||
expect(
|
||||
shouldNormalizeGoogleGenerativeAiProviderConfig("custom", {
|
||||
api: "openai-completions",
|
||||
models: [{ api: "openai-completions" }],
|
||||
}),
|
||||
).toBe(false);
|
||||
expect(
|
||||
shouldNormalizeGoogleGenerativeAiProviderConfig("google", {
|
||||
api: "openai-completions",
|
||||
models: [{ api: "openai-completions" }],
|
||||
}),
|
||||
).toBe(false);
|
||||
expect(
|
||||
shouldNormalizeGoogleGenerativeAiProviderConfig("google-vertex", {
|
||||
baseUrl: "https://aiplatform.googleapis.com",
|
||||
}),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("detects native Google Vertex hosts by hostname only", () => {
|
||||
expect(isGoogleVertexHostname("aiplatform.googleapis.com")).toBe(true);
|
||||
expect(isGoogleVertexHostname("us-central1-aiplatform.googleapis.com")).toBe(true);
|
||||
expect(isGoogleVertexHostname("generativelanguage.googleapis.com")).toBe(false);
|
||||
expect(isGoogleVertexHostname("evil-aiplatform.googleapis.com.attacker.com")).toBe(false);
|
||||
expect(
|
||||
isGoogleVertexBaseUrl(
|
||||
"https://generativelanguage.googleapis.com/v1beta/proxy/aiplatform.googleapis.com",
|
||||
),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("normalizes transport baseUrls only for Google Generative AI", () => {
|
||||
expect(
|
||||
resolveGoogleGenerativeAiTransport({
|
||||
provider: "google",
|
||||
api: undefined,
|
||||
baseUrl: "https://generativelanguage.googleapis.com",
|
||||
}),
|
||||
).toEqual({
|
||||
api: "google-generative-ai",
|
||||
baseUrl: "https://generativelanguage.googleapis.com/v1beta",
|
||||
});
|
||||
expect(
|
||||
resolveGoogleGenerativeAiTransport({
|
||||
api: "google-generative-ai",
|
||||
baseUrl: "https://generativelanguage.googleapis.com",
|
||||
}),
|
||||
).toEqual({
|
||||
api: "google-generative-ai",
|
||||
baseUrl: "https://generativelanguage.googleapis.com/v1beta",
|
||||
});
|
||||
expect(
|
||||
resolveGoogleGenerativeAiTransport({
|
||||
api: "openai-completions",
|
||||
baseUrl: "https://generativelanguage.googleapis.com",
|
||||
}),
|
||||
).toEqual({
|
||||
api: "openai-completions",
|
||||
baseUrl: "https://generativelanguage.googleapis.com",
|
||||
});
|
||||
expect(
|
||||
resolveGoogleGenerativeAiTransport({
|
||||
provider: "google-vertex",
|
||||
api: undefined,
|
||||
baseUrl: "https://us-central1-aiplatform.googleapis.com",
|
||||
}),
|
||||
).toEqual({
|
||||
api: "google-vertex",
|
||||
baseUrl: "https://us-central1-aiplatform.googleapis.com",
|
||||
});
|
||||
expect(
|
||||
resolveGoogleGenerativeAiTransport({
|
||||
provider: "google-vertex",
|
||||
api: "openai-completions",
|
||||
baseUrl:
|
||||
"https://aiplatform.googleapis.com/v1/projects/test/locations/us-central1/endpoints/openapi",
|
||||
}),
|
||||
).toEqual({
|
||||
api: "openai-completions",
|
||||
baseUrl:
|
||||
"https://aiplatform.googleapis.com/v1/projects/test/locations/us-central1/endpoints/openapi",
|
||||
});
|
||||
});
|
||||
|
||||
it("normalizes google-vertex model ids without rewriting the OpenAI-compatible baseUrl", () => {
|
||||
expect(
|
||||
normalizeGoogleProviderConfig("google-vertex", {
|
||||
api: "openai-completions",
|
||||
baseUrl:
|
||||
"https://aiplatform.googleapis.com/v1/projects/test/locations/us-central1/endpoints/openapi",
|
||||
models: [
|
||||
{
|
||||
id: "gemini-3.1-flash-lite",
|
||||
name: "Gemini Flash Lite",
|
||||
input: ["text"],
|
||||
reasoning: false,
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 1,
|
||||
maxTokens: 1,
|
||||
},
|
||||
],
|
||||
}),
|
||||
).toEqual({
|
||||
api: "openai-completions",
|
||||
baseUrl:
|
||||
"https://aiplatform.googleapis.com/v1/projects/test/locations/us-central1/endpoints/openapi",
|
||||
models: [
|
||||
{
|
||||
contextWindow: 1,
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
id: "gemini-3.1-flash-lite",
|
||||
input: ["text"],
|
||||
maxTokens: 1,
|
||||
name: "Gemini Flash Lite",
|
||||
reasoning: false,
|
||||
},
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
it("derives the Gemini API origin without duplicating /v1beta", () => {
|
||||
expect(resolveGoogleGenerativeAiApiOrigin()).toBe("https://generativelanguage.googleapis.com");
|
||||
expect(resolveGoogleGenerativeAiApiOrigin("https://generativelanguage.googleapis.com")).toBe(
|
||||
"https://generativelanguage.googleapis.com",
|
||||
);
|
||||
expect(
|
||||
resolveGoogleGenerativeAiApiOrigin("https://generativelanguage.googleapis.com/v1beta"),
|
||||
).toBe("https://generativelanguage.googleapis.com");
|
||||
});
|
||||
|
||||
it("parses project-aware oauth auth payloads into bearer headers", () => {
|
||||
expect(
|
||||
parseGeminiAuth(JSON.stringify({ token: "oauth-token", projectId: "project-1" })),
|
||||
).toEqual({
|
||||
headers: {
|
||||
Authorization: "Bearer oauth-token",
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("falls back to API key headers for raw tokens", () => {
|
||||
expect(parseGeminiAuth("api-key-123")).toEqual({
|
||||
headers: {
|
||||
"x-goog-api-key": "api-key-123",
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("builds shared Google Generative AI HTTP request config", () => {
|
||||
const oauthConfig = resolveGoogleGenerativeAiHttpRequestConfig({
|
||||
apiKey: JSON.stringify({ token: "oauth-token" }),
|
||||
baseUrl: "https://generativelanguage.googleapis.com",
|
||||
capability: "audio",
|
||||
transport: "media-understanding",
|
||||
});
|
||||
expect(oauthConfig.baseUrl).toBe("https://generativelanguage.googleapis.com/v1beta");
|
||||
expect(oauthConfig.allowPrivateNetwork).toBe(false);
|
||||
expect(Object.fromEntries(new Headers(oauthConfig.headers).entries())).toEqual({
|
||||
authorization: "Bearer oauth-token",
|
||||
"content-type": "application/json",
|
||||
});
|
||||
|
||||
const apiKeyConfig = resolveGoogleGenerativeAiHttpRequestConfig({
|
||||
apiKey: "api-key-123",
|
||||
capability: "image",
|
||||
transport: "http",
|
||||
});
|
||||
expect(apiKeyConfig.baseUrl).toBe("https://generativelanguage.googleapis.com/v1beta");
|
||||
expect(apiKeyConfig.allowPrivateNetwork).toBe(false);
|
||||
expect(Object.fromEntries(new Headers(apiKeyConfig.headers).entries())).toEqual({
|
||||
"content-type": "application/json",
|
||||
"x-goog-api-key": "api-key-123",
|
||||
});
|
||||
});
|
||||
|
||||
it("preserves explicit OpenAI-compatible Google endpoints during provider normalization", () => {
|
||||
expect(
|
||||
resolveGoogleGenerativeAiTransport({
|
||||
api: "openai-completions",
|
||||
baseUrl: "https://generativelanguage.googleapis.com/v1beta/openai",
|
||||
}),
|
||||
).toEqual({
|
||||
api: "openai-completions",
|
||||
baseUrl: "https://generativelanguage.googleapis.com/v1beta/openai",
|
||||
});
|
||||
});
|
||||
|
||||
it("strips URL credentials during Google base URL normalization", () => {
|
||||
const normalized = normalizeGoogleApiBaseUrl(
|
||||
"https://user:secret@generativelanguage.googleapis.com/v1beta/openai?x=1#frag",
|
||||
);
|
||||
expect(normalized).toBe("https://generativelanguage.googleapis.com/v1beta/openai");
|
||||
});
|
||||
|
||||
it("rejects non-Google Gemini base URLs and honors explicit private-network opt-in", () => {
|
||||
expect(() =>
|
||||
resolveGoogleGenerativeAiHttpRequestConfig({
|
||||
apiKey: "api-key-123",
|
||||
baseUrl: "https://proxy.example.com/v1beta",
|
||||
capability: "image",
|
||||
transport: "http",
|
||||
}),
|
||||
).toThrow("Google Generative AI baseUrl must use https://generativelanguage.googleapis.com");
|
||||
|
||||
expect(() =>
|
||||
resolveGoogleGenerativeAiHttpRequestConfig({
|
||||
apiKey: "api-key-123",
|
||||
baseUrl: "http://generativelanguage.googleapis.com/v1beta",
|
||||
capability: "image",
|
||||
transport: "http",
|
||||
}),
|
||||
).toThrow("Google Generative AI baseUrl must use https://generativelanguage.googleapis.com");
|
||||
|
||||
const config = resolveGoogleGenerativeAiHttpRequestConfig({
|
||||
apiKey: "api-key-123",
|
||||
baseUrl: "https://generativelanguage.googleapis.com/v1beta",
|
||||
capability: "image",
|
||||
transport: "http",
|
||||
request: { allowPrivateNetwork: true },
|
||||
});
|
||||
expect(config.allowPrivateNetwork).toBe(true);
|
||||
});
|
||||
});
|
||||
94
extensions/google/api.ts
Normal file
94
extensions/google/api.ts
Normal file
@@ -0,0 +1,94 @@
|
||||
// Google API module exposes the plugin public contract.
|
||||
import {
|
||||
resolveProviderHttpRequestConfig,
|
||||
type ProviderRequestTransportOverrides,
|
||||
} from "openclaw/plugin-sdk/provider-http";
|
||||
import { parseGeminiAuth } from "./gemini-auth.js";
|
||||
export { parseGeminiAuth };
|
||||
export { applyGoogleGeminiModelDefault, GOOGLE_GEMINI_DEFAULT_MODEL } from "./onboard.js";
|
||||
import {
|
||||
DEFAULT_GOOGLE_API_BASE_URL,
|
||||
normalizeGoogleGenerativeAiBaseUrl,
|
||||
} from "./provider-policy.js";
|
||||
export { normalizeAntigravityModelId, normalizeGoogleModelId } from "./model-id.js";
|
||||
export {
|
||||
createGoogleThinkingPayloadWrapper,
|
||||
createGoogleThinkingStreamWrapper,
|
||||
isGoogleGemini3FlashModel,
|
||||
isGoogleGemini3ProModel,
|
||||
isGoogleGemini3ThinkingLevelModel,
|
||||
isGoogleThinkingRequiredModel,
|
||||
resolveGoogleGemini3ThinkingLevel,
|
||||
sanitizeGoogleThinkingPayload,
|
||||
stripInvalidGoogleThinkingBudget,
|
||||
type GoogleThinkingInputLevel,
|
||||
type GoogleThinkingLevel,
|
||||
} from "./thinking-api.js";
|
||||
export {
|
||||
buildGoogleGenerativeAiParams,
|
||||
createGoogleGenerativeAiTransportStreamFn,
|
||||
} from "./transport-stream.js";
|
||||
export {
|
||||
DEFAULT_GOOGLE_API_BASE_URL,
|
||||
isGoogleGenerativeAiApi,
|
||||
isGoogleVertexBaseUrl,
|
||||
isGoogleVertexHostname,
|
||||
normalizeGoogleApiBaseUrl,
|
||||
normalizeGoogleGenerativeAiBaseUrl,
|
||||
normalizeGoogleProviderConfig,
|
||||
resolveGoogleGenerativeAiApiOrigin,
|
||||
resolveGoogleGenerativeAiTransport,
|
||||
shouldNormalizeGoogleGenerativeAiProviderConfig,
|
||||
shouldNormalizeGoogleProviderConfig,
|
||||
} from "./provider-policy.js";
|
||||
export { buildGoogleGeminiCliProvider } from "./gemini-cli-provider.js";
|
||||
export { buildGoogleProvider } from "./provider-registration.js";
|
||||
|
||||
type GoogleGenerativeAiRequestOverrides = ProviderRequestTransportOverrides & {
|
||||
allowPrivateNetwork?: boolean;
|
||||
};
|
||||
|
||||
function resolveTrustedGoogleGenerativeAiBaseUrl(baseUrl?: string): string {
|
||||
const normalized =
|
||||
normalizeGoogleGenerativeAiBaseUrl(baseUrl ?? DEFAULT_GOOGLE_API_BASE_URL) ??
|
||||
DEFAULT_GOOGLE_API_BASE_URL;
|
||||
let url: URL;
|
||||
try {
|
||||
url = new URL(normalized);
|
||||
} catch {
|
||||
throw new Error(
|
||||
"Google Generative AI baseUrl must be a valid https URL on generativelanguage.googleapis.com",
|
||||
);
|
||||
}
|
||||
if (
|
||||
url.protocol !== "https:" ||
|
||||
url.hostname.toLowerCase() !== "generativelanguage.googleapis.com"
|
||||
) {
|
||||
throw new Error(
|
||||
"Google Generative AI baseUrl must use https://generativelanguage.googleapis.com",
|
||||
);
|
||||
}
|
||||
return normalized;
|
||||
}
|
||||
|
||||
export function resolveGoogleGenerativeAiHttpRequestConfig(params: {
|
||||
apiKey: string;
|
||||
baseUrl?: string;
|
||||
headers?: Record<string, string>;
|
||||
request?: GoogleGenerativeAiRequestOverrides;
|
||||
capability: "image" | "audio" | "video";
|
||||
transport: "http" | "media-understanding";
|
||||
}) {
|
||||
return resolveProviderHttpRequestConfig({
|
||||
baseUrl: resolveTrustedGoogleGenerativeAiBaseUrl(params.baseUrl),
|
||||
defaultBaseUrl: DEFAULT_GOOGLE_API_BASE_URL,
|
||||
allowPrivateNetwork: params.request?.allowPrivateNetwork,
|
||||
headers: params.headers,
|
||||
request: params.request,
|
||||
defaultHeaders: parseGeminiAuth(params.apiKey).headers,
|
||||
provider: "google",
|
||||
api: "google-generative-ai",
|
||||
capability: params.capability,
|
||||
transport: params.transport,
|
||||
});
|
||||
}
|
||||
406
extensions/google/cli-backend-auth.runtime.ts
Normal file
406
extensions/google/cli-backend-auth.runtime.ts
Normal file
@@ -0,0 +1,406 @@
|
||||
import crypto from "node:crypto";
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import type { CliBackendPreparedExecution } from "openclaw/plugin-sdk/cli-backend";
|
||||
import { isRecord } from "openclaw/plugin-sdk/string-coerce-runtime";
|
||||
import { resolvePreferredOpenClawTmpDir } from "openclaw/plugin-sdk/temp-path";
|
||||
import {
|
||||
GOOGLE_GEMINI_CLI_PROVIDER_ID,
|
||||
resolveGeminiCliProfileHome as resolveGeminiCliProfileHomePath,
|
||||
} from "./gemini-cli-auth-home.js";
|
||||
|
||||
const GEMINI_CLI_PROVIDER_ID = GOOGLE_GEMINI_CLI_PROVIDER_ID;
|
||||
const GOOGLE_PROVIDER_ID = "google";
|
||||
const VERCEL_AI_GATEWAY_PROVIDER_ID = "vercel-ai-gateway";
|
||||
const GEMINI_CLI_CREDENTIALS_FILENAME = "gemini-credentials.json";
|
||||
const GEMINI_CLI_GCA_AUTH_ENV = [
|
||||
"GOOGLE_GENAI_USE_GCA",
|
||||
"GOOGLE_CLOUD_ACCESS_TOKEN",
|
||||
"GOOGLE_APPLICATION_CREDENTIALS",
|
||||
"GEMINI_FORCE_ENCRYPTED_FILE_STORAGE",
|
||||
"GEMINI_FORCE_FILE_STORAGE",
|
||||
];
|
||||
const GEMINI_CLI_API_KEY_AUTH_ENV = [
|
||||
...GEMINI_CLI_GCA_AUTH_ENV,
|
||||
"GOOGLE_GENAI_USE_VERTEXAI",
|
||||
"GOOGLE_API_KEY",
|
||||
"GOOGLE_CLOUD_PROJECT",
|
||||
"GOOGLE_CLOUD_PROJECT_ID",
|
||||
"GOOGLE_CLOUD_QUOTA_PROJECT",
|
||||
"GOOGLE_CLOUD_LOCATION",
|
||||
"GOOGLE_GEMINI_BASE_URL",
|
||||
"GEMINI_CLI_CUSTOM_HEADERS",
|
||||
"GEMINI_API_KEY_AUTH_MECHANISM",
|
||||
];
|
||||
const GEMINI_CLI_PROFILE_AUTH_ENV = [...GEMINI_CLI_API_KEY_AUTH_ENV, "GEMINI_API_KEY"];
|
||||
const GEMINI_CLI_PROFILE_SETTINGS_ENV = ["GEMINI_CLI_SYSTEM_SETTINGS_PATH"];
|
||||
|
||||
type GeminiAuthProfileCredential = {
|
||||
type: "api_key" | "oauth" | "token";
|
||||
provider: string;
|
||||
key?: string;
|
||||
token?: string;
|
||||
access?: string;
|
||||
refresh?: string;
|
||||
expires?: number;
|
||||
idToken?: string;
|
||||
projectId?: string;
|
||||
};
|
||||
|
||||
type GeminiOAuthCredential = GeminiAuthProfileCredential & {
|
||||
type: "oauth";
|
||||
provider: typeof GEMINI_CLI_PROVIDER_ID;
|
||||
access: string;
|
||||
refresh: string;
|
||||
expires: number;
|
||||
};
|
||||
|
||||
type GeminiApiKeyCredential = GeminiAuthProfileCredential & {
|
||||
type: "api_key";
|
||||
provider: typeof GEMINI_CLI_PROVIDER_ID | typeof GOOGLE_PROVIDER_ID;
|
||||
key: string;
|
||||
};
|
||||
|
||||
type GeminiCliAuthHomeContext = {
|
||||
agentDir?: string;
|
||||
authProfileId?: string;
|
||||
systemSettingsPath?: string;
|
||||
};
|
||||
|
||||
type GeminiCliAuthSelectedType = "oauth-personal" | "gemini-api-key";
|
||||
|
||||
function normalizeString(value: string | undefined): string | undefined {
|
||||
const trimmed = value?.trim();
|
||||
return trimmed ? trimmed : undefined;
|
||||
}
|
||||
|
||||
function throwUnsupportedGeminiCredential(credential: GeminiAuthProfileCredential): never {
|
||||
if (credential.provider === VERCEL_AI_GATEWAY_PROVIDER_ID) {
|
||||
throw new Error(
|
||||
"Gemini CLI execution cannot use a vercel-ai-gateway auth profile. Use the OpenClaw vercel-ai-gateway provider instead.",
|
||||
);
|
||||
}
|
||||
throw new Error("Gemini CLI execution requires a google-gemini-cli auth profile.");
|
||||
}
|
||||
|
||||
function throwUnstageableSelectedGeminiProfile(
|
||||
ctx: GeminiCliAuthHomeContext,
|
||||
credential: GeminiAuthProfileCredential | undefined,
|
||||
): never {
|
||||
const authProfileId = normalizeString(ctx.authProfileId);
|
||||
if (!authProfileId) {
|
||||
throw new Error("Gemini CLI execution requires a selected auth profile.");
|
||||
}
|
||||
if (!credential) {
|
||||
throw new Error(
|
||||
"Gemini CLI auth profile was selected but no credential material was found. Re-authenticate with `openclaw models auth login --provider google-gemini-cli --force`.",
|
||||
);
|
||||
}
|
||||
if (credential.provider !== GEMINI_CLI_PROVIDER_ID) {
|
||||
throwUnsupportedGeminiCredential(credential);
|
||||
}
|
||||
throw new Error(
|
||||
"Gemini CLI execution supports google-gemini-cli OAuth or API-key auth profiles. Re-authenticate with `openclaw models auth login --provider google-gemini-cli --force`.",
|
||||
);
|
||||
}
|
||||
|
||||
function requireGeminiOAuthCredential(
|
||||
credential: GeminiAuthProfileCredential | undefined,
|
||||
): GeminiOAuthCredential | null {
|
||||
if (!credential) {
|
||||
return null;
|
||||
}
|
||||
if (credential.type !== "oauth") {
|
||||
return null;
|
||||
}
|
||||
if (credential.provider !== GEMINI_CLI_PROVIDER_ID) {
|
||||
throwUnsupportedGeminiCredential(credential);
|
||||
}
|
||||
|
||||
const access = normalizeString(credential.access);
|
||||
const refresh = normalizeString(credential.refresh);
|
||||
if (
|
||||
!access ||
|
||||
!refresh ||
|
||||
typeof credential.expires !== "number" ||
|
||||
!Number.isFinite(credential.expires)
|
||||
) {
|
||||
throw new Error(
|
||||
"Gemini CLI OAuth profile is missing usable token material. Re-authenticate with `openclaw models auth login --provider google-gemini-cli --force`.",
|
||||
);
|
||||
}
|
||||
|
||||
return {
|
||||
...credential,
|
||||
type: "oauth",
|
||||
provider: GEMINI_CLI_PROVIDER_ID,
|
||||
access,
|
||||
refresh,
|
||||
expires: credential.expires,
|
||||
idToken: normalizeString(credential.idToken),
|
||||
projectId: normalizeString(credential.projectId),
|
||||
};
|
||||
}
|
||||
|
||||
function requireGeminiApiKeyCredential(
|
||||
credential: GeminiAuthProfileCredential | undefined,
|
||||
): GeminiApiKeyCredential | null {
|
||||
if (!credential) {
|
||||
return null;
|
||||
}
|
||||
if (credential.type !== "api_key") {
|
||||
return null;
|
||||
}
|
||||
if (
|
||||
credential.provider !== GEMINI_CLI_PROVIDER_ID &&
|
||||
credential.provider !== GOOGLE_PROVIDER_ID
|
||||
) {
|
||||
throwUnsupportedGeminiCredential(credential);
|
||||
}
|
||||
|
||||
const key = normalizeString(credential.key);
|
||||
if (!key) {
|
||||
throw new Error("Gemini CLI API-key profile is missing usable key material.");
|
||||
}
|
||||
|
||||
return {
|
||||
...credential,
|
||||
type: "api_key",
|
||||
provider: credential.provider,
|
||||
key,
|
||||
};
|
||||
}
|
||||
|
||||
function resolveGeminiCliProfileHome(ctx: GeminiCliAuthHomeContext): {
|
||||
home: string;
|
||||
geminiDir: string;
|
||||
} {
|
||||
const agentDir = normalizeString(ctx.agentDir);
|
||||
if (!agentDir) {
|
||||
throw new Error("Gemini CLI auth profile execution requires an agent directory.");
|
||||
}
|
||||
const authProfileId = normalizeString(ctx.authProfileId);
|
||||
if (!authProfileId) {
|
||||
throw new Error("Gemini CLI auth profile execution requires a selected auth profile.");
|
||||
}
|
||||
|
||||
const home = resolveGeminiCliProfileHomePath(agentDir, authProfileId);
|
||||
return { home, geminiDir: path.join(home, ".gemini") };
|
||||
}
|
||||
|
||||
function readGeminiAuthProfileCredential(
|
||||
credential: unknown,
|
||||
): GeminiAuthProfileCredential | undefined {
|
||||
if (!isRecord(credential)) {
|
||||
return undefined;
|
||||
}
|
||||
return credential as GeminiAuthProfileCredential;
|
||||
}
|
||||
|
||||
async function readGeminiCliJsonObject(
|
||||
filePath: string | undefined,
|
||||
): Promise<Record<string, unknown>> {
|
||||
const normalized = normalizeString(filePath);
|
||||
if (!normalized) {
|
||||
return {};
|
||||
}
|
||||
try {
|
||||
const parsed = JSON.parse(await fs.readFile(normalized, "utf8")) as unknown;
|
||||
if (!isRecord(parsed)) {
|
||||
throw new Error(`Gemini CLI system settings must be a JSON object: ${normalized}`);
|
||||
}
|
||||
return { ...parsed };
|
||||
} catch (error) {
|
||||
if (
|
||||
error &&
|
||||
typeof error === "object" &&
|
||||
"code" in error &&
|
||||
(error as { code?: unknown }).code === "ENOENT"
|
||||
) {
|
||||
return {};
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
function buildGeminiCliAuthSettings(
|
||||
selectedType: GeminiCliAuthSelectedType,
|
||||
): Record<string, unknown> {
|
||||
return { security: { auth: { selectedType } } };
|
||||
}
|
||||
|
||||
async function buildGeminiCliSystemSettings(
|
||||
ctx: GeminiCliAuthHomeContext,
|
||||
selectedType: GeminiCliAuthSelectedType,
|
||||
): Promise<Record<string, unknown>> {
|
||||
const base = await readGeminiCliJsonObject(ctx.systemSettingsPath);
|
||||
const security = isRecord(base.security) ? { ...base.security } : {};
|
||||
const auth = isRecord(security.auth) ? { ...security.auth } : {};
|
||||
const enforcedType = normalizeString(
|
||||
typeof auth.enforcedType === "string" ? auth.enforcedType : undefined,
|
||||
);
|
||||
if (enforcedType && enforcedType !== selectedType) {
|
||||
throw new Error(
|
||||
`Gemini CLI system settings enforce ${enforcedType} auth, but the selected OpenClaw profile requires ${selectedType}.`,
|
||||
);
|
||||
}
|
||||
security.auth = { ...auth, selectedType };
|
||||
return {
|
||||
...base,
|
||||
security,
|
||||
};
|
||||
}
|
||||
|
||||
async function writeGeminiCliJson(filePath: string, value: unknown): Promise<void> {
|
||||
const tempPath = path.join(
|
||||
path.dirname(filePath),
|
||||
`.${path.basename(filePath)}.${process.pid}.${crypto.randomUUID()}.tmp`,
|
||||
);
|
||||
await fs.writeFile(tempPath, `${JSON.stringify(value, null, 2)}\n`, {
|
||||
encoding: "utf8",
|
||||
mode: 0o600,
|
||||
});
|
||||
await fs.chmod(tempPath, 0o600);
|
||||
await fs.rename(tempPath, filePath);
|
||||
await fs.chmod(filePath, 0o600);
|
||||
}
|
||||
|
||||
async function prepareGeminiCliProfileHome(
|
||||
ctx: GeminiCliAuthHomeContext,
|
||||
selectedType: GeminiCliAuthSelectedType,
|
||||
): Promise<{
|
||||
home: string;
|
||||
geminiDir: string;
|
||||
systemSettingsPath: string;
|
||||
beforeExecution: () => Promise<void>;
|
||||
cleanup: () => Promise<void>;
|
||||
}> {
|
||||
const { home, geminiDir } = resolveGeminiCliProfileHome(ctx);
|
||||
const settings = buildGeminiCliAuthSettings(selectedType);
|
||||
const systemSettings = await buildGeminiCliSystemSettings(ctx, selectedType);
|
||||
const systemSettingsDir = await fs.mkdtemp(
|
||||
path.join(resolvePreferredOpenClawTmpDir(), "openclaw-gemini-cli-"),
|
||||
);
|
||||
await fs.chmod(systemSettingsDir, 0o700);
|
||||
const systemSettingsPath = path.join(systemSettingsDir, "settings.json");
|
||||
return {
|
||||
home,
|
||||
geminiDir,
|
||||
systemSettingsPath,
|
||||
beforeExecution: async () => {
|
||||
await fs.mkdir(geminiDir, { recursive: true, mode: 0o700 });
|
||||
await fs.chmod(home, 0o700);
|
||||
await fs.chmod(geminiDir, 0o700);
|
||||
await Promise.all([
|
||||
writeGeminiCliJson(path.join(geminiDir, "settings.json"), settings),
|
||||
writeGeminiCliJson(path.join(home, "settings.json"), settings),
|
||||
writeGeminiCliJson(systemSettingsPath, systemSettings),
|
||||
]);
|
||||
},
|
||||
cleanup: async () => {
|
||||
await fs.rm(systemSettingsDir, { recursive: true, force: true });
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function clearGeminiCliCachedCredentials(geminiDir: string): Promise<void> {
|
||||
// Gemini prefers its token store over oauth_creds.json. Rebuild that store
|
||||
// from the selected OpenClaw profile each run so stale CLI auth cannot win.
|
||||
await fs.rm(path.join(geminiDir, GEMINI_CLI_CREDENTIALS_FILENAME), { force: true });
|
||||
}
|
||||
|
||||
function buildGeminiCliProjectEnv(projectId: string | undefined): Record<string, string> {
|
||||
const normalized = normalizeString(projectId);
|
||||
if (!normalized) {
|
||||
return {};
|
||||
}
|
||||
return {
|
||||
GOOGLE_CLOUD_PROJECT: normalized,
|
||||
GOOGLE_CLOUD_PROJECT_ID: normalized,
|
||||
GOOGLE_CLOUD_QUOTA_PROJECT: normalized,
|
||||
};
|
||||
}
|
||||
|
||||
async function prepareGeminiCliOAuthHome(
|
||||
ctx: GeminiCliAuthHomeContext,
|
||||
credential: GeminiAuthProfileCredential | undefined,
|
||||
): Promise<CliBackendPreparedExecution | null> {
|
||||
const oauth = requireGeminiOAuthCredential(credential);
|
||||
if (!oauth) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const profileHome = await prepareGeminiCliProfileHome(ctx, "oauth-personal");
|
||||
const idToken = normalizeString(oauth.idToken);
|
||||
const oauthCreds: Record<string, string | number> = {
|
||||
access_token: oauth.access,
|
||||
refresh_token: oauth.refresh,
|
||||
expiry_date: oauth.expires,
|
||||
token_type: "Bearer",
|
||||
};
|
||||
if (idToken) {
|
||||
oauthCreds.id_token = idToken;
|
||||
}
|
||||
|
||||
return {
|
||||
env: {
|
||||
GEMINI_CLI_HOME: profileHome.home,
|
||||
GEMINI_CLI_SYSTEM_SETTINGS_PATH: profileHome.systemSettingsPath,
|
||||
GEMINI_FORCE_FILE_STORAGE: "true",
|
||||
...buildGeminiCliProjectEnv(oauth.projectId),
|
||||
},
|
||||
clearEnv: [...GEMINI_CLI_PROFILE_AUTH_ENV, ...GEMINI_CLI_PROFILE_SETTINGS_ENV],
|
||||
beforeExecution: async () => {
|
||||
await profileHome.beforeExecution();
|
||||
await clearGeminiCliCachedCredentials(profileHome.geminiDir);
|
||||
await writeGeminiCliJson(path.join(profileHome.geminiDir, "oauth_creds.json"), oauthCreds);
|
||||
},
|
||||
cleanup: profileHome.cleanup,
|
||||
};
|
||||
}
|
||||
|
||||
async function prepareGeminiCliApiKeyHome(
|
||||
ctx: GeminiCliAuthHomeContext,
|
||||
credential: GeminiAuthProfileCredential | undefined,
|
||||
): Promise<CliBackendPreparedExecution | null> {
|
||||
const apiKey = requireGeminiApiKeyCredential(credential);
|
||||
if (!apiKey) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const profileHome = await prepareGeminiCliProfileHome(ctx, "gemini-api-key");
|
||||
return {
|
||||
env: {
|
||||
GEMINI_CLI_HOME: profileHome.home,
|
||||
GEMINI_CLI_SYSTEM_SETTINGS_PATH: profileHome.systemSettingsPath,
|
||||
GEMINI_FORCE_FILE_STORAGE: "true",
|
||||
GEMINI_API_KEY: apiKey.key,
|
||||
},
|
||||
clearEnv: [...GEMINI_CLI_PROFILE_AUTH_ENV, ...GEMINI_CLI_PROFILE_SETTINGS_ENV],
|
||||
beforeExecution: async () => {
|
||||
await profileHome.beforeExecution();
|
||||
await Promise.all([
|
||||
fs.rm(path.join(profileHome.geminiDir, "oauth_creds.json"), { force: true }),
|
||||
clearGeminiCliCachedCredentials(profileHome.geminiDir),
|
||||
]);
|
||||
},
|
||||
cleanup: profileHome.cleanup,
|
||||
};
|
||||
}
|
||||
|
||||
export async function prepareGeminiCliAuthHome(
|
||||
ctx: GeminiCliAuthHomeContext,
|
||||
credential: unknown,
|
||||
): Promise<CliBackendPreparedExecution | null> {
|
||||
const authCredential = readGeminiAuthProfileCredential(credential);
|
||||
const prepared =
|
||||
(await prepareGeminiCliOAuthHome(ctx, authCredential)) ??
|
||||
(await prepareGeminiCliApiKeyHome(ctx, authCredential));
|
||||
if (prepared) {
|
||||
return prepared;
|
||||
}
|
||||
if (normalizeString(ctx.authProfileId)) {
|
||||
throwUnstageableSelectedGeminiProfile(ctx, authCredential);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
129
extensions/google/cli-backend.ts
Normal file
129
extensions/google/cli-backend.ts
Normal file
@@ -0,0 +1,129 @@
|
||||
import type { CliBackendPlugin } from "openclaw/plugin-sdk/cli-backend";
|
||||
import {
|
||||
CLI_FRESH_WATCHDOG_DEFAULTS,
|
||||
CLI_RESUME_WATCHDOG_DEFAULTS,
|
||||
} from "openclaw/plugin-sdk/cli-backend";
|
||||
|
||||
const GEMINI_MODEL_ALIASES: Record<string, string> = {
|
||||
pro: "gemini-3.1-pro-preview",
|
||||
flash: "gemini-3.1-flash-preview",
|
||||
"flash-lite": "gemini-3.1-flash-lite",
|
||||
};
|
||||
const GEMINI_CLI_DEFAULT_MODEL_REF = "google-gemini-cli/gemini-3-flash-preview";
|
||||
|
||||
type GeminiCliBackendConfig = CliBackendPlugin["config"];
|
||||
type GeminiCliOutputMode = NonNullable<GeminiCliBackendConfig["output"]>;
|
||||
|
||||
function mapGeminiCliOutputFormat(value: string | undefined): GeminiCliOutputMode | undefined {
|
||||
if (value === "stream-json") {
|
||||
return "jsonl";
|
||||
}
|
||||
if (value === "json" || value === "text") {
|
||||
return value;
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function readGeminiCliOutputFormat(args: readonly string[] | undefined): GeminiCliOutputMode {
|
||||
for (let index = 0; index < (args?.length ?? 0); index += 1) {
|
||||
const arg = args?.[index];
|
||||
if (arg === "--output-format" || arg === "-o") {
|
||||
return mapGeminiCliOutputFormat(args?.[index + 1]) ?? "text";
|
||||
}
|
||||
const inline = arg?.startsWith("--output-format=")
|
||||
? arg.slice("--output-format=".length)
|
||||
: arg?.startsWith("-o=")
|
||||
? arg.slice("-o=".length)
|
||||
: undefined;
|
||||
const mapped = mapGeminiCliOutputFormat(inline);
|
||||
if (mapped) {
|
||||
return mapped;
|
||||
}
|
||||
}
|
||||
return "text";
|
||||
}
|
||||
|
||||
function normalizeGeminiCliBackendConfig(config: GeminiCliBackendConfig): GeminiCliBackendConfig {
|
||||
const output = readGeminiCliOutputFormat(config.args);
|
||||
const resumeOutput = readGeminiCliOutputFormat(config.resumeArgs ?? config.args);
|
||||
const usesStreamJson = output === "jsonl" || resumeOutput === "jsonl";
|
||||
return {
|
||||
...config,
|
||||
output,
|
||||
resumeOutput,
|
||||
jsonlDialect: usesStreamJson ? "gemini-stream-json" : undefined,
|
||||
};
|
||||
}
|
||||
|
||||
export function buildGoogleGeminiCliBackend(): CliBackendPlugin {
|
||||
return {
|
||||
id: "google-gemini-cli",
|
||||
modelProvider: "google",
|
||||
liveTest: {
|
||||
defaultModelRef: GEMINI_CLI_DEFAULT_MODEL_REF,
|
||||
defaultImageProbe: true,
|
||||
defaultMcpProbe: true,
|
||||
docker: {
|
||||
npmPackage: "@google/gemini-cli",
|
||||
binaryName: "gemini",
|
||||
},
|
||||
},
|
||||
bundleMcp: true,
|
||||
bundleMcpMode: "gemini-system-settings",
|
||||
nativeToolMode: "always-on",
|
||||
authEpochMode: "profile-only",
|
||||
normalizeConfig: normalizeGeminiCliBackendConfig,
|
||||
prepareExecution: async (ctx) => {
|
||||
const { prepareGeminiCliAuthHome } = await import("./cli-backend-auth.runtime.js");
|
||||
return await prepareGeminiCliAuthHome(
|
||||
{
|
||||
agentDir: ctx.agentDir,
|
||||
authProfileId: ctx.authProfileId,
|
||||
systemSettingsPath:
|
||||
(ctx as typeof ctx & { env?: Record<string, string> }).env
|
||||
?.GEMINI_CLI_SYSTEM_SETTINGS_PATH ?? process.env.GEMINI_CLI_SYSTEM_SETTINGS_PATH,
|
||||
},
|
||||
(ctx as typeof ctx & { authCredential?: unknown }).authCredential,
|
||||
);
|
||||
},
|
||||
config: {
|
||||
command: "gemini",
|
||||
args: [
|
||||
"--skip-trust",
|
||||
"--approval-mode",
|
||||
"auto_edit",
|
||||
"--output-format",
|
||||
"stream-json",
|
||||
"--prompt",
|
||||
"{prompt}",
|
||||
],
|
||||
resumeArgs: [
|
||||
"--skip-trust",
|
||||
"--approval-mode",
|
||||
"auto_edit",
|
||||
"--resume",
|
||||
"{sessionId}",
|
||||
"--output-format",
|
||||
"stream-json",
|
||||
"--prompt",
|
||||
"{prompt}",
|
||||
],
|
||||
output: "jsonl",
|
||||
input: "arg",
|
||||
jsonlDialect: "gemini-stream-json",
|
||||
imageArg: "@",
|
||||
imagePathScope: "workspace",
|
||||
modelArg: "--model",
|
||||
modelAliases: GEMINI_MODEL_ALIASES,
|
||||
sessionMode: "existing",
|
||||
sessionIdFields: ["session_id", "sessionId"],
|
||||
reliability: {
|
||||
watchdog: {
|
||||
fresh: { ...CLI_FRESH_WATCHDOG_DEFAULTS },
|
||||
resume: { ...CLI_RESUME_WATCHDOG_DEFAULTS },
|
||||
},
|
||||
},
|
||||
serialize: true,
|
||||
},
|
||||
};
|
||||
}
|
||||
116
extensions/google/default-model.test.ts
Normal file
116
extensions/google/default-model.test.ts
Normal file
@@ -0,0 +1,116 @@
|
||||
// Google tests cover default model plugin behavior.
|
||||
import type { OpenClawConfig } from "openclaw/plugin-sdk/provider-onboard";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { applyGoogleGeminiModelDefault, GOOGLE_GEMINI_DEFAULT_MODEL } from "./api.js";
|
||||
|
||||
describe("google default model", () => {
|
||||
it("sets defaults when model is unset", () => {
|
||||
const cfg: OpenClawConfig = { agents: { defaults: {} } };
|
||||
const applied = applyGoogleGeminiModelDefault(cfg);
|
||||
expect(applied.changed).toBe(true);
|
||||
expect(applied.next.agents?.defaults?.model).toEqual({ primary: GOOGLE_GEMINI_DEFAULT_MODEL });
|
||||
});
|
||||
|
||||
it("overrides existing models", () => {
|
||||
const applied = applyGoogleGeminiModelDefault({
|
||||
agents: { defaults: { model: { primary: "anthropic/claude-opus-4-6" } } },
|
||||
} as OpenClawConfig);
|
||||
expect(applied.changed).toBe(true);
|
||||
expect(applied.next.agents?.defaults?.model).toEqual({ primary: GOOGLE_GEMINI_DEFAULT_MODEL });
|
||||
});
|
||||
|
||||
it("normalizes retired Gemini model map keys when applying the default", () => {
|
||||
const applied = applyGoogleGeminiModelDefault({
|
||||
agents: {
|
||||
defaults: {
|
||||
model: {
|
||||
primary: "google/gemini-3-pro-preview",
|
||||
fallbacks: ["google/gemini-3-pro-preview"],
|
||||
},
|
||||
models: {
|
||||
"google/gemini-3-pro-preview": { alias: "gemini" },
|
||||
},
|
||||
},
|
||||
},
|
||||
} as OpenClawConfig);
|
||||
|
||||
expect(applied.changed).toBe(true);
|
||||
expect(applied.next.agents?.defaults?.model).toEqual({
|
||||
primary: "google/gemini-3.1-pro-preview",
|
||||
fallbacks: ["google/gemini-3.1-pro-preview"],
|
||||
});
|
||||
expect(applied.next.agents?.defaults?.models).toEqual({
|
||||
"google/gemini-3.1-pro-preview": { alias: "gemini" },
|
||||
});
|
||||
});
|
||||
|
||||
it("normalizes retired Gemini model maps even when the primary is already current", () => {
|
||||
const applied = applyGoogleGeminiModelDefault({
|
||||
agents: {
|
||||
defaults: {
|
||||
model: {
|
||||
primary: GOOGLE_GEMINI_DEFAULT_MODEL,
|
||||
fallbacks: ["google/gemini-3-pro-preview"],
|
||||
},
|
||||
models: {
|
||||
"google/gemini-3-pro-preview": { alias: "gemini" },
|
||||
},
|
||||
},
|
||||
},
|
||||
} as OpenClawConfig);
|
||||
|
||||
expect(applied.changed).toBe(true);
|
||||
expect(applied.next.agents?.defaults?.model).toEqual({
|
||||
primary: GOOGLE_GEMINI_DEFAULT_MODEL,
|
||||
fallbacks: [GOOGLE_GEMINI_DEFAULT_MODEL],
|
||||
});
|
||||
expect(applied.next.agents?.defaults?.models).toEqual({
|
||||
[GOOGLE_GEMINI_DEFAULT_MODEL]: { alias: "gemini" },
|
||||
});
|
||||
});
|
||||
|
||||
it("normalizes retired Gemini provider catalog rows when the primary is already current", () => {
|
||||
const applied = applyGoogleGeminiModelDefault({
|
||||
agents: {
|
||||
defaults: {
|
||||
model: {
|
||||
primary: GOOGLE_GEMINI_DEFAULT_MODEL,
|
||||
},
|
||||
},
|
||||
},
|
||||
models: {
|
||||
providers: {
|
||||
google: {
|
||||
api: "google-generative-ai",
|
||||
baseUrl: "https://generativelanguage.googleapis.com/v1beta",
|
||||
models: [
|
||||
{
|
||||
id: "google/gemini-3-pro-preview",
|
||||
name: "Gemini 3 Pro",
|
||||
contextWindow: 1_000_000,
|
||||
maxTokens: 8192,
|
||||
input: ["text"],
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
reasoning: true,
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
} as OpenClawConfig);
|
||||
|
||||
expect(applied.changed).toBe(true);
|
||||
expect(applied.next.models?.providers?.google?.models?.map((model) => model.id)).toEqual([
|
||||
GOOGLE_GEMINI_DEFAULT_MODEL,
|
||||
]);
|
||||
});
|
||||
|
||||
it("no-ops when already on the target default", () => {
|
||||
const cfg = {
|
||||
agents: { defaults: { model: { primary: GOOGLE_GEMINI_DEFAULT_MODEL } } },
|
||||
} as OpenClawConfig;
|
||||
const applied = applyGoogleGeminiModelDefault(cfg);
|
||||
expect(applied.changed).toBe(false);
|
||||
expect(applied.next).toEqual(cfg);
|
||||
});
|
||||
});
|
||||
19
extensions/google/doctor-contract-api.ts
Normal file
19
extensions/google/doctor-contract-api.ts
Normal file
@@ -0,0 +1,19 @@
|
||||
// Google API module exposes the plugin public contract.
|
||||
import type { DoctorSessionRouteStateOwner } from "openclaw/plugin-sdk/runtime-doctor";
|
||||
|
||||
export const sessionRouteStateOwners: DoctorSessionRouteStateOwner[] = [
|
||||
{
|
||||
id: "google",
|
||||
label: "Google",
|
||||
providerIds: ["google", "google-antigravity", "google-gemini-cli", "google-vertex"],
|
||||
runtimeIds: ["google-gemini-cli"],
|
||||
cliSessionKeys: ["google-gemini-cli", "gemini-cli"],
|
||||
authProfilePrefixes: [
|
||||
"google:",
|
||||
"google-antigravity:",
|
||||
"google-gemini-cli:",
|
||||
"google-vertex:",
|
||||
"gemini-cli:",
|
||||
],
|
||||
},
|
||||
];
|
||||
246
extensions/google/embedding-batch.test.ts
Normal file
246
extensions/google/embedding-batch.test.ts
Normal file
@@ -0,0 +1,246 @@
|
||||
// Google tests cover embedding batch bounded JSON response reads.
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { runGeminiEmbeddingBatches } from "./embedding-batch.js";
|
||||
import type { GeminiEmbeddingClient } from "./embedding-provider.js";
|
||||
|
||||
// Pass-through so onResponse receives real Response objects (required by
|
||||
// readProviderJsonResponse which needs a real .body ReadableStream).
|
||||
vi.mock("openclaw/plugin-sdk/memory-core-host-engine-embeddings", async (importOriginal) => {
|
||||
const actual =
|
||||
await importOriginal<typeof import("openclaw/plugin-sdk/memory-core-host-engine-embeddings")>();
|
||||
return {
|
||||
...actual,
|
||||
withRemoteHttpResponse: async <T>(params: {
|
||||
url: string;
|
||||
ssrfPolicy?: unknown;
|
||||
init?: RequestInit;
|
||||
onResponse: (response: Response) => Promise<T>;
|
||||
}): Promise<T> => {
|
||||
const response = await fetch(params.url, params.init);
|
||||
return await params.onResponse(response);
|
||||
},
|
||||
};
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
vi.unstubAllGlobals();
|
||||
});
|
||||
|
||||
function fetchInputUrl(input: RequestInfo | URL): string {
|
||||
if (typeof input === "string") {
|
||||
return input;
|
||||
}
|
||||
if (input instanceof URL) {
|
||||
return input.href;
|
||||
}
|
||||
return input.url;
|
||||
}
|
||||
|
||||
function jsonResponse(body: unknown, status = 200): Response {
|
||||
return new Response(JSON.stringify(body), {
|
||||
status,
|
||||
headers: { "Content-Type": "application/json" },
|
||||
});
|
||||
}
|
||||
|
||||
function makeGeminiClient(): GeminiEmbeddingClient {
|
||||
return {
|
||||
baseUrl: "https://gemini-compatible.example/v1beta",
|
||||
model: "text-embedding-004",
|
||||
modelPath: "models/text-embedding-004",
|
||||
headers: { "x-goog-api-key": "test-key" },
|
||||
apiKeys: ["test-key"],
|
||||
ssrfPolicy: undefined,
|
||||
};
|
||||
}
|
||||
|
||||
type GeminiBatchRequest = Parameters<typeof runGeminiEmbeddingBatches>[0]["requests"][number];
|
||||
|
||||
function singleRequest(): GeminiBatchRequest[] {
|
||||
return [
|
||||
{
|
||||
custom_id: "r0",
|
||||
request: {
|
||||
model: "models/text-embedding-004",
|
||||
content: { parts: [{ text: "hello" }] },
|
||||
taskType: "RETRIEVAL_DOCUMENT",
|
||||
},
|
||||
},
|
||||
];
|
||||
}
|
||||
|
||||
function makeOversizedResponse(): {
|
||||
response: Response;
|
||||
getReadCount: () => number;
|
||||
wasCanceled: () => boolean;
|
||||
} {
|
||||
const chunkSize = 1024 * 1024;
|
||||
const chunkCount = 20; // 20 MiB — over 16 MiB cap
|
||||
let readCount = 0;
|
||||
let canceled = false;
|
||||
return {
|
||||
response: new Response(
|
||||
new ReadableStream<Uint8Array>({
|
||||
pull(controller) {
|
||||
if (readCount >= chunkCount) {
|
||||
controller.close();
|
||||
return;
|
||||
}
|
||||
readCount += 1;
|
||||
controller.enqueue(new Uint8Array(chunkSize));
|
||||
},
|
||||
cancel() {
|
||||
canceled = true;
|
||||
},
|
||||
}),
|
||||
{ status: 200, headers: { "Content-Type": "application/json" } },
|
||||
),
|
||||
getReadCount: () => readCount,
|
||||
wasCanceled: () => canceled,
|
||||
};
|
||||
}
|
||||
|
||||
describe("Google embedding-batch bounded JSON reads", () => {
|
||||
it("bounds oversized file-upload JSON response and cancels the stream", async () => {
|
||||
const streamed = makeOversizedResponse();
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(async (input: RequestInfo | URL) => {
|
||||
if (fetchInputUrl(input).includes("/upload/")) {
|
||||
return streamed.response;
|
||||
}
|
||||
return new Response("unexpected", { status: 500 });
|
||||
}),
|
||||
);
|
||||
|
||||
await expect(
|
||||
runGeminiEmbeddingBatches({
|
||||
gemini: makeGeminiClient(),
|
||||
agentId: "main",
|
||||
requests: singleRequest(),
|
||||
wait: true,
|
||||
concurrency: 1,
|
||||
pollIntervalMs: 50,
|
||||
timeoutMs: 5_000,
|
||||
}),
|
||||
).rejects.toThrow(/gemini\.batch-file-upload/);
|
||||
|
||||
expect(streamed.wasCanceled()).toBe(true);
|
||||
expect(streamed.getReadCount()).toBeLessThan(20);
|
||||
});
|
||||
|
||||
it("bounds oversized batch-create JSON response and cancels the stream", async () => {
|
||||
const streamed = makeOversizedResponse();
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(async (input: RequestInfo | URL) => {
|
||||
const url = fetchInputUrl(input);
|
||||
if (url.includes("/upload/")) {
|
||||
return jsonResponse({ name: "files/f-ok" });
|
||||
}
|
||||
if (url.includes(":asyncBatchEmbedContent")) {
|
||||
return streamed.response;
|
||||
}
|
||||
return new Response("unexpected", { status: 500 });
|
||||
}),
|
||||
);
|
||||
|
||||
await expect(
|
||||
runGeminiEmbeddingBatches({
|
||||
gemini: makeGeminiClient(),
|
||||
agentId: "main",
|
||||
requests: singleRequest(),
|
||||
wait: true,
|
||||
concurrency: 1,
|
||||
pollIntervalMs: 50,
|
||||
timeoutMs: 5_000,
|
||||
}),
|
||||
).rejects.toThrow(/gemini\.batch-create/);
|
||||
|
||||
expect(streamed.wasCanceled()).toBe(true);
|
||||
expect(streamed.getReadCount()).toBeLessThan(20);
|
||||
});
|
||||
|
||||
it("bounds oversized batch-status poll JSON response and cancels the stream", async () => {
|
||||
const streamed = makeOversizedResponse();
|
||||
let statusCalled = false;
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(async (input: RequestInfo | URL) => {
|
||||
const url = fetchInputUrl(input);
|
||||
if (url.includes("/upload/")) {
|
||||
return jsonResponse({ name: "files/f-ok" });
|
||||
}
|
||||
if (url.includes(":asyncBatchEmbedContent")) {
|
||||
return jsonResponse({ name: "batches/b-0", state: "PENDING" });
|
||||
}
|
||||
if (url.includes("/batches/") && !statusCalled) {
|
||||
statusCalled = true;
|
||||
return streamed.response;
|
||||
}
|
||||
return new Response("unexpected", { status: 500 });
|
||||
}),
|
||||
);
|
||||
|
||||
await expect(
|
||||
runGeminiEmbeddingBatches({
|
||||
gemini: makeGeminiClient(),
|
||||
agentId: "main",
|
||||
requests: singleRequest(),
|
||||
wait: true,
|
||||
concurrency: 1,
|
||||
pollIntervalMs: 50,
|
||||
timeoutMs: 5_000,
|
||||
}),
|
||||
).rejects.toThrow(/gemini\.batch-status/);
|
||||
|
||||
expect(streamed.wasCanceled()).toBe(true);
|
||||
expect(streamed.getReadCount()).toBeLessThan(20);
|
||||
});
|
||||
|
||||
it("parses small responses on all three JSON paths correctly", async () => {
|
||||
// Use a unit-length vector so sanitizeAndNormalizeEmbedding preserves values.
|
||||
const outputLine = JSON.stringify({
|
||||
key: "r0",
|
||||
embedding: { values: [1, 0, 0] },
|
||||
});
|
||||
let statusCalled = false;
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(async (input: RequestInfo | URL) => {
|
||||
const url = fetchInputUrl(input);
|
||||
if (url.includes("/upload/")) {
|
||||
return jsonResponse({ name: "files/f-ok" });
|
||||
}
|
||||
if (url.includes(":asyncBatchEmbedContent")) {
|
||||
return jsonResponse({ name: "batches/b-0", state: "PENDING" });
|
||||
}
|
||||
if (url.includes("/batches/") && !statusCalled) {
|
||||
statusCalled = true;
|
||||
return jsonResponse({
|
||||
name: "batches/b-0",
|
||||
state: "SUCCEEDED",
|
||||
outputConfig: { file: "files/out-0" },
|
||||
});
|
||||
}
|
||||
if (url.includes(":download")) {
|
||||
return new Response(outputLine, { status: 200 });
|
||||
}
|
||||
return new Response("unexpected", { status: 500 });
|
||||
}),
|
||||
);
|
||||
|
||||
const result = await runGeminiEmbeddingBatches({
|
||||
gemini: makeGeminiClient(),
|
||||
agentId: "main",
|
||||
requests: singleRequest(),
|
||||
wait: true,
|
||||
concurrency: 1,
|
||||
pollIntervalMs: 50,
|
||||
timeoutMs: 5_000,
|
||||
});
|
||||
|
||||
expect(result.get("r0")).toEqual([1, 0, 0]);
|
||||
});
|
||||
});
|
||||
387
extensions/google/embedding-batch.ts
Normal file
387
extensions/google/embedding-batch.ts
Normal file
@@ -0,0 +1,387 @@
|
||||
// Google plugin module implements embedding batch behavior.
|
||||
import crypto from "node:crypto";
|
||||
import {
|
||||
buildEmbeddingBatchGroupOptions,
|
||||
runEmbeddingBatchGroups,
|
||||
buildBatchHeaders,
|
||||
debugEmbeddingsLog,
|
||||
normalizeBatchBaseUrl,
|
||||
sanitizeAndNormalizeEmbedding,
|
||||
withRemoteHttpResponse,
|
||||
} from "openclaw/plugin-sdk/memory-core-host-engine-embeddings";
|
||||
import {
|
||||
createProviderHttpError,
|
||||
readProviderJsonResponse,
|
||||
} from "openclaw/plugin-sdk/provider-http";
|
||||
import { normalizeStringEntries } from "openclaw/plugin-sdk/string-coerce-runtime";
|
||||
import type { GeminiEmbeddingClient, GeminiTextEmbeddingRequest } from "./embedding-provider.js";
|
||||
|
||||
type EmbeddingBatchExecutionParams = {
|
||||
wait: boolean;
|
||||
pollIntervalMs: number;
|
||||
timeoutMs: number;
|
||||
concurrency: number;
|
||||
debug?: (message: string, data?: Record<string, unknown>) => void;
|
||||
};
|
||||
|
||||
type GeminiBatchRequest = {
|
||||
custom_id: string;
|
||||
request: GeminiTextEmbeddingRequest;
|
||||
};
|
||||
|
||||
type GeminiBatchStatus = {
|
||||
name?: string;
|
||||
state?: string;
|
||||
outputConfig?: { file?: string; fileId?: string };
|
||||
metadata?: {
|
||||
output?: {
|
||||
responsesFile?: string;
|
||||
};
|
||||
};
|
||||
error?: { message?: string };
|
||||
};
|
||||
|
||||
type GeminiBatchOutputLine = {
|
||||
key?: string;
|
||||
custom_id?: string;
|
||||
request_id?: string;
|
||||
embedding?: { values?: number[] };
|
||||
response?: {
|
||||
embedding?: { values?: number[] };
|
||||
error?: { message?: string };
|
||||
};
|
||||
error?: { message?: string };
|
||||
};
|
||||
|
||||
const GEMINI_BATCH_MAX_REQUESTS = 50000;
|
||||
function hashText(text: string): string {
|
||||
return crypto.createHash("sha256").update(text).digest("hex");
|
||||
}
|
||||
|
||||
function getGeminiUploadUrl(baseUrl: string): string {
|
||||
if (baseUrl.includes("/v1beta")) {
|
||||
return baseUrl.replace(/\/v1beta\/?$/, "/upload/v1beta");
|
||||
}
|
||||
return `${baseUrl.replace(/\/$/, "")}/upload`;
|
||||
}
|
||||
|
||||
function buildGeminiUploadBody(params: { jsonl: string; displayName: string }): {
|
||||
body: Blob;
|
||||
contentType: string;
|
||||
} {
|
||||
const boundary = `openclaw-${hashText(params.displayName)}`;
|
||||
const jsonPart = JSON.stringify({
|
||||
file: {
|
||||
displayName: params.displayName,
|
||||
mimeType: "application/jsonl",
|
||||
},
|
||||
});
|
||||
const delimiter = `--${boundary}\r\n`;
|
||||
const closeDelimiter = `--${boundary}--\r\n`;
|
||||
const parts = [
|
||||
`${delimiter}Content-Type: application/json; charset=UTF-8\r\n\r\n${jsonPart}\r\n`,
|
||||
`${delimiter}Content-Type: application/jsonl; charset=UTF-8\r\n\r\n${params.jsonl}\r\n`,
|
||||
closeDelimiter,
|
||||
];
|
||||
const body = new Blob([parts.join("")], { type: "multipart/related" });
|
||||
return {
|
||||
body,
|
||||
contentType: `multipart/related; boundary=${boundary}`,
|
||||
};
|
||||
}
|
||||
|
||||
async function submitGeminiBatch(params: {
|
||||
gemini: GeminiEmbeddingClient;
|
||||
requests: GeminiBatchRequest[];
|
||||
agentId: string;
|
||||
}): Promise<GeminiBatchStatus> {
|
||||
const baseUrl = normalizeBatchBaseUrl(params.gemini);
|
||||
const jsonl = params.requests
|
||||
.map((request) =>
|
||||
JSON.stringify({
|
||||
key: request.custom_id,
|
||||
request: request.request,
|
||||
}),
|
||||
)
|
||||
.join("\n");
|
||||
const displayName = `memory-embeddings-${hashText(String(Date.now()))}`;
|
||||
const uploadPayload = buildGeminiUploadBody({ jsonl, displayName });
|
||||
|
||||
const uploadUrl = `${getGeminiUploadUrl(baseUrl)}/files?uploadType=multipart`;
|
||||
debugEmbeddingsLog("memory embeddings: gemini batch upload", {
|
||||
uploadUrl,
|
||||
baseUrl,
|
||||
requests: params.requests.length,
|
||||
});
|
||||
const filePayload = await withRemoteHttpResponse({
|
||||
url: uploadUrl,
|
||||
ssrfPolicy: params.gemini.ssrfPolicy,
|
||||
init: {
|
||||
method: "POST",
|
||||
headers: {
|
||||
...buildBatchHeaders(params.gemini, { json: false }),
|
||||
"Content-Type": uploadPayload.contentType,
|
||||
},
|
||||
body: uploadPayload.body,
|
||||
},
|
||||
onResponse: async (fileRes) => {
|
||||
if (!fileRes.ok) {
|
||||
const text = await fileRes.text();
|
||||
throw new Error(`gemini batch file upload failed: ${fileRes.status} ${text}`);
|
||||
}
|
||||
return readProviderJsonResponse<{ name?: string; file?: { name?: string } }>(
|
||||
fileRes,
|
||||
"gemini.batch-file-upload",
|
||||
);
|
||||
},
|
||||
});
|
||||
const fileId = filePayload.name ?? filePayload.file?.name;
|
||||
if (!fileId) {
|
||||
throw new Error("gemini batch file upload failed: missing file id");
|
||||
}
|
||||
|
||||
const batchBody = {
|
||||
batch: {
|
||||
displayName: `memory-embeddings-${params.agentId}`,
|
||||
inputConfig: {
|
||||
file_name: fileId,
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
const batchEndpoint = `${baseUrl}/${params.gemini.modelPath}:asyncBatchEmbedContent`;
|
||||
debugEmbeddingsLog("memory embeddings: gemini batch create", {
|
||||
batchEndpoint,
|
||||
fileId,
|
||||
});
|
||||
return await withRemoteHttpResponse({
|
||||
url: batchEndpoint,
|
||||
ssrfPolicy: params.gemini.ssrfPolicy,
|
||||
init: {
|
||||
method: "POST",
|
||||
headers: buildBatchHeaders(params.gemini, { json: true }),
|
||||
body: JSON.stringify(batchBody),
|
||||
},
|
||||
onResponse: async (batchRes) => {
|
||||
if (batchRes.ok) {
|
||||
return readProviderJsonResponse<GeminiBatchStatus>(batchRes, "gemini.batch-create");
|
||||
}
|
||||
const text = await batchRes.text();
|
||||
if (batchRes.status === 404) {
|
||||
throw new Error(
|
||||
"gemini batch create failed: 404 (asyncBatchEmbedContent not available for this model/baseUrl). Disable remote.batch.enabled or switch providers.",
|
||||
);
|
||||
}
|
||||
throw new Error(`gemini batch create failed: ${batchRes.status} ${text}`);
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
async function fetchGeminiBatchStatus(params: {
|
||||
gemini: GeminiEmbeddingClient;
|
||||
batchName: string;
|
||||
}): Promise<GeminiBatchStatus> {
|
||||
const baseUrl = normalizeBatchBaseUrl(params.gemini);
|
||||
const name = params.batchName.startsWith("batches/")
|
||||
? params.batchName
|
||||
: `batches/${params.batchName}`;
|
||||
const statusUrl = `${baseUrl}/${name}`;
|
||||
debugEmbeddingsLog("memory embeddings: gemini batch status", { statusUrl });
|
||||
return await withRemoteHttpResponse({
|
||||
url: statusUrl,
|
||||
ssrfPolicy: params.gemini.ssrfPolicy,
|
||||
init: {
|
||||
headers: buildBatchHeaders(params.gemini, { json: true }),
|
||||
},
|
||||
onResponse: async (res) => {
|
||||
if (!res.ok) {
|
||||
throw await createProviderHttpError(res, "gemini batch status failed");
|
||||
}
|
||||
return readProviderJsonResponse<GeminiBatchStatus>(res, "gemini.batch-status");
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
async function fetchGeminiFileContent(params: {
|
||||
gemini: GeminiEmbeddingClient;
|
||||
fileId: string;
|
||||
}): Promise<string> {
|
||||
const baseUrl = normalizeBatchBaseUrl(params.gemini);
|
||||
const file = params.fileId.startsWith("files/") ? params.fileId : `files/${params.fileId}`;
|
||||
const downloadUrl = `${baseUrl}/${file}:download`;
|
||||
debugEmbeddingsLog("memory embeddings: gemini batch download", { downloadUrl });
|
||||
return await withRemoteHttpResponse({
|
||||
url: downloadUrl,
|
||||
ssrfPolicy: params.gemini.ssrfPolicy,
|
||||
init: {
|
||||
headers: buildBatchHeaders(params.gemini, { json: true }),
|
||||
},
|
||||
onResponse: async (res) => {
|
||||
if (!res.ok) {
|
||||
throw await createProviderHttpError(res, "gemini batch file content failed");
|
||||
}
|
||||
return await res.text();
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function parseGeminiBatchOutput(text: string): GeminiBatchOutputLine[] {
|
||||
if (!text.trim()) {
|
||||
return [];
|
||||
}
|
||||
return normalizeStringEntries(text.split("\n")).map(
|
||||
(line) => JSON.parse(line) as GeminiBatchOutputLine,
|
||||
);
|
||||
}
|
||||
|
||||
async function waitForGeminiBatch(params: {
|
||||
gemini: GeminiEmbeddingClient;
|
||||
batchName: string;
|
||||
wait: boolean;
|
||||
pollIntervalMs: number;
|
||||
timeoutMs: number;
|
||||
debug?: (message: string, data?: Record<string, unknown>) => void;
|
||||
initial?: GeminiBatchStatus;
|
||||
}): Promise<{ outputFileId: string }> {
|
||||
const start = Date.now();
|
||||
let current: GeminiBatchStatus | undefined = params.initial;
|
||||
while (true) {
|
||||
const status =
|
||||
current ??
|
||||
(await fetchGeminiBatchStatus({
|
||||
gemini: params.gemini,
|
||||
batchName: params.batchName,
|
||||
}));
|
||||
const state = status.state ?? "UNKNOWN";
|
||||
if (["SUCCEEDED", "COMPLETED", "DONE"].includes(state)) {
|
||||
const outputFileId =
|
||||
status.outputConfig?.file ??
|
||||
status.outputConfig?.fileId ??
|
||||
status.metadata?.output?.responsesFile;
|
||||
if (!outputFileId) {
|
||||
throw new Error(`gemini batch ${params.batchName} completed without output file`);
|
||||
}
|
||||
return { outputFileId };
|
||||
}
|
||||
if (["FAILED", "CANCELLED", "CANCELED", "EXPIRED"].includes(state)) {
|
||||
const message = status.error?.message ?? "unknown error";
|
||||
throw new Error(`gemini batch ${params.batchName} ${state}: ${message}`);
|
||||
}
|
||||
if (!params.wait) {
|
||||
throw new Error(`gemini batch ${params.batchName} still ${state}; wait disabled`);
|
||||
}
|
||||
if (Date.now() - start > params.timeoutMs) {
|
||||
throw new Error(`gemini batch ${params.batchName} timed out after ${params.timeoutMs}ms`);
|
||||
}
|
||||
params.debug?.(`gemini batch ${params.batchName} ${state}; waiting ${params.pollIntervalMs}ms`);
|
||||
await new Promise((resolve) => {
|
||||
setTimeout(resolve, params.pollIntervalMs);
|
||||
});
|
||||
current = undefined;
|
||||
}
|
||||
}
|
||||
|
||||
export async function runGeminiEmbeddingBatches(
|
||||
params: {
|
||||
gemini: GeminiEmbeddingClient;
|
||||
agentId: string;
|
||||
requests: GeminiBatchRequest[];
|
||||
} & EmbeddingBatchExecutionParams,
|
||||
): Promise<Map<string, number[]>> {
|
||||
return await runEmbeddingBatchGroups({
|
||||
...buildEmbeddingBatchGroupOptions(params, {
|
||||
maxRequests: GEMINI_BATCH_MAX_REQUESTS,
|
||||
debugLabel: "memory embeddings: gemini batch submit",
|
||||
}),
|
||||
runGroup: async ({ group, groupIndex, groups, byCustomId, pollIntervalMs, timeoutMs }) => {
|
||||
const batchInfo = await submitGeminiBatch({
|
||||
gemini: params.gemini,
|
||||
requests: group,
|
||||
agentId: params.agentId,
|
||||
});
|
||||
const batchName = batchInfo.name ?? "";
|
||||
if (!batchName) {
|
||||
throw new Error("gemini batch create failed: missing batch name");
|
||||
}
|
||||
|
||||
params.debug?.("memory embeddings: gemini batch created", {
|
||||
batchName,
|
||||
state: batchInfo.state,
|
||||
group: groupIndex + 1,
|
||||
groups,
|
||||
requests: group.length,
|
||||
});
|
||||
|
||||
if (
|
||||
!params.wait &&
|
||||
batchInfo.state &&
|
||||
!["SUCCEEDED", "COMPLETED", "DONE"].includes(batchInfo.state)
|
||||
) {
|
||||
throw new Error(
|
||||
`gemini batch ${batchName} submitted; enable remote.batch.wait to await completion`,
|
||||
);
|
||||
}
|
||||
|
||||
const completed =
|
||||
batchInfo.state && ["SUCCEEDED", "COMPLETED", "DONE"].includes(batchInfo.state)
|
||||
? {
|
||||
outputFileId:
|
||||
batchInfo.outputConfig?.file ??
|
||||
batchInfo.outputConfig?.fileId ??
|
||||
batchInfo.metadata?.output?.responsesFile ??
|
||||
"",
|
||||
}
|
||||
: await waitForGeminiBatch({
|
||||
gemini: params.gemini,
|
||||
batchName,
|
||||
wait: params.wait,
|
||||
pollIntervalMs,
|
||||
timeoutMs,
|
||||
debug: params.debug,
|
||||
initial: batchInfo,
|
||||
});
|
||||
if (!completed.outputFileId) {
|
||||
throw new Error(`gemini batch ${batchName} completed without output file`);
|
||||
}
|
||||
|
||||
const content = await fetchGeminiFileContent({
|
||||
gemini: params.gemini,
|
||||
fileId: completed.outputFileId,
|
||||
});
|
||||
const outputLines = parseGeminiBatchOutput(content);
|
||||
const errors: string[] = [];
|
||||
const remaining = new Set(group.map((request) => request.custom_id));
|
||||
|
||||
for (const line of outputLines) {
|
||||
const customId = line.key ?? line.custom_id ?? line.request_id;
|
||||
if (!customId) {
|
||||
continue;
|
||||
}
|
||||
remaining.delete(customId);
|
||||
if (line.error?.message) {
|
||||
errors.push(`${customId}: ${line.error.message}`);
|
||||
continue;
|
||||
}
|
||||
if (line.response?.error?.message) {
|
||||
errors.push(`${customId}: ${line.response.error.message}`);
|
||||
continue;
|
||||
}
|
||||
const embedding = sanitizeAndNormalizeEmbedding(
|
||||
line.embedding?.values ?? line.response?.embedding?.values ?? [],
|
||||
);
|
||||
if (embedding.length === 0) {
|
||||
errors.push(`${customId}: empty embedding`);
|
||||
continue;
|
||||
}
|
||||
byCustomId.set(customId, embedding);
|
||||
}
|
||||
|
||||
if (errors.length > 0) {
|
||||
throw new Error(`gemini batch ${batchName} failed: ${errors.join("; ")}`);
|
||||
}
|
||||
if (remaining.size > 0) {
|
||||
throw new Error(`gemini batch ${batchName} missing ${remaining.size} embedding responses`);
|
||||
}
|
||||
},
|
||||
});
|
||||
}
|
||||
265
extensions/google/embedding-provider.test.ts
Normal file
265
extensions/google/embedding-provider.test.ts
Normal file
@@ -0,0 +1,265 @@
|
||||
// Google tests cover embedding provider plugin behavior.
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
vi.mock("openclaw/plugin-sdk/memory-core-host-engine-embeddings", async (importOriginal) => {
|
||||
const actual =
|
||||
await importOriginal<typeof import("openclaw/plugin-sdk/memory-core-host-engine-embeddings")>();
|
||||
return {
|
||||
...actual,
|
||||
withRemoteHttpResponse: (async <T>(params: {
|
||||
url: string;
|
||||
init?: RequestInit;
|
||||
onResponse: (response: Response) => Promise<T>;
|
||||
}): Promise<T> => {
|
||||
const response = await fetch(params.url, params.init);
|
||||
return await params.onResponse(response);
|
||||
}) satisfies typeof actual.withRemoteHttpResponse,
|
||||
};
|
||||
});
|
||||
|
||||
import {
|
||||
buildGeminiEmbeddingRequest,
|
||||
buildGeminiTextEmbeddingRequest,
|
||||
createGeminiEmbeddingProvider,
|
||||
DEFAULT_GEMINI_EMBEDDING_MODEL,
|
||||
GEMINI_EMBEDDING_2_MODELS,
|
||||
isGeminiEmbedding2Model,
|
||||
normalizeGeminiModel,
|
||||
resolveGeminiOutputDimensionality,
|
||||
} from "./embedding-provider.js";
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
vi.unstubAllGlobals();
|
||||
});
|
||||
|
||||
function installFetchMock(
|
||||
handler: (input: RequestInfo | URL, init?: RequestInit) => unknown,
|
||||
): ReturnType<typeof vi.fn> {
|
||||
const fetchMock = vi.fn(async (input: RequestInfo | URL, init?: RequestInit) => {
|
||||
return new Response(JSON.stringify(handler(input, init)), {
|
||||
status: 200,
|
||||
headers: { "Content-Type": "application/json" },
|
||||
});
|
||||
});
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
return fetchMock;
|
||||
}
|
||||
|
||||
function fetchJsonBody(fetchMock: ReturnType<typeof vi.fn>, index: number): unknown {
|
||||
const init = fetchMock.mock.calls[index]?.[1] as RequestInit | undefined;
|
||||
const body = init?.body;
|
||||
if (typeof body !== "string") {
|
||||
throw new Error("Expected JSON string request body.");
|
||||
}
|
||||
return JSON.parse(body) as unknown;
|
||||
}
|
||||
|
||||
function requireFirstFetchInput(fetchMock: ReturnType<typeof vi.fn>): RequestInfo | URL {
|
||||
const [call] = fetchMock.mock.calls;
|
||||
if (!call) {
|
||||
throw new Error("expected Gemini embedding fetch call");
|
||||
}
|
||||
return call[0] as RequestInfo | URL;
|
||||
}
|
||||
|
||||
describe("Gemini embedding request helpers", () => {
|
||||
it("builds requests and resolves model settings", () => {
|
||||
expect(
|
||||
buildGeminiTextEmbeddingRequest({
|
||||
text: "hello",
|
||||
taskType: "RETRIEVAL_DOCUMENT",
|
||||
modelPath: "models/gemini-embedding-2-preview",
|
||||
outputDimensionality: 1536,
|
||||
}),
|
||||
).toEqual({
|
||||
model: "models/gemini-embedding-2-preview",
|
||||
content: { parts: [{ text: "hello" }] },
|
||||
taskType: "RETRIEVAL_DOCUMENT",
|
||||
outputDimensionality: 1536,
|
||||
});
|
||||
expect(
|
||||
buildGeminiEmbeddingRequest({
|
||||
input: {
|
||||
text: "Image file: diagram.png",
|
||||
parts: [
|
||||
{ type: "text", text: "Image file: diagram.png" },
|
||||
{ type: "inline-data", mimeType: "image/png", data: "abc123" },
|
||||
],
|
||||
},
|
||||
taskType: "RETRIEVAL_DOCUMENT",
|
||||
modelPath: "models/gemini-embedding-2-preview",
|
||||
outputDimensionality: 1536,
|
||||
}),
|
||||
).toEqual({
|
||||
model: "models/gemini-embedding-2-preview",
|
||||
content: {
|
||||
parts: [
|
||||
{ text: "Image file: diagram.png" },
|
||||
{ inlineData: { mimeType: "image/png", data: "abc123" } },
|
||||
],
|
||||
},
|
||||
taskType: "RETRIEVAL_DOCUMENT",
|
||||
outputDimensionality: 1536,
|
||||
});
|
||||
expect(GEMINI_EMBEDDING_2_MODELS.has("gemini-embedding-2-preview")).toBe(true);
|
||||
expect(isGeminiEmbedding2Model("gemini-embedding-2-preview")).toBe(true);
|
||||
expect(isGeminiEmbedding2Model("gemini-embedding-001")).toBe(false);
|
||||
expect(isGeminiEmbedding2Model("text-embedding-004")).toBe(false);
|
||||
expect(resolveGeminiOutputDimensionality("gemini-embedding-001")).toBeUndefined();
|
||||
expect(resolveGeminiOutputDimensionality("text-embedding-004")).toBeUndefined();
|
||||
expect(resolveGeminiOutputDimensionality("gemini-embedding-2-preview")).toBe(3072);
|
||||
expect(resolveGeminiOutputDimensionality("gemini-embedding-2-preview", 768)).toBe(768);
|
||||
expect(resolveGeminiOutputDimensionality("gemini-embedding-2-preview", 1536)).toBe(1536);
|
||||
expect(resolveGeminiOutputDimensionality("gemini-embedding-2-preview", 3072)).toBe(3072);
|
||||
expect(() => resolveGeminiOutputDimensionality("gemini-embedding-2-preview", 512)).toThrow(
|
||||
/Invalid outputDimensionality 512/,
|
||||
);
|
||||
expect(() => resolveGeminiOutputDimensionality("gemini-embedding-2-preview", 1024)).toThrow(
|
||||
/Valid values: 768, 1536, 3072/,
|
||||
);
|
||||
expect(normalizeGeminiModel("models/gemini-embedding-2-preview")).toBe(
|
||||
"gemini-embedding-2-preview",
|
||||
);
|
||||
expect(normalizeGeminiModel("gemini/gemini-embedding-2-preview")).toBe(
|
||||
"gemini-embedding-2-preview",
|
||||
);
|
||||
expect(normalizeGeminiModel("google/gemini-embedding-2-preview")).toBe(
|
||||
"gemini-embedding-2-preview",
|
||||
);
|
||||
expect(normalizeGeminiModel("")).toBe(DEFAULT_GEMINI_EMBEDDING_MODEL);
|
||||
});
|
||||
});
|
||||
|
||||
describe("Gemini embedding provider", () => {
|
||||
it("handles legacy and v2 request/response behavior", async () => {
|
||||
const fetchMock = installFetchMock((input) => {
|
||||
const url = input instanceof URL ? input.href : typeof input === "string" ? input : input.url;
|
||||
return url.endsWith(":batchEmbedContents")
|
||||
? {
|
||||
embeddings: Array.from({ length: 2 }, () => ({
|
||||
values: [0, 0, 5],
|
||||
})),
|
||||
}
|
||||
: { embedding: { values: [3, 4, 0] } };
|
||||
});
|
||||
|
||||
const { provider } = await createGeminiEmbeddingProvider({
|
||||
config: {} as never,
|
||||
provider: "gemini",
|
||||
remote: { apiKey: "test-key" },
|
||||
model: "gemini-embedding-2-preview",
|
||||
outputDimensionality: 768,
|
||||
taskType: "SEMANTIC_SIMILARITY",
|
||||
fallback: "none",
|
||||
});
|
||||
|
||||
await expect(provider.embedQuery(" ")).resolves.toStrictEqual([]);
|
||||
await expect(provider.embedBatch([])).resolves.toStrictEqual([]);
|
||||
await expect(provider.embedQuery("test query")).resolves.toEqual([0.6, 0.8, 0]);
|
||||
|
||||
const structuredBatch = await provider.embedBatchInputs?.([
|
||||
{
|
||||
text: "Image file: diagram.png",
|
||||
parts: [
|
||||
{ type: "text", text: "Image file: diagram.png" },
|
||||
{ type: "inline-data", mimeType: "image/png", data: "img" },
|
||||
],
|
||||
},
|
||||
{
|
||||
text: "Audio file: note.wav",
|
||||
parts: [
|
||||
{ type: "text", text: "Audio file: note.wav" },
|
||||
{ type: "inline-data", mimeType: "audio/wav", data: "aud" },
|
||||
],
|
||||
},
|
||||
]);
|
||||
expect(structuredBatch).toEqual([
|
||||
[0, 0, 1],
|
||||
[0, 0, 1],
|
||||
]);
|
||||
|
||||
expect(requireFirstFetchInput(fetchMock)).toBe(
|
||||
"https://generativelanguage.googleapis.com/v1beta/models/gemini-embedding-2-preview:embedContent",
|
||||
);
|
||||
expect(fetchJsonBody(fetchMock, 0)).toEqual({
|
||||
outputDimensionality: 768,
|
||||
taskType: "SEMANTIC_SIMILARITY",
|
||||
content: { parts: [{ text: "test query" }] },
|
||||
});
|
||||
expect(fetchJsonBody(fetchMock, 1)).toEqual({
|
||||
requests: [
|
||||
{
|
||||
model: "models/gemini-embedding-2-preview",
|
||||
content: {
|
||||
parts: [
|
||||
{ text: "Image file: diagram.png" },
|
||||
{ inlineData: { mimeType: "image/png", data: "img" } },
|
||||
],
|
||||
},
|
||||
taskType: "SEMANTIC_SIMILARITY",
|
||||
outputDimensionality: 768,
|
||||
},
|
||||
{
|
||||
model: "models/gemini-embedding-2-preview",
|
||||
content: {
|
||||
parts: [
|
||||
{ text: "Audio file: note.wav" },
|
||||
{ inlineData: { mimeType: "audio/wav", data: "aud" } },
|
||||
],
|
||||
},
|
||||
taskType: "SEMANTIC_SIMILARITY",
|
||||
outputDimensionality: 768,
|
||||
},
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
it("rejects non-object successful embedding responses", async () => {
|
||||
installFetchMock(() => []);
|
||||
|
||||
const { provider } = await createGeminiEmbeddingProvider({
|
||||
config: {} as never,
|
||||
provider: "gemini",
|
||||
remote: { apiKey: "test-key" },
|
||||
model: "gemini-embedding-001",
|
||||
fallback: "none",
|
||||
});
|
||||
|
||||
await expect(provider.embedQuery("test query")).rejects.toThrow(
|
||||
"gemini embeddings failed: malformed JSON response",
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects wrong single embedding vector shapes", async () => {
|
||||
installFetchMock(() => ({ embedding: { values: [1, "bad"] } }));
|
||||
|
||||
const { provider } = await createGeminiEmbeddingProvider({
|
||||
config: {} as never,
|
||||
provider: "gemini",
|
||||
remote: { apiKey: "test-key" },
|
||||
model: "gemini-embedding-001",
|
||||
fallback: "none",
|
||||
});
|
||||
|
||||
await expect(provider.embedQuery("test query")).rejects.toThrow(
|
||||
"gemini embeddings failed: malformed JSON response",
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects batch embedding count mismatches", async () => {
|
||||
installFetchMock(() => ({ embeddings: [{ values: [1, 2] }] }));
|
||||
|
||||
const { provider } = await createGeminiEmbeddingProvider({
|
||||
config: {} as never,
|
||||
provider: "gemini",
|
||||
remote: { apiKey: "test-key" },
|
||||
model: "gemini-embedding-001",
|
||||
fallback: "none",
|
||||
});
|
||||
|
||||
await expect(provider.embedBatch(["one", "two"])).rejects.toThrow(
|
||||
"gemini embeddings failed: malformed JSON response",
|
||||
);
|
||||
});
|
||||
});
|
||||
439
extensions/google/embedding-provider.ts
Normal file
439
extensions/google/embedding-provider.ts
Normal file
@@ -0,0 +1,439 @@
|
||||
// Google provider module implements model/runtime integration.
|
||||
import {
|
||||
buildRemoteBaseUrlPolicy,
|
||||
debugEmbeddingsLog,
|
||||
sanitizeAndNormalizeEmbedding,
|
||||
withRemoteHttpResponse,
|
||||
type EmbeddingInput,
|
||||
type MemoryEmbeddingProvider,
|
||||
type MemoryEmbeddingProviderCreateOptions,
|
||||
} from "openclaw/plugin-sdk/memory-core-host-engine-embeddings";
|
||||
import { resolveMemorySecretInputString } from "openclaw/plugin-sdk/memory-core-host-secret";
|
||||
import {
|
||||
collectProviderApiKeysForExecution,
|
||||
executeWithApiKeyRotation,
|
||||
requireApiKey,
|
||||
resolveApiKeyForProvider,
|
||||
} from "openclaw/plugin-sdk/provider-auth-runtime";
|
||||
import {
|
||||
createProviderHttpError,
|
||||
providerOperationRetryConfig,
|
||||
readProviderJsonObjectResponse,
|
||||
} from "openclaw/plugin-sdk/provider-http";
|
||||
import type { SsrFPolicy } from "openclaw/plugin-sdk/ssrf-runtime";
|
||||
import {
|
||||
asOptionalRecord as asRecord,
|
||||
normalizeOptionalString,
|
||||
} from "openclaw/plugin-sdk/string-coerce-runtime";
|
||||
|
||||
export type GeminiEmbeddingClient = {
|
||||
baseUrl: string;
|
||||
headers: Record<string, string>;
|
||||
ssrfPolicy?: SsrFPolicy;
|
||||
model: string;
|
||||
modelPath: string;
|
||||
apiKeys: string[];
|
||||
outputDimensionality?: number;
|
||||
};
|
||||
|
||||
export const DEFAULT_GEMINI_EMBEDDING_MODEL = "gemini-embedding-001";
|
||||
const DEFAULT_GOOGLE_API_BASE_URL = "https://generativelanguage.googleapis.com/v1beta";
|
||||
const GEMINI_MAX_INPUT_TOKENS: Record<string, number> = {
|
||||
"text-embedding-004": 2048,
|
||||
"gemini-embedding-001": 2048,
|
||||
"gemini-embedding-2-preview": 8192,
|
||||
};
|
||||
|
||||
function parseGeminiAuth(apiKey: string): { headers: Record<string, string> } {
|
||||
if (apiKey.startsWith("{")) {
|
||||
try {
|
||||
const parsed = JSON.parse(apiKey) as { token?: string };
|
||||
if (typeof parsed.token === "string" && parsed.token) {
|
||||
return {
|
||||
headers: {
|
||||
Authorization: `Bearer ${parsed.token}`,
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
};
|
||||
}
|
||||
} catch {
|
||||
// Fall back to API-key auth below.
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
headers: {
|
||||
"x-goog-api-key": apiKey,
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
type GeminiTaskType = NonNullable<MemoryEmbeddingProviderCreateOptions["taskType"]>;
|
||||
|
||||
// --- gemini-embedding-2-preview support ---
|
||||
|
||||
export const GEMINI_EMBEDDING_2_MODELS = new Set([
|
||||
"gemini-embedding-2-preview",
|
||||
// Add the GA model name here once released.
|
||||
]);
|
||||
|
||||
const GEMINI_EMBEDDING_2_DEFAULT_DIMENSIONS = 3072;
|
||||
const GEMINI_EMBEDDING_2_VALID_DIMENSIONS = [768, 1536, 3072] as const;
|
||||
|
||||
type GeminiTextPart = { text: string };
|
||||
type GeminiInlinePart = {
|
||||
inlineData: { mimeType: string; data: string };
|
||||
};
|
||||
type GeminiPart = GeminiTextPart | GeminiInlinePart;
|
||||
type GeminiEmbeddingInputPart = NonNullable<EmbeddingInput["parts"]>[number];
|
||||
type GeminiEmbeddingRequest = {
|
||||
content: { parts: GeminiPart[] };
|
||||
taskType: GeminiTaskType;
|
||||
outputDimensionality?: number;
|
||||
model?: string;
|
||||
};
|
||||
export type GeminiTextEmbeddingRequest = GeminiEmbeddingRequest;
|
||||
|
||||
function malformedGeminiEmbeddingResponse(): Error {
|
||||
return new Error("gemini embeddings failed: malformed JSON response");
|
||||
}
|
||||
|
||||
function readGeminiEmbeddingValues(value: unknown): number[] {
|
||||
if (!Array.isArray(value)) {
|
||||
throw malformedGeminiEmbeddingResponse();
|
||||
}
|
||||
for (const entry of value) {
|
||||
if (typeof entry !== "number" || !Number.isFinite(entry)) {
|
||||
throw malformedGeminiEmbeddingResponse();
|
||||
}
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function readGeminiSingleEmbedding(payload: Record<string, unknown>): number[] {
|
||||
const embedding = asRecord(payload.embedding);
|
||||
if (!embedding) {
|
||||
throw malformedGeminiEmbeddingResponse();
|
||||
}
|
||||
return readGeminiEmbeddingValues(embedding.values);
|
||||
}
|
||||
|
||||
function readGeminiBatchEmbeddings(
|
||||
payload: Record<string, unknown>,
|
||||
expectedCount: number,
|
||||
): number[][] {
|
||||
if (!Array.isArray(payload.embeddings) || payload.embeddings.length !== expectedCount) {
|
||||
throw malformedGeminiEmbeddingResponse();
|
||||
}
|
||||
return payload.embeddings.map((entry) => {
|
||||
const embedding = asRecord(entry);
|
||||
if (!embedding) {
|
||||
throw malformedGeminiEmbeddingResponse();
|
||||
}
|
||||
return readGeminiEmbeddingValues(embedding.values);
|
||||
});
|
||||
}
|
||||
|
||||
/** Builds the text-only Gemini embedding request shape used across direct and batch APIs. */
|
||||
export function buildGeminiTextEmbeddingRequest(params: {
|
||||
text: string;
|
||||
taskType: GeminiTaskType;
|
||||
outputDimensionality?: number;
|
||||
modelPath?: string;
|
||||
}): GeminiTextEmbeddingRequest {
|
||||
return buildGeminiEmbeddingRequest({
|
||||
input: { text: params.text },
|
||||
taskType: params.taskType,
|
||||
outputDimensionality: params.outputDimensionality,
|
||||
modelPath: params.modelPath,
|
||||
});
|
||||
}
|
||||
|
||||
export function buildGeminiEmbeddingRequest(params: {
|
||||
input: EmbeddingInput;
|
||||
taskType: GeminiTaskType;
|
||||
outputDimensionality?: number;
|
||||
modelPath?: string;
|
||||
}): GeminiEmbeddingRequest {
|
||||
const request: GeminiEmbeddingRequest = {
|
||||
content: {
|
||||
parts: params.input.parts?.map((part: GeminiEmbeddingInputPart) =>
|
||||
part.type === "text"
|
||||
? ({ text: part.text } satisfies GeminiTextPart)
|
||||
: ({
|
||||
inlineData: { mimeType: part.mimeType, data: part.data },
|
||||
} satisfies GeminiInlinePart),
|
||||
) ?? [{ text: params.input.text }],
|
||||
},
|
||||
taskType: params.taskType,
|
||||
};
|
||||
if (params.modelPath) {
|
||||
request.model = params.modelPath;
|
||||
}
|
||||
if (params.outputDimensionality != null) {
|
||||
request.outputDimensionality = params.outputDimensionality;
|
||||
}
|
||||
return request;
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns true if the given model name is a gemini-embedding-2 variant that
|
||||
* supports `outputDimensionality` and extended task types.
|
||||
*/
|
||||
export function isGeminiEmbedding2Model(model: string): boolean {
|
||||
return GEMINI_EMBEDDING_2_MODELS.has(model);
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate and return the `outputDimensionality` for gemini-embedding-2 models.
|
||||
* Returns `undefined` for older models (they don't support the param).
|
||||
*/
|
||||
export function resolveGeminiOutputDimensionality(
|
||||
model: string,
|
||||
requested?: number,
|
||||
): number | undefined {
|
||||
if (!isGeminiEmbedding2Model(model)) {
|
||||
return undefined;
|
||||
}
|
||||
if (requested == null) {
|
||||
return GEMINI_EMBEDDING_2_DEFAULT_DIMENSIONS;
|
||||
}
|
||||
const valid: readonly number[] = GEMINI_EMBEDDING_2_VALID_DIMENSIONS;
|
||||
if (!valid.includes(requested)) {
|
||||
throw new Error(
|
||||
`Invalid outputDimensionality ${requested} for ${model}. Valid values: ${valid.join(", ")}`,
|
||||
);
|
||||
}
|
||||
return requested;
|
||||
}
|
||||
function resolveRemoteApiKey(remoteApiKey: unknown): string | undefined {
|
||||
const trimmed = resolveMemorySecretInputString({
|
||||
value: remoteApiKey,
|
||||
path: "agents.*.memorySearch.remote.apiKey",
|
||||
});
|
||||
if (!trimmed) {
|
||||
return undefined;
|
||||
}
|
||||
if (trimmed === "GOOGLE_API_KEY" || trimmed === "GEMINI_API_KEY") {
|
||||
return process.env[trimmed]?.trim();
|
||||
}
|
||||
return trimmed;
|
||||
}
|
||||
|
||||
export function normalizeGeminiModel(model: string): string {
|
||||
const trimmed = model.trim();
|
||||
if (!trimmed) {
|
||||
return DEFAULT_GEMINI_EMBEDDING_MODEL;
|
||||
}
|
||||
const withoutPrefix = trimmed.replace(/^models\//, "");
|
||||
if (withoutPrefix.startsWith("gemini/")) {
|
||||
return withoutPrefix.slice("gemini/".length);
|
||||
}
|
||||
if (withoutPrefix.startsWith("google/")) {
|
||||
return withoutPrefix.slice("google/".length);
|
||||
}
|
||||
return withoutPrefix;
|
||||
}
|
||||
|
||||
async function fetchGeminiEmbeddingPayload(params: {
|
||||
client: GeminiEmbeddingClient;
|
||||
endpoint: string;
|
||||
body: unknown;
|
||||
signal?: AbortSignal;
|
||||
}): Promise<Record<string, unknown>> {
|
||||
return await executeWithApiKeyRotation({
|
||||
provider: "google",
|
||||
apiKeys: params.client.apiKeys,
|
||||
transientRetry: providerOperationRetryConfig("read"),
|
||||
execute: async (apiKey) => {
|
||||
const authHeaders = parseGeminiAuth(apiKey);
|
||||
const headers = {
|
||||
...authHeaders.headers,
|
||||
...params.client.headers,
|
||||
};
|
||||
return await withRemoteHttpResponse({
|
||||
url: params.endpoint,
|
||||
ssrfPolicy: params.client.ssrfPolicy,
|
||||
signal: params.signal,
|
||||
init: {
|
||||
method: "POST",
|
||||
headers,
|
||||
body: JSON.stringify(params.body),
|
||||
},
|
||||
onResponse: async (res) => {
|
||||
if (!res.ok) {
|
||||
throw await createProviderHttpError(res, "gemini embeddings failed");
|
||||
}
|
||||
return await readProviderJsonObjectResponse(res, "gemini embeddings failed");
|
||||
},
|
||||
});
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function normalizeGeminiBaseUrl(raw: string): string {
|
||||
const trimmed = raw.replace(/\/+$/, "");
|
||||
const openAiIndex = trimmed.indexOf("/openai");
|
||||
if (openAiIndex > -1) {
|
||||
return normalizeGoogleApiBaseUrl(trimmed.slice(0, openAiIndex));
|
||||
}
|
||||
return normalizeGoogleApiBaseUrl(trimmed);
|
||||
}
|
||||
|
||||
function buildGeminiModelPath(model: string): string {
|
||||
return model.startsWith("models/") ? model : `models/${model}`;
|
||||
}
|
||||
|
||||
function normalizeGoogleApiBaseUrl(baseUrl: string): string {
|
||||
const trimmed = baseUrl.trim().replace(/\/+$/, "");
|
||||
if (!trimmed) {
|
||||
return DEFAULT_GOOGLE_API_BASE_URL;
|
||||
}
|
||||
try {
|
||||
const url = new URL(trimmed);
|
||||
url.hash = "";
|
||||
url.search = "";
|
||||
if (
|
||||
url.origin.toLowerCase() === "https://generativelanguage.googleapis.com" &&
|
||||
url.pathname.replace(/\/+$/, "") === ""
|
||||
) {
|
||||
url.pathname = "/v1beta";
|
||||
}
|
||||
return url.toString().replace(/\/+$/, "");
|
||||
} catch {
|
||||
return trimmed;
|
||||
}
|
||||
}
|
||||
|
||||
export async function createGeminiEmbeddingProvider(
|
||||
options: MemoryEmbeddingProviderCreateOptions,
|
||||
): Promise<{ provider: MemoryEmbeddingProvider; client: GeminiEmbeddingClient }> {
|
||||
const client = await resolveGeminiEmbeddingClient(options);
|
||||
const baseUrl = client.baseUrl.replace(/\/$/, "");
|
||||
const embedUrl = `${baseUrl}/${client.modelPath}:embedContent`;
|
||||
const batchUrl = `${baseUrl}/${client.modelPath}:batchEmbedContents`;
|
||||
const isV2 = isGeminiEmbedding2Model(client.model);
|
||||
const outputDimensionality = client.outputDimensionality;
|
||||
|
||||
const embedQuery = async (
|
||||
text: string,
|
||||
callOptions?: { signal?: AbortSignal },
|
||||
): Promise<number[]> => {
|
||||
if (!text.trim()) {
|
||||
return [];
|
||||
}
|
||||
const payload = await fetchGeminiEmbeddingPayload({
|
||||
client,
|
||||
endpoint: embedUrl,
|
||||
body: buildGeminiTextEmbeddingRequest({
|
||||
text,
|
||||
taskType: options.taskType ?? "RETRIEVAL_QUERY",
|
||||
outputDimensionality: isV2 ? outputDimensionality : undefined,
|
||||
}),
|
||||
signal: callOptions?.signal,
|
||||
});
|
||||
return sanitizeAndNormalizeEmbedding(readGeminiSingleEmbedding(payload));
|
||||
};
|
||||
|
||||
const embedBatchInputs = async (
|
||||
inputs: EmbeddingInput[],
|
||||
callOptions?: { signal?: AbortSignal },
|
||||
): Promise<number[][]> => {
|
||||
if (inputs.length === 0) {
|
||||
return [];
|
||||
}
|
||||
const payload = await fetchGeminiEmbeddingPayload({
|
||||
client,
|
||||
endpoint: batchUrl,
|
||||
body: {
|
||||
requests: inputs.map((input) =>
|
||||
buildGeminiEmbeddingRequest({
|
||||
input,
|
||||
modelPath: client.modelPath,
|
||||
taskType: options.taskType ?? "RETRIEVAL_DOCUMENT",
|
||||
outputDimensionality: isV2 ? outputDimensionality : undefined,
|
||||
}),
|
||||
),
|
||||
},
|
||||
signal: callOptions?.signal,
|
||||
});
|
||||
const embeddings = readGeminiBatchEmbeddings(payload, inputs.length);
|
||||
return embeddings.map((values) => sanitizeAndNormalizeEmbedding(values));
|
||||
};
|
||||
|
||||
const embedBatch = async (
|
||||
texts: string[],
|
||||
optionsLocal?: { signal?: AbortSignal },
|
||||
): Promise<number[][]> => {
|
||||
return await embedBatchInputs(
|
||||
texts.map((text) => ({
|
||||
text,
|
||||
})),
|
||||
optionsLocal,
|
||||
);
|
||||
};
|
||||
|
||||
return {
|
||||
provider: {
|
||||
id: "gemini",
|
||||
model: client.model,
|
||||
maxInputTokens: GEMINI_MAX_INPUT_TOKENS[client.model],
|
||||
embedQuery,
|
||||
embedBatch,
|
||||
embedBatchInputs,
|
||||
},
|
||||
client,
|
||||
};
|
||||
}
|
||||
|
||||
async function resolveGeminiEmbeddingClient(
|
||||
options: MemoryEmbeddingProviderCreateOptions,
|
||||
): Promise<GeminiEmbeddingClient> {
|
||||
const remote = options.remote;
|
||||
const remoteApiKey = resolveRemoteApiKey(remote?.apiKey);
|
||||
const remoteBaseUrl = remote?.baseUrl?.trim();
|
||||
|
||||
const apiKey = remoteApiKey
|
||||
? remoteApiKey
|
||||
: requireApiKey(
|
||||
await resolveApiKeyForProvider({
|
||||
provider: "google",
|
||||
cfg: options.config,
|
||||
agentDir: options.agentDir,
|
||||
}),
|
||||
"google",
|
||||
);
|
||||
|
||||
const providerConfig = options.config.models?.providers?.google;
|
||||
const rawBaseUrl =
|
||||
remoteBaseUrl ||
|
||||
normalizeOptionalString(providerConfig?.baseUrl) ||
|
||||
DEFAULT_GOOGLE_API_BASE_URL;
|
||||
const baseUrl = normalizeGeminiBaseUrl(rawBaseUrl);
|
||||
const ssrfPolicy = buildRemoteBaseUrlPolicy(baseUrl);
|
||||
const headerOverrides = Object.assign({}, providerConfig?.headers, remote?.headers);
|
||||
const headers: Record<string, string> = {
|
||||
...headerOverrides,
|
||||
};
|
||||
const apiKeys = collectProviderApiKeysForExecution({
|
||||
provider: "google",
|
||||
primaryApiKey: apiKey,
|
||||
});
|
||||
const model = normalizeGeminiModel(options.model);
|
||||
const modelPath = buildGeminiModelPath(model);
|
||||
const outputDimensionality = resolveGeminiOutputDimensionality(
|
||||
model,
|
||||
options.outputDimensionality,
|
||||
);
|
||||
debugEmbeddingsLog("memory embeddings: gemini client", {
|
||||
rawBaseUrl,
|
||||
baseUrl,
|
||||
model,
|
||||
modelPath,
|
||||
outputDimensionality,
|
||||
embedEndpoint: `${baseUrl}/${modelPath}:embedContent`,
|
||||
batchEndpoint: `${baseUrl}/${modelPath}:batchEmbedContents`,
|
||||
});
|
||||
return { baseUrl, headers, ssrfPolicy, model, modelPath, apiKeys, outputDimensionality };
|
||||
}
|
||||
21
extensions/google/gemini-auth.ts
Normal file
21
extensions/google/gemini-auth.ts
Normal file
@@ -0,0 +1,21 @@
|
||||
// Google plugin module implements gemini auth behavior.
|
||||
import { parseGoogleOauthApiKey } from "./oauth-token-shared.js";
|
||||
|
||||
export function parseGeminiAuth(apiKey: string): { headers: Record<string, string> } {
|
||||
const parsed = apiKey.startsWith("{") ? parseGoogleOauthApiKey(apiKey) : null;
|
||||
if (parsed?.token) {
|
||||
return {
|
||||
headers: {
|
||||
Authorization: `Bearer ${parsed.token}`,
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
headers: {
|
||||
"x-goog-api-key": apiKey,
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
};
|
||||
}
|
||||
9
extensions/google/gemini-cli-auth-home.ts
Normal file
9
extensions/google/gemini-cli-auth-home.ts
Normal file
@@ -0,0 +1,9 @@
|
||||
import crypto from "node:crypto";
|
||||
import path from "node:path";
|
||||
|
||||
export const GOOGLE_GEMINI_CLI_PROVIDER_ID = "google-gemini-cli";
|
||||
|
||||
export function resolveGeminiCliProfileHome(agentDir: string, profileId: string): string {
|
||||
const profileHash = crypto.createHash("sha256").update(profileId).digest("hex").slice(0, 24);
|
||||
return path.join(agentDir, `${GOOGLE_GEMINI_CLI_PROVIDER_ID}-home`, "profiles", profileHash);
|
||||
}
|
||||
149
extensions/google/gemini-cli-provider.ts
Normal file
149
extensions/google/gemini-cli-provider.ts
Normal file
@@ -0,0 +1,149 @@
|
||||
import { createLazyRuntimeModule } from "openclaw/plugin-sdk/lazy-runtime";
|
||||
// Google provider module implements model/runtime integration.
|
||||
import type {
|
||||
OpenClawPluginApi,
|
||||
ProviderAuthContext,
|
||||
ProviderFetchUsageSnapshotContext,
|
||||
} from "openclaw/plugin-sdk/plugin-entry";
|
||||
import { buildOauthProviderAuthResult } from "openclaw/plugin-sdk/provider-auth-result";
|
||||
import type { ProviderPlugin } from "openclaw/plugin-sdk/provider-model-shared";
|
||||
import { fetchGeminiUsage } from "openclaw/plugin-sdk/provider-usage";
|
||||
import { GOOGLE_GEMINI_CLI_PROVIDER_ID } from "./gemini-cli-auth-home.js";
|
||||
import { formatGoogleOauthApiKey, parseGoogleUsageToken } from "./oauth-token-shared.js";
|
||||
import { GOOGLE_GEMINI_PROVIDER_HOOKS } from "./provider-hooks.js";
|
||||
import { isModernGoogleModel, resolveGoogleGeminiForwardCompatModel } from "./provider-models.js";
|
||||
|
||||
const PROVIDER_ID = GOOGLE_GEMINI_CLI_PROVIDER_ID;
|
||||
const PROVIDER_LABEL = "Gemini CLI OAuth";
|
||||
const DEFAULT_MODEL = "google/gemini-3.1-pro-preview";
|
||||
const ENV_VARS = [
|
||||
"OPENCLAW_GEMINI_OAUTH_CLIENT_ID",
|
||||
"OPENCLAW_GEMINI_OAUTH_CLIENT_SECRET",
|
||||
"GEMINI_CLI_OAUTH_CLIENT_ID",
|
||||
"GEMINI_CLI_OAUTH_CLIENT_SECRET",
|
||||
] as const;
|
||||
|
||||
const loadOauthRuntimeModule = createLazyRuntimeModule(() => import("./oauth.runtime.js"));
|
||||
|
||||
async function fetchGeminiCliUsage(ctx: ProviderFetchUsageSnapshotContext) {
|
||||
return await fetchGeminiUsage(ctx.token, ctx.timeoutMs, ctx.fetchFn, PROVIDER_ID);
|
||||
}
|
||||
|
||||
export function buildGoogleGeminiCliProvider(): ProviderPlugin {
|
||||
return {
|
||||
id: PROVIDER_ID,
|
||||
label: PROVIDER_LABEL,
|
||||
docsPath: "/providers/models",
|
||||
aliases: ["gemini-cli"],
|
||||
envVars: [...ENV_VARS],
|
||||
auth: [
|
||||
{
|
||||
id: "oauth",
|
||||
label: "Google OAuth",
|
||||
hint: "PKCE + localhost callback",
|
||||
kind: "oauth",
|
||||
run: async (ctx: ProviderAuthContext) => {
|
||||
await ctx.prompter.note(
|
||||
[
|
||||
"This is an unofficial integration and is not endorsed by Google.",
|
||||
"Some users have reported account restrictions or suspensions after using third-party Gemini CLI and Antigravity OAuth clients.",
|
||||
"Proceed only if you understand and accept this risk.",
|
||||
].join("\n"),
|
||||
"Google Gemini CLI caution",
|
||||
);
|
||||
|
||||
const proceed = await ctx.prompter.confirm({
|
||||
message: "Continue with Google Gemini CLI OAuth?",
|
||||
initialValue: false,
|
||||
});
|
||||
if (!proceed) {
|
||||
await ctx.prompter.note("Skipped Google Gemini CLI OAuth setup.", "Setup skipped");
|
||||
return { profiles: [] };
|
||||
}
|
||||
|
||||
const spin = ctx.prompter.progress("Starting Gemini CLI OAuth…");
|
||||
try {
|
||||
const { loginGeminiCliOAuth } = await loadOauthRuntimeModule();
|
||||
const result = await loginGeminiCliOAuth({
|
||||
isRemote: ctx.isRemote,
|
||||
openUrl: ctx.openUrl,
|
||||
log: (msg) => ctx.runtime.log(msg),
|
||||
note: ctx.prompter.note,
|
||||
prompt: async (message) => ctx.prompter.text({ message }),
|
||||
progress: spin,
|
||||
});
|
||||
|
||||
spin.stop("Gemini CLI OAuth complete");
|
||||
return buildOauthProviderAuthResult({
|
||||
providerId: PROVIDER_ID,
|
||||
defaultModel: DEFAULT_MODEL,
|
||||
access: result.access,
|
||||
refresh: result.refresh,
|
||||
expires: result.expires,
|
||||
email: result.email,
|
||||
configPatch: {
|
||||
agents: {
|
||||
defaults: {
|
||||
models: {
|
||||
[DEFAULT_MODEL]: { agentRuntime: { id: PROVIDER_ID } },
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
...(result.projectId ? { credentialExtra: { projectId: result.projectId } } : {}),
|
||||
...(result.projectId
|
||||
? {
|
||||
notes: [
|
||||
"If requests fail, set GOOGLE_CLOUD_PROJECT or GOOGLE_CLOUD_PROJECT_ID.",
|
||||
],
|
||||
}
|
||||
: {}),
|
||||
});
|
||||
} catch (err) {
|
||||
spin.stop("Gemini CLI OAuth failed");
|
||||
await ctx.prompter.note(
|
||||
"Trouble with OAuth? Ensure your Google account has Gemini CLI access.",
|
||||
"OAuth help",
|
||||
);
|
||||
throw err;
|
||||
}
|
||||
},
|
||||
},
|
||||
],
|
||||
wizard: {
|
||||
setup: {
|
||||
choiceId: "google-gemini-cli",
|
||||
choiceLabel: "Gemini CLI OAuth",
|
||||
choiceHint: "Google OAuth with project-aware token payload",
|
||||
methodId: "oauth",
|
||||
},
|
||||
},
|
||||
resolveDynamicModel: (ctx) =>
|
||||
resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: PROVIDER_ID,
|
||||
ctx,
|
||||
}),
|
||||
...GOOGLE_GEMINI_PROVIDER_HOOKS,
|
||||
isModernModelRef: ({ modelId }) => isModernGoogleModel(modelId),
|
||||
formatApiKey: (cred) => formatGoogleOauthApiKey(cred),
|
||||
refreshOAuth: async (cred) => {
|
||||
const { refreshGeminiCliOAuthToken } = await loadOauthRuntimeModule();
|
||||
return await refreshGeminiCliOAuthToken(cred);
|
||||
},
|
||||
resolveUsageAuth: async (ctx) => {
|
||||
const auth = await ctx.resolveOAuthToken();
|
||||
if (!auth) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
...auth,
|
||||
token: parseGoogleUsageToken(auth.token),
|
||||
};
|
||||
},
|
||||
fetchUsageSnapshot: async (ctx) => await fetchGeminiCliUsage(ctx),
|
||||
};
|
||||
}
|
||||
|
||||
export function registerGoogleGeminiCliProvider(api: OpenClawPluginApi) {
|
||||
api.registerProvider(buildGoogleGeminiCliProvider());
|
||||
}
|
||||
122
extensions/google/generation-provider-metadata.ts
Normal file
122
extensions/google/generation-provider-metadata.ts
Normal file
@@ -0,0 +1,122 @@
|
||||
// Google provider module implements model/runtime integration.
|
||||
import type { MusicGenerationProvider } from "openclaw/plugin-sdk/music-generation";
|
||||
import { isProviderApiKeyConfigured } from "openclaw/plugin-sdk/provider-auth";
|
||||
import type {
|
||||
VideoGenerationProvider,
|
||||
VideoGenerationProviderConfiguredContext,
|
||||
} from "openclaw/plugin-sdk/video-generation";
|
||||
|
||||
export const DEFAULT_GOOGLE_MUSIC_MODEL = "lyria-3-clip-preview";
|
||||
export const GOOGLE_PRO_MUSIC_MODEL = "lyria-3-pro-preview";
|
||||
export const GOOGLE_MAX_INPUT_IMAGES = 10;
|
||||
|
||||
export const DEFAULT_GOOGLE_VIDEO_MODEL = "veo-3.1-fast-generate-preview";
|
||||
export const GOOGLE_VIDEO_ALLOWED_DURATION_SECONDS = [4, 6, 8] as const;
|
||||
export const GOOGLE_VIDEO_MIN_DURATION_SECONDS = GOOGLE_VIDEO_ALLOWED_DURATION_SECONDS[0];
|
||||
export const GOOGLE_VIDEO_MAX_DURATION_SECONDS =
|
||||
GOOGLE_VIDEO_ALLOWED_DURATION_SECONDS[GOOGLE_VIDEO_ALLOWED_DURATION_SECONDS.length - 1];
|
||||
|
||||
function isGoogleProviderConfigured(
|
||||
ctx: { agentDir?: string } | VideoGenerationProviderConfiguredContext,
|
||||
): boolean {
|
||||
return isProviderApiKeyConfigured({
|
||||
provider: "google",
|
||||
agentDir: ctx.agentDir,
|
||||
});
|
||||
}
|
||||
|
||||
export function createGoogleMusicGenerationProviderMetadata(): Omit<
|
||||
MusicGenerationProvider,
|
||||
"generateMusic"
|
||||
> {
|
||||
return {
|
||||
id: "google",
|
||||
label: "Google",
|
||||
defaultModel: DEFAULT_GOOGLE_MUSIC_MODEL,
|
||||
models: [DEFAULT_GOOGLE_MUSIC_MODEL, GOOGLE_PRO_MUSIC_MODEL],
|
||||
isConfigured: isGoogleProviderConfigured,
|
||||
capabilities: {
|
||||
generate: {
|
||||
maxTracks: 1,
|
||||
supportsLyrics: true,
|
||||
supportsInstrumental: true,
|
||||
supportsFormat: true,
|
||||
supportedFormatsByModel: {
|
||||
[DEFAULT_GOOGLE_MUSIC_MODEL]: ["mp3"],
|
||||
[GOOGLE_PRO_MUSIC_MODEL]: ["mp3", "wav"],
|
||||
},
|
||||
},
|
||||
edit: {
|
||||
enabled: true,
|
||||
maxTracks: 1,
|
||||
maxInputImages: GOOGLE_MAX_INPUT_IMAGES,
|
||||
supportsLyrics: true,
|
||||
supportsInstrumental: true,
|
||||
supportsFormat: true,
|
||||
supportedFormatsByModel: {
|
||||
[DEFAULT_GOOGLE_MUSIC_MODEL]: ["mp3"],
|
||||
[GOOGLE_PRO_MUSIC_MODEL]: ["mp3", "wav"],
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export function createGoogleVideoGenerationProviderMetadata(): Omit<
|
||||
VideoGenerationProvider,
|
||||
"generateVideo"
|
||||
> {
|
||||
return {
|
||||
id: "google",
|
||||
label: "Google",
|
||||
defaultModel: DEFAULT_GOOGLE_VIDEO_MODEL,
|
||||
models: [
|
||||
DEFAULT_GOOGLE_VIDEO_MODEL,
|
||||
"veo-3.1-generate-preview",
|
||||
"veo-3.1-lite-generate-preview",
|
||||
"veo-3.0-fast-generate-001",
|
||||
"veo-3.0-generate-001",
|
||||
"veo-2.0-generate-001",
|
||||
],
|
||||
isConfigured: isGoogleProviderConfigured,
|
||||
capabilities: {
|
||||
generate: {
|
||||
maxVideos: 1,
|
||||
maxDurationSeconds: GOOGLE_VIDEO_MAX_DURATION_SECONDS,
|
||||
supportedDurationSeconds: [...GOOGLE_VIDEO_ALLOWED_DURATION_SECONDS],
|
||||
aspectRatios: ["16:9", "9:16"],
|
||||
resolutions: ["720P", "1080P"],
|
||||
supportsAspectRatio: true,
|
||||
supportsResolution: true,
|
||||
supportsSize: true,
|
||||
supportsAudio: false,
|
||||
},
|
||||
imageToVideo: {
|
||||
enabled: true,
|
||||
maxVideos: 1,
|
||||
maxInputImages: 1,
|
||||
maxDurationSeconds: GOOGLE_VIDEO_MAX_DURATION_SECONDS,
|
||||
supportedDurationSeconds: [...GOOGLE_VIDEO_ALLOWED_DURATION_SECONDS],
|
||||
aspectRatios: ["16:9", "9:16"],
|
||||
resolutions: ["720P", "1080P"],
|
||||
supportsAspectRatio: true,
|
||||
supportsResolution: true,
|
||||
supportsSize: true,
|
||||
supportsAudio: false,
|
||||
},
|
||||
videoToVideo: {
|
||||
enabled: true,
|
||||
maxVideos: 1,
|
||||
maxInputVideos: 1,
|
||||
maxDurationSeconds: GOOGLE_VIDEO_MAX_DURATION_SECONDS,
|
||||
supportedDurationSeconds: [...GOOGLE_VIDEO_ALLOWED_DURATION_SECONDS],
|
||||
aspectRatios: ["16:9", "9:16"],
|
||||
resolutions: ["720P", "1080P"],
|
||||
supportsAspectRatio: true,
|
||||
supportsResolution: true,
|
||||
supportsSize: true,
|
||||
supportsAudio: false,
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
9
extensions/google/google-genai-runtime.ts
Normal file
9
extensions/google/google-genai-runtime.ts
Normal file
@@ -0,0 +1,9 @@
|
||||
// Google plugin module implements google genai runtime behavior.
|
||||
import { GoogleGenAI } from "@google/genai";
|
||||
|
||||
export type GoogleGenAIClient = InstanceType<typeof GoogleGenAI>;
|
||||
type GoogleGenAIOptions = ConstructorParameters<typeof GoogleGenAI>[0];
|
||||
|
||||
export function createGoogleGenAI(options: GoogleGenAIOptions): GoogleGenAIClient {
|
||||
return new GoogleGenAI(options);
|
||||
}
|
||||
213
extensions/google/google.live.test.ts
Normal file
213
extensions/google/google.live.test.ts
Normal file
@@ -0,0 +1,213 @@
|
||||
import { resolveFfmpegBin } from "openclaw/plugin-sdk/media-runtime";
|
||||
// Google tests cover google plugin behavior.
|
||||
import {
|
||||
registerProviderPlugin,
|
||||
requireRegisteredProvider,
|
||||
} from "openclaw/plugin-sdk/plugin-test-runtime";
|
||||
import { normalizeTranscriptForMatch } from "openclaw/plugin-sdk/provider-test-contracts";
|
||||
import { isLiveTestEnabled } from "openclaw/plugin-sdk/test-env";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import plugin from "./index.js";
|
||||
import { createGeminiWebSearchProvider } from "./src/gemini-web-search-provider.js";
|
||||
|
||||
const GOOGLE_API_KEY =
|
||||
process.env.GEMINI_API_KEY?.trim() ||
|
||||
process.env.GOOGLE_API_KEY?.trim() ||
|
||||
process.env.GEMINI_PROVIDER_API_KEY?.trim() ||
|
||||
"";
|
||||
const LIVE = isLiveTestEnabled() && GOOGLE_API_KEY.length > 0;
|
||||
const describeLive = LIVE ? describe : describe.skip;
|
||||
|
||||
async function withGoogleApiEnvUnset<T>(fn: () => Promise<T>): Promise<T> {
|
||||
const geminiApiKey = process.env.GEMINI_API_KEY;
|
||||
const googleApiKey = process.env.GOOGLE_API_KEY;
|
||||
delete process.env.GEMINI_API_KEY;
|
||||
delete process.env.GOOGLE_API_KEY;
|
||||
try {
|
||||
return await fn();
|
||||
} finally {
|
||||
if (geminiApiKey === undefined) {
|
||||
delete process.env.GEMINI_API_KEY;
|
||||
} else {
|
||||
process.env.GEMINI_API_KEY = geminiApiKey;
|
||||
}
|
||||
if (googleApiKey === undefined) {
|
||||
delete process.env.GOOGLE_API_KEY;
|
||||
} else {
|
||||
process.env.GOOGLE_API_KEY = googleApiKey;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function isTransientGeminiSearchError(error: unknown): boolean {
|
||||
if (!(error instanceof Error)) {
|
||||
return false;
|
||||
}
|
||||
if (error.name === "AbortError") {
|
||||
return true;
|
||||
}
|
||||
const message = error.message.toLowerCase();
|
||||
return message.includes("timeout") || message.includes("aborted");
|
||||
}
|
||||
|
||||
function hasTrustedFfmpegForLiveVoiceNote(): boolean {
|
||||
try {
|
||||
resolveFfmpegBin();
|
||||
return true;
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
if (message.includes("ffmpeg not found in trusted system directories")) {
|
||||
console.warn("[google:live] skip voice-note transcode: ffmpeg unavailable");
|
||||
return false;
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
const registerGooglePlugin = () =>
|
||||
registerProviderPlugin({
|
||||
plugin,
|
||||
id: "google",
|
||||
name: "Google Provider",
|
||||
});
|
||||
|
||||
describeLive("google plugin live", () => {
|
||||
it("synthesizes speech through the registered provider", async () => {
|
||||
const { speechProviders } = await registerGooglePlugin();
|
||||
const provider = requireRegisteredProvider(speechProviders, "google");
|
||||
|
||||
const audioFile = await provider.synthesize({
|
||||
text: "OpenClaw Google text to speech integration test OK.",
|
||||
cfg: { plugins: { enabled: true } } as never,
|
||||
providerConfig: { apiKey: GOOGLE_API_KEY },
|
||||
target: "audio-file",
|
||||
timeoutMs: 90_000,
|
||||
});
|
||||
|
||||
expect(audioFile.outputFormat).toBe("wav");
|
||||
expect(audioFile.fileExtension).toBe(".wav");
|
||||
expect(audioFile.audioBuffer.byteLength).toBeGreaterThan(512);
|
||||
}, 120_000);
|
||||
|
||||
it("transcodes speech to Opus for voice-note targets", async () => {
|
||||
if (!hasTrustedFfmpegForLiveVoiceNote()) {
|
||||
return;
|
||||
}
|
||||
|
||||
const { speechProviders } = await registerGooglePlugin();
|
||||
const provider = requireRegisteredProvider(speechProviders, "google");
|
||||
|
||||
const audioFile = await provider.synthesize({
|
||||
text: "OpenClaw Google voice note integration test OK.",
|
||||
cfg: { plugins: { enabled: true } } as never,
|
||||
providerConfig: { apiKey: GOOGLE_API_KEY },
|
||||
target: "voice-note",
|
||||
timeoutMs: 90_000,
|
||||
});
|
||||
|
||||
expect(audioFile.outputFormat).toBe("opus");
|
||||
expect(audioFile.fileExtension).toBe(".opus");
|
||||
expect(audioFile.voiceCompatible).toBe(true);
|
||||
expect(audioFile.audioBuffer.byteLength).toBeGreaterThan(128);
|
||||
}, 120_000);
|
||||
|
||||
it("transcribes synthesized speech through the media provider", async () => {
|
||||
const { mediaProviders, speechProviders } = await registerGooglePlugin();
|
||||
const speechProvider = requireRegisteredProvider(speechProviders, "google");
|
||||
const mediaProvider = requireRegisteredProvider(mediaProviders, "google");
|
||||
|
||||
const phrase = "Testing Google audio transcription with pineapple.";
|
||||
const audioFile = await speechProvider.synthesize({
|
||||
text: phrase,
|
||||
cfg: { plugins: { enabled: true } } as never,
|
||||
providerConfig: { apiKey: GOOGLE_API_KEY },
|
||||
target: "audio-file",
|
||||
timeoutMs: 90_000,
|
||||
});
|
||||
|
||||
const transcript = await mediaProvider.transcribeAudio?.({
|
||||
buffer: audioFile.audioBuffer,
|
||||
fileName: "google-live.wav",
|
||||
mime: "audio/wav",
|
||||
apiKey: GOOGLE_API_KEY,
|
||||
timeoutMs: 90_000,
|
||||
});
|
||||
|
||||
const normalized = normalizeTranscriptForMatch(transcript?.text ?? "");
|
||||
expect(normalized).toContain("google");
|
||||
expect(normalized).toContain("pineapple");
|
||||
}, 180_000);
|
||||
|
||||
it("runs Gemini web search through the registered provider tool", async () => {
|
||||
const provider = createGeminiWebSearchProvider();
|
||||
const tool = provider.createTool?.({
|
||||
config: {},
|
||||
searchConfig: { gemini: { apiKey: GOOGLE_API_KEY }, cacheTtlMinutes: 0, timeoutSeconds: 90 },
|
||||
} as never);
|
||||
|
||||
let result: { provider?: string; content?: unknown; citations?: unknown } | undefined;
|
||||
let lastError: unknown;
|
||||
for (let attempt = 0; attempt < 2; attempt += 1) {
|
||||
try {
|
||||
result = await tool?.execute({ query: "OpenClaw GitHub", count: 1 });
|
||||
lastError = undefined;
|
||||
break;
|
||||
} catch (error) {
|
||||
lastError = error;
|
||||
if (!isTransientGeminiSearchError(error) || attempt === 1) {
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (lastError) {
|
||||
throw toLintErrorObject(lastError, "Non-Error thrown");
|
||||
}
|
||||
|
||||
expect(result?.provider).toBe("gemini");
|
||||
expect(typeof result?.content).toBe("string");
|
||||
expect((result!.content as string).length).toBeGreaterThan(20);
|
||||
expect(Array.isArray(result?.citations)).toBe(true);
|
||||
}, 120_000);
|
||||
|
||||
it("runs Gemini web search through the Google model provider config fallback", async () => {
|
||||
await withGoogleApiEnvUnset(async () => {
|
||||
const provider = createGeminiWebSearchProvider();
|
||||
const tool = provider.createTool?.({
|
||||
config: {
|
||||
models: {
|
||||
providers: {
|
||||
google: {
|
||||
apiKey: GOOGLE_API_KEY,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
searchConfig: { provider: "gemini", cacheTtlMinutes: 0, timeoutSeconds: 90 },
|
||||
} as never);
|
||||
|
||||
const result = await tool?.execute({ query: "OpenClaw GitHub", count: 1 });
|
||||
|
||||
expect(process.env.GEMINI_API_KEY).toBeUndefined();
|
||||
expect(process.env.GOOGLE_API_KEY).toBeUndefined();
|
||||
expect(result?.provider).toBe("gemini");
|
||||
expect(typeof result?.content).toBe("string");
|
||||
expect((result!.content as string).length).toBeGreaterThan(20);
|
||||
expect(Array.isArray(result?.citations)).toBe(true);
|
||||
expect((result!.citations as unknown[]).length).toBeGreaterThan(0);
|
||||
});
|
||||
}, 120_000);
|
||||
});
|
||||
|
||||
function toLintErrorObject(value: unknown, fallbackMessage: string): Error {
|
||||
if (value instanceof Error) {
|
||||
return value;
|
||||
}
|
||||
if (typeof value === "string") {
|
||||
return new Error(value);
|
||||
}
|
||||
const error = new Error(fallbackMessage, { cause: value });
|
||||
if ((typeof value === "object" && value !== null) || typeof value === "function") {
|
||||
Object.assign(error, value);
|
||||
}
|
||||
return error;
|
||||
}
|
||||
572
extensions/google/image-generation-provider.test.ts
Normal file
572
extensions/google/image-generation-provider.test.ts
Normal file
@@ -0,0 +1,572 @@
|
||||
// Google tests cover image generation provider plugin behavior.
|
||||
import * as providerAuthRuntime from "openclaw/plugin-sdk/provider-auth-runtime";
|
||||
import * as providerHttp from "openclaw/plugin-sdk/provider-http";
|
||||
import { mockPinnedHostnameResolution } from "openclaw/plugin-sdk/test-env";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { buildGoogleImageGenerationProvider } from "./image-generation-provider.js";
|
||||
import { testing as geminiWebSearchTesting } from "./src/gemini-web-search-provider.js";
|
||||
|
||||
let ssrfMock: { mockRestore: () => void } | undefined;
|
||||
|
||||
function jsonResponse(payload: unknown): Response {
|
||||
return new Response(JSON.stringify(payload), {
|
||||
status: 200,
|
||||
headers: { "Content-Type": "application/json" },
|
||||
});
|
||||
}
|
||||
|
||||
function mockGoogleApiKeyAuth() {
|
||||
vi.spyOn(providerAuthRuntime, "resolveApiKeyForProvider").mockResolvedValue({
|
||||
apiKey: "google-test-key",
|
||||
source: "env",
|
||||
mode: "api-key",
|
||||
});
|
||||
}
|
||||
|
||||
function installGoogleFetchMock(params?: {
|
||||
data?: string;
|
||||
mimeType?: string;
|
||||
inlineDataKey?: "inlineData" | "inline_data";
|
||||
}) {
|
||||
const mimeType = params?.mimeType ?? "image/png";
|
||||
const data = params?.data ?? "png-data";
|
||||
const inlineDataKey = params?.inlineDataKey ?? "inlineData";
|
||||
const fetchMock = vi.fn().mockResolvedValue(
|
||||
jsonResponse({
|
||||
candidates: [
|
||||
{
|
||||
content: {
|
||||
parts: [
|
||||
{
|
||||
[inlineDataKey]: {
|
||||
[inlineDataKey === "inlineData" ? "mimeType" : "mime_type"]: mimeType,
|
||||
data: Buffer.from(data).toString("base64"),
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
return fetchMock;
|
||||
}
|
||||
|
||||
function fetchRequest(fetchMock: ReturnType<typeof vi.fn>): {
|
||||
body?: string;
|
||||
headers?: HeadersInit;
|
||||
method?: string;
|
||||
url: string;
|
||||
} {
|
||||
const [url, init] = fetchMock.mock.calls[0] as [string, RequestInit | undefined];
|
||||
expect(typeof url).toBe("string");
|
||||
if (!init) {
|
||||
throw new Error("Expected fetch init");
|
||||
}
|
||||
return {
|
||||
body: typeof init.body === "string" ? init.body : undefined,
|
||||
headers: init.headers,
|
||||
method: init.method,
|
||||
url,
|
||||
};
|
||||
}
|
||||
|
||||
function postJsonRequestOptions(spy: unknown): {
|
||||
allowPrivateNetwork?: boolean;
|
||||
pinDns?: boolean;
|
||||
ssrfPolicy?: { allowRfc2544BenchmarkRange?: boolean };
|
||||
} {
|
||||
const options = (spy as { mock?: { calls?: Array<[unknown]> } }).mock?.calls?.[0]?.[0];
|
||||
if (!options) {
|
||||
throw new Error("Expected postJsonRequest options");
|
||||
}
|
||||
return options as {
|
||||
allowPrivateNetwork?: boolean;
|
||||
pinDns?: boolean;
|
||||
ssrfPolicy?: { allowRfc2544BenchmarkRange?: boolean };
|
||||
};
|
||||
}
|
||||
|
||||
describe("Google image-generation provider", () => {
|
||||
beforeEach(() => {
|
||||
ssrfMock = mockPinnedHostnameResolution();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
ssrfMock?.mockRestore();
|
||||
ssrfMock = undefined;
|
||||
vi.restoreAllMocks();
|
||||
vi.unstubAllGlobals();
|
||||
});
|
||||
|
||||
it("generates image buffers from the Gemini generateContent API", async () => {
|
||||
vi.spyOn(providerAuthRuntime, "resolveApiKeyForProvider").mockResolvedValue({
|
||||
apiKey: "google-test-key",
|
||||
source: "env",
|
||||
mode: "api-key",
|
||||
});
|
||||
const fetchMock = vi.fn().mockResolvedValue(
|
||||
jsonResponse({
|
||||
candidates: [
|
||||
{
|
||||
content: {
|
||||
parts: [
|
||||
{ text: "generated" },
|
||||
{
|
||||
inlineData: {
|
||||
mimeType: "image/png",
|
||||
data: Buffer.from("png-data").toString("base64"),
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
|
||||
const provider = buildGoogleImageGenerationProvider();
|
||||
const result = await provider.generateImage({
|
||||
provider: "google",
|
||||
model: "gemini-3.1-flash-image-preview",
|
||||
prompt: "draw a cat",
|
||||
cfg: {},
|
||||
size: "1536x1024",
|
||||
});
|
||||
|
||||
const request = fetchRequest(fetchMock);
|
||||
expect(request.url).toBe(
|
||||
"https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-flash-image-preview:generateContent",
|
||||
);
|
||||
expect(request.method).toBe("POST");
|
||||
expect(JSON.parse(request.body ?? "")).toEqual({
|
||||
contents: [
|
||||
{
|
||||
role: "user",
|
||||
parts: [{ text: "draw a cat" }],
|
||||
},
|
||||
],
|
||||
generationConfig: {
|
||||
responseModalities: ["TEXT", "IMAGE"],
|
||||
imageConfig: {
|
||||
aspectRatio: "3:2",
|
||||
imageSize: "2K",
|
||||
},
|
||||
},
|
||||
});
|
||||
expect(result).toEqual({
|
||||
images: [
|
||||
{
|
||||
buffer: Buffer.from("png-data"),
|
||||
mimeType: "image/png",
|
||||
fileName: "image-1.png",
|
||||
},
|
||||
],
|
||||
model: "gemini-3.1-flash-image-preview",
|
||||
});
|
||||
});
|
||||
|
||||
it("passes request SSRF policy to the provider HTTP helper", async () => {
|
||||
mockGoogleApiKeyAuth();
|
||||
const postJsonRequest = vi.spyOn(providerHttp, "postJsonRequest").mockResolvedValue({
|
||||
response: new Response(
|
||||
JSON.stringify({
|
||||
candidates: [
|
||||
{
|
||||
content: {
|
||||
parts: [
|
||||
{
|
||||
inlineData: {
|
||||
mimeType: "image/png",
|
||||
data: Buffer.from("png-data").toString("base64"),
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
}),
|
||||
{ status: 200, headers: { "Content-Type": "application/json" } },
|
||||
),
|
||||
finalUrl:
|
||||
"https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-flash-image-preview:generateContent",
|
||||
release: async () => {},
|
||||
});
|
||||
|
||||
const provider = buildGoogleImageGenerationProvider();
|
||||
await provider.generateImage({
|
||||
provider: "google",
|
||||
model: "gemini-3.1-flash-image-preview",
|
||||
prompt: "draw a cat",
|
||||
cfg: {},
|
||||
ssrfPolicy: { allowRfc2544BenchmarkRange: true },
|
||||
});
|
||||
|
||||
expect(postJsonRequestOptions(postJsonRequest).ssrfPolicy).toEqual({
|
||||
allowRfc2544BenchmarkRange: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("wraps wrong-shape successful Gemini image responses", async () => {
|
||||
mockGoogleApiKeyAuth();
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn().mockResolvedValue(jsonResponse({ candidates: { content: { parts: [] } } })),
|
||||
);
|
||||
|
||||
const provider = buildGoogleImageGenerationProvider();
|
||||
await expect(
|
||||
provider.generateImage({
|
||||
provider: "google",
|
||||
model: "gemini-3.1-flash-image-preview",
|
||||
prompt: "draw a cat",
|
||||
cfg: {},
|
||||
}),
|
||||
).rejects.toThrow("Google image generation response malformed");
|
||||
});
|
||||
|
||||
it("rejects invalid inline image data in successful Gemini responses", async () => {
|
||||
mockGoogleApiKeyAuth();
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn().mockResolvedValue(
|
||||
jsonResponse({
|
||||
candidates: [
|
||||
{
|
||||
content: {
|
||||
parts: [{ inlineData: { mimeType: "image/png", data: "not-base64!" } }],
|
||||
},
|
||||
},
|
||||
],
|
||||
}),
|
||||
),
|
||||
);
|
||||
|
||||
const provider = buildGoogleImageGenerationProvider();
|
||||
await expect(
|
||||
provider.generateImage({
|
||||
provider: "google",
|
||||
model: "gemini-3.1-flash-image-preview",
|
||||
prompt: "draw a cat",
|
||||
cfg: {},
|
||||
}),
|
||||
).rejects.toThrow("Google image generation response malformed");
|
||||
});
|
||||
|
||||
it("accepts OAuth JSON auth and inline_data responses", async () => {
|
||||
vi.spyOn(providerAuthRuntime, "resolveApiKeyForProvider").mockResolvedValue({
|
||||
apiKey: JSON.stringify({ token: "oauth-token" }),
|
||||
source: "profile",
|
||||
mode: "token",
|
||||
});
|
||||
const fetchMock = vi.fn().mockResolvedValue(
|
||||
jsonResponse({
|
||||
candidates: [
|
||||
{
|
||||
content: {
|
||||
parts: [
|
||||
{
|
||||
inline_data: {
|
||||
mime_type: "image/jpeg",
|
||||
data: Buffer.from("jpg-data").toString("base64"),
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
|
||||
const provider = buildGoogleImageGenerationProvider();
|
||||
const result = await provider.generateImage({
|
||||
provider: "google",
|
||||
model: "gemini-3.1-flash-image-preview",
|
||||
prompt: "draw a dog",
|
||||
cfg: {},
|
||||
});
|
||||
|
||||
const request = fetchRequest(fetchMock);
|
||||
expect(request.url.length).toBeGreaterThan(0);
|
||||
expect(request.headers).toBeInstanceOf(Headers);
|
||||
expect(new Headers(request.headers).get("authorization")).toBe("Bearer oauth-token");
|
||||
expect(result).toEqual({
|
||||
images: [
|
||||
{
|
||||
buffer: Buffer.from("jpg-data"),
|
||||
mimeType: "image/jpeg",
|
||||
fileName: "image-1.jpg",
|
||||
},
|
||||
],
|
||||
model: "gemini-3.1-flash-image-preview",
|
||||
});
|
||||
});
|
||||
|
||||
it("accepts valid multi-image inline JSON responses above the generic provider JSON cap", async () => {
|
||||
mockGoogleApiKeyAuth();
|
||||
const imageBytes = Buffer.alloc(6 * 1024 * 1024, 1);
|
||||
const imagePayload = imageBytes.toString("base64");
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn().mockResolvedValue(
|
||||
jsonResponse({
|
||||
candidates: [
|
||||
{
|
||||
content: {
|
||||
parts: Array.from({ length: 3 }, () => ({
|
||||
inlineData: {
|
||||
mimeType: "image/png",
|
||||
data: imagePayload,
|
||||
},
|
||||
})),
|
||||
},
|
||||
},
|
||||
],
|
||||
}),
|
||||
),
|
||||
);
|
||||
|
||||
const provider = buildGoogleImageGenerationProvider();
|
||||
const result = await provider.generateImage({
|
||||
provider: "google",
|
||||
model: "gemini-3.1-flash-image-preview",
|
||||
prompt: "draw a cat",
|
||||
cfg: {},
|
||||
});
|
||||
|
||||
expect(result.images).toHaveLength(3);
|
||||
expect(result.images.map((image) => image.buffer.byteLength)).toEqual([
|
||||
imageBytes.byteLength,
|
||||
imageBytes.byteLength,
|
||||
imageBytes.byteLength,
|
||||
]);
|
||||
});
|
||||
|
||||
it("still rejects oversized Google image JSON responses", async () => {
|
||||
mockGoogleApiKeyAuth();
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn().mockResolvedValue(
|
||||
jsonResponse({
|
||||
candidates: [
|
||||
{
|
||||
content: {
|
||||
parts: [{ text: "x".repeat(35 * 1024 * 1024) }],
|
||||
},
|
||||
},
|
||||
],
|
||||
}),
|
||||
),
|
||||
);
|
||||
|
||||
const provider = buildGoogleImageGenerationProvider();
|
||||
await expect(
|
||||
provider.generateImage({
|
||||
provider: "google",
|
||||
model: "gemini-3.1-flash-image-preview",
|
||||
prompt: "draw a cat",
|
||||
cfg: {},
|
||||
}),
|
||||
).rejects.toThrow("google.image-generation: JSON response exceeds");
|
||||
});
|
||||
|
||||
it("sends reference images and explicit resolution for edit flows", async () => {
|
||||
mockGoogleApiKeyAuth();
|
||||
const fetchMock = installGoogleFetchMock();
|
||||
|
||||
const provider = buildGoogleImageGenerationProvider();
|
||||
await provider.generateImage({
|
||||
provider: "google",
|
||||
model: "gemini-3-pro-image-preview",
|
||||
prompt: "Change only the sky to a sunset.",
|
||||
cfg: {},
|
||||
resolution: "4K",
|
||||
inputImages: [
|
||||
{
|
||||
buffer: Buffer.from("reference-bytes"),
|
||||
mimeType: "image/png",
|
||||
fileName: "reference.png",
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
const request = fetchRequest(fetchMock);
|
||||
expect(request.url).toBe(
|
||||
"https://generativelanguage.googleapis.com/v1beta/models/gemini-3-pro-image-preview:generateContent",
|
||||
);
|
||||
expect(request.method).toBe("POST");
|
||||
expect(JSON.parse(request.body ?? "")).toEqual({
|
||||
contents: [
|
||||
{
|
||||
role: "user",
|
||||
parts: [
|
||||
{
|
||||
inlineData: {
|
||||
mimeType: "image/png",
|
||||
data: Buffer.from("reference-bytes").toString("base64"),
|
||||
},
|
||||
},
|
||||
{ text: "Change only the sky to a sunset." },
|
||||
],
|
||||
},
|
||||
],
|
||||
generationConfig: {
|
||||
responseModalities: ["TEXT", "IMAGE"],
|
||||
imageConfig: {
|
||||
imageSize: "4K",
|
||||
},
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("forwards explicit aspect ratio without forcing a default when size is omitted", async () => {
|
||||
mockGoogleApiKeyAuth();
|
||||
const fetchMock = installGoogleFetchMock();
|
||||
|
||||
const provider = buildGoogleImageGenerationProvider();
|
||||
await provider.generateImage({
|
||||
provider: "google",
|
||||
model: "gemini-3-pro-image-preview",
|
||||
prompt: "portrait photo",
|
||||
cfg: {},
|
||||
aspectRatio: "9:16",
|
||||
});
|
||||
|
||||
const request = fetchRequest(fetchMock);
|
||||
expect(request.url).toBe(
|
||||
"https://generativelanguage.googleapis.com/v1beta/models/gemini-3-pro-image-preview:generateContent",
|
||||
);
|
||||
expect(request.method).toBe("POST");
|
||||
expect(JSON.parse(request.body ?? "")).toEqual({
|
||||
contents: [
|
||||
{
|
||||
role: "user",
|
||||
parts: [{ text: "portrait photo" }],
|
||||
},
|
||||
],
|
||||
generationConfig: {
|
||||
responseModalities: ["TEXT", "IMAGE"],
|
||||
imageConfig: {
|
||||
aspectRatio: "9:16",
|
||||
},
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("disables DNS pinning for Google image generation requests", async () => {
|
||||
mockGoogleApiKeyAuth();
|
||||
installGoogleFetchMock();
|
||||
const postJsonRequestSpy = vi.spyOn(providerHttp, "postJsonRequest");
|
||||
|
||||
const provider = buildGoogleImageGenerationProvider();
|
||||
await provider.generateImage({
|
||||
provider: "google",
|
||||
model: "gemini-3.1-flash-image-preview",
|
||||
prompt: "draw a fox",
|
||||
cfg: {},
|
||||
});
|
||||
|
||||
expect(postJsonRequestOptions(postJsonRequestSpy).pinDns).toBe(false);
|
||||
});
|
||||
|
||||
it("honors configured private-network opt-in for Google image generation", async () => {
|
||||
mockGoogleApiKeyAuth();
|
||||
installGoogleFetchMock();
|
||||
const postJsonRequestSpy = vi.spyOn(providerHttp, "postJsonRequest");
|
||||
|
||||
const provider = buildGoogleImageGenerationProvider();
|
||||
await provider.generateImage({
|
||||
provider: "google",
|
||||
model: "gemini-3.1-flash-image-preview",
|
||||
prompt: "draw a fox",
|
||||
cfg: {
|
||||
models: {
|
||||
providers: {
|
||||
google: {
|
||||
baseUrl: "https://generativelanguage.googleapis.com/v1beta",
|
||||
request: { allowPrivateNetwork: true },
|
||||
models: [],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
expect(postJsonRequestOptions(postJsonRequestSpy).allowPrivateNetwork).toBe(true);
|
||||
});
|
||||
|
||||
it("normalizes a configured bare Google host to the v1beta API root", async () => {
|
||||
mockGoogleApiKeyAuth();
|
||||
const fetchMock = installGoogleFetchMock();
|
||||
|
||||
const provider = buildGoogleImageGenerationProvider();
|
||||
await provider.generateImage({
|
||||
provider: "google",
|
||||
model: "gemini-3-pro-image-preview",
|
||||
prompt: "draw a cat",
|
||||
cfg: {
|
||||
models: {
|
||||
providers: {
|
||||
google: {
|
||||
baseUrl: "https://generativelanguage.googleapis.com",
|
||||
models: [],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
const request = fetchRequest(fetchMock);
|
||||
expect(request.url).toBe(
|
||||
"https://generativelanguage.googleapis.com/v1beta/models/gemini-3-pro-image-preview:generateContent",
|
||||
);
|
||||
expect(typeof request.method).toBe("string");
|
||||
});
|
||||
|
||||
it("strips a configured /openai suffix before calling the native Gemini image API", async () => {
|
||||
mockGoogleApiKeyAuth();
|
||||
const fetchMock = installGoogleFetchMock();
|
||||
|
||||
const provider = buildGoogleImageGenerationProvider();
|
||||
await provider.generateImage({
|
||||
provider: "google",
|
||||
model: "gemini-3-pro-image-preview",
|
||||
prompt: "draw a fox",
|
||||
cfg: {
|
||||
models: {
|
||||
providers: {
|
||||
google: {
|
||||
baseUrl: "https://generativelanguage.googleapis.com/v1beta/openai",
|
||||
models: [],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
const request = fetchRequest(fetchMock);
|
||||
expect(request.url).toBe(
|
||||
"https://generativelanguage.googleapis.com/v1beta/models/gemini-3-pro-image-preview:generateContent",
|
||||
);
|
||||
expect(typeof request.method).toBe("string");
|
||||
});
|
||||
|
||||
it("prefers scoped configured Gemini API keys over environment fallbacks", () => {
|
||||
expect(
|
||||
geminiWebSearchTesting.resolveGeminiApiKey({
|
||||
apiKey: "gemini-secret",
|
||||
}),
|
||||
).toBe("gemini-secret");
|
||||
});
|
||||
|
||||
it("falls back to the default Gemini model when unset or blank", () => {
|
||||
expect(geminiWebSearchTesting.resolveGeminiModel()).toBe("gemini-2.5-flash");
|
||||
expect(geminiWebSearchTesting.resolveGeminiModel({ model: " " })).toBe("gemini-2.5-flash");
|
||||
expect(geminiWebSearchTesting.resolveGeminiModel({ model: "gemini-2.5-pro" })).toBe(
|
||||
"gemini-2.5-pro",
|
||||
);
|
||||
});
|
||||
});
|
||||
294
extensions/google/image-generation-provider.ts
Normal file
294
extensions/google/image-generation-provider.ts
Normal file
@@ -0,0 +1,294 @@
|
||||
// Google provider module implements model/runtime integration.
|
||||
import {
|
||||
generatedImageAssetFromBase64,
|
||||
resolveInlineImageJsonResponseMaxBytes,
|
||||
type GeneratedImageAsset,
|
||||
type ImageGenerationProvider,
|
||||
} from "openclaw/plugin-sdk/image-generation";
|
||||
import { MAX_IMAGE_BYTES } from "openclaw/plugin-sdk/media-runtime";
|
||||
import { parseStrictPositiveInteger } from "openclaw/plugin-sdk/number-runtime";
|
||||
import { isProviderApiKeyConfigured } from "openclaw/plugin-sdk/provider-auth";
|
||||
import { resolveApiKeyForProvider } from "openclaw/plugin-sdk/provider-auth-runtime";
|
||||
import {
|
||||
assertOkOrThrowHttpError,
|
||||
postJsonRequest,
|
||||
readProviderJsonResponse,
|
||||
sanitizeConfiguredModelProviderRequest,
|
||||
} from "openclaw/plugin-sdk/provider-http";
|
||||
import {
|
||||
isRecord,
|
||||
normalizeLowercaseStringOrEmpty,
|
||||
normalizeOptionalString,
|
||||
} from "openclaw/plugin-sdk/string-coerce-runtime";
|
||||
import { normalizeGoogleModelId, resolveGoogleGenerativeAiHttpRequestConfig } from "./api.js";
|
||||
|
||||
const DEFAULT_GOOGLE_IMAGE_MODEL = "gemini-3.1-flash-image-preview";
|
||||
const DEFAULT_IMAGE_TIMEOUT_MS = 180_000;
|
||||
const DEFAULT_OUTPUT_MIME = "image/png";
|
||||
const GOOGLE_MAX_IMAGE_RESULTS = 4;
|
||||
const MB = 1024 * 1024;
|
||||
const GOOGLE_SUPPORTED_SIZES = [
|
||||
"1024x1024",
|
||||
"1024x1536",
|
||||
"1536x1024",
|
||||
"1024x1792",
|
||||
"1792x1024",
|
||||
] as const;
|
||||
const GOOGLE_SUPPORTED_ASPECT_RATIOS = [
|
||||
"1:1",
|
||||
"2:3",
|
||||
"3:2",
|
||||
"3:4",
|
||||
"4:3",
|
||||
"4:5",
|
||||
"5:4",
|
||||
"9:16",
|
||||
"16:9",
|
||||
"21:9",
|
||||
] as const;
|
||||
|
||||
const GOOGLE_IMAGE_MALFORMED_RESPONSE = "Google image generation response malformed";
|
||||
|
||||
function normalizeGoogleImageModel(model: string | undefined): string {
|
||||
const trimmed = model?.trim();
|
||||
return normalizeGoogleModelId(trimmed || DEFAULT_GOOGLE_IMAGE_MODEL);
|
||||
}
|
||||
|
||||
function resolveGeneratedImageMaxBytes(req: {
|
||||
cfg: { agents?: { defaults?: { mediaMaxMb?: number } } };
|
||||
}): number {
|
||||
const configured = req.cfg.agents?.defaults?.mediaMaxMb;
|
||||
if (typeof configured === "number" && Number.isFinite(configured) && configured > 0) {
|
||||
return Math.floor(configured * MB);
|
||||
}
|
||||
return MAX_IMAGE_BYTES;
|
||||
}
|
||||
|
||||
function mapSizeToImageConfig(
|
||||
size: string | undefined,
|
||||
): { aspectRatio?: string; imageSize?: "2K" | "4K" } | undefined {
|
||||
const trimmed = size?.trim();
|
||||
if (!trimmed) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
const normalized = normalizeLowercaseStringOrEmpty(trimmed);
|
||||
const mapping = new Map<string, string>([
|
||||
["1024x1024", "1:1"],
|
||||
["1024x1536", "2:3"],
|
||||
["1536x1024", "3:2"],
|
||||
["1024x1792", "9:16"],
|
||||
["1792x1024", "16:9"],
|
||||
]);
|
||||
const aspectRatio = mapping.get(normalized);
|
||||
|
||||
const [widthRaw, heightRaw] = normalized.split("x");
|
||||
const width = parseStrictPositiveInteger(widthRaw);
|
||||
const height = parseStrictPositiveInteger(heightRaw);
|
||||
if (width === undefined || height === undefined) {
|
||||
return undefined;
|
||||
}
|
||||
const longestEdge = Math.max(width, height);
|
||||
const imageSize = longestEdge >= 3072 ? "4K" : longestEdge >= 1536 ? "2K" : undefined;
|
||||
|
||||
if (!aspectRatio && !imageSize) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
return {
|
||||
...(aspectRatio ? { aspectRatio } : {}),
|
||||
...(imageSize ? { imageSize } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
function googleResponseParts(payload: unknown): unknown[] {
|
||||
if (!isRecord(payload)) {
|
||||
throw new Error(GOOGLE_IMAGE_MALFORMED_RESPONSE);
|
||||
}
|
||||
const candidates = payload.candidates;
|
||||
if (candidates === undefined || candidates === null) {
|
||||
return [];
|
||||
}
|
||||
if (!Array.isArray(candidates)) {
|
||||
throw new Error(GOOGLE_IMAGE_MALFORMED_RESPONSE);
|
||||
}
|
||||
|
||||
const parts: unknown[] = [];
|
||||
for (const candidate of candidates) {
|
||||
if (!isRecord(candidate)) {
|
||||
throw new Error(GOOGLE_IMAGE_MALFORMED_RESPONSE);
|
||||
}
|
||||
const content = candidate.content;
|
||||
if (content === undefined || content === null) {
|
||||
continue;
|
||||
}
|
||||
if (!isRecord(content)) {
|
||||
throw new Error(GOOGLE_IMAGE_MALFORMED_RESPONSE);
|
||||
}
|
||||
const candidateParts = content.parts;
|
||||
if (candidateParts === undefined || candidateParts === null) {
|
||||
continue;
|
||||
}
|
||||
if (!Array.isArray(candidateParts)) {
|
||||
throw new Error(GOOGLE_IMAGE_MALFORMED_RESPONSE);
|
||||
}
|
||||
parts.push(...candidateParts);
|
||||
}
|
||||
return parts;
|
||||
}
|
||||
|
||||
function googleInlineDataFromPart(part: unknown): Record<string, unknown> | undefined {
|
||||
if (!isRecord(part)) {
|
||||
throw new Error(GOOGLE_IMAGE_MALFORMED_RESPONSE);
|
||||
}
|
||||
const inline = part.inlineData ?? part.inline_data;
|
||||
if (inline === undefined || inline === null) {
|
||||
return undefined;
|
||||
}
|
||||
if (!isRecord(inline)) {
|
||||
throw new Error(GOOGLE_IMAGE_MALFORMED_RESPONSE);
|
||||
}
|
||||
return inline;
|
||||
}
|
||||
|
||||
export function buildGoogleImageGenerationProvider(): ImageGenerationProvider {
|
||||
return {
|
||||
id: "google",
|
||||
label: "Google",
|
||||
defaultModel: DEFAULT_GOOGLE_IMAGE_MODEL,
|
||||
models: [DEFAULT_GOOGLE_IMAGE_MODEL, "gemini-3-pro-image-preview"],
|
||||
isConfigured: ({ agentDir }) =>
|
||||
isProviderApiKeyConfigured({
|
||||
provider: "google",
|
||||
agentDir,
|
||||
}),
|
||||
capabilities: {
|
||||
generate: {
|
||||
maxCount: GOOGLE_MAX_IMAGE_RESULTS,
|
||||
supportsSize: true,
|
||||
supportsAspectRatio: true,
|
||||
supportsResolution: true,
|
||||
},
|
||||
edit: {
|
||||
enabled: true,
|
||||
maxCount: GOOGLE_MAX_IMAGE_RESULTS,
|
||||
maxInputImages: 5,
|
||||
supportsSize: true,
|
||||
supportsAspectRatio: true,
|
||||
supportsResolution: true,
|
||||
},
|
||||
geometry: {
|
||||
sizes: [...GOOGLE_SUPPORTED_SIZES],
|
||||
aspectRatios: [...GOOGLE_SUPPORTED_ASPECT_RATIOS],
|
||||
resolutions: ["1K", "2K", "4K"],
|
||||
},
|
||||
},
|
||||
async generateImage(req) {
|
||||
const auth = await resolveApiKeyForProvider({
|
||||
provider: "google",
|
||||
cfg: req.cfg,
|
||||
agentDir: req.agentDir,
|
||||
store: req.authStore,
|
||||
});
|
||||
if (!auth.apiKey) {
|
||||
throw new Error("Google API key missing");
|
||||
}
|
||||
|
||||
const model = normalizeGoogleImageModel(req.model);
|
||||
const { baseUrl, allowPrivateNetwork, headers, dispatcherPolicy } =
|
||||
resolveGoogleGenerativeAiHttpRequestConfig({
|
||||
apiKey: auth.apiKey,
|
||||
baseUrl: req.cfg?.models?.providers?.google?.baseUrl,
|
||||
request: sanitizeConfiguredModelProviderRequest(
|
||||
req.cfg?.models?.providers?.google?.request,
|
||||
),
|
||||
capability: "image",
|
||||
transport: "http",
|
||||
});
|
||||
const imageConfig = mapSizeToImageConfig(req.size);
|
||||
const inputParts = (req.inputImages ?? []).map((image) => ({
|
||||
inlineData: {
|
||||
mimeType: image.mimeType,
|
||||
data: image.buffer.toString("base64"),
|
||||
},
|
||||
}));
|
||||
const resolvedImageConfig = {
|
||||
...imageConfig,
|
||||
...(req.aspectRatio?.trim() ? { aspectRatio: req.aspectRatio.trim() } : {}),
|
||||
...(req.resolution ? { imageSize: req.resolution } : {}),
|
||||
};
|
||||
|
||||
const { response: res, release } = await postJsonRequest({
|
||||
url: `${baseUrl}/models/${model}:generateContent`,
|
||||
headers,
|
||||
body: {
|
||||
contents: [
|
||||
{
|
||||
role: "user",
|
||||
parts: [...inputParts, { text: req.prompt }],
|
||||
},
|
||||
],
|
||||
generationConfig: {
|
||||
responseModalities: ["TEXT", "IMAGE"],
|
||||
...(Object.keys(resolvedImageConfig).length > 0
|
||||
? { imageConfig: resolvedImageConfig }
|
||||
: {}),
|
||||
},
|
||||
},
|
||||
timeoutMs: req.timeoutMs ?? DEFAULT_IMAGE_TIMEOUT_MS,
|
||||
fetchFn: fetch,
|
||||
pinDns: false,
|
||||
allowPrivateNetwork,
|
||||
ssrfPolicy: req.ssrfPolicy,
|
||||
dispatcherPolicy,
|
||||
});
|
||||
|
||||
try {
|
||||
await assertOkOrThrowHttpError(res, "Google image generation failed");
|
||||
|
||||
const payload = await readProviderJsonResponse(res, "google.image-generation", {
|
||||
maxBytes: resolveInlineImageJsonResponseMaxBytes(
|
||||
GOOGLE_MAX_IMAGE_RESULTS,
|
||||
resolveGeneratedImageMaxBytes(req),
|
||||
),
|
||||
});
|
||||
let imageIndex = 0;
|
||||
const images: GeneratedImageAsset[] = [];
|
||||
for (const part of googleResponseParts(payload)) {
|
||||
const inline = googleInlineDataFromPart(part);
|
||||
if (!inline) {
|
||||
continue;
|
||||
}
|
||||
const data = normalizeOptionalString(inline.data);
|
||||
if (!data) {
|
||||
throw new Error(GOOGLE_IMAGE_MALFORMED_RESPONSE);
|
||||
}
|
||||
const image = generatedImageAssetFromBase64({
|
||||
base64: data,
|
||||
index: imageIndex,
|
||||
mimeType:
|
||||
normalizeOptionalString(inline.mimeType) ??
|
||||
normalizeOptionalString(inline.mime_type) ??
|
||||
DEFAULT_OUTPUT_MIME,
|
||||
});
|
||||
if (!image) {
|
||||
throw new Error(GOOGLE_IMAGE_MALFORMED_RESPONSE);
|
||||
}
|
||||
imageIndex += 1;
|
||||
images.push(image);
|
||||
}
|
||||
|
||||
if (images.length === 0) {
|
||||
throw new Error("Google image generation response missing image data");
|
||||
}
|
||||
|
||||
return {
|
||||
images,
|
||||
model,
|
||||
};
|
||||
} finally {
|
||||
await release();
|
||||
}
|
||||
},
|
||||
};
|
||||
}
|
||||
447
extensions/google/index.test.ts
Normal file
447
extensions/google/index.test.ts
Normal file
@@ -0,0 +1,447 @@
|
||||
// Google tests cover index plugin behavior.
|
||||
import { mkdtemp, writeFile } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import type { Context, Model } from "openclaw/plugin-sdk/llm";
|
||||
import type {
|
||||
ProviderReplaySessionEntry,
|
||||
ProviderSanitizeReplayHistoryContext,
|
||||
} from "openclaw/plugin-sdk/plugin-entry";
|
||||
import { createTestPluginApi } from "openclaw/plugin-sdk/plugin-test-api";
|
||||
import {
|
||||
registerProviderPlugin,
|
||||
requireRegisteredProvider,
|
||||
} from "openclaw/plugin-sdk/plugin-test-runtime";
|
||||
import { createCapturedThinkingConfigStream } from "openclaw/plugin-sdk/provider-test-contracts";
|
||||
import type { RealtimeVoiceProviderPlugin } from "openclaw/plugin-sdk/realtime-voice";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { registerGoogleGeminiCliProvider } from "./gemini-cli-provider.js";
|
||||
import googlePlugin from "./index.js";
|
||||
import googleProviderDiscovery from "./provider-discovery.js";
|
||||
import { registerGoogleProvider } from "./provider-registration.js";
|
||||
|
||||
const googleProviderPlugin = {
|
||||
register(api: Parameters<typeof registerGoogleProvider>[0]) {
|
||||
registerGoogleProvider(api);
|
||||
registerGoogleGeminiCliProvider(api);
|
||||
},
|
||||
};
|
||||
|
||||
const refreshGeminiCliOAuthTokenMock = vi.hoisted(() => vi.fn());
|
||||
|
||||
vi.mock("./oauth.runtime.js", () => ({
|
||||
refreshGeminiCliOAuthToken: refreshGeminiCliOAuthTokenMock,
|
||||
}));
|
||||
|
||||
describe("google provider plugin hooks", () => {
|
||||
it("owns replay policy and reasoning mode for the direct Gemini provider", async () => {
|
||||
const { providers } = await registerProviderPlugin({
|
||||
plugin: googleProviderPlugin,
|
||||
id: "google",
|
||||
name: "Google Provider",
|
||||
});
|
||||
const provider = requireRegisteredProvider(providers, "google");
|
||||
const customEntries: ProviderReplaySessionEntry[] = [];
|
||||
|
||||
expect(
|
||||
provider.buildReplayPolicy?.({
|
||||
provider: "google",
|
||||
modelApi: "google-generative-ai",
|
||||
modelId: "gemini-3.1-pro-preview",
|
||||
} as never),
|
||||
).toEqual({
|
||||
sanitizeMode: "full",
|
||||
sanitizeToolCallIds: true,
|
||||
toolCallIdMode: "strict",
|
||||
sanitizeThoughtSignatures: {
|
||||
allowBase64Only: true,
|
||||
includeCamelCase: true,
|
||||
},
|
||||
repairToolUseResultPairing: true,
|
||||
applyAssistantFirstOrderingFix: true,
|
||||
validateGeminiTurns: true,
|
||||
validateAnthropicTurns: false,
|
||||
allowSyntheticToolResults: true,
|
||||
});
|
||||
|
||||
expect(
|
||||
provider.resolveReasoningOutputMode?.({
|
||||
provider: "google",
|
||||
modelApi: "google-generative-ai",
|
||||
modelId: "gemini-3.1-pro-preview",
|
||||
} as never),
|
||||
).toBe("native");
|
||||
expect(
|
||||
provider.resolveReasoningOutputMode?.({
|
||||
provider: "google",
|
||||
modelId: "gemini-3.1-pro-preview",
|
||||
} as never),
|
||||
).toBe("native");
|
||||
|
||||
const sanitized = await Promise.resolve(
|
||||
provider.sanitizeReplayHistory?.({
|
||||
provider: "google",
|
||||
modelApi: "google-generative-ai",
|
||||
modelId: "gemini-3.1-pro-preview",
|
||||
sessionId: "session-1",
|
||||
messages: [
|
||||
{
|
||||
role: "assistant",
|
||||
content: [{ type: "text", text: "hello" }],
|
||||
},
|
||||
],
|
||||
sessionState: {
|
||||
getCustomEntries: () => customEntries,
|
||||
appendCustomEntry: (customType: string, data: unknown) => {
|
||||
customEntries.push({ customType, data });
|
||||
},
|
||||
},
|
||||
} as ProviderSanitizeReplayHistoryContext),
|
||||
);
|
||||
|
||||
const bootstrapMessage = sanitized?.[0] as
|
||||
| { role?: string; content?: unknown; timestamp?: unknown }
|
||||
| undefined;
|
||||
expect(bootstrapMessage?.role).toBe("user");
|
||||
expect(bootstrapMessage?.content).toBe("(session bootstrap)");
|
||||
expect(typeof bootstrapMessage?.timestamp).toBe("number");
|
||||
expect(sanitized?.[1]).toEqual({
|
||||
role: "assistant",
|
||||
content: [{ type: "text", text: "hello" }],
|
||||
});
|
||||
expect(customEntries).toHaveLength(1);
|
||||
expect(customEntries[0]?.customType).toBe("google-turn-ordering-bootstrap");
|
||||
});
|
||||
|
||||
it("keeps google-gemini-cli on tagged reasoning mode", async () => {
|
||||
const { providers } = await registerProviderPlugin({
|
||||
plugin: googleProviderPlugin,
|
||||
id: "google",
|
||||
name: "Google Provider",
|
||||
});
|
||||
const cliProvider = requireRegisteredProvider(providers, "google-gemini-cli");
|
||||
expect(
|
||||
cliProvider.resolveReasoningOutputMode?.({
|
||||
provider: "google-gemini-cli",
|
||||
modelApi: "google-gemini-cli",
|
||||
modelId: "gemini-2.5-pro",
|
||||
} as never),
|
||||
).toBe("tagged");
|
||||
});
|
||||
|
||||
it("keeps google-antigravity hook aliases on tagged reasoning mode", async () => {
|
||||
const { providers } = await registerProviderPlugin({
|
||||
plugin: googleProviderPlugin,
|
||||
id: "google",
|
||||
name: "Google Provider",
|
||||
});
|
||||
const provider = requireRegisteredProvider(providers, "google-antigravity");
|
||||
expect(
|
||||
provider.resolveReasoningOutputMode?.({
|
||||
provider: "google-antigravity",
|
||||
modelApi: "openai-completions",
|
||||
modelId: "gemini-3-pro-low",
|
||||
} as never),
|
||||
).toBe("tagged");
|
||||
});
|
||||
|
||||
it("keeps google-vertex hook aliases on native reasoning mode", async () => {
|
||||
const { providers } = await registerProviderPlugin({
|
||||
plugin: googleProviderPlugin,
|
||||
id: "google",
|
||||
name: "Google Provider",
|
||||
});
|
||||
const provider = requireRegisteredProvider(providers, "google-vertex");
|
||||
expect(
|
||||
provider.resolveReasoningOutputMode?.({
|
||||
provider: "google-vertex",
|
||||
modelApi: "google-vertex",
|
||||
modelId: "gemini-3.1-pro-preview",
|
||||
} as never),
|
||||
).toBe("native");
|
||||
expect(
|
||||
provider.resolveReasoningOutputMode?.({
|
||||
provider: "google-vertex",
|
||||
modelId: "gemini-3.1-pro-preview",
|
||||
} as never),
|
||||
).toBe("native");
|
||||
});
|
||||
|
||||
it("resolves Google Vertex ADC auth evidence to the config marker", async () => {
|
||||
const tempDir = await mkdtemp(path.join(os.tmpdir(), "openclaw-google-vertex-config-key-"));
|
||||
const credentialsPath = path.join(tempDir, "application_default_credentials.json");
|
||||
await writeFile(
|
||||
credentialsPath,
|
||||
JSON.stringify({
|
||||
type: "authorized_user",
|
||||
client_id: "client-id",
|
||||
client_secret: "client-secret",
|
||||
refresh_token: "refresh-token",
|
||||
}),
|
||||
"utf8",
|
||||
);
|
||||
const { providers } = await registerProviderPlugin({
|
||||
plugin: googleProviderPlugin,
|
||||
id: "google",
|
||||
name: "Google Provider",
|
||||
});
|
||||
const provider = requireRegisteredProvider(providers, "google-vertex");
|
||||
|
||||
expect(
|
||||
provider.resolveConfigApiKey?.({
|
||||
provider: "google-vertex",
|
||||
env: {
|
||||
GOOGLE_APPLICATION_CREDENTIALS: credentialsPath,
|
||||
GOOGLE_CLOUD_PROJECT: "vertex-project",
|
||||
GOOGLE_CLOUD_LOCATION: "global",
|
||||
},
|
||||
}),
|
||||
).toBe("gcp-vertex-credentials");
|
||||
expect(
|
||||
provider.resolveConfigApiKey?.({
|
||||
provider: "google-vertex",
|
||||
env: {
|
||||
GOOGLE_APPLICATION_CREDENTIALS: credentialsPath,
|
||||
GOOGLE_CLOUD_PROJECT: "",
|
||||
GCLOUD_PROJECT: "vertex-project",
|
||||
GOOGLE_CLOUD_LOCATION: "global",
|
||||
},
|
||||
}),
|
||||
).toBe("gcp-vertex-credentials");
|
||||
expect(
|
||||
googleProviderDiscovery.resolveConfigApiKey?.({
|
||||
provider: "google-vertex",
|
||||
env: {
|
||||
GOOGLE_APPLICATION_CREDENTIALS: credentialsPath,
|
||||
GOOGLE_CLOUD_PROJECT: "vertex-project",
|
||||
GOOGLE_CLOUD_LOCATION: "global",
|
||||
},
|
||||
}),
|
||||
).toBe("gcp-vertex-credentials");
|
||||
});
|
||||
|
||||
it("owns Gemini tool schema normalization for direct and CLI providers", async () => {
|
||||
const { providers } = await registerProviderPlugin({
|
||||
plugin: googleProviderPlugin,
|
||||
id: "google",
|
||||
name: "Google Provider",
|
||||
});
|
||||
const providerIds = ["google", "google-gemini-cli"] as const;
|
||||
|
||||
for (const providerId of providerIds) {
|
||||
const provider = requireRegisteredProvider(providers, providerId);
|
||||
const [tool] =
|
||||
provider.normalizeToolSchemas?.({
|
||||
provider: providerId,
|
||||
tools: [
|
||||
{
|
||||
name: "write_file",
|
||||
description: "Write a file",
|
||||
parameters: {
|
||||
type: "object",
|
||||
additionalProperties: false,
|
||||
properties: {
|
||||
path: { type: "string", pattern: "^src/" },
|
||||
},
|
||||
},
|
||||
},
|
||||
],
|
||||
} as never) ?? [];
|
||||
|
||||
expect(tool).toEqual({
|
||||
name: "write_file",
|
||||
description: "Write a file",
|
||||
parameters: {
|
||||
type: "object",
|
||||
properties: {
|
||||
path: { type: "string" },
|
||||
},
|
||||
},
|
||||
});
|
||||
expect(tool?.parameters).not.toHaveProperty("additionalProperties");
|
||||
expect(
|
||||
(tool?.parameters as { properties?: { path?: Record<string, unknown> } })?.properties?.path,
|
||||
).not.toHaveProperty("pattern");
|
||||
expect(
|
||||
provider.inspectToolSchemas?.({
|
||||
provider: providerId,
|
||||
tools: [tool],
|
||||
} as never),
|
||||
).toEqual([]);
|
||||
}
|
||||
});
|
||||
|
||||
it("wires google-thinking stream hooks for direct and Gemini CLI providers", async () => {
|
||||
const { providers } = await registerProviderPlugin({
|
||||
plugin: googleProviderPlugin,
|
||||
id: "google",
|
||||
name: "Google Provider",
|
||||
});
|
||||
const googleProvider = requireRegisteredProvider(providers, "google");
|
||||
const cliProvider = requireRegisteredProvider(providers, "google-gemini-cli");
|
||||
const capturedStream = createCapturedThinkingConfigStream();
|
||||
|
||||
const runCase = (provider: typeof googleProvider, providerId: string) => {
|
||||
const wrapped = provider.wrapStreamFn?.({
|
||||
provider: providerId,
|
||||
modelId: "gemini-3.1-pro-preview",
|
||||
thinkingLevel: "high",
|
||||
streamFn: capturedStream.streamFn,
|
||||
} as never);
|
||||
|
||||
void wrapped?.(
|
||||
{
|
||||
api: "google-generative-ai",
|
||||
provider: providerId,
|
||||
id: "gemini-3.1-pro-preview",
|
||||
} as Model<"google-generative-ai">,
|
||||
{ messages: [] } as Context,
|
||||
{},
|
||||
);
|
||||
|
||||
const capturedPayload = capturedStream.getCapturedPayload();
|
||||
expect(capturedPayload).toEqual({
|
||||
config: {
|
||||
thinkingConfig: {
|
||||
thinkingLevel: "HIGH",
|
||||
},
|
||||
},
|
||||
});
|
||||
const thinkingConfig = (
|
||||
(capturedPayload as Record<string, unknown>).config as Record<string, unknown>
|
||||
).thinkingConfig as Record<string, unknown>;
|
||||
expect(thinkingConfig).not.toHaveProperty("thinkingBudget");
|
||||
};
|
||||
|
||||
runCase(googleProvider, "google");
|
||||
runCase(cliProvider, "google-gemini-cli");
|
||||
});
|
||||
|
||||
it("wires Vertex transport before request-time metadata ADC detection", async () => {
|
||||
const { providers } = await registerProviderPlugin({
|
||||
plugin: googleProviderPlugin,
|
||||
id: "google",
|
||||
name: "Google Provider",
|
||||
});
|
||||
const provider = requireRegisteredProvider(providers, "google");
|
||||
|
||||
expect(
|
||||
provider.createStreamFn?.({
|
||||
model: {
|
||||
api: "google-vertex",
|
||||
provider: "google",
|
||||
id: "gemini-2.5-pro",
|
||||
},
|
||||
} as never),
|
||||
).toEqual(expect.any(Function));
|
||||
});
|
||||
|
||||
it("advertises adaptive thinking for Gemini dynamic thinking", async () => {
|
||||
const { providers } = await registerProviderPlugin({
|
||||
plugin: googleProviderPlugin,
|
||||
id: "google",
|
||||
name: "Google Provider",
|
||||
});
|
||||
const provider = requireRegisteredProvider(providers, "google");
|
||||
if (!provider.resolveThinkingProfile) {
|
||||
throw new Error("expected Google provider thinking profile resolver");
|
||||
}
|
||||
const resolveThinkingProfile = provider.resolveThinkingProfile;
|
||||
const gemini3Profile = resolveThinkingProfile({
|
||||
provider: "google",
|
||||
modelId: "gemini-3.1-pro-preview",
|
||||
} as never);
|
||||
const gemini25Profile = resolveThinkingProfile({
|
||||
provider: "google",
|
||||
modelId: "gemini-2.5-flash",
|
||||
} as never);
|
||||
|
||||
expect(gemini3Profile?.levels).toEqual([
|
||||
{ id: "off" },
|
||||
{ id: "low" },
|
||||
{ id: "adaptive" },
|
||||
{ id: "high" },
|
||||
]);
|
||||
expect(gemini25Profile?.levels).toEqual([
|
||||
{ id: "off" },
|
||||
{ id: "minimal" },
|
||||
{ id: "low" },
|
||||
{ id: "medium" },
|
||||
{ id: "adaptive" },
|
||||
{ id: "high" },
|
||||
]);
|
||||
});
|
||||
|
||||
it("shares Gemini replay and stream hooks across Google provider variants", async () => {
|
||||
const { providers } = await registerProviderPlugin({
|
||||
plugin: googleProviderPlugin,
|
||||
id: "google",
|
||||
name: "Google Provider",
|
||||
});
|
||||
const googleProvider = requireRegisteredProvider(providers, "google");
|
||||
const cliProvider = requireRegisteredProvider(providers, "google-gemini-cli");
|
||||
|
||||
expect(googleProvider.buildReplayPolicy).toBe(cliProvider.buildReplayPolicy);
|
||||
expect(googleProvider.wrapStreamFn).toBe(cliProvider.wrapStreamFn);
|
||||
});
|
||||
|
||||
it("buffers early realtime audio while the lazy Google bridge loads", () => {
|
||||
let realtimeProvider: RealtimeVoiceProviderPlugin | undefined;
|
||||
googlePlugin.register(
|
||||
createTestPluginApi({
|
||||
registerRealtimeVoiceProvider(provider) {
|
||||
realtimeProvider = provider;
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
const bridge = realtimeProvider?.createBridge({
|
||||
providerConfig: { apiKey: "gemini-key" },
|
||||
onAudio() {},
|
||||
onClearAudio() {},
|
||||
});
|
||||
|
||||
if (!bridge) {
|
||||
throw new Error("expected Google realtime bridge");
|
||||
}
|
||||
expect(bridge.sendAudio(Buffer.alloc(160))).toBeUndefined();
|
||||
expect(bridge.setMediaTimestamp(20)).toBeUndefined();
|
||||
expect(bridge.sendUserMessage?.("hello")).toBeUndefined();
|
||||
});
|
||||
|
||||
it("refreshes Gemini CLI OAuth through the provider-owned refresh hook", async () => {
|
||||
refreshGeminiCliOAuthTokenMock.mockResolvedValueOnce({
|
||||
type: "oauth",
|
||||
provider: "google-gemini-cli",
|
||||
access: "fresh-access",
|
||||
refresh: "fresh-refresh",
|
||||
expires: Date.now() + 60_000,
|
||||
email: "user@example.com",
|
||||
projectId: "project-1",
|
||||
});
|
||||
|
||||
const { providers } = await registerProviderPlugin({
|
||||
plugin: googleProviderPlugin,
|
||||
id: "google",
|
||||
name: "Google Provider",
|
||||
});
|
||||
const provider = requireRegisteredProvider(providers, "google-gemini-cli");
|
||||
const credential = {
|
||||
type: "oauth" as const,
|
||||
provider: "google-gemini-cli",
|
||||
access: "stale-access",
|
||||
refresh: "stale-refresh",
|
||||
expires: Date.now() - 60_000,
|
||||
email: "user@example.com",
|
||||
projectId: "project-1",
|
||||
};
|
||||
|
||||
await expect(provider.refreshOAuth?.(credential)).resolves.toMatchObject({
|
||||
access: "fresh-access",
|
||||
refresh: "fresh-refresh",
|
||||
email: "user@example.com",
|
||||
projectId: "project-1",
|
||||
});
|
||||
expect(refreshGeminiCliOAuthTokenMock).toHaveBeenCalledWith(credential);
|
||||
});
|
||||
});
|
||||
355
extensions/google/index.ts
Normal file
355
extensions/google/index.ts
Normal file
@@ -0,0 +1,355 @@
|
||||
// Google plugin entrypoint registers its OpenClaw integration.
|
||||
import type { ImageGenerationProvider } from "openclaw/plugin-sdk/image-generation";
|
||||
import type { MediaUnderstandingProvider } from "openclaw/plugin-sdk/media-understanding";
|
||||
import type { MusicGenerationProvider } from "openclaw/plugin-sdk/music-generation";
|
||||
import { definePluginEntry } from "openclaw/plugin-sdk/plugin-entry";
|
||||
import type {
|
||||
RealtimeVoiceBridge,
|
||||
RealtimeVoiceBridgeCreateRequest,
|
||||
RealtimeVoiceProviderConfig,
|
||||
RealtimeVoiceProviderPlugin,
|
||||
} from "openclaw/plugin-sdk/realtime-voice";
|
||||
import { normalizeResolvedSecretInputString } from "openclaw/plugin-sdk/secret-input";
|
||||
import { normalizeOptionalString } from "openclaw/plugin-sdk/string-coerce-runtime";
|
||||
import type { VideoGenerationProvider } from "openclaw/plugin-sdk/video-generation";
|
||||
import { buildGoogleGeminiCliBackend } from "./cli-backend.js";
|
||||
import { registerGoogleGeminiCliProvider } from "./gemini-cli-provider.js";
|
||||
import {
|
||||
createGoogleMusicGenerationProviderMetadata,
|
||||
createGoogleVideoGenerationProviderMetadata,
|
||||
} from "./generation-provider-metadata.js";
|
||||
import { geminiMemoryEmbeddingProviderAdapter } from "./memory-embedding-adapter.js";
|
||||
import { registerGoogleProvider } from "./provider-registration.js";
|
||||
import { buildGoogleSpeechProvider } from "./speech-provider.js";
|
||||
import { createGeminiWebSearchProvider } from "./src/gemini-web-search-provider.js";
|
||||
|
||||
let googleImageGenerationProviderPromise: Promise<ImageGenerationProvider> | null = null;
|
||||
let googleMediaUnderstandingProviderPromise: Promise<MediaUnderstandingProvider> | null = null;
|
||||
let googleMusicGenerationProviderPromise: Promise<MusicGenerationProvider> | null = null;
|
||||
let googleRealtimeVoiceProviderPromise: Promise<RealtimeVoiceProviderPlugin> | null = null;
|
||||
let googleVideoGenerationProviderPromise: Promise<VideoGenerationProvider> | null = null;
|
||||
|
||||
type GoogleMediaUnderstandingProvider = Required<
|
||||
Pick<
|
||||
MediaUnderstandingProvider,
|
||||
"describeImage" | "describeImages" | "transcribeAudio" | "describeVideo"
|
||||
>
|
||||
>;
|
||||
|
||||
async function loadGoogleImageGenerationProvider(): Promise<ImageGenerationProvider> {
|
||||
if (!googleImageGenerationProviderPromise) {
|
||||
googleImageGenerationProviderPromise = import("./image-generation-provider.js").then((mod) =>
|
||||
mod.buildGoogleImageGenerationProvider(),
|
||||
);
|
||||
}
|
||||
return await googleImageGenerationProviderPromise;
|
||||
}
|
||||
|
||||
async function loadGoogleMediaUnderstandingProvider(): Promise<MediaUnderstandingProvider> {
|
||||
if (!googleMediaUnderstandingProviderPromise) {
|
||||
googleMediaUnderstandingProviderPromise = import("./media-understanding-provider.js").then(
|
||||
(mod) => mod.googleMediaUnderstandingProvider,
|
||||
);
|
||||
}
|
||||
return await googleMediaUnderstandingProviderPromise;
|
||||
}
|
||||
|
||||
async function loadGoogleMusicGenerationProvider(): Promise<MusicGenerationProvider> {
|
||||
if (!googleMusicGenerationProviderPromise) {
|
||||
googleMusicGenerationProviderPromise = import("./music-generation-provider.js").then((mod) =>
|
||||
mod.buildGoogleMusicGenerationProvider(),
|
||||
);
|
||||
}
|
||||
return await googleMusicGenerationProviderPromise;
|
||||
}
|
||||
|
||||
async function loadGoogleRealtimeVoiceProvider(): Promise<RealtimeVoiceProviderPlugin> {
|
||||
if (!googleRealtimeVoiceProviderPromise) {
|
||||
googleRealtimeVoiceProviderPromise = import("./realtime-voice-provider.js").then((mod) =>
|
||||
mod.buildGoogleRealtimeVoiceProvider(),
|
||||
);
|
||||
}
|
||||
return await googleRealtimeVoiceProviderPromise;
|
||||
}
|
||||
|
||||
async function loadGoogleVideoGenerationProvider(): Promise<VideoGenerationProvider> {
|
||||
if (!googleVideoGenerationProviderPromise) {
|
||||
googleVideoGenerationProviderPromise = import("./video-generation-provider.js").then((mod) =>
|
||||
mod.buildGoogleVideoGenerationProvider(),
|
||||
);
|
||||
}
|
||||
return await googleVideoGenerationProviderPromise;
|
||||
}
|
||||
|
||||
async function loadGoogleRequiredMediaUnderstandingProvider(): Promise<GoogleMediaUnderstandingProvider> {
|
||||
const provider = await loadGoogleMediaUnderstandingProvider();
|
||||
if (
|
||||
!provider.describeImage ||
|
||||
!provider.describeImages ||
|
||||
!provider.transcribeAudio ||
|
||||
!provider.describeVideo
|
||||
) {
|
||||
throw new Error("google media understanding provider missing required handlers");
|
||||
}
|
||||
return provider as GoogleMediaUnderstandingProvider;
|
||||
}
|
||||
|
||||
function createLazyGoogleImageGenerationProvider(): ImageGenerationProvider {
|
||||
return {
|
||||
id: "google",
|
||||
label: "Google",
|
||||
defaultModel: "gemini-3.1-flash-image-preview",
|
||||
models: ["gemini-3.1-flash-image-preview", "gemini-3-pro-image-preview"],
|
||||
capabilities: {
|
||||
generate: {
|
||||
maxCount: 4,
|
||||
supportsSize: true,
|
||||
supportsAspectRatio: true,
|
||||
supportsResolution: true,
|
||||
},
|
||||
edit: {
|
||||
enabled: true,
|
||||
maxCount: 4,
|
||||
maxInputImages: 5,
|
||||
supportsSize: true,
|
||||
supportsAspectRatio: true,
|
||||
supportsResolution: true,
|
||||
},
|
||||
geometry: {
|
||||
sizes: ["1024x1024", "1024x1536", "1536x1024", "1024x1792", "1792x1024"],
|
||||
aspectRatios: ["1:1", "2:3", "3:2", "3:4", "4:3", "4:5", "5:4", "9:16", "16:9", "21:9"],
|
||||
resolutions: ["1K", "2K", "4K"],
|
||||
},
|
||||
},
|
||||
generateImage: async (req) => (await loadGoogleImageGenerationProvider()).generateImage(req),
|
||||
};
|
||||
}
|
||||
|
||||
function createLazyGoogleMediaUnderstandingProvider(): MediaUnderstandingProvider {
|
||||
return {
|
||||
id: "google",
|
||||
capabilities: ["image", "audio", "video"],
|
||||
defaultModels: {
|
||||
image: "gemini-3-flash-preview",
|
||||
audio: "gemini-3-flash-preview",
|
||||
video: "gemini-3-flash-preview",
|
||||
},
|
||||
autoPriority: { image: 30, audio: 40, video: 10 },
|
||||
nativeDocumentInputs: ["pdf"],
|
||||
describeImage: async (...args) =>
|
||||
await (await loadGoogleRequiredMediaUnderstandingProvider()).describeImage(...args),
|
||||
describeImages: async (...args) =>
|
||||
await (await loadGoogleRequiredMediaUnderstandingProvider()).describeImages(...args),
|
||||
transcribeAudio: async (...args) =>
|
||||
await (await loadGoogleRequiredMediaUnderstandingProvider()).transcribeAudio(...args),
|
||||
describeVideo: async (...args) =>
|
||||
await (await loadGoogleRequiredMediaUnderstandingProvider()).describeVideo(...args),
|
||||
};
|
||||
}
|
||||
|
||||
function createLazyGoogleMusicGenerationProvider(): MusicGenerationProvider {
|
||||
return {
|
||||
...createGoogleMusicGenerationProviderMetadata(),
|
||||
generateMusic: async (...args) =>
|
||||
await (await loadGoogleMusicGenerationProvider()).generateMusic(...args),
|
||||
};
|
||||
}
|
||||
|
||||
function resolveGoogleRealtimeProviderConfig(
|
||||
rawConfig: RealtimeVoiceProviderConfig,
|
||||
cfg?: { models?: { providers?: { google?: { apiKey?: unknown } } } },
|
||||
): RealtimeVoiceProviderConfig {
|
||||
const providers =
|
||||
typeof rawConfig.providers === "object" &&
|
||||
rawConfig.providers !== null &&
|
||||
!Array.isArray(rawConfig.providers)
|
||||
? (rawConfig.providers as Record<string, unknown>)
|
||||
: undefined;
|
||||
const nested = providers?.google;
|
||||
const raw =
|
||||
typeof nested === "object" && nested !== null && !Array.isArray(nested)
|
||||
? (nested as Record<string, unknown>)
|
||||
: typeof rawConfig.google === "object" &&
|
||||
rawConfig.google !== null &&
|
||||
!Array.isArray(rawConfig.google)
|
||||
? (rawConfig.google as Record<string, unknown>)
|
||||
: rawConfig;
|
||||
return {
|
||||
...raw,
|
||||
...(raw.apiKey === undefined
|
||||
? cfg?.models?.providers?.google?.apiKey === undefined
|
||||
? {}
|
||||
: {
|
||||
apiKey: normalizeResolvedSecretInputString({
|
||||
value: cfg.models.providers.google.apiKey,
|
||||
path: "models.providers.google.apiKey",
|
||||
}),
|
||||
}
|
||||
: {
|
||||
apiKey: normalizeResolvedSecretInputString({
|
||||
value: raw.apiKey,
|
||||
path: "plugins.entries.voice-call.config.realtime.providers.google.apiKey",
|
||||
}),
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
function resolveGoogleRealtimeEnvApiKey(): string | undefined {
|
||||
return (
|
||||
normalizeOptionalString(process.env.GEMINI_API_KEY) ??
|
||||
normalizeOptionalString(process.env.GOOGLE_API_KEY)
|
||||
);
|
||||
}
|
||||
|
||||
const GOOGLE_REALTIME_LAZY_MAX_PENDING_AUDIO_CHUNKS = 320;
|
||||
|
||||
function createLazyGoogleRealtimeVoiceBridge(
|
||||
req: RealtimeVoiceBridgeCreateRequest,
|
||||
): RealtimeVoiceBridge {
|
||||
let bridge: RealtimeVoiceBridge | undefined;
|
||||
let bridgePromise: Promise<RealtimeVoiceBridge> | undefined;
|
||||
let closed = false;
|
||||
let latestMediaTimestamp: number | undefined;
|
||||
let pendingGreeting: string | undefined;
|
||||
const pendingAudio: Buffer[] = [];
|
||||
const pendingUserMessages: string[] = [];
|
||||
const loadBridge = async () => {
|
||||
if (!bridgePromise) {
|
||||
bridgePromise = loadGoogleRealtimeVoiceProvider().then((provider) =>
|
||||
provider.createBridge(req),
|
||||
);
|
||||
}
|
||||
bridge = await bridgePromise;
|
||||
return bridge;
|
||||
};
|
||||
const requireBridge = () => {
|
||||
if (!bridge) {
|
||||
throw new Error("Google realtime voice bridge is not connected");
|
||||
}
|
||||
return bridge;
|
||||
};
|
||||
const flushPending = (loadedBridge: RealtimeVoiceBridge) => {
|
||||
if (typeof latestMediaTimestamp === "number") {
|
||||
loadedBridge.setMediaTimestamp(latestMediaTimestamp);
|
||||
}
|
||||
for (const audio of pendingAudio.splice(0)) {
|
||||
loadedBridge.sendAudio(audio);
|
||||
}
|
||||
for (const text of pendingUserMessages.splice(0)) {
|
||||
loadedBridge.sendUserMessage?.(text);
|
||||
}
|
||||
if (pendingGreeting !== undefined) {
|
||||
const greeting = pendingGreeting;
|
||||
pendingGreeting = undefined;
|
||||
loadedBridge.triggerGreeting?.(greeting);
|
||||
}
|
||||
};
|
||||
return {
|
||||
supportsToolResultContinuation: true,
|
||||
connect: async () => {
|
||||
const loadedBridge = await loadBridge();
|
||||
if (closed) {
|
||||
loadedBridge.close();
|
||||
return;
|
||||
}
|
||||
await loadedBridge.connect();
|
||||
flushPending(loadedBridge);
|
||||
},
|
||||
sendAudio: (audio) => {
|
||||
if (bridge) {
|
||||
bridge.sendAudio(audio);
|
||||
return;
|
||||
}
|
||||
if (!closed) {
|
||||
if (pendingAudio.length >= GOOGLE_REALTIME_LAZY_MAX_PENDING_AUDIO_CHUNKS) {
|
||||
pendingAudio.shift();
|
||||
}
|
||||
pendingAudio.push(audio);
|
||||
}
|
||||
},
|
||||
setMediaTimestamp: (ts) => {
|
||||
latestMediaTimestamp = ts;
|
||||
bridge?.setMediaTimestamp(ts);
|
||||
},
|
||||
sendUserMessage: (text) => {
|
||||
if (bridge) {
|
||||
bridge.sendUserMessage?.(text);
|
||||
return;
|
||||
}
|
||||
if (!closed) {
|
||||
pendingUserMessages.push(text);
|
||||
}
|
||||
},
|
||||
triggerGreeting: (instructions) => {
|
||||
if (bridge) {
|
||||
bridge.triggerGreeting?.(instructions);
|
||||
return;
|
||||
}
|
||||
if (!closed) {
|
||||
pendingGreeting = instructions;
|
||||
}
|
||||
},
|
||||
handleBargeIn: (options) => requireBridge().handleBargeIn?.(options),
|
||||
submitToolResult: (callId, result, options) =>
|
||||
requireBridge().submitToolResult(callId, result, options),
|
||||
acknowledgeMark: () => requireBridge().acknowledgeMark(),
|
||||
close: () => {
|
||||
closed = true;
|
||||
pendingAudio.length = 0;
|
||||
pendingUserMessages.length = 0;
|
||||
pendingGreeting = undefined;
|
||||
bridge?.close();
|
||||
},
|
||||
isConnected: () => bridge?.isConnected() ?? false,
|
||||
};
|
||||
}
|
||||
|
||||
function createLazyGoogleRealtimeVoiceProvider(): RealtimeVoiceProviderPlugin {
|
||||
return {
|
||||
id: "google",
|
||||
label: "Google Live Voice",
|
||||
autoSelectOrder: 20,
|
||||
resolveConfig: ({ cfg, rawConfig }) => resolveGoogleRealtimeProviderConfig(rawConfig, cfg),
|
||||
isConfigured: ({ cfg, providerConfig }) =>
|
||||
Boolean(
|
||||
normalizeOptionalString(providerConfig.apiKey) ??
|
||||
normalizeOptionalString(cfg?.models?.providers?.google?.apiKey) ??
|
||||
resolveGoogleRealtimeEnvApiKey(),
|
||||
),
|
||||
createBridge: createLazyGoogleRealtimeVoiceBridge,
|
||||
createBrowserSession: async (req) => {
|
||||
const provider = await loadGoogleRealtimeVoiceProvider();
|
||||
if (!provider.createBrowserSession) {
|
||||
throw new Error("Google realtime voice browser sessions are unavailable");
|
||||
}
|
||||
return await provider.createBrowserSession(req);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function createLazyGoogleVideoGenerationProvider(): VideoGenerationProvider {
|
||||
return {
|
||||
...createGoogleVideoGenerationProviderMetadata(),
|
||||
generateVideo: async (...args) =>
|
||||
await (await loadGoogleVideoGenerationProvider()).generateVideo(...args),
|
||||
};
|
||||
}
|
||||
|
||||
export default definePluginEntry({
|
||||
id: "google",
|
||||
name: "Google Plugin",
|
||||
description: "Bundled Google plugin",
|
||||
register(api) {
|
||||
api.registerCliBackend(buildGoogleGeminiCliBackend());
|
||||
registerGoogleGeminiCliProvider(api);
|
||||
registerGoogleProvider(api);
|
||||
api.registerMemoryEmbeddingProvider(geminiMemoryEmbeddingProviderAdapter);
|
||||
api.registerImageGenerationProvider(createLazyGoogleImageGenerationProvider());
|
||||
api.registerMediaUnderstandingProvider(createLazyGoogleMediaUnderstandingProvider());
|
||||
api.registerMusicGenerationProvider(createLazyGoogleMusicGenerationProvider());
|
||||
api.registerRealtimeVoiceProvider(createLazyGoogleRealtimeVoiceProvider());
|
||||
api.registerSpeechProvider(buildGoogleSpeechProvider());
|
||||
api.registerVideoGenerationProvider(createLazyGoogleVideoGenerationProvider());
|
||||
api.registerWebSearchProvider(createGeminiWebSearchProvider());
|
||||
},
|
||||
});
|
||||
105
extensions/google/manifest.test.ts
Normal file
105
extensions/google/manifest.test.ts
Normal file
@@ -0,0 +1,105 @@
|
||||
// Google tests cover manifest plugin behavior.
|
||||
import { readFileSync } from "node:fs";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
type GoogleManifest = {
|
||||
modelIdNormalization?: {
|
||||
providers?: Record<
|
||||
string,
|
||||
{
|
||||
aliases?: Record<string, string>;
|
||||
}
|
||||
>;
|
||||
};
|
||||
modelCatalog?: {
|
||||
suppressions?: Array<{
|
||||
provider?: string;
|
||||
model?: string;
|
||||
reason?: string;
|
||||
}>;
|
||||
};
|
||||
};
|
||||
|
||||
const RETIRED_GEMINI_CHAT_MODELS = [
|
||||
"gemini-1.5-flash",
|
||||
"gemini-1.5-flash-8b",
|
||||
"gemini-1.5-pro",
|
||||
"gemini-2.0-flash-exp",
|
||||
"gemini-2.0-flash-exp-image-generation",
|
||||
"gemini-2.0-flash-live-001",
|
||||
"gemini-2.0-flash-lite-preview",
|
||||
"gemini-2.0-flash-lite-preview-02-05",
|
||||
"gemini-2.0-flash-preview-image-generation",
|
||||
"gemini-2.0-flash-thinking-exp",
|
||||
"gemini-2.0-flash-thinking-exp-01-21",
|
||||
"gemini-2.0-flash-thinking-exp-1219",
|
||||
"gemini-2.0-pro-exp",
|
||||
"gemini-2.0-pro-exp-02-05",
|
||||
"gemini-2.5-flash-exp-native-audio-thinking-dialog",
|
||||
"gemini-2.5-flash-image-preview",
|
||||
"gemini-2.5-flash-lite-preview-06-17",
|
||||
"gemini-2.5-flash-lite-preview-09-25",
|
||||
"gemini-2.5-flash-lite-preview-09-2025",
|
||||
"gemini-2.5-flash-preview-04-17",
|
||||
"gemini-2.5-flash-preview-05-20",
|
||||
"gemini-2.5-flash-preview-09-25",
|
||||
"gemini-2.5-flash-preview-09-2025",
|
||||
"gemini-2.5-flash-preview-native-audio-dialog",
|
||||
"gemini-2.5-pro-exp-03-25",
|
||||
"gemini-2.5-pro-preview-03-25",
|
||||
"gemini-2.5-pro-preview-05-06",
|
||||
"gemini-2.5-pro-preview-06-05",
|
||||
"gemini-3-pro-preview",
|
||||
"gemini-3.1-pro-preview-customtools",
|
||||
"gemini-live-2.5-flash",
|
||||
"gemini-live-2.5-flash-preview",
|
||||
"gemini-live-2.5-flash-preview-native-audio",
|
||||
] as const;
|
||||
|
||||
const GOOGLE_CHAT_PROVIDERS = ["google", "google-gemini-cli", "google-vertex"] as const;
|
||||
|
||||
function loadManifest(): GoogleManifest {
|
||||
return JSON.parse(readFileSync(new URL("./openclaw.plugin.json", import.meta.url), "utf8"));
|
||||
}
|
||||
|
||||
describe("google manifest model catalog", () => {
|
||||
it("suppresses retired Gemini chat model identifiers for all Google chat providers", () => {
|
||||
const manifest = loadManifest();
|
||||
const suppressionRefs = new Set(
|
||||
(manifest.modelCatalog?.suppressions ?? []).map(
|
||||
(suppression) => `${suppression.provider}/${suppression.model}`,
|
||||
),
|
||||
);
|
||||
|
||||
for (const provider of GOOGLE_CHAT_PROVIDERS) {
|
||||
for (const model of RETIRED_GEMINI_CHAT_MODELS) {
|
||||
expect(suppressionRefs).toContain(`${provider}/${model}`);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
it("does not suppress still-callable Google replacement models", () => {
|
||||
const manifest = loadManifest();
|
||||
const suppressionRefs = new Set(
|
||||
(manifest.modelCatalog?.suppressions ?? []).map(
|
||||
(suppression) => `${suppression.provider}/${suppression.model}`,
|
||||
),
|
||||
);
|
||||
|
||||
expect(suppressionRefs).not.toContain("google/gemini-2.0-flash");
|
||||
expect(suppressionRefs).not.toContain("google/gemini-2.5-flash");
|
||||
expect(suppressionRefs).not.toContain("google/gemini-2.5-flash-lite");
|
||||
expect(suppressionRefs).not.toContain("google/gemini-2.5-pro");
|
||||
expect(suppressionRefs).not.toContain("google/gemini-3.1-pro-preview");
|
||||
});
|
||||
|
||||
it("normalizes retired Gemini 3 Pro aliases for all Google chat providers", () => {
|
||||
const manifest = loadManifest();
|
||||
|
||||
for (const provider of GOOGLE_CHAT_PROVIDERS) {
|
||||
const aliases = manifest.modelIdNormalization?.providers?.[provider]?.aliases;
|
||||
expect(aliases?.["gemini-3-pro"]).toBe("gemini-3.1-pro-preview");
|
||||
expect(aliases?.["gemini-3-pro-preview"]).toBe("gemini-3.1-pro-preview");
|
||||
}
|
||||
});
|
||||
});
|
||||
164
extensions/google/media-understanding-provider.ts
Normal file
164
extensions/google/media-understanding-provider.ts
Normal file
@@ -0,0 +1,164 @@
|
||||
// Google provider module implements model/runtime integration.
|
||||
import {
|
||||
describeImageWithModel,
|
||||
describeImagesWithModel,
|
||||
type AudioTranscriptionRequest,
|
||||
type AudioTranscriptionResult,
|
||||
type MediaUnderstandingProvider,
|
||||
type VideoDescriptionRequest,
|
||||
type VideoDescriptionResult,
|
||||
} from "openclaw/plugin-sdk/media-understanding";
|
||||
import {
|
||||
assertOkOrThrowProviderError,
|
||||
postJsonRequest,
|
||||
readProviderJsonResponse,
|
||||
type ProviderRequestTransportOverrides,
|
||||
} from "openclaw/plugin-sdk/provider-http";
|
||||
import {
|
||||
DEFAULT_GOOGLE_API_BASE_URL,
|
||||
normalizeGoogleModelId,
|
||||
resolveGoogleGenerativeAiHttpRequestConfig,
|
||||
} from "./runtime-api.js";
|
||||
|
||||
const DEFAULT_GOOGLE_AUDIO_MODEL = "gemini-3-flash-preview";
|
||||
const DEFAULT_GOOGLE_VIDEO_MODEL = "gemini-3-flash-preview";
|
||||
const DEFAULT_GOOGLE_AUDIO_PROMPT = "Transcribe the audio.";
|
||||
const DEFAULT_GOOGLE_VIDEO_PROMPT = "Describe the video.";
|
||||
|
||||
async function generateGeminiInlineDataText(params: {
|
||||
buffer: Buffer;
|
||||
mime?: string;
|
||||
apiKey: string;
|
||||
baseUrl?: string;
|
||||
headers?: Record<string, string>;
|
||||
request?: ProviderRequestTransportOverrides;
|
||||
model?: string;
|
||||
prompt?: string;
|
||||
timeoutMs: number;
|
||||
fetchFn?: typeof fetch;
|
||||
defaultBaseUrl: string;
|
||||
defaultModel: string;
|
||||
defaultPrompt: string;
|
||||
defaultMime: string;
|
||||
httpErrorLabel: string;
|
||||
missingTextError: string;
|
||||
}): Promise<{ text: string; model: string }> {
|
||||
const fetchFn = params.fetchFn ?? fetch;
|
||||
const model = (() => {
|
||||
const trimmed = params.model?.trim();
|
||||
if (!trimmed) {
|
||||
return params.defaultModel;
|
||||
}
|
||||
return normalizeGoogleModelId(trimmed);
|
||||
})();
|
||||
const { baseUrl, allowPrivateNetwork, headers, dispatcherPolicy } =
|
||||
resolveGoogleGenerativeAiHttpRequestConfig({
|
||||
apiKey: params.apiKey,
|
||||
baseUrl: params.baseUrl,
|
||||
headers: params.headers,
|
||||
request: params.request,
|
||||
capability: params.defaultMime.startsWith("audio/") ? "audio" : "video",
|
||||
transport: "media-understanding",
|
||||
});
|
||||
const resolvedBaseUrl = baseUrl ?? params.defaultBaseUrl;
|
||||
const url = `${resolvedBaseUrl}/models/${model}:generateContent`;
|
||||
|
||||
const prompt = (() => {
|
||||
const trimmed = params.prompt?.trim();
|
||||
return trimmed || params.defaultPrompt;
|
||||
})();
|
||||
|
||||
const body = {
|
||||
contents: [
|
||||
{
|
||||
role: "user",
|
||||
parts: [
|
||||
{ text: prompt },
|
||||
{
|
||||
inline_data: {
|
||||
mime_type: params.mime ?? params.defaultMime,
|
||||
data: params.buffer.toString("base64"),
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
const { response: res, release } = await postJsonRequest({
|
||||
url,
|
||||
headers,
|
||||
body,
|
||||
timeoutMs: params.timeoutMs,
|
||||
fetchFn,
|
||||
allowPrivateNetwork,
|
||||
dispatcherPolicy,
|
||||
});
|
||||
|
||||
try {
|
||||
await assertOkOrThrowProviderError(res, params.httpErrorLabel);
|
||||
|
||||
const payload = await readProviderJsonResponse<{
|
||||
candidates?: Array<{
|
||||
content?: { parts?: Array<{ text?: string }> };
|
||||
}>;
|
||||
}>(res, params.httpErrorLabel);
|
||||
const parts = payload.candidates?.[0]?.content?.parts ?? [];
|
||||
const text = parts
|
||||
.map((part) => part?.text?.trim())
|
||||
.filter(Boolean)
|
||||
.join("\n");
|
||||
if (!text) {
|
||||
throw new Error(params.missingTextError);
|
||||
}
|
||||
return { text, model };
|
||||
} finally {
|
||||
await release();
|
||||
}
|
||||
}
|
||||
|
||||
export async function transcribeGeminiAudio(
|
||||
params: AudioTranscriptionRequest,
|
||||
): Promise<AudioTranscriptionResult> {
|
||||
const { text, model } = await generateGeminiInlineDataText({
|
||||
...params,
|
||||
defaultBaseUrl: DEFAULT_GOOGLE_API_BASE_URL,
|
||||
defaultModel: DEFAULT_GOOGLE_AUDIO_MODEL,
|
||||
defaultPrompt: DEFAULT_GOOGLE_AUDIO_PROMPT,
|
||||
defaultMime: "audio/wav",
|
||||
httpErrorLabel: "Audio transcription failed",
|
||||
missingTextError: "Audio transcription response missing text",
|
||||
});
|
||||
return { text, model };
|
||||
}
|
||||
|
||||
export async function describeGeminiVideo(
|
||||
params: VideoDescriptionRequest,
|
||||
): Promise<VideoDescriptionResult> {
|
||||
const { text, model } = await generateGeminiInlineDataText({
|
||||
...params,
|
||||
defaultBaseUrl: DEFAULT_GOOGLE_API_BASE_URL,
|
||||
defaultModel: DEFAULT_GOOGLE_VIDEO_MODEL,
|
||||
defaultPrompt: DEFAULT_GOOGLE_VIDEO_PROMPT,
|
||||
defaultMime: "video/mp4",
|
||||
httpErrorLabel: "Video description failed",
|
||||
missingTextError: "Video description response missing text",
|
||||
});
|
||||
return { text, model };
|
||||
}
|
||||
|
||||
export const googleMediaUnderstandingProvider: MediaUnderstandingProvider = {
|
||||
id: "google",
|
||||
capabilities: ["image", "audio", "video"],
|
||||
defaultModels: {
|
||||
image: DEFAULT_GOOGLE_VIDEO_MODEL,
|
||||
audio: DEFAULT_GOOGLE_AUDIO_MODEL,
|
||||
video: DEFAULT_GOOGLE_VIDEO_MODEL,
|
||||
},
|
||||
autoPriority: { image: 30, audio: 40, video: 10 },
|
||||
nativeDocumentInputs: ["pdf"],
|
||||
describeImage: describeImageWithModel,
|
||||
describeImages: describeImagesWithModel,
|
||||
transcribeAudio: transcribeGeminiAudio,
|
||||
describeVideo: describeGeminiVideo,
|
||||
};
|
||||
226
extensions/google/media-understanding-provider.video.test.ts
Normal file
226
extensions/google/media-understanding-provider.video.test.ts
Normal file
@@ -0,0 +1,226 @@
|
||||
// Google tests cover media understanding provider.video plugin behavior.
|
||||
import { createServer, type Server } from "node:http";
|
||||
import {
|
||||
createRequestCaptureJsonFetch,
|
||||
installPinnedHostnameTestHooks,
|
||||
withFetchPreconnect,
|
||||
} from "openclaw/plugin-sdk/test-env";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { describeGeminiVideo, transcribeGeminiAudio } from "./media-understanding-provider.js";
|
||||
import { resolveGoogleGenerativeAiHttpRequestConfig } from "./runtime-api.js";
|
||||
|
||||
installPinnedHostnameTestHooks();
|
||||
|
||||
const LOOPBACK_RESPONSE_BYTES = 18 * 1024 * 1024;
|
||||
|
||||
async function listenLoopbackServer(server: Server): Promise<number> {
|
||||
return await new Promise((resolve, reject) => {
|
||||
server.once("error", reject);
|
||||
server.listen(0, "127.0.0.1", () => {
|
||||
server.off("error", reject);
|
||||
const address = server.address();
|
||||
if (!address || typeof address === "string") {
|
||||
reject(new Error("expected loopback TCP address"));
|
||||
return;
|
||||
}
|
||||
resolve(address.port);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
function createOversizedJsonServer(): { server: Server; closed: Promise<number> } {
|
||||
let resolveClosed: (sentBytes: number) => void = () => {};
|
||||
const closed = new Promise<number>((resolve) => {
|
||||
resolveClosed = resolve;
|
||||
});
|
||||
const server = createServer((_req, res) => {
|
||||
let sentBytes = 0;
|
||||
const chunk = Buffer.alloc(64 * 1024, 0x20);
|
||||
res.writeHead(200, { "content-type": "application/json" });
|
||||
const timer = setInterval(() => {
|
||||
if (sentBytes >= LOOPBACK_RESPONSE_BYTES) {
|
||||
clearInterval(timer);
|
||||
res.end();
|
||||
return;
|
||||
}
|
||||
sentBytes += chunk.length;
|
||||
res.write(chunk);
|
||||
}, 1);
|
||||
res.on("close", () => {
|
||||
clearInterval(timer);
|
||||
resolveClosed(sentBytes);
|
||||
});
|
||||
});
|
||||
return { server, closed };
|
||||
}
|
||||
|
||||
describe("describeGeminiVideo", () => {
|
||||
it("respects case-insensitive x-goog-api-key overrides", async () => {
|
||||
let seenKey: string | null = null;
|
||||
const fetchFn = withFetchPreconnect(async (_input: RequestInfo | URL, init?: RequestInit) => {
|
||||
const headers = new Headers(init?.headers);
|
||||
seenKey = headers.get("x-goog-api-key");
|
||||
return new Response(
|
||||
JSON.stringify({
|
||||
candidates: [{ content: { parts: [{ text: "video ok" }] } }],
|
||||
}),
|
||||
{ status: 200, headers: { "content-type": "application/json" } },
|
||||
);
|
||||
});
|
||||
|
||||
const result = await describeGeminiVideo({
|
||||
buffer: Buffer.from("video"),
|
||||
fileName: "clip.mp4",
|
||||
apiKey: "test-key",
|
||||
timeoutMs: 1000,
|
||||
headers: { "X-Goog-Api-Key": "override" },
|
||||
fetchFn,
|
||||
});
|
||||
|
||||
expect(seenKey).toBe("override");
|
||||
expect(result.text).toBe("video ok");
|
||||
});
|
||||
|
||||
it("keeps private-network disabled for the default Google media endpoint", async () => {
|
||||
expect(
|
||||
resolveGoogleGenerativeAiHttpRequestConfig({
|
||||
apiKey: "test-key",
|
||||
capability: "video",
|
||||
transport: "media-understanding",
|
||||
}).allowPrivateNetwork,
|
||||
).toBe(false);
|
||||
|
||||
const fetchFn = withFetchPreconnect(async () => {
|
||||
return new Response(
|
||||
JSON.stringify({
|
||||
candidates: [{ content: { parts: [{ text: "video ok" }] } }],
|
||||
}),
|
||||
{ status: 200, headers: { "content-type": "application/json" } },
|
||||
);
|
||||
});
|
||||
|
||||
await describeGeminiVideo({
|
||||
buffer: Buffer.from("video"),
|
||||
fileName: "clip.mp4",
|
||||
apiKey: "test-key",
|
||||
timeoutMs: 1000,
|
||||
fetchFn,
|
||||
});
|
||||
});
|
||||
|
||||
it("builds the expected request payload", async () => {
|
||||
const { fetchFn, getRequest } = createRequestCaptureJsonFetch({
|
||||
candidates: [
|
||||
{
|
||||
content: {
|
||||
parts: [{ text: "first" }, { text: " second " }, { text: "" }],
|
||||
},
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
const result = await describeGeminiVideo({
|
||||
buffer: Buffer.from("video-bytes"),
|
||||
fileName: "clip.mp4",
|
||||
apiKey: "test-key",
|
||||
timeoutMs: 1500,
|
||||
baseUrl: "https://generativelanguage.googleapis.com/v1beta/",
|
||||
model: "gemini-3-pro",
|
||||
headers: { "X-Other": "1" },
|
||||
fetchFn,
|
||||
});
|
||||
const { url: seenUrl, init: seenInit } = getRequest();
|
||||
|
||||
expect(result.model).toBe("gemini-3.1-pro-preview");
|
||||
expect(result.text).toBe("first\nsecond");
|
||||
expect(seenUrl).toBe(
|
||||
"https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-pro-preview:generateContent",
|
||||
);
|
||||
expect(seenInit?.method).toBe("POST");
|
||||
expect(seenInit?.signal).toBeInstanceOf(AbortSignal);
|
||||
|
||||
const headers = new Headers(seenInit?.headers);
|
||||
expect(headers.get("x-goog-api-key")).toBe("test-key");
|
||||
expect(headers.get("content-type")).toBe("application/json");
|
||||
expect(headers.get("x-other")).toBe("1");
|
||||
|
||||
const bodyText =
|
||||
typeof seenInit?.body === "string"
|
||||
? seenInit.body
|
||||
: Buffer.isBuffer(seenInit?.body)
|
||||
? seenInit.body.toString("utf8")
|
||||
: "";
|
||||
const body = JSON.parse(bodyText);
|
||||
expect(body.contents?.[0]?.parts?.[0]?.text).toBe("Describe the video.");
|
||||
expect(body.contents?.[0]?.parts?.[1]?.inline_data?.mime_type).toBe("video/mp4");
|
||||
expect(body.contents?.[0]?.parts?.[1]?.inline_data?.data).toBe(
|
||||
Buffer.from("video-bytes").toString("base64"),
|
||||
);
|
||||
});
|
||||
|
||||
it("bounds oversized video JSON responses and closes the stream early", async () => {
|
||||
const { server, closed } = createOversizedJsonServer();
|
||||
const port = await listenLoopbackServer(server);
|
||||
const fetchFn = withFetchPreconnect(async () =>
|
||||
fetch(`http://127.0.0.1:${port}/google-video-json`),
|
||||
);
|
||||
|
||||
try {
|
||||
await expect(
|
||||
describeGeminiVideo({
|
||||
buffer: Buffer.from("video-bytes"),
|
||||
fileName: "clip.mp4",
|
||||
apiKey: "test-key",
|
||||
timeoutMs: 1500,
|
||||
fetchFn,
|
||||
}),
|
||||
).rejects.toThrow(/JSON response exceeds 16777216 bytes/u);
|
||||
await expect(closed).resolves.toBeLessThan(LOOPBACK_RESPONSE_BYTES);
|
||||
} finally {
|
||||
server.close();
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects non-Google video base URLs before sending authenticated requests", async () => {
|
||||
await expect(
|
||||
describeGeminiVideo({
|
||||
buffer: Buffer.from("video-bytes"),
|
||||
fileName: "clip.mp4",
|
||||
apiKey: "test-key",
|
||||
timeoutMs: 1500,
|
||||
baseUrl: "https://example.com/v1beta/",
|
||||
fetchFn: async () => {
|
||||
throw new Error("fetch should not run");
|
||||
},
|
||||
}),
|
||||
).rejects.toThrow(
|
||||
"Google Generative AI baseUrl must use https://generativelanguage.googleapis.com",
|
||||
);
|
||||
});
|
||||
|
||||
it("formats Google audio transcription HTTP errors with provider details", async () => {
|
||||
await expect(
|
||||
transcribeGeminiAudio({
|
||||
buffer: Buffer.from("audio-bytes"),
|
||||
fileName: "clip.wav",
|
||||
apiKey: "test-key",
|
||||
timeoutMs: 1500,
|
||||
fetchFn: async () =>
|
||||
new Response(
|
||||
JSON.stringify({
|
||||
error: {
|
||||
message: "Unsupported audio",
|
||||
status: "INVALID_ARGUMENT",
|
||||
},
|
||||
}),
|
||||
{
|
||||
status: 400,
|
||||
headers: { "x-request-id": "google_audio_req" },
|
||||
},
|
||||
),
|
||||
}),
|
||||
).rejects.toThrow(
|
||||
"Audio transcription failed (400): Unsupported audio [code=INVALID_ARGUMENT] [request_id=google_audio_req]",
|
||||
);
|
||||
});
|
||||
});
|
||||
80
extensions/google/memory-embedding-adapter.ts
Normal file
80
extensions/google/memory-embedding-adapter.ts
Normal file
@@ -0,0 +1,80 @@
|
||||
// Google plugin module implements memory embedding adapter behavior.
|
||||
import {
|
||||
hasNonTextEmbeddingParts,
|
||||
isMissingEmbeddingApiKeyError,
|
||||
mapBatchEmbeddingsByIndex,
|
||||
sanitizeEmbeddingCacheHeaders,
|
||||
type MemoryEmbeddingProviderAdapter,
|
||||
} from "openclaw/plugin-sdk/memory-core-host-engine-embeddings";
|
||||
import { runGeminiEmbeddingBatches } from "./embedding-batch.js";
|
||||
import {
|
||||
buildGeminiEmbeddingRequest,
|
||||
createGeminiEmbeddingProvider,
|
||||
DEFAULT_GEMINI_EMBEDDING_MODEL,
|
||||
} from "./embedding-provider.js";
|
||||
|
||||
function supportsGeminiMultimodalEmbeddings(model: string): boolean {
|
||||
const normalized = model
|
||||
.trim()
|
||||
.replace(/^models\//, "")
|
||||
.replace(/^(gemini|google)\//, "");
|
||||
return normalized === "gemini-embedding-2-preview";
|
||||
}
|
||||
|
||||
export const geminiMemoryEmbeddingProviderAdapter: MemoryEmbeddingProviderAdapter = {
|
||||
id: "gemini",
|
||||
defaultModel: DEFAULT_GEMINI_EMBEDDING_MODEL,
|
||||
transport: "remote",
|
||||
authProviderId: "google",
|
||||
autoSelectPriority: 30,
|
||||
allowExplicitWhenConfiguredAuto: true,
|
||||
supportsMultimodalEmbeddings: ({ model }) => supportsGeminiMultimodalEmbeddings(model),
|
||||
shouldContinueAutoSelection: isMissingEmbeddingApiKeyError,
|
||||
create: async (options) => {
|
||||
const { provider, client } = await createGeminiEmbeddingProvider({
|
||||
...options,
|
||||
provider: "gemini",
|
||||
fallback: "none",
|
||||
});
|
||||
return {
|
||||
provider,
|
||||
runtime: {
|
||||
id: "gemini",
|
||||
cacheKeyData: {
|
||||
provider: "gemini",
|
||||
baseUrl: client.baseUrl,
|
||||
model: client.model,
|
||||
outputDimensionality: client.outputDimensionality,
|
||||
headers: sanitizeEmbeddingCacheHeaders(client.headers, [
|
||||
"authorization",
|
||||
"x-goog-api-key",
|
||||
]),
|
||||
},
|
||||
batchEmbed: async (batch) => {
|
||||
if (batch.chunks.some((chunk) => hasNonTextEmbeddingParts(chunk.embeddingInput))) {
|
||||
return null;
|
||||
}
|
||||
const byCustomId = await runGeminiEmbeddingBatches({
|
||||
gemini: client,
|
||||
agentId: batch.agentId,
|
||||
requests: batch.chunks.map((chunk, index) => ({
|
||||
custom_id: String(index),
|
||||
request: buildGeminiEmbeddingRequest({
|
||||
input: chunk.embeddingInput ?? { text: chunk.text },
|
||||
taskType: "RETRIEVAL_DOCUMENT",
|
||||
modelPath: client.modelPath,
|
||||
outputDimensionality: client.outputDimensionality,
|
||||
}),
|
||||
})),
|
||||
wait: batch.wait,
|
||||
concurrency: batch.concurrency,
|
||||
pollIntervalMs: batch.pollIntervalMs,
|
||||
timeoutMs: batch.timeoutMs,
|
||||
debug: batch.debug,
|
||||
});
|
||||
return mapBatchEmbeddingsByIndex(byCustomId, batch.chunks.length);
|
||||
},
|
||||
},
|
||||
};
|
||||
},
|
||||
};
|
||||
49
extensions/google/model-id.test.ts
Normal file
49
extensions/google/model-id.test.ts
Normal file
@@ -0,0 +1,49 @@
|
||||
// Google tests cover model id plugin behavior.
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { normalizeAntigravityModelId, normalizeGoogleModelId } from "./api.js";
|
||||
|
||||
describe("google model id helpers", () => {
|
||||
it.each(["gemini-3-pro", "gemini-3.1-pro", "gemini-3-1-pro"])(
|
||||
"adds default -low suffix to bare antigravity pro id: %s",
|
||||
(id) => {
|
||||
expect(normalizeAntigravityModelId(id)).toBe(`${id}-low`);
|
||||
},
|
||||
);
|
||||
|
||||
it.each([
|
||||
"gemini-3-pro-low",
|
||||
"gemini-3-pro-high",
|
||||
"gemini-3.1-flash",
|
||||
"claude-opus-4-6-thinking",
|
||||
])("keeps already-tiered and non-pro ids unchanged: %s", (id) => {
|
||||
expect(normalizeAntigravityModelId(id)).toBe(id);
|
||||
});
|
||||
|
||||
it("maps the deprecated 3.1 flash alias to the real preview model", () => {
|
||||
expect(normalizeGoogleModelId("gemini-3.1-flash")).toBe("gemini-3-flash-preview");
|
||||
expect(normalizeGoogleModelId("gemini-3.1-flash-preview")).toBe("gemini-3-flash-preview");
|
||||
});
|
||||
|
||||
it("keeps bare Gemini 3.1 Pro as an alias for Google's preview-suffixed API id", () => {
|
||||
expect(normalizeGoogleModelId("gemini-3-pro")).toBe("gemini-3.1-pro-preview");
|
||||
expect(normalizeGoogleModelId("gemini-3-pro-preview")).toBe("gemini-3.1-pro-preview");
|
||||
expect(normalizeGoogleModelId("gemini-3.1-pro")).toBe("gemini-3.1-pro-preview");
|
||||
expect(normalizeGoogleModelId("gemini-3.1-pro-preview")).toBe("gemini-3.1-pro-preview");
|
||||
});
|
||||
|
||||
it("normalizes provider-prefixed Gemini 3 Pro config ids", () => {
|
||||
expect(normalizeGoogleModelId("google/gemini-3-pro-preview")).toBe(
|
||||
"google/gemini-3.1-pro-preview",
|
||||
);
|
||||
});
|
||||
|
||||
it("keeps GA gemini-3.1-flash-lite unchanged and maps old preview name to GA", () => {
|
||||
expect(normalizeGoogleModelId("gemini-3.1-flash-lite")).toBe("gemini-3.1-flash-lite");
|
||||
expect(normalizeGoogleModelId("gemini-3.1-flash-lite-preview")).toBe("gemini-3.1-flash-lite");
|
||||
});
|
||||
|
||||
it("maps the old Gemma 4 26B shorthand to Google's canonical API id", () => {
|
||||
expect(normalizeGoogleModelId("gemma-4-26b")).toBe("gemma-4-26b-a4b-it");
|
||||
expect(normalizeGoogleModelId("google/gemma-4-26b")).toBe("google/gemma-4-26b-a4b-it");
|
||||
});
|
||||
});
|
||||
46
extensions/google/model-id.ts
Normal file
46
extensions/google/model-id.ts
Normal file
@@ -0,0 +1,46 @@
|
||||
// Google plugin module implements model id behavior.
|
||||
const ANTIGRAVITY_BARE_PRO_IDS = new Set(["gemini-3-pro", "gemini-3.1-pro", "gemini-3-1-pro"]);
|
||||
const GOOGLE_PROVIDER_PREFIX = "google/";
|
||||
|
||||
export function stripGoogleProviderPrefix(id: string): string {
|
||||
return id.startsWith(GOOGLE_PROVIDER_PREFIX) ? id.slice(GOOGLE_PROVIDER_PREFIX.length) : id;
|
||||
}
|
||||
|
||||
export function normalizeGoogleModelId(id: string): string {
|
||||
if (id.startsWith(GOOGLE_PROVIDER_PREFIX)) {
|
||||
const modelId = stripGoogleProviderPrefix(id);
|
||||
const normalizedModelId = normalizeGoogleModelId(modelId);
|
||||
return normalizedModelId === modelId ? id : `${GOOGLE_PROVIDER_PREFIX}${normalizedModelId}`;
|
||||
}
|
||||
if (id === "gemini-3-pro" || id === "gemini-3-pro-preview") {
|
||||
return "gemini-3.1-pro-preview";
|
||||
}
|
||||
if (id === "gemini-3-flash") {
|
||||
return "gemini-3-flash-preview";
|
||||
}
|
||||
// Google exposes Gemini 3.1 Pro in the Gemini API as the preview-suffixed id.
|
||||
// Keep the bare form as a user convenience alias, not as a canonical API id.
|
||||
if (id === "gemini-3.1-pro") {
|
||||
return "gemini-3.1-pro-preview";
|
||||
}
|
||||
// Gemini 3.1 Flash Lite graduated to GA on 2026-05-07; the -preview
|
||||
// endpoint is deprecated (shutdown 2026-05-25). Map old preview name
|
||||
// to the stable GA id.
|
||||
if (id === "gemini-3.1-flash-lite-preview") {
|
||||
return "gemini-3.1-flash-lite";
|
||||
}
|
||||
if (id === "gemini-3.1-flash" || id === "gemini-3.1-flash-preview") {
|
||||
return "gemini-3-flash-preview";
|
||||
}
|
||||
if (id === "gemma-4-26b") {
|
||||
return "gemma-4-26b-a4b-it";
|
||||
}
|
||||
return id;
|
||||
}
|
||||
|
||||
export function normalizeAntigravityModelId(id: string): string {
|
||||
if (ANTIGRAVITY_BARE_PRO_IDS.has(id)) {
|
||||
return `${id}-low`;
|
||||
}
|
||||
return id;
|
||||
}
|
||||
282
extensions/google/music-generation-provider.test.ts
Normal file
282
extensions/google/music-generation-provider.test.ts
Normal file
@@ -0,0 +1,282 @@
|
||||
// Google tests cover music generation provider plugin behavior.
|
||||
import { afterAll, afterEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
const { createGoogleGenAIMock, generateContentMock } = vi.hoisted(() => {
|
||||
const generateContentMockLocal = vi.fn();
|
||||
const createGoogleGenAIMockLocal = vi.fn(() => {
|
||||
return {
|
||||
models: {
|
||||
generateContent: generateContentMockLocal,
|
||||
},
|
||||
};
|
||||
});
|
||||
return {
|
||||
createGoogleGenAIMock: createGoogleGenAIMockLocal,
|
||||
generateContentMock: generateContentMockLocal,
|
||||
};
|
||||
});
|
||||
|
||||
vi.mock("./google-genai-runtime.js", () => ({
|
||||
createGoogleGenAI: createGoogleGenAIMock,
|
||||
}));
|
||||
|
||||
import * as providerAuthRuntime from "openclaw/plugin-sdk/provider-auth-runtime";
|
||||
import { expectExplicitMusicGenerationCapabilities } from "openclaw/plugin-sdk/provider-test-contracts";
|
||||
import { buildGoogleMusicGenerationProvider } from "./music-generation-provider.js";
|
||||
|
||||
type GoogleGenAIConfig = {
|
||||
apiKey?: string;
|
||||
httpOptions?: {
|
||||
baseUrl?: string;
|
||||
};
|
||||
};
|
||||
|
||||
type GenerateContentRequest = {
|
||||
model?: string;
|
||||
config?: unknown;
|
||||
};
|
||||
|
||||
function lastGoogleGenAIConfig(): GoogleGenAIConfig {
|
||||
const calls = createGoogleGenAIMock.mock.calls as unknown[][];
|
||||
const config = calls.at(-1)?.[0];
|
||||
if (!config) {
|
||||
throw new Error("Expected GoogleGenAI config");
|
||||
}
|
||||
return config as GoogleGenAIConfig;
|
||||
}
|
||||
|
||||
function firstGenerateContentRequest(): GenerateContentRequest {
|
||||
const calls = generateContentMock.mock.calls as unknown[][];
|
||||
const request = calls[0]?.[0];
|
||||
if (!request) {
|
||||
throw new Error("Expected generateContent request");
|
||||
}
|
||||
return request as GenerateContentRequest;
|
||||
}
|
||||
|
||||
describe("google music generation provider", () => {
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
generateContentMock.mockReset();
|
||||
createGoogleGenAIMock.mockClear();
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
vi.doUnmock("./google-genai-runtime.js");
|
||||
vi.resetModules();
|
||||
});
|
||||
|
||||
it("declares explicit mode capabilities", () => {
|
||||
expectExplicitMusicGenerationCapabilities(buildGoogleMusicGenerationProvider());
|
||||
});
|
||||
|
||||
it("submits generation and returns inline audio bytes plus lyrics", async () => {
|
||||
vi.spyOn(providerAuthRuntime, "resolveApiKeyForProvider").mockResolvedValue({
|
||||
apiKey: "google-key",
|
||||
source: "env",
|
||||
mode: "api-key",
|
||||
});
|
||||
generateContentMock.mockResolvedValue({
|
||||
candidates: [
|
||||
{
|
||||
content: {
|
||||
parts: [
|
||||
{ text: "wake the city up" },
|
||||
{
|
||||
inlineData: {
|
||||
data: Buffer.from("mp3-bytes").toString("base64"),
|
||||
mimeType: "audio/mpeg",
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
const provider = buildGoogleMusicGenerationProvider();
|
||||
const result = await provider.generateMusic({
|
||||
provider: "google",
|
||||
model: "lyria-3-clip-preview",
|
||||
prompt: "upbeat synthpop anthem",
|
||||
cfg: {},
|
||||
instrumental: true,
|
||||
});
|
||||
|
||||
const generateRequest = firstGenerateContentRequest();
|
||||
expect(generateRequest.model).toBe("lyria-3-clip-preview");
|
||||
expect(generateRequest.config).toEqual({
|
||||
responseModalities: ["AUDIO", "TEXT"],
|
||||
});
|
||||
expect(result.tracks).toHaveLength(1);
|
||||
expect(result.tracks[0]?.mimeType).toBe("audio/mpeg");
|
||||
expect(result.lyrics).toEqual(["wake the city up"]);
|
||||
expect(lastGoogleGenAIConfig().apiKey).toBe("google-key");
|
||||
});
|
||||
|
||||
it("strips /v1beta suffix from configured baseUrl before passing to GoogleGenAI SDK", async () => {
|
||||
vi.spyOn(providerAuthRuntime, "resolveApiKeyForProvider").mockResolvedValue({
|
||||
apiKey: "google-key",
|
||||
source: "env",
|
||||
mode: "api-key",
|
||||
});
|
||||
generateContentMock.mockResolvedValue({
|
||||
candidates: [
|
||||
{
|
||||
content: {
|
||||
parts: [
|
||||
{
|
||||
inlineData: {
|
||||
data: Buffer.from("mp3-bytes").toString("base64"),
|
||||
mimeType: "audio/mpeg",
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
const provider = buildGoogleMusicGenerationProvider();
|
||||
await provider.generateMusic({
|
||||
provider: "google",
|
||||
model: "lyria-3-clip-preview",
|
||||
prompt: "ambient ocean",
|
||||
cfg: {
|
||||
models: {
|
||||
providers: {
|
||||
google: { baseUrl: "https://generativelanguage.googleapis.com/v1beta", models: [] },
|
||||
},
|
||||
},
|
||||
},
|
||||
instrumental: true,
|
||||
});
|
||||
|
||||
expect(lastGoogleGenAIConfig().httpOptions?.baseUrl).toBe(
|
||||
"https://generativelanguage.googleapis.com",
|
||||
);
|
||||
});
|
||||
|
||||
it("does NOT strip /v1beta when it appears mid-path (end-anchor proof)", async () => {
|
||||
vi.spyOn(providerAuthRuntime, "resolveApiKeyForProvider").mockResolvedValue({
|
||||
apiKey: "google-key",
|
||||
source: "env",
|
||||
mode: "api-key",
|
||||
});
|
||||
generateContentMock.mockResolvedValue({
|
||||
candidates: [
|
||||
{
|
||||
content: {
|
||||
parts: [
|
||||
{ inlineData: { data: Buffer.from("x").toString("base64"), mimeType: "audio/mpeg" } },
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
const provider = buildGoogleMusicGenerationProvider();
|
||||
await provider.generateMusic({
|
||||
provider: "google",
|
||||
model: "lyria-3-clip-preview",
|
||||
prompt: "test",
|
||||
cfg: {
|
||||
models: {
|
||||
providers: { google: { baseUrl: "https://proxy.example.com/v1beta/route", models: [] } },
|
||||
},
|
||||
},
|
||||
instrumental: true,
|
||||
});
|
||||
|
||||
expect(lastGoogleGenAIConfig().httpOptions?.baseUrl).toBe(
|
||||
"https://proxy.example.com/v1beta/route",
|
||||
);
|
||||
});
|
||||
|
||||
it("passes baseUrl unchanged when no /v1beta suffix is present", async () => {
|
||||
vi.spyOn(providerAuthRuntime, "resolveApiKeyForProvider").mockResolvedValue({
|
||||
apiKey: "google-key",
|
||||
source: "env",
|
||||
mode: "api-key",
|
||||
});
|
||||
generateContentMock.mockResolvedValue({
|
||||
candidates: [
|
||||
{
|
||||
content: {
|
||||
parts: [
|
||||
{ inlineData: { data: Buffer.from("x").toString("base64"), mimeType: "audio/mpeg" } },
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
const provider = buildGoogleMusicGenerationProvider();
|
||||
await provider.generateMusic({
|
||||
provider: "google",
|
||||
model: "lyria-3-clip-preview",
|
||||
prompt: "test",
|
||||
cfg: {
|
||||
models: {
|
||||
providers: {
|
||||
google: { baseUrl: "https://generativelanguage.googleapis.com", models: [] },
|
||||
},
|
||||
},
|
||||
},
|
||||
instrumental: true,
|
||||
});
|
||||
|
||||
expect(lastGoogleGenAIConfig().httpOptions?.baseUrl).toBe(
|
||||
"https://generativelanguage.googleapis.com",
|
||||
);
|
||||
});
|
||||
|
||||
it("does not set baseUrl when none is configured", async () => {
|
||||
vi.spyOn(providerAuthRuntime, "resolveApiKeyForProvider").mockResolvedValue({
|
||||
apiKey: "google-key",
|
||||
source: "env",
|
||||
mode: "api-key",
|
||||
});
|
||||
generateContentMock.mockResolvedValue({
|
||||
candidates: [
|
||||
{
|
||||
content: {
|
||||
parts: [
|
||||
{ inlineData: { data: Buffer.from("x").toString("base64"), mimeType: "audio/mpeg" } },
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
const provider = buildGoogleMusicGenerationProvider();
|
||||
await provider.generateMusic({
|
||||
provider: "google",
|
||||
model: "lyria-3-clip-preview",
|
||||
prompt: "test",
|
||||
cfg: {},
|
||||
instrumental: true,
|
||||
});
|
||||
|
||||
expect(lastGoogleGenAIConfig().httpOptions?.baseUrl).toBeUndefined();
|
||||
});
|
||||
|
||||
it("rejects unsupported wav output on clip model", async () => {
|
||||
vi.spyOn(providerAuthRuntime, "resolveApiKeyForProvider").mockResolvedValue({
|
||||
apiKey: "google-key",
|
||||
source: "env",
|
||||
mode: "api-key",
|
||||
});
|
||||
const provider = buildGoogleMusicGenerationProvider();
|
||||
|
||||
await expect(
|
||||
provider.generateMusic({
|
||||
provider: "google",
|
||||
model: "lyria-3-clip-preview",
|
||||
prompt: "ambient ocean",
|
||||
cfg: {},
|
||||
format: "wav",
|
||||
}),
|
||||
).rejects.toThrow("supports mp3 output");
|
||||
});
|
||||
});
|
||||
177
extensions/google/music-generation-provider.ts
Normal file
177
extensions/google/music-generation-provider.ts
Normal file
@@ -0,0 +1,177 @@
|
||||
// Google provider module implements model/runtime integration.
|
||||
import { extensionForMime } from "openclaw/plugin-sdk/media-mime";
|
||||
import type {
|
||||
GeneratedMusicAsset,
|
||||
MusicGenerationProvider,
|
||||
MusicGenerationRequest,
|
||||
} from "openclaw/plugin-sdk/music-generation";
|
||||
import { resolveApiKeyForProvider } from "openclaw/plugin-sdk/provider-auth-runtime";
|
||||
import { normalizeOptionalString } from "openclaw/plugin-sdk/string-coerce-runtime";
|
||||
import { resolveGoogleGenerativeAiApiOrigin } from "./api.js";
|
||||
import {
|
||||
createGoogleMusicGenerationProviderMetadata,
|
||||
DEFAULT_GOOGLE_MUSIC_MODEL,
|
||||
GOOGLE_MAX_INPUT_IMAGES,
|
||||
GOOGLE_PRO_MUSIC_MODEL,
|
||||
} from "./generation-provider-metadata.js";
|
||||
import { createGoogleGenAI } from "./google-genai-runtime.js";
|
||||
|
||||
const DEFAULT_TIMEOUT_MS = 180_000;
|
||||
|
||||
type GoogleInlineDataPart = {
|
||||
mimeType?: string;
|
||||
mime_type?: string;
|
||||
data?: string;
|
||||
};
|
||||
|
||||
type GoogleGenerateMusicResponse = {
|
||||
candidates?: Array<{
|
||||
content?: {
|
||||
parts?: Array<{
|
||||
text?: string;
|
||||
inlineData?: GoogleInlineDataPart;
|
||||
inline_data?: GoogleInlineDataPart;
|
||||
}>;
|
||||
};
|
||||
}>;
|
||||
};
|
||||
|
||||
function resolveConfiguredGoogleMusicBaseUrl(req: MusicGenerationRequest): string | undefined {
|
||||
const configured = normalizeOptionalString(req.cfg?.models?.providers?.google?.baseUrl);
|
||||
return configured ? resolveGoogleGenerativeAiApiOrigin(configured) : undefined;
|
||||
}
|
||||
|
||||
function buildMusicPrompt(req: MusicGenerationRequest): string {
|
||||
const parts = [req.prompt.trim()];
|
||||
const lyrics = normalizeOptionalString(req.lyrics);
|
||||
if (req.instrumental === true) {
|
||||
parts.push("Instrumental only. No vocals, no sung lyrics, no spoken word.");
|
||||
}
|
||||
if (lyrics) {
|
||||
parts.push(`Lyrics:\n${lyrics}`);
|
||||
}
|
||||
return parts.join("\n\n");
|
||||
}
|
||||
|
||||
function resolveSupportedFormats(model: string): readonly string[] {
|
||||
return model === GOOGLE_PRO_MUSIC_MODEL ? ["mp3", "wav"] : ["mp3"];
|
||||
}
|
||||
|
||||
function resolveTrackFileName(params: { index: number; mimeType: string; model: string }): string {
|
||||
const ext =
|
||||
extensionForMime(params.mimeType)?.replace(/^\./u, "") ||
|
||||
(params.model === GOOGLE_PRO_MUSIC_MODEL ? "wav" : "mp3");
|
||||
return `track-${params.index + 1}.${ext}`;
|
||||
}
|
||||
|
||||
function extractTracks(params: { payload: GoogleGenerateMusicResponse; model: string }): {
|
||||
tracks: GeneratedMusicAsset[];
|
||||
lyrics: string[];
|
||||
} {
|
||||
const lyrics: string[] = [];
|
||||
const tracks: GeneratedMusicAsset[] = [];
|
||||
for (const candidate of params.payload.candidates ?? []) {
|
||||
for (const part of candidate.content?.parts ?? []) {
|
||||
const text = normalizeOptionalString(part.text);
|
||||
if (text) {
|
||||
lyrics.push(text);
|
||||
continue;
|
||||
}
|
||||
const inline = part.inlineData ?? part.inline_data;
|
||||
const data = normalizeOptionalString(inline?.data);
|
||||
if (!data) {
|
||||
continue;
|
||||
}
|
||||
const mimeType =
|
||||
normalizeOptionalString(inline?.mimeType) ||
|
||||
normalizeOptionalString(inline?.mime_type) ||
|
||||
"audio/mpeg";
|
||||
tracks.push({
|
||||
buffer: Buffer.from(data, "base64"),
|
||||
mimeType,
|
||||
fileName: resolveTrackFileName({
|
||||
index: tracks.length,
|
||||
mimeType,
|
||||
model: params.model,
|
||||
}),
|
||||
});
|
||||
}
|
||||
}
|
||||
return { tracks, lyrics };
|
||||
}
|
||||
|
||||
export function buildGoogleMusicGenerationProvider(): MusicGenerationProvider {
|
||||
return {
|
||||
...createGoogleMusicGenerationProviderMetadata(),
|
||||
async generateMusic(req) {
|
||||
if ((req.inputImages?.length ?? 0) > GOOGLE_MAX_INPUT_IMAGES) {
|
||||
throw new Error(
|
||||
`Google music generation supports at most ${GOOGLE_MAX_INPUT_IMAGES} reference images.`,
|
||||
);
|
||||
}
|
||||
const auth = await resolveApiKeyForProvider({
|
||||
provider: "google",
|
||||
cfg: req.cfg,
|
||||
agentDir: req.agentDir,
|
||||
store: req.authStore,
|
||||
});
|
||||
if (!auth.apiKey) {
|
||||
throw new Error("Google API key missing");
|
||||
}
|
||||
|
||||
const model = normalizeOptionalString(req.model) || DEFAULT_GOOGLE_MUSIC_MODEL;
|
||||
if (req.format) {
|
||||
const supportedFormats = resolveSupportedFormats(model);
|
||||
if (!supportedFormats.includes(req.format)) {
|
||||
throw new Error(
|
||||
`Google music generation model ${model} supports ${supportedFormats.join(", ")} output.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const client = createGoogleGenAI({
|
||||
apiKey: auth.apiKey,
|
||||
httpOptions: {
|
||||
...(resolveConfiguredGoogleMusicBaseUrl(req)
|
||||
? { baseUrl: resolveConfiguredGoogleMusicBaseUrl(req) }
|
||||
: {}),
|
||||
timeout: req.timeoutMs ?? DEFAULT_TIMEOUT_MS,
|
||||
},
|
||||
});
|
||||
const response = (await client.models.generateContent({
|
||||
model,
|
||||
contents: [
|
||||
{ text: buildMusicPrompt(req) },
|
||||
...(req.inputImages ?? []).map((image) => ({
|
||||
inlineData: {
|
||||
mimeType: normalizeOptionalString(image.mimeType) || "image/png",
|
||||
data: image.buffer?.toString("base64") ?? "",
|
||||
},
|
||||
})),
|
||||
],
|
||||
config: {
|
||||
responseModalities: ["AUDIO", "TEXT"],
|
||||
},
|
||||
})) as GoogleGenerateMusicResponse;
|
||||
|
||||
const { tracks, lyrics } = extractTracks({
|
||||
payload: response,
|
||||
model,
|
||||
});
|
||||
if (tracks.length === 0) {
|
||||
throw new Error("Google music generation response missing audio data");
|
||||
}
|
||||
return {
|
||||
tracks,
|
||||
...(lyrics.length > 0 ? { lyrics } : {}),
|
||||
model,
|
||||
metadata: {
|
||||
inputImageCount: req.inputImages?.length ?? 0,
|
||||
instrumental: req.instrumental === true,
|
||||
...(normalizeOptionalString(req.lyrics) ? { requestedLyrics: true } : {}),
|
||||
...(req.format ? { requestedFormat: req.format } : {}),
|
||||
},
|
||||
};
|
||||
},
|
||||
};
|
||||
}
|
||||
40
extensions/google/oauth-token-shared.test.ts
Normal file
40
extensions/google/oauth-token-shared.test.ts
Normal file
@@ -0,0 +1,40 @@
|
||||
// Google tests cover oauth token shared plugin behavior.
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
formatGoogleOauthApiKey,
|
||||
parseGoogleOauthApiKey,
|
||||
parseGoogleUsageToken,
|
||||
} from "./oauth-token-shared.js";
|
||||
|
||||
describe("google oauth token helpers", () => {
|
||||
it("formats oauth credentials with project-aware payloads", () => {
|
||||
expect(
|
||||
formatGoogleOauthApiKey({
|
||||
type: "oauth",
|
||||
access: "token-123",
|
||||
projectId: "project-abc",
|
||||
}),
|
||||
).toBe(JSON.stringify({ token: "token-123", projectId: "project-abc" }));
|
||||
});
|
||||
|
||||
it("returns an empty string for non-oauth credentials", () => {
|
||||
expect(formatGoogleOauthApiKey({ type: "token", access: "token-123" })).toBe("");
|
||||
});
|
||||
|
||||
it("parses project-aware oauth payloads for usage auth", () => {
|
||||
expect(parseGoogleUsageToken(JSON.stringify({ token: "usage-token" }))).toBe("usage-token");
|
||||
});
|
||||
|
||||
it("parses structured oauth payload fields", () => {
|
||||
expect(
|
||||
parseGoogleOauthApiKey(JSON.stringify({ token: "usage-token", projectId: "proj-1" })),
|
||||
).toEqual({
|
||||
token: "usage-token",
|
||||
projectId: "proj-1",
|
||||
});
|
||||
});
|
||||
|
||||
it("falls back to the raw token when the payload is not JSON", () => {
|
||||
expect(parseGoogleUsageToken("raw-token")).toBe("raw-token");
|
||||
});
|
||||
});
|
||||
43
extensions/google/oauth-token-shared.ts
Normal file
43
extensions/google/oauth-token-shared.ts
Normal file
@@ -0,0 +1,43 @@
|
||||
// Google plugin module implements oauth token shared behavior.
|
||||
import { readStringValue } from "openclaw/plugin-sdk/string-coerce-runtime";
|
||||
|
||||
type GoogleOauthApiKeyCredential = {
|
||||
type?: string;
|
||||
access?: string;
|
||||
projectId?: string;
|
||||
};
|
||||
|
||||
export function parseGoogleOauthApiKey(apiKey: string): {
|
||||
token?: string;
|
||||
projectId?: string;
|
||||
} | null {
|
||||
try {
|
||||
const parsed = JSON.parse(apiKey) as { token?: unknown; projectId?: unknown };
|
||||
return {
|
||||
token: readStringValue(parsed.token),
|
||||
projectId: readStringValue(parsed.projectId),
|
||||
};
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export function formatGoogleOauthApiKey(cred: GoogleOauthApiKeyCredential): string {
|
||||
if (cred.type !== "oauth" || typeof cred.access !== "string" || !cred.access.trim()) {
|
||||
return "";
|
||||
}
|
||||
return JSON.stringify({
|
||||
token: cred.access,
|
||||
projectId: cred.projectId,
|
||||
});
|
||||
}
|
||||
|
||||
export function parseGoogleUsageToken(apiKey: string): string {
|
||||
const parsed = parseGoogleOauthApiKey(apiKey);
|
||||
if (parsed?.token) {
|
||||
return parsed.token;
|
||||
}
|
||||
|
||||
// Keep the raw token when the stored credential is not a project-aware JSON payload.
|
||||
return apiKey;
|
||||
}
|
||||
364
extensions/google/oauth.credentials.ts
Normal file
364
extensions/google/oauth.credentials.ts
Normal file
@@ -0,0 +1,364 @@
|
||||
// Google plugin module implements oauth.credentials behavior.
|
||||
import { existsSync, readFileSync, readdirSync, realpathSync } from "node:fs";
|
||||
import type { Dirent } from "node:fs";
|
||||
import { delimiter, dirname, join } from "node:path";
|
||||
import { lowercasePreservingWhitespace } from "openclaw/plugin-sdk/string-coerce-runtime";
|
||||
import { CLIENT_ID_KEYS, CLIENT_SECRET_KEYS } from "./oauth.shared.js";
|
||||
|
||||
type CredentialFs = {
|
||||
existsSync: (path: Parameters<typeof existsSync>[0]) => ReturnType<typeof existsSync>;
|
||||
readFileSync: (path: Parameters<typeof readFileSync>[0], encoding: "utf8") => string;
|
||||
realpathSync: (path: Parameters<typeof realpathSync>[0]) => string;
|
||||
readdirSync: (
|
||||
path: Parameters<typeof readdirSync>[0],
|
||||
options: { withFileTypes: true },
|
||||
) => Dirent[];
|
||||
};
|
||||
|
||||
const defaultFs: CredentialFs = {
|
||||
existsSync,
|
||||
readFileSync,
|
||||
realpathSync,
|
||||
readdirSync,
|
||||
};
|
||||
|
||||
let credentialFs: CredentialFs = defaultFs;
|
||||
const GEMINI_CLI_TREE_SEARCH_DEPTH = 10;
|
||||
|
||||
type GeminiCliCredentialExtractDiagnostics = {
|
||||
searchedPaths: string[];
|
||||
recursiveSearchRoots: string[];
|
||||
parseFailures: string[];
|
||||
readErrors: string[];
|
||||
};
|
||||
|
||||
function resolveEnv(keys: string[]): string | undefined {
|
||||
for (const key of keys) {
|
||||
const value = process.env[key]?.trim();
|
||||
if (value) {
|
||||
return value;
|
||||
}
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
let cachedGeminiCliCredentials: { clientId: string; clientSecret: string } | null = null;
|
||||
let geminiCliCredentialExtractError: string | null = null;
|
||||
|
||||
export function clearCredentialsCache(): void {
|
||||
cachedGeminiCliCredentials = null;
|
||||
geminiCliCredentialExtractError = null;
|
||||
}
|
||||
|
||||
export function setOAuthCredentialsFsForTest(overrides?: Partial<CredentialFs>): void {
|
||||
credentialFs = overrides ? { ...defaultFs, ...overrides } : defaultFs;
|
||||
}
|
||||
|
||||
export function extractGeminiCliCredentials(): { clientId: string; clientSecret: string } | null {
|
||||
if (cachedGeminiCliCredentials) {
|
||||
return cachedGeminiCliCredentials;
|
||||
}
|
||||
|
||||
geminiCliCredentialExtractError = null;
|
||||
const diagnostics: GeminiCliCredentialExtractDiagnostics = {
|
||||
searchedPaths: [],
|
||||
recursiveSearchRoots: [],
|
||||
parseFailures: [],
|
||||
readErrors: [],
|
||||
};
|
||||
|
||||
try {
|
||||
const geminiPath = findInPath("gemini");
|
||||
if (!geminiPath) {
|
||||
geminiCliCredentialExtractError =
|
||||
"Gemini CLI binary was not found in PATH during OAuth credential extraction.";
|
||||
return null;
|
||||
}
|
||||
|
||||
const resolvedPath = credentialFs.realpathSync(geminiPath);
|
||||
const geminiCliDirs = resolveGeminiCliDirs(geminiPath, resolvedPath);
|
||||
|
||||
for (const geminiCliDir of geminiCliDirs) {
|
||||
const directCredentials = readGeminiCliCredentialsFromKnownPaths(geminiCliDir, diagnostics);
|
||||
if (directCredentials) {
|
||||
cachedGeminiCliCredentials = directCredentials;
|
||||
return directCredentials;
|
||||
}
|
||||
|
||||
const bundledCredentials = readGeminiCliCredentialsFromBundle(geminiCliDir, diagnostics);
|
||||
if (bundledCredentials) {
|
||||
cachedGeminiCliCredentials = bundledCredentials;
|
||||
return bundledCredentials;
|
||||
}
|
||||
|
||||
diagnostics.recursiveSearchRoots.push(geminiCliDir);
|
||||
const discoveredCredentials = findGeminiCliCredentialsInTree(
|
||||
geminiCliDir,
|
||||
GEMINI_CLI_TREE_SEARCH_DEPTH,
|
||||
diagnostics,
|
||||
);
|
||||
if (discoveredCredentials) {
|
||||
cachedGeminiCliCredentials = discoveredCredentials;
|
||||
return discoveredCredentials;
|
||||
}
|
||||
}
|
||||
geminiCliCredentialExtractError = formatGeminiCliCredentialExtractError({
|
||||
geminiPath,
|
||||
resolvedPath,
|
||||
diagnostics,
|
||||
});
|
||||
} catch (error) {
|
||||
geminiCliCredentialExtractError = `Unexpected error while extracting Gemini CLI OAuth credentials: ${formatError(error)}`;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function formatGeminiCliCredentialExtractError({
|
||||
geminiPath,
|
||||
resolvedPath,
|
||||
diagnostics,
|
||||
}: {
|
||||
geminiPath: string;
|
||||
resolvedPath: string;
|
||||
diagnostics: GeminiCliCredentialExtractDiagnostics;
|
||||
}): string {
|
||||
const prefix = [
|
||||
"Found Gemini CLI in PATH, but could not extract OAuth credentials.",
|
||||
`geminiPath=${geminiPath}`,
|
||||
`resolvedPath=${resolvedPath}`,
|
||||
];
|
||||
|
||||
if (diagnostics.parseFailures.length > 0) {
|
||||
return [
|
||||
...prefix,
|
||||
"Candidate credential files did not contain a parseable OAuth client id/secret.",
|
||||
`candidates=${diagnostics.parseFailures.join(", ")}`,
|
||||
].join(" ");
|
||||
}
|
||||
|
||||
if (diagnostics.readErrors.length > 0) {
|
||||
return [
|
||||
...prefix,
|
||||
"Unexpected errors occurred while reading candidate credential files/directories.",
|
||||
`errors=${diagnostics.readErrors.join(", ")}`,
|
||||
].join(" ");
|
||||
}
|
||||
|
||||
return [
|
||||
...prefix,
|
||||
"Could not locate oauth2.js or bundled credential source.",
|
||||
`searched=${diagnostics.searchedPaths.join(", ") || "(none)"}`,
|
||||
`recursiveSearchRoots=${diagnostics.recursiveSearchRoots.join(", ") || "(none)"}`,
|
||||
`recursiveSearchDepth=${GEMINI_CLI_TREE_SEARCH_DEPTH}`,
|
||||
].join(" ");
|
||||
}
|
||||
|
||||
function formatError(error: unknown): string {
|
||||
return error instanceof Error ? error.message : String(error);
|
||||
}
|
||||
|
||||
function resolveGeminiCliDirs(geminiPath: string, resolvedPath: string): string[] {
|
||||
const binDir = dirname(geminiPath);
|
||||
const candidates = [
|
||||
dirname(dirname(resolvedPath)),
|
||||
join(dirname(resolvedPath), "node_modules", "@google", "gemini-cli"),
|
||||
join(binDir, "node_modules", "@google", "gemini-cli"),
|
||||
join(dirname(binDir), "node_modules", "@google", "gemini-cli"),
|
||||
join(dirname(binDir), "lib", "node_modules", "@google", "gemini-cli"),
|
||||
];
|
||||
|
||||
const deduped: string[] = [];
|
||||
const seen = new Set<string>();
|
||||
for (const candidate of candidates) {
|
||||
for (const searchDir of resolveGeminiCliSearchDirs(candidate)) {
|
||||
const key =
|
||||
process.platform === "win32"
|
||||
? lowercasePreservingWhitespace(searchDir.replace(/\\/g, "/"))
|
||||
: searchDir;
|
||||
if (seen.has(key)) {
|
||||
continue;
|
||||
}
|
||||
seen.add(key);
|
||||
deduped.push(searchDir);
|
||||
}
|
||||
}
|
||||
return deduped;
|
||||
}
|
||||
|
||||
function resolveGeminiCliSearchDirs(candidate: string): string[] {
|
||||
const searchDirs = [
|
||||
candidate,
|
||||
join(candidate, "node_modules", "@google", "gemini-cli"),
|
||||
join(candidate, "lib", "node_modules", "@google", "gemini-cli"),
|
||||
];
|
||||
return searchDirs.filter(looksLikeGeminiCliDir);
|
||||
}
|
||||
|
||||
function looksLikeGeminiCliDir(candidate: string): boolean {
|
||||
return (
|
||||
credentialFs.existsSync(join(candidate, "package.json")) ||
|
||||
credentialFs.existsSync(join(candidate, "node_modules", "@google", "gemini-cli-core"))
|
||||
);
|
||||
}
|
||||
|
||||
function findInPath(name: string): string | null {
|
||||
const exts = process.platform === "win32" ? [".cmd", ".bat", ".exe", ""] : [""];
|
||||
for (const dir of (process.env.PATH ?? "").split(delimiter)) {
|
||||
for (const ext of exts) {
|
||||
const path = join(dir, name + ext);
|
||||
if (credentialFs.existsSync(path)) {
|
||||
return path;
|
||||
}
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function readGeminiCliCredentialsFile(
|
||||
path: string,
|
||||
diagnostics: GeminiCliCredentialExtractDiagnostics,
|
||||
): { clientId: string; clientSecret: string } | null {
|
||||
try {
|
||||
const credentials = parseGeminiCliCredentials(credentialFs.readFileSync(path, "utf8"));
|
||||
if (!credentials) {
|
||||
diagnostics.parseFailures.push(path);
|
||||
}
|
||||
return credentials;
|
||||
} catch (error) {
|
||||
diagnostics.readErrors.push(`${path}: ${formatError(error)}`);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function parseGeminiCliCredentials(
|
||||
content: string,
|
||||
): { clientId: string; clientSecret: string } | null {
|
||||
const clientId =
|
||||
content.match(/OAUTH_CLIENT_ID\s*=\s*["']([^"']+)["']/)?.[1] ??
|
||||
content.match(/(\d+-[a-z0-9]+\.apps\.googleusercontent\.com)/)?.[1];
|
||||
const clientSecret =
|
||||
content.match(/OAUTH_CLIENT_SECRET\s*=\s*["']([^"']+)["']/)?.[1] ??
|
||||
content.match(/(GOCSPX-[A-Za-z0-9_-]+)/)?.[1];
|
||||
if (!clientId || !clientSecret) {
|
||||
return null;
|
||||
}
|
||||
return { clientId, clientSecret };
|
||||
}
|
||||
|
||||
function readGeminiCliCredentialsFromKnownPaths(
|
||||
geminiCliDir: string,
|
||||
diagnostics: GeminiCliCredentialExtractDiagnostics,
|
||||
): { clientId: string; clientSecret: string } | null {
|
||||
const searchPaths = [
|
||||
join(
|
||||
geminiCliDir,
|
||||
"node_modules",
|
||||
"@google",
|
||||
"gemini-cli-core",
|
||||
"dist",
|
||||
"src",
|
||||
"code_assist",
|
||||
"oauth2.js",
|
||||
),
|
||||
join(
|
||||
geminiCliDir,
|
||||
"node_modules",
|
||||
"@google",
|
||||
"gemini-cli-core",
|
||||
"dist",
|
||||
"code_assist",
|
||||
"oauth2.js",
|
||||
),
|
||||
];
|
||||
diagnostics.searchedPaths.push(...searchPaths);
|
||||
|
||||
for (const path of searchPaths) {
|
||||
if (!credentialFs.existsSync(path)) {
|
||||
continue;
|
||||
}
|
||||
const credentials = readGeminiCliCredentialsFile(path, diagnostics);
|
||||
if (credentials) {
|
||||
return credentials;
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
function readGeminiCliCredentialsFromBundle(
|
||||
geminiCliDir: string,
|
||||
diagnostics: GeminiCliCredentialExtractDiagnostics,
|
||||
): { clientId: string; clientSecret: string } | null {
|
||||
const bundleDir = join(geminiCliDir, "bundle");
|
||||
if (!credentialFs.existsSync(bundleDir)) {
|
||||
return null;
|
||||
}
|
||||
|
||||
try {
|
||||
for (const entry of credentialFs.readdirSync(bundleDir, { withFileTypes: true })) {
|
||||
if (!entry.isFile() || !entry.name.endsWith(".js")) {
|
||||
continue;
|
||||
}
|
||||
const credentials = readGeminiCliCredentialsFile(join(bundleDir, entry.name), diagnostics);
|
||||
if (credentials) {
|
||||
return credentials;
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
diagnostics.readErrors.push(`${bundleDir}: ${formatError(error)}`);
|
||||
// Preserve the read error for diagnostics and fall back to the recursive search.
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
function findGeminiCliCredentialsInTree(
|
||||
dir: string,
|
||||
depth: number,
|
||||
diagnostics: GeminiCliCredentialExtractDiagnostics,
|
||||
): { clientId: string; clientSecret: string } | null {
|
||||
if (depth <= 0) {
|
||||
return null;
|
||||
}
|
||||
try {
|
||||
for (const entry of credentialFs.readdirSync(dir, { withFileTypes: true })) {
|
||||
const path = join(dir, entry.name);
|
||||
if (entry.isFile() && entry.name === "oauth2.js") {
|
||||
const credentials = readGeminiCliCredentialsFile(path, diagnostics);
|
||||
if (credentials) {
|
||||
return credentials;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (entry.isDirectory() && !entry.name.startsWith(".")) {
|
||||
const found = findGeminiCliCredentialsInTree(path, depth - 1, diagnostics);
|
||||
if (found) {
|
||||
return found;
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
diagnostics.readErrors.push(`${dir}: ${formatError(error)}`);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
export function resolveOAuthClientConfig(): { clientId: string; clientSecret?: string } {
|
||||
const envClientId = resolveEnv(CLIENT_ID_KEYS);
|
||||
const envClientSecret = resolveEnv(CLIENT_SECRET_KEYS);
|
||||
if (envClientId) {
|
||||
return { clientId: envClientId, clientSecret: envClientSecret };
|
||||
}
|
||||
|
||||
const extracted = extractGeminiCliCredentials();
|
||||
if (extracted) {
|
||||
return extracted;
|
||||
}
|
||||
|
||||
const detail = geminiCliCredentialExtractError
|
||||
? ` Details: ${geminiCliCredentialExtractError}`
|
||||
: "";
|
||||
throw new Error(
|
||||
`Gemini CLI not found. Install it first: brew install gemini-cli (or npm install -g @google/gemini-cli), or set GEMINI_CLI_OAUTH_CLIENT_ID.${detail}`,
|
||||
);
|
||||
}
|
||||
62
extensions/google/oauth.flow.ts
Normal file
62
extensions/google/oauth.flow.ts
Normal file
@@ -0,0 +1,62 @@
|
||||
// Google plugin module implements oauth.flow behavior.
|
||||
import { generateHexPkceVerifierChallenge } from "openclaw/plugin-sdk/provider-auth";
|
||||
import {
|
||||
generateOAuthState,
|
||||
parseOAuthCallbackInput,
|
||||
waitForLocalOAuthCallback,
|
||||
} from "openclaw/plugin-sdk/provider-auth-runtime";
|
||||
import { isWSL2Sync } from "openclaw/plugin-sdk/runtime-env";
|
||||
import { resolveOAuthClientConfig } from "./oauth.credentials.js";
|
||||
import { AUTH_URL, REDIRECT_URI, SCOPES } from "./oauth.shared.js";
|
||||
|
||||
export { generateOAuthState };
|
||||
|
||||
export function shouldUseManualOAuthFlow(isRemote: boolean): boolean {
|
||||
return isRemote || isWSL2Sync();
|
||||
}
|
||||
|
||||
export function generatePkce(): { verifier: string; challenge: string } {
|
||||
return generateHexPkceVerifierChallenge();
|
||||
}
|
||||
|
||||
export function buildAuthUrl(challenge: string, state: string): string {
|
||||
const { clientId } = resolveOAuthClientConfig();
|
||||
const params = new URLSearchParams({
|
||||
client_id: clientId,
|
||||
response_type: "code",
|
||||
redirect_uri: REDIRECT_URI,
|
||||
scope: SCOPES.join(" "),
|
||||
code_challenge: challenge,
|
||||
code_challenge_method: "S256",
|
||||
state,
|
||||
access_type: "offline",
|
||||
prompt: "consent",
|
||||
});
|
||||
return `${AUTH_URL}?${params.toString()}`;
|
||||
}
|
||||
|
||||
export function parseCallbackInput(
|
||||
input: string,
|
||||
): { code: string; state: string } | { error: string } {
|
||||
return parseOAuthCallbackInput(input, {
|
||||
missingState: "Missing 'state' parameter. Paste the full URL.",
|
||||
invalidInput: "Paste the full redirect URL, not just the code.",
|
||||
});
|
||||
}
|
||||
|
||||
export async function waitForLocalCallback(params: {
|
||||
expectedState: string;
|
||||
timeoutMs: number;
|
||||
onProgress?: (message: string) => void;
|
||||
}): Promise<{ code: string; state: string }> {
|
||||
return await waitForLocalOAuthCallback({
|
||||
expectedState: params.expectedState,
|
||||
timeoutMs: params.timeoutMs,
|
||||
port: 8085,
|
||||
callbackPath: "/oauth2callback",
|
||||
redirectUri: REDIRECT_URI,
|
||||
successTitle: "Gemini CLI OAuth complete",
|
||||
progressMessage: `Waiting for OAuth callback on ${REDIRECT_URI}…`,
|
||||
onProgress: params.onProgress,
|
||||
});
|
||||
}
|
||||
121
extensions/google/oauth.http.proxy.test.ts
Normal file
121
extensions/google/oauth.http.proxy.test.ts
Normal file
@@ -0,0 +1,121 @@
|
||||
// Google tests cover oauth.http proxy-mode selection for the Gemini CLI OAuth
|
||||
// token-exchange/identity calls (issue openclaw#46184).
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { TOKEN_URL } from "./oauth.shared.js";
|
||||
|
||||
const fetchWithSsrFGuardMock = vi.fn();
|
||||
|
||||
vi.mock("openclaw/plugin-sdk/ssrf-runtime", async () => {
|
||||
const actual = await vi.importActual<typeof import("openclaw/plugin-sdk/ssrf-runtime")>(
|
||||
"openclaw/plugin-sdk/ssrf-runtime",
|
||||
);
|
||||
return {
|
||||
...actual,
|
||||
fetchWithSsrFGuard: (params: unknown) => fetchWithSsrFGuardMock(params),
|
||||
};
|
||||
});
|
||||
|
||||
const { fetchWithTimeout } = await import("./oauth.http.js");
|
||||
|
||||
const PROXY_ENV_KEYS = [
|
||||
"HTTP_PROXY",
|
||||
"HTTPS_PROXY",
|
||||
"ALL_PROXY",
|
||||
"NO_PROXY",
|
||||
"http_proxy",
|
||||
"https_proxy",
|
||||
"all_proxy",
|
||||
"no_proxy",
|
||||
] as const;
|
||||
|
||||
const savedEnv = new Map<string, string | undefined>();
|
||||
|
||||
type ProxyEnvOverrides = {
|
||||
HTTP_PROXY?: string;
|
||||
HTTPS_PROXY?: string;
|
||||
ALL_PROXY?: string;
|
||||
NO_PROXY?: string;
|
||||
};
|
||||
|
||||
function setProxyEnv(values: ProxyEnvOverrides): void {
|
||||
for (const key of PROXY_ENV_KEYS) {
|
||||
delete process.env[key];
|
||||
}
|
||||
if (values.HTTP_PROXY !== undefined) {
|
||||
process.env.HTTP_PROXY = values.HTTP_PROXY;
|
||||
}
|
||||
if (values.HTTPS_PROXY !== undefined) {
|
||||
process.env.HTTPS_PROXY = values.HTTPS_PROXY;
|
||||
}
|
||||
if (values.ALL_PROXY !== undefined) {
|
||||
process.env.ALL_PROXY = values.ALL_PROXY;
|
||||
}
|
||||
if (values.NO_PROXY !== undefined) {
|
||||
process.env.NO_PROXY = values.NO_PROXY;
|
||||
}
|
||||
}
|
||||
|
||||
function lastGuardedOptions(): Record<string, unknown> {
|
||||
const call = fetchWithSsrFGuardMock.mock.calls.at(-1)?.[0];
|
||||
if (!call || typeof call !== "object") {
|
||||
throw new Error("Expected fetchWithSsrFGuard to be called");
|
||||
}
|
||||
return call as Record<string, unknown>;
|
||||
}
|
||||
|
||||
describe("oauth.http fetchWithTimeout proxy selection", () => {
|
||||
beforeEach(() => {
|
||||
for (const key of PROXY_ENV_KEYS) {
|
||||
savedEnv.set(key, process.env[key]);
|
||||
}
|
||||
fetchWithSsrFGuardMock.mockReset();
|
||||
fetchWithSsrFGuardMock.mockResolvedValue({
|
||||
response: new Response("{}", { status: 200 }),
|
||||
finalUrl: TOKEN_URL,
|
||||
release: async () => {},
|
||||
});
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
for (const [key, value] of savedEnv) {
|
||||
if (value === undefined) {
|
||||
delete process.env[key];
|
||||
} else {
|
||||
process.env[key] = value;
|
||||
}
|
||||
}
|
||||
savedEnv.clear();
|
||||
});
|
||||
|
||||
it("routes the Google token exchange through the env proxy when configured", async () => {
|
||||
setProxyEnv({ HTTPS_PROXY: "http://127.0.0.1:7897", HTTP_PROXY: "http://127.0.0.1:7897" });
|
||||
|
||||
await fetchWithTimeout(TOKEN_URL, { method: "POST", body: "grant_type=refresh_token" });
|
||||
|
||||
expect(lastGuardedOptions().mode).toBe("trusted_env_proxy");
|
||||
});
|
||||
|
||||
it("keeps the strict default when no proxy is configured", async () => {
|
||||
setProxyEnv({});
|
||||
|
||||
await fetchWithTimeout(TOKEN_URL, { method: "POST" });
|
||||
|
||||
expect(lastGuardedOptions().mode).toBeUndefined();
|
||||
});
|
||||
|
||||
it("keeps the strict default when NO_PROXY bypasses the target host", async () => {
|
||||
setProxyEnv({ HTTPS_PROXY: "http://127.0.0.1:7897", NO_PROXY: "googleapis.com" });
|
||||
|
||||
await fetchWithTimeout(TOKEN_URL, { method: "POST" });
|
||||
|
||||
expect(lastGuardedOptions().mode).toBeUndefined();
|
||||
});
|
||||
|
||||
it("keeps the strict default for ALL_PROXY-only environments", async () => {
|
||||
setProxyEnv({ ALL_PROXY: "http://127.0.0.1:7897" });
|
||||
|
||||
await fetchWithTimeout(TOKEN_URL, { method: "POST" });
|
||||
|
||||
expect(lastGuardedOptions().mode).toBeUndefined();
|
||||
});
|
||||
});
|
||||
172
extensions/google/oauth.http.test.ts
Normal file
172
extensions/google/oauth.http.test.ts
Normal file
@@ -0,0 +1,172 @@
|
||||
// Google tests cover oauth.http body-byte-cap for the Gemini CLI OAuth
|
||||
// token-exchange/identity calls.
|
||||
import http from "node:http";
|
||||
import type { AddressInfo } from "node:net";
|
||||
import { readResponseWithLimit } from "openclaw/plugin-sdk/response-limit-runtime";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { TOKEN_URL } from "./oauth.shared.js";
|
||||
|
||||
const fetchWithSsrFGuardMock = vi.fn();
|
||||
const releaseMock = vi.fn(async () => undefined);
|
||||
|
||||
vi.mock("openclaw/plugin-sdk/ssrf-runtime", async () => {
|
||||
const actual = await vi.importActual<typeof import("openclaw/plugin-sdk/ssrf-runtime")>(
|
||||
"openclaw/plugin-sdk/ssrf-runtime",
|
||||
);
|
||||
return {
|
||||
...actual,
|
||||
fetchWithSsrFGuard: (params: unknown) => fetchWithSsrFGuardMock(params),
|
||||
};
|
||||
});
|
||||
|
||||
const { fetchWithTimeout } = await import("./oauth.http.js");
|
||||
|
||||
describe("oauth.http fetchWithTimeout body byte cap", () => {
|
||||
beforeEach(() => {
|
||||
fetchWithSsrFGuardMock.mockReset();
|
||||
releaseMock.mockClear();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it("caps oversized response body at 16 MiB with labeled overflow error", async () => {
|
||||
// Build a Response with a body that exceeds the 16 MiB cap.
|
||||
// 1 MiB chunks × 18 chunks = 18 MiB queued; the bounded reader reads
|
||||
// up to the 16 MiB cap (16 chunks = 16777216 bytes) and one extra
|
||||
// chunk before throwing on overflow, so the labeled `size` is the
|
||||
// cap plus the trailing chunk: 16777216 + 1048576 = 17825792 bytes.
|
||||
const CHUNK = 1024 * 1024;
|
||||
let sent = 0;
|
||||
const body = new ReadableStream({
|
||||
pull(controller) {
|
||||
if (sent < 18) {
|
||||
controller.enqueue(new Uint8Array(CHUNK));
|
||||
sent++;
|
||||
} else {
|
||||
controller.close();
|
||||
}
|
||||
},
|
||||
});
|
||||
fetchWithSsrFGuardMock.mockResolvedValue({
|
||||
response: new Response(body, {
|
||||
status: 200,
|
||||
headers: { "content-type": "application/json" },
|
||||
}),
|
||||
finalUrl: TOKEN_URL,
|
||||
release: releaseMock,
|
||||
});
|
||||
|
||||
await expect(fetchWithTimeout(TOKEN_URL, { method: "POST" })).rejects.toThrow(
|
||||
/google HTTP fetch: body exceeds 16777216 bytes \(got 17825792\)/,
|
||||
);
|
||||
expect(releaseMock).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it("returns a Response for normal-size bodies", async () => {
|
||||
fetchWithSsrFGuardMock.mockResolvedValue({
|
||||
response: new Response('{"access_token":"abc","expires_in":3600}', {
|
||||
status: 200,
|
||||
headers: { "content-type": "application/json" },
|
||||
}),
|
||||
finalUrl: TOKEN_URL,
|
||||
release: releaseMock,
|
||||
});
|
||||
|
||||
const res = await fetchWithTimeout(TOKEN_URL, { method: "POST" });
|
||||
expect(res.status).toBe(200);
|
||||
expect(await res.json()).toEqual({ access_token: "abc", expires_in: 3600 });
|
||||
expect(releaseMock).toHaveBeenCalledOnce();
|
||||
});
|
||||
});
|
||||
|
||||
// Real-wire loopback proof. These tests bypass `fetchWithSsrFGuard` (which
|
||||
// blocks 127.0.0.1 by design) and exercise `readResponseWithLimit` directly
|
||||
// against a real `http.createServer` listener — the same helper that
|
||||
// `fetchWithTimeout` calls inside its try/finally block. Captured vitest
|
||||
// output for these two tests is the ClawSweeper "real behavior proof" required
|
||||
// before merge.
|
||||
describe("oauth.http bounded-read real wire proof (loopback http.createServer)", () => {
|
||||
it("caps an oversized body streamed chunked over real wire", async () => {
|
||||
const CHUNK = 1024 * 1024;
|
||||
const MAX = 16 * 1024 * 1024;
|
||||
const TOTAL = 18 * 1024 * 1024;
|
||||
const server = http.createServer((req, res) => {
|
||||
res.writeHead(200, { "content-type": "application/octet-stream" });
|
||||
let sent = 0;
|
||||
const tick = setInterval(() => {
|
||||
if (sent < 18) {
|
||||
res.write(Buffer.alloc(CHUNK));
|
||||
sent++;
|
||||
} else {
|
||||
clearInterval(tick);
|
||||
res.end();
|
||||
}
|
||||
}, 1);
|
||||
});
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
server.once("error", reject);
|
||||
server.listen(0, "127.0.0.1", () => resolve());
|
||||
});
|
||||
const port = (server.address() as AddressInfo).port;
|
||||
|
||||
let captured: Error | undefined;
|
||||
try {
|
||||
const response = await fetch(`http://127.0.0.1:${port}/`);
|
||||
// Wire framing merges TCP packets, so the exact reported size varies by
|
||||
// runtime. The stable invariant is that the cap fires after MAX.
|
||||
try {
|
||||
await readResponseWithLimit(response, MAX, {
|
||||
onOverflow: ({ size, maxBytes }) =>
|
||||
new Error(`real wire: body exceeds ${maxBytes} bytes (got ${size})`),
|
||||
});
|
||||
} catch (err) {
|
||||
captured = err as Error;
|
||||
}
|
||||
expect(captured).toBeInstanceOf(Error);
|
||||
const match = captured!.message.match(/real wire: body exceeds \d+ bytes \(got (\d+)\)/);
|
||||
expect(match).not.toBeNull();
|
||||
const got = Number(match![1]);
|
||||
expect(got).toBeGreaterThan(MAX);
|
||||
// Print to vitest stdout for PR-body real behavior proof capture.
|
||||
console.log(
|
||||
`[oauth.http loopback proof] oversized path: cap=${MAX} reported=${got} server_total=${TOTAL}`,
|
||||
);
|
||||
} finally {
|
||||
await new Promise<void>((resolve) => {
|
||||
server.close(() => resolve());
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
it("returns a Buffer for normal-size responses on real wire", async () => {
|
||||
const bodyText = '{"access_token":"loopback","expires_in":3600}';
|
||||
const server = http.createServer((req, res) => {
|
||||
res.writeHead(200, { "content-type": "application/json" });
|
||||
res.end(bodyText);
|
||||
});
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
server.once("error", reject);
|
||||
server.listen(0, "127.0.0.1", () => resolve());
|
||||
});
|
||||
const port = (server.address() as AddressInfo).port;
|
||||
|
||||
try {
|
||||
const response = await fetch(`http://127.0.0.1:${port}/`);
|
||||
const body = await readResponseWithLimit(response, 16 * 1024 * 1024, {
|
||||
onOverflow: ({ size, maxBytes }) =>
|
||||
new Error(`real wire: body exceeds ${maxBytes} bytes (got ${size})`),
|
||||
});
|
||||
expect(body.byteLength).toBe(Buffer.byteLength(bodyText, "utf8"));
|
||||
expect(new TextDecoder("utf-8").decode(body)).toBe(bodyText);
|
||||
console.log(
|
||||
`[oauth.http loopback proof] normal path: cap=16777216 returned=${body.byteLength} body=${JSON.stringify(new TextDecoder("utf-8").decode(body))}`,
|
||||
);
|
||||
} finally {
|
||||
await new Promise<void>((resolve) => {
|
||||
server.close(() => resolve());
|
||||
});
|
||||
}
|
||||
});
|
||||
});
|
||||
50
extensions/google/oauth.http.ts
Normal file
50
extensions/google/oauth.http.ts
Normal file
@@ -0,0 +1,50 @@
|
||||
// Google plugin module implements oauth.http behavior.
|
||||
import {
|
||||
shouldUseEnvHttpProxyForUrl,
|
||||
withTrustedEnvProxyGuardedFetchMode,
|
||||
} from "openclaw/plugin-sdk/fetch-runtime";
|
||||
import { readResponseWithLimit } from "openclaw/plugin-sdk/response-limit-runtime";
|
||||
import { fetchWithSsrFGuard } from "openclaw/plugin-sdk/ssrf-runtime";
|
||||
import { DEFAULT_FETCH_TIMEOUT_MS } from "./oauth.shared.js";
|
||||
|
||||
const GOOGLE_OAUTH_BODY_MAX_BYTES = 16 * 1024 * 1024;
|
||||
|
||||
export async function fetchWithTimeout(
|
||||
url: string,
|
||||
init: RequestInit,
|
||||
timeoutMs = DEFAULT_FETCH_TIMEOUT_MS,
|
||||
): Promise<Response> {
|
||||
const guardedOptions = { url, init, timeoutMs };
|
||||
const { response, release } = await fetchWithSsrFGuard(
|
||||
shouldUseEnvHttpProxyForUrl(url)
|
||||
? withTrustedEnvProxyGuardedFetchMode(guardedOptions)
|
||||
: guardedOptions,
|
||||
);
|
||||
try {
|
||||
// 16 MiB cap. A hostile or broken Google OAuth endpoint (or any
|
||||
// accounts.google.com mirror / enterprise proxy) cannot force the
|
||||
// runtime to buffer an unbounded body before the caller sees it.
|
||||
// Complements #97587, which caps at the call site — this is the
|
||||
// shared entry-point cap.
|
||||
const body = await readResponseWithLimit(response, GOOGLE_OAUTH_BODY_MAX_BYTES, {
|
||||
onOverflow: ({ size, maxBytes }) =>
|
||||
new Error(`google HTTP fetch: body exceeds ${maxBytes} bytes (got ${size})`),
|
||||
});
|
||||
// `readResponseWithLimit` returns a `Buffer` (Node Uint8Array view). The
|
||||
// global `Response` constructor accepts `BufferSource` (Uint8Array /
|
||||
// ArrayBuffer) as a body; cast through `BodyInit` because `Buffer.buffer`
|
||||
// is typed as `ArrayBufferLike` (could be `ArrayBuffer` or
|
||||
// `SharedArrayBuffer`), but the helper always returns a regular `Buffer`
|
||||
// backed by an `ArrayBuffer` with no shared-memory paths. The same
|
||||
// wrap-shape is used by the googlechat google-auth helper at
|
||||
// extensions/googlechat/src/google-auth.runtime.ts:454.
|
||||
const bodyBytes = new Uint8Array(body.buffer, body.byteOffset, body.byteLength);
|
||||
return new Response(bodyBytes as unknown as BodyInit, {
|
||||
status: response.status,
|
||||
statusText: response.statusText,
|
||||
headers: response.headers,
|
||||
});
|
||||
} finally {
|
||||
await release();
|
||||
}
|
||||
}
|
||||
70
extensions/google/oauth.local-login.test.ts
Normal file
70
extensions/google/oauth.local-login.test.ts
Normal file
@@ -0,0 +1,70 @@
|
||||
// Google tests cover oauth.local login plugin behavior.
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
const AUTH_URL = "https://accounts.google.com/o/oauth2/v2/auth?state=state-123";
|
||||
|
||||
const exchangeCodeForTokensMock = vi.hoisted(() =>
|
||||
vi.fn(async () => ({
|
||||
access: "access-token",
|
||||
refresh: "refresh-token",
|
||||
expires: 123,
|
||||
})),
|
||||
);
|
||||
const waitForLocalCallbackMock = vi.hoisted(() =>
|
||||
vi.fn(async () => ({ code: "oauth-code", state: "state-123" })),
|
||||
);
|
||||
|
||||
vi.mock("./oauth.flow.js", () => ({
|
||||
buildAuthUrl: () => AUTH_URL,
|
||||
generateOAuthState: () => "state-123",
|
||||
generatePkce: () => ({ challenge: "pkce-challenge", verifier: "pkce-verifier" }),
|
||||
parseCallbackInput: vi.fn(),
|
||||
shouldUseManualOAuthFlow: (isRemote: boolean) => isRemote,
|
||||
waitForLocalCallback: waitForLocalCallbackMock,
|
||||
}));
|
||||
|
||||
vi.mock("./oauth.token.js", () => ({
|
||||
exchangeCodeForTokens: exchangeCodeForTokensMock,
|
||||
}));
|
||||
|
||||
describe("loginGeminiCliOAuth local browser flow", () => {
|
||||
beforeEach(() => {
|
||||
exchangeCodeForTokensMock.mockClear();
|
||||
waitForLocalCallbackMock.mockClear();
|
||||
});
|
||||
|
||||
it("prints the auth URL before attempting best-effort browser launch", async () => {
|
||||
const events: string[] = [];
|
||||
const { loginGeminiCliOAuth } = await import("./oauth.js");
|
||||
const openUrl = vi.fn(async () => {
|
||||
events.push("open");
|
||||
});
|
||||
const log = vi.fn((message: string) => {
|
||||
events.push(`log:${message}`);
|
||||
});
|
||||
|
||||
const result = await loginGeminiCliOAuth({
|
||||
isRemote: false,
|
||||
openUrl,
|
||||
log,
|
||||
note: async () => {},
|
||||
prompt: async () => "",
|
||||
progress: { update: () => {}, stop: () => {} },
|
||||
});
|
||||
|
||||
expect(result).toEqual({
|
||||
access: "access-token",
|
||||
refresh: "refresh-token",
|
||||
expires: 123,
|
||||
});
|
||||
expect(log).toHaveBeenCalledWith(expect.stringContaining(AUTH_URL));
|
||||
expect(openUrl).toHaveBeenCalledWith(AUTH_URL);
|
||||
expect(events.findIndex((event) => event.startsWith("log:"))).toBeLessThan(
|
||||
events.indexOf("open"),
|
||||
);
|
||||
expect(waitForLocalCallbackMock).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ expectedState: "state-123" }),
|
||||
);
|
||||
expect(exchangeCodeForTokensMock).toHaveBeenCalledWith("oauth-code", "pkce-verifier");
|
||||
});
|
||||
});
|
||||
235
extensions/google/oauth.project.ts
Normal file
235
extensions/google/oauth.project.ts
Normal file
@@ -0,0 +1,235 @@
|
||||
// Google plugin module implements oauth.project behavior.
|
||||
import { fetchWithTimeout } from "./oauth.http.js";
|
||||
import {
|
||||
CODE_ASSIST_ENDPOINT_PROD,
|
||||
LOAD_CODE_ASSIST_ENDPOINTS,
|
||||
TIER_FREE,
|
||||
TIER_LEGACY,
|
||||
TIER_STANDARD,
|
||||
USERINFO_URL,
|
||||
} from "./oauth.shared.js";
|
||||
|
||||
const LOAD_CODE_ASSIST_METADATA = {
|
||||
ideType: "IDE_UNSPECIFIED",
|
||||
platform: "PLATFORM_UNSPECIFIED",
|
||||
pluginType: "GEMINI",
|
||||
} as const;
|
||||
|
||||
async function getUserEmail(accessToken: string): Promise<string | undefined> {
|
||||
try {
|
||||
const response = await fetchWithTimeout(USERINFO_URL, {
|
||||
headers: { Authorization: `Bearer ${accessToken}` },
|
||||
});
|
||||
if (response.ok) {
|
||||
const data = (await response.json()) as { email?: string };
|
||||
return data.email;
|
||||
}
|
||||
} catch {
|
||||
// ignore
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function isVpcScAffected(payload: unknown): boolean {
|
||||
if (!payload || typeof payload !== "object") {
|
||||
return false;
|
||||
}
|
||||
const error = (payload as { error?: unknown }).error;
|
||||
if (!error || typeof error !== "object") {
|
||||
return false;
|
||||
}
|
||||
const details = (error as { details?: unknown[] }).details;
|
||||
if (!Array.isArray(details)) {
|
||||
return false;
|
||||
}
|
||||
return details.some(
|
||||
(item) =>
|
||||
typeof item === "object" &&
|
||||
item &&
|
||||
(item as { reason?: string }).reason === "SECURITY_POLICY_VIOLATED",
|
||||
);
|
||||
}
|
||||
|
||||
function getDefaultTier(
|
||||
allowedTiers?: Array<{ id?: string; isDefault?: boolean }>,
|
||||
): { id?: string } | undefined {
|
||||
if (!allowedTiers?.length) {
|
||||
return { id: TIER_LEGACY };
|
||||
}
|
||||
return allowedTiers.find((tier) => tier.isDefault) ?? { id: TIER_LEGACY };
|
||||
}
|
||||
|
||||
async function pollOperation(
|
||||
endpoint: string,
|
||||
operationName: string,
|
||||
headers: Record<string, string>,
|
||||
): Promise<{ done?: boolean; response?: { cloudaicompanionProject?: { id?: string } } }> {
|
||||
for (let attempt = 0; attempt < 24; attempt += 1) {
|
||||
await new Promise((resolve) => {
|
||||
setTimeout(resolve, 5000);
|
||||
});
|
||||
const response = await fetchWithTimeout(`${endpoint}/v1internal/${operationName}`, {
|
||||
headers,
|
||||
});
|
||||
if (!response.ok) {
|
||||
continue;
|
||||
}
|
||||
const data = (await response.json()) as {
|
||||
done?: boolean;
|
||||
response?: { cloudaicompanionProject?: { id?: string } };
|
||||
};
|
||||
if (data.done) {
|
||||
return data;
|
||||
}
|
||||
}
|
||||
throw new Error("Operation polling timeout");
|
||||
}
|
||||
|
||||
export async function resolveGoogleOAuthIdentity(accessToken: string): Promise<{
|
||||
email?: string;
|
||||
projectId?: string;
|
||||
}> {
|
||||
const email = await getUserEmail(accessToken);
|
||||
const projectId = await discoverProject(accessToken);
|
||||
return { email, projectId };
|
||||
}
|
||||
|
||||
export async function resolveGooglePersonalOAuthIdentity(accessToken: string): Promise<{
|
||||
email?: string;
|
||||
projectId?: string;
|
||||
}> {
|
||||
return { email: await getUserEmail(accessToken) };
|
||||
}
|
||||
|
||||
async function discoverProject(accessToken: string): Promise<string> {
|
||||
const envProject = process.env.GOOGLE_CLOUD_PROJECT || process.env.GOOGLE_CLOUD_PROJECT_ID;
|
||||
const headers = {
|
||||
Authorization: `Bearer ${accessToken}`,
|
||||
"Content-Type": "application/json",
|
||||
"User-Agent": "google-api-nodejs-client/9.15.1",
|
||||
"X-Goog-Api-Client": `gl-node/${process.versions.node}`,
|
||||
"Client-Metadata": JSON.stringify(LOAD_CODE_ASSIST_METADATA),
|
||||
};
|
||||
|
||||
const loadBody = {
|
||||
...(envProject ? { cloudaicompanionProject: envProject } : {}),
|
||||
metadata: {
|
||||
...LOAD_CODE_ASSIST_METADATA,
|
||||
...(envProject ? { duetProject: envProject } : {}),
|
||||
},
|
||||
};
|
||||
|
||||
let data: {
|
||||
currentTier?: { id?: string };
|
||||
cloudaicompanionProject?: string | { id?: string };
|
||||
allowedTiers?: Array<{ id?: string; isDefault?: boolean }>;
|
||||
} = {};
|
||||
let activeEndpoint = CODE_ASSIST_ENDPOINT_PROD;
|
||||
let loadError: Error | undefined;
|
||||
for (const endpoint of LOAD_CODE_ASSIST_ENDPOINTS) {
|
||||
try {
|
||||
const response = await fetchWithTimeout(`${endpoint}/v1internal:loadCodeAssist`, {
|
||||
method: "POST",
|
||||
headers,
|
||||
body: JSON.stringify(loadBody),
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const errorPayload = await response.json().catch(() => null);
|
||||
if (isVpcScAffected(errorPayload)) {
|
||||
data = { currentTier: { id: TIER_STANDARD } };
|
||||
activeEndpoint = endpoint;
|
||||
loadError = undefined;
|
||||
break;
|
||||
}
|
||||
loadError = new Error(`loadCodeAssist failed: ${response.status} ${response.statusText}`);
|
||||
continue;
|
||||
}
|
||||
|
||||
data = (await response.json()) as typeof data;
|
||||
activeEndpoint = endpoint;
|
||||
loadError = undefined;
|
||||
break;
|
||||
} catch (err) {
|
||||
loadError = err instanceof Error ? err : new Error("loadCodeAssist failed", { cause: err });
|
||||
}
|
||||
}
|
||||
|
||||
const hasLoadCodeAssistData =
|
||||
Boolean(data.currentTier) ||
|
||||
Boolean(data.cloudaicompanionProject) ||
|
||||
Boolean(data.allowedTiers?.length);
|
||||
if (!hasLoadCodeAssistData && loadError) {
|
||||
if (envProject) {
|
||||
return envProject;
|
||||
}
|
||||
throw loadError;
|
||||
}
|
||||
|
||||
if (data.currentTier) {
|
||||
const project = data.cloudaicompanionProject;
|
||||
if (typeof project === "string" && project) {
|
||||
return project;
|
||||
}
|
||||
if (typeof project === "object" && project?.id) {
|
||||
return project.id;
|
||||
}
|
||||
if (envProject) {
|
||||
return envProject;
|
||||
}
|
||||
throw new Error(
|
||||
"This account requires GOOGLE_CLOUD_PROJECT or GOOGLE_CLOUD_PROJECT_ID to be set.",
|
||||
);
|
||||
}
|
||||
|
||||
const tier = getDefaultTier(data.allowedTiers);
|
||||
const tierId = tier?.id || TIER_FREE;
|
||||
if (tierId !== TIER_FREE && !envProject) {
|
||||
throw new Error(
|
||||
"This account requires GOOGLE_CLOUD_PROJECT or GOOGLE_CLOUD_PROJECT_ID to be set.",
|
||||
);
|
||||
}
|
||||
|
||||
const onboardBody: Record<string, unknown> = {
|
||||
tierId,
|
||||
metadata: {
|
||||
...LOAD_CODE_ASSIST_METADATA,
|
||||
},
|
||||
};
|
||||
if (tierId !== TIER_FREE && envProject) {
|
||||
onboardBody.cloudaicompanionProject = envProject;
|
||||
(onboardBody.metadata as Record<string, unknown>).duetProject = envProject;
|
||||
}
|
||||
|
||||
const onboardResponse = await fetchWithTimeout(`${activeEndpoint}/v1internal:onboardUser`, {
|
||||
method: "POST",
|
||||
headers,
|
||||
body: JSON.stringify(onboardBody),
|
||||
});
|
||||
|
||||
if (!onboardResponse.ok) {
|
||||
throw new Error(`onboardUser failed: ${onboardResponse.status} ${onboardResponse.statusText}`);
|
||||
}
|
||||
|
||||
let lro = (await onboardResponse.json()) as {
|
||||
done?: boolean;
|
||||
name?: string;
|
||||
response?: { cloudaicompanionProject?: { id?: string } };
|
||||
};
|
||||
|
||||
if (!lro.done && lro.name) {
|
||||
lro = await pollOperation(activeEndpoint, lro.name, headers);
|
||||
}
|
||||
|
||||
const projectId = lro.response?.cloudaicompanionProject?.id;
|
||||
if (projectId) {
|
||||
return projectId;
|
||||
}
|
||||
if (envProject) {
|
||||
return envProject;
|
||||
}
|
||||
|
||||
throw new Error(
|
||||
"Could not discover or provision a Google Cloud project. Set GOOGLE_CLOUD_PROJECT or GOOGLE_CLOUD_PROJECT_ID.",
|
||||
);
|
||||
}
|
||||
2
extensions/google/oauth.runtime.ts
Normal file
2
extensions/google/oauth.runtime.ts
Normal file
@@ -0,0 +1,2 @@
|
||||
// Google plugin module implements oauth behavior.
|
||||
export { loginGeminiCliOAuth, refreshGeminiCliOAuthToken } from "./oauth.js";
|
||||
73
extensions/google/oauth.settings.ts
Normal file
73
extensions/google/oauth.settings.ts
Normal file
@@ -0,0 +1,73 @@
|
||||
// Google plugin module implements oauth.settings behavior.
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import { homedir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { isRecord, normalizeOptionalString } from "openclaw/plugin-sdk/string-coerce-runtime";
|
||||
|
||||
type OAuthSettingsFs = {
|
||||
existsSync: (path: Parameters<typeof existsSync>[0]) => ReturnType<typeof existsSync>;
|
||||
readFileSync: (path: Parameters<typeof readFileSync>[0], encoding: "utf8") => string;
|
||||
homedir: typeof homedir;
|
||||
};
|
||||
|
||||
const defaultFs: OAuthSettingsFs = {
|
||||
existsSync,
|
||||
readFileSync,
|
||||
homedir,
|
||||
};
|
||||
|
||||
let oauthSettingsFs: OAuthSettingsFs = defaultFs;
|
||||
|
||||
type GeminiCliAuthSettings = {
|
||||
security?: {
|
||||
auth?: {
|
||||
selectedType?: unknown;
|
||||
enforcedType?: unknown;
|
||||
};
|
||||
};
|
||||
selectedAuthType?: unknown;
|
||||
enforcedAuthType?: unknown;
|
||||
};
|
||||
|
||||
function readSettingsFile(): GeminiCliAuthSettings | null {
|
||||
const settingsPath = join(oauthSettingsFs.homedir(), ".gemini", "settings.json");
|
||||
if (!oauthSettingsFs.existsSync(settingsPath)) {
|
||||
return null;
|
||||
}
|
||||
try {
|
||||
const parsed = JSON.parse(oauthSettingsFs.readFileSync(settingsPath, "utf8")) as unknown;
|
||||
return isRecord(parsed) ? (parsed as GeminiCliAuthSettings) : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export function setOAuthSettingsFsForTest(overrides?: Partial<OAuthSettingsFs>): void {
|
||||
oauthSettingsFs = overrides ? { ...defaultFs, ...overrides } : defaultFs;
|
||||
}
|
||||
|
||||
export function resolveGeminiCliSelectedAuthType(): string | undefined {
|
||||
const settings = readSettingsFile();
|
||||
if (settings) {
|
||||
const security = isRecord(settings.security) ? settings.security : undefined;
|
||||
const auth = isRecord(security?.auth) ? security.auth : undefined;
|
||||
const selectedAuthType =
|
||||
normalizeOptionalString(auth?.selectedType) ??
|
||||
normalizeOptionalString(auth?.enforcedType) ??
|
||||
normalizeOptionalString(settings.selectedAuthType) ??
|
||||
normalizeOptionalString(settings.enforcedAuthType);
|
||||
if (selectedAuthType) {
|
||||
return selectedAuthType;
|
||||
}
|
||||
}
|
||||
|
||||
if (process.env.GOOGLE_GENAI_USE_GCA === "true") {
|
||||
return "oauth-personal";
|
||||
}
|
||||
|
||||
return undefined;
|
||||
}
|
||||
|
||||
export function isGeminiCliPersonalOAuth(): boolean {
|
||||
return resolveGeminiCliSelectedAuthType() === "oauth-personal";
|
||||
}
|
||||
45
extensions/google/oauth.shared.ts
Normal file
45
extensions/google/oauth.shared.ts
Normal file
@@ -0,0 +1,45 @@
|
||||
// Google plugin module implements oauth.shared behavior.
|
||||
export const CLIENT_ID_KEYS = ["OPENCLAW_GEMINI_OAUTH_CLIENT_ID", "GEMINI_CLI_OAUTH_CLIENT_ID"];
|
||||
export const CLIENT_SECRET_KEYS = [
|
||||
"OPENCLAW_GEMINI_OAUTH_CLIENT_SECRET",
|
||||
"GEMINI_CLI_OAUTH_CLIENT_SECRET",
|
||||
];
|
||||
export const REDIRECT_URI = "http://localhost:8085/oauth2callback";
|
||||
export const AUTH_URL = "https://accounts.google.com/o/oauth2/v2/auth";
|
||||
export const TOKEN_URL = "https://oauth2.googleapis.com/token";
|
||||
export const USERINFO_URL = "https://www.googleapis.com/oauth2/v1/userinfo?alt=json";
|
||||
export const CODE_ASSIST_ENDPOINT_PROD = "https://cloudcode-pa.googleapis.com";
|
||||
const CODE_ASSIST_ENDPOINT_DAILY = "https://daily-cloudcode-pa.sandbox.googleapis.com";
|
||||
const CODE_ASSIST_ENDPOINT_AUTOPUSH = "https://autopush-cloudcode-pa.sandbox.googleapis.com";
|
||||
export const LOAD_CODE_ASSIST_ENDPOINTS = [
|
||||
CODE_ASSIST_ENDPOINT_PROD,
|
||||
CODE_ASSIST_ENDPOINT_DAILY,
|
||||
CODE_ASSIST_ENDPOINT_AUTOPUSH,
|
||||
];
|
||||
export const DEFAULT_FETCH_TIMEOUT_MS = 10_000;
|
||||
export const SCOPES = [
|
||||
"https://www.googleapis.com/auth/cloud-platform",
|
||||
"https://www.googleapis.com/auth/userinfo.email",
|
||||
"https://www.googleapis.com/auth/userinfo.profile",
|
||||
];
|
||||
|
||||
export const TIER_FREE = "free-tier";
|
||||
export const TIER_LEGACY = "legacy-tier";
|
||||
export const TIER_STANDARD = "standard-tier";
|
||||
|
||||
export type GeminiCliOAuthCredentials = {
|
||||
access: string;
|
||||
refresh: string;
|
||||
expires: number;
|
||||
email?: string;
|
||||
projectId?: string;
|
||||
};
|
||||
|
||||
export type GeminiCliOAuthContext = {
|
||||
isRemote: boolean;
|
||||
openUrl: (url: string) => Promise<void>;
|
||||
log: (msg: string) => void;
|
||||
note: (message: string, title?: string) => Promise<void>;
|
||||
prompt: (message: string) => Promise<string>;
|
||||
progress: { update: (msg: string) => void; stop: (msg?: string) => void };
|
||||
};
|
||||
1120
extensions/google/oauth.test.ts
Normal file
1120
extensions/google/oauth.test.ts
Normal file
File diff suppressed because it is too large
Load Diff
161
extensions/google/oauth.token.ts
Normal file
161
extensions/google/oauth.token.ts
Normal file
@@ -0,0 +1,161 @@
|
||||
// Google plugin module implements oauth.token behavior.
|
||||
import {
|
||||
asDateTimestampMs,
|
||||
resolveExpiresAtMsFromDurationSeconds,
|
||||
} from "openclaw/plugin-sdk/number-runtime";
|
||||
import { readResponseTextLimited } from "openclaw/plugin-sdk/provider-http";
|
||||
import { resolveOAuthClientConfig } from "./oauth.credentials.js";
|
||||
import { fetchWithTimeout } from "./oauth.http.js";
|
||||
import { resolveGoogleOAuthIdentity, resolveGooglePersonalOAuthIdentity } from "./oauth.project.js";
|
||||
import { isGeminiCliPersonalOAuth } from "./oauth.settings.js";
|
||||
import { REDIRECT_URI, TOKEN_URL, type GeminiCliOAuthCredentials } from "./oauth.shared.js";
|
||||
|
||||
const TOKEN_EXPIRY_BUFFER_MS = 5 * 60 * 1000;
|
||||
const GOOGLE_OAUTH_TOKEN_ERROR_BODY_LIMIT_BYTES = 8 * 1024;
|
||||
|
||||
async function requestTokenGrant(body: URLSearchParams): Promise<{
|
||||
access_token?: string;
|
||||
refresh_token?: string;
|
||||
expires_in?: unknown;
|
||||
}> {
|
||||
const response = await fetchWithTimeout(TOKEN_URL, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/x-www-form-urlencoded;charset=UTF-8",
|
||||
Accept: "*/*",
|
||||
"User-Agent": "google-api-nodejs-client/9.15.1",
|
||||
},
|
||||
body,
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
const errorText = await readResponseTextLimited(
|
||||
response,
|
||||
GOOGLE_OAUTH_TOKEN_ERROR_BODY_LIMIT_BYTES,
|
||||
);
|
||||
throw new Error(`Token exchange failed: ${errorText}`);
|
||||
}
|
||||
|
||||
return (await response.json()) as {
|
||||
access_token?: string;
|
||||
refresh_token?: string;
|
||||
expires_in?: unknown;
|
||||
};
|
||||
}
|
||||
|
||||
function resolveExpiredTokenTimestampMs(nowMs: number): number {
|
||||
return asDateTimestampMs(nowMs - TOKEN_EXPIRY_BUFFER_MS) ?? nowMs;
|
||||
}
|
||||
|
||||
function resolveTokenExpiresAt(value: unknown): number {
|
||||
const nowMs = asDateTimestampMs(Date.now());
|
||||
if (nowMs === undefined) {
|
||||
return 0;
|
||||
}
|
||||
return (
|
||||
resolveExpiresAtMsFromDurationSeconds(value, { nowMs, bufferMs: TOKEN_EXPIRY_BUFFER_MS }) ??
|
||||
resolveExpiredTokenTimestampMs(nowMs)
|
||||
);
|
||||
}
|
||||
|
||||
async function buildGeminiCliCredentials(params: {
|
||||
tokenResponse: {
|
||||
access_token?: string;
|
||||
refresh_token?: string;
|
||||
expires_in?: unknown;
|
||||
};
|
||||
refreshTokenFallback?: string;
|
||||
existing?: Pick<GeminiCliOAuthCredentials, "email" | "projectId">;
|
||||
allowIdentityFallback?: boolean;
|
||||
}): Promise<GeminiCliOAuthCredentials> {
|
||||
const accessToken = params.tokenResponse.access_token;
|
||||
if (!accessToken) {
|
||||
throw new Error("No access token received. Please try again.");
|
||||
}
|
||||
|
||||
let identity: { email?: string; projectId?: string } = params.existing ?? {};
|
||||
try {
|
||||
if (!identity.email || !identity.projectId) {
|
||||
const discovered = await resolveGeminiCliIdentity(accessToken);
|
||||
identity = {
|
||||
email: identity.email ?? discovered.email,
|
||||
projectId: identity.projectId ?? discovered.projectId,
|
||||
};
|
||||
}
|
||||
} catch (error) {
|
||||
if (!params.allowIdentityFallback || (!params.existing?.email && !params.existing?.projectId)) {
|
||||
throw error;
|
||||
}
|
||||
// If identity discovery is temporarily unavailable during refresh, keep the
|
||||
// already-stored identity binding instead of failing token renewal.
|
||||
}
|
||||
|
||||
const expiresAt = resolveTokenExpiresAt(params.tokenResponse.expires_in);
|
||||
|
||||
return {
|
||||
refresh: params.tokenResponse.refresh_token ?? params.refreshTokenFallback ?? "",
|
||||
access: accessToken,
|
||||
expires: expiresAt,
|
||||
projectId: identity.projectId,
|
||||
email: identity.email,
|
||||
};
|
||||
}
|
||||
|
||||
async function resolveGeminiCliIdentity(
|
||||
accessToken: string,
|
||||
): Promise<{ email?: string; projectId?: string }> {
|
||||
return isGeminiCliPersonalOAuth()
|
||||
? await resolveGooglePersonalOAuthIdentity(accessToken)
|
||||
: await resolveGoogleOAuthIdentity(accessToken);
|
||||
}
|
||||
|
||||
export async function exchangeCodeForTokens(
|
||||
code: string,
|
||||
verifier: string,
|
||||
): Promise<GeminiCliOAuthCredentials> {
|
||||
const { clientId, clientSecret } = resolveOAuthClientConfig();
|
||||
const body = new URLSearchParams({
|
||||
client_id: clientId,
|
||||
code,
|
||||
grant_type: "authorization_code",
|
||||
redirect_uri: REDIRECT_URI,
|
||||
code_verifier: verifier,
|
||||
});
|
||||
if (clientSecret) {
|
||||
body.set("client_secret", clientSecret);
|
||||
}
|
||||
|
||||
const refreshed = await buildGeminiCliCredentials({
|
||||
tokenResponse: await requestTokenGrant(body),
|
||||
});
|
||||
if (!refreshed.refresh) {
|
||||
throw new Error("No refresh token received. Please try again.");
|
||||
}
|
||||
return refreshed;
|
||||
}
|
||||
|
||||
export async function refreshTokensForGeminiCli(credentials: {
|
||||
refresh: string;
|
||||
email?: string;
|
||||
projectId?: string;
|
||||
}): Promise<GeminiCliOAuthCredentials> {
|
||||
const { clientId, clientSecret } = resolveOAuthClientConfig();
|
||||
const body = new URLSearchParams({
|
||||
client_id: clientId,
|
||||
grant_type: "refresh_token",
|
||||
refresh_token: credentials.refresh,
|
||||
});
|
||||
if (clientSecret) {
|
||||
body.set("client_secret", clientSecret);
|
||||
}
|
||||
|
||||
return await buildGeminiCliCredentials({
|
||||
tokenResponse: await requestTokenGrant(body),
|
||||
refreshTokenFallback: credentials.refresh,
|
||||
existing: {
|
||||
email: credentials.email,
|
||||
projectId: credentials.projectId,
|
||||
},
|
||||
allowIdentityFallback: true,
|
||||
});
|
||||
}
|
||||
106
extensions/google/oauth.ts
Normal file
106
extensions/google/oauth.ts
Normal file
@@ -0,0 +1,106 @@
|
||||
// Google plugin module implements oauth behavior.
|
||||
import type { OAuthCredential } from "openclaw/plugin-sdk/provider-auth";
|
||||
import { clearCredentialsCache, extractGeminiCliCredentials } from "./oauth.credentials.js";
|
||||
import {
|
||||
buildAuthUrl,
|
||||
generateOAuthState,
|
||||
generatePkce,
|
||||
parseCallbackInput,
|
||||
shouldUseManualOAuthFlow,
|
||||
waitForLocalCallback,
|
||||
} from "./oauth.flow.js";
|
||||
import type { GeminiCliOAuthContext, GeminiCliOAuthCredentials } from "./oauth.shared.js";
|
||||
import { exchangeCodeForTokens, refreshTokensForGeminiCli } from "./oauth.token.js";
|
||||
|
||||
export { clearCredentialsCache, extractGeminiCliCredentials };
|
||||
export type { GeminiCliOAuthContext, GeminiCliOAuthCredentials };
|
||||
|
||||
export async function loginGeminiCliOAuth(
|
||||
ctx: GeminiCliOAuthContext,
|
||||
): Promise<GeminiCliOAuthCredentials> {
|
||||
const needsManual = shouldUseManualOAuthFlow(ctx.isRemote);
|
||||
await ctx.note(
|
||||
needsManual
|
||||
? [
|
||||
"You are running in a remote/VPS environment.",
|
||||
"A URL will be shown for you to open in your LOCAL browser.",
|
||||
"After signing in, copy the redirect URL and paste it back here.",
|
||||
].join("\n")
|
||||
: [
|
||||
"Browser will open for Google authentication.",
|
||||
"Sign in with your Google account for Gemini CLI access.",
|
||||
"The callback will be captured automatically on localhost:8085.",
|
||||
].join("\n"),
|
||||
"Gemini CLI OAuth",
|
||||
);
|
||||
|
||||
const { verifier, challenge } = generatePkce();
|
||||
const state = generateOAuthState();
|
||||
const authUrl = buildAuthUrl(challenge, state);
|
||||
|
||||
if (needsManual) {
|
||||
return manualFlow(ctx, authUrl, state, verifier);
|
||||
}
|
||||
|
||||
ctx.progress.update("Complete sign-in in browser...");
|
||||
ctx.log(`\nOpen this URL in your browser:\n\n${authUrl}\n`);
|
||||
try {
|
||||
await ctx.openUrl(authUrl);
|
||||
} catch {
|
||||
// The URL is already visible; browser launch is best-effort.
|
||||
}
|
||||
|
||||
try {
|
||||
const { code } = await waitForLocalCallback({
|
||||
expectedState: state,
|
||||
timeoutMs: 5 * 60 * 1000,
|
||||
onProgress: (msg) => ctx.progress.update(msg),
|
||||
});
|
||||
ctx.progress.update("Exchanging authorization code for tokens...");
|
||||
return await exchangeCodeForTokens(code, verifier);
|
||||
} catch (err) {
|
||||
if (
|
||||
err instanceof Error &&
|
||||
(err.message.includes("EADDRINUSE") ||
|
||||
err.message.includes("port") ||
|
||||
err.message.includes("listen"))
|
||||
) {
|
||||
ctx.progress.update("Local callback server failed. Switching to manual mode...");
|
||||
return manualFlow(ctx, authUrl, state, verifier, err);
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
async function manualFlow(
|
||||
ctx: GeminiCliOAuthContext,
|
||||
authUrl: string,
|
||||
state: string,
|
||||
verifier: string,
|
||||
cause?: Error,
|
||||
): Promise<GeminiCliOAuthCredentials> {
|
||||
ctx.progress.update("OAuth URL ready");
|
||||
ctx.log(`\nOpen this URL in your LOCAL browser:\n\n${authUrl}\n`);
|
||||
ctx.progress.update("Waiting for you to paste the callback URL...");
|
||||
const callbackInput = await ctx.prompt("Paste the redirect URL here: ");
|
||||
const parsed = parseCallbackInput(callbackInput);
|
||||
if ("error" in parsed) {
|
||||
throw new Error(parsed.error, cause ? { cause } : undefined);
|
||||
}
|
||||
if (parsed.state !== state) {
|
||||
throw new Error("OAuth state mismatch - please try again", cause ? { cause } : undefined);
|
||||
}
|
||||
ctx.progress.update("Exchanging authorization code for tokens...");
|
||||
return exchangeCodeForTokens(parsed.code, verifier);
|
||||
}
|
||||
|
||||
export async function refreshGeminiCliOAuthToken(
|
||||
credentials: Pick<GeminiCliOAuthCredentials, "refresh" | "email" | "projectId">,
|
||||
): Promise<OAuthCredential> {
|
||||
const refreshed = await refreshTokensForGeminiCli(credentials);
|
||||
return {
|
||||
type: "oauth",
|
||||
provider: "google-gemini-cli",
|
||||
...refreshed,
|
||||
};
|
||||
}
|
||||
79
extensions/google/onboard.ts
Normal file
79
extensions/google/onboard.ts
Normal file
@@ -0,0 +1,79 @@
|
||||
// Google setup module handles plugin onboarding behavior.
|
||||
import {
|
||||
applyAgentDefaultModelPrimary,
|
||||
type OpenClawConfig,
|
||||
} from "openclaw/plugin-sdk/provider-onboard";
|
||||
|
||||
export const GOOGLE_GEMINI_DEFAULT_MODEL = "google/gemini-3.1-pro-preview";
|
||||
const RETIRED_GOOGLE_GEMINI_MODEL_REFS = new Set([
|
||||
"gemini-3-pro",
|
||||
"gemini-3-pro-preview",
|
||||
"google/gemini-3-pro",
|
||||
"google/gemini-3-pro-preview",
|
||||
]);
|
||||
|
||||
function isRetiredGeminiModelRef(value: unknown): boolean {
|
||||
if (typeof value !== "string") {
|
||||
return false;
|
||||
}
|
||||
const modelRef = value.trim();
|
||||
if (RETIRED_GOOGLE_GEMINI_MODEL_REFS.has(modelRef)) {
|
||||
return true;
|
||||
}
|
||||
return modelRef.endsWith("/gemini-3-pro") || modelRef.endsWith("/gemini-3-pro-preview");
|
||||
}
|
||||
|
||||
function hasRetiredGeminiDefaultModelRefs(cfg: OpenClawConfig): boolean {
|
||||
const defaults = cfg.agents?.defaults;
|
||||
const model = defaults?.model as unknown;
|
||||
if (model && typeof model === "object") {
|
||||
const fallbacks = (model as { fallbacks?: unknown }).fallbacks;
|
||||
if (
|
||||
Array.isArray(fallbacks) &&
|
||||
fallbacks.some((fallback) => isRetiredGeminiModelRef(fallback))
|
||||
) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
const models = defaults?.models;
|
||||
if (
|
||||
models &&
|
||||
typeof models === "object" &&
|
||||
Object.keys(models).some((modelRef) => isRetiredGeminiModelRef(modelRef))
|
||||
) {
|
||||
return true;
|
||||
}
|
||||
|
||||
const providerConfigs = cfg.models?.providers;
|
||||
if (!providerConfigs) {
|
||||
return false;
|
||||
}
|
||||
return Object.values(providerConfigs).some((providerConfig) =>
|
||||
Array.isArray(providerConfig.models)
|
||||
? providerConfig.models.some((providerModel) => isRetiredGeminiModelRef(providerModel.id))
|
||||
: false,
|
||||
);
|
||||
}
|
||||
|
||||
export function applyGoogleGeminiModelDefault(cfg: OpenClawConfig): {
|
||||
next: OpenClawConfig;
|
||||
changed: boolean;
|
||||
} {
|
||||
const current = cfg.agents?.defaults?.model as unknown;
|
||||
const currentPrimary =
|
||||
typeof current === "string"
|
||||
? current.trim() || undefined
|
||||
: current &&
|
||||
typeof current === "object" &&
|
||||
typeof (current as { primary?: unknown }).primary === "string"
|
||||
? ((current as { primary: string }).primary || "").trim() || undefined
|
||||
: undefined;
|
||||
if (currentPrimary === GOOGLE_GEMINI_DEFAULT_MODEL && !hasRetiredGeminiDefaultModelRefs(cfg)) {
|
||||
return { next: cfg, changed: false };
|
||||
}
|
||||
return {
|
||||
next: applyAgentDefaultModelPrimary(cfg, GOOGLE_GEMINI_DEFAULT_MODEL),
|
||||
changed: true,
|
||||
};
|
||||
}
|
||||
719
extensions/google/openclaw.plugin.json
Normal file
719
extensions/google/openclaw.plugin.json
Normal file
@@ -0,0 +1,719 @@
|
||||
{
|
||||
"id": "google",
|
||||
"icon": "https://cdn.simpleicons.org/google",
|
||||
"activation": {
|
||||
"onStartup": false
|
||||
},
|
||||
"enabledByDefault": true,
|
||||
"providers": ["google", "google-gemini-cli", "google-vertex"],
|
||||
"providerCatalogEntry": "./provider-discovery.ts",
|
||||
"autoEnableWhenConfiguredProviders": ["google-gemini-cli"],
|
||||
"modelIdNormalization": {
|
||||
"providers": {
|
||||
"google": {
|
||||
"aliases": {
|
||||
"gemini-3-pro": "gemini-3.1-pro-preview",
|
||||
"gemini-3-pro-preview": "gemini-3.1-pro-preview",
|
||||
"gemini-3-flash": "gemini-3-flash-preview",
|
||||
"gemini-3.1-pro": "gemini-3.1-pro-preview",
|
||||
"gemini-3.1-flash-lite-preview": "gemini-3.1-flash-lite",
|
||||
"gemini-3.1-flash": "gemini-3-flash-preview",
|
||||
"gemini-3.1-flash-preview": "gemini-3-flash-preview"
|
||||
}
|
||||
},
|
||||
"google-gemini-cli": {
|
||||
"aliases": {
|
||||
"gemini-3-pro": "gemini-3.1-pro-preview",
|
||||
"gemini-3-pro-preview": "gemini-3.1-pro-preview",
|
||||
"gemini-3-flash": "gemini-3-flash-preview",
|
||||
"gemini-3.1-pro": "gemini-3.1-pro-preview",
|
||||
"gemini-3.1-flash-lite-preview": "gemini-3.1-flash-lite",
|
||||
"gemini-3.1-flash": "gemini-3-flash-preview",
|
||||
"gemini-3.1-flash-preview": "gemini-3-flash-preview"
|
||||
}
|
||||
},
|
||||
"google-vertex": {
|
||||
"aliases": {
|
||||
"gemini-3-pro": "gemini-3.1-pro-preview",
|
||||
"gemini-3-pro-preview": "gemini-3.1-pro-preview",
|
||||
"gemini-3-flash": "gemini-3-flash-preview",
|
||||
"gemini-3.1-pro": "gemini-3.1-pro-preview",
|
||||
"gemini-3.1-flash-lite-preview": "gemini-3.1-flash-lite",
|
||||
"gemini-3.1-flash": "gemini-3-flash-preview",
|
||||
"gemini-3.1-flash-preview": "gemini-3-flash-preview"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"modelCatalog": {
|
||||
"suppressions": [
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-1.5-flash",
|
||||
"reason": "Google shut down Gemini 1.5 Flash on 2025-09-29. Use google/gemini-2.5-flash."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-1.5-flash-8b",
|
||||
"reason": "Google shut down Gemini 1.5 Flash-8B on 2025-09-29. Use google/gemini-2.5-flash-lite."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-1.5-pro",
|
||||
"reason": "Google shut down Gemini 1.5 Pro on 2025-09-29. Use google/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.0-flash-exp",
|
||||
"reason": "Google shut down this Gemini 2.0 experimental model. Use google/gemini-2.5-flash."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.0-flash-exp-image-generation",
|
||||
"reason": "Google shut down this Gemini 2.0 image preview. Use google/gemini-2.5-flash-image."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.0-flash-live-001",
|
||||
"reason": "Google shut down this Gemini Live model on 2025-12-09. Use google/gemini-3.1-flash-live-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.0-flash-lite-preview",
|
||||
"reason": "Google shut down this Gemini 2.0 Flash-Lite preview on 2025-12-09. Use google/gemini-2.5-flash-lite."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.0-flash-lite-preview-02-05",
|
||||
"reason": "Google shut down this Gemini 2.0 Flash-Lite preview on 2025-12-09. Use google/gemini-2.5-flash-lite."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.0-flash-preview-image-generation",
|
||||
"reason": "Google shut down this Gemini 2.0 image preview. Use google/gemini-2.5-flash-image."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.0-flash-thinking-exp",
|
||||
"reason": "Google shut down this Gemini 2.0 thinking experiment. Use google/gemini-2.5-flash."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.0-flash-thinking-exp-01-21",
|
||||
"reason": "Google shut down this Gemini 2.0 thinking experiment. Use google/gemini-2.5-flash."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.0-flash-thinking-exp-1219",
|
||||
"reason": "Google shut down this Gemini 2.0 thinking experiment. Use google/gemini-2.5-flash."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.0-pro-exp",
|
||||
"reason": "Google shut down this Gemini 2.0 Pro experiment. Use google/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.0-pro-exp-02-05",
|
||||
"reason": "Google shut down this Gemini 2.0 Pro experiment. Use google/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.5-flash-exp-native-audio-thinking-dialog",
|
||||
"reason": "Google shut down this Gemini native-audio preview. Use google/gemini-3.1-flash-live-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.5-flash-image-preview",
|
||||
"reason": "Google shut down this Gemini image preview on 2026-01-15. Use google/gemini-2.5-flash-image."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.5-flash-lite-preview-06-17",
|
||||
"reason": "Google shut down this Gemini 2.5 Flash-Lite preview on 2025-11-18. Use google/gemini-2.5-flash-lite."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.5-flash-lite-preview-09-25",
|
||||
"reason": "Google shut down this Gemini 2.5 Flash-Lite preview on 2026-03-31. Use google/gemini-3.1-flash-lite."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.5-flash-lite-preview-09-2025",
|
||||
"reason": "Google shut down this Gemini 2.5 Flash-Lite preview on 2026-03-31. Use google/gemini-3.1-flash-lite."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.5-flash-preview-04-17",
|
||||
"reason": "Google shut down this Gemini 2.5 Flash preview on 2025-07-15. Use google/gemini-2.5-flash."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.5-flash-preview-05-20",
|
||||
"reason": "Google shut down this Gemini 2.5 Flash preview on 2025-11-18. Use google/gemini-2.5-flash."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.5-flash-preview-09-25",
|
||||
"reason": "Google shut down this Gemini 2.5 Flash preview on 2026-02-17. Use google/gemini-3-flash-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.5-flash-preview-09-2025",
|
||||
"reason": "Google shut down this Gemini 2.5 Flash preview on 2026-02-17. Use google/gemini-3-flash-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.5-flash-preview-native-audio-dialog",
|
||||
"reason": "Google shut down this Gemini native-audio preview. Use google/gemini-3.1-flash-live-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.5-pro-exp-03-25",
|
||||
"reason": "Google shut down this Gemini 2.5 Pro experiment. Use google/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.5-pro-preview-03-25",
|
||||
"reason": "Google shut down this Gemini 2.5 Pro preview on 2025-12-02. Use google/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.5-pro-preview-05-06",
|
||||
"reason": "Google shut down this Gemini 2.5 Pro preview on 2025-12-02. Use google/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-2.5-pro-preview-06-05",
|
||||
"reason": "Google shut down this Gemini 2.5 Pro preview on 2025-12-02. Use google/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-3-pro-preview",
|
||||
"reason": "Google shut down Gemini 3 Pro Preview on 2026-03-09. Use google/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-3.1-pro-preview-customtools",
|
||||
"reason": "This is not a public Google Gemini chat model ID. Use google/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-live-2.5-flash",
|
||||
"reason": "This is not a current public Gemini chat model ID. Use google/gemini-3.1-flash-live-preview for Live API."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-live-2.5-flash-preview",
|
||||
"reason": "Google shut down this Gemini Live model on 2025-12-09. Use google/gemini-3.1-flash-live-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google",
|
||||
"model": "gemini-live-2.5-flash-preview-native-audio",
|
||||
"reason": "Google shut down this Gemini Live preview. Use google/gemini-3.1-flash-live-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-1.5-flash",
|
||||
"reason": "Google shut down Gemini 1.5 Flash on 2025-09-29. Use google-gemini-cli/gemini-2.5-flash."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-1.5-flash-8b",
|
||||
"reason": "Google shut down Gemini 1.5 Flash-8B on 2025-09-29. Use google-gemini-cli/gemini-2.5-flash-lite."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-1.5-pro",
|
||||
"reason": "Google shut down Gemini 1.5 Pro on 2025-09-29. Use google-gemini-cli/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.0-flash-exp",
|
||||
"reason": "Google shut down this Gemini 2.0 experimental model. Use google-gemini-cli/gemini-2.5-flash."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.0-flash-exp-image-generation",
|
||||
"reason": "Google shut down this Gemini 2.0 image preview. Use google-gemini-cli/gemini-2.5-flash-image."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.0-flash-live-001",
|
||||
"reason": "Google shut down this Gemini Live model on 2025-12-09. Use google-gemini-cli/gemini-3.1-flash-live-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.0-flash-lite-preview",
|
||||
"reason": "Google shut down this Gemini 2.0 Flash-Lite preview on 2025-12-09. Use google-gemini-cli/gemini-2.5-flash-lite."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.0-flash-lite-preview-02-05",
|
||||
"reason": "Google shut down this Gemini 2.0 Flash-Lite preview on 2025-12-09. Use google-gemini-cli/gemini-2.5-flash-lite."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.0-flash-preview-image-generation",
|
||||
"reason": "Google shut down this Gemini 2.0 image preview. Use google-gemini-cli/gemini-2.5-flash-image."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.0-flash-thinking-exp",
|
||||
"reason": "Google shut down this Gemini 2.0 thinking experiment. Use google-gemini-cli/gemini-2.5-flash."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.0-flash-thinking-exp-01-21",
|
||||
"reason": "Google shut down this Gemini 2.0 thinking experiment. Use google-gemini-cli/gemini-2.5-flash."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.0-flash-thinking-exp-1219",
|
||||
"reason": "Google shut down this Gemini 2.0 thinking experiment. Use google-gemini-cli/gemini-2.5-flash."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.0-pro-exp",
|
||||
"reason": "Google shut down this Gemini 2.0 Pro experiment. Use google-gemini-cli/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.0-pro-exp-02-05",
|
||||
"reason": "Google shut down this Gemini 2.0 Pro experiment. Use google-gemini-cli/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.5-flash-exp-native-audio-thinking-dialog",
|
||||
"reason": "Google shut down this Gemini native-audio preview. Use google-gemini-cli/gemini-3.1-flash-live-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.5-flash-image-preview",
|
||||
"reason": "Google shut down this Gemini image preview on 2026-01-15. Use google-gemini-cli/gemini-2.5-flash-image."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.5-flash-lite-preview-06-17",
|
||||
"reason": "Google shut down this Gemini 2.5 Flash-Lite preview on 2025-11-18. Use google-gemini-cli/gemini-2.5-flash-lite."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.5-flash-lite-preview-09-25",
|
||||
"reason": "Google shut down this Gemini 2.5 Flash-Lite preview on 2026-03-31. Use google-gemini-cli/gemini-3.1-flash-lite."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.5-flash-lite-preview-09-2025",
|
||||
"reason": "Google shut down this Gemini 2.5 Flash-Lite preview on 2026-03-31. Use google-gemini-cli/gemini-3.1-flash-lite."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.5-flash-preview-04-17",
|
||||
"reason": "Google shut down this Gemini 2.5 Flash preview on 2025-07-15. Use google-gemini-cli/gemini-2.5-flash."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.5-flash-preview-05-20",
|
||||
"reason": "Google shut down this Gemini 2.5 Flash preview on 2025-11-18. Use google-gemini-cli/gemini-2.5-flash."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.5-flash-preview-09-25",
|
||||
"reason": "Google shut down this Gemini 2.5 Flash preview on 2026-02-17. Use google-gemini-cli/gemini-3-flash-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.5-flash-preview-09-2025",
|
||||
"reason": "Google shut down this Gemini 2.5 Flash preview on 2026-02-17. Use google-gemini-cli/gemini-3-flash-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.5-flash-preview-native-audio-dialog",
|
||||
"reason": "Google shut down this Gemini native-audio preview. Use google-gemini-cli/gemini-3.1-flash-live-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.5-pro-exp-03-25",
|
||||
"reason": "Google shut down this Gemini 2.5 Pro experiment. Use google-gemini-cli/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.5-pro-preview-03-25",
|
||||
"reason": "Google shut down this Gemini 2.5 Pro preview on 2025-12-02. Use google-gemini-cli/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.5-pro-preview-05-06",
|
||||
"reason": "Google shut down this Gemini 2.5 Pro preview on 2025-12-02. Use google-gemini-cli/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-2.5-pro-preview-06-05",
|
||||
"reason": "Google shut down this Gemini 2.5 Pro preview on 2025-12-02. Use google-gemini-cli/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-3-pro-preview",
|
||||
"reason": "Google shut down Gemini 3 Pro Preview on 2026-03-09. Use google-gemini-cli/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-3.1-pro-preview-customtools",
|
||||
"reason": "This is not a public Google Gemini chat model ID. Use google-gemini-cli/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-live-2.5-flash",
|
||||
"reason": "This is not a current public Gemini chat model ID. Use google-gemini-cli/gemini-3.1-flash-live-preview for Live API."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-live-2.5-flash-preview",
|
||||
"reason": "Google shut down this Gemini Live model on 2025-12-09. Use google-gemini-cli/gemini-3.1-flash-live-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"model": "gemini-live-2.5-flash-preview-native-audio",
|
||||
"reason": "Google shut down this Gemini Live preview. Use google-gemini-cli/gemini-3.1-flash-live-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-1.5-flash",
|
||||
"reason": "Google retired Gemini 1.5 Flash. Use google-vertex/gemini-2.5-flash."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-1.5-flash-8b",
|
||||
"reason": "Google retired Gemini 1.5 Flash-8B. Use google-vertex/gemini-2.5-flash-lite."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-1.5-pro",
|
||||
"reason": "Google retired Gemini 1.5 Pro. Use google-vertex/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.0-flash-exp",
|
||||
"reason": "Google shut down this Gemini 2.0 experimental model. Use google-vertex/gemini-2.5-flash."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.0-flash-exp-image-generation",
|
||||
"reason": "Google shut down this Gemini 2.0 image preview. Use google-vertex/gemini-2.5-flash-image."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.0-flash-live-001",
|
||||
"reason": "Google shut down this Gemini Live model on 2025-12-09. Use google-vertex/gemini-3.1-flash-live-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.0-flash-lite-preview",
|
||||
"reason": "Google shut down this Gemini 2.0 Flash-Lite preview on 2025-12-09. Use google-vertex/gemini-2.5-flash-lite."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.0-flash-lite-preview-02-05",
|
||||
"reason": "Google shut down this Gemini 2.0 Flash-Lite preview on 2025-12-09. Use google-vertex/gemini-2.5-flash-lite."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.0-flash-preview-image-generation",
|
||||
"reason": "Google shut down this Gemini 2.0 image preview. Use google-vertex/gemini-2.5-flash-image."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.0-flash-thinking-exp",
|
||||
"reason": "Google shut down this Gemini 2.0 thinking experiment. Use google-vertex/gemini-2.5-flash."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.0-flash-thinking-exp-01-21",
|
||||
"reason": "Google shut down this Gemini 2.0 thinking experiment. Use google-vertex/gemini-2.5-flash."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.0-flash-thinking-exp-1219",
|
||||
"reason": "Google shut down this Gemini 2.0 thinking experiment. Use google-vertex/gemini-2.5-flash."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.0-pro-exp",
|
||||
"reason": "Google shut down this Gemini 2.0 Pro experiment. Use google-vertex/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.0-pro-exp-02-05",
|
||||
"reason": "Google shut down this Gemini 2.0 Pro experiment. Use google-vertex/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.5-flash-exp-native-audio-thinking-dialog",
|
||||
"reason": "Google shut down this Gemini native-audio preview. Use google-vertex/gemini-3.1-flash-live-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.5-flash-image-preview",
|
||||
"reason": "Google shut down this Gemini image preview on 2026-01-15. Use google-vertex/gemini-2.5-flash-image."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.5-flash-lite-preview-06-17",
|
||||
"reason": "Google shut down this Gemini 2.5 Flash-Lite preview on 2025-11-18. Use google-vertex/gemini-2.5-flash-lite."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.5-flash-lite-preview-09-25",
|
||||
"reason": "Google shut down this Gemini 2.5 Flash-Lite preview on 2026-03-31. Use google-vertex/gemini-3.1-flash-lite."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.5-flash-lite-preview-09-2025",
|
||||
"reason": "Google shut down this Gemini 2.5 Flash-Lite preview on 2026-03-31. Use google-vertex/gemini-3.1-flash-lite."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.5-flash-preview-04-17",
|
||||
"reason": "Google shut down this Gemini 2.5 Flash preview on 2025-07-15. Use google-vertex/gemini-2.5-flash."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.5-flash-preview-05-20",
|
||||
"reason": "Google shut down this Gemini 2.5 Flash preview on 2025-11-18. Use google-vertex/gemini-2.5-flash."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.5-flash-preview-09-25",
|
||||
"reason": "Google shut down this Gemini 2.5 Flash preview on 2026-02-17. Use google-vertex/gemini-3-flash-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.5-flash-preview-09-2025",
|
||||
"reason": "Google shut down this Gemini 2.5 Flash preview on 2026-02-17. Use google-vertex/gemini-3-flash-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.5-flash-preview-native-audio-dialog",
|
||||
"reason": "Google shut down this Gemini native-audio preview. Use google-vertex/gemini-3.1-flash-live-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.5-pro-exp-03-25",
|
||||
"reason": "Google shut down this Gemini 2.5 Pro experiment. Use google-vertex/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.5-pro-preview-03-25",
|
||||
"reason": "Google shut down this Gemini 2.5 Pro preview on 2025-12-02. Use google-vertex/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.5-pro-preview-05-06",
|
||||
"reason": "Google shut down this Gemini 2.5 Pro preview on 2025-12-02. Use google-vertex/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-2.5-pro-preview-06-05",
|
||||
"reason": "Google shut down this Gemini 2.5 Pro preview on 2025-12-02. Use google-vertex/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-3-pro-preview",
|
||||
"reason": "Google shut down Gemini 3 Pro Preview on 2026-03-09. Use google-vertex/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-3.1-pro-preview-customtools",
|
||||
"reason": "This is not a public Google Gemini chat model ID. Use google-vertex/gemini-3.1-pro-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-live-2.5-flash",
|
||||
"reason": "This is not a current public Gemini chat model ID. Use google-vertex/gemini-3.1-flash-live-preview for Live API."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-live-2.5-flash-preview",
|
||||
"reason": "Google shut down this Gemini Live model on 2025-12-09. Use google-vertex/gemini-3.1-flash-live-preview."
|
||||
},
|
||||
{
|
||||
"provider": "google-vertex",
|
||||
"model": "gemini-live-2.5-flash-preview-native-audio",
|
||||
"reason": "Google shut down this Gemini Live preview. Use google-vertex/gemini-3.1-flash-live-preview."
|
||||
}
|
||||
]
|
||||
},
|
||||
"modelPricing": {
|
||||
"providers": {
|
||||
"google-gemini-cli": {
|
||||
"openRouter": {
|
||||
"provider": "google"
|
||||
},
|
||||
"liteLLM": {
|
||||
"provider": "google"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"providerEndpoints": [
|
||||
{
|
||||
"endpointClass": "google-generative-ai",
|
||||
"hosts": ["generativelanguage.googleapis.com"]
|
||||
},
|
||||
{
|
||||
"endpointClass": "google-vertex",
|
||||
"hosts": ["aiplatform.googleapis.com"],
|
||||
"googleVertexRegion": "global"
|
||||
},
|
||||
{
|
||||
"endpointClass": "google-vertex",
|
||||
"hosts": ["aiplatform.eu.rep.googleapis.com"],
|
||||
"googleVertexRegion": "eu"
|
||||
},
|
||||
{
|
||||
"endpointClass": "google-vertex",
|
||||
"hosts": ["aiplatform.us.rep.googleapis.com"],
|
||||
"googleVertexRegion": "us"
|
||||
},
|
||||
{
|
||||
"endpointClass": "google-vertex",
|
||||
"hostSuffixes": ["-aiplatform.googleapis.com"],
|
||||
"googleVertexRegionHostSuffix": "-aiplatform.googleapis.com"
|
||||
}
|
||||
],
|
||||
"providerRequest": {
|
||||
"providers": {
|
||||
"google": {
|
||||
"family": "google"
|
||||
},
|
||||
"google-gemini-cli": {
|
||||
"family": "google"
|
||||
},
|
||||
"google-vertex": {
|
||||
"family": "google"
|
||||
}
|
||||
}
|
||||
},
|
||||
"setup": {
|
||||
"providers": [
|
||||
{
|
||||
"id": "google-vertex",
|
||||
"authMethods": ["api-key"],
|
||||
"envVars": ["GOOGLE_CLOUD_API_KEY"],
|
||||
"authEvidence": [
|
||||
{
|
||||
"type": "local-file-with-env",
|
||||
"fileEnvVar": "GOOGLE_APPLICATION_CREDENTIALS",
|
||||
"fallbackPaths": [
|
||||
"${HOME}/.config/gcloud/application_default_credentials.json",
|
||||
"${APPDATA}/gcloud/application_default_credentials.json"
|
||||
],
|
||||
"requiresAnyEnv": ["GOOGLE_CLOUD_PROJECT", "GCLOUD_PROJECT"],
|
||||
"requiresAllEnv": ["GOOGLE_CLOUD_LOCATION"],
|
||||
"credentialMarker": "gcp-vertex-credentials",
|
||||
"source": "gcloud adc"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "google",
|
||||
"envVars": ["GEMINI_API_KEY", "GOOGLE_API_KEY"]
|
||||
}
|
||||
]
|
||||
},
|
||||
"cliBackends": ["google-gemini-cli"],
|
||||
"providerAuthChoices": [
|
||||
{
|
||||
"provider": "google",
|
||||
"method": "api-key",
|
||||
"choiceId": "gemini-api-key",
|
||||
"choiceLabel": "Google Gemini API key",
|
||||
"groupId": "google",
|
||||
"groupLabel": "Google",
|
||||
"groupHint": "Gemini API key + OAuth",
|
||||
"onboardingFeatured": true,
|
||||
"optionKey": "geminiApiKey",
|
||||
"cliFlag": "--gemini-api-key",
|
||||
"cliOption": "--gemini-api-key <key>",
|
||||
"cliDescription": "Gemini API key"
|
||||
},
|
||||
{
|
||||
"provider": "google-gemini-cli",
|
||||
"method": "oauth",
|
||||
"choiceId": "google-gemini-cli",
|
||||
"choiceLabel": "Gemini CLI OAuth",
|
||||
"choiceHint": "Google OAuth with project-aware token payload",
|
||||
"groupId": "google",
|
||||
"groupLabel": "Google",
|
||||
"groupHint": "Gemini API key + OAuth",
|
||||
"onboardingFeatured": true
|
||||
}
|
||||
],
|
||||
"uiHints": {
|
||||
"webSearch.apiKey": {
|
||||
"label": "Gemini Search API Key",
|
||||
"help": "Gemini API key for Google Search grounding (fallback: GEMINI_API_KEY env var).",
|
||||
"sensitive": true,
|
||||
"placeholder": "AIza..."
|
||||
},
|
||||
"webSearch.model": {
|
||||
"label": "Gemini Search Model",
|
||||
"help": "Gemini model override for web search grounding."
|
||||
},
|
||||
"webSearch.baseUrl": {
|
||||
"label": "Gemini Search Base URL",
|
||||
"help": "Optional Gemini API base URL for web search grounding proxies."
|
||||
}
|
||||
},
|
||||
"contracts": {
|
||||
"mediaUnderstandingProviders": ["google"],
|
||||
"memoryEmbeddingProviders": ["gemini"],
|
||||
"imageGenerationProviders": ["google"],
|
||||
"musicGenerationProviders": ["google"],
|
||||
"realtimeVoiceProviders": ["google"],
|
||||
"speechProviders": ["google"],
|
||||
"videoGenerationProviders": ["google"],
|
||||
"webSearchProviders": ["gemini"]
|
||||
},
|
||||
"mediaUnderstandingProviderMetadata": {
|
||||
"google": {
|
||||
"capabilities": ["image", "audio", "video"],
|
||||
"defaultModels": {
|
||||
"image": "gemini-3-flash-preview",
|
||||
"audio": "gemini-3-flash-preview",
|
||||
"video": "gemini-3-flash-preview"
|
||||
},
|
||||
"autoPriority": {
|
||||
"image": 30,
|
||||
"audio": 40,
|
||||
"video": 10
|
||||
},
|
||||
"nativeDocumentInputs": ["pdf"]
|
||||
}
|
||||
},
|
||||
"configContracts": {
|
||||
"compatibilityRuntimePaths": ["tools.web.search.apiKey"]
|
||||
},
|
||||
"configSchema": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"webSearch": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"apiKey": {
|
||||
"type": ["string", "object"]
|
||||
},
|
||||
"model": {
|
||||
"type": "string"
|
||||
},
|
||||
"baseUrl": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
19
extensions/google/package.json
Normal file
19
extensions/google/package.json
Normal file
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"name": "@openclaw/google-plugin",
|
||||
"version": "2026.6.11",
|
||||
"private": true,
|
||||
"description": "OpenClaw Google plugin",
|
||||
"type": "module",
|
||||
"dependencies": {
|
||||
"@google/genai": "2.10.0",
|
||||
"google-auth-library": "10.9.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@openclaw/plugin-sdk": "workspace:*"
|
||||
},
|
||||
"openclaw": {
|
||||
"extensions": [
|
||||
"./index.ts"
|
||||
]
|
||||
}
|
||||
}
|
||||
29
extensions/google/provider-catalog.test.ts
Normal file
29
extensions/google/provider-catalog.test.ts
Normal file
@@ -0,0 +1,29 @@
|
||||
// Google tests cover provider catalog plugin behavior.
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
buildGoogleStaticCatalogProvider,
|
||||
buildGoogleVertexStaticCatalogProvider,
|
||||
} from "./provider-catalog.js";
|
||||
|
||||
describe("google provider catalog", () => {
|
||||
it("registers current Gemini rows for the Google Vertex provider", () => {
|
||||
const provider = buildGoogleVertexStaticCatalogProvider();
|
||||
|
||||
expect(provider.api).toBe("google-vertex");
|
||||
expect(provider.baseUrl).toBe("https://{location}-aiplatform.googleapis.com");
|
||||
expect(provider.models.map((model) => model.id)).toEqual(
|
||||
expect.arrayContaining(["gemini-2.5-pro", "gemini-3.1-pro-preview", "gemini-3.1-flash-lite"]),
|
||||
);
|
||||
expect(provider.models.find((model) => model.id === "gemini-3.1-flash-lite")).toMatchObject({
|
||||
contextWindow: 1_048_576,
|
||||
maxTokens: 65_536,
|
||||
reasoning: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps Google AI Studio and Vertex model ids aligned", () => {
|
||||
expect(buildGoogleVertexStaticCatalogProvider().models.map((model) => model.id)).toEqual(
|
||||
buildGoogleStaticCatalogProvider().models.map((model) => model.id),
|
||||
);
|
||||
});
|
||||
});
|
||||
90
extensions/google/provider-catalog.ts
Normal file
90
extensions/google/provider-catalog.ts
Normal file
@@ -0,0 +1,90 @@
|
||||
// Google provider module implements model/runtime integration.
|
||||
import type {
|
||||
ModelDefinitionConfig,
|
||||
ModelProviderConfig,
|
||||
} from "openclaw/plugin-sdk/provider-model-shared";
|
||||
|
||||
const GOOGLE_GEMINI_BASE_URL = "https://generativelanguage.googleapis.com/v1beta";
|
||||
const GOOGLE_VERTEX_BASE_URL = "https://{location}-aiplatform.googleapis.com";
|
||||
const GOOGLE_GEMINI_COST = { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 } as const;
|
||||
const GOOGLE_GEMINI_TEXT_MODELS: ModelDefinitionConfig[] = [
|
||||
{
|
||||
id: "gemini-2.5-pro",
|
||||
name: "Gemini 2.5 Pro",
|
||||
reasoning: true,
|
||||
input: ["text", "image"],
|
||||
cost: GOOGLE_GEMINI_COST,
|
||||
contextWindow: 1_048_576,
|
||||
maxTokens: 65_536,
|
||||
},
|
||||
{
|
||||
id: "gemini-2.5-flash",
|
||||
name: "Gemini 2.5 Flash",
|
||||
reasoning: true,
|
||||
input: ["text", "image"],
|
||||
cost: GOOGLE_GEMINI_COST,
|
||||
contextWindow: 1_048_576,
|
||||
maxTokens: 65_536,
|
||||
},
|
||||
{
|
||||
id: "gemini-2.5-flash-lite",
|
||||
name: "Gemini 2.5 Flash-Lite",
|
||||
reasoning: true,
|
||||
input: ["text", "image"],
|
||||
cost: GOOGLE_GEMINI_COST,
|
||||
contextWindow: 1_048_576,
|
||||
maxTokens: 65_536,
|
||||
},
|
||||
{
|
||||
id: "gemini-3.5-flash",
|
||||
name: "Gemini 3.5 Flash",
|
||||
reasoning: true,
|
||||
input: ["text", "image"],
|
||||
cost: GOOGLE_GEMINI_COST,
|
||||
contextWindow: 1_048_576,
|
||||
maxTokens: 65_536,
|
||||
},
|
||||
{
|
||||
id: "gemini-3.1-pro-preview",
|
||||
name: "Gemini 3.1 Pro Preview",
|
||||
reasoning: true,
|
||||
input: ["text", "image"],
|
||||
cost: GOOGLE_GEMINI_COST,
|
||||
contextWindow: 1_048_576,
|
||||
maxTokens: 65_536,
|
||||
},
|
||||
{
|
||||
id: "gemini-3.1-flash-lite",
|
||||
name: "Gemini 3.1 Flash Lite",
|
||||
reasoning: true,
|
||||
input: ["text", "image"],
|
||||
cost: GOOGLE_GEMINI_COST,
|
||||
contextWindow: 1_048_576,
|
||||
maxTokens: 65_536,
|
||||
},
|
||||
{
|
||||
id: "gemini-3-flash-preview",
|
||||
name: "Gemini 3 Flash Preview",
|
||||
reasoning: true,
|
||||
input: ["text", "image"],
|
||||
cost: GOOGLE_GEMINI_COST,
|
||||
contextWindow: 1_048_576,
|
||||
maxTokens: 65_536,
|
||||
},
|
||||
];
|
||||
|
||||
export function buildGoogleStaticCatalogProvider(): ModelProviderConfig {
|
||||
return {
|
||||
baseUrl: GOOGLE_GEMINI_BASE_URL,
|
||||
api: "google-generative-ai",
|
||||
models: GOOGLE_GEMINI_TEXT_MODELS,
|
||||
};
|
||||
}
|
||||
|
||||
export function buildGoogleVertexStaticCatalogProvider(): ModelProviderConfig {
|
||||
return {
|
||||
baseUrl: GOOGLE_VERTEX_BASE_URL,
|
||||
api: "google-vertex",
|
||||
models: GOOGLE_GEMINI_TEXT_MODELS,
|
||||
};
|
||||
}
|
||||
78
extensions/google/provider-contract-api.ts
Normal file
78
extensions/google/provider-contract-api.ts
Normal file
@@ -0,0 +1,78 @@
|
||||
// Google API module exposes the plugin public contract.
|
||||
import type { ProviderPlugin } from "openclaw/plugin-sdk/provider-model-shared";
|
||||
|
||||
const noopAuth = async () => ({ profiles: [] });
|
||||
|
||||
export function createGoogleProvider(): ProviderPlugin {
|
||||
return {
|
||||
id: "google",
|
||||
label: "Google AI Studio",
|
||||
docsPath: "/providers/models",
|
||||
hookAliases: ["google-antigravity", "google-vertex"],
|
||||
envVars: ["GEMINI_API_KEY", "GOOGLE_API_KEY"],
|
||||
auth: [
|
||||
{
|
||||
id: "api-key",
|
||||
kind: "api_key",
|
||||
label: "Google Gemini API key",
|
||||
hint: "AI Studio / Gemini API key",
|
||||
run: noopAuth,
|
||||
wizard: {
|
||||
choiceId: "gemini-api-key",
|
||||
choiceLabel: "Google Gemini API key",
|
||||
groupId: "google",
|
||||
groupLabel: "Google",
|
||||
groupHint: "Gemini API key + OAuth",
|
||||
},
|
||||
},
|
||||
],
|
||||
};
|
||||
}
|
||||
|
||||
export function createGoogleVertexProvider(): ProviderPlugin {
|
||||
return {
|
||||
id: "google-vertex",
|
||||
label: "Google Vertex AI",
|
||||
docsPath: "/providers/models",
|
||||
envVars: [
|
||||
"GOOGLE_CLOUD_API_KEY",
|
||||
"GOOGLE_CLOUD_PROJECT",
|
||||
"GCLOUD_PROJECT",
|
||||
"GOOGLE_CLOUD_LOCATION",
|
||||
"GOOGLE_APPLICATION_CREDENTIALS",
|
||||
],
|
||||
auth: [],
|
||||
};
|
||||
}
|
||||
|
||||
export function createGoogleGeminiCliProvider(): ProviderPlugin {
|
||||
return {
|
||||
id: "google-gemini-cli",
|
||||
label: "Gemini CLI OAuth",
|
||||
docsPath: "/providers/models",
|
||||
aliases: ["gemini-cli"],
|
||||
envVars: [
|
||||
"OPENCLAW_GEMINI_OAUTH_CLIENT_ID",
|
||||
"OPENCLAW_GEMINI_OAUTH_CLIENT_SECRET",
|
||||
"GEMINI_CLI_OAUTH_CLIENT_ID",
|
||||
"GEMINI_CLI_OAUTH_CLIENT_SECRET",
|
||||
],
|
||||
auth: [
|
||||
{
|
||||
id: "oauth",
|
||||
kind: "oauth",
|
||||
label: "Google OAuth",
|
||||
hint: "PKCE + localhost callback",
|
||||
run: noopAuth,
|
||||
},
|
||||
],
|
||||
wizard: {
|
||||
setup: {
|
||||
choiceId: "google-gemini-cli",
|
||||
choiceLabel: "Gemini CLI OAuth",
|
||||
choiceHint: "Google OAuth with project-aware token payload",
|
||||
methodId: "oauth",
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
27
extensions/google/provider-discovery.ts
Normal file
27
extensions/google/provider-discovery.ts
Normal file
@@ -0,0 +1,27 @@
|
||||
// Google provider module implements model/runtime integration.
|
||||
import type { ProviderPlugin } from "openclaw/plugin-sdk/provider-model-shared";
|
||||
import {
|
||||
buildGoogleStaticCatalogProvider,
|
||||
buildGoogleVertexStaticCatalogProvider,
|
||||
} from "./provider-catalog.js";
|
||||
import { resolveGoogleVertexConfigApiKey } from "./vertex-adc.js";
|
||||
|
||||
const googleProviderDiscovery: ProviderPlugin = {
|
||||
id: "google",
|
||||
label: "Google AI Studio",
|
||||
docsPath: "/providers/models",
|
||||
auth: [],
|
||||
resolveConfigApiKey: ({ provider, env }) =>
|
||||
provider === "google-vertex" ? resolveGoogleVertexConfigApiKey(env) : undefined,
|
||||
staticCatalog: {
|
||||
order: "simple",
|
||||
run: async () => ({
|
||||
providers: {
|
||||
google: buildGoogleStaticCatalogProvider(),
|
||||
"google-vertex": buildGoogleVertexStaticCatalogProvider(),
|
||||
},
|
||||
}),
|
||||
},
|
||||
};
|
||||
|
||||
export default googleProviderDiscovery;
|
||||
19
extensions/google/provider-hooks.ts
Normal file
19
extensions/google/provider-hooks.ts
Normal file
@@ -0,0 +1,19 @@
|
||||
// Google provider module implements model/runtime integration.
|
||||
import type {
|
||||
ProviderDefaultThinkingPolicyContext,
|
||||
ProviderThinkingProfile,
|
||||
} from "openclaw/plugin-sdk/core";
|
||||
import { buildProviderReplayFamilyHooks } from "openclaw/plugin-sdk/provider-model-shared";
|
||||
import { buildProviderToolCompatFamilyHooks } from "openclaw/plugin-sdk/provider-tools";
|
||||
import { resolveGoogleThinkingProfile } from "./provider-policy.js";
|
||||
import { createGoogleThinkingStreamWrapper } from "./thinking-api.js";
|
||||
|
||||
export const GOOGLE_GEMINI_PROVIDER_HOOKS = {
|
||||
...buildProviderReplayFamilyHooks({
|
||||
family: "google-gemini",
|
||||
}),
|
||||
...buildProviderToolCompatFamilyHooks("gemini"),
|
||||
resolveThinkingProfile: (context: ProviderDefaultThinkingPolicyContext) =>
|
||||
resolveGoogleThinkingProfile(context) satisfies ProviderThinkingProfile | undefined,
|
||||
wrapStreamFn: createGoogleThinkingStreamWrapper,
|
||||
};
|
||||
532
extensions/google/provider-models.test.ts
Normal file
532
extensions/google/provider-models.test.ts
Normal file
@@ -0,0 +1,532 @@
|
||||
// Google tests cover provider models plugin behavior.
|
||||
import type { ProviderRuntimeModel } from "openclaw/plugin-sdk/plugin-entry";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { createProviderDynamicModelContext as createContext } from "../test-support/provider-model-test-helpers.js";
|
||||
import { isModernGoogleModel, resolveGoogleGeminiForwardCompatModel } from "./provider-models.js";
|
||||
|
||||
function createTemplateModel(
|
||||
provider: string,
|
||||
id: string,
|
||||
overrides: Partial<ProviderRuntimeModel> = {},
|
||||
): ProviderRuntimeModel {
|
||||
return {
|
||||
id,
|
||||
name: id,
|
||||
provider,
|
||||
api: provider === "google-gemini-cli" ? "google-gemini-cli" : "google-generative-ai",
|
||||
baseUrl:
|
||||
provider === "google-gemini-cli"
|
||||
? "https://cloudcode-pa.googleapis.com"
|
||||
: "https://generativelanguage.googleapis.com/v1beta",
|
||||
reasoning: false,
|
||||
input: ["text", "image"],
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 200_000,
|
||||
maxTokens: 64_000,
|
||||
...overrides,
|
||||
} as ProviderRuntimeModel;
|
||||
}
|
||||
|
||||
function expectModelFields(
|
||||
model: ProviderRuntimeModel | undefined,
|
||||
fields: Partial<ProviderRuntimeModel>,
|
||||
) {
|
||||
if (!model) {
|
||||
throw new Error("expected provider model");
|
||||
}
|
||||
for (const [key, value] of Object.entries(fields)) {
|
||||
expect(model[key as keyof ProviderRuntimeModel]).toEqual(value);
|
||||
}
|
||||
}
|
||||
|
||||
describe("resolveGoogleGeminiForwardCompatModel", () => {
|
||||
it("resolves stable gemini 2.5 flash-lite from direct google templates for Gemini CLI when available", () => {
|
||||
const model = resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: "google-gemini-cli",
|
||||
ctx: createContext({
|
||||
provider: "google-gemini-cli",
|
||||
modelId: "gemini-2.5-flash-lite",
|
||||
models: [createTemplateModel("google", "gemini-2.5-flash-lite")],
|
||||
}),
|
||||
});
|
||||
|
||||
expectModelFields(model, {
|
||||
provider: "google-gemini-cli",
|
||||
id: "gemini-2.5-flash-lite",
|
||||
api: "google-generative-ai",
|
||||
reasoning: false,
|
||||
});
|
||||
});
|
||||
|
||||
it("resolves stable gemini 2.5 flash-lite from Gemini CLI templates when direct google templates are unavailable", () => {
|
||||
const model = resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: "google-gemini-cli",
|
||||
ctx: createContext({
|
||||
provider: "google-gemini-cli",
|
||||
modelId: "gemini-2.5-flash-lite",
|
||||
models: [
|
||||
createTemplateModel("google-gemini-cli", "gemini-3.1-flash-lite", {
|
||||
contextWindow: 1_048_576,
|
||||
api: "google-gemini-cli",
|
||||
baseUrl: "https://cloudcode-pa.googleapis.com",
|
||||
}),
|
||||
],
|
||||
}),
|
||||
});
|
||||
|
||||
expectModelFields(model, {
|
||||
provider: "google-gemini-cli",
|
||||
id: "gemini-2.5-flash-lite",
|
||||
api: "google-gemini-cli",
|
||||
contextWindow: 1_048_576,
|
||||
reasoning: false,
|
||||
});
|
||||
});
|
||||
|
||||
it("resolves gemini 3.1 pro for google aliases via an alternate template provider", () => {
|
||||
const model = resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: "google-vertex",
|
||||
ctx: createContext({
|
||||
provider: "google-vertex",
|
||||
modelId: "gemini-3.1-pro-preview",
|
||||
models: [createTemplateModel("google-gemini-cli", "gemini-3-pro-preview")],
|
||||
}),
|
||||
});
|
||||
|
||||
expectModelFields(model, {
|
||||
provider: "google-vertex",
|
||||
id: "gemini-3.1-pro-preview",
|
||||
api: "google-gemini-cli",
|
||||
reasoning: false,
|
||||
});
|
||||
});
|
||||
|
||||
it("canonicalizes retired Gemini 3 Pro preview requests before cloning templates", () => {
|
||||
const model = resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: "google",
|
||||
ctx: createContext({
|
||||
provider: "google",
|
||||
modelId: "gemini-3-pro-preview",
|
||||
models: [createTemplateModel("google", "gemini-3-pro-preview")],
|
||||
}),
|
||||
});
|
||||
|
||||
expectModelFields(model, {
|
||||
provider: "google",
|
||||
id: "gemini-3.1-pro-preview",
|
||||
api: "google-generative-ai",
|
||||
reasoning: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("canonicalizes provider-qualified retired Gemini 3 Pro preview requests", () => {
|
||||
const model = resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: "google",
|
||||
ctx: createContext({
|
||||
provider: "google",
|
||||
modelId: "google/gemini-3-pro-preview",
|
||||
models: [createTemplateModel("google", "gemini-3.1-pro-preview")],
|
||||
}),
|
||||
});
|
||||
|
||||
expectModelFields(model, {
|
||||
provider: "google",
|
||||
id: "google/gemini-3.1-pro-preview",
|
||||
api: "google-generative-ai",
|
||||
reasoning: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps Gemini CLI 3.1 clones sourced from CLI templates when both catalogs exist", () => {
|
||||
const model = resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: "google-gemini-cli",
|
||||
ctx: createContext({
|
||||
provider: "google-gemini-cli",
|
||||
modelId: "gemini-3.1-pro-preview",
|
||||
models: [
|
||||
createTemplateModel("google-gemini-cli", "gemini-3-pro-preview", {
|
||||
api: "google-gemini-cli",
|
||||
baseUrl: "https://cloudcode-pa.googleapis.com",
|
||||
contextWindow: 1_048_576,
|
||||
}),
|
||||
createTemplateModel("google", "gemini-3-pro-preview", {
|
||||
api: "google-generative-ai",
|
||||
baseUrl: "https://generativelanguage.googleapis.com/v1beta",
|
||||
contextWindow: 200_000,
|
||||
}),
|
||||
],
|
||||
}),
|
||||
});
|
||||
|
||||
expectModelFields(model, {
|
||||
provider: "google-gemini-cli",
|
||||
id: "gemini-3.1-pro-preview",
|
||||
api: "google-gemini-cli",
|
||||
baseUrl: "https://cloudcode-pa.googleapis.com",
|
||||
contextWindow: 1_048_576,
|
||||
});
|
||||
});
|
||||
|
||||
it("prefers current Gemini 3.1 Pro templates over retired Gemini 3 Pro templates", () => {
|
||||
const model = resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: "google-gemini-cli",
|
||||
ctx: createContext({
|
||||
provider: "google-gemini-cli",
|
||||
modelId: "gemini-3.1-pro-preview",
|
||||
models: [
|
||||
createTemplateModel("google-gemini-cli", "gemini-3-pro-preview", {
|
||||
contextWindow: 100_000,
|
||||
}),
|
||||
createTemplateModel("google-gemini-cli", "gemini-3.1-pro-preview", {
|
||||
contextWindow: 1_048_576,
|
||||
}),
|
||||
],
|
||||
}),
|
||||
});
|
||||
|
||||
expectModelFields(model, {
|
||||
provider: "google-gemini-cli",
|
||||
id: "gemini-3.1-pro-preview",
|
||||
contextWindow: 1_048_576,
|
||||
});
|
||||
});
|
||||
|
||||
it("preserves template reasoning metadata instead of forcing it on forward-compat clones", () => {
|
||||
const model = resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: "google",
|
||||
ctx: createContext({
|
||||
provider: "google",
|
||||
modelId: "gemini-3.1-flash-preview",
|
||||
models: [
|
||||
createTemplateModel("google-gemini-cli", "gemini-3-flash-preview", {
|
||||
reasoning: true,
|
||||
}),
|
||||
],
|
||||
}),
|
||||
});
|
||||
|
||||
expectModelFields(model, {
|
||||
provider: "google",
|
||||
id: "gemini-3.1-flash-preview",
|
||||
api: "google-gemini-cli",
|
||||
reasoning: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("resolves gemini 3.1 flash from direct google templates", () => {
|
||||
const model = resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: "google",
|
||||
ctx: createContext({
|
||||
provider: "google",
|
||||
modelId: "gemini-3.1-flash-preview",
|
||||
models: [
|
||||
createTemplateModel("google", "gemini-3-flash-preview", {
|
||||
reasoning: false,
|
||||
}),
|
||||
],
|
||||
}),
|
||||
});
|
||||
|
||||
expectModelFields(model, {
|
||||
provider: "google",
|
||||
id: "gemini-3.1-flash-preview",
|
||||
api: "google-generative-ai",
|
||||
reasoning: false,
|
||||
});
|
||||
});
|
||||
|
||||
it("resolves canonical gemini 3 flash from older Google flash templates when the exact row is missing", () => {
|
||||
const model = resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: "google",
|
||||
ctx: createContext({
|
||||
provider: "google",
|
||||
modelId: "gemini-3-flash-preview",
|
||||
models: [
|
||||
createTemplateModel("google", "gemini-2.5-flash", {
|
||||
contextWindow: 1_048_576,
|
||||
reasoning: true,
|
||||
}),
|
||||
],
|
||||
}),
|
||||
});
|
||||
|
||||
expectModelFields(model, {
|
||||
provider: "google",
|
||||
id: "gemini-3-flash-preview",
|
||||
api: "google-generative-ai",
|
||||
input: ["text", "image"],
|
||||
contextWindow: 1_048_576,
|
||||
reasoning: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("resolves canonical Gemini CLI 3 flash from Google flash templates when the CLI row is missing", () => {
|
||||
const model = resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: "google-gemini-cli",
|
||||
ctx: createContext({
|
||||
provider: "google-gemini-cli",
|
||||
modelId: "gemini-3-flash-preview",
|
||||
models: [
|
||||
createTemplateModel("google", "gemini-2.5-flash", {
|
||||
contextWindow: 1_048_576,
|
||||
}),
|
||||
],
|
||||
}),
|
||||
});
|
||||
|
||||
expectModelFields(model, {
|
||||
provider: "google-gemini-cli",
|
||||
id: "gemini-3-flash-preview",
|
||||
api: "google-generative-ai",
|
||||
input: ["text", "image"],
|
||||
contextWindow: 1_048_576,
|
||||
});
|
||||
});
|
||||
|
||||
it("resolves Gemini latest aliases from current Google templates", () => {
|
||||
const models = [
|
||||
createTemplateModel("google", "gemini-3-pro-preview", { reasoning: true }),
|
||||
createTemplateModel("google", "gemini-3-flash-preview", { reasoning: true }),
|
||||
createTemplateModel("google", "gemini-3.1-flash-lite", { reasoning: true }),
|
||||
];
|
||||
|
||||
expectModelFields(
|
||||
resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: "google",
|
||||
ctx: createContext({ provider: "google", modelId: "gemini-pro-latest", models }),
|
||||
}),
|
||||
{
|
||||
provider: "google",
|
||||
id: "gemini-pro-latest",
|
||||
api: "google-generative-ai",
|
||||
reasoning: true,
|
||||
},
|
||||
);
|
||||
expectModelFields(
|
||||
resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: "google",
|
||||
ctx: createContext({ provider: "google", modelId: "gemini-flash-latest", models }),
|
||||
}),
|
||||
{
|
||||
provider: "google",
|
||||
id: "gemini-flash-latest",
|
||||
api: "google-generative-ai",
|
||||
reasoning: true,
|
||||
},
|
||||
);
|
||||
expectModelFields(
|
||||
resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: "google",
|
||||
ctx: createContext({ provider: "google", modelId: "gemini-flash-lite-latest", models }),
|
||||
}),
|
||||
{
|
||||
provider: "google",
|
||||
id: "gemini-flash-lite-latest",
|
||||
api: "google-generative-ai",
|
||||
reasoning: true,
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
it("resolves Antigravity Gemini 3.1 pro customtools from the low template", () => {
|
||||
const model = resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: "google-antigravity",
|
||||
ctx: createContext({
|
||||
provider: "google-antigravity",
|
||||
modelId: "gemini-3.1-pro-preview-customtools",
|
||||
models: [
|
||||
createTemplateModel("google-antigravity", "gemini-3-pro-low", {
|
||||
api: "openai-completions",
|
||||
baseUrl: "https://antigravity.example/v1",
|
||||
contextWindow: 1_048_576,
|
||||
reasoning: true,
|
||||
}),
|
||||
],
|
||||
}),
|
||||
});
|
||||
|
||||
expectModelFields(model, {
|
||||
provider: "google-antigravity",
|
||||
id: "gemini-3.1-pro-preview-customtools",
|
||||
api: "openai-completions",
|
||||
baseUrl: "https://antigravity.example/v1",
|
||||
contextWindow: 1_048_576,
|
||||
reasoning: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("falls back to the Antigravity high template when the low template is unavailable", () => {
|
||||
const model = resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: "google-antigravity",
|
||||
ctx: createContext({
|
||||
provider: "google-antigravity",
|
||||
modelId: "gemini-3.1-pro-preview",
|
||||
models: [
|
||||
createTemplateModel("google-antigravity", "gemini-3-pro-high", {
|
||||
api: "openai-completions",
|
||||
maxTokens: 65_536,
|
||||
reasoning: true,
|
||||
}),
|
||||
],
|
||||
}),
|
||||
});
|
||||
|
||||
expectModelFields(model, {
|
||||
provider: "google-antigravity",
|
||||
id: "gemini-3.1-pro-preview",
|
||||
api: "openai-completions",
|
||||
maxTokens: 65_536,
|
||||
reasoning: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("resolves Antigravity Gemini 3.1 flash variants from the flash template", () => {
|
||||
const models = [
|
||||
createTemplateModel("google-antigravity", "gemini-3-flash", {
|
||||
api: "openai-completions",
|
||||
contextWindow: 1_048_576,
|
||||
}),
|
||||
];
|
||||
|
||||
expectModelFields(
|
||||
resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: "google-antigravity",
|
||||
ctx: createContext({
|
||||
provider: "google-antigravity",
|
||||
modelId: "gemini-3.1-flash-preview",
|
||||
models,
|
||||
}),
|
||||
}),
|
||||
{
|
||||
provider: "google-antigravity",
|
||||
id: "gemini-3.1-flash-preview",
|
||||
api: "openai-completions",
|
||||
contextWindow: 1_048_576,
|
||||
},
|
||||
);
|
||||
|
||||
expectModelFields(
|
||||
resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: "google-antigravity",
|
||||
ctx: createContext({
|
||||
provider: "google-antigravity",
|
||||
modelId: "gemini-3.1-flash-lite",
|
||||
models,
|
||||
}),
|
||||
}),
|
||||
{
|
||||
provider: "google-antigravity",
|
||||
id: "gemini-3.1-flash-lite",
|
||||
api: "openai-completions",
|
||||
contextWindow: 1_048_576,
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
it("returns undefined for Antigravity Gemini 3.1 models without a matching template", () => {
|
||||
const model = resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: "google-antigravity",
|
||||
ctx: createContext({
|
||||
provider: "google-antigravity",
|
||||
modelId: "gemini-3.1-pro-preview-customtools",
|
||||
models: [createTemplateModel("google-antigravity", "claude-opus-4-6-thinking")],
|
||||
}),
|
||||
});
|
||||
|
||||
expect(model).toBeUndefined();
|
||||
});
|
||||
|
||||
it("prefers the flash-lite template before the broader flash prefix", () => {
|
||||
const model = resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: "google-vertex",
|
||||
ctx: createContext({
|
||||
provider: "google-vertex",
|
||||
modelId: "gemini-3.1-flash-lite",
|
||||
models: [
|
||||
createTemplateModel("google-gemini-cli", "gemini-3-flash-preview", {
|
||||
contextWindow: 128_000,
|
||||
}),
|
||||
createTemplateModel("google-gemini-cli", "gemini-3.1-flash-lite", {
|
||||
contextWindow: 1_048_576,
|
||||
}),
|
||||
],
|
||||
}),
|
||||
});
|
||||
|
||||
expectModelFields(model, {
|
||||
provider: "google-vertex",
|
||||
id: "gemini-3.1-flash-lite",
|
||||
contextWindow: 1_048_576,
|
||||
reasoning: false,
|
||||
});
|
||||
});
|
||||
|
||||
it("treats gemini 2.5 ids as modern google models", () => {
|
||||
expect(isModernGoogleModel("gemini-2.5-pro")).toBe(true);
|
||||
expect(isModernGoogleModel("gemini-2.5-flash-lite")).toBe(true);
|
||||
expect(isModernGoogleModel("gemini-1.5-pro")).toBe(false);
|
||||
});
|
||||
|
||||
it("treats Gemini latest aliases as modern google models", () => {
|
||||
expect(isModernGoogleModel("gemini-pro-latest")).toBe(true);
|
||||
expect(isModernGoogleModel("gemini-flash-latest")).toBe(true);
|
||||
expect(isModernGoogleModel("gemini-flash-lite-latest")).toBe(true);
|
||||
});
|
||||
|
||||
it("treats gemma models as modern google models", () => {
|
||||
expect(isModernGoogleModel("gemma-4-26b-a4b-it")).toBe(true);
|
||||
expect(isModernGoogleModel("gemma-3-4b-it")).toBe(true);
|
||||
});
|
||||
|
||||
it("resolves Gemma 4 models with reasoning enabled regardless of template", () => {
|
||||
const model = resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: "google",
|
||||
ctx: createContext({
|
||||
provider: "google",
|
||||
modelId: "gemma-4-26b-a4b-it",
|
||||
models: [createTemplateModel("google", "gemini-3-flash-preview", { reasoning: false })],
|
||||
}),
|
||||
});
|
||||
|
||||
expectModelFields(model, {
|
||||
provider: "google",
|
||||
id: "gemma-4-26b-a4b-it",
|
||||
reasoning: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("canonicalizes Gemma 4 26B shorthand before cloning templates", () => {
|
||||
const model = resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: "google",
|
||||
ctx: createContext({
|
||||
provider: "google",
|
||||
modelId: "gemma-4-26b",
|
||||
models: [createTemplateModel("google", "gemini-3-flash-preview", { reasoning: false })],
|
||||
}),
|
||||
});
|
||||
|
||||
expectModelFields(model, {
|
||||
provider: "google",
|
||||
id: "gemma-4-26b-a4b-it",
|
||||
api: "google-generative-ai",
|
||||
reasoning: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("preserves template reasoning for non-Gemma 4 gemma models", () => {
|
||||
const model = resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: "google",
|
||||
ctx: createContext({
|
||||
provider: "google",
|
||||
modelId: "gemma-3-4b-it",
|
||||
models: [createTemplateModel("google", "gemini-3-flash-preview", { reasoning: false })],
|
||||
}),
|
||||
});
|
||||
|
||||
expectModelFields(model, {
|
||||
provider: "google",
|
||||
id: "gemma-3-4b-it",
|
||||
reasoning: false,
|
||||
});
|
||||
});
|
||||
});
|
||||
244
extensions/google/provider-models.ts
Normal file
244
extensions/google/provider-models.ts
Normal file
@@ -0,0 +1,244 @@
|
||||
// Google provider module implements model/runtime integration.
|
||||
import type {
|
||||
ProviderResolveDynamicModelContext,
|
||||
ProviderRuntimeModel,
|
||||
} from "openclaw/plugin-sdk/plugin-entry";
|
||||
import { cloneFirstTemplateModel } from "openclaw/plugin-sdk/provider-model-shared";
|
||||
import { normalizeOptionalLowercaseString } from "openclaw/plugin-sdk/string-coerce-runtime";
|
||||
import { normalizeGoogleModelId } from "./model-id.js";
|
||||
|
||||
const GOOGLE_GEMINI_CLI_PROVIDER_ID = "google-gemini-cli";
|
||||
const GOOGLE_ANTIGRAVITY_PROVIDER_ID = "google-antigravity";
|
||||
const GEMINI_2_5_PRO_PREFIX = "gemini-2.5-pro";
|
||||
const GEMINI_2_5_FLASH_LITE_PREFIX = "gemini-2.5-flash-lite";
|
||||
const GEMINI_2_5_FLASH_PREFIX = "gemini-2.5-flash";
|
||||
const GEMINI_3_1_PRO_PREFIX = "gemini-3.1-pro";
|
||||
const GEMINI_3_1_FLASH_LITE_PREFIX = "gemini-3.1-flash-lite";
|
||||
const GEMINI_3_1_FLASH_PREFIX = "gemini-3.1-flash";
|
||||
const GEMINI_3_FLASH_LITE_PREFIX = "gemini-3-flash-lite";
|
||||
const GEMINI_3_FLASH_PREFIX = "gemini-3-flash";
|
||||
const GEMINI_3_5_FLASH_PREFIX = "gemini-3.5-flash";
|
||||
const GEMINI_PRO_LATEST_ID = "gemini-pro-latest";
|
||||
const GEMINI_FLASH_LATEST_ID = "gemini-flash-latest";
|
||||
const GEMINI_FLASH_LITE_LATEST_ID = "gemini-flash-lite-latest";
|
||||
const GEMMA_PREFIX = "gemma-";
|
||||
const GEMINI_2_5_PRO_TEMPLATE_IDS = ["gemini-2.5-pro"] as const;
|
||||
const GEMINI_2_5_FLASH_LITE_TEMPLATE_IDS = ["gemini-2.5-flash-lite"] as const;
|
||||
const GEMINI_2_5_FLASH_TEMPLATE_IDS = ["gemini-2.5-flash"] as const;
|
||||
const GEMINI_3_1_PRO_TEMPLATE_IDS = ["gemini-3.1-pro-preview", "gemini-3-pro-preview"] as const;
|
||||
const GEMINI_3_1_FLASH_LITE_TEMPLATE_IDS = ["gemini-3.1-flash-lite"] as const;
|
||||
const GEMINI_3_1_FLASH_TEMPLATE_IDS = ["gemini-3-flash-preview", "gemini-2.5-flash"] as const;
|
||||
const GEMINI_3_PRO_ANTIGRAVITY_TEMPLATE_IDS = ["gemini-3-pro-low", "gemini-3-pro-high"] as const;
|
||||
const GEMINI_3_FLASH_ANTIGRAVITY_TEMPLATE_IDS = ["gemini-3-flash"] as const;
|
||||
// Gemma uses the Gemini flash template as a forward-compat approximation
|
||||
// until a dedicated Gemma template is registered in the catalog.
|
||||
const GEMMA_TEMPLATE_IDS = GEMINI_3_1_FLASH_TEMPLATE_IDS;
|
||||
const GOOGLE_PROVIDER_PREFIX = "google/";
|
||||
|
||||
function normalizeGeminiProRequestId(id: string): string {
|
||||
if (id.startsWith(GOOGLE_PROVIDER_PREFIX)) {
|
||||
const modelId = id.slice(GOOGLE_PROVIDER_PREFIX.length);
|
||||
const normalizedModelId = normalizeGeminiProRequestId(modelId);
|
||||
return normalizedModelId === modelId ? id : `${GOOGLE_PROVIDER_PREFIX}${normalizedModelId}`;
|
||||
}
|
||||
if (id === "gemini-3-pro" || id === "gemini-3-pro-preview" || id === "gemini-3.1-pro") {
|
||||
return "gemini-3.1-pro-preview";
|
||||
}
|
||||
if (id === "gemma-4-26b") {
|
||||
return normalizeGoogleModelId(id);
|
||||
}
|
||||
return id;
|
||||
}
|
||||
|
||||
function googleFamilyModelId(id: string): string {
|
||||
return id.startsWith(GOOGLE_PROVIDER_PREFIX) ? id.slice(GOOGLE_PROVIDER_PREFIX.length) : id;
|
||||
}
|
||||
|
||||
type GoogleForwardCompatFamily = {
|
||||
googleTemplateIds: readonly string[];
|
||||
cliTemplateIds: readonly string[];
|
||||
antigravityTemplateIds?: readonly string[];
|
||||
preferExternalFirstForCli?: boolean;
|
||||
};
|
||||
|
||||
type GoogleTemplateSource = {
|
||||
templateProviderId: string;
|
||||
templateIds: readonly string[];
|
||||
};
|
||||
|
||||
function cloneGoogleTemplateModel(params: {
|
||||
providerId: string;
|
||||
modelId: string;
|
||||
templateProviderId: string;
|
||||
templateIds: readonly string[];
|
||||
ctx: ProviderResolveDynamicModelContext;
|
||||
patch?: Partial<ProviderRuntimeModel>;
|
||||
}): ProviderRuntimeModel | undefined {
|
||||
return cloneFirstTemplateModel({
|
||||
providerId: params.templateProviderId,
|
||||
modelId: params.modelId,
|
||||
templateIds: params.templateIds,
|
||||
ctx: params.ctx,
|
||||
patch: {
|
||||
...params.patch,
|
||||
provider: params.providerId,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function isGoogleGeminiCliProvider(providerId: string): boolean {
|
||||
return normalizeOptionalLowercaseString(providerId) === GOOGLE_GEMINI_CLI_PROVIDER_ID;
|
||||
}
|
||||
|
||||
function isGoogleAntigravityProvider(providerId: string): boolean {
|
||||
return normalizeOptionalLowercaseString(providerId) === GOOGLE_ANTIGRAVITY_PROVIDER_ID;
|
||||
}
|
||||
|
||||
function templateIdsForProvider(
|
||||
templateProviderId: string,
|
||||
family: GoogleForwardCompatFamily,
|
||||
): readonly string[] {
|
||||
if (isGoogleGeminiCliProvider(templateProviderId)) {
|
||||
return family.cliTemplateIds;
|
||||
}
|
||||
if (isGoogleAntigravityProvider(templateProviderId)) {
|
||||
return family.antigravityTemplateIds ?? family.googleTemplateIds;
|
||||
}
|
||||
return family.googleTemplateIds;
|
||||
}
|
||||
|
||||
function buildGoogleTemplateSources(params: {
|
||||
providerId: string;
|
||||
templateProviderId?: string;
|
||||
family: GoogleForwardCompatFamily;
|
||||
}): GoogleTemplateSource[] {
|
||||
const defaultTemplateProviderId = params.templateProviderId?.trim()
|
||||
? params.templateProviderId
|
||||
: isGoogleGeminiCliProvider(params.providerId)
|
||||
? "google"
|
||||
: GOOGLE_GEMINI_CLI_PROVIDER_ID;
|
||||
const preferredExternalFirst =
|
||||
isGoogleGeminiCliProvider(params.providerId) &&
|
||||
params.family.preferExternalFirstForCli === true;
|
||||
const orderedTemplateProviderIds = preferredExternalFirst
|
||||
? [defaultTemplateProviderId, params.providerId]
|
||||
: [params.providerId, defaultTemplateProviderId];
|
||||
|
||||
const seen = new Set<string>();
|
||||
const sources: GoogleTemplateSource[] = [];
|
||||
for (const providerId of orderedTemplateProviderIds) {
|
||||
const trimmed = providerId?.trim();
|
||||
if (!trimmed || seen.has(trimmed)) {
|
||||
continue;
|
||||
}
|
||||
seen.add(trimmed);
|
||||
sources.push({
|
||||
templateProviderId: trimmed,
|
||||
templateIds: templateIdsForProvider(trimmed, params.family),
|
||||
});
|
||||
}
|
||||
return sources;
|
||||
}
|
||||
|
||||
export function resolveGoogleGeminiForwardCompatModel(params: {
|
||||
providerId: string;
|
||||
templateProviderId?: string;
|
||||
ctx: ProviderResolveDynamicModelContext;
|
||||
}): ProviderRuntimeModel | undefined {
|
||||
const trimmed = normalizeGeminiProRequestId(params.ctx.modelId.trim());
|
||||
const lower = normalizeOptionalLowercaseString(googleFamilyModelId(trimmed)) ?? "";
|
||||
|
||||
let family: GoogleForwardCompatFamily;
|
||||
let patch: Partial<ProviderRuntimeModel> | undefined;
|
||||
if (lower.startsWith(GEMINI_2_5_PRO_PREFIX)) {
|
||||
family = {
|
||||
googleTemplateIds: GEMINI_2_5_PRO_TEMPLATE_IDS,
|
||||
cliTemplateIds: GEMINI_3_1_PRO_TEMPLATE_IDS,
|
||||
preferExternalFirstForCli: true,
|
||||
};
|
||||
} else if (lower.startsWith(GEMINI_2_5_FLASH_LITE_PREFIX)) {
|
||||
family = {
|
||||
googleTemplateIds: GEMINI_2_5_FLASH_LITE_TEMPLATE_IDS,
|
||||
cliTemplateIds: GEMINI_3_1_FLASH_LITE_TEMPLATE_IDS,
|
||||
preferExternalFirstForCli: true,
|
||||
};
|
||||
} else if (lower.startsWith(GEMINI_2_5_FLASH_PREFIX)) {
|
||||
family = {
|
||||
googleTemplateIds: GEMINI_2_5_FLASH_TEMPLATE_IDS,
|
||||
cliTemplateIds: GEMINI_3_1_FLASH_TEMPLATE_IDS,
|
||||
preferExternalFirstForCli: true,
|
||||
};
|
||||
} else if (lower.startsWith(GEMINI_3_1_PRO_PREFIX) || lower === GEMINI_PRO_LATEST_ID) {
|
||||
family = {
|
||||
googleTemplateIds: GEMINI_3_1_PRO_TEMPLATE_IDS,
|
||||
cliTemplateIds: GEMINI_3_1_PRO_TEMPLATE_IDS,
|
||||
antigravityTemplateIds: GEMINI_3_PRO_ANTIGRAVITY_TEMPLATE_IDS,
|
||||
};
|
||||
if (params.providerId === "google" || params.providerId === GOOGLE_GEMINI_CLI_PROVIDER_ID) {
|
||||
patch = { reasoning: true };
|
||||
}
|
||||
} else if (
|
||||
lower.startsWith(GEMINI_3_1_FLASH_LITE_PREFIX) ||
|
||||
lower.startsWith(GEMINI_3_FLASH_LITE_PREFIX) ||
|
||||
lower === GEMINI_FLASH_LITE_LATEST_ID
|
||||
) {
|
||||
family = {
|
||||
googleTemplateIds: GEMINI_3_1_FLASH_LITE_TEMPLATE_IDS,
|
||||
cliTemplateIds: GEMINI_3_1_FLASH_LITE_TEMPLATE_IDS,
|
||||
antigravityTemplateIds: GEMINI_3_FLASH_ANTIGRAVITY_TEMPLATE_IDS,
|
||||
};
|
||||
} else if (
|
||||
lower.startsWith(GEMINI_3_1_FLASH_PREFIX) ||
|
||||
lower.startsWith(GEMINI_3_5_FLASH_PREFIX) ||
|
||||
lower.startsWith(GEMINI_3_FLASH_PREFIX) ||
|
||||
lower === GEMINI_FLASH_LATEST_ID
|
||||
) {
|
||||
family = {
|
||||
googleTemplateIds: GEMINI_3_1_FLASH_TEMPLATE_IDS,
|
||||
cliTemplateIds: GEMINI_3_1_FLASH_TEMPLATE_IDS,
|
||||
antigravityTemplateIds: GEMINI_3_FLASH_ANTIGRAVITY_TEMPLATE_IDS,
|
||||
};
|
||||
} else if (lower.startsWith(GEMMA_PREFIX)) {
|
||||
family = {
|
||||
googleTemplateIds: GEMMA_TEMPLATE_IDS,
|
||||
cliTemplateIds: GEMMA_TEMPLATE_IDS,
|
||||
};
|
||||
if (lower.startsWith("gemma-4")) {
|
||||
patch = { reasoning: true };
|
||||
}
|
||||
} else {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
for (const source of buildGoogleTemplateSources({
|
||||
providerId: params.providerId,
|
||||
templateProviderId: params.templateProviderId,
|
||||
family,
|
||||
})) {
|
||||
const model = cloneGoogleTemplateModel({
|
||||
providerId: params.providerId,
|
||||
modelId: trimmed,
|
||||
templateProviderId: source.templateProviderId,
|
||||
templateIds: source.templateIds,
|
||||
ctx: params.ctx,
|
||||
patch,
|
||||
});
|
||||
if (model) {
|
||||
return model;
|
||||
}
|
||||
}
|
||||
|
||||
return undefined;
|
||||
}
|
||||
|
||||
export function isModernGoogleModel(modelId: string): boolean {
|
||||
const lower = normalizeOptionalLowercaseString(modelId) ?? "";
|
||||
return (
|
||||
lower.startsWith("gemini-2.5") ||
|
||||
lower.startsWith("gemini-3") ||
|
||||
lower === GEMINI_PRO_LATEST_ID ||
|
||||
lower === GEMINI_FLASH_LATEST_ID ||
|
||||
lower === GEMINI_FLASH_LITE_LATEST_ID ||
|
||||
lower.startsWith(GEMMA_PREFIX)
|
||||
);
|
||||
}
|
||||
202
extensions/google/provider-policy-api.test.ts
Normal file
202
extensions/google/provider-policy-api.test.ts
Normal file
@@ -0,0 +1,202 @@
|
||||
// Google tests cover provider policy api plugin behavior.
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { normalizeConfig, resolveThinkingProfile } from "./provider-policy-api.js";
|
||||
|
||||
describe("google provider policy public artifact", () => {
|
||||
it("normalizes Google provider config without loading the full provider plugin", () => {
|
||||
expect(
|
||||
normalizeConfig({
|
||||
provider: "google",
|
||||
providerConfig: {
|
||||
baseUrl: "https://generativelanguage.googleapis.com",
|
||||
api: "google-generative-ai",
|
||||
apiKey: "GEMINI_API_KEY",
|
||||
models: [
|
||||
{
|
||||
id: "gemini-3-pro",
|
||||
name: "Gemini 3 Pro",
|
||||
reasoning: true,
|
||||
input: ["text", "image"],
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 1_048_576,
|
||||
maxTokens: 65_536,
|
||||
},
|
||||
],
|
||||
},
|
||||
}),
|
||||
).toEqual({
|
||||
api: "google-generative-ai",
|
||||
apiKey: "GEMINI_API_KEY",
|
||||
baseUrl: "https://generativelanguage.googleapis.com/v1beta",
|
||||
models: [
|
||||
{
|
||||
id: "gemini-3.1-pro-preview",
|
||||
name: "Gemini 3 Pro",
|
||||
reasoning: true,
|
||||
input: ["text", "image"],
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 1_048_576,
|
||||
maxTokens: 65_536,
|
||||
},
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
it("preserves explicit OpenAI-compatible Google endpoints during normalization", () => {
|
||||
expect(
|
||||
normalizeConfig({
|
||||
provider: "google",
|
||||
providerConfig: {
|
||||
baseUrl: "https://generativelanguage.googleapis.com/v1beta/openai",
|
||||
api: "openai-completions",
|
||||
models: [],
|
||||
},
|
||||
}),
|
||||
).toEqual({
|
||||
baseUrl: "https://generativelanguage.googleapis.com/v1beta/openai",
|
||||
api: "openai-completions",
|
||||
models: [],
|
||||
});
|
||||
});
|
||||
|
||||
it("normalizes retired Google model ids even for explicit OpenAI-compatible endpoints", () => {
|
||||
expect(
|
||||
normalizeConfig({
|
||||
provider: "google",
|
||||
providerConfig: {
|
||||
baseUrl: "https://generativelanguage.googleapis.com/v1beta/openai",
|
||||
api: "openai-completions",
|
||||
models: [
|
||||
{
|
||||
id: "google/gemini-3-pro-preview",
|
||||
name: "Gemini 3 Pro",
|
||||
reasoning: true,
|
||||
input: ["text", "image"],
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 1_048_576,
|
||||
maxTokens: 65_536,
|
||||
},
|
||||
],
|
||||
},
|
||||
}),
|
||||
).toEqual({
|
||||
baseUrl: "https://generativelanguage.googleapis.com/v1beta/openai",
|
||||
api: "openai-completions",
|
||||
models: [
|
||||
{
|
||||
id: "google/gemini-3.1-pro-preview",
|
||||
name: "Gemini 3 Pro",
|
||||
reasoning: true,
|
||||
input: ["text", "image"],
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 1_048_576,
|
||||
maxTokens: 65_536,
|
||||
},
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
it("normalizes retired Gemini CLI config model ids before emission", () => {
|
||||
expect(
|
||||
normalizeConfig({
|
||||
provider: "google-gemini-cli",
|
||||
providerConfig: {
|
||||
baseUrl: "openclaw://google-gemini-cli",
|
||||
models: [
|
||||
{
|
||||
id: "google/gemini-3-pro-preview",
|
||||
name: "Gemini CLI 3 Pro",
|
||||
reasoning: true,
|
||||
input: ["text", "image"],
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 1_048_576,
|
||||
maxTokens: 65_536,
|
||||
},
|
||||
],
|
||||
},
|
||||
}),
|
||||
).toEqual({
|
||||
baseUrl: "openclaw://google-gemini-cli",
|
||||
models: [
|
||||
{
|
||||
id: "google/gemini-3.1-pro-preview",
|
||||
name: "Gemini CLI 3 Pro",
|
||||
reasoning: true,
|
||||
input: ["text", "image"],
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 1_048_576,
|
||||
maxTokens: 65_536,
|
||||
},
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
it("preserves Gemini 3 thinking levels when catalog reasoning metadata is stale", () => {
|
||||
expect(
|
||||
resolveThinkingProfile({
|
||||
provider: "google",
|
||||
modelId: "gemini-3-flash-preview",
|
||||
reasoning: false,
|
||||
}),
|
||||
).toEqual({
|
||||
levels: [
|
||||
{ id: "off" },
|
||||
{ id: "minimal" },
|
||||
{ id: "low" },
|
||||
{ id: "medium" },
|
||||
{ id: "adaptive" },
|
||||
{ id: "high" },
|
||||
],
|
||||
preserveWhenCatalogReasoningFalse: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("preserves provider-prefixed Gemini 3 thinking levels when catalog reasoning metadata is stale", () => {
|
||||
expect(
|
||||
resolveThinkingProfile({
|
||||
provider: "google",
|
||||
modelId: "google/gemini-3-flash-preview",
|
||||
reasoning: false,
|
||||
}),
|
||||
).toMatchObject({
|
||||
levels: expect.arrayContaining([{ id: "low" }, { id: "medium" }, { id: "adaptive" }]),
|
||||
preserveWhenCatalogReasoningFalse: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("preserves normalized Gemini 3 aliases when catalog reasoning metadata is stale", () => {
|
||||
expect(
|
||||
resolveThinkingProfile({
|
||||
provider: "google",
|
||||
modelId: "google/gemini-3-pro",
|
||||
reasoning: false,
|
||||
}),
|
||||
).toEqual({
|
||||
levels: [{ id: "off" }, { id: "low" }, { id: "adaptive" }, { id: "high" }],
|
||||
preserveWhenCatalogReasoningFalse: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("preserves Gemini 3 Pro thinking levels when catalog reasoning metadata is stale", () => {
|
||||
expect(
|
||||
resolveThinkingProfile({
|
||||
provider: "google",
|
||||
modelId: "gemini-3.1-pro-preview",
|
||||
reasoning: false,
|
||||
}),
|
||||
).toEqual({
|
||||
levels: [{ id: "off" }, { id: "low" }, { id: "adaptive" }, { id: "high" }],
|
||||
preserveWhenCatalogReasoningFalse: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("honors catalog reasoning=false for non-Gemini 3 Google models", () => {
|
||||
expect(
|
||||
resolveThinkingProfile({
|
||||
provider: "google",
|
||||
modelId: "gemma-4-26b-a4b-it",
|
||||
reasoning: false,
|
||||
}),
|
||||
).toBeUndefined();
|
||||
});
|
||||
});
|
||||
12
extensions/google/provider-policy-api.ts
Normal file
12
extensions/google/provider-policy-api.ts
Normal file
@@ -0,0 +1,12 @@
|
||||
// Google API module exposes the plugin public contract.
|
||||
import type { ProviderDefaultThinkingPolicyContext } from "openclaw/plugin-sdk/core";
|
||||
import type { ModelProviderConfig } from "openclaw/plugin-sdk/provider-model-types";
|
||||
import { normalizeGoogleProviderConfig, resolveGoogleThinkingProfile } from "./provider-policy.js";
|
||||
|
||||
export function normalizeConfig(params: { provider: string; providerConfig: ModelProviderConfig }) {
|
||||
return normalizeGoogleProviderConfig(params.provider, params.providerConfig);
|
||||
}
|
||||
|
||||
export function resolveThinkingProfile(context: ProviderDefaultThinkingPolicyContext) {
|
||||
return resolveGoogleThinkingProfile(context);
|
||||
}
|
||||
245
extensions/google/provider-policy.ts
Normal file
245
extensions/google/provider-policy.ts
Normal file
@@ -0,0 +1,245 @@
|
||||
// Google provider module implements model/runtime integration.
|
||||
import type {
|
||||
ProviderDefaultThinkingPolicyContext,
|
||||
ProviderThinkingProfile,
|
||||
} from "openclaw/plugin-sdk/core";
|
||||
import type { ModelProviderConfig } from "openclaw/plugin-sdk/provider-model-types";
|
||||
import { normalizeOptionalString } from "openclaw/plugin-sdk/string-coerce-runtime";
|
||||
import { normalizeAntigravityModelId, normalizeGoogleModelId } from "./model-id.js";
|
||||
import { isGoogleGemini3ProModel, isGoogleGemini3ThinkingLevelModel } from "./thinking-api.js";
|
||||
|
||||
type GoogleApiCarrier = {
|
||||
api?: string | null;
|
||||
};
|
||||
|
||||
type GoogleProviderConfigLike = GoogleApiCarrier & {
|
||||
baseUrl?: string | null;
|
||||
models?: ReadonlyArray<GoogleApiCarrier | null | undefined> | null;
|
||||
};
|
||||
|
||||
export const DEFAULT_GOOGLE_API_BASE_URL = "https://generativelanguage.googleapis.com/v1beta";
|
||||
const GOOGLE_MODEL_ID_PROVIDERS = new Set(["google", "google-gemini-cli", "google-vertex"]);
|
||||
|
||||
function trimTrailingSlashes(value: string): string {
|
||||
return value.replace(/\/+$/, "");
|
||||
}
|
||||
|
||||
function isCanonicalGoogleApiOriginShorthand(value: string): boolean {
|
||||
return /^https:\/\/generativelanguage\.googleapis\.com\/?$/i.test(value);
|
||||
}
|
||||
|
||||
function isGoogleGenerativeAiUrl(url: URL): boolean {
|
||||
return (
|
||||
url.protocol === "https:" && url.hostname.toLowerCase() === "generativelanguage.googleapis.com"
|
||||
);
|
||||
}
|
||||
|
||||
function stripUrlUserInfo(url: URL): void {
|
||||
url.username = "";
|
||||
url.password = "";
|
||||
}
|
||||
|
||||
const GOOGLE_VERTEX_HOST = "aiplatform.googleapis.com";
|
||||
const GOOGLE_VERTEX_REGION_HOST_SUFFIX = "-aiplatform.googleapis.com";
|
||||
const GOOGLE_VERTEX_MULTI_REGION_HOSTS = new Set([
|
||||
"aiplatform.eu.rep.googleapis.com",
|
||||
"aiplatform.us.rep.googleapis.com",
|
||||
]);
|
||||
|
||||
export function isGoogleVertexHostname(hostname: string): boolean {
|
||||
const normalized = hostname.toLowerCase();
|
||||
return (
|
||||
normalized === GOOGLE_VERTEX_HOST ||
|
||||
normalized.endsWith(GOOGLE_VERTEX_REGION_HOST_SUFFIX) ||
|
||||
GOOGLE_VERTEX_MULTI_REGION_HOSTS.has(normalized)
|
||||
);
|
||||
}
|
||||
|
||||
export function isGoogleVertexBaseUrl(baseUrl?: string | null): boolean {
|
||||
const raw = normalizeOptionalString(baseUrl);
|
||||
if (!raw) {
|
||||
return false;
|
||||
}
|
||||
try {
|
||||
return isGoogleVertexHostname(new URL(raw).hostname);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
export function normalizeGoogleApiBaseUrl(baseUrl?: string): string {
|
||||
const raw = trimTrailingSlashes(normalizeOptionalString(baseUrl) || DEFAULT_GOOGLE_API_BASE_URL);
|
||||
try {
|
||||
const url = new URL(raw);
|
||||
url.hash = "";
|
||||
url.search = "";
|
||||
stripUrlUserInfo(url);
|
||||
if (isGoogleGenerativeAiUrl(url)) {
|
||||
const normalizedPath = trimTrailingSlashes(url.pathname || "");
|
||||
url.pathname = normalizedPath || "/v1beta";
|
||||
}
|
||||
return trimTrailingSlashes(url.toString());
|
||||
} catch {
|
||||
if (isCanonicalGoogleApiOriginShorthand(raw)) {
|
||||
return DEFAULT_GOOGLE_API_BASE_URL;
|
||||
}
|
||||
return raw;
|
||||
}
|
||||
}
|
||||
|
||||
export function isGoogleGenerativeAiApi(api?: string | null): boolean {
|
||||
return api === "google-generative-ai";
|
||||
}
|
||||
|
||||
export function normalizeGoogleGenerativeAiBaseUrl(baseUrl?: string): string | undefined {
|
||||
if (!baseUrl) {
|
||||
return baseUrl;
|
||||
}
|
||||
|
||||
const normalized = normalizeGoogleApiBaseUrl(baseUrl);
|
||||
try {
|
||||
const url = new URL(normalized);
|
||||
stripUrlUserInfo(url);
|
||||
if (isGoogleGenerativeAiUrl(url)) {
|
||||
url.pathname = trimTrailingSlashes(url.pathname || "").replace(/\/openai$/i, "") || "/v1beta";
|
||||
return trimTrailingSlashes(url.toString());
|
||||
}
|
||||
} catch {
|
||||
// `normalizeGoogleApiBaseUrl` already returned the best-effort input form.
|
||||
}
|
||||
|
||||
return normalized;
|
||||
}
|
||||
|
||||
export function resolveGoogleGenerativeAiTransport<TApi extends string | null | undefined>(params: {
|
||||
provider?: string;
|
||||
api: TApi;
|
||||
baseUrl?: string;
|
||||
}): { api: TApi | "google-generative-ai" | "google-vertex"; baseUrl?: string } {
|
||||
const api =
|
||||
params.api ??
|
||||
(params.provider === "google-vertex" && isGoogleVertexBaseUrl(params.baseUrl)
|
||||
? "google-vertex"
|
||||
: undefined) ??
|
||||
(params.provider === "google" && params.baseUrl ? "google-generative-ai" : params.api);
|
||||
return {
|
||||
api,
|
||||
baseUrl: isGoogleGenerativeAiApi(api)
|
||||
? normalizeGoogleGenerativeAiBaseUrl(params.baseUrl)
|
||||
: params.baseUrl,
|
||||
};
|
||||
}
|
||||
|
||||
export function resolveGoogleGenerativeAiApiOrigin(baseUrl?: string): string {
|
||||
return (
|
||||
normalizeGoogleGenerativeAiBaseUrl(baseUrl) ?? normalizeGoogleApiBaseUrl(baseUrl)
|
||||
).replace(/\/v1beta$/i, "");
|
||||
}
|
||||
|
||||
export function shouldNormalizeGoogleGenerativeAiProviderConfig(
|
||||
providerKey: string,
|
||||
provider: GoogleProviderConfigLike,
|
||||
): boolean {
|
||||
if (providerKey === "google-vertex" && isGoogleVertexBaseUrl(provider.baseUrl)) {
|
||||
return false;
|
||||
}
|
||||
if (isGoogleGenerativeAiApi(provider.api)) {
|
||||
return true;
|
||||
}
|
||||
const hasGoogleGenerativeAiModelApi =
|
||||
provider.models?.some((model) => isGoogleGenerativeAiApi(model?.api)) ?? false;
|
||||
if (hasGoogleGenerativeAiModelApi) {
|
||||
return true;
|
||||
}
|
||||
if (providerKey !== "google" && providerKey !== "google-vertex") {
|
||||
return false;
|
||||
}
|
||||
const hasExplicitNonGoogleApi = normalizeOptionalString(provider.api) !== undefined;
|
||||
return !hasExplicitNonGoogleApi;
|
||||
}
|
||||
|
||||
export function shouldNormalizeGoogleProviderConfig(
|
||||
providerKey: string,
|
||||
provider: GoogleProviderConfigLike,
|
||||
): boolean {
|
||||
return (
|
||||
providerKey === "google-antigravity" ||
|
||||
shouldNormalizeGoogleGenerativeAiProviderConfig(providerKey, provider)
|
||||
);
|
||||
}
|
||||
|
||||
function normalizeProviderModels(
|
||||
provider: ModelProviderConfig,
|
||||
normalizeId: (id: string) => string,
|
||||
): ModelProviderConfig {
|
||||
const models = provider.models;
|
||||
if (!Array.isArray(models) || models.length === 0) {
|
||||
return provider;
|
||||
}
|
||||
|
||||
let mutated = false;
|
||||
const nextModels = models.map((model) => {
|
||||
const nextId = normalizeId(model.id);
|
||||
if (nextId === model.id) {
|
||||
return model;
|
||||
}
|
||||
mutated = true;
|
||||
return Object.assign({}, model, { id: nextId });
|
||||
});
|
||||
|
||||
return mutated ? { ...provider, models: nextModels } : provider;
|
||||
}
|
||||
|
||||
export function normalizeGoogleProviderConfig(
|
||||
providerKey: string,
|
||||
provider: ModelProviderConfig,
|
||||
): ModelProviderConfig {
|
||||
let nextProvider = provider;
|
||||
const shouldNormalizeModelIds = GOOGLE_MODEL_ID_PROVIDERS.has(providerKey);
|
||||
|
||||
if (shouldNormalizeModelIds) {
|
||||
const modelNormalized = normalizeProviderModels(nextProvider, normalizeGoogleModelId);
|
||||
if (shouldNormalizeGoogleGenerativeAiProviderConfig(providerKey, modelNormalized)) {
|
||||
const normalizedBaseUrl = normalizeGoogleGenerativeAiBaseUrl(modelNormalized.baseUrl);
|
||||
nextProvider =
|
||||
normalizedBaseUrl !== modelNormalized.baseUrl
|
||||
? { ...modelNormalized, baseUrl: normalizedBaseUrl ?? modelNormalized.baseUrl }
|
||||
: modelNormalized;
|
||||
} else {
|
||||
nextProvider = modelNormalized;
|
||||
}
|
||||
}
|
||||
|
||||
if (providerKey === "google-antigravity") {
|
||||
nextProvider = normalizeProviderModels(nextProvider, normalizeAntigravityModelId);
|
||||
}
|
||||
|
||||
return nextProvider;
|
||||
}
|
||||
|
||||
export function resolveGoogleThinkingProfile({
|
||||
modelId,
|
||||
reasoning,
|
||||
}: ProviderDefaultThinkingPolicyContext): ProviderThinkingProfile | undefined {
|
||||
const normalizedModelId = normalizeGoogleModelId(modelId);
|
||||
const isGemini3ThinkingModel = isGoogleGemini3ThinkingLevelModel(normalizedModelId);
|
||||
if (reasoning === false && !isGemini3ThinkingModel) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
const levels: ProviderThinkingProfile["levels"] = isGoogleGemini3ProModel(normalizedModelId)
|
||||
? [{ id: "off" }, { id: "low" }, { id: "adaptive" }, { id: "high" }]
|
||||
: [
|
||||
{ id: "off" },
|
||||
{ id: "minimal" },
|
||||
{ id: "low" },
|
||||
{ id: "medium" },
|
||||
{ id: "adaptive" },
|
||||
{ id: "high" },
|
||||
];
|
||||
|
||||
return {
|
||||
levels,
|
||||
...(isGemini3ThinkingModel ? { preserveWhenCatalogReasoningFalse: true } : {}),
|
||||
};
|
||||
}
|
||||
68
extensions/google/provider-registration.test.ts
Normal file
68
extensions/google/provider-registration.test.ts
Normal file
@@ -0,0 +1,68 @@
|
||||
// Google tests cover provider registration plugin behavior.
|
||||
import type { Model } from "openclaw/plugin-sdk/llm";
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { buildGoogleProvider } from "./provider-registration.js";
|
||||
|
||||
const streamFns = vi.hoisted(() => ({
|
||||
createGenerativeAi: vi.fn(() => vi.fn()),
|
||||
createVertex: vi.fn(() => vi.fn()),
|
||||
}));
|
||||
|
||||
vi.mock("./transport-stream.js", () => ({
|
||||
createGoogleGenerativeAiTransportStreamFn: streamFns.createGenerativeAi,
|
||||
createGoogleVertexTransportStreamFn: streamFns.createVertex,
|
||||
}));
|
||||
|
||||
function model(overrides: Partial<Model> = {}): Model {
|
||||
return {
|
||||
id: "gemini-2.5-flash",
|
||||
name: "Gemini 2.5 Flash",
|
||||
provider: "google-vertex",
|
||||
api: "google-generative-ai",
|
||||
baseUrl: "https://aiplatform.googleapis.com",
|
||||
reasoning: true,
|
||||
input: ["text"],
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 1_048_576,
|
||||
maxTokens: 65_536,
|
||||
...overrides,
|
||||
} as Model;
|
||||
}
|
||||
|
||||
describe("buildGoogleProvider createStreamFn", () => {
|
||||
beforeEach(() => {
|
||||
streamFns.createGenerativeAi.mockClear();
|
||||
streamFns.createVertex.mockClear();
|
||||
});
|
||||
|
||||
it("routes native Vertex hosts through the Vertex transport", () => {
|
||||
const provider = buildGoogleProvider();
|
||||
|
||||
provider.createStreamFn?.({
|
||||
provider: "google-vertex",
|
||||
modelId: "gemini-2.5-flash",
|
||||
model: model(),
|
||||
} as never);
|
||||
|
||||
expect(streamFns.createVertex).toHaveBeenCalledTimes(1);
|
||||
expect(streamFns.createGenerativeAi).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("preserves explicit OpenAI-compatible Vertex endpoint configs", () => {
|
||||
const provider = buildGoogleProvider();
|
||||
|
||||
const result = provider.createStreamFn?.({
|
||||
provider: "google-vertex",
|
||||
modelId: "gemini-2.5-flash",
|
||||
model: model({
|
||||
api: "openai-completions",
|
||||
baseUrl:
|
||||
"https://aiplatform.googleapis.com/v1/projects/test/locations/us-central1/endpoints/openapi",
|
||||
}),
|
||||
} as never);
|
||||
|
||||
expect(result).toBeUndefined();
|
||||
expect(streamFns.createVertex).not.toHaveBeenCalled();
|
||||
expect(streamFns.createGenerativeAi).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
114
extensions/google/provider-registration.ts
Normal file
114
extensions/google/provider-registration.ts
Normal file
@@ -0,0 +1,114 @@
|
||||
// Google provider module implements model/runtime integration.
|
||||
import type {
|
||||
OpenClawPluginApi,
|
||||
ProviderReasoningOutputModeContext,
|
||||
} from "openclaw/plugin-sdk/plugin-entry";
|
||||
import { createProviderApiKeyAuthMethod } from "openclaw/plugin-sdk/provider-auth-api-key";
|
||||
import type { ProviderPlugin } from "openclaw/plugin-sdk/provider-model-shared";
|
||||
import { normalizeGoogleModelId } from "./model-id.js";
|
||||
import { GOOGLE_GEMINI_DEFAULT_MODEL, applyGoogleGeminiModelDefault } from "./onboard.js";
|
||||
import {
|
||||
buildGoogleStaticCatalogProvider,
|
||||
buildGoogleVertexStaticCatalogProvider,
|
||||
} from "./provider-catalog.js";
|
||||
import { GOOGLE_GEMINI_PROVIDER_HOOKS } from "./provider-hooks.js";
|
||||
import { isModernGoogleModel, resolveGoogleGeminiForwardCompatModel } from "./provider-models.js";
|
||||
import {
|
||||
isGoogleVertexBaseUrl,
|
||||
normalizeGoogleProviderConfig,
|
||||
resolveGoogleGenerativeAiTransport,
|
||||
} from "./provider-policy.js";
|
||||
import {
|
||||
createGoogleGenerativeAiTransportStreamFn,
|
||||
createGoogleVertexTransportStreamFn,
|
||||
} from "./transport-stream.js";
|
||||
import { resolveGoogleVertexConfigApiKey } from "./vertex-adc.js";
|
||||
|
||||
function resolveGoogleReasoningOutputMode(
|
||||
ctx: ProviderReasoningOutputModeContext,
|
||||
): "native" | "tagged" {
|
||||
if (ctx.provider === "google" || ctx.provider === "google-vertex") {
|
||||
const api = ctx.model?.api ?? ctx.modelApi;
|
||||
if (!api || api === "google-generative-ai" || api === "google-vertex") {
|
||||
return "native";
|
||||
}
|
||||
}
|
||||
return "tagged";
|
||||
}
|
||||
|
||||
export function buildGoogleProvider(): ProviderPlugin {
|
||||
return {
|
||||
id: "google",
|
||||
label: "Google AI Studio",
|
||||
docsPath: "/providers/models",
|
||||
hookAliases: ["google-antigravity", "google-vertex"],
|
||||
envVars: ["GEMINI_API_KEY", "GOOGLE_API_KEY"],
|
||||
auth: [
|
||||
createProviderApiKeyAuthMethod({
|
||||
providerId: "google",
|
||||
methodId: "api-key",
|
||||
label: "Google Gemini API key",
|
||||
hint: "AI Studio / Gemini API key",
|
||||
optionKey: "geminiApiKey",
|
||||
flagName: "--gemini-api-key",
|
||||
envVar: "GEMINI_API_KEY",
|
||||
promptMessage: "Enter Gemini API key",
|
||||
defaultModel: GOOGLE_GEMINI_DEFAULT_MODEL,
|
||||
expectedProviders: ["google"],
|
||||
applyConfig: (cfg) => applyGoogleGeminiModelDefault(cfg).next,
|
||||
wizard: {
|
||||
choiceId: "gemini-api-key",
|
||||
choiceLabel: "Google Gemini API key",
|
||||
groupId: "google",
|
||||
groupLabel: "Google",
|
||||
groupHint: "Gemini API key + OAuth",
|
||||
},
|
||||
}),
|
||||
],
|
||||
normalizeTransport: ({ provider, api, baseUrl }) =>
|
||||
resolveGoogleGenerativeAiTransport({ provider, api, baseUrl }),
|
||||
normalizeConfig: ({ provider, providerConfig }) =>
|
||||
normalizeGoogleProviderConfig(provider, providerConfig),
|
||||
resolveConfigApiKey: ({ provider, env }) =>
|
||||
provider === "google-vertex" ? resolveGoogleVertexConfigApiKey(env) : undefined,
|
||||
staticCatalog: {
|
||||
order: "simple",
|
||||
run: async () => ({
|
||||
providers: {
|
||||
google: buildGoogleStaticCatalogProvider(),
|
||||
"google-vertex": buildGoogleVertexStaticCatalogProvider(),
|
||||
},
|
||||
}),
|
||||
},
|
||||
normalizeModelId: ({ modelId }) => normalizeGoogleModelId(modelId),
|
||||
resolveDynamicModel: (ctx) =>
|
||||
resolveGoogleGeminiForwardCompatModel({
|
||||
providerId: ctx.provider,
|
||||
ctx,
|
||||
}),
|
||||
createStreamFn: ({ model }) => {
|
||||
if (
|
||||
model.api === "google-vertex" ||
|
||||
(model.api === "google-generative-ai" &&
|
||||
(model.provider === "google-vertex" || isGoogleVertexBaseUrl(model.baseUrl)))
|
||||
) {
|
||||
return createGoogleVertexTransportStreamFn();
|
||||
}
|
||||
if (model.api === "google-generative-ai") {
|
||||
return createGoogleGenerativeAiTransportStreamFn();
|
||||
}
|
||||
return undefined;
|
||||
},
|
||||
...GOOGLE_GEMINI_PROVIDER_HOOKS,
|
||||
// Gemini 2.5+ delivers reasoning via native thinkingParts (thinkingConfig.includeThoughts).
|
||||
// Tagged mode simultaneously injects <think>/<final> which the model opens before a tool
|
||||
// call, never closes, leaving the post-tool turn empty (payloads=0). The CLI backend keeps
|
||||
// tagged mode because it emits JSON text, not native thought parts.
|
||||
resolveReasoningOutputMode: resolveGoogleReasoningOutputMode,
|
||||
isModernModelRef: ({ modelId }) => isModernGoogleModel(modelId),
|
||||
};
|
||||
}
|
||||
|
||||
export function registerGoogleProvider(api: OpenClawPluginApi) {
|
||||
api.registerProvider(buildGoogleProvider());
|
||||
}
|
||||
4
extensions/google/provider-runtime.contract.test.ts
Normal file
4
extensions/google/provider-runtime.contract.test.ts
Normal file
@@ -0,0 +1,4 @@
|
||||
// Google tests cover provider runtime.contract plugin behavior.
|
||||
import { describeGoogleProviderRuntimeContract } from "openclaw/plugin-sdk/provider-test-contracts";
|
||||
|
||||
describeGoogleProviderRuntimeContract(() => import("./index.js"));
|
||||
940
extensions/google/realtime-voice-provider.test.ts
Normal file
940
extensions/google/realtime-voice-provider.test.ts
Normal file
@@ -0,0 +1,940 @@
|
||||
// Google tests cover realtime voice provider plugin behavior.
|
||||
import { REALTIME_VOICE_AUDIO_FORMAT_PCM16_24KHZ } from "openclaw/plugin-sdk/realtime-voice";
|
||||
import { afterAll, afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { buildGoogleRealtimeVoiceProvider } from "./realtime-voice-provider.js";
|
||||
|
||||
type MockGoogleLiveSession = {
|
||||
close: ReturnType<typeof vi.fn>;
|
||||
sendClientContent: ReturnType<typeof vi.fn>;
|
||||
sendRealtimeInput: ReturnType<typeof vi.fn>;
|
||||
sendToolResponse: ReturnType<typeof vi.fn>;
|
||||
};
|
||||
|
||||
type MockGoogleLiveConnectParams = {
|
||||
model: string;
|
||||
config: Record<string, unknown>;
|
||||
callbacks: {
|
||||
onopen: () => void;
|
||||
onmessage: (message: Record<string, unknown>) => void;
|
||||
onerror: (event: { error?: unknown; message?: string }) => void;
|
||||
onclose: (event?: { code?: number; reason?: string; wasClean?: boolean }) => void;
|
||||
};
|
||||
};
|
||||
|
||||
const { connectMock, createTokenMock, session } = vi.hoisted(() => {
|
||||
const sessionValue: MockGoogleLiveSession = {
|
||||
close: vi.fn(),
|
||||
sendClientContent: vi.fn(),
|
||||
sendRealtimeInput: vi.fn(),
|
||||
sendToolResponse: vi.fn(),
|
||||
};
|
||||
const connectMockLocal = vi.fn(async (_params: MockGoogleLiveConnectParams) => sessionValue);
|
||||
const createTokenMockLocal = vi.fn(async (_params: unknown) => ({
|
||||
name: "auth_tokens/browser-session",
|
||||
}));
|
||||
return {
|
||||
connectMock: connectMockLocal,
|
||||
createTokenMock: createTokenMockLocal,
|
||||
session: sessionValue,
|
||||
};
|
||||
});
|
||||
|
||||
vi.mock("./google-genai-runtime.js", () => ({
|
||||
createGoogleGenAI: vi.fn(() => ({
|
||||
authTokens: {
|
||||
create: createTokenMock,
|
||||
},
|
||||
live: {
|
||||
connect: connectMock,
|
||||
},
|
||||
})),
|
||||
}));
|
||||
|
||||
const ENV_KEYS = ["GEMINI_API_KEY", "GOOGLE_API_KEY"] as const;
|
||||
|
||||
let envSnapshot: Partial<Record<(typeof ENV_KEYS)[number], string>>;
|
||||
|
||||
function lastConnectParams(): MockGoogleLiveConnectParams {
|
||||
const params = connectMock.mock.calls.at(-1)?.[0];
|
||||
if (!params) {
|
||||
throw new Error("expected google live connect call");
|
||||
}
|
||||
return params;
|
||||
}
|
||||
|
||||
function sentAudio(index = 0): { data?: unknown; mimeType?: unknown } {
|
||||
const audio = session.sendRealtimeInput.mock.calls[index]?.[0]?.audio;
|
||||
if (!audio) {
|
||||
throw new Error(`Expected sent audio at index ${index}`);
|
||||
}
|
||||
return audio as { data?: unknown; mimeType?: unknown };
|
||||
}
|
||||
|
||||
function requireFirstMockArg(mock: ReturnType<typeof vi.fn>, label: string): unknown {
|
||||
const [call] = mock.mock.calls;
|
||||
if (!call) {
|
||||
throw new Error(`expected ${label}`);
|
||||
}
|
||||
return call[0];
|
||||
}
|
||||
|
||||
function requireFirstError(mock: ReturnType<typeof vi.fn>): { message?: string } {
|
||||
const error = requireFirstMockArg(mock, "Google Live error");
|
||||
if (!error || typeof error !== "object" || Array.isArray(error)) {
|
||||
throw new Error("expected Google Live error");
|
||||
}
|
||||
return error as { message?: string };
|
||||
}
|
||||
|
||||
function requireFirstAudio(mock: ReturnType<typeof vi.fn>): unknown {
|
||||
return requireFirstMockArg(mock, "Google Live audio");
|
||||
}
|
||||
|
||||
describe("buildGoogleRealtimeVoiceProvider", () => {
|
||||
beforeEach(() => {
|
||||
envSnapshot = Object.fromEntries(ENV_KEYS.map((key) => [key, process.env[key]]));
|
||||
connectMock.mockClear();
|
||||
createTokenMock.mockClear();
|
||||
session.close.mockClear();
|
||||
session.sendClientContent.mockClear();
|
||||
session.sendRealtimeInput.mockClear();
|
||||
session.sendToolResponse.mockClear();
|
||||
delete process.env.GEMINI_API_KEY;
|
||||
delete process.env.GOOGLE_API_KEY;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.useRealTimers();
|
||||
vi.restoreAllMocks();
|
||||
for (const key of ENV_KEYS) {
|
||||
const value = envSnapshot[key];
|
||||
if (value === undefined) {
|
||||
delete process.env[key];
|
||||
} else {
|
||||
process.env[key] = value;
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
vi.doUnmock("./google-genai-runtime.js");
|
||||
vi.resetModules();
|
||||
});
|
||||
|
||||
it("declares realtime Talk capabilities for catalog selection", () => {
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
|
||||
expect(provider.capabilities).toEqual({
|
||||
transports: ["provider-websocket", "gateway-relay"],
|
||||
inputAudioFormats: [
|
||||
{ encoding: "g711_ulaw", sampleRateHz: 8000, channels: 1 },
|
||||
{ encoding: "pcm16", sampleRateHz: 24000, channels: 1 },
|
||||
],
|
||||
outputAudioFormats: [
|
||||
{ encoding: "g711_ulaw", sampleRateHz: 8000, channels: 1 },
|
||||
{ encoding: "pcm16", sampleRateHz: 24000, channels: 1 },
|
||||
],
|
||||
supportsBrowserSession: true,
|
||||
supportsBargeIn: true,
|
||||
supportsToolCalls: true,
|
||||
supportsVideoFrames: true,
|
||||
supportsSessionResumption: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("normalizes provider config and cfg model-provider key fallback", () => {
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
const resolved = provider.resolveConfig?.({
|
||||
cfg: {
|
||||
models: {
|
||||
providers: {
|
||||
google: {
|
||||
apiKey: "cfg-key",
|
||||
},
|
||||
},
|
||||
},
|
||||
} as never,
|
||||
rawConfig: {
|
||||
providers: {
|
||||
google: {
|
||||
model: "gemini-live-2.5-flash-preview",
|
||||
voice: "Puck",
|
||||
temperature: 0.4,
|
||||
silenceDurationMs: 700,
|
||||
startSensitivity: "high",
|
||||
activityHandling: "no_interruption",
|
||||
turnCoverage: "turn_includes_only_activity",
|
||||
automaticActivityDetectionDisabled: false,
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
expect(resolved).toEqual({
|
||||
apiKey: "cfg-key",
|
||||
model: "gemini-live-2.5-flash-preview",
|
||||
voice: "Puck",
|
||||
temperature: 0.4,
|
||||
apiVersion: undefined,
|
||||
prefixPaddingMs: undefined,
|
||||
silenceDurationMs: 700,
|
||||
startSensitivity: "high",
|
||||
endSensitivity: undefined,
|
||||
activityHandling: "no-interruption",
|
||||
turnCoverage: "only-activity",
|
||||
automaticActivityDetectionDisabled: false,
|
||||
enableAffectiveDialog: undefined,
|
||||
sessionResumption: undefined,
|
||||
contextWindowCompression: undefined,
|
||||
thinkingLevel: undefined,
|
||||
thinkingBudget: undefined,
|
||||
});
|
||||
});
|
||||
|
||||
it("connects with Google Live setup config and tool declarations", async () => {
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
const bridge = provider.createBridge({
|
||||
providerConfig: {
|
||||
apiKey: "gemini-key",
|
||||
model: "gemini-live-2.5-flash-preview",
|
||||
voice: "Kore",
|
||||
temperature: 0.3,
|
||||
startSensitivity: "low",
|
||||
endSensitivity: "low",
|
||||
activityHandling: "no-interruption",
|
||||
turnCoverage: "only-activity",
|
||||
},
|
||||
instructions: "Speak briefly.",
|
||||
tools: [
|
||||
{
|
||||
type: "function",
|
||||
name: "lookup",
|
||||
description: "Look something up",
|
||||
parameters: {
|
||||
type: "object",
|
||||
properties: {
|
||||
query: { type: "string" },
|
||||
},
|
||||
required: ["query"],
|
||||
},
|
||||
},
|
||||
{
|
||||
type: "function",
|
||||
name: "openclaw_agent_consult",
|
||||
description: "Ask OpenClaw",
|
||||
parameters: {
|
||||
type: "object",
|
||||
properties: {
|
||||
question: { type: "string" },
|
||||
},
|
||||
required: ["question"],
|
||||
},
|
||||
},
|
||||
],
|
||||
onAudio: vi.fn(),
|
||||
onClearAudio: vi.fn(),
|
||||
});
|
||||
|
||||
await bridge.connect();
|
||||
|
||||
expect(connectMock).toHaveBeenCalledTimes(1);
|
||||
const params = lastConnectParams();
|
||||
expect(params.model).toBe("gemini-live-2.5-flash-preview");
|
||||
const config = params.config as {
|
||||
contextWindowCompression?: unknown;
|
||||
outputAudioTranscription?: unknown;
|
||||
realtimeInputConfig?: {
|
||||
activityHandling?: string;
|
||||
automaticActivityDetection?: {
|
||||
endOfSpeechSensitivity?: string;
|
||||
startOfSpeechSensitivity?: string;
|
||||
};
|
||||
turnCoverage?: string;
|
||||
};
|
||||
responseModalities?: string[];
|
||||
sessionResumption?: unknown;
|
||||
speechConfig?: { voiceConfig?: { prebuiltVoiceConfig?: { voiceName?: string } } };
|
||||
systemInstruction?: string;
|
||||
temperature?: number;
|
||||
tools?: Array<{
|
||||
functionDeclarations?: Array<{
|
||||
behavior?: string;
|
||||
description?: string;
|
||||
name?: string;
|
||||
parametersJsonSchema?: unknown;
|
||||
}>;
|
||||
}>;
|
||||
};
|
||||
expect(config.responseModalities).toEqual(["AUDIO"]);
|
||||
expect(config.temperature).toBe(0.3);
|
||||
expect(config.systemInstruction).toBe("Speak briefly.");
|
||||
expect(config.speechConfig?.voiceConfig?.prebuiltVoiceConfig?.voiceName).toBe("Kore");
|
||||
expect(config.outputAudioTranscription).toEqual({});
|
||||
expect(config.realtimeInputConfig?.activityHandling).toBe("NO_INTERRUPTION");
|
||||
expect(config.realtimeInputConfig?.automaticActivityDetection?.startOfSpeechSensitivity).toBe(
|
||||
"START_SENSITIVITY_LOW",
|
||||
);
|
||||
expect(config.realtimeInputConfig?.automaticActivityDetection?.endOfSpeechSensitivity).toBe(
|
||||
"END_SENSITIVITY_LOW",
|
||||
);
|
||||
expect(config.realtimeInputConfig?.turnCoverage).toBe("TURN_INCLUDES_ONLY_ACTIVITY");
|
||||
expect(config.sessionResumption).toEqual({});
|
||||
expect(config.contextWindowCompression).toEqual({ slidingWindow: {} });
|
||||
const declarations = config.tools?.[0]?.functionDeclarations ?? [];
|
||||
expect(declarations[0]?.name).toBe("lookup");
|
||||
expect(declarations[0]?.description).toBe("Look something up");
|
||||
expect(declarations[0]?.parametersJsonSchema).toEqual({
|
||||
type: "object",
|
||||
properties: {
|
||||
query: { type: "string" },
|
||||
},
|
||||
required: ["query"],
|
||||
});
|
||||
expect(declarations[1]?.name).toBe("openclaw_agent_consult");
|
||||
expect(declarations[1]?.description).toBe("Ask OpenClaw");
|
||||
expect(declarations[1]?.parametersJsonSchema).toEqual({
|
||||
type: "object",
|
||||
properties: {
|
||||
question: { type: "string" },
|
||||
},
|
||||
required: ["question"],
|
||||
});
|
||||
expect(declarations[1]?.behavior).toBe("NON_BLOCKING");
|
||||
});
|
||||
|
||||
it("omits zero temperature for native audio responses", async () => {
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
const bridge = provider.createBridge({
|
||||
providerConfig: {
|
||||
apiKey: "gemini-key",
|
||||
temperature: 0,
|
||||
},
|
||||
onAudio: vi.fn(),
|
||||
onClearAudio: vi.fn(),
|
||||
});
|
||||
|
||||
await bridge.connect();
|
||||
|
||||
expect(lastConnectParams().config).not.toHaveProperty("temperature");
|
||||
});
|
||||
|
||||
it("drops malformed VAD timing values before connecting", async () => {
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
const bridge = provider.createBridge({
|
||||
providerConfig: {
|
||||
apiKey: "gemini-key",
|
||||
prefixPaddingMs: -1,
|
||||
silenceDurationMs: 250.5,
|
||||
},
|
||||
onAudio: vi.fn(),
|
||||
onClearAudio: vi.fn(),
|
||||
});
|
||||
|
||||
await bridge.connect();
|
||||
|
||||
expect(lastConnectParams().config).not.toHaveProperty("realtimeInputConfig");
|
||||
});
|
||||
|
||||
it("drops malformed thinking budgets before connecting", async () => {
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
const bridge = provider.createBridge({
|
||||
providerConfig: {
|
||||
apiKey: "gemini-key",
|
||||
thinkingBudget: 24_576.5,
|
||||
},
|
||||
onAudio: vi.fn(),
|
||||
onClearAudio: vi.fn(),
|
||||
});
|
||||
|
||||
await bridge.connect();
|
||||
|
||||
expect(lastConnectParams().config).not.toHaveProperty("thinkingConfig");
|
||||
});
|
||||
|
||||
it("passes Google Live dynamic thinking budget through", async () => {
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
const bridge = provider.createBridge({
|
||||
providerConfig: {
|
||||
apiKey: "gemini-key",
|
||||
thinkingBudget: -1,
|
||||
},
|
||||
onAudio: vi.fn(),
|
||||
onClearAudio: vi.fn(),
|
||||
});
|
||||
|
||||
await bridge.connect();
|
||||
|
||||
expect(lastConnectParams().config.thinkingConfig).toEqual({ thinkingBudget: -1 });
|
||||
});
|
||||
|
||||
it("creates constrained browser sessions for Google Live Talk", async () => {
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
|
||||
const sessionLocal = await provider.createBrowserSession?.({
|
||||
providerConfig: {
|
||||
apiKey: "gemini-key",
|
||||
model: "gemini-live-2.5-flash-preview",
|
||||
voice: "Puck",
|
||||
temperature: 0.4,
|
||||
},
|
||||
instructions: "Speak briefly.",
|
||||
tools: [
|
||||
{
|
||||
type: "function",
|
||||
name: "openclaw_agent_consult",
|
||||
description: "Ask OpenClaw",
|
||||
parameters: {
|
||||
type: "object",
|
||||
properties: {
|
||||
question: { type: "string" },
|
||||
},
|
||||
required: ["question"],
|
||||
},
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
expect(createTokenMock).toHaveBeenCalledTimes(1);
|
||||
const tokenConfig = requireFirstMockArg(createTokenMock, "Google Live auth token config") as {
|
||||
config?: {
|
||||
liveConnectConstraints?: {
|
||||
config?: {
|
||||
responseModalities?: string[];
|
||||
speechConfig?: { voiceConfig?: { prebuiltVoiceConfig?: { voiceName?: string } } };
|
||||
systemInstruction?: string;
|
||||
temperature?: number;
|
||||
tools?: Array<{ functionDeclarations?: Array<{ behavior?: string; name?: string }> }>;
|
||||
};
|
||||
model?: string;
|
||||
};
|
||||
uses?: number;
|
||||
};
|
||||
};
|
||||
const liveConstraints = tokenConfig.config?.liveConnectConstraints;
|
||||
expect(tokenConfig.config?.uses).toBe(1);
|
||||
expect(liveConstraints?.model).toBe("gemini-live-2.5-flash-preview");
|
||||
expect(liveConstraints?.config?.responseModalities).toEqual(["AUDIO"]);
|
||||
expect(liveConstraints?.config?.temperature).toBe(0.4);
|
||||
expect(liveConstraints?.config?.systemInstruction).toBe("Speak briefly.");
|
||||
expect(liveConstraints?.config?.speechConfig?.voiceConfig?.prebuiltVoiceConfig?.voiceName).toBe(
|
||||
"Puck",
|
||||
);
|
||||
expect(liveConstraints?.config?.tools?.[0]?.functionDeclarations?.[0]?.name).toBe(
|
||||
"openclaw_agent_consult",
|
||||
);
|
||||
expect(liveConstraints?.config?.tools?.[0]?.functionDeclarations?.[0]?.behavior).toBe(
|
||||
"NON_BLOCKING",
|
||||
);
|
||||
expect(sessionLocal?.provider).toBe("google");
|
||||
expect(sessionLocal?.transport).toBe("provider-websocket");
|
||||
const websocketSession = sessionLocal as {
|
||||
audio: {
|
||||
inputEncoding: string;
|
||||
inputSampleRateHz: number;
|
||||
outputEncoding: string;
|
||||
outputSampleRateHz: number;
|
||||
};
|
||||
clientSecret: string;
|
||||
initialMessage: {
|
||||
setup: { generationConfig: { responseModalities: string[] }; model: string };
|
||||
};
|
||||
protocol: string;
|
||||
websocketUrl: string;
|
||||
};
|
||||
expect(websocketSession.protocol).toBe("google-live-bidi");
|
||||
expect(websocketSession.clientSecret).toBe("auth_tokens/browser-session");
|
||||
expect(websocketSession.websocketUrl).toBe(
|
||||
"wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContentConstrained",
|
||||
);
|
||||
expect(websocketSession.audio.inputEncoding).toBe("pcm16");
|
||||
expect(websocketSession.audio.inputSampleRateHz).toBe(16000);
|
||||
expect(websocketSession.audio.outputEncoding).toBe("pcm16");
|
||||
expect(websocketSession.audio.outputSampleRateHz).toBe(24000);
|
||||
expect(websocketSession.initialMessage.setup.model).toBe(
|
||||
"models/gemini-live-2.5-flash-preview",
|
||||
);
|
||||
expect(websocketSession.initialMessage.setup.generationConfig.responseModalities).toEqual([
|
||||
"AUDIO",
|
||||
]);
|
||||
});
|
||||
|
||||
it("rejects browser session expiry outside Date range", async () => {
|
||||
vi.spyOn(Date, "now").mockReturnValue(8_640_000_000_000_001);
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
|
||||
await expect(
|
||||
provider.createBrowserSession?.({
|
||||
providerConfig: {
|
||||
apiKey: "gemini-key",
|
||||
},
|
||||
}),
|
||||
).rejects.toThrow("Google realtime browser session expiry is outside the supported Date range");
|
||||
expect(createTokenMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("rejects browser session creation while the process clock is invalid", async () => {
|
||||
vi.spyOn(Date, "now").mockReturnValue(Number.NaN);
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
|
||||
await expect(
|
||||
provider.createBrowserSession?.({
|
||||
providerConfig: {
|
||||
apiKey: "gemini-key",
|
||||
},
|
||||
}),
|
||||
).rejects.toThrow("Google realtime browser session expiry is outside the supported Date range");
|
||||
expect(createTokenMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("can opt out of Google Live session resumption and context compression", async () => {
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
const bridge = provider.createBridge({
|
||||
providerConfig: {
|
||||
apiKey: "gemini-key",
|
||||
contextWindowCompression: false,
|
||||
sessionResumption: false,
|
||||
},
|
||||
onAudio: vi.fn(),
|
||||
onClearAudio: vi.fn(),
|
||||
});
|
||||
|
||||
await bridge.connect();
|
||||
|
||||
expect(lastConnectParams().config).not.toHaveProperty("contextWindowCompression");
|
||||
expect(lastConnectParams().config).not.toHaveProperty("sessionResumption");
|
||||
});
|
||||
|
||||
it("captures Google Live resumption handles and reuses them on reconnect", async () => {
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
const bridge = provider.createBridge({
|
||||
providerConfig: { apiKey: "gemini-key" },
|
||||
onAudio: vi.fn(),
|
||||
onClearAudio: vi.fn(),
|
||||
});
|
||||
|
||||
await bridge.connect();
|
||||
lastConnectParams().callbacks.onmessage({
|
||||
sessionResumptionUpdate: { resumable: true, newHandle: "resume-1" },
|
||||
});
|
||||
|
||||
await bridge.connect();
|
||||
|
||||
expect(lastConnectParams().config.sessionResumption).toEqual({ handle: "resume-1" });
|
||||
});
|
||||
|
||||
it("reconnects unexpected Google Live closes with the latest resumption handle", async () => {
|
||||
vi.useFakeTimers();
|
||||
try {
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
const onClose = vi.fn();
|
||||
const onError = vi.fn();
|
||||
const bridge = provider.createBridge({
|
||||
providerConfig: { apiKey: "gemini-key" },
|
||||
onAudio: vi.fn(),
|
||||
onClearAudio: vi.fn(),
|
||||
onClose,
|
||||
onError,
|
||||
});
|
||||
|
||||
await bridge.connect();
|
||||
lastConnectParams().callbacks.onmessage({
|
||||
setupComplete: { sessionId: "session-1" },
|
||||
sessionResumptionUpdate: { resumable: true, newHandle: "resume-1" },
|
||||
});
|
||||
lastConnectParams().callbacks.onclose({
|
||||
code: 1011,
|
||||
reason: "temporary upstream close",
|
||||
wasClean: false,
|
||||
});
|
||||
|
||||
expect(onClose).not.toHaveBeenCalled();
|
||||
const error = requireFirstError(onError);
|
||||
expect(error.message).toContain("reconnecting 1/3");
|
||||
|
||||
await vi.advanceTimersByTimeAsync(250);
|
||||
|
||||
expect(connectMock).toHaveBeenCalledTimes(2);
|
||||
expect(lastConnectParams().config.sessionResumption).toEqual({ handle: "resume-1" });
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
|
||||
it("waits for setup completion before draining audio and firing ready", async () => {
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
const onReady = vi.fn();
|
||||
const bridge = provider.createBridge({
|
||||
providerConfig: { apiKey: "gemini-key" },
|
||||
onAudio: vi.fn(),
|
||||
onClearAudio: vi.fn(),
|
||||
onReady,
|
||||
});
|
||||
|
||||
await bridge.connect();
|
||||
lastConnectParams().callbacks.onopen();
|
||||
bridge.sendAudio(Buffer.from([0xff, 0xff]));
|
||||
|
||||
expect(session.sendRealtimeInput).not.toHaveBeenCalled();
|
||||
expect(onReady).not.toHaveBeenCalled();
|
||||
|
||||
lastConnectParams().callbacks.onmessage({ setupComplete: { sessionId: "session-1" } });
|
||||
|
||||
expect(onReady).toHaveBeenCalledTimes(1);
|
||||
expect(session.sendRealtimeInput).toHaveBeenCalledTimes(1);
|
||||
const audio = sentAudio();
|
||||
expect(typeof audio.data).toBe("string");
|
||||
expect(audio.mimeType).toBe("audio/pcm;rate=16000");
|
||||
});
|
||||
|
||||
it("marks the Google audio stream complete after sustained telephony silence", async () => {
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
const bridge = provider.createBridge({
|
||||
providerConfig: { apiKey: "gemini-key", silenceDurationMs: 60 },
|
||||
onAudio: vi.fn(),
|
||||
onClearAudio: vi.fn(),
|
||||
});
|
||||
|
||||
await bridge.connect();
|
||||
lastConnectParams().callbacks.onopen();
|
||||
lastConnectParams().callbacks.onmessage({ setupComplete: { sessionId: "session-1" } });
|
||||
|
||||
const silence20ms = Buffer.alloc(160, 0xff);
|
||||
bridge.sendAudio(silence20ms);
|
||||
bridge.sendAudio(silence20ms);
|
||||
bridge.sendAudio(silence20ms);
|
||||
|
||||
expect(session.sendRealtimeInput).toHaveBeenCalledWith({ audioStreamEnd: true });
|
||||
|
||||
const callsAfterStreamEnd = session.sendRealtimeInput.mock.calls.length;
|
||||
bridge.sendAudio(silence20ms);
|
||||
expect(session.sendRealtimeInput).toHaveBeenCalledTimes(callsAfterStreamEnd);
|
||||
|
||||
session.sendRealtimeInput.mockClear();
|
||||
bridge.sendAudio(Buffer.alloc(160, 0x7f));
|
||||
bridge.sendAudio(silence20ms);
|
||||
bridge.sendAudio(silence20ms);
|
||||
bridge.sendAudio(silence20ms);
|
||||
|
||||
expect(session.sendRealtimeInput).toHaveBeenCalledWith({ audioStreamEnd: true });
|
||||
});
|
||||
|
||||
it("fuses telephony mu-law conversion into the Gemini 16 kHz PCM input frame", async () => {
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
const bridge = provider.createBridge({
|
||||
providerConfig: { apiKey: "gemini-key" },
|
||||
onAudio: vi.fn(),
|
||||
onClearAudio: vi.fn(),
|
||||
});
|
||||
|
||||
await bridge.connect();
|
||||
lastConnectParams().callbacks.onopen();
|
||||
lastConnectParams().callbacks.onmessage({ setupComplete: { sessionId: "session-1" } });
|
||||
|
||||
bridge.sendAudio(Buffer.from([0xff, 0x00]));
|
||||
|
||||
const audio = sentAudio();
|
||||
expect(typeof audio.data).toBe("string");
|
||||
expect(audio.mimeType).toBe("audio/pcm;rate=16000");
|
||||
const sent = Buffer.from(audio.data as string, "base64");
|
||||
expect(Array.from({ length: sent.length / 2 }, (_, i) => sent.readInt16LE(i * 2))).toEqual([
|
||||
0, -16062, -32124, -32124,
|
||||
]);
|
||||
});
|
||||
|
||||
it("accepts PCM16 24 kHz audio without the telephony mu-law hop", async () => {
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
const bridge = provider.createBridge({
|
||||
providerConfig: { apiKey: "gemini-key" },
|
||||
audioFormat: REALTIME_VOICE_AUDIO_FORMAT_PCM16_24KHZ,
|
||||
onAudio: vi.fn(),
|
||||
onClearAudio: vi.fn(),
|
||||
});
|
||||
|
||||
await bridge.connect();
|
||||
lastConnectParams().callbacks.onopen();
|
||||
lastConnectParams().callbacks.onmessage({ setupComplete: { sessionId: "session-1" } });
|
||||
|
||||
bridge.sendAudio(Buffer.alloc(480));
|
||||
|
||||
const audio = sentAudio();
|
||||
expect(typeof audio.data).toBe("string");
|
||||
expect(audio.mimeType).toBe("audio/pcm;rate=16000");
|
||||
const sent = Buffer.from(audio.data as string, "base64");
|
||||
expect(sent).toHaveLength(320);
|
||||
});
|
||||
|
||||
it("can disable automatic VAD for manual activity signaling experiments", async () => {
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
const bridge = provider.createBridge({
|
||||
providerConfig: {
|
||||
apiKey: "gemini-key",
|
||||
automaticActivityDetectionDisabled: true,
|
||||
},
|
||||
onAudio: vi.fn(),
|
||||
onClearAudio: vi.fn(),
|
||||
});
|
||||
|
||||
await bridge.connect();
|
||||
|
||||
const config = lastConnectParams().config as {
|
||||
realtimeInputConfig?: { automaticActivityDetection?: { disabled?: boolean } };
|
||||
};
|
||||
expect(config.realtimeInputConfig?.automaticActivityDetection?.disabled).toBe(true);
|
||||
});
|
||||
|
||||
it("sends text prompts as ordered client turns", async () => {
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
const bridge = provider.createBridge({
|
||||
providerConfig: { apiKey: "gemini-key" },
|
||||
onAudio: vi.fn(),
|
||||
onClearAudio: vi.fn(),
|
||||
});
|
||||
|
||||
await bridge.connect();
|
||||
lastConnectParams().callbacks.onopen();
|
||||
lastConnectParams().callbacks.onmessage({ setupComplete: { sessionId: "session-1" } });
|
||||
|
||||
bridge.sendUserMessage?.(" Say hello. ");
|
||||
|
||||
expect(session.sendClientContent).toHaveBeenCalledWith({
|
||||
turns: [{ role: "user", parts: [{ text: "Say hello." }] }],
|
||||
turnComplete: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("converts Google PCM output to mu-law audio", async () => {
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
const onAudio = vi.fn();
|
||||
const bridge = provider.createBridge({
|
||||
providerConfig: { apiKey: "gemini-key" },
|
||||
onAudio,
|
||||
onClearAudio: vi.fn(),
|
||||
});
|
||||
const pcm24k = Buffer.alloc(480);
|
||||
|
||||
await bridge.connect();
|
||||
lastConnectParams().callbacks.onmessage({
|
||||
setupComplete: { sessionId: "session-1" },
|
||||
serverContent: {
|
||||
modelTurn: {
|
||||
parts: [
|
||||
{
|
||||
inlineData: {
|
||||
mimeType: "audio/L16;codec=pcm;rate=24000",
|
||||
data: pcm24k.toString("base64"),
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
expect(onAudio).toHaveBeenCalledTimes(1);
|
||||
const audio = requireFirstAudio(onAudio);
|
||||
expect(audio).toBeInstanceOf(Buffer);
|
||||
expect(audio).toHaveLength(80);
|
||||
});
|
||||
|
||||
it("can keep Google PCM output as PCM16 24 kHz audio", async () => {
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
const onAudio = vi.fn();
|
||||
const bridge = provider.createBridge({
|
||||
providerConfig: { apiKey: "gemini-key" },
|
||||
audioFormat: REALTIME_VOICE_AUDIO_FORMAT_PCM16_24KHZ,
|
||||
onAudio,
|
||||
onClearAudio: vi.fn(),
|
||||
});
|
||||
const pcm24k = Buffer.alloc(480);
|
||||
|
||||
await bridge.connect();
|
||||
lastConnectParams().callbacks.onmessage({
|
||||
setupComplete: { sessionId: "session-1" },
|
||||
serverContent: {
|
||||
modelTurn: {
|
||||
parts: [
|
||||
{
|
||||
inlineData: {
|
||||
mimeType: "audio/L16;codec=pcm;rate=24000",
|
||||
data: pcm24k.toString("base64"),
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
expect(onAudio).toHaveBeenCalledTimes(1);
|
||||
expect(requireFirstAudio(onAudio)).toEqual(pcm24k);
|
||||
});
|
||||
|
||||
it("does not forward Google thought text as assistant transcript", async () => {
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
const onTranscript = vi.fn();
|
||||
const bridge = provider.createBridge({
|
||||
providerConfig: { apiKey: "gemini-key" },
|
||||
onAudio: vi.fn(),
|
||||
onClearAudio: vi.fn(),
|
||||
onTranscript,
|
||||
});
|
||||
|
||||
await bridge.connect();
|
||||
lastConnectParams().callbacks.onmessage({
|
||||
setupComplete: {},
|
||||
serverContent: {
|
||||
modelTurn: {
|
||||
parts: [{ text: "internal reasoning", thought: true }],
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
expect(onTranscript).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("forwards Live API tool calls and submits matching function responses", async () => {
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
const onToolCall = vi.fn();
|
||||
const bridge = provider.createBridge({
|
||||
providerConfig: { apiKey: "gemini-key" },
|
||||
onAudio: vi.fn(),
|
||||
onClearAudio: vi.fn(),
|
||||
onToolCall,
|
||||
});
|
||||
|
||||
await bridge.connect();
|
||||
lastConnectParams().callbacks.onmessage({
|
||||
setupComplete: { sessionId: "session-1" },
|
||||
toolCall: {
|
||||
functionCalls: [{ id: "call-1", name: "lookup", args: { query: "hi" } }],
|
||||
},
|
||||
});
|
||||
|
||||
expect(onToolCall).toHaveBeenCalledWith({
|
||||
itemId: "call-1",
|
||||
callId: "call-1",
|
||||
name: "lookup",
|
||||
args: { query: "hi" },
|
||||
});
|
||||
|
||||
bridge.submitToolResult("call-1", { result: "ok" });
|
||||
|
||||
expect(session.sendToolResponse).toHaveBeenCalledWith({
|
||||
functionResponses: [
|
||||
{
|
||||
id: "call-1",
|
||||
name: "lookup",
|
||||
response: { result: "ok" },
|
||||
},
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps Google Live consult calls open after continuing tool responses", async () => {
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
const bridge = provider.createBridge({
|
||||
providerConfig: { apiKey: "gemini-key" },
|
||||
onAudio: vi.fn(),
|
||||
onClearAudio: vi.fn(),
|
||||
onToolCall: vi.fn(),
|
||||
});
|
||||
|
||||
await bridge.connect();
|
||||
lastConnectParams().callbacks.onmessage({
|
||||
setupComplete: { sessionId: "session-1" },
|
||||
toolCall: {
|
||||
functionCalls: [
|
||||
{ id: "consult-call", name: "openclaw_agent_consult", args: { prompt: "hi" } },
|
||||
],
|
||||
},
|
||||
});
|
||||
|
||||
bridge.submitToolResult(
|
||||
"consult-call",
|
||||
{ status: "working", message: "Tell the participant you are checking." },
|
||||
{ willContinue: true },
|
||||
);
|
||||
bridge.submitToolResult("consult-call", { text: "The meeting starts at 3." });
|
||||
|
||||
expect(session.sendToolResponse).toHaveBeenNthCalledWith(1, {
|
||||
functionResponses: [
|
||||
{
|
||||
id: "consult-call",
|
||||
name: "openclaw_agent_consult",
|
||||
scheduling: "WHEN_IDLE",
|
||||
willContinue: true,
|
||||
response: { status: "working", message: "Tell the participant you are checking." },
|
||||
},
|
||||
],
|
||||
});
|
||||
expect(session.sendToolResponse).toHaveBeenNthCalledWith(2, {
|
||||
functionResponses: [
|
||||
{
|
||||
id: "consult-call",
|
||||
name: "openclaw_agent_consult",
|
||||
scheduling: "WHEN_IDLE",
|
||||
response: { text: "The meeting starts at 3." },
|
||||
},
|
||||
],
|
||||
});
|
||||
});
|
||||
|
||||
it("does not send malformed Live API tool responses without a matching call name", async () => {
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
const onError = vi.fn();
|
||||
const bridge = provider.createBridge({
|
||||
providerConfig: { apiKey: "gemini-key" },
|
||||
onAudio: vi.fn(),
|
||||
onClearAudio: vi.fn(),
|
||||
onError,
|
||||
});
|
||||
|
||||
await bridge.connect();
|
||||
|
||||
bridge.submitToolResult("missing-call", { result: "ok" });
|
||||
|
||||
expect(session.sendToolResponse).not.toHaveBeenCalled();
|
||||
const error = requireFirstError(onError);
|
||||
expect(error.message).toBe(
|
||||
"Google Live function response is missing a matching function call for missing-call",
|
||||
);
|
||||
});
|
||||
|
||||
it("reports Google Live tool response send failures without losing the call name", async () => {
|
||||
const provider = buildGoogleRealtimeVoiceProvider();
|
||||
const onError = vi.fn();
|
||||
const bridge = provider.createBridge({
|
||||
providerConfig: { apiKey: "gemini-key" },
|
||||
onAudio: vi.fn(),
|
||||
onClearAudio: vi.fn(),
|
||||
onError,
|
||||
});
|
||||
|
||||
await bridge.connect();
|
||||
lastConnectParams().callbacks.onmessage({
|
||||
setupComplete: { sessionId: "session-1" },
|
||||
toolCall: {
|
||||
functionCalls: [{ id: "call-1", name: "lookup", args: { query: "hi" } }],
|
||||
},
|
||||
});
|
||||
|
||||
const sendError = new Error("SDK send failed");
|
||||
session.sendToolResponse.mockImplementationOnce(() => {
|
||||
throw sendError;
|
||||
});
|
||||
|
||||
bridge.submitToolResult("call-1", ["retryable"]);
|
||||
|
||||
expect(onError).toHaveBeenCalledWith(sendError);
|
||||
|
||||
bridge.submitToolResult("call-1", { result: "ok" });
|
||||
|
||||
expect(session.sendToolResponse).toHaveBeenLastCalledWith({
|
||||
functionResponses: [
|
||||
{
|
||||
id: "call-1",
|
||||
name: "lookup",
|
||||
response: { result: "ok" },
|
||||
},
|
||||
],
|
||||
});
|
||||
});
|
||||
});
|
||||
972
extensions/google/realtime-voice-provider.ts
Normal file
972
extensions/google/realtime-voice-provider.ts
Normal file
@@ -0,0 +1,972 @@
|
||||
// Google provider module implements model/runtime integration.
|
||||
import { randomUUID } from "node:crypto";
|
||||
import type {
|
||||
ActivityHandling,
|
||||
Behavior,
|
||||
EndSensitivity,
|
||||
FunctionDeclaration,
|
||||
FunctionResponse,
|
||||
FunctionResponseScheduling,
|
||||
LiveConnectConfig,
|
||||
LiveServerContent,
|
||||
LiveServerMessage,
|
||||
LiveServerToolCall,
|
||||
Modality,
|
||||
RealtimeInputConfig,
|
||||
StartSensitivity,
|
||||
ThinkingConfig,
|
||||
TurnCoverage,
|
||||
} from "@google/genai";
|
||||
import {
|
||||
resolveExpiresAtMsFromDurationMs,
|
||||
timestampMsToIsoString,
|
||||
} from "openclaw/plugin-sdk/number-runtime";
|
||||
import type { OpenClawConfig } from "openclaw/plugin-sdk/provider-onboard";
|
||||
import type {
|
||||
RealtimeVoiceAudioFormat,
|
||||
RealtimeVoiceBridge,
|
||||
RealtimeVoiceBrowserSession,
|
||||
RealtimeVoiceBrowserSessionCreateRequest,
|
||||
RealtimeVoiceBridgeCreateRequest,
|
||||
RealtimeVoiceProviderConfig,
|
||||
RealtimeVoiceProviderPlugin,
|
||||
RealtimeVoiceTool,
|
||||
RealtimeVoiceToolResultOptions,
|
||||
} from "openclaw/plugin-sdk/realtime-voice";
|
||||
import {
|
||||
convertPcmToMulaw8k,
|
||||
mulawToPcm,
|
||||
REALTIME_VOICE_AUDIO_FORMAT_G711_ULAW_8KHZ,
|
||||
REALTIME_VOICE_AUDIO_FORMAT_PCM16_24KHZ,
|
||||
REALTIME_VOICE_AGENT_CONSULT_TOOL_NAME,
|
||||
resamplePcm,
|
||||
} from "openclaw/plugin-sdk/realtime-voice";
|
||||
import { normalizeResolvedSecretInputString } from "openclaw/plugin-sdk/secret-input";
|
||||
import {
|
||||
asBoolean,
|
||||
asFiniteNumber,
|
||||
normalizeOptionalString,
|
||||
} from "openclaw/plugin-sdk/string-coerce-runtime";
|
||||
import { createGoogleGenAI } from "./google-genai-runtime.js";
|
||||
|
||||
const GOOGLE_REALTIME_DEFAULT_MODEL = "gemini-2.5-flash-native-audio-preview-12-2025";
|
||||
const GOOGLE_REALTIME_DEFAULT_VOICE = "Kore";
|
||||
const GOOGLE_REALTIME_DEFAULT_API_VERSION = "v1beta";
|
||||
const GOOGLE_REALTIME_INPUT_SAMPLE_RATE = 16_000;
|
||||
const GOOGLE_REALTIME_BROWSER_API_VERSION = "v1alpha";
|
||||
const GOOGLE_REALTIME_BROWSER_WEBSOCKET_URL =
|
||||
"wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContentConstrained";
|
||||
const MAX_PENDING_AUDIO_CHUNKS = 320;
|
||||
const DEFAULT_AUDIO_STREAM_END_SILENCE_MS = 500;
|
||||
const GOOGLE_REALTIME_BROWSER_SESSION_TTL_MS = 30 * 60 * 1000;
|
||||
const GOOGLE_REALTIME_BROWSER_NEW_SESSION_TTL_MS = 60 * 1000;
|
||||
const GOOGLE_REALTIME_RECONNECT_MAX_ATTEMPTS = 3;
|
||||
const GOOGLE_REALTIME_RECONNECT_BASE_DELAY_MS = 250;
|
||||
const GOOGLE_REALTIME_RECONNECT_MAX_DELAY_MS = 2_000;
|
||||
const MULAW_LINEAR_SAMPLES = new Int16Array(256);
|
||||
|
||||
for (let i = 0; i < MULAW_LINEAR_SAMPLES.length; i += 1) {
|
||||
MULAW_LINEAR_SAMPLES[i] = decodeMulawSample(i);
|
||||
}
|
||||
|
||||
type GoogleRealtimeSensitivity = "low" | "high";
|
||||
type GoogleRealtimeThinkingLevel = "minimal" | "low" | "medium" | "high";
|
||||
type GoogleRealtimeActivityHandling = "start-of-activity-interrupts" | "no-interruption";
|
||||
type GoogleRealtimeTurnCoverage = "only-activity" | "all-input" | "audio-activity-and-all-video";
|
||||
|
||||
type GoogleRealtimeVoiceProviderConfig = {
|
||||
apiKey?: string;
|
||||
model?: string;
|
||||
voice?: string;
|
||||
temperature?: number;
|
||||
apiVersion?: string;
|
||||
prefixPaddingMs?: number;
|
||||
silenceDurationMs?: number;
|
||||
startSensitivity?: GoogleRealtimeSensitivity;
|
||||
endSensitivity?: GoogleRealtimeSensitivity;
|
||||
activityHandling?: GoogleRealtimeActivityHandling;
|
||||
turnCoverage?: GoogleRealtimeTurnCoverage;
|
||||
automaticActivityDetectionDisabled?: boolean;
|
||||
enableAffectiveDialog?: boolean;
|
||||
sessionResumption?: boolean;
|
||||
contextWindowCompression?: boolean;
|
||||
thinkingLevel?: GoogleRealtimeThinkingLevel;
|
||||
thinkingBudget?: number;
|
||||
};
|
||||
|
||||
type GoogleRealtimeLiveConfig = {
|
||||
apiKey: string;
|
||||
instructions?: string;
|
||||
tools?: RealtimeVoiceTool[];
|
||||
model?: string;
|
||||
voice?: string;
|
||||
temperature?: number;
|
||||
apiVersion?: string;
|
||||
prefixPaddingMs?: number;
|
||||
silenceDurationMs?: number;
|
||||
startSensitivity?: GoogleRealtimeSensitivity;
|
||||
endSensitivity?: GoogleRealtimeSensitivity;
|
||||
activityHandling?: GoogleRealtimeActivityHandling;
|
||||
turnCoverage?: GoogleRealtimeTurnCoverage;
|
||||
automaticActivityDetectionDisabled?: boolean;
|
||||
enableAffectiveDialog?: boolean;
|
||||
sessionResumption?: boolean;
|
||||
contextWindowCompression?: boolean;
|
||||
thinkingLevel?: GoogleRealtimeThinkingLevel;
|
||||
thinkingBudget?: number;
|
||||
};
|
||||
|
||||
type GoogleRealtimeVoiceBridgeConfig = RealtimeVoiceBridgeCreateRequest & GoogleRealtimeLiveConfig;
|
||||
|
||||
type GoogleLiveSession = {
|
||||
sendClientContent: (params: {
|
||||
turns?: Array<{ role: string; parts: Array<{ text: string }> }>;
|
||||
turnComplete?: boolean;
|
||||
}) => void;
|
||||
sendRealtimeInput: (params: {
|
||||
audio?: { data: string; mimeType: string };
|
||||
audioStreamEnd?: boolean;
|
||||
}) => void;
|
||||
sendToolResponse: (params: { functionResponses: FunctionResponse[] | FunctionResponse }) => void;
|
||||
close: () => void;
|
||||
};
|
||||
|
||||
function trimToUndefined(value: unknown): string | undefined {
|
||||
return normalizeOptionalString(value);
|
||||
}
|
||||
|
||||
function asSensitivity(value: unknown): GoogleRealtimeSensitivity | undefined {
|
||||
const normalized = normalizeOptionalString(value)?.toLowerCase();
|
||||
return normalized === "low" || normalized === "high" ? normalized : undefined;
|
||||
}
|
||||
|
||||
function asThinkingLevel(value: unknown): GoogleRealtimeThinkingLevel | undefined {
|
||||
const normalized = normalizeOptionalString(value)?.toLowerCase();
|
||||
return normalized === "minimal" ||
|
||||
normalized === "low" ||
|
||||
normalized === "medium" ||
|
||||
normalized === "high"
|
||||
? normalized
|
||||
: undefined;
|
||||
}
|
||||
|
||||
function asActivityHandling(value: unknown): GoogleRealtimeActivityHandling | undefined {
|
||||
const normalized = normalizeOptionalString(value)?.toLowerCase().replaceAll("_", "-");
|
||||
switch (normalized) {
|
||||
case "start-of-activity-interrupts":
|
||||
case "start-of-activity-interrupt":
|
||||
case "interrupt":
|
||||
case "interrupts":
|
||||
return "start-of-activity-interrupts";
|
||||
case "no-interruption":
|
||||
case "no-interruptions":
|
||||
case "none":
|
||||
return "no-interruption";
|
||||
default:
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
function asTurnCoverage(value: unknown): GoogleRealtimeTurnCoverage | undefined {
|
||||
const normalized = normalizeOptionalString(value)?.toLowerCase().replaceAll("_", "-");
|
||||
switch (normalized) {
|
||||
case "only-activity":
|
||||
case "turn-includes-only-activity":
|
||||
return "only-activity";
|
||||
case "all-input":
|
||||
case "turn-includes-all-input":
|
||||
return "all-input";
|
||||
case "audio-activity-and-all-video":
|
||||
case "turn-includes-audio-activity-and-all-video":
|
||||
return "audio-activity-and-all-video";
|
||||
default:
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
function asNonNegativeInteger(value: unknown): number | undefined {
|
||||
const number = asFiniteNumber(value);
|
||||
return number !== undefined && Number.isSafeInteger(number) && number >= 0 ? number : undefined;
|
||||
}
|
||||
|
||||
function asGoogleRealtimeThinkingBudget(value: unknown): number | undefined {
|
||||
const budget = asFiniteNumber(value);
|
||||
return budget !== undefined &&
|
||||
Number.isSafeInteger(budget) &&
|
||||
(budget === -1 || (budget >= 0 && budget <= 24_576))
|
||||
? budget
|
||||
: undefined;
|
||||
}
|
||||
|
||||
function resolveGoogleRealtimeProviderConfigRecord(
|
||||
config: Record<string, unknown>,
|
||||
): Record<string, unknown> | undefined {
|
||||
const providers =
|
||||
typeof config.providers === "object" &&
|
||||
config.providers !== null &&
|
||||
!Array.isArray(config.providers)
|
||||
? (config.providers as Record<string, unknown>)
|
||||
: undefined;
|
||||
const nested = providers?.google;
|
||||
return typeof nested === "object" && nested !== null && !Array.isArray(nested)
|
||||
? (nested as Record<string, unknown>)
|
||||
: typeof config.google === "object" && config.google !== null && !Array.isArray(config.google)
|
||||
? (config.google as Record<string, unknown>)
|
||||
: config;
|
||||
}
|
||||
|
||||
function normalizeProviderConfig(
|
||||
config: RealtimeVoiceProviderConfig,
|
||||
cfg?: OpenClawConfig,
|
||||
): GoogleRealtimeVoiceProviderConfig {
|
||||
const raw = resolveGoogleRealtimeProviderConfigRecord(config);
|
||||
return {
|
||||
apiKey: normalizeResolvedSecretInputString({
|
||||
value: raw?.apiKey ?? cfg?.models?.providers?.google?.apiKey,
|
||||
path: "plugins.entries.voice-call.config.realtime.providers.google.apiKey",
|
||||
}),
|
||||
model: trimToUndefined(raw?.model),
|
||||
voice: trimToUndefined(raw?.speakerVoice) ?? trimToUndefined(raw?.voice),
|
||||
temperature: asFiniteNumber(raw?.temperature),
|
||||
apiVersion: trimToUndefined(raw?.apiVersion),
|
||||
prefixPaddingMs: asNonNegativeInteger(raw?.prefixPaddingMs),
|
||||
silenceDurationMs: asNonNegativeInteger(raw?.silenceDurationMs),
|
||||
startSensitivity: asSensitivity(raw?.startSensitivity),
|
||||
endSensitivity: asSensitivity(raw?.endSensitivity),
|
||||
activityHandling: asActivityHandling(raw?.activityHandling),
|
||||
turnCoverage: asTurnCoverage(raw?.turnCoverage),
|
||||
automaticActivityDetectionDisabled: asBoolean(raw?.automaticActivityDetectionDisabled),
|
||||
enableAffectiveDialog: asBoolean(raw?.enableAffectiveDialog),
|
||||
sessionResumption: asBoolean(raw?.sessionResumption),
|
||||
contextWindowCompression: asBoolean(raw?.contextWindowCompression),
|
||||
thinkingLevel: asThinkingLevel(raw?.thinkingLevel),
|
||||
thinkingBudget: asGoogleRealtimeThinkingBudget(raw?.thinkingBudget),
|
||||
};
|
||||
}
|
||||
|
||||
function resolveEnvApiKey(): string | undefined {
|
||||
return trimToUndefined(process.env.GEMINI_API_KEY) ?? trimToUndefined(process.env.GOOGLE_API_KEY);
|
||||
}
|
||||
|
||||
function mapStartSensitivity(
|
||||
value: GoogleRealtimeSensitivity | undefined,
|
||||
): StartSensitivity | undefined {
|
||||
switch (value) {
|
||||
case "high":
|
||||
return "START_SENSITIVITY_HIGH" as StartSensitivity;
|
||||
case "low":
|
||||
return "START_SENSITIVITY_LOW" as StartSensitivity;
|
||||
default:
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
function mapEndSensitivity(
|
||||
value: GoogleRealtimeSensitivity | undefined,
|
||||
): EndSensitivity | undefined {
|
||||
switch (value) {
|
||||
case "high":
|
||||
return "END_SENSITIVITY_HIGH" as EndSensitivity;
|
||||
case "low":
|
||||
return "END_SENSITIVITY_LOW" as EndSensitivity;
|
||||
default:
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
function mapActivityHandling(
|
||||
value: GoogleRealtimeActivityHandling | undefined,
|
||||
): ActivityHandling | undefined {
|
||||
switch (value) {
|
||||
case "no-interruption":
|
||||
return "NO_INTERRUPTION" as ActivityHandling;
|
||||
case "start-of-activity-interrupts":
|
||||
return "START_OF_ACTIVITY_INTERRUPTS" as ActivityHandling;
|
||||
default:
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
function mapTurnCoverage(value: GoogleRealtimeTurnCoverage | undefined): TurnCoverage | undefined {
|
||||
switch (value) {
|
||||
case "only-activity":
|
||||
return "TURN_INCLUDES_ONLY_ACTIVITY" as TurnCoverage;
|
||||
case "all-input":
|
||||
return "TURN_INCLUDES_ALL_INPUT" as TurnCoverage;
|
||||
case "audio-activity-and-all-video":
|
||||
return "TURN_INCLUDES_AUDIO_ACTIVITY_AND_ALL_VIDEO" as TurnCoverage;
|
||||
default:
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
function buildThinkingConfig(config: GoogleRealtimeLiveConfig): ThinkingConfig | undefined {
|
||||
if (config.thinkingLevel) {
|
||||
return { thinkingLevel: config.thinkingLevel.toUpperCase() as ThinkingConfig["thinkingLevel"] };
|
||||
}
|
||||
if (typeof config.thinkingBudget === "number") {
|
||||
return { thinkingBudget: config.thinkingBudget };
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function buildRealtimeInputConfig(
|
||||
config: GoogleRealtimeLiveConfig,
|
||||
): RealtimeInputConfig | undefined {
|
||||
const startSensitivity = mapStartSensitivity(config.startSensitivity);
|
||||
const endSensitivity = mapEndSensitivity(config.endSensitivity);
|
||||
const activityHandling = mapActivityHandling(config.activityHandling);
|
||||
const turnCoverage = mapTurnCoverage(config.turnCoverage);
|
||||
const automaticActivityDetection = {
|
||||
...(typeof config.automaticActivityDetectionDisabled === "boolean"
|
||||
? { disabled: config.automaticActivityDetectionDisabled }
|
||||
: {}),
|
||||
...(startSensitivity ? { startOfSpeechSensitivity: startSensitivity } : {}),
|
||||
...(endSensitivity ? { endOfSpeechSensitivity: endSensitivity } : {}),
|
||||
...(typeof config.prefixPaddingMs === "number"
|
||||
? { prefixPaddingMs: config.prefixPaddingMs }
|
||||
: {}),
|
||||
...(typeof config.silenceDurationMs === "number"
|
||||
? { silenceDurationMs: config.silenceDurationMs }
|
||||
: {}),
|
||||
};
|
||||
const realtimeInputConfig = {
|
||||
...(Object.keys(automaticActivityDetection).length > 0 ? { automaticActivityDetection } : {}),
|
||||
...(activityHandling ? { activityHandling } : {}),
|
||||
...(turnCoverage ? { turnCoverage } : {}),
|
||||
};
|
||||
return Object.keys(realtimeInputConfig).length > 0 ? realtimeInputConfig : undefined;
|
||||
}
|
||||
|
||||
function buildFunctionDeclarations(tools: RealtimeVoiceTool[] | undefined): FunctionDeclaration[] {
|
||||
return (tools ?? []).map((tool) => {
|
||||
const declaration: FunctionDeclaration = {
|
||||
name: tool.name,
|
||||
description: tool.description,
|
||||
parametersJsonSchema: tool.parameters,
|
||||
};
|
||||
if (tool.name === REALTIME_VOICE_AGENT_CONSULT_TOOL_NAME) {
|
||||
declaration.behavior = "NON_BLOCKING" as Behavior;
|
||||
}
|
||||
return declaration;
|
||||
});
|
||||
}
|
||||
|
||||
function buildGoogleLiveConnectConfig(config: GoogleRealtimeLiveConfig): LiveConnectConfig {
|
||||
const functionDeclarations = buildFunctionDeclarations(config.tools);
|
||||
const realtimeInputConfig = buildRealtimeInputConfig(config);
|
||||
const thinkingConfig = buildThinkingConfig(config);
|
||||
return {
|
||||
responseModalities: ["AUDIO" as Modality],
|
||||
...(typeof config.temperature === "number" && config.temperature > 0
|
||||
? { temperature: config.temperature }
|
||||
: {}),
|
||||
speechConfig: {
|
||||
voiceConfig: {
|
||||
prebuiltVoiceConfig: {
|
||||
voiceName: config.voice ?? GOOGLE_REALTIME_DEFAULT_VOICE,
|
||||
},
|
||||
},
|
||||
},
|
||||
systemInstruction: config.instructions,
|
||||
...(functionDeclarations.length > 0 ? { tools: [{ functionDeclarations }] } : {}),
|
||||
...(realtimeInputConfig ? { realtimeInputConfig } : {}),
|
||||
inputAudioTranscription: {},
|
||||
outputAudioTranscription: {},
|
||||
...(typeof config.enableAffectiveDialog === "boolean"
|
||||
? { enableAffectiveDialog: config.enableAffectiveDialog }
|
||||
: {}),
|
||||
...(thinkingConfig ? { thinkingConfig } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
function toGoogleModelResource(model: string): string {
|
||||
return model.startsWith("models/") ? model : `models/${model}`;
|
||||
}
|
||||
|
||||
function buildBrowserInitialSetup(model: string) {
|
||||
return {
|
||||
setup: {
|
||||
model: toGoogleModelResource(model),
|
||||
generationConfig: {
|
||||
responseModalities: ["AUDIO" as Modality],
|
||||
},
|
||||
inputAudioTranscription: {},
|
||||
outputAudioTranscription: {},
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function parsePcmSampleRate(mimeType: string | undefined): number {
|
||||
const match = mimeType?.match(/(?:^|[;,\s])rate=(\d+)/i);
|
||||
const parsed = match ? Number.parseInt(match[1] ?? "", 10) : Number.NaN;
|
||||
return Number.isFinite(parsed) && parsed > 0 ? parsed : 24_000;
|
||||
}
|
||||
|
||||
function isMulawSilence(audio: Buffer): boolean {
|
||||
return audio.length > 0 && audio.every((sample) => sample === 0xff);
|
||||
}
|
||||
|
||||
function isPcm16Silence(audio: Buffer): boolean {
|
||||
const samples = Math.floor(audio.length / 2);
|
||||
if (samples === 0) {
|
||||
return false;
|
||||
}
|
||||
for (let i = 0; i < samples; i += 1) {
|
||||
if (audio.readInt16LE(i * 2) !== 0) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
function formatGoogleLiveCloseEvent(
|
||||
event:
|
||||
| {
|
||||
code?: number;
|
||||
reason?: string;
|
||||
wasClean?: boolean;
|
||||
}
|
||||
| undefined,
|
||||
): string {
|
||||
if (!event) {
|
||||
return "code=unknown reason=unknown";
|
||||
}
|
||||
const code = typeof event.code === "number" ? event.code : "unknown";
|
||||
const reason = event.reason?.trim() || "none";
|
||||
const clean = typeof event.wasClean === "boolean" ? ` clean=${event.wasClean}` : "";
|
||||
return `code=${code} reason=${reason}${clean}`;
|
||||
}
|
||||
|
||||
class GoogleRealtimeVoiceBridge implements RealtimeVoiceBridge {
|
||||
readonly supportsToolResultContinuation = true;
|
||||
|
||||
private session: GoogleLiveSession | null = null;
|
||||
private connected = false;
|
||||
private sessionConfigured = false;
|
||||
private intentionallyClosed = false;
|
||||
private pendingAudio: Buffer[] = [];
|
||||
private sessionReadyFired = false;
|
||||
private consecutiveSilenceMs = 0;
|
||||
private audioStreamEnded = false;
|
||||
private pendingFunctionNames = new Map<string, string>();
|
||||
private readonly audioFormat: RealtimeVoiceAudioFormat;
|
||||
private resumptionHandle: string | undefined;
|
||||
private reconnectAttempts = 0;
|
||||
private reconnectTimer: ReturnType<typeof setTimeout> | undefined;
|
||||
|
||||
constructor(private readonly config: GoogleRealtimeVoiceBridgeConfig) {
|
||||
this.audioFormat = config.audioFormat ?? REALTIME_VOICE_AUDIO_FORMAT_G711_ULAW_8KHZ;
|
||||
}
|
||||
|
||||
async connect(): Promise<void> {
|
||||
this.intentionallyClosed = false;
|
||||
this.sessionConfigured = false;
|
||||
this.sessionReadyFired = false;
|
||||
this.consecutiveSilenceMs = 0;
|
||||
this.audioStreamEnded = false;
|
||||
this.pendingFunctionNames.clear();
|
||||
|
||||
const ai = createGoogleGenAI({
|
||||
apiKey: this.config.apiKey,
|
||||
httpOptions: {
|
||||
apiVersion: this.config.apiVersion ?? GOOGLE_REALTIME_DEFAULT_API_VERSION,
|
||||
},
|
||||
});
|
||||
|
||||
this.session = (await ai.live.connect({
|
||||
model: this.config.model ?? GOOGLE_REALTIME_DEFAULT_MODEL,
|
||||
config: {
|
||||
...buildGoogleLiveConnectConfig(this.config),
|
||||
...(this.config.sessionResumption === false
|
||||
? {}
|
||||
: {
|
||||
sessionResumption: this.resumptionHandle ? { handle: this.resumptionHandle } : {},
|
||||
}),
|
||||
...(this.config.contextWindowCompression === false
|
||||
? {}
|
||||
: { contextWindowCompression: { slidingWindow: {} } }),
|
||||
},
|
||||
callbacks: {
|
||||
onopen: () => {
|
||||
this.connected = true;
|
||||
},
|
||||
onmessage: (message) => {
|
||||
this.handleMessage(message);
|
||||
},
|
||||
onerror: (event) => {
|
||||
const error =
|
||||
event.error instanceof Error
|
||||
? event.error
|
||||
: new Error(
|
||||
typeof event.message === "string" ? event.message : "Google Live API error",
|
||||
);
|
||||
this.config.onError?.(error);
|
||||
},
|
||||
onclose: (event) => {
|
||||
this.connected = false;
|
||||
this.sessionConfigured = false;
|
||||
this.pendingFunctionNames.clear();
|
||||
this.session = null;
|
||||
if (this.intentionallyClosed) {
|
||||
this.config.onClose?.("completed");
|
||||
return;
|
||||
}
|
||||
const closeDetails = formatGoogleLiveCloseEvent(event);
|
||||
if (this.scheduleReconnect(closeDetails)) {
|
||||
return;
|
||||
}
|
||||
this.config.onError?.(
|
||||
new Error(`Google Live session closed after reconnect attempts: ${closeDetails}`),
|
||||
);
|
||||
this.config.onClose?.("error");
|
||||
},
|
||||
},
|
||||
})) as GoogleLiveSession;
|
||||
}
|
||||
|
||||
sendAudio(audio: Buffer): void {
|
||||
if (!this.session || !this.connected || !this.sessionConfigured) {
|
||||
if (this.pendingAudio.length < MAX_PENDING_AUDIO_CHUNKS) {
|
||||
this.pendingAudio.push(audio);
|
||||
}
|
||||
return;
|
||||
}
|
||||
const silent = this.isSilence(audio);
|
||||
if (silent && this.audioStreamEnded) {
|
||||
return;
|
||||
}
|
||||
if (!silent) {
|
||||
this.consecutiveSilenceMs = 0;
|
||||
this.audioStreamEnded = false;
|
||||
}
|
||||
|
||||
const pcm16k = this.toGoogleInputPcm16k(audio);
|
||||
this.session.sendRealtimeInput({
|
||||
audio: {
|
||||
data: pcm16k.toString("base64"),
|
||||
mimeType: `audio/pcm;rate=${GOOGLE_REALTIME_INPUT_SAMPLE_RATE}`,
|
||||
},
|
||||
});
|
||||
|
||||
if (!silent) {
|
||||
return;
|
||||
}
|
||||
|
||||
const silenceThresholdMs =
|
||||
typeof this.config.silenceDurationMs === "number"
|
||||
? Math.max(0, Math.floor(this.config.silenceDurationMs))
|
||||
: DEFAULT_AUDIO_STREAM_END_SILENCE_MS;
|
||||
const bytesPerSample = this.audioFormat.encoding === "pcm16" ? 2 : 1;
|
||||
this.consecutiveSilenceMs += Math.round(
|
||||
(audio.length / bytesPerSample / this.audioFormat.sampleRateHz) * 1000,
|
||||
);
|
||||
if (!this.audioStreamEnded && this.consecutiveSilenceMs >= silenceThresholdMs) {
|
||||
this.session.sendRealtimeInput({ audioStreamEnd: true });
|
||||
this.audioStreamEnded = true;
|
||||
}
|
||||
}
|
||||
|
||||
setMediaTimestamp(_ts: number): void {}
|
||||
|
||||
sendUserMessage(text: string): void {
|
||||
const normalized = text.trim();
|
||||
if (!normalized || !this.session || !this.connected || !this.sessionConfigured) {
|
||||
return;
|
||||
}
|
||||
this.session.sendClientContent({
|
||||
turns: [{ role: "user", parts: [{ text: normalized }] }],
|
||||
turnComplete: true,
|
||||
});
|
||||
}
|
||||
|
||||
triggerGreeting(instructions?: string): void {
|
||||
const greetingPrompt =
|
||||
instructions?.trim() || "Start the call now. Greet the caller naturally and keep it brief.";
|
||||
this.sendUserMessage(greetingPrompt);
|
||||
}
|
||||
|
||||
submitToolResult(
|
||||
callId: string,
|
||||
result: unknown,
|
||||
options?: RealtimeVoiceToolResultOptions,
|
||||
): void {
|
||||
if (!this.session) {
|
||||
return;
|
||||
}
|
||||
const name = this.pendingFunctionNames.get(callId);
|
||||
if (!name) {
|
||||
this.config.onError?.(
|
||||
new Error(
|
||||
`Google Live function response is missing a matching function call for ${callId}`,
|
||||
),
|
||||
);
|
||||
return;
|
||||
}
|
||||
try {
|
||||
const isConsultTool = name === REALTIME_VOICE_AGENT_CONSULT_TOOL_NAME;
|
||||
const functionResponse: FunctionResponse = {
|
||||
id: callId,
|
||||
name,
|
||||
response:
|
||||
result && typeof result === "object" && !Array.isArray(result)
|
||||
? (result as Record<string, unknown>)
|
||||
: { output: result },
|
||||
};
|
||||
if (isConsultTool) {
|
||||
functionResponse.scheduling = "WHEN_IDLE" as FunctionResponseScheduling;
|
||||
if (options?.willContinue === true) {
|
||||
functionResponse.willContinue = true;
|
||||
}
|
||||
} else if (options?.willContinue === true) {
|
||||
this.config.onError?.(
|
||||
new Error(
|
||||
`Google Live continuation is only supported for ${REALTIME_VOICE_AGENT_CONSULT_TOOL_NAME}`,
|
||||
),
|
||||
);
|
||||
return;
|
||||
}
|
||||
this.session.sendToolResponse({
|
||||
functionResponses: [functionResponse],
|
||||
});
|
||||
if (options?.willContinue !== true) {
|
||||
this.pendingFunctionNames.delete(callId);
|
||||
}
|
||||
} catch (error) {
|
||||
this.config.onError?.(
|
||||
error instanceof Error ? error : new Error("Failed to send Google Live function response"),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
acknowledgeMark(): void {}
|
||||
|
||||
close(): void {
|
||||
this.intentionallyClosed = true;
|
||||
this.connected = false;
|
||||
this.sessionConfigured = false;
|
||||
if (this.reconnectTimer) {
|
||||
clearTimeout(this.reconnectTimer);
|
||||
this.reconnectTimer = undefined;
|
||||
}
|
||||
this.pendingAudio = [];
|
||||
this.consecutiveSilenceMs = 0;
|
||||
this.audioStreamEnded = false;
|
||||
this.pendingFunctionNames.clear();
|
||||
const session = this.session;
|
||||
this.session = null;
|
||||
session?.close();
|
||||
}
|
||||
|
||||
isConnected(): boolean {
|
||||
return this.connected && this.sessionConfigured;
|
||||
}
|
||||
|
||||
private isSilence(audio: Buffer): boolean {
|
||||
return this.audioFormat.encoding === "pcm16" ? isPcm16Silence(audio) : isMulawSilence(audio);
|
||||
}
|
||||
|
||||
private toInputPcm(audio: Buffer): Buffer {
|
||||
return this.audioFormat.encoding === "pcm16" ? audio : mulawToPcm(audio);
|
||||
}
|
||||
|
||||
private toGoogleInputPcm16k(audio: Buffer): Buffer {
|
||||
if (
|
||||
this.audioFormat.encoding === "g711_ulaw" &&
|
||||
this.audioFormat.sampleRateHz === 8_000 &&
|
||||
GOOGLE_REALTIME_INPUT_SAMPLE_RATE === 16_000
|
||||
) {
|
||||
return convertMulaw8kToPcm16k(audio);
|
||||
}
|
||||
return resamplePcm(
|
||||
this.toInputPcm(audio),
|
||||
this.audioFormat.sampleRateHz,
|
||||
GOOGLE_REALTIME_INPUT_SAMPLE_RATE,
|
||||
);
|
||||
}
|
||||
|
||||
private toOutputAudio(pcm: Buffer, sampleRate: number): Buffer {
|
||||
return this.audioFormat.encoding === "pcm16"
|
||||
? resamplePcm(pcm, sampleRate, this.audioFormat.sampleRateHz)
|
||||
: convertPcmToMulaw8k(pcm, sampleRate);
|
||||
}
|
||||
|
||||
private handleMessage(message: LiveServerMessage): void {
|
||||
this.captureSessionLifecycle(message);
|
||||
if (message.setupComplete) {
|
||||
this.handleSetupComplete();
|
||||
}
|
||||
if (message.serverContent) {
|
||||
this.handleServerContent(message.serverContent);
|
||||
}
|
||||
if (message.toolCall) {
|
||||
this.handleToolCall(message.toolCall);
|
||||
}
|
||||
}
|
||||
|
||||
private captureSessionLifecycle(message: LiveServerMessage): void {
|
||||
const raw = message as unknown as {
|
||||
goAway?: { timeLeft?: string };
|
||||
sessionResumptionUpdate?: { newHandle?: string; resumable?: boolean };
|
||||
};
|
||||
const update = raw.sessionResumptionUpdate;
|
||||
if (update?.resumable && update.newHandle) {
|
||||
this.resumptionHandle = update.newHandle;
|
||||
}
|
||||
if (raw.goAway?.timeLeft) {
|
||||
this.config.onError?.(new Error(`Google Live session goAway: ${raw.goAway.timeLeft}`));
|
||||
}
|
||||
}
|
||||
|
||||
private handleSetupComplete(): void {
|
||||
this.sessionConfigured = true;
|
||||
this.reconnectAttempts = 0;
|
||||
for (const chunk of this.pendingAudio.splice(0)) {
|
||||
this.sendAudio(chunk);
|
||||
}
|
||||
if (!this.sessionReadyFired) {
|
||||
this.sessionReadyFired = true;
|
||||
this.config.onReady?.();
|
||||
}
|
||||
}
|
||||
|
||||
private handleServerContent(content: LiveServerContent): void {
|
||||
if (content.interrupted) {
|
||||
this.config.onClearAudio();
|
||||
}
|
||||
|
||||
if (content.inputTranscription?.text) {
|
||||
this.config.onTranscript?.(
|
||||
"user",
|
||||
content.inputTranscription.text,
|
||||
content.inputTranscription.finished ?? false,
|
||||
);
|
||||
}
|
||||
|
||||
if (content.outputTranscription?.text) {
|
||||
this.config.onTranscript?.(
|
||||
"assistant",
|
||||
content.outputTranscription.text,
|
||||
content.outputTranscription.finished ?? false,
|
||||
);
|
||||
}
|
||||
|
||||
for (const part of content.modelTurn?.parts ?? []) {
|
||||
if (part.inlineData?.data) {
|
||||
const pcm = Buffer.from(part.inlineData.data, "base64");
|
||||
const sampleRate = parsePcmSampleRate(part.inlineData.mimeType);
|
||||
const audio = this.toOutputAudio(pcm, sampleRate);
|
||||
if (audio.length > 0) {
|
||||
this.config.onAudio(audio);
|
||||
this.config.onMark?.(`audio-${randomUUID()}`);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (part.thought) {
|
||||
continue;
|
||||
}
|
||||
if (!content.outputTranscription?.text && typeof part.text === "string" && part.text.trim()) {
|
||||
this.config.onTranscript?.("assistant", part.text, content.turnComplete ?? false);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private handleToolCall(toolCall: LiveServerToolCall): void {
|
||||
for (const call of toolCall.functionCalls ?? []) {
|
||||
const name = call.name?.trim();
|
||||
if (!name) {
|
||||
continue;
|
||||
}
|
||||
const callId = call.id?.trim() || `google-live-${randomUUID()}`;
|
||||
this.pendingFunctionNames.set(callId, name);
|
||||
this.config.onToolCall?.({
|
||||
itemId: callId,
|
||||
callId,
|
||||
name,
|
||||
args: call.args ?? {},
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
private scheduleReconnect(closeDetails: string): boolean {
|
||||
if (this.reconnectAttempts >= GOOGLE_REALTIME_RECONNECT_MAX_ATTEMPTS) {
|
||||
return false;
|
||||
}
|
||||
const attempt = ++this.reconnectAttempts;
|
||||
const delayMs = Math.min(
|
||||
GOOGLE_REALTIME_RECONNECT_MAX_DELAY_MS,
|
||||
GOOGLE_REALTIME_RECONNECT_BASE_DELAY_MS * 2 ** (attempt - 1),
|
||||
);
|
||||
this.config.onError?.(
|
||||
new Error(
|
||||
`Google Live session closed unexpectedly (${closeDetails}); reconnecting ${attempt}/${GOOGLE_REALTIME_RECONNECT_MAX_ATTEMPTS} in ${delayMs}ms`,
|
||||
),
|
||||
);
|
||||
this.reconnectTimer = setTimeout(() => {
|
||||
this.reconnectTimer = undefined;
|
||||
if (this.intentionallyClosed) {
|
||||
return;
|
||||
}
|
||||
this.connect().catch((error: unknown) => {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
this.config.onError?.(error instanceof Error ? error : new Error(message));
|
||||
if (!this.scheduleReconnect(`connect failed: ${message}`)) {
|
||||
this.config.onClose?.("error");
|
||||
}
|
||||
});
|
||||
}, delayMs);
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
function convertMulaw8kToPcm16k(muLaw: Buffer): Buffer {
|
||||
if (muLaw.length === 0) {
|
||||
return Buffer.alloc(0);
|
||||
}
|
||||
const pcm = Buffer.alloc(muLaw.length * 4);
|
||||
for (let i = 0; i < muLaw.length; i += 1) {
|
||||
const current = MULAW_LINEAR_SAMPLES[muLaw[i] ?? 0] ?? 0;
|
||||
const next = MULAW_LINEAR_SAMPLES[muLaw[i + 1] ?? muLaw[i] ?? 0] ?? current;
|
||||
pcm.writeInt16LE(current, i * 4);
|
||||
pcm.writeInt16LE(Math.round((current + next) / 2), i * 4 + 2);
|
||||
}
|
||||
return pcm;
|
||||
}
|
||||
|
||||
function decodeMulawSample(value: number): number {
|
||||
const muLaw = ~value & 0xff;
|
||||
const sign = muLaw & 0x80;
|
||||
const exponent = (muLaw >> 4) & 0x07;
|
||||
const mantissa = muLaw & 0x0f;
|
||||
let sample = ((mantissa << 3) + 132) << exponent;
|
||||
sample -= 132;
|
||||
return sign ? -sample : sample;
|
||||
}
|
||||
|
||||
async function createGoogleRealtimeBrowserSession(
|
||||
req: RealtimeVoiceBrowserSessionCreateRequest,
|
||||
): Promise<RealtimeVoiceBrowserSession> {
|
||||
const config = normalizeProviderConfig(req.providerConfig);
|
||||
const apiKey = config.apiKey || resolveEnvApiKey();
|
||||
if (!apiKey) {
|
||||
throw new Error("Google Gemini API key missing");
|
||||
}
|
||||
|
||||
const model = req.model ?? config.model ?? GOOGLE_REALTIME_DEFAULT_MODEL;
|
||||
const voice = req.voice ?? config.voice ?? GOOGLE_REALTIME_DEFAULT_VOICE;
|
||||
const nowMs = Date.now();
|
||||
const expiresAtMs = resolveExpiresAtMsFromDurationMs(GOOGLE_REALTIME_BROWSER_SESSION_TTL_MS, {
|
||||
nowMs,
|
||||
});
|
||||
const newSessionExpiresAtMs = resolveExpiresAtMsFromDurationMs(
|
||||
GOOGLE_REALTIME_BROWSER_NEW_SESSION_TTL_MS,
|
||||
{ nowMs },
|
||||
);
|
||||
const expireTime = timestampMsToIsoString(expiresAtMs);
|
||||
const newSessionExpireTime = timestampMsToIsoString(newSessionExpiresAtMs);
|
||||
if (expiresAtMs === undefined || !expireTime || !newSessionExpireTime) {
|
||||
throw new Error("Google realtime browser session expiry is outside the supported Date range");
|
||||
}
|
||||
const ai = createGoogleGenAI({
|
||||
apiKey,
|
||||
httpOptions: {
|
||||
apiVersion: GOOGLE_REALTIME_BROWSER_API_VERSION,
|
||||
},
|
||||
});
|
||||
const token = await ai.authTokens.create({
|
||||
config: {
|
||||
uses: 1,
|
||||
expireTime,
|
||||
newSessionExpireTime,
|
||||
liveConnectConstraints: {
|
||||
model,
|
||||
config: buildGoogleLiveConnectConfig({
|
||||
...config,
|
||||
apiKey,
|
||||
model,
|
||||
voice,
|
||||
instructions: req.instructions,
|
||||
tools: req.tools,
|
||||
}),
|
||||
},
|
||||
},
|
||||
});
|
||||
const clientSecret = token.name?.trim();
|
||||
if (!clientSecret) {
|
||||
throw new Error("Google Live browser session did not return an ephemeral token");
|
||||
}
|
||||
|
||||
return {
|
||||
provider: "google",
|
||||
transport: "provider-websocket",
|
||||
protocol: "google-live-bidi",
|
||||
clientSecret,
|
||||
websocketUrl: GOOGLE_REALTIME_BROWSER_WEBSOCKET_URL,
|
||||
audio: {
|
||||
inputEncoding: "pcm16",
|
||||
inputSampleRateHz: GOOGLE_REALTIME_INPUT_SAMPLE_RATE,
|
||||
outputEncoding: "pcm16",
|
||||
outputSampleRateHz: 24_000,
|
||||
},
|
||||
initialMessage: buildBrowserInitialSetup(model),
|
||||
model,
|
||||
voice,
|
||||
expiresAt: Math.floor(expiresAtMs / 1000),
|
||||
};
|
||||
}
|
||||
|
||||
export function buildGoogleRealtimeVoiceProvider(): RealtimeVoiceProviderPlugin {
|
||||
return {
|
||||
id: "google",
|
||||
label: "Google Live Voice",
|
||||
defaultModel: GOOGLE_REALTIME_DEFAULT_MODEL,
|
||||
autoSelectOrder: 20,
|
||||
capabilities: {
|
||||
transports: ["provider-websocket", "gateway-relay"],
|
||||
inputAudioFormats: [
|
||||
REALTIME_VOICE_AUDIO_FORMAT_G711_ULAW_8KHZ,
|
||||
REALTIME_VOICE_AUDIO_FORMAT_PCM16_24KHZ,
|
||||
],
|
||||
outputAudioFormats: [
|
||||
REALTIME_VOICE_AUDIO_FORMAT_G711_ULAW_8KHZ,
|
||||
REALTIME_VOICE_AUDIO_FORMAT_PCM16_24KHZ,
|
||||
],
|
||||
supportsBrowserSession: true,
|
||||
supportsBargeIn: true,
|
||||
supportsToolCalls: true,
|
||||
supportsVideoFrames: true,
|
||||
supportsSessionResumption: true,
|
||||
},
|
||||
resolveConfig: ({ cfg, rawConfig }) => normalizeProviderConfig(rawConfig, cfg),
|
||||
isConfigured: ({ providerConfig }) =>
|
||||
Boolean(normalizeProviderConfig(providerConfig).apiKey || resolveEnvApiKey()),
|
||||
createBridge: (req) => {
|
||||
const config = normalizeProviderConfig(req.providerConfig);
|
||||
const apiKey = config.apiKey || resolveEnvApiKey();
|
||||
if (!apiKey) {
|
||||
throw new Error("Google Gemini API key missing");
|
||||
}
|
||||
return new GoogleRealtimeVoiceBridge({
|
||||
...req,
|
||||
apiKey,
|
||||
model: config.model,
|
||||
voice: config.voice,
|
||||
temperature: config.temperature,
|
||||
apiVersion: config.apiVersion,
|
||||
prefixPaddingMs: config.prefixPaddingMs,
|
||||
silenceDurationMs: config.silenceDurationMs,
|
||||
startSensitivity: config.startSensitivity,
|
||||
endSensitivity: config.endSensitivity,
|
||||
activityHandling: config.activityHandling,
|
||||
turnCoverage: config.turnCoverage,
|
||||
automaticActivityDetectionDisabled: config.automaticActivityDetectionDisabled,
|
||||
enableAffectiveDialog: config.enableAffectiveDialog,
|
||||
sessionResumption: config.sessionResumption,
|
||||
contextWindowCompression: config.contextWindowCompression,
|
||||
thinkingLevel: config.thinkingLevel,
|
||||
thinkingBudget: config.thinkingBudget,
|
||||
});
|
||||
},
|
||||
createBrowserSession: createGoogleRealtimeBrowserSession,
|
||||
};
|
||||
}
|
||||
20
extensions/google/runtime-api.ts
Normal file
20
extensions/google/runtime-api.ts
Normal file
@@ -0,0 +1,20 @@
|
||||
// Google API module exposes the plugin public contract.
|
||||
export {
|
||||
DEFAULT_GOOGLE_API_BASE_URL,
|
||||
createGoogleThinkingPayloadWrapper,
|
||||
createGoogleThinkingStreamWrapper,
|
||||
isGoogleGemini3FlashModel,
|
||||
isGoogleGemini3ProModel,
|
||||
isGoogleGemini3ThinkingLevelModel,
|
||||
isGoogleThinkingRequiredModel,
|
||||
normalizeGoogleApiBaseUrl,
|
||||
normalizeGoogleModelId,
|
||||
parseGeminiAuth,
|
||||
buildGoogleGenerativeAiParams,
|
||||
createGoogleGenerativeAiTransportStreamFn,
|
||||
resolveGoogleGemini3ThinkingLevel,
|
||||
resolveGoogleGenerativeAiHttpRequestConfig,
|
||||
sanitizeGoogleThinkingPayload,
|
||||
stripInvalidGoogleThinkingBudget,
|
||||
} from "./api.js";
|
||||
export type { GoogleThinkingInputLevel, GoogleThinkingLevel } from "./api.js";
|
||||
564
extensions/google/setup-api.test.ts
Normal file
564
extensions/google/setup-api.test.ts
Normal file
@@ -0,0 +1,564 @@
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import type { CliBackendPlugin } from "openclaw/plugin-sdk/cli-backend";
|
||||
import type { ProviderPlugin } from "openclaw/plugin-sdk/provider-model-shared";
|
||||
import { resolvePreferredOpenClawTmpDir } from "openclaw/plugin-sdk/temp-path";
|
||||
import { withTempDir } from "openclaw/plugin-sdk/test-env";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { buildGoogleGeminiCliBackend } from "./cli-backend.js";
|
||||
import setupEntry from "./setup-api.js";
|
||||
|
||||
type GeminiPrepareContext = Parameters<
|
||||
NonNullable<ReturnType<typeof buildGoogleGeminiCliBackend>["prepareExecution"]>
|
||||
>[0] & {
|
||||
env?: Record<string, string>;
|
||||
authCredential?: {
|
||||
type: "api_key" | "oauth" | "token";
|
||||
provider: string;
|
||||
access?: string;
|
||||
refresh?: string;
|
||||
expires?: number;
|
||||
idToken?: string;
|
||||
projectId?: string;
|
||||
key?: string;
|
||||
email?: string;
|
||||
};
|
||||
};
|
||||
type GeminiPreparedExecution = Awaited<
|
||||
ReturnType<NonNullable<ReturnType<typeof buildGoogleGeminiCliBackend>["prepareExecution"]>>
|
||||
>;
|
||||
|
||||
async function stageGeminiPreparedExecution(
|
||||
prepared: GeminiPreparedExecution | null | undefined,
|
||||
): Promise<void> {
|
||||
await prepared?.beforeExecution?.();
|
||||
}
|
||||
|
||||
function buildGeminiOAuthPrepareContext(workspaceDir: string): GeminiPrepareContext {
|
||||
const agentDir = path.join(workspaceDir, "agent");
|
||||
return {
|
||||
workspaceDir,
|
||||
agentDir,
|
||||
provider: "google-gemini-cli",
|
||||
modelId: "gemini-3.1-pro-preview",
|
||||
authProfileId: "google-gemini-cli:user@example.test",
|
||||
// Private bundled-runtime bridge, not public Plugin SDK surface.
|
||||
authCredential: {
|
||||
type: "oauth",
|
||||
provider: "google-gemini-cli",
|
||||
access: "access-token",
|
||||
refresh: "refresh-token",
|
||||
expires: 1_800_000_000_000,
|
||||
idToken: "id-token",
|
||||
projectId: "profile-project",
|
||||
email: "user@example.test",
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function buildGeminiApiKeyPrepareContext(workspaceDir: string): GeminiPrepareContext {
|
||||
const agentDir = path.join(workspaceDir, "agent");
|
||||
return {
|
||||
workspaceDir,
|
||||
agentDir,
|
||||
provider: "google-gemini-cli",
|
||||
modelId: "gemini-3.1-flash-lite",
|
||||
authProfileId: "google:api-key",
|
||||
// Private bundled-runtime bridge, not public Plugin SDK surface.
|
||||
authCredential: {
|
||||
type: "api_key",
|
||||
provider: "google",
|
||||
key: "gemini-api-key",
|
||||
email: "user@example.test",
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function restoreEnv(name: string, value: string | undefined): void {
|
||||
if (value === undefined) {
|
||||
delete process.env[name];
|
||||
return;
|
||||
}
|
||||
process.env[name] = value;
|
||||
}
|
||||
|
||||
describe("google setup entry", () => {
|
||||
it("registers setup runtime providers declared by the manifest", () => {
|
||||
const providerIds: string[] = [];
|
||||
const cliBackendIds: string[] = [];
|
||||
|
||||
setupEntry.register({
|
||||
registerProvider(provider: ProviderPlugin) {
|
||||
providerIds.push(provider.id);
|
||||
},
|
||||
registerCliBackend(backend: CliBackendPlugin) {
|
||||
cliBackendIds.push(backend.id);
|
||||
},
|
||||
} as never);
|
||||
|
||||
expect(providerIds).toEqual(["google-vertex"]);
|
||||
expect(cliBackendIds).toEqual(["google-gemini-cli"]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("google gemini cli backend config", () => {
|
||||
it("keeps legacy json output overrides on the json parser", () => {
|
||||
const backend = buildGoogleGeminiCliBackend();
|
||||
const normalized = backend.normalizeConfig?.({
|
||||
...backend.config,
|
||||
args: ["--skip-trust", "--output-format", "json", "--prompt", "{prompt}"],
|
||||
resumeArgs: [
|
||||
"--skip-trust",
|
||||
"--resume",
|
||||
"{sessionId}",
|
||||
"--output-format=json",
|
||||
"--prompt",
|
||||
"{prompt}",
|
||||
],
|
||||
});
|
||||
|
||||
expect(normalized?.output).toBe("json");
|
||||
expect(normalized?.resumeOutput).toBe("json");
|
||||
expect(normalized?.jsonlDialect).toBeUndefined();
|
||||
});
|
||||
|
||||
it("keeps short stream-json output overrides on the jsonl parser", () => {
|
||||
const backend = buildGoogleGeminiCliBackend();
|
||||
const normalized = backend.normalizeConfig?.({
|
||||
...backend.config,
|
||||
args: ["--skip-trust", "-o", "stream-json", "--prompt", "{prompt}"],
|
||||
resumeArgs: [
|
||||
"--skip-trust",
|
||||
"--resume",
|
||||
"{sessionId}",
|
||||
"-o=stream-json",
|
||||
"--prompt",
|
||||
"{prompt}",
|
||||
],
|
||||
});
|
||||
|
||||
expect(normalized?.output).toBe("jsonl");
|
||||
expect(normalized?.resumeOutput).toBe("jsonl");
|
||||
expect(normalized?.jsonlDialect).toBe("gemini-stream-json");
|
||||
});
|
||||
});
|
||||
|
||||
describe("google gemini cli backend auth bridge", () => {
|
||||
it("materializes selected OpenClaw OAuth credentials into a persistent profile-scoped Gemini CLI home", async () => {
|
||||
const backend = buildGoogleGeminiCliBackend();
|
||||
const workspaceDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-test-workspace-"));
|
||||
let home: string | undefined;
|
||||
const cleanups: Array<() => Promise<void>> = [];
|
||||
|
||||
try {
|
||||
const context = buildGeminiOAuthPrepareContext(workspaceDir);
|
||||
const inheritedSettingsPath = path.join(workspaceDir, "generated-mcp-settings.json");
|
||||
await fs.writeFile(
|
||||
inheritedSettingsPath,
|
||||
`${JSON.stringify({
|
||||
security: {
|
||||
auth: {
|
||||
selectedType: "vertex-ai",
|
||||
enforcedType: "oauth-personal",
|
||||
useExternal: true,
|
||||
},
|
||||
},
|
||||
mcp: { allowed: ["openclaw"] },
|
||||
mcpServers: { openclaw: { url: "http://127.0.0.1:23119/mcp" } },
|
||||
})}\n`,
|
||||
"utf8",
|
||||
);
|
||||
context.env = { GEMINI_CLI_SYSTEM_SETTINGS_PATH: inheritedSettingsPath };
|
||||
const prepared = await backend.prepareExecution?.(context);
|
||||
if (prepared?.cleanup) {
|
||||
cleanups.push(prepared.cleanup);
|
||||
}
|
||||
await stageGeminiPreparedExecution(prepared);
|
||||
|
||||
home = prepared?.env?.GEMINI_CLI_HOME;
|
||||
const systemSettingsPath = prepared?.env?.GEMINI_CLI_SYSTEM_SETTINGS_PATH;
|
||||
expect(home).toBeTruthy();
|
||||
expect(systemSettingsPath).toBeTruthy();
|
||||
expect(systemSettingsPath).not.toBe(inheritedSettingsPath);
|
||||
expect(path.dirname(systemSettingsPath ?? "")).not.toBe(home);
|
||||
expect(
|
||||
path.relative(resolvePreferredOpenClawTmpDir(), path.dirname(systemSettingsPath ?? "")),
|
||||
).toMatch(/^openclaw-gemini-cli-/);
|
||||
expect(prepared?.env?.GEMINI_FORCE_FILE_STORAGE).toBe("true");
|
||||
expect(prepared?.env?.GOOGLE_CLOUD_PROJECT).toBe("profile-project");
|
||||
expect(prepared?.env?.GOOGLE_CLOUD_PROJECT_ID).toBe("profile-project");
|
||||
expect(prepared?.env?.GOOGLE_CLOUD_QUOTA_PROJECT).toBe("profile-project");
|
||||
if (!context.agentDir) {
|
||||
throw new Error("expected Gemini test context to include an agent directory");
|
||||
}
|
||||
expect(home).toContain(path.join(context.agentDir, "google-gemini-cli-home"));
|
||||
expect(home).not.toContain("user@example.test");
|
||||
|
||||
const raw = await fs.readFile(path.join(home ?? "", ".gemini", "oauth_creds.json"), "utf8");
|
||||
expect(JSON.parse(raw)).toEqual({
|
||||
access_token: "access-token",
|
||||
refresh_token: "refresh-token",
|
||||
id_token: "id-token",
|
||||
expiry_date: 1_800_000_000_000,
|
||||
token_type: "Bearer",
|
||||
});
|
||||
const nestedSettingsRaw = await fs.readFile(
|
||||
path.join(home ?? "", ".gemini", "settings.json"),
|
||||
"utf8",
|
||||
);
|
||||
const rootSettingsRaw = await fs.readFile(path.join(home ?? "", "settings.json"), "utf8");
|
||||
expect(JSON.parse(nestedSettingsRaw)).toEqual({
|
||||
security: { auth: { selectedType: "oauth-personal" } },
|
||||
});
|
||||
expect(JSON.parse(rootSettingsRaw)).toEqual(JSON.parse(nestedSettingsRaw));
|
||||
const systemSettingsRaw = await fs.readFile(systemSettingsPath ?? "", "utf8");
|
||||
expect(JSON.parse(systemSettingsRaw)).toEqual({
|
||||
security: {
|
||||
auth: {
|
||||
selectedType: "oauth-personal",
|
||||
enforcedType: "oauth-personal",
|
||||
useExternal: true,
|
||||
},
|
||||
},
|
||||
mcp: { allowed: ["openclaw"] },
|
||||
mcpServers: { openclaw: { url: "http://127.0.0.1:23119/mcp" } },
|
||||
});
|
||||
|
||||
const sessionMarker = path.join(home ?? "", ".gemini", "session-state.json");
|
||||
await fs.writeFile(sessionMarker, '{"keep":true}\n', "utf8");
|
||||
const cachedCredentialsPath = path.join(home ?? "", ".gemini", "gemini-credentials.json");
|
||||
await fs.writeFile(cachedCredentialsPath, "stale-cache", "utf8");
|
||||
|
||||
const preparedAgain = await backend.prepareExecution?.(context);
|
||||
if (preparedAgain?.cleanup) {
|
||||
cleanups.push(preparedAgain.cleanup);
|
||||
}
|
||||
await stageGeminiPreparedExecution(preparedAgain);
|
||||
expect(preparedAgain?.env?.GEMINI_CLI_HOME).toBe(home);
|
||||
await expect(fs.access(sessionMarker)).resolves.toBeUndefined();
|
||||
await expect(fs.access(cachedCredentialsPath)).rejects.toThrow();
|
||||
} finally {
|
||||
for (const cleanup of cleanups.toReversed()) {
|
||||
await cleanup();
|
||||
}
|
||||
await fs.rm(workspaceDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("stages Gemini CLI JSON through same-directory atomic renames", async () => {
|
||||
await withTempDir("openclaw-test-workspace-", async (workspaceDir) => {
|
||||
const backend = buildGoogleGeminiCliBackend();
|
||||
const realRename = fs.rename.bind(fs);
|
||||
const renameCalls: Array<{ from: string; to: string }> = [];
|
||||
const renameSpy = vi
|
||||
.spyOn(fs, "rename")
|
||||
.mockImplementation(async (...args: Parameters<typeof fs.rename>) => {
|
||||
renameCalls.push({ from: String(args[0]), to: String(args[1]) });
|
||||
await realRename(...args);
|
||||
});
|
||||
let prepared: GeminiPreparedExecution | null | undefined;
|
||||
|
||||
try {
|
||||
prepared = await backend.prepareExecution?.(buildGeminiOAuthPrepareContext(workspaceDir));
|
||||
await stageGeminiPreparedExecution(prepared);
|
||||
|
||||
const home = prepared?.env?.GEMINI_CLI_HOME;
|
||||
const systemSettingsPath = prepared?.env?.GEMINI_CLI_SYSTEM_SETTINGS_PATH;
|
||||
if (!home || !systemSettingsPath) {
|
||||
throw new Error("expected Gemini CLI staging paths");
|
||||
}
|
||||
const expectedTargets = [
|
||||
path.join(home, ".gemini", "settings.json"),
|
||||
path.join(home, "settings.json"),
|
||||
systemSettingsPath,
|
||||
path.join(home, ".gemini", "oauth_creds.json"),
|
||||
];
|
||||
expect(renameCalls.map((call) => call.to).toSorted()).toEqual(expectedTargets.toSorted());
|
||||
for (const call of renameCalls) {
|
||||
expect(path.dirname(call.from)).toBe(path.dirname(call.to));
|
||||
expect(path.basename(call.from).startsWith(`.${path.basename(call.to)}.`)).toBe(true);
|
||||
expect(path.basename(call.from).endsWith(".tmp")).toBe(true);
|
||||
}
|
||||
const oauthStat = await fs.stat(path.join(home, ".gemini", "oauth_creds.json"));
|
||||
expect(oauthStat.mode & 0o777).toBe(0o600);
|
||||
} finally {
|
||||
renameSpy.mockRestore();
|
||||
await prepared?.cleanup?.();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
it("prepares selected canonical Google API-key credentials and removes stale OAuth state for that profile home", async () => {
|
||||
const backend = buildGoogleGeminiCliBackend();
|
||||
const workspaceDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-test-workspace-"));
|
||||
let home: string | undefined;
|
||||
const cleanups: Array<() => Promise<void>> = [];
|
||||
|
||||
try {
|
||||
const context = buildGeminiApiKeyPrepareContext(workspaceDir);
|
||||
const firstPrepared = await backend.prepareExecution?.(context);
|
||||
if (firstPrepared?.cleanup) {
|
||||
cleanups.push(firstPrepared.cleanup);
|
||||
}
|
||||
await stageGeminiPreparedExecution(firstPrepared);
|
||||
home = firstPrepared?.env?.GEMINI_CLI_HOME;
|
||||
expect(home).toBeTruthy();
|
||||
await fs.writeFile(path.join(home ?? "", ".gemini", "oauth_creds.json"), "{}\n", "utf8");
|
||||
await fs.writeFile(
|
||||
path.join(home ?? "", ".gemini", "gemini-credentials.json"),
|
||||
"stale-cache",
|
||||
"utf8",
|
||||
);
|
||||
|
||||
const prepared = await backend.prepareExecution?.(context);
|
||||
if (prepared?.cleanup) {
|
||||
cleanups.push(prepared.cleanup);
|
||||
}
|
||||
await stageGeminiPreparedExecution(prepared);
|
||||
|
||||
home = prepared?.env?.GEMINI_CLI_HOME;
|
||||
expect(home).toBeTruthy();
|
||||
expect(prepared?.env?.GEMINI_API_KEY).toBe("gemini-api-key");
|
||||
expect(prepared?.env?.GEMINI_FORCE_FILE_STORAGE).toBe("true");
|
||||
expect(prepared?.clearEnv).toContain("GEMINI_API_KEY");
|
||||
expect(prepared?.clearEnv).toContain("GOOGLE_GENAI_USE_GCA");
|
||||
expect(prepared?.clearEnv).toContain("GOOGLE_GENAI_USE_VERTEXAI");
|
||||
expect(prepared?.clearEnv).toContain("GOOGLE_GEMINI_BASE_URL");
|
||||
|
||||
const settingsRaw = await fs.readFile(
|
||||
path.join(home ?? "", ".gemini", "settings.json"),
|
||||
"utf8",
|
||||
);
|
||||
expect(JSON.parse(settingsRaw)).toEqual({
|
||||
security: { auth: { selectedType: "gemini-api-key" } },
|
||||
});
|
||||
await expect(
|
||||
fs.access(path.join(home ?? "", ".gemini", "oauth_creds.json")),
|
||||
).rejects.toThrow();
|
||||
await expect(
|
||||
fs.access(path.join(home ?? "", ".gemini", "gemini-credentials.json")),
|
||||
).rejects.toThrow();
|
||||
} finally {
|
||||
for (const cleanup of cleanups.toReversed()) {
|
||||
await cleanup();
|
||||
}
|
||||
await fs.rm(workspaceDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects inherited Gemini system settings that enforce a different auth type", async () => {
|
||||
const backend = buildGoogleGeminiCliBackend();
|
||||
const workspaceDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-test-workspace-"));
|
||||
|
||||
try {
|
||||
const inheritedSettingsPath = path.join(workspaceDir, "generated-mcp-settings.json");
|
||||
await fs.writeFile(
|
||||
inheritedSettingsPath,
|
||||
`${JSON.stringify({
|
||||
security: { auth: { enforcedType: "gemini-api-key" } },
|
||||
})}\n`,
|
||||
"utf8",
|
||||
);
|
||||
const context = buildGeminiOAuthPrepareContext(workspaceDir);
|
||||
context.env = { GEMINI_CLI_SYSTEM_SETTINGS_PATH: inheritedSettingsPath };
|
||||
|
||||
await expect(backend.prepareExecution?.(context)).rejects.toThrow(/enforce gemini-api-key/);
|
||||
} finally {
|
||||
await fs.rm(workspaceDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("inherits process Gemini system settings when no generated settings path is present", async () => {
|
||||
const backend = buildGoogleGeminiCliBackend();
|
||||
const workspaceDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-test-workspace-"));
|
||||
const originalSystemSettingsPath = process.env.GEMINI_CLI_SYSTEM_SETTINGS_PATH;
|
||||
let prepared:
|
||||
| Awaited<ReturnType<NonNullable<typeof backend.prepareExecution>>>
|
||||
| null
|
||||
| undefined;
|
||||
|
||||
try {
|
||||
const inheritedSettingsPath = path.join(workspaceDir, "ambient-system-settings.json");
|
||||
await fs.writeFile(
|
||||
inheritedSettingsPath,
|
||||
`${JSON.stringify({
|
||||
security: {
|
||||
auth: {
|
||||
selectedType: "oauth-code-assist",
|
||||
enforcedType: "oauth-personal",
|
||||
},
|
||||
folderTrust: { enabled: true },
|
||||
},
|
||||
})}\n`,
|
||||
"utf8",
|
||||
);
|
||||
process.env.GEMINI_CLI_SYSTEM_SETTINGS_PATH = inheritedSettingsPath;
|
||||
|
||||
prepared = await backend.prepareExecution?.(buildGeminiOAuthPrepareContext(workspaceDir));
|
||||
await stageGeminiPreparedExecution(prepared);
|
||||
|
||||
const systemSettingsRaw = await fs.readFile(
|
||||
prepared?.env?.GEMINI_CLI_SYSTEM_SETTINGS_PATH ?? "",
|
||||
"utf8",
|
||||
);
|
||||
expect(JSON.parse(systemSettingsRaw)).toEqual({
|
||||
security: {
|
||||
auth: {
|
||||
selectedType: "oauth-personal",
|
||||
enforcedType: "oauth-personal",
|
||||
},
|
||||
folderTrust: { enabled: true },
|
||||
},
|
||||
});
|
||||
} finally {
|
||||
restoreEnv("GEMINI_CLI_SYSTEM_SETTINGS_PATH", originalSystemSettingsPath);
|
||||
await prepared?.cleanup?.();
|
||||
await fs.rm(workspaceDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects Vercel AI Gateway profiles for the Gemini CLI backend", async () => {
|
||||
const backend = buildGoogleGeminiCliBackend();
|
||||
const workspaceDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-test-workspace-"));
|
||||
|
||||
try {
|
||||
await expect(
|
||||
backend.prepareExecution?.({
|
||||
workspaceDir,
|
||||
agentDir: path.join(workspaceDir, "agent"),
|
||||
provider: "google-gemini-cli",
|
||||
modelId: "gemini-3.1-flash-lite",
|
||||
authProfileId: "vercel-ai-gateway:default",
|
||||
authCredential: {
|
||||
type: "api_key",
|
||||
provider: "vercel-ai-gateway",
|
||||
key: "vercel-key",
|
||||
},
|
||||
} as never),
|
||||
).rejects.toThrow(/vercel-ai-gateway auth profile/);
|
||||
} finally {
|
||||
await fs.rm(workspaceDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects selected Gemini token profiles before the CLI can use ambient auth", async () => {
|
||||
const backend = buildGoogleGeminiCliBackend();
|
||||
const workspaceDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-test-workspace-"));
|
||||
|
||||
try {
|
||||
await expect(
|
||||
backend.prepareExecution?.({
|
||||
workspaceDir,
|
||||
agentDir: path.join(workspaceDir, "agent"),
|
||||
provider: "google-gemini-cli",
|
||||
modelId: "gemini-3.1-flash-lite",
|
||||
authProfileId: "google-gemini-cli:token",
|
||||
authCredential: {
|
||||
type: "token",
|
||||
provider: "google-gemini-cli",
|
||||
token: "bearer-token",
|
||||
},
|
||||
} as never),
|
||||
).rejects.toThrow(/OAuth or API-key auth profiles/);
|
||||
} finally {
|
||||
await fs.rm(workspaceDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects selected Gemini profiles with no material before the CLI can use ambient auth", async () => {
|
||||
const backend = buildGoogleGeminiCliBackend();
|
||||
const workspaceDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-test-workspace-"));
|
||||
|
||||
try {
|
||||
await expect(
|
||||
backend.prepareExecution?.({
|
||||
workspaceDir,
|
||||
agentDir: path.join(workspaceDir, "agent"),
|
||||
provider: "google-gemini-cli",
|
||||
modelId: "gemini-3.1-flash-lite",
|
||||
authProfileId: "google-gemini-cli:missing",
|
||||
} as never),
|
||||
).rejects.toThrow(/no credential material/);
|
||||
} finally {
|
||||
await fs.rm(workspaceDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("clears inherited Gemini auth credentials when staging selected OAuth credentials", async () => {
|
||||
const backend = buildGoogleGeminiCliBackend();
|
||||
const workspaceDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-test-workspace-"));
|
||||
const originalUseGca = process.env.GOOGLE_GENAI_USE_GCA;
|
||||
const originalCloudAccessToken = process.env.GOOGLE_CLOUD_ACCESS_TOKEN;
|
||||
const originalGoogleApplicationCredentials = process.env.GOOGLE_APPLICATION_CREDENTIALS;
|
||||
const originalForceEncryptedFileStorage = process.env.GEMINI_FORCE_ENCRYPTED_FILE_STORAGE;
|
||||
const originalGeminiApiKey = process.env.GEMINI_API_KEY;
|
||||
const originalGoogleApiKey = process.env.GOOGLE_API_KEY;
|
||||
const originalQuotaProject = process.env.GOOGLE_CLOUD_QUOTA_PROJECT;
|
||||
let prepared:
|
||||
| Awaited<ReturnType<NonNullable<typeof backend.prepareExecution>>>
|
||||
| null
|
||||
| undefined;
|
||||
|
||||
process.env.GOOGLE_GENAI_USE_GCA = "true";
|
||||
process.env.GOOGLE_CLOUD_ACCESS_TOKEN = "ambient-cloud-token";
|
||||
process.env.GOOGLE_APPLICATION_CREDENTIALS = "/tmp/ambient-google-adc.json";
|
||||
process.env.GEMINI_FORCE_ENCRYPTED_FILE_STORAGE = "true";
|
||||
process.env.GEMINI_API_KEY = "ambient-gemini-key";
|
||||
process.env.GOOGLE_API_KEY = "ambient-google-key";
|
||||
process.env.GOOGLE_CLOUD_QUOTA_PROJECT = "ambient-project";
|
||||
|
||||
try {
|
||||
prepared = await backend.prepareExecution?.(buildGeminiOAuthPrepareContext(workspaceDir));
|
||||
|
||||
expect(prepared?.env?.GEMINI_CLI_HOME).toBeTruthy();
|
||||
expect(prepared?.clearEnv).toEqual([
|
||||
"GOOGLE_GENAI_USE_GCA",
|
||||
"GOOGLE_CLOUD_ACCESS_TOKEN",
|
||||
"GOOGLE_APPLICATION_CREDENTIALS",
|
||||
"GEMINI_FORCE_ENCRYPTED_FILE_STORAGE",
|
||||
"GEMINI_FORCE_FILE_STORAGE",
|
||||
"GOOGLE_GENAI_USE_VERTEXAI",
|
||||
"GOOGLE_API_KEY",
|
||||
"GOOGLE_CLOUD_PROJECT",
|
||||
"GOOGLE_CLOUD_PROJECT_ID",
|
||||
"GOOGLE_CLOUD_QUOTA_PROJECT",
|
||||
"GOOGLE_CLOUD_LOCATION",
|
||||
"GOOGLE_GEMINI_BASE_URL",
|
||||
"GEMINI_CLI_CUSTOM_HEADERS",
|
||||
"GEMINI_API_KEY_AUTH_MECHANISM",
|
||||
"GEMINI_API_KEY",
|
||||
"GEMINI_CLI_SYSTEM_SETTINGS_PATH",
|
||||
]);
|
||||
} finally {
|
||||
restoreEnv("GOOGLE_GENAI_USE_GCA", originalUseGca);
|
||||
restoreEnv("GOOGLE_CLOUD_ACCESS_TOKEN", originalCloudAccessToken);
|
||||
restoreEnv("GOOGLE_APPLICATION_CREDENTIALS", originalGoogleApplicationCredentials);
|
||||
restoreEnv("GEMINI_FORCE_ENCRYPTED_FILE_STORAGE", originalForceEncryptedFileStorage);
|
||||
restoreEnv("GEMINI_API_KEY", originalGeminiApiKey);
|
||||
restoreEnv("GOOGLE_API_KEY", originalGoogleApiKey);
|
||||
restoreEnv("GOOGLE_CLOUD_QUOTA_PROJECT", originalQuotaProject);
|
||||
await prepared?.cleanup?.();
|
||||
await fs.rm(workspaceDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("requires an agent directory for profile-owned Gemini CLI state", async () => {
|
||||
const backend = buildGoogleGeminiCliBackend();
|
||||
const workspaceDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-test-workspace-"));
|
||||
|
||||
try {
|
||||
const { agentDir: _agentDir, ...context } = buildGeminiOAuthPrepareContext(workspaceDir);
|
||||
await expect(backend.prepareExecution?.(context)).rejects.toThrow(/agent directory/);
|
||||
} finally {
|
||||
await fs.rm(workspaceDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("uses profile-only auth epochs for the private Gemini CLI bridge", () => {
|
||||
const backend = buildGoogleGeminiCliBackend();
|
||||
|
||||
expect(backend.authEpochMode).toBe("profile-only");
|
||||
expect(backend.prepareExecution).toBeTypeOf("function");
|
||||
});
|
||||
});
|
||||
14
extensions/google/setup-api.ts
Normal file
14
extensions/google/setup-api.ts
Normal file
@@ -0,0 +1,14 @@
|
||||
// Google API module exposes the plugin public contract.
|
||||
import { definePluginEntry } from "openclaw/plugin-sdk/plugin-entry";
|
||||
import { buildGoogleGeminiCliBackend } from "./cli-backend.js";
|
||||
import { createGoogleVertexProvider } from "./provider-contract-api.js";
|
||||
|
||||
export default definePluginEntry({
|
||||
id: "google",
|
||||
name: "Google Setup",
|
||||
description: "Lightweight Google setup hooks",
|
||||
register(api) {
|
||||
api.registerProvider(createGoogleVertexProvider());
|
||||
api.registerCliBackend(buildGoogleGeminiCliBackend());
|
||||
},
|
||||
});
|
||||
738
extensions/google/speech-provider.test.ts
Normal file
738
extensions/google/speech-provider.test.ts
Normal file
@@ -0,0 +1,738 @@
|
||||
// Google tests cover speech provider plugin behavior.
|
||||
import {
|
||||
getProviderHttpMocks,
|
||||
installProviderHttpMockCleanup,
|
||||
} from "openclaw/plugin-sdk/provider-http-test-mocks";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
|
||||
const transcodeAudioBufferToOpusMock = vi.hoisted(() => vi.fn());
|
||||
|
||||
vi.mock("openclaw/plugin-sdk/media-runtime", () => ({
|
||||
transcodeAudioBufferToOpus: transcodeAudioBufferToOpusMock,
|
||||
}));
|
||||
|
||||
const {
|
||||
assertOkOrThrowProviderErrorMock,
|
||||
postJsonRequestMock,
|
||||
resolveProviderHttpRequestConfigMock,
|
||||
} = getProviderHttpMocks();
|
||||
|
||||
let buildGoogleSpeechProvider: typeof import("./speech-provider.js").buildGoogleSpeechProvider;
|
||||
let testing: typeof import("./speech-provider.js").testing;
|
||||
|
||||
const GOOGLE_TTS_JSON_CAP_BYTES = 16 * 1024 * 1024;
|
||||
|
||||
beforeAll(async () => {
|
||||
({ buildGoogleSpeechProvider, testing } = await import("./speech-provider.js"));
|
||||
});
|
||||
|
||||
installProviderHttpMockCleanup();
|
||||
|
||||
function googleTtsResponse(pcm = Buffer.from([1, 0, 2, 0])) {
|
||||
return {
|
||||
ok: true,
|
||||
json: async () => ({
|
||||
candidates: [
|
||||
{
|
||||
content: {
|
||||
parts: [
|
||||
{
|
||||
inlineData: {
|
||||
mimeType: "audio/L16;codec=pcm;rate=24000",
|
||||
data: pcm.toString("base64"),
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
function installGoogleTtsRequestMock(pcm = Buffer.from([1, 0, 2, 0])) {
|
||||
postJsonRequestMock.mockResolvedValue({
|
||||
response: googleTtsResponse(pcm),
|
||||
release: vi.fn(async () => {}),
|
||||
});
|
||||
return postJsonRequestMock;
|
||||
}
|
||||
|
||||
function oversizedGoogleTtsJsonResponse(onCancel: () => void): Response {
|
||||
const response = new Response(
|
||||
new ReadableStream<Uint8Array>({
|
||||
start(controller) {
|
||||
controller.enqueue(new Uint8Array(GOOGLE_TTS_JSON_CAP_BYTES + 1));
|
||||
},
|
||||
cancel() {
|
||||
onCancel();
|
||||
},
|
||||
}),
|
||||
{ headers: { "content-type": "application/json" }, status: 200 },
|
||||
);
|
||||
Object.defineProperty(response, "json", {
|
||||
value: async () => {
|
||||
throw new Error("unbounded json reader was used");
|
||||
},
|
||||
});
|
||||
return response;
|
||||
}
|
||||
|
||||
function expectRecordFields(value: unknown, expected: Record<string, unknown>) {
|
||||
if (!value || typeof value !== "object") {
|
||||
throw new Error("Expected record");
|
||||
}
|
||||
const actual = value as Record<string, unknown>;
|
||||
for (const [key, expectedValue] of Object.entries(expected)) {
|
||||
expect(actual[key]).toEqual(expectedValue);
|
||||
}
|
||||
return actual;
|
||||
}
|
||||
|
||||
function requireFirstMockArg(mock: ReturnType<typeof vi.fn>, label: string): unknown {
|
||||
const [call] = mock.mock.calls;
|
||||
if (!call) {
|
||||
throw new Error(`Expected ${label}`);
|
||||
}
|
||||
return call[0];
|
||||
}
|
||||
|
||||
function requireFirstRecordArg(
|
||||
mock: ReturnType<typeof vi.fn>,
|
||||
label: string,
|
||||
): Record<string, unknown> {
|
||||
const value = requireFirstMockArg(mock, label);
|
||||
if (!value || typeof value !== "object" || Array.isArray(value)) {
|
||||
throw new Error(`Expected ${label}`);
|
||||
}
|
||||
return value as Record<string, unknown>;
|
||||
}
|
||||
|
||||
describe("Google speech provider", () => {
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals();
|
||||
vi.unstubAllEnvs();
|
||||
transcodeAudioBufferToOpusMock.mockReset();
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
vi.doUnmock("openclaw/plugin-sdk/media-runtime");
|
||||
vi.resetModules();
|
||||
});
|
||||
|
||||
it("synthesizes Gemini PCM as WAV and preserves audio tags in the request text", async () => {
|
||||
const requestMock = installGoogleTtsRequestMock();
|
||||
const provider = buildGoogleSpeechProvider();
|
||||
|
||||
const result = await provider.synthesize({
|
||||
text: "[whispers] The door is open.",
|
||||
cfg: {},
|
||||
providerConfig: {
|
||||
apiKey: "google-test-key",
|
||||
model: "google/gemini-3.1-flash-tts",
|
||||
voiceName: "Puck",
|
||||
},
|
||||
target: "audio-file",
|
||||
timeoutMs: 12_345,
|
||||
});
|
||||
|
||||
const request = expectRecordFields(requireFirstRecordArg(requestMock, "Google TTS request"), {
|
||||
url: "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-flash-tts-preview:generateContent",
|
||||
body: {
|
||||
contents: [
|
||||
{
|
||||
role: "user",
|
||||
parts: [{ text: "[whispers] The door is open." }],
|
||||
},
|
||||
],
|
||||
generationConfig: {
|
||||
responseModalities: ["AUDIO"],
|
||||
speechConfig: {
|
||||
voiceConfig: {
|
||||
prebuiltVoiceConfig: {
|
||||
voiceName: "Puck",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
fetchFn: fetch,
|
||||
pinDns: false,
|
||||
timeoutMs: 12_345,
|
||||
}) as { headers?: HeadersInit };
|
||||
expect(new Headers(request.headers).get("x-goog-api-key")).toBe("google-test-key");
|
||||
expect(result.outputFormat).toBe("wav");
|
||||
expect(result.fileExtension).toBe(".wav");
|
||||
expect(result.voiceCompatible).toBe(false);
|
||||
expect(result.audioBuffer.subarray(0, 4).toString("ascii")).toBe("RIFF");
|
||||
expect(result.audioBuffer.subarray(8, 12).toString("ascii")).toBe("WAVE");
|
||||
expect(result.audioBuffer.readUInt32LE(24)).toBe(testing.GOOGLE_TTS_SAMPLE_RATE);
|
||||
expect(result.audioBuffer.subarray(44)).toEqual(Buffer.from([1, 0, 2, 0]));
|
||||
expect(transcodeAudioBufferToOpusMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("bounds oversized Gemini TTS success JSON responses and cancels the stream", async () => {
|
||||
let cancelCount = 0;
|
||||
const release = vi.fn(async () => {});
|
||||
postJsonRequestMock
|
||||
.mockResolvedValueOnce({
|
||||
response: oversizedGoogleTtsJsonResponse(() => {
|
||||
cancelCount += 1;
|
||||
}),
|
||||
release,
|
||||
})
|
||||
.mockResolvedValueOnce({
|
||||
response: oversizedGoogleTtsJsonResponse(() => {
|
||||
cancelCount += 1;
|
||||
}),
|
||||
release,
|
||||
});
|
||||
const provider = buildGoogleSpeechProvider();
|
||||
|
||||
await expect(
|
||||
provider.synthesize({
|
||||
text: "oversized tts response",
|
||||
cfg: {},
|
||||
providerConfig: {
|
||||
apiKey: "google-test-key",
|
||||
},
|
||||
target: "audio-file",
|
||||
timeoutMs: 12_000,
|
||||
}),
|
||||
).rejects.toThrow("Google TTS response: JSON response exceeds 16777216 bytes");
|
||||
expect(cancelCount).toBe(2);
|
||||
expect(release).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("transcodes Gemini PCM to Opus for voice-note targets", async () => {
|
||||
installGoogleTtsRequestMock(Buffer.from([5, 0, 6, 0]));
|
||||
transcodeAudioBufferToOpusMock.mockResolvedValueOnce(Buffer.from("google-opus"));
|
||||
const provider = buildGoogleSpeechProvider();
|
||||
|
||||
const result = await provider.synthesize({
|
||||
text: "Send this as a voice note.",
|
||||
cfg: {},
|
||||
providerConfig: {
|
||||
apiKey: "google-test-key",
|
||||
},
|
||||
target: "voice-note",
|
||||
timeoutMs: 12_000,
|
||||
});
|
||||
|
||||
expect(result).toEqual({
|
||||
audioBuffer: Buffer.from("google-opus"),
|
||||
outputFormat: "opus",
|
||||
fileExtension: ".opus",
|
||||
voiceCompatible: true,
|
||||
});
|
||||
const transcodeArg = expectRecordFields(
|
||||
requireFirstRecordArg(transcodeAudioBufferToOpusMock, "Google TTS transcode request"),
|
||||
{
|
||||
inputExtension: "wav",
|
||||
tempPrefix: "tts-google-",
|
||||
timeoutMs: 12_000,
|
||||
},
|
||||
);
|
||||
expect(Buffer.isBuffer(transcodeArg.audioBuffer)).toBe(true);
|
||||
const audioBuffer = transcodeArg.audioBuffer as Buffer;
|
||||
expect(audioBuffer.subarray(0, 4).toString("ascii")).toBe("RIFF");
|
||||
expect(audioBuffer.subarray(8, 12).toString("ascii")).toBe("WAVE");
|
||||
});
|
||||
|
||||
it("advertises all documented Gemini TTS-capable models", () => {
|
||||
const provider = buildGoogleSpeechProvider();
|
||||
|
||||
expect(provider.models).toEqual(testing.GOOGLE_TTS_MODELS);
|
||||
});
|
||||
|
||||
it("renders deterministic audio-profile-v1 prompts without generating tags", async () => {
|
||||
const provider = buildGoogleSpeechProvider();
|
||||
|
||||
const prepared = await provider.prepareSynthesis?.({
|
||||
text: "[whispers] The door is open.",
|
||||
cfg: {},
|
||||
providerConfig: {
|
||||
promptTemplate: "audio-profile-v1",
|
||||
personaPrompt: "Keep a close-mic feel.",
|
||||
},
|
||||
persona: {
|
||||
id: "alfred",
|
||||
label: "Alfred",
|
||||
prompt: {
|
||||
profile: "A brilliant British butler.",
|
||||
scene: "A quiet late-night study.",
|
||||
sampleContext: "The speaker is answering a trusted operator.",
|
||||
style: "Refined and lightly amused.",
|
||||
accent: "British English.",
|
||||
pacing: "Measured.",
|
||||
constraints: ["Do not read configuration values aloud."],
|
||||
},
|
||||
},
|
||||
target: "audio-file",
|
||||
timeoutMs: 1_000,
|
||||
});
|
||||
|
||||
expect(prepared?.text).toBe(
|
||||
[
|
||||
"Synthesize speech from the TRANSCRIPT section only. Use the other sections only",
|
||||
"as performance direction. Do not read section titles, notes, labels, or",
|
||||
"configuration aloud.",
|
||||
"",
|
||||
"# AUDIO PROFILE: Alfred",
|
||||
"A brilliant British butler.",
|
||||
"",
|
||||
"## THE SCENE",
|
||||
"A quiet late-night study.",
|
||||
"",
|
||||
"### DIRECTOR'S NOTES",
|
||||
"Style: Refined and lightly amused.",
|
||||
"Accent: British English.",
|
||||
"Pacing: Measured.",
|
||||
"Constraints:",
|
||||
"- Do not read configuration values aloud.",
|
||||
"Provider notes:",
|
||||
"Keep a close-mic feel.",
|
||||
"",
|
||||
"### SAMPLE CONTEXT",
|
||||
"The speaker is answering a trusted operator.",
|
||||
"",
|
||||
"### TRANSCRIPT",
|
||||
"[whispers] The door is open.",
|
||||
].join("\n"),
|
||||
);
|
||||
});
|
||||
|
||||
it("does not wrap an OpenClaw audio-profile-v1 prompt twice", async () => {
|
||||
const provider = buildGoogleSpeechProvider();
|
||||
const text = [
|
||||
"Synthesize speech from the TRANSCRIPT section only. Use the other sections only",
|
||||
"as performance direction. Do not read section titles, notes, labels, or",
|
||||
"configuration aloud.",
|
||||
"",
|
||||
"# AUDIO PROFILE: Alfred",
|
||||
"A brilliant British butler.",
|
||||
"",
|
||||
"### TRANSCRIPT",
|
||||
"Hello.",
|
||||
].join("\n");
|
||||
|
||||
const prepared = await provider.prepareSynthesis?.({
|
||||
text,
|
||||
cfg: {},
|
||||
providerConfig: {
|
||||
promptTemplate: "audio-profile-v1",
|
||||
},
|
||||
persona: {
|
||||
id: "alfred",
|
||||
label: "Alfred",
|
||||
prompt: {
|
||||
profile: "A brilliant British butler.",
|
||||
},
|
||||
},
|
||||
target: "audio-file",
|
||||
timeoutMs: 1_000,
|
||||
});
|
||||
|
||||
expect(prepared).toBeUndefined();
|
||||
});
|
||||
|
||||
it("retries once when Gemini returns no audio payload", async () => {
|
||||
const pcm = Buffer.from([5, 0, 6, 0]);
|
||||
const requestSequence = vi
|
||||
.fn()
|
||||
.mockResolvedValueOnce({
|
||||
response: {
|
||||
ok: true,
|
||||
json: async () => ({ candidates: [{ content: { parts: [{ text: "not audio" }] } }] }),
|
||||
},
|
||||
release: vi.fn(async () => {}),
|
||||
})
|
||||
.mockResolvedValueOnce({
|
||||
response: googleTtsResponse(pcm),
|
||||
release: vi.fn(async () => {}),
|
||||
});
|
||||
postJsonRequestMock.mockImplementation(requestSequence);
|
||||
const provider = buildGoogleSpeechProvider();
|
||||
|
||||
const result = await provider.synthesize({
|
||||
text: "Retry this.",
|
||||
cfg: {},
|
||||
providerConfig: {
|
||||
apiKey: "google-test-key",
|
||||
},
|
||||
target: "audio-file",
|
||||
timeoutMs: 5_000,
|
||||
});
|
||||
|
||||
expect(requestSequence).toHaveBeenCalledTimes(2);
|
||||
expect(result.audioBuffer.subarray(44)).toEqual(pcm);
|
||||
});
|
||||
|
||||
it("retries once when Gemini TTS fetch aborts", async () => {
|
||||
const pcm = Buffer.from([7, 0, 8, 0]);
|
||||
const abortError = new Error("This operation was aborted");
|
||||
abortError.name = "AbortError";
|
||||
const requestSequence = vi
|
||||
.fn()
|
||||
.mockRejectedValueOnce(abortError)
|
||||
.mockResolvedValueOnce({
|
||||
response: googleTtsResponse(pcm),
|
||||
release: vi.fn(async () => {}),
|
||||
});
|
||||
postJsonRequestMock.mockImplementation(requestSequence);
|
||||
const provider = buildGoogleSpeechProvider();
|
||||
|
||||
const result = await provider.synthesize({
|
||||
text: "Retry aborted fetch.",
|
||||
cfg: {},
|
||||
providerConfig: {
|
||||
apiKey: "google-test-key",
|
||||
},
|
||||
target: "audio-file",
|
||||
timeoutMs: 5_000,
|
||||
});
|
||||
|
||||
expect(requestSequence).toHaveBeenCalledTimes(2);
|
||||
expect(result.audioBuffer.subarray(44)).toEqual(pcm);
|
||||
});
|
||||
|
||||
it("does not retry non-transient Gemini TTS request failures", async () => {
|
||||
const requestSequence = vi.fn().mockRejectedValueOnce(new Error("invalid request"));
|
||||
postJsonRequestMock.mockImplementation(requestSequence);
|
||||
const provider = buildGoogleSpeechProvider();
|
||||
|
||||
await expect(
|
||||
provider.synthesize({
|
||||
text: "Do not retry this.",
|
||||
cfg: {},
|
||||
providerConfig: {
|
||||
apiKey: "google-test-key",
|
||||
},
|
||||
target: "audio-file",
|
||||
timeoutMs: 5_000,
|
||||
}),
|
||||
).rejects.toThrow("invalid request");
|
||||
|
||||
expect(requestSequence).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("falls back to GEMINI_API_KEY and configured Google API base URL", async () => {
|
||||
vi.stubEnv("GEMINI_API_KEY", "env-google-key");
|
||||
const requestMock = installGoogleTtsRequestMock();
|
||||
const provider = buildGoogleSpeechProvider();
|
||||
|
||||
expect(provider.isConfigured({ providerConfig: {}, timeoutMs: 1 })).toBe(true);
|
||||
|
||||
await provider.synthesize({
|
||||
text: "Read this plainly.",
|
||||
cfg: {
|
||||
models: {
|
||||
providers: {
|
||||
google: {
|
||||
baseUrl: "https://generativelanguage.googleapis.com/v1beta/openai",
|
||||
models: [],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
providerConfig: {},
|
||||
target: "voice-note",
|
||||
timeoutMs: 10_000,
|
||||
});
|
||||
|
||||
const request = expectRecordFields(requireFirstRecordArg(requestMock, "Google TTS request"), {
|
||||
url: "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-flash-tts-preview:generateContent",
|
||||
}) as { headers?: HeadersInit };
|
||||
expect(new Headers(request.headers).get("x-goog-api-key")).toBe("env-google-key");
|
||||
});
|
||||
|
||||
it("can reuse a configured Google model-provider API key without auth profiles", async () => {
|
||||
const requestMock = installGoogleTtsRequestMock();
|
||||
const provider = buildGoogleSpeechProvider();
|
||||
const cfg = {
|
||||
models: {
|
||||
providers: {
|
||||
google: {
|
||||
apiKey: "model-provider-google-key",
|
||||
baseUrl: "https://generativelanguage.googleapis.com",
|
||||
models: [],
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
expect(provider.isConfigured({ cfg, providerConfig: {}, timeoutMs: 1 })).toBe(true);
|
||||
|
||||
await provider.synthesize({
|
||||
text: "Use the configured model provider key.",
|
||||
cfg,
|
||||
providerConfig: {},
|
||||
target: "audio-file",
|
||||
timeoutMs: 10_000,
|
||||
});
|
||||
|
||||
const request = requireFirstRecordArg(requestMock, "Google TTS request") as {
|
||||
headers?: HeadersInit;
|
||||
};
|
||||
expect(new Headers(request.headers).get("x-goog-api-key")).toBe("model-provider-google-key");
|
||||
});
|
||||
|
||||
it("returns Gemini PCM directly for telephony synthesis", async () => {
|
||||
const pcm = Buffer.from([3, 0, 4, 0]);
|
||||
installGoogleTtsRequestMock(pcm);
|
||||
const provider = buildGoogleSpeechProvider();
|
||||
|
||||
const result = await provider.synthesizeTelephony?.({
|
||||
text: "Phone call audio.",
|
||||
cfg: {},
|
||||
providerConfig: {
|
||||
apiKey: "google-test-key",
|
||||
model: "google/gemini-3.1-flash-tts",
|
||||
voice: "Kore",
|
||||
audioProfile: "Speak calmly.",
|
||||
speakerName: "Default speaker",
|
||||
},
|
||||
providerOverrides: {
|
||||
model: "google/gemini-3.1-pro-tts",
|
||||
voiceName: "Puck",
|
||||
audioProfile: "Speak brightly.",
|
||||
speakerName: "Override speaker",
|
||||
},
|
||||
timeoutMs: 5_000,
|
||||
});
|
||||
|
||||
const request = expectRecordFields(
|
||||
requireFirstRecordArg(postJsonRequestMock, "Google telephony TTS request"),
|
||||
{
|
||||
url: "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-pro-tts:generateContent",
|
||||
},
|
||||
);
|
||||
const body = request.body as {
|
||||
contents?: unknown;
|
||||
generationConfig?: { speechConfig?: unknown };
|
||||
};
|
||||
expect(body.contents).toEqual([
|
||||
{
|
||||
role: "user",
|
||||
parts: [{ text: "Speak brightly.\n\nSpeaker name: Override speaker\n\nPhone call audio." }],
|
||||
},
|
||||
]);
|
||||
expect(body.generationConfig?.speechConfig).toEqual({
|
||||
voiceConfig: {
|
||||
prebuiltVoiceConfig: {
|
||||
voiceName: "Puck",
|
||||
},
|
||||
},
|
||||
});
|
||||
expect(result).toEqual({
|
||||
audioBuffer: pcm,
|
||||
outputFormat: "pcm",
|
||||
sampleRate: 24_000,
|
||||
});
|
||||
});
|
||||
|
||||
it("prepends configured Gemini TTS profile text", async () => {
|
||||
const requestMock = installGoogleTtsRequestMock();
|
||||
const provider = buildGoogleSpeechProvider();
|
||||
|
||||
await provider.synthesize({
|
||||
text: "Status update starts now.",
|
||||
cfg: {},
|
||||
providerConfig: {
|
||||
apiKey: "google-test-key",
|
||||
audioProfile: "Speak professionally with a calm executive tone.",
|
||||
speakerName: "Alex",
|
||||
},
|
||||
target: "audio-file",
|
||||
timeoutMs: 10_000,
|
||||
});
|
||||
|
||||
const request = requireFirstRecordArg(requestMock, "Google TTS request") as {
|
||||
body?: { contents?: Array<{ parts?: Array<{ text?: string }> }> };
|
||||
};
|
||||
expect(request.body?.contents?.[0]?.parts?.[0]?.text).toBe(
|
||||
"Speak professionally with a calm executive tone.\n\n" +
|
||||
"Speaker name: Alex\n\n" +
|
||||
"Status update starts now.",
|
||||
);
|
||||
});
|
||||
|
||||
it("resolves provider config and directive overrides", () => {
|
||||
const provider = buildGoogleSpeechProvider();
|
||||
|
||||
expect(
|
||||
provider.resolveConfig?.({
|
||||
cfg: {},
|
||||
rawConfig: {
|
||||
providers: {
|
||||
google: {
|
||||
apiKey: "configured-key",
|
||||
model: "google/gemini-3.1-flash-tts-preview",
|
||||
voice: "Leda",
|
||||
audioProfile: "Speak warmly.",
|
||||
speakerName: "Narrator",
|
||||
},
|
||||
},
|
||||
},
|
||||
timeoutMs: 1,
|
||||
}),
|
||||
).toEqual({
|
||||
apiKey: "configured-key",
|
||||
audioProfile: "Speak warmly.",
|
||||
baseUrl: undefined,
|
||||
model: "gemini-3.1-flash-tts-preview",
|
||||
speakerName: "Narrator",
|
||||
voiceName: "Leda",
|
||||
});
|
||||
|
||||
expect(
|
||||
provider.parseDirectiveToken?.({
|
||||
key: "google_voice",
|
||||
value: "Aoede",
|
||||
policy: {
|
||||
enabled: true,
|
||||
allowText: true,
|
||||
allowProvider: true,
|
||||
allowVoice: true,
|
||||
allowModelId: true,
|
||||
allowVoiceSettings: true,
|
||||
allowNormalization: true,
|
||||
allowSeed: true,
|
||||
},
|
||||
}),
|
||||
).toEqual({
|
||||
handled: true,
|
||||
overrides: {
|
||||
voiceName: "Aoede",
|
||||
},
|
||||
});
|
||||
|
||||
expect(
|
||||
provider.parseDirectiveToken?.({
|
||||
key: "google_model",
|
||||
value: "gemini-3.1-flash-tts-preview",
|
||||
policy: {
|
||||
enabled: true,
|
||||
allowText: true,
|
||||
allowProvider: true,
|
||||
allowVoice: true,
|
||||
allowModelId: true,
|
||||
allowVoiceSettings: true,
|
||||
allowNormalization: true,
|
||||
allowSeed: true,
|
||||
},
|
||||
}),
|
||||
).toEqual({
|
||||
handled: true,
|
||||
overrides: {
|
||||
model: "gemini-3.1-flash-tts-preview",
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("lists Gemini prebuilt TTS voices", async () => {
|
||||
const provider = buildGoogleSpeechProvider();
|
||||
|
||||
const voices = await provider.listVoices?.({ providerConfig: {} });
|
||||
const voiceLabels = voices?.map((voice) => `${voice.id}:${voice.name}`);
|
||||
expect(voiceLabels).toContain("Kore:Kore");
|
||||
expect(voiceLabels).toContain("Puck:Puck");
|
||||
});
|
||||
|
||||
it("formats Google TTS HTTP errors with provider details", async () => {
|
||||
assertOkOrThrowProviderErrorMock.mockRejectedValue(
|
||||
new Error(
|
||||
"Google TTS failed (429): Quota exceeded [code=RESOURCE_EXHAUSTED] [request_id=google_req_123]",
|
||||
),
|
||||
);
|
||||
postJsonRequestMock.mockResolvedValue({
|
||||
response: new Response(
|
||||
JSON.stringify({
|
||||
error: {
|
||||
message: "Quota exceeded",
|
||||
status: "RESOURCE_EXHAUSTED",
|
||||
},
|
||||
}),
|
||||
{
|
||||
status: 429,
|
||||
headers: { "x-request-id": "google_req_123" },
|
||||
},
|
||||
),
|
||||
release: vi.fn(async () => {}),
|
||||
});
|
||||
const provider = buildGoogleSpeechProvider();
|
||||
|
||||
await expect(
|
||||
provider.synthesize({
|
||||
text: "Read this plainly.",
|
||||
cfg: {},
|
||||
providerConfig: { apiKey: "google-test-key" },
|
||||
target: "audio-file",
|
||||
timeoutMs: 10_000,
|
||||
}),
|
||||
).rejects.toThrow(
|
||||
"Google TTS failed (429): Quota exceeded [code=RESOURCE_EXHAUSTED] [request_id=google_req_123]",
|
||||
);
|
||||
});
|
||||
|
||||
it("honors configured private-network opt-in for Google TTS", async () => {
|
||||
installGoogleTtsRequestMock();
|
||||
|
||||
const provider = buildGoogleSpeechProvider();
|
||||
await provider.synthesize({
|
||||
text: "hello",
|
||||
cfg: {
|
||||
models: {
|
||||
providers: {
|
||||
google: {
|
||||
baseUrl: "https://generativelanguage.googleapis.com/v1beta",
|
||||
request: { allowPrivateNetwork: true },
|
||||
models: [],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
providerConfig: { apiKey: "google-test-key" },
|
||||
target: "audio-file",
|
||||
timeoutMs: 12_345,
|
||||
});
|
||||
|
||||
const requestConfig = expectRecordFields(
|
||||
requireFirstRecordArg(resolveProviderHttpRequestConfigMock, "Google TTS HTTP config request"),
|
||||
{
|
||||
allowPrivateNetwork: true,
|
||||
},
|
||||
);
|
||||
expectRecordFields(requestConfig.request, { allowPrivateNetwork: true });
|
||||
});
|
||||
|
||||
it("honors configured private-network opt-in for Google telephony TTS", async () => {
|
||||
installGoogleTtsRequestMock();
|
||||
|
||||
const provider = buildGoogleSpeechProvider();
|
||||
await provider.synthesizeTelephony?.({
|
||||
text: "hello",
|
||||
cfg: {
|
||||
models: {
|
||||
providers: {
|
||||
google: {
|
||||
baseUrl: "https://generativelanguage.googleapis.com/v1beta",
|
||||
request: { allowPrivateNetwork: true },
|
||||
models: [],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
providerConfig: { apiKey: "google-test-key" },
|
||||
timeoutMs: 12_345,
|
||||
});
|
||||
|
||||
const requestConfig = expectRecordFields(
|
||||
requireFirstRecordArg(resolveProviderHttpRequestConfigMock, "Google TTS HTTP config request"),
|
||||
{
|
||||
allowPrivateNetwork: true,
|
||||
},
|
||||
);
|
||||
expectRecordFields(requestConfig.request, { allowPrivateNetwork: true });
|
||||
});
|
||||
});
|
||||
677
extensions/google/speech-provider.ts
Normal file
677
extensions/google/speech-provider.ts
Normal file
@@ -0,0 +1,677 @@
|
||||
// Google provider module implements model/runtime integration.
|
||||
import { transcodeAudioBufferToOpus } from "openclaw/plugin-sdk/media-runtime";
|
||||
import {
|
||||
assertOkOrThrowProviderError,
|
||||
postJsonRequest,
|
||||
readProviderJsonResponse,
|
||||
sanitizeConfiguredModelProviderRequest,
|
||||
} from "openclaw/plugin-sdk/provider-http";
|
||||
import type { OpenClawConfig } from "openclaw/plugin-sdk/provider-onboard";
|
||||
import { normalizeResolvedSecretInputString } from "openclaw/plugin-sdk/secret-input";
|
||||
import type {
|
||||
SpeechDirectiveTokenParseContext,
|
||||
SpeechProviderConfig,
|
||||
SpeechProviderOverrides,
|
||||
SpeechProviderPlugin,
|
||||
SpeechSynthesisRequest,
|
||||
} from "openclaw/plugin-sdk/speech-core";
|
||||
import { asObject, trimToUndefined } from "openclaw/plugin-sdk/speech-core";
|
||||
import { normalizeOptionalString } from "openclaw/plugin-sdk/string-coerce-runtime";
|
||||
import { resolveGoogleGenerativeAiHttpRequestConfig } from "./api.js";
|
||||
|
||||
const DEFAULT_GOOGLE_TTS_MODEL = "gemini-3.1-flash-tts-preview";
|
||||
const DEFAULT_GOOGLE_TTS_VOICE = "Kore";
|
||||
const GOOGLE_TTS_SAMPLE_RATE = 24_000;
|
||||
const GOOGLE_TTS_CHANNELS = 1;
|
||||
const GOOGLE_TTS_BITS_PER_SAMPLE = 16;
|
||||
const GOOGLE_AUDIO_PROFILE_PROMPT_TEMPLATE = "audio-profile-v1";
|
||||
|
||||
const GOOGLE_TTS_MODELS = [
|
||||
"gemini-3.1-flash-tts-preview",
|
||||
"gemini-2.5-flash-preview-tts",
|
||||
"gemini-2.5-pro-preview-tts",
|
||||
] as const;
|
||||
|
||||
const GOOGLE_TTS_VOICES = [
|
||||
"Zephyr",
|
||||
"Puck",
|
||||
"Charon",
|
||||
"Kore",
|
||||
"Fenrir",
|
||||
"Leda",
|
||||
"Orus",
|
||||
"Aoede",
|
||||
"Callirrhoe",
|
||||
"Autonoe",
|
||||
"Enceladus",
|
||||
"Iapetus",
|
||||
"Umbriel",
|
||||
"Algieba",
|
||||
"Despina",
|
||||
"Erinome",
|
||||
"Algenib",
|
||||
"Rasalgethi",
|
||||
"Laomedeia",
|
||||
"Achernar",
|
||||
"Alnilam",
|
||||
"Schedar",
|
||||
"Gacrux",
|
||||
"Pulcherrima",
|
||||
"Achird",
|
||||
"Zubenelgenubi",
|
||||
"Vindemiatrix",
|
||||
"Sadachbia",
|
||||
"Sadaltager",
|
||||
"Sulafat",
|
||||
] as const;
|
||||
|
||||
type GoogleTtsProviderConfig = {
|
||||
apiKey?: string;
|
||||
baseUrl?: string;
|
||||
model: string;
|
||||
voiceName: string;
|
||||
audioProfile?: string;
|
||||
speakerName?: string;
|
||||
promptTemplate?: typeof GOOGLE_AUDIO_PROFILE_PROMPT_TEMPLATE;
|
||||
personaPrompt?: string;
|
||||
};
|
||||
|
||||
type GoogleTtsProviderOverrides = {
|
||||
model?: string;
|
||||
voiceName?: string;
|
||||
audioProfile?: string;
|
||||
speakerName?: string;
|
||||
};
|
||||
|
||||
type Maybe<T> = T | undefined;
|
||||
|
||||
type GoogleInlineDataPart = {
|
||||
mimeType?: string;
|
||||
mime_type?: string;
|
||||
data?: string;
|
||||
};
|
||||
|
||||
type GoogleGenerateSpeechResponse = {
|
||||
candidates?: Array<{
|
||||
content?: {
|
||||
parts?: Array<{
|
||||
text?: string;
|
||||
inlineData?: GoogleInlineDataPart;
|
||||
inline_data?: GoogleInlineDataPart;
|
||||
}>;
|
||||
};
|
||||
}>;
|
||||
};
|
||||
|
||||
class GoogleTtsRetryableError extends Error {
|
||||
constructor(message: string) {
|
||||
super(message);
|
||||
this.name = "GoogleTtsRetryableError";
|
||||
}
|
||||
}
|
||||
|
||||
function isGoogleTtsRetryableError(err: unknown): boolean {
|
||||
if (err instanceof GoogleTtsRetryableError) {
|
||||
return true;
|
||||
}
|
||||
if (!(err instanceof Error)) {
|
||||
return false;
|
||||
}
|
||||
if (err.name === "AbortError") {
|
||||
return true;
|
||||
}
|
||||
const message = err.message.toLowerCase();
|
||||
return (
|
||||
message.includes("aborted") ||
|
||||
message.includes("timeout") ||
|
||||
message.includes("fetch failed") ||
|
||||
message.includes("network")
|
||||
);
|
||||
}
|
||||
|
||||
function normalizeGoogleTtsModel(model: unknown): string {
|
||||
const trimmed = normalizeOptionalString(model);
|
||||
if (!trimmed) {
|
||||
return DEFAULT_GOOGLE_TTS_MODEL;
|
||||
}
|
||||
const withoutProvider = trimmed.startsWith("google/") ? trimmed.slice("google/".length) : trimmed;
|
||||
return withoutProvider === "gemini-3.1-flash-tts" ? DEFAULT_GOOGLE_TTS_MODEL : withoutProvider;
|
||||
}
|
||||
|
||||
function normalizeGoogleTtsVoiceName(voiceName: unknown): string {
|
||||
return normalizeOptionalString(voiceName) ?? DEFAULT_GOOGLE_TTS_VOICE;
|
||||
}
|
||||
|
||||
function normalizeGooglePromptTemplate(
|
||||
value: unknown,
|
||||
): typeof GOOGLE_AUDIO_PROFILE_PROMPT_TEMPLATE | undefined {
|
||||
const trimmed = normalizeOptionalString(value);
|
||||
if (!trimmed) {
|
||||
return undefined;
|
||||
}
|
||||
if (trimmed === GOOGLE_AUDIO_PROFILE_PROMPT_TEMPLATE) {
|
||||
return trimmed;
|
||||
}
|
||||
throw new Error(`Invalid Google TTS promptTemplate: ${trimmed}`);
|
||||
}
|
||||
|
||||
function resolveGoogleTtsEnvApiKey(): string | undefined {
|
||||
return (
|
||||
normalizeOptionalString(process.env.GEMINI_API_KEY) ??
|
||||
normalizeOptionalString(process.env.GOOGLE_API_KEY)
|
||||
);
|
||||
}
|
||||
|
||||
function resolveGoogleTtsModelProviderApiKey(cfg?: OpenClawConfig): string | undefined {
|
||||
return normalizeResolvedSecretInputString({
|
||||
value: cfg?.models?.providers?.google?.apiKey,
|
||||
path: "models.providers.google.apiKey",
|
||||
});
|
||||
}
|
||||
|
||||
function resolveGoogleTtsApiKey(params: {
|
||||
cfg?: OpenClawConfig;
|
||||
providerConfig: SpeechProviderConfig;
|
||||
}): string | undefined {
|
||||
return (
|
||||
readGoogleTtsProviderConfig(params.providerConfig).apiKey ??
|
||||
resolveGoogleTtsModelProviderApiKey(params.cfg) ??
|
||||
resolveGoogleTtsEnvApiKey()
|
||||
);
|
||||
}
|
||||
|
||||
function resolveGoogleTtsBaseUrl(params: {
|
||||
cfg?: OpenClawConfig;
|
||||
providerConfig: GoogleTtsProviderConfig;
|
||||
}): string | undefined {
|
||||
return (
|
||||
params.providerConfig.baseUrl ?? trimToUndefined(params.cfg?.models?.providers?.google?.baseUrl)
|
||||
);
|
||||
}
|
||||
|
||||
function resolveGoogleTtsConfigRecord(
|
||||
rawConfig: Record<string, unknown>,
|
||||
): Record<string, unknown> | undefined {
|
||||
const providers = asObject(rawConfig.providers);
|
||||
return asObject(providers?.google) ?? asObject(rawConfig.google);
|
||||
}
|
||||
|
||||
function normalizeGoogleTtsProviderConfig(
|
||||
rawConfig: Record<string, unknown>,
|
||||
): GoogleTtsProviderConfig {
|
||||
const raw = resolveGoogleTtsConfigRecord(rawConfig);
|
||||
const promptTemplate = normalizeGooglePromptTemplate(raw?.promptTemplate);
|
||||
const personaPrompt = trimToUndefined(raw?.personaPrompt);
|
||||
return {
|
||||
apiKey: normalizeResolvedSecretInputString({
|
||||
value: raw?.apiKey,
|
||||
path: "messages.tts.providers.google.apiKey",
|
||||
}),
|
||||
baseUrl: trimToUndefined(raw?.baseUrl),
|
||||
model: normalizeGoogleTtsModel(raw?.model),
|
||||
voiceName: normalizeGoogleTtsVoiceName(raw?.voiceName ?? raw?.voice),
|
||||
audioProfile: trimToUndefined(raw?.audioProfile),
|
||||
speakerName: trimToUndefined(raw?.speakerName),
|
||||
...(promptTemplate ? { promptTemplate } : {}),
|
||||
...(personaPrompt ? { personaPrompt } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
function readGoogleTtsProviderConfig(config: SpeechProviderConfig): GoogleTtsProviderConfig {
|
||||
const normalized = normalizeGoogleTtsProviderConfig({});
|
||||
const promptTemplate =
|
||||
normalizeGooglePromptTemplate(config.promptTemplate) ?? normalized.promptTemplate;
|
||||
const personaPrompt = trimToUndefined(config.personaPrompt) ?? normalized.personaPrompt;
|
||||
return {
|
||||
apiKey: trimToUndefined(config.apiKey) ?? normalized.apiKey,
|
||||
baseUrl: trimToUndefined(config.baseUrl) ?? normalized.baseUrl,
|
||||
model: normalizeGoogleTtsModel(config.model ?? normalized.model),
|
||||
voiceName: normalizeGoogleTtsVoiceName(
|
||||
config.voiceName ?? config.voice ?? normalized.voiceName,
|
||||
),
|
||||
audioProfile: trimToUndefined(config.audioProfile) ?? normalized.audioProfile,
|
||||
speakerName: trimToUndefined(config.speakerName) ?? normalized.speakerName,
|
||||
...(promptTemplate ? { promptTemplate } : {}),
|
||||
...(personaPrompt ? { personaPrompt } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
function readGoogleTtsOverrides(
|
||||
overrides: Maybe<SpeechProviderOverrides>,
|
||||
): GoogleTtsProviderOverrides {
|
||||
if (!overrides) {
|
||||
return {};
|
||||
}
|
||||
return {
|
||||
model: normalizeOptionalString(overrides.model),
|
||||
voiceName: normalizeOptionalString(overrides.voiceName ?? overrides.voice),
|
||||
audioProfile: normalizeOptionalString(overrides.audioProfile),
|
||||
speakerName: normalizeOptionalString(overrides.speakerName),
|
||||
};
|
||||
}
|
||||
|
||||
function composeGoogleTtsText(params: {
|
||||
text: string;
|
||||
audioProfile?: string;
|
||||
speakerName?: string;
|
||||
}): string {
|
||||
return [
|
||||
trimToUndefined(params.audioProfile),
|
||||
trimToUndefined(params.speakerName) ? `Speaker name: ${params.speakerName}` : undefined,
|
||||
params.text,
|
||||
]
|
||||
.filter((part): part is string => part !== undefined)
|
||||
.join("\n\n");
|
||||
}
|
||||
|
||||
function parseDirectiveToken(ctx: SpeechDirectiveTokenParseContext): {
|
||||
handled: boolean;
|
||||
overrides?: SpeechProviderOverrides;
|
||||
warnings?: string[];
|
||||
} {
|
||||
switch (ctx.key) {
|
||||
case "voicename":
|
||||
case "voice_name":
|
||||
case "google_voice":
|
||||
case "googlevoice":
|
||||
if (!ctx.policy.allowVoice) {
|
||||
return { handled: true };
|
||||
}
|
||||
return { handled: true, overrides: { voiceName: ctx.value } };
|
||||
case "google_model":
|
||||
case "googlemodel":
|
||||
if (!ctx.policy.allowModelId) {
|
||||
return { handled: true };
|
||||
}
|
||||
return { handled: true, overrides: { model: ctx.value } };
|
||||
default:
|
||||
return { handled: false };
|
||||
}
|
||||
}
|
||||
|
||||
function extractGoogleSpeechPcm(payload: GoogleGenerateSpeechResponse): Buffer {
|
||||
for (const candidate of payload.candidates ?? []) {
|
||||
for (const part of candidate.content?.parts ?? []) {
|
||||
const inline = part.inlineData ?? part.inline_data;
|
||||
const data = normalizeOptionalString(inline?.data);
|
||||
if (!data) {
|
||||
continue;
|
||||
}
|
||||
return Buffer.from(data, "base64");
|
||||
}
|
||||
}
|
||||
throw new Error("Google TTS response missing audio data");
|
||||
}
|
||||
|
||||
function normalizePromptSectionText(value: string | undefined): string | undefined {
|
||||
const trimmed = trimToUndefined(value?.replace(/\r\n?/g, "\n"));
|
||||
if (!trimmed) {
|
||||
return undefined;
|
||||
}
|
||||
let sanitized = "";
|
||||
for (const char of trimmed) {
|
||||
const code = char.charCodeAt(0);
|
||||
if (
|
||||
(code >= 0 && code <= 8) ||
|
||||
code === 11 ||
|
||||
code === 12 ||
|
||||
(code >= 14 && code <= 31) ||
|
||||
code === 127
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
sanitized += char;
|
||||
}
|
||||
return sanitized;
|
||||
}
|
||||
|
||||
function normalizePromptList(values: readonly string[] | undefined): string[] {
|
||||
return (values ?? [])
|
||||
.map((value) => normalizePromptSectionText(value))
|
||||
.filter((value): value is string => Boolean(value));
|
||||
}
|
||||
|
||||
function isOpenClawGoogleAudioProfilePrompt(text: string): boolean {
|
||||
return (
|
||||
text.includes("# AUDIO PROFILE:") &&
|
||||
text.includes("### TRANSCRIPT") &&
|
||||
text.startsWith("Synthesize speech from the TRANSCRIPT section only.")
|
||||
);
|
||||
}
|
||||
|
||||
function renderGoogleAudioProfilePrompt(params: {
|
||||
text: string;
|
||||
persona?: {
|
||||
id: string;
|
||||
label?: string;
|
||||
prompt?: {
|
||||
profile?: string;
|
||||
scene?: string;
|
||||
sampleContext?: string;
|
||||
style?: string;
|
||||
accent?: string;
|
||||
pacing?: string;
|
||||
constraints?: string[];
|
||||
};
|
||||
};
|
||||
personaPrompt?: string;
|
||||
}): string {
|
||||
const transcript = params.text.replace(/\r\n?/g, "\n").trim();
|
||||
const prompt = params.persona?.prompt;
|
||||
const profile = normalizePromptSectionText(prompt?.profile);
|
||||
const scene = normalizePromptSectionText(prompt?.scene);
|
||||
const sampleContext = normalizePromptSectionText(prompt?.sampleContext);
|
||||
const style = normalizePromptSectionText(prompt?.style);
|
||||
const accent = normalizePromptSectionText(prompt?.accent);
|
||||
const pacing = normalizePromptSectionText(prompt?.pacing);
|
||||
const constraints = normalizePromptList(prompt?.constraints);
|
||||
const personaPrompt = normalizePromptSectionText(params.personaPrompt);
|
||||
const label =
|
||||
normalizePromptSectionText(params.persona?.label) ??
|
||||
normalizePromptSectionText(params.persona?.id);
|
||||
|
||||
const sections = [
|
||||
[
|
||||
"Synthesize speech from the TRANSCRIPT section only. Use the other sections only",
|
||||
"as performance direction. Do not read section titles, notes, labels, or",
|
||||
"configuration aloud.",
|
||||
].join("\n"),
|
||||
];
|
||||
|
||||
if (label || profile) {
|
||||
sections.push([`# AUDIO PROFILE: ${label ?? "voice"}`, profile].filter(Boolean).join("\n"));
|
||||
}
|
||||
if (scene) {
|
||||
sections.push(["## THE SCENE", scene].join("\n"));
|
||||
}
|
||||
|
||||
const directorNotes: string[] = [];
|
||||
if (style) {
|
||||
directorNotes.push(`Style: ${style}`);
|
||||
}
|
||||
if (accent) {
|
||||
directorNotes.push(`Accent: ${accent}`);
|
||||
}
|
||||
if (pacing) {
|
||||
directorNotes.push(`Pacing: ${pacing}`);
|
||||
}
|
||||
if (constraints.length > 0) {
|
||||
directorNotes.push(["Constraints:", ...constraints.map((item) => `- ${item}`)].join("\n"));
|
||||
}
|
||||
if (personaPrompt) {
|
||||
directorNotes.push(["Provider notes:", personaPrompt].join("\n"));
|
||||
}
|
||||
if (directorNotes.length > 0) {
|
||||
sections.push(["### DIRECTOR'S NOTES", ...directorNotes].join("\n"));
|
||||
}
|
||||
|
||||
if (sampleContext) {
|
||||
sections.push(["### SAMPLE CONTEXT", sampleContext].join("\n"));
|
||||
}
|
||||
|
||||
sections.push(["### TRANSCRIPT", transcript].join("\n"));
|
||||
return sections.join("\n\n");
|
||||
}
|
||||
|
||||
function wrapPcm16MonoToWav(pcm: Buffer, sampleRate = GOOGLE_TTS_SAMPLE_RATE): Buffer {
|
||||
const byteRate = sampleRate * GOOGLE_TTS_CHANNELS * (GOOGLE_TTS_BITS_PER_SAMPLE / 8);
|
||||
const blockAlign = GOOGLE_TTS_CHANNELS * (GOOGLE_TTS_BITS_PER_SAMPLE / 8);
|
||||
const header = Buffer.alloc(44);
|
||||
|
||||
header.write("RIFF", 0, "ascii");
|
||||
header.writeUInt32LE(36 + pcm.length, 4);
|
||||
header.write("WAVE", 8, "ascii");
|
||||
header.write("fmt ", 12, "ascii");
|
||||
header.writeUInt32LE(16, 16);
|
||||
header.writeUInt16LE(1, 20);
|
||||
header.writeUInt16LE(GOOGLE_TTS_CHANNELS, 22);
|
||||
header.writeUInt32LE(sampleRate, 24);
|
||||
header.writeUInt32LE(byteRate, 28);
|
||||
header.writeUInt16LE(blockAlign, 32);
|
||||
header.writeUInt16LE(GOOGLE_TTS_BITS_PER_SAMPLE, 34);
|
||||
header.write("data", 36, "ascii");
|
||||
header.writeUInt32LE(pcm.length, 40);
|
||||
|
||||
return Buffer.concat([header, pcm]);
|
||||
}
|
||||
|
||||
async function synthesizeGoogleTtsPcmOnce(params: {
|
||||
text: string;
|
||||
apiKey: string;
|
||||
baseUrl?: string;
|
||||
request?: ReturnType<typeof sanitizeConfiguredModelProviderRequest>;
|
||||
model: string;
|
||||
voiceName: string;
|
||||
audioProfile?: string;
|
||||
speakerName?: string;
|
||||
timeoutMs: number;
|
||||
}): Promise<Buffer> {
|
||||
const { baseUrl, allowPrivateNetwork, headers, dispatcherPolicy } =
|
||||
resolveGoogleGenerativeAiHttpRequestConfig({
|
||||
apiKey: params.apiKey,
|
||||
baseUrl: params.baseUrl,
|
||||
request: params.request,
|
||||
capability: "audio",
|
||||
transport: "http",
|
||||
});
|
||||
|
||||
const { response: res, release } = await postJsonRequest({
|
||||
url: `${baseUrl}/models/${params.model}:generateContent`,
|
||||
headers,
|
||||
body: {
|
||||
contents: [
|
||||
{
|
||||
role: "user",
|
||||
parts: [
|
||||
{
|
||||
text: composeGoogleTtsText({
|
||||
text: params.text,
|
||||
audioProfile: params.audioProfile,
|
||||
speakerName: params.speakerName,
|
||||
}),
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
generationConfig: {
|
||||
responseModalities: ["AUDIO"],
|
||||
speechConfig: {
|
||||
voiceConfig: {
|
||||
prebuiltVoiceConfig: {
|
||||
voiceName: params.voiceName,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
timeoutMs: params.timeoutMs,
|
||||
fetchFn: fetch,
|
||||
pinDns: false,
|
||||
allowPrivateNetwork,
|
||||
dispatcherPolicy,
|
||||
});
|
||||
|
||||
try {
|
||||
if (!res.ok) {
|
||||
try {
|
||||
await assertOkOrThrowProviderError(res, "Google TTS failed");
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
if (res.status >= 500 && res.status < 600) {
|
||||
throw new GoogleTtsRetryableError(message);
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
try {
|
||||
const payload = await readProviderJsonResponse<GoogleGenerateSpeechResponse>(
|
||||
res,
|
||||
"Google TTS response",
|
||||
);
|
||||
return extractGoogleSpeechPcm(payload);
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
throw new GoogleTtsRetryableError(message);
|
||||
}
|
||||
} finally {
|
||||
await release();
|
||||
}
|
||||
}
|
||||
|
||||
async function synthesizeGoogleTtsPcm(params: {
|
||||
text: string;
|
||||
apiKey: string;
|
||||
baseUrl?: string;
|
||||
request?: ReturnType<typeof sanitizeConfiguredModelProviderRequest>;
|
||||
model: string;
|
||||
voiceName: string;
|
||||
audioProfile?: string;
|
||||
speakerName?: string;
|
||||
timeoutMs: number;
|
||||
}): Promise<Buffer> {
|
||||
let lastError: unknown;
|
||||
for (let attempt = 0; attempt < 2; attempt += 1) {
|
||||
try {
|
||||
return await synthesizeGoogleTtsPcmOnce(params);
|
||||
} catch (err) {
|
||||
lastError = err;
|
||||
if (!isGoogleTtsRetryableError(err) || attempt > 0) {
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
}
|
||||
throw lastError instanceof Error ? lastError : new Error(String(lastError));
|
||||
}
|
||||
|
||||
type GoogleTtsSynthesisRequest = Pick<
|
||||
SpeechSynthesisRequest,
|
||||
"cfg" | "providerConfig" | "providerOverrides" | "text" | "timeoutMs"
|
||||
>;
|
||||
|
||||
async function synthesizeConfiguredGoogleTts(req: GoogleTtsSynthesisRequest): Promise<Buffer> {
|
||||
const config = readGoogleTtsProviderConfig(req.providerConfig);
|
||||
const overrides = readGoogleTtsOverrides(req.providerOverrides);
|
||||
const apiKey = resolveGoogleTtsApiKey({
|
||||
cfg: req.cfg,
|
||||
providerConfig: req.providerConfig,
|
||||
});
|
||||
if (!apiKey) {
|
||||
throw new Error("Google API key missing");
|
||||
}
|
||||
return synthesizeGoogleTtsPcm({
|
||||
text: req.text,
|
||||
apiKey,
|
||||
baseUrl: resolveGoogleTtsBaseUrl({ cfg: req.cfg, providerConfig: config }),
|
||||
request: sanitizeConfiguredModelProviderRequest(req.cfg?.models?.providers?.google?.request),
|
||||
model: normalizeGoogleTtsModel(overrides.model ?? config.model),
|
||||
voiceName: normalizeGoogleTtsVoiceName(overrides.voiceName ?? config.voiceName),
|
||||
audioProfile: overrides.audioProfile ?? config.audioProfile,
|
||||
speakerName: overrides.speakerName ?? config.speakerName,
|
||||
timeoutMs: req.timeoutMs,
|
||||
});
|
||||
}
|
||||
|
||||
export function buildGoogleSpeechProvider(): SpeechProviderPlugin {
|
||||
return {
|
||||
id: "google",
|
||||
label: "Google",
|
||||
autoSelectOrder: 50,
|
||||
defaultModel: DEFAULT_GOOGLE_TTS_MODEL,
|
||||
models: GOOGLE_TTS_MODELS,
|
||||
voices: GOOGLE_TTS_VOICES,
|
||||
resolveConfig: ({ rawConfig }) => normalizeGoogleTtsProviderConfig(rawConfig),
|
||||
parseDirectiveToken,
|
||||
resolveTalkConfig: ({ baseTtsConfig, talkProviderConfig }) => {
|
||||
const base = normalizeGoogleTtsProviderConfig(baseTtsConfig);
|
||||
return {
|
||||
...base,
|
||||
...(talkProviderConfig.apiKey === undefined
|
||||
? {}
|
||||
: {
|
||||
apiKey: normalizeResolvedSecretInputString({
|
||||
value: talkProviderConfig.apiKey,
|
||||
path: "talk.providers.google.apiKey",
|
||||
}),
|
||||
}),
|
||||
...(trimToUndefined(talkProviderConfig.baseUrl) == null
|
||||
? {}
|
||||
: { baseUrl: trimToUndefined(talkProviderConfig.baseUrl) }),
|
||||
...(trimToUndefined(talkProviderConfig.modelId) == null
|
||||
? {}
|
||||
: { model: normalizeGoogleTtsModel(talkProviderConfig.modelId) }),
|
||||
...(trimToUndefined(talkProviderConfig.voiceId) == null
|
||||
? {}
|
||||
: { voiceName: normalizeGoogleTtsVoiceName(talkProviderConfig.voiceId) }),
|
||||
};
|
||||
},
|
||||
resolveTalkOverrides: ({ params }) => ({
|
||||
...(trimToUndefined(params.voiceId) == null
|
||||
? {}
|
||||
: { voiceName: normalizeGoogleTtsVoiceName(params.voiceId) }),
|
||||
...(trimToUndefined(params.modelId) == null
|
||||
? {}
|
||||
: { model: normalizeGoogleTtsModel(params.modelId) }),
|
||||
}),
|
||||
listVoices: async () => GOOGLE_TTS_VOICES.map((voice) => ({ id: voice, name: voice })),
|
||||
isConfigured: ({ cfg, providerConfig }) =>
|
||||
Boolean(resolveGoogleTtsApiKey({ cfg, providerConfig })),
|
||||
prepareSynthesis: (ctx) => {
|
||||
const config = readGoogleTtsProviderConfig(ctx.providerConfig);
|
||||
const shouldWrap =
|
||||
config.promptTemplate === GOOGLE_AUDIO_PROFILE_PROMPT_TEMPLATE ||
|
||||
Boolean(config.personaPrompt);
|
||||
if (!shouldWrap || isOpenClawGoogleAudioProfilePrompt(ctx.text)) {
|
||||
return undefined;
|
||||
}
|
||||
return {
|
||||
text: renderGoogleAudioProfilePrompt({
|
||||
text: ctx.text,
|
||||
persona: ctx.persona,
|
||||
personaPrompt: config.personaPrompt,
|
||||
}),
|
||||
};
|
||||
},
|
||||
synthesize: async (req) => {
|
||||
const pcm = await synthesizeConfiguredGoogleTts(req);
|
||||
if (req.target === "voice-note") {
|
||||
return {
|
||||
audioBuffer: await transcodeAudioBufferToOpus({
|
||||
audioBuffer: wrapPcm16MonoToWav(pcm),
|
||||
inputExtension: "wav",
|
||||
tempPrefix: "tts-google-",
|
||||
timeoutMs: req.timeoutMs,
|
||||
}),
|
||||
outputFormat: "opus",
|
||||
fileExtension: ".opus",
|
||||
voiceCompatible: true,
|
||||
};
|
||||
}
|
||||
return {
|
||||
audioBuffer: wrapPcm16MonoToWav(pcm),
|
||||
outputFormat: "wav",
|
||||
fileExtension: ".wav",
|
||||
voiceCompatible: false,
|
||||
};
|
||||
},
|
||||
synthesizeTelephony: async (req) => {
|
||||
const pcm = await synthesizeConfiguredGoogleTts(req);
|
||||
return {
|
||||
audioBuffer: pcm,
|
||||
outputFormat: "pcm",
|
||||
sampleRate: GOOGLE_TTS_SAMPLE_RATE,
|
||||
};
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export const testing = {
|
||||
DEFAULT_GOOGLE_TTS_MODEL,
|
||||
DEFAULT_GOOGLE_TTS_VOICE,
|
||||
GOOGLE_AUDIO_PROFILE_PROMPT_TEMPLATE,
|
||||
GOOGLE_TTS_MODELS,
|
||||
GOOGLE_TTS_SAMPLE_RATE,
|
||||
normalizeGoogleTtsModel,
|
||||
renderGoogleAudioProfilePrompt,
|
||||
wrapPcm16MonoToWav,
|
||||
};
|
||||
export { testing as __testing };
|
||||
375
extensions/google/src/gemini-web-search-provider.runtime.ts
Normal file
375
extensions/google/src/gemini-web-search-provider.runtime.ts
Normal file
@@ -0,0 +1,375 @@
|
||||
// Google provider module implements model/runtime integration.
|
||||
import {
|
||||
createProviderHttpError,
|
||||
formatProviderHttpErrorMessage,
|
||||
readProviderJsonObjectResponse,
|
||||
} from "openclaw/plugin-sdk/provider-http";
|
||||
import {
|
||||
buildSearchCacheKey,
|
||||
buildUnsupportedSearchFilterResponse,
|
||||
DEFAULT_SEARCH_COUNT,
|
||||
MAX_SEARCH_COUNT,
|
||||
parseWebSearchTimeFilters,
|
||||
readCachedSearchPayload,
|
||||
readConfiguredSecretString,
|
||||
readPositiveIntegerParam,
|
||||
readProviderEnvValue,
|
||||
readStringParam,
|
||||
resolveCitationRedirectUrl,
|
||||
resolveSearchCacheTtlMs,
|
||||
resolveSearchCount,
|
||||
resolveSearchTimeoutSeconds,
|
||||
type SearchConfigRecord,
|
||||
withTrustedWebSearchEndpoint,
|
||||
wrapWebContent,
|
||||
writeCachedSearchPayload,
|
||||
} from "openclaw/plugin-sdk/provider-web-search";
|
||||
import { isRecord } from "openclaw/plugin-sdk/string-coerce-runtime";
|
||||
import {
|
||||
resolveGeminiConfig,
|
||||
resolveGeminiBaseUrl,
|
||||
resolveGeminiModel,
|
||||
type GeminiConfig,
|
||||
} from "./gemini-web-search-provider.shared.js";
|
||||
|
||||
type GeminiFreshness = "day" | "week" | "month" | "year";
|
||||
|
||||
type GeminiTimeRangeFilter = {
|
||||
startTime: string;
|
||||
endTime: string;
|
||||
};
|
||||
|
||||
type GeminiGroundingResponse = {
|
||||
candidates?: Array<{
|
||||
content?: {
|
||||
parts?: Array<{
|
||||
text?: string;
|
||||
}>;
|
||||
};
|
||||
groundingMetadata?: {
|
||||
groundingChunks?: Array<{
|
||||
web?: {
|
||||
uri?: string;
|
||||
title?: string;
|
||||
};
|
||||
}>;
|
||||
};
|
||||
}>;
|
||||
error?: {
|
||||
code?: number;
|
||||
message?: string;
|
||||
status?: string;
|
||||
};
|
||||
};
|
||||
|
||||
function throwMalformedGeminiResponse(): never {
|
||||
throw new Error("Gemini API error: malformed JSON response");
|
||||
}
|
||||
|
||||
const GEMINI_FRESHNESS_DAYS: Record<GeminiFreshness, number> = {
|
||||
day: 1,
|
||||
week: 7,
|
||||
month: 30,
|
||||
year: 365,
|
||||
};
|
||||
|
||||
const GEMINI_DAY_FRESHNESS_HINT = "Prioritize web sources published in the last 24 hours.";
|
||||
|
||||
// Gemini's google_search.time_range_filter accepts second-precision RFC 3339
|
||||
// only. Despite the underlying google.protobuf.Timestamp type accepting "0, 3,
|
||||
// 6 or 9 fractional digits", the Search grounding endpoint rejects any
|
||||
// non-zero fractional component with
|
||||
// "[FIELD_INVALID] Granularity of nano is not supported".
|
||||
// Strip the fractional-second component before serializing.
|
||||
function toGeminiTimeRangeTimestamp(date: Date): string {
|
||||
return date.toISOString().replace(/\.\d+Z$/, "Z");
|
||||
}
|
||||
|
||||
function isoDateStart(value: string): string {
|
||||
return `${value}T00:00:00Z`;
|
||||
}
|
||||
|
||||
function isoDateExclusiveEnd(value: string): string {
|
||||
const end = new Date(`${value}T00:00:00Z`);
|
||||
end.setUTCDate(end.getUTCDate() + 1);
|
||||
return toGeminiTimeRangeTimestamp(end);
|
||||
}
|
||||
|
||||
function freshnessStartTime(freshness: GeminiFreshness, now: Date): string {
|
||||
const start = new Date(now);
|
||||
start.setUTCDate(start.getUTCDate() - GEMINI_FRESHNESS_DAYS[freshness]);
|
||||
return toGeminiTimeRangeTimestamp(start);
|
||||
}
|
||||
|
||||
function queryWithSoftFreshness(query: string, freshness?: "day"): string {
|
||||
if (freshness !== "day") {
|
||||
return query;
|
||||
}
|
||||
return `${query}\n\nSearch recency instruction: ${GEMINI_DAY_FRESHNESS_HINT} If no matching recent sources are available, state that limitation and use the most relevant available sources.`;
|
||||
}
|
||||
|
||||
function resolveGeminiTimeRangeFilter(
|
||||
args: Record<string, unknown>,
|
||||
now = new Date(),
|
||||
):
|
||||
| { timeRangeFilter?: GeminiTimeRangeFilter; freshness?: "day" }
|
||||
| {
|
||||
error:
|
||||
| "invalid_freshness"
|
||||
| "invalid_date"
|
||||
| "invalid_date_range"
|
||||
| "conflicting_time_filters";
|
||||
message: string;
|
||||
docs: string;
|
||||
} {
|
||||
const rawFreshness = readStringParam(args, "freshness");
|
||||
const rawDateAfter = readStringParam(args, "date_after");
|
||||
const rawDateBefore = readStringParam(args, "date_before");
|
||||
const parsedTimeFilters = parseWebSearchTimeFilters({
|
||||
rawDateAfter,
|
||||
rawDateBefore,
|
||||
rawFreshness,
|
||||
freshnessProvider: "perplexity",
|
||||
invalidFreshnessMessage:
|
||||
"freshness must be day, week, month, year, or the shortcuts pd, pw, pm, py.",
|
||||
invalidDateAfterMessage: "date_after must be YYYY-MM-DD format.",
|
||||
invalidDateBeforeMessage: "date_before must be YYYY-MM-DD format.",
|
||||
invalidDateRangeMessage: "date_after must be before date_before.",
|
||||
});
|
||||
if ("error" in parsedTimeFilters) {
|
||||
return parsedTimeFilters;
|
||||
}
|
||||
|
||||
const { freshness, dateAfter, dateBefore } = parsedTimeFilters;
|
||||
if (freshness) {
|
||||
// Gemini rejects 24-hour google_search.timeRangeFilter windows, while
|
||||
// wider freshness windows still preserve the hard grounding contract.
|
||||
if (freshness === "day") {
|
||||
return {
|
||||
freshness,
|
||||
};
|
||||
}
|
||||
return {
|
||||
timeRangeFilter: {
|
||||
startTime: freshnessStartTime(freshness, now),
|
||||
endTime: toGeminiTimeRangeTimestamp(now),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
if (!dateAfter && !dateBefore) {
|
||||
return {};
|
||||
}
|
||||
|
||||
return {
|
||||
timeRangeFilter: {
|
||||
startTime: dateAfter ? isoDateStart(dateAfter) : "1970-01-01T00:00:00Z",
|
||||
endTime: dateBefore ? isoDateExclusiveEnd(dateBefore) : toGeminiTimeRangeTimestamp(now),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export function resolveGeminiRuntimeApiKey(gemini?: GeminiConfig): string | undefined {
|
||||
return (
|
||||
readConfiguredSecretString(gemini?.apiKey, "tools.web.search.gemini.apiKey") ??
|
||||
readProviderEnvValue(["GEMINI_API_KEY"]) ??
|
||||
readConfiguredSecretString(gemini?.providerApiKey, "models.providers.google.apiKey")
|
||||
);
|
||||
}
|
||||
|
||||
async function runGeminiSearch(params: {
|
||||
query: string;
|
||||
apiKey: string;
|
||||
baseUrl: string;
|
||||
model: string;
|
||||
timeoutSeconds: number;
|
||||
signal?: AbortSignal;
|
||||
timeRangeFilter?: GeminiTimeRangeFilter;
|
||||
}): Promise<{ content: string; citations: Array<{ url: string; title?: string }> }> {
|
||||
const endpoint = `${params.baseUrl}/models/${params.model}:generateContent`;
|
||||
const googleSearch =
|
||||
params.timeRangeFilter === undefined ? {} : { timeRangeFilter: params.timeRangeFilter };
|
||||
|
||||
return withTrustedWebSearchEndpoint(
|
||||
{
|
||||
url: endpoint,
|
||||
timeoutSeconds: params.timeoutSeconds,
|
||||
signal: params.signal,
|
||||
init: {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
"x-goog-api-key": params.apiKey,
|
||||
},
|
||||
body: JSON.stringify({
|
||||
contents: [{ parts: [{ text: params.query }] }],
|
||||
tools: [{ google_search: googleSearch }],
|
||||
}),
|
||||
},
|
||||
},
|
||||
async (res) => {
|
||||
if (!res.ok) {
|
||||
const error = await createProviderHttpError(res, "Gemini API error");
|
||||
throw new Error(error.message.replace(/key=[^&\s]+/giu, "key=***"));
|
||||
}
|
||||
|
||||
const data = (await readProviderJsonObjectResponse(
|
||||
res,
|
||||
"Gemini API error",
|
||||
)) as GeminiGroundingResponse;
|
||||
|
||||
if (data.error) {
|
||||
const rawMessage = data.error.message || data.error.status || "unknown";
|
||||
throw new Error(
|
||||
formatProviderHttpErrorMessage({
|
||||
label: "Gemini API error",
|
||||
status: data.error.code ?? 0,
|
||||
detail: rawMessage.replace(/key=[^&\s]+/giu, "key=***"),
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
if (!Array.isArray(data.candidates)) {
|
||||
throwMalformedGeminiResponse();
|
||||
}
|
||||
const candidate = data.candidates[0];
|
||||
if (!isRecord(candidate) || !isRecord(candidate.content)) {
|
||||
throwMalformedGeminiResponse();
|
||||
}
|
||||
const parts = candidate.content.parts;
|
||||
if (!Array.isArray(parts)) {
|
||||
throwMalformedGeminiResponse();
|
||||
}
|
||||
const content = parts
|
||||
.map((part) => (isRecord(part) && typeof part.text === "string" ? part.text : undefined))
|
||||
.filter((text): text is string => Boolean(text))
|
||||
.join("\n");
|
||||
if (!content) {
|
||||
throwMalformedGeminiResponse();
|
||||
}
|
||||
const groundingMetadata = candidate.groundingMetadata;
|
||||
const groundingChunks =
|
||||
groundingMetadata === undefined
|
||||
? []
|
||||
: isRecord(groundingMetadata)
|
||||
? groundingMetadata.groundingChunks === undefined
|
||||
? []
|
||||
: Array.isArray(groundingMetadata.groundingChunks)
|
||||
? groundingMetadata.groundingChunks
|
||||
: undefined
|
||||
: undefined;
|
||||
if (!groundingChunks) {
|
||||
throwMalformedGeminiResponse();
|
||||
}
|
||||
const rawCitations = groundingChunks.flatMap((chunk) => {
|
||||
if (!isRecord(chunk) || !isRecord(chunk.web) || typeof chunk.web.uri !== "string") {
|
||||
return [];
|
||||
}
|
||||
return [
|
||||
{
|
||||
url: chunk.web.uri,
|
||||
title: typeof chunk.web.title === "string" ? chunk.web.title : undefined,
|
||||
},
|
||||
];
|
||||
});
|
||||
|
||||
const citations: Array<{ url: string; title?: string }> = [];
|
||||
for (let index = 0; index < rawCitations.length; index += 10) {
|
||||
const batch = rawCitations.slice(index, index + 10);
|
||||
const resolved = await Promise.all(
|
||||
batch.map(async (citation) =>
|
||||
Object.assign({}, citation, { url: await resolveCitationRedirectUrl(citation.url) }),
|
||||
),
|
||||
);
|
||||
citations.push(...resolved);
|
||||
}
|
||||
|
||||
return { content, citations };
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
export async function executeGeminiSearch(
|
||||
args: Record<string, unknown>,
|
||||
searchConfig?: SearchConfigRecord,
|
||||
context?: { signal?: AbortSignal },
|
||||
): Promise<Record<string, unknown>> {
|
||||
const unsupportedResponse = buildUnsupportedSearchFilterResponse(
|
||||
{
|
||||
country: args.country,
|
||||
language: args.language,
|
||||
},
|
||||
"gemini",
|
||||
);
|
||||
if (unsupportedResponse) {
|
||||
return unsupportedResponse;
|
||||
}
|
||||
|
||||
const timeRange = resolveGeminiTimeRangeFilter(args);
|
||||
if ("error" in timeRange) {
|
||||
return timeRange;
|
||||
}
|
||||
|
||||
const geminiConfig = resolveGeminiConfig(searchConfig);
|
||||
const apiKey = resolveGeminiRuntimeApiKey(geminiConfig);
|
||||
if (!apiKey) {
|
||||
return {
|
||||
error: "missing_gemini_api_key",
|
||||
message:
|
||||
"web_search (gemini) needs an API key. Set GEMINI_API_KEY in the Gateway environment, configure plugins.entries.google.config.webSearch.apiKey, or reuse models.providers.google.apiKey. If you do not want to configure a search API key, use web_fetch for a specific URL or the browser tool for interactive pages.",
|
||||
docs: "https://docs.openclaw.ai/tools/web",
|
||||
};
|
||||
}
|
||||
|
||||
const query = readStringParam(args, "query", { required: true });
|
||||
const count =
|
||||
readPositiveIntegerParam(args, "count", {
|
||||
max: MAX_SEARCH_COUNT,
|
||||
message: `count must be an integer from 1 to ${MAX_SEARCH_COUNT}.`,
|
||||
}) ??
|
||||
searchConfig?.maxResults ??
|
||||
undefined;
|
||||
const model = resolveGeminiModel(geminiConfig);
|
||||
const baseUrl = resolveGeminiBaseUrl(geminiConfig);
|
||||
const cacheKey = buildSearchCacheKey([
|
||||
"gemini",
|
||||
query,
|
||||
resolveSearchCount(count, DEFAULT_SEARCH_COUNT),
|
||||
baseUrl,
|
||||
model,
|
||||
timeRange.freshness,
|
||||
timeRange.timeRangeFilter?.startTime,
|
||||
timeRange.timeRangeFilter?.endTime,
|
||||
]);
|
||||
const cached = readCachedSearchPayload(cacheKey);
|
||||
if (cached) {
|
||||
return cached;
|
||||
}
|
||||
|
||||
const start = Date.now();
|
||||
const result = await runGeminiSearch({
|
||||
query: queryWithSoftFreshness(query, timeRange.freshness),
|
||||
apiKey,
|
||||
baseUrl,
|
||||
model,
|
||||
timeoutSeconds: resolveSearchTimeoutSeconds(searchConfig),
|
||||
signal: context?.signal,
|
||||
timeRangeFilter: timeRange.timeRangeFilter,
|
||||
});
|
||||
const payload = {
|
||||
query,
|
||||
provider: "gemini",
|
||||
model,
|
||||
tookMs: Date.now() - start,
|
||||
externalContent: {
|
||||
untrusted: true,
|
||||
source: "web_search",
|
||||
provider: "gemini",
|
||||
wrapped: true,
|
||||
},
|
||||
content: wrapWebContent(result.content),
|
||||
citations: result.citations,
|
||||
};
|
||||
writeCachedSearchPayload(cacheKey, payload, resolveSearchCacheTtlMs(searchConfig));
|
||||
return payload;
|
||||
}
|
||||
42
extensions/google/src/gemini-web-search-provider.shared.ts
Normal file
42
extensions/google/src/gemini-web-search-provider.shared.ts
Normal file
@@ -0,0 +1,42 @@
|
||||
// Google provider module implements model/runtime integration.
|
||||
import {
|
||||
isRecord,
|
||||
normalizeOptionalString as trimToUndefined,
|
||||
} from "openclaw/plugin-sdk/string-coerce-runtime";
|
||||
import { normalizeGoogleApiBaseUrl } from "../provider-policy.js";
|
||||
|
||||
const DEFAULT_GEMINI_WEB_SEARCH_MODEL = "gemini-2.5-flash";
|
||||
|
||||
export type GeminiConfig = {
|
||||
apiKey?: unknown;
|
||||
baseUrl?: unknown;
|
||||
model?: unknown;
|
||||
providerApiKey?: unknown;
|
||||
providerBaseUrl?: unknown;
|
||||
};
|
||||
|
||||
export function resolveGeminiConfig(searchConfig?: Record<string, unknown>): GeminiConfig {
|
||||
const gemini = searchConfig?.gemini;
|
||||
return isRecord(gemini) ? gemini : {};
|
||||
}
|
||||
|
||||
export function resolveGeminiApiKey(
|
||||
gemini?: GeminiConfig,
|
||||
env: Record<string, string | undefined> = process.env,
|
||||
): string | undefined {
|
||||
return (
|
||||
trimToUndefined(gemini?.apiKey) ??
|
||||
trimToUndefined(env.GEMINI_API_KEY) ??
|
||||
trimToUndefined(gemini?.providerApiKey)
|
||||
);
|
||||
}
|
||||
|
||||
export function resolveGeminiModel(gemini?: GeminiConfig): string {
|
||||
return trimToUndefined(gemini?.model) ?? DEFAULT_GEMINI_WEB_SEARCH_MODEL;
|
||||
}
|
||||
|
||||
export function resolveGeminiBaseUrl(gemini?: GeminiConfig): string {
|
||||
return normalizeGoogleApiBaseUrl(
|
||||
trimToUndefined(gemini?.baseUrl) ?? trimToUndefined(gemini?.providerBaseUrl),
|
||||
);
|
||||
}
|
||||
155
extensions/google/src/gemini-web-search-provider.ts
Normal file
155
extensions/google/src/gemini-web-search-provider.ts
Normal file
@@ -0,0 +1,155 @@
|
||||
// Google provider module implements model/runtime integration.
|
||||
import type { OpenClawConfig } from "openclaw/plugin-sdk/config-contracts";
|
||||
import { createLazyRuntimeModule } from "openclaw/plugin-sdk/lazy-runtime";
|
||||
import {
|
||||
createWebSearchProviderContractFields,
|
||||
mergeScopedSearchConfig,
|
||||
resolveProviderWebSearchPluginConfig,
|
||||
type WebSearchProviderPlugin,
|
||||
type WebSearchProviderToolDefinition,
|
||||
} from "openclaw/plugin-sdk/provider-web-search-config-contract";
|
||||
import { isRecord } from "openclaw/plugin-sdk/string-coerce-runtime";
|
||||
import {
|
||||
resolveGeminiApiKey,
|
||||
resolveGeminiBaseUrl,
|
||||
resolveGeminiModel,
|
||||
} from "./gemini-web-search-provider.shared.js";
|
||||
|
||||
const GEMINI_CREDENTIAL_PATH = "plugins.entries.google.config.webSearch.apiKey";
|
||||
const GOOGLE_PROVIDER_CREDENTIAL_PATH = "models.providers.google.apiKey";
|
||||
|
||||
const loadGeminiWebSearchRuntime = createLazyRuntimeModule(
|
||||
() => import("./gemini-web-search-provider.runtime.js"),
|
||||
);
|
||||
|
||||
const GEMINI_TOOL_PARAMETERS = {
|
||||
type: "object",
|
||||
properties: {
|
||||
query: { type: "string", description: "Search query string." },
|
||||
count: {
|
||||
type: "integer",
|
||||
description: "Number of results to return (1-10).",
|
||||
minimum: 1,
|
||||
maximum: 10,
|
||||
},
|
||||
country: { type: "string", description: "Not supported by Gemini." },
|
||||
language: { type: "string", description: "Not supported by Gemini." },
|
||||
freshness: {
|
||||
type: "string",
|
||||
description:
|
||||
"Filter Gemini search freshness: week, month, and year use hard Google Search time ranges; day prioritizes the last 24 hours as a recency hint.",
|
||||
},
|
||||
date_after: {
|
||||
type: "string",
|
||||
description: "Only ground with results published after this date (YYYY-MM-DD).",
|
||||
},
|
||||
date_before: {
|
||||
type: "string",
|
||||
description: "Only ground with results published before this date (YYYY-MM-DD).",
|
||||
},
|
||||
},
|
||||
required: ["query"],
|
||||
} satisfies Record<string, unknown>;
|
||||
|
||||
function createGeminiToolDefinition(
|
||||
searchConfig?: Record<string, unknown>,
|
||||
): WebSearchProviderToolDefinition {
|
||||
return {
|
||||
description:
|
||||
"Search the web using Gemini with Google Search grounding. Returns AI-synthesized answers with citations from Google Search.",
|
||||
parameters: GEMINI_TOOL_PARAMETERS,
|
||||
execute: async (args, context) => {
|
||||
const { executeGeminiSearch } = await loadGeminiWebSearchRuntime();
|
||||
return await executeGeminiSearch(args, searchConfig, context);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function resolveGoogleModelProviderConfig(
|
||||
config?: OpenClawConfig,
|
||||
): Record<string, unknown> | undefined {
|
||||
const provider = config?.models?.providers?.google;
|
||||
return isRecord(provider) ? provider : undefined;
|
||||
}
|
||||
|
||||
function getGoogleModelProviderCredentialFallback(
|
||||
config?: OpenClawConfig,
|
||||
): { path: string; value: unknown } | undefined {
|
||||
const provider = resolveGoogleModelProviderConfig(config);
|
||||
return provider && provider.apiKey !== undefined
|
||||
? { path: GOOGLE_PROVIDER_CREDENTIAL_PATH, value: provider.apiKey }
|
||||
: undefined;
|
||||
}
|
||||
|
||||
function withGoogleModelProviderFallbacks(
|
||||
searchConfig: Record<string, unknown> | undefined,
|
||||
config?: OpenClawConfig,
|
||||
): Record<string, unknown> | undefined {
|
||||
const provider = resolveGoogleModelProviderConfig(config);
|
||||
if (!provider || (provider.apiKey === undefined && provider.baseUrl === undefined)) {
|
||||
return searchConfig;
|
||||
}
|
||||
const gemini = isRecord(searchConfig?.gemini) ? { ...searchConfig.gemini } : {};
|
||||
const mergedSearchConfig: Record<string, unknown> = searchConfig
|
||||
? Object.defineProperties({}, Object.getOwnPropertyDescriptors(searchConfig))
|
||||
: {};
|
||||
const geminiDescriptor = searchConfig
|
||||
? Object.getOwnPropertyDescriptor(searchConfig, "gemini")
|
||||
: undefined;
|
||||
if (provider.apiKey !== undefined) {
|
||||
gemini.providerApiKey = provider.apiKey;
|
||||
}
|
||||
if (provider.baseUrl !== undefined) {
|
||||
gemini.providerBaseUrl = provider.baseUrl;
|
||||
}
|
||||
Object.defineProperty(mergedSearchConfig, "gemini", {
|
||||
value: gemini,
|
||||
enumerable: geminiDescriptor?.enumerable ?? false,
|
||||
configurable: true,
|
||||
writable: true,
|
||||
});
|
||||
return mergedSearchConfig;
|
||||
}
|
||||
|
||||
export function createGeminiWebSearchProvider(): WebSearchProviderPlugin {
|
||||
const contractFields = createWebSearchProviderContractFields({
|
||||
credentialPath: GEMINI_CREDENTIAL_PATH,
|
||||
searchCredential: { type: "scoped", scopeId: "gemini" },
|
||||
configuredCredential: { pluginId: "google" },
|
||||
});
|
||||
|
||||
return {
|
||||
id: "gemini",
|
||||
label: "Gemini (Google Search)",
|
||||
hint: "Requires Google Gemini API key · Google Search grounding",
|
||||
onboardingScopes: ["text-inference"],
|
||||
credentialLabel: "Google Gemini API key",
|
||||
envVars: ["GEMINI_API_KEY"],
|
||||
placeholder: "AIza...",
|
||||
signupUrl: "https://aistudio.google.com/apikey",
|
||||
docsUrl: "https://docs.openclaw.ai/tools/web",
|
||||
autoDetectOrder: 20,
|
||||
credentialPath: GEMINI_CREDENTIAL_PATH,
|
||||
...contractFields,
|
||||
getConfiguredCredentialFallback: getGoogleModelProviderCredentialFallback,
|
||||
createTool: (ctx) =>
|
||||
createGeminiToolDefinition(
|
||||
withGoogleModelProviderFallbacks(
|
||||
mergeScopedSearchConfig(
|
||||
ctx.searchConfig,
|
||||
"gemini",
|
||||
resolveProviderWebSearchPluginConfig(ctx.config, "google"),
|
||||
),
|
||||
ctx.config,
|
||||
),
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
export const testing = {
|
||||
resolveGeminiApiKey,
|
||||
resolveGeminiBaseUrl,
|
||||
resolveGeminiModel,
|
||||
withGoogleModelProviderFallbacks,
|
||||
} as const;
|
||||
export { testing as __testing };
|
||||
7
extensions/google/test-api.ts
Normal file
7
extensions/google/test-api.ts
Normal file
@@ -0,0 +1,7 @@
|
||||
// Google API module exposes the plugin public contract.
|
||||
export { buildGoogleGeminiCliBackend } from "./cli-backend.js";
|
||||
export { buildGoogleImageGenerationProvider } from "./image-generation-provider.js";
|
||||
export { buildGoogleMusicGenerationProvider } from "./music-generation-provider.js";
|
||||
export { buildGoogleSpeechProvider } from "./speech-provider.js";
|
||||
export { googleMediaUnderstandingProvider } from "./media-understanding-provider.js";
|
||||
export { buildGoogleVideoGenerationProvider } from "./video-generation-provider.js";
|
||||
15
extensions/google/thinking-api.ts
Normal file
15
extensions/google/thinking-api.ts
Normal file
@@ -0,0 +1,15 @@
|
||||
// Google API module exposes the plugin public contract.
|
||||
export {
|
||||
createGoogleThinkingPayloadWrapper,
|
||||
createGoogleThinkingStreamWrapper,
|
||||
isGoogleGemini25ThinkingBudgetModel,
|
||||
isGoogleGemini3FlashModel,
|
||||
isGoogleGemini3ProModel,
|
||||
isGoogleGemini3ThinkingLevelModel,
|
||||
isGoogleThinkingRequiredModel,
|
||||
resolveGoogleGemini3ThinkingLevel,
|
||||
sanitizeGoogleThinkingPayload,
|
||||
stripInvalidGoogleThinkingBudget,
|
||||
type GoogleThinkingInputLevel,
|
||||
type GoogleThinkingLevel,
|
||||
} from "./thinking.js";
|
||||
154
extensions/google/thinking.test.ts
Normal file
154
extensions/google/thinking.test.ts
Normal file
@@ -0,0 +1,154 @@
|
||||
// Google tests cover thinking plugin behavior.
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
resolveGoogleGemini3ThinkingLevel,
|
||||
sanitizeGoogleThinkingPayload,
|
||||
} from "./thinking-api.js";
|
||||
|
||||
describe("google thinking policy", () => {
|
||||
it.each([
|
||||
["off", "LOW"],
|
||||
["minimal", "LOW"],
|
||||
["low", "LOW"],
|
||||
["medium", "HIGH"],
|
||||
["adaptive", undefined],
|
||||
["high", "HIGH"],
|
||||
["xhigh", "HIGH"],
|
||||
] as const)("maps Gemini 3 Pro thinking level %s to %s", (thinkingLevel, expected) => {
|
||||
expect(
|
||||
resolveGoogleGemini3ThinkingLevel({
|
||||
modelId: "gemini-3.1-pro-preview",
|
||||
thinkingLevel,
|
||||
}),
|
||||
).toBe(expected);
|
||||
});
|
||||
|
||||
it.each([
|
||||
[0, "LOW"],
|
||||
[2048, "LOW"],
|
||||
[2049, "HIGH"],
|
||||
] as const)("maps Gemini 3 Pro budget %s to %s", (thinkingBudget, expected) => {
|
||||
expect(
|
||||
resolveGoogleGemini3ThinkingLevel({
|
||||
modelId: "gemini-pro-latest",
|
||||
thinkingBudget,
|
||||
}),
|
||||
).toBe(expected);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["off", "MINIMAL"],
|
||||
["minimal", "MINIMAL"],
|
||||
["low", "LOW"],
|
||||
["medium", "MEDIUM"],
|
||||
["adaptive", undefined],
|
||||
["high", "HIGH"],
|
||||
["xhigh", "HIGH"],
|
||||
] as const)("maps Gemini 3 Flash thinking level %s to %s", (thinkingLevel, expected) => {
|
||||
expect(
|
||||
resolveGoogleGemini3ThinkingLevel({
|
||||
modelId: "gemini-flash-latest",
|
||||
thinkingLevel,
|
||||
}),
|
||||
).toBe(expected);
|
||||
});
|
||||
|
||||
it.each([
|
||||
[-1, undefined],
|
||||
[0, "MINIMAL"],
|
||||
[2048, "LOW"],
|
||||
[8192, "MEDIUM"],
|
||||
[8193, "HIGH"],
|
||||
] as const)("maps Gemini 3 Flash budget %s to %s", (thinkingBudget, expected) => {
|
||||
expect(
|
||||
resolveGoogleGemini3ThinkingLevel({
|
||||
modelId: "gemini-3.1-flash-lite",
|
||||
thinkingBudget,
|
||||
}),
|
||||
).toBe(expected);
|
||||
});
|
||||
|
||||
it("removes thinkingBudget=0 for Gemini 2.5 Pro", () => {
|
||||
const payload = {
|
||||
config: {
|
||||
thinkingConfig: { thinkingBudget: 0 },
|
||||
},
|
||||
};
|
||||
|
||||
sanitizeGoogleThinkingPayload({ payload, modelId: "google/gemini-2.5-pro-preview" });
|
||||
|
||||
expect(payload.config).not.toHaveProperty("thinkingConfig");
|
||||
});
|
||||
|
||||
it("rewrites Gemini 3 thinking budgets to thinkingLevel", () => {
|
||||
const payload = {
|
||||
generationConfig: {
|
||||
thinkingConfig: { thinkingBudget: 8193, includeThoughts: true },
|
||||
},
|
||||
};
|
||||
|
||||
sanitizeGoogleThinkingPayload({
|
||||
payload,
|
||||
modelId: "gemini-3-flash-preview",
|
||||
thinkingLevel: "medium",
|
||||
});
|
||||
|
||||
expect(payload.generationConfig.thinkingConfig).toEqual({
|
||||
includeThoughts: true,
|
||||
thinkingLevel: "MEDIUM",
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps Gemini 3 adaptive thinking provider-dynamic instead of forcing a fixed level", () => {
|
||||
const payload = {
|
||||
generationConfig: {
|
||||
thinkingConfig: { thinkingBudget: 8192, includeThoughts: true },
|
||||
},
|
||||
};
|
||||
|
||||
sanitizeGoogleThinkingPayload({
|
||||
payload,
|
||||
modelId: "gemini-3-flash-preview",
|
||||
thinkingLevel: "adaptive",
|
||||
});
|
||||
|
||||
expect(payload.generationConfig.thinkingConfig).toEqual({
|
||||
includeThoughts: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("maps Gemini 2.5 adaptive thinking to dynamic thinkingBudget", () => {
|
||||
const payload = {
|
||||
config: {
|
||||
thinkingConfig: { thinkingBudget: 8192, includeThoughts: true },
|
||||
},
|
||||
};
|
||||
|
||||
sanitizeGoogleThinkingPayload({
|
||||
payload,
|
||||
modelId: "gemini-2.5-flash",
|
||||
thinkingLevel: "adaptive",
|
||||
});
|
||||
|
||||
expect(payload.config.thinkingConfig).toEqual({
|
||||
includeThoughts: true,
|
||||
thinkingBudget: -1,
|
||||
});
|
||||
});
|
||||
|
||||
it("maps Gemma 4 thinking mode without sending thinkingBudget", () => {
|
||||
const payload = {
|
||||
config: {
|
||||
thinkingConfig: { thinkingBudget: 4096 },
|
||||
},
|
||||
};
|
||||
|
||||
sanitizeGoogleThinkingPayload({
|
||||
payload,
|
||||
modelId: "gemma-4-26b-a4b-it",
|
||||
thinkingLevel: "high",
|
||||
});
|
||||
|
||||
expect(payload.config.thinkingConfig).toEqual({ thinkingLevel: "HIGH" });
|
||||
});
|
||||
});
|
||||
15
extensions/google/thinking.ts
Normal file
15
extensions/google/thinking.ts
Normal file
@@ -0,0 +1,15 @@
|
||||
// Google plugin module implements thinking behavior.
|
||||
export {
|
||||
createGoogleThinkingPayloadWrapper,
|
||||
createGoogleThinkingStreamWrapper,
|
||||
isGoogleGemini25ThinkingBudgetModel,
|
||||
isGoogleGemini3FlashModel,
|
||||
isGoogleGemini3ProModel,
|
||||
isGoogleGemini3ThinkingLevelModel,
|
||||
isGoogleThinkingRequiredModel,
|
||||
resolveGoogleGemini3ThinkingLevel,
|
||||
sanitizeGoogleThinkingPayload,
|
||||
stripInvalidGoogleThinkingBudget,
|
||||
type GoogleThinkingInputLevel,
|
||||
type GoogleThinkingLevel,
|
||||
} from "openclaw/plugin-sdk/provider-stream-shared";
|
||||
2700
extensions/google/transport-stream.test.ts
Normal file
2700
extensions/google/transport-stream.test.ts
Normal file
File diff suppressed because it is too large
Load Diff
1490
extensions/google/transport-stream.ts
Normal file
1490
extensions/google/transport-stream.ts
Normal file
File diff suppressed because it is too large
Load Diff
16
extensions/google/tsconfig.json
Normal file
16
extensions/google/tsconfig.json
Normal file
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"extends": "../tsconfig.package-boundary.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "."
|
||||
},
|
||||
"include": ["./*.ts", "./src/**/*.ts"],
|
||||
"exclude": [
|
||||
"./**/*.test.ts",
|
||||
"./dist/**",
|
||||
"./node_modules/**",
|
||||
"./src/test-support/**",
|
||||
"./src/**/*test-helpers.ts",
|
||||
"./src/**/*test-harness.ts",
|
||||
"./src/**/*test-support.ts"
|
||||
]
|
||||
}
|
||||
398
extensions/google/vertex-adc.ts
Normal file
398
extensions/google/vertex-adc.ts
Normal file
@@ -0,0 +1,398 @@
|
||||
// Google plugin module implements vertex adc behavior.
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import { readFile } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { gunzipSync } from "node:zlib";
|
||||
import {
|
||||
asDateTimestampMs,
|
||||
resolveExpiresAtMsFromDurationMs,
|
||||
resolveExpiresAtMsFromDurationSeconds,
|
||||
} from "openclaw/plugin-sdk/number-runtime";
|
||||
import { normalizeOptionalString } from "openclaw/plugin-sdk/string-coerce-runtime";
|
||||
|
||||
type GoogleAuthorizedUserCredentials = {
|
||||
type: "authorized_user";
|
||||
client_id?: string;
|
||||
client_secret?: string;
|
||||
refresh_token?: string;
|
||||
};
|
||||
|
||||
type GoogleVertexAuthorizedUserToken = {
|
||||
token: string;
|
||||
expiresAtMs: number;
|
||||
credentialsPath: string;
|
||||
refreshToken: string;
|
||||
};
|
||||
|
||||
type GoogleVertexAdcToken = {
|
||||
token: string;
|
||||
expiresAtMs: number;
|
||||
};
|
||||
|
||||
type GoogleOauthTokenResponsePayload = {
|
||||
access_token?: unknown;
|
||||
expires_in?: unknown;
|
||||
error?: unknown;
|
||||
error_description?: unknown;
|
||||
};
|
||||
|
||||
const GCP_VERTEX_CREDENTIALS_MARKER = "gcp-vertex-credentials";
|
||||
const GOOGLE_OAUTH_TOKEN_URL = "https://oauth2.googleapis.com/token";
|
||||
const GOOGLE_VERTEX_OAUTH_SCOPE = "https://www.googleapis.com/auth/cloud-platform";
|
||||
// Hold tokens slightly less long than reported expiry (Google's recommendation
|
||||
// is a 60s buffer) so we don't ship a request that's already revoked when it
|
||||
// leaves the gateway.
|
||||
const GOOGLE_VERTEX_TOKEN_EXPIRY_BUFFER_MS = 60_000;
|
||||
const GOOGLE_VERTEX_DEFAULT_TOKEN_LIFETIME_SECONDS = 3600;
|
||||
const GOOGLE_VERTEX_AUTHLIB_TOKEN_CACHE_MS = 5 * 60_000;
|
||||
|
||||
let cachedGoogleVertexAuthorizedUserToken: GoogleVertexAuthorizedUserToken | undefined;
|
||||
let cachedGoogleAuthClient:
|
||||
| {
|
||||
promise: Promise<{
|
||||
getAccessToken: () => Promise<string | null | undefined>;
|
||||
}>;
|
||||
}
|
||||
| undefined;
|
||||
let cachedGoogleVertexAdcToken: GoogleVertexAdcToken | undefined;
|
||||
|
||||
function isGoogleVertexTokenFresh(expiresAtMsRaw: number, nowRaw = Date.now()): boolean {
|
||||
const expiresAtMs = asDateTimestampMs(expiresAtMsRaw);
|
||||
const nowMs = asDateTimestampMs(nowRaw);
|
||||
if (expiresAtMs === undefined || nowMs === undefined) {
|
||||
return false;
|
||||
}
|
||||
const minFreshExpiresAtMs = resolveExpiresAtMsFromDurationMs(
|
||||
GOOGLE_VERTEX_TOKEN_EXPIRY_BUFFER_MS,
|
||||
{ nowMs },
|
||||
);
|
||||
return minFreshExpiresAtMs !== undefined && expiresAtMs > minFreshExpiresAtMs;
|
||||
}
|
||||
|
||||
function resolveAuthorizedUserTokenExpiresAtMs(value: unknown, nowRaw: number): number | undefined {
|
||||
const nowMs = asDateTimestampMs(nowRaw);
|
||||
if (nowMs === undefined) {
|
||||
return undefined;
|
||||
}
|
||||
const lifetimeSeconds =
|
||||
typeof value === "number" && Number.isFinite(value)
|
||||
? Math.max(1, value)
|
||||
: GOOGLE_VERTEX_DEFAULT_TOKEN_LIFETIME_SECONDS;
|
||||
return resolveExpiresAtMsFromDurationSeconds(lifetimeSeconds, { nowMs }) ?? nowMs;
|
||||
}
|
||||
|
||||
function resolveGoogleAuthLibraryTokenExpiresAtMs(nowRaw = Date.now()): number | undefined {
|
||||
const nowMs = asDateTimestampMs(nowRaw);
|
||||
return nowMs === undefined
|
||||
? undefined
|
||||
: resolveExpiresAtMsFromDurationMs(GOOGLE_VERTEX_AUTHLIB_TOKEN_CACHE_MS, { nowMs });
|
||||
}
|
||||
|
||||
export function resetGoogleVertexAuthorizedUserTokenCacheForTest(): void {
|
||||
cachedGoogleVertexAuthorizedUserToken = undefined;
|
||||
cachedGoogleAuthClient = undefined;
|
||||
cachedGoogleVertexAdcToken = undefined;
|
||||
}
|
||||
|
||||
export function isGoogleVertexCredentialsMarker(
|
||||
apiKey: string | undefined,
|
||||
): apiKey is undefined | typeof GCP_VERTEX_CREDENTIALS_MARKER {
|
||||
return apiKey === undefined || apiKey === GCP_VERTEX_CREDENTIALS_MARKER;
|
||||
}
|
||||
|
||||
function hasGoogleVertexProjectEnv(env: NodeJS.ProcessEnv): boolean {
|
||||
return Boolean(
|
||||
normalizeOptionalString(env.GOOGLE_CLOUD_PROJECT) ||
|
||||
normalizeOptionalString(env.GCLOUD_PROJECT),
|
||||
);
|
||||
}
|
||||
|
||||
function hasGoogleVertexLocationEnv(env: NodeJS.ProcessEnv): boolean {
|
||||
return Boolean(normalizeOptionalString(env.GOOGLE_CLOUD_LOCATION));
|
||||
}
|
||||
|
||||
function resolveGoogleApplicationCredentialsPath(
|
||||
env: NodeJS.ProcessEnv = process.env,
|
||||
): string | undefined {
|
||||
const explicit = normalizeOptionalString(env.GOOGLE_APPLICATION_CREDENTIALS);
|
||||
if (explicit) {
|
||||
return existsSync(explicit) ? explicit : undefined;
|
||||
}
|
||||
const homeDir = normalizeOptionalString(env.HOME) ?? os.homedir();
|
||||
const homeFallback = path.join(
|
||||
homeDir,
|
||||
".config",
|
||||
"gcloud",
|
||||
"application_default_credentials.json",
|
||||
);
|
||||
if (existsSync(homeFallback)) {
|
||||
return homeFallback;
|
||||
}
|
||||
const appDataDir = normalizeOptionalString(env.APPDATA);
|
||||
if (!appDataDir) {
|
||||
return undefined;
|
||||
}
|
||||
const appDataFallback = path.join(appDataDir, "gcloud", "application_default_credentials.json");
|
||||
return existsSync(appDataFallback) ? appDataFallback : undefined;
|
||||
}
|
||||
|
||||
async function readGoogleAuthorizedUserCredentials(
|
||||
credentialsPath: string,
|
||||
): Promise<GoogleAuthorizedUserCredentials | undefined> {
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(await readFile(credentialsPath, "utf8")) as unknown;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
|
||||
return undefined;
|
||||
}
|
||||
const record = parsed as Record<string, unknown>;
|
||||
if (record.type !== "authorized_user") {
|
||||
return undefined;
|
||||
}
|
||||
return {
|
||||
type: "authorized_user",
|
||||
client_id: normalizeOptionalString(record.client_id),
|
||||
client_secret: normalizeOptionalString(record.client_secret),
|
||||
refresh_token: normalizeOptionalString(record.refresh_token),
|
||||
};
|
||||
}
|
||||
|
||||
function readGoogleAdcCredentialsTypeSync(credentialsPath: string): string | undefined {
|
||||
try {
|
||||
const parsed = JSON.parse(readFileSync(credentialsPath, "utf8")) as unknown;
|
||||
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
|
||||
return undefined;
|
||||
}
|
||||
const type = (parsed as { type?: unknown }).type;
|
||||
return typeof type === "string" ? type : undefined;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns true when a file/env Application Default Credentials source usable
|
||||
* for Google Vertex AI is detectable synchronously. We still call the function
|
||||
* `...AuthorizedUserAdcSync` for backwards compatibility with older tests; the
|
||||
* predicate now also covers:
|
||||
*
|
||||
* 1. `authorized_user` credentials file (existing case - `gcloud auth
|
||||
* application-default login` produces this).
|
||||
* 2. `external_account` credentials file (Workload Identity Federation).
|
||||
* 3. `service_account` credentials file (raw GSA key - rarely used in
|
||||
* OpenClaw, included for completeness).
|
||||
* Metadata-server ADC is intentionally not detected here: `google-auth-library`
|
||||
* probes the default metadata hosts asynchronously at request time, and the
|
||||
* provider wires the Vertex transport without this sync predicate.
|
||||
*/
|
||||
export function hasGoogleVertexAuthorizedUserAdcSync(
|
||||
env: NodeJS.ProcessEnv = process.env,
|
||||
): boolean {
|
||||
const credentialsPath = resolveGoogleApplicationCredentialsPath(env);
|
||||
if (credentialsPath) {
|
||||
const type = readGoogleAdcCredentialsTypeSync(credentialsPath);
|
||||
if (type === "authorized_user" || type === "external_account" || type === "service_account") {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
export function resolveGoogleVertexConfigApiKey(
|
||||
env: NodeJS.ProcessEnv = process.env,
|
||||
): string | undefined {
|
||||
return hasGoogleVertexProjectEnv(env) &&
|
||||
hasGoogleVertexLocationEnv(env) &&
|
||||
hasGoogleVertexAuthorizedUserAdcSync(env)
|
||||
? GCP_VERTEX_CREDENTIALS_MARKER
|
||||
: undefined;
|
||||
}
|
||||
|
||||
async function refreshGoogleVertexAuthorizedUserAccessToken(params: {
|
||||
credentialsPath: string;
|
||||
credentials: GoogleAuthorizedUserCredentials;
|
||||
fetchImpl?: typeof fetch;
|
||||
}): Promise<string> {
|
||||
const clientId = normalizeOptionalString(params.credentials.client_id);
|
||||
const clientSecret = normalizeOptionalString(params.credentials.client_secret);
|
||||
const refreshToken = normalizeOptionalString(params.credentials.refresh_token);
|
||||
if (!clientId || !clientSecret || !refreshToken) {
|
||||
throw new Error(
|
||||
"Google Vertex authorized_user ADC is missing client_id, client_secret, or refresh_token.",
|
||||
);
|
||||
}
|
||||
|
||||
const cached = cachedGoogleVertexAuthorizedUserToken;
|
||||
if (
|
||||
cached?.credentialsPath === params.credentialsPath &&
|
||||
cached.refreshToken === refreshToken &&
|
||||
isGoogleVertexTokenFresh(cached.expiresAtMs)
|
||||
) {
|
||||
return cached.token;
|
||||
}
|
||||
|
||||
const body = new URLSearchParams({
|
||||
client_id: clientId,
|
||||
client_secret: clientSecret,
|
||||
refresh_token: refreshToken,
|
||||
grant_type: "refresh_token",
|
||||
});
|
||||
const response = await (params.fetchImpl ?? fetch)(GOOGLE_OAUTH_TOKEN_URL, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/x-www-form-urlencoded" },
|
||||
body,
|
||||
});
|
||||
const payload = await readGoogleOauthTokenResponsePayload(response);
|
||||
if (!response.ok) {
|
||||
const description = normalizeOptionalString(payload?.error_description);
|
||||
const code = normalizeOptionalString(payload?.error);
|
||||
throw new Error(
|
||||
`Google Vertex ADC token refresh failed: ${response.status}${code ? ` ${code}` : ""}${description ? ` (${description})` : ""}`,
|
||||
);
|
||||
}
|
||||
if (!payload) {
|
||||
throw new Error("Google Vertex ADC token refresh response could not be parsed as JSON.");
|
||||
}
|
||||
const token = normalizeOptionalString(payload?.access_token);
|
||||
if (!token) {
|
||||
throw new Error("Google Vertex ADC token refresh response did not include an access_token.");
|
||||
}
|
||||
const nowMs = Date.now();
|
||||
const expiresAtMs = resolveAuthorizedUserTokenExpiresAtMs(payload?.expires_in, nowMs);
|
||||
if (expiresAtMs !== undefined) {
|
||||
cachedGoogleVertexAuthorizedUserToken = {
|
||||
token,
|
||||
expiresAtMs,
|
||||
credentialsPath: params.credentialsPath,
|
||||
refreshToken,
|
||||
};
|
||||
}
|
||||
return token;
|
||||
}
|
||||
|
||||
async function readGoogleOauthTokenResponsePayload(
|
||||
response: Response,
|
||||
): Promise<GoogleOauthTokenResponsePayload | undefined> {
|
||||
const bytes = Buffer.from(await response.arrayBuffer());
|
||||
const text = decodeGoogleOauthTokenResponseBody(bytes, response.headers.get("content-encoding"));
|
||||
if (!text.trim()) {
|
||||
return undefined;
|
||||
}
|
||||
try {
|
||||
return JSON.parse(text) as GoogleOauthTokenResponsePayload;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
function decodeGoogleOauthTokenResponseBody(bytes: Buffer, contentEncoding: string | null): string {
|
||||
if (shouldGunzipGoogleOauthTokenResponse(bytes, contentEncoding)) {
|
||||
try {
|
||||
return gunzipSync(bytes).toString("utf8");
|
||||
} catch {
|
||||
return bytes.toString("utf8");
|
||||
}
|
||||
}
|
||||
return bytes.toString("utf8");
|
||||
}
|
||||
|
||||
function shouldGunzipGoogleOauthTokenResponse(
|
||||
bytes: Buffer,
|
||||
contentEncoding: string | null,
|
||||
): boolean {
|
||||
if (bytes[0] === 0x1f && bytes[1] === 0x8b) {
|
||||
return true;
|
||||
}
|
||||
return (contentEncoding ?? "")
|
||||
.split(",")
|
||||
.map((encoding) => encoding.trim().toLowerCase())
|
||||
.includes("gzip");
|
||||
}
|
||||
|
||||
async function resolveGoogleVertexAccessTokenViaGoogleAuth(): Promise<string> {
|
||||
// Lazy-import + cache so we don't pay the google-auth-library load cost on
|
||||
// gateway startup; only when we actually need a non-authorized_user token.
|
||||
if (!cachedGoogleAuthClient) {
|
||||
cachedGoogleAuthClient = {
|
||||
promise: import("google-auth-library").then(({ GoogleAuth }) => {
|
||||
// GoogleAuth handles every ADC variant we care about for GKE:
|
||||
// - external_account (Workload Identity Federation: STS exchange)
|
||||
// - service_account (raw GSA key: JWT-bearer)
|
||||
// - GKE Workload Identity (metadata server when no credentials file)
|
||||
// - Compute Engine / Cloud Run / GAE metadata server fallback
|
||||
// It also caches tokens internally and refreshes before expiry.
|
||||
return new GoogleAuth({
|
||||
scopes: [GOOGLE_VERTEX_OAUTH_SCOPE],
|
||||
});
|
||||
}),
|
||||
};
|
||||
}
|
||||
const auth = await cachedGoogleAuthClient.promise;
|
||||
|
||||
const cached = cachedGoogleVertexAdcToken;
|
||||
if (cached && isGoogleVertexTokenFresh(cached.expiresAtMs)) {
|
||||
return cached.token;
|
||||
}
|
||||
|
||||
const token = await auth.getAccessToken();
|
||||
const normalized = normalizeOptionalString(token);
|
||||
if (!normalized) {
|
||||
throw new Error(
|
||||
"Google Vertex ADC fallback (google-auth-library) did not return an access token. " +
|
||||
"Verify the GKE Workload Identity binding (KSA \u2192 GSA), `GOOGLE_APPLICATION_CREDENTIALS`, " +
|
||||
"or other ADC source is reachable from this pod.",
|
||||
);
|
||||
}
|
||||
// google-auth-library doesn't expose token expiry on the simple
|
||||
// `getAccessToken()` return type, so we cache for a conservative 5 minutes.
|
||||
// The library itself already refreshes well before its own internal expiry,
|
||||
// so this cache is mainly to avoid hot-loop calls into the auth client.
|
||||
const expiresAtMs = resolveGoogleAuthLibraryTokenExpiresAtMs();
|
||||
if (expiresAtMs !== undefined) {
|
||||
cachedGoogleVertexAdcToken = {
|
||||
token: normalized,
|
||||
expiresAtMs,
|
||||
};
|
||||
}
|
||||
return normalized;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve `Authorization: Bearer ...` headers for Google Vertex calls.
|
||||
*
|
||||
* We try the hand-rolled `authorized_user` refresh path first (preserves the
|
||||
* existing fetchImpl test seam and the OpenClaw upstream behaviour); when the
|
||||
* configured ADC source is anything other than `authorized_user` (the common
|
||||
* production cases on GKE: Workload Identity, Workload Identity Federation,
|
||||
* service-account JSON keys), we hand off to `google-auth-library` which
|
||||
* understands all of those natively.
|
||||
*
|
||||
* Note: the function is still named `...AuthorizedUserHeaders` to avoid a
|
||||
* symbol rename across the existing patch surface; the docstring above is
|
||||
* the truth, the name is legacy.
|
||||
*/
|
||||
export async function resolveGoogleVertexAuthorizedUserHeaders(
|
||||
fetchImpl?: typeof fetch,
|
||||
): Promise<Record<string, string>> {
|
||||
const credentialsPath = resolveGoogleApplicationCredentialsPath();
|
||||
if (credentialsPath) {
|
||||
const credentials = await readGoogleAuthorizedUserCredentials(credentialsPath);
|
||||
if (credentials) {
|
||||
const token = await refreshGoogleVertexAuthorizedUserAccessToken({
|
||||
credentialsPath,
|
||||
credentials,
|
||||
fetchImpl,
|
||||
});
|
||||
return { Authorization: `Bearer ${token}` };
|
||||
}
|
||||
}
|
||||
// No file-based authorized_user ADC. Fall back to google-auth-library which
|
||||
// handles GKE Workload Identity (metadata server), Workload Identity
|
||||
// Federation (external_account), and service-account keys.
|
||||
const token = await resolveGoogleVertexAccessTokenViaGoogleAuth();
|
||||
return { Authorization: `Bearer ${token}` };
|
||||
}
|
||||
68
extensions/google/vertex-multi-region-host.test.ts
Normal file
68
extensions/google/vertex-multi-region-host.test.ts
Normal file
@@ -0,0 +1,68 @@
|
||||
import type { Model } from "openclaw/plugin-sdk/llm";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { isGoogleVertexHostname } from "./provider-policy.js";
|
||||
import { resolveGoogleVertexBaseOrigin } from "./transport-stream.js";
|
||||
|
||||
// Minimal Vertex model whose baseUrl carries the {location} template, so the
|
||||
// base-origin resolver falls through to location-based host construction
|
||||
// (the configured-baseUrl early return only fires for a literal host).
|
||||
function buildModel(overrides: Partial<Model<"google-vertex">> = {}): Model<"google-vertex"> {
|
||||
return {
|
||||
id: "gemini-3.5-flash",
|
||||
name: "Gemini 3.5 Flash",
|
||||
api: "google-vertex",
|
||||
provider: "google-vertex",
|
||||
baseUrl: "https://{location}-aiplatform.googleapis.com",
|
||||
reasoning: true,
|
||||
input: ["text"],
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 128000,
|
||||
maxTokens: 8192,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe("Google Vertex multi-region host construction", () => {
|
||||
const model = buildModel();
|
||||
|
||||
it("routes the eu multi-region to the dedicated .rep.googleapis.com host", () => {
|
||||
expect(resolveGoogleVertexBaseOrigin(model, "eu")).toBe(
|
||||
"https://aiplatform.eu.rep.googleapis.com",
|
||||
);
|
||||
});
|
||||
|
||||
it("routes the us multi-region to the dedicated .rep.googleapis.com host", () => {
|
||||
expect(resolveGoogleVertexBaseOrigin(model, "us")).toBe(
|
||||
"https://aiplatform.us.rep.googleapis.com",
|
||||
);
|
||||
});
|
||||
|
||||
it("keeps the unprefixed host for the global location", () => {
|
||||
expect(resolveGoogleVertexBaseOrigin(model, "global")).toBe(
|
||||
"https://aiplatform.googleapis.com",
|
||||
);
|
||||
});
|
||||
|
||||
it("keeps the regional prefix for normal regions", () => {
|
||||
expect(resolveGoogleVertexBaseOrigin(model, "europe-west1")).toBe(
|
||||
"https://europe-west1-aiplatform.googleapis.com",
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("Google Vertex hostname recognition", () => {
|
||||
it("recognizes the multi-region rep host as a Vertex host", () => {
|
||||
expect(isGoogleVertexHostname("aiplatform.eu.rep.googleapis.com")).toBe(true);
|
||||
expect(isGoogleVertexHostname("aiplatform.us.rep.googleapis.com")).toBe(true);
|
||||
});
|
||||
|
||||
it("does not classify unrelated rep hosts as Vertex hosts", () => {
|
||||
expect(isGoogleVertexHostname("discoveryengine.eu.rep.googleapis.com")).toBe(false);
|
||||
expect(isGoogleVertexHostname("not-aiplatform.eu.rep.googleapis.com")).toBe(false);
|
||||
});
|
||||
|
||||
it("still recognizes the unprefixed and regional Vertex hosts", () => {
|
||||
expect(isGoogleVertexHostname("aiplatform.googleapis.com")).toBe(true);
|
||||
expect(isGoogleVertexHostname("europe-west1-aiplatform.googleapis.com")).toBe(true);
|
||||
});
|
||||
});
|
||||
761
extensions/google/video-generation-provider.test.ts
Normal file
761
extensions/google/video-generation-provider.test.ts
Normal file
@@ -0,0 +1,761 @@
|
||||
// Google tests cover video generation provider plugin behavior.
|
||||
import { mockPinnedHostnameResolution } from "openclaw/plugin-sdk/test-env";
|
||||
import { afterAll, afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
const { createGoogleGenAIMock, downloadMock, generateVideosMock, getVideosOperationMock } =
|
||||
vi.hoisted(() => {
|
||||
const generateVideosMockLocal = vi.fn();
|
||||
const getVideosOperationMockLocal = vi.fn();
|
||||
const downloadMockLocal = vi.fn();
|
||||
const createGoogleGenAIMockLocal = vi.fn(() => {
|
||||
return {
|
||||
models: {
|
||||
generateVideos: generateVideosMockLocal,
|
||||
},
|
||||
operations: {
|
||||
getVideosOperation: getVideosOperationMockLocal,
|
||||
},
|
||||
files: {
|
||||
download: downloadMockLocal,
|
||||
},
|
||||
};
|
||||
});
|
||||
return {
|
||||
createGoogleGenAIMock: createGoogleGenAIMockLocal,
|
||||
downloadMock: downloadMockLocal,
|
||||
generateVideosMock: generateVideosMockLocal,
|
||||
getVideosOperationMock: getVideosOperationMockLocal,
|
||||
};
|
||||
});
|
||||
|
||||
vi.mock("./google-genai-runtime.js", () => ({
|
||||
createGoogleGenAI: createGoogleGenAIMock,
|
||||
}));
|
||||
|
||||
import * as providerAuthRuntime from "openclaw/plugin-sdk/provider-auth-runtime";
|
||||
import { expectExplicitVideoGenerationCapabilities } from "openclaw/plugin-sdk/provider-test-contracts";
|
||||
import { buildGoogleVideoGenerationProvider } from "./video-generation-provider.js";
|
||||
|
||||
type MockWithCalls = {
|
||||
mock: { calls: unknown[][] };
|
||||
};
|
||||
|
||||
function firstObjectArg(mock: MockWithCalls): Record<string, unknown> {
|
||||
const [call] = mock.mock.calls;
|
||||
if (!call) {
|
||||
throw new Error("expected first mock call to receive an object argument");
|
||||
}
|
||||
const value = call[0];
|
||||
if (value === undefined || value === null || typeof value !== "object" || Array.isArray(value)) {
|
||||
throw new Error("expected first mock call to receive an object argument");
|
||||
}
|
||||
return value as Record<string, unknown>;
|
||||
}
|
||||
|
||||
function recordField(value: unknown, field: string): Record<string, unknown> {
|
||||
if (value === undefined || value === null || typeof value !== "object" || Array.isArray(value)) {
|
||||
throw new Error(`expected ${field} to be an object`);
|
||||
}
|
||||
return value as Record<string, unknown>;
|
||||
}
|
||||
|
||||
function firstGoogleClientHttpOptions(): Record<string, unknown> {
|
||||
return recordField(firstObjectArg(createGoogleGenAIMock).httpOptions, "httpOptions");
|
||||
}
|
||||
|
||||
function requireFetchCall(
|
||||
fetchMock: ReturnType<typeof vi.fn>,
|
||||
index: number,
|
||||
): [RequestInfo | URL, RequestInit | undefined] {
|
||||
const call = fetchMock.mock.calls[index];
|
||||
if (!call) {
|
||||
throw new Error(`expected Google video fetch call ${index}`);
|
||||
}
|
||||
return call as [RequestInfo | URL, RequestInit | undefined];
|
||||
}
|
||||
|
||||
function parseFetchJsonBody(fetchMock: ReturnType<typeof vi.fn>, index: number): unknown {
|
||||
const [, init] = requireFetchCall(fetchMock, index);
|
||||
const body = init?.body;
|
||||
if (typeof body !== "string") {
|
||||
throw new Error(`expected Google video fetch body ${index}`);
|
||||
}
|
||||
return JSON.parse(body) as unknown;
|
||||
}
|
||||
|
||||
function fetchInputUrl(fetchMock: ReturnType<typeof vi.fn>, index: number): string {
|
||||
const [input] = requireFetchCall(fetchMock, index);
|
||||
if (typeof input === "string") {
|
||||
return input;
|
||||
}
|
||||
if (input instanceof URL) {
|
||||
return input.toString();
|
||||
}
|
||||
return input.url;
|
||||
}
|
||||
|
||||
function oversizedJsonResponse(params: { chunkCount: number; chunkSize: number }): {
|
||||
response: Response;
|
||||
getReadCount: () => number;
|
||||
wasCanceled: () => boolean;
|
||||
} {
|
||||
const chunk = new Uint8Array(params.chunkSize);
|
||||
let readCount = 0;
|
||||
let canceled = false;
|
||||
return {
|
||||
response: new Response(
|
||||
new ReadableStream<Uint8Array>({
|
||||
pull(controller) {
|
||||
if (readCount >= params.chunkCount) {
|
||||
controller.close();
|
||||
return;
|
||||
}
|
||||
readCount += 1;
|
||||
controller.enqueue(chunk);
|
||||
},
|
||||
cancel() {
|
||||
canceled = true;
|
||||
},
|
||||
}),
|
||||
{
|
||||
status: 200,
|
||||
headers: { "Content-Type": "application/json" },
|
||||
},
|
||||
),
|
||||
getReadCount: () => readCount,
|
||||
wasCanceled: () => canceled,
|
||||
};
|
||||
}
|
||||
|
||||
let ssrfMock: { mockRestore: () => void } | undefined;
|
||||
|
||||
describe("google video generation provider", () => {
|
||||
beforeEach(() => {
|
||||
ssrfMock = mockPinnedHostnameResolution();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
ssrfMock?.mockRestore();
|
||||
ssrfMock = undefined;
|
||||
vi.restoreAllMocks();
|
||||
vi.unstubAllGlobals();
|
||||
downloadMock.mockReset();
|
||||
generateVideosMock.mockReset();
|
||||
getVideosOperationMock.mockReset();
|
||||
createGoogleGenAIMock.mockClear();
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
vi.doUnmock("./google-genai-runtime.js");
|
||||
vi.resetModules();
|
||||
});
|
||||
|
||||
it("declares explicit mode capabilities", () => {
|
||||
const provider = buildGoogleVideoGenerationProvider();
|
||||
expectExplicitVideoGenerationCapabilities(provider);
|
||||
expect(provider.capabilities.generate?.supportsAudio).toBe(false);
|
||||
expect(provider.capabilities.imageToVideo?.supportsAudio).toBe(false);
|
||||
expect(provider.capabilities.videoToVideo?.supportsAudio).toBe(false);
|
||||
});
|
||||
|
||||
it("submits generation and returns inline video bytes", async () => {
|
||||
vi.spyOn(providerAuthRuntime, "resolveApiKeyForProvider").mockResolvedValue({
|
||||
apiKey: "google-key",
|
||||
source: "env",
|
||||
mode: "api-key",
|
||||
});
|
||||
generateVideosMock.mockResolvedValue({
|
||||
done: true,
|
||||
name: "operations/123",
|
||||
response: {
|
||||
generatedVideos: [
|
||||
{
|
||||
video: {
|
||||
videoBytes: Buffer.from("mp4-bytes").toString("base64"),
|
||||
mimeType: "video/mp4",
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
});
|
||||
|
||||
const provider = buildGoogleVideoGenerationProvider();
|
||||
const result = await provider.generateVideo({
|
||||
provider: "google",
|
||||
model: "veo-3.1-fast-generate-preview",
|
||||
prompt: "A tiny robot watering a windowsill garden",
|
||||
cfg: {},
|
||||
aspectRatio: "16:9",
|
||||
resolution: "720P",
|
||||
durationSeconds: 3,
|
||||
audio: true,
|
||||
});
|
||||
|
||||
expect(generateVideosMock).toHaveBeenCalledTimes(1);
|
||||
const request = firstObjectArg(generateVideosMock);
|
||||
expect(request.model).toBe("veo-3.1-fast-generate-preview");
|
||||
expect(request.prompt).toBe("A tiny robot watering a windowsill garden");
|
||||
const config = recordField(request.config, "config");
|
||||
expect(config.durationSeconds).toBe(4);
|
||||
expect(config.aspectRatio).toBe("16:9");
|
||||
expect(config.resolution).toBe("720p");
|
||||
expect(config).not.toHaveProperty("generateAudio");
|
||||
expect(config).not.toHaveProperty("numberOfVideos");
|
||||
expect(result.videos).toHaveLength(1);
|
||||
expect(result.videos[0]?.mimeType).toBe("video/mp4");
|
||||
const clientOptions = firstObjectArg(createGoogleGenAIMock);
|
||||
expect(clientOptions.apiKey).toBe("google-key");
|
||||
const httpOptions = recordField(clientOptions.httpOptions, "httpOptions");
|
||||
expect(httpOptions).not.toHaveProperty("baseUrl");
|
||||
expect(httpOptions).not.toHaveProperty("apiVersion");
|
||||
});
|
||||
|
||||
it("rejects inline video bytes that exceed the configured media cap", async () => {
|
||||
vi.spyOn(providerAuthRuntime, "resolveApiKeyForProvider").mockResolvedValue({
|
||||
apiKey: "google-key",
|
||||
source: "env",
|
||||
mode: "api-key",
|
||||
});
|
||||
generateVideosMock.mockResolvedValue({
|
||||
done: true,
|
||||
response: {
|
||||
generatedVideos: [
|
||||
{
|
||||
video: {
|
||||
videoBytes: Buffer.from("too-large").toString("base64"),
|
||||
mimeType: "video/mp4",
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
});
|
||||
|
||||
const provider = buildGoogleVideoGenerationProvider();
|
||||
await expect(
|
||||
provider.generateVideo({
|
||||
provider: "google",
|
||||
model: "veo-3.1-fast-generate-preview",
|
||||
prompt: "A tiny robot watering a windowsill garden",
|
||||
cfg: { agents: { defaults: { mediaMaxMb: 0.000001 } } },
|
||||
durationSeconds: 3,
|
||||
}),
|
||||
).rejects.toThrow("Google generated video download exceeds 1 bytes");
|
||||
});
|
||||
|
||||
it("strips /v1beta suffix from configured baseUrl before passing to GoogleGenAI SDK", async () => {
|
||||
vi.spyOn(providerAuthRuntime, "resolveApiKeyForProvider").mockResolvedValue({
|
||||
apiKey: "google-key",
|
||||
source: "env",
|
||||
mode: "api-key",
|
||||
});
|
||||
generateVideosMock.mockResolvedValue({
|
||||
done: true,
|
||||
response: {
|
||||
generatedVideos: [
|
||||
{ video: { videoBytes: Buffer.from("mp4").toString("base64"), mimeType: "video/mp4" } },
|
||||
],
|
||||
},
|
||||
});
|
||||
|
||||
const provider = buildGoogleVideoGenerationProvider();
|
||||
await provider.generateVideo({
|
||||
provider: "google",
|
||||
model: "veo-3.1-fast-generate-preview",
|
||||
prompt: "A tiny robot watering a windowsill garden",
|
||||
cfg: {
|
||||
models: {
|
||||
providers: {
|
||||
google: { baseUrl: "https://generativelanguage.googleapis.com/v1beta", models: [] },
|
||||
},
|
||||
},
|
||||
},
|
||||
durationSeconds: 3,
|
||||
});
|
||||
|
||||
expect(firstGoogleClientHttpOptions().baseUrl).toBe(
|
||||
"https://generativelanguage.googleapis.com",
|
||||
);
|
||||
});
|
||||
|
||||
it("downloads MLDev direct video uri responses without routing through the Files API", async () => {
|
||||
vi.spyOn(providerAuthRuntime, "resolveApiKeyForProvider").mockResolvedValue({
|
||||
apiKey: "google-key",
|
||||
source: "env",
|
||||
mode: "api-key",
|
||||
});
|
||||
generateVideosMock.mockResolvedValue({
|
||||
done: true,
|
||||
response: {
|
||||
generatedVideos: [
|
||||
{
|
||||
video: {
|
||||
uri: "https://generativelanguage.googleapis.com/v1beta/files/generated-video:download?alt=media",
|
||||
mimeType: "video/mp4",
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
});
|
||||
const fetchMock = vi.fn(async () => {
|
||||
return new Response("direct-mp4", {
|
||||
status: 200,
|
||||
statusText: "OK",
|
||||
headers: { "content-type": "video/mp4" },
|
||||
});
|
||||
});
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
|
||||
const provider = buildGoogleVideoGenerationProvider();
|
||||
const result = await provider.generateVideo({
|
||||
provider: "google",
|
||||
model: "veo-3.1-fast-generate-preview",
|
||||
prompt: "A tiny robot watering a windowsill garden",
|
||||
cfg: {},
|
||||
durationSeconds: 3,
|
||||
});
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledTimes(1);
|
||||
const [[downloadUrl]] = fetchMock.mock.calls as unknown as [[string, RequestInit?]];
|
||||
expect(downloadUrl).toBe(
|
||||
"https://generativelanguage.googleapis.com/v1beta/files/generated-video:download?alt=media&key=google-key",
|
||||
);
|
||||
expect(downloadMock).not.toHaveBeenCalled();
|
||||
expect(result.videos[0]?.buffer).toEqual(Buffer.from("direct-mp4"));
|
||||
expect(result.videos[0]?.mimeType).toBe("video/mp4");
|
||||
});
|
||||
|
||||
it("rejects direct video uri downloads that exceed the configured media cap", async () => {
|
||||
vi.spyOn(providerAuthRuntime, "resolveApiKeyForProvider").mockResolvedValue({
|
||||
apiKey: "google-key",
|
||||
source: "env",
|
||||
mode: "api-key",
|
||||
});
|
||||
generateVideosMock.mockResolvedValue({
|
||||
done: true,
|
||||
response: {
|
||||
generatedVideos: [
|
||||
{
|
||||
video: {
|
||||
uri: "https://generativelanguage.googleapis.com/v1beta/files/generated-video:download?alt=media",
|
||||
mimeType: "video/mp4",
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
});
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(
|
||||
async () =>
|
||||
new Response("too-large", {
|
||||
status: 200,
|
||||
statusText: "OK",
|
||||
headers: { "content-type": "video/mp4" },
|
||||
}),
|
||||
),
|
||||
);
|
||||
|
||||
const provider = buildGoogleVideoGenerationProvider();
|
||||
await expect(
|
||||
provider.generateVideo({
|
||||
provider: "google",
|
||||
model: "veo-3.1-fast-generate-preview",
|
||||
prompt: "A tiny robot watering a windowsill garden",
|
||||
cfg: { agents: { defaults: { mediaMaxMb: 0.000001 } } },
|
||||
durationSeconds: 3,
|
||||
}),
|
||||
).rejects.toThrow("Google generated video download exceeds 1 bytes");
|
||||
});
|
||||
|
||||
it("downloads SDK file handles through the bounded REST media endpoint", async () => {
|
||||
vi.spyOn(providerAuthRuntime, "resolveApiKeyForProvider").mockResolvedValue({
|
||||
apiKey: "google-key",
|
||||
source: "env",
|
||||
mode: "api-key",
|
||||
});
|
||||
generateVideosMock.mockResolvedValue({
|
||||
done: true,
|
||||
response: {
|
||||
generatedVideos: [
|
||||
{
|
||||
video: {
|
||||
uri: "files/generated-video",
|
||||
mimeType: "video/mp4",
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
});
|
||||
const fetchMock = vi.fn(async () => {
|
||||
return new Response("sdk-video", {
|
||||
status: 200,
|
||||
statusText: "OK",
|
||||
headers: { "content-type": "video/mp4" },
|
||||
});
|
||||
});
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
|
||||
const provider = buildGoogleVideoGenerationProvider();
|
||||
const result = await provider.generateVideo({
|
||||
provider: "google",
|
||||
model: "veo-3.1-fast-generate-preview",
|
||||
prompt: "A tiny robot watering a windowsill garden",
|
||||
cfg: {},
|
||||
durationSeconds: 3,
|
||||
});
|
||||
|
||||
expect(fetchInputUrl(fetchMock, 0)).toBe(
|
||||
"https://generativelanguage.googleapis.com/v1beta/files/generated-video:download?alt=media&key=google-key",
|
||||
);
|
||||
expect(downloadMock).not.toHaveBeenCalled();
|
||||
expect(result.videos[0]?.buffer).toEqual(Buffer.from("sdk-video"));
|
||||
expect(result.videos[0]?.fileName).toBe("video-1.mp4");
|
||||
});
|
||||
|
||||
it("rejects SDK file-handle downloads that exceed the configured media cap", async () => {
|
||||
vi.spyOn(providerAuthRuntime, "resolveApiKeyForProvider").mockResolvedValue({
|
||||
apiKey: "google-key",
|
||||
source: "env",
|
||||
mode: "api-key",
|
||||
});
|
||||
generateVideosMock.mockResolvedValue({
|
||||
done: true,
|
||||
response: {
|
||||
generatedVideos: [
|
||||
{
|
||||
video: {
|
||||
uri: "files/generated-video",
|
||||
mimeType: "video/mp4",
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
});
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(
|
||||
async () =>
|
||||
new Response("too-large", {
|
||||
status: 200,
|
||||
statusText: "OK",
|
||||
headers: { "content-type": "video/mp4" },
|
||||
}),
|
||||
),
|
||||
);
|
||||
|
||||
const provider = buildGoogleVideoGenerationProvider();
|
||||
await expect(
|
||||
provider.generateVideo({
|
||||
provider: "google",
|
||||
model: "veo-3.1-fast-generate-preview",
|
||||
prompt: "A tiny robot watering a windowsill garden",
|
||||
cfg: { agents: { defaults: { mediaMaxMb: 0.000001 } } },
|
||||
durationSeconds: 3,
|
||||
}),
|
||||
).rejects.toThrow("Google generated video download exceeds 1 bytes");
|
||||
expect(downloadMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("falls back to REST predictLongRunning when text-only SDK video generation returns 404", async () => {
|
||||
vi.spyOn(providerAuthRuntime, "resolveApiKeyForProvider").mockResolvedValue({
|
||||
apiKey: "google-key",
|
||||
source: "env",
|
||||
mode: "api-key",
|
||||
});
|
||||
generateVideosMock.mockRejectedValue(Object.assign(new Error("sdk 404"), { status: 404 }));
|
||||
const fetchMock = vi
|
||||
.fn()
|
||||
.mockResolvedValueOnce(
|
||||
new Response(
|
||||
JSON.stringify({
|
||||
done: true,
|
||||
name: "operations/rest-123",
|
||||
response: {
|
||||
generateVideoResponse: {
|
||||
generatedSamples: [
|
||||
{
|
||||
video: {
|
||||
uri: "https://generativelanguage.googleapis.com/v1beta/files/rest-video:download?alt=media",
|
||||
mimeType: "video/mp4",
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
}),
|
||||
),
|
||||
)
|
||||
.mockResolvedValueOnce(
|
||||
new Response("rest-video", {
|
||||
status: 200,
|
||||
statusText: "OK",
|
||||
headers: { "content-type": "video/mp4" },
|
||||
}),
|
||||
);
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
|
||||
const provider = buildGoogleVideoGenerationProvider();
|
||||
const result = await provider.generateVideo({
|
||||
provider: "google",
|
||||
model: "google/models/veo-3.1-fast-generate-preview",
|
||||
prompt: "A tiny robot watering a windowsill garden",
|
||||
cfg: {},
|
||||
durationSeconds: 3,
|
||||
});
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledTimes(2);
|
||||
expect(fetchInputUrl(fetchMock, 0)).toBe(
|
||||
"https://generativelanguage.googleapis.com/v1beta/models/veo-3.1-fast-generate-preview:predictLongRunning",
|
||||
);
|
||||
expect(parseFetchJsonBody(fetchMock, 0)).toEqual({
|
||||
instances: [{ prompt: "A tiny robot watering a windowsill garden" }],
|
||||
parameters: { durationSeconds: 4 },
|
||||
});
|
||||
expect(fetchInputUrl(fetchMock, 1)).toBe(
|
||||
"https://generativelanguage.googleapis.com/v1beta/files/rest-video:download?alt=media&key=google-key",
|
||||
);
|
||||
expect(downloadMock).not.toHaveBeenCalled();
|
||||
expect(result.videos[0]?.buffer).toEqual(Buffer.from("rest-video"));
|
||||
});
|
||||
|
||||
it("bounds successful Google REST operation JSON bodies instead of buffering the whole response", async () => {
|
||||
vi.spyOn(providerAuthRuntime, "resolveApiKeyForProvider").mockResolvedValue({
|
||||
apiKey: "google-key",
|
||||
source: "env",
|
||||
mode: "api-key",
|
||||
});
|
||||
generateVideosMock.mockRejectedValue(Object.assign(new Error("sdk 404"), { status: 404 }));
|
||||
const streamed = oversizedJsonResponse({ chunkCount: 64, chunkSize: 1024 * 1024 });
|
||||
const fetchMock = vi.fn(async () => streamed.response);
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
|
||||
const provider = buildGoogleVideoGenerationProvider();
|
||||
await expect(
|
||||
provider.generateVideo({
|
||||
provider: "google",
|
||||
model: "veo-3.1-fast-generate-preview",
|
||||
prompt: "A tiny robot watering a windowsill garden",
|
||||
cfg: {},
|
||||
durationSeconds: 3,
|
||||
}),
|
||||
).rejects.toThrow("Google video operation response exceeds 16777216 bytes");
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledTimes(1);
|
||||
expect(streamed.getReadCount()).toBeLessThan(64);
|
||||
expect(streamed.wasCanceled()).toBe(true);
|
||||
});
|
||||
|
||||
it("retries transient Google REST poll failures with empty bodies", async () => {
|
||||
vi.useFakeTimers();
|
||||
vi.spyOn(providerAuthRuntime, "resolveApiKeyForProvider").mockResolvedValue({
|
||||
apiKey: "google-key",
|
||||
source: "env",
|
||||
mode: "api-key",
|
||||
});
|
||||
generateVideosMock.mockRejectedValue(Object.assign(new Error("sdk 404"), { status: 404 }));
|
||||
const fetchMock = vi
|
||||
.fn()
|
||||
.mockResolvedValueOnce(
|
||||
new Response(
|
||||
JSON.stringify({
|
||||
done: false,
|
||||
name: "operations/rest-123",
|
||||
}),
|
||||
),
|
||||
)
|
||||
.mockResolvedValueOnce(new Response("", { status: 503, statusText: "Service Unavailable" }))
|
||||
.mockResolvedValueOnce(
|
||||
new Response(
|
||||
JSON.stringify({
|
||||
done: true,
|
||||
name: "operations/rest-123",
|
||||
response: {
|
||||
generateVideoResponse: {
|
||||
generatedSamples: [
|
||||
{
|
||||
video: {
|
||||
uri: "https://generativelanguage.googleapis.com/v1beta/files/rest-video:download?alt=media",
|
||||
mimeType: "video/mp4",
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
}),
|
||||
),
|
||||
)
|
||||
.mockResolvedValueOnce(
|
||||
new Response("rest-video", {
|
||||
status: 200,
|
||||
statusText: "OK",
|
||||
headers: { "content-type": "video/mp4" },
|
||||
}),
|
||||
);
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
|
||||
const provider = buildGoogleVideoGenerationProvider();
|
||||
const resultPromise = provider.generateVideo({
|
||||
provider: "google",
|
||||
model: "veo-3.1-fast-generate-preview",
|
||||
prompt: "A tiny robot watering a windowsill garden",
|
||||
cfg: {},
|
||||
durationSeconds: 3,
|
||||
});
|
||||
await vi.advanceTimersByTimeAsync(10_250);
|
||||
const result = await resultPromise;
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledTimes(4);
|
||||
expect(fetchInputUrl(fetchMock, 1)).toBe(
|
||||
"https://generativelanguage.googleapis.com/v1beta/operations/rest-123",
|
||||
);
|
||||
expect(fetchInputUrl(fetchMock, 2)).toBe(
|
||||
"https://generativelanguage.googleapis.com/v1beta/operations/rest-123",
|
||||
);
|
||||
expect(result.videos[0]?.buffer).toEqual(Buffer.from("rest-video"));
|
||||
});
|
||||
|
||||
it("does not fall back to REST when SDK video generation with reference inputs returns 404", async () => {
|
||||
vi.spyOn(providerAuthRuntime, "resolveApiKeyForProvider").mockResolvedValue({
|
||||
apiKey: "google-key",
|
||||
source: "env",
|
||||
mode: "api-key",
|
||||
});
|
||||
generateVideosMock.mockRejectedValue(Object.assign(new Error("sdk 404"), { status: 404 }));
|
||||
const fetchMock = vi.fn();
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
|
||||
const provider = buildGoogleVideoGenerationProvider();
|
||||
await expect(
|
||||
provider.generateVideo({
|
||||
provider: "google",
|
||||
model: "veo-3.1-fast-generate-preview",
|
||||
prompt: "Animate this sketch",
|
||||
cfg: {},
|
||||
inputImages: [{ buffer: Buffer.from("img"), mimeType: "image/png" }],
|
||||
}),
|
||||
).rejects.toThrow("sdk 404");
|
||||
|
||||
expect(fetchMock).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("does NOT strip /v1beta when it appears mid-path (end-anchor proof)", async () => {
|
||||
vi.spyOn(providerAuthRuntime, "resolveApiKeyForProvider").mockResolvedValue({
|
||||
apiKey: "google-key",
|
||||
source: "env",
|
||||
mode: "api-key",
|
||||
});
|
||||
generateVideosMock.mockResolvedValue({
|
||||
done: true,
|
||||
response: {
|
||||
generatedVideos: [
|
||||
{ video: { videoBytes: Buffer.from("mp4").toString("base64"), mimeType: "video/mp4" } },
|
||||
],
|
||||
},
|
||||
});
|
||||
|
||||
const provider = buildGoogleVideoGenerationProvider();
|
||||
await provider.generateVideo({
|
||||
provider: "google",
|
||||
model: "veo-3.1-fast-generate-preview",
|
||||
prompt: "test",
|
||||
cfg: {
|
||||
models: {
|
||||
providers: { google: { baseUrl: "https://proxy.example.com/v1beta/route", models: [] } },
|
||||
},
|
||||
},
|
||||
durationSeconds: 3,
|
||||
});
|
||||
|
||||
expect(firstGoogleClientHttpOptions().baseUrl).toBe("https://proxy.example.com/v1beta/route");
|
||||
});
|
||||
|
||||
it("passes baseUrl unchanged when no /v1beta suffix is present", async () => {
|
||||
vi.spyOn(providerAuthRuntime, "resolveApiKeyForProvider").mockResolvedValue({
|
||||
apiKey: "google-key",
|
||||
source: "env",
|
||||
mode: "api-key",
|
||||
});
|
||||
generateVideosMock.mockResolvedValue({
|
||||
done: true,
|
||||
response: {
|
||||
generatedVideos: [
|
||||
{ video: { videoBytes: Buffer.from("mp4").toString("base64"), mimeType: "video/mp4" } },
|
||||
],
|
||||
},
|
||||
});
|
||||
|
||||
const provider = buildGoogleVideoGenerationProvider();
|
||||
await provider.generateVideo({
|
||||
provider: "google",
|
||||
model: "veo-3.1-fast-generate-preview",
|
||||
prompt: "test",
|
||||
cfg: {
|
||||
models: {
|
||||
providers: {
|
||||
google: { baseUrl: "https://generativelanguage.googleapis.com", models: [] },
|
||||
},
|
||||
},
|
||||
},
|
||||
durationSeconds: 3,
|
||||
});
|
||||
|
||||
expect(firstGoogleClientHttpOptions().baseUrl).toBe(
|
||||
"https://generativelanguage.googleapis.com",
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects mixed image and video inputs", async () => {
|
||||
vi.spyOn(providerAuthRuntime, "resolveApiKeyForProvider").mockResolvedValue({
|
||||
apiKey: "google-key",
|
||||
source: "env",
|
||||
mode: "api-key",
|
||||
});
|
||||
const provider = buildGoogleVideoGenerationProvider();
|
||||
|
||||
await expect(
|
||||
provider.generateVideo({
|
||||
provider: "google",
|
||||
model: "veo-3.1-fast-generate-preview",
|
||||
prompt: "Animate",
|
||||
cfg: {},
|
||||
inputImages: [{ buffer: Buffer.from("img"), mimeType: "image/png" }],
|
||||
inputVideos: [{ buffer: Buffer.from("vid"), mimeType: "video/mp4" }],
|
||||
}),
|
||||
).rejects.toThrow("Google video generation does not support image and video inputs together.");
|
||||
});
|
||||
|
||||
it("rounds unsupported durations to the nearest Veo value", async () => {
|
||||
vi.spyOn(providerAuthRuntime, "resolveApiKeyForProvider").mockResolvedValue({
|
||||
apiKey: "google-key",
|
||||
source: "env",
|
||||
mode: "api-key",
|
||||
});
|
||||
generateVideosMock.mockResolvedValue({
|
||||
done: true,
|
||||
response: {
|
||||
generatedVideos: [
|
||||
{
|
||||
video: {
|
||||
videoBytes: Buffer.from("mp4-bytes").toString("base64"),
|
||||
mimeType: "video/mp4",
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
});
|
||||
|
||||
const provider = buildGoogleVideoGenerationProvider();
|
||||
await provider.generateVideo({
|
||||
provider: "google",
|
||||
model: "veo-3.1-fast-generate-preview",
|
||||
prompt: "A tiny robot watering a windowsill garden",
|
||||
cfg: {},
|
||||
durationSeconds: 5,
|
||||
});
|
||||
|
||||
const request = firstObjectArg(generateVideosMock);
|
||||
const config = recordField(request.config, "config");
|
||||
expect(config.durationSeconds).toBe(6);
|
||||
});
|
||||
});
|
||||
625
extensions/google/video-generation-provider.ts
Normal file
625
extensions/google/video-generation-provider.ts
Normal file
@@ -0,0 +1,625 @@
|
||||
// Google provider module implements model/runtime integration.
|
||||
import { resolveApiKeyForProvider } from "openclaw/plugin-sdk/provider-auth-runtime";
|
||||
import {
|
||||
createProviderOperationDeadline,
|
||||
executeProviderOperationWithRetry,
|
||||
resolveProviderOperationTimeoutMs,
|
||||
waitProviderOperationPollInterval,
|
||||
} from "openclaw/plugin-sdk/provider-http";
|
||||
import { readResponseWithLimit } from "openclaw/plugin-sdk/response-limit-runtime";
|
||||
import { fetchWithSsrFGuard } from "openclaw/plugin-sdk/ssrf-runtime";
|
||||
import { normalizeOptionalString } from "openclaw/plugin-sdk/string-coerce-runtime";
|
||||
import type {
|
||||
GeneratedVideoAsset,
|
||||
VideoGenerationProvider,
|
||||
VideoGenerationRequest,
|
||||
} from "openclaw/plugin-sdk/video-generation";
|
||||
import { parseGeminiAuth, resolveGoogleGenerativeAiApiOrigin } from "./api.js";
|
||||
import {
|
||||
createGoogleVideoGenerationProviderMetadata,
|
||||
DEFAULT_GOOGLE_VIDEO_MODEL,
|
||||
GOOGLE_VIDEO_ALLOWED_DURATION_SECONDS,
|
||||
GOOGLE_VIDEO_MAX_DURATION_SECONDS,
|
||||
GOOGLE_VIDEO_MIN_DURATION_SECONDS,
|
||||
} from "./generation-provider-metadata.js";
|
||||
import { createGoogleGenAI, type GoogleGenAIClient } from "./google-genai-runtime.js";
|
||||
|
||||
const DEFAULT_TIMEOUT_MS = 180_000;
|
||||
const POLL_INTERVAL_MS = 10_000;
|
||||
const MAX_POLL_ATTEMPTS = 120;
|
||||
const DEFAULT_GENERATED_VIDEO_MAX_BYTES = 16 * 1024 * 1024;
|
||||
const GOOGLE_VIDEO_OPERATION_RESPONSE_MAX_BYTES = 16 * 1024 * 1024;
|
||||
const GOOGLE_VIDEO_EMPTY_RESULT_MESSAGE =
|
||||
"Google video generation response missing generated videos";
|
||||
|
||||
function resolveConfiguredGoogleVideoBaseUrl(req: VideoGenerationRequest): string | undefined {
|
||||
const configured = normalizeOptionalString(req.cfg?.models?.providers?.google?.baseUrl);
|
||||
return configured ? resolveGoogleGenerativeAiApiOrigin(configured) : undefined;
|
||||
}
|
||||
|
||||
function resolveGeneratedVideoMaxBytes(req: VideoGenerationRequest): number {
|
||||
const configured = req.cfg.agents?.defaults?.mediaMaxMb;
|
||||
if (typeof configured === "number" && Number.isFinite(configured) && configured > 0) {
|
||||
return Math.floor(configured * 1024 * 1024);
|
||||
}
|
||||
return DEFAULT_GENERATED_VIDEO_MAX_BYTES;
|
||||
}
|
||||
|
||||
function assertGeneratedVideoBufferWithinLimit(buffer: Buffer, maxBytes: number): void {
|
||||
if (buffer.length > maxBytes) {
|
||||
throw new Error(`Google generated video download exceeds ${maxBytes} bytes`);
|
||||
}
|
||||
}
|
||||
|
||||
function resolveGoogleVideoRestBaseUrl(configuredBaseUrl?: string): string {
|
||||
return `${configuredBaseUrl ?? "https://generativelanguage.googleapis.com"}/v1beta`;
|
||||
}
|
||||
|
||||
function resolveGoogleVideoRestModelPath(model: string): string {
|
||||
const trimmed = normalizeOptionalString(model) || DEFAULT_GOOGLE_VIDEO_MODEL;
|
||||
if (trimmed.startsWith("google/models/")) {
|
||||
return trimmed.slice("google/".length);
|
||||
}
|
||||
if (trimmed.startsWith("models/")) {
|
||||
return trimmed;
|
||||
}
|
||||
if (trimmed.startsWith("google/")) {
|
||||
return `models/${trimmed.slice("google/".length)}`;
|
||||
}
|
||||
return `models/${trimmed}`;
|
||||
}
|
||||
|
||||
function parseVideoSize(size: string | undefined): { width: number; height: number } | undefined {
|
||||
const trimmed = normalizeOptionalString(size);
|
||||
if (!trimmed) {
|
||||
return undefined;
|
||||
}
|
||||
const match = /^(\d+)x(\d+)$/u.exec(trimmed);
|
||||
if (!match) {
|
||||
return undefined;
|
||||
}
|
||||
const width = Number.parseInt(match[1] ?? "", 10);
|
||||
const height = Number.parseInt(match[2] ?? "", 10);
|
||||
if (!Number.isFinite(width) || !Number.isFinite(height)) {
|
||||
return undefined;
|
||||
}
|
||||
return { width, height };
|
||||
}
|
||||
|
||||
function resolveAspectRatio(params: {
|
||||
aspectRatio?: string;
|
||||
size?: string;
|
||||
}): "16:9" | "9:16" | undefined {
|
||||
const direct = normalizeOptionalString(params.aspectRatio);
|
||||
if (direct === "16:9" || direct === "9:16") {
|
||||
return direct;
|
||||
}
|
||||
const parsedSize = parseVideoSize(params.size);
|
||||
if (!parsedSize) {
|
||||
return undefined;
|
||||
}
|
||||
return parsedSize.width >= parsedSize.height ? "16:9" : "9:16";
|
||||
}
|
||||
|
||||
function resolveResolution(params: {
|
||||
resolution?: string;
|
||||
size?: string;
|
||||
}): "720p" | "1080p" | undefined {
|
||||
if (params.resolution === "720P") {
|
||||
return "720p";
|
||||
}
|
||||
if (params.resolution === "1080P") {
|
||||
return "1080p";
|
||||
}
|
||||
const parsedSize = parseVideoSize(params.size);
|
||||
if (!parsedSize) {
|
||||
return undefined;
|
||||
}
|
||||
const maxEdge = Math.max(parsedSize.width, parsedSize.height);
|
||||
return maxEdge >= 1920 ? "1080p" : maxEdge >= 1280 ? "720p" : undefined;
|
||||
}
|
||||
|
||||
function resolveDurationSeconds(durationSeconds: number | undefined): number | undefined {
|
||||
if (typeof durationSeconds !== "number" || !Number.isFinite(durationSeconds)) {
|
||||
return undefined;
|
||||
}
|
||||
const rounded = Math.min(
|
||||
GOOGLE_VIDEO_MAX_DURATION_SECONDS,
|
||||
Math.max(GOOGLE_VIDEO_MIN_DURATION_SECONDS, Math.round(durationSeconds)),
|
||||
);
|
||||
return GOOGLE_VIDEO_ALLOWED_DURATION_SECONDS.reduce((best, current) => {
|
||||
const currentDistance = Math.abs(current - rounded);
|
||||
const bestDistance = Math.abs(best - rounded);
|
||||
if (currentDistance < bestDistance) {
|
||||
return current;
|
||||
}
|
||||
if (currentDistance === bestDistance && current > best) {
|
||||
return current;
|
||||
}
|
||||
return best;
|
||||
});
|
||||
}
|
||||
|
||||
function resolveInputImage(req: VideoGenerationRequest) {
|
||||
const input = req.inputImages?.[0];
|
||||
if (!input?.buffer) {
|
||||
return undefined;
|
||||
}
|
||||
return {
|
||||
imageBytes: input.buffer.toString("base64"),
|
||||
mimeType: normalizeOptionalString(input.mimeType) || "image/png",
|
||||
};
|
||||
}
|
||||
|
||||
function resolveInputVideo(req: VideoGenerationRequest) {
|
||||
const input = req.inputVideos?.[0];
|
||||
if (!input?.buffer) {
|
||||
return undefined;
|
||||
}
|
||||
return {
|
||||
videoBytes: input.buffer.toString("base64"),
|
||||
mimeType: normalizeOptionalString(input.mimeType) || "video/mp4",
|
||||
};
|
||||
}
|
||||
|
||||
function resolveGoogleGeneratedVideoDownloadUrl(params: {
|
||||
uri: string | undefined;
|
||||
apiKey: string;
|
||||
configuredBaseUrl?: string;
|
||||
}): string | undefined {
|
||||
const trimmed = normalizeOptionalString(params.uri);
|
||||
if (!trimmed) {
|
||||
return undefined;
|
||||
}
|
||||
let url: URL;
|
||||
try {
|
||||
url = new URL(trimmed);
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
if (url.protocol !== "https:") {
|
||||
return undefined;
|
||||
}
|
||||
const allowedOrigins = new Set(["https://generativelanguage.googleapis.com"]);
|
||||
if (params.configuredBaseUrl) {
|
||||
try {
|
||||
const configuredOrigin = new URL(params.configuredBaseUrl).origin;
|
||||
if (configuredOrigin.startsWith("https://")) {
|
||||
allowedOrigins.add(configuredOrigin);
|
||||
}
|
||||
} catch {
|
||||
// Ignore invalid configured origins; resolveConfiguredGoogleVideoBaseUrl already normalizes.
|
||||
}
|
||||
}
|
||||
if (!allowedOrigins.has(url.origin)) {
|
||||
return undefined;
|
||||
}
|
||||
if (!url.searchParams.has("key")) {
|
||||
url.searchParams.set("key", params.apiKey);
|
||||
}
|
||||
return url.toString();
|
||||
}
|
||||
|
||||
function resolveGoogleGeneratedVideoFileDownloadUrl(params: {
|
||||
file: unknown;
|
||||
apiKey: string;
|
||||
configuredBaseUrl?: string;
|
||||
}): string | undefined {
|
||||
const resource = params.file as { name?: unknown; uri?: unknown } | undefined;
|
||||
const name = normalizeOptionalString(resource?.name) ?? normalizeOptionalString(resource?.uri);
|
||||
if (!name || !/^files\/[^/?#]+$/u.test(name)) {
|
||||
return undefined;
|
||||
}
|
||||
const baseUrl = resolveGoogleVideoRestBaseUrl(params.configuredBaseUrl);
|
||||
const url = new URL(`${baseUrl}/${name}:download`);
|
||||
url.searchParams.set("alt", "media");
|
||||
url.searchParams.set("key", params.apiKey);
|
||||
return url.toString();
|
||||
}
|
||||
|
||||
async function downloadGeneratedVideoFromUri(params: {
|
||||
uri: string | undefined;
|
||||
apiKey: string;
|
||||
configuredBaseUrl?: string;
|
||||
mimeType?: string;
|
||||
index: number;
|
||||
maxBytes: number;
|
||||
timeoutMs: number;
|
||||
}): Promise<GeneratedVideoAsset | undefined> {
|
||||
const downloadUrl = resolveGoogleGeneratedVideoDownloadUrl({
|
||||
uri: params.uri,
|
||||
apiKey: params.apiKey,
|
||||
configuredBaseUrl: params.configuredBaseUrl,
|
||||
});
|
||||
if (!downloadUrl) {
|
||||
return undefined;
|
||||
}
|
||||
return await executeProviderOperationWithRetry({
|
||||
provider: "google",
|
||||
stage: "download",
|
||||
operation: async () => {
|
||||
const { response, release } = await fetchWithSsrFGuard({
|
||||
url: downloadUrl,
|
||||
timeoutMs: params.timeoutMs,
|
||||
});
|
||||
try {
|
||||
if (!response.ok) {
|
||||
throw new Error(
|
||||
`Failed to download Google generated video: ${response.status} ${response.statusText}`,
|
||||
);
|
||||
}
|
||||
const buffer = await readResponseWithLimit(response, params.maxBytes, {
|
||||
chunkTimeoutMs: params.timeoutMs,
|
||||
onOverflow: ({ maxBytes }) =>
|
||||
new Error(`Google generated video download exceeds ${maxBytes} bytes`),
|
||||
onIdleTimeout: ({ chunkTimeoutMs }) =>
|
||||
new Error(`Google generated video download stalled after ${chunkTimeoutMs}ms`),
|
||||
});
|
||||
return {
|
||||
buffer,
|
||||
mimeType:
|
||||
normalizeOptionalString(response.headers.get("content-type")) ||
|
||||
normalizeOptionalString(params.mimeType) ||
|
||||
"video/mp4",
|
||||
fileName: `video-${params.index + 1}.mp4`,
|
||||
};
|
||||
} finally {
|
||||
await release();
|
||||
}
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function extractGoogleApiErrorCode(error: unknown): number | undefined {
|
||||
const status = (error as { status?: unknown } | undefined)?.status;
|
||||
if (typeof status === "number") {
|
||||
return status;
|
||||
}
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
try {
|
||||
const parsed = JSON.parse(message) as { code?: unknown; error?: { code?: unknown } };
|
||||
const code = typeof parsed.code === "number" ? parsed.code : parsed.error?.code;
|
||||
return typeof code === "number" ? code : undefined;
|
||||
} catch {
|
||||
return /\b404\b/u.test(message) ? 404 : undefined;
|
||||
}
|
||||
}
|
||||
|
||||
function extractGeneratedVideos(operation: unknown): Array<{ video?: unknown }> {
|
||||
const response = (operation as { response?: Record<string, unknown> }).response;
|
||||
const generatedVideos = response?.generatedVideos;
|
||||
if (Array.isArray(generatedVideos) && generatedVideos.length > 0) {
|
||||
return generatedVideos as Array<{ video?: unknown }>;
|
||||
}
|
||||
const generatedSamples = (response?.generateVideoResponse as { generatedSamples?: unknown })
|
||||
?.generatedSamples;
|
||||
return Array.isArray(generatedSamples) ? (generatedSamples as Array<{ video?: unknown }>) : [];
|
||||
}
|
||||
|
||||
async function requestGoogleVideoJson(params: {
|
||||
url: string;
|
||||
method: "GET" | "POST";
|
||||
headers: Record<string, string>;
|
||||
deadline: ReturnType<typeof createProviderOperationDeadline>;
|
||||
stage: "create" | "poll";
|
||||
body?: unknown;
|
||||
}): Promise<unknown> {
|
||||
function createHttpError(response: Response, detail: unknown): Error {
|
||||
const parts = [`HTTP ${response.status}`];
|
||||
const statusText = response.statusText.trim();
|
||||
if (statusText) {
|
||||
parts.push(statusText);
|
||||
}
|
||||
if (typeof detail === "string") {
|
||||
const trimmed = detail.trim();
|
||||
if (trimmed) {
|
||||
parts.push(trimmed);
|
||||
}
|
||||
} else if (detail && typeof detail === "object") {
|
||||
parts.push(JSON.stringify(detail));
|
||||
}
|
||||
const error = new Error(parts.join(": "));
|
||||
Object.assign(error, { status: response.status, statusCode: response.status });
|
||||
return error;
|
||||
}
|
||||
|
||||
return await executeProviderOperationWithRetry({
|
||||
provider: "google",
|
||||
stage: params.stage,
|
||||
operation: async () => {
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(
|
||||
() => {
|
||||
const error = new Error("request timed out");
|
||||
error.name = "TimeoutError";
|
||||
controller.abort(error);
|
||||
},
|
||||
resolveProviderOperationTimeoutMs({
|
||||
deadline: params.deadline,
|
||||
defaultTimeoutMs: DEFAULT_TIMEOUT_MS,
|
||||
}),
|
||||
);
|
||||
try {
|
||||
const { response, release } = await fetchWithSsrFGuard({
|
||||
url: params.url,
|
||||
init: {
|
||||
method: params.method,
|
||||
headers: params.headers,
|
||||
...(params.body === undefined ? {} : { body: JSON.stringify(params.body) }),
|
||||
},
|
||||
signal: controller.signal,
|
||||
});
|
||||
try {
|
||||
const buffer = await readResponseWithLimit(
|
||||
response,
|
||||
GOOGLE_VIDEO_OPERATION_RESPONSE_MAX_BYTES,
|
||||
{
|
||||
onOverflow: ({ maxBytes }) =>
|
||||
new Error(`Google video operation response exceeds ${maxBytes} bytes`),
|
||||
},
|
||||
);
|
||||
const text = new TextDecoder().decode(buffer);
|
||||
if (!response.ok) {
|
||||
let detail: unknown = text;
|
||||
if (text) {
|
||||
try {
|
||||
detail = JSON.parse(text) as unknown;
|
||||
} catch {
|
||||
detail = text;
|
||||
}
|
||||
}
|
||||
throw createHttpError(response, detail);
|
||||
}
|
||||
const payload = text ? (JSON.parse(text) as unknown) : {};
|
||||
return payload;
|
||||
} finally {
|
||||
await release();
|
||||
}
|
||||
} finally {
|
||||
clearTimeout(timeout);
|
||||
}
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
async function generateGoogleVideoViaRest(params: {
|
||||
baseUrl: string;
|
||||
headers: Record<string, string>;
|
||||
deadline: ReturnType<typeof createProviderOperationDeadline>;
|
||||
model: string;
|
||||
prompt: string;
|
||||
durationSeconds?: number;
|
||||
aspectRatio?: "16:9" | "9:16";
|
||||
resolution?: "720p" | "1080p";
|
||||
}): Promise<unknown> {
|
||||
let operation = await requestGoogleVideoJson({
|
||||
url: `${params.baseUrl}/${resolveGoogleVideoRestModelPath(params.model)}:predictLongRunning`,
|
||||
method: "POST",
|
||||
headers: params.headers,
|
||||
deadline: params.deadline,
|
||||
stage: "create",
|
||||
body: {
|
||||
instances: [{ prompt: params.prompt }],
|
||||
parameters: {
|
||||
...(typeof params.durationSeconds === "number"
|
||||
? { durationSeconds: params.durationSeconds }
|
||||
: {}),
|
||||
...(params.aspectRatio ? { aspectRatio: params.aspectRatio } : {}),
|
||||
...(params.resolution ? { resolution: params.resolution } : {}),
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
for (let attempt = 0; !((operation as { done?: boolean }).done ?? false); attempt += 1) {
|
||||
if (attempt >= MAX_POLL_ATTEMPTS) {
|
||||
throw new Error("Google video generation did not finish in time");
|
||||
}
|
||||
await waitProviderOperationPollInterval({
|
||||
deadline: params.deadline,
|
||||
pollIntervalMs: POLL_INTERVAL_MS,
|
||||
});
|
||||
const operationName = (operation as { name?: unknown }).name;
|
||||
if (typeof operationName !== "string" || !operationName) {
|
||||
throw new Error("Google video operation response missing name for polling");
|
||||
}
|
||||
operation = await requestGoogleVideoJson({
|
||||
url: `${params.baseUrl}/${operationName}`,
|
||||
method: "GET",
|
||||
headers: params.headers,
|
||||
deadline: params.deadline,
|
||||
stage: "poll",
|
||||
});
|
||||
}
|
||||
const error = (operation as { error?: unknown }).error;
|
||||
if (error) {
|
||||
throw new Error(JSON.stringify(error));
|
||||
}
|
||||
return operation;
|
||||
}
|
||||
|
||||
export function buildGoogleVideoGenerationProvider(): VideoGenerationProvider {
|
||||
return {
|
||||
...createGoogleVideoGenerationProviderMetadata(),
|
||||
async generateVideo(req) {
|
||||
if ((req.inputImages?.length ?? 0) > 1) {
|
||||
throw new Error("Google video generation supports at most one input image.");
|
||||
}
|
||||
if ((req.inputVideos?.length ?? 0) > 1) {
|
||||
throw new Error("Google video generation supports at most one input video.");
|
||||
}
|
||||
if ((req.inputImages?.length ?? 0) > 0 && (req.inputVideos?.length ?? 0) > 0) {
|
||||
throw new Error(
|
||||
"Google video generation does not support image and video inputs together.",
|
||||
);
|
||||
}
|
||||
const auth = await resolveApiKeyForProvider({
|
||||
provider: "google",
|
||||
cfg: req.cfg,
|
||||
agentDir: req.agentDir,
|
||||
store: req.authStore,
|
||||
});
|
||||
if (!auth.apiKey) {
|
||||
throw new Error("Google API key missing");
|
||||
}
|
||||
const apiKey = auth.apiKey;
|
||||
|
||||
const configuredBaseUrl = resolveConfiguredGoogleVideoBaseUrl(req);
|
||||
const restBaseUrl = resolveGoogleVideoRestBaseUrl(configuredBaseUrl);
|
||||
const authHeaders = parseGeminiAuth(apiKey).headers;
|
||||
const durationSeconds = resolveDurationSeconds(req.durationSeconds);
|
||||
const model = normalizeOptionalString(req.model) || DEFAULT_GOOGLE_VIDEO_MODEL;
|
||||
const aspectRatio = resolveAspectRatio({ aspectRatio: req.aspectRatio, size: req.size });
|
||||
const resolution = resolveResolution({ resolution: req.resolution, size: req.size });
|
||||
const hasReferenceInputs =
|
||||
(req.inputImages?.length ?? 0) > 0 || (req.inputVideos?.length ?? 0) > 0;
|
||||
const deadline = createProviderOperationDeadline({
|
||||
timeoutMs: req.timeoutMs,
|
||||
label: "Google video generation",
|
||||
});
|
||||
const client = createGoogleGenAI({
|
||||
apiKey,
|
||||
httpOptions: {
|
||||
...(configuredBaseUrl ? { baseUrl: configuredBaseUrl } : {}),
|
||||
timeout: resolveProviderOperationTimeoutMs({
|
||||
deadline,
|
||||
defaultTimeoutMs: DEFAULT_TIMEOUT_MS,
|
||||
}),
|
||||
},
|
||||
});
|
||||
let usedRestFallback = false;
|
||||
let operation;
|
||||
try {
|
||||
operation = await client.models.generateVideos({
|
||||
model,
|
||||
prompt: req.prompt,
|
||||
image: resolveInputImage(req),
|
||||
video: resolveInputVideo(req),
|
||||
config: {
|
||||
...(typeof durationSeconds === "number" ? { durationSeconds } : {}),
|
||||
...(aspectRatio ? { aspectRatio } : {}),
|
||||
...(resolution ? { resolution } : {}),
|
||||
},
|
||||
});
|
||||
} catch (error) {
|
||||
if (hasReferenceInputs || extractGoogleApiErrorCode(error) !== 404) {
|
||||
throw error;
|
||||
}
|
||||
usedRestFallback = true;
|
||||
operation = await generateGoogleVideoViaRest({
|
||||
baseUrl: restBaseUrl,
|
||||
headers: authHeaders,
|
||||
deadline,
|
||||
model,
|
||||
prompt: req.prompt,
|
||||
durationSeconds,
|
||||
aspectRatio,
|
||||
resolution,
|
||||
});
|
||||
}
|
||||
|
||||
if (!usedRestFallback) {
|
||||
let sdkOperation = operation as Awaited<
|
||||
ReturnType<GoogleGenAIClient["models"]["generateVideos"]>
|
||||
>;
|
||||
for (let attempt = 0; !(sdkOperation.done ?? false); attempt += 1) {
|
||||
if (attempt >= MAX_POLL_ATTEMPTS) {
|
||||
throw new Error("Google video generation did not finish in time");
|
||||
}
|
||||
await waitProviderOperationPollInterval({ deadline, pollIntervalMs: POLL_INTERVAL_MS });
|
||||
resolveProviderOperationTimeoutMs({ deadline, defaultTimeoutMs: DEFAULT_TIMEOUT_MS });
|
||||
sdkOperation = await executeProviderOperationWithRetry({
|
||||
provider: "google",
|
||||
stage: "poll",
|
||||
operation: () => client.operations.getVideosOperation({ operation: sdkOperation }),
|
||||
});
|
||||
}
|
||||
operation = sdkOperation;
|
||||
}
|
||||
const finalOperation = operation as { error?: unknown; name?: string };
|
||||
if (finalOperation.error) {
|
||||
throw new Error(JSON.stringify(finalOperation.error));
|
||||
}
|
||||
let generatedVideos = extractGeneratedVideos(operation);
|
||||
if (generatedVideos.length === 0 && !hasReferenceInputs && !usedRestFallback) {
|
||||
operation = await generateGoogleVideoViaRest({
|
||||
baseUrl: restBaseUrl,
|
||||
headers: authHeaders,
|
||||
deadline,
|
||||
model,
|
||||
prompt: req.prompt,
|
||||
durationSeconds,
|
||||
aspectRatio,
|
||||
resolution,
|
||||
});
|
||||
generatedVideos = extractGeneratedVideos(operation);
|
||||
}
|
||||
if (generatedVideos.length === 0) {
|
||||
throw new Error(GOOGLE_VIDEO_EMPTY_RESULT_MESSAGE);
|
||||
}
|
||||
const maxVideoBytes = resolveGeneratedVideoMaxBytes(req);
|
||||
const videos = await Promise.all(
|
||||
generatedVideos.map(async (entry, index) => {
|
||||
const inline = entry.video as
|
||||
| { videoBytes?: string; uri?: string; mimeType?: string }
|
||||
| undefined;
|
||||
if (inline?.videoBytes) {
|
||||
const buffer = Buffer.from(inline.videoBytes, "base64");
|
||||
assertGeneratedVideoBufferWithinLimit(buffer, maxVideoBytes);
|
||||
return {
|
||||
buffer,
|
||||
mimeType: normalizeOptionalString(inline.mimeType) || "video/mp4",
|
||||
fileName: `video-${index + 1}.mp4`,
|
||||
};
|
||||
}
|
||||
const directDownload = await downloadGeneratedVideoFromUri({
|
||||
uri: inline?.uri,
|
||||
apiKey,
|
||||
configuredBaseUrl,
|
||||
mimeType: inline?.mimeType,
|
||||
index,
|
||||
maxBytes: maxVideoBytes,
|
||||
timeoutMs: resolveProviderOperationTimeoutMs({
|
||||
deadline,
|
||||
defaultTimeoutMs: DEFAULT_TIMEOUT_MS,
|
||||
}),
|
||||
});
|
||||
if (directDownload) {
|
||||
return directDownload;
|
||||
}
|
||||
if (!inline) {
|
||||
throw new Error("Google generated video missing file handle");
|
||||
}
|
||||
const fileDownload = await downloadGeneratedVideoFromUri({
|
||||
uri: resolveGoogleGeneratedVideoFileDownloadUrl({
|
||||
file: inline,
|
||||
apiKey,
|
||||
configuredBaseUrl,
|
||||
}),
|
||||
apiKey,
|
||||
configuredBaseUrl,
|
||||
mimeType: inline.mimeType,
|
||||
index,
|
||||
maxBytes: maxVideoBytes,
|
||||
timeoutMs: resolveProviderOperationTimeoutMs({
|
||||
deadline,
|
||||
defaultTimeoutMs: DEFAULT_TIMEOUT_MS,
|
||||
}),
|
||||
});
|
||||
if (!fileDownload) {
|
||||
throw new Error("Google generated video missing bounded download URL");
|
||||
}
|
||||
return fileDownload;
|
||||
}),
|
||||
);
|
||||
return {
|
||||
videos,
|
||||
model,
|
||||
metadata: finalOperation.name
|
||||
? {
|
||||
operationName: finalOperation.name,
|
||||
}
|
||||
: undefined,
|
||||
};
|
||||
},
|
||||
};
|
||||
}
|
||||
2
extensions/google/web-search-contract-api.ts
Normal file
2
extensions/google/web-search-contract-api.ts
Normal file
@@ -0,0 +1,2 @@
|
||||
// Google API module exposes the plugin public contract.
|
||||
export { createGeminiWebSearchProvider } from "./src/gemini-web-search-provider.js";
|
||||
709
extensions/google/web-search-provider.test.ts
Normal file
709
extensions/google/web-search-provider.test.ts
Normal file
@@ -0,0 +1,709 @@
|
||||
// Google tests cover web search provider plugin behavior.
|
||||
import type { OpenClawConfig } from "openclaw/plugin-sdk/config-contracts";
|
||||
import { withEnv, withEnvAsync, withFetchPreconnect } from "openclaw/plugin-sdk/test-env";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { testing, createGeminiWebSearchProvider } from "./src/gemini-web-search-provider.js";
|
||||
|
||||
type TestModelProviderConfig = NonNullable<
|
||||
NonNullable<OpenClawConfig["models"]>["providers"]
|
||||
>[string];
|
||||
|
||||
function installGeminiFetch() {
|
||||
const mockFetch = vi.fn((_input?: RequestInfo | URL, _init?: RequestInit) =>
|
||||
Promise.resolve(
|
||||
new Response(
|
||||
JSON.stringify({
|
||||
candidates: [
|
||||
{
|
||||
content: { parts: [{ text: "Grounded answer" }] },
|
||||
groundingMetadata: {
|
||||
groundingChunks: [{ web: { uri: "https://example.com", title: "Example" } }],
|
||||
},
|
||||
},
|
||||
],
|
||||
}),
|
||||
),
|
||||
),
|
||||
);
|
||||
vi.stubGlobal("fetch", withFetchPreconnect(mockFetch));
|
||||
return mockFetch;
|
||||
}
|
||||
|
||||
function createGoogleModelProviderConfig(
|
||||
overrides: Partial<TestModelProviderConfig>,
|
||||
): TestModelProviderConfig {
|
||||
return {
|
||||
baseUrl: "https://generativelanguage.googleapis.com/v1beta/",
|
||||
models: [],
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
function requireFirstGeminiFetchCall(
|
||||
mockFetch: ReturnType<typeof installGeminiFetch>,
|
||||
): [RequestInfo | URL | undefined, RequestInit | undefined] {
|
||||
const [call] = mockFetch.mock.calls;
|
||||
if (!call) {
|
||||
throw new Error("expected Gemini web search fetch call");
|
||||
}
|
||||
return call as [RequestInfo | URL | undefined, RequestInit | undefined];
|
||||
}
|
||||
|
||||
function getFetchHeaders(mockFetch: ReturnType<typeof installGeminiFetch>): Record<string, string> {
|
||||
const [, init] = requireFirstGeminiFetchCall(mockFetch);
|
||||
return (init?.headers as Record<string, string> | undefined) ?? {};
|
||||
}
|
||||
|
||||
function getGeminiFetchUrl(mockFetch: ReturnType<typeof installGeminiFetch>): string | undefined {
|
||||
const [input] = requireFirstGeminiFetchCall(mockFetch);
|
||||
if (typeof input === "string") {
|
||||
return input;
|
||||
}
|
||||
if (input instanceof URL) {
|
||||
return input.toString();
|
||||
}
|
||||
return input?.url;
|
||||
}
|
||||
|
||||
function parseGeminiFetchBody(mockFetch: ReturnType<typeof installGeminiFetch>): {
|
||||
contents?: Array<{ parts?: Array<{ text?: string }> }>;
|
||||
tools?: Array<{ google_search?: { timeRangeFilter?: unknown } }>;
|
||||
} {
|
||||
const [, init] = requireFirstGeminiFetchCall(mockFetch);
|
||||
const body = init?.body;
|
||||
if (typeof body !== "string") {
|
||||
throw new Error("Expected Gemini fetch body string");
|
||||
}
|
||||
return JSON.parse(body) as {
|
||||
contents?: Array<{ parts?: Array<{ text?: string }> }>;
|
||||
tools?: Array<{ google_search?: { timeRangeFilter?: unknown } }>;
|
||||
};
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
vi.useRealTimers();
|
||||
vi.restoreAllMocks();
|
||||
vi.unstubAllGlobals();
|
||||
});
|
||||
|
||||
describe("google web search provider", () => {
|
||||
it("points missing-key users to fetch/browser alternatives", async () => {
|
||||
await withEnvAsync({ GEMINI_API_KEY: undefined }, async () => {
|
||||
const provider = createGeminiWebSearchProvider();
|
||||
const tool = provider.createTool({ config: {}, searchConfig: {} });
|
||||
if (!tool) {
|
||||
throw new Error("Expected tool definition");
|
||||
}
|
||||
|
||||
await expect(tool.execute({ query: "OpenClaw docs" })).resolves.toEqual({
|
||||
docs: "https://docs.openclaw.ai/tools/web",
|
||||
error: "missing_gemini_api_key",
|
||||
message:
|
||||
"web_search (gemini) needs an API key. Set GEMINI_API_KEY in the Gateway environment, configure plugins.entries.google.config.webSearch.apiKey, or reuse models.providers.google.apiKey. If you do not want to configure a search API key, use web_fetch for a specific URL or the browser tool for interactive pages.",
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("falls back to GEMINI_API_KEY from the environment", () => {
|
||||
withEnv({ GEMINI_API_KEY: "AIza-env-test" }, () => {
|
||||
expect(testing.resolveGeminiApiKey()).toBe("AIza-env-test");
|
||||
});
|
||||
});
|
||||
|
||||
it("prefers configured api keys over env fallbacks", () => {
|
||||
withEnv({ GEMINI_API_KEY: "AIza-env-test" }, () => {
|
||||
expect(testing.resolveGeminiApiKey({ apiKey: "AIza-configured-test" })).toBe(
|
||||
"AIza-configured-test",
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
it("uses provider api keys only after env fallbacks", () => {
|
||||
withEnv({ GEMINI_API_KEY: "AIza-env-test" }, () => {
|
||||
expect(testing.resolveGeminiApiKey({ providerApiKey: "AIza-provider-test" })).toBe(
|
||||
"AIza-env-test",
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
it("stores configured credentials at the canonical plugin config path", () => {
|
||||
const provider = createGeminiWebSearchProvider();
|
||||
const config = {} as OpenClawConfig;
|
||||
|
||||
provider.setConfiguredCredentialValue?.(config, "AIza-plugin-test");
|
||||
|
||||
expect(provider.credentialPath).toBe("plugins.entries.google.config.webSearch.apiKey");
|
||||
expect(provider.getConfiguredCredentialValue?.(config)).toBe("AIza-plugin-test");
|
||||
});
|
||||
|
||||
it("keeps model-provider fallback config runtime-only when Gemini config was injected", () => {
|
||||
const searchConfig = Object.defineProperty({ provider: "gemini" }, "gemini", {
|
||||
value: { apiKey: "AIza-plugin-test" },
|
||||
enumerable: false,
|
||||
configurable: true,
|
||||
writable: true,
|
||||
});
|
||||
|
||||
const merged = testing.withGoogleModelProviderFallbacks(searchConfig, {
|
||||
models: {
|
||||
providers: {
|
||||
google: createGoogleModelProviderConfig({
|
||||
apiKey: "AIza-provider-test",
|
||||
baseUrl: "https://generativelanguage.googleapis.com/proxy/v1beta/",
|
||||
}),
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
expect(merged?.gemini).toEqual({
|
||||
apiKey: "AIza-plugin-test",
|
||||
providerApiKey: "AIza-provider-test",
|
||||
providerBaseUrl: "https://generativelanguage.googleapis.com/proxy/v1beta/",
|
||||
});
|
||||
expect(Object.keys(merged ?? {})).toEqual(["provider"]);
|
||||
expect(Object.getOwnPropertyDescriptor(merged, "gemini")?.enumerable).toBe(false);
|
||||
});
|
||||
|
||||
it("defaults the Gemini web search model and trims explicit overrides", () => {
|
||||
expect(testing.resolveGeminiModel()).toBe("gemini-2.5-flash");
|
||||
expect(testing.resolveGeminiModel({ model: " gemini-2.5-pro " })).toBe("gemini-2.5-pro");
|
||||
});
|
||||
|
||||
it("routes Gemini web search through plugin webSearch.baseUrl", async () => {
|
||||
const mockFetch = installGeminiFetch();
|
||||
const provider = createGeminiWebSearchProvider();
|
||||
const tool = provider.createTool({
|
||||
config: {
|
||||
plugins: {
|
||||
entries: {
|
||||
google: {
|
||||
config: {
|
||||
webSearch: {
|
||||
apiKey: "AIza-plugin-test",
|
||||
baseUrl: "https://generativelanguage.googleapis.com/proxy/v1beta/",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
searchConfig: { provider: "gemini" },
|
||||
});
|
||||
|
||||
await tool?.execute({ query: "OpenClaw docs" });
|
||||
|
||||
expect(getGeminiFetchUrl(mockFetch)).toBe(
|
||||
"https://generativelanguage.googleapis.com/proxy/v1beta/models/gemini-2.5-flash:generateContent",
|
||||
);
|
||||
});
|
||||
|
||||
it("accepts Gemini success JSON with empty grounding metadata", async () => {
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
withFetchPreconnect(
|
||||
vi.fn(() =>
|
||||
Promise.resolve(
|
||||
new Response(
|
||||
JSON.stringify({
|
||||
candidates: [
|
||||
{
|
||||
content: { parts: [{ text: "Today's date is Sunday, June 7, 2026." }] },
|
||||
groundingMetadata: {},
|
||||
},
|
||||
],
|
||||
}),
|
||||
),
|
||||
),
|
||||
),
|
||||
),
|
||||
);
|
||||
const provider = createGeminiWebSearchProvider();
|
||||
const tool = provider.createTool({
|
||||
config: {
|
||||
plugins: {
|
||||
entries: {
|
||||
google: {
|
||||
config: {
|
||||
webSearch: {
|
||||
apiKey: "AIza-plugin-test",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
searchConfig: { provider: "gemini" },
|
||||
});
|
||||
|
||||
const result = await tool?.execute({ query: "current date today" });
|
||||
|
||||
expect(result).toMatchObject({
|
||||
citations: [],
|
||||
model: "gemini-2.5-flash",
|
||||
provider: "gemini",
|
||||
});
|
||||
expect(String(result?.content)).toContain("Today's date is Sunday, June 7, 2026.");
|
||||
});
|
||||
|
||||
it("reports malformed Gemini API JSON with a stable provider error", async () => {
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
withFetchPreconnect(vi.fn(() => Promise.resolve(new Response("{ nope")))),
|
||||
);
|
||||
const provider = createGeminiWebSearchProvider();
|
||||
const tool = provider.createTool({
|
||||
config: {
|
||||
plugins: {
|
||||
entries: {
|
||||
google: {
|
||||
config: {
|
||||
webSearch: {
|
||||
apiKey: "AIza-plugin-test",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
searchConfig: { provider: "gemini" },
|
||||
});
|
||||
|
||||
await expect(tool?.execute({ query: "OpenClaw docs" })).rejects.toThrow(
|
||||
"Gemini API error: malformed JSON response",
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects wrong-root Gemini success JSON with a stable provider error", async () => {
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
withFetchPreconnect(vi.fn(() => Promise.resolve(new Response(JSON.stringify([]))))),
|
||||
);
|
||||
const provider = createGeminiWebSearchProvider();
|
||||
const tool = provider.createTool({
|
||||
config: {
|
||||
plugins: {
|
||||
entries: {
|
||||
google: {
|
||||
config: {
|
||||
webSearch: {
|
||||
apiKey: "AIza-plugin-test",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
searchConfig: { provider: "gemini" },
|
||||
});
|
||||
|
||||
await expect(tool?.execute({ query: "OpenClaw docs" })).rejects.toThrow(
|
||||
"Gemini API error: malformed JSON response",
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects Gemini success JSON without candidate text", async () => {
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
withFetchPreconnect(
|
||||
vi.fn(() =>
|
||||
Promise.resolve(
|
||||
new Response(JSON.stringify({ candidates: [{ content: { parts: [] } }] })),
|
||||
),
|
||||
),
|
||||
),
|
||||
);
|
||||
const provider = createGeminiWebSearchProvider();
|
||||
const tool = provider.createTool({
|
||||
config: {
|
||||
plugins: {
|
||||
entries: {
|
||||
google: {
|
||||
config: {
|
||||
webSearch: {
|
||||
apiKey: "AIza-plugin-test",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
searchConfig: { provider: "gemini" },
|
||||
});
|
||||
|
||||
await expect(tool?.execute({ query: "OpenClaw docs" })).rejects.toThrow(
|
||||
"Gemini API error: malformed JSON response",
|
||||
);
|
||||
});
|
||||
|
||||
it("passes provider execution abort signals into the Gemini fetch", async () => {
|
||||
const mockFetch = installGeminiFetch();
|
||||
const controller = new AbortController();
|
||||
controller.abort();
|
||||
const provider = createGeminiWebSearchProvider();
|
||||
const tool = provider.createTool({
|
||||
config: {
|
||||
plugins: {
|
||||
entries: {
|
||||
google: {
|
||||
config: {
|
||||
webSearch: {
|
||||
apiKey: "AIza-plugin-test",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
searchConfig: { provider: "gemini" },
|
||||
});
|
||||
|
||||
await tool?.execute({ query: "OpenClaw docs" }, { signal: controller.signal });
|
||||
|
||||
const [, init] = requireFirstGeminiFetchCall(mockFetch);
|
||||
expect(init?.signal?.aborted).toBe(true);
|
||||
});
|
||||
|
||||
it("reuses the Google model provider key when no web search key or env key is set", async () => {
|
||||
await withEnvAsync({ GEMINI_API_KEY: undefined }, async () => {
|
||||
const mockFetch = installGeminiFetch();
|
||||
const provider = createGeminiWebSearchProvider();
|
||||
const tool = provider.createTool({
|
||||
config: {
|
||||
models: {
|
||||
providers: {
|
||||
google: createGoogleModelProviderConfig({
|
||||
apiKey: "AIza-provider-test",
|
||||
}),
|
||||
},
|
||||
},
|
||||
},
|
||||
searchConfig: { provider: "gemini" },
|
||||
});
|
||||
|
||||
await tool?.execute({ query: "OpenClaw provider key fallback" });
|
||||
|
||||
expect(getFetchHeaders(mockFetch)["x-goog-api-key"]).toBe("AIza-provider-test");
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps plugin web search keys ahead of env and provider keys", async () => {
|
||||
await withEnvAsync({ GEMINI_API_KEY: "AIza-env-test" }, async () => {
|
||||
const mockFetch = installGeminiFetch();
|
||||
const provider = createGeminiWebSearchProvider();
|
||||
const tool = provider.createTool({
|
||||
config: {
|
||||
plugins: {
|
||||
entries: {
|
||||
google: {
|
||||
config: {
|
||||
webSearch: {
|
||||
apiKey: "AIza-plugin-test",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
models: {
|
||||
providers: {
|
||||
google: createGoogleModelProviderConfig({
|
||||
apiKey: "AIza-provider-test",
|
||||
}),
|
||||
},
|
||||
},
|
||||
},
|
||||
searchConfig: { provider: "gemini" },
|
||||
});
|
||||
|
||||
await tool?.execute({ query: "OpenClaw plugin key precedence" });
|
||||
|
||||
expect(getFetchHeaders(mockFetch)["x-goog-api-key"]).toBe("AIza-plugin-test");
|
||||
});
|
||||
});
|
||||
|
||||
it("routes Gemini web search through provider-level google.baseUrl as a fallback", async () => {
|
||||
const mockFetch = installGeminiFetch();
|
||||
const provider = createGeminiWebSearchProvider();
|
||||
const tool = provider.createTool({
|
||||
config: {
|
||||
models: {
|
||||
providers: {
|
||||
google: createGoogleModelProviderConfig({
|
||||
apiKey: "AIza-provider-test",
|
||||
baseUrl: "https://generativelanguage.googleapis.com/provider/v1beta/",
|
||||
}),
|
||||
},
|
||||
},
|
||||
},
|
||||
searchConfig: { provider: "gemini" },
|
||||
});
|
||||
|
||||
await tool?.execute({ query: "OpenClaw provider baseUrl fallback" });
|
||||
|
||||
expect(getGeminiFetchUrl(mockFetch)).toBe(
|
||||
"https://generativelanguage.googleapis.com/provider/v1beta/models/gemini-2.5-flash:generateContent",
|
||||
);
|
||||
});
|
||||
|
||||
it("keeps plugin webSearch.baseUrl ahead of provider-level google.baseUrl", async () => {
|
||||
const mockFetch = installGeminiFetch();
|
||||
const provider = createGeminiWebSearchProvider();
|
||||
const tool = provider.createTool({
|
||||
config: {
|
||||
plugins: {
|
||||
entries: {
|
||||
google: {
|
||||
config: {
|
||||
webSearch: {
|
||||
apiKey: "AIza-plugin-test",
|
||||
baseUrl: "https://generativelanguage.googleapis.com/plugin/v1beta/",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
models: {
|
||||
providers: {
|
||||
google: createGoogleModelProviderConfig({
|
||||
baseUrl: "https://generativelanguage.googleapis.com/provider/v1beta/",
|
||||
}),
|
||||
},
|
||||
},
|
||||
},
|
||||
searchConfig: { provider: "gemini" },
|
||||
});
|
||||
|
||||
await tool?.execute({ query: "OpenClaw plugin baseUrl precedence" });
|
||||
|
||||
expect(getGeminiFetchUrl(mockFetch)).toBe(
|
||||
"https://generativelanguage.googleapis.com/plugin/v1beta/models/gemini-2.5-flash:generateContent",
|
||||
);
|
||||
});
|
||||
|
||||
it("uses a soft recency hint for Gemini day freshness shortcuts instead of a 24-hour range", async () => {
|
||||
const mockFetch = installGeminiFetch();
|
||||
const provider = createGeminiWebSearchProvider();
|
||||
const tool = provider.createTool({
|
||||
config: {
|
||||
plugins: {
|
||||
entries: {
|
||||
google: {
|
||||
config: {
|
||||
webSearch: {
|
||||
apiKey: "AIza-plugin-test",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
searchConfig: { provider: "gemini" },
|
||||
});
|
||||
|
||||
await tool?.execute({ query: "latest ai news timestamp precision", freshness: "pd" });
|
||||
|
||||
const body = parseGeminiFetchBody(mockFetch);
|
||||
expect(body.tools?.[0]?.google_search?.timeRangeFilter).toBeUndefined();
|
||||
expect(body.contents?.[0]?.parts?.[0]?.text).toContain(
|
||||
"Prioritize web sources published in the last 24 hours.",
|
||||
);
|
||||
});
|
||||
|
||||
it("preserves hard Gemini time ranges for wider freshness values", async () => {
|
||||
vi.useFakeTimers({ toFake: ["Date"] });
|
||||
vi.setSystemTime(new Date("2026-04-15T12:00:00.123Z"));
|
||||
const mockFetch = installGeminiFetch();
|
||||
const provider = createGeminiWebSearchProvider();
|
||||
const tool = provider.createTool({
|
||||
config: {
|
||||
plugins: {
|
||||
entries: {
|
||||
google: {
|
||||
config: {
|
||||
webSearch: {
|
||||
apiKey: "AIza-plugin-test",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
searchConfig: { provider: "gemini" },
|
||||
});
|
||||
|
||||
await tool?.execute({ query: "latest ai news timestamp precision", freshness: "week" });
|
||||
|
||||
const body = parseGeminiFetchBody(mockFetch);
|
||||
expect(body.contents?.[0]?.parts?.[0]?.text).toBe("latest ai news timestamp precision");
|
||||
expect(body.tools?.[0]?.google_search?.timeRangeFilter).toEqual({
|
||||
startTime: "2026-04-08T12:00:00Z",
|
||||
endTime: "2026-04-15T12:00:00Z",
|
||||
});
|
||||
});
|
||||
|
||||
it("partitions Gemini cache entries for soft day freshness, hard week freshness, and no freshness", async () => {
|
||||
vi.useFakeTimers({ toFake: ["Date"] });
|
||||
vi.setSystemTime(new Date("2026-04-15T12:00:00.123Z"));
|
||||
const mockFetch = installGeminiFetch();
|
||||
const provider = createGeminiWebSearchProvider();
|
||||
const tool = provider.createTool({
|
||||
config: {
|
||||
plugins: {
|
||||
entries: {
|
||||
google: {
|
||||
config: {
|
||||
webSearch: {
|
||||
apiKey: "AIza-plugin-test",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
searchConfig: { provider: "gemini" },
|
||||
});
|
||||
|
||||
await tool?.execute({ query: "same query cache partition", freshness: "day" });
|
||||
await tool?.execute({ query: "same query cache partition", freshness: "week" });
|
||||
await tool?.execute({ query: "same query cache partition" });
|
||||
|
||||
const postCalls = mockFetch.mock.calls.filter(([, init]) => typeof init?.body === "string");
|
||||
expect(postCalls).toHaveLength(3);
|
||||
const parsePostedBody = (call: (typeof postCalls)[number] | undefined) => {
|
||||
const body = call?.[1]?.body;
|
||||
if (typeof body !== "string") {
|
||||
throw new Error("Expected Gemini fetch body to be a string");
|
||||
}
|
||||
return JSON.parse(body) as {
|
||||
contents?: Array<{ parts?: Array<{ text?: string }> }>;
|
||||
tools?: Array<{ google_search?: { timeRangeFilter?: unknown } }>;
|
||||
};
|
||||
};
|
||||
const firstBody = parsePostedBody(postCalls[0]);
|
||||
const secondBody = parsePostedBody(postCalls[1]);
|
||||
const thirdBody = parsePostedBody(postCalls[2]);
|
||||
expect(firstBody.tools?.[0]?.google_search?.timeRangeFilter).toBeUndefined();
|
||||
expect(firstBody.contents?.[0]?.parts?.[0]?.text).toContain(
|
||||
"Prioritize web sources published in the last 24 hours.",
|
||||
);
|
||||
expect(secondBody.tools?.[0]?.google_search?.timeRangeFilter).toEqual({
|
||||
startTime: "2026-04-08T12:00:00Z",
|
||||
endTime: "2026-04-15T12:00:00Z",
|
||||
});
|
||||
expect(secondBody.contents?.[0]?.parts?.[0]?.text).toBe("same query cache partition");
|
||||
expect(thirdBody.tools?.[0]?.google_search?.timeRangeFilter).toBeUndefined();
|
||||
expect(thirdBody.contents?.[0]?.parts?.[0]?.text).toBe("same query cache partition");
|
||||
});
|
||||
|
||||
it("strips sub-second precision from date-range timestamps so Gemini accepts them", async () => {
|
||||
vi.useFakeTimers({ toFake: ["Date"] });
|
||||
// "now" with non-zero milliseconds. Without stripping, toISOString() emits
|
||||
// "2026-04-15T12:00:00.123Z", which Gemini's google_search.time_range_filter
|
||||
// rejects with "Granularity of nano is not supported".
|
||||
vi.setSystemTime(new Date("2026-04-15T12:00:00.123Z"));
|
||||
const mockFetch = installGeminiFetch();
|
||||
const provider = createGeminiWebSearchProvider();
|
||||
const tool = provider.createTool({
|
||||
config: {
|
||||
plugins: {
|
||||
entries: {
|
||||
google: {
|
||||
config: {
|
||||
webSearch: {
|
||||
apiKey: "AIza-plugin-test",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
searchConfig: { provider: "gemini" },
|
||||
});
|
||||
|
||||
await tool?.execute({ query: "latest ai news", date_after: "2026-04-01" });
|
||||
|
||||
const body = parseGeminiFetchBody(mockFetch);
|
||||
const filter = body.tools?.[0]?.google_search?.timeRangeFilter as
|
||||
| { startTime: string; endTime: string }
|
||||
| undefined;
|
||||
expect(filter?.startTime).not.toMatch(/\.\d+Z$/);
|
||||
expect(filter?.endTime).not.toMatch(/\.\d+Z$/);
|
||||
expect(filter).toEqual({
|
||||
startTime: "2026-04-01T00:00:00Z",
|
||||
endTime: "2026-04-15T12:00:00Z",
|
||||
});
|
||||
});
|
||||
|
||||
it("passes date ranges to Gemini Google Search grounding", async () => {
|
||||
const mockFetch = installGeminiFetch();
|
||||
const provider = createGeminiWebSearchProvider();
|
||||
const tool = provider.createTool({
|
||||
config: {
|
||||
plugins: {
|
||||
entries: {
|
||||
google: {
|
||||
config: {
|
||||
webSearch: {
|
||||
apiKey: "AIza-plugin-test",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
searchConfig: { provider: "gemini" },
|
||||
});
|
||||
|
||||
await tool?.execute({
|
||||
query: "OpenClaw release notes",
|
||||
date_after: "2026-04-01",
|
||||
date_before: "2026-04-30",
|
||||
});
|
||||
|
||||
const body = parseGeminiFetchBody(mockFetch);
|
||||
expect(body.tools?.[0]?.google_search?.timeRangeFilter).toEqual({
|
||||
startTime: "2026-04-01T00:00:00Z",
|
||||
endTime: "2026-05-01T00:00:00Z",
|
||||
});
|
||||
});
|
||||
|
||||
it("returns validation errors for invalid Gemini time filters before fetch", async () => {
|
||||
const mockFetch = installGeminiFetch();
|
||||
const provider = createGeminiWebSearchProvider();
|
||||
const tool = provider.createTool({
|
||||
config: {
|
||||
plugins: {
|
||||
entries: {
|
||||
google: {
|
||||
config: {
|
||||
webSearch: {
|
||||
apiKey: "AIza-plugin-test",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
searchConfig: { provider: "gemini" },
|
||||
});
|
||||
|
||||
await expect(
|
||||
tool?.execute({
|
||||
query: "OpenClaw release notes",
|
||||
freshness: "week",
|
||||
date_after: "2026-04-01",
|
||||
}),
|
||||
).resolves.toEqual({
|
||||
docs: "https://docs.openclaw.ai/tools/web",
|
||||
error: "conflicting_time_filters",
|
||||
message:
|
||||
"freshness and date_after/date_before cannot be used together. Use either freshness (day/week/month/year) or a date range (date_after/date_before), not both.",
|
||||
});
|
||||
expect(mockFetch).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("normalizes Gemini shorthand base URLs", () => {
|
||||
expect(
|
||||
testing.resolveGeminiBaseUrl({ baseUrl: "https://generativelanguage.googleapis.com" }),
|
||||
).toBe("https://generativelanguage.googleapis.com/v1beta");
|
||||
});
|
||||
});
|
||||
2
extensions/google/web-search-provider.ts
Normal file
2
extensions/google/web-search-provider.ts
Normal file
@@ -0,0 +1,2 @@
|
||||
// Google provider module implements model/runtime integration.
|
||||
export { createGeminiWebSearchProvider } from "./src/gemini-web-search-provider.js";
|
||||
Reference in New Issue
Block a user