Vendor OpenClaw source as Adolf fork baseline
Some checks failed
ClawSweeper Dispatch / dispatch (push) Has been cancelled
CodeQL / Security High (actions) (push) Has been cancelled
CodeQL / Security High (channel-runtime-boundary) (push) Has been cancelled
CodeQL / Security High (core-auth-secrets) (push) Has been cancelled
CodeQL / Security High (mcp-process-tool-boundary) (push) Has been cancelled
CodeQL / Security High (network-ssrf-boundary) (push) Has been cancelled
CodeQL / Security High (plugin-trust-boundary) (push) Has been cancelled
CodeQL / Security High (process-exec-boundary) (push) Has been cancelled
Docs Sync Publish Repo / sync-publish-repo (push) Has been cancelled
Docs / docs (push) Has been cancelled
OpenClaw Stable Main Closeout / Resolve stable release closeout inputs (push) Has been cancelled
OpenClaw Stable Main Closeout / Verify stable main closeout (push) Has been cancelled
Workflow Sanity / no-tabs (push) Has been cancelled
Workflow Sanity / actionlint (push) Has been cancelled
Workflow Sanity / generated-doc-baselines (push) Has been cancelled
CI / runner-admission (push) Has been cancelled
CI / preflight (push) Has been cancelled
CI / security-fast (push) Has been cancelled
CI / pnpm-store-warmup (push) Has been cancelled
CI / build-artifacts (push) Has been cancelled
CI / native-i18n (push) Has been cancelled
CI / ${{ matrix.check_name }} (push) Has been cancelled
CI / ${{ matrix.checkName }} (push) Has been cancelled
CI / checks-node-compat-node22 (push) Has been cancelled
CI / check-bundled-channel-config-metadata (push) Has been cancelled
CI / check-dependencies (push) Has been cancelled
CI / check-guards (push) Has been cancelled
CI / check-lint (push) Has been cancelled
CI / check-prod-types (push) Has been cancelled
CI / check-shrinkwrap (push) Has been cancelled
CI / check-test-types (push) Has been cancelled
CI / check-additional-boundaries-a (push) Has been cancelled
CI / check-additional-boundaries-bcd (push) Has been cancelled
CI / check-additional-extension-bundled (push) Has been cancelled
CI / check-additional-extension-channels (push) Has been cancelled
CI / check-additional-extension-package-boundary (push) Has been cancelled
CI / check-additional-runtime-topology-architecture (push) Has been cancelled
CI / check-session-accessor-boundary (push) Has been cancelled
CI / check-session-transcript-reader-boundary (push) Has been cancelled
CI / check-docs (push) Has been cancelled
CI / skills-python (push) Has been cancelled
CI / macos-swift (push) Has been cancelled
CI / ios-build (push) Has been cancelled
CI / ci-timings-summary (push) Has been cancelled
Native App Locale Refresh / Refresh native fa (push) Has been cancelled
Native App Locale Refresh / Refresh native fr (push) Has been cancelled
Native App Locale Refresh / Refresh native hi (push) Has been cancelled
Native App Locale Refresh / Refresh native id (push) Has been cancelled
Native App Locale Refresh / Refresh native it (push) Has been cancelled
Native App Locale Refresh / Refresh native ja-JP (push) Has been cancelled
Control UI Locale Refresh / plan (push) Has been cancelled
Control UI Locale Refresh / Refresh ${{ matrix.locale }} (push) Has been cancelled
Control UI Locale Refresh / Commit control UI locale refresh (push) Has been cancelled
Live Media Runner Image / Build live media runner image (push) Has been cancelled
Native App Locale Refresh / Refresh native ar (push) Has been cancelled
Native App Locale Refresh / Refresh native de (push) Has been cancelled
Native App Locale Refresh / Refresh native es (push) Has been cancelled
Native App Locale Refresh / Refresh native ko (push) Has been cancelled
Native App Locale Refresh / Refresh native nl (push) Has been cancelled
Native App Locale Refresh / Refresh native pl (push) Has been cancelled
Native App Locale Refresh / Refresh native pt-BR (push) Has been cancelled
Native App Locale Refresh / Refresh native ru (push) Has been cancelled
Native App Locale Refresh / Refresh native sv (push) Has been cancelled
Native App Locale Refresh / Refresh native th (push) Has been cancelled
Native App Locale Refresh / Refresh native tr (push) Has been cancelled
Native App Locale Refresh / Refresh native uk (push) Has been cancelled
Native App Locale Refresh / Refresh native vi (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-CN (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-TW (push) Has been cancelled
Native App Locale Refresh / Commit native locale refresh (push) Has been cancelled
Plugin Init Scaffold Validation / Validate provider scaffold (push) Has been cancelled
Plugin NPM Release / preview_plugins_npm (push) Has been cancelled
Plugin NPM Release / Validate release publish approval (push) Has been cancelled
Plugin NPM Release / preview_plugin_pack (push) Has been cancelled
Plugin NPM Release / publish_plugins_npm (push) Has been cancelled
Sandbox Common Smoke / sandbox-common-smoke (push) Has been cancelled
Website Installer Sync / static (push) Has been cancelled
Website Installer Sync / linux-docker (push) Has been cancelled
Website Installer Sync / macos-installer (push) Has been cancelled
Website Installer Sync / windows-installer (push) Has been cancelled
Website Installer Sync / sync-website (push) Has been cancelled

Adolf is a fork/vendored clone of github.com/openclaw/openclaw (v2026.6.11),
free to diverge. Tree copied sans upstream .git; upstream remote added for
future syncs. Node pinned to 24 (.nvmrc); engines already require >=22.19.
Preserves docs/ARCHITECTURE.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LeqyaxJF2nbRXJtae2kNB2
This commit is contained in:
2026-07-05 09:36:54 +00:00
parent 3216769225
commit bedb527145
21108 changed files with 6010766 additions and 0 deletions

View File

@@ -0,0 +1,49 @@
// Fal helper module supports http config behavior.
import type { AuthProfileStore, OpenClawConfig } from "openclaw/plugin-sdk/provider-auth";
import { resolveApiKeyForProvider } from "openclaw/plugin-sdk/provider-auth-runtime";
import {
resolveProviderHttpRequestConfig,
type ProviderRequestCapability,
} from "openclaw/plugin-sdk/provider-http";
import { normalizeOptionalString } from "openclaw/plugin-sdk/string-coerce-runtime";
const DEFAULT_FAL_BASE_URL = "https://fal.run";
type FalAuthenticatedRequest = {
cfg?: OpenClawConfig;
agentDir?: string;
authStore?: AuthProfileStore;
};
function resolveFalConfiguredBaseUrl(cfg?: OpenClawConfig): string | undefined {
return normalizeOptionalString(cfg?.models?.providers?.fal?.baseUrl);
}
export async function resolveFalHttpRequestConfig(params: {
req: FalAuthenticatedRequest;
baseUrl?: string;
capability: ProviderRequestCapability;
}): Promise<ReturnType<typeof resolveProviderHttpRequestConfig>> {
const auth = await resolveApiKeyForProvider({
provider: "fal",
cfg: params.req.cfg,
agentDir: params.req.agentDir,
store: params.req.authStore,
});
if (!auth.apiKey) {
throw new Error("fal API key missing");
}
return resolveProviderHttpRequestConfig({
baseUrl: params.baseUrl ?? resolveFalConfiguredBaseUrl(params.req.cfg),
defaultBaseUrl: DEFAULT_FAL_BASE_URL,
allowPrivateNetwork: false,
defaultHeaders: {
Authorization: `Key ${auth.apiKey}`,
"Content-Type": "application/json",
},
provider: "fal",
capability: params.capability,
transport: "http",
});
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,820 @@
// Fal provider module implements model/runtime integration.
import type {
GeneratedImageAsset,
ImageGenerationProvider,
ImageGenerationSourceImage,
} from "openclaw/plugin-sdk/image-generation";
import {
imageFileExtensionForMimeType,
toImageDataUrl,
} from "openclaw/plugin-sdk/image-generation";
import { isProviderApiKeyConfigured } from "openclaw/plugin-sdk/provider-auth";
import {
assertOkOrThrowHttpError,
assertOkOrThrowProviderError,
readProviderJsonResponse,
} from "openclaw/plugin-sdk/provider-http";
import { readResponseWithLimit } from "openclaw/plugin-sdk/response-limit-runtime";
import {
buildHostnameAllowlistPolicyFromSuffixAllowlist,
fetchWithSsrFGuard,
mergeSsrFPolicies,
type SsrFPolicy,
ssrfPolicyFromDangerouslyAllowPrivateNetwork,
} from "openclaw/plugin-sdk/ssrf-runtime";
import {
isRecord,
normalizeLowercaseStringOrEmpty,
normalizeOptionalString,
} from "openclaw/plugin-sdk/string-coerce-runtime";
import { resolveFalHttpRequestConfig } from "./http-config.js";
const DEFAULT_FAL_IMAGE_MODEL = "fal-ai/flux/dev";
const DEFAULT_FAL_EDIT_SUBPATH = "image-to-image";
const FAL_KREA_2_MODEL_PREFIX = "krea/v2/";
const FAL_KREA_2_MEDIUM_MODEL = "krea/v2/medium/text-to-image";
const FAL_KREA_2_LARGE_MODEL = "krea/v2/large/text-to-image";
const FAL_NANO_BANANA_MODEL = "fal-ai/nano-banana";
const FAL_NANO_BANANA_2_LITE_MODEL = "google/nano-banana-2-lite";
const FAL_GROK_IMAGINE_MODEL = "xai/grok-imagine-image";
const DEFAULT_OUTPUT_FORMAT = "png";
const GPT_IMAGE_EDIT_MAX_INPUT_IMAGES = 10;
const NANO_BANANA_LEGACY_EDIT_MAX_INPUT_IMAGES = 3;
const NANO_BANANA_EDIT_MAX_INPUT_IMAGES = 14;
const GROK_IMAGINE_EDIT_MAX_INPUT_IMAGES = 3;
const KREA_STYLE_REFERENCE_MAX_INPUT_IMAGES = 10;
const FAL_OUTPUT_FORMATS = ["png", "jpeg"] as const;
const FAL_SUPPORTED_SIZES = [
"1024x1024",
"1024x1536",
"1536x1024",
"1024x1792",
"1792x1024",
] as const;
const FAL_SUPPORTED_ASPECT_RATIOS = [
"1:1",
"2:3",
"3:2",
"2.35:1",
"3:4",
"4:3",
"4:5",
"5:4",
"9:16",
"16:9",
"21:9",
"4:1",
"1:4",
"8:1",
"1:8",
] as const;
const KREA_SUPPORTED_ASPECT_RATIOS = [
"1:1",
"4:3",
"3:2",
"16:9",
"2.35:1",
"4:5",
"2:3",
"9:16",
] as const;
const NANO_BANANA_LEGACY_SUPPORTED_ASPECT_RATIOS = [
"21:9",
"16:9",
"3:2",
"4:3",
"5:4",
"1:1",
"4:5",
"3:4",
"2:3",
"9:16",
] as const;
const NANO_BANANA_SUPPORTED_ASPECT_RATIOS = [
"21:9",
"16:9",
"3:2",
"4:3",
"5:4",
"1:1",
"4:5",
"3:4",
"2:3",
"9:16",
"4:1",
"1:4",
"8:1",
"1:8",
] as const;
const GROK_IMAGINE_SUPPORTED_ASPECT_RATIOS = [
"2:1",
"20:9",
"19.5:9",
"16:9",
"4:3",
"3:2",
"1:1",
"2:3",
"3:4",
"9:16",
"9:19.5",
"9:20",
"1:2",
] as const;
const GROK_IMAGINE_SUPPORTED_RESOLUTIONS: readonly ("1K" | "2K" | "4K")[] = ["1K", "2K"] as const;
const KREA_CREATIVITY_LEVELS = ["raw", "low", "medium", "high"] as const;
const FAL_IMAGE_MALFORMED_RESPONSE = "fal image generation response malformed";
const DEFAULT_GENERATED_IMAGE_MAX_BYTES = 6 * 1024 * 1024;
type FalImageSize = string | { width: number; height: number };
type FalEditEndpointSuffix = "edit" | "image-to-image";
type FalImageModelSchema = {
geometry: "image_size" | "native_aspect_ratio";
aspectRatios?: readonly string[];
resolutions?: readonly ("1K" | "2K" | "4K")[];
resolutionCase?: "lower";
referenceImages: "image_url" | "image_urls" | "image_style_references";
maxInputImages: number;
referenceLimitLabel: string;
referenceLimitNoun: "reference image" | "style reference";
appendEditPath: false | FalEditEndpointSuffix;
supportsCount: boolean;
supportsOutputFormat: boolean;
defaultBody?: Record<string, unknown>;
};
type FalNetworkPolicy = {
apiPolicy?: SsrFPolicy;
trustedDownloadHostSuffix?: string;
trustedDownloadPolicy?: SsrFPolicy;
};
let falFetchGuard = fetchWithSsrFGuard;
export function setFalFetchGuardForTesting(impl: typeof fetchWithSsrFGuard | null): void {
falFetchGuard = impl ?? fetchWithSsrFGuard;
}
function matchesTrustedHostSuffix(hostname: string, trustedSuffix: string): boolean {
const normalizedHost = normalizeLowercaseStringOrEmpty(hostname);
const normalizedSuffix = normalizeLowercaseStringOrEmpty(trustedSuffix);
return normalizedHost === normalizedSuffix || normalizedHost.endsWith(`.${normalizedSuffix}`);
}
function parseFalImageGenerationResponse(payload: unknown): {
images: Record<string, unknown>[];
prompt?: string;
} {
if (!isRecord(payload)) {
throw new Error(FAL_IMAGE_MALFORMED_RESPONSE);
}
const rawImages = payload.images;
if (rawImages === undefined || rawImages === null) {
return { images: [], prompt: normalizeOptionalString(payload.prompt) };
}
if (!Array.isArray(rawImages)) {
throw new Error(FAL_IMAGE_MALFORMED_RESPONSE);
}
const images: Record<string, unknown>[] = [];
for (const entry of rawImages) {
if (!isRecord(entry)) {
throw new Error(FAL_IMAGE_MALFORMED_RESPONSE);
}
images.push(entry);
}
return { images, prompt: normalizeOptionalString(payload.prompt) };
}
function resolveFalNetworkPolicy(params: {
baseUrl: string;
allowPrivateNetwork: boolean;
}): FalNetworkPolicy {
let parsedBaseUrl: URL;
try {
parsedBaseUrl = new URL(params.baseUrl);
} catch {
return {};
}
const hostSuffix = normalizeLowercaseStringOrEmpty(parsedBaseUrl.hostname);
if (!hostSuffix || !params.allowPrivateNetwork) {
return {};
}
const hostPolicy = buildHostnameAllowlistPolicyFromSuffixAllowlist([hostSuffix]);
const privateNetworkPolicy = ssrfPolicyFromDangerouslyAllowPrivateNetwork(true);
const trustedHostPolicy = mergeSsrFPolicies(hostPolicy, privateNetworkPolicy);
return {
apiPolicy: trustedHostPolicy,
trustedDownloadHostSuffix: hostSuffix,
trustedDownloadPolicy: trustedHostPolicy,
};
}
function ensureFalModelPath(model: string | undefined, hasInputImages: boolean): string {
const trimmed = model?.trim() || DEFAULT_FAL_IMAGE_MODEL;
const schema = resolveFalImageModelSchema(trimmed);
if (!hasInputImages || schema.appendEditPath === false) {
return trimmed;
}
if (
trimmed.endsWith(`/${schema.appendEditPath}`) ||
trimmed.endsWith("/edit") ||
trimmed.endsWith(`/${DEFAULT_FAL_EDIT_SUBPATH}`) ||
trimmed.includes("/image-to-image/")
) {
return trimmed;
}
return `${trimmed}/${schema.appendEditPath}`;
}
function resolveFalImageModelSchema(model: string): FalImageModelSchema {
if (model.startsWith(FAL_KREA_2_MODEL_PREFIX)) {
return {
geometry: "native_aspect_ratio",
aspectRatios: KREA_SUPPORTED_ASPECT_RATIOS,
referenceImages: "image_style_references",
maxInputImages: KREA_STYLE_REFERENCE_MAX_INPUT_IMAGES,
referenceLimitLabel: "fal Krea 2",
referenceLimitNoun: "style reference",
appendEditPath: false,
supportsCount: false,
supportsOutputFormat: false,
defaultBody: { creativity: "medium" },
};
}
if (model === FAL_NANO_BANANA_MODEL || model.startsWith(`${FAL_NANO_BANANA_MODEL}/`)) {
return {
geometry: "native_aspect_ratio",
aspectRatios: NANO_BANANA_LEGACY_SUPPORTED_ASPECT_RATIOS,
resolutions: [],
referenceImages: "image_urls",
maxInputImages: NANO_BANANA_LEGACY_EDIT_MAX_INPUT_IMAGES,
referenceLimitLabel: "fal Nano Banana",
referenceLimitNoun: "reference image",
appendEditPath: "edit",
supportsCount: true,
supportsOutputFormat: true,
};
}
if (model.startsWith("openai/gpt-image-") || model.startsWith(`${FAL_NANO_BANANA_MODEL}-`)) {
const isNanoBanana = model.startsWith(`${FAL_NANO_BANANA_MODEL}-`);
return {
geometry: isNanoBanana ? "native_aspect_ratio" : "image_size",
...(isNanoBanana ? { aspectRatios: NANO_BANANA_SUPPORTED_ASPECT_RATIOS } : {}),
referenceImages: "image_urls",
maxInputImages: isNanoBanana
? NANO_BANANA_EDIT_MAX_INPUT_IMAGES
: GPT_IMAGE_EDIT_MAX_INPUT_IMAGES,
referenceLimitLabel: isNanoBanana ? "fal Nano Banana 2" : "fal GPT Image edit",
referenceLimitNoun: "reference image",
appendEditPath: "edit",
supportsCount: true,
supportsOutputFormat: true,
};
}
// Nano Banana 2 Lite (Gemini 3.1 Flash Lite Image) uses /edit and the same
// aspect_ratio/image_urls contracts as Nano Banana 2. Its published schema
// has no resolution field, so explicit resolution overrides fail locally.
if (model.startsWith(FAL_NANO_BANANA_2_LITE_MODEL)) {
return {
geometry: "native_aspect_ratio",
aspectRatios: NANO_BANANA_SUPPORTED_ASPECT_RATIOS,
resolutions: [],
referenceImages: "image_urls",
maxInputImages: NANO_BANANA_EDIT_MAX_INPUT_IMAGES,
referenceLimitLabel: "fal Nano Banana 2 Lite",
referenceLimitNoun: "reference image",
appendEditPath: "edit",
supportsCount: true,
supportsOutputFormat: true,
};
}
// Grok Imagine (xAI) — text-to-image at /xai/grok-imagine-image, standard
// edits at /xai/grok-imagine-image/edit. Explicit quality/edit model paths
// remain unchanged. Accepts up to 3 reference images via image_urls.
if (model.startsWith(FAL_GROK_IMAGINE_MODEL)) {
return {
geometry: "native_aspect_ratio",
aspectRatios: GROK_IMAGINE_SUPPORTED_ASPECT_RATIOS,
resolutions: GROK_IMAGINE_SUPPORTED_RESOLUTIONS,
resolutionCase: "lower",
referenceImages: "image_urls",
maxInputImages: GROK_IMAGINE_EDIT_MAX_INPUT_IMAGES,
referenceLimitLabel: "fal Grok Imagine",
referenceLimitNoun: "reference image",
appendEditPath: "edit",
supportsCount: true,
supportsOutputFormat: true,
};
}
return {
geometry: "image_size",
referenceImages: "image_url",
maxInputImages: 1,
referenceLimitLabel: "fal flux image generation currently",
referenceLimitNoun: "reference image",
appendEditPath: "image-to-image",
supportsCount: true,
supportsOutputFormat: true,
};
}
function parseSize(raw: string | undefined): { width: number; height: number } | null {
const trimmed = raw?.trim();
if (!trimmed) {
return null;
}
const match = /^(\d{2,5})x(\d{2,5})$/iu.exec(trimmed);
if (!match) {
return null;
}
const width = Number.parseInt(match[1] ?? "", 10);
const height = Number.parseInt(match[2] ?? "", 10);
if (!Number.isFinite(width) || !Number.isFinite(height) || width <= 0 || height <= 0) {
return null;
}
return { width, height };
}
function mapResolutionToEdge(resolution: "1K" | "2K" | "4K" | undefined): number | undefined {
if (!resolution) {
return undefined;
}
return resolution === "4K" ? 4096 : resolution === "2K" ? 2048 : 1024;
}
function aspectRatioToEnum(aspectRatio: string | undefined): string | undefined {
const normalized = aspectRatio?.trim();
if (!normalized) {
return undefined;
}
if (normalized === "1:1") {
return "square_hd";
}
if (normalized === "4:3") {
return "landscape_4_3";
}
if (normalized === "3:4") {
return "portrait_4_3";
}
if (normalized === "16:9") {
return "landscape_16_9";
}
if (normalized === "9:16") {
return "portrait_16_9";
}
return undefined;
}
function parseAspectRatioParts(aspectRatio: string): { widthRatio: number; heightRatio: number } {
const match = /^(\d+(?:\.\d+)?):(\d+(?:\.\d+)?)$/u.exec(aspectRatio.trim());
if (!match) {
throw new Error(`Invalid fal aspect ratio: ${aspectRatio}`);
}
const widthRatio = Number.parseFloat(match[1] ?? "");
const heightRatio = Number.parseFloat(match[2] ?? "");
if (
!Number.isFinite(widthRatio) ||
!Number.isFinite(heightRatio) ||
widthRatio <= 0 ||
heightRatio <= 0
) {
throw new Error(`Invalid fal aspect ratio: ${aspectRatio}`);
}
return { widthRatio, heightRatio };
}
function aspectRatioToDimensions(
aspectRatio: string,
edge: number,
): { width: number; height: number } {
const { widthRatio, heightRatio } = parseAspectRatioParts(aspectRatio);
if (widthRatio >= heightRatio) {
return {
width: edge,
height: Math.max(1, Math.round((edge * heightRatio) / widthRatio)),
};
}
return {
width: Math.max(1, Math.round((edge * widthRatio) / heightRatio)),
height: edge,
};
}
function resolveFalImageSize(params: {
size?: string;
resolution?: "1K" | "2K" | "4K";
aspectRatio?: string;
hasInputImages: boolean;
}): FalImageSize | undefined {
const parsed = parseSize(params.size);
if (parsed) {
return parsed;
}
const normalizedAspectRatio = params.aspectRatio?.trim();
if (normalizedAspectRatio && params.hasInputImages) {
return (
aspectRatioToEnum(normalizedAspectRatio) ??
aspectRatioToDimensions(normalizedAspectRatio, 1024)
);
}
const edge = mapResolutionToEdge(params.resolution);
if (normalizedAspectRatio && edge) {
return aspectRatioToDimensions(normalizedAspectRatio, edge);
}
if (edge) {
return { width: edge, height: edge };
}
if (normalizedAspectRatio) {
return (
aspectRatioToEnum(normalizedAspectRatio) ??
aspectRatioToDimensions(normalizedAspectRatio, 1024)
);
}
return undefined;
}
function aspectRatioScore(aspectRatio: string, targetRatio: number): number {
const { widthRatio, heightRatio } = parseAspectRatioParts(aspectRatio);
return Math.abs(Math.log(widthRatio / heightRatio) - Math.log(targetRatio));
}
function resolveClosestFalAspectRatioForSize(
imageSize: FalImageSize | undefined,
aspectRatios: readonly string[],
): string | undefined {
if (!imageSize || typeof imageSize === "string") {
return undefined;
}
const targetRatio = imageSize.width / imageSize.height;
return aspectRatios.reduce<string | undefined>((best, candidate) => {
if (!best) {
return candidate;
}
return aspectRatioScore(candidate, targetRatio) < aspectRatioScore(best, targetRatio)
? candidate
: best;
}, undefined);
}
function resolveKreaCreativity(raw: string | undefined): string {
const normalized = normalizeLowercaseStringOrEmpty(raw);
return (KREA_CREATIVITY_LEVELS as readonly string[]).includes(normalized) ? normalized : "medium";
}
function resolveFalCreativityOption(providerOptions: Record<string, unknown> | undefined): string {
const falOptions = isRecord(providerOptions?.fal) ? providerOptions.fal : undefined;
return typeof falOptions?.creativity === "string" ? falOptions.creativity : "";
}
function resolveNativeFalAspectRatio(params: {
schema: FalImageModelSchema;
aspectRatio?: string;
imageSize?: FalImageSize;
}): string | undefined {
const requestedAspectRatio = params.aspectRatio?.trim();
const allowedAspectRatios = params.schema.aspectRatios;
if (requestedAspectRatio) {
if (allowedAspectRatios && !allowedAspectRatios.includes(requestedAspectRatio)) {
throw new Error(
`${params.schema.referenceLimitLabel} supports aspectRatio values: ${allowedAspectRatios.join(", ")}`,
);
}
return requestedAspectRatio;
}
if (allowedAspectRatios) {
return resolveClosestFalAspectRatioForSize(params.imageSize, allowedAspectRatios);
}
return undefined;
}
function applyFalImageGeometry(params: {
requestBody: Record<string, unknown>;
schema: FalImageModelSchema;
imageSize?: FalImageSize;
size?: string;
aspectRatio?: string;
resolution?: "1K" | "2K" | "4K";
hasInputImages: boolean;
}) {
if (params.schema.geometry === "native_aspect_ratio") {
if (params.resolution && params.schema.referenceImages === "image_style_references") {
throw new Error("fal Krea 2 supports aspectRatio but not resolution overrides");
}
const nativeAspectRatio = resolveNativeFalAspectRatio({
schema: params.schema,
aspectRatio: params.aspectRatio,
imageSize: params.size ? params.imageSize : undefined,
});
if (nativeAspectRatio) {
params.requestBody.aspect_ratio = nativeAspectRatio;
}
if (params.resolution && params.schema.referenceImages === "image_urls") {
// Schemas may opt in to resolution validation by declaring `resolutions`.
// - `resolutions: undefined` (default, e.g. Nano Banana 2): forward the
// uppercase value unchanged, matching legacy behaviour.
// - `resolutions: ["1K", "2K"]` with `resolutionCase: "lower"` (Grok
// Imagine): validate against the allowlist and lowercase before
// sending.
// - `resolutions: []` (Nano Banana 2 Lite): reject overrides when the
// published endpoint schema has no resolution field.
const allowedResolutions = params.schema.resolutions;
if (allowedResolutions === undefined) {
params.requestBody.resolution = params.resolution;
} else if (allowedResolutions.length === 0) {
throw new Error(
`${params.schema.referenceLimitLabel} does not support resolution overrides`,
);
} else if (!allowedResolutions.includes(params.resolution)) {
throw new Error(
`${params.schema.referenceLimitLabel} supports resolution values: ${allowedResolutions.join(", ")}`,
);
} else {
params.requestBody.resolution =
params.schema.resolutionCase === "lower"
? params.resolution.toLowerCase()
: params.resolution;
}
}
return;
}
if (params.imageSize !== undefined) {
params.requestBody.image_size = params.imageSize;
}
}
function applyFalReferenceImages(params: {
requestBody: Record<string, unknown>;
schema: FalImageModelSchema;
inputImages: ImageGenerationSourceImage[];
}) {
const encoded = params.inputImages.map((img) => toImageDataUrl(img));
if (params.schema.referenceImages === "image_urls") {
params.requestBody.image_urls = encoded;
return;
}
if (params.schema.referenceImages === "image_style_references") {
params.requestBody.image_style_references = encoded.map((imageUrl) => ({
image_url: imageUrl,
}));
return;
}
const [input] = encoded;
if (!input) {
throw new Error("fal image edit request missing reference image");
}
params.requestBody.image_url = input;
}
function formatFalReferenceLimitError(
schema: FalImageModelSchema,
inputImageCount: number,
): string {
const limit = schema.maxInputImages === 1 ? "one" : String(schema.maxInputImages);
const noun =
schema.maxInputImages === 1 ? schema.referenceLimitNoun : `${schema.referenceLimitNoun}s`;
return `${schema.referenceLimitLabel} supports at most ${limit} ${noun} (requested ${inputImageCount})`;
}
function resolveGeneratedImageMaxBytes(req: {
cfg: { agents?: { defaults?: { mediaMaxMb?: number } } };
}): number {
const configured = req.cfg.agents?.defaults?.mediaMaxMb;
if (typeof configured === "number" && Number.isFinite(configured) && configured > 0) {
return Math.floor(configured * 1024 * 1024);
}
return DEFAULT_GENERATED_IMAGE_MAX_BYTES;
}
async function fetchImageBuffer(
url: string,
networkPolicy?: FalNetworkPolicy,
maxBytes = DEFAULT_GENERATED_IMAGE_MAX_BYTES,
): Promise<{ buffer: Buffer; mimeType: string }> {
const downloadPolicy = (() => {
const trustedSuffix = networkPolicy?.trustedDownloadHostSuffix;
const trustedPolicy = networkPolicy?.trustedDownloadPolicy;
if (!trustedSuffix || !trustedPolicy) {
return undefined;
}
try {
const parsed = new URL(url);
return matchesTrustedHostSuffix(parsed.hostname, trustedSuffix) ? trustedPolicy : undefined;
} catch {
return undefined;
}
})();
const { response, release } = await falFetchGuard({
url,
policy: downloadPolicy,
auditContext: "fal-image-download",
});
try {
await assertOkOrThrowProviderError(response, "fal image download failed");
const mimeType = response.headers.get("content-type")?.trim() || "image/png";
return {
buffer: await readResponseWithLimit(response, maxBytes, {
onOverflow: ({ maxBytes: maxBytesLocal }) =>
new Error(`fal generated image download exceeds ${maxBytesLocal} bytes`),
}),
mimeType,
};
} finally {
await release();
}
}
export function buildFalImageGenerationProvider(): ImageGenerationProvider {
return {
id: "fal",
label: "fal",
defaultModel: DEFAULT_FAL_IMAGE_MODEL,
models: [
DEFAULT_FAL_IMAGE_MODEL,
`${DEFAULT_FAL_IMAGE_MODEL}/${DEFAULT_FAL_EDIT_SUBPATH}`,
FAL_KREA_2_MEDIUM_MODEL,
FAL_KREA_2_LARGE_MODEL,
],
isConfigured: ({ agentDir }) =>
isProviderApiKeyConfigured({
provider: "fal",
agentDir,
}),
capabilities: {
generate: {
maxCount: 4,
supportsSize: true,
supportsAspectRatio: true,
supportsResolution: true,
},
edit: {
enabled: true,
maxCount: 4,
maxInputImages: 1,
maxInputImagesByModel: {
[FAL_NANO_BANANA_MODEL]: NANO_BANANA_LEGACY_EDIT_MAX_INPUT_IMAGES,
[`${FAL_NANO_BANANA_MODEL}/edit`]: NANO_BANANA_LEGACY_EDIT_MAX_INPUT_IMAGES,
},
maxInputImagesByModelPrefix: {
"openai/gpt-image-": GPT_IMAGE_EDIT_MAX_INPUT_IMAGES,
[FAL_KREA_2_MODEL_PREFIX]: KREA_STYLE_REFERENCE_MAX_INPUT_IMAGES,
[`${FAL_NANO_BANANA_MODEL}-`]: NANO_BANANA_EDIT_MAX_INPUT_IMAGES,
[FAL_NANO_BANANA_2_LITE_MODEL]: NANO_BANANA_EDIT_MAX_INPUT_IMAGES,
[FAL_GROK_IMAGINE_MODEL]: GROK_IMAGINE_EDIT_MAX_INPUT_IMAGES,
},
supportsSize: true,
supportsAspectRatio: true,
supportsResolution: true,
},
geometry: {
sizes: [...FAL_SUPPORTED_SIZES],
sizesByModel: {
[FAL_KREA_2_MEDIUM_MODEL]: [],
[FAL_KREA_2_LARGE_MODEL]: [],
},
aspectRatios: [...FAL_SUPPORTED_ASPECT_RATIOS],
aspectRatiosByModel: {
[FAL_NANO_BANANA_MODEL]: [...NANO_BANANA_LEGACY_SUPPORTED_ASPECT_RATIOS],
[`${FAL_NANO_BANANA_MODEL}/edit`]: [...NANO_BANANA_LEGACY_SUPPORTED_ASPECT_RATIOS],
[FAL_NANO_BANANA_2_LITE_MODEL]: [...NANO_BANANA_SUPPORTED_ASPECT_RATIOS],
[`${FAL_NANO_BANANA_2_LITE_MODEL}/edit`]: [...NANO_BANANA_SUPPORTED_ASPECT_RATIOS],
[FAL_GROK_IMAGINE_MODEL]: [...GROK_IMAGINE_SUPPORTED_ASPECT_RATIOS],
[`${FAL_GROK_IMAGINE_MODEL}/edit`]: [...GROK_IMAGINE_SUPPORTED_ASPECT_RATIOS],
[`${FAL_GROK_IMAGINE_MODEL}/quality`]: [...GROK_IMAGINE_SUPPORTED_ASPECT_RATIOS],
[`${FAL_GROK_IMAGINE_MODEL}/quality/edit`]: [...GROK_IMAGINE_SUPPORTED_ASPECT_RATIOS],
},
resolutions: ["1K", "2K", "4K"],
resolutionsByModel: {
[FAL_KREA_2_MEDIUM_MODEL]: [],
[FAL_KREA_2_LARGE_MODEL]: [],
[FAL_NANO_BANANA_MODEL]: [],
[`${FAL_NANO_BANANA_MODEL}/edit`]: [],
[FAL_NANO_BANANA_2_LITE_MODEL]: [],
[`${FAL_NANO_BANANA_2_LITE_MODEL}/edit`]: [],
[FAL_GROK_IMAGINE_MODEL]: [...GROK_IMAGINE_SUPPORTED_RESOLUTIONS],
[`${FAL_GROK_IMAGINE_MODEL}/edit`]: [...GROK_IMAGINE_SUPPORTED_RESOLUTIONS],
[`${FAL_GROK_IMAGINE_MODEL}/quality`]: [...GROK_IMAGINE_SUPPORTED_RESOLUTIONS],
[`${FAL_GROK_IMAGINE_MODEL}/quality/edit`]: [...GROK_IMAGINE_SUPPORTED_RESOLUTIONS],
},
},
output: {
formats: [...FAL_OUTPUT_FORMATS],
},
},
async generateImage(req) {
const inputImageCount = req.inputImages?.length ?? 0;
const hasInputImages = inputImageCount > 0;
const requestedModel = req.model?.trim() || DEFAULT_FAL_IMAGE_MODEL;
const schema = resolveFalImageModelSchema(requestedModel);
const imageSize = resolveFalImageSize({
size: req.size,
resolution: req.resolution,
aspectRatio: req.aspectRatio,
hasInputImages,
});
const model = ensureFalModelPath(req.model, hasInputImages);
if (hasInputImages && inputImageCount > schema.maxInputImages) {
throw new Error(formatFalReferenceLimitError(schema, inputImageCount));
}
// Flux/custom edit endpoints use the singular image_url contract.
if (hasInputImages && schema.referenceImages === "image_url") {
if (req.aspectRatio) {
throw new Error("fal flux image edit endpoint does not support aspectRatio overrides");
}
}
if (!schema.supportsCount && (req.count ?? 1) > 1) {
throw new Error(`fal ${requestedModel} supports one output image per request`);
}
if (!schema.supportsOutputFormat && req.outputFormat) {
throw new Error(`fal ${requestedModel} does not support outputFormat overrides`);
}
const { baseUrl, allowPrivateNetwork, headers, dispatcherPolicy } =
await resolveFalHttpRequestConfig({ req, capability: "image" });
const networkPolicy = resolveFalNetworkPolicy({ baseUrl, allowPrivateNetwork });
const maxImageBytes = resolveGeneratedImageMaxBytes(req);
const requestBody: Record<string, unknown> = {
prompt: req.prompt,
...(schema.supportsCount ? { num_images: req.count ?? 1 } : {}),
...(schema.supportsOutputFormat
? { output_format: req.outputFormat ?? DEFAULT_OUTPUT_FORMAT }
: {}),
...schema.defaultBody,
};
if (schema.referenceImages === "image_style_references") {
requestBody.creativity = resolveKreaCreativity(
resolveFalCreativityOption(req.providerOptions),
);
}
applyFalImageGeometry({
requestBody,
schema,
imageSize,
size: req.size,
aspectRatio: req.aspectRatio,
resolution: req.resolution,
hasInputImages,
});
if (hasInputImages) {
applyFalReferenceImages({
requestBody,
schema,
inputImages: req.inputImages ?? [],
});
}
const { response, release } = await falFetchGuard({
url: `${baseUrl}/${model}`,
init: {
method: "POST",
headers,
body: JSON.stringify(requestBody),
},
timeoutMs: req.timeoutMs,
policy: networkPolicy.apiPolicy,
dispatcherPolicy,
auditContext: "fal-image-generate",
});
try {
await assertOkOrThrowHttpError(response, "fal image generation failed");
const payload = parseFalImageGenerationResponse(
await readProviderJsonResponse(response, "fal.image-generation"),
);
const images: GeneratedImageAsset[] = [];
let imageIndex = 0;
for (const entry of payload.images) {
const url = normalizeOptionalString(entry.url);
if (!url) {
throw new Error(FAL_IMAGE_MALFORMED_RESPONSE);
}
const downloaded = await fetchImageBuffer(url, networkPolicy, maxImageBytes);
imageIndex += 1;
images.push({
buffer: downloaded.buffer,
mimeType: downloaded.mimeType,
fileName: `image-${imageIndex}.${imageFileExtensionForMimeType(
downloaded.mimeType || normalizeOptionalString(entry.content_type),
)}`,
});
}
if (images.length === 0) {
throw new Error("fal image generation response missing image data");
}
return {
images,
model,
metadata: payload.prompt ? { prompt: payload.prompt } : undefined,
};
} finally {
await release();
}
},
};
}

20
extensions/fal/index.ts Normal file
View File

@@ -0,0 +1,20 @@
// Fal plugin entrypoint registers its OpenClaw integration.
import { definePluginEntry } from "openclaw/plugin-sdk/plugin-entry";
import { buildFalImageGenerationProvider } from "./image-generation-provider.js";
import { buildFalMusicGenerationProvider } from "./music-generation-provider.js";
import { createFalProvider } from "./provider-registration.js";
import { buildFalVideoGenerationProvider } from "./video-generation-provider.js";
const PROVIDER_ID = "fal";
export default definePluginEntry({
id: PROVIDER_ID,
name: "fal Provider",
description: "Bundled fal image, video, and music generation provider",
register(api) {
api.registerProvider(createFalProvider());
api.registerImageGenerationProvider(buildFalImageGenerationProvider());
api.registerMusicGenerationProvider(buildFalMusicGenerationProvider());
api.registerVideoGenerationProvider(buildFalVideoGenerationProvider());
},
});

View File

@@ -0,0 +1,235 @@
// Fal tests cover music generation provider plugin behavior.
import { expectExplicitMusicGenerationCapabilities } from "openclaw/plugin-sdk/provider-test-contracts";
import { afterEach, describe, expect, it, vi } from "vitest";
import { buildFalMusicGenerationProvider } from "./music-generation-provider.js";
const {
assertOkOrThrowHttpErrorMock,
postJsonRequestMock,
resolveApiKeyForProviderMock,
resolveProviderHttpRequestConfigMock,
} = vi.hoisted(() => ({
assertOkOrThrowHttpErrorMock: vi.fn(async () => {}),
postJsonRequestMock: vi.fn(),
resolveApiKeyForProviderMock: vi.fn(async () => ({
apiKey: "fal-key",
source: "env",
mode: "api-key",
})),
resolveProviderHttpRequestConfigMock: vi.fn((params: Record<string, unknown>) => ({
baseUrl: params.baseUrl ?? params.defaultBaseUrl,
allowPrivateNetwork: false,
headers: new Headers(params.defaultHeaders as HeadersInit | undefined),
dispatcherPolicy: undefined,
})),
}));
vi.mock("openclaw/plugin-sdk/provider-auth-runtime", () => ({
resolveApiKeyForProvider: resolveApiKeyForProviderMock,
}));
vi.mock("openclaw/plugin-sdk/provider-http", async (importOriginal) => {
const original = await importOriginal<typeof import("openclaw/plugin-sdk/provider-http")>();
return {
...original,
assertOkOrThrowHttpError: assertOkOrThrowHttpErrorMock,
postJsonRequest: postJsonRequestMock,
resolveProviderHttpRequestConfig: resolveProviderHttpRequestConfigMock,
};
});
function postRequest(): Record<string, unknown> {
const request = postJsonRequestMock.mock.calls[0]?.[0];
if (!request || typeof request !== "object" || Array.isArray(request)) {
throw new Error("expected fal music request");
}
return request as Record<string, unknown>;
}
function streamedAudioResponse(bytes: string): Response {
return new Response(
new ReadableStream({
start(controller) {
controller.enqueue(new TextEncoder().encode(bytes));
controller.close();
},
}),
{ headers: { "content-type": "audio/mpeg" } },
);
}
function falMusicJsonResponse(value: unknown) {
return {
response: Response.json(value),
release: vi.fn(async () => {}),
};
}
describe("fal music generation provider", () => {
afterEach(() => {
assertOkOrThrowHttpErrorMock.mockClear();
postJsonRequestMock.mockReset();
resolveApiKeyForProviderMock.mockClear();
resolveProviderHttpRequestConfigMock.mockClear();
vi.unstubAllGlobals();
});
it("declares explicit mode capabilities", () => {
expectExplicitMusicGenerationCapabilities(buildFalMusicGenerationProvider());
});
it("submits MiniMax music through fal and downloads the generated track", async () => {
postJsonRequestMock.mockResolvedValue(
falMusicJsonResponse({
audio: {
url: "https://v3b.fal.media/files/b/kangaroo/out.mp3",
content_type: "audio/mpeg",
file_name: "out.mp3",
},
}),
);
const fetchMock = vi.fn(
async () =>
new Response(Buffer.from("mp3-bytes"), {
headers: { "content-type": "application/octet-stream" },
}),
);
vi.stubGlobal("fetch", fetchMock);
const result = await buildFalMusicGenerationProvider().generateMusic({
provider: "fal",
model: "",
prompt: "city pop chorus",
cfg: {},
lyrics: "[Verse]\nNeon rain",
durationSeconds: 42,
format: "mp3",
});
expect(postRequest().url).toBe("https://fal.run/fal-ai/minimax-music/v2.6");
expect(postRequest().body).toEqual({
prompt: "city pop chorus",
lyrics: "[Verse]\nNeon rain",
duration: 42,
audio_setting: {
sample_rate: 44100,
bitrate: 256000,
format: "mp3",
},
});
expect(fetchMock).toHaveBeenCalledWith(
"https://v3b.fal.media/files/b/kangaroo/out.mp3",
expect.objectContaining({ method: "GET" }),
);
expect(result.model).toBe("fal-ai/minimax-music/v2.6");
expect(result.tracks[0]?.mimeType).toBe("audio/mpeg");
expect(result.tracks[0]?.buffer).toEqual(Buffer.from("mp3-bytes"));
expect(result.tracks[0]?.fileName).toBe("out.mp3");
expect(result.metadata?.audioUrl).toBe("https://v3b.fal.media/files/b/kangaroo/out.mp3");
});
it("rejects generated music downloads that exceed the configured media cap", async () => {
postJsonRequestMock.mockResolvedValue(
falMusicJsonResponse({
audio: {
url: "https://v3b.fal.media/files/b/out.mp3",
content_type: "audio/mpeg",
},
}),
);
vi.stubGlobal(
"fetch",
vi.fn(async () => streamedAudioResponse("too-large")),
);
await expect(
buildFalMusicGenerationProvider().generateMusic({
provider: "fal",
model: "fal-ai/minimax-music/v2.6",
prompt: "short track",
cfg: { agents: { defaults: { mediaMaxMb: 0.000001 } } },
}),
).rejects.toThrow("fal generated music download exceeds 1 bytes");
});
it("rejects MiniMax lyrics requests that also ask for instrumental output", async () => {
await expect(
buildFalMusicGenerationProvider().generateMusic({
provider: "fal",
model: "fal-ai/minimax-music/v2.6",
prompt: "city pop chorus",
cfg: {},
lyrics: "[Verse]\nNeon rain",
instrumental: true,
}),
).rejects.toThrow("fal MiniMax music generation cannot use lyrics when instrumental=true.");
expect(postJsonRequestMock).not.toHaveBeenCalled();
});
it("maps ACE-Step duration and instrumental controls", async () => {
postJsonRequestMock.mockResolvedValue(
falMusicJsonResponse({
audio: { url: "https://example.com/out.wav", content_type: "audio/wav" },
seed: 42,
tags: "lofi, chill",
}),
);
vi.stubGlobal(
"fetch",
vi.fn(
async () =>
new Response(Buffer.from("wav-bytes"), {
headers: { "content-type": "audio/wav" },
}),
),
);
await buildFalMusicGenerationProvider().generateMusic({
provider: "fal",
model: "fal-ai/ace-step/prompt-to-audio",
prompt: "lofi beach loop",
cfg: {},
instrumental: true,
durationSeconds: 30,
});
expect(postRequest().url).toBe("https://fal.run/fal-ai/ace-step/prompt-to-audio");
expect(postRequest().body).toEqual({
prompt: "lofi beach loop",
instrumental: true,
duration: 30,
});
});
it("maps Stable Audio duration controls", async () => {
postJsonRequestMock.mockResolvedValue(
falMusicJsonResponse({
audio: "https://example.com/stable.wav",
}),
);
vi.stubGlobal(
"fetch",
vi.fn(
async () =>
new Response(Buffer.from("wav-bytes"), {
headers: { "content-type": "audio/wav" },
}),
),
);
await buildFalMusicGenerationProvider().generateMusic({
provider: "fal",
model: "fal-ai/stable-audio-25/text-to-audio",
prompt: "orchestral hit",
cfg: {},
durationSeconds: 12,
});
expect(postRequest().url).toBe("https://fal.run/fal-ai/stable-audio-25/text-to-audio");
expect(postRequest().body).toEqual({
prompt: "orchestral hit",
seconds_total: 12,
});
});
});

View File

@@ -0,0 +1,204 @@
// Fal provider module implements model/runtime integration.
import {
downloadGeneratedMusicAsset,
extractGeneratedMusicFileCandidates,
type MusicGenerationProvider,
type MusicGenerationRequest,
} from "openclaw/plugin-sdk/music-generation";
import { isProviderApiKeyConfigured } from "openclaw/plugin-sdk/provider-auth";
import {
assertOkOrThrowHttpError,
postJsonRequest,
readProviderJsonResponse,
} from "openclaw/plugin-sdk/provider-http";
import { normalizeOptionalString } from "openclaw/plugin-sdk/string-coerce-runtime";
import { resolveFalHttpRequestConfig } from "./http-config.js";
const DEFAULT_FAL_MUSIC_MODEL = "fal-ai/minimax-music/v2.6";
const FAL_ACE_STEP_MODEL = "fal-ai/ace-step/prompt-to-audio";
const FAL_STABLE_AUDIO_MODEL = "fal-ai/stable-audio-25/text-to-audio";
const DEFAULT_TIMEOUT_MS = 180_000;
const DEFAULT_GENERATED_MUSIC_MAX_BYTES = 16 * 1024 * 1024;
const FAL_MUSIC_MODELS = [
DEFAULT_FAL_MUSIC_MODEL,
FAL_ACE_STEP_MODEL,
FAL_STABLE_AUDIO_MODEL,
] as const;
function resolveFalMusicModel(model: string | undefined): string {
return normalizeOptionalString(model) ?? DEFAULT_FAL_MUSIC_MODEL;
}
function resolveGeneratedMusicMaxBytes(req: MusicGenerationRequest): number {
const configured = req.cfg.agents?.defaults?.mediaMaxMb;
if (typeof configured === "number" && Number.isFinite(configured) && configured > 0) {
return Math.floor(configured * 1024 * 1024);
}
return DEFAULT_GENERATED_MUSIC_MAX_BYTES;
}
function buildFalMinimaxBody(req: MusicGenerationRequest): Record<string, unknown> {
const lyrics = normalizeOptionalString(req.lyrics);
if (lyrics && req.instrumental === true) {
throw new Error("fal MiniMax music generation cannot use lyrics when instrumental=true.");
}
return {
prompt: req.prompt,
...(lyrics ? { lyrics } : {}),
...(req.instrumental === true ? { is_instrumental: true } : {}),
...(!lyrics && req.instrumental !== true ? { lyrics_optimizer: true } : {}),
...(typeof req.durationSeconds === "number" ? { duration: req.durationSeconds } : {}),
audio_setting: {
sample_rate: 44_100,
bitrate: 256_000,
format: req.format ?? "mp3",
},
};
}
function buildFalAceStepBody(req: MusicGenerationRequest): Record<string, unknown> {
if (normalizeOptionalString(req.lyrics)) {
throw new Error("fal ACE-Step music generation does not support explicit lyrics.");
}
return {
prompt: req.prompt,
...(req.instrumental === true ? { instrumental: true } : {}),
...(typeof req.durationSeconds === "number" ? { duration: req.durationSeconds } : {}),
};
}
function buildFalStableAudioBody(req: MusicGenerationRequest): Record<string, unknown> {
if (normalizeOptionalString(req.lyrics)) {
throw new Error("fal Stable Audio music generation does not support explicit lyrics.");
}
if (req.instrumental === true) {
throw new Error("fal Stable Audio music generation does not support instrumental mode.");
}
return {
prompt: req.prompt,
...(typeof req.durationSeconds === "number" ? { seconds_total: req.durationSeconds } : {}),
};
}
function buildFalMusicRequestBody(
req: MusicGenerationRequest,
model: string,
): Record<string, unknown> {
if (model === FAL_ACE_STEP_MODEL) {
return buildFalAceStepBody(req);
}
if (model === FAL_STABLE_AUDIO_MODEL) {
return buildFalStableAudioBody(req);
}
return buildFalMinimaxBody(req);
}
function resolveFalMusicMetadata(payload: unknown): Record<string, unknown> | undefined {
if (!payload || typeof payload !== "object" || Array.isArray(payload)) {
return undefined;
}
const metadata: Record<string, unknown> = {};
for (const key of ["seed", "tags"]) {
const value = (payload as Record<string, unknown>)[key];
if (value !== undefined && value !== null) {
metadata[key] = value;
}
}
return Object.keys(metadata).length > 0 ? metadata : undefined;
}
export function buildFalMusicGenerationProvider(): MusicGenerationProvider {
return {
id: "fal",
label: "fal",
defaultModel: DEFAULT_FAL_MUSIC_MODEL,
models: [...FAL_MUSIC_MODELS],
isConfigured: ({ agentDir }) =>
isProviderApiKeyConfigured({
provider: "fal",
agentDir,
}),
capabilities: {
generate: {
maxTracks: 1,
maxDurationSeconds: 240,
supportsLyrics: true,
supportsLyricsByModel: {
[FAL_ACE_STEP_MODEL]: false,
[FAL_STABLE_AUDIO_MODEL]: false,
},
supportsInstrumental: true,
supportsInstrumentalByModel: {
[FAL_STABLE_AUDIO_MODEL]: false,
},
supportsDuration: true,
supportsFormat: true,
supportedFormats: ["mp3", "wav"],
supportedFormatsByModel: {
[DEFAULT_FAL_MUSIC_MODEL]: ["mp3"],
[FAL_ACE_STEP_MODEL]: ["wav"],
[FAL_STABLE_AUDIO_MODEL]: ["wav"],
},
},
edit: {
enabled: false,
},
},
async generateMusic(req) {
if ((req.inputImages?.length ?? 0) > 0) {
throw new Error("fal music generation does not support image reference inputs.");
}
const { baseUrl, allowPrivateNetwork, headers, dispatcherPolicy } =
await resolveFalHttpRequestConfig({ req, capability: "audio" });
const model = resolveFalMusicModel(req.model);
const { response, release } = await postJsonRequest({
url: `${baseUrl}/${model}`,
headers,
body: buildFalMusicRequestBody(req, model),
timeoutMs: req.timeoutMs ?? DEFAULT_TIMEOUT_MS,
fetchFn: fetch,
allowPrivateNetwork,
dispatcherPolicy,
});
try {
await assertOkOrThrowHttpError(response, "fal music generation failed");
const payload = await readProviderJsonResponse<unknown>(response, "fal music generation");
const [candidate] = extractGeneratedMusicFileCandidates(payload);
if (!candidate) {
throw new Error("fal music generation response missing audio output");
}
const track = await downloadGeneratedMusicAsset({
candidate,
timeoutMs: req.timeoutMs ?? DEFAULT_TIMEOUT_MS,
fetchFn: fetch,
provider: "fal",
requestFailedMessage: "fal generated music download failed",
maxBytes: resolveGeneratedMusicMaxBytes(req),
});
const lyrics =
typeof payload === "object" && payload && !Array.isArray(payload)
? normalizeOptionalString((payload as Record<string, unknown>).lyrics)
: undefined;
return {
tracks: [track],
model,
...(lyrics ? { lyrics: [lyrics] } : {}),
metadata: {
...resolveFalMusicMetadata(payload),
...(track.metadata?.url ? { audioUrl: track.metadata.url } : {}),
instrumental: req.instrumental === true,
...(req.format ? { requestedFormat: req.format } : {}),
...(typeof req.durationSeconds === "number"
? { requestedDurationSeconds: req.durationSeconds }
: {}),
},
};
} finally {
await release();
}
},
};
}

22
extensions/fal/onboard.ts Normal file
View File

@@ -0,0 +1,22 @@
// Fal setup module handles plugin onboarding behavior.
import type { OpenClawConfig } from "openclaw/plugin-sdk/provider-onboard";
export const FAL_DEFAULT_IMAGE_MODEL_REF = "fal/fal-ai/flux/dev";
export function applyFalConfig(cfg: OpenClawConfig): OpenClawConfig {
if (cfg.agents?.defaults?.imageGenerationModel) {
return cfg;
}
return {
...cfg,
agents: {
...cfg.agents,
defaults: {
...cfg.agents?.defaults,
imageGenerationModel: {
primary: FAL_DEFAULT_IMAGE_MODEL_REF,
},
},
},
};
}

View File

@@ -0,0 +1,47 @@
{
"id": "fal",
"activation": {
"onStartup": false
},
"enabledByDefault": true,
"providers": ["fal"],
"setup": {
"providers": [
{
"id": "fal",
"envVars": ["FAL_KEY", "FAL_API_KEY"]
}
]
},
"providerAuthChoices": [
{
"provider": "fal",
"method": "api-key",
"choiceId": "fal-api-key",
"choiceLabel": "fal API key",
"groupId": "fal",
"groupLabel": "fal",
"groupHint": "Image, video, and music generation",
"onboardingScopes": ["image-generation", "music-generation"],
"optionKey": "falApiKey",
"cliFlag": "--fal-api-key",
"cliOption": "--fal-api-key <key>",
"cliDescription": "fal API key"
}
],
"contracts": {
"imageGenerationProviders": ["fal"],
"musicGenerationProviders": ["fal"],
"videoGenerationProviders": ["fal"]
},
"videoGenerationProviderMetadata": {
"fal": {
"referenceAudioInputs": true
}
},
"configSchema": {
"type": "object",
"additionalProperties": false,
"properties": {}
}
}

View File

@@ -0,0 +1,15 @@
{
"name": "@openclaw/fal-provider",
"version": "2026.6.11",
"private": true,
"description": "OpenClaw fal provider plugin",
"type": "module",
"devDependencies": {
"@openclaw/plugin-sdk": "workspace:*"
},
"openclaw": {
"extensions": [
"./index.ts"
]
}
}

View File

@@ -0,0 +1,32 @@
// Fal API module exposes the plugin public contract.
import type { ProviderPlugin } from "openclaw/plugin-sdk/provider-model-shared";
const PROVIDER_ID = "fal";
const FAL_DEFAULT_IMAGE_MODEL_REF = "fal/fal-ai/flux/dev";
export function createFalProvider(): ProviderPlugin {
return {
id: PROVIDER_ID,
label: "fal",
docsPath: "/providers/models",
envVars: ["FAL_KEY"],
auth: [
{
id: "api-key",
kind: "api_key",
label: "fal API key",
hint: "Image, video, and music generation API key",
run: async () => ({ profiles: [], defaultModel: FAL_DEFAULT_IMAGE_MODEL_REF }),
wizard: {
choiceId: "fal-api-key",
choiceLabel: "fal API key",
choiceHint: "Image, video, and music generation API key",
groupId: "fal",
groupLabel: "fal",
groupHint: "Image, video, and music generation",
onboardingScopes: ["image-generation", "music-generation"],
},
},
],
};
}

View File

@@ -0,0 +1,39 @@
// Fal provider module implements model/runtime integration.
import { createProviderApiKeyAuthMethod } from "openclaw/plugin-sdk/provider-auth-api-key";
import type { ProviderPlugin } from "openclaw/plugin-sdk/provider-model-shared";
import { applyFalConfig, FAL_DEFAULT_IMAGE_MODEL_REF } from "./onboard.js";
const PROVIDER_ID = "fal";
export function createFalProvider(): ProviderPlugin {
return {
id: PROVIDER_ID,
label: "fal",
docsPath: "/providers/models",
envVars: ["FAL_KEY"],
auth: [
createProviderApiKeyAuthMethod({
providerId: PROVIDER_ID,
methodId: "api-key",
label: "fal API key",
hint: "Image, video, and music generation API key",
optionKey: "falApiKey",
flagName: "--fal-api-key",
envVar: "FAL_KEY",
promptMessage: "Enter fal API key",
defaultModel: FAL_DEFAULT_IMAGE_MODEL_REF,
expectedProviders: ["fal"],
applyConfig: (cfg) => applyFalConfig(cfg),
wizard: {
choiceId: "fal-api-key",
choiceLabel: "fal API key",
choiceHint: "Image, video, and music generation API key",
groupId: "fal",
groupLabel: "fal",
groupHint: "Image, video, and music generation",
onboardingScopes: ["image-generation", "music-generation"],
},
}),
],
};
}

View File

@@ -0,0 +1,4 @@
// Fal API module exposes the plugin public contract.
export { buildFalImageGenerationProvider } from "./image-generation-provider.js";
export { buildFalMusicGenerationProvider } from "./music-generation-provider.js";
export { buildFalVideoGenerationProvider } from "./video-generation-provider.js";

View File

@@ -0,0 +1,16 @@
{
"extends": "../tsconfig.package-boundary.base.json",
"compilerOptions": {
"rootDir": "."
},
"include": ["./*.ts", "./src/**/*.ts"],
"exclude": [
"./**/*.test.ts",
"./dist/**",
"./node_modules/**",
"./src/test-support/**",
"./src/**/*test-helpers.ts",
"./src/**/*test-harness.ts",
"./src/**/*test-support.ts"
]
}

View File

@@ -0,0 +1,687 @@
// Fal tests cover video generation provider plugin behavior.
import { MAX_TIMER_TIMEOUT_MS } from "openclaw/plugin-sdk/number-runtime";
import * as providerAuth from "openclaw/plugin-sdk/provider-auth-runtime";
import * as providerHttp from "openclaw/plugin-sdk/provider-http";
import { expectExplicitVideoGenerationCapabilities } from "openclaw/plugin-sdk/provider-test-contracts";
import { afterEach, describe, expect, it, vi } from "vitest";
import {
setFalVideoFetchGuardForTesting,
buildFalVideoGenerationProvider,
} from "./video-generation-provider.js";
function createMockRequestConfig() {
return {} as ReturnType<typeof providerHttp.resolveProviderHttpRequestConfig>["requestConfig"];
}
describe("fal video generation provider", () => {
const fetchGuardMock = vi.fn();
function mockFalProviderRuntime() {
vi.spyOn(providerAuth, "resolveApiKeyForProvider").mockResolvedValue({
apiKey: "fal-key",
source: "env",
mode: "api-key",
});
vi.spyOn(providerHttp, "resolveProviderHttpRequestConfig").mockReturnValue({
baseUrl: "https://fal.run",
allowPrivateNetwork: false,
headers: new Headers({
Authorization: "Key fal-key",
"Content-Type": "application/json",
}),
dispatcherPolicy: undefined,
requestConfig: createMockRequestConfig(),
});
vi.spyOn(providerHttp, "assertOkOrThrowHttpError").mockResolvedValue(undefined);
setFalVideoFetchGuardForTesting(fetchGuardMock as never);
}
function releasedJson(value: unknown) {
return {
response: Response.json(value),
release: vi.fn(async () => {}),
};
}
function releasedVideo(params: { contentType: string; bytes: string }) {
return {
response: new Response(Buffer.from(params.bytes), {
status: 200,
headers: { "content-type": params.contentType },
}),
release: vi.fn(async () => {}),
};
}
function mockCompletedFalVideoJob(params: {
requestId: string;
statusUrl: string;
responseUrl: string;
videoUrl: string;
bytes: string;
contentType?: string;
responseExtras?: Record<string, unknown>;
}) {
fetchGuardMock
.mockResolvedValueOnce(
releasedJson({
request_id: params.requestId,
status_url: params.statusUrl,
response_url: params.responseUrl,
}),
)
.mockResolvedValueOnce(releasedJson({ status: "COMPLETED" }))
.mockResolvedValueOnce(
releasedJson({
status: "COMPLETED",
response: {
video: { url: params.videoUrl },
...params.responseExtras,
},
}),
)
.mockResolvedValueOnce(
releasedVideo({ contentType: params.contentType ?? "video/mp4", bytes: params.bytes }),
);
}
function requireFetchGuardCall(callNumber: number): { init?: RequestInit; url?: string } {
const call = fetchGuardMock.mock.calls[callNumber - 1];
if (!call) {
throw new Error(`expected fal fetch guard call ${callNumber}`);
}
const [request] = call;
if (!request || typeof request !== "object" || Array.isArray(request)) {
throw new Error(`expected fal fetch guard request ${callNumber}`);
}
return request as { init?: RequestInit; url?: string };
}
function getSubmitBody(): Record<string, unknown> {
const body = requireFetchGuardCall(1).init?.body;
if (typeof body !== "string") {
throw new Error("expected fal submit JSON body");
}
return JSON.parse(body) as Record<string, unknown>;
}
function fetchGuardUrl(callNumber: number): string | undefined {
return requireFetchGuardCall(callNumber).url;
}
afterEach(() => {
vi.restoreAllMocks();
fetchGuardMock.mockReset();
setFalVideoFetchGuardForTesting(null);
});
it("declares explicit mode capabilities", () => {
const provider = buildFalVideoGenerationProvider();
expectExplicitVideoGenerationCapabilities(provider);
expect(provider.capabilities.imageToVideo?.maxInputImages).toBe(1);
expect(
provider.capabilities.imageToVideo?.maxInputImagesByModel?.[
"bytedance/seedance-2.0/fast/reference-to-video"
],
).toBe(9);
expect(provider.capabilities.videoToVideo?.maxInputVideos).toBe(0);
expect(
Object.keys(provider.capabilities.videoToVideo?.supportedDurationSecondsByModel ?? {}),
).toEqual([
"bytedance/seedance-2.0/fast/reference-to-video",
"bytedance/seedance-2.0/reference-to-video",
]);
});
it("submits fal video jobs through the queue API and downloads the completed result", async () => {
mockFalProviderRuntime();
mockCompletedFalVideoJob({
requestId: "req-123",
statusUrl: "https://queue.fal.run/fal-ai/minimax/requests/req-123/status",
responseUrl: "https://queue.fal.run/fal-ai/minimax/requests/req-123",
videoUrl: "https://fal.run/files/video.mp4",
bytes: "webm-bytes",
contentType: "video/webm",
});
const provider = buildFalVideoGenerationProvider();
const result = await provider.generateVideo({
provider: "fal",
model: "fal-ai/minimax/video-01-live",
prompt: "A spaceship emerges from the clouds",
durationSeconds: 5,
aspectRatio: "16:9",
resolution: "720P",
cfg: {},
});
expect(fetchGuardUrl(1)).toBe("https://queue.fal.run/fal-ai/minimax/video-01-live");
const submitBody = getSubmitBody();
expect(submitBody).toEqual({
prompt: "A spaceship emerges from the clouds",
});
expect(fetchGuardUrl(2)).toBe("https://queue.fal.run/fal-ai/minimax/requests/req-123/status");
expect(fetchGuardUrl(3)).toBe("https://queue.fal.run/fal-ai/minimax/requests/req-123");
expect(result.videos).toHaveLength(1);
expect(result.videos[0]?.mimeType).toBe("video/webm");
expect(result.videos[0]?.fileName).toBe("video-1.webm");
expect(result.videos[0]?.url).toBe("https://fal.run/files/video.mp4");
expect(result.metadata).toEqual({
requestId: "req-123",
});
});
it("parses raw fal queue result payloads with top-level video output", async () => {
mockFalProviderRuntime();
fetchGuardMock
.mockResolvedValueOnce(
releasedJson({
request_id: "req-raw",
status_url: "https://queue.fal.run/fal-ai/wan/requests/req-raw/status",
response_url: "https://queue.fal.run/fal-ai/wan/requests/req-raw",
}),
)
.mockResolvedValueOnce(releasedJson({ status: "COMPLETED" }))
.mockResolvedValueOnce(
releasedJson({
video: { url: "https://fal.run/files/raw-output.mp4" },
prompt: "A calm harbor at sunrise",
seed: 443600358,
}),
)
.mockResolvedValueOnce(releasedVideo({ contentType: "video/mp4", bytes: "mp4-bytes" }));
const provider = buildFalVideoGenerationProvider();
const result = await provider.generateVideo({
provider: "fal",
model: "fal-ai/wan/v2.2-a14b/image-to-video",
prompt: "A calm harbor at sunrise",
cfg: {},
});
expect(result.videos[0]?.url).toBe("https://fal.run/files/raw-output.mp4");
expect(result.metadata).toEqual({
requestId: "req-raw",
prompt: "A calm harbor at sunrise",
seed: 443600358,
});
});
it("returns URL-only videos when generated video downloads exceed the configured media cap", async () => {
mockFalProviderRuntime();
mockCompletedFalVideoJob({
requestId: "req-123",
statusUrl: "https://queue.fal.run/fal-ai/minimax/requests/req-123/status",
responseUrl: "https://queue.fal.run/fal-ai/minimax/requests/req-123",
videoUrl: "https://fal.run/files/video.mp4",
bytes: "too-large",
contentType: "video/mp4",
});
const provider = buildFalVideoGenerationProvider();
const result = await provider.generateVideo({
provider: "fal",
model: "fal-ai/minimax/video-01-live",
prompt: "A spaceship emerges from the clouds",
cfg: { agents: { defaults: { mediaMaxMb: 0.000001 } } },
});
expect(result.videos).toEqual([
{
url: "https://fal.run/files/video.mp4",
mimeType: "video/mp4",
fileName: "video-1.mp4",
},
]);
});
it("wraps malformed successful fal submit responses", async () => {
mockFalProviderRuntime();
fetchGuardMock.mockResolvedValueOnce(releasedJson([]));
const provider = buildFalVideoGenerationProvider();
await expect(
provider.generateVideo({
provider: "fal",
model: "fal-ai/minimax/video-01-live",
prompt: "bad shape",
cfg: {},
}),
).rejects.toThrow("fal video generation response malformed");
});
it("wraps non-JSON successful fal submit responses", async () => {
mockFalProviderRuntime();
fetchGuardMock.mockResolvedValueOnce({
response: new Response("<html><body>Bad Gateway</body></html>", {
status: 200,
headers: { "content-type": "text/html" },
}),
release: vi.fn(async () => {}),
});
const provider = buildFalVideoGenerationProvider();
await expect(
provider.generateVideo({
provider: "fal",
model: "fal-ai/minimax/video-01-live",
prompt: "html body",
cfg: {},
}),
).rejects.toThrow("fal video generation response malformed");
});
it("rejects missing fal queue statuses without waiting for timeout", async () => {
mockFalProviderRuntime();
fetchGuardMock
.mockResolvedValueOnce(
releasedJson({
request_id: "req-123",
status_url: "https://queue.fal.run/fal-ai/minimax/requests/req-123/status",
response_url: "https://queue.fal.run/fal-ai/minimax/requests/req-123",
}),
)
.mockResolvedValueOnce(releasedJson({}));
const provider = buildFalVideoGenerationProvider();
await expect(
provider.generateVideo({
provider: "fal",
model: "fal-ai/minimax/video-01-live",
prompt: "missing status",
cfg: {},
}),
).rejects.toThrow("fal video generation response malformed");
expect(fetchGuardMock).toHaveBeenCalledTimes(2);
});
it("rejects unknown fal queue statuses without waiting for timeout", async () => {
mockFalProviderRuntime();
fetchGuardMock
.mockResolvedValueOnce(
releasedJson({
request_id: "req-123",
status_url: "https://queue.fal.run/fal-ai/minimax/requests/req-123/status",
response_url: "https://queue.fal.run/fal-ai/minimax/requests/req-123",
}),
)
.mockResolvedValueOnce(releasedJson({ status: "ALMOST_DONE" }));
const provider = buildFalVideoGenerationProvider();
await expect(
provider.generateVideo({
provider: "fal",
model: "fal-ai/minimax/video-01-live",
prompt: "bad status",
cfg: {},
}),
).rejects.toThrow("fal video generation response malformed");
expect(fetchGuardMock).toHaveBeenCalledTimes(2);
});
it("caps oversized fal queue operation deadlines", async () => {
mockFalProviderRuntime();
const nowSpy = vi.spyOn(Date, "now");
nowSpy
.mockReturnValueOnce(0)
.mockReturnValueOnce(0)
.mockReturnValueOnce(MAX_TIMER_TIMEOUT_MS + 1);
fetchGuardMock
.mockResolvedValueOnce(
releasedJson({
request_id: "req-123",
status_url: "https://queue.fal.run/fal-ai/minimax/requests/req-123/status",
response_url: "https://queue.fal.run/fal-ai/minimax/requests/req-123",
}),
)
.mockResolvedValueOnce(releasedJson({ status: "IN_PROGRESS" }));
try {
const provider = buildFalVideoGenerationProvider();
await expect(
provider.generateVideo({
provider: "fal",
model: "fal-ai/minimax/video-01-live",
prompt: "huge timeout",
cfg: {},
timeoutMs: Number.MAX_SAFE_INTEGER,
}),
).rejects.toThrow("fal video generation did not finish in time (last status: IN_PROGRESS)");
expect(fetchGuardMock).toHaveBeenCalledTimes(2);
} finally {
nowSpy.mockRestore();
}
});
it("rejects malformed fal completed result payloads", async () => {
mockFalProviderRuntime();
fetchGuardMock
.mockResolvedValueOnce(
releasedJson({
request_id: "req-123",
status_url: "https://queue.fal.run/fal-ai/minimax/requests/req-123/status",
response_url: "https://queue.fal.run/fal-ai/minimax/requests/req-123",
}),
)
.mockResolvedValueOnce(releasedJson({ status: "COMPLETED" }))
.mockResolvedValueOnce(releasedJson({ status: "COMPLETED", response: [] }));
const provider = buildFalVideoGenerationProvider();
await expect(
provider.generateVideo({
provider: "fal",
model: "fal-ai/minimax/video-01-live",
prompt: "bad result",
cfg: {},
}),
).rejects.toThrow("fal video generation response malformed");
});
it("exposes Seedance 2 models", () => {
const provider = buildFalVideoGenerationProvider();
expect(provider.models).toContain("fal-ai/heygen/v2/video-agent");
expect(provider.models).toContain("bytedance/seedance-2.0/fast/text-to-video");
expect(provider.models).toContain("bytedance/seedance-2.0/fast/image-to-video");
expect(provider.models).toContain("bytedance/seedance-2.0/fast/reference-to-video");
expect(provider.models).toContain("bytedance/seedance-2.0/text-to-video");
expect(provider.models).toContain("bytedance/seedance-2.0/image-to-video");
expect(provider.models).toContain("bytedance/seedance-2.0/reference-to-video");
});
it("submits HeyGen video-agent requests without unsupported fal controls", async () => {
mockFalProviderRuntime();
mockCompletedFalVideoJob({
requestId: "heygen-req-123",
statusUrl:
"https://queue.fal.run/fal-ai/heygen/v2/video-agent/requests/heygen-req-123/status",
responseUrl: "https://queue.fal.run/fal-ai/heygen/v2/video-agent/requests/heygen-req-123",
videoUrl: "https://fal.run/files/heygen.mp4",
bytes: "heygen-mp4-bytes",
});
const provider = buildFalVideoGenerationProvider();
const result = await provider.generateVideo({
provider: "fal",
model: "fal-ai/heygen/v2/video-agent",
prompt: "A founder explains OpenClaw in a concise studio video",
durationSeconds: 8,
aspectRatio: "16:9",
resolution: "720P",
audio: true,
cfg: {},
});
expect(fetchGuardUrl(1)).toBe("https://queue.fal.run/fal-ai/heygen/v2/video-agent");
expect(getSubmitBody()).toEqual({
prompt: "A founder explains OpenClaw in a concise studio video",
});
expect(result.metadata).toEqual({
requestId: "heygen-req-123",
});
});
it("submits Seedance 2 requests with fal schema fields", async () => {
mockFalProviderRuntime();
mockCompletedFalVideoJob({
requestId: "seedance-req-123",
statusUrl:
"https://queue.fal.run/bytedance/seedance-2.0/fast/text-to-video/requests/seedance-req-123/status",
responseUrl:
"https://queue.fal.run/bytedance/seedance-2.0/fast/text-to-video/requests/seedance-req-123",
videoUrl: "https://fal.run/files/seedance.mp4",
bytes: "seedance-mp4-bytes",
responseExtras: { seed: 42 },
});
const provider = buildFalVideoGenerationProvider();
const result = await provider.generateVideo({
provider: "fal",
model: "bytedance/seedance-2.0/fast/text-to-video",
prompt: "A chrome lobster drives a tiny kart across a neon pier",
durationSeconds: 7,
aspectRatio: "16:9",
resolution: "720P",
audio: false,
cfg: {},
});
expect(fetchGuardUrl(1)).toBe(
"https://queue.fal.run/bytedance/seedance-2.0/fast/text-to-video",
);
expect(getSubmitBody()).toEqual({
prompt: "A chrome lobster drives a tiny kart across a neon pier",
aspect_ratio: "16:9",
resolution: "720p",
duration: "7",
generate_audio: false,
});
expect(result.metadata).toEqual({
requestId: "seedance-req-123",
seed: 42,
});
});
it("drops unsupported Seedance 2 duration values before queue submission", async () => {
mockFalProviderRuntime();
mockCompletedFalVideoJob({
requestId: "seedance-req-123",
statusUrl:
"https://queue.fal.run/bytedance/seedance-2.0/fast/text-to-video/requests/seedance-req-123/status",
responseUrl:
"https://queue.fal.run/bytedance/seedance-2.0/fast/text-to-video/requests/seedance-req-123",
videoUrl: "https://fal.run/files/seedance.mp4",
bytes: "seedance-mp4-bytes",
});
const provider = buildFalVideoGenerationProvider();
await provider.generateVideo({
provider: "fal",
model: "bytedance/seedance-2.0/fast/text-to-video",
prompt: "A chrome lobster drives a tiny kart across a neon pier",
durationSeconds: 99,
cfg: {},
});
expect(getSubmitBody()).not.toHaveProperty("duration");
});
it("submits Seedance 2 image-to-video requests with a single image_url", async () => {
mockFalProviderRuntime();
mockCompletedFalVideoJob({
requestId: "seedance-i2v-req-123",
statusUrl:
"https://queue.fal.run/bytedance/seedance-2.0/fast/image-to-video/requests/seedance-i2v-req-123/status",
responseUrl:
"https://queue.fal.run/bytedance/seedance-2.0/fast/image-to-video/requests/seedance-i2v-req-123",
videoUrl: "https://fal.run/files/seedance-i2v.mp4",
bytes: "seedance-i2v-mp4-bytes",
});
const provider = buildFalVideoGenerationProvider();
await provider.generateVideo({
provider: "fal",
model: "bytedance/seedance-2.0/fast/image-to-video",
prompt: "Animate this product still with a slow orbit",
durationSeconds: 6,
inputImages: [{ url: "https://example.com/start-frame.png" }],
cfg: {},
});
expect(getSubmitBody()).toEqual({
prompt: "Animate this product still with a slow orbit",
image_url: "https://example.com/start-frame.png",
duration: "6",
});
});
it("submits Seedance 2 reference-to-video requests with image, video, and audio URLs", async () => {
mockFalProviderRuntime();
mockCompletedFalVideoJob({
requestId: "seedance-ref-req-123",
statusUrl:
"https://queue.fal.run/bytedance/seedance-2.0/fast/reference-to-video/requests/seedance-ref-req-123/status",
responseUrl:
"https://queue.fal.run/bytedance/seedance-2.0/fast/reference-to-video/requests/seedance-ref-req-123",
videoUrl: "https://fal.run/files/seedance-ref.mp4",
bytes: "seedance-ref-mp4-bytes",
responseExtras: { seed: 1234 },
});
const provider = buildFalVideoGenerationProvider();
const result = await provider.generateVideo({
provider: "fal",
model: "bytedance/seedance-2.0/fast/reference-to-video",
prompt: "Blend @Image1, @Image2, @Video1, @Video2, and @Audio1 into one short film",
durationSeconds: 8,
aspectRatio: "9:16",
resolution: "480P",
audio: false,
inputImages: [
{ url: "https://example.com/reference-1.png" },
{ buffer: Buffer.from("local-image"), mimeType: "image/webp" },
],
inputVideos: [
{ url: "https://example.com/reference-1.mp4" },
{ buffer: Buffer.from("local-video"), mimeType: "video/quicktime" },
],
inputAudios: [
{ url: "https://example.com/reference-1.mp3" },
{ buffer: Buffer.from("local-audio"), mimeType: "audio/wav" },
],
cfg: {},
});
expect(fetchGuardUrl(1)).toBe(
"https://queue.fal.run/bytedance/seedance-2.0/fast/reference-to-video",
);
expect(getSubmitBody()).toEqual({
prompt: "Blend @Image1, @Image2, @Video1, @Video2, and @Audio1 into one short film",
image_urls: [
"https://example.com/reference-1.png",
`data:image/webp;base64,${Buffer.from("local-image").toString("base64")}`,
],
video_urls: [
"https://example.com/reference-1.mp4",
`data:video/quicktime;base64,${Buffer.from("local-video").toString("base64")}`,
],
audio_urls: [
"https://example.com/reference-1.mp3",
`data:audio/wav;base64,${Buffer.from("local-audio").toString("base64")}`,
],
aspect_ratio: "9:16",
resolution: "480p",
duration: "8",
generate_audio: false,
});
expect(result.metadata).toEqual({
requestId: "seedance-ref-req-123",
seed: 1234,
});
});
it("rejects video, audio, and multiple image references for non-reference fal models", async () => {
const provider = buildFalVideoGenerationProvider();
await expect(
provider.generateVideo({
provider: "fal",
model: "fal-ai/minimax/video-01-live",
prompt: "Animate this",
inputImages: [
{ url: "https://example.com/one.png" },
{ url: "https://example.com/two.png" },
],
cfg: {},
}),
).rejects.toThrow("fal video generation supports at most one image reference.");
await expect(
provider.generateVideo({
provider: "fal",
model: "fal-ai/minimax/video-01-live",
prompt: "Animate this",
inputVideos: [{ url: "https://example.com/reference.mp4" }],
cfg: {},
}),
).rejects.toThrow("fal video generation does not support video reference inputs.");
await expect(
provider.generateVideo({
provider: "fal",
model: "fal-ai/minimax/video-01-live",
prompt: "Animate this",
inputAudios: [{ url: "https://example.com/reference.mp3" }],
cfg: {},
}),
).rejects.toThrow("fal video generation does not support audio reference inputs.");
});
it("rejects over-limit and audio-only Seedance reference-to-video requests", async () => {
const provider = buildFalVideoGenerationProvider();
const model = "bytedance/seedance-2.0/fast/reference-to-video";
await expect(
provider.generateVideo({
provider: "fal",
model,
prompt: "Too many images",
inputImages: Array.from({ length: 10 }, (_, index) => ({
url: `https://example.com/image-${index}.png`,
})),
cfg: {},
}),
).rejects.toThrow("fal Seedance reference-to-video supports at most 9 reference images.");
await expect(
provider.generateVideo({
provider: "fal",
model,
prompt: "Too many videos",
inputVideos: Array.from({ length: 4 }, (_, index) => ({
url: `https://example.com/video-${index}.mp4`,
})),
cfg: {},
}),
).rejects.toThrow("fal Seedance reference-to-video supports at most 3 reference videos.");
await expect(
provider.generateVideo({
provider: "fal",
model,
prompt: "Too many audios",
inputAudios: Array.from({ length: 4 }, (_, index) => ({
url: `https://example.com/audio-${index}.mp3`,
})),
cfg: {},
}),
).rejects.toThrow("fal Seedance reference-to-video supports at most 3 reference audios.");
await expect(
provider.generateVideo({
provider: "fal",
model,
prompt: "Too many total files",
inputImages: Array.from({ length: 9 }, (_, index) => ({
url: `https://example.com/image-${index}.png`,
})),
inputVideos: Array.from({ length: 3 }, (_, index) => ({
url: `https://example.com/video-${index}.mp4`,
})),
inputAudios: [{ url: "https://example.com/audio.mp3" }],
cfg: {},
}),
).rejects.toThrow("fal Seedance reference-to-video supports at most 12 total reference files.");
await expect(
provider.generateVideo({
provider: "fal",
model,
prompt: "Audio only",
inputAudios: [{ url: "https://example.com/audio.mp3" }],
cfg: {},
}),
).rejects.toThrow(
"fal Seedance reference-to-video requires at least one image or video reference when audio references are provided.",
);
});
});

View File

@@ -0,0 +1,714 @@
// Fal provider module implements model/runtime integration.
import { extensionForMime } from "openclaw/plugin-sdk/media-mime";
import { resolvePositiveTimerTimeoutMs } from "openclaw/plugin-sdk/number-runtime";
import { isProviderApiKeyConfigured } from "openclaw/plugin-sdk/provider-auth";
import {
assertOkOrThrowHttpError,
createProviderOperationDeadline,
readProviderJsonResponse,
type ProviderOperationDeadline,
} from "openclaw/plugin-sdk/provider-http";
import { readResponseWithLimit } from "openclaw/plugin-sdk/response-limit-runtime";
import {
fetchWithSsrFGuard,
type SsrFPolicy,
ssrfPolicyFromDangerouslyAllowPrivateNetwork,
} from "openclaw/plugin-sdk/ssrf-runtime";
import {
isRecord,
normalizeLowercaseStringOrEmpty,
normalizeOptionalString,
} from "openclaw/plugin-sdk/string-coerce-runtime";
import type {
GeneratedVideoAsset,
VideoGenerationProvider,
VideoGenerationRequest,
} from "openclaw/plugin-sdk/video-generation";
import { resolveFalHttpRequestConfig } from "./http-config.js";
const DEFAULT_FAL_QUEUE_BASE_URL = "https://queue.fal.run";
const DEFAULT_FAL_VIDEO_MODEL = "fal-ai/minimax/video-01-live";
const HEYGEN_VIDEO_AGENT_MODEL = "fal-ai/heygen/v2/video-agent";
const SEEDANCE_2_TEXT_IMAGE_VIDEO_MODELS = [
"bytedance/seedance-2.0/fast/text-to-video",
"bytedance/seedance-2.0/fast/image-to-video",
"bytedance/seedance-2.0/text-to-video",
"bytedance/seedance-2.0/image-to-video",
] as const;
const SEEDANCE_2_REFERENCE_VIDEO_MODELS = [
"bytedance/seedance-2.0/fast/reference-to-video",
"bytedance/seedance-2.0/reference-to-video",
] as const;
const SEEDANCE_2_VIDEO_MODELS = [
...SEEDANCE_2_TEXT_IMAGE_VIDEO_MODELS,
...SEEDANCE_2_REFERENCE_VIDEO_MODELS,
] as const;
const SEEDANCE_2_DURATION_SECONDS = [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] as const;
const SEEDANCE_REFERENCE_MAX_IMAGES = 9;
const SEEDANCE_REFERENCE_MAX_VIDEOS = 3;
const SEEDANCE_REFERENCE_MAX_AUDIOS = 3;
const SEEDANCE_REFERENCE_MAX_FILES = 12;
const SEEDANCE_REFERENCE_MAX_IMAGES_BY_MODEL = Object.fromEntries(
SEEDANCE_2_REFERENCE_VIDEO_MODELS.map((model) => [model, SEEDANCE_REFERENCE_MAX_IMAGES]),
);
const SEEDANCE_REFERENCE_MAX_VIDEOS_BY_MODEL = Object.fromEntries(
SEEDANCE_2_REFERENCE_VIDEO_MODELS.map((model) => [model, SEEDANCE_REFERENCE_MAX_VIDEOS]),
);
const SEEDANCE_REFERENCE_MAX_AUDIOS_BY_MODEL = Object.fromEntries(
SEEDANCE_2_REFERENCE_VIDEO_MODELS.map((model) => [model, SEEDANCE_REFERENCE_MAX_AUDIOS]),
);
const DEFAULT_HTTP_TIMEOUT_MS = 30_000;
const DEFAULT_OPERATION_TIMEOUT_MS = 1_200_000;
const DEFAULT_GENERATED_VIDEO_MAX_BYTES = 16 * 1024 * 1024;
const POLL_INTERVAL_MS = 5_000;
const FAL_VIDEO_MALFORMED_RESPONSE = "fal video generation response malformed";
const FAL_VIDEO_PENDING_STATUSES = new Set([
"IN_QUEUE",
"IN_PROGRESS",
"PROCESSING",
"QUEUED",
"STARTED",
]);
type FalVideoResponse = {
video?: {
url?: string;
content_type?: string;
};
videos?: Array<{
url?: string;
content_type?: string;
}>;
prompt?: string;
seed?: number;
};
type FalQueueResponse = {
status?: string;
request_id?: string;
response_url?: string;
status_url?: string;
cancel_url?: string;
detail?: string;
response?: FalVideoResponse;
prompt?: string;
error?: {
message?: string;
};
};
let falFetchGuard = fetchWithSsrFGuard;
export function setFalVideoFetchGuardForTesting(impl: typeof fetchWithSsrFGuard | null): void {
falFetchGuard = impl ?? fetchWithSsrFGuard;
}
function normalizeFalVideoUrl(value: unknown): string | undefined {
const normalized = normalizeOptionalString(value);
if (!normalized && value !== undefined && value !== null) {
throw new Error(FAL_VIDEO_MALFORMED_RESPONSE);
}
return normalized;
}
function readFalVideoPayload(payload: unknown): FalVideoResponse {
if (!isRecord(payload)) {
throw new Error(FAL_VIDEO_MALFORMED_RESPONSE);
}
const video = payload.video;
const videos = payload.videos;
if (video !== undefined && video !== null && !isRecord(video)) {
throw new Error(FAL_VIDEO_MALFORMED_RESPONSE);
}
if (videos !== undefined && videos !== null && !Array.isArray(videos)) {
throw new Error(FAL_VIDEO_MALFORMED_RESPONSE);
}
return {
video: isRecord(video)
? {
url: normalizeFalVideoUrl(video.url),
content_type: normalizeOptionalString(video.content_type),
}
: undefined,
videos: Array.isArray(videos)
? videos.map((entry) => {
if (!isRecord(entry)) {
throw new Error(FAL_VIDEO_MALFORMED_RESPONSE);
}
return {
url: normalizeFalVideoUrl(entry.url),
content_type: normalizeOptionalString(entry.content_type),
};
})
: undefined,
prompt: normalizeOptionalString(payload.prompt),
seed: typeof payload.seed === "number" ? payload.seed : undefined,
};
}
function readFalQueueResponse(payload: unknown): FalQueueResponse {
if (!isRecord(payload)) {
throw new Error(FAL_VIDEO_MALFORMED_RESPONSE);
}
const error = payload.error;
if (error !== undefined && error !== null && !isRecord(error)) {
throw new Error(FAL_VIDEO_MALFORMED_RESPONSE);
}
return {
status: normalizeOptionalString(payload.status),
request_id: normalizeOptionalString(payload.request_id),
response_url: normalizeOptionalString(payload.response_url),
status_url: normalizeOptionalString(payload.status_url),
cancel_url: normalizeOptionalString(payload.cancel_url),
detail: normalizeOptionalString(payload.detail),
response: payload.response === undefined ? undefined : readFalVideoPayload(payload.response),
prompt: normalizeOptionalString(payload.prompt),
error: isRecord(error) ? { message: normalizeOptionalString(error.message) } : undefined,
};
}
function readFalCompletedQueueResult(payload: unknown): FalQueueResponse {
if (!isRecord(payload)) {
throw new Error(FAL_VIDEO_MALFORMED_RESPONSE);
}
if (
payload.response !== undefined ||
(payload.video === undefined && payload.videos === undefined)
) {
return readFalQueueResponse(payload);
}
return {
response: readFalVideoPayload(payload),
};
}
function toDataUrl(buffer: Buffer, mimeType: string): string {
return `data:${mimeType};base64,${buffer.toString("base64")}`;
}
function buildPolicy(allowPrivateNetwork: boolean): SsrFPolicy | undefined {
return allowPrivateNetwork ? ssrfPolicyFromDangerouslyAllowPrivateNetwork(true) : undefined;
}
function extractFalVideoEntry(payload: FalVideoResponse) {
if (normalizeOptionalString(payload.video?.url)) {
return payload.video;
}
return payload.videos?.find((entry) => normalizeOptionalString(entry.url));
}
function resolveGeneratedVideoMaxBytes(req: VideoGenerationRequest): number {
const configured = req.cfg.agents?.defaults?.mediaMaxMb;
if (typeof configured === "number" && Number.isFinite(configured) && configured > 0) {
return Math.floor(configured * 1024 * 1024);
}
return DEFAULT_GENERATED_VIDEO_MAX_BYTES;
}
async function downloadFalVideo(
url: string,
policy: SsrFPolicy | undefined,
maxBytes: number,
): Promise<GeneratedVideoAsset> {
const { response, release } = await falFetchGuard({
url,
timeoutMs: DEFAULT_HTTP_TIMEOUT_MS,
policy,
auditContext: "fal-video-download",
});
try {
await assertOkOrThrowHttpError(response, "fal generated video download failed");
const mimeType = normalizeOptionalString(response.headers.get("content-type")) ?? "video/mp4";
const fileName = `video-1.${extensionForMime(mimeType)?.slice(1) ?? "mp4"}`;
let exceededMaxBytes = false;
let buffer: Buffer;
try {
buffer = await readResponseWithLimit(response, maxBytes, {
onOverflow: ({ maxBytes: maxBytesLocal }) => {
exceededMaxBytes = true;
return new Error(`fal generated video download exceeds ${maxBytesLocal} bytes`);
},
});
} catch (error) {
if (exceededMaxBytes) {
return {
url,
mimeType,
fileName,
};
}
throw error;
}
return {
url,
buffer,
mimeType,
fileName,
};
} finally {
await release();
}
}
function resolveFalQueueBaseUrl(baseUrl: string): string {
try {
const url = new URL(baseUrl);
if (url.hostname === "fal.run") {
url.hostname = "queue.fal.run";
return url.toString().replace(/\/$/, "");
}
return baseUrl.replace(/\/$/, "");
} catch {
return DEFAULT_FAL_QUEUE_BASE_URL;
}
}
function isFalMiniMaxLiveModel(model: string): boolean {
return normalizeLowercaseStringOrEmpty(model) === DEFAULT_FAL_VIDEO_MODEL;
}
function isFalSeedance2Model(model: string): boolean {
return SEEDANCE_2_VIDEO_MODELS.includes(model as (typeof SEEDANCE_2_VIDEO_MODELS)[number]);
}
function isFalSeedance2ReferenceModel(model: string): boolean {
return SEEDANCE_2_REFERENCE_VIDEO_MODELS.includes(
model as (typeof SEEDANCE_2_REFERENCE_VIDEO_MODELS)[number],
);
}
function isFalHeyGenVideoAgentModel(model: string): boolean {
return normalizeLowercaseStringOrEmpty(model) === HEYGEN_VIDEO_AGENT_MODEL;
}
function resolveFalResolution(resolution: VideoGenerationRequest["resolution"], model: string) {
if (!resolution) {
return undefined;
}
if (isFalSeedance2Model(model)) {
return resolution.toLowerCase();
}
return resolution;
}
function resolveFalDuration(
durationSeconds: number | undefined,
model: string,
): number | string | undefined {
if (typeof durationSeconds !== "number" || !Number.isFinite(durationSeconds)) {
return undefined;
}
const duration = Math.max(1, Math.round(durationSeconds));
if (isFalSeedance2Model(model)) {
return SEEDANCE_2_DURATION_SECONDS.includes(
duration as (typeof SEEDANCE_2_DURATION_SECONDS)[number],
)
? String(duration)
: undefined;
}
return duration;
}
function resolveFalReferenceUrl(
asset: NonNullable<VideoGenerationRequest["inputImages"]>[number] | undefined,
defaultMimeType: string,
label: string,
): string {
const assetUrl = normalizeOptionalString(asset?.url);
if (assetUrl) {
return assetUrl;
}
if (!asset?.buffer) {
throw new Error(`fal ${label} is missing media data.`);
}
return toDataUrl(asset.buffer, normalizeOptionalString(asset.mimeType) ?? defaultMimeType);
}
function resolveFalReferenceUrls(
assets: VideoGenerationRequest["inputImages"],
defaultMimeType: string,
label: string,
): string[] {
return (assets ?? []).map((asset) => resolveFalReferenceUrl(asset, defaultMimeType, label));
}
function applyFalSeedanceControls(params: {
req: VideoGenerationRequest;
model: string;
body: Record<string, unknown>;
}): void {
const aspectRatio = normalizeOptionalString(params.req.aspectRatio);
if (aspectRatio) {
params.body.aspect_ratio = aspectRatio;
}
const size = normalizeOptionalString(params.req.size);
if (size) {
params.body.size = size;
}
const resolution = resolveFalResolution(params.req.resolution, params.model);
if (resolution) {
params.body.resolution = resolution;
}
const duration = resolveFalDuration(params.req.durationSeconds, params.model);
if (duration) {
params.body.duration = duration;
}
if (isFalSeedance2Model(params.model) && typeof params.req.audio === "boolean") {
params.body.generate_audio = params.req.audio;
}
}
function buildFalVideoRequestBody(params: {
req: VideoGenerationRequest;
model: string;
}): Record<string, unknown> {
const requestBody: Record<string, unknown> = {
prompt: params.req.prompt,
};
if (isFalSeedance2ReferenceModel(params.model)) {
const imageUrls = resolveFalReferenceUrls(
params.req.inputImages,
"image/png",
"reference image",
);
const videoUrls = resolveFalReferenceUrls(
params.req.inputVideos,
"video/mp4",
"reference video",
);
const audioUrls = resolveFalReferenceUrls(
params.req.inputAudios,
"audio/mpeg",
"reference audio",
);
if (imageUrls.length > 0) {
requestBody.image_urls = imageUrls;
}
if (videoUrls.length > 0) {
requestBody.video_urls = videoUrls;
}
if (audioUrls.length > 0) {
requestBody.audio_urls = audioUrls;
}
applyFalSeedanceControls({ req: params.req, model: params.model, body: requestBody });
return requestBody;
}
const input = params.req.inputImages?.[0];
if (input) {
requestBody.image_url = normalizeOptionalString(input.url)
? normalizeOptionalString(input.url)
: input.buffer
? toDataUrl(input.buffer, normalizeOptionalString(input.mimeType) ?? "image/png")
: undefined;
}
// MiniMax Live on fal currently documents prompt + optional image_url only.
// Keep the default model conservative so queue requests do not hang behind
// unsupported knobs such as duration/resolution/aspect-ratio overrides.
if (isFalMiniMaxLiveModel(params.model) || isFalHeyGenVideoAgentModel(params.model)) {
return requestBody;
}
applyFalSeedanceControls({ req: params.req, model: params.model, body: requestBody });
return requestBody;
}
function validateFalVideoReferenceInputs(params: {
req: VideoGenerationRequest;
model: string;
}): void {
const imageCount = params.req.inputImages?.length ?? 0;
const videoCount = params.req.inputVideos?.length ?? 0;
const audioCount = params.req.inputAudios?.length ?? 0;
if (isFalSeedance2ReferenceModel(params.model)) {
if (imageCount > SEEDANCE_REFERENCE_MAX_IMAGES) {
throw new Error(
`fal Seedance reference-to-video supports at most ${SEEDANCE_REFERENCE_MAX_IMAGES} reference images.`,
);
}
if (videoCount > SEEDANCE_REFERENCE_MAX_VIDEOS) {
throw new Error(
`fal Seedance reference-to-video supports at most ${SEEDANCE_REFERENCE_MAX_VIDEOS} reference videos.`,
);
}
if (audioCount > SEEDANCE_REFERENCE_MAX_AUDIOS) {
throw new Error(
`fal Seedance reference-to-video supports at most ${SEEDANCE_REFERENCE_MAX_AUDIOS} reference audios.`,
);
}
const totalFiles = imageCount + videoCount + audioCount;
if (totalFiles > SEEDANCE_REFERENCE_MAX_FILES) {
throw new Error(
`fal Seedance reference-to-video supports at most ${SEEDANCE_REFERENCE_MAX_FILES} total reference files.`,
);
}
if (audioCount > 0 && imageCount === 0 && videoCount === 0) {
throw new Error(
"fal Seedance reference-to-video requires at least one image or video reference when audio references are provided.",
);
}
return;
}
if (videoCount > 0) {
throw new Error("fal video generation does not support video reference inputs.");
}
if (audioCount > 0) {
throw new Error("fal video generation does not support audio reference inputs.");
}
if (imageCount > 1) {
throw new Error("fal video generation supports at most one image reference.");
}
}
async function fetchFalJson(params: {
url: string;
init?: RequestInit;
timeoutMs: number;
policy: SsrFPolicy | undefined;
dispatcherPolicy: Parameters<typeof fetchWithSsrFGuard>[0]["dispatcherPolicy"];
auditContext: string;
errorContext: string;
}): Promise<unknown> {
const { response, release } = await falFetchGuard({
url: params.url,
init: params.init,
timeoutMs: params.timeoutMs,
policy: params.policy,
dispatcherPolicy: params.dispatcherPolicy,
auditContext: params.auditContext,
});
try {
await assertOkOrThrowHttpError(response, params.errorContext);
try {
return await readProviderJsonResponse<unknown>(response, params.errorContext);
} catch (error) {
if (error instanceof Error && error.message.endsWith(": malformed JSON response")) {
throw new Error(FAL_VIDEO_MALFORMED_RESPONSE, { cause: error });
}
throw error;
}
} finally {
await release();
}
}
async function waitForFalQueueResult(params: {
statusUrl: string;
responseUrl: string;
headers: Headers;
deadline: ProviderOperationDeadline;
policy: SsrFPolicy | undefined;
dispatcherPolicy: Parameters<typeof fetchWithSsrFGuard>[0]["dispatcherPolicy"];
}): Promise<FalQueueResponse> {
let lastStatus = "unknown";
for (;;) {
const requestTimeoutMs = resolveFalQueueRemainingMs(
params.deadline,
lastStatus,
DEFAULT_HTTP_TIMEOUT_MS,
);
const payload = readFalQueueResponse(
await fetchFalJson({
url: params.statusUrl,
init: {
method: "GET",
headers: params.headers,
},
timeoutMs: requestTimeoutMs,
policy: params.policy,
dispatcherPolicy: params.dispatcherPolicy,
auditContext: "fal-video-status",
errorContext: "fal video status request failed",
}),
);
const status = normalizeOptionalString(payload.status)?.toUpperCase();
if (!status) {
throw new Error(FAL_VIDEO_MALFORMED_RESPONSE);
}
lastStatus = status;
if (status === "COMPLETED") {
return readFalCompletedQueueResult(
await fetchFalJson({
url: params.responseUrl,
init: {
method: "GET",
headers: params.headers,
},
timeoutMs: resolveFalQueueRemainingMs(
params.deadline,
lastStatus,
DEFAULT_HTTP_TIMEOUT_MS,
),
policy: params.policy,
dispatcherPolicy: params.dispatcherPolicy,
auditContext: "fal-video-result",
errorContext: "fal video result request failed",
}),
);
}
if (status === "FAILED" || status === "CANCELLED") {
throw new Error(
normalizeOptionalString(payload.detail) ||
normalizeOptionalString(payload.error?.message) ||
`fal video generation ${normalizeLowercaseStringOrEmpty(status)}`,
);
}
if (!FAL_VIDEO_PENDING_STATUSES.has(status)) {
throw new Error(FAL_VIDEO_MALFORMED_RESPONSE);
}
const pollDelayMs = resolveFalQueueRemainingMs(params.deadline, lastStatus, POLL_INTERVAL_MS);
await new Promise((resolve) => {
setTimeout(resolve, pollDelayMs);
});
}
}
function resolveFalQueueRemainingMs(
deadline: ProviderOperationDeadline,
lastStatus: string,
defaultTimeoutMs: number,
): number {
const defaultMs = resolvePositiveTimerTimeoutMs(defaultTimeoutMs, 1);
if (typeof deadline.deadlineAtMs !== "number") {
return defaultMs;
}
const remainingMs = deadline.deadlineAtMs - Date.now();
if (remainingMs <= 0) {
throw new Error(`fal video generation did not finish in time (last status: ${lastStatus})`);
}
return Math.max(1, Math.min(defaultMs, remainingMs));
}
function extractFalVideoPayload(payload: FalQueueResponse): FalVideoResponse {
if (payload.response) {
return payload.response;
}
return readFalVideoPayload(payload);
}
export function buildFalVideoGenerationProvider(): VideoGenerationProvider {
return {
id: "fal",
label: "fal",
defaultModel: DEFAULT_FAL_VIDEO_MODEL,
models: [
DEFAULT_FAL_VIDEO_MODEL,
HEYGEN_VIDEO_AGENT_MODEL,
...SEEDANCE_2_VIDEO_MODELS,
"fal-ai/kling-video/v2.1/master/text-to-video",
"fal-ai/wan/v2.2-a14b/text-to-video",
"fal-ai/wan/v2.2-a14b/image-to-video",
],
isConfigured: ({ agentDir }) =>
isProviderApiKeyConfigured({
provider: "fal",
agentDir,
}),
capabilities: {
generate: {
maxVideos: 1,
supportedDurationSecondsByModel: Object.fromEntries(
SEEDANCE_2_VIDEO_MODELS.map((model) => [model, SEEDANCE_2_DURATION_SECONDS]),
),
supportsAspectRatio: true,
supportsResolution: true,
supportsSize: true,
supportsAudio: true,
},
imageToVideo: {
enabled: true,
maxVideos: 1,
maxInputImages: 1,
maxInputImagesByModel: SEEDANCE_REFERENCE_MAX_IMAGES_BY_MODEL,
maxInputAudiosByModel: SEEDANCE_REFERENCE_MAX_AUDIOS_BY_MODEL,
supportedDurationSecondsByModel: Object.fromEntries(
SEEDANCE_2_VIDEO_MODELS.map((model) => [model, SEEDANCE_2_DURATION_SECONDS]),
),
supportsAspectRatio: true,
supportsResolution: true,
supportsSize: true,
supportsAudio: true,
},
videoToVideo: {
enabled: true,
maxVideos: 1,
maxInputImages: 0,
maxInputImagesByModel: SEEDANCE_REFERENCE_MAX_IMAGES_BY_MODEL,
maxInputVideos: 0,
maxInputVideosByModel: SEEDANCE_REFERENCE_MAX_VIDEOS_BY_MODEL,
maxInputAudiosByModel: SEEDANCE_REFERENCE_MAX_AUDIOS_BY_MODEL,
supportedDurationSecondsByModel: Object.fromEntries(
SEEDANCE_2_REFERENCE_VIDEO_MODELS.map((model) => [model, SEEDANCE_2_DURATION_SECONDS]),
),
supportsAspectRatio: true,
supportsResolution: true,
supportsSize: true,
supportsAudio: true,
},
},
async generateVideo(req) {
const model = normalizeOptionalString(req.model) || DEFAULT_FAL_VIDEO_MODEL;
validateFalVideoReferenceInputs({ req, model });
const { baseUrl, allowPrivateNetwork, headers, dispatcherPolicy } =
await resolveFalHttpRequestConfig({ req, capability: "video" });
const requestBody = buildFalVideoRequestBody({ req, model });
const policy = buildPolicy(allowPrivateNetwork);
const queueBaseUrl = resolveFalQueueBaseUrl(baseUrl);
const submitted = readFalQueueResponse(
await fetchFalJson({
url: `${queueBaseUrl}/${model}`,
init: {
method: "POST",
headers,
body: JSON.stringify(requestBody),
},
timeoutMs: DEFAULT_HTTP_TIMEOUT_MS,
policy,
dispatcherPolicy,
auditContext: "fal-video-submit",
errorContext: "fal video generation failed",
}),
);
const statusUrl = normalizeOptionalString(submitted.status_url);
const responseUrl = normalizeOptionalString(submitted.response_url);
if (!statusUrl || !responseUrl) {
throw new Error("fal video generation response missing queue URLs");
}
const operationTimeoutMs = resolvePositiveTimerTimeoutMs(
req.timeoutMs,
DEFAULT_OPERATION_TIMEOUT_MS,
);
const operationDeadline = createProviderOperationDeadline({
timeoutMs: operationTimeoutMs,
label: "fal video generation",
});
const payload = await waitForFalQueueResult({
statusUrl,
responseUrl,
headers,
deadline: operationDeadline,
policy,
dispatcherPolicy,
});
const videoPayload = extractFalVideoPayload(payload);
const entry = extractFalVideoEntry(videoPayload);
const url = normalizeOptionalString(entry?.url);
if (!url) {
throw new Error("fal video generation response missing output URL");
}
const video = await downloadFalVideo(url, policy, resolveGeneratedVideoMaxBytes(req));
return {
videos: [video],
model,
metadata: {
...(normalizeOptionalString(submitted.request_id)
? { requestId: normalizeOptionalString(submitted.request_id) }
: {}),
...(videoPayload.prompt ? { prompt: videoPayload.prompt } : {}),
...(typeof videoPayload.seed === "number" ? { seed: videoPayload.seed } : {}),
},
};
},
};
}