Vendor OpenClaw source as Adolf fork baseline
Some checks failed
ClawSweeper Dispatch / dispatch (push) Has been cancelled
CodeQL / Security High (actions) (push) Has been cancelled
CodeQL / Security High (channel-runtime-boundary) (push) Has been cancelled
CodeQL / Security High (core-auth-secrets) (push) Has been cancelled
CodeQL / Security High (mcp-process-tool-boundary) (push) Has been cancelled
CodeQL / Security High (network-ssrf-boundary) (push) Has been cancelled
CodeQL / Security High (plugin-trust-boundary) (push) Has been cancelled
CodeQL / Security High (process-exec-boundary) (push) Has been cancelled
Docs Sync Publish Repo / sync-publish-repo (push) Has been cancelled
Docs / docs (push) Has been cancelled
OpenClaw Stable Main Closeout / Resolve stable release closeout inputs (push) Has been cancelled
OpenClaw Stable Main Closeout / Verify stable main closeout (push) Has been cancelled
Workflow Sanity / no-tabs (push) Has been cancelled
Workflow Sanity / actionlint (push) Has been cancelled
Workflow Sanity / generated-doc-baselines (push) Has been cancelled
CI / runner-admission (push) Has been cancelled
CI / preflight (push) Has been cancelled
CI / security-fast (push) Has been cancelled
CI / pnpm-store-warmup (push) Has been cancelled
CI / build-artifacts (push) Has been cancelled
CI / native-i18n (push) Has been cancelled
CI / ${{ matrix.check_name }} (push) Has been cancelled
CI / ${{ matrix.checkName }} (push) Has been cancelled
CI / checks-node-compat-node22 (push) Has been cancelled
CI / check-bundled-channel-config-metadata (push) Has been cancelled
CI / check-dependencies (push) Has been cancelled
CI / check-guards (push) Has been cancelled
CI / check-lint (push) Has been cancelled
CI / check-prod-types (push) Has been cancelled
CI / check-shrinkwrap (push) Has been cancelled
CI / check-test-types (push) Has been cancelled
CI / check-additional-boundaries-a (push) Has been cancelled
CI / check-additional-boundaries-bcd (push) Has been cancelled
CI / check-additional-extension-bundled (push) Has been cancelled
CI / check-additional-extension-channels (push) Has been cancelled
CI / check-additional-extension-package-boundary (push) Has been cancelled
CI / check-additional-runtime-topology-architecture (push) Has been cancelled
CI / check-session-accessor-boundary (push) Has been cancelled
CI / check-session-transcript-reader-boundary (push) Has been cancelled
CI / check-docs (push) Has been cancelled
CI / skills-python (push) Has been cancelled
CI / macos-swift (push) Has been cancelled
CI / ios-build (push) Has been cancelled
CI / ci-timings-summary (push) Has been cancelled
Native App Locale Refresh / Refresh native fa (push) Has been cancelled
Native App Locale Refresh / Refresh native fr (push) Has been cancelled
Native App Locale Refresh / Refresh native hi (push) Has been cancelled
Native App Locale Refresh / Refresh native id (push) Has been cancelled
Native App Locale Refresh / Refresh native it (push) Has been cancelled
Native App Locale Refresh / Refresh native ja-JP (push) Has been cancelled
Control UI Locale Refresh / plan (push) Has been cancelled
Control UI Locale Refresh / Refresh ${{ matrix.locale }} (push) Has been cancelled
Control UI Locale Refresh / Commit control UI locale refresh (push) Has been cancelled
Live Media Runner Image / Build live media runner image (push) Has been cancelled
Native App Locale Refresh / Refresh native ar (push) Has been cancelled
Native App Locale Refresh / Refresh native de (push) Has been cancelled
Native App Locale Refresh / Refresh native es (push) Has been cancelled
Native App Locale Refresh / Refresh native ko (push) Has been cancelled
Native App Locale Refresh / Refresh native nl (push) Has been cancelled
Native App Locale Refresh / Refresh native pl (push) Has been cancelled
Native App Locale Refresh / Refresh native pt-BR (push) Has been cancelled
Native App Locale Refresh / Refresh native ru (push) Has been cancelled
Native App Locale Refresh / Refresh native sv (push) Has been cancelled
Native App Locale Refresh / Refresh native th (push) Has been cancelled
Native App Locale Refresh / Refresh native tr (push) Has been cancelled
Native App Locale Refresh / Refresh native uk (push) Has been cancelled
Native App Locale Refresh / Refresh native vi (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-CN (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-TW (push) Has been cancelled
Native App Locale Refresh / Commit native locale refresh (push) Has been cancelled
Plugin Init Scaffold Validation / Validate provider scaffold (push) Has been cancelled
Plugin NPM Release / preview_plugins_npm (push) Has been cancelled
Plugin NPM Release / Validate release publish approval (push) Has been cancelled
Plugin NPM Release / preview_plugin_pack (push) Has been cancelled
Plugin NPM Release / publish_plugins_npm (push) Has been cancelled
Sandbox Common Smoke / sandbox-common-smoke (push) Has been cancelled
Website Installer Sync / static (push) Has been cancelled
Website Installer Sync / linux-docker (push) Has been cancelled
Website Installer Sync / macos-installer (push) Has been cancelled
Website Installer Sync / windows-installer (push) Has been cancelled
Website Installer Sync / sync-website (push) Has been cancelled

Adolf is a fork/vendored clone of github.com/openclaw/openclaw (v2026.6.11),
free to diverge. Tree copied sans upstream .git; upstream remote added for
future syncs. Node pinned to 24 (.nvmrc); engines already require >=22.19.
Preserves docs/ARCHITECTURE.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LeqyaxJF2nbRXJtae2kNB2
This commit is contained in:
2026-07-05 09:36:54 +00:00
parent 3216769225
commit bedb527145
21108 changed files with 6010766 additions and 0 deletions

View File

@@ -0,0 +1,12 @@
// Runway plugin entrypoint registers its OpenClaw integration.
import { definePluginEntry } from "openclaw/plugin-sdk/plugin-entry";
import { buildRunwayVideoGenerationProvider } from "./video-generation-provider.js";
export default definePluginEntry({
id: "runway",
name: "Runway Provider",
description: "Bundled Runway video provider plugin",
register(api) {
api.registerVideoGenerationProvider(buildRunwayVideoGenerationProvider());
},
});

View File

@@ -0,0 +1,39 @@
{
"id": "runway",
"activation": {
"onStartup": false
},
"enabledByDefault": true,
"setup": {
"providers": [
{
"id": "runway",
"envVars": ["RUNWAYML_API_SECRET", "RUNWAY_API_KEY"]
}
]
},
"providerAuthChoices": [
{
"provider": "runway",
"method": "api-key",
"choiceId": "runway-api-key",
"choiceLabel": "Runway API key",
"groupId": "runway",
"groupLabel": "Runway",
"groupHint": "API key",
"onboardingScopes": ["image-generation"],
"optionKey": "runwayApiKey",
"cliFlag": "--runway-api-key",
"cliOption": "--runway-api-key <key>",
"cliDescription": "Runway API key"
}
],
"contracts": {
"videoGenerationProviders": ["runway"]
},
"configSchema": {
"type": "object",
"additionalProperties": false,
"properties": {}
}
}

View File

@@ -0,0 +1,15 @@
{
"name": "@openclaw/runway-provider",
"version": "2026.6.11",
"private": true,
"description": "OpenClaw Runway video provider plugin",
"type": "module",
"devDependencies": {
"@openclaw/plugin-sdk": "workspace:*"
},
"openclaw": {
"extensions": [
"./index.ts"
]
}
}

View File

@@ -0,0 +1,16 @@
{
"extends": "../tsconfig.package-boundary.base.json",
"compilerOptions": {
"rootDir": "."
},
"include": ["./*.ts", "./src/**/*.ts"],
"exclude": [
"./**/*.test.ts",
"./dist/**",
"./node_modules/**",
"./src/test-support/**",
"./src/**/*test-helpers.ts",
"./src/**/*test-harness.ts",
"./src/**/*test-support.ts"
]
}

View File

@@ -0,0 +1,321 @@
// Runway tests cover video generation provider plugin behavior.
import {
getProviderHttpMocks,
installProviderHttpMockCleanup,
} from "openclaw/plugin-sdk/provider-http-test-mocks";
import { expectExplicitVideoGenerationCapabilities } from "openclaw/plugin-sdk/provider-test-contracts";
import { beforeAll, describe, expect, it, vi } from "vitest";
const { postJsonRequestMock, fetchWithTimeoutMock } = getProviderHttpMocks();
let buildRunwayVideoGenerationProvider: typeof import("./video-generation-provider.js").buildRunwayVideoGenerationProvider;
beforeAll(async () => {
({ buildRunwayVideoGenerationProvider } = await import("./video-generation-provider.js"));
});
installProviderHttpMockCleanup();
function firstPostJsonRequest() {
const [call] = postJsonRequestMock.mock.calls;
if (!call) {
throw new Error("expected Runway create request");
}
const [request] = call;
if (!request || typeof request !== "object") {
throw new Error("expected Runway create request options");
}
return request as { url?: string; body?: Record<string, unknown> };
}
function firstFetchWithTimeoutCall() {
const [call] = fetchWithTimeoutMock.mock.calls;
if (!call) {
throw new Error("expected Runway poll request");
}
const [url, init, timeoutMs, requestFetch] = call;
if (typeof url !== "string") {
throw new Error("expected Runway poll request URL");
}
if (!init || typeof init !== "object" || Array.isArray(init)) {
throw new Error("expected Runway poll request init");
}
if (typeof timeoutMs !== "number") {
throw new Error("expected Runway poll request timeout");
}
return {
init: init as { method?: string; headers?: unknown },
requestFetch,
timeoutMs,
url,
};
}
function streamedVideoResponse(bytes: string): Response {
return new Response(
new ReadableStream({
start(controller) {
controller.enqueue(new TextEncoder().encode(bytes));
controller.close();
},
}),
{ headers: { "content-type": "video/mp4" } },
);
}
// Response.json keeps object fixtures on the standard Response body path so create/poll
// reads exercise the byte-bounded reader instead of an unbounded res.json().
function streamedJsonResponse(payload: unknown): Response {
return Response.json(payload);
}
function streamedRawResponse(text: string): Response {
return new Response(
new ReadableStream({
start(controller) {
controller.enqueue(new TextEncoder().encode(text));
controller.close();
},
}),
{ headers: { "content-type": "application/json" } },
);
}
describe("runway video generation provider", () => {
it("declares explicit mode capabilities", () => {
expectExplicitVideoGenerationCapabilities(buildRunwayVideoGenerationProvider());
});
it("submits a text-to-video task, polls it, and downloads the output", async () => {
postJsonRequestMock.mockResolvedValue({
response: streamedJsonResponse({
id: "task-1",
}),
release: vi.fn(async () => {}),
});
fetchWithTimeoutMock
.mockResolvedValueOnce(
streamedJsonResponse({
id: "task-1",
status: "SUCCEEDED",
output: ["https://example.com/out.mp4"],
}),
)
.mockResolvedValueOnce({
arrayBuffer: async () => Buffer.from("mp4-bytes"),
headers: new Headers({ "content-type": "video/webm" }),
});
const provider = buildRunwayVideoGenerationProvider();
const result = await provider.generateVideo({
provider: "runway",
model: "gen4.5",
prompt: "a tiny lobster DJ under neon lights",
cfg: {},
durationSeconds: 4,
aspectRatio: "16:9",
});
expect(postJsonRequestMock).toHaveBeenCalledTimes(1);
const createRequest = firstPostJsonRequest();
expect(createRequest.url).toBe("https://api.dev.runwayml.com/v1/text_to_video");
expect(createRequest.body).toEqual({
model: "gen4.5",
promptText: "a tiny lobster DJ under neon lights",
ratio: "1280:720",
duration: 4,
});
const pollCall = firstFetchWithTimeoutCall();
expect(pollCall.url).toBe("https://api.dev.runwayml.com/v1/tasks/task-1");
expect(pollCall.init.method).toBe("GET");
expect(pollCall.init.headers).toBeInstanceOf(Headers);
expect(pollCall.timeoutMs).toBe(120000);
expect(pollCall.requestFetch).toBe(fetch);
expect(result.videos).toHaveLength(1);
const video = result.videos[0];
if (!video) {
throw new Error("expected Runway generated video");
}
expect(video.fileName).toBe("video-1.webm");
const metadata = result.metadata as Record<string, unknown>;
expect(metadata.taskId).toBe("task-1");
expect(metadata.status).toBe("SUCCEEDED");
expect(metadata.endpoint).toBe("/v1/text_to_video");
});
it("rejects generated video downloads that exceed the configured media cap", async () => {
postJsonRequestMock.mockResolvedValue({
response: streamedJsonResponse({ id: "task-too-large" }),
release: vi.fn(async () => {}),
});
fetchWithTimeoutMock
.mockResolvedValueOnce(
streamedJsonResponse({
id: "task-too-large",
status: "SUCCEEDED",
output: ["https://example.com/out.mp4"],
}),
)
.mockResolvedValueOnce(streamedVideoResponse("too-large"));
const provider = buildRunwayVideoGenerationProvider();
await expect(
provider.generateVideo({
provider: "runway",
model: "gen4.5",
prompt: "short video",
cfg: { agents: { defaults: { mediaMaxMb: 0.000001 } } },
}),
).rejects.toThrow("Runway generated video download exceeds 1 bytes");
});
it("does not round malformed duration values into create requests", async () => {
postJsonRequestMock.mockResolvedValue({
response: streamedJsonResponse({ id: "task-duration" }),
release: vi.fn(async () => {}),
});
fetchWithTimeoutMock
.mockResolvedValueOnce(
streamedJsonResponse({
id: "task-duration",
status: "SUCCEEDED",
output: ["https://example.com/out.mp4"],
}),
)
.mockResolvedValueOnce({
arrayBuffer: async () => Buffer.from("mp4-bytes"),
headers: new Headers({ "content-type": "video/mp4" }),
});
const provider = buildRunwayVideoGenerationProvider();
await provider.generateVideo({
provider: "runway",
model: "gen4.5",
prompt: "a tiny lobster DJ under neon lights",
cfg: {},
durationSeconds: 4.5,
aspectRatio: "16:9",
});
expect(postJsonRequestMock).toHaveBeenCalledTimes(1);
expect(firstPostJsonRequest().body?.duration).toBe(5);
});
it("accepts local image buffers by converting them into data URIs", async () => {
postJsonRequestMock.mockResolvedValue({
response: streamedJsonResponse({ id: "task-2" }),
release: vi.fn(async () => {}),
});
fetchWithTimeoutMock
.mockResolvedValueOnce(
streamedJsonResponse({
id: "task-2",
status: "SUCCEEDED",
output: ["https://example.com/out.mp4"],
}),
)
.mockResolvedValueOnce({
arrayBuffer: async () => Buffer.from("mp4-bytes"),
headers: new Headers({ "content-type": "video/mp4" }),
});
const provider = buildRunwayVideoGenerationProvider();
await provider.generateVideo({
provider: "runway",
model: "gen4_turbo",
prompt: "animate this frame",
cfg: {},
inputImages: [{ buffer: Buffer.from("png-bytes"), mimeType: "image/png" }],
aspectRatio: "1:1",
durationSeconds: 6,
});
expect(postJsonRequestMock).toHaveBeenCalledTimes(1);
const request = firstPostJsonRequest();
expect(request.url).toBe("https://api.dev.runwayml.com/v1/image_to_video");
expect(request.body?.promptImage).toMatch(/^data:image\/png;base64,/u);
expect(request.body?.ratio).toBe("960:960");
expect(request.body?.duration).toBe(6);
});
it("requires gen4_aleph for video-to-video", async () => {
const provider = buildRunwayVideoGenerationProvider();
await expect(
provider.generateVideo({
provider: "runway",
model: "gen4.5",
prompt: "restyle this clip",
cfg: {},
inputVideos: [{ url: "https://example.com/input.mp4" }],
}),
).rejects.toThrow("Runway video-to-video currently requires model gen4_aleph.");
expect(postJsonRequestMock).not.toHaveBeenCalled();
});
it("reports malformed create JSON with a provider-owned error", async () => {
const release = vi.fn(async () => {});
postJsonRequestMock.mockResolvedValue({
response: streamedRawResponse("{ not json"),
release,
});
const provider = buildRunwayVideoGenerationProvider();
await expect(
provider.generateVideo({
provider: "runway",
model: "gen4.5",
prompt: "bad create response",
cfg: {},
}),
).rejects.toThrow("Runway video generation failed: malformed JSON response");
expect(release).toHaveBeenCalledOnce();
});
it("rejects status responses missing a task status", async () => {
postJsonRequestMock.mockResolvedValue({
response: streamedJsonResponse({ id: "task-missing-status" }),
release: vi.fn(async () => {}),
});
fetchWithTimeoutMock.mockResolvedValueOnce(
streamedJsonResponse({
id: "task-missing-status",
output: ["https://example.com/out.mp4"],
}),
);
const provider = buildRunwayVideoGenerationProvider();
await expect(
provider.generateVideo({
provider: "runway",
model: "gen4.5",
prompt: "missing status",
cfg: {},
}),
).rejects.toThrow("Runway video status response missing task status");
});
it("rejects malformed completed output URLs", async () => {
postJsonRequestMock.mockResolvedValue({
response: streamedJsonResponse({ id: "task-malformed-output" }),
release: vi.fn(async () => {}),
});
fetchWithTimeoutMock.mockResolvedValueOnce(
streamedJsonResponse({
id: "task-malformed-output",
status: "SUCCEEDED",
output: "https://example.com/out.mp4",
}),
);
const provider = buildRunwayVideoGenerationProvider();
await expect(
provider.generateVideo({
provider: "runway",
model: "gen4.5",
prompt: "malformed output",
cfg: {},
}),
).rejects.toThrow("Runway video generation completed with malformed output URLs");
});
});

View File

@@ -0,0 +1,473 @@
// Runway provider module implements model/runtime integration.
import { extensionForMime } from "openclaw/plugin-sdk/media-mime";
import { isProviderApiKeyConfigured } from "openclaw/plugin-sdk/provider-auth";
import { resolveApiKeyForProvider } from "openclaw/plugin-sdk/provider-auth-runtime";
import {
assertOkOrThrowHttpError,
createProviderOperationDeadline,
createProviderOperationTimeoutResolver,
fetchProviderDownloadResponse,
fetchProviderOperationResponse,
postJsonRequest,
readProviderJsonResponse,
resolveProviderOperationTimeoutMs,
resolveProviderHttpRequestConfig,
waitProviderOperationPollInterval,
type ProviderOperationTimeoutMs,
} from "openclaw/plugin-sdk/provider-http";
import { readResponseWithLimit } from "openclaw/plugin-sdk/response-limit-runtime";
import {
isRecord,
normalizeLowercaseStringOrEmpty,
normalizeOptionalString,
} from "openclaw/plugin-sdk/string-coerce-runtime";
import type {
GeneratedVideoAsset,
VideoGenerationProvider,
VideoGenerationRequest,
VideoGenerationResult,
VideoGenerationSourceAsset,
} from "openclaw/plugin-sdk/video-generation";
const DEFAULT_RUNWAY_BASE_URL = "https://api.dev.runwayml.com";
const DEFAULT_RUNWAY_MODEL = "gen4.5";
const RUNWAY_API_VERSION = "2024-11-06";
const DEFAULT_TIMEOUT_MS = 120_000;
const POLL_INTERVAL_MS = 5_000;
const MAX_POLL_ATTEMPTS = 120;
const MAX_DURATION_SECONDS = 10;
const DEFAULT_GENERATED_VIDEO_MAX_BYTES = 16 * 1024 * 1024;
type RunwayTaskStatus = "PENDING" | "RUNNING" | "THROTTLED" | "SUCCEEDED" | "FAILED" | "CANCELLED";
type RunwayTaskCreateResponse = {
id?: unknown;
};
type RunwayTaskDetailResponse = {
id?: unknown;
status?: unknown;
output?: unknown;
failure?: unknown;
};
type RunwaySourceAsset = Pick<VideoGenerationSourceAsset, "buffer" | "mimeType" | "url">;
const TEXT_ONLY_MODELS = new Set(["gen4.5", "veo3.1", "veo3.1_fast", "veo3"]);
const IMAGE_MODELS = new Set([
"gen4.5",
"gen4_turbo",
"gen3a_turbo",
"veo3.1",
"veo3.1_fast",
"veo3",
]);
const VIDEO_MODELS = new Set(["gen4_aleph"]);
const RUNWAY_TEXT_ASPECT_RATIOS = ["16:9", "9:16"] as const;
const RUNWAY_EDIT_ASPECT_RATIOS = ["1:1", "16:9", "9:16", "3:4", "4:3", "21:9"] as const;
async function readRunwayJsonResponse<T>(response: Response, label: string): Promise<T> {
// Runway submit/poll task bodies are read through the shared byte-bounded reader
// (readResponseWithLimit, via readProviderJsonResponse) so a hostile or buggy endpoint
// that streams an unbounded JSON body cannot force the runtime to buffer the whole
// payload before parsing. Overflow cancels the stream and throws a bounded error;
// malformed JSON keeps the existing `${label}: malformed JSON response` wrapping.
const payload = await readProviderJsonResponse<unknown>(response, label);
if (!isRecord(payload)) {
throw new Error(`${label}: malformed JSON response`);
}
return payload as T;
}
function readRunwayTaskStatus(payload: RunwayTaskDetailResponse): RunwayTaskStatus {
const status = normalizeOptionalString(payload.status);
switch (status) {
case "PENDING":
case "RUNNING":
case "THROTTLED":
case "SUCCEEDED":
case "FAILED":
case "CANCELLED":
return status;
case undefined:
throw new Error("Runway video status response missing task status");
default:
throw new Error(`Runway video status response returned unknown task status: ${status}`);
}
}
function readRunwayFailureMessage(failure: unknown): string | undefined {
if (typeof failure === "string") {
return normalizeOptionalString(failure);
}
if (isRecord(failure)) {
return normalizeOptionalString(failure.message);
}
return undefined;
}
function readRunwayOutputUrls(payload: RunwayTaskDetailResponse): string[] {
if (!Array.isArray(payload.output)) {
throw new Error("Runway video generation completed with malformed output URLs");
}
const outputUrls = payload.output
.map((value) => normalizeOptionalString(value))
.filter((value): value is string => Boolean(value));
if (!outputUrls.length) {
throw new Error("Runway video generation completed without output URLs");
}
return outputUrls;
}
function resolveRunwayBaseUrl(req: VideoGenerationRequest): string {
return (
normalizeOptionalString(req.cfg?.models?.providers?.runway?.baseUrl) ?? DEFAULT_RUNWAY_BASE_URL
);
}
function resolveGeneratedVideoMaxBytes(req: VideoGenerationRequest): number {
const configured = req.cfg.agents?.defaults?.mediaMaxMb;
if (typeof configured === "number" && Number.isFinite(configured) && configured > 0) {
return Math.floor(configured * 1024 * 1024);
}
return DEFAULT_GENERATED_VIDEO_MAX_BYTES;
}
function toDataUrl(buffer: Buffer, mimeType: string): string {
return `data:${mimeType};base64,${buffer.toString("base64")}`;
}
function resolveSourceUri(
asset: RunwaySourceAsset | undefined,
fallbackMimeType: string,
): string | undefined {
if (!asset) {
return undefined;
}
const url = normalizeOptionalString(asset.url);
if (url) {
return url;
}
if (!asset.buffer) {
return undefined;
}
return toDataUrl(asset.buffer, normalizeOptionalString(asset.mimeType) ?? fallbackMimeType);
}
function resolveDurationSeconds(value: number | undefined): number {
if (typeof value !== "number" || !Number.isFinite(value)) {
return 5;
}
if (!Number.isSafeInteger(value)) {
return 5;
}
return Math.max(2, Math.min(MAX_DURATION_SECONDS, value));
}
function resolveRunwayRatio(req: VideoGenerationRequest): string {
const hasImageInput = (req.inputImages?.length ?? 0) > 0;
const requested =
normalizeOptionalString(req.size) ||
(() => {
switch (normalizeOptionalString(req.aspectRatio)) {
case "9:16":
return "720:1280";
case "16:9":
return "1280:720";
case "1:1":
return "960:960";
case "3:4":
return "832:1104";
case "4:3":
return "1104:832";
case "21:9":
return "1584:672";
default:
return undefined;
}
})();
if (requested) {
if (!hasImageInput && requested !== "1280:720" && requested !== "720:1280") {
throw new Error("Runway text-to-video currently supports only 16:9 or 9:16 output ratios.");
}
return requested;
}
return "1280:720";
}
function resolveEndpoint(
req: VideoGenerationRequest,
): "/v1/text_to_video" | "/v1/image_to_video" | "/v1/video_to_video" {
const imageCount = req.inputImages?.length ?? 0;
const videoCount = req.inputVideos?.length ?? 0;
if (imageCount > 0 && videoCount > 0) {
throw new Error("Runway video generation does not support image and video inputs together.");
}
if (imageCount > 1 || videoCount > 1) {
throw new Error("Runway video generation supports at most one input image or one input video.");
}
if (videoCount > 0) {
return "/v1/video_to_video";
}
if (imageCount > 0) {
return "/v1/image_to_video";
}
return "/v1/text_to_video";
}
function buildCreateBody(req: VideoGenerationRequest): Record<string, unknown> {
const endpoint = resolveEndpoint(req);
const duration = resolveDurationSeconds(req.durationSeconds);
const ratio = resolveRunwayRatio(req);
const model = normalizeOptionalString(req.model) ?? DEFAULT_RUNWAY_MODEL;
if (endpoint === "/v1/text_to_video") {
if (!TEXT_ONLY_MODELS.has(model)) {
throw new Error(
`Runway text-to-video does not support model ${model}. Use one of: ${[...TEXT_ONLY_MODELS].join(", ")}.`,
);
}
return {
model,
promptText: req.prompt,
ratio,
duration,
};
}
if (endpoint === "/v1/image_to_video") {
if (!IMAGE_MODELS.has(model)) {
throw new Error(
`Runway image-to-video does not support model ${model}. Use one of: ${[...IMAGE_MODELS].join(", ")}.`,
);
}
const promptImage = resolveSourceUri(req.inputImages?.[0], "image/png");
if (!promptImage) {
throw new Error("Runway image-to-video input is missing image data.");
}
return {
model,
promptText: req.prompt,
promptImage,
ratio,
duration,
};
}
if (!VIDEO_MODELS.has(model)) {
throw new Error("Runway video-to-video currently requires model gen4_aleph.");
}
const videoUri = resolveSourceUri(req.inputVideos?.[0], "video/mp4");
if (!videoUri) {
throw new Error("Runway video-to-video input is missing video data.");
}
return {
model,
promptText: req.prompt,
videoUri,
ratio,
};
}
async function pollRunwayTask(params: {
taskId: string;
headers: Headers;
timeoutMs?: number;
baseUrl: string;
fetchFn: typeof fetch;
}): Promise<RunwayTaskDetailResponse> {
const deadline = createProviderOperationDeadline({
timeoutMs: params.timeoutMs,
label: `Runway video generation task ${params.taskId}`,
});
for (let attempt = 0; attempt < MAX_POLL_ATTEMPTS; attempt += 1) {
const response = await fetchProviderOperationResponse({
stage: "poll",
url: `${params.baseUrl}/v1/tasks/${params.taskId}`,
init: {
method: "GET",
headers: params.headers,
},
timeoutMs: createProviderOperationTimeoutResolver({
deadline,
defaultTimeoutMs: DEFAULT_TIMEOUT_MS,
}),
fetchFn: params.fetchFn,
provider: "runway",
requestFailedMessage: "Runway video status request failed",
});
const payload = await readRunwayJsonResponse<RunwayTaskDetailResponse>(
response,
"Runway video status request failed",
);
const status = readRunwayTaskStatus(payload);
switch (status) {
case "SUCCEEDED":
return payload;
case "FAILED":
case "CANCELLED":
throw new Error(
readRunwayFailureMessage(payload.failure) ||
`Runway video generation ${normalizeLowercaseStringOrEmpty(status)}`,
);
default:
await waitProviderOperationPollInterval({ deadline, pollIntervalMs: POLL_INTERVAL_MS });
break;
}
}
throw new Error(`Runway video generation task ${params.taskId} did not finish in time`);
}
async function downloadRunwayVideos(params: {
urls: string[];
timeoutMs?: ProviderOperationTimeoutMs;
fetchFn: typeof fetch;
maxBytes: number;
}): Promise<GeneratedVideoAsset[]> {
const videos: GeneratedVideoAsset[] = [];
for (const [index, url] of params.urls.entries()) {
const response = await fetchProviderDownloadResponse({
url,
init: { method: "GET" },
timeoutMs: params.timeoutMs ?? DEFAULT_TIMEOUT_MS,
fetchFn: params.fetchFn,
provider: "runway",
requestFailedMessage: "Runway generated video download failed",
});
const mimeType = normalizeOptionalString(response.headers.get("content-type")) ?? "video/mp4";
const buffer = await readResponseWithLimit(response, params.maxBytes, {
onOverflow: ({ maxBytes }) =>
new Error(`Runway generated video download exceeds ${maxBytes} bytes`),
});
videos.push({
buffer,
mimeType,
fileName: `video-${index + 1}.${extensionForMime(mimeType)?.slice(1) ?? "mp4"}`,
metadata: { sourceUrl: url },
});
}
return videos;
}
export function buildRunwayVideoGenerationProvider(): VideoGenerationProvider {
return {
id: "runway",
label: "Runway",
defaultModel: DEFAULT_RUNWAY_MODEL,
models: ["gen4.5", "gen4_turbo", "gen4_aleph", "gen3a_turbo", "veo3.1", "veo3.1_fast", "veo3"],
isConfigured: ({ agentDir }) =>
isProviderApiKeyConfigured({
provider: "runway",
agentDir,
}),
capabilities: {
generate: {
maxVideos: 1,
maxDurationSeconds: MAX_DURATION_SECONDS,
aspectRatios: RUNWAY_TEXT_ASPECT_RATIOS,
supportsAspectRatio: true,
},
imageToVideo: {
enabled: true,
maxVideos: 1,
maxInputImages: 1,
maxDurationSeconds: MAX_DURATION_SECONDS,
aspectRatios: RUNWAY_EDIT_ASPECT_RATIOS,
supportsAspectRatio: true,
},
videoToVideo: {
enabled: true,
maxVideos: 1,
maxInputVideos: 1,
aspectRatios: RUNWAY_EDIT_ASPECT_RATIOS,
supportsAspectRatio: true,
},
},
async generateVideo(req): Promise<VideoGenerationResult> {
const auth = await resolveApiKeyForProvider({
provider: "runway",
cfg: req.cfg,
agentDir: req.agentDir,
store: req.authStore,
});
if (!auth.apiKey) {
throw new Error("Runway API key missing");
}
const fetchFn = fetch;
const deadline = createProviderOperationDeadline({
timeoutMs: req.timeoutMs,
label: "Runway video generation",
});
const requestBody = buildCreateBody(req);
const endpoint = resolveEndpoint(req);
const { baseUrl, allowPrivateNetwork, headers, dispatcherPolicy } =
resolveProviderHttpRequestConfig({
baseUrl: resolveRunwayBaseUrl(req),
defaultBaseUrl: DEFAULT_RUNWAY_BASE_URL,
defaultHeaders: {
Authorization: `Bearer ${auth.apiKey}`,
"Content-Type": "application/json",
"X-Runway-Version": RUNWAY_API_VERSION,
},
provider: "runway",
capability: "video",
transport: "http",
});
const { response, release } = await postJsonRequest({
url: `${baseUrl}${endpoint}`,
headers,
body: requestBody,
timeoutMs: resolveProviderOperationTimeoutMs({
deadline,
defaultTimeoutMs: DEFAULT_TIMEOUT_MS,
}),
fetchFn,
allowPrivateNetwork,
dispatcherPolicy,
});
try {
await assertOkOrThrowHttpError(response, "Runway video generation failed");
const submitted = await readRunwayJsonResponse<RunwayTaskCreateResponse>(
response,
"Runway video generation failed",
);
const taskId = normalizeOptionalString(submitted.id);
if (!taskId) {
throw new Error("Runway video generation response missing task id");
}
const completed = await pollRunwayTask({
taskId,
headers,
timeoutMs: resolveProviderOperationTimeoutMs({
deadline,
defaultTimeoutMs: DEFAULT_TIMEOUT_MS,
}),
baseUrl,
fetchFn,
});
const outputUrls = readRunwayOutputUrls(completed);
const videos = await downloadRunwayVideos({
urls: outputUrls,
timeoutMs: createProviderOperationTimeoutResolver({
deadline,
defaultTimeoutMs: DEFAULT_TIMEOUT_MS,
}),
fetchFn,
maxBytes: resolveGeneratedVideoMaxBytes(req),
});
return {
videos,
model: normalizeOptionalString(req.model) ?? DEFAULT_RUNWAY_MODEL,
metadata: {
taskId,
status: normalizeOptionalString(completed.status),
endpoint,
outputUrls,
},
};
} finally {
await release();
}
},
};
}