Vendor OpenClaw source as Adolf fork baseline
Some checks failed
ClawSweeper Dispatch / dispatch (push) Has been cancelled
CodeQL / Security High (actions) (push) Has been cancelled
CodeQL / Security High (channel-runtime-boundary) (push) Has been cancelled
CodeQL / Security High (core-auth-secrets) (push) Has been cancelled
CodeQL / Security High (mcp-process-tool-boundary) (push) Has been cancelled
CodeQL / Security High (network-ssrf-boundary) (push) Has been cancelled
CodeQL / Security High (plugin-trust-boundary) (push) Has been cancelled
CodeQL / Security High (process-exec-boundary) (push) Has been cancelled
Docs Sync Publish Repo / sync-publish-repo (push) Has been cancelled
Docs / docs (push) Has been cancelled
OpenClaw Stable Main Closeout / Resolve stable release closeout inputs (push) Has been cancelled
OpenClaw Stable Main Closeout / Verify stable main closeout (push) Has been cancelled
Workflow Sanity / no-tabs (push) Has been cancelled
Workflow Sanity / actionlint (push) Has been cancelled
Workflow Sanity / generated-doc-baselines (push) Has been cancelled
CI / runner-admission (push) Has been cancelled
CI / preflight (push) Has been cancelled
CI / security-fast (push) Has been cancelled
CI / pnpm-store-warmup (push) Has been cancelled
CI / build-artifacts (push) Has been cancelled
CI / native-i18n (push) Has been cancelled
CI / ${{ matrix.check_name }} (push) Has been cancelled
CI / ${{ matrix.checkName }} (push) Has been cancelled
CI / checks-node-compat-node22 (push) Has been cancelled
CI / check-bundled-channel-config-metadata (push) Has been cancelled
CI / check-dependencies (push) Has been cancelled
CI / check-guards (push) Has been cancelled
CI / check-lint (push) Has been cancelled
CI / check-prod-types (push) Has been cancelled
CI / check-shrinkwrap (push) Has been cancelled
CI / check-test-types (push) Has been cancelled
CI / check-additional-boundaries-a (push) Has been cancelled
CI / check-additional-boundaries-bcd (push) Has been cancelled
CI / check-additional-extension-bundled (push) Has been cancelled
CI / check-additional-extension-channels (push) Has been cancelled
CI / check-additional-extension-package-boundary (push) Has been cancelled
CI / check-additional-runtime-topology-architecture (push) Has been cancelled
CI / check-session-accessor-boundary (push) Has been cancelled
CI / check-session-transcript-reader-boundary (push) Has been cancelled
CI / check-docs (push) Has been cancelled
CI / skills-python (push) Has been cancelled
CI / macos-swift (push) Has been cancelled
CI / ios-build (push) Has been cancelled
CI / ci-timings-summary (push) Has been cancelled
Native App Locale Refresh / Refresh native fa (push) Has been cancelled
Native App Locale Refresh / Refresh native fr (push) Has been cancelled
Native App Locale Refresh / Refresh native hi (push) Has been cancelled
Native App Locale Refresh / Refresh native id (push) Has been cancelled
Native App Locale Refresh / Refresh native it (push) Has been cancelled
Native App Locale Refresh / Refresh native ja-JP (push) Has been cancelled
Control UI Locale Refresh / plan (push) Has been cancelled
Control UI Locale Refresh / Refresh ${{ matrix.locale }} (push) Has been cancelled
Control UI Locale Refresh / Commit control UI locale refresh (push) Has been cancelled
Live Media Runner Image / Build live media runner image (push) Has been cancelled
Native App Locale Refresh / Refresh native ar (push) Has been cancelled
Native App Locale Refresh / Refresh native de (push) Has been cancelled
Native App Locale Refresh / Refresh native es (push) Has been cancelled
Native App Locale Refresh / Refresh native ko (push) Has been cancelled
Native App Locale Refresh / Refresh native nl (push) Has been cancelled
Native App Locale Refresh / Refresh native pl (push) Has been cancelled
Native App Locale Refresh / Refresh native pt-BR (push) Has been cancelled
Native App Locale Refresh / Refresh native ru (push) Has been cancelled
Native App Locale Refresh / Refresh native sv (push) Has been cancelled
Native App Locale Refresh / Refresh native th (push) Has been cancelled
Native App Locale Refresh / Refresh native tr (push) Has been cancelled
Native App Locale Refresh / Refresh native uk (push) Has been cancelled
Native App Locale Refresh / Refresh native vi (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-CN (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-TW (push) Has been cancelled
Native App Locale Refresh / Commit native locale refresh (push) Has been cancelled
Plugin Init Scaffold Validation / Validate provider scaffold (push) Has been cancelled
Plugin NPM Release / preview_plugins_npm (push) Has been cancelled
Plugin NPM Release / Validate release publish approval (push) Has been cancelled
Plugin NPM Release / preview_plugin_pack (push) Has been cancelled
Plugin NPM Release / publish_plugins_npm (push) Has been cancelled
Sandbox Common Smoke / sandbox-common-smoke (push) Has been cancelled
Website Installer Sync / static (push) Has been cancelled
Website Installer Sync / linux-docker (push) Has been cancelled
Website Installer Sync / macos-installer (push) Has been cancelled
Website Installer Sync / windows-installer (push) Has been cancelled
Website Installer Sync / sync-website (push) Has been cancelled
Some checks failed
ClawSweeper Dispatch / dispatch (push) Has been cancelled
CodeQL / Security High (actions) (push) Has been cancelled
CodeQL / Security High (channel-runtime-boundary) (push) Has been cancelled
CodeQL / Security High (core-auth-secrets) (push) Has been cancelled
CodeQL / Security High (mcp-process-tool-boundary) (push) Has been cancelled
CodeQL / Security High (network-ssrf-boundary) (push) Has been cancelled
CodeQL / Security High (plugin-trust-boundary) (push) Has been cancelled
CodeQL / Security High (process-exec-boundary) (push) Has been cancelled
Docs Sync Publish Repo / sync-publish-repo (push) Has been cancelled
Docs / docs (push) Has been cancelled
OpenClaw Stable Main Closeout / Resolve stable release closeout inputs (push) Has been cancelled
OpenClaw Stable Main Closeout / Verify stable main closeout (push) Has been cancelled
Workflow Sanity / no-tabs (push) Has been cancelled
Workflow Sanity / actionlint (push) Has been cancelled
Workflow Sanity / generated-doc-baselines (push) Has been cancelled
CI / runner-admission (push) Has been cancelled
CI / preflight (push) Has been cancelled
CI / security-fast (push) Has been cancelled
CI / pnpm-store-warmup (push) Has been cancelled
CI / build-artifacts (push) Has been cancelled
CI / native-i18n (push) Has been cancelled
CI / ${{ matrix.check_name }} (push) Has been cancelled
CI / ${{ matrix.checkName }} (push) Has been cancelled
CI / checks-node-compat-node22 (push) Has been cancelled
CI / check-bundled-channel-config-metadata (push) Has been cancelled
CI / check-dependencies (push) Has been cancelled
CI / check-guards (push) Has been cancelled
CI / check-lint (push) Has been cancelled
CI / check-prod-types (push) Has been cancelled
CI / check-shrinkwrap (push) Has been cancelled
CI / check-test-types (push) Has been cancelled
CI / check-additional-boundaries-a (push) Has been cancelled
CI / check-additional-boundaries-bcd (push) Has been cancelled
CI / check-additional-extension-bundled (push) Has been cancelled
CI / check-additional-extension-channels (push) Has been cancelled
CI / check-additional-extension-package-boundary (push) Has been cancelled
CI / check-additional-runtime-topology-architecture (push) Has been cancelled
CI / check-session-accessor-boundary (push) Has been cancelled
CI / check-session-transcript-reader-boundary (push) Has been cancelled
CI / check-docs (push) Has been cancelled
CI / skills-python (push) Has been cancelled
CI / macos-swift (push) Has been cancelled
CI / ios-build (push) Has been cancelled
CI / ci-timings-summary (push) Has been cancelled
Native App Locale Refresh / Refresh native fa (push) Has been cancelled
Native App Locale Refresh / Refresh native fr (push) Has been cancelled
Native App Locale Refresh / Refresh native hi (push) Has been cancelled
Native App Locale Refresh / Refresh native id (push) Has been cancelled
Native App Locale Refresh / Refresh native it (push) Has been cancelled
Native App Locale Refresh / Refresh native ja-JP (push) Has been cancelled
Control UI Locale Refresh / plan (push) Has been cancelled
Control UI Locale Refresh / Refresh ${{ matrix.locale }} (push) Has been cancelled
Control UI Locale Refresh / Commit control UI locale refresh (push) Has been cancelled
Live Media Runner Image / Build live media runner image (push) Has been cancelled
Native App Locale Refresh / Refresh native ar (push) Has been cancelled
Native App Locale Refresh / Refresh native de (push) Has been cancelled
Native App Locale Refresh / Refresh native es (push) Has been cancelled
Native App Locale Refresh / Refresh native ko (push) Has been cancelled
Native App Locale Refresh / Refresh native nl (push) Has been cancelled
Native App Locale Refresh / Refresh native pl (push) Has been cancelled
Native App Locale Refresh / Refresh native pt-BR (push) Has been cancelled
Native App Locale Refresh / Refresh native ru (push) Has been cancelled
Native App Locale Refresh / Refresh native sv (push) Has been cancelled
Native App Locale Refresh / Refresh native th (push) Has been cancelled
Native App Locale Refresh / Refresh native tr (push) Has been cancelled
Native App Locale Refresh / Refresh native uk (push) Has been cancelled
Native App Locale Refresh / Refresh native vi (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-CN (push) Has been cancelled
Native App Locale Refresh / Refresh native zh-TW (push) Has been cancelled
Native App Locale Refresh / Commit native locale refresh (push) Has been cancelled
Plugin Init Scaffold Validation / Validate provider scaffold (push) Has been cancelled
Plugin NPM Release / preview_plugins_npm (push) Has been cancelled
Plugin NPM Release / Validate release publish approval (push) Has been cancelled
Plugin NPM Release / preview_plugin_pack (push) Has been cancelled
Plugin NPM Release / publish_plugins_npm (push) Has been cancelled
Sandbox Common Smoke / sandbox-common-smoke (push) Has been cancelled
Website Installer Sync / static (push) Has been cancelled
Website Installer Sync / linux-docker (push) Has been cancelled
Website Installer Sync / macos-installer (push) Has been cancelled
Website Installer Sync / windows-installer (push) Has been cancelled
Website Installer Sync / sync-website (push) Has been cancelled
Adolf is a fork/vendored clone of github.com/openclaw/openclaw (v2026.6.11), free to diverge. Tree copied sans upstream .git; upstream remote added for future syncs. Node pinned to 24 (.nvmrc); engines already require >=22.19. Preserves docs/ARCHITECTURE.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LeqyaxJF2nbRXJtae2kNB2
This commit is contained in:
29
scripts/AGENTS.md
Normal file
29
scripts/AGENTS.md
Normal file
@@ -0,0 +1,29 @@
|
||||
# Scripts Guide
|
||||
|
||||
This directory owns local tooling, script wrappers, and generated-artifact helper rules.
|
||||
|
||||
## Wrapper Rules
|
||||
|
||||
- Prefer existing wrappers over raw tool entrypoints when the repo already has a curated seam.
|
||||
- For tests, prefer `scripts/run-vitest.mjs` or the root `pnpm test ...` entrypoints over raw `vitest run` calls.
|
||||
- Never use bare `vitest ...` in automation; it starts local watch mode unless `run` or `--run` is explicit.
|
||||
- For lint/typecheck flows, prefer `scripts/run-oxlint.mjs` and `scripts/run-tsgo.mjs` when adding or editing package scripts or CI steps that should honor repo-local runtime behavior.
|
||||
- For changed-file verification, prefer `scripts/check-changed.mjs` and keep lane classification in `scripts/changed-lanes.mjs`. Use `node scripts/check-changed.mjs --dry-run [--staged|-- <files...>]` to inspect the plan before running anything expensive. Do not copy path-scope rules into new hooks or ad hoc CI snippets.
|
||||
- For one/few lint files, prefer direct `node scripts/run-oxlint.mjs --tsconfig <matching config> <files...>` over sharded `pnpm lint`; `check-changed.mjs` owns this targeting for core, extension, and script diffs.
|
||||
|
||||
## Local Heavy-Check Lock
|
||||
|
||||
- Respect the local heavy-check lock behavior in `scripts/lib/local-heavy-check-runtime.mjs`.
|
||||
- Do not bypass that lock for real heavy commands just to make a local loop look faster.
|
||||
- Metadata-only or explicitly narrow commands may skip the lock when the existing helper logic says that is safe.
|
||||
- If you change the lock heuristics, add or update the narrow tests under `test/scripts/`.
|
||||
|
||||
## Generated Outputs
|
||||
|
||||
- If a script writes generated artifacts, keep the source-of-truth generator, the package script, and the matching verification/check command aligned.
|
||||
- Prefer additive generator/check pairs like `*:gen` and `*:check` over one-off undocumented scripts.
|
||||
|
||||
## Scope
|
||||
|
||||
- Keep script-runner behavior, wrapper expectations, and generated-artifact guidance here.
|
||||
- Leave repo-global verification policy in the root `AGENTS.md`.
|
||||
1
scripts/CLAUDE.md
Symbolic link
1
scripts/CLAUDE.md
Symbolic link
@@ -0,0 +1 @@
|
||||
AGENTS.md
|
||||
75
scripts/README.md
Normal file
75
scripts/README.md
Normal file
@@ -0,0 +1,75 @@
|
||||
---
|
||||
summary: "Repository script entry points and compatibility notes"
|
||||
read_when:
|
||||
- Looking for an existing script before adding a new one
|
||||
- Running repository checks, tests, docs, Docker, release, or GitHub helper scripts
|
||||
- Updating package scripts or CI workflow script references
|
||||
title: "Scripts Directory"
|
||||
---
|
||||
|
||||
# Scripts Directory
|
||||
|
||||
The `scripts/` directory contains repository tooling used by local development,
|
||||
CI, docs publishing, releases, Docker proof, and maintainer operations. Prefer
|
||||
the package-script entry points in `package.json` when one exists, then read the
|
||||
underlying script before running it directly.
|
||||
|
||||
## Compatibility
|
||||
|
||||
Many scripts are stable paths referenced by `package.json`, GitHub Actions,
|
||||
docs, and maintainer runbooks. Do not move, rename, or regroup scripts only to
|
||||
improve taxonomy. A directory migration needs an explicit maintainer-approved
|
||||
compatibility plan for package scripts, workflows, docs snippets, and any raw
|
||||
script paths users may have copied.
|
||||
|
||||
This index is a discovery aid for the current flat layout. It does not define a
|
||||
new directory taxonomy.
|
||||
|
||||
## Common Entry Points
|
||||
|
||||
| Area | Prefer | Notes |
|
||||
| --------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------ |
|
||||
| Build | `pnpm build` | Runs `scripts/build-all.mjs`; use specific build scripts only when debugging a build stage. |
|
||||
| Changed checks | `pnpm changed:lanes --json`, `pnpm check:changed` | Lane classification lives in `scripts/changed-lanes.mjs`; changed-file checks live in `scripts/check-changed.mjs`. |
|
||||
| Docs | `pnpm docs:list`, `pnpm docs:check-mdx`, `pnpm docs:check-links` | Backed by `scripts/docs-list.js`, `scripts/check-docs-mdx.mjs`, and `scripts/docs-link-audit.mjs`. |
|
||||
| Formatting docs | `pnpm format:docs:check` | Uses `scripts/format-docs.mjs`; use write mode only when intentionally formatting docs. |
|
||||
| Lint | `pnpm lint`, `pnpm lint:core`, `pnpm lint:all` | Wrapper scripts keep oxlint behavior aligned with repo config. |
|
||||
| Targeted tests | `pnpm test <path-or-filter>` or `node scripts/run-vitest.mjs <path-or-filter>` | Avoid bare `vitest`; it can start watch mode. |
|
||||
| Changed tests | `pnpm test:changed` | Uses the repo's changed-test resolver instead of a broad Vitest run. |
|
||||
| Docker proof | `pnpm test:docker:all`, `pnpm test:docker:rerun`, `pnpm test:docker:timings` | Use the planner/rerun helpers before launching broad Docker work. |
|
||||
| Live proof | `pnpm test:live` | Live checks require the matching environment and credentials. |
|
||||
| Release checks | `pnpm release:check`, `pnpm release:beta`, `pnpm release:candidate` | Release scripts are maintainer workflows; read release docs before use. |
|
||||
| GitHub reads | `scripts/gh-read` | Uses a GitHub App read token when configured, leaving normal `gh` login for writes. |
|
||||
| Commits | `scripts/committer "<message>" <files...>` | Preferred scoped commit helper for OpenClaw changes. |
|
||||
| Remote proof | `node scripts/crabbox-wrapper.mjs ...` | Agent default for tests and heavy work; pre-warm by source trust, sync each run, reuse the lease. |
|
||||
|
||||
## Script Families
|
||||
|
||||
- `check-*.mjs` / `check-*.ts`: guardrails for architecture, docs, package
|
||||
contents, boundaries, workflows, and generated artifacts.
|
||||
- `run-*.mjs`: wrappers around repo runtimes or tools, such as Node, Vitest,
|
||||
oxlint, tsgo, and environment setup.
|
||||
- `test-*.mjs` / `test-*.sh` / `test-*.ts`: test planners, Docker lanes, live
|
||||
checks, and focused validation helpers.
|
||||
- `docs-*` and `check-docs-*`: docs listing, link auditing, MDX checks,
|
||||
spellcheck, sync, and i18n glossary checks.
|
||||
- `release-*`, `openclaw-npm-*`, and `plugin-*-release-*`: release preparation,
|
||||
package verification, and publishing helpers.
|
||||
- `docker-*`, `test-docker-*`, and `test-live-*-docker.sh`: Docker E2E planning,
|
||||
rerun, timing, and live/package lane helpers.
|
||||
- `gh-read*`, `label-*`, `sync-labels.ts`, and PR helpers: GitHub read,
|
||||
labeling, and maintainer workflow support.
|
||||
- `generate-*`, `write-*`, `copy-*`, and `sync-*`: generated docs, metadata,
|
||||
package surfaces, and build artifact support.
|
||||
- `lib/`: shared helpers imported by script entry points.
|
||||
|
||||
## Maintenance Rules
|
||||
|
||||
- Read `scripts/AGENTS.md` before changing scripts.
|
||||
- Keep package scripts, generators, generated-artifact checks, docs references,
|
||||
and workflow references aligned when touching a script path.
|
||||
- Prefer existing wrappers instead of introducing a raw tool invocation.
|
||||
- Add or update focused tests under `test/scripts/` when changing script
|
||||
behavior.
|
||||
|
||||
See also [Scripts](https://docs.openclaw.ai/help/scripts) for public-facing script guidance.
|
||||
10
scripts/analyze-plugin-sdk-usage.ts
Normal file
10
scripts/analyze-plugin-sdk-usage.ts
Normal file
@@ -0,0 +1,10 @@
|
||||
#!/usr/bin/env node
|
||||
// Analyze Plugin Sdk Usage script supports OpenClaw repository automation.
|
||||
import { main } from "./ts-topology.ts";
|
||||
|
||||
const forwardedArgs = process.argv.slice(2);
|
||||
const normalizedArgs = forwardedArgs[0] === "--" ? forwardedArgs.slice(1) : forwardedArgs;
|
||||
const exitCode = await main(["--scope=plugin-sdk", ...normalizedArgs]);
|
||||
if (exitCode !== 0) {
|
||||
process.exit(exitCode);
|
||||
}
|
||||
111
scripts/android-app-i18n.ts
Normal file
111
scripts/android-app-i18n.ts
Normal file
@@ -0,0 +1,111 @@
|
||||
import { readdir, readFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { NATIVE_I18N_LOCALES } from "./native-app-i18n.ts";
|
||||
|
||||
const HERE = path.dirname(fileURLToPath(import.meta.url));
|
||||
const ROOT = path.resolve(HERE, "..");
|
||||
const RESOURCE_ROOT = path.join(ROOT, "apps", "android", "app", "src", "main", "res");
|
||||
const SOURCE_ROOT = path.join(ROOT, "apps", "android", "app", "src", "main");
|
||||
const ANDROID_QUALIFIERS: Record<string, string> = {
|
||||
id: "in",
|
||||
"zh-CN": "zh-rCN",
|
||||
"zh-TW": "zh-rTW",
|
||||
"pt-BR": "pt-rBR",
|
||||
"ja-JP": "ja",
|
||||
};
|
||||
const localeDirectory = (locale: string) => `values-${ANDROID_QUALIFIERS[locale] ?? locale}`;
|
||||
const LOCALES = ["values", ...NATIVE_I18N_LOCALES.map(localeDirectory)] as const;
|
||||
const STRING_RE = /<string\s+name="([A-Za-z0-9_]+)"[^>]*>([\s\S]*?)<\/string>/gu;
|
||||
const FORMAT_RE = /%\d+\$[a-z]/giu;
|
||||
const INVALID_APOSTROPHE_RE = /(?:'|(?<!\\)')/u;
|
||||
|
||||
async function readStrings(locale: string): Promise<Map<string, string>> {
|
||||
const source = await readFile(path.join(RESOURCE_ROOT, locale, "strings.xml"), "utf8");
|
||||
return new Map(
|
||||
[...source.matchAll(STRING_RE)]
|
||||
.map((match) => [match[1], match[2]] as const)
|
||||
.filter((entry): entry is readonly [string, string] => Boolean(entry[0] && entry[1])),
|
||||
);
|
||||
}
|
||||
|
||||
async function readAndroidSource(root = SOURCE_ROOT): Promise<string> {
|
||||
const entries = await readdir(root, { withFileTypes: true });
|
||||
const sources: string[] = [];
|
||||
for (const entry of entries) {
|
||||
const fullPath = path.join(root, entry.name);
|
||||
if (entry.isDirectory()) {
|
||||
if (fullPath.startsWith(`${RESOURCE_ROOT}${path.sep}values`)) {
|
||||
continue;
|
||||
}
|
||||
sources.push(await readAndroidSource(fullPath));
|
||||
continue;
|
||||
}
|
||||
if (entry.isFile() && /\.(?:kt|kts|xml)$/u.test(entry.name)) {
|
||||
sources.push(await readFile(fullPath, "utf8"));
|
||||
}
|
||||
}
|
||||
return sources.join("\n");
|
||||
}
|
||||
|
||||
function findInvalidResourceSyntax(strings: Map<string, string>): string[] {
|
||||
return [...strings]
|
||||
.filter(([, value]) => {
|
||||
const trimmed = value.trim();
|
||||
const isQuoted = trimmed.startsWith('"') && trimmed.endsWith('"');
|
||||
return !isQuoted && INVALID_APOSTROPHE_RE.test(trimmed);
|
||||
})
|
||||
.map(([key]) => key);
|
||||
}
|
||||
|
||||
export async function checkAndroidAppI18n() {
|
||||
const [source, localeStrings] = await Promise.all([
|
||||
readAndroidSource(),
|
||||
Promise.all(LOCALES.map(readStrings)),
|
||||
]);
|
||||
const [base, ...translations] = localeStrings;
|
||||
const baseKeys = new Set(base.keys());
|
||||
const problems = translations.flatMap((strings, index) => {
|
||||
const locale = NATIVE_I18N_LOCALES[index];
|
||||
const keys = new Set(strings.keys());
|
||||
const placeholderMismatches = [...base].flatMap(([key, sourceValue]) => {
|
||||
const translatedValue = strings.get(key);
|
||||
if (!translatedValue) {
|
||||
return [];
|
||||
}
|
||||
const expected = [...sourceValue.matchAll(FORMAT_RE)].map((match) => match[0]).toSorted();
|
||||
const actual = [...translatedValue.matchAll(FORMAT_RE)].map((match) => match[0]).toSorted();
|
||||
return expected.join("\u0000") === actual.join("\u0000") ? [] : [key];
|
||||
});
|
||||
return [
|
||||
[`${locale} missing`, [...baseKeys].filter((key) => !keys.has(key))],
|
||||
[`${locale} extra`, [...keys].filter((key) => !baseKeys.has(key))],
|
||||
[`${locale} placeholders`, placeholderMismatches],
|
||||
[`${locale} syntax`, findInvalidResourceSyntax(strings)],
|
||||
] as const;
|
||||
});
|
||||
problems.push(["English syntax", findInvalidResourceSyntax(base)]);
|
||||
const unusedBaseKeys = [...baseKeys].filter(
|
||||
(key) => !source.includes(`R.string.${key}`) && !source.includes(`@string/${key}`),
|
||||
);
|
||||
problems.push(["English unused", unusedBaseKeys]);
|
||||
if (problems.some(([, keys]) => keys.length)) {
|
||||
throw new Error(
|
||||
[
|
||||
"Android app i18n resources are out of sync.",
|
||||
...problems.map(([label, keys]) => `${label}=${keys.join(",") || "none"}`),
|
||||
].join("\n"),
|
||||
);
|
||||
}
|
||||
process.stdout.write(
|
||||
`android-app-i18n: keys=${baseKeys.size} locales=${NATIVE_I18N_LOCALES.join(",")}\n`,
|
||||
);
|
||||
}
|
||||
|
||||
if (process.argv[1] && import.meta.url === `file://${path.resolve(process.argv[1])}`) {
|
||||
const [command] = process.argv.slice(2);
|
||||
if (command !== "check") {
|
||||
throw new Error("usage: node --import tsx scripts/android-app-i18n.ts check");
|
||||
}
|
||||
await checkAndroidAppI18n();
|
||||
}
|
||||
202
scripts/android-pin-version.ts
Normal file
202
scripts/android-pin-version.ts
Normal file
@@ -0,0 +1,202 @@
|
||||
// Android Pin Version script supports OpenClaw repository automation.
|
||||
import path from "node:path";
|
||||
import {
|
||||
canonicalAndroidVersionCode,
|
||||
normalizeAndroidVersionCode,
|
||||
normalizePinnedAndroidVersion,
|
||||
resolveAndroidVersion,
|
||||
resolveGatewayVersionForAndroidRelease,
|
||||
syncAndroidVersioning,
|
||||
writeAndroidVersionManifest,
|
||||
} from "./lib/android-version.ts";
|
||||
|
||||
type CliOptions = {
|
||||
explicitVersion: string | null;
|
||||
explicitVersionCode: number | null;
|
||||
fromGateway: boolean;
|
||||
rootDir: string;
|
||||
sync: boolean;
|
||||
};
|
||||
|
||||
export type PinAndroidVersionResult = {
|
||||
previousVersion: string | null;
|
||||
previousVersionCode: number | null;
|
||||
nextVersion: string;
|
||||
nextVersionCode: number;
|
||||
packageVersion: string | null;
|
||||
versionFilePath: string;
|
||||
syncedPaths: string[];
|
||||
};
|
||||
|
||||
function usage(): string {
|
||||
return [
|
||||
"Usage: node --import tsx scripts/android-pin-version.ts (--from-gateway | --version <YYYY.M.PATCH>) [--version-code <int>] [--no-sync] [--root dir]",
|
||||
"",
|
||||
"Examples:",
|
||||
" node --import tsx scripts/android-pin-version.ts --from-gateway",
|
||||
" node --import tsx scripts/android-pin-version.ts --version 2026.6.5",
|
||||
" node --import tsx scripts/android-pin-version.ts --version 2026.6.5 --version-code 2026060502",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
function parseExplicitVersionCode(raw: string): number {
|
||||
const text = raw.trim();
|
||||
if (!/^[1-9]\d*$/u.test(text)) {
|
||||
throw new Error(`Invalid value for --version-code: ${raw}. Expected a positive integer.`);
|
||||
}
|
||||
const value = Number(text);
|
||||
if (!Number.isSafeInteger(value)) {
|
||||
throw new Error(`Invalid value for --version-code: ${raw}. Expected a safe integer.`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
export function parseArgs(argv: string[]): CliOptions {
|
||||
let explicitVersion: string | null = null;
|
||||
let explicitVersionCode: number | null = null;
|
||||
let fromGateway = false;
|
||||
let rootDir = path.resolve(".");
|
||||
let sync = true;
|
||||
|
||||
for (let index = 0; index < argv.length; index += 1) {
|
||||
const arg = argv[index];
|
||||
switch (arg) {
|
||||
case "--from-gateway": {
|
||||
fromGateway = true;
|
||||
break;
|
||||
}
|
||||
case "--version": {
|
||||
explicitVersion = readOptionValue(argv, index, "--version");
|
||||
index += 1;
|
||||
break;
|
||||
}
|
||||
case "--version-code": {
|
||||
const value = readOptionValue(argv, index, "--version-code");
|
||||
explicitVersionCode = parseExplicitVersionCode(value);
|
||||
index += 1;
|
||||
break;
|
||||
}
|
||||
case "--no-sync": {
|
||||
sync = false;
|
||||
break;
|
||||
}
|
||||
case "--root": {
|
||||
const value = readOptionValue(argv, index, "--root");
|
||||
rootDir = path.resolve(value);
|
||||
index += 1;
|
||||
break;
|
||||
}
|
||||
case "-h":
|
||||
case "--help": {
|
||||
console.log(`${usage()}\n`);
|
||||
process.exit(0);
|
||||
}
|
||||
default: {
|
||||
throw new Error(`Unknown argument: ${arg}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (fromGateway === (explicitVersion !== null)) {
|
||||
throw new Error("Choose exactly one of --from-gateway or --version <YYYY.M.PATCH>.");
|
||||
}
|
||||
|
||||
if (explicitVersion !== null && !explicitVersion.trim()) {
|
||||
throw new Error("Missing value for --version.");
|
||||
}
|
||||
|
||||
return { explicitVersion, explicitVersionCode, fromGateway, rootDir, sync };
|
||||
}
|
||||
|
||||
function readOptionValue(argv: string[], index: number, flag: string): string {
|
||||
const value = argv[index + 1];
|
||||
if (value === undefined || value === "" || value.startsWith("-")) {
|
||||
throw new Error(`Missing value for ${flag}.`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
export function pinAndroidVersion(params: CliOptions): PinAndroidVersionResult {
|
||||
const rootDir = path.resolve(params.rootDir);
|
||||
let previousVersion: string | null;
|
||||
let previousVersionCode: number | null;
|
||||
try {
|
||||
const currentVersion = resolveAndroidVersion(rootDir);
|
||||
previousVersion = currentVersion.canonicalVersion;
|
||||
previousVersionCode = currentVersion.versionCode;
|
||||
} catch {
|
||||
previousVersion = null;
|
||||
previousVersionCode = null;
|
||||
}
|
||||
|
||||
const gatewayVersion = params.fromGateway
|
||||
? resolveGatewayVersionForAndroidRelease(rootDir)
|
||||
: null;
|
||||
const packageVersion = gatewayVersion?.packageVersion ?? null;
|
||||
const nextVersion =
|
||||
gatewayVersion?.pinnedAndroidVersion ??
|
||||
normalizePinnedAndroidVersion(params.explicitVersion ?? "");
|
||||
const nextVersionCode =
|
||||
params.explicitVersionCode === null
|
||||
? (gatewayVersion?.versionCode ?? canonicalAndroidVersionCode(nextVersion))
|
||||
: normalizeAndroidVersionCode(params.explicitVersionCode, nextVersion);
|
||||
const versionFilePath = writeAndroidVersionManifest(nextVersion, nextVersionCode, rootDir);
|
||||
const syncedPaths = params.sync
|
||||
? syncAndroidVersioning({ mode: "write", rootDir }).updatedPaths
|
||||
: [];
|
||||
|
||||
return {
|
||||
previousVersion,
|
||||
previousVersionCode,
|
||||
nextVersion,
|
||||
nextVersionCode,
|
||||
packageVersion,
|
||||
versionFilePath,
|
||||
syncedPaths,
|
||||
};
|
||||
}
|
||||
|
||||
export async function main(argv: string[]): Promise<number> {
|
||||
try {
|
||||
const options = parseArgs(argv);
|
||||
const result = pinAndroidVersion(options);
|
||||
const sourceText = result.packageVersion
|
||||
? ` from gateway version ${result.packageVersion}`
|
||||
: "";
|
||||
process.stdout.write(
|
||||
`Pinned Android version to ${result.nextVersion} (${result.nextVersionCode})${sourceText}.\n`,
|
||||
);
|
||||
if (
|
||||
result.previousVersion &&
|
||||
(result.previousVersion !== result.nextVersion ||
|
||||
result.previousVersionCode !== result.nextVersionCode)
|
||||
) {
|
||||
process.stdout.write(
|
||||
`Previous pinned Android version: ${result.previousVersion} (${result.previousVersionCode}).\n`,
|
||||
);
|
||||
}
|
||||
process.stdout.write(
|
||||
`Updated version manifest: ${path.relative(process.cwd(), result.versionFilePath)}\n`,
|
||||
);
|
||||
if (options.sync) {
|
||||
if (result.syncedPaths.length === 0) {
|
||||
process.stdout.write("Android versioning artifacts already up to date.\n");
|
||||
} else {
|
||||
process.stdout.write(
|
||||
`Updated Android versioning artifacts:\n- ${result.syncedPaths.map((filePath) => path.relative(process.cwd(), filePath)).join("\n- ")}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
return 0;
|
||||
} catch (error) {
|
||||
process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
if (import.meta.url === `file://${process.argv[1]}`) {
|
||||
const exitCode = await main(process.argv.slice(2));
|
||||
if (exitCode !== 0) {
|
||||
process.exit(exitCode);
|
||||
}
|
||||
}
|
||||
457
scripts/android-release-signing.mjs
Normal file
457
scripts/android-release-signing.mjs
Normal file
@@ -0,0 +1,457 @@
|
||||
#!/usr/bin/env node
|
||||
import { execFileSync } from "node:child_process";
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import process from "node:process";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const rootDir = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const defaultManifestPath = path.join(rootDir, "apps", "android", "Config", "ReleaseSigning.json");
|
||||
const requiredPropertyNames = [
|
||||
"OPENCLAW_ANDROID_STORE_FILE",
|
||||
"OPENCLAW_ANDROID_STORE_PASSWORD",
|
||||
"OPENCLAW_ANDROID_KEY_ALIAS",
|
||||
"OPENCLAW_ANDROID_KEY_PASSWORD",
|
||||
];
|
||||
const sourceRequiredPropertyNames = requiredPropertyNames.filter(
|
||||
(name) => name !== "OPENCLAW_ANDROID_STORE_FILE",
|
||||
);
|
||||
|
||||
function usage() {
|
||||
process.stdout.write(`Usage:
|
||||
scripts/android-release-signing.mjs --mode plan
|
||||
scripts/android-release-signing.mjs --mode check
|
||||
scripts/android-release-signing.mjs --mode sync-pull
|
||||
scripts/android-release-signing.mjs --mode sync-push --keystore PATH --properties PATH
|
||||
|
||||
Options:
|
||||
--manifest PATH Defaults to apps/android/Config/ReleaseSigning.json.
|
||||
--workspace PATH Defaults to <materializedRoot>/apps-signing.
|
||||
--materialized-dir PATH Defaults to materializedRoot from the manifest.
|
||||
--keystore PATH Upload keystore source for --mode sync-push.
|
||||
--properties PATH Signing properties source for --mode sync-push.
|
||||
|
||||
sync-pull and sync-push use MATCH_PASSWORD to decrypt/encrypt Android release
|
||||
signing assets in the shared apps-signing repository.
|
||||
`);
|
||||
}
|
||||
|
||||
function parseArgs(argv) {
|
||||
const options = {
|
||||
mode: "",
|
||||
manifestPath: defaultManifestPath,
|
||||
workspace: "",
|
||||
materializedDir: "",
|
||||
keystorePath: process.env.OPENCLAW_ANDROID_UPLOAD_KEYSTORE || "",
|
||||
propertiesPath: process.env.OPENCLAW_ANDROID_SIGNING_PROPERTIES || "",
|
||||
};
|
||||
|
||||
for (let index = 0; index < argv.length; index += 1) {
|
||||
const arg = argv[index];
|
||||
if (arg === "--mode") {
|
||||
options.mode = readOptionValue(argv, index, arg);
|
||||
index += 1;
|
||||
} else if (arg === "--manifest") {
|
||||
options.manifestPath = path.resolve(readOptionValue(argv, index, arg));
|
||||
index += 1;
|
||||
} else if (arg === "--workspace") {
|
||||
options.workspace = path.resolve(readOptionValue(argv, index, arg));
|
||||
index += 1;
|
||||
} else if (arg === "--materialized-dir") {
|
||||
options.materializedDir = path.resolve(readOptionValue(argv, index, arg));
|
||||
index += 1;
|
||||
} else if (arg === "--keystore") {
|
||||
options.keystorePath = path.resolve(readOptionValue(argv, index, arg));
|
||||
index += 1;
|
||||
} else if (arg === "--properties") {
|
||||
options.propertiesPath = path.resolve(readOptionValue(argv, index, arg));
|
||||
index += 1;
|
||||
} else if (arg === "-h" || arg === "--help") {
|
||||
usage();
|
||||
process.exit(0);
|
||||
} else {
|
||||
throw new Error(`Unknown argument: ${arg}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (!options.mode) {
|
||||
throw new Error("Missing required --mode.");
|
||||
}
|
||||
|
||||
return options;
|
||||
}
|
||||
|
||||
function readOptionValue(argv, index, option) {
|
||||
const value = argv[index + 1] ?? "";
|
||||
if (!value || value.startsWith("-")) {
|
||||
throw new Error(`Missing value for ${option}.`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function requireString(value, key) {
|
||||
if (typeof value !== "string" || value.trim() === "") {
|
||||
throw new Error(`Android release signing manifest missing ${key}.`);
|
||||
}
|
||||
return value.trim();
|
||||
}
|
||||
|
||||
function readManifest(manifestPath) {
|
||||
const parsed = JSON.parse(fs.readFileSync(manifestPath, "utf8"));
|
||||
const manifest = {
|
||||
signingRepo: requireString(parsed.signingRepo, "signingRepo"),
|
||||
signingBranch: requireString(parsed.signingBranch, "signingBranch"),
|
||||
assetPath: requireString(parsed.assetPath, "assetPath"),
|
||||
uploadKeystoreEncryptedFile: requireString(
|
||||
parsed.uploadKeystoreEncryptedFile,
|
||||
"uploadKeystoreEncryptedFile",
|
||||
),
|
||||
gradlePropertiesEncryptedFile: requireString(
|
||||
parsed.gradlePropertiesEncryptedFile,
|
||||
"gradlePropertiesEncryptedFile",
|
||||
),
|
||||
materializedRoot: requireString(parsed.materializedRoot, "materializedRoot"),
|
||||
gradlePropertyNames: parsed.gradlePropertyNames,
|
||||
};
|
||||
|
||||
if (
|
||||
!Array.isArray(manifest.gradlePropertyNames) ||
|
||||
manifest.gradlePropertyNames.length !== requiredPropertyNames.length ||
|
||||
!requiredPropertyNames.every((name) => manifest.gradlePropertyNames.includes(name))
|
||||
) {
|
||||
throw new Error(
|
||||
`Android release signing manifest must list Gradle properties: ${requiredPropertyNames.join(", ")}.`,
|
||||
);
|
||||
}
|
||||
|
||||
return manifest;
|
||||
}
|
||||
|
||||
function relativePath(filePath) {
|
||||
const relative = path.relative(rootDir, filePath);
|
||||
return relative && !relative.startsWith("..") ? relative : filePath;
|
||||
}
|
||||
|
||||
function resolveMaterializedDir(manifest, options) {
|
||||
return options.materializedDir || path.resolve(rootDir, manifest.materializedRoot);
|
||||
}
|
||||
|
||||
function resolveWorkspace(manifest, options) {
|
||||
return options.workspace || path.join(resolveMaterializedDir(manifest, options), "apps-signing");
|
||||
}
|
||||
|
||||
function assertWorkspaceInsideMaterialized(workspace, materializedDir) {
|
||||
const resolvedWorkspace = path.resolve(workspace);
|
||||
const resolvedMaterializedDir = path.resolve(materializedDir);
|
||||
const relative = path.relative(resolvedMaterializedDir, resolvedWorkspace);
|
||||
if (!relative || relative.startsWith("..") || path.isAbsolute(relative)) {
|
||||
throw new Error(
|
||||
`Android signing workspace must be inside ${relativePath(resolvedMaterializedDir)}.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function assetDir(workspace, manifest) {
|
||||
return path.join(workspace, manifest.assetPath);
|
||||
}
|
||||
|
||||
function encryptedKeystorePath(workspace, manifest) {
|
||||
return path.join(assetDir(workspace, manifest), manifest.uploadKeystoreEncryptedFile);
|
||||
}
|
||||
|
||||
function encryptedPropertiesPath(workspace, manifest) {
|
||||
return path.join(assetDir(workspace, manifest), manifest.gradlePropertiesEncryptedFile);
|
||||
}
|
||||
|
||||
function materializedKeystorePath(materializedDir) {
|
||||
return path.join(materializedDir, "upload-keystore.jks");
|
||||
}
|
||||
|
||||
function materializedPropertiesPath(materializedDir) {
|
||||
return path.join(materializedDir, "gradle.properties");
|
||||
}
|
||||
|
||||
function requireMatchPassword() {
|
||||
if (!process.env.MATCH_PASSWORD || process.env.MATCH_PASSWORD.trim() === "") {
|
||||
throw new Error("MATCH_PASSWORD is required for Android release signing sync.");
|
||||
}
|
||||
}
|
||||
|
||||
function run(command, args, options = {}) {
|
||||
execFileSync(command, args, {
|
||||
cwd: options.cwd,
|
||||
env: options.env || process.env,
|
||||
stdio: options.stdio || "pipe",
|
||||
});
|
||||
}
|
||||
|
||||
function runText(command, args, options = {}) {
|
||||
return execFileSync(command, args, {
|
||||
cwd: options.cwd,
|
||||
env: options.env || process.env,
|
||||
encoding: "utf8",
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
}
|
||||
|
||||
function cloneSigningRepo(manifest, workspace, materializedDir) {
|
||||
assertWorkspaceInsideMaterialized(workspace, materializedDir);
|
||||
fs.rmSync(workspace, { recursive: true, force: true });
|
||||
fs.mkdirSync(path.dirname(workspace), { recursive: true });
|
||||
run("git", ["clone", "--branch", manifest.signingBranch, manifest.signingRepo, workspace]);
|
||||
}
|
||||
|
||||
function opensslCrypt({ decrypt, inputPath, outputPath }) {
|
||||
requireMatchPassword();
|
||||
fs.mkdirSync(path.dirname(outputPath), { recursive: true });
|
||||
if (decrypt) {
|
||||
fs.rmSync(outputPath, { force: true });
|
||||
}
|
||||
const args = [
|
||||
"enc",
|
||||
"-aes-256-cbc",
|
||||
"-pbkdf2",
|
||||
"-md",
|
||||
"sha256",
|
||||
...(decrypt ? ["-d"] : ["-salt"]),
|
||||
"-in",
|
||||
inputPath,
|
||||
"-out",
|
||||
outputPath,
|
||||
"-pass",
|
||||
"env:MATCH_PASSWORD",
|
||||
];
|
||||
const previousUmask = decrypt ? process.umask(0o077) : undefined;
|
||||
try {
|
||||
run("openssl", args);
|
||||
} finally {
|
||||
if (previousUmask !== undefined) {
|
||||
process.umask(previousUmask);
|
||||
}
|
||||
}
|
||||
if (decrypt) {
|
||||
fs.chmodSync(outputPath, 0o600);
|
||||
}
|
||||
}
|
||||
|
||||
function readProperties(filePath) {
|
||||
const properties = new Map();
|
||||
for (const rawLine of fs.readFileSync(filePath, "utf8").split(/\r?\n/u)) {
|
||||
const line = rawLine.trim();
|
||||
if (!line || line.startsWith("#")) {
|
||||
continue;
|
||||
}
|
||||
const separator = line.indexOf("=");
|
||||
if (separator <= 0) {
|
||||
throw new Error(`Invalid signing properties line in ${relativePath(filePath)}.`);
|
||||
}
|
||||
const key = line.slice(0, separator).trim();
|
||||
const value = line.slice(separator + 1).trim();
|
||||
if (!key || !value) {
|
||||
throw new Error(`Invalid empty signing property in ${relativePath(filePath)}.`);
|
||||
}
|
||||
properties.set(key, value);
|
||||
}
|
||||
return properties;
|
||||
}
|
||||
|
||||
function requireProperties(properties, names, filePath) {
|
||||
const missing = names.filter((name) => !properties.get(name));
|
||||
if (missing.length > 0) {
|
||||
throw new Error(
|
||||
`${relativePath(filePath)} is missing Android signing properties: ${missing.join(", ")}.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function writeMaterializedProperties(materializedDir, sourceProperties) {
|
||||
const keystorePath = materializedKeystorePath(materializedDir);
|
||||
const propertiesPath = materializedPropertiesPath(materializedDir);
|
||||
const tempPath = `${propertiesPath}.${process.pid}.tmp`;
|
||||
const properties = new Map(sourceProperties);
|
||||
properties.set("OPENCLAW_ANDROID_STORE_FILE", keystorePath);
|
||||
requireProperties(properties, requiredPropertyNames, propertiesPath);
|
||||
|
||||
const content = [
|
||||
"# Generated by scripts/android-release-signing.mjs.",
|
||||
"# Contains decrypted Android release signing values. Do not commit.",
|
||||
...requiredPropertyNames.map((name) => `${name}=${properties.get(name)}`),
|
||||
"",
|
||||
].join("\n");
|
||||
try {
|
||||
fs.writeFileSync(tempPath, content, { mode: 0o600 });
|
||||
fs.chmodSync(tempPath, 0o600);
|
||||
fs.renameSync(tempPath, propertiesPath);
|
||||
fs.chmodSync(propertiesPath, 0o600);
|
||||
} finally {
|
||||
fs.rmSync(tempPath, { force: true });
|
||||
}
|
||||
}
|
||||
|
||||
function validateMaterializedSigning(materializedDir) {
|
||||
const keystorePath = materializedKeystorePath(materializedDir);
|
||||
const propertiesPath = materializedPropertiesPath(materializedDir);
|
||||
|
||||
if (!fs.existsSync(keystorePath) || fs.statSync(keystorePath).size === 0) {
|
||||
throw new Error(
|
||||
`Missing materialized Android upload keystore at ${relativePath(keystorePath)}.`,
|
||||
);
|
||||
}
|
||||
if (!fs.existsSync(propertiesPath)) {
|
||||
throw new Error(
|
||||
`Missing materialized Android signing properties at ${relativePath(propertiesPath)}.`,
|
||||
);
|
||||
}
|
||||
|
||||
const properties = readProperties(propertiesPath);
|
||||
requireProperties(properties, requiredPropertyNames, propertiesPath);
|
||||
if (properties.get("OPENCLAW_ANDROID_STORE_FILE") !== keystorePath) {
|
||||
throw new Error(
|
||||
`${relativePath(propertiesPath)} must point OPENCLAW_ANDROID_STORE_FILE at ${relativePath(keystorePath)}.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function writePlan(manifest, options) {
|
||||
const materializedDir = resolveMaterializedDir(manifest, options);
|
||||
process.stdout.write(`Android release signing plan
|
||||
Signing repo: ${manifest.signingRepo}
|
||||
Signing branch: ${manifest.signingBranch}
|
||||
Signing assets: ${manifest.assetPath}
|
||||
Encrypted upload keystore: ${manifest.uploadKeystoreEncryptedFile}
|
||||
Encrypted Gradle properties: ${manifest.gradlePropertiesEncryptedFile}
|
||||
Materialized output: ${relativePath(materializedDir)}
|
||||
Gradle bridge: Fastlane exports ORG_GRADLE_PROJECT_* values from the materialized properties file.
|
||||
`);
|
||||
}
|
||||
|
||||
function writeSigningRepoManifest(workspace, manifest) {
|
||||
const signingManifestPath = path.join(assetDir(workspace, manifest), "manifest.json");
|
||||
const signingManifest = {
|
||||
version: 1,
|
||||
assetPath: manifest.assetPath,
|
||||
uploadKeystoreEncryptedFile: manifest.uploadKeystoreEncryptedFile,
|
||||
gradlePropertiesEncryptedFile: manifest.gradlePropertiesEncryptedFile,
|
||||
gradlePropertyNames: requiredPropertyNames,
|
||||
};
|
||||
fs.writeFileSync(signingManifestPath, `${JSON.stringify(signingManifest, null, 2)}\n`);
|
||||
}
|
||||
|
||||
function syncPull(manifest, options) {
|
||||
const workspace = resolveWorkspace(manifest, options);
|
||||
const materializedDir = resolveMaterializedDir(manifest, options);
|
||||
const tempPropertiesPath = path.join(materializedDir, ".gradle.properties.decrypted.tmp");
|
||||
|
||||
cloneSigningRepo(manifest, workspace, materializedDir);
|
||||
if (!fs.existsSync(encryptedKeystorePath(workspace, manifest))) {
|
||||
throw new Error(
|
||||
`Missing encrypted Android upload keystore in signing repo at ${manifest.assetPath}/${manifest.uploadKeystoreEncryptedFile}.`,
|
||||
);
|
||||
}
|
||||
if (!fs.existsSync(encryptedPropertiesPath(workspace, manifest))) {
|
||||
throw new Error(
|
||||
`Missing encrypted Android signing properties in signing repo at ${manifest.assetPath}/${manifest.gradlePropertiesEncryptedFile}.`,
|
||||
);
|
||||
}
|
||||
|
||||
fs.mkdirSync(materializedDir, { recursive: true });
|
||||
opensslCrypt({
|
||||
decrypt: true,
|
||||
inputPath: encryptedKeystorePath(workspace, manifest),
|
||||
outputPath: materializedKeystorePath(materializedDir),
|
||||
});
|
||||
try {
|
||||
opensslCrypt({
|
||||
decrypt: true,
|
||||
inputPath: encryptedPropertiesPath(workspace, manifest),
|
||||
outputPath: tempPropertiesPath,
|
||||
});
|
||||
const properties = readProperties(tempPropertiesPath);
|
||||
requireProperties(properties, sourceRequiredPropertyNames, tempPropertiesPath);
|
||||
writeMaterializedProperties(materializedDir, properties);
|
||||
} finally {
|
||||
fs.rmSync(tempPropertiesPath, { force: true });
|
||||
}
|
||||
|
||||
validateMaterializedSigning(materializedDir);
|
||||
process.stdout.write(
|
||||
`Materialized Android release signing assets in ${relativePath(materializedDir)}.\n`,
|
||||
);
|
||||
}
|
||||
|
||||
function requirePushSources(options) {
|
||||
if (!options.keystorePath) {
|
||||
throw new Error(
|
||||
"Missing Android upload keystore source. Pass --keystore or set OPENCLAW_ANDROID_UPLOAD_KEYSTORE.",
|
||||
);
|
||||
}
|
||||
if (!options.propertiesPath) {
|
||||
throw new Error(
|
||||
"Missing Android signing properties source. Pass --properties or set OPENCLAW_ANDROID_SIGNING_PROPERTIES.",
|
||||
);
|
||||
}
|
||||
if (!fs.existsSync(options.keystorePath) || fs.statSync(options.keystorePath).size === 0) {
|
||||
throw new Error(
|
||||
`Android upload keystore source is missing or empty: ${relativePath(options.keystorePath)}.`,
|
||||
);
|
||||
}
|
||||
if (!fs.existsSync(options.propertiesPath)) {
|
||||
throw new Error(
|
||||
`Android signing properties source is missing: ${relativePath(options.propertiesPath)}.`,
|
||||
);
|
||||
}
|
||||
const properties = readProperties(options.propertiesPath);
|
||||
requireProperties(properties, sourceRequiredPropertyNames, options.propertiesPath);
|
||||
}
|
||||
|
||||
function syncPush(manifest, options) {
|
||||
requireMatchPassword();
|
||||
requirePushSources(options);
|
||||
|
||||
const workspace = resolveWorkspace(manifest, options);
|
||||
cloneSigningRepo(manifest, workspace, resolveMaterializedDir(manifest, options));
|
||||
fs.mkdirSync(assetDir(workspace, manifest), { recursive: true });
|
||||
opensslCrypt({
|
||||
decrypt: false,
|
||||
inputPath: options.keystorePath,
|
||||
outputPath: encryptedKeystorePath(workspace, manifest),
|
||||
});
|
||||
opensslCrypt({
|
||||
decrypt: false,
|
||||
inputPath: options.propertiesPath,
|
||||
outputPath: encryptedPropertiesPath(workspace, manifest),
|
||||
});
|
||||
writeSigningRepoManifest(workspace, manifest);
|
||||
|
||||
run("git", ["add", manifest.assetPath], { cwd: workspace });
|
||||
const status = runText("git", ["status", "--porcelain"], { cwd: workspace }).trim();
|
||||
if (!status) {
|
||||
process.stdout.write("Android release signing assets were already up to date.\n");
|
||||
return;
|
||||
}
|
||||
|
||||
run("git", ["commit", "-m", "Update Android release signing assets"], { cwd: workspace });
|
||||
run("git", ["push", "origin", manifest.signingBranch], { cwd: workspace });
|
||||
process.stdout.write("Pushed encrypted Android release signing assets.\n");
|
||||
}
|
||||
|
||||
try {
|
||||
const options = parseArgs(process.argv.slice(2));
|
||||
const manifest = readManifest(options.manifestPath);
|
||||
|
||||
if (options.mode === "plan") {
|
||||
writePlan(manifest, options);
|
||||
} else if (options.mode === "check") {
|
||||
validateMaterializedSigning(resolveMaterializedDir(manifest, options));
|
||||
process.stdout.write("Android release signing materialization is valid.\n");
|
||||
} else if (options.mode === "sync-pull") {
|
||||
syncPull(manifest, options);
|
||||
} else if (options.mode === "sync-push") {
|
||||
syncPush(manifest, options);
|
||||
} else {
|
||||
throw new Error(`Unknown mode: ${options.mode}`);
|
||||
}
|
||||
} catch (error) {
|
||||
process.stderr.write(`${error.message}\n`);
|
||||
process.exit(1);
|
||||
}
|
||||
38
scripts/android-release-upload.sh
Normal file
38
scripts/android-release-upload.sh
Normal file
@@ -0,0 +1,38 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
Usage:
|
||||
scripts/android-release-upload.sh
|
||||
|
||||
Uploads Android Play metadata, builds signed release artifacts, and uploads the
|
||||
Play AAB to Google Play internal testing by default. This does not promote the
|
||||
build to production.
|
||||
EOF
|
||||
}
|
||||
|
||||
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
source "${ROOT_DIR}/scripts/lib/android-fastlane.sh"
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--)
|
||||
shift
|
||||
;;
|
||||
-h|--help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo "Unknown argument: $1" >&2
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
(
|
||||
cd "${ROOT_DIR}/apps/android"
|
||||
run_android_fastlane android release_upload
|
||||
)
|
||||
5
scripts/android-release.sh
Normal file
5
scripts/android-release.sh
Normal file
@@ -0,0 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
exec bash "${ROOT_DIR}/scripts/android-release-upload.sh" "$@"
|
||||
358
scripts/android-screenshots.sh
Normal file
358
scripts/android-screenshots.sh
Normal file
@@ -0,0 +1,358 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
Usage:
|
||||
scripts/android-screenshots.sh [--device <adb-serial>] [--avd <name>] [--locale en-US] [--skip-build] [--skip-install] [--keep-emulator] [--dry-run]
|
||||
|
||||
Builds and installs the Play debug app, launches deterministic screenshot scenes,
|
||||
and writes raw Google Play screenshots under:
|
||||
apps/android/fastlane/metadata/android/<locale>/images/phoneScreenshots/
|
||||
|
||||
If no ADB device is connected, pass --avd or set ANDROID_SCREENSHOT_AVD to boot
|
||||
an emulator non-interactively for the screenshot run.
|
||||
EOF
|
||||
}
|
||||
|
||||
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
ANDROID_DIR="${ROOT_DIR}/apps/android"
|
||||
LOCALE="en-US"
|
||||
DEVICE="${ANDROID_SCREENSHOT_DEVICE:-}"
|
||||
AVD="${ANDROID_SCREENSHOT_AVD:-}"
|
||||
KEEP_EMULATOR="${ANDROID_SCREENSHOT_KEEP_EMULATOR:-0}"
|
||||
SKIP_BUILD=0
|
||||
SKIP_INSTALL=0
|
||||
DRY_RUN=0
|
||||
SCENES=(connect chat voice screen settings)
|
||||
EMULATOR_PID=""
|
||||
EMULATOR_LOG=""
|
||||
STARTED_EMULATOR=0
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--)
|
||||
shift
|
||||
;;
|
||||
--device)
|
||||
DEVICE="${2:-}"
|
||||
shift 2
|
||||
;;
|
||||
--avd)
|
||||
AVD="${2:-}"
|
||||
shift 2
|
||||
;;
|
||||
--locale)
|
||||
LOCALE="${2:-}"
|
||||
shift 2
|
||||
;;
|
||||
--skip-build)
|
||||
SKIP_BUILD=1
|
||||
shift
|
||||
;;
|
||||
--skip-install)
|
||||
SKIP_INSTALL=1
|
||||
shift
|
||||
;;
|
||||
--keep-emulator)
|
||||
KEEP_EMULATOR=1
|
||||
shift
|
||||
;;
|
||||
--dry-run)
|
||||
DRY_RUN=1
|
||||
shift
|
||||
;;
|
||||
-h|--help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo "Unknown argument: $1" >&2
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
validate_locale() {
|
||||
local locale="$1"
|
||||
if [[ "$locale" =~ ^[A-Za-z0-9][A-Za-z0-9_-]*$ ]]; then
|
||||
return
|
||||
fi
|
||||
echo "Invalid Android screenshot locale: ${locale}" >&2
|
||||
echo "Use a locale tag like en-US or pt-BR; path separators and dot segments are not allowed." >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
validate_locale "$LOCALE"
|
||||
|
||||
cleanup_started_emulator() {
|
||||
local stopped=0
|
||||
|
||||
if [[ "$STARTED_EMULATOR" != "1" || "$KEEP_EMULATOR" == "1" ]]; then
|
||||
return
|
||||
fi
|
||||
if [[ -n "${ADB_BIN:-}" && -n "${ADB_SERIAL:-}" ]]; then
|
||||
if "$ADB_BIN" -s "$ADB_SERIAL" emu kill >/dev/null 2>&1; then
|
||||
stopped=1
|
||||
fi
|
||||
fi
|
||||
if [[ "$stopped" != "1" && -n "$EMULATOR_PID" ]]; then
|
||||
kill "$EMULATOR_PID" >/dev/null 2>&1 || true
|
||||
fi
|
||||
}
|
||||
|
||||
cleanup_emulator_log() {
|
||||
if [[ -n "$EMULATOR_LOG" && -f "$EMULATOR_LOG" ]]; then
|
||||
rm -f "$EMULATOR_LOG"
|
||||
fi
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
cleanup_started_emulator
|
||||
cleanup_emulator_log
|
||||
}
|
||||
|
||||
trap cleanup EXIT
|
||||
|
||||
adb_bin() {
|
||||
if [[ -n "${ADB:-}" ]]; then
|
||||
printf '%s\n' "$ADB"
|
||||
return
|
||||
fi
|
||||
if command -v adb >/dev/null 2>&1; then
|
||||
command -v adb
|
||||
return
|
||||
fi
|
||||
for sdk_root in "${ANDROID_HOME:-}" "${ANDROID_SDK_ROOT:-}" "$HOME/Library/Android/sdk"; do
|
||||
if [[ -n "$sdk_root" && -x "$sdk_root/platform-tools/adb" ]]; then
|
||||
printf '%s\n' "$sdk_root/platform-tools/adb"
|
||||
return
|
||||
fi
|
||||
done
|
||||
echo "adb not found. Install Android platform-tools or set ADB." >&2
|
||||
return 127
|
||||
}
|
||||
|
||||
emulator_bin() {
|
||||
if [[ -n "${ANDROID_EMULATOR:-}" ]]; then
|
||||
printf '%s\n' "$ANDROID_EMULATOR"
|
||||
return
|
||||
fi
|
||||
if command -v emulator >/dev/null 2>&1; then
|
||||
command -v emulator
|
||||
return
|
||||
fi
|
||||
for sdk_root in "${ANDROID_HOME:-}" "${ANDROID_SDK_ROOT:-}" "$HOME/Library/Android/sdk"; do
|
||||
if [[ -n "$sdk_root" && -x "$sdk_root/emulator/emulator" ]]; then
|
||||
printf '%s\n' "$sdk_root/emulator/emulator"
|
||||
return
|
||||
fi
|
||||
done
|
||||
echo "Android emulator binary not found. Install the Android emulator or set ANDROID_EMULATOR." >&2
|
||||
return 127
|
||||
}
|
||||
|
||||
connected_devices() {
|
||||
local adb="$1"
|
||||
"$adb" devices | awk 'NR > 1 && $2 == "device" { print $1 }'
|
||||
}
|
||||
|
||||
device_count() {
|
||||
local devices="$1"
|
||||
printf '%s\n' "$devices" | sed '/^$/d' | wc -l | tr -d ' '
|
||||
}
|
||||
|
||||
wait_for_single_device() {
|
||||
local adb="$1"
|
||||
local timeout_seconds="${ANDROID_SCREENSHOT_EMULATOR_TIMEOUT_SECONDS:-180}"
|
||||
local deadline=$((SECONDS + timeout_seconds))
|
||||
local devices
|
||||
local count
|
||||
|
||||
while (( SECONDS < deadline )); do
|
||||
devices="$(connected_devices "$adb")"
|
||||
count="$(device_count "$devices")"
|
||||
if [[ "$count" == "1" ]]; then
|
||||
printf '%s\n' "$devices"
|
||||
return
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
|
||||
echo "Timed out waiting for exactly one Android emulator device." >&2
|
||||
"$adb" devices -l >&2 || true
|
||||
return 1
|
||||
}
|
||||
|
||||
wait_for_boot_completed() {
|
||||
local adb="$1"
|
||||
local serial="$2"
|
||||
local timeout_seconds="${ANDROID_SCREENSHOT_EMULATOR_TIMEOUT_SECONDS:-180}"
|
||||
local deadline=$((SECONDS + timeout_seconds))
|
||||
local boot_completed
|
||||
|
||||
"$adb" -s "$serial" wait-for-device
|
||||
while (( SECONDS < deadline )); do
|
||||
boot_completed="$("$adb" -s "$serial" shell getprop sys.boot_completed 2>/dev/null | tr -d '\r' || true)"
|
||||
if [[ "$boot_completed" == "1" ]]; then
|
||||
"$adb" -s "$serial" shell input keyevent 82 >/dev/null 2>&1 || true
|
||||
return
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
|
||||
echo "Timed out waiting for Android emulator boot completion on ${serial}." >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
wait_for_explicit_device() {
|
||||
local adb="$1"
|
||||
local serial="$2"
|
||||
local timeout_seconds="${ANDROID_SCREENSHOT_DEVICE_TIMEOUT_SECONDS:-30}"
|
||||
local deadline=$((SECONDS + timeout_seconds))
|
||||
local state
|
||||
|
||||
while (( SECONDS < deadline )); do
|
||||
state="$("$adb" devices | awk -v serial="$serial" '$1 == serial { print $2 }')"
|
||||
if [[ "$state" == "device" ]]; then
|
||||
return
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
|
||||
if [[ -n "$state" ]]; then
|
||||
echo "Android device '${serial}' did not become usable within ${timeout_seconds}s; current adb state is '${state}'." >&2
|
||||
else
|
||||
echo "Android device '${serial}' was not found within ${timeout_seconds}s." >&2
|
||||
fi
|
||||
"$adb" devices -l >&2 || true
|
||||
return 1
|
||||
}
|
||||
|
||||
stabilize_device_for_screenshots() {
|
||||
local adb="$1"
|
||||
local serial="$2"
|
||||
"$adb" -s "$serial" shell settings put global window_animation_scale 0 >/dev/null 2>&1 || true
|
||||
"$adb" -s "$serial" shell settings put global transition_animation_scale 0 >/dev/null 2>&1 || true
|
||||
"$adb" -s "$serial" shell settings put global animator_duration_scale 0 >/dev/null 2>&1 || true
|
||||
}
|
||||
|
||||
boot_emulator() {
|
||||
local adb="$1"
|
||||
local avd="$2"
|
||||
local emulator
|
||||
local emulator_args
|
||||
local extra_args
|
||||
local serial
|
||||
|
||||
emulator="$(emulator_bin)"
|
||||
EMULATOR_LOG="$(mktemp "${TMPDIR:-/tmp}/openclaw-android-screenshot-emulator.XXXXXX.log")"
|
||||
echo "No connected Android device found. Booting AVD '${avd}'." >&2
|
||||
emulator_args=(-avd "$avd" -no-window -no-audio -no-boot-anim)
|
||||
if [[ -n "${ANDROID_SCREENSHOT_EMULATOR_ARGS:-}" ]]; then
|
||||
read -r -a extra_args <<<"$ANDROID_SCREENSHOT_EMULATOR_ARGS"
|
||||
emulator_args+=("${extra_args[@]}")
|
||||
fi
|
||||
"$emulator" "${emulator_args[@]}" >"$EMULATOR_LOG" 2>&1 &
|
||||
EMULATOR_PID="$!"
|
||||
STARTED_EMULATOR=1
|
||||
|
||||
serial="$(wait_for_single_device "$adb")"
|
||||
wait_for_boot_completed "$adb" "$serial"
|
||||
stabilize_device_for_screenshots "$adb" "$serial"
|
||||
ADB_SERIAL="$serial"
|
||||
}
|
||||
|
||||
resolve_device() {
|
||||
local adb="$1"
|
||||
local devices
|
||||
local count
|
||||
|
||||
if [[ -n "$DEVICE" ]]; then
|
||||
wait_for_explicit_device "$adb" "$DEVICE"
|
||||
stabilize_device_for_screenshots "$adb" "$DEVICE"
|
||||
ADB_SERIAL="$DEVICE"
|
||||
return
|
||||
fi
|
||||
devices="$(connected_devices "$adb")"
|
||||
count="$(device_count "$devices")"
|
||||
if [[ "$count" == "1" ]]; then
|
||||
stabilize_device_for_screenshots "$adb" "$devices"
|
||||
ADB_SERIAL="$devices"
|
||||
return
|
||||
fi
|
||||
if [[ "$count" == "0" ]]; then
|
||||
if [[ -n "$AVD" ]]; then
|
||||
boot_emulator "$adb" "$AVD"
|
||||
return
|
||||
fi
|
||||
echo "No Android device or emulator is connected." >&2
|
||||
echo "Start one manually, pass --device <adb-serial>, or pass --avd <name> to boot an emulator." >&2
|
||||
else
|
||||
echo "Multiple Android devices are connected. Pass --device <adb-serial>." >&2
|
||||
fi
|
||||
return 1
|
||||
}
|
||||
|
||||
latest_play_debug_apk() {
|
||||
if [[ ! -d "${ANDROID_DIR}/app/build/outputs/apk/play/debug" ]]; then
|
||||
return 0
|
||||
fi
|
||||
find "${ANDROID_DIR}/app/build/outputs/apk/play/debug" -maxdepth 1 -name '*-play-debug.apk' -print 2>/dev/null | sort | tail -n 1
|
||||
}
|
||||
|
||||
OUTPUT_DIR="${ANDROID_DIR}/fastlane/metadata/android/${LOCALE}/images/phoneScreenshots"
|
||||
ADB_SERIAL=""
|
||||
ADB_DISPLAY="${DEVICE:-<auto>}"
|
||||
|
||||
echo "Android screenshot output: ${OUTPUT_DIR}"
|
||||
echo "Scenes: ${SCENES[*]}"
|
||||
echo "ADB device: ${ADB_DISPLAY}"
|
||||
if [[ -n "$AVD" ]]; then
|
||||
echo "Fallback AVD: ${AVD}"
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == "1" ]]; then
|
||||
echo "Dry run complete. No build, install, or capture commands were executed."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
ADB_BIN="$(adb_bin)"
|
||||
resolve_device "$ADB_BIN"
|
||||
mkdir -p "$OUTPUT_DIR"
|
||||
rm -f "$OUTPUT_DIR"/*.png
|
||||
|
||||
if [[ "$SKIP_INSTALL" != "1" ]]; then
|
||||
if [[ "$SKIP_BUILD" != "1" ]]; then
|
||||
(
|
||||
cd "$ANDROID_DIR"
|
||||
./gradlew :app:assemblePlayDebug
|
||||
)
|
||||
fi
|
||||
APK_PATH="$(latest_play_debug_apk)"
|
||||
if [[ -z "$APK_PATH" ]]; then
|
||||
echo "No existing Play debug APK found. Run without --skip-build first." >&2
|
||||
exit 1
|
||||
fi
|
||||
"$ADB_BIN" -s "$ADB_SERIAL" install -r "$APK_PATH" >/dev/null
|
||||
elif [[ "$SKIP_BUILD" != "1" ]]; then
|
||||
(
|
||||
cd "$ANDROID_DIR"
|
||||
./gradlew :app:assemblePlayDebug
|
||||
)
|
||||
fi
|
||||
|
||||
for scene in "${SCENES[@]}"; do
|
||||
output_path="${OUTPUT_DIR}/openclaw-${scene}.png"
|
||||
"$ADB_BIN" -s "$ADB_SERIAL" shell am force-stop ai.openclaw.app >/dev/null
|
||||
"$ADB_BIN" -s "$ADB_SERIAL" shell am start -W \
|
||||
-n ai.openclaw.app/.MainActivity \
|
||||
--ez openclaw.screenshotMode true \
|
||||
--es openclaw.screenshotScene "$scene" >/dev/null
|
||||
sleep "${ANDROID_SCREENSHOT_SETTLE_SECONDS:-1.5}"
|
||||
"$ADB_BIN" -s "$ADB_SERIAL" exec-out screencap -p >"$output_path"
|
||||
echo "Captured ${output_path}"
|
||||
done
|
||||
|
||||
echo "Android screenshots written to ${OUTPUT_DIR}"
|
||||
40
scripts/android-sync-versioning.ts
Normal file
40
scripts/android-sync-versioning.ts
Normal file
@@ -0,0 +1,40 @@
|
||||
// Android Sync Versioning script supports OpenClaw repository automation.
|
||||
import path from "node:path";
|
||||
import { syncAndroidVersioning } from "./lib/android-version.ts";
|
||||
import { parseVersionSyncArgs } from "./lib/version-script-args.ts";
|
||||
|
||||
export { parseVersionSyncArgs as parseArgs } from "./lib/version-script-args.ts";
|
||||
|
||||
function printUsage(): void {
|
||||
process.stdout.write(
|
||||
"Usage: node --import tsx scripts/android-sync-versioning.ts [--write|--check] [--root dir]\n",
|
||||
);
|
||||
}
|
||||
|
||||
function main(argv = process.argv.slice(2)): number {
|
||||
const options = parseVersionSyncArgs(argv);
|
||||
if (options.help) {
|
||||
printUsage();
|
||||
return 0;
|
||||
}
|
||||
|
||||
const result = syncAndroidVersioning({ mode: options.mode, rootDir: options.rootDir });
|
||||
|
||||
if (options.mode === "check") {
|
||||
process.stdout.write("Android versioning artifacts are up to date.\n");
|
||||
} else if (result.updatedPaths.length === 0) {
|
||||
process.stdout.write("Android versioning artifacts already up to date.\n");
|
||||
} else {
|
||||
process.stdout.write(
|
||||
`Updated Android versioning artifacts:\n- ${result.updatedPaths.map((filePath) => path.relative(process.cwd(), filePath)).join("\n- ")}\n`,
|
||||
);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
try {
|
||||
process.exitCode = main();
|
||||
} catch (error) {
|
||||
process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
|
||||
process.exitCode = 1;
|
||||
}
|
||||
47
scripts/android-version.ts
Normal file
47
scripts/android-version.ts
Normal file
@@ -0,0 +1,47 @@
|
||||
// Android Version script supports OpenClaw repository automation.
|
||||
import { resolveAndroidVersion } from "./lib/android-version.ts";
|
||||
import { parseVersionQueryArgs } from "./lib/version-script-args.ts";
|
||||
|
||||
function printUsage(): void {
|
||||
process.stdout.write(
|
||||
"Usage: node --import tsx scripts/android-version.ts [--json|--shell] [--field name] [--root dir]\n\n",
|
||||
);
|
||||
}
|
||||
|
||||
function main(argv = process.argv.slice(2)): number {
|
||||
const options = parseVersionQueryArgs(argv);
|
||||
if (options.help) {
|
||||
printUsage();
|
||||
return 0;
|
||||
}
|
||||
|
||||
const version = resolveAndroidVersion(options.rootDir);
|
||||
|
||||
if (options.field) {
|
||||
const value = version[options.field as keyof typeof version];
|
||||
if (value === undefined) {
|
||||
throw new Error(`Unknown Android version field '${options.field}'.`);
|
||||
}
|
||||
process.stdout.write(`${value}\n`);
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (options.format === "shell") {
|
||||
process.stdout.write(
|
||||
[
|
||||
`OPENCLAW_ANDROID_VERSION_NAME=${version.canonicalVersion}`,
|
||||
`OPENCLAW_ANDROID_VERSION_CODE=${version.versionCode}`,
|
||||
].join("\n") + "\n",
|
||||
);
|
||||
} else {
|
||||
process.stdout.write(`${JSON.stringify(version, null, 2)}\n`);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
try {
|
||||
process.exitCode = main();
|
||||
} catch (error) {
|
||||
process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
|
||||
process.exitCode = 1;
|
||||
}
|
||||
976
scripts/anthropic-prompt-probe.ts
Normal file
976
scripts/anthropic-prompt-probe.ts
Normal file
@@ -0,0 +1,976 @@
|
||||
// Anthropic Prompt Probe script supports OpenClaw repository automation.
|
||||
import { spawn } from "node:child_process";
|
||||
// Live prompt probe for Anthropic setup-token and Claude CLI prompt-path debugging.
|
||||
// Usage:
|
||||
// OPENCLAW_PROMPT_TRANSPORT=direct|gateway
|
||||
// OPENCLAW_PROMPT_MODE=extra
|
||||
// OPENCLAW_PROMPT_TEXT='...'
|
||||
// OPENCLAW_PROMPT_CAPTURE=1
|
||||
// pnpm probe:anthropic:prompt
|
||||
import { randomUUID } from "node:crypto";
|
||||
import fs from "node:fs/promises";
|
||||
import http from "node:http";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import process from "node:process";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { resolveDefaultAgentDir } from "../src/agents/agent-scope.js";
|
||||
import { ensureAuthProfileStore, type AuthProfileCredential } from "../src/agents/auth-profiles.js";
|
||||
import { normalizeProviderId } from "../src/agents/model-selection.js";
|
||||
import { validateAnthropicSetupToken } from "../src/commands/auth-token.js";
|
||||
import { callGateway } from "../src/gateway/call.js";
|
||||
import { extractPayloadText } from "../src/gateway/test-helpers.agent-results.js";
|
||||
import { getFreePortBlockWithPermissionFallback } from "../src/test-utils/ports.js";
|
||||
import {
|
||||
parseBooleanEnv,
|
||||
parseStrictIntegerOption,
|
||||
redactForDevToolLog,
|
||||
} from "./lib/dev-tooling-safety.ts";
|
||||
|
||||
const TRANSPORT = process.env.OPENCLAW_PROMPT_TRANSPORT?.trim() === "direct" ? "direct" : "gateway";
|
||||
const GATEWAY_PROMPT_MODE = "extra";
|
||||
const PROMPT_TEXT = process.env.OPENCLAW_PROMPT_TEXT?.trim() ?? "";
|
||||
const PROMPT_LIST_JSON = process.env.OPENCLAW_PROMPT_LIST_JSON?.trim() ?? "";
|
||||
const USER_PROMPT = process.env.OPENCLAW_USER_PROMPT?.trim() || "is clawd here?";
|
||||
const ENABLE_CAPTURE = parseBooleanEnv({
|
||||
fallback: false,
|
||||
name: "OPENCLAW_PROMPT_CAPTURE",
|
||||
raw: process.env.OPENCLAW_PROMPT_CAPTURE,
|
||||
});
|
||||
const INCLUDE_RAW = parseBooleanEnv({
|
||||
fallback: false,
|
||||
name: "OPENCLAW_PROMPT_INCLUDE_RAW",
|
||||
raw: process.env.OPENCLAW_PROMPT_INCLUDE_RAW,
|
||||
});
|
||||
const KEEP_TMP = parseBooleanEnv({
|
||||
fallback: false,
|
||||
name: "OPENCLAW_PROMPT_KEEP_TMP",
|
||||
raw: process.env.OPENCLAW_PROMPT_KEEP_TMP,
|
||||
});
|
||||
const CLAUDE_BIN = process.env.CLAUDE_BIN?.trim() || "claude";
|
||||
const NODE_BIN = process.env.OPENCLAW_NODE_BIN?.trim() || process.execPath;
|
||||
const TIMEOUT_MS = parseStrictIntegerOption({
|
||||
fallback: 45_000,
|
||||
label: "OPENCLAW_PROMPT_TIMEOUT_MS",
|
||||
min: 1,
|
||||
raw: process.env.OPENCLAW_PROMPT_TIMEOUT_MS,
|
||||
});
|
||||
const GATEWAY_TIMEOUT_MS = parseStrictIntegerOption({
|
||||
fallback: 120_000,
|
||||
label: "OPENCLAW_PROMPT_GATEWAY_TIMEOUT_MS",
|
||||
min: 1,
|
||||
raw: process.env.OPENCLAW_PROMPT_GATEWAY_TIMEOUT_MS,
|
||||
});
|
||||
const GATEWAY_PARENT_SIGNAL_EXIT_CODES = new Map<NodeJS.Signals, number>([
|
||||
["SIGHUP", 129],
|
||||
["SIGINT", 130],
|
||||
["SIGTERM", 143],
|
||||
]);
|
||||
const CAPTURE_PROXY_MAX_BODY_BYTES = parseStrictIntegerOption({
|
||||
fallback: 2 * 1024 * 1024,
|
||||
label: "OPENCLAW_PROMPT_CAPTURE_MAX_BODY_BYTES",
|
||||
min: 1,
|
||||
raw: process.env.OPENCLAW_PROMPT_CAPTURE_MAX_BODY_BYTES,
|
||||
});
|
||||
const GATEWAY_LOG_TAIL_BYTES = 256 * 1024;
|
||||
const SETUP_TOKEN_RAW = process.env.OPENCLAW_LIVE_SETUP_TOKEN?.trim() ?? "";
|
||||
const SETUP_TOKEN_VALUE = process.env.OPENCLAW_LIVE_SETUP_TOKEN_VALUE?.trim() ?? "";
|
||||
const SETUP_TOKEN_PROFILE = process.env.OPENCLAW_LIVE_SETUP_TOKEN_PROFILE?.trim() ?? "";
|
||||
const DIRECT_CLAUDE_ARGS = ["-p", "--append-system-prompt"];
|
||||
|
||||
type CaptureSummary = {
|
||||
url?: string;
|
||||
authScheme?: string;
|
||||
xApp?: string;
|
||||
anthropicBeta?: string;
|
||||
systemBlockCount: number;
|
||||
systemBlocks: Array<{ index: number; bytes: number; preview: string }>;
|
||||
containsPromptExact: boolean;
|
||||
bodyContainsPromptExact: boolean;
|
||||
userBytes?: number;
|
||||
userPreview?: string;
|
||||
rawBody?: string;
|
||||
};
|
||||
|
||||
type PromptResult = {
|
||||
prompt: string;
|
||||
ok: boolean;
|
||||
transport: "direct" | "gateway";
|
||||
promptMode?: "extra";
|
||||
exitCode?: number | null;
|
||||
signal?: NodeJS.Signals | null;
|
||||
status?: string;
|
||||
text?: string;
|
||||
stdout?: string;
|
||||
stderr?: string;
|
||||
error?: string;
|
||||
matchedExtraUsage400: boolean;
|
||||
capture?: CaptureSummary;
|
||||
tmpDir?: string;
|
||||
};
|
||||
|
||||
type ProxyCapture = {
|
||||
url?: string;
|
||||
authHeader?: string;
|
||||
xApp?: string;
|
||||
anthropicBeta?: string;
|
||||
systemTexts: string[];
|
||||
userText?: string;
|
||||
rawBody?: string;
|
||||
};
|
||||
|
||||
type TokenSource = {
|
||||
profileId: string;
|
||||
token: string;
|
||||
};
|
||||
|
||||
type StoppableGatewayChild = {
|
||||
exitCode: number | null;
|
||||
pid?: number;
|
||||
signalCode: NodeJS.Signals | null;
|
||||
kill(signal: NodeJS.Signals): boolean;
|
||||
once(event: "exit", listener: () => void): unknown;
|
||||
};
|
||||
|
||||
type ClosableLogFile = {
|
||||
appendFile?(data: string | Uint8Array): Promise<void>;
|
||||
close(): Promise<void>;
|
||||
};
|
||||
|
||||
function toHeaderValue(value: string | string[] | undefined): string | undefined {
|
||||
return Array.isArray(value) ? value.join(", ") : value;
|
||||
}
|
||||
|
||||
function summarizeText(text: string, max = 120): string {
|
||||
const normalized = text.replace(/\s+/g, " ").trim();
|
||||
if (normalized.length <= max) {
|
||||
return normalized;
|
||||
}
|
||||
return `${normalized.slice(0, max - 1)}…`;
|
||||
}
|
||||
|
||||
function summarizeCapture(
|
||||
capture: ProxyCapture | undefined,
|
||||
prompt: string,
|
||||
): CaptureSummary | undefined {
|
||||
if (!capture) {
|
||||
return undefined;
|
||||
}
|
||||
return {
|
||||
url: capture.url,
|
||||
authScheme: capture.authHeader?.split(/\s+/, 1)[0],
|
||||
xApp: capture.xApp,
|
||||
anthropicBeta: capture.anthropicBeta,
|
||||
systemBlockCount: capture.systemTexts.length,
|
||||
systemBlocks: capture.systemTexts.map((entry, index) => ({
|
||||
index,
|
||||
bytes: Buffer.byteLength(entry, "utf8"),
|
||||
preview: summarizeText(entry),
|
||||
})),
|
||||
containsPromptExact: capture.systemTexts.includes(prompt),
|
||||
bodyContainsPromptExact: capture.rawBody?.includes(prompt) ?? false,
|
||||
userBytes: capture.userText ? Buffer.byteLength(capture.userText, "utf8") : undefined,
|
||||
userPreview: capture.userText ? summarizeText(capture.userText) : undefined,
|
||||
rawBody: INCLUDE_RAW ? capture.rawBody : undefined,
|
||||
};
|
||||
}
|
||||
|
||||
function resolveAnthropicUpstreamUrl(
|
||||
requestUrl: string | undefined,
|
||||
upstreamBaseUrl: string,
|
||||
): string {
|
||||
const raw = requestUrl || "/";
|
||||
if (!raw.startsWith("/") || raw.startsWith("//")) {
|
||||
throw new Error(`refusing non-origin proxy request URL: ${JSON.stringify(raw)}`);
|
||||
}
|
||||
const upstream = new URL(upstreamBaseUrl);
|
||||
if (upstream.protocol !== "https:" || upstream.hostname !== "api.anthropic.com") {
|
||||
throw new Error(`refusing unexpected Anthropic upstream origin: ${upstream.origin}`);
|
||||
}
|
||||
const requestPath = new URL(raw, "http://127.0.0.1");
|
||||
return new URL(`${requestPath.pathname}${requestPath.search}`, upstream).toString();
|
||||
}
|
||||
|
||||
function matchesExtraUsage400(...parts: Array<string | undefined>): boolean {
|
||||
return parts
|
||||
.filter((value): value is string => typeof value === "string" && value.length > 0)
|
||||
.join(" ")
|
||||
.toLowerCase()
|
||||
.includes("third-party apps now draw from your extra usage");
|
||||
}
|
||||
|
||||
function promptProbeTmpResult(tmpDir: string, keepTmp = KEEP_TMP): Pick<PromptResult, "tmpDir"> {
|
||||
return keepTmp ? { tmpDir } : {};
|
||||
}
|
||||
|
||||
async function cleanupPromptProbeTmpDir(tmpDir: string, keepTmp = KEEP_TMP): Promise<void> {
|
||||
if (keepTmp) {
|
||||
return;
|
||||
}
|
||||
await fs.rm(tmpDir, { force: true, recursive: true });
|
||||
}
|
||||
|
||||
function isSetupToken(value: string): boolean {
|
||||
return value.startsWith("sk-ant-oat01-");
|
||||
}
|
||||
|
||||
function listSetupTokenProfiles(store: {
|
||||
profiles: Record<string, AuthProfileCredential>;
|
||||
}): Array<{ id: string; token: string }> {
|
||||
return Object.entries(store.profiles)
|
||||
.filter(([, cred]) => {
|
||||
if (cred.type !== "token") {
|
||||
return false;
|
||||
}
|
||||
if (normalizeProviderId(cred.provider) !== "anthropic") {
|
||||
return false;
|
||||
}
|
||||
return isSetupToken(cred.token ?? "");
|
||||
})
|
||||
.map(([id, cred]) => ({ id, token: cred.token ?? "" }));
|
||||
}
|
||||
|
||||
function pickSetupTokenProfile(candidates: Array<{ id: string; token: string }>): {
|
||||
id: string;
|
||||
token: string;
|
||||
} | null {
|
||||
const preferred = ["anthropic:setup-token-test", "anthropic:setup-token", "anthropic:default"];
|
||||
for (const id of preferred) {
|
||||
const match = candidates.find((entry) => entry.id === id);
|
||||
if (match) {
|
||||
return match;
|
||||
}
|
||||
}
|
||||
return candidates[0] ?? null;
|
||||
}
|
||||
|
||||
function validateSetupToken(value: string): string {
|
||||
const error = validateAnthropicSetupToken(value);
|
||||
if (error) {
|
||||
throw new Error(`invalid setup-token: ${error}`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function resolveSetupTokenSource(): TokenSource {
|
||||
const explicitToken =
|
||||
(SETUP_TOKEN_RAW && isSetupToken(SETUP_TOKEN_RAW) ? SETUP_TOKEN_RAW : "") || SETUP_TOKEN_VALUE;
|
||||
if (explicitToken) {
|
||||
return {
|
||||
profileId: "anthropic:default",
|
||||
token: validateSetupToken(explicitToken),
|
||||
};
|
||||
}
|
||||
|
||||
const agentDir = resolveDefaultAgentDir({});
|
||||
const store = ensureAuthProfileStore(agentDir, {
|
||||
allowKeychainPrompt: false,
|
||||
});
|
||||
const candidates = listSetupTokenProfiles(store);
|
||||
if (SETUP_TOKEN_PROFILE) {
|
||||
const match = candidates.find((entry) => entry.id === SETUP_TOKEN_PROFILE);
|
||||
if (!match) {
|
||||
throw new Error(`setup-token profile not found: ${SETUP_TOKEN_PROFILE}`);
|
||||
}
|
||||
return { profileId: match.id, token: validateSetupToken(match.token) };
|
||||
}
|
||||
const match = pickSetupTokenProfile(candidates);
|
||||
if (!match) {
|
||||
throw new Error(
|
||||
"no Anthropics setup-token profile found; set OPENCLAW_LIVE_SETUP_TOKEN_VALUE or OPENCLAW_LIVE_SETUP_TOKEN_PROFILE",
|
||||
);
|
||||
}
|
||||
return { profileId: match.id, token: validateSetupToken(match.token) };
|
||||
}
|
||||
|
||||
async function sleep(ms: number): Promise<void> {
|
||||
return await new Promise((resolve) => {
|
||||
setTimeout(resolve, ms);
|
||||
});
|
||||
}
|
||||
|
||||
async function withTimeout<T>(
|
||||
promise: Promise<T>,
|
||||
timeoutMs: number,
|
||||
fallback: () => T,
|
||||
): Promise<T> {
|
||||
return await Promise.race([promise, sleep(timeoutMs).then(() => fallback())]);
|
||||
}
|
||||
|
||||
async function readRequestBody(
|
||||
req: http.IncomingMessage,
|
||||
maxBytes = CAPTURE_PROXY_MAX_BODY_BYTES,
|
||||
): Promise<Buffer> {
|
||||
const chunks: Buffer[] = [];
|
||||
let totalBytes = 0;
|
||||
for await (const chunk of req) {
|
||||
const buffer = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk);
|
||||
totalBytes += buffer.length;
|
||||
if (totalBytes > maxBytes) {
|
||||
req.destroy();
|
||||
throw new Error(`Anthropic capture proxy request body exceeded ${maxBytes} bytes`);
|
||||
}
|
||||
chunks.push(buffer);
|
||||
}
|
||||
return Buffer.concat(chunks, totalBytes);
|
||||
}
|
||||
|
||||
function extractProxyCapture(rawBody: string, req: http.IncomingMessage): ProxyCapture {
|
||||
let parsed: {
|
||||
system?: Array<{ text?: string }>;
|
||||
messages?: Array<{ role?: string; content?: unknown }>;
|
||||
} | null;
|
||||
try {
|
||||
parsed = JSON.parse(rawBody) as typeof parsed;
|
||||
} catch {
|
||||
parsed = null;
|
||||
}
|
||||
const systemTexts = Array.isArray(parsed?.system)
|
||||
? parsed.system
|
||||
.map((entry) => (typeof entry?.text === "string" ? entry.text : ""))
|
||||
.filter(Boolean)
|
||||
: [];
|
||||
const userText = Array.isArray(parsed?.messages)
|
||||
? parsed.messages
|
||||
.filter((entry) => entry?.role === "user")
|
||||
.flatMap((entry) => {
|
||||
const content = entry?.content;
|
||||
if (typeof content === "string") {
|
||||
return [content];
|
||||
}
|
||||
if (!Array.isArray(content)) {
|
||||
return [];
|
||||
}
|
||||
return content
|
||||
.map((item) =>
|
||||
item && typeof item === "object" && "text" in item && typeof item.text === "string"
|
||||
? item.text
|
||||
: "",
|
||||
)
|
||||
.filter(Boolean);
|
||||
})
|
||||
.join("\n")
|
||||
: undefined;
|
||||
return {
|
||||
url: req.url ?? undefined,
|
||||
authHeader: toHeaderValue(req.headers.authorization),
|
||||
xApp: toHeaderValue(req.headers["x-app"]),
|
||||
anthropicBeta: toHeaderValue(req.headers["anthropic-beta"]),
|
||||
systemTexts,
|
||||
userText,
|
||||
rawBody,
|
||||
};
|
||||
}
|
||||
|
||||
async function startAnthropicProxy(params: { port: number; upstreamBaseUrl: string }) {
|
||||
let lastCapture: ProxyCapture | undefined;
|
||||
const sockets = new Set<import("node:net").Socket>();
|
||||
const server = http.createServer((req, res) => {
|
||||
void (async () => {
|
||||
try {
|
||||
const method = req.method ?? "GET";
|
||||
const requestBody = await readRequestBody(req);
|
||||
const rawBody = requestBody.toString("utf8");
|
||||
lastCapture = extractProxyCapture(rawBody, req);
|
||||
|
||||
const upstreamUrl = resolveAnthropicUpstreamUrl(req.url, params.upstreamBaseUrl);
|
||||
const headers = new Headers();
|
||||
for (const [key, value] of Object.entries(req.headers)) {
|
||||
if (value === undefined) {
|
||||
continue;
|
||||
}
|
||||
const lower = key.toLowerCase();
|
||||
if (lower === "host" || lower === "content-length") {
|
||||
continue;
|
||||
}
|
||||
headers.set(key, Array.isArray(value) ? value.join(", ") : value);
|
||||
}
|
||||
const upstreamRes = await fetch(upstreamUrl, {
|
||||
method,
|
||||
headers,
|
||||
body:
|
||||
method === "GET" || method === "HEAD" || requestBody.byteLength === 0
|
||||
? undefined
|
||||
: requestBody,
|
||||
duplex: "half",
|
||||
});
|
||||
const responseHeaders: Record<string, string> = {};
|
||||
for (const [key, value] of upstreamRes.headers.entries()) {
|
||||
const lower = key.toLowerCase();
|
||||
if (
|
||||
lower === "content-length" ||
|
||||
lower === "content-encoding" ||
|
||||
lower === "transfer-encoding" ||
|
||||
lower === "connection" ||
|
||||
lower === "keep-alive"
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
responseHeaders[key] = value;
|
||||
}
|
||||
res.writeHead(upstreamRes.status, responseHeaders);
|
||||
if (upstreamRes.body) {
|
||||
for await (const chunk of upstreamRes.body) {
|
||||
res.write(Buffer.from(chunk));
|
||||
}
|
||||
}
|
||||
res.end();
|
||||
} catch (error) {
|
||||
res.writeHead(502, { "content-type": "text/plain; charset=utf-8" });
|
||||
res.end(redactForDevToolLog(`proxy error: ${String(error)}`));
|
||||
}
|
||||
})();
|
||||
});
|
||||
server.on("connection", (socket) => {
|
||||
sockets.add(socket);
|
||||
socket.on("close", () => sockets.delete(socket));
|
||||
});
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
server.once("error", reject);
|
||||
server.listen(params.port, "127.0.0.1", () => resolve());
|
||||
});
|
||||
return {
|
||||
getLastCapture() {
|
||||
return lastCapture;
|
||||
},
|
||||
async stop() {
|
||||
for (const socket of sockets) {
|
||||
socket.destroy();
|
||||
}
|
||||
await withTimeout(
|
||||
new Promise<void>((resolve, reject) => {
|
||||
server.close((error) => (error ? reject(error) : resolve()));
|
||||
}),
|
||||
1_000,
|
||||
() => undefined,
|
||||
);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function getFreePort(): Promise<number> {
|
||||
return await getFreePortBlockWithPermissionFallback({
|
||||
offsets: [0, 1, 2, 4],
|
||||
fallbackBase: 44_000,
|
||||
});
|
||||
}
|
||||
|
||||
async function runDirectPrompt(prompt: string): Promise<PromptResult> {
|
||||
const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-direct-prompt-probe-"));
|
||||
const proxyPort = ENABLE_CAPTURE ? await getFreePort() : undefined;
|
||||
const proxy =
|
||||
ENABLE_CAPTURE && proxyPort
|
||||
? await startAnthropicProxy({ port: proxyPort, upstreamBaseUrl: "https://api.anthropic.com" })
|
||||
: undefined;
|
||||
|
||||
try {
|
||||
const stdout: string[] = [];
|
||||
const stderr: string[] = [];
|
||||
const child = spawn(CLAUDE_BIN, [...DIRECT_CLAUDE_ARGS, prompt, USER_PROMPT], {
|
||||
cwd: process.cwd(),
|
||||
env: {
|
||||
...process.env,
|
||||
...(proxyPort ? { ANTHROPIC_BASE_URL: `http://127.0.0.1:${proxyPort}` } : {}),
|
||||
ANTHROPIC_API_KEY: "",
|
||||
ANTHROPIC_API_KEY_OLD: "",
|
||||
},
|
||||
detached: process.platform !== "win32",
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
child.stdout.on("data", (chunk) => stdout.push(String(chunk)));
|
||||
child.stderr.on("data", (chunk) => stderr.push(String(chunk)));
|
||||
const exitPromise = new Promise<{ code: number | null; signal: NodeJS.Signals | null }>(
|
||||
(resolve, reject) => {
|
||||
child.once("error", reject);
|
||||
child.once("exit", (code, signal) => resolve({ code, signal }));
|
||||
},
|
||||
);
|
||||
const stopDirectChild = async (signal: NodeJS.Signals = "SIGKILL") => {
|
||||
signalGatewayPromptChildTree(child, signal);
|
||||
await waitForGatewayPromptChildTreeExit(
|
||||
child,
|
||||
exitPromise.then(() => undefined),
|
||||
1_500,
|
||||
);
|
||||
};
|
||||
const removeParentSignalHandlers = installGatewayPromptParentSignalHandlers(
|
||||
child,
|
||||
stopDirectChild,
|
||||
);
|
||||
let timeoutTimer: ReturnType<typeof setTimeout> | undefined;
|
||||
const exit = await Promise.race([
|
||||
exitPromise,
|
||||
new Promise<{ code: null; signal: NodeJS.Signals }>((resolve) => {
|
||||
timeoutTimer = setTimeout(() => {
|
||||
void stopDirectChild("SIGKILL").finally(() => {
|
||||
resolve({ code: null, signal: "SIGKILL" });
|
||||
});
|
||||
}, TIMEOUT_MS);
|
||||
}),
|
||||
]).finally(() => {
|
||||
if (timeoutTimer) {
|
||||
clearTimeout(timeoutTimer);
|
||||
}
|
||||
removeParentSignalHandlers();
|
||||
});
|
||||
const joinedStdout = stdout.join("");
|
||||
const joinedStderr = stderr.join("");
|
||||
return {
|
||||
prompt,
|
||||
ok: exit.code === 0 && !matchesExtraUsage400(joinedStdout, joinedStderr),
|
||||
transport: "direct",
|
||||
exitCode: exit.code,
|
||||
signal: exit.signal,
|
||||
stdout: redactForDevToolLog(joinedStdout.trim()) || undefined,
|
||||
stderr: redactForDevToolLog(joinedStderr.trim()) || undefined,
|
||||
matchedExtraUsage400: matchesExtraUsage400(joinedStdout, joinedStderr),
|
||||
capture: summarizeCapture(proxy?.getLastCapture(), prompt),
|
||||
...promptProbeTmpResult(tmpDir),
|
||||
};
|
||||
} finally {
|
||||
await proxy?.stop().catch(() => {});
|
||||
await cleanupPromptProbeTmpDir(tmpDir).catch(() => {});
|
||||
}
|
||||
}
|
||||
|
||||
async function startGatewayProcess(params: {
|
||||
port: number;
|
||||
gatewayToken: string;
|
||||
configPath: string;
|
||||
stateDir: string;
|
||||
agentDir: string;
|
||||
bundledPluginsDir: string;
|
||||
logPath: string;
|
||||
}) {
|
||||
const logFile = await fs.open(params.logPath, "a");
|
||||
const child = spawn(
|
||||
NODE_BIN,
|
||||
["openclaw.mjs", "gateway", "--port", String(params.port), "--bind", "loopback", "--force"],
|
||||
{
|
||||
cwd: process.cwd(),
|
||||
env: {
|
||||
...process.env,
|
||||
OPENCLAW_CONFIG_PATH: params.configPath,
|
||||
OPENCLAW_STATE_DIR: params.stateDir,
|
||||
OPENCLAW_AGENT_DIR: params.agentDir,
|
||||
OPENCLAW_GATEWAY_TOKEN: params.gatewayToken,
|
||||
OPENCLAW_SKIP_CHANNELS: "1",
|
||||
OPENCLAW_SKIP_GMAIL_WATCHER: "1",
|
||||
OPENCLAW_SKIP_CANVAS_HOST: "1",
|
||||
OPENCLAW_SKIP_BROWSER_CONTROL_SERVER: "1",
|
||||
OPENCLAW_DISABLE_BONJOUR: "1",
|
||||
OPENCLAW_SKIP_CRON: "1",
|
||||
OPENCLAW_TEST_MINIMAL_GATEWAY: "1",
|
||||
OPENCLAW_BUNDLED_PLUGINS_DIR: params.bundledPluginsDir,
|
||||
ANTHROPIC_API_KEY: "",
|
||||
ANTHROPIC_API_KEY_OLD: "",
|
||||
},
|
||||
detached: process.platform !== "win32",
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
},
|
||||
);
|
||||
const pendingLogWrites = new Set<Promise<void>>();
|
||||
const logWriteErrors: unknown[] = [];
|
||||
const trackLogWrite = (chunk: Buffer) => {
|
||||
const write = logFile.appendFile(chunk).catch((error: unknown) => {
|
||||
logWriteErrors.push(error);
|
||||
throw error;
|
||||
});
|
||||
pendingLogWrites.add(write);
|
||||
void write
|
||||
.finally(() => {
|
||||
pendingLogWrites.delete(write);
|
||||
})
|
||||
.catch(() => undefined);
|
||||
};
|
||||
child.stdout.on("data", trackLogWrite);
|
||||
child.stderr.on("data", trackLogWrite);
|
||||
let stopPromise: Promise<boolean> | undefined;
|
||||
let removeParentSignalHandlers = () => {};
|
||||
const stopOnce = async (): Promise<boolean> => {
|
||||
stopPromise ??= stopGatewayPromptChild(
|
||||
child,
|
||||
logFile,
|
||||
1_500,
|
||||
1_500,
|
||||
pendingLogWrites,
|
||||
logWriteErrors,
|
||||
).finally(() => {
|
||||
removeParentSignalHandlers();
|
||||
});
|
||||
return await stopPromise;
|
||||
};
|
||||
removeParentSignalHandlers = installGatewayPromptParentSignalHandlers(child, stopOnce);
|
||||
return {
|
||||
async stop(): Promise<boolean> {
|
||||
return await stopOnce();
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function stopGatewayPromptChild(
|
||||
child: StoppableGatewayChild,
|
||||
logFile: ClosableLogFile,
|
||||
sigintTimeoutMs = 1_500,
|
||||
sigkillTimeoutMs = 1_500,
|
||||
pendingLogWrites: Iterable<Promise<void>> = [],
|
||||
logWriteErrors: readonly unknown[] = [],
|
||||
): Promise<boolean> {
|
||||
let exited = child.exitCode !== null || child.signalCode !== null;
|
||||
const exitPromise = exited
|
||||
? Promise.resolve()
|
||||
: new Promise<void>((resolve) => {
|
||||
child.once("exit", () => {
|
||||
exited = true;
|
||||
resolve();
|
||||
});
|
||||
});
|
||||
if (!exited) {
|
||||
signalGatewayPromptChildTree(child, "SIGINT");
|
||||
}
|
||||
const exitedAfterSigint = await waitForGatewayPromptChildTreeExit(
|
||||
child,
|
||||
exitPromise,
|
||||
sigintTimeoutMs,
|
||||
);
|
||||
if (!exitedAfterSigint) {
|
||||
signalGatewayPromptChildTree(child, "SIGKILL");
|
||||
await waitForGatewayPromptChildTreeExit(child, exitPromise, sigkillTimeoutMs);
|
||||
}
|
||||
const failedLogWrite = (await Promise.allSettled(pendingLogWrites)).find(
|
||||
(result): result is PromiseRejectedResult => result.status === "rejected",
|
||||
);
|
||||
await logFile.close();
|
||||
const logWriteError = failedLogWrite?.reason ?? logWriteErrors[0];
|
||||
if (logWriteError) {
|
||||
throw new Error(`Anthropic prompt gateway log write failed: ${String(logWriteError)}`);
|
||||
}
|
||||
return exited;
|
||||
}
|
||||
|
||||
function installGatewayPromptParentSignalHandlers(
|
||||
child: StoppableGatewayChild,
|
||||
stopGateway: () => Promise<unknown>,
|
||||
): () => void {
|
||||
let parentSignalShutdownStarted = false;
|
||||
const handlers = new Map<NodeJS.Signals, () => void>();
|
||||
const removeHandlers = () => {
|
||||
for (const [signal, handler] of handlers) {
|
||||
process.off(signal, handler);
|
||||
}
|
||||
handlers.clear();
|
||||
};
|
||||
for (const signal of GATEWAY_PARENT_SIGNAL_EXIT_CODES.keys()) {
|
||||
const handler = () => {
|
||||
if (parentSignalShutdownStarted) {
|
||||
signalGatewayPromptChildTree(child, "SIGKILL");
|
||||
return;
|
||||
}
|
||||
parentSignalShutdownStarted = true;
|
||||
void stopGateway()
|
||||
.catch(() => undefined)
|
||||
.finally(() => {
|
||||
removeHandlers();
|
||||
process.exit(GATEWAY_PARENT_SIGNAL_EXIT_CODES.get(signal) ?? 1);
|
||||
});
|
||||
};
|
||||
handlers.set(signal, handler);
|
||||
process.on(signal, handler);
|
||||
}
|
||||
return removeHandlers;
|
||||
}
|
||||
|
||||
async function waitForGatewayPromptChildTreeExit(
|
||||
child: StoppableGatewayChild,
|
||||
exitPromise: Promise<void>,
|
||||
timeoutMs: number,
|
||||
): Promise<boolean> {
|
||||
let leaderExited = child.exitCode !== null || child.signalCode !== null;
|
||||
const trackedExit = exitPromise.then(() => {
|
||||
leaderExited = true;
|
||||
});
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
if (leaderExited && !gatewayPromptChildTreeIsAlive(child)) {
|
||||
return true;
|
||||
}
|
||||
const waitMs = Math.min(50, Math.max(0, deadline - Date.now()));
|
||||
if (leaderExited) {
|
||||
await sleep(waitMs);
|
||||
} else {
|
||||
await Promise.race([trackedExit, sleep(waitMs)]);
|
||||
}
|
||||
}
|
||||
return leaderExited && !gatewayPromptChildTreeIsAlive(child);
|
||||
}
|
||||
|
||||
function signalGatewayPromptChildTree(
|
||||
child: StoppableGatewayChild,
|
||||
signal: NodeJS.Signals,
|
||||
): boolean {
|
||||
if (process.platform !== "win32" && typeof child.pid === "number") {
|
||||
try {
|
||||
process.kill(-child.pid, signal);
|
||||
return true;
|
||||
} catch {
|
||||
return child.kill(signal);
|
||||
}
|
||||
}
|
||||
return child.kill(signal);
|
||||
}
|
||||
|
||||
function gatewayPromptChildTreeIsAlive(child: StoppableGatewayChild): boolean {
|
||||
if (process.platform === "win32" || typeof child.pid !== "number") {
|
||||
return false;
|
||||
}
|
||||
try {
|
||||
process.kill(-child.pid, 0);
|
||||
return true;
|
||||
} catch (error) {
|
||||
return !isMissingProcessError(error);
|
||||
}
|
||||
}
|
||||
|
||||
function isMissingProcessError(error: unknown): boolean {
|
||||
return typeof error === "object" && error !== null && "code" in error && error.code === "ESRCH";
|
||||
}
|
||||
|
||||
async function waitForGatewayReady(url: string, token: string): Promise<void> {
|
||||
const deadline = Date.now() + 45_000;
|
||||
let lastError = "gateway start timeout";
|
||||
while (Date.now() < deadline) {
|
||||
try {
|
||||
await callGateway({ url, token, method: "health", timeoutMs: 5_000 });
|
||||
return;
|
||||
} catch (error) {
|
||||
lastError = String(error);
|
||||
await sleep(500);
|
||||
}
|
||||
}
|
||||
throw new Error(lastError);
|
||||
}
|
||||
|
||||
async function readLogTail(logPath: string, maxBytes = GATEWAY_LOG_TAIL_BYTES): Promise<string> {
|
||||
if (!Number.isSafeInteger(maxBytes) || maxBytes <= 0) {
|
||||
throw new Error("maxBytes must be a positive integer");
|
||||
}
|
||||
const logFile = await fs.open(logPath, "r").catch(() => undefined);
|
||||
if (!logFile) {
|
||||
return "";
|
||||
}
|
||||
try {
|
||||
const stat = await logFile.stat();
|
||||
if (stat.size <= 0) {
|
||||
return "";
|
||||
}
|
||||
const length = Math.min(stat.size, maxBytes);
|
||||
const position = Math.max(0, stat.size - length);
|
||||
const buffer = Buffer.allocUnsafe(length);
|
||||
const { bytesRead } = await logFile.read(buffer, 0, length, position);
|
||||
const raw = buffer.subarray(0, bytesRead).toString("utf8");
|
||||
const lineAlignedRaw = position > 0 ? raw.replace(/^[^\n]*(?:\r?\n|$)/u, "") : raw;
|
||||
return redactForDevToolLog(lineAlignedRaw.split(/\r?\n/).slice(-40).join("\n").trim());
|
||||
} finally {
|
||||
await logFile.close();
|
||||
}
|
||||
}
|
||||
|
||||
async function runGatewayPrompt(prompt: string): Promise<PromptResult> {
|
||||
const tokenSource = resolveSetupTokenSource();
|
||||
const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-gateway-prompt-probe-"));
|
||||
const stateDir = path.join(tmpDir, "state");
|
||||
const agentDir = path.join(stateDir, "agents", "main", "agent");
|
||||
const bundledPluginsDir = path.join(tmpDir, "bundled-plugins-empty");
|
||||
const configPath = path.join(tmpDir, "openclaw.json");
|
||||
const logPath = path.join(tmpDir, "gateway.log");
|
||||
const gatewayToken = `gw-${randomUUID()}`;
|
||||
const port = await getFreePort();
|
||||
const proxyPort = ENABLE_CAPTURE ? await getFreePort() : undefined;
|
||||
const proxy =
|
||||
ENABLE_CAPTURE && proxyPort
|
||||
? await startAnthropicProxy({ port: proxyPort, upstreamBaseUrl: "https://api.anthropic.com" })
|
||||
: undefined;
|
||||
let gateway: Awaited<ReturnType<typeof startGatewayProcess>> | undefined;
|
||||
|
||||
try {
|
||||
await fs.mkdir(agentDir, { recursive: true });
|
||||
await fs.mkdir(bundledPluginsDir, { recursive: true });
|
||||
await fs.writeFile(
|
||||
configPath,
|
||||
`${JSON.stringify(
|
||||
{
|
||||
gateway: {
|
||||
mode: "local",
|
||||
controlUi: { enabled: false },
|
||||
tailscale: { mode: "off" },
|
||||
},
|
||||
discovery: {
|
||||
mdns: { mode: "off" },
|
||||
wideArea: { enabled: false },
|
||||
},
|
||||
...(proxyPort
|
||||
? {
|
||||
models: {
|
||||
providers: {
|
||||
anthropic: {
|
||||
baseUrl: `http://127.0.0.1:${proxyPort}`,
|
||||
api: "anthropic-messages",
|
||||
models: [],
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
: {}),
|
||||
auth: {
|
||||
profiles: { [tokenSource.profileId]: { provider: "anthropic", mode: "token" } },
|
||||
order: { anthropic: [tokenSource.profileId] },
|
||||
},
|
||||
agents: {
|
||||
defaults: {
|
||||
model: "anthropic/claude-sonnet-4-6",
|
||||
heartbeat: {
|
||||
includeSystemPromptSection: false,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
null,
|
||||
2,
|
||||
)}\n`,
|
||||
);
|
||||
await fs.writeFile(
|
||||
path.join(agentDir, "auth-profiles.json"),
|
||||
`${JSON.stringify(
|
||||
{
|
||||
version: 1,
|
||||
profiles: {
|
||||
[tokenSource.profileId]: {
|
||||
type: "token",
|
||||
provider: "anthropic",
|
||||
token: tokenSource.token,
|
||||
},
|
||||
},
|
||||
},
|
||||
null,
|
||||
2,
|
||||
)}\n`,
|
||||
);
|
||||
|
||||
gateway = await startGatewayProcess({
|
||||
port,
|
||||
gatewayToken,
|
||||
configPath,
|
||||
stateDir,
|
||||
agentDir,
|
||||
bundledPluginsDir,
|
||||
logPath,
|
||||
});
|
||||
const url = `ws://127.0.0.1:${port}`;
|
||||
await waitForGatewayReady(url, gatewayToken);
|
||||
const agentRes = await callGateway({
|
||||
url,
|
||||
token: gatewayToken,
|
||||
method: "agent",
|
||||
params: {
|
||||
sessionKey: `agent:main:prompt-probe-${randomUUID()}`,
|
||||
idempotencyKey: `idem-${randomUUID()}`,
|
||||
message: "Reply with exactly: PROMPT PROBE OK.",
|
||||
...(GATEWAY_PROMPT_MODE === "extra" ? { extraSystemPrompt: prompt } : {}),
|
||||
deliver: false,
|
||||
},
|
||||
timeoutMs: 15_000,
|
||||
clientName: "cli",
|
||||
mode: "cli",
|
||||
});
|
||||
if (typeof agentRes.runId !== "string" || agentRes.runId.trim().length === 0) {
|
||||
return {
|
||||
prompt,
|
||||
ok: false,
|
||||
transport: "gateway",
|
||||
promptMode: GATEWAY_PROMPT_MODE,
|
||||
error: redactForDevToolLog(`missing runId: ${JSON.stringify(agentRes)}`),
|
||||
matchedExtraUsage400: false,
|
||||
capture: summarizeCapture(proxy?.getLastCapture(), prompt),
|
||||
...promptProbeTmpResult(tmpDir),
|
||||
};
|
||||
}
|
||||
const waitRes = await callGateway({
|
||||
url,
|
||||
token: gatewayToken,
|
||||
method: "agent.wait",
|
||||
params: { runId: agentRes.runId, timeoutMs: GATEWAY_TIMEOUT_MS },
|
||||
timeoutMs: GATEWAY_TIMEOUT_MS + 10_000,
|
||||
clientName: "cli",
|
||||
mode: "cli",
|
||||
});
|
||||
const text = extractPayloadText(waitRes);
|
||||
const logTail = await readLogTail(logPath);
|
||||
const matched400 = matchesExtraUsage400(waitRes.error, logTail, JSON.stringify(waitRes));
|
||||
return {
|
||||
prompt,
|
||||
ok: waitRes.status === "ok" && !matched400,
|
||||
transport: "gateway",
|
||||
promptMode: GATEWAY_PROMPT_MODE,
|
||||
status: waitRes.status,
|
||||
text: text || undefined,
|
||||
error:
|
||||
waitRes.status === "ok"
|
||||
? undefined
|
||||
: redactForDevToolLog(waitRes.error || logTail || "agent.wait failed"),
|
||||
matchedExtraUsage400: matched400,
|
||||
capture: summarizeCapture(proxy?.getLastCapture(), prompt),
|
||||
...promptProbeTmpResult(tmpDir),
|
||||
};
|
||||
} finally {
|
||||
const gatewayStopped = (await gateway?.stop().catch(() => false)) ?? true;
|
||||
await proxy?.stop().catch(() => {});
|
||||
if (gatewayStopped) {
|
||||
await cleanupPromptProbeTmpDir(tmpDir).catch(() => {});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function main() {
|
||||
if (!PROMPT_TEXT && !PROMPT_LIST_JSON) {
|
||||
throw new Error("missing OPENCLAW_PROMPT_TEXT or OPENCLAW_PROMPT_LIST_JSON");
|
||||
}
|
||||
const prompts = PROMPT_LIST_JSON ? (JSON.parse(PROMPT_LIST_JSON) as string[]) : [PROMPT_TEXT];
|
||||
const results: PromptResult[] = [];
|
||||
for (const prompt of prompts) {
|
||||
results.push(
|
||||
TRANSPORT === "direct" ? await runDirectPrompt(prompt) : await runGatewayPrompt(prompt),
|
||||
);
|
||||
}
|
||||
console.log(
|
||||
JSON.stringify(
|
||||
{
|
||||
transport: TRANSPORT,
|
||||
...(TRANSPORT === "gateway" ? { promptMode: GATEWAY_PROMPT_MODE } : {}),
|
||||
capture: ENABLE_CAPTURE,
|
||||
results,
|
||||
},
|
||||
null,
|
||||
2,
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
export const testing = {
|
||||
cleanupPromptProbeTmpDir,
|
||||
installGatewayPromptParentSignalHandlers,
|
||||
matchesExtraUsage400,
|
||||
promptProbeTmpResult,
|
||||
readLogTail,
|
||||
readRequestBody,
|
||||
resolveAnthropicUpstreamUrl,
|
||||
stopGatewayPromptChild,
|
||||
summarizeCapture,
|
||||
summarizeText,
|
||||
};
|
||||
|
||||
if (import.meta.url === pathToFileURL(process.argv[1] ?? "").href) {
|
||||
await main().catch((error: unknown) => {
|
||||
console.error(redactForDevToolLog(error instanceof Error ? error.message : String(error)));
|
||||
process.exitCode = 1;
|
||||
});
|
||||
}
|
||||
228
scripts/apple-app-i18n.ts
Normal file
228
scripts/apple-app-i18n.ts
Normal file
@@ -0,0 +1,228 @@
|
||||
import { mkdir, readFile, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
import { NATIVE_I18N_LOCALES } from "./native-app-i18n.ts";
|
||||
|
||||
const HERE = path.dirname(fileURLToPath(import.meta.url));
|
||||
const ROOT = path.resolve(HERE, "..");
|
||||
const REQUIRED_LOCALES = ["en", ...NATIVE_I18N_LOCALES];
|
||||
const FORMAT_RE = /%(?:\d+\$)?[@a-z]/giu;
|
||||
const APPLE_LOCALE_DIRECTORIES: Record<string, string> = {
|
||||
"ja-JP": "ja",
|
||||
"zh-CN": "zh-Hans",
|
||||
"zh-TW": "zh-Hant",
|
||||
};
|
||||
const LOCALIZED_WRAPPER_CONTRACTS: Record<string, string[]> = {
|
||||
"apps/ios/Sources/Gateway/GatewayQuickSetupSheet.swift": [
|
||||
"fullRowToggle(_ title: LocalizedStringKey",
|
||||
],
|
||||
"apps/ios/WatchApp/Sources/WatchInboxView.swift": [
|
||||
"private struct WatchPrimaryLabel: View {\n let title: LocalizedStringKey",
|
||||
"private struct WatchSecondaryLabel: View {\n let title: LocalizedStringKey",
|
||||
"private struct WatchSecondaryButton: View {\n let title: LocalizedStringKey",
|
||||
"private struct WatchDecisionButton: View {\n let title: LocalizedStringKey",
|
||||
],
|
||||
};
|
||||
|
||||
const CATALOGS = [
|
||||
{
|
||||
path: "apps/ios/Resources/Localizable.xcstrings",
|
||||
coverage: {
|
||||
"apps/ios/ShareExtension/ShareViewController.swift": [
|
||||
"Add a message, then tap Send.",
|
||||
"Cancel",
|
||||
"Edit text, then tap Send.",
|
||||
"Invalid saved gateway URL.",
|
||||
"Message is empty.",
|
||||
"OpenClaw is not connected to a gateway yet.",
|
||||
"Preparing share…",
|
||||
"Send failed: %@",
|
||||
"Send to OpenClaw",
|
||||
"Sending to OpenClaw gateway…",
|
||||
"Sent to OpenClaw.",
|
||||
],
|
||||
"apps/ios/Sources/Design/SettingsChannelsDestination.swift": ["Logout"],
|
||||
"apps/ios/Sources/Design/ChatProTab.swift": [
|
||||
"Check OpenClaw status",
|
||||
"Help me start a realtime voice session from this phone.",
|
||||
"Help me start voice chat",
|
||||
"Show me which phone controls and device capabilities are available right now.",
|
||||
"Summarize the current OpenClaw status and tell me what needs attention.",
|
||||
"What can I control here?",
|
||||
"What would you like to work on?",
|
||||
],
|
||||
"apps/ios/Sources/Gateway/GatewayProblemView.swift": ["Done"],
|
||||
"apps/ios/Sources/Gateway/GatewayQuickSetupSheet.swift": [
|
||||
"Close",
|
||||
"Connect",
|
||||
"Connect to a Gateway?",
|
||||
"Connecting…",
|
||||
"Don’t show this again",
|
||||
"No gateways found yet. Make sure your gateway is running and Bonjour discovery is enabled.",
|
||||
"Not now",
|
||||
"Quick Setup",
|
||||
],
|
||||
"apps/ios/Sources/Gateway/GatewayTrustPromptAlert.swift": [
|
||||
"Cancel",
|
||||
"First-time TLS connection.\n\nVerify this SHA-256 fingerprint out-of-band before trusting:\n%@",
|
||||
"Trust and connect",
|
||||
"Trust this gateway?",
|
||||
],
|
||||
"apps/ios/Sources/Onboarding/OnboardingWizardView.swift": ["Save"],
|
||||
"apps/ios/Sources/RootTabs.swift": ["Agent", "Chat", "Control", "Settings", "Talk"],
|
||||
"apps/ios/WatchApp/Sources/WatchInboxView.swift": [
|
||||
"Approve",
|
||||
"Chat",
|
||||
"Continue on iPhone",
|
||||
"Deny",
|
||||
"Message OpenClaw",
|
||||
"No chat synced",
|
||||
"Open all approvals",
|
||||
"Refresh",
|
||||
"Review again",
|
||||
"Talk to Claw",
|
||||
"Tap the message pill below to start from your watch.",
|
||||
"You",
|
||||
],
|
||||
"apps/shared/OpenClawKit/Sources/OpenClawChatUI/ChatMessageViews.swift": ["Writing"],
|
||||
},
|
||||
},
|
||||
{
|
||||
path: "apps/macos/Sources/OpenClaw/Resources/Localizable.xcstrings",
|
||||
coverage: {
|
||||
"apps/macos/Sources/OpenClaw/ChannelsSettings+ChannelSections.swift": [
|
||||
"Logout",
|
||||
"Refresh",
|
||||
"Save",
|
||||
],
|
||||
"apps/macos/Sources/OpenClaw/CronSettings+Rows.swift": ["Run now"],
|
||||
},
|
||||
},
|
||||
] as const;
|
||||
|
||||
type Catalog = {
|
||||
sourceLanguage?: string;
|
||||
strings?: Record<
|
||||
string,
|
||||
{
|
||||
localizations?: Record<string, { stringUnit?: { state?: string; value?: string } }>;
|
||||
}
|
||||
>;
|
||||
};
|
||||
|
||||
function formatTokens(value: string): string[] {
|
||||
return [...value.matchAll(FORMAT_RE)].map((match) => match[0]).toSorted();
|
||||
}
|
||||
|
||||
function stringsLiteral(value: string): string {
|
||||
return JSON.stringify(value);
|
||||
}
|
||||
|
||||
export async function checkAppleAppI18n() {
|
||||
let checked = 0;
|
||||
for (const [sourcePath, contracts] of Object.entries(LOCALIZED_WRAPPER_CONTRACTS)) {
|
||||
const source = await readFile(path.join(ROOT, sourcePath), "utf8");
|
||||
const missing = contracts.filter((contract) => !source.includes(contract));
|
||||
if (missing.length) {
|
||||
throw new Error(
|
||||
`Apple i18n wrapper ${sourcePath} bypasses localized string lookup: ${missing.join(", ")}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
for (const spec of CATALOGS) {
|
||||
const catalogPath = path.join(ROOT, spec.path);
|
||||
const catalog = JSON.parse(await readFile(catalogPath, "utf8")) as Catalog;
|
||||
if (catalog.sourceLanguage !== "en" || !catalog.strings) {
|
||||
throw new Error(`invalid Apple string catalog: ${spec.path}`);
|
||||
}
|
||||
|
||||
const expectedKeys = new Set(Object.values(spec.coverage).flat());
|
||||
const actualKeys = new Set(Object.keys(catalog.strings));
|
||||
const missingKeys = [...expectedKeys].filter((key) => !actualKeys.has(key));
|
||||
const extraKeys = [...actualKeys].filter((key) => !expectedKeys.has(key));
|
||||
if (missingKeys.length || extraKeys.length) {
|
||||
throw new Error(
|
||||
[
|
||||
`Apple catalog ${spec.path} does not match its phased source coverage.`,
|
||||
`missing=${missingKeys.join(",") || "none"}`,
|
||||
`extra=${extraKeys.join(",") || "none"}`,
|
||||
].join("\n"),
|
||||
);
|
||||
}
|
||||
|
||||
for (const [sourcePath, keys] of Object.entries(spec.coverage)) {
|
||||
const source = await readFile(path.join(ROOT, sourcePath), "utf8");
|
||||
const absent = keys.filter((key) => {
|
||||
const escapedKey = JSON.stringify(key).slice(1, -1);
|
||||
return !source.includes(key) && !source.includes(escapedKey);
|
||||
});
|
||||
if (absent.length) {
|
||||
throw new Error(
|
||||
`Apple i18n coverage ${sourcePath} no longer contains: ${absent.join(", ")}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
for (const [key, entry] of Object.entries(catalog.strings)) {
|
||||
const sourceTokens = formatTokens(key);
|
||||
for (const locale of REQUIRED_LOCALES) {
|
||||
const unit = entry.localizations?.[locale]?.stringUnit;
|
||||
const value = unit?.value?.trim();
|
||||
if (!value || unit?.state !== "translated") {
|
||||
throw new Error(
|
||||
`Apple catalog ${spec.path} is missing ${locale} for ${JSON.stringify(key)}`,
|
||||
);
|
||||
}
|
||||
if (formatTokens(value).join("\u0000") !== sourceTokens.join("\u0000")) {
|
||||
throw new Error(
|
||||
`Apple catalog ${spec.path} has placeholder drift in ${locale} for ${JSON.stringify(key)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
checked += 1;
|
||||
}
|
||||
}
|
||||
process.stdout.write(
|
||||
`apple-app-i18n: catalogs=${CATALOGS.length} keys=${checked} locales=${NATIVE_I18N_LOCALES.join(",")}\n`,
|
||||
);
|
||||
}
|
||||
|
||||
export async function compileMacosLocalizations(outputDir: string) {
|
||||
await checkAppleAppI18n();
|
||||
const spec = CATALOGS[1];
|
||||
const catalog = JSON.parse(await readFile(path.join(ROOT, spec.path), "utf8")) as Catalog;
|
||||
if (!catalog.strings) {
|
||||
throw new Error(`invalid Apple string catalog: ${spec.path}`);
|
||||
}
|
||||
|
||||
for (const locale of REQUIRED_LOCALES) {
|
||||
const localeDir = APPLE_LOCALE_DIRECTORIES[locale] ?? locale;
|
||||
const lprojDir = path.join(outputDir, `${localeDir}.lproj`);
|
||||
const lines = Object.entries(catalog.strings)
|
||||
.toSorted(([left], [right]) => left.localeCompare(right))
|
||||
.map(([key, entry]) => {
|
||||
const value = entry.localizations?.[locale]?.stringUnit?.value;
|
||||
if (!value) {
|
||||
throw new Error(
|
||||
`Apple catalog ${spec.path} is missing ${locale} for ${JSON.stringify(key)}`,
|
||||
);
|
||||
}
|
||||
return `${stringsLiteral(key)} = ${stringsLiteral(value)};`;
|
||||
});
|
||||
await mkdir(lprojDir, { recursive: true });
|
||||
await writeFile(path.join(lprojDir, "Localizable.strings"), `${lines.join("\n")}\n`, "utf8");
|
||||
}
|
||||
}
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(path.resolve(process.argv[1])).href) {
|
||||
const [command, flag, value] = process.argv.slice(2);
|
||||
if (command === "check") {
|
||||
await checkAppleAppI18n();
|
||||
} else if (command === "compile-macos" && flag === "--output" && value) {
|
||||
await compileMacosLocalizations(path.resolve(value));
|
||||
} else {
|
||||
throw new Error(
|
||||
"usage: node --import tsx scripts/apple-app-i18n.ts check|compile-macos --output <dir>",
|
||||
);
|
||||
}
|
||||
}
|
||||
1036
scripts/audit-seams.mjs
Normal file
1036
scripts/audit-seams.mjs
Normal file
File diff suppressed because it is too large
Load Diff
89
scripts/auth-monitor.sh
Executable file
89
scripts/auth-monitor.sh
Executable file
@@ -0,0 +1,89 @@
|
||||
#!/bin/bash
|
||||
# Auth Expiry Monitor
|
||||
# Run via cron or systemd timer to get proactive notifications
|
||||
# before Claude Code auth expires.
|
||||
#
|
||||
# Suggested cron: */30 * * * * /path/to/openclaw/scripts/auth-monitor.sh
|
||||
#
|
||||
# Environment variables:
|
||||
# NOTIFY_PHONE - Phone number to send OpenClaw notification (e.g., +1234567890)
|
||||
# NOTIFY_NTFY - ntfy.sh topic for push notifications (e.g., openclaw-alerts)
|
||||
# WARN_HOURS - Hours before expiry to warn (default: 2)
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
CLAUDE_CREDS="$HOME/.claude/.credentials.json"
|
||||
STATE_FILE="$HOME/.openclaw/auth-monitor-state"
|
||||
|
||||
# Configuration
|
||||
WARN_HOURS="${WARN_HOURS:-2}"
|
||||
NOTIFY_PHONE="${NOTIFY_PHONE:-}"
|
||||
NOTIFY_NTFY="${NOTIFY_NTFY:-}"
|
||||
|
||||
# State tracking to avoid spam
|
||||
mkdir -p "$(dirname "$STATE_FILE")"
|
||||
LAST_NOTIFIED=$(cat "$STATE_FILE" 2>/dev/null || echo "0")
|
||||
NOW=$(date +%s)
|
||||
|
||||
# Only notify once per hour max
|
||||
MIN_INTERVAL=3600
|
||||
|
||||
send_notification() {
|
||||
local message="$1"
|
||||
local priority="${2:-default}"
|
||||
|
||||
echo "$(date '+%Y-%m-%d %H:%M:%S') - $message"
|
||||
|
||||
# Check if we notified recently
|
||||
if [ $((NOW - LAST_NOTIFIED)) -lt $MIN_INTERVAL ]; then
|
||||
echo "Skipping notification (sent recently)"
|
||||
return
|
||||
fi
|
||||
|
||||
# Send via OpenClaw if phone configured and auth still valid
|
||||
if [ -n "$NOTIFY_PHONE" ]; then
|
||||
# Check if we can still use openclaw
|
||||
if "$SCRIPT_DIR/claude-auth-status.sh" simple 2>/dev/null | grep -q "OK\|EXPIRING"; then
|
||||
echo "Sending via OpenClaw to $NOTIFY_PHONE..."
|
||||
openclaw send --to "$NOTIFY_PHONE" --message "$message" 2>/dev/null || true
|
||||
fi
|
||||
fi
|
||||
|
||||
# Send via ntfy.sh if configured
|
||||
if [ -n "$NOTIFY_NTFY" ]; then
|
||||
echo "Sending via ntfy.sh to $NOTIFY_NTFY..."
|
||||
curl -s -o /dev/null \
|
||||
-H "Title: OpenClaw Auth Alert" \
|
||||
-H "Priority: $priority" \
|
||||
-H "Tags: warning,key" \
|
||||
-d "$message" \
|
||||
"https://ntfy.sh/$NOTIFY_NTFY" || true
|
||||
fi
|
||||
|
||||
# Update state
|
||||
echo "$NOW" > "$STATE_FILE"
|
||||
}
|
||||
|
||||
# Check auth status
|
||||
if [ ! -f "$CLAUDE_CREDS" ]; then
|
||||
send_notification "Claude Code credentials missing! Run: claude setup-token" "high"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
EXPIRES_AT=$(jq -r '.claudeAiOauth.expiresAt // 0' "$CLAUDE_CREDS")
|
||||
NOW_MS=$((NOW * 1000))
|
||||
DIFF_MS=$((EXPIRES_AT - NOW_MS))
|
||||
HOURS_LEFT=$((DIFF_MS / 3600000))
|
||||
MINS_LEFT=$(((DIFF_MS % 3600000) / 60000))
|
||||
|
||||
if [ "$DIFF_MS" -lt 0 ]; then
|
||||
send_notification "Claude Code auth EXPIRED! OpenClaw is down. Run on the OpenClaw host: ${SCRIPT_DIR}/mobile-reauth.sh" "urgent"
|
||||
exit 1
|
||||
elif [ "$HOURS_LEFT" -lt "$WARN_HOURS" ]; then
|
||||
send_notification "Claude Code auth expires in ${HOURS_LEFT}h ${MINS_LEFT}m. Consider re-auth soon." "high"
|
||||
exit 0
|
||||
else
|
||||
echo "$(date '+%Y-%m-%d %H:%M:%S') - Auth OK: ${HOURS_LEFT}h ${MINS_LEFT}m remaining"
|
||||
exit 0
|
||||
fi
|
||||
1301
scripts/bench-cli-startup.ts
Normal file
1301
scripts/bench-cli-startup.ts
Normal file
File diff suppressed because it is too large
Load Diff
1690
scripts/bench-gateway-restart.ts
Normal file
1690
scripts/bench-gateway-restart.ts
Normal file
File diff suppressed because it is too large
Load Diff
1026
scripts/bench-gateway-startup.ts
Normal file
1026
scripts/bench-gateway-startup.ts
Normal file
File diff suppressed because it is too large
Load Diff
244
scripts/bench-model.ts
Normal file
244
scripts/bench-model.ts
Normal file
@@ -0,0 +1,244 @@
|
||||
// Bench Model script supports OpenClaw repository automation.
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { completeSimple, type Model } from "openclaw/plugin-sdk/llm";
|
||||
import { parseStrictIntegerOption } from "./lib/dev-tooling-safety.ts";
|
||||
|
||||
type Usage = {
|
||||
input?: number;
|
||||
output?: number;
|
||||
cacheRead?: number;
|
||||
cacheWrite?: number;
|
||||
totalTokens?: number;
|
||||
};
|
||||
|
||||
type RunResult = {
|
||||
durationMs: number;
|
||||
usage?: Usage;
|
||||
};
|
||||
|
||||
type CliOptions = {
|
||||
help: boolean;
|
||||
prompt: string;
|
||||
runs: number;
|
||||
};
|
||||
|
||||
const DEFAULT_PROMPT = "Reply with a single word: ok. No punctuation or extra text.";
|
||||
const DEFAULT_RUNS = 10;
|
||||
const BOOLEAN_FLAGS = new Set(["--help", "-h"]);
|
||||
const VALUE_FLAGS = new Set(["--prompt", "--runs"]);
|
||||
|
||||
class CliArgumentError extends Error {
|
||||
override name = "CliArgumentError";
|
||||
}
|
||||
|
||||
function readValue(argv: string[], index: number, flag: string): string {
|
||||
const value = argv[index + 1]?.trim() ?? "";
|
||||
if (!value || value.startsWith("-")) {
|
||||
throw new CliArgumentError(`${flag} requires a value`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function validateCliArgs(argv: string[]): void {
|
||||
const seenValueFlags = new Set<string>();
|
||||
for (let index = 0; index < argv.length; index += 1) {
|
||||
const arg = argv[index] ?? "";
|
||||
if (BOOLEAN_FLAGS.has(arg)) {
|
||||
continue;
|
||||
}
|
||||
if (VALUE_FLAGS.has(arg)) {
|
||||
if (seenValueFlags.has(arg)) {
|
||||
throw new CliArgumentError(`${arg} was provided more than once`);
|
||||
}
|
||||
seenValueFlags.add(arg);
|
||||
readValue(argv, index, arg);
|
||||
index += 1;
|
||||
continue;
|
||||
}
|
||||
throw new CliArgumentError(`Unknown argument: ${arg}`);
|
||||
}
|
||||
}
|
||||
|
||||
function parseArg(argv: string[], flag: string): string | undefined {
|
||||
const index = argv.indexOf(flag);
|
||||
if (index === -1) {
|
||||
return undefined;
|
||||
}
|
||||
return readValue(argv, index, flag);
|
||||
}
|
||||
|
||||
function parseRuns(raw: string | undefined): number {
|
||||
return parseStrictIntegerOption({
|
||||
fallback: DEFAULT_RUNS,
|
||||
label: "--runs",
|
||||
min: 1,
|
||||
raw,
|
||||
});
|
||||
}
|
||||
|
||||
function parseArgs(argv = process.argv.slice(2)): CliOptions {
|
||||
validateCliArgs(argv);
|
||||
return {
|
||||
help: argv.includes("--help") || argv.includes("-h"),
|
||||
prompt: parseArg(argv, "--prompt") ?? DEFAULT_PROMPT,
|
||||
runs: parseRuns(parseArg(argv, "--runs")),
|
||||
};
|
||||
}
|
||||
|
||||
function printUsage(): void {
|
||||
console.log(`OpenClaw model latency benchmark
|
||||
|
||||
Usage:
|
||||
node --import tsx scripts/bench-model.ts [options]
|
||||
|
||||
Options:
|
||||
--runs <n> Runs per model (default: ${DEFAULT_RUNS})
|
||||
--prompt <text> Prompt to send to each model
|
||||
--help, -h Show this text
|
||||
|
||||
Environment:
|
||||
ANTHROPIC_API_KEY
|
||||
MINIMAX_API_KEY
|
||||
MINIMAX_BASE_URL
|
||||
MINIMAX_MODEL
|
||||
`);
|
||||
}
|
||||
|
||||
function median(values: number[]): number {
|
||||
if (values.length === 0) {
|
||||
return 0;
|
||||
}
|
||||
const sorted = [...values].toSorted((a, b) => a - b);
|
||||
const mid = Math.floor(sorted.length / 2);
|
||||
if (sorted.length % 2 === 0) {
|
||||
return Math.round((sorted[mid - 1] + sorted[mid]) / 2);
|
||||
}
|
||||
return sorted[mid];
|
||||
}
|
||||
|
||||
async function runModel(opts: {
|
||||
label: string;
|
||||
model: Model;
|
||||
apiKey: string;
|
||||
runs: number;
|
||||
prompt: string;
|
||||
}): Promise<RunResult[]> {
|
||||
const results: RunResult[] = [];
|
||||
for (let i = 0; i < opts.runs; i += 1) {
|
||||
const started = Date.now();
|
||||
const res = await completeSimple(
|
||||
opts.model,
|
||||
{
|
||||
messages: [
|
||||
{
|
||||
role: "user",
|
||||
content: opts.prompt,
|
||||
timestamp: Date.now(),
|
||||
},
|
||||
],
|
||||
},
|
||||
{ apiKey: opts.apiKey, maxTokens: 64 },
|
||||
);
|
||||
const durationMs = Date.now() - started;
|
||||
results.push({ durationMs, usage: res.usage });
|
||||
console.log(`${opts.label} run ${i + 1}/${opts.runs}: ${durationMs}ms`);
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
async function main(argv = process.argv.slice(2)): Promise<void> {
|
||||
const options = parseArgs(argv);
|
||||
if (options.help) {
|
||||
printUsage();
|
||||
return;
|
||||
}
|
||||
|
||||
const anthropicKey = process.env.ANTHROPIC_API_KEY?.trim();
|
||||
const minimaxKey = process.env.MINIMAX_API_KEY?.trim();
|
||||
if (!anthropicKey) {
|
||||
throw new Error("Missing ANTHROPIC_API_KEY in environment.");
|
||||
}
|
||||
if (!minimaxKey) {
|
||||
throw new Error("Missing MINIMAX_API_KEY in environment.");
|
||||
}
|
||||
|
||||
const minimaxBaseUrl = process.env.MINIMAX_BASE_URL?.trim() || "https://api.minimax.io/v1";
|
||||
const minimaxModelId = process.env.MINIMAX_MODEL?.trim() || "MiniMax-M2.1";
|
||||
|
||||
const minimaxModel: Model<"openai-completions"> = {
|
||||
id: minimaxModelId,
|
||||
name: `MiniMax ${minimaxModelId}`,
|
||||
api: "openai-completions",
|
||||
provider: "minimax",
|
||||
baseUrl: minimaxBaseUrl,
|
||||
reasoning: false,
|
||||
input: ["text"],
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 200000,
|
||||
maxTokens: 8192,
|
||||
};
|
||||
const opusModel: Model<"anthropic-messages"> = {
|
||||
id: "claude-opus-4-6",
|
||||
name: "Claude Opus 4.6",
|
||||
api: "anthropic-messages",
|
||||
provider: "anthropic",
|
||||
reasoning: true,
|
||||
input: ["text", "image"],
|
||||
cost: { input: 15, output: 75, cacheRead: 1.5, cacheWrite: 18.75 },
|
||||
contextWindow: 200000,
|
||||
maxTokens: 32000,
|
||||
};
|
||||
|
||||
console.log(`Prompt: ${options.prompt}`);
|
||||
console.log(`Runs: ${options.runs}`);
|
||||
console.log("");
|
||||
|
||||
const minimaxResults = await runModel({
|
||||
label: "minimax",
|
||||
model: minimaxModel,
|
||||
apiKey: minimaxKey,
|
||||
runs: options.runs,
|
||||
prompt: options.prompt,
|
||||
});
|
||||
const opusResults = await runModel({
|
||||
label: "opus",
|
||||
model: opusModel,
|
||||
apiKey: anthropicKey,
|
||||
runs: options.runs,
|
||||
prompt: options.prompt,
|
||||
});
|
||||
|
||||
const summarize = (label: string, results: RunResult[]) => {
|
||||
const durations = results.map((r) => r.durationMs);
|
||||
const med = median(durations);
|
||||
const min = Math.min(...durations);
|
||||
const max = Math.max(...durations);
|
||||
return { label, med, min, max };
|
||||
};
|
||||
|
||||
const summary = [summarize("minimax", minimaxResults), summarize("opus", opusResults)];
|
||||
console.log("");
|
||||
console.log("Summary (ms):");
|
||||
for (const row of summary) {
|
||||
console.log(`${row.label.padEnd(7)} median=${row.med} min=${row.min} max=${row.max}`);
|
||||
}
|
||||
}
|
||||
|
||||
export const testing = {
|
||||
median,
|
||||
parseArgs,
|
||||
parseRuns,
|
||||
validateCliArgs,
|
||||
};
|
||||
|
||||
if (import.meta.url === pathToFileURL(process.argv[1] ?? "").href) {
|
||||
main().catch((err: unknown) => {
|
||||
if (err instanceof CliArgumentError) {
|
||||
console.error(err.message);
|
||||
process.exitCode = 1;
|
||||
return;
|
||||
}
|
||||
console.error(err instanceof Error ? err.stack : String(err));
|
||||
process.exitCode = 1;
|
||||
});
|
||||
}
|
||||
673
scripts/bench-sqlite-state.ts
Normal file
673
scripts/bench-sqlite-state.ts
Normal file
@@ -0,0 +1,673 @@
|
||||
// SQLite state benchmark seeds OpenClaw DBs and reports hot-query proof lines.
|
||||
import fs from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import type { DatabaseSync } from "node:sqlite";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import {
|
||||
openOpenClawAgentDatabase,
|
||||
closeOpenClawAgentDatabasesForTest,
|
||||
} from "../src/state/openclaw-agent-db.js";
|
||||
import {
|
||||
closeOpenClawStateDatabaseForTest,
|
||||
openOpenClawStateDatabase,
|
||||
} from "../src/state/openclaw-state-db.js";
|
||||
import { parseStrictIntegerOption } from "./lib/dev-tooling-safety.ts";
|
||||
|
||||
type ProfileId = "smoke" | "default" | "large";
|
||||
|
||||
type ProfileConfig = {
|
||||
agentCacheEntries: number;
|
||||
agentCount: number;
|
||||
channelIngressEvents: number;
|
||||
cronJobs: number;
|
||||
cronRunLogs: number;
|
||||
deliveryQueueEntries: number;
|
||||
pluginStateEntries: number;
|
||||
queryRuns: number;
|
||||
};
|
||||
|
||||
type TimedQuery = {
|
||||
p50Ms: number;
|
||||
p95Ms: number;
|
||||
query: string;
|
||||
rows: number;
|
||||
};
|
||||
|
||||
type BenchmarkReport = {
|
||||
integrity: {
|
||||
agent: string[];
|
||||
state: string;
|
||||
};
|
||||
node: string;
|
||||
paths: {
|
||||
agentDatabases: string[];
|
||||
artifact: string | null;
|
||||
stateDatabase: string;
|
||||
stateDir: string;
|
||||
};
|
||||
profile: ProfileId;
|
||||
queries: TimedQuery[];
|
||||
rows: {
|
||||
agentCacheEntries: number;
|
||||
agentDatabases: number;
|
||||
channelIngressEvents: number;
|
||||
cronJobs: number;
|
||||
cronRunLogs: number;
|
||||
deliveryQueueEntries: number;
|
||||
pluginStateEntries: number;
|
||||
stateRows: number;
|
||||
};
|
||||
timingsMs: {
|
||||
checkpoint: number;
|
||||
seed: number;
|
||||
total: number;
|
||||
};
|
||||
walBytes: {
|
||||
agentAfter: number[];
|
||||
agentBefore: number[];
|
||||
stateAfter: number;
|
||||
stateBefore: number;
|
||||
};
|
||||
};
|
||||
|
||||
const PROFILES: Record<ProfileId, ProfileConfig> = {
|
||||
smoke: {
|
||||
agentCacheEntries: 1_000,
|
||||
agentCount: 2,
|
||||
channelIngressEvents: 1_000,
|
||||
cronJobs: 100,
|
||||
cronRunLogs: 1_000,
|
||||
deliveryQueueEntries: 1_000,
|
||||
pluginStateEntries: 1_000,
|
||||
queryRuns: 12,
|
||||
},
|
||||
default: {
|
||||
agentCacheEntries: 20_000,
|
||||
agentCount: 5,
|
||||
channelIngressEvents: 10_000,
|
||||
cronJobs: 1_000,
|
||||
cronRunLogs: 50_000,
|
||||
deliveryQueueEntries: 50_000,
|
||||
pluginStateEntries: 20_000,
|
||||
queryRuns: 30,
|
||||
},
|
||||
large: {
|
||||
agentCacheEntries: 50_000,
|
||||
agentCount: 10,
|
||||
channelIngressEvents: 100_000,
|
||||
cronJobs: 5_000,
|
||||
cronRunLogs: 250_000,
|
||||
deliveryQueueEntries: 200_000,
|
||||
pluginStateEntries: 100_000,
|
||||
queryRuns: 40,
|
||||
},
|
||||
};
|
||||
|
||||
type CliOptions = {
|
||||
output: string | null;
|
||||
profile: ProfileId;
|
||||
stateDir: string | null;
|
||||
};
|
||||
|
||||
const BOOLEAN_FLAGS = new Set(["--help"]);
|
||||
const VALUE_FLAGS = new Set(["--output", "--profile", "--state-dir"]);
|
||||
|
||||
class CliUsageError extends Error {
|
||||
override name = "CliUsageError";
|
||||
}
|
||||
|
||||
function parseFlagValue(flag: string, argv: string[]): string | undefined {
|
||||
const index = argv.indexOf(flag);
|
||||
if (index === -1) {
|
||||
return undefined;
|
||||
}
|
||||
const value = argv[index + 1];
|
||||
if (!value || value.startsWith("-")) {
|
||||
throw new CliUsageError(`${flag} requires a value`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function hasFlag(flag: string, argv = process.argv.slice(2)): boolean {
|
||||
return argv.includes(flag);
|
||||
}
|
||||
|
||||
function validateArgs(argv: string[]): void {
|
||||
const seenValueFlags = new Set<string>();
|
||||
for (let index = 0; index < argv.length; index += 1) {
|
||||
const arg = argv[index] ?? "";
|
||||
if (BOOLEAN_FLAGS.has(arg)) {
|
||||
continue;
|
||||
}
|
||||
if (VALUE_FLAGS.has(arg)) {
|
||||
if (seenValueFlags.has(arg)) {
|
||||
throw new CliUsageError(`${arg} was provided more than once`);
|
||||
}
|
||||
seenValueFlags.add(arg);
|
||||
const value = argv[index + 1];
|
||||
if (!value || value.startsWith("-")) {
|
||||
throw new CliUsageError(`${arg} requires a value`);
|
||||
}
|
||||
index += 1;
|
||||
continue;
|
||||
}
|
||||
throw new CliUsageError(`Unknown argument: ${arg}`);
|
||||
}
|
||||
}
|
||||
|
||||
function parseProfile(raw: string | undefined): ProfileId {
|
||||
if (!raw) {
|
||||
return "default";
|
||||
}
|
||||
if (raw === "smoke" || raw === "default" || raw === "large") {
|
||||
return raw;
|
||||
}
|
||||
throw new CliUsageError(
|
||||
`--profile must be one of smoke, default, large; got ${JSON.stringify(raw)}`,
|
||||
);
|
||||
}
|
||||
|
||||
function parseOptions(argv = process.argv.slice(2)): CliOptions {
|
||||
validateArgs(argv);
|
||||
return {
|
||||
output: parseFlagValue("--output", argv) ?? null,
|
||||
profile: parseProfile(parseFlagValue("--profile", argv)),
|
||||
stateDir: parseFlagValue("--state-dir", argv) ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
function applyScale(config: ProfileConfig): ProfileConfig {
|
||||
const scale = parseStrictIntegerOption({
|
||||
fallback: 1,
|
||||
label: "SQLITE_PERF_SCALE",
|
||||
min: 1,
|
||||
raw: process.env["SQLITE_PERF_SCALE"],
|
||||
});
|
||||
if (scale === 1) {
|
||||
return config;
|
||||
}
|
||||
return {
|
||||
agentCacheEntries: config.agentCacheEntries * scale,
|
||||
agentCount: config.agentCount,
|
||||
channelIngressEvents: config.channelIngressEvents * scale,
|
||||
cronJobs: config.cronJobs * scale,
|
||||
cronRunLogs: config.cronRunLogs * scale,
|
||||
deliveryQueueEntries: config.deliveryQueueEntries * scale,
|
||||
pluginStateEntries: config.pluginStateEntries * scale,
|
||||
queryRuns: config.queryRuns,
|
||||
};
|
||||
}
|
||||
|
||||
function printUsage(): void {
|
||||
console.log(`OpenClaw SQLite state benchmark
|
||||
|
||||
Usage:
|
||||
node --import tsx scripts/bench-sqlite-state.ts [options]
|
||||
|
||||
Options:
|
||||
--profile <smoke|default|large> Data volume profile (default: default)
|
||||
--state-dir <path> Reuse a state directory instead of a temp dir
|
||||
--output <path> Write machine-readable JSON report
|
||||
--help Show this text
|
||||
|
||||
Environment:
|
||||
SQLITE_PERF_SCALE=<n> Multiplies row counts for the selected profile
|
||||
`);
|
||||
}
|
||||
|
||||
function nowMs(): number {
|
||||
return Number(process.hrtime.bigint()) / 1e6;
|
||||
}
|
||||
|
||||
function fileSize(pathname: string): number {
|
||||
try {
|
||||
return fs.statSync(pathname).size;
|
||||
} catch {
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
function walSize(pathname: string): number {
|
||||
return fileSize(`${pathname}-wal`);
|
||||
}
|
||||
|
||||
function stateRowCount(config: ProfileConfig): number {
|
||||
return (
|
||||
config.channelIngressEvents +
|
||||
config.cronJobs +
|
||||
config.cronRunLogs +
|
||||
config.deliveryQueueEntries +
|
||||
config.pluginStateEntries
|
||||
);
|
||||
}
|
||||
|
||||
function seedStateDatabase(db: DatabaseSync, config: ProfileConfig): void {
|
||||
db.exec("BEGIN IMMEDIATE;");
|
||||
try {
|
||||
seedCronJobs(db, config.cronJobs);
|
||||
seedCronRunLogs(db, config.cronRunLogs);
|
||||
seedDeliveryQueue(db, config.deliveryQueueEntries);
|
||||
seedPluginState(db, config.pluginStateEntries);
|
||||
seedChannelIngress(db, config.channelIngressEvents);
|
||||
db.exec("COMMIT;");
|
||||
} catch (err) {
|
||||
db.exec("ROLLBACK;");
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
function seedCronJobs(db: DatabaseSync, count: number): void {
|
||||
const insert = db.prepare(`
|
||||
INSERT INTO cron_jobs (
|
||||
store_key, job_id, name, description, enabled, delete_after_run, created_at_ms,
|
||||
agent_id, session_key, schedule_kind, schedule_expr, schedule_tz, every_ms,
|
||||
anchor_ms, at, stagger_ms, session_target, wake_mode, payload_kind,
|
||||
payload_message, payload_model, payload_fallbacks_json, payload_thinking,
|
||||
payload_timeout_seconds, payload_allow_unsafe_external_content,
|
||||
payload_external_content_source_json, payload_light_context, payload_tools_allow_json,
|
||||
delivery_mode, delivery_channel, delivery_to, delivery_thread_id, delivery_account_id,
|
||||
delivery_best_effort, delivery_completion_mode, delivery_completion_to,
|
||||
failure_delivery_mode, failure_delivery_channel, failure_delivery_to,
|
||||
failure_delivery_account_id, failure_alert_disabled, failure_alert_after,
|
||||
failure_alert_channel, failure_alert_to, failure_alert_cooldown_ms,
|
||||
failure_alert_include_skipped, failure_alert_mode, failure_alert_account_id,
|
||||
next_run_at_ms, running_at_ms, last_run_at_ms, last_run_status, last_error,
|
||||
last_duration_ms, consecutive_errors, consecutive_skipped, schedule_error_count,
|
||||
last_delivery_status, last_delivery_error, last_delivered, last_failure_alert_at_ms,
|
||||
job_json, state_json, runtime_updated_at_ms, schedule_identity, sort_order, updated_at
|
||||
) VALUES (
|
||||
?, ?, ?, NULL, ?, NULL, ?, ?, ?, 'every', NULL, NULL, ?, ?, NULL, NULL,
|
||||
'isolated', 'now', 'agentTurn', ?, 'openai/gpt-5.5', NULL, NULL, 60,
|
||||
0, NULL, 1, NULL, 'announce', 'telegram', ?, NULL, 'bench-account',
|
||||
1, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
|
||||
NULL, NULL, NULL, ?, NULL, ?, 'completed', NULL, ?, 0, 0, 0, 'sent',
|
||||
NULL, 1, NULL, ?, '{}', ?, ?, ?, ?
|
||||
)
|
||||
`);
|
||||
for (let i = 0; i < count; i += 1) {
|
||||
const jobId = `job-${String(i).padStart(8, "0")}`;
|
||||
const storeKey = `/state/cron/jobs-${i % 8}.json`;
|
||||
const updatedAt = 1_700_000_000_000 + i;
|
||||
insert.run(
|
||||
storeKey,
|
||||
jobId,
|
||||
`Benchmark job ${i}`,
|
||||
i % 5 === 0 ? 0 : 1,
|
||||
updatedAt - 100_000,
|
||||
`agent-${i % 16}`,
|
||||
`agent:agent-${i % 16}:main`,
|
||||
60_000 + (i % 120) * 1_000,
|
||||
updatedAt - 60_000,
|
||||
`Benchmark payload ${i}`,
|
||||
`chat-${i % 32}`,
|
||||
updatedAt + (i % 2_000) * 1_000,
|
||||
updatedAt - 1_000,
|
||||
50 + (i % 500),
|
||||
JSON.stringify({ id: jobId, seed: i }),
|
||||
updatedAt,
|
||||
`schedule-${i % 512}`,
|
||||
i,
|
||||
updatedAt,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function seedCronRunLogs(db: DatabaseSync, count: number): void {
|
||||
const insert = db.prepare(`
|
||||
INSERT INTO cron_run_logs (
|
||||
store_key, job_id, seq, ts, status, error, summary, diagnostics_summary,
|
||||
delivery_status, delivery_error, delivered, session_id, session_key, run_id,
|
||||
run_at_ms, duration_ms, next_run_at_ms, model, provider, total_tokens,
|
||||
entry_json, created_at
|
||||
) VALUES (?, ?, ?, ?, ?, NULL, ?, NULL, ?, NULL, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
`);
|
||||
for (let i = 0; i < count; i += 1) {
|
||||
const jobId = `job-${String(i % Math.max(1, Math.floor(count / 20))).padStart(8, "0")}`;
|
||||
const ts = 1_700_000_000_000 + i;
|
||||
insert.run(
|
||||
`/state/cron/jobs-${i % 8}.json`,
|
||||
jobId,
|
||||
Math.floor(i / 20),
|
||||
ts,
|
||||
i % 17 === 0 ? "failed" : "completed",
|
||||
`run ${i}`,
|
||||
i % 17 === 0 ? "failed" : "sent",
|
||||
i % 17 === 0 ? 0 : 1,
|
||||
`session-${i}`,
|
||||
`agent:agent-${i % 16}:main`,
|
||||
`run-${i}`,
|
||||
ts,
|
||||
20 + (i % 1_000),
|
||||
ts + 60_000,
|
||||
"openai/gpt-5.5",
|
||||
"openai",
|
||||
100 + (i % 2_000),
|
||||
JSON.stringify({ ts, jobId, action: "finished" }),
|
||||
ts,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function seedDeliveryQueue(db: DatabaseSync, count: number): void {
|
||||
const insert = db.prepare(`
|
||||
INSERT INTO delivery_queue_entries (
|
||||
queue_name, id, status, entry_kind, session_key, channel, target, account_id,
|
||||
retry_count, last_attempt_at, last_error, recovery_state, platform_send_started_at,
|
||||
entry_json, enqueued_at, updated_at, failed_at
|
||||
) VALUES (?, ?, ?, 'message', ?, ?, ?, ?, ?, ?, NULL, NULL, NULL, ?, ?, ?, ?)
|
||||
`);
|
||||
for (let i = 0; i < count; i += 1) {
|
||||
const status = i % 13 === 0 ? "failed" : i % 3 === 0 ? "sending" : "pending";
|
||||
const enqueuedAt = 1_700_000_000_000 + i;
|
||||
insert.run(
|
||||
"outbound",
|
||||
`delivery-${String(i).padStart(8, "0")}`,
|
||||
status,
|
||||
`agent:agent-${i % 16}:main`,
|
||||
i % 2 === 0 ? "telegram" : "discord",
|
||||
`target-${i % 256}`,
|
||||
`account-${i % 8}`,
|
||||
i % 5,
|
||||
status === "failed" ? enqueuedAt + 500 : null,
|
||||
JSON.stringify({ id: i, route: { channel: "telegram", to: `target-${i % 256}` } }),
|
||||
enqueuedAt,
|
||||
enqueuedAt + 100,
|
||||
status === "failed" ? enqueuedAt + 1_000 : null,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function seedPluginState(db: DatabaseSync, count: number): void {
|
||||
const insert = db.prepare(`
|
||||
INSERT INTO plugin_state_entries (
|
||||
plugin_id, namespace, entry_key, value_json, created_at, expires_at
|
||||
) VALUES (?, ?, ?, ?, ?, ?)
|
||||
`);
|
||||
for (let i = 0; i < count; i += 1) {
|
||||
insert.run(
|
||||
`plugin-${i % 12}`,
|
||||
`namespace-${i % 16}`,
|
||||
`entry-${String(i).padStart(8, "0")}`,
|
||||
JSON.stringify({ value: i, text: `payload ${i}` }),
|
||||
1_700_000_000_000 + i,
|
||||
i % 10 === 0 ? 1_800_000_000_000 + i : null,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function seedChannelIngress(db: DatabaseSync, count: number): void {
|
||||
const insert = db.prepare(`
|
||||
INSERT INTO channel_ingress_events (
|
||||
queue_name, event_id, channel_id, account_id, status, lane_key, payload_json,
|
||||
metadata_json, received_at, updated_at, claim_token, claim_owner, claimed_at,
|
||||
attempts, last_attempt_at, last_error, failed_reason, failed_at, completed_at,
|
||||
completed_metadata_json
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, NULL, ?, ?, NULL, NULL, NULL, ?, NULL, NULL, NULL, NULL, NULL, NULL)
|
||||
`);
|
||||
for (let i = 0; i < count; i += 1) {
|
||||
insert.run(
|
||||
"ingress",
|
||||
`event-${String(i).padStart(8, "0")}`,
|
||||
i % 2 === 0 ? "telegram" : "discord",
|
||||
`account-${i % 8}`,
|
||||
i % 11 === 0 ? "claimed" : "pending",
|
||||
`lane-${i % 128}`,
|
||||
JSON.stringify({ text: `message ${i}` }),
|
||||
1_700_000_000_000 + i,
|
||||
1_700_000_000_000 + i,
|
||||
i % 3,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function seedAgentDatabase(db: DatabaseSync, count: number, agentIndex: number): void {
|
||||
db.exec("BEGIN IMMEDIATE;");
|
||||
try {
|
||||
const insert = db.prepare(`
|
||||
INSERT INTO cache_entries (scope, key, value_json, blob, expires_at, updated_at)
|
||||
VALUES (?, ?, ?, NULL, ?, ?)
|
||||
`);
|
||||
for (let i = 0; i < count; i += 1) {
|
||||
insert.run(
|
||||
i % 4 === 0 ? "session_entries" : `scope-${i % 16}`,
|
||||
`agent-${agentIndex}-entry-${String(i).padStart(8, "0")}`,
|
||||
JSON.stringify({ agentIndex, i, value: `cache ${i}` }),
|
||||
i % 7 === 0 ? 1_800_000_000_000 + i : null,
|
||||
1_700_000_000_000 + i,
|
||||
);
|
||||
}
|
||||
db.exec("COMMIT;");
|
||||
} catch (err) {
|
||||
db.exec("ROLLBACK;");
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
function readIntegrity(db: DatabaseSync): string {
|
||||
const row = db.prepare("PRAGMA integrity_check").get() as { integrity_check?: unknown };
|
||||
return typeof row.integrity_check === "string" ? row.integrity_check : "missing";
|
||||
}
|
||||
|
||||
function checkpoint(db: DatabaseSync): void {
|
||||
db.prepare("PRAGMA wal_checkpoint(TRUNCATE)").all();
|
||||
}
|
||||
|
||||
function percentile(values: number[], pct: number): number {
|
||||
if (values.length === 0) {
|
||||
return 0;
|
||||
}
|
||||
const sorted = values.toSorted((left, right) => left - right);
|
||||
const index = Math.min(sorted.length - 1, Math.ceil((pct / 100) * sorted.length) - 1);
|
||||
return Number(sorted[index].toFixed(3));
|
||||
}
|
||||
|
||||
function runTimedQuery(
|
||||
db: DatabaseSync,
|
||||
query: string,
|
||||
params: unknown[],
|
||||
runs: number,
|
||||
): TimedQuery {
|
||||
const statement = db.prepare(query);
|
||||
const samples: number[] = [];
|
||||
let rows = 0;
|
||||
for (let i = 0; i < runs; i += 1) {
|
||||
const started = nowMs();
|
||||
rows = statement.all(...params).length;
|
||||
samples.push(nowMs() - started);
|
||||
}
|
||||
return {
|
||||
p50Ms: percentile(samples, 50),
|
||||
p95Ms: percentile(samples, 95),
|
||||
query,
|
||||
rows,
|
||||
};
|
||||
}
|
||||
|
||||
function runHotQueries(params: {
|
||||
agentDb: DatabaseSync;
|
||||
config: ProfileConfig;
|
||||
stateDb: DatabaseSync;
|
||||
}): TimedQuery[] {
|
||||
return [
|
||||
runTimedQuery(
|
||||
params.stateDb,
|
||||
`SELECT job_id, name, updated_at
|
||||
FROM cron_jobs
|
||||
WHERE store_key = ?
|
||||
ORDER BY sort_order ASC, updated_at ASC, job_id
|
||||
LIMIT 50`,
|
||||
["/state/cron/jobs-0.json"],
|
||||
params.config.queryRuns,
|
||||
),
|
||||
runTimedQuery(
|
||||
params.stateDb,
|
||||
`SELECT job_id, next_run_at_ms
|
||||
FROM cron_jobs
|
||||
WHERE store_key = ? AND enabled = 1 AND next_run_at_ms IS NOT NULL
|
||||
ORDER BY next_run_at_ms ASC, job_id
|
||||
LIMIT 50`,
|
||||
["/state/cron/jobs-0.json"],
|
||||
params.config.queryRuns,
|
||||
),
|
||||
runTimedQuery(
|
||||
params.stateDb,
|
||||
`SELECT id, entry_json
|
||||
FROM delivery_queue_entries
|
||||
WHERE queue_name = ? AND status = ?
|
||||
ORDER BY enqueued_at ASC, id
|
||||
LIMIT 100`,
|
||||
["outbound", "pending"],
|
||||
params.config.queryRuns,
|
||||
),
|
||||
runTimedQuery(
|
||||
params.stateDb,
|
||||
`SELECT entry_key, value_json
|
||||
FROM plugin_state_entries
|
||||
WHERE plugin_id = ? AND namespace = ?
|
||||
ORDER BY created_at ASC, entry_key
|
||||
LIMIT 100`,
|
||||
["plugin-0", "namespace-0"],
|
||||
params.config.queryRuns,
|
||||
),
|
||||
runTimedQuery(
|
||||
params.agentDb,
|
||||
`SELECT key, value_json
|
||||
FROM cache_entries
|
||||
WHERE scope = ?
|
||||
ORDER BY key ASC
|
||||
LIMIT 100`,
|
||||
["session_entries"],
|
||||
params.config.queryRuns,
|
||||
),
|
||||
runTimedQuery(
|
||||
params.agentDb,
|
||||
`SELECT key, expires_at
|
||||
FROM cache_entries
|
||||
WHERE scope = ? AND expires_at IS NOT NULL
|
||||
ORDER BY expires_at ASC, key
|
||||
LIMIT 100`,
|
||||
["session_entries"],
|
||||
params.config.queryRuns,
|
||||
),
|
||||
];
|
||||
}
|
||||
|
||||
function printProofLines(report: BenchmarkReport): void {
|
||||
const p95 = Math.max(...report.queries.map((query) => query.p95Ms));
|
||||
console.log(`SQLITE_PERF_PROFILE=${report.profile}`);
|
||||
console.log(`SQLITE_PERF_STATE_ROWS=${report.rows.stateRows}`);
|
||||
console.log(`SQLITE_PERF_AGENT_ROWS=${report.rows.agentCacheEntries}`);
|
||||
console.log(`SQLITE_PERF_INTEGRITY=${report.integrity.state}`);
|
||||
console.log(`SQLITE_PERF_WAL_BYTES_BEFORE=${report.walBytes.stateBefore}`);
|
||||
console.log(`SQLITE_PERF_WAL_BYTES_AFTER=${report.walBytes.stateAfter}`);
|
||||
console.log(`SQLITE_PERF_QUERY_P95_MS=${p95.toFixed(3)}`);
|
||||
if (report.paths.artifact) {
|
||||
console.log(`SQLITE_PERF_ARTIFACT=${report.paths.artifact}`);
|
||||
}
|
||||
}
|
||||
|
||||
function main(): void {
|
||||
const argv = process.argv.slice(2);
|
||||
validateArgs(argv);
|
||||
if (hasFlag("--help", argv)) {
|
||||
printUsage();
|
||||
return;
|
||||
}
|
||||
const options = parseOptions(argv);
|
||||
const config = applyScale(PROFILES[options.profile]);
|
||||
const stateDir =
|
||||
options.stateDir ?? fs.mkdtempSync(path.join(os.tmpdir(), "openclaw-sqlite-perf-"));
|
||||
const env = { OPENCLAW_STATE_DIR: stateDir };
|
||||
const started = nowMs();
|
||||
try {
|
||||
const stateDatabase = openOpenClawStateDatabase({ env });
|
||||
const agentDatabases = Array.from({ length: config.agentCount }, (_, index) =>
|
||||
openOpenClawAgentDatabase({ agentId: `perf-agent-${index}`, env }),
|
||||
);
|
||||
|
||||
const seedStarted = nowMs();
|
||||
seedStateDatabase(stateDatabase.db, config);
|
||||
const perAgentEntries = Math.ceil(config.agentCacheEntries / config.agentCount);
|
||||
agentDatabases.forEach((database, index) =>
|
||||
seedAgentDatabase(database.db, perAgentEntries, index),
|
||||
);
|
||||
const seedMs = nowMs() - seedStarted;
|
||||
|
||||
const stateWalBefore = walSize(stateDatabase.path);
|
||||
const agentWalBefore = agentDatabases.map((database) => walSize(database.path));
|
||||
const stateIntegrity = readIntegrity(stateDatabase.db);
|
||||
const agentIntegrity = agentDatabases.map((database) => readIntegrity(database.db));
|
||||
const queries = runHotQueries({
|
||||
agentDb: agentDatabases[0]?.db ?? stateDatabase.db,
|
||||
config,
|
||||
stateDb: stateDatabase.db,
|
||||
});
|
||||
|
||||
const checkpointStarted = nowMs();
|
||||
checkpoint(stateDatabase.db);
|
||||
agentDatabases.forEach((database) => checkpoint(database.db));
|
||||
const checkpointMs = nowMs() - checkpointStarted;
|
||||
|
||||
const report: BenchmarkReport = {
|
||||
integrity: {
|
||||
agent: agentIntegrity,
|
||||
state: stateIntegrity,
|
||||
},
|
||||
node: process.version,
|
||||
paths: {
|
||||
agentDatabases: agentDatabases.map((database) => database.path),
|
||||
artifact: options.output,
|
||||
stateDatabase: stateDatabase.path,
|
||||
stateDir,
|
||||
},
|
||||
profile: options.profile,
|
||||
queries,
|
||||
rows: {
|
||||
agentCacheEntries: perAgentEntries * config.agentCount,
|
||||
agentDatabases: config.agentCount,
|
||||
channelIngressEvents: config.channelIngressEvents,
|
||||
cronJobs: config.cronJobs,
|
||||
cronRunLogs: config.cronRunLogs,
|
||||
deliveryQueueEntries: config.deliveryQueueEntries,
|
||||
pluginStateEntries: config.pluginStateEntries,
|
||||
stateRows: stateRowCount(config),
|
||||
},
|
||||
timingsMs: {
|
||||
checkpoint: Number(checkpointMs.toFixed(3)),
|
||||
seed: Number(seedMs.toFixed(3)),
|
||||
total: Number((nowMs() - started).toFixed(3)),
|
||||
},
|
||||
walBytes: {
|
||||
agentAfter: agentDatabases.map((database) => walSize(database.path)),
|
||||
agentBefore: agentWalBefore,
|
||||
stateAfter: walSize(stateDatabase.path),
|
||||
stateBefore: stateWalBefore,
|
||||
},
|
||||
};
|
||||
|
||||
if (options.output) {
|
||||
fs.mkdirSync(path.dirname(options.output), { recursive: true });
|
||||
fs.writeFileSync(options.output, `${JSON.stringify(report, null, 2)}\n`, "utf8");
|
||||
}
|
||||
printProofLines(report);
|
||||
} finally {
|
||||
closeOpenClawAgentDatabasesForTest();
|
||||
closeOpenClawStateDatabaseForTest();
|
||||
if (!options.stateDir) {
|
||||
fs.rmSync(stateDir, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (import.meta.url === pathToFileURL(process.argv[1] ?? "").href) {
|
||||
try {
|
||||
main();
|
||||
} catch (error) {
|
||||
if (error instanceof CliUsageError) {
|
||||
console.error(`error: ${error.message}`);
|
||||
process.exit(2);
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
271
scripts/bench-test-changed.mjs
Normal file
271
scripts/bench-test-changed.mjs
Normal file
@@ -0,0 +1,271 @@
|
||||
// Benchmarks `pnpm test:changed` planning/runtime behavior across repeated runs.
|
||||
import { spawnSync } from "node:child_process";
|
||||
import path from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { parseFlagArgs, stringFlag } from "./lib/arg-utils.mjs";
|
||||
import { formatMs } from "./lib/vitest-report-cli-utils.mjs";
|
||||
|
||||
function parsePositiveInteger(raw, label) {
|
||||
const value = raw?.trim();
|
||||
if (!value) {
|
||||
throw new Error(`${label} requires a value`);
|
||||
}
|
||||
if (!/^\d+$/u.test(value)) {
|
||||
throw new Error(`${label} must be a positive integer`);
|
||||
}
|
||||
const parsed = Number(value);
|
||||
if (!Number.isSafeInteger(parsed) || parsed < 1) {
|
||||
throw new Error(`${label} must be a safe positive integer`);
|
||||
}
|
||||
return parsed;
|
||||
}
|
||||
|
||||
function positiveIntegerFlag(flag, key) {
|
||||
return {
|
||||
consume(argv, index) {
|
||||
if (argv[index] !== flag) {
|
||||
return null;
|
||||
}
|
||||
const rawValue = argv[index + 1];
|
||||
if (!rawValue || rawValue.startsWith("--")) {
|
||||
throw new Error(`${flag} requires a value`);
|
||||
}
|
||||
return {
|
||||
flag,
|
||||
nextIndex: index + 1,
|
||||
repeatable: false,
|
||||
apply(target) {
|
||||
target[key] = parsePositiveInteger(rawValue, flag);
|
||||
},
|
||||
};
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export function parseArgs(argv) {
|
||||
const args = parseFlagArgs(
|
||||
argv,
|
||||
{
|
||||
cwd: process.cwd(),
|
||||
ref: "origin/main",
|
||||
rss: process.platform === "darwin",
|
||||
mode: "ref",
|
||||
},
|
||||
[
|
||||
stringFlag("--cwd", "cwd"),
|
||||
stringFlag("--ref", "ref"),
|
||||
positiveIntegerFlag("--max-workers", "maxWorkers"),
|
||||
],
|
||||
{
|
||||
onUnhandledArg(arg, target) {
|
||||
if (arg === "--no-rss") {
|
||||
target.rss = false;
|
||||
return "handled";
|
||||
}
|
||||
if (arg === "--worktree") {
|
||||
target.mode = "worktree";
|
||||
return "handled";
|
||||
}
|
||||
return undefined;
|
||||
},
|
||||
},
|
||||
);
|
||||
return {
|
||||
cwd: path.resolve(args.cwd),
|
||||
mode: args.mode,
|
||||
ref: args.ref,
|
||||
rss: args.rss,
|
||||
...(typeof args.maxWorkers === "number" ? { maxWorkers: args.maxWorkers } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
function quoteArg(arg) {
|
||||
return /[^A-Za-z0-9_./:-]/.test(arg) ? JSON.stringify(arg) : arg;
|
||||
}
|
||||
|
||||
function runGitList(args, cwd) {
|
||||
const result = spawnSync("git", args, {
|
||||
cwd,
|
||||
encoding: "utf8",
|
||||
});
|
||||
if (result.status !== 0) {
|
||||
throw new Error(result.stderr || result.stdout || `git ${args.join(" ")} failed`);
|
||||
}
|
||||
return result.stdout
|
||||
.split("\n")
|
||||
.map((line) => line.trim())
|
||||
.filter((line) => line.length > 0);
|
||||
}
|
||||
|
||||
function listChangedPaths(opts) {
|
||||
if (opts.mode === "worktree") {
|
||||
return [
|
||||
...new Set([
|
||||
...runGitList(["diff", "--name-only", "--relative", "HEAD", "--"], opts.cwd),
|
||||
...runGitList(["ls-files", "--others", "--exclude-standard"], opts.cwd),
|
||||
]),
|
||||
].toSorted((left, right) => left.localeCompare(right));
|
||||
}
|
||||
return runGitList(["diff", "--name-only", `${opts.ref}...HEAD`], opts.cwd);
|
||||
}
|
||||
|
||||
export function parseMaxRssBytes(output) {
|
||||
const match = output.match(
|
||||
/(?:^|\n)[^\S\r\n]*(\d+)[^\S\r\n]+maximum resident set size[^\S\r\n]*(?:\r?\n|$)/u,
|
||||
);
|
||||
if (!match) {
|
||||
return null;
|
||||
}
|
||||
const parsed = Number(match[1]);
|
||||
return Number.isSafeInteger(parsed) && parsed >= 0 ? parsed : null;
|
||||
}
|
||||
|
||||
export function formatRss(valueBytes) {
|
||||
if (valueBytes === null) {
|
||||
return "n/a";
|
||||
}
|
||||
return `${(valueBytes / 1024 / 1024).toFixed(1)}MB`;
|
||||
}
|
||||
|
||||
export function resolveBenchRssResult({ label, output, rss, status }) {
|
||||
if (!rss) {
|
||||
return { maxRssBytes: null, output, status };
|
||||
}
|
||||
const maxRssBytes = parseMaxRssBytes(output);
|
||||
if (status === 0 && maxRssBytes === null) {
|
||||
return {
|
||||
maxRssBytes,
|
||||
output: `${output}${output.endsWith("\n") ? "" : "\n"}[bench-test-changed] ${label} missing maximum resident set size from /usr/bin/time -l output\n`,
|
||||
status: 1,
|
||||
};
|
||||
}
|
||||
return { maxRssBytes, output, status };
|
||||
}
|
||||
|
||||
function runBenchCommand(params) {
|
||||
const env = { ...process.env };
|
||||
if (typeof params.maxWorkers === "number") {
|
||||
env.OPENCLAW_VITEST_MAX_WORKERS = String(params.maxWorkers);
|
||||
}
|
||||
const startedAt = process.hrtime.bigint();
|
||||
const commandArgs = params.rss ? ["-l", ...params.command] : params.command;
|
||||
const result = spawnSync(
|
||||
params.rss ? "/usr/bin/time" : commandArgs[0],
|
||||
params.rss ? commandArgs : commandArgs.slice(1),
|
||||
{
|
||||
cwd: params.cwd,
|
||||
env,
|
||||
encoding: "utf8",
|
||||
maxBuffer: 1024 * 1024 * 32,
|
||||
},
|
||||
);
|
||||
const elapsedMs = Number(process.hrtime.bigint() - startedAt) / 1_000_000;
|
||||
const output = `${result.stdout ?? ""}${result.stderr ?? ""}`;
|
||||
const normalized = resolveBenchRssResult({
|
||||
label: params.label,
|
||||
output,
|
||||
rss: params.rss,
|
||||
status: result.status ?? 1,
|
||||
});
|
||||
return {
|
||||
elapsedMs,
|
||||
maxRssBytes: normalized.maxRssBytes,
|
||||
status: normalized.status,
|
||||
output: normalized.output,
|
||||
};
|
||||
}
|
||||
|
||||
function printRunSummary(label, result) {
|
||||
console.log(
|
||||
`${label.padEnd(8, " ")} wall=${formatMs(result.elapsedMs).padStart(9, " ")} rss=${formatRss(
|
||||
result.maxRssBytes,
|
||||
).padStart(9, " ")}`,
|
||||
);
|
||||
}
|
||||
|
||||
function main() {
|
||||
let opts;
|
||||
try {
|
||||
opts = parseArgs(process.argv.slice(2));
|
||||
} catch (error) {
|
||||
console.error(error instanceof Error ? error.message : String(error));
|
||||
process.exit(1);
|
||||
}
|
||||
const changedPaths = listChangedPaths(opts);
|
||||
if (changedPaths.length === 0) {
|
||||
console.log(
|
||||
opts.mode === "worktree"
|
||||
? "[bench-test-changed] no changed paths in worktree"
|
||||
: `[bench-test-changed] no changed paths for ${opts.ref}...HEAD`,
|
||||
);
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
console.log(
|
||||
opts.mode === "worktree"
|
||||
? "[bench-test-changed] mode=worktree"
|
||||
: `[bench-test-changed] ref=${opts.ref}`,
|
||||
);
|
||||
console.log("[bench-test-changed] changed paths:");
|
||||
for (const changedPath of changedPaths) {
|
||||
console.log(`- ${changedPath}`);
|
||||
}
|
||||
|
||||
const routedCommand =
|
||||
opts.mode === "worktree"
|
||||
? [process.execPath, "scripts/test-projects.mjs", ...changedPaths]
|
||||
: [process.execPath, "scripts/test-projects.mjs", "--changed", opts.ref];
|
||||
const rootCommand = [
|
||||
process.execPath,
|
||||
"scripts/run-vitest.mjs",
|
||||
"run",
|
||||
"--config",
|
||||
"vitest.config.ts",
|
||||
...changedPaths,
|
||||
];
|
||||
|
||||
console.log(`[bench-test-changed] routed: ${routedCommand.map(quoteArg).join(" ")}`);
|
||||
const routed = runBenchCommand({
|
||||
command: routedCommand,
|
||||
cwd: opts.cwd,
|
||||
label: "routed",
|
||||
rss: opts.rss,
|
||||
...(typeof opts.maxWorkers === "number" ? { maxWorkers: opts.maxWorkers } : {}),
|
||||
});
|
||||
if (routed.status !== 0) {
|
||||
process.stderr.write(routed.output);
|
||||
process.exit(routed.status);
|
||||
}
|
||||
|
||||
console.log(`[bench-test-changed] root: ${rootCommand.map(quoteArg).join(" ")}`);
|
||||
const root = runBenchCommand({
|
||||
command: rootCommand,
|
||||
cwd: opts.cwd,
|
||||
label: "root",
|
||||
rss: opts.rss,
|
||||
...(typeof opts.maxWorkers === "number" ? { maxWorkers: opts.maxWorkers } : {}),
|
||||
});
|
||||
if (root.status !== 0) {
|
||||
process.stderr.write(root.output);
|
||||
process.exit(root.status);
|
||||
}
|
||||
|
||||
printRunSummary("routed", routed);
|
||||
printRunSummary("root", root);
|
||||
console.log(
|
||||
`[bench-test-changed] delta wall=${formatMs(root.elapsedMs - routed.elapsedMs)} rss=${
|
||||
routed.maxRssBytes !== null && root.maxRssBytes !== null
|
||||
? formatRss(root.maxRssBytes - routed.maxRssBytes)
|
||||
: "n/a"
|
||||
}`,
|
||||
);
|
||||
}
|
||||
|
||||
const isMain =
|
||||
typeof process.argv[1] === "string" &&
|
||||
process.argv[1].length > 0 &&
|
||||
import.meta.url === pathToFileURL(path.resolve(process.argv[1])).href;
|
||||
|
||||
if (isMain) {
|
||||
main();
|
||||
}
|
||||
476
scripts/bench-web-fetch.ts
Normal file
476
scripts/bench-web-fetch.ts
Normal file
@@ -0,0 +1,476 @@
|
||||
// Web fetch benchmark covers direct response loading, HTML extraction, and fallback cleanup.
|
||||
import { mkdir, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { performance } from "node:perf_hooks";
|
||||
import { extractBasicHtmlContent } from "../src/agents/tools/web-fetch-utils.js";
|
||||
import { createWebFetchTool } from "../src/agents/tools/web-fetch.js";
|
||||
import type { OpenClawConfig } from "../src/config/types.openclaw.js";
|
||||
import type { LookupFn } from "../src/infra/net/ssrf.js";
|
||||
import { extractReadableContent } from "../src/web-fetch/content-extractors.runtime.js";
|
||||
import { stripLeadingPackageManagerSeparator } from "./lib/arg-utils.mjs";
|
||||
|
||||
type BenchmarkCaseId =
|
||||
| "tool-create"
|
||||
| "tool-text"
|
||||
| "tool-markdown"
|
||||
| "tool-html-article"
|
||||
| "tool-html-article-text"
|
||||
| "tool-html-shell"
|
||||
| "extract-readable-article"
|
||||
| "extract-readable-article-text"
|
||||
| "extract-basic-shell";
|
||||
|
||||
type BenchmarkCase = {
|
||||
id: BenchmarkCaseId;
|
||||
label: string;
|
||||
run: () => Promise<void> | void;
|
||||
};
|
||||
|
||||
type Options = {
|
||||
cases: BenchmarkCaseId[];
|
||||
json: boolean;
|
||||
output?: string;
|
||||
runs: number;
|
||||
warmup: number;
|
||||
};
|
||||
|
||||
type SummaryStats = {
|
||||
avg: number;
|
||||
max: number;
|
||||
min: number;
|
||||
p50: number;
|
||||
p95: number;
|
||||
};
|
||||
|
||||
type CaseReport = {
|
||||
id: BenchmarkCaseId;
|
||||
label: string;
|
||||
samplesMs: number[];
|
||||
summaryMs: SummaryStats;
|
||||
};
|
||||
|
||||
type BenchmarkReport = {
|
||||
cases: CaseReport[];
|
||||
node: string;
|
||||
options: Omit<Options, "json">;
|
||||
rssMb: number;
|
||||
};
|
||||
|
||||
const ALL_CASE_IDS = [
|
||||
"tool-create",
|
||||
"tool-text",
|
||||
"tool-markdown",
|
||||
"tool-html-article",
|
||||
"tool-html-article-text",
|
||||
"tool-html-shell",
|
||||
"extract-readable-article",
|
||||
"extract-readable-article-text",
|
||||
"extract-basic-shell",
|
||||
] as const satisfies readonly BenchmarkCaseId[];
|
||||
|
||||
const BOOLEAN_FLAGS = new Set(["--help", "-h", "--json"]);
|
||||
const VALUE_FLAGS = new Set(["--case", "--output", "--runs", "--warmup"]);
|
||||
|
||||
class CliArgumentError extends Error {
|
||||
override name = "CliArgumentError";
|
||||
}
|
||||
|
||||
const ARTICLE_HTML = `<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<title>Web Fetch Benchmark Article</title>
|
||||
</head>
|
||||
<body>
|
||||
<nav>${Array.from({ length: 24 }, (_, index) => `<a href="/nav-${index}">Nav ${index}</a>`).join("")}</nav>
|
||||
<main>
|
||||
<article>
|
||||
<h1>Web Fetch Benchmark Article</h1>
|
||||
${Array.from(
|
||||
{ length: 180 },
|
||||
(_, index) =>
|
||||
`<p>Paragraph ${index} carries readable benchmark content with enough prose for Readability to score it as article body text.</p>`,
|
||||
).join("\n")}
|
||||
</article>
|
||||
</main>
|
||||
<footer>Repeated footer chrome that should not dominate extracted content.</footer>
|
||||
</body>
|
||||
</html>`;
|
||||
|
||||
const SHELL_HTML = `<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<title>Shell App</title>
|
||||
</head>
|
||||
<body>
|
||||
<div id="app"></div>
|
||||
<script>window.__APP__ = true;</script>
|
||||
<noscript>Enable JavaScript to load this page.</noscript>
|
||||
</body>
|
||||
</html>`;
|
||||
|
||||
const TEXT_BODY = "OpenClaw web_fetch direct text benchmark body.".repeat(160);
|
||||
const MARKDOWN_BODY = "# Web Fetch Benchmark\n\n" + "- markdown list item\n".repeat(220);
|
||||
const OFFLINE_PROVIDER_ENV_VARS = ["FIRECRAWL_API_KEY"] as const;
|
||||
|
||||
const lookupFn: LookupFn = async () => [{ address: "93.184.216.34", family: 4 }];
|
||||
const toolConfig: OpenClawConfig = {
|
||||
tools: {
|
||||
web: {
|
||||
fetch: {
|
||||
cacheTtlMinutes: 0,
|
||||
firecrawl: { enabled: false },
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
function validateArgs(args: string[]): void {
|
||||
const seenSingularValueFlags = new Set<string>();
|
||||
for (let index = 0; index < args.length; index += 1) {
|
||||
const arg = args[index] ?? "";
|
||||
if (BOOLEAN_FLAGS.has(arg)) {
|
||||
continue;
|
||||
}
|
||||
if (VALUE_FLAGS.has(arg)) {
|
||||
if (arg !== "--case") {
|
||||
if (seenSingularValueFlags.has(arg)) {
|
||||
throw new CliArgumentError(`${arg} was provided more than once`);
|
||||
}
|
||||
seenSingularValueFlags.add(arg);
|
||||
}
|
||||
const value = args[index + 1];
|
||||
if (!value || value.startsWith("-")) {
|
||||
throw new CliArgumentError(`${arg} requires a value`);
|
||||
}
|
||||
index += 1;
|
||||
continue;
|
||||
}
|
||||
throw new CliArgumentError(`Unknown argument: ${arg}`);
|
||||
}
|
||||
}
|
||||
|
||||
function parsePositiveInteger(flag: string, fallback: number, args: string[]): number {
|
||||
const index = args.indexOf(flag);
|
||||
if (index === -1) {
|
||||
return fallback;
|
||||
}
|
||||
const raw = args[index + 1];
|
||||
const value = Number(raw);
|
||||
if (!Number.isInteger(value) || value <= 0) {
|
||||
throw new CliArgumentError(`${flag} must be a positive integer`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function parseNonNegativeInteger(flag: string, fallback: number, args: string[]): number {
|
||||
const index = args.indexOf(flag);
|
||||
if (index === -1) {
|
||||
return fallback;
|
||||
}
|
||||
const raw = args[index + 1];
|
||||
const value = Number(raw);
|
||||
if (!Number.isInteger(value) || value < 0) {
|
||||
throw new CliArgumentError(`${flag} must be a non-negative integer`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function parseCases(args: string[]): BenchmarkCaseId[] {
|
||||
const requested: string[] = [];
|
||||
for (let index = 0; index < args.length; index += 1) {
|
||||
if (args[index] === "--case") {
|
||||
requested.push(args[index + 1] ?? "");
|
||||
index += 1;
|
||||
}
|
||||
}
|
||||
if (requested.length === 0 || requested.includes("all")) {
|
||||
return [...ALL_CASE_IDS];
|
||||
}
|
||||
const valid = new Set<string>(ALL_CASE_IDS);
|
||||
for (const id of requested) {
|
||||
if (!valid.has(id)) {
|
||||
throw new CliArgumentError(
|
||||
`--case must be one of all, ${ALL_CASE_IDS.join(", ")}; got ${JSON.stringify(id)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
return requested as BenchmarkCaseId[];
|
||||
}
|
||||
|
||||
function parseFlagValue(flag: string, args: string[]): string | undefined {
|
||||
const index = args.indexOf(flag);
|
||||
return index === -1 ? undefined : args[index + 1];
|
||||
}
|
||||
|
||||
function parseOptions(args = process.argv.slice(2)): Options {
|
||||
const normalizedArgs = stripLeadingPackageManagerSeparator(args);
|
||||
validateArgs(normalizedArgs);
|
||||
return {
|
||||
cases: parseCases(normalizedArgs),
|
||||
json: normalizedArgs.includes("--json"),
|
||||
output: parseFlagValue("--output", normalizedArgs),
|
||||
runs: parsePositiveInteger("--runs", 20, normalizedArgs),
|
||||
warmup: parseNonNegativeInteger("--warmup", 3, normalizedArgs),
|
||||
};
|
||||
}
|
||||
|
||||
function printUsage(): void {
|
||||
process.stdout.write(`OpenClaw web_fetch benchmark
|
||||
|
||||
Usage:
|
||||
pnpm perf:web-fetch -- [options]
|
||||
node --import tsx scripts/bench-web-fetch.ts [options]
|
||||
|
||||
Options:
|
||||
--case <id> Case to run; repeatable. Use "all" for every case.
|
||||
--runs <n> Measured runs per case (default: 20)
|
||||
--warmup <n> Warmup runs per case (default: 3)
|
||||
--output <path> Write JSON report
|
||||
--json Print JSON report
|
||||
--help, -h Show this text
|
||||
|
||||
Cases:
|
||||
${ALL_CASE_IDS.join("\n ")}
|
||||
`);
|
||||
}
|
||||
|
||||
function round(value: number): number {
|
||||
return Math.round(value * 1000) / 1000;
|
||||
}
|
||||
|
||||
function percentile(values: number[], p: number): number {
|
||||
if (values.length === 0) {
|
||||
return 0;
|
||||
}
|
||||
const sorted = values.toSorted((left, right) => left - right);
|
||||
const index = Math.min(sorted.length - 1, Math.floor((sorted.length - 1) * p));
|
||||
return round(sorted[index] ?? 0);
|
||||
}
|
||||
|
||||
function stats(values: number[]): SummaryStats {
|
||||
if (values.length === 0) {
|
||||
return { avg: 0, max: 0, min: 0, p50: 0, p95: 0 };
|
||||
}
|
||||
const total = values.reduce((sum, value) => sum + value, 0);
|
||||
return {
|
||||
avg: round(total / values.length),
|
||||
max: round(Math.max(...values)),
|
||||
min: round(Math.min(...values)),
|
||||
p50: percentile(values, 0.5),
|
||||
p95: percentile(values, 0.95),
|
||||
};
|
||||
}
|
||||
|
||||
function installMockFetch(params: { body: string; contentType: string }) {
|
||||
const fetchImpl = (async () =>
|
||||
new Response(params.body, {
|
||||
status: 200,
|
||||
headers: {
|
||||
"content-type": params.contentType,
|
||||
},
|
||||
})) as typeof globalThis.fetch & { mock: object };
|
||||
// fetchWithSsrFGuard preserves dispatcher support unless global fetch is a
|
||||
// test double. The marker keeps this benchmark offline and deterministic.
|
||||
fetchImpl.mock = {};
|
||||
globalThis.fetch = fetchImpl;
|
||||
}
|
||||
|
||||
async function withOfflineProviderEnv<T>(run: () => Promise<T>): Promise<T> {
|
||||
const previous = new Map<string, string | undefined>();
|
||||
for (const name of OFFLINE_PROVIDER_ENV_VARS) {
|
||||
previous.set(name, process.env[name]);
|
||||
process.env[name] = "";
|
||||
}
|
||||
try {
|
||||
return await run();
|
||||
} finally {
|
||||
for (const [name, value] of previous) {
|
||||
if (value === undefined) {
|
||||
delete process.env[name];
|
||||
} else {
|
||||
process.env[name] = value;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function createTool() {
|
||||
const tool = createWebFetchTool({
|
||||
config: toolConfig,
|
||||
lookupFn,
|
||||
sandboxed: false,
|
||||
});
|
||||
if (!tool?.execute) {
|
||||
throw new Error("web_fetch tool was not created");
|
||||
}
|
||||
return tool;
|
||||
}
|
||||
|
||||
function createCases(): Record<BenchmarkCaseId, BenchmarkCase> {
|
||||
const textTool = createTool();
|
||||
const markdownTool = createTool();
|
||||
const articleTool = createTool();
|
||||
const articleTextTool = createTool();
|
||||
const shellTool = createTool();
|
||||
return {
|
||||
"tool-create": {
|
||||
id: "tool-create",
|
||||
label: "create web_fetch tool",
|
||||
run: () => {
|
||||
createTool();
|
||||
},
|
||||
},
|
||||
"tool-text": {
|
||||
id: "tool-text",
|
||||
label: "execute text/plain fetch",
|
||||
run: async () => {
|
||||
installMockFetch({ body: TEXT_BODY, contentType: "text/plain; charset=utf-8" });
|
||||
await textTool.execute("bench", { url: "https://example.com/plain" });
|
||||
},
|
||||
},
|
||||
"tool-markdown": {
|
||||
id: "tool-markdown",
|
||||
label: "execute text/markdown fetch",
|
||||
run: async () => {
|
||||
installMockFetch({ body: MARKDOWN_BODY, contentType: "text/markdown; charset=utf-8" });
|
||||
await markdownTool.execute("bench", { url: "https://example.com/markdown" });
|
||||
},
|
||||
},
|
||||
"tool-html-article": {
|
||||
id: "tool-html-article",
|
||||
label: "execute article HTML fetch",
|
||||
run: async () => {
|
||||
installMockFetch({ body: ARTICLE_HTML, contentType: "text/html; charset=utf-8" });
|
||||
await articleTool.execute("bench", { url: "https://example.com/article" });
|
||||
},
|
||||
},
|
||||
"tool-html-article-text": {
|
||||
id: "tool-html-article-text",
|
||||
label: "execute article HTML fetch as text",
|
||||
run: async () => {
|
||||
installMockFetch({ body: ARTICLE_HTML, contentType: "text/html; charset=utf-8" });
|
||||
await articleTextTool.execute("bench", {
|
||||
url: "https://example.com/article-text",
|
||||
extractMode: "text",
|
||||
});
|
||||
},
|
||||
},
|
||||
"tool-html-shell": {
|
||||
id: "tool-html-shell",
|
||||
label: "execute shell HTML fallback fetch",
|
||||
run: async () => {
|
||||
installMockFetch({ body: SHELL_HTML, contentType: "text/html; charset=utf-8" });
|
||||
await shellTool.execute("bench", { url: "https://example.com/shell" });
|
||||
},
|
||||
},
|
||||
"extract-readable-article": {
|
||||
id: "extract-readable-article",
|
||||
label: "extract readable article HTML",
|
||||
run: async () => {
|
||||
await extractReadableContent({
|
||||
html: ARTICLE_HTML,
|
||||
url: "https://example.com/article",
|
||||
extractMode: "markdown",
|
||||
config: toolConfig,
|
||||
});
|
||||
},
|
||||
},
|
||||
"extract-readable-article-text": {
|
||||
id: "extract-readable-article-text",
|
||||
label: "extract readable article HTML as text",
|
||||
run: async () => {
|
||||
await extractReadableContent({
|
||||
html: ARTICLE_HTML,
|
||||
url: "https://example.com/article-text",
|
||||
extractMode: "text",
|
||||
config: toolConfig,
|
||||
});
|
||||
},
|
||||
},
|
||||
"extract-basic-shell": {
|
||||
id: "extract-basic-shell",
|
||||
label: "extract basic shell HTML",
|
||||
run: async () => {
|
||||
await extractBasicHtmlContent({
|
||||
html: SHELL_HTML,
|
||||
extractMode: "markdown",
|
||||
});
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function measureCase(testCase: BenchmarkCase, options: Options): Promise<CaseReport> {
|
||||
for (let index = 0; index < options.warmup; index += 1) {
|
||||
await testCase.run();
|
||||
}
|
||||
const samplesMs: number[] = [];
|
||||
for (let index = 0; index < options.runs; index += 1) {
|
||||
const started = performance.now();
|
||||
await testCase.run();
|
||||
samplesMs.push(round(performance.now() - started));
|
||||
}
|
||||
return {
|
||||
id: testCase.id,
|
||||
label: testCase.label,
|
||||
samplesMs,
|
||||
summaryMs: stats(samplesMs),
|
||||
};
|
||||
}
|
||||
|
||||
function printProofLines(report: BenchmarkReport): void {
|
||||
for (const testCase of report.cases) {
|
||||
const summary = testCase.summaryMs;
|
||||
console.log(
|
||||
`WEB_FETCH_BENCH_CASE=${testCase.id} avg_ms=${summary.avg.toFixed(3)} p50_ms=${summary.p50.toFixed(3)} p95_ms=${summary.p95.toFixed(3)} max_ms=${summary.max.toFixed(3)}`,
|
||||
);
|
||||
}
|
||||
console.log(`WEB_FETCH_BENCH_RSS_MB=${report.rssMb.toFixed(1)}`);
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const args = process.argv.slice(2);
|
||||
if (args.includes("--help") || args.includes("-h")) {
|
||||
printUsage();
|
||||
return;
|
||||
}
|
||||
const options = parseOptions(args);
|
||||
const report = await withOfflineProviderEnv(async () => {
|
||||
const casesById = createCases();
|
||||
const cases: CaseReport[] = [];
|
||||
for (const caseId of options.cases) {
|
||||
cases.push(await measureCase(casesById[caseId], options));
|
||||
}
|
||||
return {
|
||||
cases,
|
||||
node: process.version,
|
||||
options: {
|
||||
cases: options.cases,
|
||||
output: options.output,
|
||||
runs: options.runs,
|
||||
warmup: options.warmup,
|
||||
},
|
||||
rssMb: Math.round((process.memoryUsage().rss / 1024 / 1024) * 10) / 10,
|
||||
};
|
||||
});
|
||||
if (options.output) {
|
||||
await mkdir(path.dirname(options.output), { recursive: true });
|
||||
await writeFile(options.output, `${JSON.stringify(report, null, 2)}\n`);
|
||||
}
|
||||
if (options.json) {
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
} else {
|
||||
printProofLines(report);
|
||||
}
|
||||
}
|
||||
|
||||
main().catch((error: unknown) => {
|
||||
if (error instanceof CliArgumentError) {
|
||||
console.error(error.message);
|
||||
process.exitCode = 2;
|
||||
return;
|
||||
}
|
||||
throw error;
|
||||
});
|
||||
629
scripts/build-all.mjs
Normal file
629
scripts/build-all.mjs
Normal file
@@ -0,0 +1,629 @@
|
||||
#!/usr/bin/env node
|
||||
// Builds OpenClaw packages and plugin SDK artifacts with cache-aware orchestration.
|
||||
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { performance } from "node:perf_hooks";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { pluginSdkEntrypoints } from "./lib/plugin-sdk-entries.mjs";
|
||||
import { resolvePnpmRunner } from "./pnpm-runner.mjs";
|
||||
|
||||
const nodeBin = process.execPath;
|
||||
const BUILD_CACHE_VERSION = 3;
|
||||
const PLUGIN_SDK_ENTRY_DTS_CACHE_ENV = [
|
||||
"OPENCLAW_BUILD_PRIVATE_QA",
|
||||
"OPENCLAW_PLUGIN_SDK_CANONICAL_DTS",
|
||||
];
|
||||
const PLUGIN_SDK_ENTRY_DTS_CACHE_INPUTS = [
|
||||
"scripts/write-plugin-sdk-entry-dts.ts",
|
||||
"scripts/lib/plugin-sdk-entries.mjs",
|
||||
"scripts/lib/plugin-sdk-entrypoints.json",
|
||||
"scripts/lib/plugin-sdk-private-local-only-subpaths.json",
|
||||
"scripts/lib/plugin-sdk-deprecated-public-subpaths.json",
|
||||
"scripts/lib/plugin-sdk-deprecated-barrel-subpaths.json",
|
||||
{ path: "dist/plugin-sdk", extensions: [".d.ts"], recursive: false },
|
||||
];
|
||||
const PLUGIN_SDK_ENTRY_DTS_CACHE_OUTPUTS = [
|
||||
"dist/plugin-sdk/webhook-path.js",
|
||||
"dist/plugin-sdk/.boundary-entry-shims.stamp",
|
||||
...pluginSdkEntrypoints.map((entry) => `packages/plugin-sdk/dist/src/plugin-sdk/${entry}.d.ts`),
|
||||
];
|
||||
const PNPM_STEP_NODE_FALLBACKS = new Map([
|
||||
["plugins:assets:build", ["scripts/bundled-plugin-assets.mjs", "--phase", "build"]],
|
||||
["plugins:assets:copy", ["scripts/bundled-plugin-assets.mjs", "--phase", "copy"]],
|
||||
["ui:build", ["scripts/ui.js", "build"]],
|
||||
]);
|
||||
export const BUILD_ALL_STEPS = [
|
||||
{ label: "plugins:assets:build", kind: "pnpm", pnpmArgs: ["plugins:assets:build"] },
|
||||
{ label: "tsdown", kind: "node", args: ["scripts/tsdown-build.mjs"] },
|
||||
{
|
||||
label: "check-cli-bootstrap-imports",
|
||||
kind: "node",
|
||||
args: ["scripts/check-cli-bootstrap-imports.mjs"],
|
||||
},
|
||||
{ label: "runtime-postbuild", kind: "node", args: ["scripts/runtime-postbuild.mjs"] },
|
||||
{ label: "build-stamp", kind: "node", args: ["scripts/build-stamp.mjs"] },
|
||||
{
|
||||
label: "runtime-postbuild-stamp",
|
||||
kind: "node",
|
||||
args: ["scripts/runtime-postbuild-stamp.mjs"],
|
||||
},
|
||||
{
|
||||
label: "write-plugin-sdk-entry-dts",
|
||||
kind: "node",
|
||||
args: ["--experimental-strip-types", "scripts/write-plugin-sdk-entry-dts.ts"],
|
||||
env: {
|
||||
OPENCLAW_PLUGIN_SDK_CANONICAL_DTS: "1",
|
||||
},
|
||||
cache: {
|
||||
env: PLUGIN_SDK_ENTRY_DTS_CACHE_ENV,
|
||||
inputs: PLUGIN_SDK_ENTRY_DTS_CACHE_INPUTS,
|
||||
outputs: PLUGIN_SDK_ENTRY_DTS_CACHE_OUTPUTS,
|
||||
restore: "always",
|
||||
},
|
||||
},
|
||||
{
|
||||
label: "check-plugin-sdk-exports",
|
||||
kind: "node",
|
||||
args: ["scripts/check-plugin-sdk-exports.mjs"],
|
||||
},
|
||||
{
|
||||
label: "plugins:assets:copy",
|
||||
kind: "pnpm",
|
||||
pnpmArgs: ["plugins:assets:copy"],
|
||||
},
|
||||
{
|
||||
label: "copy-hook-metadata",
|
||||
kind: "node",
|
||||
args: ["--experimental-strip-types", "scripts/copy-hook-metadata.ts"],
|
||||
},
|
||||
{
|
||||
label: "copy-export-html-templates",
|
||||
kind: "node",
|
||||
args: ["--experimental-strip-types", "scripts/copy-export-html-templates.ts"],
|
||||
cache: {
|
||||
inputs: [
|
||||
"scripts/copy-export-html-templates.ts",
|
||||
"scripts/lib/copy-assets.ts",
|
||||
"src/auto-reply/reply/export-html",
|
||||
],
|
||||
outputs: ["dist/export-html"],
|
||||
},
|
||||
},
|
||||
{
|
||||
label: "ui:build",
|
||||
kind: "pnpm",
|
||||
pnpmArgs: ["ui:build"],
|
||||
// No build-all cache: ui/vite.config.ts derives the Control UI build ID
|
||||
// from package.json, git HEAD, and OPENCLAW_CONTROL_UI_BUILD_ID env, so a
|
||||
// file-input signature cannot exactly invalidate generated assets and a
|
||||
// warm hit could restore stale service-worker/app cache metadata.
|
||||
cache: undefined,
|
||||
},
|
||||
{
|
||||
label: "write-build-info",
|
||||
kind: "node",
|
||||
args: ["--experimental-strip-types", "scripts/write-build-info.ts"],
|
||||
},
|
||||
{
|
||||
label: "write-cli-startup-metadata",
|
||||
kind: "node",
|
||||
args: ["--experimental-strip-types", "scripts/write-cli-startup-metadata.ts"],
|
||||
},
|
||||
{
|
||||
label: "write-cli-compat",
|
||||
kind: "node",
|
||||
args: ["--experimental-strip-types", "scripts/write-cli-compat.ts"],
|
||||
},
|
||||
];
|
||||
|
||||
export const BUILD_ALL_PROFILES = {
|
||||
full: BUILD_ALL_STEPS.map((step) => step.label),
|
||||
ciArtifacts: [
|
||||
"plugins:assets:build",
|
||||
"tsdown",
|
||||
"check-cli-bootstrap-imports",
|
||||
"runtime-postbuild",
|
||||
"build-stamp",
|
||||
"runtime-postbuild-stamp",
|
||||
"write-plugin-sdk-entry-dts",
|
||||
"check-plugin-sdk-exports",
|
||||
"plugins:assets:copy",
|
||||
"copy-hook-metadata",
|
||||
"copy-export-html-templates",
|
||||
"ui:build",
|
||||
"write-build-info",
|
||||
"write-cli-startup-metadata",
|
||||
"write-cli-compat",
|
||||
],
|
||||
gatewayWatch: [
|
||||
"tsdown",
|
||||
"check-cli-bootstrap-imports",
|
||||
"runtime-postbuild",
|
||||
"build-stamp",
|
||||
"runtime-postbuild-stamp",
|
||||
],
|
||||
qaRuntime: [
|
||||
"plugins:assets:build",
|
||||
"tsdown",
|
||||
"check-cli-bootstrap-imports",
|
||||
"runtime-postbuild",
|
||||
"build-stamp",
|
||||
"runtime-postbuild-stamp",
|
||||
],
|
||||
cliStartup: [
|
||||
"tsdown",
|
||||
"check-cli-bootstrap-imports",
|
||||
"runtime-postbuild",
|
||||
"build-stamp",
|
||||
"runtime-postbuild-stamp",
|
||||
"write-cli-startup-metadata",
|
||||
"write-cli-compat",
|
||||
],
|
||||
};
|
||||
|
||||
export const BUILD_ALL_PROFILE_STEP_ENV = {
|
||||
full: {
|
||||
tsdown: {
|
||||
OPENCLAW_PRESERVE_CLI_STARTUP_METADATA: "1",
|
||||
},
|
||||
},
|
||||
ciArtifacts: {
|
||||
tsdown: {
|
||||
OPENCLAW_RUN_NODE_SKIP_DTS_BUILD: "0",
|
||||
OPENCLAW_PRESERVE_CLI_STARTUP_METADATA: "1",
|
||||
},
|
||||
},
|
||||
gatewayWatch: {
|
||||
tsdown: {
|
||||
OPENCLAW_RUN_NODE_SKIP_DTS_BUILD: "1",
|
||||
},
|
||||
"runtime-postbuild": {
|
||||
OPENCLAW_RUNTIME_POSTBUILD_STATIC_ASSETS: "0",
|
||||
},
|
||||
},
|
||||
qaRuntime: {
|
||||
tsdown: {
|
||||
OPENCLAW_RUN_NODE_SKIP_DTS_BUILD: "1",
|
||||
},
|
||||
},
|
||||
cliStartup: {
|
||||
tsdown: {
|
||||
OPENCLAW_RUN_NODE_SKIP_DTS_BUILD: "1",
|
||||
OPENCLAW_PRESERVE_CLI_STARTUP_METADATA: "1",
|
||||
},
|
||||
"runtime-postbuild": {
|
||||
OPENCLAW_RUNTIME_POSTBUILD_STATIC_ASSETS: "0",
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
export function buildAllUsage() {
|
||||
return [
|
||||
"Usage: node scripts/build-all.mjs [profile]",
|
||||
"",
|
||||
"Builds OpenClaw artifacts for the selected profile.",
|
||||
"",
|
||||
"Profiles:",
|
||||
...Object.keys(BUILD_ALL_PROFILES).map((profile) => ` ${profile}`),
|
||||
"",
|
||||
"Options:",
|
||||
" -h, --help Show this help.",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
export function parseBuildAllArgs(argv) {
|
||||
const args = {
|
||||
help: false,
|
||||
profile: "full",
|
||||
};
|
||||
let sawProfile = false;
|
||||
for (const arg of argv) {
|
||||
if (arg === "--help" || arg === "-h") {
|
||||
args.help = true;
|
||||
} else if (arg.startsWith("-")) {
|
||||
throw new Error(`unknown argument: ${arg}\n\n${buildAllUsage()}`);
|
||||
} else if (sawProfile) {
|
||||
throw new Error(`unexpected argument: ${arg}\n\n${buildAllUsage()}`);
|
||||
} else {
|
||||
args.profile = arg;
|
||||
sawProfile = true;
|
||||
}
|
||||
}
|
||||
if (!args.help && !BUILD_ALL_PROFILES[args.profile]) {
|
||||
throw new Error(`Unknown build profile: ${args.profile}\n\n${buildAllUsage()}`);
|
||||
}
|
||||
return args;
|
||||
}
|
||||
|
||||
export function resolveBuildAllSteps(profile = "full") {
|
||||
const labels = BUILD_ALL_PROFILES[profile];
|
||||
if (!labels) {
|
||||
throw new Error(`Unknown build profile: ${profile}`);
|
||||
}
|
||||
const selected = labels.map((label) => BUILD_ALL_STEPS.find((step) => step.label === label));
|
||||
if (selected.some((step) => !step)) {
|
||||
const missing = labels.filter((label) => !BUILD_ALL_STEPS.some((step) => step.label === label));
|
||||
throw new Error(`Build profile ${profile} references unknown steps: ${missing.join(", ")}`);
|
||||
}
|
||||
const envOverrides = BUILD_ALL_PROFILE_STEP_ENV[profile] ?? {};
|
||||
return selected.map((step) => {
|
||||
const env = envOverrides[step.label];
|
||||
if (!env) {
|
||||
return step;
|
||||
}
|
||||
const mergedEnv = Object.assign({}, step.env, env);
|
||||
return Object.assign({}, step, { env: mergedEnv });
|
||||
});
|
||||
}
|
||||
|
||||
function resolveStepEnv(step, env, platform) {
|
||||
const stepEnv = step.env ? Object.assign({}, env, step.env) : env;
|
||||
if (platform !== "win32" || !step.windowsNodeOptions) {
|
||||
return stepEnv;
|
||||
}
|
||||
const currentNodeOptions = stepEnv.NODE_OPTIONS?.trim() ?? "";
|
||||
if (currentNodeOptions.includes(step.windowsNodeOptions)) {
|
||||
return stepEnv;
|
||||
}
|
||||
return {
|
||||
...stepEnv,
|
||||
NODE_OPTIONS: currentNodeOptions
|
||||
? `${currentNodeOptions} ${step.windowsNodeOptions}`
|
||||
: step.windowsNodeOptions,
|
||||
};
|
||||
}
|
||||
|
||||
export function resolveBuildAllStep(step, params = {}) {
|
||||
const platform = params.platform ?? process.platform;
|
||||
const env = resolveStepEnv(step, params.env ?? process.env, platform);
|
||||
if (step.kind === "pnpm") {
|
||||
const nodeFallbackArgs =
|
||||
env.OPENCLAW_BUILD_ALL_NO_PNPM === "1" ? PNPM_STEP_NODE_FALLBACKS.get(step.label) : undefined;
|
||||
if (nodeFallbackArgs) {
|
||||
return {
|
||||
command: params.nodeExecPath ?? nodeBin,
|
||||
args: nodeFallbackArgs,
|
||||
options: {
|
||||
stdio: "inherit",
|
||||
env,
|
||||
},
|
||||
};
|
||||
}
|
||||
const runner = resolvePnpmRunner({
|
||||
env,
|
||||
pnpmArgs: step.pnpmArgs,
|
||||
nodeExecPath: params.nodeExecPath ?? nodeBin,
|
||||
npmExecPath: params.npmExecPath ?? env.npm_execpath,
|
||||
comSpec: params.comSpec,
|
||||
platform,
|
||||
});
|
||||
return {
|
||||
command: runner.command,
|
||||
args: runner.args,
|
||||
options: {
|
||||
stdio: "inherit",
|
||||
env,
|
||||
shell: runner.shell,
|
||||
windowsVerbatimArguments: runner.windowsVerbatimArguments,
|
||||
},
|
||||
};
|
||||
}
|
||||
return {
|
||||
command: params.nodeExecPath ?? nodeBin,
|
||||
args: step.args,
|
||||
options: {
|
||||
stdio: "inherit",
|
||||
env,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function normalizeCacheEntry(entry) {
|
||||
if (typeof entry === "string") {
|
||||
return { path: entry };
|
||||
}
|
||||
return entry;
|
||||
}
|
||||
|
||||
function cacheEntryIncludesFile(entry, filePath) {
|
||||
if (!entry.extensions?.length) {
|
||||
return true;
|
||||
}
|
||||
return entry.extensions.some((extension) => filePath.endsWith(extension));
|
||||
}
|
||||
|
||||
function listFilesRecursively(rootPath, fsImpl, cacheEntry = { path: rootPath }) {
|
||||
let stat;
|
||||
try {
|
||||
stat = fsImpl.statSync(rootPath);
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
if (stat.isFile()) {
|
||||
return cacheEntryIncludesFile(cacheEntry, rootPath) ? [rootPath] : [];
|
||||
}
|
||||
if (!stat.isDirectory()) {
|
||||
return [];
|
||||
}
|
||||
const out = [];
|
||||
const entries = fsImpl.readdirSync(rootPath, { withFileTypes: true });
|
||||
const recursive = cacheEntry.recursive !== false;
|
||||
for (const dirent of entries) {
|
||||
if (dirent.name === ".DS_Store") {
|
||||
continue;
|
||||
}
|
||||
const entryPath = path.join(rootPath, dirent.name);
|
||||
if (dirent.isDirectory() && recursive) {
|
||||
out.push(...listFilesRecursively(entryPath, fsImpl, cacheEntry));
|
||||
} else if (dirent.isFile() && cacheEntryIncludesFile(cacheEntry, entryPath)) {
|
||||
out.push(entryPath);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function listCacheFiles(rootDir, entries, fsImpl) {
|
||||
return entries
|
||||
.map(normalizeCacheEntry)
|
||||
.flatMap((entry) => listFilesRecursively(path.resolve(rootDir, entry.path), fsImpl, entry))
|
||||
.toSorted();
|
||||
}
|
||||
|
||||
function portableRelativePath(rootDir, filePath) {
|
||||
return path.relative(rootDir, filePath).split(path.sep).join("/");
|
||||
}
|
||||
|
||||
function normalizePortablePath(filePath) {
|
||||
return filePath.replaceAll("\\", "/");
|
||||
}
|
||||
|
||||
function resolveCachePaths(rootDir, step) {
|
||||
const safeLabel = step.label.replace(/[^a-zA-Z0-9._-]+/g, "_");
|
||||
const cacheDir = path.resolve(rootDir, ".artifacts/build-all-cache", safeLabel);
|
||||
return {
|
||||
cacheDir,
|
||||
outputRoot: path.join(cacheDir, "outputs"),
|
||||
stampPath: path.join(cacheDir, "stamp.json"),
|
||||
};
|
||||
}
|
||||
|
||||
function hashInputFiles(rootDir, files, fsImpl, envEntries = [], env = process.env) {
|
||||
const hash = createHash("sha256");
|
||||
hash.update(`v${BUILD_CACHE_VERSION}\0`);
|
||||
for (const name of envEntries.toSorted((left, right) => left.localeCompare(right))) {
|
||||
hash.update(`env:${name}`);
|
||||
hash.update("\0");
|
||||
hash.update(env[name] ?? "");
|
||||
hash.update("\0");
|
||||
}
|
||||
for (const file of files) {
|
||||
hash.update(portableRelativePath(rootDir, file));
|
||||
hash.update("\0");
|
||||
hash.update(fsImpl.readFileSync(file));
|
||||
hash.update("\0");
|
||||
}
|
||||
return hash.digest("hex");
|
||||
}
|
||||
|
||||
function readCacheStamp(stampPath, fsImpl) {
|
||||
try {
|
||||
return JSON.parse(fsImpl.readFileSync(stampPath, "utf8"));
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
function hasAllFiles(rootDir, relativeFiles, fsImpl) {
|
||||
return relativeFiles.every((relativeFile) => {
|
||||
try {
|
||||
return fsImpl.statSync(path.resolve(rootDir, relativeFile)).isFile();
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
function copyFileSync(fsImpl, sourcePath, targetPath) {
|
||||
fsImpl.mkdirSync(path.dirname(targetPath), { recursive: true });
|
||||
fsImpl.copyFileSync(sourcePath, targetPath);
|
||||
}
|
||||
|
||||
export function resolveBuildAllStepCacheState(step, params = {}) {
|
||||
if (!step.cache) {
|
||||
return { cacheable: false, fresh: false, reason: "no-cache" };
|
||||
}
|
||||
const rootDir = params.rootDir ?? process.cwd();
|
||||
const fsImpl = params.fs ?? fs;
|
||||
const inputFiles = listCacheFiles(rootDir, step.cache.inputs, fsImpl);
|
||||
if (inputFiles.length === 0) {
|
||||
return { cacheable: true, fresh: false, reason: "missing-inputs" };
|
||||
}
|
||||
const signature = hashInputFiles(
|
||||
rootDir,
|
||||
inputFiles,
|
||||
fsImpl,
|
||||
step.cache.env ?? [],
|
||||
params.env ?? process.env,
|
||||
);
|
||||
const { outputRoot, stampPath } = resolveCachePaths(rootDir, step);
|
||||
const stamp = readCacheStamp(stampPath, fsImpl);
|
||||
const outputFiles = listCacheFiles(rootDir, step.cache.outputs, fsImpl);
|
||||
const relativeOutputFiles = outputFiles.map((file) => portableRelativePath(rootDir, file));
|
||||
const stampedOutputs = Array.isArray(stamp?.outputs)
|
||||
? stamp.outputs.map((entry) => normalizePortablePath(entry))
|
||||
: [];
|
||||
const stampMatches = stamp?.version === BUILD_CACHE_VERSION && stamp.signature === signature;
|
||||
const actualOutputsPresent =
|
||||
stampedOutputs.length > 0 && hasAllFiles(rootDir, stampedOutputs, fsImpl);
|
||||
const cachedOutputsPresent =
|
||||
stampedOutputs.length > 0 && hasAllFiles(outputRoot, stampedOutputs, fsImpl);
|
||||
const alwaysRestore = step.cache.restore === "always";
|
||||
const actualOutputsAcceptable = actualOutputsPresent && !alwaysRestore;
|
||||
const restorable =
|
||||
stampMatches && cachedOutputsPresent && (alwaysRestore || !actualOutputsPresent);
|
||||
const fresh = stampMatches && (actualOutputsAcceptable || cachedOutputsPresent);
|
||||
return {
|
||||
cacheable: true,
|
||||
fresh,
|
||||
restorable,
|
||||
reason: fresh ? (restorable ? "fresh-cache" : "fresh") : "stale",
|
||||
signature,
|
||||
outputRoot,
|
||||
stampPath,
|
||||
inputFiles: inputFiles.length,
|
||||
outputFiles: outputFiles.length,
|
||||
relativeOutputFiles,
|
||||
stampedOutputs,
|
||||
};
|
||||
}
|
||||
|
||||
export function writeBuildAllStepCacheStamp(step, cacheState, params = {}) {
|
||||
if (
|
||||
!cacheState.cacheable ||
|
||||
!cacheState.signature ||
|
||||
!cacheState.stampPath ||
|
||||
!cacheState.outputRoot ||
|
||||
!cacheState.relativeOutputFiles?.length
|
||||
) {
|
||||
return;
|
||||
}
|
||||
const fsImpl = params.fs ?? fs;
|
||||
const rootDir = params.rootDir ?? process.cwd();
|
||||
for (const relativeFile of cacheState.relativeOutputFiles) {
|
||||
copyFileSync(
|
||||
fsImpl,
|
||||
path.resolve(rootDir, relativeFile),
|
||||
path.resolve(cacheState.outputRoot, relativeFile),
|
||||
);
|
||||
}
|
||||
fsImpl.mkdirSync(path.dirname(cacheState.stampPath), { recursive: true });
|
||||
fsImpl.writeFileSync(
|
||||
cacheState.stampPath,
|
||||
`${JSON.stringify({
|
||||
version: BUILD_CACHE_VERSION,
|
||||
label: step.label,
|
||||
signature: cacheState.signature,
|
||||
outputs: cacheState.relativeOutputFiles,
|
||||
})}\n`,
|
||||
);
|
||||
}
|
||||
|
||||
export function resolveBuildAllStepCacheStampState(step, cacheState, params = {}) {
|
||||
if (!cacheState.cacheable || !cacheState.signature || !step.cache) {
|
||||
return cacheState;
|
||||
}
|
||||
const rootDir = params.rootDir ?? process.cwd();
|
||||
const fsImpl = params.fs ?? fs;
|
||||
const outputFiles = listCacheFiles(rootDir, step.cache.outputs, fsImpl);
|
||||
return {
|
||||
...cacheState,
|
||||
outputFiles: outputFiles.length,
|
||||
relativeOutputFiles: outputFiles.map((file) => portableRelativePath(rootDir, file)),
|
||||
};
|
||||
}
|
||||
|
||||
export function restoreBuildAllStepCacheOutputs(cacheState, params = {}) {
|
||||
if (!cacheState.restorable || !cacheState.outputRoot || !cacheState.stampedOutputs?.length) {
|
||||
return false;
|
||||
}
|
||||
const fsImpl = params.fs ?? fs;
|
||||
const rootDir = params.rootDir ?? process.cwd();
|
||||
for (const relativeFile of cacheState.stampedOutputs) {
|
||||
copyFileSync(
|
||||
fsImpl,
|
||||
path.resolve(cacheState.outputRoot, relativeFile),
|
||||
path.resolve(rootDir, relativeFile),
|
||||
);
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
export function formatBuildAllDuration(durationMs) {
|
||||
const clampedMs = Math.max(0, durationMs);
|
||||
if (clampedMs < 1000) {
|
||||
return `${Math.round(clampedMs)}ms`;
|
||||
}
|
||||
if (clampedMs < 10000) {
|
||||
return `${(clampedMs / 1000).toFixed(2)}s`;
|
||||
}
|
||||
return `${(clampedMs / 1000).toFixed(1)}s`;
|
||||
}
|
||||
|
||||
export function formatBuildAllTimingSummary(timings) {
|
||||
if (timings.length === 0) {
|
||||
return "[build-all] phase timings: no phases ran";
|
||||
}
|
||||
const totalMs = timings.reduce((sum, timing) => sum + timing.durationMs, 0);
|
||||
const phases = timings
|
||||
.toSorted((left, right) => right.durationMs - left.durationMs)
|
||||
.map((timing) => {
|
||||
const status = timing.status === "ran" ? "" : ` (${timing.status})`;
|
||||
return `${timing.label}${status} ${formatBuildAllDuration(timing.durationMs)}`;
|
||||
})
|
||||
.join("; ");
|
||||
return `[build-all] phase timings: total ${formatBuildAllDuration(totalMs)}; slowest ${phases}`;
|
||||
}
|
||||
|
||||
function isMainModule() {
|
||||
const argv1 = process.argv[1];
|
||||
if (!argv1) {
|
||||
return false;
|
||||
}
|
||||
return import.meta.url === pathToFileURL(argv1).href;
|
||||
}
|
||||
|
||||
if (isMainModule()) {
|
||||
let args;
|
||||
try {
|
||||
args = parseBuildAllArgs(process.argv.slice(2));
|
||||
} catch (error) {
|
||||
console.error(error instanceof Error ? error.message : String(error));
|
||||
process.exit(2);
|
||||
}
|
||||
if (args?.help) {
|
||||
console.log(buildAllUsage());
|
||||
} else {
|
||||
const timings = [];
|
||||
let exitCode = 0;
|
||||
for (const step of resolveBuildAllSteps(args.profile)) {
|
||||
const startedAt = performance.now();
|
||||
const cacheState = resolveBuildAllStepCacheState(step);
|
||||
if (process.env.OPENCLAW_BUILD_CACHE !== "0" && cacheState.fresh) {
|
||||
restoreBuildAllStepCacheOutputs(cacheState);
|
||||
const durationMs = performance.now() - startedAt;
|
||||
timings.push({ label: step.label, status: "cached", durationMs });
|
||||
console.error(`[build-all] ${step.label} (cached) ${formatBuildAllDuration(durationMs)}`);
|
||||
continue;
|
||||
}
|
||||
console.error(`[build-all] ${step.label}`);
|
||||
const invocation = resolveBuildAllStep(step);
|
||||
const result = spawnSync(invocation.command, invocation.args, invocation.options);
|
||||
const durationMs = performance.now() - startedAt;
|
||||
if (typeof result.status === "number") {
|
||||
if (result.status !== 0) {
|
||||
timings.push({ label: step.label, status: "failed", durationMs });
|
||||
console.error(
|
||||
`[build-all] ${step.label} failed after ${formatBuildAllDuration(durationMs)}`,
|
||||
);
|
||||
exitCode = result.status;
|
||||
break;
|
||||
}
|
||||
writeBuildAllStepCacheStamp(step, resolveBuildAllStepCacheStampState(step, cacheState));
|
||||
timings.push({ label: step.label, status: "ran", durationMs });
|
||||
console.error(`[build-all] ${step.label} done in ${formatBuildAllDuration(durationMs)}`);
|
||||
continue;
|
||||
}
|
||||
timings.push({ label: step.label, status: "failed", durationMs });
|
||||
console.error(`[build-all] ${step.label} failed after ${formatBuildAllDuration(durationMs)}`);
|
||||
exitCode = 1;
|
||||
break;
|
||||
}
|
||||
console.error(formatBuildAllTimingSummary(timings));
|
||||
if (exitCode !== 0) {
|
||||
process.exit(exitCode);
|
||||
}
|
||||
}
|
||||
}
|
||||
78
scripts/build-and-run-mac.sh
Executable file
78
scripts/build-and-run-mac.sh
Executable file
@@ -0,0 +1,78 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
ROOT_DIR="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
APP_DIR="$ROOT_DIR/apps/macos"
|
||||
|
||||
usage() {
|
||||
printf 'Usage: %s\n' "$(basename "$0")"
|
||||
printf 'Build, stop, and relaunch the local debug OpenClaw macOS app.\n'
|
||||
}
|
||||
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--help|-h)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
--) ;;
|
||||
*) printf 'ERROR: Unknown build-and-run-mac option: %s\n' "$arg" >&2; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
cd "$APP_DIR"
|
||||
|
||||
BUILD_PATH=".build-local"
|
||||
PRODUCT="OpenClaw"
|
||||
BIN="$BUILD_PATH/debug/$PRODUCT"
|
||||
BIN_ABS="$(pwd)/$BIN"
|
||||
APP_CWD="$(pwd -P)"
|
||||
LOG_PATH="${OPENCLAW_MAC_RUN_LOG:-$(mktemp "${TMPDIR:-/tmp}/openclaw-${PRODUCT}.XXXXXX.log")}"
|
||||
|
||||
process_cwd_matches() {
|
||||
local pid="$1"
|
||||
local cwd=""
|
||||
cwd="$(lsof -a -p "$pid" -d cwd -Fn 2>/dev/null | sed -n 's/^n//p' | head -n 1 || true)"
|
||||
[[ "$cwd" == "$APP_CWD" ]]
|
||||
}
|
||||
|
||||
local_debug_app_pids() {
|
||||
{
|
||||
pgrep -f "$BIN_ABS" 2>/dev/null || true
|
||||
pgrep -f "$BIN" 2>/dev/null || true
|
||||
} | while IFS= read -r pid; do
|
||||
[[ "$pid" =~ ^[0-9]+$ ]] || continue
|
||||
if process_cwd_matches "$pid"; then
|
||||
printf '%s\n' "$pid"
|
||||
fi
|
||||
done | sort -u
|
||||
}
|
||||
|
||||
stop_existing_local_app() {
|
||||
for _ in {1..10}; do
|
||||
local pids=""
|
||||
pids="$(local_debug_app_pids)"
|
||||
if [[ -z "$pids" ]]; then
|
||||
return 0
|
||||
fi
|
||||
while IFS= read -r pid; do
|
||||
kill "$pid" 2>/dev/null || true
|
||||
done <<< "$pids"
|
||||
sleep 0.3
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
printf "\n▶️ Building $PRODUCT (debug, build path: $BUILD_PATH)\n"
|
||||
swift build -c debug --product "$PRODUCT" --build-path "$BUILD_PATH"
|
||||
|
||||
printf "\n⏹ Stopping existing $PRODUCT...\n"
|
||||
if ! stop_existing_local_app; then
|
||||
printf "ERROR: existing local %s process did not exit: %s\n" "$PRODUCT" "$BIN_ABS" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
printf "\n🚀 Launching $BIN_ABS ...\n"
|
||||
nohup "$BIN_ABS" >"$LOG_PATH" 2>&1 &
|
||||
PID=$!
|
||||
printf "Started $PRODUCT (PID $PID). Logs: $LOG_PATH\n"
|
||||
128
scripts/build-diffs-viewer-runtime.mjs
Normal file
128
scripts/build-diffs-viewer-runtime.mjs
Normal file
@@ -0,0 +1,128 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Builds browser runtime bundles for the diffs viewer assets.
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { build } from "esbuild";
|
||||
|
||||
const modulePath = fileURLToPath(import.meta.url);
|
||||
const repoRoot = path.resolve(path.dirname(modulePath), "..");
|
||||
const pierreDiffsEmptySideEffectNamespace = "openclaw-diffs-empty-side-effect";
|
||||
const pierreDiffsEmptySideEffectPath = "pierre-diffs-parse-decorations-side-effect";
|
||||
|
||||
const targets = {
|
||||
curated: {
|
||||
entry: "extensions/diffs/src/viewer-client.ts",
|
||||
output: "extensions/diffs/assets/viewer-runtime.js",
|
||||
shikiAlias: "scripts/diffs-shiki-curated.ts",
|
||||
},
|
||||
full: {
|
||||
entry: "extensions/diffs/src/viewer-client.ts",
|
||||
output: "extensions/diffs-language-pack/assets/viewer-runtime.js",
|
||||
},
|
||||
};
|
||||
|
||||
function toPosixPath(value) {
|
||||
return String(value ?? "").replaceAll("\\", "/");
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates the esbuild plugin that neutralizes Pierre diffs' browser side-effect import.
|
||||
*/
|
||||
export function createPierreDiffsSideEffectImportPlugin() {
|
||||
return {
|
||||
name: "openclaw-diffs-pierre-side-effect-imports",
|
||||
setup(buildContext) {
|
||||
buildContext.onResolve({ filter: /^diff$/ }, (args) => {
|
||||
const importer = toPosixPath(args.importer);
|
||||
if (!importer.endsWith("/@pierre/diffs/dist/utils/parseDiffDecorations.js")) {
|
||||
return undefined;
|
||||
}
|
||||
return {
|
||||
path: pierreDiffsEmptySideEffectPath,
|
||||
namespace: pierreDiffsEmptySideEffectNamespace,
|
||||
sideEffects: true,
|
||||
};
|
||||
});
|
||||
buildContext.onLoad(
|
||||
{
|
||||
filter: /^pierre-diffs-parse-decorations-side-effect$/,
|
||||
namespace: pierreDiffsEmptySideEffectNamespace,
|
||||
},
|
||||
() => ({
|
||||
contents: "export {};\n",
|
||||
loader: "js",
|
||||
}),
|
||||
);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds one configured diffs viewer runtime target.
|
||||
*/
|
||||
export async function buildDiffsViewerRuntime(targetName) {
|
||||
const target = targets[targetName];
|
||||
if (!target) {
|
||||
throw new Error(
|
||||
`Usage: node scripts/build-diffs-viewer-runtime.mjs ${Object.keys(targets).join("|")}`,
|
||||
);
|
||||
}
|
||||
|
||||
const outputPath = path.join(repoRoot, target.output);
|
||||
await fs.mkdir(path.dirname(outputPath), { recursive: true });
|
||||
|
||||
const result = await build({
|
||||
entryPoints: [path.join(repoRoot, target.entry)],
|
||||
bundle: true,
|
||||
platform: "browser",
|
||||
target: "es2020",
|
||||
format: "esm",
|
||||
minify: true,
|
||||
define: {
|
||||
NaN: "Number.NaN",
|
||||
},
|
||||
legalComments: "none",
|
||||
outfile: outputPath,
|
||||
write: false,
|
||||
plugins: [
|
||||
createPierreDiffsSideEffectImportPlugin(),
|
||||
...(target.shikiAlias
|
||||
? [
|
||||
{
|
||||
name: "openclaw-diffs-curated-shiki",
|
||||
setup(buildContext) {
|
||||
buildContext.onResolve({ filter: /^shiki$/ }, () => ({
|
||||
path: path.join(repoRoot, target.shikiAlias),
|
||||
}));
|
||||
},
|
||||
},
|
||||
]
|
||||
: []),
|
||||
],
|
||||
});
|
||||
|
||||
const outputFile = result.outputFiles?.[0];
|
||||
if (!outputFile) {
|
||||
throw new Error(`esbuild did not produce ${target.output}`);
|
||||
}
|
||||
|
||||
const runtime = outputFile.text.replace(/[ \t]+$/gm, "");
|
||||
let previousRuntime = null;
|
||||
try {
|
||||
previousRuntime = await fs.readFile(outputPath, "utf8");
|
||||
} catch (error) {
|
||||
if (error?.code !== "ENOENT") {
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
if (previousRuntime !== runtime) {
|
||||
await fs.writeFile(outputPath, runtime);
|
||||
}
|
||||
}
|
||||
|
||||
if (process.argv[1] === modulePath) {
|
||||
await buildDiffsViewerRuntime(process.argv[2]);
|
||||
}
|
||||
15
scripts/build-docs-list.mjs
Normal file
15
scripts/build-docs-list.mjs
Normal file
@@ -0,0 +1,15 @@
|
||||
#!/usr/bin/env node
|
||||
// Writes the `bin/docs-list` wrapper used by package scripts and docs tooling.
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const binDir = path.join(root, "bin");
|
||||
const binPath = path.join(binDir, "docs-list");
|
||||
|
||||
fs.mkdirSync(binDir, { recursive: true });
|
||||
|
||||
const wrapper = `#!/usr/bin/env node\nimport { spawnSync } from "node:child_process";\nimport path from "node:path";\nimport { fileURLToPath } from "node:url";\n\nconst here = path.dirname(fileURLToPath(import.meta.url));\nconst script = path.join(here, "..", "scripts", "docs-list.js");\n\nconst result = spawnSync(process.execPath, [script], { stdio: "inherit" });\nprocess.exit(result.status ?? 1);\n`;
|
||||
|
||||
fs.writeFileSync(binPath, wrapper, { mode: 0o755 });
|
||||
1
scripts/build-stamp.d.mts
Normal file
1
scripts/build-stamp.d.mts
Normal file
@@ -0,0 +1 @@
|
||||
export { BUILD_STAMP_FILE, resolveGitHead, writeBuildStamp } from "./lib/local-build-metadata.mjs";
|
||||
17
scripts/build-stamp.mjs
Normal file
17
scripts/build-stamp.mjs
Normal file
@@ -0,0 +1,17 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Writes the local build stamp and re-exports build metadata helpers.
|
||||
import process from "node:process";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { writeBuildStamp } from "./lib/local-build-metadata.mjs";
|
||||
|
||||
export { BUILD_STAMP_FILE, resolveGitHead, writeBuildStamp } from "./lib/local-build-metadata.mjs";
|
||||
|
||||
if (import.meta.url === pathToFileURL(process.argv[1] ?? "").href) {
|
||||
try {
|
||||
writeBuildStamp();
|
||||
} catch (error) {
|
||||
console.error(error);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
59
scripts/build_icon.sh
Executable file
59
scripts/build_icon.sh
Executable file
@@ -0,0 +1,59 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Render the macOS .icon bundle to a padded .icns like Trimmy's pipeline.
|
||||
# Defaults target the OpenClaw assets so you can just run the script from repo root.
|
||||
|
||||
ROOT_DIR="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
|
||||
ICON_FILE=${1:-"$ROOT_DIR/apps/macos/Icon.icon"}
|
||||
BASENAME=${2:-OpenClaw}
|
||||
OUT_ROOT=${3:-"$ROOT_DIR/apps/macos/build/icon"}
|
||||
XCODE_APP=${XCODE_APP:-/Applications/Xcode.app}
|
||||
# Where the final .icns should live; override DEST_ICNS to change.
|
||||
DEST_ICNS=${DEST_ICNS:-"$ROOT_DIR/apps/macos/Sources/OpenClaw/Resources/OpenClaw.icns"}
|
||||
|
||||
ICTOOL="$XCODE_APP/Contents/Applications/Icon Composer.app/Contents/Executables/ictool"
|
||||
if [[ ! -x "$ICTOOL" ]]; then
|
||||
ICTOOL="$XCODE_APP/Contents/Applications/Icon Composer.app/Contents/Executables/icontool"
|
||||
fi
|
||||
if [[ ! -x "$ICTOOL" ]]; then
|
||||
echo "ictool/icontool not found. Set XCODE_APP if Xcode is elsewhere." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
ICONSET_DIR="$OUT_ROOT/${BASENAME}.iconset"
|
||||
TMP_DIR="$OUT_ROOT/tmp"
|
||||
mkdir -p "$ICONSET_DIR" "$TMP_DIR"
|
||||
|
||||
MASTER_ART="$TMP_DIR/icon_art_824.png"
|
||||
MASTER_1024="$TMP_DIR/icon_1024.png"
|
||||
|
||||
# Render inner art (no margin) with macOS Default appearance
|
||||
"$ICTOOL" "$ICON_FILE" \
|
||||
--export-preview macOS Default 824 824 1 -45 "$MASTER_ART"
|
||||
|
||||
# Pad to 1024x1024 with transparent border
|
||||
sips --padToHeightWidth 1024 1024 "$MASTER_ART" --out "$MASTER_1024" >/dev/null
|
||||
|
||||
# Generate required sizes
|
||||
sizes=(16 32 64 128 256 512 1024)
|
||||
for sz in "${sizes[@]}"; do
|
||||
out="$ICONSET_DIR/icon_${sz}x${sz}.png"
|
||||
sips -z "$sz" "$sz" "$MASTER_1024" --out "$out" >/dev/null
|
||||
if [[ "$sz" -ne 1024 ]]; then
|
||||
dbl=$((sz*2))
|
||||
out2="$ICONSET_DIR/icon_${sz}x${sz}@2x.png"
|
||||
sips -z "$dbl" "$dbl" "$MASTER_1024" --out "$out2" >/dev/null
|
||||
fi
|
||||
done
|
||||
|
||||
# 512x512@2x already covered by 1024; ensure it exists
|
||||
cp "$MASTER_1024" "$ICONSET_DIR/icon_512x512@2x.png"
|
||||
|
||||
iconutil -c icns "$ICONSET_DIR" -o "$OUT_ROOT/${BASENAME}.icns"
|
||||
|
||||
mkdir -p "$(dirname "$DEST_ICNS")"
|
||||
cp "$OUT_ROOT/${BASENAME}.icns" "$DEST_ICNS"
|
||||
|
||||
echo "Icon.icns generated at $DEST_ICNS"
|
||||
9
scripts/bundle-a2ui.mjs
Normal file
9
scripts/bundle-a2ui.mjs
Normal file
@@ -0,0 +1,9 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Runs bundled asset build hooks for the Canvas A2UI runtime.
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { runBundledPluginAssetHooks } from "./bundled-plugin-assets.mjs";
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
await runBundledPluginAssetHooks({ phase: "build", plugins: ["canvas"] });
|
||||
}
|
||||
4
scripts/bundle-a2ui.sh
Executable file
4
scripts/bundle-a2ui.sh
Executable file
@@ -0,0 +1,4 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
exec node "$ROOT_DIR/scripts/bundle-a2ui.mjs" "$@"
|
||||
187
scripts/bundled-plugin-assets.mjs
Normal file
187
scripts/bundled-plugin-assets.mjs
Normal file
@@ -0,0 +1,187 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Discovers and runs bundled plugin package asset hooks.
|
||||
import { spawnSync } from "node:child_process";
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
|
||||
const rootDir = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const VALID_PHASES = new Set(["build", "copy"]);
|
||||
|
||||
async function readJsonFile(filePath) {
|
||||
return JSON.parse(await fs.readFile(filePath, "utf8"));
|
||||
}
|
||||
|
||||
async function pathExists(filePath) {
|
||||
try {
|
||||
await fs.stat(filePath);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function packagePluginAliases(packageName) {
|
||||
if (typeof packageName !== "string") {
|
||||
return [];
|
||||
}
|
||||
const aliases = [packageName];
|
||||
const unscopedName = packageName.split("/").at(-1);
|
||||
if (unscopedName) {
|
||||
aliases.push(unscopedName);
|
||||
if (unscopedName.endsWith("-plugin")) {
|
||||
aliases.push(unscopedName.slice(0, -"-plugin".length));
|
||||
}
|
||||
}
|
||||
return aliases;
|
||||
}
|
||||
|
||||
async function resolvePluginAliases(pluginDir, packageJson) {
|
||||
const aliases = new Set([path.basename(pluginDir), ...packagePluginAliases(packageJson.name)]);
|
||||
const manifestPath = path.join(pluginDir, "openclaw.plugin.json");
|
||||
if (await pathExists(manifestPath)) {
|
||||
const manifest = await readJsonFile(manifestPath);
|
||||
if (typeof manifest.id === "string" && manifest.id) {
|
||||
aliases.add(manifest.id);
|
||||
}
|
||||
}
|
||||
return aliases;
|
||||
}
|
||||
|
||||
function resolveAssetCommand(packageJson, phase) {
|
||||
const assetScripts = packageJson.openclaw?.assetScripts;
|
||||
if (!assetScripts || typeof assetScripts !== "object") {
|
||||
return null;
|
||||
}
|
||||
const command = assetScripts[phase];
|
||||
return typeof command === "string" && command.trim() ? command.trim() : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads bundled plugin asset hook commands for a build or copy phase.
|
||||
*/
|
||||
export async function readBundledPluginAssetHooks(options = {}) {
|
||||
const repoRoot = options.rootDir ?? rootDir;
|
||||
const phase = options.phase;
|
||||
if (!VALID_PHASES.has(phase)) {
|
||||
throw new Error(`Unsupported bundled plugin asset phase: ${String(phase)}`);
|
||||
}
|
||||
|
||||
const pluginFilters = new Set((options.plugins ?? []).filter(Boolean));
|
||||
const extensionsDir = path.join(repoRoot, "extensions");
|
||||
let entries;
|
||||
try {
|
||||
entries = await fs.readdir(extensionsDir, { withFileTypes: true });
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
|
||||
const hooks = [];
|
||||
for (const entry of entries) {
|
||||
if (!entry.isDirectory()) {
|
||||
continue;
|
||||
}
|
||||
const pluginDir = path.join(extensionsDir, entry.name);
|
||||
const packagePath = path.join(pluginDir, "package.json");
|
||||
if (!(await pathExists(packagePath))) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const packageJson = await readJsonFile(packagePath);
|
||||
const aliases = await resolvePluginAliases(pluginDir, packageJson);
|
||||
if (pluginFilters.size > 0 && ![...pluginFilters].some((plugin) => aliases.has(plugin))) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const command = resolveAssetCommand(packageJson, phase);
|
||||
if (!command) {
|
||||
continue;
|
||||
}
|
||||
|
||||
hooks.push({
|
||||
aliases: [...aliases].toSorted(),
|
||||
command,
|
||||
packageName: packageJson.name,
|
||||
phase,
|
||||
pluginDir,
|
||||
pluginId: aliases.has(entry.name) ? entry.name : [...aliases][0],
|
||||
});
|
||||
}
|
||||
|
||||
return hooks.toSorted((left, right) => left.pluginDir.localeCompare(right.pluginDir));
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs bundled plugin asset hook commands for the selected phase/plugins.
|
||||
*/
|
||||
export async function runBundledPluginAssetHooks(options = {}) {
|
||||
const phase = options.phase;
|
||||
const hooks = await readBundledPluginAssetHooks(options);
|
||||
if (hooks.length === 0) {
|
||||
const scope = options.plugins?.length ? ` for ${options.plugins.join(", ")}` : "";
|
||||
console.log(`No bundled plugin asset ${phase} hooks${scope}; skipping.`);
|
||||
return;
|
||||
}
|
||||
|
||||
for (const hook of hooks) {
|
||||
console.log(`[${hook.pluginId}] ${phase}: ${hook.command}`);
|
||||
const result = spawnSync(hook.command, {
|
||||
cwd: hook.pluginDir,
|
||||
env: process.env,
|
||||
shell: true,
|
||||
stdio: "inherit",
|
||||
});
|
||||
if (result.status !== 0) {
|
||||
process.exit(result.status ?? 1);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Parses `--phase` and repeated `--plugin` flags for asset hook scripts.
|
||||
*/
|
||||
export function parseBundledPluginAssetArgs(argv) {
|
||||
const args = [...argv];
|
||||
const plugins = [];
|
||||
let phase = null;
|
||||
|
||||
while (args.length > 0) {
|
||||
const arg = args.shift();
|
||||
if (arg === "--phase") {
|
||||
phase = args.shift() ?? null;
|
||||
continue;
|
||||
}
|
||||
if (arg?.startsWith("--phase=")) {
|
||||
phase = arg.slice("--phase=".length);
|
||||
continue;
|
||||
}
|
||||
if (arg === "--plugin") {
|
||||
const plugin = args.shift();
|
||||
if (plugin) {
|
||||
plugins.push(plugin);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (arg?.startsWith("--plugin=")) {
|
||||
plugins.push(arg.slice("--plugin=".length));
|
||||
continue;
|
||||
}
|
||||
throw new Error(`Unknown bundled plugin asset argument: ${String(arg)}`);
|
||||
}
|
||||
|
||||
if (!VALID_PHASES.has(phase)) {
|
||||
throw new Error(`Expected --phase ${[...VALID_PHASES].join("|")}`);
|
||||
}
|
||||
|
||||
return { phase, plugins };
|
||||
}
|
||||
|
||||
if (import.meta.url === pathToFileURL(process.argv[1] ?? "").href) {
|
||||
try {
|
||||
await runBundledPluginAssetHooks(parseBundledPluginAssetArgs(process.argv.slice(2)));
|
||||
} catch (error) {
|
||||
console.error(error instanceof Error ? error.message : String(error));
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
627
scripts/changed-lanes.mjs
Normal file
627
scripts/changed-lanes.mjs
Normal file
@@ -0,0 +1,627 @@
|
||||
// Classifies changed files into CI lanes and release metadata scopes.
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { appendFileSync, existsSync, readFileSync } from "node:fs";
|
||||
import { booleanFlag, parseFlagArgs, stringFlag } from "./lib/arg-utils.mjs";
|
||||
import { isDirectRunUrl } from "./lib/direct-run.mjs";
|
||||
import { resolveMergeHeadDiffBase } from "./lib/merge-head-diff-base.mjs";
|
||||
|
||||
const GIT_OUTPUT_MAX_BUFFER = 64 * 1024 * 1024;
|
||||
const IMPLAUSIBLE_NO_MERGE_BASE_DIFF_PATHS = 200;
|
||||
const RAW_SYNC_CHANGED_LANES_ENV = "OPENCLAW_CHANGED_LANES_RAW_SYNC";
|
||||
|
||||
const DOCS_PATH_RE = /^(?:docs\/|README\.md$|AGENTS\.md$|.*\.mdx?$)/u;
|
||||
const APP_PATH_RE = /^(?:apps\/|Swabble\/|appcast\.xml$)/u;
|
||||
const EXTENSION_PATH_RE = /^extensions\/[^/]+(?:\/|$)/u;
|
||||
const CORE_PATH_RE = /^(?:src\/|ui\/|packages\/)/u;
|
||||
const TOOLING_PATH_RE =
|
||||
/^(?:scripts\/|test\/vitest\/|\.github\/|\.vscode\/|config\/|deploy\/|git-hooks\/|Dockerfile\.sandbox(?:-(?:browser|common))?$|Makefile$|docker-setup\.sh$|setup-podman\.sh$|openclaw\.podman\.env$|skills\/pyproject\.toml$|vitest(?:\..+)?\.config\.ts$|tsconfig.*\.json$|\.dockerignore$|\.gitignore$|\.jscpd\.json$|\.npmignore$|\.pre-commit-config\.yaml$|\.swiftformat$|\.swiftlint\.yml$|\.oxlint.*|\.oxfmt.*)/u;
|
||||
const ROOT_GLOBAL_PATH_RE =
|
||||
/^(?:package\.json$|pnpm-lock\.yaml$|pnpm-workspace\.yaml$|tsdown\.config\.ts$|vitest\.config\.ts$)/u;
|
||||
const LEGACY_ROOT_ASSET_PATH_RE = /^assets\//u;
|
||||
const LIVE_DOCKER_TOOLING_PATH_RE =
|
||||
/^(?:scripts\/test-docker-all\.mjs|scripts\/test-docker-all\.sh|scripts\/lib\/live-docker-auth\.sh|scripts\/test-live-(?:acp-bind|cli-backend|codex-harness|gateway-models|models)-docker\.sh|src\/gateway\/gateway-acp-bind\.live\.test\.ts|src\/gateway\/live-agent-probes\.test\.ts)$/u;
|
||||
const LIVE_DOCKER_PACKAGE_SCRIPT_RE = /^test:docker:live-[\w:-]+$/u;
|
||||
const TEST_PATH_RE =
|
||||
/(?:^|\/)(?:test|__tests__)\/|(?:\.|\/)(?:test|spec|e2e|browser\.test)\.[cm]?[jt]sx?$/u;
|
||||
const PUBLIC_EXTENSION_CONTRACT_RE =
|
||||
/^(?:src\/plugin-sdk\/|src\/plugins\/contracts\/|src\/channels\/plugins\/|scripts\/lib\/plugin-sdk-entrypoints\.json$|scripts\/sync-plugin-sdk-exports\.mjs$|scripts\/generate-plugin-sdk-api-baseline\.ts$)/u;
|
||||
/**
|
||||
* Files whose changes are treated as release metadata only.
|
||||
*/
|
||||
export const RELEASE_METADATA_PATHS = new Set([
|
||||
"CHANGELOG.md",
|
||||
"apps/android/CHANGELOG.md",
|
||||
"apps/android/Config/Version.properties",
|
||||
"apps/android/fastlane/metadata/android/en-US/release_notes.txt",
|
||||
"apps/android/version.json",
|
||||
"apps/ios/CHANGELOG.md",
|
||||
"apps/macos/Sources/OpenClaw/Resources/Info.plist",
|
||||
"docs/.generated/config-baseline.sha256",
|
||||
"docs/install/updating.md",
|
||||
"package.json",
|
||||
]);
|
||||
|
||||
/** @typedef {"core" | "coreTests" | "extensions" | "extensionTests" | "apps" | "docs" | "tooling" | "liveDockerTooling" | "releaseMetadata" | "all"} ChangedLane */
|
||||
|
||||
/**
|
||||
* @typedef {{
|
||||
* paths: string[];
|
||||
* lanes: Record<ChangedLane, boolean>;
|
||||
* extensionImpactFromCore: boolean;
|
||||
* docsOnly: boolean;
|
||||
* reasons: string[];
|
||||
* }} ChangedLaneResult
|
||||
*/
|
||||
|
||||
/**
|
||||
* Normalizes a changed file path into repo-relative POSIX form.
|
||||
*/
|
||||
export function normalizeChangedPath(inputPath) {
|
||||
return String(inputPath ?? "")
|
||||
.trim()
|
||||
.replaceAll("\\", "/")
|
||||
.replace(/^\.\/+/u, "");
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates the default changed-lanes result object.
|
||||
*/
|
||||
export function createEmptyChangedLanes() {
|
||||
return {
|
||||
core: false,
|
||||
coreTests: false,
|
||||
extensions: false,
|
||||
extensionTests: false,
|
||||
apps: false,
|
||||
docs: false,
|
||||
tooling: false,
|
||||
liveDockerTooling: false,
|
||||
releaseMetadata: false,
|
||||
all: false,
|
||||
};
|
||||
}
|
||||
|
||||
export function isChangedLaneTestPath(changedPath) {
|
||||
return TEST_PATH_RE.test(normalizeChangedPath(changedPath));
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {string[]} changedPaths
|
||||
* @param {{ packageJsonChangeKind?: "liveDockerTooling" | "tooling" | null }} [options]
|
||||
* @returns {ChangedLaneResult}
|
||||
*/
|
||||
/**
|
||||
* Classifies a list of changed paths into docs, app, extension, core, and tooling lanes.
|
||||
*/
|
||||
export function detectChangedLanes(changedPaths, options = {}) {
|
||||
const paths = [...new Set(changedPaths.map(normalizeChangedPath).filter(Boolean))]
|
||||
.toSorted((left, right) => left.localeCompare(right))
|
||||
.filter((changedPath) => changedPath !== "--");
|
||||
const lanes = createEmptyChangedLanes();
|
||||
const reasons = [];
|
||||
let extensionImpactFromCore = false;
|
||||
let hasNonDocs = false;
|
||||
const packageJsonIsLiveDockerTooling =
|
||||
paths.includes("package.json") && options.packageJsonChangeKind === "liveDockerTooling";
|
||||
const packageJsonIsTooling =
|
||||
paths.includes("package.json") && options.packageJsonChangeKind === "tooling";
|
||||
|
||||
if (paths.length === 0) {
|
||||
reasons.push("no changed paths");
|
||||
return { paths, lanes, extensionImpactFromCore: false, docsOnly: false, reasons };
|
||||
}
|
||||
|
||||
if (
|
||||
!packageJsonIsLiveDockerTooling &&
|
||||
!packageJsonIsTooling &&
|
||||
paths.some((changedPath) => RELEASE_METADATA_PATHS.has(changedPath)) &&
|
||||
paths.every((changedPath) => RELEASE_METADATA_PATHS.has(changedPath))
|
||||
) {
|
||||
lanes.releaseMetadata = true;
|
||||
lanes.docs = paths.some((changedPath) => DOCS_PATH_RE.test(changedPath));
|
||||
for (const changedPath of paths) {
|
||||
reasons.push(`${changedPath}: release metadata`);
|
||||
}
|
||||
return { paths, lanes, extensionImpactFromCore: false, docsOnly: false, reasons };
|
||||
}
|
||||
|
||||
for (const changedPath of paths) {
|
||||
if (DOCS_PATH_RE.test(changedPath)) {
|
||||
lanes.docs = true;
|
||||
continue;
|
||||
}
|
||||
|
||||
hasNonDocs = true;
|
||||
|
||||
if (changedPath === "package.json" && packageJsonIsLiveDockerTooling) {
|
||||
lanes.liveDockerTooling = true;
|
||||
reasons.push(`${changedPath}: live Docker package scripts`);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (changedPath === "package.json" && packageJsonIsTooling) {
|
||||
lanes.tooling = true;
|
||||
reasons.push(`${changedPath}: package scripts`);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (LIVE_DOCKER_TOOLING_PATH_RE.test(changedPath)) {
|
||||
lanes.liveDockerTooling = true;
|
||||
reasons.push(`${changedPath}: live Docker tooling surface`);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (ROOT_GLOBAL_PATH_RE.test(changedPath)) {
|
||||
lanes.all = true;
|
||||
extensionImpactFromCore = true;
|
||||
reasons.push(`${changedPath}: root config/package surface`);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (PUBLIC_EXTENSION_CONTRACT_RE.test(changedPath)) {
|
||||
lanes.core = true;
|
||||
lanes.coreTests = true;
|
||||
lanes.extensions = true;
|
||||
lanes.extensionTests = true;
|
||||
extensionImpactFromCore = true;
|
||||
reasons.push(`${changedPath}: public core/plugin contract affects extensions`);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (EXTENSION_PATH_RE.test(changedPath)) {
|
||||
if (isChangedLaneTestPath(changedPath)) {
|
||||
lanes.extensionTests = true;
|
||||
reasons.push(`${changedPath}: extension test`);
|
||||
} else {
|
||||
lanes.extensions = true;
|
||||
lanes.extensionTests = true;
|
||||
reasons.push(`${changedPath}: extension production`);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
if (CORE_PATH_RE.test(changedPath)) {
|
||||
if (isChangedLaneTestPath(changedPath)) {
|
||||
lanes.coreTests = true;
|
||||
reasons.push(`${changedPath}: core test`);
|
||||
} else {
|
||||
lanes.core = true;
|
||||
lanes.coreTests = true;
|
||||
reasons.push(`${changedPath}: core production`);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
if (APP_PATH_RE.test(changedPath)) {
|
||||
lanes.apps = true;
|
||||
reasons.push(`${changedPath}: app surface`);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (changedPath.startsWith("test/") || changedPath.startsWith("test-fixtures/")) {
|
||||
lanes.tooling = true;
|
||||
reasons.push(`${changedPath}: root test/support surface`);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (TOOLING_PATH_RE.test(changedPath)) {
|
||||
lanes.tooling = true;
|
||||
reasons.push(`${changedPath}: tooling surface`);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (LEGACY_ROOT_ASSET_PATH_RE.test(changedPath)) {
|
||||
lanes.tooling = true;
|
||||
reasons.push(`${changedPath}: legacy root asset cleanup`);
|
||||
continue;
|
||||
}
|
||||
|
||||
lanes.all = true;
|
||||
extensionImpactFromCore = true;
|
||||
reasons.push(`${changedPath}: unknown surface; fail-safe all lanes`);
|
||||
}
|
||||
|
||||
return {
|
||||
paths,
|
||||
lanes,
|
||||
extensionImpactFromCore,
|
||||
docsOnly: lanes.docs && !hasNonDocs,
|
||||
reasons,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {{ paths: string[]; base: string; head?: string; staged?: boolean; mergeHeadFirstParent?: boolean }} params
|
||||
* @returns {ChangedLaneResult}
|
||||
*/
|
||||
/**
|
||||
* Classifies changed paths with optional package.json before/after contents.
|
||||
*/
|
||||
export function detectChangedLanesForPaths(params) {
|
||||
const base = params.staged
|
||||
? params.base
|
||||
: resolveMergeHeadDiffBase({
|
||||
base: params.base,
|
||||
head: params.head ?? "HEAD",
|
||||
maxBuffer: GIT_OUTPUT_MAX_BUFFER,
|
||||
preferFirstParent: params.mergeHeadFirstParent === true,
|
||||
});
|
||||
const packageJsonChangeKind = params.paths.includes("package.json")
|
||||
? classifyPackageJsonChangeFromGit({
|
||||
base,
|
||||
head: params.head,
|
||||
staged: params.staged,
|
||||
})
|
||||
: null;
|
||||
return detectChangedLanes(params.paths, { packageJsonChangeKind });
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {{ base: string; head?: string; includeWorktree?: boolean; cwd?: string; mergeHeadFirstParent?: boolean }} params
|
||||
* @returns {string[]}
|
||||
*/
|
||||
/**
|
||||
* Lists changed paths from git for a base/head comparison.
|
||||
*/
|
||||
export function listChangedPathsFromGit(params) {
|
||||
const head = params.head ?? "HEAD";
|
||||
const cwd = params.cwd ?? process.cwd();
|
||||
const base = resolveMergeHeadDiffBase({
|
||||
base: params.base,
|
||||
head,
|
||||
cwd,
|
||||
maxBuffer: GIT_OUTPUT_MAX_BUFFER,
|
||||
preferFirstParent: params.mergeHeadFirstParent === true,
|
||||
});
|
||||
if (!base) {
|
||||
return [];
|
||||
}
|
||||
let rangePaths;
|
||||
let noMergeBase = false;
|
||||
try {
|
||||
rangePaths = runGitNameOnlyDiff([`${base}...${head}`], cwd);
|
||||
} catch (error) {
|
||||
if (!isGitNoMergeBaseError(error)) {
|
||||
throw error;
|
||||
}
|
||||
noMergeBase = true;
|
||||
rangePaths = runGitNameOnlyDiff([`${base}..${head}`], cwd);
|
||||
}
|
||||
if (params.includeWorktree === false) {
|
||||
return rangePaths;
|
||||
}
|
||||
const worktreePaths = [
|
||||
...runGitNameOnlyDiff(["--cached", "--diff-filter=ACMRD"], cwd),
|
||||
...runGitNameOnlyDiff(["--diff-filter=ACMRD"], cwd),
|
||||
...runGitLsFiles(["--others", "--exclude-standard"], cwd),
|
||||
];
|
||||
// Raw Crabbox syncs can have unrelated synthetic refs; prefer the synced
|
||||
// worktree delta instead of turning that into an accidental whole-repo gate.
|
||||
if (
|
||||
noMergeBase &&
|
||||
process.env[RAW_SYNC_CHANGED_LANES_ENV] === "1" &&
|
||||
worktreePaths.length > 0 &&
|
||||
rangePaths.length > IMPLAUSIBLE_NO_MERGE_BASE_DIFF_PATHS
|
||||
) {
|
||||
rangePaths = [];
|
||||
}
|
||||
return [...new Set([...rangePaths, ...worktreePaths])].toSorted((left, right) =>
|
||||
left.localeCompare(right),
|
||||
);
|
||||
}
|
||||
|
||||
function runGitNameOnlyDiff(extraArgs, cwd = process.cwd()) {
|
||||
const output = execFileSync("git", ["diff", "--name-only", ...extraArgs], {
|
||||
cwd,
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
encoding: "utf8",
|
||||
maxBuffer: GIT_OUTPUT_MAX_BUFFER,
|
||||
});
|
||||
return output.split("\n").map(normalizeChangedPath).filter(Boolean);
|
||||
}
|
||||
|
||||
function isGitNoMergeBaseError(error) {
|
||||
const text = [
|
||||
error?.message,
|
||||
error?.stderr?.toString?.("utf8"),
|
||||
Array.isArray(error?.output)
|
||||
? error.output.map((value) => value?.toString?.("utf8")).join("\n")
|
||||
: "",
|
||||
].join("\n");
|
||||
return text.includes("no merge base");
|
||||
}
|
||||
|
||||
function runGitLsFiles(extraArgs, cwd = process.cwd()) {
|
||||
const output = execFileSync("git", ["ls-files", ...extraArgs], {
|
||||
cwd,
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
encoding: "utf8",
|
||||
maxBuffer: GIT_OUTPUT_MAX_BUFFER,
|
||||
});
|
||||
return output.split("\n").map(normalizeChangedPath).filter(Boolean);
|
||||
}
|
||||
|
||||
/**
|
||||
* Lists staged changed paths for pre-commit checks.
|
||||
*/
|
||||
export function listStagedChangedPaths(cwd = process.cwd()) {
|
||||
const output = execFileSync("git", ["diff", "--cached", "--name-only", "--diff-filter=ACMRD"], {
|
||||
cwd,
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
encoding: "utf8",
|
||||
maxBuffer: GIT_OUTPUT_MAX_BUFFER,
|
||||
});
|
||||
return output.split("\n").map(normalizeChangedPath).filter(Boolean);
|
||||
}
|
||||
|
||||
/**
|
||||
* Classifies package.json script-only changes from git content.
|
||||
*/
|
||||
export function classifyPackageJsonChangeFromGit(params) {
|
||||
try {
|
||||
const { before, after } = readPackageJsonBeforeAfter(params);
|
||||
if (isLiveDockerPackageScriptOnlyChange(before, after)) {
|
||||
return "liveDockerTooling";
|
||||
}
|
||||
return isPackageScriptOnlyChange(before, after) ? "tooling" : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Checks whether package scripts changed only live Docker script entries.
|
||||
*/
|
||||
export function isLiveDockerPackageScriptOnlyChange(before, after) {
|
||||
const beforePackage = JSON.parse(before);
|
||||
const afterPackage = JSON.parse(after);
|
||||
const beforeAllowed = extractLiveDockerPackageScripts(beforePackage);
|
||||
const afterAllowed = extractLiveDockerPackageScripts(afterPackage);
|
||||
const beforeStripped = stripLiveDockerPackageScripts(beforePackage);
|
||||
const afterStripped = stripLiveDockerPackageScripts(afterPackage);
|
||||
|
||||
return (
|
||||
stableJson(beforeStripped) === stableJson(afterStripped) &&
|
||||
stableJson(beforeAllowed) !== stableJson(afterAllowed)
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Checks whether package.json changes are limited to scripts.
|
||||
*/
|
||||
export function isPackageScriptOnlyChange(before, after) {
|
||||
const beforePackage = JSON.parse(before);
|
||||
const afterPackage = JSON.parse(after);
|
||||
const beforeScripts = extractPackageScripts(beforePackage);
|
||||
const afterScripts = extractPackageScripts(afterPackage);
|
||||
const beforeStripped = stripPackageScripts(beforePackage);
|
||||
const afterStripped = stripPackageScripts(afterPackage);
|
||||
|
||||
return (
|
||||
stableJson(beforeStripped) === stableJson(afterStripped) &&
|
||||
stableJson(beforeScripts) !== stableJson(afterScripts)
|
||||
);
|
||||
}
|
||||
|
||||
function readPackageJsonBeforeAfter(params) {
|
||||
const before = readGitText(params.staged ? "HEAD" : params.base, "package.json");
|
||||
if (params.staged) {
|
||||
return { before, after: readGitText("INDEX", "package.json") };
|
||||
}
|
||||
|
||||
let after = readGitText(params.head ?? "HEAD", "package.json");
|
||||
if (params.includeWorktree !== false && existsSync("package.json")) {
|
||||
const worktree = readGitText("WORKTREE", "package.json");
|
||||
if (worktree !== after) {
|
||||
after = worktree;
|
||||
}
|
||||
}
|
||||
return { before, after };
|
||||
}
|
||||
|
||||
function readGitText(ref, filePath) {
|
||||
if (ref === "WORKTREE") {
|
||||
return readFileSync(filePath, "utf8");
|
||||
}
|
||||
const spec = ref === "INDEX" ? `:${filePath}` : `${ref}:${filePath}`;
|
||||
return execFileSync("git", ["show", spec], {
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
encoding: "utf8",
|
||||
maxBuffer: 16 * 1024 * 1024,
|
||||
});
|
||||
}
|
||||
|
||||
function extractLiveDockerPackageScripts(packageJson) {
|
||||
const scripts = packageJson?.scripts;
|
||||
if (!scripts || typeof scripts !== "object" || Array.isArray(scripts)) {
|
||||
return {};
|
||||
}
|
||||
return Object.fromEntries(
|
||||
Object.entries(scripts).filter(([name]) => LIVE_DOCKER_PACKAGE_SCRIPT_RE.test(name)),
|
||||
);
|
||||
}
|
||||
|
||||
function stripLiveDockerPackageScripts(packageJson) {
|
||||
const clone = structuredClone(packageJson);
|
||||
const scripts = clone.scripts;
|
||||
if (!scripts || typeof scripts !== "object" || Array.isArray(scripts)) {
|
||||
return clone;
|
||||
}
|
||||
for (const name of Object.keys(scripts)) {
|
||||
if (LIVE_DOCKER_PACKAGE_SCRIPT_RE.test(name)) {
|
||||
delete scripts[name];
|
||||
}
|
||||
}
|
||||
return clone;
|
||||
}
|
||||
|
||||
function extractPackageScripts(packageJson) {
|
||||
const scripts = packageJson?.scripts;
|
||||
return scripts && typeof scripts === "object" && !Array.isArray(scripts) ? scripts : {};
|
||||
}
|
||||
|
||||
function stripPackageScripts(packageJson) {
|
||||
const clone = structuredClone(packageJson);
|
||||
delete clone.scripts;
|
||||
return clone;
|
||||
}
|
||||
|
||||
function stableJson(value) {
|
||||
if (Array.isArray(value)) {
|
||||
return `[${value.map(stableJson).join(",")}]`;
|
||||
}
|
||||
if (value && typeof value === "object") {
|
||||
return `{${Object.keys(value)
|
||||
.toSorted((left, right) => left.localeCompare(right))
|
||||
.map((key) => `${JSON.stringify(key)}:${stableJson(value[key])}`)
|
||||
.join(",")}}`;
|
||||
}
|
||||
return JSON.stringify(value);
|
||||
}
|
||||
|
||||
/**
|
||||
* Writes changed-lane booleans to the GitHub Actions output file.
|
||||
*/
|
||||
export function writeChangedLaneGitHubOutput(result, outputPath = process.env.GITHUB_OUTPUT) {
|
||||
if (!outputPath) {
|
||||
throw new Error("GITHUB_OUTPUT is required");
|
||||
}
|
||||
for (const [lane, enabled] of Object.entries(result.lanes)) {
|
||||
appendFileSync(outputPath, `run_${toSnakeCase(lane)}=${String(enabled)}\n`, "utf8");
|
||||
}
|
||||
appendFileSync(outputPath, `docs_only=${result.docsOnly}\n`, "utf8");
|
||||
appendFileSync(
|
||||
outputPath,
|
||||
`extension_impact_from_core=${result.extensionImpactFromCore}\n`,
|
||||
"utf8",
|
||||
);
|
||||
}
|
||||
|
||||
function toSnakeCase(value) {
|
||||
return value.replace(/[A-Z]/gu, (match) => `_${match.toLowerCase()}`);
|
||||
}
|
||||
|
||||
function parseArgs(argv) {
|
||||
const separatorIndex = argv.indexOf("--");
|
||||
const flagArgv = separatorIndex === -1 ? argv : argv.slice(0, separatorIndex);
|
||||
const explicitPaths = separatorIndex === -1 ? [] : argv.slice(separatorIndex + 1);
|
||||
const args = {
|
||||
base: "origin/main",
|
||||
head: "HEAD",
|
||||
staged: false,
|
||||
mergeHeadFirstParent: false,
|
||||
json: false,
|
||||
githubOutput: false,
|
||||
help: false,
|
||||
paths: [],
|
||||
};
|
||||
const parsed = parseFlagArgs(
|
||||
flagArgv,
|
||||
args,
|
||||
[
|
||||
stringFlag("--base", "base"),
|
||||
stringFlag("--head", "head"),
|
||||
booleanFlag("--staged", "staged"),
|
||||
booleanFlag("--merge-head-first-parent", "mergeHeadFirstParent"),
|
||||
booleanFlag("--json", "json"),
|
||||
booleanFlag("--github-output", "githubOutput"),
|
||||
booleanFlag("--help", "help"),
|
||||
booleanFlag("-h", "help"),
|
||||
],
|
||||
{
|
||||
onUnhandledArg(arg, target) {
|
||||
if (arg.startsWith("-")) {
|
||||
throw new Error(`Unknown option: ${arg}`);
|
||||
}
|
||||
target.paths.push(arg);
|
||||
return "handled";
|
||||
},
|
||||
},
|
||||
);
|
||||
parsed.paths.push(...explicitPaths);
|
||||
return parsed;
|
||||
}
|
||||
|
||||
function printUsage() {
|
||||
console.log(
|
||||
[
|
||||
"Usage: node scripts/changed-lanes.mjs [options] [-- <paths...>]",
|
||||
"",
|
||||
"Options:",
|
||||
" --base <ref> Base ref for changed paths (default: origin/main)",
|
||||
" --head <ref> Head ref for changed paths (default: HEAD)",
|
||||
" --staged Inspect staged changes",
|
||||
" --json Print JSON result",
|
||||
" --github-output Append GitHub output variables",
|
||||
" -h, --help Show this help",
|
||||
].join("\n"),
|
||||
);
|
||||
}
|
||||
|
||||
function isDirectRun() {
|
||||
return isDirectRunUrl(process.argv[1], import.meta.url);
|
||||
}
|
||||
|
||||
function printHuman(result) {
|
||||
const enabled = Object.entries(result.lanes)
|
||||
.filter(([, value]) => value)
|
||||
.map(([lane]) => lane);
|
||||
console.log(`lanes: ${enabled.length > 0 ? enabled.join(", ") : "none"}`);
|
||||
if (result.docsOnly) {
|
||||
console.log("docs-only: true");
|
||||
}
|
||||
if (result.extensionImpactFromCore) {
|
||||
console.log("extension-impact-from-core: true");
|
||||
}
|
||||
if (result.paths.length > 0) {
|
||||
console.log("paths:");
|
||||
for (const changedPath of result.paths) {
|
||||
console.log(`- ${changedPath}`);
|
||||
}
|
||||
}
|
||||
if (result.reasons.length > 0) {
|
||||
console.log("reasons:");
|
||||
for (const reason of result.reasons) {
|
||||
console.log(`- ${reason}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (isDirectRun()) {
|
||||
let args;
|
||||
try {
|
||||
args = parseArgs(process.argv.slice(2));
|
||||
} catch (error) {
|
||||
console.error(error instanceof Error ? error.message : String(error));
|
||||
process.exit(1);
|
||||
}
|
||||
if (args.help) {
|
||||
printUsage();
|
||||
process.exit(0);
|
||||
}
|
||||
const paths =
|
||||
args.paths.length > 0
|
||||
? args.paths
|
||||
: args.staged
|
||||
? listStagedChangedPaths()
|
||||
: listChangedPathsFromGit({
|
||||
base: args.base,
|
||||
head: args.head,
|
||||
mergeHeadFirstParent: args.mergeHeadFirstParent,
|
||||
});
|
||||
const result = detectChangedLanesForPaths({
|
||||
paths,
|
||||
base: args.base,
|
||||
head: args.head,
|
||||
staged: args.staged,
|
||||
mergeHeadFirstParent: args.mergeHeadFirstParent,
|
||||
});
|
||||
if (args.githubOutput) {
|
||||
writeChangedLaneGitHubOutput(result);
|
||||
}
|
||||
if (args.json) {
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
} else if (!args.githubOutput) {
|
||||
printHuman(result);
|
||||
}
|
||||
}
|
||||
91
scripts/changelog-to-html.sh
Executable file
91
scripts/changelog-to-html.sh
Executable file
@@ -0,0 +1,91 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
VERSION=${1:-}
|
||||
CHANGELOG_FILE=${2:-}
|
||||
|
||||
if [[ -z "$VERSION" ]]; then
|
||||
echo "Usage: $0 <version> [changelog_file]" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
if [[ -z "$CHANGELOG_FILE" ]]; then
|
||||
if [[ -f "$SCRIPT_DIR/../CHANGELOG.md" ]]; then
|
||||
CHANGELOG_FILE="$SCRIPT_DIR/../CHANGELOG.md"
|
||||
elif [[ -f "CHANGELOG.md" ]]; then
|
||||
CHANGELOG_FILE="CHANGELOG.md"
|
||||
elif [[ -f "../CHANGELOG.md" ]]; then
|
||||
CHANGELOG_FILE="../CHANGELOG.md"
|
||||
else
|
||||
echo "Error: Could not find CHANGELOG.md" >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ ! -f "$CHANGELOG_FILE" ]]; then
|
||||
echo "Error: Changelog file '$CHANGELOG_FILE' not found" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
extract_version_section() {
|
||||
local version=$1
|
||||
local file=$2
|
||||
awk -v version="$version" '
|
||||
BEGIN { found=0 }
|
||||
/^## / {
|
||||
if ($0 ~ "^##[[:space:]]+" version "([[:space:]].*|$)") { found=1; next }
|
||||
if (found) { exit }
|
||||
}
|
||||
found { print }
|
||||
' "$file"
|
||||
}
|
||||
|
||||
markdown_to_html() {
|
||||
local text=$1
|
||||
text=$(echo "$text" | sed 's/^##### \(.*\)$/<h5>\1<\/h5>/')
|
||||
text=$(echo "$text" | sed 's/^#### \(.*\)$/<h4>\1<\/h4>/')
|
||||
text=$(echo "$text" | sed 's/^### \(.*\)$/<h3>\1<\/h3>/')
|
||||
text=$(echo "$text" | sed 's/^## \(.*\)$/<h2>\1<\/h2>/')
|
||||
text=$(echo "$text" | sed 's/^- \*\*\([^*]*\)\*\*\(.*\)$/<li><strong>\1<\/strong>\2<\/li>/')
|
||||
text=$(echo "$text" | sed 's/^- \([^*].*\)$/<li>\1<\/li>/')
|
||||
text=$(echo "$text" | sed 's/\*\*\([^*]*\)\*\*/<strong>\1<\/strong>/g')
|
||||
text=$(echo "$text" | sed 's/`\([^`]*\)`/<code>\1<\/code>/g')
|
||||
text=$(echo "$text" | sed 's/\[\([^]]*\)\](\([^)]*\))/<a href="\2">\1<\/a>/g')
|
||||
echo "$text"
|
||||
}
|
||||
|
||||
version_content=$(extract_version_section "$VERSION" "$CHANGELOG_FILE")
|
||||
if [[ -z "$version_content" ]]; then
|
||||
echo "<h2>OpenClaw $VERSION</h2>"
|
||||
echo "<p>Latest OpenClaw update.</p>"
|
||||
echo "<p><a href=\"https://github.com/openclaw/openclaw/blob/main/CHANGELOG.md\">View full changelog</a></p>"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "<h2>OpenClaw $VERSION</h2>"
|
||||
|
||||
in_list=false
|
||||
while IFS= read -r line; do
|
||||
if [[ "$line" =~ ^- ]]; then
|
||||
if [[ "$in_list" == false ]]; then
|
||||
echo "<ul>"
|
||||
in_list=true
|
||||
fi
|
||||
markdown_to_html "$line"
|
||||
else
|
||||
if [[ "$in_list" == true ]]; then
|
||||
echo "</ul>"
|
||||
in_list=false
|
||||
fi
|
||||
if [[ -n "$line" ]]; then
|
||||
markdown_to_html "$line"
|
||||
fi
|
||||
fi
|
||||
done <<< "$version_content"
|
||||
|
||||
if [[ "$in_list" == true ]]; then
|
||||
echo "</ul>"
|
||||
fi
|
||||
|
||||
echo "<p><a href=\"https://github.com/openclaw/openclaw/blob/main/CHANGELOG.md\">View full changelog</a></p>"
|
||||
245
scripts/check-architecture-smells.mjs
Normal file
245
scripts/check-architecture-smells.mjs
Normal file
@@ -0,0 +1,245 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Finds core/plugin architecture boundary smells in TypeScript sources.
|
||||
import { promises as fs } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { BUNDLED_PLUGIN_PATH_PREFIX } from "./lib/bundled-plugin-paths.mjs";
|
||||
import {
|
||||
collectModuleReferencesFromSource,
|
||||
normalizeRepoPath,
|
||||
resolveRepoSpecifier,
|
||||
writeLine,
|
||||
} from "./lib/guard-inventory-utils.mjs";
|
||||
import { mapWithConcurrency } from "./lib/source-file-scan-cache.mjs";
|
||||
import {
|
||||
collectTypeScriptFilesFromRoots,
|
||||
resolveSourceRoots,
|
||||
runAsScript,
|
||||
} from "./lib/ts-guard-utils.mjs";
|
||||
|
||||
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const scanRoots = resolveSourceRoots(repoRoot, ["src/plugin-sdk", "src/plugins/runtime"]);
|
||||
let architectureSmellsPromise;
|
||||
|
||||
function compareEntries(left, right) {
|
||||
return (
|
||||
left.category.localeCompare(right.category) ||
|
||||
left.file.localeCompare(right.file) ||
|
||||
left.line - right.line ||
|
||||
left.kind.localeCompare(right.kind) ||
|
||||
left.specifier.localeCompare(right.specifier) ||
|
||||
left.reason.localeCompare(right.reason)
|
||||
);
|
||||
}
|
||||
|
||||
function pushEntry(entries, entry) {
|
||||
entries.push(entry);
|
||||
}
|
||||
|
||||
function scanPluginSdkExtensionFacadeSmells(source, filePath) {
|
||||
const relativeFile = normalizeRepoPath(repoRoot, filePath);
|
||||
if (!relativeFile.startsWith("src/plugin-sdk/")) {
|
||||
return [];
|
||||
}
|
||||
|
||||
const entries = [];
|
||||
|
||||
for (const { kind, line, specifier } of collectModuleReferencesFromSource(source)) {
|
||||
if (kind !== "export") {
|
||||
continue;
|
||||
}
|
||||
const resolvedPath = resolveRepoSpecifier(repoRoot, specifier, filePath);
|
||||
if (!resolvedPath?.startsWith(BUNDLED_PLUGIN_PATH_PREFIX)) {
|
||||
continue;
|
||||
}
|
||||
pushEntry(entries, {
|
||||
category: "plugin-sdk-extension-facade",
|
||||
file: relativeFile,
|
||||
line,
|
||||
kind,
|
||||
specifier,
|
||||
resolvedPath,
|
||||
reason: "plugin-sdk public surface re-exports extension-owned implementation",
|
||||
});
|
||||
}
|
||||
return entries;
|
||||
}
|
||||
|
||||
function scanRuntimeTypeImplementationSmells(source, filePath) {
|
||||
const relativeFile = normalizeRepoPath(repoRoot, filePath);
|
||||
if (!/^src\/plugins\/runtime\/types(?:-[^/]+)?\.ts$/.test(relativeFile)) {
|
||||
return [];
|
||||
}
|
||||
|
||||
const entries = [];
|
||||
|
||||
for (const { kind, line, specifier } of collectModuleReferencesFromSource(source)) {
|
||||
if (kind !== "dynamic-import") {
|
||||
continue;
|
||||
}
|
||||
const resolvedPath = resolveRepoSpecifier(repoRoot, specifier, filePath);
|
||||
if (
|
||||
resolvedPath &&
|
||||
(/^src\/plugins\/runtime\/runtime-[^/]+\.ts$/.test(resolvedPath) ||
|
||||
/^extensions\/[^/]+\/runtime-api\.[^/]+$/.test(resolvedPath))
|
||||
) {
|
||||
pushEntry(entries, {
|
||||
category: "runtime-type-implementation-edge",
|
||||
file: relativeFile,
|
||||
line,
|
||||
kind: "import-type",
|
||||
specifier,
|
||||
resolvedPath,
|
||||
reason: "runtime type file references implementation shim directly",
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return entries;
|
||||
}
|
||||
|
||||
function scanRuntimeServiceLocatorSmells(source, filePath) {
|
||||
const relativeFile = normalizeRepoPath(repoRoot, filePath);
|
||||
if (
|
||||
!relativeFile.startsWith("src/plugin-sdk/") &&
|
||||
!relativeFile.startsWith("src/plugins/runtime/")
|
||||
) {
|
||||
return [];
|
||||
}
|
||||
|
||||
const entries = [];
|
||||
const exportedNames = new Set();
|
||||
const runtimeStoreCalls = [];
|
||||
const mutableStateNodes = [];
|
||||
|
||||
const lines = source.split(/\r?\n/);
|
||||
for (const [index, line] of lines.entries()) {
|
||||
const lineNumber = index + 1;
|
||||
const exportedFunction = line.match(/^\s*export\s+function\s+([A-Za-z_$][\w$]*)/);
|
||||
if (exportedFunction) {
|
||||
exportedNames.add(exportedFunction[1]);
|
||||
}
|
||||
const exportedVariable = line.match(/^\s*export\s+(?:const|let|var)\s+([A-Za-z_$][\w$]*)/);
|
||||
if (exportedVariable) {
|
||||
exportedNames.add(exportedVariable[1]);
|
||||
}
|
||||
for (const mutableMatch of line.matchAll(/^\s*let\s+([A-Za-z_$][\w$]*)/g)) {
|
||||
mutableStateNodes.push({ line: lineNumber, text: mutableMatch[1] });
|
||||
}
|
||||
if (line.includes("createPluginRuntimeStore")) {
|
||||
runtimeStoreCalls.push({ line: lineNumber });
|
||||
}
|
||||
}
|
||||
|
||||
const getterNames = [...exportedNames].filter((name) => /^get[A-Z]/.test(name));
|
||||
const setterNames = [...exportedNames].filter((name) => /^set[A-Z]/.test(name));
|
||||
|
||||
if (runtimeStoreCalls.length > 0 && getterNames.length > 0 && setterNames.length > 0) {
|
||||
for (const callNode of runtimeStoreCalls) {
|
||||
pushEntry(entries, {
|
||||
category: "runtime-service-locator",
|
||||
file: relativeFile,
|
||||
line: callNode.line,
|
||||
kind: "runtime-store",
|
||||
specifier: "createPluginRuntimeStore",
|
||||
resolvedPath: relativeFile,
|
||||
reason: `exports paired runtime accessors (${getterNames.join(", ")} / ${setterNames.join(", ")}) over module-global store state`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (mutableStateNodes.length > 0 && getterNames.length > 0 && setterNames.length > 0) {
|
||||
for (const identifier of mutableStateNodes) {
|
||||
pushEntry(entries, {
|
||||
category: "runtime-service-locator",
|
||||
file: relativeFile,
|
||||
line: identifier.line,
|
||||
kind: "mutable-state",
|
||||
specifier: identifier.text,
|
||||
resolvedPath: relativeFile,
|
||||
reason: `module-global mutable state backs exported runtime accessors (${getterNames.join(", ")} / ${setterNames.join(", ")})`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return entries;
|
||||
}
|
||||
|
||||
/**
|
||||
* Collects architecture smell findings from the configured source roots.
|
||||
*/
|
||||
export async function collectArchitectureSmells() {
|
||||
if (!architectureSmellsPromise) {
|
||||
architectureSmellsPromise = (async () => {
|
||||
const files = (await collectTypeScriptFilesFromRoots(scanRoots)).toSorted((left, right) =>
|
||||
normalizeRepoPath(repoRoot, left).localeCompare(normalizeRepoPath(repoRoot, right)),
|
||||
);
|
||||
const entriesByFile = await mapWithConcurrency(files, undefined, async (filePath) => {
|
||||
const source = await fs.readFile(filePath, "utf8");
|
||||
const entries = scanPluginSdkExtensionFacadeSmells(source, filePath);
|
||||
entries.push(...scanRuntimeTypeImplementationSmells(source, filePath));
|
||||
entries.push(...scanRuntimeServiceLocatorSmells(source, filePath));
|
||||
return entries;
|
||||
});
|
||||
return entriesByFile.flat().toSorted(compareEntries);
|
||||
})();
|
||||
try {
|
||||
return await architectureSmellsPromise;
|
||||
} catch (error) {
|
||||
architectureSmellsPromise = undefined;
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
return await architectureSmellsPromise;
|
||||
}
|
||||
|
||||
function formatInventoryHuman(inventory) {
|
||||
if (inventory.length === 0) {
|
||||
return "No architecture smells found for the configured checks.";
|
||||
}
|
||||
|
||||
const lines = ["Architecture smell inventory:"];
|
||||
let activeCategory = "";
|
||||
let activeFile = "";
|
||||
for (const entry of inventory) {
|
||||
if (entry.category !== activeCategory) {
|
||||
activeCategory = entry.category;
|
||||
activeFile = "";
|
||||
lines.push(entry.category);
|
||||
}
|
||||
if (entry.file !== activeFile) {
|
||||
activeFile = entry.file;
|
||||
lines.push(` ${activeFile}`);
|
||||
}
|
||||
lines.push(` - line ${entry.line} [${entry.kind}] ${entry.reason}`);
|
||||
lines.push(` specifier: ${entry.specifier}`);
|
||||
lines.push(` resolved: ${entry.resolvedPath}`);
|
||||
}
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
async function runArchitectureSmellsCheck(argv, io) {
|
||||
const args = argv ?? process.argv.slice(2);
|
||||
const streams = io ?? { stdout: process.stdout, stderr: process.stderr };
|
||||
const json = args.includes("--json");
|
||||
const inventory = await collectArchitectureSmells();
|
||||
|
||||
if (json) {
|
||||
writeLine(streams.stdout, JSON.stringify(inventory, null, 2));
|
||||
return 0;
|
||||
}
|
||||
|
||||
writeLine(streams.stdout, formatInventoryHuman(inventory));
|
||||
writeLine(streams.stdout, `${inventory.length} smell${inventory.length === 1 ? "" : "s"} found.`);
|
||||
return 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the architecture smell check and writes human/JSON output.
|
||||
*/
|
||||
export async function main(argv, io) {
|
||||
return await runArchitectureSmellsCheck(argv, io);
|
||||
}
|
||||
|
||||
runAsScript(import.meta.url, main);
|
||||
824
scripts/check-changed.mjs
Normal file
824
scripts/check-changed.mjs
Normal file
@@ -0,0 +1,824 @@
|
||||
// Runs the changed-file check lanes selected by `scripts/changed-lanes.mjs`.
|
||||
import {
|
||||
accessSync,
|
||||
chmodSync,
|
||||
constants,
|
||||
existsSync,
|
||||
mkdtempSync,
|
||||
rmSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { performance } from "node:perf_hooks";
|
||||
import {
|
||||
detectChangedLanesForPaths,
|
||||
isChangedLaneTestPath,
|
||||
listChangedPathsFromGit,
|
||||
listStagedChangedPaths,
|
||||
normalizeChangedPath,
|
||||
} from "./changed-lanes.mjs";
|
||||
import { shrinkwrapPackageDirsForChangedPaths } from "./generate-npm-shrinkwrap.mjs";
|
||||
import { booleanFlag, parseFlagArgs, stringFlag } from "./lib/arg-utils.mjs";
|
||||
import { printTimingSummary } from "./lib/check-timing-summary.mjs";
|
||||
import { isDirectRunUrl } from "./lib/direct-run.mjs";
|
||||
import {
|
||||
acquireLocalHeavyCheckLockSync,
|
||||
resolveLocalHeavyCheckEnv,
|
||||
} from "./lib/local-heavy-check-runtime.mjs";
|
||||
import { runManagedCommand } from "./lib/managed-child-process.mjs";
|
||||
import { createSparseTsgoSkipEnv } from "./lib/tsgo-sparse-guard.mjs";
|
||||
|
||||
const LIVE_DOCKER_AUTH_SHELL_TARGETS = [
|
||||
"scripts/lib/live-docker-auth.sh",
|
||||
"scripts/test-live-acp-bind-docker.sh",
|
||||
"scripts/test-live-cli-backend-docker.sh",
|
||||
"scripts/test-live-codex-harness-docker.sh",
|
||||
"scripts/test-live-gateway-models-docker.sh",
|
||||
"scripts/test-live-models-docker.sh",
|
||||
"scripts/test-live-subagent-announce-docker.sh",
|
||||
];
|
||||
const SHRINKWRAP_POLICY_PATH_RE =
|
||||
/^(?:npm-shrinkwrap\.json|package\.json|pnpm-lock\.yaml|pnpm-workspace\.yaml|scripts\/generate-npm-shrinkwrap\.mjs|extensions\/[^/]+\/(?:package\.json|npm-shrinkwrap\.json))$/u;
|
||||
const PROMPT_SNAPSHOT_CHECK_PATH_RE =
|
||||
/^(?:scripts\/(?:generate-prompt-snapshots\.ts|prompt-snapshot-files\.ts|sync-codex-model-prompt-fixture\.ts)|test\/helpers\/agents\/(?:happy-path-prompt-snapshots|prompt-snapshot-paths)\.ts|test\/fixtures\/agents\/prompt-snapshots\/.+)$/u;
|
||||
const PROMPT_SNAPSHOT_OWNER_TEST_PATH_RE =
|
||||
/^(?:scripts\/(?:generate-prompt-snapshots\.ts|prompt-snapshot-files\.ts|sync-codex-model-prompt-fixture\.ts)|test\/helpers\/agents\/(?:happy-path-prompt-snapshots|prompt-snapshot-paths)\.ts|test\/fixtures\/agents\/prompt-snapshots\/codex-model-catalog\/.+)$/u;
|
||||
const RUNTIME_SIDECAR_BASELINE_PATH_RE =
|
||||
/^(?:scripts\/generate-runtime-sidecar-paths-baseline\.ts|scripts\/lib\/bundled-runtime-sidecar-paths\.json|src\/plugins\/runtime-sidecar-paths(?:-baseline)?\.ts)$/u;
|
||||
const CANVAS_A2UI_NATIVE_RESOURCE_PATH_RE =
|
||||
/^(?:pnpm-lock\.yaml$|apps\/shared\/OpenClawKit\/Sources\/OpenClawKit\/Resources\/CanvasA2UI\/|extensions\/canvas\/(?:package\.json$|scripts\/bundle-a2ui\.mjs$|src\/host\/a2ui(?:\/(?:index\.html|a2ui\.bundle\.js|\.bundle\.hash)$|-app\/))|scripts\/(?:bundle-a2ui|sync-native-a2ui)\.mjs$)/u;
|
||||
const CORE_OXLINT_TS_CONFIG = "config/tsconfig/oxlint.core.json";
|
||||
const EXTENSIONS_OXLINT_TS_CONFIG = "config/tsconfig/oxlint.extensions.json";
|
||||
const SCRIPTS_OXLINT_TS_CONFIG = "config/tsconfig/oxlint.scripts.json";
|
||||
const TARGETED_LINT_PATH_LIMIT = 8;
|
||||
const LINTABLE_CORE_PATH_RE = /^(?:src|ui|packages)\/.+\.[cm]?[jt]sx?$/u;
|
||||
const LINTABLE_EXTENSION_PATH_RE = /^extensions\/[^/]+\/.+\.[cm]?[jt]sx?$/u;
|
||||
const LINTABLE_SCRIPT_PATH_RE = /^scripts\/.+\.[cm]?[jt]sx?$/u;
|
||||
const MARKDOWN_LINT_OPTIMIZATION_NEUTRAL_PATH_RE = /^(?:docs\/|README\.md$|.*\.mdx?$)/u;
|
||||
const CORE_LINT_OPTIMIZATION_NEUTRAL_PATH_RE =
|
||||
/^(?:scripts|test\/scripts)\/|^\.github\/workflows\/ci\.yml$/u;
|
||||
const EXTENSION_LINT_OPTIMIZATION_NEUTRAL_PATH_RE =
|
||||
/^(?:test\/scripts\/|\.github\/workflows\/ci\.yml$)/u;
|
||||
const SCRIPT_LINT_OPTIMIZATION_NEUTRAL_PATH_RE =
|
||||
/^(?:test\/scripts\/|\.github\/workflows\/ci\.yml$)/u;
|
||||
const ANDROID_VERSION_SYNC_PATHS = new Set([
|
||||
"apps/android/CHANGELOG.md",
|
||||
"apps/android/Config/Version.properties",
|
||||
"apps/android/fastlane/metadata/android/en-US/release_notes.txt",
|
||||
"apps/android/version.json",
|
||||
]);
|
||||
const MACOS_APP_CI_PATH_RE =
|
||||
/^(?:apps\/(?:macos|macos-mlx-tts|shared|swabble)\/|Swabble\/|scripts\/(?:codesign-mac-app|create-dmg|notarize-mac-artifact|package-mac-app|package-mac-dist)\.sh$|scripts\/lib\/(?:plistbuddy|swift-toolchain)\.sh$|test\/scripts\/(?:codesign-mac-app|create-dmg|notarize-mac-artifact|package-mac-app|package-mac-dist)\.test\.ts$)/u;
|
||||
let corepackPnpmShimDir;
|
||||
let corepackPnpmShimCleanupRegistered = false;
|
||||
|
||||
export function createChangedCheckChildEnv(baseEnv = process.env) {
|
||||
const resolvedBaseEnv = resolveLocalHeavyCheckEnv(baseEnv);
|
||||
return {
|
||||
...resolvedBaseEnv,
|
||||
OPENCLAW_OXLINT_SKIP_LOCK: "1",
|
||||
OPENCLAW_TEST_HEAVY_CHECK_LOCK_HELD: "1",
|
||||
OPENCLAW_TSGO_HEAVY_CHECK_LOCK_HELD: "1",
|
||||
};
|
||||
}
|
||||
|
||||
function isTruthyEnvFlag(value) {
|
||||
const normalized = String(value ?? "")
|
||||
.trim()
|
||||
.toLowerCase();
|
||||
return normalized !== "" && normalized !== "0" && normalized !== "false" && normalized !== "no";
|
||||
}
|
||||
|
||||
function hasAndroidVersionSyncPath(paths) {
|
||||
return paths.some((changedPath) =>
|
||||
ANDROID_VERSION_SYNC_PATHS.has(normalizeChangedPath(changedPath)),
|
||||
);
|
||||
}
|
||||
|
||||
function hasMacosAppCiPath(paths) {
|
||||
return paths.some((changedPath) => MACOS_APP_CI_PATH_RE.test(normalizeChangedPath(changedPath)));
|
||||
}
|
||||
|
||||
function executableExistsOnPath(command, env = process.env) {
|
||||
const pathValue = env.PATH ?? env.Path ?? "";
|
||||
const pathExts =
|
||||
process.platform === "win32" ? (env.PATHEXT ?? ".EXE;.CMD;.BAT;.COM").split(";") : [""];
|
||||
for (const searchPath of pathValue.split(path.delimiter)) {
|
||||
if (!searchPath) {
|
||||
continue;
|
||||
}
|
||||
for (const ext of pathExts) {
|
||||
try {
|
||||
accessSync(path.join(searchPath, `${command}${ext}`), constants.X_OK);
|
||||
return true;
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
export function shouldSkipAppLintForMissingSwiftlint(options = {}) {
|
||||
const env = options.env ?? process.env;
|
||||
const platform = options.platform ?? process.platform;
|
||||
const swiftlintAvailable = options.swiftlintAvailable ?? executableExistsOnPath("swiftlint", env);
|
||||
return platform !== "darwin" && !swiftlintAvailable;
|
||||
}
|
||||
|
||||
export function shouldDelegateChangedCheckToCrabbox(argv = [], env = process.env) {
|
||||
if (isTruthyEnvFlag(env.OPENCLAW_CHECK_CHANGED_REMOTE_CHILD)) {
|
||||
return false;
|
||||
}
|
||||
if (isTruthyEnvFlag(env.CI) || isTruthyEnvFlag(env.GITHUB_ACTIONS)) {
|
||||
return false;
|
||||
}
|
||||
if (argv.includes("--dry-run")) {
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
export function buildChangedCheckCrabboxArgs(argv = [], options = {}) {
|
||||
const delegatedArgv = buildDelegatedChangedCheckArgv(argv, options);
|
||||
return [
|
||||
"crabbox:run",
|
||||
"--",
|
||||
"--provider",
|
||||
"blacksmith-testbox",
|
||||
"--blacksmith-org",
|
||||
"openclaw",
|
||||
"--blacksmith-workflow",
|
||||
".github/workflows/ci-check-testbox.yml",
|
||||
"--blacksmith-job",
|
||||
"check",
|
||||
"--blacksmith-ref",
|
||||
"main",
|
||||
"--idle-timeout",
|
||||
"90m",
|
||||
"--ttl",
|
||||
"240m",
|
||||
"--timing-json",
|
||||
"--",
|
||||
"env",
|
||||
"OPENCLAW_CHECK_CHANGED_REMOTE_CHILD=1",
|
||||
"OPENCLAW_CHANGED_LANES_RAW_SYNC=1",
|
||||
"CI=1",
|
||||
"PNPM_CONFIG_VERIFY_DEPS_BEFORE_RUN=false",
|
||||
"corepack",
|
||||
"pnpm",
|
||||
"check:changed",
|
||||
...delegatedArgv,
|
||||
];
|
||||
}
|
||||
|
||||
function buildDelegatedChangedCheckArgv(argv, options = {}) {
|
||||
const args = parseArgs(argv);
|
||||
if (!args.staged || args.paths.length > 0) {
|
||||
return argv;
|
||||
}
|
||||
const stagedPaths = listStagedChangedPaths(options.cwd);
|
||||
const next = [];
|
||||
if (args.timed) {
|
||||
next.push("--timed");
|
||||
}
|
||||
if (stagedPaths.length === 0) {
|
||||
next.push("--no-changes");
|
||||
return next;
|
||||
}
|
||||
next.push("--base", "HEAD", "--head", "HEAD");
|
||||
next.push("--", ...stagedPaths);
|
||||
return next;
|
||||
}
|
||||
|
||||
export function shouldRunShrinkwrapGuard(paths) {
|
||||
return paths.some((changedPath) => SHRINKWRAP_POLICY_PATH_RE.test(changedPath));
|
||||
}
|
||||
|
||||
export function shouldRunPromptSnapshotCheck(paths) {
|
||||
return paths.some((changedPath) => PROMPT_SNAPSHOT_CHECK_PATH_RE.test(changedPath));
|
||||
}
|
||||
|
||||
export function shouldRunPromptSnapshotOwnerTest(paths) {
|
||||
return paths.some((changedPath) => PROMPT_SNAPSHOT_OWNER_TEST_PATH_RE.test(changedPath));
|
||||
}
|
||||
|
||||
export function shouldRunRuntimeSidecarBaselineCheck(paths) {
|
||||
return paths.some((changedPath) => RUNTIME_SIDECAR_BASELINE_PATH_RE.test(changedPath));
|
||||
}
|
||||
|
||||
export function shouldRunCanvasA2uiNativeResourceCheck(paths) {
|
||||
return paths.some((changedPath) =>
|
||||
CANVAS_A2UI_NATIVE_RESOURCE_PATH_RE.test(normalizeChangedPath(changedPath)),
|
||||
);
|
||||
}
|
||||
|
||||
export function shouldRunAppcastOwnerTest(paths) {
|
||||
return paths.some((changedPath) => normalizeChangedPath(changedPath) === "appcast.xml");
|
||||
}
|
||||
|
||||
export function shouldRunTestTempCreationReport(paths) {
|
||||
return paths.some((changedPath) => isChangedLaneTestPath(changedPath));
|
||||
}
|
||||
|
||||
export function createShrinkwrapGuardCommand(paths) {
|
||||
if (!shouldRunShrinkwrapGuard(paths)) {
|
||||
return null;
|
||||
}
|
||||
const packageDirs = shrinkwrapPackageDirsForChangedPaths(paths);
|
||||
if (packageDirs.length === 0) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
name:
|
||||
packageDirs.length === 1
|
||||
? "npm shrinkwrap guard"
|
||||
: `npm shrinkwrap guard (${packageDirs.length} packages)`,
|
||||
bin: "node",
|
||||
args: [
|
||||
"scripts/generate-npm-shrinkwrap.mjs",
|
||||
"--check",
|
||||
...packageDirs.flatMap((packageDir) => ["--package-dir", packageDir]),
|
||||
],
|
||||
};
|
||||
}
|
||||
|
||||
export async function runChangedCheckViaCrabbox(argv = [], env = process.env) {
|
||||
console.error("[check:changed] delegating to Blacksmith Testbox via `pnpm crabbox:run`.");
|
||||
return await runManagedCommand({
|
||||
bin: "pnpm",
|
||||
args: buildChangedCheckCrabboxArgs(argv),
|
||||
env,
|
||||
});
|
||||
}
|
||||
|
||||
export function createChangedCheckPlan(result, options = {}) {
|
||||
const commands = [];
|
||||
const baseEnv = createChangedCheckChildEnv(options.env ?? process.env);
|
||||
const add = (name, args, env) => {
|
||||
if (!commands.some((command) => command.name === name && sameArgs(command.args, args))) {
|
||||
commands.push({ name, args, ...(env ? { env } : {}) });
|
||||
}
|
||||
};
|
||||
const addCommand = (name, bin, args, env) => {
|
||||
if (
|
||||
!commands.some(
|
||||
(command) => command.name === name && command.bin === bin && sameArgs(command.args, args),
|
||||
)
|
||||
) {
|
||||
commands.push({ name, bin, args, ...(env ? { env } : {}) });
|
||||
}
|
||||
};
|
||||
const addTypecheck = (name, args) => add(name, args, createSparseTsgoSkipEnv(baseEnv));
|
||||
const addLint = (name, args) => add(name, args, baseEnv);
|
||||
const addTestTempCreationReport = () => {
|
||||
if (!shouldRunTestTempCreationReport(result.paths)) {
|
||||
return;
|
||||
}
|
||||
addCommand(
|
||||
"test temp creation report (warning-only)",
|
||||
"node",
|
||||
[
|
||||
"scripts/report-test-temp-creations.mjs",
|
||||
...(options.staged
|
||||
? ["--staged"]
|
||||
: ["--base", options.base ?? "origin/main", "--head", options.head ?? "HEAD"]),
|
||||
],
|
||||
baseEnv,
|
||||
);
|
||||
};
|
||||
|
||||
add("conflict markers", ["check:no-conflict-markers"]);
|
||||
add("changelog attributions", ["check:changelog-attributions"]);
|
||||
add("guarded extension wildcard re-exports", ["lint:extensions:no-guarded-wildcard-reexports"]);
|
||||
add("plugin-sdk wildcard re-exports", ["lint:extensions:no-plugin-sdk-wildcard-reexports"]);
|
||||
add("duplicate scan target coverage", ["dup:check:coverage"]);
|
||||
add("dependency pin guard", ["deps:pins:check"]);
|
||||
const shrinkwrapGuardCommand = createShrinkwrapGuardCommand(result.paths);
|
||||
if (shrinkwrapGuardCommand) {
|
||||
addCommand(
|
||||
shrinkwrapGuardCommand.name,
|
||||
shrinkwrapGuardCommand.bin,
|
||||
shrinkwrapGuardCommand.args,
|
||||
baseEnv,
|
||||
);
|
||||
}
|
||||
if (shouldRunPromptSnapshotCheck(result.paths)) {
|
||||
add("prompt snapshot drift", ["prompt:snapshots:check"]);
|
||||
}
|
||||
if (shouldRunPromptSnapshotOwnerTest(result.paths)) {
|
||||
add(
|
||||
"prompt snapshot owner test",
|
||||
["test:serial", "test/scripts/prompt-snapshots.test.ts"],
|
||||
baseEnv,
|
||||
);
|
||||
}
|
||||
if (shouldRunRuntimeSidecarBaselineCheck(result.paths)) {
|
||||
add("runtime sidecar baseline", ["runtime-sidecars:check"]);
|
||||
add(
|
||||
"runtime sidecar owner test",
|
||||
["test:serial", "src/plugins/bundled-plugin-metadata.test.ts"],
|
||||
baseEnv,
|
||||
);
|
||||
}
|
||||
if (shouldRunCanvasA2uiNativeResourceCheck(result.paths)) {
|
||||
addCommand(
|
||||
"Canvas A2UI native resource sync",
|
||||
"node",
|
||||
["scripts/sync-native-a2ui.mjs", "--check"],
|
||||
baseEnv,
|
||||
);
|
||||
}
|
||||
if (shouldRunAppcastOwnerTest(result.paths)) {
|
||||
add(
|
||||
"appcast owner tests",
|
||||
["test:serial", "test/appcast.test.ts", "test/scripts/make-appcast.test.ts"],
|
||||
baseEnv,
|
||||
);
|
||||
}
|
||||
add("package patch guard", ["deps:patches:check"]);
|
||||
|
||||
if (result.docsOnly) {
|
||||
return {
|
||||
commands,
|
||||
summary: "docs-only",
|
||||
};
|
||||
}
|
||||
|
||||
addTestTempCreationReport();
|
||||
|
||||
const lanes = result.lanes;
|
||||
const runAll = lanes.all;
|
||||
const shouldRunAndroidVersionSync = hasAndroidVersionSyncPath(result.paths);
|
||||
|
||||
if (lanes.releaseMetadata) {
|
||||
add("release metadata guard", [
|
||||
"release-metadata:check",
|
||||
...(options.staged
|
||||
? ["--staged"]
|
||||
: ["--base", options.base ?? "origin/main", "--head", options.head ?? "HEAD"]),
|
||||
]);
|
||||
add("Android version sync", ["android:version:check"]);
|
||||
add("iOS version sync", ["ios:version:check"]);
|
||||
add("config schema baseline", ["config:schema:check"]);
|
||||
add("config docs baseline", ["config:docs:check"]);
|
||||
add("root dependency ownership", ["deps:root-ownership:check"]);
|
||||
return {
|
||||
commands,
|
||||
summary: "release metadata",
|
||||
};
|
||||
}
|
||||
|
||||
if (shouldRunAndroidVersionSync) {
|
||||
add("Android version sync", ["android:version:check"]);
|
||||
}
|
||||
|
||||
if (runAll) {
|
||||
add("database-first legacy-store guard", ["check:database-first-legacy-stores"]);
|
||||
add("media download helper guard", ["check:media-download-helpers"]);
|
||||
add("runtime sidecar loader guard", ["check:runtime-sidecar-loaders"]);
|
||||
addTypecheck("typecheck all", ["tsgo:all"]);
|
||||
addLint("lint", ["lint"]);
|
||||
add("runtime import cycles", ["check:import-cycles"]);
|
||||
return {
|
||||
commands,
|
||||
summary: "all",
|
||||
};
|
||||
}
|
||||
|
||||
if (lanes.core) {
|
||||
addTypecheck("typecheck core", ["tsgo:core"]);
|
||||
}
|
||||
if (lanes.coreTests) {
|
||||
addTypecheck("typecheck core tests", ["tsgo:core:test"]);
|
||||
}
|
||||
if (lanes.extensions) {
|
||||
addTypecheck("typecheck extensions", ["tsgo:extensions"]);
|
||||
}
|
||||
if (lanes.extensionTests) {
|
||||
addTypecheck("typecheck extension tests", ["tsgo:extensions:test"]);
|
||||
}
|
||||
|
||||
if (lanes.core || lanes.coreTests) {
|
||||
const coreLintCommand = createTargetedCoreLintCommand(result.paths, baseEnv);
|
||||
if (coreLintCommand) {
|
||||
addCommand(
|
||||
coreLintCommand.name,
|
||||
coreLintCommand.bin,
|
||||
coreLintCommand.args,
|
||||
coreLintCommand.env,
|
||||
);
|
||||
} else {
|
||||
addLint("lint core", ["lint:core"]);
|
||||
}
|
||||
}
|
||||
if (
|
||||
lanes.liveDockerTooling &&
|
||||
result.paths.some((changedPath) => changedPath.startsWith("src/"))
|
||||
) {
|
||||
addTypecheck("typecheck core tests", ["tsgo:core:test"]);
|
||||
addLint("lint core", ["lint:core"]);
|
||||
}
|
||||
if (lanes.extensions || lanes.extensionTests) {
|
||||
const extensionLintCommand = createTargetedExtensionLintCommand(result.paths, baseEnv);
|
||||
if (extensionLintCommand) {
|
||||
addCommand(
|
||||
extensionLintCommand.name,
|
||||
extensionLintCommand.bin,
|
||||
extensionLintCommand.args,
|
||||
extensionLintCommand.env,
|
||||
);
|
||||
} else {
|
||||
addLint("lint extensions", ["lint:extensions"]);
|
||||
}
|
||||
}
|
||||
if (lanes.tooling || lanes.liveDockerTooling) {
|
||||
const scriptLintCommand = createTargetedScriptLintCommand(result.paths, baseEnv);
|
||||
if (scriptLintCommand) {
|
||||
addLint("lint docker-e2e", ["lint:docker-e2e"]);
|
||||
addLint("raw HTTP/2 import guard", ["lint:tmp:no-raw-http2-imports"]);
|
||||
addCommand(
|
||||
scriptLintCommand.name,
|
||||
scriptLintCommand.bin,
|
||||
scriptLintCommand.args,
|
||||
scriptLintCommand.env,
|
||||
);
|
||||
} else {
|
||||
addLint("lint scripts", ["lint:scripts"]);
|
||||
}
|
||||
}
|
||||
if (lanes.apps && shouldSkipAppLintForMissingSwiftlint({ ...options, env: baseEnv })) {
|
||||
addCommand(
|
||||
"lint apps (swiftlint unavailable on this host)",
|
||||
"node",
|
||||
[
|
||||
"-e",
|
||||
"console.error('[check:changed] Swift app lint skipped: swiftlint is unavailable on this non-macOS host; macOS CI owns SwiftLint coverage.')",
|
||||
],
|
||||
baseEnv,
|
||||
);
|
||||
} else if (lanes.apps) {
|
||||
addLint("lint apps", ["lint:apps"]);
|
||||
}
|
||||
if (hasMacosAppCiPath(result.paths)) {
|
||||
add("macOS app CI tests", ["test:macos:ci"], baseEnv);
|
||||
}
|
||||
|
||||
if (lanes.core || lanes.extensions) {
|
||||
add("database-first legacy-store guard", ["check:database-first-legacy-stores"]);
|
||||
add("media download helper guard", ["check:media-download-helpers"]);
|
||||
add("runtime sidecar loader guard", ["check:runtime-sidecar-loaders"]);
|
||||
add("runtime import cycles", ["check:import-cycles"]);
|
||||
}
|
||||
if (lanes.core) {
|
||||
add("webhook body guard", ["lint:webhook:no-low-level-body-read"]);
|
||||
add("pairing store guard", ["lint:auth:no-pairing-store-group"]);
|
||||
add("pairing account guard", ["lint:auth:pairing-account-scope"]);
|
||||
}
|
||||
|
||||
if (lanes.liveDockerTooling) {
|
||||
addCommand("live Docker shell syntax", "bash", ["-n", ...LIVE_DOCKER_AUTH_SHELL_TARGETS]);
|
||||
addCommand("live Docker scheduler dry run", "node", ["scripts/test-docker-all.mjs"], {
|
||||
...baseEnv,
|
||||
OPENCLAW_DOCKER_ALL_DRY_RUN: "1",
|
||||
OPENCLAW_DOCKER_ALL_LIVE_MODE: "only",
|
||||
});
|
||||
}
|
||||
|
||||
return {
|
||||
commands,
|
||||
summary: Object.entries(lanes)
|
||||
.filter(([, enabled]) => enabled)
|
||||
.map(([lane]) => lane)
|
||||
.join(", "),
|
||||
};
|
||||
}
|
||||
|
||||
export function createTargetedCoreLintCommand(paths, env = process.env, options = {}) {
|
||||
return createTargetedOxlintCommand({
|
||||
env,
|
||||
label: "core",
|
||||
lintablePathRe: LINTABLE_CORE_PATH_RE,
|
||||
neutralPathRe: CORE_LINT_OPTIMIZATION_NEUTRAL_PATH_RE,
|
||||
paths,
|
||||
tsconfig: CORE_OXLINT_TS_CONFIG,
|
||||
...options,
|
||||
});
|
||||
}
|
||||
|
||||
export function createTargetedExtensionLintCommand(paths, env = process.env, options = {}) {
|
||||
return createTargetedOxlintCommand({
|
||||
env,
|
||||
label: "extension",
|
||||
lintablePathRe: LINTABLE_EXTENSION_PATH_RE,
|
||||
neutralPathRe: EXTENSION_LINT_OPTIMIZATION_NEUTRAL_PATH_RE,
|
||||
paths,
|
||||
tsconfig: EXTENSIONS_OXLINT_TS_CONFIG,
|
||||
...options,
|
||||
});
|
||||
}
|
||||
|
||||
export function createTargetedScriptLintCommand(paths, env = process.env, options = {}) {
|
||||
return createTargetedOxlintCommand({
|
||||
env,
|
||||
label: "script",
|
||||
lintablePathRe: LINTABLE_SCRIPT_PATH_RE,
|
||||
neutralPathRe: SCRIPT_LINT_OPTIMIZATION_NEUTRAL_PATH_RE,
|
||||
paths,
|
||||
tsconfig: SCRIPTS_OXLINT_TS_CONFIG,
|
||||
...options,
|
||||
});
|
||||
}
|
||||
|
||||
function createTargetedOxlintCommand({
|
||||
env = process.env,
|
||||
fileExists = existsSync,
|
||||
label,
|
||||
lintablePathRe,
|
||||
neutralPathRe,
|
||||
paths,
|
||||
tsconfig,
|
||||
}) {
|
||||
if (
|
||||
paths.some(
|
||||
(changedPath) =>
|
||||
!lintablePathRe.test(changedPath) &&
|
||||
!neutralPathRe.test(changedPath) &&
|
||||
!MARKDOWN_LINT_OPTIMIZATION_NEUTRAL_PATH_RE.test(changedPath),
|
||||
)
|
||||
) {
|
||||
return null;
|
||||
}
|
||||
const targets = paths
|
||||
.filter((changedPath) => lintablePathRe.test(changedPath))
|
||||
.toSorted((left, right) => left.localeCompare(right));
|
||||
if (targets.length === 0 || targets.length > TARGETED_LINT_PATH_LIMIT) {
|
||||
return null;
|
||||
}
|
||||
if (!targets.every((target) => fileExists(target))) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
name: targets.length === 1 ? `lint ${label} changed file` : `lint ${label} changed files`,
|
||||
bin: "node",
|
||||
args: ["scripts/run-oxlint.mjs", "--tsconfig", tsconfig, ...targets],
|
||||
env,
|
||||
};
|
||||
}
|
||||
|
||||
export async function runChangedCheck(result, options = {}) {
|
||||
const baseEnv = resolveLocalHeavyCheckEnv(options.env ?? process.env);
|
||||
const childEnv = createChangedCheckChildEnv(baseEnv);
|
||||
const plan = createChangedCheckPlan(result, {
|
||||
...options,
|
||||
env: childEnv,
|
||||
});
|
||||
const releaseLock = options.dryRun
|
||||
? () => {}
|
||||
: acquireLocalHeavyCheckLockSync({
|
||||
cwd: process.cwd(),
|
||||
env: baseEnv,
|
||||
toolName: "check:changed",
|
||||
});
|
||||
|
||||
try {
|
||||
printPlan(result, plan, options);
|
||||
|
||||
if (options.dryRun) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
const timings = [];
|
||||
for (const command of plan.commands) {
|
||||
const status = await runPlanCommand(command, timings);
|
||||
if (status !== 0) {
|
||||
printSummary(timings, options);
|
||||
return status;
|
||||
}
|
||||
}
|
||||
|
||||
printSummary(timings, options);
|
||||
return 0;
|
||||
} finally {
|
||||
releaseLock();
|
||||
}
|
||||
}
|
||||
|
||||
function sameArgs(left, right) {
|
||||
return left.length === right.length && left.every((value, index) => value === right[index]);
|
||||
}
|
||||
|
||||
function printPlan(result, plan, options) {
|
||||
const prefix = options.dryRun ? "[check:changed:dry-run]" : "[check:changed]";
|
||||
console.error(`${prefix} lanes=${plan.summary || "none"}`);
|
||||
if (result.extensionImpactFromCore) {
|
||||
console.error(`${prefix} extension-impacting surface; extension typecheck included`);
|
||||
}
|
||||
for (const reason of result.reasons) {
|
||||
console.error(`${prefix} ${reason}`);
|
||||
}
|
||||
if (options.dryRun) {
|
||||
for (const command of plan.commands) {
|
||||
console.error(`${prefix} would run: ${formatPlanCommand(command)}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function runPnpm(command, timings) {
|
||||
return await runCommand(createPnpmManagedCommand(command), timings);
|
||||
}
|
||||
|
||||
async function runPlanCommand(command, timings) {
|
||||
if (command.bin) {
|
||||
return await runCommand(command, timings);
|
||||
}
|
||||
return await runPnpm(command, timings);
|
||||
}
|
||||
|
||||
function formatPlanCommand(command) {
|
||||
const argv = command.bin ? [command.bin, ...command.args] : ["pnpm", ...command.args];
|
||||
return argv.map(formatShellToken).join(" ");
|
||||
}
|
||||
|
||||
function formatShellToken(token) {
|
||||
return /^[A-Za-z0-9_./:@=-]+$/u.test(token) ? token : `'${token.replaceAll("'", "'\\''")}'`;
|
||||
}
|
||||
|
||||
export function createPnpmManagedCommand(command, env = process.env) {
|
||||
const commandEnv = command.env ?? resolveLocalHeavyCheckEnv(env);
|
||||
if (isTruthyEnvFlag(commandEnv.CI) || isTruthyEnvFlag(commandEnv.GITHUB_ACTIONS)) {
|
||||
const shimmedEnv = prependCorepackPnpmShim(commandEnv);
|
||||
return {
|
||||
...command,
|
||||
bin: "corepack",
|
||||
args: ["pnpm", ...command.args],
|
||||
env: shimmedEnv,
|
||||
};
|
||||
}
|
||||
return { ...command, bin: "pnpm", env: commandEnv };
|
||||
}
|
||||
|
||||
function prependCorepackPnpmShim(env) {
|
||||
const shimDir = ensureCorepackPnpmShimDir();
|
||||
return {
|
||||
...env,
|
||||
PATH: [shimDir, env.PATH ?? env.Path ?? ""].filter(Boolean).join(path.delimiter),
|
||||
};
|
||||
}
|
||||
|
||||
function ensureCorepackPnpmShimDir() {
|
||||
if (corepackPnpmShimDir) {
|
||||
return corepackPnpmShimDir;
|
||||
}
|
||||
const dir = mkdtempSync(path.join(tmpdir(), "openclaw-corepack-pnpm-"));
|
||||
const pnpmPath = path.join(dir, "pnpm");
|
||||
writeFileSync(pnpmPath, '#!/bin/sh\nexec corepack pnpm "$@"\n', "utf8");
|
||||
chmodSync(pnpmPath, 0o755);
|
||||
writeFileSync(path.join(dir, "pnpm.cmd"), "@echo off\r\ncorepack pnpm %*\r\n", "utf8");
|
||||
corepackPnpmShimDir = dir;
|
||||
registerCorepackPnpmShimCleanup();
|
||||
return dir;
|
||||
}
|
||||
|
||||
function registerCorepackPnpmShimCleanup() {
|
||||
if (corepackPnpmShimCleanupRegistered) {
|
||||
return;
|
||||
}
|
||||
corepackPnpmShimCleanupRegistered = true;
|
||||
process.once("exit", cleanupCorepackPnpmShimDir);
|
||||
}
|
||||
|
||||
export function cleanupCorepackPnpmShimDir() {
|
||||
if (!corepackPnpmShimDir) {
|
||||
return;
|
||||
}
|
||||
const dir = corepackPnpmShimDir;
|
||||
corepackPnpmShimDir = undefined;
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
async function runCommand(command, timings) {
|
||||
const startedAt = performance.now();
|
||||
console.error(`\n[check:changed] ${command.name}`);
|
||||
let status = 1;
|
||||
try {
|
||||
status = await runManagedCommand({
|
||||
bin: command.bin,
|
||||
args: command.args,
|
||||
env: command.env ?? resolveLocalHeavyCheckEnv(),
|
||||
});
|
||||
} catch (error) {
|
||||
console.error(error);
|
||||
}
|
||||
|
||||
timings.push({
|
||||
name: command.name,
|
||||
durationMs: performance.now() - startedAt,
|
||||
status,
|
||||
});
|
||||
return status;
|
||||
}
|
||||
|
||||
function printSummary(timings, options) {
|
||||
printTimingSummary("check:changed", timings, { skipWhenAllOk: !options.timed });
|
||||
}
|
||||
|
||||
function parseArgs(argv) {
|
||||
const separatorIndex = argv.indexOf("--");
|
||||
const flagArgv = separatorIndex === -1 ? argv : argv.slice(0, separatorIndex);
|
||||
const explicitPaths =
|
||||
separatorIndex === -1 ? [] : argv.slice(separatorIndex + 1).map(normalizeChangedPath);
|
||||
const args = {
|
||||
base: "origin/main",
|
||||
head: "HEAD",
|
||||
staged: false,
|
||||
dryRun: false,
|
||||
timed: false,
|
||||
noChanges: false,
|
||||
help: false,
|
||||
paths: [],
|
||||
};
|
||||
const parsed = parseFlagArgs(
|
||||
flagArgv,
|
||||
args,
|
||||
[
|
||||
stringFlag("--base", "base"),
|
||||
stringFlag("--head", "head"),
|
||||
booleanFlag("--staged", "staged"),
|
||||
booleanFlag("--dry-run", "dryRun"),
|
||||
booleanFlag("--timed", "timed"),
|
||||
booleanFlag("--no-changes", "noChanges"),
|
||||
booleanFlag("--help", "help"),
|
||||
booleanFlag("-h", "help"),
|
||||
],
|
||||
{
|
||||
onUnhandledArg(arg, target) {
|
||||
if (arg.startsWith("-")) {
|
||||
throw new Error(`Unknown option: ${arg}`);
|
||||
}
|
||||
target.paths.push(normalizeChangedPath(arg));
|
||||
return "handled";
|
||||
},
|
||||
},
|
||||
);
|
||||
parsed.paths.push(...explicitPaths);
|
||||
return parsed;
|
||||
}
|
||||
|
||||
function printUsage() {
|
||||
process.stdout.write(
|
||||
[
|
||||
"Usage: node scripts/check-changed.mjs [options] [-- <paths...>]",
|
||||
"",
|
||||
"Options:",
|
||||
" --base <ref> Base ref for changed paths (default: origin/main)",
|
||||
" --head <ref> Head ref for changed paths (default: HEAD)",
|
||||
" --staged Check staged paths instead of git diff paths",
|
||||
" --dry-run Print the planned checks without running them",
|
||||
" --timed Print timing summary",
|
||||
" --no-changes Treat the changed path set as empty",
|
||||
" -h, --help Show this help",
|
||||
"",
|
||||
].join("\n"),
|
||||
);
|
||||
}
|
||||
|
||||
function isDirectRun() {
|
||||
return isDirectRunUrl(process.argv[1], import.meta.url);
|
||||
}
|
||||
|
||||
if (isDirectRun()) {
|
||||
const argv = process.argv.slice(2);
|
||||
let args;
|
||||
try {
|
||||
args = parseArgs(argv);
|
||||
} catch (error) {
|
||||
console.error(error instanceof Error ? error.message : String(error));
|
||||
process.exit(1);
|
||||
}
|
||||
if (args.help) {
|
||||
printUsage();
|
||||
process.exitCode = 0;
|
||||
} else if (shouldDelegateChangedCheckToCrabbox(argv, process.env)) {
|
||||
process.exitCode = await runChangedCheckViaCrabbox(argv, process.env);
|
||||
} else {
|
||||
const paths = args.noChanges
|
||||
? []
|
||||
: args.paths.length > 0
|
||||
? args.paths
|
||||
: args.staged
|
||||
? listStagedChangedPaths()
|
||||
: listChangedPathsFromGit({ base: args.base, head: args.head });
|
||||
const result = detectChangedLanesForPaths({
|
||||
paths,
|
||||
base: args.base,
|
||||
head: args.head,
|
||||
staged: args.staged,
|
||||
});
|
||||
process.exitCode = await runChangedCheck(result, {
|
||||
...args,
|
||||
explicitPaths: args.paths.length > 0,
|
||||
});
|
||||
}
|
||||
}
|
||||
159
scripts/check-changelog-attributions.mjs
Normal file
159
scripts/check-changelog-attributions.mjs
Normal file
@@ -0,0 +1,159 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Rejects changelog thanks entries that credit bots or internal handles.
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
/**
|
||||
* Exact handles that changelog thanks entries must not credit.
|
||||
*/
|
||||
export const FORBIDDEN_CHANGELOG_THANKS_HANDLES = [
|
||||
"codex",
|
||||
"openclaw",
|
||||
"steipete",
|
||||
"clawsweeper",
|
||||
"openclaw-clawsweeper",
|
||||
"clawsweeper[bot]",
|
||||
"openclaw-clawsweeper[bot]",
|
||||
];
|
||||
/**
|
||||
* Handle prefixes that identify forbidden changelog thanks credits.
|
||||
*/
|
||||
export const FORBIDDEN_CHANGELOG_THANKS_HANDLE_PREFIXES = ["app/"];
|
||||
/**
|
||||
* Handle suffixes that identify forbidden changelog thanks credits.
|
||||
*/
|
||||
export const FORBIDDEN_CHANGELOG_THANKS_HANDLE_SUFFIXES = ["[bot]"];
|
||||
/**
|
||||
* Handles that require an explicit human credit instead.
|
||||
*/
|
||||
export const CHANGELOG_THANKS_REQUIRE_HUMAN_CREDIT_HANDLES = [
|
||||
"clawsweeper",
|
||||
"openclaw-clawsweeper",
|
||||
"clawsweeper[bot]",
|
||||
"openclaw-clawsweeper[bot]",
|
||||
];
|
||||
/**
|
||||
* Handle prefixes that require explicit human credit instead.
|
||||
*/
|
||||
export const CHANGELOG_THANKS_REQUIRE_HUMAN_CREDIT_HANDLE_PREFIXES = ["app/"];
|
||||
/**
|
||||
* Handle suffixes that require explicit human credit instead.
|
||||
*/
|
||||
export const CHANGELOG_THANKS_REQUIRE_HUMAN_CREDIT_HANDLE_SUFFIXES = ["[bot]"];
|
||||
|
||||
const THANKS_PATTERN = /\bThanks\b/iu;
|
||||
const THANKED_HANDLE_PATTERN = /@([-_/A-Za-z0-9]+(?:\[bot\])?)/giu;
|
||||
|
||||
/**
|
||||
* Reports whether a handle is forbidden in changelog thanks text.
|
||||
*/
|
||||
export function isForbiddenChangelogThanksHandle(handle, options = {}) {
|
||||
const { strictBotHandle = false } = options;
|
||||
const normalized = handle.toLowerCase();
|
||||
if (normalized === "" || normalized === "null") {
|
||||
// Empty/null input is not a GitHub handle, but the shell query path may pass it through.
|
||||
return true;
|
||||
}
|
||||
if (
|
||||
FORBIDDEN_CHANGELOG_THANKS_HANDLES.includes(normalized) ||
|
||||
FORBIDDEN_CHANGELOG_THANKS_HANDLE_PREFIXES.some((prefix) => normalized.startsWith(prefix)) ||
|
||||
FORBIDDEN_CHANGELOG_THANKS_HANDLE_SUFFIXES.some((suffix) => normalized.endsWith(suffix))
|
||||
) {
|
||||
return true;
|
||||
}
|
||||
if (strictBotHandle) {
|
||||
// PR-author checks should not reject a real human whose login merely contains a bot keyword.
|
||||
return false;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Reports whether a handle needs a separate human credit.
|
||||
*/
|
||||
export function requiresExplicitHumanChangelogThanks(handle) {
|
||||
const normalized = handle.toLowerCase();
|
||||
if (normalized === "" || normalized === "null") {
|
||||
return false;
|
||||
}
|
||||
return (
|
||||
CHANGELOG_THANKS_REQUIRE_HUMAN_CREDIT_HANDLES.includes(normalized) ||
|
||||
CHANGELOG_THANKS_REQUIRE_HUMAN_CREDIT_HANDLE_PREFIXES.some((prefix) =>
|
||||
normalized.startsWith(prefix),
|
||||
) ||
|
||||
CHANGELOG_THANKS_REQUIRE_HUMAN_CREDIT_HANDLE_SUFFIXES.some((suffix) =>
|
||||
normalized.endsWith(suffix),
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Finds changelog lines that thank forbidden handles.
|
||||
*/
|
||||
export function findForbiddenChangelogThanks(content) {
|
||||
return content
|
||||
.split(/\r?\n/u)
|
||||
.map((text, index) => {
|
||||
if (!THANKS_PATTERN.test(text)) {
|
||||
return null;
|
||||
}
|
||||
// A single changelog line may thank multiple handles; scan all of them.
|
||||
for (const match of text.matchAll(THANKED_HANDLE_PATTERN)) {
|
||||
if (isForbiddenChangelogThanksHandle(match[1])) {
|
||||
return { line: index + 1, handle: match[1].toLowerCase(), text };
|
||||
}
|
||||
}
|
||||
return null;
|
||||
})
|
||||
.filter(Boolean);
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the changelog attribution check.
|
||||
*/
|
||||
export async function main(argv = process.argv.slice(2)) {
|
||||
if (argv[0] === "--is-forbidden-handle") {
|
||||
process.exitCode = isForbiddenChangelogThanksHandle(argv[1] ?? "", {
|
||||
strictBotHandle: true,
|
||||
})
|
||||
? 0
|
||||
: 1;
|
||||
return;
|
||||
}
|
||||
|
||||
if (argv[0] === "--requires-explicit-human-thanks") {
|
||||
process.exitCode = requiresExplicitHumanChangelogThanks(argv[1] ?? "") ? 0 : 1;
|
||||
return;
|
||||
}
|
||||
|
||||
const changelogPath = argv[0] ?? "CHANGELOG.md";
|
||||
const absolutePath = path.resolve(process.cwd(), changelogPath);
|
||||
const content = fs.readFileSync(absolutePath, "utf8");
|
||||
const violations = findForbiddenChangelogThanks(content);
|
||||
if (violations.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
console.error("Forbidden changelog thanks attribution:");
|
||||
for (const violation of violations) {
|
||||
const relativePath = path.relative(process.cwd(), absolutePath) || changelogPath;
|
||||
console.error(`- ${relativePath}:${violation.line} uses Thanks @${violation.handle}`);
|
||||
}
|
||||
console.error(
|
||||
`Use a credited external GitHub username instead of ${FORBIDDEN_CHANGELOG_THANKS_HANDLES.map(
|
||||
(handle) => `@${handle}`,
|
||||
).join(", ")}.`,
|
||||
);
|
||||
process.exitCode = 1;
|
||||
}
|
||||
|
||||
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
||||
main().catch(
|
||||
/** @param {unknown} error */ (error) => {
|
||||
console.error(error);
|
||||
process.exit(1);
|
||||
},
|
||||
);
|
||||
}
|
||||
368
scripts/check-channel-agnostic-boundaries.mjs
Normal file
368
scripts/check-channel-agnostic-boundaries.mjs
Normal file
@@ -0,0 +1,368 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Checks channel-agnostic core surfaces for channel-specific coupling.
|
||||
import { promises as fs } from "node:fs";
|
||||
import path from "node:path";
|
||||
import ts from "typescript";
|
||||
import {
|
||||
collectTypeScriptFiles,
|
||||
getPropertyNameText,
|
||||
resolveRepoRoot,
|
||||
runAsScript,
|
||||
toLine,
|
||||
} from "./lib/ts-guard-utils.mjs";
|
||||
|
||||
const repoRoot = resolveRepoRoot(import.meta.url);
|
||||
|
||||
const acpCoreProtectedSources = [
|
||||
path.join(repoRoot, "src", "acp"),
|
||||
path.join(repoRoot, "src", "agents", "acp-spawn.ts"),
|
||||
path.join(repoRoot, "src", "auto-reply", "reply", "commands-acp"),
|
||||
path.join(repoRoot, "src", "infra", "outbound", "conversation-id.ts"),
|
||||
];
|
||||
|
||||
const channelCoreProtectedSources = [
|
||||
path.join(repoRoot, "src", "channels", "thread-bindings-policy.ts"),
|
||||
path.join(repoRoot, "src", "channels", "thread-bindings-messages.ts"),
|
||||
path.join(repoRoot, "src", "sessions", "send-policy.ts"),
|
||||
path.join(repoRoot, "src", "sessions", "session-chat-type-shared.ts"),
|
||||
path.join(repoRoot, "src", "utils", "delivery-context.ts"),
|
||||
];
|
||||
const acpUserFacingTextSources = [
|
||||
path.join(repoRoot, "src", "auto-reply", "reply", "commands-acp"),
|
||||
];
|
||||
const systemMarkLiteralGuardSources = [
|
||||
path.join(repoRoot, "src", "auto-reply", "reply", "commands-acp"),
|
||||
path.join(repoRoot, "src", "auto-reply", "reply", "dispatch-acp.ts"),
|
||||
path.join(repoRoot, "src", "auto-reply", "reply", "directive-handling.shared.ts"),
|
||||
path.join(repoRoot, "src", "channels", "thread-bindings-messages.ts"),
|
||||
];
|
||||
|
||||
const channelIds = [
|
||||
"discord",
|
||||
"googlechat",
|
||||
"imessage",
|
||||
"irc",
|
||||
"line",
|
||||
"mattermost",
|
||||
"matrix",
|
||||
"msteams",
|
||||
"nextcloud-talk",
|
||||
"nostr",
|
||||
"qqbot",
|
||||
"signal",
|
||||
"slack",
|
||||
"synology-chat",
|
||||
"telegram",
|
||||
"tlon",
|
||||
"twitch",
|
||||
"web",
|
||||
"whatsapp",
|
||||
"zalo",
|
||||
"zalouser",
|
||||
];
|
||||
|
||||
const channelIdSet = new Set(channelIds);
|
||||
const channelSegmentRe = new RegExp(`(^|[._/-])(?:${channelIds.join("|")})([._/-]|$)`);
|
||||
const comparisonOperators = new Set([
|
||||
ts.SyntaxKind.EqualsEqualsEqualsToken,
|
||||
ts.SyntaxKind.ExclamationEqualsEqualsToken,
|
||||
ts.SyntaxKind.EqualsEqualsToken,
|
||||
ts.SyntaxKind.ExclamationEqualsToken,
|
||||
]);
|
||||
|
||||
const allowedViolations = new Set([]);
|
||||
|
||||
function isChannelsPropertyAccess(node) {
|
||||
if (ts.isPropertyAccessExpression(node)) {
|
||||
return node.name.text === "channels";
|
||||
}
|
||||
if (ts.isElementAccessExpression(node) && ts.isStringLiteral(node.argumentExpression)) {
|
||||
return node.argumentExpression.text === "channels";
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function readStringLiteral(node) {
|
||||
if (ts.isStringLiteral(node)) {
|
||||
return node.text;
|
||||
}
|
||||
if (ts.isNoSubstitutionTemplateLiteral(node)) {
|
||||
return node.text;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function isChannelLiteralNode(node) {
|
||||
const text = readStringLiteral(node);
|
||||
return text ? channelIdSet.has(text) : false;
|
||||
}
|
||||
|
||||
function matchesChannelModuleSpecifier(specifier) {
|
||||
return channelSegmentRe.test(specifier.replaceAll("\\", "/"));
|
||||
}
|
||||
|
||||
const userFacingChannelNameRe =
|
||||
/\b(?:discord|telegram|slack|signal|imessage|whatsapp|google\s*chat|irc|line|zalo|matrix|msteams)\b/i;
|
||||
const systemMarkLiteral = "⚙️";
|
||||
|
||||
function isModuleSpecifierStringNode(node) {
|
||||
const parent = node.parent;
|
||||
if (ts.isImportDeclaration(parent) || ts.isExportDeclaration(parent)) {
|
||||
return true;
|
||||
}
|
||||
return (
|
||||
ts.isCallExpression(parent) &&
|
||||
parent.expression.kind === ts.SyntaxKind.ImportKeyword &&
|
||||
parent.arguments[0] === node
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Finds channel-specific references inside channel-agnostic protected sources.
|
||||
*/
|
||||
export function findChannelAgnosticBoundaryViolations(
|
||||
content,
|
||||
fileName = "source.ts",
|
||||
options = {},
|
||||
) {
|
||||
const checkModuleSpecifiers = options.checkModuleSpecifiers ?? true;
|
||||
const checkConfigPaths = options.checkConfigPaths ?? true;
|
||||
const checkChannelComparisons = options.checkChannelComparisons ?? true;
|
||||
const checkChannelAssignments = options.checkChannelAssignments ?? true;
|
||||
const moduleSpecifierMatcher = options.moduleSpecifierMatcher ?? matchesChannelModuleSpecifier;
|
||||
|
||||
const sourceFile = ts.createSourceFile(fileName, content, ts.ScriptTarget.Latest, true);
|
||||
const violations = [];
|
||||
|
||||
const visit = (node) => {
|
||||
if (
|
||||
checkModuleSpecifiers &&
|
||||
ts.isImportDeclaration(node) &&
|
||||
ts.isStringLiteral(node.moduleSpecifier)
|
||||
) {
|
||||
const specifier = node.moduleSpecifier.text;
|
||||
if (moduleSpecifierMatcher(specifier)) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node.moduleSpecifier),
|
||||
reason: `imports channel module "${specifier}"`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (
|
||||
checkModuleSpecifiers &&
|
||||
ts.isExportDeclaration(node) &&
|
||||
node.moduleSpecifier &&
|
||||
ts.isStringLiteral(node.moduleSpecifier)
|
||||
) {
|
||||
const specifier = node.moduleSpecifier.text;
|
||||
if (moduleSpecifierMatcher(specifier)) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node.moduleSpecifier),
|
||||
reason: `re-exports channel module "${specifier}"`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (
|
||||
checkModuleSpecifiers &&
|
||||
ts.isCallExpression(node) &&
|
||||
node.expression.kind === ts.SyntaxKind.ImportKeyword &&
|
||||
node.arguments.length > 0 &&
|
||||
ts.isStringLiteral(node.arguments[0])
|
||||
) {
|
||||
const specifier = node.arguments[0].text;
|
||||
if (moduleSpecifierMatcher(specifier)) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node.arguments[0]),
|
||||
reason: `dynamically imports channel module "${specifier}"`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (
|
||||
checkConfigPaths &&
|
||||
ts.isPropertyAccessExpression(node) &&
|
||||
channelIdSet.has(node.name.text)
|
||||
) {
|
||||
if (isChannelsPropertyAccess(node.expression)) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node.name),
|
||||
reason: `references config path "channels.${node.name.text}"`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (
|
||||
checkConfigPaths &&
|
||||
ts.isElementAccessExpression(node) &&
|
||||
ts.isStringLiteral(node.argumentExpression) &&
|
||||
channelIdSet.has(node.argumentExpression.text)
|
||||
) {
|
||||
if (isChannelsPropertyAccess(node.expression)) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node.argumentExpression),
|
||||
reason: `references config path "channels[${JSON.stringify(node.argumentExpression.text)}]"`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (
|
||||
checkChannelComparisons &&
|
||||
ts.isBinaryExpression(node) &&
|
||||
comparisonOperators.has(node.operatorToken.kind)
|
||||
) {
|
||||
if (isChannelLiteralNode(node.left) || isChannelLiteralNode(node.right)) {
|
||||
const leftText = node.left.getText(sourceFile);
|
||||
const rightText = node.right.getText(sourceFile);
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node.operatorToken),
|
||||
reason: `compares with channel id literal (${leftText} ${node.operatorToken.getText(sourceFile)} ${rightText})`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (checkChannelAssignments && ts.isPropertyAssignment(node)) {
|
||||
const propName = getPropertyNameText(node.name);
|
||||
if (propName === "channel" && isChannelLiteralNode(node.initializer)) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node.initializer),
|
||||
reason: `assigns channel id literal to "channel" (${node.initializer.getText(sourceFile)})`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
ts.forEachChild(node, visit);
|
||||
};
|
||||
|
||||
visit(sourceFile);
|
||||
return violations;
|
||||
}
|
||||
|
||||
/**
|
||||
* Finds reverse dependencies from channel core into plugin/runtime surfaces.
|
||||
*/
|
||||
export function findChannelCoreReverseDependencyViolations(content, fileName = "source.ts") {
|
||||
return findChannelAgnosticBoundaryViolations(content, fileName, {
|
||||
checkModuleSpecifiers: true,
|
||||
checkConfigPaths: false,
|
||||
checkChannelComparisons: false,
|
||||
checkChannelAssignments: false,
|
||||
moduleSpecifierMatcher: matchesChannelModuleSpecifier,
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Finds user-facing channel names in ACP-owned text sources.
|
||||
*/
|
||||
export function findAcpUserFacingChannelNameViolations(content, fileName = "source.ts") {
|
||||
const sourceFile = ts.createSourceFile(fileName, content, ts.ScriptTarget.Latest, true);
|
||||
const violations = [];
|
||||
|
||||
const visit = (node) => {
|
||||
const text = readStringLiteral(node);
|
||||
if (text && userFacingChannelNameRe.test(text) && !isModuleSpecifierStringNode(node)) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node),
|
||||
reason: `user-facing text references channel name (${JSON.stringify(text)})`,
|
||||
});
|
||||
}
|
||||
ts.forEachChild(node, visit);
|
||||
};
|
||||
|
||||
visit(sourceFile);
|
||||
return violations;
|
||||
}
|
||||
|
||||
/**
|
||||
* Finds raw system mark literals where shared constants should be used.
|
||||
*/
|
||||
export function findSystemMarkLiteralViolations(content, fileName = "source.ts") {
|
||||
const sourceFile = ts.createSourceFile(fileName, content, ts.ScriptTarget.Latest, true);
|
||||
const violations = [];
|
||||
|
||||
const visit = (node) => {
|
||||
const text = readStringLiteral(node);
|
||||
if (text && text.includes(systemMarkLiteral) && !isModuleSpecifierStringNode(node)) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node),
|
||||
reason: `hardcoded system mark literal (${JSON.stringify(text)})`,
|
||||
});
|
||||
}
|
||||
ts.forEachChild(node, visit);
|
||||
};
|
||||
|
||||
visit(sourceFile);
|
||||
return violations;
|
||||
}
|
||||
|
||||
const boundaryRuleSets = [
|
||||
{
|
||||
id: "acp-core",
|
||||
sources: acpCoreProtectedSources,
|
||||
scan: (content, fileName) => findChannelAgnosticBoundaryViolations(content, fileName),
|
||||
},
|
||||
{
|
||||
id: "channel-core-reverse-deps",
|
||||
sources: channelCoreProtectedSources,
|
||||
scan: (content, fileName) => findChannelCoreReverseDependencyViolations(content, fileName),
|
||||
},
|
||||
{
|
||||
id: "acp-user-facing-text",
|
||||
sources: acpUserFacingTextSources,
|
||||
scan: (content, fileName) => findAcpUserFacingChannelNameViolations(content, fileName),
|
||||
},
|
||||
{
|
||||
id: "system-mark-literal-usage",
|
||||
sources: systemMarkLiteralGuardSources,
|
||||
scan: (content, fileName) => findSystemMarkLiteralViolations(content, fileName),
|
||||
},
|
||||
];
|
||||
|
||||
/**
|
||||
* Runs all channel-agnostic boundary checks.
|
||||
*/
|
||||
export async function main() {
|
||||
const violations = [];
|
||||
for (const ruleSet of boundaryRuleSets) {
|
||||
const files = (
|
||||
await Promise.all(
|
||||
ruleSet.sources.map(
|
||||
async (sourcePath) =>
|
||||
await collectTypeScriptFiles(sourcePath, {
|
||||
ignoreMissing: true,
|
||||
}),
|
||||
),
|
||||
)
|
||||
).flat();
|
||||
for (const filePath of files) {
|
||||
const relativeFile = path.relative(repoRoot, filePath);
|
||||
if (
|
||||
allowedViolations.has(`${ruleSet.id}:${relativeFile}`) ||
|
||||
allowedViolations.has(relativeFile)
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
const content = await fs.readFile(filePath, "utf8");
|
||||
for (const violation of ruleSet.scan(content, relativeFile)) {
|
||||
violations.push(`${ruleSet.id} ${relativeFile}:${violation.line}: ${violation.reason}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (violations.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
console.error("Found channel-specific references in channel-agnostic sources:");
|
||||
for (const violation of violations) {
|
||||
console.error(`- ${violation}`);
|
||||
}
|
||||
console.error(
|
||||
"Move channel-specific logic to channel adapters or add a justified allowlist entry.",
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
runAsScript(import.meta.url, main);
|
||||
266
scripts/check-cli-bootstrap-imports.mjs
Normal file
266
scripts/check-cli-bootstrap-imports.mjs
Normal file
@@ -0,0 +1,266 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Checks CLI bootstrap chunks for forbidden eager imports and size regressions.
|
||||
import fs from "node:fs";
|
||||
import module from "node:module";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const DEFAULT_ENTRYPOINTS = ["dist/entry.js", "dist/cli/run-main.js"];
|
||||
const DEFAULT_GATEWAY_RUN_CHUNK_MAX_BYTES = 70 * 1024;
|
||||
const GATEWAY_RUN_CHUNK_MARKER_SETS = [
|
||||
["const GATEWAY_AUTH_MODES", "function addGatewayRunCommand"],
|
||||
["const GATEWAY_RUN_VALUE_KEYS", "function addGatewayRunCommand"],
|
||||
];
|
||||
const GATEWAY_RUN_FORBIDDEN_STATIC_IMPORTS = [
|
||||
"control-ui-assets",
|
||||
"diagnostic-stability-bundle",
|
||||
"onboard-helpers",
|
||||
"process-respawn",
|
||||
"restart-sentinel",
|
||||
"server-close",
|
||||
"server-reload-handlers",
|
||||
];
|
||||
const STATIC_IMPORT_RE =
|
||||
/\b(?:import|export)\s+(?:(?:[^'"()]*?\s+from\s+)|)["'](?<specifier>[^"']+)["']/gu;
|
||||
|
||||
function isMainModule() {
|
||||
return process.argv[1] ? path.resolve(process.argv[1]) === fileURLToPath(import.meta.url) : false;
|
||||
}
|
||||
|
||||
function isBuiltinSpecifier(specifier) {
|
||||
return specifier.startsWith("node:") || module.isBuiltin(specifier);
|
||||
}
|
||||
|
||||
function isRelativeSpecifier(specifier) {
|
||||
return specifier.startsWith("./") || specifier.startsWith("../") || specifier.startsWith("/");
|
||||
}
|
||||
|
||||
function resolveRelativeImport(importer, specifier, fsImpl = fs) {
|
||||
const base = specifier.startsWith("/")
|
||||
? specifier
|
||||
: path.resolve(path.dirname(importer), specifier);
|
||||
const candidates = [
|
||||
base,
|
||||
`${base}.js`,
|
||||
`${base}.mjs`,
|
||||
`${base}.cjs`,
|
||||
path.join(base, "index.js"),
|
||||
path.join(base, "index.mjs"),
|
||||
path.join(base, "index.cjs"),
|
||||
];
|
||||
return candidates.find((candidate) => {
|
||||
try {
|
||||
return fsImpl.statSync(candidate).isFile();
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Lists static import/export specifiers from a JavaScript source string.
|
||||
*/
|
||||
export function listStaticImportSpecifiers(source) {
|
||||
return [...source.matchAll(STATIC_IMPORT_RE)].map((match) => match.groups?.specifier ?? "");
|
||||
}
|
||||
|
||||
function walkStaticImportGraph(params) {
|
||||
const { fsImpl, rootDir } = params;
|
||||
const queue = params.roots.map((entrypoint) => path.resolve(rootDir, entrypoint));
|
||||
const visited = new Set();
|
||||
const errors = [];
|
||||
|
||||
for (const filePath of queue) {
|
||||
if (!filePath || visited.has(filePath)) {
|
||||
continue;
|
||||
}
|
||||
visited.add(filePath);
|
||||
|
||||
let source;
|
||||
try {
|
||||
source = fsImpl.readFileSync(filePath, "utf8");
|
||||
} catch {
|
||||
errors.push(
|
||||
`CLI bootstrap import guard could not read ${path.relative(rootDir, filePath) || filePath}. Run pnpm build first.`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
for (const specifier of listStaticImportSpecifiers(source)) {
|
||||
if (!specifier || isBuiltinSpecifier(specifier)) {
|
||||
continue;
|
||||
}
|
||||
if (!isRelativeSpecifier(specifier)) {
|
||||
params.onExternalSpecifier?.({ filePath, specifier, errors });
|
||||
continue;
|
||||
}
|
||||
const resolved = resolveRelativeImport(filePath, specifier, fsImpl);
|
||||
if (!resolved) {
|
||||
errors.push(
|
||||
`CLI bootstrap import guard could not resolve "${specifier}" from ${path.relative(
|
||||
rootDir,
|
||||
filePath,
|
||||
)}.`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
params.onRelativeSpecifier?.({ filePath, resolved, specifier, errors });
|
||||
if (!visited.has(resolved)) {
|
||||
queue.push(resolved);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return errors;
|
||||
}
|
||||
|
||||
/**
|
||||
* Collects forbidden external import errors for CLI bootstrap entrypoints.
|
||||
*/
|
||||
export function collectCliBootstrapExternalImportErrors(params = {}) {
|
||||
const rootDir = params.rootDir ?? process.cwd();
|
||||
const entrypoints = params.entrypoints ?? DEFAULT_ENTRYPOINTS;
|
||||
const fsImpl = params.fs ?? fs;
|
||||
const errors = walkStaticImportGraph({
|
||||
fsImpl,
|
||||
rootDir,
|
||||
roots: entrypoints,
|
||||
onExternalSpecifier: ({ filePath, specifier, errors: graphErrors }) => {
|
||||
graphErrors.push(
|
||||
`CLI bootstrap static graph imports external package "${specifier}" from ${path.relative(
|
||||
rootDir,
|
||||
filePath,
|
||||
)}.`,
|
||||
);
|
||||
},
|
||||
});
|
||||
|
||||
return errors.toSorted((left, right) => left.localeCompare(right));
|
||||
}
|
||||
|
||||
function listJsFiles(dirPath, fsImpl = fs) {
|
||||
let entries;
|
||||
try {
|
||||
entries = fsImpl.readdirSync(dirPath, { withFileTypes: true });
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
const files = [];
|
||||
for (const entry of entries) {
|
||||
const fullPath = path.join(dirPath, entry.name);
|
||||
if (entry.isDirectory()) {
|
||||
files.push(...listJsFiles(fullPath, fsImpl));
|
||||
continue;
|
||||
}
|
||||
if (entry.isFile() && entry.name.endsWith(".js")) {
|
||||
files.push(fullPath);
|
||||
}
|
||||
}
|
||||
return files;
|
||||
}
|
||||
|
||||
/**
|
||||
* Collects gateway-run chunk budget errors from built CLI output.
|
||||
*/
|
||||
export function collectGatewayRunChunkBudgetErrors(params = {}) {
|
||||
const rootDir = params.rootDir ?? process.cwd();
|
||||
const fsImpl = params.fs ?? fs;
|
||||
const distDir = path.resolve(rootDir, params.distDir ?? "dist");
|
||||
const maxBytes = params.gatewayRunChunkMaxBytes ?? DEFAULT_GATEWAY_RUN_CHUNK_MAX_BYTES;
|
||||
const chunks = [];
|
||||
|
||||
for (const filePath of listJsFiles(distDir, fsImpl)) {
|
||||
let source;
|
||||
try {
|
||||
source = fsImpl.readFileSync(filePath, "utf8");
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
if (
|
||||
GATEWAY_RUN_CHUNK_MARKER_SETS.some((markers) =>
|
||||
markers.every((marker) => source.includes(marker)),
|
||||
)
|
||||
) {
|
||||
chunks.push({ filePath, source });
|
||||
}
|
||||
}
|
||||
|
||||
if (chunks.length === 0) {
|
||||
return [
|
||||
"CLI bootstrap import guard could not find the bundled gateway run chunk. Run pnpm build first.",
|
||||
];
|
||||
}
|
||||
|
||||
const errors = [];
|
||||
for (const { filePath, source } of chunks) {
|
||||
const relativePath = path.relative(rootDir, filePath) || filePath;
|
||||
let size = Buffer.byteLength(source, "utf8");
|
||||
try {
|
||||
size = fsImpl.statSync(filePath).size;
|
||||
} catch {
|
||||
// Fall back to source byte length for in-memory test fixtures.
|
||||
}
|
||||
if (size > maxBytes) {
|
||||
errors.push(
|
||||
`Gateway run chunk ${relativePath} is ${size} bytes, above budget ${maxBytes} bytes.`,
|
||||
);
|
||||
}
|
||||
|
||||
errors.push(
|
||||
...walkStaticImportGraph({
|
||||
fsImpl,
|
||||
rootDir,
|
||||
roots: [filePath],
|
||||
onRelativeSpecifier: ({
|
||||
filePath: importerPath,
|
||||
resolved,
|
||||
specifier,
|
||||
errors: graphErrors,
|
||||
}) => {
|
||||
const resolvedRelativePath = path.relative(rootDir, resolved) || resolved;
|
||||
const coldPath = [specifier, resolvedRelativePath].find((candidate) =>
|
||||
GATEWAY_RUN_FORBIDDEN_STATIC_IMPORTS.some((forbidden) => candidate.includes(forbidden)),
|
||||
);
|
||||
if (!coldPath) {
|
||||
return;
|
||||
}
|
||||
graphErrors.push(
|
||||
`Gateway run chunk ${relativePath} static graph imports cold path "${coldPath}" from ${
|
||||
path.relative(rootDir, importerPath) || importerPath
|
||||
}.`,
|
||||
);
|
||||
},
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
return errors.toSorted((left, right) => left.localeCompare(right));
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the CLI bootstrap import and chunk-budget checks.
|
||||
*/
|
||||
export function checkCliBootstrapExternalImports(params = {}) {
|
||||
const errors = [
|
||||
...collectCliBootstrapExternalImportErrors(params),
|
||||
...collectGatewayRunChunkBudgetErrors(params),
|
||||
];
|
||||
if (errors.length === 0) {
|
||||
return;
|
||||
}
|
||||
const logger = params.logger ?? console;
|
||||
logger.error("CLI bootstrap import guard failed:");
|
||||
for (const error of errors) {
|
||||
logger.error(` - ${error}`);
|
||||
}
|
||||
throw new Error("CLI bootstrap static graph imports external packages.");
|
||||
}
|
||||
|
||||
if (isMainModule()) {
|
||||
try {
|
||||
checkCliBootstrapExternalImports();
|
||||
console.log("CLI bootstrap import guard passed.");
|
||||
} catch {
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
401
scripts/check-cli-startup-memory.mjs
Normal file
401
scripts/check-cli-startup-memory.mjs
Normal file
@@ -0,0 +1,401 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Measures CLI startup memory with an isolated home and RSS hook.
|
||||
import { spawnSync as defaultSpawnSync } from "node:child_process";
|
||||
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
|
||||
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const tmpDir = process.env.TMPDIR || process.env.TEMP || process.env.TMP || os.tmpdir();
|
||||
const MAX_RSS_MARKER = "__OPENCLAW_MAX_RSS_KB__=";
|
||||
const DEFAULT_COMMAND_TIMEOUT_MS = 60_000;
|
||||
const COMMAND_TIMEOUT_MS = readPositiveIntEnv(
|
||||
"OPENCLAW_STARTUP_MEMORY_TIMEOUT_MS",
|
||||
DEFAULT_COMMAND_TIMEOUT_MS,
|
||||
);
|
||||
let tmpHome = null;
|
||||
let rssHookPath = null;
|
||||
|
||||
function readPositiveIntEnv(name, fallback, env = process.env) {
|
||||
const value = readPositiveNumberEnv(name, fallback, env);
|
||||
if (!Number.isSafeInteger(value)) {
|
||||
throw new Error(`${name} must be a positive integer`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function readPositiveNumberEnv(name, fallback, env = process.env) {
|
||||
const raw = env[name];
|
||||
if (raw === undefined || raw === "") {
|
||||
return fallback;
|
||||
}
|
||||
const text = raw.trim();
|
||||
if (!/^(?:\d+(?:\.\d+)?|\.\d+)$/u.test(text)) {
|
||||
throw new Error(`${name} must be a positive number`);
|
||||
}
|
||||
const value = Number(text);
|
||||
if (!Number.isFinite(value) || value <= 0) {
|
||||
throw new Error(`${name} must be a positive number`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function readNonEmptyEnv(name) {
|
||||
const value = process.env[name];
|
||||
return value === undefined || value.length === 0 ? null : value;
|
||||
}
|
||||
|
||||
function readRequiredPathOption(argv, index, flag) {
|
||||
const value = argv[index + 1];
|
||||
if (!value || value.startsWith("-")) {
|
||||
throw new Error(`${flag} requires a path`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function parseArgs(argv) {
|
||||
const options = {
|
||||
jsonPath:
|
||||
readNonEmptyEnv("OPENCLAW_STARTUP_MEMORY_JSON_PATH") ??
|
||||
path.join(repoRoot, ".artifacts", "startup-memory", "startup-memory.json"),
|
||||
summaryPath:
|
||||
readNonEmptyEnv("OPENCLAW_STARTUP_MEMORY_SUMMARY_PATH") ??
|
||||
path.join(repoRoot, ".artifacts", "startup-memory", "summary.md"),
|
||||
};
|
||||
for (let index = 0; index < argv.length; index += 1) {
|
||||
const arg = argv[index];
|
||||
if (arg === "--json") {
|
||||
const value = readRequiredPathOption(argv, index, "--json");
|
||||
options.jsonPath = path.resolve(value);
|
||||
index += 1;
|
||||
continue;
|
||||
}
|
||||
if (arg === "--summary") {
|
||||
const value = readRequiredPathOption(argv, index, "--summary");
|
||||
options.summaryPath = path.resolve(value);
|
||||
index += 1;
|
||||
continue;
|
||||
}
|
||||
if (arg === "--help") {
|
||||
console.log(
|
||||
"Usage: node scripts/check-cli-startup-memory.mjs [--json <path>] [--summary <path>]",
|
||||
);
|
||||
process.exit(0);
|
||||
}
|
||||
throw new Error(`Unknown option: ${arg}`);
|
||||
}
|
||||
return options;
|
||||
}
|
||||
|
||||
function resolveDefaultLimitsMb(platform = process.platform) {
|
||||
return {
|
||||
// Linux CI is the tight startup regression signal. macOS consistently reports
|
||||
// higher RSS for the same launcher path, so keep it supported without hiding
|
||||
// Linux help-path regressions.
|
||||
help: platform === "darwin" ? 300 : 100,
|
||||
statusJson: 400,
|
||||
gatewayStatus: 500,
|
||||
};
|
||||
}
|
||||
|
||||
const DEFAULT_LIMITS_MB = resolveDefaultLimitsMb();
|
||||
|
||||
const cases = [
|
||||
{
|
||||
id: "help",
|
||||
label: "--help",
|
||||
args: ["openclaw.mjs", "--help"],
|
||||
limitMb: readPositiveNumberEnv("OPENCLAW_STARTUP_MEMORY_HELP_MB", DEFAULT_LIMITS_MB.help),
|
||||
},
|
||||
{
|
||||
id: "statusJson",
|
||||
label: "status --json",
|
||||
args: ["openclaw.mjs", "status", "--json"],
|
||||
limitMb: readPositiveNumberEnv(
|
||||
"OPENCLAW_STARTUP_MEMORY_STATUS_JSON_MB",
|
||||
DEFAULT_LIMITS_MB.statusJson,
|
||||
),
|
||||
},
|
||||
{
|
||||
id: "gatewayStatus",
|
||||
label: "gateway status",
|
||||
args: ["openclaw.mjs", "gateway", "status"],
|
||||
limitMb: readPositiveNumberEnv(
|
||||
"OPENCLAW_STARTUP_MEMORY_GATEWAY_STATUS_MB",
|
||||
DEFAULT_LIMITS_MB.gatewayStatus,
|
||||
),
|
||||
},
|
||||
];
|
||||
|
||||
function formatFixGuidance(testCase, details) {
|
||||
const command = `node ${testCase.args.join(" ")}`;
|
||||
const guidance = [
|
||||
"[startup-memory] Fix guidance",
|
||||
`Case: ${testCase.label}`,
|
||||
`Command: ${command}`,
|
||||
"Next steps:",
|
||||
`1. Run \`${command}\` locally on the built tree.`,
|
||||
"2. If this is an RSS overage, compare the startup import graph against the last passing commit and look for newly eager imports, bootstrap side effects, or plugin loading on the command path.",
|
||||
"3. If this is a non-zero exit, inspect the first transitive import/config error in stderr and fix that root cause before re-checking memory.",
|
||||
"LLM prompt:",
|
||||
`"OpenClaw startup-memory CI failed for '${testCase.label}'. Analyze this failure, identify the first runtime/import side effect that makes startup heavier or broken, and propose the smallest safe patch. Failure output:\n${details}"`,
|
||||
];
|
||||
return `${guidance.join("\n")}\n`;
|
||||
}
|
||||
|
||||
function formatFailure(testCase, message, details = "") {
|
||||
const trimmedDetails = details.trim();
|
||||
const sections = [message];
|
||||
if (trimmedDetails) {
|
||||
sections.push(trimmedDetails);
|
||||
}
|
||||
sections.push(formatFixGuidance(testCase, trimmedDetails || message));
|
||||
return sections.join("\n\n");
|
||||
}
|
||||
|
||||
function parseMaxRssMb(stderr) {
|
||||
const matches = [...stderr.matchAll(new RegExp(`^${MAX_RSS_MARKER}(\\d+)\\s*$`, "gm"))];
|
||||
const lastMatch = matches.at(-1);
|
||||
if (!lastMatch) {
|
||||
return null;
|
||||
}
|
||||
const maxRssKb = Number(lastMatch[1]);
|
||||
return Number.isFinite(maxRssKb) && maxRssKb > 0 ? maxRssKb / 1024 : null;
|
||||
}
|
||||
|
||||
function formatMb(value) {
|
||||
return typeof value === "number" && Number.isFinite(value) ? `${value.toFixed(1)} MB` : "n/a";
|
||||
}
|
||||
|
||||
function formatCaseCommand(testCase) {
|
||||
return `node ${testCase.args.join(" ")}`;
|
||||
}
|
||||
|
||||
function nodeImportSpecifierForPath(filePath) {
|
||||
return pathToFileURL(filePath).href;
|
||||
}
|
||||
|
||||
function buildBenchEnv() {
|
||||
if (!tmpHome) {
|
||||
throw new Error("temporary home is not initialized");
|
||||
}
|
||||
const env = {
|
||||
HOME: tmpHome,
|
||||
USERPROFILE: tmpHome,
|
||||
XDG_CONFIG_HOME: path.join(tmpHome, ".config"),
|
||||
XDG_DATA_HOME: path.join(tmpHome, ".local", "share"),
|
||||
XDG_CACHE_HOME: path.join(tmpHome, ".cache"),
|
||||
PATH: process.env.PATH ?? "",
|
||||
TMPDIR: tmpDir,
|
||||
TEMP: tmpDir,
|
||||
TMP: tmpDir,
|
||||
LANG: process.env.LANG ?? "C.UTF-8",
|
||||
TERM: process.env.TERM ?? "dumb",
|
||||
};
|
||||
|
||||
if (process.env.LC_ALL) {
|
||||
env.LC_ALL = process.env.LC_ALL;
|
||||
}
|
||||
if (process.env.CI) {
|
||||
env.CI = process.env.CI;
|
||||
}
|
||||
if (process.env.NODE_DISABLE_COMPILE_CACHE) {
|
||||
env.NODE_DISABLE_COMPILE_CACHE = process.env.NODE_DISABLE_COMPILE_CACHE;
|
||||
} else {
|
||||
// Keep the regression check focused on app/runtime startup, not Node's
|
||||
// one-shot compile cache overhead, which varies across runner builds.
|
||||
env.NODE_DISABLE_COMPILE_CACHE = "1";
|
||||
}
|
||||
// Keep the benchmark on a single process so RSS reflects the actual command
|
||||
// path rather than the warning-suppression respawn wrapper.
|
||||
env.OPENCLAW_NO_RESPAWN = "1";
|
||||
|
||||
return env;
|
||||
}
|
||||
|
||||
function runCase(testCase, params = {}) {
|
||||
if (!rssHookPath) {
|
||||
throw new Error("RSS hook path is not initialized");
|
||||
}
|
||||
const env = buildBenchEnv();
|
||||
const spawn = params.spawnSync ?? defaultSpawnSync;
|
||||
const timeoutMs = params.timeoutMs ?? COMMAND_TIMEOUT_MS;
|
||||
const result = spawn(
|
||||
process.execPath,
|
||||
["--import", nodeImportSpecifierForPath(rssHookPath), ...testCase.args],
|
||||
{
|
||||
cwd: repoRoot,
|
||||
env,
|
||||
encoding: "utf8",
|
||||
maxBuffer: 20 * 1024 * 1024,
|
||||
timeout: timeoutMs,
|
||||
killSignal: "SIGKILL",
|
||||
},
|
||||
);
|
||||
const stderr = result.stderr ?? "";
|
||||
const maxRssMb = parseMaxRssMb(stderr);
|
||||
const matrixBootstrapWarning = /matrix: crypto runtime bootstrap failed/i.test(stderr);
|
||||
const report = {
|
||||
id: testCase.id,
|
||||
label: testCase.label,
|
||||
command: formatCaseCommand(testCase),
|
||||
limitMb: testCase.limitMb,
|
||||
maxRssMb,
|
||||
status: "pass",
|
||||
exitCode: result.status,
|
||||
signal: result.signal ?? null,
|
||||
error: null,
|
||||
};
|
||||
|
||||
if (result.error) {
|
||||
const timedOut = result.error.code === "ETIMEDOUT";
|
||||
report.status = "fail";
|
||||
report.error = timedOut
|
||||
? `${testCase.label} timed out after ${timeoutMs}ms`
|
||||
: `${testCase.label} failed to start: ${result.error.message}`;
|
||||
return Object.assign(report, {
|
||||
failureMessage: formatFailure(testCase, report.error, stderr.trim() || result.stdout || ""),
|
||||
});
|
||||
}
|
||||
if (result.status !== 0) {
|
||||
report.status = "fail";
|
||||
const exitDetail = result.status ?? result.signal ?? "unknown";
|
||||
report.error = `${testCase.label} exited with ${String(exitDetail)}`;
|
||||
return Object.assign(report, {
|
||||
failureMessage: formatFailure(testCase, report.error, stderr.trim() || result.stdout || ""),
|
||||
});
|
||||
}
|
||||
if (maxRssMb == null) {
|
||||
report.status = "fail";
|
||||
report.error = `${testCase.label} did not report max RSS`;
|
||||
return Object.assign(report, {
|
||||
failureMessage: formatFailure(testCase, report.error, stderr),
|
||||
});
|
||||
}
|
||||
if (matrixBootstrapWarning) {
|
||||
report.status = "fail";
|
||||
report.error = `${testCase.label} triggered Matrix crypto bootstrap during startup`;
|
||||
return Object.assign(report, {
|
||||
failureMessage: formatFailure(testCase, report.error),
|
||||
});
|
||||
}
|
||||
if (maxRssMb > testCase.limitMb) {
|
||||
report.status = "fail";
|
||||
report.error = `${testCase.label} used ${maxRssMb.toFixed(1)} MB RSS (limit ${
|
||||
testCase.limitMb
|
||||
} MB)`;
|
||||
return Object.assign(report, {
|
||||
failureMessage: formatFailure(testCase, report.error),
|
||||
});
|
||||
}
|
||||
|
||||
console.log(
|
||||
`[startup-memory] ${testCase.label}: ${maxRssMb.toFixed(1)} MB RSS (limit ${testCase.limitMb} MB)`,
|
||||
);
|
||||
return report;
|
||||
}
|
||||
|
||||
function writeReport(options, results) {
|
||||
const failed = results.filter((result) => result.status !== "pass");
|
||||
const report = {
|
||||
generatedAt: new Date().toISOString(),
|
||||
platform: process.platform,
|
||||
repoRoot,
|
||||
status: failed.length === 0 ? "pass" : "fail",
|
||||
results: results.map(({ failureMessage: _failureMessage, ...result }) => result),
|
||||
};
|
||||
const lines = [
|
||||
"# OpenClaw Startup Memory",
|
||||
"",
|
||||
`Generated: ${report.generatedAt}`,
|
||||
"",
|
||||
`Status: ${report.status}`,
|
||||
"",
|
||||
...results.map(
|
||||
(result) =>
|
||||
`- ${result.label}: ${result.status} RSS ${formatMb(result.maxRssMb)} / ${formatMb(
|
||||
result.limitMb,
|
||||
)}`,
|
||||
),
|
||||
"",
|
||||
];
|
||||
if (failed.length > 0) {
|
||||
lines.push(
|
||||
"## Failures",
|
||||
"",
|
||||
...failed.map((result) => `- ${result.label}: ${result.error ?? "unknown failure"}`),
|
||||
"",
|
||||
);
|
||||
}
|
||||
mkdirSync(path.dirname(options.jsonPath), { recursive: true });
|
||||
mkdirSync(path.dirname(options.summaryPath), { recursive: true });
|
||||
writeFileSync(options.jsonPath, `${JSON.stringify(report, null, 2)}\n`, "utf8");
|
||||
writeFileSync(options.summaryPath, `${lines.join("\n")}\n`, "utf8");
|
||||
}
|
||||
|
||||
function runStartupMemoryCheck(argv = process.argv.slice(2), params = {}) {
|
||||
const platform = params.platform ?? process.platform;
|
||||
if (platform !== "linux" && platform !== "darwin") {
|
||||
console.log(`[startup-memory] Skipping on unsupported platform: ${platform}`);
|
||||
return { skipped: true, results: [] };
|
||||
}
|
||||
const options = parseArgs(argv);
|
||||
tmpHome = mkdtempSync(path.join(os.tmpdir(), "openclaw-startup-memory-"));
|
||||
rssHookPath = path.join(tmpHome, "measure-rss.mjs");
|
||||
writeFileSync(
|
||||
rssHookPath,
|
||||
[
|
||||
"process.on('exit', () => {",
|
||||
" const usage = typeof process.resourceUsage === 'function' ? process.resourceUsage() : null;",
|
||||
` if (usage && typeof usage.maxRSS === 'number') console.error('${MAX_RSS_MARKER}' + String(usage.maxRSS));`,
|
||||
"});",
|
||||
"",
|
||||
].join("\n"),
|
||||
"utf8",
|
||||
);
|
||||
const results = [];
|
||||
try {
|
||||
for (const testCase of cases) {
|
||||
results.push(runCase(testCase, params));
|
||||
}
|
||||
} finally {
|
||||
writeReport(options, results);
|
||||
if (tmpHome) {
|
||||
rmSync(tmpHome, { recursive: true, force: true });
|
||||
tmpHome = null;
|
||||
rssHookPath = null;
|
||||
}
|
||||
}
|
||||
|
||||
const failure = results.find((result) => result.status !== "pass");
|
||||
if (failure?.failureMessage) {
|
||||
throw new Error(failure.failureMessage);
|
||||
}
|
||||
return { skipped: false, results };
|
||||
}
|
||||
|
||||
/**
|
||||
* Test-only access to pure startup memory helper functions.
|
||||
*/
|
||||
export const testing = {
|
||||
cases,
|
||||
nodeImportSpecifierForPath,
|
||||
parseArgs,
|
||||
readPositiveIntEnv,
|
||||
readPositiveNumberEnv,
|
||||
repoRoot,
|
||||
resolveDefaultLimitsMb,
|
||||
runCase,
|
||||
runStartupMemoryCheck,
|
||||
};
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
try {
|
||||
runStartupMemoryCheck();
|
||||
} catch (error) {
|
||||
console.error(error instanceof Error ? error.message : String(error));
|
||||
process.exitCode = 1;
|
||||
}
|
||||
}
|
||||
160
scripts/check-codex-app-server-protocol.ts
Normal file
160
scripts/check-codex-app-server-protocol.ts
Normal file
@@ -0,0 +1,160 @@
|
||||
// Check Codex App Server Protocol script supports OpenClaw repository automation.
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import {
|
||||
generateExperimentalCodexAppServerProtocolSource,
|
||||
normalizeCodexAppServerProtocolJsonText,
|
||||
selectedCodexAppServerJsonSchemas,
|
||||
} from "./lib/codex-app-server-protocol-source.js";
|
||||
|
||||
const generatedRoot = path.resolve(
|
||||
process.cwd(),
|
||||
"extensions/codex/src/app-server/protocol-generated",
|
||||
);
|
||||
|
||||
const checks: Array<{ file: string; snippets: string[] }> = [
|
||||
{
|
||||
file: "ServerRequest.ts",
|
||||
snippets: [
|
||||
'"item/commandExecution/requestApproval"',
|
||||
'"item/fileChange/requestApproval"',
|
||||
'"item/permissions/requestApproval"',
|
||||
'"item/tool/call"',
|
||||
],
|
||||
},
|
||||
{
|
||||
file: "v2/ThreadItem.ts",
|
||||
snippets: [
|
||||
'type: "contextCompaction"',
|
||||
'type: "dynamicToolCall"',
|
||||
'type: "commandExecution"',
|
||||
'type: "mcpToolCall"',
|
||||
],
|
||||
},
|
||||
{
|
||||
file: "v2/DynamicToolSpec.ts",
|
||||
snippets: [
|
||||
'"function"',
|
||||
"& DynamicToolFunctionSpec",
|
||||
'"namespace"',
|
||||
"& DynamicToolNamespaceSpec",
|
||||
],
|
||||
},
|
||||
{
|
||||
file: "v2/DynamicToolFunctionSpec.ts",
|
||||
snippets: ["name: string", "description: string", "inputSchema: JsonValue"],
|
||||
},
|
||||
{
|
||||
file: "v2/DynamicToolNamespaceSpec.ts",
|
||||
snippets: ["name: string", "description: string", "tools: Array<DynamicToolNamespaceTool>"],
|
||||
},
|
||||
{
|
||||
file: "v2/CommandExecutionApprovalDecision.ts",
|
||||
snippets: ['"accept"', '"acceptForSession"', '"decline"', '"cancel"'],
|
||||
},
|
||||
{
|
||||
file: "v2/Account.ts",
|
||||
snippets: ['type: "apiKey"', 'type: "chatgpt"', 'type: "amazonBedrock"'],
|
||||
},
|
||||
{
|
||||
file: "v2/ThreadStartParams.ts",
|
||||
snippets: [
|
||||
"permissions?: string | null",
|
||||
"dynamicTools?: Array<DynamicToolSpec> | null",
|
||||
"experimentalRawEvents",
|
||||
],
|
||||
},
|
||||
{
|
||||
file: "v2/TurnStartParams.ts",
|
||||
snippets: ["permissions?: string | null", "serviceTier?: string | null"],
|
||||
},
|
||||
{
|
||||
file: "ReviewDecision.ts",
|
||||
snippets: ['"approved"', '"approved_for_session"', '"denied"', '"abort"'],
|
||||
},
|
||||
{
|
||||
file: "v2/PlanDeltaNotification.ts",
|
||||
snippets: ["itemId: string", "delta: string"],
|
||||
},
|
||||
{
|
||||
file: "v2/TurnPlanUpdatedNotification.ts",
|
||||
snippets: ["explanation: string | null", "plan: Array<TurnPlanStep>"],
|
||||
},
|
||||
];
|
||||
|
||||
const failures: string[] = [];
|
||||
await main().catch((error: unknown) => {
|
||||
console.error(error instanceof Error ? error.message : String(error));
|
||||
process.exit(1);
|
||||
});
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const source = await generateExperimentalCodexAppServerProtocolSource();
|
||||
|
||||
try {
|
||||
await compareGeneratedProtocolMirror(source.jsonRoot);
|
||||
|
||||
for (const check of checks) {
|
||||
const filePath = path.join(source.typescriptRoot, check.file);
|
||||
let text: string;
|
||||
try {
|
||||
text = await fs.readFile(filePath, "utf8");
|
||||
} catch (error) {
|
||||
failures.push(`${check.file}: missing (${String(error)})`);
|
||||
continue;
|
||||
}
|
||||
for (const snippet of check.snippets) {
|
||||
if (!text.includes(snippet)) {
|
||||
failures.push(`${check.file}: missing ${snippet}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
await source.cleanup();
|
||||
}
|
||||
|
||||
if (failures.length > 0) {
|
||||
console.error("Codex app-server generated protocol drift:");
|
||||
for (const failure of failures) {
|
||||
console.error(`- ${failure}`);
|
||||
}
|
||||
console.error(
|
||||
`Run \`pnpm codex-app-server:protocol:sync\` after refreshing the Codex checkout at ${source.codexRepo}.`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log(
|
||||
`Codex app-server generated protocol matches OpenClaw bridge assumptions: ${source.codexRepo}`,
|
||||
);
|
||||
}
|
||||
|
||||
async function compareGeneratedProtocolMirror(sourceJsonRoot: string): Promise<void> {
|
||||
for (const schema of selectedCodexAppServerJsonSchemas) {
|
||||
const sourcePath = path.join(sourceJsonRoot, schema);
|
||||
const targetPath = path.join(generatedRoot, "json", schema);
|
||||
let sourceValue: string;
|
||||
let target: string;
|
||||
try {
|
||||
sourceValue = await fs.readFile(sourcePath, "utf8");
|
||||
} catch (error) {
|
||||
failures.push(
|
||||
`protocol-generated/json/${schema}: missing upstream schema (${String(error)})`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
target = await fs.readFile(targetPath, "utf8");
|
||||
} catch (error) {
|
||||
failures.push(`protocol-generated/json/${schema}: missing local schema (${String(error)})`);
|
||||
continue;
|
||||
}
|
||||
if (normalizeJsonSchema(sourceValue) !== normalizeJsonSchema(target)) {
|
||||
failures.push(`protocol-generated/json/${schema}: differs from source schema`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function normalizeJsonSchema(sourceLocal: string): string {
|
||||
return normalizeCodexAppServerProtocolJsonText(sourceLocal);
|
||||
}
|
||||
81
scripts/check-composite-action-input-interpolation.py
Normal file
81
scripts/check-composite-action-input-interpolation.py
Normal file
@@ -0,0 +1,81 @@
|
||||
#!/usr/bin/env python3
|
||||
from __future__ import annotations
|
||||
|
||||
import pathlib
|
||||
import re
|
||||
import sys
|
||||
|
||||
|
||||
INPUT_INTERPOLATION_RE = re.compile(r"\$\{\{\s*inputs\.")
|
||||
RUN_LINE_RE = re.compile(r"^(\s*)run:\s*(.*)$")
|
||||
USING_COMPOSITE_RE = re.compile(r"^\s*using:\s*composite\s*$", re.MULTILINE)
|
||||
|
||||
|
||||
def indentation(line: str) -> int:
|
||||
return len(line) - len(line.lstrip(" "))
|
||||
|
||||
|
||||
def scan_file(path: pathlib.Path) -> list[tuple[int, str]]:
|
||||
text = path.read_text(encoding="utf-8")
|
||||
if not USING_COMPOSITE_RE.search(text):
|
||||
return []
|
||||
|
||||
lines = text.splitlines()
|
||||
violations: list[tuple[int, str]] = []
|
||||
line_count = len(lines)
|
||||
index = 0
|
||||
|
||||
while index < line_count:
|
||||
line = lines[index]
|
||||
match = RUN_LINE_RE.match(line)
|
||||
if not match:
|
||||
index += 1
|
||||
continue
|
||||
|
||||
run_indent = len(match.group(1))
|
||||
run_value = match.group(2).strip()
|
||||
line_no = index + 1
|
||||
|
||||
if run_value and run_value[0] not in ("|", ">"):
|
||||
if INPUT_INTERPOLATION_RE.search(run_value):
|
||||
violations.append((line_no, line.strip()))
|
||||
index += 1
|
||||
continue
|
||||
|
||||
index += 1
|
||||
while index < line_count:
|
||||
script_line = lines[index]
|
||||
if script_line.strip() == "":
|
||||
index += 1
|
||||
continue
|
||||
if indentation(script_line) <= run_indent:
|
||||
break
|
||||
if INPUT_INTERPOLATION_RE.search(script_line):
|
||||
violations.append((index + 1, script_line.strip()))
|
||||
index += 1
|
||||
|
||||
return violations
|
||||
|
||||
|
||||
def main() -> int:
|
||||
root = pathlib.Path(".github/actions")
|
||||
files = sorted(root.rglob("action.y*ml"))
|
||||
all_violations: list[tuple[pathlib.Path, int, str]] = []
|
||||
|
||||
for file_path in files:
|
||||
for line_no, line in scan_file(file_path):
|
||||
all_violations.append((file_path, line_no, line))
|
||||
|
||||
if all_violations:
|
||||
print("Disallowed direct inputs interpolation in composite run blocks:")
|
||||
for file_path, line_no, line in all_violations:
|
||||
print(f"- {file_path}:{line_no}: {line}")
|
||||
print("Use env: and reference shell variables instead.")
|
||||
return 1
|
||||
|
||||
print("No direct inputs interpolation found in composite run blocks.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
9808
scripts/check-database-first-legacy-stores.mjs
Normal file
9808
scripts/check-database-first-legacy-stores.mjs
Normal file
File diff suppressed because it is too large
Load Diff
426
scripts/check-deadcode-unused-files.mjs
Normal file
426
scripts/check-deadcode-unused-files.mjs
Normal file
@@ -0,0 +1,426 @@
|
||||
#!/usr/bin/env node
|
||||
// Runs knip unused-file detection and compares results to the allowlist.
|
||||
import { spawn } from "node:child_process";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import {
|
||||
KNIP_OPTIONAL_UNUSED_FILE_ALLOWLIST,
|
||||
KNIP_UNUSED_FILE_ALLOWLIST,
|
||||
} from "./deadcode-unused-files.allowlist.mjs";
|
||||
import { createPnpmRunnerSpawnSpec } from "./pnpm-runner.mjs";
|
||||
|
||||
const KNIP_VERSION = "6.8.0";
|
||||
/**
|
||||
* Timeout for the unused-file knip child process.
|
||||
*/
|
||||
export const KNIP_TIMEOUT_MS = 10 * 60 * 1000;
|
||||
/**
|
||||
* Grace period before force-killing a timed-out knip child process.
|
||||
*/
|
||||
export const KNIP_KILL_GRACE_MS = 5_000;
|
||||
const KNIP_PROCESS_TREE_EXIT_POLL_MS = 25;
|
||||
const KNIP_POST_FORCE_KILL_WAIT_MS = 1_000;
|
||||
/**
|
||||
* Heartbeat interval used while knip runs without output.
|
||||
*/
|
||||
export const KNIP_HEARTBEAT_MS = 60_000;
|
||||
/**
|
||||
* Maximum buffered knip output retained for diagnostics.
|
||||
*/
|
||||
export const KNIP_MAX_BUFFER_BYTES = 16 * 1024 * 1024;
|
||||
const KNIP_ARGS = [
|
||||
"--config",
|
||||
"config/knip.config.ts",
|
||||
"--production",
|
||||
"--no-progress",
|
||||
"--reporter",
|
||||
"compact",
|
||||
"--files",
|
||||
"--no-config-hints",
|
||||
];
|
||||
|
||||
function normalizeRepoPath(value) {
|
||||
return value.replaceAll("\\", "/").replace(/^\.\//u, "");
|
||||
}
|
||||
|
||||
function uniqueSorted(values) {
|
||||
return [...new Set(values.map(normalizeRepoPath))].toSorted((left, right) =>
|
||||
left.localeCompare(right),
|
||||
);
|
||||
}
|
||||
|
||||
function isLikelyRepoFilePath(value) {
|
||||
return /^(apps|docs|extensions|packages|scripts|src|test|ui)\//u.test(normalizeRepoPath(value));
|
||||
}
|
||||
|
||||
/**
|
||||
* Parses compact knip output into unused file paths.
|
||||
*/
|
||||
export function parseKnipCompactUnusedFiles(output) {
|
||||
const files = [];
|
||||
let inUnusedFilesSection = false;
|
||||
let sawUnusedFilesSection = false;
|
||||
|
||||
for (const line of output.split(/\r?\n/u)) {
|
||||
if (/^Unused files \(\d+\)$/u.test(line)) {
|
||||
inUnusedFilesSection = true;
|
||||
sawUnusedFilesSection = true;
|
||||
continue;
|
||||
}
|
||||
if (inUnusedFilesSection && line.trim() === "") {
|
||||
break;
|
||||
}
|
||||
|
||||
const separatorIndex = line.lastIndexOf(": ");
|
||||
if (separatorIndex === -1) {
|
||||
continue;
|
||||
}
|
||||
if (sawUnusedFilesSection && !inUnusedFilesSection) {
|
||||
continue;
|
||||
}
|
||||
const file = line.slice(separatorIndex + 2).trim();
|
||||
if (isLikelyRepoFilePath(file)) {
|
||||
files.push(file);
|
||||
}
|
||||
}
|
||||
|
||||
return uniqueSorted(files);
|
||||
}
|
||||
|
||||
/**
|
||||
* Compares detected unused files against the checked-in allowlist.
|
||||
*/
|
||||
export function compareUnusedFilesToAllowlist(
|
||||
actualFiles,
|
||||
allowlistFiles,
|
||||
optionalAllowlistFiles = [],
|
||||
) {
|
||||
const actual = uniqueSorted(actualFiles);
|
||||
const allowed = uniqueSorted(allowlistFiles);
|
||||
const optionalAllowed = uniqueSorted(optionalAllowlistFiles);
|
||||
const allowedOrOptionalSet = new Set([...allowed, ...optionalAllowed]);
|
||||
const actualSet = new Set(actual);
|
||||
|
||||
return {
|
||||
actual,
|
||||
allowed,
|
||||
unexpected: actual.filter((file) => !allowedOrOptionalSet.has(file)),
|
||||
stale: allowed.filter((file) => !actualSet.has(file)),
|
||||
duplicateAllowedCount: allowlistFiles.length - new Set(allowlistFiles).size,
|
||||
allowlistIsSorted:
|
||||
JSON.stringify(allowlistFiles.map(normalizeRepoPath)) === JSON.stringify(allowed),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Formats unused-file allowlist drift for CLI output.
|
||||
*/
|
||||
export function formatUnusedFileComparison(comparison) {
|
||||
const lines = [];
|
||||
if (!comparison.allowlistIsSorted) {
|
||||
lines.push("deadcode unused-file allowlist is not sorted.");
|
||||
}
|
||||
if (comparison.duplicateAllowedCount > 0) {
|
||||
lines.push(
|
||||
`deadcode unused-file allowlist contains ${comparison.duplicateAllowedCount} duplicate entr${
|
||||
comparison.duplicateAllowedCount === 1 ? "y" : "ies"
|
||||
}.`,
|
||||
);
|
||||
}
|
||||
if (comparison.unexpected.length > 0) {
|
||||
lines.push("Unexpected unused files:");
|
||||
lines.push(...comparison.unexpected.map((file) => ` ${file}`));
|
||||
}
|
||||
if (comparison.stale.length > 0) {
|
||||
lines.push("Stale allowlist entries:");
|
||||
lines.push(...comparison.stale.map((file) => ` ${file}`));
|
||||
}
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
function spawnErrorCode(error) {
|
||||
return error && typeof error === "object" && "code" in error ? String(error.code) : undefined;
|
||||
}
|
||||
|
||||
function signalProcessTree(child, signal) {
|
||||
if (!child.pid) {
|
||||
return;
|
||||
}
|
||||
try {
|
||||
if (process.platform === "win32") {
|
||||
process.kill(child.pid, signal);
|
||||
} else {
|
||||
process.kill(-child.pid, signal);
|
||||
}
|
||||
} catch {
|
||||
// The child may have exited between the timeout and signal delivery.
|
||||
}
|
||||
}
|
||||
|
||||
function processTreeAlive(child) {
|
||||
if (!child.pid) {
|
||||
return false;
|
||||
}
|
||||
if (process.platform === "win32") {
|
||||
return child.exitCode === null && child.signalCode === null;
|
||||
}
|
||||
try {
|
||||
process.kill(-child.pid, 0);
|
||||
return true;
|
||||
} catch (error) {
|
||||
return error?.code === "EPERM";
|
||||
}
|
||||
}
|
||||
|
||||
async function waitForProcessTreeExit(child, timeoutMs) {
|
||||
const deadlineAt = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadlineAt) {
|
||||
if (!processTreeAlive(child)) {
|
||||
return true;
|
||||
}
|
||||
await new Promise((resolvePoll) => {
|
||||
setTimeout(resolvePoll, KNIP_PROCESS_TREE_EXIT_POLL_MS);
|
||||
});
|
||||
}
|
||||
return !processTreeAlive(child);
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs knip and returns parsed unused-file results.
|
||||
*/
|
||||
export async function runKnipUnusedFiles(params = {}) {
|
||||
const run = params.spawnCommand ?? spawn;
|
||||
const timeoutMs = params.timeoutMs ?? KNIP_TIMEOUT_MS;
|
||||
const heartbeatMs = params.heartbeatMs ?? KNIP_HEARTBEAT_MS;
|
||||
const maxBufferBytes = params.maxBufferBytes ?? KNIP_MAX_BUFFER_BYTES;
|
||||
const killGraceMs = params.killGraceMs ?? KNIP_KILL_GRACE_MS;
|
||||
const writeStatus = params.writeStatus ?? ((message) => process.stderr.write(`${message}\n`));
|
||||
const args = [
|
||||
"--config.minimum-release-age=0",
|
||||
"dlx",
|
||||
"--package",
|
||||
`knip@${KNIP_VERSION}`,
|
||||
"knip",
|
||||
...KNIP_ARGS,
|
||||
];
|
||||
|
||||
return await new Promise((resolve) => {
|
||||
const startedAt = Date.now();
|
||||
let settled = false;
|
||||
let timedOut = false;
|
||||
let bufferExceeded = false;
|
||||
let outputBytes = 0;
|
||||
const output = [];
|
||||
let killTimer;
|
||||
let exitStatus = null;
|
||||
let exitSignal = null;
|
||||
|
||||
const pnpm = createPnpmRunnerSpawnSpec({
|
||||
detached: process.platform !== "win32",
|
||||
env: params.env,
|
||||
nodeExecPath: params.nodeExecPath,
|
||||
npmExecPath: params.npmExecPath,
|
||||
platform: params.platform,
|
||||
pnpmArgs: args,
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
const child = run(pnpm.command, pnpm.args, {
|
||||
...pnpm.options,
|
||||
detached: process.platform !== "win32",
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
const parentSignalHandlers = [];
|
||||
const cleanupParentSignalHandlers = () => {
|
||||
for (const { signal, handler } of parentSignalHandlers) {
|
||||
process.off(signal, handler);
|
||||
}
|
||||
parentSignalHandlers.length = 0;
|
||||
};
|
||||
const relayParentSignal = (signal) => {
|
||||
const handler = () => {
|
||||
signalProcessTree(child, signal);
|
||||
signalProcessTree(child, "SIGKILL");
|
||||
cleanupParentSignalHandlers();
|
||||
process.kill(process.pid, signal);
|
||||
};
|
||||
parentSignalHandlers.push({ signal, handler });
|
||||
process.once(signal, handler);
|
||||
};
|
||||
if (process.platform !== "win32") {
|
||||
relayParentSignal("SIGINT");
|
||||
relayParentSignal("SIGTERM");
|
||||
relayParentSignal("SIGHUP");
|
||||
}
|
||||
|
||||
const heartbeatTimer = setInterval(() => {
|
||||
writeStatus(
|
||||
`[deadcode] Knip unused-file scan still running after ${Math.round(
|
||||
(Date.now() - startedAt) / 1000,
|
||||
)}s.`,
|
||||
);
|
||||
}, heartbeatMs);
|
||||
|
||||
const timeoutTimer = setTimeout(() => {
|
||||
timedOut = true;
|
||||
clearInterval(heartbeatTimer);
|
||||
writeStatus(
|
||||
`[deadcode] Knip unused-file scan timed out after ${Math.round(timeoutMs / 1000)}s; terminating.`,
|
||||
);
|
||||
signalProcessTree(child, "SIGTERM");
|
||||
killTimer = setTimeout(() => signalProcessTree(child, "SIGKILL"), killGraceMs);
|
||||
}, timeoutMs);
|
||||
|
||||
const finish = (result) => {
|
||||
if (settled) {
|
||||
return;
|
||||
}
|
||||
settled = true;
|
||||
clearTimeout(timeoutTimer);
|
||||
clearInterval(heartbeatTimer);
|
||||
clearTimeout(killTimer);
|
||||
cleanupParentSignalHandlers();
|
||||
resolve({
|
||||
...result,
|
||||
output: output.join(""),
|
||||
});
|
||||
};
|
||||
const finishAfterProcessTreeCleanup = async (result) => {
|
||||
if (processTreeAlive(child)) {
|
||||
await waitForProcessTreeExit(child, killGraceMs);
|
||||
}
|
||||
if (processTreeAlive(child)) {
|
||||
signalProcessTree(child, "SIGKILL");
|
||||
await waitForProcessTreeExit(child, KNIP_POST_FORCE_KILL_WAIT_MS);
|
||||
}
|
||||
finish(result);
|
||||
};
|
||||
|
||||
const appendOutput = (chunk) => {
|
||||
if (settled) {
|
||||
return;
|
||||
}
|
||||
if (bufferExceeded) {
|
||||
return;
|
||||
}
|
||||
const buffer = Buffer.isBuffer(chunk) ? chunk : Buffer.from(String(chunk));
|
||||
const remainingBytes = maxBufferBytes - outputBytes;
|
||||
if (buffer.length <= remainingBytes) {
|
||||
output.push(buffer.toString("utf8"));
|
||||
outputBytes += buffer.length;
|
||||
return;
|
||||
}
|
||||
if (remainingBytes > 0) {
|
||||
output.push(buffer.subarray(0, remainingBytes).toString("utf8"));
|
||||
outputBytes = maxBufferBytes;
|
||||
}
|
||||
if (!bufferExceeded) {
|
||||
bufferExceeded = true;
|
||||
writeStatus(
|
||||
`[deadcode] Knip unused-file scan exceeded ${maxBufferBytes} output bytes; terminating.`,
|
||||
);
|
||||
child.stdout?.off?.("data", appendOutput);
|
||||
child.stderr?.off?.("data", appendOutput);
|
||||
child.stdout?.destroy?.();
|
||||
child.stderr?.destroy?.();
|
||||
clearInterval(heartbeatTimer);
|
||||
signalProcessTree(child, "SIGTERM");
|
||||
killTimer = setTimeout(() => signalProcessTree(child, "SIGKILL"), killGraceMs);
|
||||
}
|
||||
};
|
||||
|
||||
child.stdout?.on("data", appendOutput);
|
||||
child.stderr?.on("data", appendOutput);
|
||||
child.on("error", (error) =>
|
||||
finish({
|
||||
errorCode: spawnErrorCode(error),
|
||||
errorMessage: error.message,
|
||||
signal: null,
|
||||
status: null,
|
||||
}),
|
||||
);
|
||||
child.on("exit", (status, signal) => {
|
||||
exitStatus = status;
|
||||
exitSignal = signal;
|
||||
});
|
||||
child.on("close", (status, signal) => {
|
||||
exitStatus = exitStatus ?? status;
|
||||
exitSignal = exitSignal ?? signal;
|
||||
const elapsedSeconds = Math.round((Date.now() - startedAt) / 1000);
|
||||
if (timedOut) {
|
||||
void finishAfterProcessTreeCleanup({
|
||||
errorCode: "ETIMEDOUT",
|
||||
errorMessage: `Knip unused-file scan timed out after ${elapsedSeconds}s`,
|
||||
signal: exitSignal,
|
||||
status: exitStatus,
|
||||
});
|
||||
return;
|
||||
}
|
||||
if (bufferExceeded) {
|
||||
void finishAfterProcessTreeCleanup({
|
||||
errorCode: "ENOBUFS",
|
||||
errorMessage: `Knip unused-file scan exceeded ${maxBufferBytes} output bytes`,
|
||||
signal: exitSignal,
|
||||
status: exitStatus,
|
||||
});
|
||||
return;
|
||||
}
|
||||
finish({
|
||||
errorCode: undefined,
|
||||
errorMessage: undefined,
|
||||
signal: exitSignal,
|
||||
status: exitStatus,
|
||||
});
|
||||
});
|
||||
});
|
||||
}
|
||||
/**
|
||||
* Checks detected unused files against the current allowlist.
|
||||
*/
|
||||
export function checkUnusedFiles(
|
||||
output,
|
||||
allowlistFiles = KNIP_UNUSED_FILE_ALLOWLIST,
|
||||
optionalAllowlistFiles = KNIP_OPTIONAL_UNUSED_FILE_ALLOWLIST,
|
||||
) {
|
||||
const actual = parseKnipCompactUnusedFiles(output);
|
||||
const comparison = compareUnusedFilesToAllowlist(actual, allowlistFiles, optionalAllowlistFiles);
|
||||
return {
|
||||
ok:
|
||||
comparison.allowlistIsSorted &&
|
||||
comparison.duplicateAllowedCount === 0 &&
|
||||
comparison.unexpected.length === 0 &&
|
||||
comparison.stale.length === 0,
|
||||
comparison,
|
||||
message: formatUnusedFileComparison(comparison),
|
||||
};
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const result = await runKnipUnusedFiles();
|
||||
if (result.errorCode || result.status === null) {
|
||||
console.error(
|
||||
`deadcode unused-file scan failed: ${result.errorCode ?? result.signal ?? "unknown"}${
|
||||
result.errorMessage ? `: ${result.errorMessage}` : ""
|
||||
}`,
|
||||
);
|
||||
if (result.output) {
|
||||
console.error(result.output);
|
||||
}
|
||||
process.exitCode = 1;
|
||||
return;
|
||||
}
|
||||
const check = checkUnusedFiles(result.output);
|
||||
if (!check.ok) {
|
||||
if (check.message) {
|
||||
console.error(check.message);
|
||||
}
|
||||
process.exitCode = 1;
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(
|
||||
`[deadcode] Knip unused-file allowlist matched ${check.comparison.actual.length} intentional entries.`,
|
||||
);
|
||||
}
|
||||
|
||||
if (process.argv[1] === fileURLToPath(import.meta.url)) {
|
||||
await main();
|
||||
}
|
||||
173
scripts/check-dependency-pins.mjs
Normal file
173
scripts/check-dependency-pins.mjs
Normal file
@@ -0,0 +1,173 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Audits patched dependency pins for exact versions and drift.
|
||||
import { execFileSync } from "node:child_process";
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import YAML from "yaml";
|
||||
|
||||
const PACKAGE_DEPENDENCY_SECTIONS = ["dependencies", "devDependencies", "optionalDependencies"];
|
||||
const WORKSPACE_DEPENDENCY_SECTIONS = ["overrides"];
|
||||
const EXACT_SEMVER_PATTERN = /^\d+\.\d+\.\d+(?:-[0-9A-Za-z.-]+)?(?:\+[0-9A-Za-z.-]+)?$/u;
|
||||
const EXACT_NPM_ALIAS_PATTERN =
|
||||
/^npm:(?:@[^/\s]+\/)?[^@\s]+@\d+\.\d+\.\d+(?:-[0-9A-Za-z.-]+)?(?:\+[0-9A-Za-z.-]+)?$/u;
|
||||
const PINNED_GIT_PATTERN = /(?:#|\/commit\/)[0-9a-f]{40}$/iu;
|
||||
const PINNED_GITHUB_TARBALL_PATTERN =
|
||||
/^https:\/\/codeload\.github\.com\/[^/\s]+\/[^/\s]+\/tar\.gz\/[0-9a-f]{40}$/iu;
|
||||
|
||||
function listTrackedPackageJsonFiles(cwd) {
|
||||
return execFileSync("git", ["ls-files", "-z", "--", "*package.json"], {
|
||||
cwd,
|
||||
encoding: "utf8",
|
||||
})
|
||||
.split("\0")
|
||||
.filter(Boolean)
|
||||
.toSorted((left, right) => left.localeCompare(right));
|
||||
}
|
||||
|
||||
function readJson(filePath) {
|
||||
return JSON.parse(fs.readFileSync(filePath, "utf8"));
|
||||
}
|
||||
|
||||
function readTrackedJson(cwd, relativePath) {
|
||||
const filePath = path.join(cwd, relativePath);
|
||||
if (fs.existsSync(filePath)) {
|
||||
return readJson(filePath);
|
||||
}
|
||||
return JSON.parse(
|
||||
execFileSync("git", ["show", `:${relativePath}`], {
|
||||
cwd,
|
||||
encoding: "utf8",
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
function isAllowedPinnedSpec(spec) {
|
||||
if (typeof spec !== "string") {
|
||||
return false;
|
||||
}
|
||||
if (EXACT_SEMVER_PATTERN.test(spec) || EXACT_NPM_ALIAS_PATTERN.test(spec)) {
|
||||
return true;
|
||||
}
|
||||
if (spec === "workspace:*" || spec.startsWith("file:") || spec.startsWith("link:")) {
|
||||
return true;
|
||||
}
|
||||
if (/^(?:git\+|github:|gitlab:|bitbucket:)/u.test(spec)) {
|
||||
return PINNED_GIT_PATTERN.test(spec);
|
||||
}
|
||||
if (PINNED_GITHUB_TARBALL_PATTERN.test(spec)) {
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function collectPackageJsonViolations(cwd) {
|
||||
const violations = [];
|
||||
for (const relativePath of listTrackedPackageJsonFiles(cwd)) {
|
||||
const packageJson = readTrackedJson(cwd, relativePath);
|
||||
for (const section of PACKAGE_DEPENDENCY_SECTIONS) {
|
||||
for (const [name, spec] of Object.entries(packageJson[section] ?? {})) {
|
||||
if (!isAllowedPinnedSpec(spec)) {
|
||||
violations.push({ file: relativePath, section, name, spec });
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return violations;
|
||||
}
|
||||
|
||||
function collectDependencyMapViolations(file, section, dependencyMap, violations) {
|
||||
for (const [name, spec] of Object.entries(dependencyMap ?? {})) {
|
||||
if (!isAllowedPinnedSpec(spec)) {
|
||||
violations.push({ file, section, name, spec });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function collectWorkspaceViolations(cwd) {
|
||||
const file = "pnpm-workspace.yaml";
|
||||
const workspacePath = path.join(cwd, file);
|
||||
if (!fs.existsSync(workspacePath)) {
|
||||
return [];
|
||||
}
|
||||
const workspace = YAML.parse(fs.readFileSync(workspacePath, "utf8"));
|
||||
const violations = [];
|
||||
for (const section of WORKSPACE_DEPENDENCY_SECTIONS) {
|
||||
collectDependencyMapViolations(file, section, workspace?.[section], violations);
|
||||
}
|
||||
for (const [packageName, extension] of Object.entries(workspace?.packageExtensions ?? {})) {
|
||||
collectDependencyMapViolations(
|
||||
file,
|
||||
`packageExtensions.${packageName}.dependencies`,
|
||||
extension?.dependencies,
|
||||
violations,
|
||||
);
|
||||
}
|
||||
return violations;
|
||||
}
|
||||
|
||||
/**
|
||||
* Collects dependency pin violations for the current workspace.
|
||||
*/
|
||||
export function collectDependencyPinViolations(cwd = process.cwd()) {
|
||||
return [...collectPackageJsonViolations(cwd), ...collectWorkspaceViolations(cwd)];
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds the full dependency pin audit payload.
|
||||
*/
|
||||
export function collectDependencyPinAudit(cwd = process.cwd()) {
|
||||
const packageJsonFiles = listTrackedPackageJsonFiles(cwd);
|
||||
let packageSpecCount = 0;
|
||||
for (const relativePath of packageJsonFiles) {
|
||||
const packageJson = readTrackedJson(cwd, relativePath);
|
||||
for (const section of PACKAGE_DEPENDENCY_SECTIONS) {
|
||||
packageSpecCount += Object.keys(packageJson[section] ?? {}).length;
|
||||
}
|
||||
}
|
||||
const workspaceViolations = collectWorkspaceViolations(cwd);
|
||||
const violations = [...collectPackageJsonViolations(cwd), ...workspaceViolations];
|
||||
return {
|
||||
packageManifestCount: packageJsonFiles.length,
|
||||
packageSpecCount,
|
||||
violations,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the dependency pin check.
|
||||
*/
|
||||
export async function main() {
|
||||
const audit = collectDependencyPinAudit();
|
||||
const { violations } = audit;
|
||||
if (violations.length === 0) {
|
||||
process.stdout.write(
|
||||
`PASS direct dependency pin guard: checked ${audit.packageSpecCount} directly declared ` +
|
||||
`dependency specs across ${audit.packageManifestCount} tracked package manifests; ` +
|
||||
"0 violations.\n",
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
console.error(
|
||||
`FAIL direct dependency pin guard: ${violations.length} unpinned directly declared ` +
|
||||
"dependency specs found. Direct dependency specs must be pinned exactly outside peer " +
|
||||
"dependency contracts:",
|
||||
);
|
||||
for (const violation of violations) {
|
||||
console.error(
|
||||
`- ${violation.file}:${violation.section}:${violation.name} -> ${JSON.stringify(violation.spec)}`,
|
||||
);
|
||||
}
|
||||
process.exitCode = 1;
|
||||
}
|
||||
|
||||
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
||||
main().catch(
|
||||
/** @param {unknown} error */ (error) => {
|
||||
console.error(error);
|
||||
process.exit(1);
|
||||
},
|
||||
);
|
||||
}
|
||||
209
scripts/check-deprecated-api-usage.mjs
Normal file
209
scripts/check-deprecated-api-usage.mjs
Normal file
@@ -0,0 +1,209 @@
|
||||
#!/usr/bin/env node
|
||||
// Scans source files for usage of deprecated API markers.
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { collectDeprecatedInternalConfigApiViolations } from "./lib/deprecated-config-api-guard.mjs";
|
||||
import { buildDeprecatedPluginSdkModuleSpecifiers } from "./lib/deprecated-plugin-sdk-usage.mjs";
|
||||
import { escapeRegExp } from "./lib/regexp.mjs";
|
||||
|
||||
const repoRoot = process.cwd();
|
||||
|
||||
const sourceExtensions = new Set([".ts", ".tsx", ".js", ".mjs", ".mts"]);
|
||||
const skippedSegments = new Set(["node_modules", "dist", "build", "coverage", ".turbo"]);
|
||||
const skippedFilePatterns = [
|
||||
/\.test\.[cm]?[jt]sx?$/u,
|
||||
/\.spec\.[cm]?[jt]sx?$/u,
|
||||
/\.e2e\.[cm]?[jt]sx?$/u,
|
||||
/\.test-(?:harness|loader|support)\.[cm]?[jt]sx?$/u,
|
||||
/\.contract-test-support\.[cm]?[jt]sx?$/u,
|
||||
/(?:^|\/)test-(?:helpers|support)\.[cm]?[jt]sx?$/u,
|
||||
/(?:^|\/)(?:test-helpers|test-support)\//u,
|
||||
/^extensions\/test-support\//u,
|
||||
/^src\/channels\/plugins\/contracts\/test-helpers\//u,
|
||||
/^src\/plugins\/contracts\/tts-contract-suites\.ts$/u,
|
||||
/\.d\.ts$/u,
|
||||
];
|
||||
|
||||
function toRepoPath(filePath) {
|
||||
return path.relative(repoRoot, filePath).split(path.sep).join("/");
|
||||
}
|
||||
|
||||
function shouldSkipFile(filePath, rule) {
|
||||
const repoPath = toRepoPath(filePath);
|
||||
return (rule.skippedFilePatterns ?? skippedFilePatterns).some((pattern) =>
|
||||
pattern.test(repoPath),
|
||||
);
|
||||
}
|
||||
|
||||
function* walk(dir, rule) {
|
||||
if (!fs.existsSync(dir)) {
|
||||
return;
|
||||
}
|
||||
for (const entry of fs.readdirSync(dir, { withFileTypes: true })) {
|
||||
if (skippedSegments.has(entry.name)) {
|
||||
continue;
|
||||
}
|
||||
const entryPath = path.join(dir, entry.name);
|
||||
if (entry.isDirectory()) {
|
||||
yield* walk(entryPath, rule);
|
||||
continue;
|
||||
}
|
||||
if (!entry.isFile() || !sourceExtensions.has(path.extname(entry.name))) {
|
||||
continue;
|
||||
}
|
||||
if (!shouldSkipFile(entryPath, rule)) {
|
||||
yield entryPath;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function collectIdentifierRuleViolations(rule) {
|
||||
const allowedFiles = new Set(rule.allowedFiles ?? []);
|
||||
const pattern = new RegExp(
|
||||
`\\b(?:${rule.names.map((name) => escapeRegExp(name)).join("|")})\\b`,
|
||||
"gu",
|
||||
);
|
||||
const violations = [];
|
||||
|
||||
for (const root of rule.roots) {
|
||||
for (const filePath of walk(path.join(repoRoot, root), rule)) {
|
||||
const repoPath = toRepoPath(filePath);
|
||||
if (allowedFiles.has(repoPath)) {
|
||||
continue;
|
||||
}
|
||||
const source = fs.readFileSync(filePath, "utf8");
|
||||
for (const match of source.matchAll(pattern)) {
|
||||
const line = source.slice(0, match.index).split("\n").length;
|
||||
violations.push(`${repoPath}:${line}: ${match[0]} (${rule.message})`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return violations;
|
||||
}
|
||||
|
||||
function collectModuleSpecifierRuleViolations(rule) {
|
||||
const allowedFiles = new Set(rule.allowedFiles ?? []);
|
||||
const specifierPattern = rule.moduleSpecifiers
|
||||
.map((specifier) => escapeRegExp(specifier))
|
||||
.join("|");
|
||||
const patterns = [
|
||||
new RegExp(
|
||||
`\\bimport\\s+(?:type\\s+)?(?:[^"']+?\\s+from\\s+)?["'](${specifierPattern})["']`,
|
||||
"gu",
|
||||
),
|
||||
new RegExp(
|
||||
`\\bexport\\s+(?:type\\s+)?(?:\\*\\s+from\\s+|[^"']+?\\s+from\\s+)["'](${specifierPattern})["']`,
|
||||
"gu",
|
||||
),
|
||||
new RegExp(`\\bimport\\(\\s*["'](${specifierPattern})["']\\s*\\)`, "gu"),
|
||||
];
|
||||
const violations = [];
|
||||
|
||||
for (const root of rule.roots) {
|
||||
for (const filePath of walk(path.join(repoRoot, root), rule)) {
|
||||
const repoPath = toRepoPath(filePath);
|
||||
if (allowedFiles.has(repoPath)) {
|
||||
continue;
|
||||
}
|
||||
const source = fs.readFileSync(filePath, "utf8");
|
||||
for (const pattern of patterns) {
|
||||
for (const match of source.matchAll(pattern)) {
|
||||
const line = source.slice(0, match.index).split("\n").length;
|
||||
violations.push(`${repoPath}:${line}: ${match[1]} (${rule.message})`);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return violations;
|
||||
}
|
||||
|
||||
function collectRuleViolations(rule) {
|
||||
if (rule.collect) {
|
||||
return rule.collect();
|
||||
}
|
||||
if (rule.moduleSpecifiers) {
|
||||
return collectModuleSpecifierRuleViolations(rule);
|
||||
}
|
||||
return collectIdentifierRuleViolations(rule);
|
||||
}
|
||||
|
||||
const rules = [
|
||||
{
|
||||
id: "internal-config-api",
|
||||
collect: () => collectDeprecatedInternalConfigApiViolations(),
|
||||
},
|
||||
{
|
||||
id: "plugin-sdk-compat-subpaths",
|
||||
roots: ["src", "packages"],
|
||||
moduleSpecifiers: buildDeprecatedPluginSdkModuleSpecifiers(),
|
||||
message: "use focused non-deprecated plugin SDK subpaths",
|
||||
},
|
||||
{
|
||||
id: "extension-plugin-sdk-compat-subpaths",
|
||||
roots: ["extensions"],
|
||||
moduleSpecifiers: buildDeprecatedPluginSdkModuleSpecifiers(),
|
||||
message: "extensions must use focused non-deprecated plugin SDK subpaths",
|
||||
},
|
||||
{
|
||||
id: "message-api",
|
||||
roots: ["src", "extensions", "packages"],
|
||||
names: [
|
||||
"deliverOutboundPayloads",
|
||||
"dispatchChannelMessageReplyWithBase",
|
||||
"recordChannelMessageReplyDispatch",
|
||||
"buildChannelMessageReplyDispatchBase",
|
||||
"hasFinalChannelMessageReplyDispatch",
|
||||
"hasVisibleChannelMessageReplyDispatch",
|
||||
"resolveChannelMessageReplyDispatchCounts",
|
||||
"createChannelTurnReplyPipeline",
|
||||
"deliverDurableInboundReplyPayload",
|
||||
],
|
||||
allowedFiles: [
|
||||
"src/channels/turn/durable-delivery.ts",
|
||||
"src/channels/turn/kernel.ts",
|
||||
"src/channels/message/inbound-reply-dispatch.ts",
|
||||
"src/infra/outbound/deliver-runtime.ts",
|
||||
"src/infra/outbound/deliver.ts",
|
||||
"src/plugin-sdk/channel-message-runtime.ts",
|
||||
"src/plugin-sdk/channel-message.ts",
|
||||
"src/plugin-sdk/channel-test-helpers.ts",
|
||||
"src/plugin-sdk/inbound-reply-dispatch.ts",
|
||||
"src/plugin-sdk/outbound-runtime.ts",
|
||||
"src/plugin-sdk/test-helpers/outbound-delivery.ts",
|
||||
"src/plugin-sdk/testing.ts",
|
||||
],
|
||||
message: "use sendDurableMessageBatch or deliverInboundReplyWithMessageSendContext",
|
||||
},
|
||||
];
|
||||
|
||||
const selectedRuleIds = new Set(
|
||||
process.argv
|
||||
.slice(2)
|
||||
.filter((arg) => arg.startsWith("--rule="))
|
||||
.map((arg) => arg.slice("--rule=".length)),
|
||||
);
|
||||
|
||||
const selectedRules =
|
||||
selectedRuleIds.size === 0 ? rules : rules.filter((rule) => selectedRuleIds.has(rule.id));
|
||||
const unknownRuleIds = [...selectedRuleIds].filter((id) => !rules.some((rule) => rule.id === id));
|
||||
|
||||
if (unknownRuleIds.length > 0) {
|
||||
console.error(`Unknown deprecated API usage rule(s): ${unknownRuleIds.join(", ")}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const violations = selectedRules.flatMap((rule) =>
|
||||
collectRuleViolations(rule).map((violation) => `${rule.id}: ${violation}`),
|
||||
);
|
||||
|
||||
if (violations.length > 0) {
|
||||
console.error("Deprecated API usage guard failed:");
|
||||
for (const violation of violations) {
|
||||
console.error(`- ${violation}`);
|
||||
}
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log("deprecated API usage guard passed");
|
||||
132
scripts/check-deprecated-jsdoc.mjs
Normal file
132
scripts/check-deprecated-jsdoc.mjs
Normal file
@@ -0,0 +1,132 @@
|
||||
#!/usr/bin/env node
|
||||
// Checks deprecated JSDoc blocks for required migration details.
|
||||
import fs from "node:fs";
|
||||
import { createRequire } from "node:module";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const require = createRequire(import.meta.url);
|
||||
const ts = require("typescript");
|
||||
|
||||
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const SCAN_ROOTS = ["src", "extensions", "packages"];
|
||||
const SOURCE_FILE_RE = /\.(?:ts|tsx)$/;
|
||||
const SKIP_PATH_RE =
|
||||
/(?:^|\/)(?:node_modules|dist|build|protocol-generated)(?:\/|$)|(?:\.test|\.spec|\.e2e|\.generated)\.tsx?$/;
|
||||
const DEPRECATED_SURFACE_COMMENT_RE =
|
||||
/^(?:back-compat alias|backward-compatible alias(?:es)?|deprecated alias|legacy alias|legacy field|legacy:\s|kept for compatibility with existing imports|keep the legacy helper name exported)\b/i;
|
||||
|
||||
function walk(dir, files = []) {
|
||||
if (!fs.existsSync(dir)) {
|
||||
return files;
|
||||
}
|
||||
for (const entry of fs.readdirSync(dir, { withFileTypes: true })) {
|
||||
const filePath = path.join(dir, entry.name);
|
||||
const relativePath = path.relative(repoRoot, filePath).replaceAll(path.sep, "/");
|
||||
if (SKIP_PATH_RE.test(relativePath)) {
|
||||
continue;
|
||||
}
|
||||
if (entry.isDirectory()) {
|
||||
walk(filePath, files);
|
||||
} else if (SOURCE_FILE_RE.test(entry.name)) {
|
||||
files.push(filePath);
|
||||
}
|
||||
}
|
||||
return files;
|
||||
}
|
||||
|
||||
function leadingCommentText(sourceFile, node) {
|
||||
return (ts.getLeadingCommentRanges(sourceFile.text, node.pos) ?? [])
|
||||
.map((range) => sourceFile.text.slice(range.pos, range.end))
|
||||
.join("\n");
|
||||
}
|
||||
|
||||
function normalizeCommentText(comment) {
|
||||
return comment
|
||||
.replace(/\/\*\*?/g, "")
|
||||
.replace(/\*\//g, "")
|
||||
.split("\n")
|
||||
.map((line) => line.replace(/^\s*(?:\*|\/\/)\s?/, "").trim())
|
||||
.filter(Boolean)
|
||||
.join(" ");
|
||||
}
|
||||
|
||||
function lineOf(sourceFile, node) {
|
||||
return sourceFile.getLineAndCharacterOfPosition(node.getStart(sourceFile)).line + 1;
|
||||
}
|
||||
|
||||
function isExported(node) {
|
||||
return (
|
||||
node.modifiers?.some((modifier) => modifier.kind === ts.SyntaxKind.ExportKeyword) ||
|
||||
node.parent?.kind === ts.SyntaxKind.SourceFile
|
||||
);
|
||||
}
|
||||
|
||||
function symbolName(node) {
|
||||
const declaration = node.declarationList?.declarations?.[0] ?? node;
|
||||
return declaration.name?.getText?.() ?? "<anonymous>";
|
||||
}
|
||||
|
||||
function shouldInspectNode(node) {
|
||||
if (
|
||||
ts.isFunctionDeclaration(node) ||
|
||||
ts.isClassDeclaration(node) ||
|
||||
ts.isInterfaceDeclaration(node) ||
|
||||
ts.isTypeAliasDeclaration(node) ||
|
||||
ts.isEnumDeclaration(node) ||
|
||||
ts.isVariableStatement(node)
|
||||
) {
|
||||
return isExported(node);
|
||||
}
|
||||
return (
|
||||
ts.isPropertySignature(node) ||
|
||||
ts.isMethodSignature(node) ||
|
||||
ts.isPropertyDeclaration(node) ||
|
||||
ts.isEnumMember(node)
|
||||
);
|
||||
}
|
||||
|
||||
function collectViolations(filePath) {
|
||||
const sourceText = fs.readFileSync(filePath, "utf8");
|
||||
const sourceFile = ts.createSourceFile(filePath, sourceText, ts.ScriptTarget.Latest, true);
|
||||
const violations = [];
|
||||
|
||||
function visit(node) {
|
||||
if (shouldInspectNode(node)) {
|
||||
const comment = leadingCommentText(sourceFile, node);
|
||||
const normalizedComment = normalizeCommentText(comment);
|
||||
if (
|
||||
normalizedComment &&
|
||||
DEPRECATED_SURFACE_COMMENT_RE.test(normalizedComment) &&
|
||||
!/@deprecated\b/.test(comment)
|
||||
) {
|
||||
violations.push({
|
||||
line: lineOf(sourceFile, node),
|
||||
name: symbolName(node),
|
||||
filePath: path.relative(repoRoot, filePath).replaceAll(path.sep, "/"),
|
||||
});
|
||||
}
|
||||
}
|
||||
ts.forEachChild(node, visit);
|
||||
}
|
||||
|
||||
visit(sourceFile);
|
||||
return violations;
|
||||
}
|
||||
|
||||
const violations = SCAN_ROOTS.flatMap((root) =>
|
||||
walk(path.join(repoRoot, root)).flatMap(collectViolations),
|
||||
);
|
||||
|
||||
if (violations.length > 0) {
|
||||
console.error("Deprecated JSDoc guard failed:");
|
||||
for (const violation of violations) {
|
||||
console.error(`- ${violation.filePath}:${violation.line} ${violation.name}`);
|
||||
}
|
||||
console.error(
|
||||
"Add an @deprecated JSDoc tag or reword the comment if the symbol is not deprecated.",
|
||||
);
|
||||
process.exitCode = 1;
|
||||
} else {
|
||||
console.log("deprecated JSDoc guard passed");
|
||||
}
|
||||
135
scripts/check-docker-e2e-boundaries.mjs
Normal file
135
scripts/check-docker-e2e-boundaries.mjs
Normal file
@@ -0,0 +1,135 @@
|
||||
#!/usr/bin/env node
|
||||
// Cheap guard for Docker E2E test boundaries.
|
||||
// Docker E2E must test packaged npm tarballs and package-installed images, not
|
||||
// the source checkout copied or mounted as the app under test.
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { laneResources, laneWeight } from "./lib/docker-e2e-plan.mjs";
|
||||
import { allReleasePathLanes, mainLanes, tailLanes } from "./lib/docker-e2e-scenarios.mjs";
|
||||
|
||||
const ROOT_DIR = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const errors = [];
|
||||
const packageJson = JSON.parse(readText("package.json"));
|
||||
const packageScripts = new Set(Object.keys(packageJson.scripts ?? {}));
|
||||
// These lanes prove package-installed surfaces against live auth, so they
|
||||
// intentionally need both live credentials and a package-backed image.
|
||||
const livePackageBackedLanes = new Set([
|
||||
"install-e2e-anthropic",
|
||||
"install-e2e-openai",
|
||||
"live-codex-npm-plugin",
|
||||
"live-mcp-code-mode-gateway",
|
||||
"live-plugin-tool",
|
||||
"npm-telegram-live",
|
||||
"openai-chat-tools",
|
||||
"openwebui",
|
||||
]);
|
||||
// These lanes intentionally build a focused source-checkout image instead of
|
||||
// consuming the shared package E2E images.
|
||||
const sourceCheckoutImageLanes = new Set(["plugin-binding-command-escape"]);
|
||||
|
||||
function readText(relativePath) {
|
||||
return fs.readFileSync(path.join(ROOT_DIR, relativePath), "utf8");
|
||||
}
|
||||
|
||||
function walk(dir, out = []) {
|
||||
for (const entry of fs.readdirSync(path.join(ROOT_DIR, dir), { withFileTypes: true })) {
|
||||
const relativePath = path.join(dir, entry.name);
|
||||
if (entry.isDirectory()) {
|
||||
walk(relativePath, out);
|
||||
} else {
|
||||
out.push(relativePath);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
for (const relativePath of walk("scripts/e2e")) {
|
||||
if (!/\.(?:sh|ts|mjs|js)$/u.test(relativePath)) {
|
||||
continue;
|
||||
}
|
||||
const text = readText(relativePath);
|
||||
if (/from\s+["']\.\.\/\.\.\/src\//u.test(text) || /import\(["']\.\.\/\.\.\/src\//u.test(text)) {
|
||||
errors.push(`${relativePath}: Docker E2E harness must import built dist, not ../../src`);
|
||||
}
|
||||
if (/-v\s+["']?\$ROOT_DIR:\/app(?::|["'\s]|$)/u.test(text)) {
|
||||
errors.push(`${relativePath}: do not mount the repo root as /app in Docker E2E`);
|
||||
}
|
||||
}
|
||||
|
||||
const dockerfile = readText("scripts/e2e/Dockerfile");
|
||||
if (/^\s*(?:COPY|ADD)\s+\.\s+\/app(?:\s|$)/imu.test(dockerfile)) {
|
||||
errors.push("scripts/e2e/Dockerfile: do not copy the source checkout into /app");
|
||||
}
|
||||
|
||||
function validateUniqueLanes(label, lanes) {
|
||||
const seen = new Set();
|
||||
for (const lane of lanes) {
|
||||
if (seen.has(lane.name)) {
|
||||
errors.push(`${label}: duplicate Docker E2E lane '${lane.name}'`);
|
||||
}
|
||||
seen.add(lane.name);
|
||||
}
|
||||
}
|
||||
|
||||
function validateLane(label, lane) {
|
||||
const resources = laneResources(lane);
|
||||
const sourceCheckoutImageLane = sourceCheckoutImageLanes.has(lane.name);
|
||||
if (!lane.name || typeof lane.name !== "string") {
|
||||
errors.push(`${label}: Docker E2E lane is missing a string name`);
|
||||
}
|
||||
if (!lane.command || typeof lane.command !== "string") {
|
||||
errors.push(`${label}: Docker E2E lane '${lane.name}' is missing a string command`);
|
||||
return;
|
||||
}
|
||||
if (lane.e2eImageKind && lane.e2eImageKind !== "bare" && lane.e2eImageKind !== "functional") {
|
||||
errors.push(
|
||||
`${label}: Docker E2E lane '${lane.name}' has invalid image kind '${lane.e2eImageKind}'`,
|
||||
);
|
||||
}
|
||||
if (lane.live && lane.e2eImageKind && !livePackageBackedLanes.has(lane.name)) {
|
||||
errors.push(`${label}: live Docker E2E lane '${lane.name}' must not require a package image`);
|
||||
}
|
||||
if (!lane.live && !lane.e2eImageKind && !sourceCheckoutImageLane) {
|
||||
errors.push(`${label}: package Docker E2E lane '${lane.name}' must declare an e2e image kind`);
|
||||
}
|
||||
if (sourceCheckoutImageLane && !/\bOPENCLAW_SKIP_DOCKER_BUILD=0\b/u.test(lane.command)) {
|
||||
errors.push(
|
||||
`${label}: source-checkout Docker E2E lane '${lane.name}' must force a local image build`,
|
||||
);
|
||||
}
|
||||
if (laneWeight(lane) < 1) {
|
||||
errors.push(`${label}: Docker E2E lane '${lane.name}' must have positive weight`);
|
||||
}
|
||||
if (!resources.includes("docker")) {
|
||||
errors.push(`${label}: Docker E2E lane '${lane.name}' must include the docker resource`);
|
||||
}
|
||||
|
||||
for (const match of lane.command.matchAll(/\bpnpm\s+([^\s]+)/gu)) {
|
||||
const script = match[1];
|
||||
if (!packageScripts.has(script)) {
|
||||
errors.push(
|
||||
`${label}: Docker E2E lane '${lane.name}' references missing package script '${script}'`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const releasePathLanes = allReleasePathLanes({ includeOpenWebUI: true });
|
||||
for (const [label, lanes] of [
|
||||
["release-path", releasePathLanes],
|
||||
["main", mainLanes],
|
||||
["tail", tailLanes],
|
||||
]) {
|
||||
validateUniqueLanes(label, lanes);
|
||||
for (const lane of lanes) {
|
||||
validateLane(label, lane);
|
||||
}
|
||||
}
|
||||
|
||||
if (errors.length > 0) {
|
||||
console.error(errors.join("\n"));
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log("Docker E2E package boundary/catalog guard passed.");
|
||||
248
scripts/check-docs-i18n-glossary.mjs
Normal file
248
scripts/check-docs-i18n-glossary.mjs
Normal file
@@ -0,0 +1,248 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Validates docs i18n glossary terms against configured usage rules.
|
||||
import { execFileSync } from "node:child_process";
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
|
||||
const ROOT = process.cwd();
|
||||
const GLOSSARY_PATH = path.join(ROOT, "docs", ".i18n", "glossary.zh-CN.json");
|
||||
const DOC_FILE_RE = /^docs\/(?!zh-CN\/).+\.(md|mdx)$/i;
|
||||
const LIST_ITEM_LINK_RE = /^\s*(?:[-*]|\d+\.)\s+\[([^\]]+)\]\((\/[^)]+)\)/;
|
||||
const MAX_TITLE_WORDS = 8;
|
||||
const MAX_LABEL_WORDS = 6;
|
||||
const MAX_TERM_LENGTH = 80;
|
||||
|
||||
/**
|
||||
* @typedef {{
|
||||
* file: string;
|
||||
* line: number;
|
||||
* kind: "title" | "link label";
|
||||
* term: string;
|
||||
* }} TermMatch
|
||||
*/
|
||||
|
||||
function readRefOptionValue(argv, index, optionName) {
|
||||
const value = argv[index + 1];
|
||||
if (value === undefined || value === "" || value.startsWith("-")) {
|
||||
throw new Error(`${optionName} requires a value`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
export function parseArgs(argv) {
|
||||
/** @type {{ base: string; head: string }} */
|
||||
const args = { base: "", head: "" };
|
||||
for (let i = 0; i < argv.length; i += 1) {
|
||||
if (argv[i] === "--base") {
|
||||
args.base = readRefOptionValue(argv, i, "--base");
|
||||
i += 1;
|
||||
continue;
|
||||
}
|
||||
if (argv[i] === "--head") {
|
||||
args.head = readRefOptionValue(argv, i, "--head");
|
||||
i += 1;
|
||||
}
|
||||
}
|
||||
return args;
|
||||
}
|
||||
|
||||
function runGit(args) {
|
||||
return execFileSync("git", args, {
|
||||
cwd: ROOT,
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
encoding: "utf8",
|
||||
}).trim();
|
||||
}
|
||||
|
||||
function resolveBase(explicitBase) {
|
||||
if (explicitBase) {
|
||||
return explicitBase;
|
||||
}
|
||||
|
||||
const envBase = process.env.DOCS_I18N_GLOSSARY_BASE?.trim();
|
||||
if (envBase) {
|
||||
return envBase;
|
||||
}
|
||||
|
||||
for (const candidate of ["origin/main", "fork/main", "main"]) {
|
||||
try {
|
||||
return runGit(["merge-base", candidate, "HEAD"]);
|
||||
} catch {
|
||||
// Try the next candidate.
|
||||
}
|
||||
}
|
||||
|
||||
return "";
|
||||
}
|
||||
|
||||
function listChangedDocs(base, head) {
|
||||
const args = ["diff", "--name-only", "--diff-filter=ACMR", base];
|
||||
if (head) {
|
||||
args.push(head);
|
||||
}
|
||||
args.push("--", "docs");
|
||||
|
||||
return runGit(args)
|
||||
.split("\n")
|
||||
.map((line) => line.trim())
|
||||
.filter((line) => DOC_FILE_RE.test(line));
|
||||
}
|
||||
|
||||
function loadGlossarySources() {
|
||||
const data = fs.readFileSync(GLOSSARY_PATH, "utf8");
|
||||
const entries = JSON.parse(data);
|
||||
return new Set(entries.map((entry) => String(entry.source || "").trim()).filter(Boolean));
|
||||
}
|
||||
|
||||
function containsLatin(text) {
|
||||
return /[A-Za-z]/.test(text);
|
||||
}
|
||||
|
||||
function wordCount(text) {
|
||||
return text.trim().split(/\s+/).filter(Boolean).length;
|
||||
}
|
||||
|
||||
function unquoteScalar(raw) {
|
||||
const value = raw.trim();
|
||||
if (
|
||||
(value.startsWith('"') && value.endsWith('"')) ||
|
||||
(value.startsWith("'") && value.endsWith("'"))
|
||||
) {
|
||||
return value.slice(1, -1).trim();
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function isGlossaryCandidate(term, maxWords) {
|
||||
if (!term) {
|
||||
return false;
|
||||
}
|
||||
if (!containsLatin(term)) {
|
||||
return false;
|
||||
}
|
||||
if (term.includes("`")) {
|
||||
return false;
|
||||
}
|
||||
if (term.length > MAX_TERM_LENGTH) {
|
||||
return false;
|
||||
}
|
||||
return wordCount(term) <= maxWords;
|
||||
}
|
||||
|
||||
function readGitFile(base, relPath) {
|
||||
try {
|
||||
return runGit(["show", `${base}:${relPath}`]);
|
||||
} catch {
|
||||
return "";
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {string} file
|
||||
* @param {string} text
|
||||
* @returns {Map<string, TermMatch>}
|
||||
*/
|
||||
function extractTerms(file, text) {
|
||||
/** @type {Map<string, TermMatch>} */
|
||||
const terms = new Map();
|
||||
const lines = text.split("\n");
|
||||
|
||||
if (lines[0]?.trim() === "---") {
|
||||
for (let index = 1; index < lines.length; index += 1) {
|
||||
const line = lines[index];
|
||||
if (line.trim() === "---") {
|
||||
break;
|
||||
}
|
||||
|
||||
const match = line.match(/^title:\s*(.+)\s*$/);
|
||||
if (!match) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const title = unquoteScalar(match[1]);
|
||||
if (isGlossaryCandidate(title, MAX_TITLE_WORDS)) {
|
||||
terms.set(title, { file, line: index + 1, kind: "title", term: title });
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
for (let index = 0; index < lines.length; index += 1) {
|
||||
const match = lines[index].match(LIST_ITEM_LINK_RE);
|
||||
if (!match) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const label = match[1].trim();
|
||||
if (!isGlossaryCandidate(label, MAX_LABEL_WORDS)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!terms.has(label)) {
|
||||
terms.set(label, { file, line: index + 1, kind: "link label", term: label });
|
||||
}
|
||||
}
|
||||
|
||||
return terms;
|
||||
}
|
||||
|
||||
function main() {
|
||||
const args = parseArgs(process.argv.slice(2));
|
||||
const base = resolveBase(args.base);
|
||||
|
||||
if (!base) {
|
||||
console.warn(
|
||||
"docs:check-i18n-glossary: no merge base found; skipping glossary coverage check.",
|
||||
);
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
const changedDocs = listChangedDocs(base, args.head);
|
||||
if (changedDocs.length === 0) {
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
const glossary = loadGlossarySources();
|
||||
/** @type {TermMatch[]} */
|
||||
const missing = [];
|
||||
|
||||
for (const relPath of changedDocs) {
|
||||
const absPath = path.join(ROOT, relPath);
|
||||
if (!fs.existsSync(absPath)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const currentTerms = extractTerms(relPath, fs.readFileSync(absPath, "utf8"));
|
||||
const baseTerms = extractTerms(relPath, readGitFile(base, relPath));
|
||||
|
||||
for (const [term, match] of currentTerms) {
|
||||
if (baseTerms.has(term)) {
|
||||
continue;
|
||||
}
|
||||
if (glossary.has(term)) {
|
||||
continue;
|
||||
}
|
||||
missing.push(match);
|
||||
}
|
||||
}
|
||||
|
||||
if (missing.length === 0) {
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
console.error("docs:check-i18n-glossary: missing zh-CN glossary entries for changed doc labels:");
|
||||
for (const match of missing) {
|
||||
console.error(`- ${match.file}:${match.line} ${match.kind} "${match.term}"`);
|
||||
}
|
||||
console.error("");
|
||||
console.error(
|
||||
"Add exact source terms to docs/.i18n/glossary.zh-CN.json before rerunning docs-i18n.",
|
||||
);
|
||||
console.error(`Checked changed English docs relative to ${base}.`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (process.argv[1] && path.resolve(process.argv[1]) === path.resolve(import.meta.filename)) {
|
||||
main();
|
||||
}
|
||||
386
scripts/check-docs-mdx.mjs
Normal file
386
scripts/check-docs-mdx.mjs
Normal file
@@ -0,0 +1,386 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Validates docs MDX files for syntax and repository-specific conventions.
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { compile } from "@mdx-js/mdx";
|
||||
import {
|
||||
checkMintlifyAccordionIndentation,
|
||||
MINTLIFY_ACCORDION_INDENT_MESSAGE,
|
||||
} from "./lib/mintlify-accordion.mjs";
|
||||
|
||||
const MINTLIFY_LANGUAGE_CODES = new Set([
|
||||
"en",
|
||||
"cn",
|
||||
"zh",
|
||||
"zh-Hans",
|
||||
"zh-Hant",
|
||||
"es",
|
||||
"fr",
|
||||
"fr-CA",
|
||||
"fr-ca",
|
||||
"ja",
|
||||
"jp",
|
||||
"ja-jp",
|
||||
"pt",
|
||||
"pt-BR",
|
||||
"de",
|
||||
"ko",
|
||||
"it",
|
||||
"ru",
|
||||
"ro",
|
||||
"cs",
|
||||
"id",
|
||||
"ar",
|
||||
"tr",
|
||||
"hi",
|
||||
"sv",
|
||||
"no",
|
||||
"lv",
|
||||
"nl",
|
||||
"uk",
|
||||
"vi",
|
||||
"pl",
|
||||
"uz",
|
||||
"he",
|
||||
"ca",
|
||||
"fi",
|
||||
"hu",
|
||||
]);
|
||||
|
||||
const POISON_TEXT_PATTERNS = [
|
||||
{
|
||||
pattern: /\banalysis\s+to=functions\./iu,
|
||||
message: "Leaked tool-call channel marker.",
|
||||
},
|
||||
{
|
||||
pattern: /\b(?:commentary|final)\s+to=functions\./iu,
|
||||
message: "Leaked tool-call channel marker.",
|
||||
},
|
||||
{
|
||||
pattern: /\bfunctions\.(?:read|write|exec|search|run)\b/iu,
|
||||
message: "Leaked internal tool name.",
|
||||
},
|
||||
{
|
||||
pattern: /\b[A-Za-z_\u3400-\u9fff][\w\u3400-\u9fff-]*_input=\{/u,
|
||||
message: "Leaked tool-call input payload.",
|
||||
},
|
||||
{
|
||||
pattern: /<\/?openclaw_docs_i18n_input>/iu,
|
||||
message: "Leaked docs i18n prompt wrapper.",
|
||||
},
|
||||
{
|
||||
pattern: /\/home\/runner\/work\//u,
|
||||
message: "Leaked GitHub Actions workspace path.",
|
||||
},
|
||||
{
|
||||
pattern: /彩神马争霸/u,
|
||||
message: "Known spam/gambling text from a poisoned translation.",
|
||||
},
|
||||
];
|
||||
|
||||
function parsePositiveIntegerArg(raw, label) {
|
||||
const text = String(raw ?? "").trim();
|
||||
if (!/^\d+$/u.test(text)) {
|
||||
throw new Error(`${label} must be a positive integer`);
|
||||
}
|
||||
const value = Number(text);
|
||||
if (!Number.isSafeInteger(value) || value < 1) {
|
||||
throw new Error(`${label} must be a positive integer`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function readRequiredValue(argv, index, label) {
|
||||
const value = argv[index + 1];
|
||||
if (!value || value.startsWith("-")) {
|
||||
throw new Error(`${label} requires a value`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parses docs MDX check arguments.
|
||||
*/
|
||||
export function parseArgs(argv) {
|
||||
const roots = [];
|
||||
let jsonOut = "";
|
||||
let maxErrors = 50;
|
||||
|
||||
for (let index = 0; index < argv.length; index += 1) {
|
||||
const part = argv[index];
|
||||
if (part === "--json-out") {
|
||||
jsonOut = readRequiredValue(argv, index, "--json-out");
|
||||
index += 1;
|
||||
continue;
|
||||
}
|
||||
if (part === "--max-errors") {
|
||||
maxErrors = parsePositiveIntegerArg(
|
||||
readRequiredValue(argv, index, "--max-errors"),
|
||||
"--max-errors",
|
||||
);
|
||||
index += 1;
|
||||
continue;
|
||||
}
|
||||
if (part.startsWith("--")) {
|
||||
throw new Error(`unknown arg: ${part}`);
|
||||
}
|
||||
roots.push(part);
|
||||
}
|
||||
|
||||
return {
|
||||
roots: roots.length ? roots : ["docs"],
|
||||
jsonOut,
|
||||
maxErrors,
|
||||
};
|
||||
}
|
||||
|
||||
function walkMarkdownFiles(entryPath, out = []) {
|
||||
const stat = fs.statSync(entryPath);
|
||||
if (stat.isFile()) {
|
||||
if (/\.mdx?$/i.test(entryPath)) {
|
||||
out.push(path.resolve(entryPath));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
for (const entry of fs.readdirSync(entryPath, { withFileTypes: true })) {
|
||||
if (entry.name === "node_modules" || entry.name === ".git") {
|
||||
continue;
|
||||
}
|
||||
walkMarkdownFiles(path.join(entryPath, entry.name), out);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function stripFrontmatter(raw) {
|
||||
if (!raw.startsWith("---\n") && !raw.startsWith("---\r\n")) {
|
||||
return raw;
|
||||
}
|
||||
|
||||
const lines = raw.split(/\r?\n/u);
|
||||
for (let index = 1; index < lines.length; index += 1) {
|
||||
if (lines[index] === "---" || lines[index] === "...") {
|
||||
return lines.slice(index + 1).join("\n");
|
||||
}
|
||||
}
|
||||
return raw;
|
||||
}
|
||||
|
||||
function formatMdxError(filePath, error) {
|
||||
const place = error?.place ?? error?.position;
|
||||
const start = place?.start ?? place;
|
||||
const line = typeof start?.line === "number" ? start.line : undefined;
|
||||
const column = typeof start?.column === "number" ? start.column : undefined;
|
||||
return {
|
||||
type: "mdx",
|
||||
file: filePath,
|
||||
line,
|
||||
column,
|
||||
message: String(error?.reason ?? error?.message ?? error).split("\n")[0],
|
||||
};
|
||||
}
|
||||
|
||||
function checkMintlifyMdxStructure(filePath, raw) {
|
||||
return checkMintlifyAccordionIndentation(stripFrontmatter(raw)).map((error) => ({
|
||||
type: "mintlify-mdx",
|
||||
file: filePath,
|
||||
line: error.line,
|
||||
column: error.column,
|
||||
message: MINTLIFY_ACCORDION_INDENT_MESSAGE,
|
||||
}));
|
||||
}
|
||||
|
||||
function lineColumnForIndex(raw, offset) {
|
||||
const prefix = raw.slice(0, offset);
|
||||
const lines = prefix.split(/\r?\n/u);
|
||||
return {
|
||||
line: lines.length,
|
||||
column: lines.at(-1).length + 1,
|
||||
};
|
||||
}
|
||||
|
||||
function checkPoisonText(filePath, raw) {
|
||||
const errors = [];
|
||||
for (const { pattern, message } of POISON_TEXT_PATTERNS) {
|
||||
const match = pattern.exec(raw);
|
||||
if (!match) {
|
||||
continue;
|
||||
}
|
||||
const location = lineColumnForIndex(raw, match.index);
|
||||
errors.push({
|
||||
type: "poison-text",
|
||||
file: filePath,
|
||||
line: location.line,
|
||||
column: location.column,
|
||||
message,
|
||||
});
|
||||
}
|
||||
return errors;
|
||||
}
|
||||
|
||||
async function checkMdxFile(filePath) {
|
||||
const raw = fs.readFileSync(filePath, "utf8");
|
||||
const poisonErrors = checkPoisonText(filePath, raw);
|
||||
if (poisonErrors.length > 0) {
|
||||
return poisonErrors;
|
||||
}
|
||||
const structureErrors = checkMintlifyMdxStructure(filePath, raw);
|
||||
if (structureErrors.length > 0) {
|
||||
return structureErrors;
|
||||
}
|
||||
const value = stripFrontmatter(raw);
|
||||
await compile(
|
||||
{ path: filePath, value },
|
||||
{
|
||||
development: false,
|
||||
jsx: false,
|
||||
},
|
||||
);
|
||||
return [];
|
||||
}
|
||||
|
||||
function findDocsJsonPaths(roots) {
|
||||
const paths = new Set();
|
||||
for (const root of roots) {
|
||||
const absolute = path.resolve(root);
|
||||
if (!fs.existsSync(absolute)) {
|
||||
continue;
|
||||
}
|
||||
const stat = fs.statSync(absolute);
|
||||
if (stat.isFile() && path.basename(absolute) === "docs.json") {
|
||||
paths.add(absolute);
|
||||
continue;
|
||||
}
|
||||
if (stat.isDirectory()) {
|
||||
const docsJsonPath = path.join(absolute, "docs.json");
|
||||
if (fs.existsSync(docsJsonPath)) {
|
||||
paths.add(docsJsonPath);
|
||||
}
|
||||
}
|
||||
}
|
||||
return [...paths];
|
||||
}
|
||||
|
||||
function collectNavigationLanguages(value, out = []) {
|
||||
if (Array.isArray(value)) {
|
||||
for (const item of value) {
|
||||
collectNavigationLanguages(item, out);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
if (!value || typeof value !== "object") {
|
||||
return out;
|
||||
}
|
||||
if (typeof value.language === "string") {
|
||||
out.push(value.language);
|
||||
}
|
||||
for (const child of Object.values(value)) {
|
||||
if (child && typeof child === "object") {
|
||||
collectNavigationLanguages(child, out);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function checkDocsJson(filePath) {
|
||||
const errors = [];
|
||||
let data;
|
||||
try {
|
||||
data = JSON.parse(fs.readFileSync(filePath, "utf8"));
|
||||
} catch (error) {
|
||||
return [
|
||||
{
|
||||
type: "docs-json",
|
||||
file: filePath,
|
||||
message: `Invalid JSON: ${String(error?.message ?? error)}`,
|
||||
},
|
||||
];
|
||||
}
|
||||
|
||||
const languages = collectNavigationLanguages(data?.navigation);
|
||||
for (const language of languages) {
|
||||
if (!MINTLIFY_LANGUAGE_CODES.has(language)) {
|
||||
errors.push({
|
||||
type: "docs-json",
|
||||
file: filePath,
|
||||
message: `Unsupported Mintlify navigation language: ${language}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
return errors;
|
||||
}
|
||||
|
||||
function relativize(root, filePath) {
|
||||
const relative = path.relative(root, filePath);
|
||||
return relative && !relative.startsWith("..") ? relative : filePath;
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const startedAt = Date.now();
|
||||
const args = parseArgs(process.argv.slice(2));
|
||||
const cwd = process.cwd();
|
||||
const roots = args.roots.map((root) => path.resolve(root));
|
||||
const files = [
|
||||
...new Set(
|
||||
roots.flatMap((root) => {
|
||||
if (!fs.existsSync(root)) {
|
||||
throw new Error(`path does not exist: ${root}`);
|
||||
}
|
||||
return walkMarkdownFiles(root);
|
||||
}),
|
||||
),
|
||||
].toSorted((left, right) => left.localeCompare(right));
|
||||
|
||||
const errors = [];
|
||||
for (const docsJsonPath of findDocsJsonPaths(args.roots)) {
|
||||
errors.push(...checkDocsJson(docsJsonPath));
|
||||
}
|
||||
|
||||
for (const file of files) {
|
||||
try {
|
||||
errors.push(...(await checkMdxFile(file)));
|
||||
} catch (error) {
|
||||
errors.push(formatMdxError(file, error));
|
||||
if (errors.length >= args.maxErrors) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const report = {
|
||||
files: files.length,
|
||||
errors: errors.map((error) => Object.assign({}, error, { file: relativize(cwd, error.file) })),
|
||||
ms: Date.now() - startedAt,
|
||||
};
|
||||
|
||||
if (args.jsonOut) {
|
||||
fs.mkdirSync(path.dirname(path.resolve(args.jsonOut)), { recursive: true });
|
||||
fs.writeFileSync(args.jsonOut, `${JSON.stringify(report, null, 2)}\n`);
|
||||
}
|
||||
|
||||
if (report.errors.length === 0) {
|
||||
console.log(`Docs MDX check passed (${report.files} files, ${report.ms}ms).`);
|
||||
return;
|
||||
}
|
||||
|
||||
console.error(`Docs MDX check failed (${report.errors.length} error(s), ${report.files} files).`);
|
||||
for (const error of report.errors) {
|
||||
const location =
|
||||
error.line && error.column ? `${error.file}:${error.line}:${error.column}` : error.file;
|
||||
console.error(`- ${location}: ${error.message}`);
|
||||
}
|
||||
process.exitCode = 1;
|
||||
}
|
||||
|
||||
const isMain = process.argv[1] ? fileURLToPath(import.meta.url) === process.argv[1] : false;
|
||||
|
||||
if (isMain) {
|
||||
main().catch(
|
||||
/** @param {unknown} error */ (error) => {
|
||||
console.error(error?.stack ?? error);
|
||||
process.exit(1);
|
||||
},
|
||||
);
|
||||
}
|
||||
208
scripts/check-duplicates.mjs
Normal file
208
scripts/check-duplicates.mjs
Normal file
@@ -0,0 +1,208 @@
|
||||
#!/usr/bin/env node
|
||||
// Runs duplicate-code detection with repo-specific excludes.
|
||||
import { spawnSync } from "node:child_process";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const jscpdBin = path.join(repoRoot, "node_modules", "jscpd", "bin", "jscpd");
|
||||
|
||||
const targets = [
|
||||
"src",
|
||||
"extensions",
|
||||
"examples",
|
||||
"scripts",
|
||||
"packages",
|
||||
"ui",
|
||||
"apps",
|
||||
"docs",
|
||||
"qa",
|
||||
"security",
|
||||
"test",
|
||||
"skills",
|
||||
"openclaw.mjs",
|
||||
"config/knip.config.ts",
|
||||
"tsdown.ai.config.ts",
|
||||
"tsdown.config.ts",
|
||||
"vitest.config.ts",
|
||||
];
|
||||
|
||||
const sourceExtensions = new Set([".ts", ".tsx", ".js", ".mjs", ".cjs"]);
|
||||
const sourcePattern = "**/*.{ts,tsx,js,mjs,cjs}";
|
||||
const testPattern = "**/*.{test,e2e.test,live.test}.{ts,tsx,js,mjs,cjs}";
|
||||
// Keep local agent support trees and vendored snapshots classified but outside jscpd.
|
||||
const intentionallyUnscannedPrefixes = [".agents/", "vendor/"];
|
||||
|
||||
const generatedIgnores = [
|
||||
"extensions/qa-matrix/src/shared/**",
|
||||
"extensions/qa-matrix/src/cli-paths.ts",
|
||||
"**/node_modules/**",
|
||||
"**/dist/**",
|
||||
"**/.git/**",
|
||||
"**/coverage/**",
|
||||
"**/build/**",
|
||||
"**/.build/**",
|
||||
"**/.artifacts/**",
|
||||
"vendor/**",
|
||||
];
|
||||
|
||||
const testIgnores = [
|
||||
"**/*.test.ts",
|
||||
"**/*.test.tsx",
|
||||
"**/*.test.js",
|
||||
"**/*.test.mjs",
|
||||
"**/*.test.cjs",
|
||||
"**/*.e2e.test.ts",
|
||||
"**/*.e2e.test.tsx",
|
||||
"**/*.e2e.test.js",
|
||||
"**/*.e2e.test.mjs",
|
||||
"**/*.e2e.test.cjs",
|
||||
"**/*.live.test.ts",
|
||||
"**/*.live.test.tsx",
|
||||
"**/*.live.test.js",
|
||||
"**/*.live.test.mjs",
|
||||
"**/*.live.test.cjs",
|
||||
];
|
||||
|
||||
const commonArgs = [
|
||||
"--format",
|
||||
"typescript,javascript",
|
||||
"--gitignore",
|
||||
"--noSymlinks",
|
||||
"--min-lines",
|
||||
"50",
|
||||
"--min-tokens",
|
||||
"300",
|
||||
];
|
||||
|
||||
const json = process.argv.includes("--json");
|
||||
const coverageOnly = process.argv.includes("--coverage");
|
||||
|
||||
function normalizeRepoPath(value) {
|
||||
return value.split(path.sep).join("/");
|
||||
}
|
||||
|
||||
function isUnderPrefix(value, prefix) {
|
||||
return value === prefix.slice(0, -1) || value.startsWith(prefix);
|
||||
}
|
||||
|
||||
function isCoveredByTargets(file) {
|
||||
return targets.some((target) => {
|
||||
const normalizedTarget = normalizeRepoPath(target);
|
||||
if (file === normalizedTarget) {
|
||||
return true;
|
||||
}
|
||||
return file.startsWith(`${normalizedTarget}/`);
|
||||
});
|
||||
}
|
||||
|
||||
function listTrackedSourceFiles() {
|
||||
const result = spawnSync("git", ["ls-files", "-z"], {
|
||||
cwd: repoRoot,
|
||||
encoding: "utf8",
|
||||
maxBuffer: 64 * 1024 * 1024,
|
||||
});
|
||||
if (result.status !== 0) {
|
||||
throw new Error(result.stderr || "git ls-files failed");
|
||||
}
|
||||
return result.stdout
|
||||
.split("\0")
|
||||
.filter(Boolean)
|
||||
.map(normalizeRepoPath)
|
||||
.filter((file) => sourceExtensions.has(path.extname(file)))
|
||||
.filter((file) => !intentionallyUnscannedPrefixes.some((prefix) => isUnderPrefix(file, prefix)))
|
||||
.toSorted((left, right) => left.localeCompare(right));
|
||||
}
|
||||
|
||||
function assertTargetCoverage() {
|
||||
const uncovered = listTrackedSourceFiles().filter((file) => !isCoveredByTargets(file));
|
||||
if (uncovered.length === 0) {
|
||||
console.log(`[dup:check] target coverage ok`);
|
||||
return true;
|
||||
}
|
||||
console.error(
|
||||
"[dup:check] tracked duplicate-scan source files are outside scan targets or intentional excludes:",
|
||||
);
|
||||
for (const file of uncovered) {
|
||||
console.error(` - ${file}`);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function reportArgs(name) {
|
||||
if (!json) {
|
||||
return ["--reporters", "console"];
|
||||
}
|
||||
return ["--reporters", "json", "--output", path.join(".artifacts", "jscpd", name)];
|
||||
}
|
||||
|
||||
const scans = [
|
||||
{
|
||||
name: "production",
|
||||
targets,
|
||||
pattern: sourcePattern,
|
||||
ignore: [...testIgnores, ...generatedIgnores],
|
||||
},
|
||||
{
|
||||
name: "tests",
|
||||
targets,
|
||||
pattern: testPattern,
|
||||
ignore: generatedIgnores,
|
||||
},
|
||||
{
|
||||
name: "src-mixed",
|
||||
targets: ["src"],
|
||||
pattern: sourcePattern,
|
||||
ignore: generatedIgnores,
|
||||
},
|
||||
{
|
||||
name: "extensions-mixed",
|
||||
targets: ["extensions"],
|
||||
pattern: sourcePattern,
|
||||
ignore: generatedIgnores,
|
||||
},
|
||||
{
|
||||
name: "test-mixed",
|
||||
targets: ["test"],
|
||||
pattern: sourcePattern,
|
||||
ignore: generatedIgnores,
|
||||
},
|
||||
];
|
||||
|
||||
let failed = !assertTargetCoverage();
|
||||
if (coverageOnly) {
|
||||
process.exit(failed ? 1 : 0);
|
||||
}
|
||||
for (const scan of scans) {
|
||||
console.log(`\n[dup:check] ${scan.name}`);
|
||||
const result = spawnSync(
|
||||
process.execPath,
|
||||
[
|
||||
"--max-old-space-size=8192",
|
||||
jscpdBin,
|
||||
...scan.targets,
|
||||
...commonArgs,
|
||||
"--pattern",
|
||||
scan.pattern,
|
||||
"--ignore",
|
||||
scan.ignore.join(","),
|
||||
...reportArgs(scan.name),
|
||||
],
|
||||
{
|
||||
cwd: repoRoot,
|
||||
env: process.env,
|
||||
stdio: "inherit",
|
||||
},
|
||||
);
|
||||
if (result.status !== 0) {
|
||||
failed = true;
|
||||
}
|
||||
if (result.error) {
|
||||
console.error(result.error.message);
|
||||
failed = true;
|
||||
}
|
||||
}
|
||||
|
||||
if (failed) {
|
||||
process.exit(1);
|
||||
}
|
||||
230
scripts/check-dynamic-import-warts.mjs
Normal file
230
scripts/check-dynamic-import-warts.mjs
Normal file
@@ -0,0 +1,230 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Advises on ineffective or suspicious dynamic import patterns.
|
||||
import { promises as fs } from "node:fs";
|
||||
import path from "node:path";
|
||||
import ts from "typescript";
|
||||
import {
|
||||
collectTypeScriptFilesFromRoots,
|
||||
resolveRepoRoot,
|
||||
runAsScript,
|
||||
toLine,
|
||||
} from "./lib/ts-guard-utils.mjs";
|
||||
|
||||
const repoRoot = resolveRepoRoot(import.meta.url);
|
||||
const defaultRoots = [path.join(repoRoot, "src"), path.join(repoRoot, "extensions")];
|
||||
|
||||
function readStringLiteral(node) {
|
||||
if (ts.isStringLiteral(node) || ts.isNoSubstitutionTemplateLiteral(node)) {
|
||||
return node.text;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function isTypeOnlyImportDeclaration(node) {
|
||||
const clause = node.importClause;
|
||||
if (!clause) {
|
||||
return false;
|
||||
}
|
||||
if (clause.isTypeOnly) {
|
||||
return true;
|
||||
}
|
||||
if (clause.name) {
|
||||
return false;
|
||||
}
|
||||
const bindings = clause.namedBindings;
|
||||
return (
|
||||
Boolean(bindings) &&
|
||||
ts.isNamedImports(bindings) &&
|
||||
bindings.elements.length > 0 &&
|
||||
bindings.elements.every((element) => element.isTypeOnly)
|
||||
);
|
||||
}
|
||||
|
||||
function isTypeOnlyExportDeclaration(node) {
|
||||
if (node.isTypeOnly === true) {
|
||||
return true;
|
||||
}
|
||||
const clause = node.exportClause;
|
||||
return (
|
||||
Boolean(clause) &&
|
||||
ts.isNamedExports(clause) &&
|
||||
clause.elements.length > 0 &&
|
||||
clause.elements.every((element) => element.isTypeOnly)
|
||||
);
|
||||
}
|
||||
|
||||
function readDeclarationName(node) {
|
||||
if (
|
||||
(ts.isFunctionDeclaration(node) ||
|
||||
ts.isMethodDeclaration(node) ||
|
||||
ts.isVariableDeclaration(node)) &&
|
||||
node.name &&
|
||||
ts.isIdentifier(node.name)
|
||||
) {
|
||||
return node.name.text;
|
||||
}
|
||||
|
||||
if (ts.isPropertyAssignment(node)) {
|
||||
if (ts.isIdentifier(node.name) || ts.isStringLiteral(node.name)) {
|
||||
return node.name.text;
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
function isIgnoredTestHelperContent(content) {
|
||||
return /\bfrom\s+["']vitest["']/.test(content) || /\bfrom\s+["']@vitest\//.test(content);
|
||||
}
|
||||
|
||||
function isIgnoredTestHelperPath(filePath) {
|
||||
const normalized = filePath.split(path.sep).join("/");
|
||||
const base = path.basename(filePath);
|
||||
return (
|
||||
normalized.includes("/test/") ||
|
||||
/(?:^|[./-])test(?:[./-]|$)/.test(base) ||
|
||||
base.includes("test-support") ||
|
||||
base.includes("test-harness") ||
|
||||
base.includes("test-helper") ||
|
||||
base.includes("test-mocks")
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Finds dynamic import advisories in a single source file.
|
||||
*/
|
||||
export function findDynamicImportAdvisories(content, fileName = "source.ts") {
|
||||
const sourceFile = ts.createSourceFile(fileName, content, ts.ScriptTarget.Latest, true);
|
||||
const staticRuntimeImports = new Map();
|
||||
const dynamicImports = new Map();
|
||||
const directExecuteImports = [];
|
||||
const declarationStack = [];
|
||||
|
||||
const addLine = (map, specifier, line) => {
|
||||
const lines = map.get(specifier) ?? [];
|
||||
lines.push(line);
|
||||
map.set(specifier, lines);
|
||||
};
|
||||
|
||||
const visit = (node) => {
|
||||
const declarationName = readDeclarationName(node);
|
||||
if (declarationName) {
|
||||
declarationStack.push(declarationName);
|
||||
}
|
||||
|
||||
if (
|
||||
ts.isImportDeclaration(node) &&
|
||||
ts.isStringLiteral(node.moduleSpecifier) &&
|
||||
!isTypeOnlyImportDeclaration(node)
|
||||
) {
|
||||
addLine(staticRuntimeImports, node.moduleSpecifier.text, toLine(sourceFile, node));
|
||||
}
|
||||
|
||||
if (
|
||||
ts.isExportDeclaration(node) &&
|
||||
node.moduleSpecifier &&
|
||||
ts.isStringLiteral(node.moduleSpecifier) &&
|
||||
!isTypeOnlyExportDeclaration(node)
|
||||
) {
|
||||
addLine(staticRuntimeImports, node.moduleSpecifier.text, toLine(sourceFile, node));
|
||||
}
|
||||
|
||||
if (
|
||||
ts.isCallExpression(node) &&
|
||||
node.expression.kind === ts.SyntaxKind.ImportKeyword &&
|
||||
node.arguments.length > 0
|
||||
) {
|
||||
const specifier = readStringLiteral(node.arguments[0]);
|
||||
if (specifier) {
|
||||
const line = toLine(sourceFile, node);
|
||||
addLine(dynamicImports, specifier, line);
|
||||
if (declarationStack.includes("execute")) {
|
||||
directExecuteImports.push({
|
||||
line,
|
||||
reason: `direct dynamic import of "${specifier}" inside execute path; move it behind a cached loader`,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
ts.forEachChild(node, visit);
|
||||
if (declarationName) {
|
||||
declarationStack.pop();
|
||||
}
|
||||
};
|
||||
|
||||
visit(sourceFile);
|
||||
|
||||
const advisories = [...directExecuteImports];
|
||||
for (const [specifier, dynamicLines] of dynamicImports) {
|
||||
const staticLines = staticRuntimeImports.get(specifier);
|
||||
if (staticLines?.length) {
|
||||
advisories.push({
|
||||
line: dynamicLines[0],
|
||||
reason: `runtime static + dynamic import of "${specifier}" (static line ${staticLines[0]})`,
|
||||
});
|
||||
}
|
||||
if (dynamicLines.length > 1) {
|
||||
advisories.push({
|
||||
line: dynamicLines[0],
|
||||
reason: `repeated direct dynamic import of "${specifier}" (${dynamicLines.length} callsites: ${dynamicLines.join(", ")})`,
|
||||
});
|
||||
}
|
||||
}
|
||||
return advisories;
|
||||
}
|
||||
|
||||
/**
|
||||
* Collects dynamic import advisories across configured source roots.
|
||||
*/
|
||||
export async function collectDynamicImportAdvisories(options = {}) {
|
||||
const roots = options.roots ?? defaultRoots;
|
||||
const files = await collectTypeScriptFilesFromRoots(roots, {
|
||||
extraTestSuffixes: [".suite.ts"],
|
||||
});
|
||||
const advisories = [];
|
||||
for (const filePath of files) {
|
||||
if (isIgnoredTestHelperPath(filePath)) {
|
||||
continue;
|
||||
}
|
||||
const content = await fs.readFile(filePath, "utf8");
|
||||
if (isIgnoredTestHelperContent(content)) {
|
||||
continue;
|
||||
}
|
||||
for (const advisory of findDynamicImportAdvisories(content, filePath)) {
|
||||
advisories.push({
|
||||
path: path.relative(repoRoot, filePath),
|
||||
...advisory,
|
||||
});
|
||||
}
|
||||
}
|
||||
return advisories;
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the dynamic import advisory check.
|
||||
*/
|
||||
export async function main(argv = process.argv.slice(2)) {
|
||||
const fail = argv.includes("--fail");
|
||||
const json = argv.includes("--json");
|
||||
const advisories = await collectDynamicImportAdvisories();
|
||||
|
||||
if (json) {
|
||||
console.log(JSON.stringify({ advisories }, null, 2));
|
||||
} else if (advisories.length === 0) {
|
||||
console.log("No dynamic import advisories found.");
|
||||
} else {
|
||||
console.log(`Dynamic import advisories (${advisories.length}):`);
|
||||
for (const advisory of advisories) {
|
||||
console.log(`- ${advisory.path}:${advisory.line} ${advisory.reason}`);
|
||||
}
|
||||
console.log("Advisory only. Use --fail when ratcheting this into a hard check.");
|
||||
}
|
||||
|
||||
if (fail && advisories.length > 0) {
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
runAsScript(import.meta.url, main);
|
||||
1025
scripts/check-extension-package-tsc-boundary.mjs
Normal file
1025
scripts/check-extension-package-tsc-boundary.mjs
Normal file
File diff suppressed because it is too large
Load Diff
372
scripts/check-extension-plugin-sdk-boundary.mjs
Normal file
372
scripts/check-extension-plugin-sdk-boundary.mjs
Normal file
@@ -0,0 +1,372 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Inventories extension imports to enforce plugin SDK boundary rules.
|
||||
import { promises as fs } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import {
|
||||
BUNDLED_PLUGIN_PATH_PREFIX,
|
||||
BUNDLED_PLUGIN_ROOT_DIR,
|
||||
} from "./lib/bundled-plugin-paths.mjs";
|
||||
import { classifyBundledExtensionSourcePath } from "./lib/extension-source-classifier.mjs";
|
||||
import {
|
||||
collectModuleReferencesFromSource,
|
||||
diffInventoryEntries,
|
||||
normalizeRepoPath,
|
||||
resolveRepoSpecifier,
|
||||
writeLine,
|
||||
} from "./lib/guard-inventory-utils.mjs";
|
||||
|
||||
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const extensionsRoot = path.join(repoRoot, BUNDLED_PLUGIN_ROOT_DIR);
|
||||
|
||||
const MODES = new Set([
|
||||
"src-outside-plugin-sdk",
|
||||
"plugin-sdk-internal",
|
||||
"relative-outside-package",
|
||||
]);
|
||||
|
||||
const baselinePathByMode = {
|
||||
"src-outside-plugin-sdk": path.join(
|
||||
repoRoot,
|
||||
"test",
|
||||
"fixtures",
|
||||
"extension-src-outside-plugin-sdk-inventory.json",
|
||||
),
|
||||
"plugin-sdk-internal": path.join(
|
||||
repoRoot,
|
||||
"test",
|
||||
"fixtures",
|
||||
"extension-plugin-sdk-internal-inventory.json",
|
||||
),
|
||||
};
|
||||
|
||||
let allInventoryByModePromise;
|
||||
let extensionModuleReferencesPromise;
|
||||
|
||||
const ruleTextByMode = {
|
||||
"src-outside-plugin-sdk":
|
||||
"Rule: production bundled plugins must not import src/** outside src/plugin-sdk/**",
|
||||
"plugin-sdk-internal":
|
||||
"Rule: production bundled plugins must not import src/plugin-sdk-internal/**",
|
||||
"relative-outside-package":
|
||||
"Rule: production bundled plugins must not use relative imports that escape their own package root",
|
||||
};
|
||||
|
||||
function isCodeFile(fileName) {
|
||||
return /\.(ts|tsx|mts|cts|js|jsx|mjs|cjs)$/.test(fileName);
|
||||
}
|
||||
|
||||
function isBoundaryCanaryFile(fileName) {
|
||||
return fileName.includes("__rootdir_boundary_canary__");
|
||||
}
|
||||
|
||||
async function collectExtensionSourceFiles(rootDir) {
|
||||
const out = [];
|
||||
async function walk(dir) {
|
||||
const entries = await fs.readdir(dir, { withFileTypes: true });
|
||||
for (const entry of entries) {
|
||||
if (entry.name === "dist" || entry.name === "node_modules") {
|
||||
continue;
|
||||
}
|
||||
const fullPath = path.join(dir, entry.name);
|
||||
if (entry.isDirectory()) {
|
||||
await walk(fullPath);
|
||||
continue;
|
||||
}
|
||||
if (!entry.isFile() || !isCodeFile(entry.name) || isBoundaryCanaryFile(entry.name)) {
|
||||
continue;
|
||||
}
|
||||
const relativePath = normalizeRepoPath(repoRoot, fullPath);
|
||||
if (classifyBundledExtensionSourcePath(relativePath).isTestLike) {
|
||||
continue;
|
||||
}
|
||||
out.push(fullPath);
|
||||
}
|
||||
}
|
||||
await walk(rootDir);
|
||||
return out.toSorted((left, right) =>
|
||||
normalizeRepoPath(repoRoot, left).localeCompare(normalizeRepoPath(repoRoot, right)),
|
||||
);
|
||||
}
|
||||
|
||||
async function collectExtensionModuleReferences() {
|
||||
if (!extensionModuleReferencesPromise) {
|
||||
extensionModuleReferencesPromise = (async () => {
|
||||
const files = await collectExtensionSourceFiles(extensionsRoot);
|
||||
const referenced = await Promise.all(
|
||||
files.map(async (filePath) => {
|
||||
const source = await fs.readFile(filePath, "utf8");
|
||||
if (!mayContainModuleSpecifier(source)) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
filePath,
|
||||
references: collectModuleReferencesFromSource(source),
|
||||
};
|
||||
}),
|
||||
);
|
||||
return referenced.filter((entry) => entry && entry.references.length > 0);
|
||||
})();
|
||||
}
|
||||
return await extensionModuleReferencesPromise;
|
||||
}
|
||||
|
||||
function mayContainModuleSpecifier(source) {
|
||||
return (
|
||||
/\bfrom\s*["']/.test(source) ||
|
||||
/\bimport\s*(?:\(|["']|type\b|[\w*{])/.test(source) ||
|
||||
/\bexport\s*(?:type\s+)?(?:\*|{)[^;\n]*\bfrom\s*["']/.test(source)
|
||||
);
|
||||
}
|
||||
|
||||
function resolveExtensionRoot(filePath) {
|
||||
const relativePath = normalizeRepoPath(repoRoot, filePath);
|
||||
const segments = relativePath.split("/");
|
||||
if (segments[0] !== BUNDLED_PLUGIN_ROOT_DIR || !segments[1]) {
|
||||
return null;
|
||||
}
|
||||
return `${segments[0]}/${segments[1]}`;
|
||||
}
|
||||
|
||||
function classifyReason(mode, kind, resolvedPath, specifier) {
|
||||
const verb =
|
||||
kind === "export"
|
||||
? "re-exports"
|
||||
: kind === "dynamic-import"
|
||||
? "dynamically imports"
|
||||
: "imports";
|
||||
if (mode === "relative-outside-package") {
|
||||
if (resolvedPath?.startsWith("src/plugin-sdk/")) {
|
||||
return `${verb} plugin-sdk via relative path; use openclaw/plugin-sdk/<subpath>`;
|
||||
}
|
||||
if (resolvedPath?.startsWith("src/")) {
|
||||
return `${verb} core src path via relative path outside the extension package`;
|
||||
}
|
||||
if (resolvedPath?.startsWith(BUNDLED_PLUGIN_PATH_PREFIX)) {
|
||||
return `${verb} another bundled plugin via relative path outside the extension package`;
|
||||
}
|
||||
return `${verb} relative path ${specifier} outside the extension package`;
|
||||
}
|
||||
if (mode === "plugin-sdk-internal") {
|
||||
return `${verb} src/plugin-sdk-internal from an extension`;
|
||||
}
|
||||
if (resolvedPath.startsWith("src/plugin-sdk/")) {
|
||||
return `${verb} allowed plugin-sdk path`;
|
||||
}
|
||||
return `${verb} core src path outside plugin-sdk from an extension`;
|
||||
}
|
||||
|
||||
function compareEntries(left, right) {
|
||||
return (
|
||||
left.file.localeCompare(right.file) ||
|
||||
left.line - right.line ||
|
||||
left.kind.localeCompare(right.kind) ||
|
||||
left.specifier.localeCompare(right.specifier) ||
|
||||
left.resolvedPath.localeCompare(right.resolvedPath) ||
|
||||
left.reason.localeCompare(right.reason)
|
||||
);
|
||||
}
|
||||
|
||||
function shouldReport(mode, resolvedPath) {
|
||||
if (mode === "relative-outside-package") {
|
||||
return false;
|
||||
}
|
||||
if (!resolvedPath?.startsWith("src/")) {
|
||||
return false;
|
||||
}
|
||||
if (mode === "plugin-sdk-internal") {
|
||||
return resolvedPath.startsWith("src/plugin-sdk-internal/");
|
||||
}
|
||||
return !resolvedPath.startsWith("src/plugin-sdk/");
|
||||
}
|
||||
|
||||
function collectEntriesByModeFromModuleReferences(filePath, references) {
|
||||
const entriesByMode = {
|
||||
"src-outside-plugin-sdk": [],
|
||||
"plugin-sdk-internal": [],
|
||||
"relative-outside-package": [],
|
||||
};
|
||||
const extensionRoot = resolveExtensionRoot(filePath);
|
||||
const relativeFile = normalizeRepoPath(repoRoot, filePath);
|
||||
|
||||
function push(kind, line, specifier) {
|
||||
const resolvedPath = resolveRepoSpecifier(repoRoot, specifier, filePath);
|
||||
const baseEntry = {
|
||||
file: relativeFile,
|
||||
line,
|
||||
kind,
|
||||
specifier,
|
||||
resolvedPath,
|
||||
};
|
||||
|
||||
if (specifier.startsWith(".") && resolvedPath && extensionRoot) {
|
||||
if (!(resolvedPath === extensionRoot || resolvedPath.startsWith(`${extensionRoot}/`))) {
|
||||
entriesByMode["relative-outside-package"].push({
|
||||
...baseEntry,
|
||||
reason: classifyReason("relative-outside-package", kind, resolvedPath, specifier),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
for (const mode of ["src-outside-plugin-sdk", "plugin-sdk-internal"]) {
|
||||
if (!shouldReport(mode, resolvedPath)) {
|
||||
continue;
|
||||
}
|
||||
entriesByMode[mode].push({
|
||||
...baseEntry,
|
||||
reason: classifyReason(mode, kind, resolvedPath, specifier),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
for (const { kind, line, specifier } of references) {
|
||||
push(kind, line, specifier);
|
||||
}
|
||||
return entriesByMode;
|
||||
}
|
||||
|
||||
/**
|
||||
* Collects the current extension plugin SDK boundary inventory.
|
||||
*/
|
||||
export async function collectExtensionPluginSdkBoundaryInventory(mode) {
|
||||
if (!MODES.has(mode)) {
|
||||
throw new Error(`Unknown mode: ${mode}`);
|
||||
}
|
||||
if (!allInventoryByModePromise) {
|
||||
allInventoryByModePromise = (async () => {
|
||||
const files = await collectExtensionModuleReferences();
|
||||
const inventoryByMode = {
|
||||
"src-outside-plugin-sdk": [],
|
||||
"plugin-sdk-internal": [],
|
||||
"relative-outside-package": [],
|
||||
};
|
||||
for (const { filePath, references } of files) {
|
||||
const entriesByMode = collectEntriesByModeFromModuleReferences(filePath, references);
|
||||
for (const inventoryMode of MODES) {
|
||||
inventoryByMode[inventoryMode].push(...entriesByMode[inventoryMode]);
|
||||
}
|
||||
}
|
||||
for (const inventoryMode of MODES) {
|
||||
inventoryByMode[inventoryMode] = inventoryByMode[inventoryMode].toSorted(compareEntries);
|
||||
}
|
||||
return inventoryByMode;
|
||||
})();
|
||||
}
|
||||
const inventoryByMode = await allInventoryByModePromise;
|
||||
return inventoryByMode[mode];
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads the checked-in expected boundary inventory.
|
||||
*/
|
||||
export async function readExpectedInventory(mode) {
|
||||
try {
|
||||
return JSON.parse(await fs.readFile(baselinePathByMode[mode], "utf8"));
|
||||
} catch (error) {
|
||||
if (
|
||||
(mode === "plugin-sdk-internal" || mode === "src-outside-plugin-sdk") &&
|
||||
error &&
|
||||
typeof error === "object" &&
|
||||
"code" in error &&
|
||||
error.code === "ENOENT"
|
||||
) {
|
||||
return [];
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Diffs expected and actual boundary inventory entries.
|
||||
*/
|
||||
export function diffInventory(expected, actual) {
|
||||
return diffInventoryEntries(expected, actual, compareEntries);
|
||||
}
|
||||
|
||||
function formatInventoryHuman(mode, inventory) {
|
||||
const lines = [ruleTextByMode[mode]];
|
||||
if (inventory.length === 0) {
|
||||
lines.push("No extension plugin-sdk boundary violations found.");
|
||||
return lines.join("\n");
|
||||
}
|
||||
lines.push("Extension boundary inventory:");
|
||||
let activeFile = "";
|
||||
for (const entry of inventory) {
|
||||
if (entry.file !== activeFile) {
|
||||
activeFile = entry.file;
|
||||
lines.push(activeFile);
|
||||
}
|
||||
lines.push(` - line ${entry.line} [${entry.kind}] ${entry.reason}`);
|
||||
lines.push(` specifier: ${entry.specifier}`);
|
||||
lines.push(` resolved: ${entry.resolvedPath}`);
|
||||
}
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the boundary inventory check with CLI-style inputs and outputs.
|
||||
*/
|
||||
export async function runExtensionPluginSdkBoundaryCheck(argv, io) {
|
||||
const args = argv ?? process.argv.slice(2);
|
||||
const streams = io ?? { stdout: process.stdout, stderr: process.stderr };
|
||||
const json = args.includes("--json");
|
||||
const modeArg = args.find((arg) => arg.startsWith("--mode="));
|
||||
const mode = modeArg?.slice("--mode=".length) ?? "src-outside-plugin-sdk";
|
||||
if (!MODES.has(mode)) {
|
||||
throw new Error(`Unknown mode: ${mode}`);
|
||||
}
|
||||
|
||||
const actual = await collectExtensionPluginSdkBoundaryInventory(mode);
|
||||
if (json) {
|
||||
writeLine(streams.stdout, JSON.stringify(actual, null, 2));
|
||||
return 0;
|
||||
}
|
||||
|
||||
writeLine(streams.stdout, formatInventoryHuman(mode, actual));
|
||||
if (mode === "relative-outside-package") {
|
||||
if (actual.length === 0) {
|
||||
return 0;
|
||||
}
|
||||
writeLine(
|
||||
streams.stderr,
|
||||
`Relative outside-package violations found (${actual.length}); this mode no longer uses a baseline.`,
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
|
||||
const expected = await readExpectedInventory(mode);
|
||||
const diff = diffInventory(expected, actual);
|
||||
if (diff.missing.length === 0 && diff.unexpected.length === 0) {
|
||||
writeLine(streams.stdout, `Baseline matches (${actual.length} entries).`);
|
||||
return 0;
|
||||
}
|
||||
if (diff.missing.length > 0) {
|
||||
writeLine(streams.stderr, `Missing baseline entries (${diff.missing.length}):`);
|
||||
for (const entry of diff.missing) {
|
||||
writeLine(streams.stderr, ` - ${entry.file}:${entry.line} ${entry.reason}`);
|
||||
}
|
||||
}
|
||||
if (diff.unexpected.length > 0) {
|
||||
writeLine(streams.stderr, `Unexpected inventory entries (${diff.unexpected.length}):`);
|
||||
for (const entry of diff.unexpected) {
|
||||
writeLine(streams.stderr, ` - ${entry.file}:${entry.line} ${entry.reason}`);
|
||||
}
|
||||
}
|
||||
return 1;
|
||||
}
|
||||
|
||||
/**
|
||||
* Entrypoint wrapper for the extension plugin SDK boundary check.
|
||||
*/
|
||||
export async function main(argv, io) {
|
||||
const exitCode = await runExtensionPluginSdkBoundaryCheck(argv, io);
|
||||
if (!io) {
|
||||
process.exitCode = exitCode;
|
||||
}
|
||||
return exitCode;
|
||||
}
|
||||
|
||||
if (path.resolve(process.argv[1] ?? "") === fileURLToPath(import.meta.url)) {
|
||||
await main();
|
||||
}
|
||||
99
scripts/check-extension-wildcard-reexports.mjs
Normal file
99
scripts/check-extension-wildcard-reexports.mjs
Normal file
@@ -0,0 +1,99 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Rejects local wildcard re-exports in guarded extension API barrels.
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
||||
|
||||
const LOCAL_WILDCARD_REEXPORT_PATTERN = /^\s*export\s+(?:type\s+)?\*\s+from\s+["'](?:\.{1,2}\/)/u;
|
||||
|
||||
async function walkFiles(rootDir, predicate) {
|
||||
const files = [];
|
||||
async function visit(dir) {
|
||||
const entries = await fs.readdir(dir, { withFileTypes: true });
|
||||
for (const entry of entries) {
|
||||
if (entry.name === "node_modules" || entry.name === ".git" || entry.name === "dist") {
|
||||
continue;
|
||||
}
|
||||
const filePath = path.join(dir, entry.name);
|
||||
if (entry.isDirectory()) {
|
||||
await visit(filePath);
|
||||
continue;
|
||||
}
|
||||
if (entry.isFile() && predicate(filePath)) {
|
||||
files.push(filePath);
|
||||
}
|
||||
}
|
||||
}
|
||||
await visit(rootDir);
|
||||
return files.toSorted((left, right) => left.localeCompare(right));
|
||||
}
|
||||
|
||||
async function listGuardedFiles(rootDir = repoRoot) {
|
||||
return walkFiles(
|
||||
path.join(rootDir, "extensions"),
|
||||
(filePath) =>
|
||||
filePath.endsWith(`${path.sep}runtime-api.ts`) || filePath.endsWith(`${path.sep}api.ts`),
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Finds local wildcard re-export lines in a barrel source string.
|
||||
*/
|
||||
export function findLocalWildcardReexports(source) {
|
||||
return source
|
||||
.split(/\r?\n/u)
|
||||
.map((text, index) => ({ line: index + 1, text }))
|
||||
.filter(({ text }) => LOCAL_WILDCARD_REEXPORT_PATTERN.test(text));
|
||||
}
|
||||
|
||||
/**
|
||||
* Collects guarded extension API/runtime barrels that use wildcard re-exports.
|
||||
*/
|
||||
export async function collectExtensionWildcardReexports(rootDir = repoRoot) {
|
||||
const files = await listGuardedFiles(rootDir);
|
||||
const violations = [];
|
||||
for (const filePath of files) {
|
||||
const source = await fs.readFile(filePath, "utf8");
|
||||
for (const match of findLocalWildcardReexports(source)) {
|
||||
violations.push({
|
||||
file: path.relative(rootDir, filePath).split(path.sep).join("/"),
|
||||
line: match.line,
|
||||
text: match.text.trim(),
|
||||
});
|
||||
}
|
||||
}
|
||||
return violations;
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the extension wildcard re-export guard.
|
||||
*/
|
||||
export async function main(argv = process.argv.slice(2), io = process) {
|
||||
const json = argv.includes("--json");
|
||||
const violations = await collectExtensionWildcardReexports();
|
||||
|
||||
if (json) {
|
||||
io.stdout.write(`${JSON.stringify(violations, null, 2)}\n`);
|
||||
return violations.length === 0 ? 0 : 1;
|
||||
}
|
||||
|
||||
if (violations.length === 0) {
|
||||
io.stdout.write("No guarded extension wildcard re-exports found.\n");
|
||||
return 0;
|
||||
}
|
||||
|
||||
io.stderr.write("Found guarded extension wildcard re-exports:\n");
|
||||
for (const violation of violations) {
|
||||
io.stderr.write(`- ${violation.file}:${violation.line} ${violation.text}\n`);
|
||||
}
|
||||
io.stderr.write("Use explicit named exports so runtime and public API barrels stay pinned.\n");
|
||||
return 1;
|
||||
}
|
||||
|
||||
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
||||
const exitCode = await main();
|
||||
process.exit(exitCode);
|
||||
}
|
||||
64
scripts/check-file-utils.ts
Normal file
64
scripts/check-file-utils.ts
Normal file
@@ -0,0 +1,64 @@
|
||||
// Check File Utils helper supports OpenClaw script workflows.
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
|
||||
const DEFAULT_SKIPPED_DIR_NAMES = new Set(["node_modules", "dist", "coverage", ".generated"]);
|
||||
|
||||
export function isCodeFile(filePath: string): boolean {
|
||||
if (filePath.endsWith(".d.ts")) {
|
||||
return false;
|
||||
}
|
||||
return /\.(?:[cm]?ts|[cm]?js|tsx|jsx)$/u.test(filePath);
|
||||
}
|
||||
|
||||
export function collectFilesSync(
|
||||
rootDir: string,
|
||||
options: {
|
||||
includeFile: (filePath: string) => boolean;
|
||||
skipDirNames?: ReadonlySet<string>;
|
||||
},
|
||||
): string[] {
|
||||
const skipDirNames = options.skipDirNames ?? DEFAULT_SKIPPED_DIR_NAMES;
|
||||
const files: string[] = [];
|
||||
const stack = [rootDir];
|
||||
|
||||
while (stack.length > 0) {
|
||||
const current = stack.pop();
|
||||
if (!current) {
|
||||
continue;
|
||||
}
|
||||
let entries: fs.Dirent[];
|
||||
try {
|
||||
entries = fs.readdirSync(current, { withFileTypes: true });
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
for (const entry of entries) {
|
||||
const fullPath = path.join(current, entry.name);
|
||||
if (entry.isDirectory()) {
|
||||
if (skipDirNames.has(entry.name)) {
|
||||
continue;
|
||||
}
|
||||
stack.push(fullPath);
|
||||
continue;
|
||||
}
|
||||
if (entry.isFile() && options.includeFile(fullPath)) {
|
||||
files.push(fullPath);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return files;
|
||||
}
|
||||
|
||||
export function toPosixPath(filePath: string): string {
|
||||
if (path.sep === "/") {
|
||||
return filePath;
|
||||
}
|
||||
return filePath.replaceAll("\\", "/");
|
||||
}
|
||||
|
||||
export function relativeToCwd(filePath: string): string {
|
||||
const relativePath = path.relative(process.cwd(), filePath) || filePath;
|
||||
return toPosixPath(relativePath);
|
||||
}
|
||||
470
scripts/check-gateway-cpu-scenarios.mjs
Normal file
470
scripts/check-gateway-cpu-scenarios.mjs
Normal file
@@ -0,0 +1,470 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Runs gateway startup and QA scenarios while checking hot CPU observations.
|
||||
import { spawnSync as defaultSpawnSync } from "node:child_process";
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import process from "node:process";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { stripLeadingPackageManagerSeparator } from "./lib/arg-utils.mjs";
|
||||
import {
|
||||
parseNonNegativeInt,
|
||||
parsePositiveInt,
|
||||
parsePositiveNumber,
|
||||
} from "./lib/numeric-options.mjs";
|
||||
import {
|
||||
collectGatewayCpuObservations,
|
||||
readQaSuiteSummary,
|
||||
} from "./lib/plugin-gateway-gauntlet.mjs";
|
||||
import { createPnpmRunnerSpawnSpec } from "./pnpm-runner.mjs";
|
||||
|
||||
const DEFAULT_STARTUP_CASES = ["default", "oneInternalHook", "allInternalHooks"];
|
||||
const DEFAULT_QA_SCENARIOS = [
|
||||
"channel-chat-baseline",
|
||||
"memory-failure-fallback",
|
||||
"gateway-restart-inflight-run",
|
||||
];
|
||||
const SINGLE_VALUE_FLAGS = new Set([
|
||||
"--cpu-core-warn",
|
||||
"--hot-wall-warn-ms",
|
||||
"--output-dir",
|
||||
"--runs",
|
||||
"--warmup",
|
||||
]);
|
||||
const DEFAULT_CPU_CORE_WARN = 0.9;
|
||||
const DEFAULT_HOT_WALL_WARN_MS = 30_000;
|
||||
const PRIVATE_QA_REQUIRED_DIST_ENTRIES = [
|
||||
"dist/plugin-sdk/qa-lab.js",
|
||||
"dist/plugin-sdk/qa-runtime.js",
|
||||
];
|
||||
|
||||
function parseArgs(argv) {
|
||||
const args = stripLeadingPackageManagerSeparator(argv);
|
||||
const options = {
|
||||
outputDir: path.join(
|
||||
process.cwd(),
|
||||
".artifacts",
|
||||
"gateway-cpu-scenarios",
|
||||
new Date().toISOString().replace(/[:.]/g, "-"),
|
||||
),
|
||||
startupCases: [],
|
||||
qaScenarios: [],
|
||||
runs: 1,
|
||||
warmup: 0,
|
||||
skipStartup: false,
|
||||
skipQa: false,
|
||||
cpuCoreWarn: DEFAULT_CPU_CORE_WARN,
|
||||
hotWallWarnMs: DEFAULT_HOT_WALL_WARN_MS,
|
||||
};
|
||||
const seenSingleValueFlags = new Set();
|
||||
for (let index = 0; index < args.length; index += 1) {
|
||||
const arg = args[index];
|
||||
if (SINGLE_VALUE_FLAGS.has(arg)) {
|
||||
if (seenSingleValueFlags.has(arg)) {
|
||||
throw new Error(`${arg} was provided more than once`);
|
||||
}
|
||||
seenSingleValueFlags.add(arg);
|
||||
}
|
||||
const readValue = () => {
|
||||
const value = args[index + 1];
|
||||
if (!value || value.startsWith("-")) {
|
||||
throw new Error(`Missing value for ${arg}`);
|
||||
}
|
||||
index += 1;
|
||||
return value;
|
||||
};
|
||||
switch (arg) {
|
||||
case "--output-dir":
|
||||
options.outputDir = path.resolve(readValue());
|
||||
break;
|
||||
case "--startup-case":
|
||||
options.startupCases.push(readValue());
|
||||
break;
|
||||
case "--qa-scenario":
|
||||
options.qaScenarios.push(readValue());
|
||||
break;
|
||||
case "--runs":
|
||||
options.runs = parsePositiveInt(readValue(), "--runs");
|
||||
break;
|
||||
case "--warmup":
|
||||
options.warmup = parseNonNegativeInt(readValue(), "--warmup");
|
||||
break;
|
||||
case "--cpu-core-warn":
|
||||
options.cpuCoreWarn = parsePositiveNumber(readValue(), "--cpu-core-warn");
|
||||
break;
|
||||
case "--hot-wall-warn-ms":
|
||||
options.hotWallWarnMs = parsePositiveInt(readValue(), "--hot-wall-warn-ms");
|
||||
break;
|
||||
case "--skip-startup":
|
||||
options.skipStartup = true;
|
||||
break;
|
||||
case "--skip-qa":
|
||||
options.skipQa = true;
|
||||
break;
|
||||
case "--help":
|
||||
printHelp();
|
||||
process.exit(0);
|
||||
break;
|
||||
default:
|
||||
throw new Error(`Unknown argument: ${arg}`);
|
||||
}
|
||||
}
|
||||
if (options.startupCases.length === 0) {
|
||||
options.startupCases = [...DEFAULT_STARTUP_CASES];
|
||||
}
|
||||
if (options.qaScenarios.length === 0) {
|
||||
options.qaScenarios = [...DEFAULT_QA_SCENARIOS];
|
||||
}
|
||||
if (options.skipStartup && options.skipQa) {
|
||||
throw new Error("--skip-startup and --skip-qa cannot be used together");
|
||||
}
|
||||
return options;
|
||||
}
|
||||
|
||||
function printHelp() {
|
||||
console.log(`Usage: pnpm test:gateway:cpu-scenarios [options]
|
||||
|
||||
Runs a small gateway CPU scenario suite against built dist artifacts.
|
||||
|
||||
Options:
|
||||
--output-dir <path> Artifact directory
|
||||
--startup-case <id> Startup bench case, repeatable
|
||||
--qa-scenario <id> QA Lab scenario, repeatable
|
||||
--runs <count> Startup bench runs per case (default: 1)
|
||||
--warmup <count> Startup bench warmup runs per case (default: 0)
|
||||
--cpu-core-warn <ratio> Hot CPU observation threshold (default: 0.9)
|
||||
--hot-wall-warn-ms <ms> Minimum wall time for hot CPU observations (default: 30000)
|
||||
--skip-startup Skip startup bench
|
||||
--skip-qa Skip QA Lab scenario smoke
|
||||
`);
|
||||
}
|
||||
|
||||
function readJsonIfExists(filePath) {
|
||||
if (!fs.existsSync(filePath)) {
|
||||
return null;
|
||||
}
|
||||
return JSON.parse(fs.readFileSync(filePath, "utf8"));
|
||||
}
|
||||
|
||||
function validateStartupReport(report) {
|
||||
if (!report || typeof report !== "object" || Array.isArray(report)) {
|
||||
return "startup report must be a JSON object";
|
||||
}
|
||||
if (!Array.isArray(report.results)) {
|
||||
return "startup report missing results array";
|
||||
}
|
||||
if (report.results.length === 0) {
|
||||
return "startup report has no measured results";
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function readStartupReport(startupOutput) {
|
||||
if (!fs.existsSync(startupOutput)) {
|
||||
return {
|
||||
diagnosticFailure: "startup-report-missing",
|
||||
diagnosticDetail: `expected startup bench report at ${startupOutput}`,
|
||||
report: null,
|
||||
};
|
||||
}
|
||||
try {
|
||||
const report = readJsonIfExists(startupOutput);
|
||||
const invalidReason = validateStartupReport(report);
|
||||
if (invalidReason) {
|
||||
return {
|
||||
diagnosticFailure: "startup-report-invalid",
|
||||
diagnosticDetail: invalidReason,
|
||||
report: null,
|
||||
};
|
||||
}
|
||||
return {
|
||||
diagnosticFailure: null,
|
||||
diagnosticDetail: null,
|
||||
report,
|
||||
};
|
||||
} catch (error) {
|
||||
return {
|
||||
diagnosticFailure: "startup-report-invalid",
|
||||
diagnosticDetail: error instanceof Error ? error.message : String(error),
|
||||
report: null,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
function runStep(name, command, args, options = {}, params = {}) {
|
||||
console.error(`[gateway-cpu] start ${name}`);
|
||||
const spawn = params.spawnSync ?? defaultSpawnSync;
|
||||
const result = spawn(command, args, {
|
||||
cwd: params.cwd ?? process.cwd(),
|
||||
env: params.env ?? process.env,
|
||||
stdio: "inherit",
|
||||
...options,
|
||||
});
|
||||
const error =
|
||||
result.error instanceof Error
|
||||
? result.error.message
|
||||
: result.error
|
||||
? String(result.error)
|
||||
: null;
|
||||
const status = result.error ? 1 : (result.status ?? (result.signal ? 1 : 0));
|
||||
console.error(
|
||||
`[gateway-cpu] ${status === 0 ? "pass" : "fail"} ${name}${error ? `: ${error}` : ""}`,
|
||||
);
|
||||
return {
|
||||
name,
|
||||
status,
|
||||
signal: result.signal ?? null,
|
||||
...(error ? { error } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
function pnpmCommand(args, params = {}) {
|
||||
return createPnpmRunnerSpawnSpec({
|
||||
cwd: params.cwd ?? process.cwd(),
|
||||
env: params.env ?? process.env,
|
||||
pnpmArgs: args,
|
||||
stdio: "inherit",
|
||||
});
|
||||
}
|
||||
|
||||
function toRepoRelativePath(repoRoot, absolutePath) {
|
||||
const relativePath = path.relative(repoRoot, absolutePath);
|
||||
if (!relativePath || relativePath.startsWith("..") || path.isAbsolute(relativePath)) {
|
||||
throw new Error(`Output path must stay inside the repo root: ${absolutePath}`);
|
||||
}
|
||||
return relativePath;
|
||||
}
|
||||
|
||||
function hasPrivateQaDist(repoRoot, fsImpl = fs) {
|
||||
return PRIVATE_QA_REQUIRED_DIST_ENTRIES.every((relativePath) => {
|
||||
try {
|
||||
return fsImpl.statSync(path.join(repoRoot, relativePath)).isFile();
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
function buildPrivateQaEnv(env, qaState) {
|
||||
return {
|
||||
...env,
|
||||
...(qaState
|
||||
? {
|
||||
HOME: qaState.home,
|
||||
USERPROFILE: qaState.home,
|
||||
OPENCLAW_HOME: qaState.home,
|
||||
OPENCLAW_STATE_DIR: qaState.stateDir,
|
||||
OPENCLAW_CONFIG_PATH: qaState.configPath,
|
||||
}
|
||||
: {}),
|
||||
OPENCLAW_BUILD_PRIVATE_QA: "1",
|
||||
OPENCLAW_ENABLE_PRIVATE_QA_CLI: "1",
|
||||
OPENCLAW_RUN_NODE_SKIP_DTS_BUILD: env.OPENCLAW_RUN_NODE_SKIP_DTS_BUILD ?? "1",
|
||||
OPENCLAW_TEST_DISABLE_UPDATE_CHECK: env.OPENCLAW_TEST_DISABLE_UPDATE_CHECK ?? "1",
|
||||
PNPM_CONFIG_VERIFY_DEPS_BEFORE_RUN: env.PNPM_CONFIG_VERIFY_DEPS_BEFORE_RUN ?? "false",
|
||||
};
|
||||
}
|
||||
|
||||
function createQaState(outputDir) {
|
||||
const root = path.join(outputDir, "qa-state-root");
|
||||
const home = path.join(root, "home");
|
||||
const stateDir = path.join(root, "state");
|
||||
return {
|
||||
configPath: path.join(stateDir, "openclaw.json"),
|
||||
home,
|
||||
root,
|
||||
stateDir,
|
||||
};
|
||||
}
|
||||
|
||||
async function runGatewayCpuScenarios(options, params = {}) {
|
||||
const repoRoot = params.cwd ?? process.cwd();
|
||||
const inputEnv = params.env ?? process.env;
|
||||
const baseEnv = {
|
||||
...inputEnv,
|
||||
PNPM_CONFIG_VERIFY_DEPS_BEFORE_RUN: inputEnv.PNPM_CONFIG_VERIFY_DEPS_BEFORE_RUN ?? "false",
|
||||
};
|
||||
fs.mkdirSync(options.outputDir, { recursive: true });
|
||||
|
||||
const startupOutput = path.join(options.outputDir, "gateway-startup-bench.json");
|
||||
const qaOutputDir = path.join(options.outputDir, "qa-suite");
|
||||
const qaSummaryPath = path.join(qaOutputDir, "qa-suite-summary.json");
|
||||
const qaState = options.skipQa ? null : createQaState(options.outputDir);
|
||||
if (qaState) {
|
||||
fs.mkdirSync(qaState.home, { recursive: true });
|
||||
fs.mkdirSync(qaState.stateDir, { recursive: true });
|
||||
}
|
||||
const qaBuildEnv = buildPrivateQaEnv(baseEnv, qaState);
|
||||
const qaOutputArg = toRepoRelativePath(repoRoot, qaOutputDir);
|
||||
const steps = [];
|
||||
|
||||
if (!options.skipStartup) {
|
||||
const startupBuild = runStep(
|
||||
"startup build",
|
||||
process.execPath,
|
||||
["scripts/ensure-cli-startup-build.mjs"],
|
||||
{ env: baseEnv },
|
||||
params,
|
||||
);
|
||||
steps.push(startupBuild);
|
||||
steps.push(
|
||||
startupBuild.status === 0
|
||||
? runStep(
|
||||
"startup bench",
|
||||
process.execPath,
|
||||
[
|
||||
"--import",
|
||||
"tsx",
|
||||
"scripts/bench-gateway-startup.ts",
|
||||
"--runs",
|
||||
String(options.runs),
|
||||
"--warmup",
|
||||
String(options.warmup),
|
||||
"--output",
|
||||
startupOutput,
|
||||
...options.startupCases.flatMap((id) => ["--case", id]),
|
||||
],
|
||||
{ env: baseEnv },
|
||||
params,
|
||||
)
|
||||
: { name: "startup bench", signal: null, status: 1 },
|
||||
);
|
||||
}
|
||||
|
||||
let privateQaBuildFailed = false;
|
||||
if (!options.skipQa && !hasPrivateQaDist(repoRoot, params.fs ?? fs)) {
|
||||
const privateQaBuild = runStep(
|
||||
"private QA build",
|
||||
process.execPath,
|
||||
["scripts/build-all.mjs", "qaRuntime"],
|
||||
{ env: qaBuildEnv },
|
||||
params,
|
||||
);
|
||||
steps.push(privateQaBuild);
|
||||
privateQaBuildFailed = privateQaBuild.status !== 0;
|
||||
}
|
||||
|
||||
let qaStep = null;
|
||||
if (!options.skipQa) {
|
||||
const qaCommand = pnpmCommand(
|
||||
[
|
||||
"openclaw",
|
||||
"qa",
|
||||
"suite",
|
||||
"--provider-mode",
|
||||
"mock-openai",
|
||||
"--concurrency",
|
||||
"1",
|
||||
"--output-dir",
|
||||
qaOutputArg,
|
||||
...options.qaScenarios.flatMap((id) => ["--scenario", id]),
|
||||
],
|
||||
{ cwd: repoRoot, env: qaBuildEnv },
|
||||
);
|
||||
qaStep = privateQaBuildFailed
|
||||
? { name: "qa suite", signal: null, status: 1 }
|
||||
: runStep("qa suite", qaCommand.command, qaCommand.args, qaCommand.options, params);
|
||||
steps.push(qaStep);
|
||||
}
|
||||
|
||||
const startupReportResult = options.skipStartup ? null : readStartupReport(startupOutput);
|
||||
const startupReportFailure =
|
||||
steps.find((step) => step.name === "startup bench")?.status === 0
|
||||
? (startupReportResult?.diagnosticFailure ?? null)
|
||||
: null;
|
||||
const startup = startupReportResult?.report ?? null;
|
||||
const qaSummaryResult = options.skipQa ? null : readQaSuiteSummary(qaSummaryPath);
|
||||
const qaSummaryFailure =
|
||||
qaStep?.status === 0 ? (qaSummaryResult?.diagnosticFailure ?? null) : null;
|
||||
const qa = qaSummaryResult?.summary ?? null;
|
||||
const observations = collectGatewayCpuObservations({
|
||||
startup,
|
||||
qa,
|
||||
cpuCoreWarn: options.cpuCoreWarn,
|
||||
hotWallWarnMs: options.hotWallWarnMs,
|
||||
});
|
||||
const summary = {
|
||||
generatedAt: new Date().toISOString(),
|
||||
outputDir: options.outputDir,
|
||||
startupOutput: fs.existsSync(startupOutput) ? startupOutput : null,
|
||||
qaSummary: fs.existsSync(qaSummaryPath) ? qaSummaryPath : null,
|
||||
...(startupReportFailure
|
||||
? {
|
||||
startupReportFailure,
|
||||
startupReportFailureDetail: startupReportResult?.diagnosticDetail ?? null,
|
||||
}
|
||||
: {}),
|
||||
...(qaSummaryFailure
|
||||
? {
|
||||
qaSummaryFailure,
|
||||
qaSummaryFailureDetail: qaSummaryResult?.diagnosticDetail ?? null,
|
||||
}
|
||||
: {}),
|
||||
options: {
|
||||
startupCases: options.startupCases,
|
||||
qaScenarios: options.qaScenarios,
|
||||
runs: options.runs,
|
||||
warmup: options.warmup,
|
||||
cpuCoreWarn: options.cpuCoreWarn,
|
||||
hotWallWarnMs: options.hotWallWarnMs,
|
||||
qaStateDir: qaState?.stateDir ?? null,
|
||||
},
|
||||
steps,
|
||||
observations,
|
||||
};
|
||||
const summaryPath = path.join(options.outputDir, "summary.json");
|
||||
fs.writeFileSync(summaryPath, `${JSON.stringify(summary, null, 2)}\n`);
|
||||
if (!params.silent) {
|
||||
console.log(JSON.stringify(summary, null, 2));
|
||||
}
|
||||
if (observations.length > 0) {
|
||||
console.error(
|
||||
`[gateway-cpu] fail hot CPU observations: ${observations
|
||||
.map((observation) => `${observation.kind}:${observation.id}`)
|
||||
.join(", ")}`,
|
||||
);
|
||||
}
|
||||
if (qaSummaryFailure) {
|
||||
console.error(`[gateway-cpu] fail QA summary: ${qaSummaryResult?.diagnosticDetail}`);
|
||||
}
|
||||
if (startupReportFailure) {
|
||||
console.error(`[gateway-cpu] fail startup report: ${startupReportResult?.diagnosticDetail}`);
|
||||
}
|
||||
|
||||
const exitCode =
|
||||
steps.some((step) => step.status !== 0) ||
|
||||
observations.length > 0 ||
|
||||
qaSummaryFailure ||
|
||||
startupReportFailure
|
||||
? 1
|
||||
: 0;
|
||||
return { exitCode, summary };
|
||||
}
|
||||
|
||||
async function main(params = {}) {
|
||||
const options = parseArgs(params.argv ?? process.argv.slice(2));
|
||||
const result = await runGatewayCpuScenarios(options, params);
|
||||
if (result.exitCode !== 0) {
|
||||
process.exitCode = 1;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Test-only access to the gateway CPU scenario parser and runner helpers.
|
||||
*/
|
||||
export const testing = {
|
||||
hasPrivateQaDist,
|
||||
parseArgs,
|
||||
readStartupReport,
|
||||
runGatewayCpuScenarios,
|
||||
validateStartupReport,
|
||||
};
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
main().catch(
|
||||
/** @param {unknown} error */ (error) => {
|
||||
console.error(error instanceof Error ? error.message : String(error));
|
||||
process.exitCode = 1;
|
||||
},
|
||||
);
|
||||
}
|
||||
988
scripts/check-gateway-watch-regression.mjs
Normal file
988
scripts/check-gateway-watch-regression.mjs
Normal file
@@ -0,0 +1,988 @@
|
||||
#!/usr/bin/env node
|
||||
// Measures gateway watch idle CPU and dist/runtime artifact churn.
|
||||
|
||||
import { spawn, spawnSync } from "node:child_process";
|
||||
import fs from "node:fs";
|
||||
import net from "node:net";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import process from "node:process";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { stripLeadingPackageManagerSeparator } from "./lib/arg-utils.mjs";
|
||||
import {
|
||||
BUILD_STAMP_FILE,
|
||||
writeBuildStamp,
|
||||
writeRuntimePostBuildStamp,
|
||||
} from "./lib/local-build-metadata.mjs";
|
||||
import { sleep } from "./lib/sleep.mjs";
|
||||
import { resolveBuildRequirement } from "./run-node.mjs";
|
||||
|
||||
const DEFAULTS = {
|
||||
outputDir: path.join(process.cwd(), ".local", "gateway-watch-regression"),
|
||||
windowMs: 10_000,
|
||||
readyTimeoutMs: 20_000,
|
||||
readySettleMs: 500,
|
||||
sigkillGraceMs: 10_000,
|
||||
sigkillExitGraceMs: 2_000,
|
||||
cpuWarnMs: 1_000,
|
||||
cpuFailMs: 8_000,
|
||||
distRuntimeFileGrowthMax: 200,
|
||||
distRuntimeByteGrowthMax: 2 * 1024 * 1024,
|
||||
keepLogs: true,
|
||||
skipBuild: false,
|
||||
};
|
||||
|
||||
const WATCH_GATEWAY_SKIP_ENV = {
|
||||
OPENCLAW_DISABLE_BONJOUR: "1",
|
||||
OPENCLAW_SKIP_ACPX_RUNTIME: "1",
|
||||
OPENCLAW_SKIP_ACPX_RUNTIME_PROBE: "1",
|
||||
OPENCLAW_SKIP_BROWSER_CONTROL_SERVER: "1",
|
||||
OPENCLAW_SKIP_CANVAS_HOST: "1",
|
||||
OPENCLAW_SKIP_CHANNELS: "1",
|
||||
OPENCLAW_SKIP_CRON: "1",
|
||||
OPENCLAW_SKIP_GMAIL_WATCHER: "1",
|
||||
OPENCLAW_RUNTIME_POSTBUILD_STATIC_ASSETS: "0",
|
||||
OPENCLAW_TEST_MINIMAL_GATEWAY: "1",
|
||||
NODE_ENV: "test",
|
||||
};
|
||||
|
||||
/**
|
||||
* Maximum retained stdout/stderr text for gateway watch diagnostics.
|
||||
*/
|
||||
export const WATCH_LOG_CAPTURE_MAX_CHARS = 2 * 1024 * 1024;
|
||||
const WATCH_BUILD_DETECTION_MAX_CHARS = 4096;
|
||||
const NON_NEGATIVE_INTEGER_PATTERN = /^(0|[1-9]\d*)$/u;
|
||||
const ANSI_ESCAPE_PATTERN = new RegExp(`${String.fromCharCode(27)}\\[[0-?]*[ -/]*[@-~]`, "g");
|
||||
|
||||
/**
|
||||
* Appends watch output while preserving only the diagnostic tail.
|
||||
*/
|
||||
export function appendBoundedWatchLog(current, chunk, maxChars = WATCH_LOG_CAPTURE_MAX_CHARS) {
|
||||
const next = `${current}${String(chunk)}`;
|
||||
if (next.length <= maxChars) {
|
||||
return { text: next, truncated: false };
|
||||
}
|
||||
return { text: next.slice(-maxChars), truncated: true };
|
||||
}
|
||||
|
||||
function formatCapturedWatchLog(text, truncated) {
|
||||
return truncated
|
||||
? `[openclaw] log truncated to last ${WATCH_LOG_CAPTURE_MAX_CHARS} chars\n${text}`
|
||||
: text;
|
||||
}
|
||||
|
||||
/**
|
||||
* Updates bounded watch-build detection state from new output.
|
||||
*/
|
||||
export function updateWatchBuildDetection(state, chunk) {
|
||||
const combined = `${state.buffer ?? ""}${String(chunk)}`;
|
||||
const next = appendBoundedWatchLog("", combined, WATCH_BUILD_DETECTION_MAX_CHARS);
|
||||
const reason = detectWatchBuildReason(combined, "");
|
||||
const triggered = state.triggered || combined.includes("Building TypeScript (dist is stale");
|
||||
return {
|
||||
buffer: next.text,
|
||||
triggered,
|
||||
reason: state.reason ?? reason,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Parses a safe non-negative integer CLI value.
|
||||
*/
|
||||
export function readNonNegativeInteger(value, label) {
|
||||
const raw = String(value).trim();
|
||||
if (!NON_NEGATIVE_INTEGER_PATTERN.test(raw)) {
|
||||
throw new Error(`${label} must be a non-negative integer`);
|
||||
}
|
||||
const parsed = Number(raw);
|
||||
if (!Number.isSafeInteger(parsed)) {
|
||||
throw new Error(`${label} must be a safe integer`);
|
||||
}
|
||||
return parsed;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parses gateway watch regression CLI arguments.
|
||||
*/
|
||||
export function parseArgs(argv) {
|
||||
const args = stripLeadingPackageManagerSeparator(argv);
|
||||
const options = { ...DEFAULTS };
|
||||
for (let i = 0; i < args.length; i += 1) {
|
||||
const arg = args[i];
|
||||
const next = args[i + 1];
|
||||
const readValue = () => {
|
||||
if (!next) {
|
||||
throw new Error(`Missing value for ${arg}`);
|
||||
}
|
||||
i += 1;
|
||||
return next;
|
||||
};
|
||||
switch (arg) {
|
||||
case "--output-dir":
|
||||
options.outputDir = path.resolve(readValue());
|
||||
break;
|
||||
case "--window-ms":
|
||||
options.windowMs = readNonNegativeInteger(readValue(), "--window-ms");
|
||||
break;
|
||||
case "--ready-timeout-ms":
|
||||
options.readyTimeoutMs = readNonNegativeInteger(readValue(), "--ready-timeout-ms");
|
||||
break;
|
||||
case "--ready-settle-ms":
|
||||
options.readySettleMs = readNonNegativeInteger(readValue(), "--ready-settle-ms");
|
||||
break;
|
||||
case "--sigkill-grace-ms":
|
||||
options.sigkillGraceMs = readNonNegativeInteger(readValue(), "--sigkill-grace-ms");
|
||||
break;
|
||||
case "--sigkill-exit-grace-ms":
|
||||
options.sigkillExitGraceMs = readNonNegativeInteger(readValue(), "--sigkill-exit-grace-ms");
|
||||
break;
|
||||
case "--cpu-warn-ms":
|
||||
options.cpuWarnMs = readNonNegativeInteger(readValue(), "--cpu-warn-ms");
|
||||
break;
|
||||
case "--cpu-fail-ms":
|
||||
options.cpuFailMs = readNonNegativeInteger(readValue(), "--cpu-fail-ms");
|
||||
break;
|
||||
case "--dist-runtime-file-growth-max":
|
||||
options.distRuntimeFileGrowthMax = readNonNegativeInteger(
|
||||
readValue(),
|
||||
"--dist-runtime-file-growth-max",
|
||||
);
|
||||
break;
|
||||
case "--dist-runtime-byte-growth-max":
|
||||
options.distRuntimeByteGrowthMax = readNonNegativeInteger(
|
||||
readValue(),
|
||||
"--dist-runtime-byte-growth-max",
|
||||
);
|
||||
break;
|
||||
case "--skip-build":
|
||||
options.skipBuild = true;
|
||||
break;
|
||||
default:
|
||||
throw new Error(`Unknown argument: ${arg}`);
|
||||
}
|
||||
}
|
||||
return options;
|
||||
}
|
||||
|
||||
function ensureDir(dirPath) {
|
||||
fs.mkdirSync(dirPath, { recursive: true });
|
||||
}
|
||||
|
||||
function removePathIfExists(targetPath) {
|
||||
fs.rmSync(targetPath, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
function lstatIfExists(targetPath) {
|
||||
try {
|
||||
return fs.lstatSync(targetPath);
|
||||
} catch (error) {
|
||||
if (error?.code === "ENOENT") {
|
||||
return null;
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
function normalizePath(filePath) {
|
||||
return filePath.replaceAll("\\", "/");
|
||||
}
|
||||
|
||||
function listTreeEntries(rootName) {
|
||||
const rootPath = path.join(process.cwd(), rootName);
|
||||
if (!fs.existsSync(rootPath)) {
|
||||
return [`${rootName} (missing)`];
|
||||
}
|
||||
|
||||
const entries = [rootName];
|
||||
const queue = [rootPath];
|
||||
while (queue.length > 0) {
|
||||
const current = queue.pop();
|
||||
if (!current) {
|
||||
continue;
|
||||
}
|
||||
let dirents;
|
||||
try {
|
||||
dirents = fs.readdirSync(current, { withFileTypes: true });
|
||||
} catch (error) {
|
||||
if (error?.code === "ENOENT") {
|
||||
continue;
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
for (const dirent of dirents) {
|
||||
const fullPath = path.join(current, dirent.name);
|
||||
const relativePath = normalizePath(path.relative(process.cwd(), fullPath));
|
||||
entries.push(relativePath);
|
||||
if (dirent.isDirectory()) {
|
||||
queue.push(fullPath);
|
||||
}
|
||||
}
|
||||
}
|
||||
return entries.toSorted((a, b) => a.localeCompare(b));
|
||||
}
|
||||
|
||||
function humanBytes(bytes) {
|
||||
if (bytes < 1024) {
|
||||
return `${bytes}B`;
|
||||
}
|
||||
if (bytes < 1024 * 1024) {
|
||||
return `${(bytes / 1024).toFixed(1)}K`;
|
||||
}
|
||||
if (bytes < 1024 * 1024 * 1024) {
|
||||
return `${(bytes / (1024 * 1024)).toFixed(1)}M`;
|
||||
}
|
||||
return `${(bytes / (1024 * 1024 * 1024)).toFixed(1)}G`;
|
||||
}
|
||||
|
||||
function snapshotTree(rootName) {
|
||||
const rootPath = path.join(process.cwd(), rootName);
|
||||
const stats = {
|
||||
exists: fs.existsSync(rootPath),
|
||||
files: 0,
|
||||
directories: 0,
|
||||
symlinks: 0,
|
||||
entries: 0,
|
||||
apparentBytes: 0,
|
||||
};
|
||||
|
||||
if (!stats.exists) {
|
||||
return stats;
|
||||
}
|
||||
|
||||
const queue = [rootPath];
|
||||
while (queue.length > 0) {
|
||||
const current = queue.pop();
|
||||
if (!current) {
|
||||
continue;
|
||||
}
|
||||
const currentStats = lstatIfExists(current);
|
||||
if (!currentStats) {
|
||||
continue;
|
||||
}
|
||||
stats.entries += 1;
|
||||
if (currentStats.isDirectory()) {
|
||||
stats.directories += 1;
|
||||
for (const dirent of fs.readdirSync(current, { withFileTypes: true })) {
|
||||
queue.push(path.join(current, dirent.name));
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (currentStats.isSymbolicLink()) {
|
||||
stats.symlinks += 1;
|
||||
continue;
|
||||
}
|
||||
if (currentStats.isFile()) {
|
||||
stats.files += 1;
|
||||
stats.apparentBytes += currentStats.size;
|
||||
}
|
||||
}
|
||||
|
||||
return stats;
|
||||
}
|
||||
|
||||
function writeSnapshot(snapshotDir) {
|
||||
ensureDir(snapshotDir);
|
||||
const pathEntries = [...listTreeEntries("dist"), ...listTreeEntries("dist-runtime")];
|
||||
fs.writeFileSync(path.join(snapshotDir, "paths.txt"), `${pathEntries.join("\n")}\n`, "utf8");
|
||||
|
||||
const dist = snapshotTree("dist");
|
||||
const distRuntime = snapshotTree("dist-runtime");
|
||||
const snapshot = {
|
||||
generatedAt: new Date().toISOString(),
|
||||
dist,
|
||||
distRuntime,
|
||||
};
|
||||
fs.writeFileSync(
|
||||
path.join(snapshotDir, "snapshot.json"),
|
||||
`${JSON.stringify(snapshot, null, 2)}\n`,
|
||||
);
|
||||
fs.writeFileSync(
|
||||
path.join(snapshotDir, "stats.txt"),
|
||||
[
|
||||
`generated_at: ${snapshot.generatedAt}`,
|
||||
"",
|
||||
"[dist]",
|
||||
`files: ${dist.files}`,
|
||||
`directories: ${dist.directories}`,
|
||||
`symlinks: ${dist.symlinks}`,
|
||||
`entries: ${dist.entries}`,
|
||||
`apparent_bytes: ${dist.apparentBytes}`,
|
||||
`apparent_human: ${humanBytes(dist.apparentBytes)}`,
|
||||
"",
|
||||
"[dist-runtime]",
|
||||
`files: ${distRuntime.files}`,
|
||||
`directories: ${distRuntime.directories}`,
|
||||
`symlinks: ${distRuntime.symlinks}`,
|
||||
`entries: ${distRuntime.entries}`,
|
||||
`apparent_bytes: ${distRuntime.apparentBytes}`,
|
||||
`apparent_human: ${humanBytes(distRuntime.apparentBytes)}`,
|
||||
"",
|
||||
].join("\n"),
|
||||
"utf8",
|
||||
);
|
||||
return snapshot;
|
||||
}
|
||||
|
||||
function runCheckedCommand(command, args) {
|
||||
const result = spawnSync(command, args, {
|
||||
cwd: process.cwd(),
|
||||
stdio: "inherit",
|
||||
env: process.env,
|
||||
});
|
||||
if (typeof result.status === "number" && result.status === 0) {
|
||||
return;
|
||||
}
|
||||
throw new Error(`${command} ${args.join(" ")} failed with status ${result.status ?? "unknown"}`);
|
||||
}
|
||||
|
||||
function parsePsCpuTimeMs(timeText) {
|
||||
const [maybeDays, clockText] = timeText.includes("-") ? timeText.split("-", 2) : ["0", timeText];
|
||||
const days = Number(maybeDays);
|
||||
const parts = clockText.split(":");
|
||||
if (!Number.isFinite(days) || parts.length < 2 || parts.length > 3) {
|
||||
return null;
|
||||
}
|
||||
const seconds = Number(parts.at(-1));
|
||||
const minutes = Number(parts.at(-2));
|
||||
const hours = parts.length === 3 ? Number(parts[0]) : 0;
|
||||
if (![seconds, minutes, hours].every(Number.isFinite)) {
|
||||
return null;
|
||||
}
|
||||
return Math.round(((days * 24 + hours) * 60 * 60 + minutes * 60 + seconds) * 1000);
|
||||
}
|
||||
|
||||
function readProcessTreeCpuMs(rootPid) {
|
||||
if (!Number.isInteger(rootPid) || rootPid <= 0) {
|
||||
return null;
|
||||
}
|
||||
const result = spawnSync("ps", ["-eo", "pid=,ppid=,time="], {
|
||||
cwd: process.cwd(),
|
||||
encoding: "utf8",
|
||||
stdio: ["ignore", "pipe", "ignore"],
|
||||
});
|
||||
if (result.status !== 0) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const rows = [];
|
||||
for (const line of result.stdout.split("\n")) {
|
||||
const match = line.trim().match(/^(\d+)\s+(\d+)\s+(\S+)$/);
|
||||
if (!match) {
|
||||
continue;
|
||||
}
|
||||
const pid = Number(match[1]);
|
||||
const ppid = Number(match[2]);
|
||||
const cpuMs = parsePsCpuTimeMs(match[3]);
|
||||
if (!Number.isInteger(pid) || !Number.isInteger(ppid) || cpuMs == null) {
|
||||
continue;
|
||||
}
|
||||
rows.push({ pid, ppid, cpuMs });
|
||||
}
|
||||
|
||||
const childrenByParent = new Map();
|
||||
const cpuByPid = new Map();
|
||||
for (const row of rows) {
|
||||
cpuByPid.set(row.pid, row.cpuMs);
|
||||
const children = childrenByParent.get(row.ppid) ?? [];
|
||||
children.push(row.pid);
|
||||
childrenByParent.set(row.ppid, children);
|
||||
}
|
||||
if (!cpuByPid.has(rootPid)) {
|
||||
return null;
|
||||
}
|
||||
|
||||
let totalCpuMs = 0;
|
||||
const seen = new Set();
|
||||
const stack = [rootPid];
|
||||
while (stack.length > 0) {
|
||||
const pid = stack.pop();
|
||||
if (!pid || seen.has(pid)) {
|
||||
continue;
|
||||
}
|
||||
seen.add(pid);
|
||||
totalCpuMs += cpuByPid.get(pid) ?? 0;
|
||||
for (const childPid of childrenByParent.get(pid) ?? []) {
|
||||
stack.push(childPid);
|
||||
}
|
||||
}
|
||||
return totalCpuMs;
|
||||
}
|
||||
|
||||
/**
|
||||
* Reports whether gateway watch output contains a ready marker.
|
||||
*/
|
||||
export function hasGatewayReadyLog(text) {
|
||||
const normalized = text.replaceAll(ANSI_ESCAPE_PATTERN, "");
|
||||
return /\[gateway\] (?:http server listening|ready(?:\b|\s*\())/.test(normalized);
|
||||
}
|
||||
|
||||
async function waitForGatewayReady(readText, timeoutMs) {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
if (hasGatewayReadyLog(readText())) {
|
||||
return true;
|
||||
}
|
||||
await sleep(100);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
async function allocateLoopbackPort() {
|
||||
return await new Promise((resolve, reject) => {
|
||||
const server = net.createServer();
|
||||
server.once("error", reject);
|
||||
server.listen(0, "127.0.0.1", () => {
|
||||
const address = server.address();
|
||||
if (!address || typeof address === "string") {
|
||||
server.close(() => reject(new Error("Failed to allocate watch regression port")));
|
||||
return;
|
||||
}
|
||||
const { port } = address;
|
||||
server.close((closeErr) => {
|
||||
if (closeErr) {
|
||||
reject(closeErr instanceof Error ? closeErr : new Error(String(closeErr)));
|
||||
return;
|
||||
}
|
||||
resolve(port);
|
||||
});
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
function buildTimedWatchCommand(pidFilePath, timeFilePath, isolatedHomeDir, port) {
|
||||
const isolatedStateDir = path.join(isolatedHomeDir, ".openclaw");
|
||||
const isolatedConfigPath = path.join(isolatedStateDir, "openclaw.json");
|
||||
const shellSource = [
|
||||
'echo "$$" > "$OPENCLAW_WATCH_PID_FILE"',
|
||||
'mkdir -p "$OPENCLAW_STATE_DIR"',
|
||||
`printf '%s\n' '{"gateway":{"controlUi":{"enabled":false}},"plugins":{"enabled":false}}' > "$OPENCLAW_CONFIG_PATH"`,
|
||||
`exec node scripts/watch-node.mjs gateway --force --allow-unconfigured --port ${String(port)} --token watch-regression-token`,
|
||||
].join("\n");
|
||||
const env = {
|
||||
OPENCLAW_WATCH_PID_FILE: pidFilePath,
|
||||
HOME: isolatedHomeDir,
|
||||
OPENCLAW_HOME: isolatedHomeDir,
|
||||
OPENCLAW_CONFIG_PATH: isolatedConfigPath,
|
||||
OPENCLAW_STATE_DIR: isolatedStateDir,
|
||||
XDG_CONFIG_HOME: path.join(isolatedHomeDir, ".config"),
|
||||
...WATCH_GATEWAY_SKIP_ENV,
|
||||
};
|
||||
|
||||
if (process.platform === "darwin") {
|
||||
return {
|
||||
command: "/usr/bin/time",
|
||||
args: ["-lp", "-o", timeFilePath, "/bin/sh", "-lc", shellSource],
|
||||
env,
|
||||
};
|
||||
}
|
||||
|
||||
if (!fs.existsSync("/usr/bin/time")) {
|
||||
return {
|
||||
command: "/bin/sh",
|
||||
args: ["-lc", shellSource],
|
||||
env,
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
command: "/usr/bin/time",
|
||||
args: [
|
||||
"-f",
|
||||
"__TIMING__ user=%U sys=%S elapsed=%e",
|
||||
"-o",
|
||||
timeFilePath,
|
||||
"/bin/sh",
|
||||
"-lc",
|
||||
shellSource,
|
||||
],
|
||||
env,
|
||||
};
|
||||
}
|
||||
|
||||
function parseTimingFile(timeFilePath) {
|
||||
const text = fs.readFileSync(timeFilePath, "utf8");
|
||||
if (process.platform === "darwin") {
|
||||
const user = Number(text.match(/^user\s+([0-9.]+)/m)?.[1] ?? "NaN");
|
||||
const sys = Number(text.match(/^sys\s+([0-9.]+)/m)?.[1] ?? "NaN");
|
||||
const elapsed = Number(text.match(/^real\s+([0-9.]+)/m)?.[1] ?? "NaN");
|
||||
return {
|
||||
userSeconds: user,
|
||||
sysSeconds: sys,
|
||||
elapsedSeconds: elapsed,
|
||||
};
|
||||
}
|
||||
|
||||
const match = text.match(/__TIMING__ user=([0-9.]+) sys=([0-9.]+) elapsed=([0-9.]+)/);
|
||||
return {
|
||||
userSeconds: Number(match?.[1] ?? "NaN"),
|
||||
sysSeconds: Number(match?.[2] ?? "NaN"),
|
||||
elapsedSeconds: Number(match?.[3] ?? "NaN"),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs a bounded gateway watch process and captures timing/log artifacts.
|
||||
*/
|
||||
export async function runTimedWatch(options, outputDir, deps = {}) {
|
||||
const allocatePort = deps.allocateLoopbackPort ?? allocateLoopbackPort;
|
||||
const parseTiming = deps.parseTimingFile ?? parseTimingFile;
|
||||
const readCpuMs = deps.readProcessTreeCpuMs ?? readProcessTreeCpuMs;
|
||||
const sleepMs = deps.sleep ?? sleep;
|
||||
const spawnCommand = deps.spawn ?? spawn;
|
||||
const stopChild = deps.stopTimedWatchChild ?? stopTimedWatchChild;
|
||||
const waitReady = deps.waitForGatewayReady ?? waitForGatewayReady;
|
||||
const pidFilePath = path.join(outputDir, "watch.pid");
|
||||
const timeFilePath = path.join(outputDir, "watch.time.log");
|
||||
const isolatedHomeDir = fs.mkdtempSync(path.join(os.tmpdir(), "openclaw-gateway-watch-"));
|
||||
fs.writeFileSync(path.join(outputDir, "watch.home.txt"), `${isolatedHomeDir}\n`, "utf8");
|
||||
try {
|
||||
const stdoutPath = path.join(outputDir, "watch.stdout.log");
|
||||
const stderrPath = path.join(outputDir, "watch.stderr.log");
|
||||
for (const stalePath of [pidFilePath, timeFilePath, stdoutPath, stderrPath]) {
|
||||
removePathIfExists(stalePath);
|
||||
}
|
||||
const port = await allocatePort();
|
||||
fs.writeFileSync(path.join(outputDir, "watch.port.txt"), `${String(port)}\n`, "utf8");
|
||||
const { command, args, env } = buildTimedWatchCommand(
|
||||
pidFilePath,
|
||||
timeFilePath,
|
||||
isolatedHomeDir,
|
||||
port,
|
||||
);
|
||||
const child = spawnCommand(command, args, {
|
||||
cwd: process.cwd(),
|
||||
env: { ...process.env, ...env },
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
let stdoutTruncated = false;
|
||||
let stderrTruncated = false;
|
||||
let buildDetection = { buffer: "", triggered: false, reason: null };
|
||||
child.stdout?.on("data", (chunk) => {
|
||||
const next = appendBoundedWatchLog(stdout, chunk);
|
||||
stdout = next.text;
|
||||
stdoutTruncated ||= next.truncated;
|
||||
buildDetection = updateWatchBuildDetection(buildDetection, chunk);
|
||||
});
|
||||
child.stderr?.on("data", (chunk) => {
|
||||
const next = appendBoundedWatchLog(stderr, chunk);
|
||||
stderr = next.text;
|
||||
stderrTruncated ||= next.truncated;
|
||||
buildDetection = updateWatchBuildDetection(buildDetection, chunk);
|
||||
});
|
||||
|
||||
let spawnError = null;
|
||||
const spawnErrorExit = new Promise((resolve) => {
|
||||
child.once("error", (error) => {
|
||||
spawnError = error;
|
||||
resolve({ code: null, signal: null, error: error.message });
|
||||
});
|
||||
});
|
||||
const raceSpawnError = async (operation) =>
|
||||
await Promise.race([
|
||||
Promise.resolve(operation).then((value) => ({ type: "value", value })),
|
||||
spawnErrorExit.then((value) => ({ type: "spawn-error", value })),
|
||||
]);
|
||||
|
||||
let watchPid = null;
|
||||
let exit = null;
|
||||
for (let attempt = 0; attempt < 50; attempt += 1) {
|
||||
if (fs.existsSync(pidFilePath)) {
|
||||
watchPid = Number(fs.readFileSync(pidFilePath, "utf8").trim());
|
||||
break;
|
||||
}
|
||||
const waitResult = await raceSpawnError(sleepMs(100));
|
||||
if (waitResult.type === "spawn-error") {
|
||||
exit = waitResult.value;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
let readyBeforeWindow = false;
|
||||
let idleCpuStartMs = null;
|
||||
let idleCpuEndMs = null;
|
||||
if (!exit) {
|
||||
const readyResult = await raceSpawnError(
|
||||
waitReady(() => `${stdout}\n${stderr}`, options.readyTimeoutMs),
|
||||
);
|
||||
if (readyResult.type === "spawn-error") {
|
||||
exit = readyResult.value;
|
||||
} else {
|
||||
readyBeforeWindow = readyResult.value;
|
||||
}
|
||||
}
|
||||
if (!exit && readyBeforeWindow && options.readySettleMs > 0) {
|
||||
const settleResult = await raceSpawnError(sleepMs(options.readySettleMs));
|
||||
if (settleResult.type === "spawn-error") {
|
||||
exit = settleResult.value;
|
||||
}
|
||||
}
|
||||
if (!exit) {
|
||||
idleCpuStartMs = watchPid ? readCpuMs(watchPid) : null;
|
||||
const windowResult = await raceSpawnError(sleepMs(options.windowMs));
|
||||
if (windowResult.type === "spawn-error") {
|
||||
exit = windowResult.value;
|
||||
} else {
|
||||
idleCpuEndMs = watchPid ? readCpuMs(watchPid) : null;
|
||||
}
|
||||
}
|
||||
if (!exit) {
|
||||
const stopResult = await raceSpawnError(stopChild(child, watchPid, options));
|
||||
exit = stopResult.value;
|
||||
}
|
||||
|
||||
fs.writeFileSync(stdoutPath, formatCapturedWatchLog(stdout, stdoutTruncated), "utf8");
|
||||
fs.writeFileSync(stderrPath, formatCapturedWatchLog(stderr, stderrTruncated), "utf8");
|
||||
const timingFileMissing = !fs.existsSync(timeFilePath);
|
||||
const timing = timingFileMissing
|
||||
? { userSeconds: Number.NaN, sysSeconds: Number.NaN, elapsedSeconds: Number.NaN }
|
||||
: parseTiming(timeFilePath);
|
||||
|
||||
return {
|
||||
exit,
|
||||
spawnError: spawnError ? spawnError.message : null,
|
||||
timingFileMissing,
|
||||
timing,
|
||||
readyBeforeWindow,
|
||||
idleCpuMs:
|
||||
idleCpuStartMs == null || idleCpuEndMs == null
|
||||
? null
|
||||
: Math.max(0, idleCpuEndMs - idleCpuStartMs),
|
||||
stdoutPath,
|
||||
stderrPath,
|
||||
timeFilePath,
|
||||
watchTriggeredBuild: buildDetection.triggered,
|
||||
watchBuildReason: buildDetection.reason,
|
||||
};
|
||||
} finally {
|
||||
fs.rmSync(isolatedHomeDir, { force: true, recursive: true });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Stops the timed watch child process with TERM/KILL fallback.
|
||||
*/
|
||||
export async function stopTimedWatchChild(child, watchPid, options, deps = {}) {
|
||||
const killProcess = deps.killProcess ?? ((pid, signal) => process.kill(pid, signal));
|
||||
const currentExit = () =>
|
||||
child.exitCode !== null || child.signalCode !== null
|
||||
? { code: child.exitCode, signal: child.signalCode }
|
||||
: null;
|
||||
const exited = new Promise((resolve) => {
|
||||
child.once("exit", (code, signal) => resolve({ code, signal }));
|
||||
});
|
||||
const waitForExit = async (ms) =>
|
||||
currentExit() ?? (await Promise.race([exited, sleep(ms).then(() => null)]));
|
||||
const signalWatchProcess = (signal) => {
|
||||
if (!watchPid) {
|
||||
return;
|
||||
}
|
||||
try {
|
||||
killProcess(watchPid, signal);
|
||||
} catch {
|
||||
// ignore
|
||||
}
|
||||
};
|
||||
|
||||
const existingExit = currentExit();
|
||||
if (existingExit) {
|
||||
return existingExit;
|
||||
}
|
||||
|
||||
signalWatchProcess("SIGTERM");
|
||||
const gracefulExit = await waitForExit(options.sigkillGraceMs);
|
||||
if (gracefulExit) {
|
||||
return gracefulExit;
|
||||
}
|
||||
|
||||
signalWatchProcess("SIGKILL");
|
||||
const killedExit = await waitForExit(options.sigkillExitGraceMs ?? DEFAULTS.sigkillExitGraceMs);
|
||||
if (killedExit) {
|
||||
return killedExit;
|
||||
}
|
||||
|
||||
releaseUnsettledWatchChild(child);
|
||||
return { code: null, signal: "SIGKILL" };
|
||||
}
|
||||
|
||||
function releaseUnsettledWatchChild(child) {
|
||||
child.stdin?.destroy?.();
|
||||
child.stdout?.destroy?.();
|
||||
child.stderr?.destroy?.();
|
||||
child.unref?.();
|
||||
}
|
||||
|
||||
function parsePathFile(filePath) {
|
||||
return fs
|
||||
.readFileSync(filePath, "utf8")
|
||||
.split("\n")
|
||||
.map((line) => line.trimEnd())
|
||||
.filter(Boolean);
|
||||
}
|
||||
|
||||
function writeDiffArtifacts(outputDir, preDir, postDir) {
|
||||
const diffDir = path.join(outputDir, "diff");
|
||||
ensureDir(diffDir);
|
||||
const prePaths = parsePathFile(path.join(preDir, "paths.txt"));
|
||||
const postPaths = parsePathFile(path.join(postDir, "paths.txt"));
|
||||
const preSet = new Set(prePaths);
|
||||
const postSet = new Set(postPaths);
|
||||
const added = postPaths.filter((entry) => !preSet.has(entry));
|
||||
const removed = prePaths.filter((entry) => !postSet.has(entry));
|
||||
|
||||
fs.writeFileSync(path.join(diffDir, "added-paths.txt"), `${added.join("\n")}\n`, "utf8");
|
||||
fs.writeFileSync(path.join(diffDir, "removed-paths.txt"), `${removed.join("\n")}\n`, "utf8");
|
||||
return { added, removed };
|
||||
}
|
||||
|
||||
function fail(message) {
|
||||
console.error(`FAIL: ${message}`);
|
||||
}
|
||||
|
||||
function warn(message) {
|
||||
console.error(`WARN: ${message}`);
|
||||
}
|
||||
|
||||
function detectWatchBuildReason(stdout, stderr) {
|
||||
const combined = `${stdout}\n${stderr}`;
|
||||
const match = combined.match(/Building TypeScript \(dist is stale: ([a-z_]+)/);
|
||||
return match?.[1] ?? null;
|
||||
}
|
||||
|
||||
function buildRunNodeDeps(env) {
|
||||
const cwd = process.cwd();
|
||||
return {
|
||||
cwd,
|
||||
env,
|
||||
fs,
|
||||
spawnSync,
|
||||
distRoot: path.join(cwd, "dist"),
|
||||
distEntry: path.join(cwd, "dist", "/entry.js"),
|
||||
buildStampPath: path.join(cwd, "dist", BUILD_STAMP_FILE),
|
||||
sourceRoots: ["src", "extensions"].map((sourceRoot) => ({
|
||||
name: sourceRoot,
|
||||
path: path.join(cwd, sourceRoot),
|
||||
})),
|
||||
configFiles: ["tsconfig.json", "package.json", "tsdown.config.ts"].map((filePath) =>
|
||||
path.join(cwd, filePath),
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Reports whether restored CI artifacts need fresh build stamps.
|
||||
*/
|
||||
export function shouldRefreshBuildStampForRestoredArtifacts(params) {
|
||||
return (
|
||||
params.skipBuild === true &&
|
||||
params.buildRequirement?.shouldBuild === true &&
|
||||
params.buildRequirement.reason === "config_newer"
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Writes build and runtime-postbuild stamps for the current artifact set.
|
||||
*/
|
||||
export function writeBuildAndRuntimePostBuildStamps(params = {}) {
|
||||
const cwd = params.cwd ?? process.cwd();
|
||||
writeBuildStamp({ cwd });
|
||||
writeRuntimePostBuildStamp({ cwd });
|
||||
}
|
||||
|
||||
/**
|
||||
* Collects pass/fail findings for the bounded gateway watch regression run.
|
||||
*/
|
||||
export function collectGatewayWatchFindings(params) {
|
||||
const {
|
||||
cpuMs,
|
||||
distRuntimeByteGrowth,
|
||||
distRuntimeFileGrowth,
|
||||
options,
|
||||
watchBuildReason,
|
||||
watchResult,
|
||||
watchTriggeredBuild,
|
||||
} = params;
|
||||
const failures = [];
|
||||
const warnings = [];
|
||||
if (watchResult.spawnError) {
|
||||
failures.push(`gateway:watch failed to start: ${watchResult.spawnError}`);
|
||||
}
|
||||
if (!watchResult.readyBeforeWindow) {
|
||||
failures.push("gateway:watch did not report ready before the idle CPU window");
|
||||
}
|
||||
if (watchResult.timingFileMissing && !Number.isFinite(watchResult.idleCpuMs)) {
|
||||
failures.push(
|
||||
"failed to collect CPU timing from the bounded gateway:watch run; timing artifact is missing",
|
||||
);
|
||||
} else if (watchResult.timingFileMissing) {
|
||||
warnings.push(
|
||||
"bounded gateway:watch timing artifact is missing; using process-tree idle CPU sample",
|
||||
);
|
||||
}
|
||||
if (watchTriggeredBuild && watchBuildReason === "dirty_watched_tree") {
|
||||
failures.push(
|
||||
"gateway:watch invalid local run: dirty watched source tree forced a rebuild during the watch window",
|
||||
);
|
||||
}
|
||||
if (distRuntimeFileGrowth > options.distRuntimeFileGrowthMax) {
|
||||
failures.push(
|
||||
`dist-runtime file growth ${distRuntimeFileGrowth} exceeded max ${options.distRuntimeFileGrowthMax}`,
|
||||
);
|
||||
}
|
||||
if (distRuntimeByteGrowth > options.distRuntimeByteGrowthMax) {
|
||||
failures.push(
|
||||
`dist-runtime apparent byte growth ${distRuntimeByteGrowth} exceeded max ${options.distRuntimeByteGrowthMax}`,
|
||||
);
|
||||
}
|
||||
if (!Number.isFinite(cpuMs)) {
|
||||
failures.push("failed to parse CPU timing from the bounded gateway:watch run");
|
||||
} else if (cpuMs > options.cpuFailMs) {
|
||||
failures.push(
|
||||
`LOUD ALARM: gateway:watch used ${cpuMs}ms CPU in ${options.windowMs}ms window, above loud-alarm threshold ${options.cpuFailMs}ms`,
|
||||
);
|
||||
} else if (cpuMs > options.cpuWarnMs) {
|
||||
warnings.push(
|
||||
`gateway:watch used ${cpuMs}ms CPU in ${options.windowMs}ms window, above target ${options.cpuWarnMs}ms`,
|
||||
);
|
||||
}
|
||||
return { failures, warnings };
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const options = parseArgs(process.argv.slice(2));
|
||||
ensureDir(options.outputDir);
|
||||
if (!options.skipBuild) {
|
||||
runCheckedCommand("node", ["scripts/build-all.mjs", "gatewayWatch"]);
|
||||
// The watch harness must start from a completed dist/runtime baseline.
|
||||
// Refresh both stamps after the gateway build finishes so run-node does not
|
||||
// leave stale local artifact metadata after the bounded watch window.
|
||||
writeBuildAndRuntimePostBuildStamps();
|
||||
} else {
|
||||
// Restored CI artifacts can be older than the fresh checkout mtimes.
|
||||
// Refresh the local artifact stamps so run-node trusts the already-built dist.
|
||||
writeBuildAndRuntimePostBuildStamps();
|
||||
}
|
||||
|
||||
let preflightBuildRequirement = resolveBuildRequirement(buildRunNodeDeps(process.env));
|
||||
if (
|
||||
shouldRefreshBuildStampForRestoredArtifacts({
|
||||
skipBuild: options.skipBuild,
|
||||
buildRequirement: preflightBuildRequirement,
|
||||
})
|
||||
) {
|
||||
// CI's skip-build path restores a built dist artifact after checkout.
|
||||
// Refresh the stamps so checkout mtimes for package/config files do not
|
||||
// force a duplicate build during the bounded gateway:watch window.
|
||||
writeBuildAndRuntimePostBuildStamps();
|
||||
preflightBuildRequirement = resolveBuildRequirement(buildRunNodeDeps(process.env));
|
||||
}
|
||||
if (
|
||||
preflightBuildRequirement.shouldBuild &&
|
||||
preflightBuildRequirement.reason === "dirty_watched_tree"
|
||||
) {
|
||||
const summary = {
|
||||
windowMs: options.windowMs,
|
||||
invalidated: true,
|
||||
invalidationReason: preflightBuildRequirement.reason,
|
||||
invalidationMessage:
|
||||
"gateway-watch-regression cannot run on a dirty watched tree because run-node will intentionally rebuild during the watch window.",
|
||||
};
|
||||
fs.writeFileSync(
|
||||
path.join(options.outputDir, "summary.json"),
|
||||
`${JSON.stringify(summary, null, 2)}\n`,
|
||||
);
|
||||
console.log(JSON.stringify(summary, null, 2));
|
||||
fail(
|
||||
"gateway-watch-regression invalid local run: dirty watched source tree would force a rebuild inside the watch window",
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const preDir = path.join(options.outputDir, "pre");
|
||||
const pre = writeSnapshot(preDir);
|
||||
|
||||
const watchDir = path.join(options.outputDir, "watch");
|
||||
ensureDir(watchDir);
|
||||
const watchResult = await runTimedWatch(options, watchDir);
|
||||
|
||||
const postDir = path.join(options.outputDir, "post");
|
||||
const post = writeSnapshot(postDir);
|
||||
const diff = writeDiffArtifacts(options.outputDir, preDir, postDir);
|
||||
|
||||
const distRuntimeAddedPaths = diff.added.filter((entry) =>
|
||||
entry.startsWith("dist-runtime/"),
|
||||
).length;
|
||||
const distRuntimeFileGrowth = distRuntimeAddedPaths;
|
||||
const distRuntimeByteGrowth =
|
||||
distRuntimeAddedPaths === 0
|
||||
? 0
|
||||
: post.distRuntime.apparentBytes - pre.distRuntime.apparentBytes;
|
||||
const totalCpuMs = Math.round(
|
||||
(watchResult.timing.userSeconds + watchResult.timing.sysSeconds) * 1000,
|
||||
);
|
||||
const cpuMs = watchResult.idleCpuMs ?? totalCpuMs;
|
||||
const watchTriggeredBuild = watchResult.watchTriggeredBuild;
|
||||
const watchBuildReason = watchResult.watchBuildReason;
|
||||
|
||||
const summary = {
|
||||
windowMs: options.windowMs,
|
||||
watchTriggeredBuild,
|
||||
watchBuildReason,
|
||||
cpuMs,
|
||||
totalCpuMs,
|
||||
readyBeforeWindow: watchResult.readyBeforeWindow,
|
||||
cpuWarnMs: options.cpuWarnMs,
|
||||
cpuFailMs: options.cpuFailMs,
|
||||
distRuntimeFileGrowth,
|
||||
distRuntimeFileGrowthMax: options.distRuntimeFileGrowthMax,
|
||||
distRuntimeByteGrowth,
|
||||
distRuntimeByteGrowthMax: options.distRuntimeByteGrowthMax,
|
||||
distRuntimeAddedPaths,
|
||||
addedPaths: diff.added.length,
|
||||
removedPaths: diff.removed.length,
|
||||
watchExit: watchResult.exit,
|
||||
spawnError: watchResult.spawnError,
|
||||
timingFileMissing: watchResult.timingFileMissing,
|
||||
timing: watchResult.timing,
|
||||
};
|
||||
fs.writeFileSync(
|
||||
path.join(options.outputDir, "summary.json"),
|
||||
`${JSON.stringify(summary, null, 2)}\n`,
|
||||
);
|
||||
|
||||
console.log(JSON.stringify(summary, null, 2));
|
||||
|
||||
const { failures, warnings } = collectGatewayWatchFindings({
|
||||
cpuMs,
|
||||
distRuntimeByteGrowth,
|
||||
distRuntimeFileGrowth,
|
||||
options,
|
||||
watchBuildReason,
|
||||
watchResult,
|
||||
watchTriggeredBuild,
|
||||
});
|
||||
|
||||
for (const message of warnings) {
|
||||
warn(message);
|
||||
}
|
||||
|
||||
if (failures.length > 0) {
|
||||
for (const message of failures) {
|
||||
fail(message);
|
||||
}
|
||||
if (!failures.every((message) => message.includes("dirty watched source tree"))) {
|
||||
fail(
|
||||
"Possible duplicate dist-runtime graph regression: this can reintroduce split runtime personalities where plugins and core observe different global state, including Telegram missing /voice, /phone, or /pair.",
|
||||
);
|
||||
}
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
if (import.meta.url === pathToFileURL(process.argv[1] ?? "").href) {
|
||||
await main();
|
||||
}
|
||||
167
scripts/check-import-cycles.ts
Normal file
167
scripts/check-import-cycles.ts
Normal file
@@ -0,0 +1,167 @@
|
||||
#!/usr/bin/env node
|
||||
// Check Import Cycles script supports OpenClaw repository automation.
|
||||
import { readFileSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import ts from "typescript";
|
||||
import {
|
||||
collectSourceFiles,
|
||||
collectStronglyConnectedComponents,
|
||||
formatCycle,
|
||||
} from "./lib/import-cycle-graph.ts";
|
||||
|
||||
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const scanRoots = ["src", "extensions", "scripts"] as const;
|
||||
const sourceExtensions = [".ts", ".tsx", ".mts", ".cts", ".js", ".mjs", ".cjs"] as const;
|
||||
const testSourcePattern = /(?:\.test|\.e2e\.test)\.[cm]?[tj]sx?$/;
|
||||
const generatedSourcePattern = /\.(?:generated|bundle)\.[tj]s$/;
|
||||
const declarationSourcePattern = /\.d\.[cm]?ts$/;
|
||||
const ignoredPathPartPattern =
|
||||
/(^|\/)(node_modules|dist|build|coverage|\.artifacts|\.git|assets)(\/|$)/;
|
||||
|
||||
function shouldSkipRepoPath(repoPath: string): boolean {
|
||||
return (
|
||||
ignoredPathPartPattern.test(repoPath) ||
|
||||
testSourcePattern.test(repoPath) ||
|
||||
generatedSourcePattern.test(repoPath) ||
|
||||
declarationSourcePattern.test(repoPath)
|
||||
);
|
||||
}
|
||||
|
||||
function createSourceResolver(files: readonly string[]) {
|
||||
const fileSet = new Set(files);
|
||||
const pathMap = new Map<string, string>();
|
||||
for (const file of files) {
|
||||
const parsed = path.posix.parse(file);
|
||||
const extensionless = path.posix.join(parsed.dir, parsed.name);
|
||||
pathMap.set(extensionless, file);
|
||||
if (file.endsWith(".ts")) {
|
||||
pathMap.set(`${extensionless}.js`, file);
|
||||
} else if (file.endsWith(".tsx")) {
|
||||
pathMap.set(`${extensionless}.jsx`, file);
|
||||
} else if (file.endsWith(".mts")) {
|
||||
pathMap.set(`${extensionless}.mjs`, file);
|
||||
} else if (file.endsWith(".cts")) {
|
||||
pathMap.set(`${extensionless}.cjs`, file);
|
||||
}
|
||||
}
|
||||
return (importer: string, specifier: string): string | null => {
|
||||
if (!specifier.startsWith(".")) {
|
||||
return null;
|
||||
}
|
||||
const base = path.posix.normalize(path.posix.join(path.posix.dirname(importer), specifier));
|
||||
const candidates = [
|
||||
base,
|
||||
...sourceExtensions.map((extension) => `${base}${extension}`),
|
||||
`${base}/index.ts`,
|
||||
`${base}/index.tsx`,
|
||||
`${base}/index.js`,
|
||||
`${base}/index.mjs`,
|
||||
];
|
||||
for (const candidate of candidates) {
|
||||
if (fileSet.has(candidate)) {
|
||||
return candidate;
|
||||
}
|
||||
const mapped = pathMap.get(candidate);
|
||||
if (mapped) {
|
||||
return mapped;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
};
|
||||
}
|
||||
|
||||
function importDeclarationHasRuntimeEdge(node: ts.ImportDeclaration): boolean {
|
||||
if (!node.importClause) {
|
||||
return true;
|
||||
}
|
||||
if (node.importClause.isTypeOnly) {
|
||||
return false;
|
||||
}
|
||||
const bindings = node.importClause.namedBindings;
|
||||
if (node.importClause.name || !bindings || ts.isNamespaceImport(bindings)) {
|
||||
return true;
|
||||
}
|
||||
return bindings.elements.some((element) => !element.isTypeOnly);
|
||||
}
|
||||
|
||||
function exportDeclarationHasRuntimeEdge(node: ts.ExportDeclaration): boolean {
|
||||
if (!node.moduleSpecifier || node.isTypeOnly) {
|
||||
return false;
|
||||
}
|
||||
const clause = node.exportClause;
|
||||
if (!clause || ts.isNamespaceExport(clause)) {
|
||||
return true;
|
||||
}
|
||||
return clause.elements.some((element) => !element.isTypeOnly);
|
||||
}
|
||||
|
||||
function collectRuntimeStaticImports(
|
||||
file: string,
|
||||
resolveSource: ReturnType<typeof createSourceResolver>,
|
||||
) {
|
||||
const sourceFile = ts.createSourceFile(
|
||||
file,
|
||||
readFileSync(path.join(repoRoot, file), "utf8"),
|
||||
ts.ScriptTarget.Latest,
|
||||
true,
|
||||
);
|
||||
const imports: string[] = [];
|
||||
const visit = (node: ts.Node) => {
|
||||
let specifier: string | undefined;
|
||||
let include = false;
|
||||
if (ts.isImportDeclaration(node) && ts.isStringLiteral(node.moduleSpecifier)) {
|
||||
specifier = node.moduleSpecifier.text;
|
||||
include = importDeclarationHasRuntimeEdge(node);
|
||||
} else if (
|
||||
ts.isExportDeclaration(node) &&
|
||||
node.moduleSpecifier &&
|
||||
ts.isStringLiteral(node.moduleSpecifier)
|
||||
) {
|
||||
specifier = node.moduleSpecifier.text;
|
||||
include = exportDeclarationHasRuntimeEdge(node);
|
||||
}
|
||||
if (include && specifier) {
|
||||
const resolved = resolveSource(file, specifier);
|
||||
if (resolved) {
|
||||
imports.push(resolved);
|
||||
}
|
||||
}
|
||||
ts.forEachChild(node, visit);
|
||||
};
|
||||
visit(sourceFile);
|
||||
return imports.toSorted((left, right) => left.localeCompare(right));
|
||||
}
|
||||
|
||||
function main(): number {
|
||||
const files = scanRoots.flatMap((root) =>
|
||||
collectSourceFiles(path.join(repoRoot, root), {
|
||||
repoRoot,
|
||||
sourceExtensions,
|
||||
shouldSkipRepoPath,
|
||||
}),
|
||||
);
|
||||
const resolveSource = createSourceResolver(files);
|
||||
const graph = new Map(
|
||||
files.map((file): [string, string[]] => [
|
||||
file,
|
||||
collectRuntimeStaticImports(file, resolveSource),
|
||||
]),
|
||||
);
|
||||
const components = collectStronglyConnectedComponents(graph);
|
||||
|
||||
console.log(`Import cycle check: ${components.length} runtime value cycle(s).`);
|
||||
if (components.length === 0) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
console.error("\nRuntime value import cycles:");
|
||||
for (const component of components) {
|
||||
console.error(`\n# component size ${component.length}`);
|
||||
console.error(formatCycle(component, graph));
|
||||
}
|
||||
console.error("\nBreak the cycle or convert type-only edges to `import type`.");
|
||||
return 1;
|
||||
}
|
||||
|
||||
process.exitCode = main();
|
||||
49
scripts/check-ingress-agent-owner-context.mjs
Normal file
49
scripts/check-ingress-agent-owner-context.mjs
Normal file
@@ -0,0 +1,49 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Ensures ingress agent command callsites pass explicit owner context.
|
||||
import path from "node:path";
|
||||
import ts from "typescript";
|
||||
import { bundledPluginFile } from "./lib/bundled-plugin-paths.mjs";
|
||||
import { runCallsiteGuard } from "./lib/callsite-guard.mjs";
|
||||
import {
|
||||
collectCallExpressionLines,
|
||||
runAsScript,
|
||||
unwrapExpression,
|
||||
} from "./lib/ts-guard-utils.mjs";
|
||||
|
||||
const sourceRoots = ["src/gateway", bundledPluginFile("discord", "src/voice")];
|
||||
const enforcedFiles = new Set([
|
||||
bundledPluginFile("discord", "src/voice/manager.ts"),
|
||||
"src/gateway/openai-http.ts",
|
||||
"src/gateway/openresponses-http.ts",
|
||||
"src/gateway/server-methods/agent.ts",
|
||||
"src/gateway/server-node-events.ts",
|
||||
]);
|
||||
|
||||
/**
|
||||
* Finds legacy `agentCommand(...)` call lines in ingress-owned source.
|
||||
*/
|
||||
export function findLegacyAgentCommandCallLines(content, fileName = "source.ts") {
|
||||
const sourceFile = ts.createSourceFile(fileName, content, ts.ScriptTarget.Latest, true);
|
||||
return collectCallExpressionLines(ts, sourceFile, (node) => {
|
||||
const callee = unwrapExpression(node.expression);
|
||||
return ts.isIdentifier(callee) && callee.text === "agentCommand" ? callee : null;
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the ingress owner-context guard.
|
||||
*/
|
||||
export async function main() {
|
||||
await runCallsiteGuard({
|
||||
importMetaUrl: import.meta.url,
|
||||
sourceRoots,
|
||||
findCallLines: findLegacyAgentCommandCallLines,
|
||||
skipRelativePath: (relPath) => !enforcedFiles.has(relPath.replaceAll(path.sep, "/")),
|
||||
header: "Found ingress callsites using local agentCommand() (must be explicit owner-aware):",
|
||||
footer:
|
||||
"Use agentCommandFromIngress(...) and pass senderIsOwner explicitly at ingress boundaries.",
|
||||
});
|
||||
}
|
||||
|
||||
runAsScript(import.meta.url, main);
|
||||
384
scripts/check-kysely-guardrails.mjs
Normal file
384
scripts/check-kysely-guardrails.mjs
Normal file
@@ -0,0 +1,384 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Enforces Kysely and SQLite guardrails in infrastructure code.
|
||||
import { promises as fs } from "node:fs";
|
||||
import { createRequire } from "node:module";
|
||||
import path from "node:path";
|
||||
import {
|
||||
collectTypeScriptFilesFromRoots,
|
||||
getPropertyNameText,
|
||||
resolveRepoRoot,
|
||||
runAsScript,
|
||||
toLine,
|
||||
unwrapExpression,
|
||||
} from "./lib/ts-guard-utils.mjs";
|
||||
|
||||
const require = createRequire(import.meta.url);
|
||||
const ts = require("typescript");
|
||||
|
||||
const repoRoot = resolveRepoRoot(import.meta.url);
|
||||
const sourceRoots = [path.join(repoRoot, "src")];
|
||||
|
||||
const kyselyRawAllowPaths = new Set([
|
||||
"src/infra/kysely-node-sqlite.test.ts",
|
||||
"src/infra/kysely-sync.ts",
|
||||
]);
|
||||
|
||||
const compiledRawAllowPaths = new Set([
|
||||
"src/infra/kysely-node-sqlite.ts",
|
||||
"src/infra/kysely-node-sqlite.test.ts",
|
||||
]);
|
||||
|
||||
const rawSqliteAllowPathGroups = {
|
||||
"native Kysely adapter and sync execution": [
|
||||
"src/infra/kysely-node-sqlite.ts",
|
||||
"src/infra/kysely-sync.ts",
|
||||
],
|
||||
"SQLite database lifecycle, schema, transactions, and pragmas": [
|
||||
"src/infra/node-sqlite.ts",
|
||||
"src/infra/sqlite-integrity.ts",
|
||||
"src/infra/sqlite-pragma.test-support.ts",
|
||||
"src/infra/sqlite-transaction.ts",
|
||||
"src/infra/sqlite-user-version.ts",
|
||||
"src/infra/sqlite-wal.ts",
|
||||
"src/state/openclaw-agent-db.ts",
|
||||
"src/state/openclaw-state-db.ts",
|
||||
"src/state/sqlite-schema-shape.test-support.ts",
|
||||
],
|
||||
"backup snapshot maintenance": ["src/commands/backup-verify.ts", "src/infra/backup-create.ts"],
|
||||
"agent auth profile read-only bootstrap": ["src/agents/auth-profiles/sqlite.ts"],
|
||||
"read-only SQLite status probes": ["src/commands/status.scan.shared.ts"],
|
||||
"doctor legacy state migration": [
|
||||
"src/commands/doctor/cron/migration-ledger.ts",
|
||||
"src/infra/state-migrations.ts",
|
||||
"src/infra/state-migrations.debug-proxy.ts",
|
||||
],
|
||||
"shared database stores with direct DatabaseSync access": ["src/proxy-capture/store.sqlite.ts"],
|
||||
"Kysely-backed stores that own a DatabaseSync boundary": [
|
||||
"src/acp/event-ledger.ts",
|
||||
"src/agents/subagent-registry.store.ts",
|
||||
"src/cron/run-log.ts",
|
||||
"src/cron/run-log/sqlite-store.ts",
|
||||
"src/cron/store.ts",
|
||||
"src/infra/outbound/current-conversation-bindings.ts",
|
||||
"src/media/store.ts",
|
||||
"src/plugin-sdk/memory-core-host-engine-storage.ts",
|
||||
"src/plugins/installed-plugin-index-record-reader.ts",
|
||||
"src/plugins/installed-plugin-index-store.ts",
|
||||
"src/plugin-state/plugin-state-store.sqlite.ts",
|
||||
"src/tasks/task-flow-registry.store.sqlite.ts",
|
||||
"src/tasks/task-registry.store.sqlite.ts",
|
||||
"src/tui/tui-last-session.ts",
|
||||
],
|
||||
};
|
||||
|
||||
const rawSqliteAllowPathReasons = new Map();
|
||||
for (const [reason, paths] of Object.entries(rawSqliteAllowPathGroups)) {
|
||||
for (const allowedPath of paths) {
|
||||
if (rawSqliteAllowPathReasons.has(allowedPath)) {
|
||||
throw new Error(`Duplicate raw SQLite allowlist path: ${allowedPath}`);
|
||||
}
|
||||
rawSqliteAllowPathReasons.set(allowedPath, reason);
|
||||
}
|
||||
}
|
||||
|
||||
function lineText(sourceFile, node) {
|
||||
const line = toLine(sourceFile, node);
|
||||
return sourceFile.text.split("\n")[line - 1] ?? "";
|
||||
}
|
||||
|
||||
function hasAllowComment(sourceFile, node, token) {
|
||||
const line = lineText(sourceFile, node);
|
||||
if (line.includes(token)) {
|
||||
return true;
|
||||
}
|
||||
const leading = ts.getLeadingCommentRanges(sourceFile.text, node.pos) ?? [];
|
||||
return leading.some((range) => sourceFile.text.slice(range.pos, range.end).includes(token));
|
||||
}
|
||||
|
||||
function importSource(node) {
|
||||
const moduleSpecifier = node.moduleSpecifier;
|
||||
return ts.isStringLiteral(moduleSpecifier) ? moduleSpecifier.text : "";
|
||||
}
|
||||
|
||||
function collectImports(sourceFile) {
|
||||
const kyselySqlNames = new Set();
|
||||
const compiledQueryNames = new Set();
|
||||
const syncHelperNames = new Set();
|
||||
let hasKyselyContext = false;
|
||||
let hasSqliteContext = false;
|
||||
|
||||
for (const statement of sourceFile.statements) {
|
||||
if (!ts.isImportDeclaration(statement)) {
|
||||
continue;
|
||||
}
|
||||
const source = importSource(statement);
|
||||
const clause = statement.importClause;
|
||||
const namedBindings = clause?.namedBindings;
|
||||
|
||||
if (source === "kysely") {
|
||||
hasKyselyContext = true;
|
||||
if (namedBindings && ts.isNamedImports(namedBindings)) {
|
||||
for (const element of namedBindings.elements) {
|
||||
const importedName = element.propertyName?.text ?? element.name.text;
|
||||
if (importedName === "sql") {
|
||||
kyselySqlNames.add(element.name.text);
|
||||
}
|
||||
if (importedName === "CompiledQuery") {
|
||||
compiledQueryNames.add(element.name.text);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (source.endsWith("kysely-sync.js") || source.endsWith("kysely-node-sqlite.js")) {
|
||||
hasKyselyContext = true;
|
||||
if (namedBindings && ts.isNamedImports(namedBindings)) {
|
||||
for (const element of namedBindings.elements) {
|
||||
const importedName = element.propertyName?.text ?? element.name.text;
|
||||
if (
|
||||
importedName === "executeSqliteQuerySync" ||
|
||||
importedName === "executeSqliteQueryTakeFirstSync"
|
||||
) {
|
||||
syncHelperNames.add(element.name.text);
|
||||
}
|
||||
if (importedName === "getNodeSqliteKysely") {
|
||||
hasKyselyContext = true;
|
||||
hasSqliteContext = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (
|
||||
source === "node:sqlite" ||
|
||||
source.endsWith("node-sqlite.js") ||
|
||||
source.endsWith("sqlite-transaction.js") ||
|
||||
source.endsWith("sqlite-wal.js") ||
|
||||
source.endsWith("openclaw-state-db.js")
|
||||
) {
|
||||
hasSqliteContext = true;
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
compiledQueryNames,
|
||||
hasKyselyContext,
|
||||
hasSqliteContext,
|
||||
kyselySqlNames,
|
||||
syncHelperNames,
|
||||
};
|
||||
}
|
||||
|
||||
function addViolation(violations, sourceFile, node, message) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node),
|
||||
message,
|
||||
});
|
||||
}
|
||||
|
||||
function isIdentifierNamed(node, names) {
|
||||
const unwrapped = unwrapExpression(node);
|
||||
return ts.isIdentifier(unwrapped) && names.has(unwrapped.text);
|
||||
}
|
||||
|
||||
function isTestPath(relativePath) {
|
||||
return (
|
||||
/\.(?:test|spec|e2e)\.ts$/u.test(relativePath) ||
|
||||
relativePath.includes(".test-helpers.") ||
|
||||
relativePath.includes(".test-support.")
|
||||
);
|
||||
}
|
||||
|
||||
function isSqliteStorePath(relativePath) {
|
||||
return relativePath.endsWith(".sqlite.ts") || relativePath.includes(".store.sqlite.ts");
|
||||
}
|
||||
|
||||
function isLikelySqliteReceiver(expression) {
|
||||
const unwrapped = unwrapExpression(expression);
|
||||
if (ts.isIdentifier(unwrapped)) {
|
||||
return /^(?:db|database|legacyDb|stateDb|agentDb)$/u.test(unwrapped.text);
|
||||
}
|
||||
return ts.isPropertyAccessExpression(unwrapped) && getPropertyNameText(unwrapped.name) === "db";
|
||||
}
|
||||
|
||||
function isPersistedRowExpression(expression) {
|
||||
const unwrapped = unwrapExpression(expression);
|
||||
if (ts.isPropertyAccessExpression(unwrapped)) {
|
||||
const owner = unwrapExpression(unwrapped.expression);
|
||||
return ts.isIdentifier(owner) && /^(?:row|record|entry)$/u.test(owner.text);
|
||||
}
|
||||
if (ts.isElementAccessExpression(unwrapped)) {
|
||||
const owner = unwrapExpression(unwrapped.expression);
|
||||
return ts.isIdentifier(owner) && /^(?:row|record|entry)$/u.test(owner.text);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function isPersistedStringCastType(typeText) {
|
||||
return [
|
||||
/\bTaskRecord\["(?:runtime|scopeKind|status|deliveryStatus|notifyPolicy|terminalOutcome)"\]/u,
|
||||
/\bTaskFlowRecord\["(?:status|notifyPolicy)"\]/u,
|
||||
/\bTaskFlowSyncMode\b/u,
|
||||
/\bVirtualAgentFsEntryKind\b/u,
|
||||
/\b[A-Z][A-Za-z0-9]*(?:Status|Kind|Mode|Policy|Runtime|Outcome)\b/u,
|
||||
].some((pattern) => pattern.test(typeText));
|
||||
}
|
||||
|
||||
/**
|
||||
* Collects Kysely/raw SQLite violations from one source file.
|
||||
*/
|
||||
export function collectKyselyGuardrailViolations(content, relativePath) {
|
||||
const sourceFile = ts.createSourceFile(relativePath, content, ts.ScriptTarget.Latest, true);
|
||||
const imports = collectImports(sourceFile);
|
||||
const violations = [];
|
||||
|
||||
function visit(node) {
|
||||
if (
|
||||
isSqliteStorePath(relativePath) &&
|
||||
(ts.isAsExpression(node) || ts.isTypeAssertionExpression(node)) &&
|
||||
isPersistedStringCastType(node.type.getText(sourceFile)) &&
|
||||
isPersistedRowExpression(node.expression) &&
|
||||
!hasAllowComment(sourceFile, node, "sqlite-allow-persisted-cast")
|
||||
) {
|
||||
addViolation(
|
||||
violations,
|
||||
sourceFile,
|
||||
node,
|
||||
"persisted SQLite enum-like values must be parsed through closed validators, not cast",
|
||||
);
|
||||
}
|
||||
|
||||
if (
|
||||
ts.isCallExpression(node) &&
|
||||
ts.isIdentifier(node.expression) &&
|
||||
imports.syncHelperNames.has(node.expression.text) &&
|
||||
node.typeArguments?.length &&
|
||||
!hasAllowComment(sourceFile, node, "kysely-allow-raw")
|
||||
) {
|
||||
addViolation(
|
||||
violations,
|
||||
sourceFile,
|
||||
node,
|
||||
"sync helper row generic at call site; let Kysely infer builder result rows",
|
||||
);
|
||||
}
|
||||
|
||||
if (
|
||||
ts.isTaggedTemplateExpression(node) &&
|
||||
node.typeArguments?.length &&
|
||||
isIdentifierNamed(node.tag, imports.kyselySqlNames) &&
|
||||
!kyselyRawAllowPaths.has(relativePath) &&
|
||||
!hasAllowComment(sourceFile, node, "kysely-allow-raw")
|
||||
) {
|
||||
addViolation(
|
||||
violations,
|
||||
sourceFile,
|
||||
node,
|
||||
"typed raw sql snippet needs a small helper or allowlisted boundary",
|
||||
);
|
||||
}
|
||||
|
||||
if (
|
||||
ts.isCallExpression(node) &&
|
||||
ts.isPropertyAccessExpression(node.expression) &&
|
||||
isIdentifierNamed(node.expression.expression, imports.kyselySqlNames) &&
|
||||
["ref", "table", "id", "raw"].includes(getPropertyNameText(node.expression.name) ?? "") &&
|
||||
!hasAllowComment(sourceFile, node, "kysely-allow-raw")
|
||||
) {
|
||||
addViolation(
|
||||
violations,
|
||||
sourceFile,
|
||||
node,
|
||||
"raw Kysely identifier helper requires a closed-set validator and local allow comment",
|
||||
);
|
||||
}
|
||||
|
||||
if (
|
||||
imports.hasKyselyContext &&
|
||||
ts.isPropertyAccessExpression(node) &&
|
||||
getPropertyNameText(node.name) === "dynamic" &&
|
||||
!hasAllowComment(sourceFile, node, "kysely-allow-raw")
|
||||
) {
|
||||
addViolation(
|
||||
violations,
|
||||
sourceFile,
|
||||
node,
|
||||
"Kysely dynamic refs bypass literal reference checking; use only behind closed unions",
|
||||
);
|
||||
}
|
||||
|
||||
if (
|
||||
ts.isCallExpression(node) &&
|
||||
ts.isPropertyAccessExpression(node.expression) &&
|
||||
isIdentifierNamed(node.expression.expression, imports.compiledQueryNames) &&
|
||||
getPropertyNameText(node.expression.name) === "raw" &&
|
||||
!compiledRawAllowPaths.has(relativePath) &&
|
||||
!hasAllowComment(sourceFile, node, "kysely-allow-raw")
|
||||
) {
|
||||
addViolation(
|
||||
violations,
|
||||
sourceFile,
|
||||
node,
|
||||
"CompiledQuery.raw is only allowed in the native SQLite dialect/test boundary",
|
||||
);
|
||||
}
|
||||
|
||||
if (
|
||||
imports.hasSqliteContext &&
|
||||
!isTestPath(relativePath) &&
|
||||
ts.isCallExpression(node) &&
|
||||
ts.isPropertyAccessExpression(node.expression) &&
|
||||
["prepare", "exec"].includes(getPropertyNameText(node.expression.name) ?? "") &&
|
||||
isLikelySqliteReceiver(node.expression.expression) &&
|
||||
!rawSqliteAllowPathReasons.has(relativePath) &&
|
||||
!hasAllowComment(sourceFile, node, "sqlite-allow-raw")
|
||||
) {
|
||||
addViolation(
|
||||
violations,
|
||||
sourceFile,
|
||||
node,
|
||||
"new raw node:sqlite access requires Kysely or an explicit raw SQLite allowlist entry",
|
||||
);
|
||||
}
|
||||
|
||||
ts.forEachChild(node, visit);
|
||||
}
|
||||
|
||||
visit(sourceFile);
|
||||
return violations;
|
||||
}
|
||||
|
||||
/**
|
||||
* Collects Kysely guardrail violations across configured source roots.
|
||||
*/
|
||||
export async function collectKyselyGuardrails() {
|
||||
const files = await collectTypeScriptFilesFromRoots(sourceRoots, { includeTests: true });
|
||||
const violations = [];
|
||||
for (const filePath of files) {
|
||||
const relativePath = path.relative(repoRoot, filePath).split(path.sep).join("/");
|
||||
const content = await fs.readFile(filePath, "utf8");
|
||||
for (const violation of collectKyselyGuardrailViolations(content, relativePath)) {
|
||||
violations.push({ path: relativePath, ...violation });
|
||||
}
|
||||
}
|
||||
return violations;
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the Kysely guardrail check.
|
||||
*/
|
||||
export async function main() {
|
||||
const violations = await collectKyselyGuardrails();
|
||||
if (violations.length === 0) {
|
||||
console.log("Kysely guardrails OK");
|
||||
return;
|
||||
}
|
||||
console.error("Kysely guardrail violations:");
|
||||
for (const violation of violations) {
|
||||
console.error(`- ${violation.path}:${violation.line}: ${violation.message}`);
|
||||
}
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
runAsScript(import.meta.url, main);
|
||||
26
scripts/check-live-cache.ts
Normal file
26
scripts/check-live-cache.ts
Normal file
@@ -0,0 +1,26 @@
|
||||
// Check Live Cache script supports OpenClaw repository automation.
|
||||
import { runLiveCacheRegression } from "../src/agents/live-cache-regression-runner.js";
|
||||
import { LIVE_CACHE_TEST_ENABLED, logLiveCache } from "../src/agents/live-cache-test-support.js";
|
||||
|
||||
if (!LIVE_CACHE_TEST_ENABLED) {
|
||||
logLiveCache("skipped; set OPENCLAW_LIVE_TEST=1 and OPENCLAW_LIVE_CACHE_TEST=1");
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
const result = await runLiveCacheRegression();
|
||||
if (result.warnings.length > 0) {
|
||||
process.stderr.write("\n[live-cache] non-blocking cache observations:\n");
|
||||
for (const warning of result.warnings) {
|
||||
process.stderr.write(`- ${warning}\n`);
|
||||
}
|
||||
}
|
||||
if (result.regressions.length > 0) {
|
||||
process.stderr.write("\n[live-cache] regressions detected:\n");
|
||||
for (const regression of result.regressions) {
|
||||
process.stderr.write(`- ${regression}\n`);
|
||||
}
|
||||
process.exit(1);
|
||||
} else {
|
||||
process.stderr.write("\n[live-cache] all regression floors satisfied\n");
|
||||
process.exit(0);
|
||||
}
|
||||
120
scripts/check-madge-import-cycles.ts
Normal file
120
scripts/check-madge-import-cycles.ts
Normal file
@@ -0,0 +1,120 @@
|
||||
#!/usr/bin/env node
|
||||
// Check Madge Import Cycles script supports OpenClaw repository automation.
|
||||
import { readFileSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import ts from "typescript";
|
||||
import {
|
||||
collectSourceFiles,
|
||||
collectStronglyConnectedComponents,
|
||||
} from "./lib/import-cycle-graph.ts";
|
||||
|
||||
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const scanRoots = ["src", "extensions", "ui"] as const;
|
||||
const sourceExtensions = [".ts"] as const;
|
||||
const ignoredPathPartPattern =
|
||||
/(^|\/)(node_modules|dist|build|coverage|\.artifacts|\.git|assets)(\/|$)/;
|
||||
|
||||
function shouldSkipRepoPath(repoPath: string): boolean {
|
||||
return ignoredPathPartPattern.test(repoPath);
|
||||
}
|
||||
|
||||
function loadCompilerOptions(): ts.CompilerOptions {
|
||||
const configPath = path.join(repoRoot, "tsconfig.json");
|
||||
const config = ts.readConfigFile(configPath, (filePath) => ts.sys.readFile(filePath));
|
||||
if (config.error) {
|
||||
throw new Error(ts.flattenDiagnosticMessageText(config.error.messageText, "\n"));
|
||||
}
|
||||
return ts.parseJsonConfigFileContent(config.config, ts.sys, repoRoot).options;
|
||||
}
|
||||
|
||||
function collectStaticModuleSpecifiers(sourceFile: ts.SourceFile): string[] {
|
||||
const specifiers: string[] = [];
|
||||
const visit = (node: ts.Node) => {
|
||||
if (ts.isImportDeclaration(node) && ts.isStringLiteral(node.moduleSpecifier)) {
|
||||
specifiers.push(node.moduleSpecifier.text);
|
||||
} else if (
|
||||
ts.isExportDeclaration(node) &&
|
||||
node.moduleSpecifier &&
|
||||
ts.isStringLiteral(node.moduleSpecifier)
|
||||
) {
|
||||
specifiers.push(node.moduleSpecifier.text);
|
||||
}
|
||||
ts.forEachChild(node, visit);
|
||||
};
|
||||
visit(sourceFile);
|
||||
return specifiers;
|
||||
}
|
||||
|
||||
function createImportGraph(files: readonly string[]): Map<string, string[]> {
|
||||
const compilerOptions = loadCompilerOptions();
|
||||
const compilerHost = ts.createCompilerHost(compilerOptions, false);
|
||||
const resolutionCache = ts.createModuleResolutionCache(
|
||||
repoRoot,
|
||||
(value) => value,
|
||||
compilerOptions,
|
||||
);
|
||||
const absoluteToRepoPath = new Map(
|
||||
files.map((file): [string, string] => [path.resolve(repoRoot, file), file]),
|
||||
);
|
||||
const graph = new Map<string, string[]>();
|
||||
|
||||
for (const file of files) {
|
||||
const absoluteFile = path.join(repoRoot, file);
|
||||
const sourceFile = ts.createSourceFile(
|
||||
file,
|
||||
readFileSync(absoluteFile, "utf8"),
|
||||
ts.ScriptTarget.Latest,
|
||||
true,
|
||||
);
|
||||
const imports = collectStaticModuleSpecifiers(sourceFile).flatMap((specifier) => {
|
||||
const resolved = ts.resolveModuleName(
|
||||
specifier,
|
||||
absoluteFile,
|
||||
compilerOptions,
|
||||
compilerHost,
|
||||
resolutionCache,
|
||||
).resolvedModule?.resolvedFileName;
|
||||
if (!resolved) {
|
||||
return [];
|
||||
}
|
||||
const repoPath = absoluteToRepoPath.get(path.resolve(resolved));
|
||||
return repoPath ? [repoPath] : [];
|
||||
});
|
||||
graph.set(
|
||||
file,
|
||||
imports.toSorted((left, right) => left.localeCompare(right)),
|
||||
);
|
||||
}
|
||||
|
||||
return graph;
|
||||
}
|
||||
|
||||
function main(): number {
|
||||
const files = scanRoots.flatMap((root) =>
|
||||
collectSourceFiles(path.join(repoRoot, root), {
|
||||
repoRoot,
|
||||
sourceExtensions,
|
||||
shouldSkipRepoPath,
|
||||
}),
|
||||
);
|
||||
const graph = createImportGraph(files);
|
||||
const cycles = collectStronglyConnectedComponents(graph);
|
||||
|
||||
console.log(`Madge import cycle check: ${cycles.length} cycle(s).`);
|
||||
if (cycles.length === 0) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
console.error("\nMadge circular dependencies:");
|
||||
for (const [index, cycle] of cycles.entries()) {
|
||||
console.error(`\n# cycle ${index + 1}`);
|
||||
console.error(` ${cycle.join("\n -> ")}`);
|
||||
}
|
||||
console.error(
|
||||
"\nBreak the cycle or extract a leaf contract instead of routing through a barrel.",
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
|
||||
process.exitCode = main();
|
||||
67
scripts/check-media-download-helper-roundtrip.mjs
Normal file
67
scripts/check-media-download-helper-roundtrip.mjs
Normal file
@@ -0,0 +1,67 @@
|
||||
#!/usr/bin/env node
|
||||
// Checks extension media downloads keep helper-read/save calls close together.
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
|
||||
const WINDOW_LINES = 80;
|
||||
const READ_HELPER_RE = /\b(?:readRemoteMediaBuffer|fetchRemoteMedia)\s*\(/;
|
||||
const SAVE_BUFFER_RE = /(?:\.|\b)saveMediaBuffer\s*\(/;
|
||||
|
||||
function listTrackedExtensionSources() {
|
||||
return execFileSync("git", ["ls-files", "extensions/**/*.ts"], {
|
||||
encoding: "utf8",
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
})
|
||||
.split("\n")
|
||||
.filter(Boolean)
|
||||
.filter((file) => file.includes("/src/"))
|
||||
.filter((file) => existsSync(file))
|
||||
.filter((file) => !isTestOrFixture(file));
|
||||
}
|
||||
|
||||
function isTestOrFixture(file) {
|
||||
return (
|
||||
file.endsWith(".test.ts") ||
|
||||
file.endsWith(".e2e.test.ts") ||
|
||||
file.endsWith(".test-harness.ts") ||
|
||||
file.endsWith(".test-utils.ts") ||
|
||||
file.endsWith("/test-runtime.ts") ||
|
||||
file.endsWith("/test-helpers.ts") ||
|
||||
file.includes("/test-support/") ||
|
||||
file.includes("/fixtures/")
|
||||
);
|
||||
}
|
||||
|
||||
const findings = [];
|
||||
|
||||
for (const file of listTrackedExtensionSources()) {
|
||||
const lines = readFileSync(file, "utf8").split("\n");
|
||||
for (let index = 0; index < lines.length; index += 1) {
|
||||
if (!READ_HELPER_RE.test(lines[index])) {
|
||||
continue;
|
||||
}
|
||||
const end = Math.min(lines.length, index + WINDOW_LINES);
|
||||
for (let nextIndex = index; nextIndex < end; nextIndex += 1) {
|
||||
if (!SAVE_BUFFER_RE.test(lines[nextIndex])) {
|
||||
continue;
|
||||
}
|
||||
findings.push({
|
||||
file,
|
||||
line: index + 1,
|
||||
saveLine: nextIndex + 1,
|
||||
});
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (findings.length > 0) {
|
||||
console.error("Avoid remote-media buffer/store round trips in plugin production code.");
|
||||
console.error(
|
||||
"Use saveRemoteMedia(...) for URL-to-store or saveResponseMedia(...) for fetched Response objects.",
|
||||
);
|
||||
for (const finding of findings) {
|
||||
console.error(`- ${finding.file}:${finding.line} -> saveMediaBuffer at ${finding.saveLine}`);
|
||||
}
|
||||
process.exitCode = 1;
|
||||
}
|
||||
901
scripts/check-memory-fd-repro.mjs
Normal file
901
scripts/check-memory-fd-repro.mjs
Normal file
@@ -0,0 +1,901 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Reproduces memory-search file descriptor retention with a synthetic workspace.
|
||||
import { spawn, spawnSync } from "node:child_process";
|
||||
import fs from "node:fs";
|
||||
import net from "node:net";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import process from "node:process";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { stripLeadingPackageManagerSeparator } from "./lib/arg-utils.mjs";
|
||||
import { readBoundedResponseText } from "./lib/bounded-response.mjs";
|
||||
|
||||
const ISSUE_FILE_COUNTS = [
|
||||
["memory/transcripts", 9394],
|
||||
["memory/transcripts.archived", 1695],
|
||||
["memory/structured-md/lessons", 268],
|
||||
["memory/structured-md/decisions", 215],
|
||||
["memory/structured-md/lessons.archived", 214],
|
||||
["memory/structured-md/procedures", 213],
|
||||
["memory/structured-md/decisions.archived", 151],
|
||||
["memory/structured-md/procedures.archived", 126],
|
||||
["memory/structured-md/projects", 81],
|
||||
["memory/structured-md/projects.archived", 34],
|
||||
];
|
||||
|
||||
const ISSUE_MEMORY_FILE_COUNT = ISSUE_FILE_COUNTS.reduce((sum, [, count]) => sum + count, 0);
|
||||
const DEFAULT_FILE_COUNT = 512;
|
||||
const DEFAULT_MAX_WORKSPACE_REG_FDS = process.platform === "darwin" ? 8 : 64;
|
||||
const MAX_TIMER_TIMEOUT_MS = 2_147_000_000;
|
||||
/**
|
||||
* Maximum gateway-ready output tail retained while waiting for startup.
|
||||
*/
|
||||
export const GATEWAY_READY_OUTPUT_MAX_CHARS = 128 * 1024;
|
||||
/**
|
||||
* Maximum bytes read from the memory_search HTTP response.
|
||||
*/
|
||||
export const MEMORY_SEARCH_RESPONSE_MAX_BYTES = 256 * 1024;
|
||||
/**
|
||||
* Probe query expected to hit the synthetic top-level memory file.
|
||||
*/
|
||||
export const MEMORY_SEARCH_PROBE_QUERY = "Top-level memory file";
|
||||
|
||||
const SKIP_GATEWAY_ENV = {
|
||||
NODE_ENV: "test",
|
||||
OPENCLAW_DISABLE_BONJOUR: "1",
|
||||
OPENCLAW_NO_RESPAWN: "1",
|
||||
OPENCLAW_SKIP_ACPX_RUNTIME: "1",
|
||||
OPENCLAW_SKIP_ACPX_RUNTIME_PROBE: "1",
|
||||
OPENCLAW_SKIP_BROWSER_CONTROL_SERVER: "1",
|
||||
OPENCLAW_SKIP_CANVAS_HOST: "1",
|
||||
OPENCLAW_SKIP_CHANNELS: "1",
|
||||
OPENCLAW_SKIP_CRON: "1",
|
||||
OPENCLAW_SKIP_GMAIL_WATCHER: "1",
|
||||
OPENCLAW_SKIP_PROVIDERS: "1",
|
||||
};
|
||||
|
||||
function usage() {
|
||||
return `
|
||||
Usage: node scripts/check-memory-fd-repro.mjs [options]
|
||||
|
||||
Options:
|
||||
--full Use the issue-sized 12,391-file memory tree.
|
||||
--files <count> Number of memory/**/*.md files to generate. Default: ${DEFAULT_FILE_COUNT}.
|
||||
--mode <fixed|leak|report> fixed fails on FD fan-out; leak expects it; report never fails. Default: fixed.
|
||||
--max-workspace-reg-fds <n> Fixed-mode maximum retained workspace Markdown REG FDs. Default: ${DEFAULT_MAX_WORKSPACE_REG_FDS}.
|
||||
--min-leaked-fds <n> Leak-mode minimum retained workspace Markdown REG FDs. Default: min(files, 64).
|
||||
--invoke-timeout-ms <n> Abort the memory_search HTTP call after this long. Default: 30000.
|
||||
--sample-delay-ms <n> First post-invoke FD sample delay. Default: 1000.
|
||||
--settle-delay-ms <n> Final FD sample delay after invoke settles. Default: 5000.
|
||||
--output-dir <path> Artifact directory. Default: .artifacts/memory-fd-repro/<timestamp>.
|
||||
--keep Keep the synthetic OPENCLAW_HOME and workspace after the run.
|
||||
--allow-non-darwin Run on non-macOS platforms. lsof REG counts are most meaningful on macOS.
|
||||
--help Show this help.
|
||||
`.trim();
|
||||
}
|
||||
|
||||
const NON_NEGATIVE_INTEGER_PATTERN = /^(0|[1-9]\d*)$/u;
|
||||
const ARGUMENT_FLAGS = new Set([
|
||||
"--allow-non-darwin",
|
||||
"--expect-leak",
|
||||
"--files",
|
||||
"--full",
|
||||
"--help",
|
||||
"--invoke-timeout-ms",
|
||||
"--keep",
|
||||
"--max-workspace-reg-fds",
|
||||
"--min-leaked-fds",
|
||||
"--mode",
|
||||
"--output-dir",
|
||||
"--report-only",
|
||||
"--sample-delay-ms",
|
||||
"--settle-delay-ms",
|
||||
]);
|
||||
|
||||
function stripPackageManagerSeparatorForKnownFlags(argv) {
|
||||
return argv[0] === "--" && ARGUMENT_FLAGS.has(argv[1])
|
||||
? stripLeadingPackageManagerSeparator(argv)
|
||||
: argv;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parses a safe non-negative integer option.
|
||||
*/
|
||||
export function readNumber(value, label) {
|
||||
const raw = String(value).trim();
|
||||
if (!NON_NEGATIVE_INTEGER_PATTERN.test(raw)) {
|
||||
throw new Error(`${label} must be a non-negative integer`);
|
||||
}
|
||||
const parsed = Number(raw);
|
||||
if (!Number.isSafeInteger(parsed)) {
|
||||
throw new Error(`${label} must be a safe integer`);
|
||||
}
|
||||
return parsed;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parses a safe positive integer option.
|
||||
*/
|
||||
export function readPositiveNumber(value, label) {
|
||||
const parsed = readNumber(value, label);
|
||||
if (parsed <= 0) {
|
||||
throw new Error(`${label} must be greater than 0`);
|
||||
}
|
||||
return parsed;
|
||||
}
|
||||
|
||||
function readNumberEnv(name, fallback) {
|
||||
const raw = process.env[name];
|
||||
return raw == null || raw.trim() === "" ? fallback : readNumber(raw, name);
|
||||
}
|
||||
|
||||
function readPositiveNumberEnv(name, fallback) {
|
||||
const raw = process.env[name];
|
||||
return raw == null || raw.trim() === "" ? fallback : readPositiveNumber(raw, name);
|
||||
}
|
||||
|
||||
function clampTimerTimeoutMs(valueMs, minMs = 1) {
|
||||
const min = Math.max(0, Math.floor(minMs));
|
||||
const value = Number.isFinite(valueMs) ? valueMs : min;
|
||||
return Math.min(Math.max(Math.floor(value), min), MAX_TIMER_TIMEOUT_MS);
|
||||
}
|
||||
|
||||
function readTimerTimeoutNumber(value, label, minMs = 1) {
|
||||
const parsed = minMs > 0 ? readPositiveNumber(value, label) : readNumber(value, label);
|
||||
return clampTimerTimeoutMs(parsed, minMs);
|
||||
}
|
||||
|
||||
function readTimerTimeoutNumberEnv(name, fallback, minMs = 1) {
|
||||
const raw = process.env[name];
|
||||
return raw == null || raw.trim() === ""
|
||||
? clampTimerTimeoutMs(fallback, minMs)
|
||||
: readTimerTimeoutNumber(raw, name, minMs);
|
||||
}
|
||||
|
||||
/**
|
||||
* Parses memory FD repro CLI arguments and environment fallbacks.
|
||||
*/
|
||||
export function parseArgs(argv) {
|
||||
const args = stripPackageManagerSeparatorForKnownFlags(argv);
|
||||
const stamp = new Date().toISOString().replace(/[:.]/g, "-");
|
||||
const options = {
|
||||
fileCount: undefined,
|
||||
mode: process.env.OPENCLAW_MEMORY_FD_REPRO_MODE || "fixed",
|
||||
maxWorkspaceRegFds: undefined,
|
||||
minLeakedFds: undefined,
|
||||
invokeTimeoutMs: undefined,
|
||||
sampleDelayMs: undefined,
|
||||
settleDelayMs: undefined,
|
||||
outputDir: path.resolve(".artifacts", "memory-fd-repro", stamp),
|
||||
keep: process.env.OPENCLAW_MEMORY_FD_REPRO_KEEP === "1",
|
||||
allowNonDarwin: process.env.OPENCLAW_MEMORY_FD_REPRO_ALLOW_NON_DARWIN === "1",
|
||||
};
|
||||
|
||||
parseArgv: for (let i = 0; i < args.length; i += 1) {
|
||||
const arg = args[i];
|
||||
const next = args[i + 1];
|
||||
const readValue = () => {
|
||||
if (!next || next.startsWith("-")) {
|
||||
throw new Error(`Missing value for ${arg}`);
|
||||
}
|
||||
i += 1;
|
||||
return next;
|
||||
};
|
||||
|
||||
switch (arg) {
|
||||
case "--":
|
||||
break parseArgv;
|
||||
case "--help":
|
||||
console.log(usage());
|
||||
process.exit(0);
|
||||
case "--full":
|
||||
options.fileCount = ISSUE_MEMORY_FILE_COUNT;
|
||||
break;
|
||||
case "--files":
|
||||
options.fileCount = readPositiveNumber(readValue(), "--files");
|
||||
break;
|
||||
case "--mode":
|
||||
options.mode = readValue();
|
||||
break;
|
||||
case "--expect-leak":
|
||||
options.mode = "leak";
|
||||
break;
|
||||
case "--report-only":
|
||||
options.mode = "report";
|
||||
break;
|
||||
case "--max-workspace-reg-fds":
|
||||
options.maxWorkspaceRegFds = readNumber(readValue(), "--max-workspace-reg-fds");
|
||||
break;
|
||||
case "--min-leaked-fds":
|
||||
options.minLeakedFds = readPositiveNumber(readValue(), "--min-leaked-fds");
|
||||
break;
|
||||
case "--invoke-timeout-ms":
|
||||
options.invokeTimeoutMs = readTimerTimeoutNumber(readValue(), "--invoke-timeout-ms");
|
||||
break;
|
||||
case "--sample-delay-ms":
|
||||
options.sampleDelayMs = readTimerTimeoutNumber(readValue(), "--sample-delay-ms", 0);
|
||||
break;
|
||||
case "--settle-delay-ms":
|
||||
options.settleDelayMs = readTimerTimeoutNumber(readValue(), "--settle-delay-ms", 0);
|
||||
break;
|
||||
case "--output-dir":
|
||||
options.outputDir = path.resolve(readValue());
|
||||
break;
|
||||
case "--keep":
|
||||
options.keep = true;
|
||||
break;
|
||||
case "--allow-non-darwin":
|
||||
options.allowNonDarwin = true;
|
||||
break;
|
||||
default:
|
||||
throw new Error(`Unknown argument: ${arg}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (!["fixed", "leak", "report"].includes(options.mode)) {
|
||||
throw new Error('--mode must be "fixed", "leak", or "report"');
|
||||
}
|
||||
options.fileCount ??= readPositiveNumberEnv("OPENCLAW_MEMORY_FD_REPRO_FILES", DEFAULT_FILE_COUNT);
|
||||
options.maxWorkspaceRegFds ??= readNumberEnv(
|
||||
"OPENCLAW_MEMORY_FD_REPRO_MAX_WORKSPACE_REG_FDS",
|
||||
DEFAULT_MAX_WORKSPACE_REG_FDS,
|
||||
);
|
||||
options.invokeTimeoutMs ??= readTimerTimeoutNumberEnv("OPENCLAW_MEMORY_FD_REPRO_TIMEOUT_MS", 30_000);
|
||||
options.sampleDelayMs ??= readTimerTimeoutNumberEnv(
|
||||
"OPENCLAW_MEMORY_FD_REPRO_SAMPLE_DELAY_MS",
|
||||
1_000,
|
||||
0,
|
||||
);
|
||||
options.settleDelayMs ??= readTimerTimeoutNumberEnv(
|
||||
"OPENCLAW_MEMORY_FD_REPRO_SETTLE_DELAY_MS",
|
||||
5_000,
|
||||
0,
|
||||
);
|
||||
if (!Number.isFinite(options.fileCount) || options.fileCount <= 0) {
|
||||
throw new Error("file count must be greater than 0");
|
||||
}
|
||||
if (!Number.isFinite(options.maxWorkspaceRegFds) || options.maxWorkspaceRegFds < 0) {
|
||||
throw new Error("max workspace REG FD threshold must be non-negative");
|
||||
}
|
||||
if (options.minLeakedFds === undefined) {
|
||||
options.minLeakedFds = Math.min(options.fileCount, 64);
|
||||
}
|
||||
return options;
|
||||
}
|
||||
|
||||
function logStep(message) {
|
||||
console.log(`[memory-fd-repro] ${message}`);
|
||||
}
|
||||
|
||||
function sleep(ms) {
|
||||
return new Promise((resolve) => {
|
||||
setTimeout(resolve, clampTimerTimeoutMs(ms, 0));
|
||||
});
|
||||
}
|
||||
|
||||
async function getFreePort() {
|
||||
return await new Promise((resolve, reject) => {
|
||||
const server = net.createServer();
|
||||
server.unref();
|
||||
server.on("error", reject);
|
||||
server.listen(0, "127.0.0.1", () => {
|
||||
const address = server.address();
|
||||
const port = typeof address === "object" && address ? address.port : 0;
|
||||
server.close(() => (port > 0 ? resolve(port) : reject(new Error("no free port"))));
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
function distributeFileCounts(total) {
|
||||
const exact = ISSUE_FILE_COUNTS.map(([dir, count]) => ({
|
||||
dir,
|
||||
count: Math.floor((count / ISSUE_MEMORY_FILE_COUNT) * total),
|
||||
remainder: (count / ISSUE_MEMORY_FILE_COUNT) * total,
|
||||
}));
|
||||
let assigned = exact.reduce((sum, entry) => sum + entry.count, 0);
|
||||
for (const entry of exact.toSorted((a, b) => b.remainder - a.remainder)) {
|
||||
if (assigned >= total) {
|
||||
break;
|
||||
}
|
||||
entry.count += 1;
|
||||
assigned += 1;
|
||||
}
|
||||
return exact.filter((entry) => entry.count > 0).map(({ dir, count }) => [dir, count]);
|
||||
}
|
||||
|
||||
function writeSyntheticWorkspace(workspaceDir, fileCount) {
|
||||
fs.mkdirSync(workspaceDir, { recursive: true });
|
||||
fs.writeFileSync(
|
||||
path.join(workspaceDir, "MEMORY.md"),
|
||||
"# Memory\n\nTop-level memory file for FD repro.\n",
|
||||
);
|
||||
|
||||
for (const [relativeDir, count] of distributeFileCounts(fileCount)) {
|
||||
const dir = path.join(workspaceDir, relativeDir);
|
||||
fs.mkdirSync(dir, { recursive: true });
|
||||
for (let index = 1; index <= count; index += 1) {
|
||||
const name = `${String(index).padStart(5, "0")}.md`;
|
||||
fs.writeFileSync(
|
||||
path.join(dir, name),
|
||||
`# ${relativeDir} ${index}\n\nSynthetic memory note ${index}.\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Writes isolated OpenClaw config for the synthetic memory workspace.
|
||||
*/
|
||||
export function writeConfig({ homeDir, workspaceDir, port, token }) {
|
||||
const configDir = path.join(homeDir, ".openclaw");
|
||||
fs.mkdirSync(configDir, { recursive: true });
|
||||
const configPath = path.join(configDir, "openclaw.json");
|
||||
const config = {
|
||||
agents: {
|
||||
defaults: {
|
||||
workspace: workspaceDir,
|
||||
memorySearch: {
|
||||
provider: "none",
|
||||
model: "",
|
||||
store: {
|
||||
vector: { enabled: false },
|
||||
},
|
||||
sync: {
|
||||
watch: true,
|
||||
onSessionStart: false,
|
||||
onSearch: false,
|
||||
},
|
||||
},
|
||||
},
|
||||
list: [
|
||||
{
|
||||
id: "main",
|
||||
default: true,
|
||||
tools: { allow: ["memory_search"] },
|
||||
},
|
||||
],
|
||||
},
|
||||
plugins: { allow: ["memory-core"] },
|
||||
gateway: {
|
||||
mode: "local",
|
||||
bind: "loopback",
|
||||
port,
|
||||
auth: { mode: "token", token },
|
||||
},
|
||||
};
|
||||
fs.writeFileSync(configPath, `${JSON.stringify(config, null, 2)}\n`);
|
||||
return configPath;
|
||||
}
|
||||
|
||||
function formatTail(text, maxChars = 4096) {
|
||||
return text.length > maxChars ? text.slice(-maxChars) : text;
|
||||
}
|
||||
|
||||
function preindexSyntheticMemory(env) {
|
||||
logStep("preindex start");
|
||||
const result = spawnSync(
|
||||
process.execPath,
|
||||
["scripts/run-node.mjs", "memory", "index", "--force", "--agent", "main"],
|
||||
{
|
||||
cwd: process.cwd(),
|
||||
encoding: "utf8",
|
||||
env,
|
||||
maxBuffer: 10 * 1024 * 1024,
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
},
|
||||
);
|
||||
if (result.status !== 0) {
|
||||
throw new Error(
|
||||
[
|
||||
`memory preindex failed with exit ${result.status ?? result.signal}`,
|
||||
formatTail(result.stdout || ""),
|
||||
formatTail(result.stderr || ""),
|
||||
]
|
||||
.filter(Boolean)
|
||||
.join("\n"),
|
||||
);
|
||||
}
|
||||
logStep("preindex complete");
|
||||
}
|
||||
|
||||
/**
|
||||
* Updates bounded gateway-ready output state from a stdout/stderr chunk.
|
||||
*/
|
||||
export function updateGatewayReadyOutputState(
|
||||
state,
|
||||
chunk,
|
||||
maxChars = GATEWAY_READY_OUTPUT_MAX_CHARS,
|
||||
) {
|
||||
const text = String(chunk);
|
||||
const combined = `${state.tail ?? ""}${text}`;
|
||||
return {
|
||||
tail: combined.length > maxChars ? combined.slice(-maxChars) : combined,
|
||||
readySeen: Boolean(state.readySeen || combined.includes("[gateway] ready")),
|
||||
};
|
||||
}
|
||||
|
||||
function runLsofForPid(pid) {
|
||||
const result = spawnSync("lsof", ["-nP", "-p", String(pid)], {
|
||||
encoding: "utf8",
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
if (result.status !== 0) {
|
||||
throw new Error(`lsof failed: ${result.stderr || result.stdout}`);
|
||||
}
|
||||
return result.stdout;
|
||||
}
|
||||
|
||||
function findGatewayPid(port) {
|
||||
const result = spawnSync("lsof", ["-nP", `-iTCP:${port}`, "-sTCP:LISTEN", "-t"], {
|
||||
encoding: "utf8",
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
if (result.status !== 0 && result.stdout.trim() === "") {
|
||||
return null;
|
||||
}
|
||||
const pid = Number(result.stdout.trim().split(/\s+/)[0]);
|
||||
return Number.isFinite(pid) && pid > 0 ? pid : null;
|
||||
}
|
||||
|
||||
function sampleFds({ label, pid, workspaceRealPath }) {
|
||||
const output = runLsofForPid(pid);
|
||||
const workspacePrefix = `${workspaceRealPath}${path.sep}`;
|
||||
const workspaceMarkdownPaths = [];
|
||||
let total = 0;
|
||||
let reg = 0;
|
||||
|
||||
for (const line of output.split("\n").slice(1)) {
|
||||
if (!line.trim()) {
|
||||
continue;
|
||||
}
|
||||
total += 1;
|
||||
const columns = line.trim().split(/\s+/);
|
||||
const type = columns[4];
|
||||
const filePath = columns[columns.length - 1];
|
||||
if (type === "REG") {
|
||||
reg += 1;
|
||||
}
|
||||
if (
|
||||
type === "REG" &&
|
||||
filePath?.startsWith(workspacePrefix) &&
|
||||
(filePath === path.join(workspaceRealPath, "MEMORY.md") ||
|
||||
(filePath.startsWith(path.join(workspaceRealPath, "memory") + path.sep) &&
|
||||
filePath.endsWith(".md")))
|
||||
) {
|
||||
workspaceMarkdownPaths.push(filePath);
|
||||
}
|
||||
}
|
||||
|
||||
const sample = {
|
||||
label,
|
||||
totalFds: total,
|
||||
regFds: reg,
|
||||
workspaceMarkdownRegFds: workspaceMarkdownPaths.length,
|
||||
uniqueWorkspaceMarkdownRegFds: new Set(workspaceMarkdownPaths).size,
|
||||
sampledAt: new Date().toISOString(),
|
||||
};
|
||||
logStep(
|
||||
`${label}: total=${sample.totalFds} reg=${sample.regFds} workspace_md_reg=${sample.workspaceMarkdownRegFds} unique_workspace_md_reg=${sample.uniqueWorkspaceMarkdownRegFds}`,
|
||||
);
|
||||
return sample;
|
||||
}
|
||||
|
||||
/**
|
||||
* Reports whether a spawned child has already exited.
|
||||
*/
|
||||
export function hasChildExited(child) {
|
||||
return child.exitCode !== null || child.signalCode !== null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Waits until gateway output and listener state both indicate readiness.
|
||||
*/
|
||||
export async function waitForGatewayReady({ child, port, logPath, timeoutMs }) {
|
||||
const startedAt = Date.now();
|
||||
let outputState = { tail: "", readySeen: false };
|
||||
const append = (chunk) => {
|
||||
const text = chunk.toString();
|
||||
outputState = updateGatewayReadyOutputState(outputState, text);
|
||||
fs.appendFileSync(logPath, text);
|
||||
};
|
||||
child.stdout.on("data", append);
|
||||
child.stderr.on("data", append);
|
||||
|
||||
while (Date.now() - startedAt < timeoutMs) {
|
||||
if (outputState.readySeen && findGatewayPid(port)) {
|
||||
return;
|
||||
}
|
||||
if (hasChildExited(child)) {
|
||||
throw new Error(`gateway exited before ready; see ${logPath}`);
|
||||
}
|
||||
await sleep(100);
|
||||
}
|
||||
throw new Error(`gateway did not become ready within ${timeoutMs}ms; see ${logPath}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Stops the gateway child using the default process/runtime hooks.
|
||||
*/
|
||||
export async function stopGateway({ child, port }) {
|
||||
return stopGatewayWithRuntime({
|
||||
child,
|
||||
port,
|
||||
findGatewayPidFn: findGatewayPid,
|
||||
killProcess: (pid, signal) => process.kill(pid, signal),
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Stops the gateway child and any remaining listener process.
|
||||
*/
|
||||
export async function stopGatewayWithRuntime({
|
||||
child,
|
||||
childExitPollIntervalMs = 100,
|
||||
childExitPolls = 50,
|
||||
port,
|
||||
findGatewayPidFn,
|
||||
killProcess,
|
||||
listenerSettleDelayMs = 500,
|
||||
}) {
|
||||
if (!hasChildExited(child)) {
|
||||
signalChild(child, "SIGINT");
|
||||
await waitForChildExit(child, { intervalMs: childExitPollIntervalMs, polls: childExitPolls });
|
||||
}
|
||||
const listenerPid = findGatewayPidFn(port);
|
||||
if (listenerPid) {
|
||||
try {
|
||||
killProcess(listenerPid, "SIGTERM");
|
||||
} catch {}
|
||||
await sleep(listenerSettleDelayMs);
|
||||
const stillListening = findGatewayPidFn(port);
|
||||
if (stillListening) {
|
||||
try {
|
||||
killProcess(stillListening, "SIGKILL");
|
||||
} catch {}
|
||||
}
|
||||
}
|
||||
if (!hasChildExited(child)) {
|
||||
signalChild(child, "SIGKILL");
|
||||
await waitForChildExit(child, { intervalMs: childExitPollIntervalMs, polls: childExitPolls });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads an HTTP response body up to a configured byte limit.
|
||||
*/
|
||||
export { readBoundedResponseText };
|
||||
|
||||
function signalChild(child, signal) {
|
||||
try {
|
||||
child.kill(signal);
|
||||
} catch {}
|
||||
}
|
||||
|
||||
async function waitForChildExit(child, { intervalMs, polls }) {
|
||||
for (let i = 0; i < polls; i += 1) {
|
||||
if (hasChildExited(child)) {
|
||||
return true;
|
||||
}
|
||||
await sleep(intervalMs);
|
||||
}
|
||||
return hasChildExited(child);
|
||||
}
|
||||
|
||||
function parseJsonValue(text) {
|
||||
try {
|
||||
return JSON.parse(text);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function asRecord(value) {
|
||||
return value && typeof value === "object" && !Array.isArray(value) ? value : null;
|
||||
}
|
||||
|
||||
function readStringProperty(record, key) {
|
||||
const value = record?.[key];
|
||||
return typeof value === "string" && value.trim() ? value : undefined;
|
||||
}
|
||||
|
||||
function parseToolTextContent(result) {
|
||||
const content = Array.isArray(result?.content) ? result.content : [];
|
||||
for (const entry of content) {
|
||||
const text = entry?.type === "text" && typeof entry.text === "string" ? entry.text : null;
|
||||
if (!text) {
|
||||
continue;
|
||||
}
|
||||
const parsed = asRecord(parseJsonValue(text));
|
||||
if (parsed) {
|
||||
return parsed;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Classifies the memory_search HTTP response into success/error details.
|
||||
*/
|
||||
export function classifyMemorySearchInvokeResponse({ httpOk, status, bodyText }) {
|
||||
const parsedBody = parseJsonValue(bodyText);
|
||||
const body = asRecord(parsedBody);
|
||||
if (!httpOk) {
|
||||
const errorRecord = asRecord(body?.error);
|
||||
return {
|
||||
ok: false,
|
||||
httpOk,
|
||||
status,
|
||||
gatewayOk: body?.ok === true ? true : body?.ok === false ? false : undefined,
|
||||
error:
|
||||
readStringProperty(errorRecord, "message") ??
|
||||
readStringProperty(body, "error") ??
|
||||
`memory_search HTTP request failed with status ${status}`,
|
||||
};
|
||||
}
|
||||
if (!body) {
|
||||
return {
|
||||
ok: false,
|
||||
httpOk,
|
||||
status,
|
||||
error: "memory_search response was not JSON",
|
||||
};
|
||||
}
|
||||
|
||||
const gatewayOk = body.ok === true ? true : body.ok === false ? false : undefined;
|
||||
if (gatewayOk === false) {
|
||||
const errorRecord = asRecord(body.error);
|
||||
return {
|
||||
ok: false,
|
||||
httpOk,
|
||||
status,
|
||||
gatewayOk,
|
||||
error:
|
||||
readStringProperty(errorRecord, "message") ??
|
||||
readStringProperty(body, "error") ??
|
||||
"memory_search gateway invocation failed",
|
||||
};
|
||||
}
|
||||
|
||||
const result = asRecord(body.result);
|
||||
const details = asRecord(result?.details);
|
||||
const directResult = Array.isArray(result?.results) ? result : null;
|
||||
const directBody =
|
||||
Array.isArray(body.results) || body.disabled === true || body.unavailable === true
|
||||
? body
|
||||
: null;
|
||||
const payload = details ?? parseToolTextContent(result) ?? directResult ?? directBody;
|
||||
if (!payload) {
|
||||
return {
|
||||
ok: false,
|
||||
httpOk,
|
||||
status,
|
||||
gatewayOk,
|
||||
error: "memory_search result payload missing or invalid",
|
||||
};
|
||||
}
|
||||
const resultCount = Array.isArray(payload.results) ? payload.results.length : undefined;
|
||||
const toolDisabled = payload.disabled === true;
|
||||
const toolUnavailable = payload.unavailable === true;
|
||||
const toolError = readStringProperty(payload, "error");
|
||||
const ok = gatewayOk === true && !toolDisabled && !toolUnavailable && !toolError;
|
||||
|
||||
return {
|
||||
ok,
|
||||
httpOk,
|
||||
status,
|
||||
gatewayOk,
|
||||
resultCount,
|
||||
toolDisabled,
|
||||
toolUnavailable,
|
||||
...(toolError ? { toolError } : {}),
|
||||
...(ok
|
||||
? {}
|
||||
: {
|
||||
error:
|
||||
toolError ??
|
||||
(toolDisabled || toolUnavailable
|
||||
? "memory_search returned disabled/unavailable"
|
||||
: "memory_search result payload missing or invalid"),
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
export async function invokeMemorySearch({ port, token, timeoutMs }) {
|
||||
const resolvedTimeoutMs = clampTimerTimeoutMs(timeoutMs);
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), resolvedTimeoutMs);
|
||||
const startedAt = Date.now();
|
||||
try {
|
||||
const res = await fetch(`http://127.0.0.1:${port}/tools/invoke`, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
authorization: `Bearer ${token}`,
|
||||
"content-type": "application/json",
|
||||
},
|
||||
body: JSON.stringify({
|
||||
tool: "memory_search",
|
||||
args: {
|
||||
query: MEMORY_SEARCH_PROBE_QUERY,
|
||||
maxResults: 1,
|
||||
corpus: "memory",
|
||||
},
|
||||
sessionKey: "main",
|
||||
}),
|
||||
signal: controller.signal,
|
||||
});
|
||||
const text = await readBoundedResponseText(
|
||||
res,
|
||||
"memory_search",
|
||||
MEMORY_SEARCH_RESPONSE_MAX_BYTES,
|
||||
);
|
||||
const result = classifyMemorySearchInvokeResponse({
|
||||
httpOk: res.ok,
|
||||
status: res.status,
|
||||
bodyText: text,
|
||||
});
|
||||
return {
|
||||
...result,
|
||||
durationMs: Date.now() - startedAt,
|
||||
bodyPreview: text.slice(0, 500),
|
||||
};
|
||||
} catch (error) {
|
||||
return {
|
||||
ok: false,
|
||||
aborted: error?.name === "AbortError",
|
||||
durationMs: Date.now() - startedAt,
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
};
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
|
||||
function formatFailure({ invokePassed, options, peak }) {
|
||||
if (options.mode === "fixed" && !invokePassed) {
|
||||
return `memory_search did not complete successfully; see summary invoke details`;
|
||||
}
|
||||
if (options.mode === "fixed") {
|
||||
return `workspace Markdown REG FDs peaked at ${peak}, above max ${options.maxWorkspaceRegFds}`;
|
||||
}
|
||||
if (options.mode === "leak") {
|
||||
return `workspace Markdown REG FDs peaked at ${peak}, below leak threshold ${options.minLeakedFds}`;
|
||||
}
|
||||
return "";
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const options = parseArgs(process.argv.slice(2));
|
||||
if (process.platform !== "darwin" && !options.allowNonDarwin) {
|
||||
console.log(
|
||||
`[memory-fd-repro] skipped: lsof REG watcher counts are macOS-focused; pass --allow-non-darwin to run on ${process.platform}`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
const lsofAvailable = spawnSync("lsof", ["-v"], { stdio: "ignore" }).status === 0;
|
||||
if (!lsofAvailable) {
|
||||
throw new Error("lsof is required for memory FD repro instrumentation");
|
||||
}
|
||||
|
||||
const rootDir = fs.mkdtempSync(path.join(os.tmpdir(), "openclaw-memory-fd-repro-"));
|
||||
const homeDir = path.join(rootDir, "home");
|
||||
const workspaceDir = path.join(rootDir, "workspace");
|
||||
fs.mkdirSync(options.outputDir, { recursive: true });
|
||||
|
||||
const port = await getFreePort();
|
||||
const token = `memory-fd-repro-${process.pid}`;
|
||||
writeSyntheticWorkspace(workspaceDir, options.fileCount);
|
||||
const configPath = writeConfig({ homeDir, workspaceDir, port, token });
|
||||
const workspaceRealPath = fs.realpathSync.native(workspaceDir);
|
||||
const logPath = path.join(options.outputDir, "gateway.log");
|
||||
|
||||
const env = {
|
||||
...process.env,
|
||||
...SKIP_GATEWAY_ENV,
|
||||
HOME: homeDir,
|
||||
OPENCLAW_STATE_DIR: path.join(homeDir, ".openclaw"),
|
||||
OPENCLAW_CONFIG_PATH: configPath,
|
||||
OPENCLAW_GATEWAY_TOKEN: token,
|
||||
};
|
||||
let child;
|
||||
|
||||
const summary = {
|
||||
generatedAt: new Date().toISOString(),
|
||||
platform: process.platform,
|
||||
mode: options.mode,
|
||||
fileCount: options.fileCount,
|
||||
expectedMarkdownFiles: options.fileCount + 1,
|
||||
thresholds: {
|
||||
maxWorkspaceRegFds: options.maxWorkspaceRegFds,
|
||||
minLeakedFds: options.minLeakedFds,
|
||||
},
|
||||
rootDir,
|
||||
outputDir: options.outputDir,
|
||||
samples: [],
|
||||
invoke: null,
|
||||
};
|
||||
|
||||
try {
|
||||
preindexSyntheticMemory(env);
|
||||
child = spawn(
|
||||
process.execPath,
|
||||
[
|
||||
"scripts/run-node.mjs",
|
||||
"gateway",
|
||||
"run",
|
||||
"--port",
|
||||
String(port),
|
||||
"--auth",
|
||||
"token",
|
||||
"--token",
|
||||
token,
|
||||
"--bind",
|
||||
"loopback",
|
||||
"--allow-unconfigured",
|
||||
],
|
||||
{ cwd: process.cwd(), env, stdio: ["ignore", "pipe", "pipe"] },
|
||||
);
|
||||
logStep(`workspace=${workspaceDir}`);
|
||||
logStep(`files=${options.fileCount} mode=${options.mode} port=${port}`);
|
||||
await waitForGatewayReady({ child, port, logPath, timeoutMs: 60_000 });
|
||||
const pid = findGatewayPid(port);
|
||||
if (!pid) {
|
||||
throw new Error("gateway listener pid not found after ready");
|
||||
}
|
||||
summary.gatewayPid = pid;
|
||||
summary.samples.push(sampleFds({ label: "baseline", pid, workspaceRealPath }));
|
||||
|
||||
const invokePromise = invokeMemorySearch({ port, token, timeoutMs: options.invokeTimeoutMs });
|
||||
await sleep(options.sampleDelayMs);
|
||||
summary.samples.push(sampleFds({ label: "during", pid, workspaceRealPath }));
|
||||
summary.invoke = await invokePromise;
|
||||
logStep(`invoke=${JSON.stringify(summary.invoke)}`);
|
||||
await sleep(options.settleDelayMs);
|
||||
summary.samples.push(sampleFds({ label: "settled", pid, workspaceRealPath }));
|
||||
|
||||
const peak = Math.max(...summary.samples.map((sample) => sample.uniqueWorkspaceMarkdownRegFds));
|
||||
summary.peakUniqueWorkspaceMarkdownRegFds = peak;
|
||||
const invokePassed = Boolean(summary.invoke?.ok);
|
||||
const passed =
|
||||
options.mode === "report" ||
|
||||
(options.mode === "fixed" && invokePassed && peak <= options.maxWorkspaceRegFds) ||
|
||||
(options.mode === "leak" && peak >= options.minLeakedFds);
|
||||
summary.passed = passed;
|
||||
summary.failure = passed ? undefined : formatFailure({ invokePassed, options, peak });
|
||||
|
||||
fs.writeFileSync(
|
||||
path.join(options.outputDir, "summary.json"),
|
||||
`${JSON.stringify(summary, null, 2)}\n`,
|
||||
);
|
||||
logStep(`summary=${path.join(options.outputDir, "summary.json")}`);
|
||||
if (!passed) {
|
||||
throw new Error(summary.failure);
|
||||
}
|
||||
} finally {
|
||||
if (child) {
|
||||
await stopGateway({ child, port });
|
||||
}
|
||||
if (!options.keep) {
|
||||
fs.rmSync(rootDir, { recursive: true, force: true, maxRetries: 10, retryDelay: 50 });
|
||||
} else {
|
||||
logStep(`kept synthetic root=${rootDir}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function isMainModule() {
|
||||
const entrypoint = process.argv[1];
|
||||
return Boolean(entrypoint && import.meta.url === pathToFileURL(path.resolve(entrypoint)).href);
|
||||
}
|
||||
|
||||
if (isMainModule()) {
|
||||
main().catch(
|
||||
/** @param {unknown} error */ (error) => {
|
||||
console.error(
|
||||
`[memory-fd-repro] failed: ${error instanceof Error ? error.message : String(error)}`,
|
||||
);
|
||||
process.exit(1);
|
||||
},
|
||||
);
|
||||
}
|
||||
135
scripts/check-no-conflict-markers.mjs
Normal file
135
scripts/check-no-conflict-markers.mjs
Normal file
@@ -0,0 +1,135 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Rejects unresolved merge conflict markers in tracked files.
|
||||
import { execFileSync, spawnSync } from "node:child_process";
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const CONFLICT_MARKER_GREP_PATTERN = "^(<<<<<<< |\\|\\|\\|\\|\\|\\|\\| |=======$|>>>>>>> )";
|
||||
|
||||
function isBinaryBuffer(buffer) {
|
||||
return buffer.includes(0);
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns one-based line numbers containing merge conflict markers.
|
||||
*/
|
||||
export function findConflictMarkerLines(content) {
|
||||
const lines = content.split(/\r?\n/u);
|
||||
const matches = [];
|
||||
for (const [index, line] of lines.entries()) {
|
||||
if (
|
||||
line.startsWith("<<<<<<< ") ||
|
||||
line.startsWith("||||||| ") ||
|
||||
line === "=======" ||
|
||||
line.startsWith(">>>>>>> ")
|
||||
) {
|
||||
matches.push(index + 1);
|
||||
}
|
||||
}
|
||||
return matches;
|
||||
}
|
||||
|
||||
/**
|
||||
* Lists tracked files in the repository.
|
||||
*/
|
||||
export function listTrackedFiles(cwd = process.cwd()) {
|
||||
const output = execFileSync("git", ["ls-files", "-z"], {
|
||||
cwd,
|
||||
encoding: "utf8",
|
||||
});
|
||||
return output
|
||||
.split("\0")
|
||||
.filter(Boolean)
|
||||
.map((relativePath) => path.join(cwd, relativePath));
|
||||
}
|
||||
|
||||
/**
|
||||
* Scans files for merge conflict markers, skipping binary content.
|
||||
*/
|
||||
export function findConflictMarkersInFiles(filePaths, readFile = fs.readFileSync) {
|
||||
const violations = [];
|
||||
for (const filePath of filePaths) {
|
||||
let content;
|
||||
try {
|
||||
content = readFile(filePath);
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
if (!Buffer.isBuffer(content)) {
|
||||
content = Buffer.from(String(content));
|
||||
}
|
||||
if (isBinaryBuffer(content)) {
|
||||
continue;
|
||||
}
|
||||
const lines = findConflictMarkerLines(content.toString("utf8"));
|
||||
if (lines.length > 0) {
|
||||
violations.push({
|
||||
filePath,
|
||||
lines,
|
||||
});
|
||||
}
|
||||
}
|
||||
return violations;
|
||||
}
|
||||
|
||||
/**
|
||||
* Uses git grep to list tracked files that may contain conflict markers.
|
||||
*/
|
||||
export function listTrackedFilesWithConflictMarkerCandidates(cwd = process.cwd(), run = spawnSync) {
|
||||
const result = run(
|
||||
"git",
|
||||
["grep", "-l", "-z", "-I", "-E", CONFLICT_MARKER_GREP_PATTERN, "--", "."],
|
||||
{
|
||||
cwd,
|
||||
encoding: "buffer",
|
||||
},
|
||||
);
|
||||
if (result.status === 1) {
|
||||
return [];
|
||||
}
|
||||
if (result.status !== 0) {
|
||||
const stderr = result.stderr?.toString("utf8").trim();
|
||||
throw new Error(stderr || `git grep failed with status ${result.status ?? "unknown"}`);
|
||||
}
|
||||
return result.stdout
|
||||
.toString("utf8")
|
||||
.split("\0")
|
||||
.filter(Boolean)
|
||||
.map((relativePath) => path.join(cwd, relativePath));
|
||||
}
|
||||
|
||||
/**
|
||||
* Finds merge conflict markers in tracked repository files.
|
||||
*/
|
||||
export function findConflictMarkersInTrackedFiles(cwd = process.cwd()) {
|
||||
return findConflictMarkersInFiles(listTrackedFilesWithConflictMarkerCandidates(cwd));
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the merge conflict marker check.
|
||||
*/
|
||||
export async function main() {
|
||||
const cwd = process.cwd();
|
||||
const violations = findConflictMarkersInTrackedFiles(cwd);
|
||||
if (violations.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
console.error("Found unresolved merge conflict markers:");
|
||||
for (const violation of violations) {
|
||||
const relativePath = path.relative(cwd, violation.filePath) || violation.filePath;
|
||||
console.error(`- ${relativePath}:${violation.lines.join(",")}`);
|
||||
}
|
||||
process.exitCode = 1;
|
||||
}
|
||||
|
||||
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
||||
main().catch(
|
||||
/** @param {unknown} error */ (error) => {
|
||||
console.error(error);
|
||||
process.exit(1);
|
||||
},
|
||||
);
|
||||
}
|
||||
112
scripts/check-no-deprecated-channel-access.ts
Normal file
112
scripts/check-no-deprecated-channel-access.ts
Normal file
@@ -0,0 +1,112 @@
|
||||
// Check No Deprecated Channel Access script supports OpenClaw repository automation.
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { collectFilesSync, isCodeFile, relativeToCwd } from "./check-file-utils.js";
|
||||
import { classifyBundledExtensionSourcePath } from "./lib/extension-source-classifier.mjs";
|
||||
|
||||
type Rule = {
|
||||
label: string;
|
||||
pattern: RegExp;
|
||||
};
|
||||
|
||||
const RULES: Rule[] = [
|
||||
{
|
||||
label: "deprecated channel ingress resolver aliases",
|
||||
pattern:
|
||||
/\b(?:resolved|result|directResolved|groupResolved)\.(?:legacyAccess|senderReasonCode|commandAuthorized|shouldBlockControlCommand)\b/u,
|
||||
},
|
||||
{
|
||||
label: "inline deprecated channel ingress legacyAccess projection",
|
||||
pattern: /\)\.legacyAccess\b/u,
|
||||
},
|
||||
{
|
||||
label: "low-level compatibility ingress resolver",
|
||||
pattern: /\bresolveChannelIngressAccess\b/u,
|
||||
},
|
||||
{
|
||||
label: "deprecated channel ingress compatibility projection",
|
||||
pattern:
|
||||
/\b(?:findChannelIngressSenderReasonCode|formatChannelIngressPolicyReason|mapChannelIngressReasonCodeToDmGroupAccessReason|projectChannelIngressDmGroupAccess|projectChannelIngressSenderGroupAccess)\b/u,
|
||||
},
|
||||
{
|
||||
label: "deprecated pairing-store access helper",
|
||||
pattern: /\breadStoreAllowFromForDmPolicy\b/u,
|
||||
},
|
||||
{
|
||||
label: "deprecated DM/group access helper",
|
||||
pattern: /\bresolveDmGroupAccessWith(?:Lists|CommandGate)\b/u,
|
||||
},
|
||||
{
|
||||
label: "deprecated DM/group access reason constants",
|
||||
pattern: /\bDM_GROUP_ACCESS_REASON\b/u,
|
||||
},
|
||||
{
|
||||
label: "deprecated group policy access helper",
|
||||
pattern:
|
||||
/\b(?:resolveSenderScopedGroupPolicy|evaluateSenderGroupAccess(?:ForPolicy)?|evaluateGroupRouteAccessForPolicy|evaluateMatchedGroupAccessForPolicy)\b/u,
|
||||
},
|
||||
{
|
||||
label: "deprecated group access compatibility module",
|
||||
pattern: /from\s+["']openclaw\/plugin-sdk\/group-access["']/u,
|
||||
},
|
||||
{
|
||||
label: "deprecated command authorization helper",
|
||||
pattern: /\bresolveSenderCommandAuthorization(?:WithRuntime)?\b/u,
|
||||
},
|
||||
{
|
||||
label: "deprecated command auth SDK facade",
|
||||
pattern: /from\s+["']openclaw\/plugin-sdk\/command-auth["']/u,
|
||||
},
|
||||
{
|
||||
label: "deprecated AccessFacts command authorizers",
|
||||
pattern: /\bcommands\.authorizers\b/u,
|
||||
},
|
||||
];
|
||||
|
||||
function collectBundledPluginProductionFiles(): string[] {
|
||||
const extensionsDir = path.join(process.cwd(), "extensions");
|
||||
return collectFilesSync(extensionsDir, {
|
||||
includeFile(filePath) {
|
||||
if (!isCodeFile(filePath)) {
|
||||
return false;
|
||||
}
|
||||
const repoPath = relativeToCwd(filePath);
|
||||
const classified = classifyBundledExtensionSourcePath(repoPath);
|
||||
return classified.isProductionSource;
|
||||
},
|
||||
}).toSorted((left, right) => relativeToCwd(left).localeCompare(relativeToCwd(right)));
|
||||
}
|
||||
|
||||
function main() {
|
||||
const offenders: Array<{ file: string; line: number; label: string; text: string }> = [];
|
||||
for (const file of collectBundledPluginProductionFiles()) {
|
||||
const content = fs.readFileSync(file, "utf8");
|
||||
const lines = content.split(/\r?\n/u);
|
||||
for (const [index, line] of lines.entries()) {
|
||||
for (const rule of RULES) {
|
||||
if (rule.pattern.test(line)) {
|
||||
offenders.push({
|
||||
file: relativeToCwd(file),
|
||||
line: index + 1,
|
||||
label: rule.label,
|
||||
text: line.trim(),
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (offenders.length > 0) {
|
||||
console.error(
|
||||
"Bundled plugin production code must use modern channel access results, not deprecated compatibility seams.",
|
||||
);
|
||||
for (const offender of offenders) {
|
||||
console.error(`- ${offender.file}:${offender.line}: ${offender.label}: ${offender.text}`);
|
||||
}
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log("OK: bundled plugin production code avoids deprecated channel access seams.");
|
||||
}
|
||||
|
||||
main();
|
||||
44
scripts/check-no-extension-src-imports.ts
Normal file
44
scripts/check-no-extension-src-imports.ts
Normal file
@@ -0,0 +1,44 @@
|
||||
// Check No Extension Src Imports script supports OpenClaw repository automation.
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { collectFilesSync, isCodeFile, relativeToCwd } from "./check-file-utils.js";
|
||||
import { classifyBundledExtensionSourcePath } from "./lib/extension-source-classifier.mjs";
|
||||
|
||||
const FORBIDDEN_REPO_SRC_IMPORT = /["'](?:\.\.\/)+(?:src\/)[^"']+["']/;
|
||||
|
||||
function collectExtensionSourceFiles(rootDir: string): string[] {
|
||||
return collectFilesSync(rootDir, {
|
||||
includeFile: (filePath) =>
|
||||
isCodeFile(filePath) && classifyBundledExtensionSourcePath(filePath).isProductionSource,
|
||||
});
|
||||
}
|
||||
|
||||
function main() {
|
||||
const extensionsDir = path.join(process.cwd(), "extensions");
|
||||
const files = collectExtensionSourceFiles(extensionsDir);
|
||||
const offenders: string[] = [];
|
||||
|
||||
for (const file of files) {
|
||||
const content = fs.readFileSync(file, "utf8");
|
||||
if (FORBIDDEN_REPO_SRC_IMPORT.test(content)) {
|
||||
offenders.push(file);
|
||||
}
|
||||
}
|
||||
|
||||
if (offenders.length > 0) {
|
||||
console.error("Production extension files must not import the repo src/ tree directly.");
|
||||
for (const offender of offenders.toSorted()) {
|
||||
console.error(`- ${relativeToCwd(offender)}`);
|
||||
}
|
||||
console.error(
|
||||
"Publish a focused openclaw/plugin-sdk/<subpath> surface or use the extension's own public barrel instead.",
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log(
|
||||
`OK: production extension files avoid direct repo src/ imports (${files.length} checked).`,
|
||||
);
|
||||
}
|
||||
|
||||
main();
|
||||
360
scripts/check-no-extension-test-core-imports.ts
Normal file
360
scripts/check-no-extension-test-core-imports.ts
Normal file
@@ -0,0 +1,360 @@
|
||||
// Check No Extension Test Core Imports script supports OpenClaw repository automation.
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { collectFilesSync, isCodeFile, relativeToCwd } from "./check-file-utils.js";
|
||||
|
||||
type Offender = { file: string; hint: string; line?: number; specifier?: string };
|
||||
|
||||
const FORBIDDEN_PATTERNS: Array<{ pattern: RegExp; hint: string }> = [
|
||||
{
|
||||
pattern: /["']openclaw\/plugin-sdk["']/,
|
||||
hint: "Use openclaw/plugin-sdk/<subpath> instead of the monolithic root entry.",
|
||||
},
|
||||
{
|
||||
pattern: /["']openclaw\/plugin-sdk\/test-utils["']/,
|
||||
hint: "Use a focused plugin-sdk test subpath for the public extension test surface.",
|
||||
},
|
||||
{
|
||||
pattern: /["']openclaw\/plugin-sdk\/testing["']/,
|
||||
hint: "Use a focused plugin-sdk test subpath instead of the broad compatibility testing barrel.",
|
||||
},
|
||||
{
|
||||
pattern: /["']openclaw\/plugin-sdk\/compat["']/,
|
||||
hint: "Use a focused public plugin-sdk subpath instead of compat.",
|
||||
},
|
||||
{
|
||||
pattern: /["'](?:\.\.\/)+(?:test-utils\/)[^"']+["']/,
|
||||
hint: "Use a documented openclaw/plugin-sdk test subpath for bundled extension test helpers.",
|
||||
},
|
||||
{
|
||||
pattern: /["'](?:\.\.\/)+(?:test\/helpers\/plugins\/)[^"']+["']/,
|
||||
hint: "Use a documented openclaw/plugin-sdk test subpath instead of repo-only plugin helper bridges.",
|
||||
},
|
||||
{
|
||||
pattern: /["'](?:\.\.\/)+(?:test\/helpers\/channels\/)[^"']+["']/,
|
||||
hint: "Use openclaw/plugin-sdk/channel-test-helpers or another focused SDK test subpath instead of repo-only channel helper bridges.",
|
||||
},
|
||||
{
|
||||
pattern: /["'](?:\.\.\/)+(?:test\/helpers\/media-generation\/)[^"']+["']/,
|
||||
hint: "Use openclaw/plugin-sdk/provider-test-contracts or openclaw/plugin-sdk/provider-http-test-mocks instead of repo-only media provider helper bridges.",
|
||||
},
|
||||
{
|
||||
pattern:
|
||||
/["'](?:\.\.\/)+(?:test\/helpers\/(?:bundled-channel-entry|envelope-timestamp|pairing-reply)\.(?:js|ts))["']/,
|
||||
hint: "Use openclaw/plugin-sdk/channel-test-helpers instead of repo-only channel test helper bridges.",
|
||||
},
|
||||
{
|
||||
pattern:
|
||||
/["'](?:\.\.\/)+(?:test\/helpers\/(?:http-test-server|mock-incoming-request|temp-home)\.(?:js|ts))["']/,
|
||||
hint: "Use openclaw/plugin-sdk/test-env instead of repo-only environment/network test helper bridges.",
|
||||
},
|
||||
{
|
||||
pattern:
|
||||
/["'](?:\.\.\/)+(?:test\/helpers\/(?:bundled-plugin-paths|import-fresh|node-builtin-mocks)\.(?:js|ts))["']/,
|
||||
hint: "Use openclaw/plugin-sdk/test-fixtures instead of repo-only generic test helper bridges.",
|
||||
},
|
||||
{
|
||||
pattern:
|
||||
/["'](?:\.\.\/)+(?:test\/helpers\/(?:provider-replay-policy|stt-live-audio)\.(?:js|ts))["']/,
|
||||
hint: "Use openclaw/plugin-sdk/provider-test-contracts instead of repo-only provider test helper bridges.",
|
||||
},
|
||||
{
|
||||
pattern: /["'](?:\.\.\/)+(?:test\/helpers\/)[^"']+["']/,
|
||||
hint: "Use a documented openclaw/plugin-sdk test subpath instead of repo-only test helper bridges.",
|
||||
},
|
||||
{
|
||||
pattern: /["'](?:\.\.\/)+(?:src\/channels\/plugins\/contracts\/test-helpers\/)[^"']+["']/,
|
||||
hint: "Use openclaw/plugin-sdk/channel-test-helpers or another focused SDK test subpath instead of core-only channel contract helpers.",
|
||||
},
|
||||
{
|
||||
pattern: /["'](?:\.\.\/)+(?:src\/test-utils\/)[^"']+["']/,
|
||||
hint: "Use a documented openclaw/plugin-sdk test subpath for public surfaces.",
|
||||
},
|
||||
{
|
||||
pattern: /["'](?:\.\.\/)+(?:src\/plugins\/types\.js)["']/,
|
||||
hint: "Use public plugin-sdk/core types or documented plugin-sdk test helpers instead.",
|
||||
},
|
||||
{
|
||||
pattern: /["'](?:\.\.\/)+(?:src\/channels\/plugins\/contracts\/test-helpers\.js)["']/,
|
||||
hint: "Use openclaw/plugin-sdk/channel-contract-testing for channel contract test helpers.",
|
||||
},
|
||||
];
|
||||
|
||||
const STATIC_RELATIVE_MODULE_PATTERN = /\b(?:import|export)\b[\s\S]*?\bfrom\s*["']([^"']+)["']/g;
|
||||
const DYNAMIC_RELATIVE_MODULE_PATTERN = /\bimport\s*\(\s*["']([^"']+)["']\s*\)/g;
|
||||
const MOCK_RELATIVE_MODULE_PATTERN =
|
||||
/\bvi\.(?:mock|doMock|unmock|doUnmock)\s*\(\s*["']([^"']+)["']/g;
|
||||
|
||||
const RELATIVE_CORE_HINT =
|
||||
"Use a focused plugin-sdk test/runtime subpath instead of core internals.";
|
||||
const ROOT_TEST_SUPPORT_LOCAL_SRC_HINT =
|
||||
"Move this helper under the extension's src/test-support tree or expose a narrow test-api/runtime-api surface instead of reaching into private src from package-root test-support.";
|
||||
|
||||
// Tombstones for retired repo-only plugin helper bridge files. Keep this list so
|
||||
// deleted bridges fail loudly if they are recreated instead of using SDK subpaths.
|
||||
const RETIRED_EXTENSION_TEST_HELPER_BRIDGE_FILES = [
|
||||
"test/helpers/plugins/env.ts",
|
||||
"test/helpers/plugins/fetch-mock.ts",
|
||||
"test/helpers/plugins/frozen-time.ts",
|
||||
"test/helpers/plugins/media-understanding.ts",
|
||||
"test/helpers/plugins/mock-http-response.ts",
|
||||
"test/helpers/plugins/contracts-testkit.ts",
|
||||
"test/helpers/plugins/direct-smoke.ts",
|
||||
"test/helpers/plugins/directory.ts",
|
||||
"test/helpers/plugins/onboard-config.ts",
|
||||
"test/helpers/plugins/outbound-delivery.ts",
|
||||
"test/helpers/plugins/package-manifest-contract.ts",
|
||||
"test/helpers/plugins/plugin-api.ts",
|
||||
"test/helpers/plugins/plugin-registration-contract-cases.ts",
|
||||
"test/helpers/plugins/plugin-registration-contract.ts",
|
||||
"test/helpers/plugins/plugin-registration.ts",
|
||||
"test/helpers/plugins/plugin-runtime-mock.ts",
|
||||
"test/helpers/plugins/plugin-registry.ts",
|
||||
"test/helpers/plugins/provider-auth-contract.ts",
|
||||
"test/helpers/plugins/provider-catalog.ts",
|
||||
"test/helpers/plugins/provider-contract-suites.ts",
|
||||
"test/helpers/plugins/provider-contract.ts",
|
||||
"test/helpers/plugins/provider-discovery-contract.ts",
|
||||
"test/helpers/plugins/provider-onboard.ts",
|
||||
"test/helpers/plugins/provider-registration.ts",
|
||||
"test/helpers/plugins/provider-runtime-contract.ts",
|
||||
"test/helpers/plugins/provider-usage-fetch.ts",
|
||||
"test/helpers/plugins/provider-wizard-contract-suites.ts",
|
||||
"test/helpers/plugins/public-artifacts.ts",
|
||||
"test/helpers/plugins/public-surface-loader.ts",
|
||||
"test/helpers/plugins/runtime-taskflow.ts",
|
||||
"test/helpers/plugins/runtime-env.ts",
|
||||
"test/helpers/plugins/send-config.ts",
|
||||
"test/helpers/plugins/setup-wizard.ts",
|
||||
"test/helpers/plugins/start-account-context.ts",
|
||||
"test/helpers/plugins/start-account-lifecycle.ts",
|
||||
"test/helpers/plugins/status-issues.ts",
|
||||
"test/helpers/plugins/stream-hooks.ts",
|
||||
"test/helpers/plugins/subagent-hooks.ts",
|
||||
"test/helpers/plugins/temp-dir.ts",
|
||||
"test/helpers/plugins/temp-home.ts",
|
||||
"test/helpers/plugins/tts-contract-suites.ts",
|
||||
"test/helpers/plugins/typed-cases.ts",
|
||||
"test/helpers/plugins/web-fetch-provider-contract.ts",
|
||||
"test/helpers/plugins/web-search-provider-contract.ts",
|
||||
"test/helpers/media-generation/dashscope-video-provider.ts",
|
||||
"test/helpers/media-generation/provider-capability-assertions.ts",
|
||||
"test/helpers/media-generation/provider-http-mocks.ts",
|
||||
"test/helpers/bundled-channel-entry.ts",
|
||||
"test/helpers/bundled-plugin-paths.ts",
|
||||
"test/helpers/envelope-timestamp.ts",
|
||||
"test/helpers/http-test-server.ts",
|
||||
"test/helpers/import-fresh.ts",
|
||||
"test/helpers/mock-incoming-request.ts",
|
||||
"test/helpers/node-builtin-mocks.ts",
|
||||
"test/helpers/pairing-reply.ts",
|
||||
"test/helpers/provider-replay-policy.ts",
|
||||
"test/helpers/stt-live-audio.ts",
|
||||
"test/helpers/temp-home.ts",
|
||||
"test/helpers/agents/auth-profile-runtime-contract.ts",
|
||||
"test/helpers/agents/delivery-no-reply-runtime-contract.ts",
|
||||
"test/helpers/agents/openclaw-owned-tool-runtime-contract.ts",
|
||||
"test/helpers/agents/outcome-fallback-runtime-contract.ts",
|
||||
"test/helpers/agents/prompt-overlay-runtime-contract.ts",
|
||||
"test/helpers/agents/schema-normalization-runtime-contract.ts",
|
||||
"test/helpers/agents/transcript-repair-runtime-contract.ts",
|
||||
"test/helpers/sandbox-fixtures.ts",
|
||||
];
|
||||
|
||||
function isExtensionTestFile(filePath: string): boolean {
|
||||
return /\.test\.[cm]?[jt]sx?$/u.test(filePath) || /\.e2e\.test\.[cm]?[jt]sx?$/u.test(filePath);
|
||||
}
|
||||
|
||||
function isExtensionTestSupportFile(filePath: string): boolean {
|
||||
return (
|
||||
(/(?:^|[/\\])test-support(?:[/\\]|$)/u.test(filePath) ||
|
||||
/(?:\.|-|_)test-support\.[cm]?[jt]sx?$/u.test(filePath)) &&
|
||||
/\.[cm]?[jt]sx?$/u.test(filePath)
|
||||
);
|
||||
}
|
||||
|
||||
function collectExtensionTestFiles(rootDir: string): string[] {
|
||||
return collectFilesSync(rootDir, {
|
||||
includeFile: (filePath) =>
|
||||
isExtensionTestFile(filePath) || isExtensionTestSupportFile(filePath),
|
||||
});
|
||||
}
|
||||
|
||||
function collectPluginHelperFiles(rootDir: string): string[] {
|
||||
return collectFilesSync(rootDir, {
|
||||
includeFile: isCodeFile,
|
||||
});
|
||||
}
|
||||
|
||||
function lineNumberForOffset(content: string, offset: number): number {
|
||||
let line = 1;
|
||||
for (let index = 0; index < offset; index += 1) {
|
||||
if (content.charCodeAt(index) === 10) {
|
||||
line += 1;
|
||||
}
|
||||
}
|
||||
return line;
|
||||
}
|
||||
|
||||
function resolvesToRepoSrc(filePath: string, specifier: string): boolean {
|
||||
if (!specifier.startsWith(".")) {
|
||||
return false;
|
||||
}
|
||||
const resolved = path.resolve(path.dirname(filePath), specifier);
|
||||
const repoRelative = path.relative(process.cwd(), resolved).replaceAll(path.sep, "/");
|
||||
return repoRelative === "src" || repoRelative.startsWith("src/");
|
||||
}
|
||||
|
||||
function getExtensionRootForFile(filePath: string): string | undefined {
|
||||
const relativePath = path.relative(process.cwd(), filePath).replaceAll(path.sep, "/");
|
||||
const match = /^extensions\/[^/]+(?:\/|$)/u.exec(relativePath);
|
||||
return match ? path.resolve(process.cwd(), match[0]) : undefined;
|
||||
}
|
||||
|
||||
function isRootExtensionTestSupportFile(filePath: string): boolean {
|
||||
const relativePath = path.relative(process.cwd(), filePath).replaceAll(path.sep, "/");
|
||||
return /^extensions\/[^/]+\/test-support(?:\.[cm]?[jt]sx?|\/)/u.test(relativePath);
|
||||
}
|
||||
|
||||
function resolvesToExtensionLocalSrc(filePath: string, specifier: string): boolean {
|
||||
if (!specifier.startsWith(".")) {
|
||||
return false;
|
||||
}
|
||||
const extensionRoot = getExtensionRootForFile(filePath);
|
||||
if (!extensionRoot) {
|
||||
return false;
|
||||
}
|
||||
const resolved = path.resolve(path.dirname(filePath), specifier);
|
||||
const localSrc = path.join(extensionRoot, "src");
|
||||
return resolved === localSrc || resolved.startsWith(`${localSrc}${path.sep}`);
|
||||
}
|
||||
|
||||
function collectRelativeCoreImportOffenders(
|
||||
filePath: string,
|
||||
content: string,
|
||||
opts: { includeDynamic: boolean },
|
||||
): Offender[] {
|
||||
const offenders: Offender[] = [];
|
||||
const matches = [
|
||||
...content.matchAll(STATIC_RELATIVE_MODULE_PATTERN),
|
||||
...(opts.includeDynamic ? [...content.matchAll(DYNAMIC_RELATIVE_MODULE_PATTERN)] : []),
|
||||
...content.matchAll(MOCK_RELATIVE_MODULE_PATTERN),
|
||||
];
|
||||
for (const match of matches) {
|
||||
const specifier = match[1];
|
||||
if (!specifier || !resolvesToRepoSrc(filePath, specifier)) {
|
||||
continue;
|
||||
}
|
||||
offenders.push({
|
||||
file: filePath,
|
||||
hint: RELATIVE_CORE_HINT,
|
||||
line: lineNumberForOffset(content, match.index ?? 0),
|
||||
specifier,
|
||||
});
|
||||
}
|
||||
return offenders;
|
||||
}
|
||||
|
||||
function collectRootTestSupportLocalSrcImportOffenders(
|
||||
filePath: string,
|
||||
content: string,
|
||||
): Offender[] {
|
||||
if (!isRootExtensionTestSupportFile(filePath)) {
|
||||
return [];
|
||||
}
|
||||
const offenders: Offender[] = [];
|
||||
const matches = [
|
||||
...content.matchAll(STATIC_RELATIVE_MODULE_PATTERN),
|
||||
...content.matchAll(DYNAMIC_RELATIVE_MODULE_PATTERN),
|
||||
...content.matchAll(MOCK_RELATIVE_MODULE_PATTERN),
|
||||
];
|
||||
for (const match of matches) {
|
||||
const specifier = match[1];
|
||||
if (!specifier || !resolvesToExtensionLocalSrc(filePath, specifier)) {
|
||||
continue;
|
||||
}
|
||||
offenders.push({
|
||||
file: filePath,
|
||||
hint: ROOT_TEST_SUPPORT_LOCAL_SRC_HINT,
|
||||
line: lineNumberForOffset(content, match.index ?? 0),
|
||||
specifier,
|
||||
});
|
||||
}
|
||||
return offenders;
|
||||
}
|
||||
|
||||
function main() {
|
||||
const extensionsDir = path.join(process.cwd(), "extensions");
|
||||
const pluginHelpersDir = path.join(process.cwd(), "test/helpers/plugins");
|
||||
const retiredChannelHelpersDir = path.join(process.cwd(), "test/helpers/channels");
|
||||
const files = collectExtensionTestFiles(extensionsDir);
|
||||
const pluginHelperFiles = collectPluginHelperFiles(pluginHelpersDir);
|
||||
const retiredChannelHelperFiles = fs.existsSync(retiredChannelHelpersDir)
|
||||
? collectFilesSync(retiredChannelHelpersDir, { includeFile: isCodeFile })
|
||||
: [];
|
||||
const offenders: Offender[] = [];
|
||||
|
||||
for (const file of retiredChannelHelperFiles) {
|
||||
offenders.push({
|
||||
file,
|
||||
hint: "Keep core channel contract helpers under src/channels/plugins/contracts/test-helpers and public plugin helpers under focused openclaw/plugin-sdk test subpaths.",
|
||||
});
|
||||
}
|
||||
|
||||
for (const file of RETIRED_EXTENSION_TEST_HELPER_BRIDGE_FILES) {
|
||||
const filePath = path.join(process.cwd(), file);
|
||||
if (!fs.existsSync(filePath)) {
|
||||
continue;
|
||||
}
|
||||
offenders.push({
|
||||
file: filePath,
|
||||
hint: "Import the helper directly from a documented openclaw/plugin-sdk testing subpath instead of recreating this bridge.",
|
||||
});
|
||||
}
|
||||
|
||||
for (const file of files) {
|
||||
const content = fs.readFileSync(file, "utf8");
|
||||
for (const rule of FORBIDDEN_PATTERNS) {
|
||||
if (!rule.pattern.test(content)) {
|
||||
continue;
|
||||
}
|
||||
offenders.push({ file, hint: rule.hint });
|
||||
break;
|
||||
}
|
||||
offenders.push(
|
||||
...collectRelativeCoreImportOffenders(file, content, {
|
||||
includeDynamic: true,
|
||||
}),
|
||||
);
|
||||
offenders.push(...collectRootTestSupportLocalSrcImportOffenders(file, content));
|
||||
}
|
||||
|
||||
for (const file of pluginHelperFiles) {
|
||||
const content = fs.readFileSync(file, "utf8");
|
||||
offenders.push(
|
||||
...collectRelativeCoreImportOffenders(file, content, {
|
||||
includeDynamic: true,
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
if (offenders.length > 0) {
|
||||
console.error(
|
||||
"Extension test files and plugin test helpers must stay on public plugin-sdk surfaces.",
|
||||
);
|
||||
for (const offender of offenders.toSorted((a, b) => a.file.localeCompare(b.file))) {
|
||||
const location = offender.line
|
||||
? `${relativeToCwd(offender.file)}:${offender.line}`
|
||||
: relativeToCwd(offender.file);
|
||||
const specifier = offender.specifier ? ` (${offender.specifier})` : "";
|
||||
console.error(`- ${location}${specifier}: ${offender.hint}`);
|
||||
}
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log(
|
||||
`OK: extension test files, support helpers, and plugin test helpers avoid direct core test/internal imports (${files.length} extension files, ${pluginHelperFiles.length} plugin helpers checked).`,
|
||||
);
|
||||
}
|
||||
|
||||
main();
|
||||
161
scripts/check-no-monolithic-plugin-sdk-entry-imports.ts
Normal file
161
scripts/check-no-monolithic-plugin-sdk-entry-imports.ts
Normal file
@@ -0,0 +1,161 @@
|
||||
// Check No Monolithic Plugin Sdk Entry Imports script supports OpenClaw repository automation.
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { discoverOpenClawPlugins } from "../src/plugins/discovery.js";
|
||||
import { collectFilesSync, isCodeFile, relativeToCwd } from "./check-file-utils.js";
|
||||
|
||||
// Match exact monolithic-root specifier in any code path:
|
||||
// imports/exports, require/dynamic import, and test mocks (vi.mock/jest.mock).
|
||||
const ROOT_IMPORT_PATTERN = /["']openclaw\/plugin-sdk["']/;
|
||||
const LEGACY_COMPAT_IMPORT_PATTERN = /["']openclaw\/plugin-sdk\/compat["']/;
|
||||
const LEGACY_BROAD_SUBPATH_PATTERNS = [
|
||||
{
|
||||
pattern: /["']openclaw\/plugin-sdk\/channel-runtime["']/,
|
||||
label: "openclaw/plugin-sdk/channel-runtime",
|
||||
},
|
||||
{
|
||||
pattern: /["']openclaw\/plugin-sdk\/config-runtime["']/,
|
||||
label: "openclaw/plugin-sdk/config-runtime",
|
||||
},
|
||||
{
|
||||
pattern: /["']openclaw\/plugin-sdk\/infra-runtime["']/,
|
||||
label: "openclaw/plugin-sdk/infra-runtime",
|
||||
},
|
||||
] as const;
|
||||
|
||||
function hasMonolithicRootImport(content: string): boolean {
|
||||
return ROOT_IMPORT_PATTERN.test(content);
|
||||
}
|
||||
|
||||
function hasLegacyCompatImport(content: string): boolean {
|
||||
return LEGACY_COMPAT_IMPORT_PATTERN.test(content);
|
||||
}
|
||||
|
||||
function findLegacyBroadSubpathImports(content: string): string[] {
|
||||
return LEGACY_BROAD_SUBPATH_PATTERNS.filter(({ pattern }) => pattern.test(content)).map(
|
||||
({ label }) => label,
|
||||
);
|
||||
}
|
||||
|
||||
function collectPluginSourceFiles(rootDir: string): string[] {
|
||||
const srcDir = path.join(rootDir, "src");
|
||||
if (!fs.existsSync(srcDir)) {
|
||||
return [];
|
||||
}
|
||||
return collectFilesSync(srcDir, {
|
||||
includeFile: (filePath) => isCodeFile(filePath),
|
||||
skipDirNames: new Set(["node_modules", "dist", ".git", "coverage"]),
|
||||
});
|
||||
}
|
||||
|
||||
function collectSharedExtensionSourceFiles(): string[] {
|
||||
return collectPluginSourceFiles(path.join(process.cwd(), "extensions", "shared"));
|
||||
}
|
||||
|
||||
function collectBundledExtensionSourceFiles(): string[] {
|
||||
const extensionsDir = path.join(process.cwd(), "extensions");
|
||||
let entries: fs.Dirent[];
|
||||
try {
|
||||
entries = fs.readdirSync(extensionsDir, { withFileTypes: true });
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
|
||||
const files: string[] = [];
|
||||
for (const entry of entries) {
|
||||
if (!entry.isDirectory() || entry.name === "shared") {
|
||||
continue;
|
||||
}
|
||||
for (const srcFile of collectPluginSourceFiles(path.join(extensionsDir, entry.name))) {
|
||||
files.push(srcFile);
|
||||
}
|
||||
}
|
||||
return files;
|
||||
}
|
||||
|
||||
function main() {
|
||||
const discovery = discoverOpenClawPlugins({});
|
||||
const bundledCandidates = discovery.candidates.filter((c) => c.origin === "bundled");
|
||||
const filesToCheck = new Set<string>();
|
||||
for (const candidate of bundledCandidates) {
|
||||
filesToCheck.add(candidate.source);
|
||||
for (const srcFile of collectPluginSourceFiles(candidate.rootDir)) {
|
||||
filesToCheck.add(srcFile);
|
||||
}
|
||||
}
|
||||
for (const sharedFile of collectSharedExtensionSourceFiles()) {
|
||||
filesToCheck.add(sharedFile);
|
||||
}
|
||||
for (const extensionFile of collectBundledExtensionSourceFiles()) {
|
||||
filesToCheck.add(extensionFile);
|
||||
}
|
||||
|
||||
const monolithicOffenders: string[] = [];
|
||||
const legacyCompatOffenders: string[] = [];
|
||||
const legacyBroadSubpathOffenders = new Map<string, string[]>();
|
||||
for (const entryFile of filesToCheck) {
|
||||
let content;
|
||||
try {
|
||||
content = fs.readFileSync(entryFile, "utf8");
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
if (hasMonolithicRootImport(content)) {
|
||||
monolithicOffenders.push(entryFile);
|
||||
}
|
||||
if (hasLegacyCompatImport(content)) {
|
||||
legacyCompatOffenders.push(entryFile);
|
||||
}
|
||||
const legacyBroadSubpaths = findLegacyBroadSubpathImports(content);
|
||||
if (legacyBroadSubpaths.length > 0) {
|
||||
legacyBroadSubpathOffenders.set(entryFile, legacyBroadSubpaths);
|
||||
}
|
||||
}
|
||||
|
||||
if (
|
||||
monolithicOffenders.length > 0 ||
|
||||
legacyCompatOffenders.length > 0 ||
|
||||
legacyBroadSubpathOffenders.size > 0
|
||||
) {
|
||||
if (monolithicOffenders.length > 0) {
|
||||
console.error("Bundled plugin source files must not import monolithic openclaw/plugin-sdk.");
|
||||
for (const file of monolithicOffenders.toSorted()) {
|
||||
console.error(`- ${relativeToCwd(file)}`);
|
||||
}
|
||||
}
|
||||
if (legacyCompatOffenders.length > 0) {
|
||||
console.error(
|
||||
"Bundled plugin source files must not import legacy openclaw/plugin-sdk/compat.",
|
||||
);
|
||||
for (const file of legacyCompatOffenders.toSorted()) {
|
||||
console.error(`- ${relativeToCwd(file)}`);
|
||||
}
|
||||
}
|
||||
if (legacyBroadSubpathOffenders.size > 0) {
|
||||
console.error(
|
||||
"Bundled plugin source files must not import deprecated broad plugin-sdk subpaths.",
|
||||
);
|
||||
for (const [file, labels] of [...legacyBroadSubpathOffenders.entries()].toSorted(
|
||||
([left], [right]) => left.localeCompare(right),
|
||||
)) {
|
||||
console.error(`- ${relativeToCwd(file)} (${labels.join(", ")})`);
|
||||
}
|
||||
}
|
||||
if (
|
||||
monolithicOffenders.length > 0 ||
|
||||
legacyCompatOffenders.length > 0 ||
|
||||
legacyBroadSubpathOffenders.size > 0
|
||||
) {
|
||||
console.error(
|
||||
"Use focused openclaw/plugin-sdk/<domain> subpaths for bundled plugins; root, compat, and broad runtime barrels are legacy surfaces only.",
|
||||
);
|
||||
}
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log(
|
||||
`OK: bundled plugin source files use scoped plugin-sdk subpaths (${filesToCheck.size} checked).`,
|
||||
);
|
||||
}
|
||||
|
||||
main();
|
||||
181
scripts/check-no-pairing-store-group-auth.mjs
Normal file
181
scripts/check-no-pairing-store-group-auth.mjs
Normal file
@@ -0,0 +1,181 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Prevents direct pairing-store group auth reads outside resolver helpers.
|
||||
import ts from "typescript";
|
||||
import { createPairingGuardContext } from "./lib/pairing-guard-context.mjs";
|
||||
import {
|
||||
collectFileViolations,
|
||||
getPropertyNameText,
|
||||
runAsScript,
|
||||
toLine,
|
||||
} from "./lib/ts-guard-utils.mjs";
|
||||
|
||||
const { repoRoot, sourceRoots, resolveFromRepo } = createPairingGuardContext(import.meta.url);
|
||||
|
||||
const allowedFiles = new Set([
|
||||
resolveFromRepo("src/channels/message-access/legacy-policy.ts"),
|
||||
resolveFromRepo("src/channels/allow-from.ts"),
|
||||
// Config migration/audit logic may intentionally reference store + group fields.
|
||||
resolveFromRepo("src/security/fix.ts"),
|
||||
resolveFromRepo("src/security/audit-channel.ts"),
|
||||
]);
|
||||
|
||||
const storeIdentifierRe = /^(?:storeAllowFrom|storedAllowFrom|storeAllowList)$/i;
|
||||
const groupNameRe =
|
||||
/(?:groupAllowFrom|effectiveGroupAllowFrom|groupAllowed|groupAllow|groupAuth|groupSender)/i;
|
||||
const storeSourceCallNames = new Set([
|
||||
"readChannelAllowFromStore",
|
||||
"readChannelAllowFromStoreSync",
|
||||
"readStoreAllowFromForDmPolicy",
|
||||
]);
|
||||
const allowedResolverCallNames = new Set([
|
||||
"resolveEffectiveAllowFromLists",
|
||||
"resolveDmGroupAccessWithLists",
|
||||
"resolveMattermostEffectiveAllowFromLists",
|
||||
"resolveIrcEffectiveAllowlists",
|
||||
]);
|
||||
|
||||
function getDeclarationNameText(name) {
|
||||
if (ts.isIdentifier(name)) {
|
||||
return name.text;
|
||||
}
|
||||
if (ts.isObjectBindingPattern(name) || ts.isArrayBindingPattern(name)) {
|
||||
return name.getText();
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function containsPairingStoreSource(node) {
|
||||
let found = false;
|
||||
const visit = (current) => {
|
||||
if (found) {
|
||||
return;
|
||||
}
|
||||
if (ts.isIdentifier(current) && storeIdentifierRe.test(current.text)) {
|
||||
found = true;
|
||||
return;
|
||||
}
|
||||
if (ts.isCallExpression(current)) {
|
||||
const callName = getCallName(current);
|
||||
if (callName && storeSourceCallNames.has(callName)) {
|
||||
found = true;
|
||||
return;
|
||||
}
|
||||
}
|
||||
ts.forEachChild(current, visit);
|
||||
};
|
||||
visit(node);
|
||||
return found;
|
||||
}
|
||||
|
||||
function getCallName(node) {
|
||||
if (!ts.isCallExpression(node)) {
|
||||
return null;
|
||||
}
|
||||
if (ts.isIdentifier(node.expression)) {
|
||||
return node.expression.text;
|
||||
}
|
||||
if (ts.isPropertyAccessExpression(node.expression)) {
|
||||
return node.expression.name.text;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function isSuspiciousNormalizeWithStoreCall(node) {
|
||||
if (!ts.isCallExpression(node)) {
|
||||
return false;
|
||||
}
|
||||
if (!ts.isIdentifier(node.expression) || node.expression.text !== "normalizeAllowFromWithStore") {
|
||||
return false;
|
||||
}
|
||||
const firstArg = node.arguments[0];
|
||||
if (!firstArg || !ts.isObjectLiteralExpression(firstArg)) {
|
||||
return false;
|
||||
}
|
||||
let hasStoreProp = false;
|
||||
let hasGroupAllowProp = false;
|
||||
for (const property of firstArg.properties) {
|
||||
if (!ts.isPropertyAssignment(property)) {
|
||||
continue;
|
||||
}
|
||||
const name = getPropertyNameText(property.name);
|
||||
if (!name) {
|
||||
continue;
|
||||
}
|
||||
if (name === "storeAllowFrom" && containsPairingStoreSource(property.initializer)) {
|
||||
hasStoreProp = true;
|
||||
}
|
||||
if (name === "allowFrom" && groupNameRe.test(property.initializer.getText())) {
|
||||
hasGroupAllowProp = true;
|
||||
}
|
||||
}
|
||||
return hasStoreProp && hasGroupAllowProp;
|
||||
}
|
||||
|
||||
function findViolations(content, filePath) {
|
||||
const sourceFile = ts.createSourceFile(filePath, content, ts.ScriptTarget.Latest, true);
|
||||
const violations = [];
|
||||
|
||||
const visit = (node) => {
|
||||
if (ts.isVariableDeclaration(node) && node.initializer) {
|
||||
const name = getDeclarationNameText(node.name);
|
||||
if (name && groupNameRe.test(name) && containsPairingStoreSource(node.initializer)) {
|
||||
const callName = getCallName(node.initializer);
|
||||
if (callName && allowedResolverCallNames.has(callName)) {
|
||||
ts.forEachChild(node, visit);
|
||||
return;
|
||||
}
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node),
|
||||
reason: `group-scoped variable "${name}" references pairing-store identifiers`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (ts.isPropertyAssignment(node)) {
|
||||
const propName = getPropertyNameText(node.name);
|
||||
if (propName && groupNameRe.test(propName) && containsPairingStoreSource(node.initializer)) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node),
|
||||
reason: `group-scoped property "${propName}" references pairing-store identifiers`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (isSuspiciousNormalizeWithStoreCall(node)) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node),
|
||||
reason: "group allowlist uses normalizeAllowFromWithStore(...) with pairing-store entries",
|
||||
});
|
||||
}
|
||||
|
||||
ts.forEachChild(node, visit);
|
||||
};
|
||||
|
||||
visit(sourceFile);
|
||||
return violations;
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const violations = await collectFileViolations({
|
||||
sourceRoots,
|
||||
repoRoot,
|
||||
findViolations,
|
||||
skipFile: (filePath) => allowedFiles.has(filePath),
|
||||
});
|
||||
|
||||
if (violations.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
console.error("Found pairing-store identifiers referenced in group auth composition:");
|
||||
for (const violation of violations) {
|
||||
console.error(`- ${violation.path}:${violation.line} (${violation.reason})`);
|
||||
}
|
||||
console.error(
|
||||
"Group auth must be composed via shared resolvers (resolveDmGroupAccessWithLists / resolveEffectiveAllowFromLists).",
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
runAsScript(import.meta.url, main);
|
||||
96
scripts/check-no-random-messaging-tmp.mjs
Normal file
96
scripts/check-no-random-messaging-tmp.mjs
Normal file
@@ -0,0 +1,96 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Blocks host-random tmpdir usage in messaging/channel runtime sources.
|
||||
import ts from "typescript";
|
||||
import { bundledPluginFile } from "./lib/bundled-plugin-paths.mjs";
|
||||
import { runCallsiteGuard } from "./lib/callsite-guard.mjs";
|
||||
import {
|
||||
collectCallExpressionLines,
|
||||
runAsScript,
|
||||
unwrapExpression,
|
||||
} from "./lib/ts-guard-utils.mjs";
|
||||
|
||||
/**
|
||||
* Source roots scanned for unsafe messaging tmpdir usage.
|
||||
*/
|
||||
export const messagingTmpdirGuardSourceRoots = [
|
||||
"src/channels",
|
||||
"src/infra/outbound",
|
||||
"src/line",
|
||||
"src/media",
|
||||
"src/media-understanding",
|
||||
"extensions",
|
||||
];
|
||||
const allowedRelativePaths = new Set([bundledPluginFile("feishu", "src/dedup.ts")]);
|
||||
|
||||
function collectOsTmpdirImports(sourceFile) {
|
||||
const osModuleSpecifiers = new Set(["node:os", "os"]);
|
||||
const osNamespaceOrDefault = new Set();
|
||||
const namedTmpdir = new Set();
|
||||
for (const statement of sourceFile.statements) {
|
||||
if (!ts.isImportDeclaration(statement)) {
|
||||
continue;
|
||||
}
|
||||
if (!statement.importClause || !ts.isStringLiteral(statement.moduleSpecifier)) {
|
||||
continue;
|
||||
}
|
||||
if (!osModuleSpecifiers.has(statement.moduleSpecifier.text)) {
|
||||
continue;
|
||||
}
|
||||
const clause = statement.importClause;
|
||||
if (clause.name) {
|
||||
osNamespaceOrDefault.add(clause.name.text);
|
||||
}
|
||||
if (!clause.namedBindings) {
|
||||
continue;
|
||||
}
|
||||
if (ts.isNamespaceImport(clause.namedBindings)) {
|
||||
osNamespaceOrDefault.add(clause.namedBindings.name.text);
|
||||
continue;
|
||||
}
|
||||
for (const element of clause.namedBindings.elements) {
|
||||
if ((element.propertyName?.text ?? element.name.text) === "tmpdir") {
|
||||
namedTmpdir.add(element.name.text);
|
||||
}
|
||||
}
|
||||
}
|
||||
return { osNamespaceOrDefault, namedTmpdir };
|
||||
}
|
||||
|
||||
/**
|
||||
* Finds `os.tmpdir()` or imported `tmpdir()` call lines in source.
|
||||
*/
|
||||
export function findMessagingTmpdirCallLines(content, fileName = "source.ts") {
|
||||
const sourceFile = ts.createSourceFile(fileName, content, ts.ScriptTarget.Latest, true);
|
||||
const { osNamespaceOrDefault, namedTmpdir } = collectOsTmpdirImports(sourceFile);
|
||||
return collectCallExpressionLines(ts, sourceFile, (node) => {
|
||||
const callee = unwrapExpression(node.expression);
|
||||
if (
|
||||
ts.isPropertyAccessExpression(callee) &&
|
||||
callee.name.text === "tmpdir" &&
|
||||
ts.isIdentifier(callee.expression) &&
|
||||
osNamespaceOrDefault.has(callee.expression.text)
|
||||
) {
|
||||
return callee;
|
||||
}
|
||||
return ts.isIdentifier(callee) && namedTmpdir.has(callee.text) ? callee : null;
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the messaging tmpdir guard.
|
||||
*/
|
||||
export async function main() {
|
||||
await runCallsiteGuard({
|
||||
importMetaUrl: import.meta.url,
|
||||
sourceRoots: messagingTmpdirGuardSourceRoots,
|
||||
findCallLines: findMessagingTmpdirCallLines,
|
||||
skipRelativePath: (relativePath) => allowedRelativePaths.has(relativePath),
|
||||
header: "Found os.tmpdir()/tmpdir() usage in messaging/channel runtime sources:",
|
||||
footer:
|
||||
"Use resolvePreferredOpenClawTmpDir() or plugin-sdk temp helpers instead of host tmp defaults.",
|
||||
sortViolations: false,
|
||||
});
|
||||
}
|
||||
|
||||
runAsScript(import.meta.url, main);
|
||||
109
scripts/check-no-raw-channel-fetch.mjs
Normal file
109
scripts/check-no-raw-channel-fetch.mjs
Normal file
@@ -0,0 +1,109 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Blocks new raw fetch callsites in channel and plugin runtime sources.
|
||||
import ts from "typescript";
|
||||
import { bundledPluginCallsite } from "./lib/bundled-plugin-paths.mjs";
|
||||
import { runCallsiteGuard } from "./lib/callsite-guard.mjs";
|
||||
import {
|
||||
collectCallExpressionLines,
|
||||
runAsScript,
|
||||
unwrapExpression,
|
||||
} from "./lib/ts-guard-utils.mjs";
|
||||
|
||||
const sourceRoots = ["src/channels", "src/routing", "src/line", "extensions"];
|
||||
|
||||
// Temporary allowlist for legacy callsites. New raw fetch callsites in channel/plugin runtime
|
||||
// code should be rejected and migrated to fetchWithSsrFGuard/shared channel helpers.
|
||||
const allowedRawFetchCallsites = new Set([
|
||||
bundledPluginCallsite("browser", "src/browser/cdp.helpers.ts", 268),
|
||||
bundledPluginCallsite("browser", "src/browser/client-fetch.ts", 192),
|
||||
bundledPluginCallsite("chutes", "models.ts", 536),
|
||||
bundledPluginCallsite("chutes", "models.ts", 543),
|
||||
bundledPluginCallsite("discord", "src/monitor/gateway-plugin.ts", 417),
|
||||
bundledPluginCallsite("discord", "src/monitor/gateway-plugin.ts", 483),
|
||||
bundledPluginCallsite("discord", "src/voice-message.ts", 298),
|
||||
bundledPluginCallsite("discord", "src/voice-message.ts", 333),
|
||||
bundledPluginCallsite("elevenlabs", "speech-provider.ts", 295),
|
||||
bundledPluginCallsite("elevenlabs", "tts.ts", 74),
|
||||
bundledPluginCallsite("feishu", "src/monitor.webhook.test-helpers.ts", 25),
|
||||
bundledPluginCallsite("github-copilot", "login.ts", 80),
|
||||
bundledPluginCallsite("github-copilot", "login.ts", 112),
|
||||
bundledPluginCallsite("googlechat", "src/auth.ts", 83),
|
||||
bundledPluginCallsite("huggingface", "models.ts", 143),
|
||||
bundledPluginCallsite("kilocode", "provider-models.ts", 130),
|
||||
bundledPluginCallsite("matrix", "src/matrix/sdk/transport.ts", 112),
|
||||
bundledPluginCallsite("microsoft-foundry", "onboard.ts", 479),
|
||||
bundledPluginCallsite("microsoft", "speech-provider.ts", 140),
|
||||
bundledPluginCallsite("minimax", "oauth.ts", 82),
|
||||
bundledPluginCallsite("minimax", "oauth.ts", 123),
|
||||
bundledPluginCallsite("minimax", "tts.ts", 52),
|
||||
bundledPluginCallsite("msteams", "src/graph.ts", 47),
|
||||
bundledPluginCallsite("msteams", "src/sdk.ts", 400),
|
||||
bundledPluginCallsite("msteams", "src/sdk.ts", 441),
|
||||
bundledPluginCallsite("ollama", "src/stream.ts", 649),
|
||||
bundledPluginCallsite("openai", "tts.ts", 149),
|
||||
bundledPluginCallsite("qa-channel", "src/bus-client.ts", 41),
|
||||
bundledPluginCallsite("qa-channel", "src/bus-client.ts", 221),
|
||||
bundledPluginCallsite("qa-lab", "src/docker-up.runtime.ts", 274),
|
||||
bundledPluginCallsite("qa-lab", "src/gateway-child.ts", 489),
|
||||
bundledPluginCallsite("qa-lab", "src/suite.ts", 330),
|
||||
bundledPluginCallsite("qa-lab", "src/suite.ts", 341),
|
||||
bundledPluginCallsite("qa-lab", "web/src/app.ts", 23),
|
||||
bundledPluginCallsite("qa-lab", "web/src/app.ts", 31),
|
||||
bundledPluginCallsite("qa-lab", "web/src/app.ts", 39),
|
||||
bundledPluginCallsite("qqbot", "src/engine/api/api-client.ts", 124),
|
||||
bundledPluginCallsite("qqbot", "src/engine/api/media-chunked.ts", 554),
|
||||
bundledPluginCallsite("qqbot", "src/engine/api/token.ts", 211),
|
||||
bundledPluginCallsite("qqbot", "src/engine/tools/channel-api.ts", 178),
|
||||
bundledPluginCallsite("qqbot", "src/engine/utils/stt.ts", 87),
|
||||
bundledPluginCallsite("signal", "src/install-signal-cli.ts", 224),
|
||||
bundledPluginCallsite("slack", "src/monitor/media.ts", 106),
|
||||
bundledPluginCallsite("slack", "src/monitor/media.ts", 125),
|
||||
bundledPluginCallsite("slack", "src/monitor/media.ts", 130),
|
||||
bundledPluginCallsite("venice", "models.ts", 552),
|
||||
bundledPluginCallsite("vercel-ai-gateway", "models.ts", 181),
|
||||
bundledPluginCallsite("voice-call", "src/providers/twilio/api.ts", 23),
|
||||
]);
|
||||
|
||||
function isRawFetchCall(expression) {
|
||||
const callee = unwrapExpression(expression);
|
||||
if (ts.isIdentifier(callee)) {
|
||||
return callee.text === "fetch";
|
||||
}
|
||||
if (ts.isPropertyAccessExpression(callee)) {
|
||||
return (
|
||||
ts.isIdentifier(callee.expression) &&
|
||||
callee.expression.text === "globalThis" &&
|
||||
callee.name.text === "fetch"
|
||||
);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Finds raw `fetch(...)` and `globalThis.fetch(...)` call lines.
|
||||
*/
|
||||
export function findRawFetchCallLines(content, fileName = "source.ts") {
|
||||
const sourceFile = ts.createSourceFile(fileName, content, ts.ScriptTarget.Latest, true);
|
||||
return collectCallExpressionLines(ts, sourceFile, (node) =>
|
||||
isRawFetchCall(node.expression) ? node.expression : null,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the raw channel/plugin fetch guard.
|
||||
*/
|
||||
export async function main() {
|
||||
await runCallsiteGuard({
|
||||
importMetaUrl: import.meta.url,
|
||||
sourceRoots,
|
||||
extraTestSuffixes: [".browser.test.ts", ".node.test.ts"],
|
||||
findCallLines: findRawFetchCallLines,
|
||||
skipRelativePath: (relPath) => relPath.includes("/test-support/"),
|
||||
allowCallsite: (callsite) => allowedRawFetchCallsites.has(callsite),
|
||||
header: "Found raw fetch() usage in channel/plugin runtime sources outside allowlist:",
|
||||
footer: "Use fetchWithSsrFGuard() or existing channel/plugin SDK wrappers for network calls.",
|
||||
});
|
||||
}
|
||||
|
||||
runAsScript(import.meta.url, main);
|
||||
121
scripts/check-no-raw-http2-imports.mjs
Normal file
121
scripts/check-no-raw-http2-imports.mjs
Normal file
@@ -0,0 +1,121 @@
|
||||
// Rejects raw Node http2 imports in source and extension code.
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
const SOURCE_ROOTS = ["src", "extensions"];
|
||||
const DEFAULT_SKIPPED_DIR_NAMES = new Set(["node_modules", "dist", "coverage", ".generated"]);
|
||||
|
||||
function isCodeFile(filePath) {
|
||||
if (filePath.endsWith(".d.ts")) {
|
||||
return false;
|
||||
}
|
||||
return /\.(?:[cm]?ts|[cm]?js|tsx|jsx)$/u.test(filePath);
|
||||
}
|
||||
|
||||
function collectFilesSync(rootDir, options) {
|
||||
const skipDirNames = options.skipDirNames ?? DEFAULT_SKIPPED_DIR_NAMES;
|
||||
const files = [];
|
||||
const stack = [rootDir];
|
||||
|
||||
while (stack.length > 0) {
|
||||
const current = stack.pop();
|
||||
if (!current) {
|
||||
continue;
|
||||
}
|
||||
let entries;
|
||||
try {
|
||||
entries = fs.readdirSync(current, { withFileTypes: true });
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
for (const entry of entries) {
|
||||
const fullPath = path.join(current, entry.name);
|
||||
if (entry.isDirectory()) {
|
||||
if (!skipDirNames.has(entry.name)) {
|
||||
stack.push(fullPath);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (entry.isFile() && options.includeFile(fullPath)) {
|
||||
files.push(fullPath);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return files;
|
||||
}
|
||||
|
||||
function toPosixPath(filePath) {
|
||||
return filePath.replaceAll("\\", "/");
|
||||
}
|
||||
|
||||
const FORBIDDEN_HTTP2_MODULES = new Set(["node:http2", "http2"]);
|
||||
const ALLOWED_PRODUCTION_FILES = new Set(["src/infra/push-apns-http2.ts"]);
|
||||
|
||||
function isTestFile(relativePath) {
|
||||
return (
|
||||
/(?:^|\/)(?:test|test-fixtures)\//u.test(relativePath) ||
|
||||
/\.test\.[cm]?[jt]sx?$/u.test(relativePath)
|
||||
);
|
||||
}
|
||||
|
||||
function lineNumberForOffset(content, offset) {
|
||||
return content.slice(0, offset).split(/\r?\n/u).length;
|
||||
}
|
||||
|
||||
function collectHttp2ImportOffenders(filePath) {
|
||||
const relativePath = toPosixPath(path.relative(process.cwd(), filePath));
|
||||
if (ALLOWED_PRODUCTION_FILES.has(relativePath) || isTestFile(relativePath)) {
|
||||
return [];
|
||||
}
|
||||
|
||||
const content = fs.readFileSync(filePath, "utf8");
|
||||
const offenders = [];
|
||||
const patterns = [
|
||||
/\bimport\s+(?:type\s+)?[\s\S]*?\bfrom\s*["']([^"']+)["']/gu,
|
||||
/\bexport\s+(?:type\s+)?[\s\S]*?\bfrom\s*["']([^"']+)["']/gu,
|
||||
/\bimport\s*\(\s*["']([^"']+)["']\s*\)/gu,
|
||||
/\brequire\s*\(\s*["']([^"']+)["']\s*\)/gu,
|
||||
];
|
||||
|
||||
for (const pattern of patterns) {
|
||||
for (const match of content.matchAll(pattern)) {
|
||||
const specifier = match[1];
|
||||
if (specifier && FORBIDDEN_HTTP2_MODULES.has(specifier)) {
|
||||
offenders.push({
|
||||
file: relativePath,
|
||||
line: lineNumberForOffset(content, match.index ?? 0),
|
||||
specifier,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return offenders;
|
||||
}
|
||||
|
||||
function collectSourceFiles() {
|
||||
return SOURCE_ROOTS.flatMap((root) =>
|
||||
collectFilesSync(path.join(process.cwd(), root), {
|
||||
includeFile: isCodeFile,
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
function main() {
|
||||
const offenders = collectSourceFiles().flatMap(collectHttp2ImportOffenders);
|
||||
if (offenders.length === 0) {
|
||||
console.log("OK: raw node:http2 imports stay behind the APNs proxy wrapper.");
|
||||
return;
|
||||
}
|
||||
|
||||
console.error("Raw node:http2 imports are only allowed in src/infra/push-apns-http2.ts.");
|
||||
for (const offender of offenders.toSorted(
|
||||
(a, b) => a.file.localeCompare(b.file) || a.line - b.line,
|
||||
)) {
|
||||
console.error(`- ${offender.file}:${offender.line} imports ${offender.specifier}`);
|
||||
}
|
||||
console.error("Use connectApnsHttp2Session() so APNs HTTP/2 honors managed proxy policy.");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
main();
|
||||
93
scripts/check-no-raw-window-open.mjs
Normal file
93
scripts/check-no-raw-window-open.mjs
Normal file
@@ -0,0 +1,93 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Ensures UI code opens external URLs through the safe helper.
|
||||
import { promises as fs } from "node:fs";
|
||||
import path from "node:path";
|
||||
import ts from "typescript";
|
||||
import {
|
||||
collectTypeScriptFiles,
|
||||
resolveRepoRoot,
|
||||
runAsScript,
|
||||
toLine,
|
||||
unwrapExpression,
|
||||
} from "./lib/ts-guard-utils.mjs";
|
||||
|
||||
const repoRoot = resolveRepoRoot(import.meta.url);
|
||||
const uiSourceDir = path.join(repoRoot, "ui", "src", "ui");
|
||||
const allowedCallsites = new Set([path.join(uiSourceDir, "open-external-url.ts")]);
|
||||
|
||||
function asPropertyAccess(expression) {
|
||||
if (ts.isPropertyAccessExpression(expression)) {
|
||||
return expression;
|
||||
}
|
||||
if (typeof ts.isPropertyAccessChain === "function" && ts.isPropertyAccessChain(expression)) {
|
||||
return expression;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function isRawWindowOpenCall(expression) {
|
||||
const propertyAccess = asPropertyAccess(unwrapExpression(expression));
|
||||
if (!propertyAccess || propertyAccess.name.text !== "open") {
|
||||
return false;
|
||||
}
|
||||
|
||||
const receiver = unwrapExpression(propertyAccess.expression);
|
||||
return (
|
||||
ts.isIdentifier(receiver) && (receiver.text === "window" || receiver.text === "globalThis")
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Finds raw `window.open(...)` or `globalThis.open(...)` call lines.
|
||||
*/
|
||||
export function findRawWindowOpenLines(content, fileName = "source.ts") {
|
||||
const sourceFile = ts.createSourceFile(fileName, content, ts.ScriptTarget.Latest, true);
|
||||
const lines = [];
|
||||
|
||||
const visit = (node) => {
|
||||
if (ts.isCallExpression(node) && isRawWindowOpenCall(node.expression)) {
|
||||
lines.push(toLine(sourceFile, node.expression));
|
||||
}
|
||||
ts.forEachChild(node, visit);
|
||||
};
|
||||
|
||||
visit(sourceFile);
|
||||
return lines;
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the raw window.open guard.
|
||||
*/
|
||||
export async function main() {
|
||||
const files = await collectTypeScriptFiles(uiSourceDir, {
|
||||
extraTestSuffixes: [".browser.test.ts", ".node.test.ts"],
|
||||
ignoreMissing: true,
|
||||
});
|
||||
const violations = [];
|
||||
|
||||
for (const filePath of files) {
|
||||
if (allowedCallsites.has(filePath)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const content = await fs.readFile(filePath, "utf8");
|
||||
for (const line of findRawWindowOpenLines(content, filePath)) {
|
||||
const relPath = path.relative(repoRoot, filePath);
|
||||
violations.push(`${relPath}:${line}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (violations.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
console.error("Found raw window.open usage outside safe helper:");
|
||||
for (const violation of violations) {
|
||||
console.error(`- ${violation}`);
|
||||
}
|
||||
console.error("Use openExternalUrlSafe(...) from ui/src/lib/open-external-url.ts instead.");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
runAsScript(import.meta.url, main);
|
||||
43
scripts/check-no-register-http-handler.mjs
Normal file
43
scripts/check-no-register-http-handler.mjs
Normal file
@@ -0,0 +1,43 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Blocks deprecated plugin `registerHttpHandler` callsites.
|
||||
import ts from "typescript";
|
||||
import { runCallsiteGuard } from "./lib/callsite-guard.mjs";
|
||||
import {
|
||||
collectCallExpressionLines,
|
||||
runAsScript,
|
||||
unwrapExpression,
|
||||
} from "./lib/ts-guard-utils.mjs";
|
||||
|
||||
const sourceRoots = ["src", "extensions"];
|
||||
|
||||
function isDeprecatedRegisterHttpHandlerCall(expression) {
|
||||
const callee = unwrapExpression(expression);
|
||||
return ts.isPropertyAccessExpression(callee) && callee.name.text === "registerHttpHandler";
|
||||
}
|
||||
|
||||
/**
|
||||
* Finds deprecated `registerHttpHandler(...)` call lines.
|
||||
*/
|
||||
export function findDeprecatedRegisterHttpHandlerLines(content, fileName = "source.ts") {
|
||||
const sourceFile = ts.createSourceFile(fileName, content, ts.ScriptTarget.Latest, true);
|
||||
return collectCallExpressionLines(ts, sourceFile, (node) =>
|
||||
isDeprecatedRegisterHttpHandlerCall(node.expression) ? node.expression : null,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the deprecated HTTP handler API guard.
|
||||
*/
|
||||
export async function main() {
|
||||
await runCallsiteGuard({
|
||||
importMetaUrl: import.meta.url,
|
||||
sourceRoots,
|
||||
findCallLines: findDeprecatedRegisterHttpHandlerLines,
|
||||
header: "Found deprecated plugin API call registerHttpHandler(...):",
|
||||
footer:
|
||||
"Use registerHttpRoute({ path, auth, match, handler }) and registerPluginHttpRoute for dynamic webhook paths.",
|
||||
});
|
||||
}
|
||||
|
||||
runAsScript(import.meta.url, main);
|
||||
21
scripts/check-no-runtime-action-load-config.mjs
Normal file
21
scripts/check-no-runtime-action-load-config.mjs
Normal file
@@ -0,0 +1,21 @@
|
||||
#!/usr/bin/env node
|
||||
// Prevents runtime action paths from loading global config directly.
|
||||
import { collectRuntimeActionLoadConfigViolations } from "./lib/config-boundary-guard.mjs";
|
||||
|
||||
function main() {
|
||||
const violations = collectRuntimeActionLoadConfigViolations();
|
||||
if (violations.length === 0) {
|
||||
return 0;
|
||||
}
|
||||
console.error(
|
||||
[
|
||||
"Runtime channel send/action/client/pairing helpers must not call loadConfig().",
|
||||
"Load and resolve config at the command/gateway/monitor boundary, then pass cfg through.",
|
||||
"",
|
||||
...violations,
|
||||
].join("\n"),
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
|
||||
process.exitCode = main();
|
||||
353
scripts/check-openclaw-package-tarball.mjs
Normal file
353
scripts/check-openclaw-package-tarball.mjs
Normal file
@@ -0,0 +1,353 @@
|
||||
#!/usr/bin/env node
|
||||
// Validates the npm tarball Docker E2E lanes install.
|
||||
// This is intentionally tarball-only: the check proves Docker lanes consume the
|
||||
// prebuilt package artifact with dist inventory, not a source checkout.
|
||||
import { spawnSync } from "node:child_process";
|
||||
import fs from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { performance } from "node:perf_hooks";
|
||||
import { LOCAL_BUILD_METADATA_DIST_PATHS } from "./lib/local-build-metadata-paths.mjs";
|
||||
import {
|
||||
collectPackageDistImports,
|
||||
collectPackageDistImportErrors,
|
||||
expandPackageDistImportClosure,
|
||||
} from "./lib/package-dist-imports.mjs";
|
||||
|
||||
function usage() {
|
||||
return "Usage: node scripts/check-openclaw-package-tarball.mjs <openclaw.tgz>";
|
||||
}
|
||||
|
||||
function fail(message) {
|
||||
console.error(message);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
function parseArgs(argv) {
|
||||
const args = argv[0] === "--" ? argv.slice(1) : argv;
|
||||
const tarball = args[0]?.trim() ?? "";
|
||||
if (tarball === "--help" || tarball === "-h") {
|
||||
return { help: true, tarball: "" };
|
||||
}
|
||||
if (!tarball) {
|
||||
throw new Error(usage());
|
||||
}
|
||||
if (tarball.startsWith("-")) {
|
||||
throw new Error(`Unknown OpenClaw package tarball check option: ${tarball}`);
|
||||
}
|
||||
const extraArg = args[1]?.trim();
|
||||
if (extraArg) {
|
||||
throw new Error(`Unexpected OpenClaw package tarball check argument: ${extraArg}`);
|
||||
}
|
||||
return { help: false, tarball };
|
||||
}
|
||||
|
||||
let cliArgs;
|
||||
try {
|
||||
cliArgs = parseArgs(process.argv.slice(2));
|
||||
} catch (error) {
|
||||
fail(error instanceof Error ? error.message : String(error));
|
||||
}
|
||||
if (cliArgs.help) {
|
||||
console.log(usage());
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
const { tarball } = cliArgs;
|
||||
if (!fs.existsSync(tarball)) {
|
||||
fail(`OpenClaw package tarball does not exist: ${tarball}`);
|
||||
}
|
||||
|
||||
const phaseTimingsEnabled = process.env.OPENCLAW_PACKAGE_TARBALL_CHECK_TIMINGS !== "0";
|
||||
function runPhase(label, action) {
|
||||
const startedAt = performance.now();
|
||||
try {
|
||||
return action();
|
||||
} finally {
|
||||
if (phaseTimingsEnabled) {
|
||||
const durationMs = Math.round(performance.now() - startedAt);
|
||||
console.error(`check-openclaw-package-tarball: ${label} completed in ${durationMs}ms`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const list = runPhase("tar list", () =>
|
||||
spawnSync("tar", ["-tf", tarball], {
|
||||
encoding: "utf8",
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
}),
|
||||
);
|
||||
if (list.status !== 0) {
|
||||
fail(`tar -tf failed for ${tarball}: ${list.stderr || list.status}`);
|
||||
}
|
||||
|
||||
const extractDir = fs.mkdtempSync(path.join(os.tmpdir(), "openclaw-package-tarball-"));
|
||||
try {
|
||||
const extract = runPhase("tar extract", () =>
|
||||
spawnSync("tar", ["-xf", tarball, "-C", extractDir], {
|
||||
encoding: "utf8",
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
}),
|
||||
);
|
||||
if (extract.status !== 0) {
|
||||
fs.rmSync(extractDir, { recursive: true, force: true });
|
||||
fail(`tar -xf failed for ${tarball}: ${extract.stderr || extract.status}`);
|
||||
}
|
||||
} catch (error) {
|
||||
fs.rmSync(extractDir, { recursive: true, force: true });
|
||||
throw error;
|
||||
}
|
||||
|
||||
const entries = list.stdout
|
||||
.split(/\r?\n/u)
|
||||
.map((entry) => entry.trim())
|
||||
.filter(Boolean);
|
||||
const normalized = entries.map((entry) => entry.replace(/^package\//u, ""));
|
||||
const entrySet = new Set(normalized);
|
||||
const errors = [];
|
||||
const warnings = [];
|
||||
const REQUIRED_TARBALL_ENTRIES = ["dist/control-ui/index.html"];
|
||||
const REQUIRED_TARBALL_ENTRY_PREFIXES = ["dist/control-ui/assets/"];
|
||||
const LEGACY_PACKAGE_ACCEPTANCE_COMPAT_MAX = { year: 2026, month: 4, day: 25 };
|
||||
const LEGACY_LOCAL_BUILD_METADATA_COMPAT_MAX = { year: 2026, month: 4, day: 26 };
|
||||
const LEGACY_SHRINKWRAP_COMPAT_MAX = { year: 2026, month: 5, day: 20 };
|
||||
const FORBIDDEN_LOCAL_BUILD_METADATA_FILES = new Set(LOCAL_BUILD_METADATA_DIST_PATHS);
|
||||
|
||||
const LEGACY_OMITTED_PRIVATE_QA_INVENTORY_PREFIXES = [
|
||||
"dist/extensions/qa-channel/",
|
||||
"dist/extensions/qa-lab/",
|
||||
"dist/extensions/qa-matrix/",
|
||||
"dist/plugin-sdk/extensions/qa-channel/",
|
||||
"dist/plugin-sdk/extensions/qa-lab/",
|
||||
];
|
||||
const LEGACY_OMITTED_PRIVATE_QA_INVENTORY_FILES = new Set([
|
||||
"dist/plugin-sdk/qa-channel.d.ts",
|
||||
"dist/plugin-sdk/qa-channel.js",
|
||||
"dist/plugin-sdk/qa-channel-protocol.d.ts",
|
||||
"dist/plugin-sdk/qa-channel-protocol.js",
|
||||
"dist/plugin-sdk/qa-lab.d.ts",
|
||||
"dist/plugin-sdk/qa-lab.js",
|
||||
"dist/plugin-sdk/qa-runtime.d.ts",
|
||||
"dist/plugin-sdk/qa-runtime.js",
|
||||
"dist/plugin-sdk/src/plugin-sdk/qa-channel.d.ts",
|
||||
"dist/plugin-sdk/src/plugin-sdk/qa-channel-protocol.d.ts",
|
||||
"dist/plugin-sdk/src/plugin-sdk/qa-lab.d.ts",
|
||||
"dist/plugin-sdk/src/plugin-sdk/qa-runtime.d.ts",
|
||||
]);
|
||||
|
||||
function isLegacyOmittedPrivateQaInventoryEntry(relativePath) {
|
||||
return (
|
||||
LEGACY_OMITTED_PRIVATE_QA_INVENTORY_FILES.has(relativePath) ||
|
||||
LEGACY_OMITTED_PRIVATE_QA_INVENTORY_PREFIXES.some((prefix) => relativePath.startsWith(prefix))
|
||||
);
|
||||
}
|
||||
|
||||
function parseCalver(version) {
|
||||
const match = /^(\d{4})\.(\d{1,2})\.(\d{1,2})(?:[-+].*)?$/u.exec(version);
|
||||
if (!match) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
year: Number(match[1]),
|
||||
month: Number(match[2]),
|
||||
day: Number(match[3]),
|
||||
};
|
||||
}
|
||||
|
||||
function compareCalver(left, right) {
|
||||
for (const key of ["year", "month", "day"]) {
|
||||
if (left[key] !== right[key]) {
|
||||
return left[key] - right[key];
|
||||
}
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
function isLegacyPackageAcceptanceCompatVersion(version) {
|
||||
const parsed = parseCalver(version);
|
||||
return parsed ? compareCalver(parsed, LEGACY_PACKAGE_ACCEPTANCE_COMPAT_MAX) <= 0 : false;
|
||||
}
|
||||
|
||||
function isLegacyLocalBuildMetadataCompatVersion(version) {
|
||||
const parsed = parseCalver(version);
|
||||
return parsed ? compareCalver(parsed, LEGACY_LOCAL_BUILD_METADATA_COMPAT_MAX) <= 0 : false;
|
||||
}
|
||||
|
||||
function isLegacyShrinkwrapCompatVersion(version) {
|
||||
const parsed = parseCalver(version);
|
||||
return parsed ? compareCalver(parsed, LEGACY_SHRINKWRAP_COMPAT_MAX) <= 0 : false;
|
||||
}
|
||||
|
||||
function readTarEntry(entryPath) {
|
||||
const candidates = [
|
||||
path.join(extractDir, entryPath),
|
||||
path.join(extractDir, "package", entryPath),
|
||||
];
|
||||
for (const candidate of candidates) {
|
||||
if (fs.existsSync(candidate)) {
|
||||
return fs.readFileSync(candidate, "utf8");
|
||||
}
|
||||
}
|
||||
return "";
|
||||
}
|
||||
|
||||
for (const entry of normalized) {
|
||||
if (entry.startsWith("/") || entry.split("/").includes("..")) {
|
||||
errors.push(`unsafe tar entry: ${entry}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (!entrySet.has("package.json")) {
|
||||
errors.push("missing package.json");
|
||||
}
|
||||
if (!normalized.some((entry) => entry.startsWith("dist/"))) {
|
||||
errors.push("missing dist/ entries");
|
||||
}
|
||||
for (const requiredEntry of REQUIRED_TARBALL_ENTRIES) {
|
||||
if (!entrySet.has(requiredEntry)) {
|
||||
errors.push(`missing required tar entry ${requiredEntry}`);
|
||||
}
|
||||
}
|
||||
for (const requiredPrefix of REQUIRED_TARBALL_ENTRY_PREFIXES) {
|
||||
if (!normalized.some((entry) => entry.startsWith(requiredPrefix))) {
|
||||
errors.push(`missing required tar entries under ${requiredPrefix}`);
|
||||
}
|
||||
}
|
||||
let packageVersion = "";
|
||||
if (entrySet.has("package.json")) {
|
||||
try {
|
||||
const packageJson = JSON.parse(readTarEntry("package.json"));
|
||||
packageVersion = typeof packageJson.version === "string" ? packageJson.version : "";
|
||||
} catch {
|
||||
packageVersion = "";
|
||||
}
|
||||
}
|
||||
if (entrySet.has("package-lock.json")) {
|
||||
errors.push("package tarball must ship npm-shrinkwrap.json, not package-lock.json");
|
||||
}
|
||||
if (!entrySet.has("npm-shrinkwrap.json")) {
|
||||
if (isLegacyShrinkwrapCompatVersion(packageVersion)) {
|
||||
warnings.push("legacy package omits npm-shrinkwrap.json");
|
||||
} else {
|
||||
errors.push("missing required tar entry npm-shrinkwrap.json");
|
||||
}
|
||||
} else {
|
||||
try {
|
||||
const shrinkwrap = JSON.parse(readTarEntry("npm-shrinkwrap.json"));
|
||||
const rootPackage = shrinkwrap.packages?.[""];
|
||||
if (shrinkwrap.name !== "openclaw") {
|
||||
errors.push("npm-shrinkwrap.json root name must be openclaw");
|
||||
}
|
||||
if (shrinkwrap.version !== packageVersion) {
|
||||
errors.push(
|
||||
`npm-shrinkwrap.json version ${shrinkwrap.version ?? "<missing>"} does not match package.json version ${packageVersion || "<missing>"}`,
|
||||
);
|
||||
}
|
||||
if (!rootPackage || rootPackage.name !== "openclaw") {
|
||||
errors.push("npm-shrinkwrap.json packages root must name openclaw");
|
||||
}
|
||||
if (rootPackage?.version !== packageVersion) {
|
||||
errors.push(
|
||||
`npm-shrinkwrap.json packages root version ${rootPackage?.version ?? "<missing>"} does not match package.json version ${packageVersion || "<missing>"}`,
|
||||
);
|
||||
}
|
||||
if (rootPackage?.devDependencies) {
|
||||
errors.push("npm-shrinkwrap.json must not lock root devDependencies");
|
||||
}
|
||||
const devLockedPackages = Object.entries(shrinkwrap.packages ?? {})
|
||||
.filter(([, packageMetadata]) => packageMetadata?.dev === true)
|
||||
.map(([packagePath]) => packagePath);
|
||||
if (devLockedPackages.length > 0) {
|
||||
errors.push(
|
||||
`npm-shrinkwrap.json must not lock dev packages: ${devLockedPackages.slice(0, 5).join(", ")}`,
|
||||
);
|
||||
}
|
||||
} catch (error) {
|
||||
errors.push(
|
||||
`unreadable npm-shrinkwrap.json: ${error instanceof Error ? error.message : String(error)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
for (const forbiddenEntry of FORBIDDEN_LOCAL_BUILD_METADATA_FILES) {
|
||||
if (entrySet.has(forbiddenEntry)) {
|
||||
if (isLegacyLocalBuildMetadataCompatVersion(packageVersion)) {
|
||||
warnings.push(`legacy package includes local build metadata tar entry ${forbiddenEntry}`);
|
||||
continue;
|
||||
}
|
||||
errors.push(`forbidden local build metadata tar entry ${forbiddenEntry}`);
|
||||
}
|
||||
}
|
||||
if (!entrySet.has("dist/postinstall-inventory.json")) {
|
||||
errors.push("missing dist/postinstall-inventory.json");
|
||||
}
|
||||
let packageDistImports = null;
|
||||
if (entrySet.has("dist/postinstall-inventory.json")) {
|
||||
try {
|
||||
const allowLegacyPrivateQaInventoryOmissions =
|
||||
isLegacyPackageAcceptanceCompatVersion(packageVersion);
|
||||
const inventory = JSON.parse(readTarEntry("dist/postinstall-inventory.json"));
|
||||
if (!Array.isArray(inventory) || inventory.some((entry) => typeof entry !== "string")) {
|
||||
errors.push("invalid dist/postinstall-inventory.json");
|
||||
} else {
|
||||
const normalizedInventory = inventory.map((entry) => entry.replace(/\\/gu, "/"));
|
||||
const normalizedInventorySet = new Set(normalizedInventory);
|
||||
packageDistImports = runPhase("dist import graph", () =>
|
||||
collectPackageDistImports({
|
||||
files: normalized,
|
||||
readText: readTarEntry,
|
||||
}),
|
||||
);
|
||||
for (const inventoryEntry of inventory) {
|
||||
const normalizedEntry = inventoryEntry.replace(/\\/gu, "/");
|
||||
if (!entrySet.has(normalizedEntry)) {
|
||||
if (
|
||||
allowLegacyPrivateQaInventoryOmissions &&
|
||||
isLegacyOmittedPrivateQaInventoryEntry(normalizedEntry)
|
||||
) {
|
||||
warnings.push(
|
||||
`legacy inventory references omitted private QA tar entry ${normalizedEntry}`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
errors.push(`inventory references missing tar entry ${normalizedEntry}`);
|
||||
}
|
||||
}
|
||||
const expandedInventory = expandPackageDistImportClosure({
|
||||
files: normalized,
|
||||
seedFiles: normalizedInventory,
|
||||
readText: readTarEntry,
|
||||
imports: packageDistImports,
|
||||
});
|
||||
for (const importedEntry of expandedInventory) {
|
||||
if (!normalizedInventorySet.has(importedEntry)) {
|
||||
errors.push(`inventory omits imported dist file ${importedEntry}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
errors.push(
|
||||
`unreadable dist/postinstall-inventory.json: ${
|
||||
error instanceof Error ? error.message : String(error)
|
||||
}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
errors.push(
|
||||
...collectPackageDistImportErrors({
|
||||
files: normalized,
|
||||
readText: readTarEntry,
|
||||
imports: packageDistImports ?? undefined,
|
||||
}),
|
||||
);
|
||||
|
||||
if (errors.length > 0) {
|
||||
fs.rmSync(extractDir, { recursive: true, force: true });
|
||||
fail(`OpenClaw package tarball integrity failed:\n${errors.join("\n")}`);
|
||||
}
|
||||
|
||||
for (const warning of warnings) {
|
||||
console.warn(`OpenClaw package tarball integrity warning: ${warning}`);
|
||||
}
|
||||
fs.rmSync(extractDir, { recursive: true, force: true });
|
||||
console.log("OpenClaw package tarball integrity passed.");
|
||||
82
scripts/check-package-dist-imports.mjs
Normal file
82
scripts/check-package-dist-imports.mjs
Normal file
@@ -0,0 +1,82 @@
|
||||
#!/usr/bin/env node
|
||||
// Checks built package dist files for imports outside package boundaries.
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { collectPackageDistImportErrors } from "./lib/package-dist-imports.mjs";
|
||||
|
||||
function usage() {
|
||||
return "Usage: node scripts/check-package-dist-imports.mjs [package-root]";
|
||||
}
|
||||
|
||||
function fail(message) {
|
||||
console.error(message);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
function parseArgs(argv) {
|
||||
const args = argv[0] === "--" ? argv.slice(1) : argv;
|
||||
const packageRootArg = args[0]?.trim() ?? "";
|
||||
if (packageRootArg === "--help" || packageRootArg === "-h") {
|
||||
return { help: true, packageRoot: "" };
|
||||
}
|
||||
if (packageRootArg.startsWith("-")) {
|
||||
throw new Error(`Unknown package dist import check option: ${packageRootArg}`);
|
||||
}
|
||||
const extraArg = args[1]?.trim();
|
||||
if (extraArg) {
|
||||
throw new Error(`Unexpected package dist import check argument: ${extraArg}`);
|
||||
}
|
||||
return {
|
||||
help: false,
|
||||
packageRoot: path.resolve(packageRootArg || process.cwd()),
|
||||
};
|
||||
}
|
||||
|
||||
let cliArgs;
|
||||
try {
|
||||
cliArgs = parseArgs(process.argv.slice(2));
|
||||
} catch (error) {
|
||||
fail(error instanceof Error ? error.message : String(error));
|
||||
}
|
||||
if (cliArgs.help) {
|
||||
console.log(usage());
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
const { packageRoot } = cliArgs;
|
||||
const distRoot = path.join(packageRoot, "dist");
|
||||
if (!fs.existsSync(distRoot)) {
|
||||
fail(`missing dist directory: ${distRoot}`);
|
||||
}
|
||||
|
||||
function collectFiles(rootDir) {
|
||||
const pending = [rootDir];
|
||||
const files = [];
|
||||
while (pending.length > 0) {
|
||||
const dir = pending.pop();
|
||||
for (const entry of fs.readdirSync(dir, { withFileTypes: true })) {
|
||||
const entryPath = path.join(dir, entry.name);
|
||||
if (entry.isDirectory()) {
|
||||
pending.push(entryPath);
|
||||
continue;
|
||||
}
|
||||
if (entry.isFile()) {
|
||||
files.push(path.relative(packageRoot, entryPath).replace(/\\/gu, "/"));
|
||||
}
|
||||
}
|
||||
}
|
||||
return files;
|
||||
}
|
||||
|
||||
const errors = collectPackageDistImportErrors({
|
||||
files: collectFiles(distRoot),
|
||||
readText(relativePath) {
|
||||
return fs.readFileSync(path.join(packageRoot, relativePath), "utf8");
|
||||
},
|
||||
});
|
||||
|
||||
if (errors.length > 0) {
|
||||
fail(`OpenClaw package dist import closure failed:\n${errors.join("\n")}`);
|
||||
}
|
||||
|
||||
console.log("OpenClaw package dist import closure passed.");
|
||||
148
scripts/check-package-patches.mjs
Normal file
148
scripts/check-package-patches.mjs
Normal file
@@ -0,0 +1,148 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Guards pnpm package patches against unapproved additions.
|
||||
import { execFileSync } from "node:child_process";
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import YAML from "yaml";
|
||||
|
||||
const ALLOWED_PATCHED_DEPENDENCIES = new Map([
|
||||
// Remove after fs-safe ships pinned-write fsync with best-effort EPERM handling.
|
||||
["@openclaw/fs-safe@0.4.1", "patches/@openclaw__fs-safe@0.4.1.patch"],
|
||||
["baileys@7.0.0-rc12", "patches/baileys@7.0.0-rc12.patch"],
|
||||
["baileys@7.0.0-rc13", "patches/baileys@7.0.0-rc13.patch"],
|
||||
]);
|
||||
|
||||
const ALLOWED_PATCH_FILES = new Set(["patches/.gitkeep", ...ALLOWED_PATCHED_DEPENDENCIES.values()]);
|
||||
|
||||
function listTrackedFiles(cwd, patterns) {
|
||||
return execFileSync("git", ["ls-files", "-z", "--", ...patterns], {
|
||||
cwd,
|
||||
encoding: "utf8",
|
||||
})
|
||||
.split("\0")
|
||||
.filter(Boolean)
|
||||
.toSorted((left, right) => left.localeCompare(right));
|
||||
}
|
||||
|
||||
function readYamlFile(cwd, relativePath) {
|
||||
const filePath = path.join(cwd, relativePath);
|
||||
if (!fs.existsSync(filePath)) {
|
||||
return {};
|
||||
}
|
||||
return YAML.parse(fs.readFileSync(filePath, "utf8")) ?? {};
|
||||
}
|
||||
|
||||
function readJsonFile(cwd, relativePath) {
|
||||
const filePath = path.join(cwd, relativePath);
|
||||
if (!fs.existsSync(filePath)) {
|
||||
return undefined;
|
||||
}
|
||||
return JSON.parse(fs.readFileSync(filePath, "utf8"));
|
||||
}
|
||||
|
||||
function collectPatchedDependencyViolations(file, patchedDependencies, violations, options = {}) {
|
||||
for (const [specifier, patchPathOrHash] of Object.entries(patchedDependencies ?? {})) {
|
||||
if (
|
||||
options.allowAnyValueForLegacy === true
|
||||
? ALLOWED_PATCHED_DEPENDENCIES.has(specifier)
|
||||
: ALLOWED_PATCHED_DEPENDENCIES.get(specifier) === patchPathOrHash
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
violations.push({
|
||||
file,
|
||||
kind: "patchedDependency",
|
||||
detail: `${specifier} -> ${String(patchPathOrHash)}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
function collectWorkspacePatchViolations(cwd, violations) {
|
||||
const workspace = readYamlFile(cwd, "pnpm-workspace.yaml");
|
||||
collectPatchedDependencyViolations(
|
||||
"pnpm-workspace.yaml",
|
||||
workspace?.patchedDependencies,
|
||||
violations,
|
||||
);
|
||||
}
|
||||
|
||||
function collectLockfilePatchViolations(cwd, violations) {
|
||||
const lockfile = readYamlFile(cwd, "pnpm-lock.yaml");
|
||||
collectPatchedDependencyViolations("pnpm-lock.yaml", lockfile?.patchedDependencies, violations, {
|
||||
allowAnyValueForLegacy: true,
|
||||
});
|
||||
}
|
||||
|
||||
function collectPackageJsonPatchViolations(cwd, violations) {
|
||||
for (const relativePath of listTrackedFiles(cwd, ["*package.json"])) {
|
||||
const packageJson = readJsonFile(cwd, relativePath);
|
||||
const patchedDependencies = packageJson?.pnpm?.patchedDependencies;
|
||||
for (const [specifier, patchPath] of Object.entries(patchedDependencies ?? {})) {
|
||||
violations.push({
|
||||
file: relativePath,
|
||||
kind: "packageJsonPatchedDependency",
|
||||
detail: `${specifier} -> ${String(patchPath)}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function collectPatchFileViolations(cwd, violations) {
|
||||
for (const relativePath of listTrackedFiles(cwd, ["*.patch"])) {
|
||||
if (!fs.existsSync(path.join(cwd, relativePath))) {
|
||||
continue;
|
||||
}
|
||||
if (ALLOWED_PATCH_FILES.has(relativePath)) {
|
||||
continue;
|
||||
}
|
||||
violations.push({
|
||||
file: relativePath,
|
||||
kind: "patchFile",
|
||||
detail: "new package patch file",
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Collects disallowed package patch declarations and patch files.
|
||||
*/
|
||||
export function collectPackagePatchViolations(cwd = process.cwd()) {
|
||||
const violations = [];
|
||||
collectWorkspacePatchViolations(cwd, violations);
|
||||
collectLockfilePatchViolations(cwd, violations);
|
||||
collectPackageJsonPatchViolations(cwd, violations);
|
||||
collectPatchFileViolations(cwd, violations);
|
||||
return violations;
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the package patch guard.
|
||||
*/
|
||||
export async function main() {
|
||||
const violations = collectPackagePatchViolations();
|
||||
if (violations.length === 0) {
|
||||
process.stdout.write(
|
||||
`PASS package patch guard: no new pnpm patches; ${ALLOWED_PATCHED_DEPENDENCIES.size} approved patches allowlisted.\n`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
console.error(
|
||||
"FAIL package patch guard: new pnpm package patches are not allowed. Upstream the fix, publish a new package version, then bump the dependency instead.",
|
||||
);
|
||||
for (const violation of violations) {
|
||||
console.error(`- ${violation.file}: ${violation.kind}: ${violation.detail}`);
|
||||
}
|
||||
process.exitCode = 1;
|
||||
}
|
||||
|
||||
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
||||
main().catch(
|
||||
/** @param {unknown} error */ (error) => {
|
||||
console.error(error);
|
||||
process.exit(1);
|
||||
},
|
||||
);
|
||||
}
|
||||
101
scripts/check-pairing-account-scope.mjs
Normal file
101
scripts/check-pairing-account-scope.mjs
Normal file
@@ -0,0 +1,101 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Checks pairing config logic for account-scoped allowlist handling.
|
||||
import ts from "typescript";
|
||||
import { createPairingGuardContext } from "./lib/pairing-guard-context.mjs";
|
||||
import {
|
||||
collectFileViolations,
|
||||
getPropertyNameText,
|
||||
runAsScript,
|
||||
toLine,
|
||||
} from "./lib/ts-guard-utils.mjs";
|
||||
|
||||
const { repoRoot, sourceRoots } = createPairingGuardContext(import.meta.url);
|
||||
|
||||
function isUndefinedLikeExpression(node) {
|
||||
if (ts.isIdentifier(node) && node.text === "undefined") {
|
||||
return true;
|
||||
}
|
||||
return node.kind === ts.SyntaxKind.NullKeyword;
|
||||
}
|
||||
|
||||
function hasRequiredAccountIdProperty(node) {
|
||||
if (!ts.isObjectLiteralExpression(node)) {
|
||||
return false;
|
||||
}
|
||||
for (const property of node.properties) {
|
||||
if (ts.isShorthandPropertyAssignment(property) && property.name.text === "accountId") {
|
||||
return true;
|
||||
}
|
||||
if (!ts.isPropertyAssignment(property)) {
|
||||
continue;
|
||||
}
|
||||
if (getPropertyNameText(property.name) !== "accountId") {
|
||||
continue;
|
||||
}
|
||||
if (isUndefinedLikeExpression(property.initializer)) {
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function findViolations(content, filePath) {
|
||||
const sourceFile = ts.createSourceFile(filePath, content, ts.ScriptTarget.Latest, true);
|
||||
const violations = [];
|
||||
|
||||
const visit = (node) => {
|
||||
if (ts.isCallExpression(node) && ts.isIdentifier(node.expression)) {
|
||||
const callName = node.expression.text;
|
||||
if (callName === "readChannelAllowFromStore") {
|
||||
if (node.arguments.length < 3 || isUndefinedLikeExpression(node.arguments[2])) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node),
|
||||
reason: "readChannelAllowFromStore call must pass explicit accountId as 3rd arg",
|
||||
});
|
||||
}
|
||||
} else if (
|
||||
callName === "readLegacyChannelAllowFromStore" ||
|
||||
callName === "readLegacyChannelAllowFromStoreSync"
|
||||
) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node),
|
||||
reason: `${callName} is legacy-only; use account-scoped readChannelAllowFromStore* APIs`,
|
||||
});
|
||||
} else if (callName === "upsertChannelPairingRequest") {
|
||||
const firstArg = node.arguments[0];
|
||||
if (!firstArg || !hasRequiredAccountIdProperty(firstArg)) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node),
|
||||
reason: "upsertChannelPairingRequest call must include accountId in params",
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
ts.forEachChild(node, visit);
|
||||
};
|
||||
|
||||
visit(sourceFile);
|
||||
return violations;
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const violations = await collectFileViolations({
|
||||
sourceRoots,
|
||||
repoRoot,
|
||||
findViolations,
|
||||
});
|
||||
|
||||
if (violations.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
console.error("Found unscoped pairing-store calls:");
|
||||
for (const violation of violations) {
|
||||
console.error(`- ${violation.path}:${violation.line} (${violation.reason})`);
|
||||
}
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
runAsScript(import.meta.url, main);
|
||||
238
scripts/check-plugin-extension-import-boundary.mjs
Normal file
238
scripts/check-plugin-extension-import-boundary.mjs
Normal file
@@ -0,0 +1,238 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Inventories core plugin imports that cross into bundled extension files.
|
||||
import { promises as fs } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import ts from "typescript";
|
||||
import { BUNDLED_PLUGIN_PATH_PREFIX } from "./lib/bundled-plugin-paths.mjs";
|
||||
import {
|
||||
collectTypeScriptInventory,
|
||||
createCachedAsync,
|
||||
diffInventoryEntries,
|
||||
formatGroupedInventoryHuman,
|
||||
normalizeRepoPath,
|
||||
runBaselineInventoryCheck,
|
||||
resolveRepoSpecifier,
|
||||
visitModuleSpecifiers,
|
||||
} from "./lib/guard-inventory-utils.mjs";
|
||||
import {
|
||||
collectTypeScriptFilesFromRoots,
|
||||
resolveSourceRoots,
|
||||
runAsScript,
|
||||
toLine,
|
||||
} from "./lib/ts-guard-utils.mjs";
|
||||
|
||||
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const scanRoots = resolveSourceRoots(repoRoot, ["src/plugins"]);
|
||||
const baselinePath = path.join(
|
||||
repoRoot,
|
||||
"test",
|
||||
"fixtures",
|
||||
"plugin-extension-import-boundary-inventory.json",
|
||||
);
|
||||
|
||||
const bundledWebSearchProviders = new Set([
|
||||
"brave",
|
||||
"firecrawl",
|
||||
"gemini",
|
||||
"grok",
|
||||
"kimi",
|
||||
"perplexity",
|
||||
]);
|
||||
const bundledWebSearchPluginIds = new Set([
|
||||
"brave",
|
||||
"firecrawl",
|
||||
"google",
|
||||
"moonshot",
|
||||
"perplexity",
|
||||
"xai",
|
||||
]);
|
||||
|
||||
function compareEntries(left, right) {
|
||||
return (
|
||||
left.file.localeCompare(right.file) ||
|
||||
left.line - right.line ||
|
||||
left.kind.localeCompare(right.kind) ||
|
||||
left.specifier.localeCompare(right.specifier) ||
|
||||
left.reason.localeCompare(right.reason)
|
||||
);
|
||||
}
|
||||
|
||||
function classifyResolvedExtensionReason(kind, resolvedPath) {
|
||||
const verb =
|
||||
kind === "export"
|
||||
? "re-exports"
|
||||
: kind === "dynamic-import"
|
||||
? "dynamically imports"
|
||||
: "imports";
|
||||
if (/^extensions\/[^/]+\/src\//.test(resolvedPath)) {
|
||||
return `${verb} extension implementation from src/plugins`;
|
||||
}
|
||||
if (/^extensions\/[^/]+\/index\.[^/]+$/.test(resolvedPath)) {
|
||||
return `${verb} extension entrypoint from src/plugins`;
|
||||
}
|
||||
return `${verb} extension-owned file from src/plugins`;
|
||||
}
|
||||
|
||||
function pushEntry(entries, entry) {
|
||||
entries.push(entry);
|
||||
}
|
||||
|
||||
function scanImportBoundaryViolations(sourceFile, filePath) {
|
||||
const entries = [];
|
||||
const relativeFile = normalizeRepoPath(repoRoot, filePath);
|
||||
|
||||
visitModuleSpecifiers(ts, sourceFile, ({ kind, specifier, specifierNode }) => {
|
||||
const resolvedPath = resolveRepoSpecifier(repoRoot, specifier, filePath);
|
||||
if (!resolvedPath?.startsWith(BUNDLED_PLUGIN_PATH_PREFIX)) {
|
||||
return;
|
||||
}
|
||||
pushEntry(entries, {
|
||||
file: relativeFile,
|
||||
line: toLine(sourceFile, specifierNode),
|
||||
kind,
|
||||
specifier,
|
||||
resolvedPath,
|
||||
reason: classifyResolvedExtensionReason(kind, resolvedPath),
|
||||
});
|
||||
});
|
||||
return entries;
|
||||
}
|
||||
|
||||
function scanWebSearchRegistrySmells(sourceFile, filePath) {
|
||||
const relativeFile = normalizeRepoPath(repoRoot, filePath);
|
||||
if (relativeFile !== "src/plugins/web-search-providers.ts") {
|
||||
return [];
|
||||
}
|
||||
|
||||
const entries = [];
|
||||
const lines = sourceFile.text.split(/\r?\n/);
|
||||
for (const [index, line] of lines.entries()) {
|
||||
const lineNumber = index + 1;
|
||||
|
||||
if (line.includes("web-search-plugin-factory.js")) {
|
||||
pushEntry(entries, {
|
||||
file: relativeFile,
|
||||
line: lineNumber,
|
||||
kind: "registry-smell",
|
||||
specifier: "../agents/tools/web-search-plugin-factory.js",
|
||||
resolvedPath: "src/agents/tools/web-search-plugin-factory.js",
|
||||
reason: "imports core-owned web search provider factory into plugin registry",
|
||||
});
|
||||
}
|
||||
|
||||
const pluginMatch = line.match(/pluginId:\s*"([^"]+)"/);
|
||||
if (pluginMatch && bundledWebSearchPluginIds.has(pluginMatch[1])) {
|
||||
pushEntry(entries, {
|
||||
file: relativeFile,
|
||||
line: lineNumber,
|
||||
kind: "registry-smell",
|
||||
specifier: pluginMatch[1],
|
||||
resolvedPath: relativeFile,
|
||||
reason: "hardcodes bundled web search plugin ownership in core registry",
|
||||
});
|
||||
}
|
||||
|
||||
const providerMatch = line.match(/id:\s*"(brave|firecrawl|gemini|grok|kimi|perplexity)"/);
|
||||
if (providerMatch && bundledWebSearchProviders.has(providerMatch[1])) {
|
||||
pushEntry(entries, {
|
||||
file: relativeFile,
|
||||
line: lineNumber,
|
||||
kind: "registry-smell",
|
||||
specifier: providerMatch[1],
|
||||
resolvedPath: relativeFile,
|
||||
reason: "hardcodes bundled web search provider metadata in core registry",
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return entries;
|
||||
}
|
||||
|
||||
function shouldSkipFile(filePath) {
|
||||
const relativeFile = normalizeRepoPath(repoRoot, filePath);
|
||||
return (
|
||||
relativeFile === "src/plugins/bundled-web-search-registry.ts" ||
|
||||
relativeFile.startsWith("src/plugins/contracts/") ||
|
||||
/^src\/plugins\/runtime\/runtime-[^/]+-contract\.[cm]?[jt]s$/u.test(relativeFile)
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Cached inventory of src/plugins imports that cross into bundled extensions.
|
||||
*/
|
||||
export const collectPluginExtensionImportBoundaryInventory = createCachedAsync(async () => {
|
||||
const files = (await collectTypeScriptFilesFromRoots(scanRoots))
|
||||
.filter((filePath) => !shouldSkipFile(filePath))
|
||||
.toSorted((left, right) =>
|
||||
normalizeRepoPath(repoRoot, left).localeCompare(normalizeRepoPath(repoRoot, right)),
|
||||
);
|
||||
return await collectTypeScriptInventory({
|
||||
ts,
|
||||
files,
|
||||
compareEntries,
|
||||
collectEntries(sourceFile, filePath) {
|
||||
return [
|
||||
...scanImportBoundaryViolations(sourceFile, filePath),
|
||||
...scanWebSearchRegistrySmells(sourceFile, filePath),
|
||||
];
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
/**
|
||||
* Cached expected plugin-extension import inventory baseline.
|
||||
*/
|
||||
export const readExpectedInventory = createCachedAsync(async () =>
|
||||
JSON.parse(await fs.readFile(baselinePath, "utf8")),
|
||||
);
|
||||
|
||||
/**
|
||||
* Diffs expected and actual plugin-extension boundary inventory entries.
|
||||
*/
|
||||
export function diffInventory(expected, actual) {
|
||||
return diffInventoryEntries(expected, actual, compareEntries);
|
||||
}
|
||||
|
||||
const formatInventoryHuman = (inventory) =>
|
||||
formatGroupedInventoryHuman(
|
||||
{
|
||||
rule: "Rule: src/plugins/** must not import bundled plugin files",
|
||||
cleanMessage: "No plugin import boundary violations found.",
|
||||
inventoryTitle: "Plugin extension import boundary inventory:",
|
||||
},
|
||||
inventory,
|
||||
);
|
||||
|
||||
function formatEntry(entry) {
|
||||
return `${entry.file}:${entry.line} [${entry.kind}] ${entry.reason} (${entry.specifier} -> ${entry.resolvedPath})`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the plugin-extension import boundary baseline check.
|
||||
*/
|
||||
export async function runPluginExtensionImportBoundaryCheck(argv, io) {
|
||||
return await runBaselineInventoryCheck({
|
||||
argv: argv ?? process.argv.slice(2),
|
||||
io,
|
||||
collectActual: collectPluginExtensionImportBoundaryInventory,
|
||||
readExpected: readExpectedInventory,
|
||||
diffInventory,
|
||||
formatInventoryHuman,
|
||||
formatEntry,
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Entrypoint wrapper for the plugin-extension import boundary check.
|
||||
*/
|
||||
export async function main(argv, io) {
|
||||
const exitCode = await runPluginExtensionImportBoundaryCheck(argv, io);
|
||||
if (!io && exitCode !== 0) {
|
||||
process.exit(exitCode);
|
||||
}
|
||||
return exitCode;
|
||||
}
|
||||
|
||||
runAsScript(import.meta.url, main);
|
||||
1229
scripts/check-plugin-gateway-gauntlet.mjs
Normal file
1229
scripts/check-plugin-gateway-gauntlet.mjs
Normal file
File diff suppressed because it is too large
Load Diff
105
scripts/check-plugin-npm-runtime-builds.mjs
Normal file
105
scripts/check-plugin-npm-runtime-builds.mjs
Normal file
@@ -0,0 +1,105 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Verifies publishable plugin packages can build their npm runtime outputs.
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import {
|
||||
buildPluginNpmRuntime,
|
||||
listPluginNpmRuntimeBuildOutputs,
|
||||
listPublishablePluginPackageDirs,
|
||||
resolvePluginNpmRuntimeBuildPlan,
|
||||
} from "./lib/plugin-npm-runtime-build.mjs";
|
||||
|
||||
function readPackageArgValue(argv, index) {
|
||||
const value = argv[index + 1];
|
||||
if (value === undefined || value === "" || value.startsWith("-")) {
|
||||
throw new Error("missing value for --package");
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function usage() {
|
||||
return "usage: node scripts/check-plugin-npm-runtime-builds.mjs [--package extensions/<id> ...]";
|
||||
}
|
||||
|
||||
export function parseArgs(argv) {
|
||||
const args = argv[0] === "--" ? argv.slice(1) : argv;
|
||||
if (args[0] === "--help" || args[0] === "-h") {
|
||||
return { help: true, packageDirs: [] };
|
||||
}
|
||||
const packageDirs = [];
|
||||
for (let index = 0; index < args.length; index += 1) {
|
||||
const arg = args[index];
|
||||
if (arg === "--package") {
|
||||
packageDirs.push(readPackageArgValue(args, index));
|
||||
index += 1;
|
||||
continue;
|
||||
}
|
||||
throw new Error(usage());
|
||||
}
|
||||
return { packageDirs };
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds publishable plugin npm runtimes and verifies declared outputs exist.
|
||||
*/
|
||||
export async function checkPluginNpmRuntimeBuilds(params = {}) {
|
||||
const repoRoot = path.resolve(params.repoRoot ?? ".");
|
||||
const packageDirs =
|
||||
params.packageDirs?.length > 0
|
||||
? params.packageDirs
|
||||
: listPublishablePluginPackageDirs({ repoRoot });
|
||||
const rows = [];
|
||||
for (const packageDir of packageDirs) {
|
||||
const plan = resolvePluginNpmRuntimeBuildPlan({ repoRoot, packageDir });
|
||||
if (!plan) {
|
||||
throw new Error(`${packageDir} did not produce a package-local runtime build plan`);
|
||||
}
|
||||
const result = await buildPluginNpmRuntime({
|
||||
repoRoot,
|
||||
packageDir,
|
||||
logLevel: params.logLevel ?? "warn",
|
||||
});
|
||||
const missing = listPluginNpmRuntimeBuildOutputs(result).filter(
|
||||
(runtimePath) =>
|
||||
!fs.existsSync(path.join(result.packageDir, runtimePath.replace(/^\.\//u, ""))),
|
||||
);
|
||||
if (missing.length > 0) {
|
||||
throw new Error(`${packageDir} missing built runtime outputs: ${missing.join(", ")}`);
|
||||
}
|
||||
rows.push({
|
||||
pluginDir: result.pluginDir,
|
||||
status: "built",
|
||||
entryCount: Object.keys(result.entry).length,
|
||||
copiedStaticAssets: result.copiedStaticAssets,
|
||||
});
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
|
||||
if (import.meta.url === pathToFileURL(process.argv[1] ?? "").href) {
|
||||
try {
|
||||
const args = parseArgs(process.argv.slice(2));
|
||||
if (args.help) {
|
||||
console.log(usage());
|
||||
process.exit(0);
|
||||
}
|
||||
const rows = await checkPluginNpmRuntimeBuilds(args);
|
||||
const builtCount = rows.filter((row) => row.status === "built").length;
|
||||
console.log(`checked ${rows.length} publishable plugins; built ${builtCount} npm runtimes`);
|
||||
for (const row of rows) {
|
||||
console.log(
|
||||
[
|
||||
row.pluginDir,
|
||||
row.status,
|
||||
row.entryCount,
|
||||
row.copiedStaticAssets.length > 0 ? row.copiedStaticAssets.join(",") : "-",
|
||||
].join("\t"),
|
||||
);
|
||||
}
|
||||
} catch (error) {
|
||||
console.error(error instanceof Error ? error.message : String(error));
|
||||
process.exitCode = 1;
|
||||
}
|
||||
}
|
||||
182
scripts/check-plugin-sdk-exports.mjs
Executable file
182
scripts/check-plugin-sdk-exports.mjs
Executable file
@@ -0,0 +1,182 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
/**
|
||||
* Verifies that the root plugin-sdk runtime surface is present in the compiled
|
||||
* dist output.
|
||||
*
|
||||
* Run after `pnpm build` to catch missing root exports or leaked repo-only type
|
||||
* aliases before release.
|
||||
*/
|
||||
|
||||
import { readFileSync, existsSync, statSync } from "node:fs";
|
||||
import { resolve, dirname, relative, sep } from "node:path";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
import { publicPluginSdkEntrypoints, publicPluginSdkSubpaths } from "./lib/plugin-sdk-entries.mjs";
|
||||
|
||||
const scriptDir = dirname(fileURLToPath(import.meta.url));
|
||||
const distFile = resolve(scriptDir, "..", "dist", "plugin-sdk", "index.js");
|
||||
if (!existsSync(distFile)) {
|
||||
console.error("ERROR: dist/plugin-sdk/index.js not found. Run `pnpm build` first.");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const content = readFileSync(distFile, "utf-8");
|
||||
|
||||
// Extract the final export statement from the compiled output.
|
||||
// tsdown/rolldown emits a single `export { ... }` at the end of the file.
|
||||
const exportMatch = content.match(/export\s*\{([^}]+)\}\s*;?\s*$/);
|
||||
if (!exportMatch) {
|
||||
console.error("ERROR: Could not find export statement in dist/plugin-sdk/index.js");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const exportedNames = exportMatch[1]
|
||||
.split(",")
|
||||
.map((s) => {
|
||||
// Handle `foo as bar` aliases — the exported name is the `bar` part
|
||||
const parts = s.trim().split(/\s+as\s+/);
|
||||
return (parts[parts.length - 1] || "").trim();
|
||||
})
|
||||
.filter(Boolean);
|
||||
|
||||
const exportSet = new Set(exportedNames);
|
||||
|
||||
const requiredRuntimeShimEntries = ["compat.js", "root-alias.cjs"];
|
||||
const forbiddenPublicDeclarationSpecifiers = ["@openclaw/llm-core"];
|
||||
const MAX_PLUGIN_SDK_DECLARATION_BYTES = 5_000_000;
|
||||
const RELATIVE_DECLARATION_SPECIFIER_RE = /\b(?:from|import)\s*(?:\(\s*)?["']([^"']+)["']/gu;
|
||||
const requiredSubpathExports = {
|
||||
"secret-input-runtime": [
|
||||
"coerceSecretRef",
|
||||
"hasConfiguredSecretInput",
|
||||
"isSecretRef",
|
||||
"normalizeResolvedSecretInputString",
|
||||
"normalizeSecretInputString",
|
||||
"resolveSecretInputString",
|
||||
],
|
||||
};
|
||||
|
||||
// The root plugin-sdk entry intentionally stays tiny. Keep this list aligned
|
||||
// with src/plugin-sdk/index.ts runtime exports.
|
||||
const requiredExports = [
|
||||
"emptyPluginConfigSchema",
|
||||
"onDiagnosticEvent",
|
||||
"registerContextEngine",
|
||||
"delegateCompactionToRuntime",
|
||||
];
|
||||
|
||||
let missing = 0;
|
||||
for (const name of requiredExports) {
|
||||
if (!exportSet.has(name)) {
|
||||
console.error(`MISSING EXPORT: ${name}`);
|
||||
missing += 1;
|
||||
}
|
||||
}
|
||||
|
||||
for (const entry of publicPluginSdkSubpaths) {
|
||||
const jsPath = resolve(scriptDir, "..", "dist", "plugin-sdk", `${entry}.js`);
|
||||
const dtsPath = resolve(scriptDir, "..", "dist", "plugin-sdk", `${entry}.d.ts`);
|
||||
if (!existsSync(jsPath)) {
|
||||
console.error(`MISSING SUBPATH JS: dist/plugin-sdk/${entry}.js`);
|
||||
missing += 1;
|
||||
}
|
||||
if (!existsSync(dtsPath)) {
|
||||
console.error(`MISSING SUBPATH DTS: dist/plugin-sdk/${entry}.d.ts`);
|
||||
missing += 1;
|
||||
}
|
||||
}
|
||||
|
||||
for (const entry of requiredRuntimeShimEntries) {
|
||||
const shimPath = resolve(scriptDir, "..", "dist", "plugin-sdk", entry);
|
||||
if (!existsSync(shimPath)) {
|
||||
console.error(`MISSING RUNTIME SHIM: dist/plugin-sdk/${entry}`);
|
||||
missing += 1;
|
||||
}
|
||||
}
|
||||
|
||||
for (const [entry, names] of Object.entries(requiredSubpathExports)) {
|
||||
const jsPath = resolve(scriptDir, "..", "dist", "plugin-sdk", `${entry}.js`);
|
||||
if (!existsSync(jsPath)) {
|
||||
continue;
|
||||
}
|
||||
let runtime;
|
||||
try {
|
||||
runtime = await import(pathToFileURL(jsPath).href);
|
||||
} catch (err) {
|
||||
console.error(`BROKEN SUBPATH JS: dist/plugin-sdk/${entry}.js`);
|
||||
console.error(err instanceof Error ? err.message : String(err));
|
||||
missing += 1;
|
||||
continue;
|
||||
}
|
||||
for (const name of names) {
|
||||
if (typeof runtime[name] !== "function") {
|
||||
console.error(`MISSING SUBPATH EXPORT: dist/plugin-sdk/${entry}.js#${name}`);
|
||||
missing += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const distDir = resolve(scriptDir, "..", "dist");
|
||||
const declarationPaths = new Set();
|
||||
const declarationQueue = publicPluginSdkEntrypoints.map((entry) =>
|
||||
resolve(distDir, "plugin-sdk", `${entry}.d.ts`),
|
||||
);
|
||||
while (declarationQueue.length > 0) {
|
||||
const dtsPath = declarationQueue.pop();
|
||||
if (!dtsPath || declarationPaths.has(dtsPath)) {
|
||||
continue;
|
||||
}
|
||||
if (!existsSync(dtsPath)) {
|
||||
console.error(`MISSING PUBLIC DTS DEPENDENCY: ${relative(resolve(scriptDir, ".."), dtsPath)}`);
|
||||
missing += 1;
|
||||
continue;
|
||||
}
|
||||
declarationPaths.add(dtsPath);
|
||||
const dtsContent = readFileSync(dtsPath, "utf8");
|
||||
for (const match of dtsContent.matchAll(RELATIVE_DECLARATION_SPECIFIER_RE)) {
|
||||
const specifier = match[1];
|
||||
if (!specifier?.startsWith(".")) {
|
||||
continue;
|
||||
}
|
||||
const declarationSpecifier = specifier.endsWith(".js")
|
||||
? `${specifier.slice(0, -3)}.d.ts`
|
||||
: `${specifier}.d.ts`;
|
||||
const importedPath = resolve(dirname(dtsPath), declarationSpecifier);
|
||||
if (importedPath.startsWith(`${distDir}${sep}`)) {
|
||||
declarationQueue.push(importedPath);
|
||||
}
|
||||
}
|
||||
for (const specifier of forbiddenPublicDeclarationSpecifiers) {
|
||||
if (dtsContent.includes(`"${specifier}`) || dtsContent.includes(`'${specifier}`)) {
|
||||
console.error(
|
||||
`FORBIDDEN PUBLIC DTS SPECIFIER: ${relative(resolve(scriptDir, ".."), dtsPath)} imports ${specifier}`,
|
||||
);
|
||||
missing += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const declarationBytes = Array.from(declarationPaths).reduce(
|
||||
(total, dtsPath) => total + statSync(dtsPath).size,
|
||||
0,
|
||||
);
|
||||
if (declarationBytes > MAX_PLUGIN_SDK_DECLARATION_BYTES) {
|
||||
console.error(
|
||||
`PLUGIN SDK DTS TOO LARGE: ${declarationBytes} bytes exceeds ${MAX_PLUGIN_SDK_DECLARATION_BYTES} bytes.`,
|
||||
);
|
||||
console.error("Keep plugin SDK declarations in the canonical unified tsdown graph.");
|
||||
missing += 1;
|
||||
}
|
||||
|
||||
if (missing > 0) {
|
||||
console.error(
|
||||
`\nERROR: ${missing} required plugin-sdk artifact(s) missing (named exports or subpath files).`,
|
||||
);
|
||||
console.error("This will break published plugin-sdk artifacts.");
|
||||
console.error(
|
||||
"Check src/plugin-sdk/index.ts, generated d.ts rewrites, subpath entries, and rebuild.",
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log(`OK: All ${requiredExports.length} required plugin-sdk exports verified.`);
|
||||
143
scripts/check-plugin-sdk-subpath-exports.mjs
Normal file
143
scripts/check-plugin-sdk-subpath-exports.mjs
Normal file
@@ -0,0 +1,143 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Verifies plugin SDK subpath exports and generated entrypoint metadata.
|
||||
import { readFileSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import ts from "typescript";
|
||||
import { normalizeRepoPath, visitModuleSpecifiers } from "./lib/guard-inventory-utils.mjs";
|
||||
import {
|
||||
collectTypeScriptFilesFromRoots,
|
||||
resolveSourceRoots,
|
||||
toLine,
|
||||
} from "./lib/ts-guard-utils.mjs";
|
||||
|
||||
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const scanRoots = resolveSourceRoots(repoRoot, ["src", "extensions", "scripts", "test"]);
|
||||
|
||||
function readPackageExports() {
|
||||
const packageJson = JSON.parse(readFileSync(path.join(repoRoot, "package.json"), "utf8"));
|
||||
return new Set(
|
||||
Object.keys(packageJson.exports ?? {})
|
||||
.filter((key) => key.startsWith("./plugin-sdk/"))
|
||||
.map((key) => key.slice("./plugin-sdk/".length)),
|
||||
);
|
||||
}
|
||||
|
||||
function readEntrypoints() {
|
||||
const entrypoints = JSON.parse(
|
||||
readFileSync(path.join(repoRoot, "scripts/lib/plugin-sdk-entrypoints.json"), "utf8"),
|
||||
);
|
||||
return new Set(entrypoints.filter((entry) => entry !== "index"));
|
||||
}
|
||||
|
||||
function readPrivateLocalOnlySubpaths() {
|
||||
const subpaths = JSON.parse(
|
||||
readFileSync(
|
||||
path.join(repoRoot, "scripts/lib/plugin-sdk-private-local-only-subpaths.json"),
|
||||
"utf8",
|
||||
),
|
||||
);
|
||||
return new Set(subpaths.filter((entry) => typeof entry === "string" && !entry.includes("/")));
|
||||
}
|
||||
|
||||
function parsePluginSdkSubpath(specifier) {
|
||||
if (!specifier.startsWith("openclaw/plugin-sdk/")) {
|
||||
return null;
|
||||
}
|
||||
const subpath = specifier.slice("openclaw/plugin-sdk/".length);
|
||||
return subpath || null;
|
||||
}
|
||||
|
||||
function compareEntries(left, right) {
|
||||
return (
|
||||
left.file.localeCompare(right.file) ||
|
||||
left.line - right.line ||
|
||||
left.kind.localeCompare(right.kind) ||
|
||||
left.specifier.localeCompare(right.specifier) ||
|
||||
left.subpath.localeCompare(right.subpath)
|
||||
);
|
||||
}
|
||||
|
||||
async function collectViolations() {
|
||||
const entrypoints = readEntrypoints();
|
||||
const exports = readPackageExports();
|
||||
const privateLocalOnlySubpaths = readPrivateLocalOnlySubpaths();
|
||||
const files = (await collectTypeScriptFilesFromRoots(scanRoots, { includeTests: true })).toSorted(
|
||||
(left, right) =>
|
||||
normalizeRepoPath(repoRoot, left).localeCompare(normalizeRepoPath(repoRoot, right)),
|
||||
);
|
||||
const violations = [];
|
||||
|
||||
for (const filePath of files) {
|
||||
const sourceText = readFileSync(filePath, "utf8");
|
||||
const sourceFile = ts.createSourceFile(
|
||||
filePath,
|
||||
sourceText,
|
||||
ts.ScriptTarget.Latest,
|
||||
true,
|
||||
filePath.endsWith(".tsx") ? ts.ScriptKind.TSX : ts.ScriptKind.TS,
|
||||
);
|
||||
|
||||
function push(kind, specifierNode, specifier) {
|
||||
const subpath = parsePluginSdkSubpath(specifier);
|
||||
if (!subpath) {
|
||||
return;
|
||||
}
|
||||
if (privateLocalOnlySubpaths.has(subpath)) {
|
||||
return;
|
||||
}
|
||||
|
||||
const missingFrom = [];
|
||||
if (!entrypoints.has(subpath)) {
|
||||
missingFrom.push("scripts/lib/plugin-sdk-entrypoints.json");
|
||||
}
|
||||
if (!exports.has(subpath)) {
|
||||
missingFrom.push("package.json exports");
|
||||
}
|
||||
if (missingFrom.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
violations.push({
|
||||
file: normalizeRepoPath(repoRoot, filePath),
|
||||
line: toLine(sourceFile, specifierNode),
|
||||
kind,
|
||||
specifier,
|
||||
subpath,
|
||||
missingFrom,
|
||||
});
|
||||
}
|
||||
|
||||
visitModuleSpecifiers(ts, sourceFile, ({ kind, specifier, specifierNode }) => {
|
||||
push(kind, specifierNode, specifier);
|
||||
});
|
||||
}
|
||||
|
||||
return violations.toSorted(compareEntries);
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const violations = await collectViolations();
|
||||
if (violations.length === 0) {
|
||||
console.log("OK: all referenced openclaw/plugin-sdk/<subpath> imports are exported.");
|
||||
return;
|
||||
}
|
||||
|
||||
console.error(
|
||||
"Rule: every referenced openclaw/plugin-sdk/<subpath> must exist in the public package exports.",
|
||||
);
|
||||
for (const violation of violations) {
|
||||
console.error(
|
||||
`- ${violation.file}:${violation.line} [${violation.kind}] ${violation.specifier} missing from ${violation.missingFrom.join(" and ")}`,
|
||||
);
|
||||
}
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
main().catch(
|
||||
/** @param {unknown} error */ (error) => {
|
||||
console.error(error);
|
||||
process.exit(1);
|
||||
},
|
||||
);
|
||||
95
scripts/check-plugin-sdk-wildcard-reexports.mjs
Normal file
95
scripts/check-plugin-sdk-wildcard-reexports.mjs
Normal file
@@ -0,0 +1,95 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Rejects wildcard plugin SDK re-exports in extension API barrels.
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const extensionsRoot = path.join(repoRoot, "extensions");
|
||||
|
||||
const WILDCARD_PLUGIN_SDK_REEXPORT_PATTERN =
|
||||
/^\s*export\s+(?:type\s+)?\*\s+(?:as\s+[$\w]+\s+)?from\s+["']openclaw\/plugin-sdk\//u;
|
||||
|
||||
async function listExtensionApiFiles(rootDir = extensionsRoot) {
|
||||
const entries = await fs.readdir(rootDir, { withFileTypes: true });
|
||||
const files = [];
|
||||
for (const entry of entries) {
|
||||
if (!entry.isDirectory()) {
|
||||
continue;
|
||||
}
|
||||
for (const fileName of ["api.ts", "runtime-api.ts"]) {
|
||||
const filePath = path.join(rootDir, entry.name, fileName);
|
||||
try {
|
||||
const stat = await fs.stat(filePath);
|
||||
if (stat.isFile()) {
|
||||
files.push(filePath);
|
||||
}
|
||||
} catch (error) {
|
||||
if (!error || typeof error !== "object" || !("code" in error) || error.code !== "ENOENT") {
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return files.toSorted((left, right) => left.localeCompare(right));
|
||||
}
|
||||
|
||||
/**
|
||||
* Finds wildcard plugin SDK re-export lines in an extension API barrel.
|
||||
*/
|
||||
export function findPluginSdkWildcardReexports(source) {
|
||||
return source
|
||||
.split(/\r?\n/u)
|
||||
.map((text, index) => ({ line: index + 1, text }))
|
||||
.filter(({ text }) => WILDCARD_PLUGIN_SDK_REEXPORT_PATTERN.test(text));
|
||||
}
|
||||
|
||||
/**
|
||||
* Collects extension API barrels that wildcard re-export plugin SDK subpaths.
|
||||
*/
|
||||
export async function collectPluginSdkWildcardReexports(rootDir = repoRoot) {
|
||||
const files = await listExtensionApiFiles(path.join(rootDir, "extensions"));
|
||||
const violations = [];
|
||||
for (const filePath of files) {
|
||||
const source = await fs.readFile(filePath, "utf8");
|
||||
for (const match of findPluginSdkWildcardReexports(source)) {
|
||||
violations.push({
|
||||
file: path.relative(rootDir, filePath).split(path.sep).join("/"),
|
||||
line: match.line,
|
||||
text: match.text.trim(),
|
||||
});
|
||||
}
|
||||
}
|
||||
return violations;
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the plugin SDK wildcard re-export guard.
|
||||
*/
|
||||
export async function main(argv = process.argv.slice(2), io = process) {
|
||||
const json = argv.includes("--json");
|
||||
const violations = await collectPluginSdkWildcardReexports();
|
||||
|
||||
if (json) {
|
||||
io.stdout.write(`${JSON.stringify(violations, null, 2)}\n`);
|
||||
return violations.length === 0 ? 0 : 1;
|
||||
}
|
||||
|
||||
if (violations.length === 0) {
|
||||
io.stdout.write("No plugin-sdk wildcard re-exports found in extension API barrels.\n");
|
||||
return 0;
|
||||
}
|
||||
|
||||
io.stderr.write("Found plugin-sdk wildcard re-exports in extension API barrels:\n");
|
||||
for (const violation of violations) {
|
||||
io.stderr.write(`- ${violation.file}:${violation.line} ${violation.text}\n`);
|
||||
}
|
||||
io.stderr.write("Use explicit named exports from the narrow SDK subpath instead.\n");
|
||||
return 1;
|
||||
}
|
||||
|
||||
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
||||
const exitCode = await main();
|
||||
process.exit(exitCode);
|
||||
}
|
||||
232
scripts/check-policy-config-coverage.ts
Normal file
232
scripts/check-policy-config-coverage.ts
Normal file
@@ -0,0 +1,232 @@
|
||||
#!/usr/bin/env node
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import JSON5 from "json5";
|
||||
import {
|
||||
renderConfigDocBaselineArtifacts,
|
||||
type ConfigDocBaselineEntry,
|
||||
} from "../src/config/doc-baseline.js";
|
||||
|
||||
type ClassificationStatus = "observed" | "ignored" | "out-of-scope" | "deferred";
|
||||
|
||||
type CoverageClassification = {
|
||||
readonly pattern: string;
|
||||
readonly status: ClassificationStatus;
|
||||
readonly area: string;
|
||||
readonly policy?: string;
|
||||
readonly reason: string;
|
||||
readonly allowNoSchemaPath?: boolean;
|
||||
};
|
||||
|
||||
type CoverageConfig = {
|
||||
readonly monitored: readonly string[];
|
||||
readonly classifications: readonly CoverageClassification[];
|
||||
};
|
||||
|
||||
type ConfigDocBaseline = {
|
||||
readonly coreEntries: readonly ConfigDocBaselineEntry[];
|
||||
readonly channelEntries: readonly ConfigDocBaselineEntry[];
|
||||
readonly pluginEntries: readonly ConfigDocBaselineEntry[];
|
||||
};
|
||||
|
||||
function flattenConfigDocBaselineEntries(
|
||||
baseline: ConfigDocBaseline,
|
||||
): readonly ConfigDocBaselineEntry[] {
|
||||
return [...baseline.coreEntries, ...baseline.channelEntries, ...baseline.pluginEntries];
|
||||
}
|
||||
|
||||
type ClassifiedEntry = {
|
||||
readonly path: string;
|
||||
readonly kind: ConfigDocBaselineEntry["kind"];
|
||||
readonly classification?: CoverageClassification;
|
||||
};
|
||||
|
||||
type UnmatchedMonitoredPattern = {
|
||||
readonly pattern: string;
|
||||
};
|
||||
|
||||
const args = new Set(process.argv.slice(2));
|
||||
const json = args.has("--json");
|
||||
const check = args.has("--check");
|
||||
const showCovered = args.has("--show-covered");
|
||||
|
||||
if (args.has("--help")) {
|
||||
console.log(`Usage: pnpm policy:config-coverage [--check] [--json] [--show-covered]
|
||||
|
||||
Internal maintainer report for Policy config coverage.
|
||||
|
||||
Default mode is report-only and exits 0 even when paths are unclassified.
|
||||
Use --check when a policy maintainer intentionally wants unclassified or stale
|
||||
coverage entries to fail locally.`);
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const configPath = path.join(repoRoot, "scripts/lib/policy-config-coverage.jsonc");
|
||||
|
||||
const config = JSON5.parse(await fs.readFile(configPath, "utf8")) as CoverageConfig;
|
||||
const { baseline } = await renderConfigDocBaselineArtifacts();
|
||||
const monitoredEntries = flattenConfigDocBaselineEntries(baseline)
|
||||
.filter((entry) => !entry.hasChildren)
|
||||
.filter((entry) => matchesAny(config.monitored, entry.path))
|
||||
.toSorted((left, right) => left.path.localeCompare(right.path));
|
||||
const leafEntries = flattenConfigDocBaselineEntries(baseline).filter((entry) => !entry.hasChildren);
|
||||
const unmatchedMonitored = config.monitored
|
||||
.filter(
|
||||
(pattern) =>
|
||||
!leafEntries.some((entry) => pathMatchesPattern(pattern, entry.path)) &&
|
||||
!config.classifications.some(
|
||||
(item) => item.allowNoSchemaPath === true && pathMatchesPattern(item.pattern, pattern),
|
||||
),
|
||||
)
|
||||
.map((pattern) => ({ pattern }))
|
||||
.toSorted((left, right) => left.pattern.localeCompare(right.pattern));
|
||||
|
||||
const classified: ClassifiedEntry[] = monitoredEntries.map((entry) => ({
|
||||
path: entry.path,
|
||||
kind: entry.kind,
|
||||
classification: config.classifications.find((item) =>
|
||||
pathMatchesPattern(item.pattern, entry.path),
|
||||
),
|
||||
}));
|
||||
const unclassified = classified.filter((entry) => entry.classification === undefined);
|
||||
const stale = config.classifications.filter(
|
||||
(item) =>
|
||||
item.allowNoSchemaPath !== true &&
|
||||
!monitoredEntries.some((entry) => pathMatchesPattern(item.pattern, entry.path)),
|
||||
);
|
||||
const summaryCounts = summarize(classified);
|
||||
|
||||
if (json) {
|
||||
console.log(
|
||||
JSON.stringify(
|
||||
{
|
||||
ok: unclassified.length === 0 && stale.length === 0 && unmatchedMonitored.length === 0,
|
||||
monitoredPaths: monitoredEntries.length,
|
||||
counts: summaryCounts,
|
||||
unclassified,
|
||||
unmatchedMonitored,
|
||||
stale,
|
||||
},
|
||||
null,
|
||||
2,
|
||||
),
|
||||
);
|
||||
} else {
|
||||
printTextReport({
|
||||
monitoredPaths: monitoredEntries.length,
|
||||
counts: summaryCounts,
|
||||
unclassified,
|
||||
unmatchedMonitored,
|
||||
stale,
|
||||
classified,
|
||||
});
|
||||
}
|
||||
|
||||
if (check && (unclassified.length > 0 || stale.length > 0 || unmatchedMonitored.length > 0)) {
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
function printTextReport(input: {
|
||||
readonly monitoredPaths: number;
|
||||
readonly counts: Record<string, number>;
|
||||
readonly unclassified: readonly ClassifiedEntry[];
|
||||
readonly unmatchedMonitored: readonly UnmatchedMonitoredPattern[];
|
||||
readonly stale: readonly CoverageClassification[];
|
||||
readonly classified: readonly ClassifiedEntry[];
|
||||
}): void {
|
||||
console.log(`Policy config coverage: ${input.monitoredPaths} monitored config leaf paths`);
|
||||
for (const [key, count] of Object.entries(input.counts).toSorted(([a], [b]) =>
|
||||
a.localeCompare(b),
|
||||
)) {
|
||||
console.log(` ${key}: ${count}`);
|
||||
}
|
||||
|
||||
if (input.unclassified.length > 0) {
|
||||
console.log("\nUnclassified config paths:");
|
||||
for (const entry of input.unclassified) {
|
||||
console.log(` - ${entry.path} (${entry.kind})`);
|
||||
}
|
||||
console.log(
|
||||
"\nClassify each as observed, ignored, out-of-scope, or deferred in scripts/lib/policy-config-coverage.jsonc.",
|
||||
);
|
||||
} else {
|
||||
console.log("\nNo unclassified monitored config paths.");
|
||||
}
|
||||
|
||||
if (input.unmatchedMonitored.length > 0) {
|
||||
console.log("\nMonitored patterns with no matching config paths:");
|
||||
for (const entry of input.unmatchedMonitored) {
|
||||
console.log(` - ${entry.pattern}`);
|
||||
}
|
||||
} else {
|
||||
console.log("\nNo monitored patterns without matching config paths.");
|
||||
}
|
||||
|
||||
if (input.stale.length > 0) {
|
||||
console.log("\nStale coverage classifications:");
|
||||
for (const entry of input.stale) {
|
||||
console.log(` - ${entry.pattern} (${entry.area}, ${entry.status})`);
|
||||
}
|
||||
}
|
||||
|
||||
if (showCovered) {
|
||||
console.log("\nCovered paths:");
|
||||
for (const entry of input.classified) {
|
||||
const classification = entry.classification;
|
||||
console.log(
|
||||
` - ${entry.path}: ${classification?.area ?? "unclassified"} / ${
|
||||
classification?.status ?? "unclassified"
|
||||
}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function summarize(entries: readonly ClassifiedEntry[]): Record<string, number> {
|
||||
const counts: Record<string, number> = {};
|
||||
for (const entry of entries) {
|
||||
const key =
|
||||
entry.classification === undefined
|
||||
? "unclassified"
|
||||
: `${entry.classification.area}.${entry.classification.status}`;
|
||||
counts[key] = (counts[key] ?? 0) + 1;
|
||||
}
|
||||
return counts;
|
||||
}
|
||||
|
||||
function matchesAny(patterns: readonly string[], value: string): boolean {
|
||||
return patterns.some((pattern) => pathMatchesPattern(pattern, value));
|
||||
}
|
||||
|
||||
function pathMatchesPattern(pattern: string, value: string): boolean {
|
||||
const patternParts = pattern.split(".");
|
||||
const valueParts = value.split(".");
|
||||
return matchesParts(patternParts, valueParts);
|
||||
}
|
||||
|
||||
function matchesParts(patternParts: readonly string[], valueParts: readonly string[]): boolean {
|
||||
if (patternParts.length === 0) {
|
||||
return valueParts.length === 0;
|
||||
}
|
||||
const [head, ...tail] = patternParts;
|
||||
if (head === "**") {
|
||||
if (tail.length === 0) {
|
||||
return true;
|
||||
}
|
||||
for (let index = 0; index <= valueParts.length; index += 1) {
|
||||
if (matchesParts(tail, valueParts.slice(index))) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
if (valueParts.length === 0) {
|
||||
return false;
|
||||
}
|
||||
if (head !== "*" && head !== valueParts[0]) {
|
||||
return false;
|
||||
}
|
||||
return matchesParts(tail, valueParts.slice(1));
|
||||
}
|
||||
176
scripts/check-release-metadata-only.mjs
Normal file
176
scripts/check-release-metadata-only.mjs
Normal file
@@ -0,0 +1,176 @@
|
||||
#!/usr/bin/env node
|
||||
// Validates release metadata-only changed scopes for CI routing.
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { RELEASE_METADATA_PATHS } from "./changed-lanes.mjs";
|
||||
|
||||
const VERSION_ONLY_TEXT_PATHS = new Set([
|
||||
"apps/android/Config/Version.properties",
|
||||
"apps/android/version.json",
|
||||
"apps/macos/Sources/OpenClaw/Resources/Info.plist",
|
||||
]);
|
||||
|
||||
function normalizePath(input) {
|
||||
return String(input ?? "")
|
||||
.trim()
|
||||
.replaceAll("\\", "/")
|
||||
.replace(/^\.\/+/u, "");
|
||||
}
|
||||
|
||||
function readRefOptionValue(argv, index, optionName) {
|
||||
const value = argv[index + 1];
|
||||
if (value === undefined || value === "" || value.startsWith("-")) {
|
||||
throw new Error(`Expected ${optionName} <ref>.`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
export function parseArgs(argv) {
|
||||
const separatorIndex = argv.indexOf("--");
|
||||
const flagArgv = separatorIndex === -1 ? argv : argv.slice(0, separatorIndex);
|
||||
const explicitPaths =
|
||||
separatorIndex === -1 ? [] : argv.slice(separatorIndex + 1).map(normalizePath);
|
||||
const args = { staged: false, base: "origin/main", head: "HEAD", paths: [] };
|
||||
for (let index = 0; index < flagArgv.length; index += 1) {
|
||||
const arg = flagArgv[index];
|
||||
if (arg === "--staged") {
|
||||
args.staged = true;
|
||||
} else if (arg === "--base") {
|
||||
args.base = readRefOptionValue(flagArgv, index, arg);
|
||||
index += 1;
|
||||
} else if (arg === "--head") {
|
||||
args.head = readRefOptionValue(flagArgv, index, arg);
|
||||
index += 1;
|
||||
} else if (arg.startsWith("-")) {
|
||||
throw new Error(`Unknown option: ${arg}`);
|
||||
} else {
|
||||
args.paths.push(normalizePath(arg));
|
||||
}
|
||||
}
|
||||
args.paths.push(...explicitPaths);
|
||||
return args;
|
||||
}
|
||||
|
||||
function git(args) {
|
||||
return execFileSync("git", args, {
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
encoding: "utf8",
|
||||
maxBuffer: 16 * 1024 * 1024,
|
||||
});
|
||||
}
|
||||
|
||||
function listChangedPaths(args) {
|
||||
if (args.paths.length > 0) {
|
||||
return [...new Set(args.paths.filter(Boolean))].toSorted((left, right) =>
|
||||
left.localeCompare(right),
|
||||
);
|
||||
}
|
||||
const diffArgs = args.staged
|
||||
? ["diff", "--cached", "--name-only", "--diff-filter=ACMR"]
|
||||
: ["diff", "--name-only", "--diff-filter=ACMR", `${args.base}...${args.head}`];
|
||||
return git(diffArgs)
|
||||
.split("\n")
|
||||
.map(normalizePath)
|
||||
.filter(Boolean)
|
||||
.toSorted((left, right) => left.localeCompare(right));
|
||||
}
|
||||
|
||||
function readBlob(ref, filePath) {
|
||||
if (ref === "WORKTREE") {
|
||||
return readFileSync(filePath, "utf8");
|
||||
}
|
||||
return git(["show", `${ref}:${filePath}`]);
|
||||
}
|
||||
|
||||
function refsFor(args) {
|
||||
return args.staged ? { before: "HEAD", after: "" } : { before: args.base, after: args.head };
|
||||
}
|
||||
|
||||
function readBeforeAfter(args, filePath) {
|
||||
const refs = refsFor(args);
|
||||
const before = readBlob(refs.before, filePath);
|
||||
let after = readBlob(refs.after, filePath);
|
||||
if (!args.staged && existsSync(filePath)) {
|
||||
const worktree = readBlob("WORKTREE", filePath);
|
||||
if (worktree !== after) {
|
||||
after = worktree;
|
||||
}
|
||||
}
|
||||
return {
|
||||
before,
|
||||
after,
|
||||
};
|
||||
}
|
||||
|
||||
function stripPackageVersion(raw) {
|
||||
const parsed = JSON.parse(raw);
|
||||
delete parsed.version;
|
||||
return stableJson(parsed);
|
||||
}
|
||||
|
||||
function stableJson(value) {
|
||||
if (Array.isArray(value)) {
|
||||
return `[${value.map(stableJson).join(",")}]`;
|
||||
}
|
||||
if (value && typeof value === "object") {
|
||||
return `{${Object.keys(value)
|
||||
.toSorted((left, right) => left.localeCompare(right))
|
||||
.map((key) => `${JSON.stringify(key)}:${stableJson(value[key])}`)
|
||||
.join(",")}}`;
|
||||
}
|
||||
return JSON.stringify(value);
|
||||
}
|
||||
|
||||
function normalizeVersionText(raw) {
|
||||
return raw
|
||||
.replace(/\b20\d{2}\.\d{1,2}\.\d{1,2}(?:-beta\.\d+|-\d+)?\b/gu, "<OPENCLAW_VERSION>")
|
||||
.replace(/\b20\d{6}(?:\d{2})?\b/gu, "<OPENCLAW_BUILD>");
|
||||
}
|
||||
|
||||
function fail(message) {
|
||||
console.error(`[release-metadata] ${message}`);
|
||||
process.exitCode = 1;
|
||||
}
|
||||
|
||||
export function main(argv = process.argv.slice(2)) {
|
||||
const args = parseArgs(argv);
|
||||
const paths = listChangedPaths(args);
|
||||
|
||||
for (const filePath of paths) {
|
||||
if (!RELEASE_METADATA_PATHS.has(filePath)) {
|
||||
fail(`${filePath}: not a release metadata path; run the normal changed gate`);
|
||||
}
|
||||
}
|
||||
|
||||
if (paths.includes("package.json")) {
|
||||
const { before, after } = readBeforeAfter(args, "package.json");
|
||||
if (stripPackageVersion(before) !== stripPackageVersion(after)) {
|
||||
fail("package.json changed outside the top-level version field");
|
||||
}
|
||||
}
|
||||
|
||||
for (const filePath of paths) {
|
||||
if (!VERSION_ONLY_TEXT_PATHS.has(filePath)) {
|
||||
continue;
|
||||
}
|
||||
const { before, after } = readBeforeAfter(args, filePath);
|
||||
if (normalizeVersionText(before) !== normalizeVersionText(after)) {
|
||||
fail(`${filePath}: changed outside recognized version/build literals`);
|
||||
}
|
||||
}
|
||||
|
||||
if (process.exitCode) {
|
||||
process.exit(process.exitCode);
|
||||
}
|
||||
console.error(`[release-metadata] ok (${paths.length} files)`);
|
||||
}
|
||||
|
||||
if (process.argv[1] && path.resolve(process.argv[1]) === path.resolve(import.meta.filename)) {
|
||||
try {
|
||||
main();
|
||||
} catch (error) {
|
||||
console.error(error instanceof Error ? error.message : String(error));
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
276
scripts/check-runtime-sidecar-loaders.mjs
Normal file
276
scripts/check-runtime-sidecar-loaders.mjs
Normal file
@@ -0,0 +1,276 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Finds hidden local runtime sidecar loaders missing tsdown entries.
|
||||
import { promises as fs } from "node:fs";
|
||||
import path from "node:path";
|
||||
import ts from "typescript";
|
||||
import {
|
||||
collectTypeScriptFilesFromRoots,
|
||||
resolveRepoRoot,
|
||||
runAsScript,
|
||||
toLine,
|
||||
unwrapExpression,
|
||||
} from "./lib/ts-guard-utils.mjs";
|
||||
|
||||
const repoRoot = resolveRepoRoot(import.meta.url);
|
||||
const defaultSourceRoots = [path.join(repoRoot, "src"), path.join(repoRoot, "extensions")];
|
||||
const localRuntimeSpecifierPattern = /^\.{1,2}\/.*\.runtime\.(?:js|ts)$/;
|
||||
|
||||
function toPosixPath(value) {
|
||||
return value.split(path.sep).join("/");
|
||||
}
|
||||
|
||||
function normalizeRelativePath(value) {
|
||||
return path.posix.normalize(toPosixPath(value).replace(/^\.\//, ""));
|
||||
}
|
||||
|
||||
function unwrapInitializer(expression) {
|
||||
let current = unwrapExpression(expression);
|
||||
while (ts.isSatisfiesExpression(current)) {
|
||||
current = unwrapExpression(current.expression);
|
||||
}
|
||||
return current;
|
||||
}
|
||||
|
||||
function readStringLiteral(node) {
|
||||
if (ts.isStringLiteral(node) || ts.isNoSubstitutionTemplateLiteral(node)) {
|
||||
return node.text;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function readArrayStrings(node) {
|
||||
const expression = unwrapInitializer(node);
|
||||
if (!ts.isArrayLiteralExpression(expression)) {
|
||||
return null;
|
||||
}
|
||||
const values = [];
|
||||
for (const element of expression.elements) {
|
||||
const value = readStringLiteral(unwrapInitializer(element));
|
||||
if (value === null) {
|
||||
return null;
|
||||
}
|
||||
values.push(value);
|
||||
}
|
||||
return values;
|
||||
}
|
||||
|
||||
function isCreateRequireCall(node, createRequireNames) {
|
||||
return (
|
||||
ts.isCallExpression(node) &&
|
||||
ts.isIdentifier(node.expression) &&
|
||||
createRequireNames.has(node.expression.text)
|
||||
);
|
||||
}
|
||||
|
||||
function isLocalRuntimeSpecifier(specifier) {
|
||||
return localRuntimeSpecifierPattern.test(specifier);
|
||||
}
|
||||
|
||||
function resolveRuntimeSpecifierSource(importerPath, specifier) {
|
||||
const importerDir = path.posix.dirname(normalizeRelativePath(importerPath));
|
||||
const resolved = path.posix.normalize(path.posix.join(importerDir, specifier));
|
||||
return resolved.replace(/\.js$/, ".ts");
|
||||
}
|
||||
|
||||
function readObjectEntrySources(entry) {
|
||||
if (!entry || Array.isArray(entry) || typeof entry !== "object") {
|
||||
return [];
|
||||
}
|
||||
return Object.values(entry).filter((value) => typeof value === "string");
|
||||
}
|
||||
|
||||
/**
|
||||
* Collects explicit source entry files from tsdown configuration.
|
||||
*/
|
||||
export function collectTsdownEntrySources(config) {
|
||||
const configs = Array.isArray(config) ? config : [config];
|
||||
return new Set(
|
||||
configs.flatMap((entry) => readObjectEntrySources(entry?.entry)).map(normalizeRelativePath),
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Finds local runtime require loaders not represented as explicit tsdown entries.
|
||||
*/
|
||||
export function findRuntimeSidecarLoaderViolations(content, importerPath, explicitEntrySources) {
|
||||
const sourceFile = ts.createSourceFile(importerPath, content, ts.ScriptTarget.Latest, true);
|
||||
const createRequireNames = new Set();
|
||||
const requireNames = new Set();
|
||||
const stringConstants = new Map();
|
||||
const stringArrays = new Map();
|
||||
const forOfRuntimeValues = [];
|
||||
const violations = [];
|
||||
const seen = new Set();
|
||||
|
||||
const currentForOfValueMap = () => {
|
||||
const merged = new Map();
|
||||
for (const scope of forOfRuntimeValues) {
|
||||
for (const [name, values] of scope) {
|
||||
merged.set(name, values);
|
||||
}
|
||||
}
|
||||
return merged;
|
||||
};
|
||||
|
||||
const addSpecifier = (specifier, node) => {
|
||||
if (!isLocalRuntimeSpecifier(specifier)) {
|
||||
return;
|
||||
}
|
||||
const sourcePath = resolveRuntimeSpecifierSource(importerPath, specifier);
|
||||
if (explicitEntrySources.has(sourcePath)) {
|
||||
return;
|
||||
}
|
||||
const key = `${sourcePath}:${toLine(sourceFile, node)}`;
|
||||
if (seen.has(key)) {
|
||||
return;
|
||||
}
|
||||
seen.add(key);
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node),
|
||||
specifier,
|
||||
sourcePath,
|
||||
reason:
|
||||
`hidden local runtime loader "${specifier}" resolves to ${sourcePath}, ` +
|
||||
"but that source is not an explicit tsdown entry",
|
||||
});
|
||||
};
|
||||
|
||||
const readRequireArgumentSpecifiers = (node) => {
|
||||
const arg = node.arguments[0];
|
||||
if (!arg) {
|
||||
return [];
|
||||
}
|
||||
const unwrapped = unwrapInitializer(arg);
|
||||
const literal = readStringLiteral(unwrapped);
|
||||
if (literal !== null) {
|
||||
return [literal];
|
||||
}
|
||||
if (ts.isIdentifier(unwrapped)) {
|
||||
const loopValues = currentForOfValueMap().get(unwrapped.text);
|
||||
if (loopValues) {
|
||||
return loopValues;
|
||||
}
|
||||
const constant = stringConstants.get(unwrapped.text);
|
||||
if (constant !== undefined) {
|
||||
return [constant];
|
||||
}
|
||||
}
|
||||
return [];
|
||||
};
|
||||
|
||||
const visit = (node) => {
|
||||
if (ts.isImportDeclaration(node) && ts.isStringLiteral(node.moduleSpecifier)) {
|
||||
if (node.moduleSpecifier.text === "node:module") {
|
||||
const bindings = node.importClause?.namedBindings;
|
||||
if (bindings && ts.isNamedImports(bindings)) {
|
||||
for (const element of bindings.elements) {
|
||||
if (
|
||||
element.propertyName?.text === "createRequire" ||
|
||||
element.name.text === "createRequire"
|
||||
) {
|
||||
createRequireNames.add(element.name.text);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (ts.isVariableDeclaration(node) && ts.isIdentifier(node.name) && node.initializer) {
|
||||
const initializer = unwrapInitializer(node.initializer);
|
||||
const literal = readStringLiteral(initializer);
|
||||
if (literal !== null) {
|
||||
stringConstants.set(node.name.text, literal);
|
||||
}
|
||||
const arrayValues = readArrayStrings(initializer);
|
||||
if (arrayValues) {
|
||||
stringArrays.set(node.name.text, arrayValues);
|
||||
}
|
||||
if (isCreateRequireCall(initializer, createRequireNames)) {
|
||||
requireNames.add(node.name.text);
|
||||
}
|
||||
}
|
||||
|
||||
if (ts.isForOfStatement(node)) {
|
||||
const initializer = node.initializer;
|
||||
const expression = unwrapInitializer(node.expression);
|
||||
if (
|
||||
ts.isVariableDeclarationList(initializer) &&
|
||||
initializer.declarations.length === 1 &&
|
||||
ts.isIdentifier(initializer.declarations[0].name) &&
|
||||
ts.isIdentifier(expression)
|
||||
) {
|
||||
const values = stringArrays.get(expression.text);
|
||||
if (values) {
|
||||
forOfRuntimeValues.push(new Map([[initializer.declarations[0].name.text, values]]));
|
||||
ts.forEachChild(node.statement, visit);
|
||||
forOfRuntimeValues.pop();
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (
|
||||
ts.isCallExpression(node) &&
|
||||
ts.isIdentifier(node.expression) &&
|
||||
requireNames.has(node.expression.text)
|
||||
) {
|
||||
for (const specifier of readRequireArgumentSpecifiers(node)) {
|
||||
addSpecifier(specifier, node);
|
||||
}
|
||||
}
|
||||
|
||||
ts.forEachChild(node, visit);
|
||||
};
|
||||
|
||||
visit(sourceFile);
|
||||
return violations;
|
||||
}
|
||||
|
||||
/**
|
||||
* Collects runtime sidecar loader violations across configured roots.
|
||||
*/
|
||||
export async function collectRuntimeSidecarLoaderViolations(params) {
|
||||
const files = await collectTypeScriptFilesFromRoots(params.sourceRoots, {
|
||||
extraTestSuffixes: [".test-support.ts", ".test-helpers.ts"],
|
||||
});
|
||||
const violations = [];
|
||||
for (const filePath of files) {
|
||||
if (filePath.endsWith(".d.ts")) {
|
||||
continue;
|
||||
}
|
||||
const relativePath = normalizeRelativePath(path.relative(params.repoRoot, filePath));
|
||||
const content = await fs.readFile(filePath, "utf8");
|
||||
for (const violation of findRuntimeSidecarLoaderViolations(
|
||||
content,
|
||||
relativePath,
|
||||
params.explicitEntrySources,
|
||||
)) {
|
||||
violations.push({ path: relativePath, ...violation });
|
||||
}
|
||||
}
|
||||
return violations;
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const { default: tsdownConfig } = await import("../tsdown.config.ts");
|
||||
const violations = await collectRuntimeSidecarLoaderViolations({
|
||||
repoRoot,
|
||||
sourceRoots: defaultSourceRoots,
|
||||
explicitEntrySources: collectTsdownEntrySources(tsdownConfig),
|
||||
});
|
||||
if (violations.length === 0) {
|
||||
console.log("runtime-sidecar-loaders: local runtime sidecar loaders look OK.");
|
||||
return;
|
||||
}
|
||||
console.error("runtime-sidecar-loaders: hidden local runtime loaders found:");
|
||||
for (const violation of violations) {
|
||||
console.error(
|
||||
`- ${violation.path}:${violation.line}: ${violation.reason}. ` +
|
||||
'Use cached import("./x.runtime.js") or add the sidecar as a stable tsdown entry.',
|
||||
);
|
||||
}
|
||||
process.exitCode = 1;
|
||||
}
|
||||
|
||||
runAsScript(import.meta.url, main);
|
||||
24
scripts/check-sdk-package-extension-import-boundary.mjs
Normal file
24
scripts/check-sdk-package-extension-import-boundary.mjs
Normal file
@@ -0,0 +1,24 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Runs the SDK-package extension import boundary checker.
|
||||
import { createExtensionImportBoundaryChecker } from "./lib/extension-import-boundary-checker.mjs";
|
||||
import { runAsScript } from "./lib/ts-guard-utils.mjs";
|
||||
|
||||
const checker = createExtensionImportBoundaryChecker({
|
||||
roots: ["src/plugin-sdk", "packages"],
|
||||
boundaryLabel: "sdk/package",
|
||||
rule: "Rule: src/plugin-sdk/** and packages/** must not import bundled plugin files",
|
||||
cleanMessage: "No sdk/package import boundary violations found.",
|
||||
inventoryTitle: "SDK/package extension import boundary inventory:",
|
||||
skipSourcesWithoutBundledPluginPrefix: true,
|
||||
shouldSkipFile(relativeFile) {
|
||||
return relativeFile.startsWith("packages/plugin-sdk/dist/");
|
||||
},
|
||||
});
|
||||
|
||||
/**
|
||||
* Entrypoint for the SDK-package extension import boundary checker.
|
||||
*/
|
||||
export const main = checker.main;
|
||||
|
||||
runAsScript(import.meta.url, main);
|
||||
763
scripts/check-session-accessor-boundary.mjs
Normal file
763
scripts/check-session-accessor-boundary.mjs
Normal file
@@ -0,0 +1,763 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import ts from "typescript";
|
||||
import {
|
||||
collectFileViolations,
|
||||
resolveRepoRoot,
|
||||
resolveSourceRoots,
|
||||
runAsScript,
|
||||
toLine,
|
||||
unwrapExpression,
|
||||
} from "./lib/ts-guard-utils.mjs";
|
||||
|
||||
const legacyReaderNames = new Set([
|
||||
"loadSessionStore",
|
||||
"readSessionEntries",
|
||||
"readSessionEntry",
|
||||
"readSessionStoreReadOnly",
|
||||
"resolveSessionStoreEntry",
|
||||
]);
|
||||
const legacyWholeStoreAccessNames = new Set([
|
||||
...legacyReaderNames,
|
||||
"saveSessionStore",
|
||||
"updateSessionStore",
|
||||
]);
|
||||
const legacyWriterNames = new Set([
|
||||
"applySessionStoreEntryPatch",
|
||||
"saveSessionStore",
|
||||
"updateSessionStore",
|
||||
"updateSessionStoreEntry",
|
||||
]);
|
||||
const legacyTranscriptWriterNames = new Set([
|
||||
"appendSessionTranscriptMessage",
|
||||
"emitSessionTranscriptUpdate",
|
||||
]);
|
||||
const sessionCreateLifecycleWriterNames = new Set([
|
||||
"applySessionStoreEntryPatch",
|
||||
"saveSessionStore",
|
||||
"updateSessionStore",
|
||||
"updateSessionStoreEntry",
|
||||
"ensureSessionTranscriptFile",
|
||||
]);
|
||||
const legacyManualCompactTrimNames = new Set([
|
||||
"archiveFileOnDisk",
|
||||
"readRecentSessionTranscriptLines",
|
||||
]);
|
||||
const legacyLifecycleCleanupNames = new Set([
|
||||
"archiveRemovedSessionTranscripts",
|
||||
"cleanupArchivedSessionTranscripts",
|
||||
]);
|
||||
const sessionStoreRuntimeFileBackedCompatNames = new Set([
|
||||
"loadSessionStore",
|
||||
"readSessionEntries",
|
||||
"readSessionEntry",
|
||||
"readLatestAssistantTextFromSessionTranscript",
|
||||
"readSessionStoreReadOnly",
|
||||
"resolveAndPersistSessionFile",
|
||||
"resolveSessionFilePath",
|
||||
"resolveSessionStoreEntry",
|
||||
"saveSessionStore",
|
||||
"updateSessionStore",
|
||||
]);
|
||||
const embeddedAgentSessionFileRuntimeNames = new Set(["resolveSessionFilePath"]);
|
||||
|
||||
export const allowedSessionStoreRuntimeFileBackedCompatExports = new Set([
|
||||
"loadSessionStore",
|
||||
"readLatestAssistantTextFromSessionTranscript",
|
||||
"resolveAndPersistSessionFile",
|
||||
"resolveSessionFilePath",
|
||||
"resolveSessionStoreEntry",
|
||||
"saveSessionStore",
|
||||
"updateSessionStore",
|
||||
]);
|
||||
|
||||
export const migratedSessionAccessorFiles = new Set([
|
||||
"packages/memory-host-sdk/src/host/session-files.ts",
|
||||
"src/acp/runtime/session-meta.ts",
|
||||
"src/agents/acp-spawn.ts",
|
||||
"src/agents/auth-profiles/session-override.ts",
|
||||
"src/agents/embedded-agent-runner/compaction-successor-transcript.ts",
|
||||
"src/agents/embedded-agent-runner/run/attempt.ts",
|
||||
"src/agents/embedded-agent-runner/tool-result-truncation.ts",
|
||||
"src/agents/embedded-agent-runner/transcript-rewrite.ts",
|
||||
"src/agents/embedded-agent-runner/transcript-runtime-state.ts",
|
||||
"src/agents/live-model-switch.ts",
|
||||
"src/agents/subagent-control.ts",
|
||||
"src/agents/subagent-registry-helpers.ts",
|
||||
"src/auto-reply/reply/abort.ts",
|
||||
"src/auto-reply/reply/agent-runner-helpers.ts",
|
||||
"src/auto-reply/reply/agent-runner.ts",
|
||||
"src/auto-reply/reply/commands-subagents/action-info.ts",
|
||||
"src/auto-reply/reply/followup-runner.ts",
|
||||
"src/auto-reply/reply/queue/drain.ts",
|
||||
"src/commands/export-trajectory.ts",
|
||||
"src/commands/health.ts",
|
||||
"src/commands/sandbox-explain.ts",
|
||||
"src/commands/sessions-tail.ts",
|
||||
"src/commands/sessions.ts",
|
||||
"src/commands/status.agent-local.ts",
|
||||
"src/commands/status.summary.ts",
|
||||
"src/commands/tasks.ts",
|
||||
"src/config/sessions/combined-store-gateway.ts",
|
||||
"src/cron/isolated-agent/delivery-target.ts",
|
||||
"src/cron/service/timer.ts",
|
||||
"src/gateway/session-compaction-checkpoints.ts",
|
||||
"src/gateway/session-history-state.ts",
|
||||
"src/gateway/sessions-history-http.ts",
|
||||
"src/gateway/session-utils.ts",
|
||||
"src/gateway/managed-image-attachments.ts",
|
||||
"src/gateway/boot.ts",
|
||||
"src/gateway/server-methods/artifacts.ts",
|
||||
"src/gateway/server-methods/chat.ts",
|
||||
"src/gateway/sessions-resolve.ts",
|
||||
"src/gateway/server-methods/sessions-files.ts",
|
||||
"src/gateway/server-methods/sessions.ts",
|
||||
"src/gateway/server-session-events.ts",
|
||||
"src/gateway/session-reset-service.ts",
|
||||
"src/infra/outbound/message-action-tts.ts",
|
||||
"src/agents/tools/embedded-gateway-stub.ts",
|
||||
"src/agents/tools/session-status-tool.ts",
|
||||
"src/agents/tools/sessions-list-tool.ts",
|
||||
"src/plugins/host-hook-state.ts",
|
||||
"src/status/status-message.ts",
|
||||
"src/tui/embedded-backend.ts",
|
||||
]);
|
||||
|
||||
export const migratedBundledPluginSessionAccessorFiles = new Set([
|
||||
"extensions/codex/src/conversation-binding.ts",
|
||||
"extensions/discord/src/monitor/native-command-model-picker-ui.ts",
|
||||
"extensions/discord/src/monitor/native-command-model-picker-apply.ts",
|
||||
"extensions/discord/src/monitor/thread-session-close.ts",
|
||||
"extensions/feishu/src/reasoning-preview.ts",
|
||||
"extensions/memory-core/src/dreaming-phases.ts",
|
||||
"extensions/memory-core/src/dreaming-narrative.ts",
|
||||
"extensions/mattermost/src/mattermost/model-picker.ts",
|
||||
"extensions/matrix/src/matrix/monitor/handler.ts",
|
||||
"extensions/matrix/src/session-route.ts",
|
||||
"extensions/slack/src/monitor/slash.ts",
|
||||
"extensions/telegram/src/bot-core.ts",
|
||||
"extensions/telegram/src/bot-handlers.runtime.ts",
|
||||
"extensions/telegram/src/bot.ts",
|
||||
"extensions/telegram/src/bot-message-dispatch.ts",
|
||||
"extensions/telegram/src/bot-native-commands.ts",
|
||||
"extensions/voice-call/src/response-generator.ts",
|
||||
"extensions/whatsapp/src/auto-reply/monitor/group-activation.ts",
|
||||
]);
|
||||
|
||||
export const migratedEmbeddedAgentSessionTargetFiles = new Set([
|
||||
"extensions/voice-call/src/response-generator.ts",
|
||||
]);
|
||||
|
||||
export const migratedSessionAccessorWriteFiles = new Set([
|
||||
"src/acp/runtime/session-meta.ts",
|
||||
"src/agents/auth-profiles/session-override.ts",
|
||||
"src/agents/command/attempt-execution.shared.ts",
|
||||
"src/agents/command/session-store.ts",
|
||||
"src/agents/embedded-agent-runner/run.ts",
|
||||
"src/agents/embedded-agent-runner/run/attempt.ts",
|
||||
"src/agents/live-model-switch.ts",
|
||||
"src/agents/main-session-restart-recovery.ts",
|
||||
"src/auto-reply/reply/abort.ts",
|
||||
"src/agents/subagent-control.ts",
|
||||
"src/agents/subagent-registry-helpers.ts",
|
||||
"src/agents/tools/session-status-tool.ts",
|
||||
"src/auto-reply/reply/abort-cutoff.runtime.ts",
|
||||
"src/auto-reply/reply/agent-runner-cli-dispatch.ts",
|
||||
"src/auto-reply/reply/agent-runner-execution.ts",
|
||||
"src/auto-reply/reply/agent-runner-memory.ts",
|
||||
"src/auto-reply/reply/agent-runner-session-reset.ts",
|
||||
"src/auto-reply/reply/agent-runner.ts",
|
||||
"src/auto-reply/reply/body.ts",
|
||||
"src/auto-reply/reply/commands-acp/lifecycle.ts",
|
||||
"src/auto-reply/reply/commands-reset.ts",
|
||||
"src/auto-reply/reply/commands-session-store.ts",
|
||||
"src/auto-reply/reply/directive-handling.impl.ts",
|
||||
"src/auto-reply/reply/directive-handling.persist.ts",
|
||||
"src/auto-reply/reply/dispatch-from-config.runtime.ts",
|
||||
"src/auto-reply/reply/followup-runner.ts",
|
||||
"src/auto-reply/reply/get-reply.ts",
|
||||
"src/auto-reply/reply/model-selection.ts",
|
||||
"src/auto-reply/reply/session.ts",
|
||||
"src/auto-reply/reply/session-reset-model.ts",
|
||||
"src/auto-reply/reply/session-updates.ts",
|
||||
"src/auto-reply/reply/session-usage.ts",
|
||||
"src/commands/tasks.ts",
|
||||
"src/config/sessions/cleanup-service.ts",
|
||||
"src/gateway/boot.ts",
|
||||
"src/gateway/server-node-events.ts",
|
||||
"src/gateway/session-compaction-checkpoints.ts",
|
||||
"src/plugins/host-hook-cleanup.ts",
|
||||
"src/plugins/host-hook-state.ts",
|
||||
"src/tui/embedded-backend.ts",
|
||||
]);
|
||||
|
||||
export const migratedTranscriptWriterFiles = new Set([
|
||||
"src/agents/command/attempt-execution.ts",
|
||||
"src/agents/embedded-agent-runner/context-engine-maintenance.ts",
|
||||
"src/config/sessions/transcript.ts",
|
||||
"src/gateway/server-methods/chat.ts",
|
||||
"src/gateway/server-methods/chat-transcript-inject.ts",
|
||||
"src/sessions/user-turn-transcript.ts",
|
||||
]);
|
||||
|
||||
export const migratedSessionCompactManualTrimFiles = new Set([
|
||||
"src/gateway/server-methods/sessions.ts",
|
||||
]);
|
||||
|
||||
export const migratedSessionLifecycleCleanupFiles = new Set([
|
||||
"src/config/sessions/cleanup-service.ts",
|
||||
"src/cron/session-reaper.ts",
|
||||
"src/infra/heartbeat-runner.ts",
|
||||
]);
|
||||
|
||||
export const migratedMemoryHostSessionCorpusFiles = new Set([
|
||||
"packages/memory-host-sdk/src/host/session-files.ts",
|
||||
"packages/memory-host-sdk/src/host/session-transcript-corpus.ts",
|
||||
]);
|
||||
|
||||
const memoryHostSessionCorpusFunctionNames = new Set([
|
||||
"listSessionTranscriptCorpusEntriesForAgentSync",
|
||||
"listSessionTranscriptCorpusEntriesForAgent",
|
||||
"loadDreamingNarrativeTranscriptPathSetForAgent",
|
||||
"loadSessionTranscriptClassificationForAgent",
|
||||
"listSessionFilesForAgent",
|
||||
]);
|
||||
|
||||
const legacyMemoryHostSessionCorpusNames = new Set([
|
||||
"loadSessionTranscriptClassificationForSessionsDir",
|
||||
"readSessionTranscriptClassificationStore",
|
||||
]);
|
||||
|
||||
function normalizeRelativePath(filePath) {
|
||||
return filePath.replaceAll(path.sep, "/");
|
||||
}
|
||||
|
||||
function legacyNamesForFile(fileName) {
|
||||
const normalized = normalizeRelativePath(fileName);
|
||||
if (
|
||||
fileName === "source.ts" ||
|
||||
[...migratedBundledPluginSessionAccessorFiles].some((filePath) => normalized.endsWith(filePath))
|
||||
) {
|
||||
return legacyWholeStoreAccessNames;
|
||||
}
|
||||
return legacyReaderNames;
|
||||
}
|
||||
|
||||
function propertyAccessName(expression) {
|
||||
const unwrapped = unwrapExpression(expression);
|
||||
if (ts.isIdentifier(unwrapped)) {
|
||||
return unwrapped.text;
|
||||
}
|
||||
if (ts.isPropertyAccessExpression(unwrapped)) {
|
||||
return unwrapped.name.text;
|
||||
}
|
||||
if (ts.isElementAccessExpression(unwrapped) && ts.isStringLiteral(unwrapped.argumentExpression)) {
|
||||
return unwrapped.argumentExpression.text;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function propertyNameText(name) {
|
||||
if (ts.isIdentifier(name) || ts.isStringLiteralLike(name) || ts.isNumericLiteral(name)) {
|
||||
return name.text;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function bindingName(node) {
|
||||
if (node.propertyName && ts.isIdentifier(node.propertyName)) {
|
||||
return node.propertyName.text;
|
||||
}
|
||||
if (ts.isIdentifier(node.name)) {
|
||||
return node.name.text;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function findNamedBoundaryViolations(content, fileName, legacyNames, subject) {
|
||||
const sourceFile = ts.createSourceFile(fileName, content, ts.ScriptTarget.Latest, true);
|
||||
const violations = [];
|
||||
|
||||
const visit = (node) => {
|
||||
if (ts.isImportDeclaration(node)) {
|
||||
const namedBindings = node.importClause?.namedBindings;
|
||||
if (namedBindings && ts.isNamedImports(namedBindings)) {
|
||||
for (const specifier of namedBindings.elements) {
|
||||
const importedName = specifier.propertyName?.text ?? specifier.name.text;
|
||||
if (legacyNames.has(importedName)) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, specifier),
|
||||
reason: `imports ${subject} "${importedName}"`,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (ts.isBindingElement(node)) {
|
||||
const name = bindingName(node);
|
||||
if (name && legacyNames.has(name)) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node),
|
||||
reason: `aliases ${subject} "${name}"`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (ts.isPropertyAccessExpression(node) && legacyNames.has(node.name.text)) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node.name),
|
||||
reason: `references ${subject} "${node.name.text}"`,
|
||||
});
|
||||
}
|
||||
|
||||
if (
|
||||
ts.isElementAccessExpression(node) &&
|
||||
ts.isStringLiteral(node.argumentExpression) &&
|
||||
legacyNames.has(node.argumentExpression.text)
|
||||
) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node.argumentExpression),
|
||||
reason: `references ${subject} "${node.argumentExpression.text}"`,
|
||||
});
|
||||
}
|
||||
|
||||
if (ts.isCallExpression(node)) {
|
||||
const calleeName = propertyAccessName(node.expression);
|
||||
if (
|
||||
calleeName &&
|
||||
legacyNames.has(calleeName) &&
|
||||
ts.isIdentifier(unwrapExpression(node.expression))
|
||||
) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node.expression),
|
||||
reason: `calls ${subject} "${calleeName}"`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
ts.forEachChild(node, visit);
|
||||
};
|
||||
|
||||
visit(sourceFile);
|
||||
return violations;
|
||||
}
|
||||
|
||||
function findNamedSessionStoreViolations(content, fileName, legacyNames, legacyKind) {
|
||||
return findNamedBoundaryViolations(
|
||||
content,
|
||||
fileName,
|
||||
legacyNames,
|
||||
`legacy session store ${legacyKind}`,
|
||||
);
|
||||
}
|
||||
|
||||
export function collectSessionStoreRuntimeFileBackedCompatExports(content, fileName = "source.ts") {
|
||||
const sourceFile = ts.createSourceFile(fileName, content, ts.ScriptTarget.Latest, true);
|
||||
const exports = new Map();
|
||||
|
||||
const rememberExport = (node, exportedName, sourceName = exportedName) => {
|
||||
if (!sessionStoreRuntimeFileBackedCompatNames.has(sourceName)) {
|
||||
return;
|
||||
}
|
||||
exports.set(exportedName, {
|
||||
line: toLine(sourceFile, node),
|
||||
sourceName,
|
||||
});
|
||||
};
|
||||
|
||||
for (const statement of sourceFile.statements) {
|
||||
const isExported = statement.modifiers?.some(
|
||||
(modifier) => modifier.kind === ts.SyntaxKind.ExportKeyword,
|
||||
);
|
||||
if (isExported && ts.isVariableStatement(statement)) {
|
||||
for (const declaration of statement.declarationList.declarations) {
|
||||
if (ts.isIdentifier(declaration.name)) {
|
||||
rememberExport(declaration.name, declaration.name.text);
|
||||
}
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (isExported && ts.isFunctionDeclaration(statement) && statement.name) {
|
||||
rememberExport(statement.name, statement.name.text);
|
||||
continue;
|
||||
}
|
||||
if (
|
||||
ts.isExportDeclaration(statement) &&
|
||||
statement.exportClause &&
|
||||
ts.isNamedExports(statement.exportClause)
|
||||
) {
|
||||
for (const specifier of statement.exportClause.elements) {
|
||||
rememberExport(
|
||||
specifier,
|
||||
specifier.name.text,
|
||||
specifier.propertyName?.text ?? specifier.name.text,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return exports;
|
||||
}
|
||||
|
||||
export function findSessionStoreRuntimeFileBackedCompatExportViolations(
|
||||
content,
|
||||
fileName = "source.ts",
|
||||
) {
|
||||
const exports = collectSessionStoreRuntimeFileBackedCompatExports(content, fileName);
|
||||
const violations = [];
|
||||
for (const [exportedName, exported] of exports) {
|
||||
if (
|
||||
exportedName !== exported.sourceName ||
|
||||
!allowedSessionStoreRuntimeFileBackedCompatExports.has(exportedName)
|
||||
) {
|
||||
violations.push({
|
||||
line: exported.line,
|
||||
reason: `exports unratcheted file-backed SDK session helper "${exported.sourceName}"`,
|
||||
});
|
||||
}
|
||||
}
|
||||
return violations;
|
||||
}
|
||||
|
||||
export function findSessionAccessorBoundaryViolations(content, fileName = "source.ts") {
|
||||
const legacyNames = legacyNamesForFile(fileName);
|
||||
const legacyKind = legacyNames === legacyWholeStoreAccessNames ? "access" : "reader";
|
||||
return findNamedSessionStoreViolations(content, fileName, legacyNames, legacyKind);
|
||||
}
|
||||
|
||||
export function findEmbeddedAgentSessionTargetViolations(content, fileName = "source.ts") {
|
||||
const sourceFile = ts.createSourceFile(fileName, content, ts.ScriptTarget.Latest, true);
|
||||
const violations = findNamedBoundaryViolations(
|
||||
content,
|
||||
fileName,
|
||||
embeddedAgentSessionFileRuntimeNames,
|
||||
"legacy embedded-agent session file resolver",
|
||||
);
|
||||
|
||||
const recordDeprecatedSessionFile = (name) => {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, name),
|
||||
reason:
|
||||
'passes deprecated embedded-agent runtime identity field "sessionFile"; use sessionTarget',
|
||||
});
|
||||
};
|
||||
|
||||
const visitRunOptions = (options) => {
|
||||
for (const property of options.properties) {
|
||||
if (ts.isPropertyAssignment(property) && propertyNameText(property.name) === "sessionFile") {
|
||||
recordDeprecatedSessionFile(property.name);
|
||||
} else if (
|
||||
ts.isShorthandPropertyAssignment(property) &&
|
||||
property.name.text === "sessionFile"
|
||||
) {
|
||||
recordDeprecatedSessionFile(property.name);
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
const visit = (node) => {
|
||||
if (ts.isCallExpression(node) && propertyAccessName(node.expression) === "runEmbeddedAgent") {
|
||||
const options = unwrapExpression(node.arguments[0]);
|
||||
if (options && ts.isObjectLiteralExpression(options)) {
|
||||
visitRunOptions(options);
|
||||
}
|
||||
return;
|
||||
}
|
||||
ts.forEachChild(node, visit);
|
||||
};
|
||||
|
||||
visit(sourceFile);
|
||||
return violations;
|
||||
}
|
||||
|
||||
export function findSessionAccessorWriteBoundaryViolations(content, fileName = "source.ts") {
|
||||
return findNamedSessionStoreViolations(content, fileName, legacyWriterNames, "writer");
|
||||
}
|
||||
|
||||
export function findTranscriptWriterBoundaryViolations(content, fileName = "source.ts") {
|
||||
return findNamedBoundaryViolations(
|
||||
content,
|
||||
fileName,
|
||||
legacyTranscriptWriterNames,
|
||||
"legacy transcript writer",
|
||||
);
|
||||
}
|
||||
|
||||
export function findGatewaySessionCreateLifecycleViolations(content, fileName = "source.ts") {
|
||||
const sourceFile = ts.createSourceFile(fileName, content, ts.ScriptTarget.Latest, true);
|
||||
const violations = [];
|
||||
|
||||
const visitCreateHandler = (node) => {
|
||||
if (ts.isCallExpression(node)) {
|
||||
const calleeName = propertyAccessName(node.expression);
|
||||
if (calleeName && sessionCreateLifecycleWriterNames.has(calleeName)) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node.expression),
|
||||
reason: `calls legacy sessions.create lifecycle writer "${calleeName}"`,
|
||||
});
|
||||
}
|
||||
}
|
||||
ts.forEachChild(node, visitCreateHandler);
|
||||
};
|
||||
|
||||
const visit = (node) => {
|
||||
if (
|
||||
ts.isPropertyAssignment(node) &&
|
||||
ts.isStringLiteralLike(node.name) &&
|
||||
node.name.text === "sessions.create"
|
||||
) {
|
||||
visitCreateHandler(node.initializer);
|
||||
return;
|
||||
}
|
||||
ts.forEachChild(node, visit);
|
||||
};
|
||||
|
||||
visit(sourceFile);
|
||||
return violations;
|
||||
}
|
||||
|
||||
export function findSessionCompactManualTrimBoundaryViolations(content, fileName = "source.ts") {
|
||||
return findNamedSessionStoreViolations(
|
||||
content,
|
||||
fileName,
|
||||
legacyManualCompactTrimNames,
|
||||
"manual compact trim",
|
||||
);
|
||||
}
|
||||
|
||||
export function findSessionLifecycleCleanupBoundaryViolations(content, fileName = "source.ts") {
|
||||
return findNamedSessionStoreViolations(
|
||||
content,
|
||||
fileName,
|
||||
legacyLifecycleCleanupNames,
|
||||
"lifecycle cleanup",
|
||||
);
|
||||
}
|
||||
|
||||
function declarationName(node) {
|
||||
if (ts.isFunctionDeclaration(node) && node.name) {
|
||||
return node.name.text;
|
||||
}
|
||||
if (!ts.isVariableStatement(node)) {
|
||||
return null;
|
||||
}
|
||||
const declaration = node.declarationList.declarations[0];
|
||||
return declaration && ts.isIdentifier(declaration.name) ? declaration.name.text : null;
|
||||
}
|
||||
|
||||
function functionBodyForDeclaration(node) {
|
||||
if (ts.isFunctionDeclaration(node)) {
|
||||
return node.body ?? null;
|
||||
}
|
||||
if (!ts.isVariableStatement(node)) {
|
||||
return null;
|
||||
}
|
||||
const declaration = node.declarationList.declarations[0];
|
||||
const initializer = declaration?.initializer;
|
||||
if (initializer && (ts.isArrowFunction(initializer) || ts.isFunctionExpression(initializer))) {
|
||||
return initializer.body;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function collectTopLevelFunctionBodies(sourceFile) {
|
||||
const bodies = new Map();
|
||||
for (const statement of sourceFile.statements) {
|
||||
const name = declarationName(statement);
|
||||
const body = functionBodyForDeclaration(statement);
|
||||
if (name && body) {
|
||||
bodies.set(name, body);
|
||||
}
|
||||
}
|
||||
return bodies;
|
||||
}
|
||||
|
||||
export function findMemoryHostSessionCorpusBoundaryViolations(content, fileName = "source.ts") {
|
||||
const sourceFile = ts.createSourceFile(fileName, content, ts.ScriptTarget.Latest, true);
|
||||
const functionBodies = collectTopLevelFunctionBodies(sourceFile);
|
||||
const visitedFunctions = new Set();
|
||||
const violationKeys = new Set();
|
||||
const violations = [];
|
||||
|
||||
const visitCorpusBody = (node) => {
|
||||
if (ts.isCallExpression(node)) {
|
||||
const calleeName = propertyAccessName(node.expression);
|
||||
if (calleeName && legacyMemoryHostSessionCorpusNames.has(calleeName)) {
|
||||
const line = toLine(sourceFile, node.expression);
|
||||
const reason = `calls legacy memory-host session corpus helper "${calleeName}"`;
|
||||
const key = `${line}:${reason}`;
|
||||
if (!violationKeys.has(key)) {
|
||||
violationKeys.add(key);
|
||||
violations.push({ line, reason });
|
||||
}
|
||||
}
|
||||
if (calleeName && ts.isIdentifier(unwrapExpression(node.expression))) {
|
||||
const localBody = functionBodies.get(calleeName);
|
||||
if (localBody && !visitedFunctions.has(calleeName)) {
|
||||
visitedFunctions.add(calleeName);
|
||||
visitCorpusBody(localBody);
|
||||
}
|
||||
}
|
||||
}
|
||||
ts.forEachChild(node, visitCorpusBody);
|
||||
};
|
||||
|
||||
for (const name of memoryHostSessionCorpusFunctionNames) {
|
||||
const body = functionBodies.get(name);
|
||||
if (!body || visitedFunctions.has(name)) {
|
||||
continue;
|
||||
}
|
||||
visitedFunctions.add(name);
|
||||
visitCorpusBody(body);
|
||||
}
|
||||
|
||||
return violations;
|
||||
}
|
||||
|
||||
export async function main() {
|
||||
const repoRoot = resolveRepoRoot(import.meta.url);
|
||||
const readSourceRoots = resolveSourceRoots(repoRoot, [
|
||||
"packages/memory-host-sdk/src/host",
|
||||
"extensions/discord/src/monitor",
|
||||
"extensions/memory-core/src",
|
||||
"extensions/telegram/src",
|
||||
"extensions/voice-call/src",
|
||||
"src/acp",
|
||||
"src/agents",
|
||||
"src/auto-reply",
|
||||
"src/commands",
|
||||
"src/config/sessions",
|
||||
"src/cron",
|
||||
"src/gateway",
|
||||
"src/infra",
|
||||
"src/plugins",
|
||||
"src/tui",
|
||||
]);
|
||||
const writeSourceRoots = resolveSourceRoots(repoRoot, [
|
||||
"src/acp",
|
||||
"src/agents",
|
||||
"src/auto-reply",
|
||||
"src/commands",
|
||||
"src/config/sessions",
|
||||
"src/gateway",
|
||||
"src/plugins",
|
||||
"src/tui",
|
||||
]);
|
||||
const transcriptWriterSourceRoots = resolveSourceRoots(repoRoot, [
|
||||
"src/agents/command",
|
||||
"src/agents/embedded-agent-runner",
|
||||
"src/config/sessions",
|
||||
"src/gateway/server-methods",
|
||||
"src/sessions",
|
||||
]);
|
||||
const readViolations = await collectFileViolations({
|
||||
repoRoot,
|
||||
sourceRoots: readSourceRoots,
|
||||
skipFile: (filePath) => {
|
||||
const relativePath = normalizeRelativePath(path.relative(repoRoot, filePath));
|
||||
return (
|
||||
!migratedSessionAccessorFiles.has(relativePath) &&
|
||||
!migratedBundledPluginSessionAccessorFiles.has(relativePath)
|
||||
);
|
||||
},
|
||||
findViolations: findSessionAccessorBoundaryViolations,
|
||||
});
|
||||
const writeViolations = await collectFileViolations({
|
||||
repoRoot,
|
||||
sourceRoots: writeSourceRoots,
|
||||
skipFile: (filePath) =>
|
||||
!migratedSessionAccessorWriteFiles.has(
|
||||
normalizeRelativePath(path.relative(repoRoot, filePath)),
|
||||
),
|
||||
findViolations: findSessionAccessorWriteBoundaryViolations,
|
||||
});
|
||||
const transcriptWriterViolations = await collectFileViolations({
|
||||
repoRoot,
|
||||
sourceRoots: transcriptWriterSourceRoots,
|
||||
skipFile: (filePath) =>
|
||||
!migratedTranscriptWriterFiles.has(normalizeRelativePath(path.relative(repoRoot, filePath))),
|
||||
findViolations: findTranscriptWriterBoundaryViolations,
|
||||
});
|
||||
const sessionCreateLifecycleViolations = await collectFileViolations({
|
||||
repoRoot,
|
||||
sourceRoots: resolveSourceRoots(repoRoot, ["src/gateway/server-methods"]),
|
||||
skipFile: (filePath) =>
|
||||
normalizeRelativePath(path.relative(repoRoot, filePath)) !==
|
||||
"src/gateway/server-methods/sessions.ts",
|
||||
findViolations: findGatewaySessionCreateLifecycleViolations,
|
||||
});
|
||||
const manualCompactTrimViolations = await collectFileViolations({
|
||||
repoRoot,
|
||||
sourceRoots: resolveSourceRoots(repoRoot, ["src/gateway/server-methods"]),
|
||||
skipFile: (filePath) =>
|
||||
!migratedSessionCompactManualTrimFiles.has(
|
||||
normalizeRelativePath(path.relative(repoRoot, filePath)),
|
||||
),
|
||||
findViolations: findSessionCompactManualTrimBoundaryViolations,
|
||||
});
|
||||
const lifecycleCleanupViolations = await collectFileViolations({
|
||||
repoRoot,
|
||||
sourceRoots: readSourceRoots,
|
||||
skipFile: (filePath) =>
|
||||
!migratedSessionLifecycleCleanupFiles.has(
|
||||
normalizeRelativePath(path.relative(repoRoot, filePath)),
|
||||
),
|
||||
findViolations: findSessionLifecycleCleanupBoundaryViolations,
|
||||
});
|
||||
const memoryHostSessionCorpusViolations = await collectFileViolations({
|
||||
repoRoot,
|
||||
sourceRoots: resolveSourceRoots(repoRoot, ["packages/memory-host-sdk/src/host"]),
|
||||
skipFile: (filePath) =>
|
||||
!migratedMemoryHostSessionCorpusFiles.has(
|
||||
normalizeRelativePath(path.relative(repoRoot, filePath)),
|
||||
),
|
||||
findViolations: findMemoryHostSessionCorpusBoundaryViolations,
|
||||
});
|
||||
const embeddedAgentSessionTargetViolations = await collectFileViolations({
|
||||
repoRoot,
|
||||
sourceRoots: resolveSourceRoots(repoRoot, ["extensions/voice-call/src"]),
|
||||
skipFile: (filePath) =>
|
||||
!migratedEmbeddedAgentSessionTargetFiles.has(
|
||||
normalizeRelativePath(path.relative(repoRoot, filePath)),
|
||||
),
|
||||
findViolations: findEmbeddedAgentSessionTargetViolations,
|
||||
});
|
||||
const sessionStoreRuntimePath = path.join(repoRoot, "src/plugin-sdk/session-store-runtime.ts");
|
||||
const sessionStoreRuntimeCompatViolations =
|
||||
findSessionStoreRuntimeFileBackedCompatExportViolations(
|
||||
await fs.readFile(sessionStoreRuntimePath, "utf8"),
|
||||
sessionStoreRuntimePath,
|
||||
).map((violation) =>
|
||||
Object.assign({ path: "src/plugin-sdk/session-store-runtime.ts" }, violation),
|
||||
);
|
||||
const violations = [
|
||||
...readViolations,
|
||||
...writeViolations,
|
||||
...transcriptWriterViolations,
|
||||
...sessionCreateLifecycleViolations,
|
||||
...manualCompactTrimViolations,
|
||||
...lifecycleCleanupViolations,
|
||||
...memoryHostSessionCorpusViolations,
|
||||
...embeddedAgentSessionTargetViolations,
|
||||
...sessionStoreRuntimeCompatViolations,
|
||||
];
|
||||
|
||||
if (violations.length === 0) {
|
||||
console.log("session accessor boundary guard passed.");
|
||||
return;
|
||||
}
|
||||
|
||||
console.error("Found legacy session store usage in session-accessor migrated files:");
|
||||
for (const violation of violations) {
|
||||
console.error(`- ${violation.path}:${violation.line}: ${violation.reason}`);
|
||||
}
|
||||
console.error(
|
||||
"Use src/config/sessions/session-accessor.ts helpers for migrated read/write and transcript-writer paths. Expand file-backed SDK compatibility only as an explicit pre-SQLite migration decision.",
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
runAsScript(import.meta.url, main);
|
||||
268
scripts/check-session-transcript-reader-boundary.mjs
Normal file
268
scripts/check-session-transcript-reader-boundary.mjs
Normal file
@@ -0,0 +1,268 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import path from "node:path";
|
||||
import ts from "typescript";
|
||||
import {
|
||||
collectFileViolations,
|
||||
resolveRepoRoot,
|
||||
resolveSourceRoots,
|
||||
runAsScript,
|
||||
toLine,
|
||||
unwrapExpression,
|
||||
} from "./lib/ts-guard-utils.mjs";
|
||||
|
||||
const legacyTranscriptReaderModules = new Set([
|
||||
"../gateway/session-utils.js",
|
||||
"../gateway/session-utils.fs.js",
|
||||
"../../gateway/session-utils.js",
|
||||
"../../gateway/session-utils.fs.js",
|
||||
"./session-utils.js",
|
||||
"./session-utils.fs.js",
|
||||
"../session-utils.js",
|
||||
"../session-utils.fs.js",
|
||||
]);
|
||||
|
||||
const transcriptReaderNames = new Set([
|
||||
"attachOpenClawTranscriptMeta",
|
||||
"capArrayByJsonBytes",
|
||||
"readFirstUserMessageFromTranscript",
|
||||
"readLatestRecentSessionUsageFromTranscriptAsync",
|
||||
"readLatestSessionUsageFromTranscript",
|
||||
"readLatestSessionUsageFromTranscriptAsync",
|
||||
"readRecentSessionMessages",
|
||||
"readRecentSessionMessagesAsync",
|
||||
"readRecentSessionMessagesWithStats",
|
||||
"readRecentSessionMessagesWithStatsAsync",
|
||||
"readRecentSessionTranscriptLines",
|
||||
"readRecentSessionUsageFromTranscript",
|
||||
"readRecentSessionUsageFromTranscriptAsync",
|
||||
"readSessionMessageByIdAsync",
|
||||
"readSessionMessageCount",
|
||||
"readSessionMessageCountAsync",
|
||||
"readSessionMessages",
|
||||
"readSessionMessagesAsync",
|
||||
"readSessionMessagesWithSourceAsync",
|
||||
"readSessionPreviewItemsFromTranscript",
|
||||
"readSessionTitleFieldsFromTranscript",
|
||||
"readSessionTitleFieldsFromTranscriptAsync",
|
||||
"visitSessionMessages",
|
||||
"visitSessionMessagesAsync",
|
||||
]);
|
||||
|
||||
const storageSpecificTranscriptReaderAliasNames = new Set(["readSessionMessagesFromFileAsync"]);
|
||||
|
||||
export const migratedSessionTranscriptReaderFiles = new Set([
|
||||
"src/agents/main-session-restart-recovery.ts",
|
||||
"src/agents/subagent-announce-output.test.ts",
|
||||
"src/agents/subagent-announce-output.ts",
|
||||
"src/agents/subagent-announce.runtime.ts",
|
||||
"src/agents/subagent-orphan-recovery.test.ts",
|
||||
"src/agents/subagent-orphan-recovery.ts",
|
||||
"src/agents/tools/embedded-gateway-stub.runtime.ts",
|
||||
"src/agents/tools/embedded-gateway-stub.test.ts",
|
||||
"src/agents/tools/embedded-gateway-stub.ts",
|
||||
"src/agents/tools/sessions-history-tool.ts",
|
||||
"src/agents/tools/sessions-list-tool.ts",
|
||||
"src/gateway/cli-session-history.claude.ts",
|
||||
"src/gateway/gateway-models.profiles.live.test.ts",
|
||||
"src/gateway/managed-image-attachments.test.ts",
|
||||
"src/gateway/managed-image-attachments.ts",
|
||||
"src/gateway/server-methods/artifacts.test.ts",
|
||||
"src/gateway/server-methods/artifacts.ts",
|
||||
"src/gateway/server-methods/chat.ts",
|
||||
"src/gateway/server-methods/sessions-files.test.ts",
|
||||
"src/gateway/server-methods/sessions-files.ts",
|
||||
"src/gateway/server-methods/sessions.ts",
|
||||
"src/gateway/server-session-events.ts",
|
||||
"src/gateway/session-history-state.test.ts",
|
||||
"src/gateway/session-history-state.ts",
|
||||
"src/gateway/session-reset-service.ts",
|
||||
"src/gateway/session-utils.ts",
|
||||
"src/gateway/sessions-history-http.revocation.test.ts",
|
||||
"src/gateway/sessions-history-http.ts",
|
||||
"src/status/status-message.ts",
|
||||
"src/tui/embedded-backend.test.ts",
|
||||
"src/tui/embedded-backend.ts",
|
||||
]);
|
||||
|
||||
function normalizeRelativePath(filePath) {
|
||||
return filePath.replaceAll(path.sep, "/");
|
||||
}
|
||||
|
||||
function importedModuleName(node) {
|
||||
return node.moduleSpecifier && ts.isStringLiteral(node.moduleSpecifier)
|
||||
? node.moduleSpecifier.text
|
||||
: null;
|
||||
}
|
||||
|
||||
function bindingName(node) {
|
||||
if (node.propertyName && ts.isIdentifier(node.propertyName)) {
|
||||
return node.propertyName.text;
|
||||
}
|
||||
if (ts.isIdentifier(node.name)) {
|
||||
return node.name.text;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function destructuresLegacyNamespace(node, legacyNamespaces) {
|
||||
const pattern = node.parent;
|
||||
const declaration = pattern?.parent;
|
||||
if (
|
||||
!pattern ||
|
||||
!ts.isObjectBindingPattern(pattern) ||
|
||||
!declaration ||
|
||||
!ts.isVariableDeclaration(declaration) ||
|
||||
!declaration.initializer
|
||||
) {
|
||||
return false;
|
||||
}
|
||||
|
||||
const initializer = unwrapExpression(declaration.initializer);
|
||||
return ts.isIdentifier(initializer) && legacyNamespaces.has(initializer.text);
|
||||
}
|
||||
|
||||
export function findSessionTranscriptReaderBoundaryViolations(content, fileName = "source.ts") {
|
||||
const sourceFile = ts.createSourceFile(fileName, content, ts.ScriptTarget.Latest, true);
|
||||
const violations = [];
|
||||
const legacyNamespaces = new Set();
|
||||
|
||||
const visit = (node) => {
|
||||
if (ts.isIdentifier(node) && storageSpecificTranscriptReaderAliasNames.has(node.text)) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node),
|
||||
reason: `uses storage-specific transcript reader alias "${node.text}"`,
|
||||
});
|
||||
}
|
||||
|
||||
if (ts.isImportDeclaration(node)) {
|
||||
const moduleName = importedModuleName(node);
|
||||
const namedBindings = node.importClause?.namedBindings;
|
||||
if (moduleName && legacyTranscriptReaderModules.has(moduleName) && namedBindings) {
|
||||
if (ts.isNamedImports(namedBindings)) {
|
||||
for (const specifier of namedBindings.elements) {
|
||||
const importedName = specifier.propertyName?.text ?? specifier.name.text;
|
||||
if (transcriptReaderNames.has(importedName)) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, specifier),
|
||||
reason: `imports transcript reader "${importedName}" from legacy module "${moduleName}"`,
|
||||
});
|
||||
}
|
||||
}
|
||||
} else if (ts.isNamespaceImport(namedBindings)) {
|
||||
legacyNamespaces.add(namedBindings.name.text);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (ts.isExportDeclaration(node)) {
|
||||
const moduleName = importedModuleName(node);
|
||||
if (moduleName && legacyTranscriptReaderModules.has(moduleName)) {
|
||||
const exportClause = node.exportClause;
|
||||
if (!exportClause) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node),
|
||||
reason: `re-exports transcript readers from legacy module "${moduleName}"`,
|
||||
});
|
||||
} else if (ts.isNamedExports(exportClause)) {
|
||||
for (const specifier of exportClause.elements) {
|
||||
const exportedName = specifier.propertyName?.text ?? specifier.name.text;
|
||||
if (transcriptReaderNames.has(exportedName)) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, specifier),
|
||||
reason: `re-exports transcript reader "${exportedName}" from legacy module "${moduleName}"`,
|
||||
});
|
||||
}
|
||||
}
|
||||
} else if (ts.isNamespaceExport(exportClause)) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, exportClause),
|
||||
reason: `re-exports transcript reader namespace from legacy module "${moduleName}"`,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (ts.isBindingElement(node)) {
|
||||
const name = bindingName(node);
|
||||
if (
|
||||
name &&
|
||||
transcriptReaderNames.has(name) &&
|
||||
destructuresLegacyNamespace(node, legacyNamespaces)
|
||||
) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node),
|
||||
reason: `aliases legacy transcript reader "${name}"`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (ts.isPropertyAccessExpression(node)) {
|
||||
const receiver = unwrapExpression(node.expression);
|
||||
if (
|
||||
ts.isIdentifier(receiver) &&
|
||||
legacyNamespaces.has(receiver.text) &&
|
||||
transcriptReaderNames.has(node.name.text)
|
||||
) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node.name),
|
||||
reason: `references legacy transcript reader "${node.name.text}"`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (
|
||||
ts.isElementAccessExpression(node) &&
|
||||
ts.isIdentifier(unwrapExpression(node.expression)) &&
|
||||
legacyNamespaces.has(unwrapExpression(node.expression).text) &&
|
||||
ts.isStringLiteral(node.argumentExpression) &&
|
||||
transcriptReaderNames.has(node.argumentExpression.text)
|
||||
) {
|
||||
violations.push({
|
||||
line: toLine(sourceFile, node.argumentExpression),
|
||||
reason: `references legacy transcript reader "${node.argumentExpression.text}"`,
|
||||
});
|
||||
}
|
||||
|
||||
ts.forEachChild(node, visit);
|
||||
};
|
||||
|
||||
visit(sourceFile);
|
||||
return violations;
|
||||
}
|
||||
|
||||
export async function main() {
|
||||
const repoRoot = resolveRepoRoot(import.meta.url);
|
||||
const sourceRoots = resolveSourceRoots(repoRoot, [
|
||||
"src/agents",
|
||||
"src/gateway",
|
||||
"src/status",
|
||||
"src/tui",
|
||||
]);
|
||||
const violations = await collectFileViolations({
|
||||
repoRoot,
|
||||
sourceRoots,
|
||||
includeTests: true,
|
||||
skipFile: (filePath) =>
|
||||
!migratedSessionTranscriptReaderFiles.has(
|
||||
normalizeRelativePath(path.relative(repoRoot, filePath)),
|
||||
),
|
||||
findViolations: findSessionTranscriptReaderBoundaryViolations,
|
||||
});
|
||||
|
||||
if (violations.length === 0) {
|
||||
console.log("session transcript reader boundary guard passed.");
|
||||
return;
|
||||
}
|
||||
|
||||
console.error("Found legacy transcript reader usage in migrated files:");
|
||||
for (const violation of violations) {
|
||||
console.error(`- ${violation.path}:${violation.line}: ${violation.reason}`);
|
||||
}
|
||||
console.error(
|
||||
"Use src/gateway/session-transcript-readers.ts for migrated transcript reader paths. Expand this ratchet only after a slice migrates more files.",
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
runAsScript(import.meta.url, main);
|
||||
34
scripts/check-src-extension-import-boundary.mjs
Normal file
34
scripts/check-src-extension-import-boundary.mjs
Normal file
@@ -0,0 +1,34 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Runs the src/** extension import boundary checker.
|
||||
import { createExtensionImportBoundaryChecker } from "./lib/extension-import-boundary-checker.mjs";
|
||||
import { runAsScript } from "./lib/ts-guard-utils.mjs";
|
||||
|
||||
const ALLOWED_EXTENSION_PUBLIC_SURFACE_RE = /^extensions\/[^/]+\/(?:api|runtime-api)\.js$/;
|
||||
|
||||
const checker = createExtensionImportBoundaryChecker({
|
||||
roots: ["src"],
|
||||
boundaryLabel: "src",
|
||||
rule: "Rule: production src/** must not import bundled plugin files",
|
||||
cleanMessage: "No src import boundary violations found.",
|
||||
inventoryTitle: "Src extension import boundary inventory:",
|
||||
skipSourcesWithoutBundledPluginPrefix: true,
|
||||
allowResolvedPath(resolvedPath) {
|
||||
return ALLOWED_EXTENSION_PUBLIC_SURFACE_RE.test(resolvedPath);
|
||||
},
|
||||
shouldSkipFile(relativeFile) {
|
||||
return (
|
||||
relativeFile.endsWith(".test.ts") ||
|
||||
relativeFile.endsWith(".test.tsx") ||
|
||||
relativeFile.endsWith(".e2e.test.ts") ||
|
||||
relativeFile.endsWith(".e2e.test.tsx")
|
||||
);
|
||||
},
|
||||
});
|
||||
|
||||
/**
|
||||
* Entrypoint for the src extension import boundary checker.
|
||||
*/
|
||||
export const main = checker.main;
|
||||
|
||||
runAsScript(import.meta.url, main);
|
||||
69
scripts/check-telegram-grammy-types-imports.mjs
Normal file
69
scripts/check-telegram-grammy-types-imports.mjs
Normal file
@@ -0,0 +1,69 @@
|
||||
#!/usr/bin/env node
|
||||
// Prevents Telegram runtime imports from grammy type-only modules.
|
||||
import { readdirSync, readFileSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
|
||||
const repoRoot = path.resolve(import.meta.dirname, "..");
|
||||
const telegramRoot = path.join(repoRoot, "extensions/telegram");
|
||||
const importSpecifierPatterns = [
|
||||
/\bimport\s+(?:type\s+)?[\s\S]*?\bfrom\s*["']([^"']+)["']/gu,
|
||||
/\bexport\s+(?:type\s+)?[\s\S]*?\bfrom\s*["']([^"']+)["']/gu,
|
||||
/\bimport\s*["']([^"']+)["']/gu,
|
||||
/\bimport\s*\(\s*["']([^"']+)["']\s*\)/gu,
|
||||
/\brequire\s*\(\s*["']([^"']+)["']\s*\)/gu,
|
||||
];
|
||||
|
||||
function isForbiddenTelegramTypeSpecifier(specifier) {
|
||||
return specifier === "@grammyjs/types" || specifier.startsWith("@grammyjs/types/");
|
||||
}
|
||||
|
||||
function lineNumberForOffset(source, offset) {
|
||||
return source.slice(0, offset).split(/\r?\n/u).length;
|
||||
}
|
||||
|
||||
function collectTypeScriptFiles(root) {
|
||||
const files = [];
|
||||
for (const entry of readdirSync(root, { withFileTypes: true })) {
|
||||
if (entry.name === "dist" || entry.name === "node_modules") {
|
||||
continue;
|
||||
}
|
||||
const entryPath = path.join(root, entry.name);
|
||||
if (entry.isDirectory()) {
|
||||
files.push(...collectTypeScriptFiles(entryPath));
|
||||
continue;
|
||||
}
|
||||
if (entry.isFile() && entry.name.endsWith(".ts")) {
|
||||
files.push(entryPath);
|
||||
}
|
||||
}
|
||||
return files;
|
||||
}
|
||||
|
||||
const violations = [];
|
||||
for (const filePath of collectTypeScriptFiles(telegramRoot)) {
|
||||
const source = readFileSync(filePath, "utf8");
|
||||
for (const pattern of importSpecifierPatterns) {
|
||||
for (const match of source.matchAll(pattern)) {
|
||||
const specifier = match[1];
|
||||
if (specifier && isForbiddenTelegramTypeSpecifier(specifier)) {
|
||||
violations.push(
|
||||
`${path.relative(repoRoot, filePath)}:${lineNumberForOffset(source, match.index ?? 0)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (violations.length > 0) {
|
||||
console.error(
|
||||
[
|
||||
"Telegram source must import Telegram Bot API types from grammy/types, not @grammyjs/types.",
|
||||
"This keeps grammY Context and message/update helper types on the same dependency copy.",
|
||||
"",
|
||||
...violations,
|
||||
].join("\n"),
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log("No Telegram direct @grammyjs/types imports found.");
|
||||
310
scripts/check-temp-path-guardrails.ts
Normal file
310
scripts/check-temp-path-guardrails.ts
Normal file
@@ -0,0 +1,310 @@
|
||||
// Check Temp Path Guardrails script supports OpenClaw repository automation.
|
||||
import { execFileSync } from "node:child_process";
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
|
||||
type QuoteChar = "'" | '"' | "`";
|
||||
|
||||
type QuoteScanState = {
|
||||
quote: QuoteChar | null;
|
||||
escaped: boolean;
|
||||
};
|
||||
|
||||
type RuntimeSourceGuardrailFile = {
|
||||
relativePath: string;
|
||||
source: string;
|
||||
};
|
||||
|
||||
const WEAK_RANDOM_SAME_LINE_PATTERN =
|
||||
/(?:Date\.now[^\r\n]*Math\.random|Math\.random[^\r\n]*Date\.now)/u;
|
||||
const PATH_JOIN_CALL_PATTERN = /path\s*\.\s*join\s*\(/u;
|
||||
const OS_TMPDIR_CALL_PATTERN = /os\s*\.\s*tmpdir\s*\(/u;
|
||||
const FILE_READ_CONCURRENCY = 24;
|
||||
const DEFAULT_GUARDRAIL_SKIP_PATTERNS = [
|
||||
/\.test\.tsx?$/,
|
||||
/\.test-helpers\.tsx?$/,
|
||||
/\.test-utils\.tsx?$/,
|
||||
/\.test-harness\.tsx?$/,
|
||||
/\.test-support\.tsx?$/,
|
||||
/\.suite\.tsx?$/,
|
||||
/\.e2e\.tsx?$/,
|
||||
/\.d\.ts$/,
|
||||
/[\\/](?:__tests__|tests|test-helpers|test-utils|test-support)[\\/]/,
|
||||
/[\\/][^\\/]*test-helpers(?:\.[^\\/]+)?\.ts$/,
|
||||
/[\\/][^\\/]*test-utils(?:\.[^\\/]+)?\.ts$/,
|
||||
/[\\/][^\\/]*test-harness(?:\.[^\\/]+)?\.ts$/,
|
||||
/[\\/][^\\/]*test-support(?:\.[^\\/]+)?\.ts$/,
|
||||
];
|
||||
|
||||
function shouldSkipGuardrailRuntimeSource(relativePath: string): boolean {
|
||||
return DEFAULT_GUARDRAIL_SKIP_PATTERNS.some((pattern) => pattern.test(relativePath));
|
||||
}
|
||||
|
||||
function stripCommentsForScan(input: string): string {
|
||||
return input.replace(/\/\*[\s\S]*?\*\//g, "").replace(/(^|[^:])\/\/.*$/gm, "$1");
|
||||
}
|
||||
|
||||
function beginQuotedSection(state: QuoteScanState, ch: string): boolean {
|
||||
if (ch !== "'" && ch !== '"' && ch !== "`") {
|
||||
return false;
|
||||
}
|
||||
state.quote = ch;
|
||||
return true;
|
||||
}
|
||||
|
||||
function consumeQuotedChar(state: QuoteScanState, ch: string): boolean {
|
||||
if (!state.quote) {
|
||||
return false;
|
||||
}
|
||||
if (state.escaped) {
|
||||
state.escaped = false;
|
||||
return true;
|
||||
}
|
||||
if (ch === "\\") {
|
||||
state.escaped = true;
|
||||
return true;
|
||||
}
|
||||
if (ch === state.quote) {
|
||||
state.quote = null;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
function findMatchingParen(source: string, openIndex: number): number {
|
||||
let depth = 1;
|
||||
const quoteState: QuoteScanState = { quote: null, escaped: false };
|
||||
for (let i = openIndex + 1; i < source.length; i += 1) {
|
||||
const ch = source[i];
|
||||
if (consumeQuotedChar(quoteState, ch)) {
|
||||
continue;
|
||||
}
|
||||
if (beginQuotedSection(quoteState, ch)) {
|
||||
continue;
|
||||
}
|
||||
if (ch === "(") {
|
||||
depth += 1;
|
||||
continue;
|
||||
}
|
||||
if (ch === ")") {
|
||||
depth -= 1;
|
||||
if (depth === 0) {
|
||||
return i;
|
||||
}
|
||||
}
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
|
||||
function splitTopLevelArguments(source: string): string[] {
|
||||
const out: string[] = [];
|
||||
let current = "";
|
||||
let parenDepth = 0;
|
||||
let bracketDepth = 0;
|
||||
let braceDepth = 0;
|
||||
const quoteState: QuoteScanState = { quote: null, escaped: false };
|
||||
for (const ch of source) {
|
||||
if (quoteState.quote) {
|
||||
current += ch;
|
||||
consumeQuotedChar(quoteState, ch);
|
||||
continue;
|
||||
}
|
||||
if (beginQuotedSection(quoteState, ch)) {
|
||||
current += ch;
|
||||
continue;
|
||||
}
|
||||
if (ch === "(") {
|
||||
parenDepth += 1;
|
||||
current += ch;
|
||||
continue;
|
||||
}
|
||||
if (ch === ")") {
|
||||
if (parenDepth > 0) {
|
||||
parenDepth -= 1;
|
||||
}
|
||||
current += ch;
|
||||
continue;
|
||||
}
|
||||
if (ch === "[") {
|
||||
bracketDepth += 1;
|
||||
current += ch;
|
||||
continue;
|
||||
}
|
||||
if (ch === "]") {
|
||||
if (bracketDepth > 0) {
|
||||
bracketDepth -= 1;
|
||||
}
|
||||
current += ch;
|
||||
continue;
|
||||
}
|
||||
if (ch === "{") {
|
||||
braceDepth += 1;
|
||||
current += ch;
|
||||
continue;
|
||||
}
|
||||
if (ch === "}") {
|
||||
if (braceDepth > 0) {
|
||||
braceDepth -= 1;
|
||||
}
|
||||
current += ch;
|
||||
continue;
|
||||
}
|
||||
if (ch === "," && parenDepth === 0 && bracketDepth === 0 && braceDepth === 0) {
|
||||
out.push(current.trim());
|
||||
current = "";
|
||||
continue;
|
||||
}
|
||||
current += ch;
|
||||
}
|
||||
if (current.trim()) {
|
||||
out.push(current.trim());
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function isOsTmpdirExpression(argument: string): boolean {
|
||||
return /^os\s*\.\s*tmpdir\s*\(\s*\)$/u.test(argument.trim());
|
||||
}
|
||||
|
||||
function mightContainDynamicTmpdirJoin(source: string): boolean {
|
||||
if (!source.includes("path") || !source.includes("join") || !source.includes("tmpdir")) {
|
||||
return false;
|
||||
}
|
||||
return (
|
||||
(source.includes("path.join") || PATH_JOIN_CALL_PATTERN.test(source)) &&
|
||||
(source.includes("os.tmpdir") || OS_TMPDIR_CALL_PATTERN.test(source)) &&
|
||||
source.includes("`") &&
|
||||
source.includes("${")
|
||||
);
|
||||
}
|
||||
|
||||
function hasDynamicTmpdirJoin(source: string): boolean {
|
||||
if (!mightContainDynamicTmpdirJoin(source)) {
|
||||
return false;
|
||||
}
|
||||
|
||||
const scanSource = stripCommentsForScan(source);
|
||||
const joinPattern = /path\s*\.\s*join\s*\(/gu;
|
||||
let match: RegExpExecArray | null = joinPattern.exec(scanSource);
|
||||
while (match) {
|
||||
const openParenIndex = scanSource.indexOf("(", match.index);
|
||||
if (openParenIndex !== -1) {
|
||||
const closeParenIndex = findMatchingParen(scanSource, openParenIndex);
|
||||
if (closeParenIndex !== -1) {
|
||||
const argsSource = scanSource.slice(openParenIndex + 1, closeParenIndex);
|
||||
const args = splitTopLevelArguments(argsSource);
|
||||
if (args.length >= 2 && isOsTmpdirExpression(args[0])) {
|
||||
for (const arg of args.slice(1)) {
|
||||
const trimmed = arg.trim();
|
||||
if (trimmed.startsWith("`") && trimmed.includes("${")) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
match = joinPattern.exec(scanSource);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function listTrackedRuntimeSourceFiles(repoRoot: string): string[] {
|
||||
const stdout = execFileSync("git", ["-C", repoRoot, "ls-files", "--", "src", "extensions"], {
|
||||
encoding: "utf8",
|
||||
stdio: ["ignore", "pipe", "inherit"],
|
||||
});
|
||||
return stdout
|
||||
.split(/\r?\n/u)
|
||||
.filter(Boolean)
|
||||
.filter((relativePath) => relativePath.endsWith(".ts") || relativePath.endsWith(".tsx"))
|
||||
.filter((relativePath) => !shouldSkipGuardrailRuntimeSource(relativePath))
|
||||
.map((relativePath) => path.join(repoRoot, relativePath));
|
||||
}
|
||||
|
||||
async function readRuntimeSourceFiles(
|
||||
repoRoot: string,
|
||||
absolutePaths: string[],
|
||||
): Promise<RuntimeSourceGuardrailFile[]> {
|
||||
const output: Array<RuntimeSourceGuardrailFile | undefined> = Array.from({
|
||||
length: absolutePaths.length,
|
||||
});
|
||||
let nextIndex = 0;
|
||||
|
||||
const worker = async () => {
|
||||
for (;;) {
|
||||
const index = nextIndex;
|
||||
nextIndex += 1;
|
||||
if (index >= absolutePaths.length) {
|
||||
return;
|
||||
}
|
||||
const absolutePath = absolutePaths[index];
|
||||
if (!absolutePath) {
|
||||
continue;
|
||||
}
|
||||
let source: string;
|
||||
try {
|
||||
source = await fs.readFile(absolutePath, "utf8");
|
||||
} catch {
|
||||
// File tracked by git but deleted on disk (e.g. pending deletion).
|
||||
continue;
|
||||
}
|
||||
output[index] = {
|
||||
relativePath: path.relative(repoRoot, absolutePath),
|
||||
source,
|
||||
};
|
||||
}
|
||||
};
|
||||
|
||||
const workers = Array.from(
|
||||
{ length: Math.min(FILE_READ_CONCURRENCY, Math.max(1, absolutePaths.length)) },
|
||||
() => worker(),
|
||||
);
|
||||
await Promise.all(workers);
|
||||
return output.filter((entry): entry is RuntimeSourceGuardrailFile => entry !== undefined);
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const repoRoot = process.cwd();
|
||||
const files = await readRuntimeSourceFiles(repoRoot, listTrackedRuntimeSourceFiles(repoRoot));
|
||||
const offenders: string[] = [];
|
||||
const weakRandomMatches: string[] = [];
|
||||
|
||||
for (const file of files) {
|
||||
const source = file.source;
|
||||
const mightContainTmpdirJoin =
|
||||
source.includes("tmpdir") &&
|
||||
source.includes("path") &&
|
||||
source.includes("join") &&
|
||||
source.includes("`");
|
||||
const mightContainWeakRandom = source.includes("Date.now") && source.includes("Math.random");
|
||||
|
||||
if (!mightContainTmpdirJoin && !mightContainWeakRandom) {
|
||||
continue;
|
||||
}
|
||||
if (mightContainTmpdirJoin && hasDynamicTmpdirJoin(source)) {
|
||||
offenders.push(file.relativePath);
|
||||
}
|
||||
if (mightContainWeakRandom && WEAK_RANDOM_SAME_LINE_PATTERN.test(source)) {
|
||||
weakRandomMatches.push(file.relativePath);
|
||||
}
|
||||
}
|
||||
|
||||
if (offenders.length === 0 && weakRandomMatches.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (offenders.length > 0) {
|
||||
console.error("Dynamic os.tmpdir()/path.join() template paths found:");
|
||||
for (const offender of offenders) {
|
||||
console.error(`- ${offender}`);
|
||||
}
|
||||
}
|
||||
if (weakRandomMatches.length > 0) {
|
||||
console.error("Weak Date.now()+Math.random() same-line IDs found:");
|
||||
for (const offender of weakRandomMatches) {
|
||||
console.error(`- ${offender}`);
|
||||
}
|
||||
}
|
||||
process.exitCode = 1;
|
||||
}
|
||||
|
||||
await main();
|
||||
24
scripts/check-test-helper-extension-import-boundary.mjs
Normal file
24
scripts/check-test-helper-extension-import-boundary.mjs
Normal file
@@ -0,0 +1,24 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
// Runs the test-helper extension import boundary checker.
|
||||
import { createExtensionImportBoundaryChecker } from "./lib/extension-import-boundary-checker.mjs";
|
||||
import { runAsScript } from "./lib/ts-guard-utils.mjs";
|
||||
|
||||
const checker = createExtensionImportBoundaryChecker({
|
||||
roots: ["test/helpers"],
|
||||
boundaryLabel: "test helper",
|
||||
rule: "Rule: test/helpers/** must not import bundled plugin files directly",
|
||||
cleanMessage: "No test-helper import boundary violations found.",
|
||||
inventoryTitle: "Test-helper extension import boundary inventory:",
|
||||
});
|
||||
|
||||
/**
|
||||
* Collects test-helper extension import boundary inventory.
|
||||
*/
|
||||
export const collectTestHelperExtensionImportBoundaryInventory = checker.collectInventory;
|
||||
/**
|
||||
* Entrypoint for the test-helper extension import boundary checker.
|
||||
*/
|
||||
export const main = checker.main;
|
||||
|
||||
runAsScript(import.meta.url, main);
|
||||
4
scripts/check-timed.mjs
Normal file
4
scripts/check-timed.mjs
Normal file
@@ -0,0 +1,4 @@
|
||||
// Wraps the aggregate check command with timing behavior.
|
||||
import { main } from "./check.mjs";
|
||||
|
||||
await main([...process.argv.slice(2), "--timed"]);
|
||||
96
scripts/check-ts-max-loc.ts
Normal file
96
scripts/check-ts-max-loc.ts
Normal file
@@ -0,0 +1,96 @@
|
||||
// Check Ts Max Loc script supports OpenClaw repository automation.
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { existsSync } from "node:fs";
|
||||
import { readFile } from "node:fs/promises";
|
||||
|
||||
function writeStdoutLine(message: string): void {
|
||||
process.stdout.write(`${message}\n`);
|
||||
}
|
||||
|
||||
type ParsedArgs = {
|
||||
maxLines: number;
|
||||
};
|
||||
|
||||
function parseArgs(argv: string[]): ParsedArgs {
|
||||
let maxLines = 500;
|
||||
|
||||
for (let index = 0; index < argv.length; index++) {
|
||||
const arg = argv[index];
|
||||
if (arg === "--max") {
|
||||
const next = argv[index + 1];
|
||||
if (!next || !/^\d+$/u.test(next)) {
|
||||
throw new Error("--max requires a positive integer");
|
||||
}
|
||||
maxLines = Number(next);
|
||||
if (!Number.isSafeInteger(maxLines) || maxLines <= 0) {
|
||||
throw new Error("--max requires a positive integer");
|
||||
}
|
||||
index++;
|
||||
continue;
|
||||
}
|
||||
throw new Error(`Unknown argument: ${arg}`);
|
||||
}
|
||||
|
||||
return { maxLines };
|
||||
}
|
||||
|
||||
function gitLsFilesAll(): string[] {
|
||||
// Include untracked files too so local refactors don’t “pass” by accident.
|
||||
const stdout = execFileSync("git", ["ls-files", "--cached", "--others", "--exclude-standard"], {
|
||||
encoding: "utf8",
|
||||
});
|
||||
return stdout
|
||||
.split("\n")
|
||||
.map((line) => line.trim())
|
||||
.filter(Boolean);
|
||||
}
|
||||
|
||||
async function countLines(filePath: string): Promise<number> {
|
||||
const content = await readFile(filePath, "utf8");
|
||||
// Count physical lines. Keeps the rule simple + predictable.
|
||||
return content.split("\n").length;
|
||||
}
|
||||
|
||||
async function main(argv = process.argv.slice(2)): Promise<number> {
|
||||
// Makes `... | head` safe.
|
||||
process.stdout.on("error", (error: NodeJS.ErrnoException) => {
|
||||
if (error.code === "EPIPE") {
|
||||
process.exit(0);
|
||||
}
|
||||
throw error;
|
||||
});
|
||||
|
||||
const { maxLines } = parseArgs(argv);
|
||||
const files = gitLsFilesAll()
|
||||
.filter((filePath) => existsSync(filePath))
|
||||
.filter((filePath) => filePath.endsWith(".ts") || filePath.endsWith(".tsx"));
|
||||
|
||||
const results = await Promise.all(
|
||||
files.map(async (filePath) => ({ filePath, lines: await countLines(filePath) })),
|
||||
);
|
||||
|
||||
const offenders = results
|
||||
.filter((result) => result.lines > maxLines)
|
||||
.toSorted((a, b) => b.lines - a.lines);
|
||||
|
||||
if (!offenders.length) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
// Minimal, grep-friendly output.
|
||||
for (const offender of offenders) {
|
||||
writeStdoutLine(`${offender.lines}\t${offender.filePath}`);
|
||||
}
|
||||
|
||||
return 1;
|
||||
}
|
||||
|
||||
try {
|
||||
const exitCode = await main();
|
||||
if (exitCode !== 0) {
|
||||
process.exit(exitCode);
|
||||
}
|
||||
} catch (error) {
|
||||
process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
|
||||
process.exit(1);
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user