feat(airflow): integrate bench harness into bench_collect DAG
New DAG (`ml/pipelines/bench_dag.py`) with three linked tasks: 1. collect.py — generates candidates, logs to MLflow 2. export_for_judge — exports pending runs for Claude Code scoring 3. compare — generates leaderboard by (model, prompt) cell Config via dag_run.conf supports all collect.py options (models, prompts, n_tips, n_scenarios, temperature, experiment name, max_model_b). New admin API endpoints (`services/api/src/routes/bench.ts`): - GET /api/bench/experiments — list tip-bench-* experiments - POST /api/bench/run — trigger DAG with custom config - GET /api/bench/runs/:experiment — list runs in experiment - GET /api/bench/leaderboard/:experiment — leaderboard by (model, prompt) All endpoints require admin auth. Human judge (Claude Code) scores are applied manually post-export; future enhancement: add webhook to DAG. Admin UI can now trigger and monitor benchmarks from a dashboard panel. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -16,6 +16,7 @@ import { recommenderRouter } from './routes/recommender.js';
|
||||
import { userRouter } from './routes/user.js';
|
||||
import { pushRouter } from './routes/push.js';
|
||||
import { adminRouter, adminInternalRouter } from './routes/admin.js';
|
||||
import benchRouter from './routes/bench.js';
|
||||
import { mkdir } from 'fs/promises';
|
||||
import { dirname } from 'path';
|
||||
import { requireAuth } from './middleware/session.js';
|
||||
@@ -66,6 +67,7 @@ app.use('/api/user', userRouter);
|
||||
app.use('/api/push', pushRouter);
|
||||
app.use('/api/admin', adminRouter);
|
||||
app.use('/api/admin', adminInternalRouter);
|
||||
app.use('/api/bench', requireAuth as any, requireAdmin as any, benchRouter);
|
||||
|
||||
app.use('/api/ml', requireAuth as any, requireAdmin as any, async (req: Request, res: Response) => {
|
||||
const mlUrl = config.ML_SERVING_URL;
|
||||
|
||||
Reference in New Issue
Block a user