alvis/adolf
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
45 Commits 3 Branches 0 Tags
887d4b8d90d65916f9ab198efc5a566cd6a9da37
Commit Graph

8 Commits

Author SHA1 Message Date
alvis
887d4b8d90 voice benchmark: rename --dry-run → --no-inference, fix log extraction 
- --no-inference applies to all tiers (not just complex)
- metadata key: dry_run → no_inference
- extract_tier_from_logs: forward iteration (not reversed), updated regex
- GPU check skipped when --no-inference
- Fix TypeError in misclassified print when actual=None

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-24 07:58:05 +00:00
alvis
5b09a99a7f Routing: 100% accuracy on realistic home assistant dataset 
- router.py: skip light reply generation when no_inference=True;
  add control words (да/нет/стоп/отмена/повтори/подожди/etc.) to _LIGHT_PATTERNS
- agent.py: pass no_inference to router.route(); skip preflight IO in no_inference mode
- benchmarks/benchmark.json: replace definition-heavy queries with realistic
  Alexa/Google-Home style queries (greetings, smart home, timers, shopping,
  weather, personal memory, cooking) — 30 light / 60 medium / 30 complex

Routing benchmark: 120/120 (100%), all under 0.1s per query

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-24 07:53:01 +00:00
alvis
b7d5896076 routing benchmark: 1s strict deadline per query 
QUERY_TIMEOUT=1s — classification and routing must complete within
1 second or the query is recorded as 'timeout'.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-24 07:35:13 +00:00
alvis
47a1166be6 Merge pull request 'feat: rename --dry-run to --no-inference in run_benchmark.py' (#18) from feat/no-inference-benchmark into main 
Reviewed-on: #18
 2026-03-24 07:26:44 +00:00
alvis
0fbdbf3a5e Add run_routing_benchmark.py — dedicated routing-only benchmark 
Tests routing accuracy for all tiers with no_inference=True hardcoded.
Fast (QUERY_TIMEOUT=30s), no GPU check, shares benchmark.json dataset.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-24 07:25:16 +00:00
alvis
77db739819 Rename --dry-run to --no-inference, apply to all tiers in run_benchmark.py 
No-inference mode now skips LLM for all tiers (not just complex),
GPU check is auto-skipped, and the metadata key matches agent.py.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-24 03:49:09 +00:00
alvis
98095679be Fix benchmark log extraction: first tier match, increase log tail to 300 
- Remove reversed() from extract_tier_from_logs: first match = routing decision
  (dry-run complex logs tier=complex early, then overwrites with tier=medium at done)
- Increase log tail from 80→300 to handle concurrent log activity

Fixes #7, #10

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-24 02:42:27 +00:00
Alvis
bd951f943f Move benchmark scripts into benchmarks/ subdir 
- benchmarks/run_benchmark.py (was run_benchmark.py)
- benchmarks/run_voice_benchmark.py (was run_voice_benchmark.py)
- Scripts use Path(__file__).parent so paths resolve correctly in subdir
- .gitignore updated: ignore benchmarks/benchmark.json,
  results_latest.json, voice_results*.json, voice_audio/

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-24 02:02:46 +00:00