Benchmark: light tier over-classified as medium (tech definition queries) #8

New Issue

alvis · 2026-03-24T01:58:05Z

alvis commented

2026-03-24 01:58:05 +00:00

Problem

Queries that should be light (static tech definitions) are routed to medium:

"что такое API" → medium
"сколько бит в байте" → medium
"привет, как дела?" → medium
"что такое брандмауэр" → medium
"что такое Z-Wave" → medium
"что такое Matter протокол" → medium

Root Cause

The _LIGHT_PATTERNS regex only matches exact greetings/acks. Everything else falls through to the semantic embedder. The embedder centroids overlap for short Russian tech definitions — they embed similarly to medium-tier queries because both are short question forms.

Also _LIGHT_PATTERNS uses ^...$ anchoring but the text includes trailing punctuation (?, !) which may not always match the [\s!.?]*$ suffix.

Fix

Strengthen _LIGHT_PATTERNS to catch что такое <term> and что означает <term> patterns directly
Add more Russian tech-definition utterances to _LIGHT_UTTERANCES to pull the light centroid closer
Consider a separate regex pass for сколько <unit> в <unit> patterns

Impact

~10 light queries misclassified as medium in latest run

## Problem Queries that should be `light` (static tech definitions) are routed to `medium`: - "что такое API" → medium - "сколько бит в байте" → medium - "привет, как дела?" → medium - "что такое брандмауэр" → medium - "что такое Z-Wave" → medium - "что такое Matter протокол" → medium ## Root Cause The `_LIGHT_PATTERNS` regex only matches exact greetings/acks. Everything else falls through to the semantic embedder. The embedder centroids overlap for short Russian tech definitions — they embed similarly to medium-tier queries because both are short question forms. Also `_LIGHT_PATTERNS` uses `^...$` anchoring but the text includes trailing punctuation (`?`, `!`) which may not always match the `[\s!.?]*$` suffix. ## Fix - Strengthen `_LIGHT_PATTERNS` to catch `что такое <term>` and `что означает <term>` patterns directly - Add more Russian tech-definition utterances to `_LIGHT_UTTERANCES` to pull the light centroid closer - Consider a separate regex pass for `сколько <unit> в <unit>` patterns ## Impact ~10 light queries misclassified as medium in latest run

alvis referenced a pull request that will close this issue

2026-03-24 02:45:42 +00:00

Fix routing: add Russian tech def patterns to light, strengthen medium smart home #13

alvis referenced this issue from a commit

2026-03-24 02:45:43 +00:00

Fix routing: add Russian tech def patterns to light, strengthen medium smart home

alvis closed this issue

2026-03-24 02:51:17 +00:00

Sign in to join this conversation.