9b77d6ea67
Backend: - Replace on-the-fly Ollama calls with versioned feature store (task_features, task_edges) - Background Tokio worker drains pending rows; write path returns immediately - MLConfig versioning: changing model IDs triggers automatic backfill via next_stale() - AppState with FromRef; new GET /api/ml/status observability endpoint - Idempotent mark_pending (content hash guards), retry failed rows after 30s - Remove tracked build artifacts (backend/target/, frontend/.next/, node_modules/) Frontend: - TaskItem: items-center alignment (fixes checkbox/text offset), break-words for overflow - TaskDetailPanel: fix invisible AI context (text-gray-700→text-gray-400), show all fields - TaskDetailPanel: pending placeholder when latent_desc not yet computed, show task ID - GraphView: surface pending_count as amber pulsing "analyzing N tasks…" hint in legend - Fix Task.created_at type (number/Unix seconds, not string) - Auth gate: LoginPage + sessionStorage; fix e2e tests to bypass gate in jsdom - Fix deleteTask test assertion and '1 remaining'→'1 left' stale text Docs: - VitePress docs in docs/ with guide, MLOps pipeline, and API reference Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
91 lines
2.9 KiB
Markdown
91 lines
2.9 KiB
Markdown
# Feature Pipeline
|
||
|
||
## Worker lifecycle
|
||
|
||
```
|
||
startup
|
||
│
|
||
▼
|
||
notify.notify_one() ← wake immediately to drain any pending rows
|
||
│
|
||
▼
|
||
loop:
|
||
drain loop:
|
||
next_stale() ──► None → break
|
||
│
|
||
▼
|
||
generate_description(title)
|
||
│ error → set_failed(current model IDs), sleep 5s, break
|
||
▼
|
||
get_embedding(description)
|
||
│ error → set_failed(current model IDs), sleep 5s, break
|
||
▼
|
||
UPDATE task_features SET status='ready', embedding=blob, …
|
||
│
|
||
▼
|
||
recompute_for_task(task_id)
|
||
DELETE task_edges WHERE source=id OR target=id
|
||
load all other 'ready' embeddings
|
||
INSERT pairs with cosine_sim ≥ min_similarity
|
||
|
||
tokio::select!
|
||
notified() ← new task created/updated
|
||
sleep(60s) ← retry failed rows
|
||
```
|
||
|
||
## Content hash and cache invalidation
|
||
|
||
```
|
||
content_hash = sha256( prompt_version || "\0" || title )
|
||
```
|
||
|
||
A `task_features` row is considered **stale** when:
|
||
- `status = 'pending'` — explicitly queued
|
||
- `status = 'failed'` and `updated_at < now − 30s` — retry after backoff
|
||
- `status = 'ready'` and `desc_model ≠ config.desc_model` — model changed
|
||
- `status = 'ready'` and `embed_model ≠ config.embed_model`
|
||
- `status = 'ready'` and `prompt_version ≠ config.prompt_version`
|
||
|
||
A stale-but-ready row serves its existing data until the worker overwrites it, so the graph never shows a "hole" during recomputation.
|
||
|
||
## Changing models
|
||
|
||
Edit `backend/src/ml/config.rs`:
|
||
|
||
```rust
|
||
pub fn default() -> Self {
|
||
Self {
|
||
desc_model: "qwen2.5:7b".to_string(), // upgraded
|
||
embed_model: "nomic-embed-text".to_string(),
|
||
prompt_version: "v2".to_string(), // bump when prompt changes
|
||
min_similarity: 0.75, // wider edges
|
||
..
|
||
}
|
||
}
|
||
```
|
||
|
||
On the next startup (or `notify_one()`), `next_stale()` returns every row whose stored config no longer matches. The worker re-runs them in oldest-first order.
|
||
|
||
## Prompt versioning
|
||
|
||
The prompt template is matched on `prompt_version` in `ml/ollama.rs::render_prompt`. Old versions remain compilable — bumping the version adds a new match arm rather than overwriting the old one, so descriptions produced by `v1` can always be reproduced.
|
||
|
||
```rust
|
||
fn render_prompt(prompt_version: &str, task_title: &str) -> String {
|
||
match prompt_version {
|
||
"v2" => format!("…new prompt…{task_title}…"),
|
||
_ => format!("…v1 prompt…{task_title}…"), // "v1" and legacy
|
||
}
|
||
}
|
||
```
|
||
|
||
## Embedding storage
|
||
|
||
Embeddings are stored as raw little-endian f32 bytes in a `BLOB` column:
|
||
|
||
```
|
||
[f32 LE] [f32 LE] [f32 LE] … (768 floats for nomic-embed-text = 3072 bytes)
|
||
```
|
||
|
||
`encode_embedding` / `decode_embedding` in `ml/features.rs` handle the conversion. The `embed_dim` column records the dimension so readers don't have to hard-code the model's output size.
|