Embed Crawl4AI at all tiers, restore qwen3:4b medium, update docs

- Pre-routing URL fetch: any message with URLs gets content fetched
  async (httpx.AsyncClient) before routing via _fetch_urls_from_message()
- URL context and memories gathered concurrently with asyncio.gather
- Light tier upgraded to medium when URL content is present
- url_context injected into system prompt for medium and complex agents
- Complex agent retains web_search/fetch_url tools + receives pre-fetched content
- Medium model restored to qwen3:4b (was temporarily qwen2.5:1.5b)
- Unit tests added for _extract_urls
- ARCHITECTURE.md: added Tool Handling, Crawl4AI Integration, Memory Pipeline sections
- CLAUDE.md: updated request flow and Crawl4AI integration docs

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

CLAUDE.md

@@ -37,12 +37,18 @@ Adolf is a multi-channel personal assistant. All LLM inference is routed through
 Channel adapter → POST /message {text, session_id, channel, user_id}
                 → 202 Accepted (immediate)
                 → background: run_agent_task()
+                    → asyncio.gather(
+                        _fetch_urls_from_message()  ← Crawl4AI, concurrent
+                        _retrieve_memories()         ← openmemory search, concurrent
+                      )
                     → router.route() → tier decision (light/medium/complex)
-                    → invoke agent for tier via Bifrost
+                        if URL content fetched → upgrade light→medium
+                    → invoke agent for tier via Bifrost (url_context + memories in system prompt)
                         deepagents:8000 → bifrost:8080/v1 → ollama:11436
                     → channels.deliver(session_id, channel, reply)
                         → pending_replies[session_id] queue (SSE)
                         → channel-specific callback (Telegram POST, CLI no-op)
+                    → _store_memory() background task (openmemory)
 CLI/wiki polling → GET /reply/{session_id}  (SSE, blocks until reply)
 ```
 
@@ -59,7 +65,7 @@ Bifrost (`bifrost-config.json`) is configured with the `ollama` provider pointin
 | Tier | Model (env var) | Trigger |
 |------|-----------------|---------|
 | light | `qwen2.5:1.5b` (`DEEPAGENTS_ROUTER_MODEL`) | Regex pre-match or LLM classifies "light" — answered by router model directly, no agent invoked |
-| medium | `qwen2.5:1.5b` (`DEEPAGENTS_MODEL`) | Default for tool-requiring queries |
+| medium | `qwen3:4b` (`DEEPAGENTS_MODEL`) | Default for tool-requiring queries |
 | complex | `qwen3:8b` (`DEEPAGENTS_COMPLEX_MODEL`) | `/think ` prefix only |
 
 The router does regex pre-classification first, then LLM classification. Complex tier is blocked unless the message starts with `/think ` — any LLM classification of "complex" is downgraded to medium.
@@ -107,10 +113,14 @@ External (from `openai/` stack, host ports):
 
 The file is mounted into the bifrost container at `/app/data/config.json`. It declares one Ollama provider key pointing to `host.docker.internal:11436` with 2 retries and 300s timeout. To add fallback providers or adjust weights, edit this file and restart the bifrost container.
 
-### Agent tools
+### Crawl4AI integration
+
+Crawl4AI is embedded at all levels of the pipeline:
+
+- **Pre-routing (all tiers)**: `_fetch_urls_from_message()` detects URLs in any message via `_URL_RE`, fetches up to 3 URLs concurrently with `_crawl4ai_fetch_async()` (async httpx). URL content is injected as a system context block into enriched history before routing, and into the system prompt for medium/complex agents.
+- **Tier upgrade**: if URL content is successfully fetched, light tier is upgraded to medium (light model cannot process page content).
+- **Complex agent tools**: `web_search` (SearXNG + Crawl4AI auto-fetch of top 2 results) and `fetch_url` (single-URL Crawl4AI fetch) remain available for the complex agent's agentic loop. Complex tier also receives the pre-fetched content in system prompt to avoid redundant re-fetching.
 
-`web_search`: SearXNG search + Crawl4AI auto-fetch of top 2 results → combined snippet + full page content.
-`fetch_url`: Crawl4AI single-URL fetch.
 MCP tools from openmemory (`add_memory`, `search_memory`, `get_all_memories`) are **excluded** from agent tools — memory management is handled outside the agent loop.
 
 ### Medium vs Complex agent
@@ -122,12 +132,12 @@ MCP tools from openmemory (`add_memory`, `search_memory`, `get_all_memories`) ar
 
 ### Key files
 
-- `agent.py` — FastAPI app, lifespan wiring, `run_agent_task()`, all endpoints
+- `agent.py` — FastAPI app, lifespan wiring, `run_agent_task()`, Crawl4AI pre-fetch, memory pipeline, all endpoints
 - `bifrost-config.json` — Bifrost provider config (Ollama GPU, retries, timeouts)
 - `channels.py` — channel registry and `deliver()` dispatcher
 - `router.py` — `Router` class: regex + LLM classification, light-tier reply generation
 - `vram_manager.py` — `VRAMManager`: flush/poll/prewarm Ollama VRAM directly
-- `agent_factory.py` — `build_medium_agent` / `build_complex_agent` via `create_deep_agent()`
+- `agent_factory.py` — `build_medium_agent` (`_DirectModel`, single call) / `build_complex_agent` (`create_deep_agent`)
 - `openmemory/server.py` — FastMCP + mem0 config with custom extraction/dedup prompts
 - `wiki_research.py` — batch research pipeline using `/message` + SSE polling
 - `grammy/bot.mjs` — Telegram long-poll + HTTP `/send` endpoint