288 lines
11 KiB
Markdown
288 lines
11 KiB
Markdown
# Reasoning & Self-Reflection in Local LLM Agents
|
||
|
||
Research-backed notes on implementing multi-stage reasoning for local 4-8B models (2025).
|
||
|
||
---
|
||
|
||
## TL;DR
|
||
|
||
For local 4-8B models, **programmatic self-critique loops rarely justify their cost**.
|
||
Native thinking tokens (Qwen3 `enable_thinking=True`) or external verifiers
|
||
give better results at lower complexity. See bottom of this file for recommendations.
|
||
|
||
---
|
||
|
||
## Reasoning Patterns
|
||
|
||
### Chain-of-Thought (CoT)
|
||
|
||
Single forward pass, model thinks step-by-step before answering.
|
||
Zero implementation cost — just a prompt change.
|
||
Typical gain: +5-10pp on multi-step tasks vs no CoT.
|
||
No latency overhead beyond the extra output tokens.
|
||
|
||
### Reflexion (Shinn et al., NeurIPS 2023)
|
||
|
||
Multiple complete attempts. After each attempt, the model writes a textual critique
|
||
of what went wrong and stores it in episodic memory. Next attempt is conditioned on
|
||
that memory.
|
||
|
||
```
|
||
attempt 1 → fail → write critique → attempt 2 (reads critique) → ...
|
||
```
|
||
|
||
Key results (GPT-4): HumanEval 80% → 91% pass@1.
|
||
Cost: N complete task executions. At 30s/attempt, 5 trials = 2.5 minutes.
|
||
Implementation: https://github.com/noahshinn/reflexion
|
||
|
||
### Reflection Loop (in-turn revision)
|
||
|
||
Within a single turn: generate → critique → revise → [repeat].
|
||
Simpler than Reflexion. More common in practice.
|
||
|
||
```
|
||
Generate → Critique → Revise → [stop condition]
|
||
```
|
||
|
||
Stop condition options: max iterations, score threshold, external verifier passes.
|
||
|
||
### ReAct + Reflect
|
||
|
||
Standard ReAct (Reason + Act) with an added Reflect step after failed tool calls.
|
||
Most common production pattern. Adds 1-3 extra LLM calls per failed action.
|
||
|
||
### Tree of Thoughts (ToT)
|
||
|
||
Explore N reasoning branches simultaneously, evaluate each node, BFS/DFS search.
|
||
Branching factor 3, depth 3 = 54 LLM calls per problem. Prohibitive for local models.
|
||
Works only if the model has strong self-evaluation capability (typically ≥32B).
|
||
ToTRL-trained Qwen3-8B achieved 0.633 on AIME 2025 — but required training-time RL, not
|
||
a prompt trick.
|
||
|
||
### Graph of Thoughts (GoT)
|
||
|
||
Generalizes ToT to arbitrary DAGs: thoughts can merge, split, or loop.
|
||
62% improvement in sorting vs ToT, 31% cost reduction.
|
||
Implementation: https://github.com/spcl/graph-of-thoughts
|
||
Higher complexity than ToT; graph structure is problem-specific.
|
||
|
||
---
|
||
|
||
## Native Thinking Tokens (vs Programmatic Reflection)
|
||
|
||
Open models with built-in reasoning scratchpads:
|
||
|
||
| Model | Size | Ollama | Toggle | Notes |
|
||
|-------|------|--------|--------|-------|
|
||
| Qwen3 | 0.6B–235B | Yes | enable_thinking / think=True/False | Best option for local use |
|
||
| Qwen3-4B-Thinking-2507 | 4B | Yes | Always on | Dedicated thinking variant |
|
||
| QwQ-32B | 32B | Yes | Always on | Strong reasoning, needs VRAM |
|
||
| DeepSeek-R1 distills | 1.5B–70B | Yes | Always on | Llama/Qwen base |
|
||
|
||
### Qwen3 thinking toggle in Ollama / LangChain
|
||
|
||
```python
|
||
# LangChain
|
||
model = ChatOllama(model="qwen3:4b", think=True, num_ctx=8192)
|
||
|
||
# Prompt-level (Ollama API)
|
||
# /think — enable per-request
|
||
# /no_think — disable per-request
|
||
```
|
||
|
||
Latency: thinking mode is 2-3x slower in wall-clock time (model generates internal
|
||
`<think>...</think>` tokens before answering). Qwen3-VL 8B Thinking: 262s vs 65s on a
|
||
complex visual reasoning task — but meaningfully better output.
|
||
|
||
### Native thinking vs programmatic loop
|
||
|
||
| | Native thinking | Programmatic multi-stage |
|
||
|--|----------------|------------------------|
|
||
| API calls | 1 | N (rounds × 2) |
|
||
| Implementation | Zero | Significant |
|
||
| Quality on 4-8B | Good (capability in weights) | Poor (weak model critiques itself) |
|
||
| Transparency | Opaque (one streamed block) | Inspectable per stage |
|
||
| Controllability | thinking_budget only | Full control |
|
||
| Latency | 2-3x tokens, 1 call | N × base latency |
|
||
|
||
**For local 4-8B: native thinking almost always beats a hand-coded reflection loop.**
|
||
|
||
---
|
||
|
||
## Does Programmatic Reflection Work on Small Models?
|
||
|
||
Short answer: **mostly no without external verification**.
|
||
|
||
From the research (2024-2025):
|
||
|
||
- **"When Hindsight is Not 20/20" (arXiv 2404.09129)**: Self-reflection often makes
|
||
small models worse. A model that generated an error also lacks the capability to
|
||
identify it. It confidently accepts flawed reasoning on re-reading.
|
||
|
||
- **THINKSLM (EMNLP 2025)**: Inference-time self-critique on Llama-3.1-8B is unreliable.
|
||
Training-time distilled reasoning traces help; prompt-based self-critique does not.
|
||
|
||
- **Nature 2025 study**: Large gains (GPT-4: +18.5pp) diminish sharply for smaller models.
|
||
|
||
- **Latency cost**: Each reflection round on a local 8B adds 5-30s. A 3-round loop
|
||
= 3x latency for 0-5% gain (or regression) on most tasks.
|
||
|
||
### When it actually helps on small models
|
||
|
||
1. **External verifier**: model doesn't self-evaluate — it reads objective pass/fail
|
||
feedback (unit tests, JSON schema checker, math verifier, search result grader).
|
||
Most reliable pattern. No self-evaluation capability required.
|
||
|
||
2. **Stronger critic**: generate with 4B, critique with 32B or API model. Hybrid approach.
|
||
|
||
3. **Native thinking weights**: reflection happens in a single forward pass with
|
||
trained weights. Far more reliable than prompt-based self-critique.
|
||
|
||
4. **Structured error types**: code syntax, JSON validity, regex match — computable
|
||
error signal, not linguistic self-assessment.
|
||
|
||
---
|
||
|
||
## LangGraph Reflection Loop Implementation
|
||
|
||
LangGraph is suited for this because it supports cyclic graphs with state.
|
||
|
||
### Minimal reflection graph
|
||
|
||
```python
|
||
from langgraph.graph import StateGraph, START, END, MessagesState
|
||
from langchain_ollama import ChatOllama
|
||
|
||
llm = ChatOllama(model="qwen3:4b", think=False)
|
||
critic_llm = ChatOllama(model="qwen3:4b", think=True) # or stronger model
|
||
|
||
MAX_REFLECTIONS = 2
|
||
|
||
def generate(state):
|
||
response = llm.invoke(state["messages"])
|
||
return {"messages": [response], "iterations": state.get("iterations", 0)}
|
||
|
||
def reflect(state):
|
||
critique = critic_llm.invoke(
|
||
[{"role": "system", "content": "Critique this response. Be specific about errors."}]
|
||
+ state["messages"]
|
||
)
|
||
return {
|
||
"messages": [{"role": "user", "content": critique.content}],
|
||
"iterations": state["iterations"] + 1,
|
||
}
|
||
|
||
def should_reflect(state) -> str:
|
||
if state.get("iterations", 0) >= MAX_REFLECTIONS:
|
||
return END
|
||
# Optionally: check external verifier here
|
||
return "reflect"
|
||
|
||
graph = StateGraph(MessagesState)
|
||
graph.add_node("generate", generate)
|
||
graph.add_node("reflect", reflect)
|
||
graph.add_edge(START, "generate")
|
||
graph.add_conditional_edges("generate", should_reflect)
|
||
graph.add_edge("reflect", "generate")
|
||
agent = graph.compile()
|
||
```
|
||
|
||
### Self-Correcting RAG (CRAG pattern)
|
||
|
||
```
|
||
Retrieve → Grade documents → [rewrite query if bad] → Generate → Grade answer → [loop or END]
|
||
```
|
||
|
||
The document grader and answer grader are the "external verifiers" — they do
|
||
objective quality checks rather than linguistic self-critique.
|
||
LangChain tutorial: https://learnopencv.com/langgraph-self-correcting-agent-code-generation/
|
||
|
||
---
|
||
|
||
## Alternative Tooling
|
||
|
||
### DSPy (recommended for pipeline optimization)
|
||
|
||
DSPy treats prompts as learnable parameters. Define input/output signatures, run an
|
||
optimizer on examples, and DSPy auto-tunes prompts for your specific model.
|
||
|
||
```python
|
||
import dspy
|
||
lm = dspy.LM('ollama_chat/qwen3:4b', api_base='http://localhost:11434', api_key='')
|
||
dspy.configure(lm=lm)
|
||
|
||
class Reflect(dspy.Module):
|
||
def __init__(self):
|
||
self.gen = dspy.ChainOfThought("question -> answer")
|
||
self.critique = dspy.ChainOfThought("question, answer -> critique, improved_answer")
|
||
def forward(self, question):
|
||
first = self.gen(question=question)
|
||
return self.critique(question=question, answer=first.answer).improved_answer
|
||
```
|
||
|
||
Works with Ollama. Optimizer (BootstrapFewShot, MIPRO) tunes prompts automatically
|
||
but requires multiple LLM calls per training example — slow on local hardware.
|
||
|
||
### Outlines (structured output)
|
||
|
||
Constrained decoding — guarantees valid JSON/regex output from any model.
|
||
Use this inside a reflection loop to ensure the critic always returns structured feedback.
|
||
Works with Ollama via OpenAI-compatible API.
|
||
https://dottxt-ai.github.io/outlines/
|
||
|
||
### SGLang
|
||
|
||
High-performance GPU serving runtime (replaces Ollama for GPU inference).
|
||
Natively understands `<think>...</think>` tokens, caches KV-prefix across reflection
|
||
rounds (RadixAttention). If you replace Ollama with SGLang: reflection loops become
|
||
significantly cheaper because repeated prompt prefixes are cache-hit.
|
||
https://github.com/sgl-project/sglang
|
||
|
||
---
|
||
|
||
## Benchmarks Summary
|
||
|
||
| Setup | Task | Quality Gain | Latency Cost |
|
||
|-------|------|-------------|-------------|
|
||
| GPT-4 + Reflexion | HumanEval | +11pp (80→91%) | ~5x |
|
||
| GPT-4 + reflection | Problem solving | +18.5pp | ~3x |
|
||
| Llama-7B + programmatic self-critique | Math | +7.1% | ~3x |
|
||
| Local 8B + same-model critique (typical) | General | 0-5% (often regression) | 2-3x |
|
||
| Qwen3-8B + native thinking | AIME 2025 | Matches models 10x larger | 2-3x tokens |
|
||
| Any model + external verifier (tests) | Code | +15-26pp | 1.5-2x |
|
||
|
||
---
|
||
|
||
## Practical Recommendations for Adolf (local qwen3:4b / 8b)
|
||
|
||
| Goal | Approach | Cost |
|
||
|------|----------|------|
|
||
| Better reasoning on hard questions | `think=True` in ChatOllama | 2-3x latency, zero code |
|
||
| Code/JSON correctness | External verifier (schema check, exec) + retry loop | +1 LLM call on failure |
|
||
| Complex multi-step tasks | Route to qwen3:8b with `think=True` | model swap + 2-3x tokens |
|
||
| Full reflection loop | Only if using stronger critic model or external verifier | significant complexity |
|
||
| Avoid | Programmatic self-critique using same 4-8B model as critic | adds latency, no gain |
|
||
|
||
---
|
||
|
||
## References
|
||
|
||
- Reflexion (Shinn et al., NeurIPS 2023): https://arxiv.org/abs/2303.11366
|
||
- Tree of Thoughts: https://arxiv.org/abs/2305.10601
|
||
- ToTRL (Qwen3 RL training): https://arxiv.org/html/2505.12717v1
|
||
- Graph of Thoughts: https://arxiv.org/abs/2308.09687
|
||
- Adaptive GoT (2025): https://arxiv.org/pdf/2502.05078
|
||
- When Hindsight is Not 20/20: https://arxiv.org/html/2404.09129v1
|
||
- THINKSLM (EMNLP 2025): https://aclanthology.org/2025.emnlp-main.1659.pdf
|
||
- MAR — Multi-Agent Reflexion: https://arxiv.org/html/2512.20845
|
||
- Qwen3 technical report: https://arxiv.org/pdf/2505.09388
|
||
- Qwen3 thinking cost measurement: https://medium.com/@frankmorales_91352/the-computational-cost-of-cognitive-depth-qwen3-vl-8b-instruct-vs-thinking-2517b677ba29
|
||
- DeepSeek-R1: https://arxiv.org/abs/2501.12948
|
||
- LangChain reflection blog: https://blog.langchain.com/reflection-agents/
|
||
- LangGraph CRAG: https://learnopencv.com/langgraph-self-correcting-agent-code-generation/
|
||
- DSPy: https://dspy.ai/
|
||
- Outlines: https://dottxt-ai.github.io/outlines/
|
||
- SGLang: https://github.com/sgl-project/sglang
|
||
- graph-of-thoughts: https://github.com/spcl/graph-of-thoughts
|
||
- reflexion (original code): https://github.com/noahshinn/reflexion
|