Notes and research on how we measure AI.
I keep daily notes on AI evaluation: what benchmarks measure, how capability indices are built, and where they break. The gaps I find turn into research and, eventually, new benchmarks.
Eval notes
All notes →- 📖【Paper Deep Dive】IntentDelta and Evolving User Intentions A close reading of IntentDelta: human-authored intention updates, priority-aware evaluation, and the gap between recovering requirements and executing them.
- Model evaluation vs. harness evaluation What changes when the thing under test is the whole system around the model, why a bad agent result is not always a bad model, and where model and agent evaluation still fall short.
- Benchmarks saturate in months; four ways to summarize capability Why evaluations stop being useful so quickly, how ECI, the AA Intelligence Index, Arena and LiveBench each compress "capability" into one number, and where to look for benchmark data.
Research
All research →- Do agent benchmarks model real users? A comparison of τ-bench, τ²-bench, REAL, SWE-bench and TheAgentCompany on how realistically they represent the person asking for help. Most still assume the user already knows what they want.
- Which agentic benchmarks are saturated? A saturation score for 108 agentic benchmarks on BenchLM, combining distance to the ceiling with whether the top five models can still be told apart.
- Simulated users for multi-turn evaluation A harness where a simulated user, distilled from a real conversation, talks to the model under test. The conversations it produces are the evaluation set, and difficulty is six behavioral dials instead of one.