Eval notes
Daily notes on how AI systems are measured: benchmarks, indices, methods and their limits.
- 📖【Paper Deep Dive】IntentDelta and Evolving User Intentions A close reading of IntentDelta: human-authored intention updates, priority-aware evaluation, and the gap between recovering requirements and executing them.
- Model evaluation vs. harness evaluation What changes when the thing under test is the whole system around the model, why a bad agent result is not always a bad model, and where model and agent evaluation still fall short.
- Benchmarks saturate in months; four ways to summarize capability Why evaluations stop being useful so quickly, how ECI, the AA Intelligence Index, Arena and LiveBench each compress "capability" into one number, and where to look for benchmark data.