Research
Projects that turn those notes into data, tools and new benchmarks.
- Do agent benchmarks model real users? A comparison of τ-bench, τ²-bench, REAL, SWE-bench and TheAgentCompany on how realistically they represent the person asking for help. Most still assume the user already knows what they want.
- Which agentic benchmarks are saturated? A saturation score for 108 agentic benchmarks on BenchLM, combining distance to the ceiling with whether the top five models can still be told apart.
- Simulated users for multi-turn evaluation A harness where a simulated user, distilled from a real conversation, talks to the model under test. The conversations it produces are the evaluation set, and difficulty is six behavioral dials instead of one.