UK AI Security Institute publishes benchmark results with full setup details
The UK's AI Security Institute is using EvalEval's open reporting format so others can see exactly how AI benchmark scores for six frontier models were produced.
By Eric Powlesland 3 min read
The UK AI Security Institute (AISI), a research group inside the UK government's Department for Science, Innovation and Technology, has started publishing its AI test results through open infrastructure built by the EvalEval Coalition. The aim is to make benchmark scores reproducible: anyone looking at a result should be able to see how it was produced.
As the Hugging Face Blog reported, AISI is releasing verified results, context and configuration details through EvalEval's Evaluation Cards platform. The data covers five benchmarks from the main experiment of an AISI paper titled "How Inference Compute Shapes Frontier LLM Evaluation." Six models are included: Claude Opus 4, Claude Opus 4.5, Claude Opus 4.6, GPT-5, GPT-5.2 and GPT-5.4. Results from two cyber security tests, called Cyber CTFs and The Last Ones, are also included, using a partly overlapping set of models.
EvalEval maintains a shared format for recording test results, called Every Eval Ever, and AISI's feedback helped shape it. The two groups first worked together at a workshop held alongside the NeurIPS 2025 research conference.
The underlying problem is simple. AI benchmark results are published in many formats and places, often without enough detail to repeat the test. Rerunning tests can also be very expensive. AISI's paper looks at how much scores depend on two things: how much computing effort a model is allowed to spend while answering, and the rules of the test itself.
One example from the post involves a benchmark called Humanity's Last Exam. When models were told after each attempt whether their answer was correct, they kept solving more tasks as they were allowed to use more processing. In other words, the same model on the same test can post noticeably different results depending on the setup. The post also shows AISI's results on another benchmark, Terminal-Bench 2.0, alongside results others reported for the same models under different conditions.
The post does not name all five benchmarks in the main experiment, and it does not give specific scores, so it is not yet clear how large the differences are across every test.
What it means for your business
Most small and mid-sized businesses will never read an Evaluation Card. The practical lesson still applies to anyone choosing AI tools.
When a vendor says its model beat a competitor on some benchmark, that number may depend heavily on how the test was run. A higher score achieved with more computing time, extra attempts or hints along the way is not the same as a higher score under ordinary conditions. EvalEval's own description makes the point: its tools help show when similar-looking scores came from meaningfully different conditions.
A few sensible habits follow from this:
- Treat headline benchmark comparisons as a rough signal, not proof that one tool will work better for you.
- If a vendor leans on a score, ask how the test was run and whether the setup matches how you would actually use the product.
- Your own trial on your own tasks is still the most reliable test.
The longer-term benefit is indirect. If more testers publish results in a shared format with setup details attached, it becomes easier for researchers, journalists and buyers to spot comparisons that are not like for like.
What to watch
EvalEval is asking model developers to report verified results and asking benchmark creators to use the Every Eval Ever format. Whether major AI companies follow AISI's lead will decide how useful this becomes. Because AISI is a government body, its adoption could also influence how AI testing is documented in future policy work, though the post makes no specific claims about that.
