Outside, parity; inside, a lottery begins
The most important fact here is uncomfortably simple: on the same application on the same day, the models diverged by 48 points in a white-box test. In the original test run report, it looks stark: Kimi K3 scored 88/100, Claude Opus 4.8 70/100, GPT-5.5 40/100.
Meanwhile, external reconnaissance showed almost no difference. All three models produced the same set: 7 findings, 0 CVEs, 5 IP addresses, and 10 ports. In other words, basic recon has truly become a commodity: the scanner found, the model retold.
But once access was granted, chaos erupted. Kimi K3, according to the test author, first read the frontend, extracted the API structure, and navigated it like a map: 20 steps, 20 findings, 3 live exploits, 0 failures.
Claude Opus 4.8 appeared more stable in behavior but weaker in coverage: 11 steps, 2 findings, 0 failures. It confirmed access, read one data source, and simply stopped. This is particularly frustrating in red teaming: formally, the model didn't crash, but in essence it under-checked the target.
GPT-5.5, by contrast, moved the longest but went off course. The report says the model early on decided the credentials weren't working, never revisited that decision, drifted to the public surface, and ultimately gave a false conclusion: 18 steps, 8 findings, 3 failures.
Separately, the cost of the process is striking. According to the same notes, Kimi consumed about 23k tokens per finding, Claude about 280k. If those numbers are accurately recorded, it's not just a cost difference but a difference in how the model maintains a working trajectory at all.
Why this matters for practice
The key takeaway here isn't about which model "won". It's that the phrases "found nothing" and "the model couldn't properly check" can easily disguise each other in a report.
Therefore, I'd look not at the final severity score but at the run mechanics: how many steps taken, where the model got stuck, whether it returned to hypotheses, what exactly it covered after login. A low score does not equal security. It often means only one thing: the critical path wasn't proven.
And that's no longer an academic nuance. If one model rates "medium" and another on the same target extracts a critical scenario with live exploits, the problem isn't the aesthetics of a benchmark but a false sense of mitigated risk.
From this case, I'd draw a very grounded rule: authenticated testing cannot be reduced to a single agent and a single verdict. Outside, LLMs are already fairly even, but inside an application they still have too much randomness to confuse the model's silence with the absence of vulnerability.