Hacktron benchmark: open-weight GLM 5.2 and DeepSeek V4 trail frontier LLMs at finding Fider vulnerabilities by a narrowing margin
Hacktron tested eight LLMs, open-weight and frontier, in its fixed pentest and code-review orchestration. The target was Fider, an open-source Go and React feedback app. The test measured how many of 15 vulnerabilities patched between v0.33.0 and v0.36.1 (originally reported by XBOW and Aikido) each model could rediscover. The patches post-date the models' training cutoffs, which limits data-poisoning risk. Frontier models (GPT 5.5, Gemini 3.5 Flash) still led on critical and high-severity bugs, while open-weight DeepSeek V4 (8/15) and GLM 5.2 (7/15) beat Claude Opus 4.8 (6/15) in this run. Hacktron estimates open-weight models are roughly 3 to 6 months behind the frontier. This is vendor-run benchmarking on a small app, not a threat incident.
Technical Details
- Only GPT 5.5 and Gemini 3.5 Flash found the critical mass-assignment pre-auth account takeover (patch 74a26a31). Sonnet 4.6, Opus 4.8 and all open-weight models missed it. The bug relies on Go behaviour where exported struct fields can be overwritten by request JSON unless binding is explicitly restricted.
- The same models run inside Hacktron's orchestration outperformed GPT 5.5 in Codex and Opus 4.8 in Claude Code. The native agents missed the critical bug. The authors argue that deterministic orchestration gives better codebase coverage and avoids a false sense of coverage.
- Agents had no internet access and only restricted tools to read code and write test scripts.
- The authors suspect hidden guardrails or over-reasoning degrade Opus 4.8, but they state they are unsure of the cause.
- Caveats: the test uses a single small application, and the findings come from an internal run by the vendor.
Remediation
No remediation applies. For defenders using LLMs for code review, the article suggests using orchestration that ensures full coverage, validating findings, and using frontier models for critical or hard cases. Open-weight models can handle bulk workloads where cost and vendor dependency matter.
Linha do tempo
-
Hacktron benchmark: open-weight GLM 5.2 and DeepSeek V4 trail frontier LLMs at finding Fider vulnerabilities by a narrowing margin
ler a análise desta fonte ocultar
Hacktron tested eight LLMs, open-weight and frontier, in its fixed pentest and code-review orchestration. The target was Fider, an open-source Go and React feedback app. The test measured how many of 15 vulnerabilities patched between v0.33.0 and v0.36.1 (originally reported by XBOW and Aikido) each model could rediscover. The patches post-date the models' training cutoffs, which limits data-poisoning risk. Frontier models (GPT 5.5, Gemini 3.5 Flash) still led on critical and high-severity bugs, while open-weight DeepSeek V4 (8/15) and GLM 5.2 (7/15) beat Claude Opus 4.8 (6/15) in this run. Hacktron estimates open-weight models are roughly 3 to 6 months behind the frontier. This is vendor-run benchmarking on a small app, not a threat incident.
Technical Details
- Only GPT 5.5 and Gemini 3.5 Flash found the critical mass-assignment pre-auth account takeover (patch 74a26a31). Sonnet 4.6, Opus 4.8 and all open-weight models missed it. The bug relies on Go behaviour where exported struct fields can be overwritten by request JSON unless binding is explicitly restricted.
- The same models run inside Hacktron's orchestration outperformed GPT 5.5 in Codex and Opus 4.8 in Claude Code. The native agents missed the critical bug. The authors argue that deterministic orchestration gives better codebase coverage and avoids a false sense of coverage.
- Agents had no internet access and only restricted tools to read code and write test scripts.
- The authors suspect hidden guardrails or over-reasoning degrade Opus 4.8, but they state they are unsure of the cause.
- Caveats: the test uses a single small application, and the findings come from an internal run by the vendor.
Remediation
No remediation applies. For defenders using LLMs for code review, the article suggests using orchestration that ensures full coverage, validating findings, and using frontier models for critical or hard cases. Open-weight models can handle bulk workloads where cost and vendor dependency matter.