Frenos report: AI models plan OT attacks in ~90% of tests but inconsistently, with reliable capability estimated within two years
Matéria consolidada. Reúne 2 matérias que o Tifon tinha publicado separadamente sobre a mesma história. A IA reescreveu o relatório com o que as 2 fontes trouxeram em 27/10/2026 00:04; os links antigos trazem para cá.
Frenos, an OT security company, published its inaugural report on AI in OT security, based on 16,980 tests of frontier and open-weight models on attack planning against industrial systems. Three of six frontier models produced workable attack plans in roughly 90% of attempts, but results were inconsistent across repeated runs (industrialcyber.co, theregister.com). Frenos estimates operators have about two years before AI can reliably plan such attacks (industrialcyber.co). The findings do not show autonomous attacks. The models relied on target information supplied by Frenos and on skilled human oversight (industrialcyber.co). At a media panel before publication, Steve Swick (ex-CISO of American Electric Power), Paul Shaver (Armadin, ex-Mandiant) and Harry Thomas (Frenos) said AI does not yet turn script kiddies into nation-state attackers, but "the potential is there". They said AI speeds up reconnaissance, documentation analysis and tool development against utilities and manufacturing plants, eroding "security by obscurity" and possibly widening the pool of capable threat actors (theregister.com).
Detalhes técnicos
- Testing ran from October 2025 to September 2026 and totalled 16,980 tests: 13,884 decision attempts, 2,520 planning attempts and 576 simulated attacks/runs (theregister.com, industrialcyber.co). Industrialcyber.co describes more than 500 simulated OT attack scenarios.
- Models tested: six frontier models, two US open-weight models and two foreign open-weight models. Names were withheld (theregister.com, industrialcyber.co).
- Decision tests: pass rates ranged from 6.0% to 46.8% depending on the model, against about 39.5% for random selection. In 16.3% of questions the stated goal did not match the scoring rule (industrialcyber.co).
- Planning tests: frontier models F01, F04 and F07 succeeded in 91.9%, 89.5% and 92.9% of attempts (industrialcyber.co). Only 60%, 62.9% and 74.3% of their plans passed every repeat (theregister.com), or about two-thirds of repeated runs under identical conditions (industrialcyber.co).
- Simulated scenarios: models reached the goal in 100.0% of runs in one test set and 90.5% in the other, with step-by-step feedback (industrialcyber.co).
- Common failures: skipped steps (such as a data-transfer step) and assumptions about access that earlier steps had not established (theregister.com, industrialcyber.co).
- The strongest foreign open-weight model (X03) reached a 36.2% success rate, varying between 28.6% and 54.3% across rounds. It beat the strongest US open-weight model and three US frontier models (31.9%, 18.8%, 17.9%), though it trailed the leading frontier models (theregister.com, industrialcyber.co).
- The harness was simple and no human corrected the submitted plans. Real attackers with better tooling or human correction could do better (theregister.com, industrialcyber.co).
- Frenos concludes that model choice matters, that AI-generated plans need skilled human review, and that autonomous attacks were not demonstrated (industrialcyber.co).
Remediação
- Use layered defense in OT: network segmentation, limited remote access and user permissions, and tested business continuity and disaster recovery procedures (theregister.com).
- Prioritize detection and response, use AI to assess your own environments, and examine exposure across entire systems rather than individual vulnerabilities (industrialcyber.co).
- Treat the two-year estimate as preparation time, not a reason to wait (industrialcyber.co).
Relatos divergentes
- The report title differs: State of AI in Security (theregister.com) versus State of AI in OT Security (industrialcyber.co). The more recent industrialcyber.co (2026-10-26) text calls it the inaugural OT report.
- The 36.2% figure is called an "overall success rate" of the best open-weight model (theregister.com) but a planning-test result for model X03 (industrialcyber.co). The later source gives more detail.
- The scale of simulations is given as 576 simulated attacks (theregister.com) and as more than 500 scenarios and 576 runs (industrialcyber.co). These are probably consistent, but the sources do not reconcile them.
- The two-year estimate for reliable AI attack planning appears only in industrialcyber.co. The Register describes the potential without a timeline.
Linha do tempo
-
Frenos report: AI models plan OT attacks in ~90% of tests but inconsistently; reliable capability estimated within two years
ler a análise desta fonte ocultar
Frenos, an OT security company, published its inaugural State of AI in OT Security report. It estimates that critical infrastructure operators have about two years before AI models can reliably plan cyberattacks against industrial systems. The research tested frontier and open-weight models across more than 500 simulated OT attack scenarios, totalling 16,980 tests. Three of six frontier models produced workable attack plans in roughly 90% of attempts, but only about two-thirds of repeated runs passed under identical conditions. The findings do not show autonomous attacks. The models relied on target information supplied by Frenos and on skilled human oversight.
Technical Details
- Test volume: 13,884 decision attempts, 2,520 planning attempts and 576 simulated runs.
- Decision tests: pass rates ranged from 6.0% to 46.8% depending on the model, against about 39.5% for random selection. In 16.3% of questions the stated goal did not match the scoring rule.
- Planning tests: models F01, F04 and F07 succeeded in 91.9%, 89.5% and 92.9% of attempts. Model X03 succeeded in 36.2% and varied widely between rounds (28.6%–54.3%).
- Simulated scenarios: models reached the goal in 100.0% of runs in one test set and 90.5% in the other, with step-by-step feedback.
- Common failures included skipped steps (such as a data-transfer step) and assumptions about access that earlier steps had not established.
- The strongest foreign open-weight model outperformed the strongest U.S. open-weight model, though both trailed the leading frontier models.
- Model names were withheld, and no human corrected the submitted plans.
- Frenos concludes that model choice matters, that AI-generated plans need skilled human review, and that autonomous attacks were not demonstrated.
Remediation
Frenos recommends that defenders prioritize detection and response, use AI to assess their own environments, and examine exposure across entire systems rather than individual vulnerabilities. It also advises treating the two-year estimate as preparation time, not a reason to wait.
-
Frenos report: AI lowers barriers to OT attack planning, but frontier models remain inconsistent and need human oversight
ler a análise desta fonte ocultar
OT security experts warned at a media panel ahead of the Frenos State of AI in Security report that AI does not yet turn script kiddies into nation-state attackers, but "the potential is there". Panelists Steve Swick (former CISO of American Electric Power), Paul Shaver (Armadin, ex-Mandiant) and Harry Thomas (Frenos) said AI speeds up reconnaissance, documentation analysis and tool development against utilities and manufacturing plants. This erodes the "security by obscurity" that critical infrastructure operators have relied on and may increase the number of capable threat actors. Frenos tested AI models on attack planning against industrial systems and found them promising but unreliable without human judgment.
Technical Details
- Frenos ran 16,980 tests between October 2025 and September 2026: 13,884 decision attempts, 2,520 planning attempts and 576 simulated attacks.
- It tested six frontier models, two US open-weight models and two foreign open-weight models, whose names were not disclosed.
- Pass rates on single-action choices ranged from 6% to 47% depending on the model.
- The three strongest frontier models produced a workable attack plan in about 90% of planning attempts, but only 60%, 62.9% and 74.3% of plans passed every repeat. Plans often skipped steps or assumed access that had not been established.
- The best open-weight model, a non-US one, reached a 36.2% overall success rate, higher than three US frontier models (31.9%, 18.8%, 17.9%).
- The tests used a simple harness and no human feedback loop, so real-world attackers with a more capable harness or human correction could do better.
Remediation
Frenos recommends layered defense for OT environments: network segmentation, limiting remote access and user permissions, and testing business continuity and disaster recovery procedures.