The Guardrail Paradox: When Safety Measures Become the Threat
AI cybersecurity guardrails were designed with a clear purpose: prevent malicious actors from weaponising powerful language models to launch cyberattacks at scale. But an unintended consequence is rapidly becoming a serious problem for the security community. Legitimate offensive cybersecurity researchers — professionals whose entire job is to find vulnerabilities before criminals do — are finding that the same restrictions blocking bad actors are also blocking them. And the fallout is pushing some of the most security-conscious professionals in the industry toward AI alternatives that carry their own set of risks.
The tension has been building for months. Major AI labs including Anthropic and OpenAI have constructed elaborate vetting programs and usage restrictions on their frontier models, particularly around anything touching offensive security research. The programs — OpenAI's Trusted Access for Cyber and Anthropic's Cyber Verification Program — require researchers to apply, get approved, and even then operate within restricted boundaries. For researchers who deal in zero-days, exploit development, and vulnerability analysis, the friction is creating real workflow problems. As TechCrunch reports, the guardrails are not just inconvenient — they may be counterproductive to the very security goals they aim to protect.
What Offensive Security Researchers Actually Need AI For
To understand why guardrails create such friction, it helps to understand how offensive security professionals use AI in practice. The work is not about asking a chatbot to "hack this server." Rather, AI tools are being used across a range of technical tasks where speed and pattern recognition genuinely accelerate research: reverse engineering unfamiliar code, understanding the structure of compiled binaries, building supporting automation tools, and — critically — confirming whether a discovered bug constitutes a real, exploitable vulnerability.
Chris Anley, chief scientist at NCC Group, one of the world's leading security consulting firms, articulates the core problem clearly. Asking an AI model to attempt to exploit a known bug is a legitimate and essential step in determining whether that bug is worth patching. If the model refuses because a guardrail flags the request as potentially malicious, the result is not safety — it's obstruction of the defensive process itself.
"Fix this code as a prompt is both an essential mechanism for defense but also a roadmap for finding critical vulnerabilities in the code base. The same tool is both an offensive tool and a defensive tool, and the two can't really be unpicked."
— Chris Anley, Chief Scientist, NCC GroupAnley's analogy is apt: it is like a hammer. You cannot build a house without one, but it is also, irreducibly, a weapon. The dual-use nature of security tooling is not a design flaw — it is an inherent property of the discipline. Guardrails that cannot account for this duality end up penalising legitimate defenders while determined bad actors simply find workarounds.
Paolo Stagno, CTO at CrowdFense — a firm that develops, acquires, and sells previously unknown vulnerabilities to government agencies — adds another dimension to the debate. He and his team do use frontier AI models, but only for reverse engineering tasks. For vulnerability discovery and exploit development, they deliberately avoid cloud-based models entirely. The concern is not guardrails; it is data sovereignty. Feeding sensitive vulnerability research into a cloud-hosted model risks that data being absorbed into future training runs or simply leaking through API infrastructure. Instead, CrowdFense runs open-source models locally, keeping sensitive work air-gapped from external servers.
Why Inconsistent Enforcement May Be a Bigger Problem Than Strict Rules
Perhaps the most damaging criticism of current AI guardrail systems is not that they are too strict, but that they are unpredictable. Chris Thompson, CEO of cybersecurity firm RemoteThreat and founder of Offensive AI Con — a conference dedicated to the intersection of offensive security and AI — describes an experience that will resonate with any developer who has tried to use frontier models for security work: the guardrails behave differently from session to session, even within the supposedly loosened boundaries of vetted programs.
"The practical impact is you spend a lot of time negotiating with the model instead of working on the core security program. Instead of analyzing a vulnerability, you're trying to find why you're getting inconsistent results or why models are over-sanitizing the output."
— Chris Thompson, CEO, RemoteThreatThis inconsistency is operationally corrosive. Security research demands reproducibility. If an AI model helps you analyse a code path one day and refuses the next, it cannot be integrated into a reliable workflow. The result is wasted time on meta-level prompt engineering rather than actual security work — a cost that falls entirely on the defender, not the attacker.
One anonymous researcher at a smartphone-component manufacturer described a particularly stark example. Their employer is not enrolled in Anthropic's Cyber Verification Program, meaning they operate under standard guardrails. The outcome is that AI tools become essentially non-functional for security purposes: "If it catches wind we're doing anything security related, it just stops and isn't usable," the researcher said. For a company that presumably has significant interest in securing its own products, this represents a meaningful capability gap.
The Digital Sovereignty Risk: Researchers Turning to Chinese Open-Source Models
The practical consequence of over-restrictive or inconsistent guardrails is one that should concern policymakers and IT decision-makers far more than a missed prompt response: legitimate Western security researchers are turning to Chinese open-source models. Thompson specifically names GLM — a family of freely downloadable language models that can be run locally without vetting requirements or usage restrictions — as the destination researchers are being pushed toward.
From a pure workflow perspective, the appeal is obvious. Local deployment eliminates data sovereignty concerns. No vetting process means no waiting. No guardrails means no negotiating with the model. For researchers working on sensitive vulnerability research, this combination is genuinely attractive. But the implications are significant. Running security research through models developed and maintained by Chinese organisations introduces its own set of supply chain and data security considerations — considerations that, paradoxically, may outweigh the risks the AI labs' guardrails were designed to prevent.
This dynamic maps directly onto broader debates around AI regulation and digital sovereignty that European technology policy professionals are grappling with. The EU AI Act, which came into full effect for high-risk systems, establishes risk-based classifications for AI applications — but it does not specifically resolve the dual-use dilemma in cybersecurity. When guardrail regimes push security professionals toward less regulated alternatives, the net effect on digital sovereignty and national security may be negative, regardless of intent.
Export Controls and the Anthropic Model Controversy
The political dimension of this debate became explicit when the U.S. government imposed export control restrictions on Anthropic's AI models Mythos and Fable. The move followed a report suggesting it was possible to bypass the models' guardrails in ways that could enable cyberattacks. The export controls on Fable 5 and Mythos 5 have since been lifted — Fable 5 returned to general access on July 1, while Mythos 5 has been reintroduced only to vetted U.S. organisations as part of an ongoing government review process.
The episode illustrates a feedback loop that should worry security professionals and policy analysts alike. AI labs market their most powerful models as uniquely dangerous, requiring strict controls and careful vetting. Governments respond to that framing with export restrictions and oversight mechanisms. The result is a regulatory environment built on the premise that these models are inherently dual-use weapons — which then makes it harder to use them for legitimate defensive purposes. As Brookings Institution analysis on AI governance has noted, the framing of AI capabilities as weapons creates governance challenges that differ fundamentally from traditional arms control.
Mark Dowd, a veteran security researcher with decades of experience finding and selling zero-day vulnerabilities to Western governments, frames the underlying governance issue bluntly. Speaking on a cybersecurity podcast, he expressed discomfort with large commercial companies making unilateral decisions about what constitutes safe security research. His point is not that guardrails are inherently wrong — it is that the entities drawing the lines have no particular qualification to do so, and their commercial incentives do not necessarily align with the security community's needs.
Open-Source AI as a Data Sovereignty Tool — With Caveats
The shift toward locally-run open-source models among security researchers is not simply a workaround for guardrails. For organisations handling sensitive vulnerability data, it reflects a principled data sovereignty decision. Stagno's reasoning at CrowdFense is instructive: cloud-based frontier models, regardless of their guardrail policies, represent a potential data exfiltration vector for work that is highly sensitive by nature. Feeding an undisclosed zero-day into any external API — regardless of the provider's terms of service — creates risk.
This perspective aligns with growing European interest in sovereign AI infrastructure. Initiatives like the European Commission's AI strategy emphasise the importance of European organisations maintaining control over their data and computational infrastructure. For IT decision-makers evaluating AI tools for security workflows, the calculus is not simply "which model is most capable" — it is "which model can be used without creating new attack surfaces or data sovereignty exposures."
However, the open-source alternative is not without its own risks. Running models locally requires significant infrastructure investment. Model quality for specialised security tasks may lag behind frontier models. And as Thompson's concern about GLM illustrates, the provenance of open-source models matters — a model developed and maintained by a foreign government-adjacent organisation introduces supply chain risks that are difficult to audit. The ENISA Threat Landscape report has consistently flagged supply chain compromise as one of the most significant and growing vectors in European cybersecurity — a risk that applies to AI model supply chains as much as software dependencies.
| AI Tool Category | Guardrail Restrictions | Data Sovereignty Risk | Suitability for Security Research |
|---|---|---|---|
| Frontier models (standard) | Originally reported by TechCrunch. Summarised and curated by European Purpose.
|
