The rapid integration of artificial intelligence (AI) and large language models (LLMs) into enterprise environments has triggered a gold rush in innovation and a Wild, Wild West in cybersecurity.
Take OpenClaw, for instance. This agentic AI system has spooked security professionals with concerns ranging from over-permissive agents autonomously deleting user files to the unchecked spread of shadow AI. While the adage “prevention is better than cure” still holds true, security teams are struggling to catch up with preventive measures.
It was only in 2024, for instance, that multiple Southeast Asian countries besides Singapore introduced cybersecurity bills regulating critical infrastructure, mandating the need to model how attackers could compromise systems. This practice is known as threat modeling.
Threat Modeling for Executives
At the heart of threat modeling are four foundational questions, encapsulated in the Threat Modeling Manifesto:
- What are we working on?
- What can go wrong?
- What are we going to do about it?
- Did we do a good enough job?
AI challenges all four questions in ways that require us to rethink our threat modelling practices.
One of the most successful threat modeling methodologies is STRIDE, created by Loren Kohnfelder and Praerit Garg in 1999 and popularized by Microsoft through a book in 2004.
Threat modelling has become a crucial, proactive cybersecurity activity in software development. But with the proliferation of AI systems, a critical question emerges: Are traditional threat modeling methodologies sufficient to proactively identify threats specific to AI?
A Broad Umbrella
To answer that, we must understand that “AI” is a broad umbrella. Born at the 1956 Dartmouth Summer Research Project, AI has oscillated between hype and “winters” of stalled progress. It wasn’t until the 1990s and 2000s that sub-fields like machine learning (ML), computer vision and natural language processing (NLP) began to yield practical successes. Eventually, ideas from ML such as generative adversarial networks (GANs) were combined with NLP advancements to form generative AI (GenAI).
But it was the LLM that provided a direct, natural-language gateway for laymen to access AI, causing its popularity to explode over the last three years. GenAI’s open-ended, non-deterministic natural language inputs breaks traditional threat modeling.
How the Threat Modeling Practice Evolved
Traditional models like STRIDE were built on the basis that software is deterministic and attacks result in the loss of cybersecurity properties. However, GenAI introduces two new complications:
- GenAI systems are probabilistic. Outcomes are not reproducible across independent trials, but security properties are deterministic.
- Natural language inputs make it difficult to distinguish between data and information, and result in a malleable context window that introduces more complexity of human factors into technical solutions.
One example that exemplifies this is the Grandma attack where the attacker manipulates the chatbot through a “grandma persona” to solicit information, create harmful responses or write malicious code.
These attacks are executed through a technique called prompt injection. In traditional deterministic software, an attack like SQL injection exploits rigid, syntactical flaws. But prompt injection exploits the intended functionality of the LLM, which leads to the heart of the issue: prompt injection is the result of a conscious trade-off for functionality over security, and it is unlikely that prompt injection is fully mitigable.
Given that, we can only provide statistical security assurances for AI and even this comes with large variance of uncertainty, unlike the deterministic advice we used to give in solving traditional security vulnerabilities such as the usage of parameterized queries to solve SQL injection vulnerabilities. From a threat modeling perspective, we also need new frameworks to understand AI systems to model them and understand where we should spend our limited cybersecurity budget.
New Threat Modeling Frameworks
While cybersecurity always plays catch-up with innovation, the pace of cybersecurity has been much quicker too. The cybersecurity industry introduced a taxonomy of threats specific to AI systems known as MITRE ATLAS (Adversarial Threat Landscape for AI Systems), which illustrate how adversaries can attack AI systems (Q2 of the Threat Modeling Manifesto). Other frameworks such as OWASP Top 10 for LLMs/ML exist, but taxonomy alone is insufficient.
To effectively apply ATLAS across complex enterprise architectures, organizations have begun adopting frameworks like MAESTRO that help structure our understanding of AI systems (Q1 of the Threat Modeling Manifesto), map out the different layers of abstraction of agentic AI and arrive at a risk posture said AI system poses.
Similar to how MITRE ATT&CK has a complementary framework, MITRE D3FEND, security researchers have come up with a framework known as AIDEFEND that pools together mitigations against threats identified with adversary-based threat modelling frameworks mentioned above (Q3 of the Threat Modeling Manifesto).
Finally, security for AI systems is not a one-and-done checklist; it lives in tandem with the entire lifecycle of the AI system’s deployment. This requires us to continuously consider how risky the AI system is in production, bringing us to the final question: Q4: Did we do a good enough job?
This is where the National Institute of Standards and Technology (NIST) AI Risk Management Framework (AI RMF) becomes crucial, setting out a basis to build a RMF based around governance and three important pillars: map, measure and manage. The RMF goes beyond security, targeting other aspects of AI governance such as privacy, safety, fairness and explainability. These are also critical as such properties can be compromised through offensive tradecraft such as prompt injections and model inversion.
The below table summarizes the four-step approach, at a high level, on how we can visualize the frameworks that serve as a starting point to build a threat modeling program within your organization.
| Manifesto Question | The AI Security Challenge | Possible Frameworks | Purpose |
| 1. What are we working on? | AI is a “black box” with complex, non-deterministic workflows and agentic layers. | MAESTRO | Decomposes the AI system into distinct layers of abstraction (infrastructure, models, agents, plugins) to map the attack surface. |
| 2. What can go wrong | Attackers use natural language (Prompt Injection, Data Poisoning) instead of traditional code exploits. | MITRE ATLAS / OWASP Top 10 (for GenAI/LLMs/ML) | Provides a comprehensive taxonomy of AI-specific adversary tactics, techniques, and vulnerabilities. |
| 3. What are we going to do about it? | Identifying a threat in theory does not stop an attack in runtime production. | AIDEFEND | Operationalizes security by providing the active mitigations (semantic filtering, rate limiting, agent monitoring) needed to block attacks. |
| 4. Did we do a good enough job? | AI risk is continuous and involves complex human, privacy, and explainability factors. | NIST AI RMF | Provides the governance rubric to continuously measure, map, and manage AI risks (privacy, security, fairness) across the lifecycle. |
Call to Action: Structure Threat Modeling for AI Systems
The rapid adoption of AI means security teams can no longer rely on legacy methodologies designed for deterministic software. However, the security community has managed to catch up through a variety of frameworks such as MAESTRO, MITRE ATLAS, AIDEFEND and the NIST AI RMF.
Contextualizing these to the Threat Modeling Manifesto enables us to build a robust, end-to-end security program that incorporates AI systems to enable innovation without compromising security objectives. In other words, the security community is not helpless in catching up with what appears to be a frenetic pace of AI innovation.