AI Threat Modelling: What It Is, and How to Actually Practice It
Machine learning systems fail in ways that traditional threat modelling frameworks were never built to catch. A SQL injection checklist won't help you spot a prompt injection vector. A network diagram won't show you where your training data can be poisoned. As more products ship an "AI feature" bolted onto a web app, security teams need a way to reason about the AI-specific attack surface — not just the infrastructure around it. That's the gap TryHackMe's learning path is built to close, and the AI Threat Modelling Assessment room (an easy, ~15-minute practical) is the capstone: an interactive application where you apply the methodology hands-on and capture two flags to prove you can do it, not just describe it.
Why AI systems need their own threat model
Classic threat modelling (STRIDE, PASTA, attack trees) assumes a mostly deterministic system: the same input produces the same output, and the boundaries between "trusted" and "untrusted" data are usually clear. AI systems break both assumptions.
A few things that make AI threat surfaces distinct:
The model itself is an asset. It can be stolen (model extraction), reverse-engineered, or degraded through targeted queries — attacks that have no equivalent in a normal codebase.
Training data is an attack surface. If an adversary can influence what a model learns from, they can poison it — quietly steering outputs, planting backdoors, or degrading accuracy for specific inputs.
Inputs and instructions share a channel. In a traditional app, code and data are separate. In an LLM-based system, both often arrive as plain text in the same prompt, which is what makes prompt injection possible — a user (or a document, or a webpage the model reads) can smuggle in instructions the system wasn't meant to follow.
Outputs can be actions. When a model has tool access — sending emails, querying databases, executing code — a manipulated output isn't just wrong text, it's an unauthorized action.
Non-determinism complicates testing. The same prompt can behave differently across runs, so "it worked once" isn't the same as "it's safe."
A practical framework for assessing an AI system
When you're threat modelling an AI-integrated application, it helps to walk the system asset by asset rather than trying to reason about "the AI" as one monolithic thing.
Map the data flow. Where does training data come from? Where do runtime prompts and user inputs enter? What does the model have read/write access to (files, APIs, databases, other services)?
Identify trust boundaries. Anywhere untrusted content — user input, a fetched webpage, an uploaded document, a retrieved database record — reaches the model is a boundary worth scrutinizing, because the model may not reliably distinguish "instructions from my developer" from "text I was asked to summarize."
Enumerate AI-specific threats at each stage:
Data collection/training → data poisoning, biased or backdoored training sets
Model → model theft/extraction, inversion attacks that leak training data, adversarial examples that flip a classification
Prompt/input layer → direct and indirect prompt injection, jailbreaks
Output/action layer → excessive agency (the model taking actions beyond what's needed), insecure output handling (treating model output as safe to execute or render), leaking sensitive data pulled into context
Supply chain → compromised pre-trained models, malicious plugins, poisoned fine-tuning datasets
Rate and prioritize. Not every theoretical AI attack matters for every system — a model with no tool access and no sensitive data in context has a much smaller blast radius than an autonomous agent wired into production systems. Weight your findings by actual impact and exploitability.
Map mitigations to the specific threat, not generically — input/output filtering for injection, least-privilege tool scoping for excessive agency, data provenance checks for poisoning, rate-limiting and query monitoring for extraction attempts.
This maps closely to frameworks like the OWASP Top 10 for LLM Applications, which is worth having open side-by-side while you work through any AI threat modelling exercise — it's become something of a shared vocabulary for this exact set of risks.
Turning it into a skill, not just a checklist
Reading about prompt injection and actually recognizing it in a live application are different skills. That's the value of a hands-on assessment: it forces you to apply the framework under the same ambiguity you'd face in a real engagement, where the vulnerability isn't labeled for you.
If you want to test yourself the same way, TryHackMe's path builds up to exactly that:
Two prerequisite rooms cover the fundamentals of AI/ML and how those concepts translate into real vulnerabilities, plus a structured methodology for treating an AI system as an attack surface.
The AI Threat Modelling Assessment room then hands you an interactive application with no hand-holding — you apply what you learned to capture two flags, which is TryHackMe's way of verifying you can actually do threat modelling, not just recite it.
It's rated easy and takes about 15 minutes, but that's deceptive — most of the work happened in the modules before it. The room itself is where you find out if the methodology actually stuck.
The takeaway
AI threat modelling isn't a replacement for traditional application security — it's an addition to it. You still need to worry about the API, the database, the auth layer. But the model, the training data, the prompt boundary, and the actions the AI is allowed to take are a new set of assets and a new set of failure modes, and they deserve the same rigor a network diagram or a data flow diagram would get in a conventional threat model. Practicing on a deliberately built assessment — where you have to find the gap yourself — is a good way to make sure that rigor is actually a skill you have, not just a framework you've read about.