I've been thinking quite a bit about how we build feedback loops into AI-powered threat modeling systems.
We're getting pretty good at using AI to generate threats. Give a model an architecture diagram, some information about the components and dataflows, and a little context about the application, and it can produce a reasonably convincing list of potential security issues.
Some of those threats will be useful. Others will be generic, redundant, or based on assumptions that don't hold up when you look at the actual architecture.
That's not particularly surprising. Security architects deal with the same problem when reviewing threat libraries and automated findings. The difference is that AI can generate a lot of content very quickly.
What interests me is what happens after someone reviews those findings.
If a security architect rejects a threat, changes its priority, or rewrites the attack scenario, what does the system do with that information?
In most cases, the answer is probably not much. The threat gets updated or removed, and everyone moves on.
But there's potentially a lot of value in those decisions, provided we can capture and interpret them correctly.
A rejected threat isn't necessarily a bad threat
Let's say an AI system generates a threat suggesting that an attacker could invoke an AWS Lambda function without authentication.
The security architect reviewing the model knows that the function isn't publicly exposed. It's invoked through an authenticated internal event source, so they reject the finding.
At first glance, that seems like a useful piece of feedback. The AI made an incorrect assumption, and the user corrected it.
But what exactly should the system learn?
It certainly shouldn't conclude that Lambda functions aren't susceptible to unauthorized invocation. A compromised identity, excessive IAM permissions, or a malicious event source could still create a legitimate attack path.
The actual issue was that the generated threat assumed an entry point that didn't exist in the architecture.
That's a much more useful lesson.
And it highlights one of the challenges with building feedback loops into threat generation: we need to understand the reasoning behind a decision, not just the decision itself.
A rejected threat could mean it wasn't applicable. It could mean the organization has already implemented sufficient controls. It could be a duplicate, an accepted risk, or simply something the reviewer doesn't consider important.
Those are very different outcomes.
If we treat every rejection as evidence that a threat is invalid, we're going to teach the system some bad habits.
We need better feedback than accept or reject
I think one of the first things we need to reconsider is how threat modeling systems collect feedback.
Accepting, rejecting, and editing threats are useful interactions, but they don't tell us enough on their own.
When someone rejects a threat, we should ideally understand why:
- Was an architectural assumption incorrect?
- Was an attack precondition missing?
- Was the threat already addressed by an existing control?
- Was the risk simply accepted?
We also need to preserve the context that led to the original recommendation.
If we know which components, dataflows, trust boundaries, assets, and security controls were considered, we can start connecting the reviewer's decision to the conditions under which the threat was generated.
That gives us something much more useful than a history of user actions.
It gives us examples of when particular threats do and don't apply.
Of course, there's a balance here. We can't expect security architects to complete a detailed questionnaire every time they dismiss a finding. The feedback process needs to fit naturally into the review workflow.
A few well-designed rejection reasons, combined with the architectural context the system already knows, could go a long way.
Does this mean we need to retrain the AI?
Not necessarily.
When people talk about AI learning from feedback, the conversation often jumps straight to fine-tuning.
That may eventually make sense, but I don't think it's where most threat modeling platforms should start.
There are several ways to use feedback to improve future results without changing the underlying language model.
One approach is to build a knowledge base of validated threat scenarios and architectural conditions. When the system encounters a similar architecture, it can retrieve relevant examples and use them to inform its analysis.
Another is to turn recurring feedback into applicability rules.
If a particular attack scenario requires an externally accessible endpoint, for example, the system should establish whether that endpoint exists before presenting the threat as applicable.
We can also use feedback to improve the instructions and reasoning processes used during threat generation.
If the model repeatedly makes incorrect assumptions about authentication, permissions, or trust boundaries, that's something we can address directly and then test.
Fine-tuning becomes more interesting once we've accumulated a meaningful collection of validated examples. Even then, we'd need to demonstrate that the trained model performs better on architectures it hasn't seen before.
The important thing is that the feedback has somewhere useful to go.
Collecting thousands of corrections doesn't accomplish much if we never turn them into improvements.
There's a bigger opportunity in understanding attack paths
I also think feedback loops become much more interesting when we move beyond generating threats against individual components.
It's relatively straightforward to associate a database with data disclosure threats or an API with authentication and authorization concerns.
But that's only part of the picture.
An attacker might enter through an exposed service, exploit a weak trust relationship, move through several components, and eventually reach a sensitive asset.
Whether that attack is realistic depends on how those components are connected, what permissions exist, where trust boundaries are crossed, and which controls are actually enforced.
Those relationships matter just as much as the individual technologies.
Now imagine a security architect reviewing an AI-generated attack path and identifying that one of the assumed connections doesn't exist, or that a particular control prevents the proposed movement between components.
That correction tells us something about the architecture and the conditions necessary for the attack.
Over time, collecting and validating this kind of feedback could help systems become better at distinguishing plausible attack paths from scenarios that merely sound plausible.
To me, that's a much more interesting direction than simply generating increasingly comprehensive lists of threats.
Not all feedback should carry the same weight
There's another challenge here that I don't think we can ignore.
Just because someone changes a threat doesn't mean the change is technically correct.
A developer might dismiss a finding because they believe a control is implemented. A security architect might disagree. Two organizations might assess the same threat differently because their risk tolerance, operating environments, or compensating controls are different.
We need to account for that.
I'd distinguish between feedback that represents a user preference, feedback that corrects an architectural fact, and feedback that has been independently validated.
A rejected threat with no explanation is still useful information, but I wouldn't want it automatically changing how the system behaves for everyone else.
A correction supported by architectural evidence or a confirmed security finding is a different story.
We also have to be careful about applying lessons across customers. An organization's architecture, security policies, and risk decisions aren't necessarily appropriate training material for another organization, even when the underlying technologies are similar.
The system needs some way to separate broadly applicable security knowledge from decisions that are specific to an environment.
How do we know it's actually getting better?
This is probably the most important part of the entire process.
If we're going to claim that a threat generation system improves through feedback, we need to be able to demonstrate it.
A lower rejection rate might be encouraging, but it doesn't necessarily mean the system is finding better threats. It could simply mean it's generating fewer findings or avoiding scenarios that are difficult to evaluate.
Likewise, producing more threats doesn't mean we're getting better coverage.
I'd want to measure how often generated threats are actually relevant, how well the system identifies known threats in representative architectures, and whether its proposed attack paths are supported by the system design.
I'd also pay attention to the amount of time security architects spend reviewing and correcting the output.
After all, one of the reasons we're using AI in the first place is to make threat modeling more efficient without sacrificing quality.
These measurements need to use architectures and scenarios that weren't part of the improvement process. Otherwise, we risk building a system that's very good at repeating examples it's already seen.
And we need to watch for regressions. Improving threat relevance for a typical web application doesn't help much if the same change causes the system to miss important risks in cloud infrastructure or operational technology environments.
Eventually, the feedback should extend beyond the threat model
There's an opportunity to connect these learning loops to what happens after threat modeling is complete.
If a generated threat results in a security test, a remediation task, or a code change, that downstream activity could provide additional context about the original recommendation.
Did testing confirm the attack path? Was a weakness actually identified? Was the recommended control implemented and verified?
We shouldn't assume that a threat is invalid simply because testing didn't uncover a vulnerability. But these outcomes can still help us understand which recommendations were useful and where the original analysis may have been incomplete.
I think connecting threat generation to real engineering and security outcomes is an important part of making these systems more useful over time.
Where I think this is heading
We're going to continue seeing improvements in how quickly AI can analyze architectures and generate potential threats.
But I'm not convinced that faster generation, by itself, is where the greatest value will come from.
A system that produces 100 threats and requires a security architect to spend hours cleaning them up may be less useful than one that produces 25 well-supported findings that accurately reflect the architecture.
What I'd really like to see is threat modeling systems that become better at understanding the environments they're analyzing.
Systems that can recognize when an attack precondition isn't satisfied, when an architectural assumption is unsupported, or when a control materially changes the feasibility of an attack.
Getting there will require more than better prompts or larger models.
We'll need to think carefully about how we collect feedback, how we validate it, and how we apply what we've learned without reinforcing incorrect assumptions.
The goal shouldn't just be to generate more threats. It should be to get better at identifying the threats that actually matter.
And I think building effective feedback loops is going to be a big part of getting there.