Survivability Engineering, Part 4: Threat Modeling with Reality

Threat modeling should help you understand how things break, not just how systems are designed. The goal is not to produce a clean diagram or check every category. The goal is to make better decisions about risk in an environment where behavior is messy, systems change, and failure is guaranteed at some point.

Start with outcomes. What actually hurts the business. Data loss, loss of control, operational disruption, irreversible actions. Keep it concrete. If it does not matter when it breaks, it is not worth modeling. This keeps the focus on real impact instead of theoretical threats.

From there, map the paths to those outcomes. Not individual weaknesses, but sequences. How does something small turn into something big. What steps would an attacker or a misbehaving agent take to get from low impact to high impact. Where are the pivots between systems, identities, and data. This is where most risk hides.

Bring in susceptibility. Where can influence happen. What inputs are trusted. What integrations expand exposure. Where can a human or an agent be nudged into doing the wrong thing. Assume that inputs will be manipulated and that behavior can drift over time.

Then look at damage. If that path is successful, what can actually happen. What does the system allow. What permissions exist. What actions can be taken without friction. This is where blast radius is defined. If something goes wrong, how far does it spread.

Now focus on recovery time. If that same path is triggered, how quickly do you know. How quickly can you stop it. How quickly can you return to a known good state. Detection without response does not reduce impact. Response without speed does not reduce disruption.

Ask the same question at every step. When something goes wrong, how bad does it get and how long does it last.

Once that is clear, apply controls with intent. Limit what can be reached and what can be done. Keep permissions tight and scoped. Break high impact workflows so they cannot be completed in a single path without checks. Add visibility where behavior matters, not just where systems log events. Build the ability to interrupt actions in real time. Make rollback and recovery part of the design, not an afterthought.

Treat humans and AI agents as part of the system, not exceptions to it. Model how they behave, how they can be influenced, and how they might fail. Apply the same controls you would use for a high risk operator. Least privilege, separation of duties, monitoring of behavior, and the ability to step in quickly when something looks wrong.

Keep the model alive. Systems change, so the model should change with them. Revisit it when new integrations are added, when permissions expand, when workflows change. This is not a one time exercise. It is part of how you operate.

Keep it simple enough to use. If it becomes a heavy process, it will be ignored or it will lag behind reality. The value is in how often and how quickly it can be applied, not how detailed it looks on paper.

This is threat modeling meets reality. It accepts that attackers chain actions, that behavior is unpredictable, and that failure will happen. It focuses on limiting impact and shortening recovery instead of trying to predict every possible path.

Take a simple but real scenario. A company deploys an AI agent that can read emails, pull data from internal systems, and create or update support tickets. It can also trigger actions in a CRM and a billing system to “help” operations move faster.

Start with outcomes. What actually hurts. Exposure of sensitive customer data. Unauthorized changes to billing. Creation of fraudulent transactions. Loss of control over internal systems. Those are the things that matter.

Now map a path, not a category. An attacker sends a well crafted email that looks like a normal customer request. The agent reads it, extracts instructions, and follows them. The email includes subtle prompt injection that tells the agent to pull additional data “for context” and then update a record in the billing system. Nothing here looks obviously malicious at a single step. It is just an email, a lookup, and an update. Chained together, it becomes data exposure and unauthorized action.

Look at susceptibility. The agent trusts inbound email. It has access to multiple systems. It can interpret and act on instructions without strong validation. It is easy to influence because it is designed to be helpful and responsive.

Then look at damage. The agent can read customer records, modify billing details, and trigger downstream workflows. That is a large blast radius. A single influenced action can expose data and change financial records across systems.

Now recovery time. Would anyone notice quickly. If the agent logs actions but no one is watching in real time, this could run for hours. If changes are not easy to trace or roll back, recovery becomes slow and painful. The longer it runs, the worse it gets.

Ask the question. When something goes wrong, how bad does it get and how long does it last. In this case, both answers are not good.

Now apply controls that change that outcome. Limit what the agent can do in one step. Do not allow it to both read sensitive data and make billing changes in the same flow. Require a second validation for financial actions. Scope its access so it can only retrieve the minimum data needed for a task. Treat email input as untrusted and constrain what instructions can be executed from it. Monitor behavior in real time and flag when the agent accesses data outside its normal pattern. Add the ability to pause or revoke its access immediately. Make billing changes reversible with clear audit trails.

Run the same path again with those controls. The email still arrives. The agent still reads it. But it cannot pull large amounts of unrelated data. It cannot directly execute a billing change without a check. If it starts behaving outside its normal pattern, it is flagged and stopped quickly. The blast radius is smaller and the recovery time is shorter.

That is the difference. You did not try to predict every possible injection or label it under a category. You followed a realistic path, assumed behavior could be influenced, and applied controls that reduce impact and time to recover.

Go to Part 5.