Security Brutalism Under Real Conditions, Part 4: Building the Program

Part 3 covered how to build a working inventory and a consequence map, and with those in hand the work turns concrete. Everything from here gets prioritized against the top of that list.

Most security programs add before they subtract. A new threat surfaces, a vendor shows up with a solution, or a compliance framework adds a requirement and the stack grows, each addition seeming justified on its own. The cumulative effect leaves an environment harder to understand, harder to operate, and harder to defend than it needs to be. Hardening starts with removal.

Take the security tool stack and apply a single test to each item. Does it reduce the susceptibility of the systems at the top of the consequence map, limit the blast radius if those systems are compromised, or reduce the time it takes to detect and recover from compromise of those systems? If no one can answer with evidence, that tool becomes a candidate for removal.

This isn't a comfortable exercise. Tools got purchased for reasons, often good ones at the time, some tied to a compliance framework, some bought under pressure after a previous incident, or some added because a vendor relationship made it easy. The question is whether a tool demonstrably improves survivability now. Every tool that can't answer that question adds complexity, consumes maintenance time, and represents potential supply chain exposure. Remember that security tooling has already served as the entry point in several high-profile compromises. The burden of proof for keeping something in the environment should run high.

Access follows the same logic. List every standing permission, long-lived credential, and access grant to consequential systems, and require a current documented business need for each. Anything that can't produce one gets revoked. Access accumulates silently through various factors like team changes or projects that end but leave their service accounts behind. A quarterly entitlement review is the mechanism that pushes back against this consistently by asking, regularly, whether the access that exists today is the access that should exist today.

After the subtractive work come structural decisions, following directly from the survivability questions. Consequential systems carry no standing access, with access granted for specific tasks, scoped to minimum necessary permissions, with a defined expiration and a full audit trail, since compromising a credential today shouldn't automatically yield persistent access granted months ago and never reviewed. Just-in-time access workflows implement this, and the reasoning behind it is that persistent access is the mechanism by which stolen credentials convert into sustained intrusion.

High-consequence actions need separation of duties. Anything that can cause irreversible damage, deleting production data, modifying access controls, deploying changes that can't roll back, and moving significant funds, should require a second human or a mandatory review step before execution. This slows attacker progression and opens a detection window, because the review requirement is exactly where detection can catch an attacker attempting an irreversible action on a stolen credential.

Blast radius by design means every consequential system gets segmented so full compromise of a neighboring system doesn't automatically yield access to it. The architecture question is what an attacker could do if everything with a trust relationship to a system were fully owned, and an answer of everything the system can do signals that segmentation needs to change. The focus is on bounding what happens once compromise occurs, since compromises will occur.

Friction on high-consequence paths, re-authentication before high-consequence actions, step-up verification before irreversible operations, adds blockers for legitimate users too, and that tradeoff is correct as long as the friction stays proportional to the consequence of the action being protected.

Seeing is Detecting

The detection standard isn't whether a logging platform is deployed or alert volumes look active, but rather whether you know when consequential systems are under attack before the attacker reaches the objective.

Most programs fall short of this in practice, carrying high alert volumes nobody reads. A lot of unneeded logging becomes a normalized habit of ignoring alerts once the signal-to-noise ratio gets too low to act on all of them. That normalization is the most dangerous condition in the detection stack, since real signals disappear into the background once teams stop reading alerts.

Detection architecture should build against the attack paths identified during the inventory phase, instrumenting the chokepoints on realistic paths toward each high-consequence system rather than chasing uniform coverage everywhere. Behavioral baselines on those systems form the foundation because normal access follows specific patterns of identity, timing, and operation, and any deviation should trigger an immediate alert rather than a weekly summary. First-time access from any identity, access at unusual hours, unusual data volume, changes to access controls or audit logging, and new process execution on critical infrastructure all deserve that immediate treatment.

Detecting lateral movement requires instrumenting the paths between systems, not just the endpoints, so an attacker pivoting from a workstation through an internal service toward a database gets caught at each stage rather than only at arrival.

Deception assets rank among the highest signal-to-noise detection investments available per unit of engineering effort. Honeytokens, canary credentials, and honeydocuments placed where legitimate users would never go, old backup directories, decommissioned service account configurations, and forgotten internal documentation, produce near-zero maintenance and near-zero false positives, and any activation deserves priority investigation regardless of other workload. Deploying them at multiple layers, for example a canary cloud credential buried in a build artifact, a fake API key in a wiki page, or a canary database credential in an unused config file, means an alert from any of them signals someone actively looking.

Alert volume discipline belongs in detection engineering, not as an operational afterthought. A team that has normalized skipping alerts because volume runs too high needs its alert rules pruned until that stops, even if that means dropping some low-confidence signals in exchange for every high-confidence signal getting immediate attention. A small set of reliable alerts that always get investigated beats a large set of noisy ones nobody reads.

What Recovery Plans Assume Versus What They Deliver

This is where most programs become a bit more... Fictional. Backup systems exist but have never been restored under realistic conditions, recovery procedures sit documented but untested under pressure, and kill switch processes exist on paper but have never been timed. These weaknesses stay invisible until an incident reveals them, and by then the cost of discovering them runs orders of magnitude higher than the cost of finding them in a test.

The survivability test for each consequential system covers four measurements, all requiring evidence rather than estimates.

Assume compromise right now, and ask what a realistic attacker could do with current access, what data they could reach, what actions they could take, and what else they could pivot to. That defines the current worst-case blast radius, and if it hasn't been answered from the inventory and hardening work already, start there.

Time to detect measures, based on actual monitoring in the current environment, how long before a human is looking at the right data and understands what's happening, including escalation time, queue depth, and the accuracy of the first response, not theoretical alert times. Time to contain measures how long it takes to revoke all access, isolate the system, and stop further damage, tested by actually revoking access and timing the process end-to-end rather than reading the documentation. If revoking a compromised service account takes hours of coordination across teams, that duration is the real blast radius window.

Time to restore asks whether you can restore from backup to a verified working state and how long that takes, tested by actually running the restoration in a test environment under realistic conditions rather than confirming the backup exists. Running this quarterly, rotating through consequential systems so each gets tested at least once a year, means what breaks in a scheduled test gets fixed before it becomes important, while what breaks during an actual incident under time pressure becomes a very different problem.

What surfaces in recovery exercises is almost never technical. It's poor process documentation, or unclear ownership when multiple teams get involved, or slow escalation because nobody holds clear authority under pressure, and communication failures because the people who know the system aren't the people authorized to act. Those weaknesses decide actual recovery time, and they only surface through realistic exercises built around real attack paths, not generic tabletop scenarios.

Chaos engineering extends this to the infrastructure level, introducing controlled failures in non-production environments and measuring actual detection and response, time to alert, time to human acknowledgment, time to accurate understanding, and time to containment. Every assumption about detection and response speed deserves validation by evidence rather than faith. Whether the SIEM will catch something or credentials can be revoked in five minutes should get demonstrated, not assumed.

Cadence and Entropy

None of this has a finish line. Security degrades the moment a system goes live, permissions accumulate as teams change and integrations get added, firewall exceptions get approved and never revisited, service accounts outlive their purpose, credentials go unreviewed, and alert coverage drifts as systems evolve. This is the natural behavior of any production environment under continuous change.

The operational response should be a predictable cadence that pushes back against that drift before it grows. Quarterly, every access grant to consequential systems gets reviewed against current business need, every integration against current use, and anything unjustified gets revoked, alongside a restoration test for at least one consequential system, timed, with weaknesses documented. Annually, a red team exercise scoped to the actual consequence map, not CVE hunting or perimeter testing, simulates attack paths against the systems that would actually damage the business, measuring real detection and response under sustained pressure. Continuously, every proposed tool, integration, or access grant gets evaluated against the three survivability questions before approval, and anything that can't justify its contribution doesn't get added. Monthly, one consequential system gets walked through, current access, current network paths, current detection coverage, asking what changed since the last review. Something always has, and the discipline lies in catching drift while it's still small.

Metrics

The metrics most commonly reported describe the existence of controls, not their effectiveness. Tool coverage percentages, vulnerability counts, and compliance scores describe inputs, but none of them answer whether the program actually works.

Time to detect, measured from an event occurring to a human understanding it, shows whether detection functions. Time to contain and time to restore, measured from actual incidents and test exercises rather than runbook estimates, show whether recovery capability is real. Blast radius per consequential system, tested rather than assumed, shows the actual scope of a compromise. Alert signal quality, the proportion of alerts representing real activity worth investigating, shows whether detection produces signal or noise.

These metrics take operational discipline to produce, and they're also the only ones that answer the question the program exists to answer: how long do we stay failed, and is that number getting smaller?

If you're interested in building a stronger security program along these lines, you can reach out at Black Arrows.