What I look for first when I open a production system

When I open a production system for the first time, I have five days. Not five days to fix it. Five days to map it well enough to say, with evidence, where it fails first and what that failure costs.

Five days against a codebase that took years to grow. I cannot read every line, and I am not supposed to. I rank the questions by the cost of being wrong, then spend the week in that order.

The order I look in is not a checklist. It is a ranking by blast radius.

None of the checks is secret. You can find all five in a decent audit checklist. What the checklist cannot carry is the order: what to look at first, what to leave unread, and why.

I did not design that order on a whiteboard. It came from finding the widest blast radius in the same four surfaces: exposure, access, recovery, and delivery. The code tells me more once those are mapped.

On day one, I do not yet share the team's assumptions. That advantage expires quickly. By day four, the strange choices have explanations, and I have started seeing the system through the team's eyes. So day one goes to the things that can take the whole system down. The things that merely annoy a developer can wait until day five, or forever.

Here is the order, and why each thing sits where it does.

First, what a stranger can reach

Not the code. The exposure.

The first question is what the internet can touch, because that is the only surface where a stranger gets a vote. Before I reason about what an attacker could do inside, I need to know how hard it is to get in at all.

What I find there is rarely exotic. A security group opened to the world during a debugging session and never closed. SSH on port 22, reachable from anywhere, because it always has been. A storage bucket that is public because public was the fastest way to ship. An admin panel on a path someone assumed nobody would guess. A database listening on a public address because a bastion felt like one step too many that week. DNS records pointing at machines nobody remembers owning.

None of it is broken. All of it works, which is exactly why it is still there.

A single rule here can expose the entire system. So it goes first.

Second, who can act, and who decided that

Inside the perimeter, the question changes from "can you get in" to "what can you do once you are in."

Identity in a production system is rarely designed. It accumulates: a key minted years ago for a one-off script, a role widened during an incident and never narrowed, the same credential living in three .env files, an admin scope granted just for now. Every grant was reasonable on the day it was made. But nobody has read the sum, because nobody wrote the sum. It happened one sensible decision at a time, and the decisions are still in force long after their reasons are gone.

So I read the current permission map end to end: who can reach production, who can read the data, who can change the rules. The dangerous answers are almost never malicious. They are just old.

Third, whether the data survives being needed

Everyone I audit has backups. What I ask for is the restore: run on purpose, into a clean environment, against a clock, recently enough to mean anything. A backup that has never been restored proves the job runs, and nothing else.

I cannot wait for an outage to settle the question, so I ask for the closest available evidence: a recent restore drill. Often it does not exist. That is what turns the room quiet. Nothing has broken. Nobody can show that recovery works.

Fourth, how code becomes production

The deploy path changes production on every release. That alone makes it part of the reliability model. The pipeline also records how the team actually ships, not how the runbook says it ships.

Can you roll back in one step, or is rollback a manual scramble nobody has rehearsed. Is there a path to production that skips review, and who uses it. Does the artifact that was tested ship, or is it rebuilt on the way out into something slightly different. Which user does the pipeline run as, and with how much of the permission map from step two. Does configuration have one source of truth, or does it live in two places that are allowed to disagree, where the wrong copy wins on release day.

Somewhere in the deploy path, there is one environment variable nobody can explain and everybody is afraid of. There always is.

Last, the code itself

Only now do I read code, and through one lens: what here works only because nothing has tested it.

Not coverage numbers. Places. The path that only runs at month end. The queue that has never been full. The retry that has never actually retried. The error handler that has never seen the error it handles. Untested code that happens to work is a demo that has not failed yet.

The code goes last because it has by far the largest surface and no useful priority order of its own. It would eat all five days if I let it. After the first four steps, I know which paths can expose data, block recovery, or take production down, and I read those.

What five days produce

Not a list of bugs. A map.

Where the system fails first, ranked by what being wrong would cost, with the recovery paths that do not exist marked as clearly as the ones that do. It is written for the person who owns the system, so they can make three decisions with evidence instead of folklore: what to fix now, what to fund next, and what to knowingly accept.

The findings are the easy part. The order is the work.

Most teams already know at least one of these is true about their system. The audit is not the discovery. It is the moment it stops being optional to know.