Security / Kubernetes

It Worked Yesterday. Kubernetes Rejected It Today.

How a namespace security policy change turned a previously working Kubernetes workload into a rejected deployment.

The problem

The deployment had been working.

Then Kubernetes started rejecting it.

Nothing had obviously changed on the application side. It was the same container image that had been running fine for a while. The workload hadn't quietly decided to become insecure overnight.

So the question was pretty simple. What changed?

The first question: what changed?

Whenever something that worked yesterday stops working today, and you haven't touched it, the useful question usually isn't "what's wrong with this thing." It's "what changed around it." A few obvious candidates come to mind first:

• Did the application change?
• Did the container image change?
• Did the deployment manifest change?
• Did the namespace change?
• Did the cluster configuration change?
• Did someone change a security policy?

I went through most of that list before landing on the actual answer, which turned out to be the last one.

Admission rejection is not a crash

Before getting into what actually happened, it's worth being clear about the shape of the failure, because it changes where you should even be looking.

A runtime failure looks like this. The Pod gets created, and then something inside it goes wrong:

Deployment
    ↓
Pod
    ↓
Container
    ↓
Application
    ↓
Crash

This wasn't that. What I was looking at was an admission rejection: the workload never got far enough to run in the first place.

Deployment
    ↓
Kubernetes API
    ↓
Admission
    ↓
Pod Security Admission
    ↓
REJECTED

That distinction matters more than it sounds like it should. It took me a moment to stop looking at the application and start looking at the platform.

Where I looked first

The application itself didn't seem like a productive place to start, since it wasn't getting far enough to run at all. I checked the container image next, mostly out of habit, but it was the same image that had already been deployed successfully before. Resource limits crossed my mind too, but the rejection didn't look like a CPU or memory problem at all, it looked like something was actively refusing the deployment rather than the deployment struggling to run.

Kubernetes events ended up being the more useful thread to pull, since whatever was happening was clearly happening at the platform or admission layer rather than inside the container. That's roughly when the namespace itself started looking suspicious. Once I actually read the rejection message instead of skimming past it, it was pointing at security context requirements, not anything about images, resources, or the deployment spec itself. That should probably have been the first thing I read carefully, not the third or fourth.

Asking around

At some point I just asked a colleague whether anyone had recently changed anything about the namespace. Sometimes that's faster than reconstructing the whole timeline yourself.

The answer was yes. The namespace had been changed to enforce the restricted Pod Security Standard.

I want to be clear about something here, because it's easy to read a story like this as "someone broke production." That's not really the point, and it's not fair to whoever made the change either. A security policy change had a legitimate effect on a workload that hadn't been migrated to satisfy that policy yet. That's a rollout problem, not a mistake by one person.

The namespace policy

The namespace had labels roughly equivalent to this:

metadata:
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest

pod-security.kubernetes.io/enforce: restricted tells Pod Security Admission to enforce the restricted Pod Security Standard for everything in that namespace. A workload that violates the policy can be rejected outright.

pod-security.kubernetes.io/enforce-version: latest controls which version of the Pod Security Standards is used to evaluate that policy. It's worth being precise here: latest doesn't mean the policy silently changes itself every time Kubernetes changes. It refers to the latest Pod Security Standards version supported by the cluster's Kubernetes release. The actual trigger in this incident was the enforce label being set to restricted on the namespace, not some background version drift.

You can check a namespace's labels directly:

kubectl get namespace <namespace> --show-labels

That's the command that would have shown this from the start, if I'd thought to run it earlier.

What Pod Security Admission actually is

Pod Security Admission is a built-in Kubernetes admission controller. It can enforce Pod Security Standards on workloads as they come in.

Pod request
     ↓
Kubernetes API
     ↓
Pod Security Admission
     ↓
Evaluate security requirements
     ↓
Allow / Warn / Audit / Reject

It works through three modes, and they don't all do the same thing:

enforce

Violations can actually get the workload rejected. This is the mode that bit me.

warn

The user gets a warning back, but the workload is still admitted. Useful for finding problems before they become blocking ones.

audit

Violations get recorded for auditing, but they don't reject anything on their own. Good for quietly building a picture of what's out of compliance.

That distinction between the three modes turns out to matter a lot for how you roll a stricter policy out safely, more on that later.

What restricted actually means

Pod Security Standards define three profiles, and restricted is the strictest one:

Restricted is intentionally the most security-conscious profile. That doesn't make it the correct default for every namespace everywhere. What's appropriate depends on the workload and the environment it runs in.

Broadly, restricted brings requirements around things like privilege escalation, Linux capabilities, running as a non-root user, seccomp, and a handful of other security-sensitive settings. The exact list depends on the Pod Security Standards version in use. I'm not going to reproduce the whole specification here, mostly because the part that actually mattered for this incident was one specific piece of it.

Where seccomp comes in

seccomp stands for secure computing mode. At a high level, it restricts which system calls a process is allowed to make to the Linux kernel.

Application
     ↓
System call
     ↓
Linux kernel

A seccomp profile decides whether a given call gets through:

Application
     ↓
Allowed system calls
     ↓
Kernel

Application
     ↓
Restricted system calls
     ↓
Blocked / filtered

You don't need to know Linux kernel internals to get the point. A container that only needs a normal, boring set of system calls doesn't need access to the whole kernel surface, and cutting that surface down is one of the cheaper ways to reduce what an attacker can do if they end up inside a container.

The common seccomp profile types in Kubernetes

RuntimeDefault uses the container runtime's own default seccomp profile. For most workloads that don't need anything unusual, this is the reasonable starting point.

Localhost points at a custom profile that has to already exist on the node. It works, but it adds an operational dependency: the profile has to be there wherever the workload gets scheduled.

Unconfined means no seccomp filtering at all. It's the least restrictive option, and it's exactly the kind of thing a policy like restricted is designed to catch.

The fix

In this case, the workload needed to explicitly satisfy the security requirements the namespace was now enforcing. That meant setting an actual seccomp profile instead of relying on whatever the default had been before:

spec:
  securityContext:
    seccompProfile:
      type: RuntimeDefault

Exactly where that goes, and whatever else needs adjusting alongside it, depends on the shape of the workload. I'm not going to pretend this one block is a universal fix for every seccomp rejection. It's what closed the gap for this particular workload against this particular policy.

Restricted usually expects more than just a seccomp profile though. Running as non-root, dropping Linux capabilities down to nothing extra, and disallowing privilege escalation tend to show up alongside it. Whether a given workload actually needs to touch any of those depends entirely on what it was configured with before, so I'm deliberately not listing exact values here. The point isn't the specific keys, it's that restricted checks more than one thing, and seccomp just happened to be the piece that mattered for this workload.

Why not just turn the policy off

The tempting shortcut here is obvious. Remove the restricted label, or set the workload to Unconfined and move on with your day. I'd push back on that as a first response.

The better path looks like this:

Policy rejects workload
        ↓
Understand the violation
        ↓
Determine what the application actually needs
        ↓
Remove unnecessary privileges
        ↓
Configure secure defaults
        ↓
Deploy again

Not this:

Policy rejects workload
        ↓
Disable policy
        ↓
Deploy

The real issue in a story like this one is rarely the policy itself. It's introducing a stricter policy without first checking which workloads in the namespace depend on the old, looser one.

How to roll this out without breaking things

If a namespace already has workloads that don't satisfy restricted, flipping straight to enforce is basically asking for this exact incident. A safer path usually looks something like this:

Existing workloads
       ↓
Evaluate compliance
       ↓
warn / audit
       ↓
Fix workloads
       ↓
Verify
       ↓
Enable enforce

Using warn and audit first gives you visibility into which workloads would fail before anything actually starts failing. It's not mandatory everywhere, and some environments will reasonably decide the risk of skipping straight to enforce is worth it. But it's a much gentler way to find out what's going to break.

The commands I actually used

Namespace labels, to see what policy is actually configured:

kubectl get namespace <namespace> --show-labels

A closer look at the namespace itself:

kubectl describe namespace <namespace>

The deployment, to confirm nothing on the workload side had actually drifted:

kubectl get deployment <deployment> -o yaml

Pods in the namespace, to see what state things were actually in:

kubectl get pods -n <namespace>

A specific pod, for more detail:

kubectl describe pod <pod> -n <namespace>

And events, sorted by time, which is what actually pointed at admission being the layer where this was happening:

kubectl get events -n <namespace> --sort-by=.lastTimestamp

The real root cause

Before

Namespace
    │
    └── Existing security policy
             │
             ▼
        Workload accepted


After

Namespace
    │
    └── enforce=restricted
             │
             ▼
    Pod Security Admission
             │
             ▼
     Workload evaluated
             │
             ▼
      Security violation
             │
             ▼
          REJECTED

The workload didn't suddenly become broken.

The namespace's security requirements became stricter.

That's really the whole incident.

What "it worked yesterday" actually means

It's worth being careful with the title, because it's easy to misread. "It worked yesterday" doesn't mean Kubernetes changed overnight on its own. It means the platform contract around the workload changed while the workload itself stayed exactly the same.

That contract can shift in a bunch of ways:

• Namespace policy
• Cluster configuration
• Admission configuration
• Kubernetes version
• Container runtime
• The workload manifest itself

In this specific case, the known trigger was the namespace's Pod Security policy being changed to enforce restricted. Nothing more exotic than that.

Infrastructure is a contract

This is the bigger lesson underneath the specific incident. A Kubernetes manifest doesn't get evaluated in isolation. Its behavior depends on everything around it:

Workload
   +
Namespace
   +
Admission policies
   +
Cluster configuration
   +
Runtime
   +
Cloud platform

A deployment that works fine in one environment isn't guaranteed to work in another one with a different security posture, even if the manifest is byte for byte identical. The manifest is only half the story. The namespace and the admission policy are the other half, and they can change without the manifest changing at all.

Security versus developer experience

Security teams reasonably want restricted, least privilege, non-root, seccomp, dropped capabilities, all of it. Developers mostly just want kubectl apply to work. Neither side is wrong.

The platform team's actual job sits in between: make the secure path understandable and repeatable, so following it isn't harder than not following it. A platform that gets this right makes the secure configuration the easy configuration, not an obstacle course you have to know exists before you hit it.

This incident is a small example of that balance going slightly wrong in the ordinary way it usually goes wrong: not through bad intentions, but through a policy change landing without a way for the people running workloads in that namespace to see it coming.

What I learned

When something that worked yesterday stops working today, check what changed around it before you assume something inside it broke.

An admission rejection is not a runtime bug, and it's worth treating it differently from one.

Namespace security policy is part of what a workload has to satisfy, just as much as the manifest itself is.

Loosening a policy to make an error go away just moves the problem somewhere less visible.

And if you're the one introducing a stricter policy, warn and audit are there so you don't find out what breaks by breaking it.

Debugging checklist

When a previously working Kubernetes workload is suddenly rejected:

• ☐Read the admission error carefully.
• ☐Determine whether the Pod was even created.
• ☐Check namespace labels.
• ☐Check the Pod Security Admission configuration.
• ☐Check enforce / warn / audit labels.
• ☐Check enforce-version.
• ☐Check the pod's securityContext.
• ☐Check the seccompProfile.
• ☐Check container-level securityContext.
• ☐Check recent namespace changes.
• ☐Check recent cluster changes.
• ☐Check recent Kubernetes upgrades.
• ☐Check admission policy changes.
• ☐Check whether the workload actually needs the capability being denied.
• ☐Fix the workload rather than immediately weakening the policy.
• ☐Verify the workload after the fix.

Conclusion

The workload didn't suddenly become broken. The namespace policy became stricter, and the workload no longer satisfied it.

Once you see it laid out like that, it's a pretty unremarkable incident. Getting there just took asking the right person the right question.

Next time something Kubernetes-shaped suddenly rejects a workload that was fine the day before, I know where I'm looking first: not at the workload, but at whatever changed around it.