Victor
All notes

Kubernetes / Networking

Why HTTP Requests Were Hanging in Kubernetes

A strange networking problem eventually led me down the MTU rabbit hole.

Victor Oloan
KubernetesNetworkingMTUAKSKata ContainersDebuggingInfrastructure

Quick summary: some HTTP requests inside a Kubernetes environment were hanging until they timed out, while everything else (pods, health checks, DNS, TCP) looked completely normal. The cause turned out to be a mismatch between the pod's configured MTU and the actual MTU the network path could support, introduced by an extra layer of encapsulation. This is the investigation that got me there.

The problem

Everything looked healthy.

The pods were Running.

Kubernetes wasn't reporting anything obviously wrong. No CrashLoopBackOff, no failing readiness probes, nothing that stood out in the usual places.

Small HTTP requests worked.

Then some requests started hanging. Not failing with an error. Hanging. The client would sit there until it eventually timed out, and there was nothing useful in the application logs to explain why. No stack trace, no exception, no obvious 5xx. Just silence until the timeout kicked in.

Pod is Running
      ↓
TCP connection works
      ↓
Small HTTP request works
      ↓
Larger HTTP request hangs
      ↓
Timeout

That progression is basically the whole story in miniature. Everything up to a certain point worked exactly as expected. The failure only showed up at the very last step, and only for a subset of requests. That combination is what made this one take a while to pin down.

The environment

The workload was running in a Kubernetes environment on AKS, communicating over plain HTTP between services. Some workloads in this environment ran with an additional layer of isolation, the kind of setup where each pod gets its own lightweight VM boundary instead of sharing the host kernel directly with other pods. Kata Containers is one implementation of that pattern, and it turns out to be relevant here, though not in the way I first expected. I won't walk through the full cluster architecture. Most of it isn't relevant to this story, and the parts that are will come up naturally as the investigation gets there.

The first clues

The behavior, once I actually started paying attention, looked roughly like this:

That last point is what made this look like an application problem at first. When something hangs instead of failing loudly, the instinct is to go look at the code that's doing the hanging.

What I initially suspected

None of these were unreasonable places to start, and this is roughly the order I checked them in.

Application problem

Maybe a specific code path was slow: a synchronous call somewhere, a lock, something blocking on a larger payload. Worth checking first, since it's usually the cheapest thing to rule out.

DNS

Maybe resolution was intermittently slow or failing, and the request was stuck waiting on a lookup rather than the request itself.

Service / Ingress

Maybe Kubernetes networking was misrouting a subset of traffic. Stale endpoints, an unhealthy pod still receiving traffic, that kind of thing.

Database or downstream dependency

Maybe the request was actually waiting on a backend call that itself was slow, and the hang had nothing to do with the network path in front of it.

Network connectivity

Maybe there was plain packet loss somewhere between the pods, and this was just a garden-variety flaky-network problem.

All reasonable guesses. All wrong, or at least not the actual cause. I want to be honest about that, because the MTU explanation looks obvious in hindsight, and it very much was not obvious at the time.

The investigation

The application-layer theories fell apart first. The endpoints that hung weren't consistently slow. The same endpoint would work fine on one call and hang on the next, which doesn't match a code path that's just slow. DNS resolution timing looked normal when I measured it directly. And the database wasn't even seeing the query in some of the hanging cases, the request never made it that far.

That pushed the investigation down the stack, roughly following this path:

Application
    ↓
HTTP request
    ↓
TCP connection
    ↓
Kubernetes networking
    ↓
Network interface
    ↓
MTU

TCP connections were establishing fine. The handshake completed, the socket was open, so this wasn't a routing or firewall problem in the way I'd normally think about one. Whatever was happening was happening after the connection was already established, which narrowed things down quite a bit.

The MTU clue

The detail that actually mattered was one I'd been staring at for a while without registering its significance: request size.

Small request
     ↓
Works

Large request
     ↓
Fails / hangs

The clue

Small requests worked. Larger requests didn't. That was the point where the investigation stopped looking like an application problem and started looking like a network problem.

Once I could reliably reproduce the failure just by increasing payload size, the question stopped being "why does this endpoint hang" and became "what happens to a packet somewhere between here and there once it gets big enough." That's a much more answerable question.

What MTU actually means

MTU (Maximum Transmission Unit) is the largest packet size that can be transmitted over a network interface without fragmentation. Every hop in a network path has one, and in theory they should all agree, or at least the lowest one along the path should win.

Here's the part that trips people up: MTU is a network-layer concept. TCP works out its own Maximum Segment Size (MSS) based on the MTU it's aware of, but that's typically the MTU of the local interface, not necessarily the true MTU of the entire path the packet will actually travel. That gap between what the interface reports and what the path can actually carry is exactly what caused the problem here.

Every additional layer a packet passes through (an overlay network, a VPN, a tunnel, a virtualization boundary) usually adds its own headers. Those headers take up space inside the same outer packet, which means the payload budget gets smaller even though nothing told the sender that.

Original MTU
      -
Encapsulation overhead
      =
Effective MTU
┌───────────────────────────────────────┐
│ Original packet                        │
│                                         │
│ 1500 bytes                             │
└───────────────────────────────────────┘

              ↓ encapsulation

┌───────────────────────────────────────┐
│ Outer headers │ Inner packet           │
└───────────────────────────────────────┘
       ↑
   extra overhead

Effective payload size becomes smaller.

I'm not going to put an exact overhead number here, because it depends entirely on which encapsulation is involved and how many layers are stacked. The number that mattered in this environment isn't necessarily the number that would matter in yours. The concept is what transfers, not the specific byte count.

Why encapsulation matters in Kubernetes

Kubernetes networking can involve more layers than it looks like from the outside. Depending on the CNI and the environment, traffic from a pod can pass through something like this before it reaches another node:

Pod
 ↓
veth
 ↓
CNI
 ↓
overlay / encapsulation
 ↓
node network
 ↓
cloud network

If any layer in that chain adds encapsulation, an overlay network wrapping packets to route them between nodes, for example, the effective MTU for that path can end up lower than what any single interface reports.

Important

An interface reporting MTU 1500 doesn't guarantee that 1500-byte packets can successfully traverse the entire network path. It only tells you what that one interface will accept.

That's the core lesson of this whole investigation, and honestly the thing I'd underestimated going in.

Kata Containers and the extra layer

I mentioned earlier that some workloads in this environment ran with an additional VM-based isolation boundary. I want to be careful about how I describe this part, because it's easy to misread it as "Kata is broken." That's not what this was.

Running a pod inside its own lightweight VM boundary, which is the pattern Kata Containers implements, adds another hop into the packet path between the application and the physical or cloud network. Conceptually:

Application
   ↓
Container
   ↓
Kata / VM boundary
   ↓
Virtual network
   ↓
Host networking
   ↓
Cloud networking

Every one of those boundaries is a place where headers can get added and an MTU mismatch can get introduced. None of that is a defect in the technology. It's just a consequence of having more layers between the application and the wire. In this environment, that extra layer meant the pod's idea of its own MTU didn't match what the rest of the path could actually carry.

How I confirmed it

Theory is cheap. I wanted to actually see the boundary, not just infer it. The tools here are ordinary:

bash
ip link
ip addr
ip route

These tell you what the interface believes about itself: its configured MTU, its addresses, its routes. They don't tell you what the network between here and there will actually allow through, which is really the whole point of this story.

For connectivity, ping is the obvious first tool, but it's not enough on its own. A successful ping doesn't prove that HTTP traffic at a larger size will succeed. What I actually needed was a way to test specific packet sizes with fragmentation disabled:

bash
ping -M do -s <size> <destination>

-M do sets the "do not fragment" flag, and -s controls the payload size. By gradually increasing <size>, you can find the point where packets stop getting through. That boundary is a strong signal for where the effective path MTU actually sits. I won't give a single number as "the" correct value here, because it really does depend on the environment. What matters is the methodology, not a magic constant.

tcpdump filled in the rest of the picture: retransmissions of the same oversized segment, over and over, with no corresponding ICMP response coming back to explain why.

Why it looked like hanging instead of an error

This part is the actual mechanism, and it's worth getting precise about, because "MTU problem" alone doesn't explain why the symptom was a hang rather than a clean failure.

Normally, when a packet is too large for a hop along the path, a router is supposed to send back an ICMP message: "Fragmentation Needed," effectively saying "this didn't fit, and you asked me not to fragment it, so try again smaller." This is Path MTU Discovery, and when it works, TCP quietly adjusts and the connection carries on without the application ever noticing.

The failure mode that actually hurts is when that ICMP message never makes it back to the sender, often because something along the path is filtering ICMP entirely, which is a common default in cloud networking and security groups. When that happens, the oversized packet gets silently dropped, the sender never learns it needs to shrink its segments, and it just keeps retransmitting the same packet at the same size. This is usually called a PMTU black hole, and it produces exactly the symptom I was seeing: no error, no explanation, just a connection that goes quiet until something upstream (the application, the load balancer, the client) eventually gives up and times out.

HTTP request
      ↓
TCP segment
      ↓
Packet too large
      ↓
Fragmentation / drop
      ↓
Retransmission
      ↓
Application waits
      ↓
Timeout

The exact behavior here depends heavily on the network path and configuration involved. Whether ICMP is blocked, where in the path it's blocked, how the OS-level TCP stack handles the retransmissions, all of that varies. I'm describing the general mechanism, not claiming this is the only way an MTU mismatch can show up.

Why this was difficult to diagnose

The frustrating part of this whole investigation is that nothing was actually lying to me. Every individual signal I checked was accurate. It just wasn't telling the whole story on its own.

Kubernetes:
✓ Pod Running

Application:
✓ Process Running

TCP:
✓ Connection established

Small requests:
✓ Working

Large requests:
✗ Hanging

Nothing looked completely broken. Only a specific class of traffic exposed the problem, which meant it survived normal application monitoring without tripping anything. Health checks are usually small. Most day-to-day traffic in this environment was small too. The failure only showed up for the subset of requests large enough to cross a boundary that nobody had actually verified end to end.

The fix

The fix, once the cause was clear, was to make the pod's effective MTU match what the network path could actually support. In practice, that meant lowering the pod's network interface MTU to a safe value.

bash
ip link set eth0 mtu <safe-mtu>

In Kubernetes, the practical way to apply that is with an init container that runs before the application starts, adjusts the interface, and exits:

yaml
initContainers:
  - name: fix-mtu
    image: <minimal-image-with-iproute2>
    securityContext:
      capabilities:
        add:
          - NET_ADMIN
    command:
      - sh
      - -c
      - ip link set eth0 mtu <safe-mtu>

The specific number isn't really the interesting part, and I'm not going to present one as universally correct. It depends on how much encapsulation overhead exists in your particular path. The important part is the principle: make the pod's configured MTU match what the path can actually carry, instead of trusting the interface's default.

Security considerations

Modifying a network interface from inside a container requires elevated privileges. It's tempting to just reach for privileged: true and move on, but that grants far more than this actually needs. The specific capability required here is NET_ADMIN, and scoping the init container to exactly that, rather than full privileged access, keeps the blast radius of that container as small as it can reasonably be.

yaml
securityContext:
  capabilities:
    add:
      - NET_ADMIN

What's actually required and permitted here depends on the Kubernetes runtime and the environment's policies. This is the principle to apply, not a universal recipe to copy without checking it against your own constraints.

How I verified the fix

Verification followed roughly the same shape as the original reproduction, just checking that the failure mode was gone:

Before

Small HTTP request     ✓
Large HTTP request     ✗

After

Small HTTP request     ✓
Large HTTP request     ✓

Beyond that basic before and after, I also re-checked application behavior under normal load, watched packet captures for the retransmission pattern that had been there before, and confirmed pod networking stayed stable over time rather than just working once. I'm not going to attach specific numbers to any of that. The point of the verification was confirming the failure mode was actually gone, not producing a metric for a report.

What I learned

1. "Running" doesn't mean "working."

A Kubernetes pod can be perfectly healthy by every signal Kubernetes checks, while a specific network path underneath it is quietly broken.

2. Test different packet sizes.

If small requests work and larger requests fail, think about MTU before you think about anything more exotic.

3. Look beyond Kubernetes.

Sometimes the Kubernetes configuration itself is completely fine, and the problem lives in the network path underneath it, a place Kubernetes doesn't really have visibility into.

4. Encapsulation has a cost.

Every additional network layer (overlays, tunnels, VM boundaries) can eat into the MTU budget. That cost stays invisible until something forces a packet close to the limit.

5. Network problems often look like application problems.

Timeouts and hanging requests don't automatically mean the application is broken. Sometimes the application is doing exactly what it's supposed to do, and it's just waiting on a network that isn't going to answer.

Debugging checklist

When HTTP requests hang in Kubernetes, this is roughly the order I'd work through now:

Conclusion

The interesting part of infrastructure work is that the failure isn't always where the symptom shows up.

In this case, an HTTP request that looked like an application problem eventually led all the way down to packet size and network encapsulation.

That's probably one of the better reminders that when something behaves strangely, it helps to just keep moving down the stack.

More engineering notes

Back to all notes