Engineering By Hsin, Head of Product 22 August 2026

Agent sandboxes on Kubernetes: gVisor breaks kubectl, not SSH

Turn on gVisor for hard kernel isolation around an agent's sandbox and kubectl port-forward stops reaching the pod. That part isn't a Containarium bug, and it isn't even ours to fix — it's an open gap in tooling most of the ecosystem builds on. We hit it independently while testing our own Kubernetes backend, went to prove our SSH gateway sidesteps it, and instead found our own gateway was broken too — for two completely unrelated reasons, neither one gVisor's fault. This is the whole chain: what breaks, what we got wrong the first time we checked, and what's actually fixed and proven now.

Where this comes from: we build an agent sandbox, so this is partly a "look what our product does" post. But every claim below traces to a specific GitHub issue with commands and output attached — including the upstream gap that isn't ours to close, and the two bugs in our own code that were. Links throughout, not just at the end.

The gap isn't ours, and it isn't new

kubernetes-sigs/agent-sandbox#158 — filed against the same CRD our Kubernetes backend (and a growing share of the K8s-native agent-runtime ecosystem) builds on — documents the exact failure against agent-sandbox's own reference Python client: kubectl port-forward does not reach a pod scheduled under a gVisor (runsc) RuntimeClass. It's still open, marked "planned to be supported later," with no committed timeline.

We shipped a RuntimeClass knob for our own agent-box pods a while back (PR #1122) — set it to runsc and a box schedules behind gVisor's userspace kernel instead of sharing the host kernel. That PR proved the pod really boots on the gVisor kernel. It didn't yet prove the box actually works under gVisor — SSH, MCP, running commands, writing files, persistent storage, network policy — as opposed to just standing up. So we tested it properly.

What actually works under gVisor, and what doesn't

We rebuilt a real kind cluster with gVisor installed as a containerd runtime handler, scheduled a box under runsc, and ran it through the same checks we'd run against a normal box. Full matrix and raw command output in issue #1489.

What works and what doesn't when a box runs under gVisor
Check	Under `runsc`
Pod actually runs on the gVisor kernel	✅ pass
SSH / dropbear handshake + forced-command pin	✅ pass
MCP `initialize` round-trip	✅ pass
`shell_exec` (fork/pipe/exec)	✅ pass
File read/write	✅ pass
PVC-backed storage permissions	✅ pass — identical to `runc`
Default-deny NetworkPolicy enforcement	✅ pass, both directions
`kubectl exec` straight into the pod	✅ pass — the surprise
`kubectl port-forward` straight to the pod	❌ fails

Everything the box is actually for — an agent reaching it over SSH, running an MCP session, executing commands, reading and writing files — works. So does kubectl exec, which we initially assumed would fail alongside port-forward — wrong, and worth saying plainly rather than quietly fixing in an old draft. exec spawns a fresh process against the pod via the CRI's exec path; it never touches gVisor's sandboxed netstack at all, so it was never exposed to the limitation in the first place. port-forward, by contrast, dials an existing listening socket from outside the sandbox — that's specifically the path gVisor's sandbox doesn't expose the same way.

What we got wrong trying to prove it

The table above was the easy part. The interesting failure showed up when we went to record the gateway actually carrying an agent's connection end to end, the way it does in production. It didn't work. Not because of gVisor — because our own gateway had two bugs, unrelated to each other and unrelated to anything above:

A label typo blocked all gateway traffic. Our Helm chart labeled the gateway pod one way; our own NetworkPolicy expected another. On any NetworkPolicy-enforcing CNI, every box was unreachable from the gateway, silently — invisible on the CNI most people test with locally, which doesn't enforce NetworkPolicy at all. (#1492)
A missing keypair made every login fail. With no upstream credential configured — the documented default — the gateway silently fell back to password auth against a box that only accepts keys. The client's key was accepted every time; the connection still failed, reported back as a generic key rejection that had nothing to do with the actual key. We root-caused this with a 15-minute reproduction using nothing but the bare gateway binary — no cluster required — once we stopped assuming the bug lived in Kubernetes. (#1496)

Both are fixed now: the label is corrected, and the daemon refuses to start with gateway routing on and no keypair configured, rather than silently producing a connection that can never authenticate. We're not leading with this story because it flatters us — it's here because "we tested it and it worked" is a weaker claim than "we tried to prove it, broke it, fixed it, and proved it again." The second one is what actually happened.

See it, don't just read it

Same box, same gVisor sandbox. kubectl port-forward refuses it; SSH through the real gateway — the actual helm install path, both bugs above fixed, no workarounds — reaches straight into a working MCP session on the exact same pod, seconds later.

Terminal recording: kubectl port-forward failing against a gVisor-scheduled agent-box pod, then a real SSH session through the sshpiper gateway succeeding on the same pod.

Live on a real kind + gVisor cluster, deployed via the documented Helm chart — not staged, not a workaround. sshpiper's own log confirms it authenticated with the configured keypair, not the broken password fallback from #1496. Full command list in #1489.

Why this wasn't luck

Much agent-sandbox tooling reaches a pod via kubectl port-forward — dialing an existing listening socket inside the pod's network namespace from the node side. That's precisely the path gVisor's sandboxed netstack doesn't expose the same way — real pod-to-pod traffic reaches the sandbox fine (it comes in through the CNI's veth into gVisor's virtual NIC), but a direct namespace-entering dial from outside does not.

Containarium's agent access was never built on that mechanism. A box is reached over SSH, routed by an in-cluster sshpiper gateway, which opens an ordinary outbound connection to the box's real pod IP — the same real pod-to-pod path that works fine under gVisor. We didn't design around this specific gap; we designed the access model around SSH for unrelated reasons (a portable, credential-scoped, non-kube-apiserver foothold — see the design doc), and it turned out to sidestep this one too — once our own two bugs stopped getting in the way of proving it.

Try it, and the honest caveats

Turning on hard isolation is one Helm value: runtimeClass: runsc, on a node pool with gVisor installed and a matching RuntimeClass in the cluster. Nothing changes for the agent — same SSH command, same MCP session. docs/KIND-QUICKSTART.md walks the whole thing on a local kind cluster — including generating the gateway's two keypairs, now a required, documented step rather than an easy-to-skip default (that's the #1496 fix from above, made permanent in the quickstart itself).

Three things we're not going to round up:

Don't reach a gVisor box by port-forwarding straight to its pod — it won't work. Go through the gateway (a NodePort/LoadBalancer, or a port-forward to svc/sshpiper itself, which isn't gVisor-scheduled) the same way production traffic does — which we've now actually shown working, not just asserted.
Kata Containers is unevaluated here. gVisor covers the "userspace kernel" isolation tier; a VM-per-pod boundary is a separate, undone design conversation.
If upstream ships a fix for #158, this pitch gets smaller, not wrong. The SSH/gateway path is the production access model regardless of whether kubectl port-forward also happens to start working — a future fix removes one debugging caveat, it doesn't change the architecture.