Engineering By Hsin, Head of Product 26 August 2026

Agent sandbox density: we lost by 2x, then found out why

We ran a real, live density benchmark: how many isolated agent sandboxes fit on one host, Kubernetes + gVisor + upstream agent-sandbox versus the identical setup routed through Containarium. The first result was 186 for us against 373 for agent-sandbox — half. We didn't round that up or bury it. We root-caused it to a real gap in our own CLI, shipped a one-flag fix, and re-ran the exact same comparison. 373 vs. 373. Exact match.

Where this comes from: we build an agent sandbox, so this is partly a "look what our product does" post. But every number below traces to a live run with raw output attached — including the run where we lost, the issue that explains why, and the fix that closed it. Links throughout, not just at the end.

Round one: 186 vs. 373, and why

Same host, same hard resource cap (48GiB RAM / 20 vCPU), same sandbox profile family, both under Kubernetes with the identical runsc (gVisor) RuntimeClass. The first time we ran this, Containarium's boxes only reached 186 against agent-sandbox's 373 — roughly half.

The cause traced to how each side declares its sandbox's memory:

agent-sandbox's pods request 128Mi, with a 256Mi limit — standard Burstable QoS sizing.
Containarium's create had no way to set those two numbers separately — --memory 256Mi set both request and limit to 256Mi, exactly 2× agent-sandbox's request.

Kubernetes' scheduler reserves the full declared request against node capacity the instant a pod is scheduled, whether the pod ever touches that memory or not. At the stopping point, node memory requests hit 47984Mi/48GiB (99%) for agent-sandbox and 47856Mi/48GiB (99%) for Containarium — both sides essentially saturated the same total memory budget, just divided into differently-sized chunks: 373 × 128Mi ≈ 46.6GiB, 186 × 256Mi ≈ 46.5GiB. Halving the request size, not halving anything else, produced almost exactly half the count. We re-ran it independently on a fresh cluster to rule out a fluke — 186 ready, 189 attempted, an exact match down to the node memory snapshot. It was a real, solid, reproducible number. It just wasn't the whole story yet.

The fix: a missing CLI flag, not an architecture problem

ResourceLimits.memory was always applied as both request and limit — there was no field, no flag, no way to ask for a smaller request under a bigger limit, the way Burstable QoS pods (agent-sandbox's included) are normally sized. Kubernetes itself supports request ≠ limit fine; our own create API had simply never exposed the second number.

We added it — issue #1557, shipped as PR #1560: containarium create gained --memory-request/--cpu-request, separate from --memory/--cpu (which stay the limit). Empty preserves the old behavior exactly — additive, not a breaking change.

Round two: 373 vs. 373, exact match

With the fix landed, we re-ran the comparison on a fresh cluster — Containarium's boxes now declared at the same profile as agent-sandbox's pods (request 128Mi / limit 256Mi memory, 25m/50m CPU) — and, since only the runtime class was the remaining variable, we also took the opportunity to isolate gVisor's own cost cleanly: the identical matched profile, run once under gVisor (runsc) and once under plain runc, on the same cluster.

Density benchmark results, gVisor vs. plain runc, matched resource profile
	gVisor (runsc)	plain runc
Sandboxes reached RUNNING	373	373
Attempted (incl. failures)	376	376
Node memory requests at stop	47984Mi/48GiB (99%)	47984Mi/48GiB (99%)
Wall-clock for the density loop	~77 min	~40 min

Exact match, identical node memory snapshot down to the Mi, and it lands within a rounding error of agent-sandbox's own 373. With the request/limit split available, Containarium's k8s+gVisor path doesn't just close the gap with agent-sandbox — it matches it. gVisor's own per-pod density cost is not measurable at this sandbox size against a 48GiB host: the entire 186-vs-373 gap really was the declared-size asymmetry. Worth naming honestly: agent-sandbox's pods still run a minimal busybox image against Containarium's real, heavier agent-box runtime in this run too — that asymmetry never went away, and the counts matched exactly anyway, because density here is governed by the declared memory request, not image size or pull time.

The one real, measurable difference between the two legs wasn't density — it was time. ~77 minutes under gVisor vs. ~40 minutes under plain runc to reach the same 373, roughly 2×. gVisor's per-sandbox startup overhead is real; it shows up in how fast you get to a given density, not in how dense you can get.

If the pods are identical, what is Containarium actually doing?

Fair question once you've seen 373 vs. 373 twice. Past the entry point, a bare kubectl apply against agent-sandbox's own Sandbox CR and a containarium create call build the exact same pod, through the exact same upstream controller, inside the exact same gVisor boundary — that convergence is why the density numbers matched once the memory request did. The difference is everything Containarium's daemon does besides writing that CR.

Right of the Sandbox CR, the two paths are identical — same controller, same pod spec, same gVisor boundary, which is why 373 vs. 373 made sense once the memory request matched. Left of it, only Containarium's path also builds the namespace, network policy, and SSH key material every tenant box needs.

This is where Containarium is actually doing work: the four green boxes. A bare kubectl apply against the upstream Sandbox CR gets you an isolated gVisor pod and nothing else — no per-tenant namespace boundary, no default-deny network policy, no SSH key material provisioned for you. containarium create builds all of that alongside the identical CR, every time, so a box is reachable and isolated from other tenants the moment it exists, not after you've hand-wired the surrounding Kubernetes objects yourself.

One thing this specific benchmark run did not exercise: Containarium's sshpiper SSH gateway, which is what actually lets a human or agent connect through those provisioned secrets. It was deliberately left disabled for this run (see the README's "gVisor access path" section) — the density loop only needed to confirm each box reached RUNNING, not to actually SSH into 373 of them. The secrets and Service above are still provisioned regardless; only the gateway that consumes them sat idle for this run.

A different deployment mode: 929, no Kubernetes at all

Containarium also ships a native LXC/Incus backend: no Kubernetes, no gVisor, sandboxes run directly on the host. Run the same benchmark there and you get 929 — but this isn't a gVisor-vs-gVisor result, because there's no gVisor and no Kubernetes admission control in this path at all. Kubernetes reserves what's declared, always. Incus's limits.memory, with no Kubernetes scheduler in front of it, is a cgroup ceiling, not a reservation: through most of this run, boxes declared at 256Mi were only actually using something like 90–100MiB — and only that much counted against the host. No admission backpressure means nothing tells the platform "stop" until the host actually runs out. When we pushed far enough, that's exactly what happened: the host ran out of real RAM at ~930 sandboxes, full stop.

Zoom out from a single sandbox to the daemon that provisions it, and the two k8s-based paths above converge somewhere the native LXC path never goes at all. Containarium's k8s deployment mode is conceptually a Service in front of a Deployment running the containarium-sentinel (traffic forwarding) in front of a StatefulSet running the containarium-daemon itself, the one that actually provisions boxes — three Kubernetes-native scheduling and networking layers before a single sandbox exists. The native LXC backend is one process on the host talking directly to Incus. That's fixed, per-deployment overhead, not a per-sandbox cost — but it's real infrastructure the LXC path simply never carries.

Approach 2 provisions every box through three k8s-native layers before a sandbox exists at all — a Service, a sentinel Deployment doing traffic forwarding, and a StatefulSet running the daemon that actually provisions boxes. Approach 3's daemon is a single process on the host talking directly to Incus. Fixed, per-deployment overhead, paid once rather than per sandbox — but real k8s infrastructure approach 3 simply doesn't carry.

This is the general shape of the architecture, not the literal chart benchmarked in this run. The Helm chart actually deployed for this benchmark (charts/containarium-k8s/) runs the daemon as a single Deployment with sshpiper as its gateway — no separate sentinel component and no StatefulSet. The sentinel/StatefulSet split above is how Containarium's k8s deployment is conceptually intended to work; if it changes the fixed-overhead comparison materially, that's worth a follow-up benchmark note, not an assumption.

Getting to 929 took two real fixes, found live: containarium list didn't scale (fixed with a get command for the one-container case, #1543), and a background traffic-attribution cache had the same bug independent of the benchmark (fixed to only re-fetch what changed, #1546). Full detail, every raw number, every commit: RESULTS.md in the benchmark folder.

What this actually means

If you need gVisor-grade isolation under Kubernetes, Containarium's sandboxes now pack exactly as densely as agent-sandbox's — 373 vs. 373, confirmed twice. That wasn't true when we started this investigation: it took finding a real gap and shipping a real fix to get here, and we're telling that whole story rather than only the flattering ending. The one remaining cost is time-to-density under gVisor (~2× slower to fill the same host), not density itself. If your agents are idle-heavy — most agent sandboxes spend most of their time waiting, not computing — the lever that matters for either system is the same regardless: your actual memory usage is probably far below whatever you've declared as the request. Lower it, or tune a VPA to track reality.

If you're willing to leave Kubernetes and gVisor's isolation model behind, Containarium's native LXC backend can pack meaningfully more — 929 vs. 373 — because it isn't reserving headroom nobody's using. That's a real, different trade-off (weaker isolation boundary, no admission backpressure, a hard wall when you finally exhaust real memory), not a strictly-better number, and not the same comparison as the gVisor-matched result above.

Reproduce it yourself, or tell us where the comparison should go next.

Full methodology, every host spec, every raw number, and the complete bug-by-bug investigation are in the open-source repo.

Read the full methodology How the agent sandbox works