Security By Hsin, Head of Product 29 August 2026 繁體中文

The day the secret store gets scraped: encryption boundaries, not firewalls

A PaaS platform leaked tenant environment variables, and a pile of OpenAI and Anthropic API keys ended up sitting in the open. This post covers three things: what to do in the first hour, why "better firewalls" was never going to prevent this, and the seven rules of a security boundary we build Containarium by.

If your keys might be on the leaked list, stop reading. Before you start rotating anything, do the five-minute thing first: go to your AI provider's console and set daily/monthly spend to a hard cap — "disable over the limit", not "notify over the limit". Once a credential is already out, that cap is the only control left that actually bounds the damage. Then come back for root cause.

The leak path is not the one you're picturing

When an incident like this lands, the reflex answer is "harden the firewalls between clusters." Not wrong, exactly — but the leak path in this class of multi-tenant incident is usually not cluster A moving laterally into cluster B.

It's a single control plane or secret store, scraped for every tenant at once. One over-privileged token, one unlocked internal API, one backup file — the attacker never touches your network perimeter. They're already standing at the one point where every tenant's data converges.

Network isolation cannot stop this. You can drop every cross-tenant packet perfectly and it changes nothing about the fact that the secrets sit in plaintext, in one place. The isolation lives at the network layer; the leak happens at the storage layer. They're not even on the same axis.

Update, 31 August 2026: this paragraph was too absolute. The investigation detail showed the route to the convergence point was lateral movement — out of a shared cluster that was being decommissioned and still held internal access to the core database. Network isolation cannot stop this by itself; it is one of two gates, and we understated the other one. The text above is left exactly as published.

The axis that matters is the encryption boundary

Against "the store got scraped wholesale", the effective axis isn't more firewalls — it's the encryption boundary. It comes down to three properties:

The three properties of an encryption boundary

1. Per-tenant keys — one tenant's breach is one tenant's breach, not the whole platform's.
2. The control plane moves only ciphertext — a compromised admin plane yields ciphertext.
3. Decryption at the point of use, bound to workload identity — a platform engineer can restart your service, but cannot read your secrets.

Self-test: if your secret store were copied wholesale tonight, is tomorrow a PR catastrophe — or a status-page post?

Get those three right, and "the store got stolen" downgrades from a catastrophe to an incident: the attacker walks away with a pile of ciphertext, and the keys that open it live in a different trust boundary.

One aside: moving keys out of environment variables into Vault, then expanding them all back into environment variables at service startup, accomplishes nothing — the secrets still sit in plaintext in every process's memory and in /proc; you've just adopted a Vault to babysit. What an encryption boundary is about is where decryption happens and what identity it's bound to — not which brand of store the ciphertext sits in.

Five questions to ask your PaaS

Do my secrets have their own key, or does the whole platform share one?
Who — under what identity — is able to decrypt my secrets?
Where does decryption happen: in the control plane, or next to my workload?
If your backups are stolen, is the haul plaintext or ciphertext?
Can your admin token read tenant secrets?

The question that gets the most awkward answer is where your real boundary is.

Second move: swap long-lived real keys for short-lived credentials

The encryption boundary handles "the store got stolen". The second thing to build is an AI gateway, which handles "the runtime got popped": replace "long-lived real keys scattered across N customer containers" with "the real key exists only inside one controlled boundary, and containers get short-lived, instantly-revocable credentials".

One move solves four problems:

A compromised container yields a ticket that expires in minutes and can be revoked immediately — not the real key.
The bill has a hard ceiling: quotas are enforced, not reconciled after the fact.
After an incident, the platform holds its own forensic data — no asking customers to dig through an upstream provider's console.
All model traffic passes one point, so it's observable.

There's an easily-missed corollary here: an egress allowlist cannot save a key that is already inside the container. The model provider's endpoint is on the allowlist by definition — that's exactly the road an exfiltrated key travels. So the ideal gateway isn't just a proxy — it must be the only road. A gateway that can't block the direct route to the provider is a suggested route, bypassed the moment it's inconvenient. The fix isn't a tighter network; it's a key that never enters the container, and a direct route that doesn't exist.

What the gateway should watch

Once there's a single controlled boundary, detection finally has somewhere to stand. The signals worth watching, strongest first:

The same credential active simultaneously from geographically exclusive locations — the strongest signal; there's almost no benign explanation.

Usage spikes relative to that tenant's own baseline, not an absolute threshold — every tenant's normal looks different.

A sudden switch to the most expensive model — the classic monetization move.

A change in endpoint usage: a workload that only ever did chat suddenly calling embeddings or fine-tuning.

An abnormal authentication-failure ratio — someone probing the quota boundary.

And don't build a two-state response of "normal / disabled". Observe → throttle → alert → circuit-break → revoke — the middle states matter, because one false positive that hard-cuts a paying customer will teach you to never turn automated response on again.

An honest reminder: the order is not reversible

The gateway concentrates every tenant's real keys, which makes it the single highest-value target on the platform. Without the encryption boundary in place first, a gateway merely relocates the risk — and concentrates it. You will have hand-built an even more attractive secret store. That applies just as much to bolting a standalone AI-gateway product onto a platform with no encryption boundary underneath.

So the order is: first make "the store got scraped" yield only ciphertext, then pull the real keys into one controlled boundary. Doing it the other way around is a self-inflicted wound.

The seven rules of a security boundary

We run a multi-tenant platform, so the threat model above is our own threat model. Rather than list features, here are the rules we build Containarium by — they hold without our product, and they work just as well for building your own platform as for auditing the one you're already on.

1. Design the blast radius per tenant. The most any single key, token, or permission can open is one tenant. Anything that can decrypt everything platform-wide isn't a backlog item — it's a thing that must not exist.

2. Secrets move only as ciphertext. Whatever the control plane carries, backs up, or replicates is ciphertext; the ability to decrypt is bound to workload identity, at the point of use. An administrator's power is to operate the service, not to read its secrets.

3. Runtimes hold only short-lived credentials. Real keys live in exactly one controlled place; what a container holds is measured in minutes and revocable immediately. Stealing a nearly-expired ticket is not a win.

4. A bypass route must not exist. A control point only counts if the network layer makes it the only road. A gateway that can't block the direct route is a suggested route.

5. Fail closed. When the encryption machinery is unavailable, refuse to boot and refuse to serve — never fall back to plaintext silently. Better down than naked.

6. Prove the boundary by attacking it — put a red team in CI. Isolation is never configured-then-trusted. And a red team shouldn't just be an annual pentest report: it should be a standing attacker living in your pipeline — on every change it genuinely attempts to cross tenants, and the moment it gets through, the merge is blocked. Defenders get tired, rotate, and leave; an attacker written into CI doesn't. A claimed boundary is no boundary.

7. Don't even leak existence. Cross-tenant access returns "not found" (404), never "no permission" (403). Error messages are data too.

We hold ourselves to these seven — imperfectly, like everyone. And what's true of any platform on any given day changes faster than a blog post does. That's exactly why rules beat feature lists: features go stale; rules can be checked at any moment, against any vendor. Put these seven to us — and to any platform you're evaluating.

Disclosure: Containarium is our product. The open-source version is Apache 2.0, and the implementations of these rules — envelope encryption, eBPF isolation, the cross-tenant sentry — are visible in the OSS daemon: you can self-host and verify without taking our word for it. Also: every vendor in this space (us included) writes architecture docs prettier than reality. The way to judge a platform is not what it claims, but how it talks about what it hasn't done yet.

Go and find out where your data's boundary actually is.

The five questions above are free to ask, and the answers tell you most of what you need. If you'd rather walk through it with someone — where the boundary sits today, what actually protects what — we'll do a free check-up consultation with you.