Security By Hsin, Head of Product 30 August 2026 繁體中文

Day two of the breach: custody, not trust

Yesterday's post was about encryption boundaries and seven rules. On day two, the same PaaS leak got an official statement — and a very different set of community rumors. The gap between them is wide, and you can verify neither. That's what this post is about: when you hand your most important keys to a platform you can't see into, you don't just hand over secrets — you hand over the ability to answer "what did I lose?"

If you run services on the affected platform, get one thing straight first: disclosure obligations don't transfer with hosting. If your service stored your users' personal data and that data may have been touched, the legal duty to notify sits with you, not the platform. Inventory what you put there — where the database connection strings point, what's in the environment variables besides API keys — then decide whether you owe your own users an announcement.

The official attack chain follows the script, step by step

Start with the path in the platform's own statement: one internally leaked cloud admin key, into a shared cluster, VPN access to the control plane from inside the cluster, into the main database — then a deliberate, targeted query-and-export of user environment variables, specifically hunting AI API keys.

Hold that chain up against yesterday's rules and every link is a textbook demonstration: one key opens the whole platform (rule 1: blast radius per tenant); secrets could be queried and exported straight out of a database (rule 2: secrets move only as ciphertext); and through the whole thing the attacker never touched a single customer's service — they hit the one point where everyone's secrets converge. Exactly yesterday's thesis: the leak path isn't lateral movement, it's the convergence point scraped once.

Update, 31 August 2026: the full path deserves a sharper reading than this. The route to the convergence point was lateral movement, out of a shared cluster that was being decommissioned and still held internal access to the core database — so network isolation was one of two gates here, not an irrelevance.

One of yesterday's five questions to ask your PaaS was: "Can your admin token read tenant secrets?" This incident is what the wrong answer looks like.

The real problem: you cannot verify

The official version: only environment variables were exported, everything else was at most counted and sized. The version circulating in the community: database, source code, and infrastructure control, all in the wild. Which is true? We don't know. The point is — the affected users don't know either, and they have no way to check for themselves.

That's the full price of handing over custody. You didn't just hand over keys and data; you handed over incident response itself. The audit logs are theirs. The database access records are theirs. The forensics are done by a firm they hired. Your forensic ceiling is their transparency, and the only move you have left is to wait for the next announcement and decide whether to believe the official version or the rumor.

Rotate your keys, of course. But rotation doesn't fix custody: keys can be replaced, copied data cannot be un-copied, and to the question "what else was taken?" you will only ever get a second-hand answer.

Trust → verify → custody

There are three ways to live with a platform, and the difference shows on the day something breaks — in what you're holding:

Trust: closed-source, fully hosted. Your code, data, and secrets live in their account; your security equals their honesty times their competence, and you can inspect neither variable. On the bad day, all you hold is their announcement.

Verify: open-source, self-hosted. You can read the boundary's code and run the cross-tenant tests yourself; trust becomes something checkable. On the bad day, you hold the code and your own logs.

Custody (BYOC — bring your own compute): the platform orchestrates, but compute, data, and secrets stay in your cloud account or on your machines — your IAM, your audit logs, your jurisdiction. On the bad day, you're reading your own CloudTrail, not refreshing their status page.

These aren't "good vs. bad" — they're how much verification you're willing to trade for convenience. But this incident demonstrates the floor of the trust tier: when you can't see into the platform, even "how bad is it?" is a variable you can only guess at.

The four rules of real BYOC

"BYOC" can be marketing-washed onto anything, so — same as yesterday — we define it with rules, not a feature list. BYOC that actually returns custody to you must do all four:

1. Secrets and data never transit the platform. The control plane sees metadata — names, sizes, lifecycle events — never contents. Your environment variables must not appear in any database that isn't yours, ciphertext included.

2. The data plane runs in your account, under your IAM. Revoking the platform's access is something you can do unilaterally — no ticket, no waiting for their business hours. Who can touch your machines is ultimately decided by your IAM, not by their employee roster.

3. What runs on your machines is open source and auditable. You must be able to read the agent's code and know what it sends out and what commands it listens for. "Trust our agent" is the same sentence as "trust our platform."

4. A breach of the platform must not cascade into you. The platform should hold only workload-scoped join credentials — never your cloud admin keys. BYOC done wrong — "hand us your AWS admin key to manage" — just moves the same bomb from their house into yours, with the fuse in someone else's hand.

Rule 4 deserves a second read: this entire incident began with one admin key. Don't let your BYOC vendor be holding the next one.

An honest reminder: custody isn't free

Take custody back and the responsibility comes back with it: IAM hygiene, host patching, key handling are yours again. The seed of this incident — one leaked admin key — can happen in your account too. What BYOC changes is the blast radius: from "every customer of the platform goes down together" to "only you" — and this time, you can measure the damage yourself.

It also doesn't replace yesterday's seven rules — encryption boundaries, short-lived credentials, fail-closed, the red team in CI: on your own machines, not one of them is optional. Custody decides who can answer the questions when something breaks; the seven rules decide whether it breaks.

Disclosure: Containarium sells BYOC, so of course we have a position. But the four rules above hold without our product — use them to audit any BYOC vendor, including us: the open-source version is Apache 2.0, and every line that runs on your machines is readable. The way to judge a platform was never what it claims — it's what you can verify.

Who has the final say — you, or your platform?

Worth answering before the day it matters: what can you decide unilaterally, and what needs their permission? If you'd like to go through it with someone — what you've granted, and what you could take back — we'll do a free check-up consultation with you.