Updating egress policies on running sandboxes.

Every Tensorlake sandbox that carries a network policy sits behind an egress firewall on the host, and we want to explain how it works. This post walks you through the whole enforcement path and highlights two recent improvements: policies you can update while the sandbox runs, and hostname rules enforced per connection instead of per resolution.

The agents our users run inside sandboxes usually have two phases:

• Setup: the sandbox clones a repo, runs npm install or pip install, pulls a browser, maybe fetches a model. This phase usually needs unrestricted internet access.
• Run: then the agent starts working, often on code nobody has reviewed, with credentials in the environment. This phase wants the opposite: a handful of named endpoints, or nothing at all.

Sandboxes have always taken an egress policy (allow_out, deny_out, or no network at all), but a policy was something you chose at creation and couldn't change later. Supporting the two sandbox modes above meant taking a snapshot between those two phases and restoring it with different network rules.

Now a sandbox can have its policy replaced while it runs: the new rules install as one atomic transaction, and there is no window where egress is unenforced. Every connection allowed by the previous policy is analyzed again as part of the update. We do this on purpose, so a stricter policy closes existing connections.

Hostname rules got stricter at the same time (allow_out = ["api.example.com"], deny_out = ["*.tracker.example"]) and are now evaluated per connection, against where the name points now. A destination whose IP addresses change often can't drift out of an allow rule, or slip past a deny.

We've described what these policies do in the docs, but never how the firewall behind them is built. This post covers the how.

Hostname allowlists

The simple implementation would resolve each name when the policy is applied, and write the resulting IP rules into the firewall. For a service with stable addresses, that works, and plenty of products do this. But this approach breaks when IPs change.

For example, a CDN may give you a DNS name that resolves to multiple IPs. That list is not stable; the CDN may rotate it at any time. If the system never resolves the name again, traffic to the new IPs gets blocked, even though the hostname is still in the allowlist, because the rules never got updated.

This is worse with a denylist, because a denied name can gain a new IP address the deny rule has never heard of. Traffic to it goes through, because no rule blocks the new IP.

There is also an attack: a bad actor can use a DNS name that alternates a legitimate IP with a denied one, such as the cloud metadata service. The attacker sets the DNS name's TTL to one second and gives it two IPs, one legitimate and one that points at the target. There is a chance the DNS resolver accepts the name when it returns the legitimate IP, but the client's actual connect() call resolves to the attacked IP a moment later. Under this attack, the system allows traffic to the DNS name without realizing it now points at a forbidden IP.

Under this model, DNS answers are runtime state, not fixed configuration, so the firewall reevaluates them for every new connection.

Two design constraints

When we built the current system, we faced two main challenges.

A packet carries a destination IP, not the hostname the client used to reach it. It says 203.0.113.42, an address that may serve fifty hostnames today and a different fifty tomorrow. The name lives in the DNS query the client made, in the SNI of a TLS ClientHello, or in an HTTP Host header. Hostname enforcement means collecting that evidence without requiring or trusting the guest to report it.

Policy has to change under live traffic. Changing the firewall rules is the easy part. It won't re-analyze existing connections on its own: a flow admitted under the old policy stays alive, because conntrack remembers the old verdict.

Approaches we ruled out

We ruled out three alternatives:

• Put an agent inside the guest: the guest is the untrusted party, so nothing it runs can be part of the enforcement boundary.
• Build a TLS-terminating middlebox: terminating TLS breaks certificate pinning, adds a CA we'd have to protect, and puts us in the business of reading customer traffic, which we want no part of today (we may need to, later, to inject secrets into HTTPS requests).
• Assume control of the underlying network: for the same reason as our ingress path. We run on AWS, GCP, and bare-metal providers, under both Firecracker and gVisor, and the enforcement point has to port across all of them unchanged.

We built it at the host boundary instead. Every sandbox's egress traffic goes through a known veth (virtual network interface) with a stable, sandbox-specific source address. Firecracker guests are translated to a unique host-side address inside their slot's network namespace, and gVisor guests already carry one. That source address is the identity every rule keys on, and we assign it, not the guest.

How TCP connections are evaluated

Say a sandbox with a hostname policy runs curl https://api.example.com under allow_out = ["api.example.com"]. The connection takes the following path:

1. [01]The packet hits a prerouting flow-generation guard before anything else. New flows get the sandbox's current policy generation stamped into the upper 16 bits of their conntrack mark, and flows stamped with an older generation are dropped on the spot. This is how a new policy applies to new and existing connections alike.
2. [02]A per-sandbox NAT chain redirects the sandbox's TCP to a standalone egress proxy on the host, all traffic, not just ports 80 and 443. This chain exists only when the policy actually contains a hostname; a pure IP/CIDR policy never pays for the userspace hop, and is enforced entirely in the kernel. If the proxy is down, the redirect points at a closed port and the kernel resets the connection, so hostname-based egress fails closed.
3. [03]The proxy identifies the sandbox from the peer address and recovers the original destination from conntrack. It then loads the sandbox's policy from a RocksDB database it reads as a secondary, so it never makes an RPC call to the control plane per connection: enforcement of new connections continues even while the dataplane process restarts or is briefly down. An unknown or unparseable policy row refuses the connection.
4. [04]Once the proxy knows what to do with the connection, it waits up to 500 ms for the client's first bytes and peeks at up to 8 KiB with MSG_PEEK. These bytes stay in the socket buffer and are never consumed; we use them only to find the destination hostname, inspecting the TLS ClientHello's SNI or the HTTP/1.x Host header. Anything else, an encrypted ClientHello, a server-first protocol, is treated as "no readable name," never as a partial one.
5. [05]With the policy, original destination, and any readable hostname in hand, the proxy evaluates the connection deny-first: a matching deny_out entry wins over both allow_out and the policy's default action. For proxied traffic, internal destinations, loopback, link-local addresses such as the cloud metadata service at 169.254.169.254, RFC 1918, RFC 6598 carrier-grade NAT, multicast, other sandbox slots, and their IPv6 equivalents, are refused unless an IP or CIDR literal in allow_out explicitly covers the address. A hostname match alone never permits a connection to an internal address.
6. [06]Resolving the hostname inside the proxy, rather than trusting the guest's own lookup, addresses both DNS churn and domain fronting. The proxy resolves the name from the host detected in the connection data, backed by a short host-side answer cache measured in tens of seconds rather than a fresh lookup on every connection, re-checks every DNS answer against internal IP ranges and deny rules, and dials a currently valid address, preserving the original port. If the name doesn't resolve, or every current answer is prohibited, the connection is refused; it never falls back to the address the client picked.
7. [07]Once the proxy has inspected and allowed the connection, it splices the two sockets together and gets out of the way, on the same zero-copy relay engine our ingress path uses. A default global cap of 4,096 connections and a per-sandbox cap of 64 keep one sandbox's connection loop from affecting others.

One consequence is easy to miss: a denied proxied connection still completes its TCP handshake. The proxy has to accept the connection to read the SNI it will go on to refuse. Inside the sandbox, connect() succeeds and the first write gets a reset. When testing a policy, try to transfer data; a successful connect() call by itself proves nothing.

Traffic without hostname information

TCP can carry SNI or a Host header. ICMP, UDP, and QUIC carry nothing, so there we rely on DNS instead.

Queries to the sandbox's admitted resolver are intercepted by the same egress proxy. For an allowed name, the proxy validates the upstream DNS answers, matching query ID and question, following CNAME chains only from the name that was asked, and the order of the next two steps matters:

1. [01]It installs the returned A/AAAA addresses into a TTL-bound nftables set.
2. [02]Only if that step succeeded does it hand the answer to the client. The proxy returns a DNS answer only after the matching firewall entry is already installed.

Those set entries last exactly as long as the DNS record said: an address stays permitted only for the TTL the DNS response returned, rechecked per packet so a UDP conntrack entry can't outlive it. A zero-TTL answer still reaches the guest, but installs nothing. Denied names get REFUSED, but the proxy may still resolve them privately and put their current addresses into a deny set, so even address-literal traffic toward a denied name's infrastructure gets blocked. Exact deny_out hostnames get one more bit of help: when the policy is applied, we resolve them best-effort and seed those addresses into the deny set with a 30-second timeout, so the deny covers UDP and QUIC from the first packet instead of waiting for the sandbox to look the name up itself. Once that bridge expires, intercepted DNS is what keeps the deny current. The seed is best-effort on purpose: a failure to resolve a denied hostname never blocks the sandbox from starting.

Hard-coded addresses, DNS-over-HTTPS, and answers from an unapproved resolver carry no hostname evidence. That traffic is judged on literals, IPs and CIDRs, and the policy's default verdict, which under any non-empty allowlist is to drop.

Applying policy updates to existing connections

nftables makes rule replacement atomic; the harder problem is existing flows.

To make a policy update apply to existing connections too, every update advances a per-sandbox generation number. The transition runs as a write-ahead sequence:

1. [01]Validate and render the complete new nft script before touching live state. The API rejects a malformed destination with a 400 before anything persists, so a running sandbox never sees it. What we deliberately don't check here is whether a hostname resolves: allowlisted hostnames aren't resolved again when the policy is applied, so allow_out = ["no-such-host.invalid"] becomes the active policy and simply denies every connection made under it, per connection, until you replace it. From this step onward, any failure that leaves enforcement unprovable terminates the sandbox instead of reverting to the previous generation, since reverting could reauthorize connections the update meant to close.
2. [02]Persist the new policy as a pending record. This is the recovery point if anything below is interrupted.
3. [03]Fence the proxy: relays admitted under older generations are cancelled, and a watermark stops any in-flight lookup from registering a new relay under the old policy.
4. [04]Apply the nft transaction. From this instant, new flows are stamped with the new generation, and every packet from a flow stamped with an older one is dropped at the prerouting guard, before it can reach the forward path or the proxy. The guard also refuses forgeries: a mid-stream ACK, FIN, or RST claiming to start a new flow is dropped outright, since a real TCP connection starts with SYN.
5. [05]Commit the pending record and synchronize the proxy's view. Reconnects now read the new policy, and unrelated sandboxes never notice.

Try it

Start an unrestricted sandbox, install everything it needs, then replace its policy (PATCH /v1/namespaces/{"{ns}"}/sandboxes/{"{id}"} with a new network object) and hand it to the agent locked down. Harbor drives exactly this per trial phase, setup public, agent on an allowlist, verifier with no network, without restarting the sandbox.