feat: default network policy per SandboxTemplate by vicentefb · Pull Request #287 · kubernetes-sigs/agent-sandbox

vicentefb · 2026-02-05T06:47:16Z

fixes #263

This PR updates the sandboxclaim_controller to enforce a "Secure by Default" network posture and introduces a highly scalable NetworkPolicy architecture.

Previous Behavior:
If a SandboxTemplate did not specify a NetworkPolicy, the controller created no policy, effectively granting the Sandbox unrestricted network access (depending on the cluster CNI default). This allowed untrusted workloads to potentially access the Node Metadata Server, the host network, or laterally scan other pods in the cluster. Furthermore, policies were envisioned as 1:1 per Sandbox, which would cause significant overhead at scale.

New Behavior:

Shared Network Policies & Reduced API Contention: Network Policies are now created per SandboxTemplate, rather than per SandboxClaim. When 1,000 sandboxes are spawned from the same template, they now share a single NetworkPolicy object. To prevent API Server 409 conflicts and reduce load during mass-reconciliation, the controller utilizes a cached Get -> Create flow (ignoring AlreadyExists) rather than CreateOrUpdate.
Label Management & Backward Compatibility: The controller injects a template-hash label into Sandbox pods to ensure the shared NetworkPolicy correctly selects them (supporting both Warm Pool adoption and cold starts). Label strings have been carefully maintained to ensure existing WarmPool pods are not orphaned during controller upgrades.

Secure By Default Baseline:
If spec.networkPolicy is omitted in the SandboxTemplate, the controller automatically creates a strict default NetworkPolicy with:

Ingress: Allows traffic only from the sandbox-router. All other pod-to-pod ingress is strictly denied.
Egress (Strict Isolation): Allows egress to the Public Internet, but strictly blocks traffic to Private LAN ranges (RFC1918), the Node Metadata Server, and Internal Cluster DNS (CoreDNS). This prevents lateral movement, credential theft, and internal service enumeration probing.
DNS Resolution (CoreDNS Bypass): Because internal DNS is blocked, the controller automatically injects dnsPolicy: None and public nameservers (8.8.8.8, 1.1.1.1) into the Sandbox pod spec. This allows agents to resolve public domains out-of-the-box without exposing the cluster's internal DNS architecture.
Pass-Through Customization: If a user does provide a spec.networkPolicy, the controller respects it as-is, automatically wrapping it with the correct PolicyTypes and Pod Selectors. Users can utilize hostAliases in their templates for local LLM routing without opening DNS holes (example added to docs).

example:

kubectl get networkpolicy -o yaml
apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata:
    name: limited-egress-template-network-policy
    namespace: default
    ownerReferences:
    - apiVersion: extensions.agents.x-k8s.io/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: SandboxTemplate
      name: limited-egress-template
      uid: 332fc874-157f-4f78-b040-64e3fa02a646
  spec:
    egress:
    - to:
      - ipBlock:
          cidr: 0.0.0.0/0
          except:
          - 10.0.0.0/8
          - 172.16.0.0/12
          - 192.168.0.0/16
          - 169.254.0.0/16
      - ipBlock:
          cidr: ::/0
          except:
          - fc00::/7
    ingress:
    - from:
      - podSelector:
          matchLabels:
            app: sandbox-router
    podSelector:
      matchLabels:
        agents.x-k8s.io/template-name-hash: 181aaabd
    policyTypes:
    - Ingress
    - Egress

netlify · 2026-02-05T06:47:22Z

✅ Deploy Preview for agent-sandbox canceled.

Name	Link
🔨 Latest commit	`2725b31`
🔍 Latest deploy log	https://app.netlify.com/projects/agent-sandbox/deploys/699f6065e4a9d3000895e872

vicentefb · 2026-02-05T06:50:25Z

/ok-to-test

extensions/controllers/sandboxclaim_controller.go

vicentefb · 2026-02-18T00:42:28Z

extensions/controllers/sandboxwarmpool_controller.go

 const (
 	poolLabel              = "agents.x-k8s.io/pool"
-	sandboxTemplateRefHash = "agents.x-k8s.io/sandbox-template-ref-hash"
+	sandboxTemplateRefHash = "agents.x-k8s.io/template-name-hash"


I renamed the label key to agents.x-k8s.io/template-name-hash to match the SandboxClaim controller. This fixes a bug where warm pool pods were not being selected by the default "Deny-All" NetworkPolicy, leaving them unsecured.

It's not safe to rename labels. How are old labels on existing resources handled?

sorry, I have reverted the label string back to its original value ("agents.x-k8s.io/sandbox-template-ref-hash") across both the warmpool and claim controllers.

The new template-level NetworkPolicy will now use this original label for its PodSelector. This ensures that any existing pods sitting in the warm pool prior to this upgrade will be seamlessly selected and secured by the new default NetworkPolicy without being orphaned. I also ensured we no longer delete() this label during pod adoption, so the policy remains attached for the pod's entire lifecycle.

extensions/controllers/sandboxclaim_controller.go

mtaufen · 2026-02-23T20:52:36Z

extensions/api/v1alpha1/sandboxtemplate_types.go

+	// created from this template. A single shared NetworkPolicy is created per Template.
+	// If this field is omitted (nil), the controller applies a SECURE DEFAULT policy:
+	// - Ingress: Allow traffic ONLY from the Sandbox Router. All other ingress is denied.
+	// - Egress: Allow DNS Only (UDP/TCP Port 53). All other traffic (including Internet and Metadata Server) is blocked.


Comment needs update now that we allow internet egress but not internal DNS

+1, and PR description needs to be updated as well

Done. I've updated the comments in sandboxtemplate_types.go and the PR description to accurately reflect the new egress posture (Public Internet allowed, Internal/Private CIDRs blocked).

mtaufen · 2026-02-23T20:53:22Z

/lgtm

vicentefb · 2026-02-23T21:11:13Z

/assign @janetkuo

janetkuo · 2026-02-24T01:15:47Z

extensions/api/v1alpha1/sandboxtemplate_types.go

+	// created from this template. A single shared NetworkPolicy is created per Template.
+	// If this field is omitted (nil), the controller applies a SECURE DEFAULT policy:
+	// - Ingress: Allow traffic ONLY from the Sandbox Router. All other ingress is denied.
+	// - Egress: Allow DNS Only (UDP/TCP Port 53). All other traffic (including Internet and Metadata Server) is blocked.


+1, and PR description needs to be updated as well

janetkuo · 2026-02-24T01:21:15Z

extensions/controllers/sandboxclaim_controller.go

@@ -549,65 +554,109 @@ func (r *SandboxClaimReconciler) SetupWithManager(mgr ctrl.Manager) error {
 func (r *SandboxClaimReconciler) reconcileNetworkPolicy(ctx context.Context, claim *extensionsv1alpha1.SandboxClaim, template *extensionsv1alpha1.SandboxTemplate) error {


After this change, network policy becomes a shared resource, and having every individual SandboxClaim reconciliation attempt to CreateOrUpdate this shared object can lead to significant 409 conflicts from the API server, and unnecessary Get calls when scaling to thousands of sandboxes.

Could this be cached locally in an informer to avoid the contention?

I've replaced CreateOrUpdate with a cached Get -> Create flow. The controller now checks the local informer cache first. If the policy exists, it assumes it's valid and moves on. If it's missing, it attempts to Create it, safely catching and ignoring IsAlreadyExists errors to handle race conditions between claims. This eliminates the 409 conflicts and unnecessary Update calls.

janetkuo · 2026-02-24T01:23:20Z

extensions/controllers/sandboxclaim_controller.go

+								"172.16.0.0/12",  // Block Private Class B
+								"192.168.0.0/16", // Block Private Class C
+								"169.254.0.0/16", // Block Link-Local (Metadata Server)
+							},


Some Kubernetes environments use non-standard ranges for Pod or Service CIDRs. Consider making them configurable or dynamically detecting the cluster's private ranges to avoid bypasses.

These are all already configurable via the sandbox template, aren't they?

Yes, our strategy here is to provide a "Secure by Default" baseline targeting standard RFC1918 private ranges. If a cluster administrator is running non-standard IP ranges (like public IPs for pods), they can easily override this default by explicitly defining the networkPolicy block in their SandboxTemplate. I've added a note to the README mentioning this exact scenario.

janetkuo · 2026-02-24T01:25:06Z

extensions/controllers/sandboxwarmpool_controller.go

 const (
 	poolLabel              = "agents.x-k8s.io/pool"
-	sandboxTemplateRefHash = "agents.x-k8s.io/sandbox-template-ref-hash"
+	sandboxTemplateRefHash = "agents.x-k8s.io/template-name-hash"


It's not safe to rename labels. How are old labels on existing resources handled?

janetkuo · 2026-02-24T01:28:13Z

extensions/controllers/sandboxclaim_controller.go

+			// Public Internet Access (Strict Isolation)
+			// This rule allows all ports to PUBLIC IPs, but explicitly blocks private LAN ranges.
+			// NOTE: This intentionally blocks internal cluster DNS (CoreDNS) by default to prevent
+			// agents from probing for service discovery and leaking internal service names.


This will cause sandboxes to fail any domain resolution by default and potentially breaking most agent workloads. Being so strict by default could potentially have the risk of users just copy-pasting allow-all policies just to make their agents work.

This should still allow resolution of public domains by default.

I'm much more worried about all agents ending up in an allow-all posture by default than I am about users overriding with overly-permissive policy in their sandbox templates, though it's a valid concern. With the latter at least they had to explicitly open the door.

To fix this without allowing internal DNS enumeration, I've added a secure default to createSandbox. If the user hasn't specified a DNSPolicy, we automatically inject DNSPolicy: None and set the Nameservers to ["8.8.8.8", "1.1.1.1"].

This completely bypasses CoreDNS. The agent can instantly resolve public internet domains out of the box, but it is physically incapable of resolving internal .cluster.local addresses.

janetkuo · 2026-02-24T01:39:27Z

extensions/api/v1alpha1/sandboxtemplate_types.go

 	// NetworkPolicy defines the network policy to be applied to the sandboxes
-	// created from this template.
+	// created from this template. A single shared NetworkPolicy is created per Template.
+	// If this field is omitted (nil), the controller applies a SECURE DEFAULT policy:


This is a breaking change. It requires a release note if we need to change the default.

What's the change policy in this repo? AIUI this is all currently alpha/experimental?

Even though it still alpha, we should still communicate breaking changes to users to avoid surprises.

Ack, sounds good. What's the process for adding release notes?

I've added the release-note-action-required label. This can follow #191 to add a block that will be published in the release notes.

nit update update to fix e2e tests policy per template nit updated api comments update update update nit updated update update update

k8s-ci-robot · 2026-02-25T20:49:42Z

New changes are detected. LGTM label has been removed.

k8s-ci-robot · 2026-02-25T20:49:47Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: aditya-shantanu, mtaufen, vicentefb
Once this PR has been reviewed and has the lgtm label, please ask for approval from janetkuo. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

extensions/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

dhenkel92 · 2026-02-27T10:56:39Z

extensions/controllers/sandboxclaim_controller.go

+	if template.Spec.NetworkPolicy == nil {
+		np.Spec = buildDefaultNetworkPolicySpec(template.Name)


suggestion: Add a feature toggle

This change forces people to use a shared network policy for the whole pool, which might not be desirable.
For example: we are currently using a network policy per sandbox to be able to control something like internet connectivity for each sandbox individually. So having a shared policy that is generated by default will cause some interesting side effects in our system.

If we want to stay with building default specs, I would at least recommend making it a feature toggle, so that people can opt out if they want/need to.

k8s-ci-robot requested review from barney-s and janetkuo February 5, 2026 06:47

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 5, 2026

vicentefb force-pushed the defaultNetworkPolicy branch from bd0846b to 2d4716d Compare February 5, 2026 06:49

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Feb 5, 2026

acsoto reviewed Feb 5, 2026

View reviewed changes

extensions/controllers/sandboxclaim_controller.go Outdated Show resolved Hide resolved

vicentefb force-pushed the defaultNetworkPolicy branch 2 times, most recently from 6fe4b6e to 2b17f78 Compare February 11, 2026 02:52

vicentefb requested a review from acsoto February 11, 2026 03:50

vicentefb force-pushed the defaultNetworkPolicy branch 3 times, most recently from 0c92531 to 10b3d5b Compare February 12, 2026 01:03

vicentefb changed the title ~~feat: default network policy~~ feat: default network policy per SandboxTemplate Feb 12, 2026

aditya-shantanu reviewed Feb 12, 2026

View reviewed changes

extensions/controllers/sandboxclaim_controller.go Outdated Show resolved Hide resolved

aditya-shantanu approved these changes Feb 12, 2026

View reviewed changes

vicentefb force-pushed the defaultNetworkPolicy branch 4 times, most recently from 39461fb to 081c27f Compare February 18, 2026 00:39

vicentefb commented Feb 18, 2026

View reviewed changes

vicentefb force-pushed the defaultNetworkPolicy branch from 081c27f to b335c0c Compare February 18, 2026 00:47

mtaufen reviewed Feb 18, 2026

View reviewed changes

extensions/controllers/sandboxclaim_controller.go Outdated Show resolved Hide resolved

vicentefb force-pushed the defaultNetworkPolicy branch 2 times, most recently from 8f7b17f to b5d6e64 Compare February 23, 2026 19:03

mtaufen reviewed Feb 23, 2026

View reviewed changes

k8s-ci-robot assigned mtaufen Feb 23, 2026

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 23, 2026

mtaufen approved these changes Feb 23, 2026

View reviewed changes

k8s-ci-robot assigned janetkuo Feb 23, 2026

janetkuo reviewed Feb 24, 2026

View reviewed changes

janetkuo added the release-note-action-required Denotes a PR that introduces potentially breaking changes that require user action. label Feb 24, 2026

feat: default network policy

2725b31

nit update update to fix e2e tests policy per template nit updated api comments update update update nit updated update update update

vicentefb force-pushed the defaultNetworkPolicy branch from b5d6e64 to 2725b31 Compare February 25, 2026 20:49

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 25, 2026

dhenkel92 reviewed Feb 27, 2026

View reviewed changes

		@@ -549,65 +554,109 @@ func (r *SandboxClaimReconciler) SetupWithManager(mgr ctrl.Manager) error {
		func (r SandboxClaimReconciler) reconcileNetworkPolicy(ctx context.Context, claim extensionsv1alpha1.SandboxClaim, template *extensionsv1alpha1.SandboxTemplate) error {

		if template.Spec.NetworkPolicy == nil {
		np.Spec = buildDefaultNetworkPolicySpec(template.Name)

Conversation

vicentefb commented Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for agent-sandbox canceled.

Uh oh!

vicentefb commented Feb 5, 2026

Uh oh!

Uh oh!

Uh oh!

vicentefb Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mtaufen commented Feb 23, 2026

Uh oh!

vicentefb commented Feb 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Feb 25, 2026

Uh oh!

k8s-ci-robot commented Feb 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

vicentefb commented Feb 5, 2026 •

edited

Loading

netlify bot commented Feb 5, 2026 •

edited

Loading

vicentefb Feb 18, 2026 •

edited

Loading