attempt to fix cyclonus tests: detect cyclonus pod stalls and gate on aws-node readiness

[jupdec]

Issue #, if available:

Post-#606 non-skipped PR Cyclonus Tests runs hung the cyclonus Job pod in ContainerCreating for the full 5 h timeout (noticed 5 cases). Under bash -e, the follow-up kubectl logs failure aborted the script before kubectl get pods -A -owide could run — wiping all evidence. The branch turns that into a fast, verbose failure and gates on real aws-node readiness so the submission doesn't happen while the daemonset is still rolling.

Description of changes:

scripts/lib/network-policy.sh (+62) — new function:

• wait_for_aws_node_settled() — polls until every aws-node pod is Ready with restartCount==0 for a settle window. Defaults: 300 s ceiling, 30 s
  window; overridable via AWS_NODE_SETTLE_TIMEOUT / AWS_NODE_SETTLE_WINDOW. Distroless-safe (no kubectl exec into containers). Ignores pods with
  deletionTimestamp so it correctly handles rollouts.

scripts/lib/tests.sh (+86 / −5) — two new functions, one modified:

• dump_cyclonus_diagnostics() — dumps pods (all-namespaces), events in netpol, describe job/pods, aws-node per-container restart counts,
  describe and logs --tail=200 for both aws-node and aws-eks-nodeagent containers. Each line has || true so nothing aborts the dump.
• wait_for_cyclonus_pod_running() — polls the cyclonus pod's .status.phase. Returns success on Running/Succeeded, or fails with a full
  diagnostics dump if the pod is still not Running after a bounded timeout (default 300 s, overridable via CYCLONUS_POD_START_TIMEOUT).
• run_cyclonus_tests() modified — calls wait_for_cyclonus_pod_running before the polling loop, so a stuck pod fails fast (~5 min) instead of
  hanging for 5 h. Also softens the kubectl logs / kubectl get pods / kubectl delete calls with || true / --ignore-not-found=true so set -euoE
  pipefail can't abort the script mid-dump. On job timeout, marks TEST_FAILED=true and calls dump_cyclonus_diagnostics.

scripts/run-cyclonus-tests.sh (+5) — inserts a call to wait_for_aws_node_settled between the addon-install path and run_cyclonus_tests. Aborts
with exit 1 if aws-node hasn't quiesced, refusing to submit cyclonus against a not-ready dataplane.

Testing

Ran cyclonus tests locally successfully

  ┌────────────────────┬─────────────────────────────────────────────────────────────┐
  │                    │                            Value                            │
  ├────────────────────┼─────────────────────────────────────────────────────────────┤
  │ Status             │ complete                                                    │
  ├────────────────────┼─────────────────────────────────────────────────────────────┤
  │ TEST_FAILED        │ false ✅                                                    │
  ├────────────────────┼─────────────────────────────────────────────────────────────┤
  │ Job condition      │ Complete=True                                               │
  ├────────────────────┼─────────────────────────────────────────────────────────────┤
  │ Test cases         │ 112 / 112 all passed                                        │
  ├────────────────────┼─────────────────────────────────────────────────────────────┤
  │ Cyclonus wallclock │ 3h 10m (05:00 → 08:10 UTC)                                  │
  ├────────────────────┼─────────────────────────────────────────────────────────────┤
  │ Watcher wallclock  │ 3013s (50 min, waited for cyclonus to finish then verified) │
  ├────────────────────┼─────────────────────────────────────────────────────────────┤
  │ Completed at       │ 2026-08-14 08:10:52 UTC                                     │
  └────────────────────┴─────────────────────────────────────────────────────────────┘

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[jupdec]

/test cyclonus 2c9a265

[github-actions]

✅ Cyclonus tests triggered by @jupdec for 2c9a265d19ac49394af08547d12b72dcd9a25f05. View run

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 31.63%. Comparing base (7b17620) to head (2c9a265).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #653      +/-   ##
==========================================
+ Coverage   31.60%   31.63%   +0.02%     
==========================================
  Files          27       27              
  Lines        3490     3490              
==========================================
+ Hits         1103     1104       +1     
+ Misses       2285     2284       -1     
  Partials      102      102              

Flag	Coverage Δ
unittest	31.63% <ø> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

Cyclonus Test Results: ✅ Passed

• Commit: 2c9a265d19ac49394af08547d12b72dcd9a25f05
• K8s Version: 1.35
• Image: 443709043722.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-network-policy-agent:2c9a265d19ac49394af08547d12b72dcd9a25f05
• Full logs