Fix pod network rollback issue 911 by Lokesh2Arvind · Pull Request #1032 · krkn-chaos/krkn

Lokesh2Arvind · 2025-12-27T10:49:35Z

Type of change

Refactor
New feature
Bug fix
Optimization

Description

This PR adds a stateless rollback mechanism for the pod_network_outage scenario.
Previously, this scenario had no rollback support, which could leave behind:

OpenFlow (ovs-ofctl) rules
IFB interfaces
Temporary modtools-* pods

Since the engine does not track detailed flow metadata, the rollback does not attempt to reverse the exact rules created.
Instead, it performs a safe best-effort cleanup that restores cluster networking to a clean baseline.

✔ Key Improvements

Added rollback_pod_network_outage function
Removes all OpenFlow rules with priority 65535
Removes IFB interfaces (modprobe -r ifb)
Cleans up leftover modtools-* pods
Uses temporary privileged cleanup pods to operate on nodes
Added unit test for rollback behavior
Addressed Copilot/IDE suggestions (unused imports, misplaced rollback calls, duplicate registration, formatting fixes)

This significantly improves the reliability of the chaos scenario and prevents networking side effects after failures.

Related Tickets & Documents

Related Issue: Pod Network Scenario Rollback #911
Closes: Pod Network Scenario Rollback #911

Documentation

Is documentation needed for this update?

If checked, a documentation PR must be created and merged in the website repository.

Checklist before requesting a review

I have performed a self-review of my code.
If it is a core feature, I have added thorough tests.

Copilot

Pull request overview

This PR addresses pod network rollback issue #911 by implementing a stateless rollback mechanism for pod network chaos scenarios. The changes enable automatic cleanup of network configurations when chaos experiments fail or complete.

Key Changes

Adds rollback_pod_network_outage function to clean up OpenFlow rules, IFB interfaces, and modtools pods
Integrates rollback registration into three network chaos scenarios: pod_network_outage, pod_egress_shaping, and pod_ingress_shaping
Adds comprehensive unit tests for the rollback functionality
Fixes package directory configuration from "kraken" to "krkn" in setup.cfg

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 17 comments.

File	Description
`krkn/scenario_plugins/native/pod_network_outage/pod_network_outage_plugin.py`	Implements stateless rollback function and integrates rollback handlers into three network chaos scenarios; adds helper functions for UUID resolution and handler initialization
`tests/rollback_scenario_plugins/test_pod_network_outage_rollback.py`	New test file validating rollback cleanup operations including pod creation, command execution, and modtools pod deletion
`setup.cfg`	Corrects package directory from "kraken" to "krkn"

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

krkn/scenario_plugins/native/pod_network_outage/pod_network_outage_plugin.py

tests/rollback_scenario_plugins/test_pod_network_outage_rollback.py

krkn/scenario_plugins/native/pod_network_outage/pod_network_outage_plugin.py

tsebastiani · 2026-01-02T11:26:57Z

@Lokesh2Arvind Thank you very much for your contributions, could you please solve the issues identified by copilot? Thanks a lot!

Lokesh2Arvind · 2026-01-02T13:31:39Z

Sure @tsebastiani , I will try to fix the pending errors as soon as possible.
Thank you

setup.cfg

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

krkn/scenario_plugins/native/pod_network_outage/pod_network_outage_plugin.py

Copilot

Pull request overview

Copilot reviewed 7 out of 8 changed files in this pull request and generated 14 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-06T13:49:21Z

krkn/scenario_plugins/native/pod_network_outage/pod_network_outage_plugin.py

+        # Register rollback before applying chaos
+        run_uuid = _resolve_run_uuid(params.kraken_config)
+        rollback_handler = _init_rollback_handler(run_uuid, SCENARIO_TYPE)
+        rollback_handler.set_rollback_callable(
+            rollback_pod_network_outage,
+            RollbackContent(
+                namespace=test_namespace,
+                resource_identifier=ROLLBACK_RESOURCE_IDENTIFIER,
+            ),
+        )


The test creates 3 rollback handlers (one per scenario function: pod_outage, pod_egress_shaping, pod_ingress_shaping) with the same run_uuid and scenario_type. This could lead to duplicate rollback registrations if multiple scenario types are executed in the same run. Each invocation creates a new handler and registers the same rollback function, potentially causing the rollback to be executed multiple times. Consider registering the rollback once per run, or ensuring the handler doesn't register duplicate callbacks.

Copilot · 2026-01-06T13:49:22Z

krkn/scenario_plugins/native/pod_network_outage/pod_network_outage_plugin.py

+        # Register rollback before applying chaos
+        run_uuid = _resolve_run_uuid(params.kraken_config)
+        rollback_handler = _init_rollback_handler(run_uuid, SCENARIO_TYPE)
+        rollback_handler.set_rollback_callable(
+            rollback_pod_network_outage,
+            RollbackContent(
+                namespace=test_namespace,
+                resource_identifier=ROLLBACK_RESOURCE_IDENTIFIER,
+            ),
+        )


The test creates 3 rollback handlers (one per scenario function: pod_outage, pod_egress_shaping, pod_ingress_shaping) with the same run_uuid and scenario_type. This could lead to duplicate rollback registrations if multiple scenario types are executed in the same run. Each invocation creates a new handler and registers the same rollback function, potentially causing the rollback to be executed multiple times. Consider registering the rollback once per run, or ensuring the handler doesn't register duplicate callbacks.

krkn/scenario_plugins/native/pod_network_outage/pod_network_outage_plugin.py

CI/config/pod_config.yaml

CI/config/customapp_pod_config.yaml

krkn/scenario_plugins/native/pod_network_outage/pod_network_outage_plugin.py

Copilot · 2026-01-06T13:49:25Z

krkn/scenario_plugins/native/pod_network_outage/pod_network_outage_plugin.py

+            except Exception:
+                logging.exception(f"[Rollback] Failed during cleanup operations on node '{node}'")
+
+            finally:
+                # Delete cleanup pod only if it was created
+                if pod_created:
+                    try:
+                        logging.info(f"[Rollback] Deleting cleanup pod: {cleanup_pod}")
+                        kubecli.delete_pod(cleanup_pod, namespace="default")
+                    except Exception:
+                        logging.exception(f"[Rollback] Failed to delete cleanup pod '{cleanup_pod}'")


The rollback function swallows exceptions with broad 'except Exception' clauses and only logs them. While this prevents the rollback from stopping on individual node failures, it may hide important errors. Consider collecting and returning a summary of failures so callers can determine if the rollback was partially successful and take appropriate action.

CI/config/pvc_config.yaml

CI/config/node_config.yaml

FETCH_HEAD

Copilot

Pull request overview

Copilot reviewed 21 out of 23 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

krkn/scenario_plugins/node_actions/docker_node_scenarios.py

.github/workflows/tests.yml

CI/tests/test_node.sh

krkn/scenario_plugins/pod_disruption/pod_disruption_scenario_plugin.py

krkn/scenario_plugins/native/pod_network_outage/pod_network_outage_plugin.py

krkn/scenario_plugins/node_actions/ibmcloud_node_scenarios.py

Copilot

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated 10 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

krkn/scenario_plugins/native/pod_network_outage/pod_network_outage_plugin.py

tests/rollback_scenario_plugins/test_pod_network_outage_rollback.py

krkn/scenario_plugins/native/pod_network_outage/pod_network_outage_plugin.py

setup.cfg

krkn/scenario_plugins/native/pod_network_outage/pod_network_outage_plugin.py

tests/rollback_scenario_plugins/test_pod_network_outage_rollback.py

krkn/scenario_plugins/native/pod_network_outage/pod_network_outage_plugin.py

…h testing - Add rollback handler caching to prevent duplicate registrations across pod_outage, pod_egress_shaping, and pod_ingress_shaping - Implement stateless rollback_pod_network_outage function with pod readiness checks (60s timeout) - Validate node names (DNS-1123) before creating privileged cleanup pods - Clean OpenFlow rules (priority=65535) on br-int and br0 bridges - Unload IFB kernel module to remove ingress shaping virtual interfaces - Delete leftover modtools pods created during chaos execution - Add UUID-based fallback for run_uuid generation - Add missing image argument to apply_net_policy function - Improve error handling with exception logging and tracebacks - Add comprehensive unit tests for rollback logic with mock pod readiness - Update .gitignore to ignore generated CI/config YAML files (keep only common_test_config.yaml) Fixes krkn-chaos#911 Signed-off-by: Lokesh2Arvind <lokiarvstud.gn@gmail.com>

Copilot AI review requested due to automatic review settings December 27, 2025 10:49

Lokesh2Arvind requested review from chaitanyaenr, paigerube14 and tsebastiani as code owners December 27, 2025 10:49

Copilot started reviewing on behalf of Lokesh2Arvind December 27, 2025 10:49 View session

Copilot AI reviewed Dec 27, 2025

View reviewed changes

tsebastiani added the pending discussions label Jan 2, 2026

paigerube14 reviewed Jan 2, 2026

View reviewed changes

setup.cfg Outdated Show resolved Hide resolved

Copilot AI review requested due to automatic review settings January 3, 2026 06:48

Copilot started reviewing on behalf of Lokesh2Arvind January 3, 2026 06:49 View session

Copilot AI reviewed Jan 3, 2026

View reviewed changes

Lokesh2Arvind force-pushed the fix-pod-network-rollback-issue-911 branch from 4a59a9d to b2a81f1 Compare January 6, 2026 06:17

Lokesh2Arvind requested review from Copilot and paigerube14 January 6, 2026 13:42

Copilot started reviewing on behalf of Lokesh2Arvind January 6, 2026 13:42 View session

Copilot AI reviewed Jan 6, 2026

View reviewed changes

paigerube14 reviewed Jan 6, 2026

View reviewed changes

FETCH_HEAD Outdated Show resolved Hide resolved

Lokesh2Arvind force-pushed the fix-pod-network-rollback-issue-911 branch from 431cdf7 to 1f7d74d Compare January 7, 2026 19:48

Lokesh2Arvind requested review from Copilot and paigerube14 January 7, 2026 19:50

Copilot started reviewing on behalf of Lokesh2Arvind January 7, 2026 19:54 View session

Copilot AI reviewed Jan 7, 2026

View reviewed changes

Lokesh2Arvind force-pushed the fix-pod-network-rollback-issue-911 branch from 1f7d74d to fd554e9 Compare January 7, 2026 20:16

Lokesh2Arvind requested a review from Copilot January 7, 2026 20:16

Copilot started reviewing on behalf of Lokesh2Arvind January 7, 2026 20:17 View session

Copilot AI reviewed Jan 7, 2026

View reviewed changes

Lokesh2Arvind force-pushed the fix-pod-network-rollback-issue-911 branch from fd554e9 to fa3ebf4 Compare January 7, 2026 20:27

Lokesh2Arvind force-pushed the fix-pod-network-rollback-issue-911 branch from fa3ebf4 to 649bbff Compare January 25, 2026 14:21

Merge branch 'main' into fix-pod-network-rollback-issue-911

1751442

Conversation

Lokesh2Arvind commented Dec 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Type of change

Description

✔ Key Improvements

Related Tickets & Documents

Documentation

Related Documentation PR (if applicable)

Checklist before requesting a review

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Key Changes

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tsebastiani commented Jan 2, 2026

Uh oh!

Lokesh2Arvind commented Jan 2, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Lokesh2Arvind commented Dec 27, 2025 •

edited

Loading