[Bug] build-scheduler.sh may hang forever waiting for an exact Pending-pods count that may never be reached

[Mapalmeira]

When running the emulation, build-scheduler.sh starts the custom scheduler and then waits until the number of Pending pods in the cluster reaches an exact target (DESIRED_PENDING_PODS). In practice, this condition can fail to ever become true due to normal scheduling variation and delays in measurement. As a result, the script can loop indefinitely and never proceed to the next steps, effectively stalling the whole emulation.

Sometimes (seemingly by chance), the number of Pending pods drops below the expected value and never hits it exactly again. Example:

[SCHEDULER] [INFO] There are still 1613 pending pods. Expected 1614 pending pods. Checking again in 30 seconds...
[SCHEDULER] [INFO] There are still 1608 pending pods. Expected 1614 pending pods. Checking again in 30 seconds...
[SCHEDULER] [INFO] There are still 1601 pending pods. Expected 1614 pending pods. Checking again in 30 seconds...
...
[SCHEDULER] [INFO] There are still 1445 pending pods. Expected 1614 pending pods. Checking again in 30 seconds...
[SCHEDULER] [INFO] There are still 924 pending pods. Expected 1614 pending pods. Checking again in 30 seconds...
[SCHEDULER] [INFO] There are still 567 pending pods. Expected 1614 pending pods. Checking again in 30 seconds...

[Mapalmeira]

Suggested fix: check manually if every pod in pods_and_nodes_map.csv was scheduled instead of depending on pod count.

# for each pod in pods_and_nodes_map, check if it's scheduled
all_initial_pods_scheduled () {
    while IFS=',' read -r pod namespace	node
    do
      # skip empty lines and headers
      [[ -z "$pod" || -z "$namespace" ]] && continue
      [[ "$pod" == "pod" && "$namespace" == "namespace" ]] && continue  
    
      node_name=$(kubectl -n "$namespace" get pod "$pod" -o jsonpath='{.spec.nodeName}')
      if [[ -z "$node_name" ]]; then
          echo "[SCHEDULER] [WARN] $pod has not been scheduled yet."
          return 1
      fi
    done < "$PODS_AND_NODES_MAP"
    
    return 0
}

while true; do
    if all_initial_pods_scheduled; then
        echo "[SCHEDULER] [INFO] All initial pending pods have been scheduled. Killing scheduler."
        kill "$SCHEDULER_PID"
        break
    fi

    echo "[SCHEDULER] [INFO] Not all initial pods were scheduled. Checking again in 10 seconds..."
    sleep 10
done