[bug] DataSourceGCELocal fails on GCE c3-metal instances due to NIC not ready at first boot

[goldberl]

Bug report

C3-metal instances on Google Cloud fail to boot properly because cloud-init runs before the network interface (NIC) is up. As a result, cloud-init cannot detect any instance datasource.

Steps to reproduce the problem

1. Launch a c3-standard-192-metal machine on Google cloud.

gcloud compute instances create goldberl-c3-test-01 \
  --project=ubuntu-os-support \
  --zone=europe-west1-c \
  --machine-type=c3-standard-192-metal \
  --maintenance-policy=TERMINATE \
  --provisioning-model=STANDARD \
  --image=projects/ubuntu-os-support/global/images/melissa-questing-serial-eu \
  --boot-disk-size=10GB \
  --boot-disk-auto-delete

2. ssh into the instance

gcloud compute ssh goldberl-c3-test-01 --zone=europe-west1-c

3. See the cloud-init status --long

ubuntu@goldberl-c3-test-01:~$ cloud-init status --long
status: error
extended_status: error - done
boot_status_code: enabled-by-generator
last_update: Thu, 01 Jan 1970 00:00:28 +0000
detail: Cloud-init enabled by systemd cloud-init-generator
errors:
	- No instance datasource found.
	- Can not apply stage config, no datasource found! Likely bad things to come!
	- Can not apply stage final, no datasource found! Likely bad things to come!
recoverable_errors:
WARNING:
	- required key instance-id returned nothing. not GCE
	- No instance datasource found! Likely bad things to come!
	- Getting data from <class 'cloudinit.sources.DataSourceGCE.DataSourceGCELocal'> failed
	- Getting data from <class 'cloudinit.sources.DataSourceGCE.DataSourceGCELocal'> failed
	- Can not apply stage config, no datasource found! Likely bad things to come!
	- Can not apply stage final, no datasource found! Likely bad things to come!

Other relevant logs

cloud-init analyze show
-- Boot Record 01 --
The total time elapsed since completing an event is printed after the "@" character.
The time the event takes is printed after the "+" character.
 
Starting stage: init-local
|`->no cache found @00.00300s +00.00100s
|`->no local data found from DataSourceGCELocal @00.01100s +00.00000s
Finished stage: (init-local) 00.01700 seconds
 
Starting stage: init-network
|`->no cache found @00.49000s +00.00000s
|`->no network data found from DataSourceGCE @00.49000s +05.01900s
Finished stage: (init-network) 08.86900 seconds
 
Starting stage: modules-config
|`->no cache found @10.74300s +00.00000s
Finished stage: (modules-config) 00.02900 seconds
 
Starting stage: modules-final
|`->no cache found @12.58300s +00.00100s
Finished stage: (modules-final) 00.02800 seconds
 
Total Time: 8.94300 seconds
 
1 boot records analyzed 

cloud-init analyze blame
-- Boot Record 01 --
     05.01900s (init-network/search-GCE)
     00.00100s (modules-final/check-cache)
     00.00100s (init-local/check-cache)
     00.00000s (modules-config/check-cache)
     00.00000s (init-network/check-cache)
     00.00000s (init-local/search-GCELocal) 

cloud-init analyze boot
-- Most Recent Boot Record --
    Kernel Started at: 2026-02-03 03:26:06.222816+00:00
    Kernel ended boot at: 2026-02-03 03:26:18.999937+00:00
    Kernel time to boot (seconds): 12.77712106704712
    Cloud-init activated by systemd at: 2026-02-03 03:26:22.327605+00:00
    Time between Kernel end boot and Cloud-init activation (seconds): 3.3276679515838623
    Cloud-init start: 2026-02-03 03:26:22.388000+00:00
successful 

ubuntu@goldberl-c3-test-01:~$ systemctl status cloud-init-network.service
× cloud-init-network.service - Cloud-init: Network Stage
     Loaded: loaded (/usr/lib/systemd/system/cloud-init-network.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Mon 2026-02-09 20:39:29 UTC; 4min 48s ago
 Invocation: 63fe73bcd7da4604bfeed67d104ef249
    Process: 2615 ExecStart=sh -c echo "start" | nc -Uu -W1 /run/cloud-init/share/network.sock -s /run/cloud-init/share/network-return.sock>
   Main PID: 2615 (code=exited, status=1/FAILURE)
   Mem peak: 1.7M
        CPU: 15ms

Feb 09 20:39:20 localhost systemd[1]: Starting cloud-init-network.service - Cloud-init: Network Stage...
Feb 09 20:39:29 localhost sh[2618]: Completed socket interaction for boot stage network
Feb 09 20:39:29 localhost systemd[1]: cloud-init-network.service: Main process exited, code=exited, status=1/FAILURE
Feb 09 20:39:29 localhost systemd[1]: cloud-init-network.service: Failed with result 'exit-code'.
Feb 09 20:39:29 localhost systemd[1]: Failed to start cloud-init-network.service - Cloud-init: Network Stage.

Environment details

• Cloud-init version: cloud-init 25.3~2g890873f5-0ubuntu2
• Operating System Distribution: Ubuntu Questing (25.10)
• Cloud provider, platform or installer type: Google

cloud-init logs

gcloud-c3-cloud-init.tar.gz

Workarounds

Adding a 5-second delay to cloud-init-local.service allows the VM to boot correctly

Additional info

• Only affects certain instance types (c3-metal); c4d-metal works consistently.
• NOT reproducible on 22.04,
• Somewhat consistently reproducible on 24.04, 25.04, and 25.10
• This issue seems similar to [bug]: EC2 datasource failing on first boot due to missing NICs #6697

[blackboxsw]

Thank you for this bug @goldberl and improving cloud-init experience for users.

There are two things in logs that are indicative of this problem:

1. GCE is typically detected in cloud-init's 'init-local' stage when it should have visibility to the supported network devices when it walks the kernel's representation in /sys/class/net in order to attempt to run a sandboxed DHCP client to reach the IMDS before system network configuration is fully brought up. in cloud-init init-local we see only the loopback device present, no network or emulated devices present with link up.

2026-02-03 03:26:22,399 - sources[DEBUG]: Datasource list set to use a single datasource DataSourceGCELocal.
2026-02-03 03:26:22,399 - util.py[DEBUG]: Reading from /sys/class/net/lo/address (quiet=False)
2026-02-03 03:26:22,399 - util.py[DEBUG]: Reading 18 bytes from /sys/class/net/lo/address
2026-02-03 03:26:22,399 - util.py[DEBUG]: Reading from /sys/class/net/lo/device/device (quiet=False)
2026-02-03 03:26:22,399 - DataSourceGCE.py[DEBUG]: Looking for the primary NIC in: []
2026-02-03 03:26:22,399 - performance.py[DEBUG]: Getting metadata took 0.001 seconds
2026-02-03 03:26:22,400 - log_util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceGCE.DataSourceGCELocal'> failed

2. systemd-networkd-wait-online.service is being skipped, which indicates that necessary NIC drivers for this instance type are not yet loaded either and the skip of system-networkd-wait-online allows cloud-init 'init-network' stage to proceed, and fail detection because it ran "too early" on this instance type.
   Feb 03 03:26:22.857932 localhost systemd[1]: systemd-networkd-wait-online.service - Wait for Network to be Configured was skipped because of an unmet condition check

Your attached journalctl log shows that network appears to be brought up by idpf driver (after cloud-init init-network runs). Ideally, images which we want to boot on this hardware would have required device drivers built into the kernel or initramfs, cloud-init, which runs very early in boot, cannot know whether such hardware exists and will come up with link UP.

Feb 03 03:26:29.971447 localhost gce-nic-naming_2631[5282]: No network devices found
Feb 03 03:26:29.974839 localhost kernel: idpf 0000:05:00.0 enp5s0f0: renamed from eth0
m eth0
Feb 03 03:26:29.975393 localhost systemd-networkd[2593]: eth0: Interface name change detected, renamed to enp5s0f0.
Feb 03 03:26:29.989049 localhost systemd-networkd[2593]: enp5s0f0: Configuring with /run/systemd/network/10-netplan-id0.network (dropins: /etc/systemd/network/10-netplan-id0.network.d/override.conf).
Feb 03 03:26:30.024219 localhost systemd-networkd[2593]: enp5s0f0: Link UP