Buildkite artifacts still slow to upload #4298
Description
I am running into slow buildkite artifact uploads (of the .tar files) more frequently. I am not sure if this is a buildkite issue or due to latency on central but we should figure out a more concise way to prevent all of the artifacts being uploaded.
See example log of ~20 MB taking 15 minutes to upload
2026-02-12 11:41:00 INFO Found 6 files that match "diagnostic_edmfx_rico_box/output_active/*"
2026-02-12 11:41:00 INFO Uploading to default Buildkite artifact storage
2026-02-12 11:41:00 INFO Creating (0-6)/6 artifacts
2026-02-12 11:41:01 INFO Uploading 019c535f-0022-45ea-869a-15df11bf9c50 diagnostic_edmfx_rico_box/output_active/diagnostic_edmfx_rico_box.yml (4.2 KiB)
2026-02-12 11:41:03 INFO Uploading 019c535f-0023-4e0b-b40b-680ab730e492 diagnostic_edmfx_rico_box/output_active/diagnostic_edmfx_rico_box_parameters.toml (61 KiB)
2026-02-12 11:41:05 INFO Uploading 019c535f-0024-4a21-b213-c18d9c8bef44 diagnostic_edmfx_rico_box/output_active/graceful_exit.dat (1 B)
2026-02-12 11:41:05 INFO Uploading 019c535f-0024-4512-baaa-10302d0dc932 diagnostic_edmfx_rico_box/output_active/hdf5_files.tar part 1/2 (~7.1 MiB starting at ~0 B)
2026-02-12 11:44:22 INFO Uploading 019c535f-0024-4512-baaa-10302d0dc932 diagnostic_edmfx_rico_box/output_active/hdf5_files.tar part 2/2 (~7.1 MiB starting at ~7.1 MiB)
2026-02-12 11:47:54 INFO Uploading 019c535f-0025-4503-ae03-f3fe863bc85b diagnostic_edmfx_rico_box/output_active/nc_files.tar part 1/3 (~6.2 MiB starting at ~0 B)
2026-02-12 11:51:03 INFO Uploading 019c535f-0025-4503-ae03-f3fe863bc85b diagnostic_edmfx_rico_box/output_active/nc_files.tar part 2/3 (~6.2 MiB starting at ~6.2 MiB)
2026-02-12 11:54:16 INFO Uploading 019c535f-0025-4503-ae03-f3fe863bc85b diagnostic_edmfx_rico_box/output_active/nc_files.tar part 3/3 (~6.2 MiB starting at ~12 MiB)