Vtune times out even when run with collection paused

[bcaddy]

Point of Contact

Robert Caddy

Contact Details

rcaddy@princeton.edu

Vendor/ALCF/other tickets/IDs

VASP-33391

Reproducer Path

/lus/flare/projects/CoreCollapseModel/rcaddy/vtune_issue
/lus/flare/projects/Tools/jkwack-tools-reproducer/Robert_Caddy/vtune_issue

Status

Open

Details

Summary

Running vtune on my code (Fornax) takes an extremely long time, like 6+ hours, before timing out. At the suggestion of Patrick Steinbrecher at Intel I set vtune to start with collection paused then only unpause during the sections of code that I actually wanted to profile. That didn't help so I tried just leaving collection paused for the entire run and it still timed out.

In all cases vtune itself will finish eventually, though that takes 20+ minutes even with collection paused on a single rank, single GPU tile run that takes about 15s on its own. What times out is copying the results back from node local store to Flare; if I try to run directly out of Flare then vtune times out instead. Even trying to delete what does get copied to flare takes 30-60 minutes of running rm non-stop.

How to Build

cd into the Fornax directory within the vtune_issue, source /lus/flare/projects/CoreCollapseModel/rcaddy/vtune_issue/Fornax/machines/alcf/aurora.env and run make.

How to Run

Copy the fornax executable into run_directory which is next to the Fornax directory. The submission script is at /lus/flare/projects/CoreCollapseModel/rcaddy/vtune_issue/run_directory/submission_script.sh and it should have everything set up to run Fornax

Is this a priority/blocking bug?

• Priority

ETA

2026.0

[colleeneb]

Thanks for the report and the nice reproducer!

I was able to reproduce. The filesystem on Aurora can be slow indeed. I chatted with @jkwack and one option could be to tar the vtune results directory and copy it to your home/flare and then untar it after the submission script is over. I tried it on the reproducer you added by adding a tar and copying that over:

tar -zcf /tmp/out.tar.gz "/tmp/${output_dir}"
time scp -r /tmp/out.tar.gz "${working_directory}"                                                                                                                                                                                                   
#time rsync -ah /tmp/out.tar.gz "${working_directory}"  

and it was much much faster since the file size is much smaller (94 MB vs 2.3 GB). (I used scp instead of rsync but it should be basically the same time). Is this an acceptable workaround that you can use? Or are there issues with it?

Thanks!

[bcaddy]

I actually tried that a few weeks ago and it just timed out on the tar command instead; I'll try it again.

Did untaring take a reasonable amount of time?

If this works it's fine as a work around but there's definitely still an issue with vtune if it creates so many files/directories even when started paused that it causes a filesystem to slow down that much.

[colleeneb]

The taring was fast when I tried but the untarring was slow indeed.

Did untaring take a reasonable amount of time?

Untarring took a long time, like 23 min. I guess it would be better to do the untarring not in the submission script, but after it finishes since it only needs a login node.

[bcaddy]

No luck when I tried it. With collection paused it's taken 30 minutes and counting to untar.

The culprit appears to be that within the vtune_output/sqlite-db/timeline directory where vtune generates nearly a million files and directories, even when paused for the entire run of the target application.

I tried it with collection only enabled for 2 kernels, each run only 6 times, and it wasn't done untaring after several hours when I stopped it. It only takes about 15 seconds to untar when done out of a head nodes /tmp directory, though copying it back to flare after takes a very long time.

I can just run it all out of /tmp for now but it's definitely a bug in vtune that it can generate so many files and directories that it takes this long to deal with the output; and it's doubly a bug that it will do that even when collection is paused.

[colleeneb]

With collection paused it's taken 30 minutes and counting to untar.

The filesystem might have been under more stress when you tried it possibly. Untarring will probably always take awhile. To be clear, are you untarring in the submission script where it's still part of the submitted job? (The untarring should be done on a login node after the submission ends since it's so slow and this at least is a workaround so the job doesn't time out.)

I tried it with collection only enabled for 2 kernels, each run only 6 times, and it wasn't done untaring after several hours when I stopped it.

How long was the runtime of this test without vtune and how much time did you request for the vtune run?

it's definitely a bug in vtune that it can generate so many files and directories that it takes this long to deal with the output; and it's doubly a bug that it will do that even when collection is paused.

We'll pass this to the vtune people and see what they say -- I'm not familiar with the internals of vtune or why it's doing what it does. Since it's a product of Intel it may take them time, so it's best to get some kind of workaround for now.

[bcaddy]

I'm untarring on a head node.

The test run without vtune takes about 15 seconds, single MPI rank and single GPU tile. I requested a job time of an hour and it finished in about 40 minutes (not including the untarring of course)

I might have figured out part of the issue. I got the vtune-backend tool running with the data in /tmp on a head node so I could look at results. It looks like it's not actually paused when profiling GPU stuff. It says it's paused in the timeline but it still has data for all the kernels.

[jkwack]

Thanks for sharing the result. In my understanding, pausing collection cannot pause all the collection, so it is why we still see big overhead even with pause. Can you share the VTune result folder? I would like to share it with Intel.

[bcaddy]

It's at /tmp/rcaddy/collection-enabled.tar.gz on head node 11.

[jkwack]

Great. Thanks!
I copied it to /lus/flare/projects/Tools/jkwack-tools-reproducer/Robert_Caddy/vtune_issue for Colleen and Brian.
You can remove the file on /tmp.

[colleeneb]

Just so it isn't missed: @brianwhitney59 kindly reported a JIRA to Intel (he put the number in the issue statement).

[vladimir-tsymbal]

I can take look at the result, but it would be more convenient to copy it from Aurora to my side. For some reason, I cannot copy anything from aurora-uan-**** to auroravendor.alcf.anl.gov:~
@jkwack may be you have some hints on what changed in a procedure of coping data from uan to an external place?
Thank you.

[jkwack]

@vladimir-tsymbal I copied the tar file of the VTune result folder to the following location on Aurora:
/lus/flare/projects/Tools/jkwack-tools-reproducer/Robert_Caddy/vtune_issue/collection-enabled.tar.gz
I think you can access it, since you are in Tools project. If not, please let me know.

[vladimir-tsymbal]

@jkwack Thank you. It appeared to be an rsa key problem - I re-scanned servers from login node and now can copy files. I would take some time though to untar the file.
Preliminary, yes not all data can be stopped during collection on GPU (it's just not being displayed in a result). We will try to figure out how to help in such case, but first thing that came to my mind - using a "computing-tasks-of-interest" knob for a kernel of interest.