Quantify copy time from the processing pod to nsf vs bucket

[katilp]

When the job processing ends (step runpfnano-template in the workflow, it copies the resulting file to the shared area (NSF vs bucket, depending on the workflow).

Quantify how long it takes to copy a typical output file to NSF and bucket, respectively.

To do this,

• upload an example file to the pod (use e.g. an existing file with a desired size from the CERN open data area, and upload it with xrdcp, see an example)
  • take any of the files from here https://opendata.cern.ch/record/31311/files/MuonEG_PFNano_29-Feb-24_Run2016G-UL2016_MiniAODv2_PFNanoAODv1_root_file_index.txt
  • check the size with ls -l
• print an output message when the upload is ready
• commented out the processing in the workflow (the line starting with cmsRun)
• measure the time that it takes to copy the at the end of the step

[katilp]

For nfs, check the exit timestamp for the argo job with

argo logs pfnano-process-<NNN> -n argo | grep exited | grep pfnano

(pls check)

and compare that to the closing message of the last file, for example

argo logs pfnano-process-<NNN> -n argo | grep -5 "MessageLogger Summary" | grep Closed

Probably grep Closed | tail -1 would do as well.

For the bucket workflow, you get this from the job logs, e.g.

time="2024-08-20T13:24:35.190Z" level=info msg="sub-process exited" argo=true error="<nil>"
time="2024-08-20T13:24:35.306Z" level=info msg="/code/scatter -> /var/run/argo/outputs/artifacts/code/scatter.tgz" argo=true
time="2024-08-20T13:24:35.308Z" level=info msg="Taring /code/scatter"
time="2024-08-20T13:25:15.325Z" level=info msg="archived 2 files/dirs in /code/scatter"

but do check that there's no difference in doing as for nfs