Compare streaming vs file upload

[katilp]

Compare runtime and total time between

• streaming: https://github.com/cms-dpoa/cloud-processing/blob/main/standard-gke-cluster-gcs/argo/argo_bucket_run.yaml
• input file upload: https://github.com/cms-dpoa/cloud-processing/blob/main/standard-gke-cluster-gcs/argo/argo_bucket_upload.yaml

This does not depend on whether gcs or nfs as the input files are uploaded to the container's local disk.

[katilp]

To read with the root: protocol locally, in metadata-template, change file:/... to `root://localhost//

        awk -F/ '{ printf "root://localhost//code/files/"; print $NF }' bucket/files_{{inputs.parameters.recid}}.txt > bucket/files_local_{{inputs.parameters.recid}}.txt;

To set up a local xrootd server:

xrootd -l - -b /code 

before sed statements.

[tomcordruw]

I did a test using the root: protocol as instructed and the workflow completed successfully, but it didn't seem to have a big impact.

For reference:
argo_bucket_upload.yaml (standard way):

• e2-standard-4: 4 hours 55 minutes (5 hours 17 minutes)

argo_bucket_upload.yaml (using root:):

• e2-standard-4: 5 hours 5 minutes (5 hours 28 minutes)

Edit:
argo_bucket_run.yaml (streaming):

• e2-standard-4: 4 hours 33 minutes (4 hours 57 minutes)

[katilp]

Thanks! I saw the same in a shorter test. Somewhat unexpected but now confirmed.
Do you also have the time for the same run using streaming (i.e. without the file upload)?

[tomcordruw]

Ah yes, it was otherwise identical as in the cost comparison in Issue #46 , so I now added the time for streaming to my comment above.