Timeout errors when request is in progress

[eliocamp]

I'm getting these timeout errors coming from curl_fetch_memory()

Error in (function (.x, .f, ..., .progress = FALSE)  : i In index: 1.
Caused by error in `curl::curl_fetch_memory()`:
! Timeout was reached [cds.climate.copernicus.eu]:
Connection timed out after 10001 milliseconds

This happens when the requests is being processed/queued, so it seems to be happening when checking the status of the request, not when downloading the file. I was running multiple concurrent requests and when I change to a single request at a time the issue seems to have gone away; although I'm still testing.

This might be due to some throttling on the part of CDS maybe? I noticed that they only allow for 1 request at a time (at least for era5) so they might also be taking longer to respond to requests.

[khufkens]

CDS is exceedingly bad at load balancing (check the forums for the mess it is). You used to be able to run 20 concurrent queries, but these days are long gone. I think this will just be poor load balancing and them rate limiting to alleviate things.

You can bump the httr timeout on the status check to 30s to see if this fixes it.

response <- httr::GET(
 file.path(rs_server(),"task", private$name),
 httr::add_headers(
   Authorization = paste("Bearer", private$token),
   "Content-Type" = "application/json"),
  httr::timeout(30)
)