Handle network instability during file upload in the Editor

[vchendrix]

Describe the bug
There seems to be an issue with data file uploads where the upload fails consistently for files >= 150mb. I tried this out on dev.nceas.ucsb.edu.

To Reproduce
Steps to reproduce the behavior:

1. Go to dev.nceas.ucsb.edu
2. Click on Submit
3. Upload a 200mb file from a connection upload speed of ~40 Mbps using WIFI (with and without VPN)
4. See error
5. Upload a couple of smaller files (they should succeed)
6. Save dataset

Your permissions should be lost and the user is no longer able to edit the dataset.

Expected behavior
Should be able to upload files of several GB files with no problems over a strong wifi connection. I have not had any problems uploading files over wifi using other web applications. I do wireless backups to the cloud, upload to google drive and submit data via curl to metacat with no problems. All using the same wireless connection.

Screenshots
On NCEAS dev ui

On nceas dev during upload

On nceas after save
I was able to upload two 100GB files but the 200GB file failed

Desktop (please complete the following information):

• OS: MacOS
• Browser: Chrome
• Version: Version 113.0.5672.92 (Official Build) (x86_64)

Additional context
I plugged my laptop directly into the router and had no problems uploading files. I was able to upload up to 1GB size of files (See https://dev.nceas.ucsb.edu/view/urn%3Auuid%3A052ac749-d225-40f4-a72a-6f2d5c3968ff).

So, this might be an issue in how metacatui handles uploading from flaky wifi connections. We are seeing an increasing number of users having this problem on ESS-DIVE.

I think there at least tow major issues:

1. Not handling upload errors gracefully (missing resource map, loss of permissions)
2. Flaky upload support for strong wifi connections

[mbjones]

I just tried this on dev.nceas.ucsb,.edu, first with a single 1Gb file, and then with two 1Gb files. They all uploaded fine, no errors from my home wifi with a cable modem connection rated at 300Mbps down/10 Mbps up. I did this on Mac with FF 112.0.2. Could be browser differences. I think figuring out the problem is going to take some sleuthing. @vchendrix are there any errors or warnings in the browser javascript console that give additional info we could explore?

[vchendrix]

Using Safari (MacOS 13.2.1 )

[vchendrix]

Firefox. Not really any interesting messages

[mbjones]

All of those console log errors indicate connection lost or connect reset types of messages. I think tracking down why the connection was reset will be critical to figuring out the cause. The app failure is a side effect of the connection reset.

[vchendrix]

All of those console log errors indicate connection lost or connect reset types of messages. I think tracking down why the connection was reset will be critical to figuring out the cause. The app failure is a side effect of the connection reset.

Agreed. I am trying to see if there is something in my OS network settings. One thing to note is that my WIFI is the only problem. A wired connecion on my router works fine.

[robyngit]

@vchendrix Did you determine if this issue and issue #2135 are local network or VPN problems? It looks like MetacatUI gave an appropriate error message. How would you envision the Metadata Editor handling these issues more gracefully?

[mbjones]

@robyngit We discussed this on the ess dive meeting, let's chat tomorrow on our dev call on issues to consider.

[robyngit]

During today's dev call, Matt provided more details on this issue, including ways in which we could handle network instability better.

In order to lessen the likelihood of getting web connection error in the first place, a longer-term solution could involve adopting newer technologies, like as HTTP/2 (allows multiple data transfers to occur simultaneously on a single connection) or web sockets (allows a more reliable connection between the user and the server). However, these improvements would require changes to both MetacatUI and our server-side systems.

At a minimum, we should improve MetacatUI's ability to handle interrupted file transfers more gracefully. The objective for this issue is to retain the state of the user's data package in the event of a network interruption during upload. Here's what that might look like:

• Even if the upload fails, a user should still be able to save the metadata entered thus far, along with any files that have been successfully uploaded.
• Failed uploads should be removed from the resource map, and the resource map should not get lost or become corrupted/invalid. Permissions should not be changed.
• The user should have the option to retry failed uploads. (or perhaps MetacatUI should automatically retry failed uploads?)

[mbjones]

@robyngit your proposed bulleted list seems like a good first pass to prevent data package corruption. I think we can pass on the longer-term solution, and simply work from your list of error-recovery fixes that would keep package editing working correctly. So, I repeat them here as a list to check off as we work through them. I think the first and second would need to be implemented together. I think the third item should be fine as is because the user can always add new files to a dataset -- but we should test that it works properly.

• Even if the upload fails, a user should still be able to save the metadata entered thus far, along with the resource map entries for any files that have been successfully uploaded.
• Failed uploads should be removed from the resource map, and the resource map should not get lost or become corrupted/invalid. Permissions should not be changed.
• TEST: The user should have the option to retry failed uploads. (or perhaps MetacatUI should automatically retry failed uploads?)

Could you work on prioritizing this in your bug fixes please? We can discuss whether this might be a good item for @rushirajnenuji when he is done his folder work.