Zenodo: Student project: Optimisation of large file download/upload performance over HTTP

Use case

• A) A researcher wants to upload his/her 260GB research dataset to Zenodo. The researcher is only in possession of his/her browser to do the job. Variation: The researchers knows a bit of Python and can write a script that uploads a file to Zenodo via the API.
• B) A researcher wants to download a 260GB research dataset from Zenodo.

We're talking about large files here (minimum 100mb,

Goal
Optimise performance of upload and/or download of data to/from Zenodo.

Upload
Move data from to Zenodo as fast as possible. Using 1) JavaScript (Browser) and/or 2) Python (API).

_Client side_
JavaScript: chunk file (many existing libraries support this - e.g. PLUpload), upload chunks in parallel (haven't seen this in any where - could use e.g. javascript web workers).

Python: Same thing as JavaScript - chunk/parallelize + look into HTTP pipeline and TCP layer as well.

Issues to take into account:

• file integrity (how do we make sure that the file the user have is exactly what has uploaded (checksum the file, but if you upload in parallel your checksum algorithm should support it).
• latency
• browser support for features being used.

_Server side_

• Process model (blocking/non-blocking IO) - we're using Gunicorn to run our Python processes which have different event models.
• Performance analysis on the server side (e.g can the server pipe the files straight to file storage or does it need to keep the chunk in memory).
• How can we increase the number of concurrent connections.

Download
Example 1: 260GB dataset in 1000 files. How can a researcher download the files easily without having to click 1000 links.
Example 2: 260GB dataset in 4 files. What happens if network connectivity is lost (i.e. resumable downloads).

_Client side_
Perhaps possible to write a JavaScript app that can help download the 1000 files. Same app could help with resumable downloads. Again, Javascript web worker model could possibly be used to download the file in chunks.

_Server side_
Implement support for HTTP range requests. Like upload (process model, and concurrent downloads). E.g. slow clients taking time to download a file will fill up the line.

Out-of-scope (but should be discussed): Other methods to optimise file delivery like Content Delivery Networks.

Python part will primarily go into inveniosoftware/invenio-files-rest
JavaScript parts will be separate repositories.