You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently a MD5 hash of every upload to S3 is calculated before starting the upload. This can consume a large amount of time and no progress bar can be given during that operation therefor the upload time estimate is useless.
I agree. Calculating the hash on the fly would be an improvment. The only downside is that we need a second request when we still want to set the value of the MD5 in the metadata of the file as we currently do (see md5-hash in metadata).
If the property s3.upload.metadata.md5 is set to true (false is default), then set the Content-MD5 header and let S3 check the integrity of the upload. Otherwise, we calculate the MD5 on the fly during the upload and compare it to the ETag returned for the upload.
Currently a MD5 hash of every upload to S3 is calculated before starting the upload. This can consume a large amount of time and no progress bar can be given during that operation therefor the upload time estimate is useless.
I suggest to calculate the MD5 hash during the upload when reading from the stream. See for an example: http://stackoverflow.com/questions/304268/using-java-to-get-a-files-md5-checksum
Now S3 will not return an error for a corrupted upload since it has no hash to compare. Instead the returned ETag from S3 has to be used to verify that the upload was successful: http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectPOST.html
Alternatively it would be good to have at least the option to disable the hash computation since there are cases where the overhead is not justified.
The text was updated successfully, but these errors were encountered: