Cyberduck Mountain Duck CLI

Opened 4 months ago

Last modified 4 months ago

#10278 new enhancement

Optimize Checksum Calculation

Reported by: allklier Owned by: dkocher
Priority: normal Milestone:
Component: s3 Version: 6.4.1
Severity: normal Keywords:
Cc: Architecture:
Platform: macOS 10.12

Description

Two suggestions to optimize checksum calculation while uploading to S3.

I frequently upload very large files (75-100GB) to S3 and the checksum calculation adds a significant delay in a time sensitive workflow. I was just uploading a 75GB file, and the checksum calculation took 10min before the actual upload started. Actual upload time is 32min, so that adds a 33% time penalty in uploading, which is significant and very unfortunate.

  • Compute the checksum during the upload, rather than a separate pre-calc pass. Yes, that reduces redundancy of the checksum because it becomes a single read, but errors are more likely during upload than local disk read.
  • The algorithm for reading the file for checksum calculation seems slow. My primary storage (RAID5) supports read bandwidth in excess of 400MB/s, yet during the calculation of the checksum the read speed never exceeds 120MB/s, so checksum calculation is limited by code not I/O bandwidth.

Change History (1)

comment:1 Changed 4 months ago by dkocher

  • Component changed from core to s3
  • Owner set to dkocher
Note: See TracTickets for help on using tickets.
swiss made software