Cyberduck Mountain Duck CLI

#8919 closed defect (worksforme)

Mismatch between MD5 hash and ETag for multipart upload

Reported by: collector Owned by: dkocher
Priority: high Milestone: 4.7.3
Component: s3 Version: 4.7.1
Severity: blocker Keywords: Mismatch MD5 ETag large upload
Cc: Architecture: Intel
Platform: Mac OS X 10.10

Description (last modified by dkocher)

For several months since buying CyberDuck I've been unable to upload large files to Amazon S3 (greater than 4 or 5 GB) about 95% of the time. The error returned is

Mismatch between MD5 hash (long string) of uploaded data and ETAG (long string) returned by the server.

I've uploaded with and without a VPN and from different locations and different computers (All Mac). Attached is the log file from yesterday (folder names obscured for security). The first upload of 10.45GB failed about 8GB into it. I retried it and it worked (VERY rare). The second file failed 17.7GB into a 26.1GB file.

GET /?max-keys=1000&prefix=XXXX.XX.-converted.videotapes%2F&delimiter=%2F HTTP/1.1
Date: Wed, 08 Jul 2015 18:58:28 GMT
x-amz-request-payer: requester
Authorization: AWS AKIAICYF5VD7M5F7J5JQ:Dcza6UzeiOyuSSUzCRFcuW2BGzI=
Host: XXXX-XXXXXX-videos.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.7.1.17920 (Mac OS X/10.10.3) (x86_64)
HTTP/1.1 200 OK
x-amz-id-2: y4am2JuQYXUQnnPrdhgja1qpAixHb6jgDE/KGd6e1u/aM/RujV2n6pMk8pdMBWNo
x-amz-request-id: 17127AE0D31F8BA9
Date: Wed, 08 Jul 2015 18:58:29 GMT
x-amz-bucket-region: us-east-1
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3
GET /?versioning HTTP/1.1
Date: Wed, 08 Jul 2015 18:58:28 GMT
x-amz-request-payer: requester
Authorization: AWS AKIAICYF5VD7M5F7J5JQ:beq57viq5PVQL1DiDQj8fwCf0Sw=
Host: XXX-XXXXXX-videos.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.7.1.17920 (Mac OS X/10.10.3) (x86_64)
HTTP/1.1 200 OK
x-amz-id-2: it75FkKMTGARsIevWq1VuUA4sStEWQoakCDe7TtyAfQoQaq6dVsj2GkksqB350Ne
x-amz-request-id: C4DEC4D20395AF2A
Date: Wed, 08 Jul 2015 18:58:29 GMT
Transfer-Encoding: chunked
Server: AmazonS3
GET /?max-keys=1000&prefix=XXXX.XX.-converted.videotapes%2F&delimiter=%2F HTTP/1.1
Date: Wed, 08 Jul 2015 18:58:29 GMT
x-amz-request-payer: requester
Authorization: AWS AKIAICYF5VD7M5F7J5JQ:fGhb848vvpg0OKN45HM+cZR1Km0=
Host: XXXX-XXXXXX-videos.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.7.1.17920 (Mac OS X/10.10.3) (x86_64)
HTTP/1.1 200 OK
x-amz-id-2: Fwz+JjYrmOV9fsKlLJTNnI/I+ZbulOUdv0Fzdzoeb7bEJ4BdX6Y2SPuzV8n17k7o
x-amz-request-id: 4CD54467246DE112
Date: Wed, 08 Jul 2015 18:58:29 GMT
x-amz-bucket-region: us-east-1
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3
GET /?max-keys=1000&prefix=XXXX.XX.-converted.videotapes%2F&delimiter=%2F HTTP/1.1
Date: Wed, 08 Jul 2015 20:27:52 GMT
x-amz-request-payer: requester
Authorization: AWS AKIAICYF5VD7M5F7J5JQ:NU0ekJA7ykAcmt5QzPB16oS7OmM=
Host: XXXX-XXXXXX-videos.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.7.1.17920 (Mac OS X/10.10.3) (x86_64)
HTTP/1.1 200 OK
x-amz-id-2: 1kLo6DGW4sLS2iaowUSql91seqA+m1fAymLUbj7xJqv6s24QrquuVo3A6WwEeMgc
x-amz-request-id: 9C05ED42504A1C68
Date: Wed, 08 Jul 2015 20:27:53 GMT
x-amz-bucket-region: us-east-1
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3

Attachments (1)

CyberDuck log.rtf (2.7 KB) - added by collector on Jul 9, 2015 at 1:30:36 PM.
Log File

Download all attachments as: .zip

Change History (7)

Changed on Jul 9, 2015 at 1:30:36 PM by collector

Log File

comment:1 Changed on Jul 9, 2015 at 1:31:30 PM by collector

Pasted text didn't keep line breaks. Attached log file

comment:2 Changed on Jul 9, 2015 at 1:42:25 PM by dkocher

  • Description modified (diff)

comment:3 Changed on Jul 9, 2015 at 1:46:59 PM by dkocher

  • Milestone set to 4.8
  • Status changed from new to assigned
  • Summary changed from Failed On Large Uploads: Mismatch between MD5 hash and ETag to Mismatch between MD5 hash and ETag for multipart upload

comment:4 Changed on Jul 24, 2015 at 12:29:13 PM by dkocher

  • Description modified (diff)

comment:5 Changed on Jul 24, 2015 at 12:29:25 PM by dkocher

#8948 closed as duplicate.

comment:6 Changed on Oct 15, 2015 at 1:48:45 PM by dkocher

  • Resolution set to worksforme
  • Status changed from assigned to closed

Please update to the latest snapshot build available. We have made a change in r18136 that possibly affects write performance to S3.

Note: See TracTickets for help on using tickets.