Cyberduck Mountain Duck CLI

#8263 closed defect (fixed)

Downloading gzipped files decompresses and truncates the content

Reported by: thuettner Owned by: dkocher
Priority: high Milestone: 4.6.4
Component: s3 Version: 4.5.2
Severity: normal Keywords: AWS, S3, CloudTrail
Cc: Architecture: Intel
Platform: Windows 8.1

Description

When I download CLoudTrail files from AWS S3, the files get decompressed and truncated. For instance, the file AWSLogs/*/CloudTrail/*/*/*/*/*_CloudTrail_*_*_*.json.gz has a size of 32.5KB. Downloading it, the file becomes plain text (decompressed) and has a length of 32.5KB. Of course, when you decompress it it should have a bigger length afterwards, not the compressed length.

Btw, decompressing should be an option. Is really nice to have, but not useful in all cases.

Attachments (4)

cyberduck transfers.png (53.5 KB) - added by thuettner on Oct 17, 2014 at 10:22:07 AM.
Screenshot of transfer window
478983378254_CloudTrail_eu-west-1_20140617T0005Z_38m28JInQHaNeCTz.json.gz (4.9 KB) - added by thuettner on Oct 17, 2014 at 10:23:43 AM.
File downloaded by CyberDuck, decompressed automatically and truncated
478983378254_CloudTrail_eu-west-1_20140617T0005Z_38m28JInQHaNeCTz.json (53.7 KB) - added by thuettner on Oct 17, 2014 at 10:27:41 AM.
Same file when directly downloaded from S3 (and then decompressed)
478983378254_CloudTrail_eu-west-1_20140617T0005Z_38m28JInQHaNeCTz.json.2.gz (4.9 KB) - added by thuettner on Oct 17, 2014 at 10:30:16 AM.
Same file when downloaded with another tool

Download all attachments as: .zip

Change History (30)

comment:1 Changed on Oct 17, 2014 at 8:13:49 AM by dkocher

  • Component changed from core to s3
  • Milestone set to 4.5.3
  • Owner set to dkocher
  • Priority changed from normal to high
  • Status changed from new to assigned

comment:2 follow-up: Changed on Oct 17, 2014 at 10:02:28 AM by dkocher

  • Resolution set to worksforme
  • Status changed from assigned to closed

I cannot reproduce this issue. Added test in r15379. Can you please post the transcript from the Transfers window (Ctrl-L) if you reopen this issue. If you have choosen to open the downloaded file with the default application it could be uncompressed after the download is complete. Refer to Preferences → Transfers → Downloads → Open downloaded files with default application.

Changed on Oct 17, 2014 at 10:22:07 AM by thuettner

Screenshot of transfer window

Changed on Oct 17, 2014 at 10:23:43 AM by thuettner

File downloaded by CyberDuck, decompressed automatically and truncated

Changed on Oct 17, 2014 at 10:27:41 AM by thuettner

Same file when directly downloaded from S3 (and then decompressed)

Changed on Oct 17, 2014 at 10:30:16 AM by thuettner

Same file when downloaded with another tool

comment:3 in reply to: ↑ 2 ; follow-up: Changed on Oct 17, 2014 at 10:38:08 AM by thuettner

I added a few files so you can see my results. I don't believe that the default application has something to do with it. When I try to decompress the files with 7zip I get an error message; and text editors can open the *.gz document and display it. For me this looks like CyberDuck is doing the decompression. (This does not happen when I use an alternative tool to download from S3.) I am available for an online session if you want to. Let me know how to contact you.

Last edited on Oct 17, 2014 at 11:22:07 AM by thuettner (previous) (diff)

comment:4 Changed on Oct 17, 2014 at 10:42:14 AM by thuettner

  • Resolution worksforme deleted
  • Status changed from closed to reopened

comment:5 Changed on Oct 17, 2014 at 12:46:54 PM by dkocher

  • Summary changed from Downloading AWS CloudTrail files decompresses and truncates the content to Downloading gzipped files decompresses and truncates the content

comment:6 follow-up: Changed on Oct 19, 2014 at 8:56:44 AM by dkocher

  • Platform set to Windows 7

comment:7 in reply to: ↑ 3 ; follow-up: Changed on Oct 19, 2014 at 8:59:15 AM by dkocher

Replying to thuettner:

I don't believe that the default application has something to do with it.

Can you you let me know the setting in Preferences → Transfers → Downloads → Open downloaded files with default application. and try to disable the feature if it is currently enabled.

comment:8 Changed on Oct 19, 2014 at 7:47:27 PM by dkocher

  • Milestone 4.5.3 deleted

comment:9 in reply to: ↑ 7 Changed on Oct 20, 2014 at 7:34:45 AM by thuettner

The flag was not checked and there is no default application defined.

Replying to dkocher:

Replying to thuettner:

I don't believe that the default application has something to do with it.

Can you you let me know the setting in Preferences → Transfers → Downloads → Open downloaded files with default application. and try to disable the feature if it is currently enabled.

comment:10 in reply to: ↑ 6 Changed on Oct 20, 2014 at 7:35:44 AM by thuettner

I have Windows 8.1 (not Windows 7).

comment:11 Changed on Oct 20, 2014 at 7:47:50 AM by dkocher

  • Platform changed from Windows 7 to Windows 8.1

comment:12 Changed on Oct 21, 2014 at 2:13:05 PM by dkocher

  • Resolution set to worksforme
  • Status changed from reopened to closed

Still cannot reproduce the issue using your test file. I must assume there is another process that touches the file after the download is complete.

comment:13 Changed on Jan 14, 2015 at 12:59:44 PM by Mikhail C

Guys,

I got the same thing. Gzipped files are decompressed and truncated to the size of the archive file, when downloading from S3.

Platform: Windows 7. Version: 4.6.1 (tried to update to the current snapshot, 4.6.2. Didn't help).

comment:14 follow-up: Changed on Jan 14, 2015 at 1:41:14 PM by dkocher

  • Resolution worksforme deleted
  • Status changed from closed to reopened

comment:15 in reply to: ↑ 14 Changed on Jan 14, 2015 at 1:54:28 PM by dkocher

Replying to dkocher:

Also noted in https://groups.google.com/forum/#!topic/cyberduck/yo7YldedY9E.

Can you confirm that your use case is manually compressing the content and setting the Content-Encoding header in S3.

comment:16 Changed on Jan 14, 2015 at 2:04:52 PM by Mikhail C

No, I can't, unfortunately. I'm a consumer of those files. They are uploaded by other people.

Metadata-Info tab says this: Content-Encoding: gzip Content-Type: text/csv

P.S. S3Browser downloads the files as is, without unzipping, as well as my self written java tool. That's why i'm sure that the files are valid, and something's wrong on Cyberduck side.

comment:17 Changed on Jan 14, 2015 at 3:04:38 PM by dkocher

  • Milestone set to 4.7

comment:18 Changed on Jan 25, 2015 at 10:58:53 AM by dkocher

  • Milestone changed from 4.7 to 4.6.4
  • Status changed from reopened to new

I can reproduce the bug here with files in S3 that are compressed with a Content-Encoding: gzip custom header set using metadata. The problem is that we limit reading from the known deflated size of an object which works in general for WebDAV because the Content-Encoding will be applied on the fly when serving the file. The file is stored on the server uncompressed and its length is known and we will read up the n bytes of the uncompressed file from the deflated stream. Compared to S3, the file is always compressed and the deflated size is not known. We only read the n bytes equal the compressed object from the deflated stream.

As a resolution I think we best disable the detection of Content-Encoding when connected to S3 instead of fixing the issue as otherwise users will have downloaded .gz files that are already decompressed. We may better want to retrieve the compressed file as is (and advertised in the object key extension).

comment:20 Changed on Jan 25, 2015 at 11:01:04 AM by dkocher

  • Resolution set to fixed
  • Status changed from new to closed

In r16575.

comment:21 Changed on Jan 26, 2015 at 8:50:21 AM by Mikhail C

Thank you guys! It's working fine now.

comment:22 Changed on Jan 26, 2015 at 12:19:54 PM by thuettner

I just tried it out with the latest version and downloaded CloudTrail files from S3. The files still get decompressed to plain text and then cut.

comment:23 Changed on Jan 26, 2015 at 1:56:29 PM by Mikhail C

Have you updated to the snapshot build? The fix hasn't been released yet.

Maybe this can help you: https://trac.cyberduck.io/wiki/help/en/howto/preferences#Update

comment:24 Changed on Jan 26, 2015 at 2:03:34 PM by thuettner

You are right, that solves the issue.

comment:25 Changed on Jan 26, 2015 at 2:10:57 PM by dkocher

Thanks for confirming the issue is resolved.

comment:26 Changed on Mar 13, 2015 at 9:07:45 PM by dkocher

#8263 is a duplicate.

Note: See TracTickets for help on using tickets.
swiss made software