Cyberduck Mountain Duck CLI

#10612 assigned defect

Failure proposed upload exceeds the maximum allowed size

Reported by: MarkBlaise Owned by: dkocher
Priority: normal Milestone: 7.0
Component: cli Version: 6.9.0
Severity: normal Keywords:
Cc: emby@… Architecture: Intel
Platform: Windows 8.1

Description

It may just be network hiccups, but Cyberduck v6.9.0 (29768) seems to have a lot of trouble uploading large files to my Wasabi cloud account.

I have been using both Cyberduck and the Duck CLI, and have seen the issue with both.

For example, this morning Duck failed to upload a 290 gb file, after 3 hours of trying. I have a gigabit fiber optic connection, so speed is not the issue.

I tried to enable debugging by following directions on https://trac.cyberduck.io/wiki/help/en/faq#Enabledebuglogging.

  • I created the file C:\Users\Mark\AppData\Roaming\Cyberduck\default.properties It has 1 line of text (logging=debug) - attached.
  • I also defined the system environment variable logging set equal to debug.

When I run a Duck command I get:

C:\Users\Mark>duck -l wasabisys://T5ZHIOV9VGMNBAKU1HQ4@Blaise-Archive/Magni/

  • log4j:WARN No appenders could be found for logger (ch.cyberduck.core.preferences.Preferences).

So it would seem I can't get Duck to log.

While the file is copying, there are 2 entries in the Cyberduck listing in gray text; 1 in the target folder, and 1 in the bucket root (see attached screen shot). These entries do not display a size. Once an upload completes the gray entry in the bucket root is gone, and the entry in the target folder is black, and displays a size.

When an upload fails, the two gray entries remain. If I attempt the upload 3 times and have 3 failures, I will have 6 gray entries (3 in the root, 3 in the target folder), all with the same name.

I guess the gray entries are temp files. Regrettably, when a failed upload is tried again, Duck does not "pickup where it left off" - it starts from the beginning again.

So, I have to delete the grayed root entry, which will also remove the entry in the target folder. One cannot delete the entry in the target folder.

Unfortunately, Wasabi cloud storage charges accrue for 3 months for any file uploaded, even if it is deleted prior to 3 months. This means that I will be paying for every failed file upload for 3 months of storage - and then also paying for the actual file once successfully uploaded.

I would love to figure out why these uploads fail and correct the issue, so I can stop paying for 3 months of storage for each failed upload file fragment.

I would also like to get logging working for Duck.

Right now another copy attempt is being made for this morning's failure, After that, I'll try a large upload from the Cyberduck GUI to see if that is logging.

Attachments (3)

default.properties (15 bytes) - added by MarkBlaise on Feb 13, 2019 at 6:33:06 PM.
my default properties file
GrayedEntries.jpg (108.4 KB) - added by MarkBlaise on Feb 13, 2019 at 6:33:38 PM.
screen shot of grayed entries
Copy-122gb-File.zip (2.1 MB) - added by MarkBlaise on Feb 16, 2019 at 2:13:33 PM.
log output of failed 122 GiB file upload to Wasabi via Duck -v

Change History (13)

Changed on Feb 13, 2019 at 6:33:06 PM by MarkBlaise

my default properties file

Changed on Feb 13, 2019 at 6:33:38 PM by MarkBlaise

screen shot of grayed entries

comment:1 Changed on Feb 14, 2019 at 7:34:30 AM by dkocher

  • Component changed from core to cli

I can confirm the grayed file entries are pending uploads. We will try to reproduce the issue with uploads not resuming using the --existing resume flag.

comment:2 Changed on Feb 14, 2019 at 2:17:06 PM by dkocher

  • Milestone set to 6.9.3
  • Owner set to dkocher
  • Status changed from new to assigned

comment:3 Changed on Feb 14, 2019 at 2:28:39 PM by MarkBlaise

Thank you Any ideas about why logging is not occurring?

Last edited on Feb 14, 2019 at 5:08:41 PM by MarkBlaise (previous) (diff)

comment:4 Changed on Feb 15, 2019 at 12:08:10 PM by dkocher

  • Milestone changed from 6.9.3 to 7.0

Ticket retargeted after milestone closed

comment:5 Changed on Feb 15, 2019 at 1:52:16 PM by dkocher

Can you run with the --verbose flag and look for any concluding error message.

Changed on Feb 16, 2019 at 2:13:33 PM by MarkBlaise

log output of failed 122 GiB file upload to Wasabi via Duck -v

comment:6 Changed on Feb 16, 2019 at 2:14:42 PM by MarkBlaise

Two things ...

(1) Last night a large file upload to the Wasabi cloud failed. I was working with Wasabi support, and I had turned on Wasabi account logging, Here's what they said:

I and one of the engineering team member who is working on this case looked into the logs and found that when the upload is almost complete, Cyberduck sends a PUT request of the whole file which in turn is rejected by our API as it is above the threshold of the multi-part upload for a single part. The error generated is given below:

<Error> 
 <Code>EntityTooLarge</Code> 
 <Message>Your proposed upload exceeds the maximum allowed size</Message> 
 <ProposedSize>297379400337</ProposedSize> 
 <MaxSizeAllowed>5368709120</MaxSizeAllowed> 
 <RequestId>03FF43F2B13795F8</RequestId>
 <HostId>l1BlqtLcUAY/sAC1U8mJhc7dsAlJf+qNqRz3MwqBwZh3DqbSuDaCP64ZkiCRtzkaGYdlLEva+/uH</HostId> 
</Error>

(2) Note that I am currently using a Wasabi trial account to test the feasibility of using their service and Cyberduck for our backup purposes. The trial has a 1 TB storage limit.

This morning I attempted to upload a 122 GiB file via Duck with the -v switch. There was more than 300 GiB of space available. The upload failed. Here is the command line used:

Z:\Backup-Staging\Uploaded>duck  -v -P -e rename --upload wasabisys://HKIAYRYIVIYKGMHKPSN8@Bax-Backu
p/Test/ \\odin\Backup-Staging\Uploaded\20190209-Guinan-Full-Bax-Bck_20190210015533.nbd > Copy-122gb-
File.log
log4j:WARN No appenders could be found for logger (ch.cyberduck.core.preferences.Preferences).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Z:\Backup-Staging\Uploaded>

I have attached a ZIP of the log to which output was redirected, Copy-122gb-File.zip

If I understand the Wasabi log correctly, this is a problem Cyberduck has uploading large files (to Wasabi); it is not about available space.

Please let me know what you determine.

Also, why can I not get Duck logging working?
log4j:WARN No appenders could be found for logger (ch.cyberduck.core.preferences.Preferences)
If I understand the information at the link in the error message
http://logging.apache.org/log4j/1.2/faq.html#noconfig
it has to do with Duck/Cyberduck Java configuration.

comment:7 Changed on Feb 16, 2019 at 6:12:20 PM by dkocher

  • Summary changed from Failure to upload large files (200+ gb) to Failure proposed upload exceeds the maximum allowed size

comment:8 Changed on Feb 16, 2019 at 6:12:52 PM by dkocher

[▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮  ] 117.1 GiB (125,683,690,321 bytes) of 117.1 GiB (99%, 15.2 MB/sec, 1 seconds remaining)
< HTTP/1.1 200 OK
< Date: Sat, 16 Feb 2019 13:34:06 GMT
< ETag: "9bb0b914fb0a373e78e56d6a9e54d658"
< Server: WasabiS3/3.3.791-2019-02-06-b50720e (head05)
< x-amz-id-2: GzvYZEabUcTsMzOwIEbHDXO79ARrfZ8/PLqbVxmWpkAfuDkA6CUJQBQG+N3BCQNUIFX7STb1azOq
< x-amz-request-id: 3390275C746FBE5D
< Content-Length: 0
< HTTP/1.1 200 OK
< Date: Sat, 16 Feb 2019 13:34:06 GMT
< ETag: "1085654e1bee3fba264da3dc40e963e4"
< Server: WasabiS3/3.3.791-2019-02-06-b50720e (head04)
< x-amz-id-2: f48ge9U0jwYPjRvNWddZSJFrivID4IdTi+RzmC+HlNN41h2fwmZwJxt69y2bKlAhxerXW4VpdvUw
< x-amz-request-id: C9AD360D45D9980F
< Content-Length: 0
> PUT /Bax-Backup/Test/20190209-Guinan-Full-Bax-Bck_20190210015533.nbd HTTP/1.1
> Date: Sat, 16 Feb 2019 13:34:06 GMT
> Expect: 100-continue
> Content-Type: application/octet-stream
> x-amz-content-sha256: fed4d2ddbbdc9213fe7f7ddb5fbac6df82020731316745e878af092bbe46eabf
> Host: s3.wasabisys.com
> x-amz-date: 20190216T133406Z
> Authorization: ********
> Content-Length: 125684796350
> Connection: Keep-Alive
> User-Agent: Cyberduck/6.9.0.29768 (Windows 8.1/6.3) (x86)
Transfer incomplete…
Upload 20190209-Guinan-Full-Bax-Bck_20190210015533.nbd failed. Your proposed upload exceeds the maximum allowed size. Please contact your web hosting service provider for assistance.

comment:9 Changed on Feb 16, 2019 at 9:11:26 PM by dkocher

I cannot yet reproduce it with smaller multipart uploads where the expected POST request to complete the upload is sent.

comment:10 Changed on Feb 16, 2019 at 9:51:46 PM by MarkBlaise

I could be wrong, but my admittedly limited understanding is that the file is uploaded, part by part, but in the end Cyberduck executes a PUT of the entire file.

From the Wasabi log, it seems that the maximum size of a "part" is 5 GiB.

I can say that I have successfully uploaded files as large as 35 GiB, perhaps even 90 GiB. But these files are also larger than the 5 GiB maximum part size, so it seems that Cyberduck does not always POST the entire file - otherwise an upload of any file larger than 5 GiB would fail.

When and why Cyberduck POSTs the entire file at the end of the upload, I cannot say.

Note: See TracTickets for help on using tickets.
swiss made software