Cyberduck Mountain Duck CLI

#9734 closed enhancement (worksforme)

B2: Upload/Sync - Check fileLength if there is no SHA1 in response

Reported by: cyborgduck Owned by:
Priority: normal Milestone: 5.2
Component: b2 Version: 5.1
Severity: normal Keywords: b2
Cc: Architecture:
Platform:

Description

What is the idea about:
Made a ticket recently (#9733), regarding SHA1 verification.

But, there is no SHA1 for large files.

Implementation:
0) Ask BackBlaze dev support about what is a LARGE FILE.
Maybe CyberDuck already has a limit, and we know, or even better, we define it.
See: https://www.backblaze.com/b2/docs/large_files.html

1) IF the localfile is SMALLER than the LARGE_FILE size, CALCULATE our SHA1 sum. This is a MUST, first step!

2) Submit our SHA1 properly, check if the uploaded file has our SHA1.

3) IF our localfile is BIGGER than LARGE FILE: Get fileLength! We already have this, it's "filelength()" of File, in Java.

4) Store this length, do the upload, then check the uploaded file's length with get_file_info. But maybe we get this back in the response.

ORDER depends on Upload/Download.

Ask away!

Attachments (1)

Screen Shot 2016-10-16 at 21.47.48.png (41.5 KB) - added by dkocher on Oct 16, 2016 at 7:59:43 PM.

Download all attachments as: .zip

Change History (16)

comment:1 follow-up: Changed on Oct 16, 2016 at 5:36:55 PM by cyborgduck

I can work on this, and the other one.
2 questions:
First: Is there a Coding Style for the project? Or what cli tool, or program can I use to auto-format my mess so it becomes human-readable for the project devs?
Second: What IDE should I use to tame the code?

For code I guess I have to submit a pull request on Github.

comment:2 Changed on Oct 16, 2016 at 7:05:37 PM by dkocher

The threshold for large file uploads in B2 is currently set at 200MB. See property b2.upload.largeobject.threshold.

comment:3 Changed on Oct 16, 2016 at 7:07:07 PM by dkocher

  • Milestone set to 6.0

We do file comparison using checksum, file length and modification date (in that order). See ComparisonServiceFilter.

Last edited on Oct 16, 2016 at 7:18:15 PM by dkocher (previous) (diff)

comment:4 in reply to: ↑ 1 Changed on Oct 16, 2016 at 7:08:02 PM by dkocher

Replying to cyborgduck:

I can work on this, and the other one.
2 questions:
First: Is there a Coding Style for the project? Or what cli tool, or program can I use to auto-format my mess so it becomes human-readable for the project devs?
Second: What IDE should I use to tame the code?

For code I guess I have to submit a pull request on Github.

We welcome pull requests on https://g.iterate.ch/projects/ITERATE/repos/cyberduck/browse.

comment:5 Changed on Oct 16, 2016 at 7:10:02 PM by dkocher

We always include the X-Bz-Content-Sha header for uploads to B2. See also https://github.com/iterate-ch/backblaze-b2-java-api.

comment:6 Changed on Oct 16, 2016 at 7:18:22 PM by dkocher

  • Resolution set to worksforme
  • Status changed from new to closed

comment:7 Changed on Oct 16, 2016 at 8:14:40 PM by cyborgduck

  • Resolution worksforme deleted
  • Status changed from closed to reopened

Sorry to reopen it Mr. Dkocher.

I have a spare machine now with quite some bandwidth, and I just tried the Beta today.
Once I uploaded around 30gigs, I used the Synchronize feature to continue the transfer.
It failed. The transfer was fully restarted. Again and again.

Now the fix was: Enable upload timestamps. Even to just continue the transfers.
Which is weird. I mean, if I had the same files correctly uploaded (about 100 files up, 100 remaining), and CyberDuck kept overwriting them simply.
Only and only enabling the timestamps for Uploads - which is off by default - fixed the problem.

This is why I thought there is no length check. (I did check the code, I don't want to rob your precious time.)

Also I never seen that picture pop up that you posted. Is it in Nightly only? Are you sure it's available on b2?
I know CyberDuck is an extremely versatile client on many platforms and that the b2 implementation is new, that's why I wonder.

Thank you, and thank you for helping out. (I would have posted half my stuff in a Forum if there was one, but I can understand why some people dislike forums.)

comment:8 follow-up: Changed on Oct 16, 2016 at 8:23:20 PM by cyborgduck

Ah yes, I am using Windows as platform, not OS X. But since the core is Java, I don't think that should matter... but maybe I am wrong, I don't know. Just though I should note that.

comment:9 in reply to: ↑ 8 Changed on Oct 17, 2016 at 6:51:36 AM by dkocher

Replying to cyborgduck:

Ah yes, I am using Windows as platform, not OS X. But since the core is Java, I don't think that should matter... but maybe I am wrong, I don't know. Just though I should note that.

Yes, it doesn't make a difference.

comment:10 Changed on Oct 17, 2016 at 7:22:45 AM by cyborgduck

Then I am clueless, sir.
Could you check, please?
[[br]

Simply upload a few files to b2, then try continuing the transfer with Sync or Upload.
Do NOT enable timestamps, leave everything on default. (Beta and Stable version are the ones I tested.)

Thank you!

comment:11 follow-up: Changed on Oct 17, 2016 at 9:17:06 AM by dkocher

If you upload a file that is transferred as a large upload it will not be visible to synchronize when interrupted. Please restart the upload to resume a large file upload.

comment:12 Changed on Oct 17, 2016 at 12:58:34 PM by cyborgduck

Okay today I had to grab a few files (download). A few of them downloaded, then I stopped transfer.
And then after drag n' dropping the same set of files, this window popped up.

So it's there for downloads, but missing for uploads?
I will continue to investigate.

comment:13 in reply to: ↑ 11 Changed on Oct 17, 2016 at 1:00:50 PM by cyborgduck

Replying to dkocher:

If you upload a file that is transferred as a large upload it will not be visible to synchronize when interrupted. Please restart the upload to resume a large file upload.



I didn't mean resume like that.
Suppose you have 10 files. Small, and big ones.
You upload 6. But you have to abort/restart, something.

You stop the transfer, the continue.
It starts from 0 again, transferring every file. But see my previous comment I will try to track this down.

comment:14 Changed on Oct 18, 2016 at 1:17:50 PM by dkocher

  • Resolution set to worksforme
  • Status changed from reopened to closed

comment:15 Changed on Oct 19, 2016 at 2:17:19 PM by dkocher

  • Milestone changed from 6.0 to 5.2

Milestone renamed

Note: See TracTickets for help on using tickets.
swiss made software