Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

B2: Upload/Sync - Check fileLength if there is no SHA1 in response #9734

Closed
cyberduck opened this issue Oct 16, 2016 · 13 comments
Closed

B2: Upload/Sync - Check fileLength if there is no SHA1 in response #9734

cyberduck opened this issue Oct 16, 2016 · 13 comments
Labels
b2 Backblaze B2 Protocol Implementation enhancement worksforme
Milestone

Comments

@cyberduck
Copy link
Collaborator

00353a4 created the issue

What is the idea about:[[br]]
Made a ticket recently (#9733), regarding SHA1 verification.[[br]][[br]]

But, there is no SHA1 for large files.[[br]][[br]]

-Implementation:* [[br]]
0) Ask BackBlaze dev support about what is a LARGE FILE.[[br]]Maybe CyberDuck already has a limit, and we know, or even better, we define it.[[br]]
See: (https://www.backblaze.com/b2/docs/large_files.html)[[br]][[br]]

  1. IF the localfile is SMALLER than the LARGE_FILE size, CALCULATE our SHA1 sum. This is a MUST, first step!

  2. Submit our SHA1 properly, check if the uploaded file has our SHA1.[[br]]

  3. IF our localfile is BIGGER than LARGE FILE: Get fileLength! We already have this, it's "filelength()" of File, in Java.[[br]][[br]]

  4. Store this length, do the upload, then check the uploaded file's length with get_file_info. But maybe we get this back in the response.[[br]]
    [[br]]

ORDER depends on Upload/Download.[[br]][[br]]

Ask away!


Attachments

@cyberduck
Copy link
Collaborator Author

00353a4 commented

I can work on this, and the other one.[[br]]
2 questions:[[br]]
First: Is there a Coding Style for the project? Or what cli tool, or program can I use to auto-format my mess so it becomes human-readable for the project devs?[[br]]
Second: What IDE should I use to tame the code?[[br]][[br]]
For code I guess I have to submit a pull request on Github.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

The threshold for large file uploads in B2 is currently set at 200MB. See property b2.upload.largeobject.threshold.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

We do file comparison using checksum, file length and modification date (in that order). See ComparisonServiceFilter.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:1 cyborgduck]:

I can work on this, and the other one.[[br]]
2 questions:[[br]]
First: Is there a Coding Style for the project? Or what cli tool, or program can I use to auto-format my mess so it becomes human-readable for the project devs?[[br]]
Second: What IDE should I use to tame the code?[[br]][[br]]
For code I guess I have to submit a pull request on Github.

We welcome pull requests on (https://g.iterate.ch/projects/ITERATE/repos/cyberduck/browse).

@cyberduck
Copy link
Collaborator Author

@dkocher commented

We always include the X-Bz-Content-Sha header for uploads to B2. See also (https://github.com/iterate-ch/backblaze-b2-java-api).

@cyberduck
Copy link
Collaborator Author

00353a4 commented

Sorry to reopen it Mr. Dkocher.[[br]][[br]]

I have a spare machine now with quite some bandwidth, and I just tried the Beta today.[[br]]
Once I uploaded around 30gigs, I used the Synchronize feature to continue the transfer.[[br]]
It failed. The transfer was fully restarted. Again and again.[[br]][[br]]

Now the fix was: Enable upload timestamps. Even to just continue the transfers.[[br]]
Which is weird. I mean, if I had the same files correctly uploaded (about 100 files up, 100 remaining), and CyberDuck kept overwriting them simply.[[br]]
Only and only enabling the timestamps for Uploads - which is off by default - fixed the problem.[[br]][[br]]

This is why I thought there is no length check. (I did check the code, I don't want to rob your precious time.)[[br]][[br]]

Also I never seen that picture pop up that you posted. Is it in Nightly only? Are you sure it's available on b2?[[br]]
I know CyberDuck is an extremely versatile client on many platforms and that the b2 implementation is new, that's why I wonder.[[br]][[br]]

Thank you, and thank you for helping out. (I would have posted half my stuff in a Forum if there was one, but I can understand why some people dislike forums.)

@cyberduck
Copy link
Collaborator Author

00353a4 commented

Ah yes, I am using Windows as platform, not OS X. But since the core is Java, I don't think that should matter... but maybe I am wrong, I don't know. Just though I should note that.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:8 cyborgduck]:

Ah yes, I am using Windows as platform, not OS X. But since the core is Java, I don't think that should matter... but maybe I am wrong, I don't know. Just though I should note that.
Yes, it doesn't make a difference.

@cyberduck
Copy link
Collaborator Author

00353a4 commented

Then I am clueless, sir.[[br]]
Could you check, please?[[br]][[br]

Simply upload a few files to b2, then try continuing the transfer with Sync or Upload.[[br]]
Do NOT enable timestamps, leave everything on default. (Beta and Stable version are the ones I tested.)[[br]][[br]]

Thank you!

@cyberduck
Copy link
Collaborator Author

@dkocher commented

If you upload a file that is transferred as a large upload it will not be visible to synchronize when interrupted. Please restart the upload to resume a large file upload.

@cyberduck
Copy link
Collaborator Author

00353a4 commented

Okay today I had to grab a few files (download). A few of them downloaded, then I stopped transfer.[[br]]
And then after drag n' dropping the same set of files, this window popped up.[[br]][[br]]

So it's there for downloads, but missing for uploads?[[br]]
I will continue to investigate.

@cyberduck
Copy link
Collaborator Author

00353a4 commented

Replying to [comment:11 dkocher]:

If you upload a file that is transferred as a large upload it will not be visible to synchronize when interrupted. Please restart the upload to resume a large file upload.
[[br]][[br]]
I didn't mean resume like that.[[br]]
Suppose you have 10 files. Small, and big ones.[[br]]
You upload 6. But you have to abort/restart, something.[[br]][[br]]

You stop the transfer, the continue.[[br]]
It starts from 0 again, transferring every file. But see my previous comment I will try to track this down.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Milestone renamed

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
b2 Backblaze B2 Protocol Implementation enhancement worksforme
Projects
None yet
Development

No branches or pull requests

1 participant