Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

how to use duck CLI to download from an https source? #11600

Closed
cyberduck opened this issue Mar 2, 2021 · 1 comment
Closed

how to use duck CLI to download from an https source? #11600

cyberduck opened this issue Mar 2, 2021 · 1 comment
Labels
cli Command Line Interface feature thirdparty Issue caused by third party

Comments

@cyberduck
Copy link
Collaborator

b96ee3e created the issue

unable to download from an HTTPS source using CLI on Centos linux server.

Command:

duck --verbose --download https://github.com/owid/covid-19-data/blob/master/public/data/owid-covid-data.json ./test.json

wget command on same file works!
downloading from an AWS S3 bucket works using duck CLI and Amazon S3 duck protocol.
I believe https is not a protocol. WebDAV (HTTPS) protocol does not work in this case.

working HTTPS source: https://zlib.net/pigz/pigz-2.6.tar.gz

duck --verbose --download https://zlib.net/pigz/pigz-2.6.tar.gz .

What protocol should I use? Any other suggestions?
@cyberduck
Copy link
Collaborator Author

@dkocher commented

The issue here is that we are a WebDAV client and attempt to read file attributes using the PROPFIND method. We do have however a fallback implemented for interoperability for downloads with regular HTTP servers to use HEAD when PROPFIND fails with 405 Method Not Allowed shown below.

> PROPFIND /pigz/pigz-2.6.tar.gz HTTP/1.1
> Depth: 0
> Content-Type: text/xml; charset=utf-8
> Content-Length: 297
> Host: zlib.net
> Connection: Keep-Alive
> User-Agent: Cyberduck/7.8.2.34203 (Mac OS X/11.2.2) (x86_64)
> Accept-Encoding: gzip,deflate
> Authorization: ********
< HTTP/1.1 405 Method Not Allowed
< Date: Tue, 02 Mar 2021 21:44:50 GMT
< Server: Apache
< Strict-Transport-Security: max-age=63072000; includeSubDomains
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Allow: GET,POST,OPTIONS,HEAD
< Content-Length: 348
< Keep-Alive: timeout=3, max=499
< Connection: Keep-Alive
< Content-Type: text/html; charset=iso-8859-1
> HEAD /pigz/pigz-2.6.tar.gz HTTP/1.1
> Host: zlib.net
> Connection: Keep-Alive
> User-Agent: Cyberduck/7.8.2.34203 (Mac OS X/11.2.2) (x86_64)
> Accept-Encoding: gzip,deflate
> Authorization: ********
< HTTP/1.1 200 OK
< Date: Tue, 02 Mar 2021 21:44:50 GMT
< Server: Apache
< Strict-Transport-Security: max-age=63072000; includeSubDomains
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Last-Modified: Sat, 06 Feb 2021 09:03:00 GMT
< ETag: "de896d-1a158-5baa7341a5900"
< Accept-Ranges: bytes
< Content-Length: 106840
< Keep-Alive: timeout=3, max=498
< Connection: Keep-Alive

This fails for github.com as the server responds with an unexpected 404 response.

> PROPFIND /owid/covid-19-data/blob/master/public/data/owid-covid-data.json HTTP/1.1
> Depth: 0
> Content-Type: text/xml; charset=utf-8
> Content-Length: 297
> Host: github.com
> Connection: Keep-Alive
> User-Agent: Cyberduck/7.8.2.34203 (Mac OS X/11.2.2) (x86_64)
> Cookie: _gh_sess=YxCGJR%2BtqrOzgHmhbHp1aH2iS%2FmJQr7tGLOM%2BTQO46kon7VcLEkHpOCm27gYGqd1B9906%2FG4fH92F57kmZLpT54eBT0PvfdVqOu%2FDC7Y%2FqdcLUkW59Tbdo70AdfRCqrZWsUsiDLVyZTdlIfo0rqRK7ZCTwT%2FOnTClQ4n3KmcfJXlsMJo%2F0DeNPkUrna6x%2FzPDI99kanUktpbqKf46A0lWwlzTaH6IwYsdW9%2F%2FmrKMz80uePYN7AIYzl2ABugs8UG0yPrxm9YJAhEcq%2Fni3zmOQ%3D%3D--oVa7VdlZ4A235P%2FM--vGnym%2F0Cu6IBVZmbLRMA2g%3D%3D
> Accept-Encoding: gzip,deflate
> Authorization: ********
< HTTP/1.1 404 Not Found
< Server: GitHub.com
< Date: Tue, 02 Mar 2021 21:46:09 GMT
< Content-Type: application/json; charset=utf-8
< Transfer-Encoding: chunked
< Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
< X-Frame-Options: deny
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
< Expect-CT: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"
< Content-Security-Policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com cdn.optimizely.com logx.optimizely.com/v1/events; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com collector.githubapp.com avatars.githubusercontent.com github-cloud.s3.amazonaws.com user-images.githubusercontent.com/; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com
< Vary: Accept-Encoding, Accept, X-Requested-With
< Content-Encoding: gzip
< X-GitHub-Request-Id: E833:0601:71D8D2:74D012:603EB221

There is currently no workaround.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 27, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
cli Command Line Interface feature thirdparty Issue caused by third party
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cyberduck