Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support upload to S3 buckets with Object Lock enabled #15557

Closed
yoni-yad2 opened this issue Jan 18, 2024 · 6 comments · Fixed by #15620 or #15699
Closed

Support upload to S3 buckets with Object Lock enabled #15557

yoni-yad2 opened this issue Jan 18, 2024 · 6 comments · Fixed by #15620 or #15699
Assignees
@dkocher
Labels
enhancement s3 AWS S3 Protocol Implementation
Milestone
8.8.0

Comments

@yoni-yad2
Copy link

yoni-yad2 commented Jan 18, 2024 •
edited by dkocher

Is your feature request related to a problem? Please describe.
Currently when trying to upload a file into an S3 bucket which has Object Lock enabled, this error shows:

InvalidRequest: Content-MD5 HTTP header is required for Put Object requests with Object Lock parameters

Describe the solution you'd like
Calculate MD5 hash for the file and upload it in the request header along with the file.

Describe alternatives you've considered
Disabling Object Lock on the S3 bucket temporarily to be able to upload the file.

Additional context
Discussion about workarounds for this issue
@dkocher dkocher added the s3 AWS S3 Protocol Implementation label Jan 18, 2024
@dkocher dkocher self-assigned this Jan 18, 2024
@dkocher
Copy link
Contributor

dkocher commented Jan 22, 2024

This would need to be set conditionally depending on the object lock configuration 1 of the bucket.

Footnotes

  1. https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectLockConfiguration.html ↩

@dkocher
Copy link
Contributor

dkocher commented Feb 8, 2024

This would need to be set conditionally depending on the object lock configuration 1 of the bucket.

Footnotes

  1. https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectLockConfiguration.html ↩

It would probably be easier to just set the Content-MD5 unconditionally for al uploads.

@dkocher
Copy link
Contributor

dkocher commented Feb 8, 2024

Reference Using Content-MD5 when uploading objects 1

Footnotes

  1. https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html#checking-object-integrity-md5 ↩

dkocher added a commit that referenced this issue Feb 8, 2024
@dkocher
Fix #15557.
9e73dba
@dkocher
Copy link
Contributor

dkocher commented Feb 8, 2024

Looks like instead we can use one of

x-amz-checksum-crc32: ChecksumCRC32
x-amz-checksum-crc32c: ChecksumCRC32C
x-amz-checksum-sha1: ChecksumSHA1
x-amz-checksum-sha256: ChecksumSHA256

@dkocher dkocher mentioned this issue Feb 8, 2024
Merged
@dkocher dkocher added this to the 8.7.4 milestone Feb 8, 2024
ylangisc added a commit that referenced this issue Feb 12, 2024
@ylangisc
Merge pull request #15620 from iterate-ch/feature/GH-15557
c345dd2 
Add x-amz-checksum-sha256 request header
@dkocher
Copy link
Contributor

dkocher commented Mar 7, 2024

Fix does not apply for multipart uploads.

@dkocher dkocher reopened this Mar 7, 2024
@dkocher
Copy link
Contributor

dkocher commented Mar 7, 2024

Fix does not apply for multipart uploads.

We previously came to the conclusion in f7977d0 that the x-amz-checksum-sha256 must be omitted for parts.

dkocher added a commit that referenced this issue Mar 8, 2024
@dkocher
Fix "Content-MD5 OR x-amz-checksum- HTTP header is required for Put P…
230c28d 
…art requests with Object Lock parameters" error. Added "Content-MD5" header as x-amz-checksum-sha256 causes "Checksum Type mismatch occurred, expected checksum Type: null, actual checksum Type: sha256." for parts. Fix #15557.
@dkocher dkocher linked a pull request Mar 8, 2024 that will close this issue
Fix #15557 for multipart uploads #15699
Merged
dkocher added a commit that referenced this issue Mar 8, 2024
@dkocher
Fix "Content-MD5 OR x-amz-checksum- HTTP header is required for Put P…
a4826c8 
…art requests with Object Lock parameters" error. Added "Content-MD5" header as x-amz-checksum-sha256 causes "Checksum Type mismatch occurred, expected checksum Type: null, actual checksum Type: sha256." for parts. Fix #15557.
dkocher added a commit that referenced this issue Mar 8, 2024
@dkocher
Merge pull request #15699 from iterate-ch/bugfix/GH-15557-Content-MD5
a480ac3 
Fix #15557 for multipart uploads
@dkocher dkocher mentioned this issue Apr 22, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dkocher dkocher

Labels
enhancement s3 AWS S3 Protocol Implementation
Projects
None yet
Milestone
8.8.0
2 participants
@dkocher @yoni-yad2