Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

User Rate Limit Exceeded error when uploading many files #10103

Closed
cyberduck opened this issue Oct 7, 2017 · 14 comments
Closed

User Rate Limit Exceeded error when uploading many files #10103

cyberduck opened this issue Oct 7, 2017 · 14 comments
Assignees
Labels
bug fixed googledrive Google Drive Protocol Implementation
Milestone

Comments

@cyberduck
Copy link
Collaborator

1eff184 created the issue

When trying to upload a lot of files (500gb) on/off (testing and trying to find the best solution) I, at some point, got the error 403: User Rate Limit Exceeded when uploading to google drive. I'm uploading to a Cryptomator vault. When I read online the problem could be solved by changing the API settings maybe...

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Potential improvement in 3a98fc6 reducing the number of requests for uploads.

@cyberduck
Copy link
Collaborator Author

1eff184 commented

The latest update seems to've mostly fixed the issue for individual file upload (still some issues sometimes) but the error still occurs every time when drag 'n dropping folders with multiple files or simply multiple files at a time in one transfer (not with one file per transfer).

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Ticket retargeted after milestone closed

@cyberduck
Copy link
Collaborator Author

@dkocher commented

We can reproduce this error in our nightly integration tests.

ch.cyberduck.core.exception.RetriableAccessDeniedException: Cannot create file RVIgMgGk.
	at ch.cyberduck.core.cryptomator.CopyWorkerTest.testCopyDirectoryIntoVault(CopyWorkerTest.java:214)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 
403 Forbidden
{
  "code": 403,
  "errors": [
    {
      "domain": "usageLimits",
      "message": "User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=996125414232",
      "reason": "userRateLimitExceeded",
      "extendedHelp": "https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=996125414232"
    }
  ],
  "message": "User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=996125414232"
}
	at ch.cyberduck.core.cryptomator.CopyWorkerTest.testCopyDirectoryIntoVault(CopyWorkerTest.java:214)

@cyberduck
Copy link
Collaborator Author

@dkocher commented

We could successfully increase the Requests per 100 seconds per user. By default, it is set to 100 requests per 100 seconds per user. We could now set it to the maximum allowed limit of 1,000.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Google Drive API
Thank you for submitting Case # (ID:20411254) to Google Cloud Platform support for the following quota:
Change Queries per 100 seconds per user from 1,000 to 10,000
Your request is being processed and you should receive an email confirmation for your request. Should you need further assistance, you can respond to that email.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:12 dkocher]:

Google Drive API
Thank you for submitting Case # (ID:20411254) to Google Cloud Platform support for the following quota:
Change Queries per 100 seconds per user from 1,000 to 10,000
Your request is being processed and you should receive an email confirmation for your request. Should you need further assistance, you can respond to that email.

Your quota request for api-project-996125414232 has been approved and your project quota has been adjusted accordingly.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Rate limiting implemented in 5e126f4.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Duplicate in #10867.

@cyberduck
Copy link
Collaborator Author

e95bf7f commented

I'm still getting many 403 errors from Mountain Duck due to rate exceeded with Google Drive. Using Mountain Duck 3.2.3 15107

from the log

{
  "code" : 403,
  "errors" : [ {
    "domain" : "usageLimits",
    "message" : "Rate Limit Exceeded",
    "reason" : "rateLimitExceeded"
  } ],
  "message" : "Rate Limit Exceeded"
}'}

@cyberduck
Copy link
Collaborator Author

@dkocher commented

#10856 closed as duplicate.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

#10867 closed as duplicate.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Additional backoff in 637a36b.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

#11736 closed as duplicate.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug fixed googledrive Google Drive Protocol Implementation
Projects
None yet
Development

No branches or pull requests

2 participants