Cyberduck Mountain Duck CLI

#10103 reopened defect

User Rate Limit Exceeded error when uploading many files

Reported by: vll_dk Owned by: dkocher
Priority: normal Milestone: 7.1
Component: googledrive Version: 6.2.6
Severity: major Keywords: Google drive; Cryptomator
Cc: densejes@… Architecture: Intel
Platform: Windows 10

Description (last modified by dkocher)

When trying to upload a lot of files (500gb) on/off (testing and trying to find the best solution) I, at some point, got the error 403: User Rate Limit Exceeded when uploading to google drive. I'm uploading to a Cryptomator vault. When I read online the problem could be solved by changing the API settings maybe...

Change History (18)

comment:1 Changed on Oct 7, 2017 at 8:58:43 PM by vll_dk

  • Cc densejes@… added

comment:2 Changed on Oct 8, 2017 at 7:54:06 AM by dkocher

  • Summary changed from Gdrive - 403: User Rate Limit Exceeded to 403: User Rate Limit Exceeded when uploading many files

comment:3 Changed on Oct 8, 2017 at 7:54:49 AM by dkocher

  • Milestone set to 6.3
  • Owner changed from Me to dkocher
  • Status changed from new to assigned

comment:4 Changed on Oct 8, 2017 at 7:55:05 AM by dkocher

Potential improvement in r42695 reducing the number of requests for uploads.

comment:5 Changed on Oct 28, 2017 at 11:07:54 AM by vll_dk

The latest update seems to've mostly fixed the issue for individual file upload (still some issues sometimes) but the error still occurs every time when drag 'n dropping folders with multiple files or simply multiple files at a time in one transfer (not with one file per transfer).

comment:6 Changed on Nov 13, 2017 at 4:24:53 PM by dkocher

  • Milestone changed from 6.3 to 7.0

Ticket retargeted after milestone closed

comment:7 Changed on Jan 13, 2019 at 8:43:20 PM by dkocher

  • Description modified (diff)
  • Summary changed from 403: User Rate Limit Exceeded when uploading many files to User Rate Limit Exceeded error when uploading many files

comment:8 Changed on Jan 13, 2019 at 8:44:12 PM by dkocher

We can reproduce this error in our nightly integration tests.

ch.cyberduck.core.exception.RetriableAccessDeniedException: Cannot create file RVIgMgGk.
	at ch.cyberduck.core.cryptomator.CopyWorkerTest.testCopyDirectoryIntoVault(CopyWorkerTest.java:214)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 
403 Forbidden
{
  "code": 403,
  "errors": [
    {
      "domain": "usageLimits",
      "message": "User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=996125414232",
      "reason": "userRateLimitExceeded",
      "extendedHelp": "https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=996125414232"
    }
  ],
  "message": "User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=996125414232"
}
	at ch.cyberduck.core.cryptomator.CopyWorkerTest.testCopyDirectoryIntoVault(CopyWorkerTest.java:214)

comment:9 Changed on May 31, 2019 at 1:49:26 PM by dkocher

  • Milestone 7.0 deleted

comment:10 Changed on Jul 26, 2019 at 9:12:17 AM by dkocher

We could successfully increase the Requests per 100 seconds per user. By default, it is set to 100 requests per 100 seconds per user. We could now set it to the maximum allowed limit of 1,000.

comment:11 Changed on Aug 21, 2019 at 10:20:53 AM by dkocher

  • Milestone set to 7.1

comment:12 follow-up: Changed on Aug 22, 2019 at 11:06:15 AM by dkocher

Google Drive API
Thank you for submitting Case # (ID:20411254) to Google Cloud Platform support for the following quota:
Change Queries per 100 seconds per user from 1,000 to 10,000
Your request is being processed and you should receive an email confirmation for your request. Should you need further assistance, you can respond to that email.

comment:13 in reply to: ↑ 12 Changed on Aug 22, 2019 at 12:52:20 PM by dkocher

Replying to dkocher:

Google Drive API
Thank you for submitting Case # (ID:20411254) to Google Cloud Platform support for the following quota:
Change Queries per 100 seconds per user from 1,000 to 10,000
Your request is being processed and you should receive an email confirmation for your request. Should you need further assistance, you can respond to that email.

Your quota request for api-project-996125414232 has been approved and your project quota has been adjusted accordingly.

comment:14 Changed on Aug 22, 2019 at 12:53:03 PM by dkocher

  • Resolution set to fixed
  • Status changed from assigned to closed

Rate limiting implemented in r47580.

comment:15 Changed on Nov 6, 2019 at 2:37:49 PM by dkocher

Duplicate in #10867.

comment:16 Changed on Nov 12, 2019 at 4:43:30 PM by Statick

  • Resolution fixed deleted
  • Status changed from closed to reopened

I'm still getting many 403 errors from Mountain Duck due to rate exceeded with Google Drive. Using Mountain Duck 3.2.3 15107

Version 0, edited on Nov 12, 2019 at 4:43:30 PM by Statick (next)

comment:17 Changed on Nov 25, 2019 at 3:38:57 PM by dkocher

#10856 closed as duplicate.

comment:18 Changed on Nov 25, 2019 at 3:39:29 PM by dkocher

#10867 closed as duplicate.

Note: See TracTickets for help on using tickets.
swiss made software