Jump to content
  • 0

GSuite Upload Limit


feerlessleadr

Question

Hi - first of all, thanks for the software, it works great.

 

I'm interested in creating a new cloud drive on my Gsuite account to backup my data, and I wasn't sure how CloudDrive will handle the new upload limits being enforced by Google (soft 750Gb cap per day). 

 

I have a 1Gbps line, so I hit the limit between 2-3 hours using rclone, at which point I stop rclone for 24 hours and begin the upload process again. I'm curious however how these new upload limits are handled with CloudDrive? Will CloudDrive stop trying to upload data once the limit is reached and wait the 24 hour period? Or will it keep trying over and over until it can start uploading again? 

 

I'm not sure how Google handles these api limits (meaning is it 24 hour ban from when the soft cap is hit, or 24 hours from the last time you try to upload)?  Obviously the latter scenario would mean that once the soft cap is reached, unless CloudDrive stops the upload, you would effectively be banned unless CloudDrive is shut down, so that option doesn't seem likely.

 

Before I create the new drive and try to start uploading, I was hoping someone might have some info for me.

 

Thanks!

Link to comment
Share on other sites

13 answers to this question

Recommended Posts

  • 0

Thanks for the response.

 

I guess my question really relates to the following: Once I hit the daily upload limit, after 24 hours passes and the limit is reset, will CloudDrive start uploading again, or will the fact that it is continually trying to upload after the limit is reached cause a longer ban on the upload?

 

It will start uploading again. The retries will not lengthen the API ban. It will still just start working again in at whatever time your reset is. Generally, you likely won't even notice, as long as your cache drive isn't full. 

Link to comment
Share on other sites

  • 0

From what I've read, the API ban only affects upload.  And if that's the case, you will see API/throttling errors (thread was aborted), but the drive should continue to download data, and work. 

 

That said, we do have a bandwidth throttling option.  So you can set that (unfortunately .... 75mbps over 24 hours is that 750GB cap....)

 

 

That said, some of the recent changes in the beta version optimize the API calls and data usage, so it should help avoid hitting the limit. 

Link to comment
Share on other sites

  • 0

Thanks for the response.

 

I guess my question really relates to the following: Once I hit the daily upload limit, after 24 hours passes and the limit is reset, will CloudDrive start uploading again, or will the fact that it is continually trying to upload after the limit is reached cause a longer ban on the upload?

Link to comment
Share on other sites

  • 0

It will start uploading again. The retries will not lengthen the API ban. It will still just start working again in at whatever time your reset is. Generally, you likely won't even notice, as long as your cache drive isn't full. 

 

Is this behavior only on the beta? I was on the main release before and I remember the drive entirely dismounted my first night I left a transfer running.

Also can someone clarify some information on the ban? I've read that it's based off the number of API calls but I've also read that it's based on total bandwidth used. Even so I've heard that it's at around 750GB, but I've only managed an average of 500GB a day over the time I've been uploading. Are there two bans based on both API calls and bandwidth?

Link to comment
Share on other sites

  • 0

Is this behavior only on the beta? I was on the main release before and I remember the drive entirely dismounted my first night I left a transfer running.

Also can someone clarify some information on the ban? I've read that it's based off the number of API calls but I've also read that it's based on total bandwidth used. Even so I've heard that it's at around 750GB, but I've only managed an average of 500GB a day over the time I've been uploading. Are there two bans based on both API calls and bandwidth?

 

No. All releases of CloudDrive, at least since 1.0, will behave the same way. It will simply keep retrying and eventually Google will let it upload data. It won't dismount, there is no reason for it to dismount. It's just an upload error. All of the data is still accessible--including the data in the cloud. As long as the cache drive isn't full, you can still read and write to the drive as normal, it just won't be able to update the data in the cloud until Google permits it to again. You *may* run into problems if your cache drive fills up completely, but even then, I don't think it'll disconnect. I'm not sure on that, though. Writes to the drive get so slow once CloudDrive starts throttling them (when the drive has, I believe, less than 5GB of space left) that I'm not even sure you can fill the cache drive in a 24 hour period!

 

You are describing two different bans. And I wouldn't really call the upload limit a ban--though they do use the same error code to enforce it. There is an API ban that can occur if your exceed some number of API calls in a certain period of time. But CloudDrive does not have this problem. As long as an application implements Google's suggested backoff protocols (which CloudDrive does) it's essentially impossible to get an API ban. The other "ban" (which really isn't) is an upload limit that they enforce by failing API calls after the threshold is met. So once you upload 750GB in a day, all API requests to upload additional data will fail. But they don't actually ban you from the API. All *other* API calls will work as normal. Again, the latter isn't *really* a "ban" at all. Google just uses the same error code (403, user rate limit exceeded) to enforce the upload limit. But they don't actually ban you from additional API calls wholesale, like they do with the real API ban. They don't consider it abuse, they just enforce a server-side limit. 

Link to comment
Share on other sites

  • 0

No. All releases of CloudDrive, at least since 1.0, will behave the same way. It will simply keep retrying and eventually Google will let it upload data. It won't dismount, there is no reason for it to dismount. It's just an upload error. All of the data is still accessible--including the data in the cloud. As long as the cache drive isn't full, you can still read and write to the drive as normal, it just won't be able to update the data in the cloud until Google permits it to again. You *may* run into problems if your cache drive fills up completely, but even then, I don't think it'll disconnect. I'm not sure on that, though. Writes to the drive get so slow once CloudDrive starts throttling them (when the drive has, I believe, less than 5GB of space left) that I'm not even sure you can fill the cache drive in a 24 hour period!

 

I believe what you say, but I guess I'm just confused by what I experienced then. My first day using this software, I downloaded the stable release, copied over about half a terabyte and left it running. At some point, I looked at it, there was an error in the Windows copy dialog, the drive had been detached, and there was an error screen up on the CloudDrive UI. I don't remember exactly what it said, but it was  something to do with gdrive and from what I recall, it sounded like some issue with the API or uploading. It must've been like that all night, and by that point I was able to reattach the drive and continue uploading by telling the prompt in the copy window to resume.

 

A few other things:

  • If the upload requests are what's being denied, why do I sometimes see the throttling symbol (the turtle thing) when it's downloading?
  • How much needs to be in the cache for it to start uploading? Sometimes I see it upload with only a hundred megabytes or so, but sometimes I have dozens of gigabytes sitting in there and it sits at 0 bps for like half an hour or something.
  • I mentioned this in the other thread, but what's this "thread was being aborted" error that occurs over and over intermittently but I can resume uploading almost immediately after (therefore must not be an API/upload related ban). I never saw this once on the main release and only started seeing it once I switched to the beta.

Sorry, I'm pretty new to all this. I'm trying to figure out how everything works and find a pattern to it, but some things seem really erratic and it's confusing me big time. . . Thanks for taking the time to answer all my questions. :P

Link to comment
Share on other sites

  • 0

 

 

I believe what you say, but I guess I'm just confused by what I experienced then. My first day using this software, I downloaded the stable release, copied over about half a terabyte and left it running. At some point, I looked at it, there was an error in the Windows copy dialog, the drive had been detached, and there was an error screen up on the CloudDrive UI. I don't remember exactly what it said, but it was  something to do with gdrive and from what I recall, it sounded like some issue with the API or uploading. It must've been like that all night, and by that point I was able to reattach the drive and continue uploading by telling the prompt in the copy window to resume.

 

I believe the application can dismount the drive for a number of reasons, all of which, I believe, relate to consistent connection errors (reaching out to servers and getting no response, as opposed to the API saying "I'm not going to do that right now, try again later"). The most *common* of these are the consecutive I/O errors, as far as I know. You can adjust this condition by changing the default value of "3" to something higher using the CloudFsDisk_MaximumConsecutiveIoFailures setting in the advanced settings. I've had pretty good luck coping with typical network quirks on Google Drive with settings between 5 and 8, but I wouldn't go much higher than 10 or so, at the most, because it can cause a system lockup in case of a genuine problem. In any case, none of this has anything to do with the API banning you or throttling your downloads. 

 

All of this being said, it's still a little unclear to me *why* we get these I/O errors. Perhaps Christopher can shed some like on that, or maybe only Alex knows. I don't know. But adjusting this setting can help solve frequent mounting issues, if you're having them.

 

 

 

  • If the upload requests are what's being denied, why do I sometimes see the throttling symbol (the turtle thing) when it's downloading?

 

 

Because you're being throttled. The upload threshold API rejections are not the only reason you might be throttled. You may also be throttled because CloudDrive has had to make too many API calls per second and Google has requested exponential backoff, which CloudDrive complies with. So you'll see that symbol until Google stops asking CloudDrive to back off. That symbol doesn't indicate that you've been banned or that you've hit a threshold. It just indicates that your requests are being throttled. Most of the time that's temporary and it'll clear up in a few seconds or minutes. It's just a coincidence that it also indicates when you've hit your upload threshold, because CloudDrive does the same thing when you've hit your threshold that it does for any other API rejection from Google's servers: it delays for an amount of time, tries again, delays for even longer, tries again, delays for even longer, tries again, etc...until it works. That's called exponential backoff and that functionality existed before Google even implemented this new threshold. 

 

 

 

  • How much needs to be in the cache for it to start uploading? Sometimes I see it upload with only a hundred megabytes or so, but sometimes I have dozens of gigabytes sitting in there and it sits at 0 bps for like half an hour or something.

 

 

That's an adjustable setting, actually. See the "upload threshold" setting in the i/o performance settings menu. It can be adjusted from "off" to 10MB. What I suspect you're noticing, though, is that it sometimes just seems to pause. I don't know the technical details behind why that is. It is not, in any case, anything to worry about. CloudDrive does not seem to be designed to require a constant upload. It can (and does) do a lot of work on the cache drive. It'll upload the data when it needs to, and it'll keep functioning as normal until it does. I've noticed, on a few occasions, that heavy drive usage (like copying data into the CloudDrive) tends to make it pause. Again, I don't know why though.

 

 

 

  • I mentioned this in the other thread, but what's this "thread was being aborted" error that occurs over and over intermittently but I can resume uploading almost immediately after (therefore must not be an API/upload related ban). I never saw this once on the main release and only started seeing it once I switched to the beta.

 

I don't know what causes them, but they've been around long before recent BETAs.

 

 

 

Sorry, I'm pretty new to all this. I'm trying to figure out how everything works and find a pattern to it, but some things seem really erratic and it's confusing me big time. . . Thanks for taking the time to answer all my questions.

 

 

You're fine. If I'm being honest, I think CloudDrive's error messages are a bit vague, and that's causing you to over-worry about things you can probably ignore. The application isn't very clear about *what* is causing the errors that you sometimes see. But as long as you aren't getting warnings and errors in the feedback notifications (upper left corner of the UI), and the drive is working, I wouldn't worry about all of the little notes that show up in the technical details and such. It'll let you know if there is something broken that needs to be addressed. Maybe Chrisopher can add to this for you. I'm not sure.

 

I'm happy to help, though. I've done a lot of playing around, so I'm glad to share what I've learned. 

 

EDIT: Google also sometimes just has issues. I don't know if that's your provider, but any provider, I would imagine, is subject to the occasional hiccup. Sometimes Google just isn't working right, and your drive dismounts or shows a bunch of errors. But there isn't anything you can do about that. And, in any case, it shouldn't cause any long-term problems. It'll pick back up when things are fixed. 

Link to comment
Share on other sites

  • 0

Thanks again for the information. I can live with all the errors and such. Honestly the upload pausing issue bothers me the most just cuz I don't have a lot of room for cache and I'm trying to get everything online as quickly as possible. It's weird cuz when it does pause for a while, it keeps intermittently jumping to 16.8 mbps up and then stops again. Always exactly 16.8 mbps.

 

EDIT: Just confirming what you said earlier. it's definitely not the setting. for example, there's 81GB sitting in my "To upload" cache right now and it's jumping beween going off at a few hundred mbps to pausing entirely (except for those 16.8mbps spikes).

EDIT 2: You mentioned some advanced settings. Couldn't find any info about this online, but did some digging in the filesystem and found "C:\Program Files\StableBit\CloudDrive\CloudDrive.UI.exe.config". I'm guessing that's what you're talking about as the tag you mentioned is in there. Any other settings worth looking at in there?

Link to comment
Share on other sites

  • 0

It looks like I just hit the Google drive limit too.  I'm currently initializing backups, so there is alot going up now, but it wont be that way all the time.  I was wondering if CloudDrive could track daily upload size and if you get within X % of the cap, start throttling based on time left in the day, and what speed would keep you under the limit.  I dont want to throttle the upload if I am not going to meet the limit on a particular day...

Link to comment
Share on other sites

  • 0

I'm hitting also hitting the limits daily, as I just got gigabit fiber. I'm trying to backup 15TB in my Gdrive but I've only gotten upto 3TB so far. What would be the best way to go about this? After the backup I should be fine as I never upload more than 20GB-50GB Daily. I tried entering a upload throttling amount but that doesn't really work. I've tried upload threshold but that only goes to 10mb. I tried to enter 740GB, but it say the max is 10mb. Do I put 1440 Minutes in a day? Do I change the size of my cache? I've set that to 1TB. 

Link to comment
Share on other sites

  • 0
5 hours ago, GdriveUnknown said:

I'm hitting also hitting the limits daily, as I just got gigabit fiber. I'm trying to backup 15TB in my Gdrive but I've only gotten upto 3TB so far. What would be the best way to go about this? After the backup I should be fine as I never upload more than 20GB-50GB Daily. I tried entering a upload throttling amount but that doesn't really work. I've tried upload threshold but that only goes to 10mb. I tried to enter 740GB, but it say the max is 10mb. Do I put 1440 Minutes in a day? Do I change the size of my cache? I've set that to 1TB. 

Your upload threshold is the point at which CloudDrive will upload new data to the cloud. It isn't a limit, it's a delay. I think you're sorta confusing what that setting does. Your upload threshold says "start uploading new data to the cloud when I have X amount of new data, or it has been Y minutes since the last upload--whichever comes first." It is not a limit that says, "upload X amount of data, then stop."

The uploading throttling, on the other hand, is the setting that will limit your upload speed. And it works fine to keep your usage below Google's API limits. So I'm not sure what you mean that it doesn't really work. I use it every day. A throttle limit of roughly 70mbps will keep you below Google's 750GB/day limitation.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...