Google Drive: The limit for this folder's number of children (files and folders) has been exceeded

[KiLLeRRaT]

Hi All,

I've recently noticed that my drive has 40GB to be uploaded, and saw there are errors in the top left.  Clicking that showed 

Quote

The limit for this folder's number of children (files and folders) has been exceeded

What's the deal here, and is this going to be an ongoing issue with growing drives?

My drive is a 15TB drive, with 6TB free (so only using 9TB).

Attached is my drive's info screen.

EDIT: Cloud Drive Version: 1.1.5.1249 on Windows Server 2016.

 

Cheers,

[Firerouge]

I'm guessing this latest beta changelog is referencing the solution to this

.1305
* Added a progress indicator when performing drive upgrades.
* [Issue #28394] Implemented a migration process for Google Drive cloud drives to hierarchical chunk organization:
    - Large drives with > 490,000 chunks will be automatically migrated.
        - Can be disabled by setting GoogleDrive_UpgradeChunkOrganizationForLargeDrives to false.
    - Any drive can be migrated by setting GoogleDrive_ForceUpgradeChunkOrganization to true.
    - The number of concurrent requests to use when migrating can be set with GoogleDrive_ConcurrentRequestCount (defaults to 10).
    - Migration can be interrupted (e.g. system shutdown) and will resume from where it left off on the next mount.
    - Once a drive is migrated (or in progress), an older version of StableBit CloudDrive cannot be used to access it.
* [Issue #28394] All new Google Drive cloud drives will use hierarchical chunk organization with a limit of no more than 100,000 children per folder.

Some questions, seeing as the limit appears to be around 500,000, is there an option to set the new hierarchical chunk organization folder limit to something higher than 100,000?

Has anyone performed the migration yet, what is the approximate time it takes to transfer a 500,000 chunk drive to the new format? Seeing as there are concurrency limit options, does the process also entail a large amount of upload or download bandwidth?

After migrating, is there any performance difference compared to the prior non hierarchical chunk organization?

Edit: if the chunk limit is 500,000, and if chunks are 20Mb, shouldn't this be occurring on all drives over 10Tb in size?

Note, I haven't actually experienced this issue and I have a few large drives under my own api key, so it may be a very slow rollout or an A/B test.

[steffenmand]

4 hours ago, Firerouge said:

I'm guessing this latest beta changelog is referencing the solution to this

.1305
* Added a progress indicator when performing drive upgrades.
* [Issue #28394] Implemented a migration process for Google Drive cloud drives to hierarchical chunk organization:
    - Large drives with > 490,000 chunks will be automatically migrated.
        - Can be disabled by setting GoogleDrive_UpgradeChunkOrganizationForLargeDrives to false.
    - Any drive can be migrated by setting GoogleDrive_ForceUpgradeChunkOrganization to true.
    - The number of concurrent requests to use when migrating can be set with GoogleDrive_ConcurrentRequestCount (defaults to 10).
    - Migration can be interrupted (e.g. system shutdown) and will resume from where it left off on the next mount.
    - Once a drive is migrated (or in progress), an older version of StableBit CloudDrive cannot be used to access it.
* [Issue #28394] All new Google Drive cloud drives will use hierarchical chunk organization with a limit of no more than 100,000 children per folder.

Some questions, seeing as the limit appears to be around 500,000, is there an option to set the new hierarchical chunk organization folder limit to something higher than 100,000?

Has anyone performed the migration yet, what is the approximate time it takes to transfer a 500,000 chunk drive to the new format? Seeing as there are concurrency limit options, does the process also entail a large amount of upload or download bandwidth?

After migrating, is there any performance difference compared to the prior non hierarchical chunk organization?

 

Note, I haven't actually experienced this issue and I have a few large drives under my own api key, so it may be a very slow rollout or an A/B test.

i would guess chunks just gets moved and not actually downloaded/reuploaded! only upload is to update chunk db i suspect

[JulesTop]

5 hours ago, steffenmand said:

i would guess chunks just gets moved and not actually downloaded/reuploaded! only upload is to update chunk db i suspect

I'm going through the process now.
It started at 6:45am this morning and is currently at 2.62% complete... I think this will take some time. But I don't think it actually downloads/re-uploads the chunks, it just moves them.

I have 55TB in google drive BTW.

 

Also, as far as I can tell, the drive is unusable during the process, and there is no way of pausing the process to access the drive content. However, it will be able to resume from where it left off after a shutdown.

 

Having said all that, I have to say thanks to @Alex and @Christopher (Drashna) for mobilizing quickly on this, before it becomes a really serious problem. At least right now, we can migrate on our schedule, and until then there is a workaround with the default API keys.

[Paco]

Hi to all,

I have a problem (or at least looks like) with the new migration process.

The progress indicator keeps restarting every once in a while back to 0.00%

Is this normal? @Alex @Christopher (Drashna)

Thanks and kind regards

 

[kird]

Have any formal announcements been made by the developers? How much hurry to migrate to a new system with a beta version that you don't even know what it leads to... I do not intend to adopt this system, if that is the case, without first making an official announcement by developers, with 130 TB I do not think it is a good idea.  That said, with my current version, which is by no means the last stable one officially released, I have no problem with Gdrive at the moment, everything is perfect. To say that I have the standard google api. 

[steffenmand]

Is anyone having issues creating drives on .1305 ? 

It seems to create the drives, but are unable to display it in the application!

 

I also notice that all of you get a % counter on your "Drive is upgrading"... mine doesnt show that - is there something up with my installation?

EDIT: A reboot made the new drive appear, but i can't unlock it. Service logs is spammed with:

0:02:04.0: Warning: 0 : [ApiHttp:64] Server is temporarily unavailable due to either high load or maintenance. HTTP protocol exception (Code=ServiceUnavailable).

 

EDIT 2:

After ouputting verbose i can see:

0:04:42.8: Information: 0 : [ApiGoogleDrive:69] Google Drive returned error (userRateLimitExceeded): User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=962388314550

As im not using my own API keys, the issue must be with Stablebit's

Did the drive updates make Stablebit's API keys go crazy?

[JulesTop]

2 hours ago, steffenmand said:

Is anyone having issues creating drives on .1305 ? 

It seems to create the drives, but are unable to display it in the application!

 

I also notice that all of you get a % counter on your "Drive is upgrading"... mine doesnt show that - is there something up with my installation?

EDIT: A reboot made the new drive appear, but i can't unlock it. Service logs is spammed with:

0:02:04.0: Warning: 0 : [ApiHttp:64] Server is temporarily unavailable due to either high load or maintenance. HTTP protocol exception (Code=ServiceUnavailable).

 

EDIT 2:

After ouputting verbose i can see:

0:04:42.8: Information: 0 : [ApiGoogleDrive:69] Google Drive returned error (userRateLimitExceeded): User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=962388314550

As im not using my own API keys, the issue must be with Stablebit's

Did the drive updates make Stablebit's API keys go crazy?

I was wondering if this would happen as more and more people begin the upgrade process...

The issue is, I don't think you can re-authorize the drive once the upgrade process begins... at least I don't see a way to do this. Otherwise, I would switch back to my own key.

[Chase]

On 6/24/2020 at 6:00 PM, JulesTop said:

I'm going through the process now.
It started at 6:45am this morning and is currently at 2.62% complete... I think this will take some time. But I don't think it actually downloads/re-uploads the chunks, it just moves them.

I have 55TB in google drive BTW.

 

Also, as far as I can tell, the drive is unusable during the process, and there is no way of pausing the process to access the drive content. However, it will be able to resume from where it left off after a shutdown.

 

Having said all that, I have to say thanks to @Alex and @Christopher (Drashna) for mobilizing quickly on this, before it becomes a really serious problem. At least right now, we can migrate on our schedule, and until then there is a workaround with the default API keys.

I have two drives with 10TB on one and 20TB on the other. I'm at about 17 hours in and one the smaller drive is at 58% and the bigger drive is at 30%.

 

How long did it take for you? Did everything work when it was done? I have a lot of large files (~7GB...media) but not many small files.

[JulesTop]

7 minutes ago, Chase said:

I have two drives with 10TB on one and 20TB on the other. I'm at about 17 hours in and one the smaller drive is at 58% and the bigger drive is at 30%.

 

How long did it take for you? Did everything work when it was done? I have a lot of large files (~7GB...media) but not many small files.

I'm currently at 12.02%, and I started on June 24th at 6:45am... so it's been over 2 days.

I have generally larger files... from 15GB to 60GB, but I'm not sure what affect that has.

[Chase]

1 hour ago, JulesTop said:

I'm currently at 12.02%, and I started on June 24th at 6:45am... so it's been over 2 days.

I have generally larger files... from 15GB to 60GB, but I'm not sure what affect that has.

I just restarted my computer...the percentage started over. Not sure if the whole process is starting again or if this is the percentage of what is left. FML

[steffenmand]

3 minutes ago, Chase said:

I just restarted my computer...the percentage started over. Not sure if the whole process is starting again or if this is the percentage of what is left. FML

My guess it is the % of the remaining. = it will increase faster than initially from every reboot.

 

Mine is stuck however with the user limit

[Chase]

42 minutes ago, steffenmand said:

My guess it is the % of the remaining. = it will increase faster than initially from every reboot.

 

Mine is stuck however with the user limit

What user limit are you referring to? It doesn't appear that it is downloading or uploading anything so it shouldn't effect the daily upload cap.

 

Thanks in advance.

[steffenmand]

Just now, Chase said:

What user limit are you referring to? It doesn't appear that it is downloading or uploading anything so it shouldn't effect the daily upload cap.

 

Thanks in advance.

I get a userlimitexceeded error. So API calls are not processed

[steffenmand]

so weird all of you see a %! i see nothing

[Paco]

36 minutes ago, steffenmand said:

so weird all of you see a %! i see nothing

Try to install the latest version 1305

[steffenmand]

23 minutes ago, Paco said:

Try to install the latest version 1305

Thats the one i got :-)

[steffenmand]

Just now, steffenmand said:

Thats the one i got :-)

 

[chaostheory]

I was seeing the percentage when I ran 1305 version for the first time, but after reboot it no longer shows progress/percentage. 

[Chase]

7 hours ago, JulesTop said:

I was wondering if this would happen as more and more people begin the upgrade process...

The issue is, I don't think you can re-authorize the drive once the upgrade process begins... at least I don't see a way to do this. Otherwise, I would switch back to my own key.

I was on the stablebit API key and it locked me out for the limits. So I created my own API key, saved it in the provider settings json, and reset my machine. The race is back on now.

 

it started the percent over again the same as it did when i restarted my PC earlier. And it's not climbing any faster as if the new percent level is "of what was left"...its more like it completely started over the full process.

[steffenmand]

1 hour ago, Chase said:

I was on the stablebit API key and it locked me out for the limits. So I created my own API key, saved it in the provider settings json, and reset my machine. The race is back on now.

 

it started the percent over again the same as it did when i restarted my PC earlier. And it's not climbing any faster as if the new percent level is "of what was left"...its more like it completely started over the full process.

What field does the key go in? And is it in ProviderSettings.json?

[Chase]

33 minutes ago, steffenmand said:

What field does the key go in? And is it in ProviderSettings.json?

That is where I put it.

 

Now upon closer look the log is still giving me the same error but it is no longer telling me it cant mount the drive and the percentage is going up still where it wasn't before.

[steffenmand]

33 minutes ago, Chase said:

That is where I put it.

 

Now upon closer look the log is still giving me the same error but it is no longer telling me it cant mount the drive and the percentage is going up still where it wasn't before.

Seems to make no difference here...

Same error and still no % counter. My guess is that you guys got to a certain point before the limit was reached, where mine didnt get started. Anyway i hope they will have a fix for this soon - just sucks that stuff like this always have to happen in weekends

[darkly]

Soo..... I'm still on version 1.1.2.1178 (been reluctant to upgrade with the various reports I keep seeing), but my gdrive has over 100TB of data on it and I've never seen the error about the number of children being exceeded. I use my own API keys. Have I just be extremely lucky up until this point?

[kird]

2 hours ago, darkly said:

Soo..... I'm still on version 1.1.2.1178 (been reluctant to upgrade with the various reports I keep seeing), but my gdrive has over 100TB of data on it and I've never seen the error about the number of children being exceeded. I use my own API keys. Have I just be extremely lucky up until this point?

my advice, don't even think about upgrading.  

[ezek1el3000]

too late... still upgrading; slowly. Do these instructions still apply? Use your own Google Drive API keys