Google Drive: The limit for this folder's number of children (files and folders) has been exceeded

[KiLLeRRaT]

Hi All,

I've recently noticed that my drive has 40GB to be uploaded, and saw there are errors in the top left.  Clicking that showed 

Quote

The limit for this folder's number of children (files and folders) has been exceeded

What's the deal here, and is this going to be an ongoing issue with growing drives?

My drive is a 15TB drive, with 6TB free (so only using 9TB).

Attached is my drive's info screen.

EDIT: Cloud Drive Version: 1.1.5.1249 on Windows Server 2016.

 

Cheers,

[Jellepepe]

I had the wisdom 2 years ago when one of the ssds in my server corrupted and took almost a full 50TB of data in a clouddrive with it, that i would mirror all my data over multiple clouddrives and multiple google accounts, i am very happy with my past self at this stage.

I will start a staged rollout of this process on my drives and keep you updated if i find any issues.

[ezek1el3000]

1 hour ago, Chase said:

...

this morning I awoke with back pain and two of my drives mounted and functioning. The original drive which had completed the upgrade, had been working, and then went into "drive initializing"...is now working again. The drive that I had tried to mount and said it was upgrading with no % given has mounted (FYI 15TB Drive with 500GB on it). However my largest drive is still sitting at "Drive queued to perform recovery".

Did you use your own API keys or the default ones?

[Burrskie]

2 hours ago, Chase said:

Unintentional Guinea Pig Diaries.

Day 8 - Entry 1

I spent the rest of yesterday licking my wounds and contemplating a future without my data. I could probably write a horror movie script on those thoughts but it would be too dark for the people in this world. I must learn from my transgressions. In an act of self punishment and an effort to see the world from a different angle I slept in the dogs bed last night. He never sleeps there anyways but he would never have done the things I did. For that I have decided he holds more wisdom than his human. This must have pleased the Data God's because this morning I awoke with back pain and two of my drives mounted and functioning. The original drive which had completed the upgrade, had been working, and then went into "drive initializing"...is now working again. The drive that I had tried to mount and said it was upgrading with no % given has mounted (FYI 15TB Drive with 500GB on it). However my largest drive is still sitting at "Drive queued to perform recovery".

Maybe one more night in the dogs bed will complete the offering required to the Data God's

End Diary entry.

 

(P.S. Just in case you wondered...that spoiled dog has "The Big One" Lovesac as a dog bed.. In a pretty ironic fashion their website is down. #Offering) 

Are you using your own api key or the api key from stablebit?

[Chase]

1 hour ago, ezek1el3000 said:

Did you use your own API keys or the default ones?

Uh. Yes.

When this process first started I was on the default API. At that time I had two drives going through the update. In the middle of that process I added my own API but since you aren't able to reauthorize during the update it didn't actually switch to my API. When my first drive finished I was able to reauth. My second drive that is still having issues has never been able to reauth so my understanding is that it is still on the default API. The third drive that I mounted and said it was updating without showing a % was after I added my own API and therefore authorized on my API.

[Firerouge]

I'm still holding off patiently on the conversion, it sounds like it works, but waiting to get a better idea at the of the time it takes by drive data size.

 

I've noticed that without any changed settings these past few days I've gotten a couple yellow I/O error warnings about user upload rate limit exceeded (which otherwise haven't been problems), and I've noticed gdrive side upload throttling at a lower than normal concurrency, only 4 workers at 70mbit.

 

I'm guessing some of these rate limit errors people may be seeing in converting are transient from gdrive being under high load.

[Chase]

I may have fixed my drive that is stuck in "queued for recovery". But damn did it make me nervous and I'm far from out of the woods. In my understanding of the system the "cache" is just a local file of your pinned data and a COPY of the last #GB of data that was received or transferred between your machine and the cloud drives. It also seems that it holds the information for the CloudDrive program to say...yep this is a drive that we have mounted. So I shut down the windows service for CloudDrive and deleted the cache file for my problem drive. After restarting my computer when CloudDrive opened up it only showed the two working drives as mounted and the problem drive was available to mount. Sweet! Mount that sucker.

Seems to be working. It now says that the drive is upgrading and I get a percentage actually shown. Albeit it is way slower compared to last time I went through this stage. Probably because this latest Beta has less I/O interactions. It did NOT give me the option to reauth so I assume this drive is still using the StableBit API.

At this rate it should complete by Monday-ish? 

[ezek1el3000]

6 minutes ago, Chase said:

...

and deleted the cache file for my problem drive

nice find. what is that file called? and where can i find it?

Right now 2/6 drives are queued to perform recovery and the rest is in "Drive cannot be mounted" mode

Edith: did you refer to the cache files stored at the designated CD cache drive? If so, there are over 100 files. How did you know which one to delete?

Edited July 2, 2020 by ezek1el3000
reasons

[trillex]

Decided to try for the beta as well, now that I have holiday to babysit the process.

2 drives started doing the upgrade (Drive is upgrading), 2 are stuck in "Drive initializing" and the 5 others seemed to work fine.

Due to Windows updates, I restarted the server after checking this thread and after seeing some weird issues (the mounted ones started saying it can't find the file when looking in performance settings, I forgot the phrasing of it) and found a bit of a grizzly sight.

The 5 drives that were "fine" are now unable to be mounted and my cache drive is completely filled to the brim (44k free). This could be due to some automation filling the earlier mounted drives, but I have never seen it be filled below the 5 GB limit. Now, they most likely cannot be mounted, hopefully, due to no space left on the cache and thus not possible to write to it, but as I don't have a functioning drive or any means to actually empty the cache drive, I'm stuck in a bit of a place where I am unsure if everything is lost or I just have to be patient.

Log says: [CloudDrives] Cannot mount cloud part xyz. It was force unmounted.

I'll try for patience for now.

So, be careful, out there.

[srcrist]

1 hour ago, Chase said:

In my understanding of the system the "cache" is just a local file of your pinned data and a COPY of the last #GB of data that was received or transferred between your machine and the cloud drives.

The cache also includes information that has been modified but not yet uploaded. Everyone should be *very* careful before simply deleting the local cache. Note that any modifications that have not yet been uploaded to the cloud provider will be permanently lost, and you could potentially corrupt your drive as a result. I believe that you are theoretically correct as long as none of the information in the cache has yet to be uploaded, but extreme caution should be used before following this example. Anyone who deletes their cache with data in the "to upload" state will, A), definitely lose that data for good and, B), potentially corrupt their drive depending on what that data is (read: file system data, for example).

Edited July 2, 2020 by srcrist

[srcrist]

4 minutes ago, trillex said:

The 5 drives that were "fine" are now unable to be mounted and my cache drive is completely filled to the brim (44k free). This could be due to some automation filling the earlier mounted drives, but I have never seen it be filled below the 5 GB limit.

It sounds like we might be stumbling on some buggy cache code, between this and the previous notes from Chase about deleting the cache. Make sure you guys are submitting actual tickets and troubleshooter dumps as well, so Alex and Christopher can take a look at the related code and your logs.

[Chase]

1 hour ago, ezek1el3000 said:

nice find. what is that file called? and where can i find it?

Right now 2/6 drives are queued to perform recovery and the rest is in "Drive cannot be mounted" mode

Edith: did you refer to the cache files stored at the designated CD cache drive? If so, there are over 100 files. How did you know which one to delete?

To answer your question..I deleted the main Cache file which for me is stored on my C:/ Drive. It starts with Clouddrive and is followed by your drive UID.

 

as SRCRIST said....it is very risky and my drive may be totally corrupt when it finishes. That was a risk I was okay with taking. You make your own decision.

 

I did get a message back from Christopher on the Covecube team. They looked at my path of issues and whats happened plus troubleshooter data and his message was this...

"We have found a bug in the migration code, that may be the problem here.We should have a fix out soon, that should be much faster (or even instantaneous) and handle failure better. "

 

[sausages]

New update: now at 67% (on an 8TB drive) and the web UI has been showing files being moved as well. So there is hope yet.

[Firerouge]

New beta looks to have major improvements to the migration process, make sure you're on it before reporting any additional bugs or doing something crazy like delete local cache.

.1307
* Added detailed logging to the Google Drive migration process that is enabled by default.
* Redesigned the Google Drive migration process to be quicker in most cases:
    - For drives that have not run into the 500,000 files per folder limit, the upgrade will be nearly instantaneous.
    - Is able to resume from where the old migration left off.
* [Issue #28410] Output a descriptive Warning to the log when a storage provider's data organization upgrade fails.

 

[sausages]

2 hours ago, Firerouge said:

New beta looks to have major improvements to the migration process, make sure you're on it before reporting any additional bugs or doing something crazy like delete local cache.

.1307
* Added detailed logging to the Google Drive migration process that is enabled by default.
* Redesigned the Google Drive migration process to be quicker in most cases:
    - For drives that have not run into the 500,000 files per folder limit, the upgrade will be nearly instantaneous.
    - Is able to resume from where the old migration left off.
* [Issue #28410] Output a descriptive Warning to the log when a storage provider's data organization upgrade fails.

 

Upgraded to this version and my drive instantly mounted. If I understand the logs correctly, the new version simply doesn't bother moving chunks if the drive is under the 490k limit? I'm assuming upgrading part way through the prior version moving the files hasn't broken anything but I am entirely not sure.

[Chase]

Upgraded to the latest beta this morning. That drive I was having issues with and had deleted the cache was showing about 15% completed at the time of upgrade. The percent started over but it only took about 35 minutes to complete and mount my drive. (56TB with 20TB of data). Everything works on all drives except that annoying error when I try to change the I/O settings I talked about previously. Again...the changes take it just pops up with an error that the drive isn't mounted.
 

I've got family in for the next several days. Be safe guys. I'm very happy with the latest beta fix. All of my drives are functional.

 

[ezek1el3000]

The .1307 update seems to work better. One out of six drives is back and usable, i/o settings error included. Drives 2-4 are still in "queued to perform recovery" mode. Drive 6 cannot be mounted.

[Simon83]

I use .1307 and i am by 19% after about 24h. Never had a error like this, but I had nothing better to do, than update to the latest beta and know I hate myself.

You think, that deleting the cache would speed it up?

[JulesTop]

20 minutes ago, Simon83 said:

I use .1307 and i am by 19% after about 24h. Never had a error like this, but I had nothing better to do, than update to the latest beta and know I hate myself.

You think, that deleting the cache would speed it up?

I would not delete the cache. I have a 270TB drive with 55TB used and on 1307 I am upgrading at about 10% per 24 hour period. Which is faster than before.

I would stay the course if I were you.

[Simon83]

1 hour ago, JulesTop said:

I would not delete the cache. I have a 270TB drive with 55TB used and on 1307 I am upgrading at about 10% per 24 hour period. Which is faster than before.

I would stay the course if I were you.

Ok, then I will be patient.

Fyi, I tried to set the GoogleDrive_ConcurrentRequestCount to 3, but after a while I had errors from google...so 2 seams to be the maximum.

[Firerouge]

3 hours ago, JulesTop said:

I would not delete the cache. I have a 270TB drive with 55TB used and on 1307 I am upgrading at about 10% per 24 hour period. Which is faster than before.

I would stay the course if I were you.

Whoa, that's way slower than I expected. You're seeing only about 5.5TB migrated per day!

What sort of system specs, or resource consumption are you seeing, does it seem bottlenecked by anything other than Google's concurrency limit?

[Simon83]

6 minutes ago, Firerouge said:

Whoa, that's way slower than I expected. You're seeing only about 5.5TB migrated per day!

What sort of system specs, or resource consumption are you seeing, does it seem bottlenecked by anything other than Google's concurrency limit?

Because there are no files beein moved on your disk, the system specs doesn´t matter. Clouddrive just sends commands to the cloud, to move files on the cloud.

At least this is my understanding of this.

[ezek1el3000]

I am back to "Drive cannot be mounted". 3/6 Drives are in that particular stage. 2 are mounted, available but not accessible. One is in recovery mode. I can't take any more of this tension

[Burrskie]

6 hours ago, JulesTop said:

I would not delete the cache. I have a 270TB drive with 55TB used and on 1307 I am upgrading at about 10% per 24 hour period. Which is faster than before.

I would stay the course if I were you.

I have been seeing about the same speed, roughly 10% per day unfortunately. Although mine is now showing progress, which is better than before.

[Burrskie]

My drive went back to not showing a percentage either, so now I am back to both of my drives being in the "queue" to be recovered with neither of them currently being recovered from what I can tell.

[Chase]

All of my drives are working and I have no errors...EXCEPT the last beta that changed the GoogleDrive_ConcurrentRequestCount to "2" appears to be causing issues. My drives tend to do a lot of reading and writing...it's for a plex server. And 2 concurrent causes CloudDrive to tell me I don't have enough bandwidth. I have tried to change the setting in the "settings.json" but it isn't accepting the change.