Edrock200

Thanks. I think I figured it out my issue. First, I reattached the drive with 20mb min download limit set. Initially this still did not help. Next, and I could be wrong on this, but prefetch does not appear to respect the max thread setting. Ie if I set it to 5 max threads, and had prefetch set to 100mb in 20mb chunks, any prefetch calls would cause a connection spike to 10x. I also found that my Google drive client, which was fully synced, had reinitiated about 1tb of download, due to the fact that my other pc with Google drive had a file scan operation "touch" all the files thereby modifying the file date, those this PC was trying to download these "new" files. It seems that Google not only throttles by api id, but also by ip. It seems to start throttling everything when sustained traffic to/from one source stays steady over 600mbs. I noticed when I paused Google drive, clouddrive jumped back up to full speed with prefetch working great. Then when i unpaused Google drive, it took nearly an hour to download a 1gb file, as though Google drive was now throttling. And this wasn't a bandwidth constraint on my end, I have 1gig link to the internet with speed tests running about 900/880mbs, and with both running I was at about 250 up and 400 down. To resolve this my final settings are as follows, but please note my drive content is primarily large video media: Min download 20mb Download chunk cache 100mb Download/upload throttle cloud drive 250mbs Download/upload threads 5/5 Prefetch 10mb/100mb/120s Google drive download throttle 99mbs (max throttle value allowed) From what I can tell, if Google sees the same IP pushing and/or pulling a sustained rate greater than 600mbs it gets much more strict on throttling and threads. The above settings seem stable now and I even tested 5 simultaneous hd streams without issue. But launching the Google music sync client, which literally just opened a few file comparison threads, caused the numbers to drop almost instantly. Closed it, and they went right back up. Throttled it to 10mbs and all was happy again. I do believe the prefetch threading issue should be given a closer look though. I specifically set mine to 100mbs because with 20mb download chunks, each prefetch will cause a Max of 5 new threads. Couple that with the existing Max of 5 dl threads and I'm still in a reasonable thread range of 10. When I tested 200mbs, it still worked, but the first 100 prefetched almost instantly, then downloads started to slow and the last 100 came in throttled. Tested 300 and it made it to 200, then slowed to the single digits, got to 360, slowed to kilobit range, then my drive disconnected. This is all very unscientific and just my experience. The other tweak I did was turn off windows built in drive indexing for the cloud drive to minimize "idle thrashing." The good news for me is my initial copy to the cloud drive is 95% complete, and should finish this weekend so all this bandwidth excess and throttling issues should subside and future writes will only be a few gb/day. Thanks all for the info and advice.

Thanks for that explanation. If I'm understanding it correctly, I think many who have been using clouddrive for plex do not understand the time window setting. There are numerous "tutorial" articles that suggest setting the time window to 1mb/400mb/3600seconds, which would cause massive, neverending prefetching. The tutorial writers seem to think the time window is how long the prefetch data will be stored in cache for. Did you get a chance to look at the logs I uploaded to see why prefetching causes things to throttle? Right now, if I turn prefetching on with any settings, even the defaults, my drive will disconnect within 5 to 10 minutes with an error that it cannot read from the provider.

Sweet. Thanks. I'll try that

Thank you. Where is that set? I see prefetch trigger but I though that's the threshold that initiates prefetching

Chris, Not sure what you mean by posted in a ticket, I posted it via the dropbox link you provided. How do I see your response in the ticket? I have kept prefetch off, as the drive is unusable when its on. It works ok with prefetch off but I still hit a lot of buffering, so I'm trying to get prefetch working to alleviate this.

fwiw, heres a snip of the service logs as well. starts with prefetch on, I turn it off around 17:18 17:16:52.3: Warning: 0 : [ApiGoogleDrive:132] Google Drive returned error (downloadQuotaExceeded): The download quota for this file has been exceeded. 17:16:52.3: Warning: 0 : [ApiHttp:132] HTTP protocol exception (Code=Forbidden). 17:16:52.3: Warning: 0 : [ioManager:132] HTTP error (Forbidden) performing I/O operation on provider. 17:16:52.3: Warning: 0 : [ioManager:132] Error performing I/O operation on provider. Retrying. The download quota for this file has been exceeded. 17:17:08.8: Warning: 0 : [ApiGoogleDrive:114] Google Drive returned error (downloadQuotaExceeded): The download quota for this file has been exceeded. 17:17:08.8: Warning: 0 : [ApiHttp:114] HTTP protocol exception (Code=Forbidden). 17:17:08.8: Warning: 0 : [ioManager:114] HTTP error (Forbidden) performing I/O operation on provider. 17:17:08.8: Warning: 0 : [ioManager:114] Error performing I/O operation on provider. Retrying. The download quota for this file has been exceeded. 17:17:18.6: Warning: 0 : [ApiGoogleDrive:129] Google Drive returned error (downloadQuotaExceeded): The download quota for this file has been exceeded. 17:17:18.6: Warning: 0 : [ApiHttp:129] HTTP protocol exception (Code=Forbidden). 17:17:18.6: Warning: 0 : [WholeChunkIoImplementation:129] Error on read when performing master partial write. The download quota for this file has been exceeded. 17:17:18.6: Warning: 0 : [WholeChunkIoImplementation:129] Error when performing master partial write. The download quota for this file has been exceeded. 17:17:18.6: Warning: 0 : [ApiGoogleDrive:67] Google Drive returned error (downloadQuotaExceeded): The download quota for this file has been exceeded. 17:17:18.6: Warning: 0 : [ApiHttp:67] HTTP protocol exception (Code=Forbidden). 17:17:18.6: Warning: 0 : [ioManager:129] HTTP error (Forbidden) performing I/O operation on provider. 17:17:18.6: Warning: 0 : [ioManager:129] Error performing I/O operation on provider. Retrying. The download quota for this file has been exceeded. 17:17:18.7: Warning: 0 : [WholeChunkIoImplementation:67] Error on read when performing master partial write. The download quota for this file has been exceeded. 17:17:18.7: Warning: 0 : [WholeChunkIoImplementation:67] Error when performing master partial write. The download quota for this file has been exceeded. 17:17:18.7: Warning: 0 : [ioManager:67] HTTP error (Forbidden) performing I/O operation on provider. 17:17:18.7: Warning: 0 : [ioManager:67] Error performing I/O operation on provider. Retrying. The download quota for this file has been exceeded. 17:17:21.5: Warning: 0 : [ApiGoogleDrive:108] Google Drive returned error (downloadQuotaExceeded): The download quota for this file has been exceeded. 17:17:21.5: Warning: 0 : [ApiHttp:108] HTTP protocol exception (Code=Forbidden). 17:17:21.5: Warning: 0 : [ioManager:108] HTTP error (Forbidden) performing I/O operation on provider. 17:17:21.5: Warning: 0 : [ioManager:108] Error performing I/O operation on provider. Retrying. The download quota for this file has been exceeded. 17:17:38.5: Warning: 0 : [ApiGoogleDrive:114] Google Drive returned error (downloadQuotaExceeded): The download quota for this file has been exceeded. 17:17:38.5: Warning: 0 : [ApiHttp:114] HTTP protocol exception (Code=Forbidden). 17:17:38.5: Warning: 0 : [ioManager:114] HTTP error (Forbidden) performing I/O operation on provider. 17:17:38.5: Warning: 0 : [ioManager:114] Error performing I/O operation on provider. Retrying. The download quota for this file has been exceeded. 17:18:31.1: Information: 0 : [ioManager:3] All writes complete. 17:19:05.6: Information: 0 : [ioManager:3] All writes complete.

Done. Both submitted

Hey all, I've run into an odd issue with CloudDrive and PreFetch. Essentially, with prefetch off, I get 300MB/s up and ~150MB/s down (when something is being pushed/pulled.) when I turn prefetch on, the upload downloads come to a screeching halt, uploads bob up and down from 0 to 15mb's and downloads generally stay under 1mb/s although they may occasionally go up to 8mb/s for a few seconds. Once I turn prefetch off and the time expires, the numbers spike back to their normal throughput. Here are my settings: Google Drive attached 256TB AES Encrypted mount 10TB in use 20MB chunks 10MB/1 minute upload trigger 500/500 Internet with Gig-E link Upload/download threads set to 5/5 (have tried 3/3 and 10/10 as well with same results.) Background I/O checked Prefetch - Trigger 1MB, Forward 400MB, Time 1800 Seconds (tried 10/100/600, 1/1000/3600 with same results) Upload Download limited to 250MB/250MB, although tried removing these limits for testing, same results 30GB expandable cache set on local SSD drive (tried increasing, reducing, changing to fixed or dynamic with same results) Upload queues are at 0 prior to prefetch being enabled, but once enabled they start to grow due to the speed decrease One thing I've noticed, I'f I'm understanding the summary screen correctly, is above the upload and download arrows it will show numbers like 3x or 5x. I'm assuming this is the number of threads being utilized. With prefetch off, this number never exceeds my max i/o settings for upload/download. However, with prefetch turned on, the download threads will go up, I've seen it go up to 12x for example. Does prefetch not respect the max i/o settings or am I missing something? Also, the prefetch fills slowly. Is the prefetch duration saying "if 1mb of same file is read, fetch next 400MB steadily/evenly spread over the next 1800 seconds" or is it saying "if 1mb of same file is read, fetch next 400MB as fast as possible keep for 1800 seconds."? Also, what is the significance of the "Background I/O" checkbox next to upload threads? What does turning this option off and on do? Thanks in advance, -Ed

About 4 hours after my previous post, the drive did mount successfully. Ironically, the server bug-checked at some point during the night, causing a "dirty dismount." Upon restarting this morning, the recovery remount completed within about 15 minutes. So far all is good. I'm not sure if this is related, but anytime I've tried to manually "detach" the drive, it never fully completes. It will say detaching, then go back to "green" like it's mounted. However, going to "My Computer" the drive just shows as "NTFS" with no size/used" data. Trying to open it says "Access Denied." This occurred yesterday just prior to the long mounting process, so I'm guessing they are somehow related. I saw in another thread you recommended ver 857 to potentially solve the dismount problem, so I've installed it to see if it helps.

Wow I didn't know that! Was contemplating buying another one or two cloud drive licenses, this makes the decision much easier. Side note to the OP. To work around the one-pc|one drive limitation, I installed "sync thing" on my nodes. It is free and works sort of like a dropbox or google drive client but without the central cloud storage piece, all data syncs and resides solely between your equipment. So I can sync directly between nodes, or even do one way syncs, exclude files, etc. It's very customizable.

I just ran into this as well. Oddly enough, the last two times my system crashed, causing a dirty disconnect and recovery on restart, this process completed within 5 minutes or so. This time, it was a clean detach, shutdown, reboot, reattach, but it's been running for over an hour. Services are running, and service log just keeps cycling different enumerations of the snippet below, but each cycle has different chunk id's. So, I'm assuming it's doing what it's supposed to, but confused as to why a clean detach/attach would cause this to kick off.: 1:04:49.3: Information: 0 : [ChunkIdSQLiteStorage:34] chunks IDs: 317205,317207,317208,317209,317210,317211,317212,317213,317214,317215,317216,317217,317218,317219,317220,317221,317222,317223,317224,317225,317226,317227,317228,317229,317230,317231,317232,317233,317234,317235,317236,317237,317238,317239,317240,317241,317242,317243,317244,317245,317246,317247,317248,317249,317250,317251,317252,317253,317254,317255,317256,317257,317258,317259,317260,317261,317262,317263,317264,317265,317266,317267,317268,317269,317270,317271,317272,317273,317274,317275,317276,317277,317278,317279,317280,317281,317282,317283,317284,317285,317286,317287,317288,317289,317290,317291,317292,317293,317294,317295,317296,317297,317298,317299,317300,317302,317304,317305,317306,317308