srcrist
-
Posts
466 -
Joined
-
Last visited
-
Days Won
36
Posts posted by srcrist
-
-
The "cloud used" amount refers to the space used on your provider, not the amount of data stored on the drive. When you delete something on your CloudDrive drive, it doesn't remove the drive architecture that has already been uploaded from from the provider. So, if it's a 500GB drive and you've already created and uploaded 500GB worth of chunks to the provider, those will remain, just like a real drive, to be used (overwritten) later. This is why you can use recovery software to recover data on your CloudDrive drive just like a physical hard drive. If you want to remove the additional space, you'll need to shrink the drive, which you can do from the "manage drive" options.
-
I think the prefetch functionality remains a little unclear in the documentation. I'd love for Alex to give an account of how it works in more detail as well. It would help a lot with respect to optimization. One question that I still have is whether or not defragmenting the drive would help the prefetcher to obtain contiguous data while streaming. Based on my current understanding of how the process works, I think it would. But that's still a bit unclear. Perhaps Christopher could ask for a nuts and bolts post on the feature?
-
The window does not appear to function like that. You can change the window and watch the prefetched data get held in the UI for the time you set. Try setting the window to 30 minutes and set a timer, it'll hold that prefetched data for 30 minutes and then drop it. How would 1 MB in 30 seconds work? If it did work how you are saying, how do you access 1 MB in 30 seconds? It would take like a couple seconds at MOST to access that much data so it would never trigger that "window". The other issue with a low trigger size is that if Plex or any other program is analyzing file metadata, it triggers the prefetcher and clogs up it up even if it only needs to read parts of an entire file.I don't know what to tell you. It does function like that. Please see the documentation at https://stablebit.com/Support/CloudDrive/Manual?Section=I/O%20Performance. Your concerns are also misguided. It is a maximum window, not a required window. If it accesses 1MB during any time *less* than the window it will trigger the prefetch. So 1MB in 30secs is very easy to hit. An 8 mbps video file would trigger the prefetch in one second with a 1MB trigger; a 4mbps video file would trigger it in 2 secs.
Do you have any proof that the Transcoder default throttle buffer is CPU time? "Amount in seconds to buffer before throttling back the transcoder speed." is what the Plex site says and I can't find any other sources other than suggested amounts on the Rclone forums of 10 minutes. Even if it is CPU time, it tells Plex to keep trancoding additional data and the Plex client will buffer and cache it.You can see a Plex employee explain the setting here: https://forums.plex.tv/discussion/216069/verify-transcoder-throttle-buffer
It is, in any case, not a necessary setting to modify if your CloudDrive settings are correct. CloudDrive is plenty responsive enough, particularly on a 1gbps connection, to download the necessary data in chunks without affecting your streaming. Let the server throttle down to make room for another stream. It'll unthrottle when it needs to regardless, so you're not really gaining anything.
In any case, I am certainly getting better performance with these settings. I can see spikes of data streaming / trancoding when the large prefetch gets downloaded and then processed by Plex and sent to the client. I watched 4x 1080p streams get handled flawlessly. That being said, I do have gigabit and 20x 2 GHz cores on the server.You'd likely get even more performance if you optimized things correctly. My server is only an E3 1245v2 and it easily handles 5-8 simultaneous streams on a daily basis, and most of my media is stored in Remux quality (20-35mbps or so). Yours should be able to handle far, far more than 4 1080p streams with your specs.
I suggest these settings for Plex based on a lot of testing and experimentation:
Download threads: 10
Upload threads: 5
Upload threshold: Irrelevant, really, but 1MB/5mins
Minimum Download Size: 10MB
Prefetch trigger: 1MB
Prefetch Forward: 150MB
Prefetch time window: 30secs
20MB storage chunks on gdrive, largest RAM chunk cache and expandable cache you can accommodate with your setup.
That should provide you with a responsive drive with more than enough speed to stream basically anything you throw at it.
EDIT: Just saw that Christopher responded at the same time. His suggestion of a 10MB trigger is probably fine too. I'd imagine 1MB or 10MB makes little difference with high quality media. You want it to prefetch often for Plex, in any case.
-
The window is actually a threshold that triggers the prefetch if the trigger amount of data is requested within that time. For plex, I would leave your trigger at 1MB and the window at something like 30 secs. 150MB forward prefetch should be sufficient for even the highest bitrate video files.
You also should definitely lower that Plex transcoder throttle time. That's 10 minutes of CPU time, not 10 minutes of normal old clock time. That's *waaaaaay* too much. Leave it at like 60 secs to 120 secs.
-
I figured out how to grab 930, but the problem still persists. Even on a restart, it is stuck in an infinite loop "Synchronizing cloud drives..." and the UI bounces back and forth between "This drive is attempting recovery" and "This drive is connecting".
Submitted a ticket on this too.
I would just go ahead and do the ticket process to troubleshoot. Whatever problem that is, it isn't the same thing I was having. Mine was just enumerating the files every time the system rebooted.
-
You might be able to use something like security auditing to see how and when the file is deleted: http://www.monitorware.com/common/en/articles/audit_file_deletion.php
-
The new BETA (930) seems to have solved the enumeration problem on reboot. Just FYI, if anyone else was having that problem.
-
Yeah. I once had a windows update corrupt a 20TB ReFS drive. Lost a ton of data there too.
-
Whatever you do, if you want to recover the data, don't format it. Is the disk showing up as "RAW"? What was the format before? ReFS? NTFS?
-
CloudDrive just doesn't interact with the drive at the filesystem level though. It is not even, generally, aware of what files are being accessed on the drive. Is there some issue with the drive integrity? Does chkdsk report damage or anything? CloudDrive just creates a drive image. Files going missing without any sort of problem with the integrity of that image points to a problem outside of CloudDrive. It just doesn't work that way. If CloudDrive was losing data, it wouldn't be in the form of files. It would be random sectors and clusters of the drive.
-
CloudDrive doesn't really interact with files. What else is accessing your drive? My guess is that some automation software is deleting them.
-
I believe what you say, but I guess I'm just confused by what I experienced then. My first day using this software, I downloaded the stable release, copied over about half a terabyte and left it running. At some point, I looked at it, there was an error in the Windows copy dialog, the drive had been detached, and there was an error screen up on the CloudDrive UI. I don't remember exactly what it said, but it was something to do with gdrive and from what I recall, it sounded like some issue with the API or uploading. It must've been like that all night, and by that point I was able to reattach the drive and continue uploading by telling the prompt in the copy window to resume.I believe the application can dismount the drive for a number of reasons, all of which, I believe, relate to consistent connection errors (reaching out to servers and getting no response, as opposed to the API saying "I'm not going to do that right now, try again later"). The most *common* of these are the consecutive I/O errors, as far as I know. You can adjust this condition by changing the default value of "3" to something higher using the CloudFsDisk_MaximumConsecutiveIoFailures setting in the advanced settings. I've had pretty good luck coping with typical network quirks on Google Drive with settings between 5 and 8, but I wouldn't go much higher than 10 or so, at the most, because it can cause a system lockup in case of a genuine problem. In any case, none of this has anything to do with the API banning you or throttling your downloads.
All of this being said, it's still a little unclear to me *why* we get these I/O errors. Perhaps Christopher can shed some like on that, or maybe only Alex knows. I don't know. But adjusting this setting can help solve frequent mounting issues, if you're having them.
- If the upload requests are what's being denied, why do I sometimes see the throttling symbol (the turtle thing) when it's downloading?
Because you're being throttled. The upload threshold API rejections are not the only reason you might be throttled. You may also be throttled because CloudDrive has had to make too many API calls per second and Google has requested exponential backoff, which CloudDrive complies with. So you'll see that symbol until Google stops asking CloudDrive to back off. That symbol doesn't indicate that you've been banned or that you've hit a threshold. It just indicates that your requests are being throttled. Most of the time that's temporary and it'll clear up in a few seconds or minutes. It's just a coincidence that it also indicates when you've hit your upload threshold, because CloudDrive does the same thing when you've hit your threshold that it does for any other API rejection from Google's servers: it delays for an amount of time, tries again, delays for even longer, tries again, delays for even longer, tries again, etc...until it works. That's called exponential backoff and that functionality existed before Google even implemented this new threshold.
- How much needs to be in the cache for it to start uploading? Sometimes I see it upload with only a hundred megabytes or so, but sometimes I have dozens of gigabytes sitting in there and it sits at 0 bps for like half an hour or something.
That's an adjustable setting, actually. See the "upload threshold" setting in the i/o performance settings menu. It can be adjusted from "off" to 10MB. What I suspect you're noticing, though, is that it sometimes just seems to pause. I don't know the technical details behind why that is. It is not, in any case, anything to worry about. CloudDrive does not seem to be designed to require a constant upload. It can (and does) do a lot of work on the cache drive. It'll upload the data when it needs to, and it'll keep functioning as normal until it does. I've noticed, on a few occasions, that heavy drive usage (like copying data into the CloudDrive) tends to make it pause. Again, I don't know why though.
- I mentioned this in the other thread, but what's this "thread was being aborted" error that occurs over and over intermittently but I can resume uploading almost immediately after (therefore must not be an API/upload related ban). I never saw this once on the main release and only started seeing it once I switched to the beta.
I don't know what causes them, but they've been around long before recent BETAs.
Sorry, I'm pretty new to all this. I'm trying to figure out how everything works and find a pattern to it, but some things seem really erratic and it's confusing me big time. . . Thanks for taking the time to answer all my questions.You're fine. If I'm being honest, I think CloudDrive's error messages are a bit vague, and that's causing you to over-worry about things you can probably ignore. The application isn't very clear about *what* is causing the errors that you sometimes see. But as long as you aren't getting warnings and errors in the feedback notifications (upper left corner of the UI), and the drive is working, I wouldn't worry about all of the little notes that show up in the technical details and such. It'll let you know if there is something broken that needs to be addressed. Maybe Chrisopher can add to this for you. I'm not sure.
I'm happy to help, though. I've done a lot of playing around, so I'm glad to share what I've learned.
EDIT: Google also sometimes just has issues. I don't know if that's your provider, but any provider, I would imagine, is subject to the occasional hiccup. Sometimes Google just isn't working right, and your drive dismounts or shows a bunch of errors. But there isn't anything you can do about that. And, in any case, it shouldn't cause any long-term problems. It'll pick back up when things are fixed.
-
Is this behavior only on the beta? I was on the main release before and I remember the drive entirely dismounted my first night I left a transfer running.
Also can someone clarify some information on the ban? I've read that it's based off the number of API calls but I've also read that it's based on total bandwidth used. Even so I've heard that it's at around 750GB, but I've only managed an average of 500GB a day over the time I've been uploading. Are there two bans based on both API calls and bandwidth?
No. All releases of CloudDrive, at least since 1.0, will behave the same way. It will simply keep retrying and eventually Google will let it upload data. It won't dismount, there is no reason for it to dismount. It's just an upload error. All of the data is still accessible--including the data in the cloud. As long as the cache drive isn't full, you can still read and write to the drive as normal, it just won't be able to update the data in the cloud until Google permits it to again. You *may* run into problems if your cache drive fills up completely, but even then, I don't think it'll disconnect. I'm not sure on that, though. Writes to the drive get so slow once CloudDrive starts throttling them (when the drive has, I believe, less than 5GB of space left) that I'm not even sure you can fill the cache drive in a 24 hour period!
You are describing two different bans. And I wouldn't really call the upload limit a ban--though they do use the same error code to enforce it. There is an API ban that can occur if your exceed some number of API calls in a certain period of time. But CloudDrive does not have this problem. As long as an application implements Google's suggested backoff protocols (which CloudDrive does) it's essentially impossible to get an API ban. The other "ban" (which really isn't) is an upload limit that they enforce by failing API calls after the threshold is met. So once you upload 750GB in a day, all API requests to upload additional data will fail. But they don't actually ban you from the API. All *other* API calls will work as normal. Again, the latter isn't *really* a "ban" at all. Google just uses the same error code (403, user rate limit exceeded) to enforce the upload limit. But they don't actually ban you from additional API calls wholesale, like they do with the real API ban. They don't consider it abuse, they just enforce a server-side limit.
-
Other than this reindexing issue (assuming it'll get fixed eventually) and gdrive's upload caps (got plenty of time on my hands), any serious downsides to what I'm doing then? Basically, if there's no good reason to switch TO that method, are there any good reasons to get OFF OF this method?
Well, for the peace of mind you could go with 55TB drives so that chkdsk is sure to work. Beyond that, though, I don't think there is a huge benefit. I don't use ReFS only because I had a ReFS drive corrupt to raw from a windows patch and lost several TB of data. But if ReFS is working for you then chkdsk is unnecessary--ReFS is far more inherently resilient than NTFS. One thing that is somewhat of an issue with the multiple drive solution is that, with the way that CloudDrive presently handles caching, multiple drives sharing the same cache drive can slow things down depending on the situation. It also multiplies the amount of cache *space* required as well, since you need an appropriately sized cache for each drive. I wish, at the least, that the download cache could be shared, so that it would be essentially the same as one cache for a single larger drive. Just some things to think about. It does work though. No complaints about performance. I'm using 1 92TB drive (the remnants of the old 256TB drive), and 2 55TB drives, all combined into one DrivePool drive. It'll be a long while before I need space, but I'll probably just add another 55TB drive if I do.
If you haven't already, could you open a ticket about the issue at https://stablebit.com/Contact ?
And if you have, PM me the ticket number, so I can bump it.
Yeah, the new limits have been frustrating, to say the least.
Yeah, Christopher. I'll open a ticket and PM you tomrrow. I downgraded back to 914 though, since it takes hours every time the system reboots and I don't want to have the server down that long. I haven't tested it on 914 yet. I'll do so before I open the ticket.
EDIT: BETA 914 also enumerates after every reboot. BETA 906 does not. If you are having this problem, downgrading to 906 does not enumerate even after a hard reboot for me. You may want to downgrade for now--particularly if you have a large drive that takes hours to enumerate.
-
@srcrist, any updates on your experience since pooling 64TB drives instead of using one large one? I'm currently using a massive ReFS volume and I'm considering switching over to what you did. Worth the time?
Not worth the time. In fact, I was unable to complete that migration. The new upload limitations Google introduced make migrations of mass data impractical. I simply gave up.
-
Since upgrading to beta 929, I am also seeing this issue again. It enumerates every time the system reboots. This issue had been fixed in the last beta that I was using (902). It looks like something might have reintroduced this problem.
-
Thanks for the response.
I guess my question really relates to the following: Once I hit the daily upload limit, after 24 hours passes and the limit is reset, will CloudDrive start uploading again, or will the fact that it is continually trying to upload after the limit is reached cause a longer ban on the upload?
It will start uploading again. The retries will not lengthen the API ban. It will still just start working again in at whatever time your reset is. Generally, you likely won't even notice, as long as your cache drive isn't full.
-
When I first started using Plex with StableBit CloudDrive, it ate up a lot of data as well, without me doing anything. You want to disable some thing's in Plex, so it wont do that on it's own. Now I enjoy watching thing's through Kodi using a Plex plugin. Try disabling what's listed in here, it worked for me.
https://github.com/ncw/rclone/issues/1518#issuecomment-312930264
- Library → Update my library automatically
- Library → Run a partial scan when changes are detected
- Library → Update my library periodically
- Scheduled Tasks → Perform extensive media analysis during maintenance
- Scheduled Tasks → Update all libraries during maintenance
- Scheduled Tasks → Upgrade media analysis during maintenance
None of this should have any effect on outbound data. All of those are read operations. You should be able to leave all of that enabled and run Plex just fine, even if you have a very large library.
-
That's great, Christopher. Honestly, though, the efficiency changes have also done wonders to simply make sure that it doesn't have to enumerate every time there is an unclean shutdown, as well. So I actually haven't have to work around this issue since those changes several weeks ago. Good news, in nay case.
-
BTW is the 750GB/day cap enforced on a per user account basis, or is it enforced amongst the whole user group?
That actually seems to depend. I use a edu account (legit edu account via my Alma Mater), for example, and I seem to get my own personal 750GB/day quota that exists independently of any other users. But *most* people, who are using their own personal GSuites accounts on their own domain, seem to get 750gb/day for all accounts on that domain. So it would seem that Google is treating different organizations differently. I'm sure they're just smart enough to recognize that 750GB/day is very low for larger organizations, but I don't know at what threshold they start treating it differently.
-
If you're getting the "User rate limit exceeded", yes, it will block ALL API calls to the provider (that aren't from it's official app), and that will prevent you from downloading from the provider.
To be clear, Christopher: that actually is not the case here. When you hit the threshold it will begin to give you the 403 errors (User rate limit exceeded), but ONLY on uploads. It will still permit downloaded data to function as normal. It isn't actually an API limit, wholesale. They're just using the API to enforce an upload quota. Once you exceed the quota, all of your upload API calls fail but your download calls will succeed (unless you hit the 10TB/day quota on those as well).
AH HA! it's because I have Arq running backups to the same Gsuite account. must be hitting the limit there.
Right. For the people who think they aren't hitting the 750GB/day limit, remember that it's a total limit across *every* application that is accessing that drive. So your aggregate cannot be more than that. But enough people have confirmed that the limit is somewhere right around 750GB/day, at this point, that I think we can call that settled.
-
so is there really any difference to restricting upload to 70 Mb/s (~750 GB / day) vs letting it upload uncapped until the user rate limit and letting it auto-resume once the limit goes away? seems you can upload the same amount in either case, just faster in the latter case, right?
The main difference would be the flexibility of being to address unforeseen situations wherein it might be nice to have the ability to write to the cloud. I think, on the balance, it's probably better for the drive to have the ability to read and write at any time than requiring it to wait until it has connectivity again. But that probably depends on your caching situation. It will not, in any case, affect the operation of the drive in the long-run.
-
I just know that there is an unpublished 10TB/day download limit as well, and limits on drive to drive transfers. I know they publish other details, but it seems like their fair use policies (if you want to call them that) are more obscure for some reason. Their own support folks don't even seem to be aware. It is, in any case, a concerning practice.
-
Detaching the drives before a reboot will prevent the reindexing process, yes. Though, if you're not on the latest betas, I've noticed that the drives have been far better at recovering from an unclean shutdown than they used to be.
Drive with Plex updating libraries very slowly
in General
Posted
Chupa, my library is many thousands of titles and only takes a minute or two to update. Are you, maybe, referring to the initial adding of new media from a directory to your library? It does an analysis scan on the files during that addition that can take some time, but a normal scan where it simply looks for changes and additions should be a matter of minutes, no matter how large your library, even if you don't have the "run partial scan" option enabled.