Jump to content
Covecube Inc.


  • Content Count

  • Joined

  • Last visited

  • Days Won


Everything posted by srcrist

  1. I actually see multiple problems here, related to prompt uploads and data rate. The first is that, yes, I'm not sure that your prefetcher settings make much sense for what you've described your use case to be--and the second is that you've actually configured a delay here on uploads--contrary to your actual objective to upload data as fast as possible. But, maybe most importantly, you've also disabled the minimum download, which is the most important setting that increases throughput. So let's tackle that delay first. Your "upload threshold" setting should be lower. That setting says "start uploading when you have X amount of modified data, or it has been Y minutes since the last upload." You've actually raised the values here from their lower defaults of 1MB or 5 minutes. So your drive will actually be more delayed relative to drives created with the default settings. Your minimum download should be some reasonable amount of data that roughly corresponds to the amount of data you expect the system to need in short order whenever any amount of data is requested. So, for example, if you're dealing with files of 10-20MB, and you need the data asap, you'd probably want a minimum download of around 15MB so that any time the system makes a request of the drive, it pulls all of the data that might be quickly needed to the cache immediately. Note that if you let it ClouddDrive will only pull the amount of data that your system requests of it for a particular file read. That means, say, a few hundred KB at a time of a music file, a few dozen megabits of a high quality video file, or the data for a first few pages of a document. Just like a hard drive. But, unlike a hard drive, CloudDrive has to renegotiate an API connection and request data every time you system requests any amount. You absolutely cannot let it do that. That is tremendous overhead and very slow. You need to figure out what your actual data needs are, and force CloudDrive to pull that data asap when a request is made. I would suggest (and this is based on the still somewhat cryptic use case you've described thus far) 10 or 20MB for your minimum download. You'll notice that your current minimum download recommended speed is presently only 14mbps. Compare that to the estimated data rate for these settings: With respect to your prefetcher, I'd probably need to know a bit more about both your intended purpose for the drive and your actual drive settings to give you better advice. For now, suffice it to say that I actually think that, based on your intention to load 10-20MB files as quickly as possible, the prefetcher probably doesn't need to be enabled at all. As long as your drive is structured correctly and the other settings are correct. If you want some more help here, you're going to have to provide a more detailed account of what you're actually doing with the drive, and preferably all of the structural information about the drive such as this information here: I'd also note that if you want to prioritize writes to the drive over read speed, you'd probably want to enable background I/O as well. Turning it off makes it easier for drive reads to slow the write process. And, relatedly, another thing that isn't clear to me: since data is available from the CloudDrive drive as soon as its added to the drive, what is the need for the rapid upload to the provider? That is, the data is already accessible, and the speed with which the data is uploaded to the provider has no bearing on that. So why the need for many MB/s to move what effectively amounts to duplicate data to your host? Just trying to understand your actual objective here. And upload verification is not a relevant setting with respect to drive throughput, but, if data integrity is something you care about, I do suggest turning it on. It should effectively eliminate the potential for Google issues to ever cause data loss again, since every byte uploaded is verified to exist on Google's storage before CloudDrive will move on to new data and remove the uploaded data from the cache.
  2. 10MB chunks may also impact your maximum throughput as well. The more chunks your data is stored in, if you accessing more than that amount of data, the more overhead those chunks will add to the data speed. The API limits are for your account, not the API key. So they will all share your allotment of API calls per user. It does not matter that they use different API keys to access your account. That may be, but note that Google Drive is not, regardless of whether or not it's a business account, an enterprise grade cloud storage service. Google does offer such a service (Google Cloud Storage), but at a much higher price. This remains true regardless of what you consider their service to be. CloudDrive also supports GCS APIs as well, if you actually need that level of service. Google Drive, even on a business account, is intended to provide consumer grade storage for individual users--not enterprise grade storage for high data volume, high data rate, or high availability purposes. See Google's own statement on the matter here: https://cloud.google.com/products/storage I think I understood what you were saying, actually. I don't really understand the use case that would necessitate what you're looking for, and I'm not entirely sure how you're testing the maximum throughput, but I do understand what you are seeing and what you are thinking is a problem. 110mbps is pretty low relative to my experience, assuming your system is actually requesting the amounts of data from the drive that would lead CloudDrive to max out the throughput. I am, for example, presently seeing these speeds: And that is with a largely idle workload. I believe upload verification is the only load on the drive at the moment. I don't, unfortunately, really know how to help you solve it, though. If your system is trying to pull a lot of data, and your I/O settings and your prefetcher settings are configured reasonably, speeds to Google should certainly be faster than what you are experiencing. Maybe not 100MB/s, but better than 110mbps. Have you considered a peering issue between your system and Google itself? To be clear: have you actually tested the performance in a real-world way? Trying to read large amounts of data off of the drive, as opposed to looking at the network metrics in the UI? What speed does Windows report, for example, if you try to copy a large file off of the drive that isn't stored in the cache? And, on a related note, have you tested to see if your data is being pulled from the cache? 40GB every 12 hours is not a ton of data. If your cache is larger than that, it's probably all being cached locally. If that is the case, CloudDrive won't need to actually download the data from your provider in order to provide it to your system.
  3. Is this a legacy drive created on the beta? I believe support for chunk sizes larger than 20MB was removed from the Google Drive provider in .450, unless that change was reverted and I am unaware. 20MB should be the maximum for any Google Drive drive created since. 50mbps might be a little slow if you are, in fact, using 100MB chunks--but you should double-check that. Gotcha. Yeah, if you set additional limitations on the API key on Google's end, then you'd have to create a key without those restrictions. And CloudDrive is the only application accessing that Google Drive space? Nothing else is using the API or accessing the data on your Google Drive? Those yellow arrows indicate that Google is throttling you--and that is handled on a user by user basis--so that's the angle to explore. I think you might just be running into a difference in expectation. I'm not sure how many users would consider 50mbps * X threads to be slow. I think that if most users hit 500mbps they would probably consider that to be relatively fast. So the only people who are saying, "yeah, my drive is slow," are the people whose drives are bottlenecked by I/O, rather than network throughput. It really sounds like you were looking for CloudDrive to provide effectively enterprise grade throughput on a consumer data storage service, and I don't think that was necessarily a design goal. You might be able to get the sorts of data speeds you're looking for, somewhere in the 100s of MB/s, from rClone if you configure it to operate on the data in parallel--so maybe look into that. Sure thing. Good luck finding something that works for you!
  4. Sometimes the threads-in-use lags behind the actual total. As long as you aren't getting throttled anymore, you're fine. Netdrive and rClone are not appropriate comparisons. They are uploading/downloading several gigbytes of contiguous data at a time. The throughput of that sort of data stream will always be measured as larger. CloudDrive pulls smaller chunks of data as needed and also no data at all if it's using data that is stored in the local cache. You're not going to get a measured 50MB/s per thread with CloudDrive from Google. It just isn't going to happen. 50mbps for a single thread with a 20MB chunk size is about what I would expect, accounting for network and API overhead. That's about 500mbps with 10 download threads--or roughly 62MB/s in the aggregate. A larger chunk size could raise that throughput, but Google's chunk size is limited because it creates other problems. You are hitting the expected and typical performance threshold already. Your aggregate thread count would already exceed the 50MB/s that you get from the applications making larger contiguous data reads. You just have to understand how the two applications differ in functionality. CloudDrive is not accessing a 10GB file. It is accessing X number of 20MB files simultaneously. That has performance implications for a single API thread. The aggregate performance should be similar. I am not 100% sure what this means. But, if I am interpreting correctly, I think you are confusing the API key's functionality. The API key has nothing to do with the account that you access with said key. That is: an API key created with any google account can (at least in theory) be used to access any other account's data, as long as a user with the appropriate credentials authorizes said access. So much to say that you can use the API key you create to authorize CloudDrive to access any Google account to which you have access, as long as you sign in with the appropriate credentials when you authorize the drive. Note that Google does place limits on the number of accounts that the API key can access without submitting it to an approval process, but those limits are much higher than anyone using a key for genuine personal use would ever need to worry about. It's something like 100 accounts. Note that, by default, you were already using an API key created on Stablebit's Google account to access YOUR data. Sure. Ultimately, CloudDrive isn't really designed to provide the fastest possible throughput with respect to storing data on Google or any other provider. If your needs are genuinely to have multiple 50MB/s simultaneous downloads at once, on a regular basis, this probably just isn't the tool for you. The most efficient performance configuration for CloudDrive is to have one large drive per Google account, and give that one drive the maximum number of threads as discussed above, leaving you with that 60ish MB/s aggregate performance. Your present configuration is sort of self-defeating with respect to speed to begin with. You've got multiple drives all pulling chunks from the same provider account, and they all have to compete for API access and data throughput.
  5. I mean, the short answer is probably not using your API key. But using your own key also isn't the solution to your real problem, so I'm just sorta setting it aside. It's likely an issue with the way you've configured the key. If you don't see whatever custom name you created in the Google API dashboard when you go to authorize your drive, then you're still using the default Stablebit keys. But figuring out exactly where the mistake is might require you to open a ticket with support. Again, though, the standard API keys really do not need to be changed. They are not, in any case, causing the performance and API issues that you're seeing.
  6. Any setting that is only available by editing the JSON is considered "advanced." See: https://wiki.covecube.com/StableBit_CloudDrive_Advanced_Settings That is more, in one drive, than Google's entire API limits will allow per account. You'll have to make adjustments. I'm not 100% sure the exact number, but the limit is somewhere around 10-15 simultaneous API connections at any given time. If you exceed that, Google will start requesting exponential back-off and CloudDrive will comply. This will significantly impact performance, and you will see those throttling errors that you are seeing. Note that Google's limits are across the entire google user, so, if your drives are all stored on the same account, your total number of upload and download threads across all of the drives on that account cannot exceed the 15 or so that Google will permit you to have. I use 10 down and 5 up, with the presumption that all 15 will rarely be used at once, and rarely run into throttling issues. It should not be. My server is only on a 1gbps connection, not 10gbps like yours, and I can still *easily* hit over 500mbps with 10 download threads. 20 upload threads is pointless, since Google will only allow you to upload 750GB per day, which averages out to around 70mbps or so, a rate that even two upload threads can easily saturate. Ultimately, though, your numbers are simply excessive. You're quite a bit over the recommended values. Drop it so that you're at less than 15 threads per Google account and you'll stop getting those errors. Then we can take a look at your throughput issues.
  7. Once you change the API key in the advanced settings, you will need to reauthorize the drive (or detach it and reattach it). It will not immediately switch to the new key until it recreates the API authentication. Did you do that? Also, how many upload and download threads are you using? 11 download threads per second is likely just hitting Google's API limit, and it is likely why you're getting throttled--especially since it looks like you're also allowing around 10 upload threads at the same time. In my experience, any more than 10-15 threads, accounting for both directions, will lead to exponential backoff and API throttling on Google's part. Try something like 8-10 download threads and 3-5 upload threads and see if it reduces the throttling or makes it disappear.
  8. Yessir. This article on the wiki details the process: https://wiki.covecube.com/StableBit_CloudDrive_Q7941147 Note both that the process on Google's end is a little outdated (though you can still probably navigate the process from that article), and that switching to a personal key isn't really necessary and carries new limitations relative to Stablebit's developer keys (for example, people using their personal keys were already getting these per-folder limitations back in June, while Stablebit's key has just had this limitation enforced this month). So, caveat emptor.
  9. If you're still having issues converting a drive with any release newer than .1314, you should absolutely open a ticket.
  10. User Rate Limit Exceeded is an API congestion error from Google. If you're being given that error, and you're using your own key, you're hitting the usage limits on your key. If you're using Stablebit's default key, then they are hitting their limits--which is possible, especially if many people are having to do a lot of API calls at once to convert drives. That's just speculation, though. If it's stablebit's key, you'll want to open a ticket and let them know so they can request a raise on their limit. You can also try reauthorizing the drive, as stablebit has several API keys already--or switching to your own.
  11. The fix for these issues is several months old. It was fixed back in July. .1318 is the latest stable release on the site, and it has the fix (which was in .1314). You don't need to use a beta version, if you don't want.
  12. Open a ticket using the link provided in the post you quoted. They'll get you sorted.
  13. It looks like Google's Enterprise Standard plan still has unlimited for even a single user for $20 a month. So this change appears to be little more than a price increase for people who were previously using their unlimited business plan. That remains, to my knowledge, the only provider comparable to Google's own previous plan.
  14. CloudDrive itself operates at a block level, so it isn't aware of what files in the file system are being written to or read by the applications on your computer. Much like the firmware for a HDD or SDD is unaware of that information as well. So, that is to say, that it isn't possible via CloudDrive. Windows Resource Monitor or Process Explorer or another tool to look at Windows' file system usage would be required--as it sounds like you discovered.
  15. Neither of these errors/issues are likely related to the topic of this thread. If the drive is RAW, the file system is likely corrupted. The change this thread references only restructures the way the data is stored on the provider, it isn't changing any of the data itself. If it was corrupting data on the provider, we'd be seeing that manifest in other ways for other users--and that isn't the case. I would suggest opening a ticket with support here: https://stablebit.com/Contact in order to try and troubleshoot how your file system was actually corrupted. Though even Windows Updates have corrupted ReFS volumes on a regular basis. It just isn't a stable file system, and that's why the functionality has been rolled back in Windows 10 for now. So if you blinked or breathed wrong, that might be the culprit. And a User Rate Limit Exceeded error is an API limit error. It means the API key you're using is hitting the limit of calls per time period, and Google won't allow it to access the API any more. Are you using the default API key, or your own?
  16. If you're following the instructions correctly, you are simply reshuffling data around on the same drives. They are file system level changes, and will not require any data to be reuploaded. It should complete in a matter of seconds. And the purpose of the multiple partitions is covered above: 1) to keep each volume below the 60TB limit for VSS and Chkdsk, and, 2) to avoid multiple caches thrashing the same drive and to make bandwidth management easier. If you have multiple cache drives then you can, of course, use multiple CloudDrive drives instead of volumes on the same drive, but make sure that you can actually support that overhead. I'd suggest an SSD cache drive per CloudDrive drive--the cache can be surprisingly resource heavy, depending on your usage. In any case, though, there isn't really a compelling need to use multiple CloudDrive drives over volumes--especially if the drives will all be stored on the same provider space. There just isn't really any advantage to doing so.
  17. srcrist

    Change cluster size

    You cannot make any changes to the cluster size server-side. Changing the cluster size modifies the drive structure within the chunk files that are stored on your provider. All of the data would have to be downloaded and reuploaded. You CAN, however, change the cluster size per-volume (partition), so if you can just leave your existing data in place, you can create a new volume with a different cluster size.
  18. Several people have confirmed that the "Enterprise Standard Plan" is $20 a month and retains the unlimited with even a single user. So this isn't likely to translate to much more than a price increase. See, for an example: https://old.reddit.com/r/DataHoarder/comments/j61wcg/g_suite_becomes_google_workspace_12month/g7wfre3/
  19. There should be no reason to change any advanced settings, and I would discourage people from doing so unless you specifically need to change those settings for some reason. The per-folder limitation issue should be resolved by using a beta newer than .1314 with the default settings.
  20. Note: You guys should not have to do anything specific. The advanced options setting is to force CloudDrive to convert the entire drive--including existing data--to a hierarchical format. That should not be necessary to fix your problem, and will take many hours (often several days) for a large drive. Any release later than .1314 should automatically use a hierarchy for any new data added to the drive, and move chunks to the hierarchical structure when and if they are modified. If you were still using a .12XX release, you will have to migrate to a BETA. You can find BETA releases here: http://dl.covecube.com/CloudDriveWindows/beta/download/ Using any beta newer than .1314 should immediately fix the per-folder limit issue. If it does not, I would definitely open a ticket before doing anything else: https://stablebit.com/Contact
  21. I don't have any info on this other than to say that I am not experiencing these issues, and that I haven't experienced any blue screens related to those settings. That user isn't suggesting a rollback, they're suggesting that you edit the advanced settings to force your drive to convert to the newer hierarchical format. I should also note that I do not work for Covecube--so aside from a lot of technical experience with the product, I'm probably not the person to consult about new issues. I think we might need to wait on Christopher here. My understanding, though, was that those errors were fixed with release .1314. It presumes that existing data is fine as-is, and begins using a hierarchical structure for any NEW data that you add to the drive. That should solve the problem. So make sure that you're on .1314 or later for sure. Relevant changelog: .1314 * [Issue #28415] Created a new chunk organization for Google Drive called Hierarchical Pure. - All new drives will be Hierarchical Pure. - Flat upgraded drives will be Hierarchical, which is now a hybrid Flat / Hierarchical mode. - Upgrading from Flat -> Hierarchical is very fast and involves no file moves. * Tweaked Web UI object synchronization throttling rules. .1312 * Added the drive status bar to the Web UI. .1310 * Tuned statistics reporting intervals to enable additional statistics in the StableBit Cloud. .1307 * Added detailed logging to the Google Drive migration process that is enabled by default. * Redesigned the Google Drive migration process to be quicker in most cases: - For drives that have not run into the 500,000 files per folder limit, the upgrade will be nearly instantaneous. - Is able to resume from where the old migration left off.
  22. Note that deleting files off of the drive and deleting the drive itself are not the same thing. Simply removing files off of the file system of the drive will not remove the drive data from your provider, because a drive of X capacity still exists, just like removing files off of your physical drive does not cause it to physically shrink. If you want to remove the storage files located on the provider then you have to either shrink the drive or destroy it. Also note that, with CloudDrive, the "files" stored on the provider are not representative of the files stored on the drive in the first place. They are chunks of data that comprise the drive structure that CloudDrive uses to store information. Once they exist on the provider, there is no need to delete them unless the drive structure itself changes. CloudDrive will simply overwrite and modify them as data is added and removed.
  23. At least there is a silver lining. That sucks that CenturyLink's service and hardware is so locked down and convoluted, though. I use a wired router and separate wireless access points, personally, and that would be a nightmare for my setup. Hope it works out.
  24. Generally speaking, even if other router functions are locked down by the ISP on their hardware, consumers typically have access to change the router to bridge mode themselves. Are you sure that you don't have access to that via the web configuration for the router? Is the fiber gateway also integrated into the router, or can you just replace it with your own and bypass their hardware entirely?
  25. There really isn't anything unique about CloudDrive's network usage that should impact the router that you use over and above any other high-use network application. So any general router recommendations should be fine. I personally rely on Small Net Builder for reviews and ratings for home network hardware. Might be worth a look for you.
  • Create New...