Jump to content

srcrist

Members
  • Posts

    466
  • Joined

  • Last visited

  • Days Won

    36

Posts posted by srcrist

  1. On 5/18/2019 at 10:56 PM, Bowsa said:

    Nope. Chkdsk totally scrambled all my files into video142.mkv, so recovering 12TB is as good as useless. CHKDSK did more harm than good

    Chkdsk can never do more harm than good. If it recovered your files to a Lost.dir or the data is corrupted than the data was unrecoverable. Chkdsk restores your file system to stability, even at the cost of discarding corrupt data. Nothing can magically restore corrupt data to an uncorrupt state. The alternative, not using chkdsk, just means you would continue to corrupt additional data by working with an unhealthy volume.

    Chkdsk may restore files depending on what is wrong with the volume, but it's never guaranteed. No tool can work miracles. If the volume is damaged enough, nothing can repair it from RAW to NTFS. You'll have to use file recovery or start over. 

  2. This might actually be on the provider's side. I once deleted about 100TB from Google Drive and it took months for the data to actually be removed from Google's servers. It didn't matter, since I didn't have a limit, but you might be running into a similar situation here. Contact your administrator and see if there is something they can do. 

  3. 8 hours ago, JulesTop said:

    Hi, I have a similar situation...

    My OS is on my SSD, and I am using the same SSD as the cache for my 3 clouddrives. However, two out of the three are used for duplication. Is there a way to select one of them as the drive that the data gets read from all the time... because right now, although it looks like the data is always pulled from the same drive, I have no way of guaranteeing it (as far as I can tell).

    That's a good question, and one probably better asked in the DrivePool forum, because it's really a DrivePool question. I'm not sure if there is any way to tell DrivePool to prefer reads off of a specific drive. I know that it's smart enough to prefer local drives over cloud drives if the same data exists on both, but I'm not aware, off the top of my head, of a way to tell it to prefer a specific drive that falls within the same tier of the built-in hierarchy. That doesn't mean it doesn't exist, though.

    One work-around that I can think of would be to simply point your applications at the hidden folder on the drive, rather than pointing them at the pool itself. That would work. 

  4. 6 hours ago, NMonky said:

    Okay, I am currently using Cloudflare's 1.1.1.1, I will switch it to Google's DNS service and see if that helps at all.

     

    Sure thing. And also do not be scared to simply use the DNS servers provided by your ISP to test them. They're likely to be the most local servers for you, and the most likely to provide you with efficient routing to Google--despite the other flaws with using ISP provided DNS.

     

    6 hours ago, NMonky said:

    Or, is there a more efficient method you would recommend?

    No, I think that's fine. I used DrivePool to consolidate when I recognized this problem myself. You can always click the little arrows next to the status bar in the DrivePool UI to give the transfer a higher priority, if you're doing other things on the drives. That may help speed up the process somewhat.

     

    6 hours ago, NMonky said:

    My plan now is to wait for a few weeks for it all to consolidate onto one CloudDrive, use the other CloudDrives for duplication in case one fails, switch to Google DNS, and once all this is done get back to you and let you know if it is all working or not.

    Just make sure that they aren't all caching on the same physical drive. Move the other two to another cache drive, even if you're only using them for duplication.

     

    6 hours ago, NMonky said:

    Are you happy for me just to comment on this thread again in a few weeks time?

    That's fine.

     

    6 hours ago, NMonky said:

    Also, most importantly, thanks again for your help with all this. If you're ever in Kuala Lumpur let me know, I owe you a beer.

     

    Any time.

  5. 10 hours ago, NMonky said:

    Okay, I did some quick tests last night. This has definitely moved us in the right direction. There are still some issues with pixelation and a little bit of buffering, but they are definitely less pronounced. Are there any setting that you would recommend tweaking further?

    Also, I noticed that one of my three Cloud Drives is set to 10mb chunks instead of 20. Is there any quicker and easier way to fix this without having to download all the files and reupload them?

    So, to be clear, there isn't anything that CloudDrive can do that should ever really cause pixelation. That just isn't how digital video works. Buffering, maybe. But not pixelation. That would be a Plex thing, and a result of how Plex is encoding or decoding the video. The drive throughput is either going to work or not. It's going to be fast enough to keep up, or not. It won't pixelate the video, it just won't work. I suppose Plex, as a client, might display pixelation if it isn't able to get data fast enough, as a sort of stop-gap to keep the stream going--so that might be why you're seeing that. 

    The settings I gave you were the ones I use, and am able to stream 4K video with at original quality. So that's effectively all the settings we can tweak within CloudDrive--at least in the short term. There are some tweaks we might be able to make to optimize for you later, but those should be working and they're a good starting point for your connection speed.

    At this point, it's probably best to start looking at the network, if you've ruled out the possibility that there is some issue with the file. Play with your DNS servers and see if you're being sent to a non-optimal Google server, and check your network settings across the board. You should easily be able to hit 400mbps or so with the CloudDrive settings I gave you. If you aren't, there is something wrong.

    As far as the maximum chunk size goes, there isn't any way to change that without recreating the drive. Though you may want to consider that anyway, as you cannot use chkdsk on volume sizes of 100TB, and it's a lot less optimal to use 3 separate CloudDrive drives than 1 drive partitioned into multiple volumes. In fact, if all three of those drives are all caching from the same SSD, that might be the cause of all of your issues here, as well. The cache can be very disk usage heavy, and I don't generally recommend any more than a single virtual drive caching to a single physical drive. I had problems with that myself. Note that a single 1PB CloudDrive drive can be divided up into many volumes, and all can share a single cache. It's probably better to make these changes now, while there is still relatively little on the drives, than later, when it can take months. Each volume should be smaller than 60TB to work with Volume Shadow Copy and CHKDSK. 

  6. 1 minute ago, NMonky said:

    Will test the above. Quick question; what exactly do you mean by drive structure? 

    Chunk size was the highest allowed. I think that was 20.

    I mostly meant the chunk size, but drive structure would be things like chunk size, cluster size, sector size, file system type. The things you chose when you created the drive. But really only the chunk size is relevant here. 

  7. Some of this may seem counter-intuitive, but try this: Drop your download threads to ten, drop your prefetch window to 150-175MB, raise your prefetch trigger to 15MB and drop the window to 10 seconds. Try that, test it, and report back.

    Also, what is your drive structure? Most importantly, what is the chunk size that you specified at drive creation?

    Also, to rule out one more factor, have you tried playing the files off of the CloudDrive with a local media player like VLC?

  8. 5 hours ago, Bowsa said:

    It does make a difference, CloudDrive doesn't start uploading at full speed until a certain amount of data is copied/cached, so having to go through the process for every 400GB isn't ideal.

    That simply isn't true. Are you sure that you aren't running into some sort of I/O issue? CloudDrive will upload without restriction as soon as your upload threshold is met in the performance settings. Leave "Background I/O" enabled to ensure that writes are prioritized over reads in Windows' I/O and see if that fixes your problem. Or try disabling it if it's already enabled and see if other processes are simply getting in the way. 

    I know this, btw, because I transferred 90TB from one drive to another, and my cache was full for months. So I know from experience that the cache being full does not throttle upstream performance. 

  9. CloudDrive has vastly higher overhead than many other transfer protocols like FTP and Usenet. 700mbps is probably about right with respect to a maximum theoretical transfer rate on a 1gbps connection. It isn't built for raw transfer throughput, and it adds additional overhead on top of the protocols being used to transfer the data. I think what is required here is an adjustment of expectations, more than an adjustment of settings in the application. Remember that CloudDrive is also using whatever protocols your provider requires to transfer its data, so, if you were using FTP, for example, you need to account for the standard FTP overhead AND the additional overhead above that amount that CloudDrive requires as well. 

    If transferring files from one location to another is the primary goal, you'd be better off setting aside the accessory benefits of CloudDrive, and simply completing that transfer using a dedicated file transfer protocol. It will effectively always be faster. 

    Remember, also, that overhead is additive per transfer, so CloudDrive's mechanism of chunking your data means that you add X overhead per chunk rather than X overhead per file--as you would if you were using, say, FTP. You simply will not see transfer rates of 90-95% of your maximum throughput, and that isn't the goal of this particular application. 

  10. You'll need to use the CreateDrive_AllowCacheOnDrivePool setting in the advanced settings to enable this functionality. See this wiki page for more information. The cache being full will not limit your upload, only your copy speed. It will throttle transfers to the speed of your upstream bandwidth, so it should make effectively no difference, aside from the fact that you won't be able to copy new data into the drive faster. That data will still upload at the same rate either way. 

  11. It's possible, but provides no benefit over simply using the SSD as the cache drive. The cache is stored as a single file, and will not be split among multiple drives in your pool. Just use the SSD as your cache drive, and the cached content will always be at SSD speeds. 

  12. 6 hours ago, Knxwledge said:

    Does Chunk Size really matter? I have mine set to 10 but I see everyone setting it to 20. Is 20MB better? Am I able to change the chunk size of a drive?

    That isn't a question with an objective, universal answer. The benefits to different chunk sizes vary depending on what you want to use your drive for. A lot of people use their CloudDrive drives to store large media libraries. So the maximum chunk size for Google, which is 20MB, allows for the highest capacity drive, and the highest throughput for streaming media off of the drive. 10MB chunks are *also* probably fine for this purpose, but the drive cannot be as large (though it can still be huge), and the theoretical maximum downstream speed will be lower. 

    You cannot, in any case, change the chunk size of a drive once it has been created. That's one of the settings that has to be chosen upon drive creation, and changing it requires a fundamental change in the structure of the drive as it is stored on the provider, so it can't be changed later. 

  13. 52 minutes ago, steffenmand said:

    Nobody said it was CPU threads. HOWEVER your CPU has a limitation as to how many threads it can run as a system at a time - thus putting it higher than the CPU capacity will only build queues. Not comparing it to the same type of threads it still means a limitation in concurrent operations.

     

    This is incorrect. Adjusting the number of threads in CloudDrive has no appreciable impact on CPU usage. Threads, in this case, simply refers to the number of simultaneous upload and download requests that the application will let itself make at any given time. CloudDrive has limits on the number of threads that one can set in the application, so it isn't possible to test whether or not thousands of threads may eventually lead to a degradation of CPU performance, but, under the limitations that exist today, you will hit the API limitations of the provider long before any impact on CPU usage is worthy of consideration. In short: Do not adjust CloudDrive's threads based on CPU related concerns, and there is no relationship between "threads" as related to processor scheduling and "threads" as they are used in this context--other than the name.

    57 minutes ago, steffenmand said:

    Also faily certain that the "minimal download size" will never download anything larger than the chunk size. So if the chunk size is set to 1 MB, then 1 MB is the max download. Im pretty sure that the feature was made for partial downloads of data from larger chunks - feel free to correct me here.

    As far as I know, this is also incorrect. A minimum download size larger than the chunk size will still download multiple chunks of contiguous data until the specified threshold is met, again, as far as I know. There is nothing conclusive in the documentation about exactly how this functions, but I have seen Christopher recommend settings higher than 20MB for Google Drive, such as he did in this case and this case, for specific situations. That would lead me to believe that higher settings do, in fact, have an impact beyond a single chunk. This is, of course, with the caveat that, as explained in the user manual, CloudDrive can still request less data than the "minimum" if the system is also requesting less (total) data.

    In any case, it is true, either way, that a larger minimum download will reduce the responsiveness of the drive, and that speeds in excess of 600-700mbps can be achieved with a minimum download of 10-20MB using the settings I recommended above. So if the problem is responsiveness, dropping it may help. 

    The base settings that I would recommend anyone try if they are having performance issues are the following, and then play with them from there:

    • 5 Upload Threads
    • 10 Download Threads
    • Upload Throttling enabled at approx 70mbps (to stay within Google's 750GB/day limitation)
    • Upload Threshold 1MB/5 Mins
    • Minimum Download 10-20MB
    • Disable Background IO if you'll be streaming media so that downstream performance is not impacted by upstream requests. 

    If that works, you're great, and, if not, then you come back and people can try to help. But these are settings that have worked for a lot of people here over the last several years. 

  14. On 4/24/2019 at 7:43 PM, Lex said:

    @Christopher (Drashna) Quick follow up on the above question:  In DrivePool you can replicate data and drag it manually into a newly created DP folder, then (iirc) use that with file duplication.  If you were to duplicate all encrypted data in a clouddrive, then create a second empty clouddrive and move all the duplicated encrypted files/folders to the new clouddrive folder would it then be possible to use the duplicate (CD X 2 + DrivePool)?  In this case the second clouddrive would have its own ID.  Or does clouddrive encrypt each file in a different way between two drives --  file A on CD1 is encrypted differently than file A on CD2?

    The question is a little bit confusing, but if you're asking if you can simply use DrivePool to duplicate at the file system level, even if the data is encrypted in the cloud by CloudDrive, then yes. That's fine. What Christopher is talking about above is duplicating the raw data on your provider. Don't do that for any reason, unless you really understand the longer term implications. The same file *will* be encrypted and stored differently on both drives. 

  15. 4 hours ago, Chupa said:

    Thanks for the answer, lately I tried this configuration which seems good and stable, but I wanted to ask why at the same time I started to have corrupt filesystems and I thought that the cause could be this sharing. Thank you

    The file corruption was almost certainly because there was a data rollback on provider side, if you are using Google, and not a result of anything you did at all. 

  16. 12 hours ago, skapy said:

    Hello,

    i have a gdrive business account and my created clouddrive is 99% full.

    Can i create a 2nd clouddrive on the same gdrive account without any problems?

    If possible: Any thinks i should take care for during creating a 2nd clouddrive?

     

    thanks in advantage

    You can, but if you're going to be using it for the same purpose, I would suggest enlarging the drive and adding a second volume to the same drive instead. This make cache and performance management easier, and it's all the same to CloudDrive. It doesn't care if the information is on two volumes or twenty. 

  17. The threads in CloudDrive are not CPU threads, and are not CPU limited. They are just simultaneous upload and download requests. They are limited by frequency and your provider's API limitations. Getting the correct value, for you, will be a matter of experimentation. I suggest starting with 10 download threads and 5 upload, and adjusting from there. You should *never* need more than a handful of upload threads, in any case, unless your network is misconfigured. Even 2 or 3 upload threads is capable of hitting the 70-75mpbs throughput that Google allows--permitting that your connection speed is capable of doing so. See the manual page here for more information. 

    11 hours ago, pedges said:

    Is there an obvious way to tell how many threads my CPU can handle? I'm up to 10 threads and 100MB minimum download and it's still taking forever to download even smaller amounts of data.

     

    This is the exact opposite of what you want. That's why. The larger your minimum download, the less responsive your drive will be. A 100 MB minmum download means that even if an application only requests 1 MB of data, CloudDrive is still downloading 100MB. There is effectively no reason for your minimum download to ever be larger than your chunk size, for most use cases. Raising your minimum is actually working against you here. Go with my above recommendations for thread count, and drop the minimum download to somewhere between 10 and 20 MB. See how that performs. We can make adjustments to your prefetcher to accommodate larger data requests over and above that amount. 

     

    EDIT: Also disable Background IO on the drive in the performance settings if read response is important.  

  18. 4 hours ago, fly said:

    edit: Can I 'backup' the same data from the same CloudDrive to multiple storage providers?

    Sorta. But I think you might have it backwards. You wouldn't back up the data that sits on the provider to other providers, because the drive structure changes somewhat by provider. You would, instead, mount multiple providers with CloudDrive and use a tool like DrivePool to mirror the volumes to one another. 

  19. 12 hours ago, steffenmand said:

    You are not counting the thread response time into the equation. The delay you get there will end up giving you slower responses. In my case that is 2500-2800 ms. Ie. i lose 2,5 second per verification. But ofc. if you work with very few threads it is the case, but with a high amount of threads you cant really increase it further without getting throttling :)

    A 2500-2800ms response time to google is almost *certainly* a network issue on your part, and not one that needs to be considered as a general concern. It should not take that long to send an internet packet around the entire planet, and, if you're seeing that sort of latency, you probably have bad hardware between you and your destination. A round trip to a satellite, which is the lengthiest hop in existence, is only 500ms plus the terrestrial hops. So unless you are, for some reason, beaming data to space and back several times, 2500-2800ms is simply a broken connection. I think it's important for you to consider just how unrepresentative that is, of the typical internet user, when considering whether or not someone is speaking to your issues specifically, or when requesting that changes be made to accommodate your situation. You're talking about latency that exceeds 5x even the typical satellite internet connection. The number of people dealing with such a situation are a fraction of a fraction of a percent. 

    Regardless of your situation, though, Google themselves have a hard limit of 750gb/day. A single thread, for most users, can hit that limit. In any case, my above response is perfectly compatible with slower response times. It's very easy to hit the maximum 70-75 mbps. No matter how many threads are taken up by reads, you can always add additional threads to make up for the read threads. If there are two read threads outstanding, you add two more for uploads. If there are 4 read threads outstanding, you add 4. If your responses are that slow, you won't get throttled no matter how many threads you add--which means that adding threads can always compensate for your connection delays. If you're still experiencing throughput issues, the problem is something else. 

  20. 15 minutes ago, steffenmand said:

    But as far as i saw the new upload thread doesnt start before the old one was verified - am i wrong here?

    No, you're right. It will use the upload thread count to download the data, so that will take up some number of slots for a time before those threads will be available for a new upload chunk. The mistake, though, is translating that to a loss of throughput, which isn't (or at least does not necessarily need to be) true. That is: you can achieve the same upload speeds with more threads, if you need to. 5 upload threads should be more than enough for most users to still max out the 70-75mbps of average data that Google allows you to upload each day, while also reading back the chunks in real-time.

    What *is* true is that if you theoretically only had one upload thread, it would have to use the same thread to read the data back each time, and that would slow things down somewhat. But, even for residential connections, that wouldn't generally translate to a 50% reduction in throughput, because your downstream capacity is generally a lot faster than your upstream on residential connections. So it would take a lot less time to *read* each chunk than to upload it to the provider. If your connection is symmetrical, then the question simply becomes about your overall throughput. Anything over 100 mbps or so should not generally notice a difference aside from downstream congestion as a result of the verification process. 

    Bottom line: If you *do* suffer a performance drop with upload verification enabled, and you're working with a high-bandwidth connection (100 mbps symmetrical or more), you just need to add a few upload threads to accommodate the increased workload. As long as you can hit 70ish mbps upload, though, you're capable of maxing out Google's limitations regardless. So all of this only sort of matters in a theoretical sense. 

  21. On 4/16/2019 at 11:35 AM, Bowsa said:

    My issue is that the 10TB drive is about to fill-up, and I wanted to opt against re-sizing the drive to avoid the situation I faced earlier which is 3 DAYS of chkdsk running to diagnose the disk (susceptible to power surge interruptions, and such). Ideally I'm going to be creating another 10TB CloudDrive, but my question is how does pooling work for multiple CloudDrives?

    So chkdsk is not affected by the size of the *drive*. Only the size of the *volume* (aka, generally, partition). You can have multiple volumes on the SAME CloudDrive drive. So you can still expand the size the drive and create a second VOLUME smaller than 55TB or so, and chkdsk will work with it just fine. I would, in fact, recommend this over a second drive entirely, so that you don't have multiple caches. 

    Aside from this note, DrivePool will operate identically regardless of whether or not you're using a volume on a local disk, a CloudDrive disk, or multiple CloudDrive volumes on the same disk. DrivePool does not operate at the block level, so it doesn't concern itself with the chunks. That will all still be handled at the CloudDrive level. You can customize the DrivePool settings to configure the file placement however you want. There is no impact on performance, as DrivePool simply forwards requests to the underlying drives--CloudDrive or otherwise. 

    On 4/16/2019 at 11:35 AM, Bowsa said:

    Lastly, if my new "CloudDrive Pool" is my new "main drive", does that mean that CHKDSK is something that would have to be run on the Pool, or the individual drives?

    Chkdsk is a file system tool. So it operates on volumes, not drives. But, to fix issues, you'd use chkdsk on the volumes that underlie your pool. 

  22. On 4/16/2019 at 11:24 AM, Bowsa said:

    also have UPLOAD VERIFICATION turned on. It cuts upload speeds in half... but it's worth not losing data.

    To be clear: this is not necessarily true. It depends on the speed of your connection. All the upload verification means is that it will download every byte that it uploads before clearing the data from the local cache. If your downstream bandwidth is at least as large as your upstream bandwidth, it should not theoretically slow down your upload speeds at all. There will just be a delay before the data is removed from local storage while it reads the data back from the provider. CloudDrive can (and will) still move on to the next chunk upload while this process is completed. 

×
×
  • Create New...