steffenmand
-
Posts
418 -
Joined
-
Last visited
-
Days Won
20
Posts posted by steffenmand
-
-
17 hours ago, Bowsa said:
Pertinent Information:
I had a major data loss weeks ago when Google had their downtime and made sure not to repeat the same mistake by enabling upload verification. I noticed the other day that Google was having issues, and the CloudUI reported the error more than 250 times.Today I checked and saw that a huge chunk of my are missing. I saw that my Cloud "Pool" was still working and there were a number of files, but then I had the intuition that this wasn't normal data loss like last time; it was isolated to one of the disks in the pool.
Pool Setup: Initially I had a 10TB Drive that had filled up, and to enlarge it, I went through the steps in the UI and it created another partition instead of enlarging the existing one. I pooled these two together to have them under the same Name and Drive Letter for organization purposes. In addition, I removed drive letters for the "pool drives" to prevent clutter.
Keeping the above in mind, I checked DrivePool, and it confirmed that the first Drive was "missing" and checking in Disk Management, the Drive is marked as "RAW". The first thing I did was follow these steps:It doesn't seem to have done anything, as the drive is still RAW. I also assigned a Drive Letter to the problem disk, and ran "CHKDSK /f". It deleted index entries, and didn't complete due to "insufficient disk space to insert the index entry" it also placed unindexed files in "lost and found".
I need some advice on how to go forward with this, I don't want to lose 10 TBs of Data again. What can I do to recover my data?
Should I detach the disk, and reattach it?
Recuva can recover it. It takes time, but most if not all should be recoverable
-
1 hour ago, Bowsa said:
Thing is, if I set the Minimum DL to like 40 MB, enable 18 threads, and 3000 MB of pre-fetch, I get 600-700 Mbps
The more threads, the more concurrent chunks it downloads, thus increases speeds. But then we just end up getting throttled
Im almost certain that minimal download is meant more for partial downloads. So setting it above chunk size don't seem to have any impact (atleast on my drives)
This is because other providers offer up to 100 MB chunks, so people might want to download only a partial chunk instead of the full 100 MB.
Logically 40 MB would also mean two downloads, thus two threads anyway
-
9 hours ago, Bowsa said:
Adding on to this:
My Pool Consists of x3 8TB HDDs, and x2 500GB SSDs. I currently have 2 TB "Free", but can't define the pool as a cache drive and have to dedicated another SSD I have dedicated as cache for the role, but its 500GB of storage limits the speed at which I can upload files.
For example if I'm uploading a bunch of files, it has to wait to clear out (upload) the SSD and then write more files which are effectively throttled. I guess I'm going to have to clear at least a good 4-6 TB of data, and assign a drive letter to one of the mechanical drives so I can set it as a cache drive for efficient uploads.
Just remember your hard upload max of 750 GB /day at GDrive. So if upload is your concern, you won't be able to do more on a single account pr. day
-
My guess is that you have the same "issue" as me where the response times pr. download thread is long, causing the time pr. thread to increase and thus slowing down speeds.
As GDrive works with maximum 20 MB chunks, it has to request a chunk and await the reply from google. in my case this takes 2300-2800 ms pr. chunk.
You can see details under "Technical details" (Found top right on the settings wheel) and then select your drive. Details are shown in the bottom.
I have requested 100 MB chunks for GDrive with a minimum required download size of 20 MB or more to help speed it up for us high-speed users.
-
2 hours ago, srcrist said:
This is incorrect. Adjusting the number of threads in CloudDrive has no appreciable impact on CPU usage. Threads, in this case, simply refers to the number of simultaneous upload and download requests that the application will let itself make at any given time. CloudDrive has limits on the number of threads that one can set in the application, so it isn't possible to test whether or not thousands of threads may eventually lead to a degradation of CPU performance, but, under the limitations that exist today, you will hit the API limitations of the provider long before any impact on CPU usage is worthy of consideration. In short: Do not adjust CloudDrive's threads based on CPU related concerns, and there is no relationship between "threads" as related to processor scheduling and "threads" as they are used in this context--other than the name.
As far as I know, this is also incorrect. A minimum download size larger than the chunk size will still download multiple chunks of contiguous data until the specified threshold is met, again, as far as I know. There is nothing conclusive in the documentation about exactly how this functions, but I have seen Christopher recommend settings higher than 20MB for Google Drive, such as he did in this case and this case, for specific situations. That would lead me to believe that higher settings do, in fact, have an impact beyond a single chunk. This is, of course, with the caveat that, as explained in the user manual, CloudDrive can still request less data than the "minimum" if the system is also requesting less (total) data.
In any case, it is true, either way, that a larger minimum download will reduce the responsiveness of the drive, and that speeds in excess of 600-700mbps can be achieved with a minimum download of 10-20MB using the settings I recommended above. So if the problem is responsiveness, dropping it may help.
The base settings that I would recommend anyone try if they are having performance issues are the following, and then play with them from there:
- 5 Upload Threads
- 10 Download Threads
- Upload Throttling enabled at approx 70mbps (to stay within Google's 750GB/day limitation)
- Upload Threshold 1MB/5 Mins
- Minimum Download 10-20MB
- Disable Background IO if you'll be streaming media so that downstream performance is not impacted by upstream requests.
If that works, you're great, and, if not, then you come back and people can try to help. But these are settings that have worked for a lot of people here over the last several years.
Minimal download size above 20 MB on google drive makes no difference on my system
-
6 hours ago, srcrist said:
The threads in CloudDrive are not CPU threads, and are not CPU limited. They are just simultaneous upload and download requests. They are limited by frequency and your provider's API limitations. Getting the correct value, for you, will be a matter of experimentation. I suggest starting with 10 download threads and 5 upload, and adjusting from there. You should *never* need more than a handful of upload threads, in any case, unless your network is misconfigured. Even 2 or 3 upload threads is capable of hitting the 70-75mpbs throughput that Google allows--permitting that your connection speed is capable of doing so. See the manual page here for more information.
This is the exact opposite of what you want. That's why. The larger your minimum download, the less responsive your drive will be. A 100 MB minmum download means that even if an application only requests 1 MB of data, CloudDrive is still downloading 100MB. There is effectively no reason for your minimum download to ever be larger than your chunk size, for most use cases. Raising your minimum is actually working against you here. Go with my above recommendations for thread count, and drop the minimum download to somewhere between 10 and 20 MB. See how that performs. We can make adjustments to your prefetcher to accommodate larger data requests over and above that amount.
EDIT: Also disable Background IO on the drive in the performance settings if read response is important.
srcrist.
Nobody said it was CPU threads. HOWEVER your CPU has a limitation as to how many threads it can run as a system at a time - thus putting it higher than the CPU capacity will only build queues. Not comparing it to the same type of threads it still means a limitation in concurrent operations.
Also faily certain that the "minimal download size" will never download anything larger than the chunk size. So if the chunk size is set to 1 MB, then 1 MB is the max download. Im pretty sure that the feature was made for partial downloads of data from larger chunks - feel free to correct me here.
-
9 hours ago, skapy said:
Hello,
i have a gdrive business account and my created clouddrive is 99% full.
Can i create a 2nd clouddrive on the same gdrive account without any problems?
If possible: Any thinks i should take care for during creating a 2nd clouddrive?
thanks in advantage
you can do several drives with no problem. just note the limitations for upload, api calls etc will be shared between the drives
-
8 hours ago, pedges said:
Is there an obvious way to tell how many threads my CPU can handle? I'm up to 10 threads and 100MB minimum download and it's still taking forever to download even smaller amounts of data.
This is the server I use: https://www.ebay.com/itm/Dell-Precision-T5600-16-Core-2-20GHz-E5-2660-16GB-RAM-2x-500GB-HDD/182388025949?hash=item2a772c6a5d:g:B4YAAOSw3Z9asP2S
my guess is that you made the drive with 1 mb chunks.
your cpu has 16 cores and can run up to 32 threads total. so 14 threads should be fine.
for high speeds always remember to do large chunk drives - else you cant get proper speeds.
-
5 minutes ago, pedges said:
I increased download threads to 4 (it was at 2) and minimum download size to 40MB.
Should I increase those two even further, or how do I know the limit to which I can increase them to?
The more threads, the more it downloads at the same time, thus faster speed. However of course you should not do more threads than your CPU can handle - as well there can be an upper limit from the cloud provider for max api calls.
I use 14 threads
-
26 minutes ago, pedges said:
I have about 5TB worth of data in a CloudDrive (via Google Drive) that I'm trying to copy to a physical hard drive and it is taking forever to do so.
I have a gigabit internet connection, so I don't think that is affecting CloudDrive's ability to download data to be able to move it.
Is there a faster way to do this that I'm not aware of? So far it's been running ~12 hours and has only been able to move ~30GB to the physical hard drive.
Did you do big chunks when creating the drive ?
Also remember to increase the thread count in I/O operations.
-
Hmm seems stablebit is accessing it for some reason. I would try taking it offline again and back online.
Chkdsk /f should fix it (or atleast tell you if they couldnt be recovered)
If stuff has to be recovered, you can use Recuva Deep Search to find and recover the files - it is a bigger task though
-
9 minutes ago, Orbitzer said:
Installed a fresh install of Windows Server 2016, remounted my cloud drives, both of them seem to be correct in size (roughly 13TB), but both are missing large amounts of files as the NTFS system just sees empty folders. It appears that a lot of the files that were not in separate subfolders are still intact. I attempted to clear my cache, then turn the drive back online and this didn't seem to help. So far file recovery software (EaseUS (unless this is not one I should be using for this specific case)) can only see some of the items. Any ideas? I would be really surprised if both clouddrive were somehow corrupted.
did you try chkdsk /f ?
i use recuva for recovery
-
18 minutes ago, Chupa said:
Thanks for the answer, I thought I could corrupt the file system in this way, however I think there is something wrong with GDrive or the cloud drive engine because like other people here in the forum we have disks with filesystems corrupted almost simultaneously I have 4 disks in 4 different accounts and one after the other they are all corrupting, I hope the developers will investigate the issue. Thank you
Been using this product for several years. Usually a drive can be fixed by either:
1. Putting the disk offline and then removing the cache. Put it online and it will try to download the filesystem files again
2. Run chkdsk DRIVELETTER: /f to fix the filesystem.
i only tried once having lost data - and that was way way back.
The last incident was due to a google drive issue causing people to get corrupted files locally (i suspect)
But as others mention, remember that this is not a back-up solution.
-
As long as it is only attached at one place at a time, you can move it around as you want
-
Detach and then reattach choosing the new cache drive
-
On 4/18/2019 at 10:03 PM, srcrist said:
A 2500-2800ms response time to google is almost *certainly* a network issue on your part, and not one that needs to be considered as a general concern. It should not take that long to send an internet packet around the entire planet, and, if you're seeing that sort of latency, you probably have bad hardware between you and your destination. A round trip to a satellite, which is the lengthiest hop in existence, is only 500ms plus the terrestrial hops. So unless you are, for some reason, beaming data to space and back several times, 2500-2800ms is simply a broken connection. I think it's important for you to consider just how unrepresentative that is, of the typical internet user, when considering whether or not someone is speaking to your issues specifically, or when requesting that changes be made to accommodate your situation. You're talking about latency that exceeds 5x even the typical satellite internet connection. The number of people dealing with such a situation are a fraction of a fraction of a percent.
Regardless of your situation, though, Google themselves have a hard limit of 750gb/day. A single thread, for most users, can hit that limit. In any case, my above response is perfectly compatible with slower response times. It's very easy to hit the maximum 70-75 mbps. No matter how many threads are taken up by reads, you can always add additional threads to make up for the read threads. If there are two read threads outstanding, you add two more for uploads. If there are 4 read threads outstanding, you add 4. If your responses are that slow, you won't get throttled no matter how many threads you add--which means that adding threads can always compensate for your connection delays. If you're still experiencing throughput issues, the problem is something else.
Thanks for your thorough explaination.
but no its not a hardware issue. running enterprise level gear in a top end datacenter. the latency is based on google response times and not the internet itself
-
9 hours ago, srcrist said:
No, you're right. It will use the upload thread count to download the data, so that will take up some number of slots for a time before those threads will be available for a new upload chunk. The mistake, though, is translating that to a loss of throughput, which isn't (or at least does not necessarily need to be) true. That is: you can achieve the same upload speeds with more threads, if you need to. 5 upload threads should be more than enough for most users to still max out the 70-75mbps of average data that Google allows you to upload each day, while also reading back the chunks in real-time.
What *is* true is that if you theoretically only had one upload thread, it would have to use the same thread to read the data back each time, and that would slow things down somewhat. But, even for residential connections, that wouldn't generally translate to a 50% reduction in throughput, because your downstream capacity is generally a lot faster than your upstream on residential connections. So it would take a lot less time to *read* each chunk than to upload it to the provider. If your connection is symmetrical, then the question simply becomes about your overall throughput. Anything over 100 mbps or so should not generally notice a difference aside from downstream congestion as a result of the verification process.
Bottom line: If you *do* suffer a performance drop with upload verification enabled, and you're working with a high-bandwidth connection (100 mbps symmetrical or more), you just need to add a few upload threads to accommodate the increased workload. As long as you can hit 70ish mbps upload, though, you're capable of maxing out Google's limitations regardless. So all of this only sort of matters in a theoretical sense.
You are not counting the thread response time into the equation. The delay you get there will end up giving you slower responses. In my case that is 2500-2800 ms. Ie. i lose 2,5 second per verification. But ofc. if you work with very few threads it is the case, but with a high amount of threads you cant really increase it further without getting throttling
-
11 minutes ago, srcrist said:
To be clear: this is not necessarily true. It depends on the speed of your connection. All the upload verification means is that it will download every byte that it uploads before clearing the data from the local cache. If your downstream bandwidth is at least as large as your upstream bandwidth, it should not theoretically slow down your upload speeds at all. There will just be a delay before the data is removed from local storage while it reads the data back from the provider. CloudDrive can (and will) still move on to the next chunk upload while this process is completed.
But as far as i saw the new upload thread doesnt start before the old one was verified - am i wrong here?
-
40 minutes ago, fly said:
I'm confused and concerned. I use CloudDrive to backup important client data for my business, as I assumed it was the perfect backup solution. Are people actually losing data?
If data is lost by the provider it can fuck up - but most can be recovered then using chkdsk - it just takes a long time when you have a lot stored
-
On 4/16/2019 at 5:26 PM, Bowsa said:
I have symmetrical 1000 gbps, and have seen my downloads reach 850-960 mbps. Are you set-up correctly with the threads?
i think you mean mbit :-P
Yes. It all depends on the response time you have. Speed is not the issue, it's my response time to google's servers
You're just lucky to be closer.
Plus i got upload verification on, that also cuts off speeds on uploads I get around 2500-2800 ms response time pr. thread and then instant download. So the less calls and the bigger download would do wonders for me
-
5 hours ago, Christopher (Drashna) said:
There is an issue here, though. With Google Drive, there is a hard limit on the number of times a file can be accessed. That's why Google Drive is limited to 20MB. So, if we allow a larger size, we would have to enforce a minimum download size proportional to the chunk size.
and this is what would fix it! people using 100 MB chunks are usually high bandwidth users, so forcing minimum download of 20 MB or more should be fine! right now we just have a bottleneck with the response times slowing down downloads.
i would go for for 100 mb downloads anyway
-
21 hours ago, srcrist said:
Again, other providers *can* still use larger chunks.
Please see the changelog:
This was because of issue 24914, documented here.
Again, this isn't really correct. The problem, as documented above, is that larger chunks results in more retrieval calls to particular chunks, thus triggering Google's download quota limitations. That is the problem that I could not remember. It was not because of concerns about the speed, and it was not a general problem with all providers.
EDIT: It looks like the issue with Google Drive might be resolved with an increase in the partial read size as you discussed in this post, but the code change request for that is still incomplete. So this prerequisite still isn't met. Maybe something to follow up with Christopher and Alex about.
Look at my Edit, i did acknowledge it was there - remember so as well - but they removed in an update or two later (updates almost every other day back then).
Yes, but that was with a use-case of partial downloads in mind. I do remember them highlighting issues with responsiveness as being the reason why we couldn't go above 20 MB, but the limitations of max amount of reads pr. day. pr file. is of course a valid argument as well. Was unaware that they kept it for the remaining providers.
Remember saying that those block sizes should be intended for high bandwidth use ie. minimal download of 50 MB-> 100MB. Still think that is the case today.In general the desire is to allow high bandwidth to utilize speeds. The bottleneck today is the responsetime pr. thread causing speed to massively drop!
A good solution is 100 MB chunks = minimal download atleast of 20 MB, again a lot of "if that then that"
I will reach out to Christopher about getting my old feature back on track
-
Would also love a feature to make a "raid" over cloud accounts.
Would also make it possible to utilize multiple gdrive accounts to increase the thread count or just avoid throttling in general. Plus its a great protection vs data corruption. But ofc. it would require more bandwidth, but for high-end 1 gbit or 10 gbit users that shouldnt make a big difference
-
On 4/11/2019 at 7:16 PM, srcrist said:
The maximum chunk size is actually a per-provider limitation. Some providers *can* use chunks larger than 20MB. During Beta, Google could use chunks as large as 100MB, if I remember right, but that caused some sort of issue, which escapes me, with Google's service and API limitations. So this isn't really a matter of CloudDrive's features, but those supported by the provider you're using.
We never had bigger chunks than 20 MB (me who early on pushed for higher blocks than 1 MB) EDIT: Or actually we did, but they removed it within no time. If it really was a provider issue, it should be an easy "if that then that" solution as they know what provider we are building drives on
Their worries was the drive being not responsive as chunks would take time to download - which wouldnt be the case with the right setup
Just seeing the limitations now from using that block size as both servers and internet get faster as well as the data getting bigger and bigger
Request: support for dynamic disks(software raid)
in General
Posted
Would be great if dynamic disks could be used as the cache drive - i use it for mirroring to avoid corruption but is unable to use it for clouddrive.
I understand why USB drives shouldn't be used, but dynamic discs is pretty much just a group of normal discs!