srcrist
-
Posts
466 -
Joined
-
Last visited
-
Days Won
36
Posts posted by srcrist
-
-
17 hours ago, steffenmand said:
We never had bigger chunks than 20 MB (me who early on pushed for higher blocks than 1 MB) EDIT: Or actually we did, but they removed it within no time. If it really was a provider issue, it should be an easy "if that then that" solution as they know what provider we are building drives on
Again, other providers *can* still use larger chunks.
Please see the changelog:
Quote.421 * For the Local Disk provider, checksumming can now be optionally enabled for chunks up to 100 MB in size. * Chunks up to 100 MB in size are now enabled for: - Amazon Cloud Drive (experimental) - Amazon S3 - Box - Dropbox - Google Cloud Storage - Google Drive - Microsoft Azure Storage - OneDrive (experimental).450 * [Issue #24914] Do not allow chunk sizes > 20 MB for any newly created cloud drives using the Google Drive provider.
This was because of issue 24914, documented here.
17 hours ago, steffenmand said:Their worries was the drive being not responsive as chunks would take time to download - which wouldnt be the case with the right setup
Just seeing the limitations now from using that block size as both servers and internet get faster as well as the data getting bigger and bigger
Again, this isn't really correct. The problem, as documented above, is that larger chunks results in more retrieval calls to particular chunks, thus triggering Google's download quota limitations. That is the problem that I could not remember. It was not because of concerns about the speed, and it was not a general problem with all providers.
EDIT: It looks like the issue with Google Drive might be resolved with an increase in the partial read size as you discussed in this post, but the code change request for that is still incomplete. So this prerequisite still isn't met. Maybe something to follow up with Christopher and Alex about.
-
3 hours ago, Christopher (Drashna) said:
Not explicitly. The google outage recently caused a rash of this, but we're no exactly sure why. If the data was corrupted/modified, the checksum would catch that. If the file just wasn't available, then it would error out and retry.
So, it's a very weird issue, that we can't really reproduce now, so it makes it VERY difficult to identify what went wrong.
Christopher, I'm sure you've spoken with Alex about this issue. I'm just wondering if there's been any discussion of infrastructural changes that might be able to improve the reliability of the file system data? I was wondering if, for example, CloudDrive could store a periodic local mirror of the file system data which could be restored in case of corruption? I don't know enough about NTFS and how the journal and such are stored on the drive to know if this is feasible or not. It just seems to me that almost everyone (who had an issue) saw file system corruption, but not corruption of the actual data on the drive. Which makes sense, because that data is frequently modified and is, as such, more vulnerable to inconsistencies on Google's part. So if that data could be given some sort of added redundancy...it might help to prevent future issues of this sort.
Do you have any thoughts on that? Or maybe Alex could chime in? My basic thought is that I'd rather have corruption of file data for individual files, which can be replaced if necessary, than lose an entire multi-terabyte volume because the file system itself (which comprises a very small minority of the actual data on the drive) gets borked. I'd love some features to take extra care with that data.
-
3 hours ago, steffenmand said:
Recently got 10 gbit internet with my server, however are seeing that the maximum size of 20 MB in blocks are causing speeds to be limited around 40 MB/s due to the response time delays pr. thread
Would love to request that it gets possible to increase the block size further, to acommodate increasing internet speeds over the coming years.
maybe a 50 MB block size and ofc. a warning showcasing that high internet speeds are required (just like with the minimal download size)
The maximum chunk size is actually a per-provider limitation. Some providers *can* use chunks larger than 20MB. During Beta, Google could use chunks as large as 100MB, if I remember right, but that caused some sort of issue, which escapes me, with Google's service and API limitations. So this isn't really a matter of CloudDrive's features, but those supported by the provider you're using.
-
4 minutes ago, pedges said:
I have two Cloud Drives saved to that Google account, and one is working. Is there any way to differentiate between the two drives and restore only data for the drive that is experiencing issues?
You'd have to figure out which drive is contained in which folder on your drive. If you open the technical details in CloudDrive, and open the drive details window, the "Uid" under the "Overview" heading will correspond to the folder name on your Google Drive. You'll have to restore EVERYTHING under that folder.
-
3 minutes ago, pedges said:
Got it.
Just a thought - I have the ability to restore the data in my Google Drive account to a certain point in time. Could I use that functionality to restore the data on my drive to point where it can be used?
You should consider that if it DOESN'T work, though, it may render the entire drive unrecoverable even using photorec or recuva. Once that data is missing from Google's servers, there's no getting it back.
-
2 minutes ago, pedges said:
Got it.
Just a thought - I have the ability to restore the data in my Google Drive account to a certain point in time. Could I use that functionality to restore the data on my drive to point where it can be used?
Now that's an interesting possibility. Maybe? Sure. Maybe. You'd want to detach the CloudDrive first, probably. It might be worth a shot. The Google outage was March 13th, so a date before that would be your best shot. If this works, it would definitely help to confirm that some sort of partial rollback is the cause of this issue.
-
well that's odd. There are other options, like photorec. Try that instead.
-
10 minutes ago, pedges said:
Would doing this ensure that all data on the drives is backed up?
Yes...with the caveat that it didn't prevent the google corruption that happened last month even if people used multiple accounts. The problem appears to be that Google rolled back data to an older version of some of the files. This is obviously fine for the actual file data itself, since that doesn't really change. But the chunks containing the filesystem data DO change. Often. So everybody's file systems were corrupted. If you mirror the pool to another pool that is on another account, and google has a similar issue, both pools will both be being modified basically simultaneously, and both pools would be corrupted if they did another rollback. It would actually be better to mirror it to an entirely different provider, or to mirror it locally.
-
8 minutes ago, pedges said:
Do I format it first, and then move forward with Recuva, or use Recuva while it is still formatted as raw?
You can just go ahead and use recuva. It's going to scan the drive sector by sector, so it doesn't matter if the file system is screwed.
-
6 minutes ago, pedges said:
Okay. I've tried Chris's suggestion and that didn't work. I also ran chkdsk and am receiving the error "Insufficient disk space to insert the index entry."
This comes up after a few lines of saying "Inserting an index entry into index $0 of file 19."
I should note that the drive I'm trying this on is a 10TB drive and has ~5TB of free space (less than half full).
I'm afraid I don't have good news for you...
I did all of the research I could, and, as far as I could tell, that just means the drive is borked. That error would usually indicate a failing hard drive, but that's obviously not the case here. It's just unrecoverable corruption.
The data on the drive is probably recoverable with recuva. I could recover mine that way, at least. Ultimately, though, I didn't have anything irreplaceable on my drive, so I just opted to wipe it and start over rather than go through and rename everything. Any files recovered will have arbitrary names. That data on the drive should be fine, though, even though the file system is trashed--if you have anything important.
-
Nobody in the community is 100% positive why this is happening (including Alex and Christopher). Christopher has said that the only way this *should* be able to happen is if the chunks were modified in the cloud. Google had a pretty significant service outage on March 13th, and we started seeing reports of these corruption issues on the forum immediately after that. My best personal theory is that whatever Google's issue was, they did a rollback, and restored older versions of some of the chunks in the cloud. This would obviously corrupt a CloudDrive drive.
The above post covers the only known process with a chance of recovery, but outcomes have not, unfortunately, been great. I too, did not notice any corruption at first, but, after noticing that files began disappearing over time, also ultimately simply wiped a 250TB CloudDrive and started over. The above process did not work for me, and, in fact, caused volumes to show up RAW--but, to be clear, they were losing files before that, and were obviously corrupt. This process did not *cause* the corruption.
-
To be clear: None of those show how much of the drive is actually used by data in the same way that the system sees it. CloudDrive simply cannot tell you that.
For some providers, like the local disk provider, all of the chunks are created when the drive is created--so "Cloud Unused" isn't a relevant piece of data. It creates the chunks to represent the entire drive all at once, so the amount of space you specify is also always the amount of space used on your storage device--in this case, a local drive.
For some providers, like Google Drive, CloudDrive does not create all of the chunks at drive creation. It only creates the chunks and uploads them to the cloud provider when data is actually written to them by the system. But the "Cloud Used" number is not the amount of space used by the system, it's the amount of space out of the total limit of the drive that you created--and it will not drop if data is deleted from the drive. Once the chunks are uploaded, they will be modified in place if data is removed from the drive.
It may be helpful to use an example. Let's say you create a 100GB drive on Google Drive, and you set a 5GB fixed cache. At first it will tell you that a few hundred MB are used on the cloud, and that the cloud unused is 99.9GB or something similar. Then let's say we put 100GB of data on the drive. The local amount will basically be our cache, or 5GB, the cloud used will now be 100GB, and the cloud unused will be some number very close to 0. Then let's say we delete 50GB of data from the drive. The local will still probably be 5GB, the cloud used will still be 100GB, and the cloud unused will still be something close to 0. Why? Because all of those chunks are already created, and they still exist on the cloud provider. CloudDrive doesn't know if those chunks contain available space or not. CloudDrive just knows that there are X number of chunks representing Y amount of space stored on the cloud provider. Your system is what knows whether or not that data is available for use--because it's the OS that manages the file system that tracks that data. Windows Explorer will report 50GB free on the 100GB drive, but CloudDrive will still be reporting the space that it has used and available on the provider. Note: Chunks are not removed from the provider once they have been created unless you resize the drive itself, because there isn't any reason to remove them unless you actually change the size of the drive.
-
12 hours ago, kuba4you said:
Thanks for the explanation. Generally I find it a bit confusing that you specify a drive size for the CloudDrive although you will never know how much space is taken. So basically I have to upload data and at some point I will get the message that the drive is full. This might be a feature request, but CloudDrive could check how much data is already on the CloudDrive and compare it to the specified drive size. Otherwise I have to do it manually by checking via FTP-Software how much data I have uploaded.
Again, CloudDrive itself has no way to know that. You specify a size because that is the amount of space that CloudDrive creates in order to store things, but whether or not space is available for USE is simply not something that the drive infrastructure is aware of. The file system, at the OS level, handles that information. You can always, of course, simply open up Windows Explorer to see how much space is available on your drive. But at the level at which CloudDrive operates, that information simply is not available.
Furthermore, the drive size can contain multiple volumes--so it can't really just look at some particular volume and compare it to the amount of data on the disk, even if the amount of data on the disk WERE representative of the amount of space available for new information. Which, again, it is not--because of how NTFS works. It would have to look at ALL volumes on your drive and compare them to the maximum size, and even knowing what volumes are on what drives requires access to the file system which it, again, does not have.
You're talking about adding entirely new levels of infrastructure to the application to accomplish something that can be accomplished by looking at ANY other disk tool in Windows. Simply looking at Windows Explorer, Disk Management, or Resource Monitor can provide you with volume usage information. The charts provided in CloudDrive are for the purpose of monitoring the usage on the provider, not the drive. Other tools exist for that, but no other tool exists to provide information about the provider usage.
-
11 hours ago, kuba4you said:
It would be great to see in the UI how much of the given CloudDrive I have created is actually still free and at which point I have to create another one or increase the existing. Currently I have to check manually how much data is already stored there. Further I find the graph for the CloudDrive not very useful if you don´t know how much space is taken. DrivePool for example does show that with the local drives.
I think there might be a fundamental misunderstanding of how CloudDrive operates here. Christopher can correct me if I'm wrong, but my understanding is that CloudDrive, as an application, is simply not aware of the filesystem usage on the drive. Think of the CloudDrive software as analogous to the firmware that operates a hard drive. It might be able to tell you if a particular portion of the drive has been written to at least once, but it can't tell you how much space is available on the drive because it simply doesn't operate at that level. In order for CloudDrive, or your HDD's firmware, to be able to tell you how much space is available for a particular purpose, it would have to somehow communicate with the file system--and neither does that. DrivePool, on the other hand, does. It operates at a file system level, and, as such, is aware of how much of the space on the disk is actually currently in use.
Another way to consider it is this: NTFS does not generally modify data on delete. So if you delete a file, NTFS simply marks that file as deleted and remembers that the space used by that file can now be used for new data. As far as the drive is concerned, nothing has changed in that area of the drive, but NTFS still considers it available. If that makes sense. So from the drive's perspective, that space is still used, even though the system doesn't actually look at it that way.
This is one of the distinctions between a tool like CloudDrive, which operates at a block-based level like a local disk drive, and a tool like Google File Stream or rClone, which operate at a file-based level, and are aware of the file system itself.
-
1 hour ago, MandalorePatriot said:
What is even the gain of using so many threads? More connections, sure, but doesn't Google throttle bandwidth after a certain amount? And it also depends on your upload speed, I'm capped by IPS @ 40Mbps roughly, so it seems only 3 upload threads is plenty.
To my knowledge, Google does not throttle bandwidth at all, no. But they do have the upload limit of 750GB/day, which means that a large number of upload threads is relatively pointless if you're constantly uploading large amounts of data. It's pretty easy to hit 75mbps or so with only 2 or 3 upload threads, and anything more than that will exceed Google's upload limit anyway. If you *know* that you're uploading less than 750GB that day anyway, though, you could theoretically get several hundred mbps performance out of 10 threads. So it's sort of situational.
Many of us do use servers with 1gbps synchronous pipes, in any case, so there is a performance benefit to more threads...at least in the short term.
But, ultimately, I'm mostly just interested in understanding the technical details from Christopher so that I can experiment and tweak. I just feel like I have a fundamental misunderstanding of how the API limits work.
-
3 hours ago, Christopher (Drashna) said:
Thread count is fine. We really haven't seen issues with 10.
However, the settings you have set WILL cause bottlenecking and issues.
Download threads: 10
Upload threads: 10
Minimum download size: 20MBPrefetch trigger: 5MB
Prefetch forward: 150 MB
Prefetch time windows: 30 secondsThe Prefetch forward should be roughly 75% of download threads x minimum download size. If you can set a higher minimum size, then you can increase the forward.
Out of curiosity, does Google set different limits for the upload and download threads in the API? I've always assumed that since I see throttling around 12-15 threads in one direction, that the total number of threads in both directions needed to be less than that. Are you saying it should be fine with 10 in each direction even though 20 in one direction would get throttled?
-
3 hours ago, Christopher (Drashna) said:
We've had a rash of these, unfortunately.
And actually, the best way to recover from this may be to do this:
- Take the drive offline in disk management
- Turn off all pinning in the CloudDrive UI
- Clear the local cache, and wait until it's down to 0 bytes (literally empty)
- Bring the cloud drive back online from disk management
You may need to repeat step #3 a couple of times to get this to work
If that doesn't help, then CHKDSK or data recovery may be needed.
Glad to see an official response on this.
Christopher, are you able to provide a quick explanation of *why* that process would help? What exactly is going on with these RAW errors, and can they be prevented in case of a Google outage in the future? Would turning on file upload verification help?
-
What result did chkdsk give you? Does it report that the volume is fine? Or is it giving you some other error?
Also open an actual support ticket here: https://stablebit.com/Contact
And run the troubleshooter and attach your support ticket number after you submit that request. The troubleshooter is located here: http://wiki.covecube.com/StableBit_Troubleshooter
This is probably a result from Google's issues a few weeks ago, but different people are experiencing different levels of corruption from that. So we'll need to figure out your specific situation to get a solution--if one exists.
-
It won't really limit your ability to upload larger amounts of data, it just throttles writes to the drive when the cache drive fills up. So if you have 150GB of local disk space on the cache drive, but you copy 200GB of data to it, the first roughly 145GB of data will copy at essentially full speed, as if you're just copying from one local drive to another, and then it will throttle the drive writes so that the last 55GB of data will slowly copy to the CloudDrive drive as chunks are uploaded from your local cache to the cloud provider.
Long story short: it isn't a problem unless high speeds are a concern. As long as you're fine copying data at roughly the speed of your upload, it will work fine no matter how much data you're writing to the CloudDrive drive.
-
SSD. Disk usage for the cache, particularly with a larger drive, can be heavy. I always suggest an SSD cache drive. You'll definitely notice a significant impact. Aside from upload space, most drives don't need or generally benefit from a cache larger than 50-100GB or so. You'll definitely get diminishing returns with anything larger than that. So speed is far more important than size.
-
I'm not sure why it would need to reindex all of the files on the drive like that, but if it does, indeed, need to search everything once, you could probably use an application like WinDirStat to do it all in one go. It'll touch every file on the drive in a few minutes.
-
No, you can only throttle on a per-drive basis.
-
That's just a warning. You thread count is a bit too high, and you're probably getting throttled. Google only allows around 15 simultaneous threads at a time. Try dropping your upload threads to 5 and keeping your download threads where they are. That warning will probably go away.
Ultimately, though, even temporary network hiccups can occasionally cause those warnings. So it might also be nothing. It's only something to worry about if it happens regularly and frequently.
-
Right. That is how chkdsk works. It repairs the corrupted volume information and will discard entries if it needs to. Now you have a healthy volume, but you need to recover the files if you can. That is a separate process.
It's important to understand how your file system works if you're going to be managing terabytes of data in the cloud.
The alternative would have been operating with an unhealthy volume and continuing to corrupt data every time you wrote to the drive.
Here is some additional information that may help you: https://www.minitool.com/data-recovery/recover-data-after-chkdsk.html
GDrive - Mounting as RAW this morning.
in General
Posted
I mean, this assumption is faulty, and important data should never be trusted exclusively to a single backup solution--cloud or otherwise. There is no such thing as the perfect backup solution. There is a reason that the 3-2-1 backup rule exists. 3 copies, 2 different types of media, at least 1 offsite location.
If you're relying exclusively on CloudDrive or any other storage solution as your only backup option, you're going to have a bad time. Google rolled back data and corrupted many CloudDrive volumes (we think), and they may do the same in the future. Google Drive is a consumer solution, after all. It is not intended to meet the needs of sensitive business data. CloudDrive is a remarkable product and takes a great number of steps to oversee the integrity of your data, but it can't work miracles. Cloud storage problems exist, and should not be relied upon exclusively.
Using CloudDrive with an enterprise provider like Google Cloud Storage, Amazon S3, or Microsoft Azure should be the first step, at a minimum, to store sensitive and important business data. Not a consumer provider like Google Drive, which is where we saw corruption last month. Linux ISOs and media files are one thing, but I wouldn't even use CloudDrive with Google Drive to store copies of family photographs that I did not have another backup for.