Jump to content
Covecube Inc.

srcrist

Members
  • Content Count

    247
  • Joined

  • Last visited

  • Days Won

    16

srcrist last won the day on April 15

srcrist had the most liked content!

About srcrist

  • Rank
    Advanced Member

Recent Profile Visitors

316 profile views
  1. EDIT: Never mind. I understand what you're looking at now and where you're getting those numbers.
  2. Sorta. But I think you might have it backwards. You wouldn't back up the data that sits on the provider to other providers, because the drive structure changes somewhat by provider. You would, instead, mount multiple providers with CloudDrive and use a tool like DrivePool to mirror the volumes to one another.
  3. A 2500-2800ms response time to google is almost *certainly* a network issue on your part, and not one that needs to be considered as a general concern. It should not take that long to send an internet packet around the entire planet, and, if you're seeing that sort of latency, you probably have bad hardware between you and your destination. A round trip to a satellite, which is the lengthiest hop in existence, is only 500ms plus the terrestrial hops. So unless you are, for some reason, beaming data to space and back several times, 2500-2800ms is simply a broken connection. I think it's important for you to consider just how unrepresentative that is, of the typical internet user, when considering whether or not someone is speaking to your issues specifically, or when requesting that changes be made to accommodate your situation. You're talking about latency that exceeds 5x even the typical satellite internet connection. The number of people dealing with such a situation are a fraction of a fraction of a percent. Regardless of your situation, though, Google themselves have a hard limit of 750gb/day. A single thread, for most users, can hit that limit. In any case, my above response is perfectly compatible with slower response times. It's very easy to hit the maximum 70-75 mbps. No matter how many threads are taken up by reads, you can always add additional threads to make up for the read threads. If there are two read threads outstanding, you add two more for uploads. If there are 4 read threads outstanding, you add 4. If your responses are that slow, you won't get throttled no matter how many threads you add--which means that adding threads can always compensate for your connection delays. If you're still experiencing throughput issues, the problem is something else.
  4. No, you're right. It will use the upload thread count to download the data, so that will take up some number of slots for a time before those threads will be available for a new upload chunk. The mistake, though, is translating that to a loss of throughput, which isn't (or at least does not necessarily need to be) true. That is: you can achieve the same upload speeds with more threads, if you need to. 5 upload threads should be more than enough for most users to still max out the 70-75mbps of average data that Google allows you to upload each day, while also reading back the chunks in real-time. What *is* true is that if you theoretically only had one upload thread, it would have to use the same thread to read the data back each time, and that would slow things down somewhat. But, even for residential connections, that wouldn't generally translate to a 50% reduction in throughput, because your downstream capacity is generally a lot faster than your upstream on residential connections. So it would take a lot less time to *read* each chunk than to upload it to the provider. If your connection is symmetrical, then the question simply becomes about your overall throughput. Anything over 100 mbps or so should not generally notice a difference aside from downstream congestion as a result of the verification process. Bottom line: If you *do* suffer a performance drop with upload verification enabled, and you're working with a high-bandwidth connection (100 mbps symmetrical or more), you just need to add a few upload threads to accommodate the increased workload. As long as you can hit 70ish mbps upload, though, you're capable of maxing out Google's limitations regardless. So all of this only sort of matters in a theoretical sense.
  5. So chkdsk is not affected by the size of the *drive*. Only the size of the *volume* (aka, generally, partition). You can have multiple volumes on the SAME CloudDrive drive. So you can still expand the size the drive and create a second VOLUME smaller than 55TB or so, and chkdsk will work with it just fine. I would, in fact, recommend this over a second drive entirely, so that you don't have multiple caches. Aside from this note, DrivePool will operate identically regardless of whether or not you're using a volume on a local disk, a CloudDrive disk, or multiple CloudDrive volumes on the same disk. DrivePool does not operate at the block level, so it doesn't concern itself with the chunks. That will all still be handled at the CloudDrive level. You can customize the DrivePool settings to configure the file placement however you want. There is no impact on performance, as DrivePool simply forwards requests to the underlying drives--CloudDrive or otherwise. Chkdsk is a file system tool. So it operates on volumes, not drives. But, to fix issues, you'd use chkdsk on the volumes that underlie your pool.
  6. To be clear: this is not necessarily true. It depends on the speed of your connection. All the upload verification means is that it will download every byte that it uploads before clearing the data from the local cache. If your downstream bandwidth is at least as large as your upstream bandwidth, it should not theoretically slow down your upload speeds at all. There will just be a delay before the data is removed from local storage while it reads the data back from the provider. CloudDrive can (and will) still move on to the next chunk upload while this process is completed.
  7. I mean, this assumption is faulty, and important data should never be trusted exclusively to a single backup solution--cloud or otherwise. There is no such thing as the perfect backup solution. There is a reason that the 3-2-1 backup rule exists. 3 copies, 2 different types of media, at least 1 offsite location. If you're relying exclusively on CloudDrive or any other storage solution as your only backup option, you're going to have a bad time. Google rolled back data and corrupted many CloudDrive volumes (we think), and they may do the same in the future. Google Drive is a consumer solution, after all. It is not intended to meet the needs of sensitive business data. CloudDrive is a remarkable product and takes a great number of steps to oversee the integrity of your data, but it can't work miracles. Cloud storage problems exist, and should not be relied upon exclusively. Using CloudDrive with an enterprise provider like Google Cloud Storage, Amazon S3, or Microsoft Azure should be the first step, at a minimum, to store sensitive and important business data. Not a consumer provider like Google Drive, which is where we saw corruption last month. Linux ISOs and media files are one thing, but I wouldn't even use CloudDrive with Google Drive to store copies of family photographs that I did not have another backup for.
  8. Again, other providers *can* still use larger chunks. Please see the changelog: This was because of issue 24914, documented here. Again, this isn't really correct. The problem, as documented above, is that larger chunks results in more retrieval calls to particular chunks, thus triggering Google's download quota limitations. That is the problem that I could not remember. It was not because of concerns about the speed, and it was not a general problem with all providers. EDIT: It looks like the issue with Google Drive might be resolved with an increase in the partial read size as you discussed in this post, but the code change request for that is still incomplete. So this prerequisite still isn't met. Maybe something to follow up with Christopher and Alex about.
  9. Christopher, I'm sure you've spoken with Alex about this issue. I'm just wondering if there's been any discussion of infrastructural changes that might be able to improve the reliability of the file system data? I was wondering if, for example, CloudDrive could store a periodic local mirror of the file system data which could be restored in case of corruption? I don't know enough about NTFS and how the journal and such are stored on the drive to know if this is feasible or not. It just seems to me that almost everyone (who had an issue) saw file system corruption, but not corruption of the actual data on the drive. Which makes sense, because that data is frequently modified and is, as such, more vulnerable to inconsistencies on Google's part. So if that data could be given some sort of added redundancy...it might help to prevent future issues of this sort. Do you have any thoughts on that? Or maybe Alex could chime in? My basic thought is that I'd rather have corruption of file data for individual files, which can be replaced if necessary, than lose an entire multi-terabyte volume because the file system itself (which comprises a very small minority of the actual data on the drive) gets borked. I'd love some features to take extra care with that data.
  10. The maximum chunk size is actually a per-provider limitation. Some providers *can* use chunks larger than 20MB. During Beta, Google could use chunks as large as 100MB, if I remember right, but that caused some sort of issue, which escapes me, with Google's service and API limitations. So this isn't really a matter of CloudDrive's features, but those supported by the provider you're using.
  11. You'd have to figure out which drive is contained in which folder on your drive. If you open the technical details in CloudDrive, and open the drive details window, the "Uid" under the "Overview" heading will correspond to the folder name on your Google Drive. You'll have to restore EVERYTHING under that folder.
  12. You should consider that if it DOESN'T work, though, it may render the entire drive unrecoverable even using photorec or recuva. Once that data is missing from Google's servers, there's no getting it back.
  13. Now that's an interesting possibility. Maybe? Sure. Maybe. You'd want to detach the CloudDrive first, probably. It might be worth a shot. The Google outage was March 13th, so a date before that would be your best shot. If this works, it would definitely help to confirm that some sort of partial rollback is the cause of this issue.
  14. well that's odd. There are other options, like photorec. Try that instead.
  15. Yes...with the caveat that it didn't prevent the google corruption that happened last month even if people used multiple accounts. The problem appears to be that Google rolled back data to an older version of some of the files. This is obviously fine for the actual file data itself, since that doesn't really change. But the chunks containing the filesystem data DO change. Often. So everybody's file systems were corrupted. If you mirror the pool to another pool that is on another account, and google has a similar issue, both pools will both be being modified basically simultaneously, and both pools would be corrupted if they did another rollback. It would actually be better to mirror it to an entirely different provider, or to mirror it locally.
×
×
  • Create New...