Jump to content
Covecube Inc.
  • 0

I give up


KingMotley
 Share

Question

Ok, I really wanted to like CloudDrive.  The concept is pretty awesome, but unfortunately, it's just too unstable for my liking to consider using it any longer.  In the two months I've been using it, I've had numerous "failures":

 

  • I've had my drive (Using Google Drive) require a complete check and reupload of all the cached data (Which took the drive offline for ~17 hours), and since that is part of my DrivePool, it then made the entire DrivePool read-only.
  • It has also disconnected from Google Drive three times, and it doesn't/didn't auto reconnect, so my server was offline until I could get back to it.
  • Once it went into a never ending loop of trying to move files off the drive that hosted the write cache for the CloudDrive to the CloudDrive to make space (which then just puts it in the write cache for the cloud drive which is on the same physical disk, so no free space was ever gained). So then it'd try another file. And another.  Until I finally caught it with ~4TB of cached data to upload and disabled all the balancers.
  • Tonight, I went to rearrange one of my raid drives that hosted the cache portion of the cloudrive, and I couldn't use any of the normal utilities to change the cluster size because I'm guessing it didn't like a 63TB worth of sparse files on the drive.  It only dawned on me that was the reason the utilities were failing because one of them was saying not enough free space when the drive only had a few TB used on a 9TB parition, and after I gave up, I went to go move the files temporarily to another drive so I could reformat it and put them back and... Explorer came back and said my drive with 6TB free space couldn't hold it.  I'll admit that the utilities and file explorer should be able to handle sparse files, but programs that actually use it are so rare, most things don't account for it, and want to write the entire 63TB instead of the 2-3TB that is actually used.
  • So I gave up, and figured I'd just detach the drive and reattach it on a different drive temporarily instead of copying over the cache files... And when I reattached it, one of the folders was "corrupt".  DrivePool told me it was corrupt, chkdsk told me it was corrupt, and now half the 10TB of data I had in the cloud is borked because the folder metadata is semi-scrambled and it can't read the entire folder or any of it's subfolders.  CHKDSK can't fix it, and none of the NTFS utils I tried can fix it, but a few can actually see the correct files, but they all want me to copy them to another drive... UGH.  It took me 3 months to get that data up in to the cloud, and now half of it has to come back down.  Even when I get that half down, I'm still sort of borked because no utility I have can actually DELETE the folder, so it's going to drive DrivePool crazy.  The only way to "fix" it (other than sitting down with a disk hex editor and trying to fix it by hand) is to download the ENTIRE 10TB, wipe the drive, and then re-upload it AGAIN.

 

So far I've tried Paragon's utilities, Acronis's, EASE US's drw, Recovery4all, Auslogics File Recovery, Active Parition Recovery, NTFS Recovery Toolkit, and cgsecurity's Testdisk.  Some can see the files, but will only restore to a different drive (Meaning it has to be downloaded then reuploaded), and some will try to find them by doing a "deep scan" (which requires reading the entire 63TB drive).  Considering that I only got ~10TB up over a few months, once it's downloaded, I've just decided it isn't worth the headaches, and I'll just buy another 10-16TB drive and be done with it, without any of the extra headaches that come with the cloud drive's quirks.

 

Oh, and Plex doesn't like it when half it's library disappears/turns read-only, and NzbDrone/Sonarr/Radarr really don't like it -- it sees the files as gone, and then dutifully tries to go get them again, queueing up a zillion downloads that I already have until I catch it and shut everything down until CloudDrive comes back and then spend half a day trying to clean up the mess.

 

It's possible I would have better luck if I had an array of CloudDrives all with redundant copies, and a symmetrical connection, but even then, it'd have to contend with Google throttling issues (I could only ever get ~150mbs download speed from the drive on a 330mbs/30mbs connection), and with that the drive was very laggy, and causing some other system stability issues -- Opening file explorer taking a whole minute, copies to OTHER drives somehow getting delays occasionally, some random performance issues copying to the write cache occasionally, etc etc.  It's CLOSE, but just not close enough for my use case.  Again, it might work better as a real-time backup using DrivePool to put multiple copies up on many different CloudDrives to keep the impact of losing a drive lower, and increase the performance with smaller chunk sizes rather than a tiered storage solution I was going for SSD -> RAID -> Cloud automatically moving seldom used files (minus metadata) that it can to slower and slower tiers and moving them back up as necessary.  I had hoped with DrivePool's balancing API could I throw together a quick balancer for my needs to do just that and handle some personal edge cases like never ever put 4K video in the cloud because it can't download it fast enough to serve it, and try to keep unwatched stuff to local, and only push watched stuff go to the cloud.

 

I hope this saves someone the pain I did, and gets them to really think about how they want their system to work, and how CloudDrive might (or might not) fit in that design the right way rather than all the wrong ways I did it.

 

--Robert 

Link to comment
Share on other sites

3 answers to this question

Recommended Posts

  • 0

Hi Robert

 

I have never used cloud drive but I have had really bad experiences with cloud based storage in the past which is why I stayed away from cloud drive it's just too risky a venture for me to store my own data on personally

 

I think we are a long way off what we would want to use it for it works well with a few tb's of data but when you start installing 50 tb's I think the likes of amazon and google start pulling the plug and you imagine if cloud drive has a 1000 customers all wanting to store 50tb's that's a lot of storage and equipment needed it's just not viable from a business point of view plus for me in the uk our internet is slow compared to the rest of the world so it's just a no go I have 300/30 but I would say it's only half that 90% of the time

Link to comment
Share on other sites

  • 0

About the complete recheck of the Google Drive, there are two things you can be referring to. 

Do you mean indexing of the drive? If so, this only happens once (unless the chunk ID database is damaged/lost/deleted). 

The other is if the system/service doesn't properly shutdown, and has to verify the entire cache.   This is not normal (well the verification is, but the unsafe shutdown of the service/system isn't).

 

 

The reason for the dismounts is that you were likely seeing too many errors in a short period of time.  The 16 download threads may be the cause of that, as ... that's a bit high.  Set it to 5-10 and see if that helps. 

Additionally, there is an advanced setting that may help with this, as well. 

http://wiki.covecube.com/StableBit_CloudDrive_Advanced_Settings

Set "CloudFsDisk_MaximumConsecutiveIoFailures" to a higher value.  The caveat here is that increasing the value may increase the likelihood of the system temporarily hanging/locking up. 

 

 

 

As for the sparse file stuff, yes, that could be the cause. But in this case, this isn't really a problem with our software. We use sparse files in a very regular and well documented way. We are not doing anything "weird" or abnormal with them, actually. 

So I'm not sure why these utilities would have issues with the spare files that we're using for the cache.  

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
 Share

×
×
  • Create New...