Tried FlexRAID and Frustrated - Will DrivePool work for me?

[Parallax Abstraction]

Hey all. Forgive me if this post is long but I'm having a hard time making a decision here.

 

So I have about 9TB of data that I am keeping on a small Windows 10 home server. It's mostly media, video projects and emulation stuff (I run a retro gaming YouTube channel and have tons of emulation sets.) It's about 485,000 files in total. Until recently, it was all kept on a Windows Storage Spaces JBOD volume through a 4-bay MediaSonic ProBox that was connected directly to my main PC. Problem is, because CrashPlan has such a horribly written client, backing all this up to the cloud was taking over 3GB of memory at any given time. So I built this home server and moved stuff there. The problem was, some of the data was still residing on my desktop PC and again because of CrashPlan's crap client, it was constantly rescanning terabytes of data off my main machine over the network which also was no good. So I decided to move everything to my home server.

 

I bought 4 new 3TB WD Red drives and decided to try FlexRAID RAID-F. I copied everything off the old Storage Spaces volume to a bunch of other drives I had lying around, installed the new drives and created a FlexRAID volume. 3 data drives, 1 parity drive. This was way more effort than it should have been as FlexRAID's UI isn't great, the documentation is terrible (tons of pages on their wiki are outdated), their forums are useless and the developer won't even talk to you unless you pay a ridiculous amount of money. Still, I got it working. Then I spent the better part of 2 days copying all the data back. Then I discovered that because of the way CrashPlan reads the file system, it is not capable of doing real-time backups with FlexRAID RAID-F and can only discover changes when everything is rescanned. Because of the size and number of files in my backup, this takes literally hours each time. That's no good, I pay for CrashPlan in part for real-time backup. I'm fed up with FlexRAID and am ready to copy everything back off it again and dump it before the trial ends.

 

I've been looking at alternatives and am intrigued by DrivePool. What you seem to offer isn't RAID per ce, it's storage pooling but with the option to have some or all of the data duplicated across multiple drives to protect against failure. As I understand it, this can't heal itself from a failure like a RAID can but it is possible to have a drive die without losing the whole pool. Truth be told, I think having a full RAID plus CrashPlan is probably overdoing it for my scenario. I have 12TB of storage available with this pool of Red drives but right now, only have access to 9TB of it as I have 1 drive being used for parity. If I convert the whole thing into DrivePool, I will have an extra 3TB I can use to duplicate the most important stuff, while entrusting the rest to CrashPlan. Hopefully that explains what I'm looking for.

 

So, before I take the plunge and copy all this data twice yet again, here's my questions for confirmation:

• If a drive fails, I can remove/replace that drive without it taking out the rest of the pool?
• If I pool all 4 of these 3TB drives together, I'll get a combined pool of roughly 12TB?
• Will DrivePool appear as a normal NTFS volume so that CrashPlan can back it up in real-time? FlexRAID RAID-F does not but their T-RAID option apparently does.
• I read a recent thread where someone talked about it being a nightmare to restore FROM CrashPlan after a drive failure, largely because of CrashPlan's crap client again. Should I run into a failure and it's with unduplicated data, is this what I can expect to deal with? It's not a deal breaker, I just want to know. I hate CrashPlan's client but unfortunately, they're the only truly unlimited option I have available.

Again, sorry for the long post but after wasting so many hours on FlexRAID, I really want to make sure what I choose next will do the job. As I said, I think with CrashPlan, I don't necessarily need full RAID with this data. But I want to make sure I'm not missing anything. Thank you very much!

[Christopher (Drashna)]

Ugh, CrashPlan. Why do you have to use Java as the base?! 

But yeah.....

 

 

As for healing, no. Not currently. Implementation something like that is very complex, and prone to issues. However, we do flag mismatched files, when they are detected. 

 

 

• Yes. If a drive fails, .... well, it depends on how. If it no longer shows up in the system (or you pull it), it is flagged as missing and the pool is set as read only (to prevent potential corruption/mismatched data).  Once you've removed the missing disk, it checks a few things, reduplicates data as needed and rebalances the data as needed. 
  It immediately continues to work, though. 
  However, if you have StableBit Scanner installed on the system as well, it may detect issues before the drive outright fails. And in this case, it may trigger the software to evacuate the contents of the problem drive, hopefully before it becomes an issue. 
• Yes.  Well, rather, since each 3TB drive would show up as 2.74 TBs, you'll have roughly 10.8TBs of usable space. 
  However, this doesn't account for duplication.  This just shows the raw capacity.  And duplicated data will take up twice the space. So if you're duplicating everything, you'll have roughly 5.4TB of usable space. 
• Yes.  It shows up as a normal NTFS volume in almost all regards (I say almost, because we don't support VSS snapshots). CrashPlan should work on the pool properly. 
• It depends on how you're backing up the data. If you're backing up the pooled disks and not the pool, then yes, it definitely becomes more of an issue, because of the balancing.  
  However, I'm not really well versed in CrashPlan, as I haven't used it much.  Your best bet for answers is with the people that have used it more extensively.  Sorry. 
   

And for all of the questions, not a problem!

[Parallax Abstraction]

Thanks for the answers! So I was losing my mind with FlexRAID and got it out of my life last weekend. I'll never touch that product ever again. I decided to go ahead with a DrivePool trial and I gotta' say, I'm pretty damn impressed so far.

 

There's a real test going on as I type though. So I have 4 WD 3TB Red drives and also added in a 4TB and 3TB Seagate drive I had from a previous Windows Storage Spaces pool. I was also running a trial of StableBit Scanner. Well, this morning, Scanner informed me that the 3TB Seagate is starting to fail. I'm not surprised, it's Seagate and I got it for free originally. I've I didn't have any duplication on yet but was seriously just about to enable it when this happened. Yeah, this conversion's been a nightmare. I've asked DrivePool to remove the Seagate (which has a lot less on it than the other volumes) and it's trying but it looks like that drive is failing very fast and it's really struggling. It says "Removing drive..." but also shows no read or write activity on the pool so I don't know if it's actually working. If it can manage to actually get the data off and let me passively remove it, that'll be pretty amazing.

 

I think this combined with CrashPlan will be a great fit for my setup. My plan is to expand the pool enough that I can do full duplication so the CrashPlan issue that the other user had shouldn't really be a factor for me. We'll see how things go with this busted drive but I'll very likely be purchasing DrivePool and Scanner. I'm so glad I never got to the point of actually spending money on FlexRAID.

[Christopher (Drashna)]

I'm sorry to hear about the bad experience with FlexRAID (though, you're far from the only one, apparently ....)

 

As for the removal, there is more than just moving files around. It does a check of stuff first too. So it can take a while.  And the performance isn't listed in it's UI. You'd be better looking at Scanner or running "resmon" to check.

 

That said, the "force damaged drive removal" option may be more helpful here.  It does leave problem files behind, but it does run a bit quicker. 

 

Additionally, you can use the balancing system to accomplish this. (Pool Options -> Balancing.... -> Balancers tab). 

You can enable evacuation for SMART warnings, which will empty the Seagate drive, most likely... but this setting can be a bit aggressive (eg, if your disk is overheating, high LCC, etc).

The other is to use the "Drive Usage Limiter", and uncheck both the duplicated and unduplicated option for the disk in question.

 

Additionally, there are ways to immediately drop the disk from the pool, and then you can manually move the data off of it. It's more risky, but it may be an option worth using. 

[Parallax Abstraction]

I do have it set to evacuate on SMART but of course, because Seagate, the drive wasn't reporting SMART failure and I still don't think it is. However, it's having major read issues in spite of that. I'll leave the drive removing until I get home from work tonight and see where it's at. Most of what's on this drive is already on CrashPlan so if I have to just yank it, it's not the end of the world. Fingers crossed.

[Christopher (Drashna)]

Only if you want.  Personally, I do, because it's helpful.  But it's not because it's overly aggressive by default (which is fine by me, but not for everyone). 

[Parallax Abstraction]

Happy to say that it chugged and appeared frozen for a while but DrivePool did pull out the busted Seagate and I didn't lose anything! It's out of the pool now and when my new 8-bay MediaSonic enclosure arrives, I'll be putting in two other drives, which will give me the means to use full duplication. I am super impressed with DrivePool. When I get paid next, I'll be buying licenses for both it and Scanner and CloudDrive seems neat too and I'll be trying that out soon as well. Great work guys, I should have gone with this from the start!

[Christopher (Drashna)]

Well, glad to hear that it eventually removed the drive! 

 

And glad that the software is working very well for you! 

 

And just FYI, the StableBit Bundle is the best deal.  You basically get StableBit DrivePool and StableBit Scanner at the normal (discounted) price, and StableBit CloudDrive for $5. 

 

 

If you need anything else or have any more questions, don't hesitate to ask!

[Parallax Abstraction]

Duly noted! Even if I don't use CloudDrive, for $5 why not.

[Christopher (Drashna)]

Exactly.  

 

That, and you could use StableBit CloudDrive to create disks on a local disk, similar to how TrueCrypt/VeraCrypt works.