Do you duplicate everything?

[Thronic]

In other words, do you use file duplication on entire pools? And if so, what makes the 50% capacity cost worth it to you?

I'm just procrastinating a little..

 

I want some redundancy but parity doesn't exist (I have no interest in the manual labor and handholding snapraid requires and Storage Spaces is not practical). So basically if I want parity I'd split the server role into another machine (I'm not interested in a virtualized setup) and dedicate the case holding drives to something like unraid as dedicated NAS - nothing else. Unraid also having the benefit of only loosing data on damaged drives, if parity should fail. Snapraid and Unraid, as far as I know, are the only parity solutions that has that advantage.

 

This matters to me as I can't really afford a complete backup set of drives of everything I will store - at least not if already doing mirror/parity - so I'm floating back to thinking about simple pool duplication as a somewhat decent and comfortable middle road against loss (most important things are backed up in other ways). 

I would counter some of the loss by using high level HEVC/H265 compression on media via automation tools I've already created. And perhaps in the future head towards AV1 when it's better supported.

[Shane]

Yes; x2 on all but the criticals (e.g. financials) which are x3 (they don't take much space anyway).

It is, as you say, a somewhat decent and comfortable middle road against loss (in addition to the backup machine of course). Drive dies? Entire pool is still readable while I'm getting a replacement.

I do plan to eventually - unless StableBit releases a "DrivePool v3" with parity built in - move to a ZFS NAS or similar but today is not that day (nor is this month, maybe not this year either). 

[Thronic]

Thx for replying, Shane.

Having played around with unraid the last couple of weeks, I like the zfs raidz1 performance when using pools instead of the default array.

Using 3x 1TB e-waste drives laying around, I get ~20-30MB/s writing speed on the default unraid array with single parity. This increases to ~60MB/s with turbo write/reconstruct write. It all tanks down to ~5MB/s if I do a parity check at the same time - in contrast with people on /r/DataHoarder claiming it will have no effect. I'm not sure if the flexibility is worth the performance hit for me, and I don't wanna use cache/mover to make up for it. I want storage I can write directly to in real time.

Using a raidz1 vdev of the same drives, also in unraid, I get a consistent ~112 MB/s writing speed - even when running a scrub operation at the same time. I then tried raidz2 with the same drives, just to have done it - same speedy result, which kinda impressed me quite a bit.

This is over SMB from a Windows 11 client, without any tweaks, tuning settings or modifications. 

If I park the mindset of using drivepool duplication as a middle road of not needing backups (from just a - trying to save money - standpoint), parity becomes a lot more interesting - because then I'd want a backup anyway just because of the more inherent dangerous nature of striping and how all data relies on it. 

DDDDDDPP
DDDDDDPP
D=data, P=parity (raidz2)

Would just cost 25% storage and have backup, with even backup having parity. Critical things also in 3rd party provider cloud somewhere. The backup could be located anywhere, be incremental, etc. Comparing it with just having a single duplicated DrivePool without backups it becomes a 25% cost if you consider row1=main files, row2=duplicates. Kinda worth it perhaps, but also more complex.

If I stop viewing duplication in DrivePool as a middle road for not having backups at all and convince myself I need complete backups as well, the above is rather enticing - if wanting to keep some kind of uptime redundancy. Even if I need to plan multiple drives, sizes etc. On the other hand, with zfs, all drives would be very active all the time - haven't quite made my mind up about that yet.

[Christopher (Drashna)]

I'm with shane, on the duplicating important files.  Very important stuff with x3.   But bulk media?  Unless it's personal stuff, I don't bother.  The amount of data I have makes it not worth bothering.

As for the performance hit, that's not too surprising.  There is a lot that goes into that, and even on a bare drive, you can see drops like that.   Also, you said "e-waste drives".  If these are previously used, the chances of issues are much higher, and may impact performance, too.

 

And if it makes you feel any better, I've switched most of my systems over to linux, but the server stays on Windows plus StableBit DrivePool.   There are other solutions out there, for sure, but StableBit DrivePool is just drop dead simple. 

[Thronic]

Thx for taking the time to reply Chris.

All drives were in perfect SMART condition and zero passed to trigger any potential issues before using. I used the term e-waste loosely because it's what many of my colleagues say for drives that size lately...

I do recovery and cloning professionally, so I have access to many used - but not damaged, low capacity drives. This gives me also the special benefit of actively create mechanical damage while spinning, so I can see how the systems react to true SMART problems.

Thanks for mentioning your own preferences. There are of course things (tuneables) I would need to study more if using GNU/Linux based software RAIDs, but I've used DrivePool and Scanner so many years that I just trust certain things by now. E.g. how the scanner not only is a very adamant monitoring tool, but also how it will tell you exactly what files were affected upon a pending sector that's not recoverable. This means the difference of replacing all drive data vs a single file. You're absolutely right, simplicity goes a very long way.

Still a big fan of StableBit, even if I consider  other waters once in a while.

[Christopher (Drashna)]

14 hours ago, Thronic said:

All drives were in perfect SMART condition and zero passed to trigger any potential issues before using. I used the term e-waste loosely because it's what many of my colleagues say for drives that size lately...

Ah, okay. So just another term like "spinning rust"  

14 hours ago, Thronic said:

I do recovery and cloning professionally, so I have access to many used - but not damaged, low capacity drives. This gives me also the special benefit of actively create mechanical damage while spinning, so I can see how the systems react to true SMART problems.

And that's ... very neat! 

14 hours ago, Thronic said:

Thanks for mentioning your own preferences. There are of course things (tuneables) I would need to study more if using GNU/Linux based software RAIDs, but I've used DrivePool and Scanner so many years that I just trust certain things by now. E.g. how the scanner not only is a very adamant monitoring tool, but also how it will tell you exactly what files were affected upon a pending sector that's not recoverable. This means the difference of replacing all drive data vs a single file. You're absolutely right, simplicity goes a very long way.

You are very welcome! 

And I'm in that same boat, TBH.  Zfs and btrfs look like good pooling solutions, but a lot that goes into them.  Unraid is another option, but honestly, one I'm not fond of, mainly because of how the licensing works (I ... may be spoiled by our own licensing, I admit).  

And yeah, the recovery and such for the software is great. A lot of time and effort has gone into making it so easy, and we're glad you appreciate that!  And the file recovery does make things nice, when you're in a pinch!

And I definitely understand why people keep asking for a linux and/or mac version of our software.   There is a lot to say about something that "just works".