Refs and storage spaces

[lee1978]

Hi Chris

 

I spent most of today in a meeting with a small tech company demonstrating refs and storage spaces that was the main focus however other software was also used in the demo drivepool was one of them.

 

While I think due to reading negative reviews I have never even looked at storage spaces I have to say I was very impressed with how they presented it disconnecting drives pulling the power plug, taking a drive out and connecting to another machine to show the data is still readable and to find it's very much on par with drivepool. Also the read and write speeds were good.The closets software they compared it to was drivepool which they had much praise for the only negative comment was the lack of refs support in drivepool and the benefits of using refs so is refs support on the todo list and can it be fully integrated into drivepool to take advantage of its file healing this would eliminate the need for the file safe program we have been asking for so a bit of a trade off.

 

Lee

[jmone]

Lee, I'm new here (evaluating a move from Drive Bender) and have been using ReFS for the last couple of years with great results.  Since moving my pools over to ReFS I've not had a single data integrate issue where previously I've lost an entire pool under NTFS & CHKDSK.  

 

I've been testing Drive Pool (2.2.0.659 Beta) on one of my pools on a Windows 10 PC and discussing this with Chris about ReFS support.  This  pool is 44TB and comprises of 9 x ReFS formatted drives and have been testing it by comparing the contents of this pool against my Backup pool as I:

- Added a drive

- Removed a drive

- Moved content between drives (using the balancer and remove disk option & started/stopped these operations and rebooted the PC many times during this process)

 

I'll let Chris outline StableBits official position on ReFS support (instead of relaying it), but I have seen no issues and so far it "just works" out of the box.

 

I really really like ReFS!

 

Thanks

Nathan

[Christopher (Drashna)]

Well, glad to see that our software was being demo'ed and displayed!

 

As for ReFS support.... it's unofficially (and unintentionally) enabled by default in the beta builds (no switch needed or anything, which is not intended). From what we've seen, it seems to work just fine (but only on the 2012R2 version, due to technical limitations). 

 

The other thing, is the "self healing" part... This feature specifically requires Storage Spaces. It's able to detect errors, but without the secondary block of data (via mirroring or parity) the file system isn't capable of repairing itself properly. 

 

That said, the way the file system works makes it much less prone to corruption and damage. And normal file system recovery (aka chkdsk) doesn't require taking the entire volume offline, like it does with NTFS.  This is accomplished because there is a greater level of separation between the various parts of the storage subsystems. 

 

 

The other issues with ReFS is that there are very few tools to recover data from a failing/damaged drive, and they are much more expensive.  As well as performance issues with "Integrity checking" enabled (which shouldn't really be a problem for "Write Once, Read Many" storage) (and is enabled by default), but that may be more due to using Storage Spaces with ReFS than anything else.

 

 

However, the advantage of ReFS ... is that it apparently compares pretty well with ZFS and btffs (linux file systems, designed for pooling and better data integrity).

[lee1978]

Hi Nathan

 

Thanks that's good to hear I am going to start re formatting my spare drives and start the migration.

 

I think i just got mindset by the lot of negative comments you read on the net on both storagespaces and ReFS, but I should have known better tbh I have 16 wd greens in my pool and some are now reaching the 5 year mark with no problems at all but people say don't touch them with a barge pole I think it's all down to individual usage and like you say the demonstration I had yesterday showed it just works and works very well.

 

I think I will set up another identical pool using storage spaces which I will use for backup and do a lot of testing in either case should the worst happen to either drivepool or storagespaces i would then just be able to redirect to either one I shall keep you informed.

 

Thx

Lee

[jmone]

 

From what we've seen, it seems to work just fine (but only on the 2012R2 version, due to technical limitations). 

 

FYI the same revision of ReFS used in 2012R2 is also included on 64-Bit versions of Win8.1 / Win 10.  The only limitation with the Desktop OS implementation is that you can not format drives as ReFS (without a temporary registry hack).  I'm testing DrivePool on a ReFS poll running on Win10 64-Bit currently.

 

Note: My understanding is the DrivePool requires the ReFS revision introduced in 2012R2 / Win8.1 or later, as prior ReFS versions did not support "alternate data streams".    

[lee1978]

Hi Chris

 

They had good things to say about your software but they did point out many times the advantage of ReFS it might be worth a little more of Alex's time to get it working with drivepool like it does with storagespaces.

 

The prospect of my pool self healing is very much at the top of my list it is something and some other drivepool users have been pushing for over a year i have had the odd corrupt file and it's very annoying to think my pool has a good duplicated copy on it I can't use (my workaround on this is to pull the drive out with the corrupt file let the pool rebuild then format add it back it does work but a pain to do).

 

It would be nice to get some feed back from Alex on this as the more I see not read it's becoming ever more attractive.

 

But like said in my previous post I will run an identical copy of my pool with storagespaces and see how well it works for me.

[jmone]

Hi Lee, couple of things;

- Don't mix ReFS and NTFS drives in the same pool (I believe that DrivePool at some point will require all drives in a pool to be one or the other not a mix)

- I had many WD 2TB drives and it is only a sub-model that all failed early (WD20EARS- the first of the 4K Sector Drives).  Models before (WD20EADS) and after (WDEARX) were fine.

 

The "best" drives I've had are the Hitachi 4TB.  I'm hoping the Seagate 8TB Archives turn out to be reliable given their density and price performance for media pools.

 

Thanks

Nathan

[Christopher (Drashna)]

FYI the same revision of ReFS used in 2012R2 is also included on 64-Bit versions of Win8.1 / Win 10.  The only limitation with the Desktop OS implementation is that you can not format drives as ReFS (without a temporary registry hack).  I'm testing DrivePool on a ReFS poll running on Win10 64-Bit currently.

 

Note: My understanding is the DrivePool requires the ReFS revision introduced in 2012R2 / Win8.1 or later, as prior ReFS versions did not support "alternate data streams".    

Yes, specifically, this is why. 

Use store some of the pool information in Alternate Data Streams. And we store the duplication information "tags" as ADS as well (in fact, this is why you may see empty directory structures on the pool.... they may/do have ADS files on the directories, that mark the status and inheritance)

 

The 2012/8.0 version does NOT support Alternate Data Streams, and our software isn't the only one that doesn't like that.

 

Hi Chris

 

They had good things to say about your software but they did point out many times the advantage of ReFS it might be worth a little more of Alex's time to get it working with drivepool like it does with storagespaces.

 

The prospect of my pool self healing is very much at the top of my list it is something and some other drivepool users have been pushing for over a year i have had the odd corrupt file and it's very annoying to think my pool has a good duplicated copy on it I can't use (my workaround on this is to pull the drive out with the corrupt file let the pool rebuild then format add it back it does work but a pain to do).

 

It would be nice to get some feed back from Alex on this as the more I see not read it's becoming ever more attractive.

 

But like said in my previous post I will run an identical copy of my pool with storagespaces and see how well it works for me.

 

Absolutely. However, Alex has been busy working on StableBit CloudDrive. As it is a very complex bit of code, that's what he's been spending most of his time on (Also, switching between projects can be detrimental to workflow, especially for a product that hasn't be officially/stable released yet). 

 

We do plan on adding ReFS support. 

 

As for the "self healing" part... since we work on top of the file system, it makes it a lot more complicated. Storage Spaces works in between the physical disk layer and the ... "software" layer (VDS). Because if this difference, adding a self healing aspect would be much more difficult. But it is something we've actually discussed (if we added something like this, it would be for NTFS as well, actually). And it would be a significant chunk of code to do this. 

 

Hi Lee, couple of things;

- Don't mix ReFS and NTFS drives in the same pool (I believe that DrivePool at some point will require all drives in a pool to be one or the other not a mix)

- I had many WD 2TB drives and it is only a sub-model that all failed early (WD20EARS- the first of the 4K Sector Drives).  Models before (WD20EADS) and after (WDEARX) were fine.

 

The "best" drives I've had are the Hitachi 4TB.  I'm hoping the Seagate 8TB Archives turn out to be reliable given their density and price performance for media pools.

 

Thanks

Nathan

 

 

This.  At least for compatibility reasons, you don't want to mix file systems.  

 

Alex enabled the option so I could do some testing (and then we got busy, sorry), 

And we plan on adding some better handling of the drives, so that you won't be able to mix and match file systems at all, and so that the drive shows up as ReFS for pools with ReFS drives. 

 

 

 

 

Hi Nathan

 

Thanks that's good to hear I am going to start re formatting my spare drives and start the migration.

 

I think i just got mindset by the lot of negative comments you read on the net on both storagespaces and ReFS, but I should have known better tbh I have 16 wd greens in my pool and some are now reaching the 5 year mark with no problems at all but people say don't touch them with a barge pole I think it's all down to individual usage and like you say the demonstration I had yesterday showed it just works and works very well.

 

I think I will set up another identical pool using storage spaces which I will use for backup and do a lot of testing in either case should the worst happen to either drivepool or storagespaces i would then just be able to redirect to either one I shall keep you informed.

 

Thx

Lee

 

As for the performance issues... a big part of that is the architecture and what the file system does.  When you write to the drives, it has to compute and write the checksum for each sector. That means additional CPU and memory overhead, for each sector written. This means additional time, and heavily depends on the hardware in question. 

 

Hardware RAID gets around this by having a dedicated processor for number crunching, caching (and in some cases, batteries, for integrity).  But since Storage Spaces and ReFS is all software based, it relies on the CPU and RAM in the system heavily. 

 

Lower end systems are going to feel this impact a lot harder than higher end systems, though (in theory).  And heavy IOPS applications (such as databases) are really going to feel this.  

However, for storage solutions (write once, read many), this isn't as much of an issue... at all. 

 

 

Ironically, this is kind of the same issue with the Seagate Archive 8TB drives.  

And in case it matters, you can add a Storage Spaces array to a StableBit DrivePool pool, if you want. And in this case, you could use SSD Optimizer balancer plugin as a caching mechanism for the storage spaces array, if the performance is very bad. 

 

 

 

 

 

 

The "best" drives I've had are the Hitachi 4TB.  I'm hoping the Seagate 8TB Archives turn out to be reliable given their density and price performance for media pools.

I've heard lots of good things about them.

 

As for the Archive drives, I have 5 of these drives, and haven't had any issues with them. However, the oldest is a little under a year in age. 

[lee1978]

The 8tb drives are very tempting but that's a lot of data to lose if 1 dies and if you lose 2 I think I would pass out lol 3tb is my largest I have the space for 48 drives I have 20 at the moment and just add when I need to.

 

I have set up a new mirrored pool with refs under storage-spaces and it's currently being populated the transfer speeds are high running between 100/130, the new pool will run out of space before it completes but I want to test what happens when it does and how easy it is to solve but on the whole so far nothing seems any different

[jmone]

Yup - the underling format makes no obvious usage difference.  Not keen on Storage Spaces however but that is all 2nd hand info so I'll be interested in how you go.  What OS are you using?

 

 

FYI - Nothing beats a full backup (which is why I have a backup pool on a different PC in a different part of the house to also minimise risk of fire / left as well as drive or full pool failure).  I also preview any changes prior to writing to this second pool in case I've done some inadvertent move/add/change on my primary pool.  The downside is the extra $.

[jmone]

Just testing the read capabilities of the 8TB Seagate (ReFS) in DrivePool - No problems saturating my Gigabyte Network with large files.  Now if only Intel and MS would get their act together as I lost Link Aggregation on Win10 a couple of months ago and the "fix" is still pending.

 

https://downloadcenter.intel.com/download/25016/Network-Adapter-Driver-for-Windows-10

Important Note:  Creating Intel® ANS teams and VLANs on Microsoft Windows® 10 is currently not supported.  As a result, when created, teams and VLANs do not pass traffic.  We expect that ANS will be supported on Microsoft Windows 10 client in a future release.

[Christopher (Drashna)]

The 8tb drives are very tempting but that's a lot of data to lose if 1 dies and if you lose 2 I think I would pass out lol 3tb is my largest I have the space for 48 drives I have 20 at the moment and just add when I need to.

 

I have set up a new mirrored pool with refs under storage-spaces and it's currently being populated the transfer speeds are high running between 100/130, the new pool will run out of space before it completes but I want to test what happens when it does and how easy it is to solve but on the whole so far nothing seems any different

 

Oh, I absolutely understand that. 

 

However, I had a lot of storage before I starting grabbing these drives, and have been doing one every couple of months, so if something does happen, I should be fine.  Also for RMA's, I tend to do the advanced replacement option, so I get the drive first and then ship it back to WD/Seagate. 

 

And if you're filling the Storage Spaces array/pool all the way up, be careful. I've seen threads where doing so has caused some pretty nasty issues. I suspect that this has been since fixed, but ... just in case.

 

Yup - the underling format makes no obvious usage difference.  Not keen on Storage Spaces however but that is all 2nd hand info so I'll be interested in how you go.  What OS are you using?

 

 

FYI - Nothing beats a full backup (which is why I have a backup pool on a different PC in a different part of the house to also minimise risk of fire / left as well as drive or full pool failure).  I also preview any changes prior to writing to this second pool in case I've done some inadvertent move/add/change on my primary pool.  The downside is the extra $.

 

I'm using Server 2012R2 Essentials. 

 

And yes, multiple copies in different locations is a very good idea. Especially for any critical data.

Just testing the read capabilities of the 8TB Seagate (ReFS) in DrivePool - No problems saturating my Gigabyte Network with large files.  Now if only Intel and MS would get their act together as I lost Link Aggregation on Win10 a couple of months ago and the "fix" is still pending.

 

https://downloadcenter.intel.com/download/25016/Network-Adapter-Driver-for-Windows-10

Important Note:  Creating Intel® ANS teams and VLANs on Microsoft Windows® 10 is currently not supported.  As a result, when created, teams and VLANs do not pass traffic.  We expect that ANS will be supported on Microsoft Windows 10 client in a future release.

 

Well, for the Seagate Archive 8TB drives, the max sustained speed as listed by Seagate is 190MB/s, and I can definitely attest to that speed, actually.  

While this doesn't apply to you, as ReFS uses by default, but for anyone using NTFS, it's worth using the largest allocation unit size for these drives. The trade off for allocation unit size is negligible for drives of this capacity. 

 

 

As for the Intel stuff, that's ... unfortunate.

 

[lee1978]

hi chaps

 

Ok well this morning the new pool is full as expected it automatically goes offline now I had two options I could bring it back online line read only or add another drive so i did while adding the new drive I get the option to rebalance the pool which I think is neat so after doing that the pool becomes read/write and is now rebalancing.

 

So far so good I love drivepool and scanner but the prospect of a self healing pool has my curiosity and is something I personally have been after for the last few years.

 

I stayed away from storage spaces mainly due to the negative comments but so far so good and while I have never had any data loss with drivepool I have had a few corrupt files (I do have backups) which the only option is to delete and replace from backups even with 3x duplication on the pool, which is so frustrating so the option to have exactly the same setup with self healing abilities is getting the better of me I suppose the big question is to try prove it works easyer said than done.

[Christopher (Drashna)]

Yeah, going offline is that "nasty issue" I mentioned. It's really not graceful behavior, IMO. 

 

As for the rebalance, IIRC, that's the only time you get the option.  And that's been one of the biggest complaint for a lot people. It's being added as a powershell (only) command in the Windows Server 2016 version, but that's not for existing versions. 

 

 

And I can definitely understand the desire for self healing storage. 

[jmone]

I believe the Powershell command (Optimize-StoragePool -FriendlyName "Storage pool") is also supported in Windows 10 and Server 2106.  It is however limited to Simple and Mirrored (but not Parity Spaces)

 

https://stebet.net/microsoft-finally-adds-rebalancing-to-storage-spaces/

[Christopher (Drashna)]

That article indicates that this command is *only* available in Windows 10 and Server 2016. 

 

Not "also". 

[jmone]

Sorry poor grammar - I should have said "Win10 as well as WS2016".

 

Anyway continuing to do BitSum checks between my ReFS DrivePool and DriveBender pools and so far only found one non match (and it the damaged file was on the Backup DriveBender pool).  ReFS on DrivePool still looking good.

[lee1978]

Yeah it's got the better of me, I am going to have a backup pool on storagespaces 1:1 mirror to allow the self healing to work and I am now busy converting to refs on drivepool once done I am going to split the acces to the pools between my computers and media streamers and the first 1 to get a corrupt file loses.

[lee1978]

Been doing lots and lots of Googleing and reading today and the more you read the more

You notice apart from the odd couple of problems which Some how seem to become the only topics you find the deeper you dig there is nothing but praise for both refs and ss.

 

I found one interesting article that explicitly says refs is designed to replace ntfs and is released as a fully functioning file system and has been put through 10 of thousands of test's outlining 3 stages:

 

Stage 1

 

refs will be available and implemented in to server solutions with its main focus on file system storage.

 

Stage 2

 

Refs will be become available for client based systems to allow consumers to group together storage media for a more resilient solution

 

Stage 3

 

Refs will replace ntfs for c:

 

It goes on to say that they are supporting all 3rd party venders to help with the transition and implementation of refs.

 

There was also a short part on the sceptics where the writer says they remember when ntfs replaced fat32 "it never will" "why fix something that isn't broke" "it's not as good fat32".

 

So I hope Alex can find the time to look into this and it would be amazing if he could get the auto heal to work another part I read was. When refs detects a corrupt file it signals ss to replace with the duplicate file so that might be something that can be tapped into.

[jmone]

FYI - checked another bunch of files and it all passed with no differences .  It would take a week to check the entire pool, but it would seem that file corruption for me has been very low (if at all) on these ReFS drives.  

 

I've so far only found 1 difference between the two pools in a sample set of around 4TB / 50K files and even then I think it may have been there prior to my conversion from NTFS to ReFS on this pool as the audio file in question was put on the 2nd backup pool 4(?) years ago and I went ReFS 2 years ago on this pool.

 

Edit - Got through another compare of 2.3TB / 400K files = All good

[Christopher (Drashna)]

@Lee:

 

The biggest issues with ReFS right now are performance, and support. 

 

The biggest feature of ReFS is the integrity (error detection) aspect of the file system. The problem is that there is a huge write penalty for doing so, because the OS (and therefore) the CPU must run calculations for each sector/cluster written.  If you take a look at some of the benchmarks, there is a HUGE penalty here (I've seen as low as ~30% of the normal speed).   
However, this feature can be turned off, but you lose that detection issue. There are some other improvements that still make ReFS worth using, though. 

 

As for support, it doesn't support a lot of the file system features that some programs expect.

The 2012 version removed Alternate Data Streams (which makes that version usable for us, without a drastic rewrite), but was re-added in the 2012R2 version (because of MS SQL, actually.... oops). 

 

But compression, file encryption, hard links extended attributes and QUOTAS are not supported on ReFS.  That last one is a big thing, especially for enterprises.  Though dropping support for hard links is great for us, as we can't support that on the Pool anyways. 

 

The biggest issue IMO, though, is that there is a underwhelming number of recovery tools for ReFS. Especially consumer based recovery tools.

 

If you want a good read, a lot of this is covered on Wikipedia:

https://en.wikipedia.org/wiki/ReFS

Including other issues, and pro and cons. 

[lee1978]

Hi Chris

 

My pool has been all refs now for 2 day's I am not seeing any performance hit infact I don't see any difference at all everything is working exactly the same even with ss there is also the option to optimise the pool in ss which is the same as rebalance I think it's had a lot of feedback and there adding new features and improvements every update.

[Christopher (Drashna)]

Well, that definitely depends on the setup, and config.  But if you reformatted with NTFS, or disabled the integrity streams, you will most likely see a difference in performance. 

 

 

As for the rebalancing, that's only in Server 2016 as far as I'm aware, or under certain circumstances (such as adding additional disks). 
But then again, it's been a long while since i've used Storage Spaces.

[jmone]

I think we are mixing terms of what bit does what,

- ReFS : Just a better disk format in terms of reliability in writing and managing files than NTFS.

- ReFS and Storage Spaces : Give the ability to pool disks as Simple, Mirror, or Parity Storage Spaces.  Mirror or Parity offers background scrubbing and error correction.  Parity Storage Spaces is described by MS for "where you want to maximise capacity and you're OK with lower write performance" 

 

For me DP on ReFS is the better soln as, I just want a "Simple" pool (as I backup to another pool), and

- While I'd like the background scrubbing and error correction, I'm not willing to lose 50% of my HDD space if using SS Mirror or the performance hit if using SS Parity on my main pool but I would consider using SS Parity on my Server 2012R2 Backup Pool.

- DP offers much finer level of control regarding how pools work than SS.  

- I look forward to "official" ReFS support in Drive Pool before I'd recommend the combination generally.

 

Thanks

Nathan

[BinaryData]

After reading all of this, I'm quite disappointed. I couldn't get Storage Spaces to create a pool with my Western Digital RED NAS 3TB Drives. I couldn't remember the error off the top of my head, but TechNews had 0 support on it, and said that it may not support the drives at all. Thus why I'm using StableBit on NTFS. Also, I couldn't format as ReFS on Server 2012 R2, the option didn't even appear.

 

As for 8TB drives, hah. 10TB was just announced, and my company LOVES to be the Guinea Pig for new hardware. I won't release the name due to not knowing you guys. We recently purchased a few new Dell FX2s, and I hate the things. We also just swapped out 4x 36 Bay Supermicro's 8TB drives for 10TB ones. According to the Storage Team, they are happy. R/W is a little slower, however not enough to turn back.

 

My biggest concern about DP is redundancy. I'm new to the storage game, as I'm just a Data Center Tech, I don't get much experience with RAID or anything. I just swap parts out. My largest pool will be about 27TB of space. About 3 - 10TB will be used, and the rest will be expansion. I looked into Storage Pools because it was free (Sorry Alex!), however when I google'd about it, most of the articles were 3+ Years old, and I found no recent information.

 

 

@ Alex / Chris - You guys should allow a cache configured drive, i.e SATA/PCI-E Sata drives as the cache point. It would speed up media serving.

[Christopher (Drashna)]

After reading all of this, I'm quite disappointed. I couldn't get Storage Spaces to create a pool with my Western Digital RED NAS 3TB Drives. I couldn't remember the error off the top of my head, but TechNews had 0 support on it, and said that it may not support the drives at all. Thus why I'm using StableBit on NTFS. Also, I couldn't format as ReFS on Server 2012 R2, the option didn't even appear.

 

As for 8TB drives, hah. 10TB was just announced, and my company LOVES to be the Guinea Pig for new hardware. I won't release the name due to not knowing you guys. We recently purchased a few new Dell FX2s, and I hate the things. We also just swapped out 4x 36 Bay Supermicro's 8TB drives for 10TB ones. According to the Storage Team, they are happy. R/W is a little slower, however not enough to turn back.

 

My biggest concern about DP is redundancy. I'm new to the storage game, as I'm just a Data Center Tech, I don't get much experience with RAID or anything. I just swap parts out. My largest pool will be about 27TB of space. About 3 - 10TB will be used, and the rest will be expansion. I looked into Storage Pools because it was free (Sorry Alex!), however when I google'd about it, most of the articles were 3+ Years old, and I found no recent information.

 

 

@ Alex / Chris - You guys should allow a cache configured drive, i.e SATA/PCI-E Sata drives as the cache point. It would speed up media serving.

 

Well, for Storage Spaces, not really a lot has changed. The 2012R2 version didn't really bring any improvements to it, IIRC.

And that's very odd that you had issues with the RED drives. 

 

As for cache, well, if you have duplication (redundancy) enabled in StableBit DrivePool, it does do read striping (no, this isn't as fast as it would be with RAID, but we do a number of optimizations, based on the system:

http://stablebit.com/Support/DrivePool/2.X/Manual?Section=Performance%20Options

 

Additionally, we do have the "SSD Optimizer" that works as a write cache. 

https://stablebit.com/DrivePool/Plugins

 

But as for a read cache? The problem is that it would still have to read from the hard drives, and that will still slow it down, and would really make it take longer. 

 

 

And as for redundancy, the term we use is "duplication", specifically.  And this can be configured "globally", and it can be configured per folder.  Additionally, you can set it higher than the default "x2" duplication, for stuff that you want well protected.

http://stablebit.com/Support/DrivePool/2.X/Manual?Section=File%20Protection