Jump to content
Covecube Inc.

Question

With alternate streams now available on ReFS in Windows 2012 R2 and 8.1 will Drivepool support ReFS volumes in the future?

 

NTFS has been a solid platform for many years, but MS has, with the introduction of ReFS in 2012 provided a real alternative with many new benefits, so I think a lot of us are considering a move to ReFS in the foreseable future especially for the frequent write-once-read-seldom scenarios that most of us have with media libraries and file servers. This is actually not so very different from the ZFS/BTRFS vs. EXT3/EXT4 discussions the linux community has.

 

I am wondering what thoughts you guys have about ReFS and where do you think DrivePool/Scanner is headed with regards to ReFS?

 

regards

F. Godskesen

 

 

 

"Microsoft Windows 8.1 adds support for ReFS volumes and, compared with Microsoft Windows Server 2012, both Windows Server 2012 R2 and Windows 8.1 add additional features when using ReFS, including alternate data streams and automatic correction of corruption when integrity streams are used on parity spaces."

http://en.wikipedia.org/wiki/ReFS

 

http://www.petri.co.il/4-reasons-refs-is-better-than-ntfs.htm

 

http://forum.covecube.com/discussion/1031/refs-support/p1

 

 

 

 

Share this post


Link to post
Share on other sites

24 answers to this question

Recommended Posts

  • 0

Good catch. Submitting official feature request to Alex:

https://stablebit.com/Admin/IssueAnalysis/2175

 

 

Preliminary support for ReFS has been added to the software. It is present in the beta versions, but may change. 

 

We need to investigate exactly how to implement ReFS, and what we can do with it specifically. There are some serious opportunities for advanced handling of the file system, and we want to make sure that this is done right.  For instance: 

  • Checking the integrity streams when accessing files or running a duplication pass
  • Replacing a file that has failed the integrity check with a known good copy on another disk, when duplication is enabled
  • Showing the pool as ReFS instead of NTFS
  • (more for Scanner, but) run regular integrity checks on the disks.
  • Other, unforeseen complications or features that could be added. 

 

Needless to say, this isn't as simple as just turning on the ability to use ReFS and will definitely take some time to investigate. 

 

Unfortunately, we are a small company, and our resources are limited. That means that this won't happen until we have an RC or release version of StableBit CloudDrive (which we hope is soon). 

Edited by Christopher (Drashna)

Share this post


Link to post
Share on other sites
  • 0

Alex had been very busy on another project, so he hasn't been able to get to this.

 

Actually, I was just talking to Alex about this the other day, so this is definitely still on our minds, and we would definitely like to support it.

 

However, just FYI, if we do support it, it will likely only be for Server 2012R2 version, as the Server 2012 version doesn't support some of the file system features that we use for DrivePool. Otherwise, it would require a significant redesign to implement on Server 2012.

Share this post


Link to post
Share on other sites
  • 0

Has there been any development for support of REFS in Server 2012 R2?    Drivebender has support for REFS (it's in their published FAQS), but I'd rather use Drive Pool since I'm already familiar with it on WHS 2011.   

 

Also, are there any limits on Pool Size with Drive Pool?   I'm planning to build an archive server that theoretically could grow to petabyte size over time. 

Share this post


Link to post
Share on other sites
  • 0

Once StableBit CloudDrive is released, then Alex will have a lot more time to look into supporting ReFS.

 

ANd yes, we would like to add ReFS support, but it will take time to verify that we can and change the code to be able to properly support it. 

And part of that verification is that it supports all of the file system features that we need. If it doesn't, it may not be feasible, or will require significant changes to implement support 

 

 

 

And there should be no limit to the size that the pool can grow.

Share this post


Link to post
Share on other sites
  • 0

Noted,  and I have talked to Alex about this recently (I keep on pushing it, because ... well ReFS).  

 

If we do implement it, the entire pool will have to be ReFS (eg, all the disks in it... for simplicity, as mixing file systems is a nightmare).  And it would be Server 2012R2 and up (do to the file system features we require)

Share this post


Link to post
Share on other sites
  • 0

If we do implement it, the entire pool will have to be ReFS (eg, all the disks in it... for simplicity, as mixing file systems is a nightmare).  And it would be Server 2012R2 and up (do to the file system features we require)

 

I see ReFS support in the latest DP beta release notes.  Is there anything we need to know about how to use it?

Share this post


Link to post
Share on other sites
  • 0

I see ReFS support in the latest DP beta release notes.  Is there anything we need to know about how to use it?

I was wondering when somebody would pick that up. :)

 

This is preliminary support for ReFS.  It is untested, and VERY beta. Since this is a drastically different file system, there can (and likely will be) issues with it. 

 

We do plan on testing it out, so we can add proper support for it in the future.

 

 

 

 

As for using it, we'd recommend using all ReFS disks for this, and not place anything critical on it, for the time being.  But if you want to test it out, by all means do so, and let us know if you run into any issues.

Share this post


Link to post
Share on other sites
  • 0

Any updates on this front?

 

I wonder if DrivePool can make use of ReFS's integrity streams to determine which duplicate copy to write over the other.

Share this post


Link to post
Share on other sites
  • 0

I second that I'll happily pay for version 3.0 of drivepool with those added features having lost files with a good duplicate it's becoming annoying to say the least.

Share this post


Link to post
Share on other sites
  • 0

Any updates on this front?

 

I wonder if DrivePool can make use of ReFS's integrity streams to determine which duplicate copy to write over the other.

 

Not yet. And there really, there won't be until we have a release version of StableBit CloudDrive, and even then... a while after that.

 

And as for the integrity streams think, that's one of the reasons it will take a while. We'll definitely have to look into this, and see exactly how to handle this.  Since ReFS introduces a number of new features, we have to see about how to handle all of it. Such as checking *all* of the integrity streams when running a duplication pass, or checking the integrity streams when files are accessed, or replacing a file with a good duplicate when a bad file is found, etc. 

 

This is stuff that we've discussed internally, and that we know won't really be simple to implement. 

 

There are also some other outstanding issues and requests for StableBit DrivePool, as well as StableBit Scanner that we need to address as well. 

 

I second that I'll happily pay for version 3.0 of drivepool with those added features having lost files with a good duplicate it's becoming annoying to say the least.

 

Well, sorry to hear about the lost files, but glad to hear that you'd be willing to pay for a new version (though, we don't charge for new versions, and don't have plans on changing that any time, but it's still nice to hear!). 

Share this post


Link to post
Share on other sites
  • 0

I would really like to see ReFS fully supported (2012 R2 version), and wouldn't mind helping you test it out if you could provide some way to get you helpful data from my environment. I installed the latest beta (2.2.0.665) and created a pool, files went on ok... but not sure how I can be more helpful.

Share this post


Link to post
Share on other sites
  • 0

There isn't any else to do for testing right now.

 

We've had a number of people verify that under normal usage, it appears that it does work just fine. 

 

When there is an update about the ReFS stuff, I'll make sure to post it. 

Share this post


Link to post
Share on other sites
  • 0

I've just begun using ReFS on a new pool with integrity streams enabled.  Is there any reason I would need to turn off the "Bypass file system filters" setting?  I would hope that what ReFS is doing with checksums (for both writes and reads) is at a low enough level that I can leave this setting on, but I figured I should double check.

Share this post


Link to post
Share on other sites
  • 0

No, you shouldn't need to turn off that setting. 

 

The integrity checking is done in the ReFS file system driver, meaning it's done in the kernel, when accessing files.  

 

 

There is a performance hit for leaving ReFS' integrity streams enabled. But it's pretty small, IIRC.  Even if it wasn't, I'd still recommend it, because it's a large part of WHY you'd want to use ReFS. 

Share this post


Link to post
Share on other sites
  • 0

Thanks.  Glad it works as I hoped.

 

Integrity streams are the whole reason I'm trying out ReFS on this pool, and a performance hit is basically irrelevant for my purposes.

 

One more question: if a balancer moves files between ReFS drives, is there any risk of unnoticed corruption?  That is, could a file's integrity stream on the target drive be different from on the source drive?

 

I'm guessing the answer depends on whether such corruption is possible in general between independent ReFS volumes.  If corruption is possible, perhaps DrivePool could compare integrity streams when moving/copying files.

Share this post


Link to post
Share on other sites
  • 0

Well,  right now, StableBit DrivePool uses the built in API to move files (eg, the same API that is used for drag and drops in Explorer), so it should be very stable.   

So, no, this really shouldn't change when moving between disks, I believe. 

 

 

However, while the beta versions of StableBit DrivePool do allow you to use ReFS, it's not fully supported.  There are a bunch of additional things that we could do because of the ReFS file system. But since we haven't implemented them yet.... 

 

 

And as for balancing, the files actually get copied over (as "copytemp" files, which you may or may not have seen), and then the old file is deleted once the copy is complete.  This reduces the likelihood that anything bad would happen (corruption or anything else) 

Share this post


Link to post
Share on other sites
  • 0

Oh, yeah, I understand that ReFS support is a use-at-my-own-risk feature.

 

Honestly, the data on this pool is nowhere near critical.  I was just curious whether unnoticeable corruption could occur with balancing enabled, even if the chance is low.  It sounds like it comes down to whether the built in API ensures that integrity streams match on copied files.  I briefly looked for info about that, but couldn't come up with anything.

 

Regardless, I'm happy with the current state of DrivePool's ReFS support.  Anything that further takes advantage of ReFS capabilities would be a welcome bonus.

Share this post


Link to post
Share on other sites
  • 0

Honestly, the data on this pool is nowhere near critical.  I was just curious whether unnoticeable corruption could occur with balancing enabled, even if the chance is low.  It sounds like it comes down to whether the built in API ensures that integrity streams match on copied files.  I briefly looked for info about that, but couldn't come up with anything.

 

Regardless of the status, no corruption should occur when balancing occurs. 

The integrity stream wouldn't get corrupted or changed, but would be generated for the new disk, as it's stored as part of the data structure for the files. 

 

Hopefully, in the future, when we implement proper ReFS support, we'll add a bunch more functionality to leverage the ReFS file system. 

Share this post


Link to post
Share on other sites
  • 0

+1 on the ReFS support I'm converting a server to hyper-v core 2016 all disk are ReFS and using the latest drivepool beta. Storage pool by ms is such a convoluted mess. It works when set up but using feeder ssd and parity you have to be prepared for powershell. Besides I like more control with a ui that stablebit has just hate powershell can't remember all the commands and have to reference notes. The pool you can't see what's going on or move files to disk like the ui works with drivepool and of course the scanner that emails when there is a problem on individual disk and notifications from drivepool. You can set a script in powershell for storage pools but unless you're into maintenance and a lot of fiddlin well there you go. I like to leave it alone and it works just me. KISS of drive pool and scanner is great. Now for more ReFS support from stablebit drivepool will be good cause ReFS is the future and with 2016 latest versions it is pretty good. NTFS is on it way out the door Ha!.

 

Also I would not mind paying a little more for drivepool with the self healing properties from bit rot and corruption that a mirrored pool can have with ReFS. 

Share this post


Link to post
Share on other sites
  • 0

Yeah I second that I have been using refs for a while now but only getting half the benefits from it is a bummer we know Alex is busy with cloud drive but it would be nice to even know if it is possible to get the self healing side of refs any chance of yes,no, maybe

Share this post


Link to post
Share on other sites
  • 0

Yeah I second that I have been using refs for a while now but only getting half the benefits from it is a bummer we know Alex is busy with cloud drive but it would be nice to even know if it is possible to get the self healing side of refs any chance of yes,no, maybe

 

Well, it wouldn't be the same "self healing" functionality of Storage Spaces, if we do implement anything like that. 

 

But yeah, I'm pushing for it, because it would be very, very nice to have.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...