Jump to content
Covecube Inc.
  • 0
Sign in to follow this  
DaveBetTech

Trial user with some pre-purchase questions (involves ReFs & bit rot)

Question

Hello new-solution-provider-I-just-heard-about. I just started the 30 day trial and had a few questions that I couldn't find the answers within the forum. Let me start by providing some history, which might be unnecessary:

============================================================================================
I've been using Storage Spaces w/ReFs & file integrity for a couple years now. I had five 2tb old drives and two 10tb drives for about 13.6tb effective after 2 way mirror. I have it all in a single volume and it wasn't over provisioned.

Recently I went into StorageSpaces and saw that my usage was 100% while the drive was only at 62% used space of 13.6 tb. Strange. It wouldn't let me drop/remove a drive in Storage Spaces, so I physically removed a 10 tb drive, formatted & moved all my data across the network to that 10 tb drive.

I then deleted that storage space and recreated it. I started a new WD extended scan on the new drive in the new Storage Space and started testing / verifying file integrity was set by a few CLI commands. Well, while doing this, I noticed that while there was ZERO data on my storage space, somehow it is now showing 40% used. Same issue.

I noticed some other weird shit too, like showing 98.3% used on some drives but not others, and Optimize drive usage not working like it historically has. I don't think I can trust Storage Spaces, which pisses me off because I just spent $125 on upgrading to W10 Workstation for better ReFs support.
============================================================================================

KEY DIFFERENCES / SIMILAR FEATURES: NOW, while researching new solutions, including Ubuntu + ZFS, I came across Stablebit DrivePool & Scanner. It seems to operate similar to Storage Spaces with some key intriguing differences:

  1. I think you can see where each copy of a file is located on which physical disk. Can't do that w/Storage Spaces AFAIK.
  2. All your drives show up in My PC with DrivePool *PLUS* your JBOD virtualized drive vs Storage Spaces which just shows the virtualized one
  3. I think it handles bit rot by using Scanner to periodically scan your files which should trigger ReFs recalculation of the CRC. Storage Spaces does this automatically, so this is just a similar feature, not different.
  4. Scanner seems to have a better GUI for evaluating health of drives. I have some old suckers in my array. I use CrystalDiskInfo and HWinfo to monitor usually.
  5. Scanner can eject drives with issues from the pool. Holy crapballs this seems awesome.

QUESTIONS FOR THE BIG BRAINS / THOSE FAMILIAR:

  1. Is my above statement correct about showing where each copy of a file is located when you enable redundancy?
  2. Is my above statement correct regarding how BitRot is handled with ReFs & Scanner? If not, how would I be able to protect against this using DrivePool &/or Scanner?

GENERALLY I'M LOOKING FOR SOMETHING THAT FITS THESE CRITERIA:

  • Ease of administration. I want to avoid using Linux if I can. I'm between a novice and an intermediate Linux user.
  • Has to be on Windows or Ubuntu because of GPU acceleration in Plex. I don't want to buy a separate NAS.
  • Has some level of redundancy and protection against Bit Rot. I want to "set it and forget it"
  • Would be 'nice' to see which physical disk houses each copy of a file.

Share this post


Link to post
Share on other sites

6 answers to this question

Recommended Posts

  • 0
1 hour ago, DaveBetTech said:
  1. I think you can see where each copy of a file is located on which physical disk. Can't do that w/Storage Spaces AFAIK.
  2. All your drives show up in My PC with DrivePool *PLUS* your JBOD virtualized drive vs Storage Spaces which just shows the virtualized one
  3. I think it handles bit rot by using Scanner to periodically scan your files which should trigger ReFs recalculation of the CRC. Storage Spaces does this automatically, so this is just a similar feature, not different.
  4. Scanner seems to have a better GUI for evaluating health of drives. I have some old suckers in my array. I use CrystalDiskInfo and HWinfo to monitor usually.
  5. Scanner can eject drives with issues from the pool. Holy crapballs this seems awesome.

QUESTIONS FOR THE BIG BRAINS / THOSE FAMILIAR:

  1. Is my above statement correct about showing where each copy of a file is located when you enable redundancy?
  2. Is my above statement correct regarding how BitRot is handled with ReFs & Scanner? If not, how would I be able to protect against this using DrivePool &/or Scanner?

GENERALLY I'M LOOKING FOR SOMETHING THAT FITS THESE CRITERIA:

  • Ease of administration. I want to avoid using Linux if I can. I'm between a novice and an intermediate Linux user.
  • Has to be on Windows or Ubuntu because of GPU acceleration in Plex. I don't want to buy a separate NAS.
  • Has some level of redundancy and protection against Bit Rot. I want to "set it and forget it"
  • Would be 'nice' to see which physical disk houses each copy of a file.


I do not think you can tell which disks contain a certain file from within DrivePool, but they are stored as normal files and visible in Windows Explorer (if you mount the individual drives to a folder, or give them drive letters) - so using Search would tell you.

I am not certain that Scanner helps with BitRot - that is why many people use SnapRAID in combination with Drivepool. It's a free software tool that allows you to add parity to your DrivePools. I don't use it myself though.

Share this post


Link to post
Share on other sites
  • 0
15 hours ago, Chris Downs said:

I do not think you can tell which disks contain a certain file from within DrivePool, but they are stored as normal files and visible in Windows Explorer (if you mount the individual drives to a folder, or give them drive letters) - so using Search would tell you.

I am not certain that Scanner helps with BitRot - that is why many people use SnapRAID in combination with Drivepool. It's a free software tool that allows you to add parity to your DrivePools. I don't use it myself though.

Thanks Chris for the reply. That's good to know about the drives being visible / accessible. I haven't added the drives to the DrivePool yet, so I haven't seen this. I assume there is still a 'drive' that shows the cumulative space across the drives (e.g. 10 tb if you have five 4tb drives in the pool w/duplicate on everything).

I'm now curious how it would use the bit rot protection in ReFs. There's a CRC that's calculated originally and recalculated each time it reads the file & compared to the original. If it's bad on a future read or scan, then it copies over the 'good' copy over the 'bad' copy. I'm now wondering if DrivePool works with ReFs in this manner. Or perhaps there's another method DrivePool & Scanner uses?

Share this post


Link to post
Share on other sites
  • 0
On 7/13/2020 at 3:45 AM, DaveBetTech said:

Thanks Chris for the reply. That's good to know about the drives being visible / accessible. I haven't added the drives to the DrivePool yet, so I haven't seen this. I assume there is still a 'drive' that shows the cumulative space across the drives (e.g. 10 tb if you have five 4tb drives in the pool w/duplicate on everything).

I'm now curious how it would use the bit rot protection in ReFs. There's a CRC that's calculated originally and recalculated each time it reads the file & compared to the original. If it's bad on a future read or scan, then it copies over the 'good' copy over the 'bad' copy. I'm now wondering if DrivePool works with ReFs in this manner. Or perhaps there's another method DrivePool & Scanner uses?

Yes, Drivepool creates a new drive for the pool. It's auto-assigned a drive letter but you can change it using the usual Windows method. It appears as a (mostly) normal drive, though disk management might only show it as "2048GB", as it has to report a value, and since the pool can grow and shrink, it supplies a fixed value. You still see the correct values in anything that isn't a partition manager though. The physical drives can have their letters removed if you want, or mounted to folders instead - I use C:\DP\ to mount all mine in case I want to access them (unlikely). Keeps the drive letter clutter down.

I have no idea about ReFS, I will tag in @Christopher (Drashna) for that part :) Or maybe someone else can answer.

Share this post


Link to post
Share on other sites
  • 0
5 hours ago, Chris Downs said:

Yes, Drivepool creates a new drive for the pool. It's auto-assigned a drive letter but you can change it using the usual Windows method. It appears as a (mostly) normal drive, though disk management might only show it as "2048GB", as it has to report a value, and since the pool can grow and shrink, it supplies a fixed value. You still see the correct values in anything that isn't a partition manager though. The physical drives can have their letters removed if you want, or mounted to folders instead - I use C:\DP\ to mount all mine in case I want to access them (unlikely). Keeps the drive letter clutter down.

I have no idea about ReFS, I will tag in @Christopher (Drashna) for that part :) Or maybe someone else can answer.

Thank you again @Chris Downs for your feedback.

I'm definitely interested in knowing how I can address bit rot and corruption with DrivePool. I'm not clear how both / either are addressed. I see how it scans for bad sectors, but more curious about periodic checking of files against a CRC like in ReFs or maybe compared against each other, dunno. Just want to maintain data integrity over the long term. I have had some of these files since the mid 90s.

Thanks in advance for anyone who can provide information about this!

Share this post


Link to post
Share on other sites
  • 0
On 7/15/2020 at 5:07 PM, DaveBetTech said:

Thank you again @Chris Downs for your feedback.

I'm definitely interested in knowing how I can address bit rot and corruption with DrivePool. I'm not clear how both / either are addressed. I see how it scans for bad sectors, but more curious about periodic checking of files against a CRC like in ReFs or maybe compared against each other, dunno. Just want to maintain data integrity over the long term. I have had some of these files since the mid 90s.

Thanks in advance for anyone who can provide information about this!

No problem. Here is a somewhat old thread on ReFS: https://community.covecube.com/index.php?/topic/3296-refs/#comment-22766

There seemed to be some concern over different ReFS versions not being compatible (yikes!), so not sure if that is still a concern?

Share this post


Link to post
Share on other sites
  • 0
4 hours ago, Chris Downs said:

No problem. Here is a somewhat old thread on ReFS: https://community.covecube.com/index.php?/topic/3296-refs/#comment-22766

There seemed to be some concern over different ReFS versions not being compatible (yikes!), so not sure if that is still a concern?

Thanks again for your input Mr. @Chris Downs. I think I'd be fine with using NTFS vs ReFs as long as there was some way of preventing bit flip / bit rot / corrupted files.

While waiting for something in here, I have went ahead and installed Win2019 and fooling around with storage pools. It's a REAL PAIN to use. I tried to add two 10tb and 3tb drives (9.09 & 1.81 effective) using 2-way-mirror and it only gave me a storage pool of 5.37 max. I obviously wanted around 11.8 tb effective, half of the space. I even tried to do 2x10tb & 2x3tb, but nope, gave me 4tb effective.

 

With StableBit DrivePool, I'm pretty sure I'd get exactly what I was looking for. I'm just looking for some kind of protection against bit rot/flip/corruption. It's not clear to me that it has this feature, but am kind of hoping since there's not a lot of flexibility w/o strictly adhering to these storage requirements.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
Sign in to follow this  

×
×
  • Create New...