DrivePool and SnapRAID

[saiyan]

Does anyone here use SnapRAID to create parity data stored on hard drives in DrviePool's pool?

 

What kind of balancing strategy do you use? 

 

I imagine if I use a balancing strategy that move files around between different drives too often, I would need to run SnapRAID to sync parity data more often even if I don't update any data on those hard drives.

 

Thanks.

 

[saiyan]

I noticed he web page for the "Ordered File Placement" plug-in says the following:

 

 

By default, StableBit DrivePool always places new files onto the disk with the most free space. This tends to equalize the amount of free space across all the disks in the pool.

 

Is this the default balancing behavior when all balancing plug-ins are turned off (unchecked) ?

 

Thanks.

[Shane]

Yes, that is (or should be) the default behaviour if you turn off all the plug-ins.

[Christopher (Drashna)]

If all the balancers are turned off, then yes, that should be the default behavior.

[mvd]

SnapRAID + DrivePool working perfectly here.  I dont use any disk balancing plugins, though I'm trying to get the Ordered File Placement plugin to work correctly.

 

Ofcourse the trick is you have to point Snapraid at the physical disks and not the pool drive letter obviously.   In my case I have mounted my drives to folders (i.e. C:\MOUNT\Disk01, C:\MOUNT\Disk02, etc) and then the Snapraid.conf file contains the full path including the PoolPart* folder, like so:

 

 

disk d1 C:\MOUNT\Disk01\PoolPart.f261d10c-c5de-46c2-9957-625406f74cc2\
disk d2 C:\MOUNT\Disk02\PoolPart.aef49183-777f-4efc-8692-8333a700d125\
disk d3 C:\MOUNT\Disk03\PoolPart.4511417f-2412-4621-b307-64fb0cc176d1\
...etc

 

And for the existing data I had on those disks before joining them to the DrivePool, I un-hid the PoolPart folders at the commandline with "attrib -h poolpart<TAB to autocomplete foldername>", and moved the existing folders into that folder so they show up in the pool drive letter.   So initial setup is a bit of hassle but it works.

[saiyan]

Thanks for the information.

After some more testing, I think turning off all balancing plug-ins is the best strategy if I want to use SnapRaid.

[p3x-749]

As SnapRaid is offering offline parity, in what context does it matter if the pool is balanced in any way or state?

I'd say, as long as all files are covered in the SnapRaid sync, you will be able to recover them...redo balancing after a restore later, if need be.

[mvd]

As SnapRaid is offering offline parity, in what context does it matter if the pool is balanced in any way or state?

I'd say, as long as all files are covered in the SnapRaid sync, you will be able to recover them...redo balancing after a restore later, if need be.

 

Because its creating unnecessary work and burden on SnapRAID.. Making it chase a moving target.  Because as soon as files get moved around the parity is out of sync, and if you've only scheduled syncing for once or twice a day then that's a problem as its unnecessary risk.  Not to mention unnecessary wear and tear on the disks. 

 

Bottom line:  Snapshot parity benefits from files on your disk staying as static as possible, not getting shuffled around constantly to maintain equal free space on all disks (who cares).  Better to use Ordered File Placement to fill up disk at a time.

[Christopher (Drashna)]

getting shuffled around constantly to maintain equal free space on all disks (who cares)

You do mean other than the drive must be a certain percent free to defragment it? And a number of other operations (such as backups) need a certain amount of free space (for backups, for the snapshot). Those are both good reasons... Also, there is the persistent rumor/report that NTFS suffers performance issues when ~90% full or more (I'm not sure about this one... though I have seen a lot more issues when the drive has less than 1GB free...)

 

 

Otherwise, you are pretty much spot on.

[p3x-749]

Because its creating unnecessary work and burden on SnapRAID.. Making it chase a moving target.  Because as soon as files get moved around the parity is out of sync, and if you've only scheduled syncing for once or twice a day then that's a problem as its unnecessary risk.  Not to mention unnecessary wear and tear on the disks. 

 

Bottom line:  Snapshot parity benefits from files on your disk staying as static as possible, not getting shuffled around constantly to maintain equal free space on all disks (who cares).  Better to use Ordered File Placement to fill up disk at a time.

 

Well..I simply assumed that this would apply to a pool holding "static data", like a box serving recordings, movies, mp3 files, where each file would not be changed or moved around a lot....and where SnapRaid would run only after files have been addedd...daily, at most maybe.

In another scenario I'd simply would not use SnapRaid and go for a real filer.

[saiyan]

Well..I simply assumed that this would apply to a pool holding "static data", like a box serving recordings, movies, mp3 files, where each file would not be changed or moved around a lot....and where SnapRaid would run only after files have been addedd...daily, at most maybe.

In another scenario I'd simply would not use SnapRaid and go for a real filer.

 

If SnapRAID is properly configured, it will read files from individual hard drives or partitions and not the pooled volume.  So mvd is correct that if files are moved around between different hard drives by balancer plugins, SnapRAID will need to do extra work the next time you sync.  In this case, SnapRAID would think some files were deleted from one drive and new files are added to another drive.  That means when you run SnapRAID sync again, the program will need to read those files and calculate parity data even if those files have not be modified since the previously SnapRAID sync.

[p3x-749]

..just to report that snapraid works very nicely for my setup.

I have configured an iSCSI target on my ZFS-NAS and did set this up inside my Win7 box to serve as parity for snapraid on my pool-drives.

This works perfect for me, with the parity disk being part of my ZFS Array where it gets advanced availability through my online Raid setup and it is also included in my backups.

[mrstop]

I'm trying to get SnapRAID to work with Drivepool, but I'm not having much luck.  Can anyone provide some simple instructions for setup on a Windows 7 system?

[Christopher (Drashna)]

I'm not seeing any instructions at all.

 

Give me a day or two, and I'll see what I can do.

[mrstop]

I'm not seeing any instructions at all.

 

Give me a day or two, and I'll see what I can do.

 

Thanks for looking into this.  I have tried another pooling option, but really like Drivepool and Scanner. 

 

I posted in another forum and a member provided a solution that was working for him.  He essentially removed the hidden and system attributes the Drivepool applies to the Poolpart folders.  He didn't have the exact syntax off the top of his head, but he believed it was running the following on  "attrib -H -S [Poolpart folder location]  /D".  I'm a little over my skis hear, but we probably want to apply to subdirectories also.  If this is the case, I should enter the following in an elavated command prompt. 

 

1. Run command prompt as Administrator
3. Change directory to pooled drive (not sure if this is needed or not):  Enter "[Pool Drive]:" where [Pool Drive] is the letter of the drive with your Poolpart folder
5. Remove hidden & system attributes from Poolpart folder:  Enter "attrib -H -S /D /S [Poolpart foldername]
7. Repeat for each pooled drive

Questions:

 

1. Does the above sound right?
3. Are there any issues or watchouts with removing the hidden & system attributes from the Poolpart folder?
5. Will Drivepool change this back (do I need to keep changing)?
7. Is there a better solution?

[Christopher (Drashna)]

Yes, it does look like that is "right". Though it *should* be "attrib -H -S D:\PoolPart.GUID /D".

 

There should be no issues with doing this, that I'm aware of, but Alex may have input on that.

And I don't think it will change it back, but I'm not certain.

 

As for a better solution, I think FlexRAID may work better here. Less issues, I believe.

[Shawn]

The newest version of SnapRAID works fine with my hidden PoolPart directories. Doublecheck your snapraid config file. There is a setting:

 

# Excludes hidden files and directories (uncomment to enable).

#nohidden

 

Maybe you've uncomment'ed the setting, which would make snapraid skip over the hidden PoolPart directories. I tested a snapraid restore a few weeks ago and it worked fine, it restored it to the proper drive.

[Christopher (Drashna)]

Shawn, that's good to know. 
And glad to hear that it works well with DrivePool.

[mrstop]

Yes, it does look like that is "right". Though it *should* be "attrib -H -S D:\PoolPart.GUID /D".

 

There should be no issues with doing this, that I'm aware of, but Alex may have input on that.

And I don't think it will change it back, but I'm not certain.

 

As for a better solution, I think FlexRAID may work better here. Less issues, I believe.

 

Thanks for the correction.  I played a bit with FlexRAID before I moved to Drivepool for my extender.  I couldn't get it to update properly before my trial expired which sent me to search for another solution.  I may go back if Snapraid doesn't work for me.

 

The newest version of SnapRAID works fine with my hidden PoolPart directories. Doublecheck your snapraid config file. There is a setting:

 

# Excludes hidden files and directories (uncomment to enable).

#nohidden

 

Maybe you've uncomment'ed the setting, which would make snapraid skip over the hidden PoolPart directories. I tested a snapraid restore a few weeks ago and it worked fine, it restored it to the proper drive.

 

Thanks for point this out.  I checked my config file and it appears as you show above.  How would you comment/uncomment the setting?

[Shawn]

Thanks for point this out.  I checked my config file and it appears as you show above.  How would you comment/uncomment the setting?

Normally when running Drivepool, you would keep the config file the same and leave the # sign in front of nohidden, but if you remove the # sign in front of nohidden you are then telling SnapRAID to skip over the hidden PoolPart directories - unless of course you've made those folders un-hidden manually.

[Asturia]

I'm using this script with snapraid:

http://sourceforge.net/p/snapraid/discussion/1677233/thread/3afc8cb4/?limit=25&page=1#9a07

 

And it works extremely well.

[jjw7mc]

Asturia,

 

Do you happen to have that script? The files are gone from the link you provided.

[Asturia]

Here you go!

 

https://www.dropbox.com/s/1xab4mu8fff0jdh/snapraid-helper.ini

https://www.dropbox.com/s/qhjzwzwg24h1dez/snapraid-helper.ps1

[jjw7mc]

Thanks Asturia!!

[Therealjmc]

I'll post a new version of the script soon in the linked sourceforge thread - the new 6.0 Beta of Snapraid includes access to diff and list, status etc.. without spinning the parity drives. In my case Parity is powered on/off with pre/post script. And I created some functions to clean up the code a bit.

 

Hope to put my DrivePool Licence to use - but I need the truecrypt support. Just stumbled across this and wanted to let you know that there will be a new version of the powershell script

[Zombo]

Can you tell me, where the latest version of the script is? Or are the Dropbox Links from "Posted 27 February 2014 - 12:24 PM" http://community.covecube.com/index.php?/topic/94-drivepool-and-snapraid/page-2&do=findComment&comment=3055 actual?