Snapraid & Drivepool adding new disk

[RobinM]

Hi all,

Currently I have 5 disks of which 4 are Pooled in drivepool. The 5th drive acts as parity drive for Snapraid.

Now I want to add a new drive to the pool.

I can do 2 things, first thing is to simple add new drive to pool and add drive to snapraid config. As drivepool will fill the drives equally I expect all new files going to the newly added drive. My pool is not balancing automatically.

I actually want drivepool to spread all curent data from 4 to 5 drives. But this apposes a risk when moving files and a drive would die, from documentation of Snapraid to move files to new disk they advise following:

In case you need to move files from one data disk of the array to another disk also in the array, the 100% safe way to proceed is:

1. Copy the files to the new location.
2. Run a 'diff' command to verify that the just copied files are correctly identified as 'copy'.
3. Run a 'sync' command.
4. Delete the files from the original location.
5. Run again a 'sync' command.

This process gives the guarantee that even if disks die during the sync process, you will be able to recover everything. It also verifies that the hashes of all the moved files are matching the original ones.

I dont think I can achieve above with Drivepool as I do not  have option to let drivepool copy files to new disk, then stop balancing process. Run diff and sync command and after resume balancing process so it will remove the copied files from the original location.

Only thing I think of to achieve above is to manually copy files to new disk from existing drives. Run diff and sync command and then manually delete the files from the original location. But this is a lot of work.

Please let me know your thoughts about above or provide me suggestions how I may can achieve what I want

 

 

[Umfriend]

-

[Jaga]

When using SnapRAID it isn't advised to have a large number of changes to files on the drives.  It tends to bomb out the sync process when you do something like you're proposing (moving a lot of files between drives SnapRAID does parity for).  Even when they have instructions like that, it usually only works for small subsets of manual copies (not thousands of files).  I am saying this from experience in dealing with a large number of file changes on drives in a SnapRAID array (9 in total with 4 2-part parity drives).  I've had SnapRAID totally crap out after making large changes to the files (both deletions and a re-balance in DP), and it's simply easier to just re-create your parity from scratch.

If you're really set on adding a drive and spreading out your pool files equally afterwards, I'd recommend increasing the Drivepool balancing priority to get it done more quickly, and then completely re-doing SnapRAID parity from scratch afterwards (i.e. deleting your parity and re-building it with a new sync).  It is the fastest operation and has the least chance of getting messed up.

Yes, you are left without the parity data for a short period of time (from when you delete it to when it's re-created), but I'd rather have that situation than force the drives to do all the work you outlined and then discover that the modified/updated parity may not be any good (or watch SnapRAID crash mid-sync).  Plus - new parity creation only does read operations on the data drives, which is the least stressful to them overall.

[RobinM]

Hey Jaga,

Thank you for your response and advice. I guess your way is the only way if I really want to spread the data. We are not talking about critical data that cannot ever be replaced. Although it would take a lot of time to replace the lost files if a drive would die..

When I would just add a new drive and continue all the new files will go onto this drive until it is filled up. For a media server with multiple people streaming at the same time I don't like the idea all new files will go on the same drive as new files will be accessed the most. Spreading data means spreading load  

Hoping 1 day 10TB SSD's will get affordable

[Jaga]

Something that might be a nice feature addition for DP would be a slower, more incremental type of balancing, that does say...  3% of the pool per day.  That way you wouldn't overwhelm the on-demand parity system with massive changes, and risk losing the parity altogether.

The other way to do it is source a temporary drive (external?) that's the same size or larger than your parity drive and copy the parity info to it, and then spread files around and re-compute new parity.  Although this really only saves the parity data - with massive changes across all drives, at some point it still becomes useless.

Most drives can handle the read requests of several users simultaneously (for when you have an empty drive accepting all new content until it 'balances' with the Pool).  The only exception I can think of is either highly fragmented volumes (which a new drive filling up would not be), or several UHD media streams at the same time.  Most other cases it should be a non-issue for, and honestly - banging hard on a new drive is always a *good* test of it's durability during the warranty period.  They usually die either when new, or very old.

I think SnapRAID is a bit "rough around the edges" when it comes to dealing with a large number of changes, as the recommended way of handling it shows.  But for established sets of drives it excels.

[RobinM]

The idea of balancing only a small portion each day would be perfect. Yet I do not see this feature coming any time soon. 

[Christopher (Drashna)]

@Jaga In theory, the balancing ratio does that.