Drivepool With Snapraid

[happimeal]

So I am very familiar with using mergerfs and snapraid, just moved my media center from an OMV with unionfs and snapraid setup back to windows. I have drivepool setup with 4 x 4tb hard drives working perfectly. I will be adding a 5tb parity drive and setting up snapraid any day now, just waiting for my drive to get here. In the meantime I do have a backblaze backup and an local backup. My question is, I do not want to have each drive in the pool have to have a drive letter, so can anyone tell me if there is a different way to do this in windows? The plugin for OMV , kinda took care of the parity, content and the configuration file, if its possible to not have each drive have to have a drive letter, does anyone have a snapraid config they could share, and maybe any tips/best practices?

 

Thanks in advance, any help is very appreciated.

[exterrestris]

DrivePool doesn't care about the drives in the pool having drive letters, so you'll be fine there. SnapRAID on the other hand needs to be able to access the drives - if you don't want drive letters for each, then you can mount each drive in an empty folder instead (as long as the drive is NTFS).

I use the following folder structure:

C:\Snapraid
    \Data
        \1
        \2
        \3
        \4
    \Parity
        \1
    \Cache
        \1
        \2

Each of the numbered folders is a mount point for a disk - SnapRAID is then configured to point at each of the data drives and the parity drive. Best practice is to set the "root" of each data disk in the SnapRAID config to the hidden Drivepool poolpart directory (because if/when you swap out a disk, Drivepool will assign a new ID to the poolpart directory, so the "root" would change). Also make sure that Drivepool isn't going to do any balancing between the data drives, as that will just cause an excessive number of changes each time SnapRAID runs.

Since SnapRAID doesn't like it when you write to the disk whilst it's doing a sync/scrub, I also use two cache drives with the SSD Optimizer plugin. These are not protected by SnapRAID, but are instead set up as another Drivepool with duplication enabled, which is then part of the main pool, like so:

SnapRAID (Pool) - Mounted as E:\
 - C:\Snapraid\Data\1
 - C:\Snapraid\Data\2
 - C:\Snapraid\Data\3
 - C:\Snapraid\Data\4
 - Data Cache (Pool)
    - C:\Snapraid\Cache\1
    - C:\Snapraid\Cache\2

This means that new files are always written to the cache drives, and will not interfere with SnapRAID if it happens to be running, but those files still have protection against disk failure. The SSD Optimizer is the only balancer enabled on the main pool (well, except for the Scanner drive evacuation balancer), and balances into the main pool once a day, so the cache disks only need to be big enough to hold the largest file that is likely to be written to the pool.

[happimeal]

17 hours ago, exterrestris said:

DrivePool doesn't care about the drives in the pool having drive letters, so you'll be fine there. SnapRAID on the other hand needs to be able to access the drives - if you don't want drive letters for each, then you can mount each drive in an empty folder instead (as long as the drive is NTFS).

I use the following folder structure:

C:\Snapraid
    \Data
        \1
        \2
        \3
        \4
    \Parity
        \1
    \Cache
        \1
        \2

Each of the numbered folders is a mount point for a disk - SnapRAID is then configured to point at each of the data drives and the parity drive. Best practice is to set the "root" of each data disk in the SnapRAID config to the hidden Drivepool poolpart directory (because if/when you swap out a disk, Drivepool will assign a new ID to the poolpart directory, so the "root" would change). Also make sure that Drivepool isn't going to do any balancing between the data drives, as that will just cause an excessive number of changes each time SnapRAID runs.

Since SnapRAID doesn't like it when you write to the disk whilst it's doing a sync/scrub, I also use two cache drives with the SSD Optimizer plugin. These are not protected by SnapRAID, but are instead set up as another Drivepool with duplication enabled, which is then part of the main pool, like so:

SnapRAID (Pool) - Mounted as E:\
 - C:\Snapraid\Data\1
 - C:\Snapraid\Data\2
 - C:\Snapraid\Data\3
 - C:\Snapraid\Data\4
 - Data Cache (Pool)
    - C:\Snapraid\Cache\1
    - C:\Snapraid\Cache\2

This means that new files are always written to the cache drives, and will not interfere with SnapRAID if it happens to be running, but those files still have protection against disk failure. The SSD Optimizer is the only balancer enabled on the main pool (well, except for the Scanner drive evacuation balancer), and balances into the main pool once a day, so the cache disks only need to be big enough to hold the largest file that is likely to be written to the pool.

Thank you for the amazing and thorough write up, it is thoroughly appreciated, I think I understand all of this, it seems pretty straight forward. Any chance you could post your snapraid.conf file, I would like to see what one looks like with a setup like this?

 

Again, thank you!!

[happimeal]

17 hours ago, exterrestris said:

DrivePool doesn't care about the drives in the pool having drive letters, so you'll be fine there. SnapRAID on the other hand needs to be able to access the drives - if you don't want drive letters for each, then you can mount each drive in an empty folder instead (as long as the drive is NTFS).

I use the following folder structure:

C:\Snapraid
    \Data
        \1
        \2
        \3
        \4
    \Parity
        \1
    \Cache
        \1
        \2

Each of the numbered folders is a mount point for a disk - SnapRAID is then configured to point at each of the data drives and the parity drive. Best practice is to set the "root" of each data disk in the SnapRAID config to the hidden Drivepool poolpart directory (because if/when you swap out a disk, Drivepool will assign a new ID to the poolpart directory, so the "root" would change). Also make sure that Drivepool isn't going to do any balancing between the data drives, as that will just cause an excessive number of changes each time SnapRAID runs.

Since SnapRAID doesn't like it when you write to the disk whilst it's doing a sync/scrub, I also use two cache drives with the SSD Optimizer plugin. These are not protected by SnapRAID, but are instead set up as another Drivepool with duplication enabled, which is then part of the main pool, like so:

SnapRAID (Pool) - Mounted as E:\
 - C:\Snapraid\Data\1
 - C:\Snapraid\Data\2
 - C:\Snapraid\Data\3
 - C:\Snapraid\Data\4
 - Data Cache (Pool)
    - C:\Snapraid\Cache\1
    - C:\Snapraid\Cache\2

This means that new files are always written to the cache drives, and will not interfere with SnapRAID if it happens to be running, but those files still have protection against disk failure. The SSD Optimizer is the only balancer enabled on the main pool (well, except for the Scanner drive evacuation balancer), and balances into the main pool once a day, so the cache disks only need to be big enough to hold the largest file that is likely to be written to the pool.

Thank you for the amazing and thorough write up, it is thoroughly appreciated, I think I understand all of this, it seems pretty straight forward. Any chance you could post your snapraid.conf file, I would like to see what one looks like with a setup like this? I am also guessing I should have all by balancers unchecked except the SSD and the StableBit Scanner? With them all turned off Drivepool with just equally fill up each drive as files are copied to it by default, correct, which is the behaviour I want, I do not want any duplication across my main pool, as I will have snapraid, and I have backblaze and a local nas  backup as well.

As far as my settings page, should I have "Do not balance automatically checked" for the behavior I am looking for?

 

Again, thank you!!

 

[exterrestris]

My snapraid.conf is pretty standard - I haven't really changed any of the defaults (so I haven't included them). I choose to keep a copy of the content file on every disk, but that's not strictly necessary.

# Defines the file to use as parity storage
# It must NOT be in a data disk
# Format: "parity FILE [,FILE] ..."
parity C:\Snapraid\Parity\1\snapraid.parity

# Defines the files to use as content list
# You can use multiple specification to store more copies
# You must have least one copy for each parity file plus one. Some more don't hurt
# They can be in the disks used for data, parity or boot,
# but each file must be in a different disk
# Format: "content FILE"
content C:\Snapraid\Parity\1\snapraid.content
content C:\Snapraid\Data\1\snapraid.content
content C:\Snapraid\Data\2\snapraid.content
content C:\Snapraid\Data\3\snapraid.content
content C:\Snapraid\Data\4\snapraid.content

# Defines the data disks to use
# The name and mount point association is relevant for parity, do not change it
# WARNING: Adding here your boot C:\ disk is NOT a good idea!
# SnapRAID is better suited for files that rarely changes!
# Format: "data DISK_NAME DISK_MOUNT_POINT"
data d1 C:\Snapraid\Data\1\PoolPart.a5f57749-53fb-4595-9bad-5912c1cfb277
data d2 C:\Snapraid\Data\2\PoolPart.7d66fe3d-5e5b-4aaf-a261-306e864c34fa
data d3 C:\Snapraid\Data\3\PoolPart.a081b030-04dc-4eb5-87ba-9fd5f38deb7b
data d4 C:\Snapraid\Data\4\PoolPart.65ea70d5-2de5-4b78-bd02-f09f32ed4426

# Excludes hidden files and directories (uncomment to enable).
#nohidden

# Defines files and directories to exclude
# Remember that all the paths are relative at the mount points
# Format: "exclude FILE"
# Format: "exclude DIR\"
# Format: "exclude \PATH\FILE"
# Format: "exclude \PATH\DIR\"
exclude *.unrecoverable
exclude Thumbs.db
exclude \$RECYCLE.BIN
exclude \System Volume Information
exclude \Program Files\
exclude \Program Files (x86)\
exclude \Windows\
exclude \.covefs

As for DrivePool balancers, yes, turn them all off. The Scanner is useful to keep if you want automatic evacuation of a failing drive, but not essential, and the SSD Optimiser is only necessary if you have a cache drive to use as a landing zone. If you don't use a landing zone, then you can disable automatic balancing, but if you do then you need it to balance periodically - once a day rather than immediately is best, as you ideally want the SnapRAID sync to happen shortly after the balance completes.

I'm not sure what the default behaviour of DrivePool is supposed to be when all balancers are disabled, but I think it does split evenly across the disks.

[happimeal]

@exterrestris Thank you again for the very well explained answer, this should be a breeze. Appreciate the time typing up such a well explained setup.

[happimeal]

@exterrestris hey, so I got a couple extra ssd's, just wondering the advantages of making them there own pool, but more specifically, making them part of the main pool, why not just leave them out of the main pool?