Jump to content

Question

Posted

I have 4x 2TB nvme SSDs and 1x 4TB SATA SSD in one pool. Ideally, I'd like all my files to be balanced evenly across the 4x nvme disks, and the duplicates stored on the SATA SSD. Eventually, I plan to add another 1x 4TB SATA SSD to equal the total volume of the 4x nvme disks, but for now that's not necessary for the amount of storage my files are taking up.

I've got it mostly working as intended, but for some reason one of the nvme disks is holding an inordinately large amount of data and I can't figure out why or how to get the files to balance out more evenly:

stor4-too-full.thumb.jpg.e778d6b5387016ca4dc45d700bd3e35b.jpg

I've got my balancing plugins setup as so:

ssd-optimizer.thumb.jpg.536c10a9dc00c9273f7c23ef2f28ec0f.jpg

disk-space-equalizer.thumb.jpg.db112add95f56a228a4693681c800d96.jpg

 

Can anyone provide advice on how I can achieve what I'm hoping to do?

Thank you!

 

7 answers to this question

Recommended Posts

  • 0
Posted

"I have 4x 2TB nvme SSDs and 1x 4TB SATA SSD in one pool. Ideally, I'd like all my files to be balanced evenly across the 4x nvme disks, and the duplicates stored on the SATA SSD. Eventually, I plan to add another 1x 4TB SATA SSD to equal the total volume of the 4x nvme disks, but for now that's not necessary for the amount of storage my files are taking up."

Hi GoreMaker! If this is the goal I would be recommending a multi-pool arrangement, not the SSD Optimizer - the latter is intended for using faster disks as cache rather than  as storage and will want to empty the NVMe disks to fill the SATA disk(s).

Example #1 (let drivepool handle the duplication and the deciding of whether user IO will be to/from your NVMe or your SATA disks):

  1. Create a pool (let's call it N) and add your 4 NVMe disks to it. Set this pool's balancing to evenly distribute via the Disk Space Equalizer plugin.
  2. Create a pool (let's call it S) and add your 1 SATA disk (and later, the second SATA disk) to it. Set it as above.
  3. Create a pool (let's call it P) and add your pools N and S to it. Set it to x2 duplication. You can now put files on P and they will be evenly distributed across your NVMe disks with their duplicates distributed on your SATA disk(s).

Example #2 (you decide whether user IO will be to/from your NVMe or SATA disks):

  1. Create a pool (let's call it N) and add your 4 NVMe disks to it. Set this pool's balancing to evenly distribute via the Disk Space Equalizer plugin.
  2. Create a pool (let's call it S) and add your 1 SATA disk (and later, the second SATA disk) to it. Set it as above.
  3. Set up a scheduled task or similar to mirror the content of your N pool to your S pool (e.g. robocopy N:\ S:\ /mir /dcopy:dat /xa:s). You can now put files on N and they will be mirrored on S whenever the task runs. Or you could set up a two-way sync via third-party utility if you wished to have N and S synchronising bidirectionally.
  • 0
Posted

That worked perfectly. I chose a variant of the first option you suggested:

- a 4x 2TB NVME pool at L: with...
     > Duplication Space Optimizer plugin enabled,
     > Disk Space Equalizer plugin enabled,
     > duplication enabled,
     > real-time duplication disabled,
     > read striping enabled

- a 1x 4TB SATA pool at M: with...
     > Disk Space Equalizer plugin enabled (for when I add disks to this pool in the future),
     > duplication disabled,
     > read-striping enabled (for when I add disks to this pool in the future)

- a pool at N: made up of pools L: and M: with...
     > the Ordered File Placement plugin enabled prioritizing the L:NVME pool for writing new files,
     > duplication enabled,
     > real-time duplication disabled,
     > read-striping disabled (I don't want to include slow M:SATA disks in a read stripe)

With these settings:
- sequential writing to pool N: happens as fast as the max speed of one NVME disk (7000+ MBps)
- sequential reading for unduplicated files happens as fast as the max speed of one NVME disk (+/- 7000 MBps)
- sequential reading for duplicated files happens almost twice as fast as the max speed of one NVME disk (12,000+ MBps)
- my usable NVME pool capacity is 3.65TB (half of 7.3TB formatted)
- all files are hosted and duplicated on the L:NVME pool
- a 3rd copy of all files exists on the M:SATA pool
- everything can be expanded or repaired at a later date by adding or replacing disks as needed
- this all happens seamlessly through a single partition at N:

I'm particularly impressed that the M:SATA pool is intelligently only keeping 1 copy of every unique file from the L:NVME pool, and not duplicating duplicates. It just lists the duplicated files on L: as "Other".

I'm using this pool to host all my Windows User directory's libraries (Documents, Pictures, Videos, Music, etc, with the sole exception of AppData). So far, this is almost as good as ZFS in some ways, and better in other ways. It blows Intel RST, Microsoft Storage Spaces, or Windows Disk Management arrays right out of the water.

I'm annoyed I haven't discovered DrivePool sooner.

  • 0
Posted

Great to hear. However, be aware that currently FileID does not behave as expected on pools, so software that assumes FileID is perfect may break badly after a reboot. Link.

  • Currently there is no ETA on a fix from StableBit.
  • In summary, generally a FileID is presumed by apps that use it to be unique and persistent on a given volume that reports itself as NTFS (collisions are actually possible albeit astronomically unlikely), however DrivePool's implementation is such that collisions after a reboot are effectively guaranteed on a given pool.
  • Affected software is that which decides that historical file A (pre-reboot) is current file B (post-reboot) because they have the same FileID and proceeds to read or write the wrong file.

TLDR, if you're not using such apps (so far some backup/sync tools, e.g. OneDrive, some have also reported FreeFileSync) you're unaffected, if you are then you should be careful not to use them with anything you keep on a pool.

  • 0
Posted

I don't really need those tools because I also setup a drive hosted on my home server via SMB using StableBit CloudDrive, and that drive is now part of the M:SATA pool for lazy (not real-time) duplication of files that are hosted on the SATA disk. That means I have a current copy of all my pool's files on a different computer. I also run a nightly backup of the directories hosted on the N: pool as a historical backup, which is accomplished with Duplicati and stored off-site. I've come across no issues with Duplicati and DrivePool so far. The resulting backups can be restored without problems.

  • 0
Posted

Based on the latest post here, I recommend disabling read-striping on all pools as a general precaution unless you can confirm your system is not affected by the hashing error.

  • 0
Posted
7 minutes ago, Shane said:

Based on the latest post here, I recommend disabling read-striping on all pools as a general precaution unless you can confirm your system is not affected by the hashing error.

Yup, can confirm this is definitely an issue. It corrupted thousands of my photos. Luckily, I have excellent backups of everything.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...