File Duplication with regards to drive failures

[D4LIONS]

Greetings,

I am new to Drivepool, after terrible repeat failures with MS storage spaces. I lost 20yrs of my Plex collection and I am slowly rebuilding

I have 6 drives in my pool with main content on a drive within the pool.

I have 3TB of data on 40TB total space.

I have enabled 4x duplication as I have plenty of space, but per the manual I don't see where I can protect against multiple HD failures.

In addition on the drive pool, it only says 352B are duplicated? Shouldn't it show 3TBx4 ?

 

-A-

[Shane]

Hi! If 4x duplication is enabled for your pool and you have 3TB of data in the pool, it should indeed show 12TB of "Duplicated" data under the pie graph if everything is duplicated.

> "I have 6 drives in my pool with main content on a drive within the pool."

As a basic explanation: when you create a pool from your drives it is formed by creating, and virtually merging, a hidden poolpart folder with a unique name in the root of each drive. The pool then gets its own drive letter.

Normally that pool's drive letter is how you should interact with the pool (so that DrivePool can ensure the data goes where it's supposed to on the physical drives, gets duplicated, etc).

However as you may already be guessing from the above explanation any data that was already on those drives does not automatically get put in the pool. In the DrivePool GUI it will show up in the pie chart as "Other". If you want to include that data in the new pool, you can either "move" it from the physical drive to the pool drive via their respective drive letters (e.g. "d:\stuff" to "p:\stuff") which actually copies the data across (because as far as Windows is concerned different drive letters mean different drives), which can take a long time, or you can "seed" the pool by manually moving the data on a drive directly into that drive's hidden poolpart folder.

I recommend that you actually move the data into a new temporary folder under the poolpart folder, rather than directly into the poolpart folder itself, to avoid accidentally overlapping any existing content (or overwriting any configuration data such as duplication settings) in the pool, then finding that new folder in the pool via the normal pool drive letter to subsequently move your data to where you actually want it to be in the pool.

For example:

You have drives D, E, F and so on and you create a pool of them that gets the drive letter P.

DrivePool will create hidden folders "D:\poolpart.1" and "E:\poolpart.2" and "F:\poolpart.3" and so on.

So if you already had a lot of data in a folder "D:\stuff" you could go into "D:\poolpart.1\" and create a "D:\poolpart.1\tempunique" folder (by tempunique I mean a folder which does not already exist in the pool, i.e. there was not already a "P:\tempunique" folder) and then moving "stuff" from under "D:\" to under "D:\poolpart.1\tempunique\" so that it becomes present in the pool as "P:\tempunique\stuff". Then from there you can move it to where you want it in the pool normally.

Note that even if you have real-time duplication enabled it may not duplicate immediately as you bypassed the pool drive, but if you don't want to wait for DrivePool's nightly duplication check you can manually tell it to do so via the GUI -> cog icon top right -> Troubleshooting -> Recheck duplication.

Via the DrivePool GUI, in Manage Pool -> File protection -> Folder duplication, you can check/alter the current duplication levels for folders in the pool ("Size" is the size currently occupied without duplication, "Duplication" is the additional size currently occupied by duplication; for example if you have 4x duplication then "Duplication" should equal three times the "Size" if the duplication is up to date).

---

Tips: I recommend ensuring that in the DrivePool GUI, under Manage Pool -> Performance, that "Read striping" is NOT ticked, at least with current versions of DrivePool, as some users have reported data corruption issues with Read striping.

Also note that DrivePool pools do not allow* hardlinks, which I believe may be an issue with some methods of using Plex. It does support symlinks, if that helps.

*It's technically possible to instead hardlink directly to the files via the hidden poolparts but as that bypasses DrivePool, is incompatible with DrivePool's balancing feature and breaks duplication if the files are ever modified, it would be entirely at your own risk.

[D4LIONS]

 

I recreated the pool, now leaving the data on a single drive outside of the pool, assuming that it needed to be copied directly to the pool. 

Currently, one of my main issues is that I have Pool drives at D and all my main files on E: which is outside of the pool

 

Currently using teracopy to copy the files from E to D for back up, the teracopy keeps crashing?? I have no idea why.

 

I dont specifically have to use the pool/ D for plex, as I can connect it to E, just as a backup of data, but as the database grows it was formerly 20TB it will become problematic.

 

I have been through unraid, MS storage spaces, QNAP NAS (Outgrew it) etc.. This is my latest attempt without moving to a large commercial solution which just seems overkill. I have built my own server with 20bay expansion, I just need the right SW solution to manage backup and rebuilding from drive failures.

 

 

[Shane]

No idea why teracopy would be crashing, I don't use it sorry; have you tried copymastro?

If it helps any, I typically use Veeam for image backups of boot drives and FreeFileSync or Syncthing (depending on scenario) for backing up data drives.

For general Windows disk pooling software, my pick is DrivePool. For Windows-based pooling that must have hardlinks and/or fileid support, I would look at a CloudDrive-on-DrivePool arrangement (running a local clouddrive on at least a x3 duplicated pool).

For NAS solutions (non-Windows) you might be interested in taking a look at TrueNAS community edition. I figure at some unknown point in the future I will switch from DrivePool to TrueNAS or similar, if only because the latter has bitrot protection built in, unless StableBit comes out with a DP3 that has it.

And of course, no matter what gets used, have regular backups. "RAID is not backup" still holds true for Unraid / Storage Spaces / DrivePool / TrueNAS / etc.