Duplicating files without hiding them

[Salim]

Hello, I'm new to DrivePool but read great things about it so I'm considering using it to duplicate data onto 2 drives in real time.

But the fact that drivepool hides "pooled" data is kinda a drawback to me because I don't want my data to be split into hidden "pooled" data when writing to the pool and unhidden regular data when writing directly to the disks. I want any data I write to any of the disks to be written to the 2 disks simultaneously without being hidden.

The reason I wanna do this is because 1 of my drives is faster than the other, so when reading, I want to read directly from the fast drive instead of reading from the pool, and having to unhide to read is kinda a pita.

So is there a way to set up drivepool to write to 2 disks simultaneously no matter which disks I write to and without placing the files in a hidden folder?

Hope this setup can be achieved.

Thanks

[Shane]

Hi Salim! Let's take an example where you have three physical drives, D and E and F. You create a pool P - a virtual drive - and you add D and E and F to this new pool.

DrivePool handles this behind the scenes by creating a hidden "D:\PoolPart.GUID1" folder and a hidden "E:\PoolPart.GUID2" folder and a hidden "F:\PoolPart.GUID3" folder.

Any file you then put in P is stored in a special hidden folder on D or E or F. Or if you have real-time duplication enabled it will store the file simultaneously in two (or more if you want) of those hidden folders. For example, "P:\mystuff\myfile.txt" might be stored in "D:\PoolPart.GUID1\mystuff\myfile.txt" and "F:\PoolPart.GUID3\mystuff\myfile.txt".

As to which of the D, E and F drives it picks, the default is the drive(s) with the most free space at the time. It is easy to tell DrivePool to "only use these (two or more) drives to store duplicates" but getting it to "always make sure this one drive has a duplicate of what's on (two or more) other drives" is more complicated (you have to nest pools).

You use the virtual pool drive to read and write files in the pool, and if duplication is enabled DrivePool will automatically try to read from the fastest drive that holds the files. You do not need to unhide anything; in fact normally you should never access the hidden PoolPart folders yourself, as DrivePool needs to manage those autonomously to work properly.

(note: some versions of DrivePool start with Manage Pool -> Performance -> Read Striping ticked, however that feature is buggy and you should check and turn it OFF until a fix is released).

Hope this helps!

[Salim]

Thanks Shane. I'm well aware how it works. My question is if there's a way to configure it to not hide the folder and place everything unhidden on the root of the drives if real time duplication is enabled on all files and all disks.

Because if real time duplication is enabled on all files and all disks, then there's no point in storing files in PoolPart.GUID hidden folders because all files will be on all drives.

[Shane]

Ah, got it, I was confused by "having to unhide to read".

It's not possible to avoid the hidden poolpart folders, DrivePool is built on them.

[Salim]

Ok thanks. Does drivepool still not support hardlinks even if all files are duplicated on all drives? That's another reason that's deterring me from using drivepool. I understand that drivepool doesn't support hardlinks because files need to be on the same drive for hardlinks to be possible, but if all files are duplicated onto all drives, then all hardlinks will be on the same drive, so it should be possible.

[Shane]

Correct. Symbolic and junction links are supported for the pool drive, hard links are not supported (and cannot be created) for the pool drive.

Linking directly into a poolpart folder (e.g. mklink /h d:\links\file1.txt d:\poolpart.guidx\file1.txt) is technically possible but not supported as any subsequent writes via that link will result in the poolpart no longer matching with any duplicate(s) on the other drive(s) in the pool (and DrivePool's nightly consistency check will then throw up an error about it along with an option to sync the newest version of the file).

E.g. I use a set of junction links to my pool's poolparts so I can take advantage of USN journaling for rapid searches, but I never write via those junctions.

[Salim]

Ok thanks. Although drivepool seemed promising, it doesn't seem to suit my needs unfortunately.

Basically, I have 2 drives that I want to be in sync at all times, I currently use a syncing tool to sync one onto the other. But syncing is a time and resource consuming operation because every time a sync is executed, it has to compare all files between the 2 drives and then copy the missing files from drive a to drive b.

I am looking to avoid all this by writing my files onto both drives simultaneously instead of copying from 1 drive onto the other. Drivepool can do that but it doesn't support hardlinks (I have many of those) and it stores files in a hidden subfolder which I don't want.

Any idea if there's a tool out there that can do what I'm asking? I don't want to use RAID or Storage Spaces (1 of the drives is a storage space already) because I want my files to be exposed on both drives so I can access them independently (similar to how drivepool stores files).

[Shane]

Nothing I personally know that will bidirectionally sync (even near enough to) simultaneously AND expose the drives independently AND replicate hardlinks.

I'm told SyncThing can be run as two instances on different ports for bidirectional local sync (I haven't done it myself) but I'm pretty sure it doesn't replicate hardlinks.

... After quite a bit of googling: Dimio DSynchronize might be what you're after, as it claims to support bidirectional sync, transactional sync and hardlinks?

You've got me curious as to what needs both drives to be synced and independently simultaneously accessible when they're on the same machine.

[Salim]

Thanks, I'm already using SyncBack for syncing and it does replicate hardlinks, but as I said earlier, syncing solutions involve scanning both disks and then writing the differences from 1 drive to the other. This is both time and resource exhaustive.

I'm trying to write the files simultaneously from memory onto both drives at time of initial writing instead of writing onto the main drive and then syncing to the other.

The reason I want both drives to be synced and independently accessible is for backup purposes and ease of access in case of failure. I want my files to be accessible right away on the other drive if the main drive fails, without having to perform any restores or rebuilds or unhide folders.

The main drive is already a 4 disk 2-way mirror storage space, but this only protects against 1 drive failure, if for any reason 2 drives fail, my data is nuked. So I am syncing this storage space onto another single drive, so if the storage space fails for any reason, my files are immediately accessible on the single drive (and vice versa) without having to perform any restore or rebuild or unhide folders first.

So I am wanting to write the data simultaneously onto both drives (the storage space and the single drive) instead of writing to one and then syncing to the other.

[Shane]

The transactional sync should mean DSynchronize doesn't need to scan the whole drive to find changed files, just rely on the USN journal notification, but I haven't personally tested it.

Truly simultaneous writes to both drives would require some kind of virtual drive or share or RAID equivalent.

You might want to look into a hardware-based RAID6 array or switching to a ZFS-based OS/DAS/NAS (e.g. TrueNAS, Unraid, QNAP, Synology etc) if you have a need for zero-downtime despite two concurrent drive failures (and if enough money is riding on it, also looking into high-availability clusters where you've got duplicate machines not just duplicate drives).

[Salim]

Quote

The transactional sync should mean DSynchronize doesn't need to scan the whole drive to find changed files, just rely on the USN journal notification, but I haven't personally tested it.

Thanks, that's probably how SyncBack does it as well, but I'm trying to avoid syncing altogether and trying to find a way to write simultaneously at initial write.

Quote

Truly simultaneous writes to both drives would require some kind of virtual drive or RAID equivalent, or maybe Windows DFS?

Yeah I'm looking for a virtual drive solution that does that, Windows DFS might be what I'm looking for, I'll explore it but I don't think it's supported on non-server editions of Windows, it only works on Windows Server afaik and I'm running Windows 11 Pro.

Quote

You might want to look into a hardware-based RAID6 array or switching to a ZFS-based OS/DAS/NAS (e.g. TrueNAS, Unraid, QNAP, Synology etc) if you have a need for zero-downtime despite two concurrent drive failures (and if enough money is riding on it, also looking into high-availability clusters where you've got duplicate machines not just duplicate drives).

I don't want hardware-based RAID because I only have 4 drives for the main volume and I'm already using storage spaces which is similar to RAID10, I just want my data to be written on 2 separate volumes and still be exposed/accessible per volume. ZFS won't work either because it's Linux based afaik, I need a solution that woks on Windows. High-availability clusters are not needed.