Duplicating a pool then keeping it in sync across a network

[ethanol]

I want to duplicate my current pool in a different location (without paying for a cloud service).  I managed to pick up a 5 Bay Orico external USB enclosure for a really good price. I have a separate building in which I will eventually connect the Orico to a different computer than my current server which is running Drivepool. These will be connected over a home network (Cat 6) with 1GB LAN.

My current pool is 20TB (10TB duplicated). I now have enough drives/space in the Orico and want to achieve the following : 

- Full initial copy of the current pool by connecting the Orico to my current Server

- Ongoing mirror so that both the Server with the internal drives and existing DrivePool and the remote computer with the Orico attached are exact replicas

- Be able to read locally from the DrivePool at the new computer (ie its not only for Backup)

 

Now my questions : 

- Whats the best way to initially duplicate the pool? Set-up a new Pool and then just copy in File Manager?

- How should I set-up the remote sync - would I need cloud Drive for that?

- Would I also need DrivePool on the new computer (if I'm using Cloud Drive)?

- Is there a deal on the bundle (DP, Scanner, CD) for existing customers?  

 

[Shane]

- Whats the best way to initially duplicate the pool? Set-up a new Pool and then just copy in File Manager?

I'd go with that, yes.

- How should I set-up the remote sync - would I need cloud Drive for that?

Cloud Drive isn't designed for multiple simultaneous clients. You'd need something else, e.g. SyncThing (also check out its GUI wrapper, SyncTrazor).

- Would I also need DrivePool on the new computer (if I'm using Cloud Drive)?

If you intend to access the drives as a pool at the second site you'd want an instance of DrivePool there too.

- Is there a deal on the bundle (DP, Scanner, CD) for existing customers?

As I understand it, if you're an existing customer you get a discount on each product by entering your existing activation ID and the discount is bigger if your existing activation ID covers the product you're buying; just having activation ID(s) for DP and Scanner without one for CloudDrive is enough to match the bundle price offered to new customers.

TLDR: for what you're describing I'd use two installs of DrivePool (one for each machine) plus SyncThing/SyncTrazor to mirror in near-time (I'm presuming there's reasons you're not just opening a network share to the pool over the LAN).

EDIT: make sure to exclude system folders (e.g. System Volume Information) from the sync between the two pools, and make sure to sync at the pool level - not the poolpart level. If you're using SyncThing/SyncTrazor solely across a LAN then I'd suggest disabling relaying and global discovery, and/or using direct addresses, for additional privacy/security/efficiency.

[ethanol]

1 hour ago, Shane said:

TLDR: for what you're describing I'd use two installs of DrivePool (one for each machine) plus SyncThing/SyncTrazor to mirror in near-time (I'm presuming there's reasons you're not just opening a network share to the pool over the LAN)

Thanks for the suggestions.

To be honest, hadn't considered a network share option. What would that look like? I assume both computers would have DP on them. Wouldn't that just change whether I was "pulling" or "pushing" in terms of SyncThing?

[Shane]

I meant the network share option (if you just wanted all devices in both buildings to able to access one pool via the 1GB LAN you mentioned) as an alternative to the SyncThing option (where you'd have two pools sync'd to each other, one in each building). I figured there was a reason you weren't doing that.

But yeah, you could use a network share for syncing too - e.g. with the real-time syncing option of FreeFileSync - in which case you might only need to have the syncing software on just one of the machines. There'd be pros and cons to each option.

[ethanol]

OK. Yes, I have the network share option today and it works fine. The reason I was duplicating the pool was more for redundancy in the event of a disaster (fire, theft, ...) of the primary server/pool. 

[ethanol]

Does it matter whether SyncThing is installed on Source (Server) or Target (remote PC with Orico external attached) or does it need to be installed on both?

[Shane]

SyncThing needs to be installed on both; each instance scans its own content and compares notes with the other(s) to detect changes. This is different to FreeFileSync where it goes on one machine and scans both its own content and the remote content to detect differences. The former is better in slower networks, busier networks or involving large numbers of files (the issues compound each other), as it involves much less back-and-forth of network traffic, but FreeFileSync can compare a surprisingly large number of files on a fast network (e.g. about fifty thousand files per minute when my 1Gbps LAN is idle) and I feel its GUI is rather more user-friendly.

Whichever option you go for, I'd suggest creating a test pool to trial it before committing your real pool - and you could make two test pools and try both.

[ethanol]

Great - thanks for all the help!

 

[mikernet]

@Shane Is it actually safe to use SyncThing in light of the FileID issues DrivePool currently has?

It says it uses FileWatcher to detect changes, which seems like a recipe for disaster here. I am trying to accomplish the same thing as OP, and it will be for syncing fairly large storage pools with a lot of files that get moved / deleted / renamed constantly in real-time.

[Shane]

Hi mikernet. For whatever it is worth, I am successfully using SyncThing with my primary pool (currently only ~ 16TB before duplication) without any fine-tuning, but the content is reasonably static (e.g. photo albums, family videos, financial records, various documents) and I have DrivePool's Read Striping disabled.

Note that SyncThing is not intended for situations where changes are likely to occur faster than the ability to distribute those changes between devices (e.g. if device A renames a file and device B also renames that file before it can receive the update from device A, then SyncThing is going to complain).

Diving in...

DrivePool FileID Notification bug: as I understand it, this risk involves both excessive notifications (reads being classified as writes) and insufficient notifications (e.g. moves not being reported), to which applications that rely on FileID notifications are vulnerable. The original thread for reference.

SyncThing uses two independent mechanisms to detect changes, a scheduled full scan (default hourly) and a watcher that listens for notifications (default enabled); link to documentation. This means SyncThing's scheduled scans will catch any moves that are missed by its watcher, resulting in at most a delay of one hour (or as configured) to update other nodes. I do not know how well its watcher would handle excessive notifications if those occur, but it can be easily disabled if the load becomes a problem.

DrivePool FileID Generation bug: as I understand it, this risk involves DrivePool's use of an incrementing-from-zero counter that resets on reboot, resulting in data corruption/loss for any application (local or remote) that relies on FileID having cross-boot permanence for identifying files (q.v. above original thread link).

As far as I have determined SyncThing should not be affected by this bug so long as its watcher is not being used to monitor pool content via a remote share (e.g. if for some strange esoteric reason you mapped a share "\\pc\pool" to a drive "X:" and then told SyncThing to add "X:\stufftosync" as a folder instead of following the instruction to only add local folders). I'm... actually not sure if the watcher can even do that, but if so that's the only risk.

DrivePool Read Striping bug: as I understand, this risk involves DrivePool sometimes returning invalid hashes or even invalid content to certain applications when this feature is enabled to read a duplicated file via concurrent access to the drives that file is stored on. Some systems apparently do not experience this bug despite using the same application as others. The original thread for reference.

I have NOT tested striping with SyncThing, as I keep DrivePool's Read Striping pre-emptively disabled rather than have to exhaustively test all the apps I use (and given the possibility that the problem could involve some kind of race condition, which are a PITA to avoid false negatives on, I am happy to continue erring on the side of safety over performance).

[mikernet]

@Shane Thank you very much for your reply!

I definitely understand that making changes to a file on both hosts at the same time would be bad. I have designed the system such that one of the hosts is designated the master host which is the only one allowed to make changes to files, which are then expected to asynchronously replicate and become "eventually consistent" on the other secondary host. The secondary host can be used for read-only access to load-balance reads, and it can be promoted as the master host if the other host goes down, at which point it will be the only one making changes to files going forward.

Thanks for that link to the SyncThing docs that detail how the sync works. Based on that, I believe it should work fine for how I will be using it. I figured that it might use cached FileIDs to check if a file was modified during scans but those docs cleared that up as a non-issue as far as I can tell.

I won't be using file duplication in DrivePool so read striping isn't an issue. The underlying "disks" I am passing to DrivePool are RAID 10 arrays so they already have redundancy. I'm just using it to make a bunch of them look like 1 big seamless drive. Each individual disk enclosure is setup as one big RAID 10 array, and when I need to expand storage, I just add a whole new disk enclosure setup as a RAID 10 volume and add it to the DrivePool pool.