10GB connection between two pools - performance tips?

[jmone]

I'm thinking of upgrading my switch and connection between my two Servers (Main and Backup) to 10GB.  This should move the performance bottleneck from my current 1GB link to the speed of my HDD giving a bit of a boost, but it will be nowhere near 10GB/sec on a single copy from one drive to another over the network.  Given I have 8+ drives in each pool is there any thoughts on how to do copy that can do simultaneous reads from multiple drives on Pool 1 --> write to multiple drives on Pool2?

 

Thanks

Nathan

[Christopher (Drashna)]

Well, 10gb networking is very expensive. 

 

And you should see 1.25GB/s as the max theoretical throughput for the network, but this depends on a number of factors (such as cables used, the switch, etc. 

 

As for simultaneous reads and writes, you want something that is multithreaded.  Each thread handles a copy process, so you can increase the throughput, theoretically.

 

Robocopy and RichCopy definitely do this, but I'm not sure what else does. 

[jmone]

Free File Sync is also multi threaded, but it kind of matters in what order the files are being read and written as if there is a thread for each physical drive the increased throughput would be high, but if it is all from the same drive then it will max out at the speed of the one drive.  I've not yet got the equipment (and may not for some time), but the last time I had to duplicate my 40TB over the network it took a week.  10GB "could" cut that down to under a day but only if not bottlenecked.  

 

If/When I get the equipment I'll give a real world update....just thinking out loud for now.

 

Thanks

Nathan

[SantiagoDraco]

I don't think you need (or would benefit from) 10G for DrivePool based drives since unless you are duplicating more than 2x you will only see a possible boost in read performance from DPs performance boosts from split reads.

 

I have a Synology 3615xs NAS with 2 10G ports and I have that directly wired to another 10G adapter in my Windows 2012 Server Essentials box.   The Synology is running 10 drives in Raid 6 with 2 SSD cache drives and I host all of my high data rate files (blu-ray etc) there.   All of my lower overhead data resides on a DrivePool.

 

So long story short unless you are running SSDs under DrivePool I don't see you saturating 1G with spindle drives on a DrivePool "pool".   Certainly the cost for a switch is overly excessive and you will not see the benefit of that expense.   The cheapest switch will run you about 750.  Better to run direct with adapters and run another 1G back out to the LAN if you are focused on file transfers between servers.

[Christopher (Drashna)]

 

Free File Sync is also multi threaded, but it kind of matters in what order the files are being read and written as if there is a thread for each physical drive the increased throughput would be high, but if it is all from the same drive then it will max out at the speed of the one drive.  I'

 

The placement of the drives, and the duplication level would really matter here. 

 

The Read Striping feature would absolutely help with reads here. While it can (and tries) to read from multiple disks at the same time and cache that into memory... it can also read from a single disk, if the other disk is overloaded.

 

http://stablebit.com/Support/DrivePool/2.X/Manual?Section=Performance%20Options

[jmone]

So got it all installed and working.  With large files copies, I used to get a max of 110MB/s between the pools over the network (eg maxing out a single Gigabit Link).  I now get a max of 150MB/s, some 36% faster (eg maxing out the single HDD at each end).  But as I now how heaps of 10Gb goodness, I also can now kick off a 2nd (or 3rd, 4th etc) copy between other HDD and get another max of 150MB/s per disk.

 

I've had to twice copy all the contents from one pool to another (40TB mix of large, med and small files).  It took almost a week over Gigabit.  Just the extra 36% would cut days of that.... but in addition I could kick off a copy (outside DrivePool) for each drive in my pool and such a task would then be well under a day.

 

It would be nice to have "FreeFileSync" style compare/sync between pools in DrivePool as it could then support a thread per physical drive instead of one per pool.

 

Thanks

Nathan

 

Note: I started to look at 10Gb when Intel / MS no longer supported teaming / VLAN on their Dual NIC cards under Windows 10.   Look like they have now fixed this (MS released patches, Intel has verified, and plan to issue an updated driver in June).

[Christopher (Drashna)]

Well, glad to hear that it seems to be working very well! 

 

And yeah, a "pool aware" utility would be very nice for something like this.  Something that would more intelligently queue files from the pool based on their physical location. 

[jmone]

Yup,  such a "pool aware" utility would be perfect!

[Christopher (Drashna)]

Yup,  such a "pool aware" utility would be perfect!

Well, that would be dependant on the app itself, and IIRC, not too difficult to do.

 

And I believe that there is already an API built into the driver for this (which is what the DPCMD utility uses), so we'd really just need to document this (and Alex has mentioned possibly doing this).  So, it's definitely not outside of the realm of possibility. But I think it would require  running "locally" to accomplish this. 

[jmone]

Nothing is every straightforward.  Turns out these adapters don't support WOL (so can't wake it up using the Lights Out wigit for example).  Tired to see if I could use the Mobo Ethernet adapter just for WOL.... but of course WinServer doesn't like Consumer NIC much (edit - tried a few drivers and it has now disappeared from Device Manager so may have stuffed this port altogether.)... Thinking thinking...

[Christopher (Drashna)]

Nope, it never is.

 

Though, I can't say that I'm surprised that they don't support WOL, as that is more of a consumer technology (enterprises just don't turn off servers....) 

 

As for the onboard NIC, IIRC, there is a hack to force it to work, but it requires disabling driver signing enforcement. If you're okay with that, it may be worth trying. 

It's basically this: 

https://foxdeploy.com/2013/09/12/hacking-an-intel-network-card-to-work-on-server-2012-r2/

[jmone]

thanks - ... another thing with the setup, these cards are 4x PCIe, and my slot on one PC was auto configured to 1x.  I was "only" getting 2.5Gb/s transfer between the two PCs.  Changed it to 4x and throughput jumped to 9+Gb/s - Nice!

[Christopher (Drashna)]

Yikes, that's not good!  And yeah, that definitely makes a HUGE difference!  And glad to see that it really jumped up the speeds!

[BinaryData]

I work in a Data Center, 10Gbit Network is standard here, in the home? Yeah, nope. Wouldn't waste your money, it'd cost you $800 per device, cards are $300 - $1,000 a pop. Just isn't worth it.

 

I would, if I were you and moving large amounts of data, build an 8 Bay SSD intermediary device. Since Platter drives are by far slower than SSD, everything would get moved from Outgoing_Server -> SSD Array -> Storage_Array. It'd help quite a bit, I plan on doing it with my PLEX Media Server. Looking at 6x 512GB or 1TB SSD Samsung Drives. Should be plenty fast enough and keep the bottleneck down. I also suffer from slow network speeds, but that's just because my desktop is 7 Years old

[jmone]

It's not for all but I certainly did not pay anywhere neat $800 per device!  

 

There are a heap of 2nd hand 10Gb PCI-e adapters staring around US$100 / Transceivers for $20.  New pre-terminated cables are also reasonably priced.  A few desktop boards are now also shipping with 10Gbe.  10Gb switches are another matter (they are still $$$$) but my existing switch already had two SFP+ ports so I thought... why not.

[Christopher (Drashna)]

Well, what if I want a 10Gbps connection between my file server and emby/plex server?

 

Also, ebay/CL/friends that work in a datacenter?

[alternativesurfer]

I just installed these in my file server / plex server:

http://www.amazon.com/Mellanox-Connectx-2-PCI-Epress-Interface-MNPA19-XTR/dp/B017O9VPYY?ie=UTF8&psc=1&redirect=true&ref_=oh_aui_search_detailpage

and connected them with this cable: http://www.amazon.com/10Gtek%C2%AE-1-Meter-10GBASE-CU-Passive-SFP-H10GB-CU1M/dp/B00WHS3NCA?ie=UTF8&psc=1&redirect=true&ref_=oh_aui_search_detailpage

 

10Gbps goodness, its beautiful.

[jmone]

Nice!  ... and cheap!

 

10G is great  

[Umfriend]

So what are the chances I'll be able to run 10G over my Cat5e cables with max length of, say, 25 meters?

[jmone]

I looked at 10GBASE-T  as I too had existing (mix) of Cat5e/6 runs but after reading decided it was just as cheap and more reliable to go fibre.  The spec for 10GBASE-T Lengths is:

- Cat 6a : 100m

- Cat 6 : 55m

- Cat 5e : Not officially supported but some claim you will be fine for small runs but to check the error rates.

[Umfriend]

Yes, well, I don't want to replace the cabling, quite a bit of work thaty would be.

 

Are errors checked/corrected? I mean, if I get to 5GB instread of 10 due to error correction that'd be fine as well.

[jmone]

Only one way to fine out!

[Christopher (Drashna)]

I looked at 10GBASE-T  as I too had existing (mix) of Cat5e/6 runs but after reading decided it was just as cheap and more reliable to go fibre.  The spec for 10GBASE-T Lengths is:

- Cat 6a : 100m

- Cat 6 : 55m

- Cat 5e : Not officially supported but some claim you will be fine for small runs but to check the error rates.

This is why I bought a spool of Cat6, the last time I needed cable.  Might as well go for the better stuff... and 55m is more than enough for my house. Especially as the network closet is dead center of the house, basically. 

 

Just need to get 10gig hardware.

[jmone]

Got your 10GBe yet?  All working well for me  

[Christopher (Drashna)]

Nope, it's on the backburner for now.

Tight on funds, and other issues (like getting my server stable...)