Jump to content

  • Log in with Twitter Log in with Windows Live Log In with Google      Sign In   
  • Create Account

Photo

10GB connection between two pools - performance tips?


  • Please log in to reply
24 replies to this topic

#1 jmone

jmone

    Advanced Member

  • Members
  • PipPipPip
  • 113 posts

Posted 10 April 2016 - 12:08 PM

I'm thinking of upgrading my switch and connection between my two Servers (Main and Backup) to 10GB.  This should move the performance bottleneck from my current 1GB link to the speed of my HDD giving a bit of a boost, but it will be nowhere near 10GB/sec on a single copy from one drive to another over the network.  Given I have 8+ drives in each pool is there any thoughts on how to do copy that can do simultaneous reads from multiple drives on Pool 1 --> write to multiple drives on Pool2?

 

Thanks

Nathan



#2 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,208 posts
  • LocationSan Diego, CA, USA

Posted 10 April 2016 - 09:34 PM

Well, 10gb networking is very expensive. 

 

And you should see 1.25GB/s as the max theoretical throughput for the network, but this depends on a number of factors (such as cables used, the switch, etc. 

 

As for simultaneous reads and writes, you want something that is multithreaded.  Each thread handles a copy process, so you can increase the throughput, theoretically.

 

Robocopy and RichCopy definitely do this, but I'm not sure what else does. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#3 jmone

jmone

    Advanced Member

  • Members
  • PipPipPip
  • 113 posts

Posted 10 April 2016 - 10:05 PM

Free File Sync is also multi threaded, but it kind of matters in what order the files are being read and written as if there is a thread for each physical drive the increased throughput would be high, but if it is all from the same drive then it will max out at the speed of the one drive.  I've not yet got the equipment (and may not for some time), but the last time I had to duplicate my 40TB over the network it took a week.  10GB "could" cut that down to under a day but only if not bottlenecked.  

 

If/When I get the equipment I'll give a real world update....just thinking out loud for now.

 

Thanks

Nathan



#4 SantiagoDraco

SantiagoDraco

    Member

  • Members
  • PipPip
  • 27 posts

Posted 11 April 2016 - 07:41 PM

I don't think you need (or would benefit from) 10G for DrivePool based drives since unless you are duplicating more than 2x you will only see a possible boost in read performance from DPs performance boosts from split reads.

 

I have a Synology 3615xs NAS with 2 10G ports and I have that directly wired to another 10G adapter in my Windows 2012 Server Essentials box.   The Synology is running 10 drives in Raid 6 with 2 SSD cache drives and I host all of my high data rate files (blu-ray etc) there.   All of my lower overhead data resides on a DrivePool.

 

So long story short unless you are running SSDs under DrivePool I don't see you saturating 1G with spindle drives on a DrivePool "pool".   Certainly the cost for a switch is overly excessive and you will not see the benefit of that expense.   The cheapest switch will run you about 750.  Better to run direct with adapters and run another 1G back out to the LAN if you are focused on file transfers between servers.



#5 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,208 posts
  • LocationSan Diego, CA, USA

Posted 13 April 2016 - 01:14 AM

 

Free File Sync is also multi threaded, but it kind of matters in what order the files are being read and written as if there is a thread for each physical drive the increased throughput would be high, but if it is all from the same drive then it will max out at the speed of the one drive.  I'

 

The placement of the drives, and the duplication level would really matter here. 

 

The Read Striping feature would absolutely help with reads here. While it can (and tries) to read from multiple disks at the same time and cache that into memory... it can also read from a single disk, if the other disk is overloaded.

 

http://stablebit.com...ormance Options


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#6 jmone

jmone

    Advanced Member

  • Members
  • PipPipPip
  • 113 posts

Posted 16 April 2016 - 03:28 AM

So got it all installed and working.  With large files copies, I used to get a max of 110MB/s between the pools over the network (eg maxing out a single Gigabit Link).  I now get a max of 150MB/s, some 36% faster (eg maxing out the single HDD at each end).  But as I now how heaps of 10Gb goodness, I also can now kick off a 2nd (or 3rd, 4th etc) copy between other HDD and get another max of 150MB/s per disk.

 

I've had to twice copy all the contents from one pool to another (40TB mix of large, med and small files).  It took almost a week over Gigabit.  Just the extra 36% would cut days of that.... but in addition I could kick off a copy (outside DrivePool) for each drive in my pool and such a task would then be well under a day.

 

It would be nice to have "FreeFileSync" style compare/sync between pools in DrivePool as it could then support a thread per physical drive instead of one per pool.

 

Thanks

Nathan

 

Note: I started to look at 10Gb when Intel / MS no longer supported teaming / VLAN on their Dual NIC cards under Windows 10.   Look like they have now fixed this (MS released patches, Intel has verified, and plan to issue an updated driver in June).


  • Christopher (Drashna) likes this

#7 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,208 posts
  • LocationSan Diego, CA, USA

Posted 17 April 2016 - 11:35 PM

Well, glad to hear that it seems to be working very well! 

 

And yeah, a "pool aware" utility would be very nice for something like this.  Something that would more intelligently queue files from the pool based on their physical location. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#8 jmone

jmone

    Advanced Member

  • Members
  • PipPipPip
  • 113 posts

Posted 18 April 2016 - 06:37 AM

Yup,  such a "pool aware" utility would be perfect!


  • Christopher (Drashna) likes this

#9 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,208 posts
  • LocationSan Diego, CA, USA

Posted 20 April 2016 - 09:36 PM

Yup,  such a "pool aware" utility would be perfect!

Well, that would be dependant on the app itself, and IIRC, not too difficult to do.

 

And I believe that there is already an API built into the driver for this (which is what the DPCMD utility uses), so we'd really just need to document this (and Alex has mentioned possibly doing this).  So, it's definitely not outside of the realm of possibility. But I think it would require  running "locally" to accomplish this. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#10 jmone

jmone

    Advanced Member

  • Members
  • PipPipPip
  • 113 posts

Posted 29 April 2016 - 08:52 AM

Nothing is every straightforward.  Turns out these adapters don't support WOL (so can't wake it up using the Lights Out wigit for example).  Tired to see if I could use the Mobo Ethernet adapter just for WOL.... but of course WinServer doesn't like Consumer NIC much (edit - tried a few drivers and it has now disappeared from Device Manager so may have stuffed this port altogether.)... Thinking thinking...



#11 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,208 posts
  • LocationSan Diego, CA, USA

Posted 30 April 2016 - 08:48 PM

Nope, it never is. :(

 

Though, I can't say that I'm surprised that they don't support WOL, as that is more of a consumer technology (enterprises just don't turn off servers....) 

 

As for the onboard NIC, IIRC, there is a hack to force it to work, but it requires disabling driver signing enforcement. If you're okay with that, it may be worth trying. 

It's basically this: 

https://foxdeploy.co...server-2012-r2/


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#12 jmone

jmone

    Advanced Member

  • Members
  • PipPipPip
  • 113 posts

Posted 01 May 2016 - 10:46 PM

thanks - ... another thing with the setup, these cards are 4x PCIe, and my slot on one PC was auto configured to 1x.  I was "only" getting 2.5Gb/s transfer between the two PCs.  Changed it to 4x and throughput jumped to 9+Gb/s - Nice!



#13 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,208 posts
  • LocationSan Diego, CA, USA

Posted 03 May 2016 - 07:34 PM

Yikes, that's not good!  And yeah, that definitely makes a HUGE difference!  And glad to see that it really jumped up the speeds! :)


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#14 BinaryData

BinaryData

    Member

  • Members
  • PipPip
  • 18 posts

Posted 07 May 2016 - 03:34 PM

I work in a Data Center, 10Gbit Network is standard here, in the home? Yeah, nope. Wouldn't waste your money, it'd cost you $800 per device, cards are $300 - $1,000 a pop. Just isn't worth it.

 

I would, if I were you and moving large amounts of data, build an 8 Bay SSD intermediary device. Since Platter drives are by far slower than SSD, everything would get moved from Outgoing_Server -> SSD Array -> Storage_Array. It'd help quite a bit, I plan on doing it with my PLEX Media Server. Looking at 6x 512GB or 1TB SSD Samsung Drives. Should be plenty fast enough and keep the bottleneck down. I also suffer from slow network speeds, but that's just because my desktop is 7 Years old :P



#15 jmone

jmone

    Advanced Member

  • Members
  • PipPipPip
  • 113 posts

Posted 07 May 2016 - 10:05 PM

It's not for all but I certainly did not pay anywhere neat $800 per device!  

 

There are a heap of 2nd hand 10Gb PCI-e adapters staring around US$100 / Transceivers for $20.  New pre-terminated cables are also reasonably priced.  A few desktop boards are now also shipping with 10Gbe.  10Gb switches are another matter (they are still $$$$) but my existing switch already had two SFP+ ports so I thought... why not.



#16 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,208 posts
  • LocationSan Diego, CA, USA

Posted 07 May 2016 - 10:31 PM

Well, what if I want a 10Gbps connection between my file server and emby/plex server? :)

 

Also, ebay/CL/friends that work in a datacenter? :)


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#17 alternativesurfer

alternativesurfer

    Newbie

  • Members
  • Pip
  • 7 posts

Posted 07 June 2016 - 09:43 PM

I just installed these in my file server / plex server:

http://www.amazon.co...arch_detailpage

and connected them with this cable: http://www.amazon.co...arch_detailpage

 

10Gbps goodness, its beautiful.



#18 jmone

jmone

    Advanced Member

  • Members
  • PipPipPip
  • 113 posts

Posted 09 June 2016 - 11:54 AM

Nice!  ... and cheap!

 

10G is great :) 



#19 Umfriend

Umfriend

    Advanced Member

  • Members
  • PipPipPip
  • 373 posts

Posted 11 June 2016 - 03:04 PM

So what are the chances I'll be able to run 10G over my Cat5e cables with max length of, say, 25 meters?



#20 jmone

jmone

    Advanced Member

  • Members
  • PipPipPip
  • 113 posts

Posted 12 June 2016 - 12:47 AM

I looked at 10GBASE-T  as I too had existing (mix) of Cat5e/6 runs but after reading decided it was just as cheap and more reliable to go fibre.  The spec for 10GBASE-T Lengths is:

- Cat 6a : 100m

- Cat 6 : 55m

- Cat 5e : Not officially supported but some claim you will be fine for small runs but to check the error rates.






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users