Major performance issues copying files from server to server

[Sonicmojo]

Continuing my trial testing. Using standard Windows Explorer file copy - this morning I sent a 1GB file to the pool (on new server) from my workstation and was getting an average speed of 115MB/s - which is about as fast as I have ever seen files copy here.

 

Now I am testing a "share to share" copy of 15GB of photos from the old server to the new - using Robocopy - and the pool performance is simply crawling along at approx 5.75 MB/s. It's taken about 35 minutes to copy just 12GB.

 

What is going on here? Clearly I am missing something...

 

Ideas?

 

Sonic.

[Christopher (Drashna)]

Unfortunately, that behavior is pretty much expected. 

 

Not just with StableBit DrivePool, but for any drive.  The large file is creating a single entry in the NTFS partition, and copying the data over, so rather simple.  However, the images.... I'm guessing that most of these are less than a MB each? If so, you need to keep in mind that an NTFS entry is created for each of these files (at the beginning of the disk) and then the data is copied to the disk, and repeated hundreds of times.   

 

Essentially, the disk is writing to the beginning of the drive, and then to different points, and going back and forth. This adds a significant penalty to the write speed and is why you're seeing the drop in performance. 

 

 

If you're using Robocopy, what may help is to use the "/mt:" switch to enable multithreaded copies, as this may boost the speed of the copy, by using multiple read and write threads.  

 

 

 

Another way to fix/circumvent this issue is by using the SSD Optimizer, and using an SSD (or two). This way, new files are written to the SSD and then moved out, and shouldn't see this sort of performance hit (at least not as drastically)

[Sonicmojo]

Chris,

 

I have to disagree as I ran the test in a different way and saw much better results. For yesterdays test - I ran the Robocopy script FROM the server with Drivepool on it - and watched the times suffer greatly.

 

However moments ago - I did another test - this time sending 10GB of images from my workstation TO the Drivepool server - and the entire lot copied over in just under 2 mins. I was seeing DP write speeds back up in the 75-90MB/s range.

 

Also - the MT switch wreaks havoc with the original modified date on folders being copied so I cannot use that.

 

It is bizarre that the script running in the reverse direction would log such a brutal write throughput - but I know better now.

 

Cheers!

 

Sonic.

[Christopher (Drashna)]

To clarify, this is an issue when copying from the pool, but not to it? 

 

If so, that's ... a bit odd. 

 

Are you using the Network IO Boost feature? 
Also, could you try toggling the "read striping" feature?

Both of these are in the UI, under "Pool Options" -> "Performance". 

 

Also, could you grab logs?

http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

 

Additionally, do you have any antivirus, backup or disks tools installed on either system? 

And could you post the results of: http://wiki.covecube.com/StableBit_DrivePool_Q2159701

[Sonicmojo]

To clarify, this is an issue when copying from the pool, but not to it? 

 

If so, that's ... a bit odd. 

 

Are you using the Network IO Boost feature? 

Also, could you try toggling the "read striping" feature?

Both of these are in the UI, under "Pool Options" -> "Performance". 

 

Also, could you grab logs?

http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

 

Additionally, do you have any antivirus, backup or disks tools installed on either system? 

And could you post the results of: http://wiki.covecube.com/StableBit_DrivePool_Q2159701

 

1. To clarify - I was seeing the brutal speed - when running Robocopy ON the server and copying the files to the Pool

 

Update: Ran the script again and this time - no issues. Speed was right up fluxing between 60-90 MB/s with bursts up over 115 MB/s

 

2. Read striping is on - network boost was not. Will turn it on and test again.

 

Update: Dis not turn this on since the copy was already flying.

 

3. Will turn on logging and run the same Robocopy script from the new server - again copying (pull) TO the pool.

 

Update: Activated logging but again - this copy was extremely fast now?

 

4. There are no additional tools installed on this box - just the OS (Windows Server 2012R2) and Drivepool/Scanner

 

5. Here you go....

 

Microsoft Windows [Version 6.3.9600]

© 2013 Microsoft Corporation. All rights reserved.

 

C:\Users\Administrator>fltmc

 

Filter Name                     Num Instances    Altitude    Frame

------------------------------  -------------  ------------  -----

luafv                                   1       135000         0

npsvctrig                            1        46000         0

 

 

For now - let's chalk this up as an anomaly as I cannot see any real issues at the moment.

 

I will be re-building the server tomorrow with it's real hardware and final config. If I see any issues there when testing begins - we can revisit this.

 

Cheers!

 

Sonic.

[Christopher (Drashna)]

That's ... well, really odd.  And naturally ... things tend to work properly when you're monitoring them and hoping for them to go wrong!  (aka, the "tech support effect). 

 

 

There are a number of reasons that this could have happened, but it's hard to know which, after the fact.

[Sonicmojo]

That's ... well, really odd.  And naturally ... things tend to work properly when you're monitoring them and hoping for them to go wrong!  (aka, the "tech support effect). 

 

 

There are a number of reasons that this could have happened, but it's hard to know which, after the fact.

 

The server now has 2 of it's 4 final drives and I have been moving data for the last day or so.

 

I will be installing the other two drives over the next few days and will devote a bit of time this week to monitoring copy speed etc and come back here when I have something concrete to report.

 

Cheers!

 

Sonic.

[Christopher (Drashna)]

Well, glad to hear it.

 

If you want, StableBit Scanner has a "burst test" feature that is useful for stress testing the connection to the disk (it reads the same data from the drive, so it reads it from the disk's cache).  

 

This is a good idea of you suspect that one or more of the disks are having connectivity issues.  (such as a bad cable, a controller issue, or a drive whose PCB board is shorting).