Jump to content
Covecube Inc.
  • 0
Rob Manderson

Duplication time is extremely long!

Question

My motherboard recently died and it became time to move from DP on WHS2011 to DP on Server 2016 Essentials.  So I did the install, added appropriate drivers for my eSata cards etc, installed DP and up it comes.  Everything working fine.  The entire array is set to duplication.

At which point I decided to add a 4TB drive to DP and withdraw, one at a time, a pair of 2TB drives so I could assign them as a separate pool for use as a backup target folder.  So I added the 4TB drive, no problem.  And then I think I shot myself in the foot!  I've replaced bad drives a number of times over the years, and I've always done the immediate removal followed by automatic duplication and it's always worked.  So, even though the drive wasn't bad, I did the same.

It now reports (for a total pool size of 25.5 TB) the following when measure runs.

Free space 10.1TB

Unduplicated 1.31 TB

Duplicated 14.1 TB

Other 6.53 GB

The numbers look about right.

However, after the measure, it goes into checking and at about the 25% point it starts trying to duplicate.  And then it sits there for what feels like forever.  If I hover over the bouncing bar at the bottom of the GUI I can see it's working on a particular folder in the Video area.  When I switch to that folder there's a file named (random guid).copytemp where random guid really is just a random guid.  I can see, by comparing reported file sizes, that it's a copy of a particular file in the same folder.  The file is about 250 MB in size.  After about 15 minutes it finishes with that folder and moves on to another, and I see a new file in that folder with random guid etc.  Go back to the first folder and sure enough the copytemp file is gone.

The problem isn't any particular folder, it seems to be that as it's catching up with duplication it really is duplicating, just frustratingly slow.  If it's taking about 15 minutes to duplicate a 250 MB file and I have 1.31 TB to duplicate I'm wondering if it'll finish in time for me to see it (I'm in my mid 60's! :-)  ).

Specific data.  I'm using 2.2.3.939 beta on Windows Server 2016 Essentials x64 .  I had initially installed 2.2.2.934 with the same behaviour.  My storage is 3 Mediasonic Probox enclosures with 9 sata drives of varying sizes (a mix of 2TB, 3TB and 4TB drives).  I'm using Mediasonic ProBox HP1-SS3 2 Port External SATA III PCI Express Cards (asm1061x drivers).  These are the same enclosures and cards I've been using successfully with WH2011 for about 5 years.  I doubt any of the drives is that old though (I seem to replace one or two per year but they're cheap and the great thing about DP is you don't much care given how painless a replacement is).  I have realtime duplication turned off.

The asm 1061x drivers are actually the same drivers I used on WHS2011, i.e., for server 2008 but, apart from duplication, my system is having no problems.  No instability, no BSODs,  throughput in the 50 MB/s range.  Serviio (a DLNA media server running on the same machine) finds and serves up videos just fine.  I doubt it's a hardware issue.  I had been using the WHS2011 version of DP (somewhere in the 1.3.5.xxx range) on the old machine and that's what I'm comparing current behaviour against.

Regarding that throughput figure above - that was measured copying a 2GB file from the pool to a non pooled drive across the network while the duplication is in progress. I don't think throughput is the issue here!  I should add that when I'm using files on DP (i.e., something other than DP is working on the pool) I see disk activity on the drive enclosures.  When DP is the only thing touching the pool I see no activity for minutes on end, followed by a brief flurry of activity and then a long lull again.  Pressing the 'increase priority' chevron in the DP GUI doesn't seem to change that at all.

Any ideas on what the problem might be?

 

Edit:  I left it running ov ernight - I now have 1.28TB unduplicated and 14.2 TB duplicated.  So it seems it *is* duplicating - just taking a very long time over it.

Share this post


Link to post
Share on other sites

11 answers to this question

Recommended Posts

  • 0

Yes, the duplication is slow.  This is because it runs in a background IO priority, meaning that any other activity on the disks basically pauses duplication.  And it's essentially copying the files, when it's duplicating the data. 

There should be a ">>" icon when it's duplicating, that can increase the priority, and make it quicker, but may slow down access to the pool. 

Also, you can permanently set this:
http://wiki.covecube.com/StableBit_DrivePool_2.x_Advanced_Settings#Settings.json

Set the override value for "FileDuplication_BackgroundIO" to "false" and restart the service. 

Share this post


Link to post
Share on other sites
  • 0
On 10/10/2018 at 12:36 PM, Christopher (Drashna) said:

Set the override value for "FileDuplication_BackgroundIO" to "true" and restart the service. 

 

Don't you mean 'false,' to make it use foreground IO? I just saw this advice in another thread, as well.

Share this post


Link to post
Share on other sites
  • 0

Follow up.  I think I found the reason why duplication was soooo slow.  My legendary cheapness caught me out!

My array had consisted of a mixture of 2, 3 & 4TB drives.  The new 4tb drive mentioned in the first post above was a Seagate ST4000DM004 drive, which is an SMR drive.  Apparently these drives are intended for the archive market and have very slow write speeds if you're pumping a lot of data into them in a short time.  This is one reason they're cheap (and why I chose them). Such a pity the Amazon product page makes no mention of this.  Indeed I only found out by searching on the model number and reading a lot of posts by other users.

I replaced it with a Toshiba 4TB model and duplication (and write) speeds returned to normal.  (This is a couple of months ago now and I apologise for taking so long to update this post).

Given the slow write speeds I'm at a loss to think of a usable scenario for this product.  Surely commercial users don't take weeks or months to fill a 4TB drive?  My measurements show that from clean (new volume/partition) they manage about 20GB of writing before significant slowdown occurs.  After that write speeds can drop to less than a MB/s for large files (> 250MB).  For small files (< 1MB) throughput can drop to under 100KB/s.  Let the drive sit for a few minutes still powered up while it does it's thing and write speeds return until the next time you overflow their cache.

I have four of these things!

Share this post


Link to post
Share on other sites
  • 0
8 hours ago, Rob Manderson said:

Given the slow write speeds I'm at a loss to think of a usable scenario for this product

For the Archive drives, right?  If so, that's because you're doing a lot of modification. These drives are meant for "write once, read many" type of storage. 

The ideal usage with StableBit DrivePool would be to use the SSD Optimizer balancer, actually.  It's what I do, and it works great for all 16 of these drives. 

Share this post


Link to post
Share on other sites
  • 0

Ah, ok.  So you recommend that I add (say two) of these drives to the existing array plus an SSD.  My normal writes go first to the SSD and in the background it replicates to the SMR drives.  Since those are empty they'll receive the writes until they match the other disks and after that writes will start consuming space on the existing drives?  I say two drives because I have duplication turned on (one of the best features of DP).  Since I have duplication on do I need to have two SSD's?

Share this post


Link to post
Share on other sites
  • 0

Yes, to all of that.  

Though, keep in mind that modifications of existing files go straight to the existing disk they're on, so the SMR drive. So, if you have a database, it may cause issues. 

And yes, you definitely need 2x SSDs (or "fast" drives) if you have duplication enabled. 

Share this post


Link to post
Share on other sites
  • 0

Whatever you all do, don't wait 3 years and 8,000 movies (taking up 50TB), later to decide duplication would be a good idea. When I noticed my pool was getting full, it finally dawned on me I'd have a miserable time replacing lost movies if even one of the 15 WD40EFRX 4TB drives went south. Not only did it blast a hole in my wallet this week, to fill the remainder of my RPC-4224 case with 8x new WD80EFAX 8TB and 1x new WD100EFAX 10TB drive (experimental), it appears it will take a month of Sundays to get the job done. I probably doesn't help than I'm doing this on an old WHS2011 machine with 3x AOC-SASLP2-MV8 controllers, one of which is running in a 4x slot. I just hope I don't kill something in the process. I honestly didn't think the 10TB drive would work. I had it initialize, partition and format it on a newer PC for some reason. So I'm still not 100% sure how reliable its going to be.

After 4 hours, it actually looks like its copying about 500GB per hour. So maybe it won't a full month of Sundays... :unsure:

Share this post


Link to post
Share on other sites
  • 0

Heh - I have triplication turned on - for the same reason you went duplicated.  Started out with ~8TB duplicated and went to triplication - it took 3 days and that was without a single one of those damn SMR drives installed.

I've given up on the SMR drives - nothing will make them behave.  Yep, other peoples experience may differ - I can only go by what I see and what I see is that anywhere between 20 and 100 GB of continuous writes is all they'll take before they slow to ~500KB/s write speeds.  Leave them powered up for 24 hours with no further writes and they're *still* sitting at ~500KB/s.  48 hours?  Still ~500KB/s.  Patience ran out at that point.

I won't be buying another SMR drive any time soon and the 4 I have are sitting there unused on a shelf.  I think I'll end up throwing them out.  The thing is that the SSD optimizer didn't seem to help.  After all, if I add them to a pool the expectation is that I can do some balancing to spread the data around.  What if one of my non SMR drives fails and needs to be removed?  The SMR drives will be the targets and the same issue arises - it will take weeks (months?) to remove the failing drive because copying to the SMR drives takes so long.  For my use case the SMR drives are totally unsuitable.

I ended up spending the extra bucks (which wasn't all that much) and installing Toshiba X300 4TB drives.

Share this post


Link to post
Share on other sites
  • 0

Yes, once you get past the 20GB and a bit, you'd need to leave them idle for a bit to get decent writes again. If you keep throwing data at them, once the non-SMR cache is full, it'll be slow without recovery. SSD with optimizer may help but then the write must still be less than your SSD + 20GB and a bit. I have had great sustained writes on the SMR drives but that was in cases of real sequential nature.

You got 4 SMR HDDs? I might be interested. 4 would allow fast writes of 80GB in one go already! ;)

Share this post


Link to post
Share on other sites
  • 0

I have to admit I was pleasantly surprised with the duplication process. There was 48.6 TB to be duplicated and it took 123 hours (5 days, 3 hrs) to accomplish. I did tweak the settings file to "FileDuplication_BackgroundIO=False", which moved things along. I have several people streaming Plex videos from across the country and nobody noticed problems. Now I just need to add a couple more WD100EFAX drives so I have some free pool space and I can just leave the server alone for a few months.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×