Speed Up Duplication?

[TeeeBear]

Is there a way to speed up duplication? Previously I had everything duplicated on a WHS v1 box, but have now wiped those drives and added them to my main WHS 2011 box and enabled duplication within DrivePool.  A week later it's only at 600gb duplicated. At this rate, it looks like it'll take months and I'm without backups in the mean time. I did try turning off balancing hoping duplication would be quicker, then it could balance at a later date, but that doesn't appear to have helped. Any suggestions?

 

thanks!

[Umfriend]

I would second that.

 

In my experience, any migration/reinstall etc is way faster when one holds the files outside the Pool (one way or the other), creates a Pool with duplication and then moves/copies the data to the Pool. Sure, I have little data (600GB or so) so it is feasible but for those who have like 10TB of data or so it may virtually impossible.

 

It almost appears as if balancing/checking/duplicating runs sooo deep in the background that it hardly takes any resources at all. It would be helpfull if, one way or the other, a user could temporarily allow DP high priority or something. That is assuming my impressions are anywhere near correct of course.

 

@TeeeBear: how are you without backups? WHS 2011 Server Backup does not backup Pools, only the underlying harddisks. So _if_ you are running backups on those HDDs _then_ you should have backups (and those increasingly would backup up duplicate files).

[daveyboy37]

Is there a way to speed up duplication? Previously I had everything duplicated on a WHS v1 box, but have now wiped those drives and added them to my main WHS 2011 box and enabled duplication within DrivePool.  A week later it's only at 600gb duplicated. At this rate, it looks like it'll take months and I'm without backups in the mean time. I did try turning off balancing hoping duplication would be quicker, then it could balance at a later date, but that doesn't appear to have helped. Any suggestions?

 

thanks!

 

I'm pretty surprised by this.. Earlier today I duplicated 400GB of Flac files in an afternoon. Last night I duplicated all my TV series. They amount to 2TB of data. I set the duplication going at 8PM. I had finished when i got up the following morning.. So I cant say how long it actually took...But less than 12 hours. 

 

Would you happen to know what the write speeds are on all your drives? 

[Umfriend]

I would look at (and perhaps post some screenprints) of resmon.exe

 

I should have writespeeds way over 100Mb/s but don't get anywhere near that when duplicating. Sure, lots of little files will cause lower speeds perhaps but active times / disk queue length reported indicated not a lot was happening. How's that with you?

[pjakesson]

It almost appears as if balancing/checking/duplicating runs sooo deep in the background that it hardly takes any resources at all. It would be helpfull if, one way or the other, a user could temporarily allow DP high priority or something. That is assuming my impressions are anywhere near correct of course.

 

I replaced a 2 TB disk and my pool had to be re-duplicated, and it took ages.

 

But asking/looking in this forum a found a DP config file setting that would let the

duplication process run with higher priority, Sorry I don't remember the settings name,

but search the forum.

 

Then the duplication ended within hours.

 

So I think default priority is too low.

 

regards

 

Per

[Shane]

The DP config particulars can be found here: http://wiki.covecube.com/StableBit_DrivePool_Advanced_Settings

 

Try setting FileDuplication_BackgroundIO to False but be aware this may cause some lag in other disk operations (so even if it works, you may want to change it back after it's done its job).

[Christopher (Drashna)]

Yup, Shane is absolutely correct here. 

[TeeeBear]

@TeeeBear: how are you without backups? WHS 2011 Server Backup does not backup Pools, only the underlying harddisks. So _if_ you are running backups on those HDDs _then_ you should have backups (and those increasingly would backup up duplicate files).

Previously, using robocopy, i was duplicating all the data from WHS 2011 to WHS v1, as stated above. WHS 2011 has server backups turned on but since i wiped all the drives from WHS V1, none of the server folders are backed up. I know duplication isn't a backup, but since this is mostly movies, i'm ok with duplication. I do have another back-up of my photos and important documents. Since, I have had 3 seagate drives fail in 6 months, I don't want to rebuild my collection!

 

I would look at (and perhaps post some screenprints) of resmon.exe

 

I should have writespeeds way over 100Mb/s but don't get anywhere near that when duplicating. Sure, lots of little files will cause lower speeds perhaps but active times / disk queue length reported indicated not a lot was happening. How's that with you?

I was getting around 30Mb/s

 

I replaced a 2 TB disk and my pool had to be re-duplicated, and it took ages.

 

But asking/looking in this forum a found a DP config file setting that would let the

duplication process run with higher priority, Sorry I don't remember the settings name,

but search the forum.

 

Then the duplication ended within hours.

 

So I think default priority is too low.

 

regards

 

Per

 

 

Yup, Shane is absolutely correct here. 

 

 

The DP config particulars can be found here: http://wiki.covecube.com/StableBit_DrivePool_Advanced_Settings

 

Try setting FileDuplication_BackgroundIO to False but be aware this may cause some lag in other disk operations (so even if it works, you may want to change it back after it's done its job).

 

thanks!! I am now seeing disk I/O up to 300Mb/s.

[TeeeBear]

Two new questions…

1) On the DrivePool Overview tab, under disks, it only shows two of my drives dark blue/light blue for pooled and duplicated. The rest of the drives are gray. Previously all of the drives showed as pooled data. What is happening? It seems to be using all of the drives, as percent full is increasing, so I'm not too alarmed, but it is weird.

2) One of the newly added pool disks still shows 0.0% full, when the others are filling up at an even rate, currently 21.4%. I have drive balancing turned off for all plug-ins except StableBit Scanner, which reports the drive as healthy. Any thoughts?

Edit: Never mind #2. The drive was smaller than the rest.

[Christopher (Drashna)]

TeeeBear, is this for StableBit DrivePool 1.X?

If so, run the "reindex" option, and it should fix the issue.

https://app.box.com/s/yp4kbypf2t0ada23qb0z

 

If that doesn't help, the manually rebuild the index:

http://wiki.covecube.com/StableBit_DrivePool_Q7200705B

 

 

If you're using DrivePool 2.X, then under Pool Options, use the "remeasure" option.