McFaul Posted June 9, 2015 Share Posted June 9, 2015 Hi, I have 28 drives in my pool. I've enabled duplication for the first 50TB of my files, and it is duplicating away. However duplication seems to be single threaded, for want of a better description. Its only working on on file at a time. I can see in stablebit scanner that its reading from one disk and writing to another. But that leaves the other 26 drives doing nothing. Is there a way to change this? Since they all also have lots of un-duplicated files on them, they could be copying to/from each other too? i.e. files could be simultaneously duplicated form disk 1 to disk 2 ; and from disk 3 to disk 4 and from disk 5 to disk 6; all at the same time. at this rate its going to take a VERY long time to copy 50Tb of data one file at a time with 26 of my drives idle at any given time. Chris Quote Link to comment Share on other sites More sharing options...
0 Christopher (Drashna) Posted June 9, 2015 Share Posted June 9, 2015 Honestly, I'm not sure about that. I'll ask Alex (the developer). However, the priority can be boosted UI, which helps with copy speeds. And there is an advanced setting to disable "background IO" for duplication: http://wiki.covecube.com/StableBit_DrivePool_2.x_Advanced_Settings set "FileDuplication_BackgroundIO" to "False" and restart the "StableBit DrivePool Service" (run "services.msc" to do so, or reboot the system). Quote Link to comment Share on other sites More sharing options...
0 McFaul Posted June 9, 2015 Author Share Posted June 9, 2015 yeah i already spotted the boost - that upped it from 30mb/sec to 150mb/sec.. but it still seems such a waste to only be using 2 out of my 28 disks perhaps its non-trivial to make it parallel - since the UI tells you which specific directory it is working on, but its definitely something worth considering for Drivepool v3 as it would certainly speed up my duplication (which has been running since friday night and is still going...) Quote Link to comment Share on other sites More sharing options...
0 Christopher (Drashna) Posted June 9, 2015 Share Posted June 9, 2015 Yeah, by default, we use background IO, which means it's slower but won't interfere with normal usage. Disabling that or bumping the priority will definitely copy faster. As for multithreaded, yeah, we are definitely running the duplication task as a single threaded process. I can make a good guess why. The placement of the files are important. They need to not violate the balancing settings and the file placement rules (if any), and the copies have to be on different physical disks. By making it multi threaded, it will mean that there is more work to be done in the backend to track where the files are ending up, and where new ones should be copied to. This would definitely make the code handling duplication a lot more complicated. However, it sounds like Alex would definitely like to implement this in the future. Quote Link to comment Share on other sites More sharing options...
Question
McFaul
Hi,
I have 28 drives in my pool.
I've enabled duplication for the first 50TB of my files, and it is duplicating away.
However duplication seems to be single threaded, for want of a better description. Its only working on on file at a time.
I can see in stablebit scanner that its reading from one disk and writing to another.
But that leaves the other 26 drives doing nothing. Is there a way to change this?
Since they all also have lots of un-duplicated files on them, they could be copying to/from each other too?
i.e. files could be simultaneously duplicated form disk 1 to disk 2 ; and from disk 3 to disk 4 and from disk 5 to disk 6; all at the same time.
at this rate its going to take a VERY long time to copy 50Tb of data one file at a time with 26 of my drives idle at any given time.
Chris
Link to comment
Share on other sites
3 answers to this question
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.