Ultradianguy

I'm trying to figure out the best way to split my existing pool into subpools for two purposes. I recently bought a drive that is much bigger than all my existing ones by more than double, which of course means I can't use all that space for duplication. It occurred to me that I could deal with this by creating pools of a few drives at a time that are about the same size and then use those pools in my primary pool. That way each subpool could be used for duplication. I also saw a suggestion from someone that this was a good way to deal with end of life drives by creating a pool of old drives to be used for ONLY duplicated data. If the EOL drives are mixed into the overall pool, there's a risk that all copies of a file could end up on EOL drives. By putting them all in one subpool and using duplication only ACROSS subpools, this risk is eliminated My question is how do I move existing drives in my one very large pool - into separate pools to do this. Is there a way to do it without having to remove all existing content, remove from the existing pool and then recreate them? This will take a LONG time and will be a bit of a puzzle in order to move things around in the right order. I have about 40TB of data right now so it's not a trivial amount of data to move around.

Hi @Christopher (Drashna) I just added a new 20TB drive and wanted to have it scanned immediately before putting it into use. Unfortunately, there seems to be no way to tell Scanner - scan this drive now. I clicked Start Check (it was in automatic mode before), but that just starts scanning according to whatever rules Scanner uses to prioritize. It's working on other drives that have been previously scanned and still hasn't touched the new one. It would be very helpful to be able to right click on a drive (or whatever) and tell Scanner to do that one immediately - and thus override whatever other options I've set that might prevent immediate scanning. Thanks, Michael

@methejuggler @Shane I had a couple questions/requests regarding this very helpful balancer (these address gaps that were present in the official covecube balancers as well). Currently, there is no way to do what seems most useful to me - have equal space used for unduplicated data across drives. That is, if I have 20TB of data , and 4 drives, I'd like 5 TB on each drive, so that if one goes, I minimize the loss of unduplicated files. Currently, you can do this by % of space used, which does the same thing if the drives are about the same size. But over time, I've added larger drives as prices have dropped. So I have a mix of 2TB, 4TB, 6tB , 8TB and now a 20TB drive. If I use equal % used or equal empty space, I end up with far more data on the larger drives. This seems like a pretty obvious use case, and would be easily resolved by an option to equalize by actual space used, rather than percent. There is currently no way to equalize by absolute amount of data. The other issue I'm having was with the covecube balancers. I just switched to All in One so it remains to be seen if it will still be an issue But I found that when I set the balancers to equalize by percent used for unduplicated (unchecked duplicated), that the balancing ignored that and equalized total space used. As a result there were very large differences in unduplicated data across drives. I wasn't sure if the new all in one uses different code or just puts the existing code into a better UI . Thanks for the work on this!

I second that request. I would also like to see a rebalancing window in time rather than just a time to start balancing each day. Covecube folks - it's disappointing to see how little development has taken place on Drivepool for many years now. It's still a solid product and much appreciated, but it seems that the recent upgrades have been related to Stablebit Cloud, which is of course a subscription service. I paid for that to support you guys even though I don't think it adds that much - but if I'm going to pay for a subscription service, I'd rather pay for the products I need (Drivepool and Scanner) and see improvements in those. Drivepool has needed improvements in options for years now and almost none of the requests have found their way to production. I would happily pay a reasonable subscription cost if it would mean more development.

Curious about this @methejuggler. In another thread, I thought I saw that leaving Archive and SSD both unchecked means nothing will get put on the drive.

Same experience here.

I just added a new drive and Drivepool is rebalancing. So far - it seems to be doing things right (though very slowly as always) - been a couple days already. However, I noticed that another drive (that had been in the pool before) has the unduplicated marker at zero as if I had set the drive to have no unduplicated data. But in fact, I did not turn off unduplicated on this drive nor are file placement rules limiting that drive to folders that are duplicated. I also checked drive health in Stablebit Scanner - there are not damage indications that would trigger removing unduplicated data. Why does drivepool want to prevent unduplicated files on this drive? Michael

Hi - I noticed there is a new balancer called All in One that seems to combine the SSD balancer with a few others. Has anyone used this one? It seems it wasn't developed by Covecube so I'm wondering if it's reliable. Also - I see there is a checkbox for archive drive in addition to Duplicate and Unduplicated. I'm assuming if you pick SSD, it's to indicate you want to write cache - otherwise you can just treat it like any other drive. So why would you check Archive? Thanks

According to Christopher, this is expected behavior. I could use the ignore command from CMD but since the drive was damaged, I just removed it permanently, so not worried about it getting added back. But apparently the Remove command does nor prevent DP from recognizing a drive and adding it back in.

Thanks for the input. Due to other computer problems unrelated to DP, I ended up having to restore a system image from a few days before the problem with the drivepool service started, so that fixed the issue with no service.

Wow - all of that? I was just going to uninstall and reinstall. Why did you remove everything from the pools and re-create?

Just a bit more information - it turned out that although the pool drive folders were all there, they appeared empty, though the files were still on the underlying drives. However, when I run the service from the command line, it does work, and the pooled drive does become fully accessible. So - the question is how do I get the service re-registered into the Services panel - and why does the installer fail? Is it safe to uninstall and then try reinstalling? (I can never use the phrase "is it safe" without thinking of the movie Marathon Man). Thanks!

Hi - I've had drivepool running fine for quite some time (other than being slow). Today I noticed that the UI was stopped looking for the service. Went into Services to restart and the service was gone. Determined that Bit Defender had quarantined the service executable, which I restored. Couldn't get it to show in Services panel though. Tried running the service from command line and it started successfully, but still doesn't show up in the Services panel. I've tried restarting the system - no help Figured I should reinstall drivepool, but the installer doesn't get past initializing, and eventually says it failed, with an error code about "The pipe is closing". I'm attaching the log file. Strangely, the pooled drive is still accessible despite the service not running - I don't know if that is expected or not. I haven't tried uninstalling drivepool yet - wanted to make sure that wouldn't make anything work. It clearly started with Bitdefender quaranting the service, but not clear to me why I can't get things reset. StableBit_DrivePool_(64_bit)_20191126144326.log

Hi Chris and others - I'm pretty confused about the SSD optimizer and how plugins work together. I've read in some older threads that if you use the SSD optimizer you should disable other balancers - but that means losing a lot of important functionality, which is not at all redundant with SSD optimizer. I thought the point of the SSD plugin was that files would initially go to the SSD, and then from there, would gradually be moved to other disks, duplicated on the archive disks and balanced according to rules. It doesn't make sense to me that these would be in conflict with the SSD optimizer . This would imply you must choose between faster writing to the pool or having files end up where you want. Also - I read that you need to have as many SSDs as you your max duplication - is that right? Again - this doesn't seem to make sense if the SSD is a temporary location. If I turn off real time duplication, do I still need multiple SSDs? If SSD optimizer is highest prioirty - shouldn't off loading to archives happen before all the other balancers? It seems to me that this plugin should be designed to not worry about duplication since that would reasonably be for long-term storage. Or at least there should be an option to say - "don't worry about duplication until you get stuff onto the archive (non-SSD) disks." Also - someone had said that using multiple balancers result in them "fighting with each other". Again - this seems to contradict the prioritization. If two balancers would result in different arrangements - shouldn't the one with higher priority win in situations where they conflict? So - for instance - if Drive Usage is first - and it says put unduplicated files on drives 1 and 2 and 3 but not 4, and the next balancer is equalize distribution across disks based on unduplicated files percentage, my assumption is that DP will balance unduplicated files across disks 1,2 and 3 but NOT 4 since the higher priority plugin says don't put them there at all. However, one poster implied that after doing the first balancing, DP would then re-balance according to the next balancer - even if it conflicts with the previous. If so - it isn't really prioritization - it's just sequencing. If that's really the case, then one really should only use one balancer, which again, makes them much less useful. I'm hoping what I read was incorrect. Is this not the way it works? If it isn't, we need some better documentation on how these interact. I have assumed that I can use as many plugins as I'd like, with the understanding that prioritization position determines which have the most "say" over arrangement. I

I'm setting up snapraid to use along with drivepool - lots of threads on here I've seen. However, I'm running up against a couple things I haven't seen mentioned. Unlike some others, I AM using some duplication and some DP balancing. I realize that snapraid won't be able to complete parity until the balancing settles down - that's fine. Once it's all rebalanced, the drives I'm using with snapraid will have static media files. Question 1: I have some files that will work better outside of the pool - such as image backups. I've seen people recommend setting the base folder for snapraid as the poolpart folder rather than the root of the drive - though I've never understood the reason for this. But doing that would presumably exclude files outside the pool. Would I be better off just setting the data drive to the root rather than the hidden poolpart file? (Maybe if I understood the reason for doing the latter, this would be clear). Also - as snapraid does the sync, it's reporting files that it thinks are copies but have different file data than the "other file" with the same date and size. Those files appear to be file backups of a database that gets saved frequently in dated folders. Snapraid says if it's a "false positive" I can rerun sync with a nocopy switch. What's unclear to me is whether snapraid is saying "I'm not going to count these as different files unless you use that option , so if you don't want me to ignore the additional file, use "nocopy". Thanks !

Got an old 640GB drive from a system being tossed (WD Blue) - manufactured 2009. Obviously it's pretty small, so it's not a big concern, but I was surprised to find that Scanner started marking every sector as bad - I stopped it after something like 30,000 bad sectors. I ran chkdsk with the full scan and fix, and it reported no problems at all. ( I also did a full format and again ran it through scanner - same thing. And again chkdsk says all good). The drive was plugged into a USB dock that I've had no previous problems with. It also said no SMART data was available - but I assume that's just due its ancient vintage. Are there known differences in older drives that would cause Scanner to think everything was bad?

Same here, with current version of Scanner .

So I've been having trouble getting a potentially damaged drive evacuated and removed. I've posted other places about that so I won't go into those issues. However, after manually removing unduplicated files, I physically took the drive out of the system. When DP said there was a Missing Drive, I clicked Remove, and since it was absent, it pretty quickly got removed. I then plugged the drive back in and it immediately appeared back in the pool - rather than in unpooled drives. This is not what I would have expected - I'm wondering if this is intentional or a problem. My assumption is that once I click Remove and DP says it was successful, the drive will not re-appear in the pool unless it's deliberately added. I'm guessing that this is related to having used the Remove command when the drive was missing - but still, I wouldn't expect this. I couldn't use the Remove command while the drive was still present as there was no progress even after a couple days, despite there being only a few hundred GB of files on the drive. (Note my pool has been measuring for about a week, with minimal progress).

I've been puzzled why unduplicated content on a drive with SMART warnings and damage was not being removed, despite setting various balancers to do so. I discovered today part of the reason - possibly all of it. I believe all the remaining unduplicated content were backup images from Macrium Reflect. Reflect has an anti-ransomware function that can prevent any changes to backup files. This was preventing drivepool from removing them. I realized this when I shut down drivepool service and tried to manually move those files to another drive. I'd have expected drivepool to report that files were inaccessible - but apparently it did not know other software was actually blocking it - which brings me to the next issue. Stablebit Scanner reported 20 unreadable sectors on this drive and 3 damaged files. SMART had indicated 5 pending sectors. I decided to re-run the scan after disabling the Macrium Image Guard - so far, it appears the unreadable sectors may have been caused by Image Guard and may not be bad. Remains to be seen whether the 5 pending sectors will end up being reallocated or become readable. The damaged files however were NOT image backups, so it's unclear if there was any connection. Bottom line: don't use Macrium Image Guard (or any other similar software) with pooled files. I may just move my image files out of the pool to avoid the issue.

I've resolved a part of this but I'm going to start a couple new threads so people can find in the future.

the disconnects were actually on two other drives - not the damaged ones. I'm not sure I'm understanding the dpcmd ignore -- I don't know which files on this drive are unduplicated and which are duplicates - so it's not clear to me how I should manually remove the files. That's why I set the drive usage limiter to remove only unduplicated files - but that hasn't happened at all. Feel free to respond on my ticket if you prefer.

Also , as I've explained in our private messages on my ticket, I tried the force damaged disk removal - its not that it's getting stuck on damaged files -it's that it hasn't even begun removing files from this drive because it's been measuring the pool for days. It did at one point start to balance but only moved a few files on other drives and barely touched this one.

Hi Christopher - I don't think this has anything to do with the damage on the disk. It's only 20 sectors and 3 files, of which I was able to recover 1. I can open folders on the drive, open files, copy files, etc And drive pool is generally taking forever to measure and balance the pool - it goes way beyond this particular drive.

I'm experiencing the exact same thing, except it's been going on for nearly a week. Did you resolve this? I've not been able to get drivepool to clear off a damaged drive by Remove or by using the File Usage balancer. Measuring is taking forever, and almost no balancing has occurred. I've also experienced the drive disconnected message and then reconnected a few minutes later. And each time this happens, DP clears all it's info and starts measuring from scratch.

I've been communicating with Christopher about these same issues - and it's disappointing to see that the exact same discussion was going on 4 years ago and pretty much nothing has changed. I love Drivepool and I think it does a very good job at it's primary intent - but when something is off - when a drive needs removal or a significant rebalancing is needed - the same issues discussed here are still problematic. I've been trying to evacuate the unduplicated files on a damaged drive for several days now - there are less than 500Gb of undupe files - but the process is still moving glacially. According to Scanner - there are 3 damaged files - I was able to recover one, and I don't care about the other two . Using the Remove function locked the pool and almost nothing happened after nearly 24 hours. Using the approach of using the balancing plugins and setting the drive usage limiter to zero as a way to evacuate the drive and/or using the Scanner balancer to remove files from a damaged drive has also barely removed any files. Worse yet, it seems that each time you abort any process, Drivepool seems to re-measure the pool, which itself has been taking longer than a day. At first, i was having performance problems due to a bad Windows update, but since resolving that, all of the above are still true. I I wholeheartedly agree that it would be incredibly helpful to know which files are in the unduplicated group. I'm going to try moving some of the files out of the pooled drive and onto an individual drive using Windows explorer - hopefully that will be more efficient. I know you guys have been focusing on other products - and I'm happy to see you developing the business and having success - but Drivepool needs some real attention.