Feature requests: Backup disk, keep awake and Balancing

[PoolBoy]

Backup
It would be nice to have an option to set which drives are prefered for duplicates/backups.
There could be various reasons for doing so.
- Just liking to have all backups in one place so other software can mirror them.
- Based on speed. I would prefer to have my backups on my slowest drive.

The last one may need some explanation. If understood correctly DrivePool uses both the original and duplicate files to speed up transfers. That's great if all files are duplicated. But not if not everything is duplicated.
Example: The pool contains duplicated data which automatically gets a speed boost. But it's also used for temp files, replaceble data etc. It would be nice that those files would not end up on the slowest disk.

The above should be a strict guideline, not a unbreakable rule. If DrivePool runs out of backup space it starts using other drives. Maybe a drive that's ranked as a second choice backup drive. Optionally DrivePool could give a warning the backup drive is full.

Keep awake
I have USB disk attached that go to sleep and take sometimes 10+ seconds to wake up. To stop the drives going to sleep I run a little program that simply accesses the disks every few minutes.That functionality would be nice to have in DrivePool. Possibly with a schedule.

Balancing
I understand the need and benefits for balancing and just keep doing as needed.
Still I have a request.
Drive contents can change greatly. Like lots of deleted or added data.
A empty drive can be added to the pool. Etc. Etc.
My suggestion can take hours to complete and therefore possibly should be scheduled at night or manually.

Maybe I one will unlearn this preference but I prefer my files places as there were when the drive was added to the DrivePool.
I know I can lock folders from moving, but I think I should give DrivePool some breathing room to do it's stuff and not cripple it with a lot of rules.

I want to suggest that whenever possible the balancer merges the folders of the same root when possible. Even if that means it has to shuffle around 5TB of data. If not possible reduce the amount of breaking them down as much as possible.

[Umfriend]

For "Backup":

Manage Pool -> Balancing -> Balancers -> Drive Usage Limiter -> Check / uncheck Duplicated / Unduplicated. If you uncheck Unduplicated for the slower HDDs then the unduplicated files will end up at the faster HDDs. However, a duplicated files may still end up at two faster HDDs. There is no way to direct one to a fast and the other to a slow HDD. You can not do this because DP does not actually make "backups" of duplicated files. It makes "duplicates". The difference being that all copies of a file have the same status, there is no "original" vs "backup" distinction.

You could work with hierarchal Pools though, that would make it work at the expense of being a bit less flexible with space usage. You might be able to use the File Placement Rules but I don't use it and have no clue how it should work for you. The File Placement Rules, as I understand it, would however be very helpful wrt to keeping folders/data on specific HDDs (your 3rd Q).

 

[Jaga]

Keep awake - get a copy of Stablebit Scanner and tell it not to throttle SMART queries so the drive stays awake.  Or use the USB drive's utilities to set it's sleep timer.  Drivepool just manages the pool and allows disks to spin down if that's how their firmware or Windows is set. 

As far as balancing goes - I turn off automatic balancing so files always stay where they were originally put.  Less wear and tear, less of a performance hassle if the server is being used.  New files are added to the disks with the most available space, bringing it back in line with continuing to be balanced.  If I ever see the balance on the disks being less optimal than I'd like, I kickoff a manual re-balance with the Disk Space Equalizer plugin and go to bed.  Really the only thing that kicks the pool out of balance for me is the Monthly Full & 3xx/week Differential backups I put on the drives.  And I don't care if it's slightly unbalanced because of them - it wipes the fulls every month, and it only keeps two copies of the differentials at any one time.  Plus - you can force those to specific disk(s) with file placement rules.

If you -want- to schedule periodic (nightly) rebalances, turn on "Balance every day at.." in the Balancing UI.

I think that Drivepool does a great job of balancing on it's own, and it's something that is easy for people to overthink.  Just disable automatics and manually do it when you want.

Edit:  I attached a screenshot of my pool drives - I haven't done a manual re-balance on it for over a month now, and probably won't for a long time.

[Umfriend]

So I am not entirely sure but I think it works like this:

1. Scanner gets a SMART warning and tells DP to evacuate the relevant HDD

2. Files are evacuated to another HDD.

3. Now some files may be unreadable on the relevant HDD. I *think* you first need to remove the HDD from the Pool and then have the Pool remeasured. Should DP find some files that are to be duplicated x times are no longer duplicated x times, it will re-duplicate.

But it may be a bot more automated even.

[PoolBoy]

I have Scanner installed now. 
I can't find the throttle SMART option. I did find the Standby time -> Standby enabled.
I can switch that off but it gets turned back on automatically.

Good tip on the balancer.
What's the best way to move files back to their original disk? Shut down the service and then go inside the hidden PoolPartxxxx folders and just move?

 

But that was only part of what I requested about the balancer.
As per your suggestion I turn of the balancer.
One of my disk has a root folder called HomeVideos.
a] What happens if I write new stuff to that folder? Will it be on the same disk until that disk is 100% full?
b] If the disk is 100% full then DrivePool obviously is forced to place new content on another disk.

 

About a] I want new content always on the same disk.
About b] That's good/logical behavior, but leads me to my original request. If I delete stuff that's on the original disk, I want stuff from other disks to be moved to make use of that empty space. 

So if HomeVideos is on D: and because D: is full some new content is placed on E:
If I delete 10GB from D: it would be nice to move 10GB from E: to D:

[PoolBoy]

19 hours ago, Umfriend said:

For "Backup":
....
You can not do this because DP does not actually make "backups" of duplicated files. It makes "duplicates". The difference being that all copies of a file have the same status, there is no "original" vs "backup" distinction.

Understood, but that raises another question :-)

Let's assume Scanner reports a disk is going south.
Is removing the disk from the pool (in the DiskPool software) all it takes to move both duplicated and unduplicated files?
Will DrivePool reduplicate duplicated files that are no longer accessible due to bad sectors?
If the disk is totally dead will DrivePool start duplicating the files that were on the crashed disk?
 

Those questions are quite relevant for me because Scanner gets quite nervous on several of my disk.  For example one disk parked for over 340,000 times while it's rated at 300,000 times.
Some disks are over 5 years old, but no bad sectors.
 

I've read this: https://stablebit.com/Support/DrivePool/2.X/Manual?Section=Removing a Drive from the Pool

But it's still not clear if DrivePool reduplicates files it can't read, or if the whole disk is dead.

[PoolBoy]

Quote

3. Now some files may be unreadable on the relevant HDD. I *think* you first need to remove the HDD from the Pool and then have the Pool remeasured. Should DP find some files that are to be duplicated x times are no longer duplicated x times, it will re-duplicate.

But it may be a bot more automated even.

That's clear.
I'll just remove and remeasure, until I know 100% sure remeasure isn't needed. Better safe than sorry.
Looks like I have to start shopping for a new disk because I have not enough space to evacuate my largest disk. Unless I can limit how much space DrivePool uses on a certain drive.

 

Feature request:
Because evacuation is so important an option to temporarily evacuate to another pool or disks not being part of any pool would be a great option if the pool the dying disk is in doesn't have enough space to evacuate all data.
Maybe prioritize non-duplicated data?
 

[Christopher (Drashna)]

9 hours ago, PoolBoy said:

Because evacuation is so important an option to temporarily evacuate to another pool or disks not being part of any pool would be a great option if the pool the dying disk is in doesn't have enough space to evacuate all data.
 Maybe prioritize non-duplicated data?

That ... could get super complicated, really fast.  

What happens if the system disk is what is failing?  Or if you have databases?  Etc.  THis could absolutely break configs and cause a huge load of trouble.  That's why we've let StableBit DrivePool handle this.

 

17 hours ago, PoolBoy said:

But that was only part of what I requested about the balancer.
 As per your suggestion I turn of the balancer.
 One of my disk has a root folder called HomeVideos.
a] What happens if I write new stuff to that folder? Will it be on the same disk until that disk is 100% full?
b] If the disk is 100% full then DrivePool obviously is forced to place new content on another disk.

Likely not.  The default placement strategy is to write the data to the disk with the most available free space.   If you're using a balancer plugin, that changes the order.  Also, if you're using file placement rules, and have targeted this folder, then it depends on the rule. 

That depends, but yeah, it should use a different disk.