Shane
-
Posts
1035 -
Joined
-
Last visited
-
Days Won
105
Posts posted by Shane
-
-
A way to do it would be Manage Pool -> Balancing -> Settings -> tick "File placement rules respect real-time file placement limits set by the balancing plug-ins", then Manage Pool -> Balancing -> File Placement -> add a Rule for "*" with all drives ticked except the one you want to leave empty.
-
If your goal is to maximise the available free space for incoming files, you could use the SSD Optimizer balancing plugin to set the largest drive(s) as "SSD" and the rest as "Archive", with high Fill values, then set Automatic balancing to "immediately" and the Triggers to "100%" / a very small file size, so that files come into the pool via the "SSD" drives and then automatically migrate to the smaller "Archive" drives (you could also use the Ordered placement section of the SSD Optimizer so that migrated files go to the smallest "Archive" drives first).
"I could split these larger drives into multiple partitions to reduce the backup size making the pool more viable but then the structure between the drives becomes more complex requiring links or mount points and more backups to schedule. Not undoable."
Depending on the nature of the backup source, perhaps you could use another backup method that doesn't need to create singular massive backup files? E.g. I use Veeam for single-image backups of my family's laptop PCs but use FreeFileSync for file-by-file backups of the family pool (lots of archives, docs, photos, mail, videos, etc).
-
Yes, if all files are recoverable (along with which folders they belong to) the pool can be rebuilt from scratch (otherwise it'd be a bit of a jigsaw puzzle). Files are instanced, not striped, across the disks in the pool (thus the loss of any number of disks has no effect on any files stored in any remaining disks).
Pool content is entirely stored as normal folders and files in each poolpart folder, while the pool guid (the bit that basically says "this particular disk belongs to pool XYZ") and duplication levels are stored in alternate data streams attached to each instance of the poolpart folders and the content folders they contain. Only the user preferences (pool management, balancing and placement configurations, none of which are necessary to a pool's survival) are stored in the application folders of the host machine.
Basically you can take a pool's drives straight out of one machine running DrivePool and plug them into any other machine running DrivePool and it'll go "oh hey, a new pool with content already in it", you'd just need to redo your preferences - and if every copy of DrivePool and/or Windows mysteriously disappeared you could still just plug those drives into any other OS that can read a NTFS volume and all your stuff would be there (though in the latter case if you'd done anything fancy with reparse points, e.g. symlinks or junctions, I think those might need to be redone).
-
Hi, if the damage is to the file table and/or pool metadata on that drive then it could cause that kind of problem (the very wrong size suggests at least the former has happened).
What I'd do:
- Shutdown the system and disconnect/depower the damaged drive.
-
Restart the system and check that the pool and Explorer are functional again (the pool should be in read-only mode due to the "missing" damaged drive).
-
If it's functional and your pool was fully duplicated:
- Remove the "missing" damaged drive from the pool. DrivePool should resume normal operations and proceed to re-duplicate.
- Attempt a long (not quick) format of the damaged drive on a separate system, then use Scanner to verify its status.
- If it comes back good then you can decide whether to add the "new" no-longer-damaged drive to the pool. You're done.
-
If it comes back bad then you can decide whether to add a new replacement drive to the pool. You're done.
-
If it's functional but your pool was not fully duplicated:
- Do a repair (e.g. with chkdsk) of the damaged drive on a separate machine. Make note of any files it is unable to recover in case you can restore those from backup.
- If the damaged drive wasn't repairable, remove its "missing" entry from the pool, replace the damaged drive with a new one and restore the lost content from your last backup of the pool (if any). You're done.
-
If the damaged drive was "repaired", it's still possible that its pool metadata remains damaged (since chkdsk doesn't fix that) and it may still have bad sectors:
- If you'd prefer to rely on your last backup, see step 4.2 above.
-
If you'd prefer to attempt recovery from the external drive:
- Before reconnecting it to your system, rename its hidden poolpart folder (e.g. rename the "Poolpart" text before the GUID string to "Suspect").
- Remove the "missing" repaired drive from the pool. The pool should resume normal operation.
- Connect the repaired drive to your system and manually copy the suspect poolpart's content into the corresponding folder tree in your pool (do not copy it directly into another poolpart) - or alternatively you may wish to copy it into a temporary folder / compare it with your pool to decide later.
- See step 3.2 - not to be confused with step 4.3.2 - regarding formatting and verifying the damaged drive's suitability for continued use.
Hope the above helps!
-
"1. With duplication on...I should still have one good/in tact version of the file on a different drive somewhere? Is that correct?"
A1. Yes. The normal procedure in the event of a bad drive, if the entire pool is duplicated, would be to physically remove/replace the bad drive then tell DrivePool to remove the missing drive so it can reduplicate using the good drives. However, see 2.
"2. I have tried recovering the file with Scanner and it didn't work. It finished with a "partial file recovery" message...but I assume that essentially just means it didn't work?"
A2. I believe that means it was able to recover only the undamaged parts of the file. I'm not actually sure what that means for the good/intact version on the other drive (replaced? unchanged? I'd guess the latter if both instances still have the original last-modified date). The best option is as you mentioned to attempt A1 anyway and test the result to see if it still works (in this case I guess by watching the video).
"3. ...could I just manually go into the hidden pool folder on the hard drive that has the bad sectors and manually delete the affected file? Then with that file gone...it's obviously no longer duplicated, so I'd just let drivepool redo the duplication and that would essentially recopy the good version of that file into the pool in a different location to get the duplication back? I assume those bad/unreadable sectors are already marked as such and the HDD will avoid trying to write there for any future writes??"
A3. You could indeed just manually delete the bad instance inside the affected poolpart and let DrivePool automatically reduplicate from the good instance elsewhere, and that should be what happens, but as you've already realised it is at your own risk (whether more sectors fail on that drive - or on any other in the pool).
P.S. Incidentally for very large static files in addition to duplication I use MultiPar to create parity files of them so that in the black swan event of any bitrot or damaged sectors somehow managing to slip through duplication (e.g. by being moved across to a whole new pool) I still have a good chance of repairing the damage via the par2 file. YMMV!
-
Hi, in the settings.json file - see https://wiki.covecube.com/StableBit_DrivePool_2.x_Advanced_Settings - you can try setting the override value for FileBalance_BackgroundIO to False so that DrivePool doesn't use the lower background priority for balancing (similarly FileDuplication_BackgroundIO for any scheduled duplication).
You can safely stop the current balancing by pressing the 'x' button in the GUI next to the pool organization bar.
-
Just to confirm, in each of your relevant File Placement rules have you selected the option "Never allow files to be placed on any other disks"?
The default option is "Allow files to be placed on other disks if all of the selected disks are this full" with a slider defaulting to 90%.
-
Hi, DrivePool reports the total available space of the pool to the OS for reasons including:
1) the pool is a NTFS volume and the operating system expects to see total space = (total) used space + (total) free space.
2) explorer and various utilities will refuse to perform a dialog copy operation if the total size of the files to be copied is larger than the free space reported by the target - which means otherwise there would be situations where you'd have plenty of room to receive the files but reporting only the space of only one of the disks would result in explorer/utilities thinking it can't be done.
For example, let's say we had a pool of two 2 TB disks with 1 TB free on each disk:
- if DrivePool reported only the 1 TB free, then Explorer would refuse to copy a thousand 1.5 GB files even though there's plenty of room.
- if DrivePool reported the 2 TB free, then Explorer would proceed to copy them.
- either way Explorer would still fail to copy a 1.5 TB file to the pool.
So DrivePool reports the total free space to the OS and uses the GUI to tell the user the free space per disk.
-
I'd say so.
-
My guess is because he only had two disks in his pool and those disks were only 233 GB each in size, while you've got nine disks and those are 2.73 TB each in size. Larger disks can mean more Other even when empty.
I can see you're using a bunch of 2.73 TB drives so I've grabbed one of mine, formatted it and made a pool of 1 disk; Other is 240 MB. Nine times that is about what you're seeing on yours.
-
Looking forward to seeing the difference if any as the pool is refilled.
-
I'm about out of ideas. Other is basically "not DrivePool's monkeys" and from the screenshots I can't see what could be causing it. The disks are just normally formatted NTFS, default values e.g. 4k cluster size allocation, yes?
-
Yeah, that would seem to rule out sysvol and recycle. And I can see there isn't much difference between the Logical and Physical sizes (slack space).
There's also the possibility that the Other needs to be remeasured (e.g. if the pool has been seeded via copying directly into poolparts) which you could manually trigger via the DrivePool GUI -> Manage Pool -> Re-measure...
@Christopher (Drashna) is there a way to get DrivePool to show a breakdown of Other? I tried increasing the File Size Tracking, Pool Metrics, Pool Part Metrics and Pool Statistics details in the Service Log to Verbose for a Re-measure but it still only displayed the PoolPart file sizes for the various duplication levels, so I'm wondering if it just does something simple like "Other = Total of Disk Used Space Sizes - Total of PoolPart File Sizes"?
-
Instead of using WinDirStat to look in E: (your pool drive) try looking at R: to Z: (the disks that form it). Each of them will have their own $RECYCLE.BIN folder that is separate to the pool (and thus part of Other).
-
If it's mostly photos I would guess your Other is too high to just be overhead/slack.
I would suggest trying a measuring program such as TreeSizeFree or WinDirStat (either run as Administrator) on the disks that form the pool to see where the Other is lurking.
-
If your "Other" data seems large, check the "$RECYCLE.BIN" system folders of the disks; sometimes they don't empty via the Recycle Bin icon but you can safely manually delete their contents (or even the folder itself; Windows will recreate it when it needs it).
For example, on a 39.1 TB pool I'm looking at right now "Other" is at 150.2 GB while "Duplicated" is at 30.1 TB and "Unduplicated" is at 1.30 MB; after manually deleting everything in the recycle bins on the pool disks "Other" is now 10.7 GB.
Another possibility is that it could be due to large amounts of metadata and/or slack space, which accumulates for example due to having a very large number of files.
-
Unless an application is deliberately bypassing DrivePool to directly access a pool's internal poolparts it shouldn't (so far as I know?) be encountering MAX_PATH problems any more than it would with any other drive - i.e. as far as any app accessing the pool drive is concerned, the path of anything in the pool should begin with "p:\" not "p:\poolpart.guidstring\".
If you do have a need to use such an app to directly access the poolparts, and you can't find an alternative nor get the app's developer to fix its limitation (e.g. if the app isn't being maintained anymore?), I'd suggest opening a ticket with StableBit to make the feature request; note however that the guidstring is part of how DrivePool separates and distinguishes between poolparts across multiple pools even when moved between machines; using a short incremental counter instead would require finding a new way to implement that.
Have you run into a situation where an app with a MAX_PATH limitation needed to directly access the poolparts?
EDIT: possible workarounds also include creating 8dot3 short names for the poolparts via the fsutil command or creating junctions with short names pointing to the poolparts via the mklink command (be careful with the latter to avoid creating any recursive loop).
-
Hi Chris, I was under the impression that the Disk Space Equalizer balancer is not capable of real-time file placement (since it's not mentioned in its tooltip)?
-
Good to see. I'd suggest opening a ticket with StableBit to report your WinDbg analysis of the BSOD with 2.3.11 and that reverting to 2.3.8 stopped it, in case it's useful / they can ask for the dumps from the crash to trace the problem.
-
If you uninstall DrivePool 2.3.11.1663 and install DrivePool 2.3.8.1600 (download folder link) does the BSOD stop happening?
-
Hi sparkrdom, did you delete the "System Volume Information" folder that is within the hidden poolpart folder on the disks?
E.g. if e: is a drive in pool p: then there would be an "e:\System Volume Information\WPSettings.dat" file and an "e:\PoolPart.guidstring\System Volume Information\WPSettings.dat" file (where guidstring can be different for each drive) and it's the second one that you would need to delete.
-
Hi Fox01, just to confirm what version of DrivePool are you running?
-
(skip to the TLDR at the bottom if you just want a specific solution)
Volume Equalization is for any physical disk that has been partitioned into multiple volumes that are part of the pool, while the Disk Space Equalizer is a scheduled balancer (if you rest the mouse cursor over a balancer the tool-tip mentions "can set file placement limits" only if it's able to act in real-time) so unfortunately neither would help here.
(Note that DrivePool preferring to write to the drive with the most free space is because it's impossible for a file system to guarantee knowing the size of a new file, the software's design prefers reliability over performance when it has to choose).
That said, it's not impossible - just a little fiddly - to get DrivePool to write separate files to separate drives in the pool at the same time. First you need to have separate copy operations happening at the same time: if you use Windows Explorer to highlight two folders and drag them to the pool Explorer will only copy the contents to the pool one file at a time, whereas if you drag each folder separately to the pool drive Explorer copies them concurrently.
Second - this is the fiddly part - you need to either (1) have close enough amounts of free space on the pool's drives so that DrivePool picks a different drive in the pool for each operation as each drive's free space drops from receiving incoming files or (2) set File Placement rules such that DrivePool is forced to use different drives (e.g. setting a rule that folders/files starting with A go on disk 1 and a rule that folders/files starting with B go on disk 2).
TLDR ~~~> In your scenario, you could create two folders on the destination pool, e.g. "Source 1" and "Source 2" then use File Placement to allocate different disks in the pool to those folders; then you copy from one source into Source 1 and from the other source into Source 2. Once you've got everything into the pool you can turn off those rules or move the content into other folders on the pool, and let the pool balance overnight or whatever.
-
DrivePool's default behaviour is to write to whichever drive* has the most free space; given two empty drives, one large and one small, it will write to the large until the free space is less than the small.
Presuming that you started with an empty pool on empty drives, and the stats in your post are the end result, notice that the bytes free on the two drives are very close.
*(if you have real-time duplication enabled, drives plural)
Stablebit Scanner refuses to start now after HDD failure
in General
Posted
Hi, I'd suggest opening a support ticket with StableBit to investigate. Please let us know how it goes.