Shane
-
Posts
1034 -
Joined
-
Last visited
-
Days Won
104
Posts posted by Shane
-
-
Hi! The short answer is that the goal of "all new files distribute across the storage array in real time while the location of old files in the array remain unchanged whien balancing" can't generally be accomplished because DrivePool considers "old files" to be "all files already in the pool" while SnapRaid considers "old files" to be "any file that is neither new nor changed since the last sync".
Before continuing, it may be useful to understand the following about DrivePool:
- Placement of existing files on the pool, a.k.a. balancing, is controlled by the Balancers and the File Placement rules.
-
Placement of new files on the pool, a.k.a. real-time file placement, defaults to whichever drive(s) have the most free space at the time.
- This is done primarily to minimise the risk of having insufficient free space to complete the writing of a new file.
- The real-time file placement limits that can be set, by the File Placement rules and by some Balancers, can override this.
- If you hover your mouse cursor over each of the Balancers the tool-tip will tell you if it can set real-time file placement limits.
- Unfortunately the Drive Space Equalizer balancer is not one of those able to set a real-time file placement limit.
So it's generally recommended that DrivePool's balancers be turned off (except maybe Scanner) when used together with SnapRaid, to avoid the problems it causes with syncing. DrivePool will still distribute new files across a pool's disks according to their free space (or under whatever placement limits you've set) and that still results in an even distribution of used space across those disks over time if they are of the same size.
In your case however you have one data disk that is significantly larger than the rest, and that skews the new file distribution accordingly.
I'd suggest turning off the balancers and either:
- Accept the skew; once the free space remaining on the 8TB data disks falls to the same level as the 4TB data disks then additional new files will tend to spread evenly over time, there'll just always be 4TB extra data on the disk that is 4TB bigger than the others. The simplest alternative.
- Shrink the partition on the 8TB disk to 4TB for now or replace the 8TB disk with a 4TB disk; new files will spread evenly without the skew. The next simplest alternative.
I hope this helps.
-
Hi Lanti! DrivePool only reports the used, free and total size of a pool drive from the totals of the disks that form it, and the size of any given file as if it were only on one disk. There are multiple reasons for this: it's much simpler and faster to calculate, DrivePool can't guarantee that per-folder duplication settings won't be changed by the user, DrivePool's metadata is stored on the disks at varying duplication levels depending on the number of disks in the pool and this may not match the user's level(s), and reporting any other size for individual files or a pool drive can prevent Windows file operations from working properly.
I hope that helps.
-
18 minutes ago, Shooter3k said:
I'm not sure what you mean by "folder protection options". Do you mean Manage Pool -> Balancing -> File Placement? If so, there's nothing out of the ordinary there and no plus signs.
Manage Pool -> File Protection -> Folder Duplication... shows you the duplication status of folders in the pool (and allows you to change them).
18 minutes ago, Shooter3k said:When I run "dpcmd set-duplication-recursive" from a command line, a box pops up and closes faster than I could ever imagine reading it. Is that intended?
That sounds maybe like you might be running the command line as a standard user but have UAC disabled. dpcmd needs to be run in a command line that's already being run as an administrator - otherwise its console output goes to a temporary window if it works at all - so try opening the command line as an administrator (e.g. via Windows Start Menu, Command Prompt, Run as administrator) and then running dpcmd from that.
-
C:\Windows\System32\dpcmd.exe
You should access it by opening a Command Prompt, run as Administrator, and simply entering "dpcmd" or "dpcmd [command] [parameter1 [parameter2 ...]]" where "[command] [parameter1 [parameter2 ...]]" are what you're wanting it to do (and you'll get the list of those options by entering "dpcmd" by itself).
-
Yes, a volume needs to be Simple and its disk needs to be Basic to be usable by DrivePool.
Windows Disk Management unfortunately does NOT support converting an in-use disk from Dynamic back to Basic without erasing it first.
To perform the conversion without data loss you would need to use a third-party partition manager with that capability, and there is still a risk; I would strongly advise having a backup first.
-
Just confirming, you've tried GUI -> Cog icon -> Troubleshooting -> Recheck Duplication...?
-
I've only seen the "unable to enumerate folder" error be caused by an invalid/corrupt ownership or permission setting on a poolpart or subfolder, but I can't say there aren't other possibilities.
-
I don't know if it has been resolved (and, if the cause is the use of "virtual-unfriendly" function calls by the third party tools, it would not be possible to resolve except by the makers of those tools).
If read striping is enabled and the CRC errors are reported by hashing/verification utilities you rely on, I would turn off read striping.
If read striping is not enabled, then there may be issues with your hardware.
-
9 hours ago, KeithA said:
I was thinking of moving the files out of the hidden folder DP creates and deleting the .covefs folder. Once done I can try and recreate the pool. I haven't done it because I don't know what other problems that may cause. Does anyone have experience with this?
It shouldn't cause problems (on the other hand, I don't know why DP is having this problem!).
Recreating the pool (assumes you only have the one):
- Make note of your settings (performance, balancing plugins, file placement, etc) if you wish to keep them.
- Uninstall DrivePool.
-
Optionally, delete the following folders if they remain:
- "C:\Program Files\StableBit\DrivePool\"
- "C:\ProgramData\StableBit DrivePool\"
- On each drive in the pool rename the prefix of the hidden poolpart folder (e.g. from "d:\poolpart.xyz\" to "d:\oldpart.xyz\").
- Reinstall DrivePool.
-
Create your new pool and add the drives.
- Optionally, restart the computer after adding each individual drive to see if the problem returns even with a new, empty pool, in case a particular drive is involved.
- Disable the Automatic balancing (if used).
- Stop the main DrivePool service.
-
On each drive, move your content from the old oldpart.xxxx folder to the new poolpart.xxxx folder.
- Leave behind - do not move - the "System Volume Information" and "$RECYCLE.BIN" and ".covefs" folders.
-
Optionally, restart the computer after moving each drive's content to see if the problem returns, in case particular content is involved.
- If you're doing this, remember to stop the service before each move.
- Start the main DrivePool service.
- Perform a Re-measure if DrivePool has not initiated one.
- Put your preferred settings back and re-enable the Automatic balancing (if used).
- If everything is behaving and where it should be, delete the oldpart.xxxx folders.
-
Hi SteveMc, you could try the steps I describe in this post: https://community.covecube.com/index.php?/topic/5810-ntfs-permissions-and-drivepool/&do=findComment&comment=34550
-
1) Adding a disk to a pool only makes its free space available to the pool, via creating and using a hidden PoolPart folder on the disk; it does not automatically add that disk's existing content to the pool so any existing files stay where they are.
If you want to quickly move those files into the pool rather than copying them "between" disks (because Windows treats the pool drive just like any other drive) then you can "seed" the pool as per this article - except that in step 4, where it says "For example, find the "G:\Files\" folder, and move the "Files" folder to "G:\PoolPart.xxxx\" you should first ensure that no other disk you plan to seed contains a folder or file at the same level with the same name. Otherwise, DrivePool will think they are duplicates rather than unique files.
To avoid that risk, if you are seeding I suggest instead creating unique subfolders in the poolparts, e.g. moving "G:\Files\" into "G:\PoolPart.xxxx\G\" and moving "H:\Files\" folder into "H:\PoolPart.xxxx\H\", etcetera, so that you see "P:\G\Files" and "P:\H\Files" in the pool and can rearrange them at your leasire once they're in the pool.
2) DrivePool's File Placement rules let you assign folders to prefer or exclusively use specific disks, e.g. you might decide that the "Piano" folder can only be stored on disk A while the "Violin" folder can only be stored on disk B. However, there is no way to "automate" this for new folders (although any subfolders of an assigned folder will default to inheriting the assigned folder's placement).
On that note, DrivePool does not save its own separate record of which files are on which disk in a pool; it simply queries Windows' own file tables in memory for each disk. This means if a disk should fail then you cannot ask DrivePool "what files were on that missing disk", so if you want to keep a record then you will have to do so via some other utility.
-
I'd suggest opening a support ticket to ask for help. Please let us know if there's a general fix?
-
If you're rotating a set of drives (e.g. maybe you've got 4 backup drives, one for each week of the month) then normally Windows should assign the same letter to a given drive once it re-encounters it in the rotation (unless for some reason that letter is already in use - or was in use but Windows decided for whatever reason that it won't free the letter). You may need to manually assign each drive the letter via Windows Disk Management (pick one that is unlikely to be used by other drives connected at the same time, e.g. if your usual drives occupy "C" to "K", pick "U") before this will stick.
If it's always a brand new drive that you're using in the cradle* then normally Windows will assign the first free letter it has available to it (e.g. if you have drives C, D, E and plug it in it should get F) but if that's not happening (because Windows) and you don't want to do it manually then you'd need a program/script that assigns a letter or path based on the cradle's port number or device ID or similar fixed identifier.
... and I've just found this program: https://www.uwe-sieber.de/usbdlm_e.html which may be what you're looking for, I haven't tried it myself.
*(I've also seen it called a dock or caddy, or hub if it can take multiple drives)
-
1 hour ago, one-liner said:
One parity file is placed on a disk outside the pool while the other on a disk inside the pool. The space used by both parity files equals the actual space used by the data they are backing up.
As I understand it, SnapRAID requires that the usable space for each of its parity disk(s) must all remain equal to or larger than the largest used space of each of its data disks to avoid running out of disk space for parity calculations. How does that work with your setup?
-
The only way I know of to stop an active remeasure is to stop the StableBit DrivePool service; any pools will continue to function but balancing won't occur until the service is started again.
You can set DrivePool to remeasure somewhat quicker by clicking the little ">>" icon to the right of the organizational bar in the GUI. You can also adjust the settings.json file (CoveFs_MeasureBackgroundTaskPriority and DrivePool_BackgroundTasksPriority - be sure to read this link first) if you wish to change the priority further.
There is an option in settings.json that is only described in the changelog - "Added PoolPartUpdates_AlwaysConsistent config option to disable the remeasure and duplication check passes after a missing disk comes back online" - but I do not know if it would apply to remeasuring after a forced reset.
-
No idea why it happened but I'd suggest removing it from the pool and recreating the partition at full size yourself, with a full (not quick) format, just to ensure there's nothing off with the drive.
-
As DrivePool relies on the file system as to which drives contain which files, if you didn't have duplication enabled then you'd have to compare with a recent backup or saved listing you've made if any to see what files are missing.
-
A faulty disk could interefere, yes. Best wishes.
-
Hi, if you go to the cog icon -> Troubleshooting -> Service log, is there any activity particularly warnings or errors?
You can also use Windows Resource Monitor to see what files DrivePool.Service.exe is scanning/writing re the duplicating.
If there's nothing obvious, you might also try stopping the duplication, running a Manage Pool -> Remeasure and then cog icon -> Ttroubleshooting -> Recheck duplication to see if that helps.
-
For a NVR server I'd think it would be better to use the default idle priority and single thread count for background tasks (such as remeasures), or even further reducing background task priorities (e.g. DrivePool_BackgroundTasksPriority in the settings.json file) so as to minimise their interference with application disk writes - it doesn't matter that the measuring takes longer if that prevents it from causing problems.
If the recordings are being sent to the server via network share (i.e. the application doing the recording is on another device and writing to \\someserver\someshare rather than on the server and writing to pooldriveletter:\somefolder) then the app's access to the pool can be further prioritised via GUI -> Manage Pool -> Performance -> Network I/O Boost.
For large arrays where read/write performance is crucial and even idle-priority disk activity can cause interference, consider whether using RAID to stripe sets of disks (e.g. RAID5 for a mix of extra speed and some ability to repair bit errors and survive a disk failure at the tradeoff of reduced capacity and flexibility, or RAID0 for pure speed at the expense of all else) is desirable.
-
The only con/issue that comes to mind is that there's no "one-click" way of splitting an existing pool into two pools let alone into a pool of two pools.
If you stick to using the GUI, you have to remove half your drives (problematic if your pool is over half full!), create a new pool using them, then create the new super pool that adds the other two pools, then transfer all your content from the original pool into the super pool (which, because the pools are separate drives as far as Windows is concerned, involves the slow copy-across-drives-then-delete-original process rather than the fast moving-within-a-drive process). But if you don't mind the wait involved in basically emptying half of the drives into the other half and then filling them back up again, it is definitely the simplest procedure.
Alternatively, if you're comfortable "opening the hood" and messing with hidden folders, you can manually seed the super pool instead - it is much quicker but also more fiddly (and thus a risk of making mistakes).
Note also that nested duplication is multiplicative; if the super pool folder that will show up in your hub pools when setting per-folder duplication is x2 and your super pool is itself x2, your total duplication of files in the super pool will be x4. So I'd suggest setting each hub pool's super pool folder to x1, setting the super pool itself to x2 and only then commencing the transfer of your content from the hub pools to the super pool. I hope that makes sense.
-
Hi rorick, to do that you'd have to set each hub up as its own pool (e.g. X and Y) and create a super pool (e.g. Z) formed from the two hub pools; enabling duplication in Z would have its content duplicate across and pull from the two hubs.
-
Yes, as per your edit set the Override value to your preferred time.
-
Re 1, DrivePool does not scrub duplicates at the content level, only by size and last modified date; it relies on the file system / hardware for content integrity. Some users make use of SnapRaid to do content integrity.
Re 2, DrivePool attempts to fulfill any duplication requirements when evacuating a bad disk from the pool. It appears to override file placement rules to do so (which I feel is a good thing, YMMV). However, your wording prompted me to test the Drive Usage Limiter balancer (I don't use it) and I found that it overrides evacuation by the StableBit Scanner balancer even when the latter is set to a higher priority. @Christopher (Drashna)
Re 3, I'd also like to know *hint hint*
Drivepool does not distribute files across the array in real time?
in General
Posted
Basically with all balancers/placement off, DrivePool prefers to fill whatever disk has the most free space at the time of accepting a file.
"now my 8TB contain 4.45TB data and my other 4TB contain 0.82TB data so drivepool will work correctly when the 8TB reach 4.82 TB if I turn off all the balancers?"
Yes, after that point DrivePool will tend to alternate between them as each gets less free space than the other from receiving new files.
"Or which balancer you recommended to use it for minimal balancing while using snapraid?"
With Snapraid, and considering no other factors? Only the Scanner balancer, if at all. I feel the extra load during a sync from dealing with an evacuated drive is preferable to having to rebuild a drive from parity, but that's just my feeling and your use case and mileage may vary.
"For your Point 2, if i did not get it wrong, i can cut out 2 4TB partitions from the 8TB and add them all to the pool so that i should get what I want right XD?"
If you're just wanting to get all the bar graphs in DrivePool to line up prettily, yep that's a way. Just note you'd then need to use a minimum of two parity levels of protection instead of one in your SnapRaid config (I'm assuming you currently have a single 8TB parity disk or multiple split disks totalling 8TB of single parity somewhere since you have an 8TB data disk) because if your 8TB disk partitioned as 2x4TB partitions failed you'd simultaneously lose two "disks" from SnapRaid's perspective.
If however you were wanting to maximise that free space on the 8TB to keep in reserve for some reason... maybe you could do something like use the SSD Optimizer balancer and time/script it so that your regular SnapRaid syncs happen just after the 8TB disk finishes emptying the new files into all the 4TB disks. And/or put in a feature request for the Drive Space Equalizer to be able to set real-time file placement limits and hope that's feasible.
DrivePool and SnapRaid can work very well together... so long as you are careful to line them up nicely.