jsta
-
Posts
28 -
Joined
-
Last visited
Posts posted by jsta
-
-
Ah, if you're using File Placement Rules and balancers, it can make things complicated.
You may want to (read: should) take a look at this:
https://stablebit.com/Support/DrivePool/2.X/Manual?Section=Balancing Settings#File Placement SettingsChances are, there is a conflict here.
I only used File Placement rules after the balancer failed to do its job.
Balancer did not remove data from disks after de-selecting duplicated and unduplicated for them, then I updated to latest version which experienced the same issue. So I used file placement rules to get the data evacuated.
I have not previously used file placement rules, and was only used to evacuate the data off the disks since the balancer did not work properly.
-
There was a bug recently that may have caused this. If you're on the release version (2.2.1.934, IIRC), and you're still seeing this issue, then let me know.
Experiencing the issue with 2.2.2.934, which I got directly from the DP download page yesterday.
Upgrading was the first thing I tried
File placement rules worked okay, but had to remember to go back in and select the two new disks for every folder.
-
do you have the 'unless the drive is being emptied' rule turned on, by any chance?
It is on, yes, but it's under File placement settings and a sub-option of "Balancing plug-ins respect file placement rules."
Reads to me like it would only make an impact when using File Placement rules, though, which I don't (normally) use.I did use it yesterday to de-select the 2 drives being replaced though, and it's almost done moving data off.
-
This doesn't happen anymore. We completely overhauled the removal process.
Duplicated files remain read only, but the pool should actually be writable.
.757 * When a drive is being removed from the pool, and there are in use unduplicated files preventing the drive removal, a dialog will show exactly which files are in use and by which process. * [D] Fixed crash when cloning an exceptionally long path due to running out of stack space. * [D] Rewrote how drive removal works: - The pool no longer needs to go into a read-only mode when a drive is being removed from the pool. - Open duplicated files can continue to function normally, even if one of their file parts is on the drive that's being removed. - Open unduplicated files that reside on the drive that's being removed will need to be closed in order for drive removal to complete successfully. An option to automatically force close existing unduplicated files is now available on the drive removal dialog. When not enabled, the service will move everything that's not in use before aborting drive removal. * [D] Added the ability to close handles on the pool, either by file name, pool part, or a combination of those. * [D] Added reference counting and locking to open contexts.So, you should be able to just remove the disk. No extra work.
But that said... you could use the "Disk Usage Limiter" balancer. Uncheck the "duplicated" and "unduplicated" option for the disk in question, and it will move everything out of the drive.
2 things:
First: huh, guess I should have read some release notes huh? Good to know as I was not aware. My fault for not paying attention to notes.Second: I did try to use Disk Usage Limiter. Settings do not get honored, and no data gets evacuated from the disks. That is what I swear I did previously, but is currently not working. Even upgraded to latest official release. Stated in OP
-
BTW, not worth an extra post, but does DrivePool balance the recycle bin!? Is there a way to make it ignore it? There might be 100's of GB in my recycle bin at any one time.
Almost positive Recycle Bin is part of "Other" data, which is stuff not managed by DP. So it shouldn't get balanced.
The hidden/system $RECYCLE.BIN folder is at the root of each disk, so is outside the DrivePool folders. -
There might be another way, but I used file placement rules, and unchecked the disk I wanted to remove from every folder.
Worked great.
Interesting. Looks like it might should at least work better than the part that isn't currently working.
Thanks, I'll give that a shot.
Still would be nice to know why it isn't evacuating with the Drive Usage Limiter.
-
Looking to replace a couple of my drives, and I swear this worked before.
Because the removal process locks the pool to read-only, and can take hours (at least in my case it has taken hours), I was previously using the Disk Usage Limiter balancer, and de-selecting Duplicated and Unduplicated for the drives that are being replaced. That would move the data off the drive(s), then the actual removal process would only take a few minutes.
Currently though, the balancer settings are not being applied.
I was on 2.2.0 something, so went ahead and upgraded to the latest release; 2.2.2.934.
Still, de-selecting Duplicated and Unduplicated for the 2 disks, data is not being moved off those drives even after manually starting another rebalance.The GUI does report "New file placement limit (0.0%)" for both disks.
Though "Duplicated target for rebalancing" shows -11kb and -37kb (not sure what this actually means though, notation is at the end of the percentage bar)Is there a more efficient way to evacuate data off specific disks prior to replacement, so that the pool doesn't get locked in read-only state for hours on end?
-
Looks like to can disregard my file lock issue.
It apparently is, actually, Plex.
3 different PlexScriptHost processes are (randomly) keeping files open.
-
Just found that the Windows cmd prompt has an openfiles command to display open files.
Seems the option to maintain the object list has to be enabled first, and requires a reboot.
Once the Troubleshooter upload submission is complete, I'll reboot for the option to get fully enabled.
Next time the issue occurs, I'll query with the openfiles command to see if it will report the actual process that might be locking the file.
Or apparently resource monitor can display that info too.
-
I RDP to my server all the time; but catching the issue can sometimes be problematic since I have to wake up my desktop then RDP, yadda yadda.
Regardless of that...
File system logging was enabled yesterday evening.
Issue is currently occurring; attempting to delete a file, via Plex, a couple dozen times over the last 10 minutes.
DrivePool was in the process of "Checking" but I'm unsure if that was already running when I first attempted to delete the file.
It is now in the process of duplicating and issue persists.
Duplication has completed, but still unable to delete.
Just to note that.
Also unable to delete directly from DrivePool volume from the server; so it's not a Plex issue.
Plex processes don't run under SYSTEM.
I don't appear to have any processes or services running as SYSTEM, outside of DrivePool, that should impact file access.
At least, nothing that would seemingly hold a file open for >20 minutes (still unable to delete file, and been at least 24 minutes now.)
I can still read/stream the file.
Troubleshooter is in the process of collecting and uploading data with 3044 as the ID.
Service folder has been zipped and has been uploaded to the provided Dropbox link.
Also, just to note, the 4GB "Other" data seemingly disappeared again.
-
If you're seeing this, enable logging, and reproduce this:
http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection
Upload the logs to us and let us know, as that really shouldn't be happening. The logs may help indicate why it is.
What version are you on?
If you're on any version prior to 2.2.0.710, then that may be the problem.
Both of the public releases have known issues with balancing. The public beta is actually worse due to some changes to correct issues, ... which cause more inconsistencies, as well as some drift issues.
Please make sure you're on one of the more recent builds. I linked the 2.2.0.798 build below, but that or any newer build should be fine.
If remeasuring fixes the issue, let me know. If it doesn't, then enable logging, and remeasure.
http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection
These builds (the 700+ builds) have some additional logging when it comes to measuring to help identify issues.
However, if the issue is that the measuring data is "drifting" and getting out of syn, make sure that you don't have any files open on the pool (eg, check for access via "resmon", and see if there are a lot of writes happening, and reads). Try closing the files and see if that helps.
And if you could, run the StableBit Troubleshooter
http://wiki.covecube.com/StableBit_Troubleshooter
If you're seeing this, enable logging, and reproduce this:
http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection
Upload the logs to us and let us know, as that really shouldn't be happening. The logs may help indicate why it is.
I upgraded to 2.2.0.798 when you suggested it with your first response to this thread
I'll do my best to upload the logs while the error is occurring.
Unfortunately, I tend to only notice it when I'm streaming with Plex and tend to have my desktop in sleep.
My server running DP is headless.
It may also be coincidental. I am not 100% positive it is being caused by DP, but I have been unable to determine what's accessing the file.
All I know is that it tells me the file is in use by SYSTEM.
But I'm not seeing the file lock on the network, as Windows does not display the affected file as being open via network connection.
-
...really don't know if we are in the same situation here.
At least the way both problems look alike made me post in this thread...but Christopher thinks that this is not the case.
EDIT: running that elevated command brought down the "other" from +8TB to 2.7TB now...still way too much. See -> http://imgur.com/a/rI0oH
I dunno, your issue does appear to be consistent with the issue I had.
~12.9TB worth of data in your PoolPart folders per WinDirStat.
DP reporting only 10.3TB of data between Duplicated and Unduplicated.
That leaves the 2.6TB Other unaccounted for; and DP reports 2.64TB Other.
So... yeah, it appears to be what I was seeing.
-
...I'll second that.
Without having touched a single file, my Pool changed the measurement dramatically as well, see here: http://community.covecube.com/index.php?/topic/3068-files-not-accessible-via-pool-but-still-there-in-pool-part-folders/&do=findComment&comment=21178
You're also having file access issues?
Even with 2.2.0.798, I still occasionally get access denied when deleting a file; and it doesn't matter if it's over the network or directly from the DP volume.
I know that was supposed to be fixed in a prior version, but I still have that issue.
When it occurs again, I'll have to see if I can still read the file since I haven't tested that.
You might try re-measuring.
Administrative/elevated command prompt:
dpcmd remeasure-pool PoolDriveLetter
-
I stated I was expecting discrepancies with C: and D: because I know there would be data in the SVI and other OS-protected folders.
Again, zero restore points, VSS, etc running. NONE of that. The server is not running any backups of itself.
That was just additional "proof" that data usage reported by Disk properties is all-inclusive, regardless of file permissions.
Regardless, I forced a re-measure of the pool with dpcmd and "Other" is down to 4.01GB.
DP now reports each disk having less than 500MB "Other" data. Including the disk that previously had zero "Other" data.
So, yes, there was something wrong with the information DP was reporting.
Forcing a re-measure with dpcmd seems to have corrected the reported usage.
-
Do/will one of the command utilities provide me any info?
Size vs Size on Disk has very little difference. As in, the difference is only 7MB.
And it is shown that data usage reported in Disk properties will consist of the data that is within files/folders the user account does not have read access to.
Therefore, with data usage in disk properties and data usage of the folder itself being relatively close (not 100GB difference), the "Other" data being reported must be contained within the PoolPart folder.
My DPDisk03:
Disk properties used space: 818GB
PoolPart folder size: 813GB
Other data: 5.43GB
DP reports 813GB duplicated data.
That one matches up with reported info.
DPDisk04:
Disk properties used space: 818GBPoolPart folder size: 808GB
Other data: 6.85GB
DPReports 811GB duplicated data.
Those are pretty close.
WinDirStat provides the same numbers I've provided.
DPDisk06 numbers do not match up.
Reported data for DPDisk06 is not consistent with my other disks, when said data is gathered in the same manner.
-
If I had to wager a guess, the "other" is from VSS Snapshots, and from the directory entries on the drive. Both of these can/will contribute to the "Other" data.
No snapshots. No "Previous Versions" available.
VSS is not being used, not configured, no tasks related to it.
Will Windows even write to that folder, if the disk doesn't have a drive letter assigned? Tried searching, but can't seem to find an answer.
Reason being, that 100GB showed up out of nowhere, when the disk had not been assigned a drive letter in a while.
Unfortuantely SVI always shows as Zero size and unless you take ownership of the folder from "system" you can see how big it is - and its not included in your calcs. As chris said its probably VSS snapshots or similar
Even if that folder is not accessible with a non-system account, any data contained within would be accounted for when viewing data usage of the physical disk.
Data usage reported at the disk level is only 3GB more than data usage within the PoolPart folder.
I verified this by examining the C: (OS) drive.
Used space reported at the disk level: 65,229,778,944 (60.7GB)
Used space reported by all (accessible) folders: 27,613,713,072 (25.7GB)
Difference of: 37,616,065,872
Or about 35GB
Same with my D: (application) drive.
Used space reported at disk level: 244,965,253,120 (228GB)
Used space reported by all (accessible) folders: 147,397,794,179 (137GB)
Difference of: 97,567,458,941
Or about 91GB.
Therefore, on this disk in the pool, if the "other" data was contained within the SVI folder, then reported data usage at the disk level would be much greater than data usage at the folder level.
-
If you look at the screenshot I attached, I do have show protected folders and show hidden files/folders.
Also, you can see that the PoolPart.GUID folder contains all but 3GB (closer to 2.5GB really) of the data that's stored on the disk.
Yet, DrivePool reports 100GB worth of other data.
That would mean either DrivePool is wrong, or there is 97GB worth of data in the pool that it is considering "Other."
-
My understanding of "Other" data, is data not contained within the pool.
My "Other" data had finally dropped off, and became non-existent; the "Other" item in the legend of the GUI even disappeared.
Well, now I have in total >200GB of "Other" data.
I found a drive that's reported as having 100GB of other data.
Yet, when I compare the data usage of the PoolPart folder and drive itself, the difference is less than 3GB.
Which would mean it would contain maybe 3GB worth of "Other" data.
So what would make up the other 97GB?
Drive contains: 2,337,532,772,352 bytes
PoolPart folder contains: 2,334,729,968,183 bytes
A difference of only: 2,802,804,169 bytes
which is not even 3GB.
Been this way shortly after my previous post.
This is running 2.2.0.798
Is there a way to report/display any of this?
Find out what data it is reporting as Other?
-
yeah I figured out the pool part thing when I output more detail. Makes sense.
I also found an errant DP folder on a disk I had apparently removed from the pool and readded.
Folder only contained ~700 meg.
Deleted the folder, and suddenly "Other" data was less than half; from 10.2GB to 4.3GB.
-
So I upgraded... since this includes the check-pool-fileparts I figured I'd give that a run...
If I'm reading this correctly, most of the stuff in this folder is duplicated more than configured?
This folder is only 2x duplication, but output shows, eg:
+ [10x/2x I] Z:\movies\+ [6x/2x I] Z:\movies\The Shack (2017)Top folder is 10. Sub-folders range from 2-8But results shows nothing is inconsistent.Am I understanding correctly:[{0}/{1} IM] {2}{0} - The number of file parts that were found for this file / directory.{1} - The expected duplication count for this file / directory.Means the first number is the actual number of duplicates, the second number is the expected duplication count (that's configured on the folder), and the I means the expected count is inherited from parent folder?The 5'ish minutes it took to rebalance, after upgrading, the amount of "Other" data changed quite a bit.
5 min of balancing:Other (GB): 10.2 -> 10.5 -> 10.2 -> 11.7 -> 10.6 -> 10.2 -> 11.5 -> 10.1 -> 11.4 -> 10.5 -> 10.1 -> 10.4 -> 10.3 -> 10.2Edit: Looks like it's just the folders that are getting duplicated upwards of 10 times? Not the actual files.Due to the files being placed on different drives? eg, movie file gets put on disk 1 and 2, but subtitle file gets placed on disk 3 and 4, therefore the folder will exist on 4 disks? -
Having more than a couple megs of "Other" data has been unusual in my particular setup, for the 8 months I've been using DP.
As stated, I removed the drive letters from all the disks, so they are not directly accessible from Explorer.
I also have not stored any data directly on any of the disks.They are strictly for the drive pool.
It was also strange that the amount of unduplicated and other greatly increased when adding disk 11, but then went away after that disk was removed (as it also never balanced properly to the new disk.)
As for the duplicate folders; your response would fit the bill. These were 2TB WD Green drives that had excessive LCC/head parking, and I had removed them from the pool so that I could change that idle setting via wdidle. So that would explain that.
I was curious if I should update to one of the beta builds. I'll give it a shot.
Thanks.
-
I recently added/removed/replaced some of the drives in my pool with some shucked 8TB WD REDs.
Prior to this, there was only duplicated data with less than 1k of unduplicated data.
Now, though, I have 13.6GB "Other" data.
No data is written directly to a disk. They were formatted, added to the pool, and drive letter removed.
All data is configured with at least 2x, with some files at 3x duplication.
Previously, when I had 11 drives, it was reporting around 2TB "other" data and a bunch of unduplicated. I get re-running "balance" but it did not do anything and continued to report "consistency" issues.
I removed that drive, and it seemed to go back to showing all data was duplicated.
Also recently, I occasionally have issues deleting files via Plex or Windows; I get a permission denied.
Regardless, even with that error, sometimes when (attempting) to delete via Windows, despite the error, if I refresh the Explorer window the folder disappears.
Well, it disappears from Explorer, but it's not actually gone.
But then a couple/few minutes later I am able to delete it; so not an actual file permission issue.
It also seems that it's more frequently "checking" the pool.
I've been running DP for 8 months, and it seems these issues were not persistent prior to my pool getting to >40TB.
I'm running the latest GR, 2.1.1.561
Also, whether or not it's relevant, after I removed the old 2TB WD Green drives, they all had 2 PoolPart.??? folders. Though it doesn't appear that any of my current drives do (spot-checked half of them)
-
Actually looks like I installed the patches Friday, but I digress.
Who knows which individual patch it may have been, since MS moved to the cumulative patch model.
KB3201845 is the one that installed Friday.
Nothing in the notes really relates to the issue though.
Who knows... maybe it was just Windows being Windows and the reboot cleared it up.
-
Thanks, I'll give that a shot and report back.
Edit: yup, that was it. Thanks!
Server 2016 Essentials and "Server Folders" Not Seeing DrivePool
in General
Posted
So it's been more than 2 years but... 2016 Essentials is again unable to "see" the GPT DrivePool disk with certain actions.
Poking around server health, it reported a number of my "serverfolders" as missing, but they do exist and are accessible.
Oddly, not all of them are reported as missing.
If I "recreate the folder" it creates it on the D: drive without giving me a choice.
I can then try to "move" the folder, but it only allows me to select the C: drive as a new location.
If I try to "create new folder" it does allow me to add to the DrivePool disk.
I imagine a Windows issue, but it's bazaar.
Not sure a reboot clears it up, as the server was just rebooted yesterday.