Folder Duplication Changed on it's own last night

[Beaker1024]

Ok so I've had folder duplication on for a while now with most being 2x some 3x and one share folder (Recorded TV) as 1x.

 

Everything was fine until this morning I happened to notice Drivepool (on WHS2011 but using version 2.1.1.561) was doing "Duplicating" way past it's normal time-frame.

 

Checked and all the "Server Shares" folders went from my individually set dupilcation amounts to being 3x at the root and rest being inherited.

 

So that means it was trying to do 3x on my 1x Recorded TV share!!!  As you can imagine that is not good.

 

I don't have room for that.

 

What could have caused it to change on it's own?  I don't have much installed at all on my WHS2011 (do not even have remote web access turned on ever).   I have the "Order File placement Plugin" installed and the Drive Scanner installed.   They all seem to be the current version.

 

This is my first reall bug/issue with your software (which I was signing praises of to my wife last night!  Go figure.).

 

Anything I can do to make sure this doesn't happen again?

 

I had not gone into WHS2011 Dashboard or RDP into WHS2011 to load DrivePool software to where the settings could have been manually changed (via GUI) in the last few days!

 

Only thing that happened different last night than any other night is my wife and I were working (opening, editing, creating, saving) files on the same share and same folders at the same time last night working things.  We normally are not having this occur. (see 2nd post for details on last night's usage)

[Beaker1024]

Actually last night when I was moving some files from a folder to a subfolder I had one thing look strange in Win7 File explorer.  First time I had it look like the files didn't move so I tried a second time and Win7 explorer (client working on WHS2011 share) said the files are no longer there (gave path).  Had to moving out and into folder again to see that they were moved.   Normally file explorer looks/works correctly.  

 

Could I have over taxed the realtime duplicating of DrivePool with too many moves, edits at once (2 PCs same share)? Causing DrivePool duplication to go nuts and change duplication quanity to all be the same as that one folder (was is 3x) I was working on? (Only for the root folder of "ServerShares" not for the "recycle bin" or "volumeInfo" folders in DrivePool.)

[Christopher (Drashna)]

If you changed the "Pool File Duplication", (the root of the pool), then this setting is inherited to all folders unless explicitly changed.

 

That means if you had the Recorded TV at the default setting (no duplication), then setting the pool file duplciation to x3, would (in theory) set the Recorded TV to x3 as well. 

You would need to manually change it to x1 (turn off duplication) to avoid this.

 

I suspect that this is the case, and what you are seeing here.

 

 

 

Otherwise, the duplication information is stored on the disks themselves as hidden NTFFS metadata. If a disk error occurred and damaged this data (eg, file system damage), then it could have reset the status on one or more of the folders. Though, this is very unlikely. 

 

 

As for "over taxing", no. The StableBit DrivePool service is multi-threaded, which means it can handle doing many things at once.

 

However, just in case, could you grab the logs, just in case:

http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

Just do steps #6-8 (grabbing the files), and we'll take a look.

You may want to use the Box upload widget, or open a ticket at https://stablebit.com/Contact as the logs may list file names.

[Beaker1024]

Christopher,

 

Thank you for the information!

 

It seems like the root folder duplication got "reset" wiping out the subfolders settings as they inherited the value again some time between late last night and this morning. 

 

I've got the "Services" folder zipped up.  I used the wiki.covecube.com "box" app.

 

Log File sections that seem different than (not found in) the previous days logs:

[FileDuplication] Unable to correct file modification time after hash check on..... \Thumbs.db   Access is denied

A few log lines later:
[FileDuplication] Unable to correct file modification time after hash check on.... different DIR\Thumbs.db   Access is denied

[FileDuplication] Error checking file...... for duplication [completely different share folder]

Then some [Disks]  Got disks modify....

Then some [FsControl] Clear pool mode: PoolModeMissingDisk (key=PoolPartUpdate)

 

From here things get a little stranger.  All of sudden my *.wtv files are getting logged by the "FileMover" starts saying my *.wtv aren't pattern limited and I bet thats when duplication of my unduplicated Recorded TV happened.

 

Ok that was fun looking through the log files.  Interested in hearing what you and Alex make of the log files and what happened.

 

Thanks again!

[Christopher (Drashna)]

Well, from the stuff you posted here...

 

The "Thumbs.db" file ... these get locked by Windows a lot, so we can't modify them when they're locked (well, we could but that could cause data corruption, so we just don't).

 

As for the error checking for duplication, that could indicate an issue, depending. May be related.

The "disk modify" stuff is just disk info being updated/discovered (such as when you reboot).

 

The "file mover" stuff and specifically "pattern limited" indicates that they are not limited to a specific disk by the File Placement Rules, I believe.

 

I'll have a look at the logs in detail later today (or flag them for Alex, the developer) to take a look at.

[Beaker1024]

Christopher,

Thanks for looking into it!  

 

This has really gotten me thinking that I should get enough HDD space to have at least 2x on everything.  At the moment if I do that i'm at 95% full so I leave my biggest share "Recorded TV" as 1x.  But some tiny shares (like size of 2 TV show Hours) as 3x.

 

Can you do tell me what the strange files in the "Store" subfolder of "Service" for the "Program Data\DrivePool" are and what type of information they contain? Some kind of Hashes for the Hard Diskdrives or the information on the PoolParts so the virtual drive can pull everything together?  I'm always curious how things work. 

[Christopher (Drashna)]

The "Store" folder is the "settings" store. It is where all the information is stored. Balancing settings, file placement rules, disk info, Pool info, stored notifications.

 

This is also why we recommend wiping the folders in the case of the service not starting. Occasionally, if the files get corrupted (which can happen), that will fix it.

 

However, Duplication information is stored on the pooled disks DIRECTLY, as hidden NTFS metadata files (as are the Pool IDs).

 

 

 

https://stablebit.com/Admin/IssueAnalysis/14532

[Christopher (Drashna)]

Could you run a chkdsk on the disks in the pool?

 

You could upgrade to 2.X and that should handle the potential issue here better.

 

Also, as a last resort, you can create a new folder, enable duplication on that new folder, copy the contents of the old folder in there and then delete the old folder.

[Beaker1024]

Thanks for the suggestions.

 

I'm a bit concerned about running "chkdsk" on my server HDDs in the pool.  Do I have to dismount them or remove them from the pool first?

 

Anything else to be ready to do "chkdsk" for a HDD/volume that's in the DrivePool???  

 

Also concerned about how long "Chkdsk" can take and if it makes the HDDs be offline from the server?  I've done Chkdsk pleanty on small HDDs on client machines but never on a server with DrivePool (w/2TB volumes instead of 500GB).

 

I assume running chkdsk without "/r /f" switched it'll be faster.  Just looking for a check to see if an issue is there at this time.

 

When this happened I was using the current stable 2.x DrivePool.

 

I'm now on the BETA 602.

 

I have had no issues with duplication settings since this 1st time and have since gone to all 2x (added another 4TB drive) for all folders and a couple small ones as 3x.

 

So far it's solid.   I'll be getting you the log on the dashboard crashing for BETA 602 soon today or early tomorrow.

[Christopher (Drashna)]

Depends on the OS. If it's Windows 8 or Server 2012 (or up), then you can run "chkdsk x: /r /scan". This will run the scan online (it won't unmount the disks, and they'll remain accessible). The caveat, is that it can't always fix issues. You may need to run "chkdsk x: /spotfix" afterwards to take the disk offline and fix it.

If you're not using these OS's, then a normal chkdsk pass will be fine. It WILL take the disk offline (unmounts the volume, and prompts you about it). This will cause the disk to show up as "Missing" in DrivePool, which puts the pool into a Read Only mode. Once the scan is done, the disk comes back online, and DrivePool will see this and "reconnect" the missing disk to the pool.

 

And I'm not sure what you mean by the "ready" comment, but I suspect that I may have covered that already.

 

 

As for chkdsk, /r will take MUCH longer than just /f (and if you have /r, you don't need /f, as it's implied). But the /r is much more thorough and may get issues that just the fix (/f) pass may miss. It's a good idea. Just ...start it before going to bed, and that may help with the timing.

 

 

 

 

And the reason that I'm asking for the disk check here, is that we store the duplication "tags" as hidden NTFS metadata. It is possible that it got corrupted, or was unreadable for some reason. This could change the status in certain circumstances. v2 is much better about this than v1 ... so if you switched versions, that could be the reason you saw this behavior.

[Beaker1024]

Ok Thanks for the detailed info.  Yes you covered the "ready" part of my questions.

 

I did a read only "Chkdsk" on all hard-drives on the WHS2011 computer and kept the output as a Log but they all say no errors found / no issues.

 

Unless it happens again (which I doubt) I'm going to caulk this up as a fluke and be done with this issue.

 

Thanks again!

[Christopher (Drashna)]

Well, I'm glad it hasn't happened again, and it definitely sounds like it was a fluke.

 

hopefully, no more issues.

[Beaker1024]

It happened again last night.  Been about 2 weeks.  This time I've been running Beta *.659 first time was the latest stable beta.

 

Exact same thing happened.  3x applied to all folders.  simply goto folder duplication settings and try to apply 2x put all the correct 1x, 2x, 3x back to all folders. 

 

From what I posted in another thread (didn't think I'd post in this thread again):

 

Thanks for sharing the link to Alex's details.  From his post on repairs:

"If it finds an inconsistency, such as a duplication tag that's missing or incorrect, it will automatically repair all of the tags falling back to the highest duplication level detected in the case of a conflict."

 

From this I am assuming what happened in my case was a Recorded TV share was 1x but because I had a very tiny document share as 3x the TV share was set to 3x on a "repair."   This flooded my drives.  Prior to bumping folder duplication settings up the highest level could a math check on free space vs needed space (for new duplication levels) be calculated and cause the "increase" to hault (with user notification)?

 

I can understand the intent for which this was written this way but could it be possible to put in a request to Alex to modify how this repair is handled?  Can the current behavior be a option in a GUI or config file setting to do the "repair" behavior as default with the alternative option having the user receive an email alert, or something, to check and reset the folder duplication levels manually? 

 

Just seems dangerous to let the folder duplication change on it's own without the user knowing first or an intelligent free space check.

[Beaker1024]

Thinking of changing over from Folder Level to Pool level duplication (2x).  I assume that would eliminate the possibility of this bug causing me this issue.  Any advice on converting over?

[Beaker1024]

Ok set DrivePool duplication to 2x and it didn't change anything for the Folder levels.  I'm thinking it would be good to not have anything as 3x anymore (so it can't "repair" to 3x just 2x) but the "MetaData" folder doesn't let you change it's duplication level anymore. 

 

Other thought would be to go through the effort of making another DrivePool (never had but one pool) and have it be only for RecordedTV at 1x and leave original drivepool for the other shares with what ever duplication I want.

 

Do you think going back to an earlier Beta might help?  I had used *.612 for a long time without issue.

[Beaker1024]

On a hunch I forced StableBit Scanner to do a "File System Check" on all volumes.  Found one volume as "damaged"  Did the "Forced Windows Repair" and it said it was successful.  I'm hoping that this volume being damaged is what caused my folder duplication issues (repaired to all 3x).

[Christopher (Drashna)]

On a hunch I forced StableBit Scanner to do a "File System Check" on all volumes.  Found one volume as "damaged"  Did the "Forced Windows Repair" and it said it was successful.  I'm hoping that this volume being damaged is what caused my folder duplication issues (repaired to all 3x).

Actually, if there was file system damage, it could potentially cause this to happen.

 

When the service starts, it reads the duplication tags (alternate data stream objects on the pooled disks, actually), and loads those up.  If that's not reading properly, it could cause issues. 

 

If this does fix the issue, fantastic! (and let us know).

If it doesn't, then yes, please do try reverting back.  And if this fixes it, would you mind installing higher build versions to help identify about what build this started causing this to happen? 

 

This would help us out greatly, as we've not be able to reproduce this issue.