Duplication removed??

[Trayal]

I logged into my server today to check on disk status, just a check.  I hadnt noticed anything amiss.  But I noticed when I looked in the pool that it listed 2.2 TB duplicated, but 1.6 TB not duplicated.  Which is weird because all my files are duplicated.  It also said I had more "free space" than what I remember being available the last time I checked. Looking at the scan status it claimed it was duplicating and entire folder that up until today was duplicated.  I checked in Stablebit scanner, and all drives are healthy with no warnings.  Is there any reason duplication would be lost and have to be redone?

[Tim]

Same issue. When I checked the pool status last night, approximately half of my previously duplicated files had been changed to unduplicated.

 

I'm running the pool on Server 2012r2 Essentials.

[Christopher (Drashna)]

Could you grab the logs from the system?

http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

 

 

 

And specifically, the duplication status is stored on the pool, as hidden NTFS objects.  So unless there is a disk error, this shouldn't happen. 

 

 

 

Additionally, what version of StableBit DrivePool are you using, and what OS are you using?

And do you have any antivirus, backup or disk tools installed? (do this to verify: http://wiki.covecube.com/StableBit_DrivePool_Q2159701)

 

 

Also, could you check the "Folder protection" option , to see if the folder is properly inheriting the status?

[Trayal]

Well, I went to the directory, and the logs start on 12/02, it doesnt seem like any logs before that were saved, but I have had drivepool running for almost six months with no issues.  I am uploading those logs, but the issue started before that and I had already forced a measure and a balance.  All the folder duplication settings were still correct, I didn't have to make any changes, and once I initiated a remeasure it began duplicating the files in the folders that had lost duplication.  Additionally I ran updated scans with Stablebit Scanner, and there are no warnings or issues coming up for any of my physical disks. 

 

I am currently using Drivepool 2.1.1.561.  I have no backup services running, I am using Comodo Internet Security pro for AV, however I have it set to request permission before attempting to "fix" anything it flags.  Windows 7 Pro.

[Christopher (Drashna)]

Well, I went to the directory, and the logs start on 12/02, it doesnt seem like any logs before that were saved, but I have had drivepool running for almost six months with no issues.  I am uploading those logs, but the issue started before that and I had already forced a measure and a balance.  All the folder duplication settings were still correct, I didn't have to make any changes, and once I initiated a remeasure it began duplicating the files in the folders that had lost duplication.  Additionally I ran updated scans with Stablebit Scanner, and there are no warnings or issues coming up for any of my physical disks. 

 

I am currently using Drivepool 2.1.1.561.  I have no backup services running, I am using Comodo Internet Security pro for AV, however I have it set to request permission before attempting to "fix" anything it flags.  Windows 7 Pro.

It should save all the logs, as there isn't a limit. The coveFS folder may be new, though (as that's created when you enable file system logging).

 

If you've manually reset the settings, or of this is a recent install of windows, it would only have the most recent logs.

 

 

THough, another thing, could you export/save the event viewer logs? Specifically, the "Application" and "System" lists in the WIndows Logs section. 

[Beaker1024]

I just had a similar issue.  Tonight at 8pm EST I got an email from my WHS2011 that I was completely out of HD space!

 

I have Folder level duplication on for over a year without issue (some 3x (small shares); some 2x; one 1x (Recorded TV, not enought room to duplicate))

 

Well DrivePool lost it's mind and set the entire Share folders to ALL be 3x

 

Not cool.

 

Seems we see a trend that Server OSes might have been given a MS update/patch or something that reset DrivePools dupilcation settings?  Just speculation but from this thread it's Servers and within the last week or so.

 

BTW I have the latest pushed Beta of DrivePool running (2.2.0.651)

 

Also when I went in and saw all the Folders set to 3x I did a quick change duplication at the highest level to be 2x (just to get things started to be back to normal).  But the strangest thing happened.  All my Shares went back to the correct duplication settings.  My 1x recorded TV share, my 2x folders and some 3x folders.  All from just doing one change at the highest level from 3 (which was all) to 2x.

 

Also before this mess I had a clean and tidy duplication/distribution of files/folders on the hard drives.  Now I have about 0.5TB less free space and I even have some "Unusable for duplication" space now!  Which I never had before.  Can I get things back to the nice clean distribution again?

[Christopher (Drashna)]

Beaker:

 

What OS are you using? What version of StableBit DrivePool did you have installed when this happened? Has it happened in the past? 

Do you have any antivirus, backup or disk tools installed? And could you do this: http://wiki.covecube.com/StableBit_DrivePool_Q2159701

And how are the drives connected? Onboard SATA? Add-On Controller Card?  USB? And what hardware specifically?

 

Could you check the "System" section of the Event viewer for disk, controller or ntfs errors? 

 

Specifically, the duplication settings are stored on the pool as alternate data streams. Which are "super hidden" file system objects. These should not be affected unless you're seeing disk errors. 

 

 

 

As for the "unusable for duplication" may be because of how the files are rebalanced.  It may clear up after the software has time to rebalance the data.

[Beaker1024]

DrivePool & Scanner installed on:
WHS2011 fully updated/patched; No antivirus installed except on-demand free version of MalwareBytes (which I had not loaded or ran scan in 2 weeks).

 

DrivePool is: 2.2.0.651    StableBit Scanner: 2.5.3062 [All disks are checked every 30 days and are completely healthy]

 

In event viewer on the WHS2011 computer looking at the "System" section I see the "srv" warning entries for HD drive letters becoming full (~9pm 12/14).  I see no entries for NTFS, disk or controller errors earlier during 12/14 or previous day 12/13. 

 

Looking at the "Applicatoin" section I see an ERROR for VSS: (BUt this seems to have been going on for a long time (many previous days))

VssAdmin: Unable to create a shadow copy: Either the specified volume was not found or it is not a local volume.
Command-line: 'C:\Windows\system32\vssadmin.exe Create Shadow /AutoRetry=15 /For=\\?\Volume{xxxxx-xxx-xxxx-xxxx-xxxxxxxx}\'.

 

[NOTE I replaced the volume ID with "x"]

 

Next Entry is VSS shutting down due to idle timeout.

 

This morning I did move the Duplication Space Optimizer up one and did a rebalance.  Still have Unusable for duplication space that wasn't there before.  It took a long time to do a nice balance this morning but it's done.  I also had it "re-measure" afterwards.

[Christopher (Drashna)]

Okay, thank you for the information and confirmation.

 

We do have one other user that was experiencing the same issue. So it looks like there is a bug somewhere here. 

 

The only common thing I'm seeing is that everyone is on WIndows7 or Server 2008R2 (the base OS for WHS2011).  It may be related to an update or something on the specific OS.  But it could be unrelated.

 

I've already created an issue for it, but you can find it here.

https://stablebit.com/Admin/IssueAnalysis/20851

I've updated it, to let alex know that you guys are also experiencing this, and have bumped up the priority.

[Beaker1024]

Thank you for moving this forward!

 

One thing that I find interesting enough to mention again (as I think it could be a clue as to what went wrong) is that changing the top level folder to 2x again didn't set all subfolders to 2x it actually put _every_ duplication value back to what it was before the error!  Like something just "masked" my settings (some 1x some 2x and 3x) with 3x for all and changing the top folder removed the improper 3x for all setting.

[Christopher (Drashna)]

Yeah, I'm not surprised by that.

 

Specifically, it's like it's having issues reading the information, and when you try to re-write it, it actually refreshes it.

 

If you're able to reproduce this, please do get the logs:

http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

[Christopher (Drashna)]

We believes that he has a fix for this issue.  

 

If you want to read, it was a very, very rare bug that ... well, really shouldn't have happened. 

https://stablebit.com/Admin/IssueAnalysis/20851

 

http://dl.covecube.com/DrivePoolWindows/beta/download/StableBit.DrivePool_2.2.0.672_x64_BETA.exe

[Beaker1024]

Thanks for the new BETA to try to fix this issue. 

 

Just installed and gave it a go!

 

Had to immediately uninstall and say a few prayers while rebooting and going to back *.665 beta.

 

I tired 3 times with the *.672 beta and on each install/startup it did it's duplication check pass through entire pool and always come up as Warning: issues.  and gave a message box that I could expand with what files/locations weren't right.

 

I thought OK it's just duplication so I tired to move a file on that list and windows said the file wasn't at that location anymore!!!  That's what scared me.  All of a sudden I don't have access to my files anymore (without digging in pool part folders, etc...)!!

 

that message box of files / folders was too long for the expanded display and had "..." at bottom but all the ones I could see were in my "Recorded TV" folder.

 

Interesting point about my DrivePool.  The Pool is set to 2x.  BUT in Folder duplication I went into the "Recorded TV" share folder and have one subfolder named "Unduplicated" which is set to 1x.

 

I do this for HDD space considerations.  I have to manually move things to the "unduplicated" TV subfolder but that's besides the point.

 

I"m back to *.665 which I have been running since ~March 3rd or so with issue (that I know of).  Oh and yes I can actually move/access my files again in windows!  Also *.665 did it's duplication startup check and didn't through any errors/warnings at all!

 

I really do appreciate Alex's work on this duplication issue.  I hope my ramblings can be of some use.

[Christopher (Drashna)]

Could you grab the logs from the system?

 

http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

 

Just do steps #7-9 (uploading the logs), and let me know when you've done that. 

 

 

 

And if you could, could you install the newer version, enable file system logging and reproduce the issues? 

It would definitely help out here. 

[Beaker1024]

What would be of more value?  The current logs from after the uninstall and reinstall to be back to 665

OR

Install newest Beta again, enable file system logging / have issues then send you logs?  (BTW how do you enable file system logging?)

 

I'm not going to have a lot of time to work on this (hopefully first half of upcoming week) but will try to do something to help as I really appreciate your work on these issues. 

 

Oh and how about screen snippings of the messages from both DrivePool GUI on server & messages from client Win 7 when attempting to move a file?  I like screen captures.

[Christopher (Drashna)]

Well, either.

 

In this case, the affected files should be in the text logs, so that should be fine.

 

But, ideally, if you could install the newer version, enable the logging, and get it to reproduce the issue and then stop the logging and upload it.

 

As for enabling file system logging, how to enable it is listed here: http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

 

And screen captures are fine, but the logs are usually more helpful.  But having both is nice.

[Beaker1024]

You are right I was able to find the lines in the "DrivePool.Service-2016-04-01.log" text file referring to the issues.

 

BTW What type of info is logged into the ErrorReports folders file that are *.saencryptedreport ?

 

They obviously do not open in Notepad to check out like the Service log.

[Christopher (Drashna)]

 

As for the Error Reports, these are non-critical (handled) errors that occur during normal operation. 

 

Specifically, the reference lines of code, so even if they weren't encrypted, they wouldn't be useful without the source code.  But it is very useful for troubleshooting issues on our end, if they are being dumped to error reports. 

[Beaker1024]

Well then we might be in luck.  The dates for the ErrorReport *.saencryptedlog files seem to correspond exactly with all of my duplication issues (went to 3x on all, went to 1x on all, etc...).

 

I'm going to guess that those error reports are encrypted to keep your proprietary code from being leaked then.  Sorry for all the questions but I feel more comfortable knowing what kind of information I'm uploading before putting it up/out there on the net.

 

Might as well complete the list of files and ask about the files in the "Store" folder that look like pool parts or hdds volumes.

[Christopher (Drashna)]

I'm going to guess that those error reports are encrypted to keep your proprietary code from being leaked then.  

Yes, and no. I believe that it mentions the function being called, but also the line in the source code, version info and other stuff.  Since this essentially isn't useful without the source code, encrypting it isn't a bad idea (also, I think it's designed to be transmitted over the internet, so encrypting it is a good idea in general). 

 

As for the Store folder, that's the settings store. Encrypted, as well, but contains a lot of information, such as disk info, pool info, balancing settings, and so on. 

 

 

 

And if these error reports are being generated when you're seeing the issues, then please do submit them, as they may be helpful. 

[Beaker1024]

Christopher - It's done.  I uploaded the Service Folder with the ErrorReports and the Service Log file (and Store folder intact).

 

MY comments are:

1) The dates for the Error report logs in the "ErrorReports" folder appear to be from the days that I had the DrivePool duplicate change itself for all the folders.  On the older dates I had the entire pool goto 3x duplication which overfilled the drives.  The newer/more recent errors I had it goto 1x on all folders then autocorrect itself back to 2x on most (except one subfolder).

 

2) The Beta 672 was installed 2016-04-01 14:58:33Z (for the Service Log file).

 

I've also put these comments as text files into the Zip that was uploaded and in the comment box for the upload.

 

Please confirm that the uploaded Zip file was received and Ok.

 

Thank you and Alex for working hard to fix these bugs I'm really hoping that the ErrorReport files are helpful!

 

PS - There's a slight chance the same "Service.zip" file was uploaded twice.  Didn't get confirmation of uplaod in FireFox so I did it a 2nd time in IE11 and got confirmation.

[Christopher (Drashna)]

Based on the logs, I think this is essentially "expected" at this point. 

 

The warnings in the log indicate that the pool part wasn't found, and this lines up with the specific bug in question. 

 

However, i've flagged this for Alex, so we can get a better look. 

https://stablebit.com/Admin/IssueAnalysis/25987

 

 

 

And to clarify, the warnings in the UI that you got where about the Poolpart not being detected and/or about duplication issues? 

[Beaker1024]

So for the Beta 672 issue it was when I was using a Win 7 client to access (via LAN) a share folder/files.  The files (RecordedHDTV share) were all coming up as with a message from Win 7 GUI msgbox something like:

"Access denied / File not found at location specified"

 

or something like that.  So sorry I didn't get screen captures.  I just switched back to earlier beta I knew/hoped would work again.  But 672 was not completing the initial duplication check without "Warnings"

 

I had / saw no issues about a "pool part not being detected" in the DrivePool beta 672 GUI it showed all the pool parts (volumes, etc..) as being part of the pool.

 

It was all "Duplication" issues.

 

On a separate note I am hoping that the ErrorReporting folder/SAErrorReportLog files that seem to correspond with the duplication changes (from November, Jan, Feb dates) to the folders all 3x or all 1x, etc...

Crossing my fingers those error logs give insight into those.  Have you flagged that for Alex also? 

[Christopher (Drashna)]

Ah, okay. 

 

Yeah, the files are flagged. The link above includes the files and a description of the issue. But I'll relink it here:

https://stablebit.com/Admin/IssueAnalysis/25987

 

If you have time (are willing), could you install the newer version again?  
And could you run "dpcmd reset-duplication-cache", and see if it helps?

And enable file system logging and reproduce the issues?

http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

[Umfriend]

Wednesday, logging in and opening explorer I found I had way to much space free and on opening DPUI indeed balancing had dropped and was rebuilding.

 

One consequence of this that my server backups are getting screwed (but, AFAICS, no risk of dataloss).

 

Can I upload anything for you?

 

WHS 2011 - DP 2.2.0.659 BETA.

[Christopher (Drashna)]

Wednesday, logging in and opening explorer I found I had way to much space free and on opening DPUI indeed balancing had dropped and was rebuilding.

 

One consequence of this that my server backups are getting screwed (but, AFAICS, no risk of dataloss).

 

Can I upload anything for you?

 

WHS 2011 - DP 2.2.0.659 BETA.

 

Upgrade to the 672 or 673 build, as it should catch the issue.