Jump to content

  • Log in with Twitter Log in with Windows Live Log In with Google      Sign In   
  • Create Account

Photo

Duplication inconsistent


  • Please log in to reply
15 replies to this topic

#1 JazzMan

JazzMan

    Advanced Member

  • Members
  • PipPipPip
  • 51 posts

Posted 28 August 2017 - 02:54 AM

Now both my pools are showing "Duplication Inconsistent" message in the UI.

 

I think that was true on both the 2.2.0.651 I was running before I upgraded for this thread;

http://community.cov...replacebalance/

as well as 2.2.0.823 now.  The pool automatically re-measured when I rebooted post-install.

 

The drive sizes look ok now, but both (even the pool I did not work on) show duplication inconsistent.  They showed green/ok before I replaced the first drive under 2.2.0.651.  DI message has not been an issue for me before.

 

I ran chkdsk on all drives in both pools and no errors were found.

 

I ran the troubleshooter program and sent you all the data.



#2 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,203 posts
  • LocationSan Diego, CA, USA

Posted 29 August 2017 - 03:12 AM

If it's giving the warning that I think it is, it should also give an option to re-duplicate, or at least resolve the inconsistency.

 

Run that (if you don't see it, look for the "^" next to the pool condition bar at the bottom of the UI), and see if it helps.

 

If not, then let me know


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#3 JazzMan

JazzMan

    Advanced Member

  • Members
  • PipPipPip
  • 51 posts

Posted 29 August 2017 - 07:20 AM

The only option is to re-check the duplication.  This does not fix the issue.

 

https://drive.google...YWtmWmszX3ZfbzQ



#4 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,203 posts
  • LocationSan Diego, CA, USA

Posted 29 August 2017 - 06:13 PM

Sorry, yeah, the "recheck duplication" option is the one.

 

If that's not fixing the issue, then enable file system logging and re-run the "recheck" option:

http://wiki.covecube..._Log_Collection

 

And if you could, run this:

http://wiki.covecube..._Troubleshooter


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#5 JazzMan

JazzMan

    Advanced Member

  • Members
  • PipPipPip
  • 51 posts

Posted 29 August 2017 - 07:26 PM

I did run the troubleshooter back on 8/27 and I think it uploaded about 125MB of data.  Hopefully you have that.

 

Does the file system logging need to be active for the whole duplication check, or can I start it a short time after the check starts?

 

My issue is this;  I got a BSOD the first time I ticked that option and then started the re-check duplication.

 

So the system restarted, and the duplication check starts before I can go into the UI and tick the option.  Its running now with the option ticked but a short while after the duplication check had started.  Not sure if there is a way to set this in a config file so it will be active at boot time without needing UI interaction.

 

Is there not a way as an enhancement for the program to be a little more explicit about the inconsistencies it is seeing in the pool?

 

I guess I could try going back to the 2.1 version, but I know it has its own bugs.

 

I did look at the pool I have that is just 2 drives in RAID 1 and the total file size and # of files/folders matched.  What other things make a pool inconsistent .. file attributes and security settings?

 

Thanks,

David



#6 JazzMan

JazzMan

    Advanced Member

  • Members
  • PipPipPip
  • 51 posts

Posted 29 August 2017 - 11:18 PM

I uploaded the file system logging from the duplication check.



#7 JazzMan

JazzMan

    Advanced Member

  • Members
  • PipPipPip
  • 51 posts

Posted 01 September 2017 - 03:39 AM

Any thoughts from the data I uploaded?



#8 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,203 posts
  • LocationSan Diego, CA, USA

Posted 01 September 2017 - 09:23 PM

Unfortunately, I'm not really seeing the error or anything related to it here.

 

Just in case, could you try resetting the settings and see if that helps?

http://wiki.covecube...ePool_Q2299585B

 

If not, try rerunning the troubleshooter to get updated/fresh logs (at least for comparison).

And if you could, use "3093" as the contact ID when running it.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#9 JazzMan

JazzMan

    Advanced Member

  • Members
  • PipPipPip
  • 51 posts

Posted 02 September 2017 - 12:40 AM

OK. Can you give me some idea of the things that cause this "Duplication Inconsistent" message?  And if it could be a false positive?  As I mentioned, when I checked before, I saw the same number of files, folders and total file size on the 2-drive Raid 1 pool.

 

Is there any way to tell the program to not measure the pool, or to select when it measures, or something?

 

At the moment every time i reboot to try to diagnose this  it goes through the hours-long process of re-measuring my 100+ TB pool just to report again that the duplication.is inconsistent.

 

Would the "BitLocker_PoolPartUnlockDetect" have any effect on this?  I don't use bitlocker but did muck about with  that recently as a workaround to the disks-not-spinning-down issue;

http://community.cov...ever-spin-down/



#10 JazzMan

JazzMan

    Advanced Member

  • Members
  • PipPipPip
  • 51 posts

Posted 02 September 2017 - 09:26 AM

OK, I have seemed to fix this for now.  Here is what I did.

 

Downgraded to 2.1.

 

Pool D

 

On the pool that is a 2-drive RAID 1 I saw when looking at the Poolpart subdirectories on each drive that there was a subdirectory with a lock overlay icon.  I don't have any fancy permissions on this machine, and I saw from here (https://www.howtogee...r-in-windows-7/) this can be caused by not having the Users group as part of the permissions.  This was in fact missing from the security list, so I added it to both folders (windows recursed the files and subfolders) and then re-measured the pool and it is now green.

 

I did not look at how that folder was showing in the pool drive, just the poolparts, so perhaps I could have found and fixed it from the pool.  I also do not know if there was an actual difference in the security permissions for the folder or subitems between the two poolpart copies, I just know adding Users seems to have fixed it. 

 

Pool G

 

With so many drives, folders, and files it was impossible to try to find something manually like I stumbled across above for Pool D.  I found this software and loaded it to try to help me (http://cjwdev.co.uk/...ports/Info.html) and pointed it to mount points for each of the individual pooled drives.  The only difference I was seeing for permissions was related to the Recycle Bin.  There were both objects named "Recycle Bin" in the root of the drive, and variously "Recycle Bin" and $RECYCLE.BIN in the "root" of some of the poolpart directories.  The program reported these objects as having different permissions than their parent.  Although I'm guessing that in itself is normal for Windows, since it was the only type of permissions difference I was seeing I went through and deleted all the "Recycle Bin" and $RECYCLE.BIN folders/objects from each pooled drive root and poolpart folder they were in.  I then re-measured the pool and it is also now green.  Again I am not sure exactly waht this was showing in the combined pooled drive.  I don't think the pooled drive showed either a "Recycle Bin" or $RECYCLE.BIN in its root, but I'm not 100% sure.

 

I don't know if it was the permissions, or something else about the Recycle Bin such as the contents, or what causes the two different folder names, but for this pool there seemed to be something Drivepool thought was inconsistent about the replication of the recycle bin.



#11 JazzMan

JazzMan

    Advanced Member

  • Members
  • PipPipPip
  • 51 posts

Posted 06 September 2017 - 08:43 AM

I just want to add that after fixing this as above I tried to re-upgrade to various beta versions, as 2.2.0.651 had been working, but that version, along with *.823 and *.830 all don't seem stable now on my system.  Even after deleting the recycle bin on each pool it is created on, I get a BSOD usually the first time I go to re-measure a pool after the post-install measure chugs away and finishes.

 

It is really confusing/frustrating as the only change I made (intentionally at least) was to swap 2 4TB drives to 10TB in one pool.The second pool shouldn't be going inconsistent.  I seem to be the only one reporting this, but there seems to maybe be some problem with the Recycle Bin replication and/or handling of 10TB drives in the 2.2 betas.



#12 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,203 posts
  • LocationSan Diego, CA, USA

Posted 06 September 2017 - 08:59 PM

The permissions issue could/would cause this issue, but only if the SYSTEM account was removed/denied from the files/folder.

 

This is because the service itself handles the duplication issues and resolution (as well as duplication passes ... and anything that isn't real time duplication).  Normally, this account should ALWAYS have full control on everything. But there are cases that this can end up not being the case.

 

So for why, some software does this, Windows will do this in some cases (such as user folders, and folder redirection).

 

 

As for the "$RECYCLE.BIN", this is the actual folder used for the recycle bin. It's hidden by default though. 

If that got corrupted, it could cause issues, but not usually this.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#13 JazzMan

JazzMan

    Advanced Member

  • Members
  • PipPipPip
  • 51 posts

Posted 07 September 2017 - 12:06 AM

Thanks Chris,

 

I did some more looking in NTFS Permissions Reporter (on the native NTFS PoolPart mount points, not the combined pool) and found that there were some (maybe a few hundred) files and folders where it reported something to the effect that it could not translate the SID for the Owner to a user account.  (sorry I don't have the exact error any more).  I used the Windows TakeOwn command to correct that.  I set the owner to the Administrators group as that seems to be the default of how all the other files/folders were owned.

 

This was done while DP 2.1 is the installed version.  I have not tried to re-upgrade to see if 2.2.0.x behaves better now.  .

 

Again though, DP needs to report better on what inconsistencies it sees in the pool, not just that there are inconsistencies. 



#14 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,203 posts
  • LocationSan Diego, CA, USA

Posted 07 September 2017 - 12:49 AM

If you moved the pool from one system to another, and you'd used specific user accounts rather than the built in ones, then this would be normal.   

 

Also, there are some cases where this could happen anyways (yay NTFS permissions).

 

As for the takeown stuff, that does work, but we also have a guide on how to "brute force" fix this issue, actually:

http://wiki.covecube...vePool_Q5510455

 

And yes, "Administrators" is the default owner (especially on a newly formatted drive).

 

The big thing here, is that the "SYSTEM" account does need to have full control.  Not just because of our software, but there is a bunch of other Microsoft processes that run in the system account. And if they don't have access, it can cause issues. (For instance, it breaks Windows Search, as well).

 

 

As for the reporting, yeah. But for the NTFS permissions.... "nightmare" is a nice way of describing them.  If you get them working properly, they're great.... but otherwise .... case in point, they can be a nightmare to deal with otherwise. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#15 JazzMan

JazzMan

    Advanced Member

  • Members
  • PipPipPip
  • 51 posts

Posted 08 September 2017 - 07:29 AM

SYSTEM should have had access to all the folders/files.  I never had problems accessing the files, and i haven't noticed it missing.  "Users" was missing from some folders/files. 

 

I'm less concerned about how the ACLs got out of whack as I understand things happen.  I never moved the whole pool, but it is possible some of those files were moved or copied into the pool from a portable NTFS drive in such a way as the ACLs were also carried over from the source system.  Or some such thing.

 

Not to be a broken record, because you are always very helpful Chris, but if this is the root cause of the 'duplication inconsistent' or BSODs then DP should log something like "SYSTEM cannot access file \\pool\path\file.ext" not just say "Duplication Inconsistent".  Nothing in the thread above, or any other thread I have seen on the forum, or article I have found in the wiki, confirms for me that file permissions are even a thing that can cause the "Duplication Inconsistent" warning/error.  This tack is something I started on my own trying to find a root cause.

 

I might take the pooled disks offline, try the beta upgrade one more time, bring the small pool back online, and see how it goes.  Overall I'm pleased with the DP product (I've had a few licenses for 4+ years), just not so much with this "Duplication Inconsistent" message and behavior.

 

Cheers.



#16 JazzMan

JazzMan

    Advanced Member

  • Members
  • PipPipPip
  • 51 posts

Posted 08 September 2017 - 10:31 PM

OK, it took some effort but I seem now to have both pools' duplication consistent.

 

I still think there is some bug in the beta around the triggering for when to remeasure the pool and/or the actual logic that the duplication is not consistent.

 

Here is what I did after fixing all the owner and ACL issues I could find as described above.

 

- set all the drives of pool G offline
- Did not touch Pool D and it was Green/OK as far as duplication consistency
- stopped the drivepool service
- uninstalled 2.1.1.561 from control panel
- rebooted

- installed 2.2.0.833 Beta
- Received the "unable to connect to service" dialog (although the service is started)
- stopped the DrivePool service
- cleaned out the ProgramData and AppData directories for the program as described somewhere in this forum or Wiki as this seems to be a know/documented issue with 2.2 and/or upgrading
- Restarted the drivepool service
- Opened the DrivePool UI and Activated my Key
- Pool D showed as Duplication inconsistent  [WHY?]

- Re-measured pool D
- Pool D now shows Green/OK [If the pool was inconsistent why did I have to manually re-measure it? and why did it miraculously become consistent with no change on disk?]

- set all the drives of pool G online
- Reopened the drivepool UI.  Drive G showed as "Measuring"
- When done measuring, pool was "duplication inconsistent"
- Re-measured pool G
 - Pool G then showed Green/OK

 

- Rebooted cleanly.  System shut down and restarted without forcing any programs to close or BSODs or whatever

 

- Upon reboot, logged in and opened the DP UI.

- Drive D was OK/Green

- However, Pool G was "Measuring..." again [WHY????]

-  After being Measured for the Umteenth time Pool G shows Green again.

 

- Rebooted

- This time Pool G seemed to stay consistent across the reboot and was OK/Green when I logged in to the DP UI after the reboot.

 

Thanks,

David






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users