NTFS folder compression occasionally confused

[DavidUK]

I use NTFS compression and with files this works without problems so long as there is space on the disk to uncompress the file. Drivepool also seems to be unconcerned when a file is compressed on one drive and uncompressed on a second drive.

However I have found a Windows API query on folder compression can give conflicting results when comparing the Compressed Attribute with the Compressed state. These should both be the same, but on folders on pooled drives this is not always the case. It is not a critical problem, but one which can impact changing the compressed status of folders. Unlike files, which are duplicated on two disks (or more if required), the folders, can exists on all disks. It would seem that updating the compressed status of a folder, is not being populated across all the drives on which the folder exists. My guess, and if this is correct, it could explain why some files are compressed on some drives, whilst the copy is uncompressed on another drive. I can visualize this could happen when files are moved  between drives as part of the balance process, should the parent folder have a different compression state. 

I would presume this would also apply to encrypted files.

Does should the Check Duplication Consistency also check if the folders have consistent attributes across all the copies of the folders?

Or I can script a utility to periodically scan the drives and use a majority vote to decide if a folder is compressed and set all the copies of the folders to have the same setting.

Is there any document which describes how the copies of the folders are created and managed on each drive in the pool?

 

[Christopher (Drashna)]

I'll have to pass this along to Alex (the Developer) to see what he wants to do. 

That said, you can use "compact /U /S /A /I folder" to remove compresson on every file and folder and subfolders, including system and hidden objects. It's a simple, if brute forced method to "correct" everything. 

[Christopher (Drashna)]

Hmm, That sounds like a bug in the behavior, for sure.  I'll flag that, so we can take a look into the issue. 
https://stablebit.com/Admin/IssueAnalysis/28365

Though, what version of DrivePool are you using, and what OS, specifically? 

 

 

As for thow the files are stored, they're just in "PoolPart.xxxx" on each disk. 

[DavidUK]

OS/Ver : Mainly WHS2011 - 1.3.6.7585, but I have also seen the problem with WS2008 - 2.2.3.1019 but this could be because the disks were physically moved from WHS2011 to WS2008

Because Windows NTFS resolves the compression conflict on the fly with any file access, the main evidence of the conflict is with explorer's display of the compression state. When a file is created, its compression state is taken from it's parent folder. If one copy of the parent folder has the compression state, and the second parent folder is un-compressed state, then it is likely the two file copies will have a different compression state, and I assume this is how the compression conflict has been created. If I test the two files with FC, it returns identical because NTFS will uncompress the compressed file before comparing. This compression conflict could happens when the files are created, or when they are moved during balancing, or when the file or folder compression state is changed.

For a while, like years, I have been aware of the problem with the explorer display occasionally not showing the compressed state correctly, but as everything seemed to work OK I moved on... But more recently I had two windows api's returning different compression states for the same file. So I went back to the explorer display,  and investigating further, there would seem to be two types of incorrect compression states.

Firstly, some files had a different compression state colour to their properties. This seems to arise when one copy is compressed and the other is un-compressed, the explorer display has the un-compressed colour. Then a query on the file's property can return a different result. Probably because the query is accessing the compressed copy. I guess it is fairly random as to which copy is selected by the Covecube FS.

The second problem is with the file's parent folders. In a system with a pool of 4 disks and file duplication, there can be 4 folder entries, even if only two file copies. So where I have set a folder property to be compressed state, this does not seem to update all 4 folder entries, and like the files, sometimes the folder colour does not match the folder compressed state. I have not tested to see if the problem is at the point of changing the folder compression property, or when additional folder entries are created during file movements during balancing, or when other files are added to the folder resulting in new folder copies. When a file is created, it's compression state is taken from the parent folder, hence if some folders are different this could result in the file copies having a different compression state.

I have also not investigated if encrypted files and folders behave in the same way, but I assume they do as they use similar File Attribute Constants, and, if this is the case, then it could result in files which the user expects to be encrypted, may have copies which are not encrypted.

Although I have seen the explorer problem for a while, it was not causing a problem, so I did not worry. 

But recently I was scanning the pool for compressed files and I noticed a problem when searching the pool folders where the compression attribute returned by the FindFirstFile/FindNextFile API listing of the members of a folder, was different to the compression attribute returned by the GetFileAttributes API for a member. 

I then examined each disk in the pool to find the specific file and it's folder and noticed they can have different compression states. I then manually corrected the state on each pool disk.

I did not record the number of compressed files/folders in error, but my guess is about 0.1%.

This search problem was the first time I had a show stopper caused by this compression conflict. I can see this would be very difficult to fix directly, as the FindFile could have a folder opened on one pool disk, and the GetFileAttribute could open a file on a different pool disk. Hence I would like it if the Check Duplication Consistency process resolved this. If one of the file and/or folder copies is compressed/encrypted, then all copies should be compressed/encrypted. I suspect the problem is caused by inconsistency with the Folder copies, rather than with the file duplication process.