[Beware of DrivePool corruption / data leakage / file deletion / performance degradation scenarios Windows 10/11]

Sorry, was a typo. Meant elaborateness OR simpleness.

Was not judging you to be either.

[Beware of DrivePool corruption / data leakage / file deletion / performance degradation scenarios Windows 10/11]

To be fair to Stablebit I used Drivepool for the past few years and have NEVER lost a single file because of Drivepool. The elaborateness OR simpleness of how you use Drivepool within itself is not really of concern.

What is being warned of here though is if you use any special applications that might expect FileID to behave as per NTFS there will be risks with that.

My example is that I use Freefilesync quite a bit to maintain files between my pool, my htpc and another backup location. When I move files on one drive, freefilesync using fileid recognises the file was moved so syncs a "move" on the remote filesystem as well. This saves potentially hours of copying and then deleting. It does not work on Drivepool because the fileid changes on each reboot. In this case Freefilesync fails "SAFE" in that it does the copy and delete instead, so I only suffer performance issues.

What could happen though is that you use another app that say cleans old files, or moves files around that does not fail safe if a fileid is repeated for a different file etc and in doing so you do suffer file loss. This will only happen if you use some third party application that makes changes to files. It's not the type of thing a word processor or a pc game etc are going to be doing (typically in case someone jumps in with a it could be possible argument).

So generally Drivepool is safe, and for you most likely of nothing to worry about, but if you do use an application now or in the future that is for cleaning up or syncing etc then be very careful in case it uses fileid and causes data loss because of this issue.

For day to day use, in my experience you can continue to use it as is. If you want to add to the group of us that would like this improved, feel free to add your voice to the request as otherwise I don't see any update for this in the foreseeable future.

[Beware of DrivePool corruption / data leakage / file deletion / performance degradation scenarios Windows 10/11]

I agree with you Mitch, and likely the solution for me will be to dissolve my pool and go back to multiple drives. I recently reduced down to 2 large drives anyway (+1 parity for snapraid) so it's no longer worth the hassle of maintaining a drivepool for me. If this issue was sorted I would leave it setup, but the negatives are outweighing the positives for me now.

For the record i pursued Stablebit for some time last year and I attach their outcome here:

https://stablebit.com/Admin/IssueAnalysis/28847

Which in summary was, no problem here, nothing wrong with what we are doing, we are following recommended practice.

I don't agree with their assessment, or Stablebit's interpretation of Microsoft's SHOULD, which in my mind only says Should because it depends on the filesystem, as per the table below where some filesystems don't have unique and stable file_id, but NTFS certainly does. In any case, a good developer SHOULD aspire to do better and if you are trying to emulate something that Should do something, then if it were me, I would do it, not treat it as an excuse to not do it.

64 bit file ID	Generate	Stable	Unique
FAT	Yes	No	No
EXFAT	Yes	No	No
FAT32	Yes	No	No
Cdfs	No	n/a	n/a
UDFS	Yes	Yes	Yes
NTFS	Yes	Yes	Yes
ReFS	Yes	Yes	Yes

Anyway, as you said, to all users of Drivepool who deploy some application that utilizes file_id, then be very careful. It will not behave the same as a NTFS volume.

[Beware of DrivePool corruption / data leakage / file deletion / performance degradation scenarios Windows 10/11]

You are right, I don't know where I got this idea that Drivepool had object-id's for all files from.

I just checked and my files do not have an object_id. So yes, lots of work from here to have unique file_id's, agreed.

[Beware of DrivePool corruption / data leakage / file deletion / performance degradation scenarios Windows 10/11]

PS: Here are some examples from Microsoft for writing drivers that track files by guess what? Yep FileID.

A file modified tracking minifilter driver:

https://github.com/microsoft/Windows-driver-samples/tree/main/filesys/miniFilter/change

An avscan minifilter:

https://github.com/microsoft/Windows-driver-samples/tree/main/filesys/miniFilter/avscan

[Beware of DrivePool corruption / data leakage / file deletion / performance degradation scenarios Windows 10/11]

Fair enough, not the last 64bits.

If there was no Object-ID then I agree with the overhead point. However the Drivepool developer has ALREADY gone to all the trouble and overhead of generating unique 128bit Object-ID's for every file on the pool.

This is why I feel it should be trivial to now also populate the File-ID with a unique 64bit value derived from this Object-ID. All the hard work has already been done.

No argument with the beta/alpha test. I would happily test this way as well. At present though we have a broken File-ID system.

 

[Beware of DrivePool corruption / data leakage / file deletion / performance degradation scenarios Windows 10/11]

Unfortunately it has been over a year and no change.
I feel it is a case of the developer thinks they are right and the rest of the world is wrong.

To any users out there of Drivepool, the warning is clear, be VERY careful how you interact with the data on a Drivepool volume. Any software you use that interacts with fileid on a Drivepool volume can give unintended consequences, from minor performance loss through to serious data loss.

I just don’t understand why they can’t map the last 64bit of object-id to file-id. This seems like a very simple fix to me? If the 128bit object-id is unique for every file then the last 64bit are as well. Just means a limit of unique 18446744073709551615 files for the volume as per ntfs.

[Beware of DrivePool corruption / data leakage / file deletion / performance degradation scenarios Windows 10/11]

I would add that if Stablebit have gone to the effort of providing a unique and stable 128bit object-ID for every file in the pool, why not use the same code base to also provide a stable 64bit FileID?

[Beware of DrivePool corruption / data leakage / file deletion / performance degradation scenarios Windows 10/11]

Nice find. Makes sense to me.

Hopefully Chris/Alex can make Drivepool compliant with those requirements.

[Beware of DrivePool corruption / data leakage / file deletion / performance degradation scenarios Windows 10/11]

 

Quote

 

Zenju12 Dec 2023, 14:33

Let's see...
File object IDs are supported only on NTFS volumes

But for NTFS we have:
In the NTFS file system, a file keeps the same file ID until it is deleted

Ergo:
Object ID = useless
File ID = perfect as persistent file identifier (on NTFS).

 

Response from Freefilesync developer. 
 

I read through the Microsoft docs you posted earlier and others, and I agree with the Freefilesync developer. 

It appears the best way to track all files on a volume on NTFS is to use fileid which is expected to stay persistent. This requires no extra overhead or work as the Filesystem maintains FileID’s automatically. 
ObjectID requires extra overhead and is only really intended to track special files like shortcuts for link tracking etc.

 

Any software that is emulating an NTFS system should therefore provide FileID’s and guarantee they stay persistent with a file on that volume.

 

I am seeing the direct performance impact from this and agree with Mitch that there can be other adverse side affects potentially much worse than just performance issues if someone uses software that expects FileID’s to behave as per Microsoft’s documentation. 
 

Finally also note that ObjectID is not supported by the Refs filesystem, whereas FileID is.

https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/ns-ntifs-_file_objectid_information

ReFS doesn't support object IDs. ReFS uses 128-bit file IDs, so can't cleanly distinguish between file ID versus object ID when processing an open by ID. 

 

[Beware of DrivePool corruption / data leakage / file deletion / performance degradation scenarios Windows 10/11]

Thanks for the reply Chris.

Note: the beta does not fix the FileID change on rename or copy issue.

I have posted your comment on the Freefilesync forums and will see if Object-ID is an option for consideration there.

Meanwhile I'd still think that it would be better if file-id behaved more like regular ntfs volumes and stayed persistent.

From the same document you referenced....

Quote

In the NTFS file system, a file keeps the same file ID until it is deleted. You can replace one file with another file without changing the file ID by using the ReplaceFile function. However, the file ID of the replacement file, not the replaced file, is retained as the file ID of the resulting file.

It mentions that with the FAT file system it is not safe to assume file-id will not change over time, but with NTFS it appears the file-id is indeed persistent for the life of the file.

[Beware of DrivePool corruption / data leakage / file deletion / performance degradation scenarios Windows 10/11]

Quote

It depends what the application uses the FileID for but it may assume the FileID should stay the same when a file moves, as it doesn't with DrivePool this might mean it reads or backs up, or syncs the entire file again if it is moved (perf issue).

Very well explained Mitch. I just discovered this issue as well while trying to work out why my installation of Freefilesync wasn't behaving as expected.

Drivepool indeed changes fileid every time a file is renamed or moved which is not correct NTFS behaviour.

The result is that if i move say 100GB of data on my drivepool from one folder to another (or rename a large group of files) when I run freefilesync for backup instead of mirroring the file moves or renames, it needs to delete and recopy every moved or renamed file. Over the network this can take hours instead of less than a second so the impact is substantial.

[When do file ids/inodes change for files in a pool?]

I am also seeing this behaviour and it affects performance greatly of Freefilesync. This software uses fileid to track moved files and keep folders in sync.

When I move some files on the drivepool (or rename them) the fileid changes on the Drivepool. This does not happen for any of my normal or networked drives which maintain the fileid for moves, renames etc.

The result is that if i move say 100GB of data on my drivepool from one folder to another (or rename a large group of files) when I run freefilesync for backup instead of mirroring the file moves or renames it needs to delete and recopy every moved or renamed file. Over the network this can take hours instead of less than a second so the impact is substantial.