
JC_RFC
-
Posts
13 -
Joined
-
Last visited
-
Days Won
1
Reputation Activity
-
JC_RFC got a reaction from roirraWedorehT in Beware of DrivePool corruption / data leakage / file deletion / performance degradation scenarios Windows 10/11
Response from Freefilesync developer.
I read through the Microsoft docs you posted earlier and others, and I agree with the Freefilesync developer. It appears the best way to track all files on a volume on NTFS is to use fileid which is expected to stay persistent. This requires no extra overhead or work as the Filesystem maintains FileID’s automatically.
ObjectID requires extra overhead and is only really intended to track special files like shortcuts for link tracking etc. Any software that is emulating an NTFS system should therefore provide FileID’s and guarantee they stay persistent with a file on that volume. I am seeing the direct performance impact from this and agree with Mitch that there can be other adverse side affects potentially much worse than just performance issues if someone uses software that expects FileID’s to behave as per Microsoft’s documentation.
Finally also note that ObjectID is not supported by the Refs filesystem, whereas FileID is. https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/ns-ntifs-_file_objectid_information ReFS doesn't support object IDs. ReFS uses 128-bit file IDs, so can't cleanly distinguish between file ID versus object ID when processing an open by ID.
-
JC_RFC got a reaction from roirraWedorehT in Beware of DrivePool corruption / data leakage / file deletion / performance degradation scenarios Windows 10/11
Thanks for the reply Chris.
Note: the beta does not fix the FileID change on rename or copy issue.
I have posted your comment on the Freefilesync forums and will see if Object-ID is an option for consideration there.
Meanwhile I'd still think that it would be better if file-id behaved more like regular ntfs volumes and stayed persistent.
From the same document you referenced....
It mentions that with the FAT file system it is not safe to assume file-id will not change over time, but with NTFS it appears the file-id is indeed persistent for the life of the file.
-
JC_RFC got a reaction from roirraWedorehT in Beware of DrivePool corruption / data leakage / file deletion / performance degradation scenarios Windows 10/11
Very well explained Mitch. I just discovered this issue as well while trying to work out why my installation of Freefilesync wasn't behaving as expected.
Drivepool indeed changes fileid every time a file is renamed or moved which is not correct NTFS behaviour.
The result is that if i move say 100GB of data on my drivepool from one folder to another (or rename a large group of files) when I run freefilesync for backup instead of mirroring the file moves or renames, it needs to delete and recopy every moved or renamed file. Over the network this can take hours instead of less than a second so the impact is substantial.
-
JC_RFC got a reaction from Shane in Beware of DrivePool corruption / data leakage / file deletion / performance degradation scenarios Windows 10/11
To be fair to Stablebit I used Drivepool for the past few years and have NEVER lost a single file because of Drivepool. The elaborateness OR simpleness of how you use Drivepool within itself is not really of concern.
What is being warned of here though is if you use any special applications that might expect FileID to behave as per NTFS there will be risks with that.
My example is that I use Freefilesync quite a bit to maintain files between my pool, my htpc and another backup location. When I move files on one drive, freefilesync using fileid recognises the file was moved so syncs a "move" on the remote filesystem as well. This saves potentially hours of copying and then deleting. It does not work on Drivepool because the fileid changes on each reboot. In this case Freefilesync fails "SAFE" in that it does the copy and delete instead, so I only suffer performance issues.
What could happen though is that you use another app that say cleans old files, or moves files around that does not fail safe if a fileid is repeated for a different file etc and in doing so you do suffer file loss. This will only happen if you use some third party application that makes changes to files. It's not the type of thing a word processor or a pc game etc are going to be doing (typically in case someone jumps in with a it could be possible argument).
So generally Drivepool is safe, and for you most likely of nothing to worry about, but if you do use an application now or in the future that is for cleaning up or syncing etc then be very careful in case it uses fileid and causes data loss because of this issue.
For day to day use, in my experience you can continue to use it as is. If you want to add to the group of us that would like this improved, feel free to add your voice to the request as otherwise I don't see any update for this in the foreseeable future.
-
JC_RFC got a reaction from haoma in Beware of DrivePool corruption / data leakage / file deletion / performance degradation scenarios Windows 10/11
To be fair to Stablebit I used Drivepool for the past few years and have NEVER lost a single file because of Drivepool. The elaborateness OR simpleness of how you use Drivepool within itself is not really of concern.
What is being warned of here though is if you use any special applications that might expect FileID to behave as per NTFS there will be risks with that.
My example is that I use Freefilesync quite a bit to maintain files between my pool, my htpc and another backup location. When I move files on one drive, freefilesync using fileid recognises the file was moved so syncs a "move" on the remote filesystem as well. This saves potentially hours of copying and then deleting. It does not work on Drivepool because the fileid changes on each reboot. In this case Freefilesync fails "SAFE" in that it does the copy and delete instead, so I only suffer performance issues.
What could happen though is that you use another app that say cleans old files, or moves files around that does not fail safe if a fileid is repeated for a different file etc and in doing so you do suffer file loss. This will only happen if you use some third party application that makes changes to files. It's not the type of thing a word processor or a pc game etc are going to be doing (typically in case someone jumps in with a it could be possible argument).
So generally Drivepool is safe, and for you most likely of nothing to worry about, but if you do use an application now or in the future that is for cleaning up or syncing etc then be very careful in case it uses fileid and causes data loss because of this issue.
For day to day use, in my experience you can continue to use it as is. If you want to add to the group of us that would like this improved, feel free to add your voice to the request as otherwise I don't see any update for this in the foreseeable future.
-
JC_RFC got a reaction from haoma in Beware of DrivePool corruption / data leakage / file deletion / performance degradation scenarios Windows 10/11
Sorry, was a typo. Meant elaborateness OR simpleness.
Was not judging you to be either.
-
JC_RFC got a reaction from MrPapaya in Beware of DrivePool corruption / data leakage / file deletion / performance degradation scenarios Windows 10/11
Nice find. Makes sense to me.
Hopefully Chris/Alex can make Drivepool compliant with those requirements.
-
JC_RFC got a reaction from MrPapaya in Beware of DrivePool corruption / data leakage / file deletion / performance degradation scenarios Windows 10/11
Response from Freefilesync developer.
I read through the Microsoft docs you posted earlier and others, and I agree with the Freefilesync developer. It appears the best way to track all files on a volume on NTFS is to use fileid which is expected to stay persistent. This requires no extra overhead or work as the Filesystem maintains FileID’s automatically.
ObjectID requires extra overhead and is only really intended to track special files like shortcuts for link tracking etc. Any software that is emulating an NTFS system should therefore provide FileID’s and guarantee they stay persistent with a file on that volume. I am seeing the direct performance impact from this and agree with Mitch that there can be other adverse side affects potentially much worse than just performance issues if someone uses software that expects FileID’s to behave as per Microsoft’s documentation.
Finally also note that ObjectID is not supported by the Refs filesystem, whereas FileID is. https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/ns-ntifs-_file_objectid_information ReFS doesn't support object IDs. ReFS uses 128-bit file IDs, so can't cleanly distinguish between file ID versus object ID when processing an open by ID.
-
JC_RFC got a reaction from MrPapaya in Beware of DrivePool corruption / data leakage / file deletion / performance degradation scenarios Windows 10/11
Thanks for the reply Chris.
Note: the beta does not fix the FileID change on rename or copy issue.
I have posted your comment on the Freefilesync forums and will see if Object-ID is an option for consideration there.
Meanwhile I'd still think that it would be better if file-id behaved more like regular ntfs volumes and stayed persistent.
From the same document you referenced....
It mentions that with the FAT file system it is not safe to assume file-id will not change over time, but with NTFS it appears the file-id is indeed persistent for the life of the file.
-
JC_RFC got a reaction from MrPapaya in Beware of DrivePool corruption / data leakage / file deletion / performance degradation scenarios Windows 10/11
Very well explained Mitch. I just discovered this issue as well while trying to work out why my installation of Freefilesync wasn't behaving as expected.
Drivepool indeed changes fileid every time a file is renamed or moved which is not correct NTFS behaviour.
The result is that if i move say 100GB of data on my drivepool from one folder to another (or rename a large group of files) when I run freefilesync for backup instead of mirroring the file moves or renames, it needs to delete and recopy every moved or renamed file. Over the network this can take hours instead of less than a second so the impact is substantial.