Jump to content
Covecube Inc.
  • 0
insleys

File watch / timestamps unreliable

Question

Hi there

I've been using Drivepool for a couple of years now and it's been really stable and has generally great.  However, I have noticed an issue over the past few months in relation to file watches.  Specifically, in relation to Subsonic server that is running on my machine.

The scenario is as follows:  I have 7 data disks in my pool and most of my music collection is placed only on disk 1, as per a file-placement rule (see attached screenshot).  Subsonic (which is a Java application) has code that keeps a watch on the filesystem so that when new music files are added into my music folder, it will automatically detect it, scan the files and add to my music libary.  This was working fine, but I noticed that sometimes the files were never detected, even after I forced a scan.  Long story short - because disk 1 was getting full, some music files were being placed onto disks 2 and 3 when I copied them to my pool drive.  Subsonic a) didn't notice the new files and b) would not recognise the files as new even when I forced a scan.  Now, my assumption was that the software used NTFS file watches to detect new files, but that doesn't explain why files are not detected on a forced scan, which I believe from looking at the code is based on file/folder last modified timestamps.

If I manually move the music files from disks 2 or 3 on to disk 1, or clear up space so that the file placement rules move the files when I re-balance, then Subsonic detects the files.

My hypothesis is that Drivepool isn't properly reporting timestamps or filewatch system events when files are placed on to disks that are not the one(s) chosen in file placement rules.  It's taken months to work this out and I spent ages on the Subsonic forums (and talking to its developer) because I assumed that it was a problem with Subsonic.  I've now conclusively (and repeatably) shown this is a Drivepool issue.

Thanks,
Steve.

PS:  I'm running Windows Server 2012R2 with all the latest updates.  I have no AV installed at the moment (I got rid of it while testing this, just in case).  I also use Snapraid for parity across my Drivepool disks, not that I expect that makes any difference!

FilePlacementRules.png

Edited by insleys
Correction of "are" to "are not" in response to comment.

Share this post


Link to post
Share on other sites

9 answers to this question

Recommended Posts

  • 0

I don't know. What I don't understand is that you say "and b) would not recognise the files as new even when I forced a scan."but also "but that doesn't explain why files are detected on a forced scan" unless you mean that a forced scan does recognise the files, just not as new?

Otherwise, I would speculate that subsonic is somehow explicitly pointed to RAID Disk 1, perhaps through the folder-mount?

Share this post


Link to post
Share on other sites
  • 0
5 hours ago, Umfriend said:

I don't know. What I don't understand is that you say "and b) would not recognise the files as new even when I forced a scan."but also "but that doesn't explain why files are detected on a forced scan" unless you mean that a forced scan does recognise the files, just not as new?

Thanks for pointing out the error.  Of course, I meant "are not" detected on a forced scan.  I have edited the post accordingly.

Quote

Otherwise, I would speculate that subsonic is somehow explicitly pointed to RAID Disk 1, perhaps through the folder-mount?

Nope.  Subsonic is definitely pointing at the pooled drive, not one of the specific disks within it.  Checked, double-checked and triple-checked.

Share this post


Link to post
Share on other sites
  • 0

So I took another look. I don;t have subsonic or file placement rules and I use hierarchical Pools but here is my suspicion. I have found, I think, simply by looking with Explorer to the top-level Pool, the two Sub-Pools and the underlying disks that the top level shows as Date modified, the Date modified that is found for an item on the first disk of the level just below.

In your case, I speculate, that anything querying the Pool would get as date modified for the appropriate folder the date noted on Raid Disk 1. But that one gets updated only when actual files are written to that folder on that disk. It might have been a bit better if the result would have been the most recent Date modified of all instances of such folder on disks in the Pool.

Share this post


Link to post
Share on other sites
  • 0

Now for me it is of no concern but I have the default setting so it should be synchronized and I can tell you it is not, i.e., the date modified for a folder as presented in Windows Explorer is _not_ by definition the most recent date/timestamp at which a file within such a folder on any individual HDD has been written. As I indicated, it seems to present the first date/timestamp that it finds.

I think that is what OP is looking for and I submit that DP currently (2.2.2.934) does not work that way.

Share this post


Link to post
Share on other sites
  • 0
On 10/1/2019 at 11:16 PM, Christopher (Drashna) said:

You may want the "CoveFs_SynchronizeDirectoryTimes" Setting.

http://wiki.covecube.com/StableBit_DrivePool_2.x_Advanced_Settings#Editing

Hi there

Thanks for your responses.  I've been away form home for a while, so just picking this up now.  

I can confirm that the setting you mention is already set to "true" so I would anticipate the the behaviour should not be as I'm observing.  I can confirm that the Last Modified timestamp for a folder in my pool is that of the first instance of that folder in my disk pool if you enumerate pool parts in the order that disks appear in the pool.  For example, if I have a folder called "Test" spread across Disks 1, 2, 3 and 6 by file placement rules with the following last modified timestamps:
Disk1:  01/01/2019 05:00
Disk2:  01/01/2019 04:30
Disk3:  01/01/2019 11:00
Disk6:  01/01/2019 06:00
...then the timestamp reported by Drivepool to Windows is that of the first disk in the pool, i.e 01/01/2019 05:00, not the latest timestamp (11:00) as per the last actual update (caused by a file change within the folder that happens to reside on Disk3).  To check this behaviour, I temporarily removed the "Test" folder from Disk1 and the timestamp reported to Windows in the pool drive became 04:30 (for this example).

Creating or modifying files that are placed in the Test folder on disks 2, 3 or 6 does not change the folder timestamp in the pool drive.  If a file is placed on Disk1, it does.

The same behaviour is observed for folders split across disks 3, 4 and 5 - i.e. the timestamp of the folder on disk 3 (the first in the pool to contain that folder) is reported.

I believe this is a bug.  I think the correct behaviour would be for Drivepool to enumerate ALL instances of a given folder across disks and report the LATEST of those timestamps.  This bug breaks any application (including Subsonic as per my use case) that relies on checking folder Last Modified timestamps for its functionality.

I've now tested this on Windows 10 as well as on my original Win2012R2 server where I noticed the problem - same behaviour.

Share this post


Link to post
Share on other sites
  • 0

I'm so glad you looked into this and managed to break it down.

I have been experiencing the same behavior (First with subsonic) and now Airsonic for years. There were similar posts about this issue in the past but it was never resolved.

I resorted to adding the hidden folders on my pooled disks containing my music in order for Airsonic to detected them on a scan, however this meant I had 2-3 copies of everything in my library.

I am looking forward to a fix!

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...