[Rebalacing can't move files despite space availlable]

On 3/29/2020 at 12:37 AM, Christopher (Drashna) said:

Sorry for the delay in getting to you!

 

Could you open a ticket at https://stablebit.com/Contact if you haven't already? 

No problem!

Actually, since posting, I've made a few changes and the problem seems to have gone away.  While I'm not sure exactly what solved the problem, I can at least describe the changes in case anyone has a brainwave!

1. I replaced a disk that seemed to be reallocating a lot of bad sectors (6000+ before I pulled it) according to SMART readings.  I cloned the disk offline and added the new disk back into the pool, which was seamless and has never failed me!  I ran a check -a -d <diskname> and only found one small text file that had an issue.  (Phew!  ...it seems the bad sectors were recoverable from ECC)
2. I realised there were a few rules under File Placement that were for folders that no longer existed, so I deleted these rules.
3. There were approx 100GB of read only files in the pool that needn't be read only, so I set them to read/write.  I read another post that suggested Drivepool couldn't balance read-only files - not sure if that's true or not and haven't properly tested it myself.
4. I deleted a ton of empty folders directly from the PoolPart folders that were left over from files being on the "wrong" disks that had since been moved.

After all of that and another balancing run, I have a full green bar and no warnings that files couldn't be moved.  Hurrah!  As far as I can tell, nothing actually got moved in the process.  I checked each and every folder that directly related to a placement rule and everything is in the same place as it was before.  However, the only thing I now notice is that some of the discs no longer report "Other" files that did previously, only "Unduplicated".  That always bugged me as on most disks I have no files outside the pool and so there should be zero (or near enough).  I've previously re-measured after killing all processes that could be locking files (as I also read that can interfere with measuring and file placement) and it didn't change anything, but now all seems well.

So, I'm not sure which of the things I did above have "fixed it", but everything seems back to normal.

Steve.

[Rebalacing can't move files despite space availlable]

Hi

I have a strange issue where Drivepool is unable to move files to meet my file placement rules, despite there being ample free space on all disks.  My configuration is really simple:

• No duplication
• Each directory in my root folder is manually set to be placed on 1 or 2 of my 7 disks.  I want to keep related files together (e.g. videos, music, work projects, photo albums, etc)
• Balancing settings in attachment - no auto balancing (as I use Snapraid and don't want files moving all the time)
• Only using Ordered File Placement balancer that only controls New file placement.  This is just to catch files that for whatever reason are not placed according to directory rules)
• About 40-50 folder placement rules
• All rules allow placement up to 99% full or 12GB free.

All discs have at least 950GB free but every rebalance results in 195GB could not be moved because a suitable destination could not be found.  Not annoying in itself and the bar at the bottom of the main window is green.  However, every night (more or less) I get the icons to show all disks need rebalancing.  Running a rebalance achieves nothing.

I don't see how of my rules can be conflicting, so I'm curious as to what might be causing this.  It feels like a bug.

Thanks,
Steve.

[File watch / timestamps unreliable]

Anyone know how I can raise a bug report somewhere that the dev(s) will notice?  This is infuriating and I'm close to ditching DrivePool completely.  Unrelated issue, I'm also having a problem where new files copied to the pool drive on my server are not visible on other PC connected to the network share until reboot or the network card is reset.  The latter is rare (usually when server is online for 40+ days) but the main issue I reported here is consistent regardless.

[File watch / timestamps unreliable]

On 10/1/2019 at 11:16 PM, Christopher (Drashna) said:

You may want the "CoveFs_SynchronizeDirectoryTimes" Setting.

http://wiki.covecube.com/StableBit_DrivePool_2.x_Advanced_Settings#Editing

Hi there

Thanks for your responses.  I've been away form home for a while, so just picking this up now.  

I can confirm that the setting you mention is already set to "true" so I would anticipate the the behaviour should not be as I'm observing.  I can confirm that the Last Modified timestamp for a folder in my pool is that of the first instance of that folder in my disk pool if you enumerate pool parts in the order that disks appear in the pool.  For example, if I have a folder called "Test" spread across Disks 1, 2, 3 and 6 by file placement rules with the following last modified timestamps:
Disk1:  01/01/2019 05:00
Disk2:  01/01/2019 04:30
Disk3:  01/01/2019 11:00
Disk6:  01/01/2019 06:00
...then the timestamp reported by Drivepool to Windows is that of the first disk in the pool, i.e 01/01/2019 05:00, not the latest timestamp (11:00) as per the last actual update (caused by a file change within the folder that happens to reside on Disk3).  To check this behaviour, I temporarily removed the "Test" folder from Disk1 and the timestamp reported to Windows in the pool drive became 04:30 (for this example).

Creating or modifying files that are placed in the Test folder on disks 2, 3 or 6 does not change the folder timestamp in the pool drive.  If a file is placed on Disk1, it does.

The same behaviour is observed for folders split across disks 3, 4 and 5 - i.e. the timestamp of the folder on disk 3 (the first in the pool to contain that folder) is reported.

I believe this is a bug.  I think the correct behaviour would be for Drivepool to enumerate ALL instances of a given folder across disks and report the LATEST of those timestamps.  This bug breaks any application (including Subsonic as per my use case) that relies on checking folder Last Modified timestamps for its functionality.

I've now tested this on Windows 10 as well as on my original Win2012R2 server where I noticed the problem - same behaviour.

[File watch / timestamps unreliable]

5 hours ago, Umfriend said:

I don't know. What I don't understand is that you say "and b) would not recognise the files as new even when I forced a scan."but also "but that doesn't explain why files are detected on a forced scan" unless you mean that a forced scan does recognise the files, just not as new?

Thanks for pointing out the error.  Of course, I meant "are not" detected on a forced scan.  I have edited the post accordingly.

Quote

Otherwise, I would speculate that subsonic is somehow explicitly pointed to RAID Disk 1, perhaps through the folder-mount?

Nope.  Subsonic is definitely pointing at the pooled drive, not one of the specific disks within it.  Checked, double-checked and triple-checked.

[File watch / timestamps unreliable]

Hi there

I've been using Drivepool for a couple of years now and it's been really stable and has generally great.  However, I have noticed an issue over the past few months in relation to file watches.  Specifically, in relation to Subsonic server that is running on my machine.

The scenario is as follows:  I have 7 data disks in my pool and most of my music collection is placed only on disk 1, as per a file-placement rule (see attached screenshot).  Subsonic (which is a Java application) has code that keeps a watch on the filesystem so that when new music files are added into my music folder, it will automatically detect it, scan the files and add to my music libary.  This was working fine, but I noticed that sometimes the files were never detected, even after I forced a scan.  Long story short - because disk 1 was getting full, some music files were being placed onto disks 2 and 3 when I copied them to my pool drive.  Subsonic a) didn't notice the new files and b) would not recognise the files as new even when I forced a scan.  Now, my assumption was that the software used NTFS file watches to detect new files, but that doesn't explain why files are not detected on a forced scan, which I believe from looking at the code is based on file/folder last modified timestamps.

If I manually move the music files from disks 2 or 3 on to disk 1, or clear up space so that the file placement rules move the files when I re-balance, then Subsonic detects the files.

My hypothesis is that Drivepool isn't properly reporting timestamps or filewatch system events when files are placed on to disks that are not the one(s) chosen in file placement rules.  It's taken months to work this out and I spent ages on the Subsonic forums (and talking to its developer) because I assumed that it was a problem with Subsonic.  I've now conclusively (and repeatably) shown this is a Drivepool issue.

Thanks,
Steve.

PS:  I'm running Windows Server 2012R2 with all the latest updates.  I have no AV installed at the moment (I got rid of it while testing this, just in case).  I also use Snapraid for parity across my Drivepool disks, not that I expect that makes any difference!

[Adding an External drive then shucking it?]

It depends on the controller used in the external drive housing.  I've found several times that drives formatted while in the enclosure do not show up as expected in Windows once shucked and connected directly to SATA.  

My advice is to shuck the drive first, reformat directly in your OS and then use.  This would be best whether you're intending to use Drivepool or not.