File Placement Rules not being respected as pool is filling up

[servonix]

So I've been using file placement rules for years to control which folders or file types are stored on which drives, and up until recently everything has been working perfectly.

I've noticed that as my pool is getting closer to being full the file placement rules are not being respected. My main use case is ensuring media such as movies and tv shows are stored on hard drives while images and metadata in the same folder are stored on SSD's, as well as keeping other folders on drives that do not hold my media collection.

The only balancing plugin enabled is for stablebit scanner and all drives are healthy. In file placement settings the only option checked is "Balancing Plug-ins respect file placement rules"The order of my file placement rules are setup properly so that the file extension rules for images and nfo files are above the rules that place all other media file types on the selected drives. All other file placement rules for other folders are above the rules for media files and folders with the media drives excluded. 

Duplication is enabled for the entire pool. There are 9 hard drives for the media library each with around 400-450GB free space, 2 500GB SSD's for images and metadata with around 50GB used, 4 hard drives that are excluded from media and file rules with one pair of 2TB drives having 322GB used and a pair of 3TB drives having only 60GB free space, 2 1 TB SSD's for other micselaneous data that are excluded from media folder and file rules with about 250GB used before this issue started.

The SSD's that are excluded from all of the media library rules is where I'm noticing both media files as well as images and metadata being placed. These drives have more availible space than the media drives in the pool, however the media drives have more than enough free space for the files being added. It seems that these SSD's are receiving 1 copy of the media files. No automatic balancing is triggered to correct the file placement.

I've double checked all file placement rules and settings, everything is as it always has been. I do plan on upgrading a couple of my smaller media drives to larger ones in the near future, but I don't anticipate completely running out of space before then. I'm really stumped as to why this setup is suddenly having issues when it's worked so well for so long. My assumptiom was that file placement rules will always be respected assuming the correct settings and assuming there's enough free space for the data being added.

 

[Shane]

Just to confirm, in each of your relevant File Placement rules have you selected the option "Never allow files to be placed on any other disks"?

The default option is "Allow files to be placed on other disks if all of the selected disks are this full" with a slider defaulting to 90%.

[servonix]

Each file placement rule is set to never allow files to be placed on other disks.

Since posting this I have actually been able to correct the issue. I added another file placement rule to the list for .svg images in my media library as I noticed emby fetched one for a newly added movie. When Drivepool started balancing I noticed it placing files onto the SSD's excluded from the media file placement rules. I stopped the balancing, then started again and the issue started correcting itself. All media files on the SSD's were moved to the hard drives, and all images/metadata were moved to the correct SSD's.

I strongly suspect that somehow the scanner balancer combined with old file placement rules from using those SSD's with the optimizer plugin were at fault.

One of my drives had developed a couple of unreadable sectors a few weeks ago and started evacuating files onto the SSD's. At the time the SSD's were included in the file placement rules from previously being used with the ssd optimizer plugin. I had stopped the balancing, removed the culprit drive from the pool choosing to duplicate later and did a full reformat of the drive to overwrite the bad sectors. When the format completed I did a full surface scan and the drive showed as healthy, so I was comfortable re-adding it back into the pool. Drivepool reduplicated the data back onto the drive and a balancing pass was also done to correct the balancing issues from the drive being partially evacuated. The file placement rules were also changed at this time to exclude those SSD's from being used by my media folders and metadata/images. Everything was working as expected until the other day.

For whatever reason only a small portion of newly added files were actually written to the excluded SSD's. It was literally only stuff added within the last couple of days despite a significant amount of data being added since re-adding that drive back into the pool and adjusting file placement rules to exclude the SSD's that were previously used by the optimizer. It's as if some kind of flag set by the scanner balancer or the old file placement rules weren't fully cleared.

Regardless of the cause I'm glad that everything seems to be chugging along like normal now. If I start noticing any more weird behavior I intend on enabling logging to help support identify what's going on under the hood.