Umfriend

The only thing I can think of is Manage Pool -> Performance -> Network I/O Boost (even though it says it relates to read requests, not write requests).

So there is a Pool that consist of only 1 4TB drive? If so, then yes, you can shut down, power down, remove 4TB drive and connect it again when you are done. In fact, you could install DP on another machine and connect that 4TB drive and DP should recognize it as a Pool.

Maybe another ticket needs to be opened because 28526 does not address the resulting balancing behavior (it explains it but does nothing to solve it), perhaps as a feature request? Or will this not be addressed at all?

Whether it makes sense or not, a higher pool does not inherit duplication settings from a lower Pool. That might make it harder. I even think a higher Pool is rather restricted in what it can read from a lower Pool, I.e. check the duplication status. But we'll see what Covecube says.

I fear this may be hard to address actually. Whenever the Top Pool (consisting of Pool A and Pool starts to rebalance, it would have to take into account: (a) The use-gap between Pools A and Pool B (b) Then, for any file that is a candidate for move (CFM), check what the Use Of Space (UOS) is, i.e., what the duplication status is (If, for instance, you use folder level duplication, that file may have 1 or a gazillion duplicates), (c) Then, for any such CFM, determine what the UOS would be after the move. Again, the relevant duplication may be rather arbitrary. The real issue here may be that when checking the UOS, the Top Pool would actually have to read file parameters somehow. Either it would (1) have to read on a look-through basis, so an x2 duplicated file is returned to the balancer process twice or, (2) for each file, interpret the duplication settings as per Pool A/B, including folder level duplication settings. However, I suspect that the balancing process is only able to read the relevant data for itself w.r.t. duplication settings and through the Pools A and B, meaning that querying a file would only return one result (just like you only get one results when you look at Pool A or B through Explorer). I suspect that this is how they end up at "other" currently: The Top Pool queries Pools A and B but receives only one record/data item/hit per duplicated file. It also receives total space. So the difference between total space and space used [by single instances of the file because that is all the Top Pool receives when querying Pool A or B] by definition, is other. I am sure it can all be done but it does not seem simple to me and it may have an impact on performance as building the list of files to move will require some additional intelligence.

Could you take a look at Disks 6 and 9? I suspect you find a single PoolPart.* folder in the root and that the within, you'll find two PoolPart.* folders (on each disk). Is that correct? If it is, then I get it.

@zeroibis Whatever works for you. To me, it seems like a lot have administrative hassle for a remote, really remote, probability that, assuming you have backups, can be recovered from one way or the other. As I said, my setup never requires a 20TB write operation to recover 10TB of data. It basically resembles your 3rd example although I would have Drives 1 to 4 as Pool 1, drives 5 to 8 as Pool 2 and then have Pool 3 consist of Pools 1 and 2. But I agree that if two drives fail, you need no restore unless the two drives are one while 2-drive Pool. In my case, I would need to restore some data if the two failing drives are split between Pool 1 and Pool 2. I wonder whether a triple duplication with one duplicate through CloudDrive would be helpful but I don;t use Clouddrive and it may be expensive at those sizes. I am wondering however, what do you use for backup?

First of all, DP does not require drive letters. Some here have over 40 drives in a Pool. For individual access, you can map the drives to a folder. On the backup/restore thing. Yes. However, my setup does not require backing up duplicated data. I have two x1 Pools and then one x2 Pool consisting of the two x2 Pools (it is called Hierarchical Pooling). Each x1 Pool has three HDDs. I only need to backup 1 set of three drives. Now with 4 (or 6 in my case) drives, the probability of losing two drives at the same time (or at least loss of the 2nd prior to recovery of the first failure) is really small. I also wonder what the loss of two drives in one of your Pools would do if ever your Pool is larger than 2 drives. I don't know what kind of backup solution you use but I could simply recover the whole shabang to my x2 Pool and tell the recovery not to overwrite existing files in that x2 Pool. DP would have recovered files where one duplicate still exists, the other files will again be present on both sub-Pools.

I don't know the ins- and outs but some of this stands to reason. So you have Pool X (x2, consisting of X1 and X2), Pool Y (x2) and Pool Z (x1) consisting of Pool X and Y. I think the issue is that when DP finds it needs to move 100MB from X to Y, it will select 100MB, measured at x1, of files to move but it must then move both duplicates. DP will not be able to move just one duplicate from X to Y because then the duplication requirement will not be met on either X or Y. Why you get a substantial "Other" I am not sure but it may be because from the perspective of Pool Z, the files are in fact unduplicated. On X1 and X2 both you'll have a PoolPart.* folder. Files written there are part of Pool X. Within these folders, you should have another PoolPart.* folder. Files written here are part of Pool Z. My guess is that when you write a file to Pool Z, it will feed through to, say, the inner PoolPart.* folder on X1. Then DP will try to duplicate this file in Pool X (so write to X2) and I think this duplicate may be considered "Other" from the perspective of Pool Z. Not sure where that one ends up physically (inner or outer PoolPart.* folder) on X2 but it can not be part of Pool Z (with respect to measurement) because if it were, it would be a duplicated file. That would be in violation of the duplication setting of Pool Z. Generally, I think, it is better to have Pool X and Y both at x1 (so no) duplication and then tell Pool Z to have x2 duplication. One big advantage is that if you ever need to recover, you only need either the drives from Pool X or Pool Y. I am pretty sure that would avoid balancing to overshoot and measurement being unclear.

Also, what HDDs are being written to? Just wondering if you got some SMR-drives being written to (although, even then, 3 days seems like a very very long time).

No worries, it was new to me too (and I still don't know that much about it). So a cheap SAS card, like the IBM 1015 (and there are many that are called differently but are the same. I have a Dell Perc H310 SAS controller which, I am pretty sure, is the exact same card.). Anyway, the cheaper SAS-card have two ports indeed. However, those are SAS ports. There are cables that allow you to connect up to four SATA drives to just one SAS port. For example: https://www.amazon.com/Cable-Matters-Internal-SFF-8087-Breakout/dp/B012BPLYJC. If you look at the picture you see one weird connector, that's a SFF-8087, which splits into 4 SATA. Take two of these and you get 8 SATA connectors. A few things to be aware of: You want these SAS cards to have a BIOS for IT or HBA mode, not RAID. It is something you can do youself if you get a 2nd hand card in RAID mode. I found a guy who flashed it for me before shipping. Whatever card you get, check whether it is indeed using a SFF-8087 connector. No worries if it does not, it just means you need another type of breakout cables. In my case, the breakout cables included power delivery. That was a pain because the cables became less flexible than I wanted. The picture above is more to my liking, just data cables. Power to the drives seperately Which of course means you need a way to power 8, 12 or 14 HDDs... There are splitters for this as well. I think you do want to share the load a bit on the various cables that come out of the PSU Finally, these SAS cards are typically meant for servers that have plenty of airflow. The chip can run hot and most likeley only has a heatsink. You can attach a, I think, 40mm fan to it using philips screws that attach between ribs of the heatsink.

Hi Newbie! ;D, DP needs very little, I had it running on an Intel Celeron G530 and could stream 1080p to at least one device. So a cheap build with, say, a Ryzen 3 3200G, 8GB of RAM, a decent 350W PSU and W10 would work like a charm as a file/stream server. The things you'd probably look for are SATA connectors (cheap boards often have only 4). Although you could get a cheap SAS card (IBM 1015 or somesuch, used.) which would provide plenty of expandability. The other thing is the network connection. 1Gb Ethernet, I think, should be "enough for anybody". It is a bit of a bad time as CPUs are in high demand relative to production capacity. Was a time, which will come again, when you could have a satsfactory CPU for like US$60. Edit and PS: Your English is fine. Just use capitals to start and periods to end a sentence and it'll be great.

My storage needs don't really grow so I am just sticking with what I have, anything between 1.7 and 5.5 yo (6 drives). I bought them spread out over time. Mostly Toshiba and HGST as I had a few issues with Seagate and WD years ago. Only 2 HDDs of same type and purchase date so those are in seperate Pools. As long as they don;t fail, I'll run them until they do.

You can have two PoolPart.* folders on one drive at one folder level (e.g. root) but that is a symptom of something gone a bit wrong. Long story. Only one of them would be the real currently active PoolPart.* folder. A PoolPart.* folder can in fact contain yet another PoolPart.* folder in the case where you use Hierarchical Pools. But mostly, no, normally a drive has one PoolPart.* folder in the root. You're most welcome.

It's a bit of a long shot but perhaps on the E and F drives, the PoolPart.* folder is not hidden and on the D and G drives it is? You can set Windows Explorer to show hidden files and folders.

I am just guessing but as this appears to relate to one file, perhaps use the "Automatically resolve [...]" option in the first screen? Alternatively, seek out the individual copies and delete them manually, then recheck and/or rebalance?

Just note that with a 3x10TB + 3x8TB setup, hieracrichal Pools will suffer a deadweight loss. Why? Because you can not create two non-duplicated Pools of the same size. So when you then create a Hierarchical Pool for duplication, it will see one 28TB and one 26TB drive and 2TB will be unusable. You don't happen to have a 2TB HDD lying around somewhere, do you? Edit: Of, should you have, say, a 6TB HDD, you could do 3x10 and 3x8+1x6TB.

I would just consider to split the 2x2 Pools so that each Pool has one drive on each adapter and perhaps split the 1x5 Pool into two and use Hierarchical Pools for duplication and have each underlying Pools' drives on seperate adapters. Why? Should one adapter fail then you can still access all files without having to connect all HDDs to the other adapter. Sure, you will do so anyway but at least you can now read before you do that.

So gtaus did a good elaborate job but there is one thing that is not entirely correct: If you use the SSD Optimzer plugin, where you assign an SSD as a cache / first landing zone, then that will only work for files that are not to be duplicated. If you write a file to a folder that has x2 duplication, DP will write one copy to a HDD right away. If you have little duplicated data then it won't matter much though. If you do have duplication and insist that one SSD suffices then there is a way to trick this using Hierarchical Pools. Two copies would be written to the SSD initially. I consider this inadvisable (and probably a bug). Overall, I have been using DP since early 2013 (or 2012??). It's a small Pool, more for redundancy/uptime then backup (which is dealt otherwise, Duplication != backup). Never an issue and the fact that it writes files in plain NTFS format is really great for transferring Pools to other PCs, recovery etc. Just not the RAID 10 performance.

This is why last time I bought two 8TB drives but different brands Still, that risk is small and what you could do is add one, balance, add another, balance etc.

Good idea but no. And LR might be the app you need a log for. But I am speculating really.

Or an issue with the combination of the two. You might raise a ticket. I have no clue how one would investigate this, probably would need to recreate the issue first.

I find this amazing but it is what it is. I have no other suggestions, sry. It may still be a LR issue that arises when DP is running but this is all way over my head. You could raise a ticket with stablebit.

That last suggestion of Spider99 is what I was trying to accomplish. Instead of having Lightroom scan Z:\, try a scan on M:\ (no need to hide/unhide files). If that also results in Lightroom scanning C:\ and D:\ then it is not DP related (which I can not imagine anyway) because direct access to M:\ does not go through DP software. But, as Spider said, a real seperate USB drive would work better to show that the issue can be recreated and is not caused by DP at all.

I wonder. If you were to assign a drive letter to one of those 8 drives, say T:\ and then let Lightroom scan T:\, would it then also scan C:\ and D:\? If so, then it really can't be DP related IMHO. It would be even better if you can recreate the issue with a whole other disk, perhaps some USB HDD lying around.