Poor balancing performance after adding empty drives to pool

[sophvvvv]

Hi all, I've been using drivepool in conjunction with snapraid for a few years without issue, but have ran into a weird issue when adding two empty drives to my pool.

For context, my current setup is an ITX PC (4x8TB drives) + Mediasonic probox (8+10+10tb) + mediasonic probox (3x14tb). Proboxes are connected via USB 3.0 and have had no issues maxing out their bandwidth when syncing with snapraid or balancing in the past. I consistently check SMART data and have confirmed many times that all these drives are able to hit their rated speeds via file transfers, both in and out of the pool. S2D2 is SMR, but it hasn't come into play during this issue yet and it's speeds aren't that far behind the other drives anyways. All other drives are CMR.

I added 2x14TB drives to the pool, one in each of the proboxes. After re-measuring the pool, it calculated new balance targets and started balancing. It was taking an incredibly long time, only 5% after 24 hours, so I checked drive performance in Task Manager and one of the new drives (S2D1) was only doing 10-15MB/s while the other wasn't filling at all. I also found it odd that it was only balancing files from a single drive instead of multiple, but that could just be normal behaviour that I'm not familiar with. Note: I've disabled any scheduled snapraid tasks in the meantime to not interfere with the balancing.

Things that I've tried:

• increasing priority of the balancing job. No effect on transfer speed. Tried stopping the balance and restarting with that enabled; no change.
• Looking for errant processes that are using disk resources - None I could find. 
• Trying different balancing settings/plugins - same issue between all. 
• killing drivepool service and restarting it. It started filling the other new drive (S3D1) but still at 10MB/s.

Screenshots below showing the drive's performance in task manager and of the pool. 

Any ideas or suggestions are welcome! TIA!

[VapechiK]

hi. follow this link and bookmark it if you haven't already.

https://wiki.covecube.com/StableBit_DrivePool_Knowledge_base

you mentioned that you've been using DrivePool for a few years. so you must have a good idea of what your DP settings are (under the pie chart in the GUI, 'Manage Pool ^  > Balancing...' and the 3 tabs there within). before you do anything, i recommend you take screenshots of all your settings, plug-in settings, EVERYTHING under all 3 tabs that is pertinent. yes, this is a PITA, but once it's done, you're golden right? no need to do it again unless you discover that some of your settings are sub-optimal and need to be altered. if you are using the 'Drive Usage Limiter' plugin, have all the correct checkboxes been checked?

from your screenshot- S1D2, S1D4, S2D2, and S2D3 seem to be reporting erroneous info. i.e. how can a 7.28TB capacity drive have 7.46TB of data on it and still have 260GB left to fill? this craziness may all be due to DP's internal math. i would:  follow Knowledge Base Q2299585B and reset DP's settings. maybe even Q2150495 as well. this is where your screenshots will help. if you implement Q2299585B i would at the same time, in the same mouse movements after the reset is done, IMMEDIATELY click the Manage Pool ^ and under the first tab put a nipple on the titty 'Do not balance automatically' and UNCHECK 'Allow balancing plug-ins to force immediate balancing (regardless of the automatic balancing settings).' click SAVE. THEN: refer to your screenshots and set your settings back up the way you had them. then alter any settings on the first tab. SAVE. 

i hope any of this is helpful. none of what i have said has much if anything to do with the 'speed' issue you brought up initially. left to its own devices DP is just slow and steady. i would get happy with my settings then let DP just do its thing in the background.

 

[Shane]

@sophvvvv The disk transfer rate graph in the performance screenshot for S2D1 looks somewhat like what I'd expect from moving many small files; could you please test copying a folder containing a large amount of small files (e.g. photos are a good choice) from a good drive to the root folder of S2D1 (i.e. to the drive but outside the pool) and observe the performance to see if it is similar?

 

9 hours ago, VapechiK said:

from your screenshot- S1D2, S1D4, S2D2, and S2D3 seem to be reporting erroneous info. i.e. how can a 7.28TB capacity drive have 7.46TB of data on it and still have 260GB left to fill?

The screenshot shows the drive with 7.46TB on it has a capacity of 7.72 TB.

[VapechiK]

1 hour ago, Shane said:

The screenshot shows the drive with 7.46TB on it has a capacity of 7.72 TB.

yes. my point exactly. and S1D4 shows 7.95TB capacity. could these drives actually be partitions on drives larger than 8TB (file explorer will report an 8TB drive as 7.27TB and DP will show the same drive as 7.28TB in the GUI)? it is certainly a possibility. but not likely. which is why i suggested a reset of DP's settings. of course i could be totally wrong here, wouldn't surprise me a bit lol
 

[sophvvvv]

Hi all, thanks for the input and advice so far.

@VapechiK, I took your advice and reset drivepool settings and configured my balance settings to more or less the same as before. Basically, I turn off everything but disk space equalizer (by % used) and prevent drive overfill, and leave the automatic balancing settings alone. I've since started rebalancing again, and it's been plodding along at 50-80MB/s for the last 20 mins. It would start at those speeds before dropping to 10-20MB/s in the past, so fingers crossed speeds stay high.

Regarding the drives reporting weird storage numbers, that's likely due to files that have been allotted disk space but aren't using that space yet, such as torrents and other files mid-download. Below is a pic of one such file. When considering all those files on a particular disk (often a lot, I tend to queue downloads all at once and drivepools puts them all on the same disk), it totals to more than the actual available disk space. Now that I'm expanding my pool it's not a huge deal, was only really a problem when all my drives were getting close to 95%+ full. 

@Shane, I did additional tests on S3D1 and S2D1 including a closer look at what has been moved to the .poolpart folders by drivepool. When transferring many small files, performance was expectedly slow, 5-20MB/s depending on file size, while large files were around 200MB/s. When I took a look at what had been balanced to those drives however, it was all large media files (not a surprise as that's 95% of my pool) so I don't believe that's what's responsible for slow balancing performance. As noted above my performance is a bit better since resetting settings but I will revisit this with an update after a couple of days.

---

On a side note, this balancing behaviour seems a bit sub-optimal to me, and I'm curious if y'all have any input. At present, basically all of my drives need to balance in some way, either by filling up or emptying. However, only one of the empty drives has any balacing activity, and it's only pulling data from one other drive in the pool. It would definitely balance faster if it were filling both of the empty drives at the same time from multiple drives that are full. Is this a logic or file system limitation? TIA!

[sophvvvv]

Update from this morning:

I checked the status of my balance this morning after waking up, and can see that my drives are back to syncing at 10-20MB/s. I dug through and found the files that were even being transferred at that moment (they were shown as .copytemp files), and they were 50GB+ movies, which should definitely be maxing out that 200MB/s transfer speed these drives can do when working with files like that. 

So, back to square one. 😞

[VapechiK]

aaahhh the dreaded 'not knowing torrents were involved' syndrome  i have read conflicting advice here in these forums as to whether torrenting is a good idea with DrivePool.  for simplicity's sake i keep all that separate from DP.  for torrents i finally settled on 2x4TB drives in Raid1 on an Oyen Mobius Pro 2C on a separate USB adapter card from the card where DrivePool lives in a Syba 8 Bay.  i only continuously seed full seasons of some TV shows and some hard to find movies, so i have plenty of space there to work with.   

ok so snapraid has been paused while you work this out- good idea for sure.  i assume you've paused torrenting and downloading as well, if those clients d/l to the pool.  the 'math' involved in keeping a large pool balanced is a lot to say the least, even more so if you have real-time duplication enabled and/or a steady trickle or stream of torrents and downloads are present.  regardless of either, DP now has a huge task to spread data across 25.5 TB of new space, while then having to calculate which files/folders will be moved from the already present disks to meet the DSE plugin's percentage requirement.  so in this case i highly doubt you're gonna see drives running at full speed until the rebalance is complete.  maybe the existing pooled drives haven't been defragged in a while?  if you are torrenting to the pool there is a likelihood of major fragmentation there so... something to consider.

maybe @christopher or @shane will have more pertinent/better explained info.  if so you will want to have posted details of your DP settings about duplication, performance, etc. 

Edited May 4, 2023 by VapechiK
2x 14TB not 3. adjusted my math accordingly.

[Shane]

23 hours ago, sophvvvv said:

On a side note, this balancing behaviour seems a bit sub-optimal to me, and I'm curious if y'all have any input. At present, basically all of my drives need to balance in some way, either by filling up or emptying. However, only one of the empty drives has any balacing activity, and it's only pulling data from one other drive in the pool. It would definitely balance faster if it were filling both of the empty drives at the same time from multiple drives that are full. Is this a logic or file system limitation? TIA!

I think it would be a software limitation? As I understand it, DrivePool's design philosophy is "be as lean and simple as possible to minimise use of CPU/RAM/IO". So it won't try to do fancier things like that because that's less resources available for other programs to use (e.g. it doesn't want to cause lag for other apps that are accessing files from the drives).

23 hours ago, sophvvvv said:

I've since started rebalancing again, and it's been plodding along at 50-80MB/s for the last 20 mins. It would start at those speeds before dropping to 10-20MB/s in the past, so fingers crossed speeds stay high.

17 hours ago, sophvvvv said:

I checked the status of my balance this morning after waking up, and can see that my drives are back to syncing at 10-20MB/s. I dug through and found the files that were even being transferred at that moment (they were shown as .copytemp files), and they were 50GB+ movies, which should definitely be maxing out that 200MB/s transfer speed these drives can do when working with files like that.

I'd have said it was DrivePool trying to avoid its balancing interfering with normal pool access, but the fact that it initially was managing to balance at 50-80MB/s for the same sorts of files and nothing else is changing has me scratching my head. There's a bottleneck somewhere, yeah. Given 2x14TB is going to take about three weeks at 10-20MB/s, I think it might be worth opening a support ticket.

P.S. Isn't balancing normally turned off (except maybe Scanner evacuation and Prevent Drive Overfill) if you're using DrivePool+SnapRAID, to avoid big diff/sync durations?