Why is balancing so slow? Is it just me, or like this for everyone?

[gd2246]

I added two new 12 TB drives to my pool yesterday and hit Re-Balance, and it is taking forever. As you can see in the pictures, it is currently moving data from one of my 8 TB drives to one of my new 12 TB drives. It has been doing so all night at about 10 to 20 MBps. At this rate (2% completed in one day) it will take about 50 days.

Also shown in the pictures is me bypassing DrivePool by manually moving some files within the PoolParts using Windows Explorer from a 10 TB drive to the other new 12 TB drive. Stablebit Scanner shows that these speeds are normal, around 200 MBps. They are all on the same HBA, so the problem can only be caused by DrivePool.

I know the obvious solution would be to manually move all the files to the new drives myself like I demonstrated in the pictures, but that would still be several days of me babysitting the process. I paid for DrivePool, and would like to be able to use it so I can just hit the Balance button and walk away.

I tried checking and unchecking the Bypass File System Filters and Network I/O Boost settings but that had no effect, and I am not using file duplication. Is there anything else I can try?

[Umfriend]

In the DP GUI, see the two arrows to the right of the balancing status bar? If you press that, it will increase the I/O priority of DP. May help some. Other that that, ouch! Those are more like SMR-speeds.

[gd2246]

36 minutes ago, Umfriend said:

In the DP GUI, see the two arrows to the right of the balancing status bar? If you press that, it will increase the I/O priority of DP. May help some. Other that that, ouch! Those are more like SMR-speeds.

Thank you for your reply, I didn't even notice those arrows. I hope it will help. I'll try it as soon as my latest manual move is completed.

Since no one replied till now and I couldn't think of anything else, all day I've been using the DrivePool re-balance target arrows just to tell me how much needs to be moved from each drive and doing it manually in Windows Explorer, from two drives to my two new drives simultaneously which should cut my initial estimate of a few days in half. Each time I initiate a move it takes a few hours for those two drives to finish moving the data, but that's far better than what I was getting.

Just to hammer home that it is not my HBA and I don't have any SMR drives, here's another photo of the manual file-moving I've been doing. It's been going like this with no speed problems all day. I've never thought to take a screenshot of StableBit Scanner during a SnapRAID parity check, but it always goes quickly so I'd bet it can maintain these speeds across all drives. It has to be something to do with DrivePool.

Thanks again for the help, I'll post again when I have a chance to try it.

[Umfriend]

BTW, if you move files among pooled-drives, I think it is strongly advised to stop the DrivePool Service (and restart it or reboot when done). Also, I realise it is painfully slow for you (and I have no clue as to what would be the cause of that) but in my experience, DP is really a fire-and-forget program. I wouldn't care if it took a month so to say, it works just fine while it is balancing.

[gd2246]

2 hours ago, Umfriend said:

In the DP GUI, see the two arrows to the right of the balancing status bar? If you press that, it will increase the I/O priority of DP.

OMG, that was it! Thank you so much! I'm kicking myself for not seeing it.

1 hour ago, Umfriend said:

I realise it is painfully slow for you (and I have no clue as to what would be the cause of that)

Not true, you nailed it!

About moving the files without stopping DrivePool, I did notice that after each move the data I moved showed up as "Other", rather than part of the pool. A quick Re-measure got it to see it as part of the pool again though. Not sure if that's what you meant or if it could have been something more serious, but thanks for the heads up.

And yeah I know Drivepool will work fine while it balances, but it's my SnapRAID parity files I'm worried about. When I do these re-balances, it renders the parity files invalid. Some say that having more than one parity drive increases your chances to recover if the files are on different disks, but I don't want to test that. Plus I would add a lot of data in almost 2 months, and with it constantly moving from the balancing, I wouldn't even be able to do another scan. Well I could if I paused it but the parity info would be useless once I started it back up again.

Normally I don't use the balance feature for that reason, because it creates a window where I have no backups. I only do it when I add new drives. Actually not even always then, I only did it this time because I waited too long to buy new ones and the rest of my drives were over 90% full, which from what I understand isn't good for them.

Anyway you really helped me out here, thanks again!

[Umfriend]

5 minutes ago, gd2246 said:

Not true, you nailed it!

A bit by accident as I did not really expect that simple thing to make such a difference. Anyway, you are welcome.

What you say about the "Other", yeah, that is part of it. Donnow whether it can lead to more serious consequences but stopping the DP Service when you want to do your own file tranfers is best practice.

Now to nitpick, SnapRAID is not a backup. Sure, it may save you from HDD failure to some extent but not against accidental deletions, theft, fire, simultaneous multiple HDD failure (and you got a few). If you value your data, you might consider a real solution. You wouldn't want to lose your personal copy of the entire interweb (91TB, uhm, wow!)

[gd2246]

Hey you got it right so you knew more than I did, lol.

And yeah I know it's not really a backup but it's a nice balance between buying double the amount of drives I have and nothing at all. I've got two parity drives which would mean I'd have to lose 3 at the same time to experience any data loss. Except of course as you said fire, theft, etc.

And also not to nitpick but it does protect against accidental deletions, as long as you realize what you did right away you can get those files back.

But I know you're right. It's just that there's a reason my drives got so full this time, I was waiting for a sale! This gets really expensive & there's no way I could afford a true backup. Besides, what I'm hoarding can all be replaced, it would just take a year or two, lol. But all my irreplaceable stuff like family photos & videos have both offline & offsite backups. 

Thanks for your concern though, and again of course for your help!

[Cam]

I'm having the same symptoms exactly (20-25MB/s) balancing while file transfer are over 200MB/s

Unfortunately for me, even after clicking the arrow over 100x, it is still slow