Equalize by percent doesn't work for me

[ytulpan]

Hi all,

 

I recently installed DP and I'm playing with it right now. Great product.

 

I have around 50 TB of data on 25 drives, I'm now testing various configurations with a pool of 20 TB of fresh disks plus some other disks I've connected for testing. Since I'm using quite a lot of hardlinks I'm running the last beta version (2.1.0.541) with some bugs in this area corrected.

 

At this point I don't test file duplication. I started to load data into the pool by the standard balancer, and at some point I installed the Disk space Equalizer plugin in order to change balance from absolute amount to percentage left.

 

The arrows the the UI changed accordingly; however when continuing to load into the pool, it seems that the change had no effect - the balance was still by absolute space left - and very precise in that also.

 

I've put the balancer plugin at the very top of the list so it would have highest priority. How do I continue from here ?

 

I have another problem - The DrivePool service wont stop. It will stay in the "stopping" state forever (or at least until I reboot after quite a few minutes).

 

I'm running Win Server 2008R2 on a real machine (no VM).

 

Thanks 

 

Edit: After yet another reboot, I have a strange situation. The rebalancer started working, but the pool is still being filled wrong (i.e. files go to the wrong drives according to the percentage policy). So it's like the rebalancer is fighting a bad filling policy.

[Christopher (Drashna)]

Could you verify that the balancer is on the correct setting?

 

Also, when the service is "hanging" like that, could you get a memory dump of the process?

http://wiki.covecube.com/StableBit_DrivePool_Service_Memory_Dump

 

 

And could you upload the log files from the system?

http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

(namely, steps #5-7)

 

 

Also, do you have any other balancers installed on the system?

[ytulpan]

Thank you Chris,

 

Well, the balancer is on the percentage setting, unless I am blind; there are other balancers present, but only those supplied by covecube. I attach screen prints of the balancer page and the drive situation.

 

Regarding the service hang stuff, it will take me some time since if I try to stop the service I will have to reboot; I can't reboot in the next day or so.

 

Regards, Yossi

 

 

[Christopher (Drashna)]

You do have some of the "optional" balancers installed, I see.

 

But the settings do look correct. And you'll notice the dark blue arrows on each drive? that's the "target" for them. That's what DrivePool says that it wants to get the data to. So as I said, that all looks correct.

 

Now. for the "Settings" tab on the balancing settings window there... that may be the cause of the issue there. There are a number of options there, but the simplest would be to post an image of that, so I can take a look at it.

But for the most part, we recommend immediate balancing, allowing plugins to trigger it, and setting the "Ratio" slider to 90% or higher.

[ytulpan]

OK, some news.

 

Regarding the "hang", well, I can stare down the process into submission  . While preparing the trace files for sending, I was just blankly staring at the services applet for some time and bam, the process moved from "stopping to "stopped". So I tried this a few more times and it worked every time !

 

In fact, immediately after reboot, the service stops doubleplusquick. It is only after quite some data moved that it is sluggish. In my case it takes around 7 min. to stop, after the "failed to stop service" popup appears. 

 

So, do you need the traces anymore. At what time exactly do you want me to take the dump ? (memory).

 

Regarding the balancing stuff. I attach the settings screen. Since I have added some more drives to the pool, I'm waiting to see the behavior again, while continuing the fill-up.

 

Ah, one more thing. There is a file named "trace.text" growing like a baobab in my <pool drive>\temp directory. It contains lines like:

 

[0003676725,0x0026e0:0x2d08:0x838b4a60] wmi\provider\vs_trace.cxx(0360): ****************************************************************

[0003676725,0x0026e0:0x2d08:0x838b4a60] wmi\provider\vs_trace.cxx(0362): **  TRACING STARTED - ProcessId: 0x26e0, ContextId: 0x838b4a60

[0003676725,0x0026e0:0x2d08:0x838b4a60] wmi\provider\vs_trace.cxx(0371): **  Current time: 05/20/14 16:59:34

[0003676725,0x0026e0:0x2d08:0x838b4a60] wmi\provider\vs_trace.cxx(0372): **  Command-line: C:\Windows\system32\wbem\wmiprvse.exe

[0003676725,0x0026e0:0x2d08:0x838b4a60] wmi\provider\vs_trace.cxx(0374): ****************************************************************

[0003676725,0x0026e0:0x2d08:0x838b4a60] ENTER {CInstanceProv::Initialize}: 

[0003676740,0x0026e0:0x2d08:0x838b4a60] ENTER {CVssSafeCriticalSection::Init}: 

[0003676740,0x0026e0:0x2d08:0x838b4a60] EXIT  {CVssSafeCriticalSection::Init}: hr: 0x00000000

[0003676740,0x0026e0:0x2d08:0x838b4a60] EXIT  {CInstanceProv::Initialize}: hr: 0x00000000

[0003676740,0x0026e0:0x2d08:0x838b4a60] ENTER {CInstanceProv::DoCreateInstanceEnumAsync}: 

[0003676740,0x0026e0:0x2d08:0x838b4a60] ENTER {CreateClass}: 

 

 

Is it DrivePool that requested this WMI trace ? or I should look for suspects elsewhere ?

 

Respectfully, Yossi

 

 

 

[Christopher (Drashna)]

Well, I'm glad you were able to stare the service into submission!

If that keeps up, then that's great. 

 

However, if the issue becomes an issue again, then get a memory dump of the service:

http://wiki.covecube.com/StableBit_DrivePool_Service_Memory_Dump

 

 

As for the balancing... that would be why. You basically have balancing disabled there... Set it to balance immediately, or at a set time. Then give it a couple of days and it should balance everything properly.

 

As for the trace logging, I'm not exactly sure. Alex is the one that usually handles that (and is the one that wrote the code for it). However, I do believe that yes, we use WMI for the logging. 

[ytulpan]

Hello Drashna,

 

Thank you for your speedy and complete answers.

 

Well, regarding the service hang, today I tried to shut it down and it was in "stopping" state for 45 minutes at which time I rebooted.

 

While waiting, I have taken a dump of the process and uploaded to you. In the comments I mentioned "User:ytulpan"

 

I really hope that rebooting the server with a pending service stop doesn't corrupt data...

 

Regarding the original issue. I have turned on the "immediately balance flag", so, yes, DP starts balancing, but nevertheless fills the pool in a wrong way in my view. The following screen dump shows a drive that has vastly exceeded its percentage quota and is still written to. 

 

 

This looks to me a quite serious problem, since in the case a pool contains drives with a significant disparity in size, a small disk will not be used until a large disk is close to be filled. The percentage option is crucial in many uses, and I wished it worked.

 

 

Again, Kudos for a great product ! 

[Christopher (Drashna)]

Thank you for uploading the dump.  However, it does look like the issue here was that balancing setting. It does need to be set to "immediately" or to "periodically" to work properly, as the disk space equalizer doesn't affect the "real-time placement limits" (those red arrows that you may have seen).

 

As for shutting down while the service is hanging, no it shouldn't cause any issues. 

 

As for the "writing" issue you are seeing, that's expected actually. By default, StableBit DrivePool writes to the disk(s) with the most free space. There are a couple of optional balancers that change that behavior. Specifically Ordered File Placement, and SSD Optimizer. However, these are designed for special use cases. 

 

However, maybe we should have an option to write to most free space, or highest percentage of free space. This sounds like what you are referring to, and want. I've flagged that for Alex (the developer) and he'll look into it.

IAR/feature request here: https://stablebit.com/Admin/IssueAnalysis/4211

 

And I've said this a few times lately, but it's always worth saying again. We rely a lot of user feedback. While we may not always agree or add the feature, requests to help us know where we should focus development. 

And obviously, control is a big deal for us. We allow a large degree of control over how DrivePool works and places data. So I don't think this should be a problem.

[ytulpan]

OK, Thanks again,

 

Yossi

[Christopher (Drashna)]

ytulpan:

 

Alex has updated it to a feature request and should be added in the future. (the option to change real time file placement to use disk percent used instead of free space when selecting the destination for new files)

[ytulpan]

Hi Drasna,

 

I'm happy that it was found worthwhile.

 

Thanks, Yossi