Jump to content
Covecube Inc.
  • 0
DragonQ

Migrating from FlexRAID pooling to FlexRAID + DrivePool

Question

I currently use FlexRAID RAID-F for snapshot RAID, which I'm pretty happy with. I also use the FlexRAID drive pooling feature to merge 8 HDDs, which I'm not particularly happy with since it doesn't work as I'd like it to (and is also, IMO, simply broken). I'd like to try migrating my current setup to use DrivePool in conjunction with snapshot RAID. My first question is: are there any known issues with such a setup? For example, does DrivePool do anything that would invalidate daily RAID snapshots?

 

Secondly, I'd like confirmation that DrivePool can actually do what I want. Here's an example of the root directories on my drives:

  • Drive 1: Data, Audio, Backups
  • Drive 2: Video\TV
  • Drive 3: Video\TV
  • Drive 4: Video\Films
  • Drive 5: Video\Films

In this scenario, if I copied a file into the \Video\Films folder, I want it to go to either Drive 4 or Drive 5 (not really bothered which, but I suppose it'd make sense to fill one up before using the other). The only scenario that should lead to drives 1-3 having files from the \Video\Films folder on them is if Drives 4 & 5 are full. In addition, I'd like only drives 4 & 5 to spin up (ideally at the same time) when accessing the \Video\Films folder via a network share. At the moment, going to such a share takes ages since FlexRAID shoves files on random drives without being asked to and then spins all of them up one a time. Sometimes traversing subdirectories incurs additional delays as (presumably) even more drives are spun up.

 

Finally, what kind of transfer speeds should I expect when copying files into the pool? With FlexRAID pooling, I get ~35 MB/s when writing to the pool, whereas writing to a drive directly yields 150+ MB/s. How about over a gigabit network (where I'll be doing most of my transfers)?

 

Thanks for any advice!

Share this post


Link to post
Share on other sites

20 answers to this question

Recommended Posts

  • 0

Well, the balancing engine may cause issues for the parity snapshots. 

 

In fact, we have a couple of user threads that discuss this and other configuration changes to ensure that the two products will work together as expected:

http://community.covecube.com/index.php?/topic/52-faq-parity-and-duplication-and-drivepool/&do=findComment&comment=3542

 

And you can use the "File Placement Rules" to ensure that the files end up on the drives you want.

http://stablebit.com/Support/DrivePool/2.X/Manual?Section=File%20Placement

 

 

As for performance, you should get close to or better than the actual file speed. The Pool driver acts as a sort of proxy to the filesystem, so it should get least equal speed.  Additionally, we do a number of performance optimizations that may help boost this overall. 

http://stablebit.com/Support/DrivePool/2.X/Manual?Section=Performance%20Options

 

However, you won't see more than about 120MB/s over gigabit, as the max speed (not accounting for protocol overhead and other use) is 125MB/s. 

Though, this is very much anecdotal, I get about 100MB/s when copying files from the pool, over the network.  

Share this post


Link to post
Share on other sites
  • 0

Thanks for the reply. I had assumed balancing would mess with RAID snapshots, so I was always thinking that would be disabled. I basically want to define where new files go so folders stay together but I don't need the facility to have existing files moved or balanced, since they're already in the right place, so that should be fine. However, the following statement in the DrivePool user manual concerns me:

 

 

 

A file placement rule can be violated when renaming or moving a file on the pool. Such violations are detected and a background balancing pass is scheduled in order to resolve the violation by reordering the moved files.

 

So if I move a file within the pool, it'll get shoved on a random drive rather than obeying the file placement rule I've set up? That's a bit annoying, especially if I can't have balancing fix the issue automatically.

 

The answer regarding speeds is promising, I have some free time tonight so I'll try to get it set up and use the file-move trick to get my existing files in the pool, then test the file placement rules. If it works as intended I'll then do some speed testing. The only annoying thing is that moving the files will invalidate my RAID snapshot and redoing that will take more than a day (during which time I can't access my media), so I don't really want to commit to DrivePool without being confident it's a good improvement on my current solution. Particularly I want to see what happens with drive spin-ups when accessing various folders.

Share this post


Link to post
Share on other sites
  • 0

So I've created a pool and moved my existing files into it. Transfer speeds are awesome, I'm getting 115 MB/s over my network (for files on the newer, faster drives anyway)!

 

However, the file placement rules I've set up aren't being adhered to. For example, I've set my "Adam" folder to only use Disk 1 unless it's 99% full. It's currently ~90% full but when I copy a file over the network into that pool folder, it ends up on Disk 6. Is there something wrong with my setup? Do I need to restart the service or the PC itself for the rules to take effect or something? Screenshots below!

 

Note I've also tried completely disabling all of the plugins in the Balancing tab but it hasn't changed the behaviour.

post-2291-0-83781800-1452025314_thumb.png

post-2291-0-82335200-1452025315_thumb.png

post-2291-0-50262700-1452025316_thumb.png

Share this post


Link to post
Share on other sites
  • 0

So I've created a pool and moved my existing files into it. Transfer speeds are awesome, I'm getting 115 MB/s over my network (for files on the newer, faster drives anyway)!

 

However, the file placement rules I've set up aren't being adhered to. For example, I've set my "Adam" folder to only use Disk 1 unless it's 99% full. It's currently ~90% full but when I copy a file over the network into that pool folder, it ends up on Disk 6. Is there something wrong with my setup? Do I need to restart the service or the PC itself for the rules to take effect or something? Screenshots below!

 

Note I've also tried completely disabling all of the plugins in the Balancing tab but it hasn't changed the behaviour.

 

 

Make sure that the balancing is actually enabled. Unfortunately, the file placement rules are part of the balancing code, and it may require them to be enabled. 

 

I'm double checking with Alex (the developer to make sure, as I'm not 100% certain). 

 

https://stablebit.com/Admin/IssueAnalysis/22888

Share this post


Link to post
Share on other sites
  • 0

I'm not in front of the PC at the moment but what do you mean by ensuring balancing is enabled? I don't remember a specific toggle for that. Do you mean at least one balancing plugin needs to be enabled? Or that balancing needs to be run at least once to ensure file placement rules are adhered to?

 

In any case, since restarting the machine for another reason, so far it seems to be putting files in the right place. I will continue to monitor it and report any issues!

Share this post


Link to post
Share on other sites
  • 0

I see. I set it to "balance immediately" just before I left work. The progress bar went straight to 6.3% (presumably 100/16 based on the fact I have 8 HDDs in the pool?) and Resource Manager said it was moving a file from one of my nearly-full drives. Not sure where it was moving it to since the drives have obscure IDs in Resource Monitor (no drive letters) and the DrivePool GUI only says where it's moving the file from. I'll check it when I get home but I hope it doesn't start moving a bunch of files around for no reason - they should all be on the correct drives according to the file placement rules already.

Share this post


Link to post
Share on other sites
  • 0

As for the obscure IDs, yeah...

 

It should be "\Device\HarddiskVolume##\PoolPart.GUID".  Using the 'PoolPart" folder name, you can partially identify the device.

However, you can use Systernals' WinObj to get a good look at exactly which volume is which. 

Share this post


Link to post
Share on other sites
  • 0

Actually they're listed like:

 

D:3\directory\subdirectory

 

The number after "D:" is different for each disk and I can't tell which is which. The disk listed as Hard Drive 1 in Server Manager is definitely not D:1, for example.

Share this post


Link to post
Share on other sites
  • 0

No they're not off by 1. Strange but oh well, seems to be stable at the moment so I'll leave it for now.

 

However, I had to reboot my server today due to Windows Updates and noticed another issue - all of my SMB shares on the pooled drive disappear after each reboot and need manually re-adding. How can I fix this problem? If you'd like me to open a new thread for this issue please let me know.

Share this post


Link to post
Share on other sites
  • 0

Well I noticed that I couldn't connect to my shares, then when I checked the shares on the server all of the ones pointing to folders on the pool were gone. The pool drive itself was listed in Computer (P:) so I could access the folders locally that way. I re-made the shares and had to reboot a second time for more Windows Updates, then the shares disappeared once more.

 

I plan to make a batch script to create the shares because remaking them all manually is tedious. Once I've done that I'll try restarting the "Server" service after a reboot, as you suggest. I'm using the latest release build (not beta) on Windows Server 2012 by the way.

Share this post


Link to post
Share on other sites
  • 0

Rebooted once more, the shares were gone again. Restarting the "Server" service makes them reappear. So what's going on here, it sounds like the pool is being created too late in the boot process? Any way to fix this?

Share this post


Link to post
Share on other sites
  • 0

That's ... definitely odd.

 

As I said, it should re-share the files as soon as the drive is accessible.  And I've verified this recently, actually.

 

Worst case here, run "services.msc" on the system in question, find the "Server" service, right click on it and select "Properties". Set the "startup type" to "Automatic (Delayed)".  This will start the service later (2 minutes after the system has started, IIRC), and should completely bypass the issue.

Share this post


Link to post
Share on other sites
  • 0

So I've created a pool and moved my existing files into it. Transfer speeds are awesome, I'm getting 115 MB/s over my network (for files on the newer, faster drives anyway)!

 

However, the file placement rules I've set up aren't being adhered to. For example, I've set my "Adam" folder to only use Disk 1 unless it's 99% full. It's currently ~90% full but when I copy a file over the network into that pool folder, it ends up on Disk 6. Is there something wrong with my setup? Do I need to restart the service or the PC itself for the rules to take effect or something? Screenshots below!

 

Note I've also tried completely disabling all of the plugins in the Balancing tab but it hasn't changed the behaviour.

 

 

Okay, Alex has clarified here.

 

Specifically, the File Placement Rules should be respected by the real-time placement code in the file system driver (the "covefs.sys" driver, used for the pool).  

 

But existing files may not respect the rules, unless a manual balancing pass is run. 

The pool condition should indicate if this is necessary, and allow you to manually run a balancing pass:

 

Pool-Organization1.png

Share this post


Link to post
Share on other sites
  • 0

So those little symbols should mean some files aren't in the correct place according to the file placement rules if I have all other balancing plugins disabled? That's interesting, I might try that and see what it shows - I know from looking at the drives themselves that the data is all in the correct place right now.

 

By the way, setting the "Server" service to start delayed seems to have fixed the missing pool shares issue, thanks.

Share this post


Link to post
Share on other sites
  • 0

Yeah, the icons do indicate that.  And the pool condition at the bottom will allow you to manually rebalance the files, if it detects that the files are "out of balance" basically.

 

 

And glad to hear that the delay has fixed the shares issue.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...