Setting up a hierarchical pool using an existing Pool

[IanT]

I have a question about hierarchical pools. I have an existing Pool containing 9 disks, with a variety of duplication on various folders. I also have available several SSD's which I would like to use as accelerators for the pool, but I would prefer to avoid 'wasting' space on these by duplicating. So, thinking hierarchical pools could solve this I set out to build the following set of pools .....           

                                                                                                           Pool H (new pool, the root of my hierarchy)

                                                                                                              /     \

                                                                                                            /         \

                                                                              Pool A (new SSDs)           Pool B (existing pool 9 disks)

 

Pool H would not be duplicated, but would mark Pool A as being an SSD drive, and Pool B as being an Archive drive.

When I did this and opened a file browser on Pool H, there was nothing showing!! However when I browse Pool B, all my data is there. 

 

The question is how do I "populate" Pool H with the data from Pool B?? and if this involves physically moving files on Pool B, how do I retain the replication behavior on Pool B?

 

Any hints? Or am I completely misunderstanding how hierarchical pools work?

                                                                                                                                     

 

[Shane]

Let's see if I have this right:

B is your old pool with all your stuff in it, and you want to keep its various per-folder duplication settings in the new pool.
A is your new pool of SSDs, empty, and you want to use it as cache for your stuff via the Optimizer balancer, without any duplication.
H is the new pool you created, added A and B to it, found the stuff in B didn't show up in H, you want that to happen, and you haven't put any files in H pool yet.

If so, then:

Tricky but it's doable thanks to DrivePool's file placement and per-folder duplication features being able to see (in-situ parts of) a super-pool as its own folder tree in the selected sub-pool with its own settings for the latter. And if that doesn't make sense yet then don't worry about it, it'll become clear.

• Simple way that'll probably take a very long time but should be safe (in the sense that it doesn't mess with poolparts):
   
  1. Temporarily remove A pool from H pool.
  2. Run this command to recreate B pool's folder structure in H pool:

     robocopy b:\ h:\ /e /dcopy:dat /xf * /xd "$RECYCLE.BIN" "System Volume Information" ".covefs" /r:1 /w:1

     The reason we're doing this is so we can see all the folders in the next step. I'm assuming here that you do NOT have any symbolic links, special access controls or other NTFS clever stuff here. If you do, customise as needed. EDIT: Or maybe 'un-clever' your pool B first and 're-clever' pool H after the entire way is complete.

  3. In the DrivePool GUI select pool B, then select Manage Pool -> File protection -> Folder duplication... You should see pool H "within" pool B; here is where you manually copy B pool's per-folder duplication settings for H pool. For example:
     
     B pool:
     • documents x2
       • criticaldocs x4
     • H pool:
       • documents x1 -> x2
         • criticaldocs x1 -> x4
       • music x1 -> x3
     • music x3
        
  4. Move your files from B pool's folder tree to H pool's folder tree. This will take ages because Windows will think they're on different physical drives even though they're not, and if you're using Explorer you'll have to do so in chunks no bigger than what Windows believes is the free space available on the target pool (I'd suggest chunks no bigger than the free space available on the physical drive with the most free space).
  5. That should (eventually) be it; your stuff will now all show up in pool H but it will be duplicated across the drives of pool B in accordance with your preferences there. Now you can re-add A pool to H pool and configure H pool's SSD Optimizer to use it as cache.

 

• Alternate way that should be much quicker - unless you make a mistake, because it's messing with poolparts:
   
  1. USE AT OWN RISK. Edit: For example, I'm assuming here that you do NOT have any symbolic links, special access controls or other NTFS clever stuff here. If you do, customise as needed. You may want to 'un-clever' your B pool before you use this way and then 're-clever' your H pool afterwards.
  2. Do NOT do this at night, because we want to avoid any risk of tripping over DrivePool's nightly background duplication check.
  3. Make a record of all your per-folder duplication settings in B pool.
  4. Temporarily remove A pool from H pool.
  5. In the DrivePool GUI select pool B and turn off its automatic balancing, then select pool H and turn off its automatic balancing.
  6. In the DrivePool GUI select pool B.
     1. Now select Manage Pool -> Performance -> Real-time duplication and make sure it is TICKED.
     2. Now select Manage Pool -> File protection -> Folder duplication... you should see pool H "within" pool B. Click on it, then Change its duplication level to be the same as the number of drives in pool B.
     3. Now select Manage Pool -> Performance -> Real-time duplication again and make sure it is NOT ticked any more.
  7. So every drive in pool B should have a hidden poolpart folder in its top directory and now thanks to the previous step each of those should itself have a hidden poolpart folder inside - the latter are the poolparts for pool H. So in each drive, drag your stuff - not any Windows or CoveFS stuff - from the former to the latter. Do NOT drag any folders named "$RECYCLE.BIN", "System Volume Information" or ".covefs".
     
     So if you had:
     
     x:\PoolPart.long-string-of-hexadecimals\ <-- this should be B pool's.
     -> $RECYCLE.BIN
     -> documents
     -> pictures
     -> PoolPart.different-long-string-of-hexadecimals\ <-- this should be H pool's.
     -> System Volume Information
     -> videos
     
     You would drag only "documents", "pictures" and "videos" into the PoolPart.different-long-string-of-hexadecimals folder.
  8. Confirm that all your stuff is now showing up in pool H.
  9. Open a command prompt as an administrator and run these two commands:

     dpcmd reset-duplication-cache b:
     dpcmd reset-duplication-cache h:

  10. In the DrivePool GUI select pool H, then select Manage Pool -> File protection -> Folder duplication..., make sure all duplication is off (fix it to off if needed).
  11. In the DrivePool GUI select pool B, then:
      1. select Manage Pool -> File protection -> Folder duplication..., recreate from the record you made earlier.
      2. select Manage Pool -> Performance -> Real-time duplication and make sure it is TICKED again ONLY IF you normally use real-time duplication.
  12. That should be it; your stuff will now all show up in pool H but it will be duplicated across the drives of pool B in accordance with your preferences there. Now you can turn back on automatic balancing (if you use it) for pools B and H, re-add A pool to H pool and configure H pool's SSD Optimizer to use it as cache.

Whew.

Edited March 12 by Shane
Edited with additional warnings.

[IanT]

'Whew' indeed!!! 

First of all, I cannot believe that you went to that amount of trouble! Secondly, it will take me a while to "process" this 😄.

I have to say that in the 'simple' approach above, I actually do have symbolic links, and some NTFS 'clever stuff', and then in the 'alternate' way, I'm still wrapping my head around step 6.2, especially the phrase "you should see pool H "under" pool B". I'll need to perform some experiments before this all drops into place for me.

Nevertheless, thankyou, thankyou, thankyou for confirming in my mind that this may well be the most impressive software I own, and that this group of users (and volunteers) is without doubt the best!

[Shane]

Sorry, be aware that the warning about symbolic links and other NTFS clever stuff applies to the alternate way too; I'm not sure what might break. And "under" in the alternate way == "within" in the first way (I just didn't happen to use the same word when typing it out). I'll edit both.

The biggest danger I can think of, and that you do NOT want to end up with, is a symlink/junction/whatever that creates a loop between pools. As I've added in the edit, you may wish to 'un-clever' your old pool before starting and only 're-clever' your new pool once it's all there.

[IanT]

Got it!! As I said much earlier, this is a 'nice to have', I don't actually need to do it as I have enough spare SSD;s to cover the minimum need. So in fact I have lots of time to run a variety of experiments, and may even choose to not bother (probably not what you hoped for given how much thought you've put into this).

 

Ian