Jump to content

  • Log in with Twitter Log in with Windows Live Log In with Google      Sign In   
  • Create Account

Photo

Subpools or Drive Groups Within a Pool


  • Please log in to reply
25 replies to this topic

#1 Guynamedbilly

Guynamedbilly

    Newbie

  • Members
  • Pip
  • 4 posts

Posted 22 May 2016 - 05:43 PM

Hello again.  I'm trying to figure out if there is any way to subdivide drives within a single pool so that I can prioritize which ones duplicate data first. 

 

My scenario is this.  I've got two external USB3 JBOD drive carriages that hold 4 disks each.  If I just add all of the disks to a Drivepool, the data is spread across each of the drives fairly evenly no matter which carriage they are in.  I'd like to have the duplicated data be written to each seperate carriage whenever there is enough available space to do so.  This would provide better performance since they are not all going through the same USB interface, and also better security since if one carriage was damaged then the other would have a better chance to survive.

 

I know that you can limit which individual drives receive duplicated data,  is there any way to do that for groups of drives?


  • KiaraEvirm likes this

#2 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,203 posts
  • LocationSan Diego, CA, USA

Posted 22 May 2016 - 07:09 PM

No, we don't have anything like that yet. 

 

Specifically we do plan on adding a "Duplication Grouping" feature to StableBit DrivePool in the near future.  Especially as it would work VERY well with StableBit CloudDrive, but also for external storage. 

 

However, we are a small company, and have limited resources.  Once StableBit CloudDrive has been released, Alex (the Developer) plans on going though the backlog of requests (including this) and addressing as many of them as we can.  

 

Unfortunately, we don't have an ETA on this. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#3 KingMotley

KingMotley

    Member

  • Members
  • PipPip
  • 11 posts

Posted 31 March 2017 - 01:58 AM

Any ideas on an ETA yet?  It's been over 10 months...



#4 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,203 posts
  • LocationSan Diego, CA, USA

Posted 10 April 2017 - 06:54 AM

It's in Alpha. :)

http://dl.covecube.c...oad/changes.txt

 

  • [D] Added hierarchical pooling support:
    - Pools are now made up of either disks or other pools.
    - Each pool handles its own separate folder duplication settings.
    - Balancing can work over a pool part, but file placement settings are limited to the entire pool part only.
    - Circular pooling is not allowed (e.g. Pool A -> Pool B AND Pool B -> Pool A). Contrary to popular belief, this does not lead to infinite disk space.

  • thnz likes this

Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#5 donnatronious

donnatronious

    Newbie

  • Members
  • Pip
  • 9 posts

Posted 21 April 2017 - 02:40 AM

Created an account here just to let you know this is only feature missing that I would love to see :-)


  • Christopher (Drashna) likes this

#6 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,203 posts
  • LocationSan Diego, CA, USA

Posted 21 April 2017 - 05:12 PM

Created an account here just to let you know this is only feature missing that I would love to see :-)

 

It's in the beta version, but this feature is very, very new. It may still have bugs (probably does). And we know that the UI needs an overhaul to handle things better, as this will get cause the UI to get complicated, fast.

 

But if you want to go ahead and test it out, by all means, do so.  Just let us know if you run into any issues with it. 

http://dl.covecube.c...wnload/?C=N;O=D

 

Grab the 2.2.0.744 or 2.2.0.745 build, and give it a spin.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#7 browned

browned

    Member

  • Members
  • PipPip
  • 20 posts

Posted 21 April 2017 - 08:31 PM

Not sure I am using this new feature but are there any recommendations as to what features should be enabled/disabled?

 

My setup

Sata card 1, 7 disks, Pool A, no duplication

Sata card 2, 7 disks, Pool B, no duplication

 

Pool A + Pool B = Pool C, with duplication enabled so two copies of all files on each sata card.

 

I have changed a disk in Pool B and it is balancing at the moment, I think this is causing issues with Duplication on Pool C as it starts runs finishes and starts again. Should I turn balancing off on Pool A & B as it really isn't required, or just let it all work itself out over time?



#8 donnatronious

donnatronious

    Newbie

  • Members
  • Pip
  • 9 posts

Posted 22 April 2017 - 12:05 AM

It's in the beta version, but this feature is very, very new. It may still have bugs (probably does). And we know that the UI needs an overhaul to handle things better, as this will get cause the UI to get complicated, fast.

 

But if you want to go ahead and test it out, by all means, do so.  Just let us know if you run into any issues with it. 

http://dl.covecube.c...wnload/?C=N;O=D

 

Grab the 2.2.0.744 or 2.2.0.745 build, and give it a spin.

Understood.  Retaining elegance as flexibility increases is, um, hard.


Not sure I am using this new feature but are there any recommendations as to what features should be enabled/disabled?

 

My setup

Sata card 1, 7 disks, Pool A, no duplication

Sata card 2, 7 disks, Pool B, no duplication

 

Pool A + Pool B = Pool C, with duplication enabled so two copies of all files on each sata card.

 

I have changed a disk in Pool B and it is balancing at the moment, I think this is causing issues with Duplication on Pool C as it starts runs finishes and starts again. Should I turn balancing off on Pool A & B as it really isn't required, or just let it all work itself out over time?

I don't have any answer for you because I'm new here.

 

But I'm curious, is your intent to have 1 copy of each file in pool A and 1 copy in pool B?  Thats why I want the feature.  I want my duplication to happen across interfaces for speed.

 

This statement makes me unclear "two copies of all files on each sata card"



#9 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,203 posts
  • LocationSan Diego, CA, USA

Posted 22 April 2017 - 12:12 AM

I'd recommend letting it sit, as over time, yes it should work itself out. 

 

That said, I'm bugging this, because it looks like we'll need to make sure that the service is much more aware of what is going on to prevent conflicts like this. 

 

Could you get tracing logs from when this is happening, though? 

http://wiki.covecube..._Log_Collection

 

 

Understood.  Retaining elegance as flexibility increases is, um, hard.


I don't have any answer for you because I'm new here.

 

But I'm curious, is your intent to have 1 copy of each file in pool A and 1 copy in pool B?  Thats why I want the feature.  I want my duplication to happen across interfaces for speed.

 

This statement makes me unclear "two copies of all files on each sata card"

 

Yeah, and this is far from a simple thing.

 

From the sounds of it, he has two controller cards.  He wants one set of files on each controller. That way if one controller "goes out", all of the data is stored on the other controller, so all of the data is still up and running. 

 

There are others that want this for internal vs external, multiple external enclosures, local vs cloud, etc.   Basically, it's a way to create "discrete units" to manage the duplication better. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#10 donnatronious

donnatronious

    Newbie

  • Members
  • Pip
  • 9 posts

Posted 22 April 2017 - 12:13 AM

zm09qq.png



#11 browned

browned

    Member

  • Members
  • PipPip
  • 20 posts

Posted 22 April 2017 - 06:46 AM

I'd recommend letting it sit, as over time, yes it should work itself out. 

 

That said, I'm bugging this, because it looks like we'll need to make sure that the service is much more aware of what is going on to prevent conflicts like this. 

 

Could you get tracing logs from when this is happening, though? 

http://wiki.covecube..._Log_Collection

 

 

 

Yeah, and this is far from a simple thing.

 

From the sounds of it, he has two controller cards.  He wants one set of files on each controller. That way if one controller "goes out", all of the data is stored on the other controller, so all of the data is still up and running. 

 

There are others that want this for internal vs external, multiple external enclosures, local vs cloud, etc.   Basically, it's a way to create "discrete units" to manage the duplication better. 

 

 

Yes sorry, one copy of each file on each sata card. Two reasons for this, the obvious duplication of data on separate cards in case of failure. The second, sending and receiving data from 2 cards at the same time should be faster than duplicating on the same card. The duplication seems to have settled down after the balancing, so I will leave the logging. But it seemed to be balancing pool a or b and duplicating pool c at the same time. It would be better if those tasks functioned individually.

 

An idea for the future, detect the drive pool within a drive pool and automatically disable features not required or prioritize the tasks based on user preferences. I.E. Disk replacement or addition will start a balancing process, in the case of a replacement disk and data is not duplicated in pool c, it would be better to duplicate pool c than to balance pool a, after successful duplication, balance pool a. This would also create less balancing work as the duplication process should use the empty drive in pool a therefore requiring less balancing.

 

At the moment, I have Pool A and B configured with all performance options enabled, balancing for drive overfill and volume equalization, no rules.  Pool C has performance options enabled except bypass system filters, pool duplication enabled, no balancing and no rules.


  • Christopher (Drashna) likes this

#12 browned

browned

    Member

  • Members
  • PipPip
  • 20 posts

Posted 22 April 2017 - 08:10 PM

Hmm monitoring a problem at the moment.

 

Pool Duplication is enabled on Pool C, I had a look at folder duplication on Pool C, set the $Recycle.bin to 1x and then noticed the unduplicated data growing. Rechecked the folder duplication and random folders were set to 1x. Tried changing a single folder back to 2x and got a task failed error. Disabled pool duplication, enabled it again and rebooted the server and checked folder duplication is all now 2x. It now seems to be duplicating the data again.



#13 donnatronious

donnatronious

    Newbie

  • Members
  • Pip
  • 9 posts

Posted 22 April 2017 - 08:37 PM

Hmm monitoring a problem at the moment.

 

Pool Duplication is enabled on Pool C, I had a look at folder duplication on Pool C, set the $Recycle.bin to 1x and then noticed the unduplicated data growing. Rechecked the folder duplication and random folders were set to 1x. Tried changing a single folder back to 2x and got a task failed error. Disabled pool duplication, enabled it again and rebooted the server and checked folder duplication is all now 2x. It now seems to be duplicating the data again.

A simple solution for what you want would be to disable all duplication within drivepool and use a separate program to sync the two pools, thereby achieving the redundancy.

 

I'll be doing this with a program I've been using for years to backup PC's to a NAS, http://www.smartsync.com/

 

The only downside I can see here is you won't get the read boost if drivepool isn't doing the duplication.



#14 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,203 posts
  • LocationSan Diego, CA, USA

Posted 22 April 2017 - 09:32 PM

Hmm monitoring a problem at the moment.

 

Pool Duplication is enabled on Pool C, I had a look at folder duplication on Pool C, set the $Recycle.bin to 1x and then noticed the unduplicated data growing. Rechecked the folder duplication and random folders were set to 1x. Tried changing a single folder back to 2x and got a task failed error. Disabled pool duplication, enabled it again and rebooted the server and checked folder duplication is all now 2x. It now seems to be duplicating the data again.

 

If you see this happen again, enable drive tracing on all of the pools and reproduce. 

http://wiki.covecube..._Log_Collection


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#15 Umfriend

Umfriend

    Advanced Member

  • Members
  • PipPipPip
  • 372 posts

Posted 24 April 2017 - 08:33 AM

Wow! That you guys are working on this really makes me happier. Been waiting for over 1.5 yrs now I think.

 

From the changes.txt, I gather that I could do something like this:

1. Create Pool1  from HDD1 & HDD2 - no duplication

2. Create Pool2 from HDD3 & HDD4 - no duplication

3. Create Pool3 from Pool1 & Pool2 - x2 duplication?

 

Could I also add HDDs to Pool3? Would seem weird but it might be nice for SSD caches. Although I think I'd rahter add an SSD cache to Pool1 and Pool2 seperately.

 

Had to laugh about "Circular pooling is not allowed (e.g. Pool A -> Pool B AND Pool B -> Pool A). Contrary to popular belief, this does not lead to infinite disk space." - :D



#16 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,203 posts
  • LocationSan Diego, CA, USA

Posted 24 April 2017 - 07:39 PM

Yeah, it's something that Alex has been wanting to add for a while (or something to allow "grouping" in general). 

 

As for the pools, yes, you could do that, exactly.  You could set the duplication status on the underlying pools as well.

 

And yes, you should be able to add drives to "Pool 3" (the pool of pools), as well. Just remember, that the file systems have to match, so the "pool of pools" will be NTFS, as that is what the pool registers as.  

 

And yeah... You notice how it skipped two builds?  I believe that Alex didn't add a check for circular pooling in the first build, so this was a bit of self deprecating humor. :)


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#17 Umfriend

Umfriend

    Advanced Member

  • Members
  • PipPipPip
  • 372 posts

Posted 25 April 2017 - 08:57 AM

And how does DPCMD behave when you want to get a dump of the MOAP (Mother Of All Pools)?



#18 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 8,203 posts
  • LocationSan Diego, CA, USA

Posted 25 April 2017 - 03:52 PM

Not sure I am using this new feature but are there any recommendations as to what features should be enabled/disabled?

 

My setup

Sata card 1, 7 disks, Pool A, no duplication

Sata card 2, 7 disks, Pool B, no duplication

 

Pool A + Pool B = Pool C, with duplication enabled so two copies of all files on each sata card.

 

I have changed a disk in Pool B and it is balancing at the moment, I think this is causing issues with Duplication on Pool C as it starts runs finishes and starts again. Should I turn balancing off on Pool A & B as it really isn't required, or just let it all work itself out over time?

 

Have you seen this again, or are able to replicate it at will? 

 

 

And how does DPCMD behave when you want to get a dump of the MOAP (Mother Of All Pools)?

 

It dumps one pool at a time. As per normal. You specify which pool you want to run this on, and it grabs the info from THAT pool only. 

 

That's how it's always worked.  So, if you want a "master list", I guess you'd want to run it on the main pool, and then run it on the sub-pools.

 

Otherwise, the pool of pools will treat the underlying pools as normal disk. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#19 Umfriend

Umfriend

    Advanced Member

  • Members
  • PipPipPip
  • 372 posts

Posted 25 April 2017 - 04:48 PM

So DPCMD on the POP will show two devices which are in fact Pool1 and Pool2? Not the actualy underlying physical devices?



#20 browned

browned

    Member

  • Members
  • PipPip
  • 20 posts

Posted 26 April 2017 - 03:30 AM

Have you seen this again, or are able to replicate it at will? 

 

 

 

I have just upgraded to 746. Changed the $recycle.bin to 1 x dup and checked the pool and folder duplication was still 2x for everything else. Also,  the task failed error has gone as per the 746 change log. I am doing some logging at the moment so will post it through it I see any odd behavior.

 

Edit: Actually, something I am seeing a lot is that the real time duplication doesn't seem to work. This may sound more drastic than it actually is. I am seeing large files copied or moved or overwritten in the pool and then the master pool will say, as an example 8.81GB is not duplicated, so it kicks off a duplication run, when it is finished I have 5xMB unduplicated. I am wondering it the <random file names>.copytemp from pool A and Pool B are being picked up when and if those pools balance themselves and if a balance in Pool A or Pool B will affect the Duplication in the Master Pool. I will post some logs when I have the time to work on this.

 

Edit 2: Logs uploaded for continual duplication runs.






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users