Drivepool with Local Disks and Parity via CloudDrive

[red]

Hey,

I've set up a small test with 3x physical drives in DrivePool, 1 SSD drive and 2 regular 4TB drives. I'd like to make a set up where these three drives can be filled up to their brim and any contents are duplicated only on a fourth drive: a CloudDrive. No regular writes nor reads should be done from the CloudDrive, it should only function as parity for the 3 drives.

Am I better off making a separate CloudDrive and scheduling an rsync to mirror the DrivePool contents to CloudDrive, or can this be done with DrivePool (or DrivePools) + CloudDrive combo? I'm running latest beta for both.

What I tried so far didn't work too well, immediately some files I were moving were being actually written on the parity drive even though I set it to only contain duplicated content. I got that to stop by going into File Placement and unticking parity drive from every folder (but this is an annoying thing to have to maintain whenever new folders are added).

1) 

2) 

[red]

I imagine what might be happening, is that my SSD "buffer" gets full quite fast, and the overflow is just randomly shoved to remaining disks and maybe later balanced to respect set rules (instead of never utilizing the cloud drive called "parity")

[Christopher (Drashna)]

You need two drives for "duplicated".  Otherwise, you're telling the system that there is only one disk that is valid for both copies of the data. 

The best way to do this would be go create two pools.  One with F:, E: and D:.  Add this pool and I: to another pool.  Enable duplication on that pool, but not the pool in this pool

[red]

Thanks Drashna, I made that now. Can I cut & paste files from my first drivepool and put it under the PoolPart folder that was now created inside it (fast), or do I need to move them from drive to drive the usual way (slow)?

Second question: If now copy my data into this "hybridpool", it seems to be spreading partly in the localpool and partly in the cloudpool. How can I have the cloudpool part only be there for parity and have disk reads and initial writes from the localpool side (so that basicly the cloudpool is just for the copies and nothing more).

[Christopher (Drashna)]

3 hours ago, red said:

Thanks Drashna, I made that now. Can I cut & paste files from my first drivepool and put it under the PoolPart folder that was now created inside it (fast), or do I need to move them from drive to drive the usual way (slow)?

Yes, absolutely! 
http://wiki.covecube.com/StableBit_DrivePool_Q4142489

3 hours ago, red said:

Second question: If now copy my data into this "hybridpool", it seems to be spreading partly in the localpool and partly in the cloudpool. How can I have the cloudpool part only be there for parity and have disk reads and initial writes from the localpool side (so that basicly the cloudpool is just for the copies and nothing more).

for reads, update to the latest beta, it will do this automatically. 
http://dl.covecube.com/DrivePoolWindows/beta/download/StableBit.DrivePool_2.2.0.878_x64_BETA.exe

Quote

.866
* [Issue #27655] Added an indicator to the pooled disk tooltip to show when a drive is detected to be external.
* [D] [Issue #27655] When a file is opened, each open context is given a priority. Read striping will always work over the highest 
                     available priority class. 
                     Priority classes from lowest to highest are:
                        - CloudDrive, CloudPool, LocalExternalDrive, NonCloudPool, StandardDrive

 

As for writes, there isn't a good way to do this, aside from the SSD Optimizer. 

The reason is that StableBit DrivePool writes all copies out, at the same time (real time duplication), so there isn't a way to really stop this.  However, it shouldn't be a problem, IMO. 

[red]

Thanks for the great reply. I'll  return here if I need more help in further optimizing the setup!

 

[Christopher (Drashna)]

You're very welcome! 

[red]

I'm back for more questions, but first; here are my HybridPool balancing settings:

 

1) What I'm now trying to achieve is that when I copy new files to my HybridPool, they would be written on LocalPool first (has SSD - fast), then at night, duplicated to the CloudDrive. What is happening, it would seem, instead, is that DrivePool writes some data directly to the CloudDrive and balances it back to LocalPool (no SSD - slow).Is there any way to achieve said functionality?

2) So duplication is enabled, but I've set a handful of very large folders to only be placed on the CloudDrive, and have changed duplication settings for these folders to be 1x. Balancing doesn't seem to like this, as I've tested for two nights in a row to leave the HybridPool balancing, but when I've come to the PC in the morning, the percentage is dead on in the same spot, and the balance of drives is the same -- any suggestions what might be causing this? Am I better off moving this "pure cloud content" away from the pool and just symlinking it to my HybridPool (I literally need it to be accessible from the same folder path - even if it's stored in the cloud)?

 

 

This is how I've set File placement for the data I want only to be kept on cloud storage, but available in the same pool, just via CloudDrive streaming & local caching.

[Christopher (Drashna)]

If you are using both the "Ordered File Placement" balancer and the "File Placement rules", then you need to do this:
    * UNCHECK "File placement rules respect real-time file placement limits set by the balancing plug-ins."
    * CHECK "Balancing plug-ins respect file placement rules."
    * UNCHECK "Unless the drive is being emptied."
 

[ostateblazer]

Hey sorry to revive an old thread - I can post a new one if necessary, but I'm in a very similar situation to Red. I followed the above instructions, and how have pools as follows: 

• A:\ = physical disk, 8 TB
• B:\ = physical disk, 8 TB
• D:\ = drive pool of A (physical) and B (physical), 16 TB (using manufacturers sizes for ease)
• Z:\ = could drive, 16 TB
• E:\ = drive pool of D (pool) and Z (cloud), 32 TB

What I'm going for is a mirror of D on Z, accomplished by the E pool.  I've already followed the steps with disabling the service and copying into the new pool folder, twice, such that:

• My files are nested in A and B as \PoolPart.xxx\PoolPart.yyy\files\
• In D and Z as \PoolPart.zzz\files\
• And show up in E as I would hope, E:\files\

What I can't figure out is the proper balancing to accomplish my goal of mirroring.  I've tried setting Drive Usage Limiter as follows: 

• D = [X] Duplicated (checked) - [  ] Unduplicated (unchecked)
• Z = [X] Duplicated (checked) - [  ] Unduplicated (unchecked)

I have 8 TB of actual data in my D:\ pool (4 TB each on A and B).  When I set the above balancing setting, it puts the "Un-duplicated target for rebalancing" at 4 TB per drive (i.e., 4 TB on physical disks, 4 TB in cloud):

This seems like it means to delete 4 TB from my disks, and upload that 4 TB to the cloud?  I want the full 8 TB to remain, and a complete copy (8 TB) uploaded to the cloud. 

When I also check "Unduplicated" on D, it thinks the optimal allocation is everything on D and nothing on Z:

(The 65 GB is from my tests of manually copying files over.) 

Just to see what happened, I tried only "Unduplicated" on D, and only "Duplicated" on Z:

I feel like what I want is for that damn yellow triangle to slide over to the right end on both drives (i.e., up to 16 TB (or 14.6) available on each, with a current target of 8 TB duplicated), which I would expect by checking only "Duplicated" for both D and Z.

Any idea what I'm doing wrong here?

 

[Jaga]

50 minutes ago, ostateblazer said:

Any idea what I'm doing wrong here?

You may be overworking the solution. 

What may be helpful is all of your pool files to be only on A and B, with the Cloud drive empty.  You have your child pool setup as D, then you create the master pool E by adding in the child pool D and the Cloud drive Z, and immediately give E full 2x pool duplication: when viewing the E pool in DP, click the up-arrow next to Manage Pool, choose File Protection, then Pool File Duplication.  That will allow you to enable 2x duplication on the E pool, using the child pool and the cloud drive.

You don't need to enable the drive usage limiter at all, if you setup as I described.  D and Z would end up being exact mirrors of each other - both are masters in a 2x full pool duplication.  And pool D doesn't need any duplication or limiting at all, unless you want certain folders/files inside of it to get extra protection using those two drives.  

As said above, I wouldn't start with a pre-populated Cloud drive for mirroring, I'd have it empty and allow Drivepool to locally populate it (which it does after enabling pool duplication on E). 

[ostateblazer]

That's that I was missing - thanks!!

Just to be clear, you mean I should disable the "Drive Usage Limiter" under Manage Pool-->Balancing?  I only ask because it seems to be enabled by default, and I'm not comfortable enough with the software to change away from default settings without clear instruction... 

[Jaga]

Yep - if you don't specifically need it, I'd go ahead and disable it.  Your pool drives (both the child pool and mirror/master pool) are either un-mirrored or 2x mirrored at the pool level, and I don't think you have any file/folder mirroring going on, so you don't have any specific need for what it can do.

[ostateblazer]

Thanks again!  5 hours later and I'm 4.0% done... 

Do I need to leave it alone until it finishes, or can I turn it off and/or pause the CloudDrive syncing, and have it resume later?  

[red]

Btw, I started experiencing quite long read delays with my 100GB SSD cache and 20TB files in my "HybridPool". Despite having local copies of files, double clicking on a video would just wait a good 3-4 seconds before starting playback, even though I could see from Drivepool the file was being accessed only locally and no bits were being downloaded. Same thing with browsing the folder structure. White background in explorer after opening a folder for many seconds.

I mainly use my setup with Plex, so I just pointed Plex to read everything from the LocalPool directly. The CouldDrive part just acts as a simple parity for if a disk dies. Cheaper and easier than raid. Just remember you can't delete files from the LocalPool directly or they'll just re-appear on next re-balance/duplicate.

[Jaga]

9 hours ago, ostateblazer said:

Thanks again!  5 hours later and I'm 4.0% done... 

Do I need to leave it alone until it finishes, or can I turn it off and/or pause the CloudDrive syncing, and have it resume later?  

I've never tried a Cloud drive that large, so I can't really advise.  But if you can keep the computer on and uploading, I'd do it.  I would think with the block-storage nature of Stablebit Clouddrive, it would simply start up where it last left off when you turn the machine back on.

 

8 hours ago, red said:

I mainly use my setup with Plex, so I just pointed Plex to read everything from the LocalPool directly. The CouldDrive part just acts as a simple parity for if a disk dies. Cheaper and easier than raid. Just remember you can't delete files from the LocalPool directly or they'll just re-appear on next re-balance/duplicate.

Good tip.   

[Christopher (Drashna)]

On 9/28/2018 at 1:27 AM, ostateblazer said:

Thanks again!  5 hours later and I'm 4.0% done... 

Do I need to leave it alone until it finishes, or can I turn it off and/or pause the CloudDrive syncing, and have it resume later?  

Yeah, you'd want to just let it sit and do it's thing. 

On 9/28/2018 at 3:13 AM, red said:

Btw, I started experiencing quite long read delays with my 100GB SSD cache and 20TB files in my "HybridPool". Despite having local copies of files, double clicking on a video would just wait a good 3-4 seconds before starting playback, even though I could see from Drivepool the file was being accessed only locally and no bits were being downloaded. Same thing with browsing the folder structure. White background in explorer after opening a folder for many seconds.

I mainly use my setup with Plex, so I just pointed Plex to read everything from the LocalPool directly. The CouldDrive part just acts as a simple parity for if a disk dies. Cheaper and easier than raid. Just remember you can't delete files from the LocalPool directly or they'll just re-appear on next re-balance/duplicate.

Could you open a ticket about this issue, if you haven't already? 

https://stablebit.com/Contact

[red]

I opened one now.

[Christopher (Drashna)]

Awesome, I've responded to it, too. 

[feerlessleadr]

Just to follow up on Red's problem/question - I am going to be setting up something very similar. My drives are set up as follows:

• V:\ - Pool set up of multiple physical disks of varying sizes - 25TB total
• F:\ - Pool set up of 2 cloud drive Partitions (Google Drive, 60TB each) - 120TB total
• Z:\ - Pool of both V (physical pool) & F (cloud pool)

Based on the response from Jaga above, I'm pretty clear on how to accomplish my goal of having F be a mirror of V (and consequently vice-versa). My question comes though in relation to emby, which currently uses the V:\ pool for managing my movies/TV shows. I will oftentimes delete unwanted/watched/etc. movies/shows from the V:\ pool, and If I'm understanding correctly, if I were to continue doing this, if I were to delete a file from the V:\ pool, overnight when balancing takes place (or whenever that is triggered to happen), the deleted files would be added back to V:\ from F:\, since I would have 2x duplication turned on in the new pool Z:\. Is that correct?

Assuming that is correct, the obvious solution is to change emby to point to the new Z:\ pool, rather than the V:\ pool. Before I do this however, I was wondering if there was a way to have my programs prioritize reading/writing to the Z:\ pool using the physical disks first (the V:\ pool), then duplicating to the cloud disks (the F:\ pool), since writing to the physical disks would obviously be much quicker.

I see that Red was also having challenges with slow read/write times using the combined pool (in my case this would be pool Z:\). His solution to just point to the local pool is great, but that would mean that if I wanted to delete something through emby, I wouldn't be able to, and I would need to use the file explorer to delete the files through the hybrid pool.

Just wanted to see if this is something that was fixed either in a new beta release of drive pool/cloud drive, or if there are specific settings that would need to be adjusted to be able to have emby point to the hybrid pool so that I can keep everything clean and tidy.

Thanks!

[Christopher (Drashna)]

Yeah, that's basically correct. 

On 10/13/2018 at 6:43 AM, feerlessleadr said:

, I was wondering if there was a way to have my programs prioritize reading/writing to the Z:\ pool using the physical disks first (the V:\ pool), then duplicating to the cloud disks (the F:\ pool), since writing to the physical disks would obviously be much quicker.

The "Read Striping" feature should do this, actually.  However, directory enumeration will cause activatity on the cloud, but reads should happen from the local disks. 

On 10/13/2018 at 6:43 AM, feerlessleadr said:

 I see that Red was also having challenges with slow read/write times using the combined pool (in my case this would be pool Z:\). His solution to just point to the local pool is great, but that would mean that if I wanted to delete something through emby, I wouldn't be able to, and I would need to use the file explorer to delete the files through the hybrid pool.

Writes may be slower, because it's writing to both pools, and therefore the CloudDrive disks. So it would heavily depend on how fast those drives are.

Also, if you were to use the SSD Optimizer with the Z:\ pool, that would help too. 

 

[feerlessleadr]

4 minutes ago, Christopher (Drashna) said:

Yeah, that's basically correct. 

The "Read Striping" feature should do this, actually.  However, directory enumeration will cause activatity on the cloud, but reads should happen from the local disks. 

Writes may be slower, because it's writing to both pools, and therefore the CloudDrive disks. So it would heavily depend on how fast those drives are.

Also, if you were to use the SSD Optimizer with the Z:\ pool, that would help too. 

 

Thanks - I'll set this all up once I can get my hands on a good sized SSD drive to use as my cache for the clouddrive.

[feerlessleadr]

8 minutes ago, Christopher (Drashna) said:

Yeah, that's basically correct. 

The "Read Striping" feature should do this, actually.  However, directory enumeration will cause activatity on the cloud, but reads should happen from the local disks. 

Writes may be slower, because it's writing to both pools, and therefore the CloudDrive disks. So it would heavily depend on how fast those drives are.

Also, if you were to use the SSD Optimizer with the Z:\ pool, that would help too. 

 

1 more question on the SSD optimizer option - would I be able to use the same SSD that I'm using as a cache for the cloud drive as the SSD cache for the Z:\ pool? 

If so, is there a minimum size I should be looking for? I would be duplicating (initially anyway) about 24 TB worth of data, so not sure if it would really make a difference having a 128/256/512 GB SSD.

[Christopher (Drashna)]

21 hours ago, feerlessleadr said:

1 more question on the SSD optimizer option - would I be able to use the same SSD that I'm using as a cache for the cloud drive as the SSD cache for the Z:\ pool? 

Yes.  Just keep in mind that you're splitting the usage between things, and you may see a performance hit for doing so. 

21 hours ago, feerlessleadr said:

 If so, is there a minimum size I should be looking for? I would be duplicating (initially anyway) about 24 TB worth of data, so not sure if it would really make a difference having a 128/256/512 GB SSD.

That depends on your usage.  I'm personally using a couple of 120GB SSDs and they work great for my needs. 

But if you want to use the CloudDrive cache on them, then the larger, the better.   As in, get the largest size you can reasonably justify buying.

[dburnnzz]

On 11/20/2017 at 3:07 PM, Christopher (Drashna) said:

Yes, absolutely! 
http://wiki.covecube.com/StableBit_DrivePool_Q4142489

for reads, update to the latest beta, it will do this automatically. 
http://dl.covecube.com/DrivePoolWindows/beta/download/StableBit.DrivePool_2.2.0.878_x64_BETA.exe

 

As for writes, there isn't a good way to do this, aside from the SSD Optimizer. 

The reason is that StableBit DrivePool writes all copies out, at the same time (real time duplication), so there isn't a way to really stop this.  However, it shouldn't be a problem, IMO. 

Hi  there, I know this is super old but I'm currently working on a very similar setup.

I have multiple drives in a pool currently. This is called "A:"

I just purchased CloudDrive (Assigned to "H:" and would like to get it setup to mirror that pool.

I  made another pool assigned "D:" which has "A:" in it, along with the cloud "H:". 

When i turn on duplication for "D:", nothing happens. It shows all the data in "A:" as "other". 

Any advice?

[dburnnzz]

52 minutes ago, dburnnzz said:

Hi  there, I know this is super old but I'm currently working on a very similar setup.

I have multiple drives in a pool currently. This is called "A:"

I just purchased CloudDrive (Assigned to "H:" and would like to get it setup to mirror that pool.

I  made another pool assigned "D:" which has "A:" in it, along with the cloud "H:". 

When i turn on duplication for "D:", nothing happens. It shows all the data in "A:" as "other". 

Any advice?

I think I actually figured it out... I followed that link for the moving of the data and reassigned the letters (So my Plex server didn't have to rescan all the media).

"D:" now shows the data available for duplication in the "A:" pool and I've set it up to not balance. Looks like it's duplicating to the "H:" CloudDrive now successfully!