Duplication and C:/ drive?

[hammy]

Hey everyone, 

New to the drivepool game here, absolutely loving this software. Has made storage and backup so much easier. Sorry if these are stupid questions here. 

1. I seem to be a bit confused how the "Folder duplication" process works. I have one of my movies folders set to duplicate, it's around 2.5tb. However, when I turn on duplication, the duplication across all my drives is around ~5.7tb. I don't have any other folders set for duplication, why would this figure be so high? I guess I was expecting duplication to be around the same size as the folder I was duplicating (2.5tb). Is the duplication figure the size of original folder plus the size of the duplication?

2. Am I able to safely add my C:/ drive to my pool? It's a 1TB SSD, I was hoping to harness some of its speed as a cache drive as it's mostly empty right now. Is this safe/possible? 

 

Thanks again for the phenomenal product, probably the happiest I've been with a software purchase in a long time. 

[Jaga]

What level of folder duplication did you choose?  2x, 3x, 4x, more?  It will determine the number of copies across your drives, and the resulting used space for duplicates.  If you enabled both folder/file duplication and drive duplication (on the entire pool) at the same time it complicates figuring out total space used.  More information about your level of duplication for everything would help to know.

While you can add the C drive to the pool (to take advantage of left-over space), you may have a hard time controlling the pool's use of that drive, and it may fill up the hidden poolpart folder more quickly than you want.  You can create file placement filters to try and help tell DrivePool what you want on there, to keep it under control.  The current contents of the C: drive will never show in a pool.

 

[Jaga]

8 minutes ago, hammy said:

Thanks so much for the help. I did 2x for the individual movies folder, everything else is at 1x (except the metadata folder at 3x). As far as I can see, file duplication is not on/I didn't turn it on. 

Would it be better to possibly partition my C drive then add that portion to the pool? Thanks again!

That's normal for the Metadata.  Sounds like your duplication is as expected, just taking more space than you'd hoped due to the metadata usage.

Partitioning is a great way to control use on a drive, yes.  If you can manage to shrink the C: volume enough, drop another partition on there solely for DrivePool.  OR you can consider that second SSD partition for use with the DrivePool SSD Optimizer plugin (to speed up the pool as a whole) (https://stablebit.com/DrivePool/Plugins).  It effectively adds the SSD partition to the Pool as the landing point for all new files (for faster access), then slowly offloads them to the rest of the pool over time and with use.  That's probably how I'd use the space if I were you.

[Jaga]

From what I understand, the metadata is simply information describing the system, it's files and the overall folder structure of your Pool, including the duplicated items.  With rounding numbers, it sounds close to what it should be.  The extra space may not even be used by the metadata - it could be a miscalculation by Drivepool on space used.

You can always force a re-calculation of duplicates by clicking the little gear in the upper right corner, choose "Troubleshooting", and then "Recheck duplication...".  If there were any inconsistencies, it may iron them out.

[Jaga]

12 minutes ago, hammy said:

It just seems odd that the "duplicated files" are so close to exactly double! It definitely doesn't count "duplicated" files twice (ie, once for their original file, and again for the duplicate counterpart)?  

Also, just curious, in your opinion is it actually worth it to have duplication enabled for read i/o? I have (at most) 5-6 people accessing my home server at once just so family can watch some movies off the drives. Worth it for that or is that not an intensive enough scenario for it to matter?

I'm definitely not the expert in duplication and metadata.  Perhaps one of the developers can check over the numbers and offer an opinion.  Normally when you set 2x duplication on a folder, once Drivepool -finishes- duplicating it, the files are *both* masters and completely identical (size, contents, permissions, flags, etc).

If you can spare the space for the duplicates (and it looks like you can), then read i/o can definitely help with performance.  If you had all 5-6 people hitting the same drive, and more than one was streaming UHD..  that drive would be under a bit too much stress.  Not a high likelihood, but possible.  And it's super easy to disable folder duplication later on, regaining the lost space.

[Christopher (Drashna)]

This may be relevant:

But basically, the "size" and "size on disk" are often not the same.

If you have a lot of small files on the disk, then this can happen. Especially if you formatted the disks with different "allocation unit" sizes.

 

[Christopher (Drashna)]

5 hours ago, hammy said:

2. Am I able to safely add my C:/ drive to my pool? It's a 1TB SSD, I was hoping to harness some of its speed as a cache drive as it's mostly empty right now. Is this safe/possible? 

Yes, you can.  It doesn't move the existing data, so you're set. 

However, I would generally recommend against it, since it is extra wear and tear on the drive. But if you're okay with that, then there should be no problem.

 

Additionally, if you plan on using the SSD Optimizer balancer plugin, if you're using duplication, you'd want to have 2 SSDs to really gain an advantage here. 

[hammy]

1 minute ago, Jaga said:

What level of folder duplication did you choose?  2x, 3x, 4x, more?  It will determine the number of copies across your drives, and the resulting used space for duplicates.  If you enabled both folder/file duplication and drive duplication (on the entire pool) at the same time it complicates figuring out total space used.  More information about your level of duplication for everything would help to know.

While you can add the C drive to the pool (to take advantage of left-over space), you may have a hard time controlling the pool's use of that drive, and it may fill up the hidden poolpart folder more quickly than you want.  You can create file placement filters to try and help tell DrivePool what you want on there, to keep it under control.  The current contents of the C: drive will never show in a pool.

 

Thanks so much for the help. I did 2x for the individual movies folder, everything else is at 1x (except the metadata folder at 3x). As far as I can see, file duplication is not on/I didn't turn it on. 

 

Would it be better to possibly partition my C drive then add that portion to the pool? Thanks again!

[hammy]

2 minutes ago, Jaga said:

That's normal for the Metadata.  Sounds like your duplication is as expected, just taking more space than you'd hoped due to the metadata usage.

Partitioning is a great way to control use on a drive, yes.  If you can manage to shrink the C: volume enough, drop another partition on there solely for DrivePool.  OR you can consider that second SSD partition for use with the [url=https://stablebit.com/DrivePool/Plugins]DrivePool SSD Optimizer[/url] plugin (to speed up the pool as a whole).  It effectively adds the SSD partition to the Pool as the landing point for all new files (for faster access), then slowly offloads them to the rest of the pool over time and with use.  That's probably how I'd use the space if I were you.

Jaga, thank you so much! So that's not entirely out of the ordinary for Metadata to be so large? My whole pool is around 36TB, almost all of it is taken up by uncompressed blurays, the folder I was trying to duplicate was just the compressed versions of films so I could stream them outside my home faster (with the read i/o boost).  

I will absolutely use that SSD optimizer, that's brilliant! I love it! Thanks so much for your help!

[hammy]

48 minutes ago, Jaga said:

From what I understand, the metadata is simply information describing the system, it's files and the overall folder structure of your Pool, including the duplicated items.  With rounding numbers, it sounds close to what it should be.  The extra space may not even be used by the metadata - it could be a miscalculation by Drivepool on space used.

You can always force a re-calculation of duplicates by clicking the little gear in the upper right corner, choose "Troubleshooting", and then "Recheck duplication...".  If there were any inconsistencies, it may iron them out.

It just seems odd that the "duplicated files" are so close to exactly double! It definitely doesn't count "duplicated" files twice (ie, once for their original file, and again for the duplicate counterpart)?  

Also, just curious, in your opinion is it actually worth it to have duplication enabled for read i/o? I have (at most) 5-6 people accessing my home server at once just so family can watch some movies off the drives. Worth it for that or is that not an intensive enough scenario for it to matter?

[hammy]

On 7/17/2018 at 5:42 PM, Christopher (Drashna) said:

Yes, you can.  It doesn't move the existing data, so you're set. 

However, I would generally recommend against it, since it is extra wear and tear on the drive. But if you're okay with that, then there should be no problem.

 

Additionally, if you plan on using the SSD Optimizer balancer plugin, if you're using duplication, you'd want to have 2 SSDs to really gain an advantage here. 

Christopher thank you so much! What is the benefit in having two SSDs? This is all super fascinating and I really appreciate the help. Just bought another suite of licenses for my PC, considering upgrading my whole office. Just love the software!

[Christopher (Drashna)]

On 7/17/2018 at 11:21 AM, hammy said:

It just seems odd that the "duplicated files" are so close to exactly double! It definitely doesn't count "duplicated" files twice (ie, once for their original file, and again for the duplicate counterpart)?  

Yeah. But there are always variations on how the files are placed, and such.  As well as metadata that you don't see. 
Also .... if you're using reparse points, and a LOT of them, .... that may be why too. 

On 7/17/2018 at 11:21 AM, hammy said:

 Also, just curious, in your opinion is it actually worth it to have duplication enabled for read i/o? I have (at most) 5-6 people accessing my home server at once just so family can watch some movies off the drives. Worth it for that or is that not an intensive enough scenario for it to matter?

Depends on the usage, but yeah, it can help with I/O load, as it may spread it out between the disks better. 
And the Read Striping feature helps to optimize the reads, too. 

19 hours ago, hammy said:

Christopher thank you so much! What is the benefit in having two SSDs? This is all super fascinating and I really appreciate the help. Just bought another suite of licenses for my PC, considering upgrading my whole office. Just love the software!

Duplicated data is written out to all destination disks.  So if you're using the SSD Optimizer balancer, and duplication, then you need two SSDs, or the second copy is written out to an "archive" drive, and may/will slow down the write speeds.    having 2x SSDs, (or ... a number of SSDs equal to the highest level of duplication that you're using) means that both/all copies are written out to the SSD drives, and not the archive drives. 

Real time duplication can be disabled, but that means that new files won't be protected by duplication for up to 24 hours!