DrivePool + Deduplication issue

[tomm1ed]

StableBit DrivePool DrivePool 2.2.0.701 Beta running on Windows 2012 R2 Standard with 2 Pools. 

 

One pool containing 4 harddisk partitions (on 4 separate harddisks) which works perfectly.

One pool containing 2 SSD partitions (on 2 seperate SSDs) with (VDI) deduplication enabled, Bypass fill system drivers is disabled.

 

Pool is configured to mirror one specific directory containing VHDx files which deduplicates quiet nicely:

FreeSpace    SavedSpace   OptimizedFiles     InPolicyFiles      Volume

---------    ----------   --------------     -------------      ------

190.59 GB    166.58 GB    9                  9                  E:

120.26 GB    166.38 GB    9                  9                  F:

 

Since the moment that {SavedSpace of E: and F:} > {FreeSpace of E: and F:} DrivePool no longer shows Statistics. It says 'Statistics are incomplete' and I am never able to rebalance anything because of this.

 

Both Read Striping and duplication is working so it seems it is just a cosmetical error.

I actually think I understand what is happening here, probably DrivePool does not expect there to be more data on the drive than can physically fit.

Is there any way to fix this issue?

[Christopher (Drashna)]

The issue with measuring should be fixed now.
 
However, the space reported is a problem.  This is because of how Deduplication works.  We had partial size reported, but .... 
 
There isn't a good way to handle deduplication here. Period.  Most of the data is stored outside of the pool folder strucuture, in the "System Volume Information" folder specifically.   So it's not actually pooled. 
 
Additionally, beacuse of how NTFS reports the data .... it's far from ideal. 
 
 

I fixed the dedup measurement issue in .699, but here's the problem: 

Deduplicated files don't report how much disk space they are actually taking up, so that's what the measurement pass sees. But it's not crashing anymore.
Technically speaking, that is actually "accurate", because any files that are 100% deduplicated are actually moved into System Volume Information, and only the reparse point exists on the pool part.
But this is misleading. If you place 2 identical files on the pool part, *both* of them will report 0 bytes used on the disk. This happens even though, technically, both files take up the space of what one files should take up (due to deduplication). But the MS filter reports the size as 0 for both files. So the size on disk statistics are actually incorrect.

 

 

So, this is a known issue, but there really isn't a good way to properly handle this.  It's partially a cosmetic issue, but it's partially a technical issue. And one we aren't really sure how to handle. 

 

For now, just keep in mind that deduplication will do this, but it should work fine otherwise. 

[XAlpha]

Ditto.  I had to undo all deduplication (that took about a week).  DP does not like dedup yet, so it would seem.

[tomm1ed]

The issue with measuring should be fixed now.

 

However, the space reported is a problem.  This is because of how Deduplication works.  We had partial size reported, but .... 

 

There isn't a good way to handle deduplication here. Period.  Most of the data is stored outside of the pool folder strucuture, in the "System Volume Information" folder specifically.   So it's not actually pooled. 

 

Additionally, beacuse of how NTFS reports the data .... it's far from ideal. 

 

 

 

 

So, this is a known issue, but there really isn't a good way to properly handle this.  It's partially a cosmetic issue, but it's partially a technical issue. And one we aren't really sure how to handle. 

 

For now, just keep in mind that deduplication will do this, but it should work fine otherwise. 

 

Logged back into my server and with 708 I am again getting statistics:

 

Great work Alex!

[Christopher (Drashna)]

Glad to hear it. 

 

And yeah, Alex has been hammering away at some of the bigger issues with StableBit DrivePool, including the measuring issues.  We've ran into some weird behavior. 

 

But glad to see that it's working (mostly) properly for you. 

 

It looks like you're entire pool isn't duplicated though, but you're not using "pool file duplication", so that should be fine.