• Create Account

Subpools or Drive Groups Within a Pool

25 replies to this topic

#21 Christopher (Drashna)

Christopher (Drashna)

Customer and Technical Support

• 8,208 posts
• LocationSan Diego, CA, USA

Posted 26 April 2017 - 11:23 PM

So DPCMD on the POP will show two devices which are in fact Pool1 and Pool2? Not the actualy underlying physical devices?

Yes.  I believe that is the case, I haven't had a chance to really play around with this, but based on how I know the software works... this should be the case.  (basically, it treats the pools like any other, normal disk)

I have just upgraded to 746. Changed the $recycle.bin to 1 x dup and checked the pool and folder duplication was still 2x for everything else. Also, the task failed error has gone as per the 746 change log. I am doing some logging at the moment so will post it through it I see any odd behavior. Edit: Actually, something I am seeing a lot is that the real time duplication doesn't seem to work. This may sound more drastic than it actually is. I am seeing large files copied or moved or overwritten in the pool and then the master pool will say, as an example 8.81GB is not duplicated, so it kicks off a duplication run, when it is finished I have 5xMB unduplicated. I am wondering it the <random file names>.copytemp from pool A and Pool B are being picked up when and if those pools balance themselves and if a balance in Pool A or Pool B will affect the Duplication in the Master Pool. I will post some logs when I have the time to work on this. Edit 2: Logs uploaded for continual duplication runs. If your'e able to reproduce this, could you run the "dpcmd X:\$RECYCLE.BIN", where "X:" is the top level pool.   both before and after, to verify the duplication level.

Ignore the list of folders, and look at the "expected number of copies" value.

If you can reproduce this, then enable file system logging, reproduce and then turn off logging. Upload the logs,

As for the real time duplication, I'll see about reproducing this later.

But could you use the "dpcmd list-pool-fileparts" command on the specific file, to verify if it's at both locations?

If that's having issues, then please do the above (enable logging, reproduce the issue, turn off logging and upload the logs).

As for "copytemp", this is what we use for balancing and duplication. This way, if something happens (bSOD, crash, sudden power loss, etc), that there isn't a partial file on the destination, leaving you with a corrupted file.  So if you're seeing this, it means "it's doing something".

Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

This is my server

#22 browned

browned

Member

• Members
• 20 posts

Posted 28 April 2017 - 07:42 PM

Ok. I am about to upload some more logs. This is the process I have been doing to cause the pool issues. My Setup again. Pool A and Pool B 7 disks each no duplication. Pool C contains Pool A and B with duplication enabled.

There have been files in the pool that according to drivepool have not been duplicated, I thought the directory structure might be to long so renamed a folder say from Stuff to St, note the contents of the folder was around 100k+ small files ranging from1kb to 500kb. This seems to kick off some events. The other time I have seen this process I was deleting a large directory structure full of 500k metadata files, txt, xml, jpg etc.

My visual monitoring suggests

- Pool A and B are fine.

- Pool C has issues, a duplication run starts.

- I see the unduplicated space growing.

- I check the folder duplication, even though I use pool duplication, random folders are set to 1x not 2x.

- I wait and eventually the unduplicated space stops growing.

- Checking the folder duplication again I see that all folders are back to 2x.

- Then duplication starts again after drivepool checking and saying it is not consistent.

- I have manually check the files and folder structure and see there are missing files and folders.

- After a day or so the data is duplicated again.

So it looks like large processes to delete or rename or move files cause strange events to happen with duplication settings. Hopefully someone can reproduce this issue as I think it has happened about 4 or more times now.

#23 browned

browned

Member

• Members
• 20 posts

Posted 02 May 2017 - 09:52 PM

Well after leaving the server for a few days it seems to have stopped the repeated duplication runs. There is only 4kb that is unduplicated now.

Does Drivepool support the Windows 2016 long path names feature, they have finally got rid of the 260 char limit.

Can I use DPCMD to find the "other" files on a in a pool or pool drive?

#24 Christopher (Drashna)

Christopher (Drashna)

Customer and Technical Support

• 8,208 posts
• LocationSan Diego, CA, USA

Posted 03 May 2017 - 07:02 PM

for the long file names, StableBit DrivePool doesn't care. We use UNC paths (\\?\Volume{GUID}\PoolPart.GUID2\path\to\content.txt), which has a 32k character limit.

the 260 character path limit is a limitation of the Win32 API, which explorer relies on.

As for the 4k remaining...

http://community.cov...pool-fileparts/

This should pick up what file is not duplicated.

Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

This is my server

#25 Umfriend

Umfriend

• Members
• 373 posts

Posted 09 May 2017 - 07:43 AM

So I moved to 2.2.0.746, all good. I now have two unduplicated Pools of single 4TB HDDs, partitioned as 2x2TB and a third Pool that consists of the two with duplication.

Big upside for me: the issue where duplicates can be stored on the same physical HDD is now "solved" (well, worked around). So because this now allows me to continue to use 2TB volumes (given the limitation of WHS2011 Server Backup) with larger Pools, I can keep WHS2011 for quite some time. This is a real money saver for me. Thanks!

I would have chosen a different implementation as I now need three Pools to accomplish this. I would rather have had the option to define strings within a single Pool. However, the current implementation has the benefit that it uses, I would think, sort of the same code as opposed to something new and the additonal overhead (I assume that for each I/O, the service now needs to make three calls) presumably is very small anyway.

• Christopher (Drashna) likes this

#26 Christopher (Drashna)

Christopher (Drashna)

Customer and Technical Support

• 8,208 posts
• LocationSan Diego, CA, USA

Posted 09 May 2017 - 09:50 PM

And yeah, the groups within the pool would be easier in a number of ways.  But this allows for a lot more flexibility without adding massive complexity to the balancing code.

As for the IO, its less of "calls", but yeah. The IO calls are redirected to the underlying disks (much like a reverse proxy for a web server).

Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

This is my server