Jump to content
  • 0

check-pool-fileparts


Alex

Question

If you're not familiar with dpcmd.exe, it's the command line interface to StableBit DrivePool's low level file system and was originally designed for troubleshooting the pool. It's a standalone EXE that's included with every installation of StableBit DrivePool 2.X and is available from the command line.
 
If you have StableBit DrivePool 2.X installed, go ahead and open up the Command Prompt with administrative access (hold Ctrl + Shift from the Start menu), and type in dpcmd to get some usage information.
 
Previously, I didn't recommend that people mess with this command because it wasn't really meant for public consumption. But the latest internal build of StableBit DrivePool, 2.2.0.659, includes a completely rewritten dpcmd.exe which now has some more useful functions for more advanced users of StableBit DrivePool, and I'd like to talk about some of these here.
 
Let's start with the new check-pool-fileparts command.
 
This command can be used to:

  • Check the duplication consistency of every file on the pool and show you any inconsistencies.
  • Report any inconsistencies found to StableBit DrivePool for corrective actions.
  • Generate detailed audit logs, including the exact locations where each file part is stored of each file on the pool.

Now let's see how this all works. The new dpcmd.exe includes detailed usage notes and examples for some of the more complicated commands like this one.
 
To get help on this command type: dpcmd check-pool-fileparts
 
Here's what you will get:
 

dpcmd - StableBit DrivePool command line interface

Version 2.2.0.659

The command 'check-pool-fileparts' requires at least 1 parameters.


Usage:

  dpcmd check-pool-fileparts [parameter1 [parameter2 ...]]

Command:

  check-pool-fileparts - Checks the file parts stored on the pool for consistency.

Parameters:

  poolPath - A path to a directory or a file on the pool.
  detailLevel - Detail level to output (0 to 4). (optional)
  isRecursive - Is this a recursive listing? (TRUE / false) (optional)

Detail levels:
  
  0 - Summary
  1 - Also show directory duplication status
  2 - Also show inconsistent file duplication details, if any (default)
  3 - Also show all file duplication details
  4 - Also show all file part details

Examples:
  
  - Perform a duplication check over the entire pool, show any inconsistencies, and inform StableBit DrivePool
    
    >dpcmd check-pool-fileparts P:\

  - Perform a full duplication check and output all file details to a log file

    >dpcmd check-pool-fileparts P:\ 3 > Check-Pool-FileParts.log

  - Perform a full duplication check and just show a summary

    >dpcmd check-pool-fileparts P:\ 0

  - Perform a check on a specific directory and its sub-directories

    >dpcmd check-pool-fileparts P:\MyFolder

  - Perform a check on a specific directory and NOT its sub-directories

    >dpcmd check-pool-fileparts "P:\MyFolder\Specific Folder To Check" 2 false

  - Perform a check on one specific file

    >dpcmd check-pool-fileparts "P:\MyFolder\File To Check.exe"

The above help text includes some concrete examples on how to use this commands for various scenarios. To perform a basic check of an entire pool and get a summary back, you would simply type:
dpcmd check-pool-fileparts P:\
 
This will scan your entire pool and make sure that the correct number of file parts exist for each file. At the end of the scan you will get a summary:

Scanning...
 
! Error: Can't get duplication information for '\\?\p:\System Volume Information\storageconfiguration.xml'. Access is denied
 
Summary:
  Directories: 3,758
  Files: 47,507 3.71 TB (4,077,933,565,417 B)
  File parts: 48,240 3.83 TB (4,214,331,221,046 B)
 
  * Inconsistent directories: 0
  * Inconsistent files: 0
  * Missing file parts: 0 0 B (0 B)
 
  ! Error reading directories: 0
  ! Error reading files: 1

Any inconsistent files will be reported here, and any scan errors will be as well. For example, in this case I can't scan the System Volume Information folder because as an Administrator, I don't have the proper access to do that (LOCAL SYSTEM does).
 
Another great use for this command is actually something that has been requested often, and that is the ability to generate audit logs. People want to be absolutely sure that each file on their pool is properly duplicated, and they want to know exactly where it's stored. This is where the maximum detail level of this command comes in handy:
dpcmd check-pool-fileparts P:\ 4
 
This will show you how many copies are stored of each file on your pool, and where they're stored.
 
The output looks something like this:

Detail level: File Parts
 
Listing types:
 
  + Directory
  - File
  -> File part
  * Inconsistent duplication
  ! Error
 
Listing format:
 
  [{0}/{1} IM] {2}
    {0} - The number of file parts that were found for this file / directory.
    {1} - The expected duplication count for this file / directory.
    I   - This directory is inheriting its duplication count from its parent.
    M   - At least one sub-directory may have a different duplication count.
    {2} - The name and size of this file / directory.
 
... 

+ [3x/2x] p:\Media
  -> \Device\HarddiskVolume2\PoolPart.5823dcd3-485d-47bf-8cfa-4bc09ffca40e\Media [Device 0]
  -> \Device\HarddiskVolume3\PoolPart.6a76681a-3600-4af1-b877-a31815b868c8\Media [Device 0]
  -> \Device\HarddiskVolume8\PoolPart.d1033a47-69ef-453a-9fb4-337ec00b1451\Media [Device 2]
    - [2x/2x] p:\Media\commandN Episode 123.mov (80.3 MB - 84,178,119 B)
      -> \Device\HarddiskVolume2\PoolPart.5823dcd3-485d-47bf-8cfa-4bc09ffca40e\Media\commandN Episode 123.mov [Device 0]
      -> \Device\HarddiskVolume8\PoolPart.d1033a47-69ef-453a-9fb4-337ec00b1451\Media\commandN Episode 123.mov [Device 2]
    - [2x/2x] p:\Media\commandN Episode 124.mov (80.3 MB - 84,178,119 B)
      -> \Device\HarddiskVolume2\PoolPart.5823dcd3-485d-47bf-8cfa-4bc09ffca40e\Media\commandN Episode 124.mov [Device 0]
      -> \Device\HarddiskVolume8\PoolPart.d1033a47-69ef-453a-9fb4-337ec00b1451\Media\commandN Episode 124.mov [Device 2]

...

The listing format and listing types are explained at the top, and then for each folder and file on the pool, a record like the above one is generated.
 
Of course like any command output, it could always be piped into a log file like so:
dpcmd check-pool-fileparts P:\ 4 > check-pool-fileparts.log
 
I'm sure with a bit of scripting, people will be able to generate daily audit logs of their pool :)

 

Now this is essentially the first version of this command, so if you have an idea on how to improve it, please let us know.

 

Also, check out set-duplication-recursive. It lets you set the duplication count on multiple folders at once using a file pattern rule (or a regular expression). It's pretty cool.

 

That's all for now.

Link to comment
Share on other sites

12 answers to this question

Recommended Posts

  • 0

Does it also do a binary comparison of the file parts, or does it only check, if the file is there and timestamp and size match?

This utility doesn't check. It's mostly just enumerating the files in the pool. 

 

However, as for how duplicated files are handled:

When it does a duplication pass, or when files are access, It does a timestamp and size comparison. If these matches, it moves on. Otherwise, it runs a checksum on both parts. 

Link to comment
Share on other sites

  • 0

Could you please add an option to BINARY compare the files for integrity? This possiblilty was also often discussed in other threads since a couple of months. Currently there is no means of really verifying integrity.

If I have 3x duplication and one file is differs from the other, then it is easy to have the third copy replaced by the others ones. Also with 2x duplication a conflict should be detectable.

 

I also had the problem, that I copied some 100 GB of data from one disk to another and a binary comparison afterwards showed differences. This problem cannot be detected by DrivePool currently and should be fixed, at least with the command line option and even better in the management interface, to have a scheduled full or partial verification.

Link to comment
Share on other sites

  • 0

Well, it does do a binary compare if the other information does not match.

 

However, doing a binary compare of the entire pool.... not only will this be very disk and CPU intensive, it may take days or even weeks to complete, depending on the size of your pool and how many files exist in it. 

That is why we do not currently do this by default.  Because it would mean on larger pools, any time a remeasure is initiated, it could be weeks potentially before the balancing data is correct. 

 

 

However, this entire topic is something that absolutely has been on our mind, and something we do want to resolve and handle in a better way. But it means a significant overhaul to StableBit DrivePool, or another bit of software.  So, we've been pushing it off until we have at least a release candidate for StableBit CloudDrive.  And that's part of why the "patch job" for the auditing. We needed something, so alex whipped up something quickly. 

 

 

 

I also had the problem, that I copied some 100 GB of data from one disk to another and a binary comparison afterwards showed differences. This problem cannot be detected by DrivePool currently and should be fixed, at least with the command line option and even better in the management interface, to have a scheduled full or partial verification.

Again, this is something we definitely plan on addressing. It's just not going to be simple. It's going to be a lot of coding involved in doing so. 

 

 

However, if you're experiencing differences in file content when copying from drives, that's another issue altogether, and indicates a bigger problem. If this is happening when copying to the pool, then please do let us know.

 

Also, if this ever happens, the first thing to check is the file system filters on the system. These intercept all IO operations on all disks normally (they can be bypassed, and the pool driver does when writing to the pooled disks), and can modify these operations.  This is actually how real time antivirus protection works, and how pre-emptive defragmentation works. But it can cause issues, if the filter doesn't work 100% right.  This could absolutely account for the issue you are/were seeing.

 

But regardless, we definitely understand your concern about this, and we definitely want to improve how integrity is handled.

Link to comment
Share on other sites

  • 0

Hi,

 

Just installed the Beta (v2.2.651), but my "dpcmd" doesn't have the "check-pool-fileparts" option.

 

Am I missing something? Thanks...

 

Version 2.2.0.651


Usage:


  dpcmd [command] [parameter1 [parameter2 ...]]


Commands:


  get-duplication - Outputs duplication related information about the specified path.
    parameter1 - Path to a file or folder on the pool.


  set-duplication - Sets the duplication count for a folder.
    parameter1 - Path to a folder on the pool.
    parameter2 - The duplication count (>= 1).


  list-poolparts - Lists all of the pool parts that make up a pool.
    parameter1 - Path to the pool.


  add-poolpart - Tries to add a pool part to the pool from a path to a volume.
    parameter1 - Path to a volume.
    parameter2 - Path to the pool or leave blank to create a new pool.


  hint-poolpart - Sends a hint to the pooling file system to check this volume for a pool part.
    parameter1 - Path to the volume.
Link to comment
Share on other sites

  • 0

Upgrade to 2.2.0.659.

http://dl.covecube.com/DrivePoolWindows/beta/download/

 

 

.659

* Added usage information and examples for set-duplication-recursive.

* Added check-pool-fileparts to dpcmd.exe. It's able audit your duplicated files and find inconsistencies.
    - For usage and examples: "dpcmd check-pool-fileparts"
.655
* [issue #21848] Rewrote dpcmd.exe to accommodate for a more structured command module layout and added a bunch of new 
                 commands (requires command prompt with Administrative rights).
    - Added:
        - ignore-poolpart - Immediately disconnects a pool part from the pool and prevents it from participating in 
                            future pools by placing an ignore tag on it.
        - refresh-all-poolparts - Forces the re-enumeration of all the pool parts and pools.
        - remeasure-pool - Recomputes a pool's file usage statistics.
        - set-duplication-recursive - Recursively sets the duplication count on one or more folders on the pool matching a 
                                      pattern (regex or file pattern).
        - unignore-poolpart - Removes the ignore tag from a pool part, and re-adds it to the pool.
Link to comment
Share on other sites

  • 0
16 hours ago, Christopher (Drashna) said:

It would need to export in a CVS type format,  which it doesn't, currently. 

Well, at the moment output is sorted by folder, I thought it shouldn't be too hard to have that changed to hard disk?..

Link to comment
Share on other sites

  • 0
On 5/4/2018 at 12:39 PM, siulas said:

Well, at the moment output is sorted by folder, I thought it shouldn't be too hard to have that changed to hard disk?..

This would be quite a useful feature.  When i lose a disk - this would be a great way to know what was on a specific disk...

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...