Jump to content
  • 0

A question on DrivePool duplication


banderon

Question

Howdy,

I have a question on how DP's duplication works.

My setup:

  • Pool A: 4x (new) 8TB drives, no duplication, no mounting.
  • Pool B: ~20x (old and used) 750GB - 2TB range drives, no duplication, no mounting. Pool A ~ Pool B in terms of space.
  • Pool C: Pool A & Pool B, 2x duplication, mounted to a drive letter and network shared.
  • (pending): 4x (new) 8TB drives, offsite, with a nightly sync with Pool C.

I want full duplication, and I want to make use of all the old drives I have, but I also don't particularly trust the old drives. As such, I want a copy of data to exist on both the new drives and the old drives. If I'd just gone with one pool, there'd be the likely probability that some data would be mirrored across just two of the old drives, which have a higher chance of mutual failure than an old drive and one of the newer drives.

In the last week, I've had one of the old drives show SMART errors (that aren't particularly alarming, not enough to remove the drive from the pool), and another drive has apparently reported unreadable sectors.

It's the latter I'd like to ask about. Scanner is reporting the drive as "Damaged", and DP naturally started moving data off of the drive onto other drives in Pool B (it's nearly done at this point). The drive reports as having 2.5KB of missing data due to 5 unreadable sectors, and that this data is likely permanently lost; since Pool B itself is not duplicated, Scanner rationally sees any data that gets lost from this pool as gone forever. However, this data is in fact mirrored on Pool A (via the duplication of Pool C).

My question is this: what will DP do now, in regards to Pool C? Will it scan Pool A and Pool B, see the discrepancy in mirroring, and copy over the missing data from Pool A to Pool B? Or will it sync Pool A to match Pool B, and remove the 2.5KB of data on Pool A as well?

Link to comment
Share on other sites

5 answers to this question

Recommended Posts

  • 0
On 2/16/2018 at 9:55 AM, banderon said:

But I would imagine the a 512B missing chunk would render the encrypted container unopenable. I'm also not overly concerned since it's actually an outdated copy of another I have elsewhere, but this would be scary if it were an only copy of the file.

That depends on how the encryption works for the container.  Best case, it would corrupt a chunk of the data. Worst case, it would corrupt the entire thing, rendering it unusable. 

But this is why we do generally recommend enabling duplication.  It's when, not if a drives fails or exhibits issues.  Many refer to hard drives as "spinning rust" for this reason, actually. 

On 2/16/2018 at 9:55 AM, banderon said:

DP is currently still removing the data (it has about 60GB to go). However, the parent pool has already run a consistency check, and noticed a discrepancy. I was concerned because of the notice given on the parent pool (screenshot attached). It makes it sound like DP had detected the inconsistency in the files, and its desired solution is to remove the missing file parts from the mirror as well, rather than to replace the missing part from the mirror.

Ah, yeah, I can understand that being a but confusing.  Specifically, it deletes the the older part, and then reduplicates the data.  But that should be made clearer. 

(https://stablebit.com/Admin/IssueAnalysis/27743)

Link to comment
Share on other sites

  • 0
10 hours ago, banderon said:

I want full duplication, and I want to make use of all the old drives I have, but I also don't particularly trust the old drives. As such, I want a copy of data to exist on both the new drives and the old drives. If I'd just gone with one pool, there'd be the likely probability that some data would be mirrored across just two of the old drives, which have a higher chance of mutual failure than an old drive and one of the newer drives.

Hierarchical pooling is the way to do this, as it was designed for this sort of configuration specifically. (not necessarily, good vs bad drives, but to control where duplication ends up). 

Otherwise, in a single pool, there wasn't a good way to implement this without a massive code change. 

10 hours ago, banderon said:

In the last week, I've had one of the old drives show SMART errors (that aren't particularly alarming, not enough to remove the drive from the pool), and another drive has apparently reported unreadable sectors.

Unreadable sectors are a problem, and any drive displaying this should be decommissioned. Uncorrectable Sector Count is even worse.

10 hours ago, banderon said:

It's the latter I'd like to ask about. Scanner is reporting the drive as "Damaged", and DP naturally started moving data off of the drive onto other drives in Pool B (it's nearly done at this point). The drive reports as having 2.5KB of missing data due to 5 unreadable sectors, and that this data is likely permanently lost; since Pool B itself is not duplicated, Scanner rationally sees any data that gets lost from this pool as gone forever. However, this data is in fact mirrored on Pool A (via the duplication of Pool C).

If the drives is NTFS formatted, run the file recovery option.  This will tell you if that damage is in unused space, or part of a file.  And if it's part of a file, it will tell you which.   

And worst case, it can attempt to recover the file, as well.  But if it's duplicated .... then let the balancing finish (clearing out the drive) and then physically pull it from the system.  Then remove the now "missing" disk from the system.  The software will then check the duplication status and reduplicate data, as needed. 
(you may need to power off the system to remove the disk physically)

10 hours ago, banderon said:

My question is this: what will DP do now, in regards to Pool C? Will it scan Pool A and Pool B, see the discrepancy in mirroring, and copy over the missing data from Pool A to Pool B? Or will it sync Pool A to match Pool B, and remove the 2.5KB of data on Pool A as well?

Yes.  Once the disk is removed, it runs a "pool consistency" check, specifically.  This checks the duplication status, and the balancing config.  From there, it will take whatever action is needed, up to and including reduplicating data, or rebalancing the pool. 

Link to comment
Share on other sites

  • 0

Yup, this is why I created a hierarchical pool. That feature being added a few months back was what pushed me towards DrivePool because I wanted to be able to create this exact setup. I was just explaining the reasoning so that the architecture makes sense.

The drives are all NTFS. I was given the option to attempt recovery of the files. A scan found a 512B chunk missing from a 50GB encryption container, and a 2.00KB chunk missing from a 7GB video. The video is replaceable if anything, so not worried about that. But I would imagine the a 512B missing chunk would render the encrypted container unopenable. I'm also not overly concerned since it's actually an outdated copy of another I have elsewhere, but this would be scary if it were an only copy of the file.

DP is currently still removing the data (it has about 60GB to go). However, the parent pool has already run a consistency check, and noticed a discrepancy. I was concerned because of the notice given on the parent pool (screenshot attached). It makes it sound like DP had detected the inconsistency in the files, and its desired solution is to remove the missing file parts from the mirror as well, rather than to replace the missing part from the mirror.

 

Screen Shot 2018-02-16 at 12.52.30.png

Link to comment
Share on other sites

  • 0

Yup, the ease of duplication was one of the main reasons I was attracted to DrivePool. Also, thank you; that clears things up: the "older parts" that will be deleted are the remaining parts of the damaged files, so that the intact copies can be re-replicated.

Edit: I was going to mark your reply as the "best answer", but due to the architecture of this forum, that takes it out of the flow of the thread to be displayed at the top which I imagine would make it difficult to follow the conversation for anyone else. So I'll leave things as they are. Thanks!

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...