Lingering files in PoolPart after a successful Removal from Pool?

[vfsrecycle_kid]

Hi folks,

 

I have a 40TB NAS and due to some interest SMART results on 2 drives, I've taken the precaution to remove them formally through DrivePool, and pass the data off to the other drives in the pool.

 

(2x Global Duplication, 3x Duplication for "My Photos")

 

The first drive of the 2 has finished the Removal Process, but it seems that it has left maybe ~20 folders (with a bunch of files) in the now unhidden PoolPart folder.

 

I performed a simple SHA1 check on 1 of the files and compared it to the version that is still in the pool and it is an exact match. So I'm curious as to why it is seemingly random that a few files weren't removed from the removed drive.

 

The PoolPart folder on the removed drive is around 400GB, whereas it was originally 4TB.

 

Any insight as to what happened? I don't want to delete this folder unless I'm certain that this 400GB of content exists in the pool still, but not sure how to automatically compare.

 

Thanks!

 

edit: Just a thought, could it be the files were somehow in use so DrivePool merely copied to another drive them instead of moving them?

edit2: Same thing with the 2nd removed drive. Once removal completed around 400GB of the original 4TB remains on the drive. The contents appear to be in the pool as expected, as well.

[dannybuoy]

I had the same thing recently. Most of the files I could get away with moving manually. A small percentage however gave me access denied when trying to read them, I assume they were corrupt as changing the owner/permissions did nothing to help. Whether that is the fault of Drivepool I do not know, but I had an instance where I plugged in a new drive, balanced the files then remove the disk - and was left with a couple of corrupt unreadable files. I'm now in two minds whether to continue using this solution... But at the moment I have Snapraid set up and auto balancing disabled, so I should at least be able to repair any silent corruption in future.

[vfsrecycle_kid]

I had the same thing recently. Most of the files I could get away with moving manually. A small percentage however gave me access denied when trying to read them, I assume they were corrupt as changing the owner/permissions did nothing to help. Whether that is the fault of Drivepool I do not know, but I had an instance where I plugged in a new drive, balanced the files then remove the disk - and was left with a couple of corrupt unreadable files. I'm now in two minds whether to continue using this solution... But at the moment I have Snapraid set up and auto balancing disabled, so I should at least be able to repair any silent corruption in future.

 

Not quite sure if this is the same issue I am encountering, but is useful to know so it is appreciated. I have yet to confirm if any files are corrupt on either the pool end or either of the removed drives.

 

Currently I'm manually running a checksum comparison between Removed Drives and the Pool for any missing stuff. Should be done in around 18 hours. I'm basically using Teracopy with the Verify Function ONLY (No move, no copy) to compare Source (Removed Drive 1 + Removed Drive 2) == Target (Drive Pool).

 

However a new interesting discovery, both removed drives have the SAME ~400GB in them, is this a telling sign?

[Christopher (Drashna)]

Are you using any of the options for removal (force damaged disk removal, or duplicate data later)?

 

Also, how many disks are in your pool? 

 

Also, what version of StableBit DrivePool are you using specifically?

 

 

And could you grab the log files from the system? 

http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

Just do steps #7-9

[vfsrecycle_kid]

Version: 2.1.1.561

 

Drives in Pool:

4x8TB

2x6TB (these are the ones that were removed)

 

Options:

No forced removal

No duplicate data later

(Aka, both unchecked)

 

The second drive to be removed was "queued for removal" in DrivePool

 

See attached logfiles. Removal started around Sep5.

 

I have just finished my checksum and everything outside of the pool on those 2 drives matches the 400GB of data inside of the pool (CRC).

[Christopher (Drashna)]

Okay, thanks for the info!

 

To clarify, one drive was set to be removed, and while that was happening, you queued the second drive to be removed? 

 

And the left over data was on both drives? Or the queued one? or?

 

 

 

As for the logs, I do see two things.... 

DrivePool.Service.exe Warning 0 [RemoveDriveFromPool] Last write time mismatch with file on pool. 06/01/2007 00:09:29 vs 03/12/2015 00:49:37 for '\\?\Volume{769f9718-65d7-4e11-aa7d-e2aa91f0668c}\PoolPart.870d1db5-e45c-4155-8ea3-1e33ef73ed91\[removed].jpg'. 2016-09-06 12:59:35Z 3438878361084
DrivePool.Service.exe Warning 0 [RemoveDriveFromPool] Last write time mismatch with file on pool. 06/05/2015 23:27:55 vs 06/05/2015 23:31:48 for '\\?\Volume{769f9718-65d7-4e11-aa7d-e2aa91f0668c}\PoolPart.870d1db5-e45c-4155-8ea3-1e33ef73ed91\[removed].flac'. 2016-09-06 13:05:13Z 3439668949777
DrivePool.Service.exe Warning 0 [RemoveDriveFromPool] Last write time mismatch with file on pool. 06/05/2015 20:30:53 vs 06/05/2015 20:34:42 for '\\?\Volume{769f9718-65d7-4e11-aa7d-e2aa91f0668c}\PoolPart.870d1db5-e45c-4155-8ea3-1e33ef73ed91\[removed].jpg'. 2016-09-06 13:05:15Z 3439673298905
DrivePool.Service.exe Warning 0 [RemoveDriveFromPool] Last write time mismatch with file on pool. 06/05/2015 20:51:06 vs 06/05/2015 20:50:51 for '\\?\Volume{769f9718-65d7-4e11-aa7d-e2aa91f0668c}\PoolPart.870d1db5-e45c-4155-8ea3-1e33ef73ed91\[removed].jpg'. 2016-09-06 13:05:15Z 3439673631263

I removed the path/file names from the files here, but these did pop up.  

However, I don't think that this would cause the issue you were seeing.  

 

 

But this would: 

DrivePool.Service.exe Warning 0 [RemoveDriveFromPool] Error on final delete after drive removal for \\?\Volume{769f9718-65d7-4e11-aa7d-e2aa91f0668c}\PoolPart.870d1db5-e45c-4155-8ea3-1e33ef73ed91\. Unable to delete all files after removing pool part. 2016-09-06 13:05:25Z 3439698062727
DrivePool.Service.exe Warning 0 [RemoveDriveFromPool] Error on final delete after drive removal for \\?\Volume{d6510e8b-adc3-40d6-88e6-45a8875aba42}\PoolPart.a24745dd-66bd-4b08-a5c5-8b50cfa09580\. Unable to delete all files after removing pool part. 2016-09-07 04:44:21Z 3571735118268

By chance did attempt to delete anything off of the pool yourself? 

 

If so, that may be why, as it creates a "bad state" in some cases. An issue that has been fixed in later builds already (public beta build).

 

If not, it may be that the move itself has triggered the above behavior. 

 

 

 

Either way, in this case, what you're seeing is normal, and would be expected.  however, yes, we should have better feedback about what is going on here. 

[vfsrecycle_kid]

Hello,

 

Thanks for the feedback. 

 

You are correct, while the first drive was mid-removal (lets say 1% or something), I queued up the second drive to be removed.

 

I was not actively removing anything from the pool, but that is not to say something automated/service-like was accessing something on the system.

 

And also to clarify:

The lingering 400GB on both removed drives were equivalent, and match CRC on the pool post-removal.

 

My initial issue was I didn't want to delete the PoolPart folders on the 2 removed drives unless I was 100% sure they were still persisted within the pool. After my own check it is clear that everything is good, and I can clean up the two newly removed drives.

 

I'm currently out of the country and since removals are rare, I'm not going to attempt to upgrade to the beta version remotely. So hopefully all is well.

 

Thanks for the insight.

[Christopher (Drashna)]

I don't the queued removal is the issue here, but it's something to test out, just in case. 

 

To be honest, the "Unable to delete all files after removing pool part."  is the issue here. It looks like it wasn't able to properly clean up the disk.   

 

You mention that the CRC comparison shows that the data is still in the pool. That seems to confirm that everything was copied from the disk, but just didn't get cleaned up properly.   (and whenever possible, we err on the side of caution, so that if data loss may occur, we take the option that should reduce the likelihood of it happening).

[vfsrecycle_kid]

Sounds good. Thanks for the background info.

 

Successfully pulled out and formatted the two removed drives, and replaced them with 2 new 8TBs. No issues to report.

 

Thanks again for the great product. Made a potentially scary problem very easy to handle.