Drivepool keeps checking and duplicating

[btb66]

I swapped out a drive for one with a higher capacity, I used the 'remove drive' option, when complete I then added the larger capacity drive to the pool. Initially it took time to repopulate the new drive and now the pool doesn't seem to have settled and it keeps checking, then duplicating, after duplication has finished it shows the pool is good only for a short time, about ten minutes I would say, then the cycle repeats, checking again and more duplicating, over and over.

 

I've attached a screenshot of some notifications, (only these few because I cleared them after reading earlier).

 

The pool seems to fine except for this issue, I'm fairly sure before swapping the drive this didn't happen, is there anything I can do?

 

I'm using beta v. 2.2.0.765

 

Edited to add another screenshot, cycle continues.

[Christopher (Drashna)]

Could you run this: 

http://wiki.covecube.com/StableBit_Troubleshooter

[btb66]

Thank you, I've done that and added a link to this thread.

[browned]

Mine does the same, Got 24 notifications from the 24 checks done every 1 hour for the last day.

[Christopher (Drashna)]

Thank you, I've done that and added a link to this thread.

 

Not setting it, but I think there is an issue on our end, so let me get back to you on that. 

 

 

Mine does the same, Got 24 notifications from the 24 checks done every 1 hour for the last day.

 

What version are you on, specifically?

[Spider99]

Getting a lot of "Ensure file duplication consistency.." on one pool every hour or so and also it keeps having unduplicated data - from a few hundred meg to 6GB - varies which drive its on when checking each drive from main GUI

 

pool is set to full duplication and all folders show x2

 

did not have this on 742 just updated to 769 in last couple of days when this started

 

new bug?

 

Also need a feature pronto to actually show which are not duplicated so you can check as the program knows 

 

Have run troubleshooter and its submitting now

[Christopher (Drashna)]

It could be.   We did change some significant stuff, so there is an increased likelihood of issues. 

 

That said, please rerun the troubleshooter.  There was an issue with it, but it should be fixed now.  

(the storage backend ..... ugh, but it's fixed now).

 

Also need a feature pronto to actually show which are not duplicated so you can check as the program knows 

 

Have run troubleshooter and its submitting now

 

Well, the duplication settings page calculates this in real time, so you can use that.

 

Otherwise, this:

http://community.covecube.com/index.php?/topic/1587-check-pool-fileparts/

[Spider99]

Chris

 

currently its not showing any unduplicated - but i guess it will be back if its a bug - do you still want me to re run the troubleshoot even if its not showing at the moment or wait?

 

"Well, the duplication settings page calculates this in real time, so you can use that." - could you explain a bit more please? not sure i follow

[Christopher (Drashna)]

Wait until the issue comes back.   

 

that or if you can trigger it to occur, enable drive tracing, and repro it. 

 

 

As for the second part,  in the "Folder Duplication" window, it will scan the folder selected and display it's status (how much data, and display it, rather than relying on the previously measured data). 

[btb66]

I ran the troubleshooter again and linked it to this thread, but its resolved now.

 

In my case it was down to a lot of writes to the pool, at the same time as adding more capacity I also moved a folder to the pool which is constantly written to, I've now moved it out again and the checking/duplicating has stopped and the pool has settled.

[Spider99]

Actually my issue was also after a large number of files were amended on the pool - editing, copying and moving thousands of photos (via a script - powershell)

 

I wonder if in this situation the duplication on Dp falls behind and causes the checking and duplication

 

This would be odd as my ssd cache is two samsung 850 pro's and the total volume of data amended was only a few GB

[Christopher (Drashna)]

I ran the troubleshooter again and linked it to this thread, but its resolved now.

 

In my case it was down to a lot of writes to the pool, at the same time as adding more capacity I also moved a folder to the pool which is constantly written to, I've now moved it out again and the checking/duplicating has stopped and the pool has settled.

 

Odd.

 

Well, flagged for Alex, anyways.  

https://stablebit.com/Admin/IssueAnalysis/27560

 

 

 

 

Actually my issue was also after a large number of files were amended on the pool - editing, copying and moving thousands of photos (via a script - powershell)

 

I wonder if in this situation the duplication on Dp falls behind and causes the checking and duplication

 

This would be odd as my ssd cache is two samsung 850 pro's and the total volume of data amended was only a few GB

 

Well, these should happen "in parallel", so it should really trigger this to occur.

 

If I had to guess, one or more of the disks are not showing up in time, or "disappearing" temporarily, triggering this to occur.   But I'm not certain about that, which is part of why I've flagged the issue for Alex.   He is the developer, so he should have a better idea. 

[Spider99]

Chris - no missing drives or drives going missing

 

server had been on for several days with no issues - numerous backups to the pool worked fine - not large changes

 

run a lot of changes - can trigger the issue though - then balancing fails to work correctly - or runs but does not move any files off the ssd's?

[Spider99]

Ok i have unduplicated files again 363MB on disk 02 of my e: pool

 

Also the ssd's are unbalanced - one has 4gb of duplicated files the other 18.5GB

 

a duplication run is in progress - but the ssd imbalance was there before the run started - so its not that some has been moved off

 

see image

 

 

 

Scanner sees both the ssd's as healthy

 

Lol - the unduplicated just went up to 444MB while doing the balance now affecting disks 04,09,10,11,13,14,15,16,17,18 !!! which were fine before

 

Duplicating is progressing (very slowly) and although the % is increasing nothing appears to be being moved.

 

I will wait for it to finish the balance and then run the troubleshooter and report back

[Spider99]

Ok it finished duplicating and i now have over 7GB of unduplicated files!

 

It wanted to duplicate again but i aborted it and am running the trouble shooter now - so should finish uploading in 10-20mins

 

The balance slowed down for ages then got to about 90% and finished in 10-20 seconds - BUT it moved no files - ssd's still unbalanced as in screenshot from previous post

 

unduplicated files are now across all but two of 20 drives!

 

once the info has finished uploading i will try another balance and if that does not work a remeasure - which has fixed it in the past.

 

If it comes back i think i will move back to 742 where i was before

[Spider99]

Troubleshooter info has been submitted

[lee1978]

Have you tried running the dos commands I can't for the life of me remember what's it's called but you can get it to check duplication and it will list the files not duplicated and it's fairly quick

 

Edit "DPCMD"

[Spider99]

I now have a pool that i have remeasured - no change on the undup files

 

if i run a balance it thinks about it for a few seconds then stops (no error) as it thinks its balanced when it is far from it!!!

yep good idea will re run that to see if it agrees with the GUI

[Spider99]

ok have run the dpcmd check-pool-fileparts on my e:\ pool with option 0 

 

its found 1,289 missing file parts!!!

 

i ran this two days ago and it found none.

 

going to run it again with more detail to see whats unduplicated

[Spider99]

Ok DPCmd has found some of the files

 

The odd thing is these files that were duplicated two days ago are now unduplicated - i have a full dpcmd log from the 13/7/2017.

 

The files (photos) have not even been looked at by me since November last year - checked last access date

 

So is DP - deleting files?

So is DP - losing files?

 

the directories which contain the files exist on 4x to 14x drives according the dpcmd - so i doubt its a hardware issue

 

All the disks are fine - nothing in error logs - scanner happy etc

 

This is worrying if we cant trust DP to be consistent over a couple of days

[Spider99]

this is wierd i have run two dpcmd runs

 

and the files unduplicated are different

 

the summary level 0 has 1289 files highlighted and the level 2 has 262 - total size is slightly different

 

i am going to re run the summary one again

[Spider99]

ok second run agrees with the level 2 run - so a sneaky balance must have happened in between

 

i have done a remeasure with no change

 

duplication runs for a few seconds then terminates

 

doing a re -check now to see if that does anything

[Spider99]

phewwwwwwwwwwwwwwwwwwwwwww

 

and relax

 

the re-check appears to have worked and the pool is rebalanced again - although it has not finished yet - and still have data on my ssd's

 

Looks like it maybe a dp cache thing then?

 

Chris - does the dpcmd - option to flush the dp cache - do the same thing as a "re-check" as this option does not always appear

Edited July 16, 2017 by Spider99

[Spider99]

ok by resetting my settings for DP - i finally got DP to balance properly again - took a very long time for 20gb to finally move off my ssd

 

i have only the ssd plugin loaded and all default balances off apart from scanner

 

have reset my settings and so far things appear back to normal

 

Chris - any comments???

[Spider99]

Could you run this: 

http://wiki.covecube.com/StableBit_Troubleshooter

Chris - any news from the upload from a couple of days ago?

[Christopher (Drashna)]

Well, the last bit sounds like the balancing may have gotten stuck/corrupted, and resetting the settings would have fixed that.

 

Newer versions should be much better about not getting into "bad states" like this.  But if this was on a newer version, then please do confirm that. 

 

 

Chris - any news from the upload from a couple of days ago?

 

Not yet. Sorry.