Spider99
-
Posts
544 -
Joined
-
Last visited
-
Days Won
39
Posts posted by Spider99
-
-
Ah - i just did a test copy of a large file to the pool
and the file was "saved" to a HDD not the ssd's - hence why i am not getting duplication - real time duplication is enabled in the gui
so checked config page of ssd optimiser and the two ssd's are defined as ssd's - hmm
a bit more testing is needed - will report back
-
I can almost create this at will now
I ran a simple script to update the "title" field in a few thousand MKV files
now i have over 400GB of "unduplicated files" - checking the logs - no issues for these files - they are not listed (unlike some others that i spotted earlier - which i have cleaned up)
Doing a remeasure at the moment to confirm (looking like it) the unduplicated files
Question - why are the error logs encrypted/compressed for DP? other logs are not.
-
if you create a folder by hand - i.e. not a copy or script
is it also slow to appear?
-
this is all on v769
doubt very much its a permission issue as if that were the case it would not clear the issue - more likely dp getting itself confused.
i will run the disk check - but it will take a while for 18 disks
can alex point to particular disk(s)
-
Could you run this:
Chris - any news from the upload from a couple of days ago?
-
ok by resetting my settings for DP - i finally got DP to balance properly again - took a very long time for 20gb to finally move off my ssd
i have only the ssd plugin loaded and all default balances off apart from scanner
have reset my settings and so far things appear back to normal
Chris - any comments???
-
phewwwwwwwwwwwwwwwwwwwwwww
and relax
the re-check appears to have worked and the pool is rebalanced again - although it has not finished yet - and still have data on my ssd's
Looks like it maybe a dp cache thing then?
Chris - does the dpcmd - option to flush the dp cache - do the same thing as a "re-check" as this option does not always appear
-
ok second run agrees with the level 2 run - so a sneaky balance must have happened in between
i have done a remeasure with no change
duplication runs for a few seconds then terminates
doing a re -check now to see if that does anything
-
this is wierd i have run two dpcmd runs
and the files unduplicated are different
the summary level 0 has 1289 files highlighted and the level 2 has 262 - total size is slightly different
i am going to re run the summary one again
-
Ok DPCmd has found some of the files
The odd thing is these files that were duplicated two days ago are now unduplicated - i have a full dpcmd log from the 13/7/2017.
The files (photos) have not even been looked at by me since November last year - checked last access date
So is DP - deleting files?
So is DP - losing files?
the directories which contain the files exist on 4x to 14x drives according the dpcmd - so i doubt its a hardware issue
All the disks are fine - nothing in error logs - scanner happy etc
This is worrying if we cant trust DP to be consistent over a couple of days
-
ok have run the dpcmd check-pool-fileparts on my e:\ pool with option 0
its found 1,289 missing file parts!!!
i ran this two days ago and it found none.
going to run it again with more detail to see whats unduplicated
-
is the user that runs the script the same as the one that you used to install dp and/or it runs under?
-
I now have a pool that i have remeasured - no change on the undup files
if i run a balance it thinks about it for a few seconds then stops (no error) as it thinks its balanced when it is far from it!!!
yep good idea will re run that to see if it agrees with the GUI
-
Troubleshooter info has been submitted
-
Ok it finished duplicating and i now have over 7GB of unduplicated files!
It wanted to duplicate again but i aborted it and am running the trouble shooter now - so should finish uploading in 10-20mins
The balance slowed down for ages then got to about 90% and finished in 10-20 seconds - BUT it moved no files - ssd's still unbalanced as in screenshot from previous post
unduplicated files are now across all but two of 20 drives!
once the info has finished uploading i will try another balance and if that does not work a remeasure - which has fixed it in the past.
If it comes back i think i will move back to 742 where i was before
-
Ok i have unduplicated files again 363MB on disk 02 of my e: pool
Also the ssd's are unbalanced - one has 4gb of duplicated files the other 18.5GB
a duplication run is in progress - but the ssd imbalance was there before the run started - so its not that some has been moved off
see image
Scanner sees both the ssd's as healthy
Lol - the unduplicated just went up to 444MB while doing the balance now affecting disks 04,09,10,11,13,14,15,16,17,18 !!! which were fine before
Duplicating is progressing (very slowly) and although the % is increasing nothing appears to be being moved.
I will wait for it to finish the balance and then run the troubleshooter and report back
-
Chris - no missing drives or drives going missing
server had been on for several days with no issues - numerous backups to the pool worked fine - not large changes
run a lot of changes - can trigger the issue though - then balancing fails to work correctly - or runs but does not move any files off the ssd's?
-
Actually my issue was also after a large number of files were amended on the pool - editing, copying and moving thousands of photos (via a script - powershell)
I wonder if in this situation the duplication on Dp falls behind and causes the checking and duplication
This would be odd as my ssd cache is two samsung 850 pro's and the total volume of data amended was only a few GB
-
Chris
currently its not showing any unduplicated - but i guess it will be back if its a bug - do you still want me to re run the troubleshoot even if its not showing at the moment or wait?
"Well, the duplication settings page calculates this in real time, so you can use that." - could you explain a bit more please? not sure i follow
-
Getting a lot of "Ensure file duplication consistency.." on one pool every hour or so and also it keeps having unduplicated data - from a few hundred meg to 6GB - varies which drive its on when checking each drive from main GUI
pool is set to full duplication and all folders show x2
did not have this on 742 just updated to 769 in last couple of days when this started
new bug?
Also need a feature pronto to actually show which are not duplicated so you can check as the program knows
Have run troubleshooter and its submitting now
-
Ace thanks!
-
a counter view
I find it hard to understand using DP without duplication - kind of the point as you are looking for redundancy - having one big drive with minimal or selective duplication is the way to problems as you describe if you dont know what drive has what - you "almost" might as well have a JBOD or Raid0
I dont care which of my files are on which drive as a failure is not a big issue - unlike potentially with raid - one of the main reasons i also use DP - minimal rebuild time much less chance of a second disk failing when re-dupping my files - if i want more than one level of redundancy i can go to x3,or x4 duplication etc or as i have done two pools. (I have used raid for years and still have a QNAP with 8*4TB that i still use daily as it has a few advantages so well aware of the pitfalls and benefits and still i think on balance DP is a better solution)
If you must micro manage your files - use the balancers and specify what goes where - if its media files they are not going to move around once the pool is balanced after they are added - unless you keep tinkering with the settings!
As an alternative copy the files directly into the pool part directory on a specific drive - much more likely to stay together then and you know where they are.
Oh and its easy to say you can re rip your blu-ray collection if things go south until you have to do it and spend days doing it even with a relatively small collection.
Have fun
puts on tin hat
-
-
My HP Sas expander works fine with 6tb drives
Drivepool keeps checking and duplicating
in General
Posted
Ok - i have tried the following
uninstalled the ssd plugin - upgraded to 773 - reinstalled ssd plugin - with associated reboots
This resulted in no change - i.e. the ssd cache does not work and files are saved to the HDD directly
I appear to have fixed it by removing the ssd's from the pool, i reformatted them and added them back - this appears to have made the ssd plugin work again/the pool to use the ssd's
Observations
1) DP needs some better and more visible error reporting to the user - as a lot of this time and effort could have been saved by some informative error reports
2) Resetting to defaults - does not reset everything
3) DP Error reports should not be compressed/encrypted so the user can read them