Spider99

Ah - i just did a test copy of a large file to the pool and the file was "saved" to a HDD not the ssd's - hence why i am not getting duplication - real time duplication is enabled in the gui so checked config page of ssd optimiser and the two ssd's are defined as ssd's - hmm a bit more testing is needed - will report back

I can almost create this at will now I ran a simple script to update the "title" field in a few thousand MKV files now i have over 400GB of "unduplicated files" - checking the logs - no issues for these files - they are not listed (unlike some others that i spotted earlier - which i have cleaned up) Doing a remeasure at the moment to confirm (looking like it) the unduplicated files Question - why are the error logs encrypted/compressed for DP? other logs are not.

if you create a folder by hand - i.e. not a copy or script is it also slow to appear?

this is all on v769 doubt very much its a permission issue as if that were the case it would not clear the issue - more likely dp getting itself confused. i will run the disk check - but it will take a while for 18 disks can alex point to particular disk(s)

Chris - any news from the upload from a couple of days ago?

ok by resetting my settings for DP - i finally got DP to balance properly again - took a very long time for 20gb to finally move off my ssd i have only the ssd plugin loaded and all default balances off apart from scanner have reset my settings and so far things appear back to normal Chris - any comments???

phewwwwwwwwwwwwwwwwwwwwwww and relax the re-check appears to have worked and the pool is rebalanced again - although it has not finished yet - and still have data on my ssd's Looks like it maybe a dp cache thing then? Chris - does the dpcmd - option to flush the dp cache - do the same thing as a "re-check" as this option does not always appear

ok second run agrees with the level 2 run - so a sneaky balance must have happened in between i have done a remeasure with no change duplication runs for a few seconds then terminates doing a re -check now to see if that does anything

this is wierd i have run two dpcmd runs and the files unduplicated are different the summary level 0 has 1289 files highlighted and the level 2 has 262 - total size is slightly different i am going to re run the summary one again

Ok DPCmd has found some of the files The odd thing is these files that were duplicated two days ago are now unduplicated - i have a full dpcmd log from the 13/7/2017. The files (photos) have not even been looked at by me since November last year - checked last access date So is DP - deleting files? So is DP - losing files? the directories which contain the files exist on 4x to 14x drives according the dpcmd - so i doubt its a hardware issue All the disks are fine - nothing in error logs - scanner happy etc This is worrying if we cant trust DP to be consistent over a couple of days

ok have run the dpcmd check-pool-fileparts on my e:\ pool with option 0 its found 1,289 missing file parts!!! i ran this two days ago and it found none. going to run it again with more detail to see whats unduplicated

is the user that runs the script the same as the one that you used to install dp and/or it runs under?

I now have a pool that i have remeasured - no change on the undup files if i run a balance it thinks about it for a few seconds then stops (no error) as it thinks its balanced when it is far from it!!! yep good idea will re run that to see if it agrees with the GUI

Troubleshooter info has been submitted

Ok it finished duplicating and i now have over 7GB of unduplicated files! It wanted to duplicate again but i aborted it and am running the trouble shooter now - so should finish uploading in 10-20mins The balance slowed down for ages then got to about 90% and finished in 10-20 seconds - BUT it moved no files - ssd's still unbalanced as in screenshot from previous post unduplicated files are now across all but two of 20 drives! once the info has finished uploading i will try another balance and if that does not work a remeasure - which has fixed it in the past. If it comes back i think i will move back to 742 where i was before

Ok i have unduplicated files again 363MB on disk 02 of my e: pool Also the ssd's are unbalanced - one has 4gb of duplicated files the other 18.5GB a duplication run is in progress - but the ssd imbalance was there before the run started - so its not that some has been moved off see image Scanner sees both the ssd's as healthy Lol - the unduplicated just went up to 444MB while doing the balance now affecting disks 04,09,10,11,13,14,15,16,17,18 !!! which were fine before Duplicating is progressing (very slowly) and although the % is increasing nothing appears to be being moved. I will wait for it to finish the balance and then run the troubleshooter and report back

Chris - no missing drives or drives going missing server had been on for several days with no issues - numerous backups to the pool worked fine - not large changes run a lot of changes - can trigger the issue though - then balancing fails to work correctly - or runs but does not move any files off the ssd's?

Actually my issue was also after a large number of files were amended on the pool - editing, copying and moving thousands of photos (via a script - powershell) I wonder if in this situation the duplication on Dp falls behind and causes the checking and duplication This would be odd as my ssd cache is two samsung 850 pro's and the total volume of data amended was only a few GB

Chris currently its not showing any unduplicated - but i guess it will be back if its a bug - do you still want me to re run the troubleshoot even if its not showing at the moment or wait? "Well, the duplication settings page calculates this in real time, so you can use that." - could you explain a bit more please? not sure i follow

Getting a lot of "Ensure file duplication consistency.." on one pool every hour or so and also it keeps having unduplicated data - from a few hundred meg to 6GB - varies which drive its on when checking each drive from main GUI pool is set to full duplication and all folders show x2 did not have this on 742 just updated to 769 in last couple of days when this started new bug? Also need a feature pronto to actually show which are not duplicated so you can check as the program knows Have run troubleshooter and its submitting now

Ace thanks!

a counter view I find it hard to understand using DP without duplication - kind of the point as you are looking for redundancy - having one big drive with minimal or selective duplication is the way to problems as you describe if you dont know what drive has what - you "almost" might as well have a JBOD or Raid0 I dont care which of my files are on which drive as a failure is not a big issue - unlike potentially with raid - one of the main reasons i also use DP - minimal rebuild time much less chance of a second disk failing when re-dupping my files - if i want more than one level of redundancy i can go to x3,or x4 duplication etc or as i have done two pools. (I have used raid for years and still have a QNAP with 8*4TB that i still use daily as it has a few advantages so well aware of the pitfalls and benefits and still i think on balance DP is a better solution) If you must micro manage your files - use the balancers and specify what goes where - if its media files they are not going to move around once the pool is balanced after they are added - unless you keep tinkering with the settings! As an alternative copy the files directly into the pool part directory on a specific drive - much more likely to stay together then and you know where they are. Oh and its easy to say you can re rip your blu-ray collection if things go south until you have to do it and spend days doing it even with a relatively small collection. Have fun puts on tin hat

Try here - all the beta downloads for DP http://dl.covecube.com/DrivePoolWindows/beta/download/

My HP Sas expander works fine with 6tb drives

Hi Red pro are noisy and run hot although a good drive - but expensive for what you get if you are not going to use them in a NAS i have HGST (4tb) drives and they are fast at 170 + MB/s when lightly filled when full they drop down to ~ 120 MB/s with large files SSD's obviously significantly faster Have you thought of Raid 0 for stream - as long as you have a good backup strategy to protect your data - then cheaper/smaller SSD's could be used as a failure will take out the whole raid as 0 has no redundancy