Spider99

hi zip - works if i drag and drop a file on it - the machine has winRar on it if that might make a difference CSV is not going to work with more than 500k of files as Excel 2007 - drops all records above 1048576 Csv has 1438463 lines log has 2250006 lines Can the unicode stuff have a switch in the script as i dont need it and would like to keep the files smaller they are big enough already the directory i use is in the root of d: and the user is my netadmin account which also created the scheduled task so permissions should not be an issue - permissions look ok will give 1.5 a bash

Hi 1. i would guess that D: is a separate partition on the "C" drive - i.e. drive was split into two - probably when the os was installed 2. You can swap drive letters using "disk management" - have to change one to say H first then move f to d and then d to f 3. To move its just a copy/move as you would do for any file - no magic involved - or add f to the pool and use the "quick and dirty method" to move the files into the poolpart directory 4. Nope os remains on c - cant move that 5. attached to computer power up - windows will see the new drive and then so will dp - just click add - few seconds later drive will be part of the pool. Have fun Quick and dirty method http://wiki.covecube.com/StableBit_DrivePool_Q4142489

Hi Yes i know you did not write the dpcmd utility - what i meant was could you add a line in the PS script to convert it to single byte - maybe an optional switch?? I ran 1.4 last night CSV looks fine now appears to have all files - ~1.4m lines but it failed to zip again - have the zip - size 1k is there anything to check my end on the zip front - i posted my .net and PS version a couple of days ago?

Hi One thing i spotted today is the log file is Unicode which means its twice the size it needs to be - if you convert to ANSI (single byte per character) then the files are about half the size - saving 350MB per file is no small saving and will save on the zip files as well could you add this to the script? you can use the "type" command in windows but i guess there would be something in PS to do this as well - not looked Running 1,4 now have a basic gui working but needs more work before its fully useable

I would do something a little different and take the drives out of the enclosures and put them into one or two new enclosure and or cases - i have looked at those external drives myself as a way to get some cheap storage to add to my pool but did not in the end as wanted different drives i use an orico 5 bay usb3 enclosure with out issue - but not with DP - its for backup etc - but scanner sees the drives and scans them with no problem - so dp will work ok also - some have problems with usb dropping out or other issues but i have never had that issue ever depends what you might want to spend to make this work - an old pc case with just a power supply - and using a sas/sata hba means they could be connected to your pc and they would appear as a if they were actually attached esata is another option again there are enclosures

not had a chance to run 1.4 yet probabaly tomorrow

Hi v1.2 was run on 2012 r2 essentials i believe its upto date PS version 4 0 -1 -1 net 4.6.1 Just working on a gui for log file etc will run 1.4 later - need fooooooood

get it replaced under warranty other blocks are likely to begin to fail usually they will send you one out and as long as you send the old one back in a defined time you will not be charged - can depend where you are in the world

update no real change apart from drive 19 is now approx 20% full

Hi well it ran all the way through - i left it running - took slightly longer to create the csv than the log took to create ??? 1hr25m vs 1h33m it needs a couple of bug fixes 1. it only lists one file location per file - should have 2 as have x2 dup and a few files have 3x 2. Does not list directory locations - useful if you want to find all your files and copy them to a new disk etc 3. zip does not work - empty file total number of lines appears to match file totals going to look at some alternative ways of processing the log file

dont know not let a drive get that far gone

well got bored waiting and killed the process for Powershell - it was just cycling between high and low memory every 20 seconds or so i guess it was doing something but very slowly could you change your script to write as it goes for testing so i can see if its doing what we expect

Summary: Directories: (14,957) Files: (719,212) 21.8 TB (23,990,698,777,117 File parts: (1,438,423) 43.6 TB (47,979,629,929,044 * Inconsistent directories: 0 * Inconsistent files: 1 * Missing file parts: 1 1.65 GB (1,767,625,190 ! Error reading directories: 1 ! Error reading files: 0 Error was sys vol info Produced a 706 MB files in 1h 25mins approx I think Powershell is not liking the task you have set it though - keeps taking 1.5 GB of memory and falls back to 200 MB and climbs again to 1.5GB and back to 200mB - watched it do this several times already - only taking about 6% of cpu though - will leave it for a while to if its actually doing anything

Current pool view Disk 19 is being emptied it was full yesterday now approx half looking like its going back to zero Disk 9 and 10 set to zero 1 full and 2 5% full does not make any sense to me pool is approx 48% full - so most if not all disks should be approx that full also?

Disabled all made no difference just have ssd Opt running still have drives being emptied and other set to store no files - using the blur triangles as a guide i stopped the balance and it restarted and carried on as before minimal if any change any other ideas

ah right ok no problem ill wait till the end to see how long this takes etc did not look at you script in detail yet

Not seeing any disk models or serial no in the output volume and poolpart and device no though is that what you meant? dpcmd - StableBit DrivePool command line interface Version 2.2.0.734 Detail level: File Parts Listing types: + Directory - File -> File part * Inconsistent duplication ! Error Listing format: [{0}/{1} IM] {2} {0} - The number of file parts that were found for this file / directory. {1} - The expected duplication count for this file / directory. I - This directory is inheriting its duplication count from its parent. M - At least one sub-directory may have a different duplication count. {2} - The name and size of this file / directory. Scanning... + [28x/2x M] E:\ -> \Device\HarddiskVolume1\PoolPart.24a10d83-5eb3-45d8-a126-fb04f6e3d1a2\ [Device 0] -> \Device\HarddiskVolume2\PoolPart.09244e60-99ae-4044-867c-c729d26ea73b\ [Device 1] -> \Device\HarddiskVolume3\PoolPart.4e8eb030-e003-4345-808d-8c916b0631ab\ [Device 2] -> \Device\HarddiskVolume4\PoolPart.ba54be14-6f03-488e-b8df-92623fba5e0a\ [Device 3] -> \Device\HarddiskVolume5\PoolPart.c3a46a7c-287f-4dd8-85c6-0059c52a227a\ [Device 4] -> \Device\HarddiskVolume6\PoolPart.a22c77ba-1cb8-4733-8d03-7f6782da4241\ [Device 5] -> \Device\HarddiskVolume7\PoolPart.0aed21a3-07d5-4b20-adfc-385626021e54\ [Device 6] -> \Device\HarddiskVolume8\PoolPart.711da57f-1c69-445c-b925-b2de11d9311b\ [Device 7] -> \Device\HarddiskVolume9\PoolPart.158beb49-b7f2-4635-97ff-e51bedbc6a0f\ [Device 8] -> \Device\HarddiskVolume10\PoolPart.5a8a5b05-62c8-4c91-9d04-6e69420e1e5b\ [Device 9] -> \Device\HarddiskVolume11\PoolPart.2613d588-9056-43f9-8410-31c4f267e2ca\ [Device 10] -> \Device\HarddiskVolume13\PoolPart.0486ff3a-604f-4c04-ada8-d7d2851cf13f\ [Device 11] -> \Device\HarddiskVolume15\PoolPart.bd847b99-74f9-4723-bed8-aba3a7739d93\ [Device 12] -> \Device\HarddiskVolume17\PoolPart.23cd6461-b6c9-4bff-b566-3ac9282ca68a\ [Device 13] -> \Device\HarddiskVolume19\PoolPart.f118843a-fc50-4494-bb97-7f31ce6565b0\ [Device 14] -> \Device\HarddiskVolume21\PoolPart.00baa63a-cf5f-4ed2-af44-c19624546738\ [Device 15] -> \Device\HarddiskVolume22\PoolPart.1408546a-e189-4f79-b03a-f66c1d241de3\ [Device 16] -> \Device\HarddiskVolume23\PoolPart.272e4bde-7f23-4bbb-9981-2d3123b86381\ [Device 17] -> \Device\HarddiskVolume24\PoolPart.b1d8fa8c-1210-40c0-b3d3-91c79ac81084\ [Device 18] -> \Device\HarddiskVolume25\PoolPart.656efc36-5b4f-417c-a7c7-e811697d3506\ [Device 19] -> \Device\HarddiskVolume27\PoolPart.e95092e8-fbdf-4701-9e2e-33d190be8786\ [Device 20] -> \Device\HarddiskVolume29\PoolPart.f6a548a2-8d41-4144-8537-1909ce861df1\ [Device 21] -> \Device\HarddiskVolume31\PoolPart.242cea5e-859c-4959-ae4a-0bf7e255623c\ [Device 22] -> \Device\HarddiskVolume33\PoolPart.640e49c6-45ca-4768-8544-0bfb3fbb1277\ [Device 23] -> \Device\HarddiskVolume39\PoolPart.13adb443-d50e-413a-bc4a-d251a7e68463\ [Device 25] -> \Device\HarddiskVolume40\PoolPart.2680be39-1dbe-4451-8c57-1a0996a98aa5\ [Device 26] -> \Device\HarddiskVolume41\PoolPart.aa108639-369f-42a7-9aef-cf6573cdfd81\ [Device 27] -> \Device\HarddiskVolume42\PoolPart.6bd8f30d-cc74-4355-ad74-e1afbdeed0d6\ [Device 28] + [28x/2x I] E:\ServerFolders -> \Device\HarddiskVolume1\PoolPart.24a10d83-5eb3-45d8-a126-fb04f6e3d1a2\ServerFolders [Device 0] -> \Device\HarddiskVolume2\PoolPart.09244e60-99ae-4044-867c-c729d26ea73b\ServerFolders [Device 1] -> \Device\HarddiskVolume3\PoolPart.4e8eb030-e003-4345-808d-8c916b0631ab\ServerFolders [Device 2] -> \Device\HarddiskVolume4\PoolPart.ba54be14-6f03-488e-b8df-92623fba5e0a\ServerFolders [Device 3] -> \Device\HarddiskVolume5\PoolPart.c3a46a7c-287f-4dd8-85c6-0059c52a227a\ServerFolders [Device 4] -> \Device\HarddiskVolume6\PoolPart.a22c77ba-1cb8-4733-8d03-7f6782da4241\ServerFolders [Device 5] -> \Device\HarddiskVolume7\PoolPart.0aed21a3-07d5-4b20-adfc-385626021e54\ServerFolders [Device 6] -> \Device\HarddiskVolume8\PoolPart.711da57f-1c69-445c-b925-b2de11d9311b\ServerFolders [Device 7] -> \Device\HarddiskVolume9\PoolPart.158beb49-b7f2-4635-97ff-e51bedbc6a0f\ServerFolders [Device 8] -> \Device\HarddiskVolume10\PoolPart.5a8a5b05-62c8-4c91-9d04-6e69420e1e5b\ServerFolders [Device 9] -> \Device\HarddiskVolume11\PoolPart.2613d588-9056-43f9-8410-31c4f267e2ca\ServerFolders [Device 10] -> \Device\HarddiskVolume13\PoolPart.0486ff3a-604f-4c04-ada8-d7d2851cf13f\ServerFolders [Device 11] -> \Device\HarddiskVolume15\PoolPart.bd847b99-74f9-4723-bed8-aba3a7739d93\ServerFolders [Device 12] -> \Device\HarddiskVolume17\PoolPart.23cd6461-b6c9-4bff-b566-3ac9282ca68a\ServerFolders [Device 13] -> \Device\HarddiskVolume19\PoolPart.f118843a-fc50-4494-bb97-7f31ce6565b0\ServerFolders [Device 14] -> \Device\HarddiskVolume21\PoolPart.00baa63a-cf5f-4ed2-af44-c19624546738\ServerFolders [Device 15] -> \Device\HarddiskVolume22\PoolPart.1408546a-e189-4f79-b03a-f66c1d241de3\ServerFolders [Device 16] -> \Device\HarddiskVolume23\PoolPart.272e4bde-7f23-4bbb-9981-2d3123b86381\ServerFolders [Device 17] -> \Device\HarddiskVolume24\PoolPart.b1d8fa8c-1210-40c0-b3d3-91c79ac81084\ServerFolders [Device 18] -> \Device\HarddiskVolume25\PoolPart.656efc36-5b4f-417c-a7c7-e811697d3506\ServerFolders [Device 19] -> \Device\HarddiskVolume27\PoolPart.e95092e8-fbdf-4701-9e2e-33d190be8786\ServerFolders [Device 20] -> \Device\HarddiskVolume29\PoolPart.f6a548a2-8d41-4144-8537-1909ce861df1\ServerFolders [Device 21] -> \Device\HarddiskVolume31\PoolPart.242cea5e-859c-4959-ae4a-0bf7e255623c\ServerFolders [Device 22] -> \Device\HarddiskVolume33\PoolPart.640e49c6-45ca-4768-8544-0bfb3fbb1277\ServerFolders [Device 23] -> \Device\HarddiskVolume39\PoolPart.13adb443-d50e-413a-bc4a-d251a7e68463\ServerFolders [Device 25] -> \Device\HarddiskVolume40\PoolPart.2680be39-1dbe-4451-8c57-1a0996a98aa5\ServerFolders [Device 26] -> \Device\HarddiskVolume41\PoolPart.aa108639-369f-42a7-9aef-cf6573cdfd81\ServerFolders [Device 27] -> \Device\HarddiskVolume42\PoolPart.6bd8f30d-cc74-4355-ad74-e1afbdeed0d6\ServerFolders [Device 28] + [25x/2x I] E:\ServerFolders\Backup -> \Device\HarddiskVolume1\PoolPart.24a10d83-5eb3-45d8-a126-fb04f6e3d1a2\ServerFolders\Backup [Device 0] -> \Device\HarddiskVolume3\PoolPart.4e8eb030-e003-4345-808d-8c916b0631ab\ServerFolders\Backup [Device 2] -> \Device\HarddiskVolume4\PoolPart.ba54be14-6f03-488e-b8df-92623fba5e0a\ServerFolders\Backup [Device 3] -> \Device\HarddiskVolume5\PoolPart.c3a46a7c-287f-4dd8-85c6-0059c52a227a\ServerFolders\Backup [Device 4] -> \Device\HarddiskVolume6\PoolPart.a22c77ba-1cb8-4733-8d03-7f6782da4241\ServerFolders\Backup [Device 5] -> \Device\HarddiskVolume7\PoolPart.0aed21a3-07d5-4b20-adfc-385626021e54\ServerFolders\Backup [Device 6] -> \Device\HarddiskVolume8\PoolPart.711da57f-1c69-445c-b925-b2de11d9311b\ServerFolders\Backup [Device 7] -> \Device\HarddiskVolume9\PoolPart.158beb49-b7f2-4635-97ff-e51bedbc6a0f\ServerFolders\Backup [Device 8] -> \Device\HarddiskVolume10\PoolPart.5a8a5b05-62c8-4c91-9d04-6e69420e1e5b\ServerFolders\Backup [Device 9] -> \Device\HarddiskVolume15\PoolPart.bd847b99-74f9-4723-bed8-aba3a7739d93\ServerFolders\Backup [Device 12] -> \Device\HarddiskVolume17\PoolPart.23cd6461-b6c9-4bff-b566-3ac9282ca68a\ServerFolders\Backup [Device 13] -> \Device\HarddiskVolume19\PoolPart.f118843a-fc50-4494-bb97-7f31ce6565b0\ServerFolders\Backup [Device 14] -> \Device\HarddiskVolume21\PoolPart.00baa63a-cf5f-4ed2-af44-c19624546738\ServerFolders\Backup [Device 15] -> \Device\HarddiskVolume22\PoolPart.1408546a-e189-4f79-b03a-f66c1d241de3\ServerFolders\Backup [Device 16] -> \Device\HarddiskVolume23\PoolPart.272e4bde-7f23-4bbb-9981-2d3123b86381\ServerFolders\Backup [Device 17] -> \Device\HarddiskVolume24\PoolPart.b1d8fa8c-1210-40c0-b3d3-91c79ac81084\ServerFolders\Backup [Device 18] -> \Device\HarddiskVolume25\PoolPart.656efc36-5b4f-417c-a7c7-e811697d3506\ServerFolders\Backup [Device 19] -> \Device\HarddiskVolume27\PoolPart.e95092e8-fbdf-4701-9e2e-33d190be8786\ServerFolders\Backup [Device 20] -> \Device\HarddiskVolume29\PoolPart.f6a548a2-8d41-4144-8537-1909ce861df1\ServerFolders\Backup [Device 21] -> \Device\HarddiskVolume31\PoolPart.242cea5e-859c-4959-ae4a-0bf7e255623c\ServerFolders\Backup [Device 22] -> \Device\HarddiskVolume33\PoolPart.640e49c6-45ca-4768-8544-0bfb3fbb1277\ServerFolders\Backup [Device 23] -> \Device\HarddiskVolume39\PoolPart.13adb443-d50e-413a-bc4a-d251a7e68463\ServerFolders\Backup [Device 25] -> \Device\HarddiskVolume40\PoolPart.2680be39-1dbe-4451-8c57-1a0996a98aa5\ServerFolders\Backup [Device 26] -> \Device\HarddiskVolume41\PoolPart.aa108639-369f-42a7-9aef-cf6573cdfd81\ServerFolders\Backup [Device 27] -> \Device\HarddiskVolume42\PoolPart.6bd8f30d-cc74-4355-ad74-e1afbdeed0d6\ServerFolders\Backup [Device 28] - [2x/2x] E:\ServerFolders\Backup\Capture.PNG (126 KB - 129,380 -> \Device\HarddiskVolume21\PoolPart.00baa63a-cf5f-4ed2-af44-c19624546738\ServerFolders\Backup\Capture.PNG [Device 15] -> \Device\HarddiskVolume22\PoolPart.1408546a-e189-4f79-b03a-f66c1d241de3\ServerFolders\Backup\Capture.PNG [Device 16] - [2x/2x] E:\ServerFolders\Backup\Capture2.PNG (54.2 KB - 55,484 -> \Device\HarddiskVolume21\PoolPart.00baa63a-cf5f-4ed2-af44-c19624546738\ServerFolders\Backup\Capture2.PNG [Device 15] -> \Device\HarddiskVolume24\PoolPart.b1d8fa8c-1210-40c0-b3d3-91c79ac81084\ServerFolders\Backup\Capture2.PNG [Device 18] - [2x/2x] E:\ServerFolders\Backup\Capture3.PNG (6.11 KB - 6,261 -> \Device\HarddiskVolume21\PoolPart.00baa63a-cf5f-4ed2-af44-c19624546738\ServerFolders\Backup\Capture3.PNG [Device 15] -> \Device\HarddiskVolume24\PoolPart.b1d8fa8c-1210-40c0-b3d3-91c79ac81084\ServerFolders\Backup\Capture3.PNG [Device 18] - [2x/2x] E:\ServerFolders\Backup\Capture4.PNG (8.58 KB - 8,784 -> \Device\HarddiskVolume21\PoolPart.00baa63a-cf5f-4ed2-af44-c19624546738\ServerFolders\Backup\Capture4.PNG [Device 15] -> \Device\HarddiskVolume24\PoolPart.b1d8fa8c-1210-40c0-b3d3-91c79ac81084\ServerFolders\Backup\Capture4.PNG [Device 18] - [2x/2x] E:\ServerFolders\Backup\Capture5.PNG (40.7 KB - 41,643 -> \Device\HarddiskVolume21\PoolPart.00baa63a-cf5f-4ed2-af44-c19624546738\ServerFolders\Backup\Capture5.PNG [Device 15] -> \Device\HarddiskVolume24\PoolPart.b1d8fa8c-1210-40c0-b3d3-91c79ac81084\ServerFolders\Backup\Capture5.PNG [Device 18]

Well its generated 150 mb file in 20 minutes so not huge but it has a million or more to run at did your tests have the full unc path including the volume name - as i dont have my drives lettered as i have more than 26 @dj80 also will the csv have the unc path or will it drop that to device 1,2 ,3 etc?

ok boys its running on my 2012r2 essentials i changed a couple of paths so i could put it on my d drive and have created a separate directory for the logs i have initiated it from within task Scheduler and its off to the races Now it has 40+TB to run through (duplicated pool) which is balancing at present - so will be interesting to see if i get files in more than one location in the log Also will i get two instances at 1:00 as i guess it will not have finished by then Couple of thoughts 1. Yes a UI would be nice but not essential 2. More pressing with large pools is the size of file its going to create - i'm thinking put the info into a free db program e.g. SQL Lite - or something more powerful - depends on performance- rather than use Excel 3. Some sort of summary file at the end - or thats generated form the db 4. Not looked whats required but there is an api for doing plugins for DP - could be integrated that way perhaps?? Will report back when the machine breaks or finishes Have experience of the above not a big task just needs a bit of time to code it up

interesting i will have a play with v1.1 later and see how i go

Yes hence why the old drives are giving up being worked very hard as both are doing the same thing they - will only get worse though

how old is the drive? if its new (say less than 3 months) you might be lucky and its the only block that will fail - you need to keep a very close eye on it - disk do have spare blocks that they remap to - usually you suffer poor performance when that block is trying to be read as its in another part of the drive. I have one drive thats several years old that has just one bad block which it got soon after i got it but none since. If the drive is old then this is the first indication that its going to die - how long it will last to unknown but it will get worse As for using it - how much do you value your data and the risk of corruption etc????

this is still happening i have disabled most of the plugins Using 1. SSD Opt - have ordered the drives the same for dup and undup manually i.e. 1-24 2. Scanner 3. Disk space equal 4. Prevent overfill No file placement rules Have drives being filled up to max (10GB left) others marked as Zero space to be used been like this for a few days - then overnight get filled up to 80-90% Majority are "correct" in having approx 50% fill limit and seem to be stable - as you can monitor the actual file placement easily difficult to tell. DP is balancing a large pool 40+TB with 12 new drives added about a week ago - took a few days to do the duplication and has been at the balance for a few days now (would have been much quicker to do it myself i think!) Any ideas as i am coming to the opinion this is a logic bug you only see with large volumes of data

bump

exactly that - you have two copies of each file on different disks - in windows explorer you see only one copy - no magic sauce just a merge of the files structure to hide the duplicate entries for all disk in the pool if you have a ssd cache drive setup - the file is copied to the cache from you client machine as it does so a copy is being created on one of the archive drives at the same time or very shortly afterwards depending on the speed of the archive drive - first copy will then be copied to another archive drive at some point later depending on your balancer settings. If no ssd cache then an archive drive is chosen for the landing zone - rest is the same. you have four disks i would make one pool - turn on duplication 2x or more and copy over a large sample directory and see for yourself rather than dup everything you could do it at the folder level as well - i.e. only duplicate folders you care about or x4 those you cannot loose - max is number of disk you have in the pool obviously Oh i would also remove the drive letters from your pool disks (mount them to a directory instead) makes for a cleaner interface - you can still access them directly but it prevents an accidental mistake DP takes the next available drive letter when created so in my case E: - not tried to change this but its emulating a real disk so changing the drive letter should be fine - did you try in disk management?