Wiidesire

The problem is that you were still on an affected version 3216. By upgrading to the newest version the Stablebit Scanner Service is forcefully shut down, thus the DiskId files can get corrupted in the upgrade process. Now that you are on version 3246 which fixed the problem it shouldn't happen anymore on your next upgrade/reboot/crash.
I agree wholeheartedly though that we should get a way to backup the scan status of drives just in case. A scheduled automatic backup would be great. The files are extremely small and don't take a lot of space so don't see a reason not to implement it feature wise.

The problem is that you were still on an affected version 3216. By upgrading to the newest version the Stablebit Scanner Service is forcefully shut down, thus the DiskId files can get corrupted in the upgrade process. Now that you are on version 3246 which fixed the problem it shouldn't happen anymore on your next upgrade/reboot/crash.
I agree wholeheartedly though that we should get a way to backup the scan status of drives just in case. A scheduled automatic backup would be great. The files are extremely small and don't take a lot of space so don't see a reason not to implement it feature wise.

The cause of the issue is fixed with Beta 3246. Since my machine is not frequently crashing, maybe once per month, it will take a while to verify it on my side.

I've had it happen with normal reboots as well, just not as often as with crashes. It just depends on the timing. Imagine what happens on a reboot. Windows is forcefully shutting down services, including the Stablebit Scanner Service. So if this service gets shutdown at the timeframe where it is writing new DiskId files the files can end up corrupted, thus after a reboot the Service creates new DiskId files meaning all previous scan status is lost.
Now the DiskId are not written literally every second anymore (which increases the risk that the service gets killed at the time of writing files significantly) but instead 20-40 minutes (don't know the exact interval now) . That's a reduction of a factor of 1200 to 2400 so the risk that you reboot at the exact time the files are written should basically be zero now.

To keep everyone up-to-date:
With the help of Alex I've identified the root cause of the issue. The LastSeen variable inside the DiskId files is changed literally every second. This means that the DiskId files are constantly being changed and in the event of a crash there is a high chance that it crashes while the new file is being written thus the DiskId files get corrupted.
The LastSmartUpdate variable inside the SmartPersistentInfo files is updated in a more reasonable one minute interval so I'm hoping it is a quick fix to simply adjust the write interval of the LastSeen variable.
Besides changing the interval there would have to be backup diskid files to completely eliminate the issue. So instead of creating new DiskId files when corrupt files have been detected it should copy over an older backup file of the DiskId file(s) in question. Or the LastSeen value gets completely eliminated from the DiskId file and moved somewhere else to avoid changing the DiskId files at all.

​It's too late now. It wouldn't matter anyway because in the default configuration DrivePool splits your media so much across the drives that it's way too time consuming to try to replace single files manually. I'm using DrivePool for the same reason (media library) as you so I can give you two critical tips to avoid this in the future:
​
​1) Install the Ordered File Placement plugin from here. After that go to your pool options, activate the Ordered File Placement balancer and deactivate all others. Then prioritize your drives. This way when you add a TV show or a new season, it will all land on the same drive and will not be spread across the pool which is critical for a possible replacement unless you use capacity wasting duplication.
Configurable ​Folder Split Levels would be an more ideal solution but the sole DrivePool developer "Alex" personally doesn't see the need for it -> low on priority list and will most likely never happen.
​2) Use the tool WinCatalog 2017 to keep track on what drive your media is. I'm very impressed with the tool so far. It not only archives the location and size of your files but also details like resolution/bitrate/audio tracks etc.

I have to agree with Wiidesire on this one. To be blunt, Alex is so completely wrong on this I don't even know where to begin. I currently use unRAID right now even though I own two DP licenses. My reason? Folder split level. I don't have the need or desire to go into hidden folders every time I want to drop new episodes of TV Show X into my pool. In this day and age, it's silly to expect the user to do that. I would prefer to be running Windows and DP since the rest of my network is all windows but I will not be transferring all my data back till this folder split level is resolved. That's not a threat in any way shape or form. I even bought a cloud drive license even though I don't use it. I bought it to support Alex and the continued development of SB products.
 
I usually stay pretty quiet on here but Alex saying he doesn't see a need for it kinda pisses me off. Makes me curious if he forgot who pays for this software. That would be us, the customers. Again, to be blunt, I honestly don't care if Alex sees a need for it or not. If it gets requested frequently, like Chris has said it does, then Alex needs to address it. It's that simple.
 
 
Sent from my iPhone using Tapatalk

Thanks for pumping this thread, I've been using DrivePool since yesterday and my discs are being constantly accessed (my HDD enclosure has LEDs which show individual disc activity -> constant blinking with DrivePool).
​
​Explanation:
​http://community.covecube.com/index.php?/topic/1784-drivepool-beta-constant-disk-activity/&do=findComment&comment=13602
​
​I'll try the fix tomorrow, hopefully it works.

I am sorry for this post, but it needs to be said.
I was so exited when I found DrivePool and Scanner about 2 years ago, that I bought the whole "package" right then and there. I think I even own a licence for CloudDrive, though I never tried it 
 
I've been keeping myself from posting a rant for a looong time now. But after reading this thread, it's just too much...
Long story short - my setup is 50+ drives in one pool, with a million plus files. A lot of points made below stem from this - having lots of drives with a shitload of files.
 
 
Scattered all over files
 1. DrivePool, for some ungodly reason, scatters files willy-nilly, like a shot from the shotgun (I know there are reason(s), I read something by Alex somewhere, but I don't think, they're good enough to warrant this situation). At least it's by default. And actually that's a "preferred" type of operation, as I understand. Why there isn't an option to keep files together in the same folder structure, is beyond me. For example how unRAID does it. Further-more unRAID could get away with scattering files wherever, because it uses parity for protection, not duplication. Drivepool, on the other hand - cannot get away with this (more on that later).

IMHO, this is the biggest flaw in DrivePool, so it bears repeating. No folders splitting. Or rather a really good system to setup the folders splitting. Again - look at unRAID and do what they're doing. i.e.: splitting of TV Shows when copying to the pool. Now - all the shows have most of the episodes scattered on all 50+ drives. Stupid.

2. Simple real life example (my home server). At the "height" of my excitement with DrivePool, and by that I mean the days just after I found/bough/installed it on my server, I put all my spinners (on my server) in a pool. That's 50+ drives in a single pool. I loved the convenience of the pool and navigating/finding everything easily. I have setup 2x duplication on some folders, not all the of pool (not all files need protecting). Anyway, some weeks later my first drive died, while using DrivePool. Fuck. Not because drive died - no - because the first thing I obviously needed to know was - what files have I lost. I came to the forums, and quickly found, that even though DrivePool's half of functionality is file security (by a way duplication) - it does not keep track of files on the pool, in any shape or form. Fakme. Now that was stupid, I thought to myself. Especially, if you gonna throw my files around the disks, where I have no idea where they actually are - how the fuck can you not keep track of them, so you could tell me, what I've lost if a drive dies? And up to this day, I have no idea what I've actually lost, just that it was ~3TB of stuff. Love it 

After that happened, I had a thought and stopped using duplication. Actually, of course, the first thing I've done was to setup auto-scanning of the DrivePool pool every 6 hours with WhereIsIt, so the next time a drive dies, I would at least be able to know what the hell I've lost. Once again - I need a third party tool to keep track of my files. I'm sorry, but this is inexcusable.
Please, do not mention the "Ordered File Placement" plug-in. Of course, drives need to be filled evenly, not one at a time. For speed, for wear and tear, what have you. Just simply, don't scatter files - keep them together.

3. A shitload of empty folders in the drives are driving me nuts. With this scattering of files there are probably more freaking empty folders on the drives than actual files. Dafuk.
4. At the very least - implement file and folder access in the balancer development (IIRC, now you can't), so I could write my own implementation of keeping files and folder on the same disks, if you don't want to...  Because this madness with file placement is making the program unusable. Reasons - below.
5. I have been using DrivePool for the last 1,5, at least - just because I am too lazy to disband the pool and then manually restore the order in all that mess that DrivePool created. But I'm finishing up writing a little program, that will do it for me, so fingers crossed... I might get back some sanity in my file system, soon. Anyone know of a tool which would help in this? Really lazy to code now...
 
Duplication
6. I've been eagerly waiting for grouping of drives, to finally make duplication usable. Yes, I said it. Because as it is now - duplication is unusable. Not even close. Anyone having 2x duplication and lots of drives and lots of files is actually bound to loose a shitload of files in case more than 1 drive fails at the same time. And with quality of drives nowadays - good luck... And if you don't believe me, think about it - I'll describe my reasoning in the following points:

Like I mentioned before, after loosing my first drive in the pool, I disabled the duplication. The reason was simple - I did some thinking. I have 50+ drives, probably a million plus files. DrivePool scatters files to as many drives as it can, whenever it can. If 2 drives die - how many files would actually have both copies on those drives (1 copy on each drive with 2x duplication). The answer is not zero. 100% - some files would get lost. And if someone offers me to do 3x duplication, then can I haz some of your money, plz... It's been too long for me to remember how to actually do the probability math, but simple logic dictates - that duplication would not help me at all to keep all my files safe, in case 2 drives go tits up at the same time. Basically, the more drives you have, the more files you have - the more fucked you are with 2x duplication. And remember... on top of that you will have no fucking idea, what files you have just lost... enjoy...

7. Now I see the drive groups arrived. And it's still does not fix the above problem. Am I understanding this correctly - you can make same old-style pools which can have duplication settings inside them and separately from one another. And then you pool those pools together? So they do not duplicate from one pool to another pool? Why the hell not? Isn't this, like, the first and only thought process, how to do this type of drives grouping, with the duplication in mind? What the hell, seriously... However many drives you have - split them in the middle by the age (how old they are) and that's your 2 groups. This way, all the oldest drives are duplicated into the all the newest drives. And you are sure, that each file has at least one copy in a new drive. Isn't that obvious? Want more than 2x duplication - split into more groups, but again, use common sense and group them drives by age. And duplicate between groups!
8. And anyway, duplication of 100TB+ or more of media files, seriously? Would love to be that rich, so it wouldn't hurt the wallet. How about implementing a SnapRAID over the pool. With cache/landing disk (or landing disks group) that would be easy. To keep all the files always secured:

files get copied to duplicated landing disks.
Then, in the background they get copied to the snapraid protected pool drives.
 Snapshot gets updated.
Now that files are properly secured on the pool by snapraid - they can be deleted from the landing disks.
Result - no wasted money and drives on a stupid duplication and files are protected way way better with multiple parities, if you want, and checksumming against bitrot and some-such. If it's impossible for you to use another free program like that in your paid software - give as an option to do it ourselves, by keeping the fucking files together. Because, I would've done things this way ages ago, if not for the stupid shit DrivePool does whenever it wants.

 
Balancing
9. Balancing. Don't get me started... Stop this fucking Apple-like magic, where program does whatever it wants, and I have no clue what the hell is going on. Instead, show it to me. Because magic-like operation is nice, sometimes, but not here. And it must work perfectly 100% of the time. If not - I'm an idiot who doesn't know what the hell is going on with his own files on his own computer. And DrivePool is far from 100% perfect. And I'm not expecting it to be. It is a mirracle, as it is, with only 1 dev working on it.
10. This bears repeating. Show me what the fuck DP is doing and especially - why. Not the same single meaningless phrase on the GUI - "moving files because whatever". But actually, moving these files from this drive to this drive, because of this balancer, or these file placement rules changes, etc...
11. Add an option to simulate balancing and let me choose what or if I wanna do it.
12. Add option to force re-balancing. Better still, to force re-balancing on individual balancer.
13. For the love of god - give me an  option to manually control (start/stop) the balancing. Let me select which drives I want to balance, with which balancer(s), so when I fire up the balancer, it would not go and do some shit I do not care for and waste time and my patience. When I change the file placement rules, I probably always want to balance something specific. Just... let me.
14. It would be nice, to able to prioritize balancing actions. Like a queue. This would be very helpful on big re-balancing jobs. That's, if  we ever get control of balancing.
 
Settings
15. SSD Optimizer title for a balancer is confusing. Change it to Landing Disk or Cache Drive or something similar. Because that's was it does, basically.
16. The minimum free space should be set in one and only one place . Multiple balancers have it - why!? Also, there is percent and size based settings at the same time - again - why? In short, leave the free space settings only in Prevent Drive Overfill and that's it, only with ability to set free space per drive basis, if we want. Drives now come in 1-10TB, most commonly, which is a huge difference.
17. Filter (file/folder placement rules) also by size. eg: maybe I would want all the metadata for movies and tv shows to be placed on the ssd, so the spinners would contain only big video files. Less fragmentation. Speed accessing metadata. That'd be neat.
 
 
UI
18. OK, let's move on onto another awful thing. The GUI. Talk about style over substance. Should I drop a screenshot of the DrivePool GUI in full-screen on a 21:9 ultra-wide screen monitor, or is it easy enough for anyone to actually imagine, how ridiculously unpractical it is, without even seeing it? Starting with having a program in 3440x1440 resolution and still not being able to see even half of my drives in a list. Stupid wasted desktop real estate galore. I have never ever seen anything like this before, ever.
19. The drives list is also wasted space. And of course - there's no way to see all your drives at once. Have too many drives - scroll through this one listview, never-mind there's plenty of space all around for a different view displaying more/all the drives in a wrap panel.
20. Disk performance view at the bottom right - you will see activity for only 5 files - and that's it. That's all you get. What the hell? I seriously wonder, how this decision got made. The main reason the program was created, was to manage files - but if you want to know anything about what's going on with those files - I'll show you only 5. Fuck the rest. You have 10 transfers... more...? Does not matter - 5, I say. Better look at that huge pie-chart, pretty, isn't it?...
21. Want to change some settings for the pool? Here's a universal icon for settings at the top on the right, nice and pretty intuitive... oh, wait... shit. That is not the options - what the hell? Oh here's options below that huge pancake in the middle. Oh, it's not a button to open the Settings - it's a menu, with sub-menus? Oh, I see, some menus are check-boxes and some actually open the settings window. What? There are multiple separate settings windows for all kinds of things.. Couldn't all of them be in one settings window, as per usual?  Oh, it would be too easy, intuitive and too simple to use, you say? OK, then...
22. I could go on like this for quite a while longer, but fuck it. There's more important things. During probably almost 2 years of using DrivePool and Scanner - I can confidently say, that 8 out 10 times I tried to control DP or Scanner on my server from my main rig - I could not. The trial on main rig expired ages ago. The drop-down menu selection to connect to server - there's no selection  but the local rig. 8/10 times there's no server option. I can't connect and control the actual DP and Scanner that I use. So now I have 2 programs installed in my main rig for no fucking reason. Beautiful.
23. I have a shit load of drives. I want to keep some folder on a single drive. I go into settings > File Placement > and either Folders or Rules, does not matter. Then I need to un-check all my drives, but one. One by fucking one. Where are the "Check All" or "Un-Check All" buttons? Which is a standard pretty much everywhere nowadays. But here - you wish, just click away... Stop using that stupid fucking Telerik (I gather it's Telerik, from the looks of it). Take Caliburn.micro and MahApps and use listviews which support all keyboard shortcuts of selecting out of the box. Done.
24. Let's say I am doing the file placement setup. Why the hell do you have 2 separate tabs for Folder and Rules? If you set the rules in Folder tab, they show up in the rules tab, which may have some other wildcard based rules as well. Is it for sorting? Because, you know - you could put it all in one tab...
25. And as always with DrivePool - the drives list in File Placement rules is in one column - scroll away... How about WrapPanel, at least here?
26. Folder tab at least shows how much of that folder is on what drive. Now why it scans for files every single time you select the folder in the list - nobody knows, I bet. Every single fucking time - pick two folders and click away between them - every time same shit - calculating. Calculating fucking what? How many minutes it would take to code one fucking button like - Scan - which would scan the whole pool once and then keep that info until you done with the settings? Fuck me. Off course here's that fucking problem, that DP does not keep the list of files and where they are on which drives. And it cannot display it to me, when it need to... Ashamed of the fucking mess it has made, probably...
27. So, because spreading the files all over the place is simply fucking stupid, I want to actually manage my file placement. I have a list of folders on my pool on the left, I have a list of drives on the right. OK. I select one of the folders from the pool on the left, and... fucking calculating... and... I get another fucking list of drives down below. Because... reasons. Now, if I want to check where actually are my files, on what disks, compared to the settings - I need to dart my tired eyes back and forth between two lists. Now, both lists have different font sizes and font weight (maybe even font family, who the fuck knows at this point), which makes it so much harder to "quickly match up". Because... reasons. Why the fuck would you not use the same list of drives on the right side, that you already have? How much easier would be to actually see straight away, which drives have been checked on/off and which actually have files from that folder in them? Elementary, my dear Watson. Of course, because it's DrivePool we are talking about - I need to mention straight away, that when I say - show the status of the files in the drive list on the right - I mean really fucking show it! Not some ridiculously small progress bar, like in the main window, where you could never see anything, if you let's say have 10GB of files on a 10TB drive. That shit would be indiscernible. Well, in this case, maybe the size column would help to fix this, at least. You could make it that it would be easy to spot if drives have any files from selected folder - change the drive item background color, or name color, whatever. Big green dot, if the drive has any files from that folder whatsoever. Anything to make it intuitive and easy to use.
28. Can I at least sort that second drive list down below? Nope. Sorted only by size taken. Imagine 50+ drives and finding some drive by name to see if it's even on the fucking list!
29. Wanna see the folder size - need to switch between a list and a pie-chart (thosetiny buttons at the top of folders list). Would it be so hard to show folder size by the folder name in the list up top? And whats with that pie-chart selection? It shows me the same size 2 times. One with a round blob, one with a tiny line. Am I missing something here?
30. Filter (file/folder placement rules) also on disks. ie: I select a disk and see/set which folders can go onto that disk.  Now, it is a pain to unselect every disk but one for every folder that I want to filter on. Especially, if I want it this way, I need to remember to go into settings and set filters on every new folder  I create in the root of the pool in the future. This filtering on disks could be an additional (opposite) way to  filter and set the rules.
31. Give us WebUI and let us control DP and others from the browser from wherever we want! It's not middle ages. And it's not that kind of program or the UI (especially as it is now) that would warrant you using WPF, and most importantly forcing me install 2 programs just to control them on another pc. That is whack.

I have 1 license of all 3 programs - DrivePool, Scanner and CloudDrive. But both DrivePool and Scanner are running on my home server, which I actually want and do control from my main rig. Now, of course, I need to install both DrivePool and Scanner on my main rig, as-well. Just to control them on the server. Fuck that...

32. WPF would be warranted for a program, which needs a hugely fast update times. All of the web UIs I use (and they are numerous) are faster than this WPF GUI now. Seriously. The GUIs now for both DP and SCanner are fucking slow as snails.
33. Constant Not Responding when opening GUIs. This embarrassing, dude... Async.
34. Give us full featured (and I mean full featured - every single command and data point) json Web API and see the community blow your socks of with their imagination and awesome web UIs. You're a 1 dev team. Use your head and delegate. With unRAID waking from the development sleep with these ground-breaking new features and FreeNAS catching up to unraid lately in the last versions (even though it's still a mess, with coral, etc...), if DrivePool wants to stay alive and relevant - make an API and let people make your program "great again"... lol But seriously... not a joke, just do it.
Please, don't suggest that there is that dpcmd thing. It's capabilities are weak. And the output format is crazy (why would you not make it easily consumable)?
35. Or if you hell bent on WPF - still, with full API, we can make a kick ass WPF UI. MahApps and Caliburn.Micro - both free and awesome - fuck that stupid Telerik.
36. Would be nice to have an option to see the log in the UI. With good filtering.
 
Other
37. At least give us as simple text log files. So I could tail them when I wanna know whats going on in DrivePool real time.
38. Probably a bug. Setting the priority of the running balancer or duplication is not displayed properly in the UI, if using multiple UIs on multiple computers. And priority resets to normal, it seems, if you close/re-open the UI which is totally annoying.
39. The last stable version was, when exactly? All I know it was before 07-Jul-2014, judging by the betas numbers in the list. And it's even buggy enough, that I never used it, and started by using beta straight away. Not by my own whim, mind you, by Chris'es recommendation. I know, there was CloudDrive to release. But 2 years? That's how software dies.
40. I'm not sure, if this crap still not solved - but you cannot save your settings with the pool, or export/import them. At least that was the case, last I checked. That is seriously messed up. If this is solved already, sorry... my bad. But I do not care to look for it, as it is especially hard to look for something that may not exist. I tried finding the info on dpcmd for about 20 minutes, when I was writing about it - could not find it.
 
 
Scanner
41. And... Scanner. I'm too tired already to go through and remember every single shit that's bugging me about scanner. But most of what's UI-wise and stuff - applies here too. Scanners UI is crap. Once again - only a single view with a single lit. Oh, and could the drives list items be even bigger, so I could see even less number of drives at the same time and would need to scroll more?..
42. Allow setting concurrent scans per case. In my case every backplane has a separate link, and could use 4 links on the LSI HBA, 1 for each backplane.
43. Or better still - allow creating scanning groups. Most people probably use HBAs or RAID cards. Let us control the scanning settings, so we could achieve the optimum performance. Stop your fucking magic I-know-better and I do-it-auto-magically, which does not work properly, like most of the time.
44. UI does not collapse the list of drives. Actually it collapses the list and immediately expands it back again. wtf?
45. Yesterday I tried to open Scanner UI, and it took more than 2 minutes to open. It always takes long, way longer than it should (for what it is), but 2 minutes - fuck that shit. I'm asking you again, would you, please, gives us an API and let us make an awesome UI? Pretty fucking please... Because this shit is embarrassing (2+ minutes). Or at least Async that shit. Just for good measure - fuck Telerik and it's shit controls, again.
 
Afterthoughts
All, but the UI stuff, which could anyone argue is more of the preference kinda thing, IMHO is essential features, which methinks DrivePool must have, being the software that it is, as in managing your files on your file system. It needs to stop being magic that does not work perfectly, and instead - gives us control and choices. On the other hand, when it comes to UI, I stand by my opinions. Give me full featured API and I'll take care of my problems myself. That's all I ask Or again, wake up and let the community write your UI and balancers and what-have-you, for you...
 
Phew... This one's got away from me
This started as a reply to the pinned thread about pool groups, but I quickly realized, that the frustration started pouring a bit too much and I had to post it as a new thread.
Sorry for all the swearing. It's because I love this company and the software. It is 100% true. I would love to be able to make a living as a one man software developer. And I have already said it in this post and in numerous others - imho, Alex is awesome, to have built these things alone. And I do not forget Chris, you, my friend are an exceptional person too. The job you 2 have done and keep doing is an inspiration. But from this love, and the want for your software to be better for you and us, your customers, this frustration had to be let out.
Please, take it as a feature request and not an attack.
I promise, I will cool off, return and edit all the fucks to something more cultured. Although, I don't want to take out all the emotion from the rant completely, and English is not my native tongue, so we'll see how that goes
 
P.S. If, whoever accused me of loving to hear my own voice as a reason for my long post last time, gonna come back with the same shit - I most definitely do not. I hate it. It takes a shitload of time for me. Like I said - English is not my strong suite. And I make typos in like every second word, and need to keep editing endlessly. Lastly, I - maybe - write 1 rant like this a year. Not 1/year here - 1 a year total. Maybe less. And only with good reason. Check my freakin post count, dude.
 
Peace.