[Questions: duplication, SSDs, free space, small files and other]

A couple of days ago, I tried DrivePool and Scanner for the second time. About a year ago, scanner was not able to read SMART from my HBA and enclosures, so that was a bust and I has trouble with a bug in a release version of drivepool with saving settings while having multiple pools. So I decided no to...
I have never used any raid or parity solution before to protect my data. I tried unRaid from time to time years ago, did not like it. Recently I tried tRAID for the 3rd time and did not like it too, again. Then I looked into Snapraid + DrivePool and decided, nah.
I was always categorically against backup solution because of cost, but finally, I decided - enough. Drivepool and duplication, let's try this out.
 
After a couple of days, I have some questions:
 

1. How safe is to have 2x duplication? With Scanner running safety. I mean, over the past 10 years I've been reading again and again about all kinds of ways of data protection, from hardware raids to parity, to zfs, etc... 99% of the time you always come to the same warning - raid/parity/etc is not safety - backup is. Hence the question. Is Scanner + DrivePool's 2x duplication a way to feel sure that I won't loose data, the next time I'm gonna loose a drive?
 
2. Is drivepool's 2x duplication more secure (because of scanner) or less secure (bugs, whatever), than running 2 pools and syncing between them?
 
3. I was copying TBs of data to the pool from unpooled drives and noticed problems with small files. While copying big folders with lots of tiny files inside, like images, mp3 or even smaller, the speed was climbing up to impossible 300MB/s-400MB/s and then suddenly stopping and hanging for sometimes even a couple of minutes. Then again - copying resumes, speed climbs fast to impossible (I was copying from a single spinner, hence speed should be max ~100MB) and then hangs again. Here's the possible reasons, that come to mind:
a) I am running StableBit.DrivePool_2.2.0.639_x64_BETA. I looked through forums extensively, to find which beta is considered "stable", iirc this was said to be. But still, it's beta...
b.) I was copying from a drive with SMART sector warnings.
c) I am using Directory Opus for all my file management, not the windows Explorer. Dopus is the best thing after sliced bread, imho. I hope there is no problems using dopus and drivepool together. If it's between dopus and drivepool - dopus hands down, sorry. It's too powerful and time saving for me to get back to using Explorer.
d) Maybe it's some settings I should disable/enable to avoid this behavior?

e) Edit #1. Very important! Similar thing happens with big files as well. I was copying large files from the pool to outside the pool. Again - this time from the pool to outside. The files were ~1.5GB in size. For the first 5-10 minutes Dopus was copying at about normal rate ~100MB+/s, but it was stoping and completely hanging every 5-10 seconds or so. I mean, it showed "not responding" when the copying hanged, then it resumed again, like nothing happened. And again and again, for ~10 minutes. Then it normalized and did not stop/hang anymore, but the copying speed dropped to about half of the normal ~50MB/s. Basically, the copying was done from WD Red 3TB to Hitachi 4TB drive, and the only other activity on any of them was Kodi was streaming an episode from WD Red (source) drive, which is insignificant, imho... Both drives are quite new and aOK.

Is this a normal drivepool behavior? Coz it does not look good and is annoying. Hanging the whole Dopus is enough by itself. But having only half the speed of drives while reading from the pool? Not good at all. I hope it's something on my system, that I can fix, please let me know.

Edit #2. 15 minutes later. Copying from the same place to the same place (from the pool, which happens to be same WD 3TB drive to the same Hitachi 4TB) and everything looks ok. Full speed ~100MB+/s and no stops and hangs.

Edit #3. Nah, after a cuple of minutes, speed drops to about half and stays there, again.

WTF is going on? Should I stop using Beta version of drivepool or is it Dopus (it shouldn't be), or this something else?
 
4. I was considering adding an SSD to the pool and using file placement rules to put all metadata for my movies and tv shows on it. I mean using filters like "*.jpg", "*.nfo", etc. I was thinking, the media files are big, and they pretty much never change, but meta is small and gets regularly updated, so that would possibly help with defragmentation. Having meta files separated onto SSD, would it be a better? Or possibly worse, as it would most definitely create an enormous amount of additional "merging work" for the pooling driver?
 
5. Let's say I put all meta onto SSD with file placement filters. If I enable duplication, that probably means I need 2 SSDs to be able to duplicate the meta and keep the SSD bonuses while using the meta files in the pool?
 
6. Today arrived my order of new 6TB drives. My first 6TBs ever. I was wondering, how much free space should I leave on these so not to get NTFS in trouble? I have 4TB, 3TB & 2TB drives in the pool atm. The setting that I use now is set to leave 200GB of free space. Would 200GB be enough for 6TB drive. Or, basically, the age old question: how much free space to leave on the drives? Should I leave the same amount in GBs on all size drives, or should I set the option to leave eg: 5%? I would not like to leave 10%-15%, like some people say, because I'd hate to loose that much space. Especially considering how much space is already lost on duplication. Jeez.
 
7. There are 2 balancers that have an option to set free space to be left on the drives:
Drive Usage Limiter, which only has percent setting. I have this balancer disabled atm.
Prevent Drive Overfill, which I have set to: try not to fill above: 90% or 170GB and Empty to 85% or 200GB. From the info describing how it works, I gather my settings ignore percent and use 170GB and 200GB settings. Am I correct?
 
8. If I want to enable Drive Usage Limiter, but keep size based free space setting, I should  move Prevent Drive Overfill to be above Drive Usage Limiter, right?
 
9. Long story short: I have 4 "groups" of drives:
oldest: 10+ a mix of old 2TB drives
older: 8x Seagate 3TB drives
newer: 16x WD Red 3TB drives
brand new: 6x WD Red 6TB drives
What I would like to achieve is to "control" how the duplication woks. I would like to set "drive groups" of drives:
a) a group that holds non-duplicated files. That's the newest drives (my thinking, the newer the drives, the less chance to fail). That should be easy enough, I guess: just use Drive Usage Limiter and uncheck Duplicated for those drives?
b.) now this second scenario I gather is not possible atm, please confirm. I would like to set 2 groups for duplication: one group of newer drives for "primary copy" and second group for "duplicates". This way I could be sure, that all files have one copy on newer drives. As it stands atm, I guess drivepool would duplicate files onto random drives, hence I would have files with both copies on newer drives and with both copies on oldies...?
c) I would very much like to ask for this feature to set drive groups be added to drivepool in the future in a form of balancer, most probably. From what I gather how drivepool works, that should not be difficult, methinks. Additionally, you could enable setting groups as "primary" and "copies". This would have additional positive effect: you could disable real-time duplication and would always use "primary" group of newer, faster drives in real-time everyday file management, and files would duplicate to slower drives in the background...
 
10.  I would like to ask for a feature in the drivepool's UI to display additional "free space" statistic. What I mean, is maybe it's me, still not used to pooled drives, but it's kinda confusing how much free space I really have in the pool. My pool shows 17.6 TB of free space, but considering that I have 32 drives in the pool atm, all keeping 200GB of free space , I always need to remember and do the math to realize, that actually I have FreeSpace - NumberOfDrives*200GB = RealFreeSpace. That's 6.4 TB in my case atm. But as I'm gonna constantly add drives and sometimes take some out, this number will constantly change and I'll constantly need to do the math.
I propose additional pie chart section, something like "Reserved Free Space" and then drivepool could show Free Space as in now (full) and additionally, calculated real free space left from FreeSpace-Files-ReservedFreeSpace=HowMuchSpaceIhaveLeftOnThePoolForRealz
 
11. I've read someplace on the forums, that exporting and importing of drivepool settings is coming. Very nice to hear, as this feature imho should be in every software, period. Also, why not save pool settings on the pool drives themselves? Duplicate for safety. Then, after windows reinstall, or after moving to a new machine, just pop-in the drives, and pool spins up with all the settings...
I was really surprised when I first realized that it wasn't this way already. I mean it as a compliment, actually. DrivePool and Scanner just have this quality feel and a whiff of some very good thinking and work put into in to it. Hence, I'm seriously baffled about annoying inability to quickly get away with the settings.
 
 
Oh, and CloudDrive? Seriously? One programmer doing this? All I can say - F**k you, Bitcasa. Anyone who knows what bitcasa was?is?whocares..., knows what I mean. F**k you, Bitcasa, eat your heart out...

[Questions: SMART, predictability, concurrent scans, recovering files and other]

I have been trying out Scanner for the past 3 days. Love it. Tried it about a year ago, it couldn't get SMART for all my drives from the HBA and the external enclosures. Now it does. Woot woot.
I have a couple of questions though:

1. How safe is having no duplication in the DrivePool while running Scanner? I know there cannot be a definitive answer, but just from experience... How much chance is there that scanner will be able to warn me before the drive goes R.I.P.? I am asking for the very cost conscientious reason: do I or don't I duplicate my 50TB+ movies collection? Movies are pretty easy to replace. Although, it takes time and some tedious work.
Case A: If there is little to no chance that scanner will warn me in time, I may need to bite the bullet and buy a bunch more hdds, which gets too expensive quite fast... At least I save on time and avoid potential headaches.
Case B: Scanner is quite good at predicting disasters and I save a ton of money. That would be awesome.

2. Why is there no indication of SMART warning in the list of drives? Is the system tray notification the only place for it, and if so, is it reliable 100%? I'd hate to miss some disaster warnings.

 

3. Most of my drives are in 4 external 16 bay enclosures, connected via 4 external mini SAS SFF-8088 cables to the LSI 9201-16e HBA, which to my knowledge has 4 dedicated 6Gb/s links, one for each port. In this case scanner should be able to run 4 concurrent scans on the HBA without penalty, right? Please correct me, if I'm wrong.

 

4. Which option in the settings is for setting more than one concurrent scan job per controller? Advanced Settings > Configuration Properties > ScanMaximumConcurent ?

 

5. Is there any way to set "drive groups", so that scanner would not run concurrent jobs on the same link?

 

6. Or maybe scanner is smart enough and is able to test and auto-magically detect which drives to scan when, so they would not bog down one another? I see something probably related to this in Settings > Throttling > Do not interfere with other disks on the same controller (also Clear performance data..., whatever that means).

 

7. How does "ignoring SMART warnings" work? I see there is an option on the system tray popup window: Never warn me about SMART warnings/failures. I guess this one is pretty self-explanatory - I check this and never ever get warned about SMART issues, yes?

 

8. There is a button Ignore warning... at the bottom of the SMART popup window. What does this one do exactly? If I press it, would I ignore only current warning(s) or all future warnings for this drive? What about this same warning, but getting worse, eg: increased pending, reallocated or uncorrectable sector counts?

 

9 . About bad sectors. I have one drive's SMART reporting:

Reallocated Sector Count: 0

Current Pending Sector Count: 1

Uncorrectable Sector Count: 1

What should I do? I have read many people opinions, that when u start getting bad sectors, it's a slippery slope, kinda like snowball effect and the drive is pretty much considered doomed... How true is that? Should I bin it? If no, how do I fix it? Full format or chkdsk /r or maybe scanner is actually able to fix it? I moved all files but 1 from the drive already. 1 was giving me cyclic redundancy errors. I was thinking of using SpinRite to try and recover it, if you have any suggestion for a better way to recover the file, please let me know.

 

10. Scanner warned me about another 2 driver with way more severe sector counts:

Reallocated Sector Count: 0

Current Pending Sector Count: 331

Uncorrectable Sector Count: 0

and

Reallocated Sector Count: 0

Current Pending Sector Count: 1464

Uncorrectable Sector Count: 1025

What is confusing, is both drives, even the second one shows the same messages for all sector count parameters: "The drive is operating within manufacturer specified tolerances." What the hell? Does this mean these drives are considered in good condition and usable? I was not able to move ~30 files from the second one, cyclic errors. Gonna need to try and recover also.

 

11. From reading about scanner in this blogpost: Why using StableBit Scanner is a good idea, I gather scanner has some ability to try and recover unreadable files. How does that work? If it succeeds, where does it put the files? How do I do it? Is there a special command/task to launch? Should I try spinrite if scanner fails to recover? Or is it futile?

 

 

 

Regarding questions 5 & 6: if it is not possible to set drive groups and scanner is not smart enough on it's own, I would very much like to add a request for this feature to be added to scanner in the future. I am thinking in the future terms, having 60+ 6TB or 8TB drives on one controller, scanner won't be able to complete scans on all drives in a period of 30 days...

[Re-installing: new pools, seeding, folder rules, no balancing, beta?]

I had troubles with my first attempt of using DrivePool: file placement rules gone, and Christopher (thank you) advised me to use the newest beta to fix this. So, now I have to reinstall/upgrade the DrivePool program and I would like to recreate my pools from scratch, changing some of the settings. But before hand, I would like to ask some questions, to double-check a couple of things, as I'm gonna be running DrivePool on my main server full of data, which I very much wouldn't like to loose. And running beta version is making me even more uncomfortable.
 
My server:

• Intel Xeon E3-1230 V2
• SUPERMICRO MBD-X9SCM-F-O
• Kingston 32GB KVR16E11K4/32
• LSI 9201-16e
• 4x SGI Rackable SE3016
• ~50 drives atm (~70 eventualy): 4TB, 3TB, 2TB and a couple of SSDs
• Win 7 x64 atm, will re-install win 8.1 x64 next week probably
• Usage: downloading, serving media to HTPCs, backing up other PCs around the household.

What I want to accomplish with DrivePool:

• I will NOT be using any duplication, probably ever. I use syncthing for backups.
• I am gonna make 4-5 pools of various sizes (from 3 to max 16 drives each).
• I don't want any balancing, except StableBit Scanner evacuating the drive, if it shows signs of trouble. StableBit Scanner looks pretty good for now, and probably gonna buy a licence for that one, too.
• I want DrivePool NOT to split folders between the drives, if possible. I explain that one below:

This is an example of how DrivePool split my TV shows on my first attempt of pooling, which is wrong and inconvenient for many reasons. For this example, let's assume my File Placement rules for root folder "TV Shows" is to put files on drives 1 & 2:

Drive 1
    TV Shows (this root folder can be split over drives, obviously)
        Blacklist (this folder should NOT be split over drives, but it is...)
            season 1
                episode 1
                episode 3
                episode 5
                ...
Drive 2
    TV Shows
        Blacklist
            season 1
                episode 2
                episode 4
                episode 6
                ...

How I would like it to be, is to keep files that belong to the same folder on the same drive:

Drive 1
    TV Shows
        Blacklist (now it is not split, as it should be...)
            season 1
                episode 1
                episode 2
                episode 3
                episode 4
                episode 5
                episode 6
                ...
Drive 2
    TV Shows
        Some other show...

Questions:

• The most important question: is there any situation, where by using DrivePool I could loose files? I am talking drivepool bugs (especially using beta version), etc? I tried unRaid years ago, but stopped using, I kept checking in on FlexRaid and now tRaid, but never properly tried them, because of many many people with troubles on the forum. Now finally I chose to try DrivePool, because on the surface it is simple enough and especially, without using duplication, on paper it should be reasonably safe. So I am just double checking...?
• Files are stored as regular NTFS files? ie: if for any reason - machine craps out, windows re-install, stop using DrivePool all together, etc - I simply unhide PoolPart.xxx folders, move my files out to the root of the drives, delete empty PoolPart.xxxx folders and that's that?
• How do I go about upgrading to latest beta? Install beat over current version, or uninstall current version first?
• Very important: I would very much like to do "seeding" of the data to the pool, instead of adding empty drives and copying >100TBs over for no good reason. All the data on all my drives is properly sorted and I would like to keep it that way. Is it actually safe to use "seeding" method, described here: StableBit DrivePool Q4142489 - Covecube - Wiki? I mean is it simply to safeguard people from making some stupid mistakes, or is it actually unsafe and maybe be buggy, or some-such?
• In the wiki (above) in the step 2 I would stop the DrivePool service. But later on, in step 4, it says, that when I move the files to the PoolPart.xxxx folders, I can see the changes and files in the pool right away. How come? Does DrivePool UI work even without the service? I'm just curious, so what does service do?
• The same wiki, step 6 is to start the service, but step 5 points to StableBit DrivePool Q2299585B - Covecube - Wiki, which says, when you "Reset all settings..." the service restarts by itself and starts rechecking. So... I guess, I'll see?
• If I don't wan't any of the files moved after seeding the pool, and I would prefer all the balancers, except Scanner, disabled, would't that mess up my system, if DrivePool starts measuring, and balancing the files automatically, after the service restarts?
• Is it possible to achieve that kind of file placement folder rules, that I have described in the above example? I looked through all the balancing plugins, should I disable most of them?
• StableBit Scanner - always on, right?
• Volume Equalization - disable?
• Drive Usage Limiter - this should not interfere with my organization system, I guess, enabled?
• Prevent Drive Overfill - this one's kinda does the same as the one above, enabled?
• Duplication Space Optimizer - disabled?
• SSD Optimizer - nope.
• Disk Space Equalizer - no, no, no.
• Ordered File Placement - this one sounds like the kind I should install, but I gather I MUST change the settings according to: OrderedFilePlacement/Notes.txt?:
  • UNCHECK "File placement rules respect real-time file placement limits set by the balancing plug-ins."
  • CHECK "Balancing plug-ins respect file placement rules."
  • UNCHECK "Unless the drive is being emptied."
• If I uncheck "Unless the drive is being emptied.", won't it prevent Scanner from moving the files out of dying drives? Or am I misunderstanding what "emptying the drive" means?
• Which balancer actually was responsible of placing episodes from the same folder on separate drives?
• I could not find any 3rd party plugins, are there any? All I could find was those 3 by Covecube...
• If it's not possible to achieve file placement and organization the way I'm looking for, I would be interested in writing a balancer which would ensure files would not be scattered on different drives. You could set "parent" folders in the settings. Parent folders could be spanned across multiple drives, but child folders of those parents, should never be split, except if the drive is full, etc. Would that be possible?
• Actually what takes precedence over what? Balancers or File Placement rules? Because it is a little confusing, the options UI, as it stands now, maybe just because I'm new to this.
• Finally, I am a little worried, that the reason why I'm forced to use Beta is basically, wanting to have multiple pools (as the bug is related to multiple pools). I mean, multiple pools is a core feature, imho, and the fix was probably trivial, looking at the release notes, why not release a new version? It's been, what, 7 months since last release? I'm simply trying to determine, if DrivePool is an active peace of software or is it like for example, unRaid - pretty much one guy calling himself "we/us" and "company", and most importantly, him being gone from the project for years, while everyone's waiting for the ages ago promised new version...?

 

Anyway, if anyone's still reading, this overly long post, thank you for your patience

And thanks in advance for any answers...

[Help: all the File Placement rules are gone from one of the pools]

I am new to DrivePool, started using it 5 days ago, so please forgive my potential ignorance.

 

I have 2 pools defined now, planning to have 4 or 5:

• Movies: 14 drives (28.2 TB). This was my first pool. It was 9 drives before and the problem happened when I added 5 new drives to it.
• TV Shows 4 drives (10.9 TB) This one was 2nd.

This is my first time ever pooling experience, I was looking into FlexRaid for a long time, but never tried it, because I was always worried with it not being stable enough with all them people having all kind of troubles (judging from the forums). Finally I decided to try DrivePool, it looked simple and stable enough.

 

So, everything worked perfectly and I was very happy with DrivePool, until today. My 1st pool (Movies) had 9 drives in it and I had about ~15 "File Placement" rules defined for all the "root" folders on that pool. I used those rules to manage my categories, ie: what movies were placed on what drives (movie sets, animation, etc...).

Today I added 5 new, freshly formatted and empty drives to that pool. I went to the Balancing > File Placement to set the rules for those new drives and found that all the previous rules that I had set up were gone!

 

To be honest, this makes me quite uncomfortable. I have a shitload of drives on the server (>50) and I don't wanna "play" with that much data.

 

Questions:

1. What happened? Is this a bug?
2. Is there something I don't know and did wrong? Like the rules are reset if new drives are added, etc (hope not, because that'd be stupid)?

3. Related to #2: are there any scenarios, where rules get auto reset or changed? I mean is there anything I should not be doing?

Thanks in advance for any help.