Christopher (Drashna)

Well, that would totally do it! Though, I'm glad it was a simple fix. And don't worry, stuff like this happens. As for the serial number, grab the "DirectIO Test" from here: http://community.covecube.com/index.php?/topic/36-how-to-contribute-test-data/ Just select a drive from the drop down (let it do everything else normally). You can click on "Identify" to verify the model and serial number. If it is mangled still, check the "NoWmi" option. If that comes up clean, enable the "Smart_NoWmi" option in the advance settings for StableBit Scanner (http://wiki.covecube.com/StableBit_Scanner_Advanced_Settings) If that doesn't work, uncheck it, and check the "UnsafeDirectIo" and see if that works.

That would be complicated to program, at best. And I believe it would require a significant rewrite to accomplish that... or to disable realtime duplication.... That or set up a script to later move it from an unduplicated folder into a duplicated one.

Won't work. DrivePool is aware of volumes on the same disk, and won't use the same disk for duplication. This is because... well, if a disk fails, then you lose both copies. That's just stupid. And bad. So this applies to the feeder disks as well, so you'd need two disks. If you have a large SSD for the system disk, you could use that and one additional SSD to get this working properly. Otherwise, one of the duplicates will fall back onto a mechanical/archive disk. Also, if you're using greater than just 2x duplication, you'll need that many disks.

Well, I'm glad you were able to stare the service into submission! If that keeps up, then that's great. However, if the issue becomes an issue again, then get a memory dump of the service: http://wiki.covecube.com/StableBit_DrivePool_Service_Memory_Dump As for the balancing... that would be why. You basically have balancing disabled there... Set it to balance immediately, or at a set time. Then give it a couple of days and it should balance everything properly. As for the trace logging, I'm not exactly sure. Alex is the one that usually handles that (and is the one that wrote the code for it). However, I do believe that yes, we use WMI for the logging.

That is great news! I'm glad to hear that everything is working great and settling down.

You do have some of the "optional" balancers installed, I see. But the settings do look correct. And you'll notice the dark blue arrows on each drive? that's the "target" for them. That's what DrivePool says that it wants to get the data to. So as I said, that all looks correct. Now. for the "Settings" tab on the balancing settings window there... that may be the cause of the issue there. There are a number of options there, but the simplest would be to post an image of that, so I can take a look at it. But for the most part, we recommend immediate balancing, allowing plugins to trigger it, and setting the "Ratio" slider to 90% or higher.

Well, you can configure the StableBit Scanner balancer to only evacuate unduplicates files, or completely disable that.... Or you could mark the sectors as "good" in Scanner itself (click the "+" next to the drive, and then click the button with the green circle with the yellowish circular arrow) Though, if you're getting SMART errors on the drive as well, then it may be worth RMAing the drive, just in case. (depending on what errors you're getting).

Could you grab the file system loggging from the system then? http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection

Not a problem. We rely on feedback from customers, so bouncing ideas is a great way to help us out as well (win-win) EFS ... Well, with BitLocker support in Windows.... why encrypt one or two files? Just encrypt the whole disk. And with how dead simple BitLocker is... Why bother with just a couple of files. That, and I do remember that it did have other issues. But it's been so long since I looked at EFS, I'd have to double check. But backwards compatibility... that's very important for Microsoft. And for good reasons. While other OS's like Apple's Mac OS X and iOS, and Google's Android ... and linux can get away with removing features... just look at the uproar when Microsoft removes "bad" features. And I suspect that's why EFS still exists. And for the VerifyOnCopy option, I'm not 100% sure why it's not enabled by default. I know (or swear that it has happened) that I have discussed this with Alex before. But my memory is horrible. However, I do know that enabling the option causes a lot more issues for the user to "resolve" manually. But I think it's best to let Alex answer this one. But as for when this happens? Any time that DrivePool moves or copies a file. So during duplication, or balancing passes. As for ReFS.... I know I'm not using Storage Spaces. But yeah, I was partially aware that some of the features in ReFS are supposed be used with Storage Spaces. Though, in theory, you should be able to use Storage Spaces and DrivePool (add the "spaces" to a pool). I don't think it hast been tested, but it's a possibility. Similar to how you can add a RAID array to the Pool. But in both cases, you would absolutely lose the ability to get SMART data from the disks. Ah, yes, specifically, ReFS can detect corruption. But it can ONLY automatically correct it if a proper storage space is configured. It has to be a mirrored or parity "Space" to do so. But it can at least detect it otherwise.

Could you verify that the balancer is on the correct setting? Also, when the service is "hanging" like that, could you get a memory dump of the process? http://wiki.covecube.com/StableBit_DrivePool_Service_Memory_Dump And could you upload the log files from the system? http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection (namely, steps #5-7) Also, do you have any other balancers installed on the system?

What happens when a disk gets full? Or you need to remove a disk, or? There are a number of good reasons for why it automatically balancers. And you may want to check out the following link for a list of reasons why: http://stablebit.com/Support/DrivePool/2.X/Manual?Section=About%20Balancing However, once the files are on the disk, for the most part, they will remain there. Additionally, we've recently added the option to gain fine control over where files end up. This is for both existing and future files. And from what you have posted, that sounds like it may be what you want. http://blog.covecube.com/2014/04/stablebit-drivepool-2-1-0-503-beta-per-folder-balancing/

If it's been more than a couple of days, please feel free to bump the thread. Or PM me directly. And specifically, if the files are in identical locations in the different "PoolPart" folders, DrivePool will see that, check the files, and "handle them" based on the duplication settings. Eg: if it's the correct number and matching files, it just moves on, otherwise it will delete a copy or create an additional copy, depending. Or notify you if they're not matching parts, etc.

Glad to hear it!

You can definitely use a RAID array for this instead of an SSD. We use "SSD" for the name because, well.... that's what most are probably going to use. And yes, you could use part of your system disk for this as well. But you will want two disks if you're enabling duplication, it will fall back onto another drive. As for optimal size, that depends on usage. But basically, at least a bit larger than the largest size file that you plan on using. More most, a couple of 60GB drives should be fine. However, if you plan on using larger files than that... well, the bigger the better, then. As for the RAM Disks... that depends on how the software works. But unless it flushes the contents to disk on shutdown and re-adds them on startup.... chances are that it will cause issues. Also, as I said, you will want to have TWO drives for this if you are using duplication. Otherwise, it will write to a HDD if it can't find enough "feeder/cache" disks.

It depends on the files. If they're in the same location on both disks, it will try to verify that they are indeed duplicate files. However, if they're in different locations.... then DrivePool .... ignores them when it comes to duplication. Specifically, duplciates are kept in identical folder structures on each disk. So they have to be matching as well to be considered as duplicates.

As long as they match the other folders exactly (including permissions, etc) then there should not be a problem. But we do tend to recommend against doing this because it can cause issues.

I'm glad to hear it. I was concerned that I may have missed what you wanted to know. If you have any other questions, don't hesitate to ask.

Yeah, the manual needs work. We'll see about cleaning that up. As for read striping, yup, definitely complicated. Very. But glad to hear that it performs very well for you. Performance is one of the things that we try to optimize for (after stability, of course). And yes, bigger pipes would be nice! If this is over the network, look into 10gb NICs. They make them, and they may work great. Not cheap though. Also, there is bridging, teaming, link aggregation (spelling?). These may help boost the max throughput for you.

I'm not sure what you mean by "delayed duplication" here. Just to clarify, if "real-time duplication" is enabled (and it is by default), then any writes that occur are done to all of the files in the "set" simultaneously. Meaning that all files that are duplicated are kept in sync. Also, Duplication isn't responsible for pushing files off of a feeder disk. That's Balancing, and an entirely different subsystem from Duplication. While the two do interact, they have very different roles and do different things. However, StableBit DrivePool does check periodically to make sure that the pool is properly duplicated, and will duplicate files as need, and flag any problem files for user resolution. And duplication does use the balancer rules to determine where it should place files. If you've disabled real time duplication, then you only write one copy and it's duplicated later (2am).

Oh boy, this is a long post (absolutely no problem with that). I'll try to answer all that I can. EFS may or may not be dead. It's still supported by NTFS, so we cant' say that it's 100% dead. At best. I didn't want to get into this before, but bitrot is a very complicated subject. Very. A large part of that is because the term is only loosely defined. Or it's defined in 10 different ways, by 10 different sources. Modern drives should be much more resilient to this random bit flipping. Improvements like HP's SMART IV technology (errors reported in SMART as "End to End errors") help prevent a lot of errors. Documentation about it here: http://h20000.www2.hp.com/bc/docs/support/SupportManual/c01159621/c01159621.pdf (Sorry for the PDF link) As for DrivePool, if it detects that the files are different, it will notify you in the UI about it, and ask you to resolve it. It will list the paths of the files in question, and outputs that to the logs as well. As for the VerifyOnCopy, this is for ANY move or copy operation done by DrivePool. Be it a duplication pass, or while balancing. It will also output the errors to the UI for you to resolve. And for adding the setting to the UI, it's on the to-do list, and I'll bug Alex (the developer) about this again, because I agree that it needs to be in the UI. Additionally, if data integrity is an absolute important feature, there is an Add-in for WHS/2012Essentials that you may want to check out: http://integrity-checker.com/index.html And yes, this all comes across in good terms. It's just a complicated subject. Very complicated. As for "next gen file systems", Server 2012's "ReFS" falls into that category as well. We don't currently support it, but investigating support for it is on our "to-do" list. (if you've noticed, we have a lot that we want done, but limited resources). Adding ReFS support most likely won't be simple either.

http://stablebit.com/Support/DrivePool/2.X/Manual?Section=Performance%20Options Basically, if one or more disks are much slower, it will only read from the faster disk. As for copying from the same controller, that really depends. In fact, depending on the controller, reading from three disks on the same controller could be slower overall than reading from just one disk. It may be spreading out the IO and adding additional overhead to do so. That may not be happening, but it is a possibility. As for the different colors for the performance UI: http://stablebit.com/Support/DrivePool/2.X/Manual?Section=Performance%20UI

Well, you can tell it to scan multiple disks on the same controller at the same time. Just open StableBit Scanner, click on "Settings" and then "Scanner settings". Head over to the "Throttling" tab, and un-check the "Do not interfere with other disks on the same controller". Also, un-checking the "scan with background IO" will speed up the process, but can adversely affect performance on the system. As will changing the disk activity sensitivity. Or disabling it altogether. However, if you leave it as is... each disk is checked independently for when it needs to be scanned. This means that future scans will automatically be staggered out.

If Real-Time duplication is enabled (and it is by default), then any write that occurs, is written to both disks at the same time. You may want to check out this link: http://stablebit.com/Support/DrivePool/2.X/Manual?Section=Performance%20Options In fact, I believe that Alex (the developer) runs VMware VMs from DrivePool. And we have a few people that run HyperV VMs from the Pool as well.

Depending on how it's monitoring them, then yes, it definitely could. Though, the best way to find out would be to disable it and see if it changes behavior. Also, is anything else accessing the the disks? At all? Including StableBit Scanner? If you do have StableBit Scanner installed, throttle the SMART queries, as that *may* help. But anything that queries the drives in the pool or the Pool itself could cause that behavior. Also, by "mounting a network drive", do you mean that you map it to a drive letter? If so, I vaguely remember that this can cause access to the shares. I can't find confirmation nor relevant links, ATM though.

You want to look at the Balancer settings. There is a "StableBit Scanner" balancer there that controls all that. By default, it only moves data out of drives marked as damaged. You can then optionally enable it to move data out in the case of "SMART warnings" as well. And looks like you found it. Don't worry, that's the way it goes.