Alex

I'll see if we can put in the Dashboard integration into the EXE setup package, like we do with StableBit DrivePool 2.X. (New CCR: https://stablebit.com/Admin/IssueAnalysis/1161 ) I'm currently experimenting with a "Metro" theme for the StableBit Scanner when installed on Windows 6.2+. You can see this in action in the latest internal BETAs ( http://wiki.covecube.com/Downloads ) I've fixed the case as well in the latest internal BETAs.( Completed CCR: https://stablebit.com/Admin/IssueAnalysis/1162 )

This is a very interesting thread and I think that if there is a performance issue with StableBit DrivePool we need to narrow it down to one particular scenario. For example, as bblue has suggested in the original post, Hyper-V running WHS2011 with a few pass-through drives. As far as some of the other issues that have been raised, let me comment on those: The performance pane uses our file system driver to measure performance with some limited input from the Windows Performance Counters. Some of the earlier BETAs were combining our data with the Windows performance counters in a larger sense, but that type of presentation became a bit confusing and inconsistent. The StableBit DrivePool UI is meant to show you a quick overview of what's happening on the pool, so currently it uses its own high performance real-time logger that is only active when someone is looking at the performance data and some limited input from the disk performance counters to indicate disk activity. The balancing plug-in settings operate in the system service and have really nothing to do with real-time read/write performance. Of course, except for their ability to tell the file system to avoid certain disks (according to fill limits). For example, if you tell the system to use a disk that is connected over USB 2.0 (albeit, an extreme example), then you are slowing the pool down. The point being, there is no additional "load" or resource consumption necessary to impose certain balancing rules in real-time. Background balancing does consume resources, but if background balancing is not running, balancing setting alone do not require some extra "processing". This is the nature of the balancing system and it was designed like this for performance. As for background balancing, that is optimized to not interfere with existing I/O as much as possible. The reasons are as follows: The pool remains full accessible when balancing is running. You can safely shutdown and restart the system while balancing is taking place. You will never receive "Access denied" or any type of other "In use" error message due to balancing. The background balancer uses Background I/O disk scheduling, which allows the OS to gracefully throttle it if something else needs the use of the same disk. As a consequence, it will not operate at maximum throughput. This can be turned off if you wish using advanced settings (http://wiki.covecube.com/StableBit_DrivePool_2.X_Advanced_Settings). Ok, I think we're veering off topic a bit here, but I just wanted to make it clear that background balancing and background duplication throughput speeds are never at maximum throughput, by design. You can turn this off. As far as read / write performance, this has come up now and again. So far I've never been able to recreate a scenario, in a controlled test, that has shown the pool's performance to be slower. From a theoretical point of view, here's how writing works: An I/O request comes in from some application (or the network). This I/O request is associated with some open file (opened previously). It has some data, an offset and a length. We take this request and forward it to NTFS using a thin wrapper module, all done in the kernel. That's pretty much it. So from a purely architectural point of view, there should be no discernible delay. Now I started this post by saying that if there is a potential performance issue here, that we should concentrate on recreating one specific scenario. I think that if something is going wrong it most likely is not in the read / write path. Those things are just very straightforward. Renames are fairly complicated so they might be worth considering as the culprit. There is really no such thing as a "copy" rename in the kernel, so StableBit DrivePool always renames to the same disk which should be instantaneous. I'm going to try a test to see if I can reproduce the scenario as it's outlined in the first post. IAR: https://stablebit.com/Admin/IssueAnalysisPublic?Id=1154

The Disk Performance area shows the top file operations that are operating on the busiest disks. The file at the top is operating on the disk with the most active I/O request (i.e. the disk which is the slowest to respond because of I/O load). Each file can either be read from or written to, so I guess I'm not understanding the request for showing "write files" vs. "read files". A file can either be written to or read from, it's still the same file.

These are not auto nightly builds but are deliberately built after a Code Change Request (CCR) is completed (see: http://wiki.covecube.com/Development_Status#Development_Workflow). The build process is automatic and the publish is automatic. The difference between these internal BETAs and the BETAs that are published on StableBit.com is that the internal BETAs don't undergo any kind of functional testing before a build is published. The BETAs published on StableBit.com undergo basic functional testing and the installer's upgrade procedure is tested on every supported OS. This takes a bit of time to accomplish so it's only done for public BETAs.

That's a good suggestion. What the Scanner should probably do IMO in these cases is simply rename the bad file part to something else and let DrivePool's duplication system kick in and reduplicated the file. I'm always worried about deleting files. This way we retain all 3 file parts and the user can delete the bad one. I've flagged this as issue #86 and it will be fixed.

Hmm... could be an optimization issue with the sector map drawing code. I'll take a look at reproducing this right now. We've got an unlinked IAR open for this issue (#88). IARs are our new workflow for handling bug reports and applying fixes. Normally, IARs are linked to contact requests @ stablebit.com/contact so you can get updates on your issues that way. I should really write a blog post about our new internal workflow for handling bugs Oh, and as far as logging performance issues in the UI, yes we do. We have a sort of built-in performance profiler in all BETA builds of the StableBit Scanner. It can be shown by pressing 'P' while the focus is inside the window. If you could take a screenshot of that, along with your sector map it would might me understand what's going on. I've attached an example of what this looks like.

You have a good point, there really should be some kind of notification at the end of the removal process. I've added this to the roadmap: http://wiki.covecube.com/Development_Status#StableBit_DrivePool_2.X We will add this to the UI in the future. For now, you can look in the PoolPart... folder on the damaged drive that you just removed. Any files that were not migrated will still remain in that folder.

Bad sectors will get remapped automatically by the drive without your intervention. I've experienced this first hand just a few weeks ago. Here was my experience: A Western Digital Green drive developed a single bad sector. The SMART data confirmed the bad sector that the surface scan has detected. I was notified promptly and began a file scan. The file scan revealed that no files were affected by the damage. At that point I left the system alone for a few hours and then decided to re-check the bad sector to see whether it was still bad. The sector turned green and the SMART data no longer reflected the bad sector. Whenever something tries to write to a bad sector, the drive will automatically swap in a new known good sector from the spare sector pool and take the bad sector out of service. This is done silently and automatically. This might have been what happened in your case. Normally, the Scanner will re-scan bad sectors automatically and flag them as good after some number of day (depending on your settings). Was it a false positive? One common cause of false positives is when you abruptly disconnect a drive while it is being scanned. This will cause the Scanner to flag the currently scanning sectors as bad (since they're unreadable), right before getting notified that the drive was unplugged. The Scanner will at this point flag those "bad" sectors as "unchecked". As far as your error reports: There is an issue trying to add a firewall rule for remote control. This looks like a COM issue in Windows. If you don't use Remote Control you can ignore this, or even turn that feature off in Scanner Settings. A drive seems to have been abruptly disconnected while it was being queried for its power status, before we query it for SMART data. At least on one occasion, the system drive did not report the controller that it was on. This is the strangest of the bunch. If these errors continue and impede functionality then open up a contact request @ http://stablebit.com/Contact , mention this post to get the case forwarded to me and we can do remote support.

Performance considerations with StableBit DrivePool: For non-duplicated files with the default performance setting enabled, there should be very little overhead for read / write I/O on the pool. This is because all pooled I/O is processed in the kernel and our file system driver uses Neither I/O, Fast I/O, oplocks and a full suite of file caching. These are optional under the hood performance optimizations that are available to all kernel file system drivers and they are all implemented and utilized by our driver. While there is always some overhead introduced by any kind of software storage pooling / redundancy package, I don't expect our pooling solution to have any significant impact on performance, when comparing pooled vs. non-pooled I/O on the same physical disk. The network I/O boost feature may reduce peak throughput and introduce higher CPU usage. The network I/O boost feature was designed to increase responsiveness of media streaming clients on a busy server and doesn't necessarily increase maximum throughput. In other words, we prioritize network I/O read requests over local read requests. This requires us to perform some additional processing on each network read request and so it does consume some extra CPU resources. While this does introduce overhead, for all practical purposes, it really is minimal and should not impact file transfer speeds in any appreciable way. Here is another test on the pool with network I/O boost enabled (using the same setup as above): Version 1.3.1 OS Version: Windows 8 Processor: AMD FX(tm)-8350 Eight-Core Processor Date: 02/03/2014 Time: 12:19:20 Program Parameters: 0 High Performance Timer: 0.0000000698 Test File: \\192.168.0.2\Pool-SpeedTest\NW_SpeedTest.dat Write Time = 7.5035405 Seconds Write Speed = 533.0816880 Mbps Read Time = 7.5083370 Seconds Read Speed = 532.7411440 Mbps Test File: \\192.168.0.2\Pool-SpeedTest\NW_SpeedTest.dat Write Time = 7.1603575 Seconds Write Speed = 558.6313200 Mbps Read Time = 7.3564616 Seconds Read Speed = 543.7396640 Mbps Test File: \\192.168.0.2\Pool-SpeedTest\NW_SpeedTest.dat Write Time = 7.6143740 Seconds Write Speed = 525.3222400 Mbps Read Time = 7.5884258 Seconds Read Speed = 527.1185520 Mbps Real-time file duplication will have an impact on write performance because we have to send all write requests to more than one disk. We do this in parallel by sending the request to multiple disks at the same time. Performance considerations with the StableBit Scanner: Active disk scanning will have an impact on other read / write I/O performance to the disk being scanned. How much of an impact depends on the StableBit Scanner settings. Background I/O priority scanning ensures that Windows tags the read / write requests issued by the StableBit Scanner with the Very Low I/O priority. I/O priority is utilized in the kernel by the storage drivers in order to prioritize read / write requests to and from the disk. Scan throttling will actively pause scanning when other disk activity is taking place on the disk being scanned. This works best when Background I/O priority scanning is enabled as well. Scan throttling has multiple sensitivities, Low, Medium and High. Low means that you want the StableBit Scanner to allow some other I/O to the disk without pausing scanning. High means the opposite.Scan throttling can also monitor for bus saturation conditions, whereby scanning of one disk is slowing down I/O requests to another disk that is on the same bus. If this is the case, scanning will be paused until I/O on the other disks subsides. Quick Settings in the Stablebit Scanner and performance impact: Desktop mode - Throttling (Med) and Background I/O are enabled. Scans happen at any time. Laptop mode - Throttling (Med) and Background I/O are enabled. Scans happen at any time, but only when plugged into an A/C outlet. Server mode - Throttling is disabled, background I/O is enabled. Scans happen only at night. Tablet mode - Throttling (High, no bus saturation detection) and background I/O are enabled. Scans happen at any time, they occur more often, but only when plugged into an A/C outlet. You can obviously tweak these to your linking, but the defaults should work pretty well. All in all the system is designed to be fully automatic and very hands off.

Hey guys, I've set up and run a controlled test using the same tool of pool vs. non-pool speeds and StableBit Scanner vs. no StableBit Scanner. Here is how I set up the test server: Server to client connection is on a 1 gigabit link connected through 2 switches. Created 2 shares, one on the pool and one not on the pool. No file duplication enabled. Made sure that the non-pooled share is on the disk part of the pool with the most free space. This ensures that the same physical disk will be used to allocate the test file in both the pooled and non-pooled test cases. Initially, the server has the StableBit Scanner installed and started. Made sure that the StableBit Scanner is not actively scanning any disks at this time. StableBit DrivePool is set up with the default performance settings (no Network I/O boost). Made sure that nothing else is using the physical disk while the test is ongoing. All tests were performed with a 500 MB file. It is important to ensure proper test condition in order to prevent skewed results. Now I ran 3 tests on the non-pooled share from the client: Version 1.3.1 OS Version: Windows 8 Processor: AMD FX(tm)-8350 Eight-Core Processor Date: 02/03/2014 Time: 11:36:09 Program Parameters: 0 High Performance Timer: 0.0000000698 Test File: \\192.168.0.2\NonPool-SpeedTest\NW_SpeedTest.dat Write Time = 6.7337975 Seconds Write Speed = 594.0184560 Mbps Read Time = 7.6298019 Seconds Read Speed = 524.2600080 Mbps Test File: \\192.168.0.2\NonPool-SpeedTest\NW_SpeedTest.dat Write Time = 7.1907796 Seconds Write Speed = 556.2679200 Mbps Read Time = 7.7817525 Seconds Read Speed = 514.0230320 Mbps Test File: \\192.168.0.2\NonPool-SpeedTest\NW_SpeedTest.dat Write Time = 7.4736517 Seconds Write Speed = 535.2135920 Mbps Read Time = 7.4989738 Seconds Read Speed = 533.4063200 Mbps Then I ran 3 tests on the pooled share: Version 1.3.1 OS Version: Windows 8 Processor: AMD FX(tm)-8350 Eight-Core Processor Date: 02/03/2014 Time: 11:38:09 Program Parameters: 0 High Performance Timer: 0.0000000698 Test File: \\192.168.0.2\Pool-SpeedTest\NW_SpeedTest.dat Write Time = 7.2969775 Seconds Write Speed = 548.1721680 Mbps Read Time = 7.6963353 Seconds Read Speed = 519.7278800 Mbps Test File: \\192.168.0.2\Pool-SpeedTest\NW_SpeedTest.dat Write Time = 7.1262697 Seconds Write Speed = 561.3034800 Mbps Read Time = 7.4837478 Seconds Read Speed = 534.4915520 Mbps Test File: \\192.168.0.2\Pool-SpeedTest\NW_SpeedTest.dat Write Time = 7.2078453 Seconds Write Speed = 554.9508720 Mbps Read Time = 7.7060378 Seconds Read Speed = 519.0734960 Mbps Then I stopped the StableBit Scanner service and ran another 3 tests on the pooled share: Version 1.3.1 OS Version: Windows 8 Processor: AMD FX(tm)-8350 Eight-Core Processor Date: 02/03/2014 Time: 11:41:06 Program Parameters: 0 High Performance Timer: 0.0000000698 Test File: \\192.168.0.2\Pool-SpeedTest\NW_SpeedTest.dat Write Time = 8.1331414 Seconds Write Speed = 491.8148800 Mbps Read Time = 7.6461024 Seconds Read Speed = 523.1423520 Mbps Test File: \\192.168.0.2\Pool-SpeedTest\NW_SpeedTest.dat Write Time = 7.5164256 Seconds Write Speed = 532.1678400 Mbps Read Time = 7.5272389 Seconds Read Speed = 531.4033600 Mbps Test File: \\192.168.0.2\Pool-SpeedTest\NW_SpeedTest.dat Write Time = 7.2955172 Seconds Write Speed = 548.2818960 Mbps Read Time = 7.5415770 Seconds Read Speed = 530.3930480 Mbps As expected, the speeds are roughly equivalent in all 3 cases. I'll post some more information about Stablebit Scanner / DrivePool performance shortly.

As of 1/28/2014, in an effort to streamline the development process and to centralize the management of planned feature requests I've moved the current development status to our development wiki. You can access it here: http://wiki.covecube.com/Development_Status In addition to implementing new features we now have a separate system for identifying and tracking existing issues. See the wiki page for a bit more on that.

For unduplicated files, StableBit DrivePool forwards all file I/O to the individual disk that the file is on. If the file exists on more than one disks (i.e. is duplicated), then: All file modification requests (such as writing to the file, setting its attributes, etc...) go to all of the disks that the file is on. Read requests will either go to the first disk that the file is on (when read striping is disabled), or to one of the disks that the file is on, as determined by the read striping algorithm. A directory listing operation works by querying all of the disks in parallel. This will force NTFS to read the MFT directory indexes on all of the disks where the directory being listed exists. These indexes can be cached and can theoretically be served entirely from RAM. Opening a file is similar to a directory listing. This will tell NTFS to query its directory indexes on all of the disks part of that pool in order to locate the disks that contain that file. This also is done in parallel and can be served by the system cache. So in short, StableBit DrivePool may spin up disks in many circumstances, I haven't really done testing to see how often this is.

Absolutely awesome. USB 3.0 and Zotac are amazing. This is why I wrote DrivePool

Yes, absolutely temporary. The stablebit.com server is running on a dedicated Windows Azure box with an automatic DNS failover to another shared hosting account. Today, both failed for about a total of 4 hours. I'm not sure why the Windows Azure services failed, but restarting the servers fixed the problem. The backup host is experiencing a DDOS attack at this time and is still not working. stablebit.com is back up but bitflock.com is still down. I'm monitoring the situation closely. Sorry about the inconvenience.

Today we've had a BitFlock disruption. BitFlock.com was down for about 6 hours becasue of a domain name issues. All access is now restored and you should not experience any further issues accessing bitflock.com. The problem was at Google.

Ok, so after I posted that I realized that I didn't explain at all what I meant in terms of MBits. Most people are aware of Megabytes and Gigabytes (because that's what hard drive space is measured in). I was talking about network bandwidth. Network bandwidth is typically measured in Bits. 1 byte has 8 bits. So in order to convert bits to bytes you need to divide by 8. But the math aside, MoCA is "up to" 100 Mbits /s Which means that it can transfer "up to" 12.5 MB/s. Twelve point five mega bytes per second. Yes, megabytes the same stuff that's on your hard drive.

Mega BITS I know, most people find it very confusing.

I use MoCA myself. It gets close to 100 Mb for me. If you don't need anything faster then it's a great solution.

Kihim, Thanks for uploading the dumps. I've received dumps from 2 people with the same symptoms of stuck on "Initializing", but the issues do not appear to be related to each other. I've made some changes to the code based on both dumps to try and prevent both issues. You can get the latest internal BETAs from here (currently at 2957): http://dl.covecube.com/ScannerWhs2/beta/download/ http://dl.covecube.com/ScannerWindows/beta/download/ I haven't published these yet to stablebit.com because I'm collecting feedback as to whether the fix is effective. Thanks, Edit: Actually the .wssx of 2957 is still building, it will be up in a few minutes.

The StableBit Scanner already does a "bus" performance test in order to determine the estimated maximum throughput of the bus that each disk controller is capable of. While we don't do an automatic sequential read / write speed test, I suppose that those could be added. I'll think about how this can be implemented. The upcoming per-folder balancing feature in StableBit DrivePool should let you designate which disks to use for each folder, but that would be a manual task and I understand that what you're suggesting would be completely automatic and not really the same thing.

Future Developments for StableBit DrivePool The current development status for all of our products can now be found here: http://wiki.covecube.com/Development_Status

Future Developments for the StableBit Scanner The current development status for all of our products can now be found here: http://wiki.covecube.com/Development_Status

Future Developments for BitFlock.com The current development status for all of our products can now be found here: http://wiki.covecube.com/Development_Status

I've been thinking about how I can better communicate what is in store for each of our products, there are 3 now and another one in the works. Starting today I'll be setting up topics in each forum that I'll be updating on a regular basis. Each post will maintain what the future holds for each product. I try to keep a lot of the development driven by user feedback, but most of that feedback doesn't happen in the public forum (but usually in tech support tickets). I'd just like to take this opportunity to highlight the direction that each product is heading in, a kind of roadmap. I'll be setting up those posts today so look for them popping up soon in each respective forum.

Yeah, there's something wonky going on here. It should be checking all supported file systems I'm looking into it.