Jump to content
Covecube Inc.


  • Content Count

  • Joined

  • Last visited

  • Days Won


Posts posted by Alex

  1. How it All Works


    Since File Placement is now implemented in the latest BETAs of StableBit DrivePool 2.1.0.X, I'm going to give you a short overview as to how the whole system works and what the impacts on performance are.


    When you create a new rule, the pool that hosts that rule gets marked with a special flag indicating that a full background pattern based balancing pass is required. On the next balancing pass, StableBit DrivePool goes over every file on the drives that have any pattern rules defined and moves files matching those rules to their destination. Conflict resolution occurs at this time and any conflicts are saved for display to the user.


    At the same time, your file placement rules are sent down to the file system, telling it to respect those rules on any new file creation.


    Real-Time Rule Violations


    Sometimes a file placement rule is violated because a new file had to be created on disks other than the ones that you've designated for that file. In those cases the file system notifies the service that file pattern rules were broken. At this point a balancing pass is scheduled to fully assess the situation.




    This is an important point to make.


    How do the new file placement rules affect performance? First of all, when you have no file placement rules defined, then performance is not affected at all.


    Having one or more file placement rules enabled has these performance impacts:

    • For the StableBit DrivePool service:
      • After making any changes to file placement rules, a full file placement balancing pass is scheduled for the next automatic run. For automatic balancing, the regular balancing pass may not run if there is no need to do so.
      • Manually initiating a balancing pass using the "rebalance" command will always run the regular balancing pass and perform a full file placement balancing pass, if there are any conflicts or you've made changes to the file placement rules.
      • Overall, the balancing pass will take longer if you've made changes to the file placement rules or if there are file placement conflicts.
    • Real-time performance impacts on the file system:
      • Whenever a new file is created on a pool with one or more file placement limits, the decision where to place that file has to take those limits into account. For performance reasons that decision is made by the kernel file system driver with no involvement from user system service.
      • There is no impact on Read / Write I/O performance as a result of having file placement limits defined.
  2. And I am on your side, 1GB can be much it would be better if the Feeder disk could be completely flushed.


    Setting the slider to 100% should accomplish exactly that. It basically means "balance if any data needs to be moved".


    Also, the latest internal BETAs do a better job of moving every single file off of a disk when it's being emptied.


    In particular, here is info on that code change: https://stablebit.com/Admin/IssueAnalysis/2166

  3. Drashna, Alex, thank you for looking into this.


    Two problems...


    1. Volume ending in ...27f50 doesn't exist on my system (at all).  I've added a letter to each separate drive then used the cmd /k mountvol command to verify.

    2. I have Scanner installed and have it do a full scan every 10 days - each drive is marked healthy, no smart errors, anything.  Wouldn't CRC errors/failing platters, etc. show up?

    3. I've attempted to rename the pool again via Disk Management, properties on my pooled drive mount (D) - and still no go.


    1. That's odd, it was in the logs that I saw.
    2. Yes, absolutely. A CRC error should show up in the SMART data instantly and a surface scan will pick it up as well.
    3. We store the volume name in an alternate data stream on the pool's root directory. So it's likely that the issue has something to do with that.

    Can you capture a file trace log of the rename operation like this:

    • Under Settings -> Troubleshooting enable File system logging.
    • Perform the rename operation that results in the error.
    • Then uncheck File system logging.

    This will capture a file system log of the error that I can look at to get a better idea of where the error is coming from.


    The trace files will be saved to: C:\ProgramData\StableBit DrivePool\Service\Logs\CoveFs


    Just ZIP those up and upload them here: http://wiki.covecube.com/StableBit_DrivePool_2.x_Log_Collection


    Let me know once you have that uploaded and I'll take a look.



  4. Is there any chance any of these upgrades are going to filter to whs11 or is that version a dead product.



    You must mean StableBit DrivePool 1.X, no these features will not be added to 1.X however 2.X does run on WHS 2011.


    I am still actively making fixes to 1.X as you can see here: http://dl.covecube.com/DrivePool/beta/download/Changes.txt


    But adding major features to both versions is just not practical.

  5. Hopefully it may be in the next update  (and soon ...as we seem to have been at for a very very long time)   ;).

    I should stress that the above comment is purely wishful thinking rather than any sort of insider knowledge. 


    The main reason why we're stuck at 432 is because I was trying to fix all of the important bugs that were reported as of the last release. We have a brand new system for keeping track of reported issues, because frankly I got overwhelmed with the feedback from the last release. Since than I've been going through the bugs and have been trying to meticulously identify each issue and address it. Not that there are show-stopping bugs in, but I want the 2.1.X version to be an improvement in stability as well as add something significantly new.


    There have been a whole lot of fixes since 432 (http://dl.covecube.com/DrivePoolWindows/beta/download/changes.txt)


    The next public BETA is my priority now that per-folder (and pattern based) balancing is implemented. Hopefully it won't take more than a few weeks to get everything tested and published.


    I'm also actively working on "Product 3" which will integrate very nicely with StableBit DrivePool and will add significant value to it. So there are some amazing things in the works for the future.


    Thank you for your continued support.

  6. Well guys this is now fully implemented, and very much untested :)


    Here's the change log for this feature: https://stablebit.com/Admin/IssueAnalysis/2165


    You can download the latest internal (untested) BETA here: http://dl.covecube.com/DrivePoolWindows/beta/download


    Latest BETA is as of this writing.


    I'll have much more to say on how it all works in a future blog post, once I test it a bit more thoroughly and release a public BETA up on stablebit.com. But you can give it a whirl today if you're feeling adventurous.

  7. Piotr,

    I've examined your service dump and I believe that your crash is being caused by the Ordered File Placement plug-in v It seems to be stuck in a perpetual calculating loop. If you update that to the latest (v, I think that your issue will be resolved. But reboot your system prior to upgrading to get it "unstuck".

    Download it here: http://stablebit.com/DrivePool/Plugins

    In fact, I just looked at the change logs for that plug-in and I can see that Issue #76 was fixed in that version and it was specifically causing perpetual ratio calculation.

    In addition, I've made 2 changes to the StableBit DrivePool code:

    • Balance ratio calculation should not run using IDLE thread priority by default.
    • No balancers should ever be "trusted" to calculate the balance ratio quickly. There should be an enforced timeout for that calculation and if it expires a balancer should be aborted.

    If the updated balancer doesn't resolve your issue then you can download the latest internal BETA with those fixes from here:

    If you still continue to have the problem, you can request a remote support session here:




    IAR: https://stablebit.com/Admin/IssueAnalysis/2159

  8. Lurifax,


    Thanks for sending in that memory dump. I've analyzed it and unfortunately it indeed was a bug in build 420. It only affects x86 hosts, which is why you are seeing it.


    To resolve the issue you can download the latest public BETA right here:



    Or you can get the very latest internal BETA here:

    http://wiki.covecube.com/Downloads (these contain the very latest fixes as people report them)


    I'm going to try to get 2.1 out as a release final sooner rather than later, so the fix will be in that as well. However, we have a few other important issues to address before that happens.


    IAR: https://stablebit.com/Admin/IssueAnalysis/2157

  9. I recently upgraded from Windows Home Server 2011 to Server 2012 R2 Essentials and DrivePool provides a single executable that provides a standalone application, as well as Dashboard integration. Currently Scanner offers either/or, you can install the executable and get the standalone version, or install the wssx file and get Dashboard integration.


    Additionally, DrivePool and Scanner have different presentation UIs will this be synced?


    As a separate note, can you fix the title of the Scanner window to be just "SCANNER" or something? It looks weird to have a properly cased tab in-between the rest fo the all-caps tabs in 2012 R2 essentials.


  10. This is a very interesting thread and I think that if there is a performance issue with StableBit DrivePool we need to narrow it down to one particular scenario. For example, as bblue has suggested in the original post, Hyper-V running WHS2011 with a few pass-through drives.


    As far as some of the other issues that have been raised, let me comment on those:

    • The performance pane uses our file system driver to measure performance with some limited input from the Windows Performance Counters. Some of the earlier BETAs were combining our data with the Windows performance counters in a larger sense, but that type of presentation became a bit confusing and inconsistent. The StableBit DrivePool UI is meant to show you a quick overview of what's happening on the pool, so currently it uses its own high performance real-time logger that is only active when someone is looking at the performance data and some limited input from the disk performance counters to indicate disk activity.
    • The balancing plug-in settings operate in the system service and have really nothing to do with real-time read/write performance. Of course, except for their ability to tell the file system to avoid certain disks (according to fill limits). For example, if you tell the system to use a disk that is connected over USB 2.0 (albeit, an extreme example), then you are slowing the pool down. The point being, there is no additional "load" or resource consumption necessary to impose certain balancing rules in real-time.

      Background balancing does consume resources, but if background balancing is not running, balancing setting alone do not require some extra "processing". This is the nature of the balancing system and it was designed like this for performance.
    • As for background balancing, that is optimized to not interfere with existing I/O as much as possible.

      The reasons are as follows:
      • The pool remains full accessible when balancing is running.
      • You can safely shutdown and restart the system while balancing is taking place.
      • You will never receive "Access denied" or any type of other "In use" error message due to balancing.
    • The background balancer uses Background I/O disk scheduling, which allows the OS to gracefully throttle it if something else needs the use of the same disk. As a consequence, it will not operate at maximum throughput. This can be turned off if you wish using advanced settings (http://wiki.covecube.com/StableBit_DrivePool_2.X_Advanced_Settings).
    • Ok, I think we're veering off topic a bit here, but I just wanted to make it clear that background balancing and background duplication throughput speeds are never at maximum throughput, by design. You can turn this off.
    • As far as read / write performance, this has come up now and again. So far I've never been able to recreate a scenario, in a controlled test, that has shown the pool's performance to be slower.
    • From a theoretical point of view, here's how writing works:
      • An I/O request comes in from some application (or the network).
      • This I/O request is associated with some open file (opened previously). It has some data, an offset and a length.
      • We take this request and forward it to NTFS using a thin wrapper module, all done in the kernel. That's pretty much it.
    • So from a purely architectural point of view, there should be no discernible delay.

    Now I started this post by saying that if there is a potential performance issue here, that we should concentrate on recreating one specific scenario. I think that if something is going wrong it most likely is not in the read / write path. Those things are just very straightforward. Renames are fairly complicated so they might be worth considering as the culprit. There is really no such thing as a "copy" rename in the kernel, so StableBit DrivePool always renames to the same disk which should be instantaneous.


    I'm going to try a test to see if I can reproduce the scenario as it's outlined in the first post.


    IAR: https://stablebit.com/Admin/IssueAnalysisPublic?Id=1154

  11. A suggestion, if possible. Could the code in Disk Performance be changed so that instead of showing one filename (unknown as to whether it is the read or write file, when you hover over the up arrow, it shows the read filename, and hovering over the down arrow shows the write filename?


    Could be very useful if practical in the code.




    The Disk Performance area shows the top file operations that are operating on the busiest disks. The file at the top is operating on the disk with the most active I/O request (i.e. the disk which is the slowest to respond because of I/O load). Each file can either be read from or written to, so I guess I'm not understanding the request for showing "write files" vs. "read files". A file can either be written to or read from, it's still the same file.

  12. These almost look like auto-nightly builds. Is 472 known to be in reasonable condition?



    These are not auto nightly builds but are deliberately built after a Code Change Request (CCR) is completed (see: http://wiki.covecube.com/Development_Status#Development_Workflow). The build process is automatic and the publish is automatic.


    The difference between these internal BETAs and the BETAs that are published on StableBit.com is that the internal BETAs don't undergo any kind of functional testing before a build is published. The BETAs published on StableBit.com undergo basic functional testing and the installer's upgrade procedure is tested on every supported OS. This takes a bit of time to accomplish so it's only done for public BETAs.

  13. That's a good suggestion.


    What the Scanner should probably do IMO in these cases is simply rename the bad file part to something else and let DrivePool's duplication system kick in and reduplicated the file. I'm always worried about deleting files. This way we retain all 3 file parts and the user can delete the bad one.


    I've flagged this as issue #86 and it will be fixed.

  14. Hmm... could be an optimization issue with the sector map drawing code. I'll take a look at reproducing this right now. We've got an unlinked IAR open for this issue (#88). IARs are our new workflow for handling bug reports and applying fixes. Normally, IARs are linked to contact requests @ stablebit.com/contact so you can get updates on your issues that way.


    I should really write a blog post about our new internal workflow for handling bugs :)

    Oh, and as far as logging performance issues in the UI, yes we do. We have a sort of built-in performance profiler in all BETA builds of the StableBit Scanner. It can be shown by pressing 'P' while the focus is inside the window.


    If you could take a screenshot of that, along with your sector map it would might me understand what's going on.


    I've attached an example of what this looks like.



  15. You have a good point, there really should be some kind of notification at the end of the removal process.


    I've added this to the roadmap:



    We will add this to the UI in the future.


    For now, you can look in the PoolPart... folder on the damaged drive that you just removed. Any files that were not migrated will still remain in that folder.

  16. Bad sectors will get remapped automatically by the drive without your intervention. I've experienced this first hand just a few weeks ago.


    Here was my experience:

    • A Western Digital Green drive developed a single bad sector.
    • The SMART data confirmed the bad sector that the surface scan has detected.
    • I was notified promptly and began a file scan.
    • The file scan revealed that no files were affected by the damage.
    • At that point I left the system alone for a few hours and then decided to re-check the bad sector to see whether it was still bad.
    • The sector turned green and the SMART data no longer reflected the bad sector.

    Whenever something tries to write to a bad sector, the drive will automatically swap in a new known good sector from the spare sector pool and take the bad sector out of service. This is done silently and automatically. This might have been what happened in your case.


    Normally, the Scanner will re-scan bad sectors automatically and flag them as good after some number of day (depending on your settings).


    Was it a false positive?


    One common cause of false positives is when you abruptly disconnect a drive while it is being scanned. This will cause the Scanner to flag the currently scanning sectors as bad (since they're unreadable), right before getting notified that the drive was unplugged. The Scanner will at this point flag those "bad" sectors as "unchecked".


    As far as your error reports:

    • There is an issue trying to add a firewall rule for remote control. This looks like a COM issue in Windows. If you don't use Remote Control you can ignore this, or even turn that feature off in Scanner Settings.
    • A drive seems to have been abruptly disconnected while it was being queried for its power status, before we query it for SMART data.
    • At least on one occasion, the system drive did not report the controller that it was on. This is the strangest of the bunch.

    If these errors continue and impede functionality then open up a contact request @ http://stablebit.com/Contact , mention this post to get the case forwarded to me and we can do remote support.

  17. Performance considerations with StableBit DrivePool:

    • For non-duplicated files with the default performance setting enabled, there should be very little overhead for read / write I/O on the pool. This is because all pooled I/O is processed in the kernel and our file system driver uses Neither I/O, Fast I/O, oplocks and a full suite of file caching. These are optional under the hood performance optimizations that are available to all kernel file system drivers and they are all implemented and utilized by our driver.

      While there is always some overhead introduced by any kind of software storage pooling / redundancy package, I don't expect our pooling solution to have any significant impact on performance, when comparing pooled vs. non-pooled I/O on the same physical disk.
    • The network I/O boost feature may reduce peak throughput and introduce higher CPU usage. The network I/O boost feature was designed to increase responsiveness of media streaming clients on a busy server and doesn't necessarily increase maximum throughput. In other words, we prioritize network I/O read requests over local read requests. This requires us to perform some additional processing on each network read request and so it does consume some extra CPU resources.

      While this does introduce overhead, for all practical purposes, it really is minimal and should not impact file transfer speeds in any appreciable way.

      Here is another test on the pool with network I/O boost enabled (using the same setup as above):
      Version 1.3.1
      OS Version: Windows 8
      Processor: AMD FX(tm)-8350 Eight-Core Processor          
      Date: 02/03/2014
      Time: 12:19:20
      Program Parameters: 0
      High Performance Timer: 0.0000000698
      Test File: \\\Pool-SpeedTest\NW_SpeedTest.dat
      Write Time = 7.5035405 Seconds
      Write Speed = 533.0816880 Mbps
      Read Time = 7.5083370 Seconds
      Read Speed = 532.7411440 Mbps
      Test File: \\\Pool-SpeedTest\NW_SpeedTest.dat
      Write Time = 7.1603575 Seconds
      Write Speed = 558.6313200 Mbps
      Read Time = 7.3564616 Seconds
      Read Speed = 543.7396640 Mbps
      Test File: \\\Pool-SpeedTest\NW_SpeedTest.dat
      Write Time = 7.6143740 Seconds
      Write Speed = 525.3222400 Mbps
      Read Time = 7.5884258 Seconds
      Read Speed = 527.1185520 Mbps
    • Real-time file duplication will have an impact on write performance because we have to send all write requests to more than one disk. We do this in parallel by sending the request to multiple disks at the same time.

    Performance considerations with the StableBit Scanner:

    • Active disk scanning will have an impact on other read / write I/O performance to the disk being scanned. How much of an impact depends on the StableBit Scanner settings.
    • Background I/O priority scanning ensures that Windows tags the read / write requests issued by the StableBit Scanner with the Very Low I/O priority. I/O priority is utilized in the kernel by the storage drivers in order to prioritize read / write requests to and from the disk.
    • Scan throttling will actively pause scanning when other disk activity is taking place on the disk being scanned. This works best when Background I/O priority scanning is enabled as well.

      Scan throttling has multiple sensitivities, Low, Medium and High. Low means that you want the StableBit Scanner to allow some other I/O to the disk without pausing scanning. High means the opposite.
      • Scan throttling can also monitor for bus saturation conditions, whereby scanning of one disk is slowing down I/O requests to another disk that is on the same bus. If this is the case, scanning will be paused until I/O on the other disks subsides.

    Quick Settings in the Stablebit Scanner and performance impact:

    • Desktop mode - Throttling (Med) and Background I/O are enabled. Scans happen at any time.
    • Laptop mode - Throttling (Med) and Background I/O are enabled. Scans happen at any time, but only when plugged into an A/C outlet.
    • Server mode - Throttling is disabled, background I/O is enabled. Scans happen only at night.
    • Tablet mode - Throttling (High, no bus saturation detection) and background I/O are enabled. Scans happen at any time, they occur more often, but only when plugged into an A/C outlet.

    You can obviously tweak these to your linking, but the defaults should work pretty well. All in all the system is designed to be fully automatic and very hands off.

  18. Hey guys,


    I've set up and run a controlled test using the same tool of pool vs. non-pool speeds and StableBit Scanner vs. no StableBit Scanner.


    Here is how I set up the test server:

    • Server to client connection is on a 1 gigabit link connected through 2 switches.
    • Created 2 shares, one on the pool and one not on the pool. No file duplication enabled.
    • Made sure that the non-pooled share is on the disk part of the pool with the most free space. This ensures that the same physical disk will be used to allocate the test file in both the pooled and non-pooled test cases.
    • Initially, the server has the StableBit Scanner installed and started.
    • Made sure that the StableBit Scanner is not actively scanning any disks at this time.
    • StableBit DrivePool is set up with the default performance settings (no Network I/O boost).
    • Made sure that nothing else is using the physical disk while the test is ongoing.
    • All tests were performed with a 500 MB file.

    It is important to ensure proper test condition in order to prevent skewed results.


    Now I ran 3 tests on the non-pooled share from the client:

    Version 1.3.1
    OS Version: Windows 8
    Processor: AMD FX(tm)-8350 Eight-Core Processor          
    Date: 02/03/2014
    Time: 11:36:09
    Program Parameters: 0
    High Performance Timer: 0.0000000698
    Test File: \\\NonPool-SpeedTest\NW_SpeedTest.dat
    Write Time = 6.7337975 Seconds
    Write Speed = 594.0184560 Mbps
    Read Time = 7.6298019 Seconds
    Read Speed = 524.2600080 Mbps
    Test File: \\\NonPool-SpeedTest\NW_SpeedTest.dat
    Write Time = 7.1907796 Seconds
    Write Speed = 556.2679200 Mbps
    Read Time = 7.7817525 Seconds
    Read Speed = 514.0230320 Mbps
    Test File: \\\NonPool-SpeedTest\NW_SpeedTest.dat
    Write Time = 7.4736517 Seconds
    Write Speed = 535.2135920 Mbps
    Read Time = 7.4989738 Seconds
    Read Speed = 533.4063200 Mbps

    Then I ran 3 tests on the pooled share:

    Version 1.3.1
    OS Version: Windows 8
    Processor: AMD FX(tm)-8350 Eight-Core Processor          
    Date: 02/03/2014
    Time: 11:38:09
    Program Parameters: 0
    High Performance Timer: 0.0000000698
    Test File: \\\Pool-SpeedTest\NW_SpeedTest.dat
    Write Time = 7.2969775 Seconds
    Write Speed = 548.1721680 Mbps
    Read Time = 7.6963353 Seconds
    Read Speed = 519.7278800 Mbps
    Test File: \\\Pool-SpeedTest\NW_SpeedTest.dat
    Write Time = 7.1262697 Seconds
    Write Speed = 561.3034800 Mbps
    Read Time = 7.4837478 Seconds
    Read Speed = 534.4915520 Mbps
    Test File: \\\Pool-SpeedTest\NW_SpeedTest.dat
    Write Time = 7.2078453 Seconds
    Write Speed = 554.9508720 Mbps
    Read Time = 7.7060378 Seconds
    Read Speed = 519.0734960 Mbps

    Then I stopped the StableBit Scanner service and ran another 3 tests on the pooled share:

    Version 1.3.1
    OS Version: Windows 8
    Processor: AMD FX(tm)-8350 Eight-Core Processor          
    Date: 02/03/2014
    Time: 11:41:06
    Program Parameters: 0
    High Performance Timer: 0.0000000698
    Test File: \\\Pool-SpeedTest\NW_SpeedTest.dat
    Write Time = 8.1331414 Seconds
    Write Speed = 491.8148800 Mbps
    Read Time = 7.6461024 Seconds
    Read Speed = 523.1423520 Mbps
    Test File: \\\Pool-SpeedTest\NW_SpeedTest.dat
    Write Time = 7.5164256 Seconds
    Write Speed = 532.1678400 Mbps
    Read Time = 7.5272389 Seconds
    Read Speed = 531.4033600 Mbps
    Test File: \\\Pool-SpeedTest\NW_SpeedTest.dat
    Write Time = 7.2955172 Seconds
    Write Speed = 548.2818960 Mbps
    Read Time = 7.5415770 Seconds
    Read Speed = 530.3930480 Mbps

    As expected, the speeds are roughly equivalent in all 3 cases. I'll post some more information about Stablebit Scanner / DrivePool performance shortly.

  19. As of 1/28/2014, in an effort to streamline the development process and to centralize the management of planned feature requests I've moved the current development status to our development wiki.


    You can access it here:



    In addition to implementing new features we now have a separate system for identifying and tracking existing issues. See the wiki page for a bit more on that.

  20. So my question is, how does DrivePool handle this?


    • For unduplicated files, StableBit DrivePool forwards all file I/O to the individual disk that the file is on.
    • If the file exists on more than one disks (i.e. is duplicated), then:
      • All file modification requests (such as writing to the file, setting its attributes, etc...) go to all of the disks that the file is on.
      • Read requests will either go to the first disk that the file is on (when read striping is disabled), or to one of the disks that the file is on, as determined by the read striping algorithm.
    • A directory listing operation works by querying all of the disks in parallel. This will force NTFS to read the MFT directory indexes on all of the disks where the directory being listed exists. These indexes can be cached and can theoretically be served entirely from RAM.
    • Opening a file is similar to a directory listing. This will tell NTFS to query its directory indexes on all of the disks part of that pool in order to locate the disks that contain that file. This also is done in parallel and can be served by the system cache.

    So in short, StableBit DrivePool may spin up disks in many circumstances, I haven't really done testing to see how often this is.

  21. Yes, absolutely temporary.


    The stablebit.com server is running on a dedicated Windows Azure box with an automatic DNS failover to another shared hosting account. Today, both failed for about a total of 4 hours. I'm not sure why the Windows Azure services failed, but restarting the servers fixed the problem. The backup host is experiencing a DDOS attack at this time and is still not working.


    stablebit.com is back up but bitflock.com is still down. I'm monitoring the situation closely.


    Sorry about the inconvenience.

  • Create New...