Alex

Oh, I definitely understand that. But from a management/customer service standpoint, it's less than ideal for us. Which is why I replied to it with what I did. I wanted to make sure that anyone seeing it knew that the issue had been addressed.
 
Also, if/when the issue gets sorted out, I will try to make sure the solution is posted here as well (if you don't beat me to it). Worst case, I try to remember every issue I encounter (be it mine or a customers) and use that to help troubleshoot issues in the future.

Thank you to both yourself and Drashna for the explanation. I guess an end user with a very stable system doesnt see all the bugs that need fixing in the background.
 
 Will test out the new release as soon as i get home from work.
 
Oh and "product 3" !!!   Awesome news!

Another vote, this would be an amazing feature.  Wish I'd seen this thread sooner, I've been waiting and waiting for a new beta - checking every few days for months now in hopes of something and finally decided to check the forum and see if development was still going on.
 
Rule based file placement would be a killer feature, taking the current "ordered file placement" abilities of the current plugin to the next step in granularity. I just hate related files to get scattered all over the place.  Ordered file placement has cut down on that problem significantly since it fills up a disk at a time, but its not perfect obviously and I end up doing way more manual file relocating than I'd like to.

Thank you, that inspires us to do our best.

This has renewed my faith in product support!
 
I disabled the ordered plugin, restarted the server and installed the new version of the plugin. I was able to add the drive and it's working perfectly!
 
This is fantastic! Thank you very much for your help! 

Piotr,

I've examined your service dump and I believe that your crash is being caused by the Ordered File Placement plug-in v 1.0.0.1. It seems to be stuck in a perpetual calculating loop. If you update that to the latest (v 1.0.3.4), I think that your issue will be resolved. But reboot your system prior to upgrading to get it "unstuck".

Download it here: http://stablebit.com/DrivePool/Plugins

In fact, I just looked at the change logs for that plug-in and I can see that Issue #76 was fixed in that version and it was specifically causing perpetual ratio calculation.

In addition, I've made 2 changes to the StableBit DrivePool code:
Balance ratio calculation should not run using IDLE thread priority by default. No balancers should ever be "trusted" to calculate the balance ratio quickly. There should be an enforced timeout for that calculation and if it expires a balancer should be aborted. If the updated balancer doesn't resolve your issue then you can download the latest internal BETA with those fixes from here:
http://wiki.covecube.com/Downloads

If you still continue to have the problem, you can request a remote support session here:
http://stablebit.com/Contact
 
Regards,
IAR: https://stablebit.com/Admin/IssueAnalysis/2159

I've been thinking about how I can better communicate what is in store for each of our products, there are 3 now and another one in the works. Starting today I'll be setting up topics in each forum that I'll be updating on a regular basis. Each post will maintain what the future holds for each product.
 
I try to keep a lot of the development driven by user feedback, but most of that feedback doesn't happen in the public forum (but usually in tech support tickets). I'd just like to take this opportunity to highlight the direction that each product is heading in, a kind of roadmap.
 
I'll be setting up those posts today so look for them popping up soon in each respective forum.

Thank you that is very much appreciated

This is actually a fairly complicated topic.
 
Let's start by talking about how normal standby works without the StableBit Scanner getting involved.
 
Windows "Put the Disk to Sleep" Feature
 
Normally, Windows will monitor the disk for activity and if there is no disk activity for some preset amount of time it will put the disk to "sleep" by flushing all of the data in the cache to the disk and sending a special standby command to it. At the same time, it will remember that the disk is asleep in case any other application asks.
 
Shortly after the disk goes to sleep, the StableBit Scanner will indicate the fact that the disk is asleep in the Power column. Normally, the Scanner gets the power status of the disk by querying Windows and not the disk.
 
It does not query the disk directly for the power state because Windows considers this power query disk activity and wakes up the disk as a result.
 
Now, things get a bit more complicated if you want to include the on-disk power management in this picture.
 
Disks can optionally support these features, which can put them to sleep without Windows knowing it:
Advanced power management. Standby timer Advanced Power Management
 
This is a technology that implements power consumption profiles. For instance, if you don't care about performance but want maximum power savings, then you can tell your disk just that. Simply set the Advanced Power Management to Minimum Power Consumption. Or you can do the exact opposite by setting it to Maximum Performance (which guarantees no standby).
 
With Advanced Power Management you don't concern yourself with "sleep timeouts", like in Windows. You simply state your intent and the disk will adjust various parameters, including the standby time, according to your setting.
 
The implementation of Advanced Power Management is completely up to the manufacturer of the drive, and there are no specifications that explicitly state what each power mode does. This entire feature may not even be supported, depending on the disk model.
 
Standby Timer
 
The Standby timer is more widely supported because it is an older feature. You simply specify after how much disk inactivity you would like the disk to be put to sleep. This is similar to how things work in Windows, except that the low power mode will be initiated by the disk firmware itself.
 
Again, the implementation of this is up to the manufacturer of the drive.
 
StableBit Scanner "Put into Standby"
 
In the StableBit Scanner, you can right click on a disk and put it into standby mode. What this does is send a power down command to the disk. This type of power down is equivalent to what Advanced Power Management or the Standby Timer would do.
 
More importantly, when a disk is powered down in this way, Windows will not be aware that the disk is in a low power state, and will continue to report that the disk is still powered up. This is not an issue because the disk will simply spin up the next time that Windows tries to access it.
 
But this leaves the StableBit Scanner with a dilemma. If we can't query the disk for the power state directly, how do we report the true power state of the disk? What the StableBit Scanner implements is a power state in which it's not sure whether the disk is in standby or active, and this is what you were seeing.
 
Forcing the StableBit Scanner to Query the Power Mode from the Disk
 
If you want to use on-disk power management exclusively, and you don't care about Windows putting your disks to sleep, you can instruct the StableBit Scanner to query the power mode directly from the disk.
 

 
When this is enabled, you will no longer see the standby or active message, but Windows will never try to put that disk to sleep. That's why this is off by default.
 
SMART
 
And just to make things even more complicated, sometimes a disk will wake up when it's queried for SMART data.
 
To this end the StableBit Scanner implements some more settings to deal with this:
 

 
I hope that this clears things up.

I like writing these posts because they give me feedback as to what the community is really interested in. I can see that my last post about the Scanner was not very interesting, it was probably too technical and there's probably not much to add to what I've already said.
 
Well, this time let's talk about StableBit DrivePool. In particular, I'd like to talk about DrivePool beyond 2.0.
 
Controlling Folder Placement
 
I think that I have a few great ideas for DrivePool 2.1+ but some of them depend on the ability to control folder (or file) placement, per pool part. I've kind of hinted at this capability in the thread that talked about taking out per-folder duplication, but I think that I've figured out how we can make this work.
 
What I would like to be able to do in future versions is to give you guys the ability to associate folders with one or more disks that are part of the pool. So that any files in those folders would be stored on those pool parts only (unless they're full).
 
This should be trivial to implement on the file system level, but the balancing framework would need to be enhanced to support this, and I think that I've figured out how to make that work.
 
Theoretically, you should even be able to use wildcard patterns such as /Virtual Machines/Windows* to associate all of those files with a group of pooled disks.
 
What do you guys think, is this worthwhile doing?

As of 1/28/2014, in an effort to streamline the development process and to centralize the management of planned feature requests I've moved the current development status to our development wiki.
 
You can access it here:
http://wiki.covecube.com/Development_Status
 
In addition to implementing new features we now have a separate system for identifying and tracking existing issues. See the wiki page for a bit more on that.

Well, this is going to be as off topic as it gets and fairly personal
 
One of my personal love affairs is sci fi short stories, and I love originality. I'm not a big fan of the Sci Fi that's on TV today, but I loved Rod Serling's work on the Twilight Zone. Today, some of the best stuff that I've read online resides here: http://qntm.org/fiction
 
This site is fully programmed by one Perl programmer (named Sam) and features some amazing and mind boggling stuff. I love what he's doing and would like to support him.
 
The stories are absolutely free and the ones that I loved in particular are:
http://qntm.org/difference http://qntm.org/responsibility But all of his stuff is just amazing, especially the ongoing Ra series.

Ok, so reparse points have definitely been driving me nuts lately. I was planning on releasing the StableBit DrivePool 2.1 BETA about a week after the 2.0 final with reparse point support, but it's still not out and it's because of reparse points. I've been trying different architectures over the past few weeks and none of them panned out as expected.
 
But today, I believe that I've finally got something that will work. It will support all of the various kinds of reparse points, it will be super fast and stable.
 
So what are these "reparse points" you may be asking?
 
Well, reparse points are the underlying file system technology that enable a whole suite of functionality. Mainly they are used for creating symbolic links, junctions and mount points.
 
Essentially it's a way to redirect access from one file or folder to another. You may be wondering if I'm talking about a Shortcut? No, confusingly a shortcut is not a reparse point.
 
So how many ways does Windows have to redirect files / folders?
 
A lot. That's the problem!
 
Here they are off the top of my head:
A shortcut - A special file that is parsed by the Explorer shell that really links to another file somewhere else (as in a Start menu shortcut).

Most people are probably familiar with this because it's readily available in the Explorer UI.
  Symbolic file link - A file that points to some other file somewhere else. Confusingly, Windows Explorer also calls these "shortcuts" in the file properties dialog.

A symbolic link can be created by the mklink utility with no options.
  Symbolic directory link - These are relatively new, as they were introduced in Windows Vista. This is essentially a directory that points to another directory somewhere else.

These can be created by using mklink /D.
  Directory junction point - These are very similar to "symbolic directory links", but they were available prior to Windows Vista. Again, it is essentially a directory that points to another directory somewhere else. Some people make the mistake that a junction is only capable of pointing to another directory on the same volume, and that's not the case.

These can be created by using mklink /J.
  Mount point - Mount points allow you to designate a directory that will point to another volume. These are typically used to "mount" a number of drives as directories under some other drive letter, thus saving drive letters.

These can be created from Disk Management.
  File hard link - Yet another way to make a file point to another file. However, this method can only be used to point a file to some other file on the same volume.

These are created using mklink /H.  
Yes, that's a lot of ways that you can redirect files / folders in Windows. Try Googling these and you can see the confusion that ensues as to what the differences are between each.
 
So what is the difference between all of these?
 
Well, instead of pointing out the pros and cons, I'll tell you how each one of them works under the hood and you can decide for yourself:
A shortcut - This is the most "user friendly" way of creating a file that points to another one. Even the name makes sense, "shortcut", imagine that. It's readily available from the Windows Explorer menus and works entirely in user mode. A special .lnk file is created that the user mode shell knows how to parse. In Windows Explorer, an icon with a little arrow is shown to you to let you know that this is really a shortcut.

However, as far as the kernel and file system are concerned, there is nothing special about the .lnk file, it's just a regular file.
  Symbolic file link - Sometimes called a "symlink" or "soft link", this is a system that redirects one file to another, purely in the kernel. It involves some special metadata that is stored with the "source link" file that points to the "target destination file" and requires coordination between the file system and the Windows I/O Manager.

This system uses what are called "reparse points".
  Symbolic directory link - This is exactly the same thing as a symbolic file link, but it works on directories. The reason why I separated the two is because symbolic directory links were not available prior to Windows Vista and they must be created differently.

However, the underlying technology that enables this is exactly the same. This too uses "reparse points".
  Directory junction point - This is similar to a Symbolic directory link except that it is available prior to Windows Vista and uses an older technique. Technically speaking, the main difference between this and symbolic directory links is that directory junction points always point to an absolute path, while symbolic directory links can point to relative or absolute paths.

Surprisingly, this too uses "reparse points", but not all reparse points are the same. I'll get to that soon.
  Mount point - These are implemented in the exact same way as directory junction points, except that they point to the root of some other volume instead of some other directory.

These are implemented with the exact same "reparse points" as directory junctions.
  File hard link - This is purely a file system construct. Because of the way directory indexes work in NTFS, it is possible to add a file entry to a directory index of a file that already exists under some other directory. Essentially, you can think of the file as being in 2 (or more) places at once. While this is not quantum physics, it is NTFS. Each file has a "reference count" and that count is incremented whenever a hard link is created to it. When the count reaches 0, the file is deleted.

No other kernel subsystem is involved and no "reparse points" exists. This is the cleanest and purest way of making a file appear in 2 places at once (IMO). Wow, and all this works together reliably?
 
Yes, and that's what StableBit DrivePool is trying to preserve. You see, right now the only thing that we support on the pool from the above list are shortcuts. Everything else is not supported.
 
Some people have been requesting the support of file / directory symbolic links and junctions. Those 2 can be used by software in order to create complex directory structures, in order to organize your data better.
 
4 out of the 5 unsupported technologies use "reparse points", so it makes sense for StableBit DrivePool to implement support for them.
 
Ok, so what's a "reparse point"?
 
A reparse point is a Microsoft defined data structure that gets associated with a file or a directory. When that file or directory has a reparse point associated with it, then it becomes a kind of link to "somewhere else".
 
Essentially, when a file system encounters a reparse point, it tells the I/O Manager "these aren't the droids you're looking for, go look here". The I/O Manager is responsible for opening files, so it happily obliges.
 
That doesn't sound too complicated
 
Well, it isn't, except that there are different types of "reparse points" and each reparse point has a different meaning of where to go next.
 
For example:
File / directory symbolic links use a "symlink" reparse point. Directory junction points / mount points use a "mount point" reparse point. Any 3rd party developers can develop their own type of reparse points and their own logic as to how they work. Remember drive extender from WHS v1? Yep, those tombstones were yet another kind of reparse points. Ok, so this is complicated. But will StableBit DrivePool support reparse points?
 
I'm working hard towards that goal, and the reason why I'm writing this is because I believe that I've finally cracked the architecture that we need to support all Microsoft and 3rd party reparse points on the pool.
 
The architecture has these positive aspects to it:
It supports file / directory symbolic links, directory junction points, mount points, and 3rd party reparse points on the pool.
  It is a 100% native kernel implementation, with no dependence on the user mode service.
  It follows the 0 local metadata approach of storing everything needed on the pool itself and does not rely on something like the registry. This means that your reparse points will work when moving pools between machines (provided that you didn't link to something off of the pool that no longer exists on the new machine). Some of my previous attempts had these limitations:
Requires the user mode service to run additional periodic maintenance tasks on the pool.
  No support for directory reparse points, only file ones.
  Adding a drive to the pool would require a somewhat lengthy reparse point pass. The new architecture that I came up with has none of these limitations. All it requires is NTFS and Windows.
 
When will it be ready?
 
I'd hate to predict, but I think that it should be deployed in BETA form in a few weeks.

I'm thinking about doing a Nuts & Bolts Google+ hangout in the future. Basically, it would be an open discussion about anything Covecube / StableBit, current or future products, and technology trends (no tech support though).
 
But I'm not sure how many people would participate in something like that.
 
Would anyone like to participate?

Today we've had a BitFlock disruption. BitFlock.com was down for about 6 hours becasue of a domain name issues. All access is now restored and you should not experience any further issues accessing bitflock.com. The problem was at Google.

SIIG SC-SA0M11-S1 PCI-Express 2.0 Low Profile Ready SATA III (6.0Gb/s) 2-Port Controller Card
 
(my personal favorite inexpensive PCI-Express eSATA card)
 

 
Specifications:
Speed: SATA 3 (6.0Gbit/s) Ports: 2 x eSATA Slot: PCI-Express x1 2.0 (~4 Gbit/s x 2) Chipset: ASM1061
OverallSupports Identify: Yes Supports SMART: Yes AHCI Driver
Firmware: 0.95 AHCI ModeAHCI compatible: Yes (works out of the box with the default Windows AHCI driver) * Driver: 6.2.9200.16548 (6/21/2006)storahci.sys * Port multiplier support requires Asmedia driver. 
StableBit Scanner - Direct I/O
Methods: AtaPassThrough 

Identify: Yes SDD: Yes 

SMART Status: Yes SMART Attributes: Yes SMART Thresholds: Yes SMART Error Log: Yes 

Power Mode: Yes StableBit Scanner - WMISMART Status: Yes SMART Attributes: Yes SMART Thresholds: Yes SMART Error Log: No Tested on Windows 8 64-bit.
Asmedia Driver
Firmware: 0.95 AHCI ModeAHCI compatible: Yes (works out of the box with the default Windows AHCI driver) * Driver: 1.3.4.0 (1/5/2012)asahxp64.sys Link: http://www.siig.com/download/search/?keyword=SC-SA0M11-S1 * Port multiplier support requires Asmedia driver. 
StableBit Scanner - Direct I/O
Methods: AtaPassThrough 

Identify: Yes SDD: Yes 

SMART Status: Yes SMART Attributes: Yes SMART Thresholds: Yes SMART Error Log: Yes 

Power Mode: Yes StableBit Scanner - WMISMART Status: Yes SMART Attributes: Yes SMART Thresholds: Yes SMART Error Log: No Tested on Windows 8 64-bit.

I've been thinking about how I can better communicate what is in store for each of our products, there are 3 now and another one in the works. Starting today I'll be setting up topics in each forum that I'll be updating on a regular basis. Each post will maintain what the future holds for each product.
 
I try to keep a lot of the development driven by user feedback, but most of that feedback doesn't happen in the public forum (but usually in tech support tickets). I'd just like to take this opportunity to highlight the direction that each product is heading in, a kind of roadmap.
 
I'll be setting up those posts today so look for them popping up soon in each respective forum.

Well, this is going to be as off topic as it gets and fairly personal
 
One of my personal love affairs is sci fi short stories, and I love originality. I'm not a big fan of the Sci Fi that's on TV today, but I loved Rod Serling's work on the Twilight Zone. Today, some of the best stuff that I've read online resides here: http://qntm.org/fiction
 
This site is fully programmed by one Perl programmer (named Sam) and features some amazing and mind boggling stuff. I love what he's doing and would like to support him.
 
The stories are absolutely free and the ones that I loved in particular are:
http://qntm.org/difference http://qntm.org/responsibility But all of his stuff is just amazing, especially the ongoing Ra series.

I've been thinking about how I can better communicate what is in store for each of our products, there are 3 now and another one in the works. Starting today I'll be setting up topics in each forum that I'll be updating on a regular basis. Each post will maintain what the future holds for each product.
 
I try to keep a lot of the development driven by user feedback, but most of that feedback doesn't happen in the public forum (but usually in tech support tickets). I'd just like to take this opportunity to highlight the direction that each product is heading in, a kind of roadmap.
 
I'll be setting up those posts today so look for them popping up soon in each respective forum.

Ok, so reparse points have definitely been driving me nuts lately. I was planning on releasing the StableBit DrivePool 2.1 BETA about a week after the 2.0 final with reparse point support, but it's still not out and it's because of reparse points. I've been trying different architectures over the past few weeks and none of them panned out as expected.
 
But today, I believe that I've finally got something that will work. It will support all of the various kinds of reparse points, it will be super fast and stable.
 
So what are these "reparse points" you may be asking?
 
Well, reparse points are the underlying file system technology that enable a whole suite of functionality. Mainly they are used for creating symbolic links, junctions and mount points.
 
Essentially it's a way to redirect access from one file or folder to another. You may be wondering if I'm talking about a Shortcut? No, confusingly a shortcut is not a reparse point.
 
So how many ways does Windows have to redirect files / folders?
 
A lot. That's the problem!
 
Here they are off the top of my head:
A shortcut - A special file that is parsed by the Explorer shell that really links to another file somewhere else (as in a Start menu shortcut).

Most people are probably familiar with this because it's readily available in the Explorer UI.
  Symbolic file link - A file that points to some other file somewhere else. Confusingly, Windows Explorer also calls these "shortcuts" in the file properties dialog.

A symbolic link can be created by the mklink utility with no options.
  Symbolic directory link - These are relatively new, as they were introduced in Windows Vista. This is essentially a directory that points to another directory somewhere else.

These can be created by using mklink /D.
  Directory junction point - These are very similar to "symbolic directory links", but they were available prior to Windows Vista. Again, it is essentially a directory that points to another directory somewhere else. Some people make the mistake that a junction is only capable of pointing to another directory on the same volume, and that's not the case.

These can be created by using mklink /J.
  Mount point - Mount points allow you to designate a directory that will point to another volume. These are typically used to "mount" a number of drives as directories under some other drive letter, thus saving drive letters.

These can be created from Disk Management.
  File hard link - Yet another way to make a file point to another file. However, this method can only be used to point a file to some other file on the same volume.

These are created using mklink /H.  
Yes, that's a lot of ways that you can redirect files / folders in Windows. Try Googling these and you can see the confusion that ensues as to what the differences are between each.
 
So what is the difference between all of these?
 
Well, instead of pointing out the pros and cons, I'll tell you how each one of them works under the hood and you can decide for yourself:
A shortcut - This is the most "user friendly" way of creating a file that points to another one. Even the name makes sense, "shortcut", imagine that. It's readily available from the Windows Explorer menus and works entirely in user mode. A special .lnk file is created that the user mode shell knows how to parse. In Windows Explorer, an icon with a little arrow is shown to you to let you know that this is really a shortcut.

However, as far as the kernel and file system are concerned, there is nothing special about the .lnk file, it's just a regular file.
  Symbolic file link - Sometimes called a "symlink" or "soft link", this is a system that redirects one file to another, purely in the kernel. It involves some special metadata that is stored with the "source link" file that points to the "target destination file" and requires coordination between the file system and the Windows I/O Manager.

This system uses what are called "reparse points".
  Symbolic directory link - This is exactly the same thing as a symbolic file link, but it works on directories. The reason why I separated the two is because symbolic directory links were not available prior to Windows Vista and they must be created differently.

However, the underlying technology that enables this is exactly the same. This too uses "reparse points".
  Directory junction point - This is similar to a Symbolic directory link except that it is available prior to Windows Vista and uses an older technique. Technically speaking, the main difference between this and symbolic directory links is that directory junction points always point to an absolute path, while symbolic directory links can point to relative or absolute paths.

Surprisingly, this too uses "reparse points", but not all reparse points are the same. I'll get to that soon.
  Mount point - These are implemented in the exact same way as directory junction points, except that they point to the root of some other volume instead of some other directory.

These are implemented with the exact same "reparse points" as directory junctions.
  File hard link - This is purely a file system construct. Because of the way directory indexes work in NTFS, it is possible to add a file entry to a directory index of a file that already exists under some other directory. Essentially, you can think of the file as being in 2 (or more) places at once. While this is not quantum physics, it is NTFS. Each file has a "reference count" and that count is incremented whenever a hard link is created to it. When the count reaches 0, the file is deleted.

No other kernel subsystem is involved and no "reparse points" exists. This is the cleanest and purest way of making a file appear in 2 places at once (IMO). Wow, and all this works together reliably?
 
Yes, and that's what StableBit DrivePool is trying to preserve. You see, right now the only thing that we support on the pool from the above list are shortcuts. Everything else is not supported.
 
Some people have been requesting the support of file / directory symbolic links and junctions. Those 2 can be used by software in order to create complex directory structures, in order to organize your data better.
 
4 out of the 5 unsupported technologies use "reparse points", so it makes sense for StableBit DrivePool to implement support for them.
 
Ok, so what's a "reparse point"?
 
A reparse point is a Microsoft defined data structure that gets associated with a file or a directory. When that file or directory has a reparse point associated with it, then it becomes a kind of link to "somewhere else".
 
Essentially, when a file system encounters a reparse point, it tells the I/O Manager "these aren't the droids you're looking for, go look here". The I/O Manager is responsible for opening files, so it happily obliges.
 
That doesn't sound too complicated
 
Well, it isn't, except that there are different types of "reparse points" and each reparse point has a different meaning of where to go next.
 
For example:
File / directory symbolic links use a "symlink" reparse point. Directory junction points / mount points use a "mount point" reparse point. Any 3rd party developers can develop their own type of reparse points and their own logic as to how they work. Remember drive extender from WHS v1? Yep, those tombstones were yet another kind of reparse points. Ok, so this is complicated. But will StableBit DrivePool support reparse points?
 
I'm working hard towards that goal, and the reason why I'm writing this is because I believe that I've finally cracked the architecture that we need to support all Microsoft and 3rd party reparse points on the pool.
 
The architecture has these positive aspects to it:
It supports file / directory symbolic links, directory junction points, mount points, and 3rd party reparse points on the pool.
  It is a 100% native kernel implementation, with no dependence on the user mode service.
  It follows the 0 local metadata approach of storing everything needed on the pool itself and does not rely on something like the registry. This means that your reparse points will work when moving pools between machines (provided that you didn't link to something off of the pool that no longer exists on the new machine). Some of my previous attempts had these limitations:
Requires the user mode service to run additional periodic maintenance tasks on the pool.
  No support for directory reparse points, only file ones.
  Adding a drive to the pool would require a somewhat lengthy reparse point pass. The new architecture that I came up with has none of these limitations. All it requires is NTFS and Windows.
 
When will it be ready?
 
I'd hate to predict, but I think that it should be deployed in BETA form in a few weeks.

Ok, so reparse points have definitely been driving me nuts lately. I was planning on releasing the StableBit DrivePool 2.1 BETA about a week after the 2.0 final with reparse point support, but it's still not out and it's because of reparse points. I've been trying different architectures over the past few weeks and none of them panned out as expected.
 
But today, I believe that I've finally got something that will work. It will support all of the various kinds of reparse points, it will be super fast and stable.
 
So what are these "reparse points" you may be asking?
 
Well, reparse points are the underlying file system technology that enable a whole suite of functionality. Mainly they are used for creating symbolic links, junctions and mount points.
 
Essentially it's a way to redirect access from one file or folder to another. You may be wondering if I'm talking about a Shortcut? No, confusingly a shortcut is not a reparse point.
 
So how many ways does Windows have to redirect files / folders?
 
A lot. That's the problem!
 
Here they are off the top of my head:
A shortcut - A special file that is parsed by the Explorer shell that really links to another file somewhere else (as in a Start menu shortcut).

Most people are probably familiar with this because it's readily available in the Explorer UI.
  Symbolic file link - A file that points to some other file somewhere else. Confusingly, Windows Explorer also calls these "shortcuts" in the file properties dialog.

A symbolic link can be created by the mklink utility with no options.
  Symbolic directory link - These are relatively new, as they were introduced in Windows Vista. This is essentially a directory that points to another directory somewhere else.

These can be created by using mklink /D.
  Directory junction point - These are very similar to "symbolic directory links", but they were available prior to Windows Vista. Again, it is essentially a directory that points to another directory somewhere else. Some people make the mistake that a junction is only capable of pointing to another directory on the same volume, and that's not the case.

These can be created by using mklink /J.
  Mount point - Mount points allow you to designate a directory that will point to another volume. These are typically used to "mount" a number of drives as directories under some other drive letter, thus saving drive letters.

These can be created from Disk Management.
  File hard link - Yet another way to make a file point to another file. However, this method can only be used to point a file to some other file on the same volume.

These are created using mklink /H.  
Yes, that's a lot of ways that you can redirect files / folders in Windows. Try Googling these and you can see the confusion that ensues as to what the differences are between each.
 
So what is the difference between all of these?
 
Well, instead of pointing out the pros and cons, I'll tell you how each one of them works under the hood and you can decide for yourself:
A shortcut - This is the most "user friendly" way of creating a file that points to another one. Even the name makes sense, "shortcut", imagine that. It's readily available from the Windows Explorer menus and works entirely in user mode. A special .lnk file is created that the user mode shell knows how to parse. In Windows Explorer, an icon with a little arrow is shown to you to let you know that this is really a shortcut.

However, as far as the kernel and file system are concerned, there is nothing special about the .lnk file, it's just a regular file.
  Symbolic file link - Sometimes called a "symlink" or "soft link", this is a system that redirects one file to another, purely in the kernel. It involves some special metadata that is stored with the "source link" file that points to the "target destination file" and requires coordination between the file system and the Windows I/O Manager.

This system uses what are called "reparse points".
  Symbolic directory link - This is exactly the same thing as a symbolic file link, but it works on directories. The reason why I separated the two is because symbolic directory links were not available prior to Windows Vista and they must be created differently.

However, the underlying technology that enables this is exactly the same. This too uses "reparse points".
  Directory junction point - This is similar to a Symbolic directory link except that it is available prior to Windows Vista and uses an older technique. Technically speaking, the main difference between this and symbolic directory links is that directory junction points always point to an absolute path, while symbolic directory links can point to relative or absolute paths.

Surprisingly, this too uses "reparse points", but not all reparse points are the same. I'll get to that soon.
  Mount point - These are implemented in the exact same way as directory junction points, except that they point to the root of some other volume instead of some other directory.

These are implemented with the exact same "reparse points" as directory junctions.
  File hard link - This is purely a file system construct. Because of the way directory indexes work in NTFS, it is possible to add a file entry to a directory index of a file that already exists under some other directory. Essentially, you can think of the file as being in 2 (or more) places at once. While this is not quantum physics, it is NTFS. Each file has a "reference count" and that count is incremented whenever a hard link is created to it. When the count reaches 0, the file is deleted.

No other kernel subsystem is involved and no "reparse points" exists. This is the cleanest and purest way of making a file appear in 2 places at once (IMO). Wow, and all this works together reliably?
 
Yes, and that's what StableBit DrivePool is trying to preserve. You see, right now the only thing that we support on the pool from the above list are shortcuts. Everything else is not supported.
 
Some people have been requesting the support of file / directory symbolic links and junctions. Those 2 can be used by software in order to create complex directory structures, in order to organize your data better.
 
4 out of the 5 unsupported technologies use "reparse points", so it makes sense for StableBit DrivePool to implement support for them.
 
Ok, so what's a "reparse point"?
 
A reparse point is a Microsoft defined data structure that gets associated with a file or a directory. When that file or directory has a reparse point associated with it, then it becomes a kind of link to "somewhere else".
 
Essentially, when a file system encounters a reparse point, it tells the I/O Manager "these aren't the droids you're looking for, go look here". The I/O Manager is responsible for opening files, so it happily obliges.
 
That doesn't sound too complicated
 
Well, it isn't, except that there are different types of "reparse points" and each reparse point has a different meaning of where to go next.
 
For example:
File / directory symbolic links use a "symlink" reparse point. Directory junction points / mount points use a "mount point" reparse point. Any 3rd party developers can develop their own type of reparse points and their own logic as to how they work. Remember drive extender from WHS v1? Yep, those tombstones were yet another kind of reparse points. Ok, so this is complicated. But will StableBit DrivePool support reparse points?
 
I'm working hard towards that goal, and the reason why I'm writing this is because I believe that I've finally cracked the architecture that we need to support all Microsoft and 3rd party reparse points on the pool.
 
The architecture has these positive aspects to it:
It supports file / directory symbolic links, directory junction points, mount points, and 3rd party reparse points on the pool.
  It is a 100% native kernel implementation, with no dependence on the user mode service.
  It follows the 0 local metadata approach of storing everything needed on the pool itself and does not rely on something like the registry. This means that your reparse points will work when moving pools between machines (provided that you didn't link to something off of the pool that no longer exists on the new machine). Some of my previous attempts had these limitations:
Requires the user mode service to run additional periodic maintenance tasks on the pool.
  No support for directory reparse points, only file ones.
  Adding a drive to the pool would require a somewhat lengthy reparse point pass. The new architecture that I came up with has none of these limitations. All it requires is NTFS and Windows.
 
When will it be ready?
 
I'd hate to predict, but I think that it should be deployed in BETA form in a few weeks.

Future Developments for StableBit DrivePool
 
The current development status for all of our products can now be found here:
http://wiki.covecube.com/Development_Status

I've been thinking about how I can better communicate what is in store for each of our products, there are 3 now and another one in the works. Starting today I'll be setting up topics in each forum that I'll be updating on a regular basis. Each post will maintain what the future holds for each product.
 
I try to keep a lot of the development driven by user feedback, but most of that feedback doesn't happen in the public forum (but usually in tech support tickets). I'd just like to take this opportunity to highlight the direction that each product is heading in, a kind of roadmap.
 
I'll be setting up those posts today so look for them popping up soon in each respective forum.

Keeping this board on topic, I'd like to talk about something pretty technical that is central to how the StableBit Scanner operates, and perhaps get some feedback on the topic.
 
One of the benefits of running the StableBit Scanner is not just to predict drive failure, but to prevent it. The technical term for what the StableBit Scanner performs on your drives to prevent data loss is called Data Scrubbing (see: http://en.wikipedia.org/wiki/Data_scrubbing). By periodically scanning the entire surface of the drive you are actually causing the drive to inspect its own surface for defects and to recognize those defects before they turn into what is technically called a latent sector error (i.e. a sector that can't be read).
 
In order to do the periodic surface scan of a disk, the StableBit Scanner needs to know when it scanned a disk last, which means that it needs to identify a disk uniquely and remember which sectors it has scanned last and when. The StableBit Scanner uses sector ranges to remember exactly which parts of which disk were scanned when, but that's a whole other discussion.
 
I would like to focus this post on the issue of identifying a disk uniquely, which is absolutely required for data scrubbing to function properly, and this was overhauled in the latest BETA (2.5).
 
The original StableBit Scanner (1.0) used a very simple method to identify disks. StableBit Scanner 1.0 used the MBR signature to differentiate disks among each other.
 
For those who don't know what an MBR signature is, I'll explain it briefly here. When you buy a new disk from the store, it's probably completely blank. In other words, the disk contains all 0's written to it throughout. There is absolutely nothing written to it to differentiate it from any other disks (other than the serial number, which may not be accessible from Windows).
 
When you first connect such a blank disk to a Windows machine it will ask you whether you want to "initialize" it. This "initialization" is actually the writing of the MBR (master boot record, see: https://en.wikipedia.org/wiki/Master_boot_record), or GPT (GUID Partition Table) if you so choose. The MBR and GPT define the header (and perhaps footer) of the disk, kind of like when you write a letter to someone and you have a standard header and footer that always follows the same format.
 
One of the things that initializing a disk does is write a unique "signature" to it in the MBR or GPT. It's simply a long random number that identifies a disk uniquely. The problem with a MBR signature is that the random number is not large enough and so it is only meant to be unique on one single system. So if you connect a disk from a different computer, the disk signature on the foreign disk has a miniscule chance of being the same as a disk on the system that its being connected to.
 
Well, for the StableBit Scanner 1.0 this would be a problem. It would recognize the new disk as being the old disk, which would cause all sorts of issues. For one, you can't have the same disk connected to the same computer twice. That's simply not possible and we would write out an error report and crash.
 
StableBit Scanner 2.0 improved things a bit by utilizing the GPT signature, which was guaranteed to be unique across multiple systems. The only problem with using the GPT disk signature to identify disks uniquely is that disk cloning software is capable of placing the same signature on 2 different physical disks which would end up causing the same problem. In addition, many disks still utilize MBR, so we can't solely rely on GPT to resolve this issue.
 
As you can see this has not been an easy problem to solve
 
In the latest StableBit Scanner 2.5 BETA I've completely overhauled how we associate disk scan history (and other persistent settings) with each disk in the system. This is a major change from how things used to work before.
 
In 2.5 we now have a brand new Disk ID system. The Disk ID system is heuristic based and picks the best disk scan history that it knows of based on the available information. We no longer rely on a single factor such as a MBR or GPT. Instead, we survey a combination of disk identifiers and pick the disk scan history that fits the available data best.
 
Here is the list of factors that we use, starting from the highest priority:
Direct I/O disk serial number GPT Signature + WMI Serial number + Disk size GPT Signature + WMI Model + Disk size GPT Signature + Disk size MBR Signature + WMI Serial number + Disk size MBR Signature + WMI Model + Disk size MBR Signature + Disk size See the change log for more info on what this change entails. I hope that you give the new build a try.
 
This post has definitely been on the technical side, but that's what this forum is all about
 
Let me know if you have any comments or suggestions (or can find a better way to identify disks uniquely).

Yep, that's what I was thinking as well when I made the initial post. But many people seem to love the feature, so it's not going away