Christopher (Drashna)

Ok, so reparse points have definitely been driving me nuts lately. I was planning on releasing the StableBit DrivePool 2.1 BETA about a week after the 2.0 final with reparse point support, but it's still not out and it's because of reparse points. I've been trying different architectures over the past few weeks and none of them panned out as expected.
 
But today, I believe that I've finally got something that will work. It will support all of the various kinds of reparse points, it will be super fast and stable.
 
So what are these "reparse points" you may be asking?
 
Well, reparse points are the underlying file system technology that enable a whole suite of functionality. Mainly they are used for creating symbolic links, junctions and mount points.
 
Essentially it's a way to redirect access from one file or folder to another. You may be wondering if I'm talking about a Shortcut? No, confusingly a shortcut is not a reparse point.
 
So how many ways does Windows have to redirect files / folders?
 
A lot. That's the problem!
 
Here they are off the top of my head:
A shortcut - A special file that is parsed by the Explorer shell that really links to another file somewhere else (as in a Start menu shortcut).

Most people are probably familiar with this because it's readily available in the Explorer UI.
  Symbolic file link - A file that points to some other file somewhere else. Confusingly, Windows Explorer also calls these "shortcuts" in the file properties dialog.

A symbolic link can be created by the mklink utility with no options.
  Symbolic directory link - These are relatively new, as they were introduced in Windows Vista. This is essentially a directory that points to another directory somewhere else.

These can be created by using mklink /D.
  Directory junction point - These are very similar to "symbolic directory links", but they were available prior to Windows Vista. Again, it is essentially a directory that points to another directory somewhere else. Some people make the mistake that a junction is only capable of pointing to another directory on the same volume, and that's not the case.

These can be created by using mklink /J.
  Mount point - Mount points allow you to designate a directory that will point to another volume. These are typically used to "mount" a number of drives as directories under some other drive letter, thus saving drive letters.

These can be created from Disk Management.
  File hard link - Yet another way to make a file point to another file. However, this method can only be used to point a file to some other file on the same volume.

These are created using mklink /H.  
Yes, that's a lot of ways that you can redirect files / folders in Windows. Try Googling these and you can see the confusion that ensues as to what the differences are between each.
 
So what is the difference between all of these?
 
Well, instead of pointing out the pros and cons, I'll tell you how each one of them works under the hood and you can decide for yourself:
A shortcut - This is the most "user friendly" way of creating a file that points to another one. Even the name makes sense, "shortcut", imagine that. It's readily available from the Windows Explorer menus and works entirely in user mode. A special .lnk file is created that the user mode shell knows how to parse. In Windows Explorer, an icon with a little arrow is shown to you to let you know that this is really a shortcut.

However, as far as the kernel and file system are concerned, there is nothing special about the .lnk file, it's just a regular file.
  Symbolic file link - Sometimes called a "symlink" or "soft link", this is a system that redirects one file to another, purely in the kernel. It involves some special metadata that is stored with the "source link" file that points to the "target destination file" and requires coordination between the file system and the Windows I/O Manager.

This system uses what are called "reparse points".
  Symbolic directory link - This is exactly the same thing as a symbolic file link, but it works on directories. The reason why I separated the two is because symbolic directory links were not available prior to Windows Vista and they must be created differently.

However, the underlying technology that enables this is exactly the same. This too uses "reparse points".
  Directory junction point - This is similar to a Symbolic directory link except that it is available prior to Windows Vista and uses an older technique. Technically speaking, the main difference between this and symbolic directory links is that directory junction points always point to an absolute path, while symbolic directory links can point to relative or absolute paths.

Surprisingly, this too uses "reparse points", but not all reparse points are the same. I'll get to that soon.
  Mount point - These are implemented in the exact same way as directory junction points, except that they point to the root of some other volume instead of some other directory.

These are implemented with the exact same "reparse points" as directory junctions.
  File hard link - This is purely a file system construct. Because of the way directory indexes work in NTFS, it is possible to add a file entry to a directory index of a file that already exists under some other directory. Essentially, you can think of the file as being in 2 (or more) places at once. While this is not quantum physics, it is NTFS. Each file has a "reference count" and that count is incremented whenever a hard link is created to it. When the count reaches 0, the file is deleted.

No other kernel subsystem is involved and no "reparse points" exists. This is the cleanest and purest way of making a file appear in 2 places at once (IMO). Wow, and all this works together reliably?
 
Yes, and that's what StableBit DrivePool is trying to preserve. You see, right now the only thing that we support on the pool from the above list are shortcuts. Everything else is not supported.
 
Some people have been requesting the support of file / directory symbolic links and junctions. Those 2 can be used by software in order to create complex directory structures, in order to organize your data better.
 
4 out of the 5 unsupported technologies use "reparse points", so it makes sense for StableBit DrivePool to implement support for them.
 
Ok, so what's a "reparse point"?
 
A reparse point is a Microsoft defined data structure that gets associated with a file or a directory. When that file or directory has a reparse point associated with it, then it becomes a kind of link to "somewhere else".
 
Essentially, when a file system encounters a reparse point, it tells the I/O Manager "these aren't the droids you're looking for, go look here". The I/O Manager is responsible for opening files, so it happily obliges.
 
That doesn't sound too complicated
 
Well, it isn't, except that there are different types of "reparse points" and each reparse point has a different meaning of where to go next.
 
For example:
File / directory symbolic links use a "symlink" reparse point. Directory junction points / mount points use a "mount point" reparse point. Any 3rd party developers can develop their own type of reparse points and their own logic as to how they work. Remember drive extender from WHS v1? Yep, those tombstones were yet another kind of reparse points. Ok, so this is complicated. But will StableBit DrivePool support reparse points?
 
I'm working hard towards that goal, and the reason why I'm writing this is because I believe that I've finally cracked the architecture that we need to support all Microsoft and 3rd party reparse points on the pool.
 
The architecture has these positive aspects to it:
It supports file / directory symbolic links, directory junction points, mount points, and 3rd party reparse points on the pool.
  It is a 100% native kernel implementation, with no dependence on the user mode service.
  It follows the 0 local metadata approach of storing everything needed on the pool itself and does not rely on something like the registry. This means that your reparse points will work when moving pools between machines (provided that you didn't link to something off of the pool that no longer exists on the new machine). Some of my previous attempts had these limitations:
Requires the user mode service to run additional periodic maintenance tasks on the pool.
  No support for directory reparse points, only file ones.
  Adding a drive to the pool would require a somewhat lengthy reparse point pass. The new architecture that I came up with has none of these limitations. All it requires is NTFS and Windows.
 
When will it be ready?
 
I'd hate to predict, but I think that it should be deployed in BETA form in a few weeks.

Kayot,
 
The main thing that keeps DrivePool from running on XP is the driver we use for the pool.
And as for what keeps us from writing an XP driver for DrivePool is that XP has exited mainstream support in 2009, and it's hitting End of Life in almost six months:
http://support.microsoft.com/lifecycle/?p1=3223
(April 2014)
 
By the time we have a fully stable version, the OS will cease getting even critical security updates.

It's nice to see your setup. 
 
Very small, and likely fairly power efficent (especially compared to mine).
And I'm glad that you're using a POWERED USB hub. 

Hi all.
First of all, thanks to developers for such a simple yet great tool! I've been using it only for two days, but liked it very much.
I have some propositions for the further product improvements, and will describe them in the forum
First of them, that could make life easier, is displaying some kind of overlay icons on the folders in the windows files explorer, like in the duplication tree, that will show that folder is x2 or x1+ level of duplication. (it's not always useful to open duplication option dialog to find the "safe" folders.
Of course x1 is useless overlay except the case it's not inherit. Well, I bet you know what to do better than me

I've been thinking about how I can better communicate what is in store for each of our products, there are 3 now and another one in the works. Starting today I'll be setting up topics in each forum that I'll be updating on a regular basis. Each post will maintain what the future holds for each product.
 
I try to keep a lot of the development driven by user feedback, but most of that feedback doesn't happen in the public forum (but usually in tech support tickets). I'd just like to take this opportunity to highlight the direction that each product is heading in, a kind of roadmap.
 
I'll be setting up those posts today so look for them popping up soon in each respective forum.

I've certainly forgot. 

Instead of removing the drive letters completely, I'd recommend mounted the drives to NTFS folders, that way you can easily run chkdsk on the drives if/when you need to.
 
http://wiki.covecube.com/StableBit_DrivePool_Q4822624
This is basically what you want, but it is a bit more manual... (but simple)
 
As for the Add disk, if it's not initialized or formatted, DrivePool will do that automatically, actually. And the Virtual Disk Service assigns a new drive letter automatically. 

Have you ever tried Auslogics Disk Defrag?
 
I use it on a number of my machines around the house.  But haven't put it on my WHS2011 machine (using DrivePool 1.x and Scanner).

I just have to mention that, personally, I've used SyncBack SE (not Pro) for years. It's a great piece of software and reasonably priced too.

I'd like the ability to say "keep together all files in any|each folder of folderpath" where "any" and "each" are modifiers affecting which files to keep together.
 
So given an example where folderpath was p:\media\sketches\* and I had files in p:\media\sketches\2012 and p:\media\sketches\2013
 
if the "any" modifier was selected 
then the files in sketches\2012 and sketches\2013 and any subfolders thereof would be kept all together
 
but if the "each" modifier was selected
and the files in sketches\2012 and any subfolders thereof would be kept together
and the files in sketches\2013 and any subfolders thereof would be kept together
yet sketches\2012 and sketches\2013 would be stored independently of each other.
 
Also, something along the lines of if folderpath ends in "folder\" it also applies to any files within that folder itself, while if it ends in "folder\*" it only applies to files within that folder's subfolders.
 
I hope this makes sense.

At firtst I wasn't sure I would have any use for such a feature, but the more I think about it, the more I like the idea of having more control over what goes where.  It could be very useful as my children get old and skilled enough to use the server.  Bring on the progress!
 
Oh, yeah, and thanks so much for all your hard work Alex!  Drivepool and Scanner are awesome and the WS 2012 E integration in the last update felt like an early Christmas present.

Yes, the bandwidth to upload/download from the cloud is seldom mentioned, the promoters seem to want the public to forget about the cost of that. Example, tablet users with "free" storage space on the web, using cellular connection with limited bandwidth.
The company I work at is looking at updating our SAN storage, what cost us $600K seven years ago cost less than $300K with greatly increased bandwidth and much better software [tiered] with SSDs on the front.
So yes, I agree storage is making great strides, but the bottleneck will be for the foreseeable future the connection from home to remote storage. Around dinner time in my neighborhood eveything slows to a crawl as the drones connect to netflix.

This is something that I have hoped for, for a very long time. I really have never been keen on having things scattered around the pool. Especially with music... 10 album tracks scattered over 5 or 6 drives.. But then I'm probably a bit O.C.D.
 
I'm sure for people who have various devices streaming to different rooms this must be a good thing. Knowing that all the Disney films are all on one hard drive for the rug rats, and the teenage daughter can watch her twilight knowing its on a separate drive so no risk of intensive I/O and so on. And yes i know that this could be achieved by organising multiple pools. However when you get to 13 drives and around 22TB of data creating new pools seems like a hassle. 
 
First thought is that this would eliminate the need.
Second thought is that once implemented the folder placement would to my mind then simplify the operation of creating separate pools and may actually lead me to do it, instead of just thinking about it 
 
I'm all for it!!!
 
.

I can be a bit wordy by my very nature. But yes, that's exactly what I'm talking about, controlling which files go onto which pool part.
 
And, in the future, a pool part may not necessarily represent a single local physical disk, which would make this even more interesting

Oh and thank you
 
I'm no expert at public speaking, but I do like to talk about the products that I've built, which I believe in very strongly.

Just to follow up on this,
 
I've done extensive troubleshooting on this issue with Kihim in a support ticket. It seems like this is caused by the system getting bogged down at certain times leading to a periodic slowdown in the UI. At least that's what I can see from the Scanner's built-in UI performance profiler (accessed by pressing P in a BETA build).
 
I don't see any Scanner UI task running away with the CPU. I've also set up a local test rig with similar specs, 26 disks and a 2Ghz 8-core AMD server, and have run the Scanner UI for hours looking for any slowdowns and I couldn't find any.
 
But I think the bottom line here is that the Scanner uses WPF to render its UI, and WPF does require some more CPU resources than your typical Win32 application. I think that that's really the core issue here. So in the future I would like to offer a "lite" version of the UI for situations where CPU resources are scarce. I imagine that there will be a simple drop down that will let you switch between "standard" and "lite" versions of the UI, letting you run whichever one you want.

I've found "positive pressure" filtered cases can noticeably reduce dust and salt intake; you just have to remember to clean the filters every so often to maintain the pressure ratio (and also so that the case doesn't overheat - and for this I can plug Scanner, with it's emailed and spoken overheat warnings).

Well I'm so impressed with it (especially after trying the Scanner too) that I've bought the 2 app deal after only 2 days trial
 
Using this on my N54L Microserver which is purely used as my media store/Plex server + torrents at home. I'm finding it much easier to understand/use than FlexRAID Pooling I purchased last year. For this server its more about max storage space than redundancy.
 
I only really need Folder Duplication for critical frequently changed/added files (photos/Home Movies), as I backup to another Microserver on a schedule, so no big deal getting my large Movie files back off the backup server if a drive fails.
 
 
Couple of quick questions. Will Scanner, with its installed defaults, still allow Windows Hard Drive standby or do I need to untick Advanced Power Management which is currently set to Maximum performance (no standby) as installed? This server is idling most of the time consuming only ~25W so need to ensure drive spin-down working as before (currently set in Windows8 to 30 minutes). I also use Lights-out to put the whole system in hibernation at set times based on Calendar when its only then consuming 1w. Will the "prevent-standby" type options clash with those in Lights-out? For example, with uTorrent I set Lights-out to not Hibernate if the uTorrent process is still running. Would I need to set a similar kind of rule to make sure it doesn't hibernate in Balancing/Scanning drives etc.?
 
I also have a G8 Microserver with a proper P222/512 RAID card currently with 4x3TB RAID5 (used as my lab & Media Backup from other server). I'm assuming Drivepool/Scanner doesn't really add anything to this unless I ran as individual drives ??
 
Also, my drive pool of currently a 4TB & 1.5TB show as 5TB usable pool. Why does Windows Drive Management show it as 2047.97GB and also in the Task Manager graphs also shows it as only 2TB Capacity for Covecube Virtual Disk?? Also, in Task Manager the DrivePool disk shows no Activity while the pooled disks themselves do. Does the Virtual DrivePool not interact correctly with Task Manager etc.?

Yes, you can definitely configure Scanner to send you email notifications. 

I have stablebit scanner installed, along with server smart to evaluate hard drive health. Drive pool is also installed, and there are 3 hard drives in my pool.  I mapped 2 of the DrivePool folders to my workstations, and use those folders all the time (one holds all my photos, the other holds all my videos).
 
           The other day I received a warning on the dashboard along with an email about one of the drives parking too many times which indicates a possible failure. This warning was only send from Scanner, not from server smart, so I looked into it and the drive seemed to be OK as far as performance, I thought maybe this Scanner is just a little too sensitive to some sort of reading because server smart had all check marks and did not show anything wrong with the drive. Needless to say, the drive failed about a month later. Not only did the one product prove to be a great value (Scanner), but the second product saved my data due to me not listening to Scanner’s warnings.  Drive pool showed the drive as missing after it failed, I chose the remove drive option, and suffered no loss of any data. The remaining 2 drives balanced out and all is good.
 
I just wanted to post this little story and to thank the team that has created and continues to support these products. I also feel that this post will help others decide to put faith into the product and have some piece of mind that their data is safe with proper use of the software.
Thank YOU! 
 
 

Thanks!  I was not aware of this setting.  By the way, you don't have to go through all that for RDM anymore.  All you need to do is the following and then any disk (not USB) you plugin in thereafter will be available for RDM:
In the ESXi Console window, highlight your server Go to the Configuration tab Under Software, click Advanced Settings Click RdmFilter Uncheck the box for RdmFilter.HbaIsShared Click OK

Hi Val3ntin,
 
It is feasible, as it works for me.  I have WSE2012 in ESXi 5.1.
 
1. Link your physical disks to ESXi using the RDM technique described in this blog (http://blog.davidwarburton.net/2010/10/25/rdm-mapping-of-local-sata-storage-for-esxi/)
2. In file "C:\Program Files (x86)\StableBit\Scanner\Service\Scanner.Service.exe.config", change setting "UnsafeDirectIo" to "True" - don't forget to reboot !
 
 
Good luck.

Actually, if the pool is missing *all* of the drives, then DrivePool "won't know about it".  
Once you re-attach one or all of the disks, then it will "discover" it and remount it. Making it accessible again.

WIndows Storage Server 2008R2 Essentials is basically just WHS2011, but with 25 users and computers. It's also OEM only, and much more "regulated". It's very hard to find without using Technet/MSDN, IIRC.
 
 
And .... WHS isn't dead. In fact, the entire WHS2011 code is *what* makes Server 2012 Essentials what it is. Literally.
 
 
Also, end of mainstream support means that it only gets critical security updates, IIRC. And that will last as long as Server 2008R2 is supported.
Mainstream support for 2008R2 is Jan 2015... while extended support (aka critical updates) last until 2020. So that's a good long while.
 
And Server 2012 R2 *is* Server 8.1. As Server 2012 is Server 8.
Also, the "Essentials" Role (aka the code that made WHS) will be available on Standard and Datacenter versions of Server 2012R2, IIRC.
 
 
 
And I'd wait for Server 2012 R2 Essentials, if you want to go this route (which isn't a bad idea). Namely, because it handles bare metal restores better (can create a recovery disk from scratch, and may integrate drivers, as well as supports PXE, aka network booting of the recovery image).

If the Remote Access website is enabled on the second, you could log into that and download the rdp file, and just keep that. It would be the simplest way.
 
Otherwise, you could just use Remote Desktop normally. (mstsc /v:servername)