Christopher (Drashna)

In part because it's a way to generate more interaction on the forum here, and because it's come up a few times...
 
A couple of pieces of software I highly recommend (well, aside from DrivePool and Scanner, that is):
 
Subsonic:
It's media streaming software. It has a "standalone" version and "WAR" version that you can install on your server (or desktop) so you can stream your music and video to the web. Or even to your mobile device. It works great with DrivePool, as well. It does require a yearly subscription for the video, and the app support, but it's dirt cheap.
Personally, I use a fork of it called "Madsonic", as it's much more actively developed (more than one update a year).  ESET:
I use this on all of my computers. I've been a long time supporter of the company and love their product. In fact, I have it installed  on my server (Server 2012 Essentials), though I am using the "ESET File Server" edition. It's cheap, and has all the features that you'd want for a file server. Otherwise, NOD32 works great on all desktop Windows versions, as well as Windows Home Server. 
Their licensing is also "version independent", so you can look for a cheap copy of an older version, and update to the newest version out. It is a good, cheap way to get multiple PCs protected. And it also uses Windows PE to create an offline scanner. Raxco Perfect DIsk:
While it may be debatable if defragmentation is needed... I've seen it greatly improve performance. Especially on WHSv1.  It's even SSD aware, and has it's own method for optimizing SSDs.  And personally, I find that the "Performance Aggressive" SMART Placement option does seem to help with performance.

Yes it is.
 
Right now, Alex is busy trying to add Reparse Point support before adding this.

@dbailey: yeah, Comcast... well, they have their own term for horribleness: "Comcastic". 
But in their defense, it's easier to charge more for bandwidth than it is to spend millions to actually upgrade the infrastructure. Just saying.
 
My ISP is only slightly better than them. And they have a worst name, one that they seem to try to live up to: Cox.
 
 
If you intend on using the Cloud a lot, or host a website on your connection, it may be worth looking into the business accounts. They tend to not have the bandwidth caps, don't block *any* ports (80 or 25 for example, which are HTTP and SMTP respectively)
 
 
 
 
Though, for the future, you could get away with dumping the output of a dir command to a text file, and do so on a schedule.

Ok, so reparse points have definitely been driving me nuts lately. I was planning on releasing the StableBit DrivePool 2.1 BETA about a week after the 2.0 final with reparse point support, but it's still not out and it's because of reparse points. I've been trying different architectures over the past few weeks and none of them panned out as expected.
 
But today, I believe that I've finally got something that will work. It will support all of the various kinds of reparse points, it will be super fast and stable.
 
So what are these "reparse points" you may be asking?
 
Well, reparse points are the underlying file system technology that enable a whole suite of functionality. Mainly they are used for creating symbolic links, junctions and mount points.
 
Essentially it's a way to redirect access from one file or folder to another. You may be wondering if I'm talking about a Shortcut? No, confusingly a shortcut is not a reparse point.
 
So how many ways does Windows have to redirect files / folders?
 
A lot. That's the problem!
 
Here they are off the top of my head:
A shortcut - A special file that is parsed by the Explorer shell that really links to another file somewhere else (as in a Start menu shortcut).

Most people are probably familiar with this because it's readily available in the Explorer UI.
  Symbolic file link - A file that points to some other file somewhere else. Confusingly, Windows Explorer also calls these "shortcuts" in the file properties dialog.

A symbolic link can be created by the mklink utility with no options.
  Symbolic directory link - These are relatively new, as they were introduced in Windows Vista. This is essentially a directory that points to another directory somewhere else.

These can be created by using mklink /D.
  Directory junction point - These are very similar to "symbolic directory links", but they were available prior to Windows Vista. Again, it is essentially a directory that points to another directory somewhere else. Some people make the mistake that a junction is only capable of pointing to another directory on the same volume, and that's not the case.

These can be created by using mklink /J.
  Mount point - Mount points allow you to designate a directory that will point to another volume. These are typically used to "mount" a number of drives as directories under some other drive letter, thus saving drive letters.

These can be created from Disk Management.
  File hard link - Yet another way to make a file point to another file. However, this method can only be used to point a file to some other file on the same volume.

These are created using mklink /H.  
Yes, that's a lot of ways that you can redirect files / folders in Windows. Try Googling these and you can see the confusion that ensues as to what the differences are between each.
 
So what is the difference between all of these?
 
Well, instead of pointing out the pros and cons, I'll tell you how each one of them works under the hood and you can decide for yourself:
A shortcut - This is the most "user friendly" way of creating a file that points to another one. Even the name makes sense, "shortcut", imagine that. It's readily available from the Windows Explorer menus and works entirely in user mode. A special .lnk file is created that the user mode shell knows how to parse. In Windows Explorer, an icon with a little arrow is shown to you to let you know that this is really a shortcut.

However, as far as the kernel and file system are concerned, there is nothing special about the .lnk file, it's just a regular file.
  Symbolic file link - Sometimes called a "symlink" or "soft link", this is a system that redirects one file to another, purely in the kernel. It involves some special metadata that is stored with the "source link" file that points to the "target destination file" and requires coordination between the file system and the Windows I/O Manager.

This system uses what are called "reparse points".
  Symbolic directory link - This is exactly the same thing as a symbolic file link, but it works on directories. The reason why I separated the two is because symbolic directory links were not available prior to Windows Vista and they must be created differently.

However, the underlying technology that enables this is exactly the same. This too uses "reparse points".
  Directory junction point - This is similar to a Symbolic directory link except that it is available prior to Windows Vista and uses an older technique. Technically speaking, the main difference between this and symbolic directory links is that directory junction points always point to an absolute path, while symbolic directory links can point to relative or absolute paths.

Surprisingly, this too uses "reparse points", but not all reparse points are the same. I'll get to that soon.
  Mount point - These are implemented in the exact same way as directory junction points, except that they point to the root of some other volume instead of some other directory.

These are implemented with the exact same "reparse points" as directory junctions.
  File hard link - This is purely a file system construct. Because of the way directory indexes work in NTFS, it is possible to add a file entry to a directory index of a file that already exists under some other directory. Essentially, you can think of the file as being in 2 (or more) places at once. While this is not quantum physics, it is NTFS. Each file has a "reference count" and that count is incremented whenever a hard link is created to it. When the count reaches 0, the file is deleted.

No other kernel subsystem is involved and no "reparse points" exists. This is the cleanest and purest way of making a file appear in 2 places at once (IMO). Wow, and all this works together reliably?
 
Yes, and that's what StableBit DrivePool is trying to preserve. You see, right now the only thing that we support on the pool from the above list are shortcuts. Everything else is not supported.
 
Some people have been requesting the support of file / directory symbolic links and junctions. Those 2 can be used by software in order to create complex directory structures, in order to organize your data better.
 
4 out of the 5 unsupported technologies use "reparse points", so it makes sense for StableBit DrivePool to implement support for them.
 
Ok, so what's a "reparse point"?
 
A reparse point is a Microsoft defined data structure that gets associated with a file or a directory. When that file or directory has a reparse point associated with it, then it becomes a kind of link to "somewhere else".
 
Essentially, when a file system encounters a reparse point, it tells the I/O Manager "these aren't the droids you're looking for, go look here". The I/O Manager is responsible for opening files, so it happily obliges.
 
That doesn't sound too complicated
 
Well, it isn't, except that there are different types of "reparse points" and each reparse point has a different meaning of where to go next.
 
For example:
File / directory symbolic links use a "symlink" reparse point. Directory junction points / mount points use a "mount point" reparse point. Any 3rd party developers can develop their own type of reparse points and their own logic as to how they work. Remember drive extender from WHS v1? Yep, those tombstones were yet another kind of reparse points. Ok, so this is complicated. But will StableBit DrivePool support reparse points?
 
I'm working hard towards that goal, and the reason why I'm writing this is because I believe that I've finally cracked the architecture that we need to support all Microsoft and 3rd party reparse points on the pool.
 
The architecture has these positive aspects to it:
It supports file / directory symbolic links, directory junction points, mount points, and 3rd party reparse points on the pool.
  It is a 100% native kernel implementation, with no dependence on the user mode service.
  It follows the 0 local metadata approach of storing everything needed on the pool itself and does not rely on something like the registry. This means that your reparse points will work when moving pools between machines (provided that you didn't link to something off of the pool that no longer exists on the new machine). Some of my previous attempts had these limitations:
Requires the user mode service to run additional periodic maintenance tasks on the pool.
  No support for directory reparse points, only file ones.
  Adding a drive to the pool would require a somewhat lengthy reparse point pass. The new architecture that I came up with has none of these limitations. All it requires is NTFS and Windows.
 
When will it be ready?
 
I'd hate to predict, but I think that it should be deployed in BETA form in a few weeks.

Kayot,
 
The main thing that keeps DrivePool from running on XP is the driver we use for the pool.
And as for what keeps us from writing an XP driver for DrivePool is that XP has exited mainstream support in 2009, and it's hitting End of Life in almost six months:
http://support.microsoft.com/lifecycle/?p1=3223
(April 2014)
 
By the time we have a fully stable version, the OS will cease getting even critical security updates.

It's nice to see your setup. 
 
Very small, and likely fairly power efficent (especially compared to mine).
And I'm glad that you're using a POWERED USB hub. 

Hi all.
First of all, thanks to developers for such a simple yet great tool! I've been using it only for two days, but liked it very much.
I have some propositions for the further product improvements, and will describe them in the forum
First of them, that could make life easier, is displaying some kind of overlay icons on the folders in the windows files explorer, like in the duplication tree, that will show that folder is x2 or x1+ level of duplication. (it's not always useful to open duplication option dialog to find the "safe" folders.
Of course x1 is useless overlay except the case it's not inherit. Well, I bet you know what to do better than me

I've been thinking about how I can better communicate what is in store for each of our products, there are 3 now and another one in the works. Starting today I'll be setting up topics in each forum that I'll be updating on a regular basis. Each post will maintain what the future holds for each product.
 
I try to keep a lot of the development driven by user feedback, but most of that feedback doesn't happen in the public forum (but usually in tech support tickets). I'd just like to take this opportunity to highlight the direction that each product is heading in, a kind of roadmap.
 
I'll be setting up those posts today so look for them popping up soon in each respective forum.

I've certainly forgot. 

Instead of removing the drive letters completely, I'd recommend mounted the drives to NTFS folders, that way you can easily run chkdsk on the drives if/when you need to.
 
http://wiki.covecube.com/StableBit_DrivePool_Q4822624
This is basically what you want, but it is a bit more manual... (but simple)
 
As for the Add disk, if it's not initialized or formatted, DrivePool will do that automatically, actually. And the Virtual Disk Service assigns a new drive letter automatically. 

Have you ever tried Auslogics Disk Defrag?
 
I use it on a number of my machines around the house.  But haven't put it on my WHS2011 machine (using DrivePool 1.x and Scanner).

I just have to mention that, personally, I've used SyncBack SE (not Pro) for years. It's a great piece of software and reasonably priced too.

I'd like the ability to say "keep together all files in any|each folder of folderpath" where "any" and "each" are modifiers affecting which files to keep together.
 
So given an example where folderpath was p:\media\sketches\* and I had files in p:\media\sketches\2012 and p:\media\sketches\2013
 
if the "any" modifier was selected 
then the files in sketches\2012 and sketches\2013 and any subfolders thereof would be kept all together
 
but if the "each" modifier was selected
and the files in sketches\2012 and any subfolders thereof would be kept together
and the files in sketches\2013 and any subfolders thereof would be kept together
yet sketches\2012 and sketches\2013 would be stored independently of each other.
 
Also, something along the lines of if folderpath ends in "folder\" it also applies to any files within that folder itself, while if it ends in "folder\*" it only applies to files within that folder's subfolders.
 
I hope this makes sense.

At firtst I wasn't sure I would have any use for such a feature, but the more I think about it, the more I like the idea of having more control over what goes where.  It could be very useful as my children get old and skilled enough to use the server.  Bring on the progress!
 
Oh, yeah, and thanks so much for all your hard work Alex!  Drivepool and Scanner are awesome and the WS 2012 E integration in the last update felt like an early Christmas present.

Yes, the bandwidth to upload/download from the cloud is seldom mentioned, the promoters seem to want the public to forget about the cost of that. Example, tablet users with "free" storage space on the web, using cellular connection with limited bandwidth.
The company I work at is looking at updating our SAN storage, what cost us $600K seven years ago cost less than $300K with greatly increased bandwidth and much better software [tiered] with SSDs on the front.
So yes, I agree storage is making great strides, but the bottleneck will be for the foreseeable future the connection from home to remote storage. Around dinner time in my neighborhood eveything slows to a crawl as the drones connect to netflix.

This is something that I have hoped for, for a very long time. I really have never been keen on having things scattered around the pool. Especially with music... 10 album tracks scattered over 5 or 6 drives.. But then I'm probably a bit O.C.D.
 
I'm sure for people who have various devices streaming to different rooms this must be a good thing. Knowing that all the Disney films are all on one hard drive for the rug rats, and the teenage daughter can watch her twilight knowing its on a separate drive so no risk of intensive I/O and so on. And yes i know that this could be achieved by organising multiple pools. However when you get to 13 drives and around 22TB of data creating new pools seems like a hassle. 
 
First thought is that this would eliminate the need.
Second thought is that once implemented the folder placement would to my mind then simplify the operation of creating separate pools and may actually lead me to do it, instead of just thinking about it 
 
I'm all for it!!!
 
.

I can be a bit wordy by my very nature. But yes, that's exactly what I'm talking about, controlling which files go onto which pool part.
 
And, in the future, a pool part may not necessarily represent a single local physical disk, which would make this even more interesting

Oh and thank you
 
I'm no expert at public speaking, but I do like to talk about the products that I've built, which I believe in very strongly.

Just to follow up on this,
 
I've done extensive troubleshooting on this issue with Kihim in a support ticket. It seems like this is caused by the system getting bogged down at certain times leading to a periodic slowdown in the UI. At least that's what I can see from the Scanner's built-in UI performance profiler (accessed by pressing P in a BETA build).
 
I don't see any Scanner UI task running away with the CPU. I've also set up a local test rig with similar specs, 26 disks and a 2Ghz 8-core AMD server, and have run the Scanner UI for hours looking for any slowdowns and I couldn't find any.
 
But I think the bottom line here is that the Scanner uses WPF to render its UI, and WPF does require some more CPU resources than your typical Win32 application. I think that that's really the core issue here. So in the future I would like to offer a "lite" version of the UI for situations where CPU resources are scarce. I imagine that there will be a simple drop down that will let you switch between "standard" and "lite" versions of the UI, letting you run whichever one you want.

I've found "positive pressure" filtered cases can noticeably reduce dust and salt intake; you just have to remember to clean the filters every so often to maintain the pressure ratio (and also so that the case doesn't overheat - and for this I can plug Scanner, with it's emailed and spoken overheat warnings).

Well I'm so impressed with it (especially after trying the Scanner too) that I've bought the 2 app deal after only 2 days trial
 
Using this on my N54L Microserver which is purely used as my media store/Plex server + torrents at home. I'm finding it much easier to understand/use than FlexRAID Pooling I purchased last year. For this server its more about max storage space than redundancy.
 
I only really need Folder Duplication for critical frequently changed/added files (photos/Home Movies), as I backup to another Microserver on a schedule, so no big deal getting my large Movie files back off the backup server if a drive fails.
 
 
Couple of quick questions. Will Scanner, with its installed defaults, still allow Windows Hard Drive standby or do I need to untick Advanced Power Management which is currently set to Maximum performance (no standby) as installed? This server is idling most of the time consuming only ~25W so need to ensure drive spin-down working as before (currently set in Windows8 to 30 minutes). I also use Lights-out to put the whole system in hibernation at set times based on Calendar when its only then consuming 1w. Will the "prevent-standby" type options clash with those in Lights-out? For example, with uTorrent I set Lights-out to not Hibernate if the uTorrent process is still running. Would I need to set a similar kind of rule to make sure it doesn't hibernate in Balancing/Scanning drives etc.?
 
I also have a G8 Microserver with a proper P222/512 RAID card currently with 4x3TB RAID5 (used as my lab & Media Backup from other server). I'm assuming Drivepool/Scanner doesn't really add anything to this unless I ran as individual drives ??
 
Also, my drive pool of currently a 4TB & 1.5TB show as 5TB usable pool. Why does Windows Drive Management show it as 2047.97GB and also in the Task Manager graphs also shows it as only 2TB Capacity for Covecube Virtual Disk?? Also, in Task Manager the DrivePool disk shows no Activity while the pooled disks themselves do. Does the Virtual DrivePool not interact correctly with Task Manager etc.?

Yes, you can definitely configure Scanner to send you email notifications. 

I have stablebit scanner installed, along with server smart to evaluate hard drive health. Drive pool is also installed, and there are 3 hard drives in my pool.  I mapped 2 of the DrivePool folders to my workstations, and use those folders all the time (one holds all my photos, the other holds all my videos).
 
           The other day I received a warning on the dashboard along with an email about one of the drives parking too many times which indicates a possible failure. This warning was only send from Scanner, not from server smart, so I looked into it and the drive seemed to be OK as far as performance, I thought maybe this Scanner is just a little too sensitive to some sort of reading because server smart had all check marks and did not show anything wrong with the drive. Needless to say, the drive failed about a month later. Not only did the one product prove to be a great value (Scanner), but the second product saved my data due to me not listening to Scanner’s warnings.  Drive pool showed the drive as missing after it failed, I chose the remove drive option, and suffered no loss of any data. The remaining 2 drives balanced out and all is good.
 
I just wanted to post this little story and to thank the team that has created and continues to support these products. I also feel that this post will help others decide to put faith into the product and have some piece of mind that their data is safe with proper use of the software.
Thank YOU! 
 
 

Thanks!  I was not aware of this setting.  By the way, you don't have to go through all that for RDM anymore.  All you need to do is the following and then any disk (not USB) you plugin in thereafter will be available for RDM:
In the ESXi Console window, highlight your server Go to the Configuration tab Under Software, click Advanced Settings Click RdmFilter Uncheck the box for RdmFilter.HbaIsShared Click OK

Hi Val3ntin,
 
It is feasible, as it works for me.  I have WSE2012 in ESXi 5.1.
 
1. Link your physical disks to ESXi using the RDM technique described in this blog (http://blog.davidwarburton.net/2010/10/25/rdm-mapping-of-local-sata-storage-for-esxi/)
2. In file "C:\Program Files (x86)\StableBit\Scanner\Service\Scanner.Service.exe.config", change setting "UnsafeDirectIo" to "True" - don't forget to reboot !
 
 
Good luck.