Jump to content

  • Log in with Twitter Log in with Windows Live Log In with Google      Sign In   
  • Create Account

Photo

StableBit DrivePool - Reparse Points


  • Please log in to reply
12 replies to this topic

#1 Alex

Alex

    Lead Programmer

  • Administrators
  • 242 posts
  • LocationNew York, USA

Posted 31 October 2013 - 07:53 PM

Ok, so reparse points have definitely been driving me nuts lately. I was planning on releasing the StableBit DrivePool 2.1 BETA about a week after the 2.0 final with reparse point support, but it's still not out and it's because of reparse points. I've been trying different architectures over the past few weeks and none of them panned out as expected.

 

But today, I believe that I've finally got something that will work. It will support all of the various kinds of reparse points, it will be super fast and stable.

 

So what are these "reparse points" you may be asking?

 

Well, reparse points are the underlying file system technology that enable a whole suite of functionality. Mainly they are used for creating symbolic links, junctions and mount points.

 

Essentially it's a way to redirect access from one file or folder to another. You may be wondering if I'm talking about a Shortcut? No, confusingly a shortcut is not a reparse point.

 

So how many ways does Windows have to redirect files / folders?

 

A lot. That's the problem!

 

Here they are off the top of my head:

  • A shortcut - A special file that is parsed by the Explorer shell that really links to another file somewhere else (as in a Start menu shortcut).

    Most people are probably familiar with this because it's readily available in the Explorer UI.
     
  • Symbolic file link - A file that points to some other file somewhere else. Confusingly, Windows Explorer also calls these "shortcuts" in the file properties dialog.

    A symbolic link can be created by the mklink utility with no options.
     
  • Symbolic directory link - These are relatively new, as they were introduced in Windows Vista. This is essentially a directory that points to another directory somewhere else.

    These can be created by using mklink /D.
     
  • Directory junction point - These are very similar to "symbolic directory links", but they were available prior to Windows Vista. Again, it is essentially a directory that points to another directory somewhere else. Some people make the mistake that a junction is only capable of pointing to another directory on the same volume, and that's not the case.

    These can be created by using mklink /J.
     
  • Mount point - Mount points allow you to designate a directory that will point to another volume. These are typically used to "mount" a number of drives as directories under some other drive letter, thus saving drive letters.

    These can be created from Disk Management.
     
  • File hard link - Yet another way to make a file point to another file. However, this method can only be used to point a file to some other file on the same volume.

    These are created using mklink /H.

 

Yes, that's a lot of ways that you can redirect files / folders in Windows. Try Googling these and you can see the confusion that ensues as to what the differences are between each.

 

So what is the difference between all of these?

 

Well, instead of pointing out the pros and cons, I'll tell you how each one of them works under the hood and you can decide for yourself:

  • A shortcut - This is the most "user friendly" way of creating a file that points to another one. Even the name makes sense, "shortcut", imagine that. It's readily available from the Windows Explorer menus and works entirely in user mode. A special .lnk file is created that the user mode shell knows how to parse. In Windows Explorer, an icon with a little arrow is shown to you to let you know that this is really a shortcut.

    However, as far as the kernel and file system are concerned, there is nothing special about the .lnk file, it's just a regular file.
     
  • Symbolic file link - Sometimes called a "symlink" or "soft link", this is a system that redirects one file to another, purely in the kernel. It involves some special metadata that is stored with the "source link" file that points to the "target destination file" and requires coordination between the file system and the Windows I/O Manager.

    This system uses what are called "reparse points".
     
  • Symbolic directory link - This is exactly the same thing as a symbolic file link, but it works on directories. The reason why I separated the two is because symbolic directory links were not available prior to Windows Vista and they must be created differently.

    However, the underlying technology that enables this is exactly the same. This too uses "reparse points".
     
  • Directory junction point - This is similar to a Symbolic directory link except that it is available prior to Windows Vista and uses an older technique. Technically speaking, the main difference between this and symbolic directory links is that directory junction points always point to an absolute path, while symbolic directory links can point to relative or absolute paths.

    Surprisingly, this too uses "reparse points", but not all reparse points are the same. I'll get to that soon.
     
  • Mount point - These are implemented in the exact same way as directory junction points, except that they point to the root of some other volume instead of some other directory.

    These are implemented with the exact same "reparse points" as directory junctions.
     
  • File hard link - This is purely a file system construct. Because of the way directory indexes work in NTFS, it is possible to add a file entry to a directory index of a file that already exists under some other directory. Essentially, you can think of the file as being in 2 (or more) places at once. While this is not quantum physics, it is NTFS. Each file has a "reference count" and that count is incremented whenever a hard link is created to it. When the count reaches 0, the file is deleted.

    No other kernel subsystem is involved and no "reparse points" exists. This is the cleanest and purest way of making a file appear in 2 places at once (IMO).

Wow, and all this works together reliably?

 

Yes, and that's what StableBit DrivePool is trying to preserve. You see, right now the only thing that we support on the pool from the above list are shortcuts. Everything else is not supported.

 

Some people have been requesting the support of file / directory symbolic links and junctions. Those 2 can be used by software in order to create complex directory structures, in order to organize your data better.

 

4 out of the 5 unsupported technologies use "reparse points", so it makes sense for StableBit DrivePool to implement support for them.

 

Ok, so what's a "reparse point"?

 

A reparse point is a Microsoft defined data structure that gets associated with a file or a directory. When that file or directory has a reparse point associated with it, then it becomes a kind of link to "somewhere else".

 

Essentially, when a file system encounters a reparse point, it tells the I/O Manager "these aren't the droids you're looking for, go look here". The I/O Manager is responsible for opening files, so it happily obliges.

 

That doesn't sound too complicated

 

Well, it isn't, except that there are different types of "reparse points" and each reparse point has a different meaning of where to go next.

 

For example:

  • File / directory symbolic links use a "symlink" reparse point.
  • Directory junction points / mount points use a "mount point" reparse point.
  • Any 3rd party developers can develop their own type of reparse points and their own logic as to how they work. Remember drive extender from WHS v1? Yep, those tombstones were yet another kind of reparse points.

Ok, so this is complicated. But will StableBit DrivePool support reparse points?

 

I'm working hard towards that goal, and the reason why I'm writing this is because I believe that I've finally cracked the architecture that we need to support all Microsoft and 3rd party reparse points on the pool.

 

The architecture has these positive aspects to it:

  • It supports file / directory symbolic links, directory junction points, mount points, and 3rd party reparse points on the pool.
     
  • It is a 100% native kernel implementation, with no dependence on the user mode service.
     
  • It follows the 0 local metadata approach of storing everything needed on the pool itself and does not rely on something like the registry. This means that your reparse points will work when moving pools between machines (provided that you didn't link to something off of the pool that no longer exists on the new machine).

Some of my previous attempts had these limitations:

  • Requires the user mode service to run additional periodic maintenance tasks on the pool.
     
  • No support for directory reparse points, only file ones.
     
  • Adding a drive to the pool would require a somewhat lengthy reparse point pass.

The new architecture that I came up with has none of these limitations. All it requires is NTFS and Windows.

 

When will it be ready?

 

I'd hate to predict, but I think that it should be deployed in BETA form in a few weeks.


  • Christopher (Drashna), Shane and Tardas-Zib like this

#2 Sontera

Sontera

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 22 November 2013 - 04:41 PM

Very interesting! I can't wait to see the beta and hopefully I can start encrypting my data with truecrypt again soon... :)



#3 gregcaulder

gregcaulder

    Member

  • Members
  • PipPip
  • 24 posts

Posted 27 November 2013 - 08:57 PM

Let me know when you have a suitable beta, and I would more than happy to test it out for you.



#4 Alex

Alex

    Lead Programmer

  • Administrators
  • 242 posts
  • LocationNew York, USA

Posted 27 November 2013 - 11:41 PM

The next BETA of StableBit DrivePool with reparse point support is here!

 

Download: http://stablebit.com...vePool/Download

 

Read about it: http://blog.covecube...reparse-points/



#5 gregcaulder

gregcaulder

    Member

  • Members
  • PipPip
  • 24 posts

Posted 02 December 2013 - 09:33 AM

The next BETA of StableBit DrivePool with reparse point support is here!

 

Download: http://stablebit.com...vePool/Download

 

Read about it: http://blog.covecube...reparse-points/

Excellent, now that this beta has been released, should a new thread to this specific beta release be created in to forum? I already have questions, but don't really need to keep adding it to this chain.



#6 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 7,628 posts
  • LocationSan Diego, CA, USA

Posted 02 December 2013 - 07:51 PM

Greg, feel free to start a new thread and ask away.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#7 pyrotechnomimus

pyrotechnomimus

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 10 December 2014 - 06:01 PM

Does this work with the smart files of Onedrive?



#8 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 7,628 posts
  • LocationSan Diego, CA, USA

Posted 06 January 2015 - 12:43 AM

Does this work with the smart files of Onedrive?

Sorry for not getting to this sooner.

 

No, it doesn't. At least not on Windows 8.

We are looking into the issue, though.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#9 thorrrr

thorrrr

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 19 March 2016 - 04:50 PM

Hi Alex / Chris

 

Have we moved forward with a solution  on using DP with symbolic links?

 

I have the same issue many have with Plex filling up my SSD C drive. 

 

Has anybody found an easy solution i have tried copying it to the pool and doing a registry tweak to point PMS but it does not work.



#10 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 7,628 posts
  • LocationSan Diego, CA, USA

Posted 20 March 2016 - 04:16 AM

Hi Alex / Chris

 

Have we moved forward with a solution  on using DP with symbolic links?

 

I have the same issue many have with Plex filling up my SSD C drive. 

 

Has anybody found an easy solution i have tried copying it to the pool and doing a registry tweak to point PMS but it does not work.

 

Symbolic links just fine on the Pool.

 

However, Plex uses hard links specifically, which are not supported on the pool, and unfortunately, may never be. Due to their nature.

 

The only solution is to not put the plex database on the pool directly, or unless plex changes things and uses a reasonable database storage method instead of crazy backwards way they're doing it. 

 

That said, you can put the library/database on a pooled disk, outside of the pool, and that would be fine. Though, it may be better to use a dedicated SSD in this case. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#11 cryodream

cryodream

    Member

  • Members
  • PipPip
  • 26 posts

Posted 17 September 2016 - 09:56 PM

Just to make sure about hard links.

All the drives in my server, that belong to the pool - they are not mounted to drive letters. I have them mounted to folders in D:\Drives

So, how would DrivePool react/handle these 2 situations, below. Bare in mind, I would create the hardlinks manually and I would not do it through the pool, but through the individual mountpoints in D:\Drives. Which I guess, should work, as I would be on the same drive?

  1. hardlinks from files inside the pool to inside the pool, eg........:   D:\Drives\drive01\PoolPart.xxxxx\Sports\F1\original.mkv  -->  D:\Drives\drive01\PoolPart.xxxxx\Sports\Whatever\linked.mkv
  2. hardlinks from files inside the pool to outside of the pool, eg..:   D:\Drives\drive01\PoolPart.xxxxx\Sports\F1\original.mkv  -->  D:\Drives\drive01\Whatever\linked.mkv

I guess in the first scenario, there may be trouble with duplication, and/or used/free space calculations?

Although in the second scenario, I can't find the reason, why it should not work without problems?

Chris, some wisdom, please :)

 

I've been trying to live without hardlinks for over the year now, since I started using DrivePool - and it sucks. TBH, if drivepool does not support hardlinks at all (even in any of these two scenarios), I'm afraid it's gonna be a deal breaker for me, sooner rather than later...

 

Thanks in advance for any help.



#12 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 7,628 posts
  • LocationSan Diego, CA, USA

Posted 21 September 2016 - 05:44 PM

Honestly, I'm not sure, I'd have to test it out to give you a proper answer.

 

Either way, chances are, t hat either will only count a small portion of the file as "pooled" and the rest would be considered "other" data.


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#13 Erik Wright

Erik Wright

    Newbie

  • Members
  • Pip
  • 7 posts
  • LocationNanaimo, BC

Posted 28 September 2016 - 03:05 AM

Just to make sure about hard links.

All the drives in my server, that belong to the pool - they are not mounted to drive letters. I have them mounted to folders in D:\Drives

So, how would DrivePool react/handle these 2 situations, below. Bare in mind, I would create the hardlinks manually and I would not do it through the pool, but through the individual mountpoints in D:\Drives. Which I guess, should work, as I would be on the same drive?

  1. hardlinks from files inside the pool to inside the pool, eg........:   D:\Drives\drive01\PoolPart.xxxxx\Sports\F1\original.mkv  -->  D:\Drives\drive01\PoolPart.xxxxx\Sports\Whatever\linked.mkv
  2. hardlinks from files inside the pool to outside of the pool, eg..:   D:\Drives\drive01\PoolPart.xxxxx\Sports\F1\original.mkv  -->  D:\Drives\drive01\Whatever\linked.mkv

I guess in the first scenario, there may be trouble with duplication, and/or used/free space calculations?

Although in the second scenario, I can't find the reason, why it should not work without problems?

Chris, some wisdom, please :)

 

I've been trying to live without hardlinks for over the year now, since I started using DrivePool - and it sucks. TBH, if drivepool does not support hardlinks at all (even in any of these two scenarios), I'm afraid it's gonna be a deal breaker for me, sooner rather than later...

 

Thanks in advance for any help.

 

I currently have 2 hardlinks between my Dropbox folder, located on the pool, and AppData folder, located on my Windows Boot SSD, not part of any pools.

The purpose is to keep a games' data folder (in AppData) syncronized with DropBox (stored on one of my pools) as it contains recording files for said game.

I also created a couple of hardlinks from folders within a pool for the purposes of testing that functionality as well.

 

I have not yet noticed any issues or oddities with this setup, nor have I noticed duplicated duplicate problems either.

Bear in mind that I have my pools mapped as drive letters, so that may affect your results.

 

Edited Update: I tried mounting my pooled drives in C:\_DISKS\d03_3TB-SERIALNO\PoolPart.PARTNO and C:\_DISKS\d04_3TB-BKP-SERIALNO\PoolPart.PARTNO to test things out.

The mapped hardlinks appeared to work just fine, both updating between folders on the pool, and from the AppData folder on my Windows Boot SSD.






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users