Symbolic Link Support

[JazzMan]

Just had a project where it would have been nice if the DrivePool volume supprted the creation of symbolic links (mklink).  I did what I needed to by having the link targets reference the individual drives that made up the pool, but having direct support for this within DrivePool would have been a plus.  Not sure how hard that would be to add or what issues there might be. 

[Pål Andreassen]

Missing support for sumbolic links are also preventing use DrivePool to host Plex Media Server. It uses symbolic links a lot for it's file based meta-database. Totally crashes when run off the pool. I can still have my media in the pool, but PMS and it's meta database has to go on a non-pooled drive. I expanded the C:-partition to make room for Plex since I don't have any non-pooled drives except the server backup drive.

[Christopher (Drashna)]

@JazzMan:  I've flagged this thread for Alex, so he can weigh in. But I'm pretty sure it's not possible, given how DrivePool work. It's a virtual drive that just passes the IO operations to the actually drives basically. (this is a vast oversimplification of how it works, though).

 

@Pal:

For Plex, do you mean it's local database/cache? 

I've used plex before with DrivePool without issue. But I have manually scrapped the data (using XBMC metadata, and a 3rd party tool), and had it installed to the default location (C:\Program Files)

[Pål Andreassen]

@Pal:

For Plex, do you mean it's local database/cache? 

I've used plex before with DrivePool without issue. But I have manually scrapped the data (using XBMC metadata, and a 3rd party tool), and had it installed to the default location (C:\Program Files)

 

Yes, it's local DB/cache uses symbolic links. I quess to link related data. The database is file-based and an actor picture is stored once but linked to from all the movies he/she is in. Works fine in the default (Program Files) location as long as I extended the partition. The default 20GB C: size was to small for me.

[JazzMan]

>> It's a virtual drive that just passes the IO operations to the actually drives basically. (this is a vast oversimplification of how it works, though).

 

With this oversimplification, it would seem possible then.  Symlinks are just another IO operaion that can be "passed to the drives".  Although, it seems like other features of NTFS are not supported; for example it looks like EFS (Encrypting File System) files are also not supported via DrivePool?  I wonder if streams are?  I guess this is all based on which NTFS features you choose to pass through the CoveFS layer and which you do not.

 

[Christopher (Drashna)]

The drivePool disk isn't a "real" drive. It's a Virtual drive, and windows itself has some rather strict requirements (and they tend to be undocumented too) about what features will work on what drives (API, file system features, etc).  For example: NFS. Both Alex and the Drive Bender team had issues getting the built in NFS Server to work on WHS2011. And Server 2012 changed something and now the built in NFS server doesn't want to work with the same settings that did on WHS2011...

 

But for the Symbolic Links... how would you create them? Where would they point to? I understand that it sounds like there are easy answers, but everything must be done within the confines of Windows API. Which while it may be very robust, it isn't exactly well documented. Nor is it as "fully featured" as many would like.

 

 

Either way, I've let Alex know. And if it's possible, and feasible... 

[Alex]

Guys, I've added preliminary support for symbolic links (reparse points). It still needs more testing, but here is the internal build with this feature:

• http://dl.covecube.com/DrivePoolWindows/beta/download/StableBit.DrivePool_2.0.0.281_x64_BETA.exe
   
• http://dl.covecube.com/DrivePoolWindows/beta/download/StableBit.DrivePool_2.0.0.281_x86_BETA.exe

[JazzMan]

I installed the 2.0.0.281_x64_BETA above on Win7.  I was able to successfully create a symlink.  However, if I try to access the files via that link I don't see any files.  If I look at the PoolPart directory the link appears to be created correctly and it does resolve to the target location.  Also; in Win Explorer, if I do a properties of the PoolPart object it shows a "Shortcut" tab, but the properties of the CoveFS object don't show a Shortcut tab.  The CoveFS object icon does show the shortcut arrow thingy though.

 

 

C:\Users\JazzMan>d:

D:\>cd pictures

D:\Pictures>dir "d:\sorted\pic jpg0" /b/a-d | find /v /c "::"
100000

D:\Pictures>mklink /d "pic jpg0" "d:\sorted\pic jpg0"
symbolic link created for pic jpg0 <<===>> d:\sorted\pic jpg0

D:\Pictures>dir "d:\pictures\pic jpg0" /b/a-d | find /v /c "::"
File Not Found
0

D:\Pictures>dir "U:\PoolPart.e142bb7a-bef5-4415-bf1a-f47a426a45c2\Pictures\pic jpg0" /b/a-d | find /v /c "::"
100000
 

[Alex]

Thanks for testing it out.

 

My initial implementation in build 281 above was based on the premise that we can reuse the reparse functionality that's already present in NTFS.

 

I've been reading up some more on exactly how this is supposed to work and playing around with some different approaches, it looks like the entire concept of reusing NTFS for this is not going to work.

 

So don't use build 281

 

I'm going to take the current reparse implementation out and rewrite it from scratch using a different approach.

 

Using this new approach, reparse points (or symbolic links) will appear as regular files or directories on the underlying NTFS volume, but will work like reparse points on the pool. This will also eliminate the burden of accounting for reparse points when duplicating or rebalancing, since they will be regular files on the underlying NTFS volume.

 

Look for that in a future build. I don't think that it will make it into the next build because that one is concentrating on updating the locking and caching model, which is a big change as it is.

[JazzMan]

OK, thanks for trying.  I don't know how/if your competition (StorageSpaces, DriveBender) handles symlinks.  Sounds like it is harder than I thought it wuld be.  No worries.

[Christopher (Drashna)]

Storage Spaces should definitely handle it. It's basically conventional RAID and functions as such.  

As for Drive Bender, I'm not sure, but I don't think they do either.

 

 

And yeah, everything with file systems seems to end up more complicated that it would seem.  I'm sure Alex has horror stories along those lines.

But like he said, it's on his to-do list.

[Alex]

JazzMan,

 

Simply put, it is much harder than I initially thought to make a reliable implementation of symbolic links.

 

As far as I can tell, we can make a reliable implementation in the future, it's just going to take some time.

 

Look for this in a future build.

 

Regards,