Alex
-
Posts
253 -
Joined
-
Last visited
-
Days Won
49
Posts posted by Alex
-
-
Ok, so reparse points have definitely been driving me nuts lately. I was planning on releasing the StableBit DrivePool 2.1 BETA about a week after the 2.0 final with reparse point support, but it's still not out and it's because of reparse points. I've been trying different architectures over the past few weeks and none of them panned out as expected.
But today, I believe that I've finally got something that will work. It will support all of the various kinds of reparse points, it will be super fast and stable.
So what are these "reparse points" you may be asking?
Well, reparse points are the underlying file system technology that enable a whole suite of functionality. Mainly they are used for creating symbolic links, junctions and mount points.
Essentially it's a way to redirect access from one file or folder to another. You may be wondering if I'm talking about a Shortcut? No, confusingly a shortcut is not a reparse point.
So how many ways does Windows have to redirect files / folders?
A lot. That's the problem!
Here they are off the top of my head:
-
A shortcut - A special file that is parsed by the Explorer shell that really links to another file somewhere else (as in a Start menu shortcut).
Most people are probably familiar with this because it's readily available in the Explorer UI.
-
Symbolic file link - A file that points to some other file somewhere else. Confusingly, Windows Explorer also calls these "shortcuts" in the file properties dialog.
A symbolic link can be created by the mklink utility with no options.
-
Symbolic directory link - These are relatively new, as they were introduced in Windows Vista. This is essentially a directory that points to another directory somewhere else.
These can be created by using mklink /D.
-
Directory junction point - These are very similar to "symbolic directory links", but they were available prior to Windows Vista. Again, it is essentially a directory that points to another directory somewhere else. Some people make the mistake that a junction is only capable of pointing to another directory on the same volume, and that's not the case.
These can be created by using mklink /J.
-
Mount point - Mount points allow you to designate a directory that will point to another volume. These are typically used to "mount" a number of drives as directories under some other drive letter, thus saving drive letters.
These can be created from Disk Management.
-
File hard link - Yet another way to make a file point to another file. However, this method can only be used to point a file to some other file on the same volume.
These are created using mklink /H.
Yes, that's a lot of ways that you can redirect files / folders in Windows. Try Googling these and you can see the confusion that ensues as to what the differences are between each.
So what is the difference between all of these?
Well, instead of pointing out the pros and cons, I'll tell you how each one of them works under the hood and you can decide for yourself:
-
A shortcut - This is the most "user friendly" way of creating a file that points to another one. Even the name makes sense, "shortcut", imagine that. It's readily available from the Windows Explorer menus and works entirely in user mode. A special .lnk file is created that the user mode shell knows how to parse. In Windows Explorer, an icon with a little arrow is shown to you to let you know that this is really a shortcut.
However, as far as the kernel and file system are concerned, there is nothing special about the .lnk file, it's just a regular file.
-
Symbolic file link - Sometimes called a "symlink" or "soft link", this is a system that redirects one file to another, purely in the kernel. It involves some special metadata that is stored with the "source link" file that points to the "target destination file" and requires coordination between the file system and the Windows I/O Manager.
This system uses what are called "reparse points".
-
Symbolic directory link - This is exactly the same thing as a symbolic file link, but it works on directories. The reason why I separated the two is because symbolic directory links were not available prior to Windows Vista and they must be created differently.
However, the underlying technology that enables this is exactly the same. This too uses "reparse points".
-
Directory junction point - This is similar to a Symbolic directory link except that it is available prior to Windows Vista and uses an older technique. Technically speaking, the main difference between this and symbolic directory links is that directory junction points always point to an absolute path, while symbolic directory links can point to relative or absolute paths.
Surprisingly, this too uses "reparse points", but not all reparse points are the same. I'll get to that soon.
-
Mount point - These are implemented in the exact same way as directory junction points, except that they point to the root of some other volume instead of some other directory.
These are implemented with the exact same "reparse points" as directory junctions.
-
File hard link - This is purely a file system construct. Because of the way directory indexes work in NTFS, it is possible to add a file entry to a directory index of a file that already exists under some other directory. Essentially, you can think of the file as being in 2 (or more) places at once. While this is not quantum physics, it is NTFS. Each file has a "reference count" and that count is incremented whenever a hard link is created to it. When the count reaches 0, the file is deleted.
No other kernel subsystem is involved and no "reparse points" exists. This is the cleanest and purest way of making a file appear in 2 places at once (IMO).
Wow, and all this works together reliably?
Yes, and that's what StableBit DrivePool is trying to preserve. You see, right now the only thing that we support on the pool from the above list are shortcuts. Everything else is not supported.
Some people have been requesting the support of file / directory symbolic links and junctions. Those 2 can be used by software in order to create complex directory structures, in order to organize your data better.
4 out of the 5 unsupported technologies use "reparse points", so it makes sense for StableBit DrivePool to implement support for them.
Ok, so what's a "reparse point"?
A reparse point is a Microsoft defined data structure that gets associated with a file or a directory. When that file or directory has a reparse point associated with it, then it becomes a kind of link to "somewhere else".
Essentially, when a file system encounters a reparse point, it tells the I/O Manager "these aren't the droids you're looking for, go look here". The I/O Manager is responsible for opening files, so it happily obliges.
That doesn't sound too complicated
Well, it isn't, except that there are different types of "reparse points" and each reparse point has a different meaning of where to go next.
For example:
- File / directory symbolic links use a "symlink" reparse point.
- Directory junction points / mount points use a "mount point" reparse point.
- Any 3rd party developers can develop their own type of reparse points and their own logic as to how they work. Remember drive extender from WHS v1? Yep, those tombstones were yet another kind of reparse points.
Ok, so this is complicated. But will StableBit DrivePool support reparse points?
I'm working hard towards that goal, and the reason why I'm writing this is because I believe that I've finally cracked the architecture that we need to support all Microsoft and 3rd party reparse points on the pool.
The architecture has these positive aspects to it:
- It supports file / directory symbolic links, directory junction points, mount points, and 3rd party reparse points on the pool.
- It is a 100% native kernel implementation, with no dependence on the user mode service.
- It follows the 0 local metadata approach of storing everything needed on the pool itself and does not rely on something like the registry. This means that your reparse points will work when moving pools between machines (provided that you didn't link to something off of the pool that no longer exists on the new machine).
Some of my previous attempts had these limitations:
- Requires the user mode service to run additional periodic maintenance tasks on the pool.
- No support for directory reparse points, only file ones.
- Adding a drive to the pool would require a somewhat lengthy reparse point pass.
The new architecture that I came up with has none of these limitations. All it requires is NTFS and Windows.
When will it be ready?
I'd hate to predict, but I think that it should be deployed in BETA form in a few weeks.
-
A shortcut - A special file that is parsed by the Explorer shell that really links to another file somewhere else (as in a Start menu shortcut).
-
Today we've had a BitFlock disruption. BitFlock.com was down for about 6 hours becasue of a domain name issues. All access is now restored and you should not experience any further issues accessing bitflock.com. The problem was at Google.
-
Ok, so after I posted that I realized that I didn't explain at all what I meant in terms of MBits.
- Most people are aware of Megabytes and Gigabytes (because that's what hard drive space is measured in).
- I was talking about network bandwidth. Network bandwidth is typically measured in Bits. 1 byte has 8 bits. So in order to convert bits to bytes you need to divide by 8.
But the math aside, MoCA is "up to" 100 Mbits /s
Which means that it can transfer "up to" 12.5 MB/s. Twelve point five mega bytes per second. Yes, megabytes the same stuff that's on your hard drive.
-
is that 100Mb or 100MB?
Mega BITSI know, most people find it very confusing.
-
I use MoCA myself. It gets close to 100 Mb for me. If you don't need anything faster then it's a great solution.
-
Kihim,
Thanks for uploading the dumps. I've received dumps from 2 people with the same symptoms of stuck on "Initializing", but the issues do not appear to be related to each other. I've made some changes to the code based on both dumps to try and prevent both issues.
You can get the latest internal BETAs from here (currently at 2957):
http://dl.covecube.com/ScannerWhs2/beta/download/
http://dl.covecube.com/ScannerWindows/beta/download/
I haven't published these yet to stablebit.com because I'm collecting feedback as to whether the fix is effective.
Thanks,
Edit: Actually the .wssx of 2957 is still building, it will be up in a few minutes.
-
The StableBit Scanner already does a "bus" performance test in order to determine the estimated maximum throughput of the bus that each disk controller is capable of. While we don't do an automatic sequential read / write speed test, I suppose that those could be added.
I'll think about how this can be implemented.
The upcoming per-folder balancing feature in StableBit DrivePool should let you designate which disks to use for each folder, but that would be a manual task and I understand that what you're suggesting would be completely automatic and not really the same thing.
-
Future Developments for StableBit DrivePool
The current development status for all of our products can now be found here:
http://wiki.covecube.com/Development_Status -
Future Developments for the StableBit Scanner
The current development status for all of our products can now be found here:
-
Future Developments for BitFlock.com
The current development status for all of our products can now be found here:
http://wiki.covecube.com/Development_Status -
I've been thinking about how I can better communicate what is in store for each of our products, there are 3 now and another one in the works. Starting today I'll be setting up topics in each forum that I'll be updating on a regular basis. Each post will maintain what the future holds for each product.
I try to keep a lot of the development driven by user feedback, but most of that feedback doesn't happen in the public forum (but usually in tech support tickets). I'd just like to take this opportunity to highlight the direction that each product is heading in, a kind of roadmap.
I'll be setting up those posts today so look for them popping up soon in each respective forum.
- gringott, dbailey75, Tardas-Zib and 3 others
-
6
-
Just thought I'd chime in to say I have one IDE drive that Scanner has never checked the file system on, even if you mark all sectors unchecked and rescan. Sure, I can use chkdsk, but the Scanner feature should do it. But I do love Scanner.
Yeah, there's something wonky going on here. It should be checking all supported file systems I'm looking into it.
-
Dough,
The first problem that I can see with this is that you may not have enough disk space on the other disks to accomplish this. Filling a disk completely to capacity can cause problems if files need to expand in size. That's why we have the option "Try not to fill a drive above...".
Let me explain what should be happening, based on your screenshots:
- The Disk Space Equalizer runs and redistributes your files equally across all disks. Not considering any other factors.
- The StableBit Scanner balancer runs and does nothing (since you have no damaged disks).
- The File Placement Limiter runs and sees that you've requested to:
- Move all unduplicated files out of drive N:\
- But never fill the other drives above 85%
Step #3 now sees that the other drives are all above 85% full and tries to move enough data back onto N:\ to satisfy condition #3.2
Based on that, is some step not working here as described, or are you trying to accomplish something else?
-
Just to followup with some more info, I've tested this on Windows 8 with UAC enabled / disabled / completely disabled (via. registry hack) and have tried it with IE and Chrome and I can't reproduce this. It works correctly every time for me.
The error seems pretty clear the "Class", being HTTPS is not registered in the registry. So it can't find the app to open HTTPS links (Chrome in your case).
I was thinking that there might be some combination of browser / UAC settings that would cause this to manifest, but that doesn't seem to be the case. i can't think of another explanation other than the class is really not registered, there is a registry corruption or perhaps a misconfiguration.
Not sure if this is relevant, but here are some people talking about "Class not registered" on Windows 8 with Chrome:
http://productforums.google.com/forum/#!msg/chrome/jxHvPMKVdKU/MpORZ-lEyIoJ
Looks like a bunch of people have something similar, but this was with Chrome 22 which is kind of dated by now.
-
I use this well known method to open web links from the BitFlock client:
http://support.microsoft.com/kb/305703
The error that I'm seeing from your system is:
[system]System.ComponentModel.Win32Exception: Class not registered
The Microsoft article does state that:
"This code is highly dependent on the application-file type associations in the HKEY_CLASSES_ROOT hive of the registry. This can lead to unexpected results and exceptions if the registry is damaged."
So that may be one cause.
However, I do see that over here:
http://stackoverflow.com/questions/502199/how-to-open-a-web-page-from-my-application
There's a comment to suggest that this has something to do with Windows 8 and some suggestions that it may have something to do with UAC. I just tried it on a fairly clean Windows 8 machine and it seems to be working as intended.
-
Announcing the StableBit Tweet to Win Giveaway!
Every week we'll be giving away copies of StableBit DrivePool and the StableBit Scanner to some lucky winners who enter the contest, no strings attached. I'll be posting more about this on the Blog and on StableBit.com, but for now here are the details.
How to Enter
To enter, you simply tweet:-
I'd like to win the #StableBit Scanner from http://StableBit.com @Covecube
-
I'd like to win #StableBit DrivePool from http://StableBit.com @Covecube
And that's it, it's that simple. The contest has already started for this week, so you can enter today.
You'll get a confirmation tweet back within a few minutes acknowledging your entry. You can enter once for each product, and if you don't win you can enter again next week.
How it Works
At the beginning of every Sunday, a secure cryptographic random number generator will pick 2 winners. The winners will get a tweet with a URL to claim their prize. When you click on the URL you can claim the prize as yourself or easily give it as a gift to someone else.
To claim a prize you'll need to enter your first and last name, and your email address. We'll send the activation details to the email that you provide.
What's the Catch?
Well, I've been thinking about how we can raise product awareness and at the same time give something back to our users and this is one of the ideas that I came up with.
The catch is that you're advertising our product when you tweet to enter the contest and that's really it. We're not collecting your emails or twitter handles for some mass marketing campaign or anything of that sort.
Even if you own one of our products and wish to support us, you can still enter the contest. We would definitely appreciate that! If you win, you will have the option to give your prize to someone else as a gift.Thanks for tweeting.
Edit: The current chances of winning are 1 in 20. The more people enter, the more people win.
-
I'd like to win the #StableBit Scanner from http://StableBit.com @Covecube
-
Ok, I got your error reports.
It seems like "https://" is not registered on your system, as if you have no default web browser installed. So in that case, just typing in the URL that's on the screen in any web browser's address bar should work.
-
FYI. This WD green hard drive I was using is a 2TB WD20EARS.
Just speaking from personal experience here,
The EARS drives were very flaky and I would not hold that against WD because they were one of the first (if not the first) drives to implement 4K sectors. But I still have a couple of these chugging along with the Scanner keeping a close watch on them.
As Drashna said, the WD RED drives are a far better solution today.
-
And Syncback Pro looks like a good backup solution. I'll definitely have to take a look into it.
I just have to mention that, personally, I've used SyncBack SE (not Pro) for years. It's a great piece of software and reasonably priced too.
-
Ok, you should find error reports in:
C:\ProgramData\BitFlock.com\ErrorReports
Please ZIP them up and attach them here and I'll take a look at why it's crashing.
Thanks,
-
Ok, I've updated the BitFlock client to write out the error reports, write out an event log entry and show a more informative error message with a link to this forum.
In addition, it is now compiled against our latest disk pass-through code. Give the new build a try and let me know if the problem continues (v. 1.0.0.545).
-
Saiyan,
Based on your StableBit Scanner screen, the problem is not the RAW value, but rather that the value has dropped below the threshold on a pre-fail type attribute.
But that screen just looks wrong. WD Green Drives should be publishing a lot more SMART attributes. So yes, it could be a false positive. But then again I do see 16 reallocation even counts.
Can you please PM me your BitFlock submission ID at the bottom of the SMART Window so that I could take a look at the raw data? Also, we can set up a remote support appointment if you'd like and I'll log in and take a look at what's going on there.
Thanks,
-
Hmm... so it seems like one partition somehow got into a state where it was not being rechecked, and yet the alert viewer was reporting that as an error.
I'll review the file system scan code to see if I could spot the cause of something like that.
-
Just to confirm,
File History doesn't seem to be working when backing up to the pool.
I've added this as a known issue and I'll see if we can make this work in the future.
Websites down?
in General
Posted
Yes, absolutely temporary.
The stablebit.com server is running on a dedicated Windows Azure box with an automatic DNS failover to another shared hosting account. Today, both failed for about a total of 4 hours. I'm not sure why the Windows Azure services failed, but restarting the servers fixed the problem. The backup host is experiencing a DDOS attack at this time and is still not working.
stablebit.com is back up but bitflock.com is still down. I'm monitoring the situation closely.
Sorry about the inconvenience.