-
Posts
990 -
Joined
-
Last visited
-
Days Won
98
Reputation Activity
-
Shane reacted to Alex in Questions regarding hard drive spindown/standby
This is actually a fairly complicated topic.
Let's start by talking about how normal standby works without the StableBit Scanner getting involved.
Windows "Put the Disk to Sleep" Feature
Normally, Windows will monitor the disk for activity and if there is no disk activity for some preset amount of time it will put the disk to "sleep" by flushing all of the data in the cache to the disk and sending a special standby command to it. At the same time, it will remember that the disk is asleep in case any other application asks.
Shortly after the disk goes to sleep, the StableBit Scanner will indicate the fact that the disk is asleep in the Power column. Normally, the Scanner gets the power status of the disk by querying Windows and not the disk.
It does not query the disk directly for the power state because Windows considers this power query disk activity and wakes up the disk as a result.
Now, things get a bit more complicated if you want to include the on-disk power management in this picture.
Disks can optionally support these features, which can put them to sleep without Windows knowing it:
Advanced power management. Standby timer Advanced Power Management
This is a technology that implements power consumption profiles. For instance, if you don't care about performance but want maximum power savings, then you can tell your disk just that. Simply set the Advanced Power Management to Minimum Power Consumption. Or you can do the exact opposite by setting it to Maximum Performance (which guarantees no standby).
With Advanced Power Management you don't concern yourself with "sleep timeouts", like in Windows. You simply state your intent and the disk will adjust various parameters, including the standby time, according to your setting.
The implementation of Advanced Power Management is completely up to the manufacturer of the drive, and there are no specifications that explicitly state what each power mode does. This entire feature may not even be supported, depending on the disk model.
Standby Timer
The Standby timer is more widely supported because it is an older feature. You simply specify after how much disk inactivity you would like the disk to be put to sleep. This is similar to how things work in Windows, except that the low power mode will be initiated by the disk firmware itself.
Again, the implementation of this is up to the manufacturer of the drive.
StableBit Scanner "Put into Standby"
In the StableBit Scanner, you can right click on a disk and put it into standby mode. What this does is send a power down command to the disk. This type of power down is equivalent to what Advanced Power Management or the Standby Timer would do.
More importantly, when a disk is powered down in this way, Windows will not be aware that the disk is in a low power state, and will continue to report that the disk is still powered up. This is not an issue because the disk will simply spin up the next time that Windows tries to access it.
But this leaves the StableBit Scanner with a dilemma. If we can't query the disk for the power state directly, how do we report the true power state of the disk? What the StableBit Scanner implements is a power state in which it's not sure whether the disk is in standby or active, and this is what you were seeing.
Forcing the StableBit Scanner to Query the Power Mode from the Disk
If you want to use on-disk power management exclusively, and you don't care about Windows putting your disks to sleep, you can instruct the StableBit Scanner to query the power mode directly from the disk.
When this is enabled, you will no longer see the standby or active message, but Windows will never try to put that disk to sleep. That's why this is off by default.
SMART
And just to make things even more complicated, sometimes a disk will wake up when it's queried for SMART data.
To this end the StableBit Scanner implements some more settings to deal with this:
I hope that this clears things up.
-
Shane got a reaction from Jamesperkins in Destination Folder Access Denied
Hi James. Yes, if a trial version of DrivePool expires, it does become read-only (so your files are still intact and accessible) until you re-activate it with a paid licence.
If you open the user interface for DrivePool, it should display how many days you have left in the trial or that it has expired.
-
Shane reacted to Christopher (Drashna) in Drivepool v2
Lee,
What specific features are you looking at?
And if you don't like the more advanced features of Server 2012 (R2) Essentials, there is always this:
http://www.tinkertry.com/how-to-make-windows-server-2012-r2-essentials-client-connector-install-behave-just-like-windows-home-server/
-
Shane reacted to Julie in Hot spare option?
As someone who has worked in the storage industry for many years, I thought I would take a moment and clarify what a hot spare is.
In a hardware raid system with mirroring or parity drives are required to be the same size. I'll use mirroring for the example, but parity just takes more drives and uses space more efficiently.
If drive 1 has "1234567" stored in sector 42, then drive 2 has "1234567" stored in sector 42. That is simple mirroring with 2 copies of the data.
If drive 1 fails then sector 42 can just be read from drive 2 and operation continues, however redundance/protection has been lost.
This is where the hot spare idea comes in. The only way to restore protection/redundance is add another drive. Keeping one online as a spare allows automatic replacement and the data can be immediately duplicated for protection.
Of course having 3 drives for evey two needed would be pretty expensive. Even mirroring gets pretty expensive in large storage facilities, hence the need for parity raid. So what really happens is there may be 1 hot spare online for 10 or 20 drives. It is ready to go when needed because the requirement is that duplication be restored asap.
With Drivepool a hot spare is totally unnecessary. It would just be wasting a disk that could be part of the pool and helping performance. (Remember reads and writes can be spread across disks.)
Why is it unnecessary? The need for a hot spare is to provide a place to duplicate the information. With Drivepool as long as you have more drives than the number of copies of your data (after a drive fails) then there is a place to recreate the copy. Of course this does require enough overall space.
Let's use a concrete example.
I have 3 drives in my system. All are 1TB drives. I have my entire pool duplicated 2x. If I have 2TB stored then each drive could be 66% full. (I'll touch on capacity and usage in a minute.)
If one drive fails then I still need at least 2TB across two drives to meet my duplication goal. I have two 1TB drives so I just recreate the duplicates and everything is safe again.
So, any unused storage space can be used to give me the same ability to restore protection that I get with a hot spare for hardware raid.
The formula is duplication count < number of drives
and data size * duplication count < sum(drive sizes) - size of largest drive
This means you are covered.
Drive utilization should normally be maintained less than 80%. (I alluded to this above.) The reason is that allocation for writes slows dramatically in most operating systems when the drive becomes more used. It takes more work to find free space and it tends to be in more small blocks spread about the disk. So you will likely alway have enough space available to handle a drive failure if you normally run <80% utilization and you will have faster disk access.
Using an extra disk in your pool instead of having a spare also provides a performance benefit.
With 2 drives and duplication of 2 then reading data can happen from both drives at the same time. The result is that read speed is 2x drive speed.
If I add a third drive, then I can get 3x drive speed for reads, if the read chunk is larger than the chunks used for duplication. (This is how storage spaces works.)
With Drivepool, I believe that entire files are duplicated so this benefit isn't as clear. However, in a server situation where there is a high likelyhood of reading multiple files at the same time we can achive a read speed of Nx where N is the number of drives.
-
Shane got a reaction from Tardas-Zib in Glossary - Alpha, Beta, RC, Release, Stable, Final, Version
Here's how I perceive the whole alpha/beta/version ball of string.
Alpha - the bare skeleton of a program, very crash-prone, full of bugs and lacking many planned features.
Beta - all major planned features and adding minor features, lots of minor bugs, often still some major bugs
RC (Release Candidate) - all major features, most/all minor features, only minor/unknown bugs should remain
Release - all major features, all minor features, no known bugs (except maybe a couple that are proving really hard to fix)
Stable - no known bugs, or at least the few remaining known bugs are well-documented edge-cases with workarounds
Final - can mean Stable, can mean Release, can mean last minor revision of a particular version on the roadmap
Version - a numerical way of distinguishing that the software is different from the last time it was published, often in the form of a date (e.g. ymd 20120616), integer (e.g. build# 1234) or revision (e.g. major.minor 2.3) or combination (e.g. ymd.build or revision.build or y.major.minor.build)
Roadmap - a list of planned features sorted by planned version, at least in theory.
For a hypothetical example, Fooware 2014 build 5.4 Beta might be a program published in the year 2014, is the 5th major version of that program, which in turn has seen 4 minor revisions (added another minor feature and/or fixed some bugs since the 3rd minor revision) that is still in Beta (has a tendency to crash for at least six different reasons along with numerous edge case problems).
To further confuse things, Alpha/Beta/etc can refer to a particular version of a program, a range of versions of a program, or be a separate tag independent of the version numbering scheme, depending on the preference of the person(s) doing the naming. For example, if you see a roadmap with Fooware 2014.5.3 Stable followed by Fooware 2014.5.4 Beta, this is likely to mean that a new minor feature has been added that may/has introduced some new bugs as well.
-
Shane got a reaction from champs777 in Cloud Backup/Restore Preperation Help
Yes, if you don't use the pool late at night, that would be a good time to have CrashPlan do the backups.
As drashna notes, if automatic balancing is on but not immediate, it defaults to 1am - but since in our scenario we're not using automatic balancing that's not a concern (the idea is that we want the pool to just do the bare minimum of balancing, and do so immediately as required, so that CrashPlan is not wasting bandwidth on files that have been moved between drives - though things might get tricky if we get close to filling all of the drives).
-
Shane got a reaction from Christopher (Drashna) in Duplicate over all disks
Q: Hi, Is it possible to set DrivePool to duplicate a folder to all disks available in the pool regardless of their number?
A: You would have to set that folder's duplication level so that it was equal to the number of disks in the pool. For example, if your pool consisted of ten disks, you would set that folder's duplication level (number of copies) to 10.
Q: And another question - is it possible to set DrivePool that way, that it keeps one copy of a folder on one specific disk? For example I have 3 disks and I want one folder to be duplicated twice and one copy I need on one specific disk and the other can be on any of that two remaining disks.
A: Not yet. It's a planned feature. http://community.covecube.com/index.php?/topic/208-stablebit-drivepool-controlling-folder-placement/
-
Shane got a reaction from champs777 in Cloud Backup/Restore Preperation Help
If our intent is to backup the entire pool via CrashPlan, and if we are NOT using duplication, then I might go with something like:
1. Set DrivePool to "do not balance automatically" and TICK "allow balancing plug-ins to force immediate balancing".
2. Turn off the "Volume Equalization" and "Duplication Space Optimizer" balancers.
3. Turn on the "Ordered File Placement" balancer.
4. If feasible, set CrashPlan to do its backups when you are not using the pool, so that it does not read files while DP is moving them around.
5. Then tell CrashPlan to back up the hidden poolpart folders on each drive that forms the pool.
This should minimise the amount of wasted traffic. There might be more optimization we can do, too.
If we ARE using duplication, then things get messy. Frankly, it would be a lot easier if CrashPlan just implemented a "skip existing files" option. -
Shane got a reaction from cmcj in Glossary - Alpha, Beta, RC, Release, Stable, Final, Version
Here's how I perceive the whole alpha/beta/version ball of string.
Alpha - the bare skeleton of a program, very crash-prone, full of bugs and lacking many planned features.
Beta - all major planned features and adding minor features, lots of minor bugs, often still some major bugs
RC (Release Candidate) - all major features, most/all minor features, only minor/unknown bugs should remain
Release - all major features, all minor features, no known bugs (except maybe a couple that are proving really hard to fix)
Stable - no known bugs, or at least the few remaining known bugs are well-documented edge-cases with workarounds
Final - can mean Stable, can mean Release, can mean last minor revision of a particular version on the roadmap
Version - a numerical way of distinguishing that the software is different from the last time it was published, often in the form of a date (e.g. ymd 20120616), integer (e.g. build# 1234) or revision (e.g. major.minor 2.3) or combination (e.g. ymd.build or revision.build or y.major.minor.build)
Roadmap - a list of planned features sorted by planned version, at least in theory.
For a hypothetical example, Fooware 2014 build 5.4 Beta might be a program published in the year 2014, is the 5th major version of that program, which in turn has seen 4 minor revisions (added another minor feature and/or fixed some bugs since the 3rd minor revision) that is still in Beta (has a tendency to crash for at least six different reasons along with numerous edge case problems).
To further confuse things, Alpha/Beta/etc can refer to a particular version of a program, a range of versions of a program, or be a separate tag independent of the version numbering scheme, depending on the preference of the person(s) doing the naming. For example, if you see a roadmap with Fooware 2014.5.3 Stable followed by Fooware 2014.5.4 Beta, this is likely to mean that a new minor feature has been added that may/has introduced some new bugs as well.
-
Shane reacted to JazJon in 1x Zotac ZBOX ID 83 Mini-PC, x3 Mediasonic USB 3.0 Enclosures, x12 WD Red 3TB Drives, x1 Anker USB 3.0 Hub
EDIT:
I'm throwing in the towel on this whole external enclosure hardware setup.
It's time I put together a REAL server with REAL sata connections.
http://community.covecube.com/index.php?/topic/419-building-a-proper-whs-server-need-advice/page-2&do=findComment&comment=3008
Here's my new setup on the way:
http://community.covecube.com/index.php?/topic/507-my-new-lian-pc-v2120b-full-tower-home-server-setup-drivepoolscanner/
Here is my rock solid fast stable setup.
I'm running Windows 8.1 x64 on all my computers including my DrivePool server.
I get greater than 100Mbps transfers across my gigabit network!
x1
ZOTAC ZBOX ID83 [ZBOX-ID83-U] w/ Intel Core i3 3120M 2.5 GHz Dual-Core
http://www.zotacusa.com/zbox-id83.html
http://r.ebay.com/DL54aU
x1
Crucial m4 128GB 2.5-Inch (9.5mm) SATA 6Gb/s Solid State Drive CT128M4SSD2
http://www.amazon.com/Crucial-128GB-2-5-Inch-9-5mm-CT128M4SSD2/dp/B004W2JKZI/
x1
Crucial 8GB Kit (4GBx2) DDR3 1333 MT/s (PC3-10600) CL9 SODIMM 204-Pin Notebook Memory Modules CT2KIT51264BC1339
http://www.amazon.com/gp/product/B003DZJQQI/
x1
Anker® Uspeed USB 3.0 4-Port Hub with 12V 2A Power Adapter and 3ft USB 3.0 Cable [VIA VL812 Chipset]
http://www.amazon.com/Anker®-Uspeed-4-Port-Adapter-Chipset/dp/B005QWY3PU/
x3
Mediasonic HF2-SU3S2 ProBox 4 Bay Hard Drive Enclosure with USB 3.0 & eSATA
http://www.amazon.com/Mediasonic-HF2-SU3S2-ProBox-Drive-Enclosure/dp/B003X26VV4/
x3 USB 3.0 to Esata Adapter Support Port Multiplier - U3esata
http://www.amazon.com/gp/product/B005DCCMII/
x3 Monoprice 108791 6-Feet SATA 6 Gbps External Shielded Cable, eSATA to eSATA Type I to Type I, Black
http://www.amazon.com/gp/product/B009GUL1VM
x11
WD Red 3 TB NAS Hard Drive: 3.5 Inch, SATA III, 64 MB Cache - WD30EFRX
http://www.amazon.com/WD-Red-NAS-Hard-Drive/dp/B008JJLW4M
It all fits in a Ikea BESTÅ BURS Wall shelf in High Gloss Black. This is all in my bed room suspended above the TV floating on the wall.
(nice and quiet, custom vented USB powered fans)
http://www.ikea.com/us/en/catalog/products/60133931/
-
Shane reacted to Alex in StableBit DrivePool - Reparse Points
Ok, so reparse points have definitely been driving me nuts lately. I was planning on releasing the StableBit DrivePool 2.1 BETA about a week after the 2.0 final with reparse point support, but it's still not out and it's because of reparse points. I've been trying different architectures over the past few weeks and none of them panned out as expected.
But today, I believe that I've finally got something that will work. It will support all of the various kinds of reparse points, it will be super fast and stable.
So what are these "reparse points" you may be asking?
Well, reparse points are the underlying file system technology that enable a whole suite of functionality. Mainly they are used for creating symbolic links, junctions and mount points.
Essentially it's a way to redirect access from one file or folder to another. You may be wondering if I'm talking about a Shortcut? No, confusingly a shortcut is not a reparse point.
So how many ways does Windows have to redirect files / folders?
A lot. That's the problem!
Here they are off the top of my head:
A shortcut - A special file that is parsed by the Explorer shell that really links to another file somewhere else (as in a Start menu shortcut).
Most people are probably familiar with this because it's readily available in the Explorer UI.
Symbolic file link - A file that points to some other file somewhere else. Confusingly, Windows Explorer also calls these "shortcuts" in the file properties dialog.
A symbolic link can be created by the mklink utility with no options.
Symbolic directory link - These are relatively new, as they were introduced in Windows Vista. This is essentially a directory that points to another directory somewhere else.
These can be created by using mklink /D.
Directory junction point - These are very similar to "symbolic directory links", but they were available prior to Windows Vista. Again, it is essentially a directory that points to another directory somewhere else. Some people make the mistake that a junction is only capable of pointing to another directory on the same volume, and that's not the case.
These can be created by using mklink /J.
Mount point - Mount points allow you to designate a directory that will point to another volume. These are typically used to "mount" a number of drives as directories under some other drive letter, thus saving drive letters.
These can be created from Disk Management.
File hard link - Yet another way to make a file point to another file. However, this method can only be used to point a file to some other file on the same volume.
These are created using mklink /H.
Yes, that's a lot of ways that you can redirect files / folders in Windows. Try Googling these and you can see the confusion that ensues as to what the differences are between each.
So what is the difference between all of these?
Well, instead of pointing out the pros and cons, I'll tell you how each one of them works under the hood and you can decide for yourself:
A shortcut - This is the most "user friendly" way of creating a file that points to another one. Even the name makes sense, "shortcut", imagine that. It's readily available from the Windows Explorer menus and works entirely in user mode. A special .lnk file is created that the user mode shell knows how to parse. In Windows Explorer, an icon with a little arrow is shown to you to let you know that this is really a shortcut.
However, as far as the kernel and file system are concerned, there is nothing special about the .lnk file, it's just a regular file.
Symbolic file link - Sometimes called a "symlink" or "soft link", this is a system that redirects one file to another, purely in the kernel. It involves some special metadata that is stored with the "source link" file that points to the "target destination file" and requires coordination between the file system and the Windows I/O Manager.
This system uses what are called "reparse points".
Symbolic directory link - This is exactly the same thing as a symbolic file link, but it works on directories. The reason why I separated the two is because symbolic directory links were not available prior to Windows Vista and they must be created differently.
However, the underlying technology that enables this is exactly the same. This too uses "reparse points".
Directory junction point - This is similar to a Symbolic directory link except that it is available prior to Windows Vista and uses an older technique. Technically speaking, the main difference between this and symbolic directory links is that directory junction points always point to an absolute path, while symbolic directory links can point to relative or absolute paths.
Surprisingly, this too uses "reparse points", but not all reparse points are the same. I'll get to that soon.
Mount point - These are implemented in the exact same way as directory junction points, except that they point to the root of some other volume instead of some other directory.
These are implemented with the exact same "reparse points" as directory junctions.
File hard link - This is purely a file system construct. Because of the way directory indexes work in NTFS, it is possible to add a file entry to a directory index of a file that already exists under some other directory. Essentially, you can think of the file as being in 2 (or more) places at once. While this is not quantum physics, it is NTFS. Each file has a "reference count" and that count is incremented whenever a hard link is created to it. When the count reaches 0, the file is deleted.
No other kernel subsystem is involved and no "reparse points" exists. This is the cleanest and purest way of making a file appear in 2 places at once (IMO). Wow, and all this works together reliably?
Yes, and that's what StableBit DrivePool is trying to preserve. You see, right now the only thing that we support on the pool from the above list are shortcuts. Everything else is not supported.
Some people have been requesting the support of file / directory symbolic links and junctions. Those 2 can be used by software in order to create complex directory structures, in order to organize your data better.
4 out of the 5 unsupported technologies use "reparse points", so it makes sense for StableBit DrivePool to implement support for them.
Ok, so what's a "reparse point"?
A reparse point is a Microsoft defined data structure that gets associated with a file or a directory. When that file or directory has a reparse point associated with it, then it becomes a kind of link to "somewhere else".
Essentially, when a file system encounters a reparse point, it tells the I/O Manager "these aren't the droids you're looking for, go look here". The I/O Manager is responsible for opening files, so it happily obliges.
That doesn't sound too complicated
Well, it isn't, except that there are different types of "reparse points" and each reparse point has a different meaning of where to go next.
For example:
File / directory symbolic links use a "symlink" reparse point. Directory junction points / mount points use a "mount point" reparse point. Any 3rd party developers can develop their own type of reparse points and their own logic as to how they work. Remember drive extender from WHS v1? Yep, those tombstones were yet another kind of reparse points. Ok, so this is complicated. But will StableBit DrivePool support reparse points?
I'm working hard towards that goal, and the reason why I'm writing this is because I believe that I've finally cracked the architecture that we need to support all Microsoft and 3rd party reparse points on the pool.
The architecture has these positive aspects to it:
It supports file / directory symbolic links, directory junction points, mount points, and 3rd party reparse points on the pool.
It is a 100% native kernel implementation, with no dependence on the user mode service.
It follows the 0 local metadata approach of storing everything needed on the pool itself and does not rely on something like the registry. This means that your reparse points will work when moving pools between machines (provided that you didn't link to something off of the pool that no longer exists on the new machine). Some of my previous attempts had these limitations:
Requires the user mode service to run additional periodic maintenance tasks on the pool.
No support for directory reparse points, only file ones.
Adding a drive to the pool would require a somewhat lengthy reparse point pass. The new architecture that I came up with has none of these limitations. All it requires is NTFS and Windows.
When will it be ready?
I'd hate to predict, but I think that it should be deployed in BETA form in a few weeks.
-
Shane reacted to CptKirk1 in Software I use and recommend
Have you ever tried Auslogics Disk Defrag?
I use it on a number of my machines around the house. But haven't put it on my WHS2011 machine (using DrivePool 1.x and Scanner).
-
Shane got a reaction from Christopher (Drashna) in StableBit DrivePool - Controlling Folder Placement
I'd like the ability to say "keep together all files in any|each folder of folderpath" where "any" and "each" are modifiers affecting which files to keep together.
So given an example where folderpath was p:\media\sketches\* and I had files in p:\media\sketches\2012 and p:\media\sketches\2013
if the "any" modifier was selected
then the files in sketches\2012 and sketches\2013 and any subfolders thereof would be kept all together
but if the "each" modifier was selected
and the files in sketches\2012 and any subfolders thereof would be kept together
and the files in sketches\2013 and any subfolders thereof would be kept together
yet sketches\2012 and sketches\2013 would be stored independently of each other.
Also, something along the lines of if folderpath ends in "folder\" it also applies to any files within that folder itself, while if it ends in "folder\*" it only applies to files within that folder's subfolders.
I hope this makes sense.
-
Shane got a reaction from JCMarsh in Removed drive not evacuated?
I've noticed that even when you _don't_ select Duplicate Later, copies of pooled files may remain on a removed disk. What seems to be happening is that, when removing a drive from the pool, DrivePool may encounter files that are being held open by Windows*. While it can't delete them, if it can still succeed in copying them, then it will continue to remove the drive from the pool.
So if the removal process completes without any error/warning messages about lost/uncopyable files, then DrivePool has successfully copied all files that were in that poolpart to the rest of the pool, the poolpart folder on the drive will be set visible instead of hidden, and the drive can be physically removed even if files are still "on" the drive.
*(I've noticed the Windows search/metadata engine in particular has a nasty habit of not bothering to release files straight away when it's done with them)
-
Shane got a reaction from Christopher (Drashna) in Slow Write Speeds to the Drivepool
I've found "positive pressure" filtered cases can noticeably reduce dust and salt intake; you just have to remember to clean the filters every so often to maintain the pressure ratio (and also so that the case doesn't overheat - and for this I can plug Scanner, with it's emailed and spoken overheat warnings).
-
Shane got a reaction from Christopher (Drashna) in FAQ - Parity and Duplication and DrivePool
The topic of adding RAID-style parity to DrivePool was raised several times on the old forum. I've posted this FAQ because (1) this is a new forum and (2) a new user asked about adding folder-level parity, which - to mangle a phrase - is the same fish but a different kettle. Since folks have varying levels of familiarity with parity I'm going to divide this post into three sections: (1) how parity works and the difference between parity and duplication, (2) the difference between drive-level and folder-level parity, (3) the TLDR conclusion for parity in DrivePool. I intend to update the post if anything changes or needs clarification (or if someone points out any mistakes I've made). Disclaimer: I do not work for Covecube/Stablebit. These are my own comments. You don't know me from Jack. No warranty, express or implied, in this or any other universe. Part 1: how parity works and the difference between parity and duplication Duplication is fast. Every file gets simultaneously written to multiple disks, so as long as all of those disks don't die the file is still there, and by splitting reads amongst the copies you can load files faster. But to fully protect against a given number of disks dying, you need that many times number of disks. That doesn't just add up fast, it multiplies fast. Parity relies on the ability to generate one or more "blocks" of a series of reversible checksums equal to the size of the largest protected "block" of content. If you want to protect three disks, each parity block requires its own disk as big as the biggest of those three disks. For every N parity blocks you have, any N data blocks can be recovered if they are corrupted or destroyed. Have twelve data disks and want to be protected against any two of them dying simultaneously? You'll only need two parity disks. Sounds great, right? Right. But there are tradeoffs. Whenever the content of any data block is altered, the corresponding checksums must be recalculated within the parity block, and if the content of any data block is corrupted or lost, the corresponding checksums must be combined with the remaining data blocks to rebuild the data. While duplication requires more disks, parity requires more time. In a xN duplication system, you xN your disks, for each file it simultaneously writes the same data to N disks, but so long as p<=N disks die, where 'p' depends on which disks died, you replace the bad disk(s) and keep going - all of your data is immediately available. The drawback is the geometric increase in required disks and the risk of the wrong N disks dying simultaneously (e.g. if you have x2 duplication, if two disks die simultaneously and one happens to be a disk that was storing duplicates of the first disk's files, those are gone for good). In a +N parity system, you add +N disks, for each file it writes the data to one disk and calculates the parity checksums which it then writes to N disks, but if any N disks die, you replace the bad disk(s) and wait while the computer recalculates and rebuilds the lost data - some of your data might still be available, but no data can be changed until it's finished (because parity needs to use the data on the good disks for its calculations). (sidenote: "snapshot"-style parity systems attempt to reduce the time cost by risking a reduction in the amount of recoverable data; the more dynamic your content, the more you risk not being able to recover) Part 2: the difference between drive-level and folder-level parity Drive-level parity, aside from the math and the effort of writing the software, can be straightforward enough for the end user: you dedicate N drives to parity that are as big as the biggest drive in your data array. If this sounds good to you, some folks (e.g. fellow forum moderator Saitoh183) use DrivePool and the FlexRAID parity module together for this sort of thing. It apparently works very well. (I'll note here that drive-level parity has two major implementation methods: striped and dedicated. In the dedicated method described above, parity and content are on separate disks, with the advantages of simplicity and readability and the disadvantage of increased wear on the parity disks and the risk that entails. In the striped method, each disk in the array contains both data and parity blocks; this spreads the wear evenly across all disks but makes the disks unreadable on other systems that don't have compatible parity software installed. There are ways to hybridise the two, but it's even more time and effort). Folder-level parity is... more complicated. Your parity block has to be as big as the biggest folder in your data array. Move a file into that folder, and your folder is now bigger than your parity block - oops. This is a solvable problem, but 'solvable' does not mean 'easy', sort of how building a skyscraper is not the same as building a two-storey home. For what it's worth, FlexRAID's parity module is (at the time of my writing this post) $39.95 and that's drive-level parity. Conclusion: the TLDR for parity in DrivePool
As I see it, DrivePool's "architectural imperative" is "elegant, friendly, reliable". This means not saddling the end-user with technical details or vast arrays of options. You pop in disks, tell the pool, done; a disk dies, swap it for a new one, tell the pool, done; a dead disk doesn't bring everything to a halt and size doesn't matter, done. My impression (again, I don't speak for Covecube/Stablebit) is that parity appears to be in the category of "it would be nice to have for some users but practically it'd be a whole new subsystem, so unless a rabbit gets pulled out of a hat we're not going to see it any time soon and it might end up as a separate product even then (so that folks who just want pooling don't have to pay twice+ as much)". -
Shane reacted to otispresley in How To: Get SMART data passed on from ESXI 5.1 Host
Thanks! I was not aware of this setting. By the way, you don't have to go through all that for RDM anymore. All you need to do is the following and then any disk (not USB) you plugin in thereafter will be available for RDM:
In the ESXi Console window, highlight your server Go to the Configuration tab Under Software, click Advanced Settings Click RdmFilter Uncheck the box for RdmFilter.HbaIsShared Click OK -
Shane reacted to Christopher (Drashna) in Scanner Scheduling?
Or if you want to keep very organized:
<setting name="Scanner_RunningFile" serializeAs="String"> <value>C:\ProgramData\StableBit Scanner\RunningScan.txt</value> -
Shane reacted to Mr_Smartepants in Migrating back to WHS2011, Drivepool v2 or v1.3?
Alex fixed it!
Turns out the virtual (pool) drive was starting in offline mode after restart. Bringing the drive "online" (in Disk Management) allowed SBDP to detect the pool and re-engage all the pooled drives.
Woohoo! Thanks Alex.
-
Shane reacted to Christopher (Drashna) in Can I set up duplication for specific child folders under a parent share folder?
nwtim,
DrivePool 2.x is currently able to install on WHS2011 just fine. But it lacks any really integration. It just adds a link to the UI in the Server Folders tab.
But 2.x is still in beta, but we are trying to get to a "stable" version as soon as we can.
-
Shane got a reaction from Christopher (Drashna) in Install fails on Windows 8
Protip: installing unofficial betas without the developer giving you the go-ahead is a quick way to end up in the scary end of "big red button" territory.
Try this link for DrivePool removal instructions: http://wiki.covecube.com/StableBit_DrivePool_Q8964978
Note that it is for the 1.x version, but it can be applied to the 2.x version as well (just ignore the Remote Desktop and Dashboard references).
-
Shane got a reaction from Christopher (Drashna) in Unable to Remove disk from Pool
After you unplug the disk, you should tell DrivePool to remove the missing disk from the pool. It will check for and reduplicate any files that are supposed to be duplicated.
-
Shane got a reaction from Christopher (Drashna) in Feature request - Parity support ?
Actually, parity support doesn't mandate that the user's content be unreadable on other machines; that depends on the implementation used.
The thing is that while it's possible to add even folder-level parity to JBOD-style pools in such a way as to maintain content readability on non-pool machines and still keep things simple for the end-user? Implementing it is way past non-trivial and I've no idea whether the performance costs would make it impractical. Sort of like hot fusion.
(but if any angel investors are out there with a loose seven figures to punt Covecube's way, maybe Alex would be happy to work on it for a few years)
-
Shane reacted to Henrik in Duplication count greater than expected
It's only the directory entries that occur in many duplicates and not the files within them.
Example setup:
+Pool of 3 drives.
+Folder X has a duplication count of 2x and contains 5 files. (10 files in total across 3 drives)
If the folder is then balanced across the 3 drives Folder X will exist in three copies because there will be files belonging to that folder stored on all drives
(For example drive A with 3 files, Drive B with 4 files and Drive C with 3 files)
So getting the duplication count on folders is a bit misleading. You should look at individual files instead.
Hope this clears it out
-
Shane got a reaction from kihimcarr in FAQ - Parity and Duplication and DrivePool
The topic of adding RAID-style parity to DrivePool was raised several times on the old forum. I've posted this FAQ because (1) this is a new forum and (2) a new user asked about adding folder-level parity, which - to mangle a phrase - is the same fish but a different kettle. Since folks have varying levels of familiarity with parity I'm going to divide this post into three sections: (1) how parity works and the difference between parity and duplication, (2) the difference between drive-level and folder-level parity, (3) the TLDR conclusion for parity in DrivePool. I intend to update the post if anything changes or needs clarification (or if someone points out any mistakes I've made). Disclaimer: I do not work for Covecube/Stablebit. These are my own comments. You don't know me from Jack. No warranty, express or implied, in this or any other universe. Part 1: how parity works and the difference between parity and duplication Duplication is fast. Every file gets simultaneously written to multiple disks, so as long as all of those disks don't die the file is still there, and by splitting reads amongst the copies you can load files faster. But to fully protect against a given number of disks dying, you need that many times number of disks. That doesn't just add up fast, it multiplies fast. Parity relies on the ability to generate one or more "blocks" of a series of reversible checksums equal to the size of the largest protected "block" of content. If you want to protect three disks, each parity block requires its own disk as big as the biggest of those three disks. For every N parity blocks you have, any N data blocks can be recovered if they are corrupted or destroyed. Have twelve data disks and want to be protected against any two of them dying simultaneously? You'll only need two parity disks. Sounds great, right? Right. But there are tradeoffs. Whenever the content of any data block is altered, the corresponding checksums must be recalculated within the parity block, and if the content of any data block is corrupted or lost, the corresponding checksums must be combined with the remaining data blocks to rebuild the data. While duplication requires more disks, parity requires more time. In a xN duplication system, you xN your disks, for each file it simultaneously writes the same data to N disks, but so long as p<=N disks die, where 'p' depends on which disks died, you replace the bad disk(s) and keep going - all of your data is immediately available. The drawback is the geometric increase in required disks and the risk of the wrong N disks dying simultaneously (e.g. if you have x2 duplication, if two disks die simultaneously and one happens to be a disk that was storing duplicates of the first disk's files, those are gone for good). In a +N parity system, you add +N disks, for each file it writes the data to one disk and calculates the parity checksums which it then writes to N disks, but if any N disks die, you replace the bad disk(s) and wait while the computer recalculates and rebuilds the lost data - some of your data might still be available, but no data can be changed until it's finished (because parity needs to use the data on the good disks for its calculations). (sidenote: "snapshot"-style parity systems attempt to reduce the time cost by risking a reduction in the amount of recoverable data; the more dynamic your content, the more you risk not being able to recover) Part 2: the difference between drive-level and folder-level parity Drive-level parity, aside from the math and the effort of writing the software, can be straightforward enough for the end user: you dedicate N drives to parity that are as big as the biggest drive in your data array. If this sounds good to you, some folks (e.g. fellow forum moderator Saitoh183) use DrivePool and the FlexRAID parity module together for this sort of thing. It apparently works very well. (I'll note here that drive-level parity has two major implementation methods: striped and dedicated. In the dedicated method described above, parity and content are on separate disks, with the advantages of simplicity and readability and the disadvantage of increased wear on the parity disks and the risk that entails. In the striped method, each disk in the array contains both data and parity blocks; this spreads the wear evenly across all disks but makes the disks unreadable on other systems that don't have compatible parity software installed. There are ways to hybridise the two, but it's even more time and effort). Folder-level parity is... more complicated. Your parity block has to be as big as the biggest folder in your data array. Move a file into that folder, and your folder is now bigger than your parity block - oops. This is a solvable problem, but 'solvable' does not mean 'easy', sort of how building a skyscraper is not the same as building a two-storey home. For what it's worth, FlexRAID's parity module is (at the time of my writing this post) $39.95 and that's drive-level parity. Conclusion: the TLDR for parity in DrivePool
As I see it, DrivePool's "architectural imperative" is "elegant, friendly, reliable". This means not saddling the end-user with technical details or vast arrays of options. You pop in disks, tell the pool, done; a disk dies, swap it for a new one, tell the pool, done; a dead disk doesn't bring everything to a halt and size doesn't matter, done. My impression (again, I don't speak for Covecube/Stablebit) is that parity appears to be in the category of "it would be nice to have for some users but practically it'd be a whole new subsystem, so unless a rabbit gets pulled out of a hat we're not going to see it any time soon and it might end up as a separate product even then (so that folks who just want pooling don't have to pay twice+ as much)".