Jaga
-
Posts
413 -
Joined
-
Last visited
-
Days Won
27
Posts posted by Jaga
-
-
- Are the eight 2TB storage drives internal also, or externally connected (eSATA, USB, etc)? Do they share the same controller as your 500GB internal?
- Are you using something like Plex to host them on the server, and if so do you have thumbnail generation turned on in Plex Media Server for your problematic libraries?
- Have you checked the eight Pool storage drives for SMART/Surface/File System errors, and are they all clean? Stablebit Scanner is useful for this, and has a free trial period.
- When you say the latency is "much less off a non-pooled external drive", how much less? Back to normal, or does it still exist in some amount?
Even without answering any of these questions, I would replace that 500GB internal as soon as possible. Drives are many times more likely to die after they start reallocating large numbers of sectors. And it's much easier to salvage the OS and data before they die than it is after.
The reason for the questions above however: the controller could be having issues if it's both SATA/eSATA and handling all your drives. Plex Media Server uses thumbnails off the C: (default install) location for seeking in movies, and to store other metadata. Depending on your duplication settings, the pool could be having issues if it's reading from more than one drive in the pool at once, and one or more of them have performance problems. And if you have an external drive (not in the pool) but still on the same external controller, it may be completely unrelated to Drivepool - it may be something to do with other hardware. It could still be an issue with Drivepool of course, but I'd look at lower layers first (hardware, OS, etc).
-
Yeah, the custom naming/sorting with it is useful. I was just hoping for an easy visual reference that was independent of sort/display order. Something that stuck and didn't change even if the display of the drives did.
-
Having recently added a lot of drives to my server, I'd like the ability to give them custom groupings that's indicated by background color on the main display. Locations inside and outside the server have varying heat/power/connection properties which I'd like to easily see in groups - coloring the background for them would be the easiest way to accomplish this I think. Just a simple RGB color definition/wheel.
It could also be used to denote drive pairings, RAID arrays, special drives you want to keep an eye on, dead drives, hot spares, etc.. whatever the user thought was useful. I know we already have definitions for things like Case, Number, Location, Bay and so on - but this coloring would be independent of the sort/display order.
Edit: as a sidenote related to coloring - it would be awesome to have a "dark" theme for Scanner. Something dark grey with light/white text. I often find myself blinded by the white themes we currently have access to.
-
1 hour ago, zeroibis said:
Also want to point out that with raid 50% of the drives could fail and you lose nothing just like it is possible for 50% of the drives to fail in the duplication pool as seen in attached image.
-snip-
Just to verify if I can pull the drive data in a program like CrystalDiskInfo than Stablebit Scanner will work? Because if I create a software raid 1 in diskmanagement I am able to see the underline drives in CrystalDiskInfo and get smart data. I assume the same would be true if I was to create a raid 0 in diskmanagement instead.
Yeah, I mentioned that in one of my edits - where it has Note: near the end. It'd be very rare, though possible.
Scanner has a free trial period - since you have the software RAID 0 in place, you could install and test. You wouldn't need (and couldn't use) evacuation in that case since it was a 2x duplicated pool, though I find that feature of Scanner highly valuable, and in your case you could implement it with three 2-pair drives in each Duplication pool (12 total drives). At the least, you could see if SMART data was being retrieved by Scanner successfully.
Very few (1 or 2?) SMART polling utilities were able to "see through" my LSI controller to the SMART info. I think they really wanted people to be using the MegaRaid software they developed for it instead.
-
1 hour ago, zeroibis said:
Interesting about your LSI 9201-16e that you need to have is pass the SMART data. Is that card flashed to IT mode so it bypasses the internal card firmware?
As far as speed goes software raid 0 can be just as fast to even faster than hardware depending on the system. If your using an intel atom your going to be better off with hardware I would suspect. I will be running on a Ryzen 3 1200 which should offer plenty of CPU for simple raid 0 especially on mechanical drives.
Yep, using the latest 20.x in IT mode. It operates in HBA mode fine, just doesn't want to pass SMART without that feature enabled in Scanner.
Interesting about software RAID 0 speeds, I may test that at some point myself just out of curiosity. It used to be that hardware implementations bypassed all types of other necessary layers, hence were faster.
21 minutes ago, Umfriend said:Wow. The notion of DrivePool not ever balancing except by selecting the appropriate disk for new files is, given my circumstances, ffing brilliant! How do you ensure DP will never rebalance (unless by user intervention)?
Just disable balancing on the pool - choose "Do not balance automatically", and make sure (if you installed it) that the "Disk Space Equalizer" plugin is turned off (it's usually on-demand anyway for immediate re-balancing). I also reduce the Automatic Balancing Triggers to around 50% for ratio, and disable "Or if at least this much data needs to be moved" (though it's redundant since Balancing is off except for plugins forcing it). The only Balancing plugin I have activated actually, is Stablebit Scanner's, for auto-evacuation. The only maintenance I do regularly on the data, is a simple defragmentation with Auslogics, so that file read speeds stay high.
-
24 minutes ago, zeroibis said:
Would the scanner still not work when using software raid 0 as the underline drives are still accessible by the system directly?
I did try to research that one, but came up without a definitive answer. I know from experience that Scanner has an "unsafe DirectIO" feature you can enable to poll smart and other data from drives/controllers that otherwise wouldn't pass the info (I use it with my LSI 9201-16e).
For Hardware RAID, it depends largely on the controller and it's driver set (I'd assume most won't pass SMART in a RAID scenario properly). For Software RAID... it's anyone's guess. I'm not sure you'd see the same performance gains in a software RAID 0 stripe as you would a hardware stripe, so from that perspective the primary advantage of it is reduced. This is what I was able to find after a cursory search:
- https://homeservershow.com/forums/topic/3210-does-stablebit-scanner-work-with-drives-in-raid-setup/?do=findComment&comment=31690
- https://community.covecube.com/index.php?/topic/1173-stablebit-or-raid-5/&do=findComment&comment=7850
Perhaps @Alex or @Christopher (Drashna) could chime in on whether or not Scanner (and the auto-evacuate feature in DP) work with software-raided drives.
-
Not sure if you'd considered it yet, but with your level of interest in reliability and redundancy you may want to run Stablebit Scanner on the same server as Drivepool. I rely on it now as a "drive watchdog" to email/text me when a disk is starting to have issues, or looks like it might. It can automatically trigger Drivepool to evacuate a disk's data to others when it senses trouble, and has configurable surface scan checks. The only caveat to using it is that you couldn't use RAID, since it would obfuscate the drives and not allow Scanner to do it's thing (no SMART data passed with RAID).
Going to try a comparison of your "Solution C" to a standard 8-Drive Pool (both with 2x pool duplication) configured just to see how it comes out with drives dying at the same time. Feel free to let me know if I miss anything.
Failure scenario: 1 drive dying
- Impact on Solution C: No Scanner monitoring, so no warning until drive dies. Takes out 1 volume in a Duplication pool. Zero data loss or impact on availability.
- Impact on 8-Drive Pool: Smart monitoring through Scanner - alerts issued before drive dies. Drive contents can be auto evacuated to any of the other 7 drives in the pool. Zero data loss or impact on availability.
Failure scenario: 2 drives dying (simultaneously)
- Impact on Solution C: No Scanner monitoring, so no warning until drive dies or data is damaged. Potential to take out 50% of all data (one entire Duplication pool).
- Impact on 8-Drive Pool: Smart monitoring through Scanner - alerts issued before drives die. Contents of both drives can be auto evacuated to any of the other 6 drives in the pool. Potential to take out only 25% of all data - pool duplication and random file distribution reduces this further.
Failure Scenario: 3 drives dying (simultaneously)
- Impact on Solution C: Same impact as 2 drives dying (potential 50% data loss). Potential for zero file redundancy after failures.
- Impact on 8-Drive Pool: Potential to take out ~37% of all data - 2x pool duplication and random file distribution reduces this further.
Failure Scenario: 4 drives dying (simultaneously)
- Impact on Solution C: Potential to take out entire data set (100% failure and 0% availability)
- Impact on 8-Drive Pool: Potential to take out ~50% of all data - 2x pool duplication and random file distribution reduces this further.
RAID 0 (Solution C) Advantages
- Read speed increases at the lowest level (drive pairs) in addition to duplication read speed gains.
- Due to being more highly structured in architecture, hardware level redundancy is increased.
8-Drive Pool Advantages
- Simple implementation - just add /remove disks to suit storage/duplication desired.
- Very flexible pool duplication rules - 3x, 4x, etc all available by changing one setting.
- Able to pass SMART drive data to a monitoring utility like Stablebit Scanner.
- Able to use file placement rules to populate a specific drive with desired files.
- Restoring a backup only requires the data on the failed disks be restored (if even necessary - auto-evacuation may have solved the issue with no restore needed).
RAID 0 (Solution C) Disadvantages
- 1 failure takes out data from two paired disks, effectively taking them both offline.
- 50% drive failure has potential for 0% availability.
- Requires a more highly structured implementation.
- Not expandable on a per-drive basis using RAID 0 (Solution C would require expansion by adding 2 more drives all at the same time, 4 if you wanted symmetry).
- A drive from a RAID 0 pair is not portable, unlike an 8-Drive Pool's drives are.
- Restoring a backup to a Duplication Pool means 4 drive's worth of data to write.
8-Drive Pool Disadvantages
- Loses the read speed gains of RAID 0 pairs, though it retains duplication read speed gains.
Note: these are all worst-case scenarios to illustrate the resiliency in the different architectures. There are best case scenarios as well, like a 4-drive failure on the Solution C causing zero impact (only the mirror drives are taken out). But since we can't rely on that, I went the way I was taught with disaster recovery planning.
I do like your architecture and planning, but weighing the pros and cons of it all seems to still be in favor of a traditional 8-drive Pool. Very interested to hear your thoughts on it as well however!
-
3 minutes ago, zeroibis said:
Ah very good to know, that setting eliminates the issue!
Good to know and also another reason just to use Primocache instead and just avoid any complication.
Yep, the SSD Optimizer definitely has it's place, but it was originally added as a feature request and isn't quite as robust as a separate product designed for full-volume block-level caching (which is also paid software).
You still might run into timing issues with files being added and your backups, but you can control all of that so you'll figure out a good schedule.
-
9 hours ago, Umfriend said:
AFAIK, the SSD Optimizer requires a number of SSDs that is equal to the level of duplication. I.e., one SSD will not work for a Pool that has x2 duplication. Having said that, you could create two x2 duplication Pools and put one on top of that, unduplicated and with the SSD as cache. That would work I think.
You're right, good catch. Click the up/right arrow below to read the specific reply from Christopher about using the SSD optimizer plugin with duplication.
I read this to mean that the SSD won't hold multiple copies - one (or more depending on duplication) will end up on mechanical drives in the pool. But due to the way Drivepool weighs reads from duplication drives, it'll usually choose the SSD for reading at that point.
-
1 hour ago, zeroibis said:
So I guess the question is if it is possible to completely disable rebalencing and have the balancing only take place when your adding files to the pool.
Yep, that's how I have mine configured currently. No rebalance passes (unless I kick them off), but native balancing when new files are added.
Quote... by default, StableBit DrivePool adds new data to the drive with the most available free space (absolute, not percentage).
-
Did a little digging and I can confirm that Backblaze will work with Drivepool. @Christopher (Drashna) tested it a few years ago.
Drivepool actually just passes regular drive commands to the drives, and they are fully portable. You can shut down, disconnect, and take one to another machine to read the files without any problems. Having said that - if you run Backblaze against the pool drives (and not the virtual Pool drive itself), you'll be able to have a replacement drive with data sent out to you at need.
-
You can definitely do it that way, especially if your safeguard is to have a backup HDD overnighted to you. Personally I'd throw a little bit more storage at it and enable either 3x or 4x duplication on a single pool (or have 2x on the master pool and 2x on each child pool). That would give the redundancy and even better speed for the 10G network, but cost slightly more. It would also almost negate the need for overnighting a drive in case of ~2 drive failures.
For your scenario (now that I see it and your contingencies better), software RAID snapshots are entirely too slow. It sounds like your storage solution has far too much turnover (dynamically changing data) for that to work well, and even the creator of SnapRAID indicates it's made more for data repositories that don't change a lot. So yeah - stick with a pooled solution and redundancy.
Sounds like you've worked out some good solutions and have a fair knowledge of how to achieve your goal, just need to determine what resources ($$) you're going to throw at it and how you want to configure it. I recently (weeks ago) put together a 88TB server with 72TB of data storage (the remainder for parity). I sourced the drives in USB enclosures at reduced cost and "shucked" them so I ended up with the bare drives which went into the enclosure. That might be advantageous for you if you wanted to crank up the duplication in Drivepool - you'd have the space to spare in that case.
-
No need to create hierarchical pools when you use full Pool duplication. Just one pool with X number of copies set. If you set that number to the number of drives you use, you have a redundancy factor equal to your drive count, though that's a bit overkill in most situations. No disaster recovery needed in this case - the pool just keeps on running and tries to evacuate the damaged disk if necessary. You -could- make multiple child pools and mirror them inside the master pool, but I don't see any advantages vs Pool duplication for you, unless parts of your data have different needs (placement, duplication factor, etc).
That's why I use SnapRAID - conventional RAID solutions just can't protect like they used to, with the size of data we deal with today. Two interesting and informative articles that show why this is true:
- https://www.zdnet.com/article/why-raid-5-stops-working-in-2009/
- https://www.zdnet.com/article/why-raid-6-stops-working-in-2019/
Consider for a moment if you have two pools setup - one a main pool with active data, and one a "mirror pool" for redundancy. If you have a drive fail on the main, AND a drive fail on the mirror at the same time, you risk losing file(s). That could happen with any number of child pools. With software RAID (parity calculations), you can lose up to 6 drives at once and still be protected in SnapRAID if you set it up with that many parity drives. One user of SnapRAID successfully recovered after a simultaneous loss of 4 drives.
Pool duplication is definitely viable, but you'd have to have a duplication factor of 5 or more on the entire pool to avoid that kind of disaster. It's doable however, if you want to use Drivepool's duplication and the extra drives instead of a solution like SnapRAID. With SnapRAID you can have a pool consisting of up to 42 data disks with just 6 parity disks, and STILL be protected. That's an efficient use of resources. A solution like SnapRAID however does require some downtime during disaster recovery, for it to rebuild replaced drives. The flipside is that you have better overall utilization of storage space, and not so many duplicates hanging around.
On the side - I wouldn't recommend software RAID in Windows for this kind of data protection/use. Between file duplication on the pool for increased read speeds, and a SSD handling L2 write caching with Primocache, you'd be set for instant offload of writes and faster reads. Most networks in business use today are still 1Gbit, so can easily be saturated by 1-2 drives' activity with larger files.
Your solution will of course depend on how easy you want your disaster recovery to be - easier requires much more investment in resources.
And yeah, Primocache is a great piece of software for all kinds of drive caching. It has a rather long free trial so you can test different setups.
-
I can't speak to BackBlaze, but I'll try with the others:
2) Yes you can use a single SSD as a pool cache, and use real-time duplication on the other pool drives. However, to get the nearly-real-time functionality you want, you would want to use a L2 SSD cache with a product like Primocache. It does real block-level caching in either RAM (L1 cache), or against a SSD (L2 cache) at whatever intervals you setup. It can cache the write commands made to the Pool and immediately transfer the data across as fast as the drives will take it. I use the software on virtually all my machines, and have been since before it was even Beta. Great stuff, worth the price.
3) See #2. I am fairly sure that with real-time duplication set on the main pool drives, any writes directly to them are literally synchronous.
4) You're right in assuming that if you have a file to be written to the pool that is larger than the space on the SSD, you'll run into issues. The reason is that the DP SSD cache is treated as another pool drive, just a front-load one so that all files land there first, THEN get migrated to the main pool drives after. From this perspective, using Primocache and telling it to cache writes for only a few seconds before writing them (and then freeing the cache space) would be ideal.
5) The read speed boost from duplication across pool drives is limited to the number of drives the file resides on. 2 drives might get you a small boost, whereas 6 drives might get you a substantial boost. It also depends on the size of the file and how long you can sustain multiple reads from all drives it sits on.
6) Correct - RAID functionality (beyond RAID 1 style duplication on either the entire pool or folders/files on it) isn't part of DrivePool. For that I'd recommend software like SnapRAID, which computes parity across all drives and stores it on separate parity drives. You can have anywhere from 1 parity drive (RAID 5) up to 6 parity drives (well beyond RAID 6 capabilities). It is on-demand parity calculation software, so you run it on a schedule to update (I do mine nightly). Works rather well, and is free. Native duplication on the DP pool works well however, if you're willing to sacrifice the extra space to do it.
7) You can mount as many drives as you want for Drivepool (and in fact SnapRAID) to use, simply use folder mount points. You don't need to limit yourself to drive letters where the software is concerned. They are assigned in the same place where you assign letters - you just create a folder for each on the drive you want them to appear before in Explorer, then assign it there.
-
7 hours ago, Methanoid said:
Related Qs
Scenario: Low powered server & download box. Want to order tasks to avoid them running at same time. e.g. Download queue empties so then execute Balance to move files and then execute SnapRAID to update.
Q1: How can I call the DP Balance function (with DP running all the time) just when the Download queue is empty (the downloader can identify and script calling once queue is empty)?
Q2: Once Balance is complete how can I then call to SnapRAID to do its stuff?
DPCMD.exe (from a powershell script) would be the way to do this, except I can't find a re-balance command among it's documented list. You'd run this command, wait for it to finish & issue a pause x command (or simply pause for x minutes if it terminates immediately), then the command(s) for SnapRAID. Perhaps Christopher/Alex has more information on whether or not dpcmd can trigger a re-balance.
However, if you have automatic on-the-fly balancing turned on, there should be little need to constantly re-balance the pool. In fact - I find it detrimental to do that regularly, since it actually hurts the way SnapRAID works. The more things SnapRAID has to re-calculate parity for (every time a file is moved it has to re-calculate across all files that moved/deleted file was linked to) the more work it does, and the greater a chance it will bomb.
Point in case: I recently upgraded a few hundred tv shows from 720p to 1080p, and the combination of deleting of the old and adding the new actually caused SnapRAID to error out and never finish. I had to delete all .parity and .content files and have it run a completely fresh parity build.
So from that perspective, if you have a higher number of additions/removals on the pool, I'd just let DP balance real-time instead of forcing it on a schedule. Run manual re-balances during low activity times seems like a better course of action.
-
The way I understood Scanner's Unsafe DirectIO to work, was that it only uses it to poll SMART info. From that perspective, it wouldn't mess with data on any of the drives it sees, and at worst case could force a BSOD (on the few controllers it doesn't work with) which conceivably *could* corrupt drive data IF Windows was in the process of writing or flushing the write buffers.
http://wiki.covecube.com/StableBit_Scanner_Advanced_Settings#Legacy
I don't see a per-controller or per-drive setting for it.
Overall chance in most cases to produce lasting damage = miniscule, in my opinion. You'd know rather quickly if you tested it (without any writing going on to the drive(s)), and avoid any potential harm.
-
3 minutes ago, Christopher (Drashna) said:
And yeah, the blinking lights are fun to watch!
On 8/8/2018 at 5:39 PM, johnnj said:Additionally, I enjoy watching the blinking lights.
Damn it - don't make me go buy a half-height rack and enclosure now guys...
-
Sounds like a good idea. Either a setting per provider, or per Cloud drive.
-
9 hours ago, chcguy88 said:
I don't know what changed recently, but speeds are much better with the latest 1.1.0.1051 update. (I just got the update the other day).
Hard to know since root cause was never really established. Could have been something as simple as Internet routing changing (i.e. to replace a bad hop's hardware), or Alex could have altered Clouddrive. Good to hear though.
-
Sounds like a bad install. I'd suggest uninstalling Drivepool, making sure you're logged in with an admin-level account in Windows, temporarily disabling your Anti-virus, cleaning out your temp files with something like CCleaner, re-downloading the latest DrivePool package, and then trying a re-install (right-click the download, choose Run as administrator).
-
5 hours ago, Sonicmojo said:
Thanks for this - I am certain I would have been right back here asking about this!
Cheers
Sonic.
How's your controller flash going Sonic?
-
7 hours ago, Umfriend said:
So I just got my SAS controller (Dell H310 flashed to IT mode). It works but it does not pass SMART to scanner. I have looked but could not find it. Isn't there a setting I should change in Scanner?
Edit: Never mind, found it. Scanner -> Settings -> Advanced Settings & Troubleshooting -> Cinfiguration Properities -> DirectIO- check Unsafe (which does not sound scary at all!)
I just went through finding this myself a few days ago. Glad you were able to get it sorted!
-
Unfortunately I tried Elucidate when I first chose SnapRAID, and it appeared to have some issues with the latest config file format. It blew my old config away when it tried to re-format it, and never showed me the current parity data. Had to start over and re-create the config from scratch.
So while I'd love to recommend -any- GUI for SnapRAID, that one's problematic from what I saw ~4 months ago. Andrea should probably remove the link until (if) it's updated again.
-
5 hours ago, Christopher (Drashna) said:
But keep in mind that each disk can only be part of a single pool.
Logical disk, right? It could simply be a formatted partition the way I understood it. It would make spreading multiple pools out over the same set of drives way easier.
Edit: Just tested this myself, it'll take logical volumes (partitions), so that you could structure multiple pools across the same set of physical drives.
Reallocated Sectors - Drive Latency
in General
Posted
As an afterthought - try browsing into one of the hidden PoolPart folders on your Pooled data drives, find a movie to test, and open it directly from there (not from the DP-mounted Pool letter). See what your latency is like for that scenario.