My system has gone nuts. Help!

[fleggett1]

A few days ago, I noticed that Explorer wasn't refreshing in real-time.  I did some Googling, found some hits, and did what was advised, which included restarting.

So, I did.  Problem is, the restart is going on TWO DAYS.  I know the system is doing >something<, as the drive activity light continues to blink furiously, which is why I didn't just pull the plug after a few minutes.

The only app I can think of (besides Windows itself) that would cause such behavior is Drivepool, as I keep this machine as lean and mean as possible.  If that's the case, what in the world could be happening?  Is it doing an impromptu rebalance?  I doubt it's an evacuation, as all the drives were fine just before I initiated the restart.

I can't invoke anything, like Task Manager, as it's too far into the restart process.  Every keystroke I've tried doesn't bring-up anything.  All I see is the black restart screen with the white circle, which has been circling the entire time.

If I get up in the morning and it's still trying to restart, I suppose I will have to pull the plug, as three days will be the end of my rope.  If that's the case, assuming the system comes back at all, is there a bulletproof way to check all the pool files to ensure that they're still intact?  This is with Windows 11.

[Shane]

If I had to guess, I'd guess either what was advised from your googling or a Windows Update has gone wrong. I doubt it's DrivePool related. Restarting should interrupt a rebalance/evacuation at least.

The only "bulletproof" way to check all files are intact (pool or otherwise) is to compare with - or revert to - a backup, but it should be highly unlikely that a restart should hose pool content outside the boot drive (as opposed to maybe needing to reinstall DrivePool if whatever went wrong has sufficiently messed up the boot drive).

[fleggett1]

I just wish I knew what the hell the system is doing, as it's definitely doing >something<.  The reason why I leveled the finger at DP is because I've only seen a rebalance or evacuation take this long.

Assuming I can get things back to some semblance of normal, is there anything I can run on the pool to check file integrity?  I know a simple chkdsk doesn't really cut it (I don't think it even works with DP).  I don't have backups for...reasons.

I'm already resigned to reinstalling Windows, which is going to be a royal PITA, as the few apps I do have installed have janky restore steps.

[Shane]

You can run chkdsk on the individual drives that make up the pool, and you can do a consistency check (cog icon -> Troubleshooting, Recheck duplication) if you're using duplication, but... that's about it without backups or the like.

P.S. for whatever it might be worth, I use Veeam to take backup images of just the boot drive on my drivepool machine, entirely separate to my pool backups, so that if I manage to brick Windows (or it bricks itself) I can just revert to yesterday's image (or last week's) without having to revert the entire pool as well.

[Christopher (Drashna)]

If you have StableBit CloudDrive installed, it does have some cleanup that has to occur before a full shutdown can take place. Namely, it has to flush all the data to disk, which is fun because flushing data t othe clouddrive disk causes writes to the cache drive , which then also need to be flushed.   However, this definitely shouldn't take two days. 

And if the system is running the "shutting down" screen, it should usually list a service that it is waiting on (like the StableBit CloudDrive Service).   

However, if you don't have that installed, StableBit DrivePool shouldn't be doing anything to halt the shutdown, especially if you're not using StableBit Cloud, as well. 

If you can still get a command prompt up, "shutdown -a" should abort the shutdown process.  But I suspect that this isn't possible. 

 

[fleggett1]

Well, my system is definitely screwed.  Not 100% toast, but enough that I'm definitely gonna have to reinstall Windows...again.

I swear, I think my old AT box I had in the 90s was more stable than this "state of of the art" contraption.

I can't get the system to perform a restart.  It'll get deep into the process, so much so that I can't bring ANYTHING up, but it just won't flip that last bit and do the actual deed.  I still see the disk light flickering furiously, but can't even begin to figure-out why.  I've checked my two Terramaster enclosures and none of their activity lights are lit, so it's something relegated to my two NVMEs.  One of them is a Samsung (my boot drive), so I'm going to install Magician to make sure it's not trying to lobotomize itself.

All that said, I still think it MIGHT be DP related.  I suspect this because I can no longer bring up the UIs for either DP or Scanner.  The pool still seems to work, but the UIs for both have gone bye-bye.  When I get the system back up, I'm gonna kill the processes and restart both to see what happens, but I'm not sure that'll do anything beyond satisfying my curiosity.  No other app is affected, as they all operate normally except for DP and Scanner, which is another reason why I think something is off with both.  Of course, it could just be coincidence, but I'm runnin' on fumes here.

Shane, I used to do boot drive images via Acronis Trueimage back in the PITA-to-install XP days.  I might go back to doing that, so thanks for the Veeam rec.  When I get everything back to working order, I'll run a chkdsk on all the pool drives (and pray).

Christopher, I'm not using any cloud-based tech.  It's just a local box running local stuff.  And like I wrote, it gets too far into the restart process for manual input, like invoking Task Manager or even a simple shell prompt.  I wish I could, as I'd REALLY like to know what Windows is trying to do or at least try to determine what app or process is holding-up the restart, but that's a no-go.

It's all very mystifying with a huge helping of frustration, as I haven't done anything to warrant this.  I got the Explorer fix from an official Windows tech on an official Windows forum, so I can't imagine it's that.  Arrrghh.  Maybe I really do need to try to learn some linux.

[Christopher (Drashna)]

21 hours ago, fleggett1 said:

I swear, I think my old AT box I had in the 90s was more stable than this "state of of the art" contraption.

I hear you on that! Less complexity does mean less that can go wrong.

For the UI, what do you mean that it has gone "bye-bye"?  Are the UI's showing up at all but with black screens?  Or not showing up at all?  Or?

And Veeam is very nice to use, for sure!

If this is StableBit DrivePool related, and only happens when it is installed, then this might be useful for us, at least: 
https://wiki.covecube.com/StableBit_DrivePool_System_Freeze

This does trigger a BSOD (so an unsafe shutdown), but dumps the active memory to disk.  This can be incredibly helpful information, if you know how to read these memory dumps (I only kinda do, but Alex, the developer, is a pro at these). 

 

[fleggett1]

Well, I might've inserted my foot squarely into my mouth this time.  I'm still rather mystified by this series of events, but it is what it is.

My system has returned to stability...without me needing to reinstall Windows.  I know that sounds ridiculous, but (this time) it's true.

Apparently, one of my apps that ends in *arr downloaded a virus.  This made the rounds on Reddit and Windows actually caught-it, but not before my system started to go haywire.  Since then, yet another Windows update came down the pipe, so I can only surmise that it, along with eliminating the virus, "cured" my box.

I know that doesn't make a hill o' beans amount of sense, but they're the only things that have occurred since my last post.

To summarize the current state of affairs, I can now restart normally and invoke both DP and Scanner's UI.  Before, I would click on the icons for both and nothing would happen.  No black screen, no crashes, just nothing would appear.  I've had that happen before with other apps and a restart will usually fix the issue.

So, it looks like neither DP or Scanner were at fault after all (and there was much rejoicing).

I dunno if I'm brave enough to do that memory dump, as it's exactly the sort of thing I try to avoid.  However, in the future, if worse comes to worst...  (I'll bookmark the url.)

Now, if my shucked WDs would quit dying on me, things would be peachy (I had to throw away three last month).

[Christopher (Drashna)]

Well, I'm sorry to hear about the virus, but glad it was .... a simple enough issue in the end! 

Also, assuming that the *arr program downloaded the virus via the download client, double check that the download client is configured to block "malicious extensions" (eg, exe, lnk, bat, etc).   Generally, it's a good idea to block any executible or script by default.

As for the shucked drives ... yeah, that's gonna happen, unfortunately.   A better alternative may be to buy recertified drives from a reputible company, as these are cheaper than new drives, and generally in very good condition.

[fleggett1]

Well, at least I THINK the virus was at least part of the issue.  Having not had one years, though, it's impossible to say.  I also don't know if the virus actually ran or was just downloaded and immediately quarantined.

But, yeah, I need to revisit my list of exclusions and add stuff like "lnk" extensions, as it was a file with a .lnk ending that was the culprit.  I also need to do this for torrents.  Just when you think you've got the system perfected, something like this idiocy rears its head.

I'm no longer shucking.  After all these failures, I trust them about as far as I can disc throw them at the Olympics.  From here on out, it's "real" OEM drives or bust.  I don't even trust recertified drives at this point.

And to pour salt into the wound, I really only "saved" a few bucks for drives that, in the end, weren't worth squat.  And then there was the pain of extracting them from their enclosures and doing the 3.3V pin mod when needed.  What nonsense.

Sigh.  I remember when WD was at the top of the reliability game.  Now, they seem to be bottom-feeding like the rest of the catfish.  We really need a manufacturer to step up and release a line of semi-expensive drives that won't crap their platters or blow their firmware after three or four years of relatively light use.  I would pay a handsome premium for 20+ TB spinner that lasted seven or eight years.

[Shane]

For whatever it's worth, I've currently got a pair of WD Red Plus at over 9 years run time and a WD Red at over 7 years run time; I've generally had good luck with Reds of all stripes outlasting their warranties by a year or more and hope that continues.

... I'm reasonably sure 20+ TB spinners first retailed back in 2021, so technically none of them have lasted over four years yet. 😜 

[fleggett1]

Well, I've got a handful of 2 and 4 TB Reds WDs that were still working when I pulled them for simply being too small.  These drives were pretty old - probably more than ten years at this point (maybe older).

At some point, once they crossed a TB per price threshold, the reliability went into the crapper.  I guess you can say that about every drive company, but it pains me to lump WD into that cesspool, as I used to recommend them to everyone and their fourth cousin.  Now, I have no earthly idea what make & model to get.

Point taken about the 2021 date. I guess what I'm wanting is someone to come out with high-capacity drives with 10 year warranties and actually mean it.  That would be a game changer for a lot of people.

[Christopher (Drashna)]

Shucking drives is not worth the effort, yeah.  I won't tell people not to do it, but I do recommend against it.  Every shucked drive I have had, has failed within 3 years of purchase.

And I can attest to the WD Red and Seagate NAS being good quality drives.   In fact.... the seagate NAS drives I have show ~2 years for their age because the SMART data rolled over at 10 years.  

[fleggett1]

That's a little surprising about Seagate, as I recall them being pretty low on the drive tier reliability list years ago.  Maybe I should give them another shot.

Problem with me trusting the Reds is that I've had two fail on me in the past six months, though one of them had the decency to spit out Smart errors so that I could proactively replace it (the other, by a stroke of cosmic luck, was empty).  I just don't trust anyone anymore.  If it wasn't for the Smart-auto-evacuate feature, I would've lost GBs and maybe even TBs of files due to failures.

And I still see the occasional drive dropout, which is beyond vexing, as I don't know if it's enclosure- or drive-related.  A restart is necessary to "resurrect" the drive so that DP recognizes it.  It doesn't happen often - maybe once every few months - but it's teeth gnashing when it does occur and makes me think dark thoughts.

I suppose I need to seriously consider duplication, which I've avoided my entire computer life due to the byte tax.  I have the two Terramasters, so designating one as the primary and the other as the backup should be easy.  Problem is investing around $1000 in sufficiently large drives to make it happen.  My 401(a) just took a GIGANTIC hit recently with the stock market trying to find the center of the Earth, so there's no relief there (if anything, I should be putting money in to shore it up).

Still, I could put three or four drives on my Amazon wishlist in the hope of pulling the trigger at some point in the not-too-distant future.

[Christopher (Drashna)]

Honestly, the low reliability was due to the floods, and then one specific line of drive (ST#000DM001) that was the issue.  Aside from those, Seagate drives have been rock solid for me (and IIRC, BackBlaze statistics bears that out too).  That said, I do use a mix of brands.

As for the drives, that sucks.  And if they were from the same batch of drives, that definitely makes sense (unfortunately).   But for any drive really, it's not "if" it fails, but "when".  And maybe ... how catastrophically. 

As for the dropout, if that is USB, then it's USB.  That's normal (and IIRC, allowed by the specs for USB).  Unfortunately.

And yeah, I hear you on the drives.  It's the only reason I don't have duplication for most of my pool.  Only the critical stuff is duplicated.   The drive capacity keeps in increasing and prices gradually drop (though, lets hope that continues, all things considered).    And IMO, it's worth getting the NAS/enterprise drives.  Between warranty and features/use-case, it's worth it.

[fleggett1]

I think I actually have a Seagate or two in my Terras, but I need to double-check.  Didn't the floods also affect WD pretty badly?

Regarding reliability, just so I'm clear, I know a drive that costs a little north of $300 won't last forever, especially if it's being written and read from almost every second of the day.  Like I wrote earlier, I'd happily pay a handsome sum for a high-capacity drive that genuinely lasted, at a minimum, 8 years.  It's the disposability/e-waste that annoys me, as people have now been conditioned to believe that hard drives should only last a few scant years before going bad.  For those of us distributing (ahem) numerous Linux distros, it's become a chronic fear that an un-duplicated drive might implode at a moment's notice, even if that drive has only seen two or three years of use.

It's kinda like what bios updating has become.  Before, the necessary blocks would be written, you'd see the progress bar, and it only took a minute and a single reboot.  Now, updating a bios is like a game of Russian Roulette, with numerous mystery reboots and highly disconcerting black screens throughout.  It's like the technology has gotten so "advanced" that it's become proportionally brittle.

Regarding the USB dropout, there's a power setting that I always turn off called "USB selective suspend setting", as it sounds like a definite no-no, but I dunno if it does what I think it does.  I also make sure to disable any sleep settings.

I think I am headed in the duplication direction, much to my chagrin.  It's more money, more hardware, more heat, more electricity, and more software complexity, which makes me highly uneasy, even though it's suppose to do the opposite, but it's the only thing I know to do to combat all these drive failures.

[Shane]

There are some SSDs with 10 year warranties, but SSDs are still a lot more expensive than HDD so...

I hear you on the brittleness of UEFI.

A midway option between N disks (no duplication) and 2N disks (2x duplication) that I've occasionally considered (but haven't really had a pool big enough to make it worth buying the cards/hubs) is pooling RAID6 arrays instead of using duplication with individual disks, basically offloading the redundancy onto the hardware.

Con: Needs one or more RAID cards (or RAID-capable DAS boxes), it's not full duplication, any given array would be vulnerable while rebuilding (though I have backups) and card management means yet another UI pane to deal with.

Pro: An array only needs 6+ disks to store 4+ disks worth of data (more disks = better ratio but more time to rebuild) and 3 disks would have to die concurrently to actually lose any data, offers a boost to read performance and RAID6 allows detection and repair of bitrot.

[fleggett1]

SSDs also don't have the capacity that I need (at least, not in the consumer space).  I'm talking at least 18 TB and more, if economically rational.

I feared RAID would come into the convo.  To expose my ignorance, I know very little about the tech and what I have gleaned scares the bejeezus outta me.  I keep hearing the mantra that "RAID is not a backup" and have read stories of entire arrays going south because a drive or two failed and the wrong strategy was employed for a rescue.  And then there's really bleeding-edge stuff like unRAID and filesystems with RAID-type features built-in, like zfs.

I just want Plain Jane NTFS so that, if something goes wrong, the rest of the pool isn't dragged into the abyss or needs a rebuild that may (or may not) succeed after a few weeks of drive hammering.  I also VERY MUCH like the fact that I can pull a drive from my DP pool, stick it in another Windows machine that doesn't have a clue what Drivepool is, and I can access all the files normally, without additional steps, and free from some exotic RAID implementation.  IOW, I like - no, NEED - a solution that's straightforward and doesn't descend into insane complexity.  I'm old and any new tricks I'm forced to learn have to be relatively simple and something I can visualize in my creaky, calcium-riddled brain.  This is why DP has fit all my criteria, as it does what it needs to do to configure the pool, present it to Windows as such, and that's it.

(I know the software is far more sophisticated under the hood than what I'm making it out to be, but I'm just speaking from the perspective of the "user experience".)

Besides, isn't hardware-specific RAID obsolete at this point given the raw power that modern machines offer, even low-spec boxes?

[Shane]

Fair.

For what it's worth the mantra "RAID is not backup" is shorthand for "RAID should never be a substitute for having independent copies of your data somewhere else" and this applies to DrivePool as well: if enough drives die you will lose data from the pool/array. TLDR, the only way to do backups is to actually have backups. The business mantra for that one is of course "3-2-1": have at least three copies*, on two different media, with at least one kept offsite. The middle part used to mean HDD versus tape, but these days also means using different HDD brands/models to minimise the risk from potential manufacturing defect.

*the original and two separate backups

Re hardware RAID being obsolete... it depends on the use-case. For home use? Basically, yeah, if the alternative is using an OS with good software RAID support. But sadly DrivePool is only available for Windows and DrivePool doesn't support dynamic drives so I can't even use Windows software RAID1 with it.