My system has gone nuts. Help!

[fleggett1]

A few days ago, I noticed that Explorer wasn't refreshing in real-time.  I did some Googling, found some hits, and did what was advised, which included restarting.

So, I did.  Problem is, the restart is going on TWO DAYS.  I know the system is doing >something<, as the drive activity light continues to blink furiously, which is why I didn't just pull the plug after a few minutes.

The only app I can think of (besides Windows itself) that would cause such behavior is Drivepool, as I keep this machine as lean and mean as possible.  If that's the case, what in the world could be happening?  Is it doing an impromptu rebalance?  I doubt it's an evacuation, as all the drives were fine just before I initiated the restart.

I can't invoke anything, like Task Manager, as it's too far into the restart process.  Every keystroke I've tried doesn't bring-up anything.  All I see is the black restart screen with the white circle, which has been circling the entire time.

If I get up in the morning and it's still trying to restart, I suppose I will have to pull the plug, as three days will be the end of my rope.  If that's the case, assuming the system comes back at all, is there a bulletproof way to check all the pool files to ensure that they're still intact?  This is with Windows 11.

[Shane]

Fair.

For what it's worth the mantra "RAID is not backup" is shorthand for "RAID should never be a substitute for having independent copies of your data somewhere else" and this applies to DrivePool as well: if enough drives die you will lose data from the pool/array. TLDR, the only way to do backups is to actually have backups. The business mantra for that one is of course "3-2-1": have at least three copies*, on two different media, with at least one kept offsite. The middle part used to mean HDD versus tape, but these days also means using different HDD brands/models to minimise the risk from potential manufacturing defect.

*the original and two separate backups

Re hardware RAID being obsolete... it depends on the use-case. For home use? Basically, yeah, if the alternative is using an OS with good software RAID support. But sadly DrivePool is only available for Windows and DrivePool doesn't support dynamic drives so I can't even use Windows software RAID1 with it.

[Shane]

If I had to guess, I'd guess either what was advised from your googling or a Windows Update has gone wrong. I doubt it's DrivePool related. Restarting should interrupt a rebalance/evacuation at least.

The only "bulletproof" way to check all files are intact (pool or otherwise) is to compare with - or revert to - a backup, but it should be highly unlikely that a restart should hose pool content outside the boot drive (as opposed to maybe needing to reinstall DrivePool if whatever went wrong has sufficiently messed up the boot drive).

[fleggett1]

I just wish I knew what the hell the system is doing, as it's definitely doing >something<.  The reason why I leveled the finger at DP is because I've only seen a rebalance or evacuation take this long.

Assuming I can get things back to some semblance of normal, is there anything I can run on the pool to check file integrity?  I know a simple chkdsk doesn't really cut it (I don't think it even works with DP).  I don't have backups for...reasons.

I'm already resigned to reinstalling Windows, which is going to be a royal PITA, as the few apps I do have installed have janky restore steps.

[Shane]

You can run chkdsk on the individual drives that make up the pool, and you can do a consistency check (cog icon -> Troubleshooting, Recheck duplication) if you're using duplication, but... that's about it without backups or the like.

P.S. for whatever it might be worth, I use Veeam to take backup images of just the boot drive on my drivepool machine, entirely separate to my pool backups, so that if I manage to brick Windows (or it bricks itself) I can just revert to yesterday's image (or last week's) without having to revert the entire pool as well.

[Christopher (Drashna)]

If you have StableBit CloudDrive installed, it does have some cleanup that has to occur before a full shutdown can take place. Namely, it has to flush all the data to disk, which is fun because flushing data t othe clouddrive disk causes writes to the cache drive , which then also need to be flushed.   However, this definitely shouldn't take two days. 

And if the system is running the "shutting down" screen, it should usually list a service that it is waiting on (like the StableBit CloudDrive Service).   

However, if you don't have that installed, StableBit DrivePool shouldn't be doing anything to halt the shutdown, especially if you're not using StableBit Cloud, as well. 

If you can still get a command prompt up, "shutdown -a" should abort the shutdown process.  But I suspect that this isn't possible. 

 

[fleggett1]

Well, my system is definitely screwed.  Not 100% toast, but enough that I'm definitely gonna have to reinstall Windows...again.

I swear, I think my old AT box I had in the 90s was more stable than this "state of of the art" contraption.

I can't get the system to perform a restart.  It'll get deep into the process, so much so that I can't bring ANYTHING up, but it just won't flip that last bit and do the actual deed.  I still see the disk light flickering furiously, but can't even begin to figure-out why.  I've checked my two Terramaster enclosures and none of their activity lights are lit, so it's something relegated to my two NVMEs.  One of them is a Samsung (my boot drive), so I'm going to install Magician to make sure it's not trying to lobotomize itself.

All that said, I still think it MIGHT be DP related.  I suspect this because I can no longer bring up the UIs for either DP or Scanner.  The pool still seems to work, but the UIs for both have gone bye-bye.  When I get the system back up, I'm gonna kill the processes and restart both to see what happens, but I'm not sure that'll do anything beyond satisfying my curiosity.  No other app is affected, as they all operate normally except for DP and Scanner, which is another reason why I think something is off with both.  Of course, it could just be coincidence, but I'm runnin' on fumes here.

Shane, I used to do boot drive images via Acronis Trueimage back in the PITA-to-install XP days.  I might go back to doing that, so thanks for the Veeam rec.  When I get everything back to working order, I'll run a chkdsk on all the pool drives (and pray).

Christopher, I'm not using any cloud-based tech.  It's just a local box running local stuff.  And like I wrote, it gets too far into the restart process for manual input, like invoking Task Manager or even a simple shell prompt.  I wish I could, as I'd REALLY like to know what Windows is trying to do or at least try to determine what app or process is holding-up the restart, but that's a no-go.

It's all very mystifying with a huge helping of frustration, as I haven't done anything to warrant this.  I got the Explorer fix from an official Windows tech on an official Windows forum, so I can't imagine it's that.  Arrrghh.  Maybe I really do need to try to learn some linux.

[Christopher (Drashna)]

21 hours ago, fleggett1 said:

I swear, I think my old AT box I had in the 90s was more stable than this "state of of the art" contraption.

I hear you on that! Less complexity does mean less that can go wrong.

For the UI, what do you mean that it has gone "bye-bye"?  Are the UI's showing up at all but with black screens?  Or not showing up at all?  Or?

And Veeam is very nice to use, for sure!

If this is StableBit DrivePool related, and only happens when it is installed, then this might be useful for us, at least: 
https://wiki.covecube.com/StableBit_DrivePool_System_Freeze

This does trigger a BSOD (so an unsafe shutdown), but dumps the active memory to disk.  This can be incredibly helpful information, if you know how to read these memory dumps (I only kinda do, but Alex, the developer, is a pro at these). 

 

[fleggett1]

Well, I might've inserted my foot squarely into my mouth this time.  I'm still rather mystified by this series of events, but it is what it is.

My system has returned to stability...without me needing to reinstall Windows.  I know that sounds ridiculous, but (this time) it's true.

Apparently, one of my apps that ends in *arr downloaded a virus.  This made the rounds on Reddit and Windows actually caught-it, but not before my system started to go haywire.  Since then, yet another Windows update came down the pipe, so I can only surmise that it, along with eliminating the virus, "cured" my box.

I know that doesn't make a hill o' beans amount of sense, but they're the only things that have occurred since my last post.

To summarize the current state of affairs, I can now restart normally and invoke both DP and Scanner's UI.  Before, I would click on the icons for both and nothing would happen.  No black screen, no crashes, just nothing would appear.  I've had that happen before with other apps and a restart will usually fix the issue.

So, it looks like neither DP or Scanner were at fault after all (and there was much rejoicing).

I dunno if I'm brave enough to do that memory dump, as it's exactly the sort of thing I try to avoid.  However, in the future, if worse comes to worst...  (I'll bookmark the url.)

Now, if my shucked WDs would quit dying on me, things would be peachy (I had to throw away three last month).

[Christopher (Drashna)]

Well, I'm sorry to hear about the virus, but glad it was .... a simple enough issue in the end! 

Also, assuming that the *arr program downloaded the virus via the download client, double check that the download client is configured to block "malicious extensions" (eg, exe, lnk, bat, etc).   Generally, it's a good idea to block any executible or script by default.

As for the shucked drives ... yeah, that's gonna happen, unfortunately.   A better alternative may be to buy recertified drives from a reputible company, as these are cheaper than new drives, and generally in very good condition.

[fleggett1]

Well, at least I THINK the virus was at least part of the issue.  Having not had one years, though, it's impossible to say.  I also don't know if the virus actually ran or was just downloaded and immediately quarantined.

But, yeah, I need to revisit my list of exclusions and add stuff like "lnk" extensions, as it was a file with a .lnk ending that was the culprit.  I also need to do this for torrents.  Just when you think you've got the system perfected, something like this idiocy rears its head.

I'm no longer shucking.  After all these failures, I trust them about as far as I can disc throw them at the Olympics.  From here on out, it's "real" OEM drives or bust.  I don't even trust recertified drives at this point.

And to pour salt into the wound, I really only "saved" a few bucks for drives that, in the end, weren't worth squat.  And then there was the pain of extracting them from their enclosures and doing the 3.3V pin mod when needed.  What nonsense.

Sigh.  I remember when WD was at the top of the reliability game.  Now, they seem to be bottom-feeding like the rest of the catfish.  We really need a manufacturer to step up and release a line of semi-expensive drives that won't crap their platters or blow their firmware after three or four years of relatively light use.  I would pay a handsome premium for 20+ TB spinner that lasted seven or eight years.

[Shane]

For whatever it's worth, I've currently got a pair of WD Red Plus at over 9 years run time and a WD Red at over 7 years run time; I've generally had good luck with Reds of all stripes outlasting their warranties by a year or more and hope that continues.

... I'm reasonably sure 20+ TB spinners first retailed back in 2021, so technically none of them have lasted over four years yet. 😜 

[fleggett1]

Well, I've got a handful of 2 and 4 TB Reds WDs that were still working when I pulled them for simply being too small.  These drives were pretty old - probably more than ten years at this point (maybe older).

At some point, once they crossed a TB per price threshold, the reliability went into the crapper.  I guess you can say that about every drive company, but it pains me to lump WD into that cesspool, as I used to recommend them to everyone and their fourth cousin.  Now, I have no earthly idea what make & model to get.

Point taken about the 2021 date. I guess what I'm wanting is someone to come out with high-capacity drives with 10 year warranties and actually mean it.  That would be a game changer for a lot of people.

[Christopher (Drashna)]

Shucking drives is not worth the effort, yeah.  I won't tell people not to do it, but I do recommend against it.  Every shucked drive I have had, has failed within 3 years of purchase.

And I can attest to the WD Red and Seagate NAS being good quality drives.   In fact.... the seagate NAS drives I have show ~2 years for their age because the SMART data rolled over at 10 years.  

[fleggett1]

That's a little surprising about Seagate, as I recall them being pretty low on the drive tier reliability list years ago.  Maybe I should give them another shot.

Problem with me trusting the Reds is that I've had two fail on me in the past six months, though one of them had the decency to spit out Smart errors so that I could proactively replace it (the other, by a stroke of cosmic luck, was empty).  I just don't trust anyone anymore.  If it wasn't for the Smart-auto-evacuate feature, I would've lost GBs and maybe even TBs of files due to failures.

And I still see the occasional drive dropout, which is beyond vexing, as I don't know if it's enclosure- or drive-related.  A restart is necessary to "resurrect" the drive so that DP recognizes it.  It doesn't happen often - maybe once every few months - but it's teeth gnashing when it does occur and makes me think dark thoughts.

I suppose I need to seriously consider duplication, which I've avoided my entire computer life due to the byte tax.  I have the two Terramasters, so designating one as the primary and the other as the backup should be easy.  Problem is investing around $1000 in sufficiently large drives to make it happen.  My 401(a) just took a GIGANTIC hit recently with the stock market trying to find the center of the Earth, so there's no relief there (if anything, I should be putting money in to shore it up).

Still, I could put three or four drives on my Amazon wishlist in the hope of pulling the trigger at some point in the not-too-distant future.

[Christopher (Drashna)]

Honestly, the low reliability was due to the floods, and then one specific line of drive (ST#000DM001) that was the issue.  Aside from those, Seagate drives have been rock solid for me (and IIRC, BackBlaze statistics bears that out too).  That said, I do use a mix of brands.

As for the drives, that sucks.  And if they were from the same batch of drives, that definitely makes sense (unfortunately).   But for any drive really, it's not "if" it fails, but "when".  And maybe ... how catastrophically. 

As for the dropout, if that is USB, then it's USB.  That's normal (and IIRC, allowed by the specs for USB).  Unfortunately.

And yeah, I hear you on the drives.  It's the only reason I don't have duplication for most of my pool.  Only the critical stuff is duplicated.   The drive capacity keeps in increasing and prices gradually drop (though, lets hope that continues, all things considered).    And IMO, it's worth getting the NAS/enterprise drives.  Between warranty and features/use-case, it's worth it.

[fleggett1]

I think I actually have a Seagate or two in my Terras, but I need to double-check.  Didn't the floods also affect WD pretty badly?

Regarding reliability, just so I'm clear, I know a drive that costs a little north of $300 won't last forever, especially if it's being written and read from almost every second of the day.  Like I wrote earlier, I'd happily pay a handsome sum for a high-capacity drive that genuinely lasted, at a minimum, 8 years.  It's the disposability/e-waste that annoys me, as people have now been conditioned to believe that hard drives should only last a few scant years before going bad.  For those of us distributing (ahem) numerous Linux distros, it's become a chronic fear that an un-duplicated drive might implode at a moment's notice, even if that drive has only seen two or three years of use.

It's kinda like what bios updating has become.  Before, the necessary blocks would be written, you'd see the progress bar, and it only took a minute and a single reboot.  Now, updating a bios is like a game of Russian Roulette, with numerous mystery reboots and highly disconcerting black screens throughout.  It's like the technology has gotten so "advanced" that it's become proportionally brittle.

Regarding the USB dropout, there's a power setting that I always turn off called "USB selective suspend setting", as it sounds like a definite no-no, but I dunno if it does what I think it does.  I also make sure to disable any sleep settings.

I think I am headed in the duplication direction, much to my chagrin.  It's more money, more hardware, more heat, more electricity, and more software complexity, which makes me highly uneasy, even though it's suppose to do the opposite, but it's the only thing I know to do to combat all these drive failures.

[Shane]

There are some SSDs with 10 year warranties, but SSDs are still a lot more expensive than HDD so...

I hear you on the brittleness of UEFI.

A midway option between N disks (no duplication) and 2N disks (2x duplication) that I've occasionally considered (but haven't really had a pool big enough to make it worth buying the cards/hubs) is pooling RAID6 arrays instead of using duplication with individual disks, basically offloading the redundancy onto the hardware.

Con: Needs one or more RAID cards (or RAID-capable DAS boxes), it's not full duplication, any given array would be vulnerable while rebuilding (though I have backups) and card management means yet another UI pane to deal with.

Pro: An array only needs 6+ disks to store 4+ disks worth of data (more disks = better ratio but more time to rebuild) and 3 disks would have to die concurrently to actually lose any data, offers a boost to read performance and RAID6 allows detection and repair of bitrot.

[fleggett1]

SSDs also don't have the capacity that I need (at least, not in the consumer space).  I'm talking at least 18 TB and more, if economically rational.

I feared RAID would come into the convo.  To expose my ignorance, I know very little about the tech and what I have gleaned scares the bejeezus outta me.  I keep hearing the mantra that "RAID is not a backup" and have read stories of entire arrays going south because a drive or two failed and the wrong strategy was employed for a rescue.  And then there's really bleeding-edge stuff like unRAID and filesystems with RAID-type features built-in, like zfs.

I just want Plain Jane NTFS so that, if something goes wrong, the rest of the pool isn't dragged into the abyss or needs a rebuild that may (or may not) succeed after a few weeks of drive hammering.  I also VERY MUCH like the fact that I can pull a drive from my DP pool, stick it in another Windows machine that doesn't have a clue what Drivepool is, and I can access all the files normally, without additional steps, and free from some exotic RAID implementation.  IOW, I like - no, NEED - a solution that's straightforward and doesn't descend into insane complexity.  I'm old and any new tricks I'm forced to learn have to be relatively simple and something I can visualize in my creaky, calcium-riddled brain.  This is why DP has fit all my criteria, as it does what it needs to do to configure the pool, present it to Windows as such, and that's it.

(I know the software is far more sophisticated under the hood than what I'm making it out to be, but I'm just speaking from the perspective of the "user experience".)

Besides, isn't hardware-specific RAID obsolete at this point given the raw power that modern machines offer, even low-spec boxes?

[drsalty]

@fleggett1 What was the virus you downloaded? How did you detect it?

[fleggett1]

I can't remember the specific virus, but it was downloaded via a torrent.  It was detected by Windows itself and quarantined, but I don't know if it had a chance to deploy before then.

I honestly don't know what caused my problems at this point.  It could've been DP, the virus, or a badly coded forced update.  I think it's really, really, REALLY weird that, while the machine was attempting to shutdown, it was obviously heavily writing data somewhere, as my drive activity light danced the entire time.  And the activity light wasn't just pulsing a meaningless repeating pattern, but acted like what it was writing had purpose and should not be interrupted, which is why I left it up for as long as I did.

Oh, I had another drive go belly-up on me three days ago.  A 10 TB shucked WD.  Problem is I might actually have to send this one in for data recovery, as I suspect it had some pretty unique data.

In case I wasn't clear before, I am NEVER NEVER NEVER relying on shucked drives again.  Those things can go to hell as far as I'm concerned.

Is there a feature that could be implemented into DP that would periodically generate a directory/file listing of each pooled drive and save it somewhere?  It would be a helluva diagnostic tool when a drive commits spontaneous seppuku.

[drsalty]

Ah okay. My system was having similar problems to yours but I simply don't know what's causing the issue... Virus is possible although nothing is being detected by Windows. Drivepool + my DAS was working great for weeks then all of a sudden all of my drives started dying. Bad sectors on everything, even brand new drives. Since no viruses show up in scans, I have to assume it's something in the software or hardware. I'm housing my drives in a Terramaster D6-320, pooled in Drivepool on my Beelink S12 Pro. The only other things it could possible be is I use a USB Hub switcher, but obviously the DAS isn't hooked up to that. I've noticed issues with USB devices when using the switcher, but I'd imagine a USB device not connected to the HUB should be unaffected by the switcher. I've contacted Beelink, Stablebit, and Terramaster and nobody is actually helpful or wants to take accountability, and everyone just points the finger at someone else. I'm just going to move away from this system all together. I'm returning everything, the drives, the DAS, the Beelink, and I'm just going to build my own NAS and run Unraid. Should have done this from the getgo tbh. 

[fleggett1]

I completely understand being soured on the whole affair.  I'm seriously thinking of dumping my pool altogether and just using two or three spinners that I can fit within a standard-size case.

Once drives start to go south and you permanently lose data, it calls the whole enterprise into question.  And it gets even worse when a drive fails so spectacularly that not even an evacuation is possible, as was the case with my latest dead disk.  I tried running dmde on it and, even with a complete surface scan, it failed to find anything salvageable.  This is why it's probably going to have to be sent in for data recovery, which I know can run a thousand dollars (or more).

This is also why it'd be fantastic if DP implemented that per-drive directory listing feature I mentioned earlier, as at least I would know what was lost and if it's worth re-sourcing.

If I were to try this whole sorry business again from the very beginning, I'd probably get a huge case that could hold 10-15 drives and throw as many fans at them that I could.  No more external enclosures with all their quirks.  OR I would pay an insane amount for a true NAS, as they seem to cool drives effectively and have evacuation strategies baked-in.  However, since I know very little about them, I'd lean toward the "big case" route first.

And no, that would NOT mean getting a server case.  I've tried that in the past with Norco cases and they just need more cooling than my ears can abide, like true Delta fans running at full throttle 24/7.

I'm curious, though - with your time with the Terras, what did you think of them?  Did you ever have to update their firmware?

[Shane]

If you have all your poolpart disks mounted to a path under the same folder, e.g. e:\disks\, then you can use the Windows Task Scheduler to set a daily task that runs a command along the lines of dir e:\disks\ /s /b > "e:\pooldirs%date%%time%.log" or similar (if you've got a LOT of files per disk, you might want to do some parallelisation).

Note that if your date and time format involves slashes or colons you'll need to use something different, e.g. %date:/=%%time::=% to strip those characters.

[fleggett1]

All my pooled disks are un-drive-lettered, so it's impossible (at least to my knowledge) to reveal their individual contents with a simple "dir" command since there's nothing to latch onto for the dir argument.  Maybe there's a more exotic method, like "dir \\PHYSICALDRIVE3 /S", but I've never tried that.  At the end of the day, I'm still just a dumb Windows user with just a small sprinkle of dangerous commandline/terminal experience.

And remember that I'd want the report to separate the file contents based on the actual disk they reside (or resided) on, not just a massive dir listing of the entire pool absent disk context.

This is why I'd prefer it to be an option inside DP since the software obviously knows how to address letter-less disks.  It would also be better in that DP could run the drivelisting report immediately after discovering that a disk is exhibiting extremely troubling issues.  This one logging feature would've been an absolute boon in my situation that happened only a few days ago, as I still don't know what was on that drive even with dmde.

I hope that makes sense.

[Shane]

In Windows Disk Management (which has a GUI) you can choose to mount a drive to a path instead of to a letter (or both). The advantage of using paths instead of letters is that this lets Explorer (and any other file manager) access any number of drives instead of only 26.

For example say you have a folder "c:\mount", you can create a bunch of empty subfolders within it called "disk1", "disk2", etc and then in Disk Management you add each drive in your pool to the subfolder you created for it (doesn't have to be disk1, etc, it could be the serial number of each drive or any other distinctive method. You can then browse those drives via those subfolders.

Then (for example) dir c:\mount /s /b > c:\mount\list.txt would output a file containing a sorted-by-disk list of all your files on those drives:

   c:\mount\disk1\poolpart.A\doc.txt
   c:\mount\disk2\poolpart.B\apple.jpg
   c:\mount\disk3\poolpart.C\video.avi

If you wanted a separate file for each drive you could create a batch that did separate dir /s /b commands for each disk.

Just don't put the mounting folder in your pool drive (e.g. if your pool uses P: don't use P:\mount) as that risks an endless loop.

Otherwise, to have DrivePool save file tables per disk as a built-in option, you could try submitting a feature request via the Contact form.