[Bug?] Prior scan data and settings not preserved on update

[Jaga]

My server picked up on the newest version of Scanner this morning (2.5.3.3191), so I let it auto-update.  At the same time I decided to run a manual update (download/install) on my primary workstation.  Both completed successfully, but both also ran into an issue.  None of the prior scan data (either file system or surface) was preserved - all drives that both copies of scanner monitors on each computer were reset to unscanned.  I manually marked all unchecked blocks good, but some people may not know how, or want their old data/schedule preserved.

Server is running W7 Ultimate 64-bit (SP1), workstation is running W10 64-bit v1709 (build 16299.431).

[Christopher (Drashna)]

http://dl.covecube.com/ScannerWindows/beta/download/StableBit.Scanner_2.5.4.3246_BETA.exe

WIth the help of one user, we were able to pinpoint the issue, and this shouldn't happen in the future.

Specifically, the issue was:
 

Quote

.3246
* DiskIds were being updated too frequently (every second).
    - If your system crashed during a DiskId write, there was a chance that the DiskId file would get corrupted which 
      would lead to a settings loss for that disk (scan history, disk name / location, etc...).

[Mathematics]

Same thing happened to me, now twice. There were two updates within the last week, so it seems. Every version of Scanner was the latest at every point of the software running on all my systems. Both times, the previous scan data, after the update, was purged. I don't see this to be such an essential problem but I'm just reporting the observation.

[Jaga]

32 minutes ago, Christopher (Drashna) said:

• The new JSON format for the setting store (located in "C:\ProgramData\StableBit Scanner\Service\Store\Json") should be much more resiliant. 
  In fact, I tested out copying the folder elsewhere, nuked the settings, and restored the JSON folder.  I did this on my production server, and can verify that it works.  So, if you backup the folder in question, you setting should be safe, from here on out.

Might be a nice option to have an import/export function for the settings in the various products.  Primocache and Primo Ram Disk have it, and I use them when re-creating configs all the time.  It has saved my bacon more than once.  Just bundle the JSON folder into a .zip and allow the user to specify export destination.  On import, unbundle and write (with optional overwrite verification).  Just a suggestion.   

[Christopher (Drashna)]

Depends on which version you're coming from. 

There was a big change in how the settings are stored (from our internal encrypted structure to mostly JSON).  So there is a conversion that is needed.  

Unfortunately, that does lose some settings (I personally suspect that it's because the files are partially corrupt), and is very ironic, because we made this change to avoid this sort of issues in the future.   So .... this shouldn't happen anymore, at least.

 

Also ... this is a known issue, but it's pretty rare, and ... we've not been able to successfully reproduce the issue, so fixing it is ... hard.

 

Edit: 
Marking this as the "best answer" for now, so that people can see my response. 
Additionally, I've talked to Alex directly about this issue, so we can look into it further, just in case. 

Edited July 9, 2018 by Christopher (Drashna)

[Mick Mickle]

This is very similar to the issue I raised at the thread below.  And once again, I updated my WS2012R2 Scanner 2.5.1.3062 with what I thought was the latest release version 2.5.2.3190 only to have the result be the loss of scanning history, individual disk scan information, and individual disk settings, including, unique names and bay locations.

I see by your post that the release replacement 2.5.2.3191 doesn't fix that problem.

I'm guessing those settings are in \ProgramData\StableBit Scanner\Service\Store, which now uses json extensions. Maybe there's a way to copy backup files from this folder back into it.

 

[Christopher (Drashna)]

If you have Previous Versions enable on the C:\ drive, or a backup of it.... you could stop the service, and restore the data and see if it works better. 

 

But if this is what I think it is, it won't help. 

[Mick Mickle]

A few years ago, I requested a basic feature to print the Scanner UI and disk details to use as a record of disk locations, details, and settings. (I don't think that feature was ever implemented.)  Such a printout could be used to manually recreate disk settings after an update that loses that data, although it wouldn't help with specific disk surface scan results from the prior version.

While writing this post, I reviewed the interface of 2.5.2.3190 and noticed that there is one disk, and only one, that retained its name, location, and health status. I think that I may have input that information more recently, while all the others haven't had disk details changed in a long time. So maybe Christopher's theory about corrupt data may be on track. Perhaps the data for that particular disk refreshed the relevant data file when saved, while the data files for the other disks remained corrupted or otherwise incompatible with the upgrade process.

I'll be restoring 2.5.1.3062, then I'll refresh the disk settings for each disk by making a small change and saving the setting. Then, I'll upgrade to 2.5.2.3191 and see if that makes a difference.

 

[Christopher (Drashna)]

@Mick Mickle  Just FYI, my theory is based on ... a SMART value that triggered a warning, and then I got flooded with HUNDREDs of SMS messages, being sent over and over and over again (every second or so), until I ignored the SMART status.  Resetting the settings fixed the issue.... but ... well, I used the above method to restore the settings (for the reasons you mentioned, actually). 

When I upgraded that system, this disk was one of them that lost all of it's settings.  

I did fill a bug report for that issue, so Alex does have the data to look at.  But I'm pretty sure these two issues are directly related.  The problem is figuring out what caused it, when it happened, and how it happened.  But then again, the newer format should help prevent the issue from occurring in the future. 

 

[Mick Mickle]

41 minutes ago, Christopher (Drashna) said:

If you have Previous Versions enable on the C:\ drive, or a backup of it.... you could stop the service, and restore the data and see if it works better. 

 

But if this is what I think it is, it won't help. 

I did have previous versions working so I restored an earlier version of the \service\store folder, but no change in UI depiction.  I then stopped the service again and copied my earlier backup of that folder over it -- big mistake. Now I'm in a trial license/license transfer loop. Hopefully, uninstalling, populating those folder with backup, and reinstalling previous version as planned will go smoother. If not, expect a ticket.

[Mick Mickle]

Okay, went back to 3062, refreshed all disk settings with save, then upgraded to 3090 (forgot about 3091). Same problem. But I still have one disk that all details carry over on. If we can figure out the difference between that disk data and all the ones that have amnesia, maybe problem can be solved. What do you need?

[Jaga]

2 hours ago, Christopher (Drashna) said:

Depends on which version you're coming from. 

There was a big change in how the settings are stored (from our internal encrypted structure to mostly JSON).  So there is a conversion that is needed.  

Unfortunately, that does lose some settings (I personally suspect that it's because the files are partially corrupt), and is very ironic, because we made this change to avoid this sort of issues in the future.   So .... this shouldn't happen anymore, at least.

 

Also ... this is a known issue, but it's pretty rare, and ... we've not been able to successfully reproduce the issue, so fixing it is ... hard.

The version I was using on both machines prior to the upgrade was StableBit.Scanner_2.5.1.3062_Release.  So...  not very old.

Good to hear the schema is improving to guard against problems like this though.

[Kirk Bate]

Same as above.  Though I do not know which version was my last.  I can't imagine it being too much of a problem, just wanted to make it known.

[Mick Mickle]

I agree that "never scanned" resets aren't much of a problem, since Scanner will simply scan again. (It's weird that my server has just 1 of 11 disks that was still "Healthy" with all other correct settings after upgrade.)  Of more concern to me are all the lost settings -- names, locations, whether to scan automatically (which would be duplicative for pass-through disks if Scanner is installed on both host & VM), etc.  Those items have to be re-entered for customization to continue.

[jergo]

I'm having the same problem. I had the latest release, got a new update available notification and updated. After the update, I didn't reboot-it didn't ask me to. I opened the interface and saw that half of my 12 drives had forgotten their already scanned status and also the custom names I had given them. Individual settings on those drives were gone as well. I just shrugged it off and renamed the drives to my liking, set the settings for the individual drives the way I like, and let the "unscanned" drives re-scan. All was well until I rebooted this morning. Now ALL of the drives are listed as unscanned, ALL custom names are lost, as well as any settings. What's going on? I'm on Windows 7 64 bit

[GameGod]

I had the same experience. Upgraded from the last release version to the latest, lost all customization, and scanned status. The scanner immediately started a full scan on all my disks after the upgrade. Re-did the settings, rebooted and it seems okay.

[Mathematics]

Just reporting this is the third time previous scan logs were purged immediately after update. Is this intentional and supposed to happen?

[Christopher (Drashna)]

Well, I don't mean to minimize the issue, at all.... but this generally happens to a small group people (or at least, that's how many report it).   So it doesn't seem to be a wide spread issue.  Which ... means it's pretty rare, and ... that it's not as easy to track down. 

That said, 

• We've looked into this issue in the past, with little success
• Alex is planning a "deep dive" to look into this issue ( @Mick Mickle, I believe Alex mentioned that to you in the ticket, actually), and we've talked about this
• The new JSON format for the setting store (located in "C:\ProgramData\StableBit Scanner\Service\Store\Json") should be much more resiliant. 
  In fact, I tested out copying the folder elsewhere, nuked the settings, and restored the JSON folder.  I did this on my production server, and can verify that it works.  So, if you backup the folder in question, you setting should be safe, from here on out.

[Mick Mickle]

Thanks, Christopher. It sounds like the problem is solved moving forward from 3090 and 3091. Still to come is data export/print capability for specific disk details, settings, and locations, some of which will be available in the future in Stablebit Cloud, I understand.

[Mathematics]

The import/export feature would be nice. I guess right clicking on the folder and 7zip'ing it, is the definitive solution, for now, until an automated process evolves.

According to Christopher's answer that it seems to be an isolated incident, I'm wondering what is it about our particular systems that is causing this purge? 

I have it running on both W7 and W10 and it purges on both. Both OSs are clean installs. Both run the same EVO500...alongside a WD spinner. Both are Dell. It seems to me that the purge is triggered by some integral part of the software once it's updated. Like an auto purge feature.

I'll be honest, I think most people are too lazy to sign up and post the issue, which makes it appear to be isolated incident, but I believe this is happening more often than we think. 

I'm on a lot of forums, and it's always the same people that help developers address bugs, by reporting them. Unless it's a functional problem, it goes unreported.

All of you...know how lazy people are. With that said, I like the idea of an integral backup and restore of the settings. 

[Christopher (Drashna)]

21 hours ago, Mathematics said:

According to Christopher's answer that it seems to be an isolated incident, I'm wondering what is it about our particular systems that is causing this purge? 

That's the question.  I believed that I mentioned it in this thread, but my storage server experienced this as well.  10 of the drives lost their settings.  One of those drives ... had previously caused issues with sending out hundreds of SMS notifications.  Which makes me suspect that the settings store is/was getting corrupted somehow, in a way that wasn't stopping the data from being loaded.   

21 hours ago, Mathematics said:

I'll be honest, I think most people are too lazy to sign up and post the issue, which makes it appear to be isolated incident, but I believe this is happening more often than we think. 

Yup.  The general rule of thumb is that for every one person complaining about the issue, there are at least 3 others that are experiencing it, but not saying anything.

 

 

[Christopher (Drashna)]

22 hours ago, Mick Mickle said:

Still to come is data export/print capability for specific disk details, settings, and locations

Still, going to add it as a feature request here:

https://stablebit.com/Admin/IssueAnalysis/27880

[jergo]

Rebooted three times and all three times it has forgotten my custom drive names and scanned status. What can I do to correct this? I can't re-do everything after every reboot.

*EDIT*

Just thought of something...my Windows temp file has been remapped to a different drive, due to having a solid state drive as my C drive and being in the process of re-encoding about 1000 movies with Handbrake. I didn't want to wear out my solid state drive, so I moved the temp file to a different, regular drive. Could this be my problem?

Edited July 14, 2018 by jergo
thought of some more detail

[Jaga]

4 hours ago, jergo said:

Just thought of something...my Windows temp file has been remapped to a different drive, due to having a solid state drive as my C drive and being in the process of re-encoding about 1000 movies with Handbrake. I didn't want to wear out my solid state drive, so I moved the temp file to a different, regular drive. Could this be my problem?

The Scanner settings JSON files (all that I can find anyway) are in the "C:\ProgramData\StableBit Scanner\Service\Store\Json" folder, so it shouldn't have anything to do with the Windows temp folder locations.  You might want to check that directory and see if they exist in there, and check some of their file creation dates.  If (after rebooting) it is re-creating the files from scratch (or wiping the folder contents), you should be able to tell.

[Christopher (Drashna)]

Jaga is correct here. 

And if it's not storing the settings at all, then there is another issue here, and ... resetting the settings manually may fix the issue:
http://wiki.covecube.com/StableBit_Scanner_Q4200749

Eg, delete the entire "C:\ProgramData\StableBit Scanner" folder, and restart the service.

[jergo]

Just a quick update...the JSON files are located in the directory that Jaga mentioned. I manually entered all my custom drive names and settings one more time and did a reboot after the latest Windows update. The settings seemed to stick this time.