Jump to content
Covecube Inc.
  • 0

Files deleting themselves..


darkly

Question

I probably won't be able to reproduce this anytime soon, but some directories entirely disappeared from my drive. I think they were files in the upload queue, but I can't be sure. I had 1TB+ in the upload queue this morning before I got API banned with around 600GB left. It's been sitting at this for a while now until the upload can resume, but I was just accessing a few directories here and there, and found that some folders were completely missing. I know every folder was present when I first completed the file transfer (not upload) because I checked the properties of the entire directory against the properties of the original source and they had the same size and number of folders/files.

I'm not going to have this much  in queue any time soon so it's going to be rather difficult to replicate.

 

 

EDIT

I've been rescanning my library in plex every day and I keep seeing random files disappearing. 4 episodes from one season of a show, the first episode of a season of another show, 3 entire seasons of another show, etc. I confirmed the total number of files copied immediately after copying, so somewhere in the time it's sitting in the upload queue to when it finally gets uploaded, files are getting deleted.

 

EDIT 2
I sent in a file system log. I know the instructions say "It is important that you perform this step and perform it quickly after reproducing the issue" but as I don't know what triggers the issue or when it happens, my only choice was to leave it running for a while. Hopefully it captures something that catches your eye. I've had a handful of files disappear again today as well.

 

EDIT 3

Just adding that I know for a fact it's not just files in the upload queue anymore. Some of the ones that disappeared were from my first batch which I didn't only copy over, but also let upload completely before verifying that I had the same number of files as the original.

Link to post
Share on other sites

Recommended Posts

  • 0

929 beta on gdrive.
as for triggers, basically this whole post I've been talking about files disappearing when I remount after clouddrive glitches/bugs force me to restart the server to make the service and UI operational again. I'm now trying to move the remaining files off the drive and I've run into ~10 such files so far that throw that error while copying.

Link to post
Share on other sites
  • 0

Okay.  Thank you for clarifying/confirming that.

 

Unforutnately.... because this appears to be happening "offline", it is going to make it a lot more difficult to track down the issue, and identify the cause.   Just simply because of how/where the issue is occurring.

 

Worse yet, is that this may simply be a server side issue, where it's modifying the data on the server side.  And if that is the case, there there isn't a lot that we can do.

 

However, I don't think that this is the case, as we'd have a lot more people with this issue occurring, rather than... what seems to be "just you".  

 

 

That said, we do have a number of in house tests for this sort of stuff (specifically, data consistency tests).  We run these for public releases, to ensure that nothing "funky" is going on.  

 

 

I've flagged the issue for Alex (the Developer) as a critical one, and we'll see what we can do.

 

 

And if you could, enable tracing and web tracing, and attempt to reproduce the issue (aka, detach and reattach a drive) 

http://wiki.covecube.com/StableBit_CloudDrive_Drive_Tracing

http://wiki.covecube.com/StableBit_CloudDrive_Web_Logs

 

Additionally, in the meanwhile, please run the StableBit Troubleshooter on the system in question.

Use "3160" as the Contact ID for the uploader, and let me know when that's done. 

 

 

 

 

In the meanwhile, please throttle the Google Drive accounts to 50/25mbps, when they are attached, and try increasing the minimum download size. 

 

 

https://stablebit.com/Admin/IssueAnalysis/27654

Link to post
Share on other sites
  • 0

Okay.  Thank you for clarifying/confirming that.

 

Unforutnately.... because this appears to be happening "offline", it is going to make it a lot more difficult to track down the issue, and identify the cause.   Just simply because of how/where the issue is occurring.

 

Worse yet, is that this may simply be a server side issue, where it's modifying the data on the server side.  And if that is the case, there there isn't a lot that we can do.

 

However, I don't think that this is the case, as we'd have a lot more people with this issue occurring, rather than... what seems to be "just you".  

 

 

That said, we do have a number of in house tests for this sort of stuff (specifically, data consistency tests).  We run these for public releases, to ensure that nothing "funky" is going on.  

 

 

I've flagged the issue for Alex (the Developer) as a critical one, and we'll see what we can do.

 

 

And if you could, enable tracing and web tracing, and attempt to reproduce the issue (aka, detach and reattach a drive) 

http://wiki.covecube.com/StableBit_CloudDrive_Drive_Tracing

http://wiki.covecube.com/StableBit_CloudDrive_Web_Logs

 

Additionally, in the meanwhile, please run the StableBit Troubleshooter on the system in question.

Use "3160" as the Contact ID for the uploader, and let me know when that's done. 

 

 

 

 

In the meanwhile, please throttle the Google Drive accounts to 50/25mbps, when they are attached, and try increasing the minimum download size. 

 

 

https://stablebit.com/Admin/IssueAnalysis/27654

Reproducing the issue is hard. It seems to happen when I have to force clouddrive to turn off by restarting the computer (due to it being unresponsive), or when clouddrive closes on its own for other reasons and I remount. I have not seen the issue occur just from dismounting and remounting the drive. That itself creates another issue in that, the only way I know to reproduce the issue is the actually close out of clouddrive entirely, which ends the logging.

 

I mentioned this somewhere else, but I *am* running windows server 2012 (not r2). I remember reading somewhere that this used an earlier implementation of ReFS, before it was updated for windows server 2012 r2. Could this have anything to do with this? As I've said, my other two drives have (to my knowledge) never lost data, and they're both NTFS.

 

I'll try to get some time to do some of the other things you suggested as well.

Link to post
Share on other sites
  • 0

Thank you for the additional information. 

 

That said, the "unsafe shutdowns" may actually be the culprit here.   

Any time that the system isn't powered down properly, it can cause issues.  Namely, because of write caching.  There are several layers of write caching that can be affected here.  Namely software (within Windows itself) and hardware (the disk itself).

 

The first one is why we have the "native" (aka) shutdown service. This stops the system from shutting down until all cached data has been written to the disk.   

If this fails to happen, it can cause data corruption.  In fact, this is exactly how/why disks show up as "RAW". The corruption happens to the volume information, rather than to a file. 

 

 

 

As for ReFS version, no, this should be "fine". There isn't a big difference between the Server 2012 to 2012R2 versions (the biggest thing was Alternate data stream support). 

 

However, it could be something to do with ReFS, since it's a copy on write file system, meaning that modifications actually copy the file, rather than simply modifying a part of the existing file.  This change could be why you're seeing the issues on ReFS and not NTFS (since it would be faster, IIRC).

 

 

I was going to mention upload verification, but it sounds like that wouldn't help here anyways. But if you want, feel free to do so. The caveat is that this setting uses more bandwidth, which is a problem for Google Drive. (since it downloads and verifies the data after it uploads it).

Link to post
Share on other sites
  • 0

Thank you for the additional information. 

 

That said, the "unsafe shutdowns" may actually be the culprit here.   

Any time that the system isn't powered down properly, it can cause issues.  Namely, because of write caching.  There are several layers of write caching that can be affected here.  Namely software (within Windows itself) and hardware (the disk itself).

 

The first one is why we have the "native" (aka) shutdown service. This stops the system from shutting down until all cached data has been written to the disk.   

If this fails to happen, it can cause data corruption.  In fact, this is exactly how/why disks show up as "RAW". The corruption happens to the volume information, rather than to a file. 

 

 

 

As for ReFS version, no, this should be "fine". There isn't a big difference between the Server 2012 to 2012R2 versions (the biggest thing was Alternate data stream support). 

 

However, it could be something to do with ReFS, since it's a copy on write file system, meaning that modifications actually copy the file, rather than simply modifying a part of the existing file.  This change could be why you're seeing the issues on ReFS and not NTFS (since it would be faster, IIRC).

 

 

I was going to mention upload verification, but it sounds like that wouldn't help here anyways. But if you want, feel free to do so. The caveat is that this setting uses more bandwidth, which is a problem for Google Drive. (since it downloads and verifies the data after it uploads it).

What I don't get is, some of the data disappearing is data that has been on this disk for AGES. So it's definitely been written to the cloud. In fact, almost all of it is data that had been sitting on there fully uploaded since the beginning.

Link to post
Share on other sites
  • 0

yeah, that's the thing.  If this is the case, then the likely culprit is that Google (or whomever) may have corrupted the data, ON THE CLOUD.

 

 

I've talked with Alex about this directly, and ... apparently, in some cases, Google will restore older versions or restore deleted files.  And it can cause issues (it's exceptionally rare, and our software is designed in such a way that this shouldn't really happen, but ....) 

 

So, he is aware of the issue, and we'll see about looking into this in more detail. 

Link to post
Share on other sites
  • 0

If you're still seeing this, it looks like Alex may have identified the cause of this issue (via another ticket, with an unrelated error), so the latest beta *should* fix this.

 

http://dl.covecube.com/CloudDriveWindows/beta/download/StableBit.CloudDrive_1.0.2.947_x64_BETA.exe

Link to post
Share on other sites
  • 0

Just wondering, since I run into this a lot when I need to reboot for troubleshooting (such as I need to do now to install that update), I've noticed that if there is an upload queue when CloudDrive shuts down (unsafe shutdowns), it will try to resume uploading when it starts up again. Is there a way to safely shut down cloud drive while there is an upload queue (because by default it waits til all items are done uploading when you try to detach a drive) because it's not the most convenient thing to have to wait for a 500GB upload when I need to restart my server :P

Link to post
Share on other sites
  • 0

Actually, it's not uploading data when it shuts down.  So it's not uploading that 500GB of data.

 

However, what it is doing is waiting for data to be flushed from the cache to the disk.  Since .... that happens twice (once to the cloud drive... and then that data to the cache drive), it can take a while.  But we don't want to interrupt this, since doing so could/would corrupt data. 

Link to post
Share on other sites
  • 0
On 10/22/2017 at 1:03 PM, Christopher (Drashna) said:

If you're still seeing this, it looks like Alex may have identified the cause of this issue (via another ticket, with an unrelated error), so the latest beta *should* fix this.

 

http://dl.covecube.com/CloudDriveWindows/beta/download/StableBit.CloudDrive_1.0.2.947_x64_BETA.exe

So this version should fix the issue of files disappearing? I installed this while ago but I've been swamped by other things since. Unfortunately at this point, it's become rather difficult for me to see if further things are deleted, as the last cycle of deletions took out roughly half my files and I've still not identified everything deleted from that incident.

Link to post
Share on other sites
  • 0
On 11/8/2017 at 1:01 PM, darkly said:

So this version should fix the issue of files disappearing? I installed this while ago but I've been swamped by other things since. Unfortunately at this point, it's become rather difficult for me to see if further things are deleted, as the last cycle of deletions took out roughly half my files and I've still not identified everything deleted from that incident.

It fixes an issue that could cause that, yes. 

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...