Understanding some things

[steffenmand]

Hi,

 

Just had yet another power surge yesterday (sad face) and i booted up with the drive going into recovery mode, which after some time turned out fine.

 

However i now see CloudDrive doing constant reads of a chunk and afterwards uploading that chunk. It can often do the same chunk several times untill the chunk id eventually goes up 1 id and all continues again.

 

This might go on for hours and then it suddently starts uploading properly.

 

It's worth noticing that while this goes on, nothing is being uploaded to the drive.

 

After stuff uploaded, i started copying new stuff to the drive and now it's started the read/write stuff yet again and i most likely will have to wait for hours again.

 

While it happens i always see a little bit cached (atm. 205 MB) although i set the cache drive to None while uploading (moving to a different drive after upload is done)

 

Is this intentional behaviour ?

 

The other day i even had "download quota of that file exceeded" which i posted logs to you guys about

[Christopher (Drashna)]

Are you running a CHKDSK pass on the CloudDrive drive? If so, that could be the cause of what you're seeing.

 

Also, I do know there are some integrity checks, to make sure there isn't any corruption, I believe. 

 

 

 

Additionally, what OS is this for, what version of StableBit CloudDrive are you using, and what Cloud Provider is in use here?

[steffenmand]

Version: .444

OS: Windows 10 Pro 64 bit

Provider: Google Drive

 

Did not start chkdsk myself - does Cloud Drive do this ?

 

It kept going the entire night untill Google gave the error "The download quota for this file has been exceeded". After that the drive stopped working as it was unable to do what was needed. Since the threads kept trying that error never went away (guess it resets the wait time every time the thread tries to download it again)

 

But in general, it seemed like it was going through each and every chunk and some got downloaded and uploaded TONS of times.

[steffenmand]

I'm seeing this on a completely new drive as well as soon as i started transferring to a drive.

 

I don't have upload verification on

 

 

After those writes finish, most likely a new read will happen perhaps followed by a new write of the chunk. Sometimes they move up an id.

 

What's the point with all these constant read/writes of the same chunks? They must use a ton of api calls, which results in these "The download quota of this file has been exceeded" leaving the drive useless untill the next day

[modplan]

I've seen this with large chunks, I think when existing data had changed, not with new data. 

 

Here was my theory: Chunk 10 has some data that has changed. To verify that we do a partial read on chunk 10 at offset X. The data has indeed changed! So Cloud Drive re-uploads chunk 10. Wait a sec, chunk 10 has more data that has changed! We do a partial read of chunk 10 at offset x+1, see the data is different, upload chunk 10 AGAIN. Then do a partial read of chunk 10 at x+2, upload chunk 10 AGAIN, etc etc until all of the 1MB blocks in Chunk 10 that have changed have been updated and chunk 10 has been uploaded over and over.

 

If this is indeed the case, it would be wildly more efficient to calculate the new chunk 10 before uploading any data and upload it a single time. 

 

Windows Server Backup caused the above to happen over and over and over to the point that in a whole 24 hours "to upload" would only drop by about 1-2GB but I had really uploaded over 200GB! For that drive I had to go back to 1MB chunks, which solves the above issue. For my streaming media cloud drives, the data never really changes, just new data is added, so I am able to still use 50MB chunks and do not see the above issue.

[steffenmand]

I'm using 100 MB chunks, so this might be the case - and data might have changed! Maybe the fix with larger or no partial(full download) reads will fix this as well for large chunks.

 

Of course it should update the data, but yeah it should not work on the same chunk tons of times!

[Christopher (Drashna)]

Anything that indexes the files may cause it to re-download the files. 

 

However, do you have a defrag program installed at all? And if so, have you excluded the CloudDrive?