What is the cause of delayed uploading?

[steffenmand]

Hi,

 

Just another small question.

 

Why does upload not start while copying?

 

Currently it only starts when the copy is complete or the disk is full and it has to free space for the rest of the data.

 

Tried transferring 600 files (1TB) and it would've been great that it was uploading while i was copying the files

 

[Christopher (Drashna)]

What OS is this for?

What Version of StableBit CloudDrive is this?

What provider is this in regards to?

 

 

 

And depending on what exactly is going on, this *can* happen. The chunks that contain the file system info may not upload if there is constant activity, as it's being constantly changed.

However, the rest *should*.  But you can see the active upload/download threads in real time. Click on the "gear" icon, open the "Troubleshooting" section, and click on "I/O Threads" and you can see what exactly it's doing. 

 

If it's not actively uploading, then try enable drive tracing, and reproduce the issue:

http://wiki.covecube.com/StableBit_CloudDrive_Log_Collection

[steffenmand]

Windows 10

.442

Google Drive

 

You can see a very few upload threads now and then, but it doesn't start up completely.

I'm copying 5000 files at around 1 TB total size, so i imagined that it would start uploading the first file after it completed

The file info only uploads when the copy is complete?

 

Will try to do as asked

[triadcool]

I have been experiencing the same thing.  When copying a large amount of data to the Google CloudDrive the "To Upload" field lags behind what is being copied to the cache and almost nothing will upload or download from the clouddrive until the "To Upload" field catches up after all of the files are copied to the cache.  If I pause a file transfer half way through, I can see the "To Upload" working on catching up to where it should be for a few minutes then the upload starts up again.  If I resume the transfer again, almost all upload and download activity stops until the "To Upload" catches up.

 

I am running Windows Server 2012 with Stablebit CloudDrive version 1.0.0.422

 

100MB Chunks

1Gbit Connection

[Christopher (Drashna)]

I think steffenmand has already upload logs to us. If so, then there is definitely an issue there (if it's the issue I think, it's related to a "invalid mime type" error). 

 

Otherwise, if you haven't already, please upload the log files, so we can take a look at exactly what is going on:

http://wiki.covecube.com/StableBit_CloudDrive_Log_Collection

[modplan]

I see this, but it is only when the source of the files, and the cache drive are the same drive. Basically this causes this queue length on that physical drive to shoot way up, causing IOPs to suffer, and I assume the drive is just too slow for Cloud Drive to be able to read and upload. As soon as the transfer is finished and the queue length drops back down to normal levels uploads resume at normal speed.

 

Probably nothing Cloud Drive can do to prevent this in this particular case.

[steffenmand]

Not the same drive in my case

[Christopher (Drashna)]

I see this, but it is only when the source of the files, and the cache drive are the same drive. Basically this causes this queue length on that physical drive to shoot way up, causing IOPs to suffer, and I assume the drive is just too slow for Cloud Drive to be able to read and upload. As soon as the transfer is finished and the queue length drops back down to normal levels uploads resume at normal speed.

 

Probably nothing Cloud Drive can do to prevent this in this particular case.

 

I see this, but it is only when the source of the files, and the cache drive are the same drive. Basically this causes this queue length on that physical drive to shoot way up, causing IOPs to suffer, and I assume the drive is just too slow for Cloud Drive to be able to read and upload. As soon as the transfer is finished and the queue length drops back down to normal levels uploads resume at normal speed.

 

Probably nothing Cloud Drive can do to prevent this in this particular case.

 

If that's the case, then yeah, there may not be much we can do to fix the issue. There are some optimizations we could do though, so it's worth looking into at least.

 

If you could, could you do this:

http://wiki.covecube.com/StableBit_CloudDrive_Log_Collection

[modplan]

If that's the case, then yeah, there may not be much we can do to fix the issue. There are some optimizations we could do though, so it's worth looking into at least.

 

If you could, could you do this:

http://wiki.covecube.com/StableBit_CloudDrive_Log_Collection

 

I have no problem with this happening since I understand it is likely a hardware limitation, i don't expect cloud drive to perform miracles Maybe you meant to quote steffenmand and tell him to collect logs?

[Christopher (Drashna)]

I have no problem with this happening since I understand it is likely a hardware limitation, i don't expect cloud drive to perform miracles Maybe you meant to quote steffenmand and tell him to collect logs?

Well, if you can grab the logs when the slowdown is happening, this is a detailed kernel level log.  So it is possible that the log may help us optimize how we access the data, to help prevent bottlenecking like this.  

 

It's a long shot, basically. But if it helps, it's worth it, right?

 

 

Also, in case it matters, using an SSD for the cache drive isn't a bad idea. In some tests with that configuration, Alex was seeing in the neighborhood of 500MB/s for the writes to the cache.  But the price per GB is much higher than a HDD, and the size is usually much slower. But hey... it may be worth it in some cases.