Jump to content
Covecube Inc.
  • 0

Upload speed when copying files


darkly

Question

Can someone explains why this happens?

Firstly, if I understand correctly, this is the process for uploading a file:

  1. I copy a file from a local disk to the virtual disk
  2. That file is copied to a cache stored on the local disk that queues the file for upload
  3. CloudDrive uploads to GDrive (my cloud provider) whatever is sitting in the upload queue

What I'm noticing is that when my upload queue has things in it and I'm not currently copying any more, I get upload speeds upwards of 600mbps, but when I start copying files to the virtual drive, the upload throttles down to 90-120mbps. What would cause this?

Using the latest beta.

Link to post
Share on other sites

8 answers to this question

Recommended Posts

  • 0

A couple of things could/would cause this.

 

There is upload throttling when adding new content.  This way, we're not trying to upload small changes, constantly.  There is a threshold it uses to halt it until a certain amount is uploaded.

 

Also, it may be the underlying disk. The more they're being used, the longer it takes everything.    This is especially true for spinning drives, and not so much for SSDs.  

 

 

Also, what version specifically are you using? 

Link to post
Share on other sites
  • 0

There is upload throttling when adding new content.  This way, we're not trying to upload small changes, constantly.  There is a threshold it uses to halt it until a certain amount is uploaded.

 

Wouldn't this only affect it if I had just started uploading? I'm talking about an upload queue that already has 50+GB of data in it, so it should already have plenty to group together and upload. By itself it'll upload at 600+mbps but if I start copying something else into the virtual drive, the speed plummets.

 

Also, it may be the underlying disk. The more they're being used, the longer it takes everything.    This is especially true for spinning drives, and not so much for SSDs.  

 

I'm not ruling this out, I just wanted to see if it was something others had reported before going through a bunch of i/o tests.

 

The exact version is 1.0.2.929

Link to post
Share on other sites
  • 0

I also forgot to mention, since upgrading to the beta, I've been seeing this error pop up several times throughout the day (usually quite a few times in a row before continuing to upload):

 

Cloud drive <my_drive_name> is having trouble uploading data to Google Drive in a timely manner.

Error: Thread was being aborted.
 

This error has occurred n times.

 

This error can be caused by having insufficient bandwidth or bandwidth throttling.

Your data will be cached locally if this persists.
 

--------

I never saw this before switching to the beta. Any ideas what causes this?

Link to post
Share on other sites
  • 0

There isn't anything that specifically should cause this to happen.  

 

However, if you have run into the bandwidth cap, it could cause this to happen.  

And likely, this is why.  So, just bad timing. 

 

Can I ask what information is collected in the logs?

 

The file system logging ... well logs that. What is accessed on the drive.  It's kernel mode logging, so it logs when the file was opened, accessed, closed, etc.  It doesn't show the data specifically, just what calls were made to the files on the drive. 

 

 

As for the web logging, which is really what the second link should be, sorry...

http://wiki.covecube.com/StableBit_CloudDrive_Web_Logs

 

 

This is exactly what it sounds like. It installs a temporary web proxy, and routes all of the system traffic through it. And I do mean *all*.

 

This includes your browser, Windows telemetrics, etc. Anything that touches the internet, really,  So, yes, it's very invasive, unfortunately.  

 

However, these logs are private (they're not uploaded automatically, ever), and we ONLY use these for diagnostic purposes.  

 

(and no judgement here... I mean .... I have almost 20TB of porn.... so, let he whom is without sin cast that first stone, because it sure as hell isn't me). 

Link to post
Share on other sites
  • 0

There isn't anything that specifically should cause this to happen.  

 

However, if you have run into the bandwidth cap, it could cause this to happen.  

And likely, this is why.  So, just bad timing. 

 

I'm not sure it's just this though, because I've hit the bandwidth cap before and the message is different It says nothing about the thread being aborted. And also, it happens rapidly like a dozen times but then continues with the upload. When I hit the bandwidth cap, I get another message entirely and I keep getting it for the rest of the day until the cap is removed and I'm able to continue uploading.

 

The file system logging ... well logs that. What is accessed on the drive.  It's kernel mode logging, so it logs when the file was opened, accessed, closed, etc.  It doesn't show the data specifically, just what calls were made to the files on the drive. 

 

 

As for the web logging, which is really what the second link should be, sorry...

http://wiki.covecube.com/StableBit_CloudDrive_Web_Logs

 

 

This is exactly what it sounds like. It installs a temporary web proxy, and routes all of the system traffic through it. And I do mean *all*.

 

 

So, if I'm understanding correctly, all information about files and their activity, as well as all web traffic? Fair enough, but is there a way (email?) to send this your way without slapping it on here for all to see?

 

 

(and no judgement here... I mean .... I have almost 20TB of porn.... so, let he whom is without sin cast that first stone, because it sure as hell isn't me).

 

I'm glad I wasn't drinking something. I think this is yours now. CS-badge.jpg

 

 

EDIT: I'm now getting the very distinct "user rate limit exceeded" error that will continue through the rest of the night until my ban time is up, unlike the "thread was being aborted" error which pops up throughout the day (sometimes in rapid succession) before allowing me to continue uploading .

Link to post
Share on other sites
  • 0

Well, as for the bandwidth stuff, that depends exactly on what's going on.  There are a number of ways that the throttling occurs. And it varies between providers, as well. 

 

 

 

As for the logs, yes.  If you check the bottom of the linked page, there is a button (link) for file submissions.  But here it is:

https://www.dropbox.com/request/lrCAeFa2EH9VSJqP3QLM

 

This uploads to our company dropbox account.  So only myself and Alex can access it. 

 

 

As for the lack of a drink, good thing then. :)
And I specifically mentioned it, because we've had a few customers that didn't want to upload logs for ... well this reason. I mean, I totally understand, but ... :)

 

 

As for the "User Rate Limit Exceeded", that's the common throttling issue popping up lately.  This is because Google implemented a rather aggressive bandwidth cap somewhat recently.  It looks like 750GB per day, is the limit.   That's .... 75mbps for a 24 hour period.   Downloads continue, but uploads are halted for a full 24 hours after hitting the limit.  

Setting a bandwidth limit may help ("Manage Drive" -> "Performance" -> "I/O Performance") to prevent hitting this in the future, but increase how long it takes to upload the data...

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...