Jump to content
Covecube Inc.
  • 1
Bowsa

Slow "Copy" transfer rate?

Question

I'm having trouble understanding this.. I have an NVME with insane read/write speeds, but when I copy a file to the Cloud Drive it's my understanding that the following happens:

  • File gets copied to local cache
  • File must be copied in its entirety for anything else to happen
  • File then gets encrypted/broken into chunks
  • File gets uploaded?

I'm not sure if these are the correct order of steps

But if the first step is the case, and I'm copying a file to the local cache first..why is it so slow?

The transfer rate is 30 MBs from my disk to local cache..but my NVME is 3200 MB/s Read, 1900 MB/s Write..so what gives?

What possible justification is there for this slow transfer speed if it's not even uploading yet.

Share this post


Link to post
Share on other sites

Recommended Posts

  • 0

Yes and no.

As soon as you write the file to the drive, the driver both encrypts the data, and breaks it into chunks.  Those encrypted chunks are written to the local cache, and then "processed".  Meaning they're uploaded, or queued for upload. 

As for your read and writes, you're listing the sequential read and write speeds.  Which is at least part of the problem. The StableBit CloudDrive works like a physical drive, so it can actually get fragmented.  And because the cache locations may not be sequential on the disk either ....

 

Additionally, what are your cache settings, how large is the cache drive, and how much space does it have free? 

 

Also, these threads may help with understanding what exactly is going on. 

 

Share this post


Link to post
Share on other sites
  • 0
5 hours ago, Christopher (Drashna) said:

Yes and no.

As soon as you write the file to the drive, the driver both encrypts the data, and breaks it into chunks.  Those encrypted chunks are written to the local cache, and then "processed".  Meaning they're uploaded, or queued for upload. 

As for your read and writes, you're listing the sequential read and write speeds.  Which is at least part of the problem. The StableBit CloudDrive works like a physical drive, so it can actually get fragmented.  And because the cache locations may not be sequential on the disk either ....

 

Additionally, what are your cache settings, how large is the cache drive, and how much space does it have free? 

 

Also, these threads may help with understanding what exactly is going on. 

 

 

Local Cache Size 10GB+ | Expandable | NVME | 250GB Free out of 500GB | 100 MB RAM Cache

Share this post


Link to post
Share on other sites
  • 0

And where is the file being copied from? 

And could you enable file system logging, and reproduce?
http://wiki.covecube.com/StableBit_CloudDrive_Drive_Tracing

And after doing that, run the StableBit Troubleshooter on the system and ues "3489" as the Contact ID:
http://wiki.covecube.com/StableBit_Troubleshooter

Share this post


Link to post
Share on other sites
  • 0
On 3/6/2018 at 12:54 PM, Christopher (Drashna) said:

And where is the file being copied from? 

And could you enable file system logging, and reproduce?
http://wiki.covecube.com/StableBit_CloudDrive_Drive_Tracing

And after doing that, run the StableBit Troubleshooter on the system and ues "3489" as the Contact ID:
http://wiki.covecube.com/StableBit_Troubleshooter

There we go. I just don't understand..chunks don't begin uploading until the entire file is copied.  So if a 15GB file is being transferred, and 7 GB are already in the local cache and ready to upload, it won't begin uploading until the entire file is in the local cache.

Also the transfer rates make absolutely no sense to me. I know it can't be my NVME causing the slowness, but I have my i7-7700k OC'd to 4.8 GHz, and that should be good for the encryption and splitting of files right?

Share this post


Link to post
Share on other sites
  • 0
7 hours ago, Bowsa said:

I just don't understand..chunks don't begin uploading until the entire file is copied

For the upload, it may be the upload threshold setting that is causing this:
http://stablebit.com/Support/CloudDrive/Manual?Section=I/O Performance#Upload Threshold

For the copy, though, that is pretty unusual. 

A good question is "where are you copying the data FROM". Because that can influence the speed as well.  Copying from a USB2.0 drive will only get you 20-30MB/s tops, no matter how fast the destination is.

So does this happen, if you copy the file to the NVMe drive directly? And if you copy from the NVMe drive to the CloudDrive disk? 

Share this post


Link to post
Share on other sites
  • 0
On 3/9/2018 at 3:20 PM, Christopher (Drashna) said:

For the upload, it may be the upload threshold setting that is causing this:
http://stablebit.com/Support/CloudDrive/Manual?Section=I/O Performance#Upload Threshold

For the copy, though, that is pretty unusual. 

A good question is "where are you copying the data FROM". Because that can influence the speed as well.  Copying from a USB2.0 drive will only get you 20-30MB/s tops, no matter how fast the destination is.

So does this happen, if you copy the file to the NVMe drive directly? And if you copy from the NVMe drive to the CloudDrive disk? 

Yes. This is my workflow Downloader program (Downloads + Unpacks) ---> Other program flags download, and begins transferring to Cloud Drive

This process is from NVME to NVME, so there should be no bottlenecks caused by the drive itself. Also I do have the trigger set to 10 minutes, but if the program is displaying that 10GB are ready to upload, that should've activated the upload trigger long before the 5 minutes lapsed. 

For now I've had to do it manually..I pause the upload, and wait for the file to "upload" and/or transfer to the Local cache, and then I enable uploading which works. I also use 10 threads upload/download, but I've noticed that not disabling uploading and letting it handle on its own works.. eventually but then I get errors about the drive not uploading/downloading in time; although harmless, they're a bit annoying to see.

Share this post


Link to post
Share on other sites
  • 0
4 hours ago, Christopher (Drashna) said:

Thanks for the clarification.  

And yeah, that's ... definitely nothing that should affect the speeds here, at all. 

So that's what I'm trying to figure out. What exactly is causing the bottle-neck from my NVME Drive to the NVME Local Cache, is it the decryption?

I'm really happy with the product, but if I could get higher transfer speeds I'd be extremely happy :)

Share this post


Link to post
Share on other sites
  • 0

I followed over from the /r/homelab thread on reddit, since I'm having the same problem. Copying a 5GB ISO file from a drivepool of mechanical drives to the cloud drive cache on a sata ssd peaks at about 30MBps. Copying the same file to the root of the same ssd averages 130MBps, roughly what I'd expect given the performance of the mechanical drives. Anecdotally, I've tried the same settings with encryption enabled and disabled and it doesn't appear to affect the disk to disk copy rate.

Share this post


Link to post
Share on other sites
  • 0

I've just got Malwarebytes free installed (on WS2012R2). Enabling bypass file system filters on the drivepool has not produced a noticeable effect. I'll look into file system logging later today.

Share this post


Link to post
Share on other sites
  • 0

Ended up destroying and recreating the cloud drive as I moved to WS2016 and added a dedicated ssd for cache. Problem persists; copying to root of ssd tops out the mechanical drives, but copying to the cloud drive pool tops out at 30ish MB/sec.

Uploaded logs to your dropbox, first name Zo.

Share this post


Link to post
Share on other sites
  • 0
28 minutes ago, Christopher (Drashna) said:

Okay, thanks.

 

Does uninstalling MalwareBytes help?

No. Still get a maximum of around 30-35MB/sec copying to the cloud drive versus 160MB/sec copying to the cache disk itself.

[edit] I should probably add that I'm getting similar performance transferring uncached files off the cloud drive as well. I have a 1000/500 fiber connection. The second time transferring the same file (ie when it's been cached) performance is much better, around 120MB/sec

Edited by Digmarx
Added more info

Share this post


Link to post
Share on other sites
  • 0
On 3/27/2018 at 4:04 PM, Digmarx said:

No. Still get a maximum of around 30-35MB/sec copying to the cloud drive versus 160MB/sec copying to the cache disk itself.

[edit] I should probably add that I'm getting similar performance transferring uncached files off the cloud drive as well. I have a 1000/500 fiber connection. The second time transferring the same file (ie when it's been cached) performance is much better, around 120MB/sec

Yeah this is the only annoyance for me... that and not being able to transfer files to the CloudDrive un-attended because it dismounts

Share this post


Link to post
Share on other sites
  • 0

I've seen the same thing as mentioned here for the past couple months and was waiting to see if there was any update as it still appears to be happening for me. Is there any update around this or additional information that is needed that I might be able to provide to help further the analysis?

Share this post


Link to post
Share on other sites
  • 0
On 8/8/2018 at 3:54 PM, Bajdman said:

I've seen the same thing as mentioned here for the past couple months and was waiting to see if there was any update as it still appears to be happening for me. Is there any update around this or additional information that is needed that I might be able to provide to help further the analysis?

I'm still experiencing the issue, so it doesn't seem to have been resolved. But neither have I explored it any further.

Share this post


Link to post
Share on other sites
  • 0

There are a couple of things to check here.

First, do you have at least 50GB free on the cache drive? 
If not, that may start to throttle the writes, and the less free space on the cache drive there is, the more throttling will occur.

 

Also, when copying to the CloudDrive disk, check the activity on the cache drive.  See what the Task Manager shows. 

Check the active time, as well as the average response time.  If both of these are high, it will adversely affect the write performance to the CloudDrive disk. 

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...