Jump to content
  • 0

File copy very slow


ufo56

Question

Hello

 

I have done windows 7 vm (in unraid) with latest clouddrive beta .797

 

Two hdds, one with windows and another is for cache (140gb cache disk). I copied about 50gb files to VM over lan and then copy paste to mounted google drive share.

 

Copy/Paste transfer speed is ~900kb/sec. What i am doing wrong ?

 

Internet connection is 300/300 but it should not matter when i copy files to drive, it first goes to cache and then it uploads to gdrive ? Clouddrive shows constantly 8mbps. It uses only 1 thread to upload, however i have set it to 10.

Link to comment
Share on other sites

Recommended Posts

  • 0

Are both of the CloudDrives using the same disk for the cache?

 

If so, then the disk may actually be getting overworked, causing the slow speeds.

 

 

 

Either way, could you enable tracing on one of the drives, and reproduce?

http://wiki.covecube.com/StableBit_CloudDrive_Drive_Tracing

Link to comment
Share on other sites

  • 0

Does copying files on the underlying drive to the other underlying drive (eg, the drives used for the cache) complete quickly or slowly?

 

Does copying files from one of the CloudDrive Disks to a completely separate disk have the same issue? For either/both drives? 

Hello

 

Sry for late answer.

 

Copying between just two hdds is fast.

Downloading from clouddrive is fast, if i copy gdrive->local

Copying local hdd > gdrive starts from 90mb/sec and after 30 sec goes to always 8,69mb/sec

Link to comment
Share on other sites

  • 0

I try to explain my setup.

 

unraid <windows-share> win7 machine with latest clouddrive

 

If i copy something from unraid to win7 machine clouddrive share then speed are every time 8.69mb/sec. And clouddrive stops uploading data to gdrive. Second after file transfer is completed clouddrive wakes up and starts uploading in full bandwith. Also, when file transfer is going on then windows share is inaccessible.

 

http://i.imgur.com/IxjAxt4.png 

Link to comment
Share on other sites

  • 0

Sry, i forget to answer.

 

Different drives. My unraid machine and clouddrive machine is separated. I mount my clouddrive share over gigabit network. And when i copy to unraid machine to clouddrive share then everything freezes. Clouddrive can't upload or download data after file copy is done.

 

It is somewhat better when i use ssd as cache drive. With HDD im starting to get bandwith errors and so on. Maybe HDD can't handle incoming LAN transfer and same time uploading/downloading data.

 

 

24/7 i get 300/300, i tried even 500/500 connection but same thing..

 

 

It would be nice if clouddrive can empty cache when it get at some %.

Link to comment
Share on other sites

  • 0

ufi56,

 

With the last release I am seeing similar - where my machine seems to freeze...A LOT.  When I FIRST purchased the software, I had set it up and was seeing some throttling issues.  I attributed that to the size of my Cache drive and the throttling that needed to happen to keep things happy.

 

Then, sometime last week (I do not recall the day), I was offered a new version.  Realizing this is still beta and that we were coming close to the cycle - I figured why not?

In my setup:

I have 13 clouddrives setup.

1 box.com, 2 dropbox, 10 Google Drive (not the paid cloud drives, but the provided ones for personal use).

 

I pooled them using DrivePool and disabled any replication / balancing / etc.  Just act as a single large drive putting files where ever there was space.

 

With this version, I now have to re-authenticate to each drive nearly every 12 hours!

v1.0.0.846

 

My fileserver (Windows Home Server 2012) was getting so bogged down and seemingly SLOW that I actually just let all the drives finish asking for re-authentication.  At this point, more than 1/2 of my drives are sitting waiting to be authenticated again - BUT my machine is now stable or responsive again.  I am assuming something in the kernel level was waiting on something and instead of giving up the processor while it waited, it just sat there hogging things.

 

BTW, DrivePOOL is unhappy with me at this time as most drives are not there.  Good thing this is not critical data!

 

--Dan

Link to comment
Share on other sites

  • 0

 

It would be nice if clouddrive can empty cache when it get at some %.

 

I'm not entirely sure what you mean by this, but it does basically.  But it actively avoids doing so if it can. 

http://community.covecube.com/index.php?/topic/1610-how-the-stablebit-clouddrive-cache-works/

 

 

Sry, i forget to answer.

 

Different drives. My unraid machine and clouddrive machine is separated. I mount my clouddrive share over gigabit network. And when i copy to unraid machine to clouddrive share then everything freezes. Clouddrive can't upload or download data after file copy is done.

 

It is somewhat better when i use ssd as cache drive. With HDD im starting to get bandwith errors and so on. Maybe HDD can't handle incoming LAN transfer and same time uploading/downloading data.

 

 

24/7 i get 300/300, i tried even 500/500 connection but same thing..

 

 

To clarify, how is everything setup? 

 

What provider are you using for StableBit CloudDrive? 

Is this the local disk provider? WIndows Share Provider? 

 

What system are you using for the copies? 

Where are you copying to and from?

 

 

.

 

ufi56,

 

With the last release I am seeing similar - where my machine seems to freeze...A LOT.  When I FIRST purchased the software, I had set it up and was seeing some throttling issues.  I attributed that to the size of my Cache drive and the throttling that needed to happen to keep things happy.

 

Then, sometime last week (I do not recall the day), I was offered a new version.  Realizing this is still beta and that we were coming close to the cycle - I figured why not?

In my setup:

I have 13 clouddrives setup.

1 box.com, 2 dropbox, 10 Google Drive (not the paid cloud drives, but the provided ones for personal use).

 

I pooled them using DrivePool and disabled any replication / balancing / etc.  Just act as a single large drive putting files where ever there was space.

 

With this version, I now have to re-authenticate to each drive nearly every 12 hours!

v1.0.0.846

 

My fileserver (Windows Home Server 2012) was getting so bogged down and seemingly SLOW that I actually just let all the drives finish asking for re-authentication.  At this point, more than 1/2 of my drives are sitting waiting to be authenticated again - BUT my machine is now stable or responsive again.  I am assuming something in the kernel level was waiting on something and instead of giving up the processor while it waited, it just sat there hogging things.

 

BTW, DrivePOOL is unhappy with me at this time as most drives are not there.  Good thing this is not critical data!

 

--Dan

 

This may be a completely unrelated issues to ufi56's issue, actually.

 

Specifically, in the new versions, we are forcibly indexing the entire contents of the folder's used for CloudDrive, on certain providers (Box, DropBox, Google Drive, Amazon CLoud Drive). 

 

This has to complete before each drive is usable.  However, this is a one time process, and shouldn't happen again. 

 

That said, if there isn't enough bandwidth, it may cause issues.  And if there are too many issues in a short period of time, the software will unmount the drive.

 

This is done explicitly to prevent the sort of freezing you've mentioned.  In fact, Windows can (will) lag/lock up when waiting on IO. Unmounting the drive will fix the issue.

 

 

That said, at least for the indexing process, it may be too sensitive to errors for your connection. This can be tweaked here: 

http://wiki.covecube.com/StableBit_CloudDrive_Advanced_Settings

Set "CloudFsDisk_MaximumConsecutiveIoFailures" to "10" and reboot the system.  This may prevent the drives from unmounting and requiring reauthorization. 

Link to comment
Share on other sites

  • 0

Im actually seeing something a bit similar to this, i use an SSD for cache for the cache, while moving data onto the clouddrive eg local disk -> ssd cache -> gdrive im seeing about ~5MB\s write speeds, and upload being somewhat shaky with everything from 1-30MB\s but generally I cannot copy fast enough to the SSD cache to actually keep the uploaded maxed out.

 

I have taken a look at this, and when im doing purely copy to the gdrive eg the ssd cache drive I can see that the SSD is running at 100% load, totally maxed out - The odd thing is that I had a samsung 840 evo that were in there before, it has run through 100+TB writes so I simply though it was dead, so I replaced it with a Kingston KC400 512GB Enterprise drive - The problem is the same, im seeing really slow performance to the ssd cache (writes) with 100% load on the drive.

 

Now if i use the SSD as cache for the drivepool, I can run 3-400MB\s writes to it without issues, the drive benchmarks to the 500+MB\s writes/reads that it is listed at.

 

It is important to note here that once the writes to the clouddrive stops, i instantly max out my upload speeds and the ssd cache load drops to 5-10% with ~50MB\s reads (I have 500Mbit internet).

 

Im seeing the same thing when I use a plain harddrive, infact I tried using my Samsung 950Pro nvme drive as cache and it shows the same thing but with slightly higher writes - Im not sure if this is something software wise, but from what I can see writing to the cache is extremely slow.

 

Im on the latest version, but the problem have persisted since I started using clouddrive - When and if the cache fils up, I have tried running the cache as a simply 6GB partition, so that it throttles after 1GB in the cache, the performance goes back to normal, I can run my max upload speed to gdrive with the local transfer locked at roughly the same speed as that.

 

Im on Windows 10 - If you need any details, let me know and I will collect them for you.

Link to comment
Share on other sites

  • 0

Okay, if you're seeing this (horrible) performance, could you download and install the Windows Performance Recorder?

https://msdn.microsoft.com/en-us/library/windows/hardware/hh448205.aspx

 

Enable drive tracing in StableBit CloudDrive, as well.

http://wiki.covecube.com/StableBit_CloudDrive_Drive_Tracing

 

 

Run the performance recorder, reproduce the issue and upload the data (use the "submit files" link/button in the above link)

Link to comment
Share on other sites

  • 0

I'm experiencing a similar issue, but I'm not sure which version it started on.

 

When I copy a file to my cloud drive, it saves it locally, and then uploads it to the cloud.

 

The problem is that it is transferring the file VERY slowly to the local drive area.

 

I was transferring @ 10 MB/sec to an SSD.

 

I decided to try another drive to see if something maybe was taxing the drive .

 

I created a VHD on another HDD I have ,attached it to my host, and let it run the Cloud Drive on there.

 

Now, I'm transferring @ 2 MB/sec, if I'm lucky, on this HDD.

 

 

If I take Stablebit Cloud Drive out of the equation and try to copy from/to these drives manually, I get 100 MB/sec and upwards.

 

Since this issue has nothing to do with the internet connection, what do you think may be causing this?

 

 

Look at the below image.

 

http://i.imgur.com/xeo3P1g.png

 

These two transfers are from the same physical source HDD to the same physical destination HDD.

The top one is being transferred via Stablebit CloudDrive's mounted drive.

The bottom one is being transferred directly to the storage where the "CloudPart" files reside.

http://i.imgur.com/j2NPdEY.png

 

If I wasn't doing them both at the same time, I would be getting ~2.5 MB/sec on the Cloud Drive transfer and ~100 MB/sec on the direct transfer.

 

 

To add,

 

I was never able to fully use 100% of my HDD speed when transferring to the local area of Stablebit CloudDrive, but I was able to use much more than 10 MB/Sec. I think it used to transfer at 60-70 MB/sec or so.

Link to comment
Share on other sites

  • 0

Okay, if you're seeing this (horrible) performance, could you download and install the Windows Performance Recorder?

https://msdn.microsoft.com/en-us/library/windows/hardware/hh448205.aspx

 

Enable drive tracing in StableBit CloudDrive, as well.

http://wiki.covecube.com/StableBit_CloudDrive_Drive_Tracing

 

 

Run the performance recorder, reproduce the issue and upload the data (use the "submit files" link/button in the above link)

Took a bit longer to look into this.

 

Im having a hard time doing it though, the "drive trace" within the clouddrive ui does absolutely 0 on my end.

 

Enabling it, reproducing and disabling does not create the "service" folder that are supposed to be created, I have unmounted the volume, restarted and even reinstalled the software without any change in behaviour.

 

Any ideas?

Link to comment
Share on other sites

  • 0

Took a bit longer to look into this.

 

Im having a hard time doing it though, the "drive trace" within the clouddrive ui does absolutely 0 on my end.

 

Enabling it, reproducing and disabling does not create the "service" folder that are supposed to be created, I have unmounted the volume, restarted and even reinstalled the software without any change in behaviour.

 

Any ideas?

 

The "Service" folder should be created regardless. It's used by the logging when happens as soon as the service is first started. 

 

Make sure you're looking at the "ProgramData" folder, and not the "Program Files" folder. This folder is hidden by default, so you either need to enable "Show hidden files" or type the folder name it. 

 

That said, you can enable tracing, then disable it and then run the "StableBit Troubleshooter", tool. This wil lgrab the logs and upload them, automatically. 

http://dl.covecube.com/Troubleshooter/StableBit.Troubleshooter.exe

 

The Performance Recorder is a bit more complicated. Grab the logs from where you've saved them and then upload them. 

Link to comment
Share on other sites

  • 0

I have actually been tinkering here, not entirely sure what changed but I downgraded the drive version and then a day after upgraded to the release version which entirely solved the problem.

 

Not entirely sure what changed between the two, nothing is really listed in regards to this issue in the release notes but for now I will mark this as "solved" :)

Link to comment
Share on other sites

  • 0

Was your issue the same as mine Fishie? Very slow transfer to local cache? 

 

If so, what exactly did you do to fix it? I'm still transferring at sub 10 MB/sec between virtualized drives where I don't have any issues if I bypass cloud drive itself.

Link to comment
Share on other sites

  • 0

It was actually, I was seeing the cache drive maxed out with writes being slow, not entirely sure why.

 

But I actually didn't do anything, I upgraded to the RC version, eg the very latest version and it is now running smoothly without any hickups :)

Link to comment
Share on other sites

  • 0

It was actually, I was seeing the cache drive maxed out with writes being slow, not entirely sure why.

 

But I actually didn't do anything, I upgraded to the RC version, eg the very latest version and it is now running smoothly without any hickups :)

If the drive that the cache resides on is full, then the slow speeds are normal. 

 

If you have less than 6GBs free on the disk, write throttling does occur. 

This is detailed here: 

http://community.covecube.com/index.php?/topic/1610-how-the-stablebit-clouddrive-cache-works/

 

This also varies slightly based on the cache type you're using. 

http://blog.covecube.com/2016/12/stablebit-clouddrive-1-0-0-777-beta/

Link to comment
Share on other sites

  • 0

I do not have any space concerns on my drive (1 TB Free). It is contained in a fixed size vhdx. I'm currently on 1.0.0.854 of Stablebit CloudDrive.

 

When I initiate a transfer from a source HDD to the HDD the VHDX resides on, it transfers at >100 MB/sec 

pJD0wJw.png

 

 

When I transfer from the same source HDD to the inside of the vhdx, it transfers at ~80 MB/sec.

 

3kdBZmj.png

 

When I transfer from the same source HDD to the mounted Stablebit CloudDrive, it transfers at  ~3-8 MB/sec. The speed bounces around and is not consistent.

 

q8JVRxB.png

 

This did not use to happen and nothing has changed in my hardware. The only thing that has changed is my stablebit CloudDrive version, which I'm not sure which specific version introduced it.

 

my cache size is set to minimal if that makes a difference as I prefer to not have anything locally saved. I used to have this option set to none or 0 when the interface looked a little different.

 

4RAKx90.png

Link to comment
Share on other sites

  • 0

Are there any files I should remove from the service folder before performing the drive tracing? I see that there's a lot of old logs and such in here and I wouldn't want something to conflict and show something else as a problem.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...