Jump to content

  • Log in with Twitter Log in with Windows Live Log In with Google      Sign In   
  • Create Account

Photo

File copy very slow


  • Please log in to reply
39 replies to this topic

#1 ufo56

ufo56

    Member

  • Members
  • PipPip
  • 18 posts

Posted 09 January 2017 - 10:37 PM

Hello

 

I have done windows 7 vm (in unraid) with latest clouddrive beta .797

 

Two hdds, one with windows and another is for cache (140gb cache disk). I copied about 50gb files to VM over lan and then copy paste to mounted google drive share.

 

Copy/Paste transfer speed is ~900kb/sec. What i am doing wrong ?

 

Internet connection is 300/300 but it should not matter when i copy files to drive, it first goes to cache and then it uploads to gdrive ? Clouddrive shows constantly 8mbps. It uses only 1 thread to upload, however i have set it to 10.


  • Ginoliggime and KiaraEvirm like this

#2 eldite

eldite

    Member

  • Members
  • PipPip
  • 16 posts

Posted 10 January 2017 - 11:48 AM

It is likely your cache disk has less than 5GB free?  When that happens it throttles the copy speed down to the connection speed. 



#3 ufo56

ufo56

    Member

  • Members
  • PipPip
  • 18 posts

Posted 10 January 2017 - 01:50 PM

No, 113GB free from 140GB.

 

http://i.imgur.com/coUEZ75.png

 

http://i.imgur.com/wfPUJNU.png

 

If something is copying to google mount then maybe even 10x outcoing connections but speed is 0 or very very low. 

 

Im using unraid with 6gb ram, 3 cores. One 50gb for OS and 140GB cache. Raw vdisk. No difference with VirtIO or Sata mode.



#4 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 7,976 posts
  • LocationSan Diego, CA, USA

Posted 10 January 2017 - 08:00 PM

Are both of the CloudDrives using the same disk for the cache?

 

If so, then the disk may actually be getting overworked, causing the slow speeds.

 

 

 

Either way, could you enable tracing on one of the drives, and reproduce?

http://wiki.covecube...e_Drive_Tracing


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#5 ufo56

ufo56

    Member

  • Members
  • PipPip
  • 18 posts

Posted 11 January 2017 - 02:53 PM

 

 

Are both of the CloudDrives using the same disk for the cache?

No different HDDs



#6 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 7,976 posts
  • LocationSan Diego, CA, USA

Posted 12 January 2017 - 10:12 PM

Does copying files on the underlying drive to the other underlying drive (eg, the drives used for the cache) complete quickly or slowly?

 

Does copying files from one of the CloudDrive Disks to a completely separate disk have the same issue? For either/both drives? 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#7 ufo56

ufo56

    Member

  • Members
  • PipPip
  • 18 posts

Posted 24 January 2017 - 04:59 PM

Does copying files on the underlying drive to the other underlying drive (eg, the drives used for the cache) complete quickly or slowly?

 

Does copying files from one of the CloudDrive Disks to a completely separate disk have the same issue? For either/both drives? 

Hello

 

Sry for late answer.

 

Copying between just two hdds is fast.

Downloading from clouddrive is fast, if i copy gdrive->local

Copying local hdd > gdrive starts from 90mb/sec and after 30 sec goes to always 8,69mb/sec



#8 ufo56

ufo56

    Member

  • Members
  • PipPip
  • 18 posts

Posted 25 January 2017 - 12:37 AM

I try to explain my setup.

 

unraid <windows-share> win7 machine with latest clouddrive

 

If i copy something from unraid to win7 machine clouddrive share then speed are every time 8.69mb/sec. And clouddrive stops uploading data to gdrive. Second after file transfer is completed clouddrive wakes up and starts uploading in full bandwith. Also, when file transfer is going on then windows share is inaccessible.

 

http://i.imgur.com/IxjAxt4.png 



#9 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 7,976 posts
  • LocationSan Diego, CA, USA

Posted 03 February 2017 - 12:16 AM

Is the drive you're copying from the one that the cache is on? 

 

As for the errors, it looks like you're running out of bandwidth.  What's your downstream bandwidth as reported by your ISP?


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#10 ufo56

ufo56

    Member

  • Members
  • PipPip
  • 18 posts

Posted 21 February 2017 - 12:02 AM

Sry, i forget to answer.

 

Different drives. My unraid machine and clouddrive machine is separated. I mount my clouddrive share over gigabit network. And when i copy to unraid machine to clouddrive share then everything freezes. Clouddrive can't upload or download data after file copy is done.

 

It is somewhat better when i use ssd as cache drive. With HDD im starting to get bandwith errors and so on. Maybe HDD can't handle incoming LAN transfer and same time uploading/downloading data.

 

 

24/7 i get 300/300, i tried even 500/500 connection but same thing..

 

 

It would be nice if clouddrive can empty cache when it get at some %.



#11 ironhead65

ironhead65

    Advanced Member

  • Members
  • PipPipPip
  • 31 posts

Posted 27 February 2017 - 09:39 PM

ufi56,

 

With the last release I am seeing similar - where my machine seems to freeze...A LOT.  When I FIRST purchased the software, I had set it up and was seeing some throttling issues.  I attributed that to the size of my Cache drive and the throttling that needed to happen to keep things happy.

 

Then, sometime last week (I do not recall the day), I was offered a new version.  Realizing this is still beta and that we were coming close to the cycle - I figured why not?

In my setup:

I have 13 clouddrives setup.

1 box.com, 2 dropbox, 10 Google Drive (not the paid cloud drives, but the provided ones for personal use).

 

I pooled them using DrivePool and disabled any replication / balancing / etc.  Just act as a single large drive putting files where ever there was space.

 

With this version, I now have to re-authenticate to each drive nearly every 12 hours!

v1.0.0.846

 

My fileserver (Windows Home Server 2012) was getting so bogged down and seemingly SLOW that I actually just let all the drives finish asking for re-authentication.  At this point, more than 1/2 of my drives are sitting waiting to be authenticated again - BUT my machine is now stable or responsive again.  I am assuming something in the kernel level was waiting on something and instead of giving up the processor while it waited, it just sat there hogging things.

 

BTW, DrivePOOL is unhappy with me at this time as most drives are not there.  Good thing this is not critical data!

 

--Dan



#12 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 7,976 posts
  • LocationSan Diego, CA, USA

Posted 04 March 2017 - 11:32 PM

 

It would be nice if clouddrive can empty cache when it get at some %.

 

I'm not entirely sure what you mean by this, but it does basically.  But it actively avoids doing so if it can. 

http://community.cov...ve-cache-works/

 

 

Sry, i forget to answer.

 

Different drives. My unraid machine and clouddrive machine is separated. I mount my clouddrive share over gigabit network. And when i copy to unraid machine to clouddrive share then everything freezes. Clouddrive can't upload or download data after file copy is done.

 

It is somewhat better when i use ssd as cache drive. With HDD im starting to get bandwith errors and so on. Maybe HDD can't handle incoming LAN transfer and same time uploading/downloading data.

 

 

24/7 i get 300/300, i tried even 500/500 connection but same thing..

 

 

To clarify, how is everything setup? 

What provider are you using for StableBit CloudDrive? 
Is this the local disk provider? WIndows Share Provider? 

 

What system are you using for the copies? 

Where are you copying to and from?

 

 

.

 

ufi56,

 

With the last release I am seeing similar - where my machine seems to freeze...A LOT.  When I FIRST purchased the software, I had set it up and was seeing some throttling issues.  I attributed that to the size of my Cache drive and the throttling that needed to happen to keep things happy.

 

Then, sometime last week (I do not recall the day), I was offered a new version.  Realizing this is still beta and that we were coming close to the cycle - I figured why not?

In my setup:

I have 13 clouddrives setup.

1 box.com, 2 dropbox, 10 Google Drive (not the paid cloud drives, but the provided ones for personal use).

 

I pooled them using DrivePool and disabled any replication / balancing / etc.  Just act as a single large drive putting files where ever there was space.

 

With this version, I now have to re-authenticate to each drive nearly every 12 hours!

v1.0.0.846

 

My fileserver (Windows Home Server 2012) was getting so bogged down and seemingly SLOW that I actually just let all the drives finish asking for re-authentication.  At this point, more than 1/2 of my drives are sitting waiting to be authenticated again - BUT my machine is now stable or responsive again.  I am assuming something in the kernel level was waiting on something and instead of giving up the processor while it waited, it just sat there hogging things.

 

BTW, DrivePOOL is unhappy with me at this time as most drives are not there.  Good thing this is not critical data!

 

--Dan

 

This may be a completely unrelated issues to ufi56's issue, actually.

 

Specifically, in the new versions, we are forcibly indexing the entire contents of the folder's used for CloudDrive, on certain providers (Box, DropBox, Google Drive, Amazon CLoud Drive). 

 

This has to complete before each drive is usable.  However, this is a one time process, and shouldn't happen again. 

 

That said, if there isn't enough bandwidth, it may cause issues.  And if there are too many issues in a short period of time, the software will unmount the drive.

 

This is done explicitly to prevent the sort of freezing you've mentioned.  In fact, Windows can (will) lag/lock up when waiting on IO. Unmounting the drive will fix the issue.

 

 

That said, at least for the indexing process, it may be too sensitive to errors for your connection. This can be tweaked here: 

http://wiki.covecube...vanced_Settings

Set "CloudFsDisk_MaximumConsecutiveIoFailures" to "10" and reboot the system.  This may prevent the drives from unmounting and requiring reauthorization. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#13 fishie

fishie

    Member

  • Members
  • PipPip
  • 16 posts

Posted 09 March 2017 - 03:33 PM

Im actually seeing something a bit similar to this, i use an SSD for cache for the cache, while moving data onto the clouddrive eg local disk -> ssd cache -> gdrive im seeing about ~5MB\s write speeds, and upload being somewhat shaky with everything from 1-30MB\s but generally I cannot copy fast enough to the SSD cache to actually keep the uploaded maxed out.

 

I have taken a look at this, and when im doing purely copy to the gdrive eg the ssd cache drive I can see that the SSD is running at 100% load, totally maxed out - The odd thing is that I had a samsung 840 evo that were in there before, it has run through 100+TB writes so I simply though it was dead, so I replaced it with a Kingston KC400 512GB Enterprise drive - The problem is the same, im seeing really slow performance to the ssd cache (writes) with 100% load on the drive.

 

Now if i use the SSD as cache for the drivepool, I can run 3-400MB\s writes to it without issues, the drive benchmarks to the 500+MB\s writes/reads that it is listed at.

 

It is important to note here that once the writes to the clouddrive stops, i instantly max out my upload speeds and the ssd cache load drops to 5-10% with ~50MB\s reads (I have 500Mbit internet).

 

Im seeing the same thing when I use a plain harddrive, infact I tried using my Samsung 950Pro nvme drive as cache and it shows the same thing but with slightly higher writes - Im not sure if this is something software wise, but from what I can see writing to the cache is extremely slow.

 

Im on the latest version, but the problem have persisted since I started using clouddrive - When and if the cache fils up, I have tried running the cache as a simply 6GB partition, so that it throttles after 1GB in the cache, the performance goes back to normal, I can run my max upload speed to gdrive with the local transfer locked at roughly the same speed as that.

 

Im on Windows 10 - If you need any details, let me know and I will collect them for you.



#14 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 7,976 posts
  • LocationSan Diego, CA, USA

Posted 09 March 2017 - 10:05 PM

Okay, if you're seeing this (horrible) performance, could you download and install the Windows Performance Recorder?

https://msdn.microso...e/hh448205.aspx

 

Enable drive tracing in StableBit CloudDrive, as well.

http://wiki.covecube...e_Drive_Tracing

 

 

Run the performance recorder, reproduce the issue and upload the data (use the "submit files" link/button in the above link)


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#15 fishie

fishie

    Member

  • Members
  • PipPip
  • 16 posts

Posted 09 March 2017 - 10:12 PM

Will do that this coming monday, im away on business atm.

 

Thanks for coming back to me.


  • Christopher (Drashna) likes this

#16 Quiks

Quiks

    Member

  • Members
  • PipPip
  • 25 posts
  • LocationSan Diego, CA

Posted 09 March 2017 - 10:22 PM

I'm experiencing a similar issue, but I'm not sure which version it started on.

 

When I copy a file to my cloud drive, it saves it locally, and then uploads it to the cloud.

 

The problem is that it is transferring the file VERY slowly to the local drive area.

 

I was transferring @ 10 MB/sec to an SSD.

 

I decided to try another drive to see if something maybe was taxing the drive .

 

I created a VHD on another HDD I have ,attached it to my host, and let it run the Cloud Drive on there.

 

Now, I'm transferring @ 2 MB/sec, if I'm lucky, on this HDD.

 

 

If I take Stablebit Cloud Drive out of the equation and try to copy from/to these drives manually, I get 100 MB/sec and upwards.

 

Since this issue has nothing to do with the internet connection, what do you think may be causing this?

 

 

Look at the below image.

 

http://i.imgur.com/xeo3P1g.png

 

These two transfers are from the same physical source HDD to the same physical destination HDD.

The top one is being transferred via Stablebit CloudDrive's mounted drive.

The bottom one is being transferred directly to the storage where the "CloudPart" files reside.

http://i.imgur.com/j2NPdEY.png

 

If I wasn't doing them both at the same time, I would be getting ~2.5 MB/sec on the Cloud Drive transfer and ~100 MB/sec on the direct transfer.

 

 

To add,

 

I was never able to fully use 100% of my HDD speed when transferring to the local area of Stablebit CloudDrive, but I was able to use much more than 10 MB/Sec. I think it used to transfer at 60-70 MB/sec or so.



#17 fishie

fishie

    Member

  • Members
  • PipPip
  • 16 posts

Posted 14 March 2017 - 10:29 AM

Okay, if you're seeing this (horrible) performance, could you download and install the Windows Performance Recorder?

https://msdn.microso...e/hh448205.aspx

 

Enable drive tracing in StableBit CloudDrive, as well.

http://wiki.covecube...e_Drive_Tracing

 

 

Run the performance recorder, reproduce the issue and upload the data (use the "submit files" link/button in the above link)

Took a bit longer to look into this.

 

Im having a hard time doing it though, the "drive trace" within the clouddrive ui does absolutely 0 on my end.

 

Enabling it, reproducing and disabling does not create the "service" folder that are supposed to be created, I have unmounted the volume, restarted and even reinstalled the software without any change in behaviour.

 

Any ideas?



#18 Christopher (Drashna)

Christopher (Drashna)

    Customer and Technical Support

  • Administrators
  • 7,976 posts
  • LocationSan Diego, CA, USA

Posted 18 March 2017 - 11:20 PM

Took a bit longer to look into this.

 

Im having a hard time doing it though, the "drive trace" within the clouddrive ui does absolutely 0 on my end.

 

Enabling it, reproducing and disabling does not create the "service" folder that are supposed to be created, I have unmounted the volume, restarted and even reinstalled the software without any change in behaviour.

 

Any ideas?

 

The "Service" folder should be created regardless. It's used by the logging when happens as soon as the service is first started. 

 

Make sure you're looking at the "ProgramData" folder, and not the "Program Files" folder. This folder is hidden by default, so you either need to enable "Show hidden files" or type the folder name it. 

 

That said, you can enable tracing, then disable it and then run the "StableBit Troubleshooter", tool. This wil lgrab the logs and upload them, automatically. 

http://dl.covecube.c...ubleshooter.exe

 

The Performance Recorder is a bit more complicated. Grab the logs from where you've saved them and then upload them. 


Christopher Courtney

aka "Drashna"

Microsoft MVP for Windows Home Server 2009-2012

Lead Moderator for We Got Served

Moderator for Home Server Show

 

This is my server

 

Lots of "Other" data on your pool? Read about what it is here.


#19 fishie

fishie

    Member

  • Members
  • PipPip
  • 16 posts

Posted 20 March 2017 - 12:29 PM

I have actually been tinkering here, not entirely sure what changed but I downgraded the drive version and then a day after upgraded to the release version which entirely solved the problem.

 

Not entirely sure what changed between the two, nothing is really listed in regards to this issue in the release notes but for now I will mark this as "solved" :)



#20 Quiks

Quiks

    Member

  • Members
  • PipPip
  • 25 posts
  • LocationSan Diego, CA

Posted 21 March 2017 - 03:32 AM

Was your issue the same as mine Fishie? Very slow transfer to local cache? 

 

If so, what exactly did you do to fix it? I'm still transferring at sub 10 MB/sec between virtualized drives where I don't have any issues if I bypass cloud drive itself.






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users