Jump to content
  • 0

Unbearable copy speed to Cloud Drive


Kyrluckechuck

Question

Hey there,

 

I read over and considered posting this in the existing similar thread, but it seemed to me to be a slightly different topic and likely a different root cause of the situation, I apologize if this should have gone there!

 

I've been needing to upgrade my Plex solution to a larger storage solution since my home bandwidth speeds just simply can't handle the amount of data needing to be uploaded for viewing 2 1080p streams simultaneously + gaming on the network and other server hosting going on (20mbps). I made the switch to a 3TBx2 solution on SoYouStart, but I'm already realizing that that won't be enough space and within a couple of months I'll be reaching capacity (and it's already biting into the other sites I hosts' space).

 

Currently I've come up with the idea of trialing ACD's 3-months of unlimited cloud storage to fool around with possibly using something like CloudDrive as a solution, but quickly realized it was going to be rough until ACD got its act together and it would likely be smart to put all of the data into a multi-pool solution where the data is on Google Drive for the next couple years while I'm still a student and get unlimited storage, and ACD would just be a secondary drive duping that one.

 

After getting everything setup and leaving it to copy for a few hours, I realized that it was going to be unbearably slow as the copy speed from the one VirtIO drive on my Windows Server 2012 R2 (which is a VM allocated 6 high-priority CPU threads from the main 12 thread Intel server where the other VM's use the rest (minimal going on right now, basically shut down)) to the Google Drive drive was going at 3MB/s. I thought maybe the general disk speed was slow so I tested just duplicating a file in the root of the drive I was copying from and it was actually going at about 60MB/s+. Copying from a Samba share from a Ubuntu VM had similar results (~3MB/s to the virt drive, ~50MB/s to anywhere else).

 

I have tried this on version .463, .631 and .748 and it is identical results (though trying to resize a 100GB drive to 10TB using .463 would result in BSOD's of the PAGE_FAULT_BEYOND_END_OF_ALLOCATION variety. This was solved by deleting and purging the drive, installing .631 and recreating it).

 

I'm honestly out of ideas as to what could cause this as I've tried various different Allocation sizes, chunk sizes, etc, and they all have the same slow result.

 

On my home server, however, the file IO is normally ~80MB/s+, and copying to the virt drives goes at about ~12-13MB/s which isn't fast but it's bearable (with occasional boosts up to 20MB/s and sometimes even 40MB/s+). (The copy speeds are the same for both ACD and GDrive on both servers, so it is likely the encryption process if I had to make a guesstimate of the cause. Why or how to fix it, that I don't know!)

 

So my dilemma is that while I've got a great upload speed on my server (1Gigabit down, 250Mbit up), the copy speed is ^&*%$# (the moment there's enough data to upload, it's uploaded as the network speed is faster than the copy speed), and a crappy internet speed at home (20Mbit up), but it's copy speed is faster than the upload (currently sitting on 450GB in queue of test data but it's a slow process).

 

D:

 

As per ACD/GDrive policy of being able to read content, of course these cloud drives are always encrypted and it would be illogical/not an option for me to ever go the non-encrypted route.

 

Any help would be absolutely loved :D

Link to comment
Share on other sites

3 answers to this question

Recommended Posts

  • 0

Hey there,

 

I actually was using a 3TBx2 configuration (RAID 1) on Proxmox which was hosting a Windows Server 2012 R2 system. I then tested various caching configurations for speed changes ranging from no cache all the way to writeback, but there was no major differences to the encrypted write speeds (though that VM did have 6 core's at it's disposal and was never more than 50% usage), though there was to the normal copy-paste write speeds.

 

On the home sever that was having the slightly better encryption speeds, it was a basic 750GB 7200RPM drive for the OS with the cloud drives caching onto the 3TBx2 Storage Spaces drive I had setup for the media. I was not really able to configure it too much, but I did try various configurations including caching on the single drive vs the Storage Spaces drive, and it was generally about the same.

 

I've come to the realization that this technology is amazing, in theory, and when push comes to shove it's just not quite ready for the prime-time for large-content storage of data for a Plex setup (though for long-term/infrequent access it's amazing), whether it's using Google Drive or Amazon Cloud Drive.

 

I'll be keeping an eye on this for developments, but in the meantime I downgraded my main server since I didn't need so much HDD space/CPU cores to handle the encrypting/decrypting and just upgraded my Internet and local hardware to host it locally!

 

Best of luck with this endeavor and I hope this matures enough after public release for me to be able to begin using it in a production-style environment! :)

Link to comment
Share on other sites

  • 0

If the performance issues you are seeing is with Plex, that may be more related to how plex handles things than anything else. 

 

We've seen some odd behavior.

 

Additionally, if you're using Amazon Cloud Drive as the provider here... there is a reason that it's an experimental provider.  A big part of that is that the provider is SEVERELY throttled.    using a different provider may provide better results here.

 

Specifically, it's throttled to 50 mbps download and 20 mbps upload.  It's also limited to 4 downstream connections and 2 upstream (and defaults to half that, IIRC). 

 

 

 

Furthermore, we've been having issues with Google Drive due to an API limit for the app in general. Something we have been working on (the latest versions should be much better about this).  However, the newer versions have issues with authentication due to changes that Amazon has made to ACD. 

 

 

That said, if you would, grabbing log files may help us to identify the issues you're seeing here. 

http://wiki.covecube.com/StableBit_CloudDrive_Drive_Tracing

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...