[Suggestion] Dual cache HDD.

[superka]

Hi everybody!

 

I've been using this fantastic software for 3 months ago and i really love it.

 

the only thing I miss is being able to configure the download cache different from the upload

Why?

With ftth 600/600 even the SSD Samsung pro 950 gets bottleneck if i copy big size files and the software is downloading and uploading at same time (upload verification)

 

It would be wonderful if you could use a mechanical hard drive for uploads and a solid hard drive for downloads. (also longer life for the SSD).

 

What do you think?

 

Javi,

 

 

[Jaga]

How large are the files you are talking about?  Small enough to fit (mostly) in memory temporarily?  If so a RAM cache might help your downloading/uploading.  I use a 20GB L1 cache (using 3rd party software) with a variable write time delay, to help avoid bottlenecks like that.

The notion of specifying up/down caches for Clouddrive is interesting (I do like the idea), just not sure they could expose & split that feature between two separate physical drives.

Or...  perhaps it's time for a NVMe?   You could even use the NVMe drive as a L2 cache in front of your Samsung SSD, to speed up all read/write ops and even reduce writes with built-in trimming.

P.S.  I'm officially jealous of your transfer speeds!       Still stuck on conventional 400/20 cable broadband here.  Fiber has been "coming" for a decade now, still not in sight!

 

[Christopher (Drashna)]

Do you mean like this: https://stablebit.com/Support/CloudDrive/Manual?Section=The Local Cache#The Cache Types

[superka]

 

How large are the files you are talking about?  Small enough to fit (mostly) in memory temporarily?  If so a RAM cache might help your downloading/uploading.  I use a 20GB L1 cache (using 3rd party software) with a variable write time delay, to help avoid bottlenecks like that.

The notion of specifying up/down caches for Clouddrive is interesting (I do like the idea), just not sure they could expose & split that feature between two separate physical drives.

Or...  perhaps it's time for a NVMe?   You could even use the NVMe drive as a L2 cache in front of your Samsung SSD, to speed up all read/write ops and even reduce writes with built-in trimming.

P.S.  I'm officially jealous of your transfer speeds!       Still stuck on conventional 400/20 cable broadband here.  Fiber has been "coming" for a decade now, still not in sight!

 

 

i do have an NVMe as primary disk, im using the samsung 950 pro as only cache,

 

In 2 months i have over 10tb's of data writen (i lots of gb's of video 4k)

 

Mechanical disk is not fast enought to use all my bandwitch with data upload verification activated(and this is important for me)

 

Do you mean like this: https://stablebit.com/Support/CloudDrive/Manual?Section=The Local Cache#The Cache Types

Nope i don't mean this.

 

Everytime i copy something to my gdrive it just copies to the SSD, then it upload to the cloud. At the same time it beging to download in the same disk (to do the verification)

 

If some1 else of my network start to streaming the video, it just buffers and transfers speed to cloud decrease.

 

Thats why if we could use one hardrive to upload and another to download would be wonderful.

[Jaga]

 

i do have an NVMe as primary disk, im using the samsung 950 pro as only cache,

In 2 months i have over 10tb's of data writen (i lots of gb's of video 4k)

Mechanical disk is not fast enought to use all my bandwitch with data upload verification activated(and this is important for me)

Well, a 64GB or 128GB RAM cache would probably solve the bottleneck issue.  But that's a bunch of money into RAM, and the motherboard to support it.

 

 

Do you mean like this: https://stablebit.com/Support/CloudDrive/Manual?Section=The Local Cache#The Cache Types

I think he means the ability to set the cache folder separately for reads and writes.  i.e. all reads would be cached on drive X, all writes would be cached on drive Y.  He's trying to increase throughput at the drive level since it's his current bottleneck for transfers.  Apparently the SSD just can't keep up with up/down at the same time.   

[superka]

 

Well, a 64GB or 128GB RAM cache would probably solve the bottleneck issue.  But that's a bunch of money into RAM, and the motherboard to support it.

 

I think he means the ability to set the cache folder separately for reads and writes.  i.e. all reads would be cached on drive X, all writes would be cached on drive Y.  He's trying to increase throughput at the drive level since it's his current bottleneck for transfers.  Apparently the SSD just can't keep up with up/down at the same time.   

exactly.

 

 

 

 

[superka]

nobody thinks its a good idea?

[Jaga]

I think you are in a unique situation and no one else has the problem. 

Usually a network connection isn't going to outpace a SSD, and if it does there are other options to fall back on.  I use a 20GB L1 RAM cache in my system to avoid bottle-necking my drives (even a Samsung 850 Pro) and speed up all disk operations, out of 64GB total RAM.  In your case, if you have (or can install) 64GB or 128GB of memory and dedicate ~80% of that to a L1 RAM cache, you might avoid most of the issues you are seeing.  A RAM cache will go a long way to help eliminate unnecessary writes (live trimming), redundant reads, reads against written data, etc.

You might also consider limiting up/down transfer speeds with Clouddrive's bandwidth limiter in the meantime, so you don't choke the drive(s) with 100% activity.

[superka]

 

I think you are in a unique situation and no one else has the problem. 

Usually a network connection isn't going to outpace a SSD, and if it does there are other options to fall back on.  I use a 20GB L1 RAM cache in my system to avoid bottle-necking my drives (even a Samsung 850 Pro) and speed up all disk operations, out of 64GB total RAM.  In your case, if you have (or can install) 64GB or 128GB of memory and dedicate ~80% of that to a L1 RAM cache, you might avoid most of the issues you are seeing.  A RAM cache will go a long way to help eliminate unnecessary writes (live trimming), redundant reads, reads against written data, etc.

You might also consider limiting up/down transfer speeds with Clouddrive's bandwidth limiter in the meantime, so you don't choke the drive(s) with 100% activity.

64GB of ram nowdays its really expensive for me , this would be the best option but is out of my budget.

 

I can also limit the up and down transfers... but its sad to do this when a software implementation can help the transfers.

 

What about who own a 1gbit simetric network? the problem will persist.

[Christopher (Drashna)]

 

Everytime i copy something to my gdrive it just copies to the SSD, then it upload to the cloud. At the same time it beging to download in the same disk (to do the verification)

 

If some1 else of my network start to streaming the video, it just buffers and transfers speed to cloud decrease.

 

Then it sounds like the two issues that you're having is capacity, and IOPS.   A good SSD (and better, an NVMe drive) will fix the second issue.  But the capacity issue is something I can't really help with. But this also sounds like a case of "throw more money at the problem". 

[superka]

 

Then it sounds like the two issues that you're having is capacity, and IOPS.   A good SSD (and better, an NVMe drive) will fix the second issue.  But the capacity issue is something I can't really help with. But this also sounds like a case of "throw more money at the problem". 

Samgun 970 Evo 500gb sounds good? or best to go with pro version?

[Jaga]

Pro has much higher endurance, and would be the recommended one for higher usage.  Samsung SSD/NVMe Pro editions are top shelf.

But don't you already have a 950 Pro that's getting bottlenecked?  Or is the 970 you are considering purchasing a NVMe?

This site can be helpful when comparing NVMe's and SSDs for speed and pricing.

[superka]

 

Pro has much higher endurance, and would be the recommended one for higher usage.  Samsung SSD/NVMe Pro editions are top shelf.

But don't you already have a 950 Pro that's getting bottlenecked?  Or is the 970 you are considering purchasing a NVMe?

This site can be helpful when comparing NVMe's and SSDs for speed and pricing.

850 pro (the sata one)

 

I've got 2 M.2 sockets, one runing pci 3.0 x4 and other one runing pci 2.0 x4.

[Jaga]

Ah, nice.  Then you can put a decent NVMe in your second socket.  Given your circumstances, I'd go for a nice Samsung NVMe, whatever you can afford.  It should fix the bottleneck problem if it's large enough and on the faster socket.

[Christopher (Drashna)]

 

Pro has much higher endurance, and would be the recommended one for higher usage.  Samsung SSD/NVMe Pro editions are top shelf.

Pro is also rated for higher IOPS, which for this is the most important attribute. 

[darkly]

On 6/15/2018 at 2:09 PM, Christopher (Drashna) said:

Then it sounds like the two issues that you're having is capacity, and IOPS.   A good SSD (and better, an NVMe drive) will fix the second issue.  But the capacity issue is something I can't really help with. But this also sounds like a case of "throw more money at the problem". 

I have to toss in a vote for allowing for multiple cache drives as well, although for different reasons. I'm not really having a I/O issue, but rather, I'd like to be able to transfer large amounts of data to my CloudDrive, such that Windows/applications see the file as being ON the drive, and leave that uploading in the background for however long it takes to actually move the data to the cloud. It's a pain when my single cache drive reaches max capacity and I can only make files visible on the drive at throttled speeds.

 

[Gurrman]

No one seems to have mentioned this, but take your Cache of the C Drive, The C Drive has a lot of background tasks in windows using IOPs and bandwidth.

I use a Sata SSD 240GB and have no issues, the Drive Dram fills up and transfer speeds drop back to 40Mb/s but upload still goes strong at 250Mbit/s and i have no issues with using it as per normal during this time.