Jump to content
Covecube Inc.
  • 0
steffenmand

Question about loading multiple files

Question

Hi,

 

 

I was wondering how cloud drive priorities threads if you open multiple files at the same time. are the threads equally divided among the files or will one be able to suck all threads leaving the others to wait or download extremely slow?

 

In my case i could see my need to access several files at a time and 5 threads could for me be enough to download faily quick so in a scenario where i had 20 threads i would hope to see 3 files getting 6-7 threads each?

 

And just a quick follow up on a former request:

 

Are bigger or no partial read sizes on the schedule for a release coming soon or should we expect that in weeks or months to come? :-) Just looking forward to seeing my bandwith in action on the download

Share this post


Link to post
Share on other sites

7 answers to this question

Recommended Posts

  • 0

I can't answer that.  It's a more technical question, and I'm not sure.

 

I've flagged this for Alex, so he can answer. 

 

 

Though, this would depend if the files are cached already. Frequently accessed data should remain in the cache, and be ready for immediate access. 

Also, depending on the file sizes, prefetching may occur, which should only require one thread for the sequential data.

 

 

But basically, "very complicated" is about the best technical answer I can give you here. Alex can expand.

 

https://stablebit.com/Admin/IssueAnalysis/24919

Share this post


Link to post
Share on other sites
  • 0

Hi,

 

 

I was wondering how cloud drive priorities threads if you open multiple files at the same time. are the threads equally divided among the files or will one be able to suck all threads leaving the others to wait or download extremely slow?

 

In my case i could see my need to access several files at a time and 5 threads could for me be enough to download faily quick so in a scenario where i had 20 threads i would hope to see 3 files getting 6-7 threads each?

 

And just a quick follow up on a former request:

 

Are bigger or no partial read sizes on the schedule for a release coming soon or should we expect that in weeks or months to come? :-) Just looking forward to seeing my bandwith in action on the download

 

Technically speaking, StableBit CloudDrive doesn't really deal with files directly. It actually emulates a virtual disk for the Operating System, and other applications or the Operating System can then use that disk for whatever purpose that suits them.

 

Normally, a file system kernel driver (such as NTFS) will be mounted on the disk and will provide file based access. So when you say that you're accessing 2 files on the cloud drive, you're actually dealing with the file system (typically NTFS). The file system then translates your file-based requests (e.g. read 10248 bytes at offset 0 of file \Test.bin) into disk-based requests (e.g. read 10248 bytes at position 12,447,510 on disk #4).

 

But you may ask, well, eventually these file-based requests must be scheduled by something, so who prioritizes which requests get serviced first?

 

Yes, and here's how that works:

  • The I/O pipeline in the kernel is inherently asynchronous. In other words, multiple I/O request can be "in-flight" at the same time.
  • Scheduling of disk-based I/O requests occurs right before our virtual disk, by the NT disk subsystem (multiple drivers involved). It determines which requests get serviced first by using the concept of I/O priorities.

To read more about this topic (probably much more than you'd ever want to know) see: 

https://www.microsoftpressstore.com/articles/article.aspx?p=2201309&seqNum=3

 

Skip to the I/O Prioritization section for the relevant information. You can also read about the concept of bandwidth reservation which deals with ensuring smooth media playback.

Share this post


Link to post
Share on other sites
  • 0

Are bigger or no partial read sizes on the schedule for a release coming soon or should we expect that in weeks or months to come? :-) Just looking forward to seeing my bandwith in action on the download

Oh, and regarding the partial reads / no partial reads issue, there is an Issue open for that in the form of a feature request. Right now, StableBit CloudDrive reads all checksummed / signed data in 1 MB units. So even if 1 byte needs to be read, 1 MB will actually be read and cached.

 

Can we increase that 1 MB to let's say 10 MB? Yes we can, but I'm afraid that it will adversely affect the read performance of slower bandwidth users. So that's something that needs to be tested and that's why this is in the form of a feature request.

Share this post


Link to post
Share on other sites
  • 0

Oh, and regarding the partial reads / no partial reads issue, there is an Issue open for that in the form of a feature request. Right now, StableBit CloudDrive reads all checksummed / signed data in 1 MB units. So even if 1 byte needs to be read, 1 MB will actually be read and cached.

 

Can we increase that 1 MB to let's say 10 MB? Yes we can, but I'm afraid that it will adversely affect the read performance of slower bandwidth users. So that's something that needs to be tested and that's why this is in the form of a feature request.

 

Thanks for the great reply! Regarding people with lower bandwidth, it could be an option with recommend speeds and make the partial read size completely defineable by the user. This being read operations, i guess this could easily be changed to a lower size if a user figures out it doesnt work properly for them and would not require a drive update :-)

 

Or are the partial read size somehow dependent on the drive?

Share this post


Link to post
Share on other sites
  • 0

Thanks for the great reply! Regarding people with lower bandwidth, it could be an option with recommend speeds and make the partial read size completely defineable by the user. This being read operations, i guess this could easily be changed to a lower size if a user figures out it doesnt work properly for them and would not require a drive update :-)

 

Or are the partial read size somehow dependent on the drive?

By "drive update", I'm assuming you mean creating a new drive?

 

If so... the issue with changing the chunk size or the checksum unit size is very similar to the physical sector size (512b vs 4kb "Advanced Format") issue. So, there isn't a good way to cover this over, without reading and writing to the entire drive. At which point, it would be simpler and more efficient to copy data to a new drive (I suspect).

Share this post


Link to post
Share on other sites
  • 0

I was unaware that the drive initial settings sets the read size, thought you would be able to change it as a setting in stablebit, this of course makes it more challenging for you :-)

 

Hopefully a warning or "hih bandwidth required" info would be enough then :-)

Share this post


Link to post
Share on other sites
  • 0

I was unaware that the drive initial settings sets the read size, thought you would be able to change it as a setting in stablebit, this of course makes it more challenging for you :-)

 

Hopefully a warning or "hih bandwidth required" info would be enough then :-)

 

I'm not really sure the point you're trying to make here. 

 

But, either way, the default settings are what we've tried to optimize for "most users".  These should work for most people.

In fact, the 1MB file size, or small checksum unit size for partial uploads/downloads  specifically to minimize the latency for "disk access".

 

And we do list a requirement for internet connection, as anything slower may cause issues with connectivity and latency. 

 

 

And yeah, all the setting such as chunk size, (in the future) checksum size, and the like are all set up when creating the drive. There are a number of reasons for this, but as I said, a lot of this is very much like the advanced formatting feature, and partially the formatting options.  So, it's not somethign that can be changed after the fact. 

 

 

The exception to this is that you can detach and reattach a drive and change the cache options. 

 

 

However, you high bandwidth people may require different settings (at least to saturate your bandwidth).  And worst case, I'll see about creating a guide on the wiki for high bandwidth users. 

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...