Jump to content
  • 0

hardlinks, chunk sizes, hashes and threads


evulhotdog

Question

1. How does CloudDrive handle hardlinks or symlinks? Does it truly act as one file, or does it just make a copy and act as two?

 

2. I was playing with different chunk sizes, and it seems like the larger chunk sizes takes significantly longer to upload the same total amount of data. What is going on here?

 

3. Does CloudDrive hash the local chunk, upload it, then check that hash against the hash that the ACD API provides? It seems like when I am actually watching the bandwith usage, it will upload, then download a similar amount of data. I'm wondering why it downloads this data if ACD provides MD5 hash support against those chunks.

 

4. Is there a config file to allow more threads? I was able to memory edit the service to allow more download threads, but not upload. And yes I know it is limited for the time being, was just trying to test with a higher thread count to improve upload times. 

 

5. When trying to create more than one drive through the application, it seems to fail formatting the drive. Look at the drive in drive management interface, and it shows as offline. Maybe an issue with the driver?

 

6. Is there an RSS feed or some sort of feed which shows the changes between the version? I found one but the last update was .401 which is a bit old.

 

 

 

Once the ACD issues get resolved and figured out, I will certainly be buying a product, because it mounts it differently than what NetDrive does, and allows applications to interact with the drive properly.

 

 

Link to comment
Share on other sites

3 answers to this question

Recommended Posts

  • 0

  1. The disks created by StableBit CloudDrive act as actual disks, because we're using a block based solution (as opposed to a file based solution such as NetDrive). This means that the disks support hard links, symbolic links, junction points, bitlocker encryption, etc. 

    However, this means that existing files on the provider cannot be added. 

     

  2. That depends heavily on the provider you're using.  If you're using Amazon Cloud Drive (which you've indicated you are), we do "upload verification" on all chunks. This is REQUIRED to ensure that the file has been properly uploaded to Amazon and that they have records of the file (this is a known issue with every app using Amazon Cloud Drive).  This may be why your'e seeing the discrepancy. 

     

  3. If you enabled the "Use checksum verification" option when creating the drive, each chunk contains a built in checksum. The larger chunks include partial chunk checksums, if the provider is using partial uploads. 

    As for the upload, and download, this is because of the upload verification. This downloads the data again after uploading and checks it. If the data checks out, it moves to the next chunk. Otherwise it will redownload it. 

    This is REQUIRED by Amazon Cloud Drive, otherwise, the data seems to get corrupted. 

     

  4. "Disk Options" -> "Performance" -> "IO Performance".  You can increase the number of threads. This is the number of concurrent uploads or downloads (specified separately), which means more activity.

    However, Amazon Cloud Drive is limited to 2 upload and 4 download threads, due to various issues. You can read about our ongoing struggles to get Amazon Cloud Drive supported here:

    http://community.covecube.com/index.php?/topic/1588-amazon-cloud-drive-why-is-it-not-supported/

     

  5. If this is with Amazon Cloud Drive, this may be an issue with the provider specifically.

    Otherwise, check the "SAN" policy. To do so, run "diskpart" and then type "san". Let me know what this is set as (it should be "OnlineAll"). 

    But you should be able to set the disk as online and partition it manually, if this fails to happen automatically. 

     

  6. http://dl.covecube.com/CloudDriveWindows/beta/download/changes.txt

    It's not an RSS Feed, but it's updated every time there is a new build. Also, if you remove the "changes.txt" part, you can find all of the downloads.

     

    However, be warned, that sometimes, new builds can and will cause stability issues. Right now, the 441 build should be very stable. 

     

And we do hope that the Amazon Cloud Drives issues do get cleared up, and soon. But that depends entirely on Amazon.

Link to comment
Share on other sites

  • 0

 

1. However, this means that existing files on the provider cannot be added. 

 

What do you mean by this?

 

 

 

2. If you're using Amazon Cloud Drive (which you've indicated you are), we do "upload verification" on all chunks. This is REQUIRED to ensure that the file has been properly uploaded to Amazon and that they have records of the file (this is a known issue with every app using Amazon Cloud Drive).  This may be why your'e seeing the discrepancy. 

I am. So the md5 hash functionality that they provide from the file that you upload vs hashing it locally is not enough to verify the data in the chunk? I just don't see the point of having the in-file checksum, if you are able to use amazon's, at least for uploaded file verification.

 

 

4. However, Amazon Cloud Drive is limited to 2 upload and 4 download threads, due to various issues. You can read about our ongoing struggles to get Amazon Cloud Drive supported here:

I've read about your woes, hence me asking :P

 

 

5. If this is with Amazon Cloud Drive, this may be an issue with the provider specifically.

 

Currently its set to:

SAN Policy  : Offline Shared

One of the issues that I seem to be running into, is that CloudDrive attempts to upload data to ACD, however, it never seems to progress. I have it set for 100tb disk, 2 upload threads, 50mb chunk, and no cache locally or for chunks. Below is a bw monitor so I could easily get it in the same shot. 

This block usage is purely from formatting the drive, and nothing else. 

Bs9IZHy.png

oh6Wpmn.png

 

 

Is there a way to run the service in a debug mode or verbose of some sort so I can actually see whats going on to at least play around and see what will actually work? Maybe a log to tail?

Link to comment
Share on other sites

  • 0

WHat I mean by existing files cannot be added to the storage, is that we store raw blocks of data on the provider, and don't display the contents of the providers.

 

 

 

Upload Verification and Checksum Verification are two different things.

  • Upload Verification makes sure that the file is properly uploaded to the provider (that it exists and didn't get corrupted during upload). This is required by Amazon Cloud Drive. 
  • Checksum Verification ensures that the file didn't get corrupted, or otherwise changed while residing on the provider. THis isn't necessarily required, by the overhead is rather small, so we do recommend it. Especially for any provider that isn't entirely stable. 

 

 

As for the SAN policy, that's why, actually "OfflineShared" means that any drive on the same BUS is set as offline by default. Since all of the CloudDrive's are connected to the "Covecube Cloud Disk Enumerator" controller.

 

Try running "SAN Policy=onlineall" in diskpart. This should prevent this issue from happening in the future. 

 

 

 

As for the performance stuff with uploading .... this is rather common with the Amazon Cloud Drive provider, unfortunately. It's part of why it's an experimental provider.

 

As for monitoring, click on the "gear" icon, select "Troubleshooting" and select the "IO Threads" option. You can see the active threads, and what is going on. 

 

Additionally, there are logs in "C:\ProgramData\StableBit CloudDrive\Service\Logs". But these are text logs.

If you're familiar with the .etl format, you can enable drive tracing, and that will create the file system logs.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...