Jump to content
Covecube Inc.

Alex

Administrators
  • Content Count

    249
  • Joined

  • Last visited

  • Days Won

    44

Everything posted by Alex

  1. Alex

    Mount Existing Files

    The reason why I chose this architecture for StableBit CloudDrive is because I think the primary #1 feature of StableBit CloudDrive is the full drive encryption. You set up your own encryption key, no one else knows it. We don't know it. Your cloud providers don't know it. This gives you absolute control over your data's security. Note that even if your cloud provider says that they're encrypting your drive, the point of control over that encryption is not under your supervision. So essentially you really don't know what's going on with that encryption. Perhaps some rogue employee may have access to those keys, etc... Given that type of approach, it made sense to store your data in opaque chunks. Letting you access your existing files in the cloud through a drive letter would require a totally different architecture of StableBit CloudDrive (and frankly, other application have already done this). I think we are relatively unique in our approach (at least I don't know of any other products in our price range that do what we do). I guess the way that I see StableBit CloudDrive is really TrueCrypt for the cloud (although, admittedly, we're not open source). The idea is that you create an opaque container, store it in the cloud, and have confidence that it's secure. Or you can do the same locally, but the architecture is optimized for the cloud.
  2. I've noticed that in the Google Drive thread some people are having issues, so I'm definitely going to be testing Google Drive more thoroughly, that's for sure.
  3. Some people have mentioned that we're not transparent enough with regards to what's going on with Amazon Cloud Drive support. That was definitely not the intention, and if that's how you guys feel, then I'm sorry about that and I want to change that. We really have nothing to gain by keeping any of this secret. Here, you will find a timeline of exactly what's happening with our ongoing communications with the Amazon Cloud Drive team, and I will keep this thread up to date as things develop. Timeline: On May 28th the first public BETA of StableBit CloudDrive was released without Amazon Cloud Drive production access enabled. At the time, I thought that the semi-automated whitelisting process that we went through was "Production Access". While this is similar to other providers, like Dropbox, it became apparent that for Amazon Cloud Drive, it's not. Upon closer reading of their documentation, it appears that the whitelisting process actually imposed "Developer Access" on us. To get upgraded to "Production Access" Amazon Cloud Drive requires direct email communications with the Amazon Cloud Drive team. We submitted our application for approval for production access originally on Aug. 15 over email: On Aug 24th I wrote in requesting a status update, because no one had replied to me, so I had no idea whether the email was read or not. On Aug. 27th I finally got an email back letting me know that our application was approved for production use. I was pleasantly surprised. On Sept 1st, after some testing, I wrote another email to Amazon letting them know that we are having issues with Amazon not respecting the Content-Type of uploads, and that we are also having issues with the new permission scopes that have been changes since we initially submitted our application for approval. No one answered this particular email until... On Sept 7th I received a panicked email from Amazon addressed to me (with CCs to other Amazon addresses) letting me know that Amazon is seeing unusual call patterns from one of our users. On Sept 11th I replied explaining that we do not keep track of what our customers are doing, and that our software can scale very well, as long as the server permits it and the network bandwidth is sufficient. Our software does respect 429 throttling responses from the server and it does perform exponential backoff, as is standard practice in such cases. Nevertheless, I offered to limit the number of threads that we use, or to apply any other limits that Amazon deems necessary on the client side. I highlighted on this page the key question that really needs answered. Also, please note that obviously Amazon knows who this user is, since they have to be their customer in order to be logged into Amazon Cloud Drive. Also note that instead of banning or throttling that particular customer, Amazon has chosen to block the entire user base of our application. On Sept. 16th I received a response from Amazon. On Sept. 21st I haven't heard anything back yet, so I sent them this. I waited until Oct. 29th and no one answered. At that point I informed them that we're going ahead with the next public BETA regardless. Some time in the beginning of November an Amazon employee, on the Amazon forums, started claiming that we're not answering our emails. This was amidst their users asking why Amazon Cloud Drive is not supported with StableBit CloudDrive. So I did post a reply to that, listing a similar timeline. One Nov. 11th I sent another email to them. On Nov 13th Amazon finally replied. I redacted the limits, I don't know if they want those public. On Nov. 19th I sent them another email asking to clarify what the limits mean exactly. On Nov 25th Amazon replied with some questions and concerns regarding the number of API calls per second: On Dec. 2nd I replied with the answers and a new build that implemented large chunk support. This was a fairly complicated change to our code designed to minimize the number of calls per second for large file uploads. You can read my Nuts & Bolts post on large chunk support here: http://community.covecube.com/index.php?/topic/1622-large-chunks-and-the-io-manager/ On Dec. 10th 2015, Amazon replied: I have no idea what they mean regarding the encrypted data size. AES-CBC is a 1:1 encryption scheme. Some number of bytes go in and the same exact number of bytes come out, encrypted. We do have some minor overhead for the checksum / authentication signature at the end of every 1 MB unit of data, but that's at most 64 bytes per 1 MB when using HMAC-SHA512 (which is 0.006 % overhead). You can easily verify this by creating an encrypted cloud drive of some size, filling it up to capacity, and then checking how much data is used on the server. Here's the data for a 5 GB encrypted drive: Yes, it's 5 GBs. To clarify the error issue here (I'm not 100% certain about this), Amazon doesn't provide a good way to ID these files. We have to try to upload them again, and then grab the error ID to get the actual file ID so we can update the file. This is inefficient and would be solved with a more robust API that included a search functionality, or a better file list call. So, this is basically "by design" and required currently. -Christopher Unfortunately, we haven't pursued this with Amazon recently. This is due to a number of big bugs that we have been following up on. However, these bugs have lead to a lot of performance, stability and reliability fixes. And a lot of users have reported that these fixes have significantly improved the Amazon Cloud Drive provider. That is something that is great to hear, as it may help to get the provider to a stable/reliable state. That said, once we get to a more stable state (after the next public beta build (after 1.0.0.463) or a stable/RC release), we do plan on pursuing this again. But in the meanwhile, we have held off on this as we want to focus on the entire product rather than a single, problematic provider. -Christopher Amazon has "snuck in" additional guidelines that don't bode well for us. https://developer.amazon.com/public/apis/experience/cloud-drive/content/developer-guide Don’t build apps that encrypt customer data What does this mean for us? We have no idea right now. Hopefully, this is a guideline and not a hard rule (other apps allow encryption, so that's hopeful, at least). But if we don't get re-approved, we'll deal with that when the time comes (though, we will push hard to get approval). - Christopher (Jan 15. 2017) If you haven't seen already, we've released a "gold" version of StableBit CloudDrive. Meaning that we have an official release! Unfortunately, because of increasing issues with Amazon Cloud Drive, that appear to be ENTIRELY server side (drive issues due to "429" errors, odd outages, etc), and that we are STILL not approved for production status (despite sending off additional emails a month ago, requesting approval or at least an update), we have dropped support Amazon Cloud Drive. This does not impact existing users, as you will still be able to mount and use your existing drives. However, we have blocked the ability to create new drives for Amazon Cloud Drive. This was not a decision that we made lightly, and while we don't regret this decision, we are saddened by it. We would have loved to come to some sort of outcome that included keeping full support for Amazon Cloud Drive. -Christopher (May 17, 2017)
  4. There was an OAuth 2.0 fix that was mainly affecting Dropbox (but it may have affected Google Drive as well). The fix was applied in 1.0.0.409. Download: http://dl.covecube.com/CloudDriveWindows/beta/download/ I haven't done any stress testing of this provider yet, which is on my todo list. Mainly I want to see how it performs with ~900 GB of data uploaded, and if there's anything that can be done (or should be done) to optimize it.
  5. Thank you for the sentiment. What I was thinking at the time, and I mentioned this to Christopher... At the heart of the problem is that our product scales... it scales really well.
  6. In other news, this was the last major bug that I have on my radar for the StableBit CloudDrive BETA (i.e. in our bug tracking system). So I'm going to be performing any final testing and prepping for a 1.0 release final. If anyone does find anything major beyond this point, please let us know (https://stablebit.com/Contact). Thanks,
  7. Thank you very much for reporting this. This was actually multiple nasty bugs which got introduced in the latest changes to the way that StableBit CloudDrive stores and loads secrets. But they should now be fixed in the latest internal BETA (1.0.0.409). In addition, there was an issue with supporting OAuth 2.0 providers that don't issue a refresh token. I believe this was Dropbox only for now, that's why this affected that provider specifically. Download: http://dl.covecube.com/CloudDriveWindows/beta/download/ I'm still testing the fixes to make absolutely sure that they're 100% effective, so this is hot off the press so to speak. Once you install this new build, you will have to Reauthorize your Dropbox cloud drives (only once this time). Issue: https://stablebit.com/Admin/IssueAnalysis/21843
  8. If you're not familiar with dpcmd.exe, it's the command line interface to StableBit DrivePool's low level file system and was originally designed for troubleshooting the pool. It's a standalone EXE that's included with every installation of StableBit DrivePool 2.X and is available from the command line. If you have StableBit DrivePool 2.X installed, go ahead and open up the Command Prompt with administrative access (hold Ctrl + Shift from the Start menu), and type in dpcmd to get some usage information. Previously, I didn't recommend that people mess with this command because it wasn't really meant for public consumption. But the latest internal build of StableBit DrivePool, 2.2.0.659, includes a completely rewritten dpcmd.exe which now has some more useful functions for more advanced users of StableBit DrivePool, and I'd like to talk about some of these here. Let's start with the new check-pool-fileparts command. This command can be used to: Check the duplication consistency of every file on the pool and show you any inconsistencies. Report any inconsistencies found to StableBit DrivePool for corrective actions. Generate detailed audit logs, including the exact locations where each file part is stored of each file on the pool. Now let's see how this all works. The new dpcmd.exe includes detailed usage notes and examples for some of the more complicated commands like this one. To get help on this command type: dpcmd check-pool-fileparts Here's what you will get: dpcmd - StableBit DrivePool command line interface Version 2.2.0.659 The command 'check-pool-fileparts' requires at least 1 parameters. Usage: dpcmd check-pool-fileparts [parameter1 [parameter2 ...]] Command: check-pool-fileparts - Checks the file parts stored on the pool for consistency. Parameters: poolPath - A path to a directory or a file on the pool. detailLevel - Detail level to output (0 to 4). (optional) isRecursive - Is this a recursive listing? (TRUE / false) (optional) Detail levels: 0 - Summary 1 - Also show directory duplication status 2 - Also show inconsistent file duplication details, if any (default) 3 - Also show all file duplication details 4 - Also show all file part details Examples: - Perform a duplication check over the entire pool, show any inconsistencies, and inform StableBit DrivePool >dpcmd check-pool-fileparts P:\ - Perform a full duplication check and output all file details to a log file >dpcmd check-pool-fileparts P:\ 3 > Check-Pool-FileParts.log - Perform a full duplication check and just show a summary >dpcmd check-pool-fileparts P:\ 0 - Perform a check on a specific directory and its sub-directories >dpcmd check-pool-fileparts P:\MyFolder - Perform a check on a specific directory and NOT its sub-directories >dpcmd check-pool-fileparts "P:\MyFolder\Specific Folder To Check" 2 false - Perform a check on one specific file >dpcmd check-pool-fileparts "P:\MyFolder\File To Check.exe" The above help text includes some concrete examples on how to use this commands for various scenarios. To perform a basic check of an entire pool and get a summary back, you would simply type: dpcmd check-pool-fileparts P:\ This will scan your entire pool and make sure that the correct number of file parts exist for each file. At the end of the scan you will get a summary: Scanning... ! Error: Can't get duplication information for '\\?\p:\System Volume Information\storageconfiguration.xml'. Access is denied Summary: Directories: 3,758 Files: 47,507 3.71 TB (4,077,933,565,417 File parts: 48,240 3.83 TB (4,214,331,221,046 * Inconsistent directories: 0 * Inconsistent files: 0 * Missing file parts: 0 0 B (0 ! Error reading directories: 0 ! Error reading files: 1 Any inconsistent files will be reported here, and any scan errors will be as well. For example, in this case I can't scan the System Volume Information folder because as an Administrator, I don't have the proper access to do that (LOCAL SYSTEM does). Another great use for this command is actually something that has been requested often, and that is the ability to generate audit logs. People want to be absolutely sure that each file on their pool is properly duplicated, and they want to know exactly where it's stored. This is where the maximum detail level of this command comes in handy: dpcmd check-pool-fileparts P:\ 4 This will show you how many copies are stored of each file on your pool, and where they're stored. The output looks something like this: Detail level: File Parts Listing types: + Directory - File -> File part * Inconsistent duplication ! Error Listing format: [{0}/{1} IM] {2} {0} - The number of file parts that were found for this file / directory. {1} - The expected duplication count for this file / directory. I - This directory is inheriting its duplication count from its parent. M - At least one sub-directory may have a different duplication count. {2} - The name and size of this file / directory. ... + [3x/2x] p:\Media -> \Device\HarddiskVolume2\PoolPart.5823dcd3-485d-47bf-8cfa-4bc09ffca40e\Media [Device 0] -> \Device\HarddiskVolume3\PoolPart.6a76681a-3600-4af1-b877-a31815b868c8\Media [Device 0] -> \Device\HarddiskVolume8\PoolPart.d1033a47-69ef-453a-9fb4-337ec00b1451\Media [Device 2] - [2x/2x] p:\Media\commandN Episode 123.mov (80.3 MB - 84,178,119 -> \Device\HarddiskVolume2\PoolPart.5823dcd3-485d-47bf-8cfa-4bc09ffca40e\Media\commandN Episode 123.mov [Device 0] -> \Device\HarddiskVolume8\PoolPart.d1033a47-69ef-453a-9fb4-337ec00b1451\Media\commandN Episode 123.mov [Device 2] - [2x/2x] p:\Media\commandN Episode 124.mov (80.3 MB - 84,178,119 -> \Device\HarddiskVolume2\PoolPart.5823dcd3-485d-47bf-8cfa-4bc09ffca40e\Media\commandN Episode 124.mov [Device 0] -> \Device\HarddiskVolume8\PoolPart.d1033a47-69ef-453a-9fb4-337ec00b1451\Media\commandN Episode 124.mov [Device 2] ... The listing format and listing types are explained at the top, and then for each folder and file on the pool, a record like the above one is generated. Of course like any command output, it could always be piped into a log file like so: dpcmd check-pool-fileparts P:\ 4 > check-pool-fileparts.log I'm sure with a bit of scripting, people will be able to generate daily audit logs of their pool Now this is essentially the first version of this command, so if you have an idea on how to improve it, please let us know. Also, check out set-duplication-recursive. It lets you set the duplication count on multiple folders at once using a file pattern rule (or a regular expression). It's pretty cool. That's all for now.
  9. They did respond, and Amazon has started examining the traffic patterns of the new BETA, so that's good news. They also gave me some numbers in terms of what they expect to find, but right now it's not exactly clear to me as to what the limits that they proposed mean, so I asked for clarification. I don't want to give out the actual numbers, because I'm not sure that it's something that Amazon wants to make public. But if the numbers that were given to me are a total limit for all of our users combined, then that would roughly mean that we could run a total of 10 drives across our entire customer base. Obviously this is not ideal if that's the case, but I could be completely wrong here, so I'm awaiting clarification.
  10. Hey everyone, First of all thanks for bugging Amazon about this Basically, it's all up to them at this point. We need them to tell us what limits they want us to implement and we will implement them. I've already implemented some limits in the latest public BETA, but those were just guesses on my part. I've sent multiple emails to them regarding this, and they have yet to reply to the last 2. I know that they definitely know my email address because they do answer sporadically, and when their servers started having issues, I got a response from them very quickly. Other than that, I can tell you that my past 2 emails have gone unanswered. I've sent another email to them today, perhaps I'll get lucky. I'm pretty frustrated with this whole process at this point. We've had to delay the current public BETA for months because of this one single issue. I think the delay was important because I do really want StableBit CloudDrive to support Amazon Cloud Drive. But then I find this on the Amazon forums, from Jamie@Amazon: In the heat of the moment, I did write a passionate response to Jamie outlining our entire communication with Amazon, or lack thereof. In any case, I think in the end, we're going to have to face reality here. They may never answer us and they may never approve us. So going forward, we're moving to release the 1.0 final with or without Amazon Cloud Drive support.
  11. Some years ago, when I first got the idea for StableBit CloudDrive, as cool as the concept sounded technically, I started thinking about its use cases. What would be the reason to use StableBit CloudDrive over any of the various "drive" applications that you can get directly from a cloud storage provider? Of course there are some obvious reasons, for example, StableBit CloudDrive is a real drive, it has an intelligent adaptive cache that stores the most frequently accessed data locally, but the main reason which really drove the overall design of StableBit CloudDrive is full drive encryption. If you want to ensure the privacy and the security of your data in the cloud, and who doesn't in today's security conscious climate, one option that you have is to have the cloud storage provider encrypt your data for you. That's exactly what many cloud storage providers are already doing, and it all sounds great in theory, but there's one fundamental flaw with this model. The service or company that is storing your data is also the entity which is responsible for ensuring the privacy and the security of your data. In some cases, or perhaps in most, they also provide a secondary way of accessing your data (a back door / key recovery mechanism), which can be misused. If there's a bad actor, or if the company is compelled to do so, they can unwillingly or willingly expose your data without your knowledge to some 3rd party or even publicly. I came to the conclusion, as I'm sure many other people have in the past, that trusting the storage provider with the task of ensuring the security and privacy of your data is fundamentally flawed. Now there are other solutions to this problem, like manually encrypting your files with a standalone encryption tool before uploading them or using a 3rd party backup application that supports encryption, but StableBit CloudDrive aims to solve this problem in the best technical way possible. By integrating full drive encryption for the cloud directly into the Operating System, you have the best of both worlds. Your encrypted cloud drive acts like a real disk, making it compatible with most existing Microsoft Windows applications, and it's fully encrypted. Moreover, you are in complete control of the encryption, not the cloud provider. Because full drive encryption is essential to securing your data in the cloud, let's get a little technical and let me show you how StableBit CloudDrive handles it: This is just a rough diagram to give you a top level overview and is not technically complete. Here are the basic steps involved when an application reads or writes data onto an encrypted cloud drive. A user mode application (or a kernel mode driver) issues a read or a write request to some file system. The file system issues a read or a write requests to the underlying disk driver, if the request can't be satisfied by the RAM Cache. For an encrypted cloud drive, any data that is passed into our disk driver is first encrypted, before anything is done to it. Any data that needs to leave our disk driver is decrypted as the last step before returning it to the file system. This means that any data on an encrypted cloud drive will always be encrypted both in the local cache and in the cloud. In fact none of the code involved in handling the I/O request even needs to know whether the data is encrypted or not. Decrypted data is never committed to your hard drive, unless the application itself saves the decrypted data to disk. For example, if you open a text file with Notepad on an encrypted cloud drive and then save that file to your desktop (which is not encrypted), then you've just committed a decrypted file to disk. The encryption algorithm that StableBit CloudDrive uses is AES-256 bit CBC, an industry standard encryption algorithm. The encryption API is Microsoft's CNG (https://msdn.microsoft.com/en-us/library/windows/desktop/aa376210(v=vs.85).aspx), which is FIPS 140-2 compliant. We definitely didn't want to reinvent the wheel when it comes to choosing how to handle the encryption. CNG is a core component of Microsoft Windows and has both a user mode component and a kernel mode component (upon which the user mode component is built). When you first create your cloud drive, you can choose among 2 methods of unlocking your encrypted drive. Either you can generate a random key, or you can enter a passphrase of varying length and have the system derive a key from the passphrase. If you choose to generate a key, we do so using a secure random number generator (CNG from user mode) and combine that data (using XOR) with data obtained from your mouse movements (which are gotten from your use of the application). This adds an extra layer of security, just in case the secure random number generator is discovered to be compromised in the future. The one and only key that is used to decrypt your data can optionally be derived from a passphrase that you enter. This key derivation is performed using a salted PBKDF2-HMAC-SHA512 (200,000 iterations) using the unmanaged CNG API from user mode. You can also choose to save the key locally for added convenience. If you choose to do this, we use the standard Microsoft Windows credential store to save a locally encrypted version of your key. Obviously if anyone ever gets physical access to your system, or obtains remote access, your key can be compromised. You should save your key locally only if you're not concerned with absolute security and want the added convenience of not having to enter your key on every reboot. Whatever unlock method you choose, you should always save your key to some permanent medium. You should print it out on a piece of paper or save it to an external thumb drive and store it in a secure place. Since only you have the key, it is absolutely critical that you don't lose it. Without your key, you will lose your data. Hopefully this post gave everyone a little insight as to why I think encryption is important in StableBit CloudDrive and how it's actually implemented.
  12. As I was writing the various providers for StableBit CloudDrive I got a sense for how well each one performs / scales and the various quirks of some cloud services. I'm going to use this thread to describe my observations. As Stablebit CloudDrive grows, I'm sure that my understanding of the various providers will improve as well. Also remember, this is from a developer's point of view. Google Cloud Storage http://cloud.google.com/storage Pros: Reasonably fast. Simple API. Reliable. Scales very well. Cons: Not the fastest provider. Especially slow when deleting existing chunks (e.g. when destroying a drive). Difficult to use and bloated SDK (a development issue really). This was the first cloud service that I wrote a StableBit CloudDrive provider for and initially I started writing the code against their SDK which I later realized was a mistake. I replaced the SDK entirely with my own API, so that improved the reliability of this provider and solved a lot of the issues that the SDK was causing. Another noteworthy thing about this provider is that it's not as fast as some of the other providers (Amazon S3 / Microsoft Azure Storage). Amazon S3 http://aws.amazon.com/s3/ Pros: Very fast. Reliable. Scales very well. Beautiful, compact and functional SDK. Cons: Configuration is a bit confusing. Here the SDK is uniquely good, it's a single DLL and super simple to use. Most importantly it's reliable. It handles multi-threading correctly and its error handling logic is straightforward. It is one of the few SDKs that StableBit CloudDrive uses out of the box. All of the other providers (except Microsoft Azure Storage) utilize custom written SDKs. This is a great place to store your mission critical data, I backup all of my code to this provider. Microsoft Azure Storage http://azure.microsoft.com/en-us/services/storage/ Pros: Very fast. Reliable. Scales very well. Easy to configure. Cons: No reasonably priced support option that makes sense. This is a great cloud service. It's definitely on par with Amazon S3 in terms of speed and seems to be very reliable from my testing. Having used Microsoft Azure services for almost all of our web sites and the database back-end, I can tell you that there is one major issue with Microsoft Azure. There is no one that you can contact when something goes wrong (and things seem to go wrong quite often), without paying a huge sum of money. For example, take a look at their support prices: http://azure.microsoft.com/en-us/support/plans/ If you want someone from Microsoft to take a look at an issue that you're having within 2 hours that will cost you $300 / month. Other than that, it's a great service to use. OneDrive for Business https://onedrive.live.com/about/en-US/business/ Pros: Reasonable throttling limits in place. Cons: Slow. API is lacking leading to reliability issues. Does not scale well, so you are limited in the amount of data that you can store before everything grinds to a halt. Especially slow when deleting existing chunks (e.g. when destroying a drive). This service is actually a rebranded version of Microsoft SharePoint hosted in the cloud for you. It has absolutely nothing to do with the "regular" OneDrive other than the naming similarity. This service does not scale well at all, and this is really a huge issue. The more data that you upload to this service, the slower it gets. After uploading about 200 GB, it really starts to slow down. It seems to be sensitive to the number of files that you have, and for that reason StableBit CloudDrive sets the chunk size to 1MB by default, in order to minimize the number of files that it creates. By default, Microsoft SharePoint expects each folder to contain no more than 5000 files, or else certain features simply stop working (including deleting said files). This is by design and here's a page that explain in detail why this limit is there and how to work around it: https://support.office.com/en-us/article/Manage-lists-and-libraries-with-many-items-11ecc804-2284-4978-8273-4842471fafb7 If you're going to use this provider to store large amounts of data, then I recommend following the instructions on the page linked above. Although, for me, it didn't really help much at all. I've worked hard to try and resolve this by utilizing a nested directory structure in order to limit the number of files in each directory, but nothing seems to make any difference. If there are any SharePoint experts out there that can figure out what we can do to speed this provider up, please let me know. OneDrive https://onedrive.live.com/ Experimental Pros: Clean API. Cons: Heavily and unreasonably throttled. From afar, OneDrive looks like the perfect storage provider. It's very fast, reliable, easy to use and has an inexpensive unlimited storage option. But after you upload / download some data you start hitting the throttling limits. The throttling limits are excessive and unreasonable, so much so, that using this provider with StableBit CloudDrive is dangerous. For this reason, the OneDrive provider is currently disabled in StableBit CloudDrive by default. What makes the throttling limits unreasonable is the amount of time that OneDrive expects you to wait before making another request. In my experience that can be as high as 20 minutes to 1 hour. Can you imagine when trying to open a document in Microsoft Windows hitting an error that reads "I see that you've opened too many documents today, please come back in 1 hour". Not only is this unreasonable, it's also technically infeasible to implement this kind of a delay on a real-time disk. Box https://www.box.com/ At this point I haven't used this provider for an extended period of time to render an opinion on how it behaves with large amounts of data. One thing that I can say is that the API is a bit quirky in how it's designed necessitating some extra HTTP traffic that other providers don't require. Dropbox http://www.dropbox.com/ Again, I haven't used this provider much so I can't speak to how well it scales or how well it performs. The API here is very robust and very easy to use. One very cool feature that they have is an "App Folder". When you authorize StableBit CloudDrive to use your Dropbox account, Dropbox creates an isolated container for StableBit CloudDrive and all of the data is stored there. This is nice because you don't see the StableBit CloudDrive data in your regular Dropbox folder, and Stablebit CloudDrive has no way to access any other data that's in your Dropbox or any data in some other app folder. Amazon Cloud Drive https://www.amazon.com/clouddrive/home Pros: Fast. Scales well. Unlimited storage option. Reasonable throttling limits. Cons: Data integrity issues. I know how important it is for StableBit CloudDrive to support this service and so I've spent many hours and days trying to make a reliable provider that works. This single provider delayed the initial public BETA of StableBit CloudDrive by at least 2 weeks. The initial issue that I had with Amazon Cloud Drive is that it returns various errors as a response to I/O requests. These errors range from 500 Internal Server Error to 400 Bad Request. Reissuing the same request seems to work, so there doesn't appear to be a problem with the actual request, but rather with the server. I later discovered a more serious issue with this service, apparently after uploading a file, sometimes (very rarely) that file cannot be downloaded. Which means that the file's data gets permanently lost (as far as I can tell). This is very rare and hard to reproduce. My test case scenario needs to run for one whole day before it can reproduce the problem. I finally solved this issue by forcing Upload Verification to be enabled in StableBit CloudDrive. When this issue occurs, upload verification will detect this scenario, delete the corrupt file and retry the upload. That apparently fixed this particular issue. The next thing that I discovered with this service (after I released the public BETA) is that some 400 Bad Request errors spawn at a later time, long after the initial upload / verification step is complete. After extensively debugging, I was able to confirm this with the Amazon Cloud Drive web interface as well, so this is not a provider code issue, rather the problem actually occurs on the server. If a file gets into this state, a 400 Bad Request error is issued, and if you examine that request, the error message in the response says 404 Not Found. Apparently, the file metadata is there, but the file's contents is gone. The short story is that this service has data integrity issues that are not limited to StableBit CloudDrive in particular, and I'm trying to identify exactly what they are, how they are triggered and apply possible workarounds. I've already applied another possible workaround in the latest internal BETA (1.0.0.284), but I'm still testing whether the fix is effective. I am considering disabling this provider in future builds, and moving it into the experimental category. Local Disk / File Share These providers don't use the cloud, so there's really nothing to say here.
  13. Microsoft Azure was able to get our original database back and I migrated all of the missing data to the new database. As of 4/30/2015 2:55 PM EST we are now fully online with no data loss. Edit: Some discrepancies have cropped up in the migration process, I'm attempting to resolve them manually.
  14. StableBit.com is now back online, it was down for 1 hr and 48 mins. As we were approaching 2 hours of downtime, I decided to restore the database from a local backup that was taken about 8 hours ago. I will look into implementing a redundant fail-over database in the future, as that would be ideal in these kinds of situations. If you've placed an order within the past 8 hours (4/30/2015 6 AM EST to 2 PM EST) and you are having issues activating your product please contact us and we will be happy to assist you. https://stablebit.com/Contact Again, I sincerely apologize for any inconvenience that this may have caused you.
  15. StableBit.com is currently down due to a Microsoft Azure (SQL Databases) issue. Microsoft is actively working on resolving the problem right now. Unfortunately, as a result, this means that StableBit licensing is down as well. I will keep monitoring the situation, and if the worst happens and all of our licensing data is lost, we will have to restore from a local backup taken 6 hours ago. I apologize for any inconvenience that this may have caused.
  16. Alex

    SSD not emptying?

    Thanks for submitting those logs. I'd just like to describe here how the balancing process gets triggered: Periodically, StableBit DrivePool calculates the balance ratio for each pool. This ratio, along with the actual count of bytes that need to be moved, are then used to determine if a full balancing pass is necessary. The ratio calculation does not require any I/O and is therefore performed rather frequently. The ratio is calculated every 5 minutes or if X number of bytes are processed by the pool (100 MB by default -- but this can be changed using advanced settings). The ratio itself is a percentage from 0% to 100%. 0% - All the data on the pool needs to be moved. 100% - No data on the pool needs to be moved. The SSD Optimizer always "wants" to move all of the data from the SSDs to the Archive drives, whenever the ratio is calculated. However, because the relative size of the data that needs to be moved, compared to the size of the pool itself, the overall balancing ratio may be very close to 100%, or actually be 100% because of the floating point calculation used. This is really not ideal, so I've gone ahead and made a code change that will always force a balancing pass if the critical balance ratio is set to 100%. Maybe the plug-in itself should handle this better, I'll think about it. The internal build with this change is available now: http://wiki.covecube.com/Downloads So in short, in order to get the SSD Optimizer to move your files to the archive drives as soon as possible: Under Balancing -> Settings, set If the balancing ratio falls below to 100%. Make sure Balance immediately is selected. Make sure that Not more often than every is not checked. Just some more miscellany: Balancing ratio calculations normally happen automatically, but can be directly forced by opening the balancing window and clicking Save. This is not really necessary, but just FYI. StableBit DrivePool's balancing algorithm has a special "scrape" mode when emptying disks completely, this is what the SSD Optimizer triggers. The intention of the scrape mode is to get every last file off of that disk onto the archive disks. This is unlike the standard balancing algorithm which uses a "best effort" approach in order to decide which files to select for re-balancing (for performance reasons).
  17. Hi Bjur, (for tl;dr see the 2 point at the end) Getting the correct power state of the disk is a little tricky. There are really 2 separate mechanisms that control whether a disk has spun down or not, the OS and the disk's firmware. Here are the tricky parts: The disk's firmware can spin a drive down at any time without the OS's knowledge. But this is typically disabled on new drives. This behavior can be modified under Disk Control in the StableBit Scanner. In order to get the actual power state of the drive we can query it directly, instead of asking Windows. The problem here is that, to Windows, this appears as disk access and it will prevent the OS from ever spinning down the drive. What the StableBit Scanner does by default is it always asks the OS and it never queries the drive directly. This ensures that the OS will spin the drive down correctly. even though we're querying it for the power state. But the issue here is that just because the OS thinks that the drive is active doesn't mean that it's actually spun up. If the disk's firmware has spun down the drive, the OS has no way to know that. The StableBit Scanner deals with this by reporting in the UI that the drive is Standby or Active. Since we can't attempt to query the drive directly without your explicit permission (this will upset Windows' power management), this is the best answer we can give you. The Query power mode directly from disk option, which is found in Disk Settings, is there to work around this shortcoming. When enabled, the StableBit Scanner will attempt to query the power mode directly from the disk. Keep in mind that this can fail if it can't establish Direct I/O to the disk, in which case we fall beck to relying on the OS. The way it works is like this: Query the OS. If the disk has spun down then this must be the case. The disk is in Standby. If the disk is Active (spun up) then we can't trust the OS because the disk firmware could have spun it down. If the user has not explicitly allowed us to query the power mode from the disk, we must assume Standby or Active. If the user has allowed us to query the power mode from the disk, query the power mode. If the query succeeds, set the mode to Active or to Standby (not both, because we know the power state for sure). If the query fails, fall back to the OS and set the mode to Standby or Active. So when should you enable Query power mode directly from disk? When you don't want to use the OS's power management features to spin the disk down. Why would you do this? Pros: You can control the spin down on a per disk basis. You get exact disk power states being reported in the StableBit Scanner with no ambiguity. Avoid disk spin up issues when querying SMART (I will explain below). Cons: Requires Direct I/O to the disk. To the OS (and to any other program that queries the OS) the disk will appear to be always spun up. When the OS spins down a disk it flushes the file system cache prior to spinning it down. This ensures that the disk is not spun up very quickly after that because it needs to write some additional data to it from the cache. When the firmware spins a disk down, this does not happen and there is a chance that the disk will be spun up very quickly after that to perform another write. From my experience, this is not common in practice. What about S.M.A.R.T. queries? In the StableBit Scanner, by default, SMART is queried from WMI first. If Direct I/O is not available then all the SMART data has to come from WMI. Typically this would not spin up a disk. If Direct I/O is possible to the disk then at least some additional SMART data will come from Direct I/O (and if WMI doesn't have the SMART data then all of the SMART data comes from Direct I/O). One potential problem here is that Windows considers any communication with the disk, disk activity. So if you're communicating with the disk to retrieve SMART every couple of minutes then Windows will not spin the disk down. You can avoid this problem in 2 ways: Don't let Windows control your disk spin down and set up a Standby timer in Disk Control (this has the pros and cons as outlined above). Set Throttle queries in Settings -> Scanner Settings -> SMART -> Throttle queries. Set the throttle minutes to something higher than the Windows disk spin down time (which can be examined in Power Options under the Windows Control Panel). The option Only query during the work window or if scanning controls SMART queries and has no effect on power queries. Again, by default power queries do not spin up a disk unless you've manually enabled Query power mode directly from disk (in which case you are effectively saying that you don't want the OS to ever spin down a disk).
  18. Alex

    Can I unmount a pool?

    JazzMan, I've fixed the issue that you were seeing with the remeasuring of the incorrect pool. It was actually, unnecessarily, scheduling all the pools for a remeasure pass when any volume went missing. Thanks for the detailed reproduction. Here's a video showing the fixed behavior: http://screencast.com/t/nu7sTvaA7lvO The fix is applied in the latest internal BETA (2.2.0.569) and you can download it here: http://wiki.covecube.com/Downloads Regards,
  19. I'm not sure of this is what's happening here but I've seen something similar happening as I'm developing our next product. It seems that the Virtual Disk Service gets confused when multiple drives spawn one after another. This was on Windows 8.1 and when my driver spawned 3 virtual drives, one after another, the disks list in disk management (which uses VDS) showed duplicate drives with the same drive index (e.g. "Drive 3" would show up twice). Since StableBit DrivePool uses VDS as well, it would presumably also see that kind of duplication. I did confirm that this is a VDS issue whereby simply restarting the VDS service the problem went away. But what's more disconcerting about this is that for each duplicated drive, the fake clones were showing up as uninitialized raw drives. Now these drives were really fully initialized drives with data on them. When I attempted to initialize them, through the fake clone, the real drive lost all of its data. So at this point my jaw hit the floor. VDS has a bug whereby it can spawn more than one representation of a drive that really points to one physical drive. Moreover, attempting to initialize the fake clone drive will wipe the data off of the real drive. I'm still experimenting with this and will see if I could put code in to work around this issue.
  20. Alex

    File/Folder Placement

    Weezywee, You can't set up a single rule that would work over a range of letters like that, but you can set up multiple rules, one for each letter. This would be simpler than creating an individual rule for each folder. For example: Pattern \TV Shows\A*​D:\ ​Pattern \TV Shows\B*D:\ Pattern: \TV Shows\C*D:\ Pattern: \TV Shows\D*E:\ Pattern: \TV Shows\E*​E:\ Pattern: \TV Shows\F*​E:\ Setting up rules like that under the File Placement -> Rules tab I think should work for what you're trying to achieve. We've talked before about having more complex rules that involve regular expressions, but that would be complicated to implement and involve additional real-time overhead.
  21. The Serial Number is not easy to get at, but it's probably one of the best ways to uniquely identify disks.
  22. The RC is very close to final. So far there has only been one more fix and it's very specific having to do with iSCSI targets, which was an issue on Windows 2012 (not R2). It should be ok to go with the 2.1 RC. To upgrade to the release final (once that's out) you can run the installer and reboot.
  23. Yes, it has been quite a while since the last release final. The 2.0 final was released Oct 3 2013. But development is still very active and the latest 2.1.0.553 RC was just released on May 30th. To see all the changes since 2.0.0.420 you can check out the full change log right here: http://stablebit.com/DrivePool/ChangeLog?Platform=win The 2.1 release final build should be out within a week or two.
  24. Alex

    File placement

    Lee, In StableBit DrivePool there is no notion of "main" file parts and "duplicated" file parts. We just have some number of identical file parts for each file. Unduplicated files have one file part, and duplicated files have 2 or more. Where those file parts are placed can be controlled with file placement. If you tell the balancing system to place a 2x duplicated folder on one drive, it will show you a warning that some file parts could not be placed on that drive. For a 2x duplicated folder, you need to select at least 2 drives in file placement in order to be able to satisfy those rules, or else the system will have to violate the rules. But I still think that you can accomplish what you want. If I understand you correctly, you want all of your music to be duplicated and to play from only the "first" file part. You can achieve this by using file placement to restrict the music folder to at least 2 drives and then to turn off Read Striping (under Pool Options -> Performance -> Read striping). I've mentioned above that all file parts are the same, so how could there be a "first" file part? For the purposes of read striping, when dealing with files that have > 1 file part, StableBit DrivePool's file system (CoveFS) does maintain a deterministic pool part order. In other words, the disks part of the pool are ordered in an invisible order and this order persists even after a reboot. The point being, when read striping is off, and you open a duplicated file, the "first" file is always opened.
  25. Alex

    NFS, Drivepool, VMware

    This has been an open known issue for a while (https://stablebit.com/Admin/IssueAnalysis/82), but I haven't been able to address it yet. I'm setting up a brand new testing lab right now with all new hardware just for such difficult cases and will address this in the future. I've tried to tackle this in my current lab and it has proven to be most difficult so I've opted to wait until the new lab is set up. The new lab is coming along nicely and I'll get to resolving this once the new lab is up and running. TL;DR: It may take a few months.
×
×
  • Create New...