Jump to content
  • 0

Amazon Cloud Drive, near future.?


steffenmand

Question

Hi,

 

Great product you have here i'll definately be a customer as soon as Amazon Cloud Drive gets working.

 

Just wanted to get some insight to expected timeframes..

 

 

1.

I could read that you got pulled out of production due to a single user running with 50 mb/s - couldn't you limit this on your end, making sure that no user can reach that again ?

 

2.

What is the possibility that more clients can get attached to the same cloud ? We often access stuff on several computers and it would be nice to have them all connected at once, instead of just detaching and attaching every time.

 

3.

What speeds can be expected pr. thread when in production (i know it varies, but currently im getting around 2 mbps and would be great to know if that could increase)

 

4.

Amazon Cloud Drive a priority? just so we know if time is being spent on getting this to work - i bet we all love the unlimited space :)

 

Thanks

Link to comment
Share on other sites

Recommended Posts

  • 0

  1. This wasn't due a single issue.  At least not on our end.  Amazon demoted our status back to development status because they're server's couldn't handle the connection speed. So instead of fixing it at our end, or giving us any sort of guideline of what was acceptable in terms of speed, they just demoted our account status. 

     

    Basically, this indicates that Amazon absolutely has an infrastructure issue in regards to Amazon Cloud Drive.  If you check their dev forums, it's definitely not just us with issues. 

     

    Their service (Amazon Cloud Drive) is broken. Period.  

     

    Additionally, there are significant data integrity issues with Amazon Cloud Drive.  So instead of fighting to get a version that may work ... we're depreciating the provider until such a time that Amazon gets their act together. 

     

  2. Depends on how you mean. 

    However, only one PC can access the cloud drive at one time, due to potential corruption issues. The only way around that would be to host intermediary servers. Which is expensive, time consuming, and ... well the opposite of our goals.

     

    However, if all the other systems are local, you can use Windows File Sharing on the drive, and share it between the systems. 

     

  3. Well, the example/reference you mentioned shows that our produce scales really, really well.  50MB/s is 400mbps. So, we can definitely hit high speeds!

     

    As for performance, it's ~1mbps per thread, I think. I'm not 100% sure, and I think this varies per system, basically. 

     

    Right now, the internal beta builds are limited to 2 threads, because Amazon Cloud Drive is broken.  It's not worth the time (we've already spent a great deal of it) to try to work around Amazon Cloud Drive's issues just to have them changed again and again, right now.  And that's likely to happen. 

     

  4. No. Not right now.  Once we're promoted back up to Production status (which took a couple of months in the FIRST PLACE), we'll re-evaluate the provider.  For now, it will be marked as experimental, and disabled by default. 

     

     

Alex plans on doing a post about this, explaining the issue (in a bit more detail).  But ... basically Amazon Cloud Drive is broken. It's not usable for us, and there isn't a lot we can do. 

 

This is very disappointing to us, because it is definitely the cheapest solution among all of the providers.  And we really wish it did work right. 

Link to comment
Share on other sites

  • 0

Really sad.. have you done some fixes to get your status changed again?

 

Guess one of the major issues could be the thousands of api calls due to the small chunk size and multiple threads. This could be solved by increasing the chunk size and limiting the amount of threads.

 

Look forward to reading the details about as this is really sad news for me :-( Stablebit Cloud Drive really is a great upcoming product and the cheap cloud space would really make it awesome product so it is both a loss for amazon and you as well!

Link to comment
Share on other sites

  • 0

Yes, we have. 

 

We've contacted them frequently about a number of issues. Not just our status.  We've received very few replies from them.

 

And some of the issues are corrupted data on the server, or data just "disappearing".  That's a server side issue, completely.  And it is why we had enabled "upload verification" for Amazon Cloud Drive already. 

 

They also recently changed the API... so that any data (regardless of how we classified it) was getting read, the first section analyzed and files tagged as video/pictures/music based on this "header information". Since all of our data is binary.... and flagged as such, it SHOULD NOT BE HANDLED this way. 

 

There are a number of issues, not just one or two, here.  And since things are clearly in flux, .... the question is "is it worth fighting every step of the way or is it worth stepping back for a while".  

 

 

 

 

 

As for the API... that's their issue. Not ours. We're using the API correctly. And small chunks are normal. How big are you image files? Music files? Documents?  

 

https://developer.amazon.com/public/apis/experience/cloud-drive/content/developer-guide

 

And when creating a drive, you can specify the chunk size. You can increase this when creating a drive, if you want. 


Look forward to reading the details about as this is really sad news for me :-( Stablebit Cloud Drive really is a great upcoming product and the cheap cloud space would really make it awesome product so it is both a loss for amazon and you as well!

Absolutely agree.  We really do want to work with Amazon and their Cloud Drive service. But ... it's just not feasible at this time.

 

As I've said, we've sent Amazon a bunch of emails about various issues we've encountered.  "We'll look into it" is what we've gotten back, when we've gotten a response.

Link to comment
Share on other sites

  • 0

While I would agree that the response times from questions sent to Amazon's support team can be a bit on the long side, I'm not sure I'd characterize their Cloud Drive service as a whole as broken.  I've been developing against it for months now.  Yes, I've seen some of the issues that you've reported in these forums, but those were eventually addressed.  Every app's mileage will vary, of course.  I can definitely say that we run our app on >300 mb/s connections with no issues, even when we were still at developer rate limits.  Yes, you have to have restraint on the number of concurrent threads you make from a single endpoint.  It is what it is.  If your app cannot work that way, then Cloud Drive probably isn't for you.  

 

I have no skin in this game.  I just wanted to clear up some misconceptions for your users.  

Link to comment
Share on other sites

  • 0

Their service isn't broken. Though it's not wholly reliable as well. 

 

The issues with it, however, are show stoppers for StableBit CloudDrive.  

 

 

And yes, we did do performance testing before releasing the public beta. 

 

And IIRC, Amazon doesn't specify an upper limit on bandwidth usage. If I'm wrong, please link me.

Link to comment
Share on other sites

  • 0

no specific use case as for files atm. but imagine 15 gb files or more in 1 mb chunks, they could be 5 mb or 10 mb chunks. with 1 mb as a limit (why not let the users choose?) you are bound to get tons and tons of calls.

 

about files that disappear i know that happens on both netdrive, expandrive and even amazons own desktop app sometimes, so it is an error on their side. Ive experienced it several times myself - in those apps the solution is to upload agsin. of course with chunks ot is more important to make sure that everything gets up correctly and i guess that upload verification is the way to go here.

 

them changing the status must be some excessive use of the api, bandwidth whatever and it sounds weird they wont specify it directly.

 

maybe amazon wants to limit the bandwidth depending on the type of file since they want to read the header and flag them, it wouldnt be possible to add a header to the chunks specifying the type? just if its required for them to know the type - while encrypted they cant read the contents anyway, which is the import thing for privacy

 

Link to comment
Share on other sites

  • 0

Actually, we do let users choose the chunk size. As for the limit to 1MB, I'd have to ask, but I'm pretty sure this is due to timing issues. The larger the chunk size, the more lag time you'll have to grab the files, and the more unresponsive the file system will be.

 

 

As for the missing chunks, that's why Amazon Cloud Drive is the only provider (IIRC) that has Upload Verification enabled by default. Specifically so it does what you mention. 

 

However, that it's losing files at ALL like this indicates a serious backend issue. It's something that Amazon needs to address, period. It's not something that developers should have to work around, IMO. 

 

 

 

 

 

them changing the status must be some excessive use of the api, bandwidth whatever and it sounds weird they wont specify it directly.

Alex (the developer) is the one that spoke with Amazon about this.  The user was using 50MB/s (400mbps) of bandwidth. That was the issue that caused them to demote us from production status to development status.   I suspect, in part because we weren't able to identify which user it was (there was some back and forth between alex and Amazon).... which is ironic, as you've probably seen that we require authorization from Amazon Cloud Drive.  

 

I can't speak to why they decided to demote us, but it really pulls the carpet out from under us.  While we're stuck on development status, our product's connection to Amazon Cloud Drive is severely throttled. The more users using it concurrently the worse it gets (which is why powerpad was able to get good speeds and most of our users can't). 

 

This is the heart of the issue though. We have too many people using it and we're in development  status.  There isnt' anything we can really do about either of these. 

 

 

 

maybe amazon wants to limit the bandwidth depending on the type of file since they want to read the header and flag them, it wouldnt be possible to add a header to the chunks specifying the type? just if its required for them to know the type - while encrypted they cant read the contents anyway, which is the import thing for privacy

 

No, they want to read the contents to categorize it.  IIRC, they specifically expanded the API and added the different categories when this started happening. They want to auto sort your data, based on the file header information. Rather than relying on the API or file name. 

 

I can understand why, but if they're going to do this, then they need have an "unsorted" and binary category. Unsorted gets read and binary is left alone.  

Link to comment
Share on other sites

  • 0

I get your problems - i just hope that the attention from an amazon employee now can help with the dialogue and get you to look into amazon again :-) its already by far better than existing apps and with increased speed it will pull users to amazon as well as new purchases of cloud drive or the bundle

Link to comment
Share on other sites

  • 0

I'm not really sure you can say for certain what the reason was that I was getting good speeds.  I was running concurrent tests that simulate your users' connections without an issue.  What I have have had issues with was making a large number of concurrent connections to their backend.  In my case, I throttle connections on my end.  For my use case, it doesn't really affect my users.  Your mileage will vary.  

Link to comment
Share on other sites

  • 0

I get your problems - i just hope that the attention from an amazon employee now can help with the dialogue and get you to look into amazon again :-) its already by far better than existing apps and with increased speed it will pull users to amazon as well as new purchases of cloud drive or the bundle

Once the production status has been restored, we absolutely plan on re-evaluating and updating the Amazon Cloud Drive provider (as needed). 

 

If only because it's the one that I've seen most used personally. Not surprising, all things considered. 

 

I'm not really sure you can say for certain what the reason was that I was getting good speeds.  I was running concurrent tests that simulate your users' connections without an issue.  What I have have had issues with was making a large number of concurrent connections to their backend.  In my case, I throttle connections on my end.  For my use case, it doesn't really affect my users.  Your mileage will vary.  

 

Exactly. I'm not sure why either.  Though, I don know that the more users use the development "version", the worse the issue gets. 

And if you're in production "status" for it, I can't comment on why we were flagged and you weren't. 

 

It's frustrating though. Because as I've said, we really do want it to work, and work well. (it clearly was when we did have production status for the short period). 

Link to comment
Share on other sites

  • 0

 

Apologies for the delay in response guys! I was on vacation, but I'm back now with some more information.

 

StableBit had some load issues that caused a few issues when they launched. Their production approval lasted about a week, but was then lowered. From what I understand, we reached out to their developers to work it out and never heard back unfortunately. This is a real shame as I can see a real need for this. I'll try and see if we can get in touch with them again; otherwise, feel free to nag them to contact us as we are more than willing to help resolve any issues they were having :D

 

Thanks!

 

Jamie

 

https://forums.developer.amazon.com/forums/thread.jspa?forumID=73&threadID=9355

Link to comment
Share on other sites

  • 0

Hey everyone,

 

First of all thanks for bugging Amazon about this :)

 

Basically, it's all up to them at this point. We need them to tell us what limits they want us to implement and we will implement them. I've already implemented some limits in the latest public BETA, but those were just guesses on my part.

 

I've sent multiple emails to them regarding this, and they have yet to reply to the last 2. I know that they definitely know my email address because they do answer sporadically, and when their servers started having issues, I got a response from them very quickly. Other than that, I can tell you that my past 2 emails have gone unanswered.

 

I've sent another email to them today, perhaps I'll get lucky.

 

I'm pretty frustrated with this whole process at this point. We've had to delay the current public BETA for months because of this one single issue. I think the delay was important because I do really want StableBit CloudDrive to support Amazon Cloud Drive. 

 

But then I find this on the Amazon forums, from Jamie@Amazon:

 

"StableBit had some load issues that caused a few issues when they launched. Their production approval lasted about a week, but was then lowered. From what I understand, we reached out to their developers to work it out and never heard back unfortunately."

 

In the heat of the moment, I did write a passionate response to Jamie outlining our entire communication with Amazon, or lack thereof.

 

In any case, I think in the end, we're going to have to face reality here. They may never answer us and they may never approve us. So going forward, we're moving to release the 1.0 final with or without Amazon Cloud Drive support.

Link to comment
Share on other sites

  • 0

I'm pretty sure the limits are quite high. I'm getting around 33 mb/s with netdrive (which is far inferior to Stablebit Cloud Drive). I think the issue must be around the max amount of threads and therefore the amount of api calls due to each thread making calls.

 

Thank god it was more people than me who showed an interest in getting Stablebit up and running which hopefully will lead to them starting to reply!

 

All in all, your product is awesome :)

Link to comment
Share on other sites

  • 0

I'm pretty sure the limits are quite high. I'm getting around 33 mb/s with netdrive (which is far inferior to Stablebit Cloud Drive). I think the issue must be around the max amount of threads and therefore the amount of api calls due to each thread making calls.

 

Thank god it was more people than me who showed an interest in getting Stablebit up and running which hopefully will lead to them starting to reply!

 

All in all, your product is awesome :)

Doesn't matter what they are. If they won't communicate with us and let us know what the limits *should* be, then we have to guess and HOPE they won't demote us to development status again.  Getting official word is better. But they'd still have to re-promote us to production status again..... 

 

 

But to Alex's point: we're more than willing to work with them. But they need to respond to their emails.  Communication only works if it's going in both directions. 

 

That .... or we EECB them (and that's .... well, not a good option). 

Link to comment
Share on other sites

  • 0

no luck in getting in contact with amazon yet ? :-)

 

I'm CC'ed in the conversations, so I should get notifications (assuming they don't remove the CC).... but the answer is "As much as usual". Which is to say, we'll get a flurry of them once another service at Amazon goes out ..... we hope...

Link to comment
Share on other sites

  • 0

They did respond, and Amazon has started examining the traffic patterns of the new BETA, so that's good news.

 

They also gave me some numbers in terms of what they expect to find, but right now it's not exactly clear to me as to what the limits that they proposed mean, so I asked for clarification.

 

I don't want to give out the actual numbers, because I'm not sure that it's something that Amazon wants to make public. But if the numbers that were given to me are a total limit for all of our users combined, then that would roughly mean that we could run a total of 10 drives across our entire customer base.

 

Obviously this is not ideal if that's the case, but I could be completely wrong here, so I'm awaiting clarification.

Link to comment
Share on other sites

  • 0

Sounds great! Then we are heading somewhere :)

 

I seriously don't hope that they limit bandwidth pr. app - it should be limited pr. user. Why should i get lower speeds because i use a tool for their drive, than using the drive itself.. no logic behind that!

 

You create an awesome product and it gets to shit because it was popular - no logic again :D

 

Netdrive must have a great deal of users and we still get good speeds (getting 30-32 mb/s upload and around 25 mb/s download), so must be something misunderstood between amazon and you guys :)

Link to comment
Share on other sites

  • 0

Well, we did get a response back from them, actually. 

 

Though, we're asking for clarification about a number of things (such as the number of API calls per second). 

 

So, some progress.  But yes, there will be some internal limitation on our end from the sounds of it (because Amazon can't seem to throttle properly at their end....)

 

 

 

And to clarify, the very low speeds in Amazon Cloud Drive right now is intentionally and whole "our fault". We want to make sure that the Amazon developers can properly test out the app, and so that all users can use it (albiet, slowly).  However, from the sounds of it, it may be a significant increase to the current speed. But we want to be cautious to make sure they don't limit us again.

Link to comment
Share on other sites

  • 0

Of course - i was just referring to the fact that Alex mentioned their numbers would result in 10 cloud drives tops - made no sence they would limit your bandwidth pr. app, this way all popular apps would end up being with crap speeds in the end.

 

I know the limitations are there now, but yea 2 mbps doesn't give you a hell lot of speed :) - look forward to seeing you getting things worked out with amazon, i'm just glad that the efforts of complaining to amazon helped getting this on track again!

Link to comment
Share on other sites

  • 0

Of course - i was just referring to the fact that Alex mentioned their numbers would result in 10 cloud drives tops - made no sence they would limit your bandwidth pr. app, this way all popular apps would end up being with crap speeds in the end.

 

I know the limitations are there now, but yea 2 mbps doesn't give you a hell lot of speed :) - look forward to seeing you getting things worked out with amazon, i'm just glad that the efforts of complaining to amazon helped getting this on track again!

Well, we'll let you know when we have confirmation.

 

From the sounds of it, it sounds like it may allow closer to 100MB/s up and down, which would be fantastic (as this is very close to native disk speeds!!)  But I could be interpreting wrong. 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...