Posts Tagged SimpleDB
Following my post yesterday about the SimpleDB outage that nobody seemed to notice, I thought a bit about why people aren’t using SimpleDB as much as other AWS services. DISCLAIMER: These are personal observations and not based on extensive research or speaking directly to AWS customers. This list also assumes the people in question have kicked the SQL ACID habit. Why people are still using SQL instead of SimpleDB is a completely separate list.
1. The 1K limit for values is too small for many scenarios. While it makes sense in terms of availability to have limited field value lengths, practically it means that SimpleDB won’t work in cases where there is a piece of data (say a product description even) that is of indeterminate length. And the option of putting that little piece of data in to S3 just makes things harder than they should be.
2. You don’t know how much it is going to cost. In time people will become more familiar with an arbitrary measurement unit (such as SimpleDB Machine Hour) for the cost of utility compute resources, but at the moment it is a bit of a wild guess. It gets even more difficult when you think that the ‘Machine Hours’ that are consumed depend on the particular query and the amount of data – meaning that you need to get all of your data in before you can get an idea of the costs. On the other hand you can quickly get an idea of how much data you can processes using something like MongoDB on an EC2 instance – which is an easier cost to figure out.
3. Backups are not built in. All data has to be backed up and even if SimpleDB data is redundant, you still need backups for updates or deletes that may be made by mistake. Currently you have to roll your own backup or use a third party tool. SimpleDB needs a snapshot incremental backup mechanism that backs up to AWS infrastructure (S3 or EBS). This should be simple, quick, low latency (within the AWS data centre) and offered as part of the AWS toolset.
4. Pain free data dumps are needed. Similar to backup, getting data out of SimpleDB needs to be simple and part of the AWS products. A simple API call or web console click to dump a domain to S3, compressed and in a readable format (say JSON or xml) would go a long way toward people feeling less nervous about putting all their data in SimpleDB.
5. The API libraries are simplistic. MongoDB has really good libraries that handle serialisation from native classes to the MongoDB format (BSON), making it feel like a natural extension to the language. SimpleDB libraries still require that you write your own serializer or code to ‘put attributes’. This makes the friction higher for developers to adopt as there is a whole lot of hand rolled mapping that needs to be done.
6. SimpleDB documentation is anaemic. The SimpleDB query language is SQL-ish, but not quite SQL. I have had to look up and scratch around to try and translate from a SQL statement in my head into the SimpleDB variant – without examples built by AWS themselves. Just describing the API is not good enough, AWS needs a lot more documentation to help people build stuff.
7. There are no big reference sites. Various NoSQL databases are popular because of the sites that are powered by them. You can find good examples for MongoDB, CouchDB, Redis or any other NoSQL platform out there. Where is the SimpleDB flagship? And don’t tell me that Amazon.com is one without saying much about how it is being used.
This says nothing about how good SimpleDB is at its primary functions – availability, durablility, scalability, consistency and all those other important things that you need in data storage. It is simply a list of things that you would expect from a database product that are obviously lacking in SimpleDB and these things that are lacking hinder adoption. It may also not be that obvious to a seasoned user that things are missing as they are long over the architectural consideration and learning curve.
Fortunately there are other storage options on AWS. S3 and RDS are great products and very popular. There are also other databases that can run on EC2 and may migrate into RDS one day. So very few applications need SimpleDB, but they may be missing out on something. I wish AWS would put some finishing touches on their product.
I barely noticed the SimpleDB outage that occurred on 13 June and lasted for a couple of hours. The lack of outcry on the interwebs is an indication that not many people use SimpleDB – unlike when EBS fell over in April 2011 and knocked over most of the startup world.
When architecting cloud solutions, you build things up based on assumptions about availability and performance of certain components. Even though we have to design for failure, we have to factor in the likelihood of failure when coming up with solutions. We are quite happy, for example, to assume that our web servers will fall over (or terminate if they are being scaled down) but the load balancer less so and we build accordingly. Can you imagine how much more difficult it would be to build an application where fundamental components, such as the DNS server or load balancer were unreliable?
SimpleDB is one of those components that should be more reliable than anything you can build yourself on EC2. On a recent project we place logging and error information in SimpleDB (and S3), despite having both RDS MySQL and MongoDB available. The argument being that logging to MongoDB or MySQL doesn’t help if the problem is a database connection and besides, because SimpleDB has a restful interface, so the entire set of application components could be down and SimpleDB can be queried. This decision was also made on assumptions about the availability, with no design or operational overhead, of SimpleDB. Does this assumption need to be re-assessed?
The AWS outages are becoming worryingly frequent. Without getting into all the complex statistics of the Mean Time Before Failure of components in series the frequent failure of individual components (EBS in April and SimpleDB in June) means that the more components you use, the more likelihood of failure occurring. So while EBS or SimpleDB may not fail again for a while, if you have a dependency on the next one, whatever that may be, your average time before failure of your application looks a bit on the low side.
I hope that AWS continues their openness in (eventually) communicating their outages, but the frequency is worrying. Is AWS succumbing to their own success and unable to handle the demand? Is there a fundamental flaw with their architecture that is showing cracks under load? Are their processes unable to scale, so they aren’t able to monitor problems in order to put remedial actions in place before failure occurs?
Let’s hope that they can sort things out. AWS is not necessarily any more or less reliable than their competitors and is definitely more reliable that your average enterprise data centre. But bad news is bad news and we don’t want to design for too much failure nor should we be feeding the private cloud trolls.