Posts Tagged Chef
While AWS was having its critical outage on April 21st some sites were more able to move their applications to alternative hosts (or even regions) than others. I can imagine the panic of operators unable to get their systems running on AWS and, at the same time, unable to deploy the application anywhere else due to the environment being fine tuned and baked in to the (somewhat) active deployment. We’ve all been there, where the production deployment has been running for so long, and is so manually configured that it is impossible to even know what to redeploy.
Using AWS to host applications generally forces us to consider automated deployment tools and processes but it seems that very few people take this seriously. Grace, our resident Chef expert, has spent many hours
beating Chef into submission getting Chef to work properly in a Windows environment (with enthusiastic help from OpsCode), which indicates that Chef is not extensively used in a Windows environment, and I doubt that Puppet or Capistrano are either.
If you have looked at the cost and determined that running hot standby in multiple regions is not for you, then you should look at automated deployment, as a minimum, in order to have some sort of recoverability for your applications. While, at least on AWS, it may not be true provider independence (due to reliance on AWS services) and it may be a bit tricky (due to different AMIs in different regions), it should be possible to have Chef scripts that allow you to redeploy to an alternative region (in AWS) or provider.
On the AWS platform, particularly if CloudFormation is also used, it should be relatively inexpensive to build and maintain scripts for multiple regions – at least cheaper than multi-region hosting. I reckon Grace could rustle up some CloudFormation configurations, Chef scripts and AMIs for our current project within a day or two – one person for a day or two every couple of months for recoverability peace of mind should be able to be squeezed in the budget. It is also easy and inexpensive to test – the pay per use model of AWS means that you can spin up your entire production platform in a different region, run some tests and shut it down again cheaply as the instances only need run for an hour or two.
While automated deployments don’t solve the whole problem (unfortunately the data still needs to be dealt with) it does go part of the way. Instead of bleating about how AWS let everybody down, we need to learn from these outages and review our use, or lack thereof, of the tools, patterns and practices that are available.
See Grace’s ‘Top 10 reasons for having a Configuration management solution‘, which was written the day before the AWS outage, for more thoughts on why automated deployment is important.