Posts Tagged MapReduce
Dryad, Microsoft’s MapReduce implementation, has finally found it’s way out of Microsoft Research and is now open to a public beta. Surprisingly it is a quiet blog announcement, with no accompanying name change that is typical of Microsoft such as ‘Windows Server 2008 R2 High Performance Distributed Compute Services’ – or something equally catchy. Unsurprisingly for an enterprise software vendor, Dryad is limited to Windows HPC (High Performance Compute) Server – which means high-end tin and expensive licences. While most of the MapReduce secret sauce for the Dryad implementation probably comes from the HPC OS, it is disappointing that there is still no generally available MapReduce on the Microsoft stack on the horizon.
I recall more that a year ago wishing for Dryad (and DryadLINQ) on Azure to query Azure Table Storage (the Azure NoSQL data store) and generally thinking that Azure would get some MapReduce functionality as a service (like Amazon Elastic MapReduce – a Hadoop implementation on AWS) out of the Dryad project. But it seems that Microsoft is focussing on the enterprise MapReduce market for now.
I’m not that sure about the market for enterprise MapReduce and defer to the experts, wherever they may be. I thought that MapReduce was about using cheap resources (commodity hardware and open source licences) in order to scale out compute power. Surely if you have to pay Microsoft and high end hardware tax the case for scale out drops and you are better off just scaling up already expensive servers? I am sure the research market will still go for Hadoop or similar, and maybe Microsoft sees something in the deep pockets of financial services.
Had Microsoft brought Dryad to Azure, as ‘MapReduce as a Service’ then there would be something worth looking into on the Microsoft stack, but until then MapReduce for the masses is likely to remain on Hadoop. Hadoop is the de facto MapReduce implementation for non-specialists and as MapReduce gains traction as a way of solving problems, Microsoft will find it impossible to catch up.