I’ve been thinking a lot about the cloud lately. Let me float a few of my latest ideas here:
The metacloud is inevitable.
The metacloud should- and almost surely will- be open in every way.
No one company/person will build the metacloud.
History will repeat itself.
What is this metacloud I speak of? Well, hang Silicon Valley up on a wall and throw a dart at it. Chances are you’ll hit a company trying to abstract public clouds, private clouds, or both to build a platform for the deluge of cloud-native, yet cloud-agnostic software everyone’s expecting. Enomaly, RightScale, cloud.com. . . the list goes on. And if you want to build your own metacloud, no worries. There’s an API for that. Dasein , libcloud, and even my own Simple Cloud API, for example. While Amazon, VMWare and Microsoft (I’ll spare you the links) feverishly add new features to their closed, proprietary clouds in an effort to differentiate their offerings, some small fries are working on smoothing over the differences.
This is where a good old Linux analogy comes in handy. Early on in the development of the kernel (version 1.2), Linus et. al. decided that portability was important. So they added support for Alpha, SPARC, and MIPS. In doing so, they freed Linux from the clutches of the x86 architecture and turned the hardware in to little more than an abstraction for developers working on top of the Linux platform.
The way I see things, EC2, Windows Azure, Rackspace Cloud Servers, and the rest of the gang are pimping processors. They add one feature after the next, while a bunch of startups are working on platforms that solidly hit the 80% of features and leave the developer/user blissfully unaware of where their virtual computers are actually running.
So you see where I’m going with this. The metacloud will be like Linux, abstracting out the differences in cloud computing architectures. And it’s inevitable. It’s just too useful not to be done.
But I have bad news for any metacloud developer who aspires to write the next Linux: Linux wasn’t built in a day, and it sure as hell wasn’t built by a single company. Its popularity came at the cost of almost total freedom. So any company could use it in any way, and many companies decided to back it by adding drivers and modules for their products that wouldn’t be there otherwise.
I believe the metacloud will be developed in the same way. Since API convergence is convenience in the cloud, I also believe that there will eventually be one API to rule them all (Yes! I finally found a place to bust out this reference where it actually makes sense!).
So, there’s a lesson to be learnt from history here. The vendors that made big bucks off of Linux did it by getting in on the community action and putting it to good use in their products, not by building it themselves.
That said, someone’s got to get the ball rolling.
,Wil
2 Comments
I like the intention behind your posting, but while I think it’s big on vision, it’s low on reality. So, I’ll provide a bit of a reality check here.
First off, penning new hyperbolic tags like ‘metacloud’, while a time-honored tradition in the tech industry, doesn’t really help make things any clearer, especially when it’s something that is relatively nebulous. And comparing it to Linux? Like some kind of ‘cloud OS’ just makes things even murkier. There is about zero chance this will happen any time soon. An OS is an abstraction for a piece of hardware. The abstractions that will evolve on top of cloud computing systems are a whole new ball game because of their distributed nature. They will behave quite a bit differently from hardware abstractions. Which brings me to point #2.
Second, the notion that common abstraction layers across different clouds makes developers lives easier is a treacherous notion. Whether you feel the clouds are pimping processors or not, the reality is that every single cloud today has fundamentally different architectures. For example, if you compare GoGrid, AWS, Rackspace Cloud, Azure, Savvis, and Hosting.com approaches to storage you will see a huge number of very different offerings, with different price points, different interfaces, and performance characteristics. You can’t just slap an abstraction on top of on-demand block devices, like AWS EBS, and call it a day. Not all block devices are made equal. Every layer of abstraction theoretically hides the underlying behavior for simplicity, but the flip side of that is that it hides differences in architecture that will cause a developer indigestion if they don’t understand them.
A much better abstraction than the simple libraries you mention, which simply provide a common programming interface and don’t handle architectural differences, are Platform-as-a-Service (PaaS) systems like Heroku and EngineYard. PaaS must deal with architectural differences in order to provide a truly repeatable development environment. This also means, unfortunately, that you wind up with an opinionated environment. That’s the only way to make these work successfully across clouds.
Put another way, the libraries you point to are more like simple circuits that can carry differing electrical currents. The clouds themselves while all providing electricity do so in different ways. Cloud A provides 50Hz 110VAC, cloud B provides 60Hz 208, etc. etc. The libraries you mention simple are wires for connecting these currents together, but you as the user must deal with the differences. PaaS are systems that have current converters in them. Meaning you’ll get 65Hz 120VAC from all of the clouds they support and it will be consistent, but if you need a different type of electricity you are SOL.
Finally, I think the notion of convergence on APIs is an old saw that hasn’t proven true. APIs, libraries, frameworks, and languages continue to proliferate at an astonishing pace. I agree that in many cases, standards evolve or win by fiat (e.g. HTML), but that appears to be the exception, not the rule. Most of the time when this occurs it’s because of market forces. Right now, we’re in the very early stages of the game. Will there be market pressure to converge over time? Certainly, but we’re probably talking 5-10 years out. TCP/IP, one of the hottest technology adoption curves ever, took 10-15 years to establish it’s dominance. We’re in like year 3 with no clear leader except perhaps Amazon, who have shown no or little reticence to allow copying of their architecture or APIs.
I applaud your intentions and passion, but while I think the vision is spot on, I think the tactical situation is much, much different from Linux. Linux could build on a single basic platform (x86) and branch from there. We don’t currently have a single open cloud that can be a reference design.
First off, I agree with almost everything you say. Taking it point by point. . .
Point #1: The name metacloud is actually appropriate in this case; I’m talking about a cloud of clouds. But you’re right in saying prefixes like ‘meta’ get thrown around a lot- often in completely inappropriate contexts- just to get attention. Whatever we call it, what I’m describing will likely come to fruition if history is any indication. Like other technologies built on top of multiple platforms, it will add a lot of value to the cloud as a whole.
The comparison to Linux is definitely far fetched. I needed an example of a technology built on multiple architectures that hides the anomalies among the architectures. So I picked the old standard: Linux. Maybe I could have done better; it’s easy to read more in to this analogy than I did.
Point #2: Actually, I have slapped an abstraction on file storage across clouds- http://simplecloud.org- and it has worked well in every application that I know of. Block devices are a very different beast that aren’t accessed through a web service, so I agree that there won’t be an easy abstraction there. In my vision, user exposure to underlying block devices would become less of an issue as storage services speed up enough to replace them in most storage use cases. And that’s a *long* way off. You’re right that most cloud technologies and services aren’t there yet. But enough of them are to start playing around with cloud portability.
You lose me on the electrical stuff. Damn it, Jim, I’m a software guy, not a hardware guy.
Final point: I’m out on a limb with the convergence point. Truth is, I just wanted to use the Lord of the Rings analogy.
) for the most part winning out over other OSs- save Windows Server if you put it in the same class, and HTML as you point out.
To be clear, I’m not talking about standards. I’m talking about ‘market forces’, as you put it well. And we do see examples of technology convergence: Eucalyptus implementing Amazon’s API in the cloud, on the OS side, Linux (here I go again
We are in the dawn of the cloud, so it’s true that the metacloud is a long way off. But the base technologies are already there for someone to start tinkering with it. I hope to find the time soon.
I’d love to hear what you think about my latest post: Programming the Metacloud.
Thanks for the insight.
,Wil
2 Trackbacks
[...] This post was mentioned on Twitter by wllm, Shanley Kane. Shanley Kane said: The Inevitable Metacloud: http://wp.me/pnB3F-1m (via @wllm) <— Clarity. It tastes so good. [...]
[...] The Inevitable Metacloud « wllm [...]