OpenStack 2014 somehow reminds me a lot of OMG/CORBA 1993. It's interesting to watch the industry repeat itself.
Checklist:
- Anybody but "x" club, where X is the market leader. Openstack has vascillated between VMware and AWS here, CORBA had Microsoft.
- Large investments advertised as a penis swinging contest between HP, IBM, and others
- Chasing aging technology architecture as a panacea instead of building the next generation.
CORBA was chasing distributed objects, but the Web and REST was what really was going to matter. Today the big guys are chasing the IaaS model when we have a new generation of platform and application clouds (Mesos, YARN, CoreOS, Docker, CF, Asgard) growing out there.
The vendors will of course try to recoup their investment through reference architectures and "best practices" that tell customers they should have a multi-layer SaaS or PaaS running on an IaaS, even though lightweight alternatives like CoreOS will chug along doing nothing of the sort.
This is similar to how CORBA's IIOP was promoted to be the way to do "real work" with C++ or Enterprise JavaBeans in your middle tier, but this "Web" HTTP thing can be relegated to the front tier. I know fixed income systems around 1999 that literally wouldn't use HTTP for their blotter updates, with the lead devs insisting such events needed to be pushed over IIOP even though the front end was an IE control in a VBX. (this was shortly before TIBCO had effectively seized the market data industry)
- persistent flame wars about governance and architecture of said standard on the social media of the day
This is not to say OpenStack is bad. There is value, potentially a lot of value, as there was with CORBA. That's all case-by-case. Just saying that vendor love fests rarely indicate a true revolution and most often are just an orchestrated marketing game by the current losers.
I'm a cloud architect (a person who assembles cloud computing systems from components). I haven't used OpenStack; I've looked at it and while the concept is fine, the implementation is pretty underwhelming. Corps just want it because if widely adopted, it will reduce their cost and increase their profit margins.
Don't forget the SOAP era; it also resembles CORBA. There was a whole region of time (~2002-2005) where Microsoft and IBM jumped on SOAP. They promoted the hell out of their stacks, and then, after everybody realized SOAP was a turd, put the projects in deprecated mode.
The next step in evolution was JSON payloads over HTTP, with methods defined using REST. This model has more or less survived for 10 years now, and is a community-created (rather than top-down-imposed by a corporation) system that has lots of experience under its belt. I expect only marginal changes at this point: move towards a more compact encoding (like BSON) and a bit more standardization on where the RPC method is defined. I expect REST itself, being so hard to understand, will only be used by people who really insist on getting their API surfaces really RESTy.
SOAP itself was Microsoft's attempt to wrest momentum away from Java, and get people to pay attention to .NET, which modestly succeeded.
It also was a the final turd that broke bullshit mountain's forces: ultimately, developers had to figure out what mattered more to the Web's success: was it tagged data (HTML/XML), and we could preserve this distributed component/object conceptual model? Or was it the combination of URI+HTTP with its uniform interface and hypermedia? The REST debate took a long time (through 2007) to answer that, even though in practice with JSON we just pretend to adopt the full architecture.
I think the next revolution in REST would be if a new media type and associated toolkits/libraries come out that helps people build simpler and evolvable systems than JSON does. There are a variety of experiments out there, but as you imply, it's hard work.
I too am a "self-declared" cloud architect...mainly Azure and Windows-based platforms for Fortune 500 companies that want MS-based cloud architectures for Exchange, SharePoint, MSSQL, and Dynamics. Sorry, but other than setting it up in a lab for my own interest and tinkering, I have not had a Client request OpenStack or any OSS if they are implementing a Microsoft technology-based cloud or hybrid-cloud architecture. Not to say F-500 companies don't use Open Source, but OSS is typically not utilized in every cloud or on-prem environment.
You're reasoning by analogy, but the analogy between CORBA and OpenStack is flawed.
CORBA is mainly a standard. A standard only succeeds if most people adopt it.
OpenStack is mainly a set of open source software. Users can pick and choose the pieces to use in their system. If "OpenStack Governance" decides that Software X should be used for job Y, but something better (and open source) comes along, people can just switch to the better software. In other words, competition can happen at the component level and it can come from the grassroots (without permission from the big wigs). It's much different from everyone having to agree on one standard.
I don't need to tell you that there are many thriving open source software ecosystems. Make an analogy with one of them and OpenStack makes much more sense (e.g. Linux and its packages, Python and its packages, WordPress and its themes/plugins).
The analogy is not 1:1, but consider a few observations.
Firstly, OpenStack is not a thriving open source community in the usual sense. It has succeeded in creating a vendor ecosystem, it has not succeeded in creating a large volume of customer successes. An OpenStack deployment is notoriously hard to get right in practice and hundreds of millions of dollars have been squandered to date on failed projects. In this way it has a lot of similarity to CORBA, which was a vendor love fest far before customer adoption, which took years of failures before the initial successes.
Secondly, you may want to look into how the OMG operated (and still operates), which not as a top-down standards body, it is more of a community facilitation organization. OMG set out RFI/RFPs for set of component/facility/service-level specifications with competition in the form of RFP responses that are voted on by the membership and then consolidated. The OMG would have been much better served by having an open source reference implementation for the various specs (avoiding many of the design flaws and ratholes the OMG ran into).
That said, once the OMG picked a recommendation for an area, customers or vendors tended to use that or implement that, and ignore alternatives. This is not much different from the OpenStack Foundation, which can and will control the direction of the OpenStack software for the (debatable) benefits of its leadership and members. The startup community is risk tolerant enough to just "try something else" that's open source without a support contract, sure, but in my experience most enterprises won't mix and match things that aren't "approved" or "recommended" by the mothership and supported by the likes of EMC, VMware, HP or IBM.
Third, consider the popularity of RedHat Linux. It is a billion dollar company because they are the de facto go-to vendor for a Linux distribution, patch network, and support contract. They curate what packages and kernel patches are included in the distribution, but generally rely on Linus' governance for the kernel and do not deviate strongly from it. If Linus doesn't want it in the main kernel, it's an uphill climb to expect to see it eventually in RHEL.
I expect a similar pattern with OpenStack, with the caveat the governance itself is not user-centered, it is vendor-centered, which is not a good sign.
Some of what you said applies, but I think the analogy doesn't go that far. Just to point out / correct some things: Openstack is not very anti-VMware. There are even backend drivers that allow you to manage VMware environment using openstack as a frontend. Never tried it in practice, but some company bothered to write / contribute it, so it must be working.
Application deployment via docker - openstack supports both docker and libvirt/lxc. Unless you meant technologies running in the management layer - in that case I think coreos and similar projects aim for a completely different use case, not just architecture. Openstack works well with not trusted, heterogeneous environments and separation of tenants. If you're running your own cluster, with your own in-house produced stack and control over processes, coreos may be a better idea.
Or maybe you even want to deploy coreos cluster on real hardware automatically? That's also something openstack could do using baremetal backend.
Basically what I'm trying to say is: openstack is a framework more than a single solution. It may not work for every use case, or it may require some specific config to be useful. But it shouldn't limit the architecture beyond: it's got an api.
Technically speaking, I agree that it's a not a single solution and can be flexible. That said, it fundamentally presumes VMs, attachable disks, and networks as "the objects that matters" for your cloud. It's a good assumption for now, but maybe not for 5 years from now.
My post was mostly about the funding and political context that is driving OpenStack momentum, and its similarity to past attempts for vendors to gang up and try to take on the market leader (AWS). The community is all about the vendors, currently. Users are growing, but still the minority. And that's a warning sign, because it may mean the users really just don't care as much as the investors in OpenStack think they will.
I think this article misunderstands HP's definition of service provider. From what I understand HP considers service providers to be SAAS, IAAS, Hosting, and Managed Service Providers, not only telecommunications and ISP. I'm working for a VAR that has made this definition of service provider into a vertical and we have seen massive growth in this space. A lot of companies are making this distinction now, after seeing their existing hardware sales to SMB drying up as services shift to the cloud.
Here's a fun test I like to do with people who want me to get started with OpenStack, the system that HP is backing:
Starting from the home page (http://openstack.org), try to download something that will let you run a test stack on your local machine, and run it. Time how long this takes.
For me, it took over 50 clicks, searches, etc. and two hours of time before I figured out how to get the stack up and running. It was one of the most frustrating technology experiences I've had in recent memory. The website is a labyrinthine, confusing morass of marketing gobbledygook that I found difficult to get important details from, at least for developers as an audience.
I think OpenStack has a long way to go in terms of technical usability before I'd consider trying it out again. That was months ago (August 2013), so maybe things have improved, but I haven't worked up the courage to try again just yet.
P.S. OpenStack devs: I'd be happy to have a conversation about it! I would love to be told that my experience was anomalous and get a reason to give it another shot.
It took a few hours of following the instructions to the letter but everything worked first time and it's fairly par for the course in terms of installing a complex bit of software.
Agreed. It's difficult at times, but it's completely reasonable for the level of software.
I am going through a complex install at the moment, and I am sitting in IRC with openstack developers to help solve issues.
I've done that with many packages in my life, so I don't think that's odd, but to be fair.. I have no idea how I'd be progressing without asking individuals in IRC on specific points.
We're using OpenStack at Bloomberg and we went ahead and published our Chef recipes[1] for building the OpenStack deployment. You can either take a look at these or use these to get a full stack running on your local machine to play with.
Hit the 'Getting Started' link at the bottom of the front page, then follow the link under 'Option 2: Local Dev Environment: devstack.org'. I knew what I was looking for, but that might be easier to find than it used to be?
Four clicks, but not super obvious, and it takes you to a completely different website. But still, that's not too bad by the standards of enterprise software companies. At least the openstack and devstack websites don't have addresses like "www8.hp.com" (www8, wtf?): http://www8.hp.com/us/en/cloud/helion-overview.html
Which four clicks? For me, I went to openstack.org, and followed the "Getting Started" link under the "Documentation" heading on the front page. From the getting started page, there are two main (on the top) headings: "Public Cloud" and "Local Dev Environment". Clicking the button under the local option goes to the devstack.org quickstart page (which is also their main landing page). That seems really optimal to me, but I'm not a UI guy, so maybe somebody could come up with a better flow. I'd love to see it!
That's not the same as getting the whole thing running of course, but getting to a simple-looking set of instructions is pretty quick.
As someone who spent 18 months of his life deploying Openstack at scale, I'm confident in pointing out that devstack is, like most of Openstack, half-broken.
It is a project that basically lets you manage a set of libvirt based host machines. Nothing cloud scale or anything like that just something for a small business.
Here is the architecture. It is too bad this doesn't get more press as it seems simpler and easier to understand.
My take on ovirt is that it is focusing on (moderately) large-scale deployment, albeit only really covering the hypervisor aspect of things , whereas openstack seems to have a broader focus.
I recently evaluated ovirt in a small business context, and one of the stumbling blocks was, "how do I specify which VMs should be automatically run at boot time, and how do I control the order in which they boot?" The current answer to that with ovirt is in effect: "Well, we expect you to have a highly-availble bunch of management nodes which will simply remember what the set of currently running VMs ought to be. If that doesn't float your boat, maybe you should use the API and roll your own control system."
From a small business context this basically amounts to: No.
Note that this is not a serious criticism of ovirt. A project has finite resource, there's only so many things they can focus on at one time, I fully expect that things will improve over time, and the stuff that is there installs and runs very smoothly.
I think of it this way: oVirt is an open source equivalent of VMware vSphere, while OpenStack et al are clones of AWS. The former's less complex, but what most "small" businesses actually need.
I just had local proponents here push OpenStack because they thought it was "cool" to have "cloud-grade" virtualization. It ended up being a pain to deal with and is now being scrapped.
I have twice tried to install DevStack/OpenStack/RDO to get a proof of concept replacement for vCenter up and running and have ran into nothing but half broken implementations. There are so many moving parts under the hood that I am afraid OpenStack will never be anything more then "Enterprise Grade" software running behind dozens of engineers in large corporate data centers. I can get almost any other Virtualization technology up and running in a cluster faster then I can get a single DevStack node up and running.
This news has killed Rackspace stock, RAX is down tremendously for the last week (although up today). Quarterly earnings come out next quarter, and those have been bad every first quarter so far if memory serves me. So probably not going to trend up. I don't own any rackspace stock and I'm not making predictions anyone should bet on. I'm just complaining because Rackspace has really good products and it's getting pummeled. Down more than 60ish% since its height a year or so ago.
In related news, rax being cheaper make it a bargain for an acquisition, P/E is around 40 now.
So I recently saw a keynote by someone on OpenStack at LOPSA-East and as a System Administrator it kind of scares me. I'm finally starting to understand the meaning, reasoning behind & the fervor behind DevOps & why some organizations have such a need for better DevOps in their organization.
At the same time I also see Dev taking more care of Ops & cutting us (Sysadmins) out entirely. How do I take advantage of this or get ahead of this curve? I know there will always be a niche for small time System Administrators who administer small offices, primarily use Windows, etc. but after that ... we're done for.
Upskilling into some lightweight dev or source management (e.g. release management) is always an option - DevOps can come down on the Dev or Ops side, depending on your lineage, ultimately it's a more holistic view of the application lifecycle.
Getting your head around how AWS, GAE and Docker deployments work and combining that with traditional sysadmin knowledge should do you in good stead.
You should focus more on security. I'm simply amazed, still, at how many devs need sa level access to the DB. I'm a dev that transitioned into admin with a background in Linux/Unix so the whole devops movement doesn't bother me.
I'm not at the openstack summit, but I'm friends with many people there. I'm actually right now trying to find what streaming they have so I can view some of the content.
Edit: If anyone is interested the keynote just got uploaded to youtube.
A private cloud seems like a good choice for larger organizations. They still have some admin and/or devops overhead that they would not have using AWS, Compute Cloud, Azure, etc. But, they might save a lot of money by providing developers with Heroku-like platform services but running on their own more cost effective hardware.
For startups with a few people, private clouds might not be a good ideas. Any supporting cases that show the opposite?
I would be interested to see co-ops where individuals and small companies join together to build and operate private clouds for the benefit of their co-op members.
Attention companies going public in the near future: this is how you raise your stock price
And why do you do this? Because you're paid in stock and actual internal projects are not going to "wow" analysts. When in doubt spend money on what's popular.
So HP will spend $1bn over two years on cloud while Google is spending twice that a quarter and Amazon and Microsoft spend that a quarter on capex mainly on cloud. It is not clear they are a serious player. Rack space has spent more than that already.
Keep in mind they (rax, Amazon) are already running multiple deployments across continents. It's quite easy to spend a lot of money just to keep that service going.
51 comments
[ 1.7 ms ] story [ 72.0 ms ] threadChecklist:
- Anybody but "x" club, where X is the market leader. Openstack has vascillated between VMware and AWS here, CORBA had Microsoft.
- Large investments advertised as a penis swinging contest between HP, IBM, and others
- Chasing aging technology architecture as a panacea instead of building the next generation.
CORBA was chasing distributed objects, but the Web and REST was what really was going to matter. Today the big guys are chasing the IaaS model when we have a new generation of platform and application clouds (Mesos, YARN, CoreOS, Docker, CF, Asgard) growing out there.
The vendors will of course try to recoup their investment through reference architectures and "best practices" that tell customers they should have a multi-layer SaaS or PaaS running on an IaaS, even though lightweight alternatives like CoreOS will chug along doing nothing of the sort.
This is similar to how CORBA's IIOP was promoted to be the way to do "real work" with C++ or Enterprise JavaBeans in your middle tier, but this "Web" HTTP thing can be relegated to the front tier. I know fixed income systems around 1999 that literally wouldn't use HTTP for their blotter updates, with the lead devs insisting such events needed to be pushed over IIOP even though the front end was an IE control in a VBX. (this was shortly before TIBCO had effectively seized the market data industry)
- persistent flame wars about governance and architecture of said standard on the social media of the day
- Vendor adoption drastically outpacing customer adoption , developing a "buying customers" effect
This is not to say OpenStack is bad. There is value, potentially a lot of value, as there was with CORBA. That's all case-by-case. Just saying that vendor love fests rarely indicate a true revolution and most often are just an orchestrated marketing game by the current losers.
Don't forget the SOAP era; it also resembles CORBA. There was a whole region of time (~2002-2005) where Microsoft and IBM jumped on SOAP. They promoted the hell out of their stacks, and then, after everybody realized SOAP was a turd, put the projects in deprecated mode.
http://www.ianfoster.org/wordpress/wp-content/uploads/2014/0... (page 23).
The next step in evolution was JSON payloads over HTTP, with methods defined using REST. This model has more or less survived for 10 years now, and is a community-created (rather than top-down-imposed by a corporation) system that has lots of experience under its belt. I expect only marginal changes at this point: move towards a more compact encoding (like BSON) and a bit more standardization on where the RPC method is defined. I expect REST itself, being so hard to understand, will only be used by people who really insist on getting their API surfaces really RESTy.
SOAP itself was Microsoft's attempt to wrest momentum away from Java, and get people to pay attention to .NET, which modestly succeeded.
It also was a the final turd that broke bullshit mountain's forces: ultimately, developers had to figure out what mattered more to the Web's success: was it tagged data (HTML/XML), and we could preserve this distributed component/object conceptual model? Or was it the combination of URI+HTTP with its uniform interface and hypermedia? The REST debate took a long time (through 2007) to answer that, even though in practice with JSON we just pretend to adopt the full architecture.
I think the next revolution in REST would be if a new media type and associated toolkits/libraries come out that helps people build simpler and evolvable systems than JSON does. There are a variety of experiments out there, but as you imply, it's hard work.
I find it odd that a self-declared cloud architect hasn't used what is easily the largest growing IAAS project, free or commercial.
I don't think we have accurate visibiliy into EC2 to compare its rate of growth compared to OpenStack.
OpenStack isn't IaaS. It's a PaaS middleware that abstracts the underlying IaaS.
Edit: a bit more clarity
CORBA is mainly a standard. A standard only succeeds if most people adopt it.
OpenStack is mainly a set of open source software. Users can pick and choose the pieces to use in their system. If "OpenStack Governance" decides that Software X should be used for job Y, but something better (and open source) comes along, people can just switch to the better software. In other words, competition can happen at the component level and it can come from the grassroots (without permission from the big wigs). It's much different from everyone having to agree on one standard.
I don't need to tell you that there are many thriving open source software ecosystems. Make an analogy with one of them and OpenStack makes much more sense (e.g. Linux and its packages, Python and its packages, WordPress and its themes/plugins).
Firstly, OpenStack is not a thriving open source community in the usual sense. It has succeeded in creating a vendor ecosystem, it has not succeeded in creating a large volume of customer successes. An OpenStack deployment is notoriously hard to get right in practice and hundreds of millions of dollars have been squandered to date on failed projects. In this way it has a lot of similarity to CORBA, which was a vendor love fest far before customer adoption, which took years of failures before the initial successes.
Secondly, you may want to look into how the OMG operated (and still operates), which not as a top-down standards body, it is more of a community facilitation organization. OMG set out RFI/RFPs for set of component/facility/service-level specifications with competition in the form of RFP responses that are voted on by the membership and then consolidated. The OMG would have been much better served by having an open source reference implementation for the various specs (avoiding many of the design flaws and ratholes the OMG ran into).
That said, once the OMG picked a recommendation for an area, customers or vendors tended to use that or implement that, and ignore alternatives. This is not much different from the OpenStack Foundation, which can and will control the direction of the OpenStack software for the (debatable) benefits of its leadership and members. The startup community is risk tolerant enough to just "try something else" that's open source without a support contract, sure, but in my experience most enterprises won't mix and match things that aren't "approved" or "recommended" by the mothership and supported by the likes of EMC, VMware, HP or IBM.
Third, consider the popularity of RedHat Linux. It is a billion dollar company because they are the de facto go-to vendor for a Linux distribution, patch network, and support contract. They curate what packages and kernel patches are included in the distribution, but generally rely on Linus' governance for the kernel and do not deviate strongly from it. If Linus doesn't want it in the main kernel, it's an uphill climb to expect to see it eventually in RHEL.
I expect a similar pattern with OpenStack, with the caveat the governance itself is not user-centered, it is vendor-centered, which is not a good sign.
Application deployment via docker - openstack supports both docker and libvirt/lxc. Unless you meant technologies running in the management layer - in that case I think coreos and similar projects aim for a completely different use case, not just architecture. Openstack works well with not trusted, heterogeneous environments and separation of tenants. If you're running your own cluster, with your own in-house produced stack and control over processes, coreos may be a better idea.
Or maybe you even want to deploy coreos cluster on real hardware automatically? That's also something openstack could do using baremetal backend.
Basically what I'm trying to say is: openstack is a framework more than a single solution. It may not work for every use case, or it may require some specific config to be useful. But it shouldn't limit the architecture beyond: it's got an api.
My post was mostly about the funding and political context that is driving OpenStack momentum, and its similarity to past attempts for vendors to gang up and try to take on the market leader (AWS). The community is all about the vendors, currently. Users are growing, but still the minority. And that's a warning sign, because it may mean the users really just don't care as much as the investors in OpenStack think they will.
Starting from the home page (http://openstack.org), try to download something that will let you run a test stack on your local machine, and run it. Time how long this takes.
For me, it took over 50 clicks, searches, etc. and two hours of time before I figured out how to get the stack up and running. It was one of the most frustrating technology experiences I've had in recent memory. The website is a labyrinthine, confusing morass of marketing gobbledygook that I found difficult to get important details from, at least for developers as an audience.
I think OpenStack has a long way to go in terms of technical usability before I'd consider trying it out again. That was months ago (August 2013), so maybe things have improved, but I haven't worked up the courage to try again just yet.
P.S. OpenStack devs: I'd be happy to have a conversation about it! I would love to be told that my experience was anomalous and get a reason to give it another shot.
Found that in about 30 secs from the OpenStack site.
It took a few hours of following the instructions to the letter but everything worked first time and it's fairly par for the course in terms of installing a complex bit of software.
I am going through a complex install at the moment, and I am sitting in IRC with openstack developers to help solve issues.
I've done that with many packages in my life, so I don't think that's odd, but to be fair.. I have no idea how I'd be progressing without asking individuals in IRC on specific points.
[1]: https://github.com/bloomberg/chef-bcpc#overview
4 Clicks takes you to http://devstack.org/ and installs with 2 commands... git clone https://github.com/openstack-dev/devstack.git then cd devstack && ./stack.sh
That's not the same as getting the whole thing running of course, but getting to a simple-looking set of instructions is pretty quick.
I was following this project instead:
http://www.ovirt.org
It is a project that basically lets you manage a set of libvirt based host machines. Nothing cloud scale or anything like that just something for a small business.
Here is the architecture. It is too bad this doesn't get more press as it seems simpler and easier to understand.
http://www.ovirt.org/Architecture
I recently evaluated ovirt in a small business context, and one of the stumbling blocks was, "how do I specify which VMs should be automatically run at boot time, and how do I control the order in which they boot?" The current answer to that with ovirt is in effect: "Well, we expect you to have a highly-availble bunch of management nodes which will simply remember what the set of currently running VMs ought to be. If that doesn't float your boat, maybe you should use the API and roll your own control system."
From a small business context this basically amounts to: No.
Note that this is not a serious criticism of ovirt. A project has finite resource, there's only so many things they can focus on at one time, I fully expect that things will improve over time, and the stuff that is there installs and runs very smoothly.
http://openstack.redhat.com/Main_Page
In related news, rax being cheaper make it a bargain for an acquisition, P/E is around 40 now.
At the same time I also see Dev taking more care of Ops & cutting us (Sysadmins) out entirely. How do I take advantage of this or get ahead of this curve? I know there will always be a niche for small time System Administrators who administer small offices, primarily use Windows, etc. but after that ... we're done for.
Getting your head around how AWS, GAE and Docker deployments work and combining that with traditional sysadmin knowledge should do you in good stead.
Edit: If anyone is interested the keynote just got uploaded to youtube.
https://www.youtube.com/watch?v=YoP0erdr7No
Don't trust anything you hear from on stage, especially the enterprise "success stories." These are largely coming from an alternate universe.
The engineering planning sessions are excellent.
For startups with a few people, private clouds might not be a good ideas. Any supporting cases that show the opposite?
I would be interested to see co-ops where individuals and small companies join together to build and operate private clouds for the benefit of their co-op members.
And why do you do this? Because you're paid in stock and actual internal projects are not going to "wow" analysts. When in doubt spend money on what's popular.