Thanks for sharing; I wrote the post and also submitted the distro to HN. In most cases (apparently per HN policy), blog posts that announce new services are removed in favor of direct links to the offering.
Is it really honest to say "Keeping Open Source Open"? Elasticsearch is 100% open source, even Apache Licensed, and you are implying that this is not the case "somehow". Sure "security" is missing, but who says that this has to be part of the open source project "Elasticsearch"?
Maybe it would be more honest to say "Forcing open source backing companies to avoid open core that is hated by AWS" ;) ?
(btw: I'm not associated with Elastic except loving their open source projects)
'Distro' makes it sound like a linux-distribution with, in this case, Elasticsearch installed on it. From what I read on the page and on the aws blog this is not the case though?
I thought the same thing. After some searching I realized that "Open Distro" isn't actually a linux distro after all. This is a non-fork release of Elasticsearch with the proprietary parts stripped out.
In what why did they deliberately make the meaning unclear? The middle of the page says "An Apache 2.0-licensed distribution of Elasticsearch". It's saying in black and white that it's a distribution of Elasticsearch, not of an operating system.
Hi, I work at AWS on compute services, but spent a lot of my life building Linux distros.
For those of us with Linux distro building experience, this may be initially confusing. But calling a curated collection of software a "distro" has been used for decades outside of Linux distributions.
"Distro" is short for distribution. Many things, not just operating systems, can have different distributions. Calling it a distro doesn't make it sound like a linux distribution at all. If it was a linux distribution it would probably talk about being an OS and not being a distribution of Elasticsearch.
Thank you for your input. I tried to google that, but I don't see any heavy usage of "distro" along with those solutions.
For example site:https://hadoop.apache.org distro nothing significant. Also, googled "Kubernetes distro" same thing.
I guess you could argue that "distro" has a distinct meaning to "distribution" with the former specifically pertaining to linux distributions, but I think most people probably just interpret it as being shorthand for the latter
I can assure you (within the k8s dev community anyway) that "distro" is used to refer to companies productised versions.
Look at any of the long term support discussions, and distros will be mentioned.
Same goes for OpenStack - we refer to the different vendor products as distros (this is complicated by most of the OpenStack in a box products being produced by Linux distros, but we mean all product versions of OpenStack when we was OpenStack distro)
just googling "hadoop distro", without limiting it to the hadoop website, finds lots of references to people and media outlets using distro as a shorthand.
At least in Kubernetes land, that lead to the foundation (CNCF) creating a certified Kubernetes conformance program, to avoid end user confusion and establish a baseline feature set.
Another example from years past, would have been openstack, with many vendors shipping/selling their own branded variants.
This is an interesting new "threat" to the "open core" business model. ES makes proprietary extensions to support their FOSS core product - then a tech behemoth clones these features for their own hosted service, but makes them Free as well. Good for the consumer, but bad for the company that originally created the core FOSS technology and best for Amazon.
Have there been any examples of this happening before?
And it doesn't appear that AWS is hosting these right now, just distributing so they must be paying some kind of licensing for ES. But I'm sure that will change.
Hi, I work at AWS on Compute services, but not directly on database services. I also work on broader open source topics at Amazon.
Amazon DocumentDB was in development for a long time, and development started well before any of the license changes that MongoDB (the company) made to MongoDB (the database).
Re OpenES, its an apache licensed distro without proprietary commercial bits.
One of the concerns I have had with the recent trending of some opensource companies (elastic, timescaledb, and others), is that those who mix in proprietary code into their opensource repos, effectively taint contributors who can't even look at git history on a project without viewing stuff thats not opensource.
I mean, this is basically what happened to commercial UNIXes, is it not? Tech behemoths decided it was more in their interests to fund the development of a free-software UNIX clone (the GNU/Linux ecosystem) than to keep paying Sun et al. The UNIX wars were, I'm sure, very meaningful for the participants, but ten years later it turned out there was more money to be made in building things on top of the OS than building the OS.
Hell, even before Linux was popular, gcc took off in the commercial Unix space because a) it didn't cost anything, while the official Sun compiler was ludicrously expensive, and b) it was more performant than the official Sun compiler.
> Good for the consumer, but bad for the company that originally created the core FOSS technology and best for Amazon.
I don't know if this is a bad pattern in general for the company that originally created the core FOSS technology. If your business model is support/consultancy this may work to your advantage.
How is elastic (the company) going to make a business going forward? I would expect that selling these enterprise add-ons (or similar ones) is a significant part of their revenue.
> all commits in the github repositories are made by AWS staff.
Given that it was just made public this morning, it would be surprising if it were otherwise. The real question is what the contribution model looks like going forward. The blog post says "Contributions are welcome, as are bug reports and feature requests"[1], but of course the devil is in the details.
Dear Ladies and Gentlemen,
it would be of extraordinary interest to I believe many people in our industries if anybody with some production experience on the topic would like to write a short comparison "Elasticsearch vs. Solr", thank you very much for your attention!
In most cases it's just a matter of preference. Elastic has some extra tools like kibana an was more focused on having statistics and aggregations working well, while Solr webui is a lot more complete than the elastic one and the full text search and relevance bits are more explained. Internally both use lucene, so you are going to be able to do the same things with both.
I worked extensively with both and I prefer the way of administering Solr. Anyway I use Elastic when it involves logging and statistics thanks to kibana.
Here is one good comparison from Sematext (Sematext provides both Solr and Elasticsearch consulting, support, and training, so we are familiar with both Solr and ES and have no horse in the race):
IMHO using "open" is just strategy to attract devs and companies that are looking for FOSS solutions, but in reality it's just another way to vendor lock-in. I hope i am wrong.
Honest question, besides not paying licenses, is there any technical reason to do this?
As long as this remains cleanly open, I'm not sure why you wouldn't run this if you want to self-host. I'm not sure how this will, or won't play out for Elastic company though. I'm thinking AGPL for server, and MIT for client libraries is probably the best way to go for someone starting any kind of *Server project.
Open-Core as a business model isn't something that the likes of AWS, Azure or GCP can really tolerate if they want to integrate SaaS that the customers want. DBaaS in particular is one of the bigger advantages of cloud providers in general.
Discloser: A family member of mine does work for Elastic, but I have no knowledge of their internal practices or policies or reaction to this.
I was trying to implement ES previously but kept running into things I thought we're included "open source" but turned out to be an extra paid feature. With this I know what I'm getting.
There is no more vendor lock-in with this than with Elastic.
Does knowing that you're running or contributing to Open Source code count? The AWS Open Source blog posted elsewhere in this topic implies that Elastic is making it hard to tell.
This is a very surprising move from AWS. In the past they haven't seemed to be willing to contribute to the OSS that they pick up and host. Even though they claim they'll contribute changes upstream, I doubt that Elastic would accept changes that are competitive with their commercial offerings. So you effectively get a fork.
I think this is a win for the Elastic community as a whole, but presents a real problem for Elastic the company. And that begs the question of what happens to the Elastic community if there's a real fork. And what happens if Elastic the company is very negatively impacted? Do we see a fracture of the community?
Looking even farther out, at this point does it make sense for any startup looking to create infrastructure software to open source it? If their project becomes successful, they'll get eaten up by the hosting providers. It makes smaller scale open source more commercially viable because you won't attract the attention of the providers that would come in a take your business.
Will be interesting to at least watch what early stage VC investment in open source companies looks like over the next 12-24 months. Will the open source pitch work for series A investments or will investors shy away?
I wonder why you say that? It's kind of obvious that somebody was going to do it sooner or later to protect themselves, no? Perhaps fearing the license change?
Oh I assumed that someone would do it. I just didn't expect it to be AWS. Elastic is popular enough and the core open source license permissive enough that I expected some sort of fork eventually. The license of the original project is an important factor in this. For example, I wouldn't expect a fork of MongoDB because of the limitations of the AGPL license make it less appealing. With that case I'd expect what AWS already did, which was to create an API compatible system. Also notable is that they're not open sourcing that.
With Elastic, this looks like it's half marketing on AWS' part. They want to start repairing their image in the OSS world so they say they're doing this for the community.
Adding another thought to this, which is that I doubt the code behind this is a direct response to Elastic's recent license changes. This is a ton of work and I'm guessing AWS has been working on the code side of this since well before last summer. By all accounts their hosted Elastic offering is wildly successful so they've probably been hard at work to add features that Elastic formerly had as closed and commercial (before the license change).
I think their open sourcing of this work is a direct response to the recent license changes. If it weren't for those, they might have just kept this work closed and in use only for the hosted product (like they do with many other projects).
They took the closed XPack code and put it into the open source Elastic repo, but under a commercial license. It muddied the waters for anyone looking to use or contribute to the open source.
Prior to this move, the default install of the Elastic Stack was 100% open source. X-Pack had to be intentionally installed as a plugin.
Now the default install includes X-Pack, and you have to go out of your way (assuming you even realize that there is difference) to install the 100% Apache 2.0 licensed version.
> Looking even farther out, at this point does it make sense for any startup looking to create infrastructure software to open source it?
Of course it still does, just not under permissive licenses. Hopefully the industry will converge on Kyle Mitchell's license for this, instead of everyone coming up with their own licenses:
https://github.com/kemitchell/api-copyleft-license
I view that as a significant negative impact to OSS. I have a strong preference for liberal licenses like MIT or Apache 2.0. Infectious licenses like copy-left put restrictions on it that make it less appealing as open source. I wrote a bit on it here: https://www.influxdata.com/blog/copyleft-and-community-licen...
While I understand your argument, I dispute that copy-left makes things inherently less appealing as open source, or that you can make any objective claim that liberal licenses ultimately result in broader utility.
1. I personally find copy-left inherently more appealing (as something to potentially contribute to) purely because of my ideological leanings.
2. While the first order calculation of use for liberally licensed software seems clearly in favor of them, it's impossible to ever know what benefits the copy-left restrictions would bring one or more orders removed from the original distribution.
> This is a very surprising move from AWS. In the past they haven't seemed to be willing to contribute to the OSS that they pick up and host
I’m actually suprised they’re moving so slowly with OSS. I assumed they were going to go a lot more aggressively in this direction when they hired Adrian Cockcroft over 2 years ago now.
> It makes smaller scale open source more commercially viable because you won't attract the attention of the providers that would come in a take your business.
Small scale open source isn't technologically viable.
> In the past they haven't seemed to be willing to contribute to the OSS
How is small scale open source not technologically viable ? There are tons of small open source companies that build great niche open source products which are technologically competitive. I run such a tech/business (XWiki) for 15 years now. They just are not known and don't interest VCs unless you are ready to start going open core. However it is possible to be sustainable, just don't expect to be startup-rich.
"
Unfortunately, since June 2018, we have witnessed significant intermingling of proprietary code into the code base. While an Apache 2.0 licensed download is still available, there is an extreme lack of clarity as to what customers who care about open source are getting and what they can depend on. For example, neither release notes nor documentation make it clear what is open source and what is proprietary. Enterprise developers may inadvertently apply a fix or enhancement to the proprietary source code. This is hard to track and govern, could lead to breach of license, and could lead to immediate termination of rights (for both proprietary free and paid). Individual code commits also increasingly contain both open source and proprietary code, making it very difficult for developers who want to only work on open source to contribute and participate. In addition, the innovation focus has shifted from furthering the open source distribution to making the proprietary distribution popular. This means that the majority of new Elasticsearch users are now, in fact, running proprietary software. We have discussed our concerns with Elastic, the maintainers of Elasticsearch, including offering to dedicate significant resources to help support a community-driven, non-intermingled version of Elasticsearch. They have made it clear that they intend to continue on their current path.
"
The commercial entities associated with the most popular open source technologies are taking measures to protect themselves from Amazon simply taking their software and turning it into a service, whether it's introducing proprietary components or new licensing. Wouldn't be surprised if we saw AWS do something similar for Redis, MongoDB, Kafka, etc.
> Wouldn't be surprised if we saw AWS do something similar for Redis, MongoDB, Kafka, etc.
They started on their own Redis fork a while ago as ElastiCache has had baked in SSL for over a year and half now. While it's possible they've added it via an stunnel type software stop it, I'd bet for efficiency and simplicity it's baked into their internal fork: https://aws.amazon.com/blogs/security/amazon-elasticache-now...
They already did their MongoDB-compatible NoSQL document database. Not sure what's under the hood but I assume an altered MongoDB engine adapted to their scalable cloud storage architecture.
When you publish open source, this is what you sign up for. As a publisher of open source software, I think "open core" does a huge disservice to the community as it feels deceptive. While there is no denying that creators needs to be paid, communities thrive on freedom and openness, and dual licensing takes that away.
Companies that publish open source must realize that they cannot make money from the "software". Open source gives companies a brand, that they can then leverage to do other services like support, consulting, hosting, merchandise, events, etc.
There was a time when service companies wanted to move into products because of high margins and artificial scarcity, but enough open source will ensure that software licensing is not a sustainable model to seek rent, and product companies must also move to services to be able to grow and sustain.
Well said. The issue is most of the core devs don't want to touch "support, consulting, hosting". It's a mental block. They think they can skip around doing only the "cool" stuff.
Looking at it rationally, is furthering open source admirable when it's done by a community activist, and loathsome when it's done by someone who has a business interest in the exact same outcome? Why isn't the end result the most important part of the equation? If a stronger, open sourced Elasticsearch comes out of this, why is that a bad thing because it was backed by Amazon?
Just be honest, and don't pretend to be open source, when you are attempting to harness goodwill towards open source in order to make a buck. As pointed out, ES builds on the work of hundreds of other contributors to other open source projects. Is Elastic compensating those developers?
In my mind, a company that makes lots of free contributes to open source is an open-source-friendly company, but this thread and the other on the Elastic post made me realize just how many people think that a True Open Source Company (whatever that means) should prefer to go out of business to dealing in proprietary software.
I personally don't care for any of these puritanical viewpoints that deny economic realities (there's only so much a person or a collective can afford to give away for free, money doesn't grow on trees, etc). Open source is good, but thriving, sustainable open source depends on a symbiotic relationship with for-profit institutions.
Reminds me exactly of an article I read this once. It's about the concept of companies commoditizing their product's complement.[1] AWS sells hosting/infra, for them lowering the entry barrier is beneficial. (I'm not taking sides just a thought)
I find it very ironic that AWS is behind this. Whenever I buy books from Amazon I am amazed at their broken search engines. Just search a topic on Amazon and then try to sort the results. LOL.
IIRC, most of that infrastructure is/was Oracle based and much of it going to another DB. I'm not sure if Elastic is even in the mix, or how well it scales to Amazon's levels.
Given the age of the offering, it might actually be the CloudSearch [0] product that is used by the Amazon.com search team.
FWIW, I've always found the Amazon store search to be pretty decent; normally I don't use a website's built in search system, opting for Google instead.
"The maintainers of open source projects have the responsibility of keeping the source destination open to everyone and not changing the rules midstream."
Since when is anybody entitled to any free software. Projects can do what they want with their code, and while free software (the Stallman kind) is nice, if projects cite you as the main reason for their change of direction, you are intellectually dishonest if you just blame them for not being willing to do volunteer work (for you) anymore.
This, to me, looks like an abusive "look what you made me do".
No one has to "not be a dick" but the world is better when you try for the goal.
What the blog is saying is "If you build an open source tool and people put their time and effort to using it, closing more and more of the toolset to paid add-ons isn't cool".
AWS just said "Hey, we can build those ourselves and give back to the community". They volunteered and made it happen.
Question, can I checkout the code from Github and build it with ./gradlew assemble as Apache License 2 or do I need to delete some modules before doing that?
I dunno if I'd be long $ESTC at this point. Elastic took $162 million in venture funding before IPOing. In the last year their Operating Income was -$43 million. A lot of that is sales/marketing related, but at some point they need to make a profit. This open distro is certainly not going to help.
As someone who's worked at many big corps on Elasticsearch installs, I've been worried about them pushing away from OSS for some time. Many big corps want some of these "Enterprise" grade features Open Distro is offering, and if Elastic doesn't already have a foothold in them, I wager they'll be using this instead of paying.
As an aside, how come people still think these Open Source/Core commercial businesses are viable? We've seen time and again that once commercialization occurs, the OSS stuff goes right out the window. (RedHat, Docker, MongoDB, Sentry etc)
Just commenting on the aside, we should be careful about painting with an overly broad brush. Each of these companies and organizations is different, at least two you've mentioned are fully opensource afaik not open core (Redhat and Sentry). Redhat continues to do tons of opensource work across many ecosystems, kubernetes, linux kernel, gnome, etc. The Redhat sw model isn't open core, because what they sell isn't proprietary afaict its a support with updates model, that they also distribute under a different brand then the opensource (fedora/centos/openshift okd). Additionally we're talking about a multi-billion dollar company, seems like a success to me. For Sentry, their more focused on Saas platform, but afaics an on premise distribution model doesn't reference a commercial product thats differential to their opensource repos aka its not open core, afaics. [update] sentry on premise install docs https://docs.sentry.io/server/installation/
RHEL, as far as I know, is not available as a nice installable package without a subscription, at least to run in production. CENTOS was started as a separate entity to make bundling the RHEL source in to easily installable distributions more straightforward, as Red Hat made it difficult.
As far as Sentry, I'm referring to the fact that they have not cut a release of their on-premise in some time, and features have started to diverge, though they assert they will reach parity with on-prem "at some point"
https://forum.sentry.io/t/when-will-a-new-version-be-tagged-... (Though again, like RedHat, the source is technically available...)
My main point is that OSS is at odds with making money.
Easily installed isn't part of the opensource definition, any more than source available means opensource. Calling companies that are pure opensource, open-core is misleading and detracts from the conversation, imo. Centos was a binary distribution because Redhat made redistributions of binary RHEL harder to distribute(images/logos/binaries/names via copyright/trademark), but that was part of their business model. The source however was available under opensource licenses, and most of that was from separate upstream communities, aka its linux distro (though they have lots of other products). Re sentry, if I can pull the repo and get the same bits under OSI approved license, its opensource to me. Again opensource isn't about release management practices and nice installers (although for sentry its just a docker run/build away). They might be part of good community of practices for sure though.
I'd say opensource business models are different and potentially more difficult, but there are plenty of companies selling/supporting/building opensource. The delta seems to be most vc backed companies sponsoring opensource have different expectations on growth/value extraction from customers. aka open-core is not really about opensource, its about selling proprietary products, so I'd prefer not to mis-label those who aren't doing that.
We can argue all day about following the spirit of something vs the letter of the law, but all four of those companies have direct business incentives to make it difficult for those wanting to use their software for free to do so. No surprise then that all have taken steps to dissuade free users or make things more challenging. You're arguing about labels, which is beside the point.
This is a good project to happen, and kudos to AWS for doing this! However, like some in the thread, I suspect AWS has been building this for a while for their own product and Elastic's license change may have influenced the decision to open-source.
Getting a good out-of-the-box granular security has been a long overdue pain point with ElasticSearch, there have been good alternatives for the other problems it is replicating.
Would very much like to hear from OP why these weren't contributed to instead of creating another alternative.
We've been working on a similar project in Golang and now well may be as good a day to open-source and put it out there: https://github.com/appbaseio/arc. It's an extremely light-weight API gateway for ElasticSearch that at the core comes with an out-of-the-box security system based on users and permissions (set granular ACLs, Rate Limits, TTL) inspired from a proprietary security system we built over the past year - https://appbase.io/features/security/.
According to the release blogpost [0], "SQL Support [...]is an improved version of the elasticsearch-sql plugin." They're not replacing the SQL extension you linked -- they're integrating it.
As an active developer and maintainer of a cluster, I strongly disapprove of this move by AWS.
By forking on an earlier instance of the codebase for ES and Kibana they're not only creating an open source version of the code but also attempting a fork of the community - plugin developers, user groups, etc. It's extremely frustrating.
EDIT: If folks actively building plugins have felt pain here because of decisions by Elastic, then my view on this point changes considerably, obviously.
I find it very hard to believe there was no compromise to be had here between the current Elastic roadmap for their product and their previous work.
How is this good for users of the current tools? How does this help create better capabilities on a common platform? Will this encourage users to build businesses on elasticsearch open source elements?
I'm not sure where the community ends up after this, so I'm not sure I can support by using AWS' elasticsearch tools.
As someone who is involved in the ecosystem and maintains some open-source projects based on ElasticSearch, I am similarly curious about the move to not build upon existing projects (there could be a good reason, but haven't heard it yet).
there's no "they", there's only us. AWS paying to use it means we are paying to use it. Between that and a truly Apache 2.0 project, I think it's a no-brainer for everyone.
More generally: why should AWS get to leverage its near-monopolistic position in the SaaS market to perform essentially hostile takeovers of the software projects it packages and runs? Simply because they have the resources to do it?
The code was open, and the company decided to close it for the new versions, so it forks. Forks are common when people don't like the corporate direction of an OSS project - It's a strength, not a weakness. Openoffice/Libreoffice, Hudson/Jenkins etc.
Imagine if Linus said that the next version of Linux would be paid-only.
Within 15 minutes there'd be a new librelinux repo that people could contribute to instead.
Someone was going to make a new fork anyway - This is Amazon putting their money where their mouth is to fund and support that new fork.
To be clear, no Elasticsearch code or features that were OSS were closed. As new Elasticsearch features are developed, some of them are now released as OSS and others are released with a commercial license.
If ES didn't do the weird license change thing, we wouldn't be here. Either Amazon wouldn't have used it, or they'd be paying. Including non-com features in the default install was not a good way to play things.
Would you feel the same if AWS built a drop in/api compatible system from scratch?
It's not just about AWS, other people need to use this as well. Why are security features a paid for addition? Why not make core features a paid for addition instead? I've always hated Elastic (the company) for this decision, please stop trying to give away free things that are not secure, and then charge for security.
From the pom.xml of Open Distro for ElasticSearch: the fork is from elasticsearch-6.5.4, which is the latest stable release. Only alternative would have been v6.6.1. EDIT: also, nothing is there to believe they would not bump to higher versions in later versions.
From the related AWS blog (so at least a few grains of salt):
>>This means that the majority of new Elasticsearch users are now, in fact, running proprietary software. We have discussed our concerns with Elastic, the maintainers of Elasticsearch, including offering to dedicate significant resources to help support a community-driven, non-intermingled version of Elasticsearch. They have made it clear that they intend to continue on their current path.
... maybe I'm misreading your comment, but it sounds like AWS tried to find a compromise, including contributing to upstream, and were shut down (and all their content about future contributing seems to say "we'll send it upstream, but it depends on whether Elastic accepts it," so seems to corroborate). It honestly seems like you get more certainty that everything you're using from AWS is open and will remain open, and Elastic is going to keep creating uncertainty / drive to proprietary source-available.
There is certainty around the open-source bits and the license - this is true. That said elastic the company has also seems to have brought the product a long way after becoming a company. Having a dedicated staff of engineers and developers behind a thing is helpful for that, and they should be able to make some money as part of this if we want to see future open source products advance similarly. (There is a difference too between consulting / services and money “at scale” - AWS is removing an at-scale component here)
That said, I also totally understand disagreements with Elastic’s decisions on charging for security, etc and why that causes concern. As a user I wish I wasn’t staring a fork in the face and I’m hopeful things get back to equilibrium here soon as a single community.
I like seeing a, hopefully, solid OSS implementation of security for ES. I understand the companies need to make money and pay developers, but I've never been very comfortable with them keeping the security features out of the OSS version.
WDYT:
Would it have been better for AWS to initiate Open Distro for ES as part of e.g. Apache Software Foundation, whose mantra is "community over code"?
(or if not ASF for some reason, then Eclipse or CNCF for example)
Yes, and I think they should have done so, just look at the responses in this thread. It's just been launched and everybody is afraid of Amazon vendor lock-in. For many techies, just the Amazon name is scary even though this is Apache 2.0 licensed.
Neither. Right now AWS seems to be acting in the best interest of the ElasticSearch community, but this doesn't extend to open source as a whole.
For the long-term, the open source community should look to projects where the center of gravity is outside of an open core or FANG company. ElasticSearch went open core a while ago, and Elastic's form of open core is more like MongoDB or Redis than it is like nginx or Docker. As for Amazon, I think I'd rank them below facebook, and I would rank facebook below google and Microsoft, as stewards of open source projects.
My guess is that Elastic is acting the best interest of Elastic and AWS in the best interest of AWS :)
I used to like a lot Elastic, we have a rather big cluster work with the OSS version, and I really disliked how they started mixing the proprietary part with the oss one. I hope we will get a good answer from them. Like splitting again the proprietary plugins and/or open sourcing some of the security features :)
In the long term their actions make it impossible to build any sort of business based directly on FOSS. You have to do support, or hosting, or something else.
Guess what, AWS has a lot more money to do both of those things.
and Amazon will keep right on going sucking up code and giving nothing back.
We need aggressive licensing changes.
Hell, API copyright might be necessary to stop this nonsense. We can still have good licensing in that scenario.
I recently decided to experiment with ES and prefer to evaluate things with Apache 2.0 licensing if available, even if I may pay for to license it at a later time. Elasticsearch Still maintains -oss suffixed docker images and builds that are true Apache 2.0. You can read more about this here: https://www.elastic.co/products/x-pack/open
I'm sharing this because I got the vibe from the linked post in the comments and the overall marketing of "Open Distro for ES" that Elasticsearch was 'not open anymore' and had just discovered the Apache 2.0 builds within the last week.
I see this as a net positive (https://xkcd.com/927/ notwithstanding): Elastic.co, imo, has iron-grip (in terms of licensing and features that it builds) on XPack and Elasticsearch. This incredible technology that rode on OSS wave is increasingly controlled by a for-profit organization that also acts like one. This is very similar to how Google dealt with numerous forks of Android by essentially moving key pieces to GooglePlayServices and other closed-source apps in addition to responding with shutting out manufacturers and SoC providers that weren't OHA signatories [0][1]. Amazon felt the burnt of that [2].
Apart from including/forking other community plugin projects (elasticsearch-sql, SearchGuardSSL) in open-distro [3], AWS seems to be investing to make this more "cloud-native", as well: This observation comes from PerformanceAnalyzer (released) and IndexStateManagement (request-for-comment).
- IndexStateManagement [4] is simply "never send a human to do a machine's job" automation of common yet frequent tasks required to keep elasticsearch humming along. If used in conjunction with AWS's/GCP's/Azure's managed-cloud offering, this feature might be an amazing sell for a lot of enterprises that want to set up things and have them running with as few dev-ops/engs as they can.
- PerformanceAnalyzer [5] could track multiple signals from servers running open-distro and may perform reactionary actions to either shed load, throttle requests, shift workload and so on, something similar to what MeltWater did [6].
The IndexStateManagement plugin is basically the same thing as Index Lifecycle Management (ILM) which was introduced in beta form in Elasticsearch 6.6. ILM is under the Elastic license though, so it’s not free-as-in-freedom, but the feature is included in the free-as-in-beer basic license.
As a heavy ES user, I've looked into getting a license to access the so-called X-Pack features, most of which are very basic and should belong in the OSS distribution anyways. Lots of those features are now available as plugins, but until now there's been no overall solution for getting the missing features.
After an almost two hour long phone call, we kind of got the pricing in a roundabout way from the salesperson; tens of thousands of dollars per year for a small cluster. We REALLY wanted to get this, which is why we dealt with the sales process, but there was just no way to justify that to our management.
Well what about enterprise support? When customers buy an enterprise solution at elastic they get support right? what would entitle them to move to this offering.
well it is not same. vendor support and third party services (aka support). you can influence the product through support and through being a direct customer.
You would think so, but that is not how it works with Elastic and lots of it customers. I have heard that from N of their (ex-)customers. I get why this happens for them. There is a certain vision and goal, and anything that doesn't align is a nuisance. This is part business part philosophy, culture, and brand you want to build in and around your business.
325 comments
[ 6.1 ms ] story [ 296 ms ] threadMaybe it would be more honest to say "Forcing open source backing companies to avoid open core that is hated by AWS" ;) ?
(btw: I'm not associated with Elastic except loving their open source projects)
> Unfortunately, since June 2018, we have witnessed significant intermingling of proprietary code into the code base.
And you mean the x-pack stuff?
https://github.com/elastic/elasticsearch/blob/master/LICENSE...
Interesting, wasn't aware of it.
In what way is being deliberatly unclear?
For those of us with Linux distro building experience, this may be initially confusing. But calling a curated collection of software a "distro" has been used for decades outside of Linux distributions.
For example, there are multiple distributions of TeX, like TeX Live, MacTeX, etc. http://www.tug.org/interest.html#free
See also the Anaconda Distribution for Python and R: https://www.anaconda.com/distribution/ (not to be confused with the Anaconda that I know and love: https://en.wikipedia.org/wiki/Anaconda_(installer) )
So what do we think will be next?
Likewise for the Hadoop project: https://wiki.apache.org/hadoop/Distributions%20and%20Commerc...
I guess you could argue that "distro" has a distinct meaning to "distribution" with the former specifically pertaining to linux distributions, but I think most people probably just interpret it as being shorthand for the latter
Look at any of the long term support discussions, and distros will be mentioned.
Same goes for OpenStack - we refer to the different vendor products as distros (this is complicated by most of the OpenStack in a box products being produced by Linux distros, but we mean all product versions of OpenStack when we was OpenStack distro)
At least in Kubernetes land, that lead to the foundation (CNCF) creating a certified Kubernetes conformance program, to avoid end user confusion and establish a baseline feature set.
Another example from years past, would have been openstack, with many vendors shipping/selling their own branded variants.
The main selling point of the project is providing open source alternatives to features that Elastic has kept proprietary.
Have there been any examples of this happening before?
And it doesn't appear that AWS is hosting these right now, just distributing so they must be paying some kind of licensing for ES. But I'm sure that will change.
Did AWS do the same thing with Mongo?
I believe in response to some licensing changes MongoDB made, so the AWS service targets the last permitted license release by Mongo.
https://aws.amazon.com/blogs/aws/new-amazon-documentdb-with-...
*I forgot to mention, they did not release the code for this as you may have been looking for apple-to-apple comparison with the ES situation.
Amazon DocumentDB was in development for a long time, and development started well before any of the license changes that MongoDB (the company) made to MongoDB (the database).
https://lukasatkinson.de/2019/mongodb-no-longer-seeks-osi-ap...
Re OpenES, its an apache licensed distro without proprietary commercial bits.
One of the concerns I have had with the recent trending of some opensource companies (elastic, timescaledb, and others), is that those who mix in proprietary code into their opensource repos, effectively taint contributors who can't even look at git history on a project without viewing stuff thats not opensource.
For example, I could build a multi-site WordPress instance, white label it, and resell it as a custom blogging platform.
I don't know if this is a bad pattern in general for the company that originally created the core FOSS technology. If your business model is support/consultancy this may work to your advantage.
- the website is copyrighted to Amazon Web Services
- all commits in the github repositories are made by AWS staff.
Given that it was just made public this morning, it would be surprising if it were otherwise. The real question is what the contribution model looks like going forward. The blog post says "Contributions are welcome, as are bug reports and feature requests"[1], but of course the devil is in the details.
1 - https://aws.amazon.com/blogs/aws/new-open-distro-for-elastic...
I worked extensively with both and I prefer the way of administering Solr. Anyway I use Elastic when it involves logging and statistics thanks to kibana.
But some real war stories would make the comparison much more enjoyable...
https://sematext.com/blog/solr-vs-elasticsearch-differences/
Honest question, besides not paying licenses, is there any technical reason to do this?
Open-Core as a business model isn't something that the likes of AWS, Azure or GCP can really tolerate if they want to integrate SaaS that the customers want. DBaaS in particular is one of the bigger advantages of cloud providers in general.
Discloser: A family member of mine does work for Elastic, but I have no knowledge of their internal practices or policies or reaction to this.
There is no more vendor lock-in with this than with Elastic.
I think this is a win for the Elastic community as a whole, but presents a real problem for Elastic the company. And that begs the question of what happens to the Elastic community if there's a real fork. And what happens if Elastic the company is very negatively impacted? Do we see a fracture of the community?
Looking even farther out, at this point does it make sense for any startup looking to create infrastructure software to open source it? If their project becomes successful, they'll get eaten up by the hosting providers. It makes smaller scale open source more commercially viable because you won't attract the attention of the providers that would come in a take your business.
Will be interesting to at least watch what early stage VC investment in open source companies looks like over the next 12-24 months. Will the open source pitch work for series A investments or will investors shy away?
I wonder why you say that? It's kind of obvious that somebody was going to do it sooner or later to protect themselves, no? Perhaps fearing the license change?
> So you effectively get a fork.
Yup, I assume that will happen, too.
With Elastic, this looks like it's half marketing on AWS' part. They want to start repairing their image in the OSS world so they say they're doing this for the community.
So it does seem to be a pattern this year.
I think their open sourcing of this work is a direct response to the recent license changes. If it weren't for those, they might have just kept this work closed and in use only for the hosted product (like they do with many other projects).
They took the closed XPack code and put it into the open source Elastic repo, but under a commercial license. It muddied the waters for anyone looking to use or contribute to the open source.
Now the default install includes X-Pack, and you have to go out of your way (assuming you even realize that there is difference) to install the 100% Apache 2.0 licensed version.
Of course it still does, just not under permissive licenses. Hopefully the industry will converge on Kyle Mitchell's license for this, instead of everyone coming up with their own licenses: https://github.com/kemitchell/api-copyleft-license
I'm no lawyer, but this sounds like complete nonsense.
I’m actually suprised they’re moving so slowly with OSS. I assumed they were going to go a lot more aggressively in this direction when they hired Adrian Cockcroft over 2 years ago now.
Small scale open source isn't technologically viable.
> In the past they haven't seemed to be willing to contribute to the OSS
They are extremely hostile to the idea.
" Unfortunately, since June 2018, we have witnessed significant intermingling of proprietary code into the code base. While an Apache 2.0 licensed download is still available, there is an extreme lack of clarity as to what customers who care about open source are getting and what they can depend on. For example, neither release notes nor documentation make it clear what is open source and what is proprietary. Enterprise developers may inadvertently apply a fix or enhancement to the proprietary source code. This is hard to track and govern, could lead to breach of license, and could lead to immediate termination of rights (for both proprietary free and paid). Individual code commits also increasingly contain both open source and proprietary code, making it very difficult for developers who want to only work on open source to contribute and participate. In addition, the innovation focus has shifted from furthering the open source distribution to making the proprietary distribution popular. This means that the majority of new Elasticsearch users are now, in fact, running proprietary software. We have discussed our concerns with Elastic, the maintainers of Elasticsearch, including offering to dedicate significant resources to help support a community-driven, non-intermingled version of Elasticsearch. They have made it clear that they intend to continue on their current path. "
They started on their own Redis fork a while ago as ElastiCache has had baked in SSL for over a year and half now. While it's possible they've added it via an stunnel type software stop it, I'd bet for efficiency and simplicity it's baked into their internal fork: https://aws.amazon.com/blogs/security/amazon-elasticache-now...
https://aws.amazon.com/documentdb/
Companies that publish open source must realize that they cannot make money from the "software". Open source gives companies a brand, that they can then leverage to do other services like support, consulting, hosting, merchandise, events, etc.
There was a time when service companies wanted to move into products because of high margins and artificial scarcity, but enough open source will ensure that software licensing is not a sustainable model to seek rent, and product companies must also move to services to be able to grow and sustain.
> but enough open source will ensure that software licensing is not a sustainable model to seek rent
So your solution to this is to just make it so no one can make money doing work.
Except Jeff. Jeff gets all the money.
In my mind, a company that makes lots of free contributes to open source is an open-source-friendly company, but this thread and the other on the Elastic post made me realize just how many people think that a True Open Source Company (whatever that means) should prefer to go out of business to dealing in proprietary software.
I personally don't care for any of these puritanical viewpoints that deny economic realities (there's only so much a person or a collective can afford to give away for free, money doesn't grow on trees, etc). Open source is good, but thriving, sustainable open source depends on a symbiotic relationship with for-profit institutions.
[1] https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/
FWIW, I've always found the Amazon store search to be pretty decent; normally I don't use a website's built in search system, opting for Google instead.
[0] https://aws.amazon.com/cloudsearch/
Since when is anybody entitled to any free software. Projects can do what they want with their code, and while free software (the Stallman kind) is nice, if projects cite you as the main reason for their change of direction, you are intellectually dishonest if you just blame them for not being willing to do volunteer work (for you) anymore.
This, to me, looks like an abusive "look what you made me do".
What the blog is saying is "If you build an open source tool and people put their time and effort to using it, closing more and more of the toolset to paid add-ons isn't cool".
AWS just said "Hey, we can build those ourselves and give back to the community". They volunteered and made it happen.
As someone who's worked at many big corps on Elasticsearch installs, I've been worried about them pushing away from OSS for some time. Many big corps want some of these "Enterprise" grade features Open Distro is offering, and if Elastic doesn't already have a foothold in them, I wager they'll be using this instead of paying.
As an aside, how come people still think these Open Source/Core commercial businesses are viable? We've seen time and again that once commercialization occurs, the OSS stuff goes right out the window. (RedHat, Docker, MongoDB, Sentry etc)
As far as Sentry, I'm referring to the fact that they have not cut a release of their on-premise in some time, and features have started to diverge, though they assert they will reach parity with on-prem "at some point" https://forum.sentry.io/t/when-will-a-new-version-be-tagged-... (Though again, like RedHat, the source is technically available...)
My main point is that OSS is at odds with making money.
I'd say opensource business models are different and potentially more difficult, but there are plenty of companies selling/supporting/building opensource. The delta seems to be most vc backed companies sponsoring opensource have different expectations on growth/value extraction from customers. aka open-core is not really about opensource, its about selling proprietary products, so I'd prefer not to mis-label those who aren't doing that.
Getting a good out-of-the-box granular security has been a long overdue pain point with ElasticSearch, there have been good alternatives for the other problems it is replicating.
1.) ElastAlert - https://github.com/yelp/elastalert for alerting, 2.) ElasticSearch SQL - https://github.com/NLPchina/elasticsearch-sql for writing SQL queries.
Would very much like to hear from OP why these weren't contributed to instead of creating another alternative.
We've been working on a similar project in Golang and now well may be as good a day to open-source and put it out there: https://github.com/appbaseio/arc. It's an extremely light-weight API gateway for ElasticSearch that at the core comes with an out-of-the-box security system based on users and permissions (set granular ACLs, Rate Limits, TTL) inspired from a proprietary security system we built over the past year - https://appbase.io/features/security/.
[0] https://aws.amazon.com/blogs/aws/new-open-distro-for-elastic...
By forking on an earlier instance of the codebase for ES and Kibana they're not only creating an open source version of the code but also attempting a fork of the community - plugin developers, user groups, etc. It's extremely frustrating. EDIT: If folks actively building plugins have felt pain here because of decisions by Elastic, then my view on this point changes considerably, obviously.
I find it very hard to believe there was no compromise to be had here between the current Elastic roadmap for their product and their previous work.
How is this good for users of the current tools? How does this help create better capabilities on a common platform? Will this encourage users to build businesses on elasticsearch open source elements?
I'm not sure where the community ends up after this, so I'm not sure I can support by using AWS' elasticsearch tools.
Why wasn't ElastAlert used / built upon for alerting? https://github.com/yelp/elastalert
Or why wasn't ElasticSearch SQL used / built upon for SQL? https://github.com/NLPchina/elasticsearch-sql
Or why wasn't SearchGuard used / built upon for Security? https://github.com/floragunncom/search-guard
All of them have Apache 2.0 license.
More generally: why should AWS get to leverage its near-monopolistic position in the SaaS market to perform essentially hostile takeovers of the software projects it packages and runs? Simply because they have the resources to do it?
The code was open, and the company decided to close it for the new versions, so it forks. Forks are common when people don't like the corporate direction of an OSS project - It's a strength, not a weakness. Openoffice/Libreoffice, Hudson/Jenkins etc.
Imagine if Linus said that the next version of Linux would be paid-only.
Within 15 minutes there'd be a new librelinux repo that people could contribute to instead.
Someone was going to make a new fork anyway - This is Amazon putting their money where their mouth is to fund and support that new fork.
Would you feel the same if AWS built a drop in/api compatible system from scratch?
https://github.com/opendistro-for-elasticsearch/security/blo...
According to the release blogpost [0], "SQL Support [...]is an improved version of the elasticsearch-sql plugin."
Sounds like they are building on it, as you suggested.
[0] https://aws.amazon.com/blogs/aws/new-open-distro-for-elastic....
Looks like it was. https://github.com/opendistro-for-elasticsearch/security-ssl...
floragunn GmbH Copyright 2015-2018 floragunn GmbH
[1] https://www.elastic.co/blog/elastic-stack-6-5-4-released
>>This means that the majority of new Elasticsearch users are now, in fact, running proprietary software. We have discussed our concerns with Elastic, the maintainers of Elasticsearch, including offering to dedicate significant resources to help support a community-driven, non-intermingled version of Elasticsearch. They have made it clear that they intend to continue on their current path.
... maybe I'm misreading your comment, but it sounds like AWS tried to find a compromise, including contributing to upstream, and were shut down (and all their content about future contributing seems to say "we'll send it upstream, but it depends on whether Elastic accepts it," so seems to corroborate). It honestly seems like you get more certainty that everything you're using from AWS is open and will remain open, and Elastic is going to keep creating uncertainty / drive to proprietary source-available.
That said, I also totally understand disagreements with Elastic’s decisions on charging for security, etc and why that causes concern. As a user I wish I wasn’t staring a fork in the face and I’m hopeful things get back to equilibrium here soon as a single community.
(or if not ASF for some reason, then Eclipse or CNCF for example)
For the long-term, the open source community should look to projects where the center of gravity is outside of an open core or FANG company. ElasticSearch went open core a while ago, and Elastic's form of open core is more like MongoDB or Redis than it is like nginx or Docker. As for Amazon, I think I'd rank them below facebook, and I would rank facebook below google and Microsoft, as stewards of open source projects.
In the long term their actions make it impossible to build any sort of business based directly on FOSS. You have to do support, or hosting, or something else.
Guess what, AWS has a lot more money to do both of those things.
and Amazon will keep right on going sucking up code and giving nothing back.
We need aggressive licensing changes.
Hell, API copyright might be necessary to stop this nonsense. We can still have good licensing in that scenario.
I jotted down instructions to run the docker containers based on a few different solutions from different places: https://gist.github.com/mcescalante/6be03751c820677cf0f15c7b...
I'm sharing this because I got the vibe from the linked post in the comments and the overall marketing of "Open Distro for ES" that Elasticsearch was 'not open anymore' and had just discovered the Apache 2.0 builds within the last week.
Apart from including/forking other community plugin projects (elasticsearch-sql, SearchGuardSSL) in open-distro [3], AWS seems to be investing to make this more "cloud-native", as well: This observation comes from PerformanceAnalyzer (released) and IndexStateManagement (request-for-comment).
- IndexStateManagement [4] is simply "never send a human to do a machine's job" automation of common yet frequent tasks required to keep elasticsearch humming along. If used in conjunction with AWS's/GCP's/Azure's managed-cloud offering, this feature might be an amazing sell for a lot of enterprises that want to set up things and have them running with as few dev-ops/engs as they can.
- PerformanceAnalyzer [5] could track multiple signals from servers running open-distro and may perform reactionary actions to either shed load, throttle requests, shift workload and so on, something similar to what MeltWater did [6].
[0] https://arstechnica.com/gadgets/2018/07/googles-iron-grip-on...
[1] https://techcrunch.com/2009/12/22/google-open-when-convenien...
[2] https://www.theguardian.com/technology/2013/oct/16/amazon-se...
[3] https://news.ycombinator.com/item?id=19361234
[4] https://github.com/opendistro-for-elasticsearch/index-manage...
[5] https://opendistro.github.io/for-elasticsearch-docs/docs/pa/...
[6] https://news.ycombinator.com/item?id=18413862
https://www.elastic.co/guide/en/elasticsearch/reference/6.6/...
As a heavy ES user, I've looked into getting a license to access the so-called X-Pack features, most of which are very basic and should belong in the OSS distribution anyways. Lots of those features are now available as plugins, but until now there's been no overall solution for getting the missing features.
After an almost two hour long phone call, we kind of got the pricing in a roundabout way from the salesperson; tens of thousands of dollars per year for a small cluster. We REALLY wanted to get this, which is why we dealt with the sales process, but there was just no way to justify that to our management.