With the rise of cloud computing, the effort by cloud providers to commoditize their complements, i.e., software, inevitably leads to this particular kind of conflict between companies backing OSS software who wish to deploy their own SaaS offerings and cloud providers with their own SaaS offerings. A common response is that OSS backers can build businesses based on support and consulting but I believe that's overly restrictive. Those kinds of businesses don't scale nearly as well as managed services and it will definitely have a chilling effect on OSS development if the kinds of companies that are permitted to be built on them are simply support and consulting. I don't believe that companies backing OSS have an exclusive right to offer managed services for their software, but the current situation with cloud providers offering managed services with only token engagement with the OSS community (and then only when forced into it by licensing restrictions) is not beneficial to OSS, either.
The post describes how Amazon offered more than a token engagement. Elastic rejected them, as such a partnership would have included "preferential treatment that would place them above our users". That's consistent with what I've seen from other OSS-backing companies; they want cloud providers to contribute to their community, but flatly refuse to go out of their way to enable it.
I didn’t see much done to address Amazon’s claims (intermingling of proprietary/open code in same conmit, etc). Mainly they just claimed FUD and reminisced a bit.
If I were Elastic I’d be quaking in my boots right now - their entire business model is under assault. Hell, the whole open core model is being called into question. Personally I’m glad to see Amazon’s distro.
Just my 2 cents. I don't care about Amazon's distro, I think it won't work. Others have tried, e.g. MariaDB vs MySQL, OpenOffice vs LibreOffice. Time will show though...
Edit: The thought was about big companies with their own agenda behind open source initiatives. Sorry about confusion.
What pattern are you seeing in those examples? Most Linux distros today ship with LibreOffice and have MariaDB in-repo. From where I sit, the open forks won.
Yes, the fork timeline/direction is different, but imho, the pattern is, big companies with their own agenda, hiding behind "open" fork. I only can imagine the amount of resources put into keeping up. Will Amazon "rebase" their version with Elastic's? I doubt that for long time. Features advantaging AWS will be pushed, for better integration with 3rd party, which basically AWS' solutions.
I hope that people will be able to use "vanilla" ES too on AWS.
> big companies with their own agenda, hiding behind "open" fork.
MariaDB is a community fork away from a controlling company. Libreoffice is run by its own foundation, again after forking away from a company-controlled version.
> I hope that people will be able to use "vanilla" ES too on AWS.
I mean, worst case just run it on EC2/ECS, which lets you run anything you want. They're not going to break that.
Won in terms of user base, maybe. But in terms of money made and commercial adoption? Mysql beats Mariadb, though I can'
t say for openoffice/libreoffice.
Is that a fair metric to judge on? The entire point of forking MariaDB was that the developers expected Oracle to make compromises for the sake of money.
I guess a big driver for me is a lot of the proprietary features (example: encryption in transit, RBAC, audit logging) seem absolutely basic to me. Not basic as in easy to implement, but basic as in as an operator I’ll need them for every type of logging/monitoring infra I’d ever spin up.
Worth noting I consider Elastic’s managed offering to be superior to AWS’ in a lot of ways. But personally I’ve always chosen to build my own setup off EC2
The difference is who is using them. In this case it's a trillion-dollar company that's also going to resell the fork as a service to their already locked-in customers. You betcha it's a threat to Elastic.
I don't think open core is being called into question. By that rationale, anything that is closed source that gets too popular is also called into question. Nothing is stopping AWS or anyone else from copying your API, features or anything else regardless of if your product is open, closed, or some combination in between.
I think if anything is called into question, it's the development of infrastructure software in a startup. Does it make economic sense anymore? As in, is it possible to profit directly off it at this point? Right now it obviously is, but it seems to be trending away from that.
That’ll (maybe) be a more compelling argument for one side or the other of this argument when it is finally done.
Especially if the Supreme Court rules, and it's not just the Federal Circuit’s interpretation of what the Ninth Circuit would say, which is not binding on lower courts (or other circuit courts, including the actual Ninth Circuit, in other cases.
Clicking through to the AWS blog article [1] we find this:
> At AWS, we believe that maintainers of an open source project have a responsibility to ensure that the primary open source distribution remains open and free of proprietary code so that the community can build on the project freely, and the distribution does not advantage any one company over another. This was part of the promise the maintainer made when they gained developers’ trust to adopt the software.
I have enormous respect for Adrian's work on cloud systems but the quote above seems very self-serving for Amazon.
Open source is bound by licenses that define the terms of open source code use. There's nothing whatsoever in the licenses that binds the project developers to continue to work on the project, to maintain similar license terms in later releases, or to take (or not take) any other action in their lives. To argue otherwise seems contrary to the notions of freedom that are the foundation of open source software.
I think it’s fine for Elastic to seek to go to a non-open-source license (whether source-available or not) for new developments or portions of their code. It’s also fine for others, including AWS, to take the open-source portions and fork/continue developing on the open source fork. In such a situation, I’d expect open to win over proprietary.
Amazon can believe whatever they want, but when they say "maintainers of an open source project have a responsibility to ensure ... " - I call bullshit.
Those maintainers owe Amazon nothing. The have no responsibility to Amazon. None.
Same as they owe me nothing.
By definition (if you use any of the more-that-two-or-so-year-old definitions of "open source"), the project is defined by "remaining free and open", in that you can always use the code and license you've been using - but there's no way anybody can make it a "responsibility" of a maintainer to to anything in the future. They could die, they could get bored, they could get an all-encompassing job, they could get a family, they could get venture capital and license any future work with a non-open source license (as Redis and Elastic have done, at least in some people's view).
The "the promise the maintainer made" was "here's some code, under GPL or aGPL, or Apache Licence, or Artistic license of BSD three clause, or whatever the promise at the time was. Take it or leave it. No future responsibility assumed or enforced. They might even have a track record of releasing more stuff under the same license/promise. But they can stop or change things whenever they want.
If Amazon wants the maintainers to do anything else (including ever committing another line of code or fixing a bug), they can wait for them to do (or not) it at their discretion, or they can pay the maintainer (or someone else) to do it (and then, depending on the license, potentially be required to release that code under the same license.
Sorry for the rant, but that line caught my eye too, and I got immediately pissed off at Amazon's sense of entitlement there. No open source maintainer owes you shit. They've shared what they've shared, and you can take it or leave it, but you taking it doesn't place any obligation or responsibility on them.
Maintainers are certainly free to adopt a "take it or leave it" attitude if they want.
But most maintainers (including Elastic) don't want that. They proudly declare that their developers and users are a community, and they're steering the community to produce a great and open project on an ongoing basis. The maintainers certainly have a responsibility to follow through on such promises - or at least, if Amazon is correct and they aren't following through, they have no right to complain about a hostile fork.
Amazon is welcome to fork and welcome to point out that they might be better for users than Elastic. But the issue here is that Amazon--like many others before them--are asserting a 'right' that simply does not exist.
I don't think Amazon's trying to claim that Elastic has done something legally or morally wrong. They just don't feel Elastic has lived up to their standards for community management, so they're going to take their ball and go start a new community with higher standards.
Promises without consideration (e.g. unpaid) aren't enforceable.
I help out by sweeping Amazon's yard everyday, for free. "Thanks!" they say, "Any time!" I reply. But years later, I decide to stop. Amazon cries out in anguish "But you have a Responsibility!", and threatens me that if I don't clean their yard... well, they'll do it themselves.
Free maintained software suits Amazon, and some organizations / foundations provide it. Seems to work as consulting services advertising (BTW which incentivizes hard-to-use software, lots of complicated features, jumbled together... ever seen a large foundation's open source project like that?)
Some businesses fund these foundations directly, but minimally, and there's a certain... reluctance. Perhaps an unmet need there...?
> Amazon can believe whatever they want, but when they say "maintainers of an open source project have a responsibility to ensure ... " - I call bullshit.
It is true that Amazon here is just stating their own interest and are not well placed to give lessons of open source implication. Elastic has paid many developers to build open source code.
However, Shay Bannon on his side, is too much using the word "open" for a company that is not running and from my POV cannot run a real open governance. Elastic is now a listed company and was VC backed. Presenting the integration of xpack in the core repositories as an "opening" when it's a move backwards in terms of open source governance, is not fully honest.
There is a fundamental issue with Open Source and Investor backed companies. The main goal is not open governance, but the main goal is the company.
Elastic is not clear about this. Actually not being clear means there is no open governance.
Shay Bannon won't go there, because I believe as the CEO of a listed company he can't, unless it's in the interest of Elastic's investors, which means the board should decide.
Elastic does not have to have open governance and can be a great OSS contributor nevertheless. And to the opposite, it is very possible that without very healthy Elastic.co, that there would be less OSS code.
However it raises questions about the future. What will be the next step ?
What I would prefer is to have Shay Bannon be fully honest about the fact that Elastic.co needs to be healthy for the project to be healthy unless other companies are heavily investing in the community and start contributing code. And that Elastic.co being healthy means to have a mixed model. And finally I would like to hear that Elastic.co cannot promise that this won't change because Elastic.co might have to change it if the investors decide it but that as long as he is CEO he will do his best to maintain the current mixed model as long as it is financially successful.
It would be more honest that saying "doubling down on open" when you are tainting open repositories with non open source code. There was no technical reasons that the xpack repositories could not be separate.
Also, there is a way to not give an advantage to any company in an open governance way while helping companies be healthy. You can decide to have an App store in which paying features are proposed only from core contributors that contribute more than a certain level. Business could be added to the goals of the open community.
It would be interesting if someone pushed a PR that contains more or less all the "secret sauce" of X-Pack, and see what reasons they'd use for rejecting it.
Clarification: Not push X-Pack itself, but features comparable. Like they suggested people do in the post.
"When companies came to us, seeing our success, and asked for special working relationship in order to collaborate on code, demanding preferential treatment that would place them above our users, we told them no. This happened numerous times over the years, and only recently again, this time with Amazon. Some have aligned and became wonderful partners to us and the community. Others, sadly, didn't follow through. We have a commitment that we will treat a single developer contributing to our products the same as others. There is no preference, and we will reject any ask to have one. Our answer has always been a constant: send a pull request, like everybody else does. The quality will speak for itself."
The way this is written, it seems like there was a difference of opinion in terms of the product architecture, and not a purely commercial decision by Amazon. I would think it would make commercial sense for the team at Elastic to align with Amazon than the other way round. On the other hand Amazon claims that Elastic was moving away towards proprietary code [1].
Overall, being in open source for a decade, I am against this trend in dual licensing (which I fear is pushed by Venture Capitalists). People forget that the fact that you have freedom and zero friction, is what makes open source awesome. There are other ways to make money, like building a kick-ass consulting practice around the open source offering.
Lot of bitterness about people forking and bundling ES. If it was so painful for Elastic maybe they should have gone for a more suitable model - you get everything open source and pay for support and priority /early access to security updates. Unlike the current model where you get basic features as open source and have to pay for and use different license if you want to use essential features like monitoring and performance analysis.
Someone else with a business model that benefits from providing these features for free and making them open source to reduce their development/maintenance burden - cloud providers like Amazon - are going to do just that. And they're well within their right to do so.
Impeding security updates seems rather unethical. Unless you mean backports to old/stable versions, which is still iffy in my mind but does at least maintain a way to stay secure for everyone.
I meant priority/early access to security updates really. Updated my post to reflect that.
For example enterprises may want quicker access to well tested bug or security fixes - they can get that first and then the patches appear on GitHub for open source users.
Seems unethical for people benefiting from something to externalize the cost of security, expecting bug hunters to work with no benefits. I could lean toward selling security fixes if it was commercial software. Not sure about free/open yet. If something like Tor, I'd rather we make exceptions regardless of general philosophy just to keep majority of people safe. Especially the kind of folks that help us out with their sacrifices.
Most software isn't like that. My current preference is going back to highly-permissive, shared-source licenses by public-benefit companies whose charters and contracts don't let them screw people in at least all the ways we know about. Perpetual, free licenses w/ no or optional support for non-commercial or small business. Cost goes up with scale from there maybe with enterprise features sold separately. Anything not a competitive advantage, esp building blocks or platform stuff, might get open-sourced. Anyone licensed can modify the software and distribute that to other licensees. Might make exception for features or code in new version so they can't skip payment with backport. Maintains most of benefits of OSS with less risk of proprietary or hybrid licensing.
Charging for security fixes could be a simple way of motivating folks to pay for improvements to the software. A preferred model that's harder to build is using methods that make nearly vulnerability-free software with a warranty against specific types of defects or at specific, defect rates. Cleanroom shops and Praxis Correct-by-Construction did that with an upfront premium applied with defect rate in warranty fixed at supplier's cost. So, still thinking on specifics of all of that or potential integrations.
This is a really great response and I'm curious how AWS will respond. It's particularly damning because AWS was arguing from a position of moral authority on the move, which I think was disingenuous. They wanted to create a fork and act like they weren't. Shay just called them out on it.
I also wrote up my own thoughts, but it's obviously not as relevant as this. I'm a little more blunt about what AWS is doing. I even throw in a comparison to Microsoft abusing their monopoly power in the 90's by distributing Explorer for free to kill off Netscape: https://www.influxdata.com/blog/aws-intends-for-their-new-pr...
Longer term, the real question is how Elastic will respond to this commercially. Will they create more closed source? Go higher up the stack into specific applications? It's going to be an interesting and tumultuous couple of years in open source infrastructure software.
Elastic doesn't have much moral authority either when they keep using the word "open" in reference to code that isn't open source. But they never say "open source" so technically correct.
EDIT: you've remarked elsewhere on the thread about commercial and economic realities, so I want to take the opportunity to qualify my statement: they're free to license and monetize their code however they want. It's what I see as deceptive intent that I'm having a (very strong) reaction to.
Iiuc, they are basically using open core model, with core app being free/open/libre under Apache license, and extensions proprietary with code available non-free/non-open/non-libre Elastic license. At least to me the situation seems pretty clear, and Amazon complaining that they can't take all of the code to create a competing service seems pretty disingenious to me. OP's post is thus in line with what I would expect - but it seems to irk you in some way. What is it that I'm missing?
I said more here[1] but basically the use of "open" around their gratis/proprietary code. Never "open source" so it's not an untruth but it's still misleading especially in a context where they're also talking about their contributions to open source, a commitment to being open, etc.
Elastic is not in the right here, Amazon is, and that should shame Elastic more than any snarky HN comment could. Yet it’s clear either they don’t understand the critique being lodged, or they believe it’s in their interests to willfully distribute sloppily licensed code and implicitly market it as open source (whether or not they use that exact phrase).
Wow elastic seems salty. I don't think this blog post was a great idea. While I understand that they need to monetize their project, I have been put off by them putting their security features behind a paywall. They literally advertised "Security" as non-existent on their XPack pricing page for their basic free plan.
Furthermore, I am still laughing about how the author litteraly wrote that Amazon "bluntly copied" them, "sadly, painfully, with critical bugs".
Aws is copying others’ efforts and earning billions. Nothing is being given back to the community, instead Amazon execs buy Bay Area luxuries. Stop aws manipulating innovative projects in the name of managed services. Elastic, mongo, Kafka, redis, all are being preyed upon.
This has been part of Amazon's business model for years, they have 10,000s of products copied from their vendors. They have even copied stuff hosted on AWS in addition to open source software deployed on AWS.
> Stop aws manipulating innovative projects in the name of managed services.
How? Redis is trying, and got backlash. Mongo is trying, and got backlash. Confluent is trying, and got backlash. We've rather got to pick our poison here: do we want companies to open their code and have it used for managed services/cloned, or do we want them to close it and keep making cool tech?
I recall reading about the model Google used to use for this sort of thing: open your source code about three to five years after you stop using it. This gives something to the community, while still allowing you to maintain a competitive advantage, run a managed service, etc. Particularly for those who run a SAAS model, it lets them gain widespread adoption before a competitor can spin up an instance of their stuff and undercut them slightly.
Oh, and for those saying "use the Red Hat model", the Red Hat model can't work for everyone. Red Hat had a new idea and essentially no competition for enterprise linux at the time, and so didn't have to deal with the competition. They got a chance to build a customer base before Oracle knocked their product off. And even then, Oracle did completely rip off their work. That's the risk you run with open-sourcing your codebase, and not every company is built to work that way; not our job to tell them they should redo their business model to allow it.
A lot of conversation around the backlash was about the insinuation that the new licenses were open when they weren't. "Apache 2 + Commons Clause" is an abuse of both the Apache license and the concept of the commons; the SSPL was submitted to the OSI for approval when it clearly did not meet the definition.
Why write emotionally charged letter instead of one backed with facts? Amazon is claiming [1][2] that Elastic has been intentionally unclear about what part of Elastic stack is open source vs proprietary. It would best serve Elastic if they respond to that with facts.
If the claims are true, and done intentionally, it's a really bad move on part of Elastic. The ambiguity around licensing hurts all their users and not just Amazon.
[1] "Unfortunately, since June 2018, we have witnessed significant intermingling of proprietary code into the code base. While an Apache 2.0 licensed download is still available, there is an extreme lack of clarity as to what customers who care about open source are getting and what they can depend on. For example, neither release notes nor documentation make it clear what is open source and what is proprietary."
[2] "Individual code commits also increasingly contain both open source and proprietary code, making it very difficult for developers who want to only work on open source to contribute and participate"
I couldn't agree more with Amazon. We build something on Elastic - and trying to work with them to get licensing was at best a major PITA. In ~2017 Elastic decided they wanted to go Enterprise Only. They started to charge $100k for x-pack, though one could get a dev license for $50K.
We quickly built a SQL parser for Elastic, an alerting engine, some other bits. These are all bits that just went poof yesterday when Amazon released opendistro - and I couldn't be happier. Sure we'll have to slice out a bunch of code, but at the end of the day it didn't have value to our core business.
I just wish Elastic as a company was easier to deal with. We'd be months ahead if not for them - and ultimately just worked around them. This is in spite of MY using Elastic for ~9 years now.
Not worth their target of $500M. Quite frankly Amazon just sunk the value of Elastic in a huge way. I'm trying to figure out if anyone here will let me pre-register for a short.
This he said, she said style debate is indeed unproductive.
If I read this correctly, Amazon tried to get a special deal from Elastic basically in order to produce a competing service. Elastic declined to cooperate and now Amazon is exercising their rights under the Apache license. So, far so good.
It's the provocative website that makes it seem like there's some huge conspiracy by Elastic that provoked Shay Banon's response. And not the fact that Amazon are doing what they are doing (i.e. redistribute the OSS parts of Elasticsearch), Elastic seems to have little problems with this because their license allows this and as Shay points out others have done this as well.
Amazon's complaints boil down to two things:
1) Elastic distributes a version of their software that includes non OSS components. You don't have to use these but you can. Actually Elastic offers rpms containing only the apache licensed components on their download page: https://www.elastic.co/guide/en/elasticsearch/reference/curr.... So this is maybe a bit less true than Amazon is portraying here. This would be what you'd install on a server if you are self hosting. Likewise there are OSS docker images from Elastic. They also have oss docker images. Elastic does this because up-selling this stuff is how they make money. Amazon's beef seems to be that it would be awfully nice for them if they could offer an alternate distribution more prominently without all this stuff so that they give that to their own customers.
2) Their repository contains some well documented directories that contain non Apache licensed source code for these components. As far as I know this is pretty well documented and if you are modifying files in a directory, presumably you know what you are doing. This is because they want to develop most of this stuff in the open and be able to accept pull requests against both their OSS and non OSS stuff. This seems pragmatic and preferable to producing binary distributions with some components from private repositories.
"Source code in this repository is variously licensed under the Apache License
Version 2.0, an Apache compatible license, or the Elastic License. Outside of
the "x-pack" folder, source code in a given file is licensed under the Apache
License Version 2.0, unless otherwise noted at the beginning of the file or a
LICENSE file present in the directory subtree declares a separate license.
Within the "x-pack" folder, source code in a given file is licensed under the
Elastic License, unless otherwise noted at the beginning of the file or a
LICENSE file present in the directory subtree declares a separate license.
The build produces two sets of binaries - one set that falls under the Elastic
License and another set that falls under Apache License Version 2.0. The
binaries that contain `-oss` in the artifact name are licensed under the Apache
License Version 2.0."
So, you might not agree with this arrangement but you can get or build pure OSS quite easily and many people do. And it is hard to argue against Apache 2.x here. The license pretty much makes it obvious how stuff is licensed. This is unsurprising. Any confusion here would be a legal bug; this stuff needs to be unambiguous and it seems to be.
BTW. I think it is great that Amazon is providing their own distribution for their customers. It would be awfully nice though if they'd update it to the current version and not ship already fixed bugs. They have a history of doing this with Elasticsearch; probably because they are not on speaki...
While as an open source software and company creator I highly connect to the emotional content of his response and feel the pain for Elastic when it comes to the competition by Amazon, I do agree that he does not address the factual issues at hand and uses badly the word "open" for stuff that is "shared source".
One aspect Elastic is also not directly attacked on and therefore not addressing is the choice of features that are not free and the price point. Elastic's business model is quite smart. You can easily develop on top of elastic, however once you have done it and want to go to production some key features are missing and they are not cheap. Particularly securing your cluster data is one of the features in xpack which is problematic not to have. I'm pretty sure you can search for open ports 9200 over the internet or company intranets and find unwanted unprotected data. And it's not cheap as you need to go for Gold subscription if I'm correct. I would be surprised nobody has proposed a pull request with competing code.
Elastic has and is investing a lot as open source and they do need a good business model that allows investment in the core open source code. The thing is I'm not sure it's missing here. Elastic.co is quite healthy. It's hard to know if it would be without this business model, but anyway there is little chance they would do differently given Elastic.co is VC backed. In the end the real fight is between the big money players, not about whether Elastic the software is sustainable. The reality is that given the community, it would very probably be even without the "xpack" package. Now it's not sure Elastic.co would be able to aggressively invest in both oss and non oss as much and therefore compete with Amazon.
OSS is tough. When you go for it, you should go in it for OSS, not for the company, because keeping the company at the level of the OSS success is hard both for small and big OSS projects.
You can put a reverse proxy in front of it to provide at least basic authentication measures and force HTTPS. Better than nothing at least.
But the main problem is that unsecured clusters by default have caused a lot of reputation loss to the brand. When every few months news hit about yet another unsecured Elasticsearch cluster that leaked huge amounts of data, it is getting harder and harder to explain to the less informed how that is the fault of those people who did not even bother running a reverse proxy, not the fault of Elasticsearch itself.
> and uses badly the word "open" for stuff that is "shared source".
This is my only complaint with what Redis/Elastic/Mongo et al are doing lately.
I 100% support their right to change their licensing terms. Applaud them even - I'm fascinated to see if their attempts at new business models will work out medium/long term.
Doing that and still claiming to be "open source" is wrong. It's co-opting a term that while not solidly defined, is universally understood to mean something different to what they're offering. They're lying to their users and customers.
Elastic are muddying the waters in a grey area - where you can build Apache2 licensed or Elastic's new non-FOSS licensed versions from the same download. I reckon they need to cole clean and make if very clear that they're "partially open source".
When ES writes effectively a response, the initial comments don't like ES's response or what they're doing either.
It may be that the collective HN vibe is critical no matter what. But also maybe "we" in aggregate legitimately don't like either option given.
I know open source devs need to make money somehow. I also wish there was a way to do it that was open source. I'm not sure there is. I'm not sure what it means for the continued viability of open source.
In the original period of open source, people generally worked on open source on the clock at their existing jobs, a building tools they needed for that job, a job not dedicated to that particular product but to getting something done that product helped with. People were getting paid to write open source, but it wasn't by "productizing" it. The results were still shared freely.
(this is how/where apache httpd, for one, came from; people with jobs who needed an http server writing one on the clock, even though their job was not "writing an http server." They needed an http server to accomplish what was needed for their employers, but their employers had no desire or capacity to be in the business of selling an http server. Why not collaborate on one we can all use?)
But those days are gone, software has just gotten _so much more complex_ and time-consuming to develop. The open source software we have and need can't be developed in someone's "spare" (even on-the-clock spare) time. Some open source is still written by people working for a large employer, who pay them to be dedicated to that product. But we see the perils of that when an employer decides "wait, why aren't we making money selling this? Why are we contributing our employee's time to it?" (thanks Oracle).
So that leaves people trying to productize it one way or another. It is not clear to me how to have a healthy open source ecosystem in that market environment. I don't think "proprietary layer on top" is actually good for open source, especially if it's gonna lead (naturally) to rejecting contributions from outside to add those same features as open source (why would we want to spend time working with a contributor to make the feature solid, when we _already wrote it_, and when accepting the contribution would _harm our business model_ because that feature is what we want to charge for?), and then resistance to the "right to fork" which I consider the fundamental basis of open source freedom.
Perhaps the thing just is that "programming open source stuff" is a shitty business model?
Build a database, not a database engine.
> Some open source is still written by people working for a large employer, who pay them to be dedicated to that product.
Take e.g. tensorflow by google. The software itself, is per se worthless, so they can release it as OSS as some kind of calling card or recruiting tool. The crown jewels they are guarding closely and are never letting any outsider access directly is their data.
> Take e.g. tensorflow by google. The software itself, is per se worthless, so they can release it as OSS as some kind of calling card or recruiting tool. The crown jewels they are guarding closely and are never letting any outsider access directly is their data.
No, I think it's a little more complicated than that. I don't think this is a good comparison.
Tensorflow is only "worthless per se" because it was completely open sourced. It's quite a sophisticated library and a substantial product could have been built using it if it was kept proprietary.
However, Google got to the space first and Google is fundamentally a software service company. The reason they open sourced Tensorflow is because they have data and recognized the data would be more valuable; hence, they decided to commoditize the complement. By open sourcing Tensorflow they made their data even more valuable (likewise for their services which can be used to build and manage a data pipeline).
> I'm not sure what it means for the continued viability of open source.
I believe you're conflating "open core" with with "open source". I don't believe that open source needs a profit motive to exist and flourish. While this may be a blow to the open core FOSS business model, there are a number of other models that continue to exist that do not involve proprietary enterprise extensions.
While these models may not be attractive to venture capitalists, they can certainly provide a way for developers to make money.
> I don't believe that open source needs a profit motive to exist and flourish.
The question is, who is going to write open source projects at a complexity/polish level similar to ElasticSearch. And how are they going to pay their rent?
People working on it need to pay the rent somehow.
I am suggesting that the way people paid the rent while working on open source historically in the "first phase" of internet open source (doing it on the clock at jobs for which the open source was useful for their job role, but producing the open source was not their job) -- is no longer working to get software written at the complexity level of 2019 software, as software gets much more complex requiring more people coordinating with themselves and spending more hours.
I agree, at least for the sake of argument, that "open core" is distasteful and problematic. I believe it was a response to the question of "how do we write open source and still earn a living." It may not be a satisfactory answer. I'm not sure what other answers exist. If there aren't better answers, open source will shrink, including by "open core" and such.
I reject the notion that Elastic is in this situation because they need rent money. In my view, they're in this situation because they accepted over 100M in venture capital, and now those investors want to see a return.
I believe that "open core" was not a response to the question of "how do we write open source and still earn a living?" but rather "how do we leverage open source to make a billion dollar company?". I think the distinction, however minor, is important.
I find that a reasonable line of inquiry, but the obvious next question is: How would they have paid themselves salary without the venture capital? How would they have done open source and still earned a living without it?
I am not sure there are obvious models/examples that are working. I think there's a reason that "open core" is an increasingly popular thing to try to do. I'm not sure if I can think of many earning a living while producing high-complexity high-quality open source software _except_ "open core".
(And I still think "open core" is problematic for the continued viability of actual open source).
Do you know how the people who work on vue.js get paid? Do they have fulltime jobs as employees? Do they get paid to work on vue.js?
I find in general it's easier to get free labor on something the newer it is. Both because it's more exciting, and because the more 'mature' something gets the _more work it is_ for _less gain_, there's more time dealing with backwards compat and filing off rough edges and keeping the existing code just working, which is all less interesting to do for free. Plus, the more mature something is, the _more work_ it _takes_ to keep it going.
Well, there is an option of selling services which is even worse, both for developers and the product itself.
Or, there are Fair Source License / Commons Clause / Business Source License /... that allow developers to capture some of the value they generate, but this of course means they are not open/free/libre because they limit some of the freedom to accomplish that.
Many of the most successful open source projects I can think of were funded by universities (Linux and Gnu) or natural monopolies (Unix). Their creators didn't become billionaires, but the sources of money were willing to pay them a comfortable living. Maybe it's the expectation that open source should be a massive fountain of wealth that is the real problem.
I have to strongly disagree here. Stuff like Kafka from LinkedIn, Envoy from Lyft, Prometheus from Soundcloud are just the tip of the iceberg of people writing tools because they need them, not because they are in the business of making infrastructure software.
I believe that selling software, be it foss, open core, or fully proprietary is not good business model, neither from business nor from ethics point of view. Sell your expertise or use the software yourself to generate value, but trying to sell it is just futile.
Funny you should mention Kafka. Jay Kreps, Jun Rao, and Neha Narkhede, formerly of LinkedIn who worked on Kafka, left to form Confluent. They provide support, managed cloud, and licenses for their own flavors of Kafka and ecosystem add-ons, and recently made license changes to their open source add-ons to block providers like AWS from offering them as SaaS as well.
Considering the direction Microsoft has taken, I think they in particular would agree. It is no coincidence that MS is more and more straight up giving away their software, or at least bundling it as a service.
The critical thoughts are the ones that generate comments. How good is a comment that says “yeah this is good, I agree”? If you are angry or unhappy, that’s when you feel inclined to comment.
Clicking through to the other blog posts, it becomes apparent the the author plays fast and loose with the word "open", to their benefit. They "opened up" by providing gratis licenses to some of their software. They don't call it "open source", but the phrase "open source" is also scattered around the articles, so that "open" benefits by proximal association.
They're free to license their software however they want; I don't use it so I really don't care. But it really rankles when someone extols the virtues of open source and waxes rhetoric about how open they are, and yet the software they offer is proprietary by default. They "double down on open", but the open source distribution is opt-in.
I cannot help but feel that the articles are deliberately constructed to mislead. That's a serious accusation and I wish I could be more charitable in how I interpret them but the terminology is so consistent that it cannot be an accident -- gratis-but-proprietary is always "open", libre is never just "open" but always "open source".
Technically correct isn't enough. No-one's explicitly staked "open" as being the same as "open source". But in the context of this conversation "open" is an obvious contraction of "open source" and to use the obvious contraction to mean something diametrically opposed is not reasonable or ok; especially when "source available" is widely understood. The same thing was attempted with the Commons License.
The intent behind words is important. Persuasive oratory is fine but using those tactics to mislead does not engender trust, especially when the beneficiary is the one speaking and not the audience.
Actions speak louder than words. Amazon's words [1] were backed up by action- specifically, the public release of Open Distro for Elasticsearch and several useful, advanced features under an Apache 2.0 license.
This post from Elastic consists of vague claims of 'FUD' and of their commercial code being 'bluntly copied', without any further clarifying details or responsive action to back them up. (E.g., if you have real evidence of Amazon copying proprietary code, which I doubt, then announce the copyright infringement lawsuit you have just filed, don't weakly allude to your grievances in a blog post.)
Conspicuously, the post did not contain any defense or even mention of Amazon's key accusations of "significant intermingling of proprietary code into the code base", or its claim that "the innovation focus has shifted from furthering the open source distribution to making the proprietary distribution popular." The only response offered is a vague, generic "we believe in open source" - again with no concrete action backing up those empty words.
If this is the best and only response Elastic is able to offer, I fully expect Amazon to keep the high ground in this controversy.
actions? I think this is speaking for itself: https://www.elastic.co/downloads/elasticsearch-oss
What has Amazon contributed to yet? Repackage other open source projects that was already existing? I don't call it a contribution.
"Repackage other open source projects that was already existing?"
Most people don't realize this yet. They are taking other, existing projects which anyone could install themselves for a long time now and framing it like they built these things for their new fork. So much of their "distribution" is just a collection of other open source projects.
In the case of the security functionality, that is backed by a company named Search Guard who have an Enterprise Version in addition to the Community Version included in the AWS fork. They forked another project, renamed it and now call it their own.
Isn't "a collection of other open source projects" pretty much exactly what a "distribution" usually implies? What you get is the curation/selection of what's included and some reassurance that all the pieces really do work together and other people are using them that way.
> actions? I think this is speaking for itself: [link]
The link is to a packaged, binary distribution of the 'OSS only' features, the assertion was regarding 'significant intermingling of proprietary code into the code base'. The two are not the same.
Also, from the release notes linked to from that page, Amazon's assertion that 'neither release notes nor documentation make it clear what is open source and what is proprietary' does appear to be valid.
Remember that AWS pays the Open Source Initiative, and that their members often post on Open Source related topics here without disclosing their conflict of interest.
Remember that AWS takes from developers, but doesn't give back anywhere near what they have made from the communities work.
Rather than supporting Elastic... they decide to make a hostile fork. AWS are a bunch of jerks.
What is the "Open Source Institute"? If you mean the Open Source Initiative, then yes, I have posted on open-source related topics here, yes, I am a member of the OSI (I joined... two weeks ago? mostly so I could vote for OSI board members who are committed to opposing corporate influence over the open source definition), and no, I do not have a conflict of interest. I paid the OSI $40 to join. I receive no money from the OSI; I certainly receive no money that Amazon donated to the OSI. I am not OSI staff, and I think the OSI only has one paid staff member. Saying there's a conflict of interest is like saying that individual ACLU members have a conflict of interest when speaking about a company that donates to the ACLU, or that Amazon Prime members have a conflict of interest when speaking about Amazon.
(I am, as it happens, rather opposed to Amazon as a company. I canceled my Prime membership; I try to shop anywhere else if possible; I celebrated the loss of NYC HQ2. But in this case, I think they're in the right.)
An Open Source advocacy group with cloud providers on their board, and cloud providers who pay them hundreds of thousands per year. Their mission is to be evangelists for their definition of Open Source. The one that benefits cloud providers.
Of course a lobby group like this should be transparent. But I don't expect a group like that with closed mailing lists to actually be Open.
I'm still not sure what group you're talking about. If you're talking about the Open Source Initiative, the description you're giving doesn't quite make sense: the board members of the Open Source Initiative are individuals, not organizations. Some of the individuals happen to be employed by Google, Microsoft, Red Hat, etc., but they don't seem to work in the cloud organizations and they're not representing their companies, anyway.
The total budget of the OSI is in the couple hundred thousand range, i.e., comparable to a single full-time software engineer. So the fact that a few companies are each giving five-digit donations that add up to a six-digit budget doesn't seem like it's a huge priority for any of these companies. Take a look at the sponsorship levels on https://opensource.org/sponsors .
In any case, I agree that the Open Source Initiative should be transparent, and my individual membership in the OSI is not an endorsement of everything the OSI does any more than having a PyCon ticket is endorsement of everything the Python Software Foundation or their corporate sponsors do. (One specific thing they haven't done is post form 990s more recently than 2016, and I do intend to ask them to post these. But if you look at the 2016 one https://opensource.org/files/2016990EZ.pdf you can see that these aren't vast sums of money, and Craigslist donated more money to the OSI than any cloud provider.)
Their definition is a branding of the Debian guidelines from 1998, not a ploy on behalf of AWS years later. What other definition should we be considering? I actually can't name any.
Edit: well, there's Microsoft's "Shared Source", but that's read-only if they like you, not particularly useful.
For the amount they make off community work, they contribute a tiny amount.
Right near the top of their list (in your link) they brag about contributing to Linux. Let's see how well their brag stacks up? Looking at the contribution reports on Linux, are AWS in the top 10?
Most of the companies on that list are hardware developers adding drivers for their own hardware, which generally benefits nobody other than their own customers. A five-line fix to scheduler performance is way more valuable to the community at large than ten thousand lines of code to support a 25GbE network card. (The 25GbE driver is hugely valuable to Amazon, though! And, depending on how you look at it, may well have been sponsored by Amazon by being a big customer of Linux in the datacenter.)
Amazon is rewriting the proprietary enterprise features of ES from scratch and giving them away for free.
Also, their MongoDB "distro" is just "Amazon DocumentDB (with MongoDB compatibility)". That's not going to be open source, because it took a lot of work to basically rewrite most of Mongo.
> When companies came to us, seeing our success, and asked for special working relationship in order to collaborate on code, demanding preferential treatment that would place them above our users, we told them no.
I think this may have been the crux of the friction between the companies. Elastic sure looks to have collaborated with GCP [0][1], if not other enterprises.
> Our answer has always been a constant: send a pull request, like everybody else does. The quality will speak for itself.
I look around and spot at least three substantial bug/fixes from AWS employees (?) [2][3][4], so I don't see any lack of effort from AWS? AWS engs were sending pull requests, just like the others. That said, any sort of heavy handedness from maintainers of elasticsearch might also play a role in forcing enterprises to take control of software they depend upon so much where legally allowed. We have seen this happen with Node.js/IO.js and MariaDB/MySQL before.
> The FUD mostly comes from large(r) companies that fear what such a movement can cause.
The irony of this statement is not lost on me. Elastic is that larger company (one that crucially controls a majority of the ecosystem around elasticsearch: kibana, logstash, beats, xpack to name a few), and AWS is an upstart here with their investment in open-distro, presumably to fight FUD.
> We all sometimes need to self reflect on what and why we did that made us successful, to make sure we stay true to our course. [..] to others out there, that face so many reasons to be distracted, keep your focus [..] And last, to express our shared commitment to continue to build great products [..] It is our true north.
I think one right way for Elastic to show commitment to opensource would be to entrust the projects to a community agreed governance model, like others before them have done [5][6][7][8].
Also, what's interesting is that its not just Amazon here, but Expedia and Netflix as well putting their weight behind open-distro [9].
Like someone pointed out, this response from Elastic does seem to be emotionally charged [10].
"Elastic sure looks to have collaborated with GCP"
AFAICT that partnership just resulted in Elastic Cloud running on GCP in addition to AWS. If I had to guess, I'd say that Elastic didn't think it was worth their time to support GCP, but Google provided some resources to encourage them to add it as an option. That way Google gets to tell prospective GCP customers that they have an option for hosted Elasticsearch without having to build it themselves.
I've never really thought that elastic search is all that great, to be honest. I use it every day at work for an application that does a ton of aggregations.
That said, I can't shake the feeling that we could get the same level of performance with RethinkDb with the benefit of a better API and less overhead.
Assuming that the use case does not require large quantities of full-text search, is there really any reason to choose ElasticSearch over RethinkDb? ElasticSearch does have a larger ecosystem, but with how good RethinkDb's API is, I'm not entirely sure any 3rd party packages would actually be necessary.
Add to that RethinkDb's ability to do joins and changefeeds and this decision becomes a no-brainer for me.
Sure, that's not a ton of development, but at least it's actually, really, open source.
In addition, the software itself doesn't really need much improvement. It does what it's supposed to do very well and the client libraries are fantastic.
I think a lot of open source people forget that large companies worry a lot about licenses and clarity around that. Without that clarity and control they get sued left and right and on this very forum we end up discussing how evil this large company has been.
Elastic search is a great product and there is no doubt about it. However I understand Amazon's viewpoint that it should be clear which part of ES are open source and which are not.
A the end of the day Amazon or Google can easily buuld a clone of ES from ground up.
I'm sure Shay et al have good intentions, but to have any degree of usability with elastic.co you need to buy x-packs which goes against the grain a little. I'd still like to have auth (not just basic-http) on personal projects.
Searchguard is a plugin for ES which gives you TLS + auth. They offer a paid version with more features (ActiveDirectory integration, HIPAA-compliant auditing, etc - actual enterprise features), but if you just need auth/ACLs/transport TLS, it's a great product.
“Any degree of usability” Really? I have used ES for years and the majority of folks have gotten by with simple nginx proxies... most don’t need document level security
130 comments
[ 2.6 ms ] story [ 206 ms ] threadIf I were Elastic I’d be quaking in my boots right now - their entire business model is under assault. Hell, the whole open core model is being called into question. Personally I’m glad to see Amazon’s distro.
Anyone have a different take?
Edit: The thought was about big companies with their own agenda behind open source initiatives. Sorry about confusion.
I hope that people will be able to use "vanilla" ES too on AWS.
MariaDB is a community fork away from a controlling company. Libreoffice is run by its own foundation, again after forking away from a company-controlled version.
> I hope that people will be able to use "vanilla" ES too on AWS.
I mean, worst case just run it on EC2/ECS, which lets you run anything you want. They're not going to break that.
I guess a big driver for me is a lot of the proprietary features (example: encryption in transit, RBAC, audit logging) seem absolutely basic to me. Not basic as in easy to implement, but basic as in as an operator I’ll need them for every type of logging/monitoring infra I’d ever spin up.
Worth noting I consider Elastic’s managed offering to be superior to AWS’ in a lot of ways. But personally I’ve always chosen to build my own setup off EC2
MySQL is still big but MariaDB is big too and definitely favored as a default by most distributions[2].
[1] https://github.com/apache/openoffice/pulse vs https://github.com/LibreOffice/core/pulse
[2] https://mariadb.com/kb/en/library/distributions-which-includ...
I think if anything is called into question, it's the development of infrastructure software in a startup. Does it make economic sense anymore? As in, is it possible to profit directly off it at this point? Right now it obviously is, but it seems to be trending away from that.
The API of a closed source, propriety and patented software can't just be copied. See Oracle vs Google.
That’ll (maybe) be a more compelling argument for one side or the other of this argument when it is finally done.
Especially if the Supreme Court rules, and it's not just the Federal Circuit’s interpretation of what the Ninth Circuit would say, which is not binding on lower courts (or other circuit courts, including the actual Ninth Circuit, in other cases.
Aren't APIs under copyright now, thanks to Oracle vs Google?
Related discussions:
https://news.ycombinator.com/item?id=19359602
https://news.ycombinator.com/item?id=19363961
> At AWS, we believe that maintainers of an open source project have a responsibility to ensure that the primary open source distribution remains open and free of proprietary code so that the community can build on the project freely, and the distribution does not advantage any one company over another. This was part of the promise the maintainer made when they gained developers’ trust to adopt the software.
I have enormous respect for Adrian's work on cloud systems but the quote above seems very self-serving for Amazon.
Open source is bound by licenses that define the terms of open source code use. There's nothing whatsoever in the licenses that binds the project developers to continue to work on the project, to maintain similar license terms in later releases, or to take (or not take) any other action in their lives. To argue otherwise seems contrary to the notions of freedom that are the foundation of open source software.
[1] https://aws.amazon.com/blogs/opensource/keeping-open-source-...
Edit: typo corrected
Those maintainers owe Amazon nothing. The have no responsibility to Amazon. None.
Same as they owe me nothing.
By definition (if you use any of the more-that-two-or-so-year-old definitions of "open source"), the project is defined by "remaining free and open", in that you can always use the code and license you've been using - but there's no way anybody can make it a "responsibility" of a maintainer to to anything in the future. They could die, they could get bored, they could get an all-encompassing job, they could get a family, they could get venture capital and license any future work with a non-open source license (as Redis and Elastic have done, at least in some people's view).
The "the promise the maintainer made" was "here's some code, under GPL or aGPL, or Apache Licence, or Artistic license of BSD three clause, or whatever the promise at the time was. Take it or leave it. No future responsibility assumed or enforced. They might even have a track record of releasing more stuff under the same license/promise. But they can stop or change things whenever they want.
If Amazon wants the maintainers to do anything else (including ever committing another line of code or fixing a bug), they can wait for them to do (or not) it at their discretion, or they can pay the maintainer (or someone else) to do it (and then, depending on the license, potentially be required to release that code under the same license.
Sorry for the rant, but that line caught my eye too, and I got immediately pissed off at Amazon's sense of entitlement there. No open source maintainer owes you shit. They've shared what they've shared, and you can take it or leave it, but you taking it doesn't place any obligation or responsibility on them.
(deep breath)
But most maintainers (including Elastic) don't want that. They proudly declare that their developers and users are a community, and they're steering the community to produce a great and open project on an ongoing basis. The maintainers certainly have a responsibility to follow through on such promises - or at least, if Amazon is correct and they aren't following through, they have no right to complain about a hostile fork.
I help out by sweeping Amazon's yard everyday, for free. "Thanks!" they say, "Any time!" I reply. But years later, I decide to stop. Amazon cries out in anguish "But you have a Responsibility!", and threatens me that if I don't clean their yard... well, they'll do it themselves.
Free maintained software suits Amazon, and some organizations / foundations provide it. Seems to work as consulting services advertising (BTW which incentivizes hard-to-use software, lots of complicated features, jumbled together... ever seen a large foundation's open source project like that?)
Some businesses fund these foundations directly, but minimally, and there's a certain... reluctance. Perhaps an unmet need there...?
It is true that Amazon here is just stating their own interest and are not well placed to give lessons of open source implication. Elastic has paid many developers to build open source code.
However, Shay Bannon on his side, is too much using the word "open" for a company that is not running and from my POV cannot run a real open governance. Elastic is now a listed company and was VC backed. Presenting the integration of xpack in the core repositories as an "opening" when it's a move backwards in terms of open source governance, is not fully honest.
There is a fundamental issue with Open Source and Investor backed companies. The main goal is not open governance, but the main goal is the company.
Elastic is not clear about this. Actually not being clear means there is no open governance.
Shay Bannon won't go there, because I believe as the CEO of a listed company he can't, unless it's in the interest of Elastic's investors, which means the board should decide.
Elastic does not have to have open governance and can be a great OSS contributor nevertheless. And to the opposite, it is very possible that without very healthy Elastic.co, that there would be less OSS code.
However it raises questions about the future. What will be the next step ?
What I would prefer is to have Shay Bannon be fully honest about the fact that Elastic.co needs to be healthy for the project to be healthy unless other companies are heavily investing in the community and start contributing code. And that Elastic.co being healthy means to have a mixed model. And finally I would like to hear that Elastic.co cannot promise that this won't change because Elastic.co might have to change it if the investors decide it but that as long as he is CEO he will do his best to maintain the current mixed model as long as it is financially successful.
It would be more honest that saying "doubling down on open" when you are tainting open repositories with non open source code. There was no technical reasons that the xpack repositories could not be separate.
Also, there is a way to not give an advantage to any company in an open governance way while helping companies be healthy. You can decide to have an App store in which paying features are proposed only from core contributors that contribute more than a certain level. Business could be added to the goals of the open community.
Clarification: Not push X-Pack itself, but features comparable. Like they suggested people do in the post.
The way this is written, it seems like there was a difference of opinion in terms of the product architecture, and not a purely commercial decision by Amazon. I would think it would make commercial sense for the team at Elastic to align with Amazon than the other way round. On the other hand Amazon claims that Elastic was moving away towards proprietary code [1].
Overall, being in open source for a decade, I am against this trend in dual licensing (which I fear is pushed by Venture Capitalists). People forget that the fact that you have freedom and zero friction, is what makes open source awesome. There are other ways to make money, like building a kick-ass consulting practice around the open source offering.
[1] https://opendistro.github.io/for-elasticsearch/faq.html
Someone else with a business model that benefits from providing these features for free and making them open source to reduce their development/maintenance burden - cloud providers like Amazon - are going to do just that. And they're well within their right to do so.
Impeding security updates seems rather unethical. Unless you mean backports to old/stable versions, which is still iffy in my mind but does at least maintain a way to stay secure for everyone.
For example enterprises may want quicker access to well tested bug or security fixes - they can get that first and then the patches appear on GitHub for open source users.
Most software isn't like that. My current preference is going back to highly-permissive, shared-source licenses by public-benefit companies whose charters and contracts don't let them screw people in at least all the ways we know about. Perpetual, free licenses w/ no or optional support for non-commercial or small business. Cost goes up with scale from there maybe with enterprise features sold separately. Anything not a competitive advantage, esp building blocks or platform stuff, might get open-sourced. Anyone licensed can modify the software and distribute that to other licensees. Might make exception for features or code in new version so they can't skip payment with backport. Maintains most of benefits of OSS with less risk of proprietary or hybrid licensing.
Charging for security fixes could be a simple way of motivating folks to pay for improvements to the software. A preferred model that's harder to build is using methods that make nearly vulnerability-free software with a warranty against specific types of defects or at specific, defect rates. Cleanroom shops and Praxis Correct-by-Construction did that with an upfront premium applied with defect rate in warranty fixed at supplier's cost. So, still thinking on specifics of all of that or potential integrations.
I also wrote up my own thoughts, but it's obviously not as relevant as this. I'm a little more blunt about what AWS is doing. I even throw in a comparison to Microsoft abusing their monopoly power in the 90's by distributing Explorer for free to kill off Netscape: https://www.influxdata.com/blog/aws-intends-for-their-new-pr...
Longer term, the real question is how Elastic will respond to this commercially. Will they create more closed source? Go higher up the stack into specific applications? It's going to be an interesting and tumultuous couple of years in open source infrastructure software.
EDIT: you've remarked elsewhere on the thread about commercial and economic realities, so I want to take the opportunity to qualify my statement: they're free to license and monetize their code however they want. It's what I see as deceptive intent that I'm having a (very strong) reaction to.
[1] https://news.ycombinator.com/item?id=19369774
https://www.theregister.co.uk/2013/03/08/amazon_copies_partn...
https://www.bloomberg.com/news/articles/2018-09-27/amazon-s-...
How? Redis is trying, and got backlash. Mongo is trying, and got backlash. Confluent is trying, and got backlash. We've rather got to pick our poison here: do we want companies to open their code and have it used for managed services/cloned, or do we want them to close it and keep making cool tech?
I recall reading about the model Google used to use for this sort of thing: open your source code about three to five years after you stop using it. This gives something to the community, while still allowing you to maintain a competitive advantage, run a managed service, etc. Particularly for those who run a SAAS model, it lets them gain widespread adoption before a competitor can spin up an instance of their stuff and undercut them slightly.
Oh, and for those saying "use the Red Hat model", the Red Hat model can't work for everyone. Red Hat had a new idea and essentially no competition for enterprise linux at the time, and so didn't have to deal with the competition. They got a chance to build a customer base before Oracle knocked their product off. And even then, Oracle did completely rip off their work. That's the risk you run with open-sourcing your codebase, and not every company is built to work that way; not our job to tell them they should redo their business model to allow it.
To be specific and on-topic, Open Distro for Elasticsearch is being given back to the community under an Apache 2.0 license.
If a creator thinks that it is wrong to take without giving, there are lots of other licenses for different philosophies.
If the claims are true, and done intentionally, it's a really bad move on part of Elastic. The ambiguity around licensing hurts all their users and not just Amazon.
[1] "Unfortunately, since June 2018, we have witnessed significant intermingling of proprietary code into the code base. While an Apache 2.0 licensed download is still available, there is an extreme lack of clarity as to what customers who care about open source are getting and what they can depend on. For example, neither release notes nor documentation make it clear what is open source and what is proprietary."
[2] "Individual code commits also increasingly contain both open source and proprietary code, making it very difficult for developers who want to only work on open source to contribute and participate"
We quickly built a SQL parser for Elastic, an alerting engine, some other bits. These are all bits that just went poof yesterday when Amazon released opendistro - and I couldn't be happier. Sure we'll have to slice out a bunch of code, but at the end of the day it didn't have value to our core business.
I just wish Elastic as a company was easier to deal with. We'd be months ahead if not for them - and ultimately just worked around them. This is in spite of MY using Elastic for ~9 years now.
If I read this correctly, Amazon tried to get a special deal from Elastic basically in order to produce a competing service. Elastic declined to cooperate and now Amazon is exercising their rights under the Apache license. So, far so good.
It's the provocative website that makes it seem like there's some huge conspiracy by Elastic that provoked Shay Banon's response. And not the fact that Amazon are doing what they are doing (i.e. redistribute the OSS parts of Elasticsearch), Elastic seems to have little problems with this because their license allows this and as Shay points out others have done this as well.
Amazon's complaints boil down to two things:
1) Elastic distributes a version of their software that includes non OSS components. You don't have to use these but you can. Actually Elastic offers rpms containing only the apache licensed components on their download page: https://www.elastic.co/guide/en/elasticsearch/reference/curr.... So this is maybe a bit less true than Amazon is portraying here. This would be what you'd install on a server if you are self hosting. Likewise there are OSS docker images from Elastic. They also have oss docker images. Elastic does this because up-selling this stuff is how they make money. Amazon's beef seems to be that it would be awfully nice for them if they could offer an alternate distribution more prominently without all this stuff so that they give that to their own customers.
2) Their repository contains some well documented directories that contain non Apache licensed source code for these components. As far as I know this is pretty well documented and if you are modifying files in a directory, presumably you know what you are doing. This is because they want to develop most of this stuff in the open and be able to accept pull requests against both their OSS and non OSS stuff. This seems pragmatic and preferable to producing binary distributions with some components from private repositories.
For reference, here's the text of the license file (https://github.com/elastic/elasticsearch/blob/master/LICENSE...):
"Source code in this repository is variously licensed under the Apache License Version 2.0, an Apache compatible license, or the Elastic License. Outside of the "x-pack" folder, source code in a given file is licensed under the Apache License Version 2.0, unless otherwise noted at the beginning of the file or a LICENSE file present in the directory subtree declares a separate license. Within the "x-pack" folder, source code in a given file is licensed under the Elastic License, unless otherwise noted at the beginning of the file or a LICENSE file present in the directory subtree declares a separate license.
The build produces two sets of binaries - one set that falls under the Elastic License and another set that falls under Apache License Version 2.0. The binaries that contain `-oss` in the artifact name are licensed under the Apache License Version 2.0."
So, you might not agree with this arrangement but you can get or build pure OSS quite easily and many people do. And it is hard to argue against Apache 2.x here. The license pretty much makes it obvious how stuff is licensed. This is unsurprising. Any confusion here would be a legal bug; this stuff needs to be unambiguous and it seems to be.
BTW. I think it is great that Amazon is providing their own distribution for their customers. It would be awfully nice though if they'd update it to the current version and not ship already fixed bugs. They have a history of doing this with Elasticsearch; probably because they are not on speaki...
One aspect Elastic is also not directly attacked on and therefore not addressing is the choice of features that are not free and the price point. Elastic's business model is quite smart. You can easily develop on top of elastic, however once you have done it and want to go to production some key features are missing and they are not cheap. Particularly securing your cluster data is one of the features in xpack which is problematic not to have. I'm pretty sure you can search for open ports 9200 over the internet or company intranets and find unwanted unprotected data. And it's not cheap as you need to go for Gold subscription if I'm correct. I would be surprised nobody has proposed a pull request with competing code.
Elastic has and is investing a lot as open source and they do need a good business model that allows investment in the core open source code. The thing is I'm not sure it's missing here. Elastic.co is quite healthy. It's hard to know if it would be without this business model, but anyway there is little chance they would do differently given Elastic.co is VC backed. In the end the real fight is between the big money players, not about whether Elastic the software is sustainable. The reality is that given the community, it would very probably be even without the "xpack" package. Now it's not sure Elastic.co would be able to aggressively invest in both oss and non oss as much and therefore compete with Amazon.
OSS is tough. When you go for it, you should go in it for OSS, not for the company, because keeping the company at the level of the OSS success is hard both for small and big OSS projects.
Ludovic Dubost (XWiki)
But the main problem is that unsecured clusters by default have caused a lot of reputation loss to the brand. When every few months news hit about yet another unsecured Elasticsearch cluster that leaked huge amounts of data, it is getting harder and harder to explain to the less informed how that is the fault of those people who did not even bother running a reverse proxy, not the fault of Elasticsearch itself.
This is my only complaint with what Redis/Elastic/Mongo et al are doing lately.
I 100% support their right to change their licensing terms. Applaud them even - I'm fascinated to see if their attempts at new business models will work out medium/long term.
Doing that and still claiming to be "open source" is wrong. It's co-opting a term that while not solidly defined, is universally understood to mean something different to what they're offering. They're lying to their users and customers.
Elastic are muddying the waters in a grey area - where you can build Apache2 licensed or Elastic's new non-FOSS licensed versions from the same download. I reckon they need to cole clean and make if very clear that they're "partially open source".
Duh, I'm glad about the direction Amazon is going.
When Amazon wrote their letter, the [HN response](https://news.ycombinator.com/item?id=19363961) seemed to tip towards not liking what Amazon had to say or was doing.
When ES writes effectively a response, the initial comments don't like ES's response or what they're doing either.
It may be that the collective HN vibe is critical no matter what. But also maybe "we" in aggregate legitimately don't like either option given.
I know open source devs need to make money somehow. I also wish there was a way to do it that was open source. I'm not sure there is. I'm not sure what it means for the continued viability of open source.
In the original period of open source, people generally worked on open source on the clock at their existing jobs, a building tools they needed for that job, a job not dedicated to that particular product but to getting something done that product helped with. People were getting paid to write open source, but it wasn't by "productizing" it. The results were still shared freely.
(this is how/where apache httpd, for one, came from; people with jobs who needed an http server writing one on the clock, even though their job was not "writing an http server." They needed an http server to accomplish what was needed for their employers, but their employers had no desire or capacity to be in the business of selling an http server. Why not collaborate on one we can all use?)
But those days are gone, software has just gotten _so much more complex_ and time-consuming to develop. The open source software we have and need can't be developed in someone's "spare" (even on-the-clock spare) time. Some open source is still written by people working for a large employer, who pay them to be dedicated to that product. But we see the perils of that when an employer decides "wait, why aren't we making money selling this? Why are we contributing our employee's time to it?" (thanks Oracle).
So that leaves people trying to productize it one way or another. It is not clear to me how to have a healthy open source ecosystem in that market environment. I don't think "proprietary layer on top" is actually good for open source, especially if it's gonna lead (naturally) to rejecting contributions from outside to add those same features as open source (why would we want to spend time working with a contributor to make the feature solid, when we _already wrote it_, and when accepting the contribution would _harm our business model_ because that feature is what we want to charge for?), and then resistance to the "right to fork" which I consider the fundamental basis of open source freedom.
Build a database, not a database engine.
> Some open source is still written by people working for a large employer, who pay them to be dedicated to that product.
Take e.g. tensorflow by google. The software itself, is per se worthless, so they can release it as OSS as some kind of calling card or recruiting tool. The crown jewels they are guarding closely and are never letting any outsider access directly is their data.
No, I think it's a little more complicated than that. I don't think this is a good comparison.
Tensorflow is only "worthless per se" because it was completely open sourced. It's quite a sophisticated library and a substantial product could have been built using it if it was kept proprietary.
However, Google got to the space first and Google is fundamentally a software service company. The reason they open sourced Tensorflow is because they have data and recognized the data would be more valuable; hence, they decided to commoditize the complement. By open sourcing Tensorflow they made their data even more valuable (likewise for their services which can be used to build and manage a data pipeline).
I believe you're conflating "open core" with with "open source". I don't believe that open source needs a profit motive to exist and flourish. While this may be a blow to the open core FOSS business model, there are a number of other models that continue to exist that do not involve proprietary enterprise extensions.
While these models may not be attractive to venture capitalists, they can certainly provide a way for developers to make money.
The question is, who is going to write open source projects at a complexity/polish level similar to ElasticSearch. And how are they going to pay their rent?
People working on it need to pay the rent somehow.
I am suggesting that the way people paid the rent while working on open source historically in the "first phase" of internet open source (doing it on the clock at jobs for which the open source was useful for their job role, but producing the open source was not their job) -- is no longer working to get software written at the complexity level of 2019 software, as software gets much more complex requiring more people coordinating with themselves and spending more hours.
I agree, at least for the sake of argument, that "open core" is distasteful and problematic. I believe it was a response to the question of "how do we write open source and still earn a living." It may not be a satisfactory answer. I'm not sure what other answers exist. If there aren't better answers, open source will shrink, including by "open core" and such.
I believe that "open core" was not a response to the question of "how do we write open source and still earn a living?" but rather "how do we leverage open source to make a billion dollar company?". I think the distinction, however minor, is important.
I am not sure there are obvious models/examples that are working. I think there's a reason that "open core" is an increasingly popular thing to try to do. I'm not sure if I can think of many earning a living while producing high-complexity high-quality open source software _except_ "open core".
(And I still think "open core" is problematic for the continued viability of actual open source).
I find in general it's easier to get free labor on something the newer it is. Both because it's more exciting, and because the more 'mature' something gets the _more work it is_ for _less gain_, there's more time dealing with backwards compat and filing off rough edges and keeping the existing code just working, which is all less interesting to do for free. Plus, the more mature something is, the _more work_ it _takes_ to keep it going.
Or, there are Fair Source License / Commons Clause / Business Source License /... that allow developers to capture some of the value they generate, but this of course means they are not open/free/libre because they limit some of the freedom to accomplish that.
Pick your poison. :)
Maybe. But I also think that kind of funding for open source you talk about is not there for high-complexity projects in the present time.
It's not that I don't like that traditional model. I am legitimately worried it is no longer feasible.
I have to strongly disagree here. Stuff like Kafka from LinkedIn, Envoy from Lyft, Prometheus from Soundcloud are just the tip of the iceberg of people writing tools because they need them, not because they are in the business of making infrastructure software.
I believe that selling software, be it foss, open core, or fully proprietary is not good business model, neither from business nor from ethics point of view. Sell your expertise or use the software yourself to generate value, but trying to sell it is just futile.
They're free to license their software however they want; I don't use it so I really don't care. But it really rankles when someone extols the virtues of open source and waxes rhetoric about how open they are, and yet the software they offer is proprietary by default. They "double down on open", but the open source distribution is opt-in.
I cannot help but feel that the articles are deliberately constructed to mislead. That's a serious accusation and I wish I could be more charitable in how I interpret them but the terminology is so consistent that it cannot be an accident -- gratis-but-proprietary is always "open", libre is never just "open" but always "open source".
Technically correct isn't enough. No-one's explicitly staked "open" as being the same as "open source". But in the context of this conversation "open" is an obvious contraction of "open source" and to use the obvious contraction to mean something diametrically opposed is not reasonable or ok; especially when "source available" is widely understood. The same thing was attempted with the Commons License.
The intent behind words is important. Persuasive oratory is fine but using those tactics to mislead does not engender trust, especially when the beneficiary is the one speaking and not the audience.
OP uses the word free twice, in both cases meaning $-free.
"We [...] open sourced most of it, and made all of it free."
From an enterprise licensing perspective, 'open source' as a subset of 'free' just means I shall have to spend more time with lawyers and accountants.
This post from Elastic consists of vague claims of 'FUD' and of their commercial code being 'bluntly copied', without any further clarifying details or responsive action to back them up. (E.g., if you have real evidence of Amazon copying proprietary code, which I doubt, then announce the copyright infringement lawsuit you have just filed, don't weakly allude to your grievances in a blog post.)
Conspicuously, the post did not contain any defense or even mention of Amazon's key accusations of "significant intermingling of proprietary code into the code base", or its claim that "the innovation focus has shifted from furthering the open source distribution to making the proprietary distribution popular." The only response offered is a vague, generic "we believe in open source" - again with no concrete action backing up those empty words.
If this is the best and only response Elastic is able to offer, I fully expect Amazon to keep the high ground in this controversy.
[1] https://aws.amazon.com/blogs/opensource/keeping-open-source-...
Most people don't realize this yet. They are taking other, existing projects which anyone could install themselves for a long time now and framing it like they built these things for their new fork. So much of their "distribution" is just a collection of other open source projects.
In the case of the security functionality, that is backed by a company named Search Guard who have an Enterprise Version in addition to the Community Version included in the AWS fork. They forked another project, renamed it and now call it their own.
https://github.com/floragunncom/search-guard-enterprise-modu...
https://github.com/opendistro-for-elasticsearch/security-adv...
They just forked the full code base and tried to remove any reference of Search Guard.
I am not sure what really happened here but this smells fishy.
The link is to a packaged, binary distribution of the 'OSS only' features, the assertion was regarding 'significant intermingling of proprietary code into the code base'. The two are not the same.
Also, from the release notes linked to from that page, Amazon's assertion that 'neither release notes nor documentation make it clear what is open source and what is proprietary' does appear to be valid.
Which is a wrong assertion, you can clearly see that it is an under Licence feature by the X-pack mention: https://www.elastic.co/guide/en/elasticsearch/reference/curr... above the name of the feature.
Remember that AWS takes from developers, but doesn't give back anywhere near what they have made from the communities work.
Rather than supporting Elastic... they decide to make a hostile fork. AWS are a bunch of jerks.
(I am, as it happens, rather opposed to Amazon as a company. I canceled my Prime membership; I try to shop anywhere else if possible; I celebrated the loss of NYC HQ2. But in this case, I think they're in the right.)
Of course a lobby group like this should be transparent. But I don't expect a group like that with closed mailing lists to actually be Open.
The total budget of the OSI is in the couple hundred thousand range, i.e., comparable to a single full-time software engineer. So the fact that a few companies are each giving five-digit donations that add up to a six-digit budget doesn't seem like it's a huge priority for any of these companies. Take a look at the sponsorship levels on https://opensource.org/sponsors .
In any case, I agree that the Open Source Initiative should be transparent, and my individual membership in the OSI is not an endorsement of everything the OSI does any more than having a PyCon ticket is endorsement of everything the Python Software Foundation or their corporate sponsors do. (One specific thing they haven't done is post form 990s more recently than 2016, and I do intend to ask them to post these. But if you look at the 2016 one https://opensource.org/files/2016990EZ.pdf you can see that these aren't vast sums of money, and Craigslist donated more money to the OSI than any cloud provider.)
Edit: well, there's Microsoft's "Shared Source", but that's read-only if they like you, not particularly useful.
For a "bunch of jerks" they sure do a lot of open source stuff.
https://aws.amazon.com/opensource/
Right near the top of their list (in your link) they brag about contributing to Linux. Let's see how well their brag stacks up? Looking at the contribution reports on Linux, are AWS in the top 10?
AWS are not even on the list. https://twitter.com/ibrahimatlinux/status/768631239683158016...
Just don't know where Amazon is going with this approach of re-packaging existing opensource tools and calling it their opensource contribution.
Also, their MongoDB "distro" is just "Amazon DocumentDB (with MongoDB compatibility)". That's not going to be open source, because it took a lot of work to basically rewrite most of Mongo.
Thank you.
I think this may have been the crux of the friction between the companies. Elastic sure looks to have collaborated with GCP [0][1], if not other enterprises.
> Our answer has always been a constant: send a pull request, like everybody else does. The quality will speak for itself.
I look around and spot at least three substantial bug/fixes from AWS employees (?) [2][3][4], so I don't see any lack of effort from AWS? AWS engs were sending pull requests, just like the others. That said, any sort of heavy handedness from maintainers of elasticsearch might also play a role in forcing enterprises to take control of software they depend upon so much where legally allowed. We have seen this happen with Node.js/IO.js and MariaDB/MySQL before.
> The FUD mostly comes from large(r) companies that fear what such a movement can cause.
The irony of this statement is not lost on me. Elastic is that larger company (one that crucially controls a majority of the ecosystem around elasticsearch: kibana, logstash, beats, xpack to name a few), and AWS is an upstart here with their investment in open-distro, presumably to fight FUD.
> We all sometimes need to self reflect on what and why we did that made us successful, to make sure we stay true to our course. [..] to others out there, that face so many reasons to be distracted, keep your focus [..] And last, to express our shared commitment to continue to build great products [..] It is our true north.
I think one right way for Elastic to show commitment to opensource would be to entrust the projects to a community agreed governance model, like others before them have done [5][6][7][8].
Also, what's interesting is that its not just Amazon here, but Expedia and Netflix as well putting their weight behind open-distro [9].
Like someone pointed out, this response from Elastic does seem to be emotionally charged [10].
---
[0] https://www.elastic.co/about/partners/google-cloud-platform
[1] blog.google "With the Elastic partnership in place, we’re kicking off our joint engineering work and will start rolling out managed Elasticsearch on GCP later this year." https://www.blog.google/products/google-cloud/google-cloud-p...
[2] https://github.com/elastic/elasticsearch/pull/27628/files
[3] https://github.com/elastic/elasticsearch/issues/31479
[4] https://github.com/elastic/elasticsearch/issues/29531
[5] GridGrain and https://ignite.apache.org
[6] Various networking companies and http://opennetworking.org
[7] Cloudera and https://impala.apache.org
[8] LinkedIn/Confluent and https://kafka.apache.org/
[9] sciurus ↗ "Elastic sure looks to have collaborated with GCP"
AFAICT that partnership just resulted in Elastic Cloud running on GCP in addition to AWS. If I had to guess, I'd say that Elastic didn't think it was worth their time to support GCP, but Google provided some resources to encourage them to add it as an option. That way Google gets to tell prospective GCP customers that they have an option for hosted Elasticsearch without having to build it themselves.
That said, I can't shake the feeling that we could get the same level of performance with RethinkDb with the benefit of a better API and less overhead.
Assuming that the use case does not require large quantities of full-text search, is there really any reason to choose ElasticSearch over RethinkDb? ElasticSearch does have a larger ecosystem, but with how good RethinkDb's API is, I'm not entirely sure any 3rd party packages would actually be necessary.
Add to that RethinkDb's ability to do joins and changefeeds and this decision becomes a no-brainer for me.
https://www.rethinkdb.com/blog/2.3.6-release/
The project may be transitioning to new governance.
I can't really say that RethinkDb NEEDS a new release.It's feature complete in my eyes. All it really needs at this point is maintenance.
https://rethinkdb.com/blog/rethinkdb-shutdown/
https://github.com/rethinkdb/rethinkdb/commits/next
https://github.com/rethinkdb/rethinkdb-python/commits/master https://github.com/rethinkdb/rethinkdb-rs/commits/next
Sure, that's not a ton of development, but at least it's actually, really, open source.
In addition, the software itself doesn't really need much improvement. It does what it's supposed to do very well and the client libraries are fantastic.
Elastic search is a great product and there is no doubt about it. However I understand Amazon's viewpoint that it should be clear which part of ES are open source and which are not.
A the end of the day Amazon or Google can easily buuld a clone of ES from ground up.