68 comments

[ 3.0 ms ] story [ 141 ms ] thread
Directionally good, though I'm sure there's a decision tree that's mostly driven by how it benefits AWS. Something like "is committing this back going to save us time maintaining it separately without giving away IP we care about?".
I would say that's true for any company that contributes to open source projects.
Yes, though there's some things that are somewhat unique where Amazon is competing with open source products. For example, they have to offer MKS, but would rather you use Kinesis, EventBridge, etc. So perhaps less incentive to make Kafka better for everyone.
It's probably not so much IP that they care about as IP that gives them some competitive advantage by keeping it secret.

You can probably overstate the degree to which Amazon didn't contribute to and interact with open source communities 5+ years ago (and probably also overstate the degree to which there's been a complete sea change) but, as you say I think it's fair to say there's been a directional shift with more substantive activity in addition to what I'd mostly describe as talking the talk previously.

The industry as a whole has also developed into this incredibly complicated coopetition landscape especially around containers in the past decade.

I feel like that kind of self-interest is the origin of open source, and always what has powered it when it's working.

The original apache httpd, for instance, was written by people working at different organizations collaborating together to build the web server that they needed. It wasn't something people did as just a hobby, although the individual contributors may also have had some altruistic and certainly cooperative motivations. It definitely wasn't something people did motivated by trying to make money off of selling apache httpd in some way.

Open source always requires self-interest to fuel it. The original model was people collaborating across organizations because it provided something useful to their organization -- exactly this. When this stops happening, because organizations cut budgets to the bone; because they gatekeep anything that might benefit competitors as well as themselves; because they can free-load off existing open source instead -- what other source of self-interest is there?

One is people trying to profit off of the open source itself, with "open core" models, or models where the license can appear open but tries to ensure that the creators maintain a monopoly on any profitable aspects. Or, we can just get lack of investment in open source everyone is trying to free-load off of, withering open source with burnt out maintainers.

Companies committing to open source because it saves them the time of maintaining it separately is the best case, and describes the environment of the "golden age" of open source.

I met an open source maintainer once working on a very popular tool everybody uses who was practically impoverished. From talking to him it became apparent that he was routinely being taken advantage of across all spheres of his life.

It made me wonder how much valuable open source is powered, quietly, by people who are just used to being exploited.

Probably almost all of it, except large projects with lots of paid devs, like Linux or RedHat.
This is insightful.

> Open source always requires self-interest to fuel it.

That self-interest doesn't necessarily have to be profit motivated either. As one of the replies to your comment suggest, probably the majority of open source is maintained as a hobby by an invested/interested user. Open source does not have to have a profit motive to make sense.

But likewise, a profit motive is also congruent with open source. It greases a lot of axles, to put it nicely. There's definitely some disparity in open source contributors in terms of money being made, almost like the income gap between wealth and poverty in general. I wish we could do better at fixing that and helping the "little guy" that is contributing their soul into a project.

Still though, open source is driven by "selfish" means, in whatever form that takes.

Sure, but why does that matter? A lot of good software got developed by itch-scratching.
Their s2n projects seem like valuable contributions to the ecosystem. Strong, well maintained libraries supporting networking and cryptography are important.
I wish that same drive would come to their devices division which do the absolutely bear minimum not to be sued for GPL Violation when they use GPL code on their Fire Devices

Edit: Also doesn't AWS have a history of Microsoft style EEE, where they embrace an open source project, make a service from it, extend that service in a non-open way and then push people to use their extended non-open service instead of improving upstream?

Hasn't this resulted in a number of projects changing their licensing to prevent it?

I think ElasticSearch license change was biggest such change, as was discussed previously on HN: https://news.ycombinator.com/item?id=25776657
Their "obsession" with contributing isnt really in evidence on their fork:

https://camo.githubusercontent.com/71f7e9147092c7d4b08df18da...

Worth bearing in mind that the only reason fork even exists was so that they didnt have to pay Elastic a cut when selling hosted ElasticSearch.

If they cared about open source they could start by not undermining the business models of companies that do more than just pretend to support it.

Having to pay for hosting an open source product? Isn’t Amazon doing what they should in this case where a company tries to block any competition.
No. Amazon was abusing their enormous market power to squeeze the company that made this software out of the market of hosting it.

(Not to mention lying about "partnering" with them)

If allowed to continue this would kill the project. If only Amazon can monetize it, it doesnt get developed. The open source project then dies.

Amazon is no more doing the right thing here than they were when they destroyed diapers.com with a strategy of sustained predatory pricing.

The fact that they want a pat on the back for being "pro open source" now after pulling this kind of crap is just icing on the cake.

They forked it when the upstream project switched to using a non-open-source license. This is a reasonable thing to do if your goal is to run FOSS and if you have the resources to maintain it.

Notably, you could build GPL software on top of OpenSearch. You could not do so on top of ElasticSearch at the moment.

switched to a non-open source licensing because AWS as leeching off the project, it was completely reasonable for ElesticSearch to change their licensing in the face of what AWS was going which while legal was IMO unethical
Wasn't that license change after AWS started playing funny buggers with Elastic?

So, Elastic then chose to change their license to try and counter it.

Amazon referring to it as "not open source" is a bit like Russia calling Crimea "Russian". This redefinition of open source is intended as a political statement - fighting against open source with restrictions.

Especially so, it seems, if those restrictions promote the growth of open source.

Elastic changed to a license that is completely open except insofar as it blocked Amazon from profiting from it.

>Notably, you could build GPL software on top of OpenSearch. You could not do so on top of ElasticSearch.

Notably you can't build MIT software based on GPL software either. This is 100% about MIT being a weak open source license with few restrictions and GPL being a strong open source license with restrictions.

Not coincidentally the very-much-also-open-source-license GPL also had a history of infuriating big tech for exactly the same reason - they hate restrictions put in place designed to help open source flourish.

E.g. Here's Google getting publicly grumpy about the AGPL: https://www.theregister.com/2011/03/31/google_on_open_source...

> It was changed to a license that is completely open except insofar as it blocked Amazon from profiting from it.

It didn’t just block Amazon, and even if it did, selectively blocking a single downstream commercial user would make it not open source.

The freedom of third-party competitive commercial downstream use is a guarantee against effective vendor lock in, and it is one of the important reasons to choose open source solutions.

Does that make it hard to monetize development of software by just rent-seeking in the “we don’t want to expend the effort to host our own things” SaaS market with a hosted implementation? Yes, but, too bad.

>It didn’t just block Amazon

No, I guess it also blocked google, oracle, etc. from monetizing their work without sharing the proceeds from the quite considerable profits they make (something they seem happy to do).

Didnt block anybody else though, as you well know. Any regular user of Elastic faced 0 additional restrictions.

>Does that make it hard to monetize development of software

No. As Amazon demonstrates the software can still be monetized by the most powerful rent seeking players in the market like Amazon.

Does it make it hard for the companies not running monopolistic rent machines that actually build open source? Yes. MUCH harder. It would have killed elastic.

>too bad.

Yes, too bad for open source.

You can be anti open source if you like (it has made a lot of people mad, not just MSFT and Amazon) but don't try to dress it up.

It is pretty well accepted that "open source" in the context of software is referring to the OSI's Open Source Definition. Elastic themselves say they're not calling Elasticsearch open source:

> Does this mean that Elasticsearch and Kibana are no longer Open Source?

> Yes. Neither the Elastic License nor SSPL have been approved by the OSI, so to prevent confusion, we no longer refer to Elasticsearch or Kibana as open source. We updated our website and our messaging to refer to these products as “Free & Open,” and when talking about the licenses directly, we describe them as “source-available.” If you notice an area we missed, please let us know, so we can correct it.

https://www.elastic.co/pricing/faq/licensing

OSI has become corporatized with their "approvals"...

I am not sure anyone should care what OSI thinks about a given license

>It is pretty well accepted that "open source" in the context of software is referring to the OSI

To the corporate funded OSI. I know.

The same corporate OSI who joined in the character assassination attacks on Richard "open source comes before profit" Stallman for very similar reasons.

Not sure I'd count that as "EEE"? I guess in a way you could see "We can't compete with AWS at running our own software" as AWS "extending" it, but I don't think it is quite that?
> do the absolutely bear minimum not to be sued for GPL Violation

Could you expand on what you mean? Are they violating the GPL? What does it mean to do the bare minimum but not violate the GPL?

It's unrealistic to expect corporations to go above and beyond when complying with something like a code license. If you want someone to do something, put it in the text of the license.

If you lower expectations to literal legal text only, then they don't even have the "it would be bad for PR and developer relations not to do it" incentive.
They drop a zip file with their open source code somewhere on the website at some point, but don't try to upstream patches, work with Linux kernel people in public, or provide a public git history

To be fair, it's a tough sell to tell your management chain that you have to spend a week every month arguing and refactoring code just to make some internet people happy about it

> it's a tough sell to tell your management chain ...

Sounds like a weird management chain then.

Generally I've seen the conversation go along the lines of "we should 'contribute' these changes back, so the upstream people can check them over for bugs, and we don't have to write this stuff again for the next product".

ah, I dunno if anyone in the linux community is really going to be going out of their way to fix MediaTek's shitty code for free as a favor to Amazon
>>It's unrealistic to expect corporations to go above and beyond when complying with something like a code license.

This is completely false as they (amazon) want to market themselves has developer friendly and supportative of open source

Their actions however say something completely different.

Further Loads of companies and organizations go beyond the legally required bare minimum in all kinds of things not just open source. You have a completely warped view of human interactions if you want to boil all humanity, and all interactions to just what is "legally required of us"

>It's unrealistic to expect corporations to go above and beyond when complying with something like a code license.

I agree up to the point where they start going on about how much they love and support open source. At that point going beyond legal necessity should be a given.

Why? Who decides they need to do that? How do you explain to a CEO or CFO that you want to spend more resources (money sure but time as well) than required doing things out of love? Who decides how much more?

If the legal requirement is X, and you do X+Y, there will still be people bitching about the fact that you didn't do Z, or that Y is really just self-serving, or something else. Someone someone will always complain about what you're doing because you're a profit-making enterprise.

>Who decides they need to do that?

The corporation decides to do that. They shouldn't advertise that they support charities if they don't actually support charities.

Who's talking about supporting charities?
> Also doesn't AWS have a history of Microsoft style EEE

AFAIK (but I'm far from knowing about all examples), no, they have a history of taking open-source software, forking it out of the control of the creators, and extending it there without bothering to sync the changes back.

It is hostile to people that want to gain anything from their software, but it is not against the license.

I’m pretty sure that is EEE
One of the things that led me to lose interest in AWS was its attitude to open source.
>Though it was always incorrect to lambast AWS for “strip-mining” open source

It really wasn't. That was their whole approach to Elastic.

The tantrum they threw at Elastic when they relicensed was particularly cringeworthy.

They didn’t throw a tantrum. They just forked and continued. A perfectly reasonable approach.

I think it’s important to describe things accurately. Disliking something isn’t a “tantrum.” I think amazons blog posts were pretty normal speech and tl:dr’d to “we don’t like it, so now we forked it and will maintain under Apache.”

And they did that, so it’s actually kind of nice as I can’t use ElasticSearch under its new license but can use open search.

Tantrum is pretty accurate I think. They voiced their extreme displeasure in a war of words with Elastic. They knew that the relicensing was targeted 100% at their gravy train and they condescendingly declared that they would "step up on behalf of open source" while they stepped up exclusively to prevent Elastic getting a cut of their ElasticSearch hosting profits.

The fork is really half hearted with about 1/10th the resources Elastic dedicates to the mainline branch:

https://camo.githubusercontent.com/71f7e9147092c7d4b08df18da...

Meanwhile, the only effective licensing difference is that you can sell hosting with their fork - something that only benefits them.

A lot of forks are created for similar reasons, it's not a fight between the great enlightened and the vile defilers. It's a for-profit (Elastic) wanting more control over their product (ElasticSearch) while also making ElasticSearch open-source under a permissive license that they used to kick-start their growth and another for-profit (Amazon) leveraging the open-source license to offer the product as a service to their customer base.

When they had a disagreement Amazon chose to fork it. You can frame Amazon as "cringe" and "throwing a tantrum" but ultimately this is hardly either of those.

The cringeworthy part was Amazon self righteously declaring that they were acting as staunch defenders of open source when all they were really doing was trying to deny an open source company a well earned cut of the profits Amazon made from hosting the software they build.

Lots of companies do similar or worse, I do agree. For instance a certain company that has obviously paid a PR company to go on an open source charm offensive (the article has all the hallmarks of pg's submarine) in this thread has strict rules about when its workers can pee. That's probably worse.

What open source company? Not Elastic, surely.

> […] obviously paid a PR company to go on an open source charm offensive […]

Please don’t do this here. Consider that others can disagree with you, or that you may simply be wrong.

> Please don't post insinuations about astroturfing, shilling, bots, brigading, foreign agents and the like. It degrades discussion and is usually mistaken. If you're worried about abuse, email hn@ycombinator.com and we'll look at the data.

>What open source company? Not Elastic, surely.

Why not? That's immediately who I thought of.

Company 1 develops an open-sources a product, and in order to make money from it, provides that product as a hosted service.

AWS uses their market dominance to provide a similar service and eats Company 1's lunch.

Company 1 changes their license so that any other company hosting the product as a service has to pay.

AWS forks it and acts like the good guy.

>Please don’t do this here. Consider that others can disagree with you, or that you may simply be wrong.

I was referring to the PR whitewashing piece submitted (written by an ex Amazon employee, no less), not to anybody on this site.

Please ask questions in future before accusing and refrain from assuming malintent.

That really depends on the reader. ElasticSearch is a company, and made its product open source as a growth strategy, it worked!

However , it is difficult in general for these open source companies to compete with Amazon's managed offering. The managed offering and its support staff eliminates the need for the tech service contracts that many of these companies to depend on.

ElasticSearch closed down the license and said "nope, no open source anymore because we have to make money". This was bad for the community, but not AWS fault

And if it wasn't AWS, it would've been Google. These sorts of open source development paid support contract based companies are having a hard time in general competing with the cloud.

I think it's fair for AWS to criticize Elastic for violating their ethos to make more money, but it's also pretty clear that the reason their OSS business model didn't work was managed cloud services like Aws.

Meanwhile, On Google Cloud, instead of using a service that uses a rebranded elastic, you just install elastic from your GCP console, and the company that develops it gets revenue.

And Mongo, Redis Enterprise, Databriks, neo4j, etc.

Their 'partner solutions' and not then directly competing with their partners is a win/win.

https://www.elastic.co/blog/elastic-and-google-cloud-announc...

Makes sense if you think about incentives. AWS has a lot of financial incentive to build on open source projects used internally (at a massive scale, I assume). Meanwhile, not a lot of incentive to maintain internal forks on a big project like PostgreSQL
Is there a good lookup list for all opensource projects and their corresponding AWS branded services?

Presto/Trino : Athena

It's Azure but, Apache Atlas : MS Purview
The author seems like a “open source strategist” who does not write code, but if one does, he will likely understand the difficulties of contributing to open source in a cloud service environment and why the observed “free ride”. It will take too many cycles to push an even trivial change upstream and sometimes you can never do that. But your customer does not wait for you, they will happily go to your competitors when they “free ride” more than you do.
It'd be nice if they open sourced some of their libs. When I was trying to debug a very odd issue with their Python Glue stuff, it was impossible to delve into, because the open source Python just wraps a Java blob you can't see.

https://github.com/awslabs/aws-glue-libs/tree/master/awsglue

> just wraps a Java blob you can't see

I ahve to admit no familiarity with Glue, but if that Java blob is bytecodes, shouldn't it be trivially decompilable?

The blob lives in Glue, so not available to decompile.
I lead client-side application development in AWS IoT and I can tell you we’re investing heavily in open source in AWS IoT. We can build features a lot faster without having to own the infra ourselves and cut out a lot of internal red tape. Customers love features and customization.

Examples: AWS IoT Application - https://github.com/awslabs/iot-application - OOB app for monitoring IoT assets.

AWS IoT App Kit - https://github.com/awslabs/iot-app-kit - Library for building AWS IoT apps.

To the commenters making embrace/extinguish overtures, I’ll say that log4shell got a lot of Amazon very spooked re. supply chain security.

There were a number of internal programs spun up with blank checks to mitigate a future log4shell. The Open Source program was one of those

Demand for a product increases when the prices of its complements decrease. For example, demand for mobile phones increases as the price of mobile OSs and applications declines.

To maximize demand for a product, make its complements free. For example, to maximize demand for mobile phones, make the operating system free and add a growing number of built-in default apps for free.

Surely, Amazon is thinking along these lines. Which Amazon products/services benefit from all the free code Amazon is contributing/releasing? AWS.

See also: https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/

I don't see software <-> aws as complements like windows <-> pc, because this software doesn't necessarily need aws like windows
Author disclosed he worked at AWS, but worth noting he led open source marketing:

https://www.linkedin.com/in/mjasay

Why is that worth noting other than he probably still has an interest in open source at AWS? He works for a competitor now, which makes this post even more remarkable.
>Though it was always incorrect to lambast AWS for “strip-mining” open source

Amazon absolutely strip-mined open source.

In 2001-2006 Amazon was all built on open source and had ditched HP and Digital Unix for Linux, RedHat, Apache, Xen, etc.

It was abusively hard to contribute back to open source projects, including the Linux kernel, and you had to get extensive signoff and review to ensure that you weren't giving away any IP with your lock order bugfix or whatever. There were also constant reminders and threats to not engage in open source discussion in any way that could leak company IP which created an atmosphere where it was much easier to just not participate at all. It was definitely very easy to pull open source into the company and very, very difficult to participate going the other direction.

Back when you were at Amazon, I was at Red Hat. (disclosure: I am at Amazon now, and have been for a bit over 13 years).

From my perspective when I was at Red Hat, Amazon was very much an IT consumer. Amazon was a _hugely_ important customer for Red Hat, even appearing on stage to give their testimony about the benefits [1]. It was a "lighthouse" customer that really opened some doors to selling Enterprise Software.

Part of what we sold to Amazon was being their connection "upstream". In 2001 it was quite unusual for an IT shop like Amazon to participate directly upstream. One benefit of being a customer of Red Hat's was: they will fix bugs and work upstream on your behalf.

It was a very reasonable approach in those days to use an existing contractual / support relationship with a vendor like Red Hat to deal with all of those details and complexities, which often required strong social connections among open source developers and maintainers working together upstream.

I think that over time as large companies shift to consume open source software directly from upstream, they have to build the capabilities and connections to participate directly rather than going through an intermediary like Red Hat. It can take a fairly long time to do that.

[1] https://www.computerworld.com/article/2577912/linuxworld--am...

So, I take it you're familiar with the history between those two then, and you're ignoring it for some reason? ;)