I agree, let me take a stab at starting the conversation.
First of all, huge respect to Slava for this writeup. Having your startup fail is hard and it is not the time you want to blog about it. RethinkDB going broke was a sad thing to see for me, I can't imagine how it felt for him.
I think the analysis of the two root causes (hard market, focus on the wrong metrics) is accurate. It is very sad to see that in the end correctness doesn't win the day, not even for databases.
Since I run a startup too I can't help but apply his criteria to GitLab.
3. A use case => we're making GitLab an integrated tool with a broad scope https://about.gitlab.com/direction/#scope A good way to develop software from idea to production.
I hope the above list doesn't come across as pretentious. I welcome pushback and the opportunity to talk more about the OP.
I think the OP premise that it is hard to make money in the cloud is accurate. We're spending hundreds of thousands of dollars to make GitLab.com run and revenue takes a long time to grow. Selling on-premises software products has higher margins than a service.
BTW I appreciate the shoutout to GitLab as one of the five exceptions that are doing well in the open source developer market.
I'll attempt to add to this. I might be stating things that are a bit controversial here.
1 thing we are always transparent with users about is how our business model works. 1 of my favorite videos on the internet about this is [1]. We are very much based on closed source for making revenue. This has actually helped with our community a lot.
We are basically focused on closed source solutions for specific verticals. The only way to get clients to pay is to make something more efficient or don't open source/core it. For open source companies there is always a trade off of community vs customers. You have to find a hard line and stick to it.
Community is great, but most of those people won't want to pay you. You can try to convert them, but that takes a lot of time and isn't the best path to market.
We strive to provide a great base line experience to our users, but a few of the things we have found in practice that helps:
1. Features have customer names on them
2. There's a mass of community interest in a specific feature
You have to be careful about where time is spent.
It's very easy to mistake "users" for "customers".
I wish slava and co the best of luck and really appreciate their post mortem. 1 thing we've learned as an infra company is "be as close to the app as possible". In our case the bulk of what we focus on is banking/telco anomaly detection.
It's a straight forward revenue making product that can benefit from deep learning. The great thing is we can grow in to other use cases later on if we find something else interesting to work on.
Even seen this backfire at many commercial companies. Adding whatever features a company is willing to pay for will often pull a product in multiple, incompatible directions.
I'll give you that. In my case, a lot of what we do is look for numbers and how well it fits our main application area.
You definitely have a valid point here though.
That actually goes hand in hand with "users look like customers". Also: "Not every customer is a good customer."
I always liked the concept of rethink db and I tried to get started with it a couple of times but in the end I just wanted to get my project(s) out of the door as quickly as possible.
This to me explains the massive success and quick rise of things like redis, mongodb etc. Sure the defaults are insecure but they work and they work immediately.
I don't know, my personal opinion was that RethinkDB had its head on straight, MongoDB is garbage and still neither is so much better than PostgreSQL that I will switch away from it.
Postgres is the default datastore (because schemas are awesome) for me, and I haven't had a use case yet where I needed something that Postgres wouldn't do. Maybe if you have a very specific need, you'd reach for another datastore (come to think of it, I have successfully used Cassandra for multi-DC deployments of a distributed read-only store), but that's not the norm for me.
Rethink, Mongo et al are scalable (at the expense of some ACID). PG is certainly a good DB (I use it in production) but it is at heart a traditional RDBMS - not designed for scalability. Apples vs Pears..
Postgres neither scales vertically or horizontally.
v9.6 finally brought some parallel scans but is still limited in scope and far behind the commercial databases. Same with replication and failover, although there are decent 3rd party extensions to get it working.
I suppose the parent comment was more about scaling up for transactional workloads - the story there is a lot better (although there's still issues left we haven't tackled, but mostly on very big machines) than for analytics workloads. But yes, while we progressed in 9.6, there's still a lot of important things lacking to scale a larger fraction of analytics queries.
The vast majority of us probably won't ever need that scalability anyway. Most places I've seen that needed something more scalable needed to just stop being so retarded with the tools they already had.
As for the clustering in general... Scalability may be not important, but replication and failover should be. I believe, every place that had existed long enough must've had theirs "ouch, our master DB host went down" moment.
I wasn't able to set it up properly some years ago and settled down with warm standby replica (streaming + WAL shipping) and manual failover. Automating it (and even automating the recovery when old master is back online and healthy) was certainly possible - and I got the overall idea how it should operate. But the effort required to set it up was just too big for me, so I decided it wasn't worth the hassle and settled down with "uh, if it dies, we'll get alerted and switch to the backup server manually" scenario.
Having something that goes in line of "you start a fresh server, tell it some other server address to peer with, and it does the magic (with exact guarantees and drawbacks noted in documentation)" would be really awesome. RethinkDB is just like that. PostgreSQL - at least 8.4 - wasn't, unless I've really missed something. I haven't yet checked newer versions' features in any detail, so not sure about 8.5/8.6.
Thanks for the link, I think I haven't saw this one before.
As for automation... Things can always go wrong, sure. But I wonder how many times HA and automatic failover had saved the day at GitHub so no outside observers had a faintest idea there was something failing in there.
> you start a fresh server, tell it some other server address to peer with, and it does the magic
This is the one thing I ask of every database software, yet we still don't really have it. 90% of problems could be solved if there was a focus on the basics like easy startup, config and clustering.
There are actually add-ons to Postgres[1] that add MongoDB protocol compatibility, and even with that overhead Postgres is still faster.
And even such benchmarks don't tell full story. I for example worked in one company that used Mongo to regional mapping (map latitude/longitude to a zip code and map IP address to ZIP). The database on Mongo was using around 30GB disk space (and RAM, because Mongo performed badly if it couldn't fit all data in RAM), mainly because Mongo was storing data without schema and also had limited number of types and indices. For example to do IP to ZIP mapping they generated every possible IPv4 as an integer, think how feasible that would be with IPv6.
With Postgres + ip4r (extension that adds a type for IP ranges) + PostGIS (extension that adds GEO capabilities (lookup by latitude/longitude that Mongo has, but PostGIS is way more powerful) things looked dramatically different.
After putting data using correct types and applying proper indices, all of that data took only ~600MB, which could fit in RAM on smallest AWS instance (Mongo required three beefy machines with large amount of RAM). Basically Postgres showed how trivial the problem really was when you store the data properly.
That's been my experience as well. Postgres handles everything until you hit an edge case on some of your data (volume writes with scale out, performance critical advanced functionality, heavy BI work with huge datasets, etc).
When you hit those edge cases or specific needs, you implement the specialty solution. Most people never come close to those edge cases though. When they do, PGs foreign data wrappers make the adjustment as smooth as possible.
I was truly hoping that RethinkDB would be the Mongo that Mongo could have been: a NoSQL database, but one with joins; a NoSQL database that is actually CP [0]; a database that comes out of the box with granular real-time updates (so you don't even need to worry at first about the extra moving parts of a Redis server or other queue). For a business starting from scratch with those needs, I'm making do with Mongo, but I was anxiously waiting for RethinkDB to get more battle-tested... alas, it wasn't meant to be. And I fear that we'll be stuck with Mongo for quite a while.
The truth is that NoSQL has great promise, but the folks with actual money to spare today are the ones who want a better SQL database, because they can afford the tradeoff of development time for reliability that SQL administration and migration provide. And people who solve the SQL pain points, such as the excellent folks at CockroachDB, and the various folks building replication layers on top of Postgres, can hopefully keep the market for database startups alive long enough for someone else to fix the NoSQL landscape as well.
I think Rethink would be that, but I don't know, I've never found SQL databases that hard to manage. I've found NoSQL databases hard to manage, because, invariably, someone will be doing a write right when you're trying to do a data migration/transformation, and now you have a single record that looks wrong, and it's a bug waiting to happen on read, rather than on write (where it should).
Rethink definitely had its use cases, I just never saw it as my primary data store.
I agree Postgres is pretty awesome, but I think the one thing it has going against it is a good GUI client. MySQL has way too many good options wherease Postgres is stuck on this front.
Please try Postage. It's free for all users, open source, comes in server and desktop distributions, and we listen to our users. You can get it at https://github.com/workflowproducts/postage
If it doesn't meet your needs please file an issue, our goal is to remove the no-good-GUI limitation that was imposed on postgres sixteen years ago.
Just downloaded it and it seems effective and well-polished. Thanks for the recommendation, mySQL Workbench has some killer bugs that made me stop using it.
I've tried really hard to like Postgres, but coming from SQL Server land, the tooling is mind blowingly atrocious. Every few months I go on a google spree trying to find the magic tool to make it less painful, but everything is either just as awful as pgadmin or costs way too much.
We'd would love to have your help on Postage. We want to be the Microsoft SQL Server Management Studio of PostgreSQL. Just file issues at Github for bugs and features, we'll do our best to keep up.
Microsoft gets a lot of shit for various and mostly good reasons. But their tooling is seriously top notch. I would murder kittens to get tools like SSMS for postgres.
Hell Python Tools for Visual Studio almost makes me want to do python dev work on a PC.
I developed and use(d) pgxplorer (github.com/pgxplorer). So, I can tell you this: the best out there is psql (CLI). There exists no use case where a GUI is better than psql.
I worked with people in the past that refused to touch postgres because it doesn't have a UI they like. They insisted on using MySQL for that sole reason.
+1. The main reason I didn't jump on to Postgress decades ago was because the tooling was really bad. Third party tools for MySQL were streets ahead, even back then.
End result is that I jumped on the MySQL bandwagon, and even all the negative posts against it now and the lauding of Postgres in here all the time cannot make me change tracks. I just have too much data and IP invested in it now.
Based on the postmortem, you weren't alone in thinking that way, it's just that all the people who did think so, didn't value the difference enough to actually pay for it. And, indeed, when something like Postgres is free...
There is bigger market forces that isn't being considered here - namely that the entire database as a service margins have been nearly driving down to zero by AWS. Why do you think firebase had to sellout to Google ? Why did Parse shut down ? Simply because the big cloud vendors figured out that providing paas services at nearly the cost of the infrastructure will win them even more compute workloads which is the real money maker (especially if your are reducing your computer pricing slower than Moore's law)
He says that the market was terrible because open source developer tools suffer from oversupply. I don't think that's what happened. Rather they chose to make a product for which the next best alternative (e.g. Mongo or Cassandra or MySQL or Postgresql) had almost zero cost. It doesn't matter how oversupplied the market is if someone is willing to give away a viable product for free. Perhaps "oversupply" is another way to say "someone was willing to sell at zero cost", I'm not sure.
Unless you can find a set of customers who see significant value in your product vs. the alternative zero-cost product, you're never going to make money.
> especially if your are reducing your computer pricing slower than Moore's law
I agree with most of your comment, but I take issue with that last part. Moore's law has no relation to the price of computing. It only makes predictions on the number of transistors in a CPU, not the performance of the CPU nor the price/performance ratio of the CPU. Even if it did, compute costs much more than the CPUs; memory, disk, power, cooling, networking, enclosures, and more all cost a significant amount relative to the silicon.
I know Google Cloud declared they were going to decrease prices at pace with Moore's law, but even if they can deliver on that it will be through innovations outside of the CPU.
Actually, I think learning to understand how these (complex) market forces come into play is exactly why Slava added the recommendation to religiously read The Economist at the very end of the article for maximum effect.
I really appreciate the deep introspection in this post. We can all learn a lot from it. I'm one of the (possible minority) of HN users that doesn't care about startups and business, so for me the takeaway was about how choosing quality and correctness over speed works out in the real world. It seems that you really can't take the idealist approach to quality that RethinkDB took. The world just isn't willing to wait, and it's not willing to take the time to understand a system and to see how clearly it distinguishes itself from the alternatives. Our decisions are driven by speed. Our culture is "always be shipping." But part of the result of this is that all our software is riddled with bugs and usability issues. Even the most basic functionality of our computers fail all the time, all the way down the stack. It's depressing that a really solid, honest attempt to produce extremely high quality software like RethinkDB can't survive in part because of these things.
> The world just isn't willing to wait, and it's not willing to take the time to understand a system and to see how clearly it distinguishes itself from the alternatives.
This world is always willing to wait, it was the RethinkDB team and/or their investors who weren't. The world will wait for you in eternity; it's just not willing to provide you with capital during the wait.
Your statement makes sense if there were a dozen competitors in the database space. With hundreds of competitors, actually evaluating them all becomes more expensive than just choosing eg. a managed SQL solution and then switching provider if it doesn't live up to its promises.
That's so much more flexible from a business perspective than evaluating a hundred different open source database systems, settling on one, and then having to rewrite your database code when the database you've chosen ceases development, and you're left as the maintainer of it.
I was a huge advocate of RethinkDB, absolutely love what Slava and the team were able to accomplish, but I am having a really hard time digesting that "Correctness", "Simplicity of the interface" and "Consistency" were the issue here. Example: Realm.
I can only imagine what the team at Realm is thinking after going through comments and the post-mortem. Realm was an open-source mobile first database for the past 3 years. It's free. Its the most correct, consistent, and simple mobile database that is running on millions of devices[1] (Yup fine, kind of my opinion but trust me mobile community will back this up). They ship FAST, they ship real USECASES. They just recently entered the cloud space with their Mobile Database Platform, and they are even experimenting with new things like a version built for Node.js.
I'm honestly not trying to make a specific point, but asking more questions... Is mobile different? Is it easier? Are mobile developers more willing to try new technology? Are they more willing to pay for technology? (Realm is free, haven't paid them a penny, so not sure about this one...)
My main point I guess is that it's not a terrible market, Realm made it work with VERY similar ideals and goals. So what else is going on here?
I'm not familiar with Realm, but after a brief glance at their site:
Yes, mobile in this context is very different. Or rather, local is very different. Building a database that resides on a local device and is primarily accessed by a single user is an entirely different class of problem than one that is on a server. Usually most of the hard problems in building a database system involves dealing with multiple people accessing or modifying data at the same time. When you have a single user, that problem either disappears or is massively reduced.
As you said, Realm does appear to have a server offering now, but from the sounds of it, it hasn't really been proven yet, so I don't think it's as easy as saying they've succeeded where Rethink failed.
Absolutely, in no way am I saying their cloud offering is successful. What I am saying is they have shown you can be successful with the very same ideals and goals that the RethinkDB team had, with a very similar developer-minded audience. The biggest factor here is obviously the platforms their database runs on - Mobile (or local as you put it nicely) vs cloud.
I don't actually think it is easier to get adoption. The major mobile platforms have SQLite built-in (under the hood in the case of Core Data). I think that's actually a strong barrier to overcome.
Furthermore, mobile is becoming in-demand or even core in both consumer and enterprise. For enterprise especially, though, you have to think of it as mobile + sync, not on-device only.
Turns out sync is the really hard part when you want to build something that's robust in the face of poor network connectivity.
It's probably too early to say that Realm made it work.
They've raised a roughly similar amount of VC money as RethinkDB did. Their last raise was a few years ago, before Parse's shutdown, so I suspect that investors are more antsy about this space today.
The fact that Realm recently entered the cloud space sounds similar to RethinkDB's trajectory too... Of course I hope Realm makes it, it's a great product. But I'm worried that they're trapped in the VC-funded model like RethinkDB was.
It's in the Worse is Better essay. Strange enough, the essay series the author drew on advised the opposite strategy because it wins in the market. Here's a simpler one from someone who developed a product in high-assurance security then worked to turn Microsoft security around:
He details why they put marketing and shipping first with constant improvements on correctness or security. On top of this, I advise just embedding the benefits into the product & designing it for easy change so components can be replaced once big revenue comes in. As development pace slows, QA pace can increase on components stabilizing.
Fascinating read, albeit a sad one: I love RethinkDB, and use it personally every day in our particular production deployment.[1] But -- and to Slava's point -- we didn't/don't/wouldn't pay for it, and in that regard, we were part of the problem. That said, the constraint on that particular problem was that it had to be open source, and if it couldn't have been RethinkDB, it would have been some other open source database -- proprietary software is a non-option for that use case (or indeed, for any of our use cases).
So I feel Slava's pain (and feel somewhat culpable as a user of RethinkDB), but I don't entirely agree with the analysis. Yes, infrastructure is a brutal space because it has been so thoroughly disrupted by open source -- but the dominance of AWS shows that open source can absolutely be monetized (just that it must be monetized as a service). Indeed, right now, I would love to deploy a RethinkDB-based service -- and if the company still existed, we (and by "we" I mean "we Samsung", not "we Joyent"[2]) would potentially be in a serious business conversation about what a supported, large-scale deployment would look like. Actually, that's not entirely true, and it leads to me to one thing that Slava didn't mention. While it kills me to bring it up, it does represent the single greatest impediment I have personally found to deep, mainstream, enterprise adoption of RethinkDB: its use of the AGPL. Anyone who has been part of a serious due diligence knows that the GPL alone is toxic to much of the enterprise world -- and the GPL is a buttoned-down corporate stooge compared to the virulent, ill-defined, never-tested AGPL. The AGPL is a complete non-starter for any purpose, and while I appreciate why the AGPL was selected by RethinkDB (and very much appreciate the explicit riders that RethinkDB put on it to make clear what it did and didn't apply to), the reality is that no one -- absolutely no one -- has built a business on AGPL software and it seems vanishingly unlikely that anyone ever will.
I still use RethinkDB and still intend to -- and I still harbor hope (crazily?) that whatever entity that owns the IP will see fit to relicense it to something that is palatable to the kind of people who care about their data. I actually believe that a business can be built on the technology -- which, to Slava's points, I have found to be delightfully sound in both design and implementation. And that RethinkDB is open source means -- for us, anyway -- it will continue to live on, despite the unfortunate demise of its corporate vessel (AGPL or no). That said, I would love to have a lot more company -- here's hoping that we see RethinkDB relicensed and a thriving business behind it!
> ... the reality is that no one -- absolutely no one -- has built a business on AGPL software ...
MongoDB server is AGPL - client drivers are Apache. To be fair the business side is based around larger scale deployment and management, and is proprietary add-ons.
I'm kind of surprised that approach wasn't mentioned for Rethinkdb - a free open core product, with paid extras dealing with pain points in larger deployments.
It is possible to buy a different non-AGPL license to MongoDB -- https://www.mongodb.com/community/licensing. With RethinkDB it seems completely impossible to do so. I begged and pleaded: http://sagemath.blogspot.com/2016/10/rethinkdb-must-relicens.... I just spent most of my time during the last two months rewriting SageMathCloud to use PostgreSQL instead of RethinkDB, with the catalyst for doing this being the AGPL licensing and some concrete enterprise customers needing a completely non-GPL'd stack for SageMathCloud. Coming out of this rewrite, and back to PostgreSQL (which I've used off and on for decades), I'm very impressed by PostgreSQL today, and the LISTEN/NOTIFY functionality is a solid building block on which to build something like changefeeds (thanks to the many HN comments on previous stories about RethinkDB for pointing this out!).
While I like that idea (paid extras to deal with pain points), it puts developers of the base product in a weird position. They are incentivized to ignore user complaints on one codebase, as that's a business driver on the other codebase.
I wish more databases were developed with modular, reusable pieces. RocksDB is already a popular storage engine for anyone building a database today. Throw iterlib[1] (shameless plug) on top of it, you get a query API for free.
Now you get to innovate on top of these modules using reactive APIs, user space TCP, Fancy serialization, Standards compliant SQL etc.
Some of the current incumbents were written in the 90s when C++ was a different programming language, which hurts innovation.
I'm also interested to hear what the future, if any, looks like for RethinkDB. Would also be very curious to explore what can be done regarding the IP/licensing as part of that future (agree with the concerns mentioned here).
RethinkDB as a product is not dead yet, you can contribute in it. Re-Licensing attempts are already in progress , ex founders had to deal with legal issues for it so it will be long.
A hosted distributed SQL-ish database running on Joyent would be amazing! I checked the available packages on Joyent cloud many times not quite believing RethinkDB _wasn't_ on the list as it seemed such a great fit. Fast native containers with RethinkDB's shardibg support would have been fun to have "precanned" support for.
Currently the pre-packaged databases on Joyent don't support both clustering and consistency. Is there anything in the works for this segment?!
> the reality is that no one -- absolutely no one -- has built a business on AGPL software and it seems vanishingly unlikely that anyone ever will.
I strongly disagree with that. Take RavenDB, which is very much comparable to RethinkDB in terms of a business is dual licensed as AGPL and has a commercial license available [1]. This forces any business doing non-OSI approved OSS work to purchase a commercial license. I don't know too much about RethinkDB but I think they must have used a very similar licensing strategy?
From an outsiders' perspective, RavenDB appears to be going strong and doing well commercially. They are hiring and expanding and have invested a lot into v.4.0, which I'm really excited about.
Interestingly, on the spectrum between MongoDB and RethinkDB in terms of "Wrong metrics of goodness" (Correctness, Simplicity of the Interface) they appear to me much closer to RethinkDB than MongoDB. In terms of stability, they may have been somewhat in the middle (e.g. not as bad as MongoDB, but probably not as good as RethinkDB). And they have moved very fast to add features and fix bugs in the product. It's a really nice DB to work with as a dev, I suppose the #1 reason they are not as successful as Mongo is that they're coming from the .NET niche...
The article talks alot about big companies that make lots of money but as a user of databases I always saw the competition to ReThink DB being Postgres, MySQL and to a lesser extent ArangoDB and MongoDB.
The core problem for Rethink being that, well, I can get as much database as I want for free. I explored many/most of the new/nosql databases and in the end always found that Postgres was the best solution - you just shouldn't bet against Postgres it seems.
I really liked RethinkDB and thought it was well implemented but it just wasn't needed.
I do wonder if ArangoDB is going to go the same way.
Exactly this. RethinkDB's value proposition wasn't really that attractive compared to Postgres.
Replication and a nice UI, sure, those are nice. But Postgres has fantastic single-host performance, and Rethink is not yet in a position to compete there, except in some horizontally scaled use cases. Rethink doesn't even have transactions, making it potentially useless for some things that require strict atomicity over a complicated data model. In general, Postgres has such a richness of features that it's hard to compete with it; PostGIS alone is a "killer app" if you do any kind of GIS stuff.
For me, there weren't any such "killer app" features. The change streaming is nice, but it's not difficult to do that with Postgres either. That's it really: Every solution offered by Rethink could be countered with "that's nice, but trivial with Postgres".
The one exception being horizontal replication. I have yet to launch a project where Postgres couldn't scale out well enough, and I suspect that if I got to that point, I would reach for bigger/different guns such as Cassandra. For our apps we typically use Elasticsearch in front as a kind of read-only mirror of Postgres; the latter being the slow, serializable path, and the former the fast, slightly-out-of-sync path. Users don't know the difference.
Firstly, to the OP, that is an excellent post-mortem. Introspective, goes deep, generally confidence inspiring even though a part of it is literally "we were incompetent and didn't know it", well, lessons learned.
I'd like to add some colour, although I've never used RethinkDB so take it all with a large pile of salt.
lobster_johnson says:
> Rethink doesn't even have transactions
The post-mortem dwells heavily on how RethinkDB focused on correctness and was beaten by MongoDB which is, to put it mildly, not concerned about correctness.
But Rethink is a JSON DB that apparently doesn't have transactions and which on its web page says "Query JSON documents with Python, Ruby, Node.js or dozens of other languages".
Here are three technologies I do not strongly associate with correctness:
* JSON
* Schemaless databases
* Scripting languages
Rethink's own FAQ says "RethinkDB is not a good choice if you need full ACID support or strong schema enforcement".
So in the database market there are of course lots of users who do strongly value correctness, but they tend to use databases and technologies that are statically typed (Java/C#) and database engines that have are ACID and provide schemas.
Very few of them want a database that doesn't provide ACID, doesn't provide schema enforcement and which appears to either not support standard industrial languages at all, or is at least distinctly uninterested in them.
In other words Rethink targeted the hipster startup market of companies building social networks, ride-hailing apps and other things where correctness is not important and low-correctness tech dominates, but tried to sell a product that was oriented around correctness. This seems like a market/product mismatch.
That's a good point. Competing with MongoDB is difficult because "correctness" alone is not a selling point. It's the "worse is better" phenomenon as a marketing challenge.
That said, I don't think lack of correctness itself is a design goal. I think it's accidental, or at least an intermediate state before we have something better.
Case in point: I'm currently working on an open-source (it will be released soon) data store that uses JSON to express "documents" of data, but also has schemas, joins and transactions. It uses PostgreSQL internally, but completely hides the underlying storage. The developers who are currently using it for actual work are pleasantly surprised about how nice it is to work with: You get to work with rich, structured data in its natural shape without needing some kind of ORM to mediate between an app's data model and the database's, and at the same time it's very strictly validated without being pushy (it has gradual typing: you can start out schemaless and then tighten the schema). When we designed it, we wanted to build something that reflected how people actually use a database, and I think we're achieving that. (I'm looking forward to make it public and share it on HN.)
So it's certainly possible to have the best of both worlds. It's just that the direction that the world moved in with "NoSQL" temporarily moved us off course for a little while. It started out as a natural reaction to scaling issues, and in that sense I think it might have been a necessary step while people figured out what they were doing. We're seeing the same evolution in the ways of people's thinking with regard to dynamically/statically typed languages, too.
It should come as no brainer, the back bone of any business is 'supply and demand', while it may sound simple, it is increasingly becoming complex and understanding the new world economics helps to grow the business and 'The economist' seems to serve lot of insights.
It's a bookend to the recurring point that building a business requires a good understanding of markets and how companies fight for their place in them.
The Economist covers a lot of macro-economic news and frames the world in terms of market forces.
I think Slava is saying the type of biz-savy "personal development" he experienced (belatedly) could have been accelerated by more of this perspective.
It helps you view things through the lens of market forces. Most regular media is just pundits channeling some moral position based on the writer's politics.
The Economist is basically pundits channelling their personal politics these days too. Sadly. I used to enjoy reading it a lot more than I do these days.
I don't think The Economist is what to read for this situation though. A book on the history of Oracle or something might have been more appropriate.
Nice article. I think he's underestimating the failure on the sales team. They needed to find the intersection of Oracle customers and customers that needed the unique OLTP features of their product. I would have tried to bring an ex-Oracle guy on, for example, someone who's sold into finance or healthcare.
Thanks to Salve for writing an incredible retrospective. One thing I'd be curious to hear is whether the RethinkDB team feels they could have create the tool and potentially the company without taking on venture capital/investors (similar to PostgresSQL, which I don't think has taken on vc).
Excellent reflection, I think he forgot the 'elephant' in the room. Whenever I wanted to choose between Nosql solution, I always come back to Postgresql, just enough to fit my limited usecases.
I started a new project in RethinkDB over the weekend, and this postmortem makes me want to start my next one in it even more so. It really is exactly what I want in a database. Relational document storage, with a sql like language, and realtime change feeds. Who WOULDN'T want to use that?!
Slava (coffeemug) is very brave to introspect publicly and whenever I interacted with him, behaved with greatest integrity and kindness. Unfortunately, his hypothesis here that devtools are a tough market does not ring true. Making a business is always tough, making one that does something new is even tougher. There are many tools and database companies doing extraordinarily well, from Atlassian to Cloudera to Docker to Elastic. The difficulties that RethinkDB faced, as it pivoted from closed source MySQL DB backend, to open source independent database with a very cool update mechanism, were in my analysis much more mundane. Simply, RethinkDB was not easy. Here is a flavor of their thinking, when they rejected a REST HTTP interface: "The RethinkDB query language is so simple, expressive, and pleasant, that trying to make it available over HTTP would be quite difficult and a disservice to users."
Developers have a big advantage when making devtools in that they are their own target market. They are, but only at the start of the project before they become embroiled in it. Ironically this feeling of making it for oneself leads to a mistaken urge to thrust complexity onto the target. As an infra engineer myself, this is something I always try to keep in mind -- I worked on a modelling language at a large company and I credit its success to its absolutely straightforward and accessible syntax, lack of dependencies and straightforward integration into workflows. The temptation is always to make sure users are aware of the choices, force them out of bad habits and make them consider the tradeoffs (this Strange Loop talk from Peter Alvaro addresses the social treatment of users https://www.youtube.com/watch?v=R2Aa4PivG0g ). For RethinkDB, it was wrong to try to force people to learn a new language to use a DB, when there are just tons of interfaces already, and the new language has incremental benefits over say SQL.
However elegant the design of software, the deployment that is the largest or most demanding for it will always find problems. There's no point seriously trying to claim great reliability or quality for a nascent project. It's unbelievable. Instead you need to focus on getting it used in some friendly project that is willing to pay the cost of these teething problems for some particular feature. Once battle tested, reasonable tech leads can choose to use it, comfortable that they won't be the biggest deployment (e.g. git bootstrapped by its use in the Linux kernel). Alternatively, and less scrupulously, you can try to bootstrap by providing a very convenient experience at the start of a simple project for junior engineers - there are many NoSQL DBs that promise the world and are incredibly easy to set up (see the https://aphyr.com/tags/jepsen Jepsen blog for many sad examples, RethinkDB proudly did well in this test).
RethinkDB refused to make the market fit compromises necessary. Market fit is about timing. Don't read the Economist. Ship early and keep your users on board through whatever engineering compromise necessary.
I spent 1.5 years writing a ton of code using RethinkDB, and the last two months rewriting all the code (and more) using PostgreSQL. You are completely right, e.g., "For RethinkDB, it was wrong to try to force people to learn a new language to use a DB, when there are just tons of interfaces already, and the new language has incremental benefits over say SQL." rings very true. SQL is incredibly expressive and the tooling around it is very mature.
So I disagree with this. One of the things I loved about RethinkDB was the fact that it did away with SQL. The problem with SQL (in high transaction applications anyway), is that performance isn't necessarily reproducible, or determinable from the query. You are at the mercy of how the query optimizer decided fetch your data. The more complicated your query, the less you can can out reproducible performance.
With RQL you could very clearly determine how rethinkdb was fetching your data, and you knew it was going to get your data in the same way every time the query ran.
FYI, the post is still in its unpublished state. It's basically right, but I've been meaning to edit it for tone and rewrite the market failure section for clarity. Didn't get the chance to do that before it made it on HN, so keep in mind that you're reading a (late) draft.
If you're open to edits, the $200k revenue/employee rule of thumb really doesn't make a lot of sense when you're comparing your three alternate business models (Hosted Rethink, DBaaS Rethink and "Hey, maybe PaaS on top of Rethink?").
Like you, I thought it would be way too hard to run a service (my own company was encouraged to do so, and we shied away for it for the same reason you stated). However, having seen it now from the inside for GCE for nearly 4 years, I can say it's much easier to "service-ize" yourself than it is to have built a product people want. And moreover, the shell scripts and operations stuff really scales (effectively, the hosted DB offering is the binary plus a bit of bash, while the other offerings require yet more "not done before" software).
Even more surprisingly, this statement:
> Managed hosting is essentially running the database for people on AWS so they don't have to. The alternative to using these services is setting up the database on AWS yourself. That's a pain, but it isn't actually that hard. So there is a very hard cap on how much managed database hosting services can charge.
turns out to just not be true in practice. People pay us (Cloud SQL) and AWS (RDS) a surprising amount of money to run MySQL in a VM for them. It turns out that people really like outsourcing even a small amount of pain, but that we're (as engineers) easily blindsided to "Um, why would I pay you XX% over the raw VM price? You're just running some shell scripts for me". Little things like automatically updating to the latest version, configuration replication, etc. are things that nobody will bat at an eye at paying you for (I believe in the DB space in particular, it's because the overall cost of your DB is dwarfed by "All the rest", so it's a good trade of people time versus $$s).
I hope this doesn't come across as Monday-morning quarterbacking, but if I did mine over again I'd take the "Build a great piece of software, and even if it's open, service-ize it for folks". It's true that you're competing with the Cloud providers, and that it seems like you wouldn't really be able to get people to pay you for it (and it's not your core competency), but I think it's the best option for this flavor of infrastructure software moving forward.
> If you're open to edits, the $200k revenue/employee rule of thumb really doesn't make a lot of sense when you're comparing your three alternate business models
Sure, but for many venture funded companies and even large enterprises top-line is the top metric you're focused on. If you're in a growth industry it's about top line revenue, capturing market share, and cost of underlying bits will eventually come down.
> turns out to just not be true in practice. People pay us (Cloud SQL) and AWS (RDS) a surprising amount of money to run MySQL in a VM for them.
For infrastructure providers to "service-ize" you can set cost as a much smaller markup on the cost of the raw infrastructure. Most large organizations (AWS/GCE) have a model where they give some credit for the underlying infrastructure back to the team that servicized things. This is quite a bit different from a smaller startup that still needs to make some reasonable margin but has significant cost of goods of the infrastructure itself.
At the same time I fully agree with the notion that you should build and run a service, but if the marketing or even product perspective of such a thing is we installed a VM for you you're entirely missing the point. Having built an as-a-service product a few times over, there is a lot of engineering and product work that goes into it. Turn key backups, point in time recovery, high availability, performance monitoring are all not just installing a VM. And it's very different from building a core database engine. While there can be many benefits from collaborating on them, they are very different skill sets and the notion of focus shouldn't be underweighted.
To back you up on this, at the last place I worked we ran our own MySQL master and slaves for years, with moving to RDS always on the back burner. Finally completing that migration and shutting down our own servers was a big achievement. Never looked back, and were happy to be paying AWS to take care of the DBs for us.
>>> People pay us (Cloud SQL) and AWS (RDS) a surprising amount of money to run MySQL in a VM for them.
As a happy RDS user, I totally agree. Surely, I could run MySQL and the result would be mostly fine, after some mistakes and lessons learned. But all the things such as upgrading, etc still take time, add a bit of stress, one more thing to care about. Could that time and energy be spent on more valuable (from business point of view) activities? Yes.
I might be wrong by I think you missed maybe the most important keyword: backup. You can't fix a missing/corrupted backup, at this point it's too late. So you need to get it right. It's not like, say, configuring nginx. If you get nginx wrong you just fix it. If you need a backup and it's not there you can't fix it, it's (likely) game over.
So outsourcing the database is a very logical thing to do: if you can trust a third party to have a reliable backup/recovery story, that's one thing that can kill your business that you don't have to worry about.
Nice write up! My outside personal perspective was that you needed to endure for longer. Things like Mongo rode the "Relational is a deadend, NoSQL to the rescue!" wave and gathered momentum before people started fully understanding the consequences of choosing Mongo. Now it seems there's a lot more caution being taken. Anecdotal accounts of people having success with things are taken with a grain of salt. People seem to be looking for more of a tidal wave of acceptance.
As an aside, lately it seems it's "OO is a deadend, Functional to the rescue!" ... so maybe you can leverage that wave instead... or deep learning... OverThinkDB maybe :)
Kudos on a magnificent post, and of course commiserations.
One thing resonated strongly with me. When we started out many years ago someone said to me that the worst market you could go after was to target developers. Case in point - every developer who's ever put up a web site has built or thought about building a CMS. There must be thousands upon thousands of them, virtually all with a price of $0. Who would want to sell a CMS product?
We heeded that advice and I often reflect on it during the sales process when we have the "lets just run your offering past the IT team" meeting. So often, the line of questioning from the smartypants dudes in the room is all about tripping us up, finding our weak spot (I'm a techy so I get it - I think like this myself).
The best you can normally come out of that meeting with is grudging acceptance from IT that a SaaS solution should be OK, given that the business is in a rush and all, and given that IT has other priorities. (To be fair its much less like this these days, but a few years ago this was the norm).
By contrast, showing your product to a business person is a breath of fresh air. They don't want to build it themselves, they don't really care what's under the bonnet, they just want good, quick solutions to their actual problems, so they can go home at night and sit down to Netflix. HR people (our target market) are even nicer.
What's more those business people want to pay for your solution. A quick way to kill the deal is to joyously tell them "its free!". They need to understand how you get your buck at the end of the day. They'll never buy until they understand that. And the best (because its the simplest) answer is "we've got the best product, so we charge good money for it".
IT people try and "trip you up" because we've been burnt so many times in the past with systems that don't do what is promised by the sales people, don't integrate well and who stiff you during the implementation phase.
HR people are some of the worst because everything is super secret and they don't want to get IT involved, then they go and buy a system that lets you insert a carriage return into the phone number field and produces CSV files that don't have primary keys for data extracts.
Everyone walks on eggshells around HR for "compliance" stuff. Anecdotally, I've not see a whole lot of compliant practices in the stuff HR wants. The high end HR solutions tend to enforce it, but there are always amazingly bad home grown tools sitting around to allow you to "pull a list of all our employees and all our data to a CSV" and then you find out the person converted that CSV to a google doc...(I've not seen anything that combined all these steps exactly this way, but I've seen each of these steps. The more legacy the tools, the worse)
And yet a bunch of them chose MongoDB (over something else, not just over RethinkDB.) Was there some point in time when it was the best option for a lot of people?
They were a fairly early entry into modern document databases. They offer sharding/mirroring, and have a really simple, relatively nice interface. In the end, they're a good (enough) fit for a lot of use cases. While RethinkDB imho is absolutely better, market share entropy counts for a lot. ElasticSearch is a similar product in the end, with lower consistency guarantees, that is another good fit for a lot of use cases. Cassandra (C*) is also a good fit for similar use cases, though more difficult to work with, it's also more tune-able to specific needs.
It really depends on what you need... Every non-sql database tends to sacrifice something for some performance gain... RethinkDB is as close as I've seen to one without sacrifice.
Yes, there was a point it was clearly the best option. Qualitative it was easy to understand. The mongo Cli looked exactly like MySQL. And it's single node performance was decent.
CouchDB was super slow. Cassandra was amazing but a bit confusing. MongoDB made sense and worked.
As a former startup founder, I totally empathize - it's always clearer with hindsight but it's important to honestly reflect so you can apply the lessons learned to the next turn of the ferris wheel.
> We set out to build a good database system, but users wanted a good way to do X.
If you replace "database system" with "[any product]," it encapsulates the reason why many startups fail. I was guilty of it too. I think it's a common pitfall for technical founders who tend to think in terms of solutions rather than problems.
If you have some free time, I would recommend the book What Customers Want by Anthony Ulwick, which offers a systematic approach to formulating a product that satisfies a market's underserved needs.
> I think it's a common pitfall for technical founders
There's a very closely related form that is common for non-technical founders which is "We're going to use technology to fix ____ industry"
A founder with experience in a particular industry may see how inefficient it is, and how much scope there is for improvement based on technological solutions, but business live-an-die by the question: "Are there enough people who will pay us to solve this problem for them?"
Superficially, it seems like "I can make your business run more efficiently" is a reasonable approximation for "I have something that you will pay money for", but it has a very similar value-gap to "I can provide you with a technically better database".
"More efficient" and "technically better" are both things that ought to be valuable to customers, but often aren't valuable enough to get them to hand over their money.
Thank you for the detailed analysis & post. I am myself embarking on a startup journey, albeit in the world of consulting & I wanted to let you know that your in depth writeup was extremely valuable & in many ways poignant to me. There were so many learnings in there and the way your bared it all, it feels it must have been excruciatingly hard to let go of your company (& co-workers) in the end. Thanks again & I wish you all the very best for your next venture - wiser & better.
The key to building developer tools is to build the ones that a developer needs but has no interest in creating. Backend for mobile, infrastructure for backend, etc.
Just don't create front end tools for front end developers, for example. Firebug had a donate button and we funneled all of it to volunteers in places where the currency was weak to make it at least seem material.
Kudos on the post and for sharing. I really enjoyed reading the candor and hearing from your experiences. I heard amazing things about RethinkDB and admire your team for what you did.
This was a really interesting post. Thanks for writing it.
First with MySQL and then with MongoDB, a pattern is emerging. There's a popular open-source database. Its feature set is awesome and the ecosystem of software on top of it is good because lots of people are using it, but it's buggy and unreliable. So it's controversial. Since it's popular, it keeps being developed, and over time, it slowly gets more reliable.
To me this post is a reminder that, just because I can see clear flaws with a competing product, it doesn't mean those flaws are exploitable. Better to focus on what your own customers are demanding than on criticisms of your competitor.
> (...) Since it's popular, it keeps being developed, and over time, it slowly gets more reliable.
Or people simply don't care/are not willing to pay for alternatives. MongoDB looks as unreliable as it was on the first day, but somehow people don't seem to care. I hope it's because their use-cases can live with the unreliability, but in fear it's because "won't happen to me, just to others".
Market timing, excitement (some of which later turns into hype), mindshare, and marketing make for a difficult combo to beat. The post-mortem covered addressed market timing and hinted at the rest, but is reluctant to call out MongoDB's marketing.
I can't help but wonder if the story had been different if the RethinkDB devs organized a bunch of codecamps and positioned their db as the 'default choice'. This is how Mongo gained a ton of mindshare, helped by other trends at the time, like the NoSQL wave, the Node.js boom, and the renaissance of client-side JS development.
The problem with MySQL early on was exactly its feature set. Fifteen years ago it was very far behind Postgres yet much more popular. I do not think it even had InnoDB/transactions at that point (or almost no one used it), definitely did not have user defined functions, procedures and many other features developers took for granted in commercial RDBMS.
Like MongoDB, it was popular mostly for being popular, which is a nice gig if you can get it.
I've read the Economist for the last 20 years and the HBR for a few months before I dropped it.
The Economist offers insight, thoughtful analysis and writes about the world: literally but also politics, culture, and science. The obituary is surprisingly fascinating for instance. You may not always agree with its position, but you know that their position will be worth engaging with and thinking about.
No, I don't work for the Economist, I just appreciate thoughtful articles. Same goes for Private Eye, Foreign Affairs, New Yorker.
Did you ever consider partnering with a larger vendor?
Figuring out the right GTM is tough, and the timing was weird, but GOOG and MSFT must have been pretty desperate for a NoSQL solution 2013-2014 to compete with DynamoDB. Firebase got scooped up in October 2014, but it wasn't until April 2015 when Azure announced DocumentDB.
Sorry i am the one who leaked .
@coffeemug we greatly appreciate your effort and rethinkDB is well architectured database in existance. We gonna keep using it and contribute to Rethinkdb .
It seems like you have a great ability to self-evaluate. Many of my friends and I, after reading your post, agree that we would be very willing to work in a team managed by people like you.
While I feel for you because failure hurts so much, congratulations on learning from your experience. I know this won't make you feel any better, but the next time you try things will be so much easier.
By the way the tone is fine given your experience.
I understand that you're waiting for a legal framework to accept donations as an organization, but there was an article recently about an individual GIMP developer being funded on patreon. Is this something you would consider? I'd be happy to contribute to someone who's actively working on RethinkDB (of course without any backer rewards or similar expectations), and perhaps others would as well.
Out of all the databases I tried throughout the years RethinkDB was the best. His bit about "worse is better" nails it I think. It was an elegant product targeted at the wrong group of people. ReQL alone was a masterful piece of design and it was great fun composing queries in Ruby instead of some weird JSON frankenstein DSL. At one point I was pushing 500k events per hour through a single node on an c4.xlarge instance with plenty of room to grow. I'm sure other databases would have handled the load just fine as well but I didn't really want to find out because RethinkDB was so much fun to play with. I was very sad when I heard they were shutting down.
The product still lives and development is continuing slowly. There is almost daily commits there. We need more contributors!
@atnnn ( ex rethinkdb developer) and community is working on it.
https://rethinkdb.slack.com/messages/open-rethinkdb
Man, I would have loved to pay for RethinkDB. Granted, the blame lies on me here, but something I was disappointed to see the first point not acknowledge that RethinkDB didn't really ask for money. I mean, I don't know if they changed it after folding, but go the site and tell me how to pay for it.
I would have paid a licensing fee to continue using it, but we've taken the time to switch over to Postgres now.
EDIT: Also, re: metrics of success, the examples listed as wrong there are _exactly_ why I enjoyed the product and was willing to invest time learning and implementing in a relatively new technology.
On the other hand, I understand why most developers wouldn't pay "even the price of a Starbucks coffee" for the product: because paying anything as a developer is hard. It requires making the case for why you need it to other people in the company. Developers aren't usually in a position to make financial decisions on a whim, especially those where the cost will probably grow in the future.
I think this is right on the money. As a vendor, you have to understand who your users are and who your customers are. In this space, they are different. Just look at the number of developer tools that are "free for non-commercial use". Developers don't pay for tools. Companies pay for tools that they are convinced they can't get any other way.
They didn't have a poor market; they didn't have any market at all. If I'm the CTO of XYZ, explain to me what I'm buying. I hate to keep coming back to it, but that's what Cygnus did so brilliantly: "You need an embedded development system. The existing solutions do nearly what you want. Pay us slightly less, and we will make GCC do exactly what you want." I know what I'm paying for.
"Build it and they will come" does not work in the open source space -- because if you have already built it, they don't need to pay you. And if you haven't, then they don't want it.
Now, if they had built a DB similar to Mongo and then sold a service that migrated people off of failing Mongo installations and onto RethinkDB, then they would be on to something. Tired of having your DB chew your data for breakfast? Want to have your reports finished the same day you ran them? For the price of a single developer, we will fix all your problems by migrating you to RethinkDB. That's something that will make money.
But who wants to charge for services? How will we get our multiples if we only charge once for our work? That's just folly!
Why not position yourself as the better mongodb? You explained pretty well in the post mortem (and to yourself in 2014) how much better rethinkdb is to mongo. All you had to do was aggressively market yourself as the better option by directly comparing yourself to it.
I remember when iPhone first launched, the first 30 min was spent just trashing then-current smartphones - the blackberrys.
I think he did mention this indirectly. Mongo performs better than Rethink on metrics that have nothing to do with what you truly want from a DB. He mentioned developers that would just whip up a script to throw thousands of writes at the database but never reads. Use cases that they would never see in real scenarios. First, you would need to educate your market that they need to care about the CAP theorem and what that is, that Rethink was damn good CP and that while Mongo might be fast, it isn't correct.
Why not call them out then? It should take only one or two blog articles to explain why mongodb benchmarks are bogus. I fail to believe it is hard to teach some database knowledge (CAP and others) to your potential customer when they are developers. Heck, it would even boost your SEO by a ton.
The database will never sell itself. You need use cases - even Oracle sells sofware solutions for ERP, CRM and SCM which come bundled with a Oracle database. Microsoft SQL server is another example of bundling. So use cases come first before correctness, consistency or excellent SQL.
It's hard to sell something as low level as a database. For a startup there needs to be a specific itch to scratch, an itch that many potential customers experience frequently.
As a systems-guy (more or less) I find that frustrating, but I have seen many times that this is so.
It still worth having the courage to try and these guys did that. It could have worked. Hats off.
What was the actual go to market strategy? Selling to enterprises? They have the money to spend but it seems like there was never an enterprise version of the software itself with features that are paid only. This is the high-margin sales that database companies build on, and then perhaps offer cloud and managed service to capture more spend. Support plans can also be great revenue, although they're not a sales driver but rather just required for big companies and likely break even for smaller ones.
That being said, it's an interesting database and I'm still looking for a solid distributed OLTP document-database with SQL/joins and multi-cluster replication... maybe someone else will try again.
I was told (by their lead engineer) a few weeks before RethinkDB shut down that they did have an enterprise version of the software with features that are paid only. However, clearly, that product introduction was too little too late.
I do find it truly sad how anti-intellectualism is pervading our industry and causing good products to die. Nowadays, a significant number of the hottest new technologies offer literally nothing of value beyond "low learning curve". It's my current belief that tech culture has lost nearly everything that once made us great.
Think about some examples.
MongoDB is nothing but the object databases of the 80's, rebranded. It's main selling point is "you don't need to learn SQL". Object databases failed and people at the time were willing to learn something new (SQL). Nowadays, who will do that?
Go's entire selling point is "faster than python, no need to learn anything new". In terms of language features it's again living in the 80's - Java got generics in 2004 and Java was way behind the times. In terms of speed it's worse than Java and Rust, maybe comparable to Haskell.
And NodeJS is nothing but "write server code, no need to learn a new language". Isomorphism simply didn't work. People think Node is fast because it can do async, but Java has been able to do threaded async for 15 years.
I wonder how much the industry could advance if we could simply persuade every developer to learn Java Postgres.
Interesting idea, but perhaps it's also possible to look at it as the industry broadening. 40 years ago, you had to be pretty knowledgeable to get anything at all done with computers. These days, it's possible for more people to do things, and yet there's still a market for people who are deeply knowledgeable.
A lot of people don't care much about the tech, and just want to build something. If easier (for them) tech lets them, that's a win, no?
Of course... that's easy to say but harder to swallow when they end up building something, making some money, and then hiring a developer to clean up their hideous mess. I've been there too.
I'm going to fundamentally disagree with at least the target of Go. Go introduced a series of concurrency primitives with syntax that's made writing and reasoning about concurrent code a joy, something that's not the case in Java/C++/C/Python (maybe up till the late 3.X in the case of Python), and it did that with a language paradigm (interface driven) that's vastly different from Python's mixture of OOP and duck-typing.
And sorry, but "I wonder how much the industry could advance if we could simply persuade every developer to learn Java Postgres."?
Plenty of other people would say "I wonder how far along the industry would be if you people would actually learn how to manage your memory and write C".
Plenty of other people would say "I wonder how far the industry would have progressed if we let go of the harmful and outdated concept of imperative programming and embraced functional paradigms."
... and on it goes.
-----------------
Additionally, you say "I do find it truly sad how anti-intellectualism is pervading our industry" only to end your post with "why can't we abandon all our experimenting and learning and just stick to good old Java which obviously would solve all our problems", which is itself MASSIVELY anti-intellectual.
> "why can't we abandon all our experimenting and learning and just stick to good old Java which obviously would solve all our problems", which is itself MASSIVELY anti-intellectual.
That's a mis-characterization. There is a difference between experimenting and learning and pushing all sorts of questionable fashion trends into production.
While true, the "pushing" being done is a very small compared to only the "development" of said technologies. The exact words of the parent where:
> I wonder how much the industry could advance if we could simply persuade every developer to learn Java Postgres.
And that is preceded by what I can only described as a bunch of undue mischaracterization of several languages and technologies, all of which definitely seems to imply that "yeah, all that other stuff is just a bunch of fads, why don't we stick to the real deal and all write Java."
Now, I don't want to be unkind to Java here. It's true that Java is an incredibly powerful language and there are a wide variety of technologies and tools that are absolutely incredible that you can only get via the Java ecosystem. But that can be said of many languages, e.g. C++/C.
I think what the GP may have meant, and which I do agree with, is that our industry would progress if we as developers spent a bit more time learning about the context and history of current and past solutions. We'd probably re-invent the wheel a bit less often.
But I don't want to limit that recommendation to "learn Java Postgres". I think Java developers might benefit from examining the benefits of a language like Haskell, Python programmers may benefit from a language with strong typing, such as C++/Rust/Go, that Unix lovers might learn a lot from examining the Windows NT kernel API's, and so on. If we examine the things that aren't close to us, the things with which we're the least familiar, that's when I believe the most fruitful learning can occur.
I took the GP in a very figurative way as well. A lot of the learning being done at the moment though is simply framework churn, there are developers out their familiar with a million web and client side frameworks that would have been much better served learning SQL. Our apps would be better if we focused more and learning about usability and our users than on CSS compilers. As an industry I think we're under serving everyone right now, people have machines a thousand times more powerful than a couple of decades ago but what they can do with them is almost identical.
Go didn't "introduce" CSP or actor like primitives. Scala had actors in 2006, I think Clojure has for a similar amount of time as well. There have been libraries for it in Java and C++ also since the early 2000's and probably earlier.
Plenty of other people would say "I wonder how far along the industry would be if you people would actually learn how to manage your memory and write C".
I don't see the industry gravitating towards bad re-implementations of C with less functionality and a flatter learning curve. The only "replacement" for C that I'm aware of is Rust, and Rust is most definitely not anti-intellectual in the same way I criticize Go.
Rust actually requires the developer to learn new concepts (borrow checker) which are not found in C, Python, Java, etc.
"why can't we abandon all our experimenting and learning and just stick to good old Java which obviously would solve all our problems", which is itself MASSIVELY anti-intellectual.
I didn't say to end experiments. I said everyone should actually learn some fundamentals before jumping on low-learning curve bandwagons which offer nothing new.
Don't forget the proliferation of electron so no one has to learn a new language and desktop toolsets.
Just for kicks I wrote a desktop todo list in GTK and C, not tools I had extensive knowledge of ever and nothing recently. It was a breath of fresh air how simple it was compared to doing the same in modern frameworks. A single ~30 line function to create the gui and the rest held together with a couple of function pointers. It was faster (to write and run) and simpler than anything I'd written in years. More responsive than the most responsive web framework too.
But is it cross-platform (Win32, Windows UWP, OS X, multiple linuxes, iOS, Android) and most of that same code works on the web?
It's okay to bemoan Electron for being bloated but multiple balls have been dropped by OS and platform makers, GUI toolkit makers, language runtime makers, and that giant amount of missing glue inbetween, to get to where we are.
It's rather sad that Electron, out of all things, is the first thing to fulfill the holy grail of write-once, run anywhere, in a way that's good enough and palatable to the plurality of users' and devs' satisfaction.
Electron certainly isn't the first thing to give "write once, run anywhere", let alone good enough to satisfy users. I groan every time I download an app only to discover it's Electron based.
Java beat Electron to write-once-run-anywhere by a couple of decades, and still does in the sense that Electron requires you to make platform-specific builds of your app even if the core code is the same (Java gives you that option but doesn't require it, you can still distribute a jar or web start file).
Meanwhile, Electron has managed the feat of being even bloatier than Java is. The DOM was never designed for GUIs and the terrible performance of web and Electron based apps is a testament to that. At least Java apps tend to have meaningful menu bars and context menus.
Electron satisfies web devs who have either never written desktop apps in other frameworks, or did decades ago and think nothing improved there. They "satisfy" users in the sense that users are rarely offered any alternative so have to suck it up and tend to judge web apps against each other vs a well coded, tightly written desktop app (office suites being a notable exception).
Yes, I used GTK2 which is cross platform (GTK3 isn't). Would work fine on Win32 (the only windows worth supporting), OSX and multiple linuxes. Won't work on android or iOS, but IME that usually sacrifices the dektop app anyway. This case would have been fine though if it were possible, the app was responsive enough to work on a nokia 6110.
The biggest takeaway though is how simple it was to build. If all platforms where that simple then having seperate UI code for each platform really wouldn't be that big a deal.
At the moment I'm somewhat hopefully the libui project (https://github.com/andlabs/libui) will be successful though. It's learned the lessons from previous attempts.
I half wrote a blog post/angry rant over Christmas. I'd post the draft/source here, but my c is very rusty and I want to at least check for memory leaks first (no valgrind on cygwin) ;)
Intellectual is the wrong target, just as much as easy. Humans have a limited capacity to understand complex systems, and once they grow to a certan point, your ability to integrate and innovate on them is limited due to overhead of the layers of incedental complexity. Easy is just shoving complexity under the rug. Simplicity, and building systems meant for our meager human intelligences is better.
I am not terribly fond of Go, but it has a decent answer in auto-implemented interfaces and duck typing. It's not about dumbing down, it's just different, and easier on the compiler, which is a significant advantage.
And as for async in java... ugh, how awful. We finally got something like promises and lambdas, but most APIs are still trapped in synchronous calls and using the JPA is a disaster of hidden thread-local state. What kind of stack are you using to make java async work? Maybe I'm just missing something.
The thing is, the modern systems are often less complex. SQL databases are generally far simpler than MongoDB - you just need to learn relational algebra rather than blindly shoveling whatever JSON you got into the DB.
As for Java + async + threads, I use Akka and it's basically seamless. But Netty + core Java Executors is fine too, just more verbose.
> you just need to learn relational algebra rather than blindly shoveling whatever JSON you got into the DB.
No
SQL per se is not hard.
But the whole data modeling, creating tables, picking types is just needless bureaucracy
Yes, I want to save a json snippet and be done with it. I don't want to create another table because I have a 1-N relationship. And while ORMs make this easier it is still not painless
Schema-less data always sounds good in the beginning until you actually try to use that data. Then you end up encoding all the validation and data munging logic into whatever process reads from it.
An internally-schema'd database just recognizes that this is absurd and lets you put the bottom three layers of validation logic inside the database, where you only have to do it once.
> Then you end up encoding all the validation and data munging logic into whatever process reads from it.
Do you think that doesn't happen in regular DBs or that schemas don't evolve?
Love for static typing is basically Stockholm's syndrome
Your system will eventually end up with a fixed schema, but until you get to that place you don't need to "stop the world" to change it. "But Postgresql can add columns without downtime" yes it can, but what about your system?
> Love for static typing is basically Stockholm's syndrome
Funny, that's sort of how I feel about my (long) time "loving" dynamic typing. Now I find it is such a relief to validate data once on the way in and completely trust it to be what it says it is thereafter. I found dynamic types required more bureaucracy (if I wanted to sleep at night) in the form of unit tests and precondition checks everywhere.
> I found dynamic types required more bureaucracy (if I wanted to sleep at night) in the form of unit tests and precondition checks everywhere.
This merits clarification
Most of programming in dynamic typing should be implicitly statically typed. Know what you're passing. Obey the implicit (duck-typed) interfaces
Basically, don't just check "if it's a list do that, if it's one element do something else, if it's a number do another thing" - this is a beginner's mistake (and I did those)
Besides that, unit tests are a good idea, and if you want "compile type checking" in Python use Pylint (it's a good idea to use it regardless of your situation).
Though it's not so much static typing that sucks, it's its implementation in the Java/C++ way. Go and Rust make it better.
> Most of programming in dynamic typing should be implicitly statically typed. Know what you're passing. Obey the implicit (duck-typed) interfaces
Yeah, I thought that too. After years of persistently seeing stupid (sometimes critical) bugs due to assuming the correct thing was being passed, I concluded that 1. the only sane thing to do is add explicit precondition checks that raise meaningful exceptions when assumptions are broken, and make sure those paths are exercised by unit tests, and that 2. I would much rather have that done statically and automatically by a compiler.
> Basically, don't just check "if it's a list do that, if it's one element do something else, if it's a number do another thing" - this is a beginner's mistake (and I did those)
I think this may be a misunderstanding of what I meant by "precondition checks" - I meant explicitly checking the assumptions being made by a method, most of which are type checks (foo.kind_of?) or (more commonly) interface checks (foo.respond_to?).
> Though it's not so much static typing that sucks, it's its implementation in the Java/C++ way. Go and Rust make it better.
I don't have much problem with Java's implementation of static typing at all, and I don't think C++'s problem is its static typing implementation. I find Go's implementation a bit inflexible and hard to work with though.
Akka doesn't solve missing asynchronous libraries. You still have to wrap JDBC requests in a threadpool, and are limited in how many database requests you can serve by pool size.
I just want to nitpick about Node a little bit--as a Java developer and Postgres user myself--because there is more to Node than just async JavaScript and I think it gets a bad rap a little bit because of that. Async is one good quality of Node, but it also has this really great debugging integration with Chrome DevTools. And even really huge Node programs start fast--usually in less than a second--so you get this really tight development feedback loop. So you have this overall pleasant experience of a great module system where mostly everything out there is already going to be async (less likely in Java), a REPL, a fantastic debugger and profiler via the Chrome DevTools, and extremely quick process restarts. I think for some people these qualities outweigh the lack of static typing and JavaScript language quirks (which a lot have been ironed out in ES6 and ES7 anyway).
It's not anti-intellectualism -- it's complete and utter lack of knowledge transfer, coupled with young people's spirit being dampened by established practices that are 50% reasonable, 50% bullshit.
This, piled on with subtle ageism, ensures that any useful lessons learned will only be learned by the time it's too late, and there's a new group of can-do people who feel stifled by the combined wisdom of the ages, so they break off and do their own thing.
I agree for most part but what is important is productivity.
Java ecosystem is just toxic waste full of abandoned projects, over engineering everything, crashing build because everyone use arbitrary versioning schemes, ring-fences code with licenses/patents and developers with C++ mindset.
Node.js is maybe build on top hack on top of hack. With some ugly code, lack of any abstraction, lack of sensible std library, debugging capabilities, minimal editors support and finally with package manager that cannot be trusted.
But I can build CRUD API working with any database under 1h in node.js. Where In Java I need to start with learning all maven quirks, write pointless JSON mappers and learn stupid annotations that every library introduce.
Go's entire selling point is "faster than python, no need to learn anything new". In terms of language features it's again living in the 80's - Java got generics in 2004 and Java was way behind the times. In terms of speed it's worse than Java and Rust, maybe comparable to Haskell.
This is not true, go is generally slightly faster than java (while using less memory_ and slightly slower than rust. The garbage collector is slower than java's default collector but has better latency but java's gc can be tuned and go gives you better tools to bypass gc partially or entirelly.
- Excellent tooling and integration with programming text editors
- Great documentation
- Excellent standard library which you can treat as a continuation of your own program
- Great design with pragmatic choices (this one is subjective, I agree)
In short, there's nothing else like that out there.
Without Go, I'd have been stuck with JVM, Haskell or Erlang for most of my projects. And I dislike all of them, so Go has been a godsend.
Incindentally, I also happen to enjoy working with Postgres and prefer it to any other database I've worked with (SQL or not).
I can open a standard library definition and quickly understand what it does. If I try to do that with Scala or Java, my eyes just glaze over. With Go I often don't notice when I venture outside of my own program and into 3rd party library code.
It sounds like you just didn't out in the work to learn Haskell. The docs are fantastic, but they do assume you are willing to get over the learning curve.
I don't think you are really disagreeing with me - go does nothing new, just lower learning curve.
That's the problem, there's no getting over the learning curve when it comes to languages like Scala or Haskell. While you do get more proficient with time, you never get to the state where the language gets out of the way and becomes your friend instead of something you have to wrestle most of the time to enjoy the benefits it offers (which are also offered by Go, BTW).
I was disagreeing with your original statement, Go is not just a faster python.
> That's the problem, there's no getting over the learning curve when it comes to languages like Scala or Haskell
For you, maybe. For plenty of others, that's clearly not true.
> While you do get more proficient with time, you never get to the state where the language gets out of the way and becomes your friend instead of something you have to wrestle most of the time to enjoy the benefits it offers (which are also offered by Go, BTW)
Go offers weaker type-safety benefits that Java (with the benefit of less type-safety ceremony), much less Scala or Haskell (against which it has less ceremony benefit, because Haskell and Scala have reduced ceremony with stronger type systems).
Go doesn't offer the benefits Haskell or Scala (or Rust) do. OTOH, some people find it more intuitive, and it does offer different benefits which may be more relevant to some use cases. The sweet spot for Go seems to be the place where, before Go, you might use Python but be upset about performance and might use C but be upset about boilerplate; Go improves over the weaknesses of either on that border, while preserving most (but not necessarily all) of their strengths.
> Go isn't just a faster Python.
Faster (and more parallel) Python, less obtrusive C, lower ceremony and more native Java -- it's not just any of these, but they sort of capture its primary strengths.
Thanks for providing an anecdote which 100% supports the claim I made: that all Go provides is an inferior language that appeals to people who don't wish to learn new things.
NoSQL's only selling point is you don't need to learn SQL? Hardly true.
Major Oracle shops aren't dumping them for Mongo/Couchbase/... because they're trying to avoid SQL. They do it for scalability, availability, performance, flexibility and more.
I don't see that the facts in the real world back you up here.
Not claiming NoSQL is a panacea. You're definitely making some trade-offs, and in many cases it's just the wrong tech. But it's absurd to say it's happening to avoid a learning curve.
It's much lower barrier to buy managed database service from Amazon or Microsoft than from a start-up.
They have the credibility. You don't question if these companies really have the people to keep the stuff running. In many cases the customers also already have a billing relationship with them. Much less friction to just add one new service to the already quite big invoice vs buying from a different company.
Competing with AWS or Google is hard, they would be still making money on the infra even if they gave the managed database out for free.
Rethinkdb is an incredible architecture. If I was in the market for something like rethinkdb/dynamodb, I would completely go with something that is built by the original inventors.
269 comments
[ 2.0 ms ] story [ 266 ms ] threadFirst of all, huge respect to Slava for this writeup. Having your startup fail is hard and it is not the time you want to blog about it. RethinkDB going broke was a sad thing to see for me, I can't imagine how it felt for him.
I think the analysis of the two root causes (hard market, focus on the wrong metrics) is accurate. It is very sad to see that in the end correctness doesn't win the day, not even for databases.
Since I run a startup too I can't help but apply his criteria to GitLab.
Good metrics to focus on:
1. Timely arrival => we try to ship great features every month on the 22nd, something that both our users https://twitter.com/PragTob/statuses/767777202045915136 and the parody account likely run by our competitors employees agree on https://twitter.com/gitlabceohere/status/768440048802947073
2. Palpable speed => we're doing OK on self hosted, really bad on GitLab.com (fixes are in https://gitlab.com/gitlab-com/infrastructure/issues/947 ). If you look beyond latency but to workflow we're doing great in integrating various parts of the process https://about.gitlab.com/2016/11/14/idea-to-production/
3. A use case => we're making GitLab an integrated tool with a broad scope https://about.gitlab.com/direction/#scope A good way to develop software from idea to production.
I hope the above list doesn't come across as pretentious. I welcome pushback and the opportunity to talk more about the OP.
I think the OP premise that it is hard to make money in the cloud is accurate. We're spending hundreds of thousands of dollars to make GitLab.com run and revenue takes a long time to grow. Selling on-premises software products has higher margins than a service.
BTW I appreciate the shoutout to GitLab as one of the five exceptions that are doing well in the open source developer market.
1 thing we are always transparent with users about is how our business model works. 1 of my favorite videos on the internet about this is [1]. We are very much based on closed source for making revenue. This has actually helped with our community a lot.
We are basically focused on closed source solutions for specific verticals. The only way to get clients to pay is to make something more efficient or don't open source/core it. For open source companies there is always a trade off of community vs customers. You have to find a hard line and stick to it.
Community is great, but most of those people won't want to pay you. You can try to convert them, but that takes a lot of time and isn't the best path to market. We strive to provide a great base line experience to our users, but a few of the things we have found in practice that helps:
1. Features have customer names on them
2. There's a mass of community interest in a specific feature
You have to be careful about where time is spent. It's very easy to mistake "users" for "customers".
I wish slava and co the best of luck and really appreciate their post mortem. 1 thing we've learned as an infra company is "be as close to the app as possible". In our case the bulk of what we focus on is banking/telco anomaly detection.
It's a straight forward revenue making product that can benefit from deep learning. The great thing is we can grow in to other use cases later on if we find something else interesting to work on.
[1]: https://www.youtube.com/watch?v=6h3RJhoqgK8
Even seen this backfire at many commercial companies. Adding whatever features a company is willing to pay for will often pull a product in multiple, incompatible directions.
That actually goes hand in hand with "users look like customers". Also: "Not every customer is a good customer."
This to me explains the massive success and quick rise of things like redis, mongodb etc. Sure the defaults are insecure but they work and they work immediately.
Postgres is the default datastore (because schemas are awesome) for me, and I haven't had a use case yet where I needed something that Postgres wouldn't do. Maybe if you have a very specific need, you'd reach for another datastore (come to think of it, I have successfully used Cassandra for multi-DC deployments of a distributed read-only store), but that's not the norm for me.
v9.6 finally brought some parallel scans but is still limited in scope and far behind the commercial databases. Same with replication and failover, although there are decent 3rd party extensions to get it working.
I wasn't able to set it up properly some years ago and settled down with warm standby replica (streaming + WAL shipping) and manual failover. Automating it (and even automating the recovery when old master is back online and healthy) was certainly possible - and I got the overall idea how it should operate. But the effort required to set it up was just too big for me, so I decided it wasn't worth the hassle and settled down with "uh, if it dies, we'll get alerted and switch to the backup server manually" scenario.
Having something that goes in line of "you start a fresh server, tell it some other server address to peer with, and it does the magic (with exact guarantees and drawbacks noted in documentation)" would be really awesome. RethinkDB is just like that. PostgreSQL - at least 8.4 - wasn't, unless I've really missed something. I haven't yet checked newer versions' features in any detail, so not sure about 8.5/8.6.
It offers easy commands to perform failover and even has an option to configure automatic one.
After reading about github issues[1], I am a bit cautious about having automatic failover though.
[1] https://github.com/blog/1261-github-availability-this-week
As for automation... Things can always go wrong, sure. But I wonder how many times HA and automatic failover had saved the day at GitHub so no outside observers had a faintest idea there was something failing in there.
This is the one thing I ask of every database software, yet we still don't really have it. 90% of problems could be solved if there was a focus on the basics like easy startup, config and clustering.
SQL Server availability groups requires Windows Server Failover Clustering, which is not quick or easy.
Horizontal scalability is important for HA which every production environment would like or need.
Here how it compares: http://www.datastax.com/wp-content/themes/datastax-2014-08/f...
The only values that has higher than rest are on page 13, except those are latency and lower is better.
This paper shows clearly that Mongo doesn't scale.
And even for single instance Mongo is slower than Postgres with JSON data: https://www.enterprisedb.com/postgres-plus-edb-blog/marc-lin...
There are actually add-ons to Postgres[1] that add MongoDB protocol compatibility, and even with that overhead Postgres is still faster.
And even such benchmarks don't tell full story. I for example worked in one company that used Mongo to regional mapping (map latitude/longitude to a zip code and map IP address to ZIP). The database on Mongo was using around 30GB disk space (and RAM, because Mongo performed badly if it couldn't fit all data in RAM), mainly because Mongo was storing data without schema and also had limited number of types and indices. For example to do IP to ZIP mapping they generated every possible IPv4 as an integer, think how feasible that would be with IPv6.
With Postgres + ip4r (extension that adds a type for IP ranges) + PostGIS (extension that adds GEO capabilities (lookup by latitude/longitude that Mongo has, but PostGIS is way more powerful) things looked dramatically different.
After putting data using correct types and applying proper indices, all of that data took only ~600MB, which could fit in RAM on smallest AWS instance (Mongo required three beefy machines with large amount of RAM). Basically Postgres showed how trivial the problem really was when you store the data properly.
[1] https://github.com/torodb/torodb
When you hit those edge cases or specific needs, you implement the specialty solution. Most people never come close to those edge cases though. When they do, PGs foreign data wrappers make the adjustment as smooth as possible.
Because of single threaded query execution, lack of MPP, or something else?
The truth is that NoSQL has great promise, but the folks with actual money to spare today are the ones who want a better SQL database, because they can afford the tradeoff of development time for reliability that SQL administration and migration provide. And people who solve the SQL pain points, such as the excellent folks at CockroachDB, and the various folks building replication layers on top of Postgres, can hopefully keep the market for database startups alive long enough for someone else to fix the NoSQL landscape as well.
[0] https://aphyr.com/posts/284-call-me-maybe-mongodb , where one finds that Mongo actually isn't guaranteed consistent during partitions, actually did hit us in production, and we had to dig through a rollback file manually (shudder); fun times. On the other hand, see: https://aphyr.com/posts/330-jepsen-rethinkdb-2-2-3-reconfigu...
Rethink definitely had its use cases, I just never saw it as my primary data store.
If it doesn't meet your needs please file an issue, our goal is to remove the no-good-GUI limitation that was imposed on postgres sixteen years ago.
Hell Python Tools for Visual Studio almost makes me want to do python dev work on a PC.
https://eggerapps.at/postico/
End result is that I jumped on the MySQL bandwagon, and even all the negative posts against it now and the lauding of Postgres in here all the time cannot make me change tracks. I just have too much data and IP invested in it now.
He says that the market was terrible because open source developer tools suffer from oversupply. I don't think that's what happened. Rather they chose to make a product for which the next best alternative (e.g. Mongo or Cassandra or MySQL or Postgresql) had almost zero cost. It doesn't matter how oversupplied the market is if someone is willing to give away a viable product for free. Perhaps "oversupply" is another way to say "someone was willing to sell at zero cost", I'm not sure.
Unless you can find a set of customers who see significant value in your product vs. the alternative zero-cost product, you're never going to make money.
I agree with most of your comment, but I take issue with that last part. Moore's law has no relation to the price of computing. It only makes predictions on the number of transistors in a CPU, not the performance of the CPU nor the price/performance ratio of the CPU. Even if it did, compute costs much more than the CPUs; memory, disk, power, cooling, networking, enclosures, and more all cost a significant amount relative to the silicon.
I know Google Cloud declared they were going to decrease prices at pace with Moore's law, but even if they can deliver on that it will be through innovations outside of the CPU.
This world is always willing to wait, it was the RethinkDB team and/or their investors who weren't. The world will wait for you in eternity; it's just not willing to provide you with capital during the wait.
Your statement makes sense if there were a dozen competitors in the database space. With hundreds of competitors, actually evaluating them all becomes more expensive than just choosing eg. a managed SQL solution and then switching provider if it doesn't live up to its promises.
That's so much more flexible from a business perspective than evaluating a hundred different open source database systems, settling on one, and then having to rewrite your database code when the database you've chosen ceases development, and you're left as the maintainer of it.
https://www.dreamsongs.com/RiseOfWorseIsBetter.html
I can only imagine what the team at Realm is thinking after going through comments and the post-mortem. Realm was an open-source mobile first database for the past 3 years. It's free. Its the most correct, consistent, and simple mobile database that is running on millions of devices[1] (Yup fine, kind of my opinion but trust me mobile community will back this up). They ship FAST, they ship real USECASES. They just recently entered the cloud space with their Mobile Database Platform, and they are even experimenting with new things like a version built for Node.js.
I'm honestly not trying to make a specific point, but asking more questions... Is mobile different? Is it easier? Are mobile developers more willing to try new technology? Are they more willing to pay for technology? (Realm is free, haven't paid them a penny, so not sure about this one...)
My main point I guess is that it's not a terrible market, Realm made it work with VERY similar ideals and goals. So what else is going on here?
[1] https://realm.io/news/jp-simard-realm-core-database-engine/
Yes, mobile in this context is very different. Or rather, local is very different. Building a database that resides on a local device and is primarily accessed by a single user is an entirely different class of problem than one that is on a server. Usually most of the hard problems in building a database system involves dealing with multiple people accessing or modifying data at the same time. When you have a single user, that problem either disappears or is massively reduced.
As you said, Realm does appear to have a server offering now, but from the sounds of it, it hasn't really been proven yet, so I don't think it's as easy as saying they've succeeded where Rethink failed.
It's easier to get adoption for mobile, I'd think.
Database component designed for use by mobile applications: something you base an app around
Database server: something you base your business around
Furthermore, mobile is becoming in-demand or even core in both consumer and enterprise. For enterprise especially, though, you have to think of it as mobile + sync, not on-device only.
Turns out sync is the really hard part when you want to build something that's robust in the face of poor network connectivity.
They've raised a roughly similar amount of VC money as RethinkDB did. Their last raise was a few years ago, before Parse's shutdown, so I suspect that investors are more antsy about this space today.
The fact that Realm recently entered the cloud space sounds similar to RethinkDB's trajectory too... Of course I hope Realm makes it, it's a great product. But I'm worried that they're trapped in the VC-funded model like RethinkDB was.
https://blogs.microsoft.com/microsoftsecure/2007/08/23/the-e...
He details why they put marketing and shipping first with constant improvements on correctness or security. On top of this, I advise just embedding the benefits into the product & designing it for easy change so components can be replaced once big revenue comes in. As development pace slows, QA pace can increase on components stabilizing.
So I feel Slava's pain (and feel somewhat culpable as a user of RethinkDB), but I don't entirely agree with the analysis. Yes, infrastructure is a brutal space because it has been so thoroughly disrupted by open source -- but the dominance of AWS shows that open source can absolutely be monetized (just that it must be monetized as a service). Indeed, right now, I would love to deploy a RethinkDB-based service -- and if the company still existed, we (and by "we" I mean "we Samsung", not "we Joyent"[2]) would potentially be in a serious business conversation about what a supported, large-scale deployment would look like. Actually, that's not entirely true, and it leads to me to one thing that Slava didn't mention. While it kills me to bring it up, it does represent the single greatest impediment I have personally found to deep, mainstream, enterprise adoption of RethinkDB: its use of the AGPL. Anyone who has been part of a serious due diligence knows that the GPL alone is toxic to much of the enterprise world -- and the GPL is a buttoned-down corporate stooge compared to the virulent, ill-defined, never-tested AGPL. The AGPL is a complete non-starter for any purpose, and while I appreciate why the AGPL was selected by RethinkDB (and very much appreciate the explicit riders that RethinkDB put on it to make clear what it did and didn't apply to), the reality is that no one -- absolutely no one -- has built a business on AGPL software and it seems vanishingly unlikely that anyone ever will.
I still use RethinkDB and still intend to -- and I still harbor hope (crazily?) that whatever entity that owns the IP will see fit to relicense it to something that is palatable to the kind of people who care about their data. I actually believe that a business can be built on the technology -- which, to Slava's points, I have found to be delightfully sound in both design and implementation. And that RethinkDB is open source means -- for us, anyway -- it will continue to live on, despite the unfortunate demise of its corporate vessel (AGPL or no). That said, I would love to have a lot more company -- here's hoping that we see RethinkDB relicensed and a thriving business behind it!
[1] https://github.com/joyent/manta-thoth
[2] https://www.joyent.com/blog/samsung-acquires-joyent-a-ctos-p...
MongoDB server is AGPL - client drivers are Apache. To be fair the business side is based around larger scale deployment and management, and is proprietary add-ons.
I'm kind of surprised that approach wasn't mentioned for Rethinkdb - a free open core product, with paid extras dealing with pain points in larger deployments.
Nothing new, just a strange place to be in.
Now you get to innovate on top of these modules using reactive APIs, user space TCP, Fancy serialization, Standards compliant SQL etc.
Some of the current incumbents were written in the 90s when C++ was a different programming language, which hurts innovation.
[1] https://github.com/facebookincubator/iterlib
Currently the pre-packaged databases on Joyent don't support both clustering and consistency. Is there anything in the works for this segment?!
source : https://docs.google.com/document/d/1f8qODp7voKIqwioQ3si69q3-...
I strongly disagree with that. Take RavenDB, which is very much comparable to RethinkDB in terms of a business is dual licensed as AGPL and has a commercial license available [1]. This forces any business doing non-OSI approved OSS work to purchase a commercial license. I don't know too much about RethinkDB but I think they must have used a very similar licensing strategy?
From an outsiders' perspective, RavenDB appears to be going strong and doing well commercially. They are hiring and expanding and have invested a lot into v.4.0, which I'm really excited about.
Interestingly, on the spectrum between MongoDB and RethinkDB in terms of "Wrong metrics of goodness" (Correctness, Simplicity of the Interface) they appear to me much closer to RethinkDB than MongoDB. In terms of stability, they may have been somewhat in the middle (e.g. not as bad as MongoDB, but probably not as good as RethinkDB). And they have moved very fast to add features and fix bugs in the product. It's a really nice DB to work with as a dev, I suppose the #1 reason they are not as successful as Mongo is that they're coming from the .NET niche...
[1] https://ayende.com/blog/4508/comments-on-ravendb-licensing
The article talks alot about big companies that make lots of money but as a user of databases I always saw the competition to ReThink DB being Postgres, MySQL and to a lesser extent ArangoDB and MongoDB.
The core problem for Rethink being that, well, I can get as much database as I want for free. I explored many/most of the new/nosql databases and in the end always found that Postgres was the best solution - you just shouldn't bet against Postgres it seems.
I really liked RethinkDB and thought it was well implemented but it just wasn't needed.
I do wonder if ArangoDB is going to go the same way.
Replication and a nice UI, sure, those are nice. But Postgres has fantastic single-host performance, and Rethink is not yet in a position to compete there, except in some horizontally scaled use cases. Rethink doesn't even have transactions, making it potentially useless for some things that require strict atomicity over a complicated data model. In general, Postgres has such a richness of features that it's hard to compete with it; PostGIS alone is a "killer app" if you do any kind of GIS stuff.
For me, there weren't any such "killer app" features. The change streaming is nice, but it's not difficult to do that with Postgres either. That's it really: Every solution offered by Rethink could be countered with "that's nice, but trivial with Postgres".
The one exception being horizontal replication. I have yet to launch a project where Postgres couldn't scale out well enough, and I suspect that if I got to that point, I would reach for bigger/different guns such as Cassandra. For our apps we typically use Elasticsearch in front as a kind of read-only mirror of Postgres; the latter being the slow, serializable path, and the former the fast, slightly-out-of-sync path. Users don't know the difference.
I'd like to add some colour, although I've never used RethinkDB so take it all with a large pile of salt.
lobster_johnson says:
> Rethink doesn't even have transactions
The post-mortem dwells heavily on how RethinkDB focused on correctness and was beaten by MongoDB which is, to put it mildly, not concerned about correctness.
But Rethink is a JSON DB that apparently doesn't have transactions and which on its web page says "Query JSON documents with Python, Ruby, Node.js or dozens of other languages".
Here are three technologies I do not strongly associate with correctness:
* JSON
* Schemaless databases
* Scripting languages
Rethink's own FAQ says "RethinkDB is not a good choice if you need full ACID support or strong schema enforcement".
So in the database market there are of course lots of users who do strongly value correctness, but they tend to use databases and technologies that are statically typed (Java/C#) and database engines that have are ACID and provide schemas.
Very few of them want a database that doesn't provide ACID, doesn't provide schema enforcement and which appears to either not support standard industrial languages at all, or is at least distinctly uninterested in them.
In other words Rethink targeted the hipster startup market of companies building social networks, ride-hailing apps and other things where correctness is not important and low-correctness tech dominates, but tried to sell a product that was oriented around correctness. This seems like a market/product mismatch.
That said, I don't think lack of correctness itself is a design goal. I think it's accidental, or at least an intermediate state before we have something better.
Case in point: I'm currently working on an open-source (it will be released soon) data store that uses JSON to express "documents" of data, but also has schemas, joins and transactions. It uses PostgreSQL internally, but completely hides the underlying storage. The developers who are currently using it for actual work are pleasantly surprised about how nice it is to work with: You get to work with rich, structured data in its natural shape without needing some kind of ORM to mediate between an app's data model and the database's, and at the same time it's very strictly validated without being pushy (it has gradual typing: you can start out schemaless and then tighten the schema). When we designed it, we wanted to build something that reflected how people actually use a database, and I think we're achieving that. (I'm looking forward to make it public and share it on HN.)
So it's certainly possible to have the best of both worlds. It's just that the direction that the world moved in with "NoSQL" temporarily moved us off course for a little while. It started out as a natural reaction to scaling issues, and in that sense I think it might have been a necessary step while people figured out what they were doing. We're seeing the same evolution in the ways of people's thinking with regard to dynamically/statically typed languages, too.
The Economist covers a lot of macro-economic news and frames the world in terms of market forces.
I think Slava is saying the type of biz-savy "personal development" he experienced (belatedly) could have been accelerated by more of this perspective.
While you're building you assume you'll have to gain your own level of proficiency. So you don't know at what point you'll need support.
Having to go back and ask for an increased budget after it has shipped is a harder internal sale.
As much as I appreciate rethink's approach, I can see it as a challenging business to be in
I don't think The Economist is what to read for this situation though. A book on the history of Oracle or something might have been more appropriate.
Developers have a big advantage when making devtools in that they are their own target market. They are, but only at the start of the project before they become embroiled in it. Ironically this feeling of making it for oneself leads to a mistaken urge to thrust complexity onto the target. As an infra engineer myself, this is something I always try to keep in mind -- I worked on a modelling language at a large company and I credit its success to its absolutely straightforward and accessible syntax, lack of dependencies and straightforward integration into workflows. The temptation is always to make sure users are aware of the choices, force them out of bad habits and make them consider the tradeoffs (this Strange Loop talk from Peter Alvaro addresses the social treatment of users https://www.youtube.com/watch?v=R2Aa4PivG0g ). For RethinkDB, it was wrong to try to force people to learn a new language to use a DB, when there are just tons of interfaces already, and the new language has incremental benefits over say SQL.
However elegant the design of software, the deployment that is the largest or most demanding for it will always find problems. There's no point seriously trying to claim great reliability or quality for a nascent project. It's unbelievable. Instead you need to focus on getting it used in some friendly project that is willing to pay the cost of these teething problems for some particular feature. Once battle tested, reasonable tech leads can choose to use it, comfortable that they won't be the biggest deployment (e.g. git bootstrapped by its use in the Linux kernel). Alternatively, and less scrupulously, you can try to bootstrap by providing a very convenient experience at the start of a simple project for junior engineers - there are many NoSQL DBs that promise the world and are incredibly easy to set up (see the https://aphyr.com/tags/jepsen Jepsen blog for many sad examples, RethinkDB proudly did well in this test).
RethinkDB refused to make the market fit compromises necessary. Market fit is about timing. Don't read the Economist. Ship early and keep your users on board through whatever engineering compromise necessary.
With RQL you could very clearly determine how rethinkdb was fetching your data, and you knew it was going to get your data in the same way every time the query ran.
FYI, the post is still in its unpublished state. It's basically right, but I've been meaning to edit it for tone and rewrite the market failure section for clarity. Didn't get the chance to do that before it made it on HN, so keep in mind that you're reading a (late) draft.
If you're open to edits, the $200k revenue/employee rule of thumb really doesn't make a lot of sense when you're comparing your three alternate business models (Hosted Rethink, DBaaS Rethink and "Hey, maybe PaaS on top of Rethink?").
Like you, I thought it would be way too hard to run a service (my own company was encouraged to do so, and we shied away for it for the same reason you stated). However, having seen it now from the inside for GCE for nearly 4 years, I can say it's much easier to "service-ize" yourself than it is to have built a product people want. And moreover, the shell scripts and operations stuff really scales (effectively, the hosted DB offering is the binary plus a bit of bash, while the other offerings require yet more "not done before" software).
Even more surprisingly, this statement:
> Managed hosting is essentially running the database for people on AWS so they don't have to. The alternative to using these services is setting up the database on AWS yourself. That's a pain, but it isn't actually that hard. So there is a very hard cap on how much managed database hosting services can charge.
turns out to just not be true in practice. People pay us (Cloud SQL) and AWS (RDS) a surprising amount of money to run MySQL in a VM for them. It turns out that people really like outsourcing even a small amount of pain, but that we're (as engineers) easily blindsided to "Um, why would I pay you XX% over the raw VM price? You're just running some shell scripts for me". Little things like automatically updating to the latest version, configuration replication, etc. are things that nobody will bat at an eye at paying you for (I believe in the DB space in particular, it's because the overall cost of your DB is dwarfed by "All the rest", so it's a good trade of people time versus $$s).
I hope this doesn't come across as Monday-morning quarterbacking, but if I did mine over again I'd take the "Build a great piece of software, and even if it's open, service-ize it for folks". It's true that you're competing with the Cloud providers, and that it seems like you wouldn't really be able to get people to pay you for it (and it's not your core competency), but I think it's the best option for this flavor of infrastructure software moving forward.
Sure, but for many venture funded companies and even large enterprises top-line is the top metric you're focused on. If you're in a growth industry it's about top line revenue, capturing market share, and cost of underlying bits will eventually come down.
> turns out to just not be true in practice. People pay us (Cloud SQL) and AWS (RDS) a surprising amount of money to run MySQL in a VM for them.
For infrastructure providers to "service-ize" you can set cost as a much smaller markup on the cost of the raw infrastructure. Most large organizations (AWS/GCE) have a model where they give some credit for the underlying infrastructure back to the team that servicized things. This is quite a bit different from a smaller startup that still needs to make some reasonable margin but has significant cost of goods of the infrastructure itself.
At the same time I fully agree with the notion that you should build and run a service, but if the marketing or even product perspective of such a thing is we installed a VM for you you're entirely missing the point. Having built an as-a-service product a few times over, there is a lot of engineering and product work that goes into it. Turn key backups, point in time recovery, high availability, performance monitoring are all not just installing a VM. And it's very different from building a core database engine. While there can be many benefits from collaborating on them, they are very different skill sets and the notion of focus shouldn't be underweighted.
As a happy RDS user, I totally agree. Surely, I could run MySQL and the result would be mostly fine, after some mistakes and lessons learned. But all the things such as upgrading, etc still take time, add a bit of stress, one more thing to care about. Could that time and energy be spent on more valuable (from business point of view) activities? Yes.
I have regretted this decision.
So outsourcing the database is a very logical thing to do: if you can trust a third party to have a reliable backup/recovery story, that's one thing that can kill your business that you don't have to worry about.
As an aside, lately it seems it's "OO is a deadend, Functional to the rescue!" ... so maybe you can leverage that wave instead... or deep learning... OverThinkDB maybe :)
One thing resonated strongly with me. When we started out many years ago someone said to me that the worst market you could go after was to target developers. Case in point - every developer who's ever put up a web site has built or thought about building a CMS. There must be thousands upon thousands of them, virtually all with a price of $0. Who would want to sell a CMS product?
We heeded that advice and I often reflect on it during the sales process when we have the "lets just run your offering past the IT team" meeting. So often, the line of questioning from the smartypants dudes in the room is all about tripping us up, finding our weak spot (I'm a techy so I get it - I think like this myself).
The best you can normally come out of that meeting with is grudging acceptance from IT that a SaaS solution should be OK, given that the business is in a rush and all, and given that IT has other priorities. (To be fair its much less like this these days, but a few years ago this was the norm).
By contrast, showing your product to a business person is a breath of fresh air. They don't want to build it themselves, they don't really care what's under the bonnet, they just want good, quick solutions to their actual problems, so they can go home at night and sit down to Netflix. HR people (our target market) are even nicer.
What's more those business people want to pay for your solution. A quick way to kill the deal is to joyously tell them "its free!". They need to understand how you get your buck at the end of the day. They'll never buy until they understand that. And the best (because its the simplest) answer is "we've got the best product, so we charge good money for it".
HR people are some of the worst because everything is super secret and they don't want to get IT involved, then they go and buy a system that lets you insert a carriage return into the phone number field and produces CSV files that don't have primary keys for data extracts.
It really depends on what you need... Every non-sql database tends to sacrifice something for some performance gain... RethinkDB is as close as I've seen to one without sacrifice.
CouchDB was super slow. Cassandra was amazing but a bit confusing. MongoDB made sense and worked.
> We set out to build a good database system, but users wanted a good way to do X.
If you replace "database system" with "[any product]," it encapsulates the reason why many startups fail. I was guilty of it too. I think it's a common pitfall for technical founders who tend to think in terms of solutions rather than problems.
If you have some free time, I would recommend the book What Customers Want by Anthony Ulwick, which offers a systematic approach to formulating a product that satisfies a market's underserved needs.
There's a very closely related form that is common for non-technical founders which is "We're going to use technology to fix ____ industry"
A founder with experience in a particular industry may see how inefficient it is, and how much scope there is for improvement based on technological solutions, but business live-an-die by the question: "Are there enough people who will pay us to solve this problem for them?"
Superficially, it seems like "I can make your business run more efficiently" is a reasonable approximation for "I have something that you will pay money for", but it has a very similar value-gap to "I can provide you with a technically better database".
"More efficient" and "technically better" are both things that ought to be valuable to customers, but often aren't valuable enough to get them to hand over their money.
Just don't create front end tools for front end developers, for example. Firebug had a donate button and we funneled all of it to volunteers in places where the currency was weak to make it at least seem material.
First with MySQL and then with MongoDB, a pattern is emerging. There's a popular open-source database. Its feature set is awesome and the ecosystem of software on top of it is good because lots of people are using it, but it's buggy and unreliable. So it's controversial. Since it's popular, it keeps being developed, and over time, it slowly gets more reliable.
To me this post is a reminder that, just because I can see clear flaws with a competing product, it doesn't mean those flaws are exploitable. Better to focus on what your own customers are demanding than on criticisms of your competitor.
Or people simply don't care/are not willing to pay for alternatives. MongoDB looks as unreliable as it was on the first day, but somehow people don't seem to care. I hope it's because their use-cases can live with the unreliability, but in fear it's because "won't happen to me, just to others".
I can't help but wonder if the story had been different if the RethinkDB devs organized a bunch of codecamps and positioned their db as the 'default choice'. This is how Mongo gained a ton of mindshare, helped by other trends at the time, like the NoSQL wave, the Node.js boom, and the renaissance of client-side JS development.
Like MongoDB, it was popular mostly for being popular, which is a nice gig if you can get it.
I have a rather naive question: Why do you recommend "the Economist" as oppose to something like the HBR?
The Economist offers insight, thoughtful analysis and writes about the world: literally but also politics, culture, and science. The obituary is surprisingly fascinating for instance. You may not always agree with its position, but you know that their position will be worth engaging with and thinking about.
No, I don't work for the Economist, I just appreciate thoughtful articles. Same goes for Private Eye, Foreign Affairs, New Yorker.
Figuring out the right GTM is tough, and the timing was weird, but GOOG and MSFT must have been pretty desperate for a NoSQL solution 2013-2014 to compete with DynamoDB. Firebase got scooped up in October 2014, but it wasn't until April 2015 when Azure announced DocumentDB.
By the way the tone is fine given your experience.
I understand that you're waiting for a legal framework to accept donations as an organization, but there was an article recently about an individual GIMP developer being funded on patreon. Is this something you would consider? I'd be happy to contribute to someone who's actively working on RethinkDB (of course without any backer rewards or similar expectations), and perhaps others would as well.
What do you think of Spolsky's "Good software takes 10 years"?
https://www.joelonsoftware.com/2001/07/21/good-software-take...
I would have paid a licensing fee to continue using it, but we've taken the time to switch over to Postgres now.
EDIT: Also, re: metrics of success, the examples listed as wrong there are _exactly_ why I enjoyed the product and was willing to invest time learning and implementing in a relatively new technology.
They didn't have a poor market; they didn't have any market at all. If I'm the CTO of XYZ, explain to me what I'm buying. I hate to keep coming back to it, but that's what Cygnus did so brilliantly: "You need an embedded development system. The existing solutions do nearly what you want. Pay us slightly less, and we will make GCC do exactly what you want." I know what I'm paying for.
"Build it and they will come" does not work in the open source space -- because if you have already built it, they don't need to pay you. And if you haven't, then they don't want it.
Now, if they had built a DB similar to Mongo and then sold a service that migrated people off of failing Mongo installations and onto RethinkDB, then they would be on to something. Tired of having your DB chew your data for breakfast? Want to have your reports finished the same day you ran them? For the price of a single developer, we will fix all your problems by migrating you to RethinkDB. That's something that will make money.
But who wants to charge for services? How will we get our multiples if we only charge once for our work? That's just folly!
I remember when iPhone first launched, the first 30 min was spent just trashing then-current smartphones - the blackberrys.
Lol. "It's not hard to win an Olympic medal in the 100m sprint. All you have to do is move your legs faster than those other guys!"
As a systems-guy (more or less) I find that frustrating, but I have seen many times that this is so.
It still worth having the courage to try and these guys did that. It could have worked. Hats off.
That being said, it's an interesting database and I'm still looking for a solid distributed OLTP document-database with SQL/joins and multi-cluster replication... maybe someone else will try again.
Think about some examples.
MongoDB is nothing but the object databases of the 80's, rebranded. It's main selling point is "you don't need to learn SQL". Object databases failed and people at the time were willing to learn something new (SQL). Nowadays, who will do that?
Go's entire selling point is "faster than python, no need to learn anything new". In terms of language features it's again living in the 80's - Java got generics in 2004 and Java was way behind the times. In terms of speed it's worse than Java and Rust, maybe comparable to Haskell.
And NodeJS is nothing but "write server code, no need to learn a new language". Isomorphism simply didn't work. People think Node is fast because it can do async, but Java has been able to do threaded async for 15 years.
I wonder how much the industry could advance if we could simply persuade every developer to learn Java Postgres.
A lot of people don't care much about the tech, and just want to build something. If easier (for them) tech lets them, that's a win, no?
Of course... that's easy to say but harder to swallow when they end up building something, making some money, and then hiring a developer to clean up their hideous mess. I've been there too.
And sorry, but "I wonder how much the industry could advance if we could simply persuade every developer to learn Java Postgres."?
Plenty of other people would say "I wonder how far along the industry would be if you people would actually learn how to manage your memory and write C".
Plenty of other people would say "I wonder how far the industry would have progressed if we let go of the harmful and outdated concept of imperative programming and embraced functional paradigms."
... and on it goes.
-----------------
Additionally, you say "I do find it truly sad how anti-intellectualism is pervading our industry" only to end your post with "why can't we abandon all our experimenting and learning and just stick to good old Java which obviously would solve all our problems", which is itself MASSIVELY anti-intellectual.
That's a mis-characterization. There is a difference between experimenting and learning and pushing all sorts of questionable fashion trends into production.
> I wonder how much the industry could advance if we could simply persuade every developer to learn Java Postgres.
And that is preceded by what I can only described as a bunch of undue mischaracterization of several languages and technologies, all of which definitely seems to imply that "yeah, all that other stuff is just a bunch of fads, why don't we stick to the real deal and all write Java."
Now, I don't want to be unkind to Java here. It's true that Java is an incredibly powerful language and there are a wide variety of technologies and tools that are absolutely incredible that you can only get via the Java ecosystem. But that can be said of many languages, e.g. C++/C.
I think what the GP may have meant, and which I do agree with, is that our industry would progress if we as developers spent a bit more time learning about the context and history of current and past solutions. We'd probably re-invent the wheel a bit less often.
But I don't want to limit that recommendation to "learn Java Postgres". I think Java developers might benefit from examining the benefits of a language like Haskell, Python programmers may benefit from a language with strong typing, such as C++/Rust/Go, that Unix lovers might learn a lot from examining the Windows NT kernel API's, and so on. If we examine the things that aren't close to us, the things with which we're the least familiar, that's when I believe the most fruitful learning can occur.
Plenty of other people would say "I wonder how far along the industry would be if you people would actually learn how to manage your memory and write C".
I don't see the industry gravitating towards bad re-implementations of C with less functionality and a flatter learning curve. The only "replacement" for C that I'm aware of is Rust, and Rust is most definitely not anti-intellectual in the same way I criticize Go.
Rust actually requires the developer to learn new concepts (borrow checker) which are not found in C, Python, Java, etc.
"why can't we abandon all our experimenting and learning and just stick to good old Java which obviously would solve all our problems", which is itself MASSIVELY anti-intellectual.
I didn't say to end experiments. I said everyone should actually learn some fundamentals before jumping on low-learning curve bandwagons which offer nothing new.
Well they do not want GC for their language hence borrow checker while Go already has an excellent GC.
As opposed to learning new things we need useful new things which couldn't be done before fancy new language.
Just for kicks I wrote a desktop todo list in GTK and C, not tools I had extensive knowledge of ever and nothing recently. It was a breath of fresh air how simple it was compared to doing the same in modern frameworks. A single ~30 line function to create the gui and the rest held together with a couple of function pointers. It was faster (to write and run) and simpler than anything I'd written in years. More responsive than the most responsive web framework too.
It's okay to bemoan Electron for being bloated but multiple balls have been dropped by OS and platform makers, GUI toolkit makers, language runtime makers, and that giant amount of missing glue inbetween, to get to where we are.
It's rather sad that Electron, out of all things, is the first thing to fulfill the holy grail of write-once, run anywhere, in a way that's good enough and palatable to the plurality of users' and devs' satisfaction.
Java beat Electron to write-once-run-anywhere by a couple of decades, and still does in the sense that Electron requires you to make platform-specific builds of your app even if the core code is the same (Java gives you that option but doesn't require it, you can still distribute a jar or web start file).
Meanwhile, Electron has managed the feat of being even bloatier than Java is. The DOM was never designed for GUIs and the terrible performance of web and Electron based apps is a testament to that. At least Java apps tend to have meaningful menu bars and context menus.
Electron satisfies web devs who have either never written desktop apps in other frameworks, or did decades ago and think nothing improved there. They "satisfy" users in the sense that users are rarely offered any alternative so have to suck it up and tend to judge web apps against each other vs a well coded, tightly written desktop app (office suites being a notable exception).
The biggest takeaway though is how simple it was to build. If all platforms where that simple then having seperate UI code for each platform really wouldn't be that big a deal.
At the moment I'm somewhat hopefully the libui project (https://github.com/andlabs/libui) will be successful though. It's learned the lessons from previous attempts.
I am not terribly fond of Go, but it has a decent answer in auto-implemented interfaces and duck typing. It's not about dumbing down, it's just different, and easier on the compiler, which is a significant advantage.
And as for async in java... ugh, how awful. We finally got something like promises and lambdas, but most APIs are still trapped in synchronous calls and using the JPA is a disaster of hidden thread-local state. What kind of stack are you using to make java async work? Maybe I'm just missing something.
As for Java + async + threads, I use Akka and it's basically seamless. But Netty + core Java Executors is fine too, just more verbose.
No
SQL per se is not hard.
But the whole data modeling, creating tables, picking types is just needless bureaucracy
Yes, I want to save a json snippet and be done with it. I don't want to create another table because I have a 1-N relationship. And while ORMs make this easier it is still not painless
An internally-schema'd database just recognizes that this is absurd and lets you put the bottom three layers of validation logic inside the database, where you only have to do it once.
Do you think that doesn't happen in regular DBs or that schemas don't evolve?
Love for static typing is basically Stockholm's syndrome
Your system will eventually end up with a fixed schema, but until you get to that place you don't need to "stop the world" to change it. "But Postgresql can add columns without downtime" yes it can, but what about your system?
Funny, that's sort of how I feel about my (long) time "loving" dynamic typing. Now I find it is such a relief to validate data once on the way in and completely trust it to be what it says it is thereafter. I found dynamic types required more bureaucracy (if I wanted to sleep at night) in the form of unit tests and precondition checks everywhere.
YMMV I suppose.
This merits clarification
Most of programming in dynamic typing should be implicitly statically typed. Know what you're passing. Obey the implicit (duck-typed) interfaces
Basically, don't just check "if it's a list do that, if it's one element do something else, if it's a number do another thing" - this is a beginner's mistake (and I did those)
Besides that, unit tests are a good idea, and if you want "compile type checking" in Python use Pylint (it's a good idea to use it regardless of your situation).
Though it's not so much static typing that sucks, it's its implementation in the Java/C++ way. Go and Rust make it better.
Yeah, I thought that too. After years of persistently seeing stupid (sometimes critical) bugs due to assuming the correct thing was being passed, I concluded that 1. the only sane thing to do is add explicit precondition checks that raise meaningful exceptions when assumptions are broken, and make sure those paths are exercised by unit tests, and that 2. I would much rather have that done statically and automatically by a compiler.
> Basically, don't just check "if it's a list do that, if it's one element do something else, if it's a number do another thing" - this is a beginner's mistake (and I did those)
I think this may be a misunderstanding of what I meant by "precondition checks" - I meant explicitly checking the assumptions being made by a method, most of which are type checks (foo.kind_of?) or (more commonly) interface checks (foo.respond_to?).
> Though it's not so much static typing that sucks, it's its implementation in the Java/C++ way. Go and Rust make it better.
I don't have much problem with Java's implementation of static typing at all, and I don't think C++'s problem is its static typing implementation. I find Go's implementation a bit inflexible and hard to work with though.
Again, YMMV!
Care to elaborate? I'd really like to know your opinion as this is an important aspect of my developer-life.
This, piled on with subtle ageism, ensures that any useful lessons learned will only be learned by the time it's too late, and there's a new group of can-do people who feel stifled by the combined wisdom of the ages, so they break off and do their own thing.
I wrote about this here [1] with nicer language.
[1] https://news.ycombinator.com/item?id=13022926
Java ecosystem is just toxic waste full of abandoned projects, over engineering everything, crashing build because everyone use arbitrary versioning schemes, ring-fences code with licenses/patents and developers with C++ mindset.
Node.js is maybe build on top hack on top of hack. With some ugly code, lack of any abstraction, lack of sensible std library, debugging capabilities, minimal editors support and finally with package manager that cannot be trusted.
But I can build CRUD API working with any database under 1h in node.js. Where In Java I need to start with learning all maven quirks, write pointless JSON mappers and learn stupid annotations that every library introduce.
To me, Go's selling points have been numerous:
- Faster than python
- Statically typed/no runtime bundling requirements
- Easy access to concurrency/multicore scaling
- Excellent tooling and integration with programming text editors
- Great documentation
- Excellent standard library which you can treat as a continuation of your own program
- Great design with pragmatic choices (this one is subjective, I agree)
In short, there's nothing else like that out there. Without Go, I'd have been stuck with JVM, Haskell or Erlang for most of my projects. And I dislike all of them, so Go has been a godsend.
Incindentally, I also happen to enjoy working with Postgres and prefer it to any other database I've worked with (SQL or not).
Can you come up with any intellectual rather than emotional reason for preferring Go?
I don't think you are really disagreeing with me - go does nothing new, just lower learning curve.
I was disagreeing with your original statement, Go is not just a faster python.
How did you come to this conclusion? Based on your own experiences? If so, why do you find it reasonable to assert your anecdotal truth generally?
> enjoy the benefits it offers (which are also offered by Go, BTW).
Are you seriously saying Go offers the entirety of features that Haskell offers. So what, you think people are just using Haskell for no real reason?
For you, maybe. For plenty of others, that's clearly not true.
> While you do get more proficient with time, you never get to the state where the language gets out of the way and becomes your friend instead of something you have to wrestle most of the time to enjoy the benefits it offers (which are also offered by Go, BTW)
Go offers weaker type-safety benefits that Java (with the benefit of less type-safety ceremony), much less Scala or Haskell (against which it has less ceremony benefit, because Haskell and Scala have reduced ceremony with stronger type systems).
Go doesn't offer the benefits Haskell or Scala (or Rust) do. OTOH, some people find it more intuitive, and it does offer different benefits which may be more relevant to some use cases. The sweet spot for Go seems to be the place where, before Go, you might use Python but be upset about performance and might use C but be upset about boilerplate; Go improves over the weaknesses of either on that border, while preserving most (but not necessarily all) of their strengths.
> Go isn't just a faster Python.
Faster (and more parallel) Python, less obtrusive C, lower ceremony and more native Java -- it's not just any of these, but they sort of capture its primary strengths.
Major Oracle shops aren't dumping them for Mongo/Couchbase/... because they're trying to avoid SQL. They do it for scalability, availability, performance, flexibility and more.
I don't see that the facts in the real world back you up here.
Not claiming NoSQL is a panacea. You're definitely making some trade-offs, and in many cases it's just the wrong tech. But it's absurd to say it's happening to avoid a learning curve.
RDS, Heroku and Compose. Google and Azure still dont do hosted postgres.
So many people pay for Dynamodb. You could be an alternative.
The point is - rethinkdb still exists. you could still do it. via preorders on Kickstarter.
They have the credibility. You don't question if these companies really have the people to keep the stuff running. In many cases the customers also already have a billing relationship with them. Much less friction to just add one new service to the already quite big invoice vs buying from a different company.
Competing with AWS or Google is hard, they would be still making money on the infra even if they gave the managed database out for free.
Rethinkdb is an incredible architecture. If I was in the market for something like rethinkdb/dynamodb, I would completely go with something that is built by the original inventors.
Just like docker for example.