Ask HN: What has been your experience with CockroachDB?

57 points by punkpeye ↗ HN
I literally cannot find a single person describing their experience using CockroachDB, which is bizarre considering how long they have been around.

Would love to hear from someone who've used them in production.

42 comments

[ 3.2 ms ] story [ 96.8 ms ] thread
I know of two scaleups that have left it. As usual, this might just be poor implementation choices rather than product or support issues.
(note the username!)

Check out Aerospike. It's the most robust and truly "zero effort scale" real time data platform I've ever seen. Properly configured and used, it whups everything in terms of TPS and performance/machine aka TCO. By a huge, huge, factor.

Downside is ... everything else. :/ Less polished developer experience, docs, &c. There are so many huge customers, but very few who are talking about it. Requires more of a time investment -- because you're learning a new platform, not using a drop in replacement for what you already know. But the reward is performance beyond anything else.

(the VLDB 2016 paper is a good high level overview of the architecture)

If you do check it out, be sure to use version 6.1 or later, in particular see what can be done combining secondary indexes with container data types.... A, uh, birdie told me that. :P

edit: Same birdie told me Aerospike has customers who replaced CockroachDB, when they found out it has trouble scaling past a point.

I really hope they can get their act together in terms of communication and marketing -- it's sadly "the most advanced solution you've never heard of".

> Same birdie told me Aerospike has customers who replaced CockroachDB, when they found out it has trouble scaling past a point.

Elaborate?

Nah, I'd rather be vague. ;-) The issue, I'm guessing, was that CockroachDB couldn't maintain the required latency bounds and availability past a certain point for the customer. Those are always where Aerospike wins (latency, availability, while scaling). milliseconds vs microseconds.

The limiting factor isn't how many operations the platform can handle, it's how many operations/transactions you can generate before the server will even break a sweat.

I'm just a lowly engineer, and while I'll say some stuff on a throwaway account, this is really the marketing department's purview.... Casual googling confirms it's public information that Schwab runs ... nearly everything on it, and it's a much smaller number of servers than you'd think.

Wow, Schwab runs an AGPL database?! Guessing they've a commercial license.
Doesn't it seem equally likely they haven't made any changes to the codebase? Or have simply contributed them to the project?
AGPL requires making 'full' source of a network service available to network users. Doubtful Schwab would do that.
If they haven't made any changes to the DB, that's just pointing them to the main website.
AGPL considers use of unmodified copies a "covered work". And it requires source of the calling code to also be disclosed.

(IANAL)

I read various interpretations of the AGPL and some people think it’s not the case.

MongoDB used to be AGPL and I believe most people didn’t interpret the license as everything calling the database must be AGPL, and then everything calling the thing calling the database too, that recursively until the entire world must be licensed under AGPL.

We'll see when AGPL gets tested in court. Until then I suspect legal departments will continue to treat AGPL, and even GPL (because of on prem), as a viral license which could require disclosing all their source.
No
No we won't see it tested in court? Or legal departments won't consider it a viral license?
The latter. Every large company on the planet uses Linux which is GPL.
GPL is widely used in SAAS because network users of SAAS aren't entitled to source. AGPL is different.
I don't really see how this competes with cockroach.

Cockroach is meant to be sql compatible, whilst Aerospike is nosql.

Is that why you commented under a different username, because you are affectively spreading misinformation?

> I don't really see how this competes with cockroach.

I never said it did.

> Is that why you commented under a different username, because you are affectively spreading misinformation?

No, asshole. Because it's a small world, and it's not exactly the smartest thing to do to.... never mind. Obviously you're gonna assume ill intent regardless of what I say.

> "whilst Aerospike is nosql"

You obviously have no idea what you're talking about, and have never used it. If you had, you wouldn't think of it as "nosql".

It's... more like a real time data processing platform, I honestly don't know of an existing category or comparable software.

At first glance it looks like a kvs database with a lot of bells and whistles. After you learn more, though, you realize it's not "just" a database -- you write your application with the Aerospike client library, using expressions, data modeling, CDTs, &c. Not using it as a black box for storing data.

Aerospike describes itself as a NoSql platform in the first sentence of it's front page.
Reported for not assuming the best - or even reasonable - interpretation of the parent comment. Don’t be so insufferable.
There is no reasonable interpretation that makes a comment suggesting an alternative which is nothing like CockroachDB, the entire point of the OP’s post, sound reasonable.

Especially when it’s so conspiratorial in nature and spreading information that is completely unverifiable.

Thanks for your enlightening comment. Now myself and others not to never use Aerospike.

A 12 year old product that still has poor documentation? Plus you say TPS = TCO. That’s a good one!

Haven’t used them in production because their license doesn’t allow operating as a service.

I have used an alternative.

What does that mean? What service do you have that is disallowed? We are evaluating it and are finding the license page confusing. Yugabyte seems friendlier but it seems even less talked about.
Cockroachdb uses the BSL which means you can't host it as a service for others. It's more of a "source available" license (that eventually converts to Apache, after a few years).
I used YugabyteDB. License was one thing: database is Apache 2 licensed and no strings attached, you can operate it as a service yourself. Not so with CockroachDB, what powerhour said.

Other reasons: it’s Postgres code under the hood, not a rewrite, so you can use existing extensions. I have even contributed the support for FDWs, which I use in production.

More here https://gruchalski.com/tags/yugabytedb/ and here https://github.com/radekg/yugabyte-db-multi-tenant-paas-demo.

Thank you for that feedback!
To be clear, you can use CockroachDB if you're a service.

What is prohibited, however, is you can't take CockroachDB and try and offer a hosted CockroachDB-as-a-Service to external organizations.

But if your organization has a DB-as-a-Service offering internally, you can totally add crdb to the mix (this is pretty common, as a matter of fact). If your org offers a service and its backed by crdb, you can totally do that, too.

Yeah I’d be glad to talk about it. Was part of a team that was asked to design a whistleblowing software platform for large corporate and government customers. The rest of the stack was RDS Postgres but this whistleblowing app had super different requirements. We would need to run in basically every region and also not exclusively in AWS.

After trying some different options it became clear that Cockroach was the best option by far. The strongly consistent nature of Cockroach made it a good choice for a SoR like whistleblowing unlike say Mongo. The ease of deployment and the stability of the Raft consensus protocol for managing ranges of data was better than expected.

For me having done a global rollout of it, I’d recommend it with very few caveats. You do need to train developers on what queries it can and cannot support, since it does have Postgres client support but obviously cannot handle the full range of queries. You also need to have lots of alerting around the Raft log, since that is your bread and butter in terms of “is this working or not”. However in aggressive testing, removing nodes with no warning under load, simulating pretty severe networking problems, etc, it has held up remarkably well.

What's your global setup like, if you may? Running software for government means you generally want your data isolated from everything else. Wondering where cockroach fits in.
So why would cockroach be a fit here?

That doesn't sound like an app that's going to have to process a lot of data at all.

Is it the fact you can get clusters synched really well across regions? And is that really a problem for low transaction apps today?

(comment deleted)
Maybe people who answer/read this also evaluated yugabytedb, which has a much friendlier license (apache 2)? We are evaluating different solutions and we depend on some Postgres compatibility as we have a lot of code to that effect. Yugabyte and cockroach come up all the time.
Good find. No experience, but adding my list to try.
Yes. AMA and I’ll do my best to answer.
(comment deleted)
We tried it for a payment system, the concurrency issues where an absolute nightmare to deal with.Especially in combination with Hibernate. In serializable isolation level some queries would just never finish, no matter how many retries you gave it.
Launched it on a t3.nano (0.5GB RAM) in a single node configuration for two extremely small tables - got oom kills in couple of hours.

Launched on a t3.micro (1GB RAM) - same.

Launched on a t3.small (2GB RAM) - finally it works and doesn't crash.

So, min 2GB RAM to serve about 10 (as in: ten, not thousands) records queried every other second.

Why don't you try swapping your existing PG server with CockroachDB and see how it goes.

Being a die hard fan of Postgresql for more than 15 years, I think time has come for me to select CockroachDB as new default.

I miss Table Partitioning (only available in enterprise edition ) and Full text search (yet to be built). Also minimum hardware I had to employ was quad core and 8 gigs of RAM.

But other than that everything is so smooth including installation, online documentation. It made me lazy.