> And yet some of the most popular OLTP databases in the world today are still highly dependent on a single node architecture.
Which databases? SQLite is the one I can think of, but it's designed for that use-case. Others start as single node but will replicate to other nodes, either as master-slave or master-master.
Pretty much agree, I honestly think that TB team should invest some more effort on evangelizing the double-entry model for non-financial scenarios, like managing stocks, booking a show ticket etc. The 1000x API improvement has been done, now people must know how to use it
> In less than a decade, the world has become at least three orders of magnitude more transactional. And yet the SQL databases we still use to power this are 20-30 years old. Can they hold up?
Errr yes. Without much sweat really.
Just because something started ~30 years ago doesn't mean it hasn't updated with the times, and doesn't mean it was built on bad foundations.
- Jepsen claimed that FDB has more rigorous testing than they could do.
- New programming language, Flow, for testing.
You probably could solve the same problems with FDB, but TigerBeetle I imagine is more optimized for its use case (I would hope...).
AFAIK - the only reason FDB isn't massively popular is because no one has bothered to write good layers on top. I do know of a few folks writing a SQS, DynamoDB and SQLite layers.
We were considering TigerBeetle, but found blockers:
* We use Cloudflare Workers. TigerBeetle client app is not supported. It might work using Cloudflare Containers, but then the reason we use Cloudflare is for the Workers. --> https://github.com/tigerbeetle/tigerbeetle/issues/3177
I really liked this article. I met the new kid on the block, the DBMS neighborhood. I also didn't know that Zig programming language existed. So many new things. Congratulations to TigerBeetle! I'm going to tell my team about it and try it out on an interesting project.
this has such great marketing zingers, I probably didn't know I needed TigerBeetle for something. I don't know yet, but I promise you, I'll find something to use it for.
> Deterministic Simulation Testing (DST) feels like the most transformational technology that the fewest developers know about. It’s a novel testing technique made popular by the FoundationDB team
It's not novel. That's how hardware (ASIC) testing has been done forever. The novelty is applying it to software.
> TigerBeetle’s VOPR is the single largest DST cluster on the planet. It runs on 1,000 CPU cores
Only if you exclude hardware, otherwise basically every chip design company has a cluster bigger than this.
I work at a bank with old old monstrous sql queries.
I thought I can make a use of tigerbeetle to simplify the system. But sadly, I just couldn't figure out how to make it work. Transaction requires lot of business logics but I couldn't convert that to make it work with combination of RDBMS + tigerbeetle.
I wish there were some realworld example that I can get insight using tigerbeetle.
While I am a fan of TigerBeetle’s general stance on correctness, coding practices, and their desire to hyper-specialize, I have some critiques on the post.
I think the paragraph about multi-node is a bit misleading. Contrary to what cloud native folk will tell you, a single beefy DB, well-tuned and with a connection pooler, can serve a dizzying amount of QPS just fine. At a former employer, during a maintenance period, I once accidentally had all traffic pointed to our single MySQL 8 RDS instance, instead of sharing it between its read replicas. That was somewhere around 80-90K QPS, and it didn’t care at all. It wasn’t even a giant instance - r6i.12xlarge - we just had a decent schema, mostly sane queries, and good tuning on both ProxySQL and MySQL. At peak, that writer and two .8xlarge read replicas handled 120K QPS without blinking.
A DB hosted on a server with node-local NVMe (you know, what used to be normal) will likely hit CPU limits before you saturate its I/O capabilities.
For redundancy, all RDBMS designed for networked activity have some form of failover / hot standby capability.
My other mild criticism is in the discussion on TigerBeetle’s consensus: yes, it seems quite clever and has no other dependencies, but it’s also not trying to deal with large rows. When you can fit 8,190 transactions into a 1 MiB packet that takes a single trip to be delivered, you can probably manage what would be impossible for a traditional RDBMS.
None of this should be taken as belittling their accomplishment; I remain extremely impressed by their product.
>we just had a decent schema
Glad someone mentioned it. In the dev ORM culture now days it is all about how to make the db look like the code. DB is essentially reduced to a object store. Often a proper decent schema can handle most of these load. You can always throw in redis layer later if your traffic is really getting to require that level of optimization.
TigerBeetle looks interesting. A whole networked database using statically allocated memory. I usually use that technique for small keyboard-shortcut triggered applications.
One question in case folks who work there see this:
This is the most technically impressive zig project I've seen so far. Do you have a blog post detailing your perspective on zig? i.e what design decisions of ziglang helped you in massive way, what were the warts, and in general any other thoughts too?
>Most teams write code fast. TigerBeetle tries to write code slow. Most teams treat testing as a necessary evil. TigerBeetle is built entirely on Deterministic Simulation Testing (DST). Most teams build their software on top of loads of other software. TigerBeetle has zero dependencies.
I believe that was more like the norm 25+ years ago. Before Google and Facebook brought in the move fast and break things mentality across software industry.
I really like the work Joran and his team have done around DST, distributed systems awareness, and performance practices. I especially love the craziness around no dependencies (well, could you consider an underlying OS a dependecy?).
But I've always felt the way they treat normal OLTP (they call OLGP) seems unfair. For example, comparisons using clear sub-optimal interactive SQL transactions for financial workloads, like locking rows rather than using condition checks at commit time, because "that's how OLTP was intended to be used when it was designed ~50(?) years ago".
In their cited https://tigerbeetle.com/#performance the lowest the slider can go is 1% contention. Do you think Stripe has 1% contention directly on an OLTP DB? Definitely not.
You can build systems that _expect contention_, and elegantly handle it at REALLY high throughput. These systems protect the DB from contention, so you can continue to scale. From talking to folks working on these systems, I roughly know the kinds of transactional (financial) throughput of DBs like Stripe's and other systems - they have _many_ more zeros behind them than their performance comparison page proposes they could possibly have at even 0.01% contention.
Their marketing largely ignores this fact, and treats everyone like they just slam the DB with junior engineer-designed interactive transactions. Most developers (I hope) are smarter than that if they're working at a payments company. There's even the title "payments engineer" for the kind of person that's thinks about scaling contention and correctness all day.
TigerBeetle is great, but I find the pattern of being quite misleading about other OLTPs off putting.
> comparisons using clear sub-optimal interactive SQL transactions for financial workloads, like locking rows rather than using condition checks at commit time
Could you give an example of how the same transaction could be written poorly with "locking rows" and then more optimally with "using condition checks at commit time"?
I wonder if the 'debits and credits' example is still realistic. Is it not that banks record all individual money transfers and keep them on record for years (often due to national laws that require this). Then the balance on a certain account is a simply a selection/view on the set of all transactions. For efficiency reasons you may want to store the result in some table. If there is a limit on withdrawals on a certain account, you do need to calculate the balance first and then a transaction mechanism is needed to assure that there are not several money transfer that individually do not surpass the limit but together do.
Why hasn't anyone mentioned the game they made? It looks fantastic. The concept of letting players try to crash a system to demonstrate its robustness is brilliant.
49 comments
[ 2.9 ms ] story [ 64.7 ms ] threadWhich databases? SQLite is the one I can think of, but it's designed for that use-case. Others start as single node but will replicate to other nodes, either as master-slave or master-master.
Errr yes. Without much sweat really.
Just because something started ~30 years ago doesn't mean it hasn't updated with the times, and doesn't mean it was built on bad foundations.
- Slow code writing.
- DST
- No dependencies
- Distributed by default in prod
- Clock fault tolerance with optimistic locking
- Jepsen claimed that FDB has more rigorous testing than they could do.
- New programming language, Flow, for testing.
You probably could solve the same problems with FDB, but TigerBeetle I imagine is more optimized for its use case (I would hope...).
AFAIK - the only reason FDB isn't massively popular is because no one has bothered to write good layers on top. I do know of a few folks writing a SQS, DynamoDB and SQLite layers.
* We use Cloudflare Workers. TigerBeetle client app is not supported. It might work using Cloudflare Containers, but then the reason we use Cloudflare is for the Workers. --> https://github.com/tigerbeetle/tigerbeetle/issues/3177
* TigerBeetle doesn't support any auth. It means the containing server (e.g. a VPS) must restrict by IP. Problem is, serverless doesn't have fixed IP. --> https://github.com/tigerbeetle/tigerbeetle/issues/3073
Since "interesting" is the very last thing that anyone sane wants in their accounting/financial/critical-stuff database.
Never understood why we turn those off. An assert failing in prod is an assert that I desperately want to know about.
(That "never understood" was rhetorical).
Rather, I would say that the most interesting database is the fastest and safest.
Or as Jim Gray always put it: “correct and fast”.
It's not novel. That's how hardware (ASIC) testing has been done forever. The novelty is applying it to software.
> TigerBeetle’s VOPR is the single largest DST cluster on the planet. It runs on 1,000 CPU cores
Only if you exclude hardware, otherwise basically every chip design company has a cluster bigger than this.
We typically hear from companies that TigerBeetle is the easier part of their stack. But to be fair, they may have a cleaner architecture.
Did you contact our solutions team for assistance with your business logic modelling? If not, please get in touch! solutions@tigerbeetle.com
I think the paragraph about multi-node is a bit misleading. Contrary to what cloud native folk will tell you, a single beefy DB, well-tuned and with a connection pooler, can serve a dizzying amount of QPS just fine. At a former employer, during a maintenance period, I once accidentally had all traffic pointed to our single MySQL 8 RDS instance, instead of sharing it between its read replicas. That was somewhere around 80-90K QPS, and it didn’t care at all. It wasn’t even a giant instance - r6i.12xlarge - we just had a decent schema, mostly sane queries, and good tuning on both ProxySQL and MySQL. At peak, that writer and two .8xlarge read replicas handled 120K QPS without blinking.
A DB hosted on a server with node-local NVMe (you know, what used to be normal) will likely hit CPU limits before you saturate its I/O capabilities.
For redundancy, all RDBMS designed for networked activity have some form of failover / hot standby capability.
My other mild criticism is in the discussion on TigerBeetle’s consensus: yes, it seems quite clever and has no other dependencies, but it’s also not trying to deal with large rows. When you can fit 8,190 transactions into a 1 MiB packet that takes a single trip to be delivered, you can probably manage what would be impossible for a traditional RDBMS.
None of this should be taken as belittling their accomplishment; I remain extremely impressed by their product.
One question in case folks who work there see this:
This is the most technically impressive zig project I've seen so far. Do you have a blog post detailing your perspective on zig? i.e what design decisions of ziglang helped you in massive way, what were the warts, and in general any other thoughts too?
I believe that was more like the norm 25+ years ago. Before Google and Facebook brought in the move fast and break things mentality across software industry.
I hope TigerBeetle gets more recognition. Worth reading its Jepsen report as well. https://news.ycombinator.com/item?id=44199592
But I've always felt the way they treat normal OLTP (they call OLGP) seems unfair. For example, comparisons using clear sub-optimal interactive SQL transactions for financial workloads, like locking rows rather than using condition checks at commit time, because "that's how OLTP was intended to be used when it was designed ~50(?) years ago".
In their cited https://tigerbeetle.com/#performance the lowest the slider can go is 1% contention. Do you think Stripe has 1% contention directly on an OLTP DB? Definitely not.
You can build systems that _expect contention_, and elegantly handle it at REALLY high throughput. These systems protect the DB from contention, so you can continue to scale. From talking to folks working on these systems, I roughly know the kinds of transactional (financial) throughput of DBs like Stripe's and other systems - they have _many_ more zeros behind them than their performance comparison page proposes they could possibly have at even 0.01% contention.
Their marketing largely ignores this fact, and treats everyone like they just slam the DB with junior engineer-designed interactive transactions. Most developers (I hope) are smarter than that if they're working at a payments company. There's even the title "payments engineer" for the kind of person that's thinks about scaling contention and correctness all day.
TigerBeetle is great, but I find the pattern of being quite misleading about other OLTPs off putting.
Could you give an example of how the same transaction could be written poorly with "locking rows" and then more optimally with "using condition checks at commit time"?