In-Memory C++ Leap in Blockchain Analysis (caudena.com)
We’re the core engineering team at Caudena (which is used globally by investigative and intelligence agencies, including: Europol, Interpol, BKA, DHS, IRS-CI, FBI, NPA and others), and we just released the technical details behind Prism - our real-time, in-memory C++ database for blockchain analysis.
To tackle the massive scale and complexity of blockchain data, we had to get creative with low-level engineering:
- We utilize barebone servers with 2TB RAM and 48 Cores.
- Implemented lock-free concurrent data structures
- Developed a custom memory management system
- Leveraging CPU-level vectorization
- Built a custom in-memory columnar/graph database from scratch
We’d love to AMA about:
- the engineering choices we made
- crazy optimizations that paid off
- pitfalls we hit
Ask us anything about scaling, memory trade-offs, building real-time analytics on immutable data, or the crypto-forensics space.
Looking forward to a great convo!
71 comments
[ 4.2 ms ] story [ 125 ms ] threadAre you running separate instances per customer to separate the information they have access to?
A linked list can be split in two in O(1). When it comes to updating the roots for all the removed nodes, there is no easy way out, but luckily:
- This process can be parallelized.
- It could be done just once for multiple clustering changes.
- This is a multi-level disjoint set, not all the levels or sub-clusters are usually affected. Upper level clustering, which is based on lower confidence level, can be rebuilt more easily.
If by undoing you mean reverting the changes, we don’t use a persistent data structure. When we need historical clustering, we use a patched forest with concurrent hash maps to track the changes, and then apply or throw them away.
We use a single instance for all clients, but when one CFD server processes new block data, it becomes fully blocked for read access. To solve this, we built a smart load balancer that redirects user requests to a secondary CFD server. This ensures there's always at least two servers running, and more if we need additional throughput.
Most people involved in crypto pretend it's the future and their business models depend on pumping up crypto. That might be the same for you all but I figure of anyone in the space, a group dedicated to tracking down where coins are moving for government agencies (I assume for scams/crime reasons) might not have the wool so pulled over their eyes.
That said, we absolutely believe that blockchain and cryptocurrency will shape the future of the financial system. When you look beyond the noise of scam tokens, speculative NFTs, and high-profile scandals, there is significant and meaningful financial innovation happening. This extends beyond DeFi to include the tokenization of RWA, where major institutions like BlackRock and JPM Chase are actively exploring and implementing blockchain-based solutions. Numerous projects are driving real progress, and there’s a slow but steady movement toward a more decentralized and transparent financial ecosystem.
It’d be really great to be able to price predictions for various RWA at a fine grain: “Will IBM be >$240 next Friday at market close?” “Will George Alleline default on his mortgage for 46 Smiley Court?”
But manually creating and marketing these nanosecurities has too much overhead. Traditionally you’d bundle up a bunch of small stuff into a derivative, but that creates a complex instrument that’s hard to price—and then you get a Global Financial Crisis.
Block chains offer the potential to make these kinds of nano trades self-executing at scale.
Without thinking too hard, Aave is shaping to be a giant by its own as lending protocol.
Circle recently had a very successful IPO.
Farcaster and Lens are attempting to compete as social network platforms (surprisingly they lack much of the toxicity that comes on the most known ones)
And lastly don’t forget Polymarket, which is pretty well known beyond the crypto space.
The list goes on and on if you care to dig a bit deeper
EDIT: Oh and did you implement from scratch? Why not use eg. the RCU implementation from folly?
You mean bare-metal servers?
To be clear: I don't mean to imply you should have done it any other way. I'm interested mainly in gaps in existing ecosystems and whether popular suggestions to "deprecate C++ for memory safe languages" (like one made by Azure CTO years ago) are realistic.
With POSIX semaphores, mutexes, and shared pointers, it is very rare to hit upon a memory issue in modern C++.
Source: Writing code in C/C++ for 30 years.
Edit: to be less glib, this is like saying “our shred-o-matic is perfectly safe due to its robust and thoroughly tested off switch.” An off switch is essential but not nearly enough. It only provides acceptable safety if the operator is perfect, and people are not. You need guards and safety interlocks that ensure, for example, that the machine can’t be turned on while Bob is inside lubricating the bearings.
Mutexes and smart pointers are important constructs but they don’t provide safety. Safety isn’t the presence of safe constructs, but the absence of unsafe ones. Smart pointers don’t save you when you manage to escape a reference beyond the lifetime of the object because C++ encourages passing parameters by reference all over the place. Mutexes and semaphores don’t save you from failing to realize that some shared state can be mutated on two threads simultaneously. And none of this saves you from indexing off the end of a vector.
You can probably pick a subset of C++ that lets you write reasonably safe code. But the presence of semaphores, mutexes, and shared pointers isn’t what does it.
Source: also writing C and C++ for 30 years.
Exactly. Here is a data point: https://spinroot.com/spin/Doc/rax.pdf
Tl;DR: This was software that ran on a spacecraft. Specifically designed to be safe, formally analyzed, and tested out the wazoo, but nonetheless failed in flight because someone did an end-run around the safe constructs to get something to work, which ended up producing a race condition.
The statement may not be correct but calling it terrifying is way melodramatic.
Especially with shared pointers you can encounter pretty terrible memory issues.
There is a mountain of evidence (two examples follow) that this is not true. Roughly two-thirds of serious security bugs in large C++ products are still memory-safety violations.
(1) https://msrc.microsoft.com/blog/2019/07/we-need-a-safer-syst... (2) https://www.chromium.org/Home/chromium-security/memory-safet...
But with this argument, the response is "well they didn't use the primitives properly so the problem is them", which shifts the blame onto the developer and away from the tools which are too easy to silently misuse.
This also ignores memory safety issues that aren't data races, like buffer overflows, UAF, etc.
The great lesson in software security of the past few decades is that you can’t just document “proper usage,” declare all other usage to be the programmer’s fault, and achieve anything close to secure software. You must have systems that either disallow unsafe constructs (e.g. rust preventing references from escaping at compile time) or can handle “improper usage” without allowing it to become a security vulnerability (e.g. sandboxing).
Correctly use your concurrency primitives and you won’t have thread safety bugs, hooray! And when was the last time you found a bug in C-family code caused by someone who didn’t correctly use concurrency primitives because the programmer incorrectly believed that a certain piece of mutable data would only be accessed on a single thread? I’ll give you my answer: it was yesterday. Quite likely the only reason it’s not today is because I have the day off.
You answered my question. My original argument was using concurrency primitives "properly" in C++ prevents memory issues and Rust isn't strictly necessary.
I have nothing against Rust. I will use it when they freeze the language and publish a ISO spec and multiple compilers are available.
Yes, I know, I addressed that. It's true by definition, and a useless statement. Improper usage will happen. If improper usage results in security vulnerabilities, that means you will have security vulnerabilities.
Note that I say this as someone who makes a very good living writing C++ and has only dabbled in rust. I like C++ and it can be a good tool, but we must be clear-eyed about its downsides. "It's safe if you write correct code" is a longer way to say "it's unsafe."
I've never had a single complaint from my customers. Well I do have bugs in logic during development but those are found and eliminated after testing. And every new backend I do I base on already battle tested C++ foundation code. Why FFS would I ever want to change it (rewrite in Rust). As a language Rust has way less features that I am accustomed to use and this safety of Rust does not provide me any business benefits. It is quite the opposite. I would just lose time, money and still have those same logical bugs to iron out.
[1] https://x.com/ID_AA_Carmack/status/1935353905149341968
I have no allegiance to either lang ecosystem, but I think it's an overly optimistic take to consider memory safety a solved problem from a tweet about fil-c, especially considering "the performance cost is not negligible" (about 2x according to a quick search?)
- Would it be possible to open source your DB in the future? I think there are challenges in blockchain analysis (e.g. internal transactions) that goes beyond the specific DB.
- Having used Chainalysis and others, your product seems superior based on your presentation. Which blockchains do you support?
- Is there a "HN Code" to test Prism?
FBI and Europol will work with the same forest (unless they are using on-premise setup), but with different "patches".
Love the C++ work, btw
It's great to see C++ resulting in orders of magnitude cost reduction anyway. Do you have more details on the various C++ tricks done for optimization?
There are many possible optimizations, but they’re all highly specific to the particular problems you’re trying to solve.
This seems like an odd place to spend your resources. What do Prism's benchmarks look like vs Memgraph, KX kdb+, Apache Ignite, TigerGraph, etc.?