There's indeed a connection between record/replay and deterministic execution, but there's a difference worth mentioning, too. Both can tell you about the past, but only deterministic execution can tell you about alternate histories. And that's very valuable both for bug search (fuzzing works better) and for debugging (see for example the graphs where we show when a bug became likely to occur, seconds before it actually occurred).
(Also, you won't be able to usefully record a hypervisor with jockey or rr, because those operate in userspace and the actual execution of guest code does not. You could probably record software cpu execution with qemu, but it would be slow)
I assume deterministic execution also lets you do failing test case reduction.
I've found this sort of high volume random testing w. test case reduction is just a game changer for compiler testing, where there's much the same effect at quickly flushing out newly introduced bugs.
I have been down this road a little bit, applying the ideas from jockey to write and ship a deterministic HFT system, so I have some understanding of the difficulties here.
We needed that for fault tolerance, so we could have a hot synced standby. We did have to record all inputs (and outputs for sanity checking) though.
We did also get a good taste of the debugging superpowers you mention in your blog article. We could pull down a trace from a days trading and replay on our own machines, and skip back and forth in time and find the root cause of anything.
It sounds like what you have done is something similar, but with your own (AMD64) virtual machine implementation, making it fully deterministic and replayable, and providing useful and custom hardware impls (networking, clock, etc).
That sounds like a lot of hard but also fun work.
I am missing something though, in that you are not using it just for lockstep sync or deterministic replays, but you are using it for fuzzing. That is, you are altering the replay somehow to find crashes or assertion failures.
Ah, I think perhaps you are running a large number of sims with a different seed (for injecting faults or whatnot) for your VM, and then just recording that seed when something fails.
> Programming in this state is like living life surrounded by a force field that protects you from all harm. [...] We deleted all of our dependencies (including Zookeeper) because they had bugs, and wrote our own Paxos implementation in very little time and it _had no bugs_.
Being able to make that statement and back it by evidence must be indeed a cool thing.
I've written formal proofs with bugs more than once. Reality is much messier than you can encode into any proof and there will ultimately be a boundary where the real systems you're trying to build can still have bugs.
Formal verification is incredibly, amazingly good if you achieve it, but it's not the same as "perfect".
You can claim that your spec doesn't violate some invariants in a finite number of steps, you can't claim that the spec contains all the invariants the real system must have and that it doesn't violate them in number of steps + 1.
If you are writing software, it is almost always trying to accomplish a goal outside of itself. It is trying to solve a problem for someone, and how that problem can or should be solved is rarely perfectly clear.
The spec is supposed to map to a real world problem, and there is never going to be a way to formalize that mapping.
> The longer I have computed, the less I seem to use Numerical Software Packages. In an ideal world this would be crazy; maybe it is even a little bit crazy today. But I've been bitten too often by bugs in those Packages. For me, it is simply too frustrating to be sidetracked while solving my own problem by the need to debug somebody else's software. So, except for linear algebra packages, I usually roll my own. It's inefficient, I suppose, but my nerves are calmer.
> The most troubling aspect of using Numerical Software Packages, however, is not their occasional goofs, but rather the way the packages inevitably hide deficiencies in a problem's formulation. We can dump a set of equations into a solver and it will usually give back a solution without complaint - even if the equations are quite poorly conditioned or have an unsuspected singularity that is distorting the answers from physical reality. Or it may give us an alternative solution that we failed to anticipate. The package helps us ignore these possibilities - or even to detect their occurrence if the execution is buried inside a larger program. Given our capacity for error-blindness, software that actually hides our errors from us is a questionable form of progress.
> And if we do detect suspicious behavior, we really can't dig into the package to find our troubles. We will simply have to reprogram the problem ourselves. We would have been better off doing so from the beginning - with a good chance that the immersion into the problem's reality would have dispelled the logical confusions before ever getting to the machine.
I suppose whether to do this depends on how rigorous one is, how rigorous certain dependencies are, and how much time one has. I'm not going to be writing my own database (too complicated, multiple well-tested options available) but if I only use a subset of the functionality of a smaller package that isn't tested well, rolling my own could make sense.
In the specific case in question, the biggest problem was that dependencies like Zookeeper weren't compatible with our testing approach, so we couldn't do true end to end tests unless we replaced them. One of the nice things about Antithesis is that because our approach to deterministic simulation is at the whole system level, we can do it against real dependencies if you can install them.
I was a co-founder of both FoundationDB and Antithesis.
That tracks well (both the quotes and your thoughts).
One example that comes to mind where I want to roll my own thing (and am in the process of doing so) is replacing our ci/cd usage of jenkins that is solely for running qa automation tests against PR's on github. Jenkins does way way more than we need. We just need github PR interaction/webhook, secure credentials management, and spawning ecs tasks on aws...
Every time I force myself to update our jenkins instance, I buckle up because there is probably some random plugin, or jenkins agent thing, or ... SOMETHING that will break and require me to spend time tracking down what broke and why. 100% surface area for issues, whilst we use <5% of what Jenkins actually provides.
I've been super interested in this field since finding out about it from the `sled` simulation guide [0] (which outlines how FoundationDB does what they do).
Currently bringing a similar kind of testing in to our workplace by writing our services to run on top of `madsim` [1]. This lets us continue writing async/await-style services in tokio but then (in tests) replace them with a deterministic executor that patches all sources of non-determinism (including dependencies that call out to the OS). It's pretty seamless.
The author of this article isn't joking when they say that the startup cost of this effort is monumental. Dealing with every possible source of non-determinism, re-writing services to be testable/sans-IO [2], etc. takes a lot of engineering effort.
Once the system is in place though, it's hard to describe just how confident you feel in your code. Combined with tools like quickcheck [3], you can test hundreds of thousands of subtle failure cases in I/O, event ordering, timeouts, dropped packets, filesystem failures, etc.
This kind of testing is an incredibly powerful tool to have in your toolbelt, if you have the patience and fortitude to invest in it.
As for Antithesis itself, it looks very very cool. Bringing the deterministic testing down the stack to below the OS is awesome. Should make it possible to test entire systems without wiring up a harness manually every time. Can’t wait to try it out!
TigerBeetle is actually another customer of ours. You might ask why, given that they have their own, very sophisticated simulation testing. The answer is that they're so fanatical about correctness, they wanted a "red team" for their own fault simulator, in case a bug in their tests might hide a bug in their database!
I gotta say, that is some next-level commitment to writing a good database.
Sure! I mentioned a few orthogonal concepts that go well together, and each of the following examples has a different combination that they employ:
- the company that developed Madsim (RisingWave) [0] [1] is tries hardest to eliminate non-determinism with the broadest scope (stubbing out syscalls, etc.)
- sled [2] itself has an interesting combo of deterministic tests combined with quickcheck+failpoints test case auto-discovery
- Dropbox [3] uses a similar approach but they talk about it a bit more abstractly.
Sans-IO is more documented in Python [4], but str0m [5] and quinn-proto [6] are the best examples in Rust I’m aware of. Note that sans-IO is orthogonal to deterministic test frameworks, but it composes well with them.
With the disclaimer that anything I comment on this site is my opinion alone, and does not reflect the company I work at —— I do work at a rust shop that has utilized these techniques on some projects.
TigerBeetle is an amazing example and I’ve looked at it before! They are really the best example of this approach outside of FoundationDB I think.
> you can test hundreds of thousands of subtle failure cases in I/O, event ordering, timeouts, dropped packets, filesystem failures, etc.
As cool as all this is, I can't stop but wonder how often the culture of micro-services and distributed computing is ill advised. So much complexity I've seen in such systems boils down to calling a "function" is: async, depends on the OS, is executed at some point or never, always returns a bunch of strings that need to be parsed to re-enter the static type system, which comes with its own set of failure modes. This makes the seemingly simple task of abstracting logic into a named component, aka a function, extremely complex. You don't need to test for any of the subtle failures you mentioned if you leave the logic inside the same process and just call a function. I know monoliths aren't always a good idea or fit, at the same time I'm highly septical whether the current prevalence of service based software architectures is justified and pays off.
> I can't stop but wonder how often the culture of micro-services and distributed computing is ill advised.
You can't get away from distributed computing, unless you get away from computing. A modern computer isn't a single unit, it's a system of computers talking to each other. Even if you go back a long time, you'll find many computers or proto-computers talking to each other, but with a lot stricter timings, as the computers are less flexible.
If you save a file to a disk, you're really asking the OS (somehow) to send a message to the computer on the storage device, asking it to store your data, and it will respond with success or failure and it might also write the data. (sometimes it will tell your os success and then proceed to throw the data away, which is always fun)
That said, keeping things together where it makes sense, is definitely a good thing.
I see your point. Even multithreading can be seen as a form of distributed programming. At the same time, in my experience these parts can often be isolated. You trust your DB to handle such issues, and I'm very happy we are getting a new era of DBs like Tigerbetle, FoundationDB and sled that are designed to survive Jepsen. But how many teams are building DBs? That point is a bit ironic, given I'm currently building an in-memory DB at work. But it's a completely different level of complexity. And your example with writing a file, that too is a somewhat solved problem, use ZFS. I'd argue there are many situations where the fault tolerant distributed requirements can be served by existing abstractions.
This sounds quite cool. Although it doesn't say so, I imagine the name is riff off Hypothesis, the testing tool that performs automatic test case simplification in a general way.
(I’m an early employee of what was, on my start date, just called “Void Star.”)
As I recall it, the name meant two things:
1. Our “autonomous testing” approach is the opposite, or the antithesis, of flaky and unreliable testing methodologies.
2. You can think of our product as standing in dialectical opposition to buggy customer software, pointing out its internal contradictions (bugs) and together synthesizing a new, bug-free software product. (N.b.: I’ve never actually read Hegel.)
We did note the resonance with Hypothesis (a library I like a lot!) at the time, but it was just an added bonus :).
I wonder if they are working on a time travel debugger. If it is truly deterministic presumably you could visit any point in time after a record is made and replay it.
Do you have any plans to let small open source teams use the project for free? Obviously you have bills to pay and your customers are happy to do that, but I was wondering if you'd allow open source projects access to your service once a week or something.
Partly because I want to play with this and I can't see my employer or client paying for it! But also it fits neatly into "DX", the Developer Experience, i.e. making the development cycle as friction free for devs as possible. I'm DevOps with a lifelong interest in UX, so DX is something I'm excited about.
Pricing suitable for small teams, and perhaps even a free tier, is absolutely on the roadmap. We decided to build the "hard", security-obsessed version of the infrastructure first -- single-tenant, with dedicated and physically isolated hardware and networking for every customer. That means there's a bit of per-customer overhead that we have to recoup.
In the future, we will probably have a multi-tenant offering that's easier for open source projects to adopt. In the meantime, if your project is cool and would benefit from our testing, you can try to get our research team interested in using it as part of the curriculum that makes our platform smarter.
We've actually done quite a bit of testing on open source projects as we've built this, and have discussed doing an on-going program of testing open source projects that have interested contributors. We'd probably find some interesting things and could do some write-ups. Reach out to us via our contact page or contact@antithesis.com and let's chat.
The system can certainly revisit a previous simulated moment and replay it. And we have some pretty cool things using that capability as a primitive. Check out the probability chart in the bug report linked from the demo page: https://antithesis.com/product/demo
Now I want a simulation-run replay scrubbing slider MIDI-connected to my Pioneer DJ rig to scratch through our troublesome tests as my homies push patched containers.
Our entire debugging frontend is built on that API. We've also started to build new advanced features that leverage that API in unique ways, like our React and Redux DevTools integration and "Jump to Code" feature:
Reminds me of the clever hack of playing back TCP dump logs from prod on a test network, but dialed up. Neat.
Naturally I’d prefer professional programmers learn the cognitive tools for manageably reasoning about nondeterminism, but they’ve been around over half a century and it hasn’t happened yet.
What’s really interesting to me is that the simulation adequately replicates the real network. One of the more popular criticisms of analytical approaches is sone variant of: yeah, but the real network isn’t going to behave like your model. Which by the way is an entirely plausible concern for anyone who has messed with that layer.
What is interesting here is that the solution could fuzz-test anything, including the network model, leading to failures even more implausible than reality.
> Naturally I’d prefer professional programmers learn the cognitive tools for manageably reasoning about nondeterminism
It’s not an either-or here, though. Part of the challenge is you’re not always thinking about all the non-determinisms in your code, and the interconnections between your code and other code (whose behavior you can sometimes only assume) can make that close to impossible. Part of that is the “your model of the network” critique, but also part of that is “your model of how people will use your software” isn’t necessarily correct either.
This might be the best introduction post I've read.
Lays the foundation (get it?) for who the people are and what they've built.
Then explains how the current thing they are building is a result of the previous thing. It feels that they actually want this problem solved for everyone because they have experienced how good the solution feels.
Then tells us about the teams (pretty big names with complex systems) that have already used it.
All of these wrapped in good writing that appeals to developers/founders. Landing page is great too!
The entire testing system they describe feels like something I can strive towards too. They make you want their solution because it offers a way of life and thinking and doing like you've never experienced before
Post author here. Sorry it was vague, but there's only so much detail you can go into in a blog post aimed at general audiences. Our documentation (https://antithesis.com/docs/) has a lot more info.
Here's my attempt at a more complete answer: think of the story of the blind men and the elephant. There's a thing, called fuzzing, invented by security researchers. There's a thing, called property-based testing, invented by functional programmers. There's a thing, called network simulation, invented by distributed systems people. There's a thing, called rare-event simulation, invented by physicists (!). But if you squint, all of these things are really the same kind of thing, which we call "autonomous testing". It's where you express high-level properties of your system, and have the computer do the grunt work to see if they're true. Antithesis is our attempt to take the best ideas from each of these fields, and turn them into something really usable for the vast majority of software.
We believe the two fundamental problems preventing widespread adoption of autonomous testing are: (1) most software is non-deterministic, but non-determinism breaks the core feedback loop that guides things like coverage-guided fuzzing. (2) the state space you're searching is inconceivably vast, and the search problem in full generality is insolubly hard. Antithesis tries to address both of these problems.
So... is it fuzzing? Sort of, except you can apply it to whole interacting networked systems, not just standalone parsers and libraries. Is it property-based testing? Sort of, except you can express properties that require a "global" view of the entire state space traversed by the system, which could never be locally asserted in code. Is it fault injection or chaos testing? Sort of, except that it can use the techniques of coverage guided fuzzing to get deep into the nooks and crannies of your software, and determinism to ensure that every bug is replayable, no matter how weird it is.
It's hard to explain, because it's hard to wrap your arms around the whole thing. But our other big goal is to make all of this easy to understand and easy to use. In some ways, that's proved to be even harder than the very hard technological problems we've faced. But we're excited and up for it, and we think the payoff could be big for our whole industry.
Your feedback about what's explained well and what's explained poorly is an important signal for us in this third very hard task. Please keep giving it to us!
I remember watching the Strange Loop video on your testing strategy, and now I need to go back and relearn how it differed from model checking (ie Promela or TLA+). Model checking is probably the big QA story that tech companies ignore because it requires dramatically more education, especially from QA departments typically seen as "inferior" to SWE.
Doesn't Antithesis rely on the fact that software is always deterministic? Reproducibility appears to be its top selling feature – something that wouldn't be possible if software were non-deterministic.
* Offer only good for x86-64 software that runs on Linux whose dependencies you can install locally or mock. The first two restrictions we will probably relax someday.
Right, by emulating a deterministic computer you can ensure that the inputs to the software are always deterministic – something traditional computing environments are unable to offer for various reasons.
However, if we pretend that software was somehow able to be non-deterministic, it would be able to evade your deterministic computer. But since software is always deterministic, you just have to guarantee determinism in the inputs.
>But since software is always deterministic, you just have to guarantee determinism in the inputs.
This is technically correct, but that's a very load-bearing "just". A lot of things would have to count as inputs. Think about execution time, for example. CPUs don't execute at the same speed all the time because of automatic throttling. Network packets have different flight times. Threads and processes get scheduled a little differently. In distributed/concurrent systems, all this matters. If you run the same workload twice, observable events will happen at different times and in different orders because of tiny deviations in initial conditions.
So yes, if you consider the time it takes to run every single machine instruction as an "input", then software is deterministic given the same inputs. But in the real world that's not actionable. Even if you had all those inputs, how are you going to pass them in? For all intents and purposes most software execution is non-deterministic.
The Antithesis simulation is deterministic in this way though. It is in charge of how long everything takes in "simulated time", right down to the running times of individual CPU instructions. Everything observable from within the simulation happens the exact same way, every time. You can compare a memory dump at the same (simulated) instant across two different runs and they will be bit-for-bit identical.
Sure. A good example. Execution time – more accurately, execution speed – isn't a property of software. For example, as you point out yourself, you can alter the execution speed without altering the software. It is, indeed, an input.
> Even if you had all those inputs, how are you going to pass them in?
Well, we know how to pass them in non-deterministically. That's how software is able to do anything.
Perhaps one could create a simulated environment that is able to control all the inputs? In fact, I'm told there is a company known as Antithesis working on exactly that.
Is the challenge here the same as with digital simulations of electronic circuits? That is, at the end of the day analog physics becomes confounding? Or are you doing deterministic simulation of random RF noise as well?
This vaguely reminds me of Jefferson's "Virtual Time" paper from 1985[1]. The underlying idea at the time didn't really take off because it required, like Zookeeper, a greenfield project: except that it kinda doesn't and today you could imagine instrumenting an entire Linux syscall table and letting any Linux container become a virtual time system -- but Linux didn't exist in 1985 and wouldn't be standard until much later.
So Jefferson just says, let's take your I/O-ful process, split it a message-passing actor model, and monitor all the messages going in and coming out. The messages coming out, they won't necessarily do what they're supposed to do yet, they'll just be recorded with a plus sign and a virtual timestamp, and by assumption eventually you'll block on some response. So we have a bunch of recorded message timestamps coming in, we have your recorded messages going out.
Well, there's a problem here, which is that if we have multiple actors we may discover that their timestamps have traveled out-of-order. You sent some message at t=532 but someone actually sent you a message at t=231 that you might have selected instead of whatever you actually selected to send the t=532 message. (For instance in the OS case, they might have literally sent a SIGKILL to your process and you might not have sent anything after that.) That's what the plus sign is for, indirectly: we can restart your process from either a known synchronization state or else from the very beginning, we know all of its inputs during its first run so we have "determinized" it up past t=231 to see what it does now. Now, it sends a new message at say t=373. So we use the opposite of +, the minus sign, to send to all the other processes the "undo" message for their t=532 message, this removes it from their message buffer: that will never be sent to them. And if they haven't hit that timestamp in their personal processing yet, no further action is needed, otherwise we need to roll them back too. Doing so you determinize the whole networked cluster.
The only other really modern implementation of these older ideas that I remember seeing was Haxl[2], a Haskell library which does something similar but rather than using a virtual time coordinate, it just uses a process-local cache: when you request any I/O, it first fetches from the cache if possible and then if that's not possible it goes out, fetches the data, and then caches it. As a result you can just offer someone a pre-populated cache which, with these recorded inputs, will regenerate the offending stack trace deterministically.
Has any thought been given to repurposing this deterministic computer for more than just autonomous testing/fuzzing? For example, given an ability to record/snapshot the state, resumable software (i.e. durable execution)?
Somebody once suggested to me that this could be very hand for the reproducible builds folks. I'm sure that now that we're out in the open, lots of people will suggest great applications for it.
My favourite application for "deterministic computer" is creating a cluster in order to have a virtual machine which is resilient to hardware failure. Potentially even "this VM will keep running even if an entire AWS region goes down" (although that would add significant latency).
This is interesting - it is kind of picking a fight with SaaS/cloud providers though, as that is the one kind of software you won't be able to import into your environment: not because it can't do the job, but because you don't have the code. So this would create an incentive to go back to PaaS.
It's definitely true though that a big problem with backend is that you can't easily treat it as a whole system for test purposes.
thanks, I'll dig in. I'm a very visual person and charts/diagrams/flows always help my grasp of something more than a wall of text. Maybe include some of those in there when you get the time?
Is there more info on how Antithesis solves problem number 2 (large state spaces)? I understand the fuzzing / workload generation part well, but there's so many different state space reduction techniques that I don't know what Antithesis is doing under the hood to combat that.
> Your feedback about what's explained well and what's explained poorly is an important signal for us in this third very hard task. Please keep giving it to us!
It's hard to understand these complex concepts via language alone.
Diagrams would be a huge help to understand how this system of testing works compared to existing testing concepts
Sure, it doesn't go into details. And that is exactly why I termed it an excellent introduction and a sales pitch.
I haven't heard of deterministic testing before. Nor have I heard of FoundationDB or the related things. And I went from knowing zero things about them to getting impressed and interested. This led me to go into their docs, blogs, landing page, etc. to know more.
It seems like marketing copy. Not a technical blog post.
It would be nice to see some actual use cases and examples.
Instead, the writer just name-dropped a few big companies and claimed to have a revolutionary product that works magically. Then include the typical buzzwords like '10x programmer' and 'stealth mode'. The latter doesn't make sense because they also name-drop clients.
It absolutely doesn’t read like typical marketing copy, and yes it’s not a dense technical blog post either. I’m sure the use cases and examples will come, but putting them in this post would have been overkill.
Also, stealth mode just means your company isn’t public, you can still have clients.
Having that context puts the post in a much better perspective. It's definitely an introduction post (the company has been developing this in stealth mode for the past few years), but it is most certainly _not_ a marketing post. These people developed extremely novel testing techniques for FoundationDB and are now generalizing them to work with any containerized application.
I'm reading the paper. It's very intriguing but so is the marketing material for the new Tesla. I need to work with things like this before I believe the claims.
Too many people here on HN are getting caught up in a barely-tested new technology and hailing it as some revolution.
No, I did mean to say potentially "unethical" tasks. Much of national defense is deemed "unethical" by many, yet a critical function that allows you to live a peaceful life.
It won't tell you whether the software works correctly, it will just tell you if it raises an exception or crashes.
Put a fuzzer on Chrome for example, you won't catch most of the issues it has, though Chrome actually has tons of bugs and issues, but you may find security issues if you devote a big enough budget to run your fuzzer long time enough to cover all the branches.
So it's good in the case where you use "exceptions as tests", where any minor out-of-scope behavior raises an exception and all the cases are pre-planned (a bit like you baked-in runtime checks, and the fuzzer explores them)
The similarity is about obtaining determinism through something like a hypervisor. The way rr works is it basically writes down the result of all the system calls, etc, basically everything that ended up on the Turing machine’s tape, so you can rewind and replay.
> The biggest effect was that it gave our tiny engineering team the productivity of a team 50x its size.
I feel like the idea of the legendary "10x" developer has been bastardized to just mean workers who work 15 hours a day 6.5 days a week to get something out the door until they burn out.
But here's your real 10x (or 50x) productivity. People who implement something very few people even considered or understood to be possible, which then gives amazing leverage to deliver working software in a fraction of the time.
Your definition is also vague. Someone still needs to do the legwork. One man armies who can do everything themselves don't really fit in standardized teams where everything is compartmentalized and work divided and spread out.
They work best on their own projects with nobody else in their way, no colleagues, no managers, but that's not most jobs. Once you're part of a team, you can't do too much work yourself no matter how good you are, as inevitably the other slower/weaker team members will slow you down as you'll fight dealing with the issues they introduce into the project or the issues from management, so every team moves at the speed of the lowest common denominator no matter their rockstars.
That rings true and is probably why the 10x engineers I have seen usually work on devops or modify the framework the other devs are using in some way. For example, an engineer who speeds up a build or test suite by an order of magnitude is easily a 10x engineer in most organizations, in terms of man hours saved.
> For example, an engineer who speeds up a build or test suite by an order of magnitude is easily a 10x engineer in most organizations, in terms of man hours saved.
Yeah but this isn't something scalable that can happen regularly as part of your job description. Like most jobs/companies don't have so many low hanging fruits to pick that someone can speed of build by orders of magnitude on a weekly basis. It's usually a one time thing. And one time things don't usually make you a 10x dev. Maybe you just got lucky once to see something others missed.
And often times at big places most people know where the low hanging fruits are and can fix them, but management, release schedules and tech debt are perpetually in the way.
IMHO what makes you a 10x dev is you always know how to unblock people no matter the issue so that the project is constantly smooth saling, not chasing orders of magnitude improvements unicorns.
Does anyone else feel like people follow these sort of industry pop-culture terms a bit too intensely? What I mean is that the existence of the term tends to bring out people trying to figure who that might be, as if it has to be 100% true.
I personally think that some people can provide “10x” (arbitrary) the value on occasion, like the low hanging fruit you said. I also believe some people are slightly more skilled than others, and get more results out of their work. That said, there are so many ways for somebody to have an impact that doesn’t have to immediate, that I find the term itself too prevalent.
Yep. No matter what you're doing, some people are more productive than others. Often it's a matter of experience and practice, sometimes ability to focus, sometimes motivation, rarely it's a lack or surplus of inherent ability. Using people effectively in the context of a team all depends on the skill of the manager though.
I think a lot of people that complain about 10x chattery in HN should take some kind of carpentry etc course or some other kind of handiwork like that with a real master.
Some of those people not only get things done much quicker, but they also get it done with better quality than an amateur, with less mistakes, throwing away less material, sometimes with more safety.
This is definitely more than 10x better. And there are some real hacks doing those kinds of jobs. I find programming to be not different than that.
Well sure, if you compare a master to a novice, there is almost always a great difference. But between masters of carpentry, there is usually not so much difference. But here with the 10x trope it is supposed to be different and I would say indeed, but it is not as common as many would like to think.
Perhaps there aren’t that many non-master carpenters (I don’t think that’s true, there’s plenty of professional incompetents), but I am 100% sure that not all professional developers are “masters”.
It really does depend on where you work. The order of magnitude improvements I'm describing involved interdisciplinary expertise involving both bespoke distributed build systems and assembly language. They're not unicorns, they do exist, but they are very rare and most engineers just aren't going to be able to find them, even with infinite time. Hence why a 10x engineer is so valuable and not everyone can be one. I myself am certainly not one, in most contexts.
>insane levels of inefficiency in literally every process.
In processes yes, not in code, and solo 10x devs alone can't fix broken processes as those are a the effect of broken management and engineering culture.
People know where the inefficiencies are, but management doesn't care.
Nothing wrong with "one man armies" in the team context. There is a long list of tasks that needs to be done.. over same time period, one person will do 5 complex tasks (with tests and documentation), while the other will do just 1 task, and then spend even more time redoing it properly.
Over time this produces funny effects, like super-big 20 point task done in few days because wrong person started working on it.
The truest 10x engineer I ever encountered was a memory firmware guy with ASIC experience who absolutely made sure to log off at 5 every day after really putting in the work. Go to guy for all parts of the codebase, even that which he didn't expressly touch.
The “10x engineer” comes from the observation that there is a 10x difference in productivity between the best and the worst engineers. By saying that you want to be a 1x engineer, you’re saying you want to be the least productive engineer possible. 1x is not the average, 1x is the worst.
i believe the original was for an entire "organizations" performance, and was also done in 1977. Since they are averages, It makes "sense" to conclude that the best of a good team is 10x better than the average of the worst team. Not really what the experimwnt concludes but what can you do.
Except you haven't explained it at all. Sackman, Erickson, and Grant found that some developers were able to complete what was effectively a programming contest in a 10th of the time of the slowest participants. This is the origin of the 10x developer idea.
You, on the other hand, are claiming that 10x engineers are 10 times more productive than the worst engineers. Completing a programming challenge in a 10th of the time is not the same as being 10 times more productive, and obviously your usage can't be an explanation, even as one you made up on the spot, as the math doesn't add up.
That was designed as a repeatable experiment, which seems entirely reasonable when you want to conduct a study. Why are you characterising that as “a programming contest”? That seems like an uncharitably distorted way of describing a study.
> Why are you characterising that as “a programming contest”?
Because it was? Do you have a better way to repeatedly test performance? And yes, the study's intent was to look at performance, not productivity. It's even right in the title. Not sure where you dreamed up the latter.
What we do know is that the worst engineers provide negative productivity. If 1x is the worst engineer, then let's for the sake of discussion denote x as -1 in order for the product to be negative. Except that means the 10x engineer provides -10 productivity, actually making them the worst engineer. Therein lies a conflict.
What we also know is that best engineer has positive productivity, so that means the multiplicand must always be positive. Which means that it is the multiplier that must go negative, meaning that a -1x and maybe even a -10x engineer exists.
You are arguing against the idea that there is a factor of ten difference in productivity between the best and the worst engineers. That’s fine if you want to do that, but that’s explicitly where the term “10x engineer” comes from and what defines its meaning. So if you disagree with the underlying concept, there is no way for you to use terms like “[n]x engineer” coherently since you disagree with its most fundamental premise. You certainly shouldn’t reinvent different meanings for these terms.
Thank you. This sounds so trivial at first, but your reductio ad absurdum at the beginning of your comment really nails it.
Throw into the mix the fact that productivity is hard to measure as soon as more than one person works on something and that doesn't even begin to consider the economical aspects of software.
And even when ignoring this point, there's that pesky short-term vs long-term thing.
Also, how do you define the term "productivity"? I was assuming that you mean somethint along the lines of (indirect, if employed) monetary output.
The problem here is the introduction of productivity.
The 10x developer originated from a study that measured performance. The 10x developer being able to do a task in a 10th of the time is quite conceivable and reflects what the study found. I'm sure we've all seen a developer take 10 hours to do a job that would take another developer just 1 hour. Nobody is doing it in negative hours, so the math works.
Measuring productivity like that in technology makes no sense because our work is not fungible; what and how we do it matters as much as how fast we do it. Time-based productivity measurement is for factory workers stamping out widgets. So in our revenue-based world, negative productivity makes sense.
Performance. That is what the study that found a 10x performance difference observed. There is no mention of productivity in the study. If anyone has tried to study productivity, they most certainly have not come up with a 10x moniker. It seems productivity was mentioned in this thread only because it also happens to start with the letter 'p' and someone got confused.
Okay, but it still doesn't work. The world's worst engineer who somehow managed to successfully contribute one line of code to something like GPT is way more productive than a great engineer who designed from top to bottom the best laid software ever conceived but was thrown away before seeing the light of day because the business changed direction.
Of course, that doesn't actually matter as the original study found a 10x difference in measuring performance, not productivity. There is nothing out there to suggest that some developers are 10x more productive outside of those who mixed up their p words. We're not actually talking about productivity; that was just a mistake. If one were to study productivity, I expect they would find that some engineers are many orders of magnitude more productive than the least productive engineers.
> The world's worst engineer who somehow managed to successfully contribute one line of code to something like GPT is way more productive than a great engineer who designed from top to bottom the best laid software ever conceived but was thrown away before seeing the light of day because the business changed direction.
I reject that assertion.
> performance, not productivity
You keep saying this like it's a slam dunk refutation, but performance and productivity are highly related.
Because you don't believe the worst developer contributed anything to GPT? Sure, in reality that's no doubt true, but it was only ever meant to be illustrative.
> but performance and productivity are highly related.
Not in any meaningful way. The study found that the fastest developer can perform a set of defined tasks in a 10th of the time of the slowest developer. That is what the 10x developer refers to. But being fast doesn't mean being productive.
Come to my backyard and we will each dig a hole of equal size. Let's assume that you can dig the hole in a 10th of the time I can – that you are the 10x hole digger. But, no matter how fast you are, neither of us will be productive.
Productivity in this case is performance on contrived tasks that don't represent day-to-day work well and for some of which negative score isn't possible.
Hmm, I never thought of it that way. I just heard 10x employees and fit it to what I knew. Which is that 90% of the work is accomplished by about 10% of workers. The other 90% really only get 10% done. So most developers are somewhere on a scale of 0.1 - 1. With 1 being a totally competent and good developer. The 10x people are just different though, it's like a pro-athlete to a regular player. It's not unique to software development, though it may stand out and be sought after more. I've noticed it in pretty much every industry. Some people are just able to achieve flow state in their work and be vastly more productive than others, be it writing code or laying sod. I don't find that there's a lot of in between 1 and 10 though.
Even if this was the origin of the term, it still doesn't make sense because the best engineers can solve problems the worst would never be able to do so. The difference between the best and worst is much more than 10x the worst. Maybe the worst who meets certain minimums at a company, but then the best would also be limited by those willing to work for what the company pays, and I hypothesis that the minimums of the lower bound and the maximums of the upper bound are correlated.
It sounds like you disagree with the concept of a 10x engineer then. In which case you should avoid using the term, rather than making up a new definition.
Concepts and words change meaning and sometimes we all need to accept that the popular meaning is not the definition we use.
This is especially common when dealing with historical or academic definitions versus common modern usage. "Evolution" particularly annoys me.
You should avoid using the term, rather than using a definition at odds with common usage. Your usage is confusing - and that is why you are getting push-back.
The definition you have given is nonsensical - it can't be consistent over time or between companies because it depends on finding a minimum in a group. And a value that is strongly dependent on the worst developer is useless because it mostly measures how bad the worst developer is - it doesn't say anything about how good the best developer is.
Do 10x engineers get 10x the wages? Somehow I feel being exceptionally better than other engineers is just unfair to both of you and the ones worse than you. I wouldn't want to be a 10x either, I'd rather just be normal engineer.
Meta compensates 10x types very well. 3x bonus multipliers, additional equity that can range from 100k-1m+, and level increases are a huge bump to comp (https://www.levels.fyi/)
Meta compensates all SWEs very well. To suppose arguendo that 10x types exist, I don't think they're really compensated linearly 10x more than everyone else. But yeah, certainly, if you are great at your job and want to make a bunch of money, Meta is a great employer for that.
3x bonus multiplies (Redefines Expectations) are extremely uncommon. Level increases certainly help but like, L7 only makes ~3x what L5 does -- not 10x. And there are few L7s and very few L8+.
I feel you. I had a semi-traumatic experience at a previous job, combined with a number of other factors which have left me feeling like a shell of my former self. Now I'm sure if I'm working at .5x or if I feel like .5x. Such is life, at times.
It seems like the industry would get a lot more 10x behavior if it was recognized and rewarded more often than it currently does. Too often, management will focus more on the guy who works 12 hour days to accomplish 8 hours of real work than the guy who gets the same thing accomplished in an 8 hour day.
Also, deviations from 'normal' are frowned upon. Taking time to improve the process isn't built into the schedule; so taking time to build a wheelbarrow is discouraged when they think you could be hauling buckets faster instead.
It's almost impossible to get executives to think in return on equity (“RoE”) for the future instead of “costs” measured in dollars and cents last quarter.
Which is weird, since so many executives are working in a VC-funded environment, and internal work should be “venture funded” as well.
Even when they are smart enough to know, they seem to have very short memories. While I don't consider myself to be a 10x engineer; I have certainly done a number of 10x things over my career.
I worked for a company where I almost single handedly built a product that resulted in tens of millions of dollars in sales. I got a nice 'atta boy' for it, but my future ideas were often overridden by someone in management who 'knew better'. After the management changed, I found myself in a downsizing event once I started criticizing them for a lack of innovation.
This is the sad part of it, many people without core competence end up in "leadership" positions and remove any "perceived" threats to their authority. I believe part of it is due to the absence of leadership training in the engineering curriculum. Colleges should encourage engineers to take up few leadership courses and get them trained on things like Influence and Power.
Knowing the difference between an overly ambitious or technically wrong proposal and when to listen to the engineer seems impossible at times. Perhaps it requires a consultant.
In my experience working for several companies, people who know and are not merely acting out of fear can call things out and explain why something is wrong or ambitious. And more often than not sensible engineers get it. Consultants are usually called in by the leadership when they can't deal with the backlash or have no core competence in the first place.
>It seems like the industry would get a lot more 10x behavior if it was recognized and rewarded more often than it currently does
I'd be happier if industry cares more for team productivity - I have witnessed how rewarding "10x" individuals may lead to perverse results on a wider scale, a la Cobra Effect. In one insidious case, our management-enabled, long-tenured "10x" rockstar fixed all the big customer-facing bugs quickly, but would create multiple smaller bugs and regressions for the 1x developers to fix while he moved to the next big problem worthy of his attention. Everyone else ended up being 0.7x - which made the curse of an engineer look even more productive comparatively!
Because he was allowed to break the rules, there was a growing portion of the codebase that only he could work on - while it wasn't Rust, imagine an org has a "No Unsafe Rust" rule that is optional to 1 guy. Organizations ought to be very careful how they measure productivity, and should certainly look beyond first-order metrics.
I try to look at these things through the lens of “software literacy” - software is a form of literacy and this story might be better viewed as “a bunch of illiterate managers are impressed with one good writer at the encyclopdia publishers, now it turns out this guy makes mistakes, but hey, what do you expect when the management cannot read or write !”
> In one insidious case, our management-enabled, long-tenured "10x" rockstar fixed all the big customer-facing bugs quickly, but would create multiple smaller bugs and regressions for the 1x developers to fix while he moved to the next big problem worthy of his attention. Everyone else ended up being 0.7x - which made the curse of an engineer look even more productive comparatively! Because he was allowed to break the rules,
bingo, well said. Worked on a team like this with a “principal” engineer who’d work very fast with bug-ridden work like this simply because he had the automatic blessing from on high to do whatever he wanted. My unfortunate task was to run along behind him and clean up, which to my credit I think I did a pretty good job at, but of course these types can only very rarely acknowledge/appreciate that.
Eventually he got super insecure/threatened and attempted to push me out along with whoever else he felt was a threat to his fiefdom.
I don’t know, eventually the working environment got too toxic and I along with two other senior engineers quit. As far as I know he’s still running his little kingdom.
Sure. A cohesive team of employees who stick together for more than 2 years is so rare. They can be so much more productive than a group of talented people who rotate a new person in and out every 3-6 months.
This reminds me of the "Parable of the Two Programmers." [1] A story about what happens to a brilliant developer given an identical task to a mediocre developer.
I can't seem to find it in a google search, maybe I'm just recalling entirely the wrong terms.
In the early computing era there was a competition. Something like take some input and produce an output. One programmer made a large program in (IIRC) Fortran with complex specifications documentation etc. The other used shell pipes, sort, and a small handful or two of other programs in a pipeline to accomplish the same task in like 10 developer min.
This is the competition I was thinking of. I must have read it in a dead-image PDF version some other time on HN. This paper isn't the one I recall but the solution is exactly the sort I vaguely recalled.
I'm trying to copy-in the program as it might have existed, with some obvious updates to work in today's shells ...
[For those who may not follow all the links: Bentley asked Knuth to write a program in Pascal (WEB) to illustrate literate programming—i.e. explaining a long complicated program—and so Knuth wrote a beautiful program with a custom data structure (hash-packed tries). Bentley then asked McIlroy to review the program. In the second half of the review, McIlroy (the inventor of Unix pipes) questioned the problem itself (the idea of writing a program for scratch), and used the opportunity to evangelize Unix and Unix pipes (at the time not widely known or available).]
"""“And who better understands the Unix-nature?” Master Foo asked. “Is it he who writes the ten thousand lines, or he who, perceiving the emptiness of the task, gains merit by not coding?”"""
I was both of those developers at different times, at least metaphorically.
I drank from the OO koolaid at one point. I was really into building things up using OOD and creating extensible, flexible code to accomplish everything.
And when I showed some code I'd written to my brother, he (rightly) scoffed and said that should have been 2-3 lines of shell script.
And I was enlightened. ;)
Like, I seriously rebuilt my programming philosophy practically from the ground up after that one comment. It's cool having a really smart brother, even if he's younger than me. :)
Without more backup I can only describe that as being fiction. Righteous fiction, where the good guy gets downtrodden and the bad guy wins to fuel the reader's resentment.
To me it is a story about managers clueless about the work. You can make all the effort in the world to imagine doing something but the taste of the soup is in the eating. I do very simple physical grunt work for a living, there it is much more obvious that it is impossible. It's truly hilarious.
They probably deserve more praise when they do guess correctly but would anyone really know when it happens?
Sometimes I'm appreciated, and managers actually realize what they have when I create something for them. Frequently I accomplish borderline miracles and a manager will look at me and say, "OK, what about this other thing?"
My first job out of college, I was working for a company run by a guy who said to me, "Programmers are a dime a dozen."
He also said to me, after I quit, after his client refused to give him any more work unless he guaranteed that I was the lead developer on it, "I can't believe you quit." I simply shrugged and thought, "Maybe you shouldn't have treated me like crap, including not even matching the other offer I got."
I've also made quite a lot of money "Rescuing Small Companies From Code Disasters. (TM)" ;) Yes, that's my catch phrase. So I've seen the messes that teams often create.
The "incompetent" team code description in the story is practically prescient. I've seen the results of exactly that kind of management and team a dozen times. Things that, given the same project description, I could have created in 1/100 the code and with much more overall flexibility. I've literally thrown out entire projects like that and replaced them with the much smaller, tighter, and faster code that does more than the original project.
So all I can say is: Find better teams to work with if you think this is fiction. This resonates with me because it contains industry Truth.
I had an idea once but when I tried to explain it people didn't understand.
I revisited earlier thought: communication is a 2 man job, one is to not make an effort to understand while the other explains things poorly. It always manages to never work out.
Periodically I thought about the puzzle and was eventually able to explain it such that people thought it was brilliant ~ tho much to complex to execute.
I thought about it some more, years went by and I eventually managed to make it easy to understand. The response: "If it was that simple someone else would have thought of it." I still find it hilarious decades later.
It pops to mind often when I rewrite some code and it goes from almost unreadable to something simple and elegant. Ah, this must be how someone else would have done it!
“Give me six hours to chop down a tree and I will spend the first four sharpening the axe.”
― Abraham Lincoln
I have started to follow this 'lately' (for a decade) and it has worked miracles. As for the anxious managers/clients, I keep them updated of the design/documentation/though process, mentioning the risks of the path-not-taken, and that maintain their peace of mind. But this depends heavily on the client and the managers.
This is unrelated to the excellent story, but it's annoying that the repost has the following "correction":
> The manager of Charles has by now [become] tired of seeing him goof off.
"The manager has tired of Charles" is as correct as "the manager has become tired of Charles". To tire is a verb. The square bracket correction is unnecessary and arguably makes the sentence worse.
> It seems like the industry would get a lot more 10x behavior if it was recognized and rewarded more often than it currently does.
I don't agree with that, there are a _lot_ of completely crap developers and they get put into positions where even the ones capable of doing so aren't allowed to because it's not on a ticket.
"Don't confuse motion with action", in other words. I think a lot of people aren't good at it because they themselves are rewarded for the opposite. This seems rife in the "just above individual contributor" management layer, but that's a biased take.
That's entirely the fundamental flaw of the Nx developer ethos to a tee. No individual will benchmark reliably against any other person of their same trade/craft perfectly over time. The mythical BS times developer is so over simplified to be a meaningless concept. Hire "unicorn" and get amazing results just isn't a guarantee. They just probably have better chance than average to make a higher impact, which is good enough for companies that are willing to pay Nx times average salaries to acquire them.
His point is that smart and productive people are generally hard working, focused and diligent, which is how they get to be so experienced and productive.
> His point is that smart and productive people are generally hard working, focused and diligent
I don't think that tracks. Smart, productive, hard working people don't work 9-5. They work every hour they can, breaking only when they have pushed themselves to the limit. The limit can be hit at any hour. There is no magical property of the universe that gives people unlimited stamina during the hours of 9-5.
> Nobody would call this person "10x".
I'm not sure they would call anyone that, to be fair. A "10x developer" who also puts in 8 hours alongside the 1x developers isn't a 10x developer, he would be called a sucker.
I know it's meant to be funny, but the number of tech people who spend zero time learning about "what's out there", are usually not the most effective developers. You won't find better solutions to existing or even new problems without an interest in industry. Maybe this particular article isn't "industry valuable fair enough", but having zero interest in refining and enhancing your craft beyond the work in front of you is almost guaranteed to end with worse outcomes.
Another flaw in his thinking: brain cycles and sub-conscious processing.
I'm in the middle of a hard problem right now. I ran out of ideas, and opened HN about half an hour ago. In that time, without "trying", I've had two new ideas - one sent me back to my notes, which revealed that my original thinking was flawed; the second sent me to documentation, which suggested a new route to pursue. I'm digesting the implications of that while I write this.
Beating my head against the problem directly for thirty minutes would have been less productive. (Though if I wasn't WFH I would have, and also been miserable, and learned less about the industry than I have from this thread. So there's that.)
I'm far from a 10x anything, but I don't have the only brain which works this way.
If I'd done that I couldn't have context-switched back to the documentation so easily!
Nah. I get what you're saying, and it's a great idea. Sometimes I do go for a walk. More often I do dishes. Those are both, however, higher-commitment, more time-consuming activities than flipping to a different window. If I remember correctly, my sticking point that day felt like a small one, and so the distraction / break I felt like I needed was correspondingly small.
Not true. I've known some very ADHD developers who are constantly context shifting and are able to fuck around on Hackernews for a while and then suddenly knock out a huge amount of work. The problem is that (speaking from personal experience) everybody with ADHD thinks they can do this and 99% cannot.
On my team, one of the main multipliers is understanding the need behind the requested implementation, and proposing alternative solutions - minimizing or avoiding code changes altogether. It helps that we work on internal tooling and are very close to the process and stakeholders.
"Hmmm, there's another way to accomplish this" being the 10x. Doing things faster is not it.
Exactly this. It’s why it’s so frustrating when product managers who think they’re above giving background run the show (the ones who think they’re your manager and are therefore too important to share that with you)
When I was in college, I've met a few people that coded _a lot_ faster than me. Typically, they started since they were 12 instead of 21 (like me). That's how 10x engineers exist, by the time they are 30, they have roughly 20 years of programming experience behind their belt instead of 10.
Also, their professional experience is much greater. Sure, their initial jobs at 15 are the occassional weird gig for the uncle/aunt or cousin/nephew but they get picked up by professional firms at 18 and do a job next to their CS studies.
At least, that's how it used to be. Not sure if this is still happening due to the new job environment, but this was the reality from around 2004 to 2018.
For 10x engineers to exist, all it takes is a few examples. To me, everyone is in agreement that they seem to be rare. I point to a public 10x engineer. He'd never say it himself, but my guess is that this person is a 10x engineer [1].
If you disagree, I'm curious how you'd disagree. I'm just a blind man touching a part of the elephant [2]. I do not claim to see the whole picture.
Yup, that's been my experience as someone who asked for a C++ compiler for my 12th birthday, worked on a bunch of random websites and webapps for friends of the family, and spent some time at age 16-17 running a Beowulf cluster and attempting to help postdocs port their code to run on MPI (with mixed success). All thru my CS education I was writing tons of toy programs, contributing (as much as I could) toward OSS, reading lots of stuff on best practices, and leaning on my much older (12 years) brother who was working in the industry. He pointed me to Java and IntelliJ, told me to read Design Patterns (Gang of Four) and Refactoring (Fowler). I read Joel on Software religiously, even though he was a Microsoft guy and I was a hardcore Linux-head.
By the time I joined my first real company at age 21, I was ready to start putting a lot of this stuff into place. I joined a small med device software company which had a great product but really no strong software engineering culture: zero unit tests, using CVS with no branches, release builds were done manually on the COO's workstation, etc.
As literally the most junior person in the company I worked through all these things and convinced my much more senior colleagues that we should start using release branches instead of "hey everybody, please don't check in any new code until we get this release out the door". I wrote automated build scripts mostly for my own benefit, until the COO realized that he didn't have to worry about keeping a dev environment on his machine, now that he didn't code any more. I wrote a junit-inspired unit testing framework for the language we were using (https://en.wikipedia.org/wiki/IDL_(programming_language) - like Matlab but weirder).
Without my work as a "10x junior engineer", the company would have been unable to scale to more than 3 or 4 developers. I got involved in hiring and made sure we were hiring people who were on board with writing tests. We finally turned into a "real" software company 2 or 3 years after I joined.
This sounds similar to the best programmer I personally know and he was an intern working at LLVM at the time. It's funny how companies treat that part of his life as "no experience". Then suddenly he goes into the HFT space and within a couple of years he has a similar rank that people have that are twice his age.
10x engineers exist. To be fair, it does depend which software engineer you see as "the standard software engineer", but if I take myself as a standard (as an employed software engineer with 5 years of experience), then 10x software engineers exist.
Yeah similar story here. Basically if you start programming 10y before your peers of the same age then it’s logical you’ll be more productive and knowledgeable. Maybe a lot of “10x engineers” are just guys who gave themselves a huge head start for whatever reason
I started learning how to write a Makefile when I was around 15, and I learned all about using tabs. My colleagues didn't touch Make until they were 25 and were extremely confused.
To work for the aunt/uncle, friend of the family etc described earlier the thread, or having an older brother helping… 10x programmers have a bunch of people giving them a head-start, not just themselves. Sure the person needs a drive and early start, but it’s moot without the support to actually convert that early start into a career. Maybe not moot, but for sure a whole lot harder.
I was coding before age 10 and delivering homework in website form any chance I got. Coding was just another video game to me. And yet, I didn’t even end up being a programmer, let alone 10x. I was given bad advice from highschool career advisors, lacked the family connections that knew better to give me odd tech jobs, had disinterested teachers at best and ones that insulted students who asked clarification questions at worst. I took an engineering class in highschool where i loved the content but I felt like I didn’t belong and dropped from it convinced tech was not for me after all (and then proceeded making games in unreal in my free time). For every person supporting my interest in CS there were 10 telling/nudging me to give up, so i was eventually worn out and gave it up. I still code 25 years later but I am far behind now instead of ahead.
Anyway, this is a longwinded way to say it takes privilege to be a 10x programmer, not everyone who gives themselves a head start will succeed.
Nick with Antithesis here with a funny story on this.
I became friends with Dave our CTO when I was 5 or 6, we were neighbors. He'd already started coding little games in Basic (this was 1985). Later in our friendship, like when I was maybe 10, I asked him if he could help me learn to code, which he did. After a week or two I had made some progress but compared what I could do to what he was doing and figured "I guess I just started too late, what's the point?".
I found out later that most people didn't start coding till late HS or college! It worked out though - I'm programmer adjacent and have taken care of the business side of our projects through the years :)
Last year, we had 2 new hires.. one is fresh out of college (and not one of the top ones), other with 15 years experience on resume in our industry.
I am not sure there is 10x difference, but there is at least 5x difference in performance, in favor of fresh college grad, and they are now working on the more complex tasks too.
The sad part is our hiring is still heavily in "senior engineer with lots of experience" phase, and intership program has been canceled.
At my previous company I had 2 first-job juniors and 4 or 5 interns that were outstanding.
(however the last one was totally terrible and totally killed my hiring credibility hah, but it was a 80% success rate still)
I find that there are too many pretenders, though. I get way too many people with 15 years of experience and ChatGPT resumes that just can't code at all :/
have hired five interns and five experienced developers. Some of the interns exceeded expectations, while some of the experienced developers also performed well.
The top-performing experienced developers outshone all the graduates. However, the less effective experienced developers were on par with the graduates, showing no significant difference in performance.
The takeaway for me is that simple anecdotes are not very informative. Over time and with a larger sample size, experienced individuals tend to perform better. Nonetheless, some graduates will also become exceptionally skilled.
Graduates are more cost-effective. Experienced professionals require less oversight. If they need substantial guidance, they don't truly qualify as senior, as opposed to their resume that says, at 25 they have been a CTO for 10 years).
The new core question for interviewers to answer is "why should I hire you over ChatGPT?" If you are just going to take the requirements and copy paste them into ChatGPT, I could do that myself just as fast and you aren't adding any value.
Some people organize their time and focus their efforts more efficiently than others. They also use tools that others might not even know or careabout.
You probably surf the internet 10x faster than your parents. Yes you've probably had more exposure than them, but you could probably teach them how to do it just as fast. But would they want to learn and would they actually adapt what you taught them?
Yes: Programmers who start at twelve are often the 10x programmers who can really program faster than the average developer by a lot.
No: It's not because they have 10 more years of experience. Read "The Mythical Man Month." That's the book that popularized the concept that some developers were 5-25x faster than others. One of the takeaways was that the speed of a developer was not correlated with experience. At all.
That said, the kind of person who can learn programming at 12 might just be the kind of person who is really good at programming.
I started learning programming concepts at 11-12. I'm not the best programmer I know, but when I started out in the industry at 22 I was working with developers with 10+ years of (real) experience on me...and I was able to come in and improve on their code to an extreme degree. I was completing my projects faster than other senior developers. With less than two years of experience in the industry I was promoted to "senior" developer and put on a project as lead (and sole) developer and my project was the only one to be completed on time, and with no defects. (This is video game industry, so it wasn't exactly a super-simple project; at the time this meant games written 100% in assembly language with all kinds of memory and performance constraints, and a single bug meant Nintendo would reject the image and make you fix the problem. We got our cartridge approved the first time through.)
Some programmers are just faster and more intuitive with programming than others. This shouldn't be a surprise. Some writers are better and faster than others. Some artists are better and faster than others. Some architects are better and faster than others. Some product designers are better and faster than others. It's not all about the number of hours of practice in any of these cases; yes, the best in a field often practices an insane amount. But the very top in each field, despite having similar numbers of hours of practice and experience, can vary in skill by an insane amount. Even some of the best in each field are vastly different in speed: You can have an artist who takes years to paint a single painting, and another who does several per week, but of similar ultimate quality. Humans have different aptitudes. This shouldn't even be controversial.
I do wonder if the "learned programming at 12" has anything to do with it: Most people will only ever be able to speak a language as fluently as a native speaker if they learn it before they're about 13-14 years old. After that the brain (again, for most people; this isn't universal) apparently becomes less flexible. In MRI studies they can actually detect differences between the parts of the brain used to learn a foreign language as an adult vs. as a tween or early teen. So there's a chance that early exposure to the right concepts actually reshapes the brain. But that's just conjecture mixed with my intuition of the situation: When I observe "normal" developers program, it really feels like I'm a native speaker and they're trying to convert between an alien way of thinking about a problem into a foreign language they're not that familiar with.
AND...there may not be a need to explicitly PROGRAM before you're 15 to be good at it as an adult. There are video games that exercise similar brain regions that could substitute for actual programming experience. AND I may be 100% wrong. Would be good for someone to fund some studies.
The association with video games in your last paragraph makes a lot of sense to me. This is how I feel solving problems.
I always thought that people who start at 12 and keep at it are good because they really love it.I see people who struggle a lot with learning, and it's because they hate it but are doing it for other reasons.
That childhood native-fluency analogy is insightful! Your experience matches mine.
I started programming at age 7 and it's true that the way code forms in my head feels similar to the way words form when I'm writing or speaking in English. In the same way that I don't stop and consciously figure out whether to use the past or present tense while I'm talking, I usually don't consciously think about, say, what kind of looping construct I'm about to use; it's just the natural-feeling way to express the idea I'm trying to convey. The idea itself is kind of already in the form of mental code in the same way that my thoughts are kind of already in English if I'm speaking.
But... maybe that's how it is for everyone, even people who learned later? I only know how it is in my own head.
I totally get the same sense that I'm just "communicating" using code. I just write out the code that expresses the concepts I have in my head.
And at least some people clearly don't. I was talking to one guy who said that even for a simple for-each loop it was way faster for him to "Google the code he needs and modify it" than to write it. This boggled me. I couldn't imagine being able to Google and parse results and find the one I wanted and copy and paste it and modify it being faster than just writing the code.
Even famous developers brag about their inability to code. DHH (RoR developer) has a tweet where he brags that he couldn't code a bubble sort without Googling it. A nested loop with a single compare and swap...and he's "proud" of the fact that he needs to Google it?
> Most people will only ever be able to speak a language as fluently as a native speaker if they learn it before they're about 13-14 years old.
Very few people both have a ton of exposure to a language and actually study the grammar and stuff as adults.
If you don't learn the grammar you will still speak like a dog after living in a country for 20 years.
A lot of people in an average company don't write hard things at their job, didn't read any textbooks etc. and spend loads of time in meetings etc.
> Very few people both have a ton of exposure to a language and actually study the grammar and stuff as adults.
Very few people actually learn to speak a language as a native speaker by "studying the grammar."
I remember people trying to learn what was and what wasn't a run-on sentence in junior high school, and being shocked that they had a hard time telling the difference.
And studying language explicitly doesn't change the brain regions used to the same that are used by a native speaker.
And that's my point. I didn't really "study" programming explicitly as much as understanding it intuitively. When exposed to a new concept, I just immediately internalize it; I don't need to use it a bunch of times and intentionally practice it. I just need to see it and it's obvious and becomes part of my tool-set.
I'm not even sure that coding _much_ faster than necessary is even required to give a 3-5x multiple on "average", let alone "worst case" developers. Some of the biggest productivity wins can be had by being able to look at requirements, knowing what's right or wrong about them, and getting everyone on the same page so the thing only needs to be made once. Being good at test and debug so problems are identified and fixed _early_ are also big wins. Lots of that is just having the experience to recognize what sort of problem you're dealing with very quickly.
Being a programming prodigy is nice, but I don't think you even really need that.
All of the things you list are the product of the experience that OP is talking about. Anyone can get there with 20 years of (sufficiently rigorous) experience by ~40, but people who start as a child have a head start and it does show.
It's probably especially obvious in the child-prodigy types because we as an industry have a tendency to force people out of IC roles by 40, so the child prodigies are the only ones who have enough time to develop 20 years of experience working directly with code.
the other factor I noticed from the days I was programming for fun vs these days where I'm programming for pay.
in those early years the tasks that you take on are probably above your skillset and fight through it, you're not accountable to anyone.
in a job you're usually hired for what you already know, +/- some margin for more gradual learning , not really that much room for moonshots. the work need to be divided into bit sized parts that you can justify to the higher-ups when needed. you have less room for exploring really non linear paths toward the solution which can be harder to explain but where you learn more.
so in the end this end up sometimes amounting to 10years of experience outside work being more impactful than 10 years at work.
,
It doesn't have to be that way. Sure, if you can get a reputation for being sharp, reliable, quick to learn, and happy to work on things you haven't seen before, it's not so hard to move into completely new tech areas.
People will hire you more for your problem solving ability, reliability, and ability to work with people than your experience with any particular tech. If you've demonstrated you can pick up a new tech area easily, it's not too hard to do it again.
That sort of thing will let you build deep experience in a lot of areas and keep you current with the latest technologies. Keeps things interesting, too. :)
I am not convinced that just starting early is all there is to it. I started Math, Sports, and Piano at like 6 years old but there are still plenty of "10x <insert activity here>" people that figuratively and literally run circles around me. Talent is a real thing.
I did spend years on a specific sport starting as a kid. I was average. There were people that first played the sport as teenagers and within a year were competitive nationally.
I was in the same math classes as some of my peers for a decade+. Some people were great, some were bad and most were somewhere in between. The kids who were exceptional at 9 were exceptional at 17.
Obviously time matters but genetics play a huge role as well. I have a family friend with 2 adopted kids and 2 biological kids. The adopted ones are average but the biological ones are very smart. Just like their parents.
That may be true of individual activities, but you trained in multiple. A fairer comparison would require the same people who best you in athletics to at least be comparable at math etc.
It's possible there are plenty of individuals 10x better than you while you are 10x better than most, due to early exposure. I wouldn't say this of sports and math necessarily, but I definitely would say it of your example of piano, language acquisition, and I would not be surprised if programming patterned with them, at least partially.
The real problem is the measurement: speed of coding (or doing any other job) or volume of worl done. Those two are actually really bad productivity measures.
> That's how 10x engineers exist, by the time they are 30, they have roughly 20 years of programming experience behind their belt instead of 10.
This is a relatively small part of it.
The majority of developers who have been programming for 20 years, maybe learned a few tricks along the way then got stuck in a local maximum.
There are a few who learn deep computer science principles and understand how to apply them to novel problems. I'm thinking of techniques like in this book:
(mainly because Peter Norvig is my goto as the paradigmatic 10x developer)
For example, in the Efficiency Issues chapter about how to optimize programs, Norvig lists writing a compiler from one language into a more efficient one. Most developers who have been working 20 years, either won't think of that, or understand how to implement it. But these are the kinds of things that can result in really outsize productivity gains.
Anyone can be a 10x engineer when they write something similar/identical to what they've written before. Other jobs are not like this. A plumber may only be 20% faster on the best days of their career.
This is the kind of half-correct pithy quote which falls down at the edge cases, e.g. if you're writing an algorithm that selects the lowest-speed car for new learner drivers and for some reason a blissfully unaware learner ends up in a McLaren GT with no brakes for lesson number 1.
> People who implement something very few people even considered or understood to be possible, which then gives amazing leverage to deliver working software in a fraction of the time.
I agree with the first part of your statement, but what really happens to such people?
In my experience (sample size greater than one), they receive some kudos, but remain underpaid, never promoted, and are given more work under tight deadlines. At least until some of them are laid off along with lower performers.
But for those who say that hard things are impossible, they seem to get along just fine. They merely declare such things as out-of-scope or lie about their roadmap.
> In my experience (sample size greater than one), they receive some kudos, but remain underpaid, never promoted, and are given more work under tight deadlines. At least until some of them are laid off along with lower performers.
100% agree, I've seen plenty of the best of the best get treated like trash and laid off at first sight of trouble on the horizon
I've always thought a x10 is one who sits back and sees a simpler way - like some math problems have an easy solution, if you can see it. Also: change the question; change the context (Alan Kay)
(And absolutely not brute-force grinding themselves away)
6.5 X 15 is only 97 hours per week not even close to the 400 hrs (5X40) per week of programming a 10X Rust programmer can provide. I jest but all this 10X stuff is getting ridiculous. They stayed in "Stealth" mode because they didn't have anything worth showing for 5 years. Doesn't sound all that productive to me. More likely what they are trying to do was hard and complicated and took a while to figure out.
They're not boasting about their current productivity, they're boasting about the one they achieved at FoundationDB when they implemented the testing, which gave them the idea to build antithesis
In my experience it often comes down to business processes. We have a guy in my extended team who knows everything about his side of the company. When I work with him I accomplish business altering deliveries in a very short amount of time, which after a week or two rarely needs to be touched again unless something in the business changes. He’s not a PO and we don’t do anything too formally because it’s just him, me and another developer + whatever business manager will benefit from the development (and a few testers from their tran). In many ways the way we work these projects are very akin to Team Topologies.
At other times I’ll be assigned projects with regular POs, Architects and business employees who barely know what it is they are doing themselves, with poorly defined tasks and all sorts of bureaucratic nonsense “agile” process methods and well spend forever delivering nothing.
So sometimes I’m a 50x developer delivering business altering changes. At other times I’m a useless cog in a sea of pseudo workers. I don’t particularly care, I get paid, but if management actually knew what was going on, and how to change it… well…
How many organisations - of any kind, startups or enterprise or unicorns or whatever - will invest so much in effort that doesn't even touch the product. Before the product exists!
I think the reluctance to invest effort in something that will give devs super-powers 6 months in the future is why we don't get all those 10x devs.
This is a perceptive observation. In my experience, so called "10x" engineers are as productive as they are because they have a process by which they practice the development on software that anticipates future problems. As a result when they check something in, they spend very little time "debugging" or "fixing bugs" with code that does what they already need it to do.
It is always very useful as an engineer to log your time, what are you working on "right now" and is it "new work" , "maintenance work", or "fixing work." Then for each log entry that isn't "new work" thinking about what you could have done that would have caught that problem before it was committed to the code base.
I find it is much better to evaluate engineers based on how often they are solving the same problem that they had before vs creating new stuff. That ratio, for me, is the essence of the Nx engineer (for 0.1 < N < 10)
The point that Wilson makes that having infrastructure/tools that push that ratio further from "repair" work to "new work" is hugely empowering to an organization.
Yep. Often I find our most accelerative work is stuff that makes testing changes easy (a very simple to bootstrap staging environment) or creates a lot of guarantees (typescript).
Was it 50x productivity due to 10x engineers, or 50x productivity due to optimized company structure? (edit: obviously, these do not need to be mutually exclusive - it's a sum of all the different parts)
It's easy to bog down even the best Nx engineers if you keep them occupied with endless bullshit tasks, meetings, (ever) changing timelines, and all that.
Kind of like having a professional driver drive a sportscar through a racetrack, versus the streets of Boston.
The 10x developer I know basically DOES seem to do 10x more than anyone else on our team. But they are working on a team that does relatively simple work and are by far the most senior person on that team.
It's like how an NBA player would be a 10x player on a college basketball team. Great to work with them but If I was in their shoes I don't know how enjoyable/engaging the work would be.
I got similar productivity boosts after learning TLA+ and Alloy.
Simulation is an interesting approach but I am curious if they ever implemented the simulation wrong would it report errors that don't happen on the target platform or fail to find errors that the target platform reports? How wide the gap is will matter... and how many possible platforms and configurations will the hypervisor cover?
I was mentally hijacked into clicking the jobs link (despite recently deciding I wasn’t going to go down that rabbit hole again!) but fortunately/unfortunately it is in-person and daily so, so flying out from Chicago a week out of the month won’t work and I don’t even have to ask!
More to the point of the story (though I do think the actual point was indeed a hiring or contracting pitch), this reminds me a lot of the internal tests the SQLite team has. I would love to hear from someone with access to those if they feel the same way.
> I was mentally hijacked into clicking the jobs link (despite recently deciding I wasn’t going to go down that rabbit hole again!) but fortunately/unfortunately it is in-person and daily so, so flying out from Chicago a week out of the month won’t work and I don’t even have to ask!
Gosh, I know it's a bit late, but I wish they'd called the product _The Prime Radiant_
Fans of Asimov's _Foundation_ series will appreciate the analogue to how this system aims to predict every eventuality based on every possible combination of events, a la psychohistory.
P.S. amazing intro post. Can't wait to try the product.
Feels like I may have brought a spoon to a gun fight, but I would have considered psychohistory to be the ultimate extrapolation of determinism, and the fact that the prime radiant is able to predict _which_ version of events will happen is because it (somehow) knows the state of the system.
Of course, to argue against myself, it would surely be based on layers of probabilities, and they say several times in the series that it can't predict low-level specific things, just high-level things. And perhaps the whole underlying question posed by the series is whether the universe really is deterministic. But anyway I don't think it's all off-base.
I'm trying to avoid diving into the hype cycle about this immediately - but this sounds like the holy grail right? Use your existing application as-is (assuming it's containerized), and simply check properties on it?
The blocker in doing that has always been the foundations of our machines: non-deterministic CPUs and operating systems. Re-building an entire vertical computing stack is practically impossible, so they just _avoid_ it by building a high-fidelity deterministic simulator.
I do wonder how they are checking for equivalence between the simulator and existing OS's, as that sounds like a non-trivial task. But, even still, I'm really bought in to this idea.
Containers are doing two jobs for us: they give our customers a convenient way to send us software to run, and they give us a convenient place to simulate the network boundary between different machines in a distributed system. The whole guest operating system running the containers is also running inside the deterministic hypervisor and under test (and it's mostly just NixOS Linux, not something weird that we wrote).
You still have to use their SDKs to write lots of integration tests (they call them “workloads”).
Then they run those tests while injecting all sorts of failures like OS failures, network issues, race and timing conditions, random number generator issues, etc.
It’s likely the only practical way today of testing for those things reliably, but you still have to write all of the tests and define your app state.
I appreciated this post. Separately from what they are talking about, I found this bit insightful:
// This limits the value of testing, because if you had the foresight to write a test for a particular case, then you probably had the foresight to make the code handle that case too.
I often felt this way when I saw developers feel a sense of doing good work and creating safe software because they wrote unit tests like expect add(2,2) = 4. There is basically a 1-1 correlation between cases you thought to test and that you coded for, which is really no better off in terms of unexplored scenarios.
I get that this has some incremental value in catching blatant miscoding and regressions down the road so it's helpful, it's just not getting at the main thing that will kill you.
I felt similarly about human QA back in my finance days that asked developers for a test plan. If the dev writes a test plan, it also only covers what the dev already thought about. So I asked my team to write the vaguest/highest level test plan possible (eg, "it should now be possible to trade a Singaporean bond" rather than "type the Singaporean bond ticker into the field, type the amount, type the yield, click buy or sell") - the vagueness made more room for the QA person to do something different (even things like tabbing vs clicking, or filling the fields out of sequence, or misreading the labels) than how the dev saw it, which is the whole point.
FoundationDB is an impressive achievement, quite possibly the only distributed database out there that lives up to its strict serializability claims (see https://jepsen.io/consistency/models/strict-serializable for a good definition). The way they wrote it is indeed very interesting and a tool that does this for other systems is immediately worth looking at.
Aphyr / Jepsen never tested FDB because, as he tweeted "their testing appears to be waaaay more rigorous than mine." We actually put a screen cap of that tweet in the blog post linked here.
Just a heads-up for anyone diving deeper into this thread - I dug into the original tweet and managed to track down the parent tweet right here: [1]. Moreover, there's a snapshot on archive.org [2] capturing the reply along with the quote in question. Interestingly, there's also a snapshot from foundationdb.com [3] that discusses the outcomes of running Jepsen tests on FDB. Worth checking out for those interested in the technical nitty-gritty.
Is it that good? I've been tasked to deploy it for sometime and it always bit me in the ass for one reason or another. And I'm not the one who use it so I don't know if it's actually good. For now I much prefer redis.
It's great, but operationally there are lots of gotchas and little guidance.
We got bitten _hard_ in production when we accidentally allowed some of the nodes to get above 90% of the storage used. The whole database collapsed into a state where it could only do a few transactions a second. Then the ops team, thinking they were clever, doubled the size of the cluster in order to give it the resources it needed to get the average utilization down to 45%; this was an unforced error as that pushed the size of the cluster outside the fdb comfort zone (120 nodes) which is itself a problem. The deed was done though and pulling nodes was not possible in this state, so slowly, slooooowly... things got fixed.
We ended up spending an entire weekend slowly, slowly getting things back into a good place. We did not lose data, but basically prod was down for the duration, and we found it necessary to _manually_ evict the full nodes one at a time over the period.
Now, this was a few years ago, and fdb has performed wickedly fast, with utter, total reliability before that and since, and to this day the ops team is butthurt about fdb.
From an engineering perspective, if you aren't using java fdb is pretty not great, since the very limited number of abstraction layers that exist are all java-centric. There are many, many issues with the maximum transaction time thing, the maximum key size and value size and total transaction size issue, the lack of pushdown predicates (e.g., filtered scans can't be done in-place which means that in AWS, they cost a lot in inter-az network charge terms and also are gated by the network performance of your instances), and so on.
What ALL of these have issues have in common is that they bite you late in the game. The storage issue bites you when you're hitting the DB hard in production and have a big data set, the lack of abstractions means that even something as finding leaked junked keys turns out to be impossible unless you were diligent to manually frame all your values so you could identify things as more than just bytes, the transaction time thing is very weird to deal with as you tend to have creeping crud aspects and the lack of libraries that instrument the transactions to give you early warning is an issue, likewise for certain kinds of key-value pairs, there's a creeping size problem - hey, this value is an index of other values; if you're not very careful up front, you _will_ eventually hit either the txn size limit or the key limit. The usual workarounds for those is to do separate transactions - a staging transaction, then essentially a swap operation and then a garbage collection transaction - but that has lots of issues overtime when coupled with application failure.
There are answers to ALL of these, manual ones. For the popular languages other than java - Go, python, maybe Ruby - there _should_ be answers for them, but there aren't. These are very sharp edges. Those java layers are _also_ _not_ _bug_ _free_. So yeah, one has a reliable storage storage layer (a topic that has come up over and over again in the last few years) but it's the layer on top of that where all the bugs are, but now with constraints and factors that are harder to reason about than the usual storage layer.
One might say, hey, SQL has all of these problems too, except no. You can bump into transaction limits, but the limits are vastly higher than fdb and the transaction time sluggishness will identify it long before you run into the "your transaction is rejected, spin retrying something that will _never_ recover" sort of issue that your average developer will eventually encounter in fdb.
That said, I love fdb as a software achievement. I just wish they had finished it. For my current project, I have designed it out. I might be able to avoid all of the sharp edges above at this point, but since we are not a java shop, I also can't rely on all the engineers to even know they exist.
It depends how you define "good". I care mostly about my distributed database being correct, living up to its consistency claims, and providing strict serializability.
Coincidence, I'm reading this and thinking about test harnesses for my package manager idea, which is really just a thin wrapper around nix, designed under the assumption that the network might partition at any moment: keep the data nearest where it's needed, refer by hash not by name, gossip metadata necessary to find the hash for you, no single points of failure.
I am thinking about state machine progressions and TLA+ style specifications which are invariants over a progression of variables.
Your package manager knows your operating system's current state and the state space of all the control flow graph through the program and configuration together can go to, it can verify that everything lines up and there will be no error when executed a bit like a compiler but without causing the Halting problem.
In TLA+ you can dump a state graph as a dot file, which I turn into a SVG and run with TLA+ graph visualiser.
Types verify possible control flow is valid at every point. We just need to add types to the operating system and file system and represent state space for deterministic verification.
You could hide packages that won't work.
The package manager would have to lookup precached state spaces or download them as part of the verification process.
To my untrained eye I get: Logs, a graph of when in time the bug happened over multiple runs and a statistical analysis which part of the application code could be invovled. The statistical analysis is nice but it is completely flat, without any hierarchical relationships making it quite hard to parse mentally.
I kind of expected more context to be provided about inputs, steps and systems that lead to the bug. Is it expected to then start adding all the logging/debugging that might be missing from the logs and re-run it to track it down? I hoped that given the deterministic systems and inputs there could be more initial hints provided.
437 comments
[ 2.8 ms ] story [ 379 ms ] threadhttps://www.cs.purdue.edu/homes/xyzhang/spring07/Papers/HPL-...
(Also, you won't be able to usefully record a hypervisor with jockey or rr, because those operate in userspace and the actual execution of guest code does not. You could probably record software cpu execution with qemu, but it would be slow)
I'm a co-founder of Antithesis.
I've found this sort of high volume random testing w. test case reduction is just a game changer for compiler testing, where there's much the same effect at quickly flushing out newly introduced bugs.
I like the subtle dig at type systems. :)
We needed that for fault tolerance, so we could have a hot synced standby. We did have to record all inputs (and outputs for sanity checking) though.
We did also get a good taste of the debugging superpowers you mention in your blog article. We could pull down a trace from a days trading and replay on our own machines, and skip back and forth in time and find the root cause of anything.
It sounds like what you have done is something similar, but with your own (AMD64) virtual machine implementation, making it fully deterministic and replayable, and providing useful and custom hardware impls (networking, clock, etc).
That sounds like a lot of hard but also fun work.
I am missing something though, in that you are not using it just for lockstep sync or deterministic replays, but you are using it for fuzzing. That is, you are altering the replay somehow to find crashes or assertion failures.
Ah, I think perhaps you are running a large number of sims with a different seed (for injecting faults or whatnot) for your VM, and then just recording that seed when something fails.
> Programming in this state is like living life surrounded by a force field that protects you from all harm. [...] We deleted all of our dependencies (including Zookeeper) because they had bugs, and wrote our own Paxos implementation in very little time and it _had no bugs_.
Being able to make that statement and back it by evidence must be indeed a cool thing.
I do not make the claim my spec has no bugs.
Formal verification is incredibly, amazingly good if you achieve it, but it's not the same as "perfect".
You can claim that your spec doesn't violate some invariants in a finite number of steps, you can't claim that the spec contains all the invariants the real system must have and that it doesn't violate them in number of steps + 1.
If you are writing software, it is almost always trying to accomplish a goal outside of itself. It is trying to solve a problem for someone, and how that problem can or should be solved is rarely perfectly clear.
The spec is supposed to map to a real world problem, and there is never going to be a way to formalize that mapping.
pp. 65-66:
> The longer I have computed, the less I seem to use Numerical Software Packages. In an ideal world this would be crazy; maybe it is even a little bit crazy today. But I've been bitten too often by bugs in those Packages. For me, it is simply too frustrating to be sidetracked while solving my own problem by the need to debug somebody else's software. So, except for linear algebra packages, I usually roll my own. It's inefficient, I suppose, but my nerves are calmer.
> The most troubling aspect of using Numerical Software Packages, however, is not their occasional goofs, but rather the way the packages inevitably hide deficiencies in a problem's formulation. We can dump a set of equations into a solver and it will usually give back a solution without complaint - even if the equations are quite poorly conditioned or have an unsuspected singularity that is distorting the answers from physical reality. Or it may give us an alternative solution that we failed to anticipate. The package helps us ignore these possibilities - or even to detect their occurrence if the execution is buried inside a larger program. Given our capacity for error-blindness, software that actually hides our errors from us is a questionable form of progress.
> And if we do detect suspicious behavior, we really can't dig into the package to find our troubles. We will simply have to reprogram the problem ourselves. We would have been better off doing so from the beginning - with a good chance that the immersion into the problem's reality would have dispelled the logical confusions before ever getting to the machine.
I suppose whether to do this depends on how rigorous one is, how rigorous certain dependencies are, and how much time one has. I'm not going to be writing my own database (too complicated, multiple well-tested options available) but if I only use a subset of the functionality of a smaller package that isn't tested well, rolling my own could make sense.
I was a co-founder of both FoundationDB and Antithesis.
One example that comes to mind where I want to roll my own thing (and am in the process of doing so) is replacing our ci/cd usage of jenkins that is solely for running qa automation tests against PR's on github. Jenkins does way way more than we need. We just need github PR interaction/webhook, secure credentials management, and spawning ecs tasks on aws...
Every time I force myself to update our jenkins instance, I buckle up because there is probably some random plugin, or jenkins agent thing, or ... SOMETHING that will break and require me to spend time tracking down what broke and why. 100% surface area for issues, whilst we use <5% of what Jenkins actually provides.
REDACTED REDACTED REDACTED REDACTED REDACTED REDACTED and REDACTED REDACTED? REDACTED REDACTED Antithesis REDACTED REDACTED REDACTED REDACTED, REDACTED REDACTED REDACTED REDACTED. REDACTED REDACTED Palantir REDACTED REDACTED REDACTED REDACTED REDACTED REDACTED REDACTED.
:-)
Currently bringing a similar kind of testing in to our workplace by writing our services to run on top of `madsim` [1]. This lets us continue writing async/await-style services in tokio but then (in tests) replace them with a deterministic executor that patches all sources of non-determinism (including dependencies that call out to the OS). It's pretty seamless.
The author of this article isn't joking when they say that the startup cost of this effort is monumental. Dealing with every possible source of non-determinism, re-writing services to be testable/sans-IO [2], etc. takes a lot of engineering effort.
Once the system is in place though, it's hard to describe just how confident you feel in your code. Combined with tools like quickcheck [3], you can test hundreds of thousands of subtle failure cases in I/O, event ordering, timeouts, dropped packets, filesystem failures, etc.
This kind of testing is an incredibly powerful tool to have in your toolbelt, if you have the patience and fortitude to invest in it.
As for Antithesis itself, it looks very very cool. Bringing the deterministic testing down the stack to below the OS is awesome. Should make it possible to test entire systems without wiring up a harness manually every time. Can’t wait to try it out!
[0]: https://sled.rs/simulation.html
[1]: https://github.com/madsim-rs/madsim?tab=readme-ov-file#madsi...
[2]: https://sans-io.readthedocs.io/
[3]: https://github.com/BurntSushi/quickcheck?tab=readme-ov-file#...
Are there public examples of what such a re-write looks like?
Also, are you working at a rust shop that's developing this way?
Final Note, TigerBeetle is another product that was written this way.
I gotta say, that is some next-level commitment to writing a good database.
Disclosure: Antithesis co-founder here.
- the company that developed Madsim (RisingWave) [0] [1] is tries hardest to eliminate non-determinism with the broadest scope (stubbing out syscalls, etc.)
- sled [2] itself has an interesting combo of deterministic tests combined with quickcheck+failpoints test case auto-discovery
- Dropbox [3] uses a similar approach but they talk about it a bit more abstractly.
Sans-IO is more documented in Python [4], but str0m [5] and quinn-proto [6] are the best examples in Rust I’m aware of. Note that sans-IO is orthogonal to deterministic test frameworks, but it composes well with them.
With the disclaimer that anything I comment on this site is my opinion alone, and does not reflect the company I work at —— I do work at a rust shop that has utilized these techniques on some projects.
TigerBeetle is an amazing example and I’ve looked at it before! They are really the best example of this approach outside of FoundationDB I think.
[0]: https://risingwave.com/blog/deterministic-simulation-a-new-e...
[1]: https://risingwave.com/blog/applying-deterministic-simulatio...
[2]: https://dropbox.tech/infrastructure/-testing-our-new-sync-en...
[3]: https://github.com/spacejam/sled
[4]: https://fractalideas.com/blog/sans-io-when-rubber-meets-road...
[5]: https://github.com/algesten/str0m
[6]: https://docs.rs/quinn-proto/0.10.6/quinn_proto/struct.Connec...
As cool as all this is, I can't stop but wonder how often the culture of micro-services and distributed computing is ill advised. So much complexity I've seen in such systems boils down to calling a "function" is: async, depends on the OS, is executed at some point or never, always returns a bunch of strings that need to be parsed to re-enter the static type system, which comes with its own set of failure modes. This makes the seemingly simple task of abstracting logic into a named component, aka a function, extremely complex. You don't need to test for any of the subtle failures you mentioned if you leave the logic inside the same process and just call a function. I know monoliths aren't always a good idea or fit, at the same time I'm highly septical whether the current prevalence of service based software architectures is justified and pays off.
You can't get away from distributed computing, unless you get away from computing. A modern computer isn't a single unit, it's a system of computers talking to each other. Even if you go back a long time, you'll find many computers or proto-computers talking to each other, but with a lot stricter timings, as the computers are less flexible.
If you save a file to a disk, you're really asking the OS (somehow) to send a message to the computer on the storage device, asking it to store your data, and it will respond with success or failure and it might also write the data. (sometimes it will tell your os success and then proceed to throw the data away, which is always fun)
That said, keeping things together where it makes sense, is definitely a good thing.
As I recall it, the name meant two things:
1. Our “autonomous testing” approach is the opposite, or the antithesis, of flaky and unreliable testing methodologies.
2. You can think of our product as standing in dialectical opposition to buggy customer software, pointing out its internal contradictions (bugs) and together synthesizing a new, bug-free software product. (N.b.: I’ve never actually read Hegel.)
We did note the resonance with Hypothesis (a library I like a lot!) at the time, but it was just an added bonus :).
Disclosure: I am a co-founder of Antithesis.
Do you have any plans to let small open source teams use the project for free? Obviously you have bills to pay and your customers are happy to do that, but I was wondering if you'd allow open source projects access to your service once a week or something.
Partly because I want to play with this and I can't see my employer or client paying for it! But also it fits neatly into "DX", the Developer Experience, i.e. making the development cycle as friction free for devs as possible. I'm DevOps with a lifelong interest in UX, so DX is something I'm excited about.
In the future, we will probably have a multi-tenant offering that's easier for open source projects to adopt. In the meantime, if your project is cool and would benefit from our testing, you can try to get our research team interested in using it as part of the curriculum that makes our platform smarter.
Disclosure: I'm an Antithesis co-founder.
The system can certainly revisit a previous simulated moment and replay it. And we have some pretty cool things using that capability as a primitive. Check out the probability chart in the bug report linked from the demo page: https://antithesis.com/product/demo
Seriously: impressive product revelation.
I did a "Learn with Jason" show discussion that covered the concepts of Replay, how to use it, and how it works:
- https://www.learnwithjason.dev/travel-through-time-to-debug-...
Not only is the debugger itself time-traveling, but those time-travel capabilities are exposed by our backend API:
- https://static.replay.io/protocol/
Our entire debugging frontend is built on that API. We've also started to build new advanced features that leverage that API in unique ways, like our React and Redux DevTools integration and "Jump to Code" feature:
- https://blog.replay.io/how-we-rebuilt-react-devtools-with-re...
- https://blog.isquaredsoftware.com/2023/10/presentations-reac...
- https://github.com/Replayio/Protocol-Examples
Naturally I’d prefer professional programmers learn the cognitive tools for manageably reasoning about nondeterminism, but they’ve been around over half a century and it hasn’t happened yet.
What’s really interesting to me is that the simulation adequately replicates the real network. One of the more popular criticisms of analytical approaches is sone variant of: yeah, but the real network isn’t going to behave like your model. Which by the way is an entirely plausible concern for anyone who has messed with that layer.
It’s not an either-or here, though. Part of the challenge is you’re not always thinking about all the non-determinisms in your code, and the interconnections between your code and other code (whose behavior you can sometimes only assume) can make that close to impossible. Part of that is the “your model of the network” critique, but also part of that is “your model of how people will use your software” isn’t necessarily correct either.
Lays the foundation (get it?) for who the people are and what they've built.
Then explains how the current thing they are building is a result of the previous thing. It feels that they actually want this problem solved for everyone because they have experienced how good the solution feels.
Then tells us about the teams (pretty big names with complex systems) that have already used it.
All of these wrapped in good writing that appeals to developers/founders. Landing page is great too!
Here's my attempt at a more complete answer: think of the story of the blind men and the elephant. There's a thing, called fuzzing, invented by security researchers. There's a thing, called property-based testing, invented by functional programmers. There's a thing, called network simulation, invented by distributed systems people. There's a thing, called rare-event simulation, invented by physicists (!). But if you squint, all of these things are really the same kind of thing, which we call "autonomous testing". It's where you express high-level properties of your system, and have the computer do the grunt work to see if they're true. Antithesis is our attempt to take the best ideas from each of these fields, and turn them into something really usable for the vast majority of software.
We believe the two fundamental problems preventing widespread adoption of autonomous testing are: (1) most software is non-deterministic, but non-determinism breaks the core feedback loop that guides things like coverage-guided fuzzing. (2) the state space you're searching is inconceivably vast, and the search problem in full generality is insolubly hard. Antithesis tries to address both of these problems.
So... is it fuzzing? Sort of, except you can apply it to whole interacting networked systems, not just standalone parsers and libraries. Is it property-based testing? Sort of, except you can express properties that require a "global" view of the entire state space traversed by the system, which could never be locally asserted in code. Is it fault injection or chaos testing? Sort of, except that it can use the techniques of coverage guided fuzzing to get deep into the nooks and crannies of your software, and determinism to ensure that every bug is replayable, no matter how weird it is.
It's hard to explain, because it's hard to wrap your arms around the whole thing. But our other big goal is to make all of this easy to understand and easy to use. In some ways, that's proved to be even harder than the very hard technological problems we've faced. But we're excited and up for it, and we think the payoff could be big for our whole industry.
Your feedback about what's explained well and what's explained poorly is an important signal for us in this third very hard task. Please keep giving it to us!
[0] https://www.youtube.com/watch?v=4fFDFbi3toc [1] https://thestrangeloop.com/2014/testing-distributed-systems-...
Doesn't Antithesis rely on the fact that software is always deterministic? Reproducibility appears to be its top selling feature – something that wouldn't be possible if software were non-deterministic.
* Offer only good for x86-64 software that runs on Linux whose dependencies you can install locally or mock. The first two restrictions we will probably relax someday.
However, if we pretend that software was somehow able to be non-deterministic, it would be able to evade your deterministic computer. But since software is always deterministic, you just have to guarantee determinism in the inputs.
>But since software is always deterministic, you just have to guarantee determinism in the inputs.
This is technically correct, but that's a very load-bearing "just". A lot of things would have to count as inputs. Think about execution time, for example. CPUs don't execute at the same speed all the time because of automatic throttling. Network packets have different flight times. Threads and processes get scheduled a little differently. In distributed/concurrent systems, all this matters. If you run the same workload twice, observable events will happen at different times and in different orders because of tiny deviations in initial conditions.
So yes, if you consider the time it takes to run every single machine instruction as an "input", then software is deterministic given the same inputs. But in the real world that's not actionable. Even if you had all those inputs, how are you going to pass them in? For all intents and purposes most software execution is non-deterministic.
The Antithesis simulation is deterministic in this way though. It is in charge of how long everything takes in "simulated time", right down to the running times of individual CPU instructions. Everything observable from within the simulation happens the exact same way, every time. You can compare a memory dump at the same (simulated) instant across two different runs and they will be bit-for-bit identical.
Sure. A good example. Execution time – more accurately, execution speed – isn't a property of software. For example, as you point out yourself, you can alter the execution speed without altering the software. It is, indeed, an input.
> Even if you had all those inputs, how are you going to pass them in?
Well, we know how to pass them in non-deterministically. That's how software is able to do anything.
Perhaps one could create a simulated environment that is able to control all the inputs? In fact, I'm told there is a company known as Antithesis working on exactly that.
Is the challenge here the same as with digital simulations of electronic circuits? That is, at the end of the day analog physics becomes confounding? Or are you doing deterministic simulation of random RF noise as well?
So Jefferson just says, let's take your I/O-ful process, split it a message-passing actor model, and monitor all the messages going in and coming out. The messages coming out, they won't necessarily do what they're supposed to do yet, they'll just be recorded with a plus sign and a virtual timestamp, and by assumption eventually you'll block on some response. So we have a bunch of recorded message timestamps coming in, we have your recorded messages going out.
Well, there's a problem here, which is that if we have multiple actors we may discover that their timestamps have traveled out-of-order. You sent some message at t=532 but someone actually sent you a message at t=231 that you might have selected instead of whatever you actually selected to send the t=532 message. (For instance in the OS case, they might have literally sent a SIGKILL to your process and you might not have sent anything after that.) That's what the plus sign is for, indirectly: we can restart your process from either a known synchronization state or else from the very beginning, we know all of its inputs during its first run so we have "determinized" it up past t=231 to see what it does now. Now, it sends a new message at say t=373. So we use the opposite of +, the minus sign, to send to all the other processes the "undo" message for their t=532 message, this removes it from their message buffer: that will never be sent to them. And if they haven't hit that timestamp in their personal processing yet, no further action is needed, otherwise we need to roll them back too. Doing so you determinize the whole networked cluster.
The only other really modern implementation of these older ideas that I remember seeing was Haxl[2], a Haskell library which does something similar but rather than using a virtual time coordinate, it just uses a process-local cache: when you request any I/O, it first fetches from the cache if possible and then if that's not possible it goes out, fetches the data, and then caches it. As a result you can just offer someone a pre-populated cache which, with these recorded inputs, will regenerate the offending stack trace deterministically.
1: https://dl.acm.org/doi/10.1145/3916.3988
2: https://github.com/facebook/Haxl
Disclosure: Antithesis co-founder.
It's definitely true though that a big problem with backend is that you can't easily treat it as a whole system for test purposes.
or starting a bidding war
Would it work for debugging, say, Notepad on Windows?
It's hard to understand these complex concepts via language alone.
Diagrams would be a huge help to understand how this system of testing works compared to existing testing concepts
I haven't heard of deterministic testing before. Nor have I heard of FoundationDB or the related things. And I went from knowing zero things about them to getting impressed and interested. This led me to go into their docs, blogs, landing page, etc. to know more.
The linked article is 3/4 about some history and rationale before it actually tells you what they build.
It's like those pesky recipe blogs that tell you about the authors childhood, when you just want to make vegan pancakes.
It would be nice to see some actual use cases and examples.
Instead, the writer just name-dropped a few big companies and claimed to have a revolutionary product that works magically. Then include the typical buzzwords like '10x programmer' and 'stealth mode'. The latter doesn't make sense because they also name-drop clients.
Also, stealth mode just means your company isn’t public, you can still have clients.
Having that context puts the post in a much better perspective. It's definitely an introduction post (the company has been developing this in stealth mode for the past few years), but it is most certainly _not_ a marketing post. These people developed extremely novel testing techniques for FoundationDB and are now generalizing them to work with any containerized application.
It's a big deal.
I'm reading the paper. It's very intriguing but so is the marketing material for the new Tesla. I need to work with things like this before I believe the claims.
Too many people here on HN are getting caught up in a barely-tested new technology and hailing it as some revolution.
Using the “unsavory” euphemism for unethical and illegal violence is somewhat of a deception, is it not?
Like this: https://docs.gitlab.com/ee/user/application_security/coverag...
It won't tell you whether the software works correctly, it will just tell you if it raises an exception or crashes.
Put a fuzzer on Chrome for example, you won't catch most of the issues it has, though Chrome actually has tons of bugs and issues, but you may find security issues if you devote a big enough budget to run your fuzzer long time enough to cover all the branches.
So it's good in the case where you use "exceptions as tests", where any minor out-of-scope behavior raises an exception and all the cases are pre-planned (a bit like you baked-in runtime checks, and the fuzzer explores them)
Not really. I have friends who work for a startup that's been in "stealth" for 20 years. Stealth is a business model not a phase.
I feel like the idea of the legendary "10x" developer has been bastardized to just mean workers who work 15 hours a day 6.5 days a week to get something out the door until they burn out.
But here's your real 10x (or 50x) productivity. People who implement something very few people even considered or understood to be possible, which then gives amazing leverage to deliver working software in a fraction of the time.
They work best on their own projects with nobody else in their way, no colleagues, no managers, but that's not most jobs. Once you're part of a team, you can't do too much work yourself no matter how good you are, as inevitably the other slower/weaker team members will slow you down as you'll fight dealing with the issues they introduce into the project or the issues from management, so every team moves at the speed of the lowest common denominator no matter their rockstars.
Yeah but this isn't something scalable that can happen regularly as part of your job description. Like most jobs/companies don't have so many low hanging fruits to pick that someone can speed of build by orders of magnitude on a weekly basis. It's usually a one time thing. And one time things don't usually make you a 10x dev. Maybe you just got lucky once to see something others missed.
And often times at big places most people know where the low hanging fruits are and can fix them, but management, release schedules and tech debt are perpetually in the way.
IMHO what makes you a 10x dev is you always know how to unblock people no matter the issue so that the project is constantly smooth saling, not chasing orders of magnitude improvements unicorns.
I personally think that some people can provide “10x” (arbitrary) the value on occasion, like the low hanging fruit you said. I also believe some people are slightly more skilled than others, and get more results out of their work. That said, there are so many ways for somebody to have an impact that doesn’t have to immediate, that I find the term itself too prevalent.
Agreed, there is too much effort going into the "superstars" theme, but there are definitely people who get 10x done in the same time as others.
Some of those people not only get things done much quicker, but they also get it done with better quality than an amateur, with less mistakes, throwing away less material, sometimes with more safety.
This is definitely more than 10x better. And there are some real hacks doing those kinds of jobs. I find programming to be not different than that.
You and I have worked at very different organizations. Everywhere I've been has had insane levels of inefficiency in literally every process.
In processes yes, not in code, and solo 10x devs alone can't fix broken processes as those are a the effect of broken management and engineering culture.
People know where the inefficiencies are, but management doesn't care.
But then you just move on to the next highest leverage task.
Over time this produces funny effects, like super-big 20 point task done in few days because wrong person started working on it.
"The truest 10x engineer I ever encountered was..."
https://www.construx.com/blog/the-origins-of-10x-how-valid-i...
You, on the other hand, are claiming that 10x engineers are 10 times more productive than the worst engineers. Completing a programming challenge in a 10th of the time is not the same as being 10 times more productive, and obviously your usage can't be an explanation, even as one you made up on the spot, as the math doesn't add up.
That study also does not exist in isolation:
https://www.construx.com/blog/the-origins-of-10x-how-valid-i...
Because it was? Do you have a better way to repeatedly test performance? And yes, the study's intent was to look at performance, not productivity. It's even right in the title. Not sure where you dreamed up the latter.
What we do know is that the worst engineers provide negative productivity. If 1x is the worst engineer, then let's for the sake of discussion denote x as -1 in order for the product to be negative. Except that means the 10x engineer provides -10 productivity, actually making them the worst engineer. Therein lies a conflict.
What we also know is that best engineer has positive productivity, so that means the multiplicand must always be positive. Which means that it is the multiplier that must go negative, meaning that a -1x and maybe even a -10x engineer exists.
Throw into the mix the fact that productivity is hard to measure as soon as more than one person works on something and that doesn't even begin to consider the economical aspects of software.
And even when ignoring this point, there's that pesky short-term vs long-term thing.
Also, how do you define the term "productivity"? I was assuming that you mean somethint along the lines of (indirect, if employed) monetary output.
The 10x developer originated from a study that measured performance. The 10x developer being able to do a task in a 10th of the time is quite conceivable and reflects what the study found. I'm sure we've all seen a developer take 10 hours to do a job that would take another developer just 1 hour. Nobody is doing it in negative hours, so the math works.
But performance is not the same as productivity.
Performance. That is what the study that found a 10x performance difference observed. There is no mention of productivity in the study. If anyone has tried to study productivity, they most certainly have not come up with a 10x moniker. It seems productivity was mentioned in this thread only because it also happens to start with the letter 'p' and someone got confused.
Of course, that doesn't actually matter as the original study found a 10x difference in measuring performance, not productivity. There is nothing out there to suggest that some developers are 10x more productive outside of those who mixed up their p words. We're not actually talking about productivity; that was just a mistake. If one were to study productivity, I expect they would find that some engineers are many orders of magnitude more productive than the least productive engineers.
I reject that assertion.
> performance, not productivity
You keep saying this like it's a slam dunk refutation, but performance and productivity are highly related.
Because you don't believe the worst developer contributed anything to GPT? Sure, in reality that's no doubt true, but it was only ever meant to be illustrative.
> but performance and productivity are highly related.
Not in any meaningful way. The study found that the fastest developer can perform a set of defined tasks in a 10th of the time of the slowest developer. That is what the 10x developer refers to. But being fast doesn't mean being productive.
Come to my backyard and we will each dig a hole of equal size. Let's assume that you can dig the hole in a 10th of the time I can – that you are the 10x hole digger. But, no matter how fast you are, neither of us will be productive.
This is especially common when dealing with historical or academic definitions versus common modern usage. "Evolution" particularly annoys me.
You should avoid using the term, rather than using a definition at odds with common usage. Your usage is confusing - and that is why you are getting push-back.
The definition you have given is nonsensical - it can't be consistent over time or between companies because it depends on finding a minimum in a group. And a value that is strongly dependent on the worst developer is useless because it mostly measures how bad the worst developer is - it doesn't say anything about how good the best developer is.
It's not about the hours.
3x bonus multiplies (Redefines Expectations) are extremely uncommon. Level increases certainly help but like, L7 only makes ~3x what L5 does -- not 10x. And there are few L7s and very few L8+.
In the end it doesn't matter, whole team could be laid off at once.
Productivity has no inherent value - like efficiency and perfection, it is necessarily of something else. Its value is entirely derived.
Which is weird, since so many executives are working in a VC-funded environment, and internal work should be “venture funded” as well.
I worked for a company where I almost single handedly built a product that resulted in tens of millions of dollars in sales. I got a nice 'atta boy' for it, but my future ideas were often overridden by someone in management who 'knew better'. After the management changed, I found myself in a downsizing event once I started criticizing them for a lack of innovation.
https://www.didgets.com
I'd be happier if industry cares more for team productivity - I have witnessed how rewarding "10x" individuals may lead to perverse results on a wider scale, a la Cobra Effect. In one insidious case, our management-enabled, long-tenured "10x" rockstar fixed all the big customer-facing bugs quickly, but would create multiple smaller bugs and regressions for the 1x developers to fix while he moved to the next big problem worthy of his attention. Everyone else ended up being 0.7x - which made the curse of an engineer look even more productive comparatively!
Because he was allowed to break the rules, there was a growing portion of the codebase that only he could work on - while it wasn't Rust, imagine an org has a "No Unsafe Rust" rule that is optional to 1 guy. Organizations ought to be very careful how they measure productivity, and should certainly look beyond first-order metrics.
bingo, well said. Worked on a team like this with a “principal” engineer who’d work very fast with bug-ridden work like this simply because he had the automatic blessing from on high to do whatever he wanted. My unfortunate task was to run along behind him and clean up, which to my credit I think I did a pretty good job at, but of course these types can only very rarely acknowledge/appreciate that.
Eventually he got super insecure/threatened and attempted to push me out along with whoever else he felt was a threat to his fiefdom.
[1] I preserved a copy of it on my (no-advertising or monetization) blog here: https://realmensch.org/2017/08/25/the-parable-of-the-two-pro...
In the early computing era there was a competition. Something like take some input and produce an output. One programmer made a large program in (IIRC) Fortran with complex specifications documentation etc. The other used shell pipes, sort, and a small handful or two of other programs in a pipeline to accomplish the same task in like 10 developer min.
[1] https://homepages.cwi.nl/~storm/teaching/reader/BentleyEtAl8... [2] https://www.google.com/search?q=knuth+vs+mcilroy
I'm trying to copy-in the program as it might have existed, with some obvious updates to work in today's shells ...
Alternately (escapes not yet tested) $ tr -cs A-Za-z \012 "${INPUTFILEHERE:-/dev/stdin}" | tr A-Z a-z | sort | uniq -c | sort -rn | sed ${MAXWORDSHERE:-100}qEdited: Removed some errors likely induced by OCR / me not catching that in the initial transcription from the browser view of the file.
[For those who may not follow all the links: Bentley asked Knuth to write a program in Pascal (WEB) to illustrate literate programming—i.e. explaining a long complicated program—and so Knuth wrote a beautiful program with a custom data structure (hash-packed tries). Bentley then asked McIlroy to review the program. In the second half of the review, McIlroy (the inventor of Unix pipes) questioned the problem itself (the idea of writing a program for scratch), and used the opportunity to evangelize Unix and Unix pipes (at the time not widely known or available).]
http://catb.org/~esr/writings/unix-koans/ten-thousand.html
"""“And who better understands the Unix-nature?” Master Foo asked. “Is it he who writes the ten thousand lines, or he who, perceiving the emptiness of the task, gains merit by not coding?”"""
I drank from the OO koolaid at one point. I was really into building things up using OOD and creating extensible, flexible code to accomplish everything.
And when I showed some code I'd written to my brother, he (rightly) scoffed and said that should have been 2-3 lines of shell script.
And I was enlightened. ;)
Like, I seriously rebuilt my programming philosophy practically from the ground up after that one comment. It's cool having a really smart brother, even if he's younger than me. :)
They probably deserve more praise when they do guess correctly but would anyone really know when it happens?
Sometimes I'm appreciated, and managers actually realize what they have when I create something for them. Frequently I accomplish borderline miracles and a manager will look at me and say, "OK, what about this other thing?"
My first job out of college, I was working for a company run by a guy who said to me, "Programmers are a dime a dozen."
He also said to me, after I quit, after his client refused to give him any more work unless he guaranteed that I was the lead developer on it, "I can't believe you quit." I simply shrugged and thought, "Maybe you shouldn't have treated me like crap, including not even matching the other offer I got."
I've also made quite a lot of money "Rescuing Small Companies From Code Disasters. (TM)" ;) Yes, that's my catch phrase. So I've seen the messes that teams often create.
The "incompetent" team code description in the story is practically prescient. I've seen the results of exactly that kind of management and team a dozen times. Things that, given the same project description, I could have created in 1/100 the code and with much more overall flexibility. I've literally thrown out entire projects like that and replaced them with the much smaller, tighter, and faster code that does more than the original project.
So all I can say is: Find better teams to work with if you think this is fiction. This resonates with me because it contains industry Truth.
I revisited earlier thought: communication is a 2 man job, one is to not make an effort to understand while the other explains things poorly. It always manages to never work out.
Periodically I thought about the puzzle and was eventually able to explain it such that people thought it was brilliant ~ tho much to complex to execute.
I thought about it some more, years went by and I eventually managed to make it easy to understand. The response: "If it was that simple someone else would have thought of it." I still find it hilarious decades later.
It pops to mind often when I rewrite some code and it goes from almost unreadable to something simple and elegant. Ah, this must be how someone else would have done it!
This is a good exclamation :D
And it's a poignant story. Thanks for sharing.
― Abraham Lincoln
I have started to follow this 'lately' (for a decade) and it has worked miracles. As for the anxious managers/clients, I keep them updated of the design/documentation/though process, mentioning the risks of the path-not-taken, and that maintain their peace of mind. But this depends heavily on the client and the managers.
> The manager of Charles has by now [become] tired of seeing him goof off.
"The manager has tired of Charles" is as correct as "the manager has become tired of Charles". To tire is a verb. The square bracket correction is unnecessary and arguably makes the sentence worse.
Or it was already there in whatever source I managed to copy it from. No idea.
I don't agree with that, there are a _lot_ of completely crap developers and they get put into positions where even the ones capable of doing so aren't allowed to because it's not on a ticket.
I've seen some thing.
Hence not wasting time on social networks.
> a 10x only has to put in a 48 minute day
Nobody would call this person "10x".
I don't think that tracks. Smart, productive, hard working people don't work 9-5. They work every hour they can, breaking only when they have pushed themselves to the limit. The limit can be hit at any hour. There is no magical property of the universe that gives people unlimited stamina during the hours of 9-5.
> Nobody would call this person "10x".
I'm not sure they would call anyone that, to be fair. A "10x developer" who also puts in 8 hours alongside the 1x developers isn't a 10x developer, he would be called a sucker.
Another flaw in his thinking: brain cycles and sub-conscious processing.
I'm in the middle of a hard problem right now. I ran out of ideas, and opened HN about half an hour ago. In that time, without "trying", I've had two new ideas - one sent me back to my notes, which revealed that my original thinking was flawed; the second sent me to documentation, which suggested a new route to pursue. I'm digesting the implications of that while I write this.
Beating my head against the problem directly for thirty minutes would have been less productive. (Though if I wasn't WFH I would have, and also been miserable, and learned less about the industry than I have from this thread. So there's that.)
I'm far from a 10x anything, but I don't have the only brain which works this way.
Nah. I get what you're saying, and it's a great idea. Sometimes I do go for a walk. More often I do dishes. Those are both, however, higher-commitment, more time-consuming activities than flipping to a different window. If I remember correctly, my sticking point that day felt like a small one, and so the distraction / break I felt like I needed was correspondingly small.
"Hmmm, there's another way to accomplish this" being the 10x. Doing things faster is not it.
Also, their professional experience is much greater. Sure, their initial jobs at 15 are the occassional weird gig for the uncle/aunt or cousin/nephew but they get picked up by professional firms at 18 and do a job next to their CS studies.
At least, that's how it used to be. Not sure if this is still happening due to the new job environment, but this was the reality from around 2004 to 2018.
For 10x engineers to exist, all it takes is a few examples. To me, everyone is in agreement that they seem to be rare. I point to a public 10x engineer. He'd never say it himself, but my guess is that this person is a 10x engineer [1].
If you disagree, I'm curious how you'd disagree. I'm just a blind man touching a part of the elephant [2]. I do not claim to see the whole picture.
[1] https://bellard.org/ (the person who created JSLinux)
[2] https://en.wikipedia.org/wiki/Blind_men_and_an_elephant - if you don't know the parable, it's a fun one!
By the time I joined my first real company at age 21, I was ready to start putting a lot of this stuff into place. I joined a small med device software company which had a great product but really no strong software engineering culture: zero unit tests, using CVS with no branches, release builds were done manually on the COO's workstation, etc.
As literally the most junior person in the company I worked through all these things and convinced my much more senior colleagues that we should start using release branches instead of "hey everybody, please don't check in any new code until we get this release out the door". I wrote automated build scripts mostly for my own benefit, until the COO realized that he didn't have to worry about keeping a dev environment on his machine, now that he didn't code any more. I wrote a junit-inspired unit testing framework for the language we were using (https://en.wikipedia.org/wiki/IDL_(programming_language) - like Matlab but weirder).
Without my work as a "10x junior engineer", the company would have been unable to scale to more than 3 or 4 developers. I got involved in hiring and made sure we were hiring people who were on board with writing tests. We finally turned into a "real" software company 2 or 3 years after I joined.
10x engineers exist. To be fair, it does depend which software engineer you see as "the standard software engineer", but if I take myself as a standard (as an employed software engineer with 5 years of experience), then 10x software engineers exist.
I was coding before age 10 and delivering homework in website form any chance I got. Coding was just another video game to me. And yet, I didn’t even end up being a programmer, let alone 10x. I was given bad advice from highschool career advisors, lacked the family connections that knew better to give me odd tech jobs, had disinterested teachers at best and ones that insulted students who asked clarification questions at worst. I took an engineering class in highschool where i loved the content but I felt like I didn’t belong and dropped from it convinced tech was not for me after all (and then proceeded making games in unreal in my free time). For every person supporting my interest in CS there were 10 telling/nudging me to give up, so i was eventually worn out and gave it up. I still code 25 years later but I am far behind now instead of ahead.
Anyway, this is a longwinded way to say it takes privilege to be a 10x programmer, not everyone who gives themselves a head start will succeed.
I became friends with Dave our CTO when I was 5 or 6, we were neighbors. He'd already started coding little games in Basic (this was 1985). Later in our friendship, like when I was maybe 10, I asked him if he could help me learn to code, which he did. After a week or two I had made some progress but compared what I could do to what he was doing and figured "I guess I just started too late, what's the point?".
I found out later that most people didn't start coding till late HS or college! It worked out though - I'm programmer adjacent and have taken care of the business side of our projects through the years :)
I am not sure there is 10x difference, but there is at least 5x difference in performance, in favor of fresh college grad, and they are now working on the more complex tasks too.
The sad part is our hiring is still heavily in "senior engineer with lots of experience" phase, and intership program has been canceled.
At my previous company I had 2 first-job juniors and 4 or 5 interns that were outstanding.
(however the last one was totally terrible and totally killed my hiring credibility hah, but it was a 80% success rate still)
I find that there are too many pretenders, though. I get way too many people with 15 years of experience and ChatGPT resumes that just can't code at all :/
The top-performing experienced developers outshone all the graduates. However, the less effective experienced developers were on par with the graduates, showing no significant difference in performance.
The takeaway for me is that simple anecdotes are not very informative. Over time and with a larger sample size, experienced individuals tend to perform better. Nonetheless, some graduates will also become exceptionally skilled.
Graduates are more cost-effective. Experienced professionals require less oversight. If they need substantial guidance, they don't truly qualify as senior, as opposed to their resume that says, at 25 they have been a CTO for 10 years).
You probably surf the internet 10x faster than your parents. Yes you've probably had more exposure than them, but you could probably teach them how to do it just as fast. But would they want to learn and would they actually adapt what you taught them?
No: It's not because they have 10 more years of experience. Read "The Mythical Man Month." That's the book that popularized the concept that some developers were 5-25x faster than others. One of the takeaways was that the speed of a developer was not correlated with experience. At all.
That said, the kind of person who can learn programming at 12 might just be the kind of person who is really good at programming.
I started learning programming concepts at 11-12. I'm not the best programmer I know, but when I started out in the industry at 22 I was working with developers with 10+ years of (real) experience on me...and I was able to come in and improve on their code to an extreme degree. I was completing my projects faster than other senior developers. With less than two years of experience in the industry I was promoted to "senior" developer and put on a project as lead (and sole) developer and my project was the only one to be completed on time, and with no defects. (This is video game industry, so it wasn't exactly a super-simple project; at the time this meant games written 100% in assembly language with all kinds of memory and performance constraints, and a single bug meant Nintendo would reject the image and make you fix the problem. We got our cartridge approved the first time through.)
Some programmers are just faster and more intuitive with programming than others. This shouldn't be a surprise. Some writers are better and faster than others. Some artists are better and faster than others. Some architects are better and faster than others. Some product designers are better and faster than others. It's not all about the number of hours of practice in any of these cases; yes, the best in a field often practices an insane amount. But the very top in each field, despite having similar numbers of hours of practice and experience, can vary in skill by an insane amount. Even some of the best in each field are vastly different in speed: You can have an artist who takes years to paint a single painting, and another who does several per week, but of similar ultimate quality. Humans have different aptitudes. This shouldn't even be controversial.
I do wonder if the "learned programming at 12" has anything to do with it: Most people will only ever be able to speak a language as fluently as a native speaker if they learn it before they're about 13-14 years old. After that the brain (again, for most people; this isn't universal) apparently becomes less flexible. In MRI studies they can actually detect differences between the parts of the brain used to learn a foreign language as an adult vs. as a tween or early teen. So there's a chance that early exposure to the right concepts actually reshapes the brain. But that's just conjecture mixed with my intuition of the situation: When I observe "normal" developers program, it really feels like I'm a native speaker and they're trying to convert between an alien way of thinking about a problem into a foreign language they're not that familiar with.
AND...there may not be a need to explicitly PROGRAM before you're 15 to be good at it as an adult. There are video games that exercise similar brain regions that could substitute for actual programming experience. AND I may be 100% wrong. Would be good for someone to fund some studies.
I always thought that people who start at 12 and keep at it are good because they really love it.I see people who struggle a lot with learning, and it's because they hate it but are doing it for other reasons.
I started programming at age 7 and it's true that the way code forms in my head feels similar to the way words form when I'm writing or speaking in English. In the same way that I don't stop and consciously figure out whether to use the past or present tense while I'm talking, I usually don't consciously think about, say, what kind of looping construct I'm about to use; it's just the natural-feeling way to express the idea I'm trying to convey. The idea itself is kind of already in the form of mental code in the same way that my thoughts are kind of already in English if I'm speaking.
But... maybe that's how it is for everyone, even people who learned later? I only know how it is in my own head.
And at least some people clearly don't. I was talking to one guy who said that even for a simple for-each loop it was way faster for him to "Google the code he needs and modify it" than to write it. This boggled me. I couldn't imagine being able to Google and parse results and find the one I wanted and copy and paste it and modify it being faster than just writing the code.
Even famous developers brag about their inability to code. DHH (RoR developer) has a tweet where he brags that he couldn't code a bubble sort without Googling it. A nested loop with a single compare and swap...and he's "proud" of the fact that he needs to Google it?
I have no words.
Very few people both have a ton of exposure to a language and actually study the grammar and stuff as adults. If you don't learn the grammar you will still speak like a dog after living in a country for 20 years. A lot of people in an average company don't write hard things at their job, didn't read any textbooks etc. and spend loads of time in meetings etc.
Very few people actually learn to speak a language as a native speaker by "studying the grammar."
I remember people trying to learn what was and what wasn't a run-on sentence in junior high school, and being shocked that they had a hard time telling the difference.
And studying language explicitly doesn't change the brain regions used to the same that are used by a native speaker.
And that's my point. I didn't really "study" programming explicitly as much as understanding it intuitively. When exposed to a new concept, I just immediately internalize it; I don't need to use it a bunch of times and intentionally practice it. I just need to see it and it's obvious and becomes part of my tool-set.
Being a programming prodigy is nice, but I don't think you even really need that.
It's probably especially obvious in the child-prodigy types because we as an industry have a tendency to force people out of IC roles by 40, so the child prodigies are the only ones who have enough time to develop 20 years of experience working directly with code.
in those early years the tasks that you take on are probably above your skillset and fight through it, you're not accountable to anyone.
in a job you're usually hired for what you already know, +/- some margin for more gradual learning , not really that much room for moonshots. the work need to be divided into bit sized parts that you can justify to the higher-ups when needed. you have less room for exploring really non linear paths toward the solution which can be harder to explain but where you learn more.
so in the end this end up sometimes amounting to 10years of experience outside work being more impactful than 10 years at work. ,
People will hire you more for your problem solving ability, reliability, and ability to work with people than your experience with any particular tech. If you've demonstrated you can pick up a new tech area easily, it's not too hard to do it again.
That sort of thing will let you build deep experience in a lot of areas and keep you current with the latest technologies. Keeps things interesting, too. :)
The combination of 10-15 years of experience and the energy/time of 20s is very powerful.
And when we're talking about sports, genetics matter as well (depending on each one)
When we're talking brains, while genetics also matter, assuming normal (whatever that is) brain, the plasticity changes a lot how it operates.
So, the 10 years thing is definitely a big if not the biggest part. In my opinion. Would love to see studies if any exist out there on this
I was in the same math classes as some of my peers for a decade+. Some people were great, some were bad and most were somewhere in between. The kids who were exceptional at 9 were exceptional at 17.
Obviously time matters but genetics play a huge role as well. I have a family friend with 2 adopted kids and 2 biological kids. The adopted ones are average but the biological ones are very smart. Just like their parents.
This is a relatively small part of it.
The majority of developers who have been programming for 20 years, maybe learned a few tricks along the way then got stuck in a local maximum.
There are a few who learn deep computer science principles and understand how to apply them to novel problems. I'm thinking of techniques like in this book:
https://www.everand.com/book/282526076/Paradigms-of-Artifici...
(mainly because Peter Norvig is my goto as the paradigmatic 10x developer)
For example, in the Efficiency Issues chapter about how to optimize programs, Norvig lists writing a compiler from one language into a more efficient one. Most developers who have been working 20 years, either won't think of that, or understand how to implement it. But these are the kinds of things that can result in really outsize productivity gains.
https://manuel.kiessling.net/2011/04/07/why-developing-witho...
How fast would you drive a car if I gave you the keys and told you „everything works perfectly fine, however the brakes have been removed“?
I agree with the first part of your statement, but what really happens to such people?
In my experience (sample size greater than one), they receive some kudos, but remain underpaid, never promoted, and are given more work under tight deadlines. At least until some of them are laid off along with lower performers.
But for those who say that hard things are impossible, they seem to get along just fine. They merely declare such things as out-of-scope or lie about their roadmap.
100% agree, I've seen plenty of the best of the best get treated like trash and laid off at first sight of trouble on the horizon
(And absolutely not brute-force grinding themselves away)
At other times I’ll be assigned projects with regular POs, Architects and business employees who barely know what it is they are doing themselves, with poorly defined tasks and all sorts of bureaucratic nonsense “agile” process methods and well spend forever delivering nothing.
So sometimes I’m a 50x developer delivering business altering changes. At other times I’m a useless cog in a sea of pseudo workers. I don’t particularly care, I get paid, but if management actually knew what was going on, and how to change it… well…
I think the reluctance to invest effort in something that will give devs super-powers 6 months in the future is why we don't get all those 10x devs.
It is always very useful as an engineer to log your time, what are you working on "right now" and is it "new work" , "maintenance work", or "fixing work." Then for each log entry that isn't "new work" thinking about what you could have done that would have caught that problem before it was committed to the code base.
I find it is much better to evaluate engineers based on how often they are solving the same problem that they had before vs creating new stuff. That ratio, for me, is the essence of the Nx engineer (for 0.1 < N < 10)
The point that Wilson makes that having infrastructure/tools that push that ratio further from "repair" work to "new work" is hugely empowering to an organization.
It's easy to bog down even the best Nx engineers if you keep them occupied with endless bullshit tasks, meetings, (ever) changing timelines, and all that.
Kind of like having a professional driver drive a sportscar through a racetrack, versus the streets of Boston.
It's like how an NBA player would be a 10x player on a college basketball team. Great to work with them but If I was in their shoes I don't know how enjoyable/engaging the work would be.
Simulation is an interesting approach but I am curious if they ever implemented the simulation wrong would it report errors that don't happen on the target platform or fail to find errors that the target platform reports? How wide the gap is will matter... and how many possible platforms and configurations will the hypervisor cover?
More to the point of the story (though I do think the actual point was indeed a hiring or contracting pitch), this reminds me a lot of the internal tests the SQLite team has. I would love to hear from someone with access to those if they feel the same way.
given their PLTR connection, probably not
Fans of Asimov's _Foundation_ series will appreciate the analogue to how this system aims to predict every eventuality based on every possible combination of events, a la psychohistory.
P.S. amazing intro post. Can't wait to try the product.
For a software not interacting with the real world, there is only one possibility for frame N+1, if you know the state of a system.
https://en.wikipedia.org/wiki/Determinism
PRNG are illusions, just misunderstood by humans.
(if you know the state of a system + all the rules then you can know the next state)
and determinism actually applies way beyond just algorithms.
Of course, to argue against myself, it would surely be based on layers of probabilities, and they say several times in the series that it can't predict low-level specific things, just high-level things. And perhaps the whole underlying question posed by the series is whether the universe really is deterministic. But anyway I don't think it's all off-base.
The blocker in doing that has always been the foundations of our machines: non-deterministic CPUs and operating systems. Re-building an entire vertical computing stack is practically impossible, so they just _avoid_ it by building a high-fidelity deterministic simulator.
I do wonder how they are checking for equivalence between the simulator and existing OS's, as that sounds like a non-trivial task. But, even still, I'm really bought in to this idea.
I'm a co-founder of Antithesis.
But it's also probably fair to describe NixOS as something weird that somebody else wrote :)
Then they run those tests while injecting all sorts of failures like OS failures, network issues, race and timing conditions, random number generator issues, etc.
It’s likely the only practical way today of testing for those things reliably, but you still have to write all of the tests and define your app state.
// This limits the value of testing, because if you had the foresight to write a test for a particular case, then you probably had the foresight to make the code handle that case too.
I often felt this way when I saw developers feel a sense of doing good work and creating safe software because they wrote unit tests like expect add(2,2) = 4. There is basically a 1-1 correlation between cases you thought to test and that you coded for, which is really no better off in terms of unexplored scenarios.
I get that this has some incremental value in catching blatant miscoding and regressions down the road so it's helpful, it's just not getting at the main thing that will kill you.
I felt similarly about human QA back in my finance days that asked developers for a test plan. If the dev writes a test plan, it also only covers what the dev already thought about. So I asked my team to write the vaguest/highest level test plan possible (eg, "it should now be possible to trade a Singaporean bond" rather than "type the Singaporean bond ticker into the field, type the amount, type the yield, click buy or sell") - the vagueness made more room for the QA person to do something different (even things like tabbing vs clicking, or filling the fields out of sequence, or misreading the labels) than how the dev saw it, which is the whole point.
Jepsen has never tested FoundationDB, not sure why you claim this and link to Jepsen's site.
Aphyr / Jepsen never tested FDB because, as he tweeted "their testing appears to be waaaay more rigorous than mine." We actually put a screen cap of that tweet in the blog post linked here.
[1]: https://twitter.com/obfuscurity/status/405016890306985984
[2]: https://web.archive.org/web/20220805112242/https://twitter.c...
[3]: https://web.archive.org/web/20150325003526/http://blog.found...
They link to the website for a definition of the term they are using.
We got bitten _hard_ in production when we accidentally allowed some of the nodes to get above 90% of the storage used. The whole database collapsed into a state where it could only do a few transactions a second. Then the ops team, thinking they were clever, doubled the size of the cluster in order to give it the resources it needed to get the average utilization down to 45%; this was an unforced error as that pushed the size of the cluster outside the fdb comfort zone (120 nodes) which is itself a problem. The deed was done though and pulling nodes was not possible in this state, so slowly, slooooowly... things got fixed.
We ended up spending an entire weekend slowly, slowly getting things back into a good place. We did not lose data, but basically prod was down for the duration, and we found it necessary to _manually_ evict the full nodes one at a time over the period.
Now, this was a few years ago, and fdb has performed wickedly fast, with utter, total reliability before that and since, and to this day the ops team is butthurt about fdb.
From an engineering perspective, if you aren't using java fdb is pretty not great, since the very limited number of abstraction layers that exist are all java-centric. There are many, many issues with the maximum transaction time thing, the maximum key size and value size and total transaction size issue, the lack of pushdown predicates (e.g., filtered scans can't be done in-place which means that in AWS, they cost a lot in inter-az network charge terms and also are gated by the network performance of your instances), and so on.
What ALL of these have issues have in common is that they bite you late in the game. The storage issue bites you when you're hitting the DB hard in production and have a big data set, the lack of abstractions means that even something as finding leaked junked keys turns out to be impossible unless you were diligent to manually frame all your values so you could identify things as more than just bytes, the transaction time thing is very weird to deal with as you tend to have creeping crud aspects and the lack of libraries that instrument the transactions to give you early warning is an issue, likewise for certain kinds of key-value pairs, there's a creeping size problem - hey, this value is an index of other values; if you're not very careful up front, you _will_ eventually hit either the txn size limit or the key limit. The usual workarounds for those is to do separate transactions - a staging transaction, then essentially a swap operation and then a garbage collection transaction - but that has lots of issues overtime when coupled with application failure.
There are answers to ALL of these, manual ones. For the popular languages other than java - Go, python, maybe Ruby - there _should_ be answers for them, but there aren't. These are very sharp edges. Those java layers are _also_ _not_ _bug_ _free_. So yeah, one has a reliable storage storage layer (a topic that has come up over and over again in the last few years) but it's the layer on top of that where all the bugs are, but now with constraints and factors that are harder to reason about than the usual storage layer.
One might say, hey, SQL has all of these problems too, except no. You can bump into transaction limits, but the limits are vastly higher than fdb and the transaction time sluggishness will identify it long before you run into the "your transaction is rejected, spin retrying something that will _never_ recover" sort of issue that your average developer will eventually encounter in fdb.
That said, I love fdb as a software achievement. I just wish they had finished it. For my current project, I have designed it out. I might be able to avoid all of the sharp edges above at this point, but since we are not a java shop, I also can't rely on all the engineers to even know they exist.
(see also https://aphyr.com/posts/283-jepsen-redis)
I care much less about how easy it is to use or deploy, but "good" is a subjective term, so other people might see things differently.
[1] https://www.saltwire.com/cape-breton/business/code-testing-s...
I am an absolute beginner at TLA+ but I really like this possible design space.
I have an idea for a package manager that combines type system with this style of deterministic testing and state space exploration.
Imagine knowing that your invocation of
Will always work because file system and OS state are part of the deterministic model.or an next gen Helm with type system and state space exploration is tested:
will always work when it comes up because all configuration state space exploration has been tested thanks to types.Tell me more about yours?
Your package manager knows your operating system's current state and the state space of all the control flow graph through the program and configuration together can go to, it can verify that everything lines up and there will be no error when executed a bit like a compiler but without causing the Halting problem.
In TLA+ you can dump a state graph as a dot file, which I turn into a SVG and run with TLA+ graph visualiser.
Types verify possible control flow is valid at every point. We just need to add types to the operating system and file system and represent state space for deterministic verification.
You could hide packages that won't work.
The package manager would have to lookup precached state spaces or download them as part of the verification process.
To my untrained eye I get: Logs, a graph of when in time the bug happened over multiple runs and a statistical analysis which part of the application code could be invovled. The statistical analysis is nice but it is completely flat, without any hierarchical relationships making it quite hard to parse mentally.
I kind of expected more context to be provided about inputs, steps and systems that lead to the bug. Is it expected to then start adding all the logging/debugging that might be missing from the logs and re-run it to track it down? I hoped that given the deterministic systems and inputs there could be more initial hints provided.