7 ways of installing it, but no deb or rpm. Is this the wonderful future of FOSS where developers don't bother working with distribution maintainers anymore?
This has been the norm for awhile. No one wants to work with distro maintainers because their model is incompatible with how people build and distribute their software.
You either get a curl sh, a tarball, or a wrapper around either of those that pretends to be a .deb or .rpm.
I've distributed a lot of software and DEB/RPM has to be the worst. I'd suggest those distros improve on their developer ergonomics if they want to stay relevant. 100% of my customers use Docker images these days as it is much much easier to use.
It's the easiest way to distribute software in a way that is controlled by the author of the software and where the author can reasonably control all of the dependencies installed.
This way you are providing a one-stop shop that can easily be run. I have all kinds of tools that are docker containers because its simpler to not have to worry about all kinds of library mismatches or locations of shared libraries, and instead ship a minimal docker container instead.
Someday someone will make a hyper-package-managers to be able to manage their packages installed via the thousands of package-managers out there today. Then, several other hyper-package-managers will be developed to cover the cases the first didn't cover. Then comes the hyper-hyper-package-managers...
Hasn't it always been rare for developers to maintain distro specific packages? That's why distro's have package maintainers, they also modify the layout and default configurations and whatnot to be consistent with the rest of the distro.
Yeah, Official debs/rpms are a thing but often completely independent of any distribution's packaging efforts (and they often have very different priorities).
Go binaries are statically-linked, and this tool doesn’t require any configuration files or other additional files.
Binaries are provided on the Releases page, so all you need to do is download one, put it in /usr/local/bin, and it’s essentially guaranteed to work.
To upgrade, you download the new release and overwrite it.
There’s an argument to be made for not having the version information available for dependency resolution (for scc as a dependency) available in your package manager, but you have to admit that it’s easy enough to install. GitHub doesn’t even mind if you `wget && chmod` from their CDN URL.
It's distributed dynamodb style keyvalue, SQL and Cypher graph database.
I feel if you want to get a project moving forward for something as large as a database, you can get something rudimentary working and extend the parser when you need those features.
SQL wise it supports Joins and where's and rudimentary full text search
It uses rockset's converged indexes for ease of query generation.
The database is partly multimodel with document storage and SQL and graph Cypher querying but I am yet to get all the models to be mutually queryable. The document storage is queryable by SQL but graphs aren't queryable by SQL or as a document.
[[edit: NO HE DIDN'T!]] If you missed it, the author does more than reuse the grammar, he used a parser generator, which I think is what
iLoveOncall is reacting to.
Personally I do agree that, when writing a language interpreter and claiming "no dependency", grabbing an existing parser is a bit of a cheat. Just like grabbing an existing bytecode VM, garbage collector, just-in-time engine, ... would be. It doesn't make the endeavor less interesting or less formidable, but it doesn't really fit "no dependencies" any more.
I don't mean to belabor the point but if you ctrl-f in the readme for "generator" you'll see:
> I only have to parse sqlite grammar, and even then probably only a fraction of it. So it looks like writing a parser generator might be plausible and by now it's really my only option if I want to get any tests passing at all.
> So here's my grammar so far. It's parsed by GrammarParser. In theory I could do this at comptime, but until we get a comptime allocator that's going to be a yak shave and I am way too many yak shaves deep already. So I laboriously but reliably write out all the rules into grammar.zig.
> The Tokenizer is written by hand and seems to be basically done - it runs without complaint on the entire test set. I might discover bugs there later, of course.
> The Parser is fun. There is a single parse function, but because it reads the rules from grammar.zig at compile time it gets specialized for each parse rule. Basically I got the same result as hand-generating the parser code, without having to splices a bunch of strings together. After the jam maybe I'll have some yak shave time to cut out grammar.zig entirely and do all the grammar stuff at comptime, and then it'll be a pretty sweet system.
> The best part about this is that I get really nice parse trees by generating rich types from the input grammar. Eg.
And also:
> I made some changes to the parser generator, added a whole bunch of debugging tools to the parser itself and then sat down and cranked on the grammar till I can parse 100% of the tests. I'm finally out of the damn tunnel.
The first dependency is a "dependency", but in the same way that an RFC you have to read with your eyes is a dependency. Except in this case, the dependency is a definition, in formal language (extended-backus-naur form, a.k.a. "e-bnf"), of the SQL language, as defined in ISO/IEC 9075:2016[0] (the 2016 version of the SQL language specification). They then use this language definition as the input to a program which generates parsers (called a parser-generator[1]), so that they can quickly get to where they have a library which they can use to parse/validate/inspect the SQL being given to them.
Copying the BFN "from an external source" is a smart move, since it means they don't have to do lots of busy work slowly reading and transcribing the specification; someone's already done that step, so why would anyone expect the author to waste time?
Using a parser generator is also a smart move since they exist already and are used all over the place (nobody hand-writes parsers for large languages; that's just a needless source of tedium and bugs). The code that's spit out of the parser generator is novel; that's newly created code which isn't taken from someone else's Github/other repo.
Ultimately, I don't see how any of what the author's done constitutes "[relying] on a dependency" given that they're not using anyone elses Zig source code in their compiled binary, they're writing lots of code for themselves to use, just very quickly, and with powerful tools.
The fraction of devs that regularly deal with databases and parsing. There are no 10x devs. There are devs with long experience in a specific category. The 10x idea is kind of stupid in terms of companies looking for them - it just means "we want people successfully trained somewhere else".
I agree there are probably no 10x devs. This person took 7 days to (almost) complete this task (which they cherry picked for their self), suggesting the average 1x dev with experience in this domain would take 70 days.
I think it would be more reasonable to call someone a 3 -sigma dev (someone 3 standard deviations above the mean. These would exist because that's how stats work)
I think the prevailing current definition is wrt. the average dev. The worst dev could be arbitrarily bad (suggesting the average dev could also be 10x).
It may be the least subjective but it’s also the least technically/physically probable even if someone were a 10x developer.
The gambit is that a mediocre programmer would take 10x as long to write the same code as the hypothetical 10x dev. But that’s not what a mediocre or even average dev does: they solve the problem in an entirely different (and, for purposes of this argument, less optimal) manner in some spam of time that may or may not be longer than a qualified hacker or a theoretical 10x dev. Almost all CS problems have multiple solutions, so you’re not likely to find one where there is only one solution and it takes one guy x and the other guy 10x as long.
Fair. I get your point. My only contention is of the magnitude - imo someone 10 times better than the average developer is a unicorn. But the term gets thrown around any time someone is 'just' really good; they wrote fairly optimal code, quickly. However, to your point, there may be an argument to be made that people who have made a huge impact on the CS landscape are 10x-ers (regardless of efficiency). Is Linus a 10x-er?
It's like questioning the existence of 10x NBA players or 10x chess players. The top super GMs are basically 10x better than most other GMs, who are themselves 10x better than most IMs. It seems strange that programming would be one of the fields that doesn't have a similar distribution of skill.
I think the actual pushback of the 10x programmer idea is that it's more often used to bully regular programmers into working longer hours, rather than actually identifying top performing programmers.
There's also the part where "developer" is more akin to "athlete" than "nba player". There's lots of different types of developer, just like there's lots of different types of athlete. A 10x NBA player will certainly not be a 10x Olympic Swimmer also, more likely .1X. Part of the problem is 10x developer gets talked about like they are going to be 10x athlete at NBA and Swimming and Golf and, and, and... that's what doesn't exist.
I used to question their existence but that was before I met a couple. I think it's just that they are quite rare, so a lot of people haven't met them.
If you also don't follow specific developers online then you might not notice any. Here are a few I've seen for which 10x is probably an underestimate!
* David Tonlay and Alex Crighton - feels like they've written half the Rust ecosystem between them.
* Eric Traut (Pyright author) - seriously go and check out Pyright's bug tracker. It's insane.
10x developers exist, but their prevalence is greatly overstated. Also, churning out code at 10x doesn't make one a 10x developer (this could even hamper everyone else).
I also propose that moniker be banned from being self-applied, and is in fact, a smell test: if you encounter a colleague calling themselves a 10x developer, start interviewing immediately. No good will come out of that.
My last boss was a 10x developer. Fantastically smart guy, socially awkward but nice, really looked out for his team but an absolute nightmare to work with technically.
The majority of the team couldn't understand his code. We'd have newly hired senior developers just leave rather than deal with it.
He'd rolled his own code generator for our data model that did everything from model generation to the web controllers.
The result was that while he could pump out work quickly, what would've otherwise been a quick fix for a graduate developer now required a deep understanding of a complex system.
This had the effect of turning what would have otherwise been a team of 1-2x developers into a team of 0.2-0.5x devs with a retention problem.
Imagine that instead of developing an app in a popular programming language, someone implements an idiosyncratic domain specific language suitable for the kind of app they need to build, and then builds the app using that. The result would work and maybe even let them be very productive churning out more features of the kind that were envisioned when the DSL was developed. If they need to extend or fix the DSL, as the original author, they can. Someone else will need to learn the DSL before they can do any work on the app.
There was a great deal of thought put into it and he could extend and modify the output really quickly.
The complexity of the system basically made it so that what would otherwise have been a simple task achievable by a graduate required a deep understanding to carry out.
> He'd rolled his own code generator for our data model that did everything from model generation to the web controllers.
Heh, I've actually done that... twice. Luckily it was a team of 1 and I wouldn't expect anyone else to understand my mess. The code generation was extra ugly since I planned to get rid of it eventually to craft out smaller details. It was great at doing repetitive work in bulk. Not sure if it was actually faster but at least it was less boring doing things that way.
As a junior, I worked with a senior who was considered "10x" and working with him was a pain: he decided the rules didn't apply to him, and management tolerated his repeated violations of conventions instituted to make our dynamic-language codebase manageable.
Anytime he "improved" a module, no one else could maintain it as that would entail additional rule-breaking, which was verboten for mortals, so only he could maintain code he touched. Combined with the fact that he didn't add any tests: the net result was he was slowly and surely subverting the codebase into his personal, brittle domain that no one else could change. He was slowing everyone else done, but all management was looking at was his velocity at closing bugs or rolling out new features while creating tech-debt. His boastful personality was just the icing on the cake.
@jamii seems super talented, but his bio says "in the past I've built database engines, query planners, compilers, developer tools and interfaces for [a...] myriad [of] consulting and personal research projects.", along with his repo's being related to SQL parers, or literal text-editors working purely on string manipulation. He is also sponsored to spent 100% of his time doing exactly this.
What I mean that he is almost definitely a 10x dev at writing SQL parsers. But ask him to write a shader that renders a neat waterbed material and he'd be likely a 0.8x dev? The overlap between experience and context is key.
I don't know anyone that would say that 10x developers don't exist at all. It's too easy to bring up someone like John Carmack or many foundational people in computer science that invented algorithms us layfolk could never imagine.
My experience with the phrase is people mean finding a "diamond in the rough" who can code circles around anyone else. It's not about finding a Norvig or Carmack, it's about finding a fresh graduate that you can stick on a problem and they will be bountifully productive.
It's basically a manager's wet dream: extremely productive but cheap. In my experience real 10x people appear to be the opposite: seemingly slow but incredibly expensive. Everyone I actually consider 10x makes millions. And of those that are friends, they didn't really reach that 10x stage until their 30s or 40s.
I agree that I think this can be an unhealthy attitude. A lot of us are working on projects where the biggest challenge is convincing the Product Manager to provide more than one sentence in the brief.
That said, I can't think of any technical domain where I could do this, even if provided with all the tests up front.
In the case you described you can become a 10x developer by developing some BA skills and having the guts to refuse to work until details are provided.
Of course it's impressive but this isn't the sort of work the average dev ends up doing day to day. We spend our time digging into dependencies (this had zero) both internal and external, interfacing with stakeholders and looking up business logic for a change. All those async tasks ultimately add up to significant headwinds even for the best "10x" dev.
To me, the more likely explanation is that this guy is an excellent developer, working on a problem he was suited to, and that's about all you can say. He came up with good strategies and insights here, and worked hard, and did a lot of other stuff right. But the idea of a generically 10x developer is a cartoonish oversimplification of how the world works.
> So parsing the bnf is kind of a mess, but I only have to parse this one bnf and not bnfs in general so I just mashed in a bunch of special cases.
Surely at that point it would've been a lot cleaner and more practical to just edit the one file you need to parse, to remove the weird line breaks, etc., rather than building special cases into your parser to work around those lines? What am I missing?
It's possible the files can't be edited within this project. I had a similar experience writing a code parsing engine. Sometimes it's best documented as code debt and the rewrite can be done at a later time.
I bet they wouldn't. They're well aware their complier often runs code inside if(false) blocks (in certain positions) and they just don't see this as important. Moving fast is more important. (Where exactly they're moving to is not quite clear.)
If parent is referring to what I think she/he is referring to, then it's something like this: https://github.com/ziglang/zig/issues/6768. Keep in mind this is only to do with comptime not runtime. So not ideal, but not catastrophic in the sense of "my program evaluated the wrong conditional branch at runtime!"
It's nice when you post actual links so we can discuss them.
* 10349: already fixed in master branch; issue remains open until we add behavior test coverage for it.
* 9810: same
* 8952: same
* 4491: same
* 3882: same
* 5230: same
* 6444: duplicate of 5230
* 7097: duplicate of 5230
I went through 100% of your links and 100% of them are solved already. The only reason they are open is because we are being responsible and not closing them until we have verified that each and every case has behavior test coverage.
This is why I don't post links to begin with, because it allows you to shift the narrative away from the way zig contributors treat these issues to "NO THESE ARE ALL FIXED!1!!".
If zig was being developed using a process where correctness mattered one jot, then the underlying issue wouldn't have kept happening and you wouldn't have kept papering over a fundamentally wrong model. While I did not compare zig to V, now that you brought it up in a related comment, you're right, they do seem to follow very similar development models: "That's not an issue, and even if it is an issue, we've already fixed it, and even if we haven't fixed it, it doesn't matter, and even if it matters, it's not our job, and even if it is, you can't prove it." Wow.
Zig is a toy language with a broken implementation. The comptime story is incoherent; it is two entirely unalike languages being written with superficially similar (but not identical) syntax, and lots of bugs in the gaps. At least V is honest about what it is and isn't.
ETA: I provided links, do you have any evidence whatsoever that any of these issues have been fixed? Because I went through these links and 0% of them have commits attached. 0% of them have comments from a developer indicating they've been fixed (other than comments posted after I shared links). 0% of them have comments from a developer saying that they're being held open in order for testcases to be added. Your claims are just more confabulation from a confirmed liar.
Do you realize that you’re conversing with the creator of zig, who has poured the last six years of his life into the project?
All software with any amount of complexity has bugs, even if there’s a published spec ratified by comittee over the course of 20 years, or whatever your unattainable standard of quality is.
Zig is free software and a community in the best sense of both.
Bugs are fine. Arrogant devs who claim that all bugs are fixed and their software is correct and anyone who says otherwise is lying are not fine.
And I do obviously know who I was interacting with; Kelley's arrogance and lies are legendary (second only to the V devs, but he appears to be consciously competing with them). It drove me away from the language when my own filed bug on this issue was met with a similar response.
You're free to your interpretation but your comments come of as being frustrated with something you've not paid for nor have any obligation to be involved with — yet you persist.
If you have a language or project as complex as Zig and can provide Andrew with some sort of guidance on better implementation / OSS issue management, I would love to see this. Otherwise, I would encourage you to not post comments criticizing work which you're not qualified to criticize.
What did we do to deserve such spite? I don't appreciate being associated with the "move fast and break things" motto.
If you look at the issue in question you'll see that it's not a bug, but a proposal to change the semantics of the language; rules about when something is comptime-known or not. If you replace `@compileError` (comptime effect) with `std.debug.print` (runtime effect) the print statement is never reached.
I would have liked to have learned more about how the query planner and evaluator work! There was almost nothing about that? Just the tests magically moving from 0% to 95%.
e.g. What table and value representation was used?
FWIW I suspect using a LALR(1) parser in Zig on the sqlite grammar would have saved some time and gotten past the parsing headache.
The sqllogictest comes directly from sqlite, so it seems like the parsing problem is mostly "port from C to Zig" (which are very similar metalanguages, or I guess meta- meta- languages in this case :) )
There are likely many ways of doing these things, but here is my vague recollection from the time when I worked on a C++ in-memory database. We used tree transformations for query planner/optimizer. The evaluator was implemented as a virtual interface that had a lot of complexity, but on most basic level it took some parameters and returned rows of data.
Optimizer/query planner worked by parsing the query as a tree of operations and then finding tree transformations which are equivalent. Most basically this involved swapping table join orders and trading between materializing and streaming operators(^0). Costing was based on histograms data histograms which essentially enabled estimating how many rows each node in the query tree is expected to return. Generally you want to reduce the number of rows you are materializing ASAP. This generally means joining tables with fewest rows first, because then you can efficiently scan larger tables. Default strategy for joining large tables was to scan smaller one and build a map of the values and then scan the other one checking the map for each record. An alternative strategy is to build a bloom filter instead of the map, but that is only more efficient if the cross-section of the join is small.
Virtual execute evaluate interface enabled a lot of custom execution strategies for selecting from row/column/external db based tables, sorting data, and tons of join strategies.
Note 0: The constraint with streaming operators is that you can't fully sort the data before next step is executed. This prohibits execution strategies such as efficient joins which require all data to be sorted.
Though I do think it's illumating to see the smallest, slowest code that implements the spec
Similar to the "500 lines or less" book, or say a TCP/IP implementation in Python
I cloned it, and it's ~3600 lines, which is very interesting. I didn't think you could implement anything that passes 95% of an sqlite test suite in that much code, as sqlite itself is at least 200K lines IIRC.
It has a ton of features built up over 20 years. Sure you save a lot by not having a disk pager, efficient data structures, and so forth, but still that's pretty small, and parsing/tokenizing is over 1000 lines of it.
That specific SQLite test suite (one of four [1]) has loads of generated SQL queries, and a long tail of more varied hand-written tests. That 95% of the test suite will be mostly generated queries that follow the same basic pattern of joins and projections with basic arithmetic and comparisons. See for example [2] and [3].
The generated tests are not designed to test a wide breadth of features of the SQL language, and passing them with a simple engine is very doable. A lot of the value of these tests is that the sheer volume of queries tends to find obscure problems in optimizers that would not easily surface otherwise. That is of course not a problem in a simple engine that does not have an optimizer.
Ah OK that makes sense, because I can't see any way you'd fit builtin functions like string matching/transformation in ~3600 minus ~1000 lines, and I think a bunch of those are part of the SQL standard.
The numbers could be misleading because the 5% of failing tests could be a large part or most of the core language (window functions?), while the 95% are some combinatorial tests that happen to take the exact same code path in an unoptimized engine
i.e. with 4.2 million tests for 2600 lines, it seems like you're going to have a lot of duplicate coverage, even accounting for state space explosion
Another thing I wonder is if Zig's C interop is good enough to simply reuse Lemon. Basically
- on the input side, write a Tokenizer in Zig and feed that into the generated state machine in C
- on the output side, consume Lemon's C data structures / function calls directly in Zig. This is probably the hard part as the grammar and semantic actions are very intertwined
That would be a cool demo, although honestly maybe not even less work
> I would have liked to have learned more about how the query planner and evaluator work! There was almost nothing about that? Just the tests magically moving from 0% to 95%.
At the veeeery bottom:
> Detailed writeup to follow probably in a week or two.
When I saw they were building their own parser I wondered why not use something like antlr? There are a lot of antlr grammars for SQL dialects. But then this is zig and I don’t know if there’s a zig antlr runtime yet.
I couldn't figure out what software is being developed to pass the sqllogittest suite here. Will this be a SQLite-like database or SQL parser implemented Zig?
96 comments
[ 3.6 ms ] story [ 69.2 ms ] threadYou either get a curl sh, a tarball, or a wrapper around either of those that pretends to be a .deb or .rpm.
One hopes.
https://hub.docker.com/r/stedolan/jq
Yes, it's despicable.
https://www.youtube.com/watch?v=umDr0mPuyQc
This way you are providing a one-stop shop that can easily be run. I have all kinds of tools that are docker containers because its simpler to not have to worry about all kinds of library mismatches or locations of shared libraries, and instead ship a minimal docker container instead.
Binaries are provided on the Releases page, so all you need to do is download one, put it in /usr/local/bin, and it’s essentially guaranteed to work.
To upgrade, you download the new release and overwrite it.
There’s an argument to be made for not having the version information available for dependency resolution (for scc as a dependency) available in your package manager, but you have to admit that it’s easy enough to install. GitHub doesn’t even mind if you `wget && chmod` from their CDN URL.
"Processing 40 TB of code from ~10M projects with a server and Go for $100 (2019)"
<https://news.ycombinator.com/item?id=33072846>
https://GitHub.com/samsquire/hash-db
It's distributed dynamodb style keyvalue, SQL and Cypher graph database.
I feel if you want to get a project moving forward for something as large as a database, you can get something rudimentary working and extend the parser when you need those features.
SQL wise it supports Joins and where's and rudimentary full text search It uses rockset's converged indexes for ease of query generation.
If you're interested in queries then you should read this blog post. https://rockset.com/blog/converged-indexing-the-secret-sauce...
The database is partly multimodel with document storage and SQL and graph Cypher querying but I am yet to get all the models to be mutually queryable. The document storage is queryable by SQL but graphs aren't queryable by SQL or as a document.
He didn't invent SQL. He had to take the grammar from somewhere.
You wanted him to handwrite it copied from the ANSI SQL paper?
Or to just think up all the possible grammars and ignore the real paper?
How would that be different or better than this?
Personally I do agree that, when writing a language interpreter and claiming "no dependency", grabbing an existing parser is a bit of a cheat. Just like grabbing an existing bytecode VM, garbage collector, just-in-time engine, ... would be. It doesn't make the endeavor less interesting or less formidable, but it doesn't really fit "no dependencies" any more.
Huh? He wrote the parser generator. https://github.com/jamii/hytradboi-jam-2022/tree/main/lib/sq...
> I only have to parse sqlite grammar, and even then probably only a fraction of it. So it looks like writing a parser generator might be plausible and by now it's really my only option if I want to get any tests passing at all.
> So here's my grammar so far. It's parsed by GrammarParser. In theory I could do this at comptime, but until we get a comptime allocator that's going to be a yak shave and I am way too many yak shaves deep already. So I laboriously but reliably write out all the rules into grammar.zig.
> The Tokenizer is written by hand and seems to be basically done - it runs without complaint on the entire test set. I might discover bugs there later, of course.
> The Parser is fun. There is a single parse function, but because it reads the rules from grammar.zig at compile time it gets specialized for each parse rule. Basically I got the same result as hand-generating the parser code, without having to splices a bunch of strings together. After the jam maybe I'll have some yak shave time to cut out grammar.zig entirely and do all the grammar stuff at comptime, and then it'll be a pretty sweet system.
> The best part about this is that I get really nice parse trees by generating rich types from the input grammar. Eg.
And also:
> I made some changes to the parser generator, added a whole bunch of debugging tools to the parser itself and then sat down and cranked on the grammar till I can parse 100% of the tests. I'm finally out of the damn tunnel.
Copying the BFN "from an external source" is a smart move, since it means they don't have to do lots of busy work slowly reading and transcribing the specification; someone's already done that step, so why would anyone expect the author to waste time?
Using a parser generator is also a smart move since they exist already and are used all over the place (nobody hand-writes parsers for large languages; that's just a needless source of tedium and bugs). The code that's spit out of the parser generator is novel; that's newly created code which isn't taken from someone else's Github/other repo.
Ultimately, I don't see how any of what the author's done constitutes "[relying] on a dependency" given that they're not using anyone elses Zig source code in their compiled binary, they're writing lots of code for themselves to use, just very quickly, and with powerful tools.
[0] - https://en.wikipedia.org/wiki/SQL:2016
[1] - https://en.wikipedia.org/wiki/Compiler-compiler
That is absolutely not true! In fact, most major programming language implementations use handwritten parsers [0].
That said:
> Using a parser generator is also a smart move since they exist already
Jamie wrote the parser generator here too. So it's all the more "from scratch".
[0] https://notes.eatonphil.com/parser-generators-vs-handwritten...
What fraction of devs could even complete this, let alone in merely 10x the time?
I think it would be more reasonable to call someone a 3 -sigma dev (someone 3 standard deviations above the mean. These would exist because that's how stats work)
Also, you're not guaranteed to have an example 3 standard deviations above the mean. It strongly depends on your distribution and sample size.
You're right about the sufficient sample size.
The gambit is that a mediocre programmer would take 10x as long to write the same code as the hypothetical 10x dev. But that’s not what a mediocre or even average dev does: they solve the problem in an entirely different (and, for purposes of this argument, less optimal) manner in some spam of time that may or may not be longer than a qualified hacker or a theoretical 10x dev. Almost all CS problems have multiple solutions, so you’re not likely to find one where there is only one solution and it takes one guy x and the other guy 10x as long.
I think the actual pushback of the 10x programmer idea is that it's more often used to bully regular programmers into working longer hours, rather than actually identifying top performing programmers.
Like 25%? But that’s really only because of the influx of people doing it for money.
If you’d asked me the same thing 15 years ago I’d have said 80%.
If you also don't follow specific developers online then you might not notice any. Here are a few I've seen for which 10x is probably an underestimate!
* David Tonlay and Alex Crighton - feels like they've written half the Rust ecosystem between them. * Eric Traut (Pyright author) - seriously go and check out Pyright's bug tracker. It's insane.
I also propose that moniker be banned from being self-applied, and is in fact, a smell test: if you encounter a colleague calling themselves a 10x developer, start interviewing immediately. No good will come out of that.
The majority of the team couldn't understand his code. We'd have newly hired senior developers just leave rather than deal with it.
He'd rolled his own code generator for our data model that did everything from model generation to the web controllers.
The result was that while he could pump out work quickly, what would've otherwise been a quick fix for a graduate developer now required a deep understanding of a complex system.
This had the effect of turning what would have otherwise been a team of 1-2x developers into a team of 0.2-0.5x devs with a retention problem.
There was a great deal of thought put into it and he could extend and modify the output really quickly.
The complexity of the system basically made it so that what would otherwise have been a simple task achievable by a graduate required a deep understanding to carry out.
Heh, I've actually done that... twice. Luckily it was a team of 1 and I wouldn't expect anyone else to understand my mess. The code generation was extra ugly since I planned to get rid of it eventually to craft out smaller details. It was great at doing repetitive work in bulk. Not sure if it was actually faster but at least it was less boring doing things that way.
Anytime he "improved" a module, no one else could maintain it as that would entail additional rule-breaking, which was verboten for mortals, so only he could maintain code he touched. Combined with the fact that he didn't add any tests: the net result was he was slowly and surely subverting the codebase into his personal, brittle domain that no one else could change. He was slowing everyone else done, but all management was looking at was his velocity at closing bugs or rolling out new features while creating tech-debt. His boastful personality was just the icing on the cake.
The difference, of course, the framework has extensive documentation.
So that was the missing ingredient.
He made a framework, but wrote no documentation. Writing documentation would have made everyone an n*x developer, for some value of n > 1.
@jamii seems super talented, but his bio says "in the past I've built database engines, query planners, compilers, developer tools and interfaces for [a...] myriad [of] consulting and personal research projects.", along with his repo's being related to SQL parers, or literal text-editors working purely on string manipulation. He is also sponsored to spent 100% of his time doing exactly this.
What I mean that he is almost definitely a 10x dev at writing SQL parsers. But ask him to write a shader that renders a neat waterbed material and he'd be likely a 0.8x dev? The overlap between experience and context is key.
My experience with the phrase is people mean finding a "diamond in the rough" who can code circles around anyone else. It's not about finding a Norvig or Carmack, it's about finding a fresh graduate that you can stick on a problem and they will be bountifully productive.
It's basically a manager's wet dream: extremely productive but cheap. In my experience real 10x people appear to be the opposite: seemingly slow but incredibly expensive. Everyone I actually consider 10x makes millions. And of those that are friends, they didn't really reach that 10x stage until their 30s or 40s.
That said, I can't think of any technical domain where I could do this, even if provided with all the tests up front.
It's something you get from other people. There's not a good test to figure out if you're 10x better than some randomly picked average developer.
To me, the more likely explanation is that this guy is an excellent developer, working on a problem he was suited to, and that's about all you can say. He came up with good strategies and insights here, and worked hard, and did a lot of other stuff right. But the idea of a generically 10x developer is a cartoonish oversimplification of how the world works.
Also, it's good if language doesn't stand in your way.
Surely at that point it would've been a lot cleaner and more practical to just edit the one file you need to parse, to remove the weird line breaks, etc., rather than building special cases into your parser to work around those lines? What am I missing?
Not always the best way, but usuaully the easiest thing that works for me.
I bet the zig project would be interested in the sha of the tree that blows up their compiler
Do you have an example? Sounds like a catastrophic edge case.
* 10349: already fixed in master branch; issue remains open until we add behavior test coverage for it.
* 9810: same
* 8952: same
* 4491: same
* 3882: same
* 5230: same
* 6444: duplicate of 5230
* 7097: duplicate of 5230
I went through 100% of your links and 100% of them are solved already. The only reason they are open is because we are being responsible and not closing them until we have verified that each and every case has behavior test coverage.
If zig was being developed using a process where correctness mattered one jot, then the underlying issue wouldn't have kept happening and you wouldn't have kept papering over a fundamentally wrong model. While I did not compare zig to V, now that you brought it up in a related comment, you're right, they do seem to follow very similar development models: "That's not an issue, and even if it is an issue, we've already fixed it, and even if we haven't fixed it, it doesn't matter, and even if it matters, it's not our job, and even if it is, you can't prove it." Wow.
Zig is a toy language with a broken implementation. The comptime story is incoherent; it is two entirely unalike languages being written with superficially similar (but not identical) syntax, and lots of bugs in the gaps. At least V is honest about what it is and isn't.
ETA: I provided links, do you have any evidence whatsoever that any of these issues have been fixed? Because I went through these links and 0% of them have commits attached. 0% of them have comments from a developer indicating they've been fixed (other than comments posted after I shared links). 0% of them have comments from a developer saying that they're being held open in order for testcases to be added. Your claims are just more confabulation from a confirmed liar.
All software with any amount of complexity has bugs, even if there’s a published spec ratified by comittee over the course of 20 years, or whatever your unattainable standard of quality is.
Zig is free software and a community in the best sense of both.
Show some respect.
And I do obviously know who I was interacting with; Kelley's arrogance and lies are legendary (second only to the V devs, but he appears to be consciously competing with them). It drove me away from the language when my own filed bug on this issue was met with a similar response.
> Zig has known bugs and even some miscompilations.
> Zig is immature. Even with Zig 0.9.0, working on a non-trivial project using Zig will likely require participating in the development process.
I don't know why you have it out for me. But as long as your HN account isn't blocked by mods for trolling I'm stuck endlessly refuting your claims.
If you have a language or project as complex as Zig and can provide Andrew with some sort of guidance on better implementation / OSS issue management, I would love to see this. Otherwise, I would encourage you to not post comments criticizing work which you're not qualified to criticize.
If you look at the issue in question you'll see that it's not a bug, but a proposal to change the semantics of the language; rules about when something is comptime-known or not. If you replace `@compileError` (comptime effect) with `std.debug.print` (runtime effect) the print statement is never reached.
Your claim is false.
e.g. What table and value representation was used?
FWIW I suspect using a LALR(1) parser in Zig on the sqlite grammar would have saved some time and gotten past the parsing headache.
The sqllogictest comes directly from sqlite, so it seems like the parsing problem is mostly "port from C to Zig" (which are very similar metalanguages, or I guess meta- meta- languages in this case :) )
Lemon is apparently a mini-yacc, just for sqlite's grammar, and is about 7K lines of C code, with no deps: https://sqlite.org/src/doc/trunk/doc/lemon.html
Optimizer/query planner worked by parsing the query as a tree of operations and then finding tree transformations which are equivalent. Most basically this involved swapping table join orders and trading between materializing and streaming operators(^0). Costing was based on histograms data histograms which essentially enabled estimating how many rows each node in the query tree is expected to return. Generally you want to reduce the number of rows you are materializing ASAP. This generally means joining tables with fewest rows first, because then you can efficiently scan larger tables. Default strategy for joining large tables was to scan smaller one and build a map of the values and then scan the other one checking the map for each record. An alternative strategy is to build a bloom filter instead of the map, but that is only more efficient if the cross-section of the join is small.
Virtual execute evaluate interface enabled a lot of custom execution strategies for selecting from row/column/external db based tables, sorting data, and tons of join strategies.
Note 0: The constraint with streaming operators is that you can't fully sort the data before next step is executed. This prohibits execution strategies such as efficient joins which require all data to be sorted.
If you want to know how sqlite does that, it has good docs about it, particularly https://sqlite.org/opcode.html
Though I do think it's illumating to see the smallest, slowest code that implements the spec
Similar to the "500 lines or less" book, or say a TCP/IP implementation in Python
I cloned it, and it's ~3600 lines, which is very interesting. I didn't think you could implement anything that passes 95% of an sqlite test suite in that much code, as sqlite itself is at least 200K lines IIRC.
It has a ton of features built up over 20 years. Sure you save a lot by not having a disk pager, efficient data structures, and so forth, but still that's pretty small, and parsing/tokenizing is over 1000 lines of it.
The generated tests are not designed to test a wide breadth of features of the SQL language, and passing them with a simple engine is very doable. A lot of the value of these tests is that the sheer volume of queries tends to find obscure problems in optimizers that would not easily surface otherwise. That is of course not a problem in a simple engine that does not have an optimizer.
[1] https://www.sqlite.org/testing.html#test_harnesses
[2] https://github.com/gregrahn/sqllogictest/blob/master/test/ra...
[3] https://raw.githubusercontent.com/gregrahn/sqllogictest/mast...
The numbers could be misleading because the 5% of failing tests could be a large part or most of the core language (window functions?), while the 95% are some combinatorial tests that happen to take the exact same code path in an unoptimized engine
i.e. with 4.2 million tests for 2600 lines, it seems like you're going to have a lot of duplicate coverage, even accounting for state space explosion
- on the input side, write a Tokenizer in Zig and feed that into the generated state machine in C
- on the output side, consume Lemon's C data structures / function calls directly in Zig. This is probably the hard part as the grammar and semantic actions are very intertwined
That would be a cool demo, although honestly maybe not even less work
At the veeeery bottom:
> Detailed writeup to follow probably in a week or two.