Ask HN: What bits of fundamental knowledge are productivity multipliers?
For example,
* Regular expressions for simple text processing.
* Parser combinators for parsing.
* Parser generators (esp. packrat variety) for parsing.
* The concept of fuzzing and property testing for testing code.
* Calculus for solving all sorts of problems.
* MCMC for solving a huge class of probability problems.
* Search algorithms for solving a variety of problems (e.g. all NP-hard problems, sudoku, HTNs, scheduling, planning).
* Gradient descent for solving a variety of optimization problems.
* Vector Space embedding as a conceptual tool for a variety of complex AI problems.
* Effect composition (Haskell's IO or Scala's ZIO) as an incredibly powerful paradigm for concurrency and parallelism.
What are some examples of 10x multipliers that come to your mind? Fundamental ideas without which you would be drastically less productive.
423 comments
[ 4.4 ms ] story [ 317 ms ] threadLambdas, async and Linq. Somewhat related to the previous point, but combining these three helps with writing expressive and powerful declarative and functional code.
It's unlikely that someone is doing task A without any calculus but another person who uses calculus is 10x more productive on the same task.
There are various cases of „programming hoorrors” stories where you have a calculator implementation that, instead of doing the ... you know, math, will actually go through all X combinations.
But it still feels more like unknown unknowns causing you to put in a lot of extra effort; I wouldn't classify these as productivity multipliers.
I always wondered, if he thought this trough a little bit further, he'd independently reinvent the arabic number systems.
https://en.wikipedia.org/wiki/Natural_sort_order
I think by "physicists" I should have said, Heisenberg. I'm talking about this: https://en.wikipedia.org/wiki/Matrix_mechanics#Heisenberg's_... https://en.wikipedia.org/wiki/Heisenberg's_entryway_to_matri...
After two weeks of increasing horror, and four attempts to write the algorithm, I sent the client something I called the "Blueberry Muffin Problem" email, as a reference to that scene in Casino. But there is no logical way to do this, I said. Anyway. This led to a series of conference calls in which we finally were able to see a clear strategy.
End result: 40 lines of immaculately clean code, roughly $10,000 at $200/hr, problem solved.
I obviously wasn't paid for writing the code in this situation; it was for spending enough time with the problem to find all the edge cases, and figure out the right questions to ask to get to the solution they needed.
---
I'm putting this down here because, even though it seems obvious, it's not; if you spend 10 or 20 hours really thinking seriously about what happens if people write early reports or late reports - the initial conclusion was just count how many they wrote into how many they sent, and tell if they need to send more, but that's not the way life works. Or people work. And it's going to create impossible bottlenecks if the last report is the most important one. The insight I had and what's going to drive this thing is: How many reports were due since the last one you sent. Again, that seems simple but it's not. Technically it doesn't mean they just skip them; it means if one was due Monday and one Wednesday, and you send one on Tuesday, that covers Monday's report; but if you sent it on Wednesday it covers both, the idea being to discard the missed ones without explicitly saying so, and drive them to do what's important today, and more important if it's the final day.
It also covers the situation where they filed Monday's report early, but only if they've filed all the needed reports prior to that; otherwise they'll need another one for Monday.
That's why this is so hard. It's not the code. It's the number of possible scenarios.
So the rule is, we start counting again from when the last report was written, in terms of priority. And when you're looking at 50 reports a day per franchise, you'd better know what the priorities are.
For myself personally, I think this is writing software that will lead to absolute catastrophe. I've always tried to align the way my software works with how I think people will be able to use it, to push them to do the right thing. Anyone can write the software I write, it's the thinking about this process that is extremely difficult if you're dealing with, like, hundreds of [redacted] and thousands of customers spread out over time, and have to figure out how to make reasonable suggestions. This is such a thing.
What I'm saying is that this is marching into creating a thing that will cause total chaos and I don't think this is the right way. We have the chance now to re-imagine the frequency and rearrange the expectations of the customers and the expectations placed on the staff, and I think we need to do that. The algorithm you asked for is done, but it will not be good for people. And it is extremely hard to understand, even for me, if you asked me why one should take priority over another, when dozens are urgent on the same day. I wrote an "urgency" algorithm on top of it to try to deduce that. The best I can say about it is that it will cause less misses and damage than any other way of looking at the list, and it took me a long time to get to. But I think you're asking for something that is going to be so inefficient, and create such high customer expectations, it's going to be negative on both sides.
The scene keeps going through my head from "Casino", where De Niro tells the chef in the hotel to make sure the same number of blueberries are in every muffin, and the chef just drops his hands and goes, "do you have any idea how long that's going to take?" This is a blueberry muffin situation. We need to think of a better way.
This is how Jon Bentley starts his classic Programming Pearls
"What are you doing...and why?"[0] is almost always the best place to start when tackling requests from customers
------------
[0] my review/reference in relation to this exact problem, of asking the right question
* Recursion - most complex problems seem to have a recursive solution which you can actually implement in a language like Scheme or Haskell that does not limit stack size.
* State machines.
* Monte Carlo methods.
* Unification - see SICP.
* Hand-rolled parsers instead of regular expressions for debug-ability and readability.
Understanding how your organization works: This is not about sucking up to 'important' people (atleast not when it concerns productivity). This has got to do with understanding which how the different organizational parts work, so that you can be most effective. e.g. There's tons of planning for resources/budgets that happen once in a year and if you are aware and can put in your requirements into that list, it would help.
Making connections across teams: In large and complex organizations/products, a lot of knowledge is tribal. It helps to have friendly relations and people whom you could chat with informally in teams that you depend on. Many conflicts can be resolved faster.
A lot of times O(n^2) is gonna be perfectly fine. Write it the simple and easy to maintain way first, and only worry about fancy optimizations if it's too slow.
But it's so worth it in the end, +1 for learning while doing.
By mastering I mean, not knowing everything, but knowing more than you knew yesterday.
A pretty underrated hint about learning keystrokes : _print Cheat Sheets_ (they are available in pdf for most tools) and keep them somewhere on your desk. You will discover tons of cools tricks while waiting for things.
Was shown how to use it effectively, and improved my efficiency by more than double on the task it's good for
I was under the fallacious understanding you only needed an impact drill if you were trying to remove really stuck-on nuts/bolts (like rusted lug nuts)
As it turns out ... they're AMAZING for driving deck and roofing screws :)
I wonder if your logic was because of the larger pneumatic "tire gun" style things. Anyhow, sorry for going off topic!
No apologies needed :)
And yes - the only impact drills I'd ever seen were pneumatic, so I made the (wrong, but (hopefully) understandable) jump that the battery-operated one was for the same type of work :)
MIT - The Missing Semester of Your CS Education
[Most] Classes teach you all about advanced topics within CS, from operating systems to machine learning, but there’s one critical subject that’s rarely covered, and is instead left to students to figure out on their own: proficiency with their tools. We’ll teach you how to master the command-line, use a powerful text editor, use fancy features of version control systems, and much more!
https://missing.csail.mit.edu
Topics include:
Shell Tools and Scripting, Editors (Vim), Data Wrangling, Command-line Environment, Version Control (Git), Debugging and Profiling, Metaprogramming, Security and Cryptography
- Exponential backoff
- Jitter
- Consistent hashing
- Gossip algorithm
- Basic “Functional” aspects of a language (map, flatMap, reduce, filter, some, find)
- Putting things in a queue and using a “serverless” consumer
But some of the more day to day productivity boosters: know your development tools (including all keyboard shortcuts, including multi line editing, refactoring, grow shrink AST based selection, know all git / bash commands without googling…)
Know your programming language / framework deeply is also a supposedly simple one that just needs practice and dedication.
Basic skills like this should be at the top of the list, including fast typing, using all available keyboard shortcuts, avoiding using the mouse but nevertheless being fast and accurate with it when required.
The amount of time people spend typing and using the mouse dwarfs any other time investment and if you improve your speed with these basic skills, the productivity multiplier can be enormous.
It's easier.
Someone may be more technically competent at a problem-space (at the start), but if you are able to work diligently at the problem over a sustained period, you will have 10x better results than those who crash/burnout/lose interest/try to sprint to the finish.
So the productivity multiplier advice is this: learn to cope and be productive at doing the kinds of work where flow is impossible/unfeasible!
(This way you'll not be already tired and behind when you get to the work for which "flow" is possible...)
Flow happens during fun.
Sometimes there are shortcuts to grinding that can only be discovered during "flow" (usually for me, it's to identify better tools)
At 38 years old, what I have discovered are ways to make myself valuable to my employer even when I’m not at my peak. The move to a more “architectural” role has been a Godsend for me. It means that I’m a let to serve to some degree in an advisory capacity for others. When I’m not able to enter “flow” with my primary project, I can usually find someone else with a problem that they’re struggling on, pair up with them, and help them get over whatever is in their way. This makes the team as a whole significantly more productive.
As a bonus, I often find that focusing on a problem of short duration helps me overcome the mental barriers to becoming engaged in my work and lets me get done what I was struggling not to avoid doing in the first place.
Always hoping to find somewhere willing to accept the inconsistency in output but haven't found it yet. Anyways it's cool to hear someone in one of these threads acknowledge this reality for a lot of people. The general HN consensus seems to be that anyone who isn't a top 10% technical star and an incredible productivity machine doesn't deserve to touch code.
Those of us who are like this are a “good value” to them - both in terms of relative pay, and because we tend to be VERY loyal when we find a “home.”
This sounds 100% like me.
I currently work for a healthcare company in the mental health space. We’re definitely hiring. Shoot me an email, and we’ll see if we can’t get you into the interview process.
nominallyanonymous@protonmail.com
You can read, for example, a Christian Apologetics work and come away with tools for public speaking and rhetoric that would be completely missed if you only got the "summary" of the apologetic argument in the religious context
* Linear Algebra, a matrix can represent lots of different things and lots of great tools around LA.
* Tree searches, 60% of all traditional AI is some sort of tree manipulation ( I made that number up ).
Asking "what fundamental knowledge" is like asking what telescope makes the best astronomers.
The problem with knowledge is applicability. You can know a lot of fundamental things, but if you cannot recognize the patterns were they are useful is dead knowledge.
My personal experience through my life (30+ years in the field) and observing and interviewing developers in many different industries tells me that curiosity, logical thinking, and emotional intelligence win the day. And these are not pieces of knowledge, but personal characteristics and skills to develop. And they apply to many careers, including software engineering.
I think that more important than knowledge accumulated is your ability to conceptualize and recognize the concepts in a different context. Knowledge is just a side effect of this.
There is knowledge you can acquire, and then there are skills that can allow you to acquire the right knowledge at the right time. For example, having a baseline understanding of logic and mathematical reasoning could help you to understand when a problem area might already be formalised, and to find that area and understand the work therein.
Or like. If someone came in asking "I'd like to make my car go faster, should I paint it green, or blue?" it's not unhelpful to point out that that's a solution to a different problem.
Plus anyway the audience is greater than just the person who initially posed the question. Conversations around this and other similar answers are vibrant today, so clearly they're helpful to someone.
A closely related skill is the ability to cut through all the abstractions and interfaces to get to the root cause of what is broken or needs changed. A technical and organizational debt buster if you will.
The proof is that a good coder spends about 10x as long thinking about it before actually committing any code. In a sentence: Be able to visualize the problem in your head first. The actual code, if it's elegant and succinct, is incidental.
This seems antithetical to the post above yours. Or at least how I'm reading it. In bigger tech co's this looks a lot like massive design documents with a dozen reviewers, all giving enormous amounts of time and thought to the idea in the pursuit of "building it right". Design processes that can take weeks or months for what is otherwise an MVP.
Obviously you have to take security into large consideration when doing things like this - but this is what qa/sandbox environments are for.
I'm a bit of a rarity in the modern world (and here in the corporate space that is HN), in that I'm the sole programmer on a bunch of large software projects in production that I also maintain and sysadmin 24/7. The only meetings needed are to understand what the clients want and bitch at them that what they want makes no sense, until they start making sense, and then trust me to build it. If something comes up from a design perspective that I think could go one of several ways, I usually have that thought in the shower and mumble to myself for awhile before framing it in terms they would understand as a series of yes/no choices.
This is generally way more efficient than hiring a team, and I know, because I've tried to hire teams to do it. There's a limit to what one coder like me can do, but it's a lot higher than what 4 people bickering can do. I'd say it's around what 8 bickering people can do.
Design and defense of design, though, is not just about ego. Not if it's done properly. The best design/code people will come to you and say this is why this is the only way to do X and lay out the chain of logic that led them to that conclusion. That's not because they'll be personally hurt if you don't go that direction; it's that they're annoyed they'll have to do extra work to make an inferior product if you don't take their advice. Promote those people.
If you saw what I write you'd say I was a 20x coder, but that's just from a youngin's perspective. I've been writing code since 1988. The basic value proposition for my clients is that they know I'll die at some point, but it's way cheaper and more efficient to pay one guy they know will get it done, and take the risk they'll have to scramble for someone else in less than 10 years (at which point they'll likely have to rewrite anyway). Also, the shit I write mostly maintains itself. I'm proud to say that my software in 24/7 production has had no actual crashes in years other than the occasional forced server upgrade. If I died, it would take awhile to notice if they didn't keep asking me for new features.
I'll say. It does not sound like you'd be very fun to work with, as a teammate or a stakeholder.
You can have bigger changes as proof of concept first, but select, properly design and launch the most important one.
How much of that waste would you attribute to 1x programmers and how much would you attribute to 0x managers?
Failed projects are inevitable though, even if you do everything well. Good teams don’t look for people to blame, which results in no one sticking their neck out to advocate for stuff. Instead they look to learn from each experience and hone their processes.
And I mean, who is responsible for curating those 1x engineers in the first place?
After a couple of years and several such reporting options someone asked, "Can't we just generate a spreadsheet and then your business analysts can do whatever reports they want from that?" and the customer responded, "You can do that?! That would be so much easier for us."
Imagine the weeks or even months of human life (both implementing but also using the less efficient reports) that could have been saved if the right solution had been implemented from the start.
That's literally time taken out of someone's life that they could have used to be with family or whatever they wanted instead of transcribing ridiculous reports.
(Of course, those reports were probably only generated to be looked at once in some recurring management meeting, only for the executives to feel good about having seen data. I don't think the reports were ever used to support any decision.
This means the entire reporting theatre was completely bullshit and the customer could have paid the business analysts to sip drinks on the beach instead with no loss of productivity. But that's just my cynical view.)
Only if the product is run by one or two people who give each other that much time off. *Much* more likely, this work not commissioned by the client, leaving space for other work to be done.
This "automation will lead to more free time" is a utter utter myth, because those with the power to pay staff will just sell their staff's time+skills to someone else/some other problem.
(There's a tiny chance the void would be filled with useful work, and that's also a good outcome in my book.)
These are unlikely to be technical issues, and more people/process issues. SPAs are popular because they optimsie for developer productivity (so arguably the opposite). Others have talked about them, but examples like focusing on the wrong problems to solve (if you're a startup making a video game, do you really need to build your own chat app? [0])
[0] https://nira.com/slack-history/
What are you referring to?
As an aside, from a developers perspective, all of the work is clearly valuable, I mean they got paid 300k+. But from a true definition of value perspective, the change increment is just not valuable to the users as evidenced by the lack of use.
Midwestern US salary caps are more like $90k-$150k.
You rarely, if ever, need Leetcode. What you need is to understand deeply what you should be building so you don’t waste everyone’s time!
As compared to places like tesla/shopify/etc where the Steve Jobs quote rules the roost:
At one time he was scared of the problem of scheduling the fleet. Yup, if describe the problem in complicated terms, then the scheduling problem gets wildly complicated, e.g., in NP Complete, etc. -- could have a big team working months on just the first version. But he needed answers right away and, to satisfy concerns of some investors on the Board, needed the answers also to apply to the whole, planned company (eventually it grew to something much bigger than the initial plans).
I was in a meeting looking for a solution with several people, and it was all confused. Finally I just announced that I'd solve it.
I was teaching computer science at Georgetown and had 6 weeks to go to the end of the semester. So, I wanted to be done in six weeks. How'd I do that? Sure, just concentrated on what the business actually needed and threw out the rest. So, in six weeks had the software putting out nice schedules.
Our two representatives of BoD member General Dynamics and major investor evaluated the schedule and announced "It's a little tight in a few places but it's flyable.". The BoD was thrilled.
At one point Smith stated that my work "solved the most important problem" facing the company. The 10X, maybe 1000X, difference was essentially just in delivering what the business needed and dropping the rest.
Edit to add that the course was “Time Management Fundamentals (2016)” by Dave Crenshaw. I completed it on LinkedIn learning. For those of you who’d like to give it a try, my advice is: do all the exercises. You won’t get the full value out of the course unless you actively incorporate the whole routine into your workflow. There’s a learning curve, but it’s absolutely worth it. Also be warned that the presenter’s personality is a little off-putting (to me, anyway), but he absolutely knows what he’s talking about, so listen to him.
I might watch it ... later :p
The "time management tips" one is a kind agglomeration of his advice on managing your time, your relationships, your reports, etc., and is a good follow-up. He has a course on having productive meetings, as well, which is well worth the time. It may make you hate meetings at your work, though XD.
Honestly his courses provided a lot of relief and validation to me, as well as hope for a future working in companies. I can't recommend them enough.
The course doesn't offer anything novel - for example it has elements of Getting Things Done. But it's presented in a more accessible way that resonates with me. It also focuses on making good habits, something I already discovered in another book recommended on HN, Tiny Habits.
Thanks for the recommendation. I really appreciate it.
"There is nothing so useless as doing efficiently that which should not be done at all."
https://en.wikipedia.org/wiki/Peter_Drucker
Damned, I had to search for the author and... it's also from Peter Drucker !
"Landing the plane" as they sometimes say.
I feels a bit like watching a chef, that has all the theoretical knowledge about food composition and such, but takes 5 minutes to chop an onion.
Is this really common? I know I'm not in a top tier job, so maybe they don't give dummies like me the hard problems, but I've never had to use calculus to solve a real world problem.
https://study.com/academy/lesson/calculus-in-the-real-world....
Yes, a calculus formula might have been used somewhere in that process, and was used by the finance person and dev. That doesn't mean I, or any other person, needed to know it.
As a dev at a financial institution, I've never needed to implement calculus formulas. Most calculations and formulas I've dealt with are algebra and stats. These usually are just a matter of using standard libraries too. So most devs aren't implementing formulas themselves, but just utilizing standard libraries.
That's odd. It loads up just fine for me with no paywall.
Check this out for more: https://allusesof.com/math/51-amazing-uses-of-calculus-in-re...
The point is that calculus is used to solve all sorts of problems. Sure, you can say "...not in my experience" but that's one experience and not representative. You just happen to a) not work in a realm where Calculus is used and/or b) work within that realm only on problems without Calculus.
That is why I questioned whether knowing calculus is really a piece of fundamental knowledge that allows a 10x performance increase. For the majority of people it's not going to help them. Even if they deal with calculus in their field, I don't see it providing a 10x increase because unless one is mathematician, the calculus portion of their job will be inconsequential and likely be automated or computed for them.
OK no argument there. The OP statement was "For example ... Calculus for solving all sorts of problems" so it appeared you were taking issue with the idea that Calculus can be used to solve all sorts of problems. Your argument is clearer now: even if Calculus can be used to solve all sorts of problems that doesn't mean that it is "fundamental knowledge" for most people.
If I'm wrong, show me I'm wrong and post a substantive comment. Snarky comments like that are not inline with HN guidelines.
Basic logic is here: https://en.wikipedia.org/wiki/Mathematical_logic
Dynamical systems has stuff that is not basic logic such as this topological concept which is applicable in understanding how stuff works without doing an actual calculation: https://en.wikipedia.org/wiki/Limit_cycle
That's calculus. Not logic, not philosophy.
Same for limit cycles - they make no mention or this type of calculation. The concept can exist in other domains such as systems thinking.
So concepts do exist outside the context of pure math. If you look at everything from a hard math perspective, then that is what you will see. You have to look at the context of the comments to understand. It's not necessarily calculus because the commenter might not be looking at it using those strict theories. There's no support for your assertion that it's not philosophy or not logic. It could coincidentally be similar to a calculus concept, but that doesn't mean that one is using calculus if they are approaching it from a different context that shares a similar concept.
https://en.m.wikipedia.org/wiki/Philosophy_of_logic
This is calculus, not logic, not philosophy.
Again, any evidence this is not logic or philosophy? Simply saying it isn't, is not a valid argument. Just because you assert a concept exists calculus, does not mean a similar one doesn't exist in another domain/context. Any response to the majority of my comment, especially around context of use? Or are you just trolling by repeating the same line in multiple comments with no substantive support for your claims? Honest question.
The insights from these topics: https://en.wikipedia.org/wiki/List_of_dynamical_systems_and_...
To your point on system dynamics, queuing theory pairs really well with calculus on that as well. Being able to somewhat predict how a system is going to behave under load based on measurements and an understanding of how the queues work is super super valuable for decision making.
To understand basic financial concepts, like the present value of an annuity, or implied volatility, requires calculus.
In boring software, simple questions like, how often will a workload miss cache, and what optimizes the cost there, can be modeled... if you use calculus.
Also, in the past, I've casually been thrown into a couple of jobs where calculus was directly applicable to the work I was doing.
I realize it’s a big ask, but is there any way you’d be willing to put together an example of when and how you’ve used calculus for a specific purpose where it really shines?
In another instance, I got a summer job working at a company that made an optics-related product, and that involved a bunch of numerical integration, Fourier transforms, some optimization problems in stuff like calibrating equipment. So naturally, there was calculus.
Fun-related, recently, I noticed that a long-term call option I was holding seemed surprisingly expensive compared to just buying the stock on margin, given current and expected future interest rates, and comparing against the option's theta and delta values. Then I realized I was assuming the stock only goes up, ignoring short-term vs. long term taxes, etc. To get a general sense of how an alternative strategy of just buying the stock and adjusting position sizes at a few price points might work out, in a back-of-the-envelope calculation, I considered this problem: on average, how many times will a Wiener process W(t) completely cross the small interval [a,a+h] in the time interval 0<t<T (for some T)? What's its asymptotic behavior with respect to h, T, and a? (This problem of hedging options with stock, surely, is studied much more in depth, but I wanted to understand the general picture.)
Edit: I'll add, that's also taking for granted a lot of stuff with infinite series in comp sci that is also part of calculus. Like, if you want to pick a random number from 1 to 5, you know you can pick from 1-8 and retry if it's too large, and with calculus you know it will average a finite number of tries (8/5, specifically) because the infinite series passes the ratio test.
What I realized a couple of days ago is that even earlier in the tournament, you don't want to optimize expected value -- you always want to give some of that for higher variance. Why? Because in the variance/EV trade-off curve, the slope of the curve is zero at the maximum value, which means to add h units of variance, you give up o(h), realistically, O(h^2), expected value. So there is some value of h where the trade-off is worth it.
(Concrete example: Golf tournaments could be modeled as a coin-flipping contest, where you get +1 point for flipping heads, -1 point for flipping tails, with 72 coin tosses. Suppose you could pay a 0.01 point cost to double the point value, i.e. you get +1.99 or -2.01 as a result of the coin toss. Then you should, because doing so on the first coin tosses will increase your chance of winning.)
You'll be surprised at how many "great developers" never move anywhere in terms of productivity because they lack fundamental communication skills.
But they are not applicable everywhere, so you should not expect your 8 hour days to turn into 48 minutes because you started using regexes, dynamic programming, and search algorithms everywhere.
The unlikable and unhelpful answer, but probably the single biggest fundamental multiplier, is unfortunately the g factor. General intelligence.
The only 10x differences that genuinely exist are between people who are 2x faster than average and people who are 5x worse. Not because they learned a trick, but for a complex combination of reasons, including having been handed down two or three standard deviations in general intelligence.
The harsh reality is that there is no single thing a 1x person can do to become 10x.
If the 'single thing' is really a whole compendium of integrated skills, then you can't easily teach it to people the same way that you can teach them regular expressions or parser combinators.
This sort of nebulous skill that very productive people have is also especially hard to transfer. You can spend a week taking math lessons from Terrence Tao and seeing how he works, but you will still not be Terrence Tao at the end of the week, no matter how many interesting mathematical tools and methods you learn.
My point is that there is No Silver Bullet that will consistently turn your 10 hour days in 1 hour day, or if there is one I have never seen it, and evidently no company has found one. Otherwise all of their employees would follow the same method and become 10x programmers. This has not happened.
It helps to learn popular patterns and algorithms, but I don't think learning more patterns will achieve the kind of order of magnitude improvement you're looking for.
And brain plasticity, and the characteristics of high performing groups or institutions.
Changing the way your group works together is going to have a more profound difference on the outcomes than focusing on individual performance.
I intuitively don't like the idea of the 10x programmer, my core argument is actually that you should not try to become a 10x programmer, and that there's no order of magnitude productivity you can gain in a vacuum, ignoring group dynamics. So you shouldn't focus on individual performance too much! There's not much you can do about it, anyways
Managing groups of people is hard, and there's often big inefficiencies that are going to be obvious to some people (engineers love to complain about management!) but that have a true multiplier effect when fixed. Because for the time invested, you have an effect on the whole team. That's a true multiplier.
I don't know what specifically you think I should learn about the fallacy of reification, but please feel free to elaborate.
The original idea of a 10x developer was between best and worst, not best and average.
Any evidence for this? You may say there's no evidence that a 1x person can work up to 10x its performance, but stating it's impossible without evidence lacks merit.
The first I sort of already touched on below is that gaining a whole order of magnitude productivity, not just on a single task where a cool pattern applies but on average, is a really big effect size. If your colleagues start completing tasks that should take a month in 3 days, you would definitely notice.
The second order effect is that if we had discovered any method that reliably let an average person gain 10x productivity, it would have generated tremendous profit for companies and spread like wildfire. We haven't seen that, so we should be very suspicious when people claim a single bit of knowledge, method, or pattern is going to improve your performance by 10x.
(The other response, which is also annoying and unsatisfying, is that I'm making the boring claim that there is nothing interesting, so the burden of proof is actually not on me. If someone thinks there exists any intervention that can increase an arbitrary developer's performance by 10x, it's up to them to collect data and reject the null hypothesis)
I actually like those little patterns a lot, there are problems where you might absolutely take 10x longer because you didn't know the algorithm or the right machine learning concept.
The OP talks about becoming "significantly more productive when solving a large class of problems", and where my interpretation differs is what you mean by large class of problem.
What's large for you is really going to be relative and will vary based on your personal experience, so I can't say you're wrong. I know I speak at least for myself I when I say that in my day to day, most of what I do is not leetcode-type problems, so despite knowing many algorithms I don't think I would be 10x slower if I didn't know them.
If your interpretation of OP is that these boosts will on average give you 10x more productivity, that's the only place where I really disagree.
The 10x productivity boost I get with regexes is being able to effectively automate out a lot of text manipulation on a large scale. A couple examples. Say I get a file with a bunch of junk in it that contains some IDs that I need to use as input to other commands. I write a simple regex in a text editor, find all, delete everything else, then use other regexes (or multiline editing) to wrap those IDs with the commands I actually want to run. A more impactful example. Several years ago I was writing a new loadtest suite, when I realized that it would look A LOT like some existing load tests we had. They weren't close enough for composition or inheritance to solve the issue completely (we were already both in a few places in the codebase to help out), so I used a similar process that I laid out above to write the new loadtest suite. I barely wrote any of the code manually, I was doing most of the work from a level of abstraction above, making broad code changes with regexes. I did it all in an afternoon.
To get the 10x multiplier from regexes you've got to basically bake them into everything you do.
I'm not trying to deny your experience, so I can accept your word for it that it would take you 10x longer to complete tasks if you weren't able to do this.
At this point the core of the argument becomes more what sort of mix of tasks your daily work consists of, I think. I agree that there are patterns that solve some problems very efficiently, but in my daily life if I try to look at people who complete projects 10x faster than average, I can't point at any single method or bit of secret knowledge.
So I think the twofold response is first that the average developer is actually able to use search-and-replace and already has some vague notion that regexes are a tool they could reach for, so this is really saying that "0.1x" productivity exists in a way. And I think that's uncontroversial: if you imagine an hypothetical programmer who is still using the `ed` text-editor and hand-writing machine code to address business problems, in a world where people throw together microservices running in K8s in 100 lines of code, they're going to be a lot slower than average.
But I think the question is really about whether starting from an average (or if you want, median) dev there's any single thing that will give you a 10x productivity boost, that's not already something normal the average person knows about.
The second answer I have, and that I keep coming back to, is that if we had such a thing it would be more popular than coffee and morning meetings. It would automatically and very quickly have become something that companies rapidly adopt and that becomes the new normal (like search and replace!).
So finally the point is not that there aren't thing like regexes that slow you down a lot if you don't know them, it's more that there aren't any such things that are also well-kept secret. If the effect size is a whole order of magnitude, you won't have to scour forum threads for secret methods that turn you into a Mythical 10x Programmer, it will just quietly already be on its way to being the new 1x, and it will most likely be a gradual improvement over time.
W. Edwards Deming said over and over again that the system in which an employee works puts a ceiling on how much output they can produce.
But you should take it a step further than just trying to maximize your output, and instead understand how your work creates value for your boss, your company, for society. Productivity is not a measure of how much you can output, it's a measure of how much value you can create, or how much closer you can come to achieving your goals.
If you can come in, spend five minutes and change the label on one button and increase sales 200%, you've been much more effective than if you spend years optimizing the performance of a component that isn't impacting overall performance of the system.
If we assume you get paid $100 to make a button, I think OP's question is how to do that in 1 minute instead of 10 minutes. You know, so you have 9 extra minutes to watch youtube.
But generally, if you can generate a lot of value for the company, and demonstrate it, you will go far, and you will make much more money, especially if you can do it consistently and threaten to leave.
Doing a jira ticket in 20 minutes instead of 2 hours, then spending the extra time watching youtube is fine, but how about becoming a manager, making 3x as much money, and spending 100% of your time watching youtube? Or better yet, watching movies in first class on the plane to the "strategic conference" in Hawaii.
I can't relate because I've been freelance since 2001, but I do remember how much I hated the fact that my bosses bought new cars and took trips and took credit for my work while I was making $8.50/hr as a junior webmonkey. Guess I never saw the future in it. But if I knew I'd be flying first class now, maybe I would reconsider...
In all seriousness, though, aren't people mostly paid what they're worth if they know how to demand it?
Yes, but the caveat is that most people don't know how to calculate their worth relative to the organization(s) they work with/for, and of those that do, the majority probably don't know how to negotiate/ask.
No - effectively all employees are paid less than their value to the company. This is where profit comes from.
(2) End of list.
But seriously, "10x" aside, there are some techniques that have come up surprisingly often and been surprisingly effective during my career. A lot of them are around caching, cache coherency, memoization, etc. and I see many of those have already been mentioned. How to write a simple parser is another one, likewise.
The one I haven't seen mentioned (and don't expect to) is "poisoning" of data or requests. Sometimes it's not very convenient to delete a piece of data or turn a request around with an error at the very moment when the need for that is detected - often because of locking or reentrancy issues. Just marking something as invalid, until some time when removal is more convenient, can be a very powerful technique. It can also make monitoring such events easier.
I've brought this idea into a codebase at least a dozen times, and more often than not it has gotten a "wow, that makes things so much easier" reaction. I can't take credit, of course. Like many things around caching or state machines, I got it from the hardware folks where it's a standard technique. In general, software folks can learn a lot from looking at how hardware folks have solved similar problems in an even more challenging environment, and "bring back" those solutions to their own domain.
* Fully understanding how SQL prepares queries; optimization strategies
* Embracing garbage collection in JITted runtimes, mainly JS for now
* Being able to diagnose and replace a squeaky belt in your car
* Know what a cat is saying when it meows at you and indicates something in the vicinity of its food / water / the door / a dead mouse.
These have all saved me a ton of time.
That might be very useful for me, but my career has taken a different track, and I have never needed that. I almost never touch SQL, and never in a situation where performance matters. However I know that I can look that up if I need it (or more likely find someone else - I expect looking it up will take months)
they had a query that ran every 5 minutes, but on non-trivial data sets (network monitoring data for more than ~200 devices on a network), the query would take more than 5 minutes to run ... throwing hardware at the problem didn't help noticeably (double the CPU speed? run time would drop by ~5%)
converting their query from OR form to IN() cut runtime by 90%
All because of how the SQL engine thought it should run comparisons (MySQL, at the time, at least, used the IN() as functionally equivalent to a jump table, whereas the OR format was treated as a series of it...then...elseif...else statements)
a) Let's speed up the SQL processing by creating 40K connections and each connection running a stored procedure. Unless your particular SQL instance has enough memory and cpu to support 40K connections, you are performing a denial of service attack on your own server.
b) I hate stored procedures and scripting; I am going to write a big ass SQL query that is 200 lines long. Your giant SQL query is too complex for the SQL engine to create an optimize query plan. It will effectively reduce your database engine from using set based logic to one row at time. One time, a developer complained SQL sucks and blow big time blah blah blah. It took me four hours to figure out his ungodly query. I was able to performance tune the query by breaking it into many small steps using temp tables and putting it all into a stored procedure. Execution time improved from 10 minutes to 9 seconds.