i think you got the wrong eponymous law, pournelle's iron law of bureaucracy (which i see happening all the time, btw.) has nothing to do with this issue.
If I had a nickel for every time I have seen a Makefile straight up copied from other projects and modified to "work" while leaving completely unrelated unnecessary build steps and targets in place.
You find the first part in your stack that is documented (e.g., make is documented, even if your makefile is not) and use that documentation to understand the undocumented part.
You then write down your findings for the next person.
If you don’t have enough time, write down whatever pieces you understood, and write down what parts “seem to work, but you don’t understand“ to help make progress towards better documentation.
If you put the documentation as comments into the file, this can make copy&pasting working examples into a reasonably solid process.
You know, a makefile is documentation. That's why you should probably never copy one (except for a single line here or there). There's space for commenting a few stuff, but your target names and variables should explain most of what is going there.
Anyway, the article and most people here seem to be talking about those autotools generated files. Or hand-built ones that look the same way. But either way, it's a bad solution caused by forcing a problem to be solved by a tool that wasn't aimed at solving it. We have some older languages without the concept of a "project" that need a lot of hand-holding for compiling, but despite make being intentionally created for that hand-holding, it's clearly not the best tool for that one task.
At my work I've noticed another contributing factor: tools/systems that devs need to interact with at some point, but otherwise provide little perceived value to learn day-to-day.
Example is build system and CI configuration. We absolutely need these but devs don't think they should be expected to deal with them day to day. CI is perceived as a system that should be "set and forget", like yeah we need it but really I have to learn all this just to build the app? Devs expect it to "just work" and if there are complexities then another team (AKA my role) deals with that. As a result, any time devs interact with the system, there's a high motivation to copy from the last working setup and move on with their day to the "real" work.
The best solution I see is meet the devs halfway. Provide them with tooling that is appropriate simple/complex for the task, provide documentation, minimise belief in "magic". Tools like Make kinda fail here because they are too complex and black-box-like.
The office coffee machine is not „set and forget”, but you wouldn’t expect the entire responsibility for it’s maintenance to be evenly distributed between all people that use it. Similarly, CI needs ownership and having it fall on the last developer that attempted to use it is not an efficient way of working.
Same goes for anything "enterprisey". Last time I set up a big project, I made a commitment that "we should be able to check out and build this whole thing, for as long as humanly possible".
The local part is my big problem too. I used azure Dev ops in work. I find clicking through the UI to be a miserable experience, Id love to have it running locally so I could view inputs and outputs on the file system. Also yaml is an awful choice, no one I know enjoys working with it. The white space issues just get worse and worse longer your files get.
I think GP got confused, it's not running the runners locally, it's running the CI steps locally (see the other sibling replies).
For example, running tests locally exactly the same way as in the runner - sometimes I have to open a debugger in the middle of a test to see what exactly went wrong. Our tests run in gitlab in a particular docker image, and I've been adding a "make test" that runs the same tests in the same image locally, with additional flags to have full interactivity so the debugger works if needed.
Spoken like someone who has not tried what you are describing. There are two moving parts to your response: a locally hosted runner awaits jobs from GitLab itself, which doesn't help running _locally_, and the other part is that -- back when it existed! -- trying $(gitlab-runner exec) was not a full fledged implementation of the GitLab CI concepts, making it the uncanny valley of "run something locally."
However, as of v16 there is no more exec https://gitlab.com/gitlab-org/gitlab/-/issues/385235 which I guess is good and bad. Good in that it not longer sets improper expectations that it could have plausibly done anything, and bad in that now it joins GitHub Actions[1] in not having any _local_ test strategy aside from "boot up gitlab/gitlab-ce && echo good luck"
1: yes, I'm acutely aware of the 3(?) implementations/forks of nektos/act that claim to do GHA but, again, readme is not software and I can tell you with the utmost certainty they do not do as advertised
Not to mention that the rest of the environment is missing: e.g. you probably can't push to e.g. the organization's Docker registry/Artifactory/whatever from you local dev machine (if it's even visible from your machine in the first place). And those are arguably the most interesting parts that you want to test the integration with.
Yeah, I think this is the real issue. Too many different tool types that need to interact, so you don't get a chance to get deep knowledge in any of them. If only every piece of software/CI/build/webapp/phone-app/OS was fully implemented in GNU make ;-) There's a tension between using the best tool for the job vs adding yet another tool/dependency.
If you think cmake is a good example of more complex than make, then you haven't seen automake/autoconf. The first thing I thought of. You can find tons of tons of configure scripts that check if you're running ancient versions of Unix, checks that a byte is 8 bits wide, and a ton of other pointless checks. They don't do anything with all that information, don't think for a moment that you can actually build the app on Irix, but the checks for it have been passed along for decades likes junk DNA.
Firstly, automake/autoconf is not `make`, but a different piece of software, and secondly, that you know all those details about it is because it is not black-box-like.
I never said it was. It's a script to generate a script to generate a makefile, more or less.
If it wasn't black box like, why do people keep blindly copying tests which check things which haven't been relevant for decades and in any case would require a ton of manual effort to actually use for something?
Strong agree. The best workflow I've seen uses CICD as a very thin wrapper around in-tree scripts or make files.
If a Dev can run some/all of the "cicd" stuff locally, they can see, control, and understand it. It helps tremendously to have a sense of ownership and calm, vs "cicd is something else, la la la".
(This doesn't always work. We had a team of two devs, who had thin-wrapper CICD, who pretended it was an alien process and refused to touch it. Weird.)
+1. The only CI tool that I've seen really organize around this principle is Buildkite, which I've used and enjoyed. I'm currently using Github Actions and it's fine but Buildkite is literally sooooo good for the reasons you've mentioned.
Make is one of the simplest build tools out there. Compared to something like Grunt, Webpack, etc. it’s a hammer compared to a mining drill.
The solution is to not use tools used by large corporations because they are used by large corporations. My unpopular opinion is that CI/CD is not needed in most places where it’s used. Figure out how to do your builds and deploys with the absolute fewest moving pieces even if it involves some extra steps. Then carefully consider the cost of streamlining any part of it. Buying into a large system just to do a simple thing is often times not worth it in the long run.
If you really do need CI/CD you will know because you will have a pain point. If that system is causing your developers pain, it isn’t the right fit.
I have observed the Makefile effect many times for LaTeX documents. Most researchers I worked with had a LaTeX file full of macros that they have been carrying from project to project for years. These were often inherited from more senior researchers, and were hammered into heavily-modified forks of article templates used in their field or thesis templates used at their institution.
This is a great example of an instance of this "Makefile effect" with a possible solution: use Markdown and Pandoc where possible. This won't work in every situation, but sometimes one can compose a basic Beamer presentation or LaTeX paper quickly using largely simple TeX and the same Markdown syntax you already know from GitHub and Reddit.
That won’t solve any problem that LaTeX macros solve. Boilerplate in LaTeX has 2 purposes.
The first is to factor frequently-used complex notations. To do this in Markdown you’d need to bolt on a macro preprocessor on top of Markdown.
The second one is to fine-tune typography and layout details (tables are a big offender). This is something that simply cannot be done in Markdown. A table is a table and if you don’t like the style (which is most of the time inadequate) then there is no solution.
On the other hand, there are cases where (beneficial/desired) verbosity prompts copy-paste and tweaking - not due to complexity but from some form of scale or size of the input.
In many cases this is a sign of something that should be dynamic data (put it in a db instead of conf) but that's not always the case and worth the tradeoff in the moment.
amazon's internal build tool experiences this same phenomena. engineers are hired based on their leetcode ability; which means the average engineer has gaps in their infrastructure and config tool knowledge/skillset. until the industrys hiring practices shift, this trend will continue.
As an undergrad, I did group projects with people who quite literally could not compile and run any actual project on their system outside of a pre-packaged classwork assignment, who essentially could not code at all outside of data structure and algorithm problem sets, who got Google internships the next semester.
But they were definitely brighter than I when it came to such problem sets. I suppose we need both sorts of engineer to make great things
I did that, then I needed to tweak things so I added options, then I needed to use the package somewhere that needed to be self-contained, so I started copy-pasting ;). I've done similar things with makefiles, tox configs, linter settings (all of which started from an initial version I wrote from scratch).
I suspect the real reason this effect exists is because there's copy-pasting is the best way to solve the problem, due to a varying mix of: there being no way of managing the dependencies, needing to avoid (unmanaged) dependencies (i.e. vendoring is the same, only we have a tool managing it), the file (or its contents) needing to exist there specifically (e.g. the various CI locations) and no real agreement on what template/templating tool to use (and a template is just as likely to include useless junk). Copy-pasting is viewed as a one-time cost, and the thing copy-pasted isn't expected to change all that much.
I guess that there's a very important difference between copying something that you understand (or at least the details of which, like syntax, you can easily remember - here comments become important),
and copying something that not only you do not understand, but you were not the one that made it in the first place, and you never understood it !
Old IBM mainframe scripting in JCL https://en.wikipedia.org/wiki/Job_Control_Language (so "OS JCL" now, I suppose) used to have a terrible reputation for this, but I've never actually touched the stuff myself.
I think Makefile is maybe the wrong analogy - the problem with most people and makefiles is they write so few of them, the general idea of what make does is at hand, but the muscle memory of how to do it from scratch is not.
But, point taken - I've seen so much code copy-pasta'd from the web, there will be like a bunch of dead stuff in it that's actually not used. A good practice here is to keep deleting stuff until you break it, then put whatever that was back... And delete as much as possible - certainly everything you're not using at the moment.
This is exactly the problem I face with many tools, Makefiles, KVM setups, docker configurations, CI/CD pipelines. My solution so far has been to create a separate repository with all my notes, shell script example programs etc, for these tool, libraries or frameworks. Every time I have to use these tools, I refer to my notes to refresh my memory, and if I learn something new in the process, I update the notes. I can even point an LLM at it now and ask it questions.
The repository is personal, and contains info on tools that are publicly available.
I keep organisation specific knowledge in a similar but separate repo, which I discard when my tenure with a client or employer ends.
I'm usually contractually obligated to destroy all client IP that I may posses at the end of an engagement. My contracts usually specify that I will retain engagement specific information for a period of six months beyond the end of the contract. If they come back within that time, then I'll have prior context. Otherwise it's gone. Occasionally, a client does come back after a year or two, but most of the knowledge would have been obsolete and outdated anyway.
As for LLMs. I have a couple of python scripts that concatenate files in the repo into a context that I pass to Google's Gemini API or Google AI studio, mostly the latter. It can get expensive in some situations. I don't usually load the whole repository. And I keep the chat context around so I can keep asking question around the same topic.
> the tool (or system) is too complicated (or annoying) to use from scratch.
Or boring: some systems require boilerplate with no added value. It's normal to copy & paste from previous works.
Makefiles are a good example. Every makefile author must write their own functionally identical "clean" target. Shouldn't there be an implicit default?
C is not immune, either. How many bits of interesting information do you spot in the following excerpt?
The printf alone is the real payload, the rest conveys no information. (Suggestion for compiler authors: since the programs that include stdio.h outnumber those that don't, wouldn't it be saner for a compiler to automatically do it for us, and accept a flag to not do it in those rare cases where we want to deviate?)
> since the programs that include stdio.h outnumber those that don't
I don't think that is true. There is a lot of embedded systems C out there, plus there are a lot of files in most projects, and include is per file not per project. The project might use stdio in a few files, and not use it in many others.
> The printf alone is the real payload, the rest conveys no information.
What are you talking about? Every line is important.
#include <stdio.h>
This means you need IO in your program. C is a general purpose language , it shouldn't include that unless asked for. You could claim it should include stuff by default, but that would go completely against what C stands for. Code shouldn't have to depend on knowing which flags you need to use to compile successfully (at least not in general like this).
int main(int argc, char** argv)
Every program requires a main function. Scripting languages pretend they don't, but they just wrap all top-level code in one. Having that be explicit, again, is important for a low level language like C. By the way, the C standard lets you declare it in a simplified manner:
int main(void)
Let's ignore the braces as you could just place them on the same line.
printf("Hello\n");
You could just use `puts` here, but apart from that, yeah that's the main payload, cool.
return 0;
The C standard actually makes this line optional. Funny but I guess it addresses your complaint that "common stuff" perhaps should not be spelled out all the time?
So, here is the actual minimalist Hello world:
#include <stdio.h>
int main(void) {
puts("Hello world\n");
}
Thank you, but this thread was not about writing good code, but rather how often one ends up acritically copying existing "legacy" parts without even attempting to understand it.
I probably used the wrong words: "conveys no information" was meant as "is less meaningful than the printf". Just like switching on the PC every morning is essential, but if you ask me what my job's about, I wouldn't mention it.
In the same vein, I'm convinced that the printf is the part that expresses the goal of the program. The rest, the #include, the main(), even with the optimizations that you suggested, is just boilerplate, the part that is usually copied and pasted, not because it's not useful and not because it's too difficult to get right, as the original article claims, but because it's boring.
> Makefiles are a good example. Every makefile author must write their own functionally identical "clean" target. Shouldn't there be an implicit default?
At some point you have to give the system something to go on, and the part where it starts deleting files seems like a good one where not to guess.
It's plenty implicit in other places. You can for example, without a Makefile even, just do `make foo` and it will do its best to figure out how to do that. If there's a foo.c you'll get a `foo` executable from that with the default settings.
Same with programming: You just copy some old code and modify it, if you have something lying around.
Same with frameworks (Angular, Spring Boot, ...). The tools even come with templates to generate new boilerplate for people who don't have existing ones somewhere.
I have an alternate theory: about 10% of developers can actually start something from scratch because they truly understand how things work (not that they always do it, but they could if needed). Another 40% can get the daily job done by copying and pasting code from local sources, Stack Overflow, GitHub, or an LLM—while kinda knowing what’s going on. That leaves 50% who don’t really know much beyond a few LeetCode puzzles and have no real grasp of what they’re copying and pasting.
Given that distribution, I’d guess that well over 50% of Makefiles are just random chunks of copied and pasted code that kinda work. If they’re lifted from something that already works, job done—next ticket.
I’m not blaming the tools themselves. Makefiles are well-known and not too verbose for smaller projects. They can be a bad choice for a 10,000-file monster—though I’ve seen some cleanly written Makefiles even for huge projects. Personally, it wouldn’t be my first choice. That said, I like Makefiles and have been using them on and off for at least 30 years.
> That leaves 50% who don’t really know much beyond a few LeetCode puzzles and have no real grasp of what they’re copying and pasting.
Small nuance: I think people often don’t know because they don’t have the time to figure it out. There are only so many battles you can fight during a day. For example if I’m a C++ programmer working on a ticket, how many layers of the stack should I know? For example, should I know how the CPU registers are called? And what should an AI researcher working always in Jupyter know? I completely encourage anyone to learn as much about the tools and stack as possible, but there is only so much time.
We can’t really call the field engineering if this is the standard. A fundamental understanding of what one’s code actually makes the machine do is necessary to write quality code regardless of how high up the abstraction stack it is
Sure if you are doing embedded programming in C. How does one do this in web development though where there are hundreds of dependencies that get updated monthly and still add functionality and keep their job?
Yes, and I should overrule half the business decisions of the company while I am at it. Oh, and I'll push back on "we need the next feature next week" and I'll calmly respond "we need to do excellent engineering practices in this company".
And everybody will clap and will listen to me, and I will get promoted.
...Get real, dude. Your comments come across a bit tone-deaf. I am glad you are in a privileged position but you seem to have fell for the filter bubble effect and are unaware to how most programmers out there have to work if they want to pay the bills.
I know a lot of people have terrible jobs at profoundly dysfunctional companies. I've had those too. That situation doesn't improve unless you, as they say, have the serenity to accept the things you cannot change, the courage to change the things you can, and the wisdom to know the difference.
Not everyone has a position where they have the autonomy to spend a lot of effort on paying down technical debt, but some people do, and almost every programmer has a little.
I think it's important to keep in view both your personal incentive system (which your boss may be lying to you about) and the interests of the company.
The serenity in question boils down to "I'll never make enough money to live peacefully and being able to take a two years sabbatical so let's just accept I'll be on the hamster wheel for life and I can never do anything about it".
No. I'll let my body wither and get spent before my spirit breaks. I refuse to just "accept" things. There's always something you can do.
BTW is that not what HN usually preaches? "Change your job to a better one" and all that generic motivational drivel [that's severely disconnected from reality]? Not throwing shade at you here in particular, just being a bit snarky for a minute. :)
RE: your final point, I lost the desire to keep view of both my personal and my company's incentive systems. Most "incentive systems" are basically "fall in line or GTFO".
Before you ask, I am working super hard to change my bubble and get a bit closer to yours. To say it's not easy would be so understated so as to compare the description of a lightning hit on you and you enduring the said lightning hit. But as said above, I am never giving up.
But... it's extremely difficult, man. Locality and your own marketing matter a lot, and when you have been focused on technical skills all your life and marketing is as foreign to you as are the musical notes of an alien civilization... it's difficult.
The current state of web development is unfortunately a perfect example of this quality crisis. The tangle of dependencies either directly causes or quickly multiplies the inefficiency and fragility we’ve all come to expect from the web. The solution is unrealistic because it involves design choices which are either not trendy enough or precluded by the platform
Yes and they’re far less efficient and require far more maintenance than an equivalent electric or even diesel engine, where equivalent power is even possible
We don’t have to assume, because we know. We can calculate and measure the efficiency of gasoline and diesel engines, and electric motors. We know that electric motors are highly efficient, and ICE engines are not.
Steam engines currently power most of the world's electrical grid. The main reason for this is that, completely contrary to what you said, they are more efficient and more reliable than diesel engines. (Electric motors of course are not a heat engine at all and so are not comparable.)
Steam engines used to be very inefficient, in part because the underlying thermodynamic principles were not understood, but also because learning to build safe ones (largely a question of metallurgy) took a long time. Does that mean that designing them before those principles were known was "not engineering"? That seems like obvious nonsense to me.
Steam engines are thoroughly obsolete in the developed world where there are natural gas pipeline networks.
People quit building coal burning power plants in North America at the same time they quit burning nuclear power plants for the same reason. The power density difference between gas turbines and steam turbines is enough that the capital cost difference is huge. It would be hard to afford steam turbines if the heat was free.
Granted people have been building pulverized coal burning power plants in places like China where they'd have to run efficient power plants on super-expensive LNG. They thought in the 1970s it might be cheaper to gasify coal and burn it in a gas turbine but it's one of those technologies that "just doesn't work".
Nuclear isn't going to be affordable unless they can perfect something like
There is some truth in what you say. Though steam engines still power most of the power grid (especially in the "developed world") their capital costs are indeed too high to be economically competitive.
However, there are also some errors.
In 02022 24% of total US electrical power generation capacity was combined-cycle gas turbines (CCGT), https://www.eia.gov/todayinenergy/detail.php?id=54539 which run the exhaust from a gas turbine through a boiler to run a steam turbine, thus increasing the efficiency by 50–60%. So in fact a lot of gas turbines are installed together with a comparable-capacity steam turbine, even today.
Syngas is not a technology that "just doesn't work". It's been in wide use for over two centuries, though its use declined precipitously in the 20th century with the advent of those natural-gas pipeline networks. The efficiency of the process has improved by an order of magnitude since the old gasworks you see the ruins of in many industrial cities. As you say, though, that isn't enough to make IGCC plants economically competitive.
The thing that makes steam engines economically uncompetitive today is renewable energy. Specifically, the precipitous drop in the price of solar power plants, especially PV modules, which are down to €0.10 per peak watt except in the US, about 15% of their cost ten years ago. This combines with rapidly dropping prices for batteries and for power electronics to undercut even the capex of thermal power generation rather badly, even (as you say) if the heat was free, whereas typically the fuel is actually about half the cost. I don't really understand what the prospects are for dramatically cheaper steam turbines, but given that the technology is over a century old, it seems likely that its cost will continue to improve only slowly.
as if 1.5 hours of storage was going to cut it. I've been looking for a detailed analysis of what the generation + storage + transmission costs of a reliable renewable grid is that's less than 20 years old covering a whole year and I haven't seen one yet.
To be honest, I don't think anyone has any idea yet (other than crude upper bounds) because it depends a lot on things like how much demand response can help. Demand response doesn't have to mean "rolling blackouts"; it could mean "running the freezer during the day when electricity is free". Will people heat their houses in the winter with sand batteries? Will desiccant air conditioning pan out? Can nickel–iron batteries compete economically with BYD's gigafactories? What about sodium-ion? Nobody has any idea.
I was pleased to calculate recently that the EV transition, if it looks something like replacing each ICE vehicle with the BYD equivalent of a Tesla Model Y, would add several hours of distributed grid-scale storage, if car owners choose to sell it back to the grid. But that's still a far cry from what you need for a calm, cloudy week. Maybe HVDC will be the key, because it's never cloudy across all of China.
Sensible-heat seasonal thermal stores for domestic climate control (in some sense the most critical application) have been demonstrated to be economically feasible at the neighborhood scale. PCM or TCES could be an order of magnitude lower mass, but would the cost be low enough?
I don't consider that an equal comparison. Obviously an engineer can never be omniscient and know things nobody else knows either. They can, and should, have an understanding of what they work with based on available state of the art, though.
If the steam engine was invented after those discoveries about steel, I would certainly hope it would be factored into the design (and perhaps used to make those early steam engines less prone to exploding).
A lot of material science was developed to make cannons not explode - that them went into making steam engines possible. The early steam engines introduced their own needed study of efficiency-
The problem is that software is much more forgiving than real life engineering project. You can't build a skyscraper with duct tape. With software, especially the simple webapps most devs work on, you don't NEED good engineering skills to get it running. It will suck of course, but it will not fall apart immediately. So of course most "engineers" will go the path of least resistance and never leave the higher abstractions to dive deep in concrete fundamentals.
This is the 40% that OP mentioned. But there's a proportion on people/engineers that are just clueless and are incapable of understanding code. I don't know the proportion so can't comment on the 50% number, but hey definitely exist.
If you never worked with them, you should count yourself lucky.
If you spend 80% of your time (and mental energy) applying the knowledge you already have and 20% learning new things, you will very quickly be able to win more battles per day than someone who spends 1% of their time learning new things.
Specifically for the examples at hand:
- at 20%, you will be able to write a Makefile from scratch within the first day of picking up the manual, rather than two or three weeks if you only invest 1%.
- if you don't know what the CPU registers are, the debugger won't be able to tell you why your C++ program dumped core, which will typically enable you to resolve the ticket in a few minutes (because most segfaults are stupid problems that are easy to fix when you see what the problem is, though the memorable ones are much hairier.) Without knowing how to use the disassembly in the debugger, you're often stuck debugging by printf or even binary search, incrementally tweaking the program until it stops crashing, incurring a dog-slow C++ build after every tweak. As often as not, a fix thus empirically derived will merely conceal the symptom of the bug, so you end up fixing it two or three times, taking several hours each time.
Sometimes the source-level debugger works well enough that you can just print out C++-level variable values, but often it doesn't, especially in release builds. And for performance regression tickets, reading disassembly is even more valuable.
(In C#, managed C++, or Python, the story is of course different. Until the Python interpreter is segfaulting.)
How long does it take to learn enough assembly to use the debugger effectively on C and C++ programs? Tens of hours, I think, not hundreds. At 20% you get there after a few dozen day-long debugging sessions, maybe a month or two. At 1% you may take years.
What's disturbing is how many programmers never get there. What's wrong with them? I don't understand it.
That’s an insightful comment, but there is a whole universe of programmers who never have to directly work in C/C++ and are productive in safe languages that can’t segfault usually. Admittedly we are a little jealous of those elite bitcrashers who unlock the unbridled power of the computer with C++… but yeah a lot of day jobs pay the bills with C#, JavaScript, or Python and are considered programmers by the rest of the industry
Both have strong limits for writing complex code. Typescript is one attempt of an answer because bad as javascript is for large programs the web forces it. I prefer a million lines of c++ to 100k lines of python - but if 5k lines of python sill do them c++ is way too much overhead. (rust likely plays better than c++ for large problems from scratch but most large probles have existing answers and throwing something else in would be hard)
Many devs are never even coming into challenging projects like that. For a large part of the dev community its just simple webapps for most of their professional life.
You make it sound easy, but I think it's hard to know where to invest your learning time. For example, I could put some energy into getting better at shell scripting but realistically I don't write enough of it that it'll stick so for me I don't think it'd be a good use of time.
Perhaps in learning more shell scripting I have a breakthrough and realise I can do lots of things I couldn't before and overnight can do 10% more, but again it's not obvious in advance that this will happen.
I agree. And there's no infallible algorithm. I think there are some good heuristics, though:
- invest more of your time in learning more about the things you are currently finding useful than in things that sound like they could potentially be useful
- invest more of your time in learning skills that have been useful for a long time (C, Make) than in skills of more recent vintage (MobX, Kubernetes), because of the Lindy Effect
- invest more of your time in skills that are broadly applicable (algorithms, software design, Python, JavaScript) rather than narrowly applicable (PARI/GP, Interactive Brokers)
- invest your time in learning to use free software (FreeCAD, Godot, Postgres) rather than proprietary software (SolidWorks, Unity, Oracle), because sooner or later you will lose access to the proprietary stuff.
- be willing to try things that may not turn out to be useful, and give them up if they don't
- spend some time every day thinking about what you've been doing. Pull up a level and put it in perspective
> invest your time in learning to use free software (FreeCAD, Godot, Postgres) rather than proprietary software (SolidWorks, Unity, Oracle), because sooner or later you will lose access to the proprietary stuff.
The think you have a solid point with Postgres v Oracle and I haven’t followed game dev in a while, but your FreeCAD recommendation is so far from industry standard that I don’t think it’s good advice.
If you need to touch CAD design in a professional setting, learn SolidWorks or OnShape. They’re what every MechE I’ve ever worked with knows and uses, and they integrate product lifecycle aspects that FreeCAD does not.
10 years ago people said the same about KiCAD and Blender :) No guarantee that FreeCAD will be the same - but you can be pretty confident it will never go away.
However, if the goal is to be a full time employee doing CAD, I would of course going with one of the most established tools. Potentially even get some certifications.
I agree. An additional perspective that I have found useful came from a presentation I saw by one of the pragmatic programmers.
They suggested thinking about investing in skills like financial investments. That is, investments run on a spectrum from low risk, low return to high risk, high return.
Low risk investments will almost always pay out, but the return is usually modest. Their example: C#
High risk investments often fail to return anything, but sometimes will yield large returns. Their example: Leaning a foreign language.
Some key ideas I took away:
- Diversify.
- Focus on low risk to stay gainfully employed.
- Put some effort into high risk, but keep expectations safe.
- Your mix may vary based on your appetite for risk.
One simple approach that the second, or at least third, time you deal with something, you invest time to learn it decently well. Then each time you come back to it, go a bit deeper.
This algorithm makes you learn the things you'll need quite well without having to understand and/or predict the future.
If it’s a tool you use every day, it’s worth understanding on a deeper level. I’ve used the shell probably every day in my professional career, and knowing how to script has saved me and my team countless hours of tedious effort with super simple one liners.
The other thing that’s worth learning is that if you can find tools that everybody uses regularly, but nobody understands, then try to understand those, you can bring enormous value to your team/org.
I feel like your working environment might be to blame: maybe your boss is too deadline-driven so that you have no time to learn; or maybe there is too much pressure to fix a certain number of tickets. I encourage you to find a better workplace that doesn't punish people who take the time to learn and improve themselves. This also keeps your skills up to date and is helpful in times of layoffs like right now.
Seriously? Yes, you should read the docs of every API you use and every tool you use.
I mean, it's sort of ok if you read somewhere how to use it and you use it in the same way, but I, for one, always check the docs and more often even the implementation to see what I can expect.
you don't have the time because you spend it bruteforcing solutions by trial and error instead of reading the manual and doing them right the first time
> I completely encourage anyone to learn as much about the tools and stack as possible, but there is only so much time.
That seems like a weird way to think about this. I mean, sure, there's no time today to learn make to complete your C++ ticket or whatever. But yesterday? Last month? Last job?
Basically, I think this matches the upthread contention perfectly. If you're a working C++ programmer who's failed to learn the Normal Stable of Related Tools (make, bash, python, yada yada) across a ~decade of education and experience, you probably never will. You're in that 50% of developers who can't start stuff from scratch. It's not a problem of time, but of curiosity.
> I mean, sure, there's no time today to learn make to complete your C++ ticket or whatever. But yesterday? Last month? Last job?
That seems like a weird way to think about this. Of course there was no time in the past to learn this stuff, if you still haven't learned it by the present moment. And even if there were, trying to figure out whether there perhaps was some free time in the past is largely pointless, as opposed to trying to schedule things in the future: you can't change the past anyhow, but the future is somewhat more malleable.
To be clear: I'm not suggesting a time machine, and I'm not listing any particular set of skills everyone must have. I'm saying that excusing the lack of core job skills by citing immediate time pressure is a smell. It tells me that that someone probably won't ever learn weird stuff. And in software development, people who don't learn weird stuff end up in that 50% bucket posited upthread.
> I'm saying that excusing the lack of core job skills by citing immediate time pressure is a smell. It tells me that that someone probably won't ever learn weird stuff. And in software development, people who don't learn weird stuff end up in that 50% bucket posited upthread.
Or the whole chain of work culture is bad and people do not have adequate down time or brain juice to pursue these. Additionally, how many do you want to learn? I have dealt with Makefile, then recently someone decided to introduce taskfile and then someone else wanted to use build.please and someone tried to rewrite a lot of CI pipelines using python because shell scripting is too arcane, while someone decided that CI were super slow and must be hosted on premises using their favorite system(was it now drone or whatever I forgot). Eventually, things become so many and chaotic, your brain learns to copy-paste what works and hope for the best as the tool you have spent time learning will be replaced in few months.
A lot of these technologies share a common base that can be pretty small. Once you learn about Make and concepts like target, recipe, dependencies,... It'd be easier to learn Ansible or Github actions even though they don't solve the same problem. It's the same when learning programming language and whatever tools of the week. But that requires to spend a bit of effort to goes under the hood and understand the common abstractions instead of memorizing patterns and words.
> Or the whole chain of work culture is bad and people do not have adequate down time or brain juice to pursue these.
And... again, I have to say that that kind of statement is absolutely of a piece with the analysis upthread. Someone who demands a "work culture" that provides "down time" or "brain juice" to learn to write a makefile... just isn't going to learn to write a makefile.
I mean, I didn't learn make during "downtime". I learned it by hacking on stuff for fun. And honed the skills later on after having written some really terrible build integration for my first/second/whatever job: a task I ended up doing because I had already learned make.
It all feeds back. Skills are the components you use to make a career, it doesn't work if you expect to get the skills like compensation.
Funny enough I'm an A̶I̶ML researcher and started in HPC (High Performance Computing).
> if I’m a C++ programmer ... should I know how the CPU registers are called?
Probably.
Especially with "low level"[0] languages knowing some basics about CPU operations goes a long way. You can definitely get away without knowing these things but this knowledge will reap rewards. This is true for a lot of system based information too. You should definitely know about things like SIMD, MIMD, etc because if you're writing anything in C/C++ these days it should be because you care a lot about performance. There's a lot of stuff that should be parallelized that isn't. Even stuff that could be trivially parallelized with OpenMP.
> what should an AI researcher working always in Jupyter know?
Depends on what they're researching. But I do wish a lot more knew some OS basics. I see lots of things in papers where they're like "we got 10x" performance on some speed measurement but didn't actually measure it correctly (e.g. you can't use time.time and be accurate because there's lots of asynchronous operations). There's lots of easy pitfalls here that are not at all obvious and will look like they are working correctly. There's things about GPUs that should be known. Things about math and statistics. Things about networking. But this is a broad field so there are of course lots of answers here. I'd at least say anyone working on AI should read at least some text on cognitive science and neuroscience because that's a super common pitfall too.
I think it is easy to not recognize that information is helpful until after you have that information. So it becomes easy to put off as not important. You are right that it is really difficult to balance everything though but I'm not convinced this is the problem with those in that category of programmers. There's quite a number of people who insist that they "don't need to" learn things or insist certain knowledge isn't useful based on their "success."
IMO the key point is that you should always be improving. Easier said than done, but it should be the goal. At worst, I think we should push back on anyone insisting that we shouldn't be (I do not think you're suggesting this).
[0] Quotes because depends who you talk to. C++ historically was considered a high level language but then what is Python, Lua, etc?
I like Makefiles, but just for me. Each time I create a new personal project, I add a Makefile at the root, even if the only target is the most basic of the corresponding language. This is because I can't remember all the variations of all the languages and frameworks build "sequences". But "$ make" is easy.
I disagree. Make is - at it's simplest form - exactly a "simple plain shell script" for your tasks, with some very nice bonus features like dependency resolution.
Not the parent, bit I usually start with a two line makefile and add new commands/variables/rules when necessary.
Make is - at its core - a tool for expressing and running short shell-scripts ("recipes", in Make parlance) with optional dependency relationships between each other.
Why would I want to spread out my build logic across a bunch of shell scripts that I have to stitch together, when Make is a nicely integrated solution to this exact problem?
`Just` is also popular in this space, but tbh I think Make is a better choice.
Make is included in most Linux distros, including the ones available for WSL. It's also included with Apple's developer tools. It's been used for decades by millions of people, and is very mature.
If you use it in a simple way, Make is almost identical to `Just`. And when/if you want Make's powerful features, they're there and ready for you. Make's documentation is also exceptional.
I've used a bunch of these kinds of tools over the years, and I've never found one that I like more than Make.
I’d be curious to hear your ratio. It really varies. In some small teams with talented people, there are hardly any “fake” developers. But in larger companies, they can make up a huge chunk.
Where I am now, it’s easily over 50%, and most of the real developers have already left.
PS: The fakes aren’t always juniors. Sometimes you have junior folks who are actually really good—they just haven’t had time yet to discover what they don’t know. It’s often absolutely clear that certain juniors will be very good just from a small contribution.
My personal experience:
- 5% geniuses. This are people who are passionate about what they do, they are always up to date. Typically humble, not loud people.
- 15% good, can do it properly. Not passionate, but at least have a strong sense of responsibility. Want to do “the right thing” or do it right. Sometimes average intelligence, but really committed.
- 80% I would not hire. People who talk a lot, and know very little. Probably do the work just because they need the money.
That applies for doctors, contractors, developers, taxi drivers, just about anything and everything. Those felt percentages had been consistent across 5 countries, 3 continents and 1/2 a century of life
PS: results are corrected for seniority. Even in the apprentice level I could tell who was in each category.
From my 40 years in the field, I see much the same trend. I wouldn’t call 5% of developers “genius”—maybe 1% are true geniuses. Those folks can be an order of magnitude better at certain tasks—doing things no one else can—but only within a limited sphere. They also bring their own baggage, like unique personalities. Still, I believe there’s always room for genius on a big team, even with all the complications.
Typically, upper management wants smooth, steady output. But the better your people are, the bumpier that output gets—and those “one-percenters” can produce some pretty extreme spikes. If you think of it like a graph, the area under the curve (the total productivity) can be way bigger for a spiky output than for a flat, low-level one. So even if those spikes look messy, they often deliver a ton of long-term value.
Actually it is trivial to write a very simple Makefile for a 10,000 file project, despite the fact that almost all Makefiles that I have ever seen in open-source projects are ridiculously complicated, far more complicated than a good Makefile would be.
In my opinion, it is a mistake almost always when you see in a Makefile an individual rule for making a single file.
Normally, there should be only generic building rules that should be used for building any file of a given type.
A Makefile should almost never contain lists of source files or of their dependencies. It should contain only a list with the directories where the source files are located.
Make should search the source directories, find the source files, classify them by type, create their dependency lists and invoke appropriate building rules. At least with GNU make, this is very simple and described in its user manual.
If you write a Makefile like this, it does not matter whether a project has 1 file or 10,000 files, the effort in creating or modifying the Makefile is equally negligible. Moreover, there is no need to update the Makefile whenever source files are created, renamed, moved or deleted.
If everything in your tree is similar, yes. I agree that's going to be a very small Makefile.
While this is true, for much larger projects, that have lived for a long time, you will have many parts, all with slight differences. For example, over time the language flavour of the day comes and goes. Structure changes in new code. Often different subtrees are there for different platforms or environments.
The Linux kernel is a good, maybe extreme, but clear example. There are hundreds of Makefiles.
Different platforms and environments are handled easily by Make "variables" (actually constants), which have platform-specific definitions, and which are sequestered into a platform-specific Makefile that contains only definitions.
Then the Makefiles that build a target file, e.g. executable or library, include the appropriate platform-specific Makefile, to get all the platform-specific definitions.
Most of my work is directed towards embedded computers with various architectures and operating systems, so multi-platform projects are the norm, not the exception.
A Makefile contains 3 kinds of lines: definitions, rules and targets (typical targets may be "all", "release", "debug", "clean" and so on).
I prefer to keep these in separate files. If you parametrize your rules and targets with enough Make variables to allow their customization for any environment and project, you must almost never touch the Makefiles with rules and targets. For each platform/environment, you write a file with appropriate definitions, like the names of the tools and their command-line options.
The simplest way to build a complex project is to build it in a directory with a subdirectory for each file that must be created. In the parent directory you put a Makefile that is the same for all projects, which just invokes all the Makefiles from the subdirectories that it finds below, passing any CLI options.
In the subdirectory for each generated file, you just put a minimal Makefile, with only a few lines, which includes the Makefiles with generic rules and targets and the Makefile with platform-specific definitions, adding the only information that is special for the generated file, i.e. what kind of file it is, e.g. executable, static library, dynamic library etc., a list of directories where to search for source files, the strings that should be passed to compilers for their include directory list and their preprocessor definition list, and optionally and infrequently you may override some Make definitions, e.g. for providing some special tool options, e.g. when you generate from a single source multiple object files.
Sure, but this will require you to know how to tell the compiler to generate your Makefile header dependencies and if you end up making a mistake, this will cause silent failures.
Being able to set up things and truly understanding how they work are quite different imo.
I agree with the idea that a lot of productive app developers would not be able to set up a new project ex novo but often it is not about particularly true understanding but rather knowing the correct set of magic rules and incantations to make many tools work well together
This only happens because people treat build code at a lower standard than app code. IMO you should treat all code with the same rigour. From build scripts to app code to test code.
Why write hacks in build tools when you wouldn’t do in your app code.
We build tool code with the same quality as the app code. That’s why most tooling we use are written in typescript: type safety, code reuse…
I would argue the main reason is that Make is just bad. There are easier to use alternatives such as scons or rake that don't have this effect applied to them.
I see this effect in Java Maven pom.xml files. It's hard to get a straightforward answer on why each build step is needed, what each attribute means, what parts are optional or mandatory, etc. There seems to be a culture of copying these XML files and tweaking a few things without truly understanding what the whole file means. I briefly looked at Ant and Gradle, and their ecosystems don't look any better. The build configuration files seem to have too much unexplainable magic in them.
Imo, the only solution is to avoid boilerplate generators and the parent poms projects like spring boot use for things like pom files: you can look at the boilerplate to get ideas for what might be necessary, but, if you’re starting a project, write the pom yourself. It’s a pain the first couple times, but it gets easier to know what you need.
Honestly for Java I really like Bazel. You should give it a shot. I have a project with a self contained jvm and jars from maven central. Its more explicit than the other options but way less magical IMO.
> I briefly looked at …Gradle… The build configuration files seem to have too much unexplainable magic in them.
This is largely due to the use of groovy. When the Kotlin DSL is used instead, it can usually be introspected by (eg) IntelliJ. Otherwise, it’s pretty opaque.
Unless you know this, there's zero way you will come up with this by typing `configure` and using just auto-completion. Might as well use Groovy and a String for the name of the thing you're configuring. Good tooling would be able to auto-complete from there whether it's Groovy or Kotlin (or Java etc).
That wasn’t my experience a few years ago with a large groovy-dsl project. Since groovy will take a look in several different namespaces to automatically resolve things in a script, editors I tried had no hope of telling me what anything was.
Also, groovy allows modification of private instance variables which leads to … un-fun situations. I converted tens of thousands of lines of groovy to Kotlin. A lot of those lines were automated. Too many were not automatable for myriad reasons.
As far as the magic in Kotlin, I can easily click through all keywords and jump to the implementation in IJ. Groovy (at the time and in the project I was in) was utterly hopeless in this regard.
Groovy closure delegates' type can be declared, giving as much information as with Kotlin. The reason you couldn't follow the code was that the people who wrote those things either didn't declare types, or IntelliJ wasn't using the type declarations (I believe Groovy support in Gradle files is less good than in general Groovy files, where the IDE does support this). You're correct that some plugins will resolve things dynamically and those cannot be resolved by the IDE. But that's not the fault of the language, if you're going to rewrite in Kotlin with types, you could just as well add types to your Groovy declarations for the same result.
We recycle known good stuff to avoid getting bogged down and introducing fresh flaws.
The admonition to know what we're doing and act deliberately applies to so much in life, but flies in the face of Milton Friedman's point in "I, Pencil" => https://youtu.be/67tHtpac5ws?si=yhheE1Y5ELfjWXs-
Is not this a very generic phenomenon? I would argue it applies broadly. For example budgeting, you usually start from last year's budget and tweak that, rather than start from scratch. Or when you write an application letter, or a ServiceNow ticket, or whatever. Now I regret that I have brought in ServiceNow in the discussion, it kills the good mood....
But as I understand it and I am not an accountant (IANAA?), for non-ZBB budgets last years budget is usually used as a starting point and increases are justified.
"Here's why I need more money to do the same things as last year, plus more money if you want me to do anything extra".
I'd be curious what our man Le Cost Cutter Elon Musk does for budgeting?
I have made conscious effort in the past to never copy/paste the initial fleshing-out of a Makefile or a PHP class, or HTML boilerplate, or whatever. Like, for years I stuck to that. Then I stopped making that effort because there is no upside. Or rather, there is no downside to copy+paste+modify. It's faster and you save your brain power for things that actually matter.
There's a subtle difference between a nice template and a fully-working implementation that you then modify though.
(e.g. in that they were designed with different goals in mind, so the former is likely to have stopped at the point where it was general enough, to save you time, but not too specific to create footguns).
Bonus points if your template explicitly has fail patterns that prevent your code from silently failing.
346 comments
[ 2.6 ms ] story [ 432 ms ] threadIt's a major pet peeve of mine.
Trial and error?
Well have fun with that :p
If you don’t have enough time, write down whatever pieces you understood, and write down what parts “seem to work, but you don’t understand“ to help make progress towards better documentation.
If you put the documentation as comments into the file, this can make copy&pasting working examples into a reasonably solid process.
You know, a makefile is documentation. That's why you should probably never copy one (except for a single line here or there). There's space for commenting a few stuff, but your target names and variables should explain most of what is going there.
Anyway, the article and most people here seem to be talking about those autotools generated files. Or hand-built ones that look the same way. But either way, it's a bad solution caused by forcing a problem to be solved by a tool that wasn't aimed at solving it. We have some older languages without the concept of a "project" that need a lot of hand-holding for compiling, but despite make being intentionally created for that hand-holding, it's clearly not the best tool for that one task.
Example is build system and CI configuration. We absolutely need these but devs don't think they should be expected to deal with them day to day. CI is perceived as a system that should be "set and forget", like yeah we need it but really I have to learn all this just to build the app? Devs expect it to "just work" and if there are complexities then another team (AKA my role) deals with that. As a result, any time devs interact with the system, there's a high motivation to copy from the last working setup and move on with their day to the "real" work.
The best solution I see is meet the devs halfway. Provide them with tooling that is appropriate simple/complex for the task, provide documentation, minimise belief in "magic". Tools like Make kinda fail here because they are too complex and black-box-like.
- They're often slow
- They're often proprietary
- They're often dealing with secrets which limits who can work on them
- You generally can't run them locally
So the feedback cycle for working on them is incredibly long. And working on them is therefore a massive pain.
I recognize that this is such a disincentive for me taking the initiative to fiddle with and learn about anything like this
GitLab CI gives you local runners. You can completely self-host CI.
For example, running tests locally exactly the same way as in the runner - sometimes I have to open a debugger in the middle of a test to see what exactly went wrong. Our tests run in gitlab in a particular docker image, and I've been adding a "make test" that runs the same tests in the same image locally, with additional flags to have full interactivity so the debugger works if needed.
However, as of v16 there is no more exec https://gitlab.com/gitlab-org/gitlab/-/issues/385235 which I guess is good and bad. Good in that it not longer sets improper expectations that it could have plausibly done anything, and bad in that now it joins GitHub Actions[1] in not having any _local_ test strategy aside from "boot up gitlab/gitlab-ce && echo good luck"
1: yes, I'm acutely aware of the 3(?) implementations/forks of nektos/act that claim to do GHA but, again, readme is not software and I can tell you with the utmost certainty they do not do as advertised
If it wasn't black box like, why do people keep blindly copying tests which check things which haven't been relevant for decades and in any case would require a ton of manual effort to actually use for something?
If a Dev can run some/all of the "cicd" stuff locally, they can see, control, and understand it. It helps tremendously to have a sense of ownership and calm, vs "cicd is something else, la la la".
(This doesn't always work. We had a team of two devs, who had thin-wrapper CICD, who pretended it was an alien process and refused to touch it. Weird.)
The solution is to not use tools used by large corporations because they are used by large corporations. My unpopular opinion is that CI/CD is not needed in most places where it’s used. Figure out how to do your builds and deploys with the absolute fewest moving pieces even if it involves some extra steps. Then carefully consider the cost of streamlining any part of it. Buying into a large system just to do a simple thing is often times not worth it in the long run.
If you really do need CI/CD you will know because you will have a pain point. If that system is causing your developers pain, it isn’t the right fit.
It's a very microsoft feeling pile of crap
That won’t solve any problem that LaTeX macros solve. Boilerplate in LaTeX has 2 purposes.
The first is to factor frequently-used complex notations. To do this in Markdown you’d need to bolt on a macro preprocessor on top of Markdown.
The second one is to fine-tune typography and layout details (tables are a big offender). This is something that simply cannot be done in Markdown. A table is a table and if you don’t like the style (which is most of the time inadequate) then there is no solution.
On the other hand, there are cases where (beneficial/desired) verbosity prompts copy-paste and tweaking - not due to complexity but from some form of scale or size of the input.
In many cases this is a sign of something that should be dynamic data (put it in a db instead of conf) but that's not always the case and worth the tradeoff in the moment.
But they were definitely brighter than I when it came to such problem sets. I suppose we need both sorts of engineer to make great things
I suspect the real reason this effect exists is because there's copy-pasting is the best way to solve the problem, due to a varying mix of: there being no way of managing the dependencies, needing to avoid (unmanaged) dependencies (i.e. vendoring is the same, only we have a tool managing it), the file (or its contents) needing to exist there specifically (e.g. the various CI locations) and no real agreement on what template/templating tool to use (and a template is just as likely to include useless junk). Copy-pasting is viewed as a one-time cost, and the thing copy-pasted isn't expected to change all that much.
and copying something that not only you do not understand, but you were not the one that made it in the first place, and you never understood it !
But, point taken - I've seen so much code copy-pasta'd from the web, there will be like a bunch of dead stuff in it that's actually not used. A good practice here is to keep deleting stuff until you break it, then put whatever that was back... And delete as much as possible - certainly everything you're not using at the moment.
The repository is personal, and contains info on tools that are publicly available.
I keep organisation specific knowledge in a similar but separate repo, which I discard when my tenure with a client or employer ends.
On a more practical note, what structure, formats and tools do you use that enable you to feed it to an LLM?
As for LLMs. I have a couple of python scripts that concatenate files in the repo into a context that I pass to Google's Gemini API or Google AI studio, mostly the latter. It can get expensive in some situations. I don't usually load the whole repository. And I keep the chat context around so I can keep asking question around the same topic.
Or boring: some systems require boilerplate with no added value. It's normal to copy & paste from previous works.
Makefiles are a good example. Every makefile author must write their own functionally identical "clean" target. Shouldn't there be an implicit default?
C is not immune, either. How many bits of interesting information do you spot in the following excerpt?
The printf alone is the real payload, the rest conveys no information. (Suggestion for compiler authors: since the programs that include stdio.h outnumber those that don't, wouldn't it be saner for a compiler to automatically do it for us, and accept a flag to not do it in those rare cases where we want to deviate?)no
I don't think that is true. There is a lot of embedded systems C out there, plus there are a lot of files in most projects, and include is per file not per project. The project might use stdio in a few files, and not use it in many others.
What are you talking about? Every line is important.
This means you need IO in your program. C is a general purpose language , it shouldn't include that unless asked for. You could claim it should include stuff by default, but that would go completely against what C stands for. Code shouldn't have to depend on knowing which flags you need to use to compile successfully (at least not in general like this). Every program requires a main function. Scripting languages pretend they don't, but they just wrap all top-level code in one. Having that be explicit, again, is important for a low level language like C. By the way, the C standard lets you declare it in a simplified manner: Let's ignore the braces as you could just place them on the same line. You could just use `puts` here, but apart from that, yeah that's the main payload, cool. The C standard actually makes this line optional. Funny but I guess it addresses your complaint that "common stuff" perhaps should not be spelled out all the time?So, here is the actual minimalist Hello world:
I probably used the wrong words: "conveys no information" was meant as "is less meaningful than the printf". Just like switching on the PC every morning is essential, but if you ask me what my job's about, I wouldn't mention it.
In the same vein, I'm convinced that the printf is the part that expresses the goal of the program. The rest, the #include, the main(), even with the optimizations that you suggested, is just boilerplate, the part that is usually copied and pasted, not because it's not useful and not because it's too difficult to get right, as the original article claims, but because it's boring.
At some point you have to give the system something to go on, and the part where it starts deleting files seems like a good one where not to guess.
It's plenty implicit in other places. You can for example, without a Makefile even, just do `make foo` and it will do its best to figure out how to do that. If there's a foo.c you'll get a `foo` executable from that with the default settings.
Same with frameworks (Angular, Spring Boot, ...). The tools even come with templates to generate new boilerplate for people who don't have existing ones somewhere.
Now AI does a great job of getting you 90-100% of the way there.
Given that distribution, I’d guess that well over 50% of Makefiles are just random chunks of copied and pasted code that kinda work. If they’re lifted from something that already works, job done—next ticket.
I’m not blaming the tools themselves. Makefiles are well-known and not too verbose for smaller projects. They can be a bad choice for a 10,000-file monster—though I’ve seen some cleanly written Makefiles even for huge projects. Personally, it wouldn’t be my first choice. That said, I like Makefiles and have been using them on and off for at least 30 years.
Who likely wouldn't have a job if it weren't for LLMs.
Small nuance: I think people often don’t know because they don’t have the time to figure it out. There are only so many battles you can fight during a day. For example if I’m a C++ programmer working on a ticket, how many layers of the stack should I know? For example, should I know how the CPU registers are called? And what should an AI researcher working always in Jupyter know? I completely encourage anyone to learn as much about the tools and stack as possible, but there is only so much time.
And everybody will clap and will listen to me, and I will get promoted.
...Get real, dude. Your comments come across a bit tone-deaf. I am glad you are in a privileged position but you seem to have fell for the filter bubble effect and are unaware to how most programmers out there have to work if they want to pay the bills.
For everything else, there's MasterCard.
RE: unfixably broken, well, not necessarily in concept but de facto you are sadly correct. Most people resist even the provably good changes.
...the very definition of brokenness :D Not much of a (good) choice there...
I'm pretty sure that the original meme was «In God we trust; for everything else, there’s American Express» (with or without cocaine).
Not everyone has a position where they have the autonomy to spend a lot of effort on paying down technical debt, but some people do, and almost every programmer has a little.
I think it's important to keep in view both your personal incentive system (which your boss may be lying to you about) and the interests of the company.
No. I'll let my body wither and get spent before my spirit breaks. I refuse to just "accept" things. There's always something you can do.
BTW is that not what HN usually preaches? "Change your job to a better one" and all that generic motivational drivel [that's severely disconnected from reality]? Not throwing shade at you here in particular, just being a bit snarky for a minute. :)
RE: your final point, I lost the desire to keep view of both my personal and my company's incentive systems. Most "incentive systems" are basically "fall in line or GTFO".
Before you ask, I am working super hard to change my bubble and get a bit closer to yours. To say it's not easy would be so understated so as to compare the description of a lightning hit on you and you enduring the said lightning hit. But as said above, I am never giving up.
But... it's extremely difficult, man. Locality and your own marketing matter a lot, and when you have been focused on technical skills all your life and marketing is as foreign to you as are the musical notes of an alien civilization... it's difficult.
Steam engines used to be very inefficient, in part because the underlying thermodynamic principles were not understood, but also because learning to build safe ones (largely a question of metallurgy) took a long time. Does that mean that designing them before those principles were known was "not engineering"? That seems like obvious nonsense to me.
People quit building coal burning power plants in North America at the same time they quit burning nuclear power plants for the same reason. The power density difference between gas turbines and steam turbines is enough that the capital cost difference is huge. It would be hard to afford steam turbines if the heat was free.
Granted people have been building pulverized coal burning power plants in places like China where they'd have to run efficient power plants on super-expensive LNG. They thought in the 1970s it might be cheaper to gasify coal and burn it in a gas turbine but it's one of those technologies that "just doesn't work".
Nuclear isn't going to be affordable unless they can perfect something like
https://www.powermag.com/what-are-supercritical-co2-power-cy...
If you count the cost of the steam turbine plus the steam generators plus the civil works to enclose those, nuclear just can't be competitive.
However, there are also some errors.
In 02022 24% of total US electrical power generation capacity was combined-cycle gas turbines (CCGT), https://www.eia.gov/todayinenergy/detail.php?id=54539 which run the exhaust from a gas turbine through a boiler to run a steam turbine, thus increasing the efficiency by 50–60%. So in fact a lot of gas turbines are installed together with a comparable-capacity steam turbine, even today.
Syngas is not a technology that "just doesn't work". It's been in wide use for over two centuries, though its use declined precipitously in the 20th century with the advent of those natural-gas pipeline networks. The efficiency of the process has improved by an order of magnitude since the old gasworks you see the ruins of in many industrial cities. As you say, though, that isn't enough to make IGCC plants economically competitive.
The thing that makes steam engines economically uncompetitive today is renewable energy. Specifically, the precipitous drop in the price of solar power plants, especially PV modules, which are down to €0.10 per peak watt except in the US, about 15% of their cost ten years ago. This combines with rapidly dropping prices for batteries and for power electronics to undercut even the capex of thermal power generation rather badly, even (as you say) if the heat was free, whereas typically the fuel is actually about half the cost. I don't really understand what the prospects are for dramatically cheaper steam turbines, but given that the technology is over a century old, it seems likely that its cost will continue to improve only slowly.
https://www.eia.gov/analysis/studies/powerplants/capitalcost...
as if 1.5 hours of storage was going to cut it. I've been looking for a detailed analysis of what the generation + storage + transmission costs of a reliable renewable grid is that's less than 20 years old covering a whole year and I haven't seen one yet.
To be honest, I don't think anyone has any idea yet (other than crude upper bounds) because it depends a lot on things like how much demand response can help. Demand response doesn't have to mean "rolling blackouts"; it could mean "running the freezer during the day when electricity is free". Will people heat their houses in the winter with sand batteries? Will desiccant air conditioning pan out? Can nickel–iron batteries compete economically with BYD's gigafactories? What about sodium-ion? Nobody has any idea.
I was pleased to calculate recently that the EV transition, if it looks something like replacing each ICE vehicle with the BYD equivalent of a Tesla Model Y, would add several hours of distributed grid-scale storage, if car owners choose to sell it back to the grid. But that's still a far cry from what you need for a calm, cloudy week. Maybe HVDC will be the key, because it's never cloudy across all of China.
Sensible-heat seasonal thermal stores for domestic climate control (in some sense the most critical application) have been demonstrated to be economically feasible at the neighborhood scale. PCM or TCES could be an order of magnitude lower mass, but would the cost be low enough?
If the steam engine was invented after those discoveries about steel, I would certainly hope it would be factored into the design (and perhaps used to make those early steam engines less prone to exploding).
If you never worked with them, you should count yourself lucky.
Specifically for the examples at hand:
- at 20%, you will be able to write a Makefile from scratch within the first day of picking up the manual, rather than two or three weeks if you only invest 1%.
- if you don't know what the CPU registers are, the debugger won't be able to tell you why your C++ program dumped core, which will typically enable you to resolve the ticket in a few minutes (because most segfaults are stupid problems that are easy to fix when you see what the problem is, though the memorable ones are much hairier.) Without knowing how to use the disassembly in the debugger, you're often stuck debugging by printf or even binary search, incrementally tweaking the program until it stops crashing, incurring a dog-slow C++ build after every tweak. As often as not, a fix thus empirically derived will merely conceal the symptom of the bug, so you end up fixing it two or three times, taking several hours each time.
Sometimes the source-level debugger works well enough that you can just print out C++-level variable values, but often it doesn't, especially in release builds. And for performance regression tickets, reading disassembly is even more valuable.
(In C#, managed C++, or Python, the story is of course different. Until the Python interpreter is segfaulting.)
How long does it take to learn enough assembly to use the debugger effectively on C and C++ programs? Tens of hours, I think, not hundreds. At 20% you get there after a few dozen day-long debugging sessions, maybe a month or two. At 1% you may take years.
What's disturbing is how many programmers never get there. What's wrong with them? I don't understand it.
Perhaps in learning more shell scripting I have a breakthrough and realise I can do lots of things I couldn't before and overnight can do 10% more, but again it's not obvious in advance that this will happen.
- invest more of your time in learning more about the things you are currently finding useful than in things that sound like they could potentially be useful
- invest more of your time in learning skills that have been useful for a long time (C, Make) than in skills of more recent vintage (MobX, Kubernetes), because of the Lindy Effect
- invest more of your time in skills that are broadly applicable (algorithms, software design, Python, JavaScript) rather than narrowly applicable (PARI/GP, Interactive Brokers)
- invest your time in learning to use free software (FreeCAD, Godot, Postgres) rather than proprietary software (SolidWorks, Unity, Oracle), because sooner or later you will lose access to the proprietary stuff.
- be willing to try things that may not turn out to be useful, and give them up if they don't
- spend some time every day thinking about what you've been doing. Pull up a level and put it in perspective
That's the most useful point. Sometimes just a few why-s can give you a clearer view on your activities and a direction for your efforts.
The think you have a solid point with Postgres v Oracle and I haven’t followed game dev in a while, but your FreeCAD recommendation is so far from industry standard that I don’t think it’s good advice.
If you need to touch CAD design in a professional setting, learn SolidWorks or OnShape. They’re what every MechE I’ve ever worked with knows and uses, and they integrate product lifecycle aspects that FreeCAD does not.
However, if the goal is to be a full time employee doing CAD, I would of course going with one of the most established tools. Potentially even get some certifications.
They suggested thinking about investing in skills like financial investments. That is, investments run on a spectrum from low risk, low return to high risk, high return.
Low risk investments will almost always pay out, but the return is usually modest. Their example: C#
High risk investments often fail to return anything, but sometimes will yield large returns. Their example: Leaning a foreign language.
Some key ideas I took away:
- Diversify.
- Focus on low risk to stay gainfully employed.
- Put some effort into high risk, but keep expectations safe.
- Your mix may vary based on your appetite for risk.
This algorithm makes you learn the things you'll need quite well without having to understand and/or predict the future.
The other thing that’s worth learning is that if you can find tools that everybody uses regularly, but nobody understands, then try to understand those, you can bring enormous value to your team/org.
I mean, it's sort of ok if you read somewhere how to use it and you use it in the same way, but I, for one, always check the docs and more often even the implementation to see what I can expect.
That seems like a weird way to think about this. I mean, sure, there's no time today to learn make to complete your C++ ticket or whatever. But yesterday? Last month? Last job?
Basically, I think this matches the upthread contention perfectly. If you're a working C++ programmer who's failed to learn the Normal Stable of Related Tools (make, bash, python, yada yada) across a ~decade of education and experience, you probably never will. You're in that 50% of developers who can't start stuff from scratch. It's not a problem of time, but of curiosity.
That seems like a weird way to think about this. Of course there was no time in the past to learn this stuff, if you still haven't learned it by the present moment. And even if there were, trying to figure out whether there perhaps was some free time in the past is largely pointless, as opposed to trying to schedule things in the future: you can't change the past anyhow, but the future is somewhat more malleable.
Or the whole chain of work culture is bad and people do not have adequate down time or brain juice to pursue these. Additionally, how many do you want to learn? I have dealt with Makefile, then recently someone decided to introduce taskfile and then someone else wanted to use build.please and someone tried to rewrite a lot of CI pipelines using python because shell scripting is too arcane, while someone decided that CI were super slow and must be hosted on premises using their favorite system(was it now drone or whatever I forgot). Eventually, things become so many and chaotic, your brain learns to copy-paste what works and hope for the best as the tool you have spent time learning will be replaced in few months.
And... again, I have to say that that kind of statement is absolutely of a piece with the analysis upthread. Someone who demands a "work culture" that provides "down time" or "brain juice" to learn to write a makefile... just isn't going to learn to write a makefile.
I mean, I didn't learn make during "downtime". I learned it by hacking on stuff for fun. And honed the skills later on after having written some really terrible build integration for my first/second/whatever job: a task I ended up doing because I had already learned make.
It all feeds back. Skills are the components you use to make a career, it doesn't work if you expect to get the skills like compensation.
In an ideal world build tools would just work and people wouldn't have to think about them at all.
Especially with "low level"[0] languages knowing some basics about CPU operations goes a long way. You can definitely get away without knowing these things but this knowledge will reap rewards. This is true for a lot of system based information too. You should definitely know about things like SIMD, MIMD, etc because if you're writing anything in C/C++ these days it should be because you care a lot about performance. There's a lot of stuff that should be parallelized that isn't. Even stuff that could be trivially parallelized with OpenMP.
Depends on what they're researching. But I do wish a lot more knew some OS basics. I see lots of things in papers where they're like "we got 10x" performance on some speed measurement but didn't actually measure it correctly (e.g. you can't use time.time and be accurate because there's lots of asynchronous operations). There's lots of easy pitfalls here that are not at all obvious and will look like they are working correctly. There's things about GPUs that should be known. Things about math and statistics. Things about networking. But this is a broad field so there are of course lots of answers here. I'd at least say anyone working on AI should read at least some text on cognitive science and neuroscience because that's a super common pitfall too.I think it is easy to not recognize that information is helpful until after you have that information. So it becomes easy to put off as not important. You are right that it is really difficult to balance everything though but I'm not convinced this is the problem with those in that category of programmers. There's quite a number of people who insist that they "don't need to" learn things or insist certain knowledge isn't useful based on their "success."
IMO the key point is that you should always be improving. Easier said than done, but it should be the goal. At worst, I think we should push back on anyone insisting that we shouldn't be (I do not think you're suggesting this).
[0] Quotes because depends who you talk to. C++ historically was considered a high level language but then what is Python, Lua, etc?
Not the parent, bit I usually start with a two line makefile and add new commands/variables/rules when necessary.
Make is - at its core - a tool for expressing and running short shell-scripts ("recipes", in Make parlance) with optional dependency relationships between each other.
Why would I want to spread out my build logic across a bunch of shell scripts that I have to stitch together, when Make is a nicely integrated solution to this exact problem?
This is primarily aimed at a “task runner” replacement rather than a “compilation with automatic file timestamp comparison replacement”
Others I stumbled across: Taskfile Mage XcFile
None of them have tempted me enough to move away from a set of bash scripts or scripts written in the language of my repo (yet).
Make is included in most Linux distros, including the ones available for WSL. It's also included with Apple's developer tools. It's been used for decades by millions of people, and is very mature.
If you use it in a simple way, Make is almost identical to `Just`. And when/if you want Make's powerful features, they're there and ready for you. Make's documentation is also exceptional.
I've used a bunch of these kinds of tools over the years, and I've never found one that I like more than Make.
Where I am now, it’s easily over 50%, and most of the real developers have already left.
PS: The fakes aren’t always juniors. Sometimes you have junior folks who are actually really good—they just haven’t had time yet to discover what they don’t know. It’s often absolutely clear that certain juniors will be very good just from a small contribution.
That applies for doctors, contractors, developers, taxi drivers, just about anything and everything. Those felt percentages had been consistent across 5 countries, 3 continents and 1/2 a century of life
PS: results are corrected for seniority. Even in the apprentice level I could tell who was in each category.
Typically, upper management wants smooth, steady output. But the better your people are, the bumpier that output gets—and those “one-percenters” can produce some pretty extreme spikes. If you think of it like a graph, the area under the curve (the total productivity) can be way bigger for a spiky output than for a flat, low-level one. So even if those spikes look messy, they often deliver a ton of long-term value.
In my opinion, it is a mistake almost always when you see in a Makefile an individual rule for making a single file.
Normally, there should be only generic building rules that should be used for building any file of a given type.
A Makefile should almost never contain lists of source files or of their dependencies. It should contain only a list with the directories where the source files are located.
Make should search the source directories, find the source files, classify them by type, create their dependency lists and invoke appropriate building rules. At least with GNU make, this is very simple and described in its user manual.
If you write a Makefile like this, it does not matter whether a project has 1 file or 10,000 files, the effort in creating or modifying the Makefile is equally negligible. Moreover, there is no need to update the Makefile whenever source files are created, renamed, moved or deleted.
While this is true, for much larger projects, that have lived for a long time, you will have many parts, all with slight differences. For example, over time the language flavour of the day comes and goes. Structure changes in new code. Often different subtrees are there for different platforms or environments.
The Linux kernel is a good, maybe extreme, but clear example. There are hundreds of Makefiles.
Then the Makefiles that build a target file, e.g. executable or library, include the appropriate platform-specific Makefile, to get all the platform-specific definitions.
Most of my work is directed towards embedded computers with various architectures and operating systems, so multi-platform projects are the norm, not the exception.
A Makefile contains 3 kinds of lines: definitions, rules and targets (typical targets may be "all", "release", "debug", "clean" and so on).
I prefer to keep these in separate files. If you parametrize your rules and targets with enough Make variables to allow their customization for any environment and project, you must almost never touch the Makefiles with rules and targets. For each platform/environment, you write a file with appropriate definitions, like the names of the tools and their command-line options.
The simplest way to build a complex project is to build it in a directory with a subdirectory for each file that must be created. In the parent directory you put a Makefile that is the same for all projects, which just invokes all the Makefiles from the subdirectories that it finds below, passing any CLI options.
In the subdirectory for each generated file, you just put a minimal Makefile, with only a few lines, which includes the Makefiles with generic rules and targets and the Makefile with platform-specific definitions, adding the only information that is special for the generated file, i.e. what kind of file it is, e.g. executable, static library, dynamic library etc., a list of directories where to search for source files, the strings that should be passed to compilers for their include directory list and their preprocessor definition list, and optionally and infrequently you may override some Make definitions, e.g. for providing some special tool options, e.g. when you generate from a single source multiple object files.
I agree with the idea that a lot of productive app developers would not be able to set up a new project ex novo but often it is not about particularly true understanding but rather knowing the correct set of magic rules and incantations to make many tools work well together
Whether they are a bad choice really depends on what are the alternatives though
Why write hacks in build tools when you wouldn’t do in your app code.
We build tool code with the same quality as the app code. That’s why most tooling we use are written in typescript: type safety, code reuse…
This is largely due to the use of groovy. When the Kotlin DSL is used instead, it can usually be introspected by (eg) IntelliJ. Otherwise, it’s pretty opaque.
Also, groovy allows modification of private instance variables which leads to … un-fun situations. I converted tens of thousands of lines of groovy to Kotlin. A lot of those lines were automated. Too many were not automatable for myriad reasons.
As far as the magic in Kotlin, I can easily click through all keywords and jump to the implementation in IJ. Groovy (at the time and in the project I was in) was utterly hopeless in this regard.
We recycle known good stuff to avoid getting bogged down and introducing fresh flaws.
The admonition to know what we're doing and act deliberately applies to so much in life, but flies in the face of Milton Friedman's point in "I, Pencil" => https://youtu.be/67tHtpac5ws?si=yhheE1Y5ELfjWXs-
But as I understand it and I am not an accountant (IANAA?), for non-ZBB budgets last years budget is usually used as a starting point and increases are justified.
"Here's why I need more money to do the same things as last year, plus more money if you want me to do anything extra".
I'd be curious what our man Le Cost Cutter Elon Musk does for budgeting?
(e.g. in that they were designed with different goals in mind, so the former is likely to have stopped at the point where it was general enough, to save you time, but not too specific to create footguns).
Bonus points if your template explicitly has fail patterns that prevent your code from silently failing.