I love this. I made an ask HN post about testing practices for games[0]. The answers I got are interesting and an opportunity to reflect. There are clearly some differences in programming cultural practices.
I’m of the opinion that applying great testing practices to unity programming is very important. It’s been fun synthesizing ideas I’ve encountered into my hobby game dev career.
I’m glad there are others out there doing the good work of generating content to popularize the benefits of good testing practice :)
Absolutely. What I find astonishing is how undervalued test feedback time is.
I don't know what drives that, but in most jobs I've had to fight to do the right thing.
Just recently I've made some optimizations to get a functional test suite from 65 minutes down to 3.6. Parallelism is my favorite lever, as it scales so well, and core count is still going up!
This is why you hire old programmers, so that you have someone to yell "WTF" when they see you have no testbench. We had that stuff figured out in the PS1 days.
I see this is about game development, but in the web/server world idk how anyone ever thinks its acceptable to make development impossible on your local laptop/computer. Everyone time I do a code review I need to ensure they didn't break local development. It blows my mind that I need to do it, like they think its ok to build and deploy for every little test. When I first joined I spent so much time just getting it to work on my laptop. I really just don't understand the mindset. This is has been the case in multiple places. Who was this first person to break local development and decide it was ok. Please fire that person, black list them from getting jobs ever again.
We have so many tools to make this easy now like docker and yet it still gets messed up.
I think this is why I love green field projects so much. More than being able to work on my own code and design from scratch, I appreciate the fact that no one has messed up how testable and runnable the code is from a developer machine.
In some environments, I imposed a rule: “the tests must continue to pass, and I must be able to use the application on my local machine in an airplane on crappy wifi”
AWS CloudFormation had one of the worst user stories related to iteration time a few years ago.
The lack of feedback between the writing of the yaml file and validation of the structure/type/format was frustratingly slow. I would pay good money for better tooling in the Infrastructure as Code(IaC) space.
Waiting for the resources to update, fail with cryptic error messages, then slowly rollback only to then fail the rollback. Now in an invalid state, manual resource creation was required before the rollback would succeed.
The AWS cdk has improved this significantly. As a result the sun shines just a little bit brighter.
How I wish a single line code change only took a 15 second build.
The java app I work on regularly takes 3 minutes for a single line change. Then you just need to pray that jrebel will work today and actually hot reload your change. Otherwise you have another 8 or so minutes of redploying weblogic.
Of course the ideas suggested in this can help, but when you need to run an integration test which also takes minutes to run you realize that 3 lines of code isn't bad for a days work.
Your problem is WebLogic, not Java. Try a lightweight app framework (Dropwizard, SpringBoot...) that embeds the "app server" right into your app. Of course if you're using WebLogic, you probably don't have a choice.
Docker helped me to significantly increase unit test speed on one project. Each test was recreating database, run dozens of DDL scripts over and over (to ensure clean environment). I reimplemented in in a way that DDL scripts were run once, then container with database was commited to an image and that image was re-used for every test. Also multiple containers were run at once, so tests could be run simultaneously. Docker is a miracle technology sometimes.
> 3. Does not allow running multiple test simultaneously.
It does with almost every form of transaction isolation mode. Read uncommitted in Postgres still behaves as read committed, so in Postgres you're never missing the mark here.
> 1. That wouldn't work if code under test uses multiple transactions.
A lot of abstractions interpret nested transactions as save points for this reason.
It's not the only valid pattern, but it is a consistently behaving one that's straightforward to implement and gets you most of the UX you want from integration test design.
I still think that tests would be flaky without complete isolation in a different database instances. Transaction isolation is nice, but for serial isolation you would get errors when you can't commit that transaction and you'll need to repeat the test (which might cause further issues) and for lesser isolation levels you would get some phantom reads which might manifest as a rare test failures which is a terrible thing, because you can't reproduce it.
Would it be possible to cp /var/lib/your-database /var/lib/your-database-clean after it's been set up, then after each test just overwrite /var/lib/your-database again? I'm sure that the database will need to be properly shut down and restarted, but that is far less expensive than rebuilding a large database each time.
Disk is cheap. CPU and memory are the expensive assets.
That's not normal, because the test is now not testing the code in its eventual context (which is : the transaction commits). In my experience unfortunately this is a very common pattern (tests are far removed from "reality").
Only if your database lacks the tools to do it effectively. "The transaction commits" isn't exclusive with "and you can roll back to before it started".
As someone writing Lisp code, this feels so archaic. Then I remind myself that Lisp is older and than those games and that the future just hasn't caught up yet.
Not full time, but documenting interfaces, speeding up tests, generating stubs and harnesses, profiling pain points - all these I find to be meditative tasks, and I do them when I'm not 100%.
I can do about 4-5 hours a day of "difficult" programming, but then I can easily do another 4-5 hours of this maintenance type work, and the cumulative effects of repeated maintenance time keeps everything running super smoothly.
I'd add that the cumulative effects of always knowing that the maintenance backlog is under control and that you always have time for the maintenance makes it much easier to consistently deliver the 4-5 hours of "difficult" programming.
I actually love chasing down other people's bugs. I often get to fix other developers new shiny features shortly after they've hit production. And I love the challenge.
It's also nice that bug chasing has less of an expectation of giving good times estimates, which the software industry is famously poor at anyway.
Maintenance type work tends to be chasing issues you don't personally care about in increasingly arcane edge cases or trying to intuit why someone else, who has moved on to something that sounds way more exciting than this, came up with a specific solution.
Even more so in a lot of orgs maintenance work gets hit by the "cost center" mentality and management only rewards not hearing about it.
If I had to order the things that make me productive, starting with the most important:
1. It’s easy to have a fair amount of confidence in changes without even needing to run them. Strong static typing, good/clean abstractions, local reasoning - little to no mutability, code is mostly pure functions with side effects pushed to the edges. Being able to just write a bunch of code without needing to run it at all (and having it “just work” the first time, most of the time) is HUGE for productivity
2. Excellent automated test suite, with a nice pyramid (small number of E2E tests, solid number of integration tests, exhaustive unit tests). Also excellent monitoring and alerts, with automated canary releases and automated rollback. Basically make it so that it’s hard for mistakes to fully reach production. Being able to be a bit of a cowboy, safely, is huge for productivity too. I’ve worked on plenty of systems without this, and then I’m MUCH more careful, but when you really do have this, it makes everyone so much faster, especially once you learn to stop worrying and trust the guardrails (for simpler changes)
3. Test suite runs quickly locally, and is not flakey
4. Running the app locally is quick. Also ideally I can run just the one component I’m working on (like a front end, or service, or whatever), but have it fit into a full system running elsewhere
5. Full build/test/deploy process on CI is quick and reliable
This article emphasizes 3 and 4, and they’re certainly very important, but I think 1 and 2 are even more important. With them in place, often I don’t even need to do 3 and 4 for more trivial changes - just “I’m pretty certain this works,” and then the tests all pass in CI, and I’m very certain.
Compared to systems without 1 and 2, where I have to basically change a line, then run tests and/or the app because I’m not sure it’ll work, then do extensive manual testing to be sure because the automated test suite sucks. Muuuuuuch slower.
3 and 4 are great, and necessary for more complex changes, but 1 and 2 let you ship simple changes super quick, without even needing to run anything locally. That’s fastest of all, and can still be quite safe in the right environment.
I agree with your list but really need 4 myself to be a happy camper. Often it's neglected once 2 is in place as most don't seem to find any value by it, or don't care to maintain it. I find that having the possibility to run the real thing (in some capacity) in a cozy development environment with debugger is very useful to quickly learn how parts in the system interact with each other. I have a hard time doing that with unit tests, perhaps because they are so boring.
Oh yeah, I definitely use 4 frequently too - for basically any bug ticket, or for any more complex change/new feature, I want to run the full app. Having this be fast/easy/reliable is huge. If it’s a service oriented app, and I’m modifying a single service, I especially love tools like https://www.telepresence.io/ - run my service locally, with an “auto recompile/reload on save” setup (and sometimes a debugger), while all the other services are running in the cloud, but communicate with my local service.
However, a lot of changes are just small, trivial tweaks, and if 1 and 2 are strong, I don’t even have to bother with 3 or 4.
> Being able to just write a bunch of code without needing to run it at all
I work with a frightening number of programmers who assume that they can do this... so much so that they never bother testing to see if they were correct in the first place.
FWIW, I always write tests for my changes, as long as they have any impact on behaviour (i.e. not writing tests for adding a little logging or metrics, but am for any new or modified business logic). Every single dev having a strong commitment to automated testing is a requirement for an environment where you can sometimes get away without manual testing.
For any changes where it's not obvious that it'll work in practice, despite the tests, I also run the full app locally, and test manually. Or for any UI changes, obviously. For backend changes, for some codebases, running the full app locally is required for ANY change, period - hard to reason about code, little-to-no type safety, weak overall test coverage, etc. But if you have a simple backend change in an easy to reason about code base, that has great guardrails (great automated test coverage, great monitoring/alerts, canary deploys with auto-rollback), there really is not much reason to actually run the full app for a simple change, which is a nice productivity boost. Just make the tweak, write/modify tests if necessary, and if everything passes on CI, you're good to go.
For the 8 years before my current job, I was working in environments that weren't safe enough to do this ever, and was horrified when devs would merge without actually manually testing their changes. But my current company/environment has a codebase with strong typing and nice abstractions/local reasoning, great automated test coverage, and great monitoring/alerts and canary deploys with auto-rollback. Took me a bit, but I've embraced the "no need to run the app for simple changes" approach, and it works great in this environment - highly productive and still safe enough.
Hah fair, I have too :) But generally have found it not hard to convert them to embracing automated testing, as long as there's a strong automated testing culture at the company.
Whilst I very much agree with the premise I do think this exposes a fallacy many programmers fall for. That passing unit tests means your code actually does what you think it does. Using unit tests this way is a crutch to avoid improving startup times, implementing hot reloading and other schemes to skip gameplay. You lose an awful lot by not testing your code for the job it’s actually expected to do in situ. In games this results in increasing the total iteration time as it’s usually now a different person who actually runs the code, find issues and submits them back.
1) Use unit tests to move quickly through to implementing The Feature
2) When The Feature is complete, run in a local environment to confirm it works
3) When that works, move The Feature to a development environment that more closely mimics Prod
4) Deploy to Prod
Each one of those steps takes an order of magnitude longer than the previous one, so should be done an order of magnitude less often. However if you are finding inconsistencies between steps, then alter local/dev to ensure more consistent testing. (i.e. if it works on Dev, it should just work on Prod, however there are always little issues).
Right but 2 often ends up being perfunctory whereas if you can test routinely against the local environment you’ll see a lot more and for example course correct earlier. It depends on what you’d consider an iteration as well. I don’t think the loop is closed until you’ve seen your code running in situ. Unit tests might let you have more confidence that things will work at that point.
Running in situ could also be integration testing FWIW. Depends on what it is you’re making.
You need both. If you don't have unit tests your feed back loop becomes way too slow, but then you of course have to test the "real" running of the program properly before shipping. And you need at least one person who didn't write the code to test it.
Unit tests are useful for some things. I can’t unit test my way to a game feeling good. For that I need a fast feedback loop to the real program. There’s more of that in most applications than most people suspect. And routes to get there. As programmers we make a lot of little decisions along the way and those don’t get proper consideration unless you can see that as the feature evolves.
No unit test should be mostly to make sure the old assumptions for code you don't know still hold and warn you if you break something.
Ofc if you do the code and the unit test, all you validate is that the code does what you assumed was right. But in 2 years, when someone touches something unrelated, he'll kiss you on the mouth that your test noticed an edge case that started failing.
If it's not your unit test catching unintended changes, it's your client. You may not want that :)
I'm not arguing against unit tests. Just that using them to speed up iteration is not a good way to approach that problem. They can of course be useful for other things. And obviously the things they don't test will still be found by the client in particular all the things you miss by not actually increasing iteration speed.
Neves Law says that "the harder a bug is to find, the smaller the fix will be". The worst Neves ratio I ever saw was about a week to two bits, where a plus (0x2b) had to be changed to a minus (0x2d).
I think the core point isn't about coherence, but tradeoffs. Some tests are not worth holding onto: ones that are slow, flaky, and test code that is of marginal importance. In those cases, the ideal tradeoff can be to delete the test. Perhaps to rewrite a better one, or perhaps to just 'eat' the risk incurred by removing it in favor of improved iteration times and the other long term benefits of not having to maintain it.
As always, mentioning that Your Mileage May Vary (YMMV) puts the emphasis/decision on the people facing this or that situation rather than blindly applying advice found on the Internet.
This is probably a controversial opinion, but I think that working in an environment where time-to-iterate is high is actually very beneficial for improving your skills. It's one of those things that may feel overly burdensome in the short term, but is better in the long term. I say this as someone who taught programming with beginners, and observed what many of them will do when given an IDE, which at the scale of the code they're writing, makes the time-to-iterate very short (press a key and you instantly see your changes). They end up getting into a "dopamine feedback loop" that causes them to continually make tiny changes and rebuild/run, and the code written as a result of this looks exactly like what you'd expect: it barely works, and is full of redundancies, "dead ends", and other evidence that its author was probably not thinking of anything more than the next line or two when writing it.
In other words, reducing the iteration time reduces the motivation to get it right the first time, and the associated deep reasoning/"mental execution" skills which are required to do so. In order to develop those skills, one should strive to write as much code as possible before running it, and a high iteration time assists with that.
Also, I'm tempted to continue the title with "...unless it's APL".
Your example is valid for beginners but once you're getting experience you don't code like that anymore.
After 13+ years of C#, on simple projects I can code for a couple hours without compiling and get the pleasant surprise that my code work the first time I run it. And I'm certainly not a genius, I'm a 1X programmer.
Eh, ten years a professional, I frequently fall into it still, where you forget to think about the best solution and just.. mash. Sometimes it works to get a solution that can then cleaned up, other times I waste an hour before I snap out of it. I saw someone post "Stop thinking and look!" as universal advice once, but I need the inverse just as often.
I still do. Of course dabbling into REPL-based solutions in my youth might have influenced that.
For me it is crucial to always have have the code in some compile-able and testable state. I iteratively work towards the simplest, most naive solution that gets the job done. After that it is refactoring time and I form it into a proper solution.
The reason I do so is because only after having coded the naive solution, I have an concrete and proven correct understanding of the task and can pick the appropriate abstractions. I always see colleagues with an less iterative style use premature-abstractions and over engineering on the basis of "we might need it" instead of knowing.
I do invest some time into thinking about the right data structure though. (Not much about algorithms, as they mostly follow from the data structures anyway.) Also if I code a well understood problem, I might use a more top down approach.
(Not a critique of your workflow, just wanted to share mine.)
I have been coding for over 30 years and I can barely write one function without at least making some typos if not logical errors. I would never code for a couple of hours without compiling/testing.
There is a cut off period though. In my current role there is one project that takes 15 minutes to clean build, and another 15 minutes to run the test suite. Incremental builds are fast, but sometimes/often the compiler decides it needs to rebuild everything.
Additionally for some reason on my machine, maybe 25% of the time it gets stuck while building and I need to kill it and start again. Usually I go and get coffee while it's building, but there's only so much coffee or Reddit you can drink. Many times I've taken a 30 minute break while it builds and tests, then come back to find it got stuck and need to start again. Anytime I have to work on this project I try to put it off as much as possible, as I know it's going to be painful...
In my experience developers develop their own "unique style" to deal with with this problem. Most often it is horrible pile of mess. Often without unit tests since they do not give much feedback. And acceptance criteria is simple "it compiles and prints ok".
With short iterative changes I have unit tests and fine grained git history, to untangle it.
This holds true for ML models too when the data is large. I've seen people(myself included) when the data is small, the iteration is faster when selecting the features but when the data is large then you have to think about the features that you want to select a bit more deeply, test it out on a sample and then submit it to the larger data set.
> They end up getting into a "dopamine feedback loop" that causes them to continually make tiny changes and rebuild/run,
Yeah that's what I'm doing all the time. Just constantly testing the thing after every little change. If the test is fast, I like it. Probably there is some dopamine thing. I have thought of it in the past as like a slot machine.
> and the code written as a result of this looks exactly like what you'd expect:
Why would you expect any particular thing?
> it barely works, and is full of redundancies, "dead ends", and other evidence that its author was probably not thinking of anything more than the next line or two when writing it.
My code isn't like that though. Maybe that code was like that because it's beginner code.
> In other words, reducing the iteration time reduces the motivation to get it right the first time
I'm always hoping it works on the first time but you know how that goes. Even when it's close there's some little thing. The hope of getting untested code right on the first try isn't really something I entertain too seriously anymore. Short iteration is lovely.
Getting a particular piece if code running from the first time is so unexpected that I'm like "Huh... that's pretty cool" everytime it happens.
no need to spend 3 more minutes looking at the code when a 2 seconds build will tell me there's a Segmentation fault right there, or when Valgrind will tell me I missed a free ().
Only a small fraction of segmentation-fault-causing errors will be revealed in 2 seconds of testing. Thinking about all the ways your code could fail, and being able to keep a mental track of the constraints you need to observe, is a useful skill to acquire.
Well, yesterday I made a rather significant change to the code. Essentially, a reimplementation of an existing function in a completely different way, to avoid a memory consumption issue that cannot be solved in the old approach. 68 insertions, 15 deletions. It, unexpectedly, worked correctly and passed all tests from the first attempt, even though there are threads involved, and they are my weak point. But I didn't have a "Huh... that's pretty cool" moment. I had an "I don't like this code, it's too complex, it must be wrong somehow" moment.
> Also, I'm tempted to continue the title with "...unless it's APL".
Came here to say that. But jokes aside, as an APLer I wouldn't be able to bear waiting for things to compile. I regularly edit functions while they're on the stack, then have them continue…
Those short cycles are great when learning a new language.
Experiment with it, try new things, understand how the language works.
But at some point you are better thinking about what to write, how to write it, which structure you will use.
And this skill has to be acquired as well.
I worked on projects with long compile times and nowadays I will often write code all day without executing anything and then spend the last hour running and fixing it. Maybe that's a skill I developed because of long compile times, but whenever I work on a project with short ones I feel much more motivated and productive. Your experience is probably biased from watching beginners learning to code.
I even see the mindset spreading to other activities, like writing an abstract, paper, proposal or preparing slides:
I regularly (more often than not) see students using their supervisors as "CI-chain", submitting iterations with lots of stupid errors, half-baked sentences, unclean structure... because they learned to hit CTRL-b and see what comes back.
Of course there are reasons to talk about half-baked ideas, but if your task is writing a proposal or a section, you should at least be happy with what you wrote before you send it to the Prof, you're wasting his time.
But I also want to say that CI, unit tests and quick iterations are still helpful, even if you know how to approach a problem (i.e. think, research, draft, test, revise, etc.)
Your stance looks self-contradictory. An introduced change either brings enhancement, or not. In the latter case you wouldn't expect anybody to be excited. It means one only gets dopamine boost when getting closer to a goal set, i.e. the feedback works absolutely right. If after a bunch of such changes one arrives at "barely working" it indicates poorly defined goal, and/or lack of underlying knowledge. And has nothing to do with length of compilation. Actually, that's what we can expect from beginners, right? With quicker feedback they, most likely, will have a chance to absorb knowledge quicker.
Isn't that just what it's like to be a beginner? I think it's difficult to grasp when you've been coding for a few decades that new developers don't immediately see what effects a piece of code has, they can't just look at a loop for 2 seconds and say "oh, there's an off-by-one there". Sure, I can worry about the big picture while I'm typing out the code because I don't need to pay much attention when I'm typing out the code. The syntax of code is a no-brainer activity. I've been adding semicolons at the ends of lines for longer than some developers have been alive, it's not going to be something I miss. A beginner needs to focus on the details of the lines, which means their code becomes myopic and line-focused. I don't think you can meaningfully widen the scope before you're able to do the simple steps automatically.
Experienced developers typically don't sit like Hari Seldon predicting branches 50k instructions into the future with some extremely elaborate grand master plan, they're so fluent in the code and architecture that they don't need to. This is tacit knowledge that is acquired along the way, I don't think it can be meaningfully practiced.
I think that final sentence is unduly pessimistic - after all, if practice did not play a big part in expert programmers becoming that way, what did? What hope is there for the novice who is frequently making off-by-one errors and the like?
Maybe you are saying that there is no deliberate method of practice, but I think there is: whenever you correct one of your mistakes, try to figure out how you might have written the correct code the first time. Often, it might be a boneheaded mistake, but a significant number of cases have something to learn from. Can you make an argument for the current version being correct, and the previous one not? If an API did not work the way you expected, did you overlook something in the documentation? (Maybe just that the documentation was ambiguous, and you might have realized that earlier.)
I think OP's point is that slow turnaround encourages that sort of thinking, and it does, though it is quite a cost to pay.
> I think that final sentence is unduly pessimistic - after all, if practice did not play a big part in expert programmers becoming that way, what did? What hope is there for the novice who is frequently making off-by-one errors and the like?
It's not that the knowledge can't be acquired, but this is a type of knowledge often can't be explicitly practiced through some training routine.
A beginner may of course learn to check off common errors they've been told about, and may arduously step through the code in their head line by line, but an expert sees the code is wrong unconsciously and understands what the code does without stepping through it. The only way to learn that is to engage with code for decades. There are no shortcuts.
I agree that competent, experienced programmers look at code differently, and much less linearly, than do novices, and there have been studies that show this. On the other hand, a person could (and people do) spend years programming without learning to look at code in this way. The people who do improve must be doing something different.
There are no shortcuts, AFAIK (though I think most people can achieve most of their potential in less than a decade), and I am sure you cannot get there through rote learning. Are you suggesting, however, that the sort of approach I am advocating is not very helpful, or just that it cannot be explicitly practiced?
> Experienced developers typically don't sit like Hari Seldon predicting branches 50k instructions into the future with some extremely elaborate grand master plan
The problem is many times, you can't really reason through the issues your encountering. For example, if you are trying to talk to a server with an undocumented API flow, or you're trying to coax an in-house library to life, you have no option but take baby steps and test every single change to see if you made progress.
In environments (like C++) where the compile times are high, and it's pretty easy to introduce almost impossible-to-trace bugs, I'm often tempted to do a bunch of changes, and then test them all at once, which will invariably result in something breaking and me having to spend an inordinate amount of time to figure out what went wrong.
Yeah, this breaks down in this case. I've wasted so much time making small changes trying to figure out how to adapt a model in the right way for it to work in both real and test code, getting a weird injection to work, etc. on systems with agonizing build times.
> Also, I'm tempted to continue the title with "...unless it's APL".
Or lisp. Or smalltalk. Or forth. There's an entire subfield of computer science, headed by some prestigious people, who believe that working in a REPL is better precisely for these reasons.
At this point I deploy the left-handed user argument: different developers probably find these tools and workflows have drastically different effects on their productivity.
Perhaps also a "code kata" thing. It may be worthwhile to make beginners do each for a few months just so they have the experience (and find out their own "handedness")
I genuinely don't know what to make of the handedness argument. On the one hand it makes a sort of intuitive sense (we're all individuals! Of course we all do things differently!) On the other we're all running on the same hardware, give or take. It strikes me as like learning styles: yes, people might have expressed preferences, but if you run the numbers we do all learn in fundamentally the same way, independently of what we say we prefer.
My longest compile times were when I was doing iOS development (native), and even then I couldn't kick the habit of relying on quickly looking at my changes and using logging for debugging. Not the best.
On the other hand, at least it's a motivator to look at the build pipeline every once in a while and see if there's anything that can be improved.
That said, working in a typed language with good IDE support definitely helps build up the confidence that things will work as intended before you hit compile.
Interesting. I was thinking that you pay more attention with manual. The point of manual shifting is not in how the car works. The point is in knowing what will happen before it will happen.
> whether looked at by human factors leading to accidents, by driver age or by accident type and age, the results all support the conclusion that, among all accidents, the accident rate for AT vehicles is twice as high as for MT vehicles except for head-on collisions
> But automatic lets you pay more attention to the road ahead, whether you are a new or experienced driver.
In my experience, drivers who primarily drive an automatic are easily distracted from the task of driving. They turn around to their children, fiddle with the radio or climate control, I've even seen many adjust their seating position and steering wheel position while in motion. Not to mention the internet-connected smartphone calling them incessantly, which is propped up close to eye level and blocking part of the windscreen.
Contrast to manual drivers, who will happily drive along without music. Far less prone to distractions and far less likely to touch the phone.
One could argue that the focused people who enjoy driving do prefer a manual transmission, to which I'll agree, but I'll counter that the manual transmission itself fosters that focus and enjoyment in the driving experience. The boring automatic is what invites distraction and does not foster enjoyment in the driving experience.
In the same vein: I'm very used to "printf debugging". When I use a language where it's harder to do that, I feel like I improve as a programmer because I spend more time reasoning and thinking about my code instead of directly trying to fix the problem I have right now.
As a more general thing: you can learn a lot by removing tools that you usually use from your toolbox. For example, going from a managed language to manual memory management. Going from an IDE to shell commands. Removing tools like language servers. Each time I did one of those things, I learned a lot. However, there are lots of things to learn and I don't know if this was the "most effective use of my time".
Another example of hard to balance learning: I'm currently working for a company. Should I focus my learning on their business and our codebase, or on general technology? General technology is a transferrable skill, and I'm scared of being "trapped" in a company if I focus on their specific parts too much. But on the other hand, companies want to recruit efficient people, and I think focusing on the business and our codebase would make me more efficient.
I'm a junior engineer, so maybe that's due to a lack of experience. If anyone has insights to share about that, I'd love to hear them.
My strategy is you spend 80% of the time on the company problem, and 20% for learning. By that you have both the time to deeper your professional skills and get new inspiration from new technology.
I've observed "trial and error coding" many times in my life. This isn't caused by a short feedback loop at all. The people who work like this will work like this with a long feedback loop too, they will just take a lot longer to do anything. These people are simply incompetent programmers.
This is the same argument in favor of manual photography with film, instead of a digital SLR, or a camera phone.
While you might learn more if forced by the constraint, the opportunity cost is quite high, and these days you'll never replicate the experience of going off with your glass plates and camera packed on a mule to photograph the Sierras.
Not only all of that... but Ansel Adams spent a TON of time in the darkroom, and pretty much invented most of the techniques that Photoshop emulated in their early versions. Even faced with limited chances at exposing a negative, he iterated the heck out of the development process when making prints.
I share your view that there is value in the focus, but I don't think that anyone can afford the opportunity costs.
If you're ramping up, you cannot write a significant amount of code without errors. If you try, it's much harder to figure out which change caused the failure of a previously working system, because you're less aware - by definition, ramping up - of the mapping between edits and failure modes.
Once you're ramped up, yes, it's more efficient to produce more code and fix the odd issue that crops up. I rely more heavily on high level, slower running functional tests on code where I've been the primary author or have worked on the system for a long time. But just be aware of what you're optimizing for: tenured employees.
I wouldn't kid yourself that it's "deep reasoning", beyond people on a dopamine kick, or beginning programmers working with simple data structures. You can't reason much about a system you don't know, because everything is details - millions of lines of code, you can't mentally evaluate it. It's not until you've built a mental model of the moving parts, and are able to mentally abstract away details not relevant to your task, that you can think things through in the large.
When I was a beginner I'd write code and run it very fast, after each small change, even though an experienced developer could probably compile it in their head and see whether it worked or not.
I think just as you get more experience you tend to write more code before you run it. I've written an entire (relatively small) application and written the tests for it before I even thought about running it. Run it, few tests fail, iterate a tiny bit, done.
It's just natural as you get more experienced that you can reason about code in your head without resorting to running it to confirm it.
> In other words, reducing the iteration time reduces the motivation to get it right the first time, and the associated deep reasoning/"mental execution" skills which are required to do so. In order to develop those skills, one should strive to write as much code as possible before running it, and a high iteration time assists with that.
A high iteration time [a.k.a long feedback loop] leads to distraction more frequently than to “deep reasoning”. Human brains seem to be wired to find distraction after about eight seconds have elapsed. [1]
Furthermore, let’s not forget that an IDE doing a lot of error checking real-time, often including partial compilation and execution, so nobody running an IDE is coding for a long period without compiling, they’re more likely coding for a long period without integrating.
If we reframe deep reasoning as “stay engaged with the problem while waiting for some feedback”, we can see that there are a lot of tricks for this: Unit tests, for example, or using a different medium like pencil and paper or a whiteboard. (I’m sure this group can come up with a great list.)
There has to be some nuance here. Feedback that comes when I'm not at a stop/review point is just a distraction. Is why aggressive spell checking is annoying.
This was a thing learning to type, for me. Our class would transcribe from paper to document. Those that watched the screen were slower, once folks learned to touch type.
That is, learning to get into deep thought is a skill that eventually doesn't want the distractions. You want to focus and produce, then review.
That is a really interesting perspective, and something I have noted about myself when working with Lambda in AWS, which has a god awful iteration time.
My two comments however would be that when you don't know your context, lots of probing is required, and that can take a lot of iterating. In industry work you're very often working on a small part of a larger software orchestration, and it's often entirely new to you when you're asked to work on it. So you don't get access or time to understand the full scope of the software to reason about. So you need to "discover" your context as it actually is at run time. This is especially true when bugsquashing.
My second comment, is that we should all have a good debugger. Having a debugger is like having a superpower. I can come into a codebase I've never encountered before, step through as it runs, gain enormously useful context, and solve the problem I wanted to solve, without ever running the code to completion or having to read every bit of logic.
If I'm working on someone else's codebase, usually I spend time reading to figure out the general architecture, then dive right in with a debugger and start stepping through. I find how to trigger the code paths related to my task, see what's happening as it happens, and get a really good understanding of how it's all running.
In what might be another controversial opinion, a working programmer without a debugger is like a working carpenter who's chosen to do everything with a knife. They may produce good work with their own way, but they could be faster with the proper tools.
That's what you'd like to think. But the quantity vs quality pottery grading story is a tested and retold one. Also this might just go to show that you've settled for a slower pace of progress. And this term 'correctness' that indicates perfection and the enemy of perfection - bad for getting things off the ground.
I would say that practicing in a long-compile-time environment is part of improving your skills, but that once you have a certain level of skill with a language and tech stack, then it is beneficial to go back to having compile times of a few seconds or less. Generally speaking, it is far better to have a rapid iteration time to quickly test that the ground you are working on is solid as you progress.
But definitely having a phase where you are dealing with long build times helps you enhance one aspect of software development!
Wow, I never quite formalised this in my head, but I think it's true!
At least for small delays, say 30s-3m, I'm not tempted to go for that (other) dopamine shot of checking messages or whatever at all. I keep thinking about the problem I'm solving, what to try if the thing I'm currently trying doesn't work, and so on. I don't remember ever feeling limited with such cycles - literally only when fiddling around with CSS, which I quite often just do in the browser's dev tools for the fastest possible feedback cycles.
I would not conflate slow feedback loops with intentional breaks
For me slow feedback loops push me into dopamine seeking activities, intentional breaks push me into relaxation and deep thought
Recently I got a treadmill desk and the physical exertion reminds me to take breaks where I can get off and relax and potentially think about a problem
Slow tests running causes me to not write tests and look at YouTube
The drawback with long time to iterate is you have to be in flow to do things. That's non-negotiable in environments with a lot of meetings.
There's also a "sketching" element with faster iteration. On slower iterations, you're engineering. The engineering skill is often underdeveloped but sketching is more valuable without a dedicated designer.
Also some things are poorly documented, often things like UI and browser code or legacy support for say, older Android OS. So even if you have your calculations right it ends up wrong.
Many early programers, Knuth etc, have described how writing programs on punch cards, submitting the job, picking up the results much later had a big influence on how they thought about programing. The lack of immediate feedback, counterintuitively, made them better programers it seems.
Or the people drawn to the tedium of early programming were just really good and mentally twisted in the right way to enjoy it.
My first programming experiences were BASIC on TRS-80, TI-99/4a, and similar machines. It was instant feedback, for the most part, and I enjoyed being able to make small changes and seeing the results. I was around 12 years old and that worked well with my attention span at that time.
> Or the people drawn to the tedium of early programming were just really good and mentally twisted in the right way to enjoy it.
Agree.
I think what was happening was the the lack of feedback fit the mental models of the people who were attracted to computers early.
For another example of this,Ken Thompson famously said that he never quite understood why anyone would want to see the whole file they were editing instead of the single line.
But instant feedback attracted more people to programing.
I'd say the opposite. The quicker the feedback loop, the faster you can learn. Just because some beginners choose to get the feedback faster doesn't mean they are doing something wrong.
That is a very good argument for controlled large feedback times in a learning environment, also, just some of the time, when thinking about your code is the focus of the exercise.
I'm one if the people that thinks that coding in paper is a great learning task. I tend to not ask people to do that in practice because it's a very effective way to lose control of a classroom, but I do use the second best task that is asking people to explain their code.
Anyway, if thinking about the code is not your focus, the increased feedback cycle is all cost and no gain.
Let me tell you, time to iterate on a huge 50 year old code base is through the roof and reaching out into interstellar space. The resulting code is pure and utter trash.
I would kill to have to have a quick feedback loop and at least be able to address quality issues through training, culture, or technical controls.
The alternative is fighting against a giant, heavy machine with a lot of inertia in the wrong direction.
I think "high iteration time" taken to the extreme is programming without a computer, like when taking a walk. You construct a very detailed mental model of how the program operates and interrogate it in your head.
This only works once you're a bit more advanced, but the one sure sign of a senior dev is the accuracy of their mental model. Which is also what allows you to write your piece without a continual back and forth with the compiler.
This all said, I greatly enjoy ping pong with a repl or instant hot reloading when doing visuals or sound.
Recently decided on using TDD for some library code and aiming for >90% test coverage at my dayjob. Frankly, things do take longer to develop this way and i needed to put in more thought into how the interfaces should be structured, vs just doing what people sometimes do in other projects - just relying on concrete implementations or using frameworks that use reflection to do what they want.
However, the experience itself feels like a positive one - even if the code takes longer to craft, it's much easier to work with, since now i don't need to worry about sloppily written if/else chains that depend on enums or runtime type checks, but can utilize different implementations as needed. Also, writing the actual tests allowed me to be sure about how everything would actually work, all the way up to discovering that Paths.get (Java) on Linux accepts almost anything as a valid path string, whereas Windows has actual validation rules, leading to many headaches in regards to the code coverage quality gates that i set up, since the coverage differs based on which platform you run on.
But i guess that my point still stands: in certain circumstances slowing down might actually be a good thing, when you really want to know how your code will work under most circumstances and make it maintainable.
Of course, on the other hand, compilation speeds and other factors in regards to the speed of iteration definitely shouldn't be overlooked either, and having everything else apart from writing tests and actually thinking about how everything will fit together be faster is a good thing! I'm not sure what i'd do if my test suite took 10 minutes to run instead of 10 seconds as it currently does. Probably drink lots of coffee.
I guess it depends on slowing down for good reasons, vs just wasting time because of tooling or other sub optimal circumstances (e.g. what was described in the article, personally i've also seen a local API service be pretty chatty with a remote DB which was slow over a VPN, yet no one had a local DB with all of the migrations in place, and a bunch of other things like that).
Offtopic: Anyone remember how fast Pascal compiled? Now that was a really nice stack to do some stuff in, it's a shame that it never got as popular as Java, or didn't have tooling like JetBrains has, it felt like a more ancient Go (which is also pretty good as far as ergonomics go).
I have somewhat the same experience with my first real TDD-based project. Progress was slow, but the amount of debugging afterwards as almost zero. It feels slower, but more predictable.
> Recently decided on using TDD for some library code and aiming for >90% test coverage at my dayjob.
I thought the point of TDD was that you don't write code unless a test requires it? In my mind (and the little experience I have) that translates to 100% coverage, with exceptions of really annoying side-effects that are near-impossible to test reliably.
> In my mind (and the little experience I have) that translates to 100% coverage, with exceptions of really annoying side-effects that are near-impossible to test reliably.
Yep, like Java supposedly being cross platform, but having different file system implementations, because using one implementation across multiple file systems just isn't entirely feasible, for example, due to how paths are validated.
Thus, no matter what i did, i couldn't get the coverage to 100% due to semantics of how i'm supposed to validate that the low level call works and because i have branches in the test, one that checks whether the correct exception is thrown in Windows and a different exception for Linux. Even if technically the code paths are covered, telling that to a subpar code coverage plugin is hard.
So, i just settled on writing code for everything myself, but having 90% coverage be enforced on a class line level to account for the weirdness.
Going from "manual testing" to automated testing is the biggest jump. Unfortunately, lots of folks (both developers and managers) fall into the trap of thinking they will "use up" their development time on automated tests, and not really thinking how much time they're already wasting just to do (incomprehensive and hand-wavy) manual tests each time.
And while this strategy might work for the folks who actually wrote the code (and thus have tacit knowledge about it), the moment they leave and/or someone new joins the team, all that "speed advantage" is lost and it turns from minutes to hours to days instantly.
Automated tests really are the only way to capture the business logic of any code for it to not to become "legacy code" before its time.
Very good points. In his book 'working effectively with legacy code' Michael Feathers actually defines 'legacy code' as code without tests. Largely because of reasons like you state.
What do you think about code being too smart for current team? I encountered that use-case three times in my career. It was C++ program which was written by very bright person, used template magic, boost and stuff. Rest of the team were ordinary C developers. After that person left, they had to rewrite his program, because it took too much time to understand how it works and to fix or improve it. Second case was when someone wrote some helper program with Haskell which happened to be important. Same story, nobody knew Haskell, nobody wanted to invest into learning, so rewrote in node.js. Third story: someone used reactive API with Java (Spring Flux) and again same story: team don't understand it and slowly rewrote it with ordinary blocking API.
> Unfortunately, lots of folks (both developers and managers) fall into the trap of thinking they will "use up" their development time on automated tests, and not really thinking how much time they're already wasting just to do (incomprehensive and hand-wavy) manual tests each time.
Actually, currently am about to be asked when the feature will be done, whereas i've decided to actually write tests for it (reusable library code across multiple projects). As far as the reality is concerned, the people who say that writing tests "uses up" time are right: being slow now will be more visible than no one understanding why development takes a whole lot of time down the line, which is when many might just shrug their shoulders and go "legacy code". What's the benefit of making others able to develop code faster, if that makes you be slower - both facts probably being visible to management, though oftentimes without the underlying reasons behind those being relevant to them?
I'd argue that that's why even governmental projects in my country (that are often contracted out to random companies) oftentimes have poor test coverage and just generally not a lot of thought is put into the quality and sustainability of those codebases - they got paid, they could iterate reasonably quickly while the contract was ongoing, why should they worry about anything? The people who take over the project then can also deliver features more slowly, not only because of the poor domain understanding, lack of ADRs, lack of documentation, but also lack of tests - and still often get paid on a time material basis. Thus, no one actually wants to improve things, apart from me wanting to tear my hair out whenever i'm expected to work on garbage like that and fix their problems.
Thus, you see some people in the industry adopt an egoistical approach to it all - join a company, focus on the speed of their own iteration without thinking about the project in the long term, spend a few years doing this and then leave for another company where they'd do the same thing. It might be a cultural thing, but working on other people's projects instead of starting my own feels like losing at this point - no READMEs, no automated CI, no Ansible, no Dockerfiles, no IDE run profiles, no common linting rules, no code static analysis, no local DBs with migrations, no local setup scripts etc. Why should i be expected to toil away without results that are visible to the business because someone else was allowed to neglect the codebase?
At this point, i use unit tests and code coverage rules defensively: to prevent someone from jumping into the project and ruining how things should work with breaking changes, which will be caught by the tests, or by introducing untested and undocumented code, which the coverage rules will catch and make CI fail. Of course, depending on the culture, you might find that other devs talk amongst themselves and before long your tests have @Ignore added to them or have been removed entirely, which is the point where it might just be easier to look for a less dysfunctional environment, despite living in a country that others outsource to and therefore quality isn't a priority.
In short: i agree with your point, but i think that there are social problems at play. You need to actually care about the code that you're developing to prevent these issues in the long term. And even then, if you care, that doesn't mean that other team members will.
From my limited experience, governmental projects tend to have the worst code quality ever. At least that's what I see here. It can easily be explained by pervasive corruption — projects go to those who agree to return the largest kickback and are written mostly to steal money from the taxpayers. With such incentives, code quality does not receive much attention.
> working on other people's projects instead of starting my own feels like losing at this point - no READMEs, no automated CI, no Ansible, no Dockerfiles, no IDE run profiles, no common linting rules, no code static analysis, no local DBs with migrations, no local setup scripts etc.
You hit the nail on its head here, my friend.
When working with others' projects, I try to add tests where I've added/modified features, so that at least I can get a sense of it all and make sure I don't break anything. And for a completely untested codebase, it might make sense to add more integration-like tests (while still using mocks for external dependencies), such that simultaneously a bigger proportion of the codebase is covered, even if at slightly longer test times.
I agree. The issue is when you don’t have the test harness or the code just isn’t testable. Then it really will “use up” your dev time for not much gain. Or at least it’s a harder sell. My “dark pattern” to sell this is to offer to refactor a small portion at a time, then read through as much code as I can on nights/weekends looking for unhandled edge cases. When I find one, that’s the part I make tests for. Wo and behold, it found bugs! Bonus points if the edge case actually causes issues in prod or loss of revenue. The manager is now convinced we need this thing, or at least thinking about giving us time to work on it.
This is where I stop understanding modern ui frameworks. Instead of making an ui inner machinery (controllers) as a set of modules which could in theory run even in a non-browser environment, they pile up everything together into dom/etc. UI could be just a thin layer over usually testable ui-controllers, whose methods could be called by either controls or offline test suites (as in most desktop frameworks). But nope, they do mountains of “smart” views and test them instead as a whole.
…I just googled for “React TDD” and the top no-money result suggests finding divs in containers and spying on dom changes. It’s like testing your cli app by hooking into tty driver’s section in /dev/mem.
For what it's worth, a popular Python testing framework uses something like `expect` under the hood to simply check that the output matches a predefined output.
Interesting, in my experience React allows me to write more testable code in the front-end than I ever could. Given how React will handle updates to the DOM given changed state, I can extract complex state manipulation code (i.e. code that's most important to test) into independent functions that take state and an event and produce new state, rather than having that all intertwined with DOM manipulation code.
And when it comes to testing what your DOM looks like, even though that does often involve spying for changes (after all, you're testing whether they occur), that's not on the actual DOM but on the virtual DOM, which makes it easy and efficient to run in a non-browser environment.
You have a “data” as a model sitting between the DOM and the controller (mainly methods and watchers). There are some gray lines, like computed properties, but otherwise the controller can run without the DOM and vice versa, you can easily force the model into a state to see how the view will look like.
A lot more testable than the old spaghetti of onClick and getElementById().addEventListener.
In a future post, I will go over how web developers needs to start taking iteration time more seriously as the influx of new tools and frameworks starts to bloat up build times.
The newer tools in web dev have crushed iteration times to a fraction of what they used to be. A combination of things like esbuild, fast refresh, yarn's offline cache, and a few other bits can get your iteration times on a site you're manually testing down to milliseconds to both build and update (literally). Decent devs have been writing tests for years; the test runners could be a bit faster but people are working on that as well.
The React app I'm working on at the moment takes well under a second to build, under 10 seconds to run the test suite (could be a sign I need more coverage..), and milliseconds to update the code in the browser in dev mode. It's nice to work with. It's not even clever or special. A default Next.js app will do that. Create-React-app 5 launched a few days ago with more of the tooling too. In Vue things like Vite are based on the new fast tooling and it works well. Vue dev work is fast. SvelteKit even goes further by using Snowpack to remove the bundling step entirely. Snowpack claims a default refresh time of 50ms.
I really hope the author's article about web dev is simply "Yeah, update your tools and you'll be fine."
Aside from looking at changes in your dev methods, this is a good way to demonstrate to your management the value of having fast, up to date hardware on your desk.
This guy is young and probably not educated in modern dev. I refuse C++ jobs when I ask how they do unit test and they reply it's too hard in C++, and take Java jobs because there the first question THEY ask is HOW I do unit test, what I think of coverage metrics (hints: they don't matter much) and how I propose to enforce ALWAYS unit testing important code.
In this blog post the guy took 3 job change to DISCOVER unit tests, it's insane :( It's not like they know but don't have time, it's that he's a "champion" just for proposing limited scope testing.
I think the main issue about automatted unit/integration testing is not the language, but the area you are writing code in. Yes, C++ not having good unit test support hurts. But it hugely depends on the problem domain and whether the outputs are machine verifiable, and the inputs are easy to simulate.
It's hard to do unit tests in a large part of the area of gamedev. How do you ensure that a picture is rendered correctly? You can take a screenshot and compare it to a stored one. But then what happens if you do artistic changes that you want to do? You have to update the stored screenshots of the testsuite. Who will review that the changes all made sense? And more importantly, there might be slight differences in the output of GPUs, depending on driver versions, model, etc.
So let's say you have a bug in your game where if you walk through a level in a specific direction, the game crashes. The error is easy to check for: just make sure that there is no crash. But how do you create a reproduction of the bug? You could record controller inputs and play them back. Then a different department changes something how quickly players move and increase their walking speed by 10%. Suddenly your player walks into a wall and the test is basically broken.
Compare this to a CRUD app where you have well defined operations and their impact is well described.
That being said, even in gamedev there are areas that are well testable. You can do a unit test of the low level networking layer by trying to make a server and a client, dropping some packets, then looking if the packets still arrived because the networking layer sent them again.
Another thing here is that often times people are so busy "developing" product that really trivial efficiency gains get overlooked. Things like simply removing unnecessary parts from the build at least when you're testing / in dev mode. Also compartmentalizing the build to the portion of the app you're working on, ensuring it's not building the entire app when you're only testing one feature (you'll be surprised how common this is even especially at large companies). It's a huge value add to the rest of your team to bite the bullet and take an hour or so to shave off some minutes from the time you have to wait for a build. That one hour saves you from many multiples more (hours / days / weeks) of fiddling your thumbs in the breakroom waiting for a build to finish.
(better yet to have someone regularly scanning for these efficiency related things)
209 comments
[ 5.5 ms ] story [ 128 ms ] threadI’m of the opinion that applying great testing practices to unity programming is very important. It’s been fun synthesizing ideas I’ve encountered into my hobby game dev career.
I’m glad there are others out there doing the good work of generating content to popularize the benefits of good testing practice :)
[0] https://news.ycombinator.com/item?id=29459292
I don't know what drives that, but in most jobs I've had to fight to do the right thing.
Just recently I've made some optimizations to get a functional test suite from 65 minutes down to 3.6. Parallelism is my favorite lever, as it scales so well, and core count is still going up!
Running tests is the same type of thing!
We have so many tools to make this easy now like docker and yet it still gets messed up.
The lack of feedback between the writing of the yaml file and validation of the structure/type/format was frustratingly slow. I would pay good money for better tooling in the Infrastructure as Code(IaC) space.
Waiting for the resources to update, fail with cryptic error messages, then slowly rollback only to then fail the rollback. Now in an invalid state, manual resource creation was required before the rollback would succeed.
The AWS cdk has improved this significantly. As a result the sun shines just a little bit brighter.
The java app I work on regularly takes 3 minutes for a single line change. Then you just need to pray that jrebel will work today and actually hot reload your change. Otherwise you have another 8 or so minutes of redploying weblogic.
Of course the ideas suggested in this can help, but when you need to run an integration test which also takes minutes to run you realize that 3 lines of code isn't bad for a days work.
One day soon I will never touch Java again.
That makes each test start at the same, clean state.
2. Rolling back a transaction is not exactly free.
3. Does not allow running multiple test simultaneously.
It does with almost every form of transaction isolation mode. Read uncommitted in Postgres still behaves as read committed, so in Postgres you're never missing the mark here.
> 1. That wouldn't work if code under test uses multiple transactions.
A lot of abstractions interpret nested transactions as save points for this reason.
It's not the only valid pattern, but it is a consistently behaving one that's straightforward to implement and gets you most of the UX you want from integration test design.
It doesn't let you test interactions of parallelized code doing independent/interacting database workloads, but most web/CRUD apps don't do that.
Disk is cheap. CPU and memory are the expensive assets.
Step 1 Make it less dumb
Step 2 Delete a part of the process
If you're not adding step in 10% of the time, you're not deleting enough
Step 3 Simplify or Optimize
Step 4 Go Faster
Step 5 Automate
Agree, continuously re-evaluate your systems, they can always be better
I can do about 4-5 hours a day of "difficult" programming, but then I can easily do another 4-5 hours of this maintenance type work, and the cumulative effects of repeated maintenance time keeps everything running super smoothly.
Why bother?
Even if the whole project has no career purpose, it's more fun to write new code.
So things can be largely a question of perspective.
It's also nice that bug chasing has less of an expectation of giving good times estimates, which the software industry is famously poor at anyway.
Even more so in a lot of orgs maintenance work gets hit by the "cost center" mentality and management only rewards not hearing about it.
1. It’s easy to have a fair amount of confidence in changes without even needing to run them. Strong static typing, good/clean abstractions, local reasoning - little to no mutability, code is mostly pure functions with side effects pushed to the edges. Being able to just write a bunch of code without needing to run it at all (and having it “just work” the first time, most of the time) is HUGE for productivity
2. Excellent automated test suite, with a nice pyramid (small number of E2E tests, solid number of integration tests, exhaustive unit tests). Also excellent monitoring and alerts, with automated canary releases and automated rollback. Basically make it so that it’s hard for mistakes to fully reach production. Being able to be a bit of a cowboy, safely, is huge for productivity too. I’ve worked on plenty of systems without this, and then I’m MUCH more careful, but when you really do have this, it makes everyone so much faster, especially once you learn to stop worrying and trust the guardrails (for simpler changes)
3. Test suite runs quickly locally, and is not flakey
4. Running the app locally is quick. Also ideally I can run just the one component I’m working on (like a front end, or service, or whatever), but have it fit into a full system running elsewhere
5. Full build/test/deploy process on CI is quick and reliable
This article emphasizes 3 and 4, and they’re certainly very important, but I think 1 and 2 are even more important. With them in place, often I don’t even need to do 3 and 4 for more trivial changes - just “I’m pretty certain this works,” and then the tests all pass in CI, and I’m very certain.
Compared to systems without 1 and 2, where I have to basically change a line, then run tests and/or the app because I’m not sure it’ll work, then do extensive manual testing to be sure because the automated test suite sucks. Muuuuuuch slower.
3 and 4 are great, and necessary for more complex changes, but 1 and 2 let you ship simple changes super quick, without even needing to run anything locally. That’s fastest of all, and can still be quite safe in the right environment.
However, a lot of changes are just small, trivial tweaks, and if 1 and 2 are strong, I don’t even have to bother with 3 or 4.
I work with a frightening number of programmers who assume that they can do this... so much so that they never bother testing to see if they were correct in the first place.
For any changes where it's not obvious that it'll work in practice, despite the tests, I also run the full app locally, and test manually. Or for any UI changes, obviously. For backend changes, for some codebases, running the full app locally is required for ANY change, period - hard to reason about code, little-to-no type safety, weak overall test coverage, etc. But if you have a simple backend change in an easy to reason about code base, that has great guardrails (great automated test coverage, great monitoring/alerts, canary deploys with auto-rollback), there really is not much reason to actually run the full app for a simple change, which is a nice productivity boost. Just make the tweak, write/modify tests if necessary, and if everything passes on CI, you're good to go.
For the 8 years before my current job, I was working in environments that weren't safe enough to do this ever, and was horrified when devs would merge without actually manually testing their changes. But my current company/environment has a codebase with strong typing and nice abstractions/local reasoning, great automated test coverage, and great monitoring/alerts and canary deploys with auto-rollback. Took me a bit, but I've embraced the "no need to run the app for simple changes" approach, and it works great in this environment - highly productive and still safe enough.
I wasn't saying you were one of those people - just saying I've worked with a lot of them.
1) Use unit tests to move quickly through to implementing The Feature 2) When The Feature is complete, run in a local environment to confirm it works 3) When that works, move The Feature to a development environment that more closely mimics Prod 4) Deploy to Prod
Each one of those steps takes an order of magnitude longer than the previous one, so should be done an order of magnitude less often. However if you are finding inconsistencies between steps, then alter local/dev to ensure more consistent testing. (i.e. if it works on Dev, it should just work on Prod, however there are always little issues).
Running in situ could also be integration testing FWIW. Depends on what it is you’re making.
Ofc if you do the code and the unit test, all you validate is that the code does what you assumed was right. But in 2 years, when someone touches something unrelated, he'll kiss you on the mouth that your test noticed an edge case that started failing.
If it's not your unit test catching unintended changes, it's your client. You may not want that :)
It moves the deletion as a means towards the coherence rather than as an end in and of itself, and also includes refactoring as an option.
I hope it helps the less experienced developers grab the intention more explicitly.
And rightfully so :)
Tend to get a boost in application boot/startup time, too
In other words, reducing the iteration time reduces the motivation to get it right the first time, and the associated deep reasoning/"mental execution" skills which are required to do so. In order to develop those skills, one should strive to write as much code as possible before running it, and a high iteration time assists with that.
Also, I'm tempted to continue the title with "...unless it's APL".
After 13+ years of C#, on simple projects I can code for a couple hours without compiling and get the pleasant surprise that my code work the first time I run it. And I'm certainly not a genius, I'm a 1X programmer.
For me it is crucial to always have have the code in some compile-able and testable state. I iteratively work towards the simplest, most naive solution that gets the job done. After that it is refactoring time and I form it into a proper solution.
The reason I do so is because only after having coded the naive solution, I have an concrete and proven correct understanding of the task and can pick the appropriate abstractions. I always see colleagues with an less iterative style use premature-abstractions and over engineering on the basis of "we might need it" instead of knowing.
I do invest some time into thinking about the right data structure though. (Not much about algorithms, as they mostly follow from the data structures anyway.) Also if I code a well understood problem, I might use a more top down approach.
(Not a critique of your workflow, just wanted to share mine.)
Additionally for some reason on my machine, maybe 25% of the time it gets stuck while building and I need to kill it and start again. Usually I go and get coffee while it's building, but there's only so much coffee or Reddit you can drink. Many times I've taken a 30 minute break while it builds and tests, then come back to find it got stuck and need to start again. Anytime I have to work on this project I try to put it off as much as possible, as I know it's going to be painful...
With short iterative changes I have unit tests and fine grained git history, to untangle it.
Yeah that's what I'm doing all the time. Just constantly testing the thing after every little change. If the test is fast, I like it. Probably there is some dopamine thing. I have thought of it in the past as like a slot machine.
> and the code written as a result of this looks exactly like what you'd expect:
Why would you expect any particular thing?
> it barely works, and is full of redundancies, "dead ends", and other evidence that its author was probably not thinking of anything more than the next line or two when writing it.
My code isn't like that though. Maybe that code was like that because it's beginner code.
> In other words, reducing the iteration time reduces the motivation to get it right the first time
I'm always hoping it works on the first time but you know how that goes. Even when it's close there's some little thing. The hope of getting untested code right on the first try isn't really something I entertain too seriously anymore. Short iteration is lovely.
no need to spend 3 more minutes looking at the code when a 2 seconds build will tell me there's a Segmentation fault right there, or when Valgrind will tell me I missed a free ().
But, seriously, I do NOT miss C/C++.
Came here to say that. But jokes aside, as an APLer I wouldn't be able to bear waiting for things to compile. I regularly edit functions while they're on the stack, then have them continue…
But at some point you are better thinking about what to write, how to write it, which structure you will use. And this skill has to be acquired as well.
I even see the mindset spreading to other activities, like writing an abstract, paper, proposal or preparing slides:
I regularly (more often than not) see students using their supervisors as "CI-chain", submitting iterations with lots of stupid errors, half-baked sentences, unclean structure... because they learned to hit CTRL-b and see what comes back.
Of course there are reasons to talk about half-baked ideas, but if your task is writing a proposal or a section, you should at least be happy with what you wrote before you send it to the Prof, you're wasting his time.
But I also want to say that CI, unit tests and quick iterations are still helpful, even if you know how to approach a problem (i.e. think, research, draft, test, revise, etc.)
Experienced developers typically don't sit like Hari Seldon predicting branches 50k instructions into the future with some extremely elaborate grand master plan, they're so fluent in the code and architecture that they don't need to. This is tacit knowledge that is acquired along the way, I don't think it can be meaningfully practiced.
Maybe you are saying that there is no deliberate method of practice, but I think there is: whenever you correct one of your mistakes, try to figure out how you might have written the correct code the first time. Often, it might be a boneheaded mistake, but a significant number of cases have something to learn from. Can you make an argument for the current version being correct, and the previous one not? If an API did not work the way you expected, did you overlook something in the documentation? (Maybe just that the documentation was ambiguous, and you might have realized that earlier.)
I think OP's point is that slow turnaround encourages that sort of thinking, and it does, though it is quite a cost to pay.
It's not that the knowledge can't be acquired, but this is a type of knowledge often can't be explicitly practiced through some training routine.
A beginner may of course learn to check off common errors they've been told about, and may arduously step through the code in their head line by line, but an expert sees the code is wrong unconsciously and understands what the code does without stepping through it. The only way to learn that is to engage with code for decades. There are no shortcuts.
There are no shortcuts, AFAIK (though I think most people can achieve most of their potential in less than a decade), and I am sure you cannot get there through rote learning. Are you suggesting, however, that the sort of approach I am advocating is not very helpful, or just that it cannot be explicitly practiced?
This made my day, thank you.
Or lisp. Or smalltalk. Or forth. There's an entire subfield of computer science, headed by some prestigious people, who believe that working in a REPL is better precisely for these reasons.
At this point I deploy the left-handed user argument: different developers probably find these tools and workflows have drastically different effects on their productivity.
Perhaps also a "code kata" thing. It may be worthwhile to make beginners do each for a few months just so they have the experience (and find out their own "handedness")
On the other hand, at least it's a motivator to look at the build pipeline every once in a while and see if there's anything that can be improved.
That said, working in a typed language with good IDE support definitely helps build up the confidence that things will work as intended before you hit compile.
Sure, with stick you become aware of how the car works, the engine, torque and clutch, and you get more control. This might be good for learning.
But automatic lets you pay more attention to the road ahead, whether you are a new or experienced driver.
https://www.researchgate.net/figure/A-comparison-of-accident...
From the paper:
> whether looked at by human factors leading to accidents, by driver age or by accident type and age, the results all support the conclusion that, among all accidents, the accident rate for AT vehicles is twice as high as for MT vehicles except for head-on collisions
Contrast to manual drivers, who will happily drive along without music. Far less prone to distractions and far less likely to touch the phone.
One could argue that the focused people who enjoy driving do prefer a manual transmission, to which I'll agree, but I'll counter that the manual transmission itself fosters that focus and enjoyment in the driving experience. The boring automatic is what invites distraction and does not foster enjoyment in the driving experience.
I know a bunch of people that prefer manual transmissions. And every single one of them also prefers loud music while they're driving.
As a more general thing: you can learn a lot by removing tools that you usually use from your toolbox. For example, going from a managed language to manual memory management. Going from an IDE to shell commands. Removing tools like language servers. Each time I did one of those things, I learned a lot. However, there are lots of things to learn and I don't know if this was the "most effective use of my time".
Another example of hard to balance learning: I'm currently working for a company. Should I focus my learning on their business and our codebase, or on general technology? General technology is a transferrable skill, and I'm scared of being "trapped" in a company if I focus on their specific parts too much. But on the other hand, companies want to recruit efficient people, and I think focusing on the business and our codebase would make me more efficient.
I'm a junior engineer, so maybe that's due to a lack of experience. If anyone has insights to share about that, I'd love to hear them.
While you might learn more if forced by the constraint, the opportunity cost is quite high, and these days you'll never replicate the experience of going off with your glass plates and camera packed on a mule to photograph the Sierras.
Not only all of that... but Ansel Adams spent a TON of time in the darkroom, and pretty much invented most of the techniques that Photoshop emulated in their early versions. Even faced with limited chances at exposing a negative, he iterated the heck out of the development process when making prints.
I share your view that there is value in the focus, but I don't think that anyone can afford the opportunity costs.
Once you're ramped up, yes, it's more efficient to produce more code and fix the odd issue that crops up. I rely more heavily on high level, slower running functional tests on code where I've been the primary author or have worked on the system for a long time. But just be aware of what you're optimizing for: tenured employees.
I wouldn't kid yourself that it's "deep reasoning", beyond people on a dopamine kick, or beginning programmers working with simple data structures. You can't reason much about a system you don't know, because everything is details - millions of lines of code, you can't mentally evaluate it. It's not until you've built a mental model of the moving parts, and are able to mentally abstract away details not relevant to your task, that you can think things through in the large.
I think just as you get more experience you tend to write more code before you run it. I've written an entire (relatively small) application and written the tests for it before I even thought about running it. Run it, few tests fail, iterate a tiny bit, done.
It's just natural as you get more experienced that you can reason about code in your head without resorting to running it to confirm it.
A high iteration time [a.k.a long feedback loop] leads to distraction more frequently than to “deep reasoning”. Human brains seem to be wired to find distraction after about eight seconds have elapsed. [1]
Furthermore, let’s not forget that an IDE doing a lot of error checking real-time, often including partial compilation and execution, so nobody running an IDE is coding for a long period without compiling, they’re more likely coding for a long period without integrating.
If we reframe deep reasoning as “stay engaged with the problem while waiting for some feedback”, we can see that there are a lot of tricks for this: Unit tests, for example, or using a different medium like pencil and paper or a whiteboard. (I’m sure this group can come up with a great list.)
[1] “Response times: The three important limits” https://www.nngroup.com/articles/response-times-3-important-...
This was a thing learning to type, for me. Our class would transcribe from paper to document. Those that watched the screen were slower, once folks learned to touch type.
That is, learning to get into deep thought is a skill that eventually doesn't want the distractions. You want to focus and produce, then review.
My two comments however would be that when you don't know your context, lots of probing is required, and that can take a lot of iterating. In industry work you're very often working on a small part of a larger software orchestration, and it's often entirely new to you when you're asked to work on it. So you don't get access or time to understand the full scope of the software to reason about. So you need to "discover" your context as it actually is at run time. This is especially true when bugsquashing.
My second comment, is that we should all have a good debugger. Having a debugger is like having a superpower. I can come into a codebase I've never encountered before, step through as it runs, gain enormously useful context, and solve the problem I wanted to solve, without ever running the code to completion or having to read every bit of logic.
If I'm working on someone else's codebase, usually I spend time reading to figure out the general architecture, then dive right in with a debugger and start stepping through. I find how to trigger the code paths related to my task, see what's happening as it happens, and get a really good understanding of how it's all running.
In what might be another controversial opinion, a working programmer without a debugger is like a working carpenter who's chosen to do everything with a knife. They may produce good work with their own way, but they could be faster with the proper tools.
But definitely having a phase where you are dealing with long build times helps you enhance one aspect of software development!
At least for small delays, say 30s-3m, I'm not tempted to go for that (other) dopamine shot of checking messages or whatever at all. I keep thinking about the problem I'm solving, what to try if the thing I'm currently trying doesn't work, and so on. I don't remember ever feeling limited with such cycles - literally only when fiddling around with CSS, which I quite often just do in the browser's dev tools for the fastest possible feedback cycles.
For me slow feedback loops push me into dopamine seeking activities, intentional breaks push me into relaxation and deep thought
Recently I got a treadmill desk and the physical exertion reminds me to take breaks where I can get off and relax and potentially think about a problem
Slow tests running causes me to not write tests and look at YouTube
https://youtu.be/f84n5oFoZBc
There's also a "sketching" element with faster iteration. On slower iterations, you're engineering. The engineering skill is often underdeveloped but sketching is more valuable without a dedicated designer.
Also some things are poorly documented, often things like UI and browser code or legacy support for say, older Android OS. So even if you have your calculations right it ends up wrong.
My first programming experiences were BASIC on TRS-80, TI-99/4a, and similar machines. It was instant feedback, for the most part, and I enjoyed being able to make small changes and seeing the results. I was around 12 years old and that worked well with my attention span at that time.
Agree.
I think what was happening was the the lack of feedback fit the mental models of the people who were attracted to computers early.
For another example of this,Ken Thompson famously said that he never quite understood why anyone would want to see the whole file they were editing instead of the single line.
But instant feedback attracted more people to programing.
I'm one if the people that thinks that coding in paper is a great learning task. I tend to not ask people to do that in practice because it's a very effective way to lose control of a classroom, but I do use the second best task that is asking people to explain their code.
Anyway, if thinking about the code is not your focus, the increased feedback cycle is all cost and no gain.
Let me tell you, time to iterate on a huge 50 year old code base is through the roof and reaching out into interstellar space. The resulting code is pure and utter trash.
I would kill to have to have a quick feedback loop and at least be able to address quality issues through training, culture, or technical controls.
The alternative is fighting against a giant, heavy machine with a lot of inertia in the wrong direction.
This only works once you're a bit more advanced, but the one sure sign of a senior dev is the accuracy of their mental model. Which is also what allows you to write your piece without a continual back and forth with the compiler.
This all said, I greatly enjoy ping pong with a repl or instant hot reloading when doing visuals or sound.
However, the experience itself feels like a positive one - even if the code takes longer to craft, it's much easier to work with, since now i don't need to worry about sloppily written if/else chains that depend on enums or runtime type checks, but can utilize different implementations as needed. Also, writing the actual tests allowed me to be sure about how everything would actually work, all the way up to discovering that Paths.get (Java) on Linux accepts almost anything as a valid path string, whereas Windows has actual validation rules, leading to many headaches in regards to the code coverage quality gates that i set up, since the coverage differs based on which platform you run on.
But i guess that my point still stands: in certain circumstances slowing down might actually be a good thing, when you really want to know how your code will work under most circumstances and make it maintainable.
Of course, on the other hand, compilation speeds and other factors in regards to the speed of iteration definitely shouldn't be overlooked either, and having everything else apart from writing tests and actually thinking about how everything will fit together be faster is a good thing! I'm not sure what i'd do if my test suite took 10 minutes to run instead of 10 seconds as it currently does. Probably drink lots of coffee.
I guess it depends on slowing down for good reasons, vs just wasting time because of tooling or other sub optimal circumstances (e.g. what was described in the article, personally i've also seen a local API service be pretty chatty with a remote DB which was slow over a VPN, yet no one had a local DB with all of the migrations in place, and a bunch of other things like that).
Offtopic: Anyone remember how fast Pascal compiled? Now that was a really nice stack to do some stuff in, it's a shame that it never got as popular as Java, or didn't have tooling like JetBrains has, it felt like a more ancient Go (which is also pretty good as far as ergonomics go).
> Recently decided on using TDD for some library code and aiming for >90% test coverage at my dayjob.
I thought the point of TDD was that you don't write code unless a test requires it? In my mind (and the little experience I have) that translates to 100% coverage, with exceptions of really annoying side-effects that are near-impossible to test reliably.
Yep, like Java supposedly being cross platform, but having different file system implementations, because using one implementation across multiple file systems just isn't entirely feasible, for example, due to how paths are validated.
Thus, no matter what i did, i couldn't get the coverage to 100% due to semantics of how i'm supposed to validate that the low level call works and because i have branches in the test, one that checks whether the correct exception is thrown in Windows and a different exception for Linux. Even if technically the code paths are covered, telling that to a subpar code coverage plugin is hard.
So, i just settled on writing code for everything myself, but having 90% coverage be enforced on a class line level to account for the weirdness.
And while this strategy might work for the folks who actually wrote the code (and thus have tacit knowledge about it), the moment they leave and/or someone new joins the team, all that "speed advantage" is lost and it turns from minutes to hours to days instantly.
Automated tests really are the only way to capture the business logic of any code for it to not to become "legacy code" before its time.
Does that code counts as legacy?
Sure it is a bit verbose but dam is it easy to understand due to sheer bluntness and lack of magic.
"legacy code is the product of ... when your team turns over faster than your code turns over." [1]
1. https://www.software-engineering-unlocked.com/legacy-code-mi...
Actually, currently am about to be asked when the feature will be done, whereas i've decided to actually write tests for it (reusable library code across multiple projects). As far as the reality is concerned, the people who say that writing tests "uses up" time are right: being slow now will be more visible than no one understanding why development takes a whole lot of time down the line, which is when many might just shrug their shoulders and go "legacy code". What's the benefit of making others able to develop code faster, if that makes you be slower - both facts probably being visible to management, though oftentimes without the underlying reasons behind those being relevant to them?
I'd argue that that's why even governmental projects in my country (that are often contracted out to random companies) oftentimes have poor test coverage and just generally not a lot of thought is put into the quality and sustainability of those codebases - they got paid, they could iterate reasonably quickly while the contract was ongoing, why should they worry about anything? The people who take over the project then can also deliver features more slowly, not only because of the poor domain understanding, lack of ADRs, lack of documentation, but also lack of tests - and still often get paid on a time material basis. Thus, no one actually wants to improve things, apart from me wanting to tear my hair out whenever i'm expected to work on garbage like that and fix their problems.
Thus, you see some people in the industry adopt an egoistical approach to it all - join a company, focus on the speed of their own iteration without thinking about the project in the long term, spend a few years doing this and then leave for another company where they'd do the same thing. It might be a cultural thing, but working on other people's projects instead of starting my own feels like losing at this point - no READMEs, no automated CI, no Ansible, no Dockerfiles, no IDE run profiles, no common linting rules, no code static analysis, no local DBs with migrations, no local setup scripts etc. Why should i be expected to toil away without results that are visible to the business because someone else was allowed to neglect the codebase?
At this point, i use unit tests and code coverage rules defensively: to prevent someone from jumping into the project and ruining how things should work with breaking changes, which will be caught by the tests, or by introducing untested and undocumented code, which the coverage rules will catch and make CI fail. Of course, depending on the culture, you might find that other devs talk amongst themselves and before long your tests have @Ignore added to them or have been removed entirely, which is the point where it might just be easier to look for a less dysfunctional environment, despite living in a country that others outsource to and therefore quality isn't a priority.
In short: i agree with your point, but i think that there are social problems at play. You need to actually care about the code that you're developing to prevent these issues in the long term. And even then, if you care, that doesn't mean that other team members will.
From my limited experience, governmental projects tend to have the worst code quality ever. At least that's what I see here. It can easily be explained by pervasive corruption — projects go to those who agree to return the largest kickback and are written mostly to steal money from the taxpayers. With such incentives, code quality does not receive much attention.
You hit the nail on its head here, my friend.
When working with others' projects, I try to add tests where I've added/modified features, so that at least I can get a sense of it all and make sure I don't break anything. And for a completely untested codebase, it might make sense to add more integration-like tests (while still using mocks for external dependencies), such that simultaneously a bigger proportion of the codebase is covered, even if at slightly longer test times.
…I just googled for “React TDD” and the top no-money result suggests finding divs in containers and spying on dom changes. It’s like testing your cli app by hooking into tty driver’s section in /dev/mem.
And when it comes to testing what your DOM looks like, even though that does often involve spying for changes (after all, you're testing whether they occur), that's not on the actual DOM but on the virtual DOM, which makes it easy and efficient to run in a non-browser environment.
You have a “data” as a model sitting between the DOM and the controller (mainly methods and watchers). There are some gray lines, like computed properties, but otherwise the controller can run without the DOM and vice versa, you can easily force the model into a state to see how the view will look like.
A lot more testable than the old spaghetti of onClick and getElementById().addEventListener.
The newer tools in web dev have crushed iteration times to a fraction of what they used to be. A combination of things like esbuild, fast refresh, yarn's offline cache, and a few other bits can get your iteration times on a site you're manually testing down to milliseconds to both build and update (literally). Decent devs have been writing tests for years; the test runners could be a bit faster but people are working on that as well.
The React app I'm working on at the moment takes well under a second to build, under 10 seconds to run the test suite (could be a sign I need more coverage..), and milliseconds to update the code in the browser in dev mode. It's nice to work with. It's not even clever or special. A default Next.js app will do that. Create-React-app 5 launched a few days ago with more of the tooling too. In Vue things like Vite are based on the new fast tooling and it works well. Vue dev work is fast. SvelteKit even goes further by using Snowpack to remove the bundling step entirely. Snowpack claims a default refresh time of 50ms.
I really hope the author's article about web dev is simply "Yeah, update your tools and you'll be fine."
FIY SvelteKit is using Vite 2 instead of Snowpack
Source: https://twitter.com/Rich_Harris/status/1367577006355976194
In this blog post the guy took 3 job change to DISCOVER unit tests, it's insane :( It's not like they know but don't have time, it's that he's a "champion" just for proposing limited scope testing.
It's hard to do unit tests in a large part of the area of gamedev. How do you ensure that a picture is rendered correctly? You can take a screenshot and compare it to a stored one. But then what happens if you do artistic changes that you want to do? You have to update the stored screenshots of the testsuite. Who will review that the changes all made sense? And more importantly, there might be slight differences in the output of GPUs, depending on driver versions, model, etc.
So let's say you have a bug in your game where if you walk through a level in a specific direction, the game crashes. The error is easy to check for: just make sure that there is no crash. But how do you create a reproduction of the bug? You could record controller inputs and play them back. Then a different department changes something how quickly players move and increase their walking speed by 10%. Suddenly your player walks into a wall and the test is basically broken.
Compare this to a CRUD app where you have well defined operations and their impact is well described.
That being said, even in gamedev there are areas that are well testable. You can do a unit test of the low level networking layer by trying to make a server and a client, dropping some packets, then looking if the packets still arrived because the networking layer sent them again.
(better yet to have someone regularly scanning for these efficiency related things)