The idea of prototyping to uncover 'unknown unknowns' resonates with me, it's like a reconnaissance mission before the actual project. This could indeed save a lot of time and effort in the long run...
In my experience, this is what actually happens. A developer makes a low-quality, low-effort prototype under the assumption it won’t ship. Someone sees it. It gets shipped. Everyone loses.
Sneaky move.... but then, in the eyes of some people, you might well appear not competent to put together a working system, so make sure you know your audience.
It helps if you have tests because when prototyping, you can write things in a way which break tests and ideally break other features. It works enough that you can validate the one idea. That way it will never pass CI.
Another war analogy was the Byzantine defense. Their policy was to avoid decisive, large-scale battles where a lot can go wrong at once. If an enemy army is incurring, first try to weaken or deter it with lighter forces that also gather intel. If they can't stop the threat, form and advance a larger army with a better-planned supply chain to stop it deeper in the territory.
Which actually doesn't work so well in AoE. The game doesn't have a concept of army supplies, so you usually just want a huge unstoppable army you can send anywhere.
Yet I find when rewriting from scratch there's a countering force of simplification where I find myself thinking "We spent way too much complexity on that part, and we can entirely do away with that other part, and we turned out never to use features X and Y."
> "...and we turned out never to use features X and Y."
THIS. Often, the best way to start the rewrite is to go straight to the database, and start asking "what allowed values for all these fields have never actually been used?".
EDIT: Whether it's rewriting, or refactoring, or adding some "if( Deprecated_Feature_Allow !== true ) ... ErrorDie( Feature_Deprecated )" logic here and there, or updating training material, or whatever - knowing that Software_Feature got ~zero real-world use is d*mned useful.
Justify wasteful things you've done and suggest them to others.
Less cheekily -- it's true that sometimes the best way to research a topic is by writing code that you might appropriately throw away later. And it is extremely important to be able to recognize when something is not worth keeping. But also, often, the most efficient strategy is to think about the problem off-screen enough that you don't subsequently waste time on-screen writing things you won't keep.
It seems strange to usually expect to throw away your first draft. Not every problem is that hard, and even for hard problems, not every first attempt is wrong.
The temptation to throw away all of your code - be it a prototype or a "grown" code base - arises often.
Often it is a bad idea. I get it, though, there is an inherent attractiveness in the idea of starting fresh from a clean slate. Except, more likely than not, you'll soon find yourself in a similar situation to the one you started from. The fundamental problem is that it is easy to underestimate the edge cases. Sure, the main functionality is easily understood and straight-forward to conceptualize. But did you remember to think through all the smaller aspects of your software, too?
Perhaps it's easier with a prototype implementation that in fact doesn't have a lot of features yet, but to completely replicate the functionality of a complex piece of software isn't an easy undertaking. Sure, getting 80% there is probably easy, but the last 20% is the part that's easy to overlook when considering a complete rewrite.
Admittedly, there's nothing sexy about refactoring. And often it may seem like it's less work to just simply scrap everything and start over. However, that's fallacy a lot of times.
Ok, perhaps don't throw it away. When I come up with an idea (for an app usually), I use "a few" A4-sheets (anything between 5 and 20), scribble and draw on them with a pencil, draw screens, buttons, data flows, activities, write notes (in various font sizes). Then I use my CamScanner and call this a v1.
Then email me the PDF and store the papers in a box, and a couple of days later I start the v2 (same process), then v3.
By v4 it's 'good'.
I also use the same method on my 9-5.
I take VERY seriously the Abraham Lincoln quote “Give me six hours to chop down a tree and I will spend the first four sharpening the axe” on almost everything.
I consider the v1, v2, v3 as the "sharpening the axe" and the v4 on the actual cutting.
In that spirit, the article has a similar approach, as the v1, v2, and v3 may take you down an (more than) imperfect path.
> The temptation to throw away all of your code - be it a prototype or a "grown" code base - arises often.
I want an editor plugin which allows me to mark sections of code as reviewed or "perfect" (depending on how honest I'm being). Then, when I'm tempted to rewrite everything, I can go through and mark what I think is good, and then focus on refactoring the rest until I think it is good as well.
I'm tempted to rewrite code because I lose track of what it's doing, or I've learned a lot since I wrote that old code and so I'm not sure if the old code is good anymore. It's not so much about rewriting the code as an exercise in getting familiar with the code I've already written. I want a tool to help me with this.
I would think a combination of git, unit tests, and comments would solve this problem?
Unit tests prove the code works as intended, and are basically examples of what the code is doing. Whether the code is actually "good" is a bit more subjective -- but tests give you the freedom to modify it without breaking it.
Checking into git frequently is also a way to give yourself some freedom. Commit at every milestone, like every time the next thing is "working". If you feel like refactoring, go for it -- you can reset back to working state in a few seconds.
And lastly, leave comments in. You can always clean it up before you push. You can even squash or interactively rebase your history so no one else sees the gory details how the hot dog was actually made.
The theory and the practice of unit tests are often different things. In theory you’ll make changes to the code, and the unit tests will show whether they had the intended effects.
In practice, because often the unit tests are mocking out all the classes they interact with (as is the recommended style for isolated tests) you’ll either have all the tests explode constantly because the interface being mocked has changed, or you’ll have tests that confidently tell you things are working despite the methods they call no longer existing.
If you could modularize code in such a way the sizable chunks could “just sit still!” then entire modules could be marked as perfect. But you wouldn’t have to since you wouldn’t naturally need to touch them (e.g. how many codebases are using a fork of core-utils, for example, … to exaggerate a conways law effect).
But unfortunately the modules don’t make themselves apparent at the start of the project. So it needs refactoring discipline.
Throwing away an established code base is a completely different thing from throwing away a few-days-old prototype, and not what the article is suggesting. The points you mention don't apply to the prototype case.
I feel this. What I do now is increase the points of the PoC (real estimated PoC time + optimization).
I get the PoC down asap then as soon as it’s proven (privately to myself) I move onto optimizing. If it fails then I communicate early to discuss a change of scope.
Assuming it succeeds, by the time I present it I have already done significant work making it production ready. So to request an extra 10-25% time to “smooth out the edges” goes over easy after the demo.
In reality I end up with 50-75% time to get it prod ready. The rest is QA + prod bug fixes to take it over the (practically scoped) finish line.
This is such a HN phenomenon it infuriates me. Read article. Then read comment that misconstrues article to be something completely different, accompanied with loud critique. Read replies all in violent agreement to obviously self-evident strawman.
The misunderstanding you have is that often the article isn't the point. Just like a football match in a pub isn't always the point. People just want to chime in - the comment section is like an ephemeral pub.
If that bugs you because you consume information on the web rather than socialize, the solution is simple: don't read comment sections and make up your own mind about the article.
> Admittedly, there's nothing sexy about refactoring.
There's the "strangler fig" (my favorite) method of refactoring. It's where you rewrite just a small portion of the software, little bits at a time. Instead of doing a full rewrite.
IMHO, it's the best way to refactor. You can switch out both "old" and "new" versions at-will until you're 100% sure you've covered all the edge cases.
I don't really feel that attraction to complete rewriting. I wonder if the people who do are very smart, and thus able to hold more state in their head, so ugly code bothers them more even if it's not actively a problem, because they are able to have background tasks in their mind to worry about it?
And at the same time, perhaps their code is less encapsulated, because they didn't optimize for abstraction, they optimized for beauty. A leaky abstraction doesn't bother them, because ALL abstractions are leaky to them, they probably have a sense of internal workings even whem using household appliances, but ugly code tucked away somewhere bothers them a lot, and they might dislike using popular large libraries even if they work great, just because they're not comfortable using what they don't understand deeply.
My evidence of this is the fact that suckless exists and people actually use it, I assume their experience of thought is very different from anything I have experienced.
Speaking personally my urge to rewrite at least partially comes from not truly understanding the problem and the solutions to it until I've written something reasonably functional. It doesn't matter how much time I put into sitting and theorizing, there's always things I didn't anticipate and assumptions that turned out to be incorrect.
This usually means that rewrites are significant improvements across the board, especially if they're done a relatively short time after the original is finished since it's all still fresh in my head.
This may be a weakness of sorts on my part though, I lack formal engineering training which might be why purely mental modeling (no code) doesn't work all that well for me.
Yeah, I find that the desire to rewrite often comes from the final (working) solution having poor code ergonomics. It's not necessarily that it's wrong, it's that it's awkward or clumsy to understand/use, because my mental model of how it worked didn't take into account the actual practical day to day usage of the code.
I get around that by writing the API first, with usage code examples and all, Or in a GUI app, the UI first. I'll iterate there if needed, and only rewrite internals if it's truly bad or I think it will cause a problem later.
A lot of situations where the code is so nasty I actually have wanted to rewrite... I wind up just throwing it away instead.
Like, one of my happiest days in coding was throwing out some code to support JACK and going all in on Pipewire. Another happy day was replacing some DIY code with a GPL licensed library, deciding that I really didn't need the option to do proprietary stuff anywhere near as much as I wanted less thinking about gstreamer.
"If it's hard to explain, it's probably a bad idea".... If I try a bunch of API variations and I can't come up with anything that doesn't require learning 3 new algorithms to use... Maybe I'm not making a library or an abstraction, I'm making a proof of work hashing scheme that users have to do manually to access an encrypted version of the complexity I'm hiding, and I need to stop before I make z80 assembly in JSON to autogenerate Vue templates that render to cobol.
Sadly, I think this is true of me — at least of me. I don't think smart people (by this definition) are that common though, so clean code is a sensible default.
Humans always make mistakes and are tired now and then, and the smartest people can still appreciate not needing to solve puzzles just to understand or debug their own code.
Of course clean and readable means very different things to different people, but I don't thing I ever regretted cleaning up code, while I can think of a lot of instances where not doing it wasted a lot of time.
There is some point regarding changes on the code where rewriting is less costly or more costly than changing the current code. For me, a big part depends on whether I depend long term or short term on it and how deep I will have to go anyway using it.
ALL abstractions are leaky - this is an objectively true statement.
As others said it's not for beauty, it to make sure if there is abstraction it fits the problem. If there is encapsulation it doesn't get in the way. For some coders they can get it on the first try and there is no reason for them to rewrite code. For the rest of us mid coders we need to explore first as well as make sure all cases we desire for Lib/API work.
> Admittedly, there's nothing sexy about refactoring. And often it may seem like it's less work to just simply scrap everything and start over. However, that's fallacy a lot of times.
Agreed. Though sometimes you need to refactor the core of an application, in a way which will touch the entire app. To do that I often make a new, empty project. Then I rewrite the core of my project into the new folder (in whatever new structure I’m trying out). When that works, I slowly copy the content from the old project into the new project - refactoring and testing along the way.
But it’s not perfect. About half the time I do this, I discover halfway through that I didn’t understand some aspect of the system. My new design is wrong or useless and I throw away all the new code. Or I figure out that I can just make the changes in place in the old project folder after all, and I bail on the new code and go back to a traditional refactor.
But no matter what happens, I don’t think I’ve ever regretted a refactoring attempt like this. I always come away feeling like I’ve learned something. How much you’ve learned from a project is measured by the number of lines of code you threw away in the process.
Rather than "throw away" I would "start from scratch" meaning I can refer to prototype but reimplement with respect to appropriate norms.
As a version control fanatic, very little gets really thrown away in my book. Just hidden.
Yeah normally when I do this, it’s because I picked bad abstractions. The main business logic I can bring over almost line by line, but I want to reshape the abstractions / data structures / interface to that logic
Game companies implicitly do this, throwing out old versions of their code (sort of). You make a game. Ship it. Start on a new game. Your previous game is now sort of a practice run for making a new game from scratch. Continue, ad infinitum.
It's weird: you rarely have to maintain something for 20+ years, and you get to always improve and iterate on how you did things last time. But, are you training yourself to write hard to maintain code, since you don't really have to maintain it past a certain period? Or does the learning-from-iteration actually make writing-maintainable-code easier?
I know some people do keep developing their games for decades, look at Dwarf Fortress, I'm just talking in general.
what you do you get good at. if you don't have to maintain there is no point in getting good at it or any reason to think you will gradually get better at it.
Nah. I think a prototype is something you definitionally should throw away. There's a huge freedom to knowing that instead of taking on tech debt, you're committed to declaring tech bankruptcy and throwing away the code. It lets you try things, cut corners, and generally experiment. It's great for thinking through what everybody really wants.
I do agree that it's generally a bad idea to just throw out a long-lived code base. But for me that's not an estimation issue. It's because the urge to throw it out is usually a response to upstream problems that haven't gotten fixed. For example, a lot of code bases are a mess due to time pressure. But if you don't fix the process problems that turn time pressure into mess, then you're just doing to end up with another mess. Possibly a bigger one, in that stopping all productive output in favor of a rewrite usually makes business stakeholders crazy, causing increased time pressure.
Granted Joel is taking about a wholesale rewrite of proven / established code. Code and has had had the advantage of time.
Personally, I end up rewriting parts of my code often. It usually takes a mile for me to find exactly the right way it should be based on how are people use it.
You absolutely should rewrite prototypes, and re-factor important chunks of code. I rewrote something 3 times today, it was better each time.
Of the reasons he says engineers want to rewrite the code, he says: badly organized code, slow processing time, and ugly code.
Maybe I’m not an engineer, because I bet perfectly pretty fast code is an indicator you should reorganize it to make it easier to adapt, add more capabilities even if it runs slow, and put some ugly fixes in there so it’s useful for the end user.
Agreed that wanting to over-engineer and perfect something that should be constantly evolving is not a good reason to rewrite it. That sounds like back office bubble pretend-work.
A possible alternative is to use a different programming language for doing the prototyping than the one which will be used in production. The urge to hang on to prototype code can be removed, because you are going to do a rewrite in a different language, regardless.
By using 2 different languages, there is more freedom to just make a demo, discuss, and then decide what to keep or what direction to go.
I've felt this, and I agree. It really pays of to do some work on technical discovery, make all the mistakes, see where all the design issues will be, validate as much as possible, and then start again. And the best way to really scope out requirements and get to know the "unknowns" is really to get your hands dirty.
Especially when in comes to business logic and domain modelling, some bad decisions at the begining can cost a LOT, to undo in an iterative matter.
(To clarify I'm talking about the timeline the author proposes of "a couple of days" build the throw away prototype)
Throw away 1st, second, third, etc... Write your code in small replaceable/disposable POCs until at some point the architecture and coherence starts to take shape.
That's the advantage of software over other work. You have 0 material sunk costs. Unlike building a bridge, which you can't just tear down and iterate, with software you can develop a workflow where you break things and iterate quickly.
There are of course software systems with a very large bit of complexity and entrenchment... least of which the complexity added in order to avoid said entrenchment in the first place.
Exactly what I thought after GPT3.5 was released.
"Delete it all. Start from scratch.
You'll learn so much. You'll do so much better.
You'll actually live up to the scale of your creations' potential."
But a decision like that would be superhuman by itself,
a transcendent experience, giving the creators a perspective that
seems impossible. Unlike what is happening instead, both publicly
and behind closed doors... A great pity!
I've never had luck with this approach. For me, iteratively building and changing the codebase as new requirements come up lends itself to having to think of good ways to modularize and future proof. The idea of throwing out the code and starting over is definitely appealing, but often the iterative approach gives me something I'm super happy with.
I think I do use the throwaway approach with simple prototyping of smaller portions of code that I will then port into an existing system, but the overall system remains stable.
If you're discovering requirements along with developing code you often end up with Frankenstein's monster because things that were intended to do one thing got retrofitted to do 4 more things. It works but if you had all 4 things in mind when designing the first approach it would have been much cleaner. And going forward it keeps getting worse.
I like to build my first version in one file with as few abstractions as possible. No helper functions, unless absolutely necessary, just straight-forward code. Building good abstractions later on is a lot easier than starting with wrong ones in the first place.
Very interesting approach. As someone who has overengineered last major project I worked on......maybe this is the way. I got so fixated on getting it "right" using current standards and abstractions that it became a bit of a mess. I despise working with that code base now.
This is an underrated way of doing things that I've come to appreciate over time. Heck, I'll even go live with the all-in-one-file thing until there's a reason to split it up. It's all peanuts anyway compared to bigger decisions like what API, DBMS, DB schema, other deps, libs, etc you use.
I've always done it like this as well. Initially I did it for the most obvious reason: it gave me the working software that I needed the fastest. Then later on, I always found it to be the best way to find natural abstractions as you go.
I’m in the middle of this with my new database startup. The alpha was written to gather requirements, so I intentionally skimped on code quality as it could be a waste of time. Now that I know the domain inside and out, the beta rewrite is worth turning up the quality. And it’s so fast and easy to write, as the old code serves as api documentation for all the low level stuff it depends on.
I've never respected this advice, even though I hear it often enough. The reasoning appears to come down to this:
> If you know that you're possibly keeping the code, you do things in a "proper" way, which means moving slower
combined with the idea that it's faster to redo work than it is to refactor.
I don't buy either of these propositions as a universal rule. For the first, it seems that a mindset of avoiding premature optimization, which you should be cultivating anyway, is enough to ward it off.
For the second, I see it as very context dependent, with my bias going towards refactoring instead of rewriting. I most often get things right enough the first time that it really only needs to be refactored going forward. Only when I end up with serious flaws do I decide to start over, often before I finish my implementation because the flaws are already apparent enough. Add to this that even very large changes can be refactored effectively and timely in my experience, and I feel like the bar for rewriting is really high to be cost-effective.
Unfortunately this essay and many others seem to take objections like mine for granted, and I should just believe the author that rewriting is faster with zero supporting evidence.
looking for gaping security holes is premature optimization. if you run into security problems down the line, use a security profiler and find the hot spots i.e. radioactive code
I think this depends greatly on the product area and ease of updating. Removing problems categorically from designs early has been valuable to me in the past to reduce surface area, avoid impossible refactoring, and eliminate systemic bug classes.
> looking for gaping security holes is premature optimization.
I won't approve a PR if I notice such a thing (and if it really is "gaping" I'd expect an automated tool to flag it and fail the build anyway).
I don't really see ensuring your code follows basic secure SDLC best practices as "optimisation" at all - in fact it often comes at a (hopefully small) cost of less than optimal performance.
Of course there may be areas of software development (hobby projects etc.) where security is not a priority at all, but if it's code you're being paid to write that runs on the cloud or your customer's machines then security is absolutely worth getting right from the get-go.
Usually I think it is unnecessary, but you might want to include the “/s” marker in this case. It is just close enough to believable that someone would think this, given the amount of insecure trash products that have been released…
> I most often get things right enough the first time that it really only needs to be refactored going forward.
I think it really depends on your values and what you’re working on.
If you have the mindset of a product engineer, the thing you care the most about is how well the software works for your users. The code is an unfortunate necessity. When I build react apps, I tend to think like this. The code I write first go is often good enough to last until the next redesign of the UI.
By contrast, if you’re thinking of the code as a home you build for your work that you will live in, then having a tidy home becomes its own reward. Good decisions today (in terms of cleaning up or refactoring code) should lead to more velocity later. When I think about building a database, an OS kernel or a compiler, I think like this. When I start, I don’t know where my code should be rigid and where it should be flexible. My first guesses are usually very wrong.
Personally I prefer the latter kind of problem. I like it when there’s no obvious way to structure my code and I have to explore while I work. The code I’m most proud of writing has probably been rewritten 5 or more times before I landed on the “final” design. The “right” abstractions are rarely obvious at first glance and the search can be incredibly rewarding.
The rich history of application programming abstractions suggests it isn’t exempt from this either. It’s just, when you’re building a UI there have been an awful lot of explorers who came before you and can show you the way.
I agree that it depends on factors like what you describe which is why I am asking those giving the advice like this to consider the context at all. I also happen believe that the contextual bar should be placed high. I have to say, what comes to mind when I read your comment was "What about this does refactoring not accomplish?" In the metaphor of building a house, refactoring is what keeps it tidy and organized, and rewriting would only appear necessary in the case of your trash chute leading into a bathtub or some other critical design flaw that could only be addressed timely by redesigning the entire house.
If you are in a context where you are in danger of making critical design flaws often, or where circumstances make it hard to make changes after committing, then I absolutely agree that rewriting is effective. That's just what I mean by a high bar, one that I personally don't encounter very often in my work.
I'm largely with you on this... do the simplest thing that gets a job done, that you assume will be replaced, and don't be surprised when it's still un use a decade or more later.
I definitely disagree with you on the first point.
Setting aside the part that's just fussbudgetry, I think "proper" coding is about doing things that pay off in the long term even if they feel burdensome now. As an example, I'm a huge fan of good automated tests for long-lived code bases. But if I'm just writing a quick, throwaway script, then building a bunch of automated tests are going to slow me down.
Throwaway prototypes are another area where you can save a lot of effort if you have really committed to throwing the code away once you've learned what you set out to learn. As a physical analogy, before the Long Now built out their bar/cafe space in San Francisco [1], we spent a day in the raw space building furniture and walls out of cardboard and tape [2]. It let us get a real sense of how the designs would work in practice for actual use of the space.
In the past for a number of projects I've done throwaway prototypes and then started fresh with "proper" engineering, which for me usually includes pairing, test-driven development, and continuous deployment. I've also done ones where we think we understand it enough and just start building. And either way, we usually end up evolving ways to prototype on top of the existing platform via feature flags, etc, so we can learn something from the real world and then decide whether or not we want to refactor to accommodate a major change.
Using boxes is great! The beauty of coding is that those boxes don't need to be discarded, we have the virtual power to transmute them into real chairs and tables via the power of refactoring. It's like Photoshop vs. watercolor. I'm going to sketch either way, it's just a matter of what the bar is for throwing it all out, and I think it's very often sufficient to decide in the moment in a virtual environment. That's all to say, going back to my first point, I don't need to decide ahead of time whether the code will be kept, I'm going to keep it loose either way.
And there are situations I'd go in assuming I'll probably rewrite the work, but I wouldn't set the criteria so broadly as to cover literally whatever the next large project I embark on will be, as the article suggests.
If that works for you, great. But generally when I see people "keep it loose" on a prototype that then gets kept when some decision point is hit, they don't actually take the time to clean it up to production grade.
And personally, although I'm very good at refactoring, I will still go the path of a disposable prototype in the future when I think it's more effective in terms of total effort. Which for me is any time I expect the prototype to yield significant information about both the surface and the substance of what we're doing.
I've been following this flow where I write a few successive versions and perform refactoring steps to move one version "towards" another version, as if there is a conceptual force of attraction. It's often unclear which version will prevail, as if I'm acting out a genetic algorithm with a small population.
I've seen the theory that this is why changing to a new framework/language seems like magic so much of the time. That it's not so much the differences in the technologies as it is your knowledge of the problem.
It's probably a little of both in reality but it's an interesting idea.
I have found that this kind of advice does not play well with managers (but it's actually not a bad thing to do).
I'm fairly grateful that I've retired from the rodent rally, and am working at pretty much my own pace and structure.
About six months ago, I tossed out the code for the frontend of a project that I'd been developing for over a year and a half (at that point). I distilled the business logic into an SDK, brought in a designer, and started over from scratch.
The result is ten thousand times better than what we had (and were considering shipping), a few months ago.
I've found this great on my personal projects where I've had all the time I've wanted. The next iteration was always orders of magnitude better than the first. In the professional setting I find that the prototype is often enough and rewriting costs unjustified amounts of money.
I rewrote a mission critical application. The original app was something else. One function had 1500+ lines of code. It was the ugliest thing I have seen in my life. We wrote cleaner code in early college years. However, that code ran the business for 12 years with almost 0 modifications. People created work around for some stuff. Error handling was poor. But the goal was achieved and business continued to grow.
I think devs in general sweat too much about code quality.
> However, that code ran the business for 12 years with almost 0 modifications
We have code like that, and it requires everything around it to twist and contort to comply with the beast of unmaintainable at the center of it all, simply because at this point it has so poor test coverage and is so incredibly mission critical now.
It can still be a lack of effort put in trying to convince product managers etc that more time is needed!
But almost all PRs I review have something in them that suggests the developer not putting as much thought/care into what they were doing as I'd like to see (and I'm just as capable of doing the same).
Some disputes about quality are simply differences of opinion or style.
There's not really a book or standard on opinionated code quality that satisfies a majority of the people in the field. There's way too many ways to build things for that to exist.
This leaves it up to individual teams to approach a shared consensus (with some wiggle room) within their domain and tool set.
One thing that is universal is how strict the enforcement of a team's quality standards. So as a developer, you have to decide whether you like strict or lax enforcement of quality standards. Everyone can decide that but I think most don't.
Code quality isnt that important if you check this code "once a year" and dont have time pressure to change somewhere in there. Or if you are seeing this code daily and know all the quirks.
Anecdote: I know a guy who works solo and throws every project away (usually games) at least five times, often more like ten.
According to him, each iteration tends to go smoother and faster than the last.
Edit:
There’s probably some nuance to whether this approach is a good idea based on type of software, size of team, experience level, personal and/or team skill set, etc…
This advice depends a lot on how your "first draft" was coded to begin with. The concept of "first draft" shouldn't exist as code. You design it properly, then you implement it properly.
Now, if for some reason, and there are plenty of good ones, you cannot do the former, then sure, you pay that price by redoing some work. But, at that point, you do the same thing you should have done: you design it properly, then you implement your design.
One could argue that implementing a first draft is a way of designing it. Which, I cannot imagine making sense unless you don't know what it is you want to end up with. Maybe if you're making computer games or other creative endeavors?
> One could argue that implementing a first draft is a way of designing it. Which, I cannot imagine making sense unless you don't know what it is you want to end up with. Maybe if you're making computer games or other creative endeavors?
It's when the problem itself has some ambiguity. Like, I design systems that other people and systems use within a large company. I try to figure out how they'll use them, but even they can't tell me exactly. New business requirements come in mid-design, etc. They might even cancel the whole project if something more important comes along. In that situation, it's best to get some MVP as quickly as possible then iterate on it. Later iterations might totally scrap and rewrite the internals, which is fine if the API sticks.
That makes sense. Like a prototype you hand out so that you can figure out what it is you/client actually want. I don't think of those as first drafts though. Maybe it's a language barrier thing for me.
First draft could mean a lot of things. If you mean something you write then rewrite before it's ever released, the same still kinda applies. You might be mid implementation when something changes and ruins whatever structure you had. Or maybe the abstractions aren't very obvious at first even if nothing is changing.
"Design" has a spectrum of level of detail, anything from a whiteboard full of information to a description of code in a fancy tool that is more detailed than the code itself.
Text (source code) as a representation of ideas can be worked on extremely efficiently at arbitrary detail. At low level of detail only beaten by a whiteboard.
If any piece of information has been missed when creating anything (any form of design or code), updating mutiple implementations of the same thing takes more effort than updating only one (the code).
Too many details in design will require frequent changes, with low level of detail it won't help much to detect missing pieces of the puzzle. Either way it adds effort.
Design has its advantages, lower total cost/effort in the short term isn't one though.
> You design it properly, then you implement it properly.
I’m extremely defensive and try to think through everything. Aside from analysis paralysis which is a real obstacle, it’s still not enough. Or rather, it’s not the best way to use a human brain. At some point for a complex project, no matter how smart and experienced you are, assumptions break down. The mental model changes based on “friction” with the gritty world. Prototyping is one way to confront wrong assumptions earlier rather than later.
All creative and intellectual endeavors have analogous process. Writers don’t structure a book and write it perfectly in a methodical fashion. Basically, turning chaos into order can – almost per definition – not be proceduralized.
229 comments
[ 3.9 ms ] story [ 267 ms ] threadIt helps if you have tests because when prototyping, you can write things in a way which break tests and ideally break other features. It works enough that you can validate the one idea. That way it will never pass CI.
Besides discovering unknown-unknowns it also helps in conveying design/idea to rest of the team.
Which actually doesn't work so well in AoE. The game doesn't have a concept of army supplies, so you usually just want a huge unstoppable army you can send anywhere.
The problem with rewriting is that sometimes you face second-system syndrome[1] where you overengineer everything.
Believe me--it happened to me at Groove Networks after Lotus Notes (ask your grandparents about that).
[1] https://en.wikipedia.org/wiki/Second-system_effect
THIS. Often, the best way to start the rewrite is to go straight to the database, and start asking "what allowed values for all these fields have never actually been used?".
EDIT: Whether it's rewriting, or refactoring, or adding some "if( Deprecated_Feature_Allow !== true ) ... ErrorDie( Feature_Deprecated )" logic here and there, or updating training material, or whatever - knowing that Software_Feature got ~zero real-world use is d*mned useful.
But this is not incompatible with refactoring instead of rewriting.
I think it just boils down to "keep writing systems and you'll get better at it." But there's no bait on that kind of link.
Less cheekily -- it's true that sometimes the best way to research a topic is by writing code that you might appropriately throw away later. And it is extremely important to be able to recognize when something is not worth keeping. But also, often, the most efficient strategy is to think about the problem off-screen enough that you don't subsequently waste time on-screen writing things you won't keep.
It seems strange to usually expect to throw away your first draft. Not every problem is that hard, and even for hard problems, not every first attempt is wrong.
Often it is a bad idea. I get it, though, there is an inherent attractiveness in the idea of starting fresh from a clean slate. Except, more likely than not, you'll soon find yourself in a similar situation to the one you started from. The fundamental problem is that it is easy to underestimate the edge cases. Sure, the main functionality is easily understood and straight-forward to conceptualize. But did you remember to think through all the smaller aspects of your software, too?
Perhaps it's easier with a prototype implementation that in fact doesn't have a lot of features yet, but to completely replicate the functionality of a complex piece of software isn't an easy undertaking. Sure, getting 80% there is probably easy, but the last 20% is the part that's easy to overlook when considering a complete rewrite.
Admittedly, there's nothing sexy about refactoring. And often it may seem like it's less work to just simply scrap everything and start over. However, that's fallacy a lot of times.
Maybe this is ok, given that time period?
or will the rewrite have all these extra bells and whistles?
or will the rewrite throw away the unneeded bells and whistles?
Then email me the PDF and store the papers in a box, and a couple of days later I start the v2 (same process), then v3.
By v4 it's 'good'.
I also use the same method on my 9-5.
I take VERY seriously the Abraham Lincoln quote “Give me six hours to chop down a tree and I will spend the first four sharpening the axe” on almost everything.
I consider the v1, v2, v3 as the "sharpening the axe" and the v4 on the actual cutting.
In that spirit, the article has a similar approach, as the v1, v2, and v3 may take you down an (more than) imperfect path.
I want an editor plugin which allows me to mark sections of code as reviewed or "perfect" (depending on how honest I'm being). Then, when I'm tempted to rewrite everything, I can go through and mark what I think is good, and then focus on refactoring the rest until I think it is good as well.
I'm tempted to rewrite code because I lose track of what it's doing, or I've learned a lot since I wrote that old code and so I'm not sure if the old code is good anymore. It's not so much about rewriting the code as an exercise in getting familiar with the code I've already written. I want a tool to help me with this.
Unit tests prove the code works as intended, and are basically examples of what the code is doing. Whether the code is actually "good" is a bit more subjective -- but tests give you the freedom to modify it without breaking it.
Checking into git frequently is also a way to give yourself some freedom. Commit at every milestone, like every time the next thing is "working". If you feel like refactoring, go for it -- you can reset back to working state in a few seconds.
And lastly, leave comments in. You can always clean it up before you push. You can even squash or interactively rebase your history so no one else sees the gory details how the hot dog was actually made.
In practice, because often the unit tests are mocking out all the classes they interact with (as is the recommended style for isolated tests) you’ll either have all the tests explode constantly because the interface being mocked has changed, or you’ll have tests that confidently tell you things are working despite the methods they call no longer existing.
But unfortunately the modules don’t make themselves apparent at the start of the project. So it needs refactoring discipline.
Still managers always push against redoing work they can see working for the happy flow.
I get the PoC down asap then as soon as it’s proven (privately to myself) I move onto optimizing. If it fails then I communicate early to discuss a change of scope.
Assuming it succeeds, by the time I present it I have already done significant work making it production ready. So to request an extra 10-25% time to “smooth out the edges” goes over easy after the demo.
In reality I end up with 50-75% time to get it prod ready. The rest is QA + prod bug fixes to take it over the (practically scoped) finish line.
If that bugs you because you consume information on the web rather than socialize, the solution is simple: don't read comment sections and make up your own mind about the article.
There's the "strangler fig" (my favorite) method of refactoring. It's where you rewrite just a small portion of the software, little bits at a time. Instead of doing a full rewrite.
IMHO, it's the best way to refactor. You can switch out both "old" and "new" versions at-will until you're 100% sure you've covered all the edge cases.
And at the same time, perhaps their code is less encapsulated, because they didn't optimize for abstraction, they optimized for beauty. A leaky abstraction doesn't bother them, because ALL abstractions are leaky to them, they probably have a sense of internal workings even whem using household appliances, but ugly code tucked away somewhere bothers them a lot, and they might dislike using popular large libraries even if they work great, just because they're not comfortable using what they don't understand deeply.
My evidence of this is the fact that suckless exists and people actually use it, I assume their experience of thought is very different from anything I have experienced.
This usually means that rewrites are significant improvements across the board, especially if they're done a relatively short time after the original is finished since it's all still fresh in my head.
This may be a weakness of sorts on my part though, I lack formal engineering training which might be why purely mental modeling (no code) doesn't work all that well for me.
A lot of situations where the code is so nasty I actually have wanted to rewrite... I wind up just throwing it away instead.
Like, one of my happiest days in coding was throwing out some code to support JACK and going all in on Pipewire. Another happy day was replacing some DIY code with a GPL licensed library, deciding that I really didn't need the option to do proprietary stuff anywhere near as much as I wanted less thinking about gstreamer.
"If it's hard to explain, it's probably a bad idea".... If I try a bunch of API variations and I can't come up with anything that doesn't require learning 3 new algorithms to use... Maybe I'm not making a library or an abstraction, I'm making a proof of work hashing scheme that users have to do manually to access an encrypted version of the complexity I'm hiding, and I need to stop before I make z80 assembly in JSON to autogenerate Vue templates that render to cobol.
Of course clean and readable means very different things to different people, but I don't thing I ever regretted cleaning up code, while I can think of a lot of instances where not doing it wasted a lot of time.
"If I had more time, I would have written a shorter letter."
As others said it's not for beauty, it to make sure if there is abstraction it fits the problem. If there is encapsulation it doesn't get in the way. For some coders they can get it on the first try and there is no reason for them to rewrite code. For the rest of us mid coders we need to explore first as well as make sure all cases we desire for Lib/API work.
Agreed. Though sometimes you need to refactor the core of an application, in a way which will touch the entire app. To do that I often make a new, empty project. Then I rewrite the core of my project into the new folder (in whatever new structure I’m trying out). When that works, I slowly copy the content from the old project into the new project - refactoring and testing along the way.
But it’s not perfect. About half the time I do this, I discover halfway through that I didn’t understand some aspect of the system. My new design is wrong or useless and I throw away all the new code. Or I figure out that I can just make the changes in place in the old project folder after all, and I bail on the new code and go back to a traditional refactor.
But no matter what happens, I don’t think I’ve ever regretted a refactoring attempt like this. I always come away feeling like I’ve learned something. How much you’ve learned from a project is measured by the number of lines of code you threw away in the process.
https://devrant.com/rants/816880/i-ve-done-it-again
It's weird: you rarely have to maintain something for 20+ years, and you get to always improve and iterate on how you did things last time. But, are you training yourself to write hard to maintain code, since you don't really have to maintain it past a certain period? Or does the learning-from-iteration actually make writing-maintainable-code easier?
I know some people do keep developing their games for decades, look at Dwarf Fortress, I'm just talking in general.
I do agree that it's generally a bad idea to just throw out a long-lived code base. But for me that's not an estimation issue. It's because the urge to throw it out is usually a response to upstream problems that haven't gotten fixed. For example, a lot of code bases are a mess due to time pressure. But if you don't fix the process problems that turn time pressure into mess, then you're just doing to end up with another mess. Possibly a bigger one, in that stopping all productive output in favor of a rewrite usually makes business stakeholders crazy, causing increased time pressure.
https://www.joelonsoftware.com/2000/04/06/things-you-should-...
Granted Joel is taking about a wholesale rewrite of proven / established code. Code and has had had the advantage of time.
Personally, I end up rewriting parts of my code often. It usually takes a mile for me to find exactly the right way it should be based on how are people use it.
You absolutely should rewrite prototypes, and re-factor important chunks of code. I rewrote something 3 times today, it was better each time.
At the same time you should be wary…
Maybe I’m not an engineer, because I bet perfectly pretty fast code is an indicator you should reorganize it to make it easier to adapt, add more capabilities even if it runs slow, and put some ugly fixes in there so it’s useful for the end user.
Agreed that wanting to over-engineer and perfect something that should be constantly evolving is not a good reason to rewrite it. That sounds like back office bubble pretend-work.
By using 2 different languages, there is more freedom to just make a demo, discuss, and then decide what to keep or what direction to go.
Especially when in comes to business logic and domain modelling, some bad decisions at the begining can cost a LOT, to undo in an iterative matter.
(To clarify I'm talking about the timeline the author proposes of "a couple of days" build the throw away prototype)
http://m.quickmeme.com/meme/3629ez
That's the advantage of software over other work. You have 0 material sunk costs. Unlike building a bridge, which you can't just tear down and iterate, with software you can develop a workflow where you break things and iterate quickly.
But a decision like that would be superhuman by itself, a transcendent experience, giving the creators a perspective that seems impossible. Unlike what is happening instead, both publicly and behind closed doors... A great pity!
I think I do use the throwaway approach with simple prototyping of smaller portions of code that I will then port into an existing system, but the overall system remains stable.
I recommend you do this too.
> If you know that you're possibly keeping the code, you do things in a "proper" way, which means moving slower
combined with the idea that it's faster to redo work than it is to refactor.
I don't buy either of these propositions as a universal rule. For the first, it seems that a mindset of avoiding premature optimization, which you should be cultivating anyway, is enough to ward it off.
For the second, I see it as very context dependent, with my bias going towards refactoring instead of rewriting. I most often get things right enough the first time that it really only needs to be refactored going forward. Only when I end up with serious flaws do I decide to start over, often before I finish my implementation because the flaws are already apparent enough. Add to this that even very large changes can be refactored effectively and timely in my experience, and I feel like the bar for rewriting is really high to be cost-effective.
Unfortunately this essay and many others seem to take objections like mine for granted, and I should just believe the author that rewriting is faster with zero supporting evidence.
I won't approve a PR if I notice such a thing (and if it really is "gaping" I'd expect an automated tool to flag it and fail the build anyway). I don't really see ensuring your code follows basic secure SDLC best practices as "optimisation" at all - in fact it often comes at a (hopefully small) cost of less than optimal performance. Of course there may be areas of software development (hobby projects etc.) where security is not a priority at all, but if it's code you're being paid to write that runs on the cloud or your customer's machines then security is absolutely worth getting right from the get-go.
I think it really depends on your values and what you’re working on.
If you have the mindset of a product engineer, the thing you care the most about is how well the software works for your users. The code is an unfortunate necessity. When I build react apps, I tend to think like this. The code I write first go is often good enough to last until the next redesign of the UI.
By contrast, if you’re thinking of the code as a home you build for your work that you will live in, then having a tidy home becomes its own reward. Good decisions today (in terms of cleaning up or refactoring code) should lead to more velocity later. When I think about building a database, an OS kernel or a compiler, I think like this. When I start, I don’t know where my code should be rigid and where it should be flexible. My first guesses are usually very wrong.
Personally I prefer the latter kind of problem. I like it when there’s no obvious way to structure my code and I have to explore while I work. The code I’m most proud of writing has probably been rewritten 5 or more times before I landed on the “final” design. The “right” abstractions are rarely obvious at first glance and the search can be incredibly rewarding.
The rich history of application programming abstractions suggests it isn’t exempt from this either. It’s just, when you’re building a UI there have been an awful lot of explorers who came before you and can show you the way.
If you are in a context where you are in danger of making critical design flaws often, or where circumstances make it hard to make changes after committing, then I absolutely agree that rewriting is effective. That's just what I mean by a high bar, one that I personally don't encounter very often in my work.
Setting aside the part that's just fussbudgetry, I think "proper" coding is about doing things that pay off in the long term even if they feel burdensome now. As an example, I'm a huge fan of good automated tests for long-lived code bases. But if I'm just writing a quick, throwaway script, then building a bunch of automated tests are going to slow me down.
Throwaway prototypes are another area where you can save a lot of effort if you have really committed to throwing the code away once you've learned what you set out to learn. As a physical analogy, before the Long Now built out their bar/cafe space in San Francisco [1], we spent a day in the raw space building furniture and walls out of cardboard and tape [2]. It let us get a real sense of how the designs would work in practice for actual use of the space.
In the past for a number of projects I've done throwaway prototypes and then started fresh with "proper" engineering, which for me usually includes pairing, test-driven development, and continuous deployment. I've also done ones where we think we understand it enough and just start building. And either way, we usually end up evolving ways to prototype on top of the existing platform via feature flags, etc, so we can learn something from the real world and then decide whether or not we want to refactor to accommodate a major change.
[1] https://theinterval.org/
[2] https://vimeo.com/73959127
And there are situations I'd go in assuming I'll probably rewrite the work, but I wouldn't set the criteria so broadly as to cover literally whatever the next large project I embark on will be, as the article suggests.
And personally, although I'm very good at refactoring, I will still go the path of a disposable prototype in the future when I think it's more effective in terms of total effort. Which for me is any time I expect the prototype to yield significant information about both the surface and the substance of what we're doing.
https://web.archive.org/web/20230804183217/https://ntietz.co...
It's probably a little of both in reality but it's an interesting idea.
I'm fairly grateful that I've retired from the rodent rally, and am working at pretty much my own pace and structure.
About six months ago, I tossed out the code for the frontend of a project that I'd been developing for over a year and a half (at that point). I distilled the business logic into an SDK, brought in a designer, and started over from scratch.
The result is ten thousand times better than what we had (and were considering shipping), a few months ago.
I think devs in general sweat too much about code quality.
We have code like that, and it requires everything around it to twist and contort to comply with the beast of unmaintainable at the center of it all, simply because at this point it has so poor test coverage and is so incredibly mission critical now.
I think if they sweated a little more then poor quality code wouldn't find itself into the codebase.
But almost all PRs I review have something in them that suggests the developer not putting as much thought/care into what they were doing as I'd like to see (and I'm just as capable of doing the same).
There's not really a book or standard on opinionated code quality that satisfies a majority of the people in the field. There's way too many ways to build things for that to exist.
This leaves it up to individual teams to approach a shared consensus (with some wiggle room) within their domain and tool set.
One thing that is universal is how strict the enforcement of a team's quality standards. So as a developer, you have to decide whether you like strict or lax enforcement of quality standards. Everyone can decide that but I think most don't.
According to him, each iteration tends to go smoother and faster than the last.
Edit:
There’s probably some nuance to whether this approach is a good idea based on type of software, size of team, experience level, personal and/or team skill set, etc…
Now, if for some reason, and there are plenty of good ones, you cannot do the former, then sure, you pay that price by redoing some work. But, at that point, you do the same thing you should have done: you design it properly, then you implement your design.
One could argue that implementing a first draft is a way of designing it. Which, I cannot imagine making sense unless you don't know what it is you want to end up with. Maybe if you're making computer games or other creative endeavors?
It's when the problem itself has some ambiguity. Like, I design systems that other people and systems use within a large company. I try to figure out how they'll use them, but even they can't tell me exactly. New business requirements come in mid-design, etc. They might even cancel the whole project if something more important comes along. In that situation, it's best to get some MVP as quickly as possible then iterate on it. Later iterations might totally scrap and rewrite the internals, which is fine if the API sticks.
Text (source code) as a representation of ideas can be worked on extremely efficiently at arbitrary detail. At low level of detail only beaten by a whiteboard.
If any piece of information has been missed when creating anything (any form of design or code), updating mutiple implementations of the same thing takes more effort than updating only one (the code).
Too many details in design will require frequent changes, with low level of detail it won't help much to detect missing pieces of the puzzle. Either way it adds effort.
Design has its advantages, lower total cost/effort in the short term isn't one though.
And how, if I may ask, does one design it properly?
I’m extremely defensive and try to think through everything. Aside from analysis paralysis which is a real obstacle, it’s still not enough. Or rather, it’s not the best way to use a human brain. At some point for a complex project, no matter how smart and experienced you are, assumptions break down. The mental model changes based on “friction” with the gritty world. Prototyping is one way to confront wrong assumptions earlier rather than later.
All creative and intellectual endeavors have analogous process. Writers don’t structure a book and write it perfectly in a methodical fashion. Basically, turning chaos into order can – almost per definition – not be proceduralized.