The question is whether updates can be made against both copies of the duplicate, requiring synchronization. It's the difference between a replicated slave copy of a database and a real distributed one. The first is easy. The second is very hard.
I'd argue if you find yourself needing to do this, it actually might be a hint that abstracting is appropriate, as it's proof there is more than superficial commonality. Like so many things, I don't think there's a way to make any hard and fast rules, and figuring this out is more art/experience/taste than science.
Not including cross-cutting concerns that modify all usages (eg, changing your logging or dependency injection library).
The "wrong" (less than optimal) abstraction might be better as long as it's rigorously documented, which is rare, of course. In the right hands, it can be an important stepping stone to a better abstraction. I'm dreaming already.
Duplication has it's own set of dangers leading down the road to a verbose mess of convoluted crap code in most cases.
Most established code bases make me want to puke before long.
The older I get the more I appreciate XP. The Rule of Three in particular becomes a bigger presence in my life as time goes on.
With two copies of the code you can’t be sure if the similarities are factual or coincidental. At three the situation begins to crystallize quite rapidly.
I agree with this so much. The way I've put it before is, the difference between under-engineering and over-engineering is that you can fix under-engineering.
Yes! I think what I've come to is: Spend more time thinking about how to do something more concisely, rather than spending time thinking of how to abstract the code. Sometimes the former leads to the latter. But less code--or rather more concise code (not to be misrepresented as "fewer characters" or "everything in one line"!)--is almost always better for maintaining the code into the future.
Under-engineering makes it hard to fix common issues because there is no common code to fix. You can find yourself scouring over thousands of lines of code never sure that you found all the places you need to find. This is why DRY is a solid principle.
I think there is always the problem that programmers want to apply all these principles, including DRY, without actually thinking about it. Once it's applied illogically you are in strange and awful territory.
You can fix under engineering only when it is under control. Too much of under-engineered code may lead to bloated codebase which quickly becomes unmanageable.
When you find yourself at his step 8 you don’t necessarily have to go back and fix all the sins of the past. There’s a cost to this which may or may not make sense to pay. You could simply not use the bad abstraction.
It should also be pointed out that when you find yourself at steps 6 and 7 you don't have to sin, and when you find yourself at step 1, you can obey more complicated heuristics than DRY, like "I will apply the rule of three if the duplicated code is pretty short and not inherently self-contained" or "if this big method looks like it will change, instead of fully abstracting it, I'll just break out the bits that look like neat little functions".
You only find yourself at step 8 after a suite of bad decisions, and possibly even bad decisions that you signed off on during code review.
The "wrong" abstraction is still a nexus of control and understanding - a point you know the code will return to under certain circumstances
this is not to say that if you rollback the code and then commit tonfinding and implementing new abstractions you won't win - but the second part is necessary or you are just digging a deeper hole.
think of it as a wrapped transduction - younhave to do the second part as well.
On the one hand, often there can be shared lines of code without a shared idea, this shouldn't be a candidate for being factored out into some new abstraction.
On the other hand, you may want to introduce an abstraction and factor out code into a common library / dependency / framework when there's a shared well-defined concern/responsibility.
That said, on the gripping hand, I say may because even if there's the opportunity to introduce a clean future-proof abstraction, introducing it may be at the cost of coupling two things that were not previously coupled. If you've got very good automated test coverage and CI for the new common dependency and its impact upon the places it is consumed, then perhaps this is fine. If the new common dependency is consumed by different projects with different pressures for change / different rates of change / different levels of software quality then factoring out a common idea that then introduces coupling may cause more harm than good.
This comment is more insightful than the original post.
The real problem is when engineers abhor duplication, and in order to reuse existing code, they simply call the same code from multiple places without thinking through the basis for this reuse.
Mindless deduplication is not an act of abstraction at all! This is a very important point, because a "wrong" abstraction that is conceptually sound is not that hard to evolve, and if the code is called from N places then you get to look at those places to understand how to evolve the abstraction. Improvements to one part of the code benefit N parts, and you save work.
The only other factor to keep in mind is the dependency graph and coupling, as my parent mentions.
Mindless deduplication is more common than you'd think, especially with bits of code like utility functions and React components. For example, you end up with a component called Button that has no defined scope of responsibility; it's just used in one way or another in 17 different places that render something that looks or acts sort of like a button. This is not the "wrong abstraction," it is code reuse without abstraction.
I know what you mean, but you need to find a different or more nuanced term. Deduplication is abstraction, it just isn't an abstraction mapped to the domain problem. Even a compression algorithm abstracts:
An abstraction can be seen as a compression process, mapping multiple different pieces of constituent data to a single piece of abstract data; [1]
Conceptual or semantic compression, yes, as the rest of that section makes clear. The very problem with deduplication without abstraction is not thinking at the conceptual level, only at the literal code level. There are lots of ways to compress code, e.g. minifying it :)
Quoting the start of the article: Abstraction in its main sense is a conceptual process where general rules and concepts are derived from the usage and classification of specific examples, literal ("real" or "concrete") signifiers, first principles, or other methods.
For the Button case, you'd have to come up with some concept of what a Button is and does, beyond what code lives in its file (e.g. an onClick handler that calls handleClick, etc.) in order to have an abstraction.
There are "wrong" abstractions (in the sense of abstractions that turn out to need to be changed later, like any code), but if you lump all deduplication into abstraction then you will have a skewed sense of the cost of changing an abstraction.
The cost of changing an abstraction also depends on your programming language; if you spend a lot of time in a dynamically-typed language, you may internalize that refactoring is tedious and error-prone and often intractable.
People often forget "copy-on-write". Coupling doesn't have to be permanent. If refactor to create a sahred component, and then you want to modify a shared component to help one client, you can fork it -- it's not worse than simply not having created the shared component in the first place.
In my experience people will most likely just hack the shared component by adding awful arbitrary if-statements or other such hacks, rather than fork the shared component. This is the path of least resistance. Once this happens a few times that shared component begins to be seen as a central component and is quite a complicated mess.
But often the fork happens too late, after the first few differences have been creatively shoehorned into the shared code. The resulting mess then tends to live on twice after the fork.
In the end, almost every conceptual way to slice up software can be viable if you are good at whatever you do, and terrible of not.
> when there's a shared well-defined concern/responsibility
I think a good test for this is if you can write a reasonable unit test for the code in question. If your unit tests essentially become two separate sets of tests, testing the different branches of code, it's probably the wrong abstraction. If your tests work and you've built a reasonable standalone library (even if it's not useful to anything but your exact product), that's at least a signal the abstraction is sustainable.
Now we simply need a formal objective definition of "reasonable code" and the industry should never have this problem again.
Even if some shared code is currently branch-free though, it may be unlikely to remain that way if the abstraction is fragile.
A red flag is vague function names like "processThing" or "afterThingHappens". If a function can't be summarized concisely, it's probably doing too many things, and the abstraction is likely to break down later when the callers' needs diverge.
> If the new common dependency is consumed by different projects with different pressures for change / different rates of change / different levels of software quality then factoring out a common idea that then introduces coupling may cause more harm than good.
This reminds me of the recent post on HN about a company migrating from microservices back to a monolith, for this exact reason.
Seeing the future? Picking the right macro strategy, programming language, database, etc ahead of time, sure, an experienced developer can usually do that. But correctly predicting the boundaries of systems, libraries, APIs—and guessing who will maintain which system and figuring out dependencies—that never goes according to plan. So in my experience YAGNI and reducing coupling are more important principles than sharing code.
If I'm writing an API to move a robot, my problem space is fairly bounded, and I know that someday I will want force control at some end effector. I know that there's a 6 axis robot I've been eyeing, etc.
I guess what I have run into is a lot of code that is agressively, needlessly abstracted for a future that will likely never come. Abstractions that perhaps would be worth it to save hundreds of copies or permutations, while I'm looking at one.
I'm all for not repeating myself, but there is a different between "usually avoid" and "never"
Copying and pasting in many situations would seem a breeze compared to the nest of abstractions required to avoid it.
Programmers have had DRY drummed into them so hard that it is almost heretical to even consider the tradeoff of increased coupling that arises from it. Coupling is good if things should change together because they are linked in some fundamental way such that it would be wrong for them to be different. Coupling is bad when things should evolve independently and the shared link is incidental.
The problem is that it is surprisingly hard to tell the difference up front. In the moment of writing the code, the evidence for the shared abstraction seems overwhelming, but the evidence of the cost of coupling is completely absent. It exists only in a hypothetical future. Unless there is strong evidence for a shared underlying conceptual link, I often consider only the 3rd repetition of a shared piece of code evidence for the existence of a true abstraction. Two times could just be chance, three is unlikely to be so.
It is not just that 3 indicates the existence of an abstraction, but seeing 3 examples improves your odds of identifying the correct abstraction to use.
> In the moment of writing the code, the evidence for the shared abstraction seems overwhelming, but the evidence of the cost of coupling is completely absent
This is actually representative of a problem in the industry as a whole I think. A lot of things have short term benefits but long term drawbacks. Because of the drastic, recent growth, orgs are bottom heavy (very few people have experienced long term drawbacks of X compared to how many people who just learnt X). Additionally, because of the extremely quick turnover of people, it's even rare that people who implement X are there when X blows up in people's face. They went on to implement Y...and will be gone before Y blows up.
So most tools, libraries, frameworks and abstractions are HEAVILY optimized for the short term. Optimized for getting a project set up quick. Optimized for the initial "Hello world". Optimized to get an API and a form in seconds. Very few tools/patterns are optimized for ease of long term (hell...these days long term means a year or two) maintenance. The ones that are generally get a bad rep.
And building stuff that's both good short AND long term is very, very hard.
Having worked with some Go programs, where one of the sayings is:
A little copying is better than a little dependency.
This is even in the case with the two copies share the same correct abstraction. It's different than in the npm world. I've taken this and applied it in microservices written with other languages/frameworks and have no regrets. Sometimes some versions are a bit less complete or featureful, but each works fine. If a bug is discovered, it's fairly easy to and patch them all.
It's usually the intermediate developers who like having rules to follow to know that they're doing well that tend to over-apply DRY and other principles. Only experience (aka pain over time) seems to show when to break (or just not apply) the rules.
Perhaps it's just the way things are taught/learned. Instead of just showing what's good and have them interpreted as rules, each should be shown as a rule of thumb with a concrete example of when it should not be applied. Even if they don't clearly understand the difference at the time, they'll always recall that there are exceptions and not feel so motivated to apply it in every instance.
As a senior engineer who recently became an engineering manager I always caution my devs about abstracting too liberally. Junior engineers are particularly bad about this. They see a handful of functions that are duplicated across a few (unrelated) projects and they want to create a new repo/library and share it. Then I direct them to the Slack channel for our platform services, which has a sizable shared library across dozens of services. That shared library is a frequent source of problems.
It takes a while, but I usually beat that primal impulse out of them.
> They see a handful of functions that are duplicated across a few (unrelated) projects and they want to create a new repo/library and share it
He is what I think you do when you do that.
You just created yet another internal API. Designing, creating and _documenting_ good API's is _hard_. The most likely result is an undocumented dodgy half finished API that doesn't fully encapsulate the thing it's supposed to deal with. So you end up with code that both uses and bypasses the API you just wrote.
If you do that and later decide that you want to move some functionality from one side of the API to the other you've just set yourself up for a hella lot of work.
The other thing is, you want to make changes to duplicated code. You can limit the risk to code base you're actually working on and not a bunch of unrelated programs.
I am continually fighting the DRY cult at work. I explain that we need to focus on whether the code shares requirements or implementation. With one, the duplicated code would evolve hand-in-hand while the other can have the code evolve separately. The problem is it requires a tipping point before you split the code up again. So instead you extend the existing code to handle disparate requirements, turning it into a god function / object.
To help weed out requirements I tell people that on their third copy / paste, they might begin to consider reducing the duplication. At that point, they both have had time to think about the code and had gained experience with it to discover what the requirements are.
Another problem with bad code reuse is code locality. Like with Instruction and memory locality helping to improve runtime performance via caching, code locality helps improve mental processing. The further you separate related pieces of code, the more you need to have a good abstraction for it so you can correctly reason about the code. Without a good abstraction, you are having to jump between far areas of code to figure out what your function does.
I got some flack from some other students learning when I had some duplicated code in a few places. I tried to explain that while they shared some common code the way some of the code that used it worked really didn't have the same goal and/or might change as opposed to other code. So while I was sharing some places, I chose not to share in others to allow each area some level of independence if / when we change that. They were all zombie "Barrrrrrgh look at me reusing code all efficient like!"
Granted we were all n00bs and nobody will see that code again so it wasn't a big deal... but the intent, direction, and possible future of the code seems like something that should be considered once you start sharing.
> but the intent, direction, and possible future of the code seems like something that should be considered once you start sharing.
Yes. Dare I say, intent is one of the most important things here. Two new pieces of code may be structurally identical, and yet represent completely different concepts. Such code should not be DRYed. Not just because there's a high chance you'll have to split the abstraction out later on - but also because any time two operations use a common piece of code, there's an implicit meaning conveyed here, that the common piece of code represents a concept that's shared by the two operations.
My jam is to wait for a few use cases before creating a new abstraction or process. I want to see how they are similar, and how they are different too in order to form a generalized solution that will serve the use cases at once. Dealing more and more with tooling for other developers, this applies especially to the tooling APIs.
I apply this to DRY, coupling, encapsulation, APIs, etc. Also, I prefer to focus on consistency and readability over most other concerns. I mentally, or physically!, note areas of code I want to improve but don't feel like the time is right now, right now. During future work that touches that code I will refactor it if a solution has presented itself.
These days I prefer languages with bomber language services and tooling to make refactor in the future as painless as possible(types). I prefer explicit over implicit(sorry Ruby and Chef), and configuration over convention(looking at you Gradle).
I can't agree with this beyond the high-level sentiment. Perhaps the example is just not good. In the listed example the error clearly lies with programmer B who decided to extend an abstraction with a branch. That's not extending an abstraction - that's squashing 2 different abstractions into one function!
but the article argues against creating the abstraction when the functionality was the same in both places, which I disagree with. To be honest, the whole topic disregards whether the two pieces of functionality are actually doing the same thing in a domain-logic sense or simply a code sense, which is the heart of the issue. If the similarity is coincidental (e.g. validation on two model types temporarily happens to be the same) then the abstraction should not occur. If it is not (e.g. two distinct operations on a model type happen to have an identical subroutine) then the abstraction should occur.
It is easier to introduce the right abstraction later. The article argues that if there will be no specific abstraction, programmer B would create the right abstraction without hacking on top of existing. This becomes more oblivious when the application is in maintenance and everyone is risk averse.
I've done exactly this where I'm just extending my code in an ugly way. Two alternative patterns to extending a function with a parameter switch are to pass in a function to the function or split the function into multiple functions around the bit thats different.
Definitely a lot harder to fix when you've done this though.
I think "the wrong abstraction" is too lofty a title this is just about oversize functions.
I had this in mind recently when doing the Cryptopals challenge, which requires you to implement the Counter mode of block encryption[1]. In that case, encryption is the same mathematical operation as decryption. I still figured I should have two different functions for encryption vs decryption to make it more obvious which variables are intended to hold a plaintext vs ciphertext.
Off topic, but I'd highly recommend that Cryptopals set of challeges for anyone that likes a coding challenge and wants to dip their hairy toes in encryption. Eudyptula one also cool; I think last time it came up there was a list of some similar ones.
I find that a particularly good way of learning (at least over plain reading/coding).
> When dealing with the wrong abstraction, the fastest way forward is back. ... Re-introduce duplication by inlining the abstracted code back into every caller.
This can be tricky if the person who wrote the abstraction takes it the wrong way. At my previous company, I've been yelled at for doing this. Some developers get emotional about their code, in which case undoing their abstraction causes offense. How do you get around this?
I find that the best approach is to talk to them first, explain your use case, ask them how to solve the problem.
Most of the time you will either be welcomed for your proper obsequence, or find out that there's a jealous guardian of that code and no matter what you do it won't alleviate that tension anyway.
Opportunity for leadership. Explain to them that you are not your code. That code is never done, and like writing, it can improve with each draft. Also, what specific behavior of theirs makes you feel like they are yelling (raised voice, standing close, interrupting). People aren't perfect, but we can help each other get closer.
one problem with duplication is that its tempting to copy the code you need and replace the appropriate parameters or values. Its pretty easy to mess that up.
After all the hemming and hawing about this, I'd really like to see a real case study on it. It's never been an issue for me. I've had functions where I added parameters later on for new use cases to fit the function to them, yes, but I never felt like I was suffering for it. Maybe DRY isn't so bad after all.
The Go community has always embraced this. On the Go Proverbs page[1], it's expressed as “A little copying is better than a little dependency” — true, it's not precisely the same idea, but close.
Just copy-paste code and add a comment pointing to the source (starting with TODO helps you find those later).
It will become obvious which duplicated code to abstract when you find yourself changing all/many at the same time or fearing you’ll forget/break something if you don’t change all instances. Writing tests is also a good motivator as it means even more code per duplication (and reduces the fear of breaking something).
It really takes a lot of duplication for this to get out of hand. Wait til it happens, you’re a software engineer you’ll figure out how to get rid of duplicated code just fine. Coming up with a great abstraction is extremely difficult before seeing at least a few examples.
One of the easiest ways to fix this is with lambda functions or callbacks. Each caller passes a lambda function or call back that is the specific case that's unique to the caller.
In the case they're describing this should almost surely not be a lambda function. Those should be reserved for when an arbitrary function makes sense, but it's unlikely that someone would have abstracted a function that could have done anything.
It's much more likely that the code in question is responsible for one particular thing, and switches between several different ways to achieve some sub-goal. Those parts should be lifted into some kind of interface, where the different variations are lifted into different implementations of that interface. A lambda function is the most general interface possible, so it's probably not the best choice, you'd eventually end up with callbacks calling each other without it being entirely clear which callback does what.
> A lambda function is the most general interface possible
A typed lambda function (i.e., with a defined arguments + return signature) is exactly as specific as any other typed interface, an arbitrary lambda function isn't, sure, but there's few languages where a static interface and an arbitrary lambda function are both available tools.
When optimizing for complexity, this is a reasonable argument. What about optimizing for performance? It might be cheaper to make a network call in one place and have multiple consumers each use the result, even if each of them needs to use the result in slightly different ways.
Nature obviously agrees. Look at the structure of the genome with all its gene duplication with minor variation. Copy-paste-hack is one of the primary mechanisms of evolution.
I still regularly fall for this trap. My instinct to "DRY" is so ingrained into me that I always find myself deduping similar looking blocks of code. I think this culture is a knee jerk reaction to dealing with code bases on the opposite end of the spectrum where entire classes are "copy and pasted" with only a single change. I've had the misfortune of dealing with these kinds of projects.
I now try to find the middle ground by remembering to "do the simple thing" even if it appears less elegant. This makes it easier to refactor in the future (if required) at which point more information will be available to design a more appropriate abstraction than would have been possible before.
One of my coworkers won’t be happy until we are all living in Death Valley. Almost nobody can follow his code, and given that we aren’t trying to save the world, pretty much nobody tries anymore.
So now he’s indispensable, a situation I assiduously avoid (you can’t work on the cool stuff when you have to be there to maintain the old thing).
While unexpected features can definitely complicate something that was merged into an abstraction, bug “fixes” also matter and they can be worse:
- Programmers may look at a bug in a simple shared function and conclude that it “obviously” should be fixed, and do so quickly without really understanding what else could go wrong. (As a completely contrived example: You “fix” something that previously couldn’t return a negative value, and move on; turns out this “fix” allows a bug somewhere else to crop up, a catastrophic improper cast from signed to unsigned, blowing up your -1 into an iteration over billions of expensive operations.)
- Bug priority levels vary between features, even if code is shared. Your abstraction may make it effectively impossible to fix just one high-priority feature, if your deployment is (say) set up to run hours or days of regression tests on all affected parts. Generally, the more segregated things actually are, the easier it is to set priorities well.
Just because something is duplicated doesn’t mean that it’s that way forever, either. At a good branch point, such as a new project, you can aggressively prune out things that won’t apply to that branch even if they helped keep things stable on the previous project.
My own read in this situation is that this problem is almost always less of an issue in code that is very explicit about what it’s doing. A function that uses nouns and verbs with a very precise meanings survives these changes better than wishywashy code.
“Generic” functions make it difficult to find all uses or even understand what scenarios they belong to. With bland say-nothing nouns and meaningless verbs like execute() or process() that appear everywhere in the code, you’re just crossing your fingers and hoping for the best.
I use the "dueling sins" model of "copying versus coupling." Early lifecycle code benefits from the flexibility of copying. Mature code benefits from the coupling forces of abstraction. Both have the capacity to do harm.
Do you follow Jim Highsmith at all? His philosophy is that there are no answers. That we are minmaxing a bunch of competing criteria and trying to do the best we can.
In his words: we are a solving problems, we are resolving paradoxes.
Good idea in general. Not sure I agree with the idea of inlining a bunch of code in order to refactor an abstraction.
I've always found the easiest way to refactor is to get really good code coverage on the outermost layer of code that uses the abstraction, remove/ignore unit tests if there are any on the abstraction itself, then refactor the abstraction with vigor until it feels appropriate and hopefully elegant or is removed if necessary. As long as all your tests still pass, you should be good to go.
I experienced a crystal clear example of this about 15 years ago. I developed a prototype of perl script that read data from one folder, filtered and enhanced it, and wrote it out to another folder, where it was picked up by another process. The script was configured by another perl file that contained the paths to read and write from. The initial (pre-production) deployment had the two folders adjacent, so the config script looked something like
At some point the script was handed off to some other dev who looked at those paths and apparently thought "that's not DRY!", and changed the code so that the config file just had
and actually append the "input" and "output" in code when needed (fairly elegantly leveraging some existing keys that already defined those two strings).
The problem was that when I developed the script the actual consumer hadn't been finalized, so that output folder path was just a placeholder. When it comes time to deploy we get the actual path which is now some NFS thing like
/mnt/other-services-host/other-service/input/
At this point I naively go to the config to update my $output_folder variable and discover the code changes made by the other dev, which have made it impossible to separate the two folders, and because of the "elegant reuse of existing keys" made it a huge pain to even change the code back, since the assumption that the last segment of the path had to match the intended use was deeply baked in. At this point I think I just started swearing for a week.
One might contemplate the question of why people use config files in some cases and command-line parameters in others. Having a section for constants at the beginning of a script resembles a config file to some extent.
Implicit assumption that other people's code is off-limits?
Most software on my machine has probably arrived from apt-get at some point or another. Even if it's all Perl scripts, I'm not going to overwrite them directly, only to have my changes either removed on update, or blowing up the package manager's consistency checksums, or [insert random reason I can't conceive, because I'm not familiar with internals of apt]. So it's either config files, command line, environmental variables, or I'm going to build a wrapper that bends installed software to my will.
Those are not really abstractions, just extracted variables. An abstraction would change the concept language.
$suffix = "put";
This is also not an abstraction, as the language of the term "suffix" comes from its role in string concatenation, which is the same as the role of the original string literal. It doesn't change the "level of abstraction".
This isn't an over-abstraction, it's an over-extraction. Each abstraction should be non-trivial.
$project_folder = "/whatever";
is a good abstraction. Looking at the string literal "/whatever", I cannot determine it's "role", but $project_folder is a good name and changes the concept language from being about string concatenation or arbitrary names into a concept language about projects and folders.
Now you've just obfuscated that string for no apparent reason. Nobody can grep for that literal string anymore and you gained an extra line of code.
Those paths are not the same data repeated twice just because they share common substrings. They are two paths that serve distinct purposes. The developer likely chose that syntax because it looks like a setting that can be changed. It could just as well have been read from a settings file.
In the spirit of the original article I think point is not whether it was too much or not enough abstraction--it was the wrong abstraction. His abstraction eases one potential change (changing the project folder, which would imply changing two path strings in the original code but one in his), but at the same time makes a whole bunch of other potential changes much harder (or even impossible) to handle. We all would have been better off if he'd just left the duplication in place.
214 comments
[ 11.7 ms ] story [ 664 ms ] threadNot including cross-cutting concerns that modify all usages (eg, changing your logging or dependency injection library).
Duplication has it's own set of dangers leading down the road to a verbose mess of convoluted crap code in most cases.
Most established code bases make me want to puke before long.
With two copies of the code you can’t be sure if the similarities are factual or coincidental. At three the situation begins to crystallize quite rapidly.
https://en.m.wikipedia.org/wiki/Rule_of_three_(computer_prog...
What definitely does make things easier is simply having less code to fix. Although measuring the 'amount' of code is at least somewhat subjective.
I think there is always the problem that programmers want to apply all these principles, including DRY, without actually thinking about it. Once it's applied illogically you are in strange and awful territory.
You only find yourself at step 8 after a suite of bad decisions, and possibly even bad decisions that you signed off on during code review.
this is not to say that if you rollback the code and then commit tonfinding and implementing new abstractions you won't win - but the second part is necessary or you are just digging a deeper hole.
think of it as a wrapped transduction - younhave to do the second part as well.
NSTAAFL
On the one hand, often there can be shared lines of code without a shared idea, this shouldn't be a candidate for being factored out into some new abstraction.
On the other hand, you may want to introduce an abstraction and factor out code into a common library / dependency / framework when there's a shared well-defined concern/responsibility.
That said, on the gripping hand, I say may because even if there's the opportunity to introduce a clean future-proof abstraction, introducing it may be at the cost of coupling two things that were not previously coupled. If you've got very good automated test coverage and CI for the new common dependency and its impact upon the places it is consumed, then perhaps this is fine. If the new common dependency is consumed by different projects with different pressures for change / different rates of change / different levels of software quality then factoring out a common idea that then introduces coupling may cause more harm than good.
The real problem is when engineers abhor duplication, and in order to reuse existing code, they simply call the same code from multiple places without thinking through the basis for this reuse.
Mindless deduplication is not an act of abstraction at all! This is a very important point, because a "wrong" abstraction that is conceptually sound is not that hard to evolve, and if the code is called from N places then you get to look at those places to understand how to evolve the abstraction. Improvements to one part of the code benefit N parts, and you save work.
The only other factor to keep in mind is the dependency graph and coupling, as my parent mentions.
Mindless deduplication is more common than you'd think, especially with bits of code like utility functions and React components. For example, you end up with a component called Button that has no defined scope of responsibility; it's just used in one way or another in 17 different places that render something that looks or acts sort of like a button. This is not the "wrong abstraction," it is code reuse without abstraction.
An abstraction can be seen as a compression process, mapping multiple different pieces of constituent data to a single piece of abstract data; [1]
There are wrong abstractions.
[1] https://en.wikipedia.org/wiki/Abstraction#Compression
Quoting the start of the article: Abstraction in its main sense is a conceptual process where general rules and concepts are derived from the usage and classification of specific examples, literal ("real" or "concrete") signifiers, first principles, or other methods.
For the Button case, you'd have to come up with some concept of what a Button is and does, beyond what code lives in its file (e.g. an onClick handler that calls handleClick, etc.) in order to have an abstraction.
There are "wrong" abstractions (in the sense of abstractions that turn out to need to be changed later, like any code), but if you lump all deduplication into abstraction then you will have a skewed sense of the cost of changing an abstraction.
The cost of changing an abstraction also depends on your programming language; if you spend a lot of time in a dynamically-typed language, you may internalize that refactoring is tedious and error-prone and often intractable.
When systems need to use newer functionality, port them to the new components.
I've had a mixed experience with this, but at some point you get the API right, and then it works.
In the end, almost every conceptual way to slice up software can be viable if you are good at whatever you do, and terrible of not.
I think a good test for this is if you can write a reasonable unit test for the code in question. If your unit tests essentially become two separate sets of tests, testing the different branches of code, it's probably the wrong abstraction. If your tests work and you've built a reasonable standalone library (even if it's not useful to anything but your exact product), that's at least a signal the abstraction is sustainable.
Now we simply need a formal objective definition of "reasonable code" and the industry should never have this problem again.
A red flag is vague function names like "processThing" or "afterThingHappens". If a function can't be summarized concisely, it's probably doing too many things, and the abstraction is likely to break down later when the callers' needs diverge.
This reminds me of the recent post on HN about a company migrating from microservices back to a monolith, for this exact reason.
Just because something hasn’t been shared yet isn’t good justification in my opinion, if you know it will be, especially if it’s a library/API.
If I'm writing an API to move a robot, my problem space is fairly bounded, and I know that someday I will want force control at some end effector. I know that there's a 6 axis robot I've been eyeing, etc.
Maybe I'm being downvoted by web devs?
I'm all for not repeating myself, but there is a different between "usually avoid" and "never"
Copying and pasting in many situations would seem a breeze compared to the nest of abstractions required to avoid it.
Programmers have had DRY drummed into them so hard that it is almost heretical to even consider the tradeoff of increased coupling that arises from it. Coupling is good if things should change together because they are linked in some fundamental way such that it would be wrong for them to be different. Coupling is bad when things should evolve independently and the shared link is incidental.
The problem is that it is surprisingly hard to tell the difference up front. In the moment of writing the code, the evidence for the shared abstraction seems overwhelming, but the evidence of the cost of coupling is completely absent. It exists only in a hypothetical future. Unless there is strong evidence for a shared underlying conceptual link, I often consider only the 3rd repetition of a shared piece of code evidence for the existence of a true abstraction. Two times could just be chance, three is unlikely to be so.
This is actually representative of a problem in the industry as a whole I think. A lot of things have short term benefits but long term drawbacks. Because of the drastic, recent growth, orgs are bottom heavy (very few people have experienced long term drawbacks of X compared to how many people who just learnt X). Additionally, because of the extremely quick turnover of people, it's even rare that people who implement X are there when X blows up in people's face. They went on to implement Y...and will be gone before Y blows up.
So most tools, libraries, frameworks and abstractions are HEAVILY optimized for the short term. Optimized for getting a project set up quick. Optimized for the initial "Hello world". Optimized to get an API and a form in seconds. Very few tools/patterns are optimized for ease of long term (hell...these days long term means a year or two) maintenance. The ones that are generally get a bad rep.
And building stuff that's both good short AND long term is very, very hard.
It's usually the intermediate developers who like having rules to follow to know that they're doing well that tend to over-apply DRY and other principles. Only experience (aka pain over time) seems to show when to break (or just not apply) the rules.
Perhaps it's just the way things are taught/learned. Instead of just showing what's good and have them interpreted as rules, each should be shown as a rule of thumb with a concrete example of when it should not be applied. Even if they don't clearly understand the difference at the time, they'll always recall that there are exceptions and not feel so motivated to apply it in every instance.
As a senior engineer who recently became an engineering manager I always caution my devs about abstracting too liberally. Junior engineers are particularly bad about this. They see a handful of functions that are duplicated across a few (unrelated) projects and they want to create a new repo/library and share it. Then I direct them to the Slack channel for our platform services, which has a sizable shared library across dozens of services. That shared library is a frequent source of problems.
It takes a while, but I usually beat that primal impulse out of them.
He is what I think you do when you do that.
You just created yet another internal API. Designing, creating and _documenting_ good API's is _hard_. The most likely result is an undocumented dodgy half finished API that doesn't fully encapsulate the thing it's supposed to deal with. So you end up with code that both uses and bypasses the API you just wrote.
If you do that and later decide that you want to move some functionality from one side of the API to the other you've just set yourself up for a hella lot of work.
The other thing is, you want to make changes to duplicated code. You can limit the risk to code base you're actually working on and not a bunch of unrelated programs.
To help weed out requirements I tell people that on their third copy / paste, they might begin to consider reducing the duplication. At that point, they both have had time to think about the code and had gained experience with it to discover what the requirements are.
Another problem with bad code reuse is code locality. Like with Instruction and memory locality helping to improve runtime performance via caching, code locality helps improve mental processing. The further you separate related pieces of code, the more you need to have a good abstraction for it so you can correctly reason about the code. Without a good abstraction, you are having to jump between far areas of code to figure out what your function does.
Granted we were all n00bs and nobody will see that code again so it wasn't a big deal... but the intent, direction, and possible future of the code seems like something that should be considered once you start sharing.
Yes. Dare I say, intent is one of the most important things here. Two new pieces of code may be structurally identical, and yet represent completely different concepts. Such code should not be DRYed. Not just because there's a high chance you'll have to split the abstraction out later on - but also because any time two operations use a common piece of code, there's an implicit meaning conveyed here, that the common piece of code represents a concept that's shared by the two operations.
I apply this to DRY, coupling, encapsulation, APIs, etc. Also, I prefer to focus on consistency and readability over most other concerns. I mentally, or physically!, note areas of code I want to improve but don't feel like the time is right now, right now. During future work that touches that code I will refactor it if a solution has presented itself.
These days I prefer languages with bomber language services and tooling to make refactor in the future as painless as possible(types). I prefer explicit over implicit(sorry Ruby and Chef), and configuration over convention(looking at you Gradle).
Definitely a lot harder to fix when you've done this though.
I think "the wrong abstraction" is too lofty a title this is just about oversize functions.
[1] https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation...
I find that a particularly good way of learning (at least over plain reading/coding).
This can be tricky if the person who wrote the abstraction takes it the wrong way. At my previous company, I've been yelled at for doing this. Some developers get emotional about their code, in which case undoing their abstraction causes offense. How do you get around this?
Most of the time you will either be welcomed for your proper obsequence, or find out that there's a jealous guardian of that code and no matter what you do it won't alleviate that tension anyway.
[1] https://go-proverbs.github.io/
It will become obvious which duplicated code to abstract when you find yourself changing all/many at the same time or fearing you’ll forget/break something if you don’t change all instances. Writing tests is also a good motivator as it means even more code per duplication (and reduces the fear of breaking something).
It really takes a lot of duplication for this to get out of hand. Wait til it happens, you’re a software engineer you’ll figure out how to get rid of duplicated code just fine. Coming up with a great abstraction is extremely difficult before seeing at least a few examples.
It's much more likely that the code in question is responsible for one particular thing, and switches between several different ways to achieve some sub-goal. Those parts should be lifted into some kind of interface, where the different variations are lifted into different implementations of that interface. A lambda function is the most general interface possible, so it's probably not the best choice, you'd eventually end up with callbacks calling each other without it being entirely clear which callback does what.
This is what lambdas are for. Going to interfaces just adds needless syntax sugar.
A typed lambda function (i.e., with a defined arguments + return signature) is exactly as specific as any other typed interface, an arbitrary lambda function isn't, sure, but there's few languages where a static interface and an arbitrary lambda function are both available tools.
I now try to find the middle ground by remembering to "do the simple thing" even if it appears less elegant. This makes it easier to refactor in the future (if required) at which point more information will be available to design a more appropriate abstraction than would have been possible before.
So now he’s indispensable, a situation I assiduously avoid (you can’t work on the cool stuff when you have to be there to maintain the old thing).
- Programmers may look at a bug in a simple shared function and conclude that it “obviously” should be fixed, and do so quickly without really understanding what else could go wrong. (As a completely contrived example: You “fix” something that previously couldn’t return a negative value, and move on; turns out this “fix” allows a bug somewhere else to crop up, a catastrophic improper cast from signed to unsigned, blowing up your -1 into an iteration over billions of expensive operations.)
- Bug priority levels vary between features, even if code is shared. Your abstraction may make it effectively impossible to fix just one high-priority feature, if your deployment is (say) set up to run hours or days of regression tests on all affected parts. Generally, the more segregated things actually are, the easier it is to set priorities well.
Just because something is duplicated doesn’t mean that it’s that way forever, either. At a good branch point, such as a new project, you can aggressively prune out things that won’t apply to that branch even if they helped keep things stable on the previous project.
“Generic” functions make it difficult to find all uses or even understand what scenarios they belong to. With bland say-nothing nouns and meaningless verbs like execute() or process() that appear everywhere in the code, you’re just crossing your fingers and hoping for the best.
In his words: we are a solving problems, we are resolving paradoxes.
I've always found the easiest way to refactor is to get really good code coverage on the outermost layer of code that uses the abstraction, remove/ignore unit tests if there are any on the abstraction itself, then refactor the abstraction with vigor until it feels appropriate and hopefully elegant or is removed if necessary. As long as all your tests still pass, you should be good to go.
The problem was that when I developed the script the actual consumer hadn't been finalized, so that output folder path was just a placeholder. When it comes time to deploy we get the actual path which is now some NFS thing like
At this point I naively go to the config to update my $output_folder variable and discover the code changes made by the other dev, which have made it impossible to separate the two folders, and because of the "elegant reuse of existing keys" made it a huge pain to even change the code back, since the assumption that the last segment of the path had to match the intended use was deeply baked in. At this point I think I just started swearing for a week.Most software on my machine has probably arrived from apt-get at some point or another. Even if it's all Perl scripts, I'm not going to overwrite them directly, only to have my changes either removed on update, or blowing up the package manager's consistency checksums, or [insert random reason I can't conceive, because I'm not familiar with internals of apt]. So it's either config files, command line, environmental variables, or I'm going to build a wrapper that bends installed software to my will.
This isn't an over-abstraction, it's an over-extraction. Each abstraction should be non-trivial.
is a good abstraction. Looking at the string literal "/whatever", I cannot determine it's "role", but $project_folder is a good name and changes the concept language from being about string concatenation or arbitrary names into a concept language about projects and folders.Those paths are not the same data repeated twice just because they share common substrings. They are two paths that serve distinct purposes. The developer likely chose that syntax because it looks like a setting that can be changed. It could just as well have been read from a settings file.