Good write-up. I've been observing this phenomenon for years and I think that one of the main causes is poor cost estimation by developers (or, often, no cost estimation).
The new project lead asks management to allow him to re-write the project. It's a 10 manyear task so the management declines.
At that point he decides to do the changes in gradual manner, completely ignoring the fact that it's still a 10 manyear task, while, realistically, his team can spend 2 manmonths a year on refactoring. Thus, the re-write will be accomplished in 60 years.
The same pattern repeats with the next project lead and so on.
There's that wonderful point in a project where you can point--point directly at!--a fix to a fix to make legacy thing work. I love that point. That's the point I start looking for new work and resenting my forerunners.
"second system" syndrome is very real, but nobody in management ever seems to appreciate "Look, technology has progressed in the last five years since this was written...we can do more with less, pleeeeaaaase let us fix this.".
Nobody looks forward to working with legacy code. The business logic is usually wrong, the business itself has changed, the original developers are gone, the market talent composition has changed, etc.
I'd honestly rather see a business embrace the ablative nature of code and instead build itself with the assumption that, every 2-3 years, chunks will be rewritten. This is an argument in favor of services and better documentation and tests, by the way.
EDIT:
And even worse is when the manager/CTO/whatever admits that the code is going to need rewriting--but not yet. Because that means that a) you're going to be wasting effort temporarily fixing something you know will be thrown out and b) tacitly admitting that they are too scared of change to embrace it and too short-sighted to figure out how to balance business requirements with technical debt.
I've been burned by that multiple times.
A man-month of unpaid overtime for rewrites, not a nano-second for legacy support!
>I'd honestly rather see a business embrace the ablative nature of code and instead build itself with the assumption that, every 2-3 years, chunks will be rewritten
>And even worse is [...] you're going to be wasting effort temporarily fixing something you know will be thrown out
Can you reconcile these two points for me? I fear I may be misunderstanding you. It sounds like you're arguing in favor of treating code as a transient thing, but then lamenting having to write something that won't last.
The subtle difference (not well articulated by me, I suppose) is that in one case, you are given the go ahead to go and rewrite everything to conform to, say, an interface spec or API or whatever. This year's implementation or architecture may be scrapped, but then you get a cleanish slate.
In the latter case, what's happening is you are usually building an adapter over something that exists, instead of throwing it away completely. What then happens is that there is a lot of stress involved ("Is this wrapping correctly?", "Does this still do what it used to?", "Have my tests covered all the use cases?", "Wait, fuck, this original thing never even had tests...so aren't my tests are kind of pissing in the wind?", etc.) and then inevitably the push is to fix the fixes, and then fix the fixes' fixes, and so on.
And sometimes that's unavoidable (maybe the legacy code is, oh I don't know, numerical simulation of a heat-exchanger in Fortan77 that was cutting edge and now the original author is quite dead and buried). But, the painful thing (as mentioned) is when the person in charge even admits that the whole thing will be rewritten anyways.
In that case, why the fuck is the rewrite being put off? It signals a lack of faith in the development team, and a lack of prior planning to deal with downtime or other issues coming up.
Unfortunately your otherwise correct and very healthy mentality on work/pay ethics is only fashionable in US + Canada + Australia + parts of Europe. The whole rest of the world (that happens to include more than 60% of the planet's population btw...) has very different views on things. And considering that return on capital grows much faster than return on labor, things are not improving worldwide, actually the reverse...
And keep in the mind that software industry tends to be on the "fair" side of the fence regarding work/pay ethics everywhere in the world... all other professions are much worse.
And sticking to the subject: lots of programmers I know would unfortunately choose the unpaid overtime instead of the working with legacy code, and, unfortunately, I've even witnessed discussions where the programmers were more or less directly offering to put in the unpaid/unclocked overtime just to convince the management to let them do the fucking rewrite from scratch!
...the world in general is much crazier than you think, and you "crazy talk" can very easily turn into "they way we usually do things down here", unfortunately.
> Look at Stack-Overflow ... they also use only static methods for performance, we should do the same ... Gordy had dismissed Ina’s love of unit testing. It’s hard to unit test code written mostly with static methods.
I also happen to like static methods quite a bit, precisely because I thought they were EASY to unit test. I thought that a static method, which is essentially just a function in the enclosing object's namespace, would be EASY to unit test. Input -> output. No state.
Am I digging my own code grave here? What am I missing?
Additionally, if you need to mock that doSomething() method's behavior at all, you can't just subclass and override the method. You would need to bring in byte code voodoo (using e.g. PowerMock in Java).
It can be a useful tool, but it causes all kinds of weirdness which can completely mitigate any advantages. On my current project I had to strip out most of our uses of PowerMock. While the tests pass on my local machine just fine, many of them fail for no good reason on the CI server.
Our CI servers are already virtual machines. Or did you mean my local machine? We have a standard ecosystem that everyone uses, but there must be some difference between the CI server and our ecosystem. I simply haven't had the time to figure out what or why yet and it's better/easier to simply remove PowerMock where the tests are broken.
> there must be some difference between the CI server and our ecosystem
Well that's why I suggested using Docker: To ensure both environments are exactly the same.
You say they are virtual machines, but how do you provision them? I assume you are using puppet/chef/salt/ansible and creating them from scratch, as opposed to uploading whatever image you used in dev, like you would with Docker.
If you can avoid it, do; PowerMock is basically only for tests that interact with badly designed code that you can't change. But sometimes you need to depend on a badly designed library, in which case having a somewhat cumbersome way to mock it is better than nothing.
"PowerMock is basically only for tests that interact with badly designed code that you can't change"
Presumably where the definition of badly designed includes "uses static methods"? That's something that I hold true, and teach to other people as true, but I have started wondering about it. Whenever I ask why it's bad I get told "it makes testing hard". Well, Powermock can make testing it possible - "but powermock is only for badly designed code". What's badly designed code? And so it goes round.
Is it really that conceptually bad? Or is it just that our testing tools are insufficient, or that the language itself did not make provisions for testing all of its features? I've seen a lot of code recently that uses dependency injection purely for testing, and uses builders instead of constructors all over the place, and it feels like a lot of effort and complexity to avoid using PowerMock.
It's not just about testing, though that's where you tend to run into it first. Statics and constructors are both second-class citizens in Java: they can't conform to interfaces, so there's no way to separate interface and implementation when using them, no way to hide information, no way to invert control or any of the other principles of good design.
(Of course, Java was designed by people doing the best they could subject to particular constraints, and is now constrained by backwards compatibility. But other languages provide the same kind of features in better ways (or provide syntax that means builders and dependency injection are much less effort, depending on which way you look at it))
Pragmatically, good design is not the only concern; if you're not writing a library with binary-compatibility concerns, then IMO the syntactic overhead of builders is often a cost that outweighs the benefits. But the design costs of statics and constructors in Java are real enough, and there are very direct costs to using PowerMock (tests are harder to maintain and more brittle as they have to specify which classes they're rewriting, and it can conflict with other things that do bytecode manipulation e.g. test coverage tools).
Fake, not Mock (if you need to Mock, you've alrrady lost), but yes, you shouldn't write anything static if you'll need to provide an alterate implementation of the interface.
Not really, because db is not static. In my example, that Database.write() method would itself be static, and thus you can't pass it in as an argument.
Basically, this concept means that it's static all the way down, from the top layer to the bottom layer.
You put it well, and most people in this thread (including GP) seem in violent agreement.
In short: static methods are easy to test, methods using static methods (whether static or not) are more difficult to test.
My takeaway: the bar for moving functionality into non-static methods (which will still be decently easy to test) is quite low. If you call a static method, be prepared to own all its functionality when testing the calling method. Taken to logical conclusion, public static methods should be quite small.
Unfortunately some Java developers don't distinguish between stateful and stateless static methods and just say all static methods are bad. (But note that if the method does I/O it's stateful; you generally want I/O calls to be replaceable.)
The problem is that any code which calls a static method is hard to test. Because the thing that is static about static methods - the reason they are called 'static' methods - is that they are static-bound. Code which calls a static method, will be compiled with a hard, static dependency on that static method. You can't mock it, stub it, or dependency inject it. It's... static.
I think this is ultimately caused by developers not quite knowing where and how to place their ambition. It's easy to look at the big legacy app and be like, "I can fix that!" Then they roll up their shoulders and go to work, management be damned. Management can't see or understand what he does, and they're paying him for his expertise, so they give him the benefit of the doubt and let him do what he thinks he needs to do.
I documented my own answer to this situation here. Basically, don't treat organizational problems as technical ones. They're paying you to do a job, everything else you do is basically an ambition. First rule of ambition is to do no harm.
If you're going to refactor, don't refactor in the direction of new. Refactor in the direction of coherency. Do things that you can finish, or, that at least, don't matter that much if you don't. Like adding tests. Nothing wrong with that.
If you want to fix the legacy app, you need to make less ambitious changes, because you need to be able to finish them. Instead of changing the data types, refactor the initialization code. The initialization code runs once, data manipulation code is basically convention and you need to change it everywhere at once or not at all. Refactor it to the point where you can change it everywhere at once and then do that. Or get really good with a text editor and macro the changes.
Giles Bowkett has a great book on it. Unfuck A Monorail For Great Justice. It's Rails-specific but the ideas can work everywhere. Write code that understands code.
But understand that there are probably better things to do with your time. You can pick out pieces of the legacy app and try to understand them well enough to rubber-duck debug them without having the source code in front of you. Once you have that then really, you don't need to be a hero. You're already a hero because you know the system better than anyone. You can focus on other things. Maybe use your new-found credibility to build some consensus among your peers and managers. Then when you make your case to replace the app, it won't just be you.
I feel like this is very sound advice but i'm not sure if i can really accept it fully.
I'm a fairly green developer (3 years professionally) working on an C# application that suffers from design schizophrenia as detailed in the article. I'm more or less the sole developer assigned to it now.
Although I am new, I feel very strongly about my 'pride as an engineer' in my application of sound design, knowledge and rigor to the best of my abilities. So it discourages me greatly when i hear a lot of "Uhh, that was a long time ago...", "We had tough deadlines..", "Well developer 'X' and me disagreed about that..." from the former maintainers of the code.
I've taken to a rash practice of just tearing out tightly coupled, untestable, duplicated and poorly designed code by the roots. I feel like i am somehow passing a 'holier-than-thou' judgment on the work of my predecessors. But due to it's tight coupling i can't have much confidence about even seemingly simple changes since they always seem to come back with a wagon of defects. The new code i write surely isn't perfect and has it's own defects that come back to me, but i feel much better about the code in many ways and i am always open to criticism and discussion over my design choices. If anyone would ever give them serious study...
I have this gut feeling that if i leave the code i touch messy that i'm not doing my job and the next poor soul to come behind me will lose weekends to trying to clean up an even deeper mess. I feel judgmental even voicing these thoughts.
> I've taken to a rash practice of just tearing out tightly coupled, untestable, duplicated and poorly designed code by the roots.
That's not a terrible approach. Depending on the context the code is running in, that might be a worthwhile way to maintain it, especially if there's not all that much different between doing that and working on it as is. If it's taking you the same amount of time, why not leave it better than you found it? The important thing is that you finish the job, and it sounds like you're doing that. You're not making a three-year plan, not telling anyone about it, and doing a little bit every day. That way lies the lava layer.
Me personally, I have like four other projects I could be working on. Give me a choice between working on old legacy and working on new hotness, I'll band-aid the legacy every time and just get on with life. I didn't use to do it this way. I wanted to make my mark on the legacy codebase, prove that I was better than it. So I'd tell my boss I'd need the rest of the week to implement this and then dive in. I'm happy I don't need to do that anymore.
> I wanted to make my mark on the legacy codebase, prove that I was better than it. So I'd tell my boss I'd need the rest of the week to implement this and then dive in.
I think this statement sums up my attitude almost perfectly. I have a notion that if I channel that line of thinking into positive structural changes in the code then I should while I can. I just don't know whether or not it is a good attitude to have.
My own experience is that it's easy for that to become either a source of a lot of regressions or an infinite death march. It's key to thoroughly understand the part to replace before getting involved in that kind of wholesale replacements.
The amount of tests available on the previous implementation can go a long way towards lighting the way and bounding the level of required efforts. Code that is deeply embedded, untested and "ugly" is just a recipe for pain however.
I need to work harder at mapping out our code to understand it as you say. I keep telling myself every codebase is different and none are perfect, but it is so nice to work in fresh or at least clean code.
My project is about 200KLOC C# 2.0; we have a single .cs file 13,000+ lines long and no, it's not generated. We have a single method 900+ lines long in our DAL containing 23 separate hand coded SQL queries and no, it is not a report. Our codebase contains exactly 14 unit tests. 10 of which were written by me. Your prayers are welcome.
Haha, nope. B2B software in the hospitality industry. It's both comforting and sad that my codebase's ugly stats might be shared by other applications out there.
What you're talking about is a bad implementation of a design stemming from its inherited technology. Essentially, attempting to refactor the technology or the design first is a bad idea - fix the implementation.
There's a book you really ought to read then. It's called "Brownfields Development" and it's sadly out of print. However, there are some free PDFs that might help you here:
> I've taken to a rash practice of just tearing out tightly coupled, untestable, duplicated and poorly designed code by the roots.
Hello me, from a parallel universe!
You've basically described the project I'm currently assigned to and the situation I'm in as well, and I share your enthusiasm for "weeding" out obvious turds and warts from the codebase.
Unfortunately I've reached a new kind of roadblock when it comes to improving and refactoring the project: I've picked all the low-hanging fruit, and now if I tug at any of the remaining lose ends the whole thing will unravel.
Basically the remaining pain points are architectural ones (that will hurt us at scale) and that require attention but at the same time are large enough that management won't approve the necessary resources to work on them.
Honestly I almost feel as if I've fucked myself over by making those improvements: the project is now mediocre as a whole, but just good enough so that nobody really cares about improvements any more. Had I just let the ship sink, maybe something better would have come out of this whole mess in the end.
> If you're going to refactor, don't refactor in the direction of new. Refactor in the direction of coherency.
1000x times this. And making things coherent will make it a lot easier to put the new hotness in too. Add in the new hotness only when you can do it in a coherent way. If you can't do it in a coherent way, you might need to refactor until you can get there.
You might never get there in a legacy codebase, but if everyone took this attitude from the start, you might be less likely to end up there.
You can put new things in and change architecture, but you always have to do so coherently -- which does mean you can put in fewer new things, and sometimes not the new things you want to put in as soon as you want to put them in.
And oh, yes, so very much: "don't treat organizational problems as technical ones." Without understanding the organizational disfunction at the root of your technical issues (it's almost always there), you're doomed. Of course, then you've got to figure out what to do about the organizational problems (or how to operate succesfully in their shadow), which is not what you find the most fun part of your job, but that's how it is.
While the example shows five different people bringing in four different ways of doing things, I was sad to recognize a lot of my own work in that story. I've been the sole developer on a smallish web site for five years.
I've personally switched patterns over the years when adding features, always telling myself that I'll go back and refactor the earlier work.
I 95% agree with this article, but think that 5% is an important disagreement.
Background: I spent most of the last decade writing, maintaining, and managing a large enterprise app that had to evolve from its .NET 1.1 foundations to the present day. Our code base looked almost precisely like the one talked about in that article, and with the very real problems mentioned there.
But...
1. Sometimes the productivity wins of a new approach are serious enough that it's worth taking the consistency hit. I'd argue that ORMs, for instance, really are worth it compared to the quasi-code-gen ADO.NET layer. Switching to those midway through the project introduced inconsistency, but all future development done with the ORM was much faster (and more reliable) than the stuff done with the old approach.
(And yes, this is a dangerous argument, because it's easy to convince yourself that it's true all the time, if you want an excuse to use a new technology. But if applied skeptically, it really is true sometimes.)
2. Developer morale is a thing. "You can only use technology invented in 2003, none of this fancy .NET 2.0 stuff, and haha LINQ, are you kidding me?" is going to crush people's spirits, whereas "sure, let's try this new module out using ASP.NET MVC" will introduce an inconsistency into the codebase, but will keep developers more engaged and learning new things (and who knows, maybe this experiment will prove to be one of those ones that gives you a big productivity win).
You have to be careful with that, because you can't literally try out every crazy new thing, but you also can't ignore everything new in the name of consistency and expect anyone to want to work on the project.
True, but the ones who are worth their salt know how to take advantage of new libraries and approaches, and how to structure their code so that it is easy to port and maintain.
A language is not the only part of the technology stack. There's worlds of difference between a JEE application and a Play application, even if both are written in Java.
> you also can't ignore everything new in the name of consistency and expect anyone to want to work on the project.
The strategy will cause a variant of the Cobol effect. You'll either have to pay developers obscene wages to put up with your consistent but antiquated codebase, or try to find developers who are willing to put up with it for normal wages. The latter probably aren't the people you want on your team.
The flipside to this is the older technologies start to become end-of-lifed or unsupported by the vendor (like .NET Remoting or .NET 1.1). In that case, there seems to be some level of continual need to push the application forward, but of course we want to avoid the kind of inconsistent layering described here. It's a tough problem.
The easy answer of "regular rewrites every 5-7 years" doesn't sit well with paying customers or management.
This is seen in the infrastructure side at any non-startup company. Let me give you an example of ACLs management.
1. First they get devices from one vendor. They set up ACLs in certain way: hardcoded IP addreses/ranges, hardcoded ports.
2. Then come devices from another vendor. This addes another layer.
3. Another vendor comes with an acl automation tool and tells team to create objects/references for ip addresses/ranges/ports. Some guys follow it; some thers don't.
4. You can see a big mess of things created by many vendors, many software automation tools, manual additions, etc.
One day, some guy changes the object/reference; all of sudden, production outage: add another layer of manual edits on the top.
That's why they preach "if it aint broke, dont fix it, but add another layer on top of it".
I think projects displaying the pattern often have a root problem that the writeup didn't discuss: An initial application with requirements created by non-technical folks who had it implemented by whatever the equivalent is of outsourced workers/students/interns/guy they hired without knowing how to hire a developer.
The initial developer doesn't understand, know, or follow basic CS knowledge or design trade trade offs and constraints, but will try to create a reasonable system using whatever has press at the time - Rails, Php + Zend, etc.
Unfortunately they will follow weird aspects of tutorials they googled for studiously while missing out on the design implication of their choices leading to monstrosities like a product display page that requires hundreds of recursively made SQL queries, an entire application where 90% of the code is in one file, database tables that are composed in the most awful ways, and more.
It's this first layer that causes everything that comes after. The root of the problem isn't that trying something new is right or wrong. It's that the original was done so badly nobody can reasonably continue down the same path and everyone who comes after can't figure out how to cope.
Of course it's typically exacerbated by deadlines. When you can add a new dependency and meet your deadline or try to do things the inscrutably bad way they were done previously and miss it (let alone have any ability to test it), it's hardly surprising that people choose to add what they know.
When confronted with something intractable of course you throw your favorite spaghetti at the wall. And of course there's not any sort of order to what sticks.
So I think the lesson is more about the criticalness of doing things remotely right to begin with.
stop blaming non-developers for the problems created by developers.
It's the result of developer who don't have the maturity to consider the long term risk of their decisions. That has nothing to do with the business leaders.
I freelance which means I see a lot of shit. In any given week I can touch/work with 10+ tech stacks across both Windows and *nix. When you do that on a regular basis it gives you a different perspective on things.
In particular, you write/design code in the same vein as the code around it. It doesn't matter what your personal values as a developer are. Consistency is more important than that. I have been known to pull people to the side and ask them "I'm about to do X, how do you normally do X in codebase Y? I want to do it exactly the way you do".
If you're going to change the way something like a DAL accesses its data you better be able to either do all of it, or you do none of it for exactly the reasons the article stated.
That is purely a technical decision, and the blame for that lies with every single developer who chose to introduce a new tech into the stack for no other reason than sensibilities.
Silly question - if we aren't to blame those poor put-upon business folks who hired the incapable, immature developer, then why is the developer accountable?
Honestly, we can't really blame the developers either. If you inherit a compilation of different techniques, there isn't much you could do to it. There isn't necessarily a code base that you could build on as the same thing is already been done in a number of different ways. So any way you're going to make changes there isn't going to make the code much better and, thus, the least worst choice would be to start gradually injecting some consistency into the codebase even if that's the fifth layer on top of the previous ones.
Maybe, just possibly, the second developer could at least theoretically continue exactly from where the first developer left off (a single person or team == a single codebase that's at least somewhat consistent) but it could often make the mess even worse if the first developer really didn't have much of a clue of what he was doing.
The managers are basically the customer who ordered the software and they can't be trusted to know anything. It's a very common case that the customer doesn't know what they need and it's the job of the developers to first figure out the customer's real problem before beginning prototyping.
Thats fine for the initial stab at the problem. But then someone who does know what he is doing come along and recommends a module be rewritten and management says no, because "it's working".
I've used this same argument for why "show me your github" isn't necessarily the best way to judge a developer.
My group uses github for all of our code and while it's private that might change in the future. If that happens, there are a lot of places where the answer to "why did you do X this way?" is "that's the way they do it" and not that I agreed with it. That's potentially more harm than good if someone was looking at it.
It's not "CS knowledge" that causes tech fashion to change over time. CS knowledge hasn't actually changed very much in the last 30 years. The people that designed SQL knew all about the advantages and tradeoffs of "NoSQL" key-value stores. The impedance mismatch of ORM Object-SQL systems has been studied for decades.
If anything changes it's the engineering "best practices", like unit testing methodologies. But mostly it's just fashion, and who has the biggest platforms and marketing budgets for pushing their framework.
Software development as a field sucks, no one really knows what they're doing. We're all just figuring it out as we go along.
As he says in the article: "We think that the current shiny best way is the end of history; that it will never be superseded or seen to be suspect with hindsight."
That doesn't mean that the knowledge wasn't there, just that there were different tradeoffs being made back then.
I disagree w/ the GP that it's purely about fashion, but certainly there distinct pendulums which swing back and forth in our field. IMO that pendulum isn't purely driven by fancy alone (i.e. fashion) but also changes in computing power.
For instance the ever driving swing of the local driven/server driven architecture. These days it's called "cloud" but it's gone by other names in the past. At some point the demands of the applications demand too much for the network traffic/network storage/etc and an increasing amount of work is offloaded to the local client. And then, at some point even further in the future there's an advance on the capabilities of remote computing (e.g. network bandwith) and the cycle repeats itself.
Throw away the first prototype version of a software and rewire v1.0 from scratch that you can use as a solid base for future development (for any application that is expected to have a long life - think years, not months) even if it's awesome for the users and works business-wise.
And always write a prototype/MVP first, and plan to throw it away, even if you end up using it in production and it ends up crucial for you business. Yes, you should try and build the MVP in weeks, not months, and you should already have its successor done when you're out of beta... but regardless of this, the pancakes rule will apply: the first one is always awful, and if try to stack the next ones on top of it, the whole pancakes tower will crumble. So throw away the first pancake and start from the second!
I think that's just one kind of initial nucleation point for the future crystals of failure.
Looking at the 10-year-old PHP codebase we have at work, the real cause is the organizational fluid around the project, a lack of accountability/ownership.
Nobody on the business-die can confidently say what ought to be current practice, so the system becomes--via circular logic--its own reference-implementation
Since no end-user can independently state "How We Do Things Here" except by saying "Whatever The System Already Does", that channels the software's future evolutionary paths: You cannot make "common sense" fixes, because the bug might be a feature.
I've seen this happen so many times, it feels like the norm. I guess it's inevitable though, it's more interesting to work on something "new" and "cool" than adjust your mindset to year old design decisions.
Would be interesting to know if there is a relationship between years of experience in programming and how likely you are going to inject the latest "cool" design pattern / framework into code. Personally I have become far more cautious about this stuff. I'd rather adapt what is already present than subject anyone else to an abstract monstrosity I just cooked up after reading the latest hacker news post on the latest hype.
I think it's more a matter of as programmer mature/learn, they overly disparage old code and idioms. It's not the new orthodoxy...
I've seen in with RPC calls (hand-coded HTTP -> Hessian -> Protobufs), serialization languages (XML -> Json -> Protobufs/Bson), ad nauseam.
I think this kind of changes is the natural state of multi-year development efforts. On a 10-15 years old code base, you'll see all sorts of things like that cropping up and it's incredibly hard to steer clear of these kinds of things.
What are the alternatives? Wholesale replacement of all uses? Wasteful in resources! Not introducing new technologies? Offends the sensibilities of the devs...
A more realistic way to fight this is to make Lava layer programs less harmful by keeping documentation and testing efforts consistent throughout the project (and back filling when previous developers didn't).
The constant narrative in the example is developers having no idea why and where things are built a certain way. With more information and verification, gradual changes and refactors can be more manageable.
Avoiding Lava layer applications is not feasible, so calling it an anti-pattern is silly.
Hmm, this has been true for some projects I worked on. When there is a stable dev lead for many many years. He needs to teach others his approach. Then it's perfect. People tend to keep things that work well. Just don't hire those who want changes:)
The best project I've ever worked on was like this. I don't think it's an anti-pattern; if anything I'd see it as a positive sign, as it indicates a project that's under continued development and willing to investigate new technology.
Just as it's vital to accept that business requirements will change during a project, and structure your methodology to accommodate this, the same is true of technical considerations. And the same solution applies, viz. Agile: break up any "redesign" into small steps that can deliver concrete benefits inside a two-week period. Don't treat the "transitional architecture" as a temporary thing that doesn't need to be good, doesn't need documenting etc.; like with a big database migration, you make sure that at every stage it's something that you can develop under, and you know what you'd do if you had to abandon the plan at any stage.
If you try to keep a codebase on 2003 tech forever, you're storing up trouble for later; devs will become harder to find and more expensive to hire, and eventually you'll be forced into a Big Rewrite, and we know how those turn out. Like with deployment, better to embrace Continuous Re-Architecting. If it hurts, you're not doing it often enough.
I think this kind of pattern was a big part of why I enjoyed it. Because it was an 8-year-old, 500-kloc codebase, everyone knew you couldn't change it instantly; you could see an archaeological history in the layering, and understood that in all probability the style that was great today would become the bad old code in three or four years' time. But the result was that it was a living system, we knew we were working on it because it made us money, and every day we were making the code a little bit better (even though that's a never-ending process), which was hugely satisfying.
I've seen incremental changes really well. But in those cases there were some important forces at play. Firstly, there was always at least one person advocating for the change. They could make improvements over time, educate other engineers, and react to feedback. Secondly, changes won out on merit. Engineers used new abstractions because they were better, not because they were new. Sometimes abstractions didn't work. And were expunged from the codebase.
I have experienced this personally in a couple of companies I worked for - Just like mentioned in the article, they had high staff turnover. What happens is that nobody truly understands the codebase and it requires many more employees to maintain. To make matters worse, it's also a self-perpetuating cycle. It's more a management issue than an engineering issue.
There isn;t and will likely never be any strategy to prevent the conditions that the author discusses. Software is biological in nature, and to take it further, subject to the same laws as everything else, in particular entropy. There is no "Maxwell's Demon" approach to writing software. There are best practices and mitigation strategies, but those are incompatible with the business of writing software.
Further, and I think this should be more and more clear to people, _any_ retained unencrypted data is _compromised_ data.
Unfortunately, our society is on the way to being completely dependent upon centralized systems and institutions which are unaware of their limitations, and in fact we live in an age where centralization is considered a virtue, when it is instead a significant liability.
When working with a poorly-written or unstable legacy app, there is a middle ground approach. Rather than rewrite the whole thing (often impossible), or introduce a new framework in the app, you can draw a circle around a core subsystem and rewrite just that component as a microservice. This frees you from the (often) monolithic app, gives you the freedom to pick an entirely different language, if you want, and leaves you with a part of the app that any new developer could completely wrap their head around.
79 comments
[ 2.5 ms ] story [ 131 ms ] threadThe new project lead asks management to allow him to re-write the project. It's a 10 manyear task so the management declines.
At that point he decides to do the changes in gradual manner, completely ignoring the fact that it's still a 10 manyear task, while, realistically, his team can spend 2 manmonths a year on refactoring. Thus, the re-write will be accomplished in 60 years.
The same pattern repeats with the next project lead and so on.
"second system" syndrome is very real, but nobody in management ever seems to appreciate "Look, technology has progressed in the last five years since this was written...we can do more with less, pleeeeaaaase let us fix this.".
Nobody looks forward to working with legacy code. The business logic is usually wrong, the business itself has changed, the original developers are gone, the market talent composition has changed, etc.
I'd honestly rather see a business embrace the ablative nature of code and instead build itself with the assumption that, every 2-3 years, chunks will be rewritten. This is an argument in favor of services and better documentation and tests, by the way.
EDIT:
And even worse is when the manager/CTO/whatever admits that the code is going to need rewriting--but not yet. Because that means that a) you're going to be wasting effort temporarily fixing something you know will be thrown out and b) tacitly admitting that they are too scared of change to embrace it and too short-sighted to figure out how to balance business requirements with technical debt.
I've been burned by that multiple times.
A man-month of unpaid overtime for rewrites, not a nano-second for legacy support!
>And even worse is [...] you're going to be wasting effort temporarily fixing something you know will be thrown out
Can you reconcile these two points for me? I fear I may be misunderstanding you. It sounds like you're arguing in favor of treating code as a transient thing, but then lamenting having to write something that won't last.
The subtle difference (not well articulated by me, I suppose) is that in one case, you are given the go ahead to go and rewrite everything to conform to, say, an interface spec or API or whatever. This year's implementation or architecture may be scrapped, but then you get a cleanish slate.
In the latter case, what's happening is you are usually building an adapter over something that exists, instead of throwing it away completely. What then happens is that there is a lot of stress involved ("Is this wrapping correctly?", "Does this still do what it used to?", "Have my tests covered all the use cases?", "Wait, fuck, this original thing never even had tests...so aren't my tests are kind of pissing in the wind?", etc.) and then inevitably the push is to fix the fixes, and then fix the fixes' fixes, and so on.
And sometimes that's unavoidable (maybe the legacy code is, oh I don't know, numerical simulation of a heat-exchanger in Fortan77 that was cutting edge and now the original author is quite dead and buried). But, the painful thing (as mentioned) is when the person in charge even admits that the whole thing will be rewritten anyways.
In that case, why the fuck is the rewrite being put off? It signals a lack of faith in the development team, and a lack of prior planning to deal with downtime or other issues coming up.
And keep in the mind that software industry tends to be on the "fair" side of the fence regarding work/pay ethics everywhere in the world... all other professions are much worse.
And sticking to the subject: lots of programmers I know would unfortunately choose the unpaid overtime instead of the working with legacy code, and, unfortunately, I've even witnessed discussions where the programmers were more or less directly offering to put in the unpaid/unclocked overtime just to convince the management to let them do the fucking rewrite from scratch!
...the world in general is much crazier than you think, and you "crazy talk" can very easily turn into "they way we usually do things down here", unfortunately.
> Look at Stack-Overflow ... they also use only static methods for performance, we should do the same ... Gordy had dismissed Ina’s love of unit testing. It’s hard to unit test code written mostly with static methods.
I also happen to like static methods quite a bit, precisely because I thought they were EASY to unit test. I thought that a static method, which is essentially just a function in the enclosing object's namespace, would be EASY to unit test. Input -> output. No state.
Am I digging my own code grave here? What am I missing?
It can be a useful tool, but it causes all kinds of weirdness which can completely mitigate any advantages. On my current project I had to strip out most of our uses of PowerMock. While the tests pass on my local machine just fine, many of them fail for no good reason on the CI server.
Offtopic: Time for start using Docker? :p
Well that's why I suggested using Docker: To ensure both environments are exactly the same.
You say they are virtual machines, but how do you provision them? I assume you are using puppet/chef/salt/ansible and creating them from scratch, as opposed to uploading whatever image you used in dev, like you would with Docker.
Presumably where the definition of badly designed includes "uses static methods"? That's something that I hold true, and teach to other people as true, but I have started wondering about it. Whenever I ask why it's bad I get told "it makes testing hard". Well, Powermock can make testing it possible - "but powermock is only for badly designed code". What's badly designed code? And so it goes round.
Is it really that conceptually bad? Or is it just that our testing tools are insufficient, or that the language itself did not make provisions for testing all of its features? I've seen a lot of code recently that uses dependency injection purely for testing, and uses builders instead of constructors all over the place, and it feels like a lot of effort and complexity to avoid using PowerMock.
(Of course, Java was designed by people doing the best they could subject to particular constraints, and is now constrained by backwards compatibility. But other languages provide the same kind of features in better ways (or provide syntax that means builders and dependency injection are much less effort, depending on which way you look at it))
Pragmatically, good design is not the only concern; if you're not writing a library with binary-compatibility concerns, then IMO the syntactic overhead of builders is often a cost that outweighs the benefits. But the design costs of statics and constructors in Java are real enough, and there are very direct costs to using PowerMock (tests are harder to maintain and more brittle as they have to specify which classes they're rewriting, and it can conflict with other things that do bytecode manipulation e.g. test coverage tools).
[1] http://en.wikipedia.org/wiki/Dependency_injection
Basically, this concept means that it's static all the way down, from the top layer to the bottom layer.
In short: static methods are easy to test, methods using static methods (whether static or not) are more difficult to test.
My takeaway: the bar for moving functionality into non-static methods (which will still be decently easy to test) is quite low. If you call a static method, be prepared to own all its functionality when testing the calling method. Taken to logical conclusion, public static methods should be quite small.
I documented my own answer to this situation here. Basically, don't treat organizational problems as technical ones. They're paying you to do a job, everything else you do is basically an ambition. First rule of ambition is to do no harm.
https://news.ycombinator.com/item?id=8768524
If you're going to refactor, don't refactor in the direction of new. Refactor in the direction of coherency. Do things that you can finish, or, that at least, don't matter that much if you don't. Like adding tests. Nothing wrong with that.
If you want to fix the legacy app, you need to make less ambitious changes, because you need to be able to finish them. Instead of changing the data types, refactor the initialization code. The initialization code runs once, data manipulation code is basically convention and you need to change it everywhere at once or not at all. Refactor it to the point where you can change it everywhere at once and then do that. Or get really good with a text editor and macro the changes.
Giles Bowkett has a great book on it. Unfuck A Monorail For Great Justice. It's Rails-specific but the ideas can work everywhere. Write code that understands code.
http://gilesbowkett.blogspot.com/2013/03/new-ebook-unfuck-mo...
But understand that there are probably better things to do with your time. You can pick out pieces of the legacy app and try to understand them well enough to rubber-duck debug them without having the source code in front of you. Once you have that then really, you don't need to be a hero. You're already a hero because you know the system better than anyone. You can focus on other things. Maybe use your new-found credibility to build some consensus among your peers and managers. Then when you make your case to replace the app, it won't just be you.
This can be generalized: refactor the code to the point where what you need to next is easy to do.
I'm a fairly green developer (3 years professionally) working on an C# application that suffers from design schizophrenia as detailed in the article. I'm more or less the sole developer assigned to it now.
Although I am new, I feel very strongly about my 'pride as an engineer' in my application of sound design, knowledge and rigor to the best of my abilities. So it discourages me greatly when i hear a lot of "Uhh, that was a long time ago...", "We had tough deadlines..", "Well developer 'X' and me disagreed about that..." from the former maintainers of the code.
I've taken to a rash practice of just tearing out tightly coupled, untestable, duplicated and poorly designed code by the roots. I feel like i am somehow passing a 'holier-than-thou' judgment on the work of my predecessors. But due to it's tight coupling i can't have much confidence about even seemingly simple changes since they always seem to come back with a wagon of defects. The new code i write surely isn't perfect and has it's own defects that come back to me, but i feel much better about the code in many ways and i am always open to criticism and discussion over my design choices. If anyone would ever give them serious study...
I have this gut feeling that if i leave the code i touch messy that i'm not doing my job and the next poor soul to come behind me will lose weekends to trying to clean up an even deeper mess. I feel judgmental even voicing these thoughts.
That's not a terrible approach. Depending on the context the code is running in, that might be a worthwhile way to maintain it, especially if there's not all that much different between doing that and working on it as is. If it's taking you the same amount of time, why not leave it better than you found it? The important thing is that you finish the job, and it sounds like you're doing that. You're not making a three-year plan, not telling anyone about it, and doing a little bit every day. That way lies the lava layer.
Me personally, I have like four other projects I could be working on. Give me a choice between working on old legacy and working on new hotness, I'll band-aid the legacy every time and just get on with life. I didn't use to do it this way. I wanted to make my mark on the legacy codebase, prove that I was better than it. So I'd tell my boss I'd need the rest of the week to implement this and then dive in. I'm happy I don't need to do that anymore.
I think this statement sums up my attitude almost perfectly. I have a notion that if I channel that line of thinking into positive structural changes in the code then I should while I can. I just don't know whether or not it is a good attitude to have.
The amount of tests available on the previous implementation can go a long way towards lighting the way and bounding the level of required efforts. Code that is deeply embedded, untested and "ugly" is just a recipe for pain however.
My project is about 200KLOC C# 2.0; we have a single .cs file 13,000+ lines long and no, it's not generated. We have a single method 900+ lines long in our DAL containing 23 separate hand coded SQL queries and no, it is not a report. Our codebase contains exactly 14 unit tests. 10 of which were written by me. Your prayers are welcome.
http://www.manning.com/baley/
Hello me, from a parallel universe!
You've basically described the project I'm currently assigned to and the situation I'm in as well, and I share your enthusiasm for "weeding" out obvious turds and warts from the codebase.
Unfortunately I've reached a new kind of roadblock when it comes to improving and refactoring the project: I've picked all the low-hanging fruit, and now if I tug at any of the remaining lose ends the whole thing will unravel.
Basically the remaining pain points are architectural ones (that will hurt us at scale) and that require attention but at the same time are large enough that management won't approve the necessary resources to work on them.
Honestly I almost feel as if I've fucked myself over by making those improvements: the project is now mediocre as a whole, but just good enough so that nobody really cares about improvements any more. Had I just let the ship sink, maybe something better would have come out of this whole mess in the end.
Not sure what the lesson here really is! :|
1000x times this. And making things coherent will make it a lot easier to put the new hotness in too. Add in the new hotness only when you can do it in a coherent way. If you can't do it in a coherent way, you might need to refactor until you can get there.
You might never get there in a legacy codebase, but if everyone took this attitude from the start, you might be less likely to end up there.
You can put new things in and change architecture, but you always have to do so coherently -- which does mean you can put in fewer new things, and sometimes not the new things you want to put in as soon as you want to put them in.
And oh, yes, so very much: "don't treat organizational problems as technical ones." Without understanding the organizational disfunction at the root of your technical issues (it's almost always there), you're doomed. Of course, then you've got to figure out what to do about the organizational problems (or how to operate succesfully in their shadow), which is not what you find the most fun part of your job, but that's how it is.
I've personally switched patterns over the years when adding features, always telling myself that I'll go back and refactor the earlier work.
Background: I spent most of the last decade writing, maintaining, and managing a large enterprise app that had to evolve from its .NET 1.1 foundations to the present day. Our code base looked almost precisely like the one talked about in that article, and with the very real problems mentioned there.
But...
1. Sometimes the productivity wins of a new approach are serious enough that it's worth taking the consistency hit. I'd argue that ORMs, for instance, really are worth it compared to the quasi-code-gen ADO.NET layer. Switching to those midway through the project introduced inconsistency, but all future development done with the ORM was much faster (and more reliable) than the stuff done with the old approach.
(And yes, this is a dangerous argument, because it's easy to convince yourself that it's true all the time, if you want an excuse to use a new technology. But if applied skeptically, it really is true sometimes.)
2. Developer morale is a thing. "You can only use technology invented in 2003, none of this fancy .NET 2.0 stuff, and haha LINQ, are you kidding me?" is going to crush people's spirits, whereas "sure, let's try this new module out using ASP.NET MVC" will introduce an inconsistency into the codebase, but will keep developers more engaged and learning new things (and who knows, maybe this experiment will prove to be one of those ones that gives you a big productivity win).
You have to be careful with that, because you can't literally try out every crazy new thing, but you also can't ignore everything new in the name of consistency and expect anyone to want to work on the project.
Those devs are, naturally, in the minority.
But c99 is pretty cool (though as the sibling suggested, c89 is rather common). Anyway, 15 years old, just right.
The strategy will cause a variant of the Cobol effect. You'll either have to pay developers obscene wages to put up with your consistent but antiquated codebase, or try to find developers who are willing to put up with it for normal wages. The latter probably aren't the people you want on your team.
The easy answer of "regular rewrites every 5-7 years" doesn't sit well with paying customers or management.
1. First they get devices from one vendor. They set up ACLs in certain way: hardcoded IP addreses/ranges, hardcoded ports.
2. Then come devices from another vendor. This addes another layer.
3. Another vendor comes with an acl automation tool and tells team to create objects/references for ip addresses/ranges/ports. Some guys follow it; some thers don't.
4. You can see a big mess of things created by many vendors, many software automation tools, manual additions, etc.
One day, some guy changes the object/reference; all of sudden, production outage: add another layer of manual edits on the top.
That's why they preach "if it aint broke, dont fix it, but add another layer on top of it".
The initial developer doesn't understand, know, or follow basic CS knowledge or design trade trade offs and constraints, but will try to create a reasonable system using whatever has press at the time - Rails, Php + Zend, etc.
Unfortunately they will follow weird aspects of tutorials they googled for studiously while missing out on the design implication of their choices leading to monstrosities like a product display page that requires hundreds of recursively made SQL queries, an entire application where 90% of the code is in one file, database tables that are composed in the most awful ways, and more.
It's this first layer that causes everything that comes after. The root of the problem isn't that trying something new is right or wrong. It's that the original was done so badly nobody can reasonably continue down the same path and everyone who comes after can't figure out how to cope.
Of course it's typically exacerbated by deadlines. When you can add a new dependency and meet your deadline or try to do things the inscrutably bad way they were done previously and miss it (let alone have any ability to test it), it's hardly surprising that people choose to add what they know.
When confronted with something intractable of course you throw your favorite spaghetti at the wall. And of course there's not any sort of order to what sticks.
So I think the lesson is more about the criticalness of doing things remotely right to begin with.
Not mutually exclusive with what you mentioned, but can exacerbate the situation or leave a codebase with even an adept programmer a mess.
It's the result of developer who don't have the maturity to consider the long term risk of their decisions. That has nothing to do with the business leaders.
I freelance which means I see a lot of shit. In any given week I can touch/work with 10+ tech stacks across both Windows and *nix. When you do that on a regular basis it gives you a different perspective on things.
In particular, you write/design code in the same vein as the code around it. It doesn't matter what your personal values as a developer are. Consistency is more important than that. I have been known to pull people to the side and ask them "I'm about to do X, how do you normally do X in codebase Y? I want to do it exactly the way you do".
If you're going to change the way something like a DAL accesses its data you better be able to either do all of it, or you do none of it for exactly the reasons the article stated.
That is purely a technical decision, and the blame for that lies with every single developer who chose to introduce a new tech into the stack for no other reason than sensibilities.
Maybe, just possibly, the second developer could at least theoretically continue exactly from where the first developer left off (a single person or team == a single codebase that's at least somewhat consistent) but it could often make the mess even worse if the first developer really didn't have much of a clue of what he was doing.
The managers are basically the customer who ordered the software and they can't be trusted to know anything. It's a very common case that the customer doesn't know what they need and it's the job of the developers to first figure out the customer's real problem before beginning prototyping.
At this point, it's management's fault.
My group uses github for all of our code and while it's private that might change in the future. If that happens, there are a lot of places where the answer to "why did you do X this way?" is "that's the way they do it" and not that I agreed with it. That's potentially more harm than good if someone was looking at it.
If anything changes it's the engineering "best practices", like unit testing methodologies. But mostly it's just fashion, and who has the biggest platforms and marketing budgets for pushing their framework.
Software development as a field sucks, no one really knows what they're doing. We're all just figuring it out as we go along.
As he says in the article: "We think that the current shiny best way is the end of history; that it will never be superseded or seen to be suspect with hindsight."
I think they could not possibly imagine fitting the whole working set into server's RAM. Which changes quite a lot.
I disagree w/ the GP that it's purely about fashion, but certainly there distinct pendulums which swing back and forth in our field. IMO that pendulum isn't purely driven by fancy alone (i.e. fashion) but also changes in computing power.
For instance the ever driving swing of the local driven/server driven architecture. These days it's called "cloud" but it's gone by other names in the past. At some point the demands of the applications demand too much for the network traffic/network storage/etc and an increasing amount of work is offloaded to the local client. And then, at some point even further in the future there's an advance on the capabilities of remote computing (e.g. network bandwith) and the cycle repeats itself.
Throw away the first prototype version of a software and rewire v1.0 from scratch that you can use as a solid base for future development (for any application that is expected to have a long life - think years, not months) even if it's awesome for the users and works business-wise.
And always write a prototype/MVP first, and plan to throw it away, even if you end up using it in production and it ends up crucial for you business. Yes, you should try and build the MVP in weeks, not months, and you should already have its successor done when you're out of beta... but regardless of this, the pancakes rule will apply: the first one is always awful, and if try to stack the next ones on top of it, the whole pancakes tower will crumble. So throw away the first pancake and start from the second!
Looking at the 10-year-old PHP codebase we have at work, the real cause is the organizational fluid around the project, a lack of accountability/ownership.
Nobody on the business-die can confidently say what ought to be current practice, so the system becomes--via circular logic--its own reference-implementation
Since no end-user can independently state "How We Do Things Here" except by saying "Whatever The System Already Does", that channels the software's future evolutionary paths: You cannot make "common sense" fixes, because the bug might be a feature.
Would be interesting to know if there is a relationship between years of experience in programming and how likely you are going to inject the latest "cool" design pattern / framework into code. Personally I have become far more cautious about this stuff. I'd rather adapt what is already present than subject anyone else to an abstract monstrosity I just cooked up after reading the latest hacker news post on the latest hype.
I've seen in with RPC calls (hand-coded HTTP -> Hessian -> Protobufs), serialization languages (XML -> Json -> Protobufs/Bson), ad nauseam.
I think this kind of changes is the natural state of multi-year development efforts. On a 10-15 years old code base, you'll see all sorts of things like that cropping up and it's incredibly hard to steer clear of these kinds of things.
What are the alternatives? Wholesale replacement of all uses? Wasteful in resources! Not introducing new technologies? Offends the sensibilities of the devs...
That makes them unable to solve the domain problems. They just follow the spec written by business analyst.
With no interesting problems to solve they get bored and look for "new" and "cool" technologies to make their work more palatable.
The constant narrative in the example is developers having no idea why and where things are built a certain way. With more information and verification, gradual changes and refactors can be more manageable.
Avoiding Lava layer applications is not feasible, so calling it an anti-pattern is silly.
Just as it's vital to accept that business requirements will change during a project, and structure your methodology to accommodate this, the same is true of technical considerations. And the same solution applies, viz. Agile: break up any "redesign" into small steps that can deliver concrete benefits inside a two-week period. Don't treat the "transitional architecture" as a temporary thing that doesn't need to be good, doesn't need documenting etc.; like with a big database migration, you make sure that at every stage it's something that you can develop under, and you know what you'd do if you had to abandon the plan at any stage.
If you try to keep a codebase on 2003 tech forever, you're storing up trouble for later; devs will become harder to find and more expensive to hire, and eventually you'll be forced into a Big Rewrite, and we know how those turn out. Like with deployment, better to embrace Continuous Re-Architecting. If it hurts, you're not doing it often enough.
Further, and I think this should be more and more clear to people, _any_ retained unencrypted data is _compromised_ data.
Unfortunately, our society is on the way to being completely dependent upon centralized systems and institutions which are unaware of their limitations, and in fact we live in an age where centralization is considered a virtue, when it is instead a significant liability.