This is an excellent way of using your (hopefully) already existing testing and CI infrastructure to change a social problem into a technical problem. As we all know, technical problems are by far the easier of the two. :)
We used this technique to great effect at my previous day job, though under the more neutral name of "ratchet".
Brilliant. It's a really good compliment to strangling certain parts of a large code base.
As always the key is getting the information to the right people at the right time, and making it more difficult to make the wrong choices then the right choices.
The idea of deprecation and death of code assumes that those using it can easily stop.
That’s not always the case.
When Flash dies Jan 12, 2021, you’ll see what I mean.
There’s been so much misinformation about it, but there’s an OS datetime check in the Flash viewer code, browsers will disable the plugin even for some older versions not just new releases and have or will remove PPAPI and NPAPI support, and Windows already has an optional update to kill Flash support that will be part of regular updates in Summer 2021.
Even with three years of warning, it’s not ok.
It’s an extreme case of deprecation, being attempted by the best in the business, but it is inherently bad, and there’s hardly a good way of doing it.
At this point if you/your company is still using Flash I find it hard to sympathize with you. How did the company come to the conclusion that using Flash was a good business decision? It's been largely unsupported/disabled in mainstream browsers for _years_ now. They also knew for 3 years it was officially going to be deprecated in 2021. Sounds like despite that they continued to develop on the technology anyway?
Sure. Can't have it both ways though: you're welcome to choose to implement on a soon to be deprecated tech stack but you don't get to bitch about it or expect the maintainers of said stack to suddenly change their minds about deprecating.
> My point is that deprecation is by its very nature going
> to hurt something that isn’t prepared for its demise.
TFA describes techniques for deprecating code paths that do their best to notify and prepare the folks working with that code.
You're describing the end of Life of Adobe Flash that hits at the end of 2020.
Deprecation is different from EOL (Flash has been deprecated for some time now), and deprecating/EOLing a creation tool/product line has little similarity with tooling to deprecate internal APIs gracefully.
Although I do think that you and TFA agree on how difficult it is to deprecate functionality that folks rely on, it seems like you're talking about two wildly different facets of that statement.
Exactly. What's nice about this approach is it factors in the "behavioural economics" of developers. As the author puts it...
> At the end of the day, everyone needs to get work done, and if they see a code-path already being used from 10 places in the code-base despite these soft warnings–it doesn’t seem crazy to introduce another.
Even the best person with the best intentions will take a short cut under some circumstances, meaning to fix it later then getting distracted before that happens.
Here, a “shitlist” tracks existing uses of deprecated behavior within a large code base. It prevents new uses of deprecated behavior without disrupting legacy code. This is a temporary measure used to facilitate refactoring large code bases.
Shitlist driven migration/refactor, maybe. If it's development it means you're always deprecating something and that's a significant organizational smell.
Maybe we have different views of what "changing" means?
Every software is different, so take my words with a grain of salt, but personally, if it's always changing its behavior/interfaces/architecture (as opposed to just growing or a healthy mix of both) to me that sounds like either:
- It's software that runs with a highly volatile and niche functional target (scrapers, certain bots and business/operation research come to mind).
- It's a snowball of technical debt asking for a rewrite that was never given a green light.
Sometimes an evolution is needed and then shitlists have a place. If you guide development through shitlists it means you always have shit to get rid of. Maybe you have a ball of shit, in that case?
For the Java Developers here, have a look at archunit. We use this to discourage certain behaviour but you can also use it to test and whitelist the usage of shitlist items => https://www.archunit.org/
To an extent this seems to be related to more general automated detection of refactor targets/"code smells" (ironically) for which there exist tools already (e.g. linters indicated in the article) - I like this simple "list" approach though.
A similar kind of automated mechanism is required in distributed systems that allow for rolling upgrades. New functionality in upgraded nodes can't break not-yet-upgraded nodes and legacy behavior in not-yet-upgraded nodes has to be tolerated in upgraded nodes but only until the entire system is upgraded and then it is prohibited. Doing this wrong results in some really hard to fix production states.
One can't always introduce a multitude of shitlists into different places in the code. This can impact performance and readability; and perhaps even more importantly - if the language used is not high-level with reflection capabilities, you will be counting on callers' participation in calling the semi-deprecated API with truthful "usage key" for looking up on the shitlist. Who's to say nobody reuses authroized keys for expediency?
In a similar vein but on a smaller scale we have a number of tests for deprecated behaviors in our application code, and whitelist the existing code based on serializing the file name, method name, and the ordered list of parsed token types of the method.
That way minor alterations like changing a string or numeric value don't remove the method from the whitelist, but alterations to the logic of the given method require you to fix the issue while you're in there poking around already to pass CI.
I think a key to doing large, multi-developer projects, is true modular design, with opaque APIs, and each module with its own project identity and lifecycle.
Not a particularly popular stance, as it means a lot more overhead in each project.
If you don't want people to use deprecated code, it's essential to document what the replacement should be. In JavaWorld, this can be done through Javadoc. Just tagging a method or class with @Deprecated and leaving it there is the sort of thing that makes me want to hurt people.
This is literally one of the topics covered in the article and the author advocates for the same solution, documenting the alternative in the error/deprecation message...
No. But when it's not explicitly mentioned as support, it can appear as an implicit contradiction to readers. We also want the discussion about an article to build on what the article proposes, and explicitly tying comments to the article help with that.
Agreed. There's been a lot more "commenting without reading the article" here recently than in the past. So when someone comments on an article without anything in the comment that refers back to the article and merely offers the exact advice that was in the article without further analysis or insight:
1. That gives the appearance of a user who hasn't read the article
2. Even if the user did read the article I'm not sure what their comment adds to discussion of the article. For comparison, imagine participating in a conversation with a few friends in real life, friend X says "It's a good idea to write tests for your code," then friend Y replies "It's a good idea to write tests for your code." Usually the way people discuss things is to reply or offer addendums to the original idea. It's not a rule, it's just good conversation.
In this case, I think the generous interpretation is that the poster simply missed that section of the article. The generous interpretation basically requires one to conclude he read it, because otherwise he wouldn't make the connection to deprecation from the title alone.
The article in question is usually not the topic of conversation on HN. It is better to consider it to be the launching point for the conversation. Some will discuss the article, certainly, but you can expect the conversation to branch out very quickly to related items (especially if those items are mentioned in the article). While it's probably not a good idea to discuss the plight of African elephants in these threads, drawing attention to a minor, but specific point in the article is perfectly cromulent.
In this case, the post has value. It does two things. First, it draws attention to a subtopic that was only an aside in the article. (One I happened to miss the first read-through.) Second, the phrasing in the article is somewhat general and vague about this, and the post is relatively more specific and direct.
It's a shame people don't actually read the articles because, assuming the article is actually insightful, you can have better discussions after the community has actually read the article.
And in the case where people don't read the article and just read the headline, many comments have nothing to do with the article, and end up discussing a less interesting misconception of the submitted article's title/headline.
It's great to be generous with our interpretations of comments here on HN. But I would like to push for a higher standard of discussion, which has always been the draw of this community, particularly for technical articles (and this is actually a high quality technical article!). So I don't think I'll ever be happy being generous to a low value comment that re-states an opinion from the submitted article without adding anything or even without adding much.
This is the practical usage, it is really language dependent, though.
C++/Wikipedia uses the definition of the original poster, while Java's @Deprecated annotation literally has a `forRemoval` boolean value attached. Personally, I am used to how it is used in semantic versioning where it is just a helpful hint for what the next release's breaking changes may be while introducing the migration behavior if possible in this release. This allows for clients to incrementally migrate instead of making it all or nothing.
Deprecation without an obvious replacement is pretty toothless, because people will keep using the deprecated thing for lack of an obvious replacement. This is how we ended up with disasters like the Python 2/3 transition.
Your comment made me think of the adage: "Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live"
I don't know about my whole brain, but my programming brain totally is.
Now that I've been programming for a nontrivial number of years, it's happened enough that I no longer trust future me to understand anything clever that I do, so I spend much more time structuring code to require minimal context. Failing that, ample documentation, because future me will appreciate having an essay to read much more than current me wants to write it.
This is depressingly familiar. I can't tell you the number of times I've got an error, searched, ended up on StackOverflow reading the answer, full of stuff I have no knowledge of. Get to the end, it was answered by me. More than twice.
Seriously.
I'm now more obsessed with documentation and the kind of code structuring you mention and have come to hate, viscerally, projects that lack documentation.
As a potential aside, I once took a course in ASP. We had a bit of time before the class started and the tutor had a chat with a couple of us. Some huge government projects were in the news for having failed and gone well over budget. He told me he'd worked for the company behind them and that they used to obfuscate the code just by stripping out all the comments so the government had to keep going back to them for work.
I think of that any time I hear "but the code is the documentation"…
Sometimes I write my future self emails (well, logs), especially when I'm carefully stepping through knowledge domains or decisions I don't understand.
It's so helpful when I do it that I probably ought to do it more, and if there weren't so much overhead I'd probably do it everywhere.
The version I've evolved, for teaching students why I care about style:
You need to communicate clearly with a number of coworkers, the most important of whom is future-nerdponx. Sometimes future-nerdponx is smarter than you, and sometimes future-nerdponx is dumber than you, but either way they know where you live, and their misery is your misery.
Love the concept, hate the name. To me this seems like the recycle bin concept. Allow the code to exist temporarily, but encourage permanent deletion. Perhaps a better name could be a "screamlist" or "needstochange" list.
All arbritary rules like this just makes the process a mess.
Also the same people doing the shitlist will most likely be the dug in senior ones responsible for all the shit code being there in the first place and include the wrong things.
On a somewhat related note: One of the biggest things I've noticed is that simpler pieces of software are less likely to be deprecated, whereas complex codebases quickly become legacy and abandoned...
... windows ? MacOS ? Literally every Adobe software ? Literally every large audio / video / 3d création software ? Every large game engine ?
I don't know, I notice pretty much the opposite in the field of software I know - smaller software come and go, but the large ones were there 20 years ago and will likely still be there in twenty years
There is a lot of inertia driving maintenance on some of these large products but there is also a ton of deprecation taking place internally relative to them.
On Windows, almost every "new" API is built on top of Win32.
Obviously, if you're writing a simple CRUD application, you'll never need to go down there. (You'll probably make a browser application or an Electron application.)
But if you're doing something unique, like a utility, or trying to hook into the OS to make it do something new, you gotta work at the lower level APIs.
And, yes, I shipped a product that included a Windows driver this year. It required working at the Win32 level in user-mode to communicate with the driver, and to test the driver. Windows recently added a new API to Windows 10 that eliminated the need for a driver, but we still had customers on Windows 7. Even then, that new API is technically Win32.
I still write new applications in Win32 (or x64). If you want to ensure your app works on as many environments out of the box as possible this is the way to go. In fact, you shouldn’t even depend on DLL version of C runtime as they may not exist on the environment.
Yes, yes I would. I'm pretty sure that Win32 will be supported in 30 years, without much maintenance effort, compared to whatever new hotness that is popular today.
If you're targeting Windows, win32 is the stable platform. Microsoft keeps presenting other options that aren't as good, and don't do as much, and keeps walking back to win32. It even works on other platforms easily(ish) through wine/proton/etc.
Everything I do is cross-platform so I use Qt, for which version 1.0 is from 1996 and which has really not changed much since then : https://www.qt.io/blog/2018/05/24/porting-from-qt-1-0 ; I write new apps with Qt on a very regular basis.
Those large ones are large and important enough to receive substantial rewrites/overhauls when necessary. Those rewrites can be very expensive, and are completely infeasible for most software projects that don't have huge backing behind them. For small and medium projects, (code) complexity is a leading factor in maintenance, and a huge amount of unmaintainable software gets abandoned regularly, or simply falls behind and is abandoned over time.
The maintenance of giants is tied to their importance rather than the complexity of their codebases. SQLite is no less likely to be around in the future because it is simple.
Codebase complexity also isn't just about size. A huge, well-written codebase is still more likely to stick around in its current form without requiring a total overhaul.
I will probably remember Sirupsen forever for his move of breaking Golang projects by renaming his github username. This led to go modules freaking out because different project used different name casing for the dependency. He has a popular logging library and go modules are angry when you rename github usernames..
Yeah, the suggestion is that you should run a caching mirror or manually vendor your dependencies.
Other unconventional ideas about go packaging is that new major versions should have a new package name or be available in a subdirectory with the old version existing in perpetuity, and the minimal version selection approach to dependency conflicts.
I'm sure these things work well inside Google, but at least for me the package management situation is a big turn off for the language.
If you have built a module-enabled Go binary yesterday with a specific version of an external dependency, you will get that exact version of the dependency tomorrow when you rebuild (IFF you have version-controlled your go.mod and go.sum, which you should). And if someone tries to be sneaky and move the release tag, the checksum will no longer match, and your build will fail.
If you want to upgrade the version of a dependency, that is doable (with plenty of toll assistance), but it requires a positive action.
Theer is, however, no good solution for the "first use". But, that is pretyt much standard.
That is probably to some extent correct. Although this would bite you on a new build, not for anything that is already running in production. My understanding is that that is/was not necessarily the case for leftpad.
It's better to use a Shiterator, which can yield pieces of shit on demand, lazily enumerate them just in time, and generate infinite streams of shit, instead of allocating all your shit up front, and having your shit together in one place.
That's elegant dungtional programming but it's not really relevant here as you are supposed to check if new crap matches any of the enumerated fecal matter; how would you do that against an infinite stream of diarrhea? I'd leave this to the type system if possible. And of course use a colon-delimited list for the occasion.
I'm more curious about how things like shitlists are implemented in different languages.
Most of my experience is C#, where calling deprecated code triggers a warning.
Considering that C# warnings will fail in CI, how would someone do a C# shitlist? Would it require some kind of #pragma, that would stick out like a sore thumb in a code review
I think if warnings can't do the job, you'd have to go the route of writing a test that greps the codebase for usages. Maybe a custom linter could also do the trick.
The ability to grandfather in existing violations, though, is something most build tools are likely to have trouble with.
Maybe another way to do it is to have warnings fail CI, but grep out the known offenders from stderr.
None of that sounds elegant to me, but it _is_ a shitlist.
147 comments
[ 2.3 ms ] story [ 189 ms ] thread> “shitlists”: a whitelist of deprecated behaviour. Existing deprecated behaviour is OK and whitelisted.
We used this technique to great effect at my previous day job, though under the more neutral name of "ratchet".
As always the key is getting the information to the right people at the right time, and making it more difficult to make the wrong choices then the right choices.
That’s not always the case.
When Flash dies Jan 12, 2021, you’ll see what I mean.
There’s been so much misinformation about it, but there’s an OS datetime check in the Flash viewer code, browsers will disable the plugin even for some older versions not just new releases and have or will remove PPAPI and NPAPI support, and Windows already has an optional update to kill Flash support that will be part of regular updates in Summer 2021.
Even with three years of warning, it’s not ok.
It’s an extreme case of deprecation, being attempted by the best in the business, but it is inherently bad, and there’s hardly a good way of doing it.
My point is that deprecation is by its very nature going to hurt something that isn’t prepared for its demise.
There is no good way to do it. There are only less bad ways to do it.
They don’t care if you sympathize. The business or institution decides how it wants to prioritize replacement, and some projects miss deadlines.
If you think that every application running in an enterprise has stopped using Flash, you’re sorely mistaken.
You're describing the end of Life of Adobe Flash that hits at the end of 2020.
Deprecation is different from EOL (Flash has been deprecated for some time now), and deprecating/EOLing a creation tool/product line has little similarity with tooling to deprecate internal APIs gracefully.
Although I do think that you and TFA agree on how difficult it is to deprecate functionality that folks rely on, it seems like you're talking about two wildly different facets of that statement.
> At the end of the day, everyone needs to get work done, and if they see a code-path already being used from 10 places in the code-base despite these soft warnings–it doesn’t seem crazy to introduce another.
Even the best person with the best intentions will take a short cut under some circumstances, meaning to fix it later then getting distracted before that happens.
And why is it an organizational smell?
Every software is different, so take my words with a grain of salt, but personally, if it's always changing its behavior/interfaces/architecture (as opposed to just growing or a healthy mix of both) to me that sounds like either:
- It's software that runs with a highly volatile and niche functional target (scrapers, certain bots and business/operation research come to mind).
- It's a snowball of technical debt asking for a rewrite that was never given a green light.
Sometimes an evolution is needed and then shitlists have a place. If you guide development through shitlists it means you always have shit to get rid of. Maybe you have a ball of shit, in that case?
It was personal because the idea didn't get a lot of attention when I brought it up.
That way minor alterations like changing a string or numeric value don't remove the method from the whitelist, but alterations to the logic of the given method require you to fix the issue while you're in there poking around already to pass CI.
I think a key to doing large, multi-developer projects, is true modular design, with opaque APIs, and each module with its own project identity and lifecycle.
Not a particularly popular stance, as it means a lot more overhead in each project.
It does reduce the need for shitlists, though.
1. That gives the appearance of a user who hasn't read the article
2. Even if the user did read the article I'm not sure what their comment adds to discussion of the article. For comparison, imagine participating in a conversation with a few friends in real life, friend X says "It's a good idea to write tests for your code," then friend Y replies "It's a good idea to write tests for your code." Usually the way people discuss things is to reply or offer addendums to the original idea. It's not a rule, it's just good conversation.
The article in question is usually not the topic of conversation on HN. It is better to consider it to be the launching point for the conversation. Some will discuss the article, certainly, but you can expect the conversation to branch out very quickly to related items (especially if those items are mentioned in the article). While it's probably not a good idea to discuss the plight of African elephants in these threads, drawing attention to a minor, but specific point in the article is perfectly cromulent.
In this case, the post has value. It does two things. First, it draws attention to a subtopic that was only an aside in the article. (One I happened to miss the first read-through.) Second, the phrasing in the article is somewhat general and vague about this, and the post is relatively more specific and direct.
And in the case where people don't read the article and just read the headline, many comments have nothing to do with the article, and end up discussing a less interesting misconception of the submitted article's title/headline.
It's great to be generous with our interpretations of comments here on HN. But I would like to push for a higher standard of discussion, which has always been the draw of this community, particularly for technical articles (and this is actually a high quality technical article!). So I don't think I'll ever be happy being generous to a low value comment that re-states an opinion from the submitted article without adding anything or even without adding much.
Deprecated means “still supported but its use is discouraged.” It does not mean “no longer supported.”
C++/Wikipedia uses the definition of the original poster, while Java's @Deprecated annotation literally has a `forRemoval` boolean value attached. Personally, I am used to how it is used in semantic versioning where it is just a helpful hint for what the next release's breaking changes may be while introducing the migration behavior if possible in this release. This allows for clients to incrementally migrate instead of making it all or nothing.
Now that I've been programming for a nontrivial number of years, it's happened enough that I no longer trust future me to understand anything clever that I do, so I spend much more time structuring code to require minimal context. Failing that, ample documentation, because future me will appreciate having an essay to read much more than current me wants to write it.
Seriously.
I'm now more obsessed with documentation and the kind of code structuring you mention and have come to hate, viscerally, projects that lack documentation.
As a potential aside, I once took a course in ASP. We had a bit of time before the class started and the tutor had a chat with a couple of us. Some huge government projects were in the news for having failed and gone well over budget. He told me he'd worked for the company behind them and that they used to obfuscate the code just by stripping out all the comments so the government had to keep going back to them for work.
I think of that any time I hear "but the code is the documentation"…
https://existentialcomics.com/comic/1
It's so helpful when I do it that I probably ought to do it more, and if there weren't so much overhead I'd probably do it everywhere.
You need to communicate clearly with a number of coworkers, the most important of whom is future-nerdponx. Sometimes future-nerdponx is smarter than you, and sometimes future-nerdponx is dumber than you, but either way they know where you live, and their misery is your misery.
All arbritary rules like this just makes the process a mess.
Also the same people doing the shitlist will most likely be the dug in senior ones responsible for all the shit code being there in the first place and include the wrong things.
I don't know, I notice pretty much the opposite in the field of software I know - smaller software come and go, but the large ones were there 20 years ago and will likely still be there in twenty years
There is a lot of inertia driving maintenance on some of these large products but there is also a ton of deprecation taking place internally relative to them.
Obviously, if you're writing a simple CRUD application, you'll never need to go down there. (You'll probably make a browser application or an Electron application.)
But if you're doing something unique, like a utility, or trying to hook into the OS to make it do something new, you gotta work at the lower level APIs.
And, yes, I shipped a product that included a Windows driver this year. It required working at the Win32 level in user-mode to communicate with the driver, and to test the driver. Windows recently added a new API to Windows 10 that eliminated the need for a driver, but we still had customers on Windows 7. Even then, that new API is technically Win32.
The maintenance of giants is tied to their importance rather than the complexity of their codebases. SQLite is no less likely to be around in the future because it is simple.
Codebase complexity also isn't just about size. A huge, well-written codebase is still more likely to stick around in its current form without requiring a total overhaul.
I can list 10 billionaires in the world, but being a billionaires is not normal...
Or is this an "unofficial but popular" library?
Other unconventional ideas about go packaging is that new major versions should have a new package name or be available in a subdirectory with the old version existing in perpetuity, and the minimal version selection approach to dependency conflicts.
I'm sure these things work well inside Google, but at least for me the package management situation is a big turn off for the language.
If you have built a module-enabled Go binary yesterday with a specific version of an external dependency, you will get that exact version of the dependency tomorrow when you rebuild (IFF you have version-controlled your go.mod and go.sum, which you should). And if someone tries to be sneaky and move the release tag, the checksum will no longer match, and your build will fail.
If you want to upgrade the version of a dependency, that is doable (with plenty of toll assistance), but it requires a positive action.
Theer is, however, no good solution for the "first use". But, that is pretyt much standard.
Most of my experience is C#, where calling deprecated code triggers a warning.
Considering that C# warnings will fail in CI, how would someone do a C# shitlist? Would it require some kind of #pragma, that would stick out like a sore thumb in a code review
The ability to grandfather in existing violations, though, is something most build tools are likely to have trouble with.
Maybe another way to do it is to have warnings fail CI, but grep out the known offenders from stderr.
None of that sounds elegant to me, but it _is_ a shitlist.
This results in syntax highlighting in Visual Studio and warnings on compilation when somebody tries to use the method.
We treat warnings as errors and use #pragmas in cases where obsolete members are used.
When marking members as obsolete we require it be commented with a Jira issue number to make sure removing it is not forgotten.
I also strongly encourage that an XML Document Comment is added with a <see cref=""/> to the replacement member which is navigable to by the IDE.
Finally, reviewers of the PRs just need to make sure these rules are applied, but it’s easy to spot #pragmas.