Yes, I'm scratching my head about this and wondering how the output method never entered into the discussion of this post. Now, the context is that "fgrep" and "egrep" have been deprecated by GNU, so that eliminates syslogs or GUI messages from the calculus at hand, but it seems like warnings on stderr are pretty normal.
Now the further context is that fgrep and egrep have been embedded into larger scripts and systems which call them in turn, so now you've got the stderr of a tool mingling with the stderr of every other tool, and the script itself. But that's why everything on stderr includes argv[0] from time immemorial, too.
The comment about rate-limiting makes me imagine someone's got plenty of "find ... -print | xargs fgrep" in there, though!
EDIT: scratch the context, since this blog post is from 2007, though cks still had over 15 years of sysadmin experience.
There are a lot of you statements in this rant.
It reads like a junior sysadmin who just learned (or failed to learn) an important lesson about using >/dev/null 2>&1 at the end of commands in their scripts (or, even better, how to push output to a log file with the logger command or similar)
It's also not rare to find wrapper scripts that check "was there any output to stderr?", and if so, consider the program to have failed.
Warnings are not errors, but also not output, so there isn't a good way to report them in the stdout/stderr model (but stderr is still better than stdout).
> If you want to print deprecation warnings for something, the system administrators of the world will thank you if you make it not the default behavior. That way people who actually care can check when they want to (or run in checking mode all the time), but you won't cause pain for others.
And so that they can have an excuse to say they have not been informed of the deprecation and call you names. Seriously, this basically amounts to never deprecating anything at all.
As long as those deprecation errors a are accompanied with an upgrade guide detailing the deprecations, you can't be blamed for anything.
I'm sure people will upgrade willy-nilly and still call you names, but at least they've had plenty of warning.
Besides, if your program doesn't work on their test environment after updating because it needs some extra config to enable legacy behaviour, their update process will have to slow down (alter config and retry) but no serious downtime will occur.
I realize that's a realistic problem. How you deal with people like that really depends on your environment.
For open source software: too bad. The AS IS part of the license should cover expectations well.
For commercial software: depends on the contract, really. If the contract hasn't specified anything, an upgrade document will work fine in many cases.
For enterprise software: your customers won't update your software anyway. I'd go with warnings just to be safe. Maybe do it in three stages, first warnings, then severe warnings, then errors.
If you really want to grab attention for your warnings, prefix them with <<COMPLIANCY RISK ALERT>> and you'll be sure to get them to read the upgrade guide!
I was about to quote the same thing and say "yes, please". If the plan is to replace what is being deprecated with something truly better, then by all means, let's do it. But if it's more about resume padding by re-writing in the latest hot language, then I'd just as soon skip it. Of course, telling the difference is never so simple.
The reason that this has come up again was a little more complex. It wasn't resume padding, and it wasn't writing in an alternative language.
It was about deprecating and then removing long standing Unix command names, with histories back at least as far as the middle/early 1980s, simply because they aren't now in the Single Unix Specification.
I agree with others here that it should be the end-users job to handle the output in a reasonable manner which limits the breakage of things, but I think a good point is implicitly made, it is very hard to manage programs when the API is just having programs emit text that you filter somehow.
On the other hand, warning messages printed just to get logged to a file somewhere (maybe inside a container), or ignored by CI/CD, and/or drowned in a deluge of superfluous INFO (or, shudder, DEBUG) level logging, might as well not exist at all.
Yes, seems like a silly opinion to me. If your system breaks upon seeing a deprecation warning (which... It shouldn't?), you can work around it by filtering or whatever, and have the remaining deprecation period to actually fix it. If your system breaks because you failed to notice the warning and the feature was actually removed, you have no choice but to actually fix it, under time pressure.
So even if it "might as well be a fatal error", at least it's a fatal error that's more likely to be manageable instead of a fire.
My thinking was also, never has a deprecation warning I added caused immediate attention to it. If people actually read warnings like this we'd have much fewer issues with API upgrades.
Seems like maybe we should create a convention by which output from a library or program can be ignore by critical processes, so that comments and notes could be inserted, like // or # notation in code? //this depreciation notice may be given to the terminal but it will be ignored by any process that digests this output// or something like that?
Yeah, a third output would be nice to have. For example, grep could use stdout to print the actual matches it finds, stderr for any files it was unable to open, and a third stream to show the progress of the search.
The warnings that M. Siebennmann hit (since this has come up again this year) were going to standard error.
If you write to standard error in a cron job, mail gets sent to root. All of the cron jobs using fgrep and egrep were suddenly mailing root, every single run, even when nothing went wrong.
Yea, this seems like a good place to use stderr. Many ci/cd systems as well as subprocess libraries will concatenate stdout and stderr, but it is better than printing the warnings to stdout.
this guy advocates for making outputting depracation messages not default and I understand but perhaps a better compromise would be at least making sure you can turn them off? like some people have mentioned, sometimes it ends up going to standard output whereas it should probably go to error or just have a flag you can turn on to squelch them
Instead of adding a deprecation warning in the new release, go ahead and throw the full-blown exception and log it as such. The exception message can contain instructions for resolving the concern, links to documentation, et. al.
I know the way we specialize and divide labor makes this kind of thing suck in many shops, but the counterpoint is that by "just fixing it now", you don't really have to worry about all of this weird cruft accumulating over time. eventually you will get rug-pulled one way or the other, so why not incrementally shore up your usage as the API changes over time?
From a legal & B2B perspective, this might also be considered desirable. If your customers have to stay closely-aligned to your happy path, you won't have to send in seal team 6 to rescue some deprecated pile of shit or otherwise worry about weird, unsupported combinations of state & configuration building up over time.
Related on the same site that just got posted today:
> This change is pointless make-work inflicted on the broad open source ecology by GNU Grep. GNU Grep's decision to cause these long-standing commands to emit new messages requires everyone else to go through making changes in order to return to the status quo. This is exactly the same kind of make work as other pointless API changes, and just like them it's hostile to the broad open source ecology.
For contemporary discussion of this back in April, including where I dug up the Debian patches and Paul Eggert's take on this, and Rob Landley's statement about ToyBox fgrep and egrep, see https://mastodon.social/@cks/110232377928840323 .
I agree with the author that this could just be a fatal error and not an obsolescence warning; the obsolescence warning has been in the spec for thirty years already.
I learned about a quarter century ago that egrep and fgrep are considered obsolescent by POSIX/SuS, which has grep -E and grep -F, and stopped using them.
That's probably before some of the complainers were weaned off diapers.
However, I disagree with the removal of these commands. A POSIX or "Unix-like" system has a good many commands that are not in the spec. Removing commands that are in the spec (or have been) but were obsolescent is completely wrongheaded.
A sane approach might be for the utilities project like Coreutils to remove the programs; distros can simulate them with scripts (that don't emit any lame chatter):
This could be /usr/bin/egrep:
#!/bin/sh
grep -E "$@"
Why should we remove egrep just because it was mentioned in the spec as obsolescent, yet keep emacs or ssh that are not in the spec?
It should be a priority to retain any command that was ever in the actual spec, regardless of whether it was obsolescent, over retaining commands that are not in the spec.
The guiding principle should be whether those commands are in wide use and whether they are harmful in any way. fgrep and egrep aren't harmful. There is a spec for them which says what they should do; we have it from POSIX that egrep is equivalent to grep -E. People use them.
The only portability issue with egrep in a script is that it might not be there, which would only be because someone had a knee-jerk reaction to wording in the spec. If a system does have egrep, it is expected to conform to the spec and be equivalent to grep -E.
I learned at one point that posix doesn't include tar, and instead has an archiving tool called "pax". I haven't seen references to pax outside of this discovery, and indeed many systems don't even include pax. If posix can't even include such a prototypically unix tool as tar, then how can it claim to standardize the open variant of unix systems?
I think the issue is that they tried, but couldn't, due to differences in tar. So they came up with pax. This is because they wanted a format that is interoperable among implementations.
However, I think there are two mistakes in the process:
1. Why should it matter that there are different flavors of tar out there which have interoperability issues? The command and some of its options could be standardized, as having an implementation-defined format. The ecosystem of POSIX-like systems will solve the interoperability problem.
2. Why consider the matter closed for so many decades; maybe the issue should be revisited? There should be an effort to have tar as a standard command, given that it's widely used. The only reason we have POSIX in the first place is that Unix is widely implemented and used.
I mean, different file systems have totally different inode data structures, right? Yet we have a standard commands that manipulate things in the file system, like ls. What your system can ls, another one might not be able to mount at all.
A simple false dichotomy. The problem is with using a single output stream for both warning and operational/fatal messages. Give the program an option to output warnings separately and the sysadmin should use that in any automation and note when it has output without ringing the pager.
There's still a problem if operational output goes to stdout and error output to stderr. Where do the warnings go that don't directly affect either? Perhaps a better solution is to have structured 'pipes' that some experimental shells/programs use.
I'm not an ops guy, but my understanding was that stdout was for ordinary program output and stderr was for logs, which in a modern application should be structured , including a level/severity field.
I’ll vote that new warning messages should be treated as one time fatal errors. When they first make their appearance the human responsible should read them and acknowledge their existence with a little introspection as to priority then do something or nothing. Otherwise what’s the point of warning of something? It’s a warning, not an informational or debut statement.
47 comments
[ 2.5 ms ] story [ 91.6 ms ] threadIsn't this why there are two output streams, stdout and stderr?
Now the further context is that fgrep and egrep have been embedded into larger scripts and systems which call them in turn, so now you've got the stderr of a tool mingling with the stderr of every other tool, and the script itself. But that's why everything on stderr includes argv[0] from time immemorial, too.
The comment about rate-limiting makes me imagine someone's got plenty of "find ... -print | xargs fgrep" in there, though!
EDIT: scratch the context, since this blog post is from 2007, though cks still had over 15 years of sysadmin experience.
And so that they can have an excuse to say they have not been informed of the deprecation and call you names. Seriously, this basically amounts to never deprecating anything at all.
I'm sure people will upgrade willy-nilly and still call you names, but at least they've had plenty of warning.
Besides, if your program doesn't work on their test environment after updating because it needs some extra config to enable legacy behaviour, their update process will have to slow down (alter config and retry) but no serious downtime will occur.
Bold of you to assume they have one of those! Personally I assume they will break production, then call you names because of it.
For open source software: too bad. The AS IS part of the license should cover expectations well.
For commercial software: depends on the contract, really. If the contract hasn't specified anything, an upgrade document will work fine in many cases.
For enterprise software: your customers won't update your software anyway. I'd go with warnings just to be safe. Maybe do it in three stages, first warnings, then severe warnings, then errors.
If you really want to grab attention for your warnings, prefix them with <<COMPLIANCY RISK ALERT>> and you'll be sure to get them to read the upgrade guide!
yeah, though from the perspective of a sysadmin I suspect this is what they would prefer
It was about deprecating and then removing long standing Unix command names, with histories back at least as far as the middle/early 1980s, simply because they aren't now in the Single Unix Specification.
They were in the SVID.
is that not the idea?
So even if it "might as well be a fatal error", at least it's a fatal error that's more likely to be manageable instead of a fire.
Or just use stderr?
If you write to standard error in a cron job, mail gets sent to root. All of the cron jobs using fgrep and egrep were suddenly mailing root, every single run, even when nothing went wrong.
Instead of adding a deprecation warning in the new release, go ahead and throw the full-blown exception and log it as such. The exception message can contain instructions for resolving the concern, links to documentation, et. al.
I know the way we specialize and divide labor makes this kind of thing suck in many shops, but the counterpoint is that by "just fixing it now", you don't really have to worry about all of this weird cruft accumulating over time. eventually you will get rug-pulled one way or the other, so why not incrementally shore up your usage as the API changes over time?
From a legal & B2B perspective, this might also be considered desirable. If your customers have to stay closely-aligned to your happy path, you won't have to send in seal team 6 to rescue some deprecated pile of shit or otherwise worry about weird, unsupported combinations of state & configuration building up over time.
> This change is pointless make-work inflicted on the broad open source ecology by GNU Grep. GNU Grep's decision to cause these long-standing commands to emit new messages requires everyone else to go through making changes in order to return to the status quo. This is exactly the same kind of make work as other pointless API changes, and just like them it's hostile to the broad open source ecology.
* https://utcc.utoronto.ca/~cks/space/blog/linux/GNUGrepVersus...
I learned about a quarter century ago that egrep and fgrep are considered obsolescent by POSIX/SuS, which has grep -E and grep -F, and stopped using them.
Here is Single Unix Spec from 1997:
https://pubs.opengroup.org/onlinepubs/007908799/xcu/egrep.ht...
That's probably before some of the complainers were weaned off diapers.
However, I disagree with the removal of these commands. A POSIX or "Unix-like" system has a good many commands that are not in the spec. Removing commands that are in the spec (or have been) but were obsolescent is completely wrongheaded.
A sane approach might be for the utilities project like Coreutils to remove the programs; distros can simulate them with scripts (that don't emit any lame chatter):
This could be /usr/bin/egrep:
Why should we remove egrep just because it was mentioned in the spec as obsolescent, yet keep emacs or ssh that are not in the spec?It should be a priority to retain any command that was ever in the actual spec, regardless of whether it was obsolescent, over retaining commands that are not in the spec.
The guiding principle should be whether those commands are in wide use and whether they are harmful in any way. fgrep and egrep aren't harmful. There is a spec for them which says what they should do; we have it from POSIX that egrep is equivalent to grep -E. People use them.
The only portability issue with egrep in a script is that it might not be there, which would only be because someone had a knee-jerk reaction to wording in the spec. If a system does have egrep, it is expected to conform to the spec and be equivalent to grep -E.
However, I think there are two mistakes in the process:
1. Why should it matter that there are different flavors of tar out there which have interoperability issues? The command and some of its options could be standardized, as having an implementation-defined format. The ecosystem of POSIX-like systems will solve the interoperability problem.
2. Why consider the matter closed for so many decades; maybe the issue should be revisited? There should be an effort to have tar as a standard command, given that it's widely used. The only reason we have POSIX in the first place is that Unix is widely implemented and used.
I mean, different file systems have totally different inode data structures, right? Yet we have a standard commands that manipulate things in the file system, like ls. What your system can ls, another one might not be able to mount at all.