That might be true culturally, but it's not true technically. Once you know a bit of Perl (including some command line switches like -e, -n, -a, -l, -p, and -i) you will be able to do just as easily everything you used to do in awk and sed, but with a more powerful foundation so you have the flexibility to go further.
> but with a more powerful foundation so you have the flexibility to go further.
The question is: should you?
I like Perl, but it itself has many warts that make maintaining a large codebase more of a nightmare than using C, and certainly more than most modern languages.
I agree with other commenters here that the tools Perl sought to replace are still used more than it. It has its niche of being excellent at text processing, and more capable than shell scripts at that task, but I'd think twice about reaching for it to build anything more complex than a shell script replacement. Especially in 2023.
If you can't manage it with Shell scripting you should take a look at Python.
I've written Perl for many years but in the end switched to Python because the internet and resources you can find are much more quantitative and qualitative than those for Perl. Especially now when Google doesn't seem to return older results anymore.
Thank you for mentioning that book - I'm very interested in how perl can replace awk and sed and even grep(!) - I use grep multiple times a day, awk maybe a few times a week, and sed very rarely. I think I'm in the target demographic for this :)
My first thought when I saw this was that I'd like to see some stats on use of the three. I personally use sed and awk regularly and never perl, but I have no idea what is typical. I have a feeling that most people (amongst those who may have a use case) don't use any of them.
Yep I responded in another comment but I use sed and awk almost daily but I never learned Perl. It's actually something that could be useful in my current role though -- is there a particularly good intro to Perl in 2023 that I might want to look at?
The wonderful thing about Perl is the docs from 2013 are still valid. All the growth/rapid-change/package-management issues that newer ecosystem suffer through Perl did in like the late 90s. 20 year old code still works (mostly). Not nearly the churn (anymore) as JS or PHP or Go or Rust.
The Perl man pages are really good! Start with "man perlintro" and then look at whatever sections in the "man perl" table of contents fit your needs.
A lot of the tutorials online in my opinion are overbaked and written for people writing large OO projects using a lot of scaffolding. But the man pages were originally targeted at sed and awk users in your position.
Google Trends has sed crossing over perl about 5 years ago. Awk seems insignificant, which matches my expectation. (I’ve used sed but never awk, I don’t even know what it does)
Worth noting “sed” is “thirst” in Spanish, which has the potential to throw off the data, especially worldwide.
awk is a fantastic command line application that is useful for filtering, basic computations and transformations that other basic utilities (e.g. cut, sort, etc.) + grep don't (readily/easily) provide. I suppose it might be a bit archaic these days, but for someone familiar with it, it's often faster to compose an awk command than to write even a very quick/short script in, e.g., python or perl, for certain things.
>awk is a fantastic command line application that is useful for filtering, basic computations and transformations that other basic utilities (e.g. cut, sort, etc.) + grep don't (readily/easily) provide.
All true.
But awk is also a programming language, not just a command line application or utility.
It has conditionals, loops, regexes, file handling, string handling, (limited) user-definable functions, hashes (associative arrays), reporting abilities and more.
In fact, the name of the original book about awk is The Awk Programming Language.
> In fact, the name of the original book about awk is The Awk Programming Language.
Great book. I read it after it was so enthusiastically endorsed here on HN. A lot of people said it was worth reading just to take in the excellent technical writing style of Brian Kernighan.
Ditto - I use sed/awk daily, but I never write Perl any more (though I do use older bash scripts that call Perl under the hood).
I'm still comfortable recommending people learn sed/awk but I don't think I'd recommend learning Perl* now. It sits in an awkward spot between the simplicity of GNU utils and the expressivity of a scripting language, but doesn't do either of those things better than the equivalent sed/Python etc.
FWIW I still write Bash scripts on a daily basis, too.
* that's no comment on Raku, which I consider a separate beast.
> simplicity of GNU utils and the expressivity of a scripting language, but doesn't do either of those things better than the equivalent sed/Python etc.
It does scripting and gluing scripts far better than python.
It's not even close.
I'm not even going to address sed, awk, bash and the like.
if you have a script that makes heavy use of sed, awk, tr and the like, then maybe translating to perl would be most natural.
but otherwise a script is either just calling a lot of commands one after the other, for which a shell script is fine. or they do a lot of data mangling without needing many external applications or none even, in which case any other languages besides perl is just as fine.
Just imagine writing bash incorporating a sed script for something you can do with 20 keystrokes, then migrating the sed part to awk until it reaches 20 lines, then rewrite the whole thing to Python or somehow bridge it to the shell script - when you could have done the whole thing in Perl without feeling that it holds you back...
it's not about stats but simply that the space between simple shell scripts and complex systems that required "big" language, which used to be filled by perl has shrunk a lot because it is filled with many other languages that are better suited for larger systems than perl is. for most uses of perl today there are a dozen other suitable languages, including some that are older than perl but were not freely available at the time.
stats wise i am sure there are still a lot of perl users. it has its fans, and there is a lot of existing systems, but it is now a choice among many, and not usually the best choice.
>for most uses of perl today there are a dozen other suitable languages, including some that are older than perl but were not freely available at the time.
Can you name some? I'm guessing Rexx is one of those you mean.
i am talking about the popular classics that have also been mentioned in other comments here, python, ruby, php, and several others that came after them.
with the older ones i mean languages like lisp and smalltalk which didn't become available for free or even usable on PCs until the 90s
i am not saying that they are all replacing perl, but that perl was used in areas where they are better suited, and with their appearance perl is no longer needed in those areas, hence the usage space for perl shrunk a lot.
It's still used heavily for internal processes and internal web sites that query databases etc. At least at some of the large publicly traded companies I know about...
Unix was always about tools doing one thing. Obviously awk and sed didn’t do one thing only, but Perl broke away from that entirely. Perl tried to replace awk, sed, and the shell all at once.
The problem with Perl is it was still tied to Unix culture to compete with Python and it’s strong library set, so it was pretty much fazed out as an awkward intermediary.
My livelihood depends on a large set of Perl scripts, and I bet I'm not the only one in that position. It's perfect for gluing things together and then not breaking for decades, while other languages come and go. It's kind of the new COBOL that way.
Well, I'm not sure about everywhere but, the work I'm finding is ASP3.0 (IIS6.0) and VB6 (more rare) also Perl and PHP. Many of these projects were started in the late 90s. They are internal systems (billing automation, internal work-tracking, custom "ERP"). Nothing exciting, just old shit that makes money in a non-tech business. I find the work on LI, and I show up when folk search for these olds skills. But I know a few others in this "maintenance" space. One just does legacy Java, one just does old Windows (NT4, 2000), etc. Generally we find work from each other.
Additionally, I think it's hard to enter the "Legacy" space. You have to un-learn some patterns, lots of: oh yea, we used to do this the hard way. The other bump is the documentation - the 1999 docs are buried under the 2009 docs which are decaying under the 2019 docs. The thing called ActiveRecord for example means like 99 things.
Do you find any of these legacy system clients are open to having their system upgraded to a modern version of the same or related language, eg. .Net for ASP? After all, it wouldn't involve a significant change in the business logic and most esablished languages are more or less backwards compatible.
I hear that. I want that. But, mostly what I see is that at least once before a vendor shit the bed on a rewrite in $ModernX and then the business play became "nurse it".
And, having been through many rewrites myself, I frequently surprised they could hose the deal. It feels like scope-creep is the killer but I only have feelings for the data, no metrics.
Recently I wanted to see if some code I wrote still worked on Java 1.x.
The only way I could get Java 1.x to run (among my fleet of existing machines) was by installing the windows JDK version to run inside Wine on my Ubuntu.
Native Windows 10 could not run Java 1.x. This surprised me, since in my previous experience Windows was pretty amazing for maintaining reverse compatibility.
(I guess I could grab a super-old Debian docker image like Debian 6 and see if the linux Java 1.x binary would run on that, but I wanted to stick with machines I already had running.)
I use Perl all of the time. If I am going to write a bash script that is more than 12 lines long I use Perl or if I need to do read text files and do regex matching things I haven't found anything better.
There's a difference between using something because you want to and using something because you have to. awk and sed are sometimes the right tool, just not always.
When I used sed and awk a lot I also made heavy use of bash which makes things a lot nicer. And Python, of course.
I used to kind of unfairly disparage perl, but now I regret never learning it. I've written a lot of scripts for munging text/files that outgrow bash, and when I rewrite in Python they get twice as long. Seems like perl would be a happy medium and let me keep using a lot of the nice bash shortcuts like `baskticks` for shell execution, etc
Yeah so many people shooting themselves in the foot by failing to take advantage of this powerful tool - as you say especially for stuff that you might have otherwise used an overly-complex bash script for.
I find Python much easier to read and write, whereas I find it hard to write Perl as anything but an unstructured blob. Its main advantage for me is that interpreter startup time is significantly faster than Python, Ruby, Node, and most Scheme implementations. So it's useful if I ever need to call an external program for some text processing inside a shell script. Otherwise I strongly prefer the maintainability of Python.
I once went looking for something like Ansible that was implemented in Perl. Does such a thing exist? I think declarative system orchestration got popular well after perl's popularity declined. In any case, I did not find anything.
Try xonsh. It's an unholy merging of python in a shelllike syntax. It can work as your system shell and/or you can write scripts in the superset of python.
I've forgotten almost everything about Perl, but one thing I dimly remember is that for just about anything exceeding your one-liner/dirty one-off script (outgrow bash??) you'd use some library anyway, even for purposes like this. Just to make it sane. In other words it came down to more or less what you do in Python post Y2K: run(['blah'], ...) Not necessarily shorter (how?), probably not better (how?), certainly not safer (nope!) and "even" to reproduce everything subprocess can do these days I think you'd rather needed 3 or 4 imports, minimum, and remember this is Perl. So we're not talking standard library. Off to CPAN you were. Not all was great, even in heyday, and eventually people jumped ship for reasons, different reasons. Still, Perl deserved better than a bizarre zombie afterlife or undead pseudo-existence hinging on hardly more than worn out UNIX yarn. Funny how no one ever recalls tcl? Such is true rest. Such is nirvana. Yet tcl got that actual object system that never arrived for Perl. Tcl could do threads when Perl hardly knew there's such a thing! Oh and yes, tcl shipped its own GUI on top, consider that, vanilla Python even comes with it today I think. That's a way to go.
It’s interesting that while macOS is weeding out scripting language runtimes as the internal tools get rewritten (in Swift, I presume), one that is likely to prevail is Perl, so deeply is it entrenched there.
Curiously enough, Ventura ships with not one but two perls, 5.18 and 5.30. I haven’t yet investigated which tools there are using Perl. (It could also be that some prominent third party applications still call out to Perl, expect it to be there, and vendors asked nicely. How else would you explain the 5.18?)
It seems to be an apt illustration of the Lindy effect more than anything.
It was fashionable in programming discourse for people to parade their skillful opinions about things they never really understood. And it still is. ^_^
Even today in 2023, some people continue to disparage bash, awk, sed, grep, and Perl as though these tools form a common nightmare for all humanity. But these are very successful tools that solve problems quickly, efficiently in memory, and across many hardware platforms.
People are free to dislike the notation and to prefer Python or Haskell or Powershell. But some detractors simply behave in a bigoted way towards languages that they have little desire to learn. And as far as their objections to the notation are concerned, the same expressive problems find similar notational solutions in Powershell and the various DSLs that Python includes (e.g. numpy, pandas).
Bash, awk, sed, grep, and Perl are robust, durable, portable tools. Their developers were/are smart people. I've found that knowing these tools well has been important in my career. And I also code in Python.
One thing to keep in mind is that this is not actually the first version of perl, it's the first public version of perl. A version 0 did exist at the JPL previously but it's likely lost to the ages at this point.
EDIT: For the curious about the history, take a look at the perlhist documentation, https://perldoc.perl.org/perlhist it's got a lot of good info about the history of perl and it's releases.
Apparently my memory on that seems to be faulty, looks oike it was shortly after he was at jpl and left for unisys/system development corp (it changed names), https://www.oreilly.com/pub/au/148
When perl came out we were living in horrific times. You had the choice of either Bourne, C or Korn shell. Automation was glued together in one of these with a series of grep, awk, sed, ls, test, commands glued together. Anything more complicated was written in C and called from one of these things.
Perl in one stroke collapsed the programming of C, text manipulation, the capabilities of all of the Unix utilities, and data structures into one system. For anything which wasn't subsumed into the monolith of Perl, you could easy access via backticks. It was very friendly in dealing with text streams, and that's what those call-outs in those back ticks spoke.
Yes, awk and sed were replaced by Perl, but more importantly, the unmaintainable nightmare that glued all of it together was wiped out.
sed and awk are still around and still used a lot especially for one-liners. Anything more than that is typically not done as much anymore, though is certainly possible.
Agreed. I would turn to Python for anything more involved that couldn’t be done very directly and efficiently using awk/sed. I would never use Perl for anything.
1980s awk and sed aren't the same as the 2020s version.
They've become much more useful because of the influence of things like perl.
To see what I mean, here's SunOS 4.1.1's man page for awk and sed as grabbed from https://github.com/ambiamber/Run-Sun3-SunOS-4.1.1/blob/main/... and ran through groffer(1) . These are from 1989 and 1987 respectively. The core stuff is there but not much else.
Awk today is very much the same language it was in 1977 when it was introduced. Comparing your links with today's POSIX awk manpage doesn't yield a single thing Perl brought to awk. At best, while JavaScript syntax is even based on awk, you could say JavaScript regexpes are based on PCRE, and it wasn't clear whether use of capture groups/back references makes Perl regexpes accidentally Turing.
POSIX, right. POSIX sucks. POSIX anything is stuck 30 years ago. Nobody in 2023 really uses POSIX awk on some commercial UNIX like IBM AIX, they use GNU awk, that's the modern one I'm talking about https://www.gnu.org/software/gawk/manual/html_node/index.htm...
What are you talking about? There is no POSIX awk implementation, just a language spec and nawk AKA the one true awk. As shipped on Mac OS, whereas Debian ships mawk in addition to gawk. gawk is just much slower and unfortunately also buggy (crashes on a non-"C" locale with complex regexpes).
You were the one that that said POSIX awk to begin with, I was using your terms. I understand what you meant, why don't you?
As far as shitting on the GMU tools, I don't think I've seen someone do that in decades, especially just referencing the bottom of the man page like that.
As far as speed, I don't know how you're using these tools, but if gawk is your bottleneck and not i/o, there's probably smarter ways to do things.
This is not a productive conversation. You can live life however you want and if you're happy than I'm happy. We are definitely at an impasse. I'm off to bed.
Well, I look at the POSIX standards whenever I want to write shell scripts and Makefiles that work on both Linux and macOS. Which, in my current role, is often enough that I am driven crazy by how far POSIX is behind GNU tooling...
Yep. Funny enough, Linux is actually not POSIX certified
POSIX was initially a bunch of the Unix vendors getting together in the 80s seeing a rise of incompatibility saying "this shit is crazy let's find something we can agree on" and then the conflicts of trying to still have a differentiating But Also compatible product. They knew that if they cannibalized themselves, IBM Novell and DEC were there to snatch up the customers
As a result, POSIX was intentionally kneecapped by all the players but not enough to make it completely worthless.
By the 2010s I decided to simply ignore it and if something breaks on FreeBSD or whatever, make the decision then whether to support it or not. Mostly it's platform testing at the top and then subbing the gnu version of things and bailing out if it's not there.
Got it. I'm the dev tools guy at my job right now, and we have a mix of Linux and Mac users, so I've gotta make sure that any shell I write works on both. For me, POSIX seems to be a good indicator of the intersection between the two OSs, even though the true intersection is neither a superset nor subset of POSIX (eg, Mac has bash, which isn't POSIX shell). Oh well. It's close enough to be useful, and I test it all manually anyway.
Mac OS ships zsh since 10.15 as default shell as I'm sure you know, with the (restricted ksh subset called) POSIX shell pretty much the intersection of what works on both Linux and Mac OS. There you have the usefulness of POSIX/SUS.
They are also constrained languages which gives many advantages. POSIX regex will always execute in O(n), Perl "regex" are potentially exponential and can be a security risk in certain situations.
Well, that's at least an actual argument, even though POSIX regex would have never proven so useful in a variety of use cases.
Other than that, I'm completely shocked that somebody would praise sed or awk in well into the 21st century. Turing-complete languages barely capable of properly solving any problem that has any sort of algorithmic complexity, let alone one that requires proper data types, variables or interacting with any sort of system. Really, you can do that in Perl (even in Raku) without your couple-of-liner horribly breaking down once you need some custom logic to it. Oh, and it will even resemble programming languages, not some hieroglyphs left behind by aliens.
Not just the language itself but the whole ecosystem it brought with it was revolutionary. CPAN was incredible. There seemed like there was a module for just about anything! Perldoc and a testing framework were built right in. Regexes, backticks to shell out to the system, reporting built in. It really was the whole package.
Yeah CPAN is easy to take for granted now that every language has its own package manager (npm, cargo, pip), and even for C/C++ you can find what you want on GitHub. At the time though CPAN was revolutionary and no one else had it. To be able to just search for a module, find one that did what you wanted, then download it to your project was pure magic.
There is some question whether CTAN (for TeX) or CPAN was the first big library. It was pretty close. I was involved tangentially with the guys in the UK who were setting up the first pass at CTAN as the administrator of the ymir.claremont.edu archive and I remember one of the things that they came up with back then was that you could do an FTP get of any directory and get back a zip archive of its contents which was pretty fancy in the 90s. That said, both CPAN and CTAN definitely show their age in that they assume a single version of dependencies will be installed on any system which causes untold nightmares when it turns out that there’s a non-backwards compatible update to something three dependencies deep that you weren’t even aware you were using.
>There is some question whether CTAN (for TeX) or CPAN was the first big library.
I'm not in this space, but the similarity in names made me wonder if these would really have started around the same time, with an unclear "dependency order", so I reached for a search engine, and according to Wikipedia, CPAN (1993) is based on CTAN (1992).
> There is some question whether CTAN (for TeX) or CPAN was the first big library.
There isn't "some question", there is a definite answer, namely in "Programming Perl" and I quote the 4th edition, page 629:
History
Toward the end of 1993, Tim Bunce, Jarkko Hietaniemi, and Andreas König set up the perl-packrats mailing list to discuss the idea of an archive for all the Perl 4 stuff floating around the Internet. Perl 5 development had started that year, and one of its main features would be an extensible module system that would allow people to extend the language without changing perl. Jared Rhine suggested he idea of a central repository, but nothing much happened. His idea had come from CTAN (http://www.ctan.org), the Comprehensive TeX Archive Network.
>assume a single version of dependencies will be installed on any system which causes untold nightmares when it turns out that there’s a non-backwards compatible update to something three dependencies deep that you weren’t even aware you were using.
I may not completely understand what you're describing, but with perlbrew managing different Perl versions, and various CPAN clients installing modules in versioned libraries - as well as the test/smoke servers that test CPAN on tons of versions of Perl for Perl CPAN authors automatically, I don't know if I've come across what you're describing.
What's still a pita is when there's an outside dependency from the Perl ecosystem that breaks a module, but I'm not sure that's Perl's fault.
It happens more with CTAN where there is no possibility of a testing framework and there is no ability for multiple versions of an artifact to coexist.
> Automation was glued together in one of these with a series of grep, awk, sed, ls, test, commands glued together. Anything more complicated was written in C and called from one of these things.
This doesn't sound that horrific to me. It's the classic Unix approach of building small tools that do one thing well, and composing them in novel ways to solve problems. For any problem that can't be solved this way you write another small tool using your programming language of choice. Rinse and repeat.
But occasionally Unix attracts users and programmers who reject this approach, and who prefer building a monolithic tool, or in the case of Larry Wall, new programming languages. To be clear, I'm a fan of Perl and think it has its place, especially in the era it came out. It inspired many modern languages, and its impact is undeniable.
Personally, I find solutions you refer to as "unmaintainable nightmare" to be simple and elegant, if used correctly. No, you probably shouldn't abuse shell scripts to build a complex system, and beyond a certain level of complexity, a programming language is the better tool. But for most simple processing pipelines, Unix tools are perfectly capable and can be used to build maintainable solutions.
The classic Knuth-Mcllroy bout[1] comes to mind. Would you rather maintain Knuth's solution or Mcllroy's?
I don’t think you’ve seen the kind of scripts the person you are responding to is talking about.
I have, and mentioned one lower down in the comments. Unix philosophy was great but does not scale well in terms of maintainability or efficiency. Invoking processes over and over again loops is godawful slow. And the horror of complicated shell scripts is legendary.
I certainly have, and might have written a few of those myself.
But this doesn't make this approach inherently wrong or obsolete. The programmer is wrong for trying to use the tools beyond their capabilities. Where that line is drawn is subjective, as is the concept of maintainability, but if you feel that you're struggling to accomplish something, and that it's becoming a chore to maintain, the path forward is choosing a more capable tool, like a programming language.
I think we are actually mostly in agreement there then.
Perl was invented because the gap from shell to more capable languages was (and is) really big. Languages like Python and Ruby didn’t exist yet, and Perl had a really, really strong sweet spot in text processing.
Perl is usually installed by default on Linux and Unix systems. Ruby might be there, it depends.
Perl is faster than Ruby. Ruby has been one of the slower scripting languages.
But Ruby has been working on performance improvements in the past few releases. I have not seen any benchmarks of the current Perl versus the current Ruby, so this may have changed.
Perl is more concise than Ruby allowing more functionality for less code.
>> As for speed it's useful to distinguish between startup and runtime.
The JVM would like a word. (It has slow startup, but can be very fast at runtime due to JIT optimization and cacheing.)
Scripts should start and run quickly.
Ruby has historically been fairly slow which is why Ruby 3 focused heavily on performance. It has been improving a lot, but I have not seen any benchmarks against other programming languages.
The thing to remember here is that speed is relative. I haven't checked but Ruby 3 is probably faster than Perl 5.0. For most scripting purposes on modern hardware Ruby is plenty fast enough. Whilst there may may be marginal speed differences between Perl, Ruby and Python the differnce is insignificant.
I used Perl then Ruby as my main language for almost a decade each. These days, I don't really write Ruby anymore; I moved on to Elixir and never looked back. But I still find myself using Perl on the command line, in contexts where Awk or Sed would also make sense. Ruby never optimized for the one-liner case IMO.
Programming environments including shells and operating systems are just tools. And every tool can been misused. I opened a can of beans with a screwdriver once. Reality is messy. That doesn’t make the tool bad.
As a self-taught coder, I've experienced many times how highly skilled software engineers groan and sweat when they encounter shell scripts. I don't understand why, but it seems like people with a CS background are never really taught shell scripts and have come to irrationally fear them. It's sort of taboo.
This results in weird behavior, such as writing a groovy (Java?) script for Jenkins to execute bazel in order to build a go binary that runs the very same commands in an exec.Command() construct. Or people who download and import pandas to grab the third field in a csv file.
During the course of learning, I've naturally written code in bash that should have been written in another language. I replaced if statements with case because they turned out to be more performant. It's a great learning experience and why I got into python and go.
IMO we should use the right tool for the job. Sometimes that tool is a combination of unix utilities that you can put in a shell script for easier maintenance. It's just procedural execution of (usually very efficient) binaries, akin to a jenkins script or gitlab pipeline. Just mind the exceptions and use exit codes.
I'm the type of person that would grab pandas to parse a CSV. Here's my reasoning
* often times, it's not just the third column I want. Sometimes it becomes "third column unless the first column is 'b' then instead grab the fourth column". Having a good data representation makes sure that I'm not mixing logic code with representation parsing code
* I don't have to care about CSV parsing edge cases. Escaped comma? Quotes? I don't care, the library will either handle it or throw an explicit error. With custom parsing code, instead of an error, I'll get some mangled result in the middle of the file that I won't even catch / notice until later down the line
* when working with CSVs, in my area (ML / scientific compute), Python is often the right context to be in.
> I've experienced many times how highly skilled software engineers groan and sweat when they encounter shell scripts. I don't understand why, but it seems like people with a CS background are never really taught shell scripts and have come to irrationally fear them. It's sort of taboo.
It's not a lack of being "taught" shell scripts. It's the fact that shell programming constructs aren't well documented, your "standard library" is basically dependent on whatever binaries happen to be available on the filesystem, error handling is almost non-existent, etc.
It's very easy to write a bad shell script that "solves" a problem as long as a bunch of assumptions aren't violated. In my experience, senior software engineers are extremely averse to hidden assumptions and very concerned with reliability of the systems they build.
Yea... "Well the script works fine in MY SYSTEM" was the most common issue with said scripts. Running across different versions of Linux was fraught with issues, much less any other operating system that could execute a shell script.
Of course this can happen with any language, especially as it ages and adds complexity.
Yeah, the actual experience of leaky abstractions and non-portable code is forgotten. Perl solved a very real problem in the 90s. Grief, I shudder to think back to the sheer complexity of my bashrc file back then.
Yes, exactly. I have seen entire backend systems written in bash. Everything was shell script, sed and awk. The owner didn't want python or perl because he only knew bash and the related tools.
Everything was needlessly hard because these tools were not built for that. Easy to talk about philosophy and the "classic Unix approach" if you don't have to build modern applications this way.
You hit it on the head with the slowness of loops when the body comprises a series of program invocations. The horror really seeps in when you realize the original author wasn't stopped by the lack of data structures: they could get around that with some creative variable names.
At the time, before everything became Linux, all these tools and the shells used to glue them together were an incoherent mess. Was your glue sh, ksh, csh, tcsh, bash or something uncommon like zsh? Did your grep, awk and sed use the same regexp syntax as your text editor? Single letter command line options, all meaning something different to each tool. Dozens of domain specific languages (shells, awk, sed etc.) meant dozens to learn and keep in your head. And you needed it in your head, because finding the information you needed in the massive single document man file reference was a pain because hypertext links had not been invented yet (well, probably in Emacs, which was another tool like Perl that people used to avoid the command line nightmare).
While there are many possible answers, it eventually boils down to Unix being a C runtime and thus has a C culture. Lisp is from outside of this section of the world, so it simply had less adoption and support inside Unix land. Other languages, like sed, awk, and shell are not C but share its heritage(essentially, they were made by people close to the making of C.)
The first big iron I had the luck to work with was an IBM 3090 , essentially a gift from IBM, it handled the university entrance exams of the entire country of some ten million people and it had 64 MB of RAM. (It was also the first computer in Hungary permanently connected to the Internet via a leased line to Austria so it had an Austrian IP address. Hungary didn't have its IP region for two more years.)
I think the first machine with 128MB was a VAX 6510 a year or two later at another university. A little bit later, in 1994, CERN had gifted a VAX 9000 with an astounding 256MB of RAM.
To compare, the first server I installed Linux on had a grand total of 4MB RAM -- and that was one of the largest computers a small department at the university had.
It would be a long, long time before "128MB" and "mine" entered the same sentence.
Shades of the Monty Python sketch here, but the following is true...
4MB?!?
My first encounter with IBM kit was a, er, darn I'm not sure cuz I'm getting old, but I think it was a 4300? Not big iron in some senses, but still with a box that was something like 6-8 feet long iirc and definitely several feet wide and high. (And a bank of about 6-8 tape decks, each as tall as me, and two disk units, each the size of a washing machine, and so on.)
Its RAM? A massive 1 MB.
That IBM kit was the heart of the super new expensive upgrade in 1980 that cost something like 5-10 million pounds iirc to build, including a brand new building to house it and a team of programmers.
The older setup, which is where I was until its last days, was an ICL system that was expanded at the end of its life to a whopping 48KB -- yes, KB -- of RAM.
And that kit ran all the systems, internal (payroll, accounting, etc., etc.) and external (sales etc.) for the largest car dealership in the UK.
128MB? 4MB? Even 1MB? That was an unimaginably insanely large amount of RAM!
(Yes, it was very weird to be working with this physically enormous setup, and dealing with keeping it all cool enough not to halt for a half hour or so, through super human efforts when the A/C broke down, when the likes of PETs, Sinclair Z80s, and Acorn Atoms were a thing...)
Ha yes the aforementioned IBM 3090 was so big for installation they removed the roof of the building it was living in, craned it in place and put the roof the back. Bringing it up the elevator or stairs was impossible.
Much later, in the second half of the 90s, I remember the four of us carrying an IBM HDD -- I think it was your normal 5.25" drive but it needed four people because it was mounted on a vibration dampening base ...
> Much later, in the second half of the 90s, I remember the four of us carrying an IBM HDD -- I think it was your normal 5.25" drive but it needed four people because it was mounted on a vibration dampening base ...
Continuing the shades of Monty Python theme[1]:
I remember one of my first few nights being in charge of the new IBM kit (I was a "computer operator" back then, in 1980), leaning back in the fancy new chair at the desk with its fancy "virtual" teletypes (a couple "terminals" displaying the status of the OS with a CICS system), and showing off to an "underling" by swinging a long plastic slide rule or something stupid like that (I no longer recall), and me accidentally banging it on the desk. Right "near" a recessed big red button. Or perhaps "on" the button? As I snapped my head around to look at the button and begin to understand what I may have just done I heard an ominous series of whirring and clicking sounds coming from the cpu box, right near where there was an 8" diskette drive that wasn't supposed to be doing anything while the OS was running (it was just for starting the OS). Then I looked at the console... Uhoh. They didn't fire me but it took months before they decided to let me be "in charge" again with someone else actually hovering over me...
Fast forward to when I was a coder (BCPL) in a small software startup, during the second half of the 80s, presumably 10 years before you were carrying your 5.25" drive monster, I vividly recall someone bringing a 700MB hard drive back from a local computer store. It cost an astonishingly paltry 700 quid or thereabouts. A pound a MB!
> I vividly recall someone bringing a 700MB hard drive back from a local computer store. It cost an astonishingly paltry 700 quid or thereabouts.
I ... do not know. That sounds very low. Look at https://jcmit.net/diskprice.htm and note the pound was 1.8-ish around this time so the price should have been well above 1000 pounds even in early 90s. We are talking of a 5.25" full height drive, here's an 1987 model http://www.bitsavers.org/pdf/maxtor/MXT8760E.pdf rare in personal computers, it was definitely for workstations / servers.
Elisp is very much a niche language. For whatever reasons, the use of Elisp outside of Emacs is basically non-existent. Elisp is quite clunky, and AFAIK there hasn’t really been any big efforts to make it usable outside of Emacs. People who wanted Lisp outside of Emacs already had Common Lisp. (And Chez Scheme, and Scheme 48, etc etc.)
I remember capturing every password at my university via "methods". Because we had a printer quota. In the summer when everyone was gone I printed out all the man pages (all the mans, the system libraries, etc) so I'd have a nice reference book. I made sure to make it so no one was charged any money.
The one thing people can't possibly fathom if they started coding after the mid-late 90s was how much we relied on the printed medium.
I still remember when we measured the documentation IBM shipped with the mainframes not in pages but in yards it occupied on the shelves. It was a lot.
By the time our lord and savior zsh appeared on the scene Perl was already at Perl 3. And, to be fair, I do not think many used zsh before 2.1 which was some time 1991 fall and by then the Camel Book was out for half a year or something like that. So the pre-Perl and the we-use-zsh days do not really overlap.
you probably shouldn't abuse shell scripts to build a complex system, and beyond a certain level of complexity, a programming language is the better tool
but the only free programming languages available at the time were C/C++, various shells, and awk. everything else was expensive or not generally usable for other reasons. all the really useful languages to build complex systems didn't really appear or become freely available until the 90s. and perl was first among those.
I'm not saying that Perl didn't have its time and place. It certainly fulfilled a need at the time for a language more capable than shell scripts, but less cumbersome than C/C++.
But the thing is that today the shell landscape is much more mature for solving simple problems, and we have C/C++ alternatives that are saner and more capable than Perl (e.g. Go). So it arguably has lost its place, as shell tools are still in widespread use, while Perl is mostly underused. Raku is interesting, but it goes in a different direction, and its adoption is practically zero.
Unfortunately. Ruby should have been Perl's natural successor. Python is the VHS of scripting languages. For a start it doesn't have a decent answer to Perl or Ruby's one-liners. Then there's the crippled lambda implementation. Python is a sad case of worse is better.
> For a start it doesn't have a decent answer to Perl or Ruby's one-liners.
This is by design. Readability is core to the design and philosophy of python. One liners are cool and fun to write, but trying to decipher someone else's incredibly dense bash or perl one-liner is absolutely awful.
>> One liners are cool and fun to write, but trying to decipher someone else's incredibly dense bash or perl one-liner is absolutely awful.
You can write hard-to-read code in any programming language.
Python lets you with mandatory whitespace so that the awfulness spans multiple lines instead.
Really talented Python programmers can do downright demonic stuff with list comprehensions.
Python appears to be simple, but is actually quite complex. I recommend reading "Effective Python" (https://effectivepython.com/) to see beneath the surface.
The readability complaint usually comes from people who never took the time to grok the language and its idioms. At least give the user the option. Advocating a language based on what it denies you doesn't make sense. Why use a scriptig language at all if belt and braces is what you're looking for?
This topic is akin to "holy wars" and since I used to think the same way (still do to some extent), I would like you to at least consider another aspect: the effort it takes to "grok the language and its idioms" is vastly different depending on the design of the language. Letting people do whatever they want, whatever way they want it isn't only about protecting them from themselves or not.
Just think of C: I'd argue its design is actually more akin to Python than Perl (and it definitely inspired languages like Go and Zig, NOT languages like C++). It's a small language and this is a very important characteristic of it: you can count on being able to actually master it or at least very well comprehend it. Other effects of a simple and literally straightforward language can be: easier implementation and evolvement, less mental load on the developer, easier portability among developers (both for general knowledge and actual code), etc. I'm not saying that C is the way it is for all these reasons but I wouldn't overlook this factor and I do think that languages like Python are deliberately building on these advantages.
Now, I don't dislike C++ at all but back when I studied it at university, I noticed that it was the first language for me that needed to be actually studied, unlike Pascal, C, Python and "oldschool" JS. Ever since, the only languages where I felt the same were Prolog (mostly because it requires a different mindset; other than that, it didn't seem bloated) and Raku. Not C#, not Java, not Erlang. I didn't really have to touch Perl but from all I know, Raku started off as a fresh take on the Perl approach. It seems somewhat more organized but huge nevertheless, to the extent that there literally isn't one person who really "groks the language" all around. In the case of Raku, I wouldn't even say it encourages you to write unreadable code (especially if you have a thing for APL look-alikes, lol) - it's just so rich that there is a good chance you will come across something in someone else's code you have never used before and don't quite remember how it will act in your specific use case.
There are different types of freedom than "do whatever you want". Like, the freedom to feel safe and confident about code. These days, humanity has aggregated an immense amount of knowledge and technology and "I will do it all by myself" is not that much of an option. And even people with such puritanistic tendencies will choose simple and straightforward tools, even if not for the "limitations".
There have always been feature-rich and bare-bones languages. No point pitting one against the other. I don't consider myself particularly clever but with the help of "Programming Perl" and "Learning Perl" I managed to get a pretty good grasp of the language with no prior programming experience. I just feel a lot of the knee-jerk response to the mention of Perl comes from people who have never put any effort into learning it.
Well I'm saying that "you just didn't bother to learn it, duh" is not an equally valid argument for different languages; it's much less valid for Perl than it would be for Python. It rewards your efforts much less. Perl is notoriously a language that had desperate criticism among its users even by the mid 90's ($[ and $] stuff comes to mind), quickly led to the creation of Ruby and famously "forked itself" with what is now known as Raku.
Now I have barely spent any time with the oldschool Perl but trust me, I have put a lot of effort into learning Raku, the language that was meant to fix Perl. Whenever something that "seemed like a good idea at first but it's actually harmful" shows up, it's usually Perl's legacy. I'm thinking of things like the conceptual mishmash between a single-element list and a scalar value (or in general, trying hard to break down variables arbitrarily into list-alikes, hash-alikes and the rest of the world), the concept of values that try to implicitly pretend they are strings and numbers at will, or the transparency of all subroutines to loop control statements which is some next level spaghetti design. If you ever actually use something like this, you introduce a brand new level of complexity, somewhere inbetween a "goto" and a "comefrom", so I would really think about if this was worth learning at all.
Oh right... from what I remember, it was also Perl that fostered this idiotic idea that a name of a concrete thing could be overloaded to be a namespace as well, and a concrete Foo::Bar could very well be something that has no logical relation to a concrete Foo. Moreover, I'm quite sure Perl invented this nonsensical distribution-module dichotomy where you are supposed to depend on modules, despite the smallest publishable and installable unit being a distribution. There are three outcomes with that:
- if the distribution contains only one module: what was the point of drawing the distinction?
- if the distribution contains tightly coupled modules: you can pretend to only depend on one of the modules but in fact you are depending on the whole distribution together
- if the distribution is a collection of unrelated modules: why are you trying to encouple the metadata when this will make the versioning meaningless?
I can only hope that it's somehow better than Raku but the whole principle is just an anomaly.
And you know, then these people move around in the world, pretending that all of this is just normal and you just have to learn it. Well guess what, there is a reason people might want to put that effort into something else.
I think it's possible that things that seem normal and inoffensive can become horrific simply from scale. You'll climb the stepladder without complaint, but then there's that radio tower in Canada...
Others like the idea of the family cow, then they see the 10,000 head feedlot from the highway.
Scale is sometimes sufficient by itself to induce horror.
csh was a decent interactive tool, but not great for scripting. Bourne shell had the right idea but there were so many bugs in various corners of it (I still sometimes end up writing "test "x$foo" = "xbar" even though shells that need that are long gone).
If you can depend on a recent bash and use shellcheck, then it's actually quite a pleasant programming environment, with fewer footguns than one might think. (I want a @#$@# "set -e" equivalent that returns non-zero from a function if any statement in the function results in non-zero).
There are some things that are more awkward than they should be though (e.g. given a glob, does it match 0, 1, or many files, or the way array expansions work).
Also, there's no builtin way to manage libraries (I don't know about Perl, but Python suffers from this as well). This results in me pasting a few dozen lines of shell at the top of any of my significant shell scripts, for quality-of-life functions. Then I have to use "command -v" to check if the various external programs I'm going to use are present. Say what you will about C, but a statically-linked C program can be dropped in anywhere.
Mostly agree. The modern shell scripting environment is much more robust than 30 years ago, with ShellCheck and some sane defaults, as you say. I also find it pleasant, once you get over some of its quirks.
As for managing libraries, that's true, but you can certainly import and reuse some common util functions.
For example, this is at the top of most of my scripts:
set -eEuxo pipefail
_scriptdir="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")"
source "${_scriptdir}/lib.sh"
This loads `lib.sh` from a common directory where my shell scripts live, which has some logging and error handling functions, so it cuts down on repetition just like a programming language would.
> This doesn't sound that horrific to me. It's the classic Unix approach of building small tools that do one thing well, and composing them in novel ways to solve problems.
This works really well if your problem can be solved in one or two liners.
It go bad very quickly when, say, you have two CSV files and want to join them the sql-way.
In sed, you have to use positional variables and think about shell escaping. In perl, you can at least name those variables and use \Q
> This works really well if your problem can be solved in one or two liners.
My personal comfort threshold is around the 100-line mark. It's even possible to write maintainable shell scripts up to 500 lines, but it mostly depends on the problem you're trying to solve, and the discipline of the programmer to follow best practices (use sane defaults, ShellCheck, etc.).
> It go bad very quickly when, say, you have two CSV files and want to join them the sql-way.
In that case we're talking about structured data, and, yeah, Perl or Python would be easier to work with. That said, depending on the complexity of the CSV, you can still go a long way with plain Bash with IFS/read(1) or tr(1) to split CSV columns. This wouldn't be very robust, but there are tools that handle CSV specifically[1], which can be composed in a shell script just fine.
So it's always a balancing act of being productive quickly with a shell script, or reaching out for a programming language once the tools aren't a good fit, or maintenance becomes an issue.
You’re discussing modern tooling in a conversation about early UNIX tooling. Back in the period being discussed, even ‘read’ was less functional. Ksh introduced a lot of the stuff we now take for granted, some of which wasn’t even available until the Ksh93 (long after Perl was released). Bash itself is a younger project than Perl. Albeit not by much.
Fair point. I'm not arguing that Perl wasn't an improvement back then, but that the approach of composing Unix tools is not inherently bad. And as the shell ecosystem evolved since then, and more capable programming languages appeared, Perl has been left by the wayside as a historical relic, rather than the replacement of Unix tools that Wall envisioned.
So I don't disagree that it was needed back then, but it's important to mention the modern context it struggles to exist in.
Perl is still a commonly used tool chain. It is far from being a “historic relic”.
I agree that there’s nothing wrong with composing UNIX tools. I mean, that was one of its key selling points. if you watch any early promo videos for UNIX you’ll see them talk heavily about the composability of the command line and shell scripting. It wasn’t an accident — it was designed that way.
The point of the conversation wasn’t to say that one shouldn’t write shell scripts, it was just to say that there was a massive and unfilled gulf between what was easy to do in Ksh, awk and sed, and what could be done in C.
The "philosophy" that I never saw in commercial UNIXes, starting with my first experience with Xenix in 1993, beyond being endless repeated in FOSS circles, which is ironic given how GNU and BSD applications work, with their endless amount of command line parameters.
The effortless composition of complex commands out of simple standalone programs is one of the best features of Unix. And yes, I admire and love it as well.
That said, imagine a metrics system for a huge networking company that used these methods to cover all automated testing or defect analysis. Those inner loops were made of greps and seds and so forth, and each one is the invocation of a new program. It wasn't uncommon for these runs to take almost a day.
Besides performance, the other nightmare was was someone described below: each script was a one-off that didn't leverage the work from others. If the author only new C shell, then you know you're going to be doing gymnastics to catch the stderr of some of those programs (you can't capture it in the same manner that Bourne variants do).
Anyway, yes, we all adore the Unix philosophy, but there are limits.
Back in the 90s I got my first 'real' job, and needed to learn well grep, awk, sed, col et. al. Or learn Perl. Perl kept us afloat until we had too much Perl and needed to switch to Python. And I still don't know awk or sed; to this day my fingers just go with 'perl -pie'.
Perl didn't replace anything. It was replaced by PHP for CGI programming (which says a lot!) and botched itself by the second worst version migration of all times.
Yes. It's comparative. The absolute value of Perl's readability, maintainability, and general intuitiveness is massively better than all comparable tools that existed when Perl came out. Perl is a massive improvement over those tools.
If you have doubts, try writing a complicated production software stack in awk. Then hand it off to a coworker.
One need not be irreplaceably good, if one is already beating the current state of the art by an order of magnitude. Perl did this.
Moreover, as often as people joke about the readability of Perl code, that's entirely a function of the developer.
I've easily written tens of thousands of lines of Perl, and not a single person has complained about difficulty reading or maintaining that code. Why? Because I apply all the usual best practices for code hygiene that apply to any language.
Frankly, I think most people are just repeating a meme they heard once, and the rest just get put off by a) sigils, and b) the use of implicit variables (which I tend to use only very sparingly, and mostly in quick one-liners).
But a developer that poorly names their functions or variables, fails to modularize appropriately, fails to document their code, abuses language features because they want to be excessively terse or clever, that kind of person is gonna write crappy, difficult-to-maintain code no matter what language they use.
In fact, I'd argue the advent and popularity of Python--which was pretty radical in how opinionated it was at the time--is a direct response to languages like Perl and C that were a lot more free-form and easier to abuse by poor coders.
Oh sure. IME Perl just pulls it all together into a much more convenient, all-in-one package that has all the libraries and tools you need to build anything from small scripts all the way up to large projects.
All nontrivial sed programs are line noise, there is no other way to program in it. In Perl you at least have a chance. Awk is not much more readable than Perl actually.
Oh yeah! Perl sure did have a bunch of ugly programs, but I believe that was because it was so easy for non-experts to program in it. There have been many companies successfully built atop of a Perl code base, and I've seen fantastic systems built with it. I've also seen one-off programs handed from manager to manager which would scare you to death.
The dozens of different versions of UNIX did not help either. A bug in their sed grep awk toolchain meant that you had to write workarounds for things like "Sequioa Unix" to support is single customer. as long as the UNIX flavor had a PERL port all of that could go away.
PERL being open source meant you were a "make" away from having the same environment on all those unixes.
even today when the dozens of UNIX distributions have collapsed into a couple of Lunux flavours, getting a shell based script to work reliably between say macOS and redhat can take some effort.
assuming that the security administrators of the systems you want to run the script on haven't followed Microsoft's 'recommendations' and basically made powershell unusable on the systems due to random GPOs ;)
PS is 'okay' as a scripting language, but it's very frustrating how Microsoft's security defaults seem to be dead-set on ensuring you can't use scripts on systems without jumping through a bunch of hoops. I get how MS got to this conclusion that it needs to be locked down, but it makes me think twice about pumping out a script since I know I might need to walk my colleagues through how to actually run the damn thing depending on what the Security Admins decided to lock down that week.
# Sometimes I wake up screaming. Famous figures are gathered in the nightmare,
# Steve Bourne, Larry Wall, the whole of the ANSI C committee. They're just
# standing there, waiting, but the truely terrifying thing is what they carry
# in their hands. At first sight each seems to bear the same thing, but it is
# not so for the forms in their grasp are ever so slightly different one from
# the other. Each is twisted in some grotesque way from the other to make each
# an unspeakable perversion impossible to perceive without the onset of madness.
# True insanity awaits anyone who perceives all of these horrors together.
For me in the current day, perl is too much like "real programming", while I'm fairly comfortable with grep and sed, and to a lesser extent, awk. Piping a bunch of stuff together is just easier to understand and gradually expand. One thing I do in a lot of scripts is read from and write to the clipboard, usually transforming text with sed and co. Stuff like turning YouTube shorts URLs into regular ones, taking just the last directory name from a long file path on my clipboard, making my clipboard lowercase, etc.
I barely get into loops and lists when trying to learn Python before losing interest/focus and not trying again for months. It feels very abstract and like it's not useful for day-to-day stuff without learning and writing a lot. Shell stuff is very short and powerful and feels more grounded in reality. To be clear, I am not using if/else and the like in my shell scripts either. Usually they're just a few lines with little to no logic, but they're immensely useful. I can't imagine learning perl and then using it for all this stuff.
Yes, writing shell scripts calling awk and sed was awful, just an aesthetic nightmare to work with. Properly quoting strings was always problematic, and while Usenet was available there wasn't a plethora of information like nowadays. This was a few years before Bash was released.
Through my use of Usenet I learned that a better reader program was available called 'rn', which I downloaded and built. It had an amazing handwritten install script (autoconf was years off) which could automatically configure and build rn on any *NIX system. All developed by a fellow named Larry Wall.
rn was truly a joy to use and made reading Usenet swift and efficient. It would get updated with the fixes that came across Usenet that I'd apply using the clever 'patch' program, also written by Larry Wall.
Based on my experience with his other software, when Larry Wall released Perl on Usenet I immediately downloaded, built, and started using it. As promised, for scripting things not requiring a C program, it was massive improvement. Version 2.0 came out and brought many great new capabilities.
I wasn't writing software while versions 3 and 4 came out; I started using it again after version 5 and the appearance of CPAN. Over the years I've used Perl extensively for task automation and data wrangling.
Python now dominates Perl's niche because it's easier to learn and interfaces better with C. It's also less flexible, which compared to Perl is a virtue. One of Perl's mottos is TMTOWTDI—there's more than one way to do it. But many of them are bad. Much of Perl's poor reputation ("line noise") stems from this.
But when Perl was released it was a revelation, like a drab day when the clouds suddenly part letting warm sunlight pour down on the land.
I will always love perl because it got my career off the ground. Anyone who doesn’t just learn it is doing themselves a disservice. It’s similar to SQL in that sense.
This. PHP was my first "real" programming language (which is a bit scary now that I think about it since it followed me learning QBasic and VB6) and by the time I was messing with it for web dev circa 2003 CGI scripts were on their way out. I suspect that it had something to do with CPanel/WHM-based shared hosting setups being reticent to deal with something as powerful as CGI and preferring to just run PHP in Safe Mode. I could be wrong though as I was also a technically-ignorant punk that fixed phpBB installer filesystem permission issues by 777'ing everything in my hosting directory :-). Good times...
Ha! Cheers! My first language was VB.net and my first paying gig was porting a VB6 app. But my career really started when I took on Perl and Oracle. Good times indeed.
You missed out. Perl is way more than the web. Nobody is grinding up csv files with PHP. But I wired up an Oracle billing system and a MSSQL invoice system in under a month as a junior dev. And it worked.
Every library I needed was at my fingertips on CPAN. And just… worked. Connect to Oracle, run SQL, create a CSV, FTP it somewhere. EZPZ.
I miss CGI internal tooling. It was always fast and just did the thing you want. Now we have bullshit recursive lambdas with nonsense UIs full of harmful whitespace.
I started with Perl CGI and have been less productive every year since then.
Seriously. I feel like 99% of the work I do in Micro-Service/Event-Driven/etc systems could realistically be a few CGI scripts hooked connected to a DB via DBI. It's sort of ironic how people often call Perl "overly complicated" or "confusing" then justify 100+ service clusters using K8's and AWS to serve 1000 users/day. AdventOfCode handles a few million users per day during the weeks it's active, all powered by a few FastCGI scripts, it's still really good technology.
I'm younger than a lot of folks on HN, but I've never touched Perl for anything more than some copy-pasted shell one-liners from stack overflow. Granted, web isn't what I get paid to do, but I still know Ruby, Python, a bit of SQL, etc... I've just never really had a use case for Perl, nor am I totally sure what that use case would look like.
Well Perl isn’t (just) a web language. And this article does a pretty good job explaining what it is. Which is a powerful, lightweight general purpose programming language.
CPAN alone is worth the price of admission. I started my career using Perl to build Excel spreadsheets from queries I ran against Oracle DB. That may sound horrible but it was actually plug and play. Really simple and just worked. I could even fire off emails from the script. Just zero hurdles (or guard rails).
I don't think comparing Perl to SQL quite works. There are currently plenty of other widely-used scripting languages that can do the same sort of thing that Perl does (if we consider it in the context of a general purpose scripting language rather than a sed/awk alternative), but the usage of SQL alternatives for querying relational databases is a rounding error.
Unpopular opinion, but I think we'd be so much better off if Netscape had just embedded a Perl interpreter instead of creating JavaScript. Perl CGI was already the dominant server side technology at the time and it has incredible string manipulation capabilities that would be beneficial for browser scripting.
JavaScript being some kind of Lisp dialect with a C syntax is good IMHO. You also get some Perl inspired features in JavaScript, like the regular expression engine, and the not so user friendly array methods like slice and splice.
The DOM API is also a bit more robust than string manipulation.
I know its history but Javascript is not in any meaningful sense a "Lisp dialect". Code is not data, code is not even an object, you have to define a function, closure or object to manipulate the code itself.
I recall the C interface to perl being exceptionally ugly. It was not designed with embedding in mind. That was one of the things that convinced me to move to ruby.. matz had exceptional taste when it came to the C api.
Should have just embedded a shell with Unix inspired (perhaps even plan 9 - 1992) command suite, implicitly treating piped HTML as plaintext corresponding to the rendered page. So the user which is 99% of people is better served rather than some random developer who happens to know perl.
I know people dump on Perl all the time, I know I do.
But it was a critical part of computing history. Perl did a tremendous job of bridging the gap between shell commands and shell scripts and “big languages”. Way back in the early 90’s I took a complicated shell script that took about 2 hours to run this huge text processing job , rewrote it in Perl and it took about 20 seconds, and was more maintainable to boot.
It really helped boost software development in the 90s, even pre-CGI script.
The Perl source itself was also very interesting, with numerous optimizations to make it fast (again, for its time).
What killed it were the never ending eccentricities you ran into, endless foot guns, and of course Perl 6 never being delivered.
During a later period (early 2000's) Perl was the only scripting language with good Unicode support, including inside regular expressions, which was revolutionary at the time. There was so much cool stuff you could only do in Perl, and then like you say it all ran into the brick wall of Perl 6.
Yes and Perl’s Unicode transition between 5.5 and 5.6* was exemplary. There were some speed bumps but generally old stuff continued to work and Unicode support could be added in a straightforward manner. In contrast to how python 3 handled it (not trying to start a language war, these days I prefer python overall). I suspect Larry Wall just understood text processing much more deeply than Guido (makes sense since he is a linguist and Perl was always more text oriented).
*technically it was 5.005 or something to 5.6 because they decided to drop a zero or two
The brick wall of Perl 6 was itself a solution to an internal nightmare inside of core Perl development. Particularly around conflicts involving Tom Christensen, which caused no progress to be made on Perl 5.6.1 despite major bugs in 5.6.0. For example:
my $x = 5 + 6;
$x .= "0";
print $x + 5; # 16, not 115.
The result in the year 2000 was a famous cup throwing incident by Jon Orwant. And Perl 6 was a plan to separate those who wanted to create their ideal language (Perl 6) from those who wanted to maintain existing Perl.
And it worked, sort of, for a few years.
I was on the Perl Grant Committee at the moment that killed Perl 6 in my opinion. We had a choice of 2 grants from Nicholas Clark that we could fund during 2006. One brought a lot of immediate benefits to Perl (Unicode fixes, reduced memory usage, etc), and the other was to get Ponie to the point where someone other than a core Perl developer could work on Perl 6.
Not like php, no. But perl has loads of excellent web frameworks(mojolicious, dancer, et al) and there is webperl.js which tries to bridge the gap to an extent, though I don't know if anyone uses it and what the quality of it is
It is ludicrous to claim that Perl didn't support web development. From Amazon to eToys to Citysearch to Stamps.com, big dot coms were all based on Perl.
Big PHP shops like Facebook, and PHP projects like WordPress, came in a later development generation for the simple reason that PHP made it easy to get started on shared webhosting.
BS. CGI.pm was part of the Perl core in 1997. And Perl was the language that dominated the dot com boom.
The thing that PHP did better than Perl was the ability to be set up on shared web hosting. Mind you, it did that mostly by not acknowledging all of the security holes that it had which allowed one user access to what should have been private for another user. But shared hosting providers had a population of people who wanted to use PHP, were forgiving of major security flaws (which PHP had many of), and would pay money.
Meanwhile Perl went the route of doing things right, and being much more secure. Which, even though it feels important to good developers, is bad for marketing. It was a classic Worse is Better situation. See https://www.dreamsongs.com/RiseOfWorseIsBetter.html if you're not familiar with that concept.
And, in the end, the commodity wins. So in the mid-2000s, PHP did catch up and beat Perl.
Is there a summary of what Tom Christensen did? I hold him up pretty high when it comes to Perl lore. I don't know from what you wrote if he's the hero or villain in this case (and if you don't remember that's fine: ancient history)
Essentially Tom was both knowledgeable and abrasive to an extreme. Technically he was brilliant and a valued asset. But socially he was a center of conflict. And the result was that Perl 5 development ground to a halt and other people left.
I don't know everything that Larry did about the situation. But it was a situation that everyone on p5p was painfully aware of at the time.
Perl still does have the best Unicode support, especially for regex's. In comparison, grep has imitation Unicode support (the -P "Perl" flag helps, but it's still nowhere near as powerful as Perl regex's). Python's standard regex library, re, is better than grep's regex's, but still not as good as Perl. Python also has a non-standard regex library, regex, in PyPi, which I think is as good as Perl.
Can you share some examples of Perl regex Unicode support that you make use of in real programs? Examples of stuff that can't be done (or easily be done) in PCRE would be awesome. :)
>> What killed it were the never ending eccentricities you ran into, endless foot guns, and of course Perl 6 never being delivered.
>> But it really was the weird shit that did it in.
The current state of Perl 5 for Python fans: ----
Perl 5: I'm not dead!
TIOBE: 'Ere! 'E says 'e's not dead!
Internet: Yes he is.
Perl 5: I'm not!
TIOBE: 'E isn't?
Internet: Well... he will be soon-- he's very ill...
Perl 5: I'm getting better!
Internet: No you're not, you'll be stone dead in a moment.
TIOBE: I can't take 'im off like that! It's against regulations!
Perl 5: I don't want to go off the chart....
Internet: Oh, don't be such a baby.
TIOBE: I can't take 'im off....
Perl 5: I feel fine!
Internet: Well, do us a favor...
TIOBE: I can't!
Internet: Can you hang around a couple of minutes? He won't be long...
TIOBE: No, gotta get to Reddit, they lost nine today.
Internet: Well, when's your next round?
TIOBE: Next year.
Perl 5: I think I'll go for a walk....
Internet: You're not fooling anyone, you know-- (to TIOBE) Look, isn't there something you can do...?
Perl 5: I feel happy! I feel happy!
----
But seriously, Perl is not dead yet.
It is usually installed by default on most Unix and Linux systems.
It also rides along with most MinGW-based toolkits like Git for Windows. If you do software development, there is a good chance you might have Perl installed and not even know it.
If you are on a Linux or Unix system try running 'perl --version' to see what version you have installed. I would be very surprised if it is not there.
Some people like to make fun of it despite its wide-spread use and utility. Most of them probably know little to nothing about Perl and have not written or maintained any Perl code. There is a good chance that they are using Perl code without knowing it.
My version of “dead” here means rarely used for new development. I relied on it for years, and even I finally gave up and do script level work in Python instead. Particularly because the many CPAN dependencies are aging and starting to rot from neglect.
I still remember seeing this line from the configure script scroll by.. Made me curious what Eunice was, which was harder than it might seem, given that this was before the web and any search engines.. (and yes, it turns out Eunice was a unix-ish environment on VAX/VMS)
The next time I need awk or sed, whatever I end up writing I’ll submit to chat gpt and ask it to rewrite it in perl so I can compare. I’m curious because so many people (well, enough people anyway) speak so highly of perl’s text processing capabilities. If it’s such a great tool, then I want to use it too.
Edit: one plus is that perl is as ubiquitous as awk and sed. It was there the whole time and I didn’t even realize it!
This sounds interesting!
The main reason I moved to perl instead of sed or AWK is for better control. Shell scripts don’t differentiate between variables and data (anyone else still having nightmares from trying to escape quotes and meta chars in a shell?).
Perl gives you all the simple features from sed or AWK and adds useful (maintainable) foreach iterators, arrays, hashes, etc.
Recommend using Strict mode if you are new to Perl. It gives good guardrails against silly mistakes like not declaring or misspelling a variables, or accessing strings or numbers that aren’t the correct datatype.
Thank you for the tip re: strict! Escaping - that's definitely a sore point. I feel like I get tripped up by embedded double quotes way too often - does perl have a good way to handle them?
Perl is/was fantastic and awful. An excellent alternative to grep, awk, sed, ls, test.
A lot of the greatness of Perl came from writings of Larry Wall, Mark Jason Dominus, Randal Schwartz, Tom Christiansen. There is so much wisdom to be gained from all their code and documentation.
My only real forays with Perl in anger was back with the original camel book and Perl 4. The secret sauce was the ability to persist the associative arrays to a dbm file.
Even though I worked on a lot of Unix boxes doing assorted data processing, Perl was never my goto for the same reason I adopted vi instead of eMacs.
As good as it was, it didn’t come stock on my clients machines, and I could not presume to install it or count on it being there. So, the litany of classic Unix tools prevailed and I simply became adept at working with those.
My few temptations to dip my toes in the modern (at the time) Perl waters just found indecipherable source code (to my ignorant eyes) and disastrous attempts to get whatever it was I was dabbling with out of CPAN. I was never very successful with it.
A while back, I wrote a two-part article highlighting use cases where Perl's rich regular expression engine, built-in functions, extensive ecosystem and portability helps. Includes examples that are easier to do with Perl compared to sed/awk:
Reading this summarizes my feelings on what kind of new language we need.
Something that makes workloads usually delegated to find, grep, sed and awk coherent and easy to use, allows streaming data between functions/commands and can still be used to glue things together.
Can I use Perl instead of bash/as my command line ? Should I start learning Perl?
I found Perl a practical replacement on the command line (not as the command line) to command line sed and awk, to the point I never learned sed and awk. Thanks to the -p and -e options (perl -pe 's/foo/bar/'). It never felt like a practical replacement to find. It would probably work as a grep replacement, but my fingers already knew grep. Today? Don't know. Back in the day though, in your standard heterogeneous environment it was the most practical way of working since your awk and sed seemed to be platform dependent in many ways (awk, gawk, nawk, proprietary extensions etc.)
I miss it. I’m too rusty to be effective with it anymore, but it’s the most pleasant to use and productive language for me for small stuff. Best community there ever was as well, mostly C and Unix system programmer types.
258 comments
[ 21.4 ms ] story [ 797 ms ] threadThe question is: should you?
I like Perl, but it itself has many warts that make maintaining a large codebase more of a nightmare than using C, and certainly more than most modern languages.
I agree with other commenters here that the tools Perl sought to replace are still used more than it. It has its niche of being excellent at text processing, and more capable than shell scripts at that task, but I'd think twice about reaching for it to build anything more complex than a shell script replacement. Especially in 2023.
I write a lot of bash for better or worse. I wonder if perl would be a better choice sometimes.
I've written Perl for many years but in the end switched to Python because the internet and resources you can find are much more quantitative and qualitative than those for Perl. Especially now when Google doesn't seem to return older results anymore.
A lot of the tutorials online in my opinion are overbaked and written for people writing large OO projects using a lot of scaffolding. But the man pages were originally targeted at sed and awk users in your position.
Worth noting “sed” is “thirst” in Spanish, which has the potential to throw off the data, especially worldwide.
https://trends.google.com/trends/explore?date=all&geo=US&q=p...
All true.
But awk is also a programming language, not just a command line application or utility.
It has conditionals, loops, regexes, file handling, string handling, (limited) user-definable functions, hashes (associative arrays), reporting abilities and more.
In fact, the name of the original book about awk is The Awk Programming Language.
Great book. I read it after it was so enthusiastically endorsed here on HN. A lot of people said it was worth reading just to take in the excellent technical writing style of Brian Kernighan.
I would offer a strong second to that.
I'm still comfortable recommending people learn sed/awk but I don't think I'd recommend learning Perl* now. It sits in an awkward spot between the simplicity of GNU utils and the expressivity of a scripting language, but doesn't do either of those things better than the equivalent sed/Python etc.
FWIW I still write Bash scripts on a daily basis, too.
* that's no comment on Raku, which I consider a separate beast.
It does scripting and gluing scripts far better than python.
It's not even close.
I'm not even going to address sed, awk, bash and the like.
but otherwise a script is either just calling a lot of commands one after the other, for which a shell script is fine. or they do a lot of data mangling without needing many external applications or none even, in which case any other languages besides perl is just as fine.
This. I always feel like bash and pipes are always holding me back and resisting.
Python is just to verbose to exec/pipe/read/write.
The real issue is the it's really easy to shoot yourself in the foot with perl.
stats wise i am sure there are still a lot of perl users. it has its fans, and there is a lot of existing systems, but it is now a choice among many, and not usually the best choice.
Can you name some? I'm guessing Rexx is one of those you mean.
with the older ones i mean languages like lisp and smalltalk which didn't become available for free or even usable on PCs until the 90s
i am not saying that they are all replacing perl, but that perl was used in areas where they are better suited, and with their appearance perl is no longer needed in those areas, hence the usage space for perl shrunk a lot.
The problem with Perl is it was still tied to Unix culture to compete with Python and it’s strong library set, so it was pretty much fazed out as an awkward intermediary.
Additionally, I think it's hard to enter the "Legacy" space. You have to un-learn some patterns, lots of: oh yea, we used to do this the hard way. The other bump is the documentation - the 1999 docs are buried under the 2009 docs which are decaying under the 2019 docs. The thing called ActiveRecord for example means like 99 things.
And, having been through many rewrites myself, I frequently surprised they could hose the deal. It feels like scope-creep is the killer but I only have feelings for the data, no metrics.
The only way I could get Java 1.x to run (among my fleet of existing machines) was by installing the windows JDK version to run inside Wine on my Ubuntu.
Native Windows 10 could not run Java 1.x. This surprised me, since in my previous experience Windows was pretty amazing for maintaining reverse compatibility.
(I guess I could grab a super-old Debian docker image like Debian 6 and see if the linux Java 1.x binary would run on that, but I wanted to stick with machines I already had running.)
When I used sed and awk a lot I also made heavy use of bash which makes things a lot nicer. And Python, of course.
Edit: this applies to any language not just Perl. I use it to learn Python things I would otherwise miss.
Curiously enough, Ventura ships with not one but two perls, 5.18 and 5.30. I haven’t yet investigated which tools there are using Perl. (It could also be that some prominent third party applications still call out to Perl, expect it to be there, and vendors asked nicely. How else would you explain the 5.18?)
It seems to be an apt illustration of the Lindy effect more than anything.
It was fashionable in programming discourse for people to parade their skillful opinions about things they never really understood. And it still is. ^_^
Even today in 2023, some people continue to disparage bash, awk, sed, grep, and Perl as though these tools form a common nightmare for all humanity. But these are very successful tools that solve problems quickly, efficiently in memory, and across many hardware platforms.
People are free to dislike the notation and to prefer Python or Haskell or Powershell. But some detractors simply behave in a bigoted way towards languages that they have little desire to learn. And as far as their objections to the notation are concerned, the same expressive problems find similar notational solutions in Powershell and the various DSLs that Python includes (e.g. numpy, pandas).
Bash, awk, sed, grep, and Perl are robust, durable, portable tools. Their developers were/are smart people. I've found that knowing these tools well has been important in my career. And I also code in Python.
EDIT: For the curious about the history, take a look at the perlhist documentation, https://perldoc.perl.org/perlhist it's got a lot of good info about the history of perl and it's releases.
As for as I know the only "official" reference to that is the "classified" and "don't ask" in perlhist.
Perl in one stroke collapsed the programming of C, text manipulation, the capabilities of all of the Unix utilities, and data structures into one system. For anything which wasn't subsumed into the monolith of Perl, you could easy access via backticks. It was very friendly in dealing with text streams, and that's what those call-outs in those back ticks spoke.
Yes, awk and sed were replaced by Perl, but more importantly, the unmaintainable nightmare that glued all of it together was wiped out.
Adding to that some popular examples are here [1][2]
[1] - https://www.commandlinefu.com/commands/matching/awk/YXdr/sor...
[2] - https://www.commandlinefu.com/commands/matching/sed/c2Vk/sor...
I still use awk and sed semi regularly. I haven’t used Perl in over a decade.
They've become much more useful because of the influence of things like perl.
To see what I mean, here's SunOS 4.1.1's man page for awk and sed as grabbed from https://github.com/ambiamber/Run-Sun3-SunOS-4.1.1/blob/main/... and ran through groffer(1) . These are from 1989 and 1987 respectively. The core stuff is there but not much else.
https://9ol.es/sed.1v.pdf
https://9ol.es/awk.1.pdf
I was incorrect about 30 years. It's actually 35.
You were the one that that said POSIX awk to begin with, I was using your terms. I understand what you meant, why don't you?
As far as shitting on the GMU tools, I don't think I've seen someone do that in decades, especially just referencing the bottom of the man page like that.
As far as speed, I don't know how you're using these tools, but if gawk is your bottleneck and not i/o, there's probably smarter ways to do things.
This is not a productive conversation. You can live life however you want and if you're happy than I'm happy. We are definitely at an impasse. I'm off to bed.
POSIX was initially a bunch of the Unix vendors getting together in the 80s seeing a rise of incompatibility saying "this shit is crazy let's find something we can agree on" and then the conflicts of trying to still have a differentiating But Also compatible product. They knew that if they cannibalized themselves, IBM Novell and DEC were there to snatch up the customers
As a result, POSIX was intentionally kneecapped by all the players but not enough to make it completely worthless.
By the 2010s I decided to simply ignore it and if something breaks on FreeBSD or whatever, make the decision then whether to support it or not. Mostly it's platform testing at the top and then subbing the gnu version of things and bailing out if it's not there.
Same. Awk and Sed are delightful little tools that have aged exceptionally well.
Other than that, I'm completely shocked that somebody would praise sed or awk in well into the 21st century. Turing-complete languages barely capable of properly solving any problem that has any sort of algorithmic complexity, let alone one that requires proper data types, variables or interacting with any sort of system. Really, you can do that in Perl (even in Raku) without your couple-of-liner horribly breaking down once you need some custom logic to it. Oh, and it will even resemble programming languages, not some hieroglyphs left behind by aliens.
I'm not in this space, but the similarity in names made me wonder if these would really have started around the same time, with an unclear "dependency order", so I reached for a search engine, and according to Wikipedia, CPAN (1993) is based on CTAN (1992).
There isn't "some question", there is a definite answer, namely in "Programming Perl" and I quote the 4th edition, page 629:
History
Toward the end of 1993, Tim Bunce, Jarkko Hietaniemi, and Andreas König set up the perl-packrats mailing list to discuss the idea of an archive for all the Perl 4 stuff floating around the Internet. Perl 5 development had started that year, and one of its main features would be an extensible module system that would allow people to extend the language without changing perl. Jared Rhine suggested he idea of a central repository, but nothing much happened. His idea had come from CTAN (http://www.ctan.org), the Comprehensive TeX Archive Network.
Edit: fix markup
I may not completely understand what you're describing, but with perlbrew managing different Perl versions, and various CPAN clients installing modules in versioned libraries - as well as the test/smoke servers that test CPAN on tons of versions of Perl for Perl CPAN authors automatically, I don't know if I've come across what you're describing.
What's still a pita is when there's an outside dependency from the Perl ecosystem that breaks a module, but I'm not sure that's Perl's fault.
This doesn't sound that horrific to me. It's the classic Unix approach of building small tools that do one thing well, and composing them in novel ways to solve problems. For any problem that can't be solved this way you write another small tool using your programming language of choice. Rinse and repeat.
But occasionally Unix attracts users and programmers who reject this approach, and who prefer building a monolithic tool, or in the case of Larry Wall, new programming languages. To be clear, I'm a fan of Perl and think it has its place, especially in the era it came out. It inspired many modern languages, and its impact is undeniable.
Personally, I find solutions you refer to as "unmaintainable nightmare" to be simple and elegant, if used correctly. No, you probably shouldn't abuse shell scripts to build a complex system, and beyond a certain level of complexity, a programming language is the better tool. But for most simple processing pipelines, Unix tools are perfectly capable and can be used to build maintainable solutions.
The classic Knuth-Mcllroy bout[1] comes to mind. Would you rather maintain Knuth's solution or Mcllroy's?
[1]: https://matt-rickard.com/instinct-and-culture
I have, and mentioned one lower down in the comments. Unix philosophy was great but does not scale well in terms of maintainability or efficiency. Invoking processes over and over again loops is godawful slow. And the horror of complicated shell scripts is legendary.
But this doesn't make this approach inherently wrong or obsolete. The programmer is wrong for trying to use the tools beyond their capabilities. Where that line is drawn is subjective, as is the concept of maintainability, but if you feel that you're struggling to accomplish something, and that it's becoming a chore to maintain, the path forward is choosing a more capable tool, like a programming language.
Perl was invented because the gap from shell to more capable languages was (and is) really big. Languages like Python and Ruby didn’t exist yet, and Perl had a really, really strong sweet spot in text processing.
Still does.
Ubiquity, speed, and conciseness.
Perl is usually installed by default on Linux and Unix systems. Ruby might be there, it depends.
Perl is faster than Ruby. Ruby has been one of the slower scripting languages. But Ruby has been working on performance improvements in the past few releases. I have not seen any benchmarks of the current Perl versus the current Ruby, so this may have changed.
Perl is more concise than Ruby allowing more functionality for less code.
The JVM would like a word. (It has slow startup, but can be very fast at runtime due to JIT optimization and cacheing.)
Scripts should start and run quickly.
Ruby has historically been fairly slow which is why Ruby 3 focused heavily on performance. It has been improving a lot, but I have not seen any benchmarks against other programming languages.
This results in weird behavior, such as writing a groovy (Java?) script for Jenkins to execute bazel in order to build a go binary that runs the very same commands in an exec.Command() construct. Or people who download and import pandas to grab the third field in a csv file.
During the course of learning, I've naturally written code in bash that should have been written in another language. I replaced if statements with case because they turned out to be more performant. It's a great learning experience and why I got into python and go.
IMO we should use the right tool for the job. Sometimes that tool is a combination of unix utilities that you can put in a shell script for easier maintenance. It's just procedural execution of (usually very efficient) binaries, akin to a jenkins script or gitlab pipeline. Just mind the exceptions and use exit codes.
* often times, it's not just the third column I want. Sometimes it becomes "third column unless the first column is 'b' then instead grab the fourth column". Having a good data representation makes sure that I'm not mixing logic code with representation parsing code
* I don't have to care about CSV parsing edge cases. Escaped comma? Quotes? I don't care, the library will either handle it or throw an explicit error. With custom parsing code, instead of an error, I'll get some mangled result in the middle of the file that I won't even catch / notice until later down the line
* when working with CSVs, in my area (ML / scientific compute), Python is often the right context to be in.
It's not a lack of being "taught" shell scripts. It's the fact that shell programming constructs aren't well documented, your "standard library" is basically dependent on whatever binaries happen to be available on the filesystem, error handling is almost non-existent, etc.
It's very easy to write a bad shell script that "solves" a problem as long as a bunch of assumptions aren't violated. In my experience, senior software engineers are extremely averse to hidden assumptions and very concerned with reliability of the systems they build.
Of course this can happen with any language, especially as it ages and adds complexity.
Everything was needlessly hard because these tools were not built for that. Easy to talk about philosophy and the "classic Unix approach" if you don't have to build modern applications this way.
You hit it on the head with the slowness of loops when the body comprises a series of program invocations. The horror really seeps in when you realize the original author wasn't stopped by the lack of data structures: they could get around that with some creative variable names.
Whippersnappers! :D
The first big iron I had the luck to work with was an IBM 3090 , essentially a gift from IBM, it handled the university entrance exams of the entire country of some ten million people and it had 64 MB of RAM. (It was also the first computer in Hungary permanently connected to the Internet via a leased line to Austria so it had an Austrian IP address. Hungary didn't have its IP region for two more years.)
I think the first machine with 128MB was a VAX 6510 a year or two later at another university. A little bit later, in 1994, CERN had gifted a VAX 9000 with an astounding 256MB of RAM.
To compare, the first server I installed Linux on had a grand total of 4MB RAM -- and that was one of the largest computers a small department at the university had.
It would be a long, long time before "128MB" and "mine" entered the same sentence.
4MB?!?
My first encounter with IBM kit was a, er, darn I'm not sure cuz I'm getting old, but I think it was a 4300? Not big iron in some senses, but still with a box that was something like 6-8 feet long iirc and definitely several feet wide and high. (And a bank of about 6-8 tape decks, each as tall as me, and two disk units, each the size of a washing machine, and so on.)
Its RAM? A massive 1 MB.
That IBM kit was the heart of the super new expensive upgrade in 1980 that cost something like 5-10 million pounds iirc to build, including a brand new building to house it and a team of programmers.
The older setup, which is where I was until its last days, was an ICL system that was expanded at the end of its life to a whopping 48KB -- yes, KB -- of RAM.
And that kit ran all the systems, internal (payroll, accounting, etc., etc.) and external (sales etc.) for the largest car dealership in the UK.
128MB? 4MB? Even 1MB? That was an unimaginably insanely large amount of RAM!
(Yes, it was very weird to be working with this physically enormous setup, and dealing with keeping it all cool enough not to halt for a half hour or so, through super human efforts when the A/C broke down, when the likes of PETs, Sinclair Z80s, and Acorn Atoms were a thing...)
Ha yes the aforementioned IBM 3090 was so big for installation they removed the roof of the building it was living in, craned it in place and put the roof the back. Bringing it up the elevator or stairs was impossible.
Much later, in the second half of the 90s, I remember the four of us carrying an IBM HDD -- I think it was your normal 5.25" drive but it needed four people because it was mounted on a vibration dampening base ...
Continuing the shades of Monty Python theme[1]:
I remember one of my first few nights being in charge of the new IBM kit (I was a "computer operator" back then, in 1980), leaning back in the fancy new chair at the desk with its fancy "virtual" teletypes (a couple "terminals" displaying the status of the OS with a CICS system), and showing off to an "underling" by swinging a long plastic slide rule or something stupid like that (I no longer recall), and me accidentally banging it on the desk. Right "near" a recessed big red button. Or perhaps "on" the button? As I snapped my head around to look at the button and begin to understand what I may have just done I heard an ominous series of whirring and clicking sounds coming from the cpu box, right near where there was an 8" diskette drive that wasn't supposed to be doing anything while the OS was running (it was just for starting the OS). Then I looked at the console... Uhoh. They didn't fire me but it took months before they decided to let me be "in charge" again with someone else actually hovering over me...
Fast forward to when I was a coder (BCPL) in a small software startup, during the second half of the 80s, presumably 10 years before you were carrying your 5.25" drive monster, I vividly recall someone bringing a 700MB hard drive back from a local computer store. It cost an astonishingly paltry 700 quid or thereabouts. A pound a MB!
[1] https://www.youtube.com/watch?v=VKHFZBUTA4k
I ... do not know. That sounds very low. Look at https://jcmit.net/diskprice.htm and note the pound was 1.8-ish around this time so the price should have been well above 1000 pounds even in early 90s. We are talking of a 5.25" full height drive, here's an 1987 model http://www.bitsavers.org/pdf/maxtor/MXT8760E.pdf rare in personal computers, it was definitely for workstations / servers.
That's what the O'Reilly books were for, especially the Nutshell series.
The one thing people can't possibly fathom if they started coding after the mid-late 90s was how much we relied on the printed medium.
Rather Waite-y.
By the time our lord and savior zsh appeared on the scene Perl was already at Perl 3. And, to be fair, I do not think many used zsh before 2.1 which was some time 1991 fall and by then the Camel Book was out for half a year or something like that. So the pre-Perl and the we-use-zsh days do not really overlap.
When Perl appeared the absolute hotness was the version of Korn shell what later became known as ksh88. https://github.com/weiss/original-bsd/tree/master/local/tool...
but the only free programming languages available at the time were C/C++, various shells, and awk. everything else was expensive or not generally usable for other reasons. all the really useful languages to build complex systems didn't really appear or become freely available until the 90s. and perl was first among those.
But the thing is that today the shell landscape is much more mature for solving simple problems, and we have C/C++ alternatives that are saner and more capable than Perl (e.g. Go). So it arguably has lost its place, as shell tools are still in widespread use, while Perl is mostly underused. Raku is interesting, but it goes in a different direction, and its adoption is practically zero.
This is by design. Readability is core to the design and philosophy of python. One liners are cool and fun to write, but trying to decipher someone else's incredibly dense bash or perl one-liner is absolutely awful.
You can write hard-to-read code in any programming language.
Python lets you with mandatory whitespace so that the awfulness spans multiple lines instead.
Really talented Python programmers can do downright demonic stuff with list comprehensions.
Python appears to be simple, but is actually quite complex. I recommend reading "Effective Python" (https://effectivepython.com/) to see beneath the surface.
Just think of C: I'd argue its design is actually more akin to Python than Perl (and it definitely inspired languages like Go and Zig, NOT languages like C++). It's a small language and this is a very important characteristic of it: you can count on being able to actually master it or at least very well comprehend it. Other effects of a simple and literally straightforward language can be: easier implementation and evolvement, less mental load on the developer, easier portability among developers (both for general knowledge and actual code), etc. I'm not saying that C is the way it is for all these reasons but I wouldn't overlook this factor and I do think that languages like Python are deliberately building on these advantages.
Now, I don't dislike C++ at all but back when I studied it at university, I noticed that it was the first language for me that needed to be actually studied, unlike Pascal, C, Python and "oldschool" JS. Ever since, the only languages where I felt the same were Prolog (mostly because it requires a different mindset; other than that, it didn't seem bloated) and Raku. Not C#, not Java, not Erlang. I didn't really have to touch Perl but from all I know, Raku started off as a fresh take on the Perl approach. It seems somewhat more organized but huge nevertheless, to the extent that there literally isn't one person who really "groks the language" all around. In the case of Raku, I wouldn't even say it encourages you to write unreadable code (especially if you have a thing for APL look-alikes, lol) - it's just so rich that there is a good chance you will come across something in someone else's code you have never used before and don't quite remember how it will act in your specific use case.
There are different types of freedom than "do whatever you want". Like, the freedom to feel safe and confident about code. These days, humanity has aggregated an immense amount of knowledge and technology and "I will do it all by myself" is not that much of an option. And even people with such puritanistic tendencies will choose simple and straightforward tools, even if not for the "limitations".
Now I have barely spent any time with the oldschool Perl but trust me, I have put a lot of effort into learning Raku, the language that was meant to fix Perl. Whenever something that "seemed like a good idea at first but it's actually harmful" shows up, it's usually Perl's legacy. I'm thinking of things like the conceptual mishmash between a single-element list and a scalar value (or in general, trying hard to break down variables arbitrarily into list-alikes, hash-alikes and the rest of the world), the concept of values that try to implicitly pretend they are strings and numbers at will, or the transparency of all subroutines to loop control statements which is some next level spaghetti design. If you ever actually use something like this, you introduce a brand new level of complexity, somewhere inbetween a "goto" and a "comefrom", so I would really think about if this was worth learning at all.
Oh right... from what I remember, it was also Perl that fostered this idiotic idea that a name of a concrete thing could be overloaded to be a namespace as well, and a concrete Foo::Bar could very well be something that has no logical relation to a concrete Foo. Moreover, I'm quite sure Perl invented this nonsensical distribution-module dichotomy where you are supposed to depend on modules, despite the smallest publishable and installable unit being a distribution. There are three outcomes with that: - if the distribution contains only one module: what was the point of drawing the distinction? - if the distribution contains tightly coupled modules: you can pretend to only depend on one of the modules but in fact you are depending on the whole distribution together - if the distribution is a collection of unrelated modules: why are you trying to encouple the metadata when this will make the versioning meaningless?
I can only hope that it's somehow better than Raku but the whole principle is just an anomaly.
And you know, then these people move around in the world, pretending that all of this is just normal and you just have to learn it. Well guess what, there is a reason people might want to put that effort into something else.
I think it's possible that things that seem normal and inoffensive can become horrific simply from scale. You'll climb the stepladder without complaint, but then there's that radio tower in Canada...
Others like the idea of the family cow, then they see the 10,000 head feedlot from the highway.
Scale is sometimes sufficient by itself to induce horror.
If you can depend on a recent bash and use shellcheck, then it's actually quite a pleasant programming environment, with fewer footguns than one might think. (I want a @#$@# "set -e" equivalent that returns non-zero from a function if any statement in the function results in non-zero).
There are some things that are more awkward than they should be though (e.g. given a glob, does it match 0, 1, or many files, or the way array expansions work).
Also, there's no builtin way to manage libraries (I don't know about Perl, but Python suffers from this as well). This results in me pasting a few dozen lines of shell at the top of any of my significant shell scripts, for quality-of-life functions. Then I have to use "command -v" to check if the various external programs I'm going to use are present. Say what you will about C, but a statically-linked C program can be dropped in anywhere.
As for managing libraries, that's true, but you can certainly import and reuse some common util functions.
For example, this is at the top of most of my scripts:
This loads `lib.sh` from a common directory where my shell scripts live, which has some logging and error handling functions, so it cuts down on repetition just like a programming language would.Moreover, it's existence being required explains why my error handling, recently, wasn't working as expected.
This works really well if your problem can be solved in one or two liners.
It go bad very quickly when, say, you have two CSV files and want to join them the sql-way. In sed, you have to use positional variables and think about shell escaping. In perl, you can at least name those variables and use \Q
My personal comfort threshold is around the 100-line mark. It's even possible to write maintainable shell scripts up to 500 lines, but it mostly depends on the problem you're trying to solve, and the discipline of the programmer to follow best practices (use sane defaults, ShellCheck, etc.).
> It go bad very quickly when, say, you have two CSV files and want to join them the sql-way.
In that case we're talking about structured data, and, yeah, Perl or Python would be easier to work with. That said, depending on the complexity of the CSV, you can still go a long way with plain Bash with IFS/read(1) or tr(1) to split CSV columns. This wouldn't be very robust, but there are tools that handle CSV specifically[1], which can be composed in a shell script just fine.
So it's always a balancing act of being productive quickly with a shell script, or reaching out for a programming language once the tools aren't a good fit, or maintenance becomes an issue.
[1]: https://miller.readthedocs.io/
So I don't disagree that it was needed back then, but it's important to mention the modern context it struggles to exist in.
I agree that there’s nothing wrong with composing UNIX tools. I mean, that was one of its key selling points. if you watch any early promo videos for UNIX you’ll see them talk heavily about the composability of the command line and shell scripting. It wasn’t an accident — it was designed that way.
The point of the conversation wasn’t to say that one shouldn’t write shell scripts, it was just to say that there was a massive and unfilled gulf between what was easy to do in Ksh, awk and sed, and what could be done in C.
Then just put them in a database and write a simple SQL query. If you use Perl it’s really very simple to do.
That sounds like a perfect use case for `join`.
That said, imagine a metrics system for a huge networking company that used these methods to cover all automated testing or defect analysis. Those inner loops were made of greps and seds and so forth, and each one is the invocation of a new program. It wasn't uncommon for these runs to take almost a day.
Besides performance, the other nightmare was was someone described below: each script was a one-off that didn't leverage the work from others. If the author only new C shell, then you know you're going to be doing gymnastics to catch the stderr of some of those programs (you can't capture it in the same manner that Bourne variants do).
Anyway, yes, we all adore the Unix philosophy, but there are limits.
I remember different CLI tools working differently on SysV and other variants. I remember AIX and HPUX.
Perl meant one thing that just goddamned worked mostly the same way. That and vi and emacs. :-)
Er, Perl replaced an unmaintainable nightmare? Perl, the language infamous for being indistinguishable from line noise?
If you have doubts, try writing a complicated production software stack in awk. Then hand it off to a coworker.
One need not be irreplaceably good, if one is already beating the current state of the art by an order of magnitude. Perl did this.
I've easily written tens of thousands of lines of Perl, and not a single person has complained about difficulty reading or maintaining that code. Why? Because I apply all the usual best practices for code hygiene that apply to any language.
Frankly, I think most people are just repeating a meme they heard once, and the rest just get put off by a) sigils, and b) the use of implicit variables (which I tend to use only very sparingly, and mostly in quick one-liners).
But a developer that poorly names their functions or variables, fails to modularize appropriately, fails to document their code, abuses language features because they want to be excessively terse or clever, that kind of person is gonna write crappy, difficult-to-maintain code no matter what language they use.
In fact, I'd argue the advent and popularity of Python--which was pretty radical in how opinionated it was at the time--is a direct response to languages like Perl and C that were a lot more free-form and easier to abuse by poor coders.
https://packaging.python.org/en/latest/overview/
<kidding> OCI (Docker) Containers were invented to overcome the problems inherent with deploying Python applications. </kidding>
I would argue the same of sed/awk; you can write an unmaintainable mess, but you don't have to.
Developers extremely seldom complain about other developers' code to their faces.
Instead they go to lenghts to avoid it if they dislike it severely; right or wrong.
Get the Perl source code for PangZero and you'll understand proper Perl code.
PERL being open source meant you were a "make" away from having the same environment on all those unixes.
even today when the dozens of UNIX distributions have collapsed into a couple of Lunux flavours, getting a shell based script to work reliably between say macOS and redhat can take some effort.
PS is 'okay' as a scripting language, but it's very frustrating how Microsoft's security defaults seem to be dead-set on ensuring you can't use scripts on systems without jumping through a bunch of hoops. I get how MS got to this conclusion that it needs to be locked down, but it makes me think twice about pumping out a script since I know I might need to walk my colleagues through how to actually run the damn thing depending on what the Security Admins decided to lock down that week.
I barely get into loops and lists when trying to learn Python before losing interest/focus and not trying again for months. It feels very abstract and like it's not useful for day-to-day stuff without learning and writing a lot. Shell stuff is very short and powerful and feels more grounded in reality. To be clear, I am not using if/else and the like in my shell scripts either. Usually they're just a few lines with little to no logic, but they're immensely useful. I can't imagine learning perl and then using it for all this stuff.
Through my use of Usenet I learned that a better reader program was available called 'rn', which I downloaded and built. It had an amazing handwritten install script (autoconf was years off) which could automatically configure and build rn on any *NIX system. All developed by a fellow named Larry Wall.
rn was truly a joy to use and made reading Usenet swift and efficient. It would get updated with the fixes that came across Usenet that I'd apply using the clever 'patch' program, also written by Larry Wall.
Based on my experience with his other software, when Larry Wall released Perl on Usenet I immediately downloaded, built, and started using it. As promised, for scripting things not requiring a C program, it was massive improvement. Version 2.0 came out and brought many great new capabilities.
I wasn't writing software while versions 3 and 4 came out; I started using it again after version 5 and the appearance of CPAN. Over the years I've used Perl extensively for task automation and data wrangling.
Python now dominates Perl's niche because it's easier to learn and interfaces better with C. It's also less flexible, which compared to Perl is a virtue. One of Perl's mottos is TMTOWTDI—there's more than one way to do it. But many of them are bad. Much of Perl's poor reputation ("line noise") stems from this.
But when Perl was released it was a revelation, like a drab day when the clouds suddenly part letting warm sunlight pour down on the land.
Every library I needed was at my fingertips on CPAN. And just… worked. Connect to Oracle, run SQL, create a CSV, FTP it somewhere. EZPZ.
I started with Perl CGI and have been less productive every year since then.
CPAN alone is worth the price of admission. I started my career using Perl to build Excel spreadsheets from queries I ran against Oracle DB. That may sound horrible but it was actually plug and play. Really simple and just worked. I could even fire off emails from the script. Just zero hurdles (or guard rails).
The DOM API is also a bit more robust than string manipulation.
But it was a critical part of computing history. Perl did a tremendous job of bridging the gap between shell commands and shell scripts and “big languages”. Way back in the early 90’s I took a complicated shell script that took about 2 hours to run this huge text processing job , rewrote it in Perl and it took about 20 seconds, and was more maintainable to boot.
It really helped boost software development in the 90s, even pre-CGI script.
The Perl source itself was also very interesting, with numerous optimizations to make it fast (again, for its time).
What killed it were the never ending eccentricities you ran into, endless foot guns, and of course Perl 6 never being delivered.
But it really was the weird shit that did it in.
*technically it was 5.005 or something to 5.6 because they decided to drop a zero or two
And it worked, sort of, for a few years.
I was on the Perl Grant Committee at the moment that killed Perl 6 in my opinion. We had a choice of 2 grants from Nicholas Clark that we could fund during 2006. One brought a lot of immediate benefits to Perl (Unicode fixes, reduced memory usage, etc), and the other was to get Ponie to the point where someone other than a core Perl developer could work on Perl 6.
We chose the immediate benefits for Perl 5.
They did not add support for encoding/decoding html entities or URLs were added. No standard modules for this.
No easy method of making html pages from a URL request besides running as CGI.
This made perl lose to php which made it very easy to make a simple "Hello, world" page.
php would never had any traction if perl developers had made web support high priority.
http://www.masonhq.com/
For years Amazon.com used Perl Mason.
Big PHP shops like Facebook, and PHP projects like WordPress, came in a later development generation for the simple reason that PHP made it easy to get started on shared webhosting.
It's certainly not in core, but look how PHP flubbed that up (html_entity_decode, htmlspecialchars_decode, htmlentities).
>No standard modules for this.
Obviously there are,
https://metacpan.org/pod/HTML::Entities
The changelog goes back only to 1998, where it states,
That wasn't a hurtle to developers, but certainly to end users who did want to just throw a php script up and have it work.Netscape Navigator was released late 1994, so this was the point of time where perl developers should have seen the light.
The thing that PHP did better than Perl was the ability to be set up on shared web hosting. Mind you, it did that mostly by not acknowledging all of the security holes that it had which allowed one user access to what should have been private for another user. But shared hosting providers had a population of people who wanted to use PHP, were forgiving of major security flaws (which PHP had many of), and would pay money.
Meanwhile Perl went the route of doing things right, and being much more secure. Which, even though it feels important to good developers, is bad for marketing. It was a classic Worse is Better situation. See https://www.dreamsongs.com/RiseOfWorseIsBetter.html if you're not familiar with that concept.
And, in the end, the commodity wins. So in the mid-2000s, PHP did catch up and beat Perl.
Well, I thank you.
Is there a summary of what Tom Christensen did? I hold him up pretty high when it comes to Perl lore. I don't know from what you wrote if he's the hero or villain in this case (and if you don't remember that's fine: ancient history)
I don't know everything that Larry did about the situation. But it was a situation that everyone on p5p was painfully aware of at the time.
https://en.wikibooks.org/wiki/Raku_Programming/Perl_History#...
And yes, I deal with non-ASCII Unicode every day.
>> But it really was the weird shit that did it in.
The current state of Perl 5 for Python fans: ----
Perl 5: I'm not dead!
TIOBE: 'Ere! 'E says 'e's not dead!
Internet: Yes he is.
Perl 5: I'm not!
TIOBE: 'E isn't?
Internet: Well... he will be soon-- he's very ill...
Perl 5: I'm getting better!
Internet: No you're not, you'll be stone dead in a moment.
TIOBE: I can't take 'im off like that! It's against regulations!
Perl 5: I don't want to go off the chart....
Internet: Oh, don't be such a baby.
TIOBE: I can't take 'im off....
Perl 5: I feel fine!
Internet: Well, do us a favor...
TIOBE: I can't!
Internet: Can you hang around a couple of minutes? He won't be long...
TIOBE: No, gotta get to Reddit, they lost nine today.
Internet: Well, when's your next round?
TIOBE: Next year.
Perl 5: I think I'll go for a walk....
Internet: You're not fooling anyone, you know-- (to TIOBE) Look, isn't there something you can do...?
Perl 5: I feel happy! I feel happy!
----
But seriously, Perl is not dead yet.
It is usually installed by default on most Unix and Linux systems.
It also rides along with most MinGW-based toolkits like Git for Windows. If you do software development, there is a good chance you might have Perl installed and not even know it.
If you are on a Linux or Unix system try running 'perl --version' to see what version you have installed. I would be very surprised if it is not there.
Some people like to make fun of it despite its wide-spread use and utility. Most of them probably know little to nothing about Perl and have not written or maintained any Perl code. There is a good chance that they are using Perl code without knowing it.
I still remember seeing this line from the configure script scroll by.. Made me curious what Eunice was, which was harder than it might seem, given that this was before the web and any search engines.. (and yes, it turns out Eunice was a unix-ish environment on VAX/VMS)
Edit: one plus is that perl is as ubiquitous as awk and sed. It was there the whole time and I didn’t even realize it!
Perl gives you all the simple features from sed or AWK and adds useful (maintainable) foreach iterators, arrays, hashes, etc.
Recommend using Strict mode if you are new to Perl. It gives good guardrails against silly mistakes like not declaring or misspelling a variables, or accessing strings or numbers that aren’t the correct datatype.
where {} can be any bracket pair.
This was a major reason I downloaded Perl 1.0 off Usenet in 1987.[1]
[1]https://news.ycombinator.com/item?id=36657782
A lot of the greatness of Perl came from writings of Larry Wall, Mark Jason Dominus, Randal Schwartz, Tom Christiansen. There is so much wisdom to be gained from all their code and documentation.
Even though I worked on a lot of Unix boxes doing assorted data processing, Perl was never my goto for the same reason I adopted vi instead of eMacs.
As good as it was, it didn’t come stock on my clients machines, and I could not presume to install it or count on it being there. So, the litany of classic Unix tools prevailed and I simply became adept at working with those.
My few temptations to dip my toes in the modern (at the time) Perl waters just found indecipherable source code (to my ignorant eyes) and disastrous attempts to get whatever it was I was dabbling with out of CPAN. I was never very successful with it.
[0] https://www.perl.com/article/perl-one-liners-part-1/
[1] https://www.perl.com/article/perl-one-liners-part-2/
Something that makes workloads usually delegated to find, grep, sed and awk coherent and easy to use, allows streaming data between functions/commands and can still be used to glue things together.
Can I use Perl instead of bash/as my command line ? Should I start learning Perl?
No. Perl is not a command line shell just as Python and Ruby are not.
>> Should I start learning Perl?
If you are doing workloads with the command line tools you list above (find, grep, sed and awk), yes.
If you are familiar with bash, sed, and awk you will find Perl familiar-looking and very handy.