While CGI scripts written in perl and shell script were all the rage, I remember using C (and ODBC) quite fondly and successfully. It wasn't until I encountered the webscr.cgi at PayPal that was more than a GB compiled and spanned millions of lines of C language that I realized the error of my ways.
No, that definitely happened. I wasn't there, but I'm pretty sure I've seen discussions from multiple companies about how they managed when their C based CGI/FCGI/HTTP+logic executable got too big to fit in 32-bit memory space; for some, amd64 came in time, for others, they had to split things out, or do terrible stuff with PAE. Adjusting the user/kernel address split was also common, IIRC.
One of the best aspects of CGI is how easily tested a CGI program is. All you have to do is set a couple environment variables and connect your request entity to stdin, expecting things on stdout. What could be better?
I remember creating a web query form for an employee search or something like that using CA Clipper and reading stdin an spitting our HTML on stdout. Worked like a charm, was fast and made that data available to whoever needed it.
That's not the fault of CGI but rather of the people who wrote bad code. You can easily have the same problem in your NodeJS express app if you're careless!
CGI still work great if you know your constraints and for whom you're building stuff. It is underrated alongside its cousins FCGI and SCGI. A ton of little tools work great as CGIs.
Shameless plug: I wrote this small CGI script in bash to serve stuff from my home server with auto-expiring links: https://github.com/tzahola/share-link
This feature is one of those things that proves that the technology industry is bullshit.
There's an entire industry, probably worth hundreds of billions of dollars, built around trying to detect insecure code and force it to be less insecure. Meanwhile, this one old language that nobody uses has a single option you enable to be [more] secure by default. And I'll bet you a billion dollars the reason other languages don't have it is "the design doesn't look very clean".
If anything no input interpretation should happen, everything should just go directly to the program, and the program should not be capable of file operations. stdin/stdout only.
Why wouldn’t it be safe? Many scripts I have are only available internally so don’t need to worry about people hacking them (unless they want to risk being fired to look at say syslog messages without having to ssh into the box).
Perl -T helps ensure there’s no tainting of variables - you take the parameter that’s passed and regex pull out the matching pattern you need to use. If your variable is a match of ([\d]+) then it’s only going to have numbers in, and you can use it fairly safely
yes, in fact the security model of Apache mod_cgi(especially with mod_userexec) will isolate scripts to an single user, which is also the level most modern reverse proxy based setups offer.
The problem was that the newbies writing cgi scripts in 1993 generally did not have much experience with input sanitation and had very few libraries to lean on to do it for them.
I had no idea what I was doing back then - copying scripts from MSA, no idea what chmod or ftp binary mode or even what the unix system I was telnetted into was (couple of years before). Managed to get things working though.
No mention of server-side includes, which were great -- you could include a nav bar on all your pages and just have one file to update, you could include a 'last updated' line at the bottom based on the file modification, you could include a visitor counter.
I did a website for a local art+book shop back in 1997, it was awful. Stopped updating it after 6 months, but it was kept online for some reason - even after the shop closed. archive.org shows the SSI counter was still incrementing in 2005. I assume the (copy-and-paste) code ran a script that opened a file, incremented the number, wrote it back to the file, then printed the number.
CGI was a wonderful paradigm for when I was first learning web development, a few years after this. A shell account somewhere, some scripts that just had to deal with stdin and stdout, and you could create amazing web experiences. Previously I only knew BASIC, but CGI let me start to share my weird little creations with the world.
There should be more simple ways for young people to write and share programs with each other. Maybe that’s what Roblox is, to an extent? I haven’t used it myself.
I do have fond memories of the simplicity of CGI, but also remember the big speedup and scalability improvement when we swapped over to NSAPI (no more forking, pooled processes, etc).
Kind of interesting that "serverless" (AWS Lambda, etc) seems to be going through the same evolution now. That is, initially liking the simplicity then hitting a wall with performance and making it more complicated.
Ah, yeah...mod_perl did make a nice bridge for Perl stuff. I had mentioned NSAPI because we had C++ CGI and NSAPI was available before things like FastCGI.
Been thinking this recently, lambda is a return to CGI, you provide a small program that runs and provides its results to something which sends them back to the client.
Sure. Wikipedia mentions FastCGI initially being a reaction to NSAPI.
"Open Market originally developed FastCGI in part as a competitive response to Netscape's proprietary, in-process application programming interfaces (APIs) (Netscape Server Application Programming Interface (NSAPI)) for developing Web applications"
Oh, I'm showing my ignorance here - I didn't know NSAPI was in-process. The Netscape server software was expensive and kinda hard to find in Europe back then. I guess FastCGI was an improvement in terms of stability - I'm assuming the NSAPI was .so/.dll-based? So if you messed up, the entire server went down.
Back in the mid 90s I worked on the Spinner/Roxen webserver that was built on an interpreted C-like language modelled after the LPC language being used for MUDs. Modules were were run in-process, but because of the interpretation a failure resulted in an exception/error, rather than a full crash. This software didn't get very popular - we sucked at marketing.
Most people kept the in-process stuff down to something that worked like Fast-CGI, just proxying requests to some non-NSAPI persistent process over a socket.
I loved spinner! It was one of my favorite web servers, even if I never did learn enough pike to do anything useful. But the graphical web server configuration interface was amazing for its time.
Same. Wrote a complete database-driven web application in C, an online bookstore. Switching to Perl 4 was a vast improvement in terms of development productivity.
C-based CGI programs that implemented a web app for doing drop shipment of phones to customers. It was supposed to be used for just a few months while they waited to roll out a more official solution. The users hated the official one, and kept using my version for another 4 years.
Same. My first C cgi program was a small Tic Tac Toe game that kept the current state in the request string. It had a little logic to play against the user. It did not, however, support number of players 0.
Mine played Othello using the same mechanism. It was cool because every square that was a valid move would change the cursor when hovered over. The CGI would make the players move, plan the computers counter move, return the board showing the players move with an automatic reload after X seconds to the URL encoding the game state plus the pre-planned machine move. Used a html table ang gifs. So fun.
I used Lincoln Stein's CGI.pm module (https://metacpan.org/pod/distribution/CGI/lib/CGI.pod#CGI.pm...) all the time back then. I wrote a Perl module for SMIL which I called SMIL.pm and emailed him to tell him and was thrilled when he responded. I'm strangely saddened to hear it has been removed from Perl.
It’s still available[1] (and maintained) for those that need it, but there are much better tools available for web work these days, like Mojolicious and Dancer.
I feel like CGI Perl scripts back in the late 90s was definitely a critical reason for me picking up on programming, web dev (as a high school kid), and eventually becoming a professional web developer, and software engineer.
The whole web dev paradigm back then starting from basic HTML files, to client JS, to server-side Perl CGI scripts, was just really easy to wrap my head around, and actually create usable products out of. It maps well to a Windows OS file system -- you have these HTML files sitting in folders, you can write them locally and open them locally to see them. Upload to a server via FTP in the exact same structure, and see it work on the internet. If some files are CGI scripts, they run some code on server before output. Input from browser comes in as stdin in the script, and output to browser is just stdout. It was simple enough that as a high school kid, I was able to start one of those "webmaster services" (providing guestbooks, counters, etc.). I have nothing but fond memories building scrappy stuff back in those days.
Not OP but things got way to complicated and not much better IMO. Usability has gone down and the number of ecosystems and methodologies has grown and continues to grow at a pace that makes few people want keep up with the latest trend. And things are only accelerating now with wasm. Featurewise the web technology has grown enormously though and one can live inside the browser and get all their needs met but I wonder if this is the direction we really want to take. Living inside the browser is also more distraction prone, not very compatible with the ones who want to do deep work
I started in the early 90s with CGI, I think modern day stuff is really good despite all the complaints. I often think people look back with rose tinted glasses, or remember specific things that were really good while forgetting the general trend of everything at the time. We ( or at least I ) were a lot more forgiving as the new new things enabled stuff we never had before, now expectations are a lot higher.
The only real objective thing I can say about it is that much of the modern ecosystem can have a pretty steep learning curve. Frontend alone, React is quite hard to grasp for a beginner. I'm a Svelte fan and I think it's a lot easier than React, but even then it's no competition to just plain old HTML files and some light JS.
On the backend, now you have this fragmentation in Node, Python, PHP, Ruby, Go, Elixir, etc. In the Matt's Script Archive days, CGI scripts were pretty much only Perl. There were other languages used by some, but they were rare. If you wanted to learn Perl CGI scripts, you pretty much downloaded MSA as well as tons of other free scripts and read them, tweak them, play with them and figured things out. (Not unlike how you simply View Source on frontend to learn HTML and JS)
Bigger sites would use c/c++ and actual shell scripts weren't unheard of either, but that was presuming you actually had an full unix account, which with the arrival of mod_PHP usually werent the case.
Todays deployment pipeline however have gone completely off track with all of the "large team, big cluster" overhead enforced on even small agile part time projects that don't need even 1 full server.
I recall writing CGI scripts in perl. I met some guys are work that used C and I was shocked. But not as shocked as my boss. It took them so long to deliver applications compared to cranking out perl.
Sure, that's also how PHP started. Thing is, with Perl and PHP, before mod_perl and mod_php, your script needed to be parsed for every single uncached request, easily dominating everything else, like URL parsing and process creation time. Hence natively compiled CGIs weren't such a bad idea. They're even used today a lot, with Apache/CGI uptimes measured in decades since neither Apache nor CGI binary updates require planned downtime on Unix. Its ultra-robustness is one of the reasons the concept of FaaS has become fashionable again.
Well, AWS kinda has made it so that almost every millisecond of CPU time (or other resources) is considered billable. While in the olden days you could run some scripts in some unused corner of a server for "free", with clouds that is less likely.
Services like Lambda are basically single process per request. They just pre-fork instances under load, similar to Apache. Because of the not insignificant instrumentation and other overhead in cloud architectures (e.g. Lambda uses VMs for isolation, Cloudflare uses WASM), a small, natively compiled CGI app probably uses less CPU and I/O than these modern solutions as starting a new process in Linux from an already cached copy of the executable doesn't really have much overhead in comparison to the layers of software involved in some of the aforementioned options, despite their other optimizations.
A lightsail server can cope with 500 or so people that need to use my server every day for $10 a month. It’s not worth thinking about the price, just throw the script up and job done. Even if you could spend a couple of hours making it twice as performant it wouldn’t matter.
The linksys WiFi router I was using until last year used cgi app for the admin interface.
My first web application was developed in 1999 in IISAPI (VC++) running obviously on IIS. I had to write a rudimentary session storage (in plain txt file!), because it was not provided by the framework.
Yep. I love Go, but I am certain it would not be half as popular as it is were it not for those simple statically linked executables that can be deployed anywhere quickly.
Actually writing CGI scripts was a bit before my time, but I have very fond memories of installing and using Newspro and Imagefolio on my old web pages.
When I got to college and got online in 1995, I pretty quickly learned about CGI scripts and the cgi-bin directory, which of course my school did not have enabled on campus-wide accounts (security risks). I really wanted cool stuff like a web hit counter on my page. A lot of web hosts at the time would advertise something like "5MB of space and your own cgi-bin!" and I eventually got to one of these, got into Perl, and start making stuff using Berkeley DB and other on-disk databases. Eventually I outgrew this, found PHP & MySQL, found a web host who supported it, went to work at that web host, ... All said, CGI's were totally the reason I got into internet programming.
I still have cgi scripts running, I assume, somewhere.
Sometimes you just need a single file that does a few things and mostly behaves like a webpage. It doesn't need to be deployed or managed or anything. I had this one that served approximately seven requests on average for over a decade. It's simple. It doesn't need any love. It's in the documentation for the overall site because I wrote the documentation, but it isn't "omg this needs an APP and a BACK END and we have to worry about SCALING."
I still have a book somewhere on CGI with Perl and C, from the 1990s. Probably due for a purge.
OK, looks like my memory isn't that good. panda came out in 1998. At the time I was working for a company that paid Y2K bonuses to dissuade us from quitting and moving to the Bay Area. I often wonder how different my life would be if I had done that.
Who am I kidding? I was having way too much fun writing motion control software to do any of that Web crap ;-)
In the early 2000s, Sun's patching support website, sunsolve, was a massive Apache/CGI empire. Perf. problems were rife and by early 2002 the whole thing was grinding to a halt under traffic.
It just happened to be intern season. In walks some fresh-faced kid and installs mod_perl. Like magic, the entire website stands up, dusts itself off and starts working again. CPU load drops by two thirds. I think the senior engineers were kind of pissed but couldn't really say anything. I think the kid got a beer or two out of the whole thing. (And of course got an offer to stay.)
Hmm, I don't have any negative associations with SSI, seems like similar ideas have existed in lots of more recent platforms (like partials in Rails, e.g.). Static site generators are doing something similar, just in advance?
99 comments
[ 2.5 ms ] story [ 153 ms ] threadThere's an entire industry, probably worth hundreds of billions of dollars, built around trying to detect insecure code and force it to be less insecure. Meanwhile, this one old language that nobody uses has a single option you enable to be [more] secure by default. And I'll bet you a billion dollars the reason other languages don't have it is "the design doesn't look very clean".
Perl -T helps ensure there’s no tainting of variables - you take the parameter that’s passed and regex pull out the matching pattern you need to use. If your variable is a match of ([\d]+) then it’s only going to have numbers in, and you can use it fairly safely
The problem was that the newbies writing cgi scripts in 1993 generally did not have much experience with input sanitation and had very few libraries to lean on to do it for them.
No mention of server-side includes, which were great -- you could include a nav bar on all your pages and just have one file to update, you could include a 'last updated' line at the bottom based on the file modification, you could include a visitor counter.
I did a website for a local art+book shop back in 1997, it was awful. Stopped updating it after 6 months, but it was kept online for some reason - even after the shop closed. archive.org shows the SSI counter was still incrementing in 2005. I assume the (copy-and-paste) code ran a script that opened a file, incremented the number, wrote it back to the file, then printed the number.
There should be more simple ways for young people to write and share programs with each other. Maybe that’s what Roblox is, to an extent? I haven’t used it myself.
Kind of interesting that "serverless" (AWS Lambda, etc) seems to be going through the same evolution now. That is, initially liking the simplicity then hitting a wall with performance and making it more complicated.
There was a transition period where people were trying to still use CGI but make it scale so you had things like CGI mod_perl in Apache.
See also: https://news.ycombinator.com/item?id=24683304 ("FastCGI – The Forgotten Treasure (2002)")
"Open Market originally developed FastCGI in part as a competitive response to Netscape's proprietary, in-process application programming interfaces (APIs) (Netscape Server Application Programming Interface (NSAPI)) for developing Web applications"
Back in the mid 90s I worked on the Spinner/Roxen webserver that was built on an interpreted C-like language modelled after the LPC language being used for MUDs. Modules were were run in-process, but because of the interpretation a failure resulted in an exception/error, rather than a full crash. This software didn't get very popular - we sucked at marketing.
https://docs.oracle.com/cd/E19146-01/821-1833/index.html
Tricky to follow where all the stuff ended up: https://en.wikipedia.org/wiki/IPlanet
I haven't written it up yet, but basically it's things which are supported by almost every browser and/or almost every web server.
Other things on the list are parts of HTML 3.2, like <p> tags, the standard format of access.log, HTTP/1.1, etc.
I try to not use anything outside of that list in my projects, some JS things being an exception (which I feature-check before using)
While I haven't written any significant Perl in a long time, I will always appreciate how much it improved things for me back then.
After that, it was Java all the way.
[1] https://metacpan.org/release/CGI
https://cgi-lib.berkeley.edu/
The whole web dev paradigm back then starting from basic HTML files, to client JS, to server-side Perl CGI scripts, was just really easy to wrap my head around, and actually create usable products out of. It maps well to a Windows OS file system -- you have these HTML files sitting in folders, you can write them locally and open them locally to see them. Upload to a server via FTP in the exact same structure, and see it work on the internet. If some files are CGI scripts, they run some code on server before output. Input from browser comes in as stdin in the script, and output to browser is just stdout. It was simple enough that as a high school kid, I was able to start one of those "webmaster services" (providing guestbooks, counters, etc.). I have nothing but fond memories building scrappy stuff back in those days.
On the backend, now you have this fragmentation in Node, Python, PHP, Ruby, Go, Elixir, etc. In the Matt's Script Archive days, CGI scripts were pretty much only Perl. There were other languages used by some, but they were rare. If you wanted to learn Perl CGI scripts, you pretty much downloaded MSA as well as tons of other free scripts and read them, tweak them, play with them and figured things out. (Not unlike how you simply View Source on frontend to learn HTML and JS)
Todays deployment pipeline however have gone completely off track with all of the "large team, big cluster" overhead enforced on even small agile part time projects that don't need even 1 full server.
Sure, forking a process per request isn't fast... but I'd love to have enough traffic for that to matter.
There’s literally 20 people that have client certificates that can access my page. I don’t care if they fork a process each.
My first web application was developed in 1999 in IISAPI (VC++) running obviously on IIS. I had to write a rudimentary session storage (in plain txt file!), because it was not provided by the framework.
Modern Web Development are needlessly complicated. Both Front End and Back End. Oh... and deployment.
Sometimes you just need a single file that does a few things and mostly behaves like a webpage. It doesn't need to be deployed or managed or anything. I had this one that served approximately seven requests on average for over a decade. It's simple. It doesn't need any love. It's in the documentation for the overall site because I wrote the documentation, but it isn't "omg this needs an APP and a BACK END and we have to worry about SCALING."
I still have a book somewhere on CGI with Perl and C, from the 1990s. Probably due for a purge.
Or maybe not. I seem to have gotten out of the web business at the right time. It's gone in a direction that makes me grind my teeth.
"Before we had these nifty LED bulbs, we burned candles."
OK, looks like my memory isn't that good. panda came out in 1998. At the time I was working for a company that paid Y2K bonuses to dissuade us from quitting and moving to the Bay Area. I often wonder how different my life would be if I had done that.
Who am I kidding? I was having way too much fun writing motion control software to do any of that Web crap ;-)
It just happened to be intern season. In walks some fresh-faced kid and installs mod_perl. Like magic, the entire website stands up, dusts itself off and starts working again. CPU load drops by two thirds. I think the senior engineers were kind of pissed but couldn't really say anything. I think the kid got a beer or two out of the whole thing. (And of course got an offer to stay.)
Good riddance.
I did write an entire CMS in PERL, once.
I still wake up screaming...
Static site generators don’t generate “live,” like SSI did.