I once got a term sheet on the basis of a "website" which was a quick-n-dirty lash-up of CGI, cron, and email. Low-code can be done not only with new tech but also with old.
I would ocassionally see dynamic web pages with .exe extensions before 2000s. I assume Windows IIS was also CGI capable and people were building dynamic web handlers in whatever language, compiling down to an executable.
It makes perfect sense but always felt out of place to me for some reason.
I recall eBay used to have ISAPI.dll in most of its request URLs. It bewildered my younger self who thought DLLs were only for a Windows machine, not a web server.
A previous employer of mine build a CMS for newspapers, written in Delphi and ran on Windows under IIS, it too was had a .dll extension. I believe it was the same IIS API (ISAPI) that made it work. It was an insane product that had simply failed to move forwards.
It's not that you can't do what you need to do using the ISAPI, but the ecosystem is just to small for things like the Delphi stuff and keeping up with expectations of customers becomes to much work.
It's impressive that anything that built the correct DLL could be leveraged by it.
But I assume ISAPI didn't have anything like CPAN or pip. It seems likely you'd have to write everything yourself - perhaps worthwhile if you were a very large corporation with the manpower. But something as simple as PHP would swamp it for the rest of us.
IIS was and still is CGI capable, but starting a process on Windows is a very heavy weight thing (compared to starting a process on linux), so it doesn't scale well and a noticeable initial delay whenever it executes. To solve that MS made ISAPI, which runs inside the webserver similar to an Apache httpd module.
In the CGI RFC, IIRC they say that what they call a script can totally be a
binary. The fact that execve on linux can take a script with a #! shebang makes
it it easy to implement, you don't have to care whehter you are exec'ing a binary
or a script.
I love CGI and still use it all the time. It's natural MVC too which is great.
Someone should make a framework like Electron or Tauri or NeutralinoJS that works like CGI. Neutralino on is probably the closest but... could be simpler. Just push everything over a pipe.
> Someone should make a framework like Electron or Tauri or WebKit that works like CGI. Don't need to use TypeScript or Rust or whatever. Just push between backend and frontend over a file stream.
I’d argue that already exists in the form of your $SHELL.
You can get graphical UIs in shells. Albeit they’re usually TUIs. But there’s nothing stopping someone from creating a more ”attractive” UI using sixel or even embedding web components via proprietary ANSI escape sequences.
Personally though, I don’t think HTML and JavaScript is a particularly nice abstraction for writing UIs. But each to their own
Recently I built a CGI script using Perl for my own pleasure. It was quite easy indeed, just a case of importing Template Toolkit, GD and CGI.pm then printing the result to stdout. Since I'm only using CGI.pm to send the HTTP header it isn't really necessary.
I found CGI to combine quite well with bleeding edge JavaScript 7 : async / await (compared to JS6 promises that left me confused as a JavaScript newbie).
I am more interested to know in which situations it's a bad idea to use CGI ?
In the old days (1990s), it was a "bad idea" to use CGI for heavily trafficked sites because there was a lot of system overhead involved in spawning a new CGI process for every request, firing up an interpreter (usually Perl), parsing the code, then executing it.
As one of the previous comments mentions, operating systems do a better job of caching now, so there's less overhead there. Fast CGI also gets rid of that problem by running your CGI script as a single persistent process.
CGI may not be the best solution for really huge applications that have hundreds of routes or very complex business logic.
That said, it's still incredibly useful. I use it on some personal sites, and it's nice to know that I can drop it into any new site without having to worry about dependencies and without having to do much configuration. It's simple and it works. Those are two big pluses in a world where things have gotten so complex.
CGI was abandoned due to performance problems, but it is surprising scalable on modern OSs, the kernel does a good job of caching the code so it's not really a full round trip to the file system and cold start on each request. Obviously you still have the overhead of an interpreter for many languages, but they also tend to do a fairly good job of caching their byte code. I absolutely wouldn't recommend it for new systems, but for a quick and dirty script it's fine.
The minute you want to do url routing and have middleware, a persistent process makes everything much simpler and keeps everything in one place. I much prefer routing inside a framework than using Mod_Rewrite.
My point really was when having url routing, such as ids/slugs in the url, it is much easier to implement in your application code than in the web server configuration. It keeps application code in one place.
Caddy doesn't support CGI out of the box. You need to use a module (which requires a custom build of it and the hope the module author continues to maintain it), or you need to run something like fcgiwrap to convert FCGI (what Caddy does support) to CGI (and now you need a process manager to make sure your fcgiwrap and caddy processes are kept running, restarted on failure, etc).
> a persistent process makes everything much simpler ...
I started with PHP and never trusted persistent processes for anything.
What if your persistent process hangs or crashes for whatever reason?
Is your whole site down then, not just the part with the bug?
The only site I ever made in persistent process paradigm was in ASP.NET and it didn't cause too many problems but it was more of an app, not website, with limited number of logged in users and pretty much no anonymous/guest part.
> What if your persistent process hangs or crashes for whatever reason?
You've got the same problem with your web server, and your operating system kernel. Simply write a program that can't hang or crash – at least, not due to an application bug. It takes attention to detail, but this is a lot easier than writing a program that always behaves correctly; if you limit the syscalls you make, use timeouts appropriately, and don't segfault, it's often achievable.
I don't think any mainstream framework in any mainstream language has these kinds of precautions.
Rails? Django? Express? .NET? Do they kill long running requests so they don't affect whole application?
Can the application still mostly work despite syntax error in one of the files?
Another thing is level of trust. I trust Linux kernel, I trust apache webserver. But some webserver written in scripting language few years ago? Not so much.
Well try in a nodejs server to put a `while(true);` in a request handler. Do you know the answer? The whole process blocks, since nodejs is single process.
I trust more the code that I wrote than the code written from others... not because I'm better than others, but because I know what it does, and I know how to fix it if it breaks.
> I started with PHP and never trusted persistent processes for anything.
Well it's (maybe was, it did get a bit nicer over years but old shit code is still there) terrible language so no wonder you have no trust.
> What if your persistent process hangs or crashes for whatever reason?
depend on language. For example in Go (in most frameworks at least) panic() in goroutine handling request will just... nuke that request and nothing else. Sure, memory leaks can be issue if the code is terrible but essentially restarting app after every request is just shitty workaround for shoddy code.
That's how most other handle code. Other, that do multi-core processing badly like Ruby, just spawn multiples of server so it isn't really that different from PHP model, you get "supervisor" that spawns few processes of app and feed them requests, if one dies it just gets restarted.
> Well it's/twas terrible language so no wonder you have no trust
The reason someone who was shaped by working in PHP might not trust long-running processes probably has less to do with any specific shortcomings of it as a language and more to do with the fact that overwhelming execution model is single-request shared-nothing.
This conceptual model works well enough (especially for the web) there's entire cloud services built around recreating it for other languages called "serverless," and they do well because it's a useful simplification which means the issues associated with persistent processes become somebody else's problem.
Devs being what we are, of course there's always those of us who prefer certain things to remain our problem. This is fortunate as someone needs to assume problems like long-running processes in order for others to make it somebody else's problem. Software development has some things in common with comparative advantage.
Fault tolerance was how Next branded it, IIRC. If an instance of a long running process doesn’t communicate with an external monitor, then it’s assumed to have hung or crashed at which point it is restarted by the external tool. Or at least that’s how things were done in the early/mid 90s. Like techniques are employed by Node apps these days, and undoubtedly many others.
I find it kind of interesting that people frowned upon CGI because it was not efficient. And yet here we are using lambas on AWS like it’s the best invention ever
No web dev here, but if memory serves, the main concerns about CGI were about safety, not performance or efficiency. If that's the case, I wonder if new more safe languages could make its use viable again.
With people reaching for kqueue, epoll and io_uring to handle the C10K problem it seems that spawning processes wouldn’t be a great idea, with the latency and memory overhead of spawning itself, as well as the meagre limits of the number of concurrent processes in a system.
One issue was that people configured CGI wrong. You had executable scripts that had free range on the entire filesystem. Ideally you'd need to limit the cgi-scripts to a chroot, but that made it difficult to use Perl, Python and PHP. OpenBSD shipped/ships a chrooted Perl for use with CGI, but other operating systems just left that bit as an exercise to the programmer.
More modern languages like Go or Rust provides a lot of safety and is pretty easy to statically compile, so it's easily chrooted. It would however be weird to do a CGI program in Go, because that language already have a really good build in webserver and more powerful/easily accessible features for interacting with requests compared to what CGI provides.
> It would however be weird to do a CGI program in Go, because that language already have a really good build in webserver and more powerful/easily accessible features for interacting with requests compared to what CGI provides.
Noting that not everyone has the ability to run standalone servers on their web hoster. Shared hosters generally enable CGIs but cannot support per-user standalone servers, in particular not on common ports like 80 or 443.
What safety concerns are you referring to? CGI scripts often had issues with SQL injection and shell command injection, but that was because of badly designed (or poorly used) libraries and unrelated to CGI itself. Or if CGI programs were written in C (as a result of performance concerns) those would of course have the usual memory safety issues of C programs.
You could write this off as a library implementation problem, but it comes about because of unfortunate mapping of HTTP parameters to well-known environment variables in CGI, and has tripped up multiple library authors.
Usually poor ops hygiene, same really as PHP problems:
* Code runs as web server user
* Code is deployed as web server user, because web monkey doesn't understand unix permission (or alternatively chmod 777 everything for same effect)
* Code gets hacked, attacker can modify any file so they just leave backdoor in code
If you deploy code as different user attacker can still of course steal all the data but at the very least they can't modify the code that is running.
On top of that, a lot of poor security practices when it comes to qouting stuff. Perl had kinda interesting feature regarding that where in special mode (recommended for apps dealing with untrusted inputs) the variable was considered tainted [1] till something "cleaned" it by passing it thru regexp. So say taking argument directly from environment (so headers for web) or stdin and passing it with no processing to system() would trigger it.
If there is another request "in the queue", a lambda that is done with a request will just pick that one up instead of closing down and letting a new lambda spin up to deal with the next request. So as soon as you have enough traffic that you have $CONCURRENCY_LIMIT amount of lambdas running (otherwise you would never build up a queue of course), the execution model of lambdas approximates fastCGI much more than regular CGI.
Personally I think the enthusiasm for lambdas stems from several things:
- A (legitimate) interest in outsourcing non-essential things like server maintenance to somewhere else with more ops experience. This frees up the devs to work on business logic instead of worrying about backups and log rotations.
- Some people are REALLY enthusiastic about the whole scale-down-to-zero thing. I personally think that that is somewhat shortsighted, since that means trading (expensive!) engineering time to save a few 100 USD per month on your AWS bill. Sometimes it is definitely worth it to cut costs, but usually it's engineers overoptimizing the one aspect of profit they have significant influence over (ie hosting costs).
- Some people are very enthusiastic about how you can instantly get to "web scale" with lambdas by just putting a very large max concurrency number. That is somewhat true, but massive overkill for the vast majority of companies. It's "engineer it like Google" syndrome but with Amazon instead.
You can use flask with CGI too ;)
I did this last month for a website that only needed to live for a few weeks so it could piggyback off an existing Apache setup and require less work to setup and run.
I first saw PHP around that time, too, and you could see it set up on CGI on some hosting providers; I think it was usually older ones who had Perl originally and then just tacked PHP on to follow demand.
I've built anything from CGI guestbooks to webrings in the '90s but IMO the sweet-spot for solo projects in terms of time to market and scalability/stability is the humble combo of Flask & Caddy.
I really like Haserl (https://haserl.sourceforge.net/).
It acts as a middleman for CGI scripts and and makes it easy to write them with
a shell or lua while keeping the look a HTTP response with a html body.
"""
Haserl is a small cgi wrapper that allows "PHP" style cgi programming, but uses a UNIX bash-like shell or Lua as the programming language. It is very small, so it can be used in embedded environments, or where something like PHP is too big.
It combines three features into a small cgi engine:
It parses POST and GET requests, placing form-elements as name=value pairs into the environment for the CGI script to use. This is somewhat like the uncgi wrapper.
It opens a shell, and translates all text into printable statements. All text within <% ... %> constructs are passed verbatim to the shell. This is somewhat like writing PHP scripts.
It can optionally be installed to drop its permissions to the owner of the script, giving it some of the security features of suexec or cgiwrapper.
"""
IIRC from when I wrote a basic http server, the problem with CGI is that you
have to store the output of the script in memory in order to insert the Content
Length field value in the http respondse header. It seems like a rather
innefficient operation, especially if repeated a million times on a server.
Else you could just read from the your CGI process in a non
blocking manner, writing back to the socket whenever output
is available. Am I correct ?
transfer-encoding: chunked, and then send data in 64kb packets. You need to tell the size of each packet at its header. Even then, it is an optional good practice, but not required by RFC.
I wrote a web app that launched from inetd to control my cd player. It worked great but wasn't super practical since I tended to use it from the same computer that it ran from. Everything web was fun in the 90s though.
Recently I enjoyed some "retrofuturistic" development with WASM and CGI. Spin[1], a webserver written in Rust, can execute WASI[2] binaries that speak CGI. You can then deploy it to Fermyon Cloud or your own server and it "just works". It's a wonderful mix of old and new. I used it for PHP (Prolog Home Page): https://github.com/guregu/php
[2]: WASI is a POSIX-ish standard for WASM that gives you all the low level stuff like standard input and output. It includes all the bits and pieces needed for CGI to work.
Kind of feels like the AssemblyScript guys were right - WASI is becoming a standard for WASM modules that foists Unix on Webassembly, including stuff that isn't really portable like symlinks.
That makes sense if you want to compile existing software to WASM but that isn't the case here is it?
This does use existing software. The onus behind this project was to test my port of Trealla Prolog (which was written by someone else) to WASM. Without a Unixy standard environment it would have been a lot harder. In theory, WASI should enable many CGI-like usages of pre-existing software, but in my experience the quirks between the WASI implementations push you towards using nonstandard things anyway. For example, trealla-js is mostly WASI but uses a couple extra imported functions to coordinate calling into JS. If I wanted to let PHP make outgoing requests I'd have to implement Spin's custom nonstandard components. I hope the WASM Components proposal makes this easier in the future. Personally, I might have preferred a more Plan 9-style approach that uses special files to do things like make outgoing connections, purely because it would be trivial to implement client-side as a simple printf vs. integrating wit-bindgen into the build system and all of that. However, I can see many benefits of the strongly typed component model as well.
I don't have a strong opinion on the AssemblyScript controversy[1]. I agree that WASI is imperfect, especially in browsers, but I'm also glad that _something_ exists with not-horrible runtime support. WASI was pretty nice to use as someone who just wanted to get a C program running on WASM.
Wasm allows you to consume as much or as little of the API surface area as you want. You are free to construct Wasm modules in whatever way you see fit for your application.
I do contract work for a company that has it's pretty customer facing site as a mix of angular/react/react native and .net core apis and the less pretty internal staff site which is vanilla html, js and c++ cgi apis, so I get to work in both on the same day. The internal c++ site has been running for ~25 years. The customer site seems to need to be re-written every time we add a new feature, because all the tech is out of support and there are breaking changes to libraries
> Personal projects remain unfinished or abandoned because I’ve wanted to do things the “right” way. So I start building it the “right” way with complicated tools and processes, then raise my hands and think it’s all too much for what I want to do. It puts all sorts of things out of reach.
This brings to mind some of Paul Virilio’s writings, who discussed this idea of speed (to include not only velocity but also the sensory assault that comes with complexity and speed combined. This has been paraphrased as “speed is a continuation of fear by other means.” How many framework decisions are made in this mindset?
Yeah, I like the analogy of a march, as a contrast to the sprints typically employed in commercial development. Like my search engine project is 2 years in now, and still not anywhere near "done". Yet it keeps getting better and more capable at a slow but steady pace.
I feel this. The other day I decided to just use Rails for one of these personal projects that I’ve been over-engineering for a while and it’s making progress.
Completely forgot how fast Rails makes everything. Based on the experience I’m probably going to force myself to just use Rails for any personal stuff now and figure if any of them actually take off I’ll rebuild if I need to.
This.
Find a tech (NOT the fotm shenanigans) that will help you get a product up and running. Then you can always go back and swap out parts as needed.
Try not to be influenced by the "omg, that tech is SOOO old!!!" crowd.
this is the exact opposite of the point the comment you replied to is making. Rails is not simple. we are talking about SIMPLE, like CGI. Rails is a classic example of a slow bloated framework.
Rails makes a lot of very complicated things, very simple.
Doing what I'm trying to do with CGI would be a huge headache by comparison. When I said "how fast rails makes things" I meant from a standpoint of productivity.
Rails will be significantly faster than a CGI script that spawns a whole new process, starts up your script interpreter, etc. with every single web request.
100%. I finish projects with Rails... Can't say this about things I start in Go with Clean Architecture and a distributed frontend written in SvelteKit etc. pp.
Some of the "right" tools are right only in context of a professional team anyway. Especially true when deciding how to organise a project.
If it is just me, for me, I will write it simply and from scratch or use a familiar general purpose framework like rails or laravel. One I have lots of experience with so the project actually gets done, instead of spending all my energy learning another new tool.
That could also be reduced down to "JUST USE RAILS".
Why not get the best of both worlds where you finish projects AND be happy about it? The "for now" makes you think it's a bad decision and you hacked something together to get it done.
But you can finish, stick with Rails and be insanely successful based on what's important to you (start, finish, maintain, profit, IPO, etc.).
> Nobody should start to undertake a large project. You start with a small trivial project, and you should never expect it to get large. If you do, you'll just overdesign and generally think it is more important than it likely is at that stage. Or worse, you might be scared away by the sheer size of the work you envision. So start small, and think about the details. Don't think about some big picture and fancy design. If it doesn't solve some fairly immediate need, it's almost certainly over-designed. And don't expect people to jump in and help you. That's not how these things work. You need to get something half-way useful first, and then others will say "hey, that almost works for me", and they'll get involved in the project.
A quote from Linus Torvalds someone posted on HN and I saved almost a year ago
"A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over, beginning with a working simple system."
What's funny is this is how Agile works in a functioning organization. You validate the idea first and quick, and only then iterate with new layers of complexity. Building something the "right" way before ever having seen the product live is called waterfall.
The thing that nobody tells junior programmers, but which you really have to pick up from experience, is this:
"The right final architecture" is never achieved by just immediately going out and building that architecture, i.e. by hooking up all the tools required to support that architecture. That's cargo-culting the architecture.
Facebook's use of Cassandra and CI lint-checks and blue-green deployments is just like military cargo planes' use of radio towers — they didn't build those first; they scaled the thing they were doing to the point that these things became necessary support structures, and then they built them.
The "right way" — the right process for engineering a solution — has very little to do with up-front architectural design. The "right way" — the way that'll be most likely to get you to that "right final architecture" eventually — is really the tenable way: the iterative approach that allows you to build up your solution while keeping only one change or consideration in your head at a time. Which means that engineering "the right way" involves not doing all those cargo-cult practices unless/until they become necessary, and even then, only adopting them one at a time. Just like you wouldn't try to make ten different refactorings in a codebase in one patch-set.
Or, to put that another way: YAGNI applies to processes and tools just as much as it does to code. Some projects never exceed 1000 lines. Do those projects need CI cyclomatic-complexity checkers? No.
Only introduce support structures to a project, as the pain of not having them starts to outweigh the pain of adding them.
> Facebook's use of Cassandra and CI lint-checks and blue-green deployments is just like military cargo planes' use of radio towers — they didn't build those first; they scaled the thing they were doing to the point that these things became necessary support structures, and then they built them.
This is so true. In fact, a lot of the technology choices companies at the scale of Facebook use are only necessary because they're so large/popular, and were added later to stop everything falling over.
Because it's so unlikely your project will ever need to contend with a billion users or whatever, you really shouldn't be designing under the assumption that's likely to happen.
As someone who wrote a large, commercial social network for schools in Perl + CGI.pm in ~ 2000, here were the problems:
* Perl was very memory inefficient so you could often only serve 4 users at the same time, given the limited RAM available (eg. 512M - 1G). Note that Perl's speed was not a problem. I suspect this one has probably gone away with modern servers with massive amounts of RAM, plus the cloud allows you to quickly scale up and down with demand. Nevertheless using a compiled language would be better. (I later wrote a vastly more efficient framework in C).
* Every action on the page had to round-trip to the server, and always involved a database check (at minimum to map the user's cookie to a user ID). This is sort of solved now that Javascript is widely supported, although that brings its own issues along. Also memcached neatly solves the database access problem. Our website was developed a little bit before memcached.
* It's very clumsy and time-consuming to write any non-trivial CRUD-style action using CGI. eg. Just having a table with update/delete buttons is going to involve writing paging code, update form, update submit form (repeat for every possible action). This is the kind of thing that Ruby on Rails solved quite nicely.
* Organizational problems interfacing with the database. We had to negotiate with the DBA for every schema change, which is a problem when the DBA is a sociopath supported by management. Devops sort of solves this (but also I appreciated not being on call).
* The general hassle of dealing with the web, like setting all the non-obvious HTTP headers to make it secure. I think modern frameworks just deal with this, although I've not used them very much.
However if I was to go back and change anything, it wouldn't have been to change the technology. It would have been to extend the service first to university students and later to the whole world :-)
"Perl was very memory inefficient so you could often only serve 4 users at the same time, given the limited RAM available (eg. 512M - 1G). Note that Perl's speed was not a problem. I suspect this one has probably gone away with modern servers with massive amounts of RAM. Nevertheless using a compiled language would be better. (I later wrote a vastly more efficient framework in C)."
I cannot even imagine the strange complexities of this circa year 2000 web application if it could only fit 4 simultaneous page loads on 1 GB of RAM. It's baffling, in fact. Perl isn't really memory hungry these days. Was it really that bad 20 years ago? I would be surprised if the Perl 5 interpreter wasn't in fact much leaner back then. At least my immediate impression of this bullet point is that it wasn't a programming language problem, but a software design problem.
Well it depends what you're doing of course. Part of the application was an email interface which I remember was a particular problem - it was likely loading whole emails into memory to do MIME parsing. But the point here is that large, slow processes gradually take over your system crowding out smaller and simpler requests. We had our own cage in a Docklands data centre, so "spinning up another server" was a weeks long process involving many people.
In modern terms Perl isn't a problem, especially when you have servers with hundreds of gigabytes.
I can safely say that it's not really an immediately impeding problem with 1 GB of RAM either. I have a bit of experience in the context as I develop and manage a very large Perl software stack that runs on a fleet of servers ranging from itty-bitty to small/medium. Some of it is inefficient CGI, some of it is FastCGI, some of it is resident in the background and some of it fires up on intervals. All of it is heavily trafficated.
But, certainly, I'll concede that if the platform is completely CGI-based and for some reason is written such that there's a minimum footprint of 50-100 MB for any single invocation, then servers will be needing a bit of memory in lieu of a software rethink.
It was definitely a software problem. Small Perl app would take few tens of MBs max back then (and now).
Also Perl FCGI existed much earlier than 2000 (looking at CPAN ~'96-97 appear to be first versions), althought obviously the code would be bit more complex than just "dump html on stdout"
CGI is slow [ spawns an OS process for every request ]. But FCGI is as fast as any other solution, like compiling the whole app in a single binary and running it as a HTTP server.
CGI is _computationally_ slow, but in practice it's faster than it needs to be for most sites. Some examples of CGI-powered sites i'm aware of include:
Many people confuse "X is slower than Y" with "X is slow in some absolute sense". Starting a new process takes well under a millisecond on most modern machines, so that would be completely lost in the latency of most web requests. The real benefits of persistent processes are in getting to reuse database connections and the like.
YMMV, but thats why i love Forth. it compiles the full app (not interpret) in ms. some think its dead or only for the embedded, but for me and my projects its a deadly combination + ultra fun.
Yeah, I don't think the speed performance of FCGI vs CGI is really about spawning in 2023: the benefit lies in the fact that the app can keep state and resources between requests. In particular, it can maintain a database connection all the time (with a retry logic for the case when it gets disconnected).
However, perhaps 15+ years ago I converted an application to an Apache module from a cgi-bin binary and the performance benefit was immense in an embedded Linux device. We certainly were not talking even about tens requests per second in the CGI version.
> the benefit lies in the fact that the app can keep state and resources between requests.
You can share state between processes using temporary files/shared memory. Like CGI itself, it doesn’t scale as well, but it’s not even a blip on resource usage unless you’re dealing with thousands of requests per second.
> In particular, it can maintain a database connection all the time (with a retry logic for the case when it gets disconnected).
You can run a daemon that performs the connection pooling for you, and have each request connect to the daemon instead.
Temporary files/shared memory won’t solve the cost of starting up a Python interpreter or a JVM or whatever and importing all your code with its dependencies. As for the connection pooling daemon, you still need to connect to it. It might be faster than talking to the database directly on a different server, but no new connection still beats new local connection.
> Temporary files/shared memory won’t solve the cost of starting up a Python interpreter or a JVM or whatever and importing all your code with its dependencies.
This is true, CGI is sensitive to startup latency. It can be addressed with pre-forking, but doing so bears the consequence of increased memory usage at idle workloads.
> As for the connection pooling daemon, you still need to connect to it. It might be faster than talking to the database directly on a different server, but no new connection still beats new local connection.
It’s a nothingburger. A connection pool’s bottleneck is in the TCP connection to the database. A UNIX socket has orders of magnitude less latency (a couple μs) and supports orders of magnitude more throughput (a few GBs/sec). Compared to spawning new processes running interpreters, it’s noise.
Some context: in 2017, I found by informal testing that I could run (just on a dev laptop) about about 180 lua no-op programs per second, 140 perl5, 35 py2, 30 py3, 15 node, 12 ruby; /usr/bin/true, as a benchmark, a bit over 500 Hz.
Doesn't necessarily extrapolate back another 20 years, but perhaps sheds some light on how "slow" CGI would've been.
I would also try benchmarking the new io_uring_spawn interface, which is said to be significantly faster than the traditional fork+exec way of spawning processes.
Spawning a process is not that slow on modern hardware. CGI is considered slow, because back in the 90's, everything was slow and resource constrained. Slow processors, limited RAM, slow disks for swap, slow internet connections... I remember developing Perl CGI scripts on a Sparc 10 with 32 megs of RAM connected to a T1 that was nearly always maxed out.
On a related note, "modern" cloud-based application development strategies (like AWS Lambda) seem no faster than CGI. We've come full circle. Though now things are more distributed, of course.
I felt exactly the same as the author. That's why I created https://trusted-cgi.reddec.net/ (currently on the way to v1 and under the heavy refactoring)
I wrote Perl cgi scripts for a self-serve customer admin interface for an ISP in the early 2000s. As users grew, we started hitting the limits of what 1-2G ram could do. Thank goodness for mod_perl and HTML::Mason, which could REALLY speed up performance. I remain a huge fan of Perl these days.
I used Perl's CGI::Application to transform an Excel spreadsheet of teachers, used for fielding telephone enquiries, into a PostgreSQL-backed web service used by 33 household brands. Everything ran on Apache CGI without a hitch for 15 years.
It might interest readers to know that the Fossil SCM's primary mode of operation is CGI, and has been since it began life in 2007: https://fossil-scm.org/home
When i first discovered fossil, in December 2007, its ability to run as a CGI was one of its two "killer features" for me, and it's still right at the top of its list of killer SCM features for me.
(Disclaimer: i'm a long-time fossil contributor, but its CGI support pre-dates my joining the project.)
I started implementing a FastCGI client library for Racket, and (though I have experience getting into the bytes and bits of protocol implementation) it seemed harder than it needed to be. So I decided to try SCGI first, and SCGI ended up working great in production:
224 comments
[ 3.4 ms ] story [ 223 ms ] threadIt makes perfect sense but always felt out of place to me for some reason.
It's not that you can't do what you need to do using the ISAPI, but the ecosystem is just to small for things like the Delphi stuff and keeping up with expectations of customers becomes to much work.
But I assume ISAPI didn't have anything like CPAN or pip. It seems likely you'd have to write everything yourself - perhaps worthwhile if you were a very large corporation with the manpower. But something as simple as PHP would swamp it for the rest of us.
Someone should make a framework like Electron or Tauri or NeutralinoJS that works like CGI. Neutralino on is probably the closest but... could be simpler. Just push everything over a pipe.
I’d argue that already exists in the form of your $SHELL.
Though I'm not sure requiring end users to install such a niche thing (assuming it exists) is a great distribution strategy.
Personally though, I don’t think HTML and JavaScript is a particularly nice abstraction for writing UIs. But each to their own
The result is here: https://www.thran.uk/cgi-win/fleg.pl
It is a flag generator affectionately titled flegger.
It even does event-based and websockets
I am more interested to know in which situations it's a bad idea to use CGI ?
As one of the previous comments mentions, operating systems do a better job of caching now, so there's less overhead there. Fast CGI also gets rid of that problem by running your CGI script as a single persistent process.
CGI may not be the best solution for really huge applications that have hundreds of routes or very complex business logic.
That said, it's still incredibly useful. I use it on some personal sites, and it's nice to know that I can drop it into any new site without having to worry about dependencies and without having to do much configuration. It's simple and it works. Those are two big pluses in a world where things have gotten so complex.
The minute you want to do url routing and have middleware, a persistent process makes everything much simpler and keeps everything in one place. I much prefer routing inside a framework than using Mod_Rewrite.
Edit: This article covers what I was thinking of:
"Myths About CGI Scalability"
http://z505.com/cgi-bin/qkcont/qkcont.cgi?p=Myths%20About%20...
Previously discussed:
https://news.ycombinator.com/item?id=32699395
2023 - Use single binary caddy. It's like 2 lines of config total to remove the extension from your CGI url's.
My point really was when having url routing, such as ids/slugs in the url, it is much easier to implement in your application code than in the web server configuration. It keeps application code in one place.
I started with PHP and never trusted persistent processes for anything.
What if your persistent process hangs or crashes for whatever reason?
Is your whole site down then, not just the part with the bug?
The only site I ever made in persistent process paradigm was in ASP.NET and it didn't cause too many problems but it was more of an app, not website, with limited number of logged in users and pretty much no anonymous/guest part.
You've got the same problem with your web server, and your operating system kernel. Simply write a program that can't hang or crash – at least, not due to an application bug. It takes attention to detail, but this is a lot easier than writing a program that always behaves correctly; if you limit the syscalls you make, use timeouts appropriately, and don't segfault, it's often achievable.
Rails? Django? Express? .NET? Do they kill long running requests so they don't affect whole application?
Can the application still mostly work despite syntax error in one of the files?
Another thing is level of trust. I trust Linux kernel, I trust apache webserver. But some webserver written in scripting language few years ago? Not so much.
Not sure about the built-in .net server, but IIS does exactly that. It supports timeout and error/time based restarts for the persistent process.
I trust more the code that I wrote than the code written from others... not because I'm better than others, but because I know what it does, and I know how to fix it if it breaks.
Well it's (maybe was, it did get a bit nicer over years but old shit code is still there) terrible language so no wonder you have no trust.
> What if your persistent process hangs or crashes for whatever reason?
depend on language. For example in Go (in most frameworks at least) panic() in goroutine handling request will just... nuke that request and nothing else. Sure, memory leaks can be issue if the code is terrible but essentially restarting app after every request is just shitty workaround for shoddy code.
That's how most other handle code. Other, that do multi-core processing badly like Ruby, just spawn multiples of server so it isn't really that different from PHP model, you get "supervisor" that spawns few processes of app and feed them requests, if one dies it just gets restarted.
The reason someone who was shaped by working in PHP might not trust long-running processes probably has less to do with any specific shortcomings of it as a language and more to do with the fact that overwhelming execution model is single-request shared-nothing.
This conceptual model works well enough (especially for the web) there's entire cloud services built around recreating it for other languages called "serverless," and they do well because it's a useful simplification which means the issues associated with persistent processes become somebody else's problem.
Devs being what we are, of course there's always those of us who prefer certain things to remain our problem. This is fortunate as someone needs to assume problems like long-running processes in order for others to make it somebody else's problem. Software development has some things in common with comparative advantage.
More modern languages like Go or Rust provides a lot of safety and is pretty easy to statically compile, so it's easily chrooted. It would however be weird to do a CGI program in Go, because that language already have a really good build in webserver and more powerful/easily accessible features for interacting with requests compared to what CGI provides.
Noting that not everyone has the ability to run standalone servers on their web hoster. Shared hosters generally enable CGIs but cannot support per-user standalone servers, in particular not on common ports like 80 or 443.
https://www.ionos.co.uk/digitalguide/server/configuration/ht...
You could write this off as a library implementation problem, but it comes about because of unfortunate mapping of HTTP parameters to well-known environment variables in CGI, and has tripped up multiple library authors.
* Code runs as web server user * Code is deployed as web server user, because web monkey doesn't understand unix permission (or alternatively chmod 777 everything for same effect) * Code gets hacked, attacker can modify any file so they just leave backdoor in code
If you deploy code as different user attacker can still of course steal all the data but at the very least they can't modify the code that is running.
On top of that, a lot of poor security practices when it comes to qouting stuff. Perl had kinda interesting feature regarding that where in special mode (recommended for apps dealing with untrusted inputs) the variable was considered tainted [1] till something "cleaned" it by passing it thru regexp. So say taking argument directly from environment (so headers for web) or stdin and passing it with no processing to system() would trigger it.
* [1] https://perldoc.perl.org/perlsec
There’s other ways to get that model, but lambda makes it pretty easy.
Personally I think the enthusiasm for lambdas stems from several things:
- A (legitimate) interest in outsourcing non-essential things like server maintenance to somewhere else with more ops experience. This frees up the devs to work on business logic instead of worrying about backups and log rotations.
- Some people are REALLY enthusiastic about the whole scale-down-to-zero thing. I personally think that that is somewhat shortsighted, since that means trading (expensive!) engineering time to save a few 100 USD per month on your AWS bill. Sometimes it is definitely worth it to cut costs, but usually it's engineers overoptimizing the one aspect of profit they have significant influence over (ie hosting costs).
- Some people are very enthusiastic about how you can instantly get to "web scale" with lambdas by just putting a very large max concurrency number. That is somewhat true, but massive overkill for the vast majority of companies. It's "engineer it like Google" syndrome but with Amazon instead.
Combined with single binary caddy and sqlite (+wal) gets you down to copy + paste production-ready deployments.
https://docs.oracle.com/cd/A97335_02/apps.102/a90099/feature...
I kinda wonder if anyone actually ever used that in production.
There's a big difference between get shit done and enjoy it.
And I started around PHP 4
""" Haserl is a small cgi wrapper that allows "PHP" style cgi programming, but uses a UNIX bash-like shell or Lua as the programming language. It is very small, so it can be used in embedded environments, or where something like PHP is too big.
It combines three features into a small cgi engine:
It parses POST and GET requests, placing form-elements as name=value pairs into the environment for the CGI script to use. This is somewhat like the uncgi wrapper.
It opens a shell, and translates all text into printable statements. All text within <% ... %> constructs are passed verbatim to the shell. This is somewhat like writing PHP scripts.
It can optionally be installed to drop its permissions to the owner of the script, giving it some of the security features of suexec or cgiwrapper. """
[1]: https://spin.fermyon.dev/
[2]: WASI is a POSIX-ish standard for WASM that gives you all the low level stuff like standard input and output. It includes all the bits and pieces needed for CGI to work.
That makes sense if you want to compile existing software to WASM but that isn't the case here is it?
I don't have a strong opinion on the AssemblyScript controversy[1]. I agree that WASI is imperfect, especially in browsers, but I'm also glad that _something_ exists with not-horrible runtime support. WASI was pretty nice to use as someone who just wanted to get a C program running on WASM.
[1] https://www.assemblyscript.org/standards-objections.html
So true it hurts.
The lesson from survivorship bias is to optimize for survival :).
Completely forgot how fast Rails makes everything. Based on the experience I’m probably going to force myself to just use Rails for any personal stuff now and figure if any of them actually take off I’ll rebuild if I need to.
Try not to be influenced by the "omg, that tech is SOOO old!!!" crowd.
Doing what I'm trying to do with CGI would be a huge headache by comparison. When I said "how fast rails makes things" I meant from a standpoint of productivity.
I need to print this out:
"JUST USE RAILS FOR NOW."
If it is just me, for me, I will write it simply and from scratch or use a familiar general purpose framework like rails or laravel. One I have lots of experience with so the project actually gets done, instead of spending all my energy learning another new tool.
That could also be reduced down to "JUST USE RAILS".
Why not get the best of both worlds where you finish projects AND be happy about it? The "for now" makes you think it's a bad decision and you hacked something together to get it done.
But you can finish, stick with Rails and be insanely successful based on what's important to you (start, finish, maintain, profit, IPO, etc.).
A quote from Linus Torvalds someone posted on HN and I saved almost a year ago
- John Gall
However, I wonder how this works for physical things, how does one create a simple version of a train or bridge? Some things have a complexity floor.
(That's a half serous question, of course.)
"The right final architecture" is never achieved by just immediately going out and building that architecture, i.e. by hooking up all the tools required to support that architecture. That's cargo-culting the architecture.
Facebook's use of Cassandra and CI lint-checks and blue-green deployments is just like military cargo planes' use of radio towers — they didn't build those first; they scaled the thing they were doing to the point that these things became necessary support structures, and then they built them.
The "right way" — the right process for engineering a solution — has very little to do with up-front architectural design. The "right way" — the way that'll be most likely to get you to that "right final architecture" eventually — is really the tenable way: the iterative approach that allows you to build up your solution while keeping only one change or consideration in your head at a time. Which means that engineering "the right way" involves not doing all those cargo-cult practices unless/until they become necessary, and even then, only adopting them one at a time. Just like you wouldn't try to make ten different refactorings in a codebase in one patch-set.
Or, to put that another way: YAGNI applies to processes and tools just as much as it does to code. Some projects never exceed 1000 lines. Do those projects need CI cyclomatic-complexity checkers? No.
Only introduce support structures to a project, as the pain of not having them starts to outweigh the pain of adding them.
This is so true. In fact, a lot of the technology choices companies at the scale of Facebook use are only necessary because they're so large/popular, and were added later to stop everything falling over.
Because it's so unlikely your project will ever need to contend with a billion users or whatever, you really shouldn't be designing under the assumption that's likely to happen.
"Tell HN: I was tired of being a perfectionist so I built an app within 24 hours"
https://news.ycombinator.com/item?id=33303269
* Perl was very memory inefficient so you could often only serve 4 users at the same time, given the limited RAM available (eg. 512M - 1G). Note that Perl's speed was not a problem. I suspect this one has probably gone away with modern servers with massive amounts of RAM, plus the cloud allows you to quickly scale up and down with demand. Nevertheless using a compiled language would be better. (I later wrote a vastly more efficient framework in C).
* Every action on the page had to round-trip to the server, and always involved a database check (at minimum to map the user's cookie to a user ID). This is sort of solved now that Javascript is widely supported, although that brings its own issues along. Also memcached neatly solves the database access problem. Our website was developed a little bit before memcached.
* It's very clumsy and time-consuming to write any non-trivial CRUD-style action using CGI. eg. Just having a table with update/delete buttons is going to involve writing paging code, update form, update submit form (repeat for every possible action). This is the kind of thing that Ruby on Rails solved quite nicely.
* Organizational problems interfacing with the database. We had to negotiate with the DBA for every schema change, which is a problem when the DBA is a sociopath supported by management. Devops sort of solves this (but also I appreciated not being on call).
* The general hassle of dealing with the web, like setting all the non-obvious HTTP headers to make it secure. I think modern frameworks just deal with this, although I've not used them very much.
However if I was to go back and change anything, it wouldn't have been to change the technology. It would have been to extend the service first to university students and later to the whole world :-)
I cannot even imagine the strange complexities of this circa year 2000 web application if it could only fit 4 simultaneous page loads on 1 GB of RAM. It's baffling, in fact. Perl isn't really memory hungry these days. Was it really that bad 20 years ago? I would be surprised if the Perl 5 interpreter wasn't in fact much leaner back then. At least my immediate impression of this bullet point is that it wasn't a programming language problem, but a software design problem.
In modern terms Perl isn't a problem, especially when you have servers with hundreds of gigabytes.
But, certainly, I'll concede that if the platform is completely CGI-based and for some reason is written such that there's a minimum footprint of 50-100 MB for any single invocation, then servers will be needing a bit of memory in lieu of a software rethink.
Also Perl FCGI existed much earlier than 2000 (looking at CPAN ~'96-97 appear to be first versions), althought obviously the code would be bit more complex than just "dump html on stdout"
CGI is _computationally_ slow, but in practice it's faster than it needs to be for most sites. Some examples of CGI-powered sites i'm aware of include:
- https://sqlite.org/forum
- https://sqlite.org/wasm (same domain, different sub-site)
- https://fossil-scm.org/home
- https://fossil-scm.org/forum (same domain, separate sub-site)
- https://fossil.wanderinghorse.net
All CGI, all the time, and _plenty_ performant for the purpose.
But if that process is Python or Java it's going to be tens to hundreds of milliseconds before it's ready to serve the request.
It tested in a loop fork/exec and vfork/exec for a dynamic binary and for a static binary with musl.
On an Intel Core i5 6600k:
regular:
fork 13179 ms / 100000 = 131 µs, 7587 per sec
vfork 14423 ms / 100000 = 144 µs, 6933 per sec
static:
fork 3173 ms / 100000 = 31 µs, 31515 per sec
vfork 3833 ms / 100000 = 38 µs, 26089 per sec
However, perhaps 15+ years ago I converted an application to an Apache module from a cgi-bin binary and the performance benefit was immense in an embedded Linux device. We certainly were not talking even about tens requests per second in the CGI version.
You can share state between processes using temporary files/shared memory. Like CGI itself, it doesn’t scale as well, but it’s not even a blip on resource usage unless you’re dealing with thousands of requests per second.
> In particular, it can maintain a database connection all the time (with a retry logic for the case when it gets disconnected).
You can run a daemon that performs the connection pooling for you, and have each request connect to the daemon instead.
This is true, CGI is sensitive to startup latency. It can be addressed with pre-forking, but doing so bears the consequence of increased memory usage at idle workloads.
> As for the connection pooling daemon, you still need to connect to it. It might be faster than talking to the database directly on a different server, but no new connection still beats new local connection.
It’s a nothingburger. A connection pool’s bottleneck is in the TCP connection to the database. A UNIX socket has orders of magnitude less latency (a couple μs) and supports orders of magnitude more throughput (a few GBs/sec). Compared to spawning new processes running interpreters, it’s noise.
Doesn't necessarily extrapolate back another 20 years, but perhaps sheds some light on how "slow" CGI would've been.
https://lwn.net/Articles/908268/
Not so uncommon these days tbh
On a related note, "modern" cloud-based application development strategies (like AWS Lambda) seem no faster than CGI. We've come full circle. Though now things are more distributed, of course.
mod_perl teaches you quickly about state.
When i first discovered fossil, in December 2007, its ability to run as a CGI was one of its two "killer features" for me, and it's still right at the top of its list of killer SCM features for me.
(Disclaimer: i'm a long-time fossil contributor, but its CGI support pre-dates my joining the project.)
https://www.mit.edu/~yandros/doc/specs/fcgi-spec.html
Easier than integrating a 3rd party tool.
http://python.ca/scgi/protocol.txt
I started implementing a FastCGI client library for Racket, and (though I have experience getting into the bytes and bits of protocol implementation) it seemed harder than it needed to be. So I decided to try SCGI first, and SCGI ended up working great in production:
https://docs.racket-lang.org/scgi/