Where are you are sanitizing the comments? You have literally provided a perfect SQL injection mechanism here and allowed arbitrary HTML to be inserted onto your page.
I can tell it's Friday because it took until the section ID until I realized this was going to be about user comments on a website not <!-- HTML comments --> LOL.
If you've got Ruby listening for updates with a postgres database all tied into a content generator it's probably time to admit you don't have a static website anymore and just look for best practice ways to go about this (with a bit more security!).
Same here... and I wouldn't call this a static page anymore if it runs JS on load, calls a HTTP endpoint which makes DB query which returns HTML that modifies the DOM.
but if you can host your site from an S3 bucket, would it not be called static? to me, static means no server side languages necessary to render the page. if your statically served JS can call a remote host to pull down some data that updates things client side, it's still considered static, from a hosting perspective.
I guess it depends on perspective. What a host service that will charge you more money for a non-static site, a client-side JS app can still be considered static.
We might also be blurring lines of static sites vs interactive sites. You can have a static site that does nothing but allows the user to scroll to view it. You can also have static sites that change and do things based on the user's input. Entire games have been made using client-side JS but is still served statically.
IMO if a web user triggered event triggers a server side page regeneration they see when the page reloads what you have is more a dynamic site with caching not a static site. "It's a write-to-file-before-serving site" isn't really a meaningful distinguisher but "the site functions the same when you only have a copy of the website files" is.
The JS doesn’t trigger a DB query, it loads a static comments file previously generated by Ruby at the time the last comment was inserted or modified. It could be replaced by a server-side include.
I’d regenerate the complete HTML page server-side whenever a comment changes, instead of only regenerating the comments section, to obviate the secondary loading via JS.
There is some advantage in splitting it in that the page itself stays cached even as the comments change and the page doesn't need to wait for the comments to be transferred to start rendering. You could keep this advantage with <iframe> if you really wanted to avoid JS though. It's a shame html imports died, they were a bit more lightweight.
If you're just slinging static HTML, there's really not that big of a difference, and if it's a frequently accessed resource where a difference might be expected to matter, it will be cached regardless.
Hilariously enough, that's no longer true in many parts of the world; according to firefox telemetry, on-device storage is frequently much slower than the network.
> the page doesn't need to wait for the comments to be transferred to start rendering.
It doesn't. Browsers will happily render half an HTML page as it's streamed in. There's an interesting HTTP 203 on this. https://web.dev/shows/http-203/LLRig4s1_yA/ if you're interested.
Server side includes work nicely for this, I think nginx was what I used (long time ago now). You can even inline them on load but reload htmx-style with minimal effort.
It's saturday here now, I've slept plenty and I still interpreted the headline the same way.
But the topic is plenty interesting too, I'm curious how far the SSG movement will take it.
This used to be relatively common. Messageboards and forums (going all the way back to Matt's cgi board, which didn't even use a database) would cache static HTML pages and only regenerate them when necessary. I think Wordpress' cache plugin works in much the same way.
I guess it's a completely foreign concept in the age of SPAs when everything is dynamically generated in the browser.
I implemented static HTML comments on my website by tail'ing the /var/logs/nginx/access.log with a perl script. No CGI, no database, just the actual text of the comments stored as a single comment per line in a .html file.
To comment a visitor takes any url on the domain and appends "/@say/". Like http://notmyurl.com/somepage.html/@say/Some response to somepage. Or "...lakephoto.jpg/@say/Cool fish! How long was it?"
The perl script sees the /@say/ in the logs and adds the parsed out and sanitized comment to an .html file. There's some nginx location hijinks for matching /@say/ URLs that goes to a confirmation page and redirects to the comment listing page.
what executes the perl script? the POST doing a CGI type thingy i have so forgotten how to utilize from not using it since the 90s? on first read, i missed that you're modifying the contents of the html and was concerned about log rotation before realizing that was just dumb to be concerned about.
I have it start when my OS starts. It runs continuously thereafter (~5MB ram). The webserver does not know or care about it. The script leans heavily on perl's POE::Wheel::FollowTail module to get updates to nginx's access.log automagically then I abuse regexes to note log entries with /@say/, extract what URL-file-path commented on, what the comment is, and prepend the line to the algorithmically appropriate .html file with some html mark-up. The webserver serves the static html as it would any html.
I've used this comment system on my tor onion services sites for the last decade. I get plenty of people trying to exploit it. It's kind of fun. If the Tor folk haven't pwned it I doubt the HN folk will. Not for lack of skill but mostly a lack of motivation relative to the tor folk.
Minor pet peeve, but it is so much nicer when people use https://example.com for example links.
I clicked on your link expecting a real site (my monkey mind having skipped over the actual URL contents) and I was redirected to some other odd Chinese (?) URL with spammy ads and NSFW content.
The normal way around this in Postgres is to use NOTIFY to merely signal that there is something new, and use a separate (perhaps transactional) work queue to claim a row to process. You’re correct that relying on parameters in the actual notification will lose work.
This doesn't deal with the deluge of spam you'll get once any page on your site gets even modestly popular. (But it seems great for simpler use cases).
It would be so nice to have a comment system like Disqus but simple like described here, that also had an easy tie-in to a spam prevention service (or would work with a trained spam prevention model).
Ok, but instead of inserting into db, it'd be cool to have the script commit the comment into source control and have your ci rebuild/deploy your static site
So, you’ve got a static-ish page, a Ruby server, Postgres and NGINX. The page both POSTs and GETs comments from the backend via an Ajax calls. All very conventional, but then you’re worried about traffic spikes, and to deal with that, the comments on the server are stored in pre-generated HTML files, updated by some wild PG triggers setup, and served up by NGINX. IMO you could go way more traditional, with the same result.
Worried that Postgres will tip over? Just read from the DB, but cache the results in memory with a short TTL on your Ruby server.
Or, if you’re worried about not just Postgres, but also your Ruby server getting overloaded, just go with a fully server rendered page, and cache the whole page. Could use NGNIX cache, as you’re already using NGINX, or a CDN, or Varnish, whatever. No JS loaded comments at all.
Either way, caching should be able to get you tonnes of scalability, easily avoid
hug of death, without resorting to PG triggers calling a Ruby server that writes files to disk.
With that being said, this approach works too, and if it’s just a personal page with you as the only dev, you understand it, so it’s all good. But I’d certainly avoid this approach at a company, largely because it’s unusual and will take new devs longer to wrap their heads around.
The approach in the article is functionally different though, in that the cache is always immediately invalidated when a new comment comes in. So it’s not the same result as using a cache with a TTL.
You can always set a super short TTL like 15s or even 1s.
It's still going to make scaling a non-issue like that and at some point the caching will be fresher then the html generation, as that needs to be done too (and read after by the http server).
It's fun to make architecture like in the article though, I'm guilty of it as well with previous hobby projects
OP’s approach has fast cache updates, but it’s still eventually consistent. Like, the whole flow of PG trigger, handled by Ruby server, then writing to a file, that happens outside the flow of POST comments, and could sometimes finish after POST comments returns to the caller.
Also, with my suggestions, you can cache invalidate too, if a short TTL isn’t good enough. Based on writing files to local disk, it seems like a “single server” scenario, so invalidating the in-memory cache on POST comments is dead simple, and can be truly immediately consistent. Likewise, for full page caching, if using Varnish as a cache, Varnish lets you invalidate easily too. NGINX and CDNs don’t (generally), but if a short TTL isn’t enough, and you need instant invalidation on new comments, then just go in-memory cache or Varnish.
> The big idea is to write the comments as static HTML, only when comments change, instead of doing a database query to display them every time
I'm using Cloudflare to do this, and any kind of db updates will update the cache. I'm new to all this, but wouldn't this be spiritually the same, e.g. if the cache would serve the latest db content instead of hammering the db? (I'm doing this because I'm using Airtable as db, and it's incredibly slow, rather than a "hug of death" kind of scenario)
The big idea is to write the comments as static HTML, only when comments change, instead of doing a database query to display them every time. This prevents the “hug of death” if you get a burst of traffic.
I believe this was the technique used (and still being used?) by 2chan, the ancestor of the infamous 4chan. It didn't even need a database, as it was entirely file-based and simply appended each post to the appropriate file. Visitors were served static HTML with no JS required.
Yep, I'm surprised the OP doesn't use such a scheme, possibly with something SSE-ish so you don't mix the content and comments on-disk but they're both retrieved as a single request.
This way all you need is a POST endpoint (which might just be a cgi script) to receive the comment and update the comment file.
Gets more complicated if you support threaded comments or pagination, but for a linear comment system it should be more than enough.
That's similar to what was my firtst message board done. It was a static HTML file with a form. The form submitted to a separate PHP script that appended the comment to the original HTML.
> Another interesting approach would be to write the comments into each HTML file directly, instead of in a separate file, so you wouldn’t need JavaScript at all.
A better approach IMO. Why require js for something as trivial as adding a small block static html to a page every time it loads? Makes no sense.
This is how the first discussion board on the web was written to work. Search for Matt's script WWWBoard. Perl script added the posted comment with date into the HTML file's insertion point.
At that time in the 90s this was heavily deployed everywhere. This was before PHP-BB and other database based boards got popular.
Yes, and we have 100x the DRAM capacity, 1000x CPU Performance if we include MultiCore, and 10,000x faster IOPS on SSD. We could have gone back to the old ways of doing things and it would ( or should ) actually be more scalable.
I've got an idea to allow comments to be appended to the HTML and counting on the fs implementation synchronization, so only allow comments around 4k (or whatever is prooved to be safe) for a complete single append. It would require the page to not have a footer and closing tags for </body></html>.
This has to be a joke.
If the Ruby implementation that saves the comment would simply contain the logic to save the static HTML file - the obvious implementation - no Postgres events are required.
Having a database separates the view from the model. If you want to modify the generated HTML, it's much easier with the comments being separate from the HTML itself.
I don’t say to get rid of the database. Of course it makes sense to store the comments there. I say get rid of 1) the database events and 2) the Ruby logic to listen to these events. This can be done by the Ruby saving logic. It’s just over engineering to use events.
Thanks for the suggestion! As I understand Hypothesis, it is a social tool for users to annotate any web page; as a website owner, there is nothing I need to do, right? In that case, it doesn't match the original intent of my blog post, but I think I should still mention it.
You're totally right that this would be something that the users would be using... perhaps the only requirement as a website owner would be to properly configure the Content Security Policy (CSP) HTTP headers [0].
Is there a point in using NOTIFY in PostgreSQL, instead of just generating the HTML file when saving a new comment? The performance cost of doing a simple SELECT and writing a file in the request handler is neglible (especially since you are talking to the database anyway), but a NOTIFY-less solution is far simpler and will work with any database, even sqlite.
84 comments
[ 3.3 ms ] story [ 148 ms ] threadIt's the right setup for write rarely, read often scenarios.
If you've got Ruby listening for updates with a postgres database all tied into a content generator it's probably time to admit you don't have a static website anymore and just look for best practice ways to go about this (with a bit more security!).
We might also be blurring lines of static sites vs interactive sites. You can have a static site that does nothing but allows the user to scroll to view it. You can also have static sites that change and do things based on the user's input. Entire games have been made using client-side JS but is still served statically.
https://simonhearne.com/2020/network-faster-than-cache/ has some discussion.
It doesn't. Browsers will happily render half an HTML page as it's streamed in. There's an interesting HTTP 203 on this. https://web.dev/shows/http-203/LLRig4s1_yA/ if you're interested.
I guess it's a completely foreign concept in the age of SPAs when everything is dynamically generated in the browser.
To comment a visitor takes any url on the domain and appends "/@say/". Like http://notmyurl.com/somepage.html/@say/Some response to somepage. Or "...lakephoto.jpg/@say/Cool fish! How long was it?"
The perl script sees the /@say/ in the logs and adds the parsed out and sanitized comment to an .html file. There's some nginx location hijinks for matching /@say/ URLs that goes to a confirmation page and redirects to the comment listing page.
I clicked on your link expecting a real site (my monkey mind having skipped over the actual URL contents) and I was redirected to some other odd Chinese (?) URL with spammy ads and NSFW content.
https://nextjs.org/docs/basic-features/data-fetching/increme...
If your application is down when the NOTIFY is triggered that comment is lost, isn’t it?
It would be so nice to have a comment system like Disqus but simple like described here, that also had an easy tie-in to a spam prevention service (or would work with a trained spam prevention model).
A separate authentication layer is entirely possible, at the loss of anonymity.
Worried that Postgres will tip over? Just read from the DB, but cache the results in memory with a short TTL on your Ruby server.
Or, if you’re worried about not just Postgres, but also your Ruby server getting overloaded, just go with a fully server rendered page, and cache the whole page. Could use NGNIX cache, as you’re already using NGINX, or a CDN, or Varnish, whatever. No JS loaded comments at all.
Either way, caching should be able to get you tonnes of scalability, easily avoid hug of death, without resorting to PG triggers calling a Ruby server that writes files to disk.
With that being said, this approach works too, and if it’s just a personal page with you as the only dev, you understand it, so it’s all good. But I’d certainly avoid this approach at a company, largely because it’s unusual and will take new devs longer to wrap their heads around.
It's fun to make architecture like in the article though, I'm guilty of it as well with previous hobby projects
Also, with my suggestions, you can cache invalidate too, if a short TTL isn’t good enough. Based on writing files to local disk, it seems like a “single server” scenario, so invalidating the in-memory cache on POST comments is dead simple, and can be truly immediately consistent. Likewise, for full page caching, if using Varnish as a cache, Varnish lets you invalidate easily too. NGINX and CDNs don’t (generally), but if a short TTL isn’t enough, and you need instant invalidation on new comments, then just go in-memory cache or Varnish.
I'm using Cloudflare to do this, and any kind of db updates will update the cache. I'm new to all this, but wouldn't this be spiritually the same, e.g. if the cache would serve the latest db content instead of hammering the db? (I'm doing this because I'm using Airtable as db, and it's incredibly slow, rather than a "hug of death" kind of scenario)
I believe this was the technique used (and still being used?) by 2chan, the ancestor of the infamous 4chan. It didn't even need a database, as it was entirely file-based and simply appended each post to the appropriate file. Visitors were served static HTML with no JS required.
This way all you need is a POST endpoint (which might just be a cgi script) to receive the comment and update the comment file.
Gets more complicated if you support threaded comments or pagination, but for a linear comment system it should be more than enough.
A better approach IMO. Why require js for something as trivial as adding a small block static html to a page every time it loads? Makes no sense.
At that time in the 90s this was heavily deployed everywhere. This was before PHP-BB and other database based boards got popular.
Here is an incomplete try: https://github.com/hadrianw/compend
Lock and unlock is there for moments when someone wants to modify the content or moderate comments while still allowing commenting.
[0] https://web.hypothes.is/
[0] https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP