I hope with the IPFS and nix stuff I've worked on, we could someday have a world where you can distribute and archive the source of a static site, and the thing can be built and rendered on demand without tons of security and reproducibility issues.
I think this is an important “feature” but find that people either get or don’t. When I describe how cool it is that the version and history of a post is included in git and therefore reliable, I get sort of pleasant nods. But then people will fixate on having bullet lists in tables or something and give up on markdown.
Not that blog posts are life changing or anything, but having a format that’s durable and reliable seems important.
I lost my blog from college back in 95 when I got my first job and didn’t think to archive my shell account. I wish I had, even as a memento.
One thing I think that factors into it is that age is kind of nice for slowly removing stuff that’s not used from existence. Posts from 20 years ago might not be good to keep around if no one is reading them. No the gradual degradation that comes naturally from new phones, computers, hosts, jobs might be a feature for some people who don’t necessarily want everything around forever.
This is pretty much how I do it. I built my SSG using Hakyll and host the resulting binary on my own Debian repo, that way I can write a post in any flavour of text that Pandoc supports (though I go with Markdown for simplicity) and within about 20 seconds my CI pipeline has built and pushed a new version of my site (using scp). Easily done by myself too but at least I know with the CI approach that I'm writing to git first and then publishing.
I'm part way through the implementation of showing the commit history for each post and linking it to the commit hosted on sourcehut. It provides another view on the content the same as hosting a gopher or gemini version would: clone it and read the things in the posts folder however you want.
My web server hosts that directory with indexing enabled, but I don't use apache for it like most examples do. There's nothing special about it, it's just a directory tree built in a way that apt likes. (https://pkg.kamelasa.dev/). In fact, the entire configuration of the repo is visible there.
There's a step in the middle where I sign the packages with my GPG key, and the public key is available on Ubuntu's keyserver (http://keyserver.ubuntu.com/).
I don't need to run this workflow very often, as it'll take about 40 minutes to rebuild and push. But if I do update my SSG I know it'll end up in my debian repo with a version bump, so I'm happy.
On a second pipeline I can just do a simple 'add-apt-repository' and 'apt install'.
The longest living websites are forgotten accidents. It's always some ancient webserver using maybe a single RAID-1 array of two tiny spinning disks, running forgotten software that is never ever updated. The uptime on those boxes are not uncommonly measured in decades. Somebody's credit card just keeps gets charged $40 a year ($10 for the domain and $30 for the website hosting), and the machine never gets touched.
Typically it sits in the back of a dusty rack for a website hosting vendor (or, rarely, a colo provider) and the gear has long since paid for itself, but also is unmaintainable due to not even having the remotest semblance of a service warranty or service parts (other than what somebody might have ordered as spares years before). If it ever loses power, everything starts back up on boot time and it keeps on chugging, defying the usual laws of computer entropy.
I have a similar setup for my blog and I'm using Netlify for both CI and hosting (free). It's been great! GitHub Actions could probably work well for you if you already have decent hosting.
Any particular guide or set of docs you recommend for that? I've heard a lot of good things about Netlify as far as hosting, and I've been wanting to set up a blog; seems like as good a time as ever to start learning.
I've evaluated Netlify for my blog at https://jdsalaro.com and I'd advise against that unless you're missing a CMS on your static site generator. Other than that I personally felt it didn't bring much to the table. I do admit that their CMS is slick, but it wasn't really a feature I needed.
Of course I'm not sure what other people are using them for, but in my case GitLab/GitHub covered all my bases pretty well since I wasn't looking for a pelican/hugo CMS and you still host somewhere else (not Netlify).
Github's main business is not Github Pages or be a CDN.
Netlify's main business is.
Netlify also provides deployment previews for commits and PRs, and a complete CI/CD pipeline that you would have to implement yourself with Github Actions or something else.
The CMS isn't what I need either, but there is so much more than just hosting.
Netlify seems to lack basic testing of their client. For example, I found that if you have a repo not hosted on GitHub, trying to publish it will cause the client to dump a stack trace and crash. That, the general whizbang design of the UI, and the state of the docs left me with a poor impression about the quality of Netlify's offerings. "Good" for a free service, maybe, but all their attempts at fancy integrations makes the whole thing less attractive, since it actually creates hassle (for a person needing to dodge it all just to make the most basic, vanilla use out of it).
The only middle-way sort of solution I found a long time ago, even before joining the company, while looking where to host https://jdsalaro.com was to use GitLab, they being open-source and quite public about their, now our, values.
Nowadays I'm planning on writing again and have tried a setup that seems to be alright:
- migrate from Pelican to Hugo since the community is more vibrant, responsive and the themes much more mature and well maintained.
- Use a mature, well-maintained documentation-oriented template as base (I based https://jdsalaro.com off Pelican's bootstrap3)
- Create an Obsidian vault out of the posts/ directory to improve note-taking and editing capabilities.
- Host same as before on GitLab
- Deploy via CI/CD copying the posts/ directory to a public repository
- Probably mirroring to GitHub
I'll be able to report back once I've gotten the above fleshed-out :)
How does having the "BSD Zero Clause" with your content work? I open sourced my blog's software, but I still keep all the actual content in a separate private repo, since I don't want my written content distributed under the same license. Also works out well, since I have the github action automation in the private content repo.
I can still easily change the private flag down the road if I were to decommission it (though, unlikely -- I'd just archive it somewhere).
I wish people would stop using github as a swiss knife CDN (free hosting, hooks, history through git repo, etc) and build higher altitude solution that could leverage git/MD/whatever but with free/opensource/selfhosted/alternative tools (like gitea instead of github and minio instead of S3 for instance).
Not really, "free/opensource/selfhosted/alternative" [0] as in "here this minio bucket that behaves like S3 but you can play with it without an amazon bill of death that may or may not be coming".
[0] Also, thanksnotreally for cherry-picking and putting words in my mouth with your use of `!M$`, screw you too.
Setting aside how weary I am of "M$" (it's just about tied with using "sheep" to describe Apple users), what on God's green earth does Microsoft have to do with the OP's point? GitHub itself is not and was never open source. It's not like Microsoft bought it and closed off the source. Do you think using Microsoft tools demonically injects anti-FSF sentiment into your work? ("I started using VS Code, and now whenever I try to work on GPL 3 software, my keyboard becomes too hot to touch until I close the window!")
Personally, I have a different take than the OP, although I'm not sure this is really a disagreement. I would argue open data formats are more important in this context. If I write text in Markdown, it doesn't matter whether I'm using Emacs or a closed-source, proprietary commercial editor; the text isn't tied in any meaningful way to the editor. Likewise, a Git repository hosted on GitHub isn't tied in any meaningful way to GitHub. It's... a git repo.
As long as GitHub isn't doing anything that I particularly object to -- and "oh woe, it is owned by Microsoft" is not an objection I share -- there's a pretty good case for continuing to use it. I'm taking a calculated risk that it's both unlikely to go away anytime soon and to markedly change business direction in an unfriendly way, but if it does? As long as my local copy is up to date, I can re-publish it anywhere and just, well, stop using GitHub.
> build higher altitude solution that could leverage git/MD/whatever but with free/opensource/selfhosted/alternative tools
Not that I disagree with this, but that goes almost directly against "Mak[ing] that source public via GitHub or another favorite long-lived, erosion-resistant host. Git’s portable, so copy or move repositories as you go."
Any self-hosted service or solution is going to not be erosion-resistant by virtue of not being for-profit.
> Any self-hosted service or solution is going to not be erosion-resistant by virtue of not being for-profit.
And for-profit services are? Often enough it's the VCs which ruin a service by pressing out money before it inevitably dies because the experience has been degraded to an extreme degree.
I used to have my blog source on GitHub, but then it turned out I didn't want my half-finished works-in-progress public. To use a private repository would rather defeat the point; using a private repo and a public fork is inviting confusion. Now I just use a private repo on my own server, cloned to my dev machine. Does anyone have a usable solution for that problem?
I knew someone would say that. It's not about imperfections per se; more that I sometimes write stuff that I then reconsider and delete before it ever sees print. Admittedly nowadays I mostly self-censor that stuff before even starting to write it, but a few years ago I wrote a lot more stuff that, after reflection, I considered it would be unwise to go ahead and publish.
I personally don't move anything into git until it's close to ready. WIP writing is kept on Bear (Markdown editor). But there are tons of other apps, including ones that sync to a cloud.
I have a branch and remote called "publish", and a branch and remote called "draft". I suppose if I ever accidentally pushed the draft branch to the publish remote, I'd just delete the branch from the remote.
I only keep the output on github. So the generator folder with my drafts, posts etc is on my local machine, and whatever output it generates is in its own folder that is a git repo.
I've been thinking about this a lot. I created Ponder so that I could quickly create drafts in my browser. Then I planned to move them to my blog (Markdown on GitHub) when I'm done.
I need more than 10 drafts (the current limit in Ponder), but I feel like I'm getting close to a solution that works better for me. I also want to be able to edit across machines (Mac, Chromebook, Phone), so I need to make the app work online.
Using git submodules might work, but it's not very convinient in my experience. I'm now thinking about having a GitHub Action in my private repo that pushes all public posts into a separate public repository automatically
I started my blog about python/django (https://simpleisbetterthancomplex.com) using Jekyll and hosting on Github Pages and it was pretty good to get started because back then I wasn't sure if I would keep it up or not.
After a year or so I migrated to a 5 USD droplet on Digital Ocean (back then GH Pages didn't offer https for custom domains) and integrated with Github webhooks to automate the deployment when pushing new markdown to the main branch.
Over the time it indeed started to degrade. The build time takes almost a minute, but after building the website is just a bunch of static html pages.
Nowadays it is annoying to write new posts because I like to write locally and refresh to browser to check if it's looking good or not. So I would say it degraded for me but for the reader it's still as fast as it was when there was just a couple of posts.
I thought about migrating the blog to something else, but because I used some custom markdown extensions for code highlight and other things, it would be painful to migrate all the blog posts. So I've been postponing it since 2019.
yes because of a bad design decision I made to create a custom "related posts", which is a code that searches all my markdowns to pick the best pages and add to the current page. so even with `--incremental` it take quite some time to regenerate
20 years ago that was my first useful custom CGI script, it took the metadata of the current page and ran a grep for pages with similar tags. Running that was really fast and flexible until I published it and got my first ten simultaneous visitors.
Then I learned about how hard it is to use a cache.
to be honest jekyll still serves the purpose really well (especially for the reader)
the site is still sitting on an inexpensive entry-level cloud server, it's relatively fast, serves 300k+ page views every month and it give me some passive income
i'm pretty sure that if one uses the framework properly it can still hold on very well
i think what left me with a mixed feeling about static site generators is that it is constantly holding you back whenever you think about growing your website. i ended up building a django api that runs on the same vpc as the blog itself, and i used it to expand some features, like reading from google analytics the total page views for a given blog post, or consuming the disqus api to list the latest comments on the home page. this kind of thing.
managing the posts in static files is quite challenging as the number of posts grows. at some point you will want to change some info, or add a certain metadata and you will need to write a script to walk the _posts dir and edit the files (maybe there's a better way of doing that :P)
I wonder if you could save some time with some caching?
Find all posts related to "django", select the top 3, pin them as the related posts for all django entries? You'd trade out some uniqueness and slightly increased memory usage to gain speed.
(I just got started blogging and found https://github.com/sunainapai/makesite python static site generator worked well for me. Obviously you're probably a bit far in to switch horses by now.)
Something similar happened to me when I was using static site generators. In fact one that I was really enjoying even switched programming languages between 1.x and 2.0.
Since that time I look for the people and community behind the project, and try to find signs of stability and long-term care. After that I look at open formats rather than open and flexible architecture-chains. For example, I'd rather use my LibreOffice HTML template and simple PHP controller on a more monolithic (but open) platform than connect a bunch of technologies together to create a build process with a bunch of moving, quickly developing, interdependent parts.
Not sure it's the best answer, but it has worked better to use more monolithic software, even blogging software that's been in steady, if slow development since the early 2000s...
It's a simple injection controller for adding links and markup. Very rough. It doesn't do much about the existing HTML, which I felt was mostly workable, or workaround-able. :-) You can see some output here:
There was a company on here the other day talking about their product, built on top of Docker. I wish I'd bookmarked it.
Their secret sauce is, effectively, partial evaluation in Docker images. They run the code to detect if any of the changes in a layer have side effects that require that layer to be rebuilt (which invariably causes every layer after to be rebuilt)
I mention this because if I'm editing a single page, I would like to be able to test that edit in log(n) time worst case. I can justify that desire. If I'm editing a cross-cutting concern, I'm altering the very fabric of the site and now nlogn seems unavoidable. Also less problematic because hopefully I've learned what works and what doesn't before the cost of failure gets too large. It would be good if these publishing tools had a cause-and-effect map of my code that can avoid boiling the ocean every time.
Correct me if I'm wrong, but don't you have to formally map out all of your inputs and outputs for partial recompilation to work?
I believe the last time I touched make was to fix an exceedingly badly mapped out chain of cause and effect that would consistently compile too much and yet occasionally miss the one thing you actually needed.
Partial evaluation works out the dependencies by evaluating all of the conditional logic and seeing which inputs interact with which outputs.
> Correct me if I'm wrong, but don't you have to formally map out all of your inputs and outputs for partial recompilation to work?
Make needs that information, but that doesn't mean you need to be the one to catalog it. I found this article pretty useful [1]; in a nutshell, many compilation tools (including gcc, clang, and erlc) are happy to write a file that lists the compile time resources they used, and you can use that file to tell Make what the dependencies are. It's a bit meta, so it made my brain hurt a bit, but it can work pretty well, and you can use it with templated rules to really slim down your Makefile.
(I've tested this with GNU Make; don't know if it's workable with BSD Make)
Don’t know about Jekyll but with Hugo you just run hugo server in the Git repo and it will give a a live preview that you can view in the browser, served locally.
It’s very fast.
I have a mockup blog with ~90 pages and it takes 190 ms to generate the whole site.
Jekyll would probably handle that in a few seconds.
But, once you approach two to four or five hundreds, it can quickly add up to a minute or two, making it impractical to say the least.
One solution is to run `jekyll build`, which will just build HTML to a directory, and then just removing old Markdowns and serving those generated HTMLs directly via nginx or something.
I've honestly given up and switched to Ghost, where I don't have to worry about that sort of stuff.
If you use "bundle exec jekyll serve" you shouldn't have too much problems locally as it just rebuilds the pages that change on every save. A minute to deploy the finished version is not terrible by any stretch for a blog IMO.
I wrote my own blogging software, and went through a similar journey. Initially I wrote a simple Perl script which would read all the posts I'd written (as markdown), insert them into a temporary SQLite database, and then using the database I'd generate output.
Having the SQLite database made cooking up views really trivial (so I wrote a plugin to generate /archive, another to write /tags, along with /tags/foo, /tags/bar, etc). But the process was very inefficient.
Towards the end of its life it was taking me 30+ seconds to rebuild from an empty starting point. I dropped the database, rewrote the generator in golang, and made it process as many things in parallel as possible. Now I get my blog rebuilt in a second, or less.
I guess these days most blogs have a standard set of pages, a date-based archive, a tag-cloud, and per-tag indexes, along with RSS feeds for them all. I was over-engineering making it possible to use SQL to make random views across all the posts, but it was still a fun learning experience!
I made goode experiences with Lektor (a static site generator). If you run it locally using "lektor server" you see your page plus admin backend. Once you click (or type) deploy the page gets statically built and copied somewhere via rsync.
You can still have the sources in a git for quick rollbacks.
What I like about lektor over most CMS solutions is that it is more easily adjustable. Basically Jinja2 and python held together with glue
Thanks for your Blog! It's a great resource - I love the generic form template and signal blog post.
It is really great and I recommend it to anyone who uses django
Some static generators like eleventy and Gatsby offer partial builds, only building the new posts etc. which should be considerably faster. Another thing would be to run an empty version of your site with only the text you're working on and when it's done, moving it to the proper site and building it then
I worked on a wiki that fell off the web; I've thought about recovering it from scrapes, and rewriting entries so that search wasn't needed, so that the Wayback Machine mirror worked better. Maybe using Kiwix tools to make a self-contained, downloadable backup, too.
Something that I feel is missing here is the problem with dead links. Even if the same content is still available on the very same blog, links might change when you decide to change your stack.
I might still be able to get around it, but it's still a pain.
This is the kind of thing pocket/instapaper do to extract the main content from a page in a format that's easier to read (and also probably to programmatically modify)
For all the (deserved) complaints people have about wordpress, it's been around since before git or markdown was even a thing. For all its faults, its been pretty resilient to time.
The real reason most websites disappear is a much more human one.
Yes, if you constantly update your installation, which is the exact opposite of what this blog post is about. Otherwise you run a pretty good risk of your installation being hijacked, overtaken by spam bots or similar after a few years.
Yep. I run a few WordPress sites and the amount of maintenance with them is crazy. The plugin model makes maintenance brutal too since it's all independent devs with different release cycles, testing practices, etc.
Following the philosophy of simple hosting + MD outlined in the OP then the WP equivalent would be something like a simple (official twenty*, or your own custom) WP theme with zero (or minimal amount of) plugins. Enable auto-updates on WP and you're done.
The more plugins you use, the faster entropy kicks in without maintenance.
Agreed, if you stick with the twenty* themes and keep it simple, you can get pretty far with very little maintenance.
For simple sites, especially where we don't do comments, I've started using Simply Static though to just generate pre-rendered pages. All the benefits of static site builder with all the conveniences of a CMS.
I've had it on my TODO-list to try out that plugin, thanks for reminding me!
How many sites have you tested it on, have you had any issues? I'm gonna play around with it now but what I'm mostly concerned about now is how you manage the site if you point the web server to the static directory.
I made a prototype at one point where you have your actual WP site in a subdirectory, like example.com/wordpress, which is IP restricted (or behind http auth). Then you crawl that site with wget (or curl) to generate static HTML and to finish it off by search/replacing the HTML files (removing the /wordpress portion). Then you serve the static files to general users.
If you happen to ever read this, I ended up with a setup similar to the hacky one I did before:
- WP moved to subdirectory (WP home and siteurl changed in DB)
- Static site served normally from (and generated to) webroot
Just needed to add sitemaps and robots.txt to the config to be included. Workflow is super smooth. Login to example.com/wordpress/wp-admin/, view your changes at example.com/wordpress and when you're happy, publish the site through the plugin.
Curious to see what kinds of issues I might be running into later on (this was just a quick test).
Same story with Jenkins, the plugin ecosystem is just ridiculous. If Linux distros did the same thing with packages, Linux would be broken literally all the time.
WordPress core yes, and it's pretty safe, but if you use many plugins (which most people do) that gets a lot riskier. Premium plugs are pretty safe but still I've had stuff break.
The kind of person whose needs would be satisfied by a static site generator doesn't seem likely to need 20 plugins, so it doesn't sound like an apples-to-apples comparison.
I don't base on number of plugins, I base on complexity personally.
I mostly agree with you, but in my experience most (not all) people who are capable of using a static site generator will do so. Those who aren't comfortable with it use WordPress. I'm not slamming WordPress, just making an observation.
> The real reason most websites disappear is a much more human one.
So true.
I have had a blog running since 2006, all was php based back then with (html) posts inside a database. The tooling shouldn't be a problem, I switched systems several times (once every few years).
Currently on Hugo with markdown posts. As long as you treat your migration carefully and take some time to migrate, every tool should suffice. It's mostly about human effort and human error when things get lost.
IMO the biggest barrier to blogging (and the cause of most blogs dying) is inconvenience, and minimizing that if the biggest advantage of Markdown + Git. If there's any inconvenience at all, it naturally drags on the process of writing, and writing takes enough time and focus that if there's any friction it's too easy to push things off to the next day.
My co-author and I use Markdown and Git as the author suggests, and one of the best things is that between simple CI/CD pipelines and effortless scaling of a static site, we don't need to do any technical work so there's no friction on my lifestyle. We've been writing for almost a year now, 4+ posts a month, and 99% of that "work" has just been writing.
On the inconvenience front, I think that also makes it clear why so much stuff that would have been blogs, say prior to 2013 to pick a pseudo-random date [1], is that the convenience of various walled gardens got very convenient. It's really easy to post an update to the walled garden social media site of your preference (Twitter, Facebook, TikTok, Tumblr, whatever), and with network effects really conveniently easy to have some sense of readership (even if it just Likes or Faves or whatever).
There are some blogs that I realize will never "come back" so long as "everyone is on Twitter these days". Because Twitter is still so much more convenient that blogs (even ones in Markdown + Git).
[1] Okay, not actually random, it was the Google Reader shutdown year. Google Reader provided a lot of convenience to RSS, including social media-like network effects, that almost brought blogs mainstream.
Any suggestions for static site generators / where to host? I'm thinking about starting a new blog with Hugo / Digital Ocean.
I have a popular Spark blog on Wordpress (https://mungingdata.com/) and can relate to your sentiment that any inconvenience can hold up the writing process. Your post is motivating me to streamline my publishing process.
Hugo is great, and they have a section with tutorials on how to host on a bunch of different platforms, so pick the one that you're most comfortable with or seems best/cheapest:
Gitlab and Github both have static site hosting options. Netlify works great as well.
We've gotten it to the point where we just push new pages via git and it all works. Anything more and you'll find yourself procrastinating ever so little at a time...
Anyone have any experience with Hugo? It started out as a fast single binary static site generator but it seems it is now dependent on Go being installed on the host...
It's not dependent on Go being installed on the host... I'm not sure why you think this is the case. Like most Go applications, it's compiled into a single static binary.
The toolchain has changed significantly in 4+ years. It started as literally a shell script invoking Gruber's original markdown.pl. Then I switched to CommonMark, etc.
But the core data hasn't "rotted" at all, which is good. Unix is data-centric, not code-centric.
Previous thread that mentions that Spolsky's blog (one of my favorites) "rotted" after 10+ years, even though it was built on his own CityDesk product (which was built on VB6 and Windows). He switched to WordPress. Not saying this is bad but just interesting. https://news.ycombinator.com/item?id=25675869
> The toolchain has changed significantly in 4+ years. It started as literally a shell script invoking Gruber's original markdown.pl. Then I switched to CommonMark, etc.
Wouldn't switching formats like that break peoples pages? That's one concern I have with using markdown for a site like this. For anything non-trivial, you either end up using HTML (in which case, what's the point) or some dielect-specific feature.
At the bottom of this article I note one incompatibility I encountered with the reference CommonMark vs. markdown.pl. But I just fixed those and have been using it for 3 years, and it's been great.
----
The key point is to have a diversity of implementations, so you don't get locked into one, which may go away in 10-20 years. In 10-20 years, I'm very confident that there will be CommonMark renderers available.
that's exactly what I did for my blog and achieved 90+ on lighthouse audit. Google is considering core web vitals to rank pages from this year (may 21). I don't want to rely on an external hosting provider who usually loads tons of tracking code, analytics scripts, irrelevant scripts to hosted sites (looking at you Wix).
Your favorite static site generator can be used with free, no-ads hosting at indie https://neocities.org. Their $5/mo plan adds custom domains and higher disk/traffic quotas.
When I last had mine out I went a step further. Just a single static HTML page, with a little CSS in the HEAD. A single request for a single file, easily saved. No build or deploy process. No javascript, no stack of dependencies, no docker containers, no dependence on github or any other third-party site.
For a full blog, I would use a static site generator, but not for something well-served by a single static page.
A bit morbid I know, but I've been thinking I would like my blog to outlive me. I have my blog on blogger.com, and while I've been tempted to switch to something more modern and fun in a container on a VPS, I've resisted, because I think blogger.com is not going away in my lifetime. A VPS and a personal domain on the other hand is going to get shutdown when someday I die and my credit card gets closed. At this stage Github Pages is probably also as likely to stick around as Blogger though.
There is a problem in the CMS industry. Everyone wants to build a headless CMS to power Jamstack sites—only the content itself lives in databases and the CMS is usually a proprietary SaaS. Doesn't leverage Git for content in any way.
This is deeply problematic. We should have dozens, if not hundreds of contenders for "the next WordPress" that leverage Git as a foundational aspect of content management. Instead we have a bunch of Contentful clones (no disrespect to Contentful) with REST and/or GraphQL APIs.
It's bananas that if you search for Git-based CMSes you have NetlifyCMS, and…wait what? Is that all??? Forestry gets mentioned a lot because it's Git-based but that's also a proprietary SaaS. I just don't understand it. Is this a VC problem or a real blind spot for CMS entrepreneurs?
One explanation comes from just looking from the content creator's point of view: they kinda don't care what previous iterations of the work really look like (draft-final-v5-oct27-FINAL.doc anyone?)
The things a content creator might be more interested in aren't really core git strengths: SEO, publishing schedules, editorial back-and-forth, social media, etc.
I agree, I had this debate recently for what to use on a static site and we went with a Git-based CMS (https://content.nuxtjs.org/) over Strapi. I use nuxt/content for my personal site as well and really enjoy working with it.
Maybe FrontAid CMS might be interesting to you, even though it is also proprietary. FrontAid CMS is similar to Forestry, but stores all your content in JSON instead of Markdown. It is therefore better applicable to (web) applications in addition to simple blogs. https://frontaid.io/
> if you search for Git-based CMSes you have NetlifyCMS, and…wait what? Is that all???
There are some more. I know of Grav which I intend to use migrating from NetlifyCMS. It has flat-file storage and Git connectivity. I think it can be installed on my shared hosting provider by just unpacking the thing (I don't have Composer / shell access).
> We should have dozens, if not hundreds of contenders for "the next WordPress" that leverage Git as a foundational aspect of content management.
If you want a new CMS, give it a shot. But nobody's made a better free version of Jenkins either. It's hard to do and completely unrewarding/unmonetizeable (as FOSS), which is probably why nobody has done it.
However, insisting it leverage a difficult source code version control system is artificially restricting. Start with an MVP of flat files for content and add a plugin system. Somebody will write Git support, but they'll also add other content backends. I'll bet you a million dollars that a custom SQL plugin that does version control will be preferred over Git by 99% of users. But they may wise up and use an S3 plugin instead. Or maybe all three. They'll have choices, and your project will become more useful.
Don't start your project with an artificial restriction and it will be better for it.
I’m under the impression there is still plenty more you can do with a database backed CMS then you can with markdown files. For instance difficulty marking pages as related to one another. You could use the slug, but if you later change the slug, you must track down all of the pages that relate to it and update the slug there too. Aka no normalisation.
If you want something to live on the web long after you've lost interest in it static HTML/GitHub pages seems like the way to go. The chess board I built in high school to learn JavaScript is still going strong: https://github.com/PJ-Finlay/JavaScript-Chess-Board
Per the timestamp, that's ~5 years. I've got a personal domain I've had for 20 years now. GitHub wasn't there, and I wouldn't bet a whole lot it will be there in another 20. I think it's likely, but nothing close to guaranteed. Git didn't exist then, either.
It's pretty easy to mirror your own git repos, so that's a nice option, and to have stuff hosted on GitHub, your own, and maybe BitBucket as well. But nothing is going to beat having your own backups--which I DO still have, from as early as 2000.
I run my blog using perhaps the most boring option - Wordpress, with close-but-not-quite the default theme (different fonts mostly). Outside of adding a cover image on every post and occasional footnotes, I don’t really need much.
However, that’s pretty lightweight on decisions I had to actually make to publish, and all the alternatives seem to be more involved. I wouldn’t mind migrating off WordPress, but just on the theming side that has a decent chance of involving a non-free theme, making the idea of hosting it on a public repo somewhat of a non starter...
I cut my teeth on Hugo and mdbook, and mkdocs was a breath of fresh air. No crazy templating, no pseudo-markdown index files, and lots of flexibility in the site and page structure.
181 comments
[ 2.0 ms ] story [ 259 ms ] threadNot that blog posts are life changing or anything, but having a format that’s durable and reliable seems important.
I lost my blog from college back in 95 when I got my first job and didn’t think to archive my shell account. I wish I had, even as a memento.
One thing I think that factors into it is that age is kind of nice for slowly removing stuff that’s not used from existence. Posts from 20 years ago might not be good to keep around if no one is reading them. No the gradual degradation that comes naturally from new phones, computers, hosts, jobs might be a feature for some people who don’t necessarily want everything around forever.
I'm part way through the implementation of showing the commit history for each post and linking it to the commit hosted on sourcehut. It provides another view on the content the same as hosting a gopher or gemini version would: clone it and read the things in the posts folder however you want.
I then scp the file into the pool directory on my server, and re-run a script that calls apt-ftparchive and regenerates the contents of the repo (https://manpages.debian.org/buster/apt-utils/apt-ftparchive....).
My web server hosts that directory with indexing enabled, but I don't use apache for it like most examples do. There's nothing special about it, it's just a directory tree built in a way that apt likes. (https://pkg.kamelasa.dev/). In fact, the entire configuration of the repo is visible there.
There's a step in the middle where I sign the packages with my GPG key, and the public key is available on Ubuntu's keyserver (http://keyserver.ubuntu.com/).
I don't need to run this workflow very often, as it'll take about 40 minutes to rebuild and push. But if I do update my SSG I know it'll end up in my debian repo with a version bump, so I'm happy.
On a second pipeline I can just do a simple 'add-apt-repository' and 'apt install'.
Typically it sits in the back of a dusty rack for a website hosting vendor (or, rarely, a colo provider) and the gear has long since paid for itself, but also is unmaintainable due to not even having the remotest semblance of a service warranty or service parts (other than what somebody might have ordered as spares years before). If it ever loses power, everything starts back up on boot time and it keeps on chugging, defying the usual laws of computer entropy.
I haven't setup CI pipeline yet.
It true whats said in the post, I used to save links thinking it won't disappear. But most the links, I had saved, it's no more.
Of course I'm not sure what other people are using them for, but in my case GitLab/GitHub covered all my bases pretty well since I wasn't looking for a pelican/hugo CMS and you still host somewhere else (not Netlify).
Netlify's main business is.
Netlify also provides deployment previews for commits and PRs, and a complete CI/CD pipeline that you would have to implement yourself with Github Actions or something else.
The CMS isn't what I need either, but there is so much more than just hosting.
They can also manage your domain/ssl which is nice so I can manage everything related to my site on Netlify.
Lmk if you need any extra details
When you feel you want more, you can link a git repo and point a domain name at it. When you update the repo, it automagically updates the site.
It's my goto for all new projects.
There is great documentation at every step.
Nowadays I'm planning on writing again and have tried a setup that seems to be alright:
- migrate from Pelican to Hugo since the community is more vibrant, responsive and the themes much more mature and well maintained.
- Use a mature, well-maintained documentation-oriented template as base (I based https://jdsalaro.com off Pelican's bootstrap3)
- Create an Obsidian vault out of the posts/ directory to improve note-taking and editing capabilities.
- Host same as before on GitLab
- Deploy via CI/CD copying the posts/ directory to a public repository
- Probably mirroring to GitHub
I'll be able to report back once I've gotten the above fleshed-out :)
I can still easily change the private flag down the road if I were to decommission it (though, unlikely -- I'd just archive it somewhere).
I'm also using Gatsby if anyone is curious (https://github.com/cbeley/beleyblog & https://chrisbeley.com).
[0] Also, thanksnotreally for cherry-picking and putting words in my mouth with your use of `!M$`, screw you too.
Personally, I have a different take than the OP, although I'm not sure this is really a disagreement. I would argue open data formats are more important in this context. If I write text in Markdown, it doesn't matter whether I'm using Emacs or a closed-source, proprietary commercial editor; the text isn't tied in any meaningful way to the editor. Likewise, a Git repository hosted on GitHub isn't tied in any meaningful way to GitHub. It's... a git repo.
As long as GitHub isn't doing anything that I particularly object to -- and "oh woe, it is owned by Microsoft" is not an objection I share -- there's a pretty good case for continuing to use it. I'm taking a calculated risk that it's both unlikely to go away anytime soon and to markedly change business direction in an unfriendly way, but if it does? As long as my local copy is up to date, I can re-publish it anywhere and just, well, stop using GitHub.
Not that I disagree with this, but that goes almost directly against "Mak[ing] that source public via GitHub or another favorite long-lived, erosion-resistant host. Git’s portable, so copy or move repositories as you go."
Any self-hosted service or solution is going to not be erosion-resistant by virtue of not being for-profit.
Of course the end-result is easy to move out but the build pipeline is more dependent on GH.
Not that it matters but it could be cool too to pipe SSG's output through a managed VPS just to show you don't need Github to host your static files.
And for-profit services are? Often enough it's the VCs which ruin a service by pressing out money before it inevitably dies because the experience has been degraded to an extreme degree.
Edit: this is easy, in fact: you make a copy of your 'publish' branch on the publishing site, call it 'draft', and then protect it.
I need more than 10 drafts (the current limit in Ponder), but I feel like I'm getting close to a solution that works better for me. I also want to be able to edit across machines (Mac, Chromebook, Phone), so I need to make the app work online.
Anyway...
https://ponder.joeldare.com
After a year or so I migrated to a 5 USD droplet on Digital Ocean (back then GH Pages didn't offer https for custom domains) and integrated with Github webhooks to automate the deployment when pushing new markdown to the main branch.
Over the time it indeed started to degrade. The build time takes almost a minute, but after building the website is just a bunch of static html pages.
Nowadays it is annoying to write new posts because I like to write locally and refresh to browser to check if it's looking good or not. So I would say it degraded for me but for the reader it's still as fast as it was when there was just a couple of posts.
I thought about migrating the blog to something else, but because I used some custom markdown extensions for code highlight and other things, it would be painful to migrate all the blog posts. So I've been postponing it since 2019.
Then I learned about how hard it is to use a cache.
to be honest jekyll still serves the purpose really well (especially for the reader)
the site is still sitting on an inexpensive entry-level cloud server, it's relatively fast, serves 300k+ page views every month and it give me some passive income
i'm pretty sure that if one uses the framework properly it can still hold on very well
i think what left me with a mixed feeling about static site generators is that it is constantly holding you back whenever you think about growing your website. i ended up building a django api that runs on the same vpc as the blog itself, and i used it to expand some features, like reading from google analytics the total page views for a given blog post, or consuming the disqus api to list the latest comments on the home page. this kind of thing.
managing the posts in static files is quite challenging as the number of posts grows. at some point you will want to change some info, or add a certain metadata and you will need to write a script to walk the _posts dir and edit the files (maybe there's a better way of doing that :P)
Find all posts related to "django", select the top 3, pin them as the related posts for all django entries? You'd trade out some uniqueness and slightly increased memory usage to gain speed.
(I just got started blogging and found https://github.com/sunainapai/makesite python static site generator worked well for me. Obviously you're probably a bit far in to switch horses by now.)
Since that time I look for the people and community behind the project, and try to find signs of stability and long-term care. After that I look at open formats rather than open and flexible architecture-chains. For example, I'd rather use my LibreOffice HTML template and simple PHP controller on a more monolithic (but open) platform than connect a bunch of technologies together to create a build process with a bunch of moving, quickly developing, interdependent parts.
Not sure it's the best answer, but it has worked better to use more monolithic software, even blogging software that's been in steady, if slow development since the early 2000s...
I’m currently on Hugo 0.80.0 and who knows, it might take years before I upgrade it.
Could you tell me more about this? I've had a similar idea but the HTML that LibreOffice generates is very gross
https://www.friendlyskies.net/fmk/
Their secret sauce is, effectively, partial evaluation in Docker images. They run the code to detect if any of the changes in a layer have side effects that require that layer to be rebuilt (which invariably causes every layer after to be rebuilt)
I mention this because if I'm editing a single page, I would like to be able to test that edit in log(n) time worst case. I can justify that desire. If I'm editing a cross-cutting concern, I'm altering the very fabric of the site and now nlogn seems unavoidable. Also less problematic because hopefully I've learned what works and what doesn't before the cost of failure gets too large. It would be good if these publishing tools had a cause-and-effect map of my code that can avoid boiling the ocean every time.
I believe the last time I touched make was to fix an exceedingly badly mapped out chain of cause and effect that would consistently compile too much and yet occasionally miss the one thing you actually needed.
Partial evaluation works out the dependencies by evaluating all of the conditional logic and seeing which inputs interact with which outputs.
Make needs that information, but that doesn't mean you need to be the one to catalog it. I found this article pretty useful [1]; in a nutshell, many compilation tools (including gcc, clang, and erlc) are happy to write a file that lists the compile time resources they used, and you can use that file to tell Make what the dependencies are. It's a bit meta, so it made my brain hurt a bit, but it can work pretty well, and you can use it with templated rules to really slim down your Makefile.
(I've tested this with GNU Make; don't know if it's workable with BSD Make)
[1] http://make.mad-scientist.net/papers/advanced-auto-dependenc...
It’s very fast.
I have a mockup blog with ~90 pages and it takes 190 ms to generate the whole site.
But, once you approach two to four or five hundreds, it can quickly add up to a minute or two, making it impractical to say the least.
One solution is to run `jekyll build`, which will just build HTML to a directory, and then just removing old Markdowns and serving those generated HTMLs directly via nginx or something.
I've honestly given up and switched to Ghost, where I don't have to worry about that sort of stuff.
It is really astonishing what power our devices have and how little that power is utilised by many tool chains.
Having the SQLite database made cooking up views really trivial (so I wrote a plugin to generate /archive, another to write /tags, along with /tags/foo, /tags/bar, etc). But the process was very inefficient.
Towards the end of its life it was taking me 30+ seconds to rebuild from an empty starting point. I dropped the database, rewrote the generator in golang, and made it process as many things in parallel as possible. Now I get my blog rebuilt in a second, or less.
I guess these days most blogs have a standard set of pages, a date-based archive, a tag-cloud, and per-tag indexes, along with RSS feeds for them all. I was over-engineering making it possible to use SQL to make random views across all the posts, but it was still a fun learning experience!
You can still have the sources in a git for quick rollbacks.
What I like about lektor over most CMS solutions is that it is more easily adjustable. Basically Jinja2 and python held together with glue
I run a Django development agency, and I've passed around links to your tutorials more times than I can reliably estimate; hundreds of times :)
It's a great resource, thank you for it.
I’d make a simple static blog generator but then grow tired of some shortcomings, which then led to abandonment.
My current blog has an HTML rendered homepage but all the posts are text files. https://blog.webb.page
I might still be able to get around it, but it's still a pain.
https://github.com/mauidude/go-readability
This is the kind of thing pocket/instapaper do to extract the main content from a page in a format that's easier to read (and also probably to programmatically modify)
The real reason most websites disappear is a much more human one.
The more plugins you use, the faster entropy kicks in without maintenance.
With WP in general though I definitely agree.
For simple sites, especially where we don't do comments, I've started using Simply Static though to just generate pre-rendered pages. All the benefits of static site builder with all the conveniences of a CMS.
How many sites have you tested it on, have you had any issues? I'm gonna play around with it now but what I'm mostly concerned about now is how you manage the site if you point the web server to the static directory.
I made a prototype at one point where you have your actual WP site in a subdirectory, like example.com/wordpress, which is IP restricted (or behind http auth). Then you crawl that site with wget (or curl) to generate static HTML and to finish it off by search/replacing the HTML files (removing the /wordpress portion). Then you serve the static files to general users.
- WP moved to subdirectory (WP home and siteurl changed in DB)
- Static site served normally from (and generated to) webroot
Just needed to add sitemaps and robots.txt to the config to be included. Workflow is super smooth. Login to example.com/wordpress/wp-admin/, view your changes at example.com/wordpress and when you're happy, publish the site through the plugin.
Curious to see what kinds of issues I might be running into later on (this was just a quick test).
I don't base on number of plugins, I base on complexity personally.
I mostly agree with you, but in my experience most (not all) people who are capable of using a static site generator will do so. Those who aren't comfortable with it use WordPress. I'm not slamming WordPress, just making an observation.
So true.
I have had a blog running since 2006, all was php based back then with (html) posts inside a database. The tooling shouldn't be a problem, I switched systems several times (once every few years).
Currently on Hugo with markdown posts. As long as you treat your migration carefully and take some time to migrate, every tool should suffice. It's mostly about human effort and human error when things get lost.
My co-author and I use Markdown and Git as the author suggests, and one of the best things is that between simple CI/CD pipelines and effortless scaling of a static site, we don't need to do any technical work so there's no friction on my lifestyle. We've been writing for almost a year now, 4+ posts a month, and 99% of that "work" has just been writing.
Writing with a partner helps a ton as well.
It's been great.
There are some blogs that I realize will never "come back" so long as "everyone is on Twitter these days". Because Twitter is still so much more convenient that blogs (even ones in Markdown + Git).
[1] Okay, not actually random, it was the Google Reader shutdown year. Google Reader provided a lot of convenience to RSS, including social media-like network effects, that almost brought blogs mainstream.
I have a popular Spark blog on Wordpress (https://mungingdata.com/) and can relate to your sentiment that any inconvenience can hold up the writing process. Your post is motivating me to streamline my publishing process.
https://gohugo.io/hosting-and-deployment/
If you have no idea, Netlify is good and with a good free tier.
We've gotten it to the point where we just push new pages via git and it all works. Anything more and you'll find yourself procrastinating ever so little at a time...
And it seems some themes are using this feature, therefore one can not say that Hugo is a single-binary static site generator anymore.
I use a bunch of old Unix tools that will be around forever to make it look nice: http://www.oilshell.org/site.html
The toolchain has changed significantly in 4+ years. It started as literally a shell script invoking Gruber's original markdown.pl. Then I switched to CommonMark, etc.
But the core data hasn't "rotted" at all, which is good. Unix is data-centric, not code-centric.
Previous thread that mentions that Spolsky's blog (one of my favorites) "rotted" after 10+ years, even though it was built on his own CityDesk product (which was built on VB6 and Windows). He switched to WordPress. Not saying this is bad but just interesting. https://news.ycombinator.com/item?id=25675869
Wouldn't switching formats like that break peoples pages? That's one concern I have with using markdown for a site like this. For anything non-trivial, you either end up using HTML (in which case, what's the point) or some dielect-specific feature.
CommonMark is a Useful, High-Quality Project http://www.oilshell.org/blog/2018/02/14.html
At the bottom of this article I note one incompatibility I encountered with the reference CommonMark vs. markdown.pl. But I just fixed those and have been using it for 3 years, and it's been great.
----
The key point is to have a diversity of implementations, so you don't get locked into one, which may go away in 10-20 years. In 10-20 years, I'm very confident that there will be CommonMark renderers available.
(Even apart from the fact that the reference cmark implementation I use is extremely compact C with no dependencies. C will outlive Perl -- the Lindy effect at work! http://www.oilshell.org/blog/2021/01/philosophy-design.html#... )
blog: https://gourav.io/blog repo: https://github.com/GorvGoyl/Personal-Site-Gourav.io/tree/mas...
Build with next.js, mdx, tailwind and deployed on vercel.
For a full blog, I would use a static site generator, but not for something well-served by a single static page.
This is deeply problematic. We should have dozens, if not hundreds of contenders for "the next WordPress" that leverage Git as a foundational aspect of content management. Instead we have a bunch of Contentful clones (no disrespect to Contentful) with REST and/or GraphQL APIs.
It's bananas that if you search for Git-based CMSes you have NetlifyCMS, and…wait what? Is that all??? Forestry gets mentioned a lot because it's Git-based but that's also a proprietary SaaS. I just don't understand it. Is this a VC problem or a real blind spot for CMS entrepreneurs?
The things a content creator might be more interested in aren't really core git strengths: SEO, publishing schedules, editorial back-and-forth, social media, etc.
Hmm, not really core PostgreSQL strengths either. What am I missing?
There are some more. I know of Grav which I intend to use migrating from NetlifyCMS. It has flat-file storage and Git connectivity. I think it can be installed on my shared hosting provider by just unpacking the thing (I don't have Composer / shell access).
https://getgrav.org/features
If you want a new CMS, give it a shot. But nobody's made a better free version of Jenkins either. It's hard to do and completely unrewarding/unmonetizeable (as FOSS), which is probably why nobody has done it.
However, insisting it leverage a difficult source code version control system is artificially restricting. Start with an MVP of flat files for content and add a plugin system. Somebody will write Git support, but they'll also add other content backends. I'll bet you a million dollars that a custom SQL plugin that does version control will be preferred over Git by 99% of users. But they may wise up and use an S3 plugin instead. Or maybe all three. They'll have choices, and your project will become more useful.
Don't start your project with an artificial restriction and it will be better for it.
It's pretty easy to mirror your own git repos, so that's a nice option, and to have stuff hosted on GitHub, your own, and maybe BitBucket as well. But nothing is going to beat having your own backups--which I DO still have, from as early as 2000.
However, that’s pretty lightweight on decisions I had to actually make to publish, and all the alternatives seem to be more involved. I wouldn’t mind migrating off WordPress, but just on the theming side that has a decent chance of involving a non-free theme, making the idea of hosting it on a public repo somewhat of a non starter...
Now to figure out what I can write about.
Though I'm definitely going to use this for internal docs at work probably. Very nice.
Edit: I'm guessing it's this? https://squidfunk.github.io/mkdocs-material/getting-started/
Here's a site using it: https://docs.libretro.com/