> There are examples on the web of the sorts of things that need to be done, but all the ones I could find were in TypeScript. That’s a hurdle I didn’t feel like trying to overcome today.
I mean, we can all have opinions about TypeScript, but converting from TS to JS is far from a hurdle.
So: developer attempts to integrate into a global distributed system without a connection to the Internet and complains he can't get it working without extra steps? Obviously there's two ways of doing this: you put everything online and then there's no extra steps, which the author doesn't want to do, or, if you want to do this offline, then extra steps are needed. Back in the day we used "dev environments" with tightly guarded ACLs. These days you can have things like Docker Compose locally or K8S clusters.
It feels like the author has their own preconceived notions about how systems ought to be "isolated"[1], regardless of the use case, keeps fighting with their half-baked networking implementation and then denounces everything, ranging from HTTPS[2] to now ActivityPub as "broken".
It triggers me a little when folks complain about HTTPS. Nowadays you have a wealth of tools to trivialize TLS certificate tasks.
Public TLS cert? LetsEncrypt + certbot (or any of the other zillion great ACME clients/libraries out there).
Self-hosted PKI for private/test use? Smallstep CLI (https://smallstep.com/docs/step-cli/) is good. And honestly, the raw `openssl` commands to barf out a root selfsigned cert aren't that difficult and number as less than 5 shell invocations that you could have in your history or a bash function or something.
Self-hosted PKI with extra steps that you already did because you have a Kubernetes cluster for some other reason? cert-manager will take care of pretty much everything.
These days the "hardest" part is sneaking your CA chain into the appropriate browsers / services you're using/testing, but even that is a few moments of inconvenience at best.
Getting a CA into a Mastodon server is an adventure of its own. Setting up the CA isn't that hard even without smallstep (copy/paste three or four commands and you're good to start testing) but getting the CA working where it needs to work is a massive pain. Everything has a different certificate store, sometimes compiled into the binaries, and getting stuff talking to each other is a massive pain.
For browsers, importing certificates comes down to just searching "certificate" in the settings. It's everything else that's broken, in my experience.
I agree. For example home assistant OS has a custom cert store that is reset on boot and there’s no way to make persistent modifications without recurring to hacks such as this one https://github.com/Athozs/hass-additional-ca
Enraging on a tool that offers user login from a webpage. You should be able to add DNS records to your gateway and not have to deal with public CAs to access it by name with TLS.
Yep. As a datapoint: I set up my own Mastodon server behind a Cloudflare tunnel and completed every step except the cert, eventually opting to just tell Cloudflare to ignore the cert config.
The default solution doesn't work on a shared host where you don't have control over running your own services and opening your own ports, only serving static pages instead using a shared web server.
> Nowadays you have a wealth of tools to trivialize TLS certificate tasks.
Trivialize what? You have to learn the tools. When you're paid for it or actually believe that signing everything is useful, you're willing to spend the time.
In the original article it looks like the poster already went over their time budget by a lot without even getting to researching if these tools exist.
The lack of a standard way to specify your trusted CAs on a system or network is one of the biggest hassles in that area. It’s typically all bespoke configuration, and often not well documented.
The principle the author is getting at is valid though. It should not be that hard to run tests. There should be easy escape hatches in systems to stand them up and tear down quickly so you can check your code against a reasonable "dummy" without having to reimplement the other side in your tests, as much ad possible.
As you say, Docker would be great for that, as they say they use in your linked [1]. Did you read it?
I wouldn't read too much into these posts. These are days apart and read more like inner-monologue from setting up systems. My process for implementing someone would probably fare the same. You wade through the cruft, complain about some things, but it gets done in the end and it's always nice to solve the pain points later.
Fair enough – there's nothing wrong with expecting systems to be able to be ran in dev mode suitable for running locally or within test containers, and that would be my expectation as well.
As you say, the inner monologue writing style of "I have this server and this thing and tried this and also I don't like this" may have led me astray as it didn't feel particularly focused, so it wasn't super clear what the complaint actually was.
Yeahh, understandable, I had to read the linked articles twice to form coherency. More time than I would have invested normally, I only did so after reading your comment xD
Makes me wonder if inner-monologue type stuff is even useful to post to the internet (something ive wanted to post more off because it sucks less energy from writing motivation). Takes more effort to parse as a reader though...
It's tricky. In practice, easy escape hatches tend to become ways people deploy to production with the safeties off.
In the past, we would then throw up our hands and say "Well boo on them for making their system hackable; sounds like their problem." But when you factor in the damage that can be done on the internet with things like botnets, it becomes everyone's problem. So sometimes people refrain from adding the escape hatches because they intend to work in an ecosystem of "You must be at least this technically savvy to play."
> developer attempts to integrate into a global distributed system without a connection to the Internet and complains he can't get it working without extra steps?
No, he's not trying to integrate into a global distributed system without a connection to the internet. He's trying to test locally an app that will eventually integrate into a global distributed system.
It's a valid complaint. If I have a local Mastodon server, and a local static website, why can't I just make the two talk without going through the public internet, or adding a self-signed CA to a bunch of trust stores?
It seems to me like it wouldn't be hard to add a --dev flag to Mastodon that allowed you to forgo the TLS certs, and it would make testing much easier. He's not trying to
This seems to be a common response to people criticizing open source projects. Basically, any criticism can be dismissed by suggesting that the critic should just fix the problem instead of complaining about it. I don't see why someone shouldn't be able to point out technical flaws without spending the time and resources to fix it themselves.
> why can't I just make the two talk without going through the public internet, or adding a self-signed CA to a bunch of trust stores?
Sure you can. Getting a valid, trusted certificate from a public entity doesn't mean that your service has to be public. Just get one and assign private IPs to the domain.
I get the idea of not doing that if you want to be a purist about the isolated approach. But then you should be ready to implement something that others don't care about as much as you do.
> you put everything online and then there's no extra steps
“put everything online” here is hand-waving the hard part. Most people are used to working in development environments that are not set up to serve inbound traffic from the public internet. There are security implications. In most of my past jobs, it wouldn’t have even been an option without violating IT policies.
I think this only applies to web clients, but I don't think ActivityPub's C2S protocol was ever designed to be used by arbitrary web clients. I suppose you could work around it by setting up some kind of proxy for requesting Mastodon URLs; you may need to do that anyway if the other server has authorised fetch enabled.
There are a whole bunch of technical implementation details that I disagree with when it comes to Mastodon. I wish the many MissKey forks would stabilise, because it seems a lot saner (yet every time I consider setting up a server, there has been a new fork thst everyone is switching to).
There are ready made ActivityPub signature algorithms for all kinds of languages. If you're planning on implementing them yourself, you could try finding an existing implementation to get inspiration from in a language that suits you but I don't agree that there are only Typescript examples.
As for the HTTPS thing: last time I messed with ActivityPub, I solved that problem with a Let's Encrypt wildcard certificate that I copy between hosts, but there are ActivityPub servers that will let you run in debug mode and federate over HTTP.
I think the problem with implementing ActivityPub is that the protocol looks deceptively simple at first glance, and people seem to expect it to be somewhat like RSS. However, when you actually start implementing it, you realise how many edge cases the protocol needs to deal with (and doesn't deal with).
The signature is a relatively small hurdle (it's an RSA public key encoded in base64, almost every language I know has a library to do the hard parts) but it's one of many. ActivityPub isn't a protocol you just tack onto your code in an afternoon, especially if you don't like using external libraries, even if it looks like it's just a bunch of JSON.
"Not being a protocol you tack onto your code in an afternoon" suggests that there is a big hole in the ActivityPub ecosystem; it should absolutely be possible to do that. You can, for example, write a basic HTTPS application server in an afternoon in Python, because libraries to implement all the hard bits have already been packaged by someone else. And similar functionality exists in other languages.
A similar level of cross-platform library support for ActivityPub would greatly ease its adoption across the universe of Internet applications.
I remember about 10-15 years ago there used be packages aimed at supporting ActivityPub, or whatever it was called back them. One of them, for Ruby, was even maintained by Steve Klabnik of Rust fame.
Perhaps there's something like that now again, but last I checked, the protocol got quite a bit more complex in its requirements, so there was no ready-to-go option, everything required a lot of manual testing and adaptation.
There are a whole bunch of small projects that turn an RSS feed into ActivityPub posts. In theory, a well-placed JSON file (the webfinger part) could direct anyone trying to subscribe to the blog to a dedicated fediverse server.
The protocol has been wrapped a whole bunch of times in various languages, but I don't know what protocol this particular blog uses, so I can't comment on what would be appropriate.
> The signature is a relatively small hurdle (it's an RSA public key encoded in base64 ...
Heh. If you look at the problem like this, I believe you have already implemented it wrong. The signature has an algorithm that you can specify, but sadly Mastodon only accepts RSA 256 keys which then rippled into the wider ecosystem and everyone is just accepting those. Sigh. A valid HTTPSignature can be built using almost any key algo, the fact that servers only accept RSA is an endless frustration of mine.
That's true, but this post is quite specifically about getting Mastodon to follow the blog. Of course you could use more modern alternatives for other platforms, but like with other federated protocols, your freedom is often restricted by the most outdated common client implementation.
Very few ActivityPub servers are actually spec-compliant, but unless Mastodon loses its leading position, I don't think we'll see spec compliance become relevant soon.
I hope I don't run into this many issues when I do the same implementation. I've been wanting to add integration with Mastodon to my static blog for a while now, as my refuge from Twitter and platforms in general. I just never use my "normal" Mastodon account so it feels weird to think I'll use this blog any more than I do that one...
Some of these issues seem avoidable though? The author seems to be diving too far into the testing rabbit hole. For my workflows, I generally find holistic integration tests to be too time consuming and not worth it, for the level of fidelity I want (I'm not NASA/bug-free). Same with trying to avoid testing on prod. It might not be "clean" but for a site like this it seems like a reasonable tradeoff.
The integration I've seen work that's simple to implement is to have a button for "show comments" that loads a particular mastodon thread, and then if people want to comment they need a mastodon account and can comment on that thread.
Click on the "Load comments", you can see that it does request to Mastodon server, in this case, twit.social to fetch the comments. Because most Mastodon servers have CORS allowed to all sites, it's pretty easy to do.
It's not 100% clear if OP wanted two-communication, eg. display replies as comments. His title seems to imply one-way communication only. That is super simple to implement.
Simplest solution is to use something like https://mastofeed.org/ which automatically posts your RSS feed to Mastodon.
If you're looking for a very simple ActivityPub implementation that lets you post, follow, and be followed, I can point you at snac: https://codeberg.org/grunfink/snac2. It's 100% C, and not a lot of code at that. It should be easy to follow and debug, and you can double-check your implementation against it pretty easily with some choice breakpoints. And you can stand it up as an individual instance to have it talk to your code if you want to test interoperatability.
(I would not really recommend it for general use, unfortunately, since it's a pile of C that's not really all that secure. But as a publicity stunt we run @ish@ish.app inside of iSH itself, and snac turned out to be excellent for this because iSH is slow and doesn't implement all of Linux, so picking something simple and lightweight was a must.)
Browsers deal with this by considering localhost a "secure context", and anything that "requires https" actually requires a secure context. [1] You can debug new web features like the Audio Output API that require a secure context [2] with http://localhost urls, and use multiple ports if you need multiple hosts. ActivityPub could do this too.
You can test on a local network. Get a free certificate for a wildcard test domain (the author already owns a domain, so that's trivial), then use whatever.test.your-domain for all your testing. The post makes it sound way more complicated than it is.
He’s able to test on a local network. It is a certificates problem, not a network one.
The article’s OP is wrong here:
> So to test this, I’d need both ends of the communcation [sic] to be on the public internet with proper certificates.
That is false. If you setup Tailscale (or Headscale, if you want a self-hosted endpoint), all communications are made with CGNAT address space (you hit neither the public internet nor your private network), and still get HTTPS certificates (from Tailscale’s ts.net ACME-friendly subdomain).
Basically: use a VPN service which handles both TLS certificates and NAT traversal, and you can do your secure local tests all you want.
To a person complaining about increased complexity introduced by mandatory usage of HTTPS you respond with an option to introduce even more complexity in the form of internal VPN?! Can you tell me how you find that reasonable?
Have you ever used Tailscale? Just installing it on each device (and logging in with an account you already own) is the only thing you have to do to create a tunnel. You enable HTTPS certificates requires clicking on two links and enabling a switch.
Caddy (a webserver which, surely, the OP of this article must be using, if he’s serious about keeping things simple) comes with Tailscale support built-in, so you get your automatic HTTPS (ACME) with one line of configuration (adding `machinename.tailnet-name.ts.net` in the Caddyfile).
For fun, here’s the full configuration file for a server which serves a “what is my IP” service over HTTPS, only served when you are connected to your VPN:
(I should probably show how to remote proxy, to stay relevant with the article’s point, but I wanted to show off my Go-template-fu.)
Do you think installing software, clicking on a few links + a button (switch) on a website, and writing one line of configuration is unreasonable or complex?
At this point Mastodon is a certified obstacle in the face of wider ActivityPub adoption due to all the warts and quirks of their implementation that ripple into the wider ecosystem.
I wish they would dedicate a modicum of attention of being a better Fediverse citizen now that they have people they employ.
The author mentions difficulties with HTTPS and trying stuff locally.
I've had some success with mkcert [1,2] to easily create certificates trusted by browsers, I can suggest to look into this. You are your own root CA, I think it can work without an internet connection.
I empathize with the author and found the post to be a interesting and concrete example of what it's _actually like_ to try to publish a blog to Mastodon, which is something that I have thought about and read about in abstract. So, thank you for writing this up.
One thing to consider would be to try to use Caddy [0], or a tool like localias [1], as a local https proxy. You would be able to run both the mastodon server and your blog software on the same computer, addressable via local-only urls like "https://blog.test" and "https://mastodon.test" and have everything work. These tools manage the certificates for you transparently and you don't need to worry about anything being exposed publicly.
I'd be curious to know why the author didn't try this, they seem to be quite knowledgeable of other web technologies so I have to assume there's a problem that I'm not seeing here.
not to dismiss the issue, but if you can procure a certificate (e.g. host nginx+acme on `local.nwalsh.com` to obtain a wildcard for `*.local.nwalsh.com`), then just put `127.0.0.1 mastodon.local.nwalsh.com` in your hosts file and you should be good, right?
but yes, non-local runtime dependencies in software which you thought you set up to be local-first have a real habit of sneaking in. the wildcard cert solution only masks that non-local runtime dependency, visible by disconnecting the server from the network for 90 days at which point your cert expires and it'll fail again.
There are so many weird suggestions in the comments. I'm surprised nobody has mentioned ngrok https://ngrok.com/ (there are other competing alternatives as well). It makes exposing local service over HTTPS trivial. It's been used heavily in most of my engineering orgs.
Lots of people are suggesting self-signed certs or a local CA; why not wildcard certs? I have a homelab with some public stuff, some internal stuff, and for the internal stuff I just have a certbot post-renewal hook that scps the wildcard cert from my public reverse proxy to the services that need it. Yeah, not as easy as not needing certs, but once you have it set up it's not too bad
54 comments
[ 2.8 ms ] story [ 116 ms ] threadI mean, we can all have opinions about TypeScript, but converting from TS to JS is far from a hurdle.
It feels like the author has their own preconceived notions about how systems ought to be "isolated"[1], regardless of the use case, keeps fighting with their half-baked networking implementation and then denounces everything, ranging from HTTPS[2] to now ActivityPub as "broken".
[1] - https://so.nwalsh.com/2024/01/06-isolation
[2] - https://so.nwalsh.com/2023/12/31-https
It triggers me a little when folks complain about HTTPS. Nowadays you have a wealth of tools to trivialize TLS certificate tasks.
Public TLS cert? LetsEncrypt + certbot (or any of the other zillion great ACME clients/libraries out there).
Self-hosted PKI for private/test use? Smallstep CLI (https://smallstep.com/docs/step-cli/) is good. And honestly, the raw `openssl` commands to barf out a root selfsigned cert aren't that difficult and number as less than 5 shell invocations that you could have in your history or a bash function or something.
Self-hosted PKI with extra steps that you already did because you have a Kubernetes cluster for some other reason? cert-manager will take care of pretty much everything.
These days the "hardest" part is sneaking your CA chain into the appropriate browsers / services you're using/testing, but even that is a few moments of inconvenience at best.
For browsers, importing certificates comes down to just searching "certificate" in the settings. It's everything else that's broken, in my experience.
The default solution doesn't work on a shared host where you don't have control over running your own services and opening your own ports, only serving static pages instead using a shared web server.
Trivialize what? You have to learn the tools. When you're paid for it or actually believe that signing everything is useful, you're willing to spend the time.
In the original article it looks like the poster already went over their time budget by a lot without even getting to researching if these tools exist.
As you say, Docker would be great for that, as they say they use in your linked [1]. Did you read it?
I wouldn't read too much into these posts. These are days apart and read more like inner-monologue from setting up systems. My process for implementing someone would probably fare the same. You wade through the cruft, complain about some things, but it gets done in the end and it's always nice to solve the pain points later.
As you say, the inner monologue writing style of "I have this server and this thing and tried this and also I don't like this" may have led me astray as it didn't feel particularly focused, so it wasn't super clear what the complaint actually was.
Makes me wonder if inner-monologue type stuff is even useful to post to the internet (something ive wanted to post more off because it sucks less energy from writing motivation). Takes more effort to parse as a reader though...
It does feel like this sometimes.
In the past, we would then throw up our hands and say "Well boo on them for making their system hackable; sounds like their problem." But when you factor in the damage that can be done on the internet with things like botnets, it becomes everyone's problem. So sometimes people refrain from adding the escape hatches because they intend to work in an ecosystem of "You must be at least this technically savvy to play."
No, he's not trying to integrate into a global distributed system without a connection to the internet. He's trying to test locally an app that will eventually integrate into a global distributed system.
It's a valid complaint. If I have a local Mastodon server, and a local static website, why can't I just make the two talk without going through the public internet, or adding a self-signed CA to a bunch of trust stores?
It seems to me like it wouldn't be hard to add a --dev flag to Mastodon that allowed you to forgo the TLS certs, and it would make testing much easier. He's not trying to
https://github.com/mastodon/mastodon/blob/main/CONTRIBUTING....
Sure you can. Getting a valid, trusted certificate from a public entity doesn't mean that your service has to be public. Just get one and assign private IPs to the domain.
I get the idea of not doing that if you want to be a purist about the isolated approach. But then you should be ready to implement something that others don't care about as much as you do.
“put everything online” here is hand-waving the hard part. Most people are used to working in development environments that are not set up to serve inbound traffic from the public internet. There are security implications. In most of my past jobs, it wouldn’t have even been an option without violating IT policies.
It almost feels as if “online” is a centralized system.
Another gripe with the technical implementation of mastodon is the CORS headers required to access the ActivityPub API [0].
Because of this issue, an activitypub-aware frontend for mastodon has to have its own mastodon server running, which adds a whole bunch of hurdles.
[0]:https://github.com/mastodon/mastodon/issues/10400
There are a whole bunch of technical implementation details that I disagree with when it comes to Mastodon. I wish the many MissKey forks would stabilise, because it seems a lot saner (yet every time I consider setting up a server, there has been a new fork thst everyone is switching to).
As for the HTTPS thing: last time I messed with ActivityPub, I solved that problem with a Let's Encrypt wildcard certificate that I copy between hosts, but there are ActivityPub servers that will let you run in debug mode and federate over HTTP.
I think the problem with implementing ActivityPub is that the protocol looks deceptively simple at first glance, and people seem to expect it to be somewhat like RSS. However, when you actually start implementing it, you realise how many edge cases the protocol needs to deal with (and doesn't deal with).
The signature is a relatively small hurdle (it's an RSA public key encoded in base64, almost every language I know has a library to do the hard parts) but it's one of many. ActivityPub isn't a protocol you just tack onto your code in an afternoon, especially if you don't like using external libraries, even if it looks like it's just a bunch of JSON.
A similar level of cross-platform library support for ActivityPub would greatly ease its adoption across the universe of Internet applications.
Perhaps there's something like that now again, but last I checked, the protocol got quite a bit more complex in its requirements, so there was no ready-to-go option, everything required a lot of manual testing and adaptation.
The protocol has been wrapped a whole bunch of times in various languages, but I don't know what protocol this particular blog uses, so I can't comment on what would be appropriate.
Heh. If you look at the problem like this, I believe you have already implemented it wrong. The signature has an algorithm that you can specify, but sadly Mastodon only accepts RSA 256 keys which then rippled into the wider ecosystem and everyone is just accepting those. Sigh. A valid HTTPSignature can be built using almost any key algo, the fact that servers only accept RSA is an endless frustration of mine.
Very few ActivityPub servers are actually spec-compliant, but unless Mastodon loses its leading position, I don't think we'll see spec compliance become relevant soon.
Some of these issues seem avoidable though? The author seems to be diving too far into the testing rabbit hole. For my workflows, I generally find holistic integration tests to be too time consuming and not worth it, for the level of fidelity I want (I'm not NASA/bug-free). Same with trying to avoid testing on prod. It might not be "clean" but for a site like this it seems like a reasonable tradeoff.
https://leo.fm/2022/02/isitover/
Click on the "Load comments", you can see that it does request to Mastodon server, in this case, twit.social to fetch the comments. Because most Mastodon servers have CORS allowed to all sites, it's pretty easy to do.
Simplest solution is to use something like https://mastofeed.org/ which automatically posts your RSS feed to Mastodon.
Of course you can also do it yourself. Posting to an existing mastodon account is just a single HTTP call with an API key: https://docs.joinmastodon.org/methods/statuses/#create
(I would not really recommend it for general use, unfortunately, since it's a pile of C that's not really all that secure. But as a publicity stunt we run @ish@ish.app inside of iSH itself, and snac turned out to be excellent for this because iSH is slow and doesn't implement all of Linux, so picking something simple and lightweight was a must.)
[1] https://developer.mozilla.org/en-US/docs/Web/Security/Secure...
[2] https://developer.mozilla.org/en-US/docs/Web/Security/Secure...
Broadly yes, except Safari which still sometimes doesn't: https://bugs.webkit.org/show_bug.cgi?id=171934
(Not relevant to this case, but eugh it's a very annoying exception)
Sounds like a feature, not a bug. What am I missing? You could generate self-signed certificates to make life a bit easier.
The article’s OP is wrong here:
> So to test this, I’d need both ends of the communcation [sic] to be on the public internet with proper certificates.
That is false. If you setup Tailscale (or Headscale, if you want a self-hosted endpoint), all communications are made with CGNAT address space (you hit neither the public internet nor your private network), and still get HTTPS certificates (from Tailscale’s ts.net ACME-friendly subdomain).
Basically: use a VPN service which handles both TLS certificates and NAT traversal, and you can do your secure local tests all you want.
Caddy (a webserver which, surely, the OP of this article must be using, if he’s serious about keeping things simple) comes with Tailscale support built-in, so you get your automatic HTTPS (ACME) with one line of configuration (adding `machinename.tailnet-name.ts.net` in the Caddyfile).
For fun, here’s the full configuration file for a server which serves a “what is my IP” service over HTTPS, only served when you are connected to your VPN:
(I should probably show how to remote proxy, to stay relevant with the article’s point, but I wanted to show off my Go-template-fu.)Do you think installing software, clicking on a few links + a button (switch) on a website, and writing one line of configuration is unreasonable or complex?
I wish they would dedicate a modicum of attention of being a better Fediverse citizen now that they have people they employ.
I've had some success with mkcert [1,2] to easily create certificates trusted by browsers, I can suggest to look into this. You are your own root CA, I think it can work without an internet connection.
[1] https://github.com/FiloSottile/mkcert/
[2] https://news.ycombinator.com/item?id=33383095
One thing to consider would be to try to use Caddy [0], or a tool like localias [1], as a local https proxy. You would be able to run both the mastodon server and your blog software on the same computer, addressable via local-only urls like "https://blog.test" and "https://mastodon.test" and have everything work. These tools manage the certificates for you transparently and you don't need to worry about anything being exposed publicly.
I'd be curious to know why the author didn't try this, they seem to be quite knowledgeable of other web technologies so I have to assume there's a problem that I'm not seeing here.
[0] https://caddyserver.com/
[1] https://github.com/peterldowns/localias
but yes, non-local runtime dependencies in software which you thought you set up to be local-first have a real habit of sneaking in. the wildcard cert solution only masks that non-local runtime dependency, visible by disconnecting the server from the network for 90 days at which point your cert expires and it'll fail again.