The payload can be picked up and managed via REST API as well.
I'm building a similar solution for a slightly different usecase, so wanted to have a look under the hood. There's nothing I can see that would stop you using this for iOS so maybe they just don't have a iOS dev on the project.
Thanks for clarifying. This could be a good solution for a non Google play type phone, or just someone who doesnt want t rely on Google servers. It would just mean in the latter case there would always be 2 open connections listening for messages...
So, their conclusion in the end that it is technically impossible to build a self-hosted notification solution on iOS due to Apple's restrictions. Shame because I would love to be able to deploy something like this internally at work, but of course, half the business is walking around with iPhones. I'm kind of curious though why it doesn't ruin every single XMPP client out there - there seem to be a few on the App store.
Either the server is sending pushes through APNS or the client is simply not getting messages reliably when the screen is locked or the app backgrounded, as the network restrictions very much apply to those clients as well (and to everything, to save power by sleeping the NICs).
Same. I wasted more time this evening setting up octobercms behind a reverse proxy (hours, not working) than getting that server handling notifications from a VPS running many small services.
I got so scared when I looked at the swagger. The authentication uses the exact same variables as mine and is somewhat similar. The name of the endpoints are also very similar and the project has resemblance to mine.
That was really really akward for a moment.
I'll use the project though ( different use-case), looks really interesting. Coincidentally it's for the same project as I mentioned.
It is worth noting that this will use a noticeable amount of power more than Google cloud messaging. The basic idea of how is works is about the same, but the devil is in the details. If you look at an idle connection over LTE, you will notice that after a while your carrier will close it. You need keepalives. If you look at the connection for Google cloud messaging, you might notice that it needs much fewer keep alives than YOU need to keep your connection alive. This means fewer wakes from sleep mode and lower power.
While it's likely true, GCM/FCM is handled by Google Play Services, which is whitelisted and has I believe system-level permissions. It has less restrictions than a regular app exempt from battery optimisations, as it's supposed to be the only app that needs to run in order to coordinate things like push messages or location updates across all apps on the device
edit: I missed that the original post was talking about keep-alives and you were probably referring to them. But how app-level optimisations can help with keep-alives?
I suspect the hard bit for normal apps is keeping the app active on the device while it is in the background. Google has their FCM receiver running as a low level service so can make sure it is always running.
Even if you have native IPv6, some carriers do connection tracking anyway to prevent random port scans from the internet waking up your phone or trying various exploits.
Not only that, but for many manufacturers this will straight up either not work after the app has been in the background for some (not long) time, or will deliver (that is, app will be able to connect and receive) push notifications with significant delay. Sadly, most reliable way I know to trigger push notifications when app is closed is using FCM - I'd love hear if there's an alternative, though.
In my experience it's not bad enough to worry about.
I have:
- Signal
- Whatsapp
- Etesync
- Conversations.im
- Syncthing
all whitelisted from battery optimisation, and I've not noticed any huge drain (I expected it to be unusable and have to go back to google play services or micro-g).
The documentation states that End2End tests are included. I didn't find them in the repository. I want to learn from the tests - Can somebody please help me and point me to the End2End tests?
Although I've been using Pushover[0], a paid service, instead of Gotify, I found a really cool project to pair a push-notification service with is huginn[1], which allows you to program bots to do things on cron jobs, among other really powerful actions.
Just to clarify, Pushover is not a "paid service" in that you need to pay for a subscription. You only have to pay once to use it past the 7-day free trial.
I'm not up to date with Android, but why does displaying notifications require an entire server or a paid service? Is it not just a matter of installing a client program and sending a curl to an endpoint?
It is not needed for displaying the notification but specifically targeting users. You would usually call an FCM function to get a token, which can be used to specifically target that user and is stored in the database. To send a notification to your customised target audience based on user-generated events, it becomes essential to use web-service that does it for you.
For example- Let's say you wish to send a notification to user A when user B sends him a message. In this scenario, you can call another FCM fn, using the token user A had generated. Of course you can use the web dashboard to send the notifications, but in this case, you would not be able to send event-based notification reliability and in real-time. Other services also work in a similar manner.
Of course FCM also has notion of channels where you can broadcast to certain topics (which apps listen to) without having to track tokens on application server side.
You need a way to tell the mobile device that a new message has been received. Since you don't have a fixed IP or anything there is no way to directly send the message.
An alternative is that the mobile device pulls from time to time, but that's quite power intensive or, when using a long interval, is slow.
Instead one has an idle connection open to which the server can send messages.
> I'm not up to date with Android, but why does displaying notifications require an entire server or a paid service? Is it not just a matter of installing a client program and sending a curl to an endpoint?
Imagine if every program on the users phone had their own schedule for checking for updates against remote hosts. That sounds terribly inefficient and wasteful of power, as the applications would need to "wake up" (as in, consume processor cycles) to initiate and perform a check.
Google's solution - Firebase Cloud Messaging is for the most part free and fairly easy to setup. Yes you can use curl to call their HTTP/REST endpoint to send a notification I believe.
If you get to the stage of sending millions of notifications per day then you'll need to start paying,
I get why that's "yuck" but if I'd hedge a bet...it's probably done because it's easier to make a GET than it is a POST or PUT request and the length of a text message will never exceed the query string uri length limits.
Not saying I disagree, but i don't know it changes a whole lot in this case.
In this case maybe, but generally GET should be avoided for any creative/transactional action because many systems will replay/cache GET requests, assuming that they’re idempotent.
Yeah the caching issue is a great reason not to use a GET, i was more just suggesting if you're being lazy and pasting a request into a browser v's using a command line (which obviously you should) then it's easy to get going quickly to kick the tires.
just to point out: there are no uri length limits. browsers typically have one (I haven't checked lately tho), but other tools generally don't (except by mistake / size-capped buffers).
the spec does not define a limit, and the widely varying implementation-specific limits are soft evidence of that.
even wget and curl are probably more limited by your CLI arg length limits than the bins themselves. e.g. I can craft a multi-megabyte GET request in a file and pass it to `curl -K` and it works just fine (I just did so to verify, a bit over 3MB. google complained a bit, but it responded). even if I screwed that up somehow / it silently truncated, I can absolutely do something with netcat and know it won't truncate.
How is a GET request easier? I assume you need to include authentication somehow, so it isn't like you are triggering it by typing the url in a browser.
If you really want to, you can in fact issue a POST with a query string and no body. It would admittedly be rather unusual.
Also, application/x-www-form-urlencoded bodies are literally identical to query strings anyway, so if you can construct a query string, you can easily construct a POST body using that Content-Type too.
I had exactly the same problem, and ended up using Signal. I like it more than normal push notifications because you can reply and have a full chatbot running.
Right but as the article shows, there are many "push technologies"... what is this project about? Someone subscribed for notifications and then this server sends them?
But yes, it's essentially implemented the same across iOS/Android and browsers like Chrome, FF, Edge etc. There are services such as Amazon SNS or Google Firebase that provide a single interface which can talk to all the slightly different implementations.
In a nutshell the device registers to the server and receives an identifier token. Then a server can send messages to that identifier and the message will be displayed on the screen. Some providers allow for Icon, Audio and some Actions to also be sent as part of the message payload.
I'm not sure, but I don't think iOS lets apps auto-start and run in the background forever. This would be needed for any alternative push notification app, else it would be too easy to miss notifications.
Awesome! I've been working on a collection of news source parsers [1] (a combination of RSS and HTML scrapers) that this will work perfect with to replace my native news app. All I have to do is pipe the lines from my parser through a Python script issuing these message POST requests and it just works out of the box.
FCM is integrated in kernel, so it cannot be killed. And it starts a persistent TCP connection to Google server. If all apps uses FCM, then all notification would route through this connection, and this significantly reduce battery consumption than the alternative, where each app uses its own web socket server.
So, assume using a degoogled android, can this notification replaced FCM if more apps use this notification server? This server uses web socket, so what does FCM use for its connection? Does web socket battery efficient enough?
It's a plain-old TCP connection in GMSCore, but has special permissions to ignore various power save modes.
I personally wish Google would work with mobile networks to replace it with something lower level (eg. Based on the same thing used to initiate phone calls), because a hanging TCP connection open for many hours gets unreliable as NAT drops it and as the phone migrates from one mobile network to another (eg. Across country boundaries)
It's handled by Play services which is whitelisted to allow network access when the device is in light/deep doze mode.
Play services can also instruct Google's servers to stop sending notifications when the device is in deep doze and re-open the connection when the device is active again -- thus saving power (messages are queued at Google's servers)
I really like the idea of Google providing a single, efficient connection for push-notifications, but having it bundled with Firebase (and therefore with Google Play Services) is an unfriendly approach.
XMPP is only one option for communicating with the FCM server. The most popular method is using HTTP/REST. XMPP is only needed when you have bidirectional messages - server to client app (downstream) and client app to server (upstream).
Why would you need a self-hosted (or really, any other push-notif service) other than the official one from Apple/Goggle?
They're optimized (for each respective platform), no need to maintain a server, and free. If you need "cross-platform", you can use Firebase(?) that abstracts that away.
Privacy? Keep in mind that not everyone intends to build a user-supporting platform on this tech. I like the concept because it's an easy way for me to push notifications from my homebrew IoT stuff and news streams without it flowing through ad companies.
You can encrypt notifications that go through Google's FCM and Apple's APNS push notification servers. This is how Pushover works - each notification is encrypted with a device-specific key, sent through the notification servers as an encrypted blob, then the app on the device is woken up and receives the blob, decrypts it, and posts it as a normal notification.
You could. But if you don't need that kind of added complexity and scale (and I don't for personal news updates or IoT device alerts) then why jump through those hoops?
I think the major reason this was created was for apps on the F-Droid store and for people who prefer not to have Google Play Services installed on their phone.
Consider that some countries do not have a reliable access to Google backend.
The problem remains when trying to use notifications with a PWA. AFAICT, there's no way to set your own notification service in Chrome/Firefox/Safari.
It's still a concern and this service will increase battery drain - this is why all the users of apps using this service will have to explicitly exclude the apps from battery optimizations.
I don't see Google Play allowing these kind of apps in massive numbers though - applications abusing background processing to poll for messages were one of the main reasons why Android got blamed for being worse than iOS in standby battery consumption. Warranted or not, Google/Android gets blamed for abuse of power saving at the end.
I would like to point out that there exists solutions tested in the field in the last 20 years to solve the same problem.
In particular I would suggest to get to know MQTT an open protocol with several (open and close) implementation that is know for its flexibility and scalability (if nothing has changed it is used to power the Messager Facebook platform, but don't quote me on that).
verneMQ is the server implementation I would suggest if you are interested in such topic.
MQTT is just a protocol, not a service. I don't think it was the developer's intention to provide an alternative to MQTT, but to offer a solution for push-notifications, which just happens to be built on top of WebSocket without MQTT. Also keep in mind that MQTT will usually get wrapped by WebSocket to avoid issues with firewalls or to easily get TLS via a proxy server like NGINX, or simply to add web browser support. VerneMQ is not the only server implementation for MQTT.
You just re-phrased everything I said in a more negative tone with zero value added.
I said myself that MQTT is a protocol and again I said that verneMQ is an implementation of the broker, the one that I suggest but not the only one.
And yes, it was a choice to leave aside the implementation detail of the use of websockets for MQTT because is a different level of abstraction.
Please let's try to add value to the community and not just add noise and entropy.
Before to comment I asked myself.
1) does this need to be said? And since people where finding this project interesting I decided that yes, the same people could be interested in knowing another more mature ecosystem that target a similar niche.
2) does this need to be said by me?
And again, yes, none mentioned it already, the post was up for a while and I do have production experience with MQTT.
My apologies on the VerneMQ comment. I misread the sentence, not realizing it being a specific suggestion by you.
Regarding the topic of MQTT in general, to me it sounded as if you were not adding information but criticizing the choice of WebSocket instead of MQTT in this project. It was a misunderstanding.
To add to the discussion, maybe you could help me a bit with an issue I'm having. I'm currently working on a service which is starting to make use of MQTT for microservices communication. Initially I was using a WebSocket server for this, then moved over to RabbitMQ a couple of years ago, and now I have added a Mosquitto server which is working, but has not yet replaced RabbitMQ. This service also has one public NodeJS websocket server for web browser clients to connect to it, this WebSocket server receives the messages via RabbitMQ and dispatches it to the clients. I also want to replace that WebSocket server by another Mosquitto instance, so that MQTT will basically become the main way for communication between servers and also with clients.
I chose Mosquitto because it have been using it at home without any problems for a while now and have a custom authentication plug-in for it which I wanted to reuse.
VerneMQ is a candidate, but Mosquitto is working just fine, so I haven't looked into it again. I like how lean Mosquitto is and how it's configured inside a container. Plus I can understand the source code and eventually modify it.
Would you advise me against using Mosquitto in a production system? Which benefit would I have by using VerneMQ?
Don't worry! We are all busy and sometimes is normal to read too quickly a message on internet.
My advice would be to have one single broker no matter what, so don't replace the websockets server with another instace of mosquitto, but just use the same old instance.
Then, the ice thing of working with a protocol is that you don't care of the implementation, hence just keep using mosquitto until it support the load and the, eventually swap verneMQ if necessary.
Be careful with your custom authentication logic, that if you want to migrate will require you to port it over.
Verne provide plugins for Auth so it is not an huge deal but you need to think a little bit ahead.
But again, if mosquitto is working fine just keep it!
Depends on the exact problem definition. This is a small, single-binary server, offering a built-in web UI & API, requiring very little configuration, working with a mobile app for Android phones. Took me 5 minutes to start getting some notifications from things around my house.
Sometimes you don't need a cargo van or a jumbo jet to get the job done when a skateboard will do.
This is what home assistant needs! The only really lacking thing about HA is the lack of native Android app, but the most important thing is notifications.
99 comments
[ 3.4 ms ] story [ 98.5 ms ] threadI only saw android support, anything for iOS in the works?
New messages in the queue are then notified (gotified) via the websocket. https://gotify.net/api-docs#/message/streamMessages
The payload can be picked up and managed via REST API as well.
I'm building a similar solution for a slightly different usecase, so wanted to have a look under the hood. There's nothing I can see that would stop you using this for iOS so maybe they just don't have a iOS dev on the project.
Edit: Ah, here's why. https://github.com/gotify/server/issues/87
Unfortunately messages are only plaintext.
It’s the go gopher... gophing about.
I got so scared when I looked at the swagger. The authentication uses the exact same variables as mine and is somewhat similar. The name of the endpoints are also very similar and the project has resemblance to mine.
That was really really akward for a moment.
I'll use the project though ( different use-case), looks really interesting. Coincidentally it's for the same project as I mentioned.
edit: I missed that the original post was talking about keep-alives and you were probably referring to them. But how app-level optimisations can help with keep-alives?
https://www.tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepaliv...
I suspect the hard bit for normal apps is keeping the app active on the device while it is in the background. Google has their FCM receiver running as a low level service so can make sure it is always running.
https://0xacab.org/leap/bitmask_android/issues/2226#note_921...
https://news.ycombinator.com/item?id=17946320
I have:
- Signal - Whatsapp - Etesync - Conversations.im - Syncthing
all whitelisted from battery optimisation, and I've not noticed any huge drain (I expected it to be unusable and have to go back to google play services or micro-g).
$ git clone https://github.com/gotify/server
$ cd server
$ env GOARCH=386 go build
[0] https://pushover.net/ [1] https://github.com/huginn/huginn
For example- Let's say you wish to send a notification to user A when user B sends him a message. In this scenario, you can call another FCM fn, using the token user A had generated. Of course you can use the web dashboard to send the notifications, but in this case, you would not be able to send event-based notification reliability and in real-time. Other services also work in a similar manner.
An alternative is that the mobile device pulls from time to time, but that's quite power intensive or, when using a long interval, is slow.
Instead one has an idle connection open to which the server can send messages.
Imagine if every program on the users phone had their own schedule for checking for updates against remote hosts. That sounds terribly inefficient and wasteful of power, as the applications would need to "wake up" (as in, consume processor cycles) to initiate and perform a check.
If you get to the stage of sending millions of notifications per day then you'll need to start paying,
I actually wound up just using an SMS gateway provider with a simple API you can just fire a get request to.
Not saying I disagree, but i don't know it changes a whole lot in this case.
Modern browsers are typically around 100-200k limit, command line tools such as curl and wget also has their limits.
IE was famously limiting GET requests to about 2k bytes.
even wget and curl are probably more limited by your CLI arg length limits than the bins themselves. e.g. I can craft a multi-megabyte GET request in a file and pass it to `curl -K` and it works just fine (I just did so to verify, a bit over 3MB. google complained a bit, but it responded). even if I screwed that up somehow / it silently truncated, I can absolutely do something with netcat and know it won't truncate.
Also, application/x-www-form-urlencoded bodies are literally identical to query strings anyway, so if you can construct a query string, you can easily construct a POST body using that Content-Type too.
I liked it so much that I even wrote a small Python library for it: https://gitlab.com/stavros/pysignald/
Exactly that. A device registers with the server and then there's a REST API or CLI binary that you can use to send it push notification messages.
But yes, it's essentially implemented the same across iOS/Android and browsers like Chrome, FF, Edge etc. There are services such as Amazon SNS or Google Firebase that provide a single interface which can talk to all the slightly different implementations.
In a nutshell the device registers to the server and receives an identifier token. Then a server can send messages to that identifier and the message will be displayed on the screen. Some providers allow for Icon, Audio and some Actions to also be sent as part of the message payload.
What is the point of building alternative push notification services that can’t even properly rise above the bar of technical feasibility?
[1] https://github.com/wybiral/stream-sources
So, assume using a degoogled android, can this notification replaced FCM if more apps use this notification server? This server uses web socket, so what does FCM use for its connection? Does web socket battery efficient enough?
What do you mean it's "integrated in kernel"? Are you saying google play services includes a kernel module?
It's a plain-old TCP connection in GMSCore, but has special permissions to ignore various power save modes.
I personally wish Google would work with mobile networks to replace it with something lower level (eg. Based on the same thing used to initiate phone calls), because a hanging TCP connection open for many hours gets unreliable as NAT drops it and as the phone migrates from one mobile network to another (eg. Across country boundaries)
Play services can also instruct Google's servers to stop sending notifications when the device is in deep doze and re-open the connection when the device is active again -- thus saving power (messages are queued at Google's servers)
https://firebase.google.com/docs/cloud-messaging/xmpp-server...
I really like the idea of Google providing a single, efficient connection for push-notifications, but having it bundled with Firebase (and therefore with Google Play Services) is an unfriendly approach.
Why would you need a self-hosted (or really, any other push-notif service) other than the official one from Apple/Goggle?
They're optimized (for each respective platform), no need to maintain a server, and free. If you need "cross-platform", you can use Firebase(?) that abstracts that away.
I don't see Google Play allowing these kind of apps in massive numbers though - applications abusing background processing to poll for messages were one of the main reasons why Android got blamed for being worse than iOS in standby battery consumption. Warranted or not, Google/Android gets blamed for abuse of power saving at the end.
In particular I would suggest to get to know MQTT an open protocol with several (open and close) implementation that is know for its flexibility and scalability (if nothing has changed it is used to power the Messager Facebook platform, but don't quote me on that).
verneMQ is the server implementation I would suggest if you are interested in such topic.
You just re-phrased everything I said in a more negative tone with zero value added.
I said myself that MQTT is a protocol and again I said that verneMQ is an implementation of the broker, the one that I suggest but not the only one.
And yes, it was a choice to leave aside the implementation detail of the use of websockets for MQTT because is a different level of abstraction.
Please let's try to add value to the community and not just add noise and entropy.
Before to comment I asked myself.
1) does this need to be said? And since people where finding this project interesting I decided that yes, the same people could be interested in knowing another more mature ecosystem that target a similar niche.
2) does this need to be said by me? And again, yes, none mentioned it already, the post was up for a while and I do have production experience with MQTT.
Please try to follow the same guidelines!
Regarding the topic of MQTT in general, to me it sounded as if you were not adding information but criticizing the choice of WebSocket instead of MQTT in this project. It was a misunderstanding.
To add to the discussion, maybe you could help me a bit with an issue I'm having. I'm currently working on a service which is starting to make use of MQTT for microservices communication. Initially I was using a WebSocket server for this, then moved over to RabbitMQ a couple of years ago, and now I have added a Mosquitto server which is working, but has not yet replaced RabbitMQ. This service also has one public NodeJS websocket server for web browser clients to connect to it, this WebSocket server receives the messages via RabbitMQ and dispatches it to the clients. I also want to replace that WebSocket server by another Mosquitto instance, so that MQTT will basically become the main way for communication between servers and also with clients.
I chose Mosquitto because it have been using it at home without any problems for a while now and have a custom authentication plug-in for it which I wanted to reuse.
VerneMQ is a candidate, but Mosquitto is working just fine, so I haven't looked into it again. I like how lean Mosquitto is and how it's configured inside a container. Plus I can understand the source code and eventually modify it.
Would you advise me against using Mosquitto in a production system? Which benefit would I have by using VerneMQ?
My apologies, again.
My advice would be to have one single broker no matter what, so don't replace the websockets server with another instace of mosquitto, but just use the same old instance.
Then, the ice thing of working with a protocol is that you don't care of the implementation, hence just keep using mosquitto until it support the load and the, eventually swap verneMQ if necessary.
Be careful with your custom authentication logic, that if you want to migrate will require you to port it over.
Verne provide plugins for Auth so it is not an huge deal but you need to think a little bit ahead.
But again, if mosquitto is working fine just keep it!
Cheers!
Sometimes you don't need a cargo van or a jumbo jet to get the job done when a skateboard will do.