It's easier to get contributors and responses when most people already have github accounts. When not many habe have gitlab, it adds some friction to contributing and a lot won't bother
i dunno. autotools is hairy, but it also solved some incredibly hairy problems over an enormous breadth for a very long time in a very reliable manner.
portability across all the different unices with their weird c compilers and differing libcs that actually worked out of the box every time was a gargantuan task.
The real use case is super simple clusters, which could be just 1 node, no etcd, no API (SSH managed), no iptables.
Simplenetes is not a living breathing thing which auto heals it self, it's a deterministic approach where you can see your cluster with all hosts in the git repo which represents the cluster.
Also, it's focused on making local development easy, so you can develop in a micro services architecture straight on your laptop with very quick iterations.
Despite whatever the k8s crowd misuses the term to be, 1 node is not a cluster. It might be a single node managed using cluster-capable technologies, but it's still not a cluster.
This is ~17k lines of shell scripting and claims to be "simple". Kuberentes is probably ~1m lines of Go. I'm not entirely convinced that this is less complex (simpler) than Kubernetes.
There are many factors in complexity, but a big one for me is what I have to hold in my head – global state. From any given point, what do I need to understand to know what's actually going on.
For Go, this is actually comparatively little. Between a fairly straightforward language, static binaries, and the focus on ease of distribution, there's not a whole lot outside of a codebase itself to think about.
However for a shell like Bash there's a relatively large amount to consider. You're very exposed to the underlying OS, to libraries and system packages, to various tools, to configurations users may have specified, to things being redefined underneath you. There's a lot of global state to consider for a Bash script.
I suspect it is simpler than Kubernetes, but I don't think it's a clear cut case.
To be fair Kubernetes itself probably has 17k lines of bash, and it is a good thing. It is the first go-to project that I use as a reference for bash scripting best practices.
edit: wc reports 47023 lines in *.sh[1], out of which 23836 are unique[2].
If we take out the argument of OS dependent setup, supporting only stateless services etc.
One of the key benefits of kubernetes are API server and custom resource definitions. If you don't like how kubernetes does stuff. You can change the default behavior and orchestrate the way you want.
I can't believe you are being serious - why are you comparing a shell script to a million-line piece of production software? Apples to oranges. This was written as a joke to make fun of a common gripe against Kubernetes - its complexity. Seems like the Kubernetes community has a bit of a thin skin.
If it is a joke, the README does a poor job of conveying that. It is a bit funny at first but then gets relatively serious and recommends the project for use with up to 20 containers, with the beta caveat.
Either way, setting aside the laudable amount of work put into this, talking about the issues of complexity and different forms it takes seems quite on topic. (I don't have a cow in this fight, not using K8s or anything of the sort.)
Thanks!!
We do use it in production, it is however fresh from the keyb so giving it some time before putting sensitive stuff in there.
About the state, it's perfectly doable, but requires some more Ops than k8s and isn't as flexible, naturally.
1. Anything that offers effective competition to Kubernetes is a Good Thing. I'm not saying this is because I haven't look at it closely enough but competition brings about improvement.
2. Why do people write "Simplenetes has a 100x less code than Kubernetes" instead of saying it has 1/100th the code?
I don't disagree with your conclusion. But your comment communicates no argument against Spring and how the trade-offs it makes make it a poor option for new projects.
Anyone with sufficient experience with JPQL and Hibernate will realize their limits and how poorly it works in the latency sensitive transactions we want now. Spring and Hibernate were designed to cache domain 'entities' and manage their dirty state client-side as an optimization. Any time you go want to go near the EntityManager you're hitting this impedance mismatch.
Asking to learn, but if you were to start a Java application in the following two scenarios:
1) Monolith with some CRUD, API, persistence to a DB + search.
2) Scalable backend with an API, persistence to a DB + some caching
Not in any particular order (choose what fits your team): Kotlin or Java and a lighter framework (e.g. Javalin is to Spring as Sinatra is to Rails, with JDBI), a newer big framework (e.g. Micronaut, maybe Play, others), Phoenix/Elixir, Rails/Ruby+Sorbet, Go with libraries.
Edit: I just did a google search for "Java Framework" and get a lot of useless top-N lists with old content retitled "in 2020" with hits like: Hibernate (not a framework), JSF (JavaServer Faces), GWT (Google Web Toolkit), Struts (The Later Version), DropWizard (seems to have stagnated and docs are all over the place. JDBI was extracted and has a life of its own).
Cmon, you can't be serious, do you really recommend Play, Micronaut, etc. over Spring ecosystem? Also if you want the benefits of GraalVM you can stay in the ecosystem today with Spring Native, moreover if you are ideologically against Spring then you should go with Quarkus because that's the next technology by maturity/popularity. How can you not go with Spring when REST services are today a solved problem? Why do you want to turn it into a problem by adopting something else that is not battle tested? In the company that I work for, Spring is ruling the microservices layer and we have a combination of Scala and Python for data pipelines that are not related to the behavior of the business logic, they only know one thing and that is data. Hibernate has not been a problem for us, I can say that I totally recommend it when you have multiple people working on a project, rather than polluting the codebase with raw SQL.
If you really want something exotic in Scala that is impossible to express in Java, Go, Ruby, Elixir etc., then take a look at the Typelevel (cats, cats-effect, http4s, doobie) ecosystem or ZIO. If purely functional programming is your cup of tea.
I'm not ideologically against Spring since I mentioned Micronaut being a modern take on it, without the outdated assumptions. I'm not familiar with Quarkus but someone else recommended it so may have a look at it. I'm currently at a Rails shop, but I see patterns evolving as teams get larger toward program construction patterns like Spring, persistence, repository, application model, etc. ActiveRecord as big as it is, is still much simpler than Spring/Hibernate with an EntityManager.
BTW, I did work at a startup for years building microservices with Spring which is where I ran into it's limitations. I basically worked around the EntityManager, JPQL, and query template caching where ever it was necessary, which was frequent. I'm not against query builders or data mappers, having worked on making one myself[0]. I'm just more performance/resource conscious than most.
Anything Java-backend-ish these days, my first 2 stops would be Quarkus [1] or Micronaut [2]. It would need special requirements that those couldn't match to move me to something else.
A new Java application? Java is getting pretty long in the tooth. You might be better off building on top of Go or Rust, especially for those specific scenarios (Go in particular was designed to build network services.)
When I did enterprise sales, we had another phrase for that: "No one ever got fired for buying IBM." (Coincidentally, I also worked for IBM, back in the day.)
While it's true that tech choices are signals that enterprises do use to determine quality, if you are relying on a particular choice of programming language to sell into major enterprises that's simply a mistake. (And enterprises have invested vast sums in technology written in c# or PHP, so clearly it's not all "lol" out there.)
Rust's and Go's capabilities of compiling into a single binary, without requiring a JVM that needs to be fed and cared for, make them a particularly compelling choice for enterprise deployments. Kubernetes itself is written in Go.
Linux itself is written in C, though everything runs on it. Who cares about the language that Kubernetes is written in (by the way, it used to be Java but it was changed politically to Go)? If you are writing Java/Scala you can use GraalVM to achieve the same thing that Go does. If your service is mainly doing data processing, then Go won't help you there because its abstractions are too low-level or even non-existent. A Java service/app won't require a JVM if you you create a self-contained application package. Applications can be deployed on fresh systems with no requirement for the JRE to be installed. The advantage is that you control the version of the JRE used by the application and the disadvantage is that application updates are the responsibility of developer and self-contained applications do not have built-in support for automatic updates. Exactly why for services I wouldn't recommend it, and neither Go or Rust. I would never trade the state-of-the-art garbage collectors that the JVM provides me with the ones of Go or the borrow checker of Rust (though I would take Rust anytime over Go). I can't help myself but only think that you comment is straight from the 90s when it comes to the JVM ecosystem.
The hate against spring on hacker news is so ridiculous. Spring boot is one of the easiest lowest boilerplate ways to build anything, period. Especially if you use a type dynamic language on top, like groovy or kotlin or whatever.
Some people are just masochistic or they just want to distract themselves by solving problems that have already been solved by Spring circa 10 years ago. Most of them don't even use the technology that they preach around here, it's borderline trolling.
If you're looking for a simple Kubernetes alternative, take a look at the HasiCorp stack. I use that in production and it's easy to set up and maintain.
The YouTube channel [0] has plenty of introduction videos, but I find the documentation [1] to be the best place to start. You should set up Consul (service mesh) first, followed by Nomad (orchestration), and Vault (secret management) if you need it. The only thing I initially struggled with was the systemd-resolved config on Ubuntu, to make the service discovery work. I plan to write an article about it, should I ever find the time for it :)
Btw. the cluster consists out of three servers on Hetzner [2] and runs our web analytics platform Pirsch [3].
I use k3s in production, and it is great. But as far as daily use, it is not any simpler than kuberenetes. Initial setup and clustering is a breeze with k3s. But after that, you are mostly just working with the same k3s api as a full kuberenetes setup.
k3s is kubernetes. Simpler to install and get a cluster running and much smaller, but still kubernetes, with an identical feature set and API. That is part of the point of kubernetes, that it is a standard and anyone can implement it, with Google's original implementation only being a reference, not canonical.
They don't mention that this was accomplished via k3s, but I happen to know it was because it was colleagues of mine that did this. The point of being able to do something like this is you can deploy a completely identical stack of admin/controller level software to systems running in data centers and systems running in airplanes. As long as they can run kubernetes, you're good to go, and with k3s, just about any piece of hardware can run kubernetes. This has huge implications for defense because it means you can develop and test software that isn't directly responsible for specialized hardware control without having to build emulators or lab equipment identical to what is installed in the actual weapons platforms. You just need to ensure they can both run kubernetes.
It doesn't have to be kubernetes, obviously, but you get the advantage of a fairly rich ecosystem that includes things like k3s, which doesn't even require bash or a shell at all to work.
Building a platform to deploy stateless services are the easy part. This might even work nicely at a smaller scale. It is gets super complicated when you want to mount volumes, re-use those volumes. This is like a nice weekend project, contrarian in nature, but it is just not a replacement for kubernetes. This is some bash script orchestration on top of podman.
Bulk of complexity in opensource distro of kubernetes comes from the fact that everything is interface driven and has to work in an generic way. For example, you don't really have to implement cri-o like thing if you are opinionated platform and only supports docker. Similarly, you don't have to build CSI, if you only support ceph or aws' storage. I love the contrarian thought, but matter of the fact is modern orchestration platforms that want to be a general purpose platform is going to be big.
It is a really nice exercise however build something like this when you want to learn about networking, containers, orchestration, storage etc. and how to tied everything together.
However, if you are just dealing with a three node cluster (Yes I know many of you are running tri-cycles with k8s stickers on it) then maybe it can be fine for many applications to not have a scheduler.
Simplenetes does however support multiple replicas of pods so if one node fails it can be OK.
If you want something simpler and use podman you can try `podc` [1] which compiles a pod yaml spec into a single executable shell script with a nice managagement CLI api to it.
Haha :) Well, the learning has been great I can confirm that. My script-fu is on a whole new level now, it was already quite up there after creating Space.sh [1]
Waste of time... Well... Let's not think too much about that. I'm at least a very happy user of this :)
17k lines of shell? Hah give me the infinity lines of Go that is kubernetes — maintaining shell scripts is a nightmare. At least Go is a proper language with an ecosystem and tests and established best practices, etc., etc.
The amount of vitriol in this comments section is astonishing. Can we not share fun projects without the professional Kubernetes experts thumbing their noses? At the very least, it does seem now like the complexity of K8s’s code base is a point of very thin skin for the community.
Seems like my click-baity title brought out some strong feelings :)
I do actually like Kubernetes. I recommend it to clients, I assess candidates who are to work with it, but I do really think it is a Beast. Because it is.
Kubernetes is like C++, extremely useful but it just keeps growing and nobody really knows all of it.
While Simplenetes is like... Lua, batteries not included, your boss will not approve, it doesnt't hit the buzzspot, but some people will secretely use it and be happy about it.
17k lines is a lot of shell... but... it bypasses all the complexity of heavyweight scripting languages, bash is a lightweight interpreter that is always there and just works and i bet it's fast.
this is a cool idea.
next up, somebody will modernize daemontools for the container/cluster era...
Why? This is not a completely novel concept, and a daemon is not a particularly complicated thing. You can use shell scripts to easily "daemonize" small programs. It's both fun and effective to do so.
actually no. there are a bunch of properties to properly daemonize in unix. things like special handling for stdin/stdout/stderr, reparenting, signal handling... look it up.
classic rookie mistake to just call a program that doesn't exit a daemon...
Running "in the background", handling of stdin/out/err, logging, detaching from tty, creating a new process group/session, closing fds, and much more is handled these days mostly by systemd. No need to put all this into your program at all.
I'll give you that signal handling still needs to be done in the target process, if the defaults don't satisfy you.
But you really don't need anything special anymore to make any old script into a daemon.
Without systemd you can use daemonize from shell, for example. I either run these kinds of service scripts via systemd in production, or via my special program that takes care of starting and monitoring, and restarting when the code of the script changes.
Why should I stuff daemonization machinery into every single script I might want to run as a daemon, especially when it's cumbersome in many languages where I want to do it, like bash or PHP, and in general less flexible than using some external tool?
maybe you're right. shrug. does your script change the working directory? do all systems have fancy core file management now? does it still cause issues with unmounting?
it is true that it's rare to see a daemonize flag these days...
This all depends on what the program does. If you have some app specific service that waits for postgresql NOTIFY messages, and does something in response, it doesn't matter that working dir is not /. You just stop the service and perform maintenance.
If it's something like sshd, it better not prevent any filesystems from umounting unless necessary.
158 comments
[ 2.7 ms ] story [ 221 ms ] thread> 17k lines of shell script
those two phrases don't belong together... to think about this, "17k lines" and "shell script" should not be used together either.
Also, you are a bit light on tests.. there only seems to be a single trivial one, which seems way too low for a codebase this size.
About the tests, can't disagree with you, they will come.
[1] https://github.com/space-sh/space
We just moved Space.sh from GitLab to GitHub (hurts in my heart, but yeah community is on GitHub).
Also we are lousy at promoting our stuff, maybe April 1st is not the best day to do this either...
I'm surprised when people say this. I thought communities were only around repos.
Wut?
Are you a Space.sh project member, or FSF functionary?
I assumed it was tongue-in-cheek?
portability across all the different unices with their weird c compilers and differing libcs that actually worked out of the box every time was a gargantuan task.
Yeah, this represent a few months of full time work.
How can you afford this?
The real use case is super simple clusters, which could be just 1 node, no etcd, no API (SSH managed), no iptables.
Simplenetes is not a living breathing thing which auto heals it self, it's a deterministic approach where you can see your cluster with all hosts in the git repo which represents the cluster.
Also, it's focused on making local development easy, so you can develop in a micro services architecture straight on your laptop with very quick iterations.
Despite whatever the k8s crowd misuses the term to be, 1 node is not a cluster. It might be a single node managed using cluster-capable technologies, but it's still not a cluster.
Main upside is that I can put Kubernetes on my resume.
There are many factors in complexity, but a big one for me is what I have to hold in my head – global state. From any given point, what do I need to understand to know what's actually going on.
For Go, this is actually comparatively little. Between a fairly straightforward language, static binaries, and the focus on ease of distribution, there's not a whole lot outside of a codebase itself to think about.
However for a shell like Bash there's a relatively large amount to consider. You're very exposed to the underlying OS, to libraries and system packages, to various tools, to configurations users may have specified, to things being redefined underneath you. There's a lot of global state to consider for a Bash script.
I suspect it is simpler than Kubernetes, but I don't think it's a clear cut case.
https://caseymuratori.com/blog_0031
I'm not sure you can with shell scripting because of all the global system state.
edit: wc reports 47023 lines in *.sh[1], out of which 23836 are unique[2].
[1]: find . -name '*.sh' | xargs cat | wc -l
[2]: find . -name '*.sh' | xargs cat | sort | uniq | wc -l
It can distinguish between actual lines of code, comments and empty lines for a lot of different languages.
It's the same as cloc, but ludicrously fast.
One of the key benefits of kubernetes are API server and custom resource definitions. If you don't like how kubernetes does stuff. You can change the default behavior and orchestrate the way you want.
Either way, setting aside the laudable amount of work put into this, talking about the issues of complexity and different forms it takes seems quite on topic. (I don't have a cow in this fight, not using K8s or anything of the sort.)
Quite an engineering feat, though, I am sure! Is this to have fun, to learn, to actually use in production?
2. Why do people write "Simplenetes has a 100x less code than Kubernetes" instead of saying it has 1/100th the code?
By contrast, Kubernetes has at least 1000x more developers working on it
It’s way too complex already.
For comparison, IIRC WebKit had like 500 contributors as of five or six years ago.
Perfect, complete and too complex.
For big and great applications, spring if the obvious choice. It makes incredible complex and difficult things easy.
But, unfortunately, makes simple and quick applications incredible complex and difficult.
As a short-list sampling, a Micronaut SQL guide[0] lists the following:
I myself chose JDBI for more recent projects using Java/Kotlin.[0] https://micronaut-projects.github.io/micronaut-sql/latest/gu...
what would you use these days?
Edit: I just did a google search for "Java Framework" and get a lot of useless top-N lists with old content retitled "in 2020" with hits like: Hibernate (not a framework), JSF (JavaServer Faces), GWT (Google Web Toolkit), Struts (The Later Version), DropWizard (seems to have stagnated and docs are all over the place. JDBI was extracted and has a life of its own).
If you really want something exotic in Scala that is impossible to express in Java, Go, Ruby, Elixir etc., then take a look at the Typelevel (cats, cats-effect, http4s, doobie) ecosystem or ZIO. If purely functional programming is your cup of tea.
BTW, I did work at a startup for years building microservices with Spring which is where I ran into it's limitations. I basically worked around the EntityManager, JPQL, and query template caching where ever it was necessary, which was frequent. I'm not against query builders or data mappers, having worked on making one myself[0]. I'm just more performance/resource conscious than most.
[0] https://github.com/karmakaze/safeql
[1] - https://quarkus.io/ [2] - https://micronaut.io/
While it's true that tech choices are signals that enterprises do use to determine quality, if you are relying on a particular choice of programming language to sell into major enterprises that's simply a mistake. (And enterprises have invested vast sums in technology written in c# or PHP, so clearly it's not all "lol" out there.)
Rust's and Go's capabilities of compiling into a single binary, without requiring a JVM that needs to be fed and cared for, make them a particularly compelling choice for enterprise deployments. Kubernetes itself is written in Go.
persistence: jooq + postgres
search: postgres full text search
cache: ehcache
Btw. the cluster consists out of three servers on Hetzner [2] and runs our web analytics platform Pirsch [3].
[0] https://www.youtube.com/channel/UC-AdvAxaagE9W2f0webyNUQ
[1] https://learn.hashicorp.com/consul (for Consul, the pages for Nomad and Vault are similar)
[2] https://www.hetzner.com/cloud
[3] https://pirsch.io/
Part of what something like k3s enables is this: https://www.acc.af.mil/News/Article-Display/Article/2557413/...
They don't mention that this was accomplished via k3s, but I happen to know it was because it was colleagues of mine that did this. The point of being able to do something like this is you can deploy a completely identical stack of admin/controller level software to systems running in data centers and systems running in airplanes. As long as they can run kubernetes, you're good to go, and with k3s, just about any piece of hardware can run kubernetes. This has huge implications for defense because it means you can develop and test software that isn't directly responsible for specialized hardware control without having to build emulators or lab equipment identical to what is installed in the actual weapons platforms. You just need to ensure they can both run kubernetes.
It doesn't have to be kubernetes, obviously, but you get the advantage of a fairly rich ecosystem that includes things like k3s, which doesn't even require bash or a shell at all to work.
Bulk of complexity in opensource distro of kubernetes comes from the fact that everything is interface driven and has to work in an generic way. For example, you don't really have to implement cri-o like thing if you are opinionated platform and only supports docker. Similarly, you don't have to build CSI, if you only support ceph or aws' storage. I love the contrarian thought, but matter of the fact is modern orchestration platforms that want to be a general purpose platform is going to be big.
It is a really nice exercise however build something like this when you want to learn about networking, containers, orchestration, storage etc. and how to tied everything together.
However, if you are just dealing with a three node cluster (Yes I know many of you are running tri-cycles with k8s stickers on it) then maybe it can be fine for many applications to not have a scheduler.
Simplenetes does however support multiple replicas of pods so if one node fails it can be OK.
Why do I need 17k lines shellscript to run my tri-cycles? I would rather recommend using docker swarm or just podman itself.
you are adding more complexity without to a very simple problem.
[1] https://github.com/simplenetes-io/podc
Waste of time... Well... Let's not think too much about that. I'm at least a very happy user of this :)
[1] https://github.com/space-sh/space
Now I’m trying to figure out if I just got too old to appreciate them, or if the good ones have all been used up.
Also namespaces are a killer feature of K8s imo
Seems like my click-baity title brought out some strong feelings :)
I do actually like Kubernetes. I recommend it to clients, I assess candidates who are to work with it, but I do really think it is a Beast. Because it is.
Kubernetes is like C++, extremely useful but it just keeps growing and nobody really knows all of it.
While Simplenetes is like... Lua, batteries not included, your boss will not approve, it doesnt't hit the buzzspot, but some people will secretely use it and be happy about it.
I was thinking why are people talking about virtio so much.
this is a cool idea.
next up, somebody will modernize daemontools for the container/cluster era...
classic rookie mistake to just call a program that doesn't exit a daemon...
proper daemons can stay up for years.
Running "in the background", handling of stdin/out/err, logging, detaching from tty, creating a new process group/session, closing fds, and much more is handled these days mostly by systemd. No need to put all this into your program at all.
I'll give you that signal handling still needs to be done in the target process, if the defaults don't satisfy you.
But you really don't need anything special anymore to make any old script into a daemon.
if someone else runs it on a system without systemd it won't work correctly?
Without systemd you can use daemonize from shell, for example. I either run these kinds of service scripts via systemd in production, or via my special program that takes care of starting and monitoring, and restarting when the code of the script changes.
Why should I stuff daemonization machinery into every single script I might want to run as a daemon, especially when it's cumbersome in many languages where I want to do it, like bash or PHP, and in general less flexible than using some external tool?
it is true that it's rare to see a daemonize flag these days...
If it's something like sshd, it better not prevent any filesystems from umounting unless necessary.