HN Explorer

Ask HN: Project ideas for learning distributed systems

2 points by mr-karan ↗ HN
As a software dev who's primary role is around Infra/DevOps @ work, what are some interesting projects that I can use to improve my programming chops in the Distributed Systems side of things. I'm looking to improve my skills around networking, consensus/fault tolerant systems and pretty much anything related. I'm comfortable with programming in Golang and want to use this as an opportunity to get better at it as well.

Thanks!

7 comments

[ 2.3 ms ] story [ 15.4 ms ] thread
PaulHoule
I always liked Hazelcast.
brutus1213
More context would be appreciated.
PaulHoule
Distributed systems need some way of keeping track of configuration and other control plane information.

Most of those systems such as Zookeeper, Kubernetes, and the IBM zSystem Parallel Sysplex Coupling Facility are more of a problem than a solution.

I know some people run Hazelcast as a cluster separate from the application cluster (think Redis) but you can embed it inside the application servers in which case it disappears as an element of infrastructure (one less problem!) and you benefit from having shared data structures that implement the Java collection interfaces. In a lot of cases you don't have to think much about how the data structures span multiple address spaces. Application servers can discover each other and self-organize the data structures required to coordinate them, whether you are running N instances of the same application server or if you are running M different services.

What is great is that it "just works" if N=1 and M=1 so you can do a lot of testing without having to spin up extra infrastructure servers (points of failure! headaches of configuration! more gigabytes of docker downloads!) to do control plane coordination.

If there isn't something like that for go there should be.

brutus1213
Thx .. pretty decent idea. You are right that that there isn't a low-effort, modern way to get reliable datastructures. My team studied hazelcast a few years back as part of evaluating the distributed system we built ourselves. It definitely has some nice features. Here is the issue we faced though .. we had a system with great performance but not partition tolerance. We joked that we should call it almost-right consensus (we used CRDTs in some nifty ways). This was a sad joke as there is no way we can publish this.
PaulHoule
It seems to me that if you are worried about partition tolerance you are never going to be happy. Perhaps you are ignorant and happy if you decide to accept it.

My take is that most people can live with occasional "split brain" problems in the control plane so long as the algorithms involved have a latch structure with a bias towards progressing. (Think of the arguments as to why Bubble Sort completes even though it has free loops.) You might get some work units stuck and performance will be less than optimal but the main fear is that the degraded state is so hard to detect that it goes on for a long time.

Data plane problems are worse because people really do expect to get the right answer. There are many things other than "distributed system BS" that lead to incorrectness, such as floating point numbers, but I like the restartability paradigm behind Spark.

brutus1213
There are some good videos on Paxos and Raft. You can also learn your way around Zookeeper. Dist systems is a huge topic .. several text books exist along with courses. Networking is also very broad but not really a dist systems specific topic. Do you want to learn SDN or how networking works at the low level or something else.