HN Explorer

Ask HN: What was your worst microservices nightmare?

4 points by mbrodersen ↗ HN
A friend of mine has taken over a legacy (20+ years) 100+ microservices application. And it is an absolute nightmare. The complexity of building and deploying it is the worst I have ever seen. 10x worse than then worst monolith I have ever worked with. Debugging is a nightmare. Synchronisation issues makes auto testing almost impossible. The only good thing is that all the microservices are written in the same programming language. Imagine if 5+ different languages and build systems were used instead? The complexity would have been out of this world.

What is your nightmare experience with microservices?

3 comments

[ 300 ms ] story [ 807 ms ] thread
yuppie_scum
Inherited about 12 Kube clusters with 200+ charts, which were 8 versions behind on kube version, running multiple ingress, metrics, secrets, routing and logging type schemes per cluster. since they were built really before Kube was functionally stable and there was a “you build it, you own it, you get laid off, nobody understands the app anymore” culture. No consistency in language, build ideology observability, or anything like that. Had to cut a new Kube platform release, spin up some separate clusters with EKS vs crappy provisioner software, create standards and docs, and slowly and painstakingly work with disinterested devs to migrate to the new single standard. The modernization and standardization work has definitely paid dividends. No more random cluster failures, costs under control, we have actual observability and certainty around compliance. It was a huge pain in the ass but I am pretty proud of the effort.
mbrodersen
Wow. How long did it take to turn the ship around?
yuppie_scum
About 2 calendar years (Pandemic probably added a lot of person hours to the effort) - still not complete. This was kinda a rogue effort/moonshot by me but I think if we didn’t do it, management was gonna do something drastic like move everything to data centers or something.