Ask HN: Is it worth getting into distributed systems after 12 years full stack?
I have done 12 years Full Stack engineering, including some DevOps. Recently I have been getting very interested in Distributed Systems, and not just at a consuming level but at the theory level. I would like to get into the gritty details of distributed systems, perhaps to the level of a Masters' Degree.
Is it worth getting into the the subject? It would take a major commitment in time and efforts. Would it help with better career opportunities?
68 comments
[ 2.6 ms ] story [ 115 ms ] threadDisclosure: MSc in Digital Currency holder.
For example a system that uses distributed algorithms for performance improvements, or a system that is distributed by design and necessity to retain the benefits of redundancy, could be considered both centralized and distributed.
This is correct. Distributed is not the same as decentralization.
To OP: I think it’s absolutely worth it. Any car is a distributed system, even smartphones (by themselves, not just as part of a network!) have so many independent devices with full-blown CPU cores in them that you can call them distributed systems, and as you probably know by now, it’s absolutely a non-trivial topic. So if it’s something that deeply interests you, there is a market.
Given your full stack background (by which I assume you mean web development?), you might first need to look for work on distributed systems that are strongly associated with the web, web services etc.
I was under the impression that full stack meant you can do what's needed to stand up a solution at literally every OSI layer.
For example, from down at the base of infrastructure, networking, hardware/cloud, servers / platforms, integration, data / database, custom and COTs applications and services to clients, UI, desktop/web/thin/fat etc. etc.
By that measurement someone who can do all that probably already understands and in a sense, works in distributed systems - you probably had to already to cover the breadth and depth of all that knowledge.
Not picking on your choice of words here ("full stack"), just mulling it over and thinking: 1) if you can do that you're qualified to work on distributed systems and 2) there's no shortage of work advancing distributed systems and that problem isn't going away any time soon.
I'd say distributed systems is challenging (I worked on a startup doing that with database federation and it was really interesting and challenging), fun and important.
I think maybe the bigger question is, are you interested and want to dedicate the effort needed to get into it, and will that make you happy? There's so many levels of working in distributed systems - from research, inventing / building the tech, to implementing existing solutions (e.g. Apache Spark, Kubernetes etc. etc.).
If that stuff is interesting I'd first decide if I wanted to be on more on the research or practical side, and then how much money you need to live, and then choose my entry point into the field.
Yeah, that's what makes this term a poor one IMO. It's not at all clear what it means. I've always taken it to mean "not merely Javascript."
Fullstack developer = someone who can do all parts of a project on their own if they wanted to.
Calling the top two layers of a very deep stack "full stack" is a bit strange.
Do you mean service oriented architectures? That's the practical 'distributed' systems many companies are using today. You don't need a degree- just go work at Amazon for a couple years.
Do you mean decentralized systems and algorithms? Paxos, Raft, Bitcoin and TOR? Then sure, you might consider a master's degree. Or just start reading papers online and messing around with code in your spare time.
For example, a system using a database with asynchronous replicas - especially if latency is not extremely low - using those replicas for read scaling, is an eventually consistent system (because of replication lag) and suffers from related issues.
Multi-master databases are even worse. You get the joys of network partitioning (netsplit or split-brain) and conflicting updates.
Sure, there are ways to work around this, but all the workarounds I know of have some rather ugly tradeoffs.
Basically, there are many opportunities to do something great, and not be bored. IMHO.
Do you recommend a full time degree in DS or actual projects?
> Java, Scala, Ruby, Python, Angular.
You should also learn C. (Isn't that a pretty important part of the "stack"? ;-))
One growth area is cloud gaming. And I think "virtual desktops" in general may finally gain some consumer traction and scale pretty soon.
Naturally this leads to an interest in Decentralized Applications as it represents an opportunity to re-build the Web.
Some resources to get started. Best of luck ;)
MIT 6.S974, Decentralized Applications, Fall 2018
http://nil.lcs.mit.edu/6.S974/papers.html
Web 3.0 Summit Playlist
https://www.youtube.com/playlist?list=PLxVihxZC42nHlc7J-UqeS...
So the utopia of interchangeable shared units of anything will only be viable if those units are completely indistinguishable. If there's any kind of distribution of quality, the incentives will cause the economy of the system to devolve into hoarding of the better units. One way to work around this is to mandate regular exchange, but that really gets you into the territory of communism. Not necessarily a bad thing in theory or in some vastly different future, but a big change.
I work on / with what I think is pretty cutting edge distributed software but outside of academia most of the work is, at the end of the day, still engineering.
Now is as good a time as ever to be a seasoned Full Stack engineer. Pursuing an MS and shifting specialties seems very expensive.
Despite all the support other commenters are giving, I believe you should be very hesitant about making such a career shift.
I suspect that, due to bias against Front End developers (and javascript :) ), hiring managers are have trouble justifying senior front end developer positions and so are resorting to overqualifying the position and unicorn-seeking.
There's also a bias in technical interview questions, since conventional algorithmic/puzzle questions hold less relevance on the front end, so qualifying experienced engineers without asking back-end appropriate questions poses difficulties. Which means that, despite a role being a Front End position, Full Stack candidates end up looking like a better fit, since they do better in the interviews.
Right. People think I’m a hater because I say “full stack engineer” is just title-inflation for webdev, but I’m just telling it how the entire rest of the world sees it.
Throughout most of the history of the web, the same people wrote the CGI scripts, and any SQL in them, as wrote the HTML; working this way is perfectly normal for a webdev for like the last 20 years. I guess we were all “full stack engineers” back then, just we didn’t have a grandiose title.
What downsides do you foresee in making the career shift? (Apart from a risk to salary)
Sure there is a quality trade off, being jack of all trades but this industry is very elastic. You can have a great frontend engineer with only 2 years experience and an ok one with 6 years experience. Time in field doesnt really gauge skill well. Its more about the challenges one takes on and the variety of problems they solve with their code.
I guess most of the 90s and early 00s web devs were full stack then, including setting up the servers ("dev ops" I guess).
As the previous poster mentioned, it's quite simple, if you can implement a feature vertically (and I'm thinking at least a 3-tier system should be considered), you could consider yourself to be a full stack developer. I've met plenty of developers in my career who refused to work outside their tier.
The fact that sales people and recruiters are making full stacks up to be some sort of mythical horses with a single horn on their head is a different story...
Yes, the MSc taught me a lot about Distributed Systems. Yes, it cost me all of the cash I had saved up (£15,000, no debt). Yes, I could have tried to go it alone... but had I done that there is a good chance I might not have learned enough to know that I do not know enough; how to explore the challenges of designing and working with Distributed Systems; how to understand and debate the tradeoffs of different systems; how to apply the theory to the pratice; how to read, analyze and even critize white papers; I could go on...
What open source projects do you suggest to get started with?
IMO the average CRUD web app doesn't have the scale to have distributed systems problems. Also there are managed service versions of everything these days even when one scales up.
May I ask what you mean? I didn't formally study distributed system, but it seems like much of my daily struggle as an engineer is trying to deal with folks who are trying to implement consistent solutions in an eventually consistent world. For me, distributed computing lies beneath every non-trivial programming challenge I'm working on. Even OPs front-end JS talking to the back-end is distributed.
I guess its an opportunity cost. What else would you be doing besides that. with 12 years of experience Id hope that you have some understanding about the subject already and arent completely in the dark. I guess with 12 years I would say it should take you no time to pick it up cause I would expect you to already be well into APIs and Oauth.
I wouldnt hurt your career opportunities. But I guess you could do something more established like get a data science degree or work with embedded systems to work with robots. Its really about what you want. But doing this is better than doing nothing.
Technically I think that a lot of people do it for a cool factor rather than for a need. You can add uncessary complexity without benefit. I dont find there to be too much deep complex ideas about it.
Basically you are performing separation of concerns across a network vs doing it by creating different class files, this allows you to offset query load to an entire set of servers which helps scaling. But your company also has to pay of another set of servers.
It depends on the demand.
How many senior web devs are there? How many senior web dev positions are there?
Guestimate: 278057 senior developers and 23050 open positions [1].
How many distributed systems people are there? How many DS positions are there?
Guestimate: 1355 distributed systems people [1] and 219 open positions [2].
In other words: try to find some data that give some intuition. Sure, the 'research' (if you want to call it that, lol) is really biased but it probably is better data than your own intuition if you have no clue about it.
------- addressing some other stuff you said -------
Why do a masters in it? Do you want to research it as well? Writing a thesis is the obligatory part of it. At the Vrije Universiteit Amsterdam there is a masters in it that is spearheaded by Andrew Tanenbaum. But when you look at the courses, you see that many courses aren't about the theoretical part of distributed systems (see [3]).
To get your feet wet I'd recommend to fully follow distributed systems from Maarten van Steen. Why? First of all, his book [5] is being recommended by other people (see [6]). He also has lecture videos on it (from 2012, but since it's from a theoretical perspective they are still relevant). Edit: he deleted his videos, but I found one source here [4]. Oh yea, and he offers it all for free (which is also why I'm comfortable posting the new URL of the DS videos).
These are my 2 cents. Goodluck!
My personal opinion, don't pursue a masters. Just scan study guides of universities and figure out a curriculum, work less as a web deveoper but keep doing it (i.e. 4 or even 3 days per week) and study it independently.
[1] I searched it via LinkedIn by typing "senior web developer" and "web developer" and "distributed systems"
[2] I typed in "distributed systems" as job title on indeed.com. LinkedIn gave nonsensical results since everyone's course Distributed Systems was counted and almost no one had a title in it.
[3] https://www.vu.nl/en/study-guide/2018-2019/master/m-r/parall...
[4] https://u.cs.biu.ac.il/~ariel/download/ds590/videos/
[5] https://www.distributed-systems.net/index.php/books/distribu...
[6] https://teachyourselfcs.com/
Read, understand, repeat.
But if you are self learner and want industry job with hands-on distributed systems, I would recommend exposing any back-end/dev-ops/networking experience in your résumé. Also obtain AWS and GCP certificates, they will require you to learn about both theoretical and practical aspects of distributed systems.
Seek companies working on large scale distributed software, you may start as full-stack or back-end engineer and then get closer to interesting projects.
Yes: I believe everyone would benefit from leaving their comfort zone and venture into more DS to expose themselves to some non-trivial work. Don't get me wrong, there are plenty of challenges on writing a working CRUD system from the bottom up but facing scalability, consistency and fault tolerance issues is a different word, and most software developers don't give a damn about performance. That's usually what happens at startups or small companies(I've been there several times). A small and focused team builds a working product, and suddenly they need to bring more experienced engineers to help then scale it once the cooking recipes from the internet aren't serving them well or because the leadership won't let the team engage these challenges. So my first advice goes here if your company has such problems try to expose yourself to them.
Practice and theory should walk side-by-side, you can read the entire library on distributed systems but still be mediocre when facing real problems because you've never encountered those problems in practice. Use the books and papers to give you a direction to where you should point and do your homework. Consistently paced challenges and exercises gives you volume(consistency) and helps you build experience. Build your glossary of concepts, nail your difficulties and once you understand the basics, go read and implement CRDTs in Rust by yourself :) - I'd also suggest you to try following a book on your own pace first before committing to any formal education. Try to find a mentor within your friends who can give you more advice's and peer review your work.
Now the sad part: Unfortunately when shifting expertise even within the same career, you give up on several years of your experience. You will look less appealing to recruiters and companies because you just turned yourself into a JR. Distributed systems engineer w/o previous experience, and those are pretty hard to find out there. If you can live with that(both checking our ego and financially) and be humble enough to accept your new position - I'd recommend you to embrace it.
I wrote a post on what a distributed system engineer is, at least from our perspective at Ably Realtime. See https://blog.ably.io/what-is-a-distributed-systems-engineer-... if you’re interested in what it’s like.
Matt, tech co-founder Ably, www.ably.io