Ask HN: What is best way to do hands-on practice for system design?
I have build many small size web apps.
I lack experience in designing medium to large application. I tried to read system design content but not able to gain confidence.
How can I do the hands-on practice of system design concepts. Please your tips, resources, plan etc.
97 comments
[ 2.4 ms ] story [ 182 ms ] threadIt's essentially an apprenticeship: you work in teams and as you gain experience, you contribute progressively more and bigger ideas. One day, you get a job where the design is your responsibility.
There's also a list here with more resource links: https://github.com/madd86/awesome-system-design
Not directly related to any specific type of system, of course, but very good at getting you to build a system, maintain it, and fix any issues with it.
Factorio is an obvious example.
Second, I think the best way to learn is to just work on a lot of systems. Sometimes your ideas will work out. Other times, you will wish you had become anything other than a software developer. Both states are equally helpful. It’s nice to write good software and have things work out. But you’ll learn a lot more from the projects you can only fix with ‘rm -rf’.
Third, try to get as wide a range of experience as possible. Be the lead designer on one project. And be extremely junior on another project. You’ll learn as much from implementing another person’s designs as you will learn from implementing you own.
And finally, have fun and be cool to everyone you work with. You will meet a lot of beautiful people and a lot of assholes. But if you love writing software, you’re part of the tribe. Make it positive and be cool to others. In twenty years, you won’t remember half of what you know now. But you will always remember developers who were kind to you when you were learning. Be one of those memorable developers…:)
Good luck. It’s a hard path but I know you can do it. You’ve proven it by starting off here.
[0] https://nealford.com/katas/#:~:text=Architectural%20Katas%20....
1. That has a functioning app
AND
2. Business is growing rapidly bringing more customers than the system can handle.
In other words you learn on the job by getting your hands burnt. I got lucky to have joined such a startup. Learnt a lot, from fixing DB queries, designing asynchronous order processor, using CDN etc. I worked on scaling up the full stack including stuff like connection pools.
So your best bet is to join such a startup. You will learn a lot by handling real user traffic. And also scaling isn’t homogeneous. For example, you make different trade offs scaling a search application Vs scaling a payment processor. So business use case and business domain does matter.
Join a start up with a legacy code base, e.g. dodgy old PHP that they’re pulling out into modern domain services.
Within a year you’ll be able to analyse performance and scalability issues and refactor them in your sleep.
But working at a company like this is not a good way to build your wealth. You're far better off working at a big tech company that has awesome benefits or a fresh startup. Startups that have been around long enough to have a legacy code base aren't a place to become wealthy.
There are plenty of valid reasons to not want to work at big tech co’s, but if you’re discounting it for expected stress level alone I’d do a bit more research as I don’t think it’s as true as you think. Obviously read reviews of Glassdoor/Blind to get a sense of the company you’re applying to…
Common software engineering mistake: focusing on solving the wrong problem.
OP wants to practice hand-on system design. OP never writes about "building wealth" and doesn't even mention money anywhere. So where do you get that this is about building wealth?
From web & app, frontend, to backend, databases and server management.
Yet I still find it hard to grasp good system design.
At best I prefer things to be as simple and clear as possible, that's about all I'm certain off.
Most systems are already so established if they get going, you are mostly patching. Once you start patching all effeciency goes out the window, especially if those who build v1 are gone. Hard, and most often a bad idea, to overhaul a system to something more stable & efficient.
Now that I think of it, probably my biggest learnings are what not to do, not what to do.
That's how it works in new fields where there's no good theorical foundations yet. I think OP is asking if we already know how to avoid getting hands burnt in the first place, as in many other fields of engineering where you don't just learn on the job and get yourself burnt a few times.
I believe not long ago, medicine was like that. Clearly, things can, and should, become more standardized and that will not in any way encumber the creativity that is also a big part of our field.
1. Almost all distributed systems has a global load balancer sitting in front, which will distribute your traffic to nearest possible server.
2. Decide how do you want to make your DB replication to happen. Is it active/active, active/passive, P2P, etc ?
3. What do you choose between Consistency and Availability when Network Partition happen? Some databases like Cassandra are eventual consistent.
4. Any asynchronous tasks will need to use queuing service.
5. Any performant API will need to use Caching and CDN.
One: How do you get better at systems design? Build stuff -- lots of stuff. If you're interested in designing particular kinds of systems (say, planet-scale web services...), then get a job at a company that does that kind of thing.
Two: How do you get better at systems design interviews? That's much easier; you can just throw money at the problem. Use one of those paid mock-interviewing platforms where you can hire FAANG interviewers to anonymously interview you and give you feedback.
I just went through this interview prep and got the offers I wanted, so I can offer you a data point. It took me < 20 mock interviews (total cost of < $10K) to go from "I have no idea what I'm doing" to "I'm not worried about this part of the on-site interview loop." Admittedly, I have a decent bit of experience architecting systems at scale, so I was mostly using this approach to refresh my memory (heading back to a Big Tech job after several years of retirement) and get familiar with the interview format.
There are some common resources recommended for this sort of thing if you want to also make sure you've read the right content. The most helpful resources for me were the DDIA book [1] and the "Systems Design Interview" YouTube channel [2]. Both are great for breadth and getting exposure to a ton of different concepts -- after that, you can follow your nose (or the book's bibliography) to get depth on whatever you're most interested in.
Good luck with it! It's a deep and really fun rabbit hole. I suggest you find a particular type of system that you're really interested in; start pulling on that thread, and you'll inevitably find your way to all kinds of other fascinating systems-design topics as you go.
[1]: https://www.amazon.com/gp/product/1449373321/
[2]: https://www.youtube.com/c/SystemDesignInterview
That's a lot of money!
I'd say for us non-wealthy mortals, the best approach is to find another engineer who will conduct mocks for you. Short of that, maybe do one or two paid mocks to calibrate yourself, then practice on your own.
The sad fact is the "system design interview" is not really like actual system design at all. It's more about following a specific format that everyone expects and knowing the ideal solution up front.
As you said ,you have a lot of small apps, mabey you could do something to link them together, and build a middle layer service for all the apps. In this progress, you should consider all the part of designing a distrubuted system, it could help you a lot, I guess.
Perhaps it's not the best way to improve the ablity for system design, it works a lot on me at least.
The great thing about smaller projects is that you can take risks with them.
Build something small that uses a message queue. Then do a project that integrates with some web APIs. Then try building something that implements GraphQL. Then try integrating full text search.
When you get the chance to work on something bigger I guarantee you'll find this experience incredibly valuable. Larger systems tend to look a lot like smaller systems, but messier and more complicated because more people have been involved in helping grow them as their requirements have changed over time.
I'm going to be exploring how to go about it, how to massively over-engineer things for the sake of learning, as well as build minimalistic aspects of the games.
There will be `Main Quests` where we build a simple game, and `Side Quests` where we go off the wall and explore new tech.
I just started tonight! So it's definitely a WIP
https://mikercampbell.gitbook.io/gotta-code-them-all/
I'm currently working on the Caching (both in memory and with Redis as an option), how to generate Pokemon with random or specific values (like NPC teams need to have hard coded moves and levels, but you want random encounters to have random data) as well as the building the Overworld, with basic keyboard controls, collision detection, and beginning to have an ECS from scratch.
I'd love it if you would alpha test my next chapter? And please provide any feedback. Let me know where I'm too verbose, or if the code isn't clear or doesn't work for you!
I added an email to my profile where you can contact me
[0] https://locust.io/
[1] https://github.com/asatarin/testing-distributed-systems
Take a fast moving input data stream, and map it to some slower moving output streams.
Use a message queue, a web tier, a worker, and implement private and public caching internally in something like Redis. Implement HTTP caching.
Implement backups, failover, and recovery. Create runbooks and checklists.
Secure everything with MFA auth for admin-type users and design and implement lesser privileged user access.
Design and implement RBAC.
Add reporting, logging, fault-tolerance.
Make it platform agnostic and support multi-tenancy, i18n, and WCAG.
Support and test for HIPAA and GDPR compliance.
Document the whole thing.
Figure out how to write automated tests for all of these aspects.
1. Not clarifying the problem. The very first thing out of your mouth should be a series of questions that helps you define the requirements more clearly. Identify use cases. Try to understand the scale; questions that start with "how many" are good ones.
2. Not being concrete. For example, I had a design problem I'd give people that usually wound up with them sending lots and lots of messages in real-time from one system to another. But when pressed, very few people were able to describe the contents of those messages in any detail.
Just thought I'd pass that along in case it's helpful to you. And I'll second what a lot of other folks here are saying: the best way to learn this is to get experience.
3. Not being specific about any technologies. Instead of using a "SQL Database" and a "cache", just say "PostgreSQL" and "Redis". It's an interview, if you're not specific, I will start thinking you don't have any experience with Redis or Postgres. If you really don't have any, I will find out anyway.
- when you pick specific technology, you also have to defend why this one specifically. For ex, why redis and not memcache or why only postgres and not redshift..
If you keep it at sql and no-sql, you can easily defend the answer and control the interview
Like if the design depends on modifying two keys atomically, a transaction is required. Some KV stores have that ability, some don’t. Do you need the keys to be in the same shard now? What kind of persistence is required here?
Redis and Memcache are quite different where the rubber meets the road. Gotta know what you’re talking about, don’t handwave over the details.
Strong no.
From the outside, it all feels random: unless you know the rules the interview is playing by, you can easily make a mistake. Each place has a different interview script and priorities.
(This is the same advice I give people when I'm coaching them for FAA oral exams, incidentally. People really have a hard time listening to a question and answering the actual question.)
Can you explain why this matters? It’s an architecture interview right? Not an API/Worker/Queue design exercise.
It’s not like I don’t expect someone to be eventually able to figure out the contents of the payload, I’d just not expect it to be at the forefront of their mind.
That being said, there are a few well known examples of systems that are somewhat complex right from the start. Ecommerce comes to mind: build a configurable ecommerce framework that supports multiple stores, multiple vendors, multiple channels, CC and cash payments, multiple currencies, international delivery etc.
I think I’m inclined to incrementally solve potential problems with the system (as introduced by the interviewer), where a lot of these interviews seem to want you to anticipate all issues.
So asking lots of questions about what they want is probably a first.
For reference we do paid 45 mins mock interviews. In case you are interested, dm me and I can share more details.
* i18n/l10n: Display labels in different language and correctly format the counter value for your user (1'000 vs 1.000). A user should be able to switch country
* Allow users to register and create their own counters
* Store the counter value in database hosted somewhere
* Store content stuff (title, labels, etc.) in a headless CMS and integrate with it.
* Set up build pipelines for your project
* ...and more :)
* Make this app highly available across geographies under heavy use.
If you want hands on experience, pick an open source app in the language you know and deploy it somewhere and load test till it breaks and see which part breaks first. It could be your load balancer can’t handle that many connections or you app server rubs out of memory or the db comes to a crawl. The more real world you can make load test queries the better.
At the very least you’ll get hands on experience.
Are you a small business owner that preparing for going viral one day? Keep improving your system bit by bit. Maybe add a caching layer today, load balancer tomorrow, a job queue somewhere down the road and scale up your database now and then. And probably YAGNI imo.
If for job hunting, other comments had better suggestion