We have come to very narrow definition of a 'system' when using terms like system design, system architect, etc., which here seems to mean a distributed computer system. That, as compared to a more open-ended definition found on, say, Wikipedia [1]. What about biological systems like a cell or a human body, a road system, a sound system, ...
Working in the automotive sector, I find it even worse. Sometimes “system” means an ECU (e.g. a radar sensor), sometimes the whole car, sometimes even includes infrastructure (cloud, other cars, etc). Sometimes it is just a software component within an ECU (e.g. the operating system).
If I could I would taboo the term. It conveys practically zero information at this point.
the Google NALSD guides are also worth reading for concrete-ising these sort of designs, too. there's some stuff in the SRE Workbook: https://sre.google/workbook/non-abstract-design/ and much more online.
This thing has 233k stars. I dare to say, without evidence, that the vast majority of devs does not know what e.g. the CAP theorem is——and they’ll never need it. The vast majority of “systems” are small-scale. Who’s this massive audience?
I would guess that it's probably aspirational to some degree. People hope to work with larger systems.
Personally I think it looks interesting and I work at a small web agency. We usually only work with a few parts of these kinds of systems, but occasionally we have clients like a large sports ticket vendor, where a few more of these patterns are helpful.
Job hoppers trying for staff+ roles at FAANGs or VC-backed mid-stage startups.
Also, to add to your comment about CAP. knowing CAP (consistency, availability, partition tolerance, choose two) can be useful when choosing which database to use for a production system, I.e. can you get away with a fancy distributed gossip database or do you need the robustness of a rdbms.
99.9% of the time a RDBMS is fine, but if you're vying for a senior staff role, you need to sound like you can talk about the 0.01% of the time where Riak or something like that makes sense as a primary data store. That's why this guide mentions it.
I fundamentally distrust these sort of guides. It was brought home reading something on the google SRE book - basically they talked about how google started with two data centres and a bunch of python scripts as automation.
Did they do "up-front" design? Of course. Did they get it horribly wrong at times ? of course.
but the large working system evolved from a small working system.
Thinking you can cunningly plan a future system to hang together like some giant zipper with pieces as yet to be built pulling together perfectly is ludicrous.
These interviews are for engineers interviewing at companies which have scale ; not for a 0-1 growth of a product like your google example.
When you're launching products at scale, you absolutely need to design large systems that are durable and can operate at said scale. And when I'm hiring, I need to filter for folks who are able to rationalize at that scale.
Well as you are the FAANG hiring manager I expect I am going to lose the argument but I wonder if my oranges are more like apples than you suspect.
I see there being two options here
apples: please take the existing set of distributed components that evolved in tandem at this company and design a new application on top of and using those components
oranges: please start from scratch and explain how you would build a set of components that will work as well as those in 1.
Apples is what we mostly look for - in any org we have a scale and ecosystem and we don't want to throw it all out. If it must evolve it evolves in tandem with rest of the ecosystem. So if we have some scalable data layer now that has chosen one side of CP/AP and then we come along and say actually throw that out We need full concistency all the time for this HR app, we are going to build our own different CP data layer globally, hold my beer, then ...
The oranges part is fine - it tells us if someone has actually understood the principles behind apples. But honestly it's a fake out of we think oranges can actually be done. And this sort of stuff gives the impression you can design the application solution by also designing the ecosystem at the same time. That's the bit Inwant to emphasise - you build the platform then design the application on the quirks of what you built
And anyone who has not used the apples components won't understand their quirks and can come unstuck when the well known but non obvious behaviour strikes.
I am being too vague here. I feel there is an interesting set of discussions to come out of it - Inwill reread the article
This is super helpful as a primer (especially the bits about knowing which kinds of infra go where and doing capacity planning BEFORE architecting anything) but is not a replacement for actually having done it somewhere, since there are so many tools and tricks that are difficult to convey in a single guide.
I say this because this guide is often used by engineers trying to job hop their way into staff+ roles without having the requisite experience to back that up.
Put another way, this guide will help you answer "design Netflix," (I.e. narrow the scope, capacity plan, diagram the ideal path, replace with infrastructure, discuss tradeoffs) but will NOT help you answer "tell me about a time where you designed a large-scale system and how you would've made it better given the chance"
I had a look at the "design a crawler" part, since that's something I have experience with. It makes some pretty sketchy assumptions and weird design choices.
First in the back-of-the-envelope calculation it overestimates the size of a document by about two orders of magnitude. It's closer to 7.5 KB than 500 KB on average, uncompressed. 500 KB is closer to the size of Homer's Odyssey, which I will state is much longer than most websites. I'm willing to give them the benefit of the doubt though, maybe I'm misunderstanding what these 500 KB is supposed to consist of.
I also don't think the crawl frontier design would work at the sort of scales they imagine. This is a design that will get you to maybe 10 million documents.
Apart from the profoundly questionable choice of using a nosql database for this, apart from the race condition in the crawling algorithm, Redis can only store 4 billion (2^32-1) items in a set, and let me tell you, crawl frontiers grow very quickly when you design them like this one has been designed. By the time you're indexing a billion documents, your crawl frontier will be of many orders of magnitude more.
The biggest mistake is to think too link-oriented. You probably want to retrieve links one domain at a time as this permits connection pooling (otherwise your biggest crawling bottleneck will be waiting for the remote end to acknowledge the connection's closure), helps with deduplication and means you don't need to re-fetch robots.txt for every single document you retrieve. It also solves many problems limiting how much you fetch from a single domain, since you can just set a hard time and link limit.
You don't have to re-fetch every document on that domain when you refresh it, just get the new ones and sample the existing ones. This also means you don't need a billion-entry priority queue, you can get away with a manageable queue of a few million instead.
I don't think it makes economic sense to crawl everything with headless browsers, given it's several orders of magnitude more expensive, and in most cases doesn't actually give you different results. (Many websites that seem like they'd not be crawlable with a dumb crawler, e.g. Medium, actually crawl just fine as long as you don't masquerade the U-A to look like Chrome)
Makes a lot more sense to sample, identify and specifically crawl the sites where a classic retrieval doesn't work with headless browser, and do classic crawling elsewhere.
I don't think "GitHub" should lead in the title here. It's a bit misleading because it makes one think this is going to be a blog series from GitHub or something like that and, in any event isn't in line with standard HN practice. Only thing I could see making sense is to put "(GitHub)" at the end of the title, but even that I'm not sure is necessary. @dang?
21 comments
[ 3.6 ms ] story [ 67.6 ms ] threadWe have come to very narrow definition of a 'system' when using terms like system design, system architect, etc., which here seems to mean a distributed computer system. That, as compared to a more open-ended definition found on, say, Wikipedia [1]. What about biological systems like a cell or a human body, a road system, a sound system, ...
[1] https://en.m.wikipedia.org/wiki/System
If I could I would taboo the term. It conveys practically zero information at this point.
No, it can also be a design between multiple services on the same machine. Or it can be a design between multiple subsystems of a single application.
https://primer.style/
Personally I think it looks interesting and I work at a small web agency. We usually only work with a few parts of these kinds of systems, but occasionally we have clients like a large sports ticket vendor, where a few more of these patterns are helpful.
Also, to add to your comment about CAP. knowing CAP (consistency, availability, partition tolerance, choose two) can be useful when choosing which database to use for a production system, I.e. can you get away with a fancy distributed gossip database or do you need the robustness of a rdbms.
99.9% of the time a RDBMS is fine, but if you're vying for a senior staff role, you need to sound like you can talk about the 0.01% of the time where Riak or something like that makes sense as a primary data store. That's why this guide mentions it.
Did they do "up-front" design? Of course. Did they get it horribly wrong at times ? of course.
but the large working system evolved from a small working system.
Thinking you can cunningly plan a future system to hang together like some giant zipper with pieces as yet to be built pulling together perfectly is ludicrous.
These interviews are for engineers interviewing at companies which have scale ; not for a 0-1 growth of a product like your google example.
When you're launching products at scale, you absolutely need to design large systems that are durable and can operate at said scale. And when I'm hiring, I need to filter for folks who are able to rationalize at that scale.
I see there being two options here
apples: please take the existing set of distributed components that evolved in tandem at this company and design a new application on top of and using those components
oranges: please start from scratch and explain how you would build a set of components that will work as well as those in 1.
Apples is what we mostly look for - in any org we have a scale and ecosystem and we don't want to throw it all out. If it must evolve it evolves in tandem with rest of the ecosystem. So if we have some scalable data layer now that has chosen one side of CP/AP and then we come along and say actually throw that out We need full concistency all the time for this HR app, we are going to build our own different CP data layer globally, hold my beer, then ...
The oranges part is fine - it tells us if someone has actually understood the principles behind apples. But honestly it's a fake out of we think oranges can actually be done. And this sort of stuff gives the impression you can design the application solution by also designing the ecosystem at the same time. That's the bit Inwant to emphasise - you build the platform then design the application on the quirks of what you built
And anyone who has not used the apples components won't understand their quirks and can come unstuck when the well known but non obvious behaviour strikes.
I am being too vague here. I feel there is an interesting set of discussions to come out of it - Inwill reread the article
I say this because this guide is often used by engineers trying to job hop their way into staff+ roles without having the requisite experience to back that up.
Put another way, this guide will help you answer "design Netflix," (I.e. narrow the scope, capacity plan, diagram the ideal path, replace with infrastructure, discuss tradeoffs) but will NOT help you answer "tell me about a time where you designed a large-scale system and how you would've made it better given the chance"
First in the back-of-the-envelope calculation it overestimates the size of a document by about two orders of magnitude. It's closer to 7.5 KB than 500 KB on average, uncompressed. 500 KB is closer to the size of Homer's Odyssey, which I will state is much longer than most websites. I'm willing to give them the benefit of the doubt though, maybe I'm misunderstanding what these 500 KB is supposed to consist of.
I also don't think the crawl frontier design would work at the sort of scales they imagine. This is a design that will get you to maybe 10 million documents.
Apart from the profoundly questionable choice of using a nosql database for this, apart from the race condition in the crawling algorithm, Redis can only store 4 billion (2^32-1) items in a set, and let me tell you, crawl frontiers grow very quickly when you design them like this one has been designed. By the time you're indexing a billion documents, your crawl frontier will be of many orders of magnitude more.
The biggest mistake is to think too link-oriented. You probably want to retrieve links one domain at a time as this permits connection pooling (otherwise your biggest crawling bottleneck will be waiting for the remote end to acknowledge the connection's closure), helps with deduplication and means you don't need to re-fetch robots.txt for every single document you retrieve. It also solves many problems limiting how much you fetch from a single domain, since you can just set a hard time and link limit.
You don't have to re-fetch every document on that domain when you refresh it, just get the new ones and sample the existing ones. This also means you don't need a billion-entry priority queue, you can get away with a manageable queue of a few million instead.
Having said that, link orientation may not matter if they have to run a headless browser rendering such performance optimizations moot.
Makes a lot more sense to sample, identify and specifically crawl the sites where a classic retrieval doesn't work with headless browser, and do classic crawling elsewhere.
There’s one more github repo called “awesome-scalability” which has the good blog posts by concept.