Ask HN: What are must-know concepts for back end development?
I’d like to learn more about backend development but most of what I find while searching are results like “Build a nodejs backend in 20 minutes” or “learn backend development in an hour.” I think part of the problem is I don’t know enough to ask the right questions.
134 comments
[ 3.1 ms ] story [ 195 ms ] threadFor you specifically, I'd recommend picking up a web development framework like Ruby on Rails. It will teach you every aspect of building websites: Interacting with databases, writing server endpoints, creating front end web pages, user authentication, deployment, and probably version control. I would consider all of these things to be the bread and butter of typical "back end" engineers (except for maybe the front end stuff.)
From there, you can broaden your knowledge in any direction that interests you. If you like building interactive applications, you can look into front end frameworks like React or Vue. If you want to focus more on back end, you can learn more about relational databases (Head First SQL is a great beginner resource.) Lots of directions you can go.
Once you have accepted a connection, you then have to decide how to handle multiple concurrent connections. Do you use threads? Sub-processes? select/poll?
You can then move on to reading/writing to shared state. Do you use memory? Files on disk? A SQL DB? If using threads/processes, is concurrent access to those resources done safely?
It's a long journey to do the above well. I highly recommend anything/everything by W. Richard Stevens https://en.wikipedia.org/wiki/W._Richard_Stevens
- Load balancers
- Web servers
- Caches (eg. Redis, memcached)
- Databases (relational, non-relational, document)
- Search datastores (eg. Elasticsearch, Solr)
- Log/event/message processors (eg. Kafka)
- Task queues/task processing libraries
- Periodic jobs (eg. cron)
If you dig into any of these there's a ton to learn, especially around looking into the underlying technologies used to build these higher-level systems.
There are also more conceptual things that are part of building/maintaining backend systems. These are a bit fuzzier, but I would say are also as important as the specific technologies used:
- Reliability
- Monitoring
- Observability
- Error/failure handling
- Migration strategies
- Data normalization/denormalization
- Horizontal vs. vertical scalability
This is by no means a complete list, but these terms are enough to get you in the right ballpark of ideas and start learning. I think highscalability.com is a great place to read about how other companies have built backend systems to solve specific problems. They have a massive list of quality articles written about various backend systems at scale.
Also what does observability mean is this context?
Something went wrong, and now your site is serving 500 server errors to everybody at the rate of 25,000 per minute. The ops team already tried "just reboot it" and it didn't help. How are you going to figure out what is going on and fix it?
It's (mostly) too late to add anything, so all you've got is the logs you already had, the metrics you already had, etc. That's the "observable" stuff in a system. There's an art to recording what it is you need to know, while at the same time recording so much that you can't find what you need in the mess.
(The "mostly" is that if you have a good enough setup, you might be able to bring up a new system and route some very small fraction of traffic to it to examine it more intensely in real-time with a debugger or something, though in my experience, on those occasions I've had the opportunity to try this, it's never been a problem that would manifest on a new system receiving a vanishing fraction of a percent of the scale of a production box. But maybe you'll get lucky.)
You certainly want to do everything you can to not be in that mess in the first place, but it won't be enough. You need a system sufficiently observable that you can find the problem and find some sort of solution.
[1] https://codeascraft.com/2011/02/15/measure-anything-measure-...
- logging (ex tools: Splunk, Sumologic, LogDNA)
- metrics (Prometheus, datadog, Grafana)
- tracing (lightstep, new relic, zipkin)
As mentioned above, observability is the data collected about a system.
As for reliability, monitoring, and error handling I've heard good things about the Google SRE book: https://landing.google.com/sre/books/
I haven't read it personally, but I've heard good things from others and looking over it briefly the advice there lines up with what I've experienced in practice.
Cron jobs are the definition of the anti pattern of treating servers like pets and not cattle. You have to worry about that one non redundant server running your cron jobs. There are others ways to skin the cat, but my favorite is Hashicorp’s Nomad. I like to call it “distributed cron”. Together with Consul for configuration it’s dead simple to schedule jobs across app servers - the jobs can be executables, Docker containers, shell scripts, anything.
I think it is worth noting for any of the systems above that there's a spectrum of possibilities around how much you automate/offload the management of them, as well as plenty of backend systems for managing those.
But I agree and concede, a user with zero back-end experience will just google "cron" which will take them to a crontab example, so they will likely be mislead into the anti-pattern, as you said.
What's old is new again. Much of clean distributed systems development is now built on, what is essentially, scheduled period operations. They're pretty much the least complicated ways to loosely couple domain logic in distributed systems that follow eventually-consistent semantics. They're also a good model for the functionality of many distributed scheduling systems like Kubernetes, AWS' ECS Scheduled Tasks, and more.
Kubernetes even goes so far to have Jobs (batch operations) and CronJobs (scheduler that creates Jobs).
there crontab is not an anti-pattern at all
Or if the server doesn’t go down and your cron job fails. Do you then implement retry logic in every job since cron can’t do retries automatically?
For example, if I had a traditionally deployed (i.e. not in K8S / a PaaS / similar) backend app that accepted file uploads, then passed those off to something else, I'd be streaming the uploads to a temporary holding directory on disk. I'd then have a CronJob that clears stale items from the temp dir. If the server fails, that's OK that the CronJob didn't run.
There are still plenty of use cases for traditional cron.
The extra complexity of any workflow manager would make it even more certain to "goes down in the middle of the night or have issues".
The server going down was never really the issue though honestly. The issue was usually a process taking more CPU/Memory than expected in that case Nomad could intelligently schedule jobs based on available resources across the fleet of app servers.
These days with AWS, I don’t use Nomad, I just use CloudWatch and for the processes that aren’t Lambda based, I use autoscaling groups with the appropriate metrics for scaling in and out.
That also means if a server goes wonky, I can just take it out of the group for troubleshooting later and another instance will automatically be launched.
Cron jobs are the definition of "if it works don't fix it" and YAGNI.
Unless the cron job is only doing some type of maintenance that only affects the local redundant server....
It sounds easy ("derp, back-end is server, front-end is client") but it's really quite a mind bender when you think about it. Take a given piece of website code in a given language. Where is it being executed? Some "front-end" techs (Node, Webpack, Uglify, etc.) will be executed invisibly on the server. Some "back-end" web techs (cookies, redirects, OPTIONS requests, TLS) enable some invisible client behavior. Some (PHP) are dual-purpose, executed in two stages, some (conditional comments) are hacks that exploit this duality. Some (OAuth2) are orchestrated dances between back-end and front-end. It's really not so simple when you think about it.
Modern backend code will usually connect to databases and call REST APIs. The databases themselves might be part of a cluster. Moreover, the backend code is rarely exposed directly to the public; there's usually a proper http server sitting in front of it, and there might be another load balancer or CDN between the user and your http server. Every connection has a client and a server, and sometimes the same component plays both roles. Remembering which role(s) each component is playing in any given context is very important for security, not to mention debugging.
What all use cases are of using web servers?
Basically these severs provide a separation of concerns and are built to be more scale-able than a naive application listening on a port for requests.
2. How to read request parameters.
3. How to get the stuff you're asked for from the database.
4. How to create and send a response.
Once you can do that you can create a backend.
5. How to not get swamped in a mess and complexity when you do all above faster than you are required to.
https://dataintensive.net
Edit: I decided to go ahead and buy it since it was pretty cheap on Amazon. The table of contents had a lot of the information from the other comments. Thanks for the recommendation!
It's probably too late for you to edit your comment, but the title is Designing Data-Intensive Applications (not Systems).
I would structure interview questions around APIs rather about distributed systems/transactions, circuit breakers, etc. - practical aspects of running multiple systems talking to each other through pipes.
there are thigns that are absolute basics:
- bash/linux scripting, understanding
- usage of basic linux commands, grep, awk, sed...
- the basics of posix (stdin/out/err pipes, stderr)
- Monitor your system: observe CPU/network
- Configure firewall, ssh services, cronjobs, set up a systemd job
And from here how to deploy anything with the stack/framework which is popular this year, kubernetes, docker or ansible or whatever infrastructure it is in the backend.
Very rarely someone works "just coding".
2. Many Web frameworks -- Django and Ruby on Rails come to mind -- treat the database as an integral part of an application. In practice, however, the database in most cases runs as a separate service, more often than not on a completely separate machine. It is useful to view a database as just another part of your distributed system, with a very clear responsibility (persistence of state), that is accessed via a special API (SQL).
[0] https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...
Topics include different joins, relationship types, full text search, indexing, JSONB, amongst many others!
Your comment reminds me of a PyCon 2017 talk by Raymond Hettinger where he details the improvements in Python dictionaries from 2.7 to 3.6 which effectively amounted to rediscovering and re-implementing what was standard in databases decades ago.
"Modern Python Dictionaries A confluence of a dozen great ideas" https://www.youtube.com/watch?v=npw4s1QTmPg&ab_channel=PyCon...
---
The failure modes; what they are and how they manifest: read contention, thundering herd, cascade, ...
Mitigation strategies and where to apply them: throttling, backpressure, load-shifting, graceful degradation - and of course how to signal the upper layers that these are taking place.
Decoupling reads and writes. (Search for material on CQRS and see where the rabbit hole takes you.)
Error handling and tracing. This is particularly devious, because by definition error path is the unhappy path. It will be more expensive when hit, and has to spend more time serialising data. See also read contention, thundering herd and mitigation strategies.
Learn and understand the differences between: telemetry, instrumentation and monitoring.
However, I don't think they're "must know concepts for back end development". Most backend development is small in scale: a few servers, maybe more, a database, maybe something running cron/equivalent jobs or a queue worker, maybe some caches. Not much besides that. While people learning to develop systems at that scale might incidentally learn some of the above, I don't think most of those concepts will be useful until people are designing systems at much greater scale. Informally, many of them are present in small systems: e.g. when your cache server is down but connections take 10sec to time out, that's a form of backpressure, technically, but not in a super formal or useful way to understand the concept.
In sum, I think those are important (essential, even) areas of knowledge for intermediate-experience back end developers, or developers looking to increase the scale of their infrastructure or projects (or work on very large-scale applications), but I don't think they're in the must-know, 101-level tier of back end knowledge. People can and do build successful, stable, easy-to-work-on back ends without knowledge of any of those things. In fact, this is the norm in our industry.
The lack of knowledge of these areas is not a significant handicap (to productivity, understanding, or code/output quality) until you get beyond small scale. Most software projects do not.
Fair enough, I admit I am looking at things from behind glasses tinted in a certain way.
But I do believe some kind of familiarity of the concepts is essential. Even at relatively small scale. Read contention in particular is really easy to hit the first time you have to deal with an increase in traffic (doesn't need to be external, could be a minor change that triples the number of cross-service calls). Simply because the first observation is that worker systems are running hot, the instinctive reaction is to add more workers.
The two core problem areas in backend development - which in this context can mean anything that requires non-trivial server side processing - are I/O latency and processing capacity. Regardless of scale. Being unaware of (or worse, ignoring) them is not tenable.
[0] http://highscalability.com/blog/2016/1/11/a-beginners-guide-...
Find out how to make your ORM log the SQL it's running, and try running that in your database directly. If you are having trouble, get the SQL working first, and then figure out how to make your ORM generate that SQL.
On the web, use View Source and/or Inspect Element to see what is really in the DOM.
Look at the Javascript you're serving. Is it what you think it should be? Maybe the problem is in your webpack config or caching or something.
If SCSS is giving you trouble, look at the CSS it's generating. Is it what you expect?
If there is a problem submitting a form, look at the Network tab to verify the right things are being sent. Understand how HTTP works here. Try putting the same request into `curl -v`. (Browsers even have a "Copy as cUrl" command nowadays.) If that looks okay, see if Rails is turning your submitting into the right params (or whatever your framework is).
One nice thing about checking the seams is you can debug via binary search: first see if the problem is in the browser or the server. Next see if it's in your app or the database. Next see if it's in your controller or your model. Etc.
This, specifically, is very good advice: using the ORM as a tool to implement a SQL-first solution. And not just good advice for beginners.
tl;dr: Keep digging, because everything's worth knowing at some point.
It's generally much easier to fix a bug in your own code or infrastructure if you're familiar with the innards of the things you depend upon directly, and learning your preferred platform beneath the current frameworks is a solid advantage because it helps you move sideways when the next New Framework comes along.
But there's also the conceptual side, the general concepts which have remained unchanged for decades, which a detailed knowledge of a specific platform (eg. Java, .NET, Node.js) doesn't necessarily aid with. These probably won't even feel relevant on a daily basis, but will definitely multiply your effectiveness over the long term.
Some things which might fall into that category:
* SQL and relational databases (especially if you're not using SQL, because you'll be reinventing it for any nontrivial amount of data)
* Consistency models and distributed systems (capabilities and limitations; some things can never be waved away by technology)
* The operating system, and principles of its design (not necessarily in any great detail, but knowing roughly how software talks to hardware is very useful, especially when assessing the marketing blurb for any cloud platform)
* Complexity theory (stringing three 'optimal' algorithms together might be textbook, but often you can apply some domain knowledge to cut out a step or two to reduce actual runtime and maintenance overhead)
I've found stuff like graph theory, principles of compiler design, and a working knowledge of assembly code to be useful too, but that's probably the far end of the bell curve for web server stuff :P
Computer Science is a subset of mathematics and it's amazing which bits of pure maths turn out to be useful in my day job. Shame I'm not very good at that stuff... but reading enough about it to recognise when it's applicable can mean the difference between 'picking a good library off the shelf' and 'hacking together a terrible solution myself'.
> Find out how to make your ORM log the SQL it's running, and try running that in your database directly. If you are having trouble, get the SQL working first, and then figure out how to make your ORM generate that SQL.
Time would be much better spent learning SQL properly. ORM's have their place, they can cut down a tonne of boilerplate, but anyone using an ORM should be somewhat competent in SQL already. If you learn SQL from ORM output then in many cases you'll be learning what not to do.
Make sure you really learn TCP and the unix toolchain for digging into network problems e.g. tcpdump. It's always important to isolate things: is it the latest push causing things or is it network problems?
- Persistence - Messaging - Publish-Subscribe / Event Handling - Latency - Separation of Concerns / Layering
https://news.ycombinator.com/item?id=18881649
https://news.ycombinator.com/item?id=16757128
https://news.ycombinator.com/item?id=13530924
- The front end can do whatever it wants. The back end is where all important data integrity and security logic should go.
- Sessions. No one has mentioned sessions? Sessions are somewhat important.
For example, i just googled for "zero downtime schema migration" and this covers the topic relatively well: https://blog.philipphauer.de/databases-challenge-continuous-...
In my experience, doing DB migrations w/o downtime is rarely worth it and involves big risks of actually ending up having long unplanned downtime due to the process being prone to errors. Large majority of schema changes can be performed very quickly and couple of seconds downtime is acceptable in most cases.
Long story short: Don't do DB migrations without downtime unless you absolutely need to.
P.S. If your business guy is requesting deployment w/o downtime, make sure to have a conversation with him to understand why is that so. In majority of cases they make it more dramatic than it really is.
As a backend engineer, what you create is basically the core of the product. Data structures and relations you define will be the limitations of your product.
Coming up with intuitive and good data structures for a complex application can actually be a very challenging task and as a backend engineer most of my time is not spent on writing js or sql, but on product design and specs, so I can create something that is stable and scalable, yet not too rigid, considering that iteration is part of the process and everything you create might be subject to change, specially in startup environments.
Data migrations make my palms sweaty (not schema migrations but the kind of migration that requires rewriting large amounts of existing data). Picking the wrong data structure is a fast path to data migrations.