HN Explorer

Ask HN: DevOps learning resources

288 points by durian89 ↗ HN
Hey guys,

I've been a developer for some time now, but I've never had to do anything with deployment, servers, and all these good things.

I'd really like to have the basics so would you guys have some nice resources to gain some general knowledge about it. Thanks!

67 comments

[ 4.8 ms ] story [ 127 ms ] thread
nikivi
I made a mind map for learning DevOps :

https://learn-anything.xyz/programming/software-development/...

Clicking on nodes with a map will go to other mind maps with resources.

mkez00
Nice. Hope you don't mind me pointing this out, but cluster manager seems like odd semantics, wouldn't that grouping be container orchestration?
nikivi
Yes, I think that is more appropriate too. Will change it now.
vacri
You might want to add in networking components as well. Throw in (OS) packaging as well (not all services are put into containers...). I'd change 'automation' to 'config management', since the former concept is also part of a number of other nodes (particularly CI and orchestration). Datastores and their efficient use are another set of nodes (databases, caches/cdns, things like s3). Secrets management (fun!) is another one, though that belongs under config management.
maruhan2
I have a question. How are automation and CI different?

I don't know much about these stuff and the only thing I have ever touched is Jenkins. So how is Jenkins actually different from Ansible for example?

oblio
Well, the best way to learn is to actually do something. Create a small service in your favourite framework. Something really small, an echo service will probably suffice.

Then stop "developing" and switch on "Ops mode".

Automate:

- builds (build & packaging scripts)

- deployments - try all 3 major approaches:

    - push deployment: running a command on a central server that orchestrates everything (Ansible, Salt, chef-solo, ...)

    - pull deployment: agents running on your target server, that pull the latest changes (Chef, Puppet, ...)

    - immutable infrastructure: VMs or containers that are never modified, only recreated (CloudFormation, Docker, ...)

  include database updates in your deployment orchestration and possibly include even environment pre-warming/pre-caching
- functional tests, especially fast smoke tests

Add:

- high availability/load balancing (Nginx, HAProxy, Apache, Elastic Load Balancer)

- detailed technical monitoring and graphing (Nagios, Zabbix, Cloudwatch)

- availability monitoring (Pingdom)

- a status page (can't give you a decent example; you can build your own, but host it somewhere else than your main "app")

- log collection and shipping (Splunk, Graylog)

Basically, for almost everything I listed google options and pick an "Ops stack". Then implement that as best you can.

Oh, and by the way, while working on the "Ops stack", only "develop" things in support of this Ops work in your "app".

toomuchtodo
> - a status page (can't give you a decent example; you can build your own, but host it somewhere else than your main "app")

Does statuspage.io have free trial accounts? Integrate with them for your demo/trial using API calls. Bonus points if you pull in applications metrics to your status page (response time, etc).

falcolas
The only thing I'd add is be sure to do it manually first - you'll need the solid foundation of understanding how it works (and how it fails) in order to automate it. Note the key word - understanding - and how that means that just googling the steps won't help you in the long run.

Set up & configure Jenkins to pull and build. Set up and configure nginx, MySQL/PostgreSQL, init files for your app, iptables to firewall off unnecessary ports, DNS, and LetsEncrypt SSL certs (and the associated nginx configs). Make sure nothing that doesn't need to be is running as root. Make sure that what is running needs to be running, and nothing else.

Then, start automating it. Toss in cloud features like ELBs, RDS instances, cloudformation (or terraform, etc), autoscaling groups, autoscaling container services, etc.

This will take you a very long time (and you will probably have enough skills to be hired as a "DevOps" engineer well before you're done), especially if you strive for understanding. But it's really the only way to get where you want to get and be good at it.

collyw
While I agree that learning by doing is the best way to start, once you get beyond the basics its nice to have some patterns and best practices outlined so you can try to do things more "properly".
vacri
Friends don't let friends configure Nagios...
oblio
How is he supposed to learn the Ops part of DevOps without pain and suffering? :D
busterarm
> - log collection and shipping (Splunk, Graylog)

I'll also throw my hat in for setting up (and securing, without Shield) an ELK stack here. You'll learn a lot along the way.

nikon
Agreed, recently set up the Elastic "stack" with fluentd shipping logs from Kubernetes nodes. Fun and very rewarding.
busterarm
I'll have to look into that.

I'm finding Beats + Logstash to be very convenient though.

oblio
I forgot to mention 3 very important things "to add":

- security (you should already know a great deal from your work as a developer, the general things such as no open ports, no plain text passwords, etc.)

- related to security: secret management (Chef, Puppet - I think, Vault)

- backups, backups, backups: for the database and basically anything which is not reproducible from source (which in general should only be your database data)

rob-olmos
Interesting only one comment made a clear call to backups. Strongly second the notion of backups as a high priority to learn and implement. Could make it more "devops-y" by also automating the verification of them with restores/testing.
[dead] jankedeen
These have packaged the process and tools but no one can tell you how to do it for yourself. Because at up to an expert level it is still a set of tools you use. A lot of the advice/terminology: 'fast smoke tests', 'google ops stack', 'host your status page elsewhere', choice of load balancers, etc.. is stupid-hoc.

Do yourself a favor if you are going to do devops practice.

1. Write it from scratch. 2. Make it work no matter what happens before and after calamity. 3. Write the docs. 4. Leave it to beaver.

theossuary
There's two major types of monitoring you should be concerned with, log aggregation and metrics collection.

For log aggregation the most mature open-source solution I know of is the ELK stack (Elasticsearch for storage, Logstash for aggregation, Kibana for visualization). Set that up (bonus points if you use Puppet/Chef/Ansible/Whatever to set it up for you), and make sure you can detect anomalies from your visualization interface.

For metrics collection, Prometheus has a special place in my heart, I love its architecture. That said there are quite a few options, InfluxDB is another (arguably simpler) option. You can then use something like Grafana to visualize the metrics.

For configuration management tools, Chef is complicated and you better know Ruby and have a few months to dig into it. Puppet is simpler, but has a lot of weird quirks. Ansible is simple, but not as powerful (or used); so pick your poison, they all are pretty similar conceptually.

Basically at the end of your journey you should be able to kill any server (or network connection, or application running on a server) and be alerted to it in near realtime. You should be able to diagnose what's wrong quickly using Kibana and Grafana, and finally you should be able to destroy and recreate the server using a few well understood commands and have it be completely reconfigured and ready to accept connections. Of course this is pie in the sky (like having perfect unit/integration testing for a very large application), but that should be the goal.

bradknowles
With respect, Puppet is push-based, not pull-based.

Chef is pull-based by default, but you can also use it in a push-oriented manner.

Having worked with both Chef and Puppet, it is my opinion that Chef is the more scalable of the two, and the one that makes more logical sense to me. YMMV.

sharjeelsayed
Here is some I collected

http://serverfault.com/questions/1046/what-is-the-single-mos...

https://github.com/Leo-G/DevopsWiki

https://learn-anything.xyz/programming/software-development/...

https://news.ycombinator.com/item?id=14153545 (What Is ‘Site Reliability Engineering’?)

https://www.reddit.com/r/homelab/wiki/software

https://www.reddit.com/r/linuxadmin/comments/65vb8s/advice_i...

https://www.reddit.com/r/sysadmin/wiki/bootcamp

https://www.reddit.com/r/sysadmin/wiki/learn

http://Learn.SharjeelSayed.com

wyclif
This is a less general answer, but if you want to work with AWS and you have a Kindle, download the manuals for free and read them during down time. I did this and it helped me a lot.
Juliate
Also, if you can afford it, the Site Reliability Engineering book is an excellent comprehensive resource on practices, theory, return from experience: https://landing.google.com/sre/book.html (and you can read it online for free, too).
flor1s
Instead of buying the book, why not get Safari (the website, not the browser) and read that book plus many more? And if you are an ACM member you already have access to it through http://learning.acm.org
atsaloli
Check out: - The DevOps Handbook - State of DevOps Report 2017 - Guide to Sysadmin Body of Knowledge www.sabok.org
ozkatz
I use pocket to collect (and tag) links related to the field. I hacked together a small UI around it that lets you filter them by tags. There's a lot of good content there IMHO.

You can check it out here: http://links.ozkatz.com/

kellet
Over at DZone, we have big community of developers sharing tutorials and content that could help introduce you to DevOps.

For starters, we have an entire web portal with thousands of free community-written articles about DevOps-related topics: https://dzone.com/devops

We also have several Refcardz (cheatsheets) on a variety of DevOps related topics:

Deployment Automation Patterns https://dzone.com/refcardz/deployment-automation-patterns

Continuous Integration Servers https://dzone.com/refcardz/continuous-integration-servers

Continuous Delivery Patterns https://dzone.com/refcardz/continuous-delivery-patterns

And for an introductory overview, we provide topical research and best practices Guides. Here is our latest 2017 DevOps Guide: https://dzone.com/guides/devops-continuous-delivery-and-auto...

samstave
Where do you live? If you'd like to meet up, I'd be happy to braindump as much as possible on you.
digitalsushi
I work at one of the large insurance companies as a DevOps role wherein I convert the various product build processes into a CI pipeline. We use Jenkins (we use all of them, but our department uses Jenkins). Everyone has roughly the same process in general - you store some code in SCM, and then something checks it out, runs various "Quality Gates" (corporate speek for tests), and then you eventually build the product and commit it to some secure artifact repository. You put some logs on there that you're somewhat convinced employees can't modify or delete, and call that an audit trail.

Ok, so that said, I would say that for an open source core like Jenkins, the source code is hands down the only documentation you can trust. And that's very, very frustrating at first. Trusting what's in a wiki is the easiest way to burn a day, believing something that was once (possibly) true to still be true. Most interfaces are not documented at all, beyond system generated documentation. A certain class of developers are fine with that, but many of us need a little push up the hill in order to script our way around these ecosystems effectively.

After accepting that the source code is the single source of truth, the job got a lot easier (and made me come across as far more of an expert). But it's a difficult sell to get other people interested in sharing this work.

organsnyder
The first thing to realize is that DevOps is an ambiguous term (at least partly by design, it seems).

My belief—shaped by many at the forefront of the DevOps movement—is that it is a cultural focus rather than a technical one. In many ways, it's an extension of agile philosophies, with a focus on fast feedback, transparency, heightened interactions between teams, etc. There is also a heavy focus on automation (CICD), but the automation is there to serve the cultural goals. Just because you do CICD doesn't mean you're necessarily doing DevOps, and you can adopt a lot of DevOps principles without doing full CICD.

Books:

* The Phoenix Project— introduces a lot of concepts (such as lean principles) that are foundational to the movement

* Effective DevOps (Oreilly)

* The DevOps Handbook

Podcasts:

* Arrested DevOps

* DevOps Cafe

Blogs:

* IT Revolution

Events:

* DevOpsDays conferences

* Local meetups

* Velocity conferences

* DevOps Enterprise Summit

Having a good grasp of both development and operations skills is helpful. But it's far from complete. If you solely focus on the technical aspects without examining the cultural, you're missing the foundation of the movement.

[dead] jankedeen
The foundation of the movement is getting rid of ops.
remilasalle
For those who've gone, is the DevOpsDays $175 price tag worth it?
organsnyder
They're offered all over the world, so the quality probably varies quite a bit (especially since it's a fairly decentralized organization). However, for the price tag, I'd say it's a steal—the networking value alone is worth more than that.

DevOpsDays has been around since the very beginning of the movement—in fact, the term was popularized at DevOpsDays Belgium in 2009.

johnpython
I recommend the following books to learn more about the work involved on the Ops side:

https://www.amazon.com/Modern-Linux-Administration-Cutting-E...

https://www.amazon.com/Practice-Cloud-System-Administration-...

https://landing.google.com/sre/book.html

https://www.amazon.com/How-Linux-Works-Superuser-Should/dp/1...

https://www.amazon.com/Effective-Monitoring-Alerting-Web-Ope...

https://www.amazon.com/Ansible-DevOps-Server-configuration-m...

https://www.amazon.com/Terraform-Running-Writing-Infrastruct...

https://www.amazon.com/Amazon-Services-Action-Andreas-Wittig...

djvdorp
Great list of books, I have quite a few in common and some to add:

https://www.amazon.com/Modern-Linux-Administration-Cutting-E... (General Linux)

https://landing.google.com/sre/book.html (SRE)

https://www.amazon.com/Ansible-DevOps-Server-configuration-m... (Ansible)

https://www.amazon.com/Puppet-4-10-Beginners-Guide-Second/dp... (Puppet, beginner)

https://www.amazon.com/Learning-Puppet-Configuration-Managem... (Puppet, intermediate)

https://www.amazon.com/Learning-SaltStack-Second-Colton-Myer... (SaltStack)

https://www.amazon.com/Networking-Systems-Administrators-Mas... (Networking)

https://leanpub.com/serversforhackers (SysAdmin)

https://leanpub.com/scalingphp (Scaling)

https://debian-handbook.info/get/ (Debian Linux)

sngz
for that linux administration book, how do you know whether its good or not? it's not out yet.
rb808
The new cloud space has changed a lot in the last few years. A short time ago I'd say you should learn about creating VMs, using ansible, scripting etc, now Kubernetes and Docker has changed much of that. Its worth looking at which space you want - with Kubernetes etc you barely need to know any Linux or scripting.

People learn differently - but I'd recommend starting there. Get the generous Google Compute intro special and write some apps. There is a bunch of free resources on the web - I can't point to anything specific. Linux academy, Safari books online are huge but not cheap.

mpdehaan2
(In case the handle is missed, I created Ansible and I'm a little opinionated on this)

DevOps isn't really a thing, but an amalgam of things. Some people think it's about culture or something, which I think is too obvious to be a thing. In the beginning when many people were using the word lightly, most people really just used it to mean automated systems administration, which is more likely called "Operations" now. That's fine. Some people use it to mean groups of people who make ops tools to allow developers to self-deploy their own stuff. That's also fine.

Most likely what you are looking for is to learn how to do IT Operations stuff they way people are currently doing it.

Reading a lot of articles is fine, trying lots of tools is fine. Talking to people at your company that DO ops is huge. Make friends with the guys who run the build systems, do security, or anything like that for starters and they can show you lots. Plus I strangely find that ops guys are much better to go to lunch with than developers. Don't know why :)

You should read up on AWS lots, as it will likely be across your career path at some time. Try a configuration management tool (or two). Learn about monitoring systems and logfile collection/analysis systems. Do a little bit of reading on computer security. Vagrant is probably useful, but optional, though you should at least get going on a virtualized Linux box. Reading up on Immutable Systems is worthwhile. Pick up either CloudFormation or Terraform, or both if on AWS. I don't know Google as well, but it has a lot of similar things.

DevOps Days conferences can be good sometimes but often they are too cultural to get down into technical bits. But they are cheap and usually close by, so they are things.

If you have a local meetup group that can be absolutely great.

The really nice thing about AWS now is there are tons of parts and it is pretty cheap to try things out, where before you probably couldn't get your IT guy to let you play with a load balancer or get you your own database instance. Now you can, so that makes it a lot easier to learn than it was before.

IMHO I don't like books because they are often written by people who don't DO things (DevOps has an unfortunate "thought leader" problem, which impacts conferences and tries to get everyone to believe the same things), and podcasts/videos are too slow for me, and my brain is a lot more random access.

Don't get caught up in assuming you must do any one particular thing. For instance, Continuous Deployment is a spectrum, it's not appropriate for everyone.

And at most people's scale, you have no need for something like Docker or Kubernetes when basic AWS instances require a lot less to keep going in your head.

ransom1538
Devops:

One word of caution. The future really will be having a docker file and finding a place to run code (docker swarm, ecs, etc). When you commit code it will be ran through a ci system (unit tested, staged) then updated.

The salts, puppets, chefs, Vaults, running your own Kubernetes (wtf), VMs, Nagios, sendmails, vpcs and all this other drama will be latin in a few years.

The future will be running the code on your macbook (with container of choice), then commiting. The end.

carlisle_
This is not particularly helpful advice. All of these things are worth learning and they are all still in widespread use and aren't going to disappear overnight. On top of that it's not like docker is some magic wand that magically makes DevOps happen. If that's the future it's not here yet and doesn't answer the OP's question.
ransom1538
not like docker is some magic wand that magically makes DevOps happen

Exactly. So I would start studying this. Like container security, how ci works with containers, how ecs works. Instead of terrible advice like 'learn how puppet works'. Further it is helpful advice. If someone asked me how they can get involved in action script coding I would warn them too.

nikon
Sorry but Kubernetes is production ready. Swarm is everything but. If you struggle with K8s, give Deis Workflow a try.

ECS is proprietary, learning how Kubernetes works gives you flexibility to move between cloud providers. GKE is a beast.

all_usernames
This could be true, from a developer's perspective. But the OP was in the spirit of expanding one's knowledge and understanding. But even if a single-interface, fully-managed container orchestration service is available to all, the value of understanding how the underlying systems actually function will not cease to exist. Anyone building moderately complex systems will still want to know about OS, networks, storage systems, access control, and all the rest, because those areas of expertise are applicable and useful across the stack.
[deleted]
(comment deleted)
southpawflo
get a digital ocean account and a domain name. start a lamp server (or equivalent), use the excellent (imo) documentation to get things running. add services as you go along. I learned a lot from just that, now I have my own webserver, source control (git server with gitea) and other things running. tons of fun and learning