Ask HN: How can a intermediate-beginner learn Unix/Linux and programming?
For a long time, I’ve been in an awkward position with my knowledge of computers. I know basic JavaScript (syntax and booleans and nothing more). I’ve learned the bare basics of Linux from setting up a Pi-hole. I understand the concept of secure hashes. I even know some regex.
The problem is, I know so little that I can’t actually do anything with this knowledge. I suppose I’m looking for a tutorial that will teach me to be comfortable with the command line and a Unix environment, while also teaching me to code a language. Where should I start?
113 comments
[ 3.2 ms ] story [ 229 ms ] threadPick one and move on to defining a realistic project that seems interesting and achievable.
Most important of all, finish it, and write what you did down somewhere. It took me 20 years to realize how important this is.
For Unix, install linux on your computer and use it daily. You’ll be forced to learn.
I tried this years ago and it didn't end well. With how complicated current-gen distros are, you need to understand what's going on under the hood before you can effectively mess with it. In my experience you'll just end up with one screen that's only at 720p and a crashed pulseaudio that you never fix because you just want to watch netflix right now, and for some reason that still works as long as you use bluethooth. Even though you didn't want to use your bluetooth headphones, it's easier than fixing the problem with your sound driver or whatever so you'll just end up using them anyway.
It might be easier to start with server management because you don't have to figure out much human IO stuff... Linux is really good at talking to Linux. VPSs are cheap and give you root access. Plus if you mess up, it's generally pretty easy to just format the VPS and start over. You can't easily do the same on a desktop. Also it forces you to learn about Linux and not whatever software your distro uses (for example, learning how to use apt is a lot more important than the ubuntu app store)
BTW if the OP (or anyone else) is wondering, now I run w10 and minimal Ubuntu with i3. The knowledge transferred basically 1:1 from server management (except for window manager stuff, but that was easy to research)
Also check github for a bunch of repos that contain biolerplate code that is used in most deamons illustrating signal handling, forking, etc.[2]
Also I suggest taking some open source projects from Daniel Bernstein (DJB) as examples on how to write secure code. qmail, djbdns, daemontools ... there are lots of great ideas there[3]
Write a couple of simple programs that utilize your code and expand from there, learn the plumbing, e.g. write a Makefile, learn autotools/autoconf, how to write tests, how to use gdb, how to create shared libs and link a program LD_LIBRARY_PATH/ldconfig, etc ...
Most important think about something that you like to write and go from there. Back in the late 90ies I studied RFC's (MIME, SMTP, etc) and then implemented things that I could actually use myself. Here is a recent project I did some weeks ago (an extreme edge-case for security maximalists which will never be used by anyone other then myself, ... but I really enjoyed writing this and learned a bunch of new stuff while doing so)[4]
if you need ideas or help with looking at your code, don't hesitate and send me a mail.
[1] https://en.wikipedia.org/wiki/Advanced_Programming_in_the_Un...
[2] https://github.com/jirihnidek/daemon
[3] https://cr.yp.to/djb.html
[4] https://github.com/DyslexicAtheist/nfq/
docker-systemctl-replacement is a (partial) reimplementation of systemd as one python script that can be run as the init process of a container that's helpful for understanding how systemd handles processes: https://github.com/gdraheim/docker-systemctl-replacement/blo...
systemd is written in C: https://github.com/systemd/systemd
> examples of how to write secure code
awesome-safety-critical > Coding Guidelines https://github.com/stanislaw/awesome-safety-critical/blob/ma...
It can be pretty overwhelming to use the terminal so you may have to some reading to get yourself really comfortable.
I usually go to books/(book pdfs) to help me start out because they have conceptual explanations about the syntax which really helps my understanding.
Maybe this one would be good? The Linux Command Line, 2nd Edition: A Complete Introduction
Once I'm comfortable based on the book, using StackOverflow and other websites becomes much, MUCH easier.
In the process, you'll learn BASH which is used for scripting.
Free PDF in the author's (William Shotts) homepage: http://www.linuxcommand.org/tlcl.php/
Keep projects small, just patch together open source code for fun, and read the open stuff you do use.
You could also switch the GUI environment of your raspberry pi. Install headless raspbian and install he GUI from the CLI. Just hack!
My koan is video poker, I’ve rewritten a whole game about 3 times. I also like writing prime number finders in new languages.
- https://cmdchallenge.com/. This set of increasing-in-difficulty challenges should help with structure in terms of learning a bit more.
- https://explainshell.com/ can be helpful as well.
- Don't forget the man pages! `man <command>` can be quite helpful! For a different approach on teaching some, bropages.org could be helpful.
- I haven't used it personally, but there is a lot of info at http://www.opsschool.org/
As for learning to program, I suggest a very good starter language is Python, and its built-in, first-party language reference/tutorial I found helpful when I learned it: https://docs.python.org/3.8/tutorial/index.html
As for "what" to program after that, I don't have any great resources for what constitutes some good initial programs. Off the top of my head, here are some ideas about places to start which are doable with only inputting and outputting text:
- FizzBuzz
- Play Rock Paper Scissors with the computer
- Play Tic Tac Toe with the Computer
- Sieve of Eratosthenes
- Conway's Game of Life
- Given a piece of text, count how many occurrences of each word there are and display the top ten.
Get familiar with the interface and the concept of a "terminal".
Then try do some simple stuff. Maybe set up nginx (or Apache) to run a web server that you can access via your host machine.
Thereafter try doing some Linux admin tasks, like creating users, switching users. Maybe even set up SSH and try accessing the virtual machine from your host machine via PuTTY.
After, maybe try spinning up a simple node web app on the virtual machine and access it from a browser on your host machine.
By this time, you'll probably get a sense of what you want to explore next! And remember, Google it's your friend for all of the above!
All the best :)
It's probably the best introduction to CS and programming online and teaches you the basics of the command line and C along the way.
Depending on the CS program, it's roughly equivalent of the first 1-3 semesters at a more average university, but the quality of the pedagogy is so high that it's very doable for anyone willing to put in the work.
[1]: https://missing.csail.mit.edu/
[2]: https://software-carpentry.org/lessons/index.html
[3]: https://mywiki.wooledge.org/ParsingLs
[4]: https://mywiki.wooledge.org/BashGuide
[5]: https://guide.bash.academy/
I feel like before any of that:
1. Do practical stuff with Python
2. Learn the semantics behind the Linux directory tree
3. CLI, piping stuff together, a little bit of bash.
is there something I am missing out by not using vim?
Being able to use vi is a very handy skill if you're a sysadmin or SSH into other computers/clusters very frequently. Since vi is part of the POSIX specification, all Unix systems would have some implementation of vi. Of course, Vim is much more powerful than a text editor like nano, but also requires more investment to learn. It won't make you a better programmer, but it might make you a faster programmer.
(Oddly enough, the default navigation used for the info pages is that of emacs.)
Someone's reading this itching to type that you can simply switch the default navigation for those yourself, which is true, but it's still helpful to know these if you have to hop on someone else's machine or ssh into a remote server.
Try one of these editors for at least a month after doing the basic built-in tutorial (e.g., vimtutor) and go through the tutorial every Sunday or so as needed to refresh your memory.
Or you could also use Anki to remember the shortcuts as you learn about them and use them.
This tutorial introduces Vim only after introducing shell and pipes. I was wrong about makefiles -- this tutorial doesn't include makefiles (I was thinking about Software Carpentry [1], which has similar goals). In any case, learning Vim isn't hard as people make it out to be, and one can easily skip that part of the tutorial to come back to at a later point without breaking the continuity.
[1]: https://software-carpentry.org/lessons/index.html
LFS is well-document and maintained, and levels you up in a jiffy on the practical basics.
It's not about programming per se, but covers quite a few concepts you're expected to "just know".
Plenty of pointers to more advanced concepts, too.
My strategy to solidify my linux skills was simple: commit to linux desktop. We spend most of our time on our desktop, may as well let that time earn you skill points. I picked ubuntu and use the same on servers. You naturally learn a bit every day through daily use. This really adds up over time without extra effort of using a side system. Getting good at windows or osx is a waste of your time unless you are specifically into those platforms.
As for a language, pick one that is (1) easy to start with (2) covers advanced use cases, (3) sound in its design, (4) general purpose, (5) concise. That rules out C (due to 1), JS (due to 3 and 4), php (2,3,4) and most things really.
For me that left me with ubuntu and python, I never looked back. (Yes I know many other things but that is my home base.)
The skills you need to run Linux on the laptop aren’t particularly close to what you’d need to run Linux as a server, IMO, particularly Linux as a VM.
I felt that I really started understanding how things work when I switched to something requiring more reading - Archlinux.
To get a running desktop it not only required doing all kinds of things I did not understand back then, it also provided great Wiki to get there. I still sometimes end up in that Wiki to quickly get up to speed about some tools or concepts.
And the fact that it is rolling release distro and was always running the latest upstream software version meant that frequently at least the UI part broke down, and that forced me to learn even more things. :)
That was only the beginning but now it feels like it opened many possibilities and career paths for me. It probably was a bit frustrating at the beginning but it is worth it.
On Ubuntu Server it will still automatically install bootloader, configure initram, full disk encryption, etc. And after installing desktop environment packages it will configure, enable and start all of the services automatically.
While it is easier and the result looks the same, you are less involved in the process and thus potentially learn and understand less.
Just try setting up Gentoo or Archlinux and then tell me that it was the same experience as Ubuntu Server. ;)
If minor things are constantly going wrong and getting fixed then the only way that can happen is the user building up a good mental model of how the system works.
An update that breaks X11 will teach you a world of information about terminals. "How do I use a messaging client in a terminal?", "What text editors work in a terminal?", "How do I browse the web from a terminal?", "What games are there for terminals", "How do I watch a movie in the terminal" (ascii).
Get forced to live like that for a week and you also learn how to fix X11 (eg, learning about the difference between 2D and 3D graphics drivers, OSS vs non-OSS drivers, kernel vs userspace). Valuable life skills. Godspeed to the Wayland devs.
When I was in college, a kid came to our * Nix User Group having fscked up his Ubuntu install by, I kid you not, `cd /; sudo rm -rf * `. He wanted help fixing it.
Another kid who was quite a bit smarter learned about fork bombs and ran one on a shared CS Linux server. It was offline for a few days. When it came back, `who` informed us that the sysadmin spent quite a bit of time running `man limits.conf`. (After graduating, I learned I was able to still login to that server using my SSH key, despite my account being deactivated in LDAP.)
It's a bit like Murphy's Law: anything that can be done on a * Nix machine will be done by a learner given sufficient time.
Regardless, even if most will eventually do the worst, isn’t it worth it? Perhaps not in the case of the server, but arguably even then one could (perhaps facetiously) just say that the sysadmin learned as much as the student. For me, one thing is certain. It made a huge impact on my future.
Disagree. There's nothing worse than needing to use a machine, but you can't since it's broken because some package maintainer pushed a bad release upstream.
There will be plenty to fix and tweak on any standard desktop or laptop when you install Linux, especially Macs. Might as well use a distro and release channel that let's you get your work done.
IMO, the hours that I wasted learning how to fix XFree86 went to waste.
It's ok to not like it, but don't let that dislike be due to lack of understanding.
Besides that, libraries tend to be made in C as well. That's because if it's in C, it's easy to make it available for other languages. Most general purpose languages will have some type of "foreign function interface" to call on C code. That means that the main documentation for these libraries will be for C as well.
I spent a lot of time in assembler too, and yes it has benefits for learning. Im also not recommending a high level toy like Logo.
I also say hand-in-hand as a bidirectional thing. I wouldn't suggest learning C without learning about Unix too. And I say that as someone who has worked extensively outside of Unix-like platforms.
Based on my experience with the switch, I've learned miles more about Linux and how computers work than I would've before. I might even say I like the experience more than I did before. On top of that, as the parent comment says, I've learned more about Linux, shell script, the directory tree, regular expressions, etc., than I think I would've otherwise by self-teaching. It might (read: "definitely will") be uncomfortable at first, but the Linux community is big and helpful and there's plenty of great open-source software that can replace your old workflows.
As far as picking a language goes, I started with Python for the reasons above, but really, starting anywhere is better than being stuck deciding. Common starting points are Python, Java, JavaScript, C#, but really, just pick any language with a large community to help you out. When you think you know enough, consider ideas for a personal project, which is where you'll likely learn the most. After that, keep learning and if you think you want to learn something else, you'll realize how much easier it will be with one language under your belt already.
That's interesting, most people I've started on C have found it easier to start out in that than say, Java, which has been on a hell of a lot of introductory courses for a decade now
What is the lifecycle of a process and their states? What is a signal? What is a file descriptor, a socket? etc...
Can you say you know about Linux without being able to answer these questions?
For programming language, I'd say Python is decent for a beginner. I've used Python as a language to teach some people programming, and the fact that they can do a wide variety of things (even games with Pygame) helps them stay engaged.
I would always recommend exploring other options as well though, to see what else is out there.
So that leave us with "sound in its design" and I'd argue the absolute unmitigated disaster of python 2 vs 3, the numerous competing competing package managers, and the abundance of really bad code examples due to the influx of the mathematics world (just look at R), I fully disagree that python is a good choice.
But then again, that's just my opinion, but setting people off on an unnecessarily biased path with the assumption that there's only one way to do things definitely sounds like a pythonic behavioral trait.
Once you reach a point where you want to do more advanced stuff or just try out other languages you can still learn about include, import etc. and start to structure your code better by splitting it into compilation units, classes, modules, what not.
Here's some BASIC questions an ubuntu user might never encounter:
Which processes are currently hogging the CPU?
Which ports are currently open?
How do you setup a remote git repo and/or add a user for limited ssh?
How do you increase an existing swap part without locking the CPU during write at peak loads?
How do you exclude/include packages from being autoupdated on apt update?
How do you ensure opaque crontabs aren't silently failing?
My advice: Setup a Linux server on aws or whatever for a paying customer with medium load. Only the stress of respondong to actual downtime will teach you anything useful.
Ubuntu is good for a server. Using 2 different distros would be good to learn how not all Unixes/Linuxes are the same and how they might differ.
I would also add: Whatever programming language you learn, you're probably going to want to learn some auxiliary languages as well. Don't get stuck thinking you can do everything with one language. Learn JavaScript, SQL, HTML and CSS as well.
Learn to write good Python. After that, learning to write good C will be much faster and easier and far less frustrating. Ultimately, on Unix you end up at C for at least some things (Python is implemented it C, after all). But these days, you can kick the can pretty darn far down the road before needing C.
learned so much about os stuff and kernel drivers, etc...
It was like diving in the deep end of a pool though.
Once you grasp how powerful this closed system of incrementally processing text is, you are sort of unstoppable at the command in a *nix system.
Pay careful attention to regular expressions, which are a universal tool in Unix for processing text. Along with Unix' treatment of files as a stream of newline terminated strings, this how Unix gets away with working with unstructured files instead of something like XML or S-expressions): that is, most Unix tools expect you to work with a file (or file descriptor, which is a pointer to any file-like thing in memory), and to process it one line at a time, and to use regular expressions to parse those lines into fields.
Working with lines using regular expressions is how the original editor, /bin/ed, was conceived, before the existence of visual editors like Vim and Emacs: you can find, print, and also edit lines in the file using regular expression syntax in ed, and this knowledge, btw, translates to fluent use of regular expressions in both sed ("stream editor", or basically a non-interactive version of /bin/ed, which essentially executes your editor commands and then exits), and egrep (which searches for lines matching your regular expression).
In fact, once you start to see these pieces come together, look for examples in Kernighan and Pike[0] which demonstrate the use of awk. Awk is an extremely powerful but also extremely simple tool for processing data in Unix, using all the concepts I just described, and not much else. It has a few extra features, like variables and loops, which you'd expect in a programming language, but you are expected to use Awk exactly like you'd use things like grep, ed, and the shell. It's just that now you've got this very compact, powerful notation for splitting records (lines) into fields (regular expression matches), and then doing arbitrary things with them to spit text back out. You could even write your own database using something like Awk, and you can read about how to do database-like tasks in the classic and accessible book on Awk by the original authors of the tool (Aho, Weinberger, and Kernighan, whose last initials together form the name AWK)[1].
Then, much later in life, when you find yourself doing some fancy stuff in Haskell and creating a DSL to do things like process pipelines, you realize that Awk was actually way ahead of its time.
[0] https://en.wikipedia.org/wiki/The_Unix_Programming_Environme...
[1] https://en.wikipedia.org/wiki/The_AWK_Programming_Language
- Customize your bash prompt, create aliases for some common commans (ls, df are good)
- Build a LAMP stack, compile everything from source tarballs. Get it working so that you're able to browse your webserver from your phone.
- Update a sound or video driver
- After you're comfortable, recompile the kernel
- Through these, do as much in the command line as you can. Get comfortable with mv rm chmod chown and so on.
Any time you have edit a text file, use vim, nothing else.
1) install the language of your choice and start a simple web site using the languages webserver. View the website in your browser on Windows. A Python Flask 'hello world' or a Node one?
2) Set up an SSH server and remote in from the Windows machine. Learn how to set it up to use keys only instead of passwords. Find out why that is a good idea
3) learn Git. Github has some nice tutorials.
4) set up NGinx to run your website above.
5) set up Nginx for https
6) set up LetsEncrypt.
7) set up a free AWS account, start a server and put your website on it using SSH and git.
8) look through the ssh log files to see login attempts, and the Nginx logs. Imagine there were millions of rows, how would you find a certain one, or the last few? (head, tail, grep, cat).
That should get you somewhere.!
Read and lightly work through The C Programming Language (K&R). Code along with the book, don’t feel you have to really deep-dive anything.
You don’t have to learn C well at this point. Just learn from it.
Then start reading The Art Of Unix Programming. Do this casually. Read a bit, code a bit. Between K&R and TAOUP, you’ll be inspired by to write at least two small projects.
To supplement K&R and TAOUP, begin reading Unix Network Programming.
If you have access to a Mac or a Linux machine, explore the man pages.
If you’re on a Mac, explore the man pages __and__ the code that lives on your system.
One way to do this is to use something like vscode’s intellisense to “Go to Definition.” E.g., when you’re coding an example from K&R, right click on one of the included headers, and click “Go to definition.” It will take you to the header file. Look through the code. Then you could look at the code from FreeBSD or OpenBSD on GitHub.
Edit: Just wanted to add, by doing the above you’ll get exposure to your system in new ways. Moreover, you’ll find yourself learning Unix out of necessity, for fun, and by accident. You can practice the art of Unix programming in almost any language at this point. C is unix-y, but so is Rust.
My second piece of advice would be to learn C from this book: https://en.wikipedia.org/wiki/The_C_Programming_Language
After finishing it and completing all the exercises, you should be ready to go learn anything else you want to.
Get yourself a bulldog clip to hold that send book open to the page you are repeated referring to. When you graduate to Stephens' UNIX Systems Programming, the bulldog clip won't be needed.
If your monitor doesn't stand tall enough, 4 reams of copy paper are much cheaper than the same thickness of X programming manuals.
Keep the terminal open, if it can be done in a terminal it may turn out you prefer it when you get used to it. Torrents, IRC, playing music, email, file explorer, whatever makes you stay in there. It makes you comfortable navigating the system and you get to see what a GUI hides from you.
Programming? You're not gonna benefit from diving into any deep ends there, continue with JS if that's what you like. Learn databases and web servers, here your new linux skills comes in use. Try some stuff and see what you like.