Ask HN: What overlooked class of tools should a self-taught programmer look into
The books went into detail and since reading them I've felt confident writing scripts I needed to scratch an itch. Over time, I grew comfortable believing I had a strong grasp of the practical details and anything I hadn't seen was likely either minor quibble, domain specific, or impractically theoretic.
This was until last year when I started working on a trading bot. I felt there should be two distinct parts to the bot, one script getting data then passing that data along to the other script for action. This seemed correct as later I might want multiple scripts serving both roles and passing data all around. Realizing the scripts would need to communicate over a network with minimal latency, I considered named pipes, Unix domain sockets, even writing files to /dev/shm but none of these solutions really fit.
Googling, I encountered something I hadn't heard of called a message queue. More specifically, the ZMQ messaging library. Seeing some examples I realized this was important. The step of then plowing through the docs was nothing short of revelatory. Every next chapter introduced another brilliant pattern. While grokking Pub/Sub, Req/Res, Push/Pull and the rest I couldn't help breaking away, staring in space, struck by how this new thing I had just read could have deftly solved some fiendish memorable problem I'd previously struggled against.
Later, I pondered the meaning of only now stumbling on something so powerful, so fundamental, so hidden in plain sight, as messaging middleware? What other great tools remain invisible to me for lack of even knowing what to look for?
My question: In the spirit of generally yet ridiculously useful things like messaging middleware, what non-obvious tools and classes of tools would you suggest a hobbyist investigate that they otherwise may never encounter?
415 comments
[ 1.8 ms ] story [ 292 ms ] threadEnterprise Integration Patterns
Just start reading some Martin Fowler books, you will be up to your ears in patterns in no time :)
Design patterns give you a vocabulary to describe design choices you made. They don't give you a set of things that inform design.
Knowing idioms that relate to certain expressive / organizational problems is a good thing (especially if you're primarily working in a static manifestly typed language), but there's a weird overcelebrated status to them, and I'm not sure I'd encourage a developer I was training to become familiar with a full catalogue of them.
All kinds of languages have patterns. Whatever kind of language you use, learn the patterns that are relevant/useful for it.
They're different than external libraries, because usually the details of your application are closely integrated with how you implement the design pattern.
There are so many ways people misunderstand and misuse SQL and relational databases, it's honestly staggering.
So, I second the OP. Learn SQL, and you'll stand out.
- https://dataintensive.net - Really deep dives into different types of data storage solutions, their history, and how they actually work.
- http://www.cattell.net/datastores/Datastores.pdf - Good paper that helps differentiate similar but different datastores. Really helpful when you're trying to pick a modern data solution.
https://philip.greenspun.com/seia/
It goes pedagogically through the way things are typically done in a relational database in such a clear way that word-for-word it's one of the best tutorials I've seen... but it also weaves a narrative of "how can this be done better/more scalably/more reliably/more flexibly-to-business-needs" in pointing to a streaming/event-sourcing architecture. You may or not need the latter right away, but it's a fantastic tool to have in your toolbox to be able to say "ah, this new requirement feels like it would benefit hugely from this architecture."
Especially for OP who's starting to think about the "why" of messaging queues, this could be a fantastically valuable first step.
[0] https://www.youtube.com/watch?v=fU9hR3kiOK0
[1] https://www.confluent.io/blog/turning-the-database-inside-ou...
[1] https://en.wikipedia.org/wiki/SOLID
They're all good, and you'll get good insight from them all, but I think that first one is more important and has provided me more value than all the rest. I think Liskov substitutability would be my second pick.
Here is an excellent introduction to unified logs and stream processing by the author of Kafka at LinkedIn:
https://engineering.linkedin.com/distributed-systems/log-wha...
E.g. these four pages are the university of Cambridge masters in computer science:
https://www.cl.cam.ac.uk/teaching/1819/part1a-75.html
https://www.cl.cam.ac.uk/teaching/1819/part1b-75.html
https://www.cl.cam.ac.uk/teaching/1819/part2-75.html
https://www.cl.cam.ac.uk/teaching/1819/part3.html
(Or a MOOC, but the links above are easy to browse text, syllabuses and lecture notes, not a load of videos.)
The Cambridge course isn't perfect, but they do a very good job of making as much teaching material as possible publicly available.
Numerous times I'd be sitting in a class and we'd go over a solution to some theoretical problem, and I'd realize that this solved a problem that had taken me days to discover on my own (and this solution was usually better than what I'd come up with).
If you are the kind of person who can work through everything on your own (including what may seem like the the boring parts), I highly recommend doing so.
I've thought about going back for my CS degree a lot but can't really justify the cost and time investment vs self teaching. But it's something that's always been in the back of my mind.
(Sadly though this exposure was not in the context of a theoretical course on data structures, but rather in the context of reading the docs for HashMap as my university dropped older courses and languages to jump on the bandwagon of becoming a "Java school".)
Another example: the automata class I took went over pushdown automata, and I immediately saw that it would solve the issues I'd been having with a finite state machine I was using to handle input for a game.
Oh and recently I needed to put different sections of a screen on different layers so that no 2 adjacent sections were on the same layer. I realized that this was basically just graph coloring, so I was able to find a solution in minutes instead of hours.
I'm sure their are people who can get through most of a CS curriculum on their own, but I'm not that disciplined. I've also never met anyone who was. It has been immensely helpful.
Anyway, either of them will let you mirror a website so you never have to worry about it going down.
https://www.cl.cam.ac.uk/teaching/material.html
I can't recommend this book enough. I have a CS background, and still had quite a few "I can't believe this thing has been hiding in plain sight!" moments while reading it.
Are there good books for data intensive desktop apps? Like games or CAD design tools?
Example: https://gitlab.com/internet-cleanup-foundation/web-security-...
Running the 'make' command in this repo will setup all requirements and then run code checks and tests. Running it again will skip the setup part unless one of the dependency files has changed (or the setup env is removed).
I had the opposite problem of wanting to develop some stuff for Windows from a Linux environment, and I settled on running a linux VM and copying binaries over by scping to WSL, which works reasonably well.
If you just want a Make, there is [2] which can be installed separately and is part of the GNUWin32 collection.
[1] https://www.msys2.org/ [2] http://gnuwin32.sourceforge.net/packages/make.htm
GNU Make even comes with a vcproj file for building a native binary with Visual Studio. Worked fine for me. Building it with Guile support though is difficult, but fortunately Eli Zaretskii provides native binaries through his ezwinports, and they worked pretty much flawlessly for me. Of course you will usually need a shell to execute recipes, but Make itself runs natively. For more information, see README.W32 in the sources.
It is easy to add command line arguments to the tasks, configure them using files (json, yaml), environment variables, to split the task definitions into several modules/namespaces.
There's always some level of bootstrapping a project (installing packages/software, compiling libraries and dependencies) where it's easier to just to write a shell script than to program python to do. E.g. How do you get fabric installed on a system?
There's also been this longevity of sorts that Make seems to have gotten right. People just keep going back to it because it's simple.
I used bash scripts for years, but for a lot of reasons made the switch:
- It was always painful to create small libraries of functions used across multiple scripts in a project
- It's difficult to consistently configure them with per-user settings. I've written bash implementations of settings management, Invoke handles this for me.
- I'd still have to reach for Python whenever I needed to do anything with arrays or dicts, etc.
- Getting error handling correct can be a chore
Invoke has a lot of nice to haves to:
- Autogenerated help output
- Autocomplete out of the box
- Very easy to add tasks, just a Python function
- Easy to run shell code when needed
- Very powerful config management when needed
- Supports namespacing, task dependencies, running multiple tasks at once and deduplicating them
It's not perfect, but it's a lot better than my hand rolled scripts were.
The first example they use to describe the tool is:
> Cufflinks is a tool to assemble transcripts, calculate abundance and conduct a differential expression analysis on RNA-Seq data. his example shows how to create a typical Cufflinks workflow with Snakemake. It assumes that mapped RNA-Seq data for four samples 101-104 is given as bam files.
This is epitome of non-programmer programming, colour me disappointed.
The big downside of Make, alas, is Windows compatibility.
You'd have to give me a _very_ compelling reason to support developers who use Windows, when Windows lacks these essential tools. Besides, don't people who develop on Windows live in WSL?
(I also write code to run on Linux, but still prefer Gradle.)
I’m just getting into Kotlin and gradle isn’t something I’ve used before since I’m mostly web, .net til now.
I can barely understand why you'd want to develop on Windows (ok, for non-Windows-only products) with it, but without it...
I've been actively using WSL for over a year along with Docker and set up the Docker CLI in WSL to talk to the Docker for Windows daemon.
Performance in that scenario is no different than running the Docker CLI in PowerShell, or do you just mean I/O performance in general in WSL? In which case once you turn off Windows defender it's very usable. WSL v2 will also apparently make I/O performance 2-20x faster depending on what you're doing.
WSL adds a lot if you're using Docker IMO. Suddenly if you want, you can run tmux and terminal Vim along with ranger while your apps run nice and efficiently in Docker. Before you know it, you're spending almost all of your time on the command line but can still reach into the Windows cookie jar for gaming and other GUI apps that aren't available on Linux and can't be run in a Windows VM.
It's acceptable for relatively infrequent file access, but will eat you alive if you're doing anything that involves lots of random file access, or batch processing of large sets of small files, or stuff like that.
That's dealing with 10k+ line projects spread across dozens of files quite often, and even transforming ~100 small JS / SCSS files through Webpack. It's all really fast even on 5 year old hardware (my source code isn't even on an SSD either).
Fast as in, Webpack CSS recompiles often take 250ms to about 1.5 second depending on how big the project is and all of the web framework code is close to instant to reload on change. Hundreds of Phoenix controller tests run in 3 seconds, etc..
VS2019 supports Clang and cmake now.
https://insights.stackoverflow.com/survey/2019#technology-_-...
Windows 47.5%
MacOS 26.8%
Linux-based 25.6%
BSD 0.1%
87,851 responses
(The Stack Overflow survey is a poor representation of the entire development community, but it's worth something, maybe the best we have.)
Isn't the big problem that you have no idea what it's doing to your system? Also that you aren't expected to be able to undo it. You can read the makefiles, of course, but it seems simpler not to have to. (Just update the necessary packages yourself, to the latest version.)
Forgive me if this is naive of me.
At that point, I use a scripting -not shell- language (which is not as implicit)
I don't program in c anymore, so my major workflow is all in one language; I write my server in the same language that do s the compilation, which is in the same language that does utilities like creating network tunnels to my lab nodes
As opposed to what exactly? Any other alternative, e.g. separate shell scripts, "npm run" scripts in package.json, running a Docker image, hell even cmake or other make-like tools - does stuff you don't know about without reading the files either.
I tend to write my Makefiles to create as much of a local dev environment as possible for every project. Using Python virtualenv/Pipenv/Poetry, Ruby vendored dirs, custom Gopath per project (using direnv), etc. Most tools support some sort of isolation/localisation, but it's often just not on by default.
I have seriously considered hiring someone to audit and prune all the random little libraries and tools I've installed over the years for that one-off time I had to process a weird file format or wanted to try something from HN.
It does sound like the right idea. This is hypocritical as someone who doesn't use it but I hope more people use it.
https://godarch.com/
Every boot is a fresh install. Any one-off only becomes persisted unless I add it to my recipes.
https://github.com/pauldotknopf/darch-recipes
GNU Make works fine on Windows. The sources come with a vcproj to build it natively, or you get it from ezwinports. At my dayjob, we have a pretty complicated build with GNU Make for cross-compiling our application to Arm and PowerPC, and it works on Windows, even with special Guile scripts to reduce the number of shell calls which are extremely slow on Windows.
VBScript works better on Windows, IMO. Also works out of the box on all Windows versions since at least 2000 (on Win9x it was shipped with IE).
Not really... not since XP, anyway. Unless you have a space in your username (which is a terrible idea for many other reasons), your "Documents" path is C:\Users\JohnSmith\Documents. "Program Files" is pretty much the only important path which is likely to have spaces, and your makefiles (hopefully!) don't need to touch that.
For a software to work fine on Windows, it must support spaces in files and paths. Also Unicode in files and paths.
VBScript does, GNU Make doesn't.
> which is a terrible idea for many other reasons
If you use make to setup stuff, it's very possible you'll need to access "c:\Users\All Users" which does contain space in username. Also "c:\Program Files (x86)\Common Files" which contain more than one.
DOCUME~1 Documents
or
<SYMLINKD> ALLUSE~1 All Users [C:\ProgramData]
That is entirely correct and really the most glaring downside of Make. In my opinion: If you have spaces in your dependency names, just stay away from Make as far as possible.
I’ve just installed Visual C++ 6.0 Professional on a WinXP VmWare machine. Took less than a minute, BTW — modern SSDs are awesome. The default installation path under program files also contain spaces, it’s "C:\Program Files\Microsoft Visual Studio\VC98"
BTW, they have a bug even on the very first welcome screen: http://const.me/tmp/vc6.png
>ezwinports
Interesting, hadn't run into this before. What's the advantage over MSYS2?
Can you break down your problem into a bunch of rules of the form “To produce this file, I need to run these shell commands, which read those other files over there as input”? If so, Make can take care of figuring out which steps actually need to be run.
http://matt.might.net/articles/intro-to-make/
https://www.youtube.com/watch?v=fkEz_oVh0B4
For learning advanced techniques: "The GNU Make Book" by John Graham-Cumming.
But the Make manual is pretty comprehensive as a guide and reference: https://www.gnu.org/software/make/manual/make.html
Also (as with most things) knowing what name some concept has makes it easy to search for references. For example the terminology of rules (target, prerequisite, recipe): https://www.gnu.org/software/make/manual/make.html#Rule-Synt...
Things I tend to google often because I forget and some are used more often than others are: automatic variables, implicit rules, conditionals and functions.
One trick that really helps making Make complete is making your own pseudo state files and understanding the dependency system. One of the best features of Make is its dependency resolving. You generally write rules because you want a target (a file or directory) to be created, based on prerequisites (dependencies) according to a recipe (shell scripts). Make figures out that if the prerequisites didn't change, it doesn't need to run the recipe again and it will reuse the target. Greatly saving on build time.
Because Make relies on file timestamps to do its dependency resolving magic if you don't have a file there is not much Make can do. So what you can do instead is create a pseudo target output yourself. For example: https://github.com/aequitas/macos-menubar-wireguard/blob/mas... Here a linter check is run which creates no output. So instead a hidden file .check is created as target. Whenever the sources change the target is invalidated and Make will run this recipe again updating the timestamp of .check. Also note the prerequisites behind the pipe (order-only prerequisites). These don't count toward the timestamp checking, but only need to be there. Ideal for environment dependencies, like in this case the presence of the swiftlint executable.
https://nostarch.com/gnumake
All in all tho, it's a fantastic place to write down long CLI commands (ex: launching a dev docker container with the right networking and volume configurations) that you use a lot when working on the project.
Our Jenkins pipeline also relies on the Makefiles, literally just invoking `make release`, which is also pretty awesome.
Make does support includes (https://www.gnu.org/software/make/manual/html_node/Include.h...) which allow for some form of code reuse across projects. But then you encounter the balance between DRY and clarity. There are always exceptions, so you try to make stuff to universal, but then its hard to grok the code. And I feel that if I start to use functions I'm using Make wrong and that kind of logic better fits in shell scripts called from the Makefile. Makefiles (the way I use them at least) should be simple to read and explain themselves. But it's often hard to balance this with the features Make provides, like implicit rules and automatic variables. And if I ever turn to generating Makefiles (other than for C projects where it kind of expected) I will probably retire.
Oh absolutely. It's fantastic for that. Our build pipeline actually relies on that; every project has a "release" target that is basically for the CI to use.
> Make includes
Yeah, I looked into that, and I think I had the same conclusion.
> scripts called from the Makefile
That's what I'm thinking is the way to level up this kind of system. Although then, why have `make init` instead of just `./bin/init` ?
In the `make init` example. It doesn't matter how many intermediate steps are involved `init` is the end-state I want to achieve. So in most of my Makefiles the `init` target will fan-out into requirements as wide and deep as it needs, including running apt to install missing system dependencies. But then the good part. If a dependency is already fulfilled Make won't have to run it again. Although sometimes its hard or clunky to convert some dependencies into 'files' so Make can do its dependency resolving work properly.
That said, I wholeheartedly agree with the comment.
It's sad that something so central to a project and so useful and important to so many people seems like it hasn't advanced ... ever.
Developers generally do the minimum with Makefiles and get out. They are similar to 1040 forms in popularity.
I've always had a dream of a redesigned "make" system... with import statements, object oriented rules, clear targets, rules and files clearly seperated, structure and organization... sigh.
That said, mileage may vary. It was originally built by Google so it has quirks. I find it is best suited for projects with compiled dependencies and large repos.
Otherwise, I was going to add that Gradle as a build system is very advanced and improves upon make in many ways.
As I'm not the most intelligent developer, I'm sure there's a better, more sophisticated way to do this but I got really frustrated and gave up.
There, fixed it for you. In particular, I'm glad that the OO cancer hasn't spread to something as basic as a build system.
It is also a nice document of what you can build and how.
I also like it because Netlify supports it, so you can get it to run make to deploy your site when you push a commit, giving you a lot of control about your CI, while keeping it simple.
https://github.com/ruby/rake
If it sounds interesting, check out https://www.npmjs.com/package/shelljs
PS: I do my primary development on windows, but my production environment is ubuntu. node apps "just work" on both environments. truely cross platform.
But of course I'm immediately skeptical of this idea a la https://xkcd.com/927/ (Standards). For instance, maybe this is what npm and all the rest thought they were doing. Certainly Rake in the ruby world tried to do this, and I never really liked it, so clearly they missed the mark somehow, at least for me. But then when I feel discouraged about the ability to improve on things, I think about how I felt this way when I first heard about Git. Why would you implement a new source control system when we already have subversion? Sure, svn has its frustrations and warts, but this new thing is just gonna have its own frustrations and warts and now we'll just have another frustrating warty thing and we haven't really gained anything. And this is totally true! Git is super frustrating and warty. Except that it's also way better than subversion, much faster and far more flexible. It was a revelation when I started using it. So I think back to Linus when he was thinking about creating git and think that he probably didn't have this discouraged uncertainty about improving things; he just had ideas for a better way and he went out and did it. (And yes, I know it was influenced by bitkeeper and other DVCs exist, so it's not like he invented the concept, but my point stands.)
So maybe someone could make a better Make?
The things you’ve learned were mostly covered in my computer science curriculum. So I am going to second Robin_messsage his message.
Skim through a CS curriculum.
Sometimes, having a limited toolset would better focus you on the problem at hand. Then, once the challenge is clarified, the search for alternative ways to architecture and implement it would become practical.
If just for fun of exploring something new, pick whatever interests you in general, language, framework, a domain. Async processing and idioms?
VSCode or PyCharm (assuming you are still a Python developer) could be a good place to start. I'm always surprised when I see professional developers coding in Sublime Text and debugging with print statements (or their equivalent). Usually you have better options than that, especially for statically-typed languages - but even for JS and Python.
Property based testing aren't quite formal methods, but I think they are a good stepping stone. And they also somewhat force your code into an algebraic/functional style which also make it amenable to refactoring, better testing and is easier to understand.
Design tools like Swagger can help one think through services w/o diving into code. Code itself is a liability and should be thought of as "spending" not creating. Code is a debt.
Refactoring and code understanding tools, if you use PyCharm (you should, it is free in all senses), learn how to navigate into your libraries. Read you libraries.
https://bigmachine.io/products/the-imposters-handbook/
Doesn't matter what the project is. Could just be tossing around a string of data that eventually gets dumped into the db and sent to the server.
Research different methods of architecting such a system. Code up a few.
By doing this, you will actually find the answer to the question you posed. And, arguably, have fun doing it.
Many of the recommendations in other comments could tie into architecting such a study project too: data structures, algorithms, communication among distributed processes..