Show HN: Jeeves – A Pythonic Alternative to GNU Make (jeeves.sh)
Write a Python file named `jeeves.py` in your project directory with contents:
import sh
def lint():
"""Lint your Python project."""
sh.mypy()
pip install jeeves-shell[all]
j lint
• Automation of routine tasks,
• Standardization of your projects,
• Implementation of best practices,
• And more :)
88 comments
[ 4.3 ms ] story [ 153 ms ] thread“Why would I consider this, given that make is perfect”
And
“Doesn’t eMacs org mode already do this?”
—
Seriously, looks nice.
I’m a big fan of Jeeves and Wooster novels and I am a little concerned that Reginald Jeeves would not like to be portrayed as a kind of murderous robot. He’s far too civil to execute people (though he did use a kosh to knock out a police officer on at least one occasion).
Perhaps this particular robot is running a much more civilized version of Skynet/Cyberdyne firmware.
And is sure to be speaking impeccable English.
By making .PHONY and .ONESHELL labels history, as a particular example.
Not even that; this doesn't seem to implement any of make's core features, so it's not actually a replacement for make. It's a replacement for shell scripts.
(And no, I don't buy that the comparison is apt because some people use make as a command runner. Sure, they do, sometimes, but that doesn't mean jeeves is a make replacement. It's not.)
/sarcasm
I would prefer something far more formal than Python.
https://github.com/casey/just
Edit: played with just and got confused about how it works. Not going to use it just yet.
No more remembering commands to build a release or deploy a docker image. Now it's 'just release' and 'just deploy'.
(1) https://waf.io/
jeeves, on the other hand, does no such thing. It is a command runner: you write a function → it is converted into a shell command.
I believe jeeves is easier to get started with, partially due to its simplicity and partially because it follows Python standards. For instance, waf uses a peculiar syntax to define command parameters:
waf employs a peculiar method to configure command options:
while jeeves relies upon function arguments and type hints (as it is based upon Typer): While jeeves is easy to start with it also promises ability to scale: with modular packages for commands, subcommands, and installable/shareable plugins.If something is marketed as a make alternative, most people would expect it to, at the very least, possess make's core feature, which is skipping targets that have already been built and whose inputs haven't changed. That's what makes something a build system and not just a command runner.
I say this as somebody that built their own python replacement for make long ago - during the building of which I learned what make was actually for, and how to use make, and thus abandoned my own project.
[1]: https://github.com/llimllib/pub
[0]: https://docs.python.org/3/library/argparse.html#quick-links-...
my.pdf: my.ps
<tab> command to convert my.pdf to my.ps
Make makes no attempt to generate scripts, or anything like that. In general, make uses file timestamps to determine that my.pdf is out of date with respect to my.ps Make does have default commands that it can use. For example, it knows that executable w can be generated from w.c If I have a directory that contains w.c I can:
make w
cc w.c -o w
Now, because w exists, and is newer than w.c doing make w again does nothing!
make w
make: 'w' is up to date
I don't think that is what jeeves is?
However, dealing with Python mostly in my time in the industry, I scarcely ever had the need to use these particular features of Make. I used it as a task runner, guilty of that, — and that's what I wanted to reimplement in Python.
Even writing in Rust, everything I have to do is `cargo build`, — there is no need to write Makefiles for building the project.
I will consider changing that wording to set proper expectations, but I still feel that for me "a Pythonic alternative to Make" perfectly conveys the use case I daily employ this thing for.
Instead of `make` as entry point for all project-specific commands (like `make lint`, `make build`, `make deploy`), users can rely upon `j` — both locally and CI.
Regarding your particular points:
* dependency graphs aren't presently at scope, I haven't yet come up with the method of expressing them in Python which would have entirely satisfied me;
* I am not so sure I would want to implement pattern rules due to complexity they bring about. Maybe just writing explicit Python code would be enough for cases where they're used. No strong opinion though.
1. Databases. We ended up calling `psql -f some_script.sql && touch $my_task_name` and tracking changes with touch files. (Putting this into database on views or materialized views proved to be unsustainable.)
2. Datasets. If you just open a sqlite file, it's changed, and GNU Make thinks you must rebuild everything downstream. Datasets are mostly treated as row-order-independent, so hashing them as is does not always work.
3. Very expensive tasks that shouldn't be called always. Like my makefile had a script that parsed a million web pages, going around captchas via Tor, and touching the upstream files was to be avoided -- or if it happened, I had to manually touch the target, to avoid re-running that part.
4. Some targets can be updated and the result will always be new -- e.g. run a query to a live database, or news website. Some may produce the same. Would appreciate if any system has such a distinction.
5. Surprisingly, lots of alternative build systems don't do partial update. They only update every item in the deps graph.
If you manage to get any of these right, you'd be praised.
the rough essence of make is arguably:
1. there are targets, and recipes (written as a shell command) to derive the target 2. targets can express dependencies on other targets 3. targets are files in the filesystem. 4. add a bunch of language features on top (pattern rules, automatic variables, built-in functions, etc) to make it easier to write and maintain the recipes
Another way of thinking about make is that it is a a way to define and evaluate gigantic pure-functional expressions, where each input and output file is a file, and where intermediate sub-expressions can be cached.
for an alternative to make that supports 1+2+3+4, see e.g. https://gittup.org/tup/index.html
for an alternative to make that supports 1+2+3 but explicitly does not aim to support 4, see e.g. https://ninja-build.org/manual.html
another alternative to make that supports 1+2+3+4 but de-emphasizes 3, "targets are files", from the ruby community, is https://github.com/ruby/rake
these days in projects it is quite common to see make used to do 1 but not 2, 3 nor 4. i.e. a makefile as a way to define a bunch of imperative shell recipes for named targets, where each named target has no relationship to a file, and dependencies between the targets aren't expressed or relevant. as a random example of this, here's the first google result searching for "makefile terraform":
https://github.com/paulRbr/terraform-makefile/blob/master/Ma...
note all the targets are ".PHONY" targets, we're telling make to disable the default behaviour where every target corresponds to a file in the file system -- to turn off "3 targets are files in the filesystem". there's very sparing use of dependencies. so this is an example of using makes support for 1 defining recipes to produce targets and a little splash of 2 dependencies between targets, but not 3 nor 4.
it looks like this "jeeves" project is aiming at this latter use case, not attempting to implement support for 2+3, the core of how make is typically used as a build tool
5. allow partial rebuilding of targets
When I checked some build systems several year ago, many turned out to re-build just everything.
Make is ugly, but it's reasonably clean "algebraically". I find Python very difficult to reason about, and ugly — it's a language that doesn't seem to have been well designed in any grand sense.
* Fabric: https://www.fabfile.org/
* Invoke: https://www.pyinvoke.org/faq.html
v1 was so much more simple and elegant.
- no type hints for docs & validation, I wanted them
- Makefile in its basic form is very concise and doesn't require a @task decorator, I didn't want it either
I didn't need dependency graph much.
So you made a Make alternative that isn't a Make alternative because you never needed Make in the first place?
- via direct function calls in Python code,
- or using Typer callback functions
…is easy.
There are potential approaches worth exploring (annotations, decorators, …). I might come up with a list of options with syntax examples and ask for community's thoughts about it, — but at this point jeeves, as an MVP, is already useful for me and a few people I work with, and that motivated me to share it with the community.
Those are really bad defaults in general, which is why I recommend everyone to avoid this library. The tty on stdout means many programs run in "interactive" rather then "batch" mode: programs which use pager get output truncated, auto-colors may get enabled and emit ESC controls into output streams (or not, depending on user's distro... fun!)
But it is _especially_ bad for general tools runner. Because of forced capture, you are not going to get any output or errors until process completes, and maybe not even then. Also interactive prompts won't work, errors would be reordered compared to regular outputs and so on...
I can't say anything about jeeves, but at least avoid "sh".
Btw does rich scrape the special formatting characters if piped?
You can specify `_out=rich.print` and it will work but AFAIK it won't scrape ASCII terminal formatting characters.
I had to fix these characters in my `sh` based scripts but do not consider that as a big deal.
Does not seem very ergonimic to me.
- command actually executed,
- snippet of stdout
- and snippet of stderr.
They can be handled using standard exception techniques.
In addition stdout can be redirected to a file, or to another command, or to a callable, — which will be called for chunks of stdout while the command is in operation.
Transparent SSH interop is why I stick to Bash rather than moving to a Python solution.
The way I'd recommend to run shell commands from jeeves is `sh` library. It has features for ssh support: https://sh.readthedocs.io/en/latest/sections/contrib.html#ss...
I think Earthly has the right idea with an Earthfile. https://docs.earthly.dev/docs/earthfile
It's a different tool for running commands, though.
I don't believe Make has plugins.
I'm not sure whether it's suggested to install it globally, as a development dependency with poetry/similar, or with pipx https://pypa.github.io/pipx/
The "import sh" thing could have some users installing this package https://pypi.org/project/sh/
This in addition to it being known as "j". It has at least 3 names, jeeves, j, and sh.
* jeeves is the name of the project which I'm advertising, which converts a Python file to a set of commands; * `j` is the name of executable which aformentioned project exposes; * and `sh` is the library (the correct link to which you have provided) which is an optional dependency of `jeeves` and provides a more convenient interface to calling processes and executing commands from Python than `subprocess.run()`.
It would be worth to see how they tackles some of the challenges you're looking into.
Blurb from the website:
SCons is an Open Source software construction tool. Think of SCons as an improved, cross-platform substitute for the classic Make utility with integrated functionality similar to autoconf/automake and compiler caches such as ccache. In short, SCons is an easier, more reliable and faster way to build software.
SCons is not idiomatic Python and it abuses things like `eval` which gives it terrible performance.
Source: I used to work for MongoDB and my full time job was to make SCons faster, which I eventually did by making it a Ninja generator (which has now been upstreamed). But the code is still pretty bad.
Using SCons however is much nicer than using make / autoconf IMO, especially now that you can farm the builds out to Ninja.
And nice work making SCons faster!! not at easy thing to do at all.
In some ways this is similar to a project I've been working on for the last month that is kind of a pythonic reimagining of Ansible and Cookiecutter. It's still a WIP, but I'm starting to successfully convert my YAML-style declarative snippet/cookiecutters into Python, and I'm working on improving the documentation now.
Where Jeeves seems to excel at collections of small "scriptlets", uPlaybook is targeting declarative automation of larger tasks, including templating of configuration files, triggering restarts on changes, etc...
https://github.com/linsomniac/uplaybook
The point of "make" was supposed to be that it defined a dependency graph, then executed only the commands required to achieve the goals. Somewhere that got lost.
Jeeves seems to be a thing where you define python functions and then Jeeves makes it so you can run them as 'subcommands' of the 'j' command line program, and to take arguments, etc. Seems like it'd be up to you to write your own code to handle figuring out if each function/subcommand has dependencies, and if those dependencies need to be run or built.
Neat project, but has nothing to do with make. Sure, some people may use make as a command runner, but that doesn't mean this is a make replacement. At least not for the vast majority of things that people use make for.
* gnu or otherwise, and why specify that in the first place anyway? If it did actually do a suitable job of replacing gnu make, would it not also suitably replace bsd or any other make? Is a person who says something like that a good source of ideas so deep and quality that they probably do improve upon those who created unix?
make may not be the final build system for the rest of eternity, but all this is, is "I don't really understand the full job make does, so here is something which only does a simpler job which I do understand, and I like python."
> did actually do a suitable job of replacing gnu make
The title does not say _a replacement_, it says _an alternative_. I never intended to replace GNU Make; if I said that anywhere — that was a mistake on my part.
> do improve upon those who created unix
I did not intend to improve upon UNIX ideas or philosophy. The improvements I am aiming for are:
- developer experience,
- conciseness and maintainability of the code.
…and these are being addressed for a narrow use case. For that use case, in my experience, Make is very often used.
I argue that within the bounds of this use case, this is an alternative which can be of use to improve productivity and everyday experience.
> I don't really understand the full job make does, so here is something which only does a simpler job which I do understand, and I like python
I classify these assumptions about my understanding or misunderstanding as ad hominem and unprofessional.
In other words, make(1) does a lot of things, one of which is to run shell commands. If this the only thing you care about and your project offers, it would be better expressed as "Pythonic way to run commands" or something, with no mention of make.
Makefile keeps dependency graph. I had a 100-entry 300-line Makefile, with graphviz drawing charts of it, and kept a huge, year-long project on my own, organized and running all partial updates smoothly.
[1] https://github.com/ekimekim/yaargh
> just `python your_file.py`
That's actually one of the points.
is shorter than and the letter `j` is what the index finger of your right hand is pointing to on the keyboard. There is usually a tactile bump on that key. It is something you can type very quickly. You can also install shell completion to improve the experience further.Ergonomics matters.
I'll agree this is some improvement in ergonomics if the executable supports tab completion.
With makefile + argh I can do this: put the script as a dependency of data file, and rebuild it if code changes.
With your package, this becomes harder -- the file name and command name now are separated and must be checked for being in sync.So, the package adds some features, but does it at some cost of other ways of interaction.
- Is putting a symlink to one of your project directories, probably residing in $HOME, to /usr/bin/ a good practice?
- As an alternative one can put a symlink into ~/bin; but what if you have multiple different projects?
- And they can have different jeeves commands and different plugins installed. For instance, `j lint` implementations in different projects are work on are completely different.
Because these are different projects.
Rather, I prefer to have jeeves automatically create the executable command in each virtual env, and its behavior in different virtual envs will be different.
Why not use native Python methods, namely endpoints, instead, and automate this away?
Anyway, my point is that your package offers conveniences like those in existing packages, and not free but at a similar cost. (Actually, I also contemplated making my own plugins system for `argh`, but just wrote a single package with my own decorator to handle just the few types I needed: pd.DataFrame and gpd.GeoDataFrame. https://github.com/culebron/erde/ )
Experiences differ, and de gustibus on est disputandum; in my bubble, virtualenv is a must-have for Python development.
I have multiple Python projects, they run different versions of Python itself and its libraries, and for many of them Docker would be an overkill; so it is hard for me to imagine my routine without virtual environments.
I employ pyenv to manage them but venv is a venv.