I’ve been working on Makefiles lately and its funny how many compatibility issues there are. I was doing Sed substitutions and found I needed to change the -i flag to make it work between Mac and Linux. I was really hoping for ultimate portability but ended up learning lots of hacks that lead to portability
But it is a good example of something that you might expect to be more standard than it is between platforms, because of its age and relative ubiquity, where the differences get highlighted when trying to create (not just with make but other tools too) a consistent build process for a project.
Yeah, variance between different system utilities is one of the biggest underrated perils of shell scripting. For a long time, I worked on a team that mostly used Macs to administer Linux servers, which led to some confusing issues when people tested certain commands locally and then tried to use them on a server.
It would be nice if shellcheck had checks to detect common BSD/GNU incompatibilities and warn about them so you don't have to catch them in the moment.
But it is a bug in your Makefile to assume a particular version of a tool (or path to a tool, etc) without first checking (or otherwise forcing the correct thing to be used).
For forcing the existence of tools, you can start your Makefile by setting up $PATH with the dependencies your script has (everything not guaranteed by POSIX, for example). E.g. with Nix
Will cause `make` to include DEPENDENCY1 and DEPENDENCY2 from the Nix store in $PATH for the rest of the Makefile. You can get more complex and pin particular versions where it matters.
Other cross-platform package management tools presumably have similar tricks that can be used, or you can bundle binaries for the tools you need. Then your build instructions only need to tell the user to have Make and the package management tool installed, and everything else can be set up in the Makefile.
That's same as `sed -i 's/foo/FOO/' ip.txt` on GNU sed.
Question is about inplace editing command that works on both GNU and MacOS (without needing a backup, with backup `-i.bkp` works on both). I don't have a Mac to check if `sed -e 's/foo/FOO/' -i 'ip.txt'` works there.
Sure it does, all the standard calls are universal, hell I wrote a _universal_ directory monitoring shell script. Try and do that with bash on windows/linux/osx.
If you're going to use PowerShell like a shell (i.e., to call external programs), you still have to worry about distributing them or variations between what's preinstalled.
If you're only going to use PowerShell functions and not external programs anyway, then PowerShell is more in competition with languages like Python, Perl, Ruby, etc., than it is with Bash.
PowerShell also falls down when it comes to the other distinguishing feature of a shell, which is that you gradually learn your scripting language just by using your computer, because PowerShell just falls flat as a daily login shell. This is mainly because it's unbelievably dog slow for a shell and lacks basic functionality like job control, but arguably also because of smaller things like its verbosity, extremely complicated profile sourcing behavior, etc.
PowerShell is still sometimes a good choice for cross-platform scripting, but I wouldn't say that's because of its compatibility.
Indeed. Also, scripts (especially Dockerfiles) which run `apt-get install foo bar baz`; usually mixed in with a bunch of other commands, and file copying/editing. That's literally what package formats like .deb, .rpm, etc. are for! For example:
- Write your dependencies in a `my-package/DEBIAN/control` file
- If you want any extra files, put those in your `my-package/` folder too (e.g. `my-package/etc/some_config_file`, `my-package/usr/bin/some_executable`, etc.)
- If you want to run other commands, you can write a `my-package/DEBIAN/preinst` and/or `my-package/DEBIAN/postinst` script
- Run `dpkg-deb --build` to make your .deb file
Voila! Using a declarative format gives us an uninstaller "for free", lets us query which version is installed, and the installation will notify us of any conflicting files. Packages also tend to be more robust across different machines/environments, and don't drag in a whole extra OS plus VM/container-manager/etc.
More features are unlocked with a little extra specification, e.g. package conflicts (to avoid known-bad setups); dependency alternatives (e.g. prefer Caddy, accept Nginx, fall back to Apache); etc. I'm sure there are more quick-wins too, e.g. Googling gives me https://www.internalpointers.com/post/build-binary-deb-packa...
Disclaimer: Whilst I used to use Debian heavily (even on my phone!) I switched to Nix/NixOS about a decade ago, which I find even better. It's just infuriating to see the industry ignoring well crafted solutions, which have been around for decades, in favour of half-baked anti-patterns like Dockerfiles. Especially when the latter is often just a layer of crap on top of the former (like running `apt-get` in a script; rather than what it's designed for!)
The main issue when it comes to using Debian or most conventional package managers (nix really deserves its own mention there) is that you also pretty much distro lock your software, which may not be always beneficial. Another problem comes now in the form of distribution; packaging is one thing, distributing is another. Debian isn't too bad about this but depending on your choice of packager[0], you could suddenly need to spend another couple days trying to figure out how to even host the thing unless you want to spend time running wget on your production machines to pull in deb/RPM/makepkg output/pick your poison files.
Nix deserves its own mention because while it pretty much singlehandedly tries to solve all major issues with dependency management, it also... does so by abandoning almost every assumption most people have about how the OS is supposed to work (not to mention that writing nix files is basically learning a whole new language which is... doable but not exactly accessible). It's definitely the stronger solution compared to dockerfiles, but you can translate any program for any distro into a dockerfile as long as it runs on your kernel (alongside the fact that a dockerfile is a really good way to deal with the "documentation abandoned 5 years ago, but it runs on Dave's machine if you follow the README and change about 50 different small settings in your OS" projects which are done a dozen with FOSS projects).
Docker is a mess technically but when it comes to software you want to use/deploy but don't want to understand all the fine details of the source code, it is often a lot easier to work with compared to Nix, which if you have something not in nixpkgs, can result in you needing to start making upstream patches to genericize build directories and generally change peoples CI flows in ways maintainers don't necessarily appreciate.
tldr; nix is good, docker is easy.
[0] It's been a few years but the worst package distribution system I've had the displeasure to use are Ubuntu PPAs. It's a system close to the Debian specification but it requires a whole set of separate commands and weird signing steps before you can upload anything, none of which is properly documented because they expect people that want to use PPAs to shove it in a makefile, which isn't useful if you don't use a language that relies on Makefiles, but I digress.
I agree with most of what you're saying, although we're describing slightly different use cases.
For example, using a .deb file is just as "distro locked" as the sort of "apt-get scripting" I was complaining about. It's pretty much a Pareto improvement to change a Dockerfile from `RUN apt-get install ...` to `RUN apt install ./my-package.deb` (we can construct the .deb wherever we like; outside the container, inside the container, in a separate container, etc.)
As for Docker being "easy"... I suppose that's subjective. Personally, it's given me nothing but pain; from trying to get the damn thing installed, to trying to understand it (lots of documentation turns out to be obsolete, and lots more assumes you're already familiar with its bizarro-world of not-invented-here concoctions).
I gave up trying to understand Docker itself. I learned much more by following the OCI specs, and building container images directly using `tar` and `sha256`. AWS ECS seems to run them just fine :)
That tool is really good to know about. I despise `sudo make install` because it can drop stuff anywhere, making it difficult to perform the inverse operation. You also always run the risk of conflicts with other applications. A package manager at best might bail out if it realizes it's going to clobber something it doesn't expect, but `sudo make install` on a different project is going to happily overwrite whatever it wants.
> I despise `sudo make install` because it can drop stuff anywhere, making it difficult to perform the inverse operation.
It can, but I can't recall seeing it install anything outside of $PREFIX in my usage. At least not in recent memory. Of course it depends on how the Makefile was written, but most things I deal with use GNU autotools which makes it easy to specify where things go in general. Also a number of Makefiles come with an uninstall target which can be handy.
I mostly agree (I was working with sudo make install in order to package neatroff). However, most of the build scripts which generate binary packages for distros etc use `make install`, usually with DESTDIR set, so it's not irrelevant.
I also think `sudo make install` should at least work as expected in this one case, especially because this project isn't widely packaged so most users will be building from source.
> In my opinion, the best solution that POSIX.1-202x should use is to require make to set the PWD macro itself. The whole SCCS stuff in POSIX already requires make to think about PWD, so I don't see why this shouldn't happen
It should't happen because make isn't the problem, it's sudo not setting PWD to what you expect by default. This isn't a make bug, it's a sudo bug. Author even explicitly says this. They just prefer to modify make, for.... reasons.
It can.... get it from the environment.... like every application run by every operating system does. PWD is only ever set by a shell. It exports it as an environment variable. Programs can read environment variables passed to them.
As you noted, I said that the problems stems from sudo not setting PWD, hence `sudo make` doesn't inherit PWD. I'm not calling that a bug in sudo, but it was a bug in the specific makefile which assumed PWD to be set always.
A good solution to the problem is for make to set the PWD macro. BSD make already does this.
> It can get if from the environment like every application does
Well no, applications either read an environment variable or use getcwd(2).
37 comments
[ 19.7 ms ] story [ 90.0 ms ] threadBut it is a good example of something that you might expect to be more standard than it is between platforms, because of its age and relative ubiquity, where the differences get highlighted when trying to create (not just with make but other tools too) a consistent build process for a project.
It would be nice if shellcheck had checks to detect common BSD/GNU incompatibilities and warn about them so you don't have to catch them in the moment.
For forcing the existence of tools, you can start your Makefile by setting up $PATH with the dependencies your script has (everything not guaranteed by POSIX, for example). E.g. with Nix
`export PATH := $(shell nix-shell -p DEPENDENCY1 DEPENDENCY2 --run 'echo $$PATH')`
Will cause `make` to include DEPENDENCY1 and DEPENDENCY2 from the Nix store in $PATH for the rest of the Makefile. You can get more complex and pin particular versions where it matters.
Other cross-platform package management tools presumably have similar tricks that can be used, or you can bundle binaries for the tools you need. Then your build instructions only need to tell the user to have Make and the package management tool installed, and everything else can be set up in the Makefile.
Of course this is a PITA.
https://pubs.opengroup.org/onlinepubs/9699919799/
https://pubs.opengroup.org/onlinepubs/9699919799/
Question is about inplace editing command that works on both GNU and MacOS (without needing a backup, with backup `-i.bkp` works on both). I don't have a Mac to check if `sed -e 's/foo/FOO/' -i 'ip.txt'` works there.
That might be a good Show HN example.
If you're only going to use PowerShell functions and not external programs anyway, then PowerShell is more in competition with languages like Python, Perl, Ruby, etc., than it is with Bash.
PowerShell also falls down when it comes to the other distinguishing feature of a shell, which is that you gradually learn your scripting language just by using your computer, because PowerShell just falls flat as a daily login shell. This is mainly because it's unbelievably dog slow for a shell and lacks basic functionality like job control, but arguably also because of smaller things like its verbosity, extremely complicated profile sourcing behavior, etc.
PowerShell is still sometimes a good choice for cross-platform scripting, but I wouldn't say that's because of its compatibility.
Tools like checkinstall are the way to go, at a minimum.
- Write your dependencies in a `my-package/DEBIAN/control` file
- If you want any extra files, put those in your `my-package/` folder too (e.g. `my-package/etc/some_config_file`, `my-package/usr/bin/some_executable`, etc.)
- If you want to run other commands, you can write a `my-package/DEBIAN/preinst` and/or `my-package/DEBIAN/postinst` script
- Run `dpkg-deb --build` to make your .deb file
Voila! Using a declarative format gives us an uninstaller "for free", lets us query which version is installed, and the installation will notify us of any conflicting files. Packages also tend to be more robust across different machines/environments, and don't drag in a whole extra OS plus VM/container-manager/etc.
More features are unlocked with a little extra specification, e.g. package conflicts (to avoid known-bad setups); dependency alternatives (e.g. prefer Caddy, accept Nginx, fall back to Apache); etc. I'm sure there are more quick-wins too, e.g. Googling gives me https://www.internalpointers.com/post/build-binary-deb-packa...
Disclaimer: Whilst I used to use Debian heavily (even on my phone!) I switched to Nix/NixOS about a decade ago, which I find even better. It's just infuriating to see the industry ignoring well crafted solutions, which have been around for decades, in favour of half-baked anti-patterns like Dockerfiles. Especially when the latter is often just a layer of crap on top of the former (like running `apt-get` in a script; rather than what it's designed for!)
Nix deserves its own mention because while it pretty much singlehandedly tries to solve all major issues with dependency management, it also... does so by abandoning almost every assumption most people have about how the OS is supposed to work (not to mention that writing nix files is basically learning a whole new language which is... doable but not exactly accessible). It's definitely the stronger solution compared to dockerfiles, but you can translate any program for any distro into a dockerfile as long as it runs on your kernel (alongside the fact that a dockerfile is a really good way to deal with the "documentation abandoned 5 years ago, but it runs on Dave's machine if you follow the README and change about 50 different small settings in your OS" projects which are done a dozen with FOSS projects).
Docker is a mess technically but when it comes to software you want to use/deploy but don't want to understand all the fine details of the source code, it is often a lot easier to work with compared to Nix, which if you have something not in nixpkgs, can result in you needing to start making upstream patches to genericize build directories and generally change peoples CI flows in ways maintainers don't necessarily appreciate.
tldr; nix is good, docker is easy.
[0] It's been a few years but the worst package distribution system I've had the displeasure to use are Ubuntu PPAs. It's a system close to the Debian specification but it requires a whole set of separate commands and weird signing steps before you can upload anything, none of which is properly documented because they expect people that want to use PPAs to shove it in a makefile, which isn't useful if you don't use a language that relies on Makefiles, but I digress.
For example, using a .deb file is just as "distro locked" as the sort of "apt-get scripting" I was complaining about. It's pretty much a Pareto improvement to change a Dockerfile from `RUN apt-get install ...` to `RUN apt install ./my-package.deb` (we can construct the .deb wherever we like; outside the container, inside the container, in a separate container, etc.)
As for Docker being "easy"... I suppose that's subjective. Personally, it's given me nothing but pain; from trying to get the damn thing installed, to trying to understand it (lots of documentation turns out to be obsolete, and lots more assumes you're already familiar with its bizarro-world of not-invented-here concoctions).
I gave up trying to understand Docker itself. I learned much more by following the OCI specs, and building container images directly using `tar` and `sha256`. AWS ECS seems to run them just fine :)
That said, some things have a install-rpm or similar target in Makefile which can be handy.
It can, but I can't recall seeing it install anything outside of $PREFIX in my usage. At least not in recent memory. Of course it depends on how the Makefile was written, but most things I deal with use GNU autotools which makes it easy to specify where things go in general. Also a number of Makefiles come with an uninstall target which can be handy.
I also think `sudo make install` should at least work as expected in this one case, especially because this project isn't widely packaged so most users will be building from source.
[0] https://porg.sourceforge.net/
It should't happen because make isn't the problem, it's sudo not setting PWD to what you expect by default. This isn't a make bug, it's a sudo bug. Author even explicitly says this. They just prefer to modify make, for.... reasons.
It can.... get it from the environment.... like every application run by every operating system does. PWD is only ever set by a shell. It exports it as an environment variable. Programs can read environment variables passed to them.
A good solution to the problem is for make to set the PWD macro. BSD make already does this.
> It can get if from the environment like every application does
Well no, applications either read an environment variable or use getcwd(2).