Some time ago I wrote a pytest plugin[1] that helps me test my long running shell scripts.
The idea was to treat the shell process as a fixture, and conditions before and after the test (input file checksums, output files, generated stdout/stderr, etc.) would be checked. It was mostly for data analysis pipelines (hence the name).
I don't use it as much these days as my days of writing long running pipelines are behind me. But for that time, it did its job pretty ok. It was also a nice way to peek into pytest's internals.
Another commenter mentioned Cram, which I would look into if checking stdout/stderr (mostly) is enough for my use case.
If I am already using a specific framework to write the pipelines, I would also look into whether it already provides ways for testing.
I am not aware of other tools that test for pre-/post- conditions, so I would probably start with mine and see if need to tweak things along the way.
One thing I am not too happy about the plugin is that sometimes setting up the process as a fixture can be quite verbose. It feels like this part can be improved with some more thought.
Note that this recommends "sh" package, which made an _interesting_ decision to run processes in tty by default. This goes against unix conventions that batch processes should not be using tty, and breaks a number of utilities like git and systemctl. This is so bad that even their example from the front page is broken(!):
$ python3
>>> import sh
>>> sh.git.show('HEAD').count('\n')
49
# Now resize your terminal window to be 10 lines tall
>>> sh.git.show('HEAD').count('\n')
9
Yep, it truncated "git show" output to the height of your terminal window. Do you want this in your code?
While you can append special flag for this, it is very easy to forget to do so, and the documentation does not help either. Their FAQ states that they are aware of the issue and won't fix it [0].
So my advice is: stay away from "sh" package, as it makes it very simple to generate unreliable code. Instead, use built-in tools which are pretty nice and just a bit more verbose:
>>> import subprocess
>>> subprocess.check_output('git show HEAD', shell=True).decode().count('\n')
49
it's not "just a bit more verbose", subprocess is much longer and complex module than sh! i constantly need to go back to the manual (is it check_output or call? or maybe i need to create a pipe for that one? never remember).
sh provides clear semantics for certain use cases - that it doesn't match your expectations on how a program shold behave is understandable, but there are knobs to change that behavior, so I don't know if that is a strong enough argument to completely "stay away from" it altogether.
i know i've been happy it's there and used it in a few projects with good success, particularly projects that need exactly that: replace a shell and have the terminal passed down correctly.
It is obviously each person's preference, but I especially hate non-reproducible, context dependent bugs. I think having a reproducible, consistent runs is key to having nice software that everyone enjoys developing.
In this light, I find the tty decision particularly bad, as it is very prone to introducing subtle changes -- for example, do you need the special flag for "git log"? what about "git for-each-ref"? what about "ls"? what about "gcc"?
For this reason, I find it way easier to just ban "sh" outright altogether. Our organization policy say "do not use sh module". When new member joins, and they way to shell stuff out, they will find out that they need to use subprocess module, grumble a bit, search the web, but then produce a working, if verbose, code.
An alternative would be to say: You can use sh, but never use it as sh's own front page recommends. If you find an article on the web which mentions sh, be aware that you cannot just copy-paste code from it, as it can be broken depending on which command you run. You need to audit which command you run to see if it cares about tty. Currently, you have to include the flag for many (but not all) git subcommands, "ls", "apt", ... but that list can change at anytime.
> i know i've been happy it's there and used it in a few projects with good success, particularly projects that need exactly that: replace a shell and have the terminal passed down correctly.
You have been lucky so far, but every time you use "sh" you are walking on landmines. I can give you a ton of plausible examples where sh breaks. Here is a shell replacement gone wrong:
$ git show HEAD | wc
49 273 2202
$ python3 -c 'import sh; print(sh.wc(sh.git.show("HEAD")))'
19 97 812
edit: have you seen "subprocess.run" in python 3.5 [0]? It gives nice all-in-one API, with readable options like `check=True' or `capture_stdout=True`
Maybe I am off here, but I do something a bit more simple
```
def issue_command(self, command):
if not str(command):
response = None
logger.error('Command argument must be string')
else:
response = os.system(command)
if response != 0:
logger.error('Command returned non-zero status: {}'.format(command))
exit(1)
return response
We did a fair bit of this kind of thing at my last company.
More specifically, we were using the PBT-like `hypothesis` framework in Python for the purposes of more exhaustively exercising the CLI tools we were creating.
18 comments
[ 3.4 ms ] story [ 51.4 ms ] threadThe idea was to treat the shell process as a fixture, and conditions before and after the test (input file checksums, output files, generated stdout/stderr, etc.) would be checked. It was mostly for data analysis pipelines (hence the name).
I don't use it as much these days as my days of writing long running pipelines are behind me. But for that time, it did its job pretty ok. It was also a nice way to peek into pytest's internals.
[1] https://github.com/bow/pytest-pipeline
Another commenter mentioned Cram, which I would look into if checking stdout/stderr (mostly) is enough for my use case.
If I am already using a specific framework to write the pipelines, I would also look into whether it already provides ways for testing.
I am not aware of other tools that test for pre-/post- conditions, so I would probably start with mine and see if need to tweak things along the way.
One thing I am not too happy about the plugin is that sometimes setting up the process as a fixture can be quite verbose. It feels like this part can be improved with some more thought.
https://bitheap.org/cram/
You can define in a small test file what is the shell code to execute and what is the expected result, including error codes
I like sh very much in Python to generate scripts, but in terms of pure testing shell commands, I think cram is brilliant.
Here is an example on tests for a shell utility I created: https://github.com/jaimebuelta/ffind/tree/master/tests
While you can append special flag for this, it is very easy to forget to do so, and the documentation does not help either. Their FAQ states that they are aware of the issue and won't fix it [0].
So my advice is: stay away from "sh" package, as it makes it very simple to generate unreliable code. Instead, use built-in tools which are pretty nice and just a bit more verbose:
[0] https://amoffat.github.io/sh/sections/faq.html#why-is-tty-ou...sh provides clear semantics for certain use cases - that it doesn't match your expectations on how a program shold behave is understandable, but there are knobs to change that behavior, so I don't know if that is a strong enough argument to completely "stay away from" it altogether.
i know i've been happy it's there and used it in a few projects with good success, particularly projects that need exactly that: replace a shell and have the terminal passed down correctly.
In this light, I find the tty decision particularly bad, as it is very prone to introducing subtle changes -- for example, do you need the special flag for "git log"? what about "git for-each-ref"? what about "ls"? what about "gcc"?
For this reason, I find it way easier to just ban "sh" outright altogether. Our organization policy say "do not use sh module". When new member joins, and they way to shell stuff out, they will find out that they need to use subprocess module, grumble a bit, search the web, but then produce a working, if verbose, code.
An alternative would be to say: You can use sh, but never use it as sh's own front page recommends. If you find an article on the web which mentions sh, be aware that you cannot just copy-paste code from it, as it can be broken depending on which command you run. You need to audit which command you run to see if it cares about tty. Currently, you have to include the flag for many (but not all) git subcommands, "ls", "apt", ... but that list can change at anytime.
> i know i've been happy it's there and used it in a few projects with good success, particularly projects that need exactly that: replace a shell and have the terminal passed down correctly.
You have been lucky so far, but every time you use "sh" you are walking on landmines. I can give you a ton of plausible examples where sh breaks. Here is a shell replacement gone wrong:
edit: have you seen "subprocess.run" in python 3.5 [0]? It gives nice all-in-one API, with readable options like `check=True' or `capture_stdout=True`[0] https://docs.python.org/3/library/subprocess.html#subprocess...
https://plumbum.readthedocs.io/en/latest/
``` def issue_command(self, command):
```More specifically, we were using the PBT-like `hypothesis` framework in Python for the purposes of more exhaustively exercising the CLI tools we were creating.
It worked pretty well.