Is this a standard anywhere outside IPython? I would love to use this sort of thing (python with seamless integration with a system shell) in place of bash scripts. Yes, you can use subprocess but it's a lot less readable.
The bang syntax is specific to ipython. I'm not sure it's really desirable outside of that context. It lacks a lot of process control features you'd want in order to make things robust for general purpose scripting purposes.
You might like Perl. The syntax to invoke and interact with processes is much more terse than Python's. And there's quite a lot of history of using it instead of shell scripts.
These days, Perl has 3 argument open, and can use a variable for a filehandle. So:
open(my $pipeline, "-|", "ls -l");
This is a good thing, because you can treat a filehandle as a standard variable, and can separate the special stuff from the command itself. This way if you're taking in user input, they don't get to mess with that. You can also do:
open(my $pipeline, "-|", "ls", "-l", $directory);
Which lets you get away from the mess of correct quoting and just pass every parameter exactly as intended.
While I haven't personally used it, another commenter mentioned Xonsh[0] and that is what it appears to be: A very smooth integration of python into a shell that still feels like a unix shell.
It depends, for other python REPL/Env not really but it is the standard way to call shell command from ed [1] for example. Julia have also a shell mode in its REPL [2]. Personally, I like to use Raku [3] instead of Python or Shell script. It make it easy to drop one-liner, call shell commands [4] and you can super easily make a script a small CLI utilities by writing a MAIN sub [5].
xonsh is absolutely the right way to go here. There are a lot of 'gotchas' when it comes to building a shell that isn't 'just' a novelty but actually usable.
Shells are most successful when you don't need to think in order to use them. Especially since everyone that's used linux in the past 30(?!) years has the same basic foundation for how to use the command line, the best Shell will feel familiar to sh and bash, but better.
Math, logic, string and value manipulation, all those things I need to google because I don't remember whether an if statement in bash needs single [ ] or double [[ ]] or '' or "" or ; or spaces or all those weird gotchas.
I understand that many people do know how to program in bash effectively. That doesn't mean it's the future. It's like the Perl vs Python thread from the other day.
Xonsh is intuitive. Xonsh is well designed. Long live Xonsh.
It checks if the variables 'ls' and 'la' exist, and if they do, it'll run it as python. If not, it'll try to parse it as bash. If it can't parse it as bash, it'll try again in python mode.
There are edge cases but IME I rarely came across commands that could be valid python and valid bash. And you'll keep those in mind when you're running things so that you won't ever name a variable after a bash command you might use.
I ran it for a couple of years while I was in school, it worked great. I wouldn't run it on a production server, this is really for experimental work
Holy crap that was my exact reaction to reading that. It’s just like bash: 99% of the time it’s an innocent thing, until suddenly you source some script that redefines something, innocently thinking it shouldn’t be a problem. That’s how you get an rm -rf.
Yeah you wouldn't source a script written for a different shell in this language, if that's a part of your workflow you'll want to either audit them first or you'll just want to drop into a bash shell.
I'm not sure how you end up rm -rf ing anything. 'rm -rf' by itself doesn't do anything, and if you do 'rm -rf .' that's no longer valid python. Not to mention that most people writing python would write 'rm - rf' if for some reason they had variables with those names (why would you name it that?) that they needed to subtract. And for some reason, you're not doing anything with the result of that subtraction.
If you're sourcing scripts without reading them, the source of the danger is that behavior, not your shell
> most people writing python would write 'rm - rf' if for some reason they had variables with those names (why would you name it that?) that they needed to subtract.
I can see myself naming variables that way, when doing a geometric computation. Being a mathematician, all my variable names are single letters. But sometimes I have variants of a name that merit a second letter, like a sub-index. Maybe I use "r" for the current radius of an object and "rm" for the "middle" radius and "rf" for the "final" radius after some polishing process. Then I want to see how much material I'm losing from the object in the middle. Thus "rm -rf" (yes, the missing space is a typo).
You need two typos in that you need to forget that space, and you also need to have forgotten to set one of those variables. You also don't actually delete anything until you name a file, so you'll need to add more valid python that's also a valid filename
That's fine. But simple things that should work in order to be painless, don't.
--
--
bash:~]$ if $var < 21; then echo "smaller"; fi
bash: 21: No such file or directory
--
--
BUT this works:
if (( $var < 21 )); then echo "smaller"; fi
smaller
--
--
BUT this DOESN'T:
if [[$var < 21 ]]; then echo "smaller"; fi
bash: 21: No such file or directory
--
--
BUT this DOES (added a space after '[['):
if [[ $var < 21 ]]; then echo "smaller"; fi
smaller
Plus the fact that having to type out 'then' and 'fi' and that semi-colons are semi-necessary are annoying. Compared to today's languages, very little of the non-posix parts of bash feel well though out. We can do better.
All these things make it necessary to understand bash. Most modern programming languages feel familiar. Bash and Sh don't anymore. They feel antiquated.
Yeah, having played with some scripting-languages-as-shells, the value that a shell adds over REPL is not insignificant... but haphazardly trying to mash together a scripting language and a shell can produce something like Powershell, which is incredibly powerful and expressive but infuriatingly inconsistent and full of gotchas.
> the value that a shell adds over REPL is not insignificant
Could you expand on this? I've been thinking about using a repl as a system shell. The piping issue is solved easily by sh.py. The main problem I can think of is startup latency.
In general it's the poor ergonomics of calling native command-line tools and manipulating their results that tend to disappoint, but I've mostly played with CSX (c# repl script) as a "repl as command-line" dabbling.
> all those things I need to google because I don't remember whether an if statement in bash needs single [ ] or double [[ ]] or '' or "" or ; or spaces or all those weird gotchas.
This made me chuckle. A few years ago, I went on a learning spree to finally memorize all of these sparse semantics so I could write bash scripts fluently. I ended up having a really good grasp, could write scripts with no flailing.
Six months later, I was back to searching "bash double quotes vs single". Unless you use them every day, for years, these semantics just fall right out of your head.
Last time I tried, it manually "passed through" nano to avoid broken keyboard handling, but you had to add micro, makepkg, sudoedit... to the list, you had to exclude every "top-level binary" that either supplied or invoked a console-based text editor, from xonsh's specialized handling.
The question to me is what gaps exist in using IPython directly versus a more dedicated shell like Xonsh?
I've inadvertently found myself using the IPython interpreter as my shell more and more. Opening vim in IPython always makes me smile. But I don't try to push it's abilities as a shell too much out of a healthy fear that it'll break unexpectedly in the worst way when I absolutely need it not to.
I've never heard of this project before, but as someone from Florida, its name is causing me great distress! Conch is pronounced konk, not kontch. So I'm just going to call this project zonk.
When I said I'd even take lua over python, that's only if I was forced to pick a high level language for a shell. Lua is really easy to conjure up higher level algos, like javascript can.
But I'm not convinced any high level language is a viable replacement for bash. It would have to be more functional than bash whilst still having good support for standard unix tools.
One very interesting idea I found was a macro library for racket that turns s-expressions from deeply nested structures into a much flatter representation. Lisp might be a good option with toy with because of flexibility like that.
I tried this a few years ago when I was younger and more optimistic. But man it was not a great experience.
Having said that I think there is still room for modern shells which are easier to script in. For a while I am using fish and thought scripting would be at least as easy and straightforward as using lua. But the syntax is really awkward. For example comparing two variables needs to be put in a
Used this a while ago and it's ok-ish, but I've also used Powershell and despite it not being as nice of a language as Python (imo) and also can suck in other ways, what is striking for this particular example is that Powershell doesn't need most of the explanation on this page because the topic is basically 'here is how to do grep/sed on plain text in a list of strings which contains file information and hope you extract the correct field' and Powershell did nail that pretty well by piping objects.
Is there a sh to powershell conversion tool? When I run a local environment in Google Colab, I find myself having to write !commands for MS and Linux filesystems. It'd be convenient to write once.
Not really (for a couple of often-used commands like cd and ls there are aliases in powershell), but I also wouldn't now how that would work. You mean to convert the complete syntax?
Babashka has picked up a lot of adherents lately by using a more hybrid approach to this idea. Use clojure to where you want to succinctly and powerfully manipulate data structures (even simple things like (assoc ...) is alien technology in bash) User the shell for everything else.
TempleOS uses (Holy)C as a system shell (and for everything else). It certainly has its limitations at the command line, but on the flip side it makes writing "shell" scripts a whole lot easier!
64 comments
[ 5.6 ms ] story [ 136 ms ] threadIs this a standard anywhere outside IPython? I would love to use this sort of thing (python with seamless integration with a system shell) in place of bash scripts. Yes, you can use subprocess but it's a lot less readable.
Edit: Things like:
And access to child exit status in a way that's familiar ($?), etc.These days, Perl has 3 argument open, and can use a variable for a filehandle. So:
This is a good thing, because you can treat a filehandle as a standard variable, and can separate the special stuff from the command itself. This way if you're taking in user input, they don't get to mess with that. You can also do: Which lets you get away from the mess of correct quoting and just pass every parameter exactly as intended.0. https://xon.sh/
[1] https://www.gnu.org/software/ed/manual/ed_manual.html#Comman...
[2] https://docs.julialang.org/en/v1/stdlib/REPL/#man-shell-mode
[3] https://raku.org/
[4] https://docs.raku.org/language/quoting#index-entry-quote_qqx...
[5] https://docs.raku.org/language/create-cli
https://xon.sh/
Shells are most successful when you don't need to think in order to use them. Especially since everyone that's used linux in the past 30(?!) years has the same basic foundation for how to use the command line, the best Shell will feel familiar to sh and bash, but better.
Math, logic, string and value manipulation, all those things I need to google because I don't remember whether an if statement in bash needs single [ ] or double [[ ]] or '' or "" or ; or spaces or all those weird gotchas.
I understand that many people do know how to program in bash effectively. That doesn't mean it's the future. It's like the Perl vs Python thread from the other day.
Xonsh is intuitive. Xonsh is well designed. Long live Xonsh.
"ls -la"
how does it know if it is "variable ls minus variable la", or "command list with command argument -la"?
https://xon.sh/tutorial.html#python-mode-vs-subprocess-mode
This is horrifying. It can break in so many unexpected and context dependent ways... Do people really run this shell on purpose?
I ran it for a couple of years while I was in school, it worked great. I wouldn't run it on a production server, this is really for experimental work
I'm not sure how you end up rm -rf ing anything. 'rm -rf' by itself doesn't do anything, and if you do 'rm -rf .' that's no longer valid python. Not to mention that most people writing python would write 'rm - rf' if for some reason they had variables with those names (why would you name it that?) that they needed to subtract. And for some reason, you're not doing anything with the result of that subtraction.
If you're sourcing scripts without reading them, the source of the danger is that behavior, not your shell
I can see myself naming variables that way, when doing a geometric computation. Being a mathematician, all my variable names are single letters. But sometimes I have variants of a name that merit a second letter, like a sub-index. Maybe I use "r" for the current radius of an object and "rm" for the "middle" radius and "rf" for the "final" radius after some polishing process. Then I want to see how much material I'm losing from the object in the middle. Thus "rm -rf" (yes, the missing space is a typo).
https://marceltheshell.org
if ls /path ; then echo foo ; fi
works just as well as the sugar wrapping "test" ( [ ] ) or extended builtins.
-- -- bash:~]$ if $var < 21; then echo "smaller"; fi
bash: 21: No such file or directory
-- -- BUT this works:
if (( $var < 21 )); then echo "smaller"; fi
smaller
-- -- BUT this DOESN'T:
if [[$var < 21 ]]; then echo "smaller"; fi
bash: 21: No such file or directory
-- -- BUT this DOES (added a space after '[['):
if [[ $var < 21 ]]; then echo "smaller"; fi
smaller
Plus the fact that having to type out 'then' and 'fi' and that semi-colons are semi-necessary are annoying. Compared to today's languages, very little of the non-posix parts of bash feel well though out. We can do better.
All these things make it necessary to understand bash. Most modern programming languages feel familiar. Bash and Sh don't anymore. They feel antiquated.
One of my biggest annoyances with shell is that there are these magic programs such as [ that pretend to be part of the language but aren't.
If if $var has an exit status (because perhaps you insanely want to have var be some program set elsewhere) your if statement would work....
These other examples have operators (test or similar expanded builtins), and those set the exist status that the if block reads.
so grep -q hacker < /etc/passwd if [ $? -eq 0 ] ; then echo hackd! ; fi
is an indirect way of saying:
if grep -q hacker < /etc/passwd ; then echo hackd! ; fi
Shell is terrible, but the way people use it makes it even worse because the sugar hides what's actually going on.
Could you expand on this? I've been thinking about using a repl as a system shell. The piping issue is solved easily by sh.py. The main problem I can think of is startup latency.
This made me chuckle. A few years ago, I went on a learning spree to finally memorize all of these sparse semantics so I could write bash scripts fluently. I ended up having a really good grasp, could write scripts with no flailing.
Six months later, I was back to searching "bash double quotes vs single". Unless you use them every day, for years, these semantics just fall right out of your head.
I've inadvertently found myself using the IPython interpreter as my shell more and more. Opening vim in IPython always makes me smile. But I don't try to push it's abilities as a shell too much out of a healthy fear that it'll break unexpectedly in the worst way when I absolutely need it not to.
(Otherwise, it looks great!)
[EDIT] Pr updating the docs https://github.com/ipython/ipython/pull/12894
- no pipe/redirection mechanism
- high overhead in version/library control(pyenv, venv)
(All my personal preference, obviously. But I'd rather have any other language than python, even lua)
But I'm not convinced any high level language is a viable replacement for bash. It would have to be more functional than bash whilst still having good support for standard unix tools.
One very interesting idea I found was a macro library for racket that turns s-expressions from deeply nested structures into a much flatter representation. Lisp might be a good option with toy with because of flexibility like that.
https://www.reddit.com/r/Racket/comments/mands9/fluent_unix_...
seriously? lua shell won't even evaluate bare expressions
file_list = !ls
delete_me = b[0]
!rm {delete_me}
Though it can't fit into a single statement due to a greedy '!'
Having said that I think there is still room for modern shells which are easier to script in. For a while I am using fish and thought scripting would be at least as easy and straightforward as using lua. But the syntax is really awkward. For example comparing two variables needs to be put in a
math "2 + 2"
Urg.
pretty nice if you know scala, still have to use regular shell(s) so I do not forget them
I can hear the smalltalkers laugh from afar though
For example, custom history command to make early commands at the top, scroll to the bottom, and show the date and command on the same line
function hist; history -t -R | pyp '["\t".join(pair) for pair in zip(lines[::2],lines[1::2])]' | less +G; end; funcsave hist
[1] https://github.com/hauntsaninja/pyp