Lately I've been paying more attention to alternative CLI tooling, extensions, etc. Our open source community uses git-flow-avh for example and it has been extremely helpful. I can't wait to try this out.
Interesting! I am also looking forward to setting aside time to play around with https://elv.sh/, which takes a completely different approach with a similar philosophy.
This looks great, but what exactly are the benefits over PowerShell? The article mentions PowerShell, then says:
> What if we could take the ideas of a structured shell and make it more functional? What if it worked on Windows, Linux, and macOS? What if it had great error messages?
- Any shell with piping is already very functional, and PowerShell has script blocks (higher-order functions). Or does this mean "functional" in the literal sense? What functionality is missing?
- PowerShell works on macOS, Linux and Windows
- The examples of error messages further down look almost exactly like PowerShells error messages (underlining the command in the pipeline that failed)
It is not clear to me what exactly the authors sought out to do different
I wanted to like PowerShell but I find the syntax choices absolutely exhausting. This looks far more comfortable to me even as someone who mostly uses Windows but also knows Unix shell.
Why not? On your own system you can use whatever shell you want and if you develop shell scripts that you plan to distribute you can just list it as a dependency.
I know the long cased commands like Foo-Bar can be alienating to Unix folks at first, but have you tried just using aliases and not caring about casing? PowerShell will accept any unambiguous case-insensitive abbreviation of commands and parameters, and has almost all frequently used commands aliased (see this thread, it's literally the same as nushell).
I often wonder if people just don't like it because all they see is PowerShell scripts, where people like to spell out commands so it's clearer what they do. Imo it's a nightmare that we write bash scripts with all these 2-3 letter commands and flags that no beginner could ever understand. (but if you want to write that way even in scripts, nothing is stopping you)
> I know the long cased commands like Foo-Bar can be alienating to Unix folks at first, but have you tried just using aliases and not caring about casing.
You'd have to alias every command and every option to make it enjoyable for interactive use.
And of course those aliases would need to be part of the standards powershell.
I'm more of a Windows and C# guy, and even I just can't get along with the naming and syntax.
I tried using aliases instead, but there is a danger with this - some of the aliases are commands on Linux. For example, if I use the `ls` alias it will work as expected on Windows, but if you run the script on Linux, it will run the Linux `ls` command, rather than whatever the Powershell alias is for.
It's not the name of the commands that bother me; it's literally everything else from the operators, the sigils, to how functions are defined.
There is no excuse to make a new language that re-invents nearly every basic syntactic principle or borrows from the worst examples (bash, perl).
Although the naming isn't great either; the use of namespace prefixes rather than a flat namespace of long names might have been a better (and more logical) choice. It would have even allowed for better auto-complete.
> This looks great, but what exactly are the benefits over PowerShell?
To my perspective, making parsing for a variety of text-based formats a default action is an advantage, because it means tools don't have to use a particular runtime (e.g., .NET) to feed objects into shell pipelines without the shell user doing explicit conversion.
The only difference is powershell uses “>” and “<” for redirection, and so uses mnemonic operators for comparisons.
Nu doesn't seem to support redirection (or even have anything other than one input and one output stream, though metadata might serve the purposes otherwise served by additional streams; it's not how a program would return something Nu would treat as metadata, though.)
I guess that with a structured-data shell, error type of info could just be returned in-band under a different tag, in any case.
Complaining about command length when PS explicitly had full fledged named commands (for readability) and default aliases (for terseness) doesn't make sense imo.
# most common in script, vs at shell
Get-ChildItem | Where-Object { $_.Length -gt 4kb }
gci | where length -gt 4kb
# longest most restrictive with namespaces,
# most stable over time,
# least risk of hitting the wrong command
# or wrong property as surroundings change
Microsoft.PowerShell.Management\Get-ChildItem | Microsoft.PowerShell.Core\Where-Object -Property Length -gt 4kb
# shortest(?)
gci|? le* -gt 4kb
# because you can
switch(gci){{$_.length-gt4kb}{$_}}
gci | & { process{ if($_.Length -gt 4kb) { $_ } } }
$larges, $smalls = (gci).where({$_.length -gt 4kb}, 'split')
[linq.enumerable]::where([string[]][io.directory]::EnumerateFiles($pwd), [func[string,bool]]{param($f) [io.fileinfo]::new($f).Length -in 0..4kb})
In Powershell you can tab complete "Length" in the filter, from the available properties of the objects coming out of gci, which is neat.
PowerShell has some great ideas, but I can think of quite a few things that can be improved in (or over) PowerShell: its abysmal speed, its verbose syntax, verb-noun structure which is much less useful than noun-verb would have been...
Now, is it "Stop-Process" or "Set-Process -Status Stop"? "Out-GridView" or, say, "Show-GridView" or
"Out-View -Grid"? The other examples are all technically legal, and other modules can and do have such inconsistencies.
With verb-noun, I can't meaningfully complete for the verb when I don't know it, I can complete for the noun if I know part of it. But I usually know the noun, so that's not so useful.
With noun-verb, I could get a meaningful verb completion and still be able to complete nouns - with some duplication of results, though I think there could be a way to have some semi-completion for the noun only. A semi-completion isn't useful in the verb-noun case, since the program can't guess the noun yet and different nouns may have different verb uses.
I feel open parsing files is misguided. Downright wrong if it profilerates into scripts, because suddenly simple scripts can't work on JSON, because opening JSON returns something different from a stream of lines.
Well I imagine it respect the shebang at the top of the script so it will be run in the right shell where it doesn't parse or do any of this fancy stuff.
> I feel open parsing files is misguided. Downright wrong if it profilerates into scripts, because suddenly simple scripts can't work on JSON, because opening JSON returns something different from a stream of lines.
Also, I suspect most simple scripts that want to work on JSON Will want to parse it, so the default makes them more easily able to work with it, not less.
The problem is not with JSON parsing. It is with the fact, that people will use open without --raw to process, for example, Python files, because it is easy and works now. And some day later shell will learn to parse them into AST with open, and today's scripts will no longer work. Or worse start doing something very wrong.
Whoa. It has been a while since I've seen something that truly feels like it can revolutionise my workflow. Amazing job to everyone involved in this project!
Really excited to try this, surprised that there is no pre-built binaries. Are these planned?
The compelling idea here is that they convert the output of common shell commands into tabular data that can be manipulated using common operators, so that you don't have the remember sorting/filtering/grouping flags that may be different for every different shell command. So, imagine being able to use the same sorting/filtering logic on the output of `ls` as you might on the output of `ps`, and without relying on hacky solutions like pipelines to `grep`, `cut`, and `sort`.
It also means shell command output can be easily transposed into JSON and CSV. Actually pretty clever!
I've been delving into bash scripting a bit more than I'd like as of late and the lack of universally available consistent structured output for the CLI really got to me. Most of the script contents end up being these obfuscating and brittle "hacky solutions" that never should have been necessary. When I thought about pursuing fixing this I felt like the task was a bit overwhelming. I'm delighted that these developers are working on this!
I've generally found you can get most of what you need done with head, tail, grep, cut, paste, sed and tr (and maybe a couple others I'm missing). Oh, and test (both bash's and coreutil's). It can be a little hacky, but once you realize the crazy output you sometimes have to parse from third party programs, you realize that the ability to handle any free form text is pretty essential.
That said, since I'm a Perl developer, I'll generally just go straight to perl's inlined implicit loop mode (-p or -n) if I need to use more than a couple of those utils, unless I want it to be portable with a minimum of requirements (as even though more binaries are used in the shell version, they're all essentially guaranteed to exist on a Linux/Unix).
You can also just use PowerShell. Its chief crime is its verbosity in naming, but it has the "universally available consistent structured output for the CLI" you're asking for.
Im speaking from a position of naivete, so dont attack me;
Would an ideal situation be “do + while”
Whereas one may write
While x is true do “blah blah” && if output is still true then end elif do &&?
Or is that basically how it already works and we just have shitty interactions with the machines to get them too do the above (unless youre a more seniorbash person ((its one of my career regrets that i didnt spend a ton of time bash scripting)))
I spent a lot of time in Powershell on Windows. Almost 3-4 years. I simply could not use it as a shell.
I enjoyed converting our many build scripts into Powershell, but using it as a shell was just uncomfortable.
It's not clear to me why. If I had to guess, besides the verbosity, it was the additional "indirection". Nothing was what it appeared on the screen. It was something else.
Bash felt far more comfortable as a daily shell because what you see is what you get.
I want to try Nu because it appears to combine that WYSIWYG simplicity of bash with the Powershell quality of passing data rather than text. Maybe tables are indeed a better data structure to use than objects.
PowerShell is more like interactive Python with very smart tab completion than "bash with objects." Basic conveniences like "grep" require some memorized scripting gymnastics because the commands got generalized a little too much.
For regular shell usage it's a bit painful because the abstractions are a bit too far removed from the file system you're operating on. But for scripting, such as managing and configuring systems, it's remarkably fluid.
The "nothing is as it appears" problem never bothered me much because it's still just data. You just specify the output when you're done crunching it. IMO being able to operate with typed data is really nice because you don't have to worry (nearly as much) about serialization between commands.
This has been taken further into human read/write domains... with mixed reception: ant, XSLT, json as a config format. After all, parsing and "the syntax business" has a practical purpose.
This is not true. If you want, you can literally pipe PowerShell output into grep (or select-string, which does the same) and it will filter by the tabular representation of each row. That can work just fine in interactive use cases and while it is less accurate and performant, it is the same bash.
The key to success in using PowerShell as an interactive shell is to memorize aliases and not caring about casing and spelling everything out. PowerShell is case-insensitive, has aliases (e.g. ? instead of where-object, or % foo to get the foo property of every object) and will accept abbreviated command and parameter names if the abbreviation is unambiguous.
It is also useful to know about the different formatting cmdlets: mostly format-table (ft), format-list (fl), and that you can pick properties with -p (-Property). While PowerShell usually does the right thing, with these you can get the most appropriate formatting for any use case.
I commented on this earlier but the defectiveness of PS as a shell hit me almost immediately when I tried to recreate my workflow of 'dirs -v', pushd and popd with a large number of paths on the stack.
The chief crime of PowerShell is actually how long it takes for a prompt to appear after starting it - especially on Windows. I still use it as a login shell on MacOS because it offers useful features, but it's (to put it bluntly) slow as hell to start.
The key flaw of UNIX philosophy is destructuring deserialization and reserialization based on lines, necessitating all manner of argument escaping and field delimiters, when pipelines should be streams of typed messages that encapsulate data in typed fields. Logs especially (logging to files is a terrible idea, because it creates log rotation headaches and each program requires a log parser because of the loss of structured information) also. Line-oriented pipeline processing is fundamentally too simple. Settling on a common data simple/universal format robust enough for all purposes, including very large data sets, to exchange between programs and is just complicated enough to eliminate escapement and delimiter headaches without throwing away flexibility (by a new/refined set of processing tools/commands) is key.
It's not so much a flaw as a tradeoff. Because it's so easy to produce lines of text, virtually every program supports it, even programs written by hackers who aren't interested in following OS communication standards.
No, that's your opinion, not mine. It's a flaw because it creates escaping magic, delimiters that are in-band (spaces) and every program has to know how to parse the output of every other program, wasting human time and processing time. Structured communication, say like an open standard for a schema-less, self-describing, flexible de/serialization like protobufs/msgpack, would be far superior, more usable, more efficient and simple but no too simple to process streams of data with structure and programmability already there.
Being able to dump structured information out of an exception directly into a log, and then from a log into a database, without any loss of information or extraneous log parsing, is a clear win. Or from a command as simple as listing files ("ls") into a database or into any other tool or program. Outputing only line-oriented strings is just throwing away type information, and creates more work for everyone else, even more so than continuing to stay with lines processing tools.
So, if you're not a fan of the UNIX philosophy, maybe check out Powershell. Or take a look at WMI and DCOM in Windows. Eschew shell scripts in favor of strongly-typed programs that process strongly-typed binary formats, or XML, or whatever. The alternatives are out there.
Nushell does not seem like a violation of the unix philosophy, or at least the version of it that I like best.
"Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface."
Perhaps I'm wrong, isn't nushell simply adding one more dimension?
Instead of a one-dimensional array of lines, the standard is now... a two-dimensional array of tabular data. Perhaps it is not strictly a "text stream" but this does not seem to me to be a violation of the spirit of the message.
Simple line-based text streams are clearly inadequate IMO. 99.99% of all Unix programs output multiple data fields on one line, and to do anything useful at all with them in a programmatic fashion you wind up needing to hack together some way of parsing out those multiple bits of data on every line.
maybe check out Powershell
I left Windows right around the time PS became popular, so I never really worked with it.
It seems like overkill for most/all things I'd ever want to do. Powershell objects seem very very powerful, but they seemed like too much.
Nushell seems like a nice compromise. Avoids the overkill functionality of Powershell.
Semantically, a one-dimensional list/array of “things” can be trivially transposed into a series of lines with a linebreak separator, but I don’t think the same holds true for a “list of lists” (2-dimensional data) or a “list of lists of lists” (3D data) etc. At least without a standard begin-end list delimiter that allows nesting.
Just thinking about a way that perhaps an old tool can be “wrapped” to be tricked into working with 2+-dimensional data by somehow divvying up the 2+ dimension input data into concurrent 1-dimensional streams, but this seems to require a way to represent more than 1 dimension of data without breaking existing utilities (unless there was, like, a wrapper/unwrapper layer that handled this...)
When perl came along it killed the 'Unix one tool philosophy' dead as a doornail. And since then and people have just kinda ignored the smell coming off the rotting corpse.
I don't write complex scripts in shell anymore because it's insanity. But ad-hoc loops and crap like that... hell yeah. At least a few a day. Sometimes dozens.
People need to be reminded, I think, that shell isn't a programming language first. It's a user interface. And when I look at Powershell scripts and other things of that nature and think about living in Powershell day in and day out I don't see the big pay-off over something like fish.
'Is this going to make it so I can get my work done faster?'
'Is this going to be more pleasant to use as a primary interface for a OS?'
When I go into fish or zsh and use curl to grab json and 'jq' to get a idea of how to use the API in python or node...
versus running 'curl' in powershell against some random API I have never used before..
I get the distinct impression that 'This is fucking garbage' in that It would take me a lot longer to figure out how to use powershell in a useful way then the time I would save by doing so in the long run.
The irony is that the very attempt to be one tool for everything caused Perl's own destruction. Perl 5 is still used by some veterans for small scripts but who wants to use Perl 6?
Unix follows the KISS principle, and that is key for success. Albert Einstein said: "Keep things as simple as possible but not too simple". In that sense Unix and Posix are well done. However, that doesn't mean that good ideas like Nushell are not welcome.
Yeah, I tried using Powershell as my shell and that's when I found out Powershell is more about the scripting language than an optimized shell used for everyday use. I was confronted with this almost immediately because one of the things I rely on most in Bash is 'dirs -v', pushd and popd. I have dirs aliased to dirs -v and I usually have 15-20 paths on my stack. I'll leave implementing the same functionality in Powershell as a user exercise.
Im confused... Why doesn't Push-Location (aliased to pushd by default), Pop-Location (aliased to popd by default), and Get-Location -Stack (trivially aliasable to dirs) not work? You can even have multiple location stacks and, if you care about the registry, you can also use the same commands to handle registry navigation as well.
‘dirs -v’ shows an enumerated list of directories. If I follow that up with pushd +11 I will be taken to the 11th path listed in the prior dirs command. As far as I know this isn’t implemented out of the box in PS
An addition to `{Push,Pop}-Location` - `Get-Location -Stack` will show the current directory stack. I don't know offhand of a way to get the equivalent of `pushd +11` though.
I'm trying to elicit more of a response from the comment author. I think they make some good points. I would like to learn more about their ideal system.
That still limits us too much due to the rather severely limited available data types (although arbitrary types could be serialized into binary/string data, I guess...)
Funny that you should mention this. I just hit that problem yesterday. The lack of a binary type is a problem. This is the same thing that hit XML. Unfortunately (or fortunately), the best solution is still the MIME stuff.
Better than nothing, but the problem with that is that you can't construct a deep, lazily evaluated pipeline. JSON can't be outputted until all the data is collected.
Sounds like the CLI tools, GNU and otherwise, could benefit from some kind of "--format" switch to define the output in detail. I mean something like ls --format "{ \"filename\": \"%{filename}\", \"size\": %{size} }" for a JSON-style output (or whatever the user wanted).
Or even something like --format-rfc4627 filename,size,permissions-numeric to get only those fields out in a valid JSON format.
This wouldn't remove the "every program has to know how to parse the output of every other program", but I am not convinced it is needed. For instance, how would e.g. grep know what field you want to process? And does e.g. "size" carry universally the same meaning and semantics in all programs there is and can ever be? Ditto for "permissions". And what about "blacklist".
The fictious "json-shunt" (for lack of better name) would pass only that input-parameter to its --cmd as an input, in this case grep command, the | part would be done for things for which the grep matched, but with all the other parameters intact. So it'd print the filenames and sizes of filenames with case-sensitive "MegaCorp", and output it in JSON.
Yes, I know there are more concise and elegant ways to express the same thing of printing out file sizes and filenames of matching filenames... Also, when scripting pipelines the verbosity doesn't matter, IMO it'd actually be a benefit to be able to read the script.
Edit: fix pre-caffeine wonky redirection and brain fart
Yes, I agree that it's a tradeoff. Although that tradeoff was made when memory was sparse and expensive, a modern shell should certainly be more structured - e.g. like PowerShell or Nushell here.
I don't think I'd go as far as typed messages as that doesn't strike me as a simple or light-weight format. Something like JSON is almost good enough, but perhaps something even simpler/lighter would be ideal here.
Comments are great for human-edited config files, but not a clear win for an stdout/in format.
The trouble is, comments are normally defined as having no effect, not part of the data model. But iff the comments contain useful information, how do you pick it out using the next tool? Imagine having to extend `jq` with operators to select "comment on line before key Foo" etc... And if it is extractable, are they still comments?
How do you even preserve comments from input to output, in tools like grep, sort, etc.
XML did make comments part of its "dataset". That is, conforming parsers expose them as part of the data. Similarly for whitespace (though it's only meaningful in rare cases like pre tag but that's up to the tool interpreting data), and some other things that "should not matter" like abbreviated prefixes used for XML namespaces).
This does allow round-tripping, but complicates all tools, most importantly by disallowing assumptions that some aspect never matters, e.g. that's it's safe re-indent.
I'd argue the depest reason JSON won over XML is that XML's dataset was so damn complicated.
is "destructuring deserialization and reserialization based on lines" really an aspect of the UNIX philosophy or just an implementation detail? I thought it was more about doing one thing and doing it well [1]. It could be argued that nushell follows the UNIX philosophy.
“This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.”
Doug McIlroy (2003). The Art of Unix Programming: Basics of the Unix Philosophy
That said, I’m looking forward to testing our nushell!
> The key flaw of UNIX philosophy is destructuring deserialization and reserialization based on lines
not so sure about your assertion there eugene :o)
interfacing with funk formatted lines of text is waay more easier because (imho) of the 'wysiayg' principle, and encourages, for lack of a better term, 'tinkering'
> Settling on a common data simple/universal format robust enough for all purposes...
output from 'ls, gcc, yacc/bison, lex/flex, grep, ...' should all follow the same fmt, sure...the good thing about standards is that we don't have enough of them to choose from, f.e. xml, sgml, s-expressions, json, csv, yaml, ...
having said that, when dealing with copious volumes of data, structure doesn't hurt, but in such cases, my guess is, there are only a handful of data-producers, and data-consumers for automatic consumption, and are all tightly controlled.
It was a decision that lead to success. Before Unix most filesystems were record oriented. Unix's philosophy of treating everything as a stream of char was different and very successful.
However, the unstructuring of data and keeping a single form of communication is what has kept UNIX around for so long and why Linux hasn't collapsed under its own weight long ago.
I was voicing your opinions exactly when I was new to the professional computing world. Over time I saw a lot of structured data schemes come and go and they all fall down with this: inflexibility and improper implementations.
Where do you think does the structure inside the data come from? You need to come up with a standard. Now you need everyone to stick to that standard. You need to parse the entire data or at least most of it to get to the information you need, finding new security footguns on the way. Soon you will realise you need to extend the standard. Now you have not only conflicting implementations (because noone ever gets them completely right) but conflicting versions.
And this needs to be done for every single fucking little filter and every single language they are written in.
Take a look at the various implementations of DER and ASN.1. The standard seems simple at first glance, but I haven't seen a single implementation that wasn't wrong, incomplete or buggy. Most of them are all of that. And DER is a very old standard that people should have understood in the meantime.
In order to get at least a bit of sanity in all of this, you need a central source of The Truth wrt. your standard. Who is that going to be for Linux and the BSDs and for macOS? Linus? Microsoft? Apple? The ISO standards board? You?
And all of this is equally true for log data.
I'm okay with tossing backwards compatibility over board. But not in favour of the horrible nightmare some people here seem to propose.
I used to think this, and am still sympathetic to the motivation behind this. But I've since changed my mind when I saw the challenges faced in Big Data environments. Structured data is nice at first when you get into it. But the need for data prep stubbornly will not go away. If only because how your stakeholders want the data structured only sometimes aligns with the pre-defined structure. And when you get those numerous requests for a different structure, you're right back to data munging, and wrangling lines of text.
In my limited experience, you're either taking the cognitive hit to learn how to grapple with line-oriented text wrangling, or taking the cognitive hit to learn how to grapple with line-oriented text wrangling AND on top of that a query language to fish out what you want from the structure first before passing it to the text wrangling. I'd sure like a general solution though, if you have a way out of this thicket.
Thats when you switch to a real programming language. Relying on output from other programs will always be brittle - as error messages etc will change every new dist release.
Seems its only one level deep, no tables of tables are shown. Powershell has a lot of issues but one cool thing is you can access output as nested object data. Seems like the logical evolution.
I agree objects are way better than text for manipulating structured data. Python is great at this, so I use Mario to do it in the shell. https://github.com/python-mario/mario
For example, to sort filenames by length
$ ls | mario apply 'sorted(x, key=len)' chain
doc
src
bench
extra
AUTHORS
LICENSE
I love the core idea of nushell, but reimplementing them seems rather insane, given the incalcuable number of programmer-hours that have gone into coreutils.
That seems like a real mistake, rather than simply having nushell's "ls" be a wrapper for coreutils "ls -la" or some such.
I understand the benefit of reimplementing everything from scratch, as that way you have a more consistent nushell experience everywhere, regardless of which version of "native" ls your system has. And allowing "^ls" is a useful and necessary hedge.
But, wow reimplementing nearly everything seems like an enormous undertaking.
(It is of course possible that I'm completely misunderstanding things! Perhaps the devs can comment?)
Check the ls man page and remove anything related to sorting, filtering, or formatting information (as under this system all this functionality is provided in a shared and generic way). There is not much left.
This is pretty much what I've idealized for a new shell. Where every program can just output a standard table of information for each action. No need to have the program itself be queried for it every time, or find all the right options. You just get all the data, and filter what you want programaticly.
> You just get all the data, and filter what you want programaticly.
There are definitely cases when you're better off with producer-side filtering. For example, `find` allows you to not traverse certain directories with -prune, which might save a lot of resources.
PowerShell has a lot of "corrections" of this kind, to get producer-side filtering. e.g. Getting all Windows Eventlogs, then filtering client side turned out to be quite bad, so Get-EventLog became Get-WinEvent with a lot more filtering controls.
Cmdlets which get data, like Get-AdUser or Get-MessageTrackingLog in Exchange, or Get-CimInstance often have a lot of common filtering parameters as well as a more general -Filter or -Query "build your own".
Big source of annoyance where you can't do that, but might want to.
It also generally applies to concept of wildcards/globs not just to file paths, but many parameters. So a cmdlet like Get-User with a -Name parameter to get a user with a certain name can also be called with John* to get all users with first name John. In my experience the filtering options are pretty consistent
I think they took one step in the right direction, where two more were needed.
The UNIX idea of piping text output from one command to the next was fine at the time, but it means that the downstream command has to parse text. With nushell, you are parsing slightly more structured text.
The two steps they could have taken in addition:
1. Instead of generating rows of a table, why not lists of strings? E.g. ("abc", "def", 123) instead of "abc | def | 123". Much easier to deal with in the command receiving this input.
2. And instead of just lists, why not objects? E.g. a process object, a file object, or, if you'd like a list as above.
I have implemented this idea: https://github.com/geophile/osh. For example, if you want to find processes in state 'S' with more than 3 descendents, and print the pid and command line of each, sorted by pid:
osh ps ^ select 'p: p.state == "S" and len(p.descendents) > 3' ^ f 'p: (p.pid, p.commandline)' ^ sort $
^ is used to denote piping. ps yields Process objects. Select examines those processes and keeps those with the right characteristics. Then f applies a function, mapping to the the pid and commandline in a list, and then sort sorts the lists, by pid (the first field of each list). $ does output.
How do you think it compares to xonsh? I ask because I had been thinking recently about investing some effort into xonsh and now nushell has appeared. I’d be very curious what any serious xonsh users think about this newcomer.
> So, imagine being able to use the same sorting/filtering logic on the output of `ls` as you might on the output of `ps`, and without relying on hacky solutions like pipelines to `grep`, `cut`, and `sort`
I too have been off and on writing a Rust shell, though I want it to support `chsh` POSIX compatibility for my Arch setup. It's no where near complete though.
A brand new shell, especially in Rust is a great thing to see and also how it tackles the problems that exist in other shells.
However, its interesting to see so many dependencies that are required to build it (73 crates last time I checked) and as shells have always been portable across many operating systems, I wonder how portable nushell would be since it wishes to replace my current shell.
I did see quite a few that double up, like 2 versions of url and a few others.
Just had a quick play ... on arch linux `yay nushell-git` already works!!!
One area where I got a bit stuck was around help. `man where` gave me nothing, `help where` also. I tries stuff like `ls | where type = File` and got a type error. I think it would be amazing if this thing onboarded people a bit nicer, "where needs a condition, to learn about where type "help where" ... stuff like that
Overall really enjoying the ideas here and I am absolutely going to be following this!
I think the JavaScript/npm world has trained a generation of devs that high dependency counts are a code smell. This can be true, of course, but that's largely because of external factors.
In a perfect world, a language's ecosystem would have a tiny stdlib and allow importing of libraries for everything else. The Linux philosophy would be followed strictly.
The problem is just that the overhead in securing, maintaining, and organizing all those libraries is pretty large, as we've seen by npm's repeated failures. Of course the *nix community seems to have largely solved the problem, but there's also a massive culture of volunteerism benefiting them.
The key difference between typical GNU/Linux distributions and the npm/cargo model is that the set of packages available in the distributions is to some extent curated.
I think there’s definitely room for mirrors of package repos that focus on a stablized and audited subset of packages, with frequent security and bug backports and medium-length cycles for new features — like what Fedora and Debian do, but for language ecosystems.
Agreed. ActiveState has had that as a commercial offering for quite some time (https://www.activestate.com/products/) and Red Hat is getting into it with (OpenShift) Application Runtimes. Given that providing such a thing is hard not-so-fun work I have a hard time seeing it getting available for free. (ASP).NET Core might be alone in being well-financed and free, something I think will lure (enterprise software) developers over long-term.
It is safe to say that it is unrealistic to expect that developers in the real world audit their dependencies.
I'd imagine a feasible solution being a permission-based system (which might already exist), where programs and their dependencies do not have access to certain facilities, like the filesystem, other processes or the network, without them being declared in a manifest file of sorts. Permissions should be inheritable when explicitly specified, and a sub-dependency could not gain a permission it didn't inherit from its parent. Unfortunately this does not work so well with a shell, since the shell needs the ability to spawn arbitrary processes. At least the shell itself would not have network access, forcing it to rely on tools already installed in the system
Also, if we resort to some magical wishful thinking, then all the tools in the system follow this permission model and the package manager is aware of the permissions. You could then query the package manager for all installed software with a certain capability, like network access, disabling tools like curl and netcat.
> However, its interesting to see so many dependencies that are required to build it (73 crates last time I checked)
The problem is that Rust has two issues:
1) In Rust, you don't pay for what you don't use. This means that a lot of stuff tends to be an external library since not using something means you can exclude it COMPLETELY. The issue is that things tend to get atomized more finely than most languages would do.
2) Rust is trying to not bake things into a standard library too early. This is a good comment about standard libraries from Python:
"The standard libraries constitute one of Python’s great features, but they tend to suck up all the oxygen. Programmers are reluctant to write libraries that duplicate their functions, so poor libraries in the standard set persist. Only a few people, like Reitz, are willing to write modules that compete with the standards."
> shells have always been portable across many operating systems
Most shells work on just one of either Unix or Windows; the blog post specifically mentions Windows support as being something on the radar of the developers, which Rust is arguably a better fit for than C/C++ (which most shells are written in) due there being a single compiler, standard library, and build tool with first-class support for both Unix and Windows in the Rust toolchain.
The risk of a supply chain attack, roughly speaking, is then multiplied by all 73 dependencies. If any of those are compromised, the shell and system are compromised.
To ameliorate the risk, the security of all 73 dependencies would need to be at least an order of magnitude greater, just to catch up to shells with no dependencies.
The irony is that this lack of "dependency-safety" is far more easily exploitable than any previous lack of "memory-safety".
Of course, this can be easily fixed if developers stop multiplying third-party dependencies, so that importing a dependency involves O(1) risk, not O(?).
> But this can be easily fixed. Developers need to stop using third-party dependencies, so that importing a dependency should involve O(1) risk, not O(N^2).
How is this an easy fix? Now developers need to develop and maintain O(N) instead of O(1) software projects.
I guess that if nushell had a zero dependency policy, it would have never happened.
And there is a significant risk to spreading your resources thin when reimplementing sensitive dependencies, like crypto or network / http libraries.
> How is this an easy fix? Now developers need to develop and maintain O(N) instead of O(1) software projects.
At some point, developers need to take responsibility for the security of their projects. If your project is larger, you will need to maintain more code, whether directly or indirectly. You can't simply import a dependency but never audit it. And if you're taking the trouble to audit a third-party dependency, at some point it becomes worth maintaining directly.
> sensitive dependencies, like crypto or network / http libraries.
I am not suggesting that developers reimplement TCP.
I've had to reinvent the wheel, over and over again, throughout my career. Sometimes it was because our environment was constrained (no cloud, needs to be up for major disasters), or because the architecture was unique (no primary, all protocols must be decentralized), or because what I needed didn't exist for my stack.
It's pain and sorrow all the way down and I find more bugs in my reinvented wheels than in the actu code I set out to write.
How about this:
0. Rebuild from latest stable dependencies, or latest updates to your LTSB, whenever you or your dependencies have a bug.
1. Prefer static linking or lib incorporation if feasible, or use a package manager that properly guarantees you the right versions.
2. Stick to popular repos with responsive maintainers, or
3. Fork the dependency yourself, audit the code and know how to debug it.
> At some point, developers need to take responsibility for the security of their projects.
Agreed, but that could be through careful selection instead of rolling your own. And it would be great if there was better tooling for that.
Imagine an ecosystem where every module author automatically gets not just a public repo and issue tracker, but also a way to deal with security issues, an indication if all contributors used 2FA for code contribution and package uploads. Then there could be a search for packages where you can say "only give me those with 2FA and a defined security disclosure process, including all transitive dependencies".
That wouldn't be perfect, but much, much better than any ecosystem I've seen so far.
This is sorta why things like Guava and Boost exist - audited / trustworthy libraries for common asks where you know more or less what to expect in terms of quality.
The approach the Rust community is taking with a sort of decoupled-STL approach is interesting, but I do wish they found a way to collect the known-good-bits in a common namespace.
You do not have to maintain your own, you can use a popular and audited one that many other projects depend on so it is typically installed. The point is not to not have any dependencies, just do not have in the hundreds, especially not if they are not shared; it would take up too much space!
Note that these files are shared between projects, you do not have to download and compile them per project. Can you do the same using Cargo? According to my experience with it from two years ago, I had to download and build over 400 dependencies for all projects, separately. I do not know how many dependencies Rust projects have on average.
Looks amazing! Unfortunately a `cargo install` on stable fails due to a dependency "futures-async-stream v0.1.0-alpha.4" with the "rawkey,clipboard" features.
I love the idea of treating output as tabular data.
I see you already have a where operator. What would fit this very well would be in-built sql-like syntax. Since tables are first class, you should have an in-built way of handling them.
A languiage similar to q/kdb - it has a great SQL dsl, but also works as a regular programming language. It's also very concise and expressive, you can write oneliners that do mind bogglingly complex things.
I like the concept a lot, especially the focus on outputting data in an organised way.
What I feel could be improved is the presentation of the tabular data. Some ideas below:
- The screenshots show two horizontal lines consisting of dashes, one at the start of the table, one at the end. To me they seem redundant and could be replaced with spaces (whitespace). I did a quick check and PostgreSQL's psql tables also don't have lines like this.
- How about using Unicode lines instead of dashes and plus sign in the header? This way there would be fewer gaps there. I'm not sure though how this would affect compatibility.
Yes, this is needed for sure. Some output didn't appear on my terminal because it is hard coded to appear as black. The author must have a light colored terminal that differs from mine.
I really like this conceptually. I feel like with Powershell, you need a bunch of commands that actually pass objects around or act upon them; and something similar in Linux would require writing a lot of extensions first.
I've been using fish shell for years and glad I started down that route. It's been incredibly useful.
I really want to give this a shot too. I feel like too often I'll open up Python, read a file and then manipulate it when I need to something more complex than simple grep/sed/cut/wc etc... This feels like it really fills in that gap. It starts with something really simple: get your unstructured output in a structure, and then build from it. I could see people writing more custom extensions and objects later on if this gets popular.
The big problem with Powershell, and it makes it borderline unusable on Linux, is its crazy error handling. The equivalent of bash's `set -e` is `$ErrorActionPreference="Stop"`.
The problem is that it treats any output to stderr as an error and ignores actual posix errors (non-zero exit codes.) What's worse is in some contexts the error it throws on stderr is impossible to catch.
I really love Powershell in a lot of ways but this makes a lot of really basic scripts in bash very cumbersome to write in Powershell.
Do you have any examples of stderr errors being impossible to catch in PS, and/or a basic script that is very easy in Bash but cumbersome in PowerShell?
Recently I was trying to automate something with gsutil (Google's storage CLI) using Powershell remoting to a Windows host. gsutil is a totally well-behaved posix app, which emits status information to stderr. The bug is specifically that when using remoting (Windows equivalent of ssh) the error is uncatchable.
Ultimately I just want Powershell to handle well-behaved posix apps in a sane way, but honestly the reaction of some of the people on the thread seems like there's a huge cultural gap between Microsoft and the Posix community. (Which, Microsoft claims to be a part of but clearly is not.)
I really enjoy seeing new ideas around interfaces. But I do wonder, rather than creating a new shell, couldn't we have a 'json/structured' mode for cli programs - and maintain compatibility with existing shells? For example if it was toggled with a SHELL_FORMAT environment variable?
PowerShell has an IMO good answer to this: if you run a classic CLI program, its output is interpreted as an array of strings (lines), so if you pipe the output, you have a way to process it into something object-like.
Having a toggle creates the problem that you have to write two implementations for your tools, or you end up having to memorize which commands can run in which modes. It's easier to have a consistent pattern and a way to up-convert existing CLI programs.
I'we always felt like there should be no file system and no shell but a relational database with a particularly suitable and extensible SQL dialect REPL.
I can't really tell how do we apply make this idea compatible with the common workflow/model of computing. In my imaginary world everything down to the level of a statement of what is a code file in the real world is a separate record in a relational database, there are no file names and no folders (only tags, commentaries and other metadata).
In the real world folder hierarchy is probably inevitable so yes, it has to be stored in a way (in a separate table perhaps so we can have many-to-many relationships sort of what hardlinks do).
> In my imaginary world everything down to the level of a statement of what is a code file in the real world is a separate record in a relational database,
It's cool, indeed. Nevertheless, besides files which contain sets of logical statements/data, there are pictures, audio and video recordings which are binary by nature. Does Prolog support this kind of data types?
I’m still with you on the complete reimagining of how a computer stores and accesses ‘code files’, but there’s still a challenge with ‘small files’, which is to say the hundreds or thousands of chunks of code that can have dozens of uses (Plex’s metadata cache, for example, or the hundreds of shared libraries that allow larger more complex programs to be built small by referencing them).
One benefit of folder hierarchies (and I don’t think there are many) is that it’s easy to delete/move hundreds of thousands of tiny bits of associated data just by deleting/moving the parent.
You know Bill Gates wanted Windows Vista to use a relational DB for all applications instead of a file system. It was one of the reasons Vista got delayed, because they realised it is just not practical and a security nightmare
I know. This was the first of the 3 times I ever felt good and even eager about a new feature [promised to be] introduced in a new version of Windows. I felt a huge pity it was cancelled. It could have been picked up by other systems (like Linux and Mac) already if it was actually released that time (provided it was engineered well enough to be usable).
The second nice thing was the new taskbar (with pinnable app buttons and without text) in Windows 7-10. I've even configured the taskbar in KDE (which I use at home) to the same mode.
The third cool thing is cross-app text input autocompletion and correction made available with physical keyboards in Windows 10. Sadly I don't know how to accept a completion suggestion without using mouse, yet the feature already helps me a huge lot (as I write a language I'm not very good at).
If not these 3 things I'd be glad to keep using Windows 95, if only there were security&stability updates and current software and hardware support - newer versions of Windows didn't add any other features I would find useful. Everything else (besides internals which have obviously improved a lot) is either bloat or a minor goody.
Windows 10 has seemingly added universal file tagging support (previous versions could only add tags to files of some formats which allow injecting them in file body) but as long as Windows, KDE and MacOS don't store the tags the same way (so I could tag files on an external drive using Windows and keep using the tags when I attach the drive to a Mac or a Linux machine) this is of little use for me.
I like this a lot but why does this have to be a completely new shell? Couldn't this also just be a set of tools that I can use in an existing shell and get the best of both worlds?
This was my take away, as well. Bourne and Bash are everywhere and zsh will be included in macOS updates. I definitely see value in a separate set of text processing tools.
The GitHub repo looks kinda weird on first glance (>1k commits but <100 stars, virtually no contributors or issues, empty LICENSE file) but yeah it is more of what I thought.
Personally, I'm not very accustomed to Python though, so I find the syntax kinda weird. I'd prefer something that feels more like a standard Un*x tool (like nushell).
375 comments
[ 1.8 ms ] story [ 258 ms ] thread> What if we could take the ideas of a structured shell and make it more functional? What if it worked on Windows, Linux, and macOS? What if it had great error messages?
- Any shell with piping is already very functional, and PowerShell has script blocks (higher-order functions). Or does this mean "functional" in the literal sense? What functionality is missing? - PowerShell works on macOS, Linux and Windows - The examples of error messages further down look almost exactly like PowerShells error messages (underlining the command in the pipeline that failed)
It is not clear to me what exactly the authors sought out to do different
https://github.com/PowerShell/PowerShell/blob/master/LICENSE...
I often wonder if people just don't like it because all they see is PowerShell scripts, where people like to spell out commands so it's clearer what they do. Imo it's a nightmare that we write bash scripts with all these 2-3 letter commands and flags that no beginner could ever understand. (but if you want to write that way even in scripts, nothing is stopping you)
You'd have to alias every command and every option to make it enjoyable for interactive use.
And of course those aliases would need to be part of the standards powershell.
Which due to the naming conventions are still too verbose.
I tried using aliases instead, but there is a danger with this - some of the aliases are commands on Linux. For example, if I use the `ls` alias it will work as expected on Windows, but if you run the script on Linux, it will run the Linux `ls` command, rather than whatever the Powershell alias is for.
There is no excuse to make a new language that re-invents nearly every basic syntactic principle or borrows from the worst examples (bash, perl).
Although the naming isn't great either; the use of namespace prefixes rather than a flat namespace of long names might have been a better (and more logical) choice. It would have even allowed for better auto-complete.
To my perspective, making parsing for a variety of text-based formats a default action is an advantage, because it means tools don't have to use a particular runtime (e.g., .NET) to feed objects into shell pipelines without the shell user doing explicit conversion.
That doesn't mean there aren't ways to improve upon PowerShell.
As a light user of PowerShell I consider it a poor implementation of a great idea.
Commands are way too long, control structures have poor syntax etc.
To make this very concrete, take this example from nu:
Now give me an equivalent in PowerShell and then let's talk about how different Nu is from PowerShell.Nu doesn't seem to support redirection (or even have anything other than one input and one output stream, though metadata might serve the purposes otherwise served by additional streams; it's not how a program would return something Nu would treat as metadata, though.)
I guess that with a structured-data shell, error type of info could just be returned in-band under a different tag, in any case.
"Get-Process | Out-GridView -PassThru | Stop-Process"
Now, is it "Stop-Process" or "Set-Process -Status Stop"? "Out-GridView" or, say, "Show-GridView" or "Out-View -Grid"? The other examples are all technically legal, and other modules can and do have such inconsistencies.
With verb-noun, I can't meaningfully complete for the verb when I don't know it, I can complete for the noun if I know part of it. But I usually know the noun, so that's not so useful.
With noun-verb, I could get a meaningful verb completion and still be able to complete nouns - with some duplication of results, though I think there could be a way to have some semi-completion for the noun only. A semi-completion isn't useful in the verb-noun case, since the program can't guess the noun yet and different nouns may have different verb uses.
I really like the concept, but the slowness ruined the user experience for me.
Doesn't “open --raw” address that?
https://book.nushell.sh/en/loading_data#opening-in-raw-mode
Also, I suspect most simple scripts that want to work on JSON Will want to parse it, so the default makes them more easily able to work with it, not less.
https://github.com/python-mario/mario
Really excited to try this, surprised that there is no pre-built binaries. Are these planned?
It also means shell command output can be easily transposed into JSON and CSV. Actually pretty clever!
That said, since I'm a Perl developer, I'll generally just go straight to perl's inlined implicit loop mode (-p or -n) if I need to use more than a couple of those utils, unless I want it to be portable with a minimum of requirements (as even though more binaries are used in the shell version, they're all essentially guaranteed to exist on a Linux/Unix).
Would an ideal situation be “do + while”
Whereas one may write
While x is true do “blah blah” && if output is still true then end elif do &&?
Or is that basically how it already works and we just have shitty interactions with the machines to get them too do the above (unless youre a more seniorbash person ((its one of my career regrets that i didnt spend a ton of time bash scripting)))
I enjoyed converting our many build scripts into Powershell, but using it as a shell was just uncomfortable.
It's not clear to me why. If I had to guess, besides the verbosity, it was the additional "indirection". Nothing was what it appeared on the screen. It was something else.
Bash felt far more comfortable as a daily shell because what you see is what you get.
I want to try Nu because it appears to combine that WYSIWYG simplicity of bash with the Powershell quality of passing data rather than text. Maybe tables are indeed a better data structure to use than objects.
For regular shell usage it's a bit painful because the abstractions are a bit too far removed from the file system you're operating on. But for scripting, such as managing and configuring systems, it's remarkably fluid.
The "nothing is as it appears" problem never bothered me much because it's still just data. You just specify the output when you're done crunching it. IMO being able to operate with typed data is really nice because you don't have to worry (nearly as much) about serialization between commands.
Scripting is programming, so you want data structures. For interaction, you want human-readable syntax.
e.g. programming languages mostly have a syntax: C, python, bash, fortran (exception: lisp), but they operate on data structures.
We can see powershell, nushell and other structured shells (e.g. some use XML) as an attempt to "get out of the syntax business", which XML began and JSON finished (https://blog.mlab.com/2011/03/why-is-json-so-popular-develop...).
This has been taken further into human read/write domains... with mixed reception: ant, XSLT, json as a config format. After all, parsing and "the syntax business" has a practical purpose.
The key to success in using PowerShell as an interactive shell is to memorize aliases and not caring about casing and spelling everything out. PowerShell is case-insensitive, has aliases (e.g. ? instead of where-object, or % foo to get the foo property of every object) and will accept abbreviated command and parameter names if the abbreviation is unambiguous. It is also useful to know about the different formatting cmdlets: mostly format-table (ft), format-list (fl), and that you can pick properties with -p (-Property). While PowerShell usually does the right thing, with these you can get the most appropriate formatting for any use case.
I commented on this earlier but the defectiveness of PS as a shell hit me almost immediately when I tried to recreate my workflow of 'dirs -v', pushd and popd with a large number of paths on the stack.
Being able to dump structured information out of an exception directly into a log, and then from a log into a database, without any loss of information or extraneous log parsing, is a clear win. Or from a command as simple as listing files ("ls") into a database or into any other tool or program. Outputing only line-oriented strings is just throwing away type information, and creates more work for everyone else, even more so than continuing to stay with lines processing tools.
"Worse is better" isn't for everyone.
"Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface."
Perhaps I'm wrong, isn't nushell simply adding one more dimension?
Instead of a one-dimensional array of lines, the standard is now... a two-dimensional array of tabular data. Perhaps it is not strictly a "text stream" but this does not seem to me to be a violation of the spirit of the message.
Simple line-based text streams are clearly inadequate IMO. 99.99% of all Unix programs output multiple data fields on one line, and to do anything useful at all with them in a programmatic fashion you wind up needing to hack together some way of parsing out those multiple bits of data on every line.
I left Windows right around the time PS became popular, so I never really worked with it.It seems like overkill for most/all things I'd ever want to do. Powershell objects seem very very powerful, but they seemed like too much.
Nushell seems like a nice compromise. Avoids the overkill functionality of Powershell.
Just thinking about a way that perhaps an old tool can be “wrapped” to be tricked into working with 2+-dimensional data by somehow divvying up the 2+ dimension input data into concurrent 1-dimensional streams, but this seems to require a way to represent more than 1 dimension of data without breaking existing utilities (unless there was, like, a wrapper/unwrapper layer that handled this...)
There's no real alternative for many profesionals, if they want to be employable.
Maybe we should accept this fact and try to make everyone's life easier than be stuck up and push people away.
There's no law that says that Unix tools can't be extended.
And heck, basic POSIX tools violate so many Unix principles...
I don't write complex scripts in shell anymore because it's insanity. But ad-hoc loops and crap like that... hell yeah. At least a few a day. Sometimes dozens.
People need to be reminded, I think, that shell isn't a programming language first. It's a user interface. And when I look at Powershell scripts and other things of that nature and think about living in Powershell day in and day out I don't see the big pay-off over something like fish.
'Is this going to make it so I can get my work done faster?'
'Is this going to be more pleasant to use as a primary interface for a OS?'
When I go into fish or zsh and use curl to grab json and 'jq' to get a idea of how to use the API in python or node...
versus running 'curl' in powershell against some random API I have never used before..
I get the distinct impression that 'This is fucking garbage' in that It would take me a lot longer to figure out how to use powershell in a useful way then the time I would save by doing so in the long run.
Unix follows the KISS principle, and that is key for success. Albert Einstein said: "Keep things as simple as possible but not too simple". In that sense Unix and Posix are well done. However, that doesn't mean that good ideas like Nushell are not welcome.
[0] https://github.com/nickcox/cd-extras#navigate-even-faster
I'm trying to elicit more of a response from the comment author. I think they make some good points. I would like to learn more about their ideal system.
[1] http://jsonlines.org/
[2] http://ndjson.org/
Or even something like --format-rfc4627 filename,size,permissions-numeric to get only those fields out in a valid JSON format.
This wouldn't remove the "every program has to know how to parse the output of every other program", but I am not convinced it is needed. For instance, how would e.g. grep know what field you want to process? And does e.g. "size" carry universally the same meaning and semantics in all programs there is and can ever be? Ditto for "permissions". And what about "blacklist".
As a completely fictious toy example:
The fictious "json-shunt" (for lack of better name) would pass only that input-parameter to its --cmd as an input, in this case grep command, the | part would be done for things for which the grep matched, but with all the other parameters intact. So it'd print the filenames and sizes of filenames with case-sensitive "MegaCorp", and output it in JSON.Yes, I know there are more concise and elegant ways to express the same thing of printing out file sizes and filenames of matching filenames... Also, when scripting pipelines the verbosity doesn't matter, IMO it'd actually be a benefit to be able to read the script.
Edit: fix pre-caffeine wonky redirection and brain fart
The trouble is, comments are normally defined as having no effect, not part of the data model. But iff the comments contain useful information, how do you pick it out using the next tool? Imagine having to extend `jq` with operators to select "comment on line before key Foo" etc... And if it is extractable, are they still comments?
How do you even preserve comments from input to output, in tools like grep, sort, etc.
XML did make comments part of its "dataset". That is, conforming parsers expose them as part of the data. Similarly for whitespace (though it's only meaningful in rare cases like pre tag but that's up to the tool interpreting data), and some other things that "should not matter" like abbreviated prefixes used for XML namespaces). This does allow round-tripping, but complicates all tools, most importantly by disallowing assumptions that some aspect never matters, e.g. that's it's safe re-indent.
I'd argue the depest reason JSON won over XML is that XML's dataset was so damn complicated.
[1] https://en.wikipedia.org/wiki/Unix_philosophy
Doug McIlroy (2003). The Art of Unix Programming: Basics of the Unix Philosophy
That said, I’m looking forward to testing our nushell!
not so sure about your assertion there eugene :o)
interfacing with funk formatted lines of text is waay more easier because (imho) of the 'wysiayg' principle, and encourages, for lack of a better term, 'tinkering'
> Settling on a common data simple/universal format robust enough for all purposes...
output from 'ls, gcc, yacc/bison, lex/flex, grep, ...' should all follow the same fmt, sure...the good thing about standards is that we don't have enough of them to choose from, f.e. xml, sgml, s-expressions, json, csv, yaml, ...
having said that, when dealing with copious volumes of data, structure doesn't hurt, but in such cases, my guess is, there are only a handful of data-producers, and data-consumers for automatic consumption, and are all tightly controlled.
However, the unstructuring of data and keeping a single form of communication is what has kept UNIX around for so long and why Linux hasn't collapsed under its own weight long ago.
I was voicing your opinions exactly when I was new to the professional computing world. Over time I saw a lot of structured data schemes come and go and they all fall down with this: inflexibility and improper implementations.
Where do you think does the structure inside the data come from? You need to come up with a standard. Now you need everyone to stick to that standard. You need to parse the entire data or at least most of it to get to the information you need, finding new security footguns on the way. Soon you will realise you need to extend the standard. Now you have not only conflicting implementations (because noone ever gets them completely right) but conflicting versions.
And this needs to be done for every single fucking little filter and every single language they are written in.
Take a look at the various implementations of DER and ASN.1. The standard seems simple at first glance, but I haven't seen a single implementation that wasn't wrong, incomplete or buggy. Most of them are all of that. And DER is a very old standard that people should have understood in the meantime.
In order to get at least a bit of sanity in all of this, you need a central source of The Truth wrt. your standard. Who is that going to be for Linux and the BSDs and for macOS? Linus? Microsoft? Apple? The ISO standards board? You?
And all of this is equally true for log data.
I'm okay with tossing backwards compatibility over board. But not in favour of the horrible nightmare some people here seem to propose.
In my limited experience, you're either taking the cognitive hit to learn how to grapple with line-oriented text wrangling, or taking the cognitive hit to learn how to grapple with line-oriented text wrangling AND on top of that a query language to fish out what you want from the structure first before passing it to the text wrangling. I'd sure like a general solution though, if you have a way out of this thicket.
https://github.com/python-mario/mario
For example, to sort filenames by length
In their examples it looks like 'ls' is built-in to their shell instead of from e.g. coreutils.
That seems like a real mistake, rather than simply having nushell's "ls" be a wrapper for coreutils "ls -la" or some such.
I understand the benefit of reimplementing everything from scratch, as that way you have a more consistent nushell experience everywhere, regardless of which version of "native" ls your system has. And allowing "^ls" is a useful and necessary hedge.
But, wow reimplementing nearly everything seems like an enormous undertaking.
(It is of course possible that I'm completely misunderstanding things! Perhaps the devs can comment?)
There are definitely cases when you're better off with producer-side filtering. For example, `find` allows you to not traverse certain directories with -prune, which might save a lot of resources.
Cmdlets which get data, like Get-AdUser or Get-MessageTrackingLog in Exchange, or Get-CimInstance often have a lot of common filtering parameters as well as a more general -Filter or -Query "build your own".
Big source of annoyance where you can't do that, but might want to.
The UNIX idea of piping text output from one command to the next was fine at the time, but it means that the downstream command has to parse text. With nushell, you are parsing slightly more structured text.
The two steps they could have taken in addition:
1. Instead of generating rows of a table, why not lists of strings? E.g. ("abc", "def", 123) instead of "abc | def | 123". Much easier to deal with in the command receiving this input.
2. And instead of just lists, why not objects? E.g. a process object, a file object, or, if you'd like a list as above.
I have implemented this idea: https://github.com/geophile/osh. For example, if you want to find processes in state 'S' with more than 3 descendents, and print the pid and command line of each, sorted by pid:
^ is used to denote piping. ps yields Process objects. Select examines those processes and keeps those with the right characteristics. Then f applies a function, mapping to the the pid and commandline in a list, and then sort sorts the lists, by pid (the first field of each list). $ does output.'where' in pwsh / nush literally looks for keys with particular values.
That would awksome.
https://github.com/dkogan/vnlog#powershell-style-filtering-o...
https://github.com/nixpulvis/oursh
`nu` seems very cool (and I love the name ;), but isn't building atm.
However, its interesting to see so many dependencies that are required to build it (73 crates last time I checked) and as shells have always been portable across many operating systems, I wonder how portable nushell would be since it wishes to replace my current shell.
That's a bit too many. But it's not often you compile your shell from scratch.
I’m always impressed when people show up with fixes like this (that I wouldn’t even know to look for).
Just had a quick play ... on arch linux `yay nushell-git` already works!!!
One area where I got a bit stuck was around help. `man where` gave me nothing, `help where` also. I tries stuff like `ls | where type = File` and got a type error. I think it would be amazing if this thing onboarded people a bit nicer, "where needs a condition, to learn about where type "help where" ... stuff like that
Overall really enjoying the ideas here and I am absolutely going to be following this!
In a perfect world, a language's ecosystem would have a tiny stdlib and allow importing of libraries for everything else. The Linux philosophy would be followed strictly.
The problem is just that the overhead in securing, maintaining, and organizing all those libraries is pretty large, as we've seen by npm's repeated failures. Of course the *nix community seems to have largely solved the problem, but there's also a massive culture of volunteerism benefiting them.
- torvalds
For a FOSS language, a large stdlib can be similar to a curated repo in practice.
I'd imagine a feasible solution being a permission-based system (which might already exist), where programs and their dependencies do not have access to certain facilities, like the filesystem, other processes or the network, without them being declared in a manifest file of sorts. Permissions should be inheritable when explicitly specified, and a sub-dependency could not gain a permission it didn't inherit from its parent. Unfortunately this does not work so well with a shell, since the shell needs the ability to spawn arbitrary processes. At least the shell itself would not have network access, forcing it to rely on tools already installed in the system
Also, if we resort to some magical wishful thinking, then all the tools in the system follow this permission model and the package manager is aware of the permissions. You could then query the package manager for all installed software with a certain capability, like network access, disabling tools like curl and netcat.
https://github.com/python-mario/mario
The problem is that Rust has two issues:
1) In Rust, you don't pay for what you don't use. This means that a lot of stuff tends to be an external library since not using something means you can exclude it COMPLETELY. The issue is that things tend to get atomized more finely than most languages would do.
2) Rust is trying to not bake things into a standard library too early. This is a good comment about standard libraries from Python:
"The standard libraries constitute one of Python’s great features, but they tend to suck up all the oxygen. Programmers are reluctant to write libraries that duplicate their functions, so poor libraries in the standard set persist. Only a few people, like Reitz, are willing to write modules that compete with the standards."
https://leancrew.com/all-this/2012/04/where-modules-go-to-di...
Most shells work on just one of either Unix or Windows; the blog post specifically mentions Windows support as being something on the radar of the developers, which Rust is arguably a better fit for than C/C++ (which most shells are written in) due there being a single compiler, standard library, and build tool with first-class support for both Unix and Windows in the Rust toolchain.
To ameliorate the risk, the security of all 73 dependencies would need to be at least an order of magnitude greater, just to catch up to shells with no dependencies.
The irony is that this lack of "dependency-safety" is far more easily exploitable than any previous lack of "memory-safety".
Of course, this can be easily fixed if developers stop multiplying third-party dependencies, so that importing a dependency involves O(1) risk, not O(?).
How is this an easy fix? Now developers need to develop and maintain O(N) instead of O(1) software projects.
I guess that if nushell had a zero dependency policy, it would have never happened.
And there is a significant risk to spreading your resources thin when reimplementing sensitive dependencies, like crypto or network / http libraries.
At some point, developers need to take responsibility for the security of their projects. If your project is larger, you will need to maintain more code, whether directly or indirectly. You can't simply import a dependency but never audit it. And if you're taking the trouble to audit a third-party dependency, at some point it becomes worth maintaining directly.
> sensitive dependencies, like crypto or network / http libraries.
I am not suggesting that developers reimplement TCP.
It's pain and sorrow all the way down and I find more bugs in my reinvented wheels than in the actu code I set out to write.
How about this:
0. Rebuild from latest stable dependencies, or latest updates to your LTSB, whenever you or your dependencies have a bug.
1. Prefer static linking or lib incorporation if feasible, or use a package manager that properly guarantees you the right versions.
2. Stick to popular repos with responsive maintainers, or
3. Fork the dependency yourself, audit the code and know how to debug it.
4. Watch upstream for security bugs.
Prefer dependencies with smaller flatter dependency trees.
Agreed, but that could be through careful selection instead of rolling your own. And it would be great if there was better tooling for that.
Imagine an ecosystem where every module author automatically gets not just a public repo and issue tracker, but also a way to deal with security issues, an indication if all contributors used 2FA for code contribution and package uploads. Then there could be a search for packages where you can say "only give me those with 2FA and a defined security disclosure process, including all transitive dependencies".
That wouldn't be perfect, but much, much better than any ecosystem I've seen so far.
The approach the Rust community is taking with a sort of decoupled-STL approach is interesting, but I do wish they found a way to collect the known-good-bits in a common namespace.
Or look at bash:
https://www.archlinux.org/packages/core/x86_64/bash/
I would say many shells that are written in C will have an extremely short list of dependencies in contrast to Rust.
As per your request:
Those files are generally installed on all major Linux distributions.For completeness:
Note that these files are shared between projects, you do not have to download and compile them per project. Can you do the same using Cargo? According to my experience with it from two years ago, I had to download and build over 400 dependencies for all projects, separately. I do not know how many dependencies Rust projects have on average.Really interested to see where this goes..
The source is pretty interesting. Here's the PS command: https://github.com/nushell/nushell/blob/master/src/commands/...
And the sort-by: https://github.com/nushell/nushell/blob/master/src/commands/...
Anything you can script in PowerShell can have unit tests and code coverage and it integrates into CI systems.
I don't really like using PowerShell, but built-in unit testing is something I wish other shells offered. It is a huge plus.
Can you imagine the cute squirrel logo? Mouth full of peanuts? Nushell sounds like an new wave band.
I see you already have a where operator. What would fit this very well would be in-built sql-like syntax. Since tables are first class, you should have an in-built way of handling them.
A languiage similar to q/kdb - it has a great SQL dsl, but also works as a regular programming language. It's also very concise and expressive, you can write oneliners that do mind bogglingly complex things.
[1] https://www.gnu.org/software/emacs/manual/html_mono/eshell.h...
What I feel could be improved is the presentation of the tabular data. Some ideas below:
- The screenshots show two horizontal lines consisting of dashes, one at the start of the table, one at the end. To me they seem redundant and could be replaced with spaces (whitespace). I did a quick check and PostgreSQL's psql tables also don't have lines like this.
- How about using Unicode lines instead of dashes and plus sign in the header? This way there would be fewer gaps there. I'm not sure though how this would affect compatibility.
Relative like "today" would be fine, if it includes the exact time today too.
I've been using fish shell for years and glad I started down that route. It's been incredibly useful.
I really want to give this a shot too. I feel like too often I'll open up Python, read a file and then manipulate it when I need to something more complex than simple grep/sed/cut/wc etc... This feels like it really fills in that gap. It starts with something really simple: get your unstructured output in a structure, and then build from it. I could see people writing more custom extensions and objects later on if this gets popular.
It's spectacular to have a shell with consistent verbiage and parameters.
https://aka.ms/getps6-linux
The problem is that it treats any output to stderr as an error and ignores actual posix errors (non-zero exit codes.) What's worse is in some contexts the error it throws on stderr is impossible to catch.
I really love Powershell in a lot of ways but this makes a lot of really basic scripts in bash very cumbersome to write in Powershell.
[Remoting Bug] https://github.com/PowerShell/PowerShell/issues/3996
[RFC for improved error handling] https://github.com/PowerShell/PowerShell-RFC/pull/88
Ultimately I just want Powershell to handle well-behaved posix apps in a sane way, but honestly the reaction of some of the people on the thread seems like there's a huge cultural gap between Microsoft and the Posix community. (Which, Microsoft claims to be a part of but clearly is not.)
https://github.com/python-mario/mario
https://takoshell.org/
https://news.ycombinator.com/newsguidelines.html
What issues could be faced with this approach?
Having a toggle creates the problem that you have to write two implementations for your tools, or you end up having to memorize which commands can run in which modes. It's easier to have a consistent pattern and a way to up-convert existing CLI programs.
https://github.com/python-mario/mario
In the real world folder hierarchy is probably inevitable so yes, it has to be stored in a way (in a separate table perhaps so we can have many-to-many relationships sort of what hardlinks do).
I think you'd like prolog https://swish.swi-prolog.org/example/examples.swinb
One benefit of folder hierarchies (and I don’t think there are many) is that it’s easy to delete/move hundreds of thousands of tiny bits of associated data just by deleting/moving the parent.
The second nice thing was the new taskbar (with pinnable app buttons and without text) in Windows 7-10. I've even configured the taskbar in KDE (which I use at home) to the same mode.
The third cool thing is cross-app text input autocompletion and correction made available with physical keyboards in Windows 10. Sadly I don't know how to accept a completion suggestion without using mouse, yet the feature already helps me a huge lot (as I write a language I'm not very good at).
If not these 3 things I'd be glad to keep using Windows 95, if only there were security&stability updates and current software and hardware support - newer versions of Windows didn't add any other features I would find useful. Everything else (besides internals which have obviously improved a lot) is either bloat or a minor goody.
Windows 10 has seemingly added universal file tagging support (previous versions could only add tags to files of some formats which allow injecting them in file body) but as long as Windows, KDE and MacOS don't store the tags the same way (so I could tag files on an external drive using Windows and keep using the tags when I attach the drive to a Mac or a Linux machine) this is of little use for me.
https://github.com/python-mario/mario
https://github.com/python-mario/mario
https://github.com/xonsh/xonsh/
Personally, I'm not very accustomed to Python though, so I find the syntax kinda weird. I'd prefer something that feels more like a standard Un*x tool (like nushell).