I would wish for jq to be a really generic tool for working with structured data on the commandline, but I have a really hard time figuring out how to do e.g. conditional-based editing etc. Can't get my head around that, and don't find any info about it on the net. Seems even something that just supports SQL (upon JSON) would be better in this regard.
Doing editing only on selected "rows" or items in a structure, based on the values of other attributes in them. Similar to what you specify in the WHERE clause in SQL-statements.
Hi there - I'm the author of `jc`. I also created `jello`[0], which works just like `jq` but uses python syntax. I find `jq` is great for many things but sometimes more complex operations are easier for me to grok in python.
So is xml. Json is relatively unstructured compared to other formats.
I think json has several advantages though. It’s a relatively lightweight and widely known serialization standard, rich enough for most cases and extensible in others, and it has easy to use parsers in all major programming languages.
Also jsonlines is a simple addition that make it easy for json to play well with non-json aware older Unix tools.
It has a few shortcomings but I think its advantages outweigh them, and it’s become a pretty widely used standard in a short time.
Last year I looked for JSON output with the dig command on Linux but found the yaml option while reading the man page. It was handy. I always wondered why yaml/JSON output is not standard with Linux and Unix utilities for scripting needs. Anyway:
Because neither yaml or JSON existed when most of these tools were written, and then people got used to the current outputs, and so it's very varying whether or not anyone has felt the pain enough to decide it's worth adding the option for another output format.
Been using this a while to pull out a bunch of server stats for a monitoring dashboard and it's been great. I just wish there were more supported services. The total mess of different outputs and config types for Linux packages is extremely annoying to deal with.
Feel free to open issues on GitHub if you would like to recommend parsers! Also, the latest version of jc supports parsing of /proc files, which may also help.
Didn't know about this, the HN dividend pays out again!
When wrestling with sed/awk in trying to parse results of a shell command, I've often thought that a shell-standard, structured outpout would be very handy. Powershell[0] has this, but it's a binary format - so not human-readable. I want something in the middle: human- and machine-readable. Without either having to do parsing gymnastics.
jc isn't quite that shell standard, but looks like it goes a long way towards it. And, of course, when JSON falls out of fashion and is replaced by <whatever>, `*c` can emerge to fill the gap. Nice.
Well, yes - powershell passes binary objects but as you can always:
1) access their properties
2) pass them downstream
3) serialize to json/csv
4) instantiate from json/csv
I think this is both human- and machine-readable enough (even through internal format is binary, but working with Powershell you are never really exposed to it).
How do you think it can be improved?
In my opinion object io IS the best part of powershell - it allows us to ditch results wrangling with sed/awk/grep entirely. I'm super interested if there's an even better way forward.
Shell would benefit from Content-Type/Accept headers. Like you can specify that cat accepts text and jq accepts Json. Then `ip a` would output corresponding type automatically.
>> Shell would benefit from Content-Type/Accept headers. Like you can specify that cat accepts text and jq accepts Json. Then `ip a` would output corresponding type automatically.
That seems unnecessary. Traditionally, shells have always used text streams. JSON is just text that follows a given convention. Couldn't what you are describing be implemented by setting environment variables or using command line flags?
For example:
PREFERRED_OUTPUT_FORMAT="JSON"
--output-format="JSON"
--input-format="JSON"
Tools that can generate and consume structured text formats are a good idea, but they should be flexible enough that they can even work with other tools that have not been written yet.
"This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface." --Doug McIlroy
I don't follow. JSON is not really readable. I don't want to see JSON output ever except for script debugging. I want to see well formatted output. But at the same time I want to be able to write something like
ip a | filter "[].address like 192.*"
So when I'm typing `ip a` I expect to get output for human and when I'm piping it to `filter` program, I expect for those programs to exchange with JSON (and ideally `filter` should use some tabular formatting as its human-readable output).
You suggesting that I should write `PREFERRED_OUTPUT_FORMAT=JSON ip a | filter "[].address like 192.*"` but that's really verbose and error-prone. It might work for scripts, but for ad-hoc shell I don't like this approach. Ideally programs should be able to communicate between pipes for their preferred formats.
I was saying that Accept Headers or "format negotiation" concepts that are typically used in client-server communications are a bit overkill for command line tools and shell pipelines.
I agree that human-readable text formats should be the default output formats for command line tools, but that easy-to-parse structured text output formats should be easy to specify with either environment variables or command line flags.
If I am writing a script and I am using tools that support a given structured output format and use environment variables or command line flags for output configuration it could work something like this:
This would mean that the command line tools default to human-readable formats, but can still generate JSON or some other structured text format when configured to do so.
Someone on this site suggested that programs open another filehandle along with stdout and stderr (stdjson) for their json output which struck me as a way to make this work in a backwards-compatible fashion.
Nushell has this too. I‘ve tried is as daily driver for a while. It‘s not there yet, but almost. After it hits 1.0 I‘m going to switch for good and leave the duck tape solutions behind.
Just like we have stdout and stderr, header lines such as those produced by `ps` should be printed to stdmeta. Curl is the worse offender here, outputing meta lines to stderr instead of stdout. A stdmeta file descriptor would make it clear what is data, what is an error, and what is _describing_ the data.
If we're adding standard output fds, maybe it would be a useful time to define any mechanism for consumers of those fds to discern the total ordering of bytes written to the three of them.
WC has a "character" flag that really just counts bytes:
$ echo דותן | wc -c
9
Note that each letter is two bytes, and the newline is an additional byte. So you could pipe (or tree) to wc to count bytes. For a hypothetical stdmeta on fd 3, that might look like this (piping stdout to devnull):
I'm not sure what you are answering. I am writing about the thing where some programs intentionally write different byte streams to stderr and stdout that will be interleaved a certain way if stdout and stderr turn out the be the same thing, but if the user wishes to distinguish which bytes were written to which stream, they can no longer recover the correct interleaving of the bytes.
The best compromise I can think of would be to prepend the line source to the combined output, so lines in the combined output can be distinguished. But depending on how you do that, you might have timing issues. In fact, any form of interleaving the outputs of stderr and stdout are technically subject to timing issues because even the source application can’t specify that bytes in one stream should follow bytes in another stream. https://stackoverflow.com/questions/12517519/how-to-redirect... talks a bit about the problem, and I’m sure someone somewhere has written a general-purpose tool, but there are timing edge cases here and often the solution is seen as keeping errors messages exclusively in stdout (or logs) or more practically, not worrying about preserving order and writing errors exclusively to stderr without worrying as much about strictly when errors interleave. If you add timestamps to your line outputs, or add some form of line numbers to the output, you can restore order after logging but it would be application-specific to do so. From a practical perspective, listening to both streams and quickly appending to a list with a data structure containing the source and message text is probably your closest approximation if writing code to handle output streams and you want an combined output that can still source which stream came from which source. (Again subject to timing errors)
You could require orderable sequence numbers before or after each line. It would be easiest as yet another VT escape sequence, but if you are pie-in-sky adding more standard "files" anyway maybe you wish for lines to be newline-delimited JSON CRDTs while you are at it.
That reminds me of something I've wanted for quite a while:
A ringbuffer filetype. Similar to a named pipe file (see: fifo(7)[^1]), but without consuming the contents on read and automatically rotating out the oldest lines.
Of course, there would be some complexities around handling read position as lines are being rotated out from under you.
Only allow sequential reads. Each reader gets it's own cursor. If writer catches up to slowest reader (buffer full) it bumps it up before it writes. Clients can query current cursor position to know if they've been bumped. Or something like that.
Great idea, but sounds like a maintenance nightmare to me. Not only that many users will complain that their favorite CLI tool isn't supported, but also a new release of any of the supported CLI tools might break the support without any kind of warning, as I don't think changes to the (human-readable) output are considered major changes.
Changes to the human-readable output of most standard Unix tools are a big deal given the amount of scripts which depends on them. They commonly are seen as fairly stable to the point that even some non-Posix tools like e.g. apt, which doesn't have a stable CLI warns about it to stderr if you try to redirect it's output to a pipe[1].
If anything, though, that's a good reason for a tool like this to exist rather than have every script that depends on these tools use their own, often hacky, parsing of the output.
[1] "WARNING: apt does not have a stable CLI interface. Use with caution in scripts."
It doesn't even need a new release. jc can already fail because of details like the system-language. With my local language, on a simple output of ls -l, it's parsing
{"filename":"drwxr-xr-x 16 root root 4096 Oct 4 11:21 ."}
This is true. However, the issue is with the tool outputting unstructured data to begin with. As the `jc` author says, the best would be if tools supported structured output formats to begin with.
Any custom parser of ls output would potentially have the same problem. Of course, it can be improved though – for example by looking at LANG – and it would be nice for such improvements to get into `jc`, so that other tools can rely on it at least more than doing the parsing directly themselves.
Hi there! `jc` author here. Yes, it is a documented caveat[0] that the `C` or `en_US.UTF-8` locales should be used for best results.
It's not unheard of for tools to require `C` locale for proper parsing:
$ LC_ALL=C ls -l | jc --ls
This is one of many inherent issues with using unstructured text as an API. That's why I believe there should be a JSON (or at least some other widely used format[1]) option for tools that have output that would be useful in scripts.
Why does this site recommend using "paru", "aura" or "yay" to install it on Arch? I have been using Arch for a decade or so but have never even heard of such tools. They don't even have pages in the Arch wiki, and only yay ("Pacman wrapper and AUR helper written in go") is available via the standard repository.
Begs the question: what is so wrong with plain pacman?
`time` needs to be the one to exec the command because they need to know when the command they are timing starts. Therefore, `time` cannot use pipes, as by then, the command being timed would've already started!
`jc` doesn't need to know anything about the command producing the output - just the format of the output. So using a pipe and stdin makes a lot of sense.
I can imagine `jc` having some detection built in, from which it determines the command/content it's being parsed. Doesn't seem to have it, yet, and I'm generally no big fan of "magic" like this, but it would remove the redundancy.
Having it as a pipe, allows for much more, though.
Hi there - author of `jc` here. I originally intended to have auto-detection but put that on the backburner to focus on creating parsers, especially after introducing the magic syntax.
I did implement auto-detection for `/proc` file parsers so you can just do:
$ cat /proc/foo | jc --proc
or
$ jc /proc/foo
But you can specify each procfile parser directly if you want to as well.
I Love using this in streams of data, but there is a lot to be said about the pit falls. Some of them; first you might want to filter with grep first and that leads to missing metadata, second error handling is a good thing but people tend to ignore errors and then not handle them. This is basically what filebeat/logstash from Elastic does, which is a beast at parsing (and impossible to use from command line).
The power of plain text pipes is that you do not interpret them and that makes them fast, that is usefull because you handle both 100 bytes, 1MB and 1TB as input. You choose what you parse keeping it simple, fast and usually error free. This tool miss the, fast, simple and human readable part of debugging pipes. Which is fine!
I am the author of SPyQL [1]. Combining JC with SPyQL you can easily query the json output and run python commands on top of it from the command-line :-) You can do aggregations and so forth in a much simpler and intuitive way than with jq.
I just wrote a blogpost [2] that illustrates it. It is more focused on CSV, but the commands would be the same if you were working with JSON.
A slightly related pet-peeve: I don't like it when "random" commands "squat" the two-letter domain, or worse, the one-letter domain: t, jq, jc etc.
In my perfect world (which, obviously doesn't exist), commands from tools "in the wild" are at least three letters long. With historical exceptions for gnutools: preferably they'd take the three-letter space, but two-letters (cd, ls, rm etc) is fine.
Two letter space outside of gnutools, is then reserved for my aliases. If jsonquery is too long to type, AND I lack autocomplete, then an alias is easy and fast to make. alias jq=jsonquery.
In the case of this tool, it will conflict with a specialised alias I have: `alias jc=javac -Werror`. Easy to solve by me with another alias, but a practical example of why I dislike tools "squatting" the two letter namespace.
I don't believe it's really an issue in practice, your `jc` alias will just take the priority and you can easily add one for jsonquery -> `path/to/bin/jc`.
I think good short command names can help adoption (like for ripgrep, fd) but it's true that we should have a race to squat all the 2 letter names.
I perceive an element of hubris when a tool claims a two-character name. That is a very small namespace, so the tool is effectively staking a claim on mental or emotional real-estate.
This seems a general symptom of unix/POSIX command naming where commands have "always" been short names, and commands are expected to be short names.
It's something I appreciate about the powershell naming conventions. A lot of people mock the verbosity of the names of powershell commands and commandlets which require the "proper" name to be Verb-Noun qebab case monstrosities, but this was chosen for exactly the reasons of your pet peeve: short command names should be user aliases for work in a shell, and longer command names are great for avoiding namespace clashes in scripts and between users. The verbs and nouns create large (discoverable) namespaces.
For instance, this tool might be powershell named ConvertTo-ParsedJson. (ConvertTo-Json is an out of the box command that converts any powershell object to JSON.) It might suggest the user alias it by adding `Set-Alias -Name jc -Value ConvertTo-ParsedJson` but generally commands in powershell only offer such aliases as suggestions rather than defaults. (Though there are a lot of out of the box aliases for common powershell commands.)
It makes sense to me that powershell encourages long names first and allows and encourages users to have far more control over short aliases.
Another vote for something like libXo as the better solution. This thing is just passing the parsing problem of interacting with an unstable API to someone else and hoping they maintain theirs better than you would maintain yours.
to make this great tool truly universal, it has to be written in c instead of python these days, then provide python|javascript|etc bindings if possible.
I'd like to use it on embedded systems, where python is too large to fit. this tool can be widely deployed just like awk|sed|etc but it has to be in C for that.
Agreed but I would change it with "any compiled language that has no external runtime" and common shared libs dependencies. I don't care that a utility is written in Go, Zig, C++ or Pascal ;)
`jc` is available today as a compiled binary with all dependencies self-contained. The binaries and OS package installers are available under the GitHub Releases:
It seems like you have multiple binaries (otherwise, I don't see how you would save space) in a context without an operating system, but do have abstractions like a linker; throwing this out there, you may be able to sort of fake having dynamically linked binaries with transparent disk compression. The shared code gets compressed, and you trade the overhead of your linker for the overhead of your compression, and probably roughly break even.
Seems to me like anywhere Python is too bulky to work, a layer serializing and deserializing JSON at every pipeline step is likely to exhaust your memory too.
135 comments
[ 2.8 ms ] story [ 184 ms ] threadBlog post with examples here: https://blog.kellybrazil.com/2020/08/30/parsing-command-outp...
Discussion: https://news.ycombinator.com/item?id=25006277 366 points | Nov 6, 2020 | 91 comments
Or pipe it into rq [2] to convert the format to yaml, toml etc.
[1]: https://stedolan.github.io/jq/tutorial/ [2]: https://github.com/dflemstr/rq#format-support-status
[0] https://github.com/kellyjonbrazil/jello
I think json has several advantages though. It’s a relatively lightweight and widely known serialization standard, rich enough for most cases and extensible in others, and it has easy to use parsers in all major programming languages.
Also jsonlines is a simple addition that make it easy for json to play well with non-json aware older Unix tools.
It has a few shortcomings but I think its advantages outweigh them, and it’s become a pretty widely used standard in a short time.
I love it
When wrestling with sed/awk in trying to parse results of a shell command, I've often thought that a shell-standard, structured outpout would be very handy. Powershell[0] has this, but it's a binary format - so not human-readable. I want something in the middle: human- and machine-readable. Without either having to do parsing gymnastics.
jc isn't quite that shell standard, but looks like it goes a long way towards it. And, of course, when JSON falls out of fashion and is replaced by <whatever>, `*c` can emerge to fill the gap. Nice.
[0]: https://learn.microsoft.com/en-us/powershell/
Well, yes - powershell passes binary objects but as you can always:
1) access their properties 2) pass them downstream 3) serialize to json/csv 4) instantiate from json/csv
I think this is both human- and machine-readable enough (even through internal format is binary, but working with Powershell you are never really exposed to it).
How do you think it can be improved?
In my opinion object io IS the best part of powershell - it allows us to ditch results wrangling with sed/awk/grep entirely. I'm super interested if there's an even better way forward.
That seems unnecessary. Traditionally, shells have always used text streams. JSON is just text that follows a given convention. Couldn't what you are describing be implemented by setting environment variables or using command line flags?
For example:
PREFERRED_OUTPUT_FORMAT="JSON"
--output-format="JSON"
--input-format="JSON"
Tools that can generate and consume structured text formats are a good idea, but they should be flexible enough that they can even work with other tools that have not been written yet.
"This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface." --Doug McIlroy
You suggesting that I should write `PREFERRED_OUTPUT_FORMAT=JSON ip a | filter "[].address like 192.*"` but that's really verbose and error-prone. It might work for scripts, but for ad-hoc shell I don't like this approach. Ideally programs should be able to communicate between pipes for their preferred formats.
I agree that human-readable text formats should be the default output formats for command line tools, but that easy-to-parse structured text output formats should be easy to specify with either environment variables or command line flags.
If I am writing a script and I am using tools that support a given structured output format and use environment variables or command line flags for output configuration it could work something like this:
This would mean that the command line tools default to human-readable formats, but can still generate JSON or some other structured text format when configured to do so.https://unix.stackexchange.com/questions/197809/propose-addi...
HTTPS://GitHub.com/lmorg/murex
* "Bringing the Unix philosophy to the 21st century (2019)" (https://blog.kellybrazil.com/2019/11/26/bringing-the-unix-ph...) - https://news.ycombinator.com/item?id=28266193 238 points | Aug 22, 2021 | 146 comments
* "Tips on adding JSON output to your CLI app" (https://blog.kellybrazil.com/2021/12/03/tips-on-adding-json-...) - https://news.ycombinator.com/item?id=29435786 183 points | 11 months ago | 110 comments
https://unix.stackexchange.com/questions/197809/propose-addi...
Just like we have stdout and stderr, header lines such as those produced by `ps` should be printed to stdmeta. Curl is the worse offender here, outputing meta lines to stderr instead of stdout. A stdmeta file descriptor would make it clear what is data, what is an error, and what is _describing_ the data.
There's a separate option for counting characters: -m
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/w...
A ringbuffer filetype. Similar to a named pipe file (see: fifo(7)[^1]), but without consuming the contents on read and automatically rotating out the oldest lines.
Of course, there would be some complexities around handling read position as lines are being rotated out from under you.
[1]: https://linux.die.net/man/7/fifo
that seems solvable to me, but multiple simultaneous readers... that seems like it might be a bit more challenging...
If anything, though, that's a good reason for a tool like this to exist rather than have every script that depends on these tools use their own, often hacky, parsing of the output.
[1] "WARNING: apt does not have a stable CLI interface. Use with caution in scripts."
Any custom parser of ls output would potentially have the same problem. Of course, it can be improved though – for example by looking at LANG – and it would be nice for such improvements to get into `jc`, so that other tools can rely on it at least more than doing the parsing directly themselves.
It's not unheard of for tools to require `C` locale for proper parsing:
This is one of many inherent issues with using unstructured text as an API. That's why I believe there should be a JSON (or at least some other widely used format[1]) option for tools that have output that would be useful in scripts.[0] https://github.com/kellyjonbrazil/jc#locale
[1] formats should have good library support across many languages and nice filter/query capabilities from the command-line
Why does this site recommend using "paru", "aura" or "yay" to install it on Arch? I have been using Arch for a decade or so but have never even heard of such tools. They don't even have pages in the Arch wiki, and only yay ("Pacman wrapper and AUR helper written in go") is available via the standard repository.
Begs the question: what is so wrong with plain pacman?
EDIT: Okay so seems they were previously on AUR and once accepted to community repository they just forgot to stop recommending an AUR wrapper for installing: https://github.com/kellyjonbrazil/jc/commit/f2dd7b8815edc92e...
EDIT2: Created a PR with the GitHub.dev editor .. Absolutely blown away by how easy it was! Feels like the future of development.. https://github.com/kellyjonbrazil/jc/pull/310
But this is 100% going in my toolbox - I can think of a couple of scripts that I can update to use this right of the bat!
`jc` doesn't need to know anything about the command producing the output - just the format of the output. So using a pipe and stdin makes a lot of sense.
I can imagine `jc` having some detection built in, from which it determines the command/content it's being parsed. Doesn't seem to have it, yet, and I'm generally no big fan of "magic" like this, but it would remove the redundancy.
Having it as a pipe, allows for much more, though.
OrI did implement auto-detection for `/proc` file parsers so you can just do:
or But you can specify each procfile parser directly if you want to as well.The power of plain text pipes is that you do not interpret them and that makes them fast, that is usefull because you handle both 100 bytes, 1MB and 1TB as input. You choose what you parse keeping it simple, fast and usually error free. This tool miss the, fast, simple and human readable part of debugging pipes. Which is fine!
I am the author of SPyQL [1]. Combining JC with SPyQL you can easily query the json output and run python commands on top of it from the command-line :-) You can do aggregations and so forth in a much simpler and intuitive way than with jq.
I just wrote a blogpost [2] that illustrates it. It is more focused on CSV, but the commands would be the same if you were working with JSON.
[1] https://github.com/dcmoura/spyql [2] https://danielcmoura.com/blog/2022/spyql-cell-towers/
https://github.com/tomnomnom/gron
In my perfect world (which, obviously doesn't exist), commands from tools "in the wild" are at least three letters long. With historical exceptions for gnutools: preferably they'd take the three-letter space, but two-letters (cd, ls, rm etc) is fine.
Two letter space outside of gnutools, is then reserved for my aliases. If jsonquery is too long to type, AND I lack autocomplete, then an alias is easy and fast to make. alias jq=jsonquery.
In the case of this tool, it will conflict with a specialised alias I have: `alias jc=javac -Werror`. Easy to solve by me with another alias, but a practical example of why I dislike tools "squatting" the two letter namespace.
E.g.:
It's something I appreciate about the powershell naming conventions. A lot of people mock the verbosity of the names of powershell commands and commandlets which require the "proper" name to be Verb-Noun qebab case monstrosities, but this was chosen for exactly the reasons of your pet peeve: short command names should be user aliases for work in a shell, and longer command names are great for avoiding namespace clashes in scripts and between users. The verbs and nouns create large (discoverable) namespaces.
For instance, this tool might be powershell named ConvertTo-ParsedJson. (ConvertTo-Json is an out of the box command that converts any powershell object to JSON.) It might suggest the user alias it by adding `Set-Alias -Name jc -Value ConvertTo-ParsedJson` but generally commands in powershell only offer such aliases as suggestions rather than defaults. (Though there are a lot of out of the box aliases for common powershell commands.)
It makes sense to me that powershell encourages long names first and allows and encourages users to have far more control over short aliases.
https://github.com/Juniper/libxo
I'd like to use it on embedded systems, where python is too large to fit. this tool can be widely deployed just like awk|sed|etc but it has to be in C for that.
https://github.com/kellyjonbrazil/jc/releases
This is still python under the hood and not as small of a binary as I would like, but it does work.
But yeah, for these types of utilities, relying on an external language runtime like Python/Node is pretty rough.
all I need is that rust's stdlib can be linked dynamically just like c/c++/java/whatever, if that happens I'm ready to switch to it.