I could see it being useful when using grep or writing awk scripts. For the latter you usually have to carry around some state if you want to deduce the path to a desired leaf node.
I could also see it useful for just finding the index of a particular array entity.
Author here. I use it all the time when working with jq [1] to find the path of the nodes that I want to select or filter. It also makes it much easier to understand the structure of deeply nested JSON files.
Have you seen gron[0]? It's similar: flattens JSON documents to make them easily greppable. But it also can revert (ie, ungron) so you can pipe json to gron, grep -v to remove some data, then ungron to get it back to json.
Maybe GP meant that jq can do selection as well, i.e. that grepping is redundant after jq. But jq is much more complicated to learn and grep works on all inputs (not just json), so it makes a lot more sense to learn and use grep properly.
jq seems very powerful. I don't deal with json all that often and my most common use case (by far) is `jq '.' -C` and it took a few tries for me to remember that syntax.
The idea of flattening, grepping, then reverting sounds very appealing and sounds like a better fit for me.
It does look like neither are needed if you pipe a file in jq, but `jq . file.json` requires the `.` and if you're pipeing into a pager, like less, you need both `.` and `-C` to get colored output (that was the case with the alias I had pulled up). I am using 1.5 and haven't looked to see if 1.6 changes this.
`-C` would be required when piping because most of the time (with the exception of piping into less) when stdout is not a terminal, it doesn't make sense to include terminal color escape sequences. You'd end up with those codes in your files, and grep would be looking at them for matches, for example.
`.` would be required when passing the file as an argument instead of stdin, because jq interprets the first argument as jq-code. If you don't include `.` it would interpret the filename as jq-code.
`.` is still needed if I'm pipeing in json--but only when I'm piping out. Otherwise help goes to stderr and nothing goes to stdout.
I do honestly think jq is a cool and powerful tool. I also appreciate little things like auto-color when appropriate--git also does this. Git also uses your pager, which might trivialize my personal use case.
I find it useful when I don't know what the json schema is.
Then you can just do a quick gron + grep and find where the interesting parts of a large json document are.
There are cases when you have some complicated json and just want to search for stuff. Then you use grep + gron.
There are cases when you want a complete json processing tool. Then you use jq.
You can probably simulate each approach with the other approach, but the code needed to this is just too tedious to write. So you use whatever tool fits your use case.
That's actually a nice lifehack. Much simplier than jq. Unfortunately, would be harder to make all kinds of logical conditions for which jq allows (even if not that intuitively).
It still feels like there must be something in between, some way to make queries with json more naturally, than with jq, yet with enough power.
jq is certainly a unique language, which makes it unfamiliar to work with. Intuitiveness and natural feeling comes when one has gotten familiar with it after a bit of practice and reading the manual, though. It's a very well thought-out language. A very nice design.
It might help to recognize how it's influenced by shell languages and XPath, if you're familiar with those.
Well, no arguing there, it is indeed. And I use it from time to time. However, it's not like I need a tool like that every day, and if I'm no using it for a week I usually need to "learn" it all over again.
While I don’t disagree with you, if you look at the comment above this you’ll see @enriquto mentions just dumping lists to a text file. Pretty sure that’s what was implied by ‘unix’: use a simple text file that is dumped to the fs that can be easily parsed by standard tools on any Unix-like.
JSON is usually mocked by people who don't get to use Javascript objects natively. The benefits aren't obvious or tangible to them.
XML is native to nothing. It's annoying to use everywhere.
UNIX tools just don't compare. Sure, they stick around, because sometimes they're the easiest tool for some job, until they aren't and you regret starting out with them.
In a few years, Javascript and Javascript-compatible languages are likely to be bigger than ever before. A whole generation of developers has been trained mainly on them. Billions of dollars have been invested into the ecosystem. Whether we like it or not, that's the reality. For that reason alone, JSON will stick around.
In the very rare cases that you really need to serialize complicated data structures, something like s-expressions or (gasp) json, or even a memory dump is perfectly appropriate. However, most of the times your data structures should be lists of numbers or lists of strings. For those, you do not really need a "format", you can simply print and scan them from a text file.
If you come from xml hell, then yes, json may even seem a reasonable option. But if you are used to the terse beauty of printf and scanf, then a json file looks like bad sarcasm.
I don't see what jsonlines has over yaml; in fact the jsonlines examples presented there are almost trivially converted to yaml.
e.g. the first example is valid yaml if you add a '- ' to each line.
And JSON is (almost) a perfect subset of yaml.
I've been using csv lately. It's reputation is overstated.
What I like is that it's far more compact than yaml or json and trivially pulled into sqlite for ad-hoc queries.
One thing I like about JSON Lines is robustness to bad data - if an individual line is corrupted, you can discard it / print a warning and move on, and the parser can start again at the next newline. This makes it useful for log messages / metrics, because if something crashes while emitting a log line, you can recover. If something crashes while emitting an item in a YAML list, you might corrupt the entire rest of the document.
Another is that it makes streaming processing a little easier. Once you have a line, you know you can attempt to process it, and you can shard processing on newlines without a full YAML processor. Tools that work on newlines or tools that can just split on lines can handle the first level of JSON Lines output.
This is really similar to the format that AWS uses to represent recursive structures (arrays, maps) as a single array of key-value pairs for their APIs. Your catj could potentially be used to create that if ever working with the raw API directly instead of a SDK.
The output format like a slightly better INI (in that it would support deeper hierarchies), although arrays would be a pain to write an index at a time like that.
And for purely aesthetic, nitpicky reasons I think the leading period in each line is redundant.
I was expecting some mention of jq for this post, and you did not disappoint. Thank you for the script -- it works great and I'm adding it to my collection.
That's insane. Here I was thinking about how much more challenging it would be to parse and reconstruct the object from jq, and you got the idea to take advantage of the syntax similarity to parse it as jq code itself. Nice. And so, that means the inverse of the jq code I posted would simply be:
Parsing is awkward in jq, but setpath(PATHS; VALUE) will create necessary structure. PATHS uses the array form, like ["movie", "cast", 5] not .movie.cast[5]. Since 1.5, jq has PCRE regex, so could remove ], and separate by . and [.
This would be more suitable to large json files if used with the `--stream` flag. Here's my take on it:
jq -c --stream '
. as $in
| select(length == 2)
| (
$in[0] | map(
if type == "number"
then "[" + tostring + "]"
else "." + .
end
) | add
) + " = " + ($in[1] | tostring)'
Using `--stream` allows jq to start before parsing the entire json file. In my experience, a 700mb json file can take up 5gb of ram in either jq or python -m json.
You're right about `--stream`, but you didn't need the variable assignment. Also, `-c`, besides the fact that it's not available in jq-1.5 which some people are using, is very pointless in this situation, since we're looking to output text, and `-c` is for outputting objects/arrays in a compact format. The fact that `-r` wasn't used causes jq to output the text encoded as json strings. So, instead of outputting:
".movie.name = Interstellar"
".movie.year = 2014"
".movie.is_released = true"
".movie.else = Christopher Nolan"
".movie.cast[0] = Matthew McConaughey"
".movie.cast[1] = Anne Hathaway"
".movie.cast[2] = Jessica Chastain"
".movie.cast[3] = Bill Irwin"
".movie.cast[4] = Ellen \\\\ Burstyn"
".movie.cast[5] = Michael Caine"
Another point is how the strings at the right of the `=` are displayed. They should be quoted. The reason why they're not is because you piped the second element to `tostring` instead of `@json`.
A better version of your suggestion would've been:
jq -r --stream '
select(length > 1)
| (
.[0] | map(
if type == "number"
then "[" + tostring + "]"
else "." + .
end
) | add
) + " = " + (.[1] | @json)
'
The use of `length > 1` instead of `length == 2` is a minor point, but if a future version jq decides to sometimes put 3 elements in these arrays, your filter would ignore those when we're likely to also want those. `length > 1` ensures what we need, that there are at least the elements that we're going to be using, while `length == 2` might filter some of those out, even if it's not right now.
Your use of `add` is neat, though. I wouldn't have thought of that.
Thanks, but I haven't really done much in jq before. Doing the above script involved looking a lot through the manual and experimenting. This exercise was a learning experience for me. I was only able to do this because the manpage is well written.
macOS appears to support multiple args just fine. Which is why it annoys me that Shellcheck bitches about using more than one arg even though I'm writing a script for macOS specifically.
ShellCheck is right insofar as compatibility is concerned. You can only rely on the shebang supporting one argument. I'd personally just ignore that warning if I were writing for MacOS specifically, but you can configure ShellCheck to ignore certain errors that you don't care about[1].
Yeah, but it's simpler just to move the additional args to `set` calls than to remember the syntax for disabling the directive. It's just irritating, and especially so because, for some reason, in VSCode it ends up highlighting literally the entire file as an error instead of just the shebang.
You whitelist against what the syntax allows for identifiers and then you blacklist reserved keywords. Writing it this way makes it easier to verify for correctness when comparing with the ECMAScript Specs. This is still a non-exhaustive blacklist and the whitelist regex lacks allowed unicode characters.
Hmm... I just downgraded to 1.5 and it seems to work. jq just has 2 numbers in its versioning[1]. The other number must be from your distribution's package building. Maybe the issue is with something your distribution did while building the package (like adding a patch)? It might also be that the syntax error is not with the script, but with the JSON you input. Sorry, without being able to reproduce the error, I can't help more.
Like another thread mentioned, shebang (#!) parsing is non-standard. In macOS, I think what you tried would work like you'd expect, but it'd work differently on linux. The reason is that in linux, after parsing the path to the executable and a space, everything else is taken as a single argument. So if you were in bash, what you did would be the equivalent of doing:
jq "--stream -rf" path/to/script
and jq doesn't know of any one option called "--stream -rf".
I haven't seen the discussions around these design decisions in the different OSes, but I imagine the crux of the matter is that you have to pick somewhere to stop, and where you chose to stop is largely arbitrary.
I mean, you can have the OS interpret shebangs with multiple arguments, but then you'll want to be able to put spaces in these arguments, so you'll want quoting, and then you'll want to put special characters like newlines inside, so you'll want escaping, etc.
The OS can implement all these things in execve()'s logic, but it might also be preferable to keep the logic simple in the interest of avoiding security-harming bugs. You know, less code, less bugs, less vulnerabilities.
If --stream had a single letter option equivalent, you could stick it together with the other ones. However, since it doesn't, your only option to make a portable script is to use a shell shebang like #!/bin/bash, and then do:
exec jq --stream -rf ...
You might feel that this single argument restriction sucks and is definitely inferior to any implementation of multiple argument shebangs. I don't know if macOS shebangs support quoting, but if they don't and simply split on spaces, then I can tell you they can't do hacky stuff like writing code in a shebang like this:
What an interesting idea! On my last contract, I had to deal with 100mb+ JSON responses from a barely-documented API. This would have come in handy when I was figuring it out.
I've used JSONExplorer for this purpose, but it is web based and doesn't handle files this large.
Extending the filesystem metaphor to JSON data and re-using the same commands strikes me as a great idea.
Did another project inspire you, or did you come up with the concept yourself?
This looks cool! I'm curious: did you consider using FUSE for this, instead of taking a shell? Using FUSE might make installation harder for less technical users, but the upside is that they could use their choice of file manager (shell commands, curses-based or GUI).
As you've already implemented most relevant commands (cd, ls, cat), it would probably be easy to make a FUSE version using fs-fuse / fuse-bindings
Hm, essentially JSON->Properties file. Cool. I wrote a little script the other day and decided Properties format was a pleasant way to define the config, maybe this could dovetail with similar future endeavors.
126 comments
[ 0.20 ms ] story [ 167 ms ] threadI could also see it useful for just finding the index of a particular array entity.
[1] https://stedolan.github.io/jq
[0] https://github.com/tomnomnom/gron
The project linked to is from 2014 with last update in 2015.and it is on NPM...
What is left to say? Thank you!
[1]:https://github.com/stedolan/jq
The idea of flattening, grepping, then reverting sounds very appealing and sounds like a better fit for me.
I don't think you really need neither `.` nor `-C`. Just `jq` seems to do the same colored output of the input by default.
`-C` would be required when piping because most of the time (with the exception of piping into less) when stdout is not a terminal, it doesn't make sense to include terminal color escape sequences. You'd end up with those codes in your files, and grep would be looking at them for matches, for example.
`.` would be required when passing the file as an argument instead of stdin, because jq interprets the first argument as jq-code. If you don't include `.` it would interpret the filename as jq-code.
I do honestly think jq is a cool and powerful tool. I also appreciate little things like auto-color when appropriate--git also does this. Git also uses your pager, which might trivialize my personal use case.
There are cases when you have some complicated json and just want to search for stuff. Then you use grep + gron.
There are cases when you want a complete json processing tool. Then you use jq.
You can probably simulate each approach with the other approach, but the code needed to this is just too tedious to write. So you use whatever tool fits your use case.
It still feels like there must be something in between, some way to make queries with json more naturally, than with jq, yet with enough power.
It might help to recognize how it's influenced by shell languages and XPath, if you're familiar with those.
XML is native to nothing. It's annoying to use everywhere.
UNIX tools just don't compare. Sure, they stick around, because sometimes they're the easiest tool for some job, until they aren't and you regret starting out with them.
In a few years, Javascript and Javascript-compatible languages are likely to be bigger than ever before. A whole generation of developers has been trained mainly on them. Billions of dollars have been invested into the ecosystem. Whether we like it or not, that's the reality. For that reason alone, JSON will stick around.
Related, if you want more of a csv-style, see JSONLines. aka "newline-delimited JSON"
http://jsonlines.org/
And JSON is (almost) a perfect subset of yaml.
I've been using csv lately. It's reputation is overstated.
What I like is that it's far more compact than yaml or json and trivially pulled into sqlite for ad-hoc queries.
Or spreadsheets, to work out a plan, and then MySQL.
Another is that it makes streaming processing a little easier. Once you have a line, you know you can attempt to process it, and you can shard processing on newlines without a full YAML processor. Tools that work on newlines or tools that can just split on lines can handle the first level of JSON Lines output.
And for purely aesthetic, nitpicky reasons I think the leading period in each line is redundant.
EDIT 2: For those who want to save this script, you can put just the jq code in an executable file with the shebang:
BTW the input json can be null, so -n works (also using process substitution):
A better version of your suggestion would've been:
The use of `length > 1` instead of `length == 2` is a minor point, but if a future version jq decides to sometimes put 3 elements in these arrays, your filter would ignore those when we're likely to also want those. `length > 1` ensures what we need, that there are at least the elements that we're going to be using, while `length == 2` might filter some of those out, even if it's not right now.Your use of `add` is neat, though. I wouldn't have thought of that.
https://gist.github.com/fernandoacorreia/4b67a41bbe227654868...
Maybe not, but I'm pretty sure every system supports a single arg. And very few (none?) support more.
[1] https://github.com/koalaman/shellcheck/wiki/Ignore
From https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...
> If the first line of a file of shell commands starts with the characters "#!", the results are unspecified.
There is no more specification for "#!/usr/bash" than "#!/usr/bin/jq -jf"
The exec page provides even fewer words about how to interpret shebangs if you thought perhaps I was linking to the wrong portion of the posix spec
[1] https://github.com/stedolan/jq/releases
tostream | select(length > 1) | ( .[0] | map( if type == "number" then "[" + tostring + "]" else "." + . end ) | join("") ) + " = " + (.[1] | @json)
`#!/usr/bin/jq -rf ` with tostream wrapper in code works fine
I haven't seen the discussions around these design decisions in the different OSes, but I imagine the crux of the matter is that you have to pick somewhere to stop, and where you chose to stop is largely arbitrary.
I mean, you can have the OS interpret shebangs with multiple arguments, but then you'll want to be able to put spaces in these arguments, so you'll want quoting, and then you'll want to put special characters like newlines inside, so you'll want escaping, etc.
The OS can implement all these things in execve()'s logic, but it might also be preferable to keep the logic simple in the interest of avoiding security-harming bugs. You know, less code, less bugs, less vulnerabilities.
If --stream had a single letter option equivalent, you could stick it together with the other ones. However, since it doesn't, your only option to make a portable script is to use a shell shebang like #!/bin/bash, and then do:
You might feel that this single argument restriction sucks and is definitely inferior to any implementation of multiple argument shebangs. I don't know if macOS shebangs support quoting, but if they don't and simply split on spaces, then I can tell you they can't do hacky stuff like writing code in a shebang like this:> https://unix.stackexchange.com/questions/365436/choose-inter...
Granted, it's bad practice, but a little cool nevertheless.
https://blog.tedivm.com/open-source/2017/05/introducing-json...
I've used JSONExplorer for this purpose, but it is web based and doesn't handle files this large.
Extending the filesystem metaphor to JSON data and re-using the same commands strikes me as a great idea.
Did another project inspire you, or did you come up with the concept yourself?
Have you done a Show HN yet?
As you've already implemented most relevant commands (cd, ls, cat), it would probably be easy to make a FUSE version using fs-fuse / fuse-bindings
https://github.com/twpayne/flatjson
The flat format is great for diffs:
jq can also do the same thing, with more flexibility. And it is possible to combine with bash alias to make it indistinguishable from catj
... or you know, you could put the jq script in an executable file and add a shebang like
or In my opinion, aliases should mostly be used to add default options only. Not really to insert whole scripts into them.https://sqlite.org/json1.html#jtree