HN Explorer

Show HN: gq – like jq or zq, but you use Go (github.com)

107 points by errnil ↗ HN
Hi HN. I've gotten pretty tired of needing to learn a custom programming language for a tool I use once a week or less. So I figured, it might be easier to pick up if `jq` used a programming language I already know. Voila, gq.

73 comments

[ 2.7 ms ] story [ 133 ms ] thread
eatonphil
Nice work! I also learned about gojq [0] yesterday too. A Go clone of jq that you can also embed in Go code.

[0] https://github.com/itchyny/gojq

Edit: oh unlike jq/gojq your tool doesn't implement the jq language it allows you to script in Go. Cool! And interesting to see your use of the Yaegi interpreter for interpreting the Go.

I'm curious, did you find it faster to interpret with Yaegi vs just writing the Go code to a temp file and build/running that? I imagine at some point for larger datasets the latter approach would be faster?

errnil
Precisely. The first pass generated a temp file and compiled/ran it, but that lead to annoying pauses when trying to run simple scripts (probably 500ms latency? Maybe a tad more?). Yaegi cut that latency dramatically. I think you’re right that this may make it slower for larger scripts or for sufficiently large inputs, however the biggest bottleneck for me with jq isn’t performance but learning it’s language repeatedly.
tedunangst
yaegi is pretty fast. Not compiled fast, but it can chug through some pretty large tree walks without trouble. (Using it for walking html processing.) Half the work is parsing, which is using the compiled library.
eatonphil
Curious what is your scenario for using yaegi and not Go itself?

You allow users to run arbitrary Go somewhere?

bachmeier
Looks nice. I might even use it if I wrote Go programs.
johnday
I'd give it maybe a day and a half before `hq` makes its way onto the front page.

No guesses which language the `h` stands for :)

ldh
Surely `rq` can't be far behind.
wey-gu
Haha 100% sure there will be rq…
eatonphil
I think the joke was that it already exists.

https://github.com/dflemstr/rq

ducktective
rq is criminally underrated. It is a universal convertor between json, avro, cbor, message-pack, toml, yaml and csv.

It misses XML though, and activity in development ;)

mynameisash
I was going to post this the other day, but my project[0] isn't quite the same as jq. I didn't even know about jq when I had a need for some easy JSONL text extraction. No aggregation or complex querying -- just extracting from arrays/objects in my data.

Sorry to disappoint? :)

[0] https://github.com/aeshirey/jsx

adolph
Two maybe: one for the systems folk and another for data scientologists
ricardobeat
Go is a fine language, but not great for scripting. The verbosity shows in the examples, it’s basically skipping five or six lines of code to read from stdin and import a functional library.

Learning the jq syntax will have a much better payoff in the long run.

nh23423fefe 

    | gq 'j.Filter(func(n *Node) bool { return strings.Contains(n.String(), "good/") })'
    | jq 'map(select(contains("good/")))'
throwaway894345
I like jq, but the parent is right that it's hard to wrap one's head around the way jq thinks about things and how that model maps to its grammar. Even in this example, I couldn't tell you why `map()` wraps everything else, and I always get tripped up on when to use `|` or how to go back and forth between a list and a stream. I don't think the answer is `gq`, but I certainly understand why people try out other languages.
nh23423fefe
every filter operates on its input. in this case the input was an array.

every map implementation ive seen has type function(a,b) -> list(a) -> list(b)

select(contains("good/")) is a function from string -> string|void

so map plus a voidable function equals a filter

| just means connect 2 filters

if you have a stream you turn it into a array with the array constructor []

if you have an array you turn it into a stream with the producer operator .[]

the api is super orthogonal and well designed. but you do have to understand the model

throwaway894345
Right, I already believe it all makes sense when you understand the model, but the model isn't super obvious and I often find myself Googling things for reference because I don't use it very often; however, I do use regular programming languages and I can reason about those more easily (although while I like Go, I would be more inclined to used a Python- or JS-like language for something like this).
errnil
No doubt this tool is lacking. However I will say that for me the most important quality in my `jq`-like is not conciseness. My main frustration is that I always forget the small details of how `jq`'s language works and spend 10 minutes googling. E.g doing operations within arrays of objects, using select, and so on. Maybe I'm just dumb, but I'm fond of tools that accommodate my dumbness!
sleepybrett
Also this seems to mean my pipeline needs to know how to compile go if i want to transform some json .. boo hiss
errnil
Actually `gq` does not require a locally installed Go compiler. It interprets the supplied Go using Yaegi.
spacemanmatt
i can get the same jq from multiple package installers that will ensure it is in my execution path. this covers windows and linux, and likely osx if i needed it.
dmitriid
Ok. Is this the new "todo list" or "hacker news clone" trend, but for tools? :)

We need competing implementations in more languages!

(Ignore my snarky tone because: why not?)

pipeline_peak
Why not just pipe into a python program that uses a mature JSON parser? These “*q” things are really boring.
eatonphil
Is go's json parser less mature than python's?
pipeline_peak
I was under the assumption that this thing is using it's own parser, I guess it doesn't, idc. The point I'm trying to make is that JQ depends on the user investing the time to know an esoteric domain language. This thing is fed by Go scripts that get interpreted, at that rate just use a powerful shell language like Python.
tedunangst
What if I'm more familiar with go and python is an esoteric language?
pipeline_peak
In the world of shell scripts, python isn't an esoteric language....
joshbaptiste
because many folks just want a single binary and not a full language library
pipeline_peak
If you are using whateverq and writing a shell script, everyone else using it needs to have whateverq installed as well....
wpietri
You could use the same logic to ask yourself why there should be any command-line tools at all, just python libraries.

Python is my daily driver, but I still use JQ when I'm doing something quick. If you try some complex examples, I'd bet you'll quickly figure out why.

sleepybrett
Why do I want to bloat my pipeline container with a fucking python install including all the libraries? I'd rather pop a single tiny binary in there. Container bloat = slower pipelines.
pipeline_peak
If you’re working in a Unix environment within a container isn’t there a high likelihood there already is Python as well?

I never said every single library, also Python scripts can run as standalone executables.

If you truly can’t have Python because it’s too bloated, then JQ in shell scripts is a fair use case, that just sounds so rare.

adolph
True, high likelyhood there is _a_ Python but maybe not the Python your script needs. The situation is better nowadays but 2.7 persisted for a really long time.
[deleted]
(comment deleted)
[deleted]
(comment deleted)
zamalek
> Im tired of learning new programming languages.

Hear hear! Building bona-fide compilers that have interesting concepts, or just for fun, is a worthwhile thing to do. But these one-off languages are becoming ridiculous.

* buildah shows just how much of a headache Dockerfiles are. We already have a whole ecosystem of awesome scripting languages, such as bash, fish, zx, etc. Instead, we have to deal with the incredibly constrained (and harmfully magic) behavior of docker build.

* the whole implerative YAML ecosystem. I have been playing around with Dagger (CUE, a language built to solve real problems), and what a joy compared implerative CI.

* lest we forget how JavaScript came to be: https://thenewstack.io/brendan-eich-on-creating-javascript-i...

yessirwhatever
> Hear hear! Building bona-fide compilers that have interesting concepts, or just for fun, is a worthwhile thing to do. But these one-off languages are becoming ridiculous.

I don't understand this mentality. Some programmers enjoy writing programs and some enjoy learning about programming languages. No one is telling all programmers they they need to learn about every programming language, but why paint the whole endeavor as frivolous?

zamalek
From what you quoted:

> Building bona-fide compilers that have interesting concepts, or just for fun, is a worthwhile thing to do.

dijit
I've been a sysadmin (now, devops, SRE, whatever) for over 15 years now, so when you say:

> No one is telling all programmers they they need to learn about every programming language

I cringe a little.

There are about 25 different DSL's in my head from configuring various desired state configuration systems or services which use DSLs, there's countless config file formats ranging for ZONE files to TOML to YAML to HCL and on and on and on.

Makefiles, Dockerfiles, Ninja files, Jenkinsfiles etc;etc;etc;etc;

DevOps of Yore had to know bash and perl, then it was Ruby, then python, now Go. The treadmill continues, it's _necessary_ to learn those things because you _must_ interact with those things.

That's before we get into special languages like AWK which had significant prominence when I was beginning life as a sysadmin too.

Nobody is forcing programmers to know every language, but honestly, there's a lot that sysadmins had to learn, and with the rise of DevOps, developers are the SysAdmins of tomorrow: so you have to know what we knew.

yessirwhatever
I feel your tiredness.

But I don't understand why you'd say it'd make you cringe.

You can have a discussion about how the industry (usually motivated by non-technical reasons) is moving towards. I was talking specifically about those "one-off" languages that some people find fun to investigate and/or work on.

dijit
I cringe because you said that programmers don’t have to learn many languages.

But in order to do a good job a sysadmin has to learn the semantics of dozens.

So you’re right, in a way, but it’s not right at all for sysadmins.

Since sysadmins are being essentially replaced by programmers, it won’t be right for programmers soon either.

cpuguy83
I agree that Dockerfile has some constraints, but I'm not sure I get the "magic" part. It's also not like it's a full language. Run a command, set an env var. There's some additional flags to set for optimizations. Otherwise minimal things to get you to an env in a container that you can do whatever you want.

Also +1 to dagger+cue. I like that it gives me essentially for full power in buildkit's llb exposed as cue configs.

zamalek
> I'm not sure I get the "magic" part

The problem I was trying to solve was container signing, as well as building AMD64+ARM64 multi-arch images without buildx having access to both contexts, i.e. on different VMs (a CircleCI limitation). When you use buildah without bud you learn just how many implicit steps are involved in a Dockerfile, and how those implicit steps have been the problem all along.

buildx works great when you're on the razor thin happy path. The abstraction is great for the majority who need it but, crucially, Dockerfiles provide no way to escape the abstractions.

forty
My latest story with Dockerfile:

my.tar contains files owned by user 0, I want them to be owned by nobody:

  USER nobody
  ADD --chown nobody:nobody my.tar /app
Damn, --chown does not work when untaring and doesn't fail either but the files end up being owned by root. Fine, I'll just chmod

  USER nobody
  ADD my.tar /app
  RUN chmod -R nobody:nobody /app
Now it fails (of course) because the files are owned by root and the command is ran as nobody.

While it all makes sense, I'm not sure why it's logic to have the ADD command always run as root and the others following the USER directive (yeah and I know, it's all my bad for using ADD rather then COPY)

zamalek
> RUN chmod -R nobody:nobody /app

Did you know: RUNning chmod increases the size of the layer by the size of the file being chmodded[1]. Just some more of that magical secret sauce.

(With buildah you choose when to create a layer, so both the COPY and the chmod can be layered together)

[1]: https://blog.vamc19.dev/posts/dockerfile-copy-chmod/

forty
Ah well, no I did not know, thanks :) in my case it's not too bad because I use kaniko's option to make a single layer of my changes, but good to know.
pensatoio
I love that you wrote this last night in response to the recent front page. :D What a cool project!
errnil
:) Thank you! It was fun to write.
voidon
This is cool and I agree about not needing to learn yet another language. I love jq, but I always have to look at the docs. I recently learned about ctrl-x-e (after 25 years of shell-use!) to edit the one-liners in $EDITOR, so verbose go code might actually work.
vemv
I keep using "My jq killer" as I commented back in 2016:

https://news.ycombinator.com/item?id=13091469

sleepybrett
Now my build/ci/cd container needs fucking ruby in it .. no thanks. I broke myself from my 'just use perl instead of awk/sed' for the very same reason. The smaller those containers are the faster my pipeline runs.
renewiltord
I use https://github.com/itchyny/gojq . I like `jq`'s functional style, but I don't like the way it breaks number precision (or used to).

However, for deep `jq` selects I use `<file.json | gron | grep key -C n | norg` where `norg` is `gron --ungron`.

I also prefer `gron` for ad-hoc pipelines and `jq` for longer-lasting ones.

sleepybrett
Firstmover goes to `jq` all these other tools might be great but honestly it might just be better to use the dev time to add features to jq.

For instance json is a subset of yaml, maybe internally jq they could use yaml libraries, consume either, output either json or input format by default and give me a flag for yaml output if I want it.

As a guy who does a bunch of devops and works with a bunch of teams to improve their pipelines etc. jq is a tool they all have and i always find in pipelines having to install the new jq flavor of the month every six months is terrible.

kitd
For instance json is a subset of yaml, maybe internally jq they could use yaml libraries, consume either, output either json or input format by default and give me a flag for yaml output if I want it.

As it happens, gojq, mentioned elsewhere in this thread, adds this feature, and quite a few others.

Jq has served the industry admirably, but there's always room for improvement.

cabirum
Do we even need a tool like that? I mean, just use the language you know to do something with the json you have.

Alternatively: the tool should use sql.

dhagz
Just load the JSON into Mongo/Cassandra and query that. /s
IceWreck
Can someone explain why people are creating so many CLI tools (with a very weird and custom syntax) to extract JSON using a very complicated one line command when you can make one off scripts to do the same using python/go/ruby/node/whatever ?
QuikAccount
I'd argue that it is less about practicality and more about learning a language by recreating a CLI tool.
UncleEntity
I wrapped boost::property_tree in a python module to read in a rss xml feed and store the items in a json file to track the TV shows I download all because the pure python version was kind of slow.

The amount of time I spent on getting all that working is probably a lot less than waiting a few extra seconds on a python script I run every few days so…

NAR8789
oneliners are generally nicer for fast data exploration than one-off scripts. With a oneliner I can get started instantly, and build incrementally. With a one-off script my first step is boilerplate input handling.

Maybe a nice middle ground would be for more language runtimes to have an equivalent to perl's `-p` flag. https://perldoc.perl.org/perlrun#-p

This flag is so convenient that I mostly use `perl -pe` instead of `sed` for regex search-and-replace.