Have you considered adding css sectors as an alternative to xpath? For many simple things a css selector is easer to write and more people already know them.
I believe it's possible to translate css selectors to xpath so it wouldn't need another selection engine.
Thank you for your support!
I didn't consider CSS selectors as an alternative to Xpath, but really like the idea. Absolutely agree CSS selectors will be more convenient :)
The issue with CSS selectors is they can only express element selection. There's no concept of picking the text of the element or the value of an attribute. There's been proposals over the years to support pseudo elements for that, so you could make something up like ::attr(name) and ::text, but nothing in the standards.
htmlq does this by having various command line options like --attribute and --text.
Quite a few scraping (such as Scrapy) or browser automated testing frameworks (like Playwright) have standardised on ::text. I would take a look around and if other tools are using the same none standard selectors for these features, it would be perfect for Xq. It's not like you are complying with standards for browser compatibility.
This is awesome. Hugo has been missing a tidy feature for a long time. I hope they use this to implement it finally or that the author might even consider creating a PR.
There is also yq [1], which attempts the same for yaml, toml and xml. (And confusingly also contains a binary named "xq" for the xml part - however, it uses jq for querying instead of xpath)
Ah ok; Namespace support before XPath 3.0 with Q{nsname}localname would make cli tool like this a bit too verbose. Of course you could store ns-prefix to ns mappings to conf file or environment variables...
I actually tried the lxml syntax "/{urn:beta}b" and that's how I learned about the panic-based error handling, but I didn't know about that Q syntax, so thank you for bringing it to my attention
$ echo '<a xmlns="urn:alpha">hello</a>' | xq --xpath '/Q{urn:alpha}a/text()'
panic: /Q{urn:alpha}a/text() has an invalid token.
goroutine 18 [running]:
github.com/antchfx/xmlquery.Find(...)
github.com/antchfx/xmlquery@v1.3.8/query.go:76
...
There aren't that many libraries available with full XPath 3.x conformance. In order to give some constructive feedback, perhaps an optional .conf file with ns-prefix->ns-name mappings could be read and passed to the XPath engine as ns-context? Even w/o the XPath functions support and only XPath selector expression this would make the tool to cover most of the usecases eg. simply finding elements.
Definitely seconding pup, love it and it's easy to use with css selectors. Often use it to parse HTML tables of data on random websites into usable CSVs / etc.
xq looks interesting for pure XML, will add it to my notes.
Regarding xmlstarlet, I don't use it much but here's some stuff from my notes:
XML filter on xpath: xmlstarlet sel -t -c configuration/appender logback.xml; echo
XML editing: xmlstarlet ed -d configuration/root -d configuration/logger -s configuration -t elem -n woof -v 'Woof!' logback.xml
The last one deletes elements given by xpath "configuration/root" and "configuration/logger", then adds a new element with the contents <woof>Woof!</woof> under the xpath "configuration".
The pup readme looks pretty great, but a heads up that the project has had limited activity over the past few years. Several raised issues about problems compiling with latest Go releases.
Yeah, 234 forks is not a good look, but that said `brew install -s pup` with `go version go1.19.3 darwin/amd64` reports a-ok so I'd guess it just doesn't have a `go.mod` like normal go developers are used to, and that's why the brew formula moves it into the old style GOPATH setup: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/...
Yes, I've generally used curl | xmllint --html --xpath for this (usually piped to awk). Not always very pretty, but seems pretty available on MacOS / Linux.
Hadn’t heard of XQilla, but looks like it’s limited to XPath 2.
You should try an XPath 3 based implementation. XQuery 3.1 has some nice features not found in XPath 2 implementations e.g. native JSON support, the arrow operator (piping), group by clause in FWLORs. XSLT 3.0 has some nice new stuff too, but I use that less
I would switch but, finding a simple binary that does what I want (input -> complex query -> output) is so frustrating. Everything is a framework or library.
Thankfully XQilla has its own json support which was all I needed so far
it works if you need to not use the structured nature of the document
for $mp in /ls:live_stream_config/ls:mountpoints/ls:mountpoint
for $server in $mp/ls:servers/ls:server
return { concat(
"https://", $server/ls:ip, "/", $mp/ls:mount, ".",
local:codec-to-ext($mp/ls:media-format/ls:audio/@codec)
) }
is the core of my xqilla work, its something I could replicate in jq, I don't think I could replicate it in xq however
According to the README on Github main page for the project, you need to provide the -m flag when working with HTML.
— The tool parses XML by default, which is much stricter than HTML. Hence the error you are seeing (in HTML5, one can omit various close tags and the HTML is still 100% valid — that's what you are encountering here)
So there are now approximately 1 million command line tools with various overlapping feature sets for extracting data from XML, JSON, YAML, TOML, and CSV-like delimited data. Is anyone ambitious enough to have constructed a feature matrix for all of them? Is there even a complete list out there?
I wonder if this space of tools is a bit like static site generators: it's almost as easy to write your own as it is to learn somebody else's.
35 comments
[ 2.3 ms ] story [ 82.8 ms ] threadHave you considered adding css sectors as an alternative to xpath? For many simple things a css selector is easer to write and more people already know them.
I believe it's possible to translate css selectors to xpath so it wouldn't need another selection engine.
htmlq does this by having various command line options like --attribute and --text.
https://github.com/mgdm/htmlq#examples
It'll be nice to try something new. swiss army style command line XML tools has been pretty neglected.
[1] https://github.com/kislyuk/yq
xq looks interesting for pure XML, will add it to my notes.
Regarding xmlstarlet, I don't use it much but here's some stuff from my notes:
XML filter on xpath: xmlstarlet sel -t -c configuration/appender logback.xml; echo
XML editing: xmlstarlet ed -d configuration/root -d configuration/logger -s configuration -t elem -n woof -v 'Woof!' logback.xml
The last one deletes elements given by xpath "configuration/root" and "configuration/logger", then adds a new element with the contents <woof>Woof!</woof> under the xpath "configuration".
It is also handy doing at some PoC work, and then one can feed it "-C" to output the backing xslt for use elsewhere
https://github.com/TomConlin/xpath2dot
xsltproc
You should try an XPath 3 based implementation. XQuery 3.1 has some nice features not found in XPath 2 implementations e.g. native JSON support, the arrow operator (piping), group by clause in FWLORs. XSLT 3.0 has some nice new stuff too, but I use that less
Thankfully XQilla has its own json support which was all I needed so far
JC – JSONifies the output of many CLI tools
https://news.ycombinator.com/item?id=33448204 2022-11-03
Which extracts XML and HTML as JSON.
Also cool: https://github.com/dbohdan/structured-text-tools/ "A list of command line tools for manipulating structured text data"
— The tool parses XML by default, which is much stricter than HTML. Hence the error you are seeing (in HTML5, one can omit various close tags and the HTML is still 100% valid — that's what you are encountering here)
HTH
I wonder if this space of tools is a bit like static site generators: it's almost as easy to write your own as it is to learn somebody else's.