This is pretty cool actually. The one thing I always hated about wikipedias .zim files is that they're not really a standard I can use elsewhere. Although I'm having trouble understanding the sorting structure with just one json file I'm looking at. I realize it's sorting by the ID, but is that just simply based off of when the ID in wikipedia was created?
However, I think the json file is extraordinarily nice to have because it's all of wikipedia in just a simple text format. If we could get wikipedia as an entire package with all the html, css, javascript but capable of being run locally, I think that would be an amazing game changer using wikipedia offline, especially on a pi! While kiwix "works" in some ways, it also has some major issues that makes using it such a pain. I'd like for wikipedia to be just a completely source of data and information that I could search and pull up just even a basic html file and read that article from looking it up just in my own directory.
It's not a hugely useful bit of information, but I found out a while ago that most Kobo eReaders are able to handle ZIM files.
And some of them can have their "on-board" storage increased, which is actually a normally non-user-accessible SD card. I wouldn't do it on a water-resistant model to avoid breaking the seal.
yeah, AFAIK the MediaWiki people spent a lot of time to write a new parser for their visual editor that creates an AST from the syntax. That should be a much better way to create whatever format you want.
OTOH this is just hearsay and I never really looked into it. So maybe this was the last ditch and OP just did not mention the failed attempts before?
tldr; There are 3 parsers for wikitext,
1. the "old" wikitext parser written in php
2. a newer effort written in js and named parsoid
3. parsoid.php which is the one currently in use on (most?) WikiMedia sites.
Disclosure: I work for the Wikimedia foundation but not directly on the parsing code.
Anything I post on hacker news does not represent the views of my employer.
Maybe one could create a full text search for Wikipedia out of this data. This would also allow us to easily gather statistical information from Wikipedia, e.g. how often does some word occur in Wikipedia. It would be great to have a Wikipedia to play and query around with.
10 comments
[ 2.6 ms ] story [ 31.3 ms ] threadHowever, I think the json file is extraordinarily nice to have because it's all of wikipedia in just a simple text format. If we could get wikipedia as an entire package with all the html, css, javascript but capable of being run locally, I think that would be an amazing game changer using wikipedia offline, especially on a pi! While kiwix "works" in some ways, it also has some major issues that makes using it such a pain. I'd like for wikipedia to be just a completely source of data and information that I could search and pull up just even a basic html file and read that article from looking it up just in my own directory.
Overall though this is a super cool project!
And some of them can have their "on-board" storage increased, which is actually a normally non-user-accessible SD card. I wouldn't do it on a water-resistant model to avoid breaking the seal.
OTOH this is just hearsay and I never really looked into it. So maybe this was the last ditch and OP just did not mention the failed attempts before?
All the important details are on the wiki page: https://www.mediawiki.org/wiki/Parsoid
Disclosure: I work for the Wikimedia foundation but not directly on the parsing code. Anything I post on hacker news does not represent the views of my employer.