This is a good question! The answer is, unfortunately, uncomputable – or so we think. But if you manage to find a more satisfying answer, you'll have solved one of the more beguiling problems of the last century, which is the basis for most of modern Information Theory. You can learn more about it here: https://homepages.cwi.nl/~paulv/kolmogorov.html
These methods could not beat the 480 token result because of the small dataset. But for a sufficiently large corpus with graph reordering and columnar parquet compression, they could be quite competitive.
The parquet files are self explanatory. For knowledge.lbdb.zst, first decompress with zstd and then run a cypher such as:
lbug> MATCH (a:E)-[t:TRIPLE]->(b:E) RETURN a.name AS subj, t.predicate AS pred, b.name AS obj, t.props AS props, t.scope AS scope;
5 comments
[ 3.9 ms ] story [ 53.2 ms ] threadThese methods could not beat the 480 token result because of the small dataset. But for a sufficiently large corpus with graph reordering and columnar parquet compression, they could be quite competitive.
The parquet files are self explanatory. For knowledge.lbdb.zst, first decompress with zstd and then run a cypher such as: