Quickwit (https://quickwit.io/) | Paris, France | Onsite and remote (based in Europe) | Full-time The company is fully remote but we also have a small office in Paris. We prefer candidates based in Europe but can make…
We indexed Common Crawl only for the purpose of this demo so this is one-time thing, we won't deal with updates.
There are more recent versions of the dataset. We used the february/march snapshot from this year and the April snapshot just came out (https://commoncrawl.org/2021/04/april-2021-crawl-archive-now...).
We store the URI of each shard making up the index and, optionally, partition key and value(s). Along with a few flags, we also store the shard size, creation and last modification time. This additional metadata is not…
What we store on S3 is a regular tantivy index and another tiny data structure that we call "turbo index", which makes queries faster on object storages. For this demo, the tantivy indexes are fairly large and we issue…
Quickwit (https://quickwit.io/) | Paris, France | Onsite and remote (based in Europe) | Full-time The company is fully remote but we also have a small office in Paris. We prefer candidates based in Europe but can make…
We indexed Common Crawl only for the purpose of this demo so this is one-time thing, we won't deal with updates.
There are more recent versions of the dataset. We used the february/march snapshot from this year and the April snapshot just came out (https://commoncrawl.org/2021/04/april-2021-crawl-archive-now...).
We store the URI of each shard making up the index and, optionally, partition key and value(s). Along with a few flags, we also store the shard size, creation and last modification time. This additional metadata is not…
What we store on S3 is a regular tantivy index and another tiny data structure that we call "turbo index", which makes queries faster on object storages. For this demo, the tantivy indexes are fairly large and we issue…