What does the author mean by while indexing? You have to send a commit message to make the newly indexed doc searchable. Its silly to do this after every document. I'm assuming that is what the author was doing to get search poor search results. You could also just use Lucene's NRT abilities in Solr w/ a simple patch. I believe it is going to be working out of the box in the next version of solr. I wrote one myself to deal with real time search. If we want a fair comparison, we should be comparing Solr using NRT and ElasticSearch.
NRT is just part of the story, since in order to provide a proper (near) real time features in a distributed architecture you need to build it around it. For example, not doing pull replication, but push replication. The whole distributed nature of elasticsearch compared to Solr was not discussed in this blog post, but its a big differentiator.
It is not silly if you have asynchronous indexing because your content is generated by your users, one document at a time. If you want real-time indexing, that's the way to do it.
This is the default behavior for IndexTank. When an indexing call returns, whatever you indexed WILL show up in search results. We don't give you the option to commit or not commit, we simply designed IndexTank to be fast enough when indexing in real time. I'd like to see us included in that comparison as well.
IndexTank :) Meaning, we wrote a search engine from the ground up. We probably should write a blog post to explain why. We did use a couple of things from Lucene:
- The tokenizers, to be compatible with the query syntax
- The index format for persistence on disk. It's a great format, no point in reinventing that. Analogous to how Open Office can save in .doc format.
Its not the only way to do it. Twitter doesn't do it that way, neither does LinkedIn. They have their own Search Engines as well build around Lucene. I haven't looked at them in a while, but I don't think they were commiting every document. They also get their content from their users.
FYI, LinkedIn open-sourced the work they did to build real-time indexing/search on top of Lucene. It's called Zoie (http://javasoze.github.com/zoie/), I've played with it and it's nice from a performance point of view...
Lucene doesn't have to commit document updates to make them searchable. You just need to grab the IndexReader via the IndexWriter.getReader() method instead of opening an IndexReader on the index directory.
We use this for NRT indexing currently on our site and it works fine - there's a small (500 millisecond or less) delay in changes appearing but it's close enough for us.
We don't use Solr, though - it's a custom search engine based on Lucene.
Granted, percolation sounds pretty cool, but if the article is referring only to single-server/single-core Solr architectures, then that speed comparison is unfair.
The article doesn't mention the setup he used for the benchmark.
Is anyone using ES on production, how is the support for things like lat/long(geo) search. If you're on django then your options are limited if you want to use django-haystack. Solr is pretty much the only way to go. xapian geo support hasn't landed yet, and waiting for solr 4 to land.
I've been using JTeam SSP 1.0 plugin(http://blog.jteam.nl) for solr, with django-haystack(https://github.com/sidmitra/django-haystack) for the same and it's been really useful and responsive so far. I still want to give ES a try, but will have to look into support with haystack + for geo stuff.
ES doesn't have haystack support, and shoe-horning it in is pretty hard to do because ES semantics are quite different from the search engines Haystack supports.
I use ES via Python, and ended up writing my own backend for it that approximates the Django ORM.
Are you using it in production anywhere? The stack i mentioned above powers the location search for teaspiller.com right now, but i'm not that invested into haystack not to look at alternatives if it suits us well.
For everything else, I'm using Sphinx. I chose Spinx because at the time (starting 2 years ago), it seemed like the best choice for real time indexing, set up was easy, and it integrated well with MySQL.
But now the Lucrene backend DBs are in decent completion with it---Elastic Search and Solandra.
I'm a bit confused by the usage of "percolation". Generally speaking that refers to, in a tiered retrieval system, a document being initially indexed in a small temporary index then gradually "percolated" upwards to larger more permanent indices. Such systems allow updates without merging into the main index.
Lucene has supported near real-time search since 2.9 or so. By calling IndexWriter.getReader(), your index reader has access to both committed and non-committed changes.
In my experience, this near real-time search picks up changes within the last 500 milliseconds or so. I think it's basically a workaround for the latency related to disk I/O on commits.
What is Twitter doing to make that faster? I'd appreciate any links you could post which explain their changes.
18 comments
[ 429 ms ] story [ 2622 ms ] threadThis is the default behavior for IndexTank. When an indexing call returns, whatever you indexed WILL show up in search results. We don't give you the option to commit or not commit, we simply designed IndexTank to be fast enough when indexing in real time. I'd like to see us included in that comparison as well.
- The tokenizers, to be compatible with the query syntax - The index format for persistence on disk. It's a great format, no point in reinventing that. Analogous to how Open Office can save in .doc format.
We use this for NRT indexing currently on our site and it works fine - there's a small (500 millisecond or less) delay in changes appearing but it's close enough for us.
We don't use Solr, though - it's a custom search engine based on Lucene.
Is anyone using ES on production, how is the support for things like lat/long(geo) search. If you're on django then your options are limited if you want to use django-haystack. Solr is pretty much the only way to go. xapian geo support hasn't landed yet, and waiting for solr 4 to land.
I've been using JTeam SSP 1.0 plugin(http://blog.jteam.nl) for solr, with django-haystack(https://github.com/sidmitra/django-haystack) for the same and it's been really useful and responsive so far. I still want to give ES a try, but will have to look into support with haystack + for geo stuff.
I use ES via Python, and ended up writing my own backend for it that approximates the Django ORM.
For everything else, I'm using Sphinx. I chose Spinx because at the time (starting 2 years ago), it seemed like the best choice for real time indexing, set up was easy, and it integrated well with MySQL.
But now the Lucrene backend DBs are in decent completion with it---Elastic Search and Solandra.
See: http://nlp.stanford.edu/IR-book/html/htmledition/dynamic-ind...
In my experience, this near real-time search picks up changes within the last 500 milliseconds or so. I think it's basically a workaround for the latency related to disk I/O on commits.
What is Twitter doing to make that faster? I'd appreciate any links you could post which explain their changes.