Ask HN: Are Lucene/Solr/ES Still Used for Search?
I casually visit jobs/freelancing sites once in a while. I don't see as much demand for Lucene/Solr/ES skills for website / text / document search or other kinds of information retrieval, as I used to about 4-5 years ago.
ES seems to be the most popular but only in its ELK avatar for devops dashboards.
What technologies are you people using for text or document or website search nowadays?
222 comments
[ 4.4 ms ] story [ 235 ms ] threadAlso solr, to although a lot less.
This expertise is very on demand.
My company personally migrated from ElasticSearch to https://vespa.ai/ and could not be happier. Faster and way easier to maintain a cluster. The "Application Packages" feature present in Vespa opened many opportunities to improve our product( Curiously we use Lucene inside our custom application for a "Search map" functionality, for something like that https://www.lexisnexis.com/en-us/products/lexis-advance/sear... ) . I highly recommend it!
For me Vespa is a absolute game-change, in features and as someone said here, ES looks like it intentionally complicated to maintain. With nodes randomly getting unhealthy. Vespa is like Redis to me. I completely forgot about maintaining it and works great.
It makes a world of difference in our product, and I take every opportunity to evangelize it.
This may be of interest to you: https://sematext.com/opensee/report/project/trend?q=ElasticS...
Would you happen to know how Vespa compares to ES in terms of memory or CPU footprint? Have you done apples to apples comparison by any chance?
It's also readily apparent that it's an ancient system that's grown out of an in-house project, and that its design has accrued a lot of oddities over the years from lack of careful, co-ordinated design. It includes a bunch of esoteric features (like the "predicate" function) that have obviously grown out of Yahoo's own internal architecture.
One curiosity is Vespa's approach to schema and configuration changes. To make any kind of change, or indeed set up an index, you have to create that "application package" containing your schema and configuration in the form of files, and then use separate REST APIs to "upload", "prepare" and "activate" it. There's a CLI tool to help perform those steps, at least.
It's nice that they're more consistent and rigid about schema and config evolution than Elasticsearch. But it's not exactly operator-friendly, at least not for first-time users with no pre-existing operations based around Vespa.
The package design also makes it more cumbersome to perform programmatic updates for a schema. I once worked on a SaaS project where we indexed data in Elasticsearch — arbitrary documents where we didn't know the schema ahead of time, because we just accepted any JSON document posted by the client. With ES, we could just use its dynamic mapping support, which automatically creates field definitions when new fields arrive (using regex-based templates). Do you know how long a package update takes in Vespa, to add, say, a single field?
The Vespa documentation is also pretty terrible, in my opinion. They explain a lot of things, but it's confusingly written, uses a lot of homegrown terminology, and neglects to collect all the reference documentation in one place. For example, you can't find an overview of the entire API — fragments of it are just scattered across a dozen or so unrelated pages.
Lastly, Vespa is Java. One of the biggest challenges maintaining Elasticsearch is controlling its resource consumption. You have to give it a lot of RAM, and it's never clear how much it needs and what configuration settings and usage patterns affect its memory use. Tuning it is something of a dark art. I don't know exactly how Vespa is implemented (is it all pure Java?), but I'm worried that, being a JVM app, it has the same shortcomings.
In my own attempt to compare the two, I found the memory consumption of Vespa was easier to predict and understand (there are formulas for it in the documentation).
If you have dynamic fields like in your SaaS example I recommend using a single map field rather than let data not under your control drive changes to the set of fields.
> Do you know how long a package update takes in Vespa, to add, say, a single field?
A few seconds. However, rather than having operators do any of this manually, set up an automatic process which deploys on each change made to the repo (i.e do CD).
> all the reference documentation in one place
https://docs.vespa.ai/documentation/api.html
Edit: Documentation says: "Accessing attributes in maps and arrays of struct in ranking is not possible". So maps aren't really usable.
Regarding how long it takes to update a field, the application I described would have to do this programmatically. It would have to keep track of all known fields in some kind of registry, and then if a new unknown field came in, it would have to perform an "application package" deploy just for that field, using the REST API. (Unless there's a less cumbersome way to do it?)
Reference docs: That's nice, but that's just a bunch of links. Good reference documentation has tables of contents. Bonus points for runnable examples in multiple languages. For an example of good reference API documentation, look at Stripe's [1].
[1] https://stripe.com/docs/api
An application's search module could flatten the location key (e.g. "location_city", "location_state") for simple attributes, but the same is not possible for the array, since there can be arbitrary array elements. And you can't split it to an array of strings:
...because queries like "firstName contains 'Bill' and lastName contains 'Clinton'" could match different records ("Bill Bryson" and "George Clinton"). Never mind deeply nested arrays of objects containing arrays containing objects containing arrays.This seems unnecessarily restrictive. A search engine should be able to index the data you already have, not force the application to contort its data to whatever shape the engine requires.
Is there no way around this?
I guess you don't see much demand because for a lot of use cases the basic setups are good enough.
I ended up trying to reinvent Solr for the client, realizing after about two days of trying to reinvent stemming and indexing, that this was stupid to do on someone else's time, and called the client to tell them that I'm moving to Solr, and I got the project done before-schedule as a result.
====
I think for 99% of usecases (involving search), Lucene/Solr/ES is perfectly fine. However, I do absolutely hate that some companies have decided to make it their primary database.
EDIT: I just want to make it clear, I think it's totally valid to try and reinvent Solr for fun, or if that's something you're paid specifically to do; nothing is perfect, and I am actually a big fan of the "if it works, break it and make it better!" mentality.
TLDR: Lucene < 7.5 won't merge segments larger than 5GB (default) unless they accumulate 50% deletions.
Delivering a conference talk [1] later this year about it.
[0]: https://www.eivindarvesen.com/blog/2018/09/16/elasticsearch-...
[1]: https://2019.javazone.no/program/3f7cd8a7-a9ea-4874-a7dd-531...
I don't think the GDPR regulatory agencies are operating at a technical level that they would make an argument that it was not a good enough deletion.
Finally I have to ask this part: assuming ES is not your primary database, how does this get around the GDPR issues? If someone wants their data erased you are supposed to erase it from wherever you store data, I suppose this means ES when it indexes a primary store and finds it has deleted data actually deletes it but if it is told to delete something it keeps it around?
If it is easy to delete specific users from the primary database, the deleted users will naturally disappear during the next ES reindex.
Edit: The old index is deleted at the file system level.
If the reindexing occurs daily or weekly, perhaps this will satisfy GDPR.
There are other good reason to not use ES as the primary data store. First, it isn’t entirely reliable. It’s good and I’ve never seen a corruption, but ES and Lucene’s history isn’t as a reliable database. Second, if you want to change how you index, it is a bit easier to do if the source data is outside of ES.
How is GDPR compliance of having data in Elastic Search influenced by it being primary vs. secondary storage?
Basically: you reindex ES periodically, so when a user is deleted from the primary, it will disappear from ES upon the next reindex. The old index is deleted at the file system level.
at what point would you be able to 'delete' data without being in violation of GDPR?
Though the EU has said it will consider intention etc. there's really no way of knowing for certain until when and if it's settled in a court case.
Fix it 'til it breaks is what I always say :D
Needless to say my 3D printer had a lot of down-time haha.
I inherited a legacy Mongo solution, and all the data is duplicated and indexed in ES, so I've always wondered why we're using both. Mongo has none of the SQL capabilities that would make my life easier, and the types of queries allowed by Mongo could be done with ES.
What are the negatives of ES alone?
The v7 upgrade to a new cluster protocol (zen2) has improved things but overall the system has a long history of losing or destroying data. It's better to have a primary OLTP system that's ACID and reliable while using ES as the secondary search source. You can also remove the _source field if you just need matches without the original content.
It's common to see pattern used with a relational database since, as you can see, Mongo doesn't buy you much else as another document-store.
Thing is, there's heavy demand for something more performant than Elasticsearch, so eventually the market will provide.
Meanwhile, Redis Enterprise is trying to grab some 'share with RediSearch, which has some severe caveats IMO that make it not a great fit for most.
https://github.com/tantivy-search/tantivy
That said - it's effectively Lucene rewritten in rust, so the main win is some performance gains. Lucene has spent a ton of time getting the details right, and it's unlikely we'll see an order of magnitude of innovation in that particular space. At the higher level for querying / query understanding it feels like there's still more technological room to grow vs the lower level details.
It is not exactly a port but yeah. tantivy is strongly inspired from Lucene.
> Lucene has spent a ton of time getting the details right, and it's unlikely we'll see an order of magnitude of innovation in that particular space.
Have you checked out the perf gain in Lucene 8.0 ? Block-WAND proved you wrong.
Tantivy is a cool project, but I have to say the part I love most about it is your blog posts on it. They're a great introduction for people who are unfamiliar with the underlying tech of search engines.
Thanks a lot! I am not a native speaker, and I often feel very bad at conveying engineering concepts. The positive feedback is actually very helpful :)
Disclaimer: I’m a previous employee but have no economic interests in this as it’s not a publicly traded company.
[0]: https://lucene.apache.org/solr/
Vespa [1] would like to have a word with you.
[1]https://vespa.ai
ES is not supposed to be your source of truth.
https://discuss.elastic.co/t/elasticsearch-as-a-primary-data...
Any changes to the schema, such as switching fields between indexed/not indexed/stored/not stored, requires reimporting the data to populate those fields, data which you're not likely to have if it was your primary store.
1. https://www.elastic.co/guide/en/elasticsearch/reference/curr...
2. https://www.elastic.co/guide/en/elasticsearch/reference/curr...
I think that we definitely have the best integrated and most full featured solution for Sitecore customers.
It's not only about querying but also having a UI framework, built-in customizable indexing, analytics tracking and access to machine learning available in one package.
Source: Am product manager for Coveo for Sitecore.
I work at a state university (with the associated purchasing ... issues :)) and we had some staffing issues, which they were very accommodating of from initial quote to subscription.
If we end up running into issues as/if we expand our usage (we're essentially only using Solr for the mandatory bits) I'll keep Coveo in mind. :)
Pros:
- Managed search engine
- Great API / Developer experience
Cons:
- Cloud only makes it hard for local development
- Expensive (I guess it depends on the usage)
Would love to hear your feedback :)
First there's the Docker + Kubernetes architecture that ES lends itself to really well. Then (depending on your use-case) there are concerns like hot/warm architecture, node types, ETL/indexing processes. ES recently moved over to openJDK, so there's a couple intricacies there (i.e. JVM heap size)
Then, there's document/query structure. In no particular order:
- Do you have any parent/child relationships?
- Do you have stop-word lists developed?
- Can search templates help your queries?
- How will you interface with ES? It has REST APIs, but it's recommended to not expose ES directly to your applications.
- Some advanced querying possibilities like customizing tokenizers, normalizers, and a bit of internationalization.
- Oh, we haven't even discussed security yet.
- Also, ES isn't meant to be a primary data storage. This is more so a "cache", but not quite like Redis. So, you'll need a DB elsewhere most of the time.
All of this changes depending on if you're using it for SIEM, e-commerce, AI/ML, etc. Also, Elastic now provides their own SIEM solution, a pre-built search solution (AppSearch + Search UI), built-in security features. Check out the new ES 7.2 update; it's kinda nuts.
My current employers uses ES - we're on 6.8, planning to move to 7 in a few months. Judging by the other replies here I'd say we have a reasonably large cluster (150+ i3.2xlarge instances, billions of documents), so tuning the cluster is very relevant to us. Could you expand on how things have changed with the move to OpenJDK?
I've seen some claims online that, contrary to what Elastic recommends in their docs, a few machines with huge heaps (100+ gb) is the way to go, rather than many machines with 20gb heaps.
Usually the recommendation is less than 32GB - this link has some more discussion about it: From https://discuss.elastic.co/t/es-lucene-32gb-heap-myth-or-fac...
It seems whether it's better or worse depends on your data set . But I would love to see tests of different kinds of workloads with large or smaller heaps.
Also, you have plan ahead and over-allocate or deal with fixed indexes/datasets. You also have to religiously monitor the garbage collection and deduce what's going on with search & indexing performance. When the situation changes you need to scale your cluster and re-index everything, which is not a trivial thing at most companies. I've seen bad situations at companies where it takes days to re-index a cluster and they're dead in the water until it's done.
And that's just the operations side. You have to make sure your data is flat (because nesting creates subindexes for Lucene and kills your search performance), that you only define in your index template the fields that you want to be searchable (and binary blob the rest), etc.
http://highscalability.com/blog/2013/1/28/duckduckgo-archite...
For commodity search workloads (general retrieval/faceting) Elastic does a fine job. It scales well and there is good documentation and support.
Lucene is the core engine behind both of these solutions.
For fun, lets look at the large Enterprise acquisitions over the years:
* Verity - bought by Autonomy
* Fast - bought by Microsoft (Also known as the Enron of Norway...)
* Autonomy - bought by HP (Look at the backstory on this deal!)
* Endeca - bought by Oracle
* Vivisimo - bought by Oracle
* Google - GSA (now Google Cloud Search, hosted solution)
Next, follow the path of online acquisitions:
* IndexTank - bought by LinkedIn
* Swiftype - bought by Elastic
There's a number of interesting independent players still. Coveo plays in the Enterprise space, but it's a hard market. Algolia is doing great in the commodity online search space and seems to be growing well.
This is an area I think is open to more competition. Especially with AI/ML technologies available around Document Understanding - the Enterprise market is open for a good on-prem upstart to really take off.
Ping me offline if you have additional questions - spent almost 20 years in the space and ran a search company of my own.
That one was painful to live through, we got forced to migrate to Windows and everything went sideways. That was almost 10 years ago with quite a big cluster (tens of nodes).
Interesting, could you elaborate on why SOLR is dominant in that space over say Elasticsearch?
Also, relevancy in retail is often influenced by other factors that cannot easily be implemented in Elastic. TFIDF/BM25 search is available in both platforms, but you may also weigh in other factors such as relationships with the vendor, stock on hand, or other ML techniques that are more easily implemented in SOLR.
These days, I would say there's very little that either product does that the other product can't do though obviously there are lots of strengths and weaknesses on both sides.
Thankfully we are moving to Azure Cloud Search services, which just work, over the next year.
Good bye manure pile, hello compost.
Source: I have the capital V from the building in my house.
(I see now atambo has already noted this typo)
While I have not used Solr recently, it has evolved along with ES as a solid product with a solid community. Nothing against it; it's a solid choice and there are probably people offering to host that as well. Either way you are using Lucene. Using Lucene directly makes no sense unless you know what you are doing and have a need to do that. If you have to ask that means it's not for you.
If search is important to your use case and things like relevance, precision, and recall have real impact on your business, you should get some specialists involved and not reinvent the wheel or make all the rookie mistakes. Somebody like me basically ;-). However, that can be expensive and if search is not that critical, just sign up for one of the search as a plug in solution type products out there and don't bother running a lot of infrastructure. E.g. Elastic offers a thing called App Search and it probably covers most of the simple needs and is stupidly easy to get started with. There are several competing products probably that I can't vouch for.
You can always upgrade to something proper later. Things like mongo and postgres also have some limited capabilities here and you can get away with doing some simplistic stuff with sql even. However, there's a point where you hit a brick wall and there's something you need that is simply hard/impossible using that or where you end up reinventing a lot of stuff that things like Elasticsearch probably do better.
I think everyone doing Elasticsearch well has to bring it in-house eventually. AWS's hosted solution is poor, Logz.io and ElasticCloud are expensive.
There's a 7-figure/yr Elastic Cloud customer I work with who is so tired of Elastic just randomly killing their clusters out of nowhere and having to spend basically triple to deal with it that they're bringing it all in house.
Once you grow larger than that though, the hosted service prices get astronomical compared to standing up a cluster, assuming you have someone who can admin it.
My point is that a ton of companies (more for cultural reasons than business requirements IME) are so freaked out by any downtime in any way that they’re going to pay for an engineer to just maintain these things themselves.
If you're just spinning up something's defaults and throwing data in it, that bill is going to due eventually and it's probably going to be ugly.
What has been poor about it in your experience?
The backup issue has been resolved I believe but the others...
On AWS I’d suggest elastic on ECS and you can use the leftover compute on the cluster to run other applications effectively for free.
If a configuration change (changing # of nodes, instance type, etc) goes wrong, your cluster indefinitely gets stuck in Processing due to a race condition. The only way to get unstuck is to file a ticket. The company I'm at doesn't pay for AWS support, so at one point we ended up completely tearing down our cluster and rebuilding a new one (via Terraform) after getting tired of waiting for the reps. (They advised us to cut off log flow to let the system get out of processing, which we did, but it didn't work because once it gets stuck in processing like that it's just completely stuck).
It's difficult to troubleshoot issues - you can get some logs via Cloudwatch, but they're hard to search through and I'm not entirely positive everything shows up there.
Amazon is always several releases behind Elasticsearch versions.
--
Elastic.co's offering looks much better, just by reading their excellent comparison article: https://www.elastic.co/blog/hosted-elasticsearch-services-ro...
(We haven't used Elastic.co but what they say makes sense and I imagine their service is much better)
--
Once you hit a big enough scale - for us, we're pushing about 2TB a day of logs (that's before accounting for replication of course), it doesn't make sense to stay on Amazon's hosted service.
I'm in the process of advocating for in-housing our Elasticsearch setup and just building on top of ec2. Elasticsearh seems like the perfect candidate for Kubernetes since rebalancing is automatic and the affinity rules are simple (every Elasticsearch instance needs its own node). Cluster autoscaling (i.e. node-level, not _horizontal_) just makes too much sense.
Unfortunately I haven't gotten the go-ahead to take it inhouse, but I've been gunning for the project for some time now, so I'm hopeful I'll get the opportunity.
--
BTW, totally unrelated but for anyone managing Elasticsearch, make sure you have your shard count tuned properly. When I came to this company, they had their data way oversharded with primary shards varying between hundreds of kb to a few gb; i.e. orders of magnitude difference. Switching to ~50 GB shards (done via simplifying the way we were indexing) massively improved performance.
Also i3 instances > anything with EBS.
[/ramble]
ES is indeed super solid if you know what you are doing. However, most users getting started with this are probably going to find out a few things the hard way though; which is why I recommend hosted solutions as it removes quite a bit of non trivial devops from the equation.
That is because Elastic Cloud is not a fully managed Elasticsearch. People often don't get that with Elastic Cloud you are still responsible for your ES cluster. That's one of the differentiators that e.g. Sematext has (disclaimer: founder).
> AWS's hosted solution is poor, Logz.io and ElasticCloud are expensive.
Right. Have a look at https://sematext.com/logsene pricing. People say Sematext compares favorably to Logz.io and Elastic Cloud.
The per gb cost on the other ones start at 1.20 $/gb and goes till 2$/gb. While almost all hosted ELK solutions start off at 3$/gb.
Im asking because i would very much like to adopt an ELK based hosted solution..but I'm not able to justify paying double. Is it that running+resource costs for ELK are so high that the extra charge needs to happen ?
You do not need an agent. You can ship directly from syslog ( https://docs.datadoghq.com/integrations/rsyslog/?tab=datadog...)
I've never had clusters being killed randomly but I did have a few self inflicted issues with cluster instability due to flooding the cluster with too much data and not having suitable data retention policies in place. With the recently added index life cycle management (an x-pack feature), this is easier to manage these days.
If you are spending seven figures a year on elastic search, you clearly are not a beginning user and there's some cost savings that you might be able to realize by taking ownership of the problem of hosting it somewhere cheaper. For that, their recently open sourced kubernetes helm charts are worth a look. Those scripts take care of a lot of things and get you a self hosted version of Elastic Cloud.
Amazon hosted clusters are indeed a bit bare-bones (as is their support for these clusters) and I would also not recommend them; you get more value for money by using Elastic Cloud.
For small projects (for some 1000s of documents), I'd probably go with Postgresql FTS if possible. Sphinx/Solr for anything with indices smaller than a couple 100GBs After that, ES seems reasonable & worth the overhead
EDIT: my biggest issue with ES is that it seems to be specifically engineered to sell you support. So get a managed version if you can.
why do you think you can't throw 1TB of data on postgresql?
The problem with PG FTS is that it doesn't have advanced search functionalities (fuzzy matching, faceting, term distance, highlighting results) and it lacks the modern relevance scoring systems so that'll be the limiting factor instead of speed.
Would love to hear your feedback.
One area where this might be less true is that the full-text search in Postgres & MySQL have matured to the point where some basic applications might reasonably decide that it's not worth using a separate service.