HN Explorer

Ask HN: What tool do you use to sync MongoDB with Elasticsearch?

1 points by anikdas ↗ HN
Currently I am performing full text search on MongoDB but I would love to use Elasticsearch for the same. My primary data source is MongoDB. Right now I am not able find a suitable/reliable sync tool for MongoDB->Elasticsearch.

There used to be a tool called elasticsearch-river-mongodb [1] which is deprecated now.

Also, there is no official logstash input plugin for MongoDB

[1] https://github.com/richardwilly98/elasticsearch-river-mongodb

What tool do you use to reliably sync MongoDB collections with Elasticsearch?

3 comments

[ 3.3 ms ] story [ 21.7 ms ] thread
FBISurveillance
Take a look at Kafka. When your backend pushes data to MongoDB, have it produce a Kafka message that contains serialized Elasticsearch JSON documents and then use either a bespoke consumer, or kafka-connect-elasticsearch, or logstash-input-kafka to consume those documents and bulk-insert them into Elasticsearch.

Bonus points: it will let you have a standby Elasticsearch cluster in another region you could failover to when needed, perform Elasticsearch cluster upgrades/maintenance without losing data, and more.

Depending on your use case this could be an overkill so YMMV.

anikdas
I could definitely do that. But my original intention was to use a standalone application which can get the job done. For example when the data is updated/deleted instead of insertion I need to write my own logic.

Any way great suggestion. Will definitely keep in consideration.