I saw these guys at Velocity in NY this year. Pretty impressive product. I felt like the query language they built was easier to work with than setting up queries and filters in elasticsearch's api.
I'm working on something similar. So far I like Apache NiFi for ingestion and Apache Flink for processing. Storage choice(s) are plenty and IMHO determined by the use-case and available expertise.
Indeed storage choices are plenty. That said, if you want to be able to do the kind of optimizations that are described in the article, then the set of candidates gets a bit smaller.
Disclaimer: I’m the CTO at Jut and have been with the company since it’s inception.
Actually Jut didn't shut down last week. We figured out a much better way to package up our tech and ended up making some big changes to execute on this strategy. Unfortunately @PurpleQuark and a number of other really great people are no longer with us.
Apologies, that tweet is mine, and was posted shortly after the company all hands that day, and reflected my understanding at the time of the nature of the reorganization. Clearly, @mdemmer 's post is more accurate. Apologies for the confusion.
Let's get this out of the way - I love it when companies are open and transparent about their architecture. Sharing intimate details like this is fantastic.
Where I'm struggling is that there are a number of questionable choices here with little justification. For example, why a HTTP front-end? This is fine for webhooks but I'm not going to let my website's backend open an HTTP connection for every event I want to send out. The decision to store the data in Elasticsearch and Cassandra is equally dubious. In my experience Elasticsearch has been a maintenance nightmare and has not been a perform any and robust reporting solution at scale.
Do you support only transitive aggregation operations? If so why not push the entire aggregation to elasticsearch/cadsandra?
How do you plan to scale cpu wise? ES and streaming engines (dont know cassandra) are cpu hogs (compared to map reduce).
I heard at devopsdays Tel Aviv that bigpanda decided to provide different sla to paying and non paying customers to balance the costs.
15 comments
[ 5.1 ms ] story [ 154 ms ] threadReally interesting to hear about the innards.
Thanks for the post.
[1] https://www.crunchbase.com/organization/jut-inc
Actually Jut didn't shut down last week. We figured out a much better way to package up our tech and ended up making some big changes to execute on this strategy. Unfortunately @PurpleQuark and a number of other really great people are no longer with us.
Where I'm struggling is that there are a number of questionable choices here with little justification. For example, why a HTTP front-end? This is fine for webhooks but I'm not going to let my website's backend open an HTTP connection for every event I want to send out. The decision to store the data in Elasticsearch and Cassandra is equally dubious. In my experience Elasticsearch has been a maintenance nightmare and has not been a perform any and robust reporting solution at scale.
How do you plan to scale cpu wise? ES and streaming engines (dont know cassandra) are cpu hogs (compared to map reduce). I heard at devopsdays Tel Aviv that bigpanda decided to provide different sla to paying and non paying customers to balance the costs.
As far as the scaling issue goes, this was designed to run on premise rather than as a SaaS service (unlike bigpanda?).
Disclaimer on last paragraph: as per @demmer's comment below, Jut's plans have changed, so it may no longer be valid or relevant.