I read all releases log since 2.3 and its always been enlightening, and I start using those features. Elasticsearch is my favorite tool and it has been very useful over the years, just wanted to share my thanks. I just hope they can survive in the cloud defying broken AWS Elasticsearch abomination.
Heh, I'm pretty sure our attempt to use AWS-managed ES went badly, and we quickly gave up on it and stuck with self-hosted. And this is at a company that's pretty keen to use AWS-managed services when we can.
Make darn well sure you aren't using any unsupported operations.
In their favor, if I were to revisit them I could use it now; however in the past they didn't support templates and a few other things which turned out to be major blockers.
If you want to get it set up and running most of the way without tuning a bunch of stuff it’s pretty good.
It also has significant cost overhead compared to self-managed and limited instance types.
My team now is working on migrating a 2.3 cluster from AWS ES to a 7.8 self-managed cluster (probably 7.9 now I guess). We do lots of indexing throughout the day and the i3 instances available are pretty weak, just switching to r5d/r6gd instances greatly reduced our indexing and search latency
We moved in the early days. Today, we are fairly advanced with regards to our understanding of the internals of Elasticsearch - data nodes, master nodes, indexing, shards, replication strategy and query performance. In hindsight, it would be fair to say that we have been extremely happy with our choice given our limited understanding of running a production scale Elasticsearch cluster 6 months back. With Amazon Elasticsearch we were able to start small, move fast, learn on the go, and fine-tune our cluster for production workloads without having to understand every single aspect of managing and scaling a production scale Elasticsearch cluster.
We've been looking at doing managed because we use ES for some projects. But at they're always running on the same server as the application. And we have nobody with the knowledge or ambition to maintain it.
But reading these comments I get wary of doing it that way.
I’m sure it depends what you’re doing - our cluster is medium-largeish, I’d guess (wish there was more info on the clusters that are out there!). Several hundred big EC2 instances, all NVMe, billions of documents, indexes in the terabytes, high update rate in addition to inserts. It requires fairly active management and optimization. We might just be outside of the sweet spot for the AWS offering.
> Several hundred big EC2 instances, all NVMe, billions of documents, indexes in the terabytes, high update rate in addition to inserts. It requires fairly active management and optimization.
Got spoiled by Splunk, I still cannot recreate the same experience in Kibana. Elastic seem to be doing well for structured data, but when your data contains a lot of free text (such as log lines), it still lacks the rich query language Splunk has.
There's so much marketingspeak in that piece, it's nauseating. I use and maintain an ELK stack and I'm not even entirely sure what most of that nonsense means.
Many of the X-Pack features are actually available in their "Basic" license and even public source: https://www.elastic.co/what-is/open-x-pack but just not under an actual OSS license.
e.g. "Data Streams" is covered under the Basic license so you can use it for free (I think? As far as I can tell at least). You can find the feature on their subscriptions page: https://www.elastic.co/subscriptions
I find the whole situation with X-Pack, Subscriptions and OSS vs Basic super confusing and the subscription page seems very bad at explaining it all.
Also agreed. To whatever extent it's helpful, this is my take on it:
Elasticsearch:
* Data streams -- an abstraction that eases management of a set of indices containing time series data.
* Wildcard data type -- makes searching for partial strings much more efficient
* EQL -- a new query language that's popular in the SIEM (security info and event mgmt) world
* Miscellaneous new aggregations
* Tableau connector for Elasticsearch
Other parts of the stack:
* free version of Workplace Search (index all your stuff from multiple sources such as Gmail, Google Drive, MS Sharepoint, Salesforce, etc.)
* free version of endpoint security from Elastic's acquisition of Endgame
* Ingest Manager/Elastic Agent -- centrally manage and configure Beats-based endpoint agents that push metrics, monitoring, etc. to Elasticsearch
* new rendering engine in Kibana
I just read through it. It's basically a summary of quite a lot of change happening to quite a lot of products that they ship as Elastic. Each of those products has their own set of release notes; which you probably should be reading if you use those products.
But the executive summary for an ELK user like you:
- Kibana just got faster (finally). I know some people in that team and they completed some internal javascript refactoring that gets rid of the annoyingly long page loads. So, this is a relatively big improvement for a 7.9.
- Elastic Agent is a new thing that can get a lot of data
into dashboards quickly. That might be relevant for you if you have a (no doubt) lovingly handcrafted logstash ingestion pipeline and matching dashboards. Also, setting up half a dozen elastic different beat agents to achieve the same was definitely a bit of a devops burden. Sounds like something I might give a try next time I set up new infrastructure.
- You can now self host Elastic Cloud in your own AWS infrastructure. This may be relevant for you if you were self hosting some home grown setup on premise (or in aws) or are using the fully managed cloud solution. Kind of a big deal for more advanced setups as a lot of companies get themselves into trouble with getting their homegrown devops for this right.
- A lot of updates to Elastic products neither you nor me use. The workplace search thing sounds interesting though. Wasn't really on my horizon so far.
I read it as: we are working hard, so that the Kibana JS bundle is over 100MB before compression.
I have 6.x and 7.x installed side by side and loading "discover" tab with clear cache takes 3.5s on old one and 14s on new one.
24 comments
[ 1.5 ms ] story [ 65.6 ms ] threadHeh, I'm pretty sure our attempt to use AWS-managed ES went badly, and we quickly gave up on it and stuck with self-hosted. And this is at a company that's pretty keen to use AWS-managed services when we can.
Make darn well sure you aren't using any unsupported operations.
In their favor, if I were to revisit them I could use it now; however in the past they didn't support templates and a few other things which turned out to be major blockers.
If you want to get it set up and running most of the way without tuning a bunch of stuff it’s pretty good.
It also has significant cost overhead compared to self-managed and limited instance types.
My team now is working on migrating a 2.3 cluster from AWS ES to a 7.8 self-managed cluster (probably 7.9 now I guess). We do lots of indexing throughout the day and the i3 instances available are pretty weak, just switching to r5d/r6gd instances greatly reduced our indexing and search latency
https://spun.io/2019/10/10/aws-elasticsearch-a-fundamentally...
https://code972.com/blog/2017/12/why-you-shouldnt-use-aws-el...
More details here - https://www.akshaysurve.com/2018/01/what-we-like-about-amazo...
But reading these comments I get wary of doing it that way.
AWS Elasticsearch may not be for power users but they did announce Petabyte and Ultrawarm for über scale clusters in run up to last year's re:Invent: https://docs.aws.amazon.com/elasticsearch-service/latest/dev... and https://aws.amazon.com/blogs/database/run-a-petabyte-scale-c...
"Data Streams sounds cool! It could help alleviate some of the management overhead that I was accomplishing with ingest pipelines."
clicks into the docs
`X-Pack only`
I get that it's complementary to their ILM functionality, which, too, is an X-Pack feature, but still, clarity would be helpful.
e.g. "Data Streams" is covered under the Basic license so you can use it for free (I think? As far as I can tell at least). You can find the feature on their subscriptions page: https://www.elastic.co/subscriptions
I find the whole situation with X-Pack, Subscriptions and OSS vs Basic super confusing and the subscription page seems very bad at explaining it all.
But the executive summary for an ELK user like you:
- Kibana just got faster (finally). I know some people in that team and they completed some internal javascript refactoring that gets rid of the annoyingly long page loads. So, this is a relatively big improvement for a 7.9.
- Elastic Agent is a new thing that can get a lot of data into dashboards quickly. That might be relevant for you if you have a (no doubt) lovingly handcrafted logstash ingestion pipeline and matching dashboards. Also, setting up half a dozen elastic different beat agents to achieve the same was definitely a bit of a devops burden. Sounds like something I might give a try next time I set up new infrastructure.
- You can now self host Elastic Cloud in your own AWS infrastructure. This may be relevant for you if you were self hosting some home grown setup on premise (or in aws) or are using the fully managed cloud solution. Kind of a big deal for more advanced setups as a lot of companies get themselves into trouble with getting their homegrown devops for this right.
- A lot of updates to Elastic products neither you nor me use. The workplace search thing sounds interesting though. Wasn't really on my horizon so far.
I feel attacked, somehow. I have made so many logstash pipelines. Some of them I love. Others I don't.