Elasticsearch anti-patterns and bad practices to be aware of

I've seen and done some things in my time. Some of those things were glorious. Some turned out to be horrible, horrible mistakes. In this video, I speak about some of those things that involved Elasticsearch.
Wonderful as it is, Elasticsearch won't solve all your problems, and may even cause you some sleepless nights if used inappropriately. You might not even need it. I'd like to help you avoid the mistakes I've made and seen while building and managing clusters. It feels good to get this off my chest.

[@samb2543]

Speaking as somebody who is setting up their first proper Elasticsearch cluster at the moment, this stuff is gold. So often you don't hear about the practical realities of new tech, people only mention the shiny bits, and it's great to go into this project with my eyes open.

[@GeorgeBridgemanData]

Thanks so much! I hope the setup goes well. Please let me know if there's anything you found difficult and would like explained; I'm always looking to hear people's experience.

[@samb2543]

@@GeorgeBridgemanData Thank you!
The thing that puzzles me most at the moment are users, roles, privileges and api keys. As I build out and plug in more things (3rd party monitoring, logstash, metricbeat, elastalert etc) I try to provision users scoped to only the features and data they need, but it's a slow process trying to figure out only the minimal permissions.
I've not really found a good explanation of the ES security model yet, if you fancied doing a vid on that sort of topic I'm sure I and others would find it very useful!

[@kickinit333]

Great tips. I ran a in-house ELK cluster for 6 years and you're absolutely right. The skill needed to setup, maintain, optimize and grow an ELK cluster in many ways does not mirror RDB at all.
Prepare for some late nights unclogging bottlenecks, upgrading and optimizing (and many times rewriting) ingest pipelines. Not to mention the network costs retrieving data from cloud providers and dealing with out-of-date plugins. Lot's to think about. Cheers.

[@GeorgeBridgemanData]

I'm really pleased it resonated with you! Elasticsearch is certainly a different beast than how it's considered.
What you mentioned about cost is a really interesting point. There are a lot of places where (financial) cost scope can creep up. Network cost is one, storage cost for lots of replica shards in a read-heavy cluster is another, VM cost per node... Bills can get to the point where people don't want to add nodes because it's so expensive, even at the cost of cluster performance.

[@jamesa8165]

Certainly can attest to point #1 - We went into our project (hundreds of millions of records, for full document text search) thinking we would have completely flattened data.
Later on in the project we saw a serious need for some form of hierarchy/relationship - I see now why ES HIGHLY recommends flattened data approaches - the hasParent/hasChild queries can be complicated and super expensive, especially for aggregations.

[@ankurleonardo]

You deserve at least 10 million subscribers. All watching with zeal.

[@deez_narts]

I can attest to the challenges you described. I’m the only one in my organization using elasticsearch. Everyone else is afraid to touch it. I do not blame them.

[@skkooledd]

Great video and thought process!

[@abdirahmann]

am currently building a project that requires semantic search, i wanted to pay someone soo quickly to get this done (i can do it myself but i don't want to spend the time), i thought ELK would be such a breeze, thanks for saving me the time and money :)

[@toughcoding]

The worst thing is when you meet Solution Architect who want you to increase index mapping total_fields limit just to fit badly design mapping. Good point!
Good sound quality. wish to have same one day

[@GeorgeBridgemanData]

There's a time and a place for increasing it. It can sometimes get you out of a bind but it's not a good long term solution and you're better off fixing the mapping as soon as practical.

[@atom6_]

These are great points - within an organisation where elastic ends up in production without really fleshing out the way it should be used and having teams onboarding their data without true understanding of the data modelling that needs to be done and understood upfront is a major pain to fix later down the line. Common fields are essential (i would say critical) to make most of it when trying to get your data out in a useful way.

[@guyholmes1231]

Hi @GeorgeBridgemanData - can Elsatic search be used for types of files that it may never have seen before and "shown" how to parse them?

[@sidalisaadi1961]

I was planning to run a elastic search image for work on docker, thinking it would be a simple process. this video is an eye opener, now I'm not sure if I should go for it or not, our use case is something like yelp with much less data

[@nathansherburne7038]

How about using Elasticsearch for only term searches and doc id queries? Bad use case?

[@eladitzkovitch6203]

Hi George, how can I contact you in person? We would like to invite you as a guest to a function of ours to talk about Elasticsearch.

[@GeorgeBridgemanData]

Hi there. Feel free to contact me via LinkedIn.

[@mksmurff]

Well that's just put me off completely

[@abdulhayy225]

hi

[@alexgavril385]

amazing lessons