What are Elasticsearch shards? Why do they matter? Elasticsearch cluster architecture explained.

แชร์
ฝัง
  • เผยแพร่เมื่อ 21 ก.ค. 2021
  • Elasticsearch is a fantastic tool but it's easy to muddle through without knowing the fundamentals. It's only a matter of time before your cluster performance drops, errors start happening, and you wonder just what a shard actually is.
    I talk and wave my hands about, explaining how we could have - maybe - built - some of - Elasticsearch ourselves. At the end of the video you'll know what a shard is, why they're so important, and want to learn more to really improve the performance of your cluster.
    Watch on (and watch the rest of the free course linked below) to figure out how to answer that mysterious question: How many primary shards do I need for my index?
    My course - Fundamentals of Elasticsearch architecture and shards - is available for free here: school.georgebridgeman.com/p/...
    You can read more about Elasticsearch and The Elastic Stack in general on my blog: georgebridgeman.com/posts/

ความคิดเห็น • 63

  • @nch77884
    @nch77884 2 ปีที่แล้ว +40

    Hands down the best explanation and introduction to Elasticsearch. Can't thank enough for making this video.

    • @GeorgeBridgemanData
      @GeorgeBridgemanData  2 ปีที่แล้ว

      Thank you for your lovely comment. I'm so pleased you enjoyed the video!

    • @zaeemahmedabbasi
      @zaeemahmedabbasi 9 หลายเดือนก่อน

      ​@@GeorgeBridgemanData⁷77777777777777777777777777777777777777777777777777777777777777777777😅😮

  • @netlob
    @netlob 2 ปีที่แล้ว +12

    Holy sh*t! This must've been the best tutorial I've ever seen on TH-cam. High production, clear presentation and well thought through. +1 subscriber for sure!

  • @HaisumUsman
    @HaisumUsman ปีที่แล้ว +2

    Man! You are not from this planet! You deserve a thousand thumbs up.

  • @sridharnuthi1
    @sridharnuthi1 ปีที่แล้ว +7

    This is a video that should go to a reference library about Elasticsearch. Thank you for putting such a good, clear and methodical overview of ES. Just brilliant!

  • @harrisonleong4283
    @harrisonleong4283 2 ปีที่แล้ว +3

    I really wish this was the 1st elasticsearch video that I had watched, so as to save me so much time watching other video which could not teach me the same level of information that I need. Thank you very much, and I shall check out your courses.

    • @GeorgeBridgemanData
      @GeorgeBridgemanData  2 ปีที่แล้ว

      That's the reaction I was hoping for - a useful first video on Elasticsearch. Thank you so much for posting!

  • @lawlade
    @lawlade ปีที่แล้ว +3

    Watched this 5 times, rewound several times and i understand it FULLY. Thanks so much for such clear explaining

    • @GeorgeBridgemanData
      @GeorgeBridgemanData  ปีที่แล้ว

      You're welcome! I'm so pleased you understood everything. I hope it helps!

  • @TheOtmane007
    @TheOtmane007 ปีที่แล้ว +2

    What a clear , and progressively explained architecture. Thank you so much

  • @crujzojam7004
    @crujzojam7004 2 ปีที่แล้ว +3

    Please post more videos…ur videos are easy to understand and quite informative….please carry on the good work

    • @GeorgeBridgemanData
      @GeorgeBridgemanData  2 ปีที่แล้ว +1

      Done! New video just posted!
      There will be more. I have lots of ideas but I'm really trying to get this my Elasticsearch course finished, and work the day job.

  • @thomasanderson8478
    @thomasanderson8478 2 ปีที่แล้ว +3

    This is the best explanation of elasticsearch I've ever seen. So many videos skip over the details, and it's been making it difficult to understand what elasticsearch is doing under the hood. I normally don't comment on videos, but this is too high quality not to. Please continue to put out content!

    • @GeorgeBridgemanData
      @GeorgeBridgemanData  2 ปีที่แล้ว

      What a lovely comment! Thank you so much. It means a lot!

  • @blossomwithcurls
    @blossomwithcurls 5 หลายเดือนก่อน

    I just started learning Elastic search and this is the best an clear information on Elastic search architecture. Thanks for sharing!

  • @yazzy9975
    @yazzy9975 10 หลายเดือนก่อน

    This video changed my life. No exaggeration.

  • @jupudivinod
    @jupudivinod 2 ปีที่แล้ว +2

    This is fantastic! Bricks till walls in a nutshell! Thanks much for this great presentation.

    • @GeorgeBridgemanData
      @GeorgeBridgemanData  2 ปีที่แล้ว +1

      You're welcome! I'm really pleased you enjoyed it.

  • @Transactional
    @Transactional ปีที่แล้ว +1

    Thank you. It feels like my brain is getting clearer.

  • @sobhan285
    @sobhan285 2 ปีที่แล้ว +1

    Wonderful. Looking forward to more courses from you.

    • @GeorgeBridgemanData
      @GeorgeBridgemanData  2 ปีที่แล้ว

      Thanks so much! Elasticsearch Engineer Essentials is in the works, and I'll be posting shorter content on here as well.

  • @tkousek1
    @tkousek1 2 ปีที่แล้ว +1

    Thank you very much sir for this information. Awesome people like you are what's good about this world!!! Much appreciated!!!

  • @user-tm6cd8sg5l
    @user-tm6cd8sg5l 2 หลายเดือนก่อน

    Many thanks for your work! It's awesome video!

  • @thsu1
    @thsu1 2 ปีที่แล้ว +1

    thanks for the clear and awesome explanation to Elasticsearch and Lucene. really appreciate this useful content

    • @GeorgeBridgemanData
      @GeorgeBridgemanData  ปีที่แล้ว

      I'm so pleased you appreciated it. It was great to make it!

  • @jhoyl
    @jhoyl 2 ปีที่แล้ว +1

    Thanks - the perfect introduction to Elasticsearch architecture.

  • @johnsonakanbi367
    @johnsonakanbi367 2 ปีที่แล้ว

    Thanks so much for this great presentation.

  • @pseudolimao
    @pseudolimao หลายเดือนก่อน

    where was thsi video 1 month ago. you should be paid by these software companies... bless your heart

  • @hnyc1986
    @hnyc1986 ปีที่แล้ว +1

    Awesome explanation about Elasticsearch!!!

  • @user-nt7of9us8p
    @user-nt7of9us8p 4 หลายเดือนก่อน

    Very clear! Thanks!

  • @riazbacchus3962
    @riazbacchus3962 8 หลายเดือนก่อน

    this is great content. thank you.

  • @arpit9163
    @arpit9163 ปีที่แล้ว +1

    Thank You for making this fantastic video !

  • @cicd
    @cicd 2 ปีที่แล้ว +1

    Great content, thanks for sharing!

  • @rakeshkush1234
    @rakeshkush1234 ปีที่แล้ว +1

    wonderful technical story.

  • @moritzlgrs401
    @moritzlgrs401 2 ปีที่แล้ว +1

    Absolutely fantastic!

  • @ucthuannguyen6432
    @ucthuannguyen6432 2 ปีที่แล้ว +1

    Wonderful. Thank you so much.

  • @hieungo770
    @hieungo770 5 หลายเดือนก่อน

    Its two year late but the lesson is extremely value

  • @PhanTanThangTH
    @PhanTanThangTH 3 หลายเดือนก่อน

    Thank you so much :)

  • @slapcanister
    @slapcanister 2 ปีที่แล้ว +1

    This is so good.

  • @danielsantiago11
    @danielsantiago11 ปีที่แล้ว +1

    Premium content, thank you!

  • @andy_ltluan
    @andy_ltluan 10 หลายเดือนก่อน +1

    I think that shard in ES has the same concept with partition in Kafka when they have all partition replicas in different nodes

  • @DrewIsFail
    @DrewIsFail 2 ปีที่แล้ว +1

    Is it fair to say you could build ES from dynamoDB? I'm trying to compare the two.
    I would love a video on the query language, does it have a mathematical basis like sql does to sets?
    It goes without saying, but I'm say it, thanks for making this clear, concise, focused high level content.

    • @GeorgeBridgemanData
      @GeorgeBridgemanData  2 ปีที่แล้ว

      Hi there. I'm really pleased you enjoyed the video.
      I'm not sure you could build an equivalent of Elasticsearch using DynamoDB. There's a *lot* more to Elasticsearch than I talked about in this video!
      There's more content coming, including an introduction to the query language. The in-depth content will be in a training course instead of TH-cam, though. I've never considered if there's a mathematical basis to the query language. I doubt there is in terms of what Elasticsearch offers, but all Elasticsearch queries are converted to Lucene query language, which may be more thoroughly researched. Interesting question!

  • @carlosroberto366
    @carlosroberto366 2 ปีที่แล้ว

    Isn't the cluster the server (i.e. AWS EC2 instance) itself? To my mind, a node is not a server because you can create several nodes in the same machine. I was expecting to see MyCluster1 and MyCluster2 each having a single node, hence, high availability via cross-cluster communication.
    11:11 node = server in his example
    15:05 node = process

    • @GeorgeBridgemanData
      @GeorgeBridgemanData  2 ปีที่แล้ว +1

      A node is an Elasticsearch process running on a host. You're right that you can run multiple nodes on the same host (even not containerised), but it's not recommended and it's widely accepted that you only run a single node on a host.
      If you did run two nodes on a single host, you could have either one or two clusters on that host. The node is configured with the cluster name it's expected to join, so you could configure each node with a different cluster name and have two clusters on that host!
      Cluster formation can get quite involved. There are configuration settings that need to be applied specifically at the formation stage. I can do a video on how that works at some stage.

  • @yazzy9975
    @yazzy9975 10 หลายเดือนก่อน

    If elasticsearch distributes the data between the shards of an index such that each lucene store roughly holds the same number of documents, when you run a search query, elasticsearch, despite the inter-node communication, only knows which shards hold that index and not which particular shard will have that document? So it has to run the query against all the shards and merge results, it cannot just search the one shard that contains that document? It does not know beforehand based on how documents are distributed among shards.

  • @akshaychawla7413
    @akshaychawla7413 ปีที่แล้ว

    I am not able to enroll for your course, tried with 2 different emails, please have a look into this.

    • @GeorgeBridgemanData
      @GeorgeBridgemanData  11 หลายเดือนก่อน

      Sorry for the very late response. I've had feedback from a couple of people using Firefox, who worked around it by using a different browser. I'm not sure if that's your issue but thought I'd mention it. Let me know if you're still having issues and I'll try responding quicker this time!

  • @Milostrosic
    @Milostrosic ปีที่แล้ว +1

    Very clear explanation!

  • @sv_n
    @sv_n 3 หลายเดือนก่อน

    1000th like 😅