Data pipeline using Kafka - Elasticsearch - Logstash - Kibana | ELK Stack | Kafka

แชร์
ฝัง
  • เผยแพร่เมื่อ 29 ธ.ค. 2024

ความคิดเห็น • 66

  • @leamon9024
    @leamon9024 4 ปีที่แล้ว +2

    Awesome. Looking forward to more in-depth contents about ELK stack

  • @praveenchakravarthula8806
    @praveenchakravarthula8806 2 ปีที่แล้ว

    This is the one of the best video tutorial I have seen for education.
    very intuitive and required level of details

    • @AshuToshSharma99
      @AshuToshSharma99 2 ปีที่แล้ว

      Facing issues with cmak unable to create topic can you help

  • @shaiovadia5766
    @shaiovadia5766 2 ปีที่แล้ว +1

    Great tutorial - extensive enough and doen't skip any step.
    excellent work !

  • @pholohohlaha10
    @pholohohlaha10 8 หลายเดือนก่อน

    Wonderful work my brother, this has helped me a lot, what a presentation.

  • @pavankumard5276
    @pavankumard5276 8 หลายเดือนก่อน

    Really good video, I have subscribed to your channel looking forward to more such videos👍

  • @rabithbo7089
    @rabithbo7089 ปีที่แล้ว

    A very useful tutorial 😘🙏

  • @ElHiro03
    @ElHiro03 3 ปีที่แล้ว

    subscribed. Great tutorial and amazing channel.
    Saludos desde Argentina.

  • @uploadvoice
    @uploadvoice 4 ปีที่แล้ว +1

    Excellent !!!
    Great video

  • @srikanthchitturi7283
    @srikanthchitturi7283 3 ปีที่แล้ว

    Excellent explanation. Good job

  • @dannotes9396
    @dannotes9396 4 หลายเดือนก่อน

    Thanks for the video bro, very informative :)
    One non-related query, what font do you configured in the jet brains IDE? :)

  • @gouravkhatter2520
    @gouravkhatter2520 3 ปีที่แล้ว +3

    Informative, easily and cleanly explained.... subscribed (you owned) 😀
    Can I have reference guide for processing data before sending to elasticsearch?

  • @ksavinash
    @ksavinash ปีที่แล้ว

    @SelfTuts
    please share the working docker-compose setup lagstash as well
    Note: The video shows logstash local setup its working fine for me in a local setup. I'm trying add logstash to docker compose file as well but I'm unable connect to Kafka

  • @AbhimanyuKumar-ko1ih
    @AbhimanyuKumar-ko1ih 4 ปีที่แล้ว +3

    Very informative thanks

  • @sakshivats7574
    @sakshivats7574 ปีที่แล้ว

    How do I start logstash using cmd or shell . Bcs systemctl doesn’t work

  • @shafiquegkhan6400
    @shafiquegkhan6400 9 หลายเดือนก่อน

    Great video. Thnx 4 sharing

  • @dheerajasnani7918
    @dheerajasnani7918 3 ปีที่แล้ว

    Great tutorial, just what i was looking for. Thanks

  • @shridarshanramanand4766
    @shridarshanramanand4766 7 หลายเดือนก่อน

    This is best video i have ever been seen . Cool nice thank you . point to point any videos for ansible script patching or provision linux server through vmware clone ?

  • @lalitshukla6585
    @lalitshukla6585 2 ปีที่แล้ว

    Thanks Sir, do we have to pay or take youtube subscription for accessing your website ?

  • @sany2k8
    @sany2k8 2 ปีที่แล้ว

    Great tutorials, subscribed

  • @nicwhites
    @nicwhites ปีที่แล้ว

    15:30 You should simply do a lambda instead of created a one line function. Other than that, good video!

  • @BhavinShahBlog
    @BhavinShahBlog ปีที่แล้ว

    Excellent post. The only problem is the pace is very fast.. I guess using AI it can be re-recorded with slower pace professionally

  • @dmcp3261
    @dmcp3261 4 ปีที่แล้ว +1

    subscribed - awesome content.

  • @anupamchowdhury0
    @anupamchowdhury0 ปีที่แล้ว

    How logstash connect to secure kafka using keycloak?

  • @edgarnaula2835
    @edgarnaula2835 3 ปีที่แล้ว

    All services get up except kafka. Any idea why that happens?

  • @satishchhatpar
    @satishchhatpar 3 ปีที่แล้ว

    Very good demo. Thank you

  • @salihuumar3138
    @salihuumar3138 ปีที่แล้ว

    A wonderful video

  • @satishchhatpar
    @satishchhatpar 3 ปีที่แล้ว

    your website is not working. Gives error "Error establishing a database connection"

  • @HindiEditzCountdown
    @HindiEditzCountdown 2 ปีที่แล้ว

    so nice tutoorial bro

  • @Vivek.1011
    @Vivek.1011 3 ปีที่แล้ว +1

    the links are broken, can you update them

  • @balinasuryachandra9337
    @balinasuryachandra9337 4 ปีที่แล้ว

    Hey SelfTuts Awesome explanation
    i have small doubt, i installed kafka, elk environment on different instances IN AWS
    to establish a connection between kafka and logstash which ip address should i provide either public IP or private IP in conf directory of logstash

  • @tatavarthyvenkatesh4390
    @tatavarthyvenkatesh4390 4 ปีที่แล้ว

    Can we monitor.. consumer lag , fetch lag and all.. using logstach and kibana?

  • @himanshuagarwal7628
    @himanshuagarwal7628 3 ปีที่แล้ว

    why your site is currently not reachable?

  • @86Saurabh1
    @86Saurabh1 3 ปีที่แล้ว

    Hello, great tutorial. I am having some issues when both containers are running. I can see that all the 5 services are running when using command "docker ps" but when I try to open the UI for either kibana or kafka_manager , both won't open, only one would open. When I run the containers one at a time then kibana or kafka_manager would open, depending upon which service is running, but not together. Any help would be appreciated.

    • @86Saurabh1
      @86Saurabh1 3 ปีที่แล้ว

      Issue seems to be with Kafka Manager service, as without this service I can run Kibana and other services. Need to find an alternative to this.

  • @skidrow406
    @skidrow406 7 หลายเดือนก่อน

    great, thanks

  • @AnthonyPinero
    @AnthonyPinero ปีที่แล้ว

    Hi nice tutorial, do you think you can share the sources in any repository?

  • @srikanthjnr
    @srikanthjnr 3 ปีที่แล้ว

    very good video! keep it up!

  • @billmcguire6128
    @billmcguire6128 3 ปีที่แล้ว

    Great video - thanks

  • @aj_abdel7113
    @aj_abdel7113 2 ปีที่แล้ว

    Best videos!

  • @ajay-lu5gh
    @ajay-lu5gh 2 ปีที่แล้ว

    Thanks a lot!

  • @harrydam_
    @harrydam_ 3 ปีที่แล้ว

    Does Your website still down?

  • @joepoptiya
    @joepoptiya 4 ปีที่แล้ว

    @selftuts, did you know your website is down?

  • @localboyvijay9628
    @localboyvijay9628 2 ปีที่แล้ว

    website is not working, Can you please check

  • @rvpandey99
    @rvpandey99 2 ปีที่แล้ว +1

    your website is down.

  • @hemantjain9850
    @hemantjain9850 3 ปีที่แล้ว +1

    website is down

  • @saisharath5515
    @saisharath5515 2 ปีที่แล้ว +1

    Your website is not reachable

  • @stockmarketBullsVsBears
    @stockmarketBullsVsBears 4 ปีที่แล้ว

    Can you add ssl info for secure data transfer

  • @krystian-magdziarz
    @krystian-magdziarz 3 ปีที่แล้ว

    Amazing.

  • @davidmadronalsanchez4377
    @davidmadronalsanchez4377 3 ปีที่แล้ว

    links are down?

  • @mosseery8118
    @mosseery8118 3 ปีที่แล้ว

    Awesome

  • @hemp_gaming
    @hemp_gaming 4 ปีที่แล้ว +1

    Hello, can you do a video about api connections in kafka,
    About real-time, data processing.

  • @freefallinu3446
    @freefallinu3446 3 ปีที่แล้ว

    Cool!

  • @rufianarchamdani3848
    @rufianarchamdani3848 3 ปีที่แล้ว

    good very cool

  • @preethikannan98
    @preethikannan98 2 ปีที่แล้ว

    Given links are not reachable

  • @lucaspendola7696
    @lucaspendola7696 3 ปีที่แล้ว +3

    version: "3.3"
    services:
    elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.4.0
    container_name: elasticsearch
    restart: always
    environment:
    - xpack.security.enabled=false
    - discovery.type=single-node
    ulimits:
    memlock:
    soft: -1
    hard: -1
    nofile:
    soft: 65536
    hard: 65536
    cap_add:
    - IPC_LOCK
    volumes:
    - elasticsearch-data:/usr/share/elasticsearch/data
    ports:
    - 9200:9200
    kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:7.4.0
    restart: always
    environment:
    - ELASTICSEARCH_HOSTS=elasticsearch:9200 # address of elasticsearch docker container which kibana will connect
    ports:
    - 5601:5601
    depends_on:
    - elasticsearch # kibana will start when elasticsearch has started
    volumes:
    elasticsearch-data:

    • @lucaspendola7696
      @lucaspendola7696 3 ปีที่แล้ว +1

      version: "3"
      services:
      zookeeper:
      image: zookeeper
      restart: always
      container_name: zookeeper
      hostname: zookeeper
      ports:
      - 2181:2181
      environment:
      ZOO_MY_ID: 1
      kafka:
      image: wurstmeister/kafka
      container_name: kafka
      ports:
      - 9092:9092
      environment:
      KAFKA_ADVERTISED_HOST_NAME: [machine-ip-address]
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      kafka_manager:
      image: hlebalbau/kafka-manager:stable
      container_name: kakfa-manager
      restart: always
      ports:
      - "9000:9000"
      environment:
      ZK_HOSTS: "zookeeper:2181"
      APPLICATION_SECRET: "random-secret"
      command: -Dpidfile.path=/dev/null

    • @vineethkumar7803
      @vineethkumar7803 2 ปีที่แล้ว +1

      @@lucaspendola7696 Thanks for sharing. Saves lot of time when the website provided in description is down.