► Follow me on IG for behind the scenes content 😊👉🏼 bit.ly/2F3LXYJ ► This video was sponsored by the Cloud Native Computing Foundation www.cncf.io/ 🙌🏼 ► More infos on Fluentd: www.fluentd.org/ ▬▬▬▬▬▬ T I M E S T A M P S ⏰ ▬▬▬▬▬▬ 0:00 - Intro 0:26 - Why we actually need log data? 1:40 - Challenges of collecting and consuming logs 3:40 - Solution - a unified logging layer 4:17 - How does Fluentd work 6:29 - How to configure Fluentd? 9:03 - Built-In Reliability 10:09 - Other Use Cases
Have had fluentd daemonset set up in my kubernetes cluster, pushing pod, service, and other logs to elasticsearch for a month or so now. Really like the simplicity and flexibility it offers.
Very interesting. I heard about fluentd in another place and came here to find some info. Glad I found your video, easy to understand and digest. Thanks!
Nice, my attention was drawn to fluentd by your Udemy course and I was looking for more information about it a few days ago and now you made a video about it 👌
Great Video Nana! I use FluentD for quite some time already collecting logs and sending them to specific indexes outisde of the cluster so they can easially be observed using query's within observability platforms :)
first of all, you are my favourite in my internet world, I learn a lot from you, thank you mam. sorry for my english. in my language which is my favourite आपको बहुत बहुत धन्यवाद।
I will actually make a follow-up video on Fluentbit as well. One of the common setups, especially in bigger clusters is, fluentbit pods as collectors and fluentd as an aggregator. Because fluentbit is more lightweight, so such setup would give you good performance while saving resources.
Can fluentd collect logs using push-model like logstash? It is convenient to just add handler/appender to e.g. logback configuration and forget about log format.
Thanks for this video. is there a way to limit nested parsing in json parser plugin @t ? as current it's parsing json in all levels to lots of field getting created. to wanted to limit josn parsing on first level.
Can you search/parse collect a specific string in fluentD like sumo logic. Can you set up dashboards on FluentD itself without pushing it to a destination system
That's a common question. Fluentd is more fit for k8s than Logstash, it integrates better and was created with k8s in mind. Plus it has some advantages, like routing the logs is much easier in Fluentd as well as Logstash needs additional service for data persistence. These are some of the main diffs.
Hello.. thanks for taking the time to put together this video. It’s excellent, it really does help with conceptualizing how to utilize fluentd.. I got it up and running and I am capturing logs from several of my back up processes and syslogs from few servers. Something I’ve not been able to figure out is how to dynamically assign labels to my Loki match blocks. I’ve used the rewrite_tag_filter plugin to push log lines to different match blocks where I have different labels but that feels like I’m hard coding the same thing over and over just to get one label to be different. I’d like to store the program or process name in a variable based on the log line and use it as a label. Is that possible? I’ve tried several different ways but I have not been successful doing that as yet. Any info would be greatly appreciated. Thanks again *edit when I say labels I’m referring to labels used in Loki and visible in Grafana
Thanks for the video...Currently I am using Fluentd but it's can't go beyond 2k/EventPerSec/PerIntance. Where I am looking for 20k/EPS/PerInstance with each average event size of 800 Byte. Do you have any suggestion for the same?
Dear Nana, when i try to deploy fluentd using helm chart by default it deploys as root user. i changed to non root user by using security context. But the problem is it errors out "[error]: #0 unexpected error error_class=Errno::EACCES error="Permission denied @ rb_sysopen - /var/log/fluentd-containers.log.pos". Can you please advise in case if you had this issue
Yes Fluentbit is more lightweight. Needs much less resources. However Fluentd has more features, plugins. So depends on what you need to configure. If Fluentbit functionality is enough, then you can go with it. I will actually make a video about Fluentbit next month :)
Hey Nana, the content of your video is always very good and easily understandable. Except for your audio level, it seems low to me. I have tried many ways to increase your audio and try different devices as well but it didnt increase the sound volume. please make sure your videos are having little high audio quality so that some one can easily listen to it without earphones or headphones. I hope it wouldn`t appear offensive to you.
Dear Nana, I like your videos they are very good with good content. But in this case it was not the quality I was expecting :) your videos are watched by IT pros you need to give more in dept knowledge instead of just high level overview. Please :) But overall I like your vides, keep up the good work :)
I am having a bad experience for the last 3 days even though I have paid for the course. Please provide me a refund if you cannot help me open the second video of this course....
Hi Ash, can you please send an email to techworld-with-nana@nnsoftware.at and provide email with which you have purchased the course? So we can check. On TH-cam comments there is no support Is this your email: goashtech@gmail.com ?
I don't think the explanation is good. it is so abstracted and I wonder if you really have configured it by your self. Is Fluentd flies along the applications like the animation you put on 4:26?
I would say that I don't consume logs, I don't eat them, I don't inhale or inject them into my bloodstream ;-) I normally just view them. I prefer consuming food.
► Follow me on IG for behind the scenes content 😊👉🏼 bit.ly/2F3LXYJ
► This video was sponsored by the Cloud Native Computing Foundation www.cncf.io/ 🙌🏼
► More infos on Fluentd: www.fluentd.org/
▬▬▬▬▬▬ T I M E S T A M P S ⏰ ▬▬▬▬▬▬
0:00 - Intro
0:26 - Why we actually need log data?
1:40 - Challenges of collecting and consuming logs
3:40 - Solution - a unified logging layer
4:17 - How does Fluentd work
6:29 - How to configure Fluentd?
9:03 - Built-In Reliability
10:09 - Other Use Cases
Have had fluentd daemonset set up in my kubernetes cluster, pushing pod, service, and other logs to elasticsearch for a month or so now. Really like the simplicity and flexibility it offers.
Never tried fluentd before, but I definitively will, thanks to this video. I love this channel. Please keep 'em coming.
Glad to hear, thank you Farid :)
wow, super good quality video, thanks for the great effort!
Very interesting. I heard about fluentd in another place and came here to find some info. Glad I found your video, easy to understand and digest. Thanks!
Nice, my attention was drawn to fluentd by your Udemy course and I was looking for more information about it a few days ago and now you made a video about it 👌
😊👍
That's amazing 🔥
We need more tutorials on EFK stack 🙏
I strugle with this a few days ago. I wish i had found this video earlier! thanks
Great Video Nana! I use FluentD for quite some time already collecting logs and sending them to specific indexes outisde of the cluster so they can easially be observed using query's within observability platforms :)
first of all, you are my favourite in my internet world, I learn a lot from you, thank you mam. sorry for my english. in my language which is my favourite आपको बहुत बहुत धन्यवाद।
Spectacular explanation!!! 👏🏽👏🏽👏🏽👏🏽👏🏽
Nice - short but informative. Thank you.
Thanks Vasily :)
The DevOps hero we didn't know we needed :)
Great. Nana. You always enlightened my knowledge. EFK becomes my target K8s logging framework. Thanks.
This content is so good it should be paid
Thanks for your videos, I'm a pretty bad english listener and I understood all you said and I learned a lot with !!!!
I do really like the way you express and deliver your thoughts, much, much appreciated!
Please go on with your videos! :)
+
BTW - how can we support you, donating anyhow? Thanks!
Dear Nana. Very interesting Your opinion. Fluentd vs Fluent-bit? And which product is better to use in 2021? Thank you 😊
I will actually make a follow-up video on Fluentbit as well. One of the common setups, especially in bigger clusters is, fluentbit pods as collectors and fluentd as an aggregator. Because fluentbit is more lightweight, so such setup would give you good performance while saving resources.
@@TechWorldwithNana Thank you Nana!
As usual Absolutely fantastic content thanks a lot for your wonderful explanation.
Nice intro. I was expecting more of a how to actually install and configure fluentd
great video 🎉
Another bang on video thank you for your help please add some slides to take notes.
You are the best woman in the world , I love ur info ❤
Great content as always. Thank you!
Nice explaination. Good job. Can you also do a video on OpenTelemetry?
As usual the best!!
Very well explained.
Thank you ! very useful content
Thank you Mark!
Thanks for your videos
what about logstash !! thanks for the video .
Fluentd is actually more fit for K8s, it was actually created with k8s in mind.
@@TechWorldwithNana Fluentd was created when such things as Kubernetes does not exist at all
A great way of explanation indeed.
Can fluentd collect logs using push-model like logstash? It is convenient to just add handler/appender to e.g. logback configuration and forget about log format.
Great content (same comment again but this content is really good)
Happy to hear, thank you Daniel 😊
Hi Anna, your video is always perfect and helpful. I wonder how you can make like this video, what are tools to do this
Thank you so much! Is very interisting
should I choose Fluentd over logstash/filebeat? @nana
very well broken down, the config snippets and animations help to visualize it a as well, thanks !
Dear Nana, could you please provide the video on design , how to get the health and pod level metrics for the applications deployed in openshift?
Thanks for this video. is there a way to limit nested parsing in json parser plugin @t ? as current it's parsing json in all levels to lots of field getting created. to wanted to limit josn parsing on first level.
great explanation as always Nana, what is the logging solution we can integrate for Prometheus to collect application logs?
Thank you for the nice presentation. What tool would you recommend using when visualizing the log data?
I've used EFK stack for that so I think Kibana is really great for visualising logs!
Thank you !!!
Thank you. The contents are same as your Fluentbit lecture. Could you please give bit precise use case info when to use which one or how to use both.
Can u please let me know how to push specific namespace logs only to elasticsearch from fluentd in k8s any specific parameters we need to use ?
Do you have lessons on linked in as well ? Couldn’t find ?
Hi Nana, Can Fluentd replace #autodiscover feature of filebeat? Which helps to get the logs of #AUTOSCALING environment
very useful
If I want anyone collecting my data... It's Fluentd!
Hey how does it compare to cribl ? I cant rely spot wheres the diffences right now.
What's the difference with ELK stack?
Need to know this too.
Subbed👍🏻
Can this be used for logs from iOS application?
start at 4:22 :)
which one is most fit for gcp cloud run logs collecting?
Can Fluentd collect pod/containers metrics ? if not, what do so in Docker & Kubernetes environments ?
It's worth watching and learning too. How it differs from Jaeger ? Please throw some light on that. Thanks in advance.
Could you share your views on how Fluentd is different then splunk for logging apart from the fact that splunk is commercial
Can you search/parse collect a specific string in fluentD like sumo logic. Can you set up dashboards on FluentD itself without pushing it to a destination system
Thank you - Crystal clear - ***** Stars
Thanks Fayaz 🙏
Filebit or logstash also does same thing then why we should go with fluentd..
And also plz explain fluentd config file in more details..
That's a common question. Fluentd is more fit for k8s than Logstash, it integrates better and was created with k8s in mind. Plus it has some advantages, like routing the logs is much easier in Fluentd as well as Logstash needs additional service for data persistence. These are some of the main diffs.
Is it worth using fluentd together with Prometheus? Or is it more like an alternative?
Amazing video, thanks a lot🔥.Could you also please make one lesson on using ArgoCD for Kubernetes as the GitOps continuous delivery tool?
Hello.. thanks for taking the time to put together this video. It’s excellent, it really does help with conceptualizing how to utilize fluentd.. I got it up and running and I am capturing logs from several of my back up processes and syslogs from few servers. Something I’ve not been able to figure out is how to dynamically assign labels to my Loki match blocks. I’ve used the rewrite_tag_filter plugin to push log lines to different match blocks where I have different labels but that feels like I’m hard coding the same thing over and over just to get one label to be different. I’d like to store the program or process name in a variable based on the log line and use it as a label. Is that possible? I’ve tried several different ways but I have not been successful doing that as yet. Any info would be greatly appreciated. Thanks again
*edit when I say labels I’m referring to labels used in Loki and visible in Grafana
Thanks for the video. is it possible fluent-logger-java integrate in android? if possible please provide details?
Thanks for the video...Currently I am using Fluentd but it's can't go beyond 2k/EventPerSec/PerIntance. Where I am looking for 20k/EPS/PerInstance with each average event size of 800 Byte. Do you have any suggestion for the same?
I'm new to all this. Is nlog just another logger like fluentd?
Dear Nana, when i try to deploy fluentd using helm chart by default it deploys as root user. i changed to non root user by using security context. But the problem is it errors out "[error]: #0 unexpected error error_class=Errno::EACCES error="Permission denied @ rb_sysopen - /var/log/fluentd-containers.log.pos". Can you please advise in case if you had this issue
thanx ❤❤
Thanks for sharing..
Though AWS documents suggesting to use FluentBit instead of FluentD..
Yes Fluentbit is more lightweight. Needs much less resources. However Fluentd has more features, plugins. So depends on what you need to configure. If Fluentbit functionality is enough, then you can go with it. I will actually make a video about Fluentbit next month :)
@@TechWorldwithNana You care for us so much :)
Hi @TechWorldwothNana when the the DevSecOps course going to be available
Hey Nana, the content of your video is always very good and easily understandable.
Except for your audio level, it seems low to me. I have tried many ways to increase your audio and try different devices as well but it didnt increase the sound volume. please make sure your videos are having little high audio quality so that some one can easily listen to it without earphones or headphones.
I hope it wouldn`t appear offensive to you.
You don’t have patreon or smth like that? You are doing an amazing job helping the rest of the community and I would like to give back smth :)
Do you have any plans on creating videos about podman, buildah, skopeo?
I didn't plan it for now, but will definitely consider them! 👍
Dear Nana, I like your videos they are very good with good content. But in this case it was not the quality I was expecting :) your videos are watched by IT pros you need to give more in dept knowledge instead of just high level overview. Please :)
But overall I like your vides, keep up the good work :)
Nana , sound volume is a bit low , :)
Thanks for the feedback!
Hello.. how is it different from Splunk?
You could compare FluentD to Splunk forwarders. Preparsing, transforms etc. It's not a full product as Splunk, just a part of a stack.
Nice video, but I can't get the difference between this tool and a broker tool as for example CAMEL
How to create Fluentd index with application name , Like Fluend-containername , deployed with helm
It should be somehow marked that this is K8s related instruction. Im looking for plain docker instructions
Suberb mam
why not looking at a more capable log management solution such as syslog-ng?
Nana, I need your help,...!!!!
I am having a bad experience for the last 3 days even though I have paid for the course. Please provide me a refund if you cannot help me open the second video of this course....
Hi Ash, can you please send an email to techworld-with-nana@nnsoftware.at and provide email with which you have purchased the course? So we can check. On TH-cam comments there is no support
Is this your email: goashtech@gmail.com ?
I don't think the explanation is good. it is so abstracted and I wonder if you really have configured it by your self. Is Fluentd flies along the applications like the animation you put on 4:26?
დიდი მადლობა მსგავსი კარგი ტუტორიალები არსად არაა)
still wonder why logstash is left behind and fluentd becomes more common
I would say that I don't consume logs, I don't eat them, I don't inhale or inject them into my bloodstream ;-) I normally just view them. I prefer consuming food.
First
it would be much better if you explained how to install and manage it than listening of 10 minuts of bla bla bla fluentd is good fluentd is the best
why pointing the finger in thumbnail ? '-[
all logs to splunk, period !
many Splunk users use Fluentd to perform the data ingestion to Splunk, there is a good value having the option to control "which" data is ingested.