Mastering Grafana Loki: Complete Guide to Installation, Configuration, and Integration | Part 1
ฝัง
- เผยแพร่เมื่อ 2 ส.ค. 2024
- 🚀AWS Cloud Practitioner Course: kode.wiki/AWSCloudPractitione...
Welcome to Part 1 of our comprehensive guide to Grafana Loki! In this video, we'll take you through everything you need to know about Grafana, the Loki architecture, step-by-step installation instructions, configuration tips, and a quick demo on integrating Grafana with Loki. Loki is a powerful logging backend, designed to be cost-effective and easy to operate. Whether you're new to Loki or looking to enhance your monitoring stack, this series is for you! Don't forget to subscribe and hit the bell icon for future updates on Part 2 and other exciting tech content!
🆓Join our Slack Community for FREE: kode.wiki/JoinOurSlackCommunity
⬇️Below are the topics we are going to discuss in this AWS Updates video:
00:00 - Introduction
01:26 - What is Loki
03:34 - Architecture of Loki
06:43 - Loki Installation
10:50 - Promtail installation
17:20 - Querying Logs
22:44 - Collecting App logs
29:56 - Conclusion
✅Start your DevOps Career Now - bit.ly/DevOpsLearningPath-YT
We have various Learning Paths to help you choose your next step and shape your DevOps & Cloud Career.
Check out our learning paths at KodeKloud to get started:
▶️ Cloud: kode.wiki/CloudLearningPath
▶️ Linux: bit.ly/LinuxLearningPath
▶️ Kubernetes: bit.ly/KubernetesLearningPath
▶️ DevOps Learning Path: bit.ly/DevOpsLearningPath-YT
▶️ Docker: bit.ly/DockerLearningPath
▶️ Infrastructure as Code(IAC): bit.ly/IACLearningPath
▶️ Programming: bit.ly/ProgrammingLearningPath
#loki #grafanaloki #lokiinstallation #devops #kodekloud
For more updates on courses and tips, follow us on:
🌐 Website: kodekloud.com/
🌐 LinkedIn: / kodekloud
🌐 Twitter: / kodekloudhq
🌐 Facebook: / kodekloudhq
🌐 Instagram: / kodekloud
🌐 Blog: kodekloud.com/blog/
Great explanation and very easy to understand and follow along. Hope to see more like this.
Appreciate your gratitude! We strive to make complex topics accessible. If there's a specific area you'd like us to cover, let us know. We're here to help!
That was really awesome demo on Loki, thanks Kodekloud team.
Thank you for the appreciation. You can check our videos related to DevOps and subscribe to our channel to stay updated with all the latest tools.
@@KodeKloud I am already a fan of Kodekloud and have taken yearly subscription 😊
Great explanation and very easy to understand!!👏
Fantastic! Very well put together
Glad you think so!
That's was really clean and clear explanation ❤
Thanks a ton!
Great explanation brother.Respect.
Glad it was helpful!
Good and straight!
Hi, we appreciate the kind comment! enjoy!
Very good video, Thank you for making it!
Glad you liked it!
Thank you so match for this demo. It was very hopefull for me
Glad it was helpful!
amazing!
Nice video bro, keep it up ☺
Thanks for the video. How do we set up to view Loki’s logs using systemd (rpa)
Great!
Hi, we appreciate the kind comment! enjoy!
Great ❤
Thanks for watching our video. Cheers!
Thanks for the video.
I am using Grafana cloud, and I want to collect logs from the Airflow. Now as you said, I am only getting the labels, not the actual logs. So if I use promtail, would I get all the logs which are present in the Airflow?
Also, if you could explain a bit more what do you mean by “basic logs” 4:18
Thanks for the video
I have a simple question regarding Promtail; I am using regexp to set labels. Are all the lines in the log that don't meet the regexp still sent to Loki? If "yes", is there any way to not do that and just send to Loki only the lines that pass the regexp?
In other words I’d like to send to Loki only a subset of log lines.
good 😬
This is awesome, can you also make video on installing loki with helm chart ?
"Hey, thank you for watching our video. We will definitely look into your suggestions. Do subscribe and stay tuned for updates on our channel. Cheers :)
"
@@KodeKloud Hey i would also be really interested on this topic , even using minikube aand grafana
/loki
This was amazing and very helpful but i have a question .. is grafana loki better option in general than splunk ?
"Grafana Loki is cost-effective, simpler, and integrates well with Grafana. It's a good choice for straightforward log management, especially in environments already using Grafana.
Splunk offers more comprehensive log management and advanced analytics features. It's better suited for complex log analysis and large-scale enterprise use, but it's typically more expensive.
Choose Loki for a simpler, more budget-friendly solution, and Splunk for advanced features and large-scale operations."
after the promtail start command, am getting the output same as here,but when i check the grafana console i cant see my log file paths or job names over there, i can only see the existing files and log paths of other servers, do i need to refresh or do anything on the grafana or loki server, please advice
"If you're not seeing your log file paths or job names in Grafana after starting Promtail, try these steps:
Check Promtail's Configuration: Ensure it's correctly set up to tail the desired log files.
Verify Log File Access: Promtail should have permission to read these files.
Look at Promtail Logs: Check for any errors or warnings.
Confirm Loki Receives Logs: Use Loki's UI or API to verify.
Check Grafana's Data Source Setup: Ensure it's correctly connected to Loki.
Refresh Grafana: Sometimes, refreshing the dashboard or creating a new panel helps.
Review Network Connectivity: If Promtail and Loki are on different servers.
Examine Grafana Queries: Make sure they are correctly formulated.
Restart Services: Restarting Promtail and/or Grafana can resolve issues.
Consult Documentation: For any missed configuration steps."
In my current role we want build the open source like loki, minor, tempo, is it possible do same videos for mimir and tempo with s3 bucket as datastore in k8s and best practice for production level
Hey, thank you for your thoughts, we might consider doing it in the future.
how can configure promtail to send multiple containers logs which are running on the same machine to loki with different labels
To configure Promtail to send logs from multiple containers on the same machine to Loki with different labels, create a Promtail configuration file and define separate scrape jobs for each container. Customize the labels and paths for each job to distinguish between the containers, and then start Promtail with this configuration. It will send logs to Loki with the specified labels for differentiation.
I have a issue in this scenario, grafana not show all pods log,
Give solution for that ...plz..
Can't we install/configure Grafana/Loki in the application server ?
Nice, when you post part 2 ?
Hi, We will be posting second part next week. Do subscribe to our channel and stay tuned!👍
Hi @KodeKloud and all....
I want your opinion about logging solution. Should we consider Grafana+Loki over ELK stack.
I am getting huge cost for the monitoring and logging with AWS. Want to suggest particularly logging solution to my organisation.
If you can provide your inputs on this please.
Thanks in advance.
The choice between Grafana + Loki and the ELK Stack should be based on your specific logging requirements, operational capabilities, and preferences. Each toolset has its strengths and is suited to different scenarios
Grafana + Loki: Good for simpler operations, integrates well with Grafana and Prometheus, more cost-effective, and easier to use. Ideal if you already operate within this ecosystem and want streamlined log management.
ELK Stack: Offers powerful search capabilities and is well-suited for complex log analysis. Requires more resources and can be more expensive to run, but it's the go-to choice if full-text search and extensive data processing are necessary.
How to get syslog into loki?
can we take logs from S3, if yes please do a tutorial or reply a article on it
Hey, please refer to community.grafana.com/t/it-is-possible-to-read-aws-s3-logs-by-loki/55579
Hi I have one scenario working on im using loki and grafana agent here i just want to monitor log files under a directory, in this case the directory has more than 1000 folders and inside that folders more than 10000 log files are there and here the issue is folders are getting created everyday so I have used * for folders and * for log files also but I reaaly don't need older folders the folders are in YYYYMMDD format is there anyway to avoid these older folders being added because it cause an error while starting
I have a issue in this scenario grafana not show all pod log, plz... Me
@@Shiv_30 both are different bro anyway I have done this using scripts
Great video!!! is there any way to store the logs in S3 bucket instead of local file system
I think you need to rewatch the video
@@therealslimaddy Yes I saw that part I tried it but I am unable to ship logs to S3 bucket
Hi, thanks for watching our vieeo. You can please refer to bluelight.co/blog/how-to-install-grafana-loki
any chace of having a full loki course?
Hi, The second part is releasing next week. Do subscribe to our channel and stay tuned for future updates.
What are the specs for the servers? Ram, cpu etc?
Hey, please refer to community.grafana.com/t/hardware-requirements-for-running-grafana-loki/54790/3
How to get response time and alerting to response time of the apps
To monitor response time and set up alerts for applications with Grafana Loki, first configure your application logs to include response time data. Use Loki to ingest these logs and create queries to extract response time information. Set up alerting rules in Grafana based on these queries, defining conditions for when response times exceed specific thresholds. Configure notification channels to receive alerts, whether through email, Slack, or other platforms. Regularly test and adjust your alerting thresholds to ensure you're notified when response times become critical. This approach allows for proactive response to performance issues and helps ensure your applications meet their service level objectives.
how to configuration loki delete log automaticaly ?
Hi, please refer to grafana.com/docs/loki/latest/operations/storage/logs-deletion/#:~:text=Configuration%20Enable%20log%20entry%20deletion,very%20careful%20when%20enabling%20retention
How to store Grafana Loki logs in Azure Blob Storage
"
To store Grafana Loki logs in Azure Blob Storage, first create an Azure Blob Storage account and configure the necessary access permissions. Then, install and configure Loki, specifying Azure Blob Storage as the backend in your Loki configuration file. Make sure to provide the Azure Blob Storage URL, container name, and the appropriate credentials. Restart the Loki service to apply the changes and verify that logs are now being stored in Azure Blob Storage. Remember to consult documentation for both Loki and Azure for specific configuration details to suit your environment and setup. You can refer more to grafana.com/docs/loki/latest/storage/
How did you switch to different nodes in one machine? This thing should also be cleared.
We utilized Vagrant to create virtual machines specifically for demonstration purposes. To access these VMs, we executed the vagrant ssh command. For convenience, we opened multiple terminal windows, each running a separate vagrant ssh session, allowing us to switch between different VM environments as required.
Is grafana Loki open source?
Yes Grafana has both open source and paid versions
Yes, Loki is an open-source log aggregation tool inspired by Prometheus architecture.
@@KodeKloud thanks a lot for your response and the video was just amazing and it helps me a lot
Low key, really cool!
You see what I did there! HA HA! Dad Zoomer 👴 joke!
Hope you enjoyed our video! We have a ton more videos like this on our channel. We hope you will join our community!
நீங்கள் தமிழ் பேசுகிறீர்களா தம்பி)?
Thanks for watching our video. Cheers!
Guide from web is disgusting. Thanks fro your guide!
Glad it was helpful!
Is there an easy way to add host name label in the promtail. Can you set a dynamic label? Something like hostname:{HOST}. Where HOST is an environment variable. When looking at logs in Grafana you would want to know the host they came from.
Yes, in Promtail, you can use dynamic labels like hostname:${HOST} in the configuration to include the host's name in logs. This helps identify log sources when viewing them in Grafana.
./loki-linux-amd64 -config.file=loki-local-config.yaml I can only access loki from Grafana when this command is entered.
when the command is killed
Unable to fetch labels from Loki (Failed to call resource), please check the server logs for more details
I get an error.
Is this what it should be or am I doing something wrong?