I would have loved to see your approach to logging from multiple processes in the same files. For example logging from a FastAPI application that uses uvicorn or gunicorn with multiple workers. Perhaps in a future video. Looking forward to that :) I really love watching your videos. They're full of useful information.
Or from multiple files/modules. I've written a function that sets up logging and call that one in every file, but have no idea if that's the way to go.
@@thomasjohanns7661 You typically only run the logger in your main function. Then in the config you can set up how it should behave for different python modules (including any imported modules. say if you want to silence, or actually increase, logging for anything in particular).
@@_DRMR_ While this is a common practice, it does not always get implemented this way. For example, if I use a custom library that might have its own logging module. Also, multiple processes could imply two or more handlers on the same logger.
We use initialized loggers. Do not forget to check official python build in: QueueHandlera and QueueListener for separation of handler (OUT end of queue) from logger(INPUT side pf queue)
You are doing great videos, but IMHO this was a bit too basic, would love to see an even more deep dive into this module. Like setting it up for a bigger project, best practices, etc. Anyway, thanks a lot and have a nice weekend! 🙂
I think its good he started off basic so those of us who dont get a chance to use this stuff can have a basic setup. I hope he makes an advanced one too!!!
I am also looking for setting it up for bigger practices and best practices. i.e. I also would like to know how you can pipeline the logs to may be ELK or anything where it can be consumable
+ 100 on this idea. Most logging ends up being a overflowed garbage can of rotten log salad and parsing it to find confirmation of correct application, business logic performance, and the mixed bag of errors is just difficult to retrofit after integrated into production. Once it rots in production awhile, logs are definitely an 8th "olfactory offense".
I see that a lot of ppl here are asking for a more complex level of logging explanation, but let me tell you that as being totally new to logging, this video was perfect! straight to the point and very very useful for we ppl that are looking for an starting point. Thanks a lot.
As it is named deep-dive I was a bit hoping the focus of the logging levels and what are the daily praxis where to log, and how to handle the multiple levels in the code. This is something after years of developing software I still struggle with. Sometimes it feels like every second line is a log, and sometimes there are 3 log lines in the full project (hyperbolic). It never found a good regular way / a principle to follow ... :) Whatever! :) Keep going, I learned a lot from you the last year I have now in my daily praxis. It helped me a lot :)
I've found talks by Kevlin Henney incredibly insightful and interesting. Here is a part from a talk (with a clickbaity title) where he addresses logging: th-cam.com/video/FyCYva9DhsI/w-d-xo.html. No concrete advice, but hope it's helpful nonetheless. My personal idea of logging is pretty minimalistic: YAGNI. I make extensive use of logging in the debugging phase, as regular debuggers are often of little help with kinds of software I'm dealing with. But after I have figured out some part of the code and it is covered with tests, logging goes out of the window, unless I have very specific scenarios in mind where I would need it for a post-mortem. For example, if I need to log user actions, I have a set of requirements for that. Otherwise, I do not need to log user actions.
I’m a terrible coder, but only do it for personal projects, and I never thought about using logging to replace the print statements I use when developing, this would save me having to find all the random print statements all over my code 😊
Great content as always Arjan! I think it would be also a good idea to show ways of not overly littering your code with logger calls (ie. using decorators, as in one of your recent videos, or perhaps using other methods) and perhaps demonstrating a real-world example of crucial logging within an app. All the best!
That's very difficult to explain and is up to the project. It's almost artistic. The best way I can describe it is log anywhere you think will help you understand what the software did without tracing the code. You want to be able to diagnose what a problem is almost from the logs alone. That expands to log any information you need to diagnose. Like user names or values that were switched on or errors encountered
You're videos are absolutely brilliant but what I really love is the code being available. It saves me a lot of time when I come back to one of your videos I've already watched a couple times.
Great video as always! Interesting to see use of rotating logs as well as multiple package log implementation. I know I ran into issues with those when I first started writing in python.
I used to work in tech risk for a FI. Regulators require all logs to be maintained and those logs once collected gets streamed to some security vendor to do help look for security issues at runtime. They ingest records at around 1K per second. That exposure changed my perception on logging
I feel as though you just glossed over some of the basics and added the SysLog as a bonus. What about getLogger(__name__) combined with propagation? What about different handlers at different levels? What about custom formatters? What about using decorators to do your logging for you? If nothing else, there are some ideas for a future show :)
Again an excellent video ! I'm actually pointing all my new devs to some of your videos so that I'm sure they pick up good practices (even old ones like me). I didn't know papertrail, and I'm now starting using it. it's perfect, though I'm a bit worried about anybody being able to send logs to my account (no real auth)
another important use is audit logs... we have some critical operations that the customers have to be able to see when they ran, how many times, with what parameters etc
I love your videos about theory and architecture....once you have been coding for a while and you are no longer a begginer this kind knowledge feels really escential.
I've used grafana + loki for very long time. Its very efficient with little to no resources. It can be setup in multiple ways and there is also a package for python called python-logging-loki. Loki has a really nice and powerful logql language with which you can make awesome grafana dashboards.
I would agree with other commenters, it's nice to get a basic intro to logging, but I'd really like to know WHAT I should be logging. In theory I could put a debug log for every line of code I write stating would should happen, but I imagine this is overkill. Or is it? I don't know where to draw the line, it would be good to get some examples of what kinds of things you log, and what level is appropriate for them. But good video, and I'm looking forward to a part 2 😉
I really appreciate your efforts in making this video and it was enjoyable to watch. Informationally however it doesn't quite live up to the expectation set by the title...
Thank you for the video. I didn't know SysLogHandler and Papertrail before. In our company, we are using Graylog instead of Papertrail. Graylog isn't only for Python apps but also for other apps creaded via Java etc. Graylog has a REST API Server. Thus, we are accessing logs in Graylog via requests library in Python. Then, we perform Fraud Detection based on logs data. There are also some libraries like loguru and structlog for logging in Python.
We have struggled with logging quite a bit in our Company and I guess the best solution if to use a central logging config object (dict or even better yaml file). There u set the Formatter, Handler and if you need to set custom attributes, you can do that pretty nice by using a filter.
Yesss. Log as JSON, ingest those logs into some central service (you could use Filebeat for ingesting the log files and use Elastic as search engine with Kibana for dashboarding and searching through your centralized logs). You can also use `jq` (a terminal application) to sort, filter, format your files, etc. It's sooooo nice.
@@NostraDavid2 Yep, we also use JSON format for the logs and a central instance where u can see all logs. This is luckily done automatically in our Kubernetes Cluster
Thanks Arjan, good start on logging. I recently struggled a lot to configure getLogger to get logging object from other module. Python documentation is not that clear on this. I even tried to use thirdparty pkg like logguru. Try doing a video on configuring multi file/module logging and best practices
I like you content very much. I'd like to know when to use logging & when to use print statement. Currently, I tend to checking "logging" via print statement.
Great video logs simple and insight full. But this content is available like everywhere, can you make a video on logging with different modules in the project or when using a design patter in projects and we have tons of files and folders in our project.
I enjoyed the video. I am currently working on doing a presentation on logging. During that process, I spent a lot of time understanding handlers and formaters. I become most unclear when looking at the best way to implement logging in a package compared to how to implement it in an application. Do you have the link to the second video you mentioned on logging?
Thanks for the video. Looks like papertrail is now solarwinds, any idea how to do the port stuff with the new set up? Would be great if you have any suggestions
Hi Arjan, thanks for this video! I was hoping to learn how I should set up logging for a package, so that people that might install my package via pip get access to the logs when executing the code if they wish so, but maybe not automatically. Do you think you could make a second video about more advanced nuances about logging? Also I saw another package (requests) set a NullHandler in __init.py__ and I was wondering what the reason behind that is.
Some tools (which can be a bit overkill) like splunk and elastic stack can really allow you to search and filter log files with ease. But this however is coming from a digital forensics background.
IMHO the logging module seems really over-complicated and easy to get lost in (handlers FTW). But now that it’s in the std lib, nothing more Pythonic gets much traction. See at all the comments asking for complex configuration examples, across multiple files. Video is a good intro to its existence but there’s only so much that can be conveyed this way. And in blog posts as well, most of them also barely scratch the surface of Python logging.
I wonder how often do large companies use an in-house logging solution. I’d imagine that would prevent some of those data leaks and would be perfectly feasible to implement given their size.
Good introductory video about logging in general. However not a deep dive on Python's logging package, as announced at the outset. In fact there is surprisingly little about Python in this video.
Im working in how to implemet logs with fastapi and just I realised logs are printed in stderr by default wich will be a problem if a cicd pipeline relies on stderr in order to continue the right flow. I hope you will explain it here :(
Hi, I am using this functionallity, but module "requests" messes up my log, as it also throws its requests, and for example I see updates of virus signatures and configuration downloads from firefox, is there any way to prevent it?, I could upscale my messages as warning, as it seems it only happens in the INFO configuration, but there has to be another way. Thanks for your great videos!
Nowadays one of the main reasons for a logging server is separation of concerns in the light of information security. If you get attacked, you may want to have a log server separated from your operational data proccessing - with no possibility to remotely delete your logs. Forensics will be very thankful.
I tend to do this because it increases the consistency of the code I write, especially as this also translates well to concurrent code. In that case, you would simply change the main function to async, and use asyncio.run under the if statement.
All your logs are in one place, so you can more quickly find problems, better log visualization features such as colored logging and various types of charts, better log search, better export and backup capabilities, better automation features. Just to name a few things off the top of my head.
Thanks! I am trying to force people to use logging framework on a company internal project, it packed full of prints and I am so sick of getting screenshots of a command prompt.
I get a bit depressed when I watch videos where I already knew everything that was explained. It's nice to watch it accurately and well explained but like following a football team you're hoping for a goal, that piece of knowledge you didn't have, but that never comes. After a while you reach the end with this programming thing and you learn that it was the learning itself you loved, not the programming.
👷 Join the FREE Code Diagnosis Workshop to help you review code more effectively using my 3-Factor Diagnosis Framework: www.arjancodes.com/diagnosis
I would have loved to see your approach to logging from multiple processes in the same files. For example logging from a FastAPI application that uses uvicorn or gunicorn with multiple workers. Perhaps in a future video. Looking forward to that :)
I really love watching your videos. They're full of useful information.
+50
Or from multiple files/modules. I've written a function that sets up logging and call that one in every file, but have no idea if that's the way to go.
@@thomasjohanns7661 You typically only run the logger in your main function. Then in the config you can set up how it should behave for different python modules (including any imported modules. say if you want to silence, or actually increase, logging for anything in particular).
@@_DRMR_ While this is a common practice, it does not always get implemented this way. For example, if I use a custom library that might have its own logging module. Also, multiple processes could imply two or more handlers on the same logger.
We use initialized loggers. Do not forget to check official python build in: QueueHandlera and QueueListener for separation of handler (OUT end of queue) from logger(INPUT side pf queue)
You are doing great videos, but IMHO this was a bit too basic, would love to see an even more deep dive into this module. Like setting it up for a bigger project, best practices, etc. Anyway, thanks a lot and have a nice weekend! 🙂
I think its good he started off basic so those of us who dont get a chance to use this stuff can have a basic setup. I hope he makes an advanced one too!!!
I am also looking for setting it up for bigger practices and best practices. i.e. I also would like to know how you can pipeline the logs to may be ELK or anything where it can be consumable
You gotta start somewhere. I loved it!
+ 100 on this idea. Most logging ends up being a overflowed garbage can of rotten log salad and parsing it to find confirmation of correct application, business logic performance, and the mixed bag of errors is just difficult to retrofit after integrated into production. Once it rots in production awhile, logs are definitely an 8th "olfactory offense".
@@OhsoLosoo Title should NOT be misleading. This is amateur/beginner logging, pretty far from Pro.
Fantastic as always, would love a deep dive on the structlog library for your next video! really takes logging to the next level
Great suggestion!
I see that a lot of ppl here are asking for a more complex level of logging explanation, but let me tell you that as being totally new to logging, this video was perfect! straight to the point and very very useful for we ppl that are looking for an starting point. Thanks a lot.
Logging throughout a module is way more complicated than you’ve shown though. Can you show how you set that up?
Was about to type this
100 % agreed
i just put "logger = logging.getLogger()" at the top of each file and call logger as usual. Works great
@@dirtdart81 but then you’re always grabbing the root logger, which is bad practice as far as I can tell
As it is named deep-dive I was a bit hoping the focus of the logging levels and what are the daily praxis where to log, and how to handle the multiple levels in the code. This is something after years of developing software I still struggle with. Sometimes it feels like every second line is a log, and sometimes there are 3 log lines in the full project (hyperbolic). It never found a good regular way / a principle to follow ... :)
Whatever! :) Keep going, I learned a lot from you the last year I have now in my daily praxis. It helped me a lot :)
I've found talks by Kevlin Henney incredibly insightful and interesting. Here is a part from a talk (with a clickbaity title) where he addresses logging: th-cam.com/video/FyCYva9DhsI/w-d-xo.html. No concrete advice, but hope it's helpful nonetheless.
My personal idea of logging is pretty minimalistic: YAGNI. I make extensive use of logging in the debugging phase, as regular debuggers are often of little help with kinds of software I'm dealing with. But after I have figured out some part of the code and it is covered with tests, logging goes out of the window, unless I have very specific scenarios in mind where I would need it for a post-mortem. For example, if I need to log user actions, I have a set of requirements for that. Otherwise, I do not need to log user actions.
*Thanks Arjan!* That is super useful 👍
Glad it was helpful!
I’m a terrible coder, but only do it for personal projects, and I never thought about using logging to replace the print statements I use when developing, this would save me having to find all the random print statements all over my code 😊
Great content as always Arjan! I think it would be also a good idea to show ways of not overly littering your code with logger calls (ie. using decorators, as in one of your recent videos, or perhaps using other methods) and perhaps demonstrating a real-world example of crucial logging within an app. All the best!
Thanks Arjan! Could you do a follow up on structured logging?
Nice. I would have loved if you had shown what's best practice to actually log and at which places in the code loog statements are supposed to be.
That's very difficult to explain and is up to the project. It's almost artistic. The best way I can describe it is log anywhere you think will help you understand what the software did without tracing the code. You want to be able to diagnose what a problem is almost from the logs alone. That expands to log any information you need to diagnose. Like user names or values that were switched on or errors encountered
You're videos are absolutely brilliant but what I really love is the code being available. It saves me a lot of time when I come back to one of your videos I've already watched a couple times.
Great video as always!
Interesting to see use of rotating logs as well as multiple package log implementation.
I know I ran into issues with those when I first started writing in python.
Would love to see more advanced logging concepts especially unified logging in a big library
I used to work in tech risk for a FI. Regulators require all logs to be maintained and those logs once collected gets streamed to some security vendor to do help look for security issues at runtime. They ingest records at around 1K per second. That exposure changed my perception on logging
I feel as though you just glossed over some of the basics and added the SysLog as a bonus. What about getLogger(__name__) combined with propagation? What about different handlers at different levels? What about custom formatters? What about using decorators to do your logging for you? If nothing else, there are some ideas for a future show :)
That was very informative. I was avoiding logging like a fire but it's not as difficult as it initially seems.
This was a lot more basic than I was expecting. Python's logging has a lot of pitfalls that I would have liked addressed.
Again an excellent video ! I'm actually pointing all my new devs to some of your videos so that I'm sure they pick up good practices (even old ones like me). I didn't know papertrail, and I'm now starting using it. it's perfect, though I'm a bit worried about anybody being able to send logs to my account (no real auth)
another important use is audit logs... we have some critical operations that the customers have to be able to see when they ran, how many times, with what parameters etc
I love your videos about theory and architecture....once you have been coding for a while and you are no longer a begginer this kind knowledge feels really escential.
Arjan, thanks for uploading this :) the date/time formatting of logs was very helpful. I've been using INFO:ROOT this whole time like a n00b.
I also find logging super important! Thanks a lot for this video with lots of good advice :)
I've used grafana + loki for very long time. Its very efficient with little to no resources. It can be setup in multiple ways and there is also a package for python called python-logging-loki. Loki has a really nice and powerful logql language with which you can make awesome grafana dashboards.
Yessss I’ve been waiting for this! Thanks Arjan!
I would agree with other commenters, it's nice to get a basic intro to logging, but I'd really like to know WHAT I should be logging.
In theory I could put a debug log for every line of code I write stating would should happen, but I imagine this is overkill. Or is it? I don't know where to draw the line, it would be good to get some examples of what kinds of things you log, and what level is appropriate for them.
But good video, and I'm looking forward to a part 2 😉
as always, great content Arjan.
Thank you so much!
Thank you for this!
As always, excellent video. Making a simple topic so robust, I want to build some apps just to log now.
Yeahhhh thank you so much!! Really appreciate a logging tutorial!
You're very welcome!
waiting for this!!!! Thanks Arjen!!
I really appreciate your efforts in making this video and it was enjoyable to watch.
Informationally however it doesn't quite live up to the expectation set by the title...
Thank you for the video. I didn't know SysLogHandler and Papertrail before.
In our company, we are using Graylog instead of Papertrail. Graylog isn't only for Python apps but also for other apps creaded via Java etc. Graylog has a REST API Server. Thus, we are accessing logs in Graylog via requests library in Python. Then, we perform Fraud Detection based on logs data.
There are also some libraries like loguru and structlog for logging in Python.
Great Job bro !! Keep it up.. Thanks for this ❤🙏🏻
Thank you for the kind words! Will do.
I am glad to have found this video. Thank you!
Glad it was helpful, William!
Amazing content Arjan, keep it up. Cool intro to Paper Trail too :)
Been playing with log2d that makes logging a bit easier.
Great job! Thank you so much
best for someone who just started to learn logging.
Thanks, Arjan! As always, amazing video!
Glad you liked it!
We have struggled with logging quite a bit in our Company and I guess the best solution if to use a central logging config object (dict or even better yaml file). There u set the Formatter, Handler and if you need to set custom attributes, you can do that pretty nice by using a filter.
Yesss. Log as JSON, ingest those logs into some central service (you could use Filebeat for ingesting the log files and use Elastic as search engine with Kibana for dashboarding and searching through your centralized logs).
You can also use `jq` (a terminal application) to sort, filter, format your files, etc. It's sooooo nice.
I use the structlog lib, which can be a pain to configure, but it's incredibly powerful and flexible.
@@NostraDavid2 Yep, we also use JSON format for the logs and a central instance where u can see all logs. This is luckily done automatically in our Kubernetes Cluster
Thanks Arjan, good start on logging.
I recently struggled a lot to configure getLogger to get logging object from other module. Python documentation is not that clear on this. I even tried to use thirdparty pkg like logguru.
Try doing a video on configuring multi file/module logging and best practices
Always good videos
I like you content very much. I'd like to know when to use logging & when to use print statement. Currently, I tend to checking "logging" via print statement.
Great video logs simple and insight full. But this content is available like everywhere, can you make a video on logging with different modules in the project or when using a design patter in projects and we have tons of files and folders in our project.
DUDE.. I was searching for Logging tips TODAY.... lol
With normal file logging you can search with grep, less and all the other standard tools, what could be easier than that?
I enjoyed the video. I am currently working on doing a presentation on logging. During that process, I spent a lot of time understanding handlers and formaters. I become most unclear when looking at the best way to implement logging in a package compared to how to implement it in an application. Do you have the link to the second video you mentioned on logging?
Thanks for the video. Looks like papertrail is now solarwinds, any idea how to do the port stuff with the new set up? Would be great if you have any suggestions
Hi Arjan, thanks for this video! I was hoping to learn how I should set up logging for a package, so that people that might install my package via pip get access to the logs when executing the code if they wish so, but maybe not automatically. Do you think you could make a second video about more advanced nuances about logging?
Also I saw another package (requests) set a NullHandler in __init.py__ and I was wondering what the reason behind that is.
Super useful, thanks!
Glad it was helpful!
Hi there, from Portugal,
Any recommendation on using python for google cloud function using google cloud logging?
Obrigado(Thanks)
you are best my friend.😍😍😍😍😍😍😍
Some tools (which can be a bit overkill) like splunk and elastic stack can really allow you to search and filter log files with ease. But this however is coming from a digital forensics background.
thanks:)
Thanks 🙏🙏🙏
Logging service, you just blowed up my mind :) I had not idea it existed. As usual thanks for the great content.
Happy to help, Daniel!
IMHO the logging module seems really over-complicated and easy to get lost in (handlers FTW). But now that it’s in the std lib, nothing more Pythonic gets much traction. See at all the comments asking for complex configuration examples, across multiple files. Video is a good intro to its existence but there’s only so much that can be conveyed this way. And in blog posts as well, most of them also barely scratch the surface of Python logging.
I wonder how often do large companies use an in-house logging solution. I’d imagine that would prevent some of those data leaks and would be perfectly feasible to implement given their size.
Good introductory video about logging in general. However not a deep dive on Python's logging package, as announced at the outset. In fact there is surprisingly little about Python in this video.
Im working in how to implemet logs with fastapi and just I realised logs are printed in stderr by default wich will be a problem if a cicd pipeline relies on stderr in order to continue the right flow. I hope you will explain it here :(
Hi, I am using this functionallity, but module "requests" messes up my log, as it also throws its requests, and for example I see updates of virus signatures and configuration downloads from firefox, is there any way to prevent it?, I could upscale my messages as warning, as it seems it only happens in the INFO configuration, but there has to be another way. Thanks for your great videos!
My question is if something goes wrong in production.. how can we back trace the problem from logs?
Nowadays one of the main reasons for a logging server is separation of concerns in the light of information security. If you get attacked, you may want to have a log server separated from your operational data proccessing - with no possibility to remotely delete your logs. Forensics will be very thankful.
how do I log to system.log on macos ventura? none of what i've found on SE or what has been provided by chatGPT works (no errors though)
I used nbdev in VSCode to write python modules. Where does the logging module write it's output when working with notebook cells?
# import reload
from importlib import reload
reload(logging)
logging.basicConfig(format='%(asctime)s %(levelname)s: %(message)s', level=logging.DEBUG, datefmt='%I:%M:%S')
those unequal length cords on your hoodie is driving me crazy
Why did papertrail mark the log message as coming from "logger" when the logger you configured was named 'arjancodes'?
What are the pro's to have an explicit main() function vs having the contents of main() under __name__ == "__main__"?
I tend to do this because it increases the consistency of the code I write, especially as this also translates well to concurrent code. In that case, you would simply change the main function to async, and use asyncio.run under the if statement.
Really important! Why is getLogger etc from the logging package camel case? It hurts my pythonic eyes
Why at the first place you need a separate logging service.??
All your logs are in one place, so you can more quickly find problems, better log visualization features such as colored logging and various types of charts, better log search, better export and backup capabilities, better automation features. Just to name a few things off the top of my head.
Thanks! I am trying to force people to use logging framework on a company internal project, it packed full of prints and I am so sick of getting screenshots of a command prompt.
what is the purpose of the "if __name__ == "__main__":" block
Logging to splunk is useful too.
Can I write some code to like all your videos?
Haha, thank you, your comment made my day ☺️.
👏🏾👏🏾👏🏾
what is the name of the compiler?
I'm liking the glass of water 🤓
Real talk where is that sweater from?
Me, an intellectual: print(x) scattered throughout my spaghetti code
It'd be cool if you showed logging config from different files, like ini and yaml. thanks!
/dev/null as a Service!
I agree that this was too basic. Mentioning Paertrail was very helpful
This dive was into this module was really shallow
nice
How do you configure your emacs this good? Can you make a video about this as well?
Anybody here uses loguru?
👍👍
Logging into a File with a Rotating FileHandler from multiple processes is not so nice tho xD
Logging is hella important. If you think logging and not well organized and built logging systems is not important, you're still very junior.
remove all of this pain with just `pip install loguru` and `from loguru import logger`... yw
I get a bit depressed when I watch videos where I already knew everything that was explained. It's nice to watch it accurately and well explained but like following a football team you're hoping for a goal, that piece of knowledge you didn't have, but that never comes.
After a while you reach the end with this programming thing and you learn that it was the learning itself you loved, not the programming.
nothing compares to print("here")
Yes, logging can do all these things, but tracing and metrics do it better. Logs are good if tracing isn't working.
Log to a database ..sqlite3 baby
It is good，but i choose loguru😂
Big mistakes with logging: Do not over-log information, just what needs to be logged. You kind of said this with the security part, but still.
Title should NOT be misleading. This is amateur/beginner logging, pretty far from Pro.
Auto translation of the video title to german totally butchered it.
Yes, we switched that feature off for more recent videos. It doesn’t work that well.
this was hardly a deep dive. this was VERY surface level