Minor thing, but that visualization of verbosity levels at 2:00 is using some backwards logic. Selecting a log level should show entries at that level or _more_ severe, not _less_ severe. What's the point in filtering out the important messages and only showing the noise?
Exactly! There are usually other level as well, like DEBUG and TRACE, which are normally filtered out during production, but are useful to find problems that you can reproduce locally.
Depends what you are looking for and if the logs are correctly categorized to begin with. It's not rare that I need to go through debug level logs ignoring error levels because those errors are either not what I'm looking for, the symptoms of the issue I'm trying to fix or just totally unrelated to what I am currently looking for and thus part of the noise. Which is why being able to select the levels shown is better than having a level cut off. And it looked like showing the selection of levels in the vid.
@@Ragnar_Oock You can always filter out entries that you're not interested in. Also, what you're describing here seems to be valid only if you can reproduce the issue on your local machine. Logs are more important when you run the system on a server (or many servers) or, if the user is running the program on their own machine. Additionally, all frameworks that I've seen work by selecting a minimum level. Even if there are some where you can pick and choose individual ones, they are the exception. Surely, it makes sense to describe best practices that can actually be used.
That was insanely informative, thankyou for the video. We currently use Better Stack for a couple of things at work, you can be sure we're going to add in some of these concepts to our existing setup.
In 3:40 I think one of the most important part is missing. Its the file name and line number. It makes a world of a difference when you have this to information. Debugging becomes super easy.
I would be careful about what information you log out, it is much safer to write logs to a secure location with a unique id logged out to the console. This allows the console to be monitored and alerts that you get from that service doesn’t have user personal information, then you can have a secure system access the information by looking up using the unique id. It is great to have multiple logging outputs: console, text file, database, etc… Each one can be used as triggers for actionable information, like having a specific console error trigger emailing the text file while the database has all the users details stored with different access credentials, and that email could have the link to your secure logging dashboards - which can be made even more secure by only allowing certain networks access to it and requiring a VPN for external access.
8:35 That's not "security by obscurity". In fact you are mixing 2 different concepts here. One is privacy and the other is security. You may be breaching your privacy policy (or in Europe, the law) by sharing personally identifiable data with a 3rd party without the user's consent. Second, "security by obscurity" is when your *insecure* system relies on the attacker not knowing about the vulnerabilities. In your example it does not matter how secure the system is because you are broadcasting your password.
You are allowed to store user data as long as you can motivate it and only keep it as long as you need. So feel free to continue logging the email address that someone is repeatedly failing to log in with, just make sure that you actually use that info for something, have a retention period that makes sense, and write it down in your privacy policy so the users are aware. If you start thinking of it as borrowing the users data and having to explain to them why you're doing it, you'll be on the right path.
just a personal note, 12 Logging BEST Practices in 12 minutes th-cam.com/video/I2mWnh66Bkg/w-d-xo.htmlsi=DtC5IvXBFwEopcM7 - ask yourself - what are your application's main business goals? - what critical operations need monitoring? - what KPIs actually matter? - provide enough context to fix the problem, think of the future self would debug - review what's useful and what's noises - the best logging strategy is not about capturing everything, it's about capturing the right thing - 4 commong level: info, warn, error, fatal - [TIMESTAMP] [LEVEL] log description - info - significant and noteworthy business event (user completed checkout) - warn - abnormal situation that may indicate future problems (payment took longer time) - error - unrecoverable errors that affect a specific operation (fail payment, server crash, failed db connect) - fatal - unrecoverable errors that affect the entire program (system out of memory) - structured logging - request id (for tracing requests across microservices) - user id (for user session context) - system state data (like database or cache status) - full error context (stack traces when relevant) - use logging libraries - log should provide 5w1h even attempt amount - "logs are your system black box recorder, make them detail enough that you can replay any scenario that happened not just know that it actually happened - instead of storing every single logs, stored a representative sample - use canonical log method (a single log entry that tells the whole story, it's like a movie scenery instead of watching all the scenes separately) one log that captures everything important. it's like a TL;DR - what's worst than having no logs, having logs everywhere - put logs in one place centralized instead of scattered it in each folders or services - retention policy - error logs for 90 days for thorough investigation, debug logs for 7 days for immediate troubleshooting needs, security audit log for 365 days for compliance requirement - recent logs always available for quick debugging - older logs are moved to cheaper cold storage - logs deleted when they're no longer needed - logs aren't equal so prioritize important - securing sensitive logs - encryption in transit - protect as they move to storage - encryption at rest - keep them secured when they're stored - access controls - junior only basic logs, senior a bit more sensitive logs, security full access - don't logs sensitive information, encrypt it - "the best sensitive data leak is the one that never happens bcs you never logged it in the first place - performance impact - choose efficient logging libraries - use sampling for high traffic path - log to separate disk - run load tests look out for bottlenecks - log is not for everything, use your logs to debug problems and use your metrics to know when and how often things have a problem - "good logging isn't about logging everything, it's about logging the right things in the right way at the right time" - review and adapt
I feel like it's important to note that while logging is important, one should not use it as a debugging tool for currently worked on code. That should be do with debbugers and breakpoints. Because we all know that "I'll remove the logs later" is a lie and now the file is 25% logs and you can make sense of it.
Logs must have at least: - Timestamp - Severity (info, warning, error, critical) - Message - Full traceback (in case of exceptions, critical errors, etc) They should also contain stamps for start and end of execution clearly
Hi! Good point, the content in this video is supported by a blog post so I have linked that. Can find it here: betterstack.com/community/guides/logging/logging-best-practices/#12-don-t-rely-on-logs-for-monitoring Contains more detail and references 🙂 - James
logging sampling is such a bad idea that it hurts my brain just delete the logs after X amount of time and call it a day. If you need to debug some user ticket and you don't have the logs (or the logs are inconsistent) because of sampling, you'll be fucked. bad idea, don't do that.
IMHO, JSON is a terrible log format. Too verbose. Which means that: a) human readability is hurt b) Being very inefficient in files which are likely to grow very big If you need structured data, CSV is a way better option for this particular use.
I didn't expect to learn so much about logging from this video
And the good part is that he is providing value while also apearing to sell something useful
Minor thing, but that visualization of verbosity levels at 2:00 is using some backwards logic. Selecting a log level should show entries at that level or _more_ severe, not _less_ severe. What's the point in filtering out the important messages and only showing the noise?
Exactly!
There are usually other level as well, like DEBUG and TRACE, which are normally filtered out during production, but are useful to find problems that you can reproduce locally.
Agreed, I originally meant to show log levels being toggled on/off, not log filtering. Thanks for catching the confusion 🙂
- James
Came here to comment the same :)
Depends what you are looking for and if the logs are correctly categorized to begin with. It's not rare that I need to go through debug level logs ignoring error levels because those errors are either not what I'm looking for, the symptoms of the issue I'm trying to fix or just totally unrelated to what I am currently looking for and thus part of the noise. Which is why being able to select the levels shown is better than having a level cut off. And it looked like showing the selection of levels in the vid.
@@Ragnar_Oock You can always filter out entries that you're not interested in.
Also, what you're describing here seems to be valid only if you can reproduce the issue on your local machine. Logs are more important when you run the system on a server (or many servers) or, if the user is running the program on their own machine.
Additionally, all frameworks that I've seen work by selecting a minimum level. Even if there are some where you can pick and choose individual ones, they are the exception. Surely, it makes sense to describe best practices that can actually be used.
That was insanely informative, thankyou for the video. We currently use Better Stack for a couple of things at work, you can be sure we're going to add in some of these concepts to our existing setup.
Great video! Logging is not a popular topic but a really important one!
In 3:40 I think one of the most important part is missing. Its the file name and line number. It makes a world of a difference when you have this to information. Debugging becomes super easy.
100% the easiest (and fastest) subscribe ive done in a long time. absolute gold dust. keep it up mate
I would be careful about what information you log out, it is much safer to write logs to a secure location with a unique id logged out to the console. This allows the console to be monitored and alerts that you get from that service doesn’t have user personal information, then you can have a secure system access the information by looking up using the unique id. It is great to have multiple logging outputs: console, text file, database, etc… Each one can be used as triggers for actionable information, like having a specific console error trigger emailing the text file while the database has all the users details stored with different access credentials, and that email could have the link to your secure logging dashboards - which can be made even more secure by only allowing certain networks access to it and requiring a VPN for external access.
8:35 That's not "security by obscurity".
In fact you are mixing 2 different concepts here. One is privacy and the other is security.
You may be breaching your privacy policy (or in Europe, the law) by sharing personally identifiable data with a 3rd party without the user's consent.
Second, "security by obscurity" is when your *insecure* system relies on the attacker not knowing about the vulnerabilities. In your example it does not matter how secure the system is because you are broadcasting your password.
Great product and UX, great work!
Fantastic video. Thankyou so much
What UI do you use to view the logs and metrics? 7:09
Thats our tool, Better Stack! betterstack.com/
Betterstack Telemetry
Nice overview!
3:36 Just fyi: You should never log user data if you ever plan to offer your app in the EU or anywhere else with basic data protection laws.
You are allowed to store user data as long as you can motivate it and only keep it as long as you need. So feel free to continue logging the email address that someone is repeatedly failing to log in with, just make sure that you actually use that info for something, have a retention period that makes sense, and write it down in your privacy policy so the users are aware.
If you start thinking of it as borrowing the users data and having to explain to them why you're doing it, you'll be on the right path.
I have to say, this is a very good video.
Love your talking speed :)
Hey, can't be the 3:40 log level "error"? Like as it is expected error in the system right?
Well structured content and a lot to learn
I also use the DEBUG Level
Good point, I tried to tackle the common ones but they vary from tool to tool. Debug and Trace great for development and debugging.
A lot of great advice here
3:43 Very important: include the plaintext password for everyone to see...
just a personal note,
12 Logging BEST Practices in 12 minutes th-cam.com/video/I2mWnh66Bkg/w-d-xo.htmlsi=DtC5IvXBFwEopcM7
- ask yourself
- what are your application's main business goals?
- what critical operations need monitoring?
- what KPIs actually matter?
- provide enough context to fix the problem, think of the future self would debug
- review what's useful and what's noises
- the best logging strategy is not about capturing everything, it's about capturing the right thing
- 4 commong level: info, warn, error, fatal
- [TIMESTAMP] [LEVEL] log description
- info - significant and noteworthy business event (user completed checkout)
- warn - abnormal situation that may indicate future problems (payment took longer time)
- error - unrecoverable errors that affect a specific operation (fail payment, server crash, failed db connect)
- fatal - unrecoverable errors that affect the entire program (system out of memory)
- structured logging
- request id (for tracing requests across microservices)
- user id (for user session context)
- system state data (like database or cache status)
- full error context (stack traces when relevant)
- use logging libraries
- log should provide 5w1h even attempt amount
- "logs are your system black box recorder, make them detail enough that you can replay any scenario that happened not just know that it actually happened
- instead of storing every single logs, stored a representative sample
- use canonical log method (a single log entry that tells the whole story, it's like a movie scenery instead of watching all the scenes separately) one log that captures everything important. it's like a TL;DR
- what's worst than having no logs, having logs everywhere
- put logs in one place centralized instead of scattered it in each folders or services
- retention policy
- error logs for 90 days for thorough investigation, debug logs for 7 days for immediate troubleshooting needs, security audit log for 365 days for compliance requirement
- recent logs always available for quick debugging
- older logs are moved to cheaper cold storage
- logs deleted when they're no longer needed
- logs aren't equal so prioritize important
- securing sensitive logs
- encryption in transit - protect as they move to storage
- encryption at rest - keep them secured when they're stored
- access controls - junior only basic logs, senior a bit more sensitive logs, security full access
- don't logs sensitive information, encrypt it
- "the best sensitive data leak is the one that never happens bcs you never logged it in the first place
- performance impact
- choose efficient logging libraries
- use sampling for high traffic path
- log to separate disk
- run load tests look out for bottlenecks
- log is not for everything, use your logs to debug problems and use your metrics to know when and how often things have a problem
- "good logging isn't about logging everything, it's about logging the right things in the right way at the right time"
- review and adapt
Thank you for this. I feel like I just reached a higher level in my craft
Just to add to the “don’t log everything”, be aware of log file sizes! I (unfortunately) have learnt this the hard way with server space filling up 😣🤪
what a great channel !!
I feel like it's important to note that while logging is important, one should not use it as a debugging tool for currently worked on code. That should be do with debbugers and breakpoints.
Because we all know that "I'll remove the logs later" is a lie and now the file is 25% logs and you can make sense of it.
can this be implemented in any type of software or app that also needs network connection?
It is possible to do all of this locally, without an internet connection
Good video! The last thing I'm thinking about is log pushing, should it be synchronous or asynchronous?
It should be always be asynchronous be it FE or BE. Use something like ELK with filebeat for logging and for tracing something like jaeger
I'd push synchronously for security events, and async for everything else
Logs must have at least:
- Timestamp
- Severity (info, warning, error, critical)
- Message
- Full traceback (in case of exceptions, critical errors, etc)
They should also contain stamps for start and end of execution clearly
Completely off topic, but what's the font used in the thumbnail?
Cal Sans! github.com/calcom/font
- James
Could you maybe putvsome references/pointers in thd video description, for easy reference? Like, you mentioned Open Telemetry a few times.
Hi! Good point, the content in this video is supported by a blog post so I have linked that. Can find it here: betterstack.com/community/guides/logging/logging-best-practices/#12-don-t-rely-on-logs-for-monitoring
Contains more detail and references 🙂
- James
Fancy logging over functionality first for me. If the log isn't pretty, the app doesn't get any effort
Thank you
Winston logging in nestjs please
Please, best logging practices in fastify api using pinojs
Great video. Thank you!
i love him, no bullshit, low subs, good f
job,
logging sampling is such a bad idea that it hurts my brain
just delete the logs after X amount of time and call it a day. If you need to debug some user ticket and you don't have the logs (or the logs are inconsistent) because of sampling, you'll be fucked.
bad idea, don't do that.
Op
IMHO, JSON is a terrible log format. Too verbose. Which means that:
a) human readability is hurt
b) Being very inefficient in files which are likely to grow very big
If you need structured data, CSV is a way better option for this particular use.
🫡
This video is all over the place. Feels like a big ad...