This video is absolutely fantastic! The presenter's clear explanations and engaging visuals made it a pleasure to learn from. I'm grateful for the valuable insights and real-world examples provided. Well done!
Absolutely fantastic video. This is the best information theoretic feature selection explaination that I have come across - so accessible, so well explained. Kudos on a fantastic job.
Excellent lecture Dr. Khushaba! I was searching for a visual explanation of these concepts, now I feel like I actually understand how MI works! Thank you so much, please make more videos like this.
Thanks a lot for the feedback, much appreciated. I have added another one on transfer entropy yesterday, just look for “finding causal relationships” in my channel. It extends further on the concepts presented here to look for causality rather than just mutual information.
Thank you for this great explanation! Helped me a lot. I have one question: In 31:20 you say what is I(X,Y,Z) in the Venn diagram. From what I understood from the example with just two variables, X and Y, the area for I(X,Y) was the envelope of the Venn diagram (at least it was marked like this). Why is it here not the envelope of the three circles? Thank you very much already in advance!
Hello, thanks for your feedback. About your question: for the case of two variables, the exterior envelope of the two circles is H(X,Y) that is the entropy of the two variables. Mutual information, I(X;Y) is given by the overlapping region between the two circles. So you start from entropies, and the joint areas are the portions that one variable knows about another and hence reducing entropy - this is mutual information. The same goes for the case of three variables. I(X;Y;Z) is the joint part between the three circles and the outside envelop for the three circles is the entropy.
So ',' and ';' both represent 'and', however, it is like ';' has higher priority than ','. They have same portion in the Venn diagram if there are only two variables.
Hi, first of all, this is an excellent video! I have a question on page 13. in each square of the 2 dimensional graph, what should be the filled in value? at the left lower square, is the value there equal to the sum of the number of data samples from the first bin of the feature represented by the horizontal axis and the number of data samples from the first bin of the feature represented by vertical axis? Thanks for your clarification!
Thank you for your feedback. The answer is not the sum of the individual values, as you just need to think multidimensional rather than single dimension. The answer for that lower left bin is a count of the number of samples from feature 1 that falls within the range of the first bin while the samples in the second feature fall within the range of the first bin of that feature. In that example it is given as how many samples from feature 1 fall within the range of 0.1 to 0.34 while at the same time the samples from feature2 had values within the range of 1 to 1.59. Think of this as driving on a highway with 4 lanes. How many times did the car on lane 1 drive on a speed of 60 to 70 km for example while, at the same time, the car on lane 2 had a speed between 80 to 90km for example. When you do individual cars you just look at car on lane 1 individually from other cars and count number of times it drove on a specific speed regardless of other cars on other lanes.
Thanks for your feedback. The conditional entropy is defined in the slides using a different formula than the one you expressed for a reason and that is for anyone interested to read and discover more ways of writing the formulas to encourage further learning and development.
You are one of the best data science/ML teachers that I've found on TH-cam and I really hope keep making videos.
Thank you so much for your feedback.
I've watched a tons of vdo about Entropy. This is the best !
Very clear, many thanks!
This video is absolutely fantastic! The presenter's clear explanations and engaging visuals made it a pleasure to learn from. I'm grateful for the valuable insights and real-world examples provided. Well done!
i have been into this topic for a month... and so so happy to find your lecture series !!! Thank you so much. So clear and precise.. Treasure :)
Absolutely fantastic video. This is the best information theoretic feature selection explaination that I have come across - so accessible, so well explained. Kudos on a fantastic job.
Thanks for the feedback, much appreciated.
What an excellent lecture. Its really hard to find such good ones. Hats off for you!
Thank you so much for your feedback, much appreciated.
Excellent lecture Dr. Khushaba! I was searching for a visual explanation of these concepts, now I feel like I actually understand how MI works! Thank you so much, please make more videos like this.
Thank you for the feedback, much appreciated.
This is very useful to me, as I'm planning to leverage this concept in my masters thesis. Thanks again!
Thanks, appreciate the feedback and good luck with your Master study.
Thanks a lot. I was really struggling with the relationship between MI and Entropy, also with the notations, until I came across your video.
Thank you, appreciate the feedback
amaizing amazing amazing woow please make more like this in information theory
Thanks for the feedback
Amazing work, many thanks Dr. Khushaba.
Thanks for the feedback
Thank you!!! This is what I was looking for!!! If you update your microphone it will be ideal.
Thank you, appreciate the feedback.
Simply put..Excellent. It is a kind request to add more of this nature. Regards
Thanks a lot for the feedback, much appreciated. I have added another one on transfer entropy yesterday, just look for “finding causal relationships” in my channel. It extends further on the concepts presented here to look for causality rather than just mutual information.
@@AlaphBeth Thank you so much. The videos and explanations are excellent. Excellent resources for learning. Many thanks. Warm Regards
Thanks a lotttt for this video!!!
I highly appreciate your time and dedication.
Thank you, appreciate your feedback
Wonderful talking! well explained!
Thank you, appreciate the feedback.
Amazing!
Thank you for this great explanation! Helped me a lot. I have one question: In 31:20 you say what is I(X,Y,Z) in the Venn diagram. From what I understood from the example with just two variables, X and Y, the area for I(X,Y) was the envelope of the Venn diagram (at least it was marked like this). Why is it here not the envelope of the three circles?
Thank you very much already in advance!
Hello, thanks for your feedback. About your question: for the case of two variables, the exterior envelope of the two circles is H(X,Y) that is the entropy of the two variables. Mutual information, I(X;Y) is given by the overlapping region between the two circles. So you start from entropies, and the joint areas are the portions that one variable knows about another and hence reducing entropy - this is mutual information.
The same goes for the case of three variables. I(X;Y;Z) is the joint part between the three circles and the outside envelop for the three circles is the entropy.
Thank you, ah yes I mixed up the entropy and the mutual information. It’s clear now, thanks!
So ',' and ';' both represent 'and', however, it is like ';' has higher priority than ','. They have same portion in the Venn diagram if there are only two variables.
Very nice!
Thank you, appreciate the feedback
you are applying Histogram Approach to discrete data or continuous data?
Hi, first of all, this is an excellent video! I have a question on page 13. in each square of the 2 dimensional graph, what should be the filled in value? at the left lower square, is the value there equal to the sum of the number of data samples from the first bin of the feature represented by the horizontal axis and the number of data samples from the first bin of the feature represented by vertical axis? Thanks for your clarification!
Thank you for your feedback. The answer is not the sum of the individual values, as you just need to think multidimensional rather than single dimension. The answer for that lower left bin is a count of the number of samples from feature 1 that falls within the range of the first bin while the samples in the second feature fall within the range of the first bin of that feature. In that example it is given as how many samples from feature 1 fall within the range of 0.1 to 0.34 while at the same time the samples from feature2 had values within the range of 1 to 1.59.
Think of this as driving on a highway with 4 lanes. How many times did the car on lane 1 drive on a speed of 60 to 70 km for example while, at the same time, the car on lane 2 had a speed between 80 to 90km for example. When you do individual cars you just look at car on lane 1 individually from other cars and count number of times it drove on a specific speed regardless of other cars on other lanes.
nice explanation. can you provide us with slides?
Wonderful work, thanks. Can you share the slides for reference?
Thank you, appreciate the feedback. I will post a link to the slides soon.
missing a formlar H(Y|X)=H(X,Y)-H(X)=p(x,y)log(1/p(x,y)) -p(x)log(1/p(x))
Thanks for your feedback. The conditional entropy is defined in the slides using a different formula than the one you expressed for a reason and that is for anyone interested to read and discover more ways of writing the formulas to encourage further learning and development.