Decision Tree Solved | Id3 Algorithm (concept and numerical) | Machine Learning (2019)

Decision Tree is a supervised learning method used for classification and regression. It is a tree which helps us by assisting us in decision-making!
Decision tree builds classification or regression models in the form of a tree structure. It breaks down a data set into smaller and smaller subsets and simultaneously decision tree is incrementally developed. The final tree is a tree with decision nodes and leaf nodes. A decision node has two or more branches. Leaf node represents a classification or decision. We cannot do more split on leaf nodes.
The topmost decision node in a tree which corresponds to the best predictor called root node. Decision trees can handle both categorical and numerical data.
#codewrestling #decisiontree #machinelearning #id3
Common terms used with Decision trees:
Root Node: It represents entire population or sample and this further gets divided into two or more homogeneous sets.
Splitting: It is a process of dividing a node into two or more sub-nodes.
Decision Node: When a sub-node splits into further sub-nodes, then it is called decision node.
Leaf/ Terminal Node: Nodes do not split is called Leaf or Terminal node.
Pruning: When we remove sub-nodes of a decision node, this process is called pruning. You can say opposite process of splitting.
Branch / Sub-Tree: A sub section of entire tree is called branch or sub-tree.
Parent and Child Node: A node, which is divided into sub-nodes is called parent node of sub-nodes whereas sub-nodes are the child of parent node.
How does Decision Tree works ?
Decision tree is a type of supervised learning algorithm (having a pre-defined target variable) that is mostly used in classification problems. It works for both categorical and continuous input and output variables. In this technique, we split the population or sample into two or more homogeneous sets (or sub-populations) based on most significant splitter / differentiator in input variables.
Advantages of Decision Tree:
1. Easy to Understand: Decision tree output is very easy to understand even for people from non-analytical background. It does not require any statistical knowledge to read and interpret them. Its graphical representation is very intuitive and users can easily relate their hypothesis.
2. Useful in Data exploration: Decision tree is one of the fastest way to identify most significant variables and relation between two or more variables. With the help of decision trees, we can create new variables / features that has better power to predict target variable. It can also be used in data exploration stage. For e.g., we are working on a problem where we have information available in hundreds of variables, there decision tree will help to identify most significant variable.
3 Decision trees implicitly perform variable screening or feature selection.
4. Decision trees require relatively little effort from users for data preparation.
5. Less data cleaning required: It requires less data cleaning compared to some other modeling techniques. It is not influenced by outliers and missing values to a fair degree.
6. Data type is not a constraint: It can handle both numerical and categorical variables. Can also handle multi-output problems.
ID3 Algorithm
Key Factors:
Entropy- It is the measure of randomness or ‘impurity’ in the dataset.
Information Gain: It is the measure of decrease in entropy after the dataset is split.
Visit our website for more Machine Learning and Artificial Intelligence blogs
www.codewrestl...
Ask me A Question:
[e-mail]
Music: www.bensound.com
For Decision Trees slides refer:
github.com/cod...
Join Us to Telegram for Free Placement and Coding Challenge Resources including Machine Learning ~ @Code Wrestling
t.me/codewrest...

[@me-hn4bs]

i have to thank indians for tutorials you are amazing guys

[@JanithGamageVEVO]

You are wayyy better with teaching than most of the so-called senior lecturers at my University.

[@CodeWrestling]

Glad it helped!
We also help in implementing projects. Reach us at [e-mail]

[@poojah4127]

The best explanation for id3 algorithm.. Well done

[@kartheekeswar8231]

he is only one who explained best and in a easy way to understand id3 in TH-cam.

[@nikhilnarayane680]

Very well explained. Most people leave it after finding the root node. Thanks for showing the entire calculation

[@anirudhbharadwaj7909]

Excellent! Your emphasis on certain points made it even more clearer to understand. Keep up the good work.

[@MohamedHassan-tk5bq]

Thank you very much; you explained things more clearly and slowly than my prof.

[@surajkulkarni9394]

I had gone through the same example in many courses, everyone explained how to select root node but after that nobody explained further splitting till leaf node. Thank you....

[@CodeWrestling]

Glad it helped!
We also help in implementing projects. Reach us at [e-mail]

[@keerthithejas.c.9098]

Thank you for this video. This is by far the best explanation of the ID3 algorithm out there. I have one question though if the Information gains of two attributes are the same then which one do we use for further splitting?

[@trojanhorse2003]

College is fucked up we have computers for doing all this stuff still they want us to do it manually using pen and paper for just a few marks

[@admail2020]

I feel that you are the one who explained to us this in Manipal course in summer 2020? Excellent. thank you

[@CodeWrestling]

We wish!!

[@sharmass3756]

I have exam tomorrow thanks for saving super explanation my teacher took 5 hours but i didn't understand u explained in 20 mins u are awesome man!!! keep going

[@CodeWrestling]

Glad it helped!
We also help in implementing projects. Reach us at [e-mail]

[@nige3218]

Wow, thank you, from a zero Math and AI background.
Looking forward to learning more

[@Virus-ke8xj]

WOW
WOW
Thanks a lot!!!!!!
Our lecturer took 3 days to explain this, thanks for making this so easy

[@CodeWrestling]

Glad it helped!

[@imaadullahafridi1928]

Thank you very much for such an easy explanation, you explain very well. Please make more and more videos like this on machine learning. I like your way teaching and concept delivery..
Thanks a lot
Imaad Ullah

[@tanishktripathi8773]

This is one hell of an algorithm, simple but time consuming

[@theprathmeshagashe]

Hard

[@jihonoh5212]

The algorithm itself is simple, the time consuming part will be done by a computer iteratively

[@ncsctsr]

FANTASTIC EXPLANATION.ANYBODY CAN UNDERSTAND.THANKS FOR THE VIDEO.PLEASE DO A RF ALGORITHM WITH NUMERICAL EXAMPLE LIKE ID3.

[@sonakshigoswami5622]

Thank you so much. Not one video explained how to calculate the gains after finding the first node.
RECOMMEND IT 10/10

[@CodeWrestling]

Glad it was helpful!

[@vineethpai4486]

You're a life saver❤️ Keep doing such contents. Thanks a lot. Waiting for rest of ML lab Algorithms.😊

[@iAndrewMontanai]

Preparing to the exam, your videos are the best. Also liked Candidate Elimination and Find-S explaination. Gotta check other videos

[@tauhait]

The best explanation on the whole internet!! Thank you,

[@midhilak8422]

Easily understood.👍 Thank you.
Please upload C4.5 algorithm also.

[@foodiez0217]

Explained well but music during the video is bad option. Some may loose concerntration while listening to the video.

[@muhammadhamzanabeel3440]

Concept clear sir.. your teaching method is very impressive. Thanks sir...

[@whiitehead]

To calculate entropy, just paste this in your browser console:
const entropy = (...numbers) => numbers.map((number) => ((number, sum) => -number / sum * Math.log2(number / sum))(number, numbers.reduce((prev, cur) => prev + cur))).reduce((prev, cur) => prev + cur)
after pasting it, you can calculate entropy of outlook with:
entropy(5, 4, 5)

[@vamsikrishnachiguluri8510]

The best and clear cut explanation that I have ever seen

[@mehmetkazanc5855]

Clear. Simply explained. Great Job. Thanks

[@furqanhaider1154]

Shukria ...
buht shukria ...
buht acha or easy explain kia Dear Borther ..
Allah apko khush rakhy ...

[@132132465465]

very clear!! pls continue doing this. you are so good

[@CodeWrestling]

Thank you 😊 soon we will update stay tuned with us.

[@vidhipunjabi3637]

Why is the "temperature" attribute not part of the final decision tree?

[@kregg34]

attributes can be irrelevant to the decision and hence don’t show up

[@naeemaanwer3619]

Superb explanation of calculating entropy and information gain. Thanks a bunch for making these videos.

[@CodeWrestling]

Thanks 😊 we will come up with more videos soon stay tuned with us...

[@swapnilagashe4715]

thanks for the detailed explanation, request you to make a similar video for Random forests covering all the maths, I haven't been able to find a good resource online for this

[@CodeWrestling]

Great suggestion!

[@hasibahammad8508]

It's indeed a great video on ID3 algorithm. Thanks @Code Wrestling.

[@CodeWrestling]

Thanks a lot!

[@Uma7473]

well said. can u clear this doubt... 1. if parent entropy is less than child entropy can we split the child node (can child node become subtree)?
2. if one branch is pure set(4 yes/ 0 no) and another branch
impure(3 yes / 3 no). what should we do now. can we split the impure node(3 yes/3no) when the parent entropy is less than child entropy?

[@omrk5622]

The best explanation of ID3!

[@roopagaur8834]

Very single video in ML cleany explained. Thanks a lot....!!!

[@CodeWrestling]

Thank you!! #CodeWrestling

[@dingshuoyang2421]

awesome video, would be better if there is no BGM when you r talking

[@rkfm2012]

Very clearly explained -- excellent -- more videos from you on machine learning please

[@CodeWrestling]

The videos are on the way.. Stay Tuned #codewrestling

[@kamleshprajapati6621]

Awesome lecture on ID3 numerical..
Thank You 🎉💝🎊

[@rajeshdevadiga2539]

awesome explaination sir,now I understood this algorithm.

[@sudhadevi3211]

Thank you so much for explaining ID3...my doubts are cleared now.

[@rohiniam4650]

Thank u so much sir and id3 concept along with problem is clearly explained

[@CodeWrestling]

Thanks for commenting and appreciating. :)

[@hedejan5000]

Indians are very smart teachers :O

[@Mohitkumar-rb5zr]

@Code_Wrestling where is the video for python implementation?

[@jaelinmccreary9217]

This helped me so incredibly much. Thank you!!

[@anilj5875]

Awesome tutorial👌...but I have one small suggestion....background music is distracting sometimes....especially when you listen to this tutorial with headphones.

[@bhumikab6475]

very good explaination sir..understood it really well

[@nagesh4utube]

Great explanation, understood the concept in a single go. Can you share the name of the music used in the background. Its stuck in my head and want to download it.

[@shibuvm2127]

The best practical explanation

[@premstein16]

Good explanation, thanks for taking patience to explain the entire information gain step :)

[@CodeWrestling]

Thank You!!

[@sunilkumarsk7670]

U taught better than our lecture😊

[@iaashu98]

*taught
Now I know the reason.

[@CodeWrestling]

:D

[@sunilkumarsk7670]

@@iaashu98 Ty for crcting

[@scaratlas3347]

I'm not getting the same entropy values through the calculator

[@naveenkrishna7742]

I don't get it for total 14 attributes outlooks in 3 groups and you got each 5 attributes

[@keremylmaz9595]

Why we arent checking if outlook is sunny,humity is high and we arent checking the temprature or windy, arent there important. How can we select the second branch

[@apoorvasavitha5512]

All the video u upload are very useful n helpful, upload all algorithms in machine learning

[@anudeepk7390]

I understood this algorithm thanks to you

[@sreevanijalagari3863]

Crystal clear

[@yaminipeddireddi8969]

great explanation

[@saibazriyazahmedmulla3701]

Bhai excellent explanation thanks a lot bro

[@higiniofuentes2551]

Best presentation ever for ID3 decision tree definition and calculation! Thank you
I found a difference with the Entropy formula, the theory is : ...log2(1/p) but you have ...log2(p), there are any reason? Thanks a million!!

[@CodeWrestling]

Thank You, and wrt to that log thing.. see the -ve sign.. if you will do reciprocal then a -ve sign will come forward according to log property

[@vib2810]

if the final classification had 3 types(here we have only 2 +ve and -ve) then if would have been log3() as entropy should be 1 if data is equally distributed

[@sidharths9416]

Thanks a ton bro. Clearly understood in depth. Excellent

[@mushfc2343]

thannnnnnnnkkkkk you brother that was very useful for me , and i will subscribe

[@xDDeeDee]

Beautifully explained, thank you! :)

[@1UniverseGames]

Why no temperature in the final decision tree? Any idea

[@abilash305]

What about the attributes Humidity and Temperature? Are they neglected.?

[@rohitchitte5614]

You deserve more subscribers

[@CodeWrestling]

Thanks, please share our videos.

[@masakkali9996]

If equal positive examples and negative example then entropy is 1 right? At 2:14...

[@ssuriset]

Exactly what I needed

[@divyarao4146]

At 15:55 , while calculating entropy , shouldnt it be 0 as n=p?

[@mohammedzia1015]

Actually, Entropy is 'ONE' when we have an equal number of Positive and Negative examples. You said Entropy is 'ZERO'.

[@shankrukulkarni3234]

Thank you bro for this Crystal clear explanation..

[@nitinshiv]

best explanation

[@Areeva2407]

Very Clear, Very systematic

[@CodeWrestling]

Glad it was helpful!

[@NeverStoppedSinging]

Excellent video ! Thank you!

[@zouhairghazi3541]

What to do if 2 average information entropy I for 2 attributes are equal and maximal ? Which one is root ?

[@jav4725]

very nice explanation.please do all machine learning topics.BTW from which book u have taken the dataset?

[@bhavikdudhrejiya852]

Best video for learning decision tree superb

[@matheusfelix5742]

Thank you so much the lecture. Great explanation.

[@ashwinchandore7676]

THank you for this video.very good explaination

[@CodeWrestling]

Glad it was helpful!

[@kanhataak1269]

very very good explain, Thanks a lot

[@kaydenpham4826]

Good explanation. However, it has too much Ads, which is so annoying

[@keshawsingh7899]

Great Video Easily understood... Could you please let me understand how you get E(Temperature=cool) = 0.811 I am getting Entropy E(Temperature=cool) = -3/4 log2 3/4 - 1/4 log2 1/4 = 2.311

[@Muddassir_Ahmed]

what should we do if gain of two features is same ? we can make root any one of them?

[@Roger14216]

Marvelous explanation 👌

[@mohammedzain4199]

Bro you explained it very good but you used Formula wrong for I(Attribute)
When we want to find attribute for complete set then we will use I(attribute) = ((pi+ni)/(p+n))*entropy (s) - ((pi+ni)/(p+n))*entropy (s)

[@CodeWrestling]

Could you please cross-verify once?

[@xiangxu8677]

@@CodeWrestling hunch.net/~coms-4771/quinlan.pdf

[@sarthakvarshney11]

Thank you

[@Ghostalking]

Life saver, thank you man!

[@shivlal121]

Very well explained

[@NitinPasumarthy]

Thanks for making this. What about numerical columns? How to create a decision boundary to compute the gain then?

[@Linaiz]

Amazing, this helped me so much!!!

[@CodeWrestling]

I'm so glad!

[@Wasif_md]

Thanks and Pls upload more videos on ML

[@exoticme4760]

the tree we make is a binary tree. Over cast has three values so three children but in code we always set left and right child , how to solve this. pls help

[@sankethkini6192]

nice and clear explanation.....want a video on instances for using different classifiers

[@leakeyinbrizzy-theitguy5947]

Hi, What is the next video please? Is it Id3 implementation using Python like you said in the video? Thanks.

[@CodeWrestling]

Yes, soon

[@xiangxu8677]

I think the formula is different than original paper, This is a great step by step walkthrough, thanks a lot, the end result of the calculation will be the same, but your Entropy should be the information, and your information should be the Entropy, I believe this is what original paper says hunch.net/~coms-4771/quinlan.pdf

[@DeathRacerSH97]

Great video learned a lot, the way i do it there can be attributes that have no positive or negative examples. in this case a 0 would appear in a log2(). What am i missing?

[@abrahamgk9707]

Please do more videos on machine learning

[@sidharthrunwal6833]

great video! thanks

[@farmeenfathima5627]

great explanation!!

[@CodeTechGyan]

Please upload the video on Ann