Thank you Aakash for giving a raw walk-through. Apart from usual documentation stuff I think the ability to google and find answers for the problems are much more important skills in the area of Data Science apart from ofcourse mathematical understanding. This walkthrough actually gave me more confidence in using pandas without worrying about typical syntax pitfalls
best project tutorial in any youtube channel. Everyone make video and cut those portion when they get stuck. On the other hand you just show the real scenario and show us how to find solution using google as well. we want similar video on data science project. Love from Bangladesh.
Do you know why I loved this video? You kept it raw and real, you are clearly portraying how a data analyst thinks and does the project, which I believe is more important than fancy coding...syntax you can get easily but analytical thinking requires the real efforts
@@shivsharma9153 open notebook through jovian run through colab with help of kaggle datset . I have written some line of code in colab .then i have close all tabs . Now how to go to that file where i have write my code.
I have seen only half the video. Couldn't stop myself from apprecating the good work. Couldn't have been done any better. Way better than the Paid education platforms
Aakash, you are one of the best teachers I have come across. Coming from a hard-core medical background and pivoting into data analytics I came across your panda's courses while preparing for my foundation in python before starting a master's in the US in analytics this Fall. Hands down you have given beginners like me a lot of handholding with your courses and videos!
@@jovianhq I went through many teachers on youtube and data camp: but truth to be told- most are ludicrously formal in their teaching methods and have a slower theoretical pace. Is there any possibility to connect with Aakash to get certain roadmap tips for a beginner who plans to venture into the US Health Business Analytics Domain?
The current dataset is updated. A high percentage of accidents occur between 3 pm to 6 pm (probably people in a hurry to get to home) Next higest percentage is 6 am to 9 am. Over 1100 cities have reported just one accident (need to investigate On Sundays, the peak occurs between 11 am and 6 pm, unlike weekdays
Sir actually New York data is present in the given data set. We get the output as False if we use: 'New York' in df ['City'] And we get the output as True if we use : 'New York' in df ['City'].unique()
Yes Jeet, you are correct. We found it later but didn't update the video to show that this type of error might happen any time during working on a project. Great work on finding it!
New York data is there, manually checked the whole data to find the city because the method you showed at 58 minutes mostly doesn't work because of the spaces, some values contain unnecessary spaces and that creates a problem.
What a great Tutorial! The df is obviously updated and I cannot find the 'New York' value in Cities, BUT there are data in cities_by_accident "cities_by_accident['New York']" and is equal to 7068
Good class of Python pandas, but in R exploratory and statistics analysis are way easier compared to Python. Example: data_frame %>% filter(City == "New York") bam!! dataset filtered. Summarize numeric data => data_frame %>% summary() !! bam!! Done!! in a totally functional way.
Both R and Python are great, you can use either one. Python is gaining more traction because it also has great packages for machine learning & deep learning.
This is a great project and I really enjoyed it. After finishing this video yesterday, I am working on other parameters to expand my analysis. I would love to see more projects from Akash. Thank you.
"New York" in df. City returns False But "New York" in df.City.uniqu() returns True. (Which I have no explanation for) And in fact, there are 4220 accident cases inside the dataset which occurred in New York inside the dataset (The dataset could be updated recently.). I don't know if it has been updated. But since the accident records fall in between the year 2016 and 2020, it would seem weird if new rows get added later on.
You are correct, there was "New York" in the dataset before as well. df.City returns a Series where if you search using the "in" operator, it will search for the indexes and not match with the values. Where as df.City.unique() creates a list and "New York" is searched within that list so you were able to find "New York".
Hi, at around 49:17 when you are checking that weather we have 'New York' data or not so in that when we are checking for the existence with, if 'New York' in df.values - it will return True And If 'New York' in df.City - False Also If 'Dublin' in df.City - False ( and for all the other cities ) So, in my preference we need to use the df.values ( it will check the whole dataset - yes might be time taking and requires unwanted computing processing as well ) Please help us to improve this part Thanks
Yes @Aman, you are correct, New York is indeed present in the dataframe. We've purposefully kept the video in it's raw format instead of editing it. This shows that it's very common to get errors like these while working on your project, one have to be very careful before making a conclusion.
@@jovianhq yes sir, thanks for the session learnt a lot from this basically for "how" to do it there is ample of resources available but "what" to do in EDA is hard to find Thanks for that
idk if they have updated the dataset , but i just tried to find whether New City is there or not and if yes then the number of records ....(referring to 56:00 ) used this code - len(df[df['City']=='New York']) and got the answer.. so , New york is there in the dataset and the number of accidents is 7068
Hey, have you tried our Machine Learning course? We have covered feature Engineering/Selection and lots of other interesting topic in that course. View the course from here -> zerotogbms.com
sir i try this code and his show True in 'New York' city first i see df.values df.values show my all data value in array form then i write this code 'New York' in df.values this line of code show True as a output.
New York is present in the data. I think you have used the wrong function to check whether a value is present, 'New York' in df. City: Using this function will always get a false result if you check for any city. The correct function to check is 'New York' in df['City'].values
I did read your exploratory analysis file for crop production analysis… and all the agroup by queries that you had run were wrong.. plz look into it and rectify them
Yes, it's absolutely fine, you're not expected to know everything, and even if you know there can be a better way of implementing the same thing. So it's totally fine to google something out.
Sir here you have a column called siverity And it tells the siverity of the accident . So what I am asking is to find out the cities with highest number of accidents can I group by function and group based on the city and siverity . I.e df.groupby('City). Siverity.sum().sort_values( ascending = False) Bcs I feel this is a better approach then using unique values . Please please reply back
Yes, you can do that, but the code should be like this, df.groupby('City')["Severity"].count().sort_values(ascending=False), Here the column severity does not matter to get the total number of accidents, so we are just counting the total number of rows in each city instead of using sum() on Severity. For better assistance post your question in the community. jovian.ai/forum
Hello Sir, 1.Can you pls tell me how to handle missing observations for the following features FEATURE SUM Precipitation(in) 510527 Wind_Chill(F) 449288 Wind_Speed(mph) 128852 Humidity(%) 45506 Visibility(mi) 44206 Weather_Condition 44001 Temperature(F) 43030 Wind_Direction 41857 Pressure(in) 36270 Weather_Timestamp 30263 Airport_Code 4248 Timezone 2302 Zipcode 935 dtype: int64 I have removed "number" feature as 70% of the data of that column was missing Can we use mean/median/ mode or is there any other technique ? 2.For the univariate analysis wouldn't it be very lengthy and time consuming to study 47 features?
36:26 I tried numerics but an error was thrown saying 'numerics' is not defined. Googled the problem and found out df.select_dtypes(include='number') works.
In df.sample() we need to pass a value denoting the number of rows of sample we want. 0.1 * len(df) means the number of rows of sample we want to see is equal to 10% of the length of df i.e if len(df) is 100 we want a sample dataframe with 10 rows.
It was a good video, I like how he didn't cut out the part when he's stuck at some problem. ⭐⭐⭐⭐⭐
Thank you Aakash for giving a raw walk-through. Apart from usual documentation stuff I think the ability to google and find answers for the problems are much more important skills in the area of Data Science apart from ofcourse mathematical understanding. This walkthrough actually gave me more confidence in using pandas without worrying about typical syntax pitfalls
Glad it was helpful!
best project tutorial in any youtube channel. Everyone make video and cut those portion when they get stuck. On the other hand you just show the real scenario and show us how to find solution using google as well. we want similar video on data science project.
Love from Bangladesh.
Do you know why I loved this video? You kept it raw and real, you are clearly portraying how a data analyst thinks and does the project, which I believe is more important than fancy coding...syntax you can get easily but analytical thinking requires the real efforts
Hi shiv Im from non technical background i was doing the given project but when we close all tabs how to reach out to the same notebook
@@snehaldamkondwar618 Hi, where did you save it locally? Which folder
@@shivsharma9153 need to save??
@@shivsharma9153 open notebook through jovian run through colab with help of kaggle datset . I have written some line of code in colab .then i have close all tabs . Now how to go to that file where i have write my code.
@@snehaldamkondwar618 try to search locally with the name of notebook you may be able to locate it
55:52 this dataset contains New York data accidents, there are around 10.000 record. I checked it using df[df['City'] == 'New York']
Yes
Thanks! Not sure why I missed it. Maybe I was using a different version of the dataset.
@@AakashNS not sure why but ["New York" in Df.City] always returns false try it with a another city it just doesn't work
@@claudiolb8552
Use:
a. >>> 'NY' in list(df.State) or
b. >>> 'New York' in list(df.City) or
c. >>> cities_by_accident['New York']
yes, new york city is there in the dataset, the state which is missing is 'Alaska'
I have seen only half the video. Couldn't stop myself from apprecating the good work. Couldn't have been done any better. Way better than the Paid education platforms
Thank you for teaching us how to problem solve and the whole EDA process!
Happy to help!
I like the way you google for help ..Appreciate your time ..learnt new things on how to articulate our work ..thanks
Glad it was helpful!
great work jovian , we need more such types of session. Thanks a lot
Glad you liked it!
Aakash, you are one of the best teachers I have come across.
Coming from a hard-core medical background and pivoting into data analytics I came across your panda's courses while preparing for my foundation in python before starting a master's in the US in analytics this Fall.
Hands down you have given beginners like me a lot of handholding with your courses and videos!
Thanks, I'm glad you found our course helpful! 😊 - Aakash
@@jovianhq I went through many teachers on youtube and data camp: but truth to be told- most are ludicrously formal in their teaching methods and have a slower theoretical pace.
Is there any possibility to connect with Aakash to get certain roadmap tips for a beginner who plans to venture into the US Health Business Analytics Domain?
NY is in the state list. The Missing states are AK(Alaska) and HI(Hawaii). It also considers DC as state
The current dataset is updated.
A high percentage of accidents occur between 3 pm to 6 pm (probably people in a hurry to get to home)
Next higest percentage is 6 am to 9 am.
Over 1100 cities have reported just one accident (need to investigate
On Sundays, the peak occurs between 11 am and 6 pm, unlike weekdays
Interesting analysis and insights Shreyas!
Great course Aakash - this and everything else you have put here. Thanks for your generosity to teach this the way you have done it. Brilliant!!!
Great sharing. I've been looking for a full guide like this one for a while. Thank you!
Glad you enjoyed it!
this EDA is so motivating to me .......we all search in google...i thought why i can't recall all of those pandas function....
Sir actually New York data is present in the given data set.
We get the output as False if we use:
'New York' in df ['City']
And we get the output as True if we use :
'New York' in df ['City'].unique()
Yes Jeet, you are correct. We found it later but didn't update the video to show that this type of error might happen any time during working on a project. Great work on finding it!
I used - 'New York' in df.values and got True
One of the best TH-cam channel for data analysis learners❤️❤️
Watching the master is how you learn...Thanks a lot for this...
I can't see an EDA explanation clearly in other videos... awesome 🎉
bro you were my senior in intermediate .ascent junior college ,vizag . got a clarity on eda btw
Glad you liked our tutorials!
This was excellent. I hope for more of these in 2023
Definitely!! Stay tuned, more interesting videos coming soon.
@@jovianhq is this the type of project that is sufficient to be included in an analytics portfolio? or does it need to be something more extensive?
Great session on EDA. We need more such sessions on different datasets.
This is just an example, we hope the viewers will be able to make better EDA projects on different datasets after watching this video.
New York data is there, manually checked the whole data to find the city because the method you showed at 58 minutes mostly doesn't work because of the spaces, some values contain unnecessary spaces and that creates a problem.
concept is crystal clear by anuj bhyia.
Thanks!
What a great Tutorial! The df is obviously updated and I cannot find the 'New York' value in Cities, BUT there are data in cities_by_accident "cities_by_accident['New York']" and is equal to 7068
Good class of Python pandas, but in R exploratory and statistics analysis are way easier compared to Python. Example: data_frame %>% filter(City == "New York") bam!! dataset filtered. Summarize numeric data => data_frame %>% summary() !! bam!! Done!! in a totally functional way.
Both R and Python are great, you can use either one. Python is gaining more traction because it also has great packages for machine learning & deep learning.
This was a superb explanation of how to do EDA. Extremely helpful, Aakash!
Thank you Aakash. Really helpful. Liked the way we committed mistakes and even learnt something new from it.
Glad you liked it
This is best video on EDA I have ever watched! Thanks Aakash :)
Thanks for the kind feedback!
Best teacher for Data scince!
This is a great project and I really enjoyed it. After finishing this video yesterday, I am working on other parameters to expand my analysis. I would love to see more projects from Akash. Thank you.
Hi do you know once we close all tabs how to work on it again
Perfect video. I was looking for such video. Thank you Sir
Thank you for sharing this! It was really insightful to see the analysis process from start to finish. It also answered a few questions I had.
Glad it was helpful!
56:02 The dataset does contain 'New York'
"" cities_by_accident['New York'] "" gives us the output as 10255
yes..but it does not show the values..just showing count...strange!!
Yes, the dataset does contain New York now.
nice video, simple explaination and the best part was it from the scratch. loved it
loved your freecodecamp course. thank you so much
"New York" in df. City returns False
But "New York" in df.City.uniqu() returns True. (Which I have no explanation for)
And in fact, there are 4220 accident cases inside the dataset which occurred in New York inside the dataset (The dataset could be updated recently.). I don't know if it has been updated. But since the accident records fall in between the year 2016 and 2020, it would seem weird if new rows get added later on.
You are correct, there was "New York" in the dataset before as well. df.City returns a Series where if you search using the "in" operator, it will search for the indexes and not match with the values. Where as df.City.unique() creates a list and "New York" is searched within that list so you were able to find "New York".
Very educational video. Please keep posting such videos.
Thank you Aakash, great work
Thank you. This is one of the best video on EDA .
Thank You...
looking forward to your upcoming videos
Thank you so much. Definitely a very useful video. ✌🏽
Glad it was helpful!
@36:30 We can also do, df.describe().shape[1]
@54:40, I got the results for new york. I did cities_by_accident.loc[“New York”]
Yes, the dataset now contains information about New York
You are doing great job. Thank you
Thank you!
Thank You Very Much learn lots of things through this session
Glad to hear that!
Big thank you for being Here 👍👍
Glad you liked it!
Thank you very much ! This video was useful and easy to understand.
Hi, at around 49:17 when you are checking that weather we have 'New York' data or not so in that when we are checking for the existence with,
if 'New York' in df.values - it will return True
And
If 'New York' in df.City - False
Also
If 'Dublin' in df.City - False ( and for all the other cities )
So, in my preference we need to use the df.values ( it will check the whole dataset - yes might be time taking and requires unwanted computing processing as well )
Please help us to improve this part
Thanks
Yes @Aman, you are correct, New York is indeed present in the dataframe. We've purposefully kept the video in it's raw format instead of editing it. This shows that it's very common to get errors like these while working on your project, one have to be very careful before making a conclusion.
@@jovianhq yes sir, thanks for the session learnt a lot from this basically for "how" to do it there is ample of resources available but "what" to do in EDA is hard to find
Thanks for that
Wow, thank you for your concise explanations.
Glad it was helpful!
awesome video! big shout out from Brazil
Hey Mauricio👋, thanks!
This was great! Thank you for the video
This is really helpful thank you Jovian
Glad you liked it!
Thank you so much! It was very educational
i liked ur naturality
A great session indeed
Glad you liked it!
finished watching
idk if they have updated the dataset , but i just tried to find whether New City is there or not and if yes then the number of records ....(referring to 56:00 )
used this code - len(df[df['City']=='New York']) and got the answer..
so , New york is there in the dataset and the number of accidents is 7068
You are correct! New York was indeed present in the dataset, but in the live session it got skipped due to some mistake in code.
aakash is knowledeble as sky .
Can't agree more!
- Jovian Team
Among the top 100 cities in number of accidents, which states do they belong to most frequently? How do we solve this question
For personal use:- 1:17:19
please make vedio of feature engirreing and selection and thanks for this content
Hey, have you tried our Machine Learning course? We have covered feature Engineering/Selection and lots of other interesting topic in that course.
View the course from here -> zerotogbms.com
Thank you so much Sir !!
Previously I felt to remember all the pandas methods but you made me confident. Thank you Bhai❤❤
Glad it was helpful! Check our other courses at jovian.ai/learn
Really great video!
Any chance I can get a copy of the jupyter file?
Thank you so much for all your efforts :)
Thank you for the comment! Glad you like the videos
You did a fantastic job buddy
Glad you liked it!
tysm for this. this tutorial is a blessing
You're so welcome!
df[df.City == 'New York']
^ shows New York in the dataset
Yes, the dataset seems to have changed to include New York now
Hey I am getting an error while installing packages. "You may need to restart the kernel to use updated packages"
It's a good video but the dataset you uploaded that is updated one . We also want the row messy dataset which u use while handling missing values
Thank You. Unfortunately the dataset was updated from Kaggle, we don't have access to the previous version to the dataset.
sir i try this code and his show True in 'New York' city
first i see df.values
df.values show my all data value in array form
then i write this code
'New York' in df.values
this line of code show True as a output.
Please sir build neural network algorithm with manual coding without keras and use study case
New York is present in the data. I think you have used the wrong function to check whether a value is present, 'New York' in df. City: Using this function will always get a false result if you check for any city. The correct function to check is 'New York' in df['City'].values
On data preparation step my data frame is crashing continuously. What to do now?
I did read your exploratory analysis file for crop production analysis… and all the agroup by queries that you had run were wrong.. plz look into it and rectify them
Great video 👏 Congrats for this awesome job
Thank you very much!
i have completed zero to pandas course
can you plz create content about sql integration project?
Sure we will definitely consider the topic for our upcoming courses.
thank you🙂
In my case when i read the file the jupyter notebook is taking way too long time
What can i do?
Shall i do for this as my mini project???
there is a difference bw the county and country!
Thanks i got the idea of doing EDA.
Can you teach us about web scraping
.
Corey Schafer post some great videos about that
Hi Milan, We are doing a workshop on web scraping next Thursday(April 15th) at 9PM IST on our TH-cam Channel.
th-cam.com/video/RKsLLG-bzEY/w-d-xo.html
at 37:59 how did we get a plot without importing matplotlib ??
time 1:29:38 of the video......there was no data regarding on which days accident occur so how the graph was formed
trying now
Is it Fine to Google Something while working on a project??Like How did u in the Video??
Yes, it's absolutely fine, you're not expected to know everything, and even if you know there can be a better way of implementing the same thing. So it's totally fine to google something out.
Sir here you have a column called siverity And it tells the siverity of the accident .
So what I am asking is to find out the cities with highest number of accidents can I group by function and group based on the city and siverity .
I.e df.groupby('City). Siverity.sum().sort_values( ascending = False)
Bcs I feel this is a better approach then using unique values .
Please please reply back
Yes, you can do that, but the code should be like this,
df.groupby('City')["Severity"].count().sort_values(ascending=False), Here the column severity does not matter to get the total number of accidents, so we are just counting the total number of rows in each city instead of using sum() on Severity.
For better assistance post your question in the community. jovian.ai/forum
@@jovianhq but why doesn't it matter bcs if you read that column description it says the sevirity if the accidents.
Wow, why am I just seeing this channel now
Better late than never!
Great Video
thank you so much
You're welcome!
it is asking permissions to use those datasets,what should i do?
Awesome sir
Hello Sir,
1.Can you pls tell me how to handle missing observations for the following features
FEATURE SUM
Precipitation(in) 510527
Wind_Chill(F) 449288
Wind_Speed(mph) 128852
Humidity(%) 45506
Visibility(mi) 44206
Weather_Condition 44001
Temperature(F) 43030
Wind_Direction 41857
Pressure(in) 36270
Weather_Timestamp 30263
Airport_Code 4248
Timezone 2302
Zipcode 935
dtype: int64
I have removed "number" feature as 70% of the data of that column was missing
Can we use mean/median/ mode or is there any other technique ?
2.For the univariate analysis wouldn't it be very lengthy and time consuming to study 47 features?
I have seen people filling median values in the empty cells
if we have two columns like Vote_now and Past_vote so id chose rjd in both the columns how to calculate percentage of that
Your question is not clear, Can you please post your question in the forum with pictures of your DataFrame?
Link: jovian.ai/forum
Great session
Glad you liked it!
36:26
I tried numerics but an error was thrown saying 'numerics' is not defined. Googled the problem and found out df.select_dtypes(include='number') works.
In his project 49 column are there but now only 47 columns , is this problem bcz of this?
At 21:03 ..I couldn't load the data ...what should I do?
Which algorithm is used
How did this work 1:49:52
sample_df=df.sample(int(0.1*len(df)))
In df.sample() we need to pass a value denoting the number of rows of sample we want. 0.1 * len(df) means the number of rows of sample we want to see is equal to 10% of the length of df i.e if len(df) is 100 we want a sample dataframe with 10 rows.
Good session