Python for-loops are very slow so you should avoid them whenever possible. Luckily pandas supports doing math directly on columns. So to add the BMI column you just need data['bmi'] = data['height']/data['weight']**2 For example, for a dataframe with 300000 rows this is over 1000 times faster on my machine than the video's method.
You're absolutely right Phsopher!! 😀 I totally messed up on this one! it's not just about speed - but also resources! aaaand 5 lines of code that you just reduced into one! 😊 Thank you so much for your comment and for letting me know - I will film a quick correction tutorial shortly, my apologies for not catching that! (it's funny how I'm instructing folks to avoid for loops in "drop_duplicates()" but encouraging them in data manipulation... 🙃 I can't even use the "brain fart" excuse because of that hahahahaha 🤣🤣🤣) Sorry once again - I'll make sure everybody knows there's a MUCH better way! 😉
@@PythonSimplified I wish there was a Code Jam platform that is constantly offering the Best Pythonic Way to do something in real time, where somebody catches the flow of code while in production so it consumes less Brain resources, just like a computer, lol. Any ideas ? Do you know of any camps that nuture Best practices other than the typical ? I would enjoy more live interaction on the code.
I am already in love with you. 😂😂😂😂 You are an incredible and amazing teacher. The smile on your face and your sense of humor makes me want to learn more. I want to be your friend.
I Made a app to store my recipes in pdf . But I cannot figure out how to make the recipe save in to a format that I want to create. Really appreciate ❤
I wouldn’t have bothered with the for loop for the bmi you can calculate the bmi by using: data[“BMI”]=data[“weight”]/data[“height]**2 Great overview though, enjoy all the videos.
In fact I was working in a project last night, automating an excel file that contains supply levels for Xerox printers, and reading about pandas, great video
Hi Mariya, This video helped me with creating a budgeting app for an assignment in a programming course. Pandas was used in my app for outputting tables of expense data. Thank you for posting videos. - Henry
Mariya, have you looked into the Polars library for Python? It's supposed to be much faster than Pandas? It might be helpful to do a comparison video on Pandas vs Polars - strengths and weaknesses. What do you think?
@10:13 if you can't remove the index column, The issue might be the default behavior of read_csv when reading from a file with a custom separator (in this case, \t). The first column name is being interpreted as the index column. To resolve this, you can explicitly set the index column in select_DF file. Adjust line 2 like this: data=pd.read_csv("bmi.csv", sep="\t", index_col=0)
It's a cloud IDE called Wayscript: www.wayscript.com/ Made by a bunch of really nice folks who were supporting this channel from the very beginning 😀 Also, Scott4WhiteRock as in White Rock, BC? We're almost neighbors! 😁 Cheers from Burnaby!
Thank you so much Charlie! 😀 I have one correction to the code I presented in this tutorial, it should have been: data["bmi"] = data["weight"]/(data["height"]**2) This saves typing, doesn't waste resources and runs much faster! (especially when dealing with enormous amounts of data)
Great video! Now I have a quesiton, when I run the line 25 at time stamp 10:26, it was report "read_csv() got an unexpected keyword argument 'index'", could you please tell me how to fix it? Thank you very much!
really nice explaination.....when i was new to programming i write a code in 3 days to group the dataframe by using index and lists, after 3-4 months i came to know that pandas have inbuilt "groupby" 🤦 sometime i still laugh for what i did🤣
hahaha I totally understand! I had the exact same situation with the pd.read_html() method!! 😁 I have a 20 minute long video showing how to web scrape a Wikipedia table with Mechanical Soup: th-cam.com/video/MkGQmZoMuRM/w-d-xo.html And then in one of the comments, somebody mentioned read_html()... took me less than 5 minutes to get the exact same results!!! hahahaha 🤣🤣🤣 But the good news is - you'll never forget "groupby" just as much as I won't forget "read_html" 😜
Doing something like data.iloc[1]["weight"] is inefficient and requires accessing the frame twice - the first access extracts a series at index 1 and the second one retrieves the value at the 'weight' index from the series. Instead, one should try to always use the multidim accessors, that is .iloc[*, *] or .loc[*, *]. There are also accessors - .ix[*, *] - that mix the integer-based acces with with the column-name based access but these should be used with extreme caution and are in fact deprecated.
I learned a couple of tricks, that's good. I believe the documentation does not recommend iteration on data frames. You're better off to use the apply, map etc functions which I use regularly. They're quick and efficient compared to iteration, especially if you're dealing with million plus lines of data.
You're absolutely right Philip! 😀 The method I've shown is not only taking up waaay too many lines of code - but its also time costly and wastes lots of resources! It should have been: data["bmi"] = data["weight"]/(data["height"]**2) instead of the unnecessary for loop, the appending and all that jazz 😟 My apologies on the mess-up, I will film a short correction video very soon! And thank you so much for your comment! I really appreciate you letting me know! 😊
Binge watching your channel today getting solutions for my TODO list on my current project. (Refactor and Mobile version of my Clownfish Genetics Calculator) Now, I'll be able to refactor my results phenotype data with Pandas. It's current a nasty spaghetti if statement...uggh! Thoroughly enjoying your channel and learning a lot (more kivyMD and adb)!
Thank u.......The reason I am thanking is ,hitherto I am confused by the result which given by the Python is error, while I doing dataframe work .......because I don't know before watching this video It is important to put dataframe D and F in caps😁😁😁
Thank you so much for caring and teaching python which is so sweet , you are so professional and affable.I'm following your youtube channel and telling my friends to follow you, Good Luck :)
hello belle, thanks for all good information you shared with us. I am a new python learner and i would like to know how or the best way to extract data from a pdf file that has inside date, numbers and letters. i would like to see how we can sort them and delete what we don't need. if it is possible . thanks
Trying to think of a use case with pandas for my Open World Text Adventure lol. Not sure if I'd benefit from it or not since I don't need to know what data exists and where, nor do I need to see it printed out (it's printed but in a more user-friendly way). But I still enjoyed watching this tutorial and will try some things out in the future! Right now I just use for loops to detect what the game is asking for in the player inventory either by selection (viewing the inventory and highlighting the item with the arrow keys) or by action (like trying to mine for Iron, we know we need a pickaxe so a for loop will check for one equipped). I want to see if pandas has anything interesting I can do with my inventory dictionaries, but like I said I have no need to know what is where or how many the player has, it is just handled with my functions and for loops.
hahahahaha trust me, it's not gonna take you 45 years if you watch my channel 😜 Checkout this vlog of mine when I talk about the roadmap of learning Python: th-cam.com/video/wFEC7VbWBZo/w-d-xo.html I have so many videos explaining these concepts, but you can always take a quick "introduction to Python" course in an online platform like Udacity, Udemy, Idx, Coursera first. They often offer free introduction courses and it's a really quick way of getting up to speed with Python 😊 Best of luck on your programming journey - it's the most exciting profession in the world, you gonna love it! 😁
A quick tip for the face cam video problems ... record both on seprately . record the screen as you do and record your facecam with a seprate camera or use a seprate recorder for the facecma
Great video, thanks... fun examples and attitude made it quite interesting throughout in a way that many tech tutorials are not, and paced at what I thought was exactly the right speed..
Every time you click your keyboard, and my ears are devastated by the thunderous roar coming through my headphones I am traumatized 😢 Great video otherwise. I’m glad these resources are freely available. Thank you for your contribution to the coding community 😊
Nice video. But I couldn't find any solution about how to convert values in one column (DataFrame has loaded by read_csv) from string to DateTime type. Спасибо!
Good question. You can try mine and check the playlist for Python. Very easy to follow fundamental tutorials for Python. And source files can be downloaded. Another playlist in the channel is R Beginning course.
Python for-loops are very slow so you should avoid them whenever possible. Luckily pandas supports doing math directly on columns. So to add the BMI column you just need
data['bmi'] = data['height']/data['weight']**2
For example, for a dataframe with 300000 rows this is over 1000 times faster on my machine than the video's method.
Thank you
or data['bmi'] = data.weight.div(data.height.pow(2))
You're absolutely right Phsopher!! 😀
I totally messed up on this one! it's not just about speed - but also resources! aaaand 5 lines of code that you just reduced into one! 😊
Thank you so much for your comment and for letting me know - I will film a quick correction tutorial shortly, my apologies for not catching that! (it's funny how I'm instructing folks to avoid for loops in "drop_duplicates()" but encouraging them in data manipulation... 🙃 I can't even use the "brain fart" excuse because of that hahahahaha 🤣🤣🤣)
Sorry once again - I'll make sure everybody knows there's a MUCH better way! 😉
@@PythonSimplified No worries :) Thanks for great content!
@@PythonSimplified I wish there was a Code Jam platform that is constantly offering the Best Pythonic Way to do something in real time, where somebody catches the flow of code while in production so it consumes less Brain resources, just like a computer, lol.
Any ideas ?
Do you know of any camps that nuture Best practices other than the typical ?
I would enjoy more live interaction on the code.
I'm new to programming, but I can swear the way of this explains it works for me perfectly.
I hope everyone appreciates the amount of time and effort you put into your videos! I know I do! Thank you very much.
Wow! I have no idea what Pythons is, or why I'd want to use it, but I'm going to learn now!!!!
I am already in love with you. 😂😂😂😂 You are an incredible and amazing teacher. The smile on your face and your sense of humor makes me want to learn more. I want to be your friend.
I'm new in Python, but with your video I got it so fast and clear.....Thank you so much. Keep the great work on!
Thia was really helpful. I just joined an advanced deep learning course. But I found out i was missing some background.
I Made a app to store my recipes in pdf . But I cannot figure out how to make the recipe save in to a format that I want to create. Really appreciate ❤
I love ur videos. They're well structured with great explanations and are very engaging.
u teach super good keep uploading data science videos in your plaYLIST I LOVE IT
2 minutes into the video I realize we think alike when problem solving. Look forward to exploring your content!
I wouldn’t have bothered with the for loop for the bmi you can calculate the bmi by using: data[“BMI”]=data[“weight”]/data[“height]**2
Great overview though, enjoy all the videos.
You're the best mentor of all time 🎉
In fact I was working in a project last night, automating an excel file that contains supply levels for Xerox printers, and reading about pandas, great video
You are a very gifted teacher Mariya.
"There you go", another great tutorial ! Thx!
Hi Mariya,
This video helped me with creating a budgeting app for an assignment in a programming course. Pandas was used in my app for outputting tables of expense data. Thank you for posting videos.
- Henry
Mariya, have you looked into the Polars library for Python? It's supposed to be much faster than Pandas? It might be helpful to do a comparison video on Pandas vs Polars - strengths and weaknesses. What do you think?
You sold me on learning pandas.
Looks like a great alternative to ETL tools like informatica.
I'll learn it well.
@10:13 if you can't remove the index column, The issue might be the default behavior of read_csv when reading from a file with a custom separator (in this case, \t).
The first column name is being interpreted as the index column.
To resolve this, you can explicitly set the index column in select_DF file. Adjust line 2 like this: data=pd.read_csv("bmi.csv", sep="\t", index_col=0)
Great! Thank you!
Thanx and highly appreciate ❤
Create more content about ML and DL
Awesome!!
I liked this video and the girl as well. She speaks and move hands as my MSc supervisor.
Love the pandas tutorials! Thank you from bitterly cold Ontario, I hope you're keeping warm.
This is the tutorial I was after for some time. Thanks a million. Fantastic presentation (as always)
Love the Stich pullover!
Thank you Mariya. Your tutorials are really simplified and awesome!!!
Thanks for the tutorial. What terminal program are you using? Looks handy.
It's a cloud IDE called Wayscript: www.wayscript.com/
Made by a bunch of really nice folks who were supporting this channel from the very beginning 😀
Also, Scott4WhiteRock as in White Rock, BC? We're almost neighbors! 😁
Cheers from Burnaby!
Excellent! Clear, efficient, simple and well explained and demonstrated.
Mam pls tell which keyboard u use? Keyboard sound is awesome..
And there you go, this is how a magnificent video is made. Thanks for so much all tricks.
Your content is gold.
Wow you are a hero, you saved my project. Thank you so much!
This is such a great video thank you. I have been getting up to speed on pandas the past few days and this is one of the best I've seen. Thank you!
Thank you so much Charlie! 😀
I have one correction to the code I presented in this tutorial, it should have been:
data["bmi"] = data["weight"]/(data["height"]**2)
This saves typing, doesn't waste resources and runs much faster! (especially when dealing with enormous amounts of data)
I didn't know that you can do querys in a database with pandas! Super-mega-ultra-useful jajaja Gracias jefa
just what i was looking for!!!
Great video! Now I have a quesiton, when I run the line 25 at time stamp 10:26, it was report "read_csv() got an unexpected keyword argument 'index'", could you please tell me how to fix it? Thank you very much!
Great tutorial
I like the sound made by your keyboard when u were typing. It sound great too.
Your for loop really helped me with some things that I couldn't do with standard pandas calculations! Thanks! Great video!
What's this light projector device? It's awesome.
adding rows also can be done via pd.Series and passing a list
Your content is amazing. I really enjoy your channel.
really nice explaination.....when i was new to programming i write a code in 3 days to group the dataframe by using index and lists, after 3-4 months i came to know that pandas have inbuilt "groupby" 🤦 sometime i still laugh for what i did🤣
hahaha I totally understand! I had the exact same situation with the pd.read_html() method!! 😁
I have a 20 minute long video showing how to web scrape a Wikipedia table with Mechanical Soup:
th-cam.com/video/MkGQmZoMuRM/w-d-xo.html
And then in one of the comments, somebody mentioned read_html()... took me less than 5 minutes to get the exact same results!!! hahahaha 🤣🤣🤣
But the good news is - you'll never forget "groupby" just as much as I won't forget "read_html" 😜
@@PythonSimplified yeah true😅😝
An excellent way to remember- self taught might make mistakes in the beginning, but become some of the most well thought out for programmers.
you are absolute genius
Thank you mariya for your lecture it was really helpful
Thanks , there is a lot of good information in this video!!
Thank you so much! I'm glad you found it useful! 😀😀😀
How am I just discovering you?! This is glorious!
Thank you for all the details in this video.
Jumping to the previous video and codes (GTA stuff) was confusing, but "pandas" material was awesome!
Great video, Mariya. Keep up the great work!!
Doing something like data.iloc[1]["weight"] is inefficient and requires accessing the frame twice - the first access extracts a series at index 1 and the second one retrieves the value at the 'weight' index from the series. Instead, one should try to always use the multidim accessors, that is .iloc[*, *] or .loc[*, *]. There are also accessors - .ix[*, *] - that mix the integer-based acces with with the column-name based access but these should be used with extreme caution and are in fact deprecated.
Really cool tricks Mariya.
Especially the bot removing trick for code-jam, that really got me... Great work...
What a great explanation!!! Thxxx!
Thank you so much Juan! 😊
I learned a couple of tricks, that's good. I believe the documentation does not recommend iteration on data frames. You're better off to use the apply, map etc functions which I use regularly. They're quick and efficient compared to iteration, especially if you're dealing with million plus lines of data.
You're absolutely right Philip! 😀
The method I've shown is not only taking up waaay too many lines of code - but its also time costly and wastes lots of resources!
It should have been:
data["bmi"] = data["weight"]/(data["height"]**2)
instead of the unnecessary for loop, the appending and all that jazz 😟
My apologies on the mess-up, I will film a short correction video very soon!
And thank you so much for your comment! I really appreciate you letting me know! 😊
Mariya, I I really thank you for your tutorials!
Binge watching your channel today getting solutions for my TODO list on my current project. (Refactor and Mobile version of my Clownfish Genetics Calculator) Now, I'll be able to refactor my results phenotype data with Pandas. It's current a nasty spaghetti if statement...uggh! Thoroughly enjoying your channel and learning a lot (more kivyMD and adb)!
Just adding my own appreciation for a well architected video, and well done too for taking on the suggestion of others for improving the code.❤
This is dope am digging these pandas 🐼
Thank u.......The reason I am thanking is ,hitherto I am confused by the result which given by the Python is error, while I doing dataframe work .......because I don't know before watching this video It is important to put dataframe D and F in caps😁😁😁
now i love pandas :)
hi! thanks for the video
Thank you so much for caring and teaching python which is so sweet , you are so professional and affable.I'm following your youtube channel and telling my friends to follow you, Good Luck :)
hello belle, thanks for all good information you shared with us. I am a new python learner and i would like to know how or the best way to extract data from a pdf file that has inside date, numbers and letters. i would like to see how we can sort them and delete what we don't need. if it is possible . thanks
you are the best!!! , great job. Thank you.
Great video. Makes me appreciate R so much more.
Amazing tutorial as usual. Thanks again Mariya
Trying to think of a use case with pandas for my Open World Text Adventure lol. Not sure if I'd benefit from it or not since I don't need to know what data exists and where, nor do I need to see it printed out (it's printed but in a more user-friendly way). But I still enjoyed watching this tutorial and will try some things out in the future!
Right now I just use for loops to detect what the game is asking for in the player inventory either by selection (viewing the inventory and highlighting the item with the arrow keys) or by action (like trying to mine for Iron, we know we need a pickaxe so a for loop will check for one equipped). I want to see if pandas has anything interesting I can do with my inventory dictionaries, but like I said I have no need to know what is where or how many the player has, it is just handled with my functions and for loops.
I must learn a absolutelly basic of computer programming first. I will be back 45 yrs later. Cheers.👽
hahahahaha trust me, it's not gonna take you 45 years if you watch my channel 😜
Checkout this vlog of mine when I talk about the roadmap of learning Python:
th-cam.com/video/wFEC7VbWBZo/w-d-xo.html
I have so many videos explaining these concepts, but you can always take a quick "introduction to Python" course in an online platform like Udacity, Udemy, Idx, Coursera first. They often offer free introduction courses and it's a really quick way of getting up to speed with Python 😊
Best of luck on your programming journey - it's the most exciting profession in the world, you gonna love it! 😁
Not nearly that difficult. Learn how to get the computer to print to screen, then how to get text from keyboard, then learn conditional statements
The queen is back
Love your videos. Well presented and clear. Thanks for taking the time to make this.
This was a brilliant tutorial, but how would I make this into a data display board?
pandas is love thanks Mariya
A quick tip for the face cam video problems ... record both on seprately . record the screen as you do and record your facecam with a seprate camera or use a seprate recorder for the facecma
This was a helpful tutorial, thanks.
Well explained 👌🏿. What do you use to make your videos?
Great stuff.
One question though:
what is faster: filtering data from db with pandas/python of just sql command select XXX where xxxx ?
There you go M !! always amazing !!!
Great video with details. I would suggest you to make the same kind of videos about Numpy, matplotlib and other visualisation libraries. Also SQL)
Thank you so much Rashid!
Will do! both Numpy and Matplotlib are definitley on my list! 😉
Great video, thanks... fun examples and attitude made it quite interesting throughout in a way that many tech tutorials are not, and paced at what I thought was exactly the right speed..
very insight full .....i love this
Thank you!
Very nice video, and since I got to see this video I also checked your channel.... I subscribed
Keep up the good work! :)
ur amazing always , can u do any video on stock market data manipulation using pandas and numpy ?
Every time you click your keyboard, and my ears are devastated by the thunderous roar coming through my headphones I am traumatized 😢
Great video otherwise. I’m glad these resources are freely available. Thank you for your contribution to the coding community 😊
Nice video. This is the kind of guide that is handy in almost every situation with pandas. I've learned new tricks. Thanks you.
Nice video. I am new to Pandas and this video has thought me a lot. Thanks!
Great video. Congrats for your channel
You have a gift for explain, thank you, I would love to know about pandas and data analytics
Pandas is the best money making library I’ve seen so far.
Hola, great job!.. thank you.
I recommend learn typing (mechanography). Bye!
This is a great video! I use excel a lot, but I am trying to use Python more so I can work with larger sets of data :) Thanks for the great tutorial!
Hey Maria, amazing job! very nice video..just a feedback the audio is out of sync with your lip movements
Thank you , for all the useful and informative content.
Thank you so much for your lovely comment! 😃
Thank you Mariya, i now who what iloc is😁
Nice video.
But I couldn't find any solution about how to convert values in one column (DataFrame has loaded by read_csv) from string to DateTime type.
Спасибо!
Hey i would like some help with a question about my assignment using python with pandas
awesome shirt haha
Thank you. Good pace and really friendly 👌
Can you explain if you need to save data to excel file?
Good question. You can try mine and check the playlist for Python. Very easy to follow fundamental tutorials for Python. And source files can be downloaded. Another playlist in the channel is R Beginning course.
Your Stitch t-shirt is awesome! :D
I like so much your explanation. Thank you for the information :D.
Thank you so much for the lovely comment Axel! 😃
thanks it's very useful for us :) love you
Thank you so much!! super happy to hear! 😁