How do I apply a function to a pandas Series or DataFrame?
ฝัง
- เผยแพร่เมื่อ 22 ส.ค. 2024
- Have you ever struggled to figure out the differences between apply, map, and applymap? In this video, I'll explain when you should use each of these methods and demonstrate a few common use cases. Watch the end of the video for three important announcements!
Subscribe to the Data School email newsletter: www.dataschool....
Join "Data School Insiders" for exclusive rewards: / dataschool
== DATA SCHOOL VIDEO TUTORIALS ==
Data analysis with pandas (30 videos): • Data analysis in Pytho...
Machine learning with scikit-learn (10 videos): • Machine learning in Py...
Version control with Git and GitHub (11 videos): • Version control with G...
== PANDAS RESOURCES ==
GitHub repository for the series: github.com/jus...
Series "map" documentation: pandas.pydata.o...
Series "apply" documentation: pandas.pydata.o...
DataFrame "apply" documentation: pandas.pydata.o...
DataFrame "applymap" documentation: pandas.pydata.o...
== RELATED PANDAS VIDEOS ==
loc, iloc, and ix: • How do I select multip...
string methods: • How do I use string me...
== JOIN THE DATA SCHOOL COMMUNITY ==
Blog: www.dataschool.io
Newsletter: www.dataschool....
Twitter: / justmarkham
Facebook: / datascienceschool
TH-cam: www.youtube.co...
Dude , you're the Corey Schafer of Pandas.
Ha! Corey is a friend of mine and a great educator, so I take that as a huge compliment 😄
haha
Awesome. I think it is especially super useful for non-English speakers as your English is clear and slow. Thank you very much.
You're very welcome!
Great Series: I have learned so much. You are one of the best teachers I have ever seen.
Wow, thank you so much! I really appreciate your compliment!
Thanks for the great series. Your videos have given me a good foundation in pandas. With what I have learned here I have already been able to generate several data clean-up scripts, and run data analysis that I have presented at an academic conference using only pandas. Great work!
Wow! That's awesome, I'm so glad to hear this series has been helpful to you!
Hi Kevin, how do I lookup a column values in one datra frame against the second data frame, say column 1 and get the corresponding data from column 2 of second data frame and add that results to first data frame.
No doubt one of the greatest teachers on TH-cam!!
So calm and graceful ‼️
You are so kind, thank you!
Thanks for everything. Every minute here invested has been most valuable.
Awesome videows. Super didactic!
Thanks!
Hands down, one of the best Pandas instructor in YT!
Thank you!
Thank you so much for the time and effort you have put into your video series, and the amazingly well documented Jupyter Notebook with everything from the series!
Thanks for appreciating!
This is amazing! The way you explain the complicated stuff is fenomenal, thank you
Thanks, I really appreciate your kind words!
Thank you again. whenever I watch this again, I am learning new.
Great to hear!
4-8 hours to make a less than 20 mins video? Gee, I appreciate your work
It's true!
you are the best pandas teacher I've ever seen
Thank you!
One of the best Pandas videos ever. If i had saw this a few days ago, it would have saved me a few days of frusteration
Glad it helped!
we can also get the first element of the list of strings by using :
train.Name.str.split( ' , ' ).str.get( 0 )
Hey Yours above solution worked fine for me but when i am trying with apply function , it is giving me error TypeError: 'float' object is not subscriptable
Also the list which i am passing is a list of strings only .
def tryfunc(list1,position):
return list1[position]
hi["new"]=hi["WebframeDesireNextYear"].str.replace(".", "").str.split(";").apply(tryfunc,position=0)
You are such an awesome tutor that I felt this is the best channel for pandas. Please keep posting. Thanks a lot.
Thanks so much!
every minute i spent here more than worthy.The way you teach to really good.
Thank you! 🙏
EVERYBODY: remember to always use vectorized operations when possible. They are many orders of magnitude faster than using "apply", which is not vectorized.
- For instance, instead of:
drinks.loc[:,'beer_servings':'wine_servings'].apply(max, axis=0)
- Use:
drinks.loc[:,'beer_servings':'wine_servings'].max()
for simple operations like len and sum, no functions are needed, for example: df ['Len'] = df [Column_with_text] .str.len (), it will be much faster. To calculate the sum or average, we can choose an axis: df ['Total_for_rows'] = df.sum (axis = 0) or df.loc ['Total_for_columns'] = df.sum (axis = 1)
I've watched a lot of tutorial videos in my time, but you my friend, are hands-down the best!
Wow, thank you so much! 🙏
Though in some cases, APPLY may not be as fast as using other approaches, that difference is usually negligible. Kevin doesn't mention that here, in the interest of time/audience; HOWEVER, he did make this very point in an insightful Stack Overflow post that I recently came across. In fact, there was a subtle advantage to using apply, instead of the other alternative, in that it preserved the index. So, yes: speed and optimization are great and all, but most of the time, writing code that is simple / readable -- which Kevin acknowledges as being subjective -- will just as effectively get the job done.
This video series is wonderful because it's not only palatable to beginners, but I find myself coming back to it and picking up certain details of why things work as they do, that may not have *stuck* the first time around. It can initially feel overwhelming that Pandas gives you so many options on HOW to do things, but I've come to appreciate the value in wielding a proverbial swiss-army knife, and I've started to recognize which situations favor using one instrument over another. Thank you, Kevin, for being a great teacher, and for presenting content -- beyond this video series, at that -- that appeals to users having a range of experience.
Thank you *so much* for your thoughtful comment and your kind words! 🙏
Sometimes when I have free time I watch your videos. Always walk away with something new and useful. Thanks for the great vids.
Awesome! That's so nice to hear.
Best pandas tutorial so far on the internet... Thanks Kevin. Great work. hope you can do more tutorial on applied Tensorflow & ML
Thanks for your kind comment, and also for your suggestion!
Saved my butt for finishing my hw today. Thx!
That's awesome to hear!
Thanks a ton for the great series, Kevin!. I have watched all your videos and Pandas series videos are still go to watch even in 2021. You are a life savior, Sir!
You are so very welcome!
Hats off to you .You made me pro in coding.Thanks a lot..keep going
Thank you!
Thank you. Your explanations are well-paced and easy to understand and most importantly sticks to memory.
Thanks for your kind words! 🙏
Corey brings me here. Your videos are great! Keep up the great work
Corey is great! 🙌
This is a great video series on pandas. Really concise and clear. Many thanks for the effort!
Thanks for your kind comment! I enjoyed creating the series :)
I just found your videos and they are awesome. Your explanations and examples are easy to understand, but also very practical. I am looking forward to watching the other 29 videos
Great to hear!
this helped me in better understanding the fundamental difference. thank you so much.
You're very welcome!
Excellent tutorial, your video series is a must watch for anyone serious in data science.
Thanks very much for your kind words!
Omg I really loooove your videos... I'm going to use everything you share, it's amazing. I quit on learning python 2 years ago and I want to give it a try again now. And by seeing your pandas videos I truly think it will woth it. Thank you so so much for your effort... keep going 💪🏽
Thank you so much! 🙌
I'm glad I found your videos! You make understanding these concepts much easier!
Thanks very much for your kind words!
Thank you for sharing! I learn so much from every video!
I'm so glad!
Incredible explanation which is very easy to understand. One can give this much of a clear explanation on stuffs only if they are confident with what they are talking about and you are nailed it buddy.
Thanks very much for your kind words!
All videos are brilliantly curated and explained thoroughly. Great way to teach and learn.
Many thanks!
Hey...great Series Sir..learning Pandas module from your side is such a boon..Thanks,Love from India!!!
Thank you!
You know you like a channel when you find yourself trying to subscribe only to find that you already have
Thank you! 🙏
Thanks Kevin for such a nice tour on pandas....I enjoyed it a lot. By the way, In future if you decide to start it again, please enlighten us with the "Reduce" function as well. Thanks again for helping people so much. I know it is really hard to make a video. It takes lot of time and effort. Take Care, Good luck to your projects
Thanks so much for your kind words! Glad the pandas series has been helpful to you. You're right, it is a ton of effort to make the videos, but definitely worth it :)
Superb tutorial. Thanks so much!
Thanks for your kind words!
Wow. Excellent job of explaining map vs apply methods not simply explained elsewhere (even by Wes)
What a nice compliment, thank you so much! :)
Just finished all 30 videos. Thank you very much Kevin! You are a great teacher!
Awesome! Congrats on finishing the series! :)
Thanks Kevin for devoting your time to provide such good series videos.
You're welcome!
exceptional videos, it has the depths and info needed to catch up with data scientist and with remarkable teacher, well done, thanks
You're very welcome! Thanks for your kind words!
I followed all the videos of this series and course content is awesome and your teaching style is tremendous
Thank you so much!
Excellent! I am much clear now on this topic...Many thanks!
You are welcome!
Thank you for your clear and concise explanations. Such a pleasure to watch your videos.
Wow, thank you so much for your kind comment!
Thanks for the series. Very helpful. Would recommend to everyone who want to kick start.
You're very welcome! And, thanks for your kind comment!
Nice, finally I understood the apply method
Many Thanks !
Great to hear!
I really appreciate how clear and thorough you are with your explanations. Thank you for this!
One question I have, when you split on the comma, I understand how it creates a list of strings. Can you explain why the strings do not show up in quotes, though? Don't most other string list elements? Thanks!
Glad you like the videos! Regarding your question, pandas doesn't display quote marks around strings.
Very good explanation
please run a case study from problem definition onwards
Great videos, perfect explanation. Best teacher on TH-cam for Data science... Waiting for your videos
Hey thanks for sharing! You lessons are clear
Thanks!
Thank you so much for this! You're so clear and concise!
Glad it was helpful!
You are really a best teacher :-) your videos are very helpful and very informative.
Thanks you so much 😊😊
Thank you! 🙏
Thanks for this , It is really helpful, but in my case, it returns 0 and 1 instead of column label
df.loc[:,'Parle':'Soap'].apply(np.argmax,axis=1)
your video helping me to much
thanks from india
@Data School, I did some testing and I have found that if you type the lambda function @10:35 a little differently, you get a more efficient processing time:
import timeit
tic = timeit.default_timer()
train.Name.apply(lambda x: x.split(',')[0]) # Method 1
toc = timeit.default_timer()
method1 = toc-tic
print('Method 1:', method1)
ticktic = timeit.default_timer()
train.Name.str.split(',').apply(lambda x: x[0]) # Method 2
toc = timeit.default_timer()
method2 = toc-tic
print('Method 2:', method2)
if method1 < method2:
print(f'Method 1 is faster by {method2/method1} times.')
else:
print(f'Method 2 is faster by {method1/method2} times.')
Thanks for sharing! 🙌
You are a sweetheart bro! Thank you so much, the whole playlist is perfect
Thank you!
Very nice teachers i ever seen.
Thank you!
You should be an English teacher. Very pleasant pronunciation and easy to understand for non-native English speakers
That's very kind of you to say, thank you!
Best python tutorials in the sea of the net you can dig. I promise subscribe it!
Thanks so much for subscribing, and for your kind comments!
Clear explanation, excellent tutoriel. Many thanks.
You're very welcome!
Really appreciate it, thank you, Kevin!
You're welcome!
Clear and concise. Thanks
You're welcome!
Thanks a lot for your time
It has been very helpful for me
You're very welcome! Thanks for joining me for the series :)
Great videos 👍 I have learnt a lot from them. I have one question it would be very great if you find time to answer this. I have a list of string values and I am trying to create a new column and the value should be 1 if the value given in the list exists in any of the column of dataframe else the value should be 0. Eg. There is a dataframe with cols A,B,C,D and I have a list names test which has some values in it. If the value exists in either A,B,C or D then the first row of new col E should have 1 else 0 and similarly for the next row
Thanks. it is a great series about DATA SCIENCE
A clarification on the apply function.
In the example where you apply it on the Name series, how is it that you can use the str.split function on the 'Name' series without the apply function in the dataframe but for 'len' you need to use the apply function?
Thanks for your effort. It never gets old.
Thanks for your videos. They really do help build the knowledge base for beginners like myself
You're very welcome! Good luck in your daat science education :)
Thanks for the great videos. I just finished watching all the 30 videos in this series and enjoyed learning Pandas. You are a great teacher! I also signed up for the newsletter. I have a question and hope that despite your busy schedule, you can either answer me, or at least point me to the right direction: How would I add a column to a data frame, which holds aggregate data from the same dataframe? Let's say I have a dataset of "expenses" over the last 5 years. I want to add two columns to the dataset, and for every row, the first column will hold the "year to date" sum of all expenses (including this one), from the beginning of the year, and the second column will hold the "month to date" sum of all expenses. How can I accomplish this? Thanks!
Thanks so much for watching the videos, and for your kind words! Regarding your question, I feel like it's possible but I'm not sure exactly how. If you are interested in joining Data School Insiders, you could post this question in our private forum and see if someone happens to know: www.patreon.com/dataschool
Thank you so much for this video and for all the other videos in your amazing channel. I just found a solution in this video for a problem I was struggling with :)
Awesome! Great to hear! :)
Really like these informative videos! I was hoping you'd get to a video on multi-index
Glad you like it! I cover multi-index a tiny bit in my DataCamp course: www.datacamp.com/courses/analyzing-police-activity-with-pandas?tap_a=5644-dce66f&tap_s=280411-a25fc8
You might like to know that I just released a video about the MultiIndex: th-cam.com/video/tcRGa2soc-c/w-d-xo.html
thank you very much. the best explanation of youtube.
Wow, thanks!
very knowledgeful thanks for this
You're welcome!
Thank you so much! All your videos are amazing!! I learned so many new things
Thank you for the kind words 🙏
This is amazing technique...Many thanks for the video
Sorry to hear that you decided to put this series on hold.
It was a hard decision, but it's the only way I have time to complete other projects, such as launching some new online courses. Thanks for joining me for the series, and thanks for understanding!
This is an awesome tutorial! LEarned a lot from this. One question, what if I want to create 2 new columns named "First Name", "Last Name" and then split the values in Name column into those 2. Ex:For 1 st row, the "First Name" = Mr. Owen Harris and "Last Name" = Braund. Could you share the workaround to get this done?
Hi Kevin,
Great Tutorials, learned a lot.
Thanks.
You're very welcome!
First of all, thank you very much for the time and effort you put into this video series it has been very helpful. Moreover, would you know how to subtract from a column a substring that is in another column?
There's probably a string method that can help you, but I don't know which one off-hand... sorry!
It was a great video series. Thank you very much for your great effort. You are great Kevin
Thanks! :)
This can be used in an efficient way for feature selection.
This was great, i did the exercise and it was great..I will sign up to your news letter.
Awesome! Here's the link to sign up for my email newsletter: www.dataschool.io/subscribe/
very good examples. Thank you so much
You're welcome!
The video is perfect, and i studied a lot from it, thanks.
You're very welcome!
thank you so much for putting so much effort.
You're welcome!
Thank you very much. You've been a great help.
Glad to hear that!
Brother you look like Dr. Sheldon Cooper from tbbt 😂
Nice explanation 😊
Thanks! 😄
Good to watch and learn
Thanks!
Hi Master Kevin ,
Actually i tried below argmax method in my data as you suggest.
# finding out the which argument is max
drink.loc[:,'beer_servings':'wine_servings'].apply(maxBeer,axis=1)
My output:
showing index value of the argument ultimately argmax do that only. So I written function externally to get string value. which written below.
# String value for index
def maxBeer(a):
if np.argmax(a) == 0:
return 'beer_servings'
elif np.argmax(a) == 1:
return 'spirit_servings'
else:
return 'wine_servings'
most helpful video man
Thanks!
Just amazing! thanks a lot for sharing your knowledge.
Thank you!
Great videos.
I have 2 questions:
1. How to display ONLY the negative values?
df [df < 0] gives a data frame with NaNs. I just need negative values.
2. Using map functions (pipe, apply, applymap), how to replace all negative values in a dataframe with the mean of all values of data frame in Python Pandas?
Something like this but using map functions:
df [df < 0] = df.values.mean( )
Thanks in advance
Sorry, I won't be able to assist, good luck!
Thank you ! 😊😊😊
You're welcome!
it would be great to see a vid of where pandas is really necessary. These practice examples, while useful, don't expose the reason why using pandas is needed over basic python
Thanks for your suggestion! In brief, you could probably do 95% of pandas code in base Python. However, you code would be a lot longer... sometimes 2x longer, sometimes 50x longer.
Lucid explanations.. i love your channel!!
Thanks!
super clear and very helpful. thanks!
You're very welcome!
Hi, I have a question. I am trying to add a column to a df with the map method. It works. Kinda ... It only copies the first part of the (quite long) string and leaves the rest of it out. The split occurs at a random point. Why is it not taking the whole strings into the dataframe?