Hope you all are all staying safe! In this video we'll be learning a lot about working with dates and time-series data in Pandas, and also look at doing some basic plotting. In the next video, we'll be learning how to load in data to Pandas from different (Excel, JSON, SQL, etc). Let me know if there is anything else you'd like me to cover in the Pandas series. I will likely be taking a break from this series after the next two videos are released just so I can focus on some different topics.
Hey Corey Will you be able to cover some machine learning libraries like scikit learn,tensorflow,...? Because matplotlib and pandas that you have covered so far is related to machine learning and we can put this libraries into practical
@@kaaaaasraa49563 I'll do my best. It's a very large topic to cover, so I've been a bit intimidated to tackle such a large course, but I know many people are wanting that, so I'll do my best to start working on something soon
Hey Corey, i was wondering if you could help do a complete EDA on one dataset ,kaggle or from anywhere in which you could cover the entire operations that we've learnt in one complete tutorial video with like a real world dataset. That could be like the end of this series video or something.
3:50 convert to datetime using to_datetime 7:45 parse dates while loading data from csv 11:20 using day_name on whole series using dt class 12:20 create column of dayname 13:20 min and max methods on datetime series 15:00 filtering by dates as string 17:20 filter by to_datetime 18:40 set date as index 19:21 filter data by just passing the date in brackets 20:05 using a slice to get specific dates data 21:00 calculating average of a slice(timeframe) 22:05 getting max value of a column on a given day 23:50 resampling(breaking down by days) a whole column into a new variable 27:00 plotting with matplotlib in pandas 28:28 resampling df with multiple columns 30:55 using agg to apply different function on different columns while resampling
For all those getting the following error: *FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime module instead.* Just do this: *from datetime import datetime* and change the *pd.datetime.strptime()* to: *datetime.srtptime()* and it should work the same.
I used the following import datetime as dparser d_parser = lambda fn : dparser.datetime.strptime(fn, '%Y-%m-%d %I-%p') this also worked. is there any advantage of your method over mine ?
@@mrrohitnayak1 its the same thing, u just named datetime as dparser, His code would call datetime.datetime.strptime() and yours call dparser.datetime.strptime()
To anyone getting the message that 'date_parser' is deprecated and will be removed in a future version... You no longer need to pass in a function, just the date string formats in the argument date_format='%Y-%m-%d %I-%p' example: df = pd.read_csv('ETH_1h.csv', parse_dates=['Date'], date_format='%Y-%m-%d %I-%p')
I learn through this video in 2023 as a new entry-level data analyst and non-native English. It's still relevant and lite but packed, especially for the `.resample()` method. It saves my time compared to using `.groupby()`. In addition, we need to keep in mind indice filtering. It's better to use `.loc[]` to avoid the deprecation issue. Thank you, Corey Schafer. Your channel is such a gem!
For anyone getting error in the first part of the video, this is what worked for me d_parser = lambda x: pd.to_datetime(x, format='%Y-%m-%d %I-%p') df = pd.read_csv('data/ETH_1h.csv', parse_dates=['Date'], date_parser=d_parser)
@@rafabws for dfx.loc['2020-01':'2020-02'] , I get KeyError: 'Value based partial slicing on non-monotonic DatetimeIndexes with non-existing keys is not allowed.; What to do with it? plz help
@@user-testo21347 I was getting same error too and I believe I found the solution. For some reason our indexes are not in order. For this reason, if we sort our indexes by writing the code dfx.sort_index(inplace=True) before writing the code dfx.loc['2020-01':'2020-02'], we get rid of the error.
@@user-testo21347 I was getting same error too and I believe I solved it. For some reason our indexes are not in order. For this reason, if we sort our indexes by writing the code dfx.sort_index(inplace=True) before writing the code dfx.loc['2020-01':'2020-02'], we get rid of the error.
Also to get the mean for resampling, the default for the mean function was changed. So to get the weekly resampling at around 30:05 of the video: df.resample('W').mean() will give an error do this instead: df.resample('W').mean(numeric_only=True)
While there are already other excellent pandas tutorials on TH-cam your videos IMHO stand out for their carefully crafted examples, clarity of explanation, information density and relevance to practical applications. I'd really love this series to be continued and cover some of the more advanced topics.
Hi Corey, thank you so much for taking the time and putting this series together. This is one of the most structured series on panda on the web. I feel confident in pandas after watching the series. We would love to see something similar on the NumPy. Please, Corey...put together a Numpy series for us.
I want to point out something. I like your videos. There is nothing boring about them. But I do not have much time so I watch them at 2x. Your normal talking speed is not slow but very clear so I can easily understand you at 2x too.
The concept of resample is so damn crazy!!! I struggle so much in other systems with this kind of aggregation and it's easy like nuts in pandas... Corey, you are great!
20:32 if you are getting error here then you first have to sort the index and use loc: # Sort the index df = df.sort_index() # Select data between January and February of 2020 df.loc['2020-01':'2020-02']
if you get this error (KeyError: 'Value based partial slicing on non-monotonic DatetimeIndexes with non-existing keys is not allowed.') try sorting your data before slicing. df = df.sort_index()
THank you so much. After this, can you slice one day of data? like at 22:44 df["2020-01-01"]? I got this error: KeyError: '2020-01-01'. I also tried df.loc["2020-01-01"], same error.
I loved the point Corey made about how much easier it is to use date as your index, however while following along I got a warning this warning: FutureWarning: Indexing a DataFrame with a datetimelike index using a single string to slice the rows, like `frame[string]`, is deprecated and will be removed in a future version. Use `frame.loc[string]` instead.
Gave a thumbs up before even started watching the tutorial..and I'm not disappointed at all. Keep up the good work Corey!!! BTW you are the only TH-camr getting likes from me.
Thanks a lot for this whole series Corey!! I've watched 3/4 other tutorials between youtube and other platforms and yours were the best by far, especially due to the fact that I'm from Italy so it was easier to follow your explanations step by step I can't wait for your plotting with Pandas tutorials too, those are gonna be pretty useful! Have a nice day a stay safe ;)
Hi Corey..I have referred to various resources to learn Pandas properly on TH-cam. And your tutorials are the best I could find. And looking forward to learn more from your matplotlib tutorials. Thanks for creating top class content!!
Please show more *errors and its corrections* for future vids. The fixes are really beneficial! Thanks for recording, editing, uploading and sharing! 👍
Thank you Corey for the wonderful tutorials. No paid courses can come close to the way you present the depth of the concept. Thank you for making it free and a big thank you for keeping up the good work.
Hi Corey! Your series on pandas is really informative and helpful. Can you make a similar series for numpy? I believe that's a topic that will be of interest to many of your viewers.
After using d_pars = lambda x: datetime.strptime(x, '%Y-%m-%d %I-%p') i got this warning: FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime module instead. I mean, i still get correct output , but if u wanna remove this warning just do: from datetime import datetime then d_pars = lambda x: datetime.strptime(x, '%Y-%m-%d %I-%p'). Treat it like curiosity or sth Btw, great videos, sir!
great job Corey, I was going to go through Wes McKinney's book but looking at the pandas section I realized that I already know most of it by now form your series. Pls do the data plotting soon! ;)
Your tutorials are the bests on the web. Before I can do the index filter (19:35 in the video), I had to sort the dataframe by index(df = df.sort_index()) I don't know why but it was given me an error (AssertionError: ). Thanks for sharing.
Well done! I didn't know that I can pass in a dict in agg. Always passed in a list and then selected the column-aggregation tuples that I needed, which is quite resource intensive and hard to read. Also I did not know I could make datetime comparisons to a string. Thank you.
Your every video is soo nice and perfect. The explanation is very friendly and in a proper sequence. Your channel works for me as a one-stop-shop. Thank you soo much man.❤️❤️ Love from India 🇮🇳
As usual your videos are great ! You explain things very clearly. Please start videos on Machine Learning and Artificial Intelligence soon. Earnest request !
Hi Corey, much appreciated for your video, just excellent for a beginner like me! But I have a question here: for the resampling of the whole DataFrame, say, if I want to get the open price at 00:00:00 for the 'open price' of the day and the close price at 23:00:00 for the 'close price' of the day, what function am I supposed to use here? Thank you !
All I can say you are awesome. The way you explained 😍✌️✌️✌️✌️. You have the best play list for pandas on web. Your efforts are truly appreciated 👍👍👍👍👍👍
Fantastic series! Thank you for providing this awesome content. I have been using Pandas quite a bit lately to process data, one thing I am struggling with and would really find helpful would be a video on taking these Pandas notebooks and functions and converting them into actual Python apps that can be reused on other projects. Pandas can really be like magic sometimes and being able to pass data into a Python app that could apply these functions would be extremely powerful, I have yet to find any good tutorials or reading on this subject, one from the master would be greatly appreciated!!
Values the time of viewers. Not sure if you really speak so fast or just increase speed in the first and last few mins of video. I just like the way it goes. Thanks for the great tutorial.
For those getting an error for accessing the datetime index, you need to sort it first now. So instead of df['2020'] make it df.sort_index()['2020'] and the same for slicing it
Fantastic series on Pandas. I have no words to express how i am thankful to you for this series. You are a great teacher. I am yet to check on the 11th video on this series and up next to the Matplotlib series (definetely). You are awesome. A request, could you please plan to do Numpy series as well.
Let me echo what plenty of others have said -- your videos are exceptionally well done and very understandable!! I knew virtually nothing about Python about two months ago and now I can do plenty of rudimentary, but useful stuff with it. In working through this tutorial with data that I've exported from ThingSpeak, I get to the point of having pandas successfully (apparently) converting the str by using: df['created_at'] = pd.to_datetime(df['created_at']). I get no errors and it seems to recognize the datetime group automatically without providing formatting schema. However, when I try to prove that it's now a datetime via the .day_name() method it throws an error and tells me my attribute is a datetime.datetime object and doesn't allow me to use that method. I get the same results if I convert the created_at column via the read_csv() method at the initial import. From forum searches I'm concluding that it got converted differently, than in your example, into a datetime object. I don't totally understand what that means, but if I try different methods on those objects, I can manage to arrive at the same destination, which is to return the day of the week for a given date. I used this strftime() method: print(datetime.strftime(df.loc[20001,'created_at'],'%A')) IDK if you're still responding to comments on a 2 yo vid, but in case others ran into this problem, that's how I solved it. I'd be great to have a video on the datetime library (module ?) and how that's different and how to best use it. Maybe you've already done that, so I'll look for that next in your extensive list of videos, but if not, just a suggestion. Again - thanks for all the amazing education efforts!!! cheers
lol -- and of course I now see a tutorial on this: Python Tutorial: Datetime Module - How to work with Dates, Times, Timedeltas, and Timezones. checking that out next!! 🤓
I got an error with the lambda function trying to parse the dates while loading from csv. The error stated pandas no longer supports pd.datetime. After checking the comments I tried @larc99's suggestion but still got an error as it was expecting a string rather than a function for the format. What gave the same output and I think is much simpler (without the lambda function) turned out to be: df = pd.read_csv('data/ETH_1h.csv', parse_dates=['Date'], date_format='%Y-%m-%d %I-%p')
It does not work by slicing the df using date such as df["2020-01":"2020-02"] at 21:40. I got this error "TypeError: cannot do slice indexing on RangeIndex with these indexers [2020-01] of type str". Anyone experienced the same? Pandas datetime is sooooo annoying and not intuitive at all.
How to solve error 'Value based partial slicing on non-monotonic DatetimeIndexes with non-existing keys is not allowed.' while using df['2020-01': '2020-02'] at timestamp 20 :17
Great tutorial! Question: How would be able to grab the last trade close price for the week in the agg demonstration? Doesn't make sense to average the daily close as the weekly. The close is the close.
This will be the video I am looking for. I am pretty sure. LET IT BE! Pretty mad how much I failed at plotting panda dataframe in matplotlib sorted by month. We have a german saying, „taking a detour increases the knowledge of your surroundings“. From 0% to 5% domain knowledge in multiple hours:/
I like all the videos u upload and u are the reason I started development and became fond of python. Can u please make some videos on how to write unit testing in python?
Hello. am using Jupyter Notebook on Ubuntu 20.04 LTS, my date and time in the setting is showing: 14 June 2020, 13: 48 and on the desktop its showing: Sun Jun 14 13:48:25 which format can i use?
Hi, New to Pandas and just a bit confused: Firstly at 6:36 , we converted all the values of the Series to datetime type from String type using: df[‘Date’] = pd.to_datetime(df[‘Date’], format = ‘%Y-%m-%d %I-%p’) but when doing the same thing while loading csv 10:24, we used a lengthier approach of using a lambda and assigning it to date_parser argument. So my doubt is whether while following the later approach, do we basically apply a function(lambda in this case) to a column('Date') so that the function is applied to each value in that column and it converts each value(String) by calling strptime() ? If so, can we assume to_datetime() method converts a whole Series to datetime type while the later approach converts each value of a Series to datetime object? Is there any way we can replicate the former way while loading the csv file? That look minimal and easy. Thanks in advence
Hope you all are all staying safe! In this video we'll be learning a lot about working with dates and time-series data in Pandas, and also look at doing some basic plotting. In the next video, we'll be learning how to load in data to Pandas from different (Excel, JSON, SQL, etc). Let me know if there is anything else you'd like me to cover in the Pandas series. I will likely be taking a break from this series after the next two videos are released just so I can focus on some different topics.
Hey Corey
Will you be able to cover some machine learning libraries like scikit learn,tensorflow,...?
Because matplotlib and pandas that you have covered so far is related to machine learning and we can put this libraries into practical
@@kaaaaasraa49563 I'll do my best. It's a very large topic to cover, so I've been a bit intimidated to tackle such a large course, but I know many people are wanting that, so I'll do my best to start working on something soon
Hey guys checkout dfcleaner package on pypi
Hey Corey, i was wondering if you could help do a complete EDA on one dataset ,kaggle or from anywhere in which you could cover the entire operations that we've learnt in one complete tutorial video with like a real world dataset. That could be like the end of this series video or something.
Please help with numpy
I bless the day I came across your channel
3:50 convert to datetime using to_datetime
7:45 parse dates while loading data from csv
11:20 using day_name on whole series using dt class
12:20 create column of dayname
13:20 min and max methods on datetime series
15:00 filtering by dates as string
17:20 filter by to_datetime
18:40 set date as index
19:21 filter data by just passing the date in brackets
20:05 using a slice to get specific dates data
21:00 calculating average of a slice(timeframe)
22:05 getting max value of a column on a given day
23:50 resampling(breaking down by days) a whole column into a new variable
27:00 plotting with matplotlib in pandas
28:28 resampling df with multiple columns
30:55 using agg to apply different function on different columns while resampling
Would it be possible for you to tell me how can I download ETH_1h.csv file? I found just row data on bit.ly/Pandas-10. Thanks in advance.
😊
Wow! Thk you very much! I appreciated a lot!
For all those getting the following error: *FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime module instead.*
Just do this:
*from datetime import datetime*
and change the *pd.datetime.strptime()* to:
*datetime.srtptime()*
and it should work the same.
I used the following
import datetime as dparser
d_parser = lambda fn : dparser.datetime.strptime(fn, '%Y-%m-%d %I-%p')
this also worked. is there any advantage of your method over mine ?
@@mrrohitnayak1 its the same thing, u just named datetime as dparser, His code would call datetime.datetime.strptime() and yours call dparser.datetime.strptime()
@@alenjose3903 any advantage in using his method over mine or vice versa
@@mrrohitnayak1 no its the same, u can name it anything you want.
Consider changing 'datetime.srtptime()' to 'datetime.strptime()'
To anyone getting the message that 'date_parser' is deprecated and will be removed in a future version...
You no longer need to pass in a function, just the date string formats in the argument date_format='%Y-%m-%d %I-%p'
example:
df = pd.read_csv('ETH_1h.csv', parse_dates=['Date'], date_format='%Y-%m-%d %I-%p')
I learn through this video in 2023 as a new entry-level data analyst and non-native English. It's still relevant and lite but packed, especially for the `.resample()` method. It saves my time compared to using `.groupby()`. In addition, we need to keep in mind indice filtering. It's better to use `.loc[]` to avoid the deprecation issue. Thank you, Corey Schafer. Your channel is such a gem!
For anyone getting error in the first part of the video, this is what worked for me
d_parser = lambda x: pd.to_datetime(x, format='%Y-%m-%d %I-%p')
df = pd.read_csv('data/ETH_1h.csv', parse_dates=['Date'], date_parser=d_parser)
or pass the date format directly
df = pd.read_csv("data/ETH_1h.csv",parse_dates=['Date'],date_format = "%Y-%m-%d %I-%p")
@@rafabws for dfx.loc['2020-01':'2020-02'] , I get KeyError: 'Value based partial slicing on non-monotonic DatetimeIndexes with non-existing keys is not allowed.; What to do with it? plz help
@@user-testo21347 I was getting same error too and I believe I found the solution. For some reason our indexes are not in order. For this reason, if we sort our indexes by writing the code dfx.sort_index(inplace=True) before writing the code dfx.loc['2020-01':'2020-02'], we get rid of the error.
@@user-testo21347 I was getting same error too and I believe I solved it. For some reason our indexes are not in order. For this reason, if we sort our indexes by writing the code dfx.sort_index(inplace=True) before writing the code dfx.loc['2020-01':'2020-02'], we get rid of the error.
@@user-testo21347 I too got this error. if you have solution pls share it with me
Also to get the mean for resampling, the default for the mean function was changed. So to get the weekly resampling at around 30:05 of the video:
df.resample('W').mean() will give an error
do this instead:
df.resample('W').mean(numeric_only=True)
where to download this time series data please
@@sahilmehra290 you can find in his github link
While there are already other excellent pandas tutorials on TH-cam your videos IMHO stand out for their carefully crafted examples, clarity of explanation, information density and relevance to practical applications.
I'd really love this series to be continued and cover some of the more advanced topics.
Hi Corey, thank you so much for taking the time and putting this series together. This is one of the most structured series on panda on the web. I feel confident in pandas after watching the series. We would love to see something similar on the NumPy. Please, Corey...put together a Numpy series for us.
Yesterday i was searching for any tutorial that explains dates and times in pandas and i couldn't find a good material
Thanks man
I want to point out something. I like your videos. There is nothing boring about them. But I do not have much time so I watch them at 2x. Your normal talking speed is not slow but very clear so I can easily understand you at 2x too.
Starting at 8:02
You may use this instead
*df = pd.read_csv('data/ETH_1h.csv',parse_dates=['Date'], date_format = '%Y-%m-%d %I-%p')*
The concept of resample is so damn crazy!!! I struggle so much in other systems with this kind of aggregation and it's easy like nuts in pandas...
Corey, you are great!
this whole series is so beautifully explained that if one is little bit aware of python and its syntax whole series can be watched without volume .
This is the only channel I can’t wait for the new video to drop. Two weeks and waiting but who’s counting. ☺️
20:32 if you are getting error here then you first have to sort the index and use loc:
# Sort the index
df = df.sort_index()
# Select data between January and February of 2020
df.loc['2020-01':'2020-02']
Thanks
thx, saved my time
Thank you. It's really worked.
Since I stumbled on Corey Schafer videos, it has become my favourite TH-cam Channel
These are the most developed, best explained videos on how to use pandas - thank you Corey for putting these together!
if you get this error (KeyError: 'Value based partial slicing on non-monotonic DatetimeIndexes with non-existing keys is not allowed.') try sorting your data before slicing.
df = df.sort_index()
THank you so much. After this, can you slice one day of data? like at 22:44 df["2020-01-01"]? I got this error: KeyError: '2020-01-01'. I also tried df.loc["2020-01-01"], same error.
Thanks brother 👍
@@looklook6075 same here
@@looklook6075 same here
@@looklook6075 try df["2020-01-01:2020-01-01"]
I loved the point Corey made about how much easier it is to use date as your index, however while following along I got a warning this warning:
FutureWarning: Indexing a DataFrame with a datetimelike index using a single string to slice the rows, like `frame[string]`, is deprecated and will be removed in a future version. Use `frame.loc[string]` instead.
Very, very, very good. The accurate and straightforward explanations are supported by great examples. Big thanks.
Gave a thumbs up before even started watching the tutorial..and I'm not disappointed at all. Keep up the good work Corey!!!
BTW you are the only TH-camr getting likes from me.
I always give you a thumbs up before i even watch the video!
Will you introduce us to machine learning in the near future?
Thank you sir. This is the best explanation out of all the videos I've watched!
Thanks a lot for this whole series Corey!! I've watched 3/4 other tutorials between youtube and other platforms and yours were the best by far, especially due to the fact that I'm from Italy so it was easier to follow your explanations step by step
I can't wait for your plotting with Pandas tutorials too, those are gonna be pretty useful! Have a nice day a stay safe ;)
Right now I'm learning flask from your playlist and it's freaking amazing!
Hi Corey..I have referred to various resources to learn Pandas properly on TH-cam. And your tutorials are the best I could find. And looking forward to learn more from your matplotlib tutorials. Thanks for creating top class content!!
You taught me pandas completely free of charge.
Thank you, sir.
Please show more *errors and its corrections* for future vids.
The fixes are really beneficial!
Thanks for recording, editing, uploading and sharing! 👍
This Video was sent by the Data Science Gods. Thank you so much
I really appreciate your detailed explanation on the errors, which help us to select the right corrective action to take. Love your videos.
My man Corey on youtube dropping bangers.
Thanks man. Amazing content.
Thank you Corey for the wonderful tutorials. No paid courses can come close to the way you present the depth of the concept. Thank you for making it free and a big thank you for keeping up the good work.
I watched this video when the likes for this video are 999 and I hit like right away to make it 1K, because I believe Corey's content!
There should be a mega-like button for this. Thank you so much!
I learnt so much from this single video. Thank you Corey
I'm happy that I found a Awesome channel to learn Python for Data Science Easily!!!! Very Nice and Crystal Clear Explanation!!👌🙂
Hi Corey! Your series on pandas is really informative and helpful. Can you make a similar series for numpy? I believe that's a topic that will be of interest to many of your viewers.
After using d_pars = lambda x: datetime.strptime(x, '%Y-%m-%d %I-%p') i got this warning: FutureWarning: The pandas.datetime class is deprecated and will be removed from pandas in a future version. Import from datetime module instead. I mean, i still get correct output , but if u wanna remove this warning just do: from datetime import datetime then d_pars = lambda x: datetime.strptime(x, '%Y-%m-%d %I-%p').
Treat it like curiosity or sth
Btw, great videos, sir!
This is the best episode from the series, well done
I used this video to backtrack over some material that I wasn't sure of. This was a great video as is your other instructional videos. Thanks so much.
great job Corey, I was going to go through Wes McKinney's book but looking at the pandas section I realized that I already know most of it by now form your series. Pls do the data plotting soon! ;)
32:28 You could have also used method 'first' for open price within agg function
Thanks a lot for these videos.
Exactly what I was thinking.
Thank you Sir for sacrificing your precious time to teach the World.
We appreciate you .
l Appreciate You.
Your tutorials are the bests on the web. Before I can do the index filter (19:35 in the video), I had to sort the dataframe by index(df = df.sort_index()) I don't know why but it was given me an error (AssertionError: ).
Thanks for sharing.
Well done! I didn't know that I can pass in a dict in agg. Always passed in a list and then selected the column-aggregation tuples that I needed, which is quite resource intensive and hard to read. Also I did not know I could make datetime comparisons to a string. Thank you.
Extremely useful. Thanks for the video Corey! You have my gratitude.
Pandas is so easy to work on after seeing this vid.
Wish my coaching teacher taught me this simpler way
Very well-explained. Thanks for this. I am requesting you to make a video on multi index in Pandas.
That's great Corey !!
The best lesson about Time series, ever.
Best tutorial series for pandas ever❤❤❤
You are a great teacher, many thanks for putting in all the effort and time in creating such a great series :)
Your every video is soo nice and perfect. The explanation is very friendly and in a proper sequence. Your channel works for me as a one-stop-shop. Thank you soo much man.❤️❤️
Love from India 🇮🇳
OMG!!!
You are awesome Corey.
I learned a lot from your videos!
Another great video Corey. Keep up the good work!!!
If anybody gets an error at 10:20 , use date_format= '%Y-%m-%d %I-%p' instead of date_parser = d_parser
thanks so much.
thank you
so what is the use of the lambda express on the line above then ??
I really like how you explain things so clearly. Super helpful for my project. Thank you so much!
As usual your videos are great ! You explain things very clearly. Please start videos on Machine Learning and Artificial Intelligence soon. Earnest request !
Hi Corey, much appreciated for your video, just excellent for a beginner like me!
But I have a question here:
for the resampling of the whole DataFrame, say, if I want to get the open price at 00:00:00 for the 'open price' of the day and the close price at 23:00:00 for the 'close price' of the day, what function am I supposed to use here?
Thank you !
Same here i have been stuck here
You're my savior I was searching for video as this thank you.
corey man, i feel obligated to say thank you , aaaaaaaaaaaaaaaaaand a big well done sir.
All I can say you are awesome. The way you explained 😍✌️✌️✌️✌️. You have the best play list for pandas on web. Your efforts are truly appreciated 👍👍👍👍👍👍
This was an absolute masterpiece!! Thank you so much. I hope you are good and safe
pandas plotting will be useful and intesting! looking forward to it! Thanks Corey!
Corey this series is fantastic, thank you!
just 11 minutes in the video, i hit the like button as hard as i can !!
This has been beyond useful! Thank you.
Hei, welcome to my playlists too, Python and R, downloadable source files (see video description)
Fantastic series! Thank you for providing this awesome content. I have been using Pandas quite a bit lately to process data, one thing I am struggling with and would really find helpful would be a video on taking these Pandas notebooks and functions and converting them into actual Python apps that can be reused on other projects. Pandas can really be like magic sometimes and being able to pass data into a Python app that could apply these functions would be extremely powerful, I have yet to find any good tutorials or reading on this subject, one from the master would be greatly appreciated!!
Values the time of viewers. Not sure if you really speak so fast or just increase speed in the first and last few mins of video.
I just like the way it goes. Thanks for the great tutorial.
Loving this series and have learned a lot. The prices of ETH in this video, wow!
டேmn true. $200 vs $4900 at peak and $1700 now
Great job Corey! Thank you!
For those getting an error for accessing the datetime index, you need to sort it first now. So instead of df['2020'] make it df.sort_index()['2020'] and the same for slicing it
Yes!, or for accessing the datetime index you can also put df.loc['2020'] but this not work for slicing!
Oh Corey, you legend! Thanks mate, this is some great stuff.
Fantastic series on Pandas. I have no words to express how i am thankful to you for this series. You are a great teacher. I am yet to check on the 11th video on this series and up next to the Matplotlib series (definetely). You are awesome. A request, could you please plan to do Numpy series as well.
Let me echo what plenty of others have said -- your videos are exceptionally well done and very understandable!! I knew virtually nothing about Python about two months ago and now I can do plenty of rudimentary, but useful stuff with it.
In working through this tutorial with data that I've exported from ThingSpeak, I get to the point of having pandas successfully (apparently) converting the str by using:
df['created_at'] = pd.to_datetime(df['created_at']). I get no errors and it seems to recognize the datetime group automatically without providing formatting schema.
However, when I try to prove that it's now a datetime via the .day_name() method it throws an error and tells me my attribute is a datetime.datetime object and doesn't allow me to use that method.
I get the same results if I convert the created_at column via the read_csv() method at the initial import.
From forum searches I'm concluding that it got converted differently, than in your example, into a datetime object. I don't totally understand what that means, but if I try different methods on those objects, I can manage to arrive at the same destination, which is to return the day of the week for a given date. I used this strftime() method:
print(datetime.strftime(df.loc[20001,'created_at'],'%A'))
IDK if you're still responding to comments on a 2 yo vid, but in case others ran into this problem, that's how I solved it. I'd be great to have a video on the datetime library (module ?) and how that's different and how to best use it. Maybe you've already done that, so I'll look for that next in your extensive list of videos, but if not, just a suggestion.
Again - thanks for all the amazing education efforts!!!
cheers
lol -- and of course I now see a tutorial on this: Python Tutorial: Datetime Module - How to work with Dates, Times, Timedeltas, and Timezones.
checking that out next!! 🤓
I got an error with the lambda function trying to parse the dates while loading from csv. The error stated pandas no longer supports pd.datetime. After checking the comments I tried @larc99's suggestion but still got an error as it was expecting a string rather than a function for the format. What gave the same output and I think is much simpler (without the lambda function) turned out to be:
df = pd.read_csv('data/ETH_1h.csv', parse_dates=['Date'], date_format='%Y-%m-%d %I-%p')
Thanks Corey!
Your videos are always lit!
😁✌️
It does not work by slicing the df using date such as df["2020-01":"2020-02"] at 21:40. I got this error "TypeError: cannot do slice indexing on RangeIndex with these indexers [2020-01] of type str". Anyone experienced the same? Pandas datetime is sooooo annoying and not intuitive at all.
Hey Corey, Shall we wait for NumPy series as well after this ?
Great! I love your lectures.
How to solve error 'Value based partial slicing on non-monotonic DatetimeIndexes with non-existing keys is not allowed.' while using df['2020-01': '2020-02'] at timestamp 20
:17
Thanks a lot for this pandas series Corey ♥️.
Great tutorial! Question: How would be able to grab the last trade close price for the week in the agg demonstration? Doesn't make sense to average the daily close as the weekly. The close is the close.
Wondering that same thing, likewise for 'open' and the first value of the newly resampled period.
What a detailed video !! Thank you man !!! 🤝
This will be the video I am looking for. I am pretty sure. LET IT BE!
Pretty mad how much I failed at plotting panda dataframe in matplotlib sorted by month.
We have a german saying, „taking a detour increases the knowledge of your surroundings“.
From 0% to 5% domain knowledge in multiple hours:/
Hi Corey, thank you so much for videos, you are doing an amazing job.
32:35 I got this error 'DataFrame' object has no attribute 'resamlpe'
okay it worked, just reloaded the page )
Amazing tutorials man ! thank you so much
Awesome stuff as always
You are the best teacher. period
At 20.08, calling df['2019'] didn't work for me. Gave an 'Assertion Error'. df.loc['2019'] worked though.
Same here !
@@nesrinebelkadi7702 print(df.loc['2019'])
I like all the videos u upload and u are the reason I started development and became fond of python. Can u please make some videos on how to write unit testing in python?
Hey there. Glad you like the videos! I do have one unit testing video you can find on my channel, but I do plan on doing more in the future
Man...you really rock! Thanks a lot for that!
Thank you thank you thank you, Finally found the help that i needed!
And yet another great lesson. Thanks
again and again awesome information
Hello.
am using Jupyter Notebook on Ubuntu 20.04 LTS, my date and time in the setting is showing: 14 June 2020, 13: 48 and on the desktop its showing: Sun Jun 14 13:48:25 which format can i use?
Hi, New to Pandas and just a bit confused:
Firstly at 6:36 , we converted all the values of the Series to datetime type from String type using:
df[‘Date’] = pd.to_datetime(df[‘Date’], format = ‘%Y-%m-%d %I-%p’)
but when doing the same thing while loading csv 10:24,
we used a lengthier approach of using a lambda and assigning it to date_parser argument.
So my doubt is whether while following the later approach, do we basically apply a function(lambda in this case) to a column('Date') so that the function is applied to each value in that column and it converts each value(String) by calling strptime() ?
If so, can we assume to_datetime() method converts a whole Series to datetime type while the later approach converts each value of a Series to datetime object?
Is there any way we can replicate the former way while loading the csv file? That look minimal and easy.
Thanks in advence
Thank you again Corey!!
This is so in depth I love it thank you!