perfect starting point into this fascinating sphere! 👌looking foreward to see deeper analysis of the data you scraped (maybe social network analysis or user sentiment analysis). 🙃
It was very helpful, but can you also make a video which states how to retrieve reply from a particular tweet, couldn't find such video on youtube..thanks
Brilliant! I would like to ask when we iterate the tweets object (in the line "for tweet in tweets"), it can easily reach the rate limit. How should we solve this problem?
Is there a way I can include 2 hashtags like : keyword = '#food' AND '#drink'? Or perhaps to get a specific location from a particular hashtag? Thanks :)
Thank you for the amazing videos with step by step guidance. Post the update of twitter api to 2.0, when i get the tweets, the retweets are truncated to 140 characters, can you suggest me a workaround
excellent stuff for researchers and analysts but i have a problem i want to extract replies on tweet without using tweepy? how is it possible please save my time and help us
What can be the max value you can give for limit in tweets = tweepy.Cursor(.....).item(limit) ? ALso Can we add a since date and until date for this method?
Maximum number of tweets depends on the method you pass to Cursor. For api.user_timeline, the maximum is 3200, and for api.search_tweets I don't think there is any particular limit of number, but I know that you cannot get tweets older than 7-days. You can also add since and until dates, but again have on mind that you only can consider for the last 7-days. If you want for older than that, create 30-days dev environment, and then use api.search_30_day method.
Hello, this is such an incredible video. Would it be possible to get tweets from a specific user with a particular hashtag? I would like to know if there is a method to combine one filter with user and hashtag. Thanks in advance for your reply.
Thanks for the tutorial. Is it possible to gather historical all tweets with a specific hashtag (for example global warming) from 2000 to 2022? In other words, is there any limit to collecting the number of tweets for a specific hashtag ?......Thanks
when we extract the names by tweet.user.name it is not the actual name of the user who tweeted on specific keywords the name is of a random person who retweeted. how to extract the name of actual person who tweeted?
Great video! I have a question about reading tweets. Do you need Elevated access in order to just read twitter id's off of twitter? Even if its tweets from a specific user?
As someone with no programming background trying to collect tweets and respective data for analysis, these videos have helped tremendously. When collecting tweets from a specific user, is there any way to only get tweets before a specific date (for ex, 100 tweets from user "X" before February 10th)? Thank you very much for your work!
Unfortunately it's not possible to directly filter user tweets based on dates. But what you could do it to get the tweet_id attribute of a tweet on the date that you want (like Feb. 10), then pass it to the function user_timeline as "max_id".
i'm having an issue with retrieving tweets with hashtags. the following error message showed up when i run the file. please help. Traceback (most recent call last): File "c:\Users\User\Documents\Tugas Akhir\analisis sentimen\twitter_data_users.py", line 50, in print(df) File "C:\Program Files\Python310\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode characters in position 105-106: character maps to
Thanks the video is very helpful. Can you also tell me how can we get tweets from a specific user and containing a particular keyword? Or maybe after getting all the tweets from a specific user, how can we filter it so that it returns all tweets containing the keyword?
@@aispectrum Thank you. Is there a way to filter the results while calling the data from the api using tweepy.Cursor(). Because i wanted all 3000 tweets that are extracted should have those keywords.
Thanks for your video! I learned a lot from it. However, I encountered a problem with the code. I did exactly as your video taught, but when trying to get data by user, the code "tweets = api.user_timeline(screen_name=user, count=limit, tweet_mode='extended')" return error 401 unauthorized. But I followed your instructions from your last video to get public_tweets, it worked perfectly. Are there any solutions about the error? Thank you very much!
Great video!! Thanks so much!! Quick question if I want to add the public_metrics (retweet count, reply count, like count and quote count) where should I add it? because I tried in the data.append but it says my attribute is empty... Could you help me with this? :) TIA!
In for tweet in tweets:, you should be able to get those attributes by, for example, writing tweet.retweet_count, tweet.favorite_count. I just tried it and could get access to those attributes.
Bro please make a video on “How to use Twitter developer account for beginners or differences between normal Twitter and Twitter developer” 🤟🏻🤟🏻🤟🏻🤟🏻🤟🏻🤟🏻🤟🏻🤟🏻🤟🏻🤟🏻🤟🏻
At the moment the only solution that comes to my mind is to loop over the uses and get their tweets, then append all the tweets to the data list, and make the dataframe. Then just simply save the dataframe into a csv file with the to_csv method.
I wouldn't say you should only use one. It is recommended to use v2 if you can. But if there are methods that are only available with v1.1, then you wouldn't have other option.
Thank you for this, was very helpful. The tweet_mode = 'extended' parameter is not working for me and I only return tweets up to 140 character in length, have you experienced this before?
Based on the tweepy's documentation, when you get the tweet from the full_text attribute, it should always return the full tweet. Are you getting the tweets from this attribute?
This possible by writing the search the query as “Word (from:Username)”. You can even use advanced search of twitter website to search for more complex criteria.
If you want get tweets older than 7 days, create a 30-days dev environment on twitter developer website, and then use api.search_30_day method in the python code.
Hi, thank you for the video, but there are two things I don't understand. I played around and searched for different hashtags and I'm getting results but the one hashtag I need says 'Empty data frame' while 100% this hashtag is used on twitter. Really weird! Also how would you request a date for each tweets? (ay, month, year) and country? I entered 'tweet.date' and 'tweet.country' but its not identifying it. Please help :(
Unfortunately I don't have a solution for the first questions. But for the second one, you can pass for example until="2022-03-19" to get tweets older than this date(just remember that you can only get 7-days old tweets). Another way is to pass since_id to get older tweets or max_id to get tweets more recent tweets. For country, try the geocode parameter to give the location of the users (pass latitude,longitude,radius)
Hi Daniela, I think you mean you'd like the date as a column next to each tweet? I found a way to do this by copying the 1st video from this channel, where he requests three columns, time, user and tweet text - and changing this from the two columns requested here.
When i try to export this to excel, following error occurs: ValueError: Excel does not support datetimes with timezones. Please ensure that datetimes are timezone unaware before writing to Excel. Is there any good way to treat date data from your tutorial codes?
how do you extract hashtags from a specific user, eg say i want to extract elon musks tweets on #dogecoin only, it would be great if you could help, Great Video.
Great video and explanation! But whenever I try to execute your script I get the error "tweepy.errors.BadRequest: 400 Bad Request 215 - Bad Authentication data." even though I have elevated access to the Twitter API and have used simpler scripts for Tweepy before that worked well. Does anyone know what might be the issue?
The most common solution to this problem is regenerating API keys and access tokens. Also just in case, make sure that you are using the latest version of tweepy.
@@aispectrum Thank you for the super quick reply. Unfortunately my API Keys and access token work if I run other scripts and I do have the latest version of tweepy 😕
hi! I try to code and there is an error msg showing "Unauthorized: 401 Unauthorized" in "for tweet in tweets: data.append([tweet.user.screen_name, tweet.full_text])" How can I fix it? Thanks !
This error is related to the credentials of your account. Double check your api keys, and if they are correct, the app should connect to the API with no problem.
For the search method, there is a 7-day limit. So you cannot get tweets older than a week. There is also a 30-day search limit that you can use. For this, you should create a dev environment on the twitter developer website for "search tweets: 30-day". Then in you program, use search_30_day method. If you want older results, you should apply for Academic Research access on the twitter developer website, and then use search_full_archive method.
@@agam6057 I don't think there is another way. As far as I know to get access to the full tweet archive Twitter requires you to be a student or part of an academic institution.
The following error occurs: PS C:\Users\Usuário> & C:/ProgramData/Anaconda3/python.exe c:/Users/Usuário/Desktop/tweepy_tests/twitter_data_users.py Traceback (most recent call last): File "c:\Users\Usuário\Desktop\tweepy_tests\twitter_data_users.py", line 9, in api_key = config['twitter']['api_key'] File "C:\ProgramData\Anaconda3\lib\configparser.py", line 963, in __getitem__ raise KeyError(key) I also tried to move the config and twitter_data files to the same path as 'configparser.py', but the error continues. Would you know how to solve it, please?
This is because the python code cannot find the config file. Make sure that the files are in the same directory, and it should solve the issue. If didn't, the easiest option is to just directly put your keys in the python file. Something like api_key = "paste the key here"
anyone knows why he can get 100 queries but when I do it I only got 20 queries max? *edit Nvm, i followed the video and it works. And I think theres a limit for 1500 tweets that i read from somewhere
Is your config file in the same folder of the python code? If not, that could be the cause of this error. Other than, you can directly type the credentials in the python file, something like api_key = "xxxxxxxxxxx"
I love you, really. I need this methodology for my Ph.D. and you've saved my life. Really, Thank you!
A video specifically downloading and analyzing a large number of past tweets related to a specific hashtag would be very helpful.
Downloading then as Csv file
perfect starting point into this fascinating sphere! 👌looking foreward to see deeper analysis of the data you scraped (maybe social network analysis or user sentiment analysis). 🙃
very useful. I am waiting for your next vedio.
It was very helpful, but can you also make a video which states how to retrieve reply from a particular tweet, couldn't find such video on youtube..thanks
Brilliant! I would like to ask when we iterate the tweets object (in the line "for tweet in tweets"), it can easily reach the rate limit. How should we solve this problem?
Thanks for your Great Video!!! Just WoW !! I am really grateful to you 🤩
Sir kindly please tell the tool names in the starting so that students like me (new) can start easily. Thanks
Is there a way I can include 2 hashtags like :
keyword = '#food' AND '#drink'?
Or perhaps to get a specific location from a particular hashtag?
Thanks :)
nice tutorial, i will apply some deep learning models such as sentiment analysis on this data! thank you!
I am really glad it was helpful!
Learning alot from ur videos.... Can u make a video explaining attributes of tweet, like city of the user, followers count, and all...
Hi! great video, but how could I exclude all the retweets?
Thank you for the video!
Thank you for the amazing videos with step by step guidance. Post the update of twitter api to 2.0, when i get the tweets, the retweets are truncated to 140 characters, can you suggest me a workaround
Quick questions, how to get when is the tweets created for user timeline?
excellent stuff for researchers and analysts but i have a problem i want to extract replies on tweet without using tweepy? how is it possible please save my time and help us
can we search tweets in other language ?? do we have any option ?
awesome video :) many thanks
What can be the max value you can give for limit in tweets = tweepy.Cursor(.....).item(limit) ? ALso Can we add a since date and until date for this method?
Maximum number of tweets depends on the method you pass to Cursor. For api.user_timeline, the maximum is 3200, and for api.search_tweets I don't think there is any particular limit of number, but I know that you cannot get tweets older than 7-days. You can also add since and until dates, but again have on mind that you only can consider for the last 7-days. If you want for older than that, create 30-days dev environment, and then use api.search_30_day method.
Great video - thank you for sharing. Is there any way this method can be used to download all tweets of a user?
Struggling with how to download the tweets from the list so as to clean them
very nice
Can you develop a hashtag recommendation system for twitter in python using twitter datasets ?
Hello, this is such an incredible video. Would it be possible to get tweets from a specific user with a particular hashtag? I would like to know if there is a method to combine one filter with user and hashtag. Thanks in advance for your reply.
Would you consider this a web scraper? Since Twitter does not allow web scrapers.
after introduction to version 2, this one seems not to work
Thanks for the tutorial. Is it possible to gather historical all tweets with a specific hashtag (for example global warming) from 2000 to 2022? In other words, is there any limit to collecting the number of tweets for a specific hashtag ?......Thanks
when we extract the names by tweet.user.name it is not the actual name of the user who tweeted on specific keywords the name is of a random person who retweeted. how to extract the name of actual person who tweeted?
Can you create a similar one for Facebook especially on getting certain keywords
Great video! I have a question about reading tweets.
Do you need Elevated access in order to just read twitter id's off of twitter? Even if its tweets from a specific user?
Thanks! Actually you don't necessarily need elevated access. To do it, you can use the Client.get_user method to get the ids.
@@aispectrum Thank you lots! Additionally, when you u search tweets by keyword, how can you retrieve the usernames from the tweets?
@@Chill-yv4gg I think you can use tweet.user.screen_name for each tweet.
As someone with no programming background trying to collect tweets and respective data for analysis, these videos have helped tremendously. When collecting tweets from a specific user, is there any way to only get tweets before a specific date (for ex, 100 tweets from user "X" before February 10th)?
Thank you very much for your work!
Unfortunately it's not possible to directly filter user tweets based on dates. But what you could do it to get the tweet_id attribute of a tweet on the date that you want (like Feb. 10), then pass it to the function user_timeline as "max_id".
@@aispectrum How would that work if I want all tweets from let's say 2019 to 2020?
i'm having an issue with retrieving tweets with hashtags. the following error message showed up when i run the file. please help.
Traceback (most recent call last):
File "c:\Users\User\Documents\Tugas Akhir\analisis sentimen\twitter_data_users.py", line 50, in
print(df)
File "C:\Program Files\Python310\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 105-106: character maps to
how you can get tweets from the year 2020 to the year 2022
Thanks the video is very helpful. Can you also tell me how can we get tweets from a specific user and containing a particular keyword? Or maybe after getting all the tweets from a specific user, how can we filter it so that it returns all tweets containing the keyword?
As you said it yourself, you can filter the results based on any keywords you like. You can try df[df['Tweet'].str.contains("hello")]
@@aispectrum Thank you. Is there a way to filter the results while calling the data from the api using tweepy.Cursor(). Because i wanted all 3000 tweets that are extracted should have those keywords.
@@awkwardak4818 Unfortunately I'm not aware of any other way. Specially because the API doesn't allow you to filter the tweets when calling the data.
absolutely you are 👌
Hello, is the tweeter API still accessible for free ?
How to pass both user name and keywords together to get the tweets posted by specific user containing a particular list of keywords?
You can do by change the keyword variable to :
keywords = 'from:username keywordYouWantToSearch'
Thanks for your video! I learned a lot from it. However, I encountered a problem with the code. I did exactly as your video taught, but when trying to get data by user, the code "tweets = api.user_timeline(screen_name=user, count=limit, tweet_mode='extended')" return error 401 unauthorized.
But I followed your instructions from your last video to get public_tweets, it worked perfectly. Are there any solutions about the error? Thank you very much!
Great video!! Thanks so much!!
Quick question if I want to add the public_metrics (retweet count, reply count, like count and quote count) where should I add it? because I tried in the data.append but it says my attribute is empty... Could you help me with this? :)
TIA!
In for tweet in tweets:, you should be able to get those attributes by, for example, writing tweet.retweet_count, tweet.favorite_count. I just tried it and could get access to those attributes.
Bro please make a video on “How to use Twitter developer account for beginners or differences between normal Twitter and Twitter developer”
🤟🏻🤟🏻🤟🏻🤟🏻🤟🏻🤟🏻🤟🏻🤟🏻🤟🏻🤟🏻🤟🏻
Hello,
thankyou so much for this video!
Would it be possible to extract this data into csv using pandas?
Sure. try out "df.to_csv('tweets.csv', index = True)"
@@aispectrum thankyou!
Is there a way to scrape tweets from multiple users at once? Then, ideally being able to download them as a single CSV file? Thanks!
At the moment the only solution that comes to my mind is to loop over the uses and get their tweets, then append all the tweets to the data list, and make the dataframe. Then just simply save the dataframe into a csv file with the to_csv method.
Good
Really Helpful. Much appreciated.
Would it be possible to extract these printed outputs into a csv or an excel document further and how?
Thanks.
Why would we use the API1.1 interface and not 2 from tweepy?
I wouldn't say you should only use one. It is recommended to use v2 if you can. But if there are methods that are only available with v1.1, then you wouldn't have other option.
Thank you for this, was very helpful. The tweet_mode = 'extended' parameter is not working for me and I only return tweets up to 140 character in length, have you experienced this before?
Based on the tweepy's documentation, when you get the tweet from the full_text attribute, it should always return the full tweet. Are you getting the tweets from this attribute?
@@aispectrum Thank you for the reply! I am using the full_text attribute in addition to the tweet_mode = 'extended' parameter. Still not working.
@@jayyee3790 That’s odd. Frankly I cannot think of a solution for this unless maybe there’s a mistake in your code, or tweepy isn’t updated.
Hola, como seria el caso de querer buscar Tweets que mencionen un Hashtag? y no por usuario. Gracias
En el apartado de q='#tuhashtag'
Can we search for multiple keywords at the same time ? like (username + good ) and how can do this
This possible by writing the search the query as “Word (from:Username)”. You can even use advanced search of twitter website to search for more complex criteria.
how can i scrape the data earlier? tweepy using api merely get data last 7 days
If you want get tweets older than 7 days, create a 30-days dev environment on twitter developer website, and then use api.search_30_day method in the python code.
what is the name of your platform that coding in it?
Visual studio code is one of the best
By searching keywords,, how can I get the number of follwers of the people who tweeted the keywords searched
You can pass the username of each account to api.get_followers to get the followers.
Hi, thank you for the video, but there are two things I don't understand. I played around and searched for different hashtags and I'm getting results but the one hashtag I need says 'Empty data frame' while 100% this hashtag is used on twitter. Really weird! Also how would you request a date for each tweets? (ay, month, year) and country? I entered 'tweet.date' and 'tweet.country' but its not identifying it.
Please help :(
Unfortunately I don't have a solution for the first questions. But for the second one, you can pass for example until="2022-03-19" to get tweets older than this date(just remember that you can only get 7-days old tweets). Another way is to pass since_id to get older tweets or max_id to get tweets more recent tweets. For country, try the geocode parameter to give the location of the users (pass latitude,longitude,radius)
Hi Daniela, I think you mean you'd like the date as a column next to each tweet? I found a way to do this by copying the 1st video from this channel, where he requests three columns, time, user and tweet text - and changing this from the two columns requested here.
When i try to export this to excel, following error occurs:
ValueError: Excel does not support datetimes with timezones. Please ensure that datetimes are timezone unaware before writing to Excel.
Is there any good way to treat date data from your tutorial codes?
Have you tried df.to_csv() instead?
how do you extract hashtags from a specific user, eg say i want to extract elon musks tweets on #dogecoin only, it would be great if you could help, Great Video.
I think it's better that you get the tweets of the user that you want. Then just filter based on the keyword that you want.
Great video and explanation! But whenever I try to execute your script I get the error "tweepy.errors.BadRequest: 400 Bad Request
215 - Bad Authentication data." even though I have elevated access to the Twitter API and have used simpler scripts for Tweepy before that worked well. Does anyone know what might be the issue?
The most common solution to this problem is regenerating API keys and access tokens. Also just in case, make sure that you are using the latest version of tweepy.
@@aispectrum Thank you for the super quick reply. Unfortunately my API Keys and access token work if I run other scripts and I do have the latest version of tweepy 😕
@@lisanneburghardt1746 See if this helps: github.com/duttashi/scrapers/issues/21
I get same issue as well, any solution to this ?
hi! I try to code and there is an error msg showing "Unauthorized: 401 Unauthorized" in
"for tweet in tweets:
data.append([tweet.user.screen_name, tweet.full_text])"
How can I fix it? Thanks !
This error is related to the credentials of your account. Double check your api keys, and if they are correct, the app should connect to the API with no problem.
i cannot use tweepy, i write and nothiung happens in 'import tweepy'
Do you mean that you only write “import tweepy” and nothing happens? Have you installed tweepy using
pip install tweepy
@@aispectrum yup, now i install tweepy in cmd
Getting error like this
KeyError: 'twitter'
Any suggestions how to solve this
It's not clear why you are getting this error. There should be more information in the console that can be helpful to find the solution.
@@aispectrum it would be helpful to understand "twitter" within the logic?
How old tweets can we extract using this code
For the search method, there is a 7-day limit. So you cannot get tweets older than a week. There is also a 30-day search limit that you can use. For this, you should create a dev environment on the twitter developer website for "search tweets: 30-day". Then in you program, use search_30_day method. If you want older results, you should apply for Academic Research access on the twitter developer website, and then use search_full_archive method.
@@aispectrum Is there any way to get the older tweets without academic research access?
@@agam6057 I don't think there is another way. As far as I know to get access to the full tweet archive Twitter requires you to be a student or part of an academic institution.
@@aispectrum Thank you for explaining, it's very useful for me
The following error occurs:
PS C:\Users\Usuário> & C:/ProgramData/Anaconda3/python.exe c:/Users/Usuário/Desktop/tweepy_tests/twitter_data_users.py
Traceback (most recent call last):
File "c:\Users\Usuário\Desktop\tweepy_tests\twitter_data_users.py", line 9, in
api_key = config['twitter']['api_key']
File "C:\ProgramData\Anaconda3\lib\configparser.py", line 963, in __getitem__
raise KeyError(key)
I also tried to move the config and twitter_data files to the same path as 'configparser.py', but the error continues.
Would you know how to solve it, please?
This is because the python code cannot find the config file. Make sure that the files are in the same directory, and it should solve the issue. If didn't, the easiest option is to just directly put your keys in the python file. Something like
api_key = "paste the key here"
anyone knows why he can get 100 queries but when I do it I only got 20 queries max?
*edit
Nvm, i followed the video and it works. And I think theres a limit for 1500 tweets that i read from somewhere
Zoom out. Ffs.
Can i have ur email 🙏🏻
I tried this; it's stuck at api_key = config['twitter']['api_key'] error of line 964, in __getitem__
raise KeyError(key)
KeyError: 'twitter'
Is your config file in the same folder of the python code? If not, that could be the cause of this error. Other than, you can directly type the credentials in the python file, something like api_key = "xxxxxxxxxxx"
query = "(#alcnft) (from:Thla1oc)" - this thing didn't work, though I have use this hashtag often
Great video! Thank you