Edit: It looks like this package is no longer working for some people (Thanks Elon) If you have issues here are some tips: From the SNScrape github page: ``` If you discover an issue with snscrape, please report it at github.com/JustAnotherArchivist/snscrape/issues. If you use the CLI, please run snscrape with -vv and include the log output in the issue. If you use snscrape as a module, please enable debug-level logging using import logging; logging.basicConfig(level = logging.DEBUG) (before using snscrape at all) and include the log output in the issue. ```
Hi Rob, I'm new to data science and I was wondering if you can make a video that shares the basics to webscraping from webpages. More specifically how to go about navigating and reading the parsed HTML texts of data to locate the strings we want like say content specific to a table or reviews on a webpages. Thanks for posting such easy to digest videos and sharing valuable tips to do things in a more efficient way!
Is scraping millions of tweets still possible because i would like to build an AI model using the text data. Please if it's possible how can I go about it?
Hello, I wrote the code for the video, but I'm getting an error. It still runs as of this date due to the issue with the APIs that need to be paid for on Twitter.
Thank you very much for the video,,, SNScrape used to work for me before but now it shows blocked 404, after some research I found that twitter now is blocking scraping without login..... any idea how to solve the problem ? Many thanx for the efforts
HI, Rob. I don't understand why i have this error: Errors: blocked (403), blocked (403), blocked (403), blocked (403) that's when i call scraper.get_items()
Hi Rob, I cannot seem to make it work, I am not sure if its something I am doing but it produces a 'Errors: blocked (404), blocked (404), blocked (404), blocked (404)' message and states 'failed, giving up', with a lot of links to twitter that show I don't have access. Do you have any suggestions? Thanks!
Everything works (in VS Code notebook) except the progress bar isn't displaying? It runs without error. Here is the code: n_tweets = 5000 for i, tweet in tqdm(enumerate(scraper.get_items()),total=n_tweets): data = [ tweet.date, tweet.id, tweet.content, tweet.user.username, tweet.likeCount, tweet.retweetCount, ] tweets.append(data) if i >n_tweets: break
Hi Rob, thank you a lot for this video. I have two questions: 1. How to proceed if we want to scrap data on twitter with more queries like : #father; #mother; #baby; #foot etc.. Are we going to write one code for each query? 2. Did you make video on scrapping Facebook, Instagram, Mastodon, Reddit, Telegram like you did for twitter?
Oh no, I haven't experienced that. Have you checked the github issues to see if others have the same problem? github.com/JustAnotherArchivist/snscrape/issues
Mate, how are u so good? What's your strategy? You code every single day? Or maybe you have many yrs of experience? What sort of courses or books do you study? How often do you read papers? I'm asking cz I learn so much jst thru recreating some of your projects and usually u have a very unique approach that I am used to, with very accurate results. I sure wanna be like you some day. Also could u keep the projects coming mate? I know there are tons of them on kaggle but it's quite frustrating to get stuck for us newbies a video explanation really helps.
Thanks for the kind words. I really don’t think of myself as being better than any other person with the same amount of experience. I think the main thing I try to do is learn constantly and be patient. Things take time to learn. Check out my video about how to get started in ML for a more detailed breakdown of the skills to learn. Good luck.
I tried but showing error after "for" statement, Error like : blocked (403), Error retrieving and message "Bad Authentication data" code 215 what I do now? Actually I have some twitter link and from that link I want like , view , RT count .. can you help me.. ?? How I code
Hi, thanks for the video super useful. I am quite new to all of this. What software are you using? And do you make those titles/write in each box separately? I am currently using PyCharm.
Thank you so much. I have one question. When I use snscrape, I can only use "tweet.content, count, date, id, index, outlinks, outlinkess, tcountlinks, tcountlinksss, url". But I want to explore like counts, image url and so on. Would you tell me how to scrape them?
I believe likes are in there as well as the attached images. But you might want to check test it out. If snscrape doesn’t work you’ll need to get a Twitter api key.
Instead of capturing tweets based on some hashtag, can we pull tweets from specific accounts? Like I have 5 accounts for whom I want the tweets data for a given time frame, is it doable from this method?
Hello,Would you mind if ask if you can demonstrate how to use a crawler to capture twitter space cations. If the audio file is very long (more than ten or twenty hours), how to quickly capture it?
Thanks! Appending to a list is much faster than appending to a dataframe so typically I prefer to just creat the dataframe after all the data is collected.
I think it's because that loop will actually gather 1002 results with n_tweets set at 1000. I think Rob was just trying to get ~1000, not 1000 exactly. Reason: enumerate starts from 0, and the loop is only broken once that number goes ABOVE n_tweets (e.g. 1001), so you end up with an extra result at both ends. Quick Fix: break the loop when i > (n_tweets - 2)
@@robmulla You're right, but given the versatility of Transformers Architecture and the actual importance on the State of the Art it's worthy to know it, I personally haven't found good materials and since your videos are awesome I'd very much like to see it.
Hi Rob! How much time it takes per 100k tweet? Just to compare with my scraper built in another language which takes approximately 10 to 15min to scape each 100k tweets.
Hi Rob. Thanks for the video. Did twitter update their security? Using your exact code from the first 3 minutes of the video, I keep getting an error retrieving api requests. Any tips on how to get past that?
@@eve9587 Hi, I am having the same problem with snscrape recently. I am quite beginner on this. Can you explain where to apply top=true? Thanks in advance.
Looks to be an issue everyone is experiencing. You can follow the github issues here and they may update. Hopefully just a temporary issue... github.com/JustAnotherArchivist/snscrape/issues
The comma after the "i" is to separate out the two things enumerate returns (index, value) in to the index "i" and value "tweet". Does that make sense?
Great question. You can add any query search terms like you would in the twitter search bar. Here is a cheat sheet: media.sproutsocial.com/uploads/2016/02/Twitter-Search-Operators-Cheatsheet-1.pdf So to search a specific user you should be able to do something like "from:Rob_Mulla". Hope that helps.
Sir, do you have a chat group where people ask questions? I followed your flight EDA, do you think predictions could be done with random forest ? Do you have something about this? (newbie here) . Thanks. BTW i like your videos.
I’m not sure. Things are changing every day at Twitter. This package has been relatively unstable lately so you might want to check out their GitHub (linked in the description)
Yes! I mentioned in a different response but you can use twitter's query to filter to specific users or dates. Check this cheat sheet: media.sproutsocial.com/uploads/2016/02/Twitter-Search-Operators-Cheatsheet-1.pdf So do something like "from:Rob_Mulla"... And setting your loop to only go through 50 would get you the latest tweets.
That’s a great question. There are ways to do this but I don’t believe snscrape can do it. You might need to apply for the Twitter api and use something like tweepy
Hi, just wanted to ask, we need twitter API access to do this scraping, right? Cause when I run the same code, it sends a request to twitter API website and then an error is displayed.
@@TemporaryForstudy I was using this code, but as of 2 days ago, it stopped working: import snscrape.modules.twitter as sntwitter import pandas as pd limit = 1000 tweets_list = [] for tweet in sntwitter.TwitterSearchScraper("NestleIndia").get_items(): if len(tweets_list) == limit: break else: tweets_list.append([tweet.date, tweet.content, tweet.user.username, tweet.user.displayname, tweet.user.description, tweet.replyCount, tweet.retweetCount, tweet.likeCount, tweet.lang]) df = pd.DataFrame(tweets_list, columns=['Date', 'Content', 'Username', 'Description', 'Language']) # Save the DataFrame to a CSV file df.to_csv('tweets_nestle.csv', index=False) print("Data saved to tweets.csv")
Yes, use something like "lang:en" in the query. Check out this cheatsheet on twitter queries. media.sproutsocial.com/uploads/2016/02/Twitter-Search-Operators-Cheatsheet-1.pdf
I got into detail about my jupyter setup towards the end of this video: th-cam.com/video/5pf0_bpNbkw/w-d-xo.html - Hope that helps. In short: jupyterlab with solarized dark theme.
Your shared videos are fantastic, Rob. Well done. By the way, I applied the above, but I was unable to scrape the data, assuming that Twitter does not allow escape without API. If anyone has done this successfully, please advise.
Unfortunately, does not work anymore after changes. Returns 404 constantly.
Edit: It looks like this package is no longer working for some people (Thanks Elon) If you have issues here are some tips:
From the SNScrape github page:
```
If you discover an issue with snscrape, please report it at github.com/JustAnotherArchivist/snscrape/issues. If you use the CLI, please run snscrape with -vv and include the log output in the issue. If you use snscrape as a module, please enable debug-level logging using import logging; logging.basicConfig(level = logging.DEBUG) (before using snscrape at all) and include the log output in the issue.
```
even I install the last version, it still won't work
Yes it's not working 😭😭
Presently doing my MSc project and this video just pushed me forward so much!
Thank youuuuu
Really glad to hear that.
Hi Rob, I'm new to data science and I was wondering if you can make a video that shares the basics to webscraping from webpages. More specifically how to go about navigating and reading the parsed HTML texts of data to locate the strings we want like say content specific to a table or reviews on a webpages.
Thanks for posting such easy to digest videos and sharing valuable tips to do things in a more efficient way!
You make it look so easy. Amazing vid!
Thank you! 😊 Hopefully you can put it to some good use.
I wish you were my lecturer/ tutor, I’m doing a conversion masters and the way you explain code is stopping me from having a breakdown 🤣🤣
Ha! Thanks, I apprecaite that. I try to make things easy to follow, glad to hear you're learning from it!
@@robmulla you’re doing an amazing job at making it easy to follow. Thank you!
Is scraping millions of tweets still possible because i would like to build an AI model using the text data. Please if it's possible how can I go about it?
Hello, I wrote the code for the video, but I'm getting an error. It still runs as of this date due to the issue with the APIs that need to be paid for on Twitter.
The code is no longer working; it reports a problem of tweets blocking. Could you please check and help solve this issue?
Yh noticed thusalso, something seems to be blocked.
Thank you very much for the video,,,
SNScrape used to work for me before but now it shows blocked 404, after some research I found that twitter now is blocking scraping without login.....
any idea how to solve the problem ?
Many thanx for the efforts
Interesting, I was just going to start using this. 😢
Awesome. Simple and straight 👌
Thanks for watching!
hi rob, i think this api doesn't work anymore. i think twitter blocked it
I'm geting 403 Forbidden error while trying to scrape tweets using the library. Does it means there is a restriction?
oh man, this is incredible and exactly what I was looking for 🤩 Let us know when you decide to launch a Patreon!
Thanks man! No Patreon for now, just share with others who might also apprecaite it.
HI, Rob. I don't understand why i have this error:
Errors: blocked (403), blocked (403), blocked (403), blocked (403)
that's when i call scraper.get_items()
same errror
Hi Rob - I tried but it is not working anymore.
Amazing, will help me a lot with a project. Keep up whit the awesome content!!
Great to hear! Thanks for commenting!
Hi guys, I have a problem with snscrape. The code works well but without any reason the process early stop at 188 tweets :/
This is awesome, nice and straight to the point!
and its not working
Hi Rob, I cannot seem to make it work, I am not sure if its something I am doing but it produces a 'Errors: blocked (404), blocked (404), blocked (404), blocked (404)' message and states 'failed, giving up', with a lot of links to twitter that show I don't have access. Do you have any suggestions? Thanks!
same issue
snscrape no longer works for twitter. Elon has changed the policies making it hard to scrape without using the official API.
Thanks for the knowledge sharing ....
When i am trying the same i am getting the 404 error .. Can you help me with the error rectification..
ok people this aint working now dont waste your time here
with this twitter x update is not working anymore? I tried to run the "for tweet in..." and it did not work "Error retrieving ...."
Yea, I think Elon broke it. :(
Everything works (in VS Code notebook) except the progress bar isn't displaying? It runs without error. Here is the code:
n_tweets = 5000
for i, tweet in tqdm(enumerate(scraper.get_items()),total=n_tweets):
data = [
tweet.date,
tweet.id,
tweet.content,
tweet.user.username,
tweet.likeCount,
tweet.retweetCount,
]
tweets.append(data)
if i >n_tweets:
break
try importing the base tqdm instead of tqdm notebook:
replace:
from tqdm.notebook import tqdm
with:
from tqdm import tqdm
@@robmulla Thanks that did the trick!!!
the coding is easy but currently, it's not working. I tried many times 🤕
Hi Rob, thank you a lot for this video. I have two questions:
1. How to proceed if we want to scrap data on twitter with more queries like : #father; #mother; #baby; #foot etc.. Are we going to write one code for each query?
2. Did you make video on scrapping Facebook, Instagram, Mastodon, Reddit, Telegram like you did for twitter?
use a for loop
why the tweets extracted showing only half text? why not full text plz help
Oh no, I haven't experienced that. Have you checked the github issues to see if others have the same problem? github.com/JustAnotherArchivist/snscrape/issues
Hi I am getting an error at line 6 for loop that Error retrieving , errors blocked 403 and request failed to giving up
Mate, how are u so good? What's your strategy? You code every single day? Or maybe you have many yrs of experience? What sort of courses or books do you study? How often do you read papers?
I'm asking cz I learn so much jst thru recreating some of your projects and usually u have a very unique approach that I am used to, with very accurate results.
I sure wanna be like you some day.
Also could u keep the projects coming mate? I know there are tons of them on kaggle but it's quite frustrating to get stuck for us newbies a video explanation really helps.
Thanks for the kind words. I really don’t think of myself as being better than any other person with the same amount of experience. I think the main thing I try to do is learn constantly and be patient. Things take time to learn. Check out my video about how to get started in ML for a more detailed breakdown of the skills to learn. Good luck.
This is gold. 6 minutes and so much information given without any unnecessary fluff... I'm amazed. Thank you so much.
Could you make it work? I guess the most recent reply is yours, I get " blocked (404) " error with Python 3.10.
@@ozgurbuldum6829 yeah, blocked!
@@ozgurbuldum6829 it does not work anymore right?
Another nice vid! Good job!
Glad you enjoyed! Share with a friend!
@@robmulla Always do!
I tried but showing error after "for" statement, Error like : blocked (403), Error retrieving and message "Bad Authentication data" code 215 what I do now? Actually I have some twitter link and from that link I want like , view , RT count .. can you help me.. ?? How I code
thanks! your progress bar is awesome!
I appreciate it! I actually have a whole video on tqdm (the progress bar package) you should check out.
Hi, thanks for the video super useful.
I am quite new to all of this. What software are you using? And do you make those titles/write in each box separately?
I am currently using PyCharm.
He's using Jupyter Notebook
is this still valid with x though?
It seems like the snscrape module doesn't work anymore, due to returns an error. Do you have any information?
Thank you so much.
I have one question.
When I use snscrape, I can only use "tweet.content, count, date, id, index, outlinks, outlinkess, tcountlinks, tcountlinksss, url".
But I want to explore like counts, image url and so on. Would you tell me how to scrape them?
I believe likes are in there as well as the attached images. But you might want to check test it out. If snscrape doesn’t work you’ll need to get a Twitter api key.
Instead of capturing tweets based on some hashtag, can we pull tweets from specific accounts? Like I have 5 accounts for whom I want the tweets data for a given time frame, is it doable from this method?
Hello,Would you mind if ask if you can demonstrate how to use a crawler to capture twitter space cations. If the audio file is very long (more than ten or twenty hours), how to quickly capture it?
Hey I am getting a 403 blocked error, I suppose the snscrape is going through API
Please create another video on this topic as these methods no longer work; they only scrape 100 tweets.
Good stuff! Why did use a list first (data) rather than a data.frame?
Thanks! Appending to a list is much faster than appending to a dataframe so typically I prefer to just creat the dataframe after all the data is collected.
When I run the code the progress bar turns red and becomes "1001/?". Did I make a mistake in the code or can I fix this?
I think it's because that loop will actually gather 1002 results with n_tweets set at 1000. I think Rob was just trying to get ~1000, not 1000 exactly.
Reason: enumerate starts from 0, and the loop is only broken once that number goes ABOVE n_tweets (e.g. 1001), so you end up with an extra result at both ends.
Quick Fix: break the loop when i > (n_tweets - 2)
Good catch! This is a bug. It can be fixed by subtracting 2 from the number of tweets in the line before the break.
hi, thank you for the tutorial. do you know how to set the data for specified location or range?
I'd really love to see you Forecasting Time Series using Transformers based Neural Nets.
I plan on making a video like this at some point. Only downside is that in my experience these models don't perform well unless the data is huge.
@@robmulla You're right, but given the versatility of Transformers Architecture and the actual importance on the State of the Art it's worthy to know it, I personally haven't found good materials and since your videos are awesome I'd very much like to see it.
Hi Rob! How much time it takes per 100k tweet? Just to compare with my scraper built in another language which takes approximately 10 to 15min to scape each 100k tweets.
I think if you watch the end of the video the progress bar shows 35 tweets per second. So like 1 hour for 100k tweets?
Is your method still working today ?
Can i use this same code in vscode
Hi Rob. Thanks for the video.
Did twitter update their security? Using your exact code from the first 3 minutes of the video, I keep getting an error retrieving api requests. Any tips on how to get past that?
Sameee. Did you work around it?
@@eve9587 I did not unfortunately. What about you?
Dude use top=True
It works now
@@eve9587 Hi, I am having the same problem with snscrape recently. I am quite beginner on this. Can you explain where to apply top=true? Thanks in advance.
Looks to be an issue everyone is experiencing. You can follow the github issues here and they may update. Hopefully just a temporary issue...
github.com/JustAnotherArchivist/snscrape/issues
What is the mark after the "for i ,"
The comma after the "i" is to separate out the two things enumerate returns (index, value) in to the index "i" and value "tweet".
Does that make sense?
What Tim said! Thanks Tim!
May I ask if I want to scrape from one specific user? How can I do that?
Great question. You can add any query search terms like you would in the twitter search bar. Here is a cheat sheet: media.sproutsocial.com/uploads/2016/02/Twitter-Search-Operators-Cheatsheet-1.pdf
So to search a specific user you should be able to do something like "from:Rob_Mulla".
Hope that helps.
@@robmulla Thanks for answering me! I'll definitely try that method 😁
Sir, do you have a chat group where people ask questions? I followed your flight EDA, do you think predictions could be done with random forest ? Do you have something about this? (newbie here) . Thanks. BTW i like your videos.
Yes I do! Join the discord: discord.gg/HZszek7DQc
What would be helpful for me would be a video on how to connect Python to a SQL Server database.!!
It's really easy with pandas. pd.read_sql() - you just need to create the connection object which depends on your database type.
Does Twitter block you if create a bot with and get tweets every day?
I’m not sure. Things are changing every day at Twitter. This package has been relatively unstable lately so you might want to check out their GitHub (linked in the description)
This with tweepy makes for a very interesting reposting bot
Interesting idea.
how can i scrape specific tweets? pls answer me it is very important .
How can I solve the 404 request error that I get when looping through the scraper object?
Any idea?
I think they’ve blocked it unfortunately
My code is not working
Amazing man
Thanks 🔥
How do I get a specific tweet and retrieve the quotes and replies?
The request was blocked by the twitter.
is there a way to pull say, latest 50 tweets from a specific user?
Yes! I mentioned in a different response but you can use twitter's query to filter to specific users or dates. Check this cheat sheet:
media.sproutsocial.com/uploads/2016/02/Twitter-Search-Operators-Cheatsheet-1.pdf
So do something like "from:Rob_Mulla"... And setting your loop to only go through 50 would get you the latest tweets.
Excellent Rob. Would you happen to have tutorial sessions? if so kindly link me to it
Yes! Subscribe and hit the bell icon and you’ll be notified when I go live. Also follow me on twitch.
it's not working for me I'm getting and error in retrieving the data 🤕🤕
very nice. Thanks
Most welcome
It is not working now days.
Do you have any other alternatives?
Does it still work with new API packages?
The data seems to be truncated. could anyone please tell me how to deal with that ?
Is it possible to scrape profile information and followers list? thank you
That’s a great question. There are ways to do this but I don’t believe snscrape can do it. You might need to apply for the Twitter api and use something like tweepy
big headstart in a project
does it still work? i doubt that. Was getting a 404 and 429 status codes last time i tried
Sir, is it legal to scrape data on twitter without using twitter api now? I heard twitter shut down their free api.
I don’t know. I’m not a lawyer but you should be ok.
@@robmulla thank you
Hi, just wanted to ask, we need twitter API access to do this scraping, right? Cause when I run the same code, it sends a request to twitter API website and then an error is displayed.
did you find any solution
@@khadijadar3156 yes, I can scrape tweets now. It was a very simple code.
@@rishavbhardwaj8044 can you tell me how you fixed that?
tell us how!
@@TemporaryForstudy
I was using this code, but as of 2 days ago, it stopped working:
import snscrape.modules.twitter as sntwitter
import pandas as pd
limit = 1000
tweets_list = []
for tweet in sntwitter.TwitterSearchScraper("NestleIndia").get_items():
if len(tweets_list) == limit:
break
else:
tweets_list.append([tweet.date, tweet.content, tweet.user.username, tweet.user.displayname,
tweet.user.description, tweet.replyCount, tweet.retweetCount, tweet.likeCount,
tweet.lang])
df = pd.DataFrame(tweets_list, columns=['Date', 'Content', 'Username', 'Description', 'Language'])
# Save the DataFrame to a CSV file
df.to_csv('tweets_nestle.csv', index=False)
print("Data saved to tweets.csv")
Hello, Thanks for the nice tutorial, I just wonder if there is a way to scrape only tweets instead of reply and tweets
I’m not sure but it should be possible in the query method.
Can u limit the tweets to only tweets in a certain language say English?
Yes, use something like "lang:en" in the query. Check out this cheatsheet on twitter queries. media.sproutsocial.com/uploads/2016/02/Twitter-Search-Operators-Cheatsheet-1.pdf
not working anymore
Why your jupyter notebook interface looks different than mine?
I got into detail about my jupyter setup towards the end of this video: th-cam.com/video/5pf0_bpNbkw/w-d-xo.html - Hope that helps. In short: jupyterlab with solarized dark theme.
@@robmulla Thanks. Imma do it with my setup
if I have the tweet's url or ID, can I get its content?
Thanks for the awesome video, are you planning on doing NLP project?
Thanks for watching! Have you seen my video on sentiment analysis? That has some NLP th-cam.com/video/QpzMWQvxXWk/w-d-xo.html
Very effective teaching! could you have a video for exploring data scrapped from twitter?
Thanks! I'd suggest checking out one of my live streams!
Just subscribed ❤
Thanks for subbing. Hope you like the videos.
How can I get follower count of different users using this??
I don’t think so. The Twitter api should give this easy enough though.
how do I get only people account and also their follower and following count?
Not sure if you can do this with snscrape, but with tweepy you could.
@@robmulla thank you sir!
please help with the 403 error if you can.
It’s possible they may have blocked this
@@robmulla Fix it
how do i scrape the english tweets alone?
Thank you for sharing this amazing tool and wonderful tutorial.
Q: How can I retrieve all replies to a selection of tweets?
Thanks. I don’t know if that’s possible with snscrape. You might need the official Twitter api.
what software do you use for write the code ?
it's look like jupyterlab
Yes, jupyterlab - I have a whole video about it!
Thank you for this wonderful tutorial.
Q: How can I retrieve infos like tweet content , username ... ,for a specific tweet using its id or url?
I have seen that you use jupyter lab in dark mode, how to activate dark mode?
Solarized dark theme! Check out my tutorial video where I explain how: th-cam.com/video/5pf0_bpNbkw/w-d-xo.html
i've try now but sntwitter cannot get any items
Oh no. Did Elon shut it down?!
i have an error "Unable to find guest token"
I'm not sure why that would be, works for me! Let me know if you get it working.
came across this error you need to initialize the scraper object after before every run of .get_items()
can we access tweets older than 7 days?
At least you used to be able to
Will I be able to scrape data using google collab?
You should be able to!
sure
how can we install snscrape?
Unfortunately, this is now dead 😒
clone directly from git
@@soumilyade1057 can you give more details on this, how to do it now
I tried too and it just gets blocked now. Shame.
@@mousaha643 there's no avenue left unblocked....! Ar kora jabenaa
@soumilyade1057 Are you saying there are now ways left to scrape twitter?
it is not working anymore !
an error 403 from twitter api occurs
at get items
Awesome
Thanks Patrick!
Your shared videos are fantastic, Rob. Well done.
By the way, I applied the above, but I was unable to scrape the data, assuming that Twitter does not allow escape without API. If anyone has done this successfully, please advise.
my case too. I think that these methods do no more work !
What shall we do for reddit,
Make one analysing WallStreetBets.
Great idea. Maybe I’ll make another video.
What is the thing you’re using to run the script? I know nothing about coding so I’m confused
Juypter notebook just google it and you’ll find it … or search anaconda
It’s not the snake I promise…
Does anyone know if this method still works today?
nope