Very useful video John. Keep them coming. If you had made this video a day earlier it would have saved lots of my time. But for the future it's a good reference.
Awesome vid with easy to understand explanations! Thanks John. Would you ever consider adding a script that would then open each article and scrape the contents? That would be super useful to see!
@@jonvincentmedenilla1296 from requests_html import AsyncHTMLSession url = 'news.google.com/topstories?hl=en-GB&gl=GB&ceid=GB:en' asession = AsyncHTMLSession() resp = await asession.get(url) await resp.html.arender(sleep=1, scrolldown=5) articles = resp.html.find('article') r = resp.html.raw_html That should be the start of the code now :D
Hey , i got some error when i run this , that is in render AttributeError : coroutine object has no attribute newPage runtimewarning coroutine launch was never awaited
@@JohnWatsonRooney ok... Also it does not renders the JS script data. I'm scraping trip.com , on this site the hotel details are rendered dynamically using AJAX . But requests-html is unable to render this data
John, thanks for making everything so easily accessible. Going through this step by step has (a) worn out my pause/play finger but (b) allowed me to understand how you’ve been building this up. having followed through it there are a couple of questions which I have as a ‘first timer’. I’m using python from within Anaconda and VS Code but the ‘render’ isn’t turning blue and it’s telling me “newsarticle” is not defined…. Any suggestions? have to admit I’m on a MacBook Pro but everything else seems to be fine. Thanks John.
great video! would it be possible to scrape the whole contenet of the news? I am doing aproject about fake news detection and I would need the whole content :)
Sorry but if i try to do that i get a bunch of errors, most importend of which is "OS Error: [WinError 14001] This application couldnt be stated, as the Side-by-Side-Konfiguration in invalid. " i guess. Did i forgot to install something?
I fixed this issue by uninstalling all of my python installations via control panel. Then I redownloaded python directly from the python website NOT the windows store.
It would be easy to do with selenium yes, but I try to stay away from that as much as possible - don’t always want the browser to pop up every time. I would be able to run this script on a Linux server every day easier like this than with selenium, for example.
Thank you for the video. When I try to run my code I get multiple error messages. One of those is that one here: "ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate" Do you know what went wrong?
I fixed it by going to python folder inside applications and click install certificates command. As far as I understand this problem is unique to Mac only and has to do with a certain library commonly used by other libraries.
Very useful video John. Keep them coming. If you had made this video a day earlier it would have saved lots of my time. But for the future it's a good reference.
Awesome vid with easy to understand explanations! Thanks John. Would you ever consider adding a script that would then open each article and scrape the contents? That would be super useful to see!
Hey did you create something like that?
This was really helpful. Thanks a lot!!
Thank you. But I have this error "Cannot use HTMLSession within an existing event loop. Use AsyncHTMLSession instead."
Are you using jupyter notebook? It doesn’t work in those unfortunately
@@JohnWatsonRooney so how do we got about it in Jupyter then?
@@jonvincentmedenilla1296
from requests_html import AsyncHTMLSession
url = 'news.google.com/topstories?hl=en-GB&gl=GB&ceid=GB:en'
asession = AsyncHTMLSession()
resp = await asession.get(url)
await resp.html.arender(sleep=1, scrolldown=5)
articles = resp.html.find('article')
r = resp.html.raw_html
That should be the start of the code now :D
I cant even start. " No module named " requests_html". Please, help me.
Hey , i got some error when i run this , that is in render
AttributeError : coroutine object has no attribute newPage
runtimewarning coroutine launch was never awaited
Thank you for the great video. How can I scrape all the news from every page, not only page 1 of the web?
+1. Is it possible to make a generic script which can scrap news from any news website?
You would have to make a separate web scraper for every website, as every website does the html in their own way.
My list seems to stop at 100 articles? Is there a way to circumvent this?
That seems to be some sort of Google soft cap, I’ll look in to it
@@JohnWatsonRooney I have encountered the same issue
I am getting this RuntimeError: Cannot use HTMLSession within an existing event loop. Use AsyncHTMLSession instead.
Are you trying to use it in a jupyter notebook? That’s the error I see usually when that’s the case
@@JohnWatsonRooney how do you overcome this in Jupyter notebook?
@@harivignesh4443 did you figure this out???
Thanks for video, can I use the same code, to return only articles with specific name in the header ?
Yea, you would need to add your own logic to match names but it’s definitely doable
where is content if i want to open each article and scrape content like title name how to do that?
Great video Jhon !¿Can you tell me what does html render does technically to our program?
Great video sir. How can we modify this to save the results in a well-structured spreadsheet?
how can i make it get only the newest news and make it run for ever
is it efficient to use a while loop?
why does r.html.render takes so much time for me...but it gets executed quickly for you
Not sure! It loads up a browser behind the scenes so does use a little bit of ram and processing power
@@JohnWatsonRooney ok... Also it does not renders the JS script data. I'm scraping trip.com , on this site the hotel details are rendered dynamically using AJAX . But requests-html is unable to render this data
John, thanks for making everything so easily accessible. Going through this step by step has (a) worn out my pause/play finger but (b) allowed me to understand how you’ve been building this up. having followed through it there are a couple of questions which I have as a ‘first timer’. I’m using python from within Anaconda and VS Code but the ‘render’ isn’t turning blue and it’s telling me “newsarticle” is not defined…. Any suggestions? have to admit I’m on a MacBook Pro but everything else seems to be fine. Thanks John.
How can I get the content of the news rather than the link
great video! would it be possible to scrape the whole contenet of the news? I am doing aproject about fake news detection and I would need the whole content :)
I got the same project bro. So is scraping the whole article possible?
How can this be used to do anything since its not a software?
I am getting same number of articles when i am using scrolldown=0 or scrolldown=5
Can anyone explain, why?
thanks sir but i think for only top headline we can just use our bs4 and return the first h1 tag text
Sorry but if i try to do that i get a bunch of errors, most importend of which is "OS Error: [WinError 14001] This application couldnt be stated, as the Side-by-Side-Konfiguration in invalid. " i guess. Did i forgot to install something?
I fixed this issue by uninstalling all of my python installations via control panel. Then I redownloaded python directly from the python website NOT the windows store.
what vscode theme are you using??
One dark pro I think
@@JohnWatsonRooney yeah i think it is thank you
I ended up getting duplicates in my list for reason. Each story title and link is listed at least 5 times each.
Try using hash table or something to manage your data
I only want it to show 10 news, how do I do that?
[:10]
@@masinde.charles Excuse me please can you explain me better
thanks sir its very damn useful
how can i get the date
Look for the element that contains the date.
date = Item.find(‘the element that contains the date’).text
How to scrap the content of the news
Thank you so much for this video. Question, wouldnt this be a perfect situation to use selenium?
It would be easy to do with selenium yes, but I try to stay away from that as much as possible - don’t always want the browser to pop up every time. I would be able to run this script on a Linux server every day easier like this than with selenium, for example.
@@JohnWatsonRooney thanks for the insight
👊
👊
did anyone else notice how my man wrote 'kink' real quick
Thank you for the video. When I try to run my code I get multiple error messages. One of those is that one here: "ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate"
Do you know what went wrong?
I fixed it by going to python folder inside applications and click install certificates command. As far as I understand this problem is unique to Mac only and has to do with a certain library commonly used by other libraries.
pmsl do any of your tutorials work lad.