John, Great tutorial... many thanks... now I know how to juggle... ;) Wanted to pass on an observation... apparently, Windows, can be cranky with asyncio/aiohttp. Your example program throws a “RuntimeError: Event loop is closed ” error. However, adding towards the bottom: asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) … on top of pages = asyncio.run(main(urls)) … solves whatever 'Event loop' issues that were present.
yes i was working on the async request and I got the similar error, the only way it works for me is if my program keeps on running. Since I am doing a constant request response after one ends, i dont get this error. but when I run my program for only one page to scrap then I get this error.
Hi John, thank you very much for this. Found this video while trying to figure out how to include an async AIOHTTP loop in some API processing script I'm writing and this was invaluable for figuring out how to structure the code.
No requests from me... just love your videos John! - Thanks for spending the time... I need this code to pull data for 188,600 items (each one is a web page...with 3 tables each) - UIPath would take about 32 days to complete. - asyncio + aiohttp should be much, much faster! Thanks for the tip Rasstag ( had the same issue in Windows)
So basically what's happening here in the whole program is that when on the event loop -- all of the tasks while the requests is being made, while it's waitiing it passes the resources to the other task functions? so on and so fort up until we got the response? I'm sorry if it's not that clear
Yeah so basically instead of the whole program waiting for a server response it gets on doing other things, ie more requests out. It means there is minimal time spent waiting and maximum time spend doing
@@JohnWatsonRooney THANK YOU! To anyone who watches his videos and does not subscribe! PLEASE DO IT NOW! Thank you man! also please smile more! I like your beard! wish I have that as well
Thank you so much, I had learn a lot from your videos, I have a question, It is possible that there is a similar option for pages in cloudflare, I currently use cloudscraper, but it has bugs, do you recommend something?
Hi John. Thank you for your video. Just one question: Is it possible to make def Pars to be asynсhronous too? I need two functions: first gets links for product card pages from 30 pages of site, and second function uses that links to get products price and other data from every card page. I have made first function asynсhronous and it works really quickly. I want second function o be asynсhronous too because the number of cards is more than 1000. But I still not know, how to gather these two functions together, how to create tasks for them both. Could you please give me your advise how to make it works
If you shoud define the best module for web scraping in terms of efficience and robust, what would be? I know selenium, requests, HTMLSessions, aiohttp, AsyncHTMLSession, scrapy, among others. What do you recommend to focus in specifically for its completeness. Thank you for your content.
I've learned a lot about scraping through your videos! What is the chances of getting blocked if I host my script on a server and scrape a website multiple times a day? Should I be implementing proxy rotation or can I get away without it?
Thanks Justin! It depends on how often is multiple times and how many requests you are making. I’d say try it first and see if you start to run into issues
Thanks! Yes you will need to use some good proxies to avoid the bans. Or I find async is great for getting data from multiple different sites at the same time
Wow, it is an amazing content. Thank you. I would like to know more about asyncio in Python. How did you learn it? Do you recommend some lecture or article?
The webpage that I'm trying to scrape using asyncio and aiohttp throws an error saying "return self._body.decode( # type: ignore[no-any-return,union-attr] UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 25051-25052: invalid continuation byte" Was working fine in requests and html. What's the solution?
I wouldbreally appreciate an answer: i have to make about 10000 api requests but the api supports only about 50 calls at a time so i need to split the whole 10k range into chunks. How do i make async requests in batches.?
Ive been trying to use Selenium to Parse the Name and Price of some coins in a crypto exchange i use very often. But i keep getting this error. driver.find_element_by_xpath("//*[@id="root"]/div/div/div/div[3]/div[2]/div/form/input").click() What do i do
ok so i have a question for you i have over 6000+ links and all i want to do is read the url get a one thing off the page and add it to the corelating link in google sheets any tips
This library is great. But how can we pass rich cookies with "path, expires, secure... etc", not just simple "key:value"? I found that documentation is extremely poor in terms of cookies.
Nice one, the third lesson learned here! Can you show some database lessons like we scraped 2 sites of shopify which have the same products, then do some price matching work to find the lowest price, thanks
One question...Since this is async, Will it wait for javascript to load fully before fetching the data from a page? Ex: If I put a brand filter in a ecommerce site which updates the page without loading it again, Will Aiohttp be able to fetch the updated the data ??
Nice! More tools in the chest! Thanks John. Any chance you can demo using an if-then-else scraping example? For example: If a set price or a range in price is found. Then scrape that object. Else, move on to the next page? Thanks for considering this request. Or, if you've done something similar in your previous Vids? Can you mention which ones? I've might've missed it... Thanks again!
Thanks for these videos. I am starting my journey on web scraping because I want to try understanding my spend on Deliveroo. (a food delivery app - like uber eats). Some questions and cheekily a tutorial, if possible, would be to learn how to manage social login (I used to google login for Deliveroo) and then be able to go down a level to a page and gather the order history.
@@JohnWatsonRooney yes sir, but i very confuse to fix it, can you some day or maybe later in the next videos. Made a easy to solve my problem forbidden 403.. Before for that.. Thank you sir.. Your replay my comment
Thank you very much. I got this error `Traceback (most recent call last): File "aiohttp.py", line 34, in results = asyncio.run(main(urls)) File "C:\Users\Future\AppData\Local\Programs\Python\Python37\lib\asyncio unners.py", line 43, in run return loop.run_until_complete(main) File "C:\Users\Future\AppData\Local\Programs\Python\Python37\lib\asyncio\base_events.py", line 584, in run_until_complete return future.result() File "aiohttp.py", line 18, in main async with aiohttp.ClientSession() as session: AttributeError: module 'aiohttp' has no attribute 'ClientSession'`. Any ideas?
The problem occurred because the file name was aiohttp.py (It seems that causes conflict with the aiohttp module). I have put the solution so as to be a reference for others that may this problem happen with.
Hello i am doing sum website scraping with one websit but i gating the one problem that website have an button function but when i click it it will give me a one name but wan i cloae the new tabe and reopen the page the name will button givea is hang every time than i can i scrape that Advance thanks and please help me 🙏
John,
Great tutorial... many thanks... now I know how to juggle... ;)
Wanted to pass on an observation... apparently, Windows, can be cranky with asyncio/aiohttp. Your example program throws a “RuntimeError: Event loop is closed ” error.
However, adding towards the bottom:
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
… on top of
pages = asyncio.run(main(urls))
… solves whatever 'Event loop' issues that were present.
yes i was working on the async request and I got the similar error, the only way it works for me is if my program keeps on running. Since I am doing a constant request response after one ends, i dont get this error. but when I run my program for only one page to scrap then I get this error.
@Rasstag, dude ur a life saver. well done!
Thank man!
Your videos and topics just keep getting better. Great job!
Hi John, thank you very much for this. Found this video while trying to figure out how to include an async AIOHTTP loop in some API processing script I'm writing and this was invaluable for figuring out how to structure the code.
Great wallpaper from Firewatch
No requests from me... just love your videos John! - Thanks for spending the time...
I need this code to pull data for 188,600 items (each one is a web page...with 3 tables each) -
UIPath would take about 32 days to complete. - asyncio + aiohttp should be much, much faster!
Thanks for the tip Rasstag ( had the same issue in Windows)
thank you very much!
Your video helped a lot, thank you
HEEY MAAAN RECENTLY WATCH YOUR VIDS ON SCRAPY!
YOU'RE SAVING MY LIFE AGAIN! THANKS1
@John Watson Rooney Good tutorial! Thanks! But the lines 23 to 26 are synchrone no?
Awesome way to get list of urls 👌😎
Thanks!
you are a wizard my friend
So basically what's happening here in the whole program is that when on the event loop -- all of the tasks while the requests is being made, while it's waitiing it passes the resources to the other task functions? so on and so fort up until we got the response? I'm sorry if it's not that clear
Yeah so basically instead of the whole program waiting for a server response it gets on doing other things, ie more requests out. It means there is minimal time spent waiting and maximum time spend doing
@@JohnWatsonRooney THANK YOU!
To anyone who watches his videos and does not subscribe! PLEASE DO IT NOW! Thank you man! also please smile more! I like your beard! wish I have that as well
Thank you so much, I had learn a lot from your videos, I have a question, It is possible that there is a similar option for pages in cloudflare, I currently use cloudscraper, but it has bugs, do you recommend something?
Subscribed. What are your thoughts on going about it this way vs something like scrapy?
Honestly I use scrapy almost all the time now, unless it’s more of a throwaway script
Hi John. Thank you for your video. Just one question: Is it possible to make def Pars to be asynсhronous too? I need two functions: first gets links for product card pages from 30 pages of site, and second function uses that links to get products price and other data from every card page. I have made first function asynсhronous and it works really quickly. I want second function o be asynсhronous too because the number of cards is more than 1000. But I still not know, how to gather these two functions together, how to create tasks for them both. Could you please give me your advise how to make it works
If you shoud define the best module for web scraping in terms of efficience and robust, what would be? I know selenium, requests, HTMLSessions, aiohttp, AsyncHTMLSession, scrapy, among others. What do you recommend to focus in specifically for its completeness. Thank you for your content.
Hi, requests-html is my go to for most smaller projects. Otherwise scrapy
Hi John, great videos by the way! I was wondering how can I scrape a website for the ASIN's, product title, stock levels and price?
thanks for the tutorial!
Cool tutorial. Just one question: what could we do, if we want to add new urls to the task list from the parsed results?
Very good! :) Thank you!
Thanks glad you enjoyed it
Always trying new ways of scraping. Great 👍🌹
Thanks!
Thank you sir!!
Hi John, thanks alot for your wonderful vedios, I was wondering which is faster async or multithreading in webscraping?
Thanks! Async is faster
Thank you it was really helpful to grasp the asyncio concept
I've learned a lot about scraping through your videos! What is the chances of getting blocked if I host my script on a server and scrape a website multiple times a day? Should I be implementing proxy rotation or can I get away without it?
Thanks Justin! It depends on how often is multiple times and how many requests you are making. I’d say try it first and see if you start to run into issues
Thank you, John! Realy nice tutorial, helped alot.
why did you put the await in return await r.text? I had to take it out in roder for my code to work
Great tutorial! How do we get around with the IP bans? Bombing the sever with async requests often gets me banned.
Thanks! Yes you will need to use some good proxies to avoid the bans. Or I find async is great for getting data from multiple different sites at the same time
Wow, it is an amazing content. Thank you. I would like to know more about asyncio in Python. How did you learn it? Do you recommend some lecture or article?
Great tutorial..Sir
Thank you very much 🥰
Really good explanation ! thks a lot !
Where is the link for the video with "session objects", please?
You're the best
Thank you!
hi john , it would be great if you could reply, how do i go about returning the url for each iteration of the event loop?
How could we use proxy dictionary with aiohttp?
Hi John, great tutorial again. What theme for VS Code is this? Best regards
thanks- i believe that is GruvBox Material. i change a lot!
The webpage that I'm trying to scrape using asyncio and aiohttp throws an error saying
"return self._body.decode( # type: ignore[no-any-return,union-attr]
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 25051-25052: invalid continuation byte"
Was working fine in requests and html. What's the solution?
I wouldbreally appreciate an answer: i have to make about 10000 api requests but the api supports only about 50 calls at a time so i need to split the whole 10k range into chunks. How do i make async requests in batches.?
Bro how to scrape sciencedirect website
Can you explain please how to scrap products price from webstore and send telegram alert when price drop? Thanks for you're video
Ive been trying to use Selenium to Parse the Name and Price of some coins in a crypto exchange i use very often. But i keep getting this error. driver.find_element_by_xpath("//*[@id="root"]/div/div/div/div[3]/div[2]/div/form/input").click()
What do i do
cool. thanks!!
great video
Will you do a trio video and cover nurseries please?
ok so i have a question for you i have over 6000+ links and all i want to do is read the url get a one thing off the page and add it to the corelating link in google sheets any tips
aka most of the items from the market on a game called runescape
This library is great. But how can we pass rich cookies with "path, expires, secure... etc", not just simple "key:value"?
I found that documentation is extremely poor in terms of cookies.
Awesome 👍😎
Thanks!
Hey John!
Also, my next attempt will be grabbing one image from each of the 120k products. Is there a video for that? :)
Where I can copy this code 🤷🏻♂️
Awesome... :-)
Nice one, the third lesson learned here! Can you show some database lessons like we scraped 2 sites of shopify which have the same products, then do some price matching work to find the lowest price, thanks
Good stuff! Yes I am planning on some more database related things to come
@@JohnWatsonRooney absolutely great guide, just scrape 36000 product using 300s, just like a movie:)
One question...Since this is async, Will it wait for javascript to load fully before fetching the data from a page?
Ex: If I put a brand filter in a ecommerce site which updates the page without loading it again, Will Aiohttp be able to fetch the updated the data ??
Nice! More tools in the chest! Thanks John.
Any chance you can demo using an if-then-else scraping example?
For example:
If a set price or a range in price is found.
Then scrape that object.
Else, move on to the next page?
Thanks for considering this request.
Or, if you've done something similar in your previous Vids? Can you mention which ones? I've might've missed it... Thanks again!
Sure sounds like a good idea I’ll have a look and see what I can come up with!
can you publish the source code?
Your videos are very informative...
Bro....can you make video on web scraping where cookies expires after 30 mins...example website like NSE etc
thanks .
Link to the quickstart page :
docs.aiohttp.org/en/stable/client_quickstart.html
Thanks for these videos. I am starting my journey on web scraping because I want to try understanding my spend on Deliveroo. (a food delivery app - like uber eats). Some questions and cheekily a tutorial, if possible, would be to learn how to manage social login (I used to google login for Deliveroo) and then be able to go down a level to a page and gather the order history.
How to fix error 403 Forbidden sir?
Try different headers - user agents and cookies to see if that works
@@JohnWatsonRooney yes sir, but i very confuse to fix it, can you some day or maybe later in the next videos. Made a easy to solve my problem forbidden 403.. Before for that.. Thank you sir.. Your replay my comment
Thank you very much. I got this error `Traceback (most recent call last):
File "aiohttp.py", line 34, in
results = asyncio.run(main(urls))
File "C:\Users\Future\AppData\Local\Programs\Python\Python37\lib\asyncio
unners.py", line 43, in run
return loop.run_until_complete(main)
File "C:\Users\Future\AppData\Local\Programs\Python\Python37\lib\asyncio\base_events.py", line 584, in run_until_complete
return future.result()
File "aiohttp.py", line 18, in main
async with aiohttp.ClientSession() as session:
AttributeError: module 'aiohttp' has no attribute 'ClientSession'`. Any ideas?
The problem occurred because the file name was aiohttp.py (It seems that causes conflict with the aiohttp module). I have put the solution so as to be a reference for others that may this problem happen with.
Great glad you worked it out!
Can you show how to scrap form the delta airlines delta.com
Sure I can have a look
@@JohnWatsonRooney thanks a lot John
Hai John, can you do example for scrap the Google search results with 15 results.(heading, url, content)
Hello i am doing sum website scraping with one websit but i gating the one problem that website have an button function but when i click it it will give me a one name but wan i cloae the new tabe and reopen the page the name will button givea is hang every time than i can i scrape that
Advance thanks and please help me 🙏
grequests is not working ??
i try treturns me none.
wers the source code