I signed up for IPRoyal after watching your videos but couldn't get the static proxies to work. It returned timeout error. I tried changing the network settings but could not solve the problem. Do you have any idea what could cause such issues? Thanks.
Hey, the video is really really helpful. Thank you very much for it! You are the go to channel for me whenever I wish to research on any topic related to web-scraping. You're doing a great job man! Also, in the end of the video you said that this is not your preferred method for scrapping infinite scroll dynamic websites. So which one is your preferred method, which is also scalable?
Thanks I’m glad you’re enjoying the content! The way I mentioned is by reverse engineering the sites backend api and making requests to it - I have a few videos on my channel that explain the basics of this idea!
I watched the section between 4:30 and 5:00 (roughly) so many times. The off-by-one space there was extremely distracting as well as satisfying when fixed. Cheers
I'm wondering if there is a way to use playwright with scrapy's shell? For me scrapy shell just seems to open the browser at the url and then block the scrapy shell from opening.
This is so timely for me @John, as I was literally building a scraper yesterday to scrape a website that used XHR. Top content! Additonally, would it be possible for you to share the java script "code" that was used in the PageMethod function?
I'm literally just getting started with python and need a fast study done for my thesis so I decided to study word usage on reddit. Should go through with it? Idk if i need any special stuff :/ I don't even have python installed. Cheers
You absolutely can - it’s not as easy to setup and use well in my opinion but fits well into a specific use case. However I don’t think it’s been updated for a while and I’ve had some people tell me it hasn’t been working for them recently. Give it a go and if it works for you then great
This is nice, but my problem with using playwright is that it the twisted reactor always leads to issues when I want to run my spiders using python scripts
you could try using Splash? It hasn't been updated in a few years but may still work. Or create your own scraping/render service separate and use that?
As a newbe... Does anyone have some experience with a PUP - a command line tool for processing HTML? Is there any way to import it to the Playwright project the same way as the HTMLParser? Thanks.
@@JohnWatsonRooney I use selenium it's powerful but some time it's some modules not working properly and thay makes me angry 😅 and I think I have move to new solutions but then I reminded myself that I have use proxies. But I don't like use of proxy I don't know why but I scares from using proxy. Is their any free proxies?
not something I've done before, but i know that the app will have a backend server/api that it makes the requests too, you;'d need to find this and reverse engineer it. or it might be possible to emulate the app on a pc/through browser?
@@JohnWatsonRooney most apps have ssl pinning security so we can not intercept it, to bypass this we can use nox player and man in the middle proxy to intercept from nox
You are a great tutor, and I suggest a video discussing and comparing all of these tools, why and when we could use them what is the best compo great work keep making tutorials
Thanks for another great video! This method seems so easy and wanted to try it myself but unfortunately, it seems that scrapy-playwright doesn't work on windows. Some sort of Linux emulation (WSL) is required. Also thanks for the iproyal discount. I was looking for such a service and your discount comes just perfect, will use it after NYE party :) PS: Everyone, a Happy new year!
@@kanwaradnan4849 for api usage it’s important to look at thenpayload that is sent. Is this a data form or a json payload? Also look good at the headers, this will fix 9/10 of your issues. Still doesn’t work? You probably forgot to fake some cookies :)
brother, i'm in a big problem. last 20 day's i'm trying to scrape one of site. but i failed all the time. I watched 100 of videos. but i failed. can you scrape a site for me. if possible plz reply to my comments. this is my final year project. you just scrape me some data. my final year defense in knocking my door. plz brother if possible reply my comments.
@@techlogger so after some digging with fiddler I have found the api and also able to get the video embedded url but couldn't get it to stream, since I have little JavaScript knowledge and unable to use the devtool (because they blocked it) this is as far as i can go for now, will try it later 😌.. and also as you mentioned this site use Obfuscated JavaScript.
@@skshaheen7506 you can unblock devtools, stream links ( m3u8 links ) are mostly restricted.. you have to pass proper headers, payload or in some case a decryption key. And yes that website is heavily obfuscated. But it's doable.
Grab IPRoyal Proxies and get 50% off with code JWR50 at iproyal.club/JWR50
Can you make a video about how to use it properly?
yes will do
@@JohnWatsonRooney thank you
I signed up for IPRoyal after watching your videos but couldn't get the static proxies to work. It returned timeout error. I tried changing the network settings but could not solve the problem. Do you have any idea what could cause such issues? Thanks.
You are the webscrape master
Hey, the video is really really helpful. Thank you very much for it! You are the go to channel for me whenever I wish to research on any topic related to web-scraping. You're doing a great job man!
Also, in the end of the video you said that this is not your preferred method for scrapping infinite scroll dynamic websites. So which one is your preferred method, which is also scalable?
Thanks I’m glad you’re enjoying the content! The way I mentioned is by reverse engineering the sites backend api and making requests to it - I have a few videos on my channel that explain the basics of this idea!
@@JohnWatsonRooney Oh alright. Yeah I’ve gone through those videos of yours and they indeed made my task a lot easier. So thanks for that too! XD
can you please do a video on your neovim configuration
Great video John, which Editor are you using ?
It’s neovim, with the basic ide config by chris@machine
@@JohnWatsonRooney Thank you bud, I've been looking for an alternative to PyCharm and vscode for a while now
Brilliant! love your attitude, admire an out-of-the-box thinker! Keep up the good work buddy!
Thank you!
I watched the section between 4:30 and 5:00 (roughly) so many times.
The off-by-one space there was extremely distracting as well as satisfying when fixed.
Cheers
Great video John
I'm wondering if there is a way to use playwright with scrapy's shell? For me
scrapy shell
just seems to open the browser at the url and then block the scrapy shell from opening.
This is so timely for me @John, as I was literally building a scraper yesterday to scrape a website that used XHR. Top content! Additonally, would it be possible for you to share the java script "code" that was used in the PageMethod function?
Thanks for reminding me I've added it to the description now!
Another great video! Thanks for showing both methods. 💯
Thanks mate!
Which is the best for JavaScript rendered websites?
Selenium or puppeteer or playwright?
I use playwright mostly now
@@JohnWatsonRooney but I'm selenium expert, Should I use playwright instead of selenium or stick with the selenium according to your expertise?
@@casual_gamer1413 selenium! as you say you already know it, they do the same thing for web scraping purposes
@@JohnWatsonRooney thank you❣️❤️
Hey, how can we scrape PDFs that are embedded to be viewed by chrome pdf preview? I think they use javascript.
I'm literally just getting started with python and need a fast study done for my thesis so I decided to study word usage on reddit. Should go through with it? Idk if i need any special stuff :/ I don't even have python installed. Cheers
instead of playwright, can we use splash for any projects? which are recommended for web scraping?
You absolutely can - it’s not as easy to setup and use well in my opinion but fits well into a specific use case. However I don’t think it’s been updated for a while and I’ve had some people tell me it hasn’t been working for them recently. Give it a go and if it works for you then great
How to run a script from playwright in jupyter notebook?
Need you to make a nvim setup video because thats cool af
Off topic, but will you do a video on websockets?
Good idea I’ll add it to my list
This is nice, but my problem with using playwright is that it the twisted reactor always leads to issues when I want to run my spiders using python scripts
you could try using Splash? It hasn't been updated in a few years but may still work. Or create your own scraping/render service separate and use that?
Next button is hidden for me. I am stuck
What is the IDE and theme used for this?. Nice explanation
It’s neovim and catpuccin theme
hiii, where is the github codes that you are using
Excellent tutorial! Thank you
Glad you enjoyed it!
Hi John.. What options do you suggest if I have to save screenshot of webpage as jpeg or as html itself.. Is it possible to with Scrapy
Yes! Where I have the pagemethods - you can add in another “screenshot” that will do it
@@JohnWatsonRooney Thanks John
Any way to get around PerimeterX?
As a newbe... Does anyone have some experience with a PUP - a command line tool for processing HTML? Is there any way to import it to the Playwright project the same way as the HTMLParser? Thanks.
why you don't use selenium?
I used to! It’s great.. I just spent more time with playwright and find it a bit easier to use
@@JohnWatsonRooney I use selenium it's powerful but some time it's some modules not working properly and thay makes me angry 😅 and I think I have move to new solutions but then I reminded myself that I have use proxies. But I don't like use of proxy I don't know why but I scares from using proxy.
Is their any free proxies?
Is that turle neck from uniqlo? It looks 👌
hah thanks - its Indicode brand.. but i got it from tk max
Hey how to scrape data from an Android application.
not something I've done before, but i know that the app will have a backend server/api that it makes the requests too, you;'d need to find this and reverse engineer it. or it might be possible to emulate the app on a pc/through browser?
@@JohnWatsonRooney most apps have ssl pinning security so we can not intercept it, to bypass this we can use nox player and man in the middle proxy to intercept from nox
Do you think chatgpt will put many of us out of commision??
Not anytime soon no
are you using lunar vim?
No but it’s the basic ide by the lunarvim author, it’s great
You are a great tutor, and I suggest a video discussing and comparing all of these tools, why and when we could use them what is the best compo
great work keep making tutorials
Great video content about webscrape. Your doing amazing bro.
Thanks for another great video!
This method seems so easy and wanted to try it myself but unfortunately, it seems that scrapy-playwright doesn't work on windows. Some sort of Linux emulation (WSL) is required.
Also thanks for the iproyal discount. I was looking for such a service and your discount comes just perfect, will use it after NYE party :)
PS: Everyone, a Happy new year!
I would really be interested on how to find api‘s 😊
i have a few videos on it on my channel! basically using the network tab in your browser to look for requests when pages are loaded
@@JohnWatsonRooney YEAH BUT IT DOES NOT WORK ALL THE TIMES.
@@kanwaradnan4849 for api usage it’s important to look at thenpayload that is sent. Is this a data form or a json payload? Also look good at the headers, this will fix 9/10 of your issues. Still doesn’t work? You probably forgot to fake some cookies :)
@@jesseroeleveld5430 yeah i found it buy watching video on this channel
@@jesseroeleveld5430 the reason I told the above statement is because I created a react app and it doesn't show my api using old tricks
John, Is it possible to scrapy an Android or iOS app
brother, i'm in a big problem. last 20 day's i'm trying to scrape one of site. but i failed all the time. I watched 100 of videos. but i failed. can you scrape a site for me. if possible plz reply to my comments. this is my final year project. you just scrape me some data. my final year defense in knocking my door. plz brother if possible reply my comments.
yes
If you're looking for a challenge, then try to scrape 9anime, It will be an interesting challenge. And also great content to watch. 🍿🍿
My friend did it with just requests.
@@techlogger hmmm yes it's possible 🤔, but you have to find the API.
@@skshaheen7506 yes. Not only api.. you have to solve heavy Java obfuscation too. It's more complicated than normal scraping
@@techlogger so after some digging with fiddler I have found the api and also able to get the video embedded url but couldn't get it to stream, since I have little JavaScript knowledge and unable to use the devtool (because they blocked it) this is as far as i can go for now, will try it later 😌.. and also as you mentioned this site use Obfuscated JavaScript.
@@skshaheen7506 you can unblock devtools, stream links ( m3u8 links ) are mostly restricted.. you have to pass proper headers, payload or in some case a decryption key. And yes that website is heavily obfuscated. But it's doable.
Can't say I'm first 😂