Python Scrapy Tutorial - 9 - Extracting data w/ CSS Selectors
ฝัง
- เผยแพร่เมื่อ 19 ต.ค. 2024
- In this video we will scrape quotes from a website and select elements that need to be scraped using CSS Selectors. We will also learn about the tool called as Selector Gadget that is going to make your life so much easier!
Summary
1) Using CSS selectors
Using scrapy shell
response.css('title')
response.css('title').extract()
response.css('title::text').extract()
response.css('title::text')[0].extract()
response.css('title::text').extract_first()
2) Selector gadget on quotes website
3) Selector gadget on amazon
Next video - Extracting data using XPATH
• Python Scrapy Tutorial...
Full playlist - • Python Web Scraping & ...
Subscribe - / @buildwithpython
Website - www.buildwithpython.com
Instagram - / buildwithpython
#python
Really great tutorial. He goes through it step by step in order, so you have a clear understanding. That helps a lot
You just made my day! From last 2-3 days I am trying to learn web scraping but there are complicated videos on other channels .Today I watched your first 3 videos and then I got it you are going to kill it🔥🔥 and now suggesting that tool ...it became 💎💎. Thank you. You have one more subscriber.
You just gave me a breather with the Chrome Extension. Amazing video series! Keep up the good work. You earned a subscribe :)
bro really you are so underrated. bro you are teaching so well that i ,a mechanical engineering is doing this like its nothing. keep the hard work on . i love your videos and your teching style
Wow. I was going to wait until the last video to comment but I had to do it now. THANK YOU for these videos! They are SUPER helpful.
oh my god ! how am I gonna pay you back. YOU JUST MADE MY DAY. Speechless. The chrome extension is damn good bro. Thank you so much for this particular video !!!!
Glad I could help.
Why am I getting an empty list when scraping Amazon?
I wish I found this video much earlier. Just saved a lot of time and effort.
I'm really finding your work helpful for a research project I'm on in the UK. A big thank you for your excellent videos
Glad I could help.
Bhiya I don't know how to thank you great job and thanks a lot,you just made selecting piece of cake,thanks again
i have subscribed you nailed it bro i am Nigerian and we loved Indians
How do you get the last command in pycharm? Up does not work here for me. I have to write response...etc all over again which is annoying.
Incredible series!!!! Thanks a lot!! The extension you recommended is extremely helpful
Your tutorial is pure magic. Thank you very much!
Once again, great tutorial! Clear and straightforward!
your explanations are amazing, very engaging and interesting stuffs
Hey, great tutorial bhai ! What i get from it is that by using shell command on the terminal we can dynamically scrape data like we do with python request and beautiful soup.
Thanks for uploading them.
This video is Gold! I'm excited to learn web scraping now :D
Glad I could help.
Subscribed! Very helpful information ! definitely keep these videos coming!
a really good hands-on tutorial, 10x alot
Nice! The video is so clear, I think you should consider a lecturer carrier! You have a gift to explain complicated things very simply.
P.S. www.buildwithpython.com does not work - it says "The account for this site no longer active.
This content is not currently available."
Yeah it's not up. I didn't know people were even checking it out!
Hey, I'm getting error 404 while scraping the amazon website which you gave. I tied finding solution but was not able to fix it. Can you please help me out on this?
Nicely explained 👍. Thanks.
Have a question. It looks like the "response" object under "Available Scrapy objects:" is responsible for response.css. is that right?
There is no "response" object in the list for the web link I try to work on. Any suggestions? Ideas? Please.
OMG! IT IS WONDERFUL!
I have tried below ..but still not displaying here anything
>>> response.css(".a-color-base.a-text-normal").extract()
[]
>>> response.css(".a-text-normal::text").extract()
[]
>>> response.css("a-text-normal").extract()
[]
Did you try it on the example website I gave?
Solved the issue in two different ways,
response.css(".a-color-base.a-text-normal::text").getall()
and
response.css(".a-color-base.a-text-normal::text").extract()
Facing the same. Worked with quotes to scrape but not with amazon.
I tried it with Flipkart and it worked
@@mihirthakur917 hey I have a separate video for Amazon in the same playlist
amazon must have found this video and decided to block scrapers...
Selector gadget is awesome. Thanks mate.
Pure Gold. Thank you!
The selector tool is magic
how do you remove blank space like
and spaces when it just has a bunch of them from it
many thanks for all your teachings
Hi, I am following the code as you guide, but I am getting a Empty list for response.css even for previous video I got empty value can you explain me why?
Did you get it now? I'm getting empty list lol.
same I too getting empty
list
Brother you're outstanding
wow your tutorial is so great! good job
Thank You So Much Sir 👍👍
Cant scrape amazon... returns empty list
>>> response.css(".acs-product-block__product-title .a-truncate-cut::text").extract()
[]
any help..?
same here
i have same error,
did u find an alternate way?
Thank you so much for your Scrapy tutorials! However at 10:36, I tried running scrapy shell command on the Amazon website, and the response came back with a 503 code. How do I fix this? And, what's the issue behind it? I am running Windows 10.
Nevermind, I fixed the issue. I reduced the concurrent requests in the settings.py file to 1(I also added a user agent for Chrome browser with the latest version in the same file)
@@imaduddinsheikh3546 THANK YOU!!! Your comment saved me from a lifetime of searching for the fix!
I am trying to scrap data from youtube but it is returning an empty list every time . please tell me what to do.
very exceptional excellent work thanks for doing this
It is giving empty list on my pc at 11:12 please help me out.
yes it is fine but it is not working for all websites returning me an empty list
this extension is perfect. thank u so much.
The series is great, although there's something wrong with the quotestoscrape website, it gives me a twisted internet error, works for every other website though. Thanks.
Wow great tutorial
How to access the previous commands in the shell.. usually when I'm in the terminal I am able to access the previous command using the up button but in the shell I am not able to do the same as shown in the video.. can anyone help me with this..
SIR, while running scrapy shell command, terminal is raising a ValueError : invalid hostname: 'http
Hi there, I had a question. I wanted to parse the alt text off of an img. How would I go about this? I appreciate any help you can give!
Use attrib("alt")
Very helpful videos. thanks a lot :)
Just want to say Thank you!
Helped a great....but after half of video .. view not clear
great video man.ver very thanku
Amazon's source could change, I can't crawl data, elements are render from script, not from sample Html
You're great instructor!
thanks
Great video! Thanks a lot!
will appreciate your help
bro i am getting 503 error code how could i fix it please tell me brother
Go out back, find the biggest stick you can find, keep hitting your pc until it works. I hope this helped!
@@teo-medesi I tried brother due to that I bought new pc (😄 I fixed the error)
@@teo-medesithanks brother for providing valuable knowledge
@@shaikhanuman8012 Any time!
@@teo-medesi tq sir
Great video, just a small correction. In 09:00 you mention [1] is the first index of the list of authors. It's the second index.
scrapy crawl quotes. -> not returning anything. Nothing is displayed on the terminal
Basically, the function parse is not getting executed. Anything else written outside parse but inside the class is getting executed.
Thanks man. 💪
Your very Good !!!!
i have some question of this video. As you know ,scrapy is have a two ways for xpath that is css and xpath. i wonder why are u using css on your video .
In the next video I use xpath. I just like CSS selectors
@@buildwithpython thank u for comment!!
Saved my neck thanks man
I am continuously getting a null array, after using selector gadget.
you're probably getting a 503 error, which means service is unavailable. I solved this by specifying a user agent in settings.py and disabling the cookies, also in settings.py . User agent can be Mozzila 5.0 etc etc ( check explanations here: www.scrapehero.com/how-to-fake-and-rotate-user-agents-using-python-3/)
10:34 im getting 503 error from terminal for amazon
it says forbidden by robots, what do i do?
same here
hey I was trying to follow along this video and I think you can no longer use response.css, because it was removed I guess,
the error I get is: AttributeError: 'function' object has no attribute 'css'
Nope it's not removed. Don't think your scrap is installed properly.
@@buildwithpython oh, i did cd quotetutorial before opening shell,my bad
@@Pandazaar Hey, I am getting same error. Can you explain what went wrong? And solution pls. Thanks
@@babuji010 just type "cd .." and then open the shell
thanks a lot!
great!
Just Wowww
You are god
thx bro
No problem
wow
It is not a list its Array
It's a list, this is not C, in python they're called like that
Should be called a css De-selector
is it only me or the entire headphones shakes and trembles when he presses his keys
think they are feared of him
please dont overuse it and give some rest for both u and your keyboard
What if response =403, I can't extract anything?