Watch this video(link below), it explains about navigating through pages Web Scraping Google Search Results Using Selenium Python th-cam.com/video/8NXbUipUt8U/w-d-xo.html
Hi, I also have videos about this topic but I did not use python more than a year so I forget selenium totally. I found your video and tried your codes but unfortunately the link part raises error and I couldn't solve it. Code is link = elem.find_element_by_xpath('.//div[@class="yuRUbf"]/a').get_attribute("href")) error is: selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":".//div[@class="yuRUbf"]/a"} It says no such element but I can find that element when I check from chrome. Do you have any idea?
By the way I tried your code. It does not raise any error but it does not writes titles,links or descriptions to excel file. I try to see what it finds with print(title,link,detail) but it does not show anything.
Hi Hasan Onur Atac thanks for reaching out, while scraping google search results note that google search results html structure is displayed differently for different languages and regions or when you browse in your browser and the chromedriver session structure could also be different. So there are two things to consider 1. Inspect elements of chrome window that opens for script session or custom in terminal to navigate 2. Test these and #comment title and detail for testing (I have tested this and working) for element in driver.find_elements_by_xpath('//div[@id="search"]//div[@class="g"]'): link = element.find_element_by_xpath('.//div[@class="yuRUbf"]/a').get_attribute('href') here is recently uploaded video for google search results scraping th-cam.com/video/8NXbUipUt8U/w-d-xo.html
@@ScrapingNinja Hey this code works! thank you. I dont know how to detect proper way. I think it worked because of for element in driver.find_elements_by_xpath('//div[@id="search"]//div[@class="g"]'): this part. I will watch your new video right now. I hope you explain well how to find proper way for all situations.
This is all what i wanted Thanks @Let’sScrapeTheWeb creator Only one question drive.find_element_by_xpath is not working for me though i got the solution on StackOverflow but why i am getting this error
@@ScrapingNinja and sir i am also not able to get the links of the particular div class see here’s my code link = driver.find_element(by = By.XPATH,value='.//div[@class="kno-rdesc"]/a').get_attribute('href')
Hi, thank you for sharing this. But how do you scrape the data beyond the first page? and also prevent google from blocking us with captcha?
Thanks for watching, new video about captcha and proxies is coming soon.. Don’t forget to subscribe
@@ScrapingNinja Okay, thanks. I will wait for that, I am having a problem scraping the data reliably, because of the captcha.
How to get search results if it contains more than 1 page like for example 100 pages. Please reply. Thanks
Watch this video(link below), it explains about navigating through pages
Web Scraping Google Search Results Using Selenium Python
th-cam.com/video/8NXbUipUt8U/w-d-xo.html
Hi, I also have videos about this topic but I did not use python more than a year so I forget selenium totally. I found your video and tried your codes but unfortunately the link part raises error and I couldn't solve it. Code is
link = elem.find_element_by_xpath('.//div[@class="yuRUbf"]/a').get_attribute("href"))
error is:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":".//div[@class="yuRUbf"]/a"}
It says no such element but I can find that element when I check from chrome. Do you have any idea?
By the way I tried your code. It does not raise any error but it does not writes titles,links or descriptions to excel file.
I try to see what it finds with print(title,link,detail) but it does not show anything.
Hi Hasan Onur Atac thanks for reaching out, while scraping google search results note that google search results html structure is displayed differently for different languages and regions or when you browse in your browser and the chromedriver session structure could also be different. So there are two things to consider
1. Inspect elements of chrome window that opens for script session or custom in terminal to navigate
2. Test these and #comment title and detail for testing (I have tested this and working)
for element in driver.find_elements_by_xpath('//div[@id="search"]//div[@class="g"]'):
link = element.find_element_by_xpath('.//div[@class="yuRUbf"]/a').get_attribute('href')
here is recently uploaded video for google search results scraping
th-cam.com/video/8NXbUipUt8U/w-d-xo.html
@@ScrapingNinja Hey this code works! thank you. I dont know how to detect proper way. I think it worked because of for element in driver.find_elements_by_xpath('//div[@id="search"]//div[@class="g"]'):
this part. I will watch your new video right now. I hope you explain well how to find proper way for all situations.
@@Programlama101 i am glad that it works for you...
From where you learn web scrapping
I am self taught from mostly stack overflow and github...
Which python version do you use ?
Its python 3.8 i am using in this video..
Good job cahchu jan❤️
Thanks dear
Do you have this code on the Github repo? I need urgently for a job assignment!
Yes it is available on github, i have added the link in video description
@@ScrapingNinja Thank you!
👍
Ali
Hi Thanks for taking the time to comment.
This is all what i wanted
Thanks @Let’sScrapeTheWeb creator
Only one question drive.find_element_by_xpath is not working for me though i got the solution on StackOverflow but why i am getting this error
I see typo here it should be driver.find_element_by_xpath
@@ScrapingNinja yes
@@ScrapingNinja and sir i am also not able to get the links of the particular div class see here’s my code
link = driver.find_element(by = By.XPATH,value='.//div[@class="kno-rdesc"]/a').get_attribute('href')
Try to print(link) and then do link.get_attribute(‘href’)