This worked for me. I had to scrape data of more than 10000 real estate agents and it was a headache for me. I watched a lot of videos but I didn't found the way out. I tried many data scrapers but they worked for sometime but then they failed. Then I watched this video and it took me half an hour to fix the issue and now it is automatically collecting my data. Thanks GOD!
I must say that this video has been an incredibly helpful tutorial. I've always been curious about how to extract and analyze data effectively, and this resource has provided me with clear guidance and valuable insights. I truly appreciate the effort put into creating such an informative and beginner-friendly tutorial.
Such a good tutorial! There wasnt a single moment where it was unclear to me what you were doing. As a beginner with webscraping I managed to make it work, thanks to this video!
Thank you so much. I was about to get my work hours reduced but thanks to you I will be able to put food on the table and pay my rent this month. I wish you the best and thanks again 🙏
This web scraping tutorial is exactly what I needed. The ability to scrape data from websites and export it to Excel will be so useful for my projects.
This was by far the best no bs video.Thank you so much. You have saved me so much headache, time and money. I liked it as it's very straightforward and easy to follow. THANK YOU. Jay - London
@@rimantasdanilevicius6754 Bro, did you manage to solve the problem with the delay? I need to scrape more than 40 thousand pages. But it's just text. cheers!!
@@rimantasdanilevicius6754 Did you ever find out what happened? My site has 985 rows but it only scraped 200. I'm thinking it only scrapped the first 10 pages and not the pages under "Next".
Wow! The way you explain is very clear and concise. This is very helpful especially for someone like me who is not really good in coding. thank you sir!
THANK YOU!!! It saved me from a lot of working, time and energy. Instead of maybe sorting my data manually for 2 weeks, this trick only took me less than 15 minutes to have it sorted!!! Genius!
Thank you! Thank you! Thank you! I have been looking for a a video like this to teach me how to scrape data and might i say, your video is absolutely amazing and straight to the point! Amazing job!
What do the "P", "C", "S" stand for when selecting something? How do we know which to use for what? Also, what are the different "Type"s you can choose from when adding a new selector?
Absolutely terrifying ! Number one ! Although the plugin is outdated and a bit different than the current version the video is TOP NOCH! I would like to see a bit more in depth explanation, but the tutorial is super ! Great work !
Well done, and thanks. I couldn't get the new pagination (beta) type to work - and was adding 10 pages at a time to the meta, which was getting boring. But I followed your instructions, setting it to 'link' first, then saved it, went back and changed it to the new pagination type, and bing0 -- 700 pages currently being scraped. The WebScraper-io tutorials on this don't explain much and I've been messing around for 2 days trying to get pagination to work. Thanks again.
Hi Azharul - Thanks for the help in scrapping. I have one problem making this work. I can only collect from 1 page. Tried mulitple times its not extracting more than 20 contacts from a page of 100 and also not collecting from other pages any help here ?
Very detailed and 100% helpful But when I export data long website appear like this, for exemple: www.youtu... and i don't know how to solve it Thanks for your help
i've tried creating my own sitemap but it seems i cannot click on 2 parent selectors. i can only click one parent selector. what seems to be the problem on that part? thanks
Awesome! So many things I hardly i understood in LinkedIn, just getting big CO's, CEO's personal emails addresses, phone numbers and how to connect to them for business are made bear naked in this app. Thanks for this because I love your work
I have watched many of these videos and it seems so easy but then when I try to do the things, the whole site is just greyed out and I can't select anything...
Hi thank you for this amazing tutorial. I just wanted to say what if we have many pages but there are only two buttons pervious and next for every page of job announcements for instance. How should we select all pages at one selector
I agree - great video but the moment I need to select, I get a black blob and can't go further. I have searched and can't find an explanation? Thank you.
Wonderful video. A question though, suppose the first business I am scraping does not have email but I require to feed email so that the tool can collect mails from businesses that have, how do I do that, Another quiz, it's my first time I am using this and when I have finished entering all the fields I require, when I click scrape, I get a notification that I need to automate web scraping, how do I go about that? Kindly help. Thank you in advance
Thanks a lot for this Video. I have one question. Can i Filter the data? Eg i would like to have car insurance in City x. How can i exclude some Post codes? Many thanks Nelly
Great tutorial. Mine is exporting empty sheets after scrapping even though the preview shows the content before scaping. What could be the problem. Everything seem to work but the data output is completely empty. Appreciate your response.
Thank you for this, it saved me tons of time!!!! Everything worked well, except the scrape didn't capture all of the websites on a page. Do you or anyone else know why...and what to do to complete that portion?
Very informative video. I did everything but am still unable to highlight my link using s hotkey. it keeps redirecting me to link i am trying to select
Hi Azharul. Thank you very much for this wonderful insight. One question. Please is this process dynamic? Can it be refreshed to get new data when the page is updated?
Great videos you have posted, very informative. Im new to all this but struggling right from get-go. Why do I have a 'haze' when trying to web scrap? When I go to the area that I want to we scrap, my cursor goes from a "pointer to a "round grey haze"
Very clear explanation.. thanks.. HOWEVER, i tired it twice and i am only getting the result for 1st page, not the rest of the pages. as soon as i get the first 50 results, it stops and the Tab that scraps disappears. Any solution? I have done the "Pages" thing exactly same as you, and then went into edit and selected both "root and pages".. nothing.
Need help with scraping data? Check this out: wher2hire.com/datascraping/ If you are facing any difficulties with scraping data from any website, you can hire someone from Fiverr to get all the data scraped from any website. You will find professionals who writes custom scripts to scrape data from any website. Tip: Before you hire any freelancer, please spend some time reading some of the reviews from their previous clients. It will help you learn more about the freelancer, and will help you find a best fit for your project.
Just open up the inspector tool again in that tab and then return to the original page. You’ll have to refresh the site map when you return to the original page. But it does work and pull the data over.
Dear Azharul Rafy, Your videos are very helpful for learning. But, when I follow this video learning, the process is not in working. web scraping option does not come in appear. will you mind to solve the problem or may I get any support from you?
Hi, great tutorial! One question for you though. The site I am trying to scrape has multiple page selectors. It has the main page selectors at the root site and another after the first link. I have set it up the way you showed in this tutorial. But I can’t figure how to scrape from the second page selector and have it continue the process. Without adding all new selectors for the information I want afterwards.
Great video. A problem I am running into is when dealing with multiple pages. Example to illustrate: Page numbers/links are displayed as follows: 1,2,3,4,5,6,7,8,...,100, next So 8 page numbers are displayed + 3 dots to show that there are more pages (unselectable - not a button/link) + the number/link of the final page (100) + the next button The next button does not go to page 9, it goes to page 2. When on page 2, pressing the next button goes to page 3 and so forth. So the scraper will only extract data from pages 1-8, and 100. 9-99 are not accessed. Is there a way to make the scraper not skip pages 9-99? I hope my example was clear. Thank you!
Great tutorial! I am having issue with getting the email addresses from Yellow Pages like you showed. I followed your instructions and it doesn't pull any data. Is there another method or work around? Thanks a bunch!
Hello Azharul. The video is very very helpful. I have a question though. Is it possible to refresh the scrapped data in excel by a click of a button or something ? If YES, how can I go about it ?
This is very helpful, but would you please tell me what to do if it's giving me an error message saying "could not establish connection. receiving end does not exist"
Hey mate, this is super useful, thank you very much What do we do if we're scrapping phone numbers that are hidden behind a button. For example, we do everything you've listed in the page but to see the phone number we have to click a button to "show" the number and then the number itself isn't text but is also a link. Thank you again
Great video but had a problem..I tried the yellowpages like your example but when doing the first selector and clicking each insurance agent, it just opened them instead of selecting and highlighting. Nothing I tried stopped that.
Much much thanks 👍👍👍👍👍 One question I would have: What if there is a Formular between? I mean on a site where services can offer ther service the name is public and the place... but for writing you need to give in your email, your name and click the AGB.. than you can open the formular ... here behind the send button should hide his email(or at least the mail for the site) or? Is there a way?
wonderful, for next one, would you to explain what to do if to extract the table data... that show per table let say 25 data but, and we need to acces next 26 - 50, 51-75 and so on ..
Hello there. I need some help. I have followed all the steps up until make selection. When I go for selecting multiple titles, it won't allow me. For example I select the first title, after that when I click on the second title, the previous selection disappears and only the last selection remains.
Hey bro, great video, it was so easy to follow however, some of the key information I need is not appearing once the pages have been scrapped, could you help with this? many thanks
Thanks for the info, but can you tell me why my cursor was more like a highlighter than a pointer. I wasn't able to select anything despite following your examples as my cursor wouldn't select - it was more like a highlighter / pointer, although the selection box did appear on the screen?
This worked for me. I had to scrape data of more than 10000 real estate agents and it was a headache for me. I watched a lot of videos but I didn't found the way out. I tried many data scrapers but they worked for sometime but then they failed. Then I watched this video and it took me half an hour to fix the issue and now it is automatically collecting my data. Thanks GOD!
What did u previously use? I need help
This is literally by far the best tutorial i've ever watched. No bs straight to the point, you gained a follower kind sir!
This tutorial changed my life.
I must say that this video has been an incredibly helpful tutorial. I've always been curious about how to extract and analyze data effectively, and this resource has provided me with clear guidance and valuable insights. I truly appreciate the effort put into creating such an informative and beginner-friendly tutorial.
Such a good tutorial! There wasnt a single moment where it was unclear to me what you were doing. As a beginner with webscraping I managed to make it work, thanks to this video!
Please help@mirtecox2872 how did you a company’s website without opening a new tab
You don't imagine how this tutorial has helped me in my new position. Thank you so much!!
hi....when i click the serial no. 1 that link open in new tab...what is this type of problem...please guide me...thanks
Thank you so much. I was about to get my work hours reduced but thanks to you I will be able to put food on the table and pay my rent this month. I wish you the best and thanks again 🙏
This web scraping tutorial is exactly what I needed. The ability to scrape data from websites and export it to Excel will be so useful for my projects.
This was by far the best no bs video.Thank you so much. You have saved me so much headache, time and money. I liked it as it's very straightforward and easy to follow. THANK YOU. Jay - London
the web scraping do not scrap all data. How you solved this problem?I tried to set delay 10 sec and still not all data have been scraped...
@@rimantasdanilevicius6754 Bro, did you manage to solve the problem with the delay? I need to scrape more than 40 thousand pages. But it's just text. cheers!!
@@rimantasdanilevicius6754 Did you ever find out what happened? My site has 985 rows but it only scraped 200. I'm thinking it only scrapped the first 10 pages and not the pages under "Next".
Wow! The way you explain is very clear and concise. This is very helpful especially for someone like me who is not really good in coding. thank you sir!
THANK YOU!!! It saved me from a lot of working, time and energy. Instead of maybe sorting my data manually for 2 weeks, this trick only took me less than 15 minutes to have it sorted!!! Genius!
i was doing a task for an internship thank you your video saved me a lot of time
Great video... simple step by step guide to follow for a non-programmer.
Thanks so much- super useful!
Thank you! Thank you! Thank you! I have been looking for a a video like this to teach me how to scrape data and might i say, your video is absolutely amazing and straight to the point! Amazing job!
Thank you for this video!!! I had to do data scrapping for 612 results and this helped me so much.
loved this tutorial, no fluff. straight to the point.
What do the "P", "C", "S" stand for when selecting something? How do we know which to use for what? Also, what are the different "Type"s you can choose from when adding a new selector?
Very comprehensive and better than the tutorial vid that accompanies the actual Chrome extension. Thanks!
Gets straight to the point, is efficient, and effective. Love this channel.
Absolutely terrifying ! Number one ! Although the plugin is outdated and a bit different than the current version the video is TOP NOCH!
I would like to see a bit more in depth explanation, but the tutorial is super ! Great work !
This video is absolutely fantastic! Thanks to it, I was able to find a way to not manually scrape 750 datapoints! Thank you for sharing this tutorial!
I only can scrape 5 pages tho
After clicking on the web scraper tab, there's no create new site tab showing. Can you assist me please?
From all the tutorials I've searched this one is the best! It helped me a lot!
Hey man, this is BY FAR the best video tutorial I've seen! Thank you!
what is the use of P,C,S buttons while selecting a link? And how to know whether to use link option or pop-up link option?
Hi Azharul
I recently started learning data analysis and your above video has been very helpful to me
Thanks a lot, really appreciate your content.
A very practical video, which saves time on writing and constantly revising!
A wonderful video that we've used as a reference for our recent additions. Your sharing is highly appreciated!
Well done, and thanks. I couldn't get the new pagination (beta) type to work - and was adding 10 pages at a time to the meta, which was getting boring. But I followed your instructions, setting it to 'link' first, then saved it, went back and changed it to the new pagination type, and bing0 -- 700 pages currently being scraped. The WebScraper-io tutorials on this don't explain much and I've been messing around for 2 days trying to get pagination to work. Thanks again.
I had the same problem the tool scrape data just from the first page then it stops did you find a solution
@@yassineouahi5361 I’m having the same problem
i can scrap page up to 11th page not more than 11th page, because 12th and 13th was not there
I had the same problem.
Hi can help, i only can scrap 5 pages
I want to thank you for showing us this tutorial. I was able to complete my dream-thesis splendidly 😂👍👍👍 thank you!
Thank you, i'm beginner in scraping and for the first time, i have followed toturials to do it, i will keep on practicing.
Hats off to your content sir. Really very helpful. Hope you get the recognition you deserve.
Hi Azharul - Thanks for the help in scrapping. I have one problem making this work. I can only collect from 1 page. Tried mulitple times its not extracting more than 20 contacts from a page of 100 and also not collecting from other pages any help here ?
Very informative VDO! Thanks! How can I select multiple Parent Selectors? I can't select _root and the pages.
Did you get any response to that
Nice tutorial, how can I scrape multiple of images? pagination --> image (multiple)? i don't understand, sorry
Need Help with Data Scrapping? Hire a Freelancer: rafys.net/DataScrapping
Get Octoparse Web Scraper: rafys.net/Octoparse
All the best,
Azharul
Very detailed and 100% helpful
But when I export data long website appear like this, for exemple: www.youtu...
and i don't know how to solve it
Thanks for your help
i've tried creating my own sitemap but it seems i cannot click on 2 parent selectors. i can only click one parent selector. what seems to be the problem on that part? thanks
Awesome tutorial. I have followed your instructions and learned it😁 everything worked! Thanks for your tutorials
Awesome! So many things I hardly i understood in LinkedIn, just getting big CO's, CEO's personal emails addresses, phone numbers and how to connect to them for business are made bear naked in this app.
Thanks for this because I love your work
Do you still have the copy for their email address?
I have watched many of these videos and it seems so easy but then when I try to do the things, the whole site is just greyed out and I can't select anything...
Hi Azharul, your videos are very helpful. Thank you so much, following from 🇰🇪 Kenya, be blessed.
This is a cheat code in life. Thanks
Hi thank you for this amazing tutorial. I just wanted to say what if we have many pages but there are only two buttons pervious and next for every page of job announcements for instance. How should we select all pages at one selector
Thank you so much for this video !
There’s an issue with getting data from new pages.
What should I do to fix it ?
You saved 50 hour of my time.
Thank you
I agree - great video but the moment I need to select, I get a black blob and can't go further. I have searched and can't find an explanation? Thank you.
Hi. I'm having the same issue. Did you find any solution? If yes, please share
amazing tutorial. you saved my 1000's of dollars.
Hi. Great video. I learned a lot. How do you select data in a simple drop drown menu?
Wonderful video. A question though, suppose the first business I am scraping does not have email but I require to feed email so that the tool can collect mails from businesses that have, how do I do that,
Another quiz, it's my first time I am using this and when I have finished entering all the fields I require, when I click scrape, I get a notification that I need to automate web scraping, how do I go about that?
Kindly help. Thank you in advance
Thanks a lot for this Video. I have one question.
Can i Filter the data? Eg i would like to have car insurance in City x. How can i
exclude some Post codes?
Many thanks Nelly
You are like a GEM in this field. Many wishes for you...
Thank you a lot for making the video. I did the same as you said. Unfortunately it only scraped the first page. where did I get it wrong?
Finally, I got what I have been looking for.
OMG.This is what I wanted😭
Thank you for your great work!!!
Great tutorial. Mine is exporting empty sheets after scrapping even though the preview shows the content before scaping. What could be the problem. Everything seem to work but the data output is completely empty. Appreciate your response.
Thank you for this, it saved me tons of time!!!! Everything worked well, except the scrape didn't capture all of the websites on a page. Do you or anyone else know why...and what to do to complete that portion?
When you select C that is child for certain IDs, does that mena you are setting them as children for the link or what?
Thanks!
This is a powerful tool! I hope I can use this to amazon website. Thank you for this great video
Man, you're a genius. Thanks for the tutorial!
Thankyou so much, I searched everywhere for my problem but you just solved it. Thankyou so much
Very informative video. I did everything but am still unable to highlight my link using s hotkey. it keeps redirecting me to link i am trying to select
Hi Azharul. Thank you very much for this wonderful insight. One question. Please is this process dynamic? Can it be refreshed to get new data when the page is updated?
Great videos you have posted, very informative. Im new to all this but struggling right from get-go.
Why do I have a 'haze' when trying to web scrap? When I go to the area that I want to we scrap, my cursor goes from a "pointer to a "round grey haze"
disable developer tools on top right side
Wow, you are saving me from having to buy a scrapper bot subscription. Your instructions were very clear. Thank You, Thank You, Thank You 😊 👍🏾👍🏾👍🏾
Very clear explanation.. thanks.. HOWEVER, i tired it twice and i am only getting the result for 1st page, not the rest of the pages. as soon as i get the first 50 results, it stops and the Tab that scraps disappears. Any solution? I have done the "Pages" thing exactly same as you, and then went into edit and selected both "root and pages".. nothing.
Need help with scraping data? Check this out: wher2hire.com/datascraping/
If you are facing any difficulties with scraping data from any website, you can hire someone from Fiverr to get all the data scraped from any website. You will find professionals who writes custom scripts to scrape data from any website.
Tip: Before you hire any freelancer, please spend some time reading some of the reviews from their previous clients. It will help you learn more about the freelancer, and will help you find a best fit for your project.
Hi, What if link window opens in another tab automatically?
There should be an option in Google Chrome settings to open page in same tab just check it out.
Just open up the inspector tool again in that tab and then return to the original page. You’ll have to refresh the site map when you return to the original page. But it does work and pull the data over.
@@themeyer2010 your technique screws up the site map
I am facing same problem
Give me solution please
Thank you for the video..Can it be used at Organization level? Is it a paid service?
Which scrapper is better between Web scrapper and Instant Data scrapper ??
Dear Azharul Rafy, Your videos are very helpful for learning. But, when I follow this video learning, the process is not in working. web scraping option does not come in appear. will you mind to solve the problem or may I get any support from you?
Hi, great tutorial! One question for you though. The site I am trying to scrape has multiple page selectors. It has the main page selectors at the root site and another after the first link. I have set it up the way you showed in this tutorial. But I can’t figure how to scrape from the second page selector and have it continue the process. Without adding all new selectors for the information I want afterwards.
Thanks brother you just took my business to another level Thanks everything is working properly .
Great video, love the simplicity and explanations!!
Do you have a video on how to use this on pages that have infinite scrolling pages? Thanks!
Great video. A problem I am running into is when dealing with multiple pages.
Example to illustrate:
Page numbers/links are displayed as follows: 1,2,3,4,5,6,7,8,...,100, next
So 8 page numbers are displayed + 3 dots to show that there are more pages (unselectable - not a button/link) + the number/link of the final page (100) + the next button
The next button does not go to page 9, it goes to page 2. When on page 2, pressing the next button goes to page 3 and so forth.
So the scraper will only extract data from pages 1-8, and 100. 9-99 are not accessed.
Is there a way to make the scraper not skip pages 9-99? I hope my example was clear.
Thank you!
I ended up watching your video 'Web Scraping Large Data' with Octoparse and so far it has solved the issue I mentioned above. Thank you again!
Wow! The way you explain is very clear and concise.
Hi Azharuf, what is the meaning of the C button when choosing "link" ...by the way..nice video
Great tutorial! I am having issue with getting the email addresses from Yellow Pages like you showed. I followed your instructions and it doesn't pull any data. Is there another method or work around? Thanks a bunch!
Hello Azharul. The video is very very helpful. I have a question though. Is it possible to refresh the scrapped data in excel by a click of a button or something ? If YES, how can I go about it ?
Please help. Azharul explains wonderfully but when I need to select a black blob appears and not a hand? Thanks
I'm having the same issue. Did you find any solution? If yes, please share
This is very helpful, but would you please tell me what to do if it's giving me an error message saying "could not establish connection. receiving end does not exist"
You’re a life saver!!! I wish I could like multiple times
Waow waow waow pure information, one of the best information that I had lastly, thank you sir, thank you a lot!
Hey mate, this is super useful, thank you very much
What do we do if we're scrapping phone numbers that are hidden behind a button.
For example, we do everything you've listed in the page but to see the phone number we have to click a button to "show" the number and then the number itself isn't text but is also a link. Thank you again
It works but only the first 5 pages and then stops. Is there a way to keep it going without stopping?
Great video but had a problem..I tried the yellowpages like your example but when doing the first selector and clicking each insurance agent, it just opened them instead of selecting and highlighting. Nothing I tried stopped that.
Much much thanks 👍👍👍👍👍
One question I would have: What if there is a Formular between?
I mean on a site where services can offer ther service the name is public and the place... but for writing you need to give in your email, your name and click the AGB.. than you can open the formular ... here behind the send button should hide his email(or at least the mail for the site) or? Is there a way?
wonderful, for next one, would you to explain what to do if to extract the table data... that show per table let say 25 data but, and we need to acces next 26 - 50, 51-75 and so on ..
It was marvellous for collecting small amounts of data. What's an easy way of scraping tens of data points from hundreds of different websites?
Thank You So Much Brother. I learn every day from your video something new.
thankyou! It's really helpfull. Can I use this for website that don't have any link except the pages? because I tried this, and it didn't work
sorry I'm newbie, but what's different between selecting P and C when we using the select feature?
This is the best tutorial for beginners to learn web scraping
fantastic video. you show that you scrape every page. selects all pages and next. but in your excel you only have page 2. is it correct?
Hello there. I need some help. I have followed all the steps up until make selection. When I go for selecting multiple titles, it won't allow me. For example I select the first title, after that when I click on the second title, the previous selection disappears and only the last selection remains.
Amazing video Azharul! Tysm
worked 100%. But in other cases when scraping data but cloudflare verification appears, how can it be executed ?
Hi Sir,
I am unable to select multiple pages from linkedin, saying - selector is required and cannot be empty.
Yellowpages fine
Please please help
Same issue for me
Hey bro, great video, it was so easy to follow however, some of the key information I need is not appearing once the pages have been scrapped, could you help with this? many thanks
I tried to select the link but my mouse cursor goes round when I leave the chrome extension to the target webpage
Can u help me with that ..?pls
Why did you click on P, C and S for some elements and not for some? when you were selecting
Thanks for the info, but can you tell me why my cursor was more like a highlighter than a pointer. I wasn't able to select anything despite following your examples as my cursor wouldn't select - it was more like a highlighter / pointer, although the selection box did appear on the screen?
What if after slelecting the link - instead it opens a new chrome tab?