Great tutorial John, just a quick fix on the for loop, you forgot to add the i to the extract function: Made the changes to this for i in range(0, 41, 10): print(f'Getting page {i} ') c = extract(i) transform(c)
Hi everyone, I had the following error message : File "/Users/admin/Downloads/Test.py", line 36, in transform (C) File "/Users/admin/Downloads/Test.py", line 22, in transform summary = item.find('div', class_='job-snippet').text.strip().replace(' ','') #trouver sommaire et remplacer new lines with noth' UnboundLocalError: local variable 'item' referenced before assignment ANY HELP ?
That's an excellent video for the following reasons: -- the flow of the tutorial is really smooth, -- the explanation is excellent so you can easily adjust the classes that existed in the time of the video to the current ones -- and the iterations are detailed so every step is easy to understand. Thank you so much for this video! Greetings from Greece! 🇬🇷
I am very surprised you only have 1500 views. This is one of the best webscraping tutorials i have come across. Can you do one for rightmove or zoopla?
Worked flawlessly, I just had to edit a few things, like the classes and the tags. Nothing wrong with the code, just the indeed website that changed since you posted this video. Thanks!
Hello, John. I have just started learning Python, and I'm trying to use it automate some daily tasks, and web scraping is my current "class". I really enjoy watching your workflow. I love watching the incremental development of the program as you work your way through. You are very fluent in the language, as well as the various libraries you demonstrate. I am still at the stage where I have to look up the syntax of tuples and dictionaries... (Is it curly brace or brackets? commas or colons?) so I find myself staring in amazement as two or three lines of code blossom into 20, and this wondrously effective program is completed in minutes... I am envious of your skill, and I wanted to let you know I appreciate your taking the time to share your knowledge. I find your content compelling. Sometimes I forget to click the like button before moving on to the next vid, so sorry about that. I just have to go watch it again, just to make sure I leave a like... Your work is very inspiring to me as a noob. I aspire to the same type of fluency as you demonstrate so charmingly. Thanks again.
Hi David! Thank you very much for the comment, I really appreciate it. It's always great to hear that the content I make is helping out. learning programming is a skill and will take time, but if you stick with it, things click and then no doubt you'll be watching my videos and commenting saying I could have done it better! (which is also absolutely fine) John
John amazing tutorial and skills, love the way you slip in sometimes different method on going about. Hope you getting big bucks for your expertise. Keep them video coming.
I couldn't thank you enough for this tutorial...I am following a Python course on Udemy for the moment, and I found the section on web scraping incomplete...I followed this tutorial and it's brilliant...The indeed page is quite different including the html code, but the logic stays the same...I will put my code in the comments, it might be of interest especially for people using indeed in french
@@oximine Yea, indeed changed their code. I had a rough time figuring it out. the job title is no longer in the 'a' tag in the new html. At 9:13 in the video you need to use: ... divs = soup.find_all('div', class_='heading4') for item in divs: title = item.find('h2').text print(title) ... Reason being, Indeed now has the title of each job title within an h2 element which is in the class starting with heading4. So the code searches for the class heading4, once it finds it will search for the 'title' item in the h2 element Just look at the html and see where the "title" of the job search is in the new code. One thing is for sure, once you figure this out and understand it, you understand what's going on.
Very thankful to your videos John, we support your channel and your popular now in youtube, I wish you can make video also scraping LinkIn or Zillow website, these are demands from freelance sites
Cheers! I'd love an updated version of this. It seems They've changed it. I have a project due soon for which I'd like to scrape Indeed as the project is a job search app.
Hello John, Thank you for the good work. It would be nice to be able to be see how the job descriptions can be added to the data collected from the webpage as well.
Good Job, but in the loop you forgot to add "i" in extract function , the data were replication of first page, thanks a lot, plus more option to make location and job titile as parameter as well
I hope this message finds you well. I wanted to reach out and let you know that I've been trying to interact with your video, but I keep receiving a 403 response instead of the expected 200 response. I have checked my code, and it seems that I am setting the User-Agent header correctly to mimic a browser request. However, despite these efforts, I am still encountering the 403 error. I wanted to ask if there's anything specific I should be aware of or if there are any additional steps I need to take to ensure proper access to your video. I appreciate your time and any guidance you can provide to help me resolve this issue. Thank you for creating such valuable content, and I look forward to your response.
Thanks John for your valuable efforts In my case I wanna scrape data inside each container where there is a table of info then loop over every link in the page So I need to click the link of first job for example and get data from a table and so on so forth for the rest of the page It would be highly appreciated if you could consider similar case in your next vids. Cheers
Hi John, Thanks for this wonderful video. I am following the steps but struggling with getting Company reviews the same way. Can not seem to find the right div class. Could you please help there.
Hi John, Tks for a great video! I am studying Python with your video, yet it keep ending up with 403 message. Any plans to update the tutorial?Thank you:)
I followed your guide and edit a few lines of code so that I can scrap the whole job description. It worked well, but after 15 pages or so, I faced a captcha page and was unable to scrap anymore. I watched your user-agent video and changed the user-agent, still no luck. Is there any way I can scrap again?
Hi. How can I scrape multiple pages? Can I just define another function to scrape another page? Ideally I would like to add all the information to one database using sqlite.
Great coding structure and explanations. However, the website underlying CSS structure has change and as a results the codes no longer works. Is there a work around?
Hi John! Great Video, however could you please update or make a new video to scrape indeed in present day? The website's html is very different now and the same code doesn't work. Would really appreciate it!
Appreciate you responding! I have also been getting a 403 status code despite trying out multiple User Agents. Being a python noob i could really use that helper video! Ty!
I am new to this so wondered if you can point me somewhere that shows how to set things up before you start typing i.e. this seems to be visual studio but how do you set that up for python? - I am stuck before even starting! : )😞
Sure no worries I have a video on my channel for setting up Python, VS code and everything else you need to get to this point, I’m sure it will help you if you look for it on my channel page!
Hi John I appreciate your great tutorial. I have a quick question. I know this video is from two years ago. I ran the first part today, and I got the 403 code. The Indeed website blocks me from getting data. Would you happen to have any suggestions for me to resolve my issue? A newer method? Your answer would help me a lot.
Hello John, thank you very much for your Tutorial ! I wanted to ask you if you know how its possible to get the Business owner & Phone number (from the website) on my Scraping list ?
@@JohnWatsonRooney thank you very much for your response, in 95% of the leads there is no business owner in the article, is there a alternative ? Im doing cold calling in Germany
![30 ชั่วโมงคนเดียว บนรถไฟชั้น 3 โซเวียต [Around The World Ep. 2]](http://i.ytimg.com/vi/MlE9boAVqDA/mqdefault.jpg)
Great tutorial John, just a quick fix on the for loop, you forgot to add the i to the extract function: Made the changes to this
for i in range(0, 41, 10):
print(f'Getting page {i} ')
c = extract(i)
transform(c)
Great thank you!
Hi everyone,
I had the following error message :
File "/Users/admin/Downloads/Test.py", line 36, in
transform (C)
File "/Users/admin/Downloads/Test.py", line 22, in transform
summary = item.find('div', class_='job-snippet').text.strip().replace('
','') #trouver sommaire et remplacer new lines with noth'
UnboundLocalError: local variable 'item' referenced before assignment
ANY HELP ?
That's an excellent video for the following reasons:
-- the flow of the tutorial is really smooth,
-- the explanation is excellent so you can easily adjust the classes that existed in the time of the video to the current ones
-- and the iterations are detailed so every step is easy to understand.
Thank you so much for this video! Greetings from Greece! 🇬🇷
Awesome, thank you very much!
I am very surprised you only have 1500 views. This is one of the best webscraping tutorials i have come across. Can you do one for rightmove or zoopla?
Heheh already at 42k today 😁 well deserved
Worked flawlessly, I just had to edit a few things, like the classes and the tags. Nothing wrong with the code, just the indeed website that changed since you posted this video. Thanks!
Awesome thank you!
Hello, John. I have just started learning Python, and I'm trying to use it automate some daily tasks, and web scraping is my current "class". I really enjoy watching your workflow. I love watching the incremental development of the program as you work your way through. You are very fluent in the language, as well as the various libraries you demonstrate. I am still at the stage where I have to look up the syntax of tuples and dictionaries... (Is it curly brace or brackets? commas or colons?) so I find myself staring in amazement as two or three lines of code blossom into 20, and this wondrously effective program is completed in minutes... I am envious of your skill, and I wanted to let you know I appreciate your taking the time to share your knowledge. I find your content compelling. Sometimes I forget to click the like button before moving on to the next vid, so sorry about that. I just have to go watch it again, just to make sure I leave a like... Your work is very inspiring to me as a noob. I aspire to the same type of fluency as you demonstrate so charmingly. Thanks again.
Hi David! Thank you very much for the comment, I really appreciate it. It's always great to hear that the content I make is helping out. learning programming is a skill and will take time, but if you stick with it, things click and then no doubt you'll be watching my videos and commenting saying I could have done it better! (which is also absolutely fine) John
John amazing tutorial and skills, love the way you slip in sometimes different method on going about. Hope you getting big bucks for your expertise. Keep them video coming.
This channel has helped me a lot. Everything I know about web scrapping is thanks to John and his to-the-point tutorials.
Honestly, This channel is marvelous. It has helped me a lot. 'a lot' is even an understatement
Your style of code is so beautiful and easy to follow.
I couldn't thank you enough for this tutorial...I am following a Python course on Udemy for the moment, and I found the section on web scraping incomplete...I followed this tutorial and it's brilliant...The indeed page is quite different including the html code, but the logic stays the same...I will put my code in the comments, it might be of interest especially for people using indeed in french
Any update on your code bud? I'm trying to scrape indeed right now and the html looks very different than what's in the video
@@oximine Yea, indeed changed their code. I had a rough time figuring it out.
the job title is no longer in the 'a' tag in the new html.
At 9:13 in the video you need to use:
...
divs = soup.find_all('div', class_='heading4')
for item in divs:
title = item.find('h2').text
print(title) ...
Reason being, Indeed now has the title of each job title within an h2 element which is in the class starting with heading4.
So the code searches for the class heading4, once it finds it will search for the 'title' item in the h2 element
Just look at the html and see where the "title" of the job search is in the new code.
One thing is for sure, once you figure this out and understand it, you understand what's going on.
@@cryptomoonmonk The code works. Thank you for sharing.
Good work John, please use the variable i in the extract function to avoid duplicate results
Thank you. You saved my entire semester!
Very thankful to your videos John, we support your channel and your popular now in youtube, I wish you can make video also scraping LinkIn or Zillow website, these are demands from freelance sites
Sure I can have a look at the best way to scrape those sites
@@JohnWatsonRooney That would be great and don't forget the github link when you do. :)
Hey. Seriously. Thank you. I just downloaded soft and I can CLEARLY see why your vid was recomnded. You're an aweso intro into
bro with your help i have finished my project
Thats great!
Great tutorial! Nice and easy flow of code! As a beginner programmer, I really enjoyed this video! Thank you a lot!
This man is from another planet .....
Dude, thanks so much. You deserve much more views and likes. I didn't understand scraping one bit before this.
Cheers! I'd love an updated version of this. It seems They've changed it. I have a project due soon for which I'd like to scrape Indeed as the project is a job search app.
Thanks, I did a new version not that long ago the code is on my GitHub (jhnwr)
@@JohnWatsonRooney Unreal. Thanks for the quick reply too.
I just did webscraping on this website and youtube recomanded this video!🤣
Damn this was top quality my man, thank you!
Sir I became a great fan of u. Really interesting. A great skill to explain things in a better way to understand. Thanks a lot
Thank you!
keep up the good work those lines of code and the logic is sure fire.
You sir are a true legend. This taught me so much! I really appreciate it!
Thanks!
Hi great , but follow the same steps but i get 403 response not 200 , any help
your video has helped me a lot, thank you!
Hello John,
Thank you for the good work.
It would be nice to be able to be see how the job descriptions can be added to the data collected from the webpage as well.
Amazing content and teaching style! Thank you.
Hey thanks very kind of you
Ima download it thanks for sharing!!
dude, you're awesome. Thank you for this.
Nice guitars btw
Thanks Fabio!
Good Job, but in the loop you forgot to add "i" in extract function , the data were replication of first page, thanks a lot, plus more option to make location and job titile as parameter as well
can you explain where to add the i in the extrct function? im dealing with this very problem right now
@@nathantyrell4898 see at time 18:43 in line#35 just make it ..... c = extract(i) instead of c = extract(0)
Do you know how I could extract the full job description ? since the url changes based on the selected job.
Thank a lot, really helpful.
I'll love to see how to automate applying to them 🤔🤔🤔
Hello John, great video but unfortunately I keep getting 403 from indeed instead of 200 so not working for me.
Hi John, how did u customize the output path, i tried so many experiments but it did not work. can u help me with that?
Like usual very helpful John. Than you!
I hope this message finds you well. I wanted to reach out and let you know that I've been trying to interact with your video, but I keep receiving a 403 response instead of the expected 200 response.
I have checked my code, and it seems that I am setting the User-Agent header correctly to mimic a browser request. However, despite these efforts, I am still encountering the 403 error. I wanted to ask if there's anything specific I should be aware of or if there are any additional steps I need to take to ensure proper access to your video.
I appreciate your time and any guidance you can provide to help me resolve this issue. Thank you for creating such valuable content, and I look forward to your response.
I think its the best tutorial!!!! big thanks
This is great, thanks!
Great job, neat explanations! Thanks a lot!
really thanks for this wonderful work
GREAT VIDEO !!! Thank you very much
That's nuts!
Thanks for the video john! It was really helpful.
Thanks John for your valuable efforts
In my case I wanna scrape data inside each container where there is a table of info then loop over every link in the page
So I need to click the link of first job for example and get data from a table and so on so forth for the rest of the page
It would be highly appreciated if you could consider similar case in your next vids.
Cheers
Great tutorial thanks
god of scraping
This is brilliant, thank you!
I really appreciate this video, you thought me a lot. Keep up the good work
Great videos. Very helpful in learning scraping. Nicely done. Thanks!
Any idea how I can render/display the response data on a browser using HTML instead of saving it into CSV?
Your Aswere is much appreciated. Thanks.
Thank you 🙏🏼
Great tutorial! thank you John
Hi John, I want to know how to solve 403 error in scrapy.If u know please give explanation.
Nice and very informative video. keep going!
is there a way to get the emails of the company?
Hi John,
Thanks for this wonderful video. I am following the steps but struggling with getting Company reviews the same way. Can not seem to find the right div class. Could you please help there.
Hi John, Tks for a great video! I am studying Python with your video, yet it keep ending up with 403 message. Any plans to update the tutorial?Thank you:)
Can't get the 200, tried lots of mimic headers, cookies. But no results. Any advice?
Thanks! It was really helpful.
I followed your guide and edit a few lines of code so that I can scrap the whole job description.
It worked well, but after 15 pages or so, I faced a captcha page and was unable to scrap anymore.
I watched your user-agent video and changed the user-agent, still no luck.
Is there any way I can scrap again?
how were you able to get the full job description? Doesn't the url changes for each selected job id
Hi. How can I scrape multiple pages? Can I just define another function to scrape another page? Ideally I would like to add all the information to one database using sqlite.
Awesome tutorial!
I know you mentioned using a while loop to run through more pages. Could you give an example of how this might look like?
Such a great video!
How to GRAB job listening email addresses to e-mail CV in BULK at ONCE???. Thanks
hi,
why my object divs = soup.find_all("div", class_ = "jobsearch...") returns empty list?
After coding with the same code it's showing me 403 error can any one help me
great content hopefully more videos to come
really good for lead generation ty
Can u please teach us how to Automate or Scrape Facebook too. Thank u again bro for ur valuable teachings. GBU
really, really good content!!
Great coding structure and explanations. However, the website underlying CSS structure has change and as a results the codes no longer works. Is there a work around?
Thanks. I haven’t revisited this one yet, I’m sure there is a way I will look into doing an update!
@@JohnWatsonRooney Many thanks for you kind reply. Indeed, it would very practical to see the code adaptation to the new CSS structure.
GREAT. Can you tell me how to go inside these jobs urls? how to get jobs urls!?
You are awesome!!! Def Subscribing
hi John , great tutorial , how would you add the time function in this particular set of code .
Hi John! Great Video, however could you please update or make a new video to scrape indeed in present day? The website's html is very different now and the same code doesn't work.
Would really appreciate it!
Sure, I’ve actually rewritten it recently I could put out a helper video soon
Appreciate you responding! I have also been getting a 403 status code despite trying out multiple User Agents. Being a python noob i could really use that helper video! Ty!
@@oximine OK cool, I'll see if I can put it in for next week
Thanks Oxamine for this question
Great video minus the single letter variables
Thanks 🙏🏻 you so so much. Actually can’t thank you enough.
its so helpful brother
Cool 👍
Thanks
Thank you very much Sir ...
Hello John, I'm not able to crawl the website because of captcha. Who should I handle it?
I got a 403 on my status code, does anyone know any potential solutions?
Thanks!
I am new to this so wondered if you can point me somewhere that shows how to set things up before you start typing i.e. this seems to be visual studio but how do you set that up for python? - I am stuck before even starting! : )😞
Sure no worries I have a video on my channel for setting up Python, VS code and everything else you need to get to this point, I’m sure it will help you if you look for it on my channel page!
@@JohnWatsonRooney Thanks John - very kind 🙂 I will have a look again tonight
Hi John
I appreciate your great tutorial.
I have a quick question. I know this video is from two years ago. I ran the first part today, and I got the 403 code. The Indeed website blocks me from getting data. Would you happen to have any suggestions for me to resolve my issue? A newer method? Your answer would help me a lot.
I'm also tying the same thing and I am getting an error 403. Please help resolve this.
very neat!
Thanks for video....realy useful content
Thanks
It returns javascript not an html tag
Is there any tips you can offer?
This is amazing
Thank u so much🙏
perfect class
Shouldn’t it be c = extract(i) instead of c = extract(0)
Can someone please explain?
Yes I think it should be - I made a mistake on this on I’m the for loop which meant the same result was repeated instead of getting the next
@@JohnWatsonRooney Thank you! My friend pointed this out. Your videos are amazing! Keep rocking Buddy!
Hello John, thank you very much for your Tutorial !
I wanted to ask you if you know how its possible to get the Business owner & Phone number (from the website) on my Scraping list ?
Hey thank you. Yes if the data is there on the site it should be possible using the same methods
@@JohnWatsonRooney thank you very much for your response, in 95% of the leads there is no business owner in the article, is there a alternative ? Im doing cold calling in Germany
Amazing video! Do you have a copy of the code you used in this video anywhere?
hello it still working on 2024? i have 0 results how about and update?
Afraid this doesn’t work anymore
@john watson Rooney - Thnx for nice Video :) .
Please share code as well on github :)
Hi John, nice job. Do you work as a freelancer? How can I contact you?
my email is on my main youtube page if you want to reach out
cool!