Right now I'm learning Bootstrap from your udemy bootstrap course. Man, it's amazing! it's not just videos or slides, it has a very comprehensive code examples which accompanies whatever Brad says in the video. Brad did a hell of a job in this course! It's wonderful! I highly recommend it. I've not tried other brad's udemy courses, but if they are as half good as his bootstrap course, I'm sold! This man is a god among us! Love you brad, the great instructor and the awesome family man! God bless you and your family.
Let's scrape ... the scraping blog! I had a good laugh. Your courses are amazing and every now and then we get a good laugh. Keep up the excellent work.
I was just looking what a crawler is some hours ago. Now logging in to see this uploaded an hour ago! Are you reading the minds of your subscribers? :)
for code 'page = response.url.split('/')[-1]'. I thought it should be page = response.url.split('/')[-2] and it works for me. But I donot know why it works with 'page = response.url.split('/')[-1]' in the vedio.
Bro ur the best, I hate my life but these vids help make it better. I do IT and dev because I like it and because I don't have anything/anyone else for me. Thenk u for helping me learn, Been into python and C# lately, as I re visit JS, it only strengthens my skills, after thinking in New paradigms
@@TraversyMedia Looking forward to them both. Have a phone screen with Amazon coming up and was really worried about my lack of experience with vanilla stuff. What great timing!
Love your tutorial man. Thank you. With scrapy can we scrape millions of data with sequenced/scheduled interval to not get blacklisted and keep updating out file?
I cant seem to make this code work on Python IDLE. upto 22:24 and it gives me the output on scrapy shell but cant make it work in Python IDLE 3.8.2 please help.
Nice and I know you've said there's lots more you could do with this, but one obvious improvement you could make to this is to collect an array or a set of URLs as you go to ensure you don't crawl the same page more than once - as I think that's what this code might end up doing, as it is right now. Right?
great tutorial, but Im having trouble following along. filename='posts-%s.html' % page fails to number the pages so i just get post-.html and it overwrites itself for page 2, i assume. also tried filename = 'posts-{}.html'.format(page) with no joy.
@@bentraje I have the same issue, were you able to solve it? is it related to kite? EDIT: Found the problem, you need to replace this line OLD: "def parse(self, response):" with this one NEW: "def parse(self, response, **kwargs)"
so weird that i just started looking at scrapy this morning and boom... this vid drops. question - i cant seem to get vs studio to launch the debugger for a scrapy file. any secrets? thx
This is great content @Brad, would it be possible for you to explain some basic topics of SEO, I feel as engineers we often lack those skills, right now I am going through the pain. Again very grateful for every piece of content you put out there.
Booyamakashi I hadn’t watched the video by the time I commented, I honestly would’ve loved it more if it was a JavaScript, I still like the video as long as brad made it.
I'm confused regarding why sometimes we specify the attribute name (div) and sometimes we don't while selecting by class. For example: 13:54: No 'div' keyword 18:16: There is 'div' keyword
The o ly difference is that when we don't select with div, all elements with class will be selected, when we write with div, we gonna select the elements with class, that are divs only. With sayi g with div, you're more specified.
I may be mistaken but I believe there is already a default method named "parse" that is overwritten here. Nothing wrong with overwriting it but it could cause unexpected behavior for someone that doesn't know.
Hello! I want to do some web scraping but to find info on a certain thing. So normally, I would use a search engine to find the urls then from there, find the data I need. How would I automate the process of obtaining the URLS? The websites are pretty much the same ( I only really end up using 4 or 5 five websites with the data being a specific spot on the site). I would really appreciate any suggestions! Web scraping is such a good tool, but I need to automate the URL gathering process to accompany the Web scraping
Can you please explain why you used yield on lines 13 and 21 for final version of code? Does this mean parse is generator function in this case? How does this work under the hood?
Can Someone Help me I am getting two exception errors while putting the command Scrapy crawl posts: 1)KeyError :posts 2)Spider not Found in Posts Thank you in Advance! (Any Help Appreciated)
@@rishabhkothari1763 sounds like you are making it search the wrong place. are you sure you are set rightfully up with your virtual environment? try going to 'debug configuration' and change the 'source path', e.g. make the last path equal to PostsSpider.py. then it should be able to find the spider. Hope it helps :)
it is used to tell the command "scrapy crawl posts" where to get the data from. It's like the variable name that you don't used in the code but you use it in the command line.
Quick question, using xpath insead of css when generating with yeild creates in json file differently, I mean it puts all the titles first, then the dates and so on. It's a there a different sintax that I need to use?
I have 2 dozens of sites with jobs in europe. I would like to crawl and scrapp several data sets from it. Is there a way to do this in a generic matter to get it all at once ?
I tried to install Scrapy it was written some error, I read the documentation for installation they recommend to use Conda, I installed Scrapy using Anaconda Prompt, then I tried to start project (scrapy startproject ) and got "Fatal error in launcher: Unable to create process using '"d:\bld\scrapy_1587736721630\_h_env\python.exe" " error now and cannot solve it. Can you help please.
Some python packages conflict with other python packages, (or their dependencies may conflict), or you may have older projects that depend on older versions of a package, and maybe some even require you use an older version of Python. Virtual environments let you import the packages you need for a project, and use the versions you need and want (for both packages and Python), without having to worry about messing things up for other projects. It's generally a good idea.
![แข่งเพิ่มน้ำหนักใน 24 ชั่วโมง!! [Ver 2025]](http://i.ytimg.com/vi/20xn0g6k-ec/mqdefault.jpg)
I did a little web scraping a while back-- this video is very timely because I was going to get back to it!! I needed a refresher, thank you!!!
@@Tolrias Good to know, thank you!!
@@Tolrias thanks. I wanted to know this. Also could you link me to python scraping with headless chrome tutorial? A blog is also fine
Awesome video. I never thought I'd learn this much in 30 mins. Every second of video is full of useful information. Thank you so much
Right now I'm learning Bootstrap from your udemy bootstrap course. Man, it's amazing! it's not just videos or slides, it has a very comprehensive code examples which accompanies whatever Brad says in the video. Brad did a hell of a job in this course! It's wonderful! I highly recommend it. I've not tried other brad's udemy courses, but if they are as half good as his bootstrap course, I'm sold! This man is a god among us! Love you brad, the great instructor and the awesome family man! God bless you and your family.
Thank you so much brad. I purchased the Django course on udemy. Awesome content. Congratulations. U will soon reach 1M subscribers. Wow.
man after watching this video and executing this video in just one morning I managed to crawl an entire website in seconds. Thank you!!!!
Let's scrape ... the scraping blog! I had a good laugh. Your courses are amazing and every now and then we get a good laugh. Keep up the excellent work.
I was just looking what a crawler is some hours ago.
Now logging in to see this uploaded an hour ago!
Are you reading the minds of your subscribers? :)
YEAH, ME TOO. What a coincidence!
A great content creator that knows what people want. Brads a legend :)
No no, he's projecting his thoughts into your mind.
abj freakin brad get out of our heads lol.
😊 maybe, i do hear that a lot
I have use scrapy for many web crawling and web scraping projects. However, I still found this tutorial very handy.
It's a little out of my league since I am only a beginner coder but it was utterly fascinating! Thank you very much!
great lesson. After doing some webscraping with selenium, this finally made a lot of sense because I was lost a month ago
I was following along with a different site I needed to scap all made sense still always grateful Brad
This video couldn't have come at a better time... Thanks a bunch Brad... God bless
Thank you for the instructions, I like how the last minutes made things clear for me...
Your videos always great.
A lot of other coding vids built on python talk about simple math for 8 hours and I learn nothing.
for code 'page = response.url.split('/')[-1]'. I thought it should be page = response.url.split('/')[-2] and it works for me. But I donot know why it works with 'page = response.url.split('/')[-1]' in the vedio.
Like => Add to Watch later => Thanks, Brad. :)
thank you so much. My first time with Scrapy and you've been really clear. Great video. Tranks mate :)
thanks for your comprehensive description. i think this is good as start point.
Bro ur the best, I hate my life but these vids help make it better.
I do IT and dev because I like it
and because I don't have anything/anyone else for me.
Thenk u for helping me learn,
Been into python and C# lately, as I re visit JS, it only strengthens my skills, after thinking in New paradigms
Love your series! Thank you always!
Thank you very much for this tutorial! It's nice, short and crisp!
Such an awesome tutorial, sir!
Great video, both compressive and concise!
Great tutorial, the copy XPath from the browser was very handy
Ridiculously awesome video! Def an amazing teaching and great start to web scraping with scrapy. Dope Stuff!
I see Brad's Video I click it even though I don't know what's going on :P . Like it anyways.
MeGaZ haha, thanks I appreciate that ❤️
Hello brad ! Please could you tell me when would yould you share the front end course for the devBootcamp backend on udemy?
After my next course (20 vanilla Projects) which will be released within 25 days or so. I will start working on it
@@TraversyMedia Looking forward to them both. Have a phone screen with Amazon coming up and was really worried about my lack of experience with vanilla stuff. What great timing!
I love me some scraping but I did it with puppeteer and something else for work. My custom API did get blocked a few months later though...
In 8:04, why not use an f-string instead of the old percent sign way?
I dont search for this but i am kind of like to watch this so thank you
how are you know my thought. i looking for web scrapper and you make a tutorial with this ? are you an alien brad
Very good, please keep doing this tutorial series :)
Next video: How to overcome captcha with Scrapy :)
Hey brad m still waiting for your new vanilla javascript course can you tell us when it will available in udemy???
Excellent tutorial, as usual. Kudos!
This is a great Tutorial for crawling data
I'm getting an Unknown command: crawl error which is at 8:49 into the video. I can;t seem to find the error.
Any help here?
Love your tutorial man. Thank you. With scrapy can we scrape millions of data with sequenced/scheduled interval to not get blacklisted and keep updating out file?
very useful video super educative and clear
I cant seem to make this code work on Python IDLE. upto 22:24 and it gives me the output on scrapy shell but cant make it work in Python IDLE 3.8.2 please help.
Great Video simple explanation Thank you
Hah I was just watching Scrappy Coco.
I haven't found any better videos for data structure & algorithm. If you know something please make a vid about it.
Hey man, pls do a course on setting up a bespoke MVC system from scratch with express server , node etc.. going over the MVC fundamentals etc.
wish you did a whole series on this
Nice and I know you've said there's lots more you could do with this, but one obvious improvement you could make to this is to collect an array or a set of URLs as you go to ensure you don't crawl the same page more than once - as I think that's what this code might end up doing, as it is right now. Right?
great tutorial, but Im having trouble following along. filename='posts-%s.html' % page fails to number the pages so i just get post-.html and it overwrites itself for page 2, i assume. also tried filename = 'posts-{}.html'.format(page) with no joy.
I have the same issue. Did you managed to solve it?
@@bentraje I have the same issue, were you able to solve it? is it related to kite?
EDIT:
Found the problem, you need to replace this line
OLD: "def parse(self, response):" with this one
NEW: "def parse(self, response, **kwargs)"
@@AdamEfrati ah gotcha. didn't solve it. thanks for the reply!
so weird that i just started looking at scrapy this morning and boom... this vid drops. question - i cant seem to get vs studio to launch the debugger for a scrapy file. any secrets? thx
have you tried turning it off and on again?
This is great content @Brad, would it be possible for you to explain some basic topics of SEO, I feel as engineers we often lack those skills, right now I am going through the pain. Again very grateful for every piece of content you put out there.
Thank you for sharing the knowledge.
What about pages that are secured with middleware can you scrap them aswell?
Hi Brad, thanks for the video. Is Scrapy also able to handle SPAs and specifically with content that is dynamically generated with Javascript?
Very clear explanations :-) Thanks a lot !
4:38 When I type 'import scrapy' I get the message 'unresolved import 'scrapy'Python(unresolved-import)' I am using vscode
This solved the issue:
www.reddit.com/r/learnpython/comments/a97p09/unresolved_import_warning_vscode/
Scrapping with js, just in the perfect time for me
Thank you brad
You clearly watched the video if you think its in js.
Booyamakashi I hadn’t watched the video by the time I commented, I honestly would’ve loved it more if it was a JavaScript, I still like the video as long as brad made it.
scrapy is python lib lol
neesyler you’re right, scrapy and beautiful soup are python, puppter is JS
This is sooooo cool! Thanks a lot Brad!
I'm confused regarding why sometimes we specify the attribute name (div) and sometimes we don't while selecting by class. For example:
13:54: No 'div' keyword
18:16: There is 'div' keyword
The o ly difference is that when we don't select with div, all elements with class will be selected, when we write with div, we gonna select the elements with class, that are divs only. With sayi g with div, you're more specified.
@@nowieszco868 oh oh I see. Thank you for explaining :)
thx for this one, helped me a lot
Awesome! Thank you Brad!
I may be mistaken but I believe there is already a default method named "parse" that is overwritten here. Nothing wrong with overwriting it but it could cause unexpected behavior for someone that doesn't know.
you're really doing good job ... keep it up buddy... joey says
Can we also take the user input for the url to scrape in scrapy?
Hi, Is it a vscode extension to see a document at 4:57 How can I use that of 'docs'?
I found the answer myself;marketplace.visualstudio.com/items?itemName=kiteco.kite
Can you make a scrapping tutorial in js? There, maybe, so many persons who are looking for web scrapping tutorials in javascript.
You are literally the best
Glad to be here
Your video is best, thank you help me a lot!
Nice video man !
which extension do you use to see scrapy help in vscode ?
it's called "Kite"
Can you do a video about *unit testing* ? Please
Is there any upcoming course for Vue with TypeScript?
th-cam.com/video/TGW-z1bIWyg/w-d-xo.html
th-cam.com/video/Ww57lUS9dF4/w-d-xo.html
Hello! I want to do some web scraping but to find info on a certain thing. So normally, I would use a search engine to find the urls then from there, find the data I need. How would I automate the process of obtaining the URLS? The websites are pretty much the same ( I only really end up using 4 or 5 five websites with the data being a specific spot on the site).
I would really appreciate any suggestions! Web scraping is such a good tool, but I need to automate the URL gathering process to accompany the Web scraping
I'm stuck at 24:50. I run the program and no data returns, and no errors, either.
i am getting error 'str' object has no attribute 'css' 19:00
Can you please explain why you used yield on lines 13 and 21 for final version of code? Does this mean parse is generator function in this case? How does this work under the hood?
Wow! It would be great if you make JIRA course and Agile development, Love all your courses here in udemy, keep going sir.
sujal khatiwada you don’t need a course in Jira. If you don’t know it you are in some ways lucky.
@@nathanlewis42 but why, JIRA is used in industry
This is great. Any plans for a Python video that calls an external API and fills models?
Can Someone Help me I am getting two exception errors while putting the command Scrapy crawl posts:
1)KeyError :posts
2)Spider not Found in Posts
Thank you in Advance! (Any Help Appreciated)
Sorry I have the same problem
@@Vincent.Esders Hey! If you get any Solution, please notify me here in the comment box. Would Really Appreciate the help!
@@rishabhkothari1763 sounds like you are making it search the wrong place. are you sure you are set rightfully up with your virtual environment? try going to 'debug configuration' and change the 'source path', e.g. make the last path equal to PostsSpider.py. then it should be able to find the spider. Hope it helps :)
Is it possible to code this normally like in pycharm or sublime without using a virtual environment?
I think PyCharm automatically creates a venv for your projects..
@@aarongonzales3765 whenever i try to code this in pycharm i run into issues
@@trinimafia001 Something like package not found? If so, that is easy to fix.
8:41 I don't understand how that works. He declared a starts_url array and then doesn't use it?
it is used to tell the command "scrapy crawl posts" where to get the data from. It's like the variable name that you don't used in the code but you use it in the command line.
Quick question, using xpath insead of css when generating with yeild creates in json file differently, I mean it puts all the titles first, then the dates and so on. It's a there a different sintax that I need to use?
For me yield doesn't do the same when is generating, is just putting all the text under a single tag for each section.
thank you
one question
why is necessary to create a virtual env?
How about if I have multiple keywords, for instance, “123”, “apple”, orange” or even with date time, can I use these before crawling it?
Hi,
Please advice me on how to improve / speed up the scrapy process
Hello Brad, When 20 vanilla Projects course will release.. Waiting for that.
To be safe, I will say within a month. Most likely sooner though
I have 2 dozens of sites with jobs in europe. I would like to crawl and scrapp several data sets from it. Is there a way to do this in a generic matter to get it all at once ?
Also tried a scraping with a node app. I don't know why but the performance was really different from this Scrapy.
Sir I have watched the while series but I got one question how to bypass 423 status code as the user agent and proxy pool isn't working
In vs code how do you execute the python code in the terminal. Like when he starts the for loop?
What is the setup on your developer tools on chrome?
made it to 28:00 but my posts.json has no data.
cant we create a spider using genspider? or we need to do it manually.
I want to do scrape using scrapy in jupyter lab. How can i do that?
Very helpful, thank you.
Great tutorial. when I import scrapy in the spider.py file I get a 'Unable to import 'scrapy'' on VS code, is there something im doing wrong?
What if the Website is heavy on JS? and how to manage the robot.txt that explicitly disallows Scrapy? :/
do u have its course ?? or playlist where are other scrapy videos
I tried to install Scrapy it was written some error, I read the documentation for installation they recommend to use Conda, I installed Scrapy using Anaconda Prompt, then I tried to start project (scrapy startproject ) and got "Fatal error in launcher: Unable to create process using '"d:\bld\scrapy_1587736721630\_h_env\python.exe" " error now and cannot solve it. Can you help please.
My VS doesn't show any of the scrapy folders. But when I cd into my project name and do Tree it shows the folders.
how do i select all texts from for example "a class="new-class"?
i dont want the text from other classes
What is the purpose of Making virtual environment ??
Please explain.
Some python packages conflict with other python packages, (or their dependencies may conflict), or you may have older projects that depend on older versions of a package, and maybe some even require you use an older version of Python. Virtual environments let you import the packages you need for a project, and use the versions you need and want (for both packages and Python), without having to worry about messing things up for other projects. It's generally a good idea.
Does it support SPA web app such as Angular?