Build a Web Scraper (super simple!)
ฝัง
- เผยแพร่เมื่อ 8 ก.ย. 2024
- ⭐ Sign up for my Full Stack Developer Course: www.codewithan...
In this video I show you how to build a Web Scrapper in a super simple beginner friendly way using Node.js.
Web scraping refers to the extraction of data from a website quickly and accurately. Many people move onto selling their web scraping tools for money, either by building them as a chrome extension or API, or selling them to data capturing companies. So, the option to make money off this tool, is there for you too.
Common questions:
- This tutorial assumes you have nodemon installed globally on your computer. If you do not, use command: npm i nodemon -g
Part 2 For Express Routing: • Get Data from Backend ...
Final code for Part 1 and Part 2: github.com/kub...
___
⭐ Use promo code ANIAKUBOW for 3 months free of WebStorm IDE here (I get no commission from this link, but am in a partnership): jb.gg/get_webs...
⭐ New to code and none of this is making sense? Watch my '12hr+ TH-cam Coding Bootcamp' in which you will learn HTML, CSS and JavaScript Fundamentals completely from scratch. It's on my channel and its 100% free.
⭐ In most videos I use Tabnine as my A.I autocompletion tool. You can download it for free here (I get no commission from this link, but am in a partnership): bit.ly/tabnine-...
⭐ You can get a blockchain domain with my affiliate link here: bit.ly/get-a-cr...
⭐ If you would like to buy me a coffee, well thank you very much that is mega kind! : www.buymeacoff...
⭐ Sign up for weekly coding tips from my newsletter partnership: bit.ly/JS-tips
You can also find me on:
Twitter: / ania_kubow
Instagram: / aniakubow
#codingbootcamp #coding
thanks Ania its working perfectly, may i ask why we cant use an arrow function inside the each on line 15 of the code when you call the cheerio ($) function i try it and i got all undefine but i cannot wrap my head around the why....
I *think* it’s because cheerio is not configured to use arrow functions - but I can’t be sure - I haven’t looked into it enough :) I will have to when I’m back from holiday. I will pin this so others can see your great question 😄
@@austinps4026 This work for me:
try {
$('.fc-item__title').each(function () {
const title = $(this).text
const link = $(this).find('a').attr('href')
articles.push({
title,
link
})
console.log(articles)
})
}catch(err){
console.log(err)
}
because the arrow function scopes the function outside of 'this'
You ca use an arrow function with the each method, but you must pass 2 parameters, an index and the element:
$(".wathever", html).each((i, el) => {
const title = $(el).attr("title")
const image = $(el).find("img").attr("src")
article.push({
titolo,
image
})
Arrow functions use this from the calling context. Before arrow functions to do this you had to use the bind method when calling your function to switch the this context of your function. You could implement bind before it existed with apply when calling to have the function run with a different this… arrow functions just make what you usually want easier which is to carry the this context forward.. however many libraries exist that take advantage of the fact that the function you pass into another function can be called in away that changes the this…
For anyone wondering… you’ll need to make sure the nodemon package is installed. You can run ‘npm i nodemon -g’ to use it globally on your machine. Or alternatively, you can run ‘npm i nodemon -D’ inside the project directory to use it as a development dependency while the project is running. Great video Ania, keep it up 👍
Thanks for sharing this!!!! You are totally right I missed explaining this part- I will make sure to cover it in my next videos :)
As a total newbie it saved me thanks.
thx, but I like my demons
thank u!! love when u have an issue that's way over your head and it's solved in the first comment u read
Ohhh that will do it. Thanks for this
This comment doesn't belong to this video, >> You saved me in my senior project last 6 months while I watched tutorial about web development ❤️❤️❤️❤️❤️
Thanks so much for coding with me 💚
Love how you just get into it without any sitcomish-intros, just straight to the point. Your videos have given me the confidence to finally start applying to dev jobs. Thank you so much, Ania!
I am a boomer coding virgin except for a minute amount of C++ for my Arduino. No one has made sense to me before as there’s usually an assumption of a greater knowledge than I have. You are the first person who was coherent to me, every step explained and details of what is happening behind the scenes. Your beautiful diction helps a lot as well, thank you.
Clear and precise english. This woman explains well all topic she talks about. Thank you for being a content creator. It fits you so well
this channel is amazing!!!!
study coding and simping at the same time
Works purrfectly and super simple code to use and follow. Can you do a part 2 where you learn us to crawl() with this scraping code? I mean the entire site for an example
Yes please Khaleesi of codes do this!
Very cool simple project! One thing I noticed is that you must have nodemon installed globally (unless it's packaged with a version of Node I don't have?). For anyone that doesn't, the "npm start" script won't work. Of course you can just install it locally into this project with "npm i nodemon" in the terminal.
My man, thank you! Had this exact problem and you solved it. :)
Thanks man! This is the command for a Mac: sudo npm install -g nodemon
thanks alot man
well done bro! installed on the project ;)
Thanks Ania! Tried already two small projects of yours, and I must say, that you're a teacher, that speaks to novices as well. The samples you made are easy to follow, and simple enough to figure it out, how the packages work with each others. Of course that also shows a great understanding, how to produce working program, but having success with working sample encourages to do more. Thank you with appreciations!
This is so lovely for you to say and has made my day :) 🥰 thank you so much Arto!
@@aniakubow ... Sorry - forgot to wish Merry Xmas! ... and now I do. Happy New Year goes with the same package ;-)
Would love a follow up to this with more advanced scraping such as a page that is behind a login wall, or something that requires a query to be filled before getting the final web content. Thanks heaps!
Just had a web scraper project on work last week so used your idea. Thanx Ania. had some problems with nademon... But just installed nodemon wit npm i nodemon
Amazing!!! I’m glad this video helped and you managed to get nodemon installed :) thanks for leaving your advice for others too!
This was a very good intro video to web scraping, thanks!
As a quick tip, when running the `npm init` command you can just append `-y` at the end so it becomes `npm init -y`, and then it will proceed to skip the checks and create the package.json file without you having to press enter several times.
Thank you for such awesome tutorial. Dziękuję Ci.
Not sure why this channel/video showed up in my feed but most definitely this lady is a damn reasonable POA
Thank you, Ania! It worked perfectly. I had no idea how to complete this task. You saved my day and gave me a lot of knowledge and fun too. I send you love from Venezuela, you are a genius! ♥
sister used 70% of the vid to set up everything huhu.
anyways, it works at the end, and its what we came here for. Thanks!
This was a fantastic tutorial, and as someone that is a lover of JS, it's nice to see this approach.
I just have to say, I know that they are no longer required, but my brain just gets a completely unnecessary comfort out of using semicolons 😅 even though I can definitely agree that it looks cleaner without. I still have the desire to go through and add them everywhere! haha
Thank you so much!! I really appreciate your comment 💚. Haha yes , I’m team no semi colons for my projects on here haha. I used to use them at work, but even then an extension added them in for me 😛
hahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahah >=[
Wow! You explained this so well and left nothing vague. A lot of other tutorials leave out so many parts, assuming everyone knows what they're telling.
It's the first time I've come across your channel, and you just gained a subscriber. Thanks a lot!
hey just here to say "thank you" !!
you put a lot of effort into these videos i hope your channel grows fast and get to the top 🙌
The mere fact that you have a couple longboards behind you convinced me to subscribe and like this video. You're a cool coder 🌝
🤮
I can listen to you all day. I hope you were my web dev lecturer.
I am assigned as data engineer and really need to scrape some data from a marketplace. this is a blessing!
Thanks for the intro. After that I was able to move around and play around to build a scraper for blogspot
Seeing couple of software out there that helps people scrape data from the website. It's great to watch this video so I can be able to do the same by just learning programming from best people over on youtube.
Hello Ania, thank you for the quality of your videos !
I have almost 4 years of experience as a js dev, but still learning with your content.
Just one question, why you install express and turn the app into a server, since you only need the http client (axios) in order to scrap the page?
Same question
same ques
I am really proud that you are my teacher, I learned a lot from you, what is really interesting is that you are beautiful in appearance and also you have a beautiful mind you can explain the information in the simplest ways. Eagerly waiting for your new one🌹
Crazy, I just happened to need do something similar for a project and here you are uploading a video that helps a ton! Thanks!
Learning from Ania's style of teaching is easy and relieving. Gonna binge this stuff
learning my first line of code because my teacher Ania is stop eating beautiful
Damn. Wish there were videos to follow like this 20+ years ago when I was starting out lol. You make it very easy to understand
I wanted to learn the power of web scraping. This video shows just that! Thank you Ania
Thanks for the tutorial =) As a few others have said, an alternative option might be to use puppeteer. It has a nice syntax and is very flexible. You can simulate natural browsing using simulated click events and run additional commands such as taking screenshots of html content based on css classes.
yes, I design a template in 4 minutes to scrape title and url using a tool based on puppeteer: th-cam.com/video/rB5BHg0XyKs/w-d-xo.html
This is a wonderful explanation of every line of code. I've learned code through online resources, mentors and college. Some of this stuff I knew I had to do but I did not know why. Thank you!
You provide an amazing technique for web scrapping. It would be good if you explain also how to manage sites using an anti scrapping technique, for example instead of just populate a text directly, they wrap every character in a different tag or even replace some digits with a similar text characters and so on.
Wonderfully clear exposition. A very effective teacher.
Interesting, whenever I think of scraper I always think of Python (beautifulsoup).
This is really a very good perspective.
thanks for sharing.
You have helped changed the lives of many, thank you for sharing your priceless knowledge
Ania makes programming very interesting!
I have subscribed to many programming channels, however you are one of those who really add value..keep it up
thanks, Ania it is very good and working perfectly.
Omg, idk what words to say, i love you, this is so easy to follow
i will build anything that beautiful Ania tells me, web scraper, 20 room house, spaceship, time portal, anything!!!
I could listen to her say scraper all day long. The part of my brain that would enjoy it doesn't have that many lines of code either.
Hi, this is my first time here, and i really love the way you teach. thank you for your very informative tutorial.
I don't know anything about programming, but I watched the clip to the end.
Ania....it is just awesome...someday if you have holiday to Bali I will treat you and show you aound
Thanks for the video. You are obviously an expert.
Thanks for update my knowledge on npm. video was short and sweet.
I subscribed today because of my friend Hesham recommended you, so don't disappoint me
Big thanks, Ania!
With your help i managed to write a webscrapper to parse through all private repositories in my organization (had to add authorization as well), "read" it's package.json file, save it to a file, then run another conversion script that forms array of strings in certain way & then save it to .csv file, to be able to create a pivot table & analyze tech stack of our product :)
That is why learning docker was important for me. We can package our program to work on any machine without worrying about breaking changes from node or express. Thank you for the nvm tip!
yes docker is a life saver especially when you work on both mac and windows
This is great that you are going through the doc during video :-) It is great because you teach people how to read the doc... I do the same on my Python videos :-) Pozdrawiam! :)
Your tutorials are very consice and to the point. Its greatly appreciated.
Great video! Only feedback is: Use chapters so we can skip the steps to install node if we already have it for example :)
Everything you explained is quite clear and simple .So very helpful to learn.Thank you 😘
Thanks so much Sue!!! It’s great to hear this feedback as I’m always trying to improve 😄💚
@@aniakubow looking forward to your new tutorial!
Loving the bit more "fullstack-oriented" content! Actually have a project in mind where I could apply this perfectly. Thanks for the inspiration! ;)
Great video. I love the longboard on the wall. Thanks for posting!!!
Thanks, this is really clear and simple to follow!
Back in the day I had to build a scrapper by hand with only perl in a unix environment. It would start at a root, parse all the html, and then traverse to the sub pages. All as an intern at Xerox back in 94.
I've been waiting for a tutorial like this! Thanks Ania!
YEAHH!!!! ILL BE SCRAPING ALL OF YOUR VIDEOS 😍
I have already creating a project with use of web scra and django. But this will be very helpful to learn from you 🔥
Which Python package are you using? I don't like Selenium and I'm with requests-html but is no longer developed. The base requests package don't execute JavaScript.
Thank you so much i needed to learn how to build a web scrapper
You are the best❤️
Hi Ania, love the videos! Would love one on setting-up a web scraper to scrape every minute or so in a Litespeed server. Keep up the great work.
I built a hierarchical object model to hold the parsed HTML, made of nodes and parms, and a Trie to hold the key words for the parser. I wrote it in every language I've learned... Python, Java, C#, VB... You can create HTML with it too, and output to a string, so it's in every web server I build. There's several different search methods, so adding to the object model is fast and easy... no more counting
I think some of us, senior developers, will follow because we would like to see a surfing tutorial. .:D ..featuring the Lara Croft voice. Great coding! ;)
at 17:15 mins, i'm surprised, that you didnt get a "Cors" Error !!.... but great tutorial !!... thank you very much for sharing it !! 😃👍
This is great Ania, thanks for your video! BTW just as an idea, would be great to have a second and a more advanced part, that shows how to do it in a site like Linkedin that requires Login or a site that has dynamic ids and classes.
Cheerios!
Really clear and concise tutorial. Have a couple questions please: 1. How can you then use that data? Would you export to a csv, or database, to then use either on your own custom webpage or app? 2. Does this method run risk of being blocked by the webserver? Or would that not be an issue, as long as you're not scraping the data consistently?
Awesome Video. Please Do a Docker Begginer friendly project with react or MERN, I have been trying to learn that, With your videos it will be so much helpful.
Awesome Video.
Clear Precise and great Audio, easy to follow and listen to.
Thanks for the video! I had some issues with the "start" script on Windows but found that "node index.js" not "nodemon index.js" worked for me.
Thanks!
Thank you so so much Kaushik!!! It means the world!
Great tutorial. Very well spoken. Very well communicated. Great Job Ania.
Thank you so much. I always wanted to learn how to scrape the web I didn't know it's this easy.
Scrape it for what?.. if you don't mind me asking.
great video and nice accent! It was pleasant listening to this tutorial, I've subscribed : )
Thanks Ania, had a little bit issues but after i got nodemon package installed and realized i was missing a . for calling the class it worked like a charm :). One thing to note i think is that windows doesnt have a NVM, so i had to als install one of those just so i could make sure i am using same version as you
Subscribed! cause your contents are really great!
There's an extension called "Selector Gadget" which lets you pick elements on a page and it'll work out the best CSS selector to reach those elements which is great for scraping like this.
short, concise and useful. THANK YOU
Omg 😱, waiting for this👩💻😊
Honestly great tutorial. And coming from me, someone who rarely comments.
Thanks so much Kevin!
You have the most mesmerizing eyes I have ever seen. Your tutorial is excellent as well, you certainly know your coding.
its 6 am and I was waiting for this I love your channel
Thanks for joining Cris!!
just what i needed i successfully got src but was stuck at chaining thank you
Just created a stock data web scraping app thanks to this video!!! Thank you so much! I feel like a real pro lol :)
That’s amazing! Thank you so much for sharing this with me 😍
I was looking for a completely different thing, but you gave a good idea with this video! 🤗 Thanx! 😁
Please do a react js tutorial Ms. Ania.😍
Thanks Ania. You are a great teacher ☺️
You make it looks so easy.
Congrats on the 200k subscribers
Thank you Tarek!
Great lesson! Special thanks for enlarging the screen with the code!
You should have a million subscribers... you're awesome
Thanks Ania, you're a wonderful teacher. As a newbie, your lesson is as simple as it can be for me to understand. I even worked around it to try some other projects too that I can think off 🤠💕
Thanks! I'm surprised at how simple it was.
i got hooked with the "scraper" pronunciation
Love it, so simple and unique way of teaching. Love from India >3000
Awesome thanks, please more of advance scrapping
I don't exactly understand why do we need to start an express server if we're trying to scrap an existing website. Is this some condition to run nodemon?
It’s for part 2 :) we are going to show the results in the browser
@@aniakubow so it can be called at endpoint right?
I was wondering the same, probably it will use a front end to get the url and display the results !
In your amazing video you forgot to say that if you don't have nodemon installed globally already, it should be installed too.
this saved me, it wasnt working for me before reading your comment, thanks!