One thing I really love about your tutorials is how you explain what might go wrong for someone who is watching in the future. No other tutorials I've watched explain in such detail. Thank you and keep up!
I have watched many programming videos, from some of my favorite programming channels with large followings, but she explains the key concepts and nuance details many leave out or barley mention. First video of hers I have watched...instant sub
@@bfunkydunk Off the top of my head i think you would use something like python, ETC for doing that on large datasets as it will be much more optimized and multithreaded to search lots of documents at once.. That being said it seems like the parser she is using here (cheerios) is just searching the dom, so if it is referenced in the dom (static) or loaded to the page i don't see why you couldn't search for a different/specific word.
Although i completely disagree with your approach of scraping data at runtime, BUT, i must say i have watched every second of this video for the only reason that you made it so perfectly precise no long no short, I wish every programming youtuber made this kind of video with a precise goal in mind, in fact i have forwarded this video to people who were asking me about a good intro to node js.
You make some of the best tutorials I have ever seen on YT. You have a great teaching style of anticipating what the student will have questions about and making sure not to skip over details that would be confusing. Thanks so much!
Won't speak for others, but this really helped me understand a lot of things even when I'm not looking to sell APIs anywhere. Please continue with your elaborative teaching style as these days it's a rare art. Thanks much.
Ania, your tutorials are excellent. Very well delivered and executed. Your elocution is perfect, so I can watch them at 2x speed and still catch every word. No waffle, all the details without diverging. Concise. You should have WAY more views than you do. One of the best. thank you - keep them coming.
Loved the video!! Advice for anyone following along: what she wrote is not a production ready code. It is merely for demonstrative purposes. Please do write a prod ready code when you're working on your APi for better support.
Ania I stopped this video three times inorder to call family members to check it out! AMAZING! You are a gift to all of the self taught developers like myself - MUCH LOVE FROM NEW ORLEANS!
I really enjoy this, not only she looks great, but i can tell for sure, if someone can explain something, and has a good way to do so, not too much blabla... Straight to the point. Its hard in a way, to get things explained well, cause there is just so much to talk about... Awesome work Ania :) will follow from now on
Your English accent is touching my heart ❤️😍 If I would learn coding from scratch again, I would definitely learn it with you. Finally I found a beautiful and knowledgeable coder that I can sharpen my skills with her.
@@helloukw I know that. Even in the past you can do with or without those ';'. For me who program in C and a lot of others languages, this just fell wrong, like something is missing. Whatever, today there aren't JavaScript programmers, just developers, most of all just use NPM and don't know what happen when build, even never seen the build package or not be able to read those.
You make it look so easy ! Thank you for having such a detailed video ! It is a perfect starter / intermediate video. You have a lot (a LOT) of details, but in the end, all I need in this video, and I don't have to open 10 new tabs to understand what is what ! As an extra, I also get to learn a few tricks ! Thx again ;)
Been a front end dev for five years and one year of back end on ServiceNow. I had no idea this was how you built an API. I've always used the api data, but never knew how it was built. I am surprised at how easy it is with express. Maybe not easy for beginners. I have some basic tutorials on my channel but this is nice. Just as good, if not better than a Udemy course.
@@prymestudio I have worked with a lot of entry level devs and they do not understand these things. They might be able to follow the code along, but they are not going to understand every bit of it. Callbacks, promises, GET, POST, these are not beginner level aspects. Express makes this easier obviously.
Realistically this is an extremely simple API with a majority of the heavy lifting API aspects handled by RapidAPI. This is more just creating requests that a custom API is using to return data, it's not building an API itself And I'm not saying this like it's a problem, but just want you to know that you probably shouldn't get API development confused with endpoint building that an API then uses
Hi Ania, Thank you for this tutorial. I first followed your 2 parts about building a simple web scraper, and then to get it working on mobile (because it didn't work due to using localhost) I followed this tutorial and got my web scraper working! A tutorial about what rapid API and Heroku are doing to understand this a bit better would also be really nice. Keep up the good work!
You speak very similar to that of Christopher Barnatt from explaining computers. I appreciate your content and love what you do! Your intellect is beautiful.
okay i cant lie, I was a bit sceptical about your video quality, but this is just beyond helpful. Its so easy to follow and understand like I hate js but this motivates me to use javascript. Thank you for this tutorial and amazing idea.
Great video. I know it might not be easy to nail down, but it would've been good to see how you can best judge the cost of hosting your API and how much to charge for access (also, I wasn't sure if you covered tokenised API access or whether that's handled elsewhere)?
You are a very good teacher, thank you, I am learning node and especially npm and you are by miles the best tutor I have watched. Also I love your accent!
Great video and definitely learned a lot. When doing the :newspaperId section, couldn't we just filter the articles array instead of filter/find on the newspapers array? That way we wouldn't be using time and resources to duplicate the effort if we already have all the articles
.filter() actually returns the object (which matches search criteria), So no need to run the filter again. Just get whichever property you want from that object.
This is the first video I from Ania I watch. Currently I am learning php/css/html and will start JS soon. I just wanted to say: ‘Ku bo’ means ‘for/with you’ in my language (Papiamentu) so basically this comment is gonna lead into a proposal: Ania tell me, can I be your main simp? 🤣
EAN codes. Each item in the world has a UUID code. Scrap Amazon, Ebay, Wallmart, etc. searching for that item. Build an API around delivery times and prices comparison between them. What do you think?
Awsomee!!! so easy to understand and so important and productive for the work!!! thanks Ania for share your knowllege. Im becoming a fan of all your videos. I just seen this blog a few days ago and watched lot of your codes!!!!
Really great video, never thought of this. Only issue I can see would be the policies of these newspaper sites being scraped. Many already have their own form of API already or probably forbid others from storing their data like this.
This was intended to be an educational video and web scraping is a quick way to achieve that. When you create your own API, you are responsible for making sure that you are legally able to do what you intend your API to do which is beyond the scope of this tutorial.
Great tutorial, many thanks for this. Question for anyone watching - Mine works at 28:30 but it cuts off the the domain of the URL and just returns the slug. Any help?
the most beautiful programmer I've ever seen, I'm still a beginner, Java script and HTML5 I'm not an expert, but seeing your explanation I became very enthusiastic LOL
Ania! Thanks for this tutorial. Exactly what I was looking for to do multisite scraping. One question for you, how would you save those results in firebase or any other type of db, so they don't get lost? And how would you then make an API from that mentioned db so they all there no matter the date?
Once you have the data that you need, you can create some code to filter out exactly what you want and store it in a hash/object or an array, or an object-array. Then, instead of creating yet another API, you can create some more code that displays the information that you want Rather than having the API’s display the information immediately.
If you want also, you can use an the date() method that gives you the current date, and store all of the information you’ve gathered based on date and other specifications. If you’d like to store all of the data regardless of date, you can simply combine everything that you scraped into an object of objects/arrays, and when you finally decide that you need to pull up any of that data regardless of date, you can always store the date along with that data so that you can search for it when needed.
Isaac Newton said "If I Have seen further, It is by standing on the shoulders of giants", Today Ania, we are standing on your shoulders to see further. Thanks for giving us that support.
In addition, don't forget you can take advantage of communicating with subprocesses in the OS (for things that run at that level), take deeper advantage of collaborative multitasking, and use queueing data structures to stash highly blocking requests that you might not want to store as pure javascript objects in case your application crashes/has bugs.
Thanks to the universe , still exist wonderful people like you in this wordl. I share your goal. The Knowledge must not be concentrated. Thanks to people like you the world could be a better place. A Big Thanks and a Huge embrace from South America, country Chile to you
28:38 When you keep refresh the page making more then 1 get request to that endpoint the data copies itself to the object again? instead of overwriting the current data like you think it should do. Hoping she teaches how to get around this. 34:40 this assumes all html data is the same across all webpages.
The declaration of the articles array should be within the scope of the request handler, not global. I would put that line of code right before the cheerio .each loop
To be honest I have been looking into dev side, however on a passive side note or a second job sort of thing. I find your videos to be quite helpful. Thanks.
Really good simple tutorial on how to use Cheerio to Scrape pages and Setup an API Endpoint, straight to the point. Kudos to that! One concern I have is that if Scraping the data from these pages are legal or not?
You have to be careful, it's bit of a grey area. Your scaping tool should respect the Robots nofollow/index meta tags, and your API should only provide a summary and link to the full content. If your API delivers the full content, or you're making money from someone else's content without their permission, there's always some risk of a lawsuit.
@James Pickering Yes, that's true. My question is about the financial aspect. Can I ask money in exchange of some else's publicly available stuff? I mean, can I ask money for providing an API which ex. scrapes your FB account?
@@thedevdavid Just because something is "publicly available" doesn't mean it's not copyrighted. In fact, this has been a massive debate in the US over the past 3-5 years... But here's a rundown on your social media content: if you own the photo (if you're the one who took it or you're the one who paid for someone to take it and have purchased the rights to that content), it is absolutely illegal for you to scrape that data and sell it. However, I will say good luck fighting large companies on that kind of stuff... They'll typically just run you through courts until you don't have any money left to fight it, unfortunately. I hope this helps :)
@@thedevdavid Haha, I didn't mean to imply that you were going to ;) I was mostly pointing out how ridiculous it is that large companies can get away with that kind of stuff with sleazy tactics, lol.
Great tutorial Ania, thanks :) My only question is whether it is 100% legal to make an API that scraps data from webpages like The Guardian(the one you show in the tutorial) and then post the API publicly?
There are no specific laws that do not permit web scraping/crawling...... and as long as you are not simply copying the "presentation" of publicly available data, e.g. cloning a web site and reusing it for commercial purposes, then you are not in breach of copyright laws either. One might also add that the scraped data links back to the source, so all credit is given. If anything, such an API will drive traffic TO the scraped sites, honestly can't see who'd complain about that :)
Nice idea, however, you should consider test automation for the data source scheme changes, and the second point is the site will register frequent access they may start blocking (which may even happen automatically by AWS routing services), which will have impact on "your API"s availabiliy, resulting in bad reviews.
You're right but I guess she aimed for an easy to understand example since there are many different topics she covered in this tutorial. Besides: I know it's an example but she should have pointed out that this api is in now way legal to run and even provide via public api considering all the newspaper outlets own copyright to these articles.
I love your videos so much while recovering from neck surgery and now is recovering and has to wear a neck brace for a while. hope you never had to wear a neck brace
this is fantastic, thank you so much for this. I just started getting my head wrapped around API. Now can you do a video on how to use this API or any API in your app?
21:23 Very cool, but I'm missing how app.get gets anything before the port is even open. Or is app.get just declaring a route to watch for and you have to create many declarations, each with a different function name?
In express everything is a middle ware, meaning those route and route handlers get executed when a request hit the server. the app.listen() will be the called synchronously
Thank you for a great tutorial Ania. Really inspires to dig a bit deeper and explore what more can be built based on such app. Though one question regarding the code. Why in "/:newspaperId" route you make a new axios get request instead of filtering matching results from the existing articles array based on provided ID in params?
Although is possible but this approach seems more desirable for optimising performance/profiling and negating code complexity issues- as an initial GEt request of all items and then running a higher order function like filter would be bulky if there were around 10000+ items in an array End of the day javascript is single threaded (:
Because it is expected normally for a RESTful api to serve /foo for "get all" and /foo/:id for "get one" Also keep in mind that you normally send articles "in packages" e.g. in sets of 50 (bc you might have 3 billion stored). so if you want some very old id you might need to reload 50 articles in a for loop on the backend over and over again until you find the correct one in the filter function
I speak Spanish and although my English is not the best, I understand you more with the subtitles and some phrases that I can understand than with other tutorials that speak Spanish 👍
![Vue.js Course for Beginners [2021 Tutorial]](/img/n.gif)
Thanks to you Mother of the Dragons for this helpful tutorial
👑🐉
😂😂😂🙂
I was trying to figure out why I recognized her!
😂😂😂😂
Lmao🤣😘
One thing I really love about your tutorials is how you explain what might go wrong for someone who is watching in the future. No other tutorials I've watched explain in such detail. Thank you and keep up!
I have watched many programming videos, from some of my favorite programming channels with large followings, but she explains the key concepts and nuance details many leave out or barley mention.
First video of hers I have watched...instant sub
sa te uiti si la mada. ea din viitor stie trecutul.
@@mybusinessnotyours2051 yeah okay buddy, you have never programmed in your life you are here for one reason
Am I the only one who feels it's a unique experience being taught by Daenerys Targaryen on how to make money from selling APIs?
🐉👑
correct
as indonesian it feels more bizzare actually, cos API in indonesian means fire 😂
@@aniakubow Could you convert this to a web scraper (specific words, etc. Not a specific site?
@@bfunkydunk Off the top of my head i think you would use something like python, ETC for doing that on large datasets as it will be much more optimized and multithreaded to search lots of documents at once..
That being said it seems like the parser she is using here (cheerios) is just searching the dom, so if it is referenced in the dom (static) or loaded to the page i don't see why you couldn't search for a different/specific word.
I came to watch something educative while I was eating my breakfast, I did not even realize that I watched the whole tutorial, excelente 🥳
Although i completely disagree with your approach of scraping data at runtime, BUT, i must say i have watched every second of this video for the only reason that you made it so perfectly precise no long no short, I wish every programming youtuber made this kind of video with a precise goal in mind, in fact i have forwarded this video to people who were asking me about a good intro to node js.
You make some of the best tutorials I have ever seen on YT. You have a great teaching style of anticipating what the student will have questions about and making sure not to skip over details that would be confusing. Thanks so much!
True and I especially like her face which is very fuckable btw.
1 year after video release but never late, you won my subscription, thank you :D
Won't speak for others, but this really helped me understand a lot of things even when I'm not looking to sell APIs anywhere. Please continue with your elaborative teaching style as these days it's a rare art. Thanks much.
This is the best tutorial on making your own API. Ania, you are too AWESOME!!
I randomly bumped into this video and so glad I did. Learned heaps and feel less scary about Node and the packages used. Thank you Ania.
I like the way you insert a pause at the end of the sentence, before the final word. It gives off a vibe of TV news.
Ania, your tutorials are excellent.
Very well delivered and executed.
Your elocution is perfect, so I can watch them at 2x speed and still catch every word.
No waffle, all the details without diverging. Concise.
You should have WAY more views than you do.
One of the best.
thank you - keep them coming.
simp, she's reading
Loved the video!!
Advice for anyone following along: what she wrote is not a production ready code. It is merely for demonstrative purposes. Please do write a prod ready code when you're working on your APi for better support.
She's done it omg. She's finally delved into the untapped market of selling API's hahahaha. Keep it up!
So that's sarcasm or what?
Ania I stopped this video three times inorder to call family members to check it out! AMAZING! You are a gift to all of the self taught developers like myself - MUCH LOVE FROM NEW ORLEANS!
Oh wow thank you so much TJ! 😍😍😍😍 this is so great to hear! Big love to you and your family too - honestly this has really made my day 🙌
What a concept! How did I not think of this??
Great work. Ania!
Yep, please create a Django version of this, I know the deployment to Heroku part but transferring Django API to RapidAPI is where I am struggling.
yeah dude ,definitely make these django API too!
Fantastic outline of the concept :)
new tools 4 the future!!
madalina stia. dar nu a vrut sa zica.
I really enjoy this, not only she looks great, but i can tell for sure, if someone can explain something, and has a good way to do so, not too much blabla... Straight to the point.
Its hard in a way, to get things explained well, cause there is just so much to talk about...
Awesome work Ania :) will follow from now on
This is one of the most interesting topics I have seen all year. I have for so long tried to demystify the true power of marketing APIs.
Are you coming from metal forging ages?
I literally finished your Async/Sync video series, and this video is the perfect follow-up. Thank you for all your vides! Please keep making them!
Gratulacje, twój akcent jest doskonały!
Easily one of the best tutorials I ever saw.
An original idea and a great, explanation, step by step, nothing left out. Subbed!
Your English accent is touching my heart ❤️😍 If I would learn coding from scratch again, I would definitely learn it with you.
Finally I found a beautiful and knowledgeable coder that I can sharpen my skills with her.
Great tutorial. I'm old with JS, since 2001. I liked what they have done with the language, but those lines without ending ';' freak me out.
You can add them to feel safe.
@@helloukw I know that. Even in the past you can do with or without those ';'. For me who program in C and a lot of others languages, this just fell wrong, like something is missing. Whatever, today there aren't JavaScript programmers, just developers, most of all just use NPM and don't know what happen when build, even never seen the build package or not be able to read those.
@@LuizFernandoSoftov yeah as a c developer too. it's a bad habit
@@LuizFernandoSoftov Thankfully Prettier adds them automatically in case I forget!
You have an amazing presence and such a clear way of explaining
Thank you Bella!
You make it look so easy ! Thank you for having such a detailed video !
It is a perfect starter / intermediate video. You have a lot (a LOT) of details, but in the end, all I need in this video, and I don't have to open 10 new tabs to understand what is what ! As an extra, I also get to learn a few tricks !
Thx again ;)
Robisz coś pozytywnego i kreatywnego dla innych :) Nie przestawaj! Tym bardziej, że widać od ludzi pozytywną reakcje
Nice! I watched at 1.25x speed and forgot I was at that speed when thinking "Wow, she talks as fast as I do"
Been a front end dev for five years and one year of back end on ServiceNow. I had no idea this was how you built an API. I've always used the api data, but never knew how it was built. I am surprised at how easy it is with express. Maybe not easy for beginners. I have some basic tutorials on my channel but this is nice. Just as good, if not better than a Udemy course.
Bro I feel a beginner would follow without any issues. As long as they know the basics of Js
@@prymestudio I have worked with a lot of entry level devs and they do not understand these things. They might be able to follow the code along, but they are not going to understand every bit of it. Callbacks, promises, GET, POST, these are not beginner level aspects. Express makes this easier obviously.
@@nick_jacob I see what you mean
Realistically this is an extremely simple API with a majority of the heavy lifting API aspects handled by RapidAPI.
This is more just creating requests that a custom API is using to return data, it's not building an API itself
And I'm not saying this like it's a problem, but just want you to know that you probably shouldn't get API development confused with endpoint building that an API then uses
Hi Ania,
Thank you for this tutorial.
I first followed your 2 parts about building a simple web scraper, and then to get it working on mobile (because it didn't work due to using localhost) I followed this tutorial and got my web scraper working!
A tutorial about what rapid API and Heroku are doing to understand this a bit better would also be really nice.
Keep up the good work!
Very happy that I have discovered this channel.
I have become a big fan of you Ania both professionally and clearly you are a wonderful person.
Thanks so much David! You rock!
Is this the best channel on YT ? why do i even ask ?! ofc it is !
Damn, the idea is so brilliant. Didn't know about API marketplace, by the way.
Thanks a lot!
the idea is so brilliant.
Straight to the concepts without beating around the bush !! Wonderful 👏
You speak very similar to that of Christopher Barnatt from explaining computers. I appreciate your content and love what you do! Your intellect is beautiful.
Thanks!
okay i cant lie, I was a bit sceptical about your video quality, but this is just beyond helpful. Its so easy to follow and understand like I hate js but this motivates me to use javascript. Thank you for this tutorial and amazing idea.
Thanks Ania for this fantastically interesting, valuable and well-crafted tutorial. You are a supernova in the ferment of programming tutorials.
I wish you know the NUMBER of sites I have scrapped after this tutorials. Thanks and Stay BLESSED Ania 🥰
Scrape responsibly. Alot of sights are putting up pay walls and bot deterrents because the bots are consuming too much bandwidth.
@@gdolphy Yes bro, definitely am doing it responsibly.
Proud Of You Dear sister , your contribution to computer science is appreciable.
Something I have been meaning to look into for a long time, the video is great and very clear. Thank you
Thank you, Ania. You are so wonderful. You are really a good instructor, especially for a newbie like me.
I wonder if there's any legal concern when we create scrapping tool API?
Less legal and more policy.
You can bribe the law with the money you made from APIs
I could watch her all day. Thanks!
Your videos are so well put together - you obviously work so hard on these 👍👍👍
Finally, I published my API... Now it's time to learn more javascript and build more... Thanks
Simply brilliant, thanks for explaining things crystal clear 👍
si pentru madalina e cristal. dar ea a baut din el.
You're a great teacher. You've earned me as a follower.
API in malay/indonesian language means Fire. Khaleesi you control the Fire (API) and taught us the way to use API. Thank u khaleesi.
🐉👑 =>🔥🔥🔥🔥🔥🔥
it also means "bees" in Italian; maybe she controls the fire bees as well 😯
Ania I am amazed on how valuable your channel is. Keep up the good job.
Big thank you ! cant imagine how much joy and fun i always have coding along with you. keep it up.
Ania is actually very great 👍
Subscribed!!!
if she is my coding professor, I will never skip her lectures
Excellent tutorial and practical overview on how to get an api up and running. Thank you.
Great video. I know it might not be easy to nail down, but it would've been good to see how you can best judge the cost of hosting your API and how much to charge for access (also, I wasn't sure if you covered tokenised API access or whether that's handled elsewhere)?
the best teacher on TH-cam 🥰
💚💚💚
Doesn't this lead to copyright infringement on the newspapers content?
Similar works Google too.
Why? You are just sending traffic to their sites. You are not copying anything.
Nice nice..thank you
@@AlejandroVivas you scrape the text only and doesn't show their ads, they don't get any revenue when you show their content in text form
You are a very good teacher, thank you, I am learning node and especially npm and you are by miles the best tutor I have watched. Also I love your accent!
Great video and definitely learned a lot.
When doing the :newspaperId section, couldn't we just filter the articles array instead of filter/find on the newspapers array? That way we wouldn't be using time and resources to duplicate the effort if we already have all the articles
.filter() actually returns the object (which matches search criteria), So no need to run the filter again. Just get whichever property you want from that object.
30:10 just a heads-up, jquery :contains only checks text. So if the url contains "climate" but text doesn't, that won't be captured.
This is the first video I from Ania I watch. Currently I am learning php/css/html and will start JS soon. I just wanted to say: ‘Ku bo’ means ‘for/with you’ in my language (Papiamentu) so basically this comment is gonna lead into a proposal: Ania tell me, can I be your main simp? 🤣
great voice for a teacher. soothing but engaging i likey
The thing is, wich API would we make? My hot take would be doing stuff and finding something you find usefull and not or badly done. Any better ideas?
start doing one. You'll quickly discover more as you go.
EAN codes. Each item in the world has a UUID code. Scrap Amazon, Ebay, Wallmart, etc. searching for that item. Build an API around delivery times and prices comparison between them. What do you think?
Awsomee!!! so easy to understand and so important and productive for the work!!! thanks Ania for share your knowllege. Im becoming a fan of all your videos. I just seen this blog a few days ago and watched lot of your codes!!!!
Really great video, never thought of this. Only issue I can see would be the policies of these newspaper sites being scraped. Many already have their own form of API already or probably forbid others from storing their data like this.
This was intended to be an educational video and web scraping is a quick way to achieve that. When you create your own API, you are responsible for making sure that you are legally able to do what you intend your API to do which is beyond the scope of this tutorial.
@@dokgu I didn't think of that 😮
Great tutorial, many thanks for this. Question for anyone watching - Mine works at 28:30 but it cuts off the the domain of the URL and just returns the slug. Any help?
the most beautiful programmer I've ever seen, I'm still a beginner, Java script and HTML5 I'm not an expert, but seeing your explanation I became very enthusiastic LOL
Lowl
You are one of the Greatest Developer
Ania! Thanks for this tutorial. Exactly what I was looking for to do multisite scraping.
One question for you, how would you save those results in firebase or any other type of db, so they don't get lost?
And how would you then make an API from that mentioned db so they all there no matter the date?
Such a good Question I am battling with the same dilemma.
Once you have the data that you need, you can create some code to filter out exactly what you want and store it in a hash/object or an array, or an object-array. Then, instead of creating yet another API, you can create some more code that displays the information that you want Rather than having the API’s display the information immediately.
If you want also, you can use an the date() method that gives you the current date, and store all of the information you’ve gathered based on date and other specifications. If you’d like to store all of the data regardless of date, you can simply combine everything that you scraped into an object of objects/arrays, and when you finally decide that you need to pull up any of that data regardless of date, you can always store the date along with that data so that you can search for it when needed.
Isaac Newton said "If I Have seen further, It is by standing on the shoulders of giants", Today Ania, we are standing on your shoulders to see further. Thanks for giving us that support.
In addition, don't forget you can take advantage of communicating with subprocesses in the OS (for things that run at that level), take deeper advantage of collaborative multitasking, and use queueing data structures to stash highly blocking requests that you might not want to store as pure javascript objects in case your application crashes/has bugs.
Thanks to the universe , still exist wonderful people like you in this wordl. I share your goal. The Knowledge must not be concentrated. Thanks to people like you the world could be a better place. A Big Thanks and a Huge embrace from South America, country Chile to you
28:38 When you keep refresh the page making more then 1 get request to that endpoint the data copies itself to the object again? instead of overwriting the current data like you think it should do. Hoping she teaches how to get around this. 34:40 this assumes all html data is the same across all webpages.
The declaration of the articles array should be within the scope of the request handler, not global. I would put that line of code right before the cheerio .each loop
To be honest I have been looking into dev side, however on a passive side note or a second job sort of thing. I find your videos to be quite helpful. Thanks.
it's okay to do the same for a school projet, but not for real projects
Yes...Truly.. Ania is Great..
Love from India.
Really good simple tutorial on how to use Cheerio to Scrape pages and Setup an API Endpoint, straight to the point. Kudos to that!
One concern I have is that if Scraping the data from these pages are legal or not?
You have to be careful, it's bit of a grey area. Your scaping tool should respect the Robots nofollow/index meta tags, and your API should only provide a summary and link to the full content. If your API delivers the full content, or you're making money from someone else's content without their permission, there's always some risk of a lawsuit.
You are maybe one of the prettiest dev I have ever seen. 😅
Aniu, bardzo dobrze tłumaczysz. Daję radę nawet ze swoim poziomem angielskiego ;-) Dziękuję 🙂
Thank you so much for sharing your expertise on how to build an API. Thanks again.
Great tutorial, Rapid API wasn’t even on my radar. Thanks 👍!
Hi Ania, thank you for an amazing tutorial!
Is web scraping legal, especially when used for commercial purposes?
Absolutely cool tutorial! Question: Is it legal to ask for money for API usage if you get the data from these sources (Guardian, etc.)?
@James Pickering Yes, that's true. My question is about the financial aspect. Can I ask money in exchange of some else's publicly available stuff? I mean, can I ask money for providing an API which ex. scrapes your FB account?
@@thedevdavid Just because something is "publicly available" doesn't mean it's not copyrighted. In fact, this has been a massive debate in the US over the past 3-5 years... But here's a rundown on your social media content: if you own the photo (if you're the one who took it or you're the one who paid for someone to take it and have purchased the rights to that content), it is absolutely illegal for you to scrape that data and sell it. However, I will say good luck fighting large companies on that kind of stuff... They'll typically just run you through courts until you don't have any money left to fight it, unfortunately. I hope this helps :)
@@thedevdavid Haha, I didn't mean to imply that you were going to ;) I was mostly pointing out how ridiculous it is that large companies can get away with that kind of stuff with sleazy tactics, lol.
You can bribe the law with the money you made from APIs 😂
I'm so glad i stumbled upon you. Subscribed! I'm extremely new to this and trying to learn.
Great tutorial Ania, thanks :) My only question is whether it is 100% legal to make an API that scraps data from webpages like The Guardian(the one you show in the tutorial) and then post the API publicly?
There are no specific laws that do not permit web scraping/crawling...... and as long as you are not simply copying the "presentation" of publicly available data, e.g. cloning a web site and reusing it for commercial purposes, then you are not in breach of copyright laws either. One might also add that the scraped data links back to the source, so all credit is given. If anything, such an API will drive traffic TO the scraped sites, honestly can't see who'd complain about that :)
always check the ToS of the website you're scraping from and consider if you want to be moral or morally flexible 😄
You explain things very well! in a way that anyone can understand... you're amazing and never stop!!
Nice idea, however, you should consider test automation for the data source scheme changes, and the second point is the site will register frequent access they may start blocking (which may even happen automatically by AWS routing services), which will have impact on "your API"s availabiliy, resulting in bad reviews.
You're right but I guess she aimed for an easy to understand example since there are many different topics she covered in this tutorial. Besides: I know it's an example but she should have pointed out that this api is in now way legal to run and even provide via public api considering all the newspaper outlets own copyright to these articles.
Simple, helpful, insightful and fantastic. Thank you so much for this course, learnt a lot
Really interested into this one...
why? can you not figure it out?
Who need Netflix for fun if they got your great videos for free.🔥 Just loved the way you explained.
I love your videos so much while recovering from neck surgery and now is recovering and has to wear a neck brace for a while. hope you never had to wear a neck brace
Instead of concatenating the urls as regular strings, you should consider constructing via the new URL() constructor which handles the base for you.
this comment is gold, thank you buddy
Thanks for the tip Donnie!
Ania you make me dream, literally!!
I started to study web dev just a few months ago, you help me a lot!
Thank you
🥰
this is fantastic, thank you so much for this. I just started getting my head wrapped around API. Now can you do a video on how to use this API or any API in your app?
it's an easy-to-follow tutorial, and such a mind blowing business idea...i never thought of this.....
21:23 Very cool, but I'm missing how app.get gets anything before the port is even open. Or is app.get just declaring a route to watch for and you have to create many declarations, each with a different function name?
In express everything is a middle ware, meaning those route and route handlers get executed when a request hit the server. the app.listen() will be the called synchronously
A great tutorial and the explanation is very clear. I've learn about scraping and developing API using NodeJS. Thank you Ania.
Thank you for a great tutorial Ania. Really inspires to dig a bit deeper and explore what more can be built based on such app. Though one question regarding the code. Why in "/:newspaperId" route you make a new axios get request instead of filtering matching results from the existing articles array based on provided ID in params?
other big question is how articles array will be updated? answer: only on server reload. btw good demo anyway
Although is possible but this approach seems more desirable for optimising performance/profiling and negating code complexity issues- as an initial GEt request of all items and then running a higher order function like filter would be bulky if there were around 10000+ items in an array
End of the day javascript is single threaded (:
@@vladbocharov3246 with every HTTP request to the server.... a frontend setInterval would do the automation
@@vladbocharov3246 great point! what about using node-cron? seems that could get a job done, e.g. by setting daily or weekly schedule
Because it is expected normally for a RESTful api to serve /foo for "get all" and /foo/:id for "get one"
Also keep in mind that you normally send articles "in packages" e.g. in sets of 50 (bc you might have 3 billion stored). so if you want some very old id you might need to reload 50 articles in a for loop on the backend over and over again until you find the correct one in the filter function
I speak Spanish and although my English is not the best, I understand you more with the subtitles and some phrases that I can understand than with other tutorials that speak Spanish 👍
Can't wait for 26 hours 😭😭... Btw great content Ma'am
how come?
@@Shoe_On_Head I mean, I posted this comment a day ago... At that point it was showing "premiering in 26 hours"