So COOOOOOL!! Thank you Mr. DevOdyssey. I was just wondering how to use Postman to scrape and in 10 minutes, I am doing it !! You are a godsend!! The website still works after 3 years. I will check out your other videos.
Thanks for watching @drnotebook! When Postman added the rendering feature "visualize" and I saw an example of what someone created, I thought it was really neat, and had to show it off. Thankfully, it wasn't too difficult to incorporate. I'm happy someone else had the same idea as me haha! Hope you enjoy my other videos and get some value out of them!
You're welcome Nicolas! Thanks for watching. Thats a fun challenge. Doing one challenge a day for Postman will really help you get acquainted with all that Postman can do. It's definitely a go to tool for me. I'm glad this video helped you get "unstuck". Best of luck on your Postman challenge!
Eye Opener! Love the Postman Visualizer finale! *Which link in your description box would you suggest for a Noob to begin with?* Never seen a Postman output before. Mind blown!
Hi @spicer41282! Thanks so much for watching 😊. Glad you loved the Visualizer finale and enjoyed video. As for the link, I would suggest the Intro to Postman playlist here, for a complete Noob: th-cam.com/play/PLtnFhV9xWljfvww1P_fNgb0cnA4H-zmZQ.html This is a good starting point if you have little to no familiarity with Postman and APIs. Even if you do have some prior knowledge, here you can scroll through the video or skip anything you wouldn’t find pertinent. If anything, I’d then start at the second or third video of the Intro to Postman playlist. Always cool to open others eyes to Postman output they haven’t seen before 😎. I hope the above link helps!
Great question @Robert Adhi Ruddyanto. Generally, this authentication isn't exactly supported, but likely possible. You can't exactly log in using this method, unless you maybe do so in some kinda pre-request script, but it should be possible to make it look like you have already authenticated. The way I would go about this is to first login / authenticate to the source that you're looking to scrape. Then once logged in, view the cookies that you have in your browser for that site. Copy those cookies and then paste them into Postman for your scraping request. Then that should be enough to authenticate you to your source. Just to note, this is not a a fool proof method, but the general concept should work. I've tried it in other ways unrelated to Postman. Additionally, be sure that the source you are logging into allows web scraping. Hope this helps!
Hi @David Irvin, Thanks for the reply. Now that definitely can work for website that do accept Basic Authorization. For example, most home router logins utilize Basic Auth, and if you send your router a Basic Auth request with admin and password, you'll effectively log into the router. Just depends on the website you are working with. Most these days will have more complex forms of user Authentication outside of Basic Auth. Thanks for mentioning this!
Amazing video, man with this. would i be able to "scrape" all products from online store? how many time could take 1000 or 20000 items? and of course i could make a lot of things with that data (like price tracker).
Thank you very much Antonio! Yes! You could do that, so long as their robots.txt allows for it, and they don’t use dynamically generated html via JavaScript. If not, you should be able to create a price tracker. When scrapping at the scale of 1000 or 20000 items, at that point I’d recommend creating a web server / crawler to do that for you, in python, nodejs, or whatever language of choice, so you can scrape with better performance, and would be quicker and easier to store the data in a database. Doing this in Postman would take longer that a web scraper/crawler. Whatever data is available via the e-commerce store, you can make whatever form of tracker, such as an inventory tracker as well! There are plenty of things you can do, it’s just a matter of what you come up with! 😊. Glad you enjoyed the video.
Thanks for watching! I’d be happy to make a part 2! Did you have something in mind you were looking for in regards to web scraping with cheerio and postman?
@BestBlackPeopleVine Gotcha, sounds like a fun video to make! Should be possible, the discord webhook part for sure. The only thing that may not be possible is webscraping from the retailer. That will really depend on the frontend stack the retailer uses for their website. If the javascript is dynamic, cheerio does not support that, and it would effectively fail. But if I can find a retailer that doesn't use dynamic javascript, then I can make a visualizer like that.
So COOOOOOL!! Thank you Mr. DevOdyssey. I was just wondering how to use Postman to scrape and in 10 minutes, I am doing it !! You are a godsend!! The website still works after 3 years. I will check out your other videos.
Thanks for watching @drnotebook! When Postman added the rendering feature "visualize" and I saw an example of what someone created, I thought it was really neat, and had to show it off. Thankfully, it wasn't too difficult to incorporate. I'm happy someone else had the same idea as me haha!
Hope you enjoy my other videos and get some value out of them!
thanks man, I am doing a 30 day challenge of Postman, and I was stucked until I saw your video.
You're welcome Nicolas! Thanks for watching.
Thats a fun challenge. Doing one challenge a day for Postman will really help you get acquainted with all that Postman can do. It's definitely a go to tool for me.
I'm glad this video helped you get "unstuck". Best of luck on your Postman challenge!
Thanks for the mention. Stoked it worked for you!
You’re welcome! Thanks for making the template! I couldn’t have made this video without you. And I had to give credit where credit is due 😊
Eye Opener! Love the Postman Visualizer finale!
*Which link in your description box would you suggest for a Noob to begin with?*
Never seen a Postman output before.
Mind blown!
Hi @spicer41282!
Thanks so much for watching 😊. Glad you loved the Visualizer finale and enjoyed video.
As for the link, I would suggest the Intro to Postman playlist here, for a complete Noob: th-cam.com/play/PLtnFhV9xWljfvww1P_fNgb0cnA4H-zmZQ.html
This is a good starting point if you have little to no familiarity with Postman and APIs. Even if you do have some prior knowledge, here you can scroll through the video or skip anything you wouldn’t find pertinent. If anything, I’d then start at the second or third video of the Intro to Postman playlist.
Always cool to open others eyes to Postman output they haven’t seen before 😎.
I hope the above link helps!
is it possible to scrape websites that require login with this postman method?
Great question @Robert Adhi Ruddyanto.
Generally, this authentication isn't exactly supported, but likely possible. You can't exactly log in using this method, unless you maybe do so in some kinda pre-request script, but it should be possible to make it look like you have already authenticated.
The way I would go about this is to first login / authenticate to the source that you're looking to scrape. Then once logged in, view the cookies that you have in your browser for that site. Copy those cookies and then paste them into Postman for your scraping request. Then that should be enough to authenticate you to your source.
Just to note, this is not a a fool proof method, but the general concept should work. I've tried it in other ways unrelated to Postman.
Additionally, be sure that the source you are logging into allows web scraping.
Hope this helps!
Hi @David Irvin,
Thanks for the reply. Now that definitely can work for website that do accept Basic Authorization. For example, most home router logins utilize Basic Auth, and if you send your router a Basic Auth request with admin and password, you'll effectively log into the router.
Just depends on the website you are working with. Most these days will have more complex forms of user Authentication outside of Basic Auth.
Thanks for mentioning this!
Amazing video, man with this.
would i be able to "scrape" all products from online store?
how many time could take 1000 or 20000 items?
and of course i could make a lot of things with that data (like price tracker).
Thank you very much Antonio! Yes! You could do that, so long as their robots.txt allows for it, and they don’t use dynamically generated html via JavaScript. If not, you should be able to create a price tracker. When scrapping at the scale of 1000 or 20000 items, at that point I’d recommend creating a web server / crawler to do that for you, in python, nodejs, or whatever language of choice, so you can scrape with better performance, and would be quicker and easier to store the data in a database. Doing this in Postman would take longer that a web scraper/crawler. Whatever data is available via the e-commerce store, you can make whatever form of tracker, such as an inventory tracker as well! There are plenty of things you can do, it’s just a matter of what you come up with! 😊. Glad you enjoyed the video.
Good video, thank you
You’re welcome! I’m glad you found it helpful 😊
Awesome
Thank you very much Ranga 😎
pt 2 pls
Thanks for watching! I’d be happy to make a part 2! Did you have something in mind you were looking for in regards to web scraping with cheerio and postman?
@@DevOdyssey Like how to webscrape from a retailer like a shop and send it to discord webhook with cheerio and postman?
@BestBlackPeopleVine Gotcha, sounds like a fun video to make! Should be possible, the discord webhook part for sure. The only thing that may not be possible is webscraping from the retailer. That will really depend on the frontend stack the retailer uses for their website. If the javascript is dynamic, cheerio does not support that, and it would effectively fail. But if I can find a retailer that doesn't use dynamic javascript, then I can make a visualizer like that.
@@DevOdyssey thank you so much bro
You’re welcome! 😊 I look forward to the challenge