API Endpoints? Get data from the web easily with PYTHON
ฝัง
- เผยแพร่เมื่อ 8 ก.ค. 2024
- This is another way to web scrape with Python. Find where the website gets its data from and then mimic this request programmatically.
Postman: www.postman.com/
Insomnia REST Client: insomnia.rest/
-------------------------------------
twitter / jhnwr
code editor code.visualstudio.com/
WSL2 (linux on windows) docs.microsoft.com/en-us/wind...
-------------------------------------
Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases
mouse amzn.to/2SH1ssK
27" monitor amzn.to/2GAH4r9
24" monitor (vertical) amzn.to/3jIFamt
dual monitor arm amzn.to/3lyFS6s
microphone amzn.to/36TbaAW
mic arm amzn.to/33NJI5v
audio interface amzn.to/2FlnfU0
keyboard amzn.to/2SKrjQA
lights amzn.to/2GN7INg
webcam amzn.to/2SJHopS
camera amzn.to/3iVIJol
gfx card amzn.to/2SKYraW
ssd amzn.to/3lAjMAy - วิทยาศาสตร์และเทคโนโลยี
Great video John - many thanks for sharing.
it's been months that I was trying to scrape a hotel booking engine and I could not do it through the classic way. I managed to do it with Selenium, by it was very slow and plenty of errors were happening.
Thanks to this video I managed to get the data straight from the API and thousands of data are scrapped within seconds and with very high accuracy.
Again many thanks and hopefully you will continue sharing these great skills !
Thanks John, you really made it simple. Liked , Subscribed and smashed the notification bell.
Thanks so much for this! This was a massive life saver for a project I am working on
Hey John Thanks for the video.
BTW, I watched the sunglass and livescores episodes. I think this way of getting data is my new one. THANKS!
you are the man who actually taught me something
[
Real interesting...alternative to what I so far have considered webscraping. COOL. Thanks for the effort.
You are a genius at explaining python!
Just watched this video and have subscribed. Exactly what I was looking for as I am learning Python now for a career change. Thank you and look forward to watching more videos in my journey
Thank you I have lots of more videos planned I think you will enjoy
@@JohnWatsonRooney Thank you excited for them. Just a quick question, I use Chrome and have been playing around with the chrome dev tool. Is there a video that explains how to optimise scraping using that and Python?
what a great video. I was having a time problem while scraping with selenium and with the help of this video I solved it thx.
bu adam goat ciddi
This video saved my life for a job. Thanks mate
Glad it helped!!
Thank you very much for the info!
Thank you so much! You are my life saver!
Thank you for this very useful video!
Thanks for this John! Smashed the like button
Much appreciated!
loading entire page with selenium is awful its very slow for scraping bunch of data specially when you need speed up work as well as you can this method it way better than selenium
thanks you again sir 🖤🙏🏻
you change my life sir thank you for this great information 🖤🖤🖤🖤🖤🖤🖤🖤🍻🍻🍻👌🏻👌🏻🍻🍻🍾🍾🍾🍾
Thanks. Been learning a lot from your videos. Question. What do you do when the API requires an auth token like a Bearer token? Thanks
Thanks, helpful!
Hey, great video - thanks! I have a question, I'm trying to build an app ontop of a website. Is it possible to locate the API used to make user accounts, for example? Would I be able to post my own data to their API to make an account. Thanks :)
Hi John, I wish you can create more example of this kind of scraping data from a website, getting data through API Endpoints, I'm stuck here now, this is the trickiest way of scraping data I think, it requires more investigation, so very much appreciated if you can create more examples like this.
Hi Mart - sure I have some videos coming up to show multiple ways to scrape the same website so this will be covered again.
How can I acces specific element from this result ? Result format is like string, There is no dictionaries or something, so I can`t acces for example if I want only 'Names' or something like this...
this the most exactly what i looking for, imsomnia doing good job, simple and work, i try postman it failed respon my curl header
Very useful! I'm stuck on one thing though. I need to extract a cookie from a network request that happens after I make my initial request. I can't make this request directly. Is there any way to inspect sub-requests like this with Python?
This is great, thanks! What are the benefits of going through postman/insomnia rather than just using the endpoint directly in requests.get(url)?
Thanks Andy! Postman makes it easier to tailor the request exactly how you want it by quickly changing parameters. Although I didn’t change anything in my example when you start to work with APIs more you see the benefits!
Help me a lot
Please do airbnb next!
thnaks man another great video! i am trying to use scrapy for a similar website (oddsportal1) but the URL of request keep changing by time and by each item so is there a way i can use the page URL and catch the req URL by a partial match? or is there another way to reach the jason data of each item in a list on the main page ? thanks again for such a good content i really feel like i owe you something :)
on point!
Hi John, I am trying to get the information from Network > XHR, but there is any request with JSON data retrieved, what can i do?
what if you don't see the json data in the webpage when you are inspecting. I notice some websites don't show it. Are some sites just hiding that data or is there a workaround for that. Thanks
That site may either be hiding it well, or not using this method. My best advice is to open the network tab and start clicking on links on the site, best ones to check are next pages or load more, and see what comes up
@@JohnWatsonRooney thanks for your response. 🙏🙏
Hi, I'm looking for an automated way to discover public API endpoints, not the manual version (inspect - network - XHR). Could you help me with any clue if there is an open source piece of software that does API endpoints search & discovery?
You can use a automated browser to load the page and have it dump out any requests made - this will show what you are after I think. I can’t remember the exact name but it’s definitely there. I have a playwright video on it on my channel
Hi Pal
I want to Build a website where all the crypto rates of staking apy are I found
all the endpoint api I have about 35 api which i want to get the data from cron 30mins
and save the data i need to my database then make a simple frontend so I can find the
best rate and the platform How can I go on about this
When I do this on Mac, I get the code but nothing in the output. My friend does the exact same on Windows, and runs it, and get the data outputted. Is there a method for mac?
Hi John, again really great video! I tries this method for a sports website but it returns a 403 error. Do you have any idea on how to solve this?
Thanks, 403 is forbidden, make sure you are copying all the headers over too including the cookie part and it should work
After both Postman and Insomnia have become more walled in tools requiring you to have an account, are there any other similar tools that you would recommend nowadays?
Yeah it’s a shame, I don’t use either anymore - I just use httpx, it has a cli tool and I do it from the terminal
how to get json url in mobile android?
which browser you are using i am hoping that aap by chrom but every thing is changed
this is old but i think its firefox, i usually use chrome now for this sort of thing though
Hi, I love your tutorials, could you help me, how can I use this wonderful information to create an API from scratch? I know programming, but I don't know where to start a Scrap API, I mean, an API of its own, where my data would come from these endpoints that you explained. Thanks John for the content
Hey! What I do is create scripts to fill a database at a certain time/times of day, then use a framework like django rest framework or flask to then serve your own api endpoints. Hope that makes sense
How can u solve if insomnia is blocked by website. I.e saying that I'm a suspected robot when doing the curl request?
Did you copy all the headers including the cookie?
So basically. I copied the API endpoint as curl pasted it into insomnia. Then I get suspected robot notification where the Json should be. Website in question is infojobs.net you can search data analyst
Why do you need postman/insomnia? I always just get the JSON with requests and get the data that way.
You don’t need it - it just helps see what is going on and a chance to see if you can change some of the parameters. Its also an easy way to generate initial code
Where to find amazon api
🙏
John. This is useful. But how to handle authentication with Insomnia? I have to sign onto the site of interest using a username and password.
I logged in using Chrome, and located the correct xhr line in Network which corresponds to the table data being returned. So I copied the request as you demonstrated via Copy -> Curl, and pasted that as a new Get request Taking a look at the data copied and pasted - it includes multiple lines including a line containing a cookie. The cookie line starts like this
-H "Cookie: __Host-nc_sameSiteCookielax=true; __Host-nc_sameSiteCookiestrict=true; oc_sessionPassphrase=
However when I send this, the response is either "401 unauthorised" or some xml saying "Current user is not logged in".
Only thing a bit different was that i could copy as :"curl (bash)" or as "curl(cmd)"
Thanks
I worked it out - i found a feature in Insomnia which fixed the problem. I pasted in the request but Insomnia gave errors. I found the "Auth" tab, and selected the "Basic" method - and Insomnia gives a dialog to add a user id and password. Then it finally connected ok with the right cookies
That is incredible. I was just looking for this info.
Thnks a knowledge for this man.
Subscribed 🫂