Web Scraping with ChatGPT Code Interpreter is Mind-Blowing!
ฝัง
- เผยแพร่เมื่อ 7 ก.ค. 2024
- In this video, we'll see how to do web scraping using ChatGPT Code Interpreter.
🔥 My FREE Cheat Sheets (ChatGPT, web scraping, data science): artificialcorner.substack.com...
Social Media
===========
✏️ Medium: thepycoach.com/
🔗 LinkedIn: / thepycoach
🐥 Twitter: / thepycoach
😺 GitHub: github.com/thepycoach
🎮 Discord: / discord
My Courses
==========
🔥 Join My Automation Course in Python: www.udemy.com/course/automate...
🔥 Join My Python for Data Science Bootcamp: www.udemy.com/course/python-f...
🔥 8-hour Web Scraping Course in Python: www.udemy.com/course/web-scra...
💰 Make money by writing about AI, programming, data science or tech: thepycoach.teachable.com/p/me...
Support My Work
==============
💵 PayPal: www.paypal.com/donate/?hosted...
Content
0:00 Web Scraping with ChatGPT Code Interpreter
4:42 Scraping Multiple Pages with Code Interpreter
7:04 Extra things to consider
Download my FREE ChatGPT Cheat Sheet: frankandrade.ck.page/08c94cf1c1
It will be nice if we can apply this approach on Facebook posts and websites?
HI , please can we still assess the the free chatGPT cheat sheet. please do share again
Mind-blowing stuff! Wonder if combining it with Proxy-Store's proxies would level up the scraping game even more. Any ChatGPT fans here?
A terrific video that has been the cornerstone for our new team members. Thank you for sharing it!
Awesome!!!! and extremely useful!
Thank you so much The PyCoach!
Thanks, that's really helpful advice - and indeed pretty mind-blowing!
thank you!!! never thought about the save page as html and then uploading
Thank you very helpful.
Very practical and helpful video with very detailed explanation!
Hey PyCoach, learning a lot from you. Thanks man!
You're welcome!!
Top tier content bro. Easy sub.
This is really cool❤
very nice! thumbs up.
You are the best, subscribed :)
Great stuff @ThePyCoach! Question now is, how do we automate this scraping for multi-pages instead of manually saving each page as a new html file?
Thanks for the video! I tried both methods to scrape Udemy and it didn't work, since the content is in a container. Do you have a solution for this?
how would you do it for the products page. So the tv plus the product pages details of the tv then so on.
Try saving all the pages at once to reduce manual repetitive work
I guess the manual download part is intended for demonstration. You'd do this programmatically.
cool
I love your contents especially this one..
Sir how can we offer this as a service. Who would need this service ?
Can you help out maybe with a resources thank your
Cool process. But what is the advantage of doing this process, instead of using pre-built scrapers already available like "Instant Data Scraper".
You can tell Chat GPT code interpreter to do the same thing for any amazon page, just upload the html and run the same process. Instant data scraper is clunky and the interface looks old, I'm sick of clicking an dragging with all these data scrapping app extensions. Also certain websites don't work well with data scraper apps to many blockers. try extracting data from harbor freight with instant data scraper. its a nightmare! no thank you.
can this be used to scrape multiple pages like 100 pages on a domain? i guess so, because one page would be silly
this is cool ! although It is probably not able to scrape any IMG/Links ?
Hi, is there any way now to scrape all posts and comments of a subreddit? I need them for my research.
Can you make a video like this that's scrapes the SEO keywords that a website is using?
YES...LIKE AHREFS
Hi Frank ty for the great walkthrough! I am trying to scrape google to deterniine companies near me that do not have a website. it appears to be a simple task to determine on the search results which ones to choose but for some reason gpt says the html structure of nesting elements is not allowing it to do so. Any ideas?
So have you done it?
Just make a list of all the businesses near you. Then identify among those which could benefit from having a website. BOOMSHAKALAKA!
If you used harpa or perplexity it will get the job done with no complications 😅
I've had a huge increase in scrapers across my entire domain portfolio.
Given the power of AI, I've managed to poison most of the scraped data and cause major headaches going forward.
What do you mean?
@@tryderrick upon detection, I feed a load of nonsense back in between legit data, and it totally ruins the dataset.
@@ronin4518 depends what you're scraping. Not my data
But not the automation I am looking for - will keep up with the options choose and one click. Maybe a long way to complete it but will do everything I need. Really python and is with some chat chatbot content writing - might use ChatGPT - but the content never passes ai detection like clause on it's first time.
You really have to do the most part of the writing for SEO, based on the 'draft' or outline of the content.
is it same work as like other language website like Hindi , Japanese , Chinese and many more language ?
the interpreter code does not appear in the beta feautures to activate it just i have only plugin why? do you have a solution please
Could have also just Did View Source, then copy paste, or maybe even use plugins with link reader and paste that viewed source URL in.
View source won’t work if the page is rendered dynamically
true that@@egauci1
This was pretty good, but I notice that none of these GPT methods for data extraction deal with pagination. How would we do this if we've got multiple pages of jobs on Glassdoor and you wanted to let chatGPT run through all of them?
Thank you for pointing this out before I went too far in this video, as I am trying to scrape dynamic content loaded with JS
So how do I scrape data from the product pages as well?
Just upload its HTML file and follow the same steps
Thanks, but what if we ask it to use link directly with help of beautiful soup and selenium? What would happen
When I checked the code I could see it's using Beautiful Soup. I'm not quite sure what would happen if you specify to use Selenium. Just remember that code interpreter is mostly parsing the HTML file
Hi, is Code Interpreter also available through the API?
Hi. I'm not quite sure
we need to all this thing for scrap? And scrap for what?
Is it not possible to provide the url and then get it to loop through the pages till it gets to the last page and then get it to export the results to csv
If you pass the URL, ChatGPT will probably tell you that it's not able to perform web scraping or that is against the ToS
you can ask chatgpt for python script to scrap that page. and then you can run that script on your computer to loop and scrap.
@@ThePyCoach I would have thought someone would have made a plug-in for this now
@@anuroop345 I tried this but it’s not as easy, Chatgpt keeps having trouble with the looping side, plus a few websites require a login to access their site
Did they remove the Code Interpreter plugin? How do you get it? I have a paid account. Thanks
Try enabling it in your settings. Open Settings > Click on "Beta Features" > Press "Code interpreter" toggle button
listening to this video gives me "Two Minute Papers" vibes. I was waiting for ""Alright everyone, hold onto your papers"
what a time to be alive
code interpreter isnt even an option was it removed? Maybe explain how to get it thatd be helpful
So basically you made a tutorial about saving a file and loading it into chatgpt?
Why would I do this? What is the use case for this
🎯 Key Takeaways for quick navigation:
00:00 🌐 Introduction to web scraping with ChatGPT Code Interpreter
01:07 📂 Saving a webpage as an HTML file
03:38 🧩 Extracting data from HTML using ChatGPT
04:08 📄 Exporting extracted data to CSV
05:03 🔁 Repeating the process for multiple pages
06:51 🌐 Scraping data from Glassdoor
08:13 🧩 Scraping specific data elements from Glassdoor
11:56 📄 Exporting Glassdoor data to CSV
12:37 🚀 Conclusion and call to action
Made with HARPA AI
This was actually slower than classic python scraping.
Hey would you be able to assist in python scraping script and looping the scape for another search ?
But it appears this only works with the paid version of chatGBT
Ok...but what I really need is for chatgpt to code a webapp like Ahrefs.
What is the purpose of doing this? What was the problem that makes your work its solution?
I don't wanna be that guy, but you can do this with a free tool called Harpa in a split of a second, it is nice to see the procedure but... 🤷♀
3:05 a empresa AVANCO SOCIAL tem o poder de transformar marcas e negócios por meio da habilidade em impulsionar redes sociais. É inspirador!
I used it to scrap some websites. Last week my account was banned, said my activity was suspicions.
How could that happen? Did you upload too many HTML files?
@@ThePyCoachNo i didn’t. I just uploaded direct link with scraper plugin on. And prompted the code interpreter to generate a new programming language that is more advanced.
that is why you need to use langchain and make your own code agent.
Also, only use interpreter to write code once, then handle it yourself.
@@curious_one1156how
Good. We don't like dodgy scrapers.
What will happen if you want to scrape 1 million pages? 😀
1 million page downloads, and 1 million ChatGPT uploads.
this video should have been a short.
This scraping is very manual. There are much better ways.
Next version of code interpreter
Could you please share them
@@izzyal-zyoud7563 webpilot plugin, can scrape any number of pages with one prompt and by just providing the page 1 link
@@mishos.2228 Won't that get you banned anytime soon using just links?
Yea this aint web scraping. this is html extracting data, where is the web scraping at?? you are doing it manual…
Dude 😂
This is more of a chatgpt-aided manual scraping. The processes aren't automated.
If you know web scraping this is really nothing :D
Watching your video made me angry
I really don’t like this title. It feels dishonest. There’s are much better ways to use chatgpt to scrape
Bro how to train a own AI 😅 and make like chatgpt
I think you need to use Langchain for that
@@ThePyCoach you can create a chatbot using Langchain how to make a chatgpt using Langchain
Not too useful. Stick to node puppeteer or perl/python command line scripts.
Clickbait 😅
Please remove this video, you wasted my 15 minutes 😢