How to Scrape Any Website in Make.com
ฝัง
- เผยแพร่เมื่อ 24 ก.พ. 2024
- GET THE BLUEPRINTS HERE FOR FREE ⤵️
leftclicker.gumroad.com/l/zttjl
GET ALL BLUEPRINTS + COMMUNITY COACHING + WEEKLY OFFICE HOURS (LIMITED) ⤵️
www.skool.com/makemoneywithma...
In this video, I'll show you how to scrape any website using Make.com and AI!
1-ON-1 PAID CONSULTING ⤵️
intro.co/nicksaraev
SUMMARY ⤵️
First, I'll demonstrate how to gather data from virtually any source and transform it into structured information that you can use for various purposes using AI. You can customize outbound emails, build simple parasite SEO campaigns, etc in minutes.
Then, I'll show you how to scrape a large multinational data source like Redfin. We'll build custom parsers, apply sneaky headers to avoid being detected and I'll show you how to dump the data to a Google Sheet for later usage!
WHAT TO WATCH NEXT 🍿
How I Make $20K/Mo on Upwork with Make: • This Make.com AI Conte...
My $21K/Mo Make.com Proposal System: • This Make.com Proposal...
Generate Content Automatically With AI: • This Simple Make.com A...
MY TOOLS, SOFTWARE DEALS & GEAR (some of these links give me kickbacks-thank you!)
🤖 MAKE: link.nicksaraev.com/make
⚙️ APIFY: link.nicksaraev.com/apify
🖱️ WEBFLOW link.nicksaraev.com/webflow
⏰ RIZE TIME TRACKING: link.nicksaraev.com/rize (25% off!)
📧 INSTANTLY: link.nicksaraev.com/instantly
🔖 PANDADOC: link.nicksaraev.com/pandadoc
👻 PHANTOMBUSTER: link.nicksaraev.com/phantombu...
🦸 SUPERHUMAN: link.nicksaraev.com/superhuman
💰 HARVEST: link.nicksaraev.com/harvest
🖥️ SITEGROUND: link.nicksaraev.com/siteground
📸 GEAR KIT: link.nicksaraev.com/kit
🟩 UPWORK link.nicksaraev.com/upwork
FOLLOW ME
✍🏻 My content writing agency: 1secondcopy.com
🦾 My automation agency: leftclick.ai
🕊️ My Twitter/X: / nicksaraev
🤙 My blog (followed by the founder of HubSpot!): nicksaraev.com
WHY ME?
If this is your first watch-hi, I’m Nick! TLDR: I spent five years building automated businesses with Make.com (most notably 1SecondCopy, a content company that hit 7 figures). Today a lot of people talk about automation, but I’ve noticed that very few have practical, real world success making money with it. So this channel is me chiming in and showing you what real systems that make real revenue look like!
Hopefully I can help you improve your business, and in doing so, the rest of your life :-)
Please like, subscribe, and leave me a comment if you have a specific request! Thanks. - วิทยาศาสตร์และเทคโนโลยี
![I Built a $1M AI App [No Code]](/img/n.gif)
My long awaited community is now live! Apply fast: bit.ly/maker-school
Limited to 400. Price increases every 40 members 🙏😤
After building OpenAI module am facing rate limit error. Even after upgrading to GPT 4o am facing same issue.
Any idea how do I fix this?
Awesome tutorial Nick... I cant emphasise enough, not only how helpful this tutorial was but also the number of ideas this tutorial has given me - top 5 Channel for me!
Really ? Who else being in a Top 4?
Thank you so much Nick! Every time there is a brilliant video!
What did he write in the User Role - "Tell me about this website in JSON formate." What did he write after that?
One the best videos I have met this year so far..Thanks!
What did he write in the User Role - "Tell me about this website in JSON formate." What did he write after that?
Hi I want to say Thank you for being a great teacher. I appreciate you taking your time in explaining things.You are very easy to follow. I always look forward to your next video.
You're very welcome Michelle!
@@nicksaraev how can one get to your mentorship/course please ?
I got the same question. It seems that it is still in construction. Because we get curriculum in the video description. @@johnringo6155
What did he write in the User Role - "Tell me about this website in JSON formate." What did he write after that?
That's amazing, how did I miss this company. You got a new customer. Great job
That was fascinating to watch and very clear explanation. Thank you for sharing. I am definitely subscribing!
Welcome aboard!
What did he write in the User Role - "Tell me about this website in JSON formate." What did he write after that?
Thank you for your sharing, it has truly benefited me a lot.
Thanks Nick, super helpful. Will set this up right away :D
Hell ya man! Let me know how it goes.
thanks for teaching us sir. Appreciate it!
Fantastic stuff, thank you so much Nick!
this is pure gold :)
3 minutes in and I know how to scrape a webpage and parse it to text. THANK YOU!!!!
Glad I could help!
Dude, I cannot overstate how mindblowing this series is. There were so many things in this video that I had absolutely no idea were possible.
Also 1 bed 4 bath is crazy.
Hell ya man! Glad I could help. And SF real estate smh
This is very good explained!!!!!!!!
BRO YOU ARE ROCKING IT!!!!!
What did he write in the User Role - "Tell me about this website in JSON formate." What did he write after that?
Nicely brought!
So happy you found value in this man.
Another brilliant video, Nick!
Would be awesome to get a more in depth tutorial about Regex or what to ask chatgpt about (What are we looking for specifically) in order to scrape. Were you a developer before? You seem to know a lot about webdev.
Thanks again!
What did he write in the User Role - "Tell me about this website in JSON formate." What did he write after that?
Great info!
Very appreciative of what you’re doing with this series 🙏🏽
It’s becoming clear that having a solid understanding of JSON and regix is a must if you intend on building anything decently complex for clients. Any resources, courses, or forums you can point us towards?
Thanks again!
You can always ask ChatGPT for help with this kind of stuff. Explains to you in plain english
Thank you Jeffrey 🙏 agree that it's important. Luckily AI is making it less so-if regex is currently the bottleneck in your flows, you can usually "cheat" by passing the input into GPT-4 with a prompt like "extract X".
To answer your q, though, my education was basically: I watched a few TH-cam videos, same as you, and now just use regex101 for fine tuning.
Most of my parsers are extremely simple and real regex pros would laugh at them (but they work for me and my wallet!) .* is your friend
Hope this helps man.
Super insightful videos, much appreciated! Just fyi, in timestamp 28:20 you're trying to expand the window size. You can do this by clicking the little symbol with the 4 arrows.
Thanks for making this video; very helpful with a few automation projects that I have. I've never heard of make before. I've been spending the last two years making a local webhook application as a side project that basically does the same thing as make, but this site is so much better.
You're very welcome! I'm a dev as well and find Make better for 99% of business use cases. The only time I build something out in code these days is when a flow is extremely operationally heavy. Keep me posted 🦾
Great job ! thank you
Really great content, thank you for this video!
If I wanted to optimize your flow, I would check if the URL is in the Google Sheets document before calling the parsed URL and extract the data on the page.
Good thinking!
Hey Nick, amazing tutorial as always, you've massively helped me on so many flows - thank you! I actually managed to build a similar flow but instead of RegEx I used an anchor tag text parser with a filter that checked for the presence of a class of "page__link" from element type since all page links had that. Would you say there's anything wrong with this if it works for the use case?
Hey Nick, love the videos!! just had a few questions. Would love if you could help us out. What is your business model like? Do you offer clients a subscription model? or a one shot payment? and what do you think we should apply to our business model as well, considering we r looking to rope in new clients and remain profitable over a period of time. I ask this since the websites we will be using have a monthly subscription fee and a limit on the API / operation requests and if the requests exceed the limit of the plan purchased, how do you tackle that? It would be of great help if you could make a short 10 min video on this or maybe a reply to the comment. Love the series!! Keep up the good work!!
Great Job!
Thank you Terry!
Super helpful, thank you. Any chance you could do a tutorial on how to scrape sites that require logging in?
Hey Nick great video!! I just have a doubt. If you this module once for one url and then put it on sleep, how do you scrape the other urls. I didnt quite get the hang of it how it happens so it would be nice of you to explain in briefly. Thanks in advance!!
AWOSOOOOME🔥🔥🔥🔥
Again, your instructional video’s are so informative. Very much appreciated! Could you post how i can visit multiple websites from sheet? Would i add a sheet at the front and another at the the end to assess the next row?
Appreciate the support! Absolutely-here are steps for plugging multiple sites in:
1. Create a Google Sheet (sheets.new) with a column labelled "URL".
2. In the Make scenario builder, search for Google Sheets connectors. You're looking specifically for "Search Rows", which has a built in iterator. Make this the trigger of your flow.
3. Authorize your account, select the sheet from earlier in the modal, etc. Set "maximum number of returned rows" to however many you need.
4. Use the output from the "URL" column as the input to the rest of the flow you see in the video. Remember that since "Search Rows" is now a trigger, if you turn this scenario on it'll run every X minutes. So if you don't have a designated flow you might want to make it "on demand" and just run whenever you need to process sites/etc. You can then make another Google Sheet to collect the output and use the "Add Rows" module to fill it up.
Hope this helps!
Thank you very much Nick for your amazing videos! I'm a beginner and this question may sound dumb but I'm running a scenario with 2 text parsers following each other. The first one runs 1 operation but the one following that's using the same input data runs way more operations. Do you know where that could be coming from? No hard feeling if you don't have time to answer ;)
Golden content
So glad you find it valuable man
came for the web scraping insights, stayed for the pearly white teeth
Brb getting a Colgate sponsorship
I am curious how you would tackle getting around a "click to reveal" phone number. It requires 2 clicks to find the phone number.
Gonna check out, wonder if could be used for comments on a post, or twitter, example someone saying they want something, than boom you can respond
Thanks Raiheen! You could, although there are probably better solutions to this. Facebook/Twitter/etc often hide comments behind a "reveal" mechanism like scrolling or a "Read More" button which makes scraping them difficult (in addition to their security and tight rate limits).
That said, anything is possible in 2024! You could run a search every X minutes using a search bar and scrape the top X comments. You'd use an intermediary DB of some kind to store the comment text, and then for every comment in your scrape, if that comment doesn't already exist, you could fire up a browser automation tool like Apify and log on to the platform in question. You'd then have GPT-4 or similar dream up a response and post it using JavaScript.
Hope this helps man 🙏
Hi Nick, how you doing?
First of all I wanna thank you for everything you are doing for us.
I tried to use this automation on different websites but there are a lot of websites where the code has 2 or 3 times almost in a row the same link for the same house/product, so when you use regex you get repeated results. How can I filter this or put some type of condition that makes me not have duplicate results and save me operations?
Thank you
Great stuff. the shared Hidden_API_Masterclass.json seems incomplete, would be great if the complete json could be shared.
Great video! What if the page you're trying to scrape requires authentification? Like the "my profile" section of uber or ny other company.
In the OpenAI does it need payment to generate an output (credits) or you just need to get the api key and that’s it?
so it scrapes but you have to sign up.. hardly feels private does it
Hi Nick thanks for the video, I'm having a problem with the parser...it's not parsing down the text for me like it show in the video. Any suggestions on this?
This is great and well explained. After watching the full length of the tutorial, I'd rather opt for using a web scrapper tool until I'm good with using regex. Btw any resources on learning regex?
Thx Tobi! Frankly I just use Regex101 for everything (regex101.com), the highlighting as you set your search up is extremely helpful. If you were to quiz me on tokens/selectors without a tool like this I'd probably know fewer than 50% of them 😂
Hey Nick, What would be a prompt of GTP4 to extract those URLS and build the regex?
How do you get past authentication to scrape for resources that require a sign in?
Hey amazing work just you should cut in post what did not work I got so lost and trying to do at my home can't make it happen :(
Managed to get a 200 reponse, on the first step. But it appears that some of the html is hidden. Seems like there is a delay in all the data being populated. I added all the header info. Thanks for the tutorial.
Unfortunately i couldn't get this to work. The parsed HTML seemed to have different data to your example and i couldn't figure out Regex. You mentioned it could be done with Chat GPT - it would be helpful to know that approach also.
Is there a way to use proxies for this? I just feel like it’d be pointless to get so deep into this without one
Thanks for the Tutorial.
Does this also work on Amazon listings?
Glad I could help. Yes it works on Amazon, though be wary that their bot detection is much more sophisticated (see another comment where I discuss how to scrape reviews).
Don’t understand why you moved the last sleep before the sheets but otherwise great explanation
amazing
Thanks so much for the tutorial! Just a question: how do you deal with pagination when scraping data?
Thanks Artem 🙏 you'd create a separate route for the scraper so it can iterate over each page, then add each page's data to an array (using the add() function or similar). On your main route you'd then add a Get Variable module and pull the array contents. Hope this helps.
@@nicksaraev Thank so much for sharing, Nick! Hopefully, I'll be able to help you somehow one day :)
Awesome
How can I code the headers for scraping data from TikTok? Is a specific type of header required to imitate a legitimate user or device?
to get it right ... u feed the whole html-content to gpt. so u pay the input-tokens for all of this. isnt it possible just feed the body or a single container ID or Class ?
I can't get past the HTML to Text module. It keeps giving me an error message: BundleValidationError. Maybe poor HTML on the website I'm scraping? Anyway, thanks for the information. So much to learn!
Great project to learn from. Thank you Nick.
my friend thank you so much for your videos, I really appreciate it, again any Go High Level Platform Review?
I will absolutely do one on GHL, I used to sell their platform as an affiliate actually. Tbh I don't like their "automations" one bit but it's important enough to go through. Probably next month as I finish the course and the rest of my videos-thank you for the idea!
@@nicksaraev thanks buddy, I tried it but it was a bit overwhelming for a beginner.
Thank you, can I use this to web scrape all reviews of a product on amazon
Absolutely, just checked for you. You have to do it in two parts:
1. Feed in the Amazon product URL to a Request module like I show in the video. Then scrape HTML and parse as text.
2. Somewhere in the resulting scrape will be a URL with a string like /product-reviews/. You need to match this (can use regex). Then make another request to that URL for product reviews.
Amazon's bot detection is very good so be careful you don't get rate limited 🙏
Hi Nick, is it possible to scrape a page that does not have an API and that you have to be logged into, please?
Have you caught wind of VideoGPT making waves? It's your ticket to seamless, professional video content.
What camera are you using, is it Lumia by any chance? Thanks!
Because of this comment & a few others, I just published a full gear list in the description! Including camera, lens, lighting, etc :-) all the best
Hi Nick! Thank you for your education! But.... How to solve the issue with Status code 403?
i think adding " " to the value section of the header's fixed this for me!
sometimes the regex say match on regex101 and then in integromat it doesn't..
Thank you for the information. BTW, your copyright has not been updated. :)
What did he write in the User Role - "Tell me about this website in JSON formate." What did he write after that?
Doesn`t make support css selectors?
How would I use this if I have to login to a site in order to scrape it? Is there a login prompt to add before the site prompt? Thanks for all the info!!!
Happy you found this valuable Daniel!
It depends on the site-sometimes you can just pass a username/password in the HTTP request module to get the cookie, other times you need to use browser automation tools like Apify. I recorded an in-depth video on authentication here if you're interested: th-cam.com/video/R8o7V39NSSY/w-d-xo.html
Hope this helps 🙏
You only logged the first listing on your Redfin search. How does it loop to the second and so on?
Great q. The flow automatically loops because the "Match Pattern" module outputs multiple bundles. When multiple bundles are output by a module, every module after that module runs anew for each respective bundle.
Hope this helps 🙏
How would you address the legality of scrapping?
After building OpenAI module am facing rate limit error. Even after upgrading to GPT 4o am facing same issue.
Any idea how do I fix this?
How about content behind a pay wall?
Just recorded a video to answer this (hidden APIs)! Hope it helps you.
but this only works if the content data you want to grab is only in text form on the web page, but if it is dynamically created let's say for a js script or something it wouldn't be able to grab the desire data... i.e... if i want to grab a price data from a web page... the content grab by Make a Request module would get something like this PRICE: $ 0.00, but in the web page it is like this PRICE: $ 3.70... but this last one is dynamically created and doesn't show this way in the make module...
Can we add proxies to these flows?
Yes, definitely. You'd just replace the URL in the HTTP Request module with whatever your proxy is and then add the proxy-specific data (most proxies will require you to send credentials, the URL you want to passthrough, etc).
@@nicksaraev can you show us an example of this?
At the 13:31 mark, could you have your ChatGPT with custom instructions entered versus writing JSON to get a better email intro?
Yes, definitely! PS the quality usually goes up if you let it output plaintext. This isn't as relevant for my purposes but something to keep in mind if you're generating content (say blogs etc)
need to scrape website for an ai web app to allow me to put Q&A, company info etc to fields on web app is that possible?
Absolutely. I did something similar for a data viz SaaS a while back. You'd have to find a way to parse each of those strings (Q&A, Company Name, Company Description, etc) and then pass them to your app db. You can use AI for this if there's no consistent pattern-something like "Categorize the following text into XYZ using the following JSON format".
Hope this helps man 🙏
Brother can I scrap translators details from translation websites?
For sure man. If you have a specific site in mind just drop it below and I'll take a peek 🙏
This does not seem to work with linkedin
Is it possible to scrape Wikipedia?
Not working following your steps
bro.. that drink was empty
I don't understand how to find a regex
Skip to 2:47
Hello,
How can I convert a super large XML FILE (literally a huge stack of file archives) into a simple site Jyson?
I'd sing at anyone's wedding if anyone can share.... lol jk but, truly would be very thankful for any suggestions
And nice channel 100 you got a new sub here ⚜️
Hi, are you available to build something like this fo me or can you refer someone? Thanks Dan
Hey Dan, happy to chat. Shoot me an email at nick@leftclick.ai and if I can't help I'll refer you.
First, you should explain what scraping a website is. 🤔
There are other videos for that this is for those taking the next step
Curious why you decided to watch the video if you didn’t know what it was
Firstly, thank you, but the title should be: Any website without Cloudflare.
Agar WhatsApp pe contact block hota hain to call nahi lagta Hain . Message nahi jata hai, aur mere pass koi number nahi hain.to hum apna matha phone pe patakage to tumko telepathy se phone lagega?
can you scrape banking account ?
Only my own 😫
facilísimo
Sono d'accordo 🙏
Where do I get the regex app?
Head over to Text Parser (orange box under Tools) and select "Match". Hope this helps man
You could have changed the greedy Regex to lazy by adding a question mark to the quantifier e. g. .*? instead of .*
How to bypass the robo.txt file blocking the scraper?
Same would like to know the answer to this
You can't with this I am pretty sure
It appears the free version of Chat GPT doesn't work with this. Still, interesting
2 THESSALONIANS 1:6 KJV!
Scraping a website WTF is that ? Could u please elaborate it
I do it with a simple python code.
Could you talk, move your cursor and type faster so we have even less of an idea what you are doing?
😂 Was flicking through the video hoping to see the end result of what he did with the the content on a new site or something.... Ssems like a whole lot excess to get some content
Could you be less ungrateful so we have even less of an idea of the kind of person you are?
Good content but please for god sake Regex is pronounced with a "g" like gallium, cause it's an abbreviation of "Regular Expression", not "Rejular Expression" :D
Ahaha this made me spit out my drink. Noted, I'll respect the regex gods from here on out.
ReJex
It would be ironic if, after pointing this out, you called a GIF a “JIF”. 😂
This is incorrect advice, if you live in an English speaking country don't do this unless you want people to think you're dumb. It's pronunced rej like the start of the word register in every company I've worked with in Europe and America lol
Sure, you are correct… But,a lil pedantic aint we ? …besides that’s rather presumptuous to think God is listening to you. There is a queue ya no.