Web Scraping in Google Sheets: I replaced importXML with Make (Integromat) and ScrapeNinja
ฝัง
- เผยแพร่เมื่อ 3 ต.ค. 2024
- In this video I develop a simple low-code Make.com scenario which iterates over Google Sheets rows and scrapes websites from each row, using ScrapeNinja.net, and puts results back to the same Google Sheet.
Why ImportXML is not perfect for web scraping: pixeljets.com/...
![Top 27 ChatGPT Make.com Automations [Automate With AI]](http://i.ytimg.com/vi/cr3plCKOnVU/mqdefault.jpg)
You changed my life man. Keep up the the great instruction.
Thanks mate, I appreciate it.
Hello,
thank you for the video.
thank you for watching!
Thanks for this, looks great. I'm just getting started and following along, but I don't see ScrapeNinja in the list when I try to 'Add Module'. I did subscribe to it via RapidAPI (free subscription to start). I must be missing a step, would appreciate any advice.
Thanks! ScrapeNinja module was approved by Make team, but it will be available in public list only in a few weeks during next Make update. To use ScrapeNinja now, you need to click invitation link from the description of this video to see the module: eu1.make.com/app/invite/6a5739aa760491ee365289b800649846
UPD: ScrapeNinja integration is now available in official Make integrations list: www.make.com/en/integrations/scrapeninja so you don't need to click invite link anymore. Yay!
Can you please share your list of automation service?
An article that compare them would be great!
sure, I have such an article: pixeljets.com/blog/zapier-make-com-pipedream-from-a-developer-perspective/
I want to get the product price, how can I do that? Please help me
Geat video.
Nice video! In my case i need to make a scrap of different news pages, blogs, etc; to collect these news, organize and compose it for my Newsletter with the Open AI API; I have been trying with ninja scrapper and make but I can’t get it to enter each of the written articles, it only takes the text that is outside, like titles, labels, etc. Any ideas? I want to automate the entire process of collecting and writing content from different websites
did you see my another video related to content extraction pipeline? th-cam.com/video/hRQqJtgYz_Q/w-d-xo.html
Hi! Can you help me? When I try to add scrapeNinja to “make” I have a problem with “creating connection”. Because I need RapidAPI key. But I don’t understand where I can generate this one.
Same... Did you find out how it works ?
I had this working a couple days ago, but not anymore. I tried everything... even started over and copied all your steps using the same URLs in my google sheet. No matter what I try, I get a ModuleTimeoutError for each operation before ScrapeNinja gives an output. Any tips?
I think we figured it out over email. Thanks for reporting!
How do I scrape data from a number of items listed on a page. For example, I want to see all companies in vc portfolio, scape name, url, etc. I'd like this in a spreadsheet. Can I use ScrapeNinja for that?
Sure, ScrapeNinja can definitely be used for this kind of task. The implementation details depend on a particular webpage.
@@pixeljets Do you have documentation I can reference. I'm new to scraping. I tried, but could not get it work with scrapeNinja.
Hi, thank you for the great video! Been looking for this kind of tutorial for web scraping. I'm running into cookie consent and cloudfare problems on some of the websites. If you havent visited the website, you're trying to scrape and they have a cookie consent, then the scraper only gets the data from the cookie consent. Or sometimes cloudfare stops the bot. Got any tips for these? Thank you again!
Hi, thank you! I would try to use playwright/puppeteer with chrome extension (like "I dont care about cookies" extension) to auto-hide the consent. playwright.dev/docs/chrome-extensions
@@pixeljets Thank you for the quick answer! I decided to go the hard way and found out that when making a scraper with python, it autopasses these problems and same with autogen + anaconda combo, which i definetly would recommend to check :)
@@nurmimika thanks for sharing your experience! Using plain python (like, using requests module) will probably open another can of worms though. do you use autogen agents for real world scraping? how its going?
How do i find my API key to add scrapeninja in Make?
If you are on this page at 1:30 , click the button labled "Subscribe to test" then click the free subscription. Go back and scroll down and to the right youll see code. Look for 'X-RapidAPI-Key" Copy that, this is your ScrapeNinja Key.
Hi! Get your key by subscribing: rapidapi.com/restyler/api/scrapeninja
looks great video,
please, how to get my api key?
For ScrapeNinja, you can get the API key here: rapidapi.com/restyler/api/scrapeninja/
@@pixeljets thanks boss
Like so many this video's volume is far too low for the hard of hearing. Using CC is the only way I can folllow what's being done.
Have you tried using headphones? If the problem still consists, i doubt it's because of the volume on this video. If i put system volume and TH-cam video volume to max. i'm going to break my eardrums.
3:06
7:18
7:52
8:45
10:18