Autonomous AI crawler
ฝัง
- เผยแพร่เมื่อ 30 ก.ค. 2024
- In this video I share my concept of autonomous AI crawler built with n8n that can navigate through the website and extract desired data.
In this tutorial I cover:
1. Concept of AI crawler built with n8n (how it works and what are its capabilities)
2. Pros and cons of the AI crawler
3. Quick walkthrough for main AI crawling agent, text tool and URL tool
4. Using Web Unlocker proxy for better result accuracy
5. Adding multiple agents to the workflow to retrieve more data
Download example n8n workflows from this tutorial:
1. Single agent with tools: n8n.io/workflows/2315-autonom...
2. Multiple agents with tools: gist.github.com/workfloows/48...
Resources:
1. Understanding agents (official n8n docs): docs.n8n.io/advanced-ai/examp...
2. Proxy used in the video (Web Unlocker): brightdata.com/products/web-u...
My other tutorials:
1. How to automate Gmail with n8n AI: • How to automate Gmail ...
2. Web scraping data with n8n and Puppeteer: • Web scraping data with...
3. Build undetectable Amazon scraper with n8n, Puppeteer and Scraping Browser: • Build undetectable Ama...
4. How to automate Notion databases using n8n: • How to automate Notion...
5. Getting started with native 🦜️🔗 LangChain nodes in n8n: • Getting started with n...
Subscribe my newsletter: workfloows.com/
Visit my n8n creator profile: n8n.io/creators/workfloows/
Visit my Gumroad profile: workfloows.gumroad.com/
Follow me on Twitter/X: / workfloows
Follow Workfloows on LinkedIn: / workfloows
Disclaimer: I cannot be held responsible for any consequences resulting from the use of the information provided in this tutorial. Make sure to obtain proper authorization before engaging in web scraping activities, or consider using proxies to protect your online presence and ensure ethical scraping practices.
Create your Bright Data account and get $10 credit: brdta.com/workfloows
Create your n8n cloud account here (affiliate): n8ngmbh.partnerlinks.io/6hvl7...
Screen recording software that I use (affiliate): www.screen.studio/@jmMwX
0:00 AI crawler concept
1:15 Pros and cons of AI crawler
1:47 Part 1: Main AI crawling agent
3:35 Part 2: Text scraper tool
5:32 Part 3: Proxy and CAPTCHA solver
6:43 Part 4: URL scraper tool
8:10 Part 5: Crawling with multiple agents
Hey, thank you for watching! This is my first video in a bit new format focusing on workflow concepts and architecture. You can quickly reproduce the workflow using the files in the description.
How do you like this format? Please let me know in the comments - your feedback will help me shape future content!
Like this format, you solved the problem in easy way!
Thank you very much for your feedback, I’m really happy you like it!
Nice workflow. Brightdata captcha solver is something that I should checkout!
Thanks Oskar for your smart explained 🚀🚀 great work.
Thank you very much!
Thanks Oskar ;)
You’re very welcome!
Great work
Thank you very much!
Awesome!
Thanks a lot my friend! 🙌
Nice format
Although you left us hanging with the no clicking problem, that would've been great to know how to go around it at the end
Maybe next video? 👀 Good work though taught me alot about n8n
Thanks for your comment! Indeed good idea for the next tutorial, definitely will think about it 😃
Thanks
You're very welcome!
very nice video thank you. and thank you for making the workflow files available.
Are you able to extend this workflow to read read reviews of the business and create a sentiment analysis overall and also create top 5 reasons why they got positive reviews and top 5 reasons why they got negative reviews. I am trying to figure out if we can get what value the business provides to their customers top 5, what problems for customers they solve top 5, what complaints customers have about their business top 5. etc. thank you
Hey, thank you for your comment!
Do you have in mind any specific source of reviews (e.g. Google reviews)? Or would you like your agent to gather all possible reviews about specific company from various sources? Both options should be doable, but approach to development will be slightly different.
@@workfloows I’m thinking if I’m going to start a new service biz , then I could first do research about what going to start. For example let’s say I want to start medical billing, this is what m in trying to start, so first research the medical billing industry, gather competition data and client pain points , what to provide as a valuable service and what to do better than competition. Then in same research process gather the completion reviews either from their site or google my biz etc and output findings into let’s say 3 main categories. 1 customers pain points , 2 what competition is talking about as value they provide and 3 gather what customers don’t like about competition.
Now I can start working on framing my offer or service to be aligned with what my icp is looking for and what they are not happy with etc. Idea is still fresh in my mind so haven’t dialed it in properly yet but this is what I’m going with for now.
So it can be used for starting new service biz or modifying existing biz offering. I think this kind of research will help narrow down what you need to do or create. What you think ?
Do you think this would work well for retreiving products from eCommerce sites?
Hey, thanks for your comment! To retrieve data for a single product from multiple ecommerce sites, I think this workflow should do the job. However, if you want to scrape dozens or hundreds of products from a single online store, I’d recommend building a dedicated scraping script/workflow that handles pagination and doesn’t rely entirely on AI. This approach will ensure better scraping accuracy and lower execution costs.
@@workfloows thanks for your reply!
Bright Data not solving Captcha:(
Hey, I apologize for my very late feedback. CAPTCHAs can be tricky as they change frequently. The best way to handle this would be to contact Bright Data support. They are very helpful and usually resolve issues quickly.