Powerful Web Scraping - Scalable, Multiple Page Crawling API for LLM Apps and AI Workflows
āļāļąāļ
- āđāļāļĒāđāļāļĢāđāđāļĄāļ·āđāļ 6 āļŠ.āļ. 2024
- Learn how to build powerful and scaleable Web Scraping for your LLM Apps and AI Workflows with scalable API. Supports Static and Dynamic Pages, Crawler for multiple pages and AI LLM based extractor. using BuildShip - a low-code Visual Backend builder with full flexibility.
Try on Live Playground: llm-web-crawler.vercel.app/
Remix links ð:
buildship.app/remix?template=...
buildship.app/remix?template=...
buildship.app/remix?template=...
buildship.app/remix?template=...
Chapters ð
00:00 Introduction to Web Scraping
01:33 Getting Started with Static Scraping
04:01 Advanced Techniques: Dynamic Scraping
07:50 Scraping Entire Websites with Web Crawlers
10:53 Revolutionizing Scraping with LLM Extraction
14:16 Building an AI Assistant with Scraped Data
18:01 Conclusion and Next Steps
Let's connect ðĪ
Chat: / discord
Tweets: / buildshipapp
LinkedIn: / buildship
#webscraping #crawler #crawling #webcrawlerapi #aiworkflow #nocode #lowcode #backend #apibuilder #cronjob #scheduledtask #backgroundjob #openai #ai #googlecloud #aiassistant
Wow! buildship just keeps getting better and better and better!! My goodness!. If there was a no-code/low-code tool that gives its users, even those who don't really understand too much code SUPER POWERS, its gotta be BUILDSHIP!!! Wow and the tutorials just keep getting better and more explanatory, even for non-technical minds! Awesome Job guys!.............This dynamic scraping method is exactly what a client needed to solve a problem, thanks for giving me the solution!ð
Awesome!
How can I add a referral code to the product links to provide them to my clients?
I need tutorials on how to create flows with AI
LLM extraction option looks insanely useful, thank you guys!
I have one question regarding the process. 11:20 - should I specify the exact names of fields to parse?
For example, I want to crawl & scrape 100 sites (collected by Google SERP scraping, with different sitemaps, structure and complexity, various JS selectors and fields naming), containing info about marketing agencies, to get the list of all agencies (name+link). Since there are many naming options for the same fields for each website, is there any possibility to give less accurate instructions like "marketing agecy name" + "marketing agecy link", instead of "name"+"link"?
If there is such option and performance won't suffer - you can't imagine how happy would I be :) Thanks in advance!
Is it possible to use proxy with LLM scrape? Or send our own scraped data for the LLM to process?
The LLM Extract node has been updated to take any text input you want instead of a URL. Extracting from URL has been moved to it's own node. So now you can just pass your own scraped content directly to the node.
@BuildShipApp this is amazing. I'm really impressed with all the templates Buildship are coming out with. Keep up the good work!
Can use that for get my bank account info?
Hi! While BuildShip can be a powerful tool for web scraping, it's important to note that accessing bank account information through unauthorized means can be illegal and unsafe. Instead, many banks offer official APIs for accessing account data securely.
@@BuildShipApp in Ecuador and other countries in iberoameric that API dont exist
@@BuildShipApp In Ibero-American countries, banks do not have APIs, so it is necessary and many companies use their RPA precisely for this thing that banks have not solved and will not do so until perhaps several decades later.