you know this is exactly why Twitter limit the amount of content people can look at lol use AI bot to do web scraping for you lol we're going to abuse social media now to the point every platform will have to set limits lmao
i am new to web scraping but i have been researching on how to make an ai webscraper that can automatically open the ispect page of a website and extract the necessary elements given to it by a user. the user only needs to specify what he wants from a website and the scraper does all the work. it can be particularly helpful for newbies
I did similar projects in the past (in php long time ago with curl and tidy, now in python with bs4 or selenium). To avoid bot detection, you can use the tor network, detect captcha request, and simply reconnect tor to get a different IP. Of course, you still need to put random sleeps here and there. Also, you need to be prepared to update or rewrite your code any time the website changes. Nice video!
"Gathering" data from other people is one of the holy grails of programming. OpenAI is the epitome of this aren't they - Gathered data from all of the internet, regurgitated it with "AI magic" and sell it back to consumers as their own.
@@sergesmitty137 You say that in quotes as if it is some gimmick yet multiple multi-billion dollar corporations weren't trying pouring a fuckton of money into similar projects and all failed or produced AI barely worth mentioning. Besides, they aren't selling the data they scrapped, that was just used to train the model, they're selling you a tool that can do practically anything digital you'd hire a minimum wage employee to do, for now.
@@DerickMasai The UserAgent string is simply an up-to-date Firefox/Chrome. What I meant was to use tor as a proxy to hide and replace your real IP quickly. tags=b.findAll('p',attrs={"class":"mt-4"}) # whatever style their captcha text is using.. if(len(tags)>0 and 'Your IP made too many requests' in tags[0].text): # we got caught print("CAPTCHA, restart tor") restart_tor() and that function does: def restart_tor(): try: print("restarting tor..") time.sleep(random.randint(10,15)) x=subprocess.run("sudo /etc/init.d/tor restart", shell=True, timeout=30) except: return False print("done, waiting") time.sleep(random.randint(25,30)) return (x.returncode==0) I hope the indents stay ok. Anyway: just detect if the target site caught you scraping, and instead of solving the captcha, just restart tor. When it reconnects, the new IP you get can redo the last request. (Also I've put my user into the sudoers so the script can do the restart part without my password.) The timing magic numbers were a bit more sophisticated, but it works.
Just an anecdote, but when I first got into a data role I built a competitor price web scraper for my company. At the time it was supposed to be a show and tell for my boss in case python work came up in the future. It worked well enough for the business to utilize for beating the comp to vendors on price. We could keep tabs on them at a quick enough pace to adjust quotes and win. Didn't always work out but being able to say a python project netted three extra contracts is really cool
I am currently developing this exact project with Power BI as dataviz tool. Currently have nearly 60 websites scraped and nearly 3300 urls daily updating. Kinda sad because everything I do is local and dont have the infrastructure to put it in the cloud or run from VMs (also I dont really know how to get more deep than this lol) The company is a multinacional and they DO NOT have market intelligence, it is bizarre
@@gustavonovakoski4867 Awesome 😎 Would you like to share your knowledge & scripts. How you are managing live data updates from 60 + websites How are you then comparing it I would love to see your video tutorials or guides Thanks 🙏
Great video! Thank you for talking about web scraping as an IT subset. What most people don't know is that it's extremely hard to build your career around it. I have been running my own web scraping software house for several years now and getting clients is more difficult than if you build apps, create websites, or do virtually anything else. People and companies simply don't have enough awareness what access to real time data can do for their businesses. Creating this awareness is actually the biggest part of my job. Writing code is just a formality afterwards.
Wouldn’t it be better to build tools that solves specific problems more than trying to convince people they need tools they don’t know about? For example I pay more than I want in my business to access a tool that does a small part of a problem I’m trying to fix. I found another tool that fixes the whole problem and I’m happy to spend $30 a month to solve that problem.
I'm trying to build as many projects as I can right now to upgrade my skills and cv. This project idea is really helpful to my resume, I'll try to do it using fastapi and mysql (if possible I'll try to track stock price). Truly appreciated for sharing this project, sir!
He literally used the sponsor's tool in the project. I would use the tool regardless if it was a sponsorship or not so in this case case he makes money at my benefit. I have no problem with this.
He is right, I work for tech consulting company. One of the client projects I work on is web scraping project that collects doctor disciplinary action data using python and they pay a lot for us to do that for them.
@@vatsalyavigyaverma5494 there's multiple way of storing data... I prefer Tinydb or SQlite, but if you work on a bigger project, I would recommend using a cloud storage like mongodb or firebase.
I totally agree, I've done a couple of web scraping scripts for some companies, and to get some specific data, however there are a lot of challenges. But in general the rate limiting for APIs and changes in DOM, or catchaps are the most common ones.
@@camilocastrillon2030react does modify the DOM. The dom is updated with the diff of the react virtual Dom. This isn't really the issue with web scraping & Dom changes though. They mean "changes to the DOM" as in the websites html just changes as the developers of the website add new features, change the layout, etc.
I've seen people exploiting online job portal sites like Upwork. They hire freelancers from the Philippines to scrap websites rather than building a program, and of course they *CHEAP AF* like we're talking about $2-3 per 100+ websites. Most of them charged $2-3 per hour which is absolutely disgusting.
Wow! I was literally thinking of doing the same thing, but really had no idea it was that hard or that some many people have already tried the same thing and failed. Before I even started facing the problems, you gave me the solutions for free... So thank you!
appreciated brother. I was actually building something very similar but yours is definitely better build. Do you recommend against doing this in Javascript ?
Great video ❤ one thing I noticed that might be helpful on your code is comments when you’re walking through what each part does. That way anyone who uses it from your GitHub doesn’t have to reference your video each time to hear what it does. Just a small detail I noticed. Otherwise, fantastic idea and project!
Tim, your videos look very professional. I see the gear/hardware you use. May I ask what software you use to record your screen and include a picture-in-picture of yourself? I did not see this info anywhere. Thanks in advance!
Funny enough I used PyQt5 to build a web scraper, I can scrape Amazon without getting block... But with this new information I can extend my web scraper a lot further to make a more complex project. Thank You Very Much Tim...
I am planning on working on something like this, but I am from africa and i dont have too much knowledge of these GDPR laws and so on, could you make a video on that please?
Clowns like you have said the same thing for years, and still remain employees. If you think one man can take the whole pie, you're very, very mistaken.
Just watched this vid about web scraping and got hit with a million-dollar project idea! Thinking of using Proxy-Store's proxies for it. Anyone else got big plans in the works?
How many pages can you scrape using this approach before Amazon blocks your scraper, and what's the best approach to avoiding detection by Amazon and other sites?
I did something like this for a company I worked for. Setting up slowly changing dimensions tables in python is p hard when there aren’t primary keys for the data you’ve scraped.
@@TannerBarcelosyou are scraping tons of data? Why would you need primary keys? Just collect the data you want, pass it into a pandas dataframe and give it whatever header names (keys) that you want
Who is buying scraped data - companies would just build it ? or is the value proposition any community that wants/needs to know if data has changed and building some kind of tool to do that?
What language would be the best to learn to create an app? Java? C#? Which one should I choose? Also, I want to know the prospects of each language. Can you tell me?
The idea is cool(the frontend part) except the "pay us to run your code and provide you the data data your code got" part, but what if you would want to scrape like 100k distinct items? The frontend will become just a long scrollbar.
Hey, thanks for the great content! Guide me please, what is the road map for learning the tools to understand what you have done in this project and be able to do it on my own?
Isn't Web Scrapping Illegal? Some time ago I had the idea to do a project just like this one and I ended up giving it up because I thought I was illegal. I read it on the website's terms and conditions.
bear in mind this is illegal in most cases, if you try to start a saas doing this you most probably end up broke if the source of your scraping is copyright protected
But Tim you know what the problem is. When a Co website have T&C that does not allow machine access to their website and your scraper crashes it then they can come after you for damages if they lost a lot of money. And you are the guy who sold this piece of software to the user. Imagine you sold dozens of such scrapers doesnt this put your legal risk very high? As freelancers we need to be very careful about the legal repercussions of our work.
If you don't want to be apart of something that can be a legit legal issue. Do what I do, steer away from it. Use API's instead so you and/or your company can safely exist without getting sued in the future.
a 20 year old idea that had been exploited to the maximum already. i myself used it to steal real time data in commercial goods trading. dont want to get specific to avoid saturation in that field next to the legal issues.
Tech With Tim, dude, big fan, suggestion, get a ton of sleep like a TON like 24 hours then eat and take a shower then a really cold shower, and then go get a professional picture made or just use your camera and then use that pic as your profile pic cause on your profile pic u look like you're going to pass out from too much hard work too many hours (which I understand being a techie myself) your eyes are giving you away. Love your stuff man, thank you tons, keep it up PLEASE, TKX!
Hey i watched some of your videos and i like what you do here ... i was thinking may be if you could do a tutorial on how to Develop an API with Python or JavaScript
I'm trying to scrap documents from a website with multi layers, so I can use those files with a logical chat bot to reference what I'm looking for, please any help will be appreciated, thanks in advance.
good day tim, i just want to mention and ask permission that i will extend this base code. for now i made a private repo and soon on the right time i will make it public and will credit you. thanks
Just wondering haven't played with the program yet. But does the program also record prices differences if items have multiple versions example Red rain coat vs the same rain coat but it's Blue and cost $1 more.
The biggest question is where to sell how to build trust I already made few tools and selling them but i have no idea how to sell web scraping data or tools Also thanks ❤ that's a good starting point Edit: 9:02 after seeing this i remember i made something similar but it scraps anime each one with it episodes and each episode with stream url and title desc... and then display and stream anime with electron ui but i never finished it cuz i have no idea how to make ui with js
Let me know any web scraping project ideas you have!
Sign up for @BrightData here and get $15 in FREE credit! brdta.com/techwithtim.
Thanks for all the videos man, I appreciate your work.
you know this is exactly why Twitter limit the amount of content people can look at lol use AI bot to do web scraping for you lol we're going to abuse social media now to the point every platform will have to set limits lmao
i am new to web scraping but i have been researching on how to make an ai webscraper that can automatically open the ispect page of a website and extract the necessary elements given to it by a user. the user only needs to specify what he wants from a website and the scraper does all the work. it can be particularly helpful for newbies
@@nevilleachi6888 If you could make such a thing I'll use it 😂
@@AR-rg2en🎉😊😢
I did similar projects in the past (in php long time ago with curl and tidy, now in python with bs4 or selenium). To avoid bot detection, you can use the tor network, detect captcha request, and simply reconnect tor to get a different IP. Of course, you still need to put random sleeps here and there. Also, you need to be prepared to update or rewrite your code any time the website changes.
Nice video!
"Gathering" data from other people is one of the holy grails of programming.
OpenAI is the epitome of this aren't they - Gathered data from all of the internet, regurgitated it with "AI magic" and sell it back to consumers as their own.
@@sergesmitty137 You say that in quotes as if it is some gimmick yet multiple multi-billion dollar corporations weren't trying pouring a fuckton of money into similar projects and all failed or produced AI barely worth mentioning. Besides, they aren't selling the data they scrapped, that was just used to train the model, they're selling you a tool that can do practically anything digital you'd hire a minimum wage employee to do, for now.
As for the original comment, thank you so much for the Tor idea! Never even thought using that as the useragent. Downloading it now.
What an idea , using Tor networking for web scaraping . Thumbs up for that 👍
@@DerickMasai The UserAgent string is simply an up-to-date Firefox/Chrome. What I meant was to use tor as a proxy to hide and replace your real IP quickly.
tags=b.findAll('p',attrs={"class":"mt-4"}) # whatever style their captcha text is using..
if(len(tags)>0 and 'Your IP made too many requests' in tags[0].text): # we got caught
print("CAPTCHA, restart tor")
restart_tor()
and that function does:
def restart_tor():
try:
print("restarting tor..")
time.sleep(random.randint(10,15))
x=subprocess.run("sudo /etc/init.d/tor restart", shell=True, timeout=30)
except:
return False
print("done, waiting")
time.sleep(random.randint(25,30))
return (x.returncode==0)
I hope the indents stay ok. Anyway: just detect if the target site caught you scraping, and instead of solving the captcha, just restart tor. When it reconnects, the new IP you get can redo the last request. (Also I've put my user into the sudoers so the script can do the restart part without my password.) The timing magic numbers were a bit more sophisticated, but it works.
Just an anecdote, but when I first got into a data role I built a competitor price web scraper for my company. At the time it was supposed to be a show and tell for my boss in case python work came up in the future. It worked well enough for the business to utilize for beating the comp to vendors on price. We could keep tabs on them at a quick enough pace to adjust quotes and win. Didn't always work out but being able to say a python project netted three extra contracts is really cool
Nice, I’m in a data role
I am currently developing this exact project with Power BI as dataviz tool. Currently have nearly 60 websites scraped and nearly 3300 urls daily updating.
Kinda sad because everything I do is local and dont have the infrastructure to put it in the cloud or run from VMs (also I dont really know how to get more deep than this lol)
The company is a multinacional and they DO NOT have market intelligence, it is bizarre
@@gustavonovakoski4867
Awesome 😎
Would you like to share your knowledge & scripts. How you are managing live data updates from 60 + websites
How are you then comparing it
I would love to see your video tutorials or guides
Thanks 🙏
@MuhammadFAH33M he won't share is soup with strangers
So this is basically an elaborate commercial
i agree. this video is very misleading. and so is every video with titles about web scraping but then uses a paid 3rd party product.
Great video! Thank you for talking about web scraping as an IT subset. What most people don't know is that it's extremely hard to build your career around it. I have been running my own web scraping software house for several years now and getting clients is more difficult than if you build apps, create websites, or do virtually anything else. People and companies simply don't have enough awareness what access to real time data can do for their businesses. Creating this awareness is actually the biggest part of my job. Writing code is just a formality afterwards.
If most of these are e-commerce only
Wouldn’t it be better to build tools that solves specific problems more than trying to convince people they need tools they don’t know about?
For example I pay more than I want in my business to access a tool that does a small part of a problem I’m trying to fix. I found another tool that fixes the whole problem and I’m happy to spend $30 a month to solve that problem.
Hey, let me know if you can connect with me if you are active in web scraping
I'm trying to build as many projects as I can right now to upgrade my skills and cv. This project idea is really helpful to my resume, I'll try to do it using fastapi and mysql (if possible I'll try to track stock price). Truly appreciated for sharing this project, sir!
How did it go
Thumbs up if sponsored tools are not your content!
But man’s gotta eat tho
@@thetruthsayer8347That’s his 1 million Dollar project (Let viewers use the sponsor) 😂
His bills won't sort themselves out
He mentioned that companies are fighting against scraping...
He literally used the sponsor's tool in the project. I would use the tool regardless if it was a sponsorship or not so in this case case he makes money at my benefit. I have no problem with this.
He is right, I work for tech consulting company. One of the client projects I work on is web scraping project that collects doctor disciplinary action data using python and they pay a lot for us to do that for them.
dude, I need money.
I'm literally scraping day and night.
is there anyway I can help ?
I'm hungry for data.
@@anonfourtyfivehow you store those data 😊
@@vatsalyavigyaverma5494 there's multiple way of storing data...
I prefer Tinydb or SQlite, but if you work on a bigger project, I would recommend using a cloud storage like mongodb or firebase.
@@anonfourtyfivewhat is your goa scraping day and night?
How many languages would you recommend someone learn inorder to work in tech consulting like you, currently learning python.
I totally agree, I've done a couple of web scraping scripts for some companies, and to get some specific data, however there are a lot of challenges. But in general the rate limiting for APIs and changes in DOM, or catchaps are the most common ones.
What does DOM stand for?
@@normallyChallengeddocument object model
Since react doesn’t modify the dom directly I’m guessing I could take an advantage and sort that out, right?
@@camilocastrillon2030react does modify the DOM. The dom is updated with the diff of the react virtual Dom.
This isn't really the issue with web scraping & Dom changes though. They mean "changes to the DOM" as in the websites html just changes as the developers of the website add new features, change the layout, etc.
@@camilocastrillon2030 now its saturated hey can u think of how free saas app can make money
I've seen people exploiting online job portal sites like Upwork. They hire freelancers from the Philippines to scrap websites rather than building a program, and of course they *CHEAP AF* like we're talking about $2-3 per 100+ websites.
Most of them charged $2-3 per hour which is absolutely disgusting.
Good idea. If u r in 2013. That's plenty of services who are provided same possibilities
this video is the reason removing the dislike button was a bad idea
17500 up, 500 down right now
Need a whole series with videos like these
+1
Agree
+1
3rd to 6th the MOTION & Request!! Thank you in advance!
Agree
I love web scrapping; web scrapping is really satisfying when you finish your project.
What is web scraping?
20$ / GB is huge and you will hit that really fast lol
youtube videos these days are becoming ads
A simple Chrome extension build would have made this so much easier for you and it's completely transparent for bot blocking.
Could you elaborate on this?
I've tried an extension easyscraper and that too got my IP blocked.
Wow! I was literally thinking of doing the same thing, but really had no idea it was that hard or that some many people have already tried the same thing and failed. Before I even started facing the problems, you gave me the solutions for free... So thank you!
Great tool Tim, i look forward to playing around with this. I still have a lot to learn about the Data industry though, keep up the amazing work.
Glad it was helpful!
More uses:
- job searches
- combine with NLP to obtain live information regarding sentiment towards products or services
What's NLP?
@@coder_117natural language processing. Analyses text
Thank you! This is the replacement for the outwit browser I've been searching for!
Very helpful looking for data will check out bright. Thank you.
Zeus Proxy ensures anonymity and privacy while performing SEO tasks, enhancing security and reliability.
Thanks for this valuable content Tim. I found it helpful
Just the cost of Bright Data makes this not applicable. I want to make 10,000 requests a day at least. Anyone found a work around?
This product would be a smash hit among Coupon Grandmas in Texas.
I tought I had some good skills on web scraping, then you showed me I'm just a baby still. Loved the idea and the fact you gave it out for free
not free, it is just an ad
I did this back then with food delivery services for example burger it will display the cheapest delivery and price from a pool of available service.
thanks for the video. I have been thinking of making something similar. Even bought a domain for it.
appreciated brother. I was actually building something very similar but yours is definitely better build. Do you recommend against doing this in Javascript ?
Don't forget the proxies while web-scraping 😄
That's awesome, keep going bro
Great video ❤ one thing I noticed that might be helpful on your code is comments when you’re walking through what each part does. That way anyone who uses it from your GitHub doesn’t have to reference your video each time to hear what it does. Just a small detail I noticed. Otherwise, fantastic idea and project!
ChatGPT will do that for you.
Please keep project ideas video coming 🙏
Tim deserves more than 1.26 million subscribers. Fantastic job!
Super helpful, thanks
Tim, your videos look very professional.
I see the gear/hardware you use. May I ask what software you use to record your screen and include a picture-in-picture of yourself?
I did not see this info anywhere.
Thanks in advance!
Amazing project scraping the web
Sir i watch your all videos and they are very helpful thankyou to provide as like that informative content.....👌
Great video. scrapers + llm apps are going to dominate data very soon (if they aren't already doing this)
What are llm?
@@tanysanchez8519 Large Language Models (machine learning algorithms)
Funny enough I used PyQt5 to build a web scraper, I can scrape Amazon without getting block... But with this new information I can extend my web scraper a lot further to make a more complex project. Thank You Very Much Tim...
This is exactly what I was looking for - thank you!
Thanks for inspiring me to learn programming, you're really cool. This is a very cool and interesting project. ⌨💪🧑💻
Man u gave us so many valuable informations,we appreciate it❤
I am planning on working on something like this, but I am from africa and i dont have too much knowledge of these GDPR laws and so on, could you make a video on that please?
Ok so it's an ad for a paid service.
Fuck that, if I wanted a paid solution I would have looked for a paid solution already, not a video tutorial.
Great stuff Bro i was wondering how to do this for long time
if it is a million dollar idea then why you don't do this and show us your success...who are you fooling it's just an ad of bright data
Calm down bruh
Clowns like you have said the same thing for years, and still remain employees. If you think one man can take the whole pie, you're very, very mistaken.
Bahagaaaa😢😢😢
😂😂😂😂😂oh ms goshhh
Bright data is making millions
very great vedio like every time
Just watched this vid about web scraping and got hit with a million-dollar project idea! Thinking of using Proxy-Store's proxies for it. Anyone else got big plans in the works?
How many pages can you scrape using this approach before Amazon blocks your scraper, and what's the best approach to avoiding detection by Amazon and other sites?
would be awesome to have a step by step tutorial on the app you have developed
I did something like this for a company I worked for. Setting up slowly changing dimensions tables in python is p hard when there aren’t primary keys for the data you’ve scraped.
This is the issue I am in right now. I am scraping tons of data but I need to create the data model but I am lacking primary keys, etc.
@@TannerBarcelos Hey, did you figure it out?
@@TannerBarcelosyou are scraping tons of data? Why would you need primary keys? Just collect the data you want, pass it into a pandas dataframe and give it whatever header names (keys) that you want
Who is buying scraped data - companies would just build it ? or is the value proposition any community that wants/needs to know if data has changed and building some kind of tool to do that?
I'm definitely adding this one to my resume
the fact he started with little subscriber and how young he was 5 years ago I made my first project thanks to him time flies
Thanks for info
bright data ok cool got it !
What language would be the best to learn to create an app? Java? C#? Which one should I choose? Also, I want to know the prospects of each language. Can you tell me?
we watch you grow up
Hey Tim, you always have amazing content. Keep it up! Greetings from Italy!
The idea is cool(the frontend part) except the "pay us to run your code and provide you the data data your code got" part, but what if you would want to scrape like 100k distinct items? The frontend will become just a long scrollbar.
I build a "free" scraper to this project that you can find in my videos.
Wouldn’t it be illegal to do if the terms and conditions of the website tells you not to scrape it?
A wonderful video that we've used as a reference for our recent additions. Your sharing is highly appreciated!
thank you my king this sounds so cool
I have no idea what’s going on but nice vid
This is just an advert
Hey, thanks for the great content!
Guide me please, what is the road map for learning the tools to understand what you have done in this project and be able to do it on my own?
Isn't Web Scrapping Illegal? Some time ago I had the idea to do a project just like this one and I ended up giving it up because I thought I was illegal. I read it on the website's terms and conditions.
Web scraping is perfectly legal. What you do with the data may or may not be legal though.
Thank you
Would you consider a full length setup and code tutorial for this project or a similar one?
!
I love this dude deeply this was the reason I started learning code❤ now some one is give me a repo
Refined marketing approach... Brilliantly done
Oh so it's just an ad. Cool
bear in mind this is illegal in most cases, if you try to start a saas doing this you most probably end up broke if the source of your scraping is copyright protected
But Tim you know what the problem is. When a Co website have T&C that does not allow machine access to their website and your scraper crashes it then they can come after you for damages if they lost a lot of money. And you are the guy who sold this piece of software to the user. Imagine you sold dozens of such scrapers doesnt this put your legal risk very high?
As freelancers we need to be very careful about the legal repercussions of our work.
If you don't want to be apart of something that can be a legit legal issue. Do what I do, steer away from it. Use API's instead so you and/or your company can safely exist without getting sued in the future.
Who is down to port this with me into NextJS/tRPC/Prisma? Great Video Tim as always! Give me some motivation to start working on a scrapper!
Thanks Tim. Any plan for a detailed tutorial?
Legend!
this is so sickk ima mess with it rn
I find your content and expertise level brilliant. Are you self taught or You have a BS in IT or attended a bootcamp?
Self taught, but I did do 5 semester of a CS degree before dropping out (I have videos discussing that)
Thanks for sharing this project with us !!! 👍
Thanks
a 20 year old idea that had been exploited to the maximum already. i myself used it to steal real time data in commercial goods trading. dont want to get specific to avoid saturation in that field next to the legal issues.
Summarizing the project:
NOTHING WORKS IF YOU DON'T USE BRIGHT DATA!!
not true, u can easily remove the dependency on bright data, in fact it's a simple as changing 2 lines of code.
@@TechWithTim pls help, how to remove the dependency? im new to this
It's a year now, How did it go Tim
Do you have a video of web scraping data using javascript and nodejs?
Great project
Man, he is getting older, its funny to remember how he used to have a baby face and now he looks like a full grown man
Tech With Tim, dude, big fan, suggestion, get a ton of sleep like a TON like 24 hours then eat and take a shower then a really cold shower, and then go get a professional picture made or just use your camera and then use that pic as your profile pic cause on your profile pic u look like you're going to pass out from too much hard work too many hours (which I understand being a techie myself) your eyes are giving you away. Love your stuff man, thank you tons, keep it up PLEASE, TKX!
Can you please build a live streaming web app & show us?
Hey i watched some of your videos and i like what you do here ... i was thinking may be if you could do a tutorial on how to Develop an API with Python or JavaScript
nice ad.
I'm trying to scrap documents from a website with multi layers, so I can use those files with a logical chat bot to reference what I'm looking for, please any help will be appreciated, thanks in advance.
good day tim, i just want to mention and ask permission that i will extend this base code. for now i made a private repo and soon on the right time i will make it public and will credit you. thanks
I think with ChatGPT interpreter, a lot of possibilities are opened
Cloudfare won't let me scrape
Can you tell, what thing did you do to monetize through webscraping
Just wondering haven't played with the program yet. But does the program also record prices differences if items have multiple versions example Red rain coat vs the same rain coat but it's Blue and cost $1 more.
The biggest question is where to sell how to build trust
I already made few tools and selling them but i have no idea how to sell web scraping data or tools
Also thanks ❤ that's a good starting point
Edit: 9:02 after seeing this i remember i made something similar but it scraps anime each one with it episodes and each episode with stream url and title desc... and then display and stream anime with electron ui but i never finished it cuz i have no idea how to make ui with js
can you do a scraper for movies?