Simple Effective Data Extraction (for Python Devs)
ฝัง
- เผยแพร่เมื่อ 10 ธ.ค. 2024
- Check Out ProxyScrape here: proxyscrape.co...
➡ JOIN MY MAILING LIST
johnwr.com
➡ COMMUNITY
/ discord
/ johnwatsonrooney
➡ PROXIES
proxyscrape.co...
➡ HOSTING (Digital Ocean)
m.do.co/c/c7c9...
If you are new, welcome. I'm John, a self taught Python developer working in the web and data space. I specialize in data extraction and automation. If you like programming and web content as much as I do, you can subscribe for weekly content.
⚠ DISCLAIMER
Some/all of the links above are affiliate links. By clicking on these links I receive a small commission should you chose to purchase any services or items.
This video was sponsored by ProxyScrape.
Please never stop making videos !
Protect this guy at all costs 🔥
Great video that has reduced my hesitancy and FUD to use scrapy. I assume you'll put this in playlist with your other Scrapy videos?
Really good explanation and end-to-end example. Much appreciated.
I started your channel with this video and being new to scrapy I had an issue initially with this error:
raise KeyError(f"Spider not found: {spider_name}")
KeyError: 'Spider not found: products'
You may mention it in other videos, but digging into the docs I found that the products.py template file created with the genspider command must be moved into the arc/spiders folder for scrapy to find it. Maybe this helps someone else.
Great video. How are you doing pasting the code in one line at a time? Great trick.
I learnt it from fireship, delete the lines and the use undo to put them back. Simple but effective!
Hi John. Just wonder is there any way to use pipeline to normalise different schemas? How to handle case when you scrape two different shops with different schemas?
Hi John, what font you used in this video?
IBM plex mono
Just wanted to know😅. Are you a freelancer or do you work for a company?. I am also really into web scraping.
I have a day job as well as scraping + youtube!
Im starting to learn web scrapping where to start first can you share a road map? Or atleast which video of yours i should watch first???
I learned from John. start with weather scraping video first. Its a good start
Thank you I'll definitely watch it
In sites that require fingerprinting how do you handle that in scrapy?
@@munchcup you need to use impersonate attribute in Scrapy for fingerprinting.
@Satish-st5le thank you
@@Satish-st5leCan’t find impersonate attribute in scrapy documentation, unless you’re talking about incorporating curl_cffi or tls_client into the scrapy solution.
@samuelgodfrey3545 you need to install it thru pip. Type Scrapy-impersonate Zenrows in Google.
@@samuelgodfrey3545 Type this: "scrapy-impersonate zenrows" in google.
🔥
Hey, I saw your videos. They're great and informative but your thumbnails are not appealing enough. I think you should hire a Professional Thumbnail Artist for your videos to increase your view count cause every impression matters. I can improve your ctr from 2-3% to 15%. Please acknowledge and share your contact details to get your thumbnail.