Darshil, very enthusiastic and excited I feel while listening your beautiful classes. Please explain different Hadoop and spark projects. That immensely help us to clear the interviews and should be good performer in job role. As You said , your data engineer course will lauch , please explain concept and how concept will be used in project. Very excited for your course. Your course and projects should meet all our needs. Thankyou darshiil. All the best.
@@rajeshramisetti83 Yes, that's the goal! I will try to make everything as much as easier as I do it on TH-cam Not only Data Engineering but I will try to cover everything I can from multiple domain Cloud/Devops/ and many more
I went through this quickly to see if I can understand your teaching style. AMAZING WORK! Now I will reduce the speed and work on it with you. Thank you so much! With videos like yours I am learning things that I want to know how to do in Python really fast!
Thank you Darshil for this amazing video. With the help of this I made Scraping from stock market data based on weekly or monthly gainer and exported it as CSV. it is helpful for swing trades or investment. Thank you for teaching us great things.
Hi, I was working on something similar at the moment. Could you tell me where to get live data from and what kind of data do I need exactly to begin with. This would really help me alot.
hey did your code work? because seems like i am getting blocked by amazons anti bot measure (which stops any bot from scraping their data ) could you help me out ?
Hi bro, You are dropping very useful and productive tutorials on yt. Generally i watch python videos specially web scraping. But i didn't find any playlist on this topic,please drop more videos on this topic and make a playlist.
Bro thanks for sharing the content as alwys it's amazing ... Can you please post a video about a sample real time data engineer projects you came across and how infrastructure decided like memory etc and what steps involve from development to production. To help us to understand a full picture of actual things happen in the organisation. Thanks much
It was a perfect learning example of the data mining technique of web usage recently learned about it and saw it practically in your video . Your videos are fantastic for anyone looking for a career in data. Thanks for all things that you provide
Hello, thank you very much for this video. In my case the line [ links = soup.find_all("a", attrs={'class':'a-link-normal s-underline-text s-underline-link-text s-link-style a-text-normal'})] it gives me emptiness and i don't understand the reason. Can you help me? please
Thanks for walking through the project. Curious to know how can you automatate this process? And what needs to be done for user-agent if we want to automate this process.
I think many people, including me are getting 503 error when trying to send request to Amazon. I tried different domains but it does not work. Any other suggestion to overcome this error?
I am trying to extract the text in for loops but it shows that de find object is a nonetype object that has not the attribute text. But if I choose just one element of the list, i can obtain the text. It just does not work in a loop
Hi Darshil, Good day ! Here you have shown us without logging in to the Amazon page. There is requirements like, after your login only you need to extract some info from the page. How to do that. For Example - I have to send daily a scheduled mail for VOIP Recharge amount to our team. I have to login each time to the page and paste 'your available amount: 20000 INR" to mail sub and body. How can I make it automated. (I have login details of the page but I do not have any API Key or link.) Hope I will get my answer.
Thank you Darshil. How to prevent from being blocked while scraping the data as I think I have been blocked by one of the websites that I have been scraping. Any help?
Respones is the output I got and after I tried to access amazon content and it is not allowing to get the contents, eventhough I used user agent. How much time we need to wait for amazon webpage to give access?
Thanks Darshil for such an amazing content. Just a query, since Twitter API is not free these days, is there an alternative to complete the Twitter Data Pipeline Project in that case ?
hello i love the video it so informative , i just have a little request , how to go from page to another page , like if scraped the page one , how to go to the next page for scrapping
Hmm, anyone having an issue with the data retrieved from the FIND_ALL() function? It retrieves the HREF links from A tags, but it doesn't do it in order & it skips the first A tagged, HREF element. Hmm
When I try to use this program to get an update on the price of the saved items in my cart, the scraper i built using the code above doesn't work. It always gives an empty list. What could I be doing wrong?
Hi, this tutorial is very easy to understand, thanks for making this tutorial. I want to ask whether the use of single (' ') or double quotation (" ") needs to be considered here?
Similar, I have a requirement to get the cost of ola and uber for certain list of lat lon ,I am facing issue to connect to the ola and uber website due to authentication and how can I pass the lat lon to the from and to location of website to extract cost. Help me, if possible
Hi Darshil, This one was very informative like your other videos. I like them because they are accurate and to the point and also project-based. I've one doubt related to web - scraping, like some of the modern web pages don't use visual pagination with the page numbers, instead they have like a show more button which renders the new results. In this case it doesn't change the webpage so how can we scrape the new results.
@@DarshilParmar You are right, I didnt realise this video was for python beginners. I am trying to find out about concurrency in data engineering and how it is done best. Ill check out your other videos, your channel helps me.
The video was too good suggestion- If you could have written the code in the video instead of reading it in second half, it could be cherry on the cake
Hello! I can see you are interested in learning how to scrape websites. I can help you get better at it. Let me know if you’d like more details or if you have any questions!
Don’t forget to hit the like button and tell me what you want see more?
Darshil, very enthusiastic and excited I feel while listening your beautiful classes. Please explain different Hadoop and spark projects. That immensely help us to clear the interviews and should be good performer in job role. As You said , your data engineer course will lauch , please explain concept and how concept will be used in project. Very excited for your course. Your course and projects should meet all our needs. Thankyou darshiil. All the best.
Please don't forget to relase projects.
@@rajeshramisetti83 Yes, that's the goal!
I will try to make everything as much as easier as I do it on TH-cam
Not only Data Engineering but I will try to cover everything I can from multiple domain Cloud/Devops/ and many more
one real time End To End Project please
awesome tutorial...
but how we can run to scrape one entire category or 1000 pages
Amazon block our bot
what to do at that time!!!
thank you
I always feel difficulty in doing web scraping, but you man makes it so easy and comfortable
Just loved it!!
I went through this quickly to see if I can understand your teaching style.
AMAZING WORK!
Now I will reduce the speed and work on it with you.
Thank you so much!
With videos like yours I am learning things that I want to know how to do in Python really fast!
Thank you Darshil for this amazing video. With the help of this I made Scraping from stock market data based on weekly or monthly gainer and exported it as CSV. it is helpful for swing trades or investment. Thank you for teaching us great things.
Hi, I was working on something similar at the moment. Could you tell me where to get live data from and what kind of data do I need exactly to begin with. This would really help me alot.
Not the 50 videos in a playlist but more data than the whole playlist appreciated man ♥
hey did your code work? because seems like i am getting blocked by amazons anti bot measure (which stops any bot from scraping their data ) could you help me out ?
I'm not able to do this, when I'm extracting the HTML code from amazon, it is not having any anchor tag, instead it is saying "something went wrong"
Your content and the way you are teaching, it's awesome brother.
Keep teaching us 😊...
Thank for this amazing tutorial. You made it so simple and easy to apply.
Please don't quite any content. Because, i am learning from your content. Upload the content like learning with projects.
I feel some skipping in a blink in the video frames. for example of getting agent details.
That is really tutorial. We need such kind of practical Python tutorial.
Hi bro,
You are dropping very useful and productive tutorials on yt. Generally i watch python videos specially web scraping. But i didn't find any playlist on this topic,please drop more videos on this topic and make a playlist.
Bro thanks for sharing the content as alwys it's amazing ... Can you please post a video about a sample real time data engineer projects you came across and how infrastructure decided like memory etc and what steps involve from development to production. To help us to understand a full picture of actual things happen in the organisation. Thanks much
Thanks man for your help, I could understand everything right from the start. You explained it very easily 😃
Hi Darshil, i tried this but for some reason, I am unable to pull in the product title? Any tips. Thank you.
same problem
It was a perfect learning example of the data mining technique of web usage recently learned about it and saw it practically in your video
. Your videos are fantastic for anyone looking for a career in data. Thanks for all things that you provide
Hey I need help while scraping do you mind sharing email id can we have google meet?
@@vishaljeshnani1352 Is it done or you still need help
Thumbnail madhi tar लइच खुश झाला भो 😂😂..
i think now amazon has changed api or something that does not accepting scraping requests as it detects scarling
any workaround?
@SahlEbrahim utilised ai
Thank you soo soo much bro, loved the way you explained, I've made a web scrapping for flipkart by just watching your video, thanks alot!!!!!
Great Zahid, I am also going to do the same 😉
Hello, thank you very much for this video. In my case the line [ links = soup.find_all("a", attrs={'class':'a-link-normal s-underline-text s-underline-link-text s-link-style a-text-normal'})] it gives me emptiness and i don't understand the reason. Can you help me? please
Same problem
Bro same problem faced
try these..
links = soup.find_all("a",class_='a-link-normal s-underline-text s-underline-link-text s-link-style a-text-normal')
Thanks for your effort.. it is such a great learning watching your videos😊
Amazing tutorial on webscraping and data preperation!
Thanks for walking through the project. Curious to know how can you automatate this process? And what needs to be done for user-agent if we want to automate this process.
bro u r a LIFESAVERRRR i am SO THANKFUL for this video and code
Useful. Pls, Continue like this. Thank you.. Really Appreciated.
Really helpful keep making more videos like this ✨
Best ever and easiest tutorial.
I can easily understand from your explanation . thanks
I found exactly what i needed! Thank you!
this video really helped me , thank you so much 😍
I think many people, including me are getting 503 error when trying to send request to Amazon. I tried different domains but it does not work. Any other suggestion to overcome this error?
got any solution
@@onkarHINDU Did you got any solution for the problem?
Hi bro , try using retry mechanism I have done the same and it is returning me the data.
@@omkarpatil9717 How to use retry mechanism? can you show how you have done
Good job , amazing Tuto !!
Thx Teacher
Thank you for this video bro make on how data engineer work on daily basis
you tut in the best i've ever seen t'ill now
I’m so glad I found you omggg
This one video is better a lot of courses
thank you so much bhai for giving me needed content
I am trying to extract the text in for loops but it shows that de find object is a nonetype object that has not the attribute text. But if I choose just one element of the list, i can obtain the text.
It just does not work in a loop
this is awesome and hopefully you do the pagination also and this will be the best tutorial for web scraping.
Hi Darshil, Good day ! Here you have shown us without logging in to the Amazon page. There is requirements like, after your login only you need to extract some info from the page. How to do that. For Example - I have to send daily a scheduled mail for VOIP Recharge amount to our team. I have to login each time to the page and paste 'your available amount: 20000 INR" to mail sub and body. How can I make it automated. (I have login details of the page but I do not have any API Key or link.) Hope I will get my answer.
Not working, can't pull the data out, ig amazon has changed something, though we can see html tags but can't pull the data
Hi , you can pull the data
@@omkarpatil9717 No they have applied preventive measures, you either need to use scrapy playwright or selenium to do it
Hey, I'm getting the response 503. What should I do?
same
@@aakif6182 Did you got any solution for the problem?
Yeah because the url response might not be providing details at that time due to load on webpage. You can try retry mechanism there and check
same problem
i am not getting where is anchor tag. when i check the code their only class a code. how to get a
have u use any proxy and ip fingerprinting use for real human beahivour for scraping large number of data?
Is Amazon not letting you to scrape data.
Thank you Darshil. How to prevent from being blocked while scraping the data as I think I have been blocked by one of the websites that I have been scraping. Any help?
This tutorial was amazing, can you do a tutorial to extract data from multiple pages, say 20 pages in amazon?
I am getting error in if_name_=_main_
NameError: name is not refined
Can anyone pls help me to resolve this error
Respones is the output I got and after I tried to access amazon content and it is not allowing to get the contents, eventhough I used user agent. How much time we need to wait for amazon webpage to give access?
I am also having same issue can anyone help?
@@itsdevilish9856 Did you got any solution for the problem?
please did you get any solution?????????? please please give meeeterad
@@juanignacio4353 Yes
Try retry mechanism here you would get the response
You are a chad. Have someone ever told you that? 💯
How can it be done for pages that need to scroll to load more products?
You're a blessing.
thank you!
one more vedio for selenium plz
Thanks Darshil for such an amazing content.
Just a query, since Twitter API is not free these days, is there an alternative to complete the Twitter Data Pipeline Project in that case ?
Sir, what is the purpose of adding an Ads campaign in the storefront in Amazon and how is it done? please reply
hello i love the video it so informative , i just have a little request , how to go from page to another page , like if scraped the page one , how to go to the next page for scrapping
How would you go about using proxies to avoid captchas and rotating proxies as well?
Dada ek number
in Links var, what did u exactly paste, my links var is not working not showing anything
Thankyou bhaiya ❣️
Awesome, thanks 👍
Hmm, anyone having an issue with the data retrieved from the FIND_ALL() function? It retrieves the HREF links from A tags, but it doesn't do it in order & it skips the first A tagged, HREF element. Hmm
Yes I've been having this issue also
Cannot we do it on Google collab?
im not getting anything. what could be the problem?
You didn't get html contents or you go 503 error then try aiohttp library
When I try to use this program to get an update on the price of the saved items in my cart, the scraper i built using the code above doesn't work. It always gives an empty list. What could I be doing wrong?
Thankyou Darshil
Hi, this tutorial is very easy to understand, thanks for making this tutorial. I want to ask whether the use of single (' ') or double quotation (" ") needs to be considered here?
doesn't matter in python
Bro is it possible to scrap purchase data of these websites ? , i mean purchase count of a particular product.
Similar, I have a requirement to get the cost of ola and uber for certain list of lat lon ,I am facing issue to connect to the ola and uber website due to authentication and how can I pass the lat lon to the from and to location of website to extract cost.
Help me, if possible
i have a task that i need to get the prices of 3000 products , but the price div is different for differnt link so i am not getting the price
I am getting error on accept language.. telling invalid syntax
i am encountering some error when i try to get text i.e. the productTitle is it because of captcha, please help
I am getting response 503 after so many tries, can anyone show/tell me that where em I doing wrong.
Did you got any solution for the problem?
You can try retry mechanism here by sending request again and again
Amazon is prompting us to login when we click see more reviews.Is there anyway to handle this
How can I scrap product price details of all products in amazon? is it possible?
Thanks a lot brother! Very helpful!!
😅 Nice explanation I'm using scrapy actually but thinking about how to get data everything from the link and is it gonna block my ip address and ..
how to to get reports from amazon seller central and amazon ads using API or we can scrape them.
At 8:09, I get an error that says "requests" is not defined. Can anyone help with this?
Pip install requests
@@niknasrullah2358 it still not working
Hi Darshil,
This one was very informative like your other videos. I like them because they are accurate and to the point and also project-based.
I've one doubt related to web - scraping, like some of the modern web pages don't use visual pagination with the page numbers, instead they have like a show more button which renders the new results.
In this case it doesn't change the webpage so how can we scrape the new results.
You will have to use Selenium
@@DarshilParmar Ok thanks.. I'll try once
Hello, newbie here..every element doesn't contain anchor tags and 'href' in the page. How to access such data in that case?
Check the tag name and you can find it by Class or ID
How do you scrape from the other next pages also
bro your jawline ☠
Captcha if I use your header
How much time is required to scrape atleast 5 pages or how quick is scarpping
cant find the CLASS in ANCHOR tag, ID is there instead of class, where you are extracting the liinks ?
PLEASE HELP !!!
They keep changing structure
@@DarshilParmar so how am i going to extract the links now ?
bro could please tell me how to get product href link and feature image href links in directory
Great tutorial.. for future tutorials, can you reduce the size of your face when writing code, it is tooo big!
it gives me empty list in link
me too
did u find the solution
yea.. Me too. Pl post the solution
you're the best
Wouldnt it be better to use threading and generators since this is I/O task?
Do you think, people who don't know P of python, if I start teaching them about Threading and Generators will understand it?
@@DarshilParmar You are right, I didnt realise this video was for python beginners. I am trying to find out about concurrency in data engineering and how it is done best. Ill check out your other videos, your channel helps me.
I am unable to get data using .find method
same here
Hi Darshil can you make a video on data pipeline and how to dockeriz the entire pipeline
I will add this in my list, thanks for suggestion
Please have a look at discord link I guess its expired I am not able to join
I am getting few errors here can you pleaese help me through google meet?
I'm getting response 503 😥 . Please help !
Are you still getting 503 error?
Did you got any solution for the problem?
The video was too good
suggestion- If you could have written the code in the video instead of reading it in second half, it could be cherry on the cake
It becomes repeatable that's why I did not do it
@@DarshilParmar Ohk Understood. Thank you. Loved your content. looking forward to more videos like this
I am getting none in all title, prize, ratings data
i got response 503 and what can i do next?
i just got the same error i had to use VPN so i can access because in my case the error is caused due to geographical restrictions
Did you got any solution for the problem?
yes I have used different method@@thedailymotivational
@@anilprajapat Can you share what solution you have done?
@@riyamodi8154 i did get any solution properly because amazon can only scraping data for one time after that they applied restrictions on request
When I try to print the links I am getting [] as output
Same issue for me have you found any solution bro..?
Check the body of soup. It's something else.
Iam not getting any output all empty lists
Hello! I can see you are interested in learning how to scrape websites. I can help you get better at it. Let me know if you’d like more details or if you have any questions!
@@japhethmutuku8508 why are some of us getting empty lists