Same here! I have not been able to get a lot of profiles. In fact, I only got like 3-4 profiles out of 200 links. :( It was not too bad yesterday, I got about 8-9 profiles out of 10 profiles in a testing round. No idea what went wrong today. Today though? It was able to crawl lots of profiles but wasn't scraping any of those, and then when I just tried now, it couldn't even crawl anything at all.
the profile.json file is empty, i followed all the steps and copied the code from The article that goes along with this video. What can be the problem?
I passed as a parameter a csv file with a list of 500 usernames to put them in the url but I only got a 19 profiles out of 500. what should I do to get them all ?
I've always had one major problem. That's getting the url links in the first place. When you want 10-20k profiles, manually doesn't make any sense. What do you think about choose one and then have it click to the next profile in their friends list?
LinkedIn has higher protections on any page that lists profiles, etc. so it is harder to scrape reliably. A popular other option is to have a scraper query Google for a person's name and LinkedIn profile using advanced queries and get the LinkedIn username that way. You could feed in a list of people/company name combinations and get all the URLs you need.
@@scrapeops I have been using that method for a little while now and can confirm it is effective at getting people's linkedin url. Most of the time, I get around 85 % of the accounts I wanted. If you really want the remaining 15 %, you will have to manually search.
How do you replace "reidhofman" in profile_list by a list of usernames stored on a CSV ? Let's say I have 500 usernames I want to scrape, all stored on an Excel / CSV. Is it possible and how do you do it ? Thank you for the video very interesting
Can you help me with this plz? Eventhough I am using the scrapeops proxy, I can't scrape the information I need. as it is asking for login, I tried to but it didn't work.... @@dailywisdomquotes518
hi , is this possible to scrape linked public listed data using selenium/scrapy WIHTOUT LOGIN ? I am using proxies but still its getting blocked. Can you please share ideas if you have any ?
Getting an error when trying to pip install scrapy (Python 3.11.1) ERROR: Failed building wheel for twisted-iocpsupport Same thing when trying to manually install Twisted. I was able to install it in a conda environment running python 3.9.15 Anyone else encountered this issue? Haven't found any relevant info online for win os
@@carlosjeanpierresaqui writing it in Jupyter notebook is a lot different to the way shown in the video as you will likely need to use CrawlProcess to run the spider instead of a scrapy project. You will need to make sure you refactor your code correctly for jupyter notebook.
Thank you very much for the tutorial. But it's not working for me .... When I "scrapy crawl linkedin_people_profile", I get the following error "KeyError: 'Spider not found: linkedin_people_profile'" 😢 Eventhough I am following the exact same steps you did Could u help me with this, as it's urgent
Thank you for your tutorial videos. When I repeat your code, I have no problems. I'm sorry, but I want to ask about one problem that does not concern your code specifically, I was looking for answers on stackoverflow, but without success for now. If Scrapy spider closing after first request to start_urls every time in various situations, how to understand what exactly is not working correctly?
If you have no errors in your console when you run your scraper, then it is likely you have set up your code not to make any subsequent requests after it finishes with the start requests.
In the user agreement of Linkedin, you are not allowed to scrape data as I remember. Is there any one who has experience, how much profiles you can scrape without being banned?
LinkedIn don't want you to scrape their pages so that is why it makes it so hard. If you are just using 1 IP address then you will start getting blocked very quickly (10 pages could start getting your requests blocked if you are scraping fast)...They are unlikely to ban you completely that quickly, instead it will start flashing the login screen and stop showing you the data.
If you're using a VPN and being sure not to log in, then you aren't explicitly violating the User Agreement/ Terms of Use. You cross the line when you log in and scrape from behind a user profile. If you are clever about this, you won't ever have to be concerned about getting banned.
If you're using a VPN and being sure not to log in, then you aren't explicitly violating the User Agreement/ Terms of Use. You cross the line when you log in and scrape from behind a user profile. If you are clever about this, you won't ever have to be concerned about getting banned.
We have 2 other tutorials covering scraping company profiles and jobs. If you mean scraping the posts that are in the main feed when you are logged in - we do not do that as you need to be logged in and that is against their terms of service
Dont work for now. Linkedin, from time to time sends a page where 141 li class, and it's not the page that we expect.
I tried this but I am not getting any output. Some script runs but I am not getting the desired output. How to solve this issue.
I am getting always the error code 500 internal server error, can anyone help me?
Same here! I have not been able to get a lot of profiles. In fact, I only got like 3-4 profiles out of 200 links. :( It was not too bad yesterday, I got about 8-9 profiles out of 10 profiles in a testing round. No idea what went wrong today.
Today though? It was able to crawl lots of profiles but wasn't scraping any of those, and then when I just tried now, it couldn't even crawl anything at all.
You make useful lessons. Thank you
Thanks for the feedback!
How can i scrape the contact info , if you can help please share me the code snippet
when i enter "scrapy list" in my terminal, the reponse is "quotes" instead of the spider. Can someone help me with this issue?
Thanks my friend
the profile.json file is empty, i followed all the steps and copied the code from The article that goes along with this video. What can be the problem?
Assalamualaikum brother
I'm also getting the same problem
Have u find the solution of that problem, if yes kindly request you to help me
I passed as a parameter a csv file with a list of 500 usernames to put them in the url but I only got a 19 profiles out of 500. what should I do to get them all ?
Do you found a solution for thath ?
It seems that You have consumed all your FREE API credits. You should upgrade to a larger plan to activate more credits!!
Ran without any errors but returned an empty list in the profile.json. Can anyone tell me what to do?
it runs with no errors for me but I don't get any data back
same it just returns an empty list
what is css selector for "open to work" status of linkedin user profiles?@scrapeops or anyone maybe can help me?
try many times but still didnt works
How to go about extracting skills and endorsements?
You just add CSS selectors to extract it from the HTML response like the other data types.
Question if I wanted to scrape all the profiles from linkedin for a specific country. Say Hungary. How would I go about doing that ?
So this bypasses the usual limits on most other tools?
I've always had one major problem. That's getting the url links in the first place. When you want 10-20k profiles, manually doesn't make any sense.
What do you think about choose one and then have it click to the next profile in their friends list?
LinkedIn has higher protections on any page that lists profiles, etc. so it is harder to scrape reliably.
A popular other option is to have a scraper query Google for a person's name and LinkedIn profile using advanced queries and get the LinkedIn username that way. You could feed in a list of people/company name combinations and get all the URLs you need.
@@scrapeops I have been using that method for a little while now and can confirm it is effective at getting people's linkedin url. Most of the time, I get around 85 % of the accounts I wanted. If you really want the remaining 15 %, you will have to manually search.
bro i copy your github code but unfortunately not scrap data and also i cant found error except fingerprint
How do you replace "reidhofman" in profile_list by a list of usernames stored on a CSV ? Let's say I have 500 usernames I want to scrape, all stored on an Excel / CSV. Is it possible and how do you do it ? Thank you for the video very interesting
yes it is, i did it
Can you help me with this plz?
Eventhough I am using the scrapeops proxy, I can't scrape the information I need. as it is asking for login, I tried to but it didn't work....
@@dailywisdomquotes518
@@dailywisdomquotes518 Hie could you please help me doing this ?
hi , is this possible to scrape linked public listed data using selenium/scrapy WIHTOUT LOGIN ? I am using proxies but still its getting blocked. Can you please share ideas if you have any ?
did he logged in?
it is still showing me the error No module named 'scrapeops_scrapy' after reactivating virtual environment
i am getting the same error too. @scrapeops could you please help?
Can we scrap linkedin followers from my linkedin page
Getting an error when trying to pip install scrapy (Python 3.11.1)
ERROR: Failed building wheel for twisted-iocpsupport
Same thing when trying to manually install Twisted. I was able to install it in a conda environment running python 3.9.15
Anyone else encountered this issue? Haven't found any relevant info online for win os
Try "pip install wheel" in your venv
The file that originates .json comes out empty, could you help me :c
Did you git clone the code in the GitHub repo and ran that? Or did your write the code yourself?
@@scrapeops I wrote it myself using Jupyter Notebook
@@carlosjeanpierresaqui writing it in Jupyter notebook is a lot different to the way shown in the video as you will likely need to use CrawlProcess to run the spider instead of a scrapy project. You will need to make sure you refactor your code correctly for jupyter notebook.
Do you know how to scrape linkedin profiles over 15k followers?
Thank you very much for the tutorial.
But it's not working for me ....
When I "scrapy crawl linkedin_people_profile", I get the following error "KeyError: 'Spider not found: linkedin_people_profile'" 😢
Eventhough I am following the exact same steps you did
Could u help me with this, as it's urgent
Thank you I have solved the problem, it was just about changing the running directory cd
Thank you for your tutorial videos.
When I repeat your code, I have no problems.
I'm sorry, but I want to ask about one problem that does not concern your code specifically,
I was looking for answers on stackoverflow, but without success for now.
If Scrapy spider closing after first request to start_urls every time in various situations,
how to understand what exactly is not working correctly?
If you have no errors in your console when you run your scraper, then it is likely you have set up your code not to make any subsequent requests after it finishes with the start requests.
do you have to login to linkedin first
No, the scraper only scrapes public pages so you don't need to login.
@@scrapeops but it shows 999 status code. what does that mean. it's usually giving 200 status code
@@azhari7968 i've got this error too, have you found any way to get around?
@@tranngocduc5653 I gave up using scrapy. I use selenium and beautifulsoup instead
I get 999 error
In the user agreement of Linkedin, you are not allowed to scrape data as I remember. Is there any one who has experience, how much profiles you can scrape without being banned?
LinkedIn don't want you to scrape their pages so that is why it makes it so hard.
If you are just using 1 IP address then you will start getting blocked very quickly (10 pages could start getting your requests blocked if you are scraping fast)...They are unlikely to ban you completely that quickly, instead it will start flashing the login screen and stop showing you the data.
@@scrapeops So using ScrapeOps Proxy could help resolving this issue?
@@xev Yes, we have proxy providers in our network that can scrape LinkedIn so if you use the proxy then you will be able to reliably scrape it.
If you're using a VPN and being sure not to log in, then you aren't explicitly violating the User Agreement/ Terms of Use. You cross the line when you log in and scrape from behind a user profile. If you are clever about this, you won't ever have to be concerned about getting banned.
If you're using a VPN and being sure not to log in, then you aren't explicitly violating the User Agreement/ Terms of Use. You cross the line when you log in and scrape from behind a user profile. If you are clever about this, you won't ever have to be concerned about getting banned.
The exact code provides the following response: 500 Internal Server Error. This didn't happen until last week.
This was an issue with an underlying proxy provider that is being used when scraping Linkedin. It should be fixed now.
@@scrapeops Thank goodness. I'll try again tomorrow. Thank you 👍👍👍
I am getting the same error. Is there a way around it?
@@hansbaumberger4234 contact them and wait. It took a few days but they eventually fixed this.
The project run correctly but the json file is empty ? Thank you
Having the same issue, could you solve it?
@@hansbaumberger4234 i get the same problem please could you contact me if you find the answer ?
is this legal?
in a gray area .. but it is what google and chatgpt do :)
can you scrape posts?
We have 2 other tutorials covering scraping company profiles and jobs. If you mean scraping the posts that are in the main feed when you are logged in - we do not do that as you need to be logged in and that is against their terms of service