Another great tutorial Samer, thanks a lot! I have started learning RSelenium some time ago, and it would have been so great to have this video at that time.
Wow this is by far the best tutorial I've ever seen on RSelenium! Everything is so helpful and making it a video really helps me follow along. Fantastic!
I have never coded in my life and was able to follow this tutorial to start what I am looking to do. I can't tell you how excited I am. Thank you so much!! Made some things a legitimate option for me to figure out. I was able to use this and apply it to a specific webpage I am looking to navigate
You are way better at teaching this than any resource on the web, and even better than my teacher! love the videos. Could you make a video scraping multiple pages and putting it into a data frame or xlsx?
Very helpful, I think that’s exactly what I needed to perform a task at my job, thanks a lot!!! Just as an FYI, it was just a tiny typo at the end, you typed jave.exe (with an e) and not java.exe Thanks again!
Hello Samer, thanks a million for the detailed explanation given to each line of code. Its been really helpful. Please at your spare time could you make a video about scrapping data from TH-cam using R (using either RSelenium or RVest) or you could refer me to any tutorial about such if you know any Thanks. Please treat as urgent.
I'm glad to hear this, Uche! Although there's lots of videos on scraping TH-cam data with Python, I am not sure about any R tutorials for that. Perhaps look into the TH-cam API for this. Anyhow, I would like to visit this topic and see if I can make a video on it. Thanks!
Great tutorial! I was wondering if you could provide advice on how to run RSelenium just like this while using a proxy? I'm unable to find any good tutorials out there on how to do this
Great video! Thanks so much. I have a question and is what if we want a specific element (like prices) for multiple pages with r selenium and no just the first page, ¿how we can get this?
Thanks Juan! You'll have to click on the next button and move to the next page. You can do so using the clickElement() function. Then put that in a loop so it loops over all the pages.
Hey Samer, thanks for your video, it helped me get started with RSelenium. Do you happen to know some way to make the steps taken by RSelenium longer? I'm using it to scrape data from multiple pages, using a for loop (enter link -> click button -> get data -> start over with another link). However, because everything happens so fast the pages can't even fully load and it can't get the data before going to the next loop. Any tips?
Hi @SamerHijjazi, Thanks for the great work. I was wondering if there's a way to use RSelenium within a ShinyApp? Any help would be greatly appreciated.
Hi Samar, How do we select a drop down menu in Rselenium. I tried using findElement drop down name and clickElement() but not sure how to select a value from that. Also the drop down is not expanding to show the options
Thank you very much @Samer Hijjazi Is me who asked you the question about the possibility to use RSelenium from Rstudio cloud during the meetup @R-ladies Tunis
@Samer Hiijazi Great tutorial, as always! Would be super helpful to also have some instructions on how to set up a VNC so that remDr$()open actually opens a Browser window!
Many thanks for putting this tutorial together! however, when I use the binman::list_versions("chromedriver"), I get the following error: binman::list_versions("chromedriver") - any suggestions?
Magnificent tutorials. Could you please teach us how to scrape Facebook, the Marketplace? Or public comments from any public group from Facebook? We would greatly appreciate it. Regards and keep going crack
Salam moh. Great question. There are several functions that help you deal with alerts. The most common being the acceptAlert() function, which is the same as clicking "OK" on the alert. You can also use the dismissAlert() function to ignore the alert. As for the frames, if you'd like to switch to the frame, use the switchToFrame() function. I recommend you check out the documentation for both of these. However, I plan to make deeper dive videos into RSelenium.
Hi. When I give command: R.4.2.0 > binman::list_versions('chromedriver'), I get: Error: C:\Users\eduardo\AppData\Local/binman/binman_chromedriver app directory not found. Am I missing something? Thanks
Oh, I figured out: We must first run a successful RSelenium::rsDriver in order to create the missing directories. Agter that the binman command works ok
@@flordemaria127 To create the needed directories, You must run for the first time: selenium = RSelenium::rsDriver( browser = "chrome" ) After that you can now run binman::list_versions('chromedriver') you can then stop selenium server selenium[["server"]]$stop() and then run the RSelenium::rsDriver command again, this time fine-tuning all the parameters you could need. I hope it helps
not sure if youll respond but I get this error: Could not open chrome browser. Client error message: Undefined error in httr call. httr output: Failed to connect to localhost port 14415: Connection refused Check server log for further details.
Hi Samer. Fantastic job. I'm trying to follow along your tutorial, but I'm getting an error in rsDriver command. Could you help me? I need to install the chrome webdriver for selenium? Thank you very much for your help.
Hi Samer, nice video. I keep getting Geckodriver timeout when I try to load driver. Do you know why everytime R runs that predownload process for different things like phantom, geckodriver ete?
Hi there! Hmmm, I wouldn't have a solution at the moment. Perhaps try to reinstall the RSelenium package. Also, make sure you have Java installed on your computer.
Hello Samer, thank you a perfect tutorial! Could you please help me with this error I'm getting when trying to set up the server (JAVA is installed, chrome version is all right): Selenium message:Unable to create new service: ChromeDriverService Could not open chrome browser. Client error message: Summary: SessionNotCreatedException Detail: A new session could not be created. Further Details: run errorDetails method Check server log for further details.
@@adamruschka Thank you for the great feedback! I was looking for some solutions via stack overflow, but then I saw you figured it out. What was the problem?
@@SamerHijjazi Hello. The problem was in the AppData\...\binman. The folders containing the .exe drivers needed to be erased and downloaded again. I'm not exactly sure why, but making the script to download them once again made it running.
Amazing tutorial Samer. However, I kept getting this error: Could not open chrome browser. Client error message: Undefined error in httr call. httr output: Failed to connect to localhost port 14415: Connection refused Check server log for further details. Warning message: In rsDriver(browser = "chrome", chromever = "114.0.5735.90", verbose = FALSE, : Could not determine server status. Any suggestions will be really appreciated.
Thank you very much for the great instructions. It helps me a lot. I works well within RStudio IDE. However, when I tried to put the script into cron job to automate it everyday, it did not work and an error message occurred "sh: netstat: command not found". I checked and the netstat is already installed from Terminal. Not sure what happened. Could you please help with this? Much appreciated! I'm using MacOS
Hello! Thanks for the excellent contribution! I used this guide on my personal computer, and everything worked fine. However, on my job computer, I'm encountering an error with the free_port function (it seems to be a language issue, the operating system must be in spanish on my laptop). When creating the "rs_driver_object," it throws the following error: "Error in gsub("\s+", ",", ActiveConnections) : input string 1 is invalid In addition: Warning message: In gsub("\s+", ",", ActiveConnections) : unable to translate ' Proto Direccin local Direccin remota Estado' to a wide string" Do you know of any other way to find a free port? Greetings from Chile!
Hi there! Thank you for the great feedback. To find a list of free ports to use, use the free_ports() function from the net stat package. That will give you a list of all the free ports you can use. Then manually enter the number in place of the port argument followed by an L. So for example, if 21483 is a free port, you want to set the port = 21483L. I hope this helps!
Hi Samer, Thanks for the amazing video. I have question. When i run rsDriver(), i get this error: "Error in if (file.access(phantompath, 1) < 0) { : argument is of length zero". I am new to RSelenium and i am not able to figure out how to fix this issue. Any suggestion would be really helpful. Thanks
Thank you for this tutorial. Well done! However, after trying two different computers (both Win10 64-bit), different internet connection(s), also with antivirus offf... and most of the proposed solutions below, I still get the message "could not open chrome [or firefox...] browser. Client error message: Undefined error in httr call. httr output: Failed to connect to localhost port 4567: Connection refused Check server log for further details. Warning message: In RSelenium::rsdriver(browser = "chrome") : Could not determine server status." Can you, please, review, if the tutorial (which is otherwise great) is still working with the current version of chrome/r/rstudio, the packages in question? I have certain preoccupation (based also on quick search elsewhere on the internet) that for many users, the suggested approach does not work. My apologies for this stupid question.
Hello! This issue has been brought to my attention recently. I have found a solution to your error and I am hoping to make an update video soon. Thank you for the feedback :)
Hello sir. I got an issue message when using the function 'findElement'. that said : Selenium message:no such element: Unable to locate element: {"method":"css selector","selector":"#pagination\-table\-container"} (Session info: chrome=107.0.5304.107)
I have struggled with this issue for the last 3 months. After doing all your steps, i get: BEGIN: POSTDOWNLOAD Error in if (file.access(phantompath, 1) < 0) { : argument is of length zero
I'm having issues getting the Chrome session to start. Every time I run the code, I get the following: Could not open chrome browser. Client error message: Summary: UnknownError Detail: An unknown server-side error occurred while processing the command. Further Details: run errorDetails method Check server log for further details. I've installed the most recent version of Java jdk but I still get this. Oh well, guess I won't be using Selenium then lol. Such a shame since this is a great video
my chrome version is 124.0.6367.92, But the only options i have in chromever is 113.0.5672.63" "114.0.5735.16" "114.0.5735.90" do i have to install an older version of chrome?
what do i do if the chrome version i have does not match up with the versions listed? this is what it shows me: > ?rsDriver > binman::list_versions("chromedriver") $win32 [1] "113.0.5672.63" "114.0.5735.16" [3] "114.0.5735.90" but the version i have of chrome is 124.0.6367.92 thanks in advance!
Hello! This issue has been brought to my attention recently. I have found a solution to your error and I am hoping to make an update video soon. Thank you for the feedback :)
DISCLAIMER: If you're having trouble running RSelenium, I would suggest you watch my installation guide here:
th-cam.com/video/GnpJujF9dBw/w-d-xo.html
Wow, I spent 6 hours failing at what this video showed me how to do in 16 minutes. 45 if you count when I paused it.
Great video Samer, the interactive nature of RSelenium in controlling the web browser is really neat.
Excellent Video on RSelenium Samer. Thanks for benefitting the R community.
Another great tutorial Samer, thanks a lot! I have started learning RSelenium some time ago, and it would have been so great to have this video at that time.
Thanks a ton for your support, Joachim! The lack of videos on RSelenium is what encouraged me to make this video haha :)
Wow this is by far the best tutorial I've ever seen on RSelenium! Everything is so helpful and making it a video really helps me follow along. Fantastic!
Thanks, Samer! My Selenium-Python bot stopped working, but R (and your tutorial) saved me!
I have never coded in my life and was able to follow this tutorial to start what I am looking to do. I can't tell you how excited I am. Thank you so much!! Made some things a legitimate option for me to figure out. I was able to use this and apply it to a specific webpage I am looking to navigate
Samer: This is great. Amazing that it is your first video. You are a natural. I look forward to many more and am subscribed.
Thank you, Peter! I really appreciate it :)
Hi Samer. You are such a great teacher. I learnt a lot from this video. Thank you for your time and effort.
My pleasure. Thank you for the kind comment!
You rock Samer! Thanks so much. This was exactly the get going information I needed to begin a work related Selenium webscrape. 🙌🙌
Thank you for the wonderful feedback. I'm glad it helped!
This video helped me a lot. I'm already using RSelenium at my job. Thank you so much
You're welcome! Glad to hear it helped you out
You are way better at teaching this than any resource on the web, and even better than my teacher! love the videos. Could you make a video scraping multiple pages and putting it into a data frame or xlsx?
Thank you for the awesome feedback! That's a pretty good idea. I'd like to make a video like that in the future :)
Really helpful tutorial for RSelenium. Thanks a lot!
Absolute Goat. This was awesome. Had some hiccups setting up chrome browser but you even had videos for that thank you.
Thx a lot Samer Hijjazi. Very nice explanation.
Very helpful, I think that’s exactly what I needed to perform a task at my job, thanks a lot!!!
Just as an FYI, it was just a tiny typo at the end, you typed jave.exe (with an e) and not java.exe
Thanks again!
Thank you for the great feedback. Glad to hear it's helped you out at your job. Grrrrr, it's always the tiny typos that get me. Great catch!
Thanks!
Thank you for the super thanks!! Really appreciate it 😊
Hello Samer, thanks a million for the detailed explanation given to each line of code. Its been really helpful.
Please at your spare time could you make a video about scrapping data from TH-cam using R (using either RSelenium or RVest) or you could refer me to any tutorial about such if you know any
Thanks. Please treat as urgent.
I'm glad to hear this, Uche! Although there's lots of videos on scraping TH-cam data with Python, I am not sure about any R tutorials for that. Perhaps look into the TH-cam API for this. Anyhow, I would like to visit this topic and see if I can make a video on it. Thanks!
Super helpful. Thank you!
thank you. It was very helpful. you're a great teacher!
This video was great, thanks!
Thank you very much, Tivan!
kind Sir. I can't thank you enough for this !
thank so much, It's the first time this works for me, great video
Great video, thanks for sharing
Great video
Great tutorial! I was wondering if you could provide advice on how to run RSelenium just like this while using a proxy? I'm unable to find any good tutorials out there on how to do this
Thank you for the kind comment! I have not tried that before, but maybe I can look into it.
Great tutorial. Thank you so much.
what do I do when my chrome version (120.0.6099.109) isn't in binman::listversions? I am using RSelenium ver 1.7.9
Great video! Thanks so much.
I have a question and is what if we want a specific element (like prices) for multiple pages with r selenium and no just the first page, ¿how we can get this?
Thanks Juan! You'll have to click on the next button and move to the next page. You can do so using the clickElement() function. Then put that in a loop so it loops over all the pages.
Great tutorial!
Hey Samer, thanks for your video, it helped me get started with RSelenium.
Do you happen to know some way to make the steps taken by RSelenium longer? I'm using it to scrape data from multiple pages, using a for loop (enter link -> click button -> get data -> start over with another link). However, because everything happens so fast the pages can't even fully load and it can't get the data before going to the next loop. Any tips?
My pleasure! Try using the Sys.sleep function between your lines of code. You can specify how long it should wait before moving to the second line
Hi @SamerHijjazi, Thanks for the great work. I was wondering if there's a way to use RSelenium within a ShinyApp? Any help would be greatly appreciated.
Hi! Thank you for the feedback. I personally do not have much experience with Shiny, so I wouldn't know.
Excellent!
Great! Subscribed!
Hi Samar, How do we select a drop down menu in Rselenium. I tried using findElement drop down name and clickElement() but not sure how to select a value from that. Also the drop down is not expanding to show the options
Make sure to inspect the elements in your dropdown after you click on it. Then use those attributes to help click on the elements.
Hi Samer thanks for the helpful video. I am getting an error httr output: Failed to connect to localhost port. Can you please guide
Hi Sukalyan. Do you have the Java Development Kit installed? You may need to install and set it up first to solve the issue you're running into.
Great video. Can you do a video on how to bypass Cloudflare and other anti-bot tactics?
Thank you! It might be a little challenging, but I'm willing to explore this idea :)
Thank you very much @Samer Hijjazi Is me who asked you the question about the possibility to use RSelenium from Rstudio cloud during the meetup @R-ladies Tunis
Thank you for the feedback! I remember you from that workshop. Glad you liked the video :)
I get this: "PATH to JAVA not found. Please check JAVA is installed.". So, I can not continue. ¡Help!
You have to install JAVA for RSelenium to work. Please refer to my RSelenium Installation Guide tutorial on my channel.
@Samer Hiijazi Great tutorial, as always! Would be super helpful to also have some instructions on how to set up a VNC so that remDr$()open actually opens a Browser window!
Many thanks for putting this tutorial together! however, when I use the binman::list_versions("chromedriver"), I get the following error: binman::list_versions("chromedriver") - any suggestions?
Thank you for the feedback. You may need to install the binman package first. See if that solves it.
Magnificent tutorials.
Could you please teach us how to scrape Facebook, the Marketplace?
Or public comments from any public group from Facebook?
We would greatly appreciate it.
Regards and keep going crack
Thank you! Social media is usually a little trickier to scrape, so I probably won't cover it on this channel
Salam Samer, Hope to explain how to deal with frames and alerts, etc
Salam moh. Great question. There are several functions that help you deal with alerts. The most common being the acceptAlert() function, which is the same as clicking "OK" on the alert. You can also use the dismissAlert() function to ignore the alert.
As for the frames, if you'd like to switch to the frame, use the switchToFrame() function. I recommend you check out the documentation for both of these. However, I plan to make deeper dive videos into RSelenium.
Hi. When I give command: R.4.2.0 > binman::list_versions('chromedriver'), I get: Error: C:\Users\eduardo\AppData\Local/binman/binman_chromedriver app directory not found. Am I missing something? Thanks
Oh, I figured out: We must first run a successful RSelenium::rsDriver in order to create the missing directories. Agter that the binman command works ok
@@tugastone How do you do that? I'm getting the same error. Thanks!
@@flordemaria127 To create the needed directories, You must run for the first time: selenium = RSelenium::rsDriver( browser = "chrome" ) After that you can now run binman::list_versions('chromedriver') you can then stop selenium server selenium[["server"]]$stop() and then run the RSelenium::rsDriver command again, this time fine-tuning all the parameters you could need. I hope it helps
@@tugastone Thank you very much!
Thanks for helping out! :)
not sure if youll respond but I get this error:
Could not open chrome browser.
Client error message:
Undefined error in httr call. httr output: Failed to connect to localhost port 14415: Connection refused
Check server log for further details.
Hi! Watch my RSelenium installation guide. It should help out: th-cam.com/video/GnpJujF9dBw/w-d-xo.html
Hi Samer. Fantastic job. I'm trying to follow along your tutorial, but I'm getting an error in rsDriver command. Could you help me? I need to install the chrome webdriver for selenium? Thank you very much for your help.
Thanks Jose! Please refer to my RSelenium Installation Guide tutorial.
@@SamerHijjazi, thank you very much. Delete the license file it's all that was needed.
Hi Samer, nice video. I keep getting Geckodriver timeout when I try to load driver. Do you know why everytime R runs that predownload process for different things like phantom, geckodriver ete?
Hi there! Hmmm, I wouldn't have a solution at the moment. Perhaps try to reinstall the RSelenium package. Also, make sure you have Java installed on your computer.
Hello Samer, thank you a perfect tutorial! Could you please help me with this error I'm getting when trying to set up the server (JAVA is installed, chrome version is all right):
Selenium message:Unable to create new service: ChromeDriverService
Could not open chrome browser.
Client error message:
Summary: SessionNotCreatedException
Detail: A new session could not be created.
Further Details: run errorDetails method
Check server log for further details.
Nevermind, I managed to solve the issue. If anyone else has the same problem, I'm happy to help.
@@adamruschka Thank you for the great feedback! I was looking for some solutions via stack overflow, but then I saw you figured it out. What was the problem?
@@SamerHijjazi Hello. The problem was in the AppData\...\binman. The folders containing the .exe drivers needed to be erased and downloaded again. I'm not exactly sure why, but making the script to download them once again made it running.
Do you know if RSelenium has stopped being maintained? It doesn't seem to support the most recent version of Chrome.
RSelenium has stopped being maintained a long time ago, although it still works. Refer to my latest video on setting it up, as the setup has changed
Amazing tutorial Samer. However, I kept getting this error:
Could not open chrome browser.
Client error message:
Undefined error in httr call. httr output: Failed to connect to localhost port 14415: Connection refused
Check server log for further details.
Warning message:
In rsDriver(browser = "chrome", chromever = "114.0.5735.90", verbose = FALSE, :
Could not determine server status.
Any suggestions will be really appreciated.
Thank you very much for the great instructions. It helps me a lot. I works well within RStudio IDE. However, when I tried to put the script into cron job to automate it everyday, it did not work and an error message occurred "sh: netstat: command not found". I checked and the netstat is already installed from Terminal. Not sure what happened. Could you please help with this? Much appreciated! I'm using MacOS
Thank you for the amazing feedback. Try removing the free_port argument from the rsDriver function. That could help
Hello! Thanks for the excellent contribution! I used this guide on my personal computer, and everything worked fine. However, on my job computer, I'm encountering an error with the free_port function (it seems to be a language issue, the operating system must be in spanish on my laptop). When creating the "rs_driver_object," it throws the following error:
"Error in gsub("\s+", ",", ActiveConnections) : input string 1 is invalid
In addition: Warning message:
In gsub("\s+", ",", ActiveConnections) :
unable to translate ' Proto Direccin local Direccin remota Estado' to a wide string"
Do you know of any other way to find a free port?
Greetings from Chile!
Hi there! Thank you for the great feedback. To find a list of free ports to use, use the free_ports() function from the net stat package. That will give you a list of all the free ports you can use. Then manually enter the number in place of the port argument followed by an L. So for example, if 21483 is a free port, you want to set the port = 21483L.
I hope this helps!
Hi Samer, Thanks for the amazing video. I have question. When i run rsDriver(), i get this error: "Error in if (file.access(phantompath, 1) < 0) { :
argument is of length zero". I am new to RSelenium and i am not able to figure out how to fix this issue. Any suggestion would be really helpful. Thanks
Hi pyarasid, thank you for feedback! Have you installed the java development kit first?
@@SamerHijjazi Hi Samer thanks for the video. I received the same error, and I do have jdk installed. I'm stumped, any thoughts?
@@MCshaneization Hi. Try this .
binman::rm_platform("phantomjs")
wdman::selenium(retcommand = TRUE)
For me, worked.
Thank you for this tutorial. Well done! However, after trying two different computers (both Win10 64-bit), different internet connection(s), also with antivirus offf... and most of the proposed solutions below, I still get the message "could not open chrome [or firefox...] browser. Client error message: Undefined error in httr call. httr output: Failed to connect to localhost port 4567: Connection refused Check server log for further details. Warning message: In RSelenium::rsdriver(browser = "chrome") : Could not determine server status." Can you, please, review, if the tutorial (which is otherwise great) is still working with the current version of chrome/r/rstudio, the packages in question? I have certain preoccupation (based also on quick search elsewhere on the internet) that for many users, the suggested approach does not work. My apologies for this stupid question.
Hello! This issue has been brought to my attention recently. I have found a solution to your error and I am hoping to make an update video soon. Thank you for the feedback :)
@@SamerHijjazi I have the same, is it related to using Mac OS?
How can I use a specific chrome profile in selenium? Thanks
I don't believe you can do that in Selenium. At least not from my understanding
Hi samer.
why getting this error in rsDriver ?
rs_obj
Please refer to my installation tutorial: th-cam.com/video/GnpJujF9dBw/w-d-xo.html
Hello sir. I got an issue message when using the function 'findElement'. that said :
Selenium message:no such element: Unable to locate element: {"method":"css selector","selector":"#pagination\-table\-container"}
(Session info: chrome=107.0.5304.107)
Make sure you're referring to the right syntax when trying to locate by either css or xpath. That's what's causing the error.
When I create the selenium server the log says there is some problem, I dont know how to solve it.
What could be happening?
Please refer to my RSelenium Installation Guide video. That should help! :)
@@SamerHijjazi I solve it few days ago with that video.
Thx!
I have struggled with this issue for the last 3 months. After doing all your steps, i get: BEGIN: POSTDOWNLOAD
Error in if (file.access(phantompath, 1) < 0) { :
argument is of length zero
Hi there! Within the rsDriver function, you'll need to set the phantomver = NULL. That should make it work.
Why didn't you explain how to set up selenium?
I made an updated video on that. th-cam.com/video/GnpJujF9dBw/w-d-xo.html
can you make R selenium for automate youtube?
Possibly! If you're trying to obtain data from TH-cam, you can use the TH-cam API
I'm having issues getting the Chrome session to start. Every time I run the code, I get the following:
Could not open chrome browser.
Client error message:
Summary: UnknownError
Detail: An unknown server-side error occurred while processing the command.
Further Details: run errorDetails method
Check server log for further details.
I've installed the most recent version of Java jdk but I still get this.
Oh well, guess I won't be using Selenium then lol. Such a shame since this is a great video
Try using Firefox. It still works the same way as Chrome. Also thank you for the wonderful feedback!
@@SamerHijjazi thanks so much for your reply! Your suggestion worked, so glad I don’t need to give up on RSelenium. Thanks so much again!
my chrome version is 124.0.6367.92,
But the only options i have in chromever is 113.0.5672.63" "114.0.5735.16" "114.0.5735.90"
do i have to install an older version of chrome?
what do i do if the chrome version i have does not match up with the versions listed?
this is what it shows me:
> ?rsDriver
> binman::list_versions("chromedriver")
$win32
[1] "113.0.5672.63" "114.0.5735.16"
[3] "114.0.5735.90"
but the version i have of chrome is 124.0.6367.92
thanks in advance!
Could you please help me resolve the following error, I am unable to connect to the server.
I typed the following code:
rD
Hello! This issue has been brought to my attention recently. I have found a solution to your error and I am hoping to make an update video soon. Thank you for the feedback :)
@@SamerHijjazi Hi I am getting the same error too and cannot solve it. An updated video soon would be awesome!