Thank you for having me, Joachim! This was a very fun video to make. I certainly enjoyed testing out RSelenium on your website. Looking forward to more collaborations with you in the future!
This video has been tremendously helpful. I'm brand new to using Selenium and was able to get things up and running and accomplish my goal for a work project thanks to the guidance provided here -- thanks so much!!
Thanks for the helpful video! I am trying to automate a website that requires a user login every time. Is there a way to save cookies so that it will bypass the login screen and go straight to the link?
hi, thank you for the tutorial. I had some problem with captcha and reChapca, they detect i am a bot, do you guys know how to avoid chaptchas that block my web scraping?
Hey Gio, unfortunately, I don't have experience with that. This thread seems to be interesting, even though it's on Python it might be useful to you: stackoverflow.com/questions/58872451/how-can-i-bypass-the-google-captcha-with-selenium-and-python
There is a way to put in a pause before you go to the next page. I thought Samer covered it. Check out Kasper Welbers "Webscraping in R" video on youtube. It is excellent.
Hi. When i run "binman::list_versions('chromedriver')" i get this error: "Error: C:\Users\0000\AppData\Local/binman/binman_chromedriver app directory not found.". can you help me please?
I have seen Samer's video on his youtube page and they are excellent. However, I am still have trouble with RSelenium on a Mac OS. I am also a bit worried about "scraping" (unless this is just to facilitate downloading - not scraping multiple files) since using robots to run through webpages is a bit dicy and borders on unethical (unless you have explicit permission to do so). Hence I am sticking with rvest and httr to scrape a few tables from Wikipedia or other sites that allow it.
Hi Haraldur, thank you for sharing your thoughts on this! Unfortunately, I do not have any experience using this on a Mac OS, so I won't be able to help with that. Regarding the ethical aspect, I definitely agree with this. Users should definitely make sure that they do not break any rules/laws when scraping data. Regards, Joachim
Hi. When I give command: R.4.2.0 > binman::list_versions('chromedriver'), I get: Error: C:\Users\eduardo\AppData\Local/binman/binman_chromedriver app directory not found. Am I missing something? Thanks
Oh, I figured out: We must first run a successful RSelenium::rsDriver in order to create the missing directories. Agter that the binman command works ok
Thank you for your reply. Yes, using the binman command returns, at the latest, version 114 of Chrome. Chrome is up to version 120 now. I have downloaded the latest chrome webdriver and get an undefined error in httr call. Do you have a better way to contact you? It doesn't seem like the package is being maintained, honestly. Or, do I need to do something?
Hey, I'm not sure if RSelenium is still updated. I think it would be best to contact the developers of the package directly via GitHub: github.com/ropensci/RSelenium I hope this helps! Joachim
Hello! Thanks for the video. I am getting this error : [1] "Connecting to remote server" Error in checkError(res) : Undefined error in httr call. httr output: Failed to connect to localhost port 14415: Connection refused by using rD
Hello Richard, Unfortunately, I am not much experienced in RSelenium. Have you checked this thread on StackOverflow: stackoverflow.com/questions/45395849/cant-execute-rsdriver-connection-refused. If it doesn't help, you can post your question on our Facebook discussion group: facebook.com/groups/statisticsglobe. This way, other users can read and help. Regards, Cansu
Hey, please note that bypassing Cloudflare's detection mechanisms using tools like RSelenium to access or scrape web content can violate terms of service and ethical guidelines. It's important to respect website security measures and seek legal ways to access data, such as using official APIs or obtaining permission from the website owner.
This didn't work for me. Pretty early in the video, I got an error than I couldn't rectify with anything in this video, or doing Google search. Basically, the way of finding my Chrome version didn't work.
Hey Nathan, could you please share your code and the error message that was returned? Without seeing your code, it will be difficult for me to find the problem. Thanks, Joachim
@@cansustatisticsglobe Hi, I got Error: C:\Users\thom\AppData\Local/binman/binman_chrome_driver app directory not found. the package seems to load fine but it looks like there is a problem with it?
@@StatisticsGlobe Hi, it's perhaps a problem with the binman package. i get ...Error: C:\Users\thomp\AppData\Local/binman/binman_chrome_driver app directory not found. Thanks,
@@nathanthompson8652 You might be right, but unfortunately, I couldn't find a solution either. I think you should post your question on our Facebook discussion group: facebook.com/groups/statisticsglobe, someone, who is more experienced in the Rselenium package, might help. Regards, Cansu
I got 4:20 into the video and have problem with the version. Current chrome version is 99.0.4844.84. I get an error setting rs_driver_object. (binman list is: "100.0.4896.20" "99.0.4844.35" "99.0.4844.51" ) Is there a "work around?"
Hi Robert, my apologies for seeing this very late. In this case, you will have to use the 99.0.4844.51 value from the binman list. You always want to use the version from binman list that is closest to your current chrome version. I hope this helps!
Thank you for having me, Joachim! This was a very fun video to make. I certainly enjoyed testing out RSelenium on your website. Looking forward to more collaborations with you in the future!
Thanks a lot for the great collaboration Samer, it was an absolute please to working with you!
This video has been tremendously helpful. I'm brand new to using Selenium and was able to get things up and running and accomplish my goal for a work project thanks to the guidance provided here -- thanks so much!!
Thanks a lot for the wonderful feedback Ryan, glad the tutorial was useful!
This is awesome, thank you so much both. Hope your talk went well on the 15th Samer, saw it too late on linkedin!
Thank you for the kind comment, glad you like the video!
Amazing, thanks for the tutorial
Hi Fernando, thanks for the positive feedback! Hope the video was helpful!
Thank you so much for the toturial!
Thanks Dela! Glad it was helpful!
Great!!!!
Thanks for the feedback! Glad you liked the video!
Thanks for the helpful video! I am trying to automate a website that requires a user login every time. Is there a way to save cookies so that it will bypass the login screen and go straight to the link?
Hello Eric,
Sorry for the late reply. Do you still need help? If so, I will contact the main author of this tutorial.
Regards,
Cansu
Thanks for this video
Most welcome Ángel, glad you like it!
hi, thank you for the tutorial. I had some problem with captcha and reChapca, they detect i am a bot, do you guys know how to avoid chaptchas that block my web scraping?
Hey Gio, unfortunately, I don't have experience with that. This thread seems to be interesting, even though it's on Python it might be useful to you: stackoverflow.com/questions/58872451/how-can-i-bypass-the-google-captcha-with-selenium-and-python
There is a way to put in a pause before you go to the next page. I thought Samer covered it. Check out Kasper Welbers "Webscraping in R" video on youtube. It is excellent.
Hi. When i run "binman::list_versions('chromedriver')" i get this error: "Error: C:\Users\0000\AppData\Local/binman/binman_chromedriver app directory not found.". can you help me please?
Hi Juan, do you still need help or have you fixed the issue? Sorry for my late response, I've been on vacation and couldn't reply sooner.
I have seen Samer's video on his youtube page and they are excellent. However, I am still have trouble with RSelenium on a Mac OS. I am also a bit worried about "scraping" (unless this is just to facilitate downloading - not scraping multiple files) since using robots to run through webpages is a bit dicy and borders on unethical (unless you have explicit permission to do so). Hence I am sticking with rvest and httr to scrape a few tables from Wikipedia or other sites that allow it.
Hi Haraldur, thank you for sharing your thoughts on this! Unfortunately, I do not have any experience using this on a Mac OS, so I won't be able to help with that. Regarding the ethical aspect, I definitely agree with this. Users should definitely make sure that they do not break any rules/laws when scraping data. Regards, Joachim
Hi. When I give command: R.4.2.0 > binman::list_versions('chromedriver'), I get: Error: C:\Users\eduardo\AppData\Local/binman/binman_chromedriver app directory not found. Am I missing something? Thanks
Oh, I figured out: We must first run a successful RSelenium::rsDriver in order to create the missing directories. Agter that the binman command works ok
Glad you found a solution Eduardo, thanks for sharing! :)
I looks like you are using windows. Will the same commands (i.e., last commands to close the connection to the server) work?
Hey Haraldur, I'm not sure about other systems. Have you already tried it, and did it cause any problems?
what are some applications of this? I suppose webscraping?
Hey, yes web scraping is one if its applications. You may also use it to automate certain tasks as a webmaster of a website.
Hi, do you know if the RSelenium package is still being maintained? It seems that it is not.
Hi Ruben. Are you having any issues using the package?
Thank you for your reply. Yes, using the binman command returns, at the latest, version 114 of Chrome. Chrome is up to version 120 now. I have downloaded the latest chrome webdriver and get an undefined error in httr call. Do you have a better way to contact you? It doesn't seem like the package is being maintained, honestly. Or, do I need to do something?
Hey, I'm not sure if RSelenium is still updated. I think it would be best to contact the developers of the package directly via GitHub: github.com/ropensci/RSelenium I hope this helps! Joachim
Hello! Thanks for the video. I am getting this error :
[1] "Connecting to remote server"
Error in checkError(res) :
Undefined error in httr call. httr output: Failed to connect to localhost port 14415: Connection refused
by using rD
Hello Richard,
Unfortunately, I am not much experienced in RSelenium. Have you checked this thread on StackOverflow: stackoverflow.com/questions/45395849/cant-execute-rsdriver-connection-refused. If it doesn't help, you can post your question on our Facebook discussion group: facebook.com/groups/statisticsglobe. This way, other users can read and help.
Regards,
Cansu
How to bypass Cloudflare detection?
Hey, please note that bypassing Cloudflare's detection mechanisms using tools like RSelenium to access or scrape web content can violate terms of service and ethical guidelines. It's important to respect website security measures and seek legal ways to access data, such as using official APIs or obtaining permission from the website owner.
This didn't work for me. Pretty early in the video, I got an error than I couldn't rectify with anything in this video, or doing Google search. Basically, the way of finding my Chrome version didn't work.
Hey Nathan, could you please share your code and the error message that was returned? Without seeing your code, it will be difficult for me to find the problem. Thanks, Joachim
Hello Nathan,
When do you exactly get the error? Could you also share the error with us?
Regards,
Cansu
@@cansustatisticsglobe Hi, I got Error: C:\Users\thom\AppData\Local/binman/binman_chrome_driver app directory not found. the package seems to load fine but it looks like there is a problem with it?
@@StatisticsGlobe Hi, it's perhaps a problem with the binman package. i get ...Error: C:\Users\thomp\AppData\Local/binman/binman_chrome_driver app directory not found.
Thanks,
@@nathanthompson8652 You might be right, but unfortunately, I couldn't find a solution either. I think you should post your question on our Facebook discussion group: facebook.com/groups/statisticsglobe, someone, who is more experienced in the Rselenium package, might help.
Regards,
Cansu
I got 4:20 into the video and have problem with the version. Current chrome version is 99.0.4844.84. I get an error setting rs_driver_object. (binman list is: "100.0.4896.20" "99.0.4844.35" "99.0.4844.51" ) Is there a "work around?"
Hey Robert, I also couldn't figure out why this is happening. I have forwarded your question to Samer. Regards, Joachim
Hi Robert, my apologies for seeing this very late. In this case, you will have to use the 99.0.4844.51 value from the binman list. You always want to use the version from binman list that is closest to your current chrome version. I hope this helps!
Same issue here but I have decided to put RSelenium to the side since I worried about the ethical side.