Hello I've followed your video carefully and am pretty sure I didn't miss any detail or do anything wrong. However, I got the error: "errorMessage": "Unable to import module 'main': No module named 'selenium'", "errorType": "Runtime.ImportModuleError", "stackTrace": [] I really hope you can help me with this, I've turn the internet upside down, but so far nothing works
1. Did you add the selenium layer? 2. Did you connect the layer to the lambda function? 3. Did you install specifically selenium 3.8.0 and with python 3.7 version with the correct path command which I have in the description? Check those 3, as I faced the same issue before recording the video, and the same checklist applied to me as well.
Hi Michael, Thank you so much for such a good video. Recently AWS lambda announced that lambda will not be maintaining Python3.7 anymore . Need to move my code to python 3.11. Not sure how to do this. Can you please make a video to use selenium chrome driver with python 3.11 in lambda? Or can you suggest me a way to solve this?
Hi thanks for the video, this was something that frustrated me in may and dropping it all together. There are plenty of sites with the information, BUT the most important element: this ONLY works with Python 3.7 and Selenium 3.8. Following the security mindset, everything back in may was trying to use the latest version =)
I used the same versions but instead I download chromedriver and headless-chromium from s3 to my tmp files (because of the limit size of layers). I got this error: Message: 'chromedriver' executable may have wrong permissions. Do you know how to fix this?
How did you manage to make headless-chromium turn from 102,6 MB uncompressed to 5,1 MB uncompressed at the 9:50 mark? It's like the number just changes without any explanation. How are you able to upload headless-chromium and chromedriver without exceeding the Layer size limit of 50MB? The exact same file uncompressed on my system is 100MB as well. I'd prefer not to have to use S3
Are you using Mac? That’s how much it is compressed using the Mac’s compression (Under 50mb). Using another software would for sure give different results.
Hello bro, Thanks for ur great work , but I got a question AWS Lambda end of support for Python 3.7 so it's not supported anymore at least for layers, do u have another way to get around with this , I would really appreciate this alot! Thanks in advance !
Hi Michael, in a few comments you talk about an updated video that doesn't rely on a deprecated version of Python. If this video exists, can you please provide a link here
Hi, I have this error: "errorMessage": "2024-03-12T04:26:45.989Z 394e1df8-6683-4e3b-aa9c-da1864b9bba9 Task timed out after 20.01 seconds". I followed all steps but this error appears.
Hello, thanks for sharing. I tested it and it worked perfectly as in the video, but I have a question. I couldn't get it to work with newer versions of the libraries. Why does this happen? Today I have several codes implemented with the latest versions and it would be a lot of work to adapt the old versions. Tks
Hello, thanks for the video, but but when i am adding the layer of chormdrive there, i got a error 'Layers consume more than the available size of 262144000 bytes'
@@alexkeila I have managed to solve that by using the exact version of selenium , however , the one run on lambda does not have the same result as I run on my local machine
aws lambda gonna end support for Python 3.7. Also it seems like the current setup we got cannot be migrated to python 3.11. I tried to migrate but chromedriver couldnt start. How should we go from here, Michael?
i've been working with python 3.9 and i am struggle with this error : "Message: Service /opt/chromedriver unexpectedly exited. Status code was: 127" Does anyone have any idea how to fix it? I have followed step by step very carefully
same issue with me as well. Strugling with this from past few days and tried almost all the options available on internet. But still can't figure out. Did you got the solution??
using the commands provided, it downloads the chromedriver and the headless chromium but it doesnt open, it always says that the file format is not supported or is broken
@@MichaelKitas How did you get past the error - Error setting up WebDriver: Message: disconnected: unable to connect to renderer? I am using a Docker image.
Very informative. Do you know if there is a way to configure your AWS lambda account, to tell Amazon a max dollar amount you want to spend, and then if you accidentally exceed that amount with your resources, they either immediately notify you, or just simply shut down your site? I'd prefer it works that way. I'd hate to get a big surprise bill from amazon, because I didn't know my code was very inefficient, and the code was spending too much time during a function invocation.
I get the message below when putting it to test: { "errorMessage": "'module' object is not callable", "errorType": "TypeError", "stackTrace": [ " File \"/var/runtime/bootstrap.py\", line 127, in handle_event_request response = request_handler(event, lambda_context) " ] } can you help me?
@@MichaelKitas Yes, and only that version works for me. But how do I know when AWS supports newer versions and which combination of python, selenium and chromedriver? I don't want to operate with 3.7 forever.
Hello. Thank you for so necessary video. How could you upload the file more than 10Mb in the layer? (chromedriver.zip 44.6Mb) System should had to make you to upload through S3. Is not it?
Hi, I keep getting the error "no chrome binary at /var/taskheadless-chromium ". I added the selenium layer, connected it to the lambda function and installed the specific versions you had. Nothing is working to fix it.
It works for me. does anyone knows if there is a way to do it in python3.9 or python3.8? i did not found a solution without the use of a 5xxmb approx docker image
@@CristóbalAbarca-j4b So if i want to do it now, Can't i do it with python 3.7 ? Please elaborate more . And if I have to use Python3.9/3.10 then What versions of chromedriver and headless chromium should i use?
@@MichaelKitas for windows, do we also need to download the headless chromium like you did on the video or just a chrome driver downloaded from the site you mentioned above is okay?if yes, where do i download the headless chromium?
I have done exactly the same as shown in the video . But when i run I get this response .-> { "errorMessage": "Syntax error in module 'main': invalid syntax (selenium_manager.py, line 46)", "errorType": "Runtime.UserCodeSyntaxError", "stackTrace": [ " File \"/opt/python/lib/python3.7/site-packages/selenium/webdriver/common/selenium_manager.py\" Line 46 if (path := os.getenv(\"SE_MANAGER_PATH\")) is not None: " ] }
![Kubernetes Tutorial for Beginners [FULL COURSE in 4 Hours]](http://i.ytimg.com/vi/X48VuDVv0do/mqdefault.jpg)
Hello
I've followed your video carefully and am pretty sure I didn't miss any detail or do anything wrong. However, I got the error:
"errorMessage": "Unable to import module 'main': No module named 'selenium'",
"errorType": "Runtime.ImportModuleError",
"stackTrace": []
I really hope you can help me with this, I've turn the internet upside down, but so far nothing works
1. Did you add the selenium layer?
2. Did you connect the layer to the lambda function?
3. Did you install specifically selenium 3.8.0 and with python 3.7 version with the correct path command which I have in the description?
Check those 3, as I faced the same issue before recording the video, and the same checklist applied to me as well.
@@MichaelKitas Nevermind, thank you so much, now it's working
@@minhthieu2466 well what was the issue
@@miclo_ssx he didn't add the selenium layer
@@minhthieu2466
Can't thank you enough for this, I am gonna write a blog and add this video there.
Thank you a lot for this 🙂
Really awesome tutorial, even I was able to get this up and running!
what an incredible tutorial, priceless information very hard to find over the internet.
SUPERB !! i just need to pay attention to the zip path structure, thanks a lot!
Glad it was helpful!
Hi Michael, can you make another video on selenium using container implementation to build lambda? It would be a life saver! Thanks in advance
Great suggestion, will make sure to add it to the list
@@MichaelKitas Yes please!!!! Thanks
Just uploaded a new video which supports any python version: th-cam.com/video/j_qOVqQDBWw/w-d-xo.html
Hi Michael,
Thank you so much for such a good video. Recently AWS lambda announced that lambda will not be maintaining Python3.7 anymore . Need to move my code to python 3.11. Not sure how to do this. Can you please make a video to use selenium chrome driver with python 3.11 in lambda? Or can you suggest me a way to solve this?
Please reply Michael
Many requested this, I've added it to the to-do list, will try to make a video asap
when will be ready this video?@@MichaelKitas
please@@MichaelKitas
Just uploaded a new video which supports any python version: th-cam.com/video/j_qOVqQDBWw/w-d-xo.html
Hi thanks for the video, this was something that frustrated me in may and dropping it all together. There are plenty of sites with the information, BUT the most important element: this ONLY works with Python 3.7 and Selenium 3.8.
Following the security mindset, everything back in may was trying to use the latest version =)
Agreed with this. Wish it is compatible with all the python and selenium version.
I used the same versions but instead I download chromedriver and headless-chromium from s3 to my tmp files (because of the limit size of layers). I got this error: Message: 'chromedriver' executable may have wrong permissions. Do you know how to fix this?
Check this out: stackoverflow.com/questions/49787327/selenium-on-mac-message-chromedriver-executable-may-have-wrong-permissions
hey. im facing the exact same issue. were u able to fix it?
How did you manage to make headless-chromium turn from 102,6 MB uncompressed to 5,1 MB uncompressed at the 9:50 mark? It's like the number just changes without any explanation. How are you able to upload headless-chromium and chromedriver without exceeding the Layer size limit of 50MB? The exact same file uncompressed on my system is 100MB as well. I'd prefer not to have to use S3
Are you using Mac? That’s how much it is compressed using the Mac’s compression (Under 50mb). Using another software would for sure give different results.
Hello bro, Thanks for ur great work , but I got a question
AWS Lambda end of support for Python 3.7
so it's not supported anymore at least for layers, do u have another way to get around with this , I would really appreciate this alot!
Thanks in advance !
Yes, making an updated video this week. Hint, check: AWS SAM CLI
Hi, Which selenium version is compatible with Python 3.9 lambda version??
thank you so much.
It was my big problem. you saved my time.!
Hi Michael, in a few comments you talk about an updated video that doesn't rely on a deprecated version of Python. If this video exists, can you please provide a link here
th-cam.com/video/j_qOVqQDBWw/w-d-xo.html
Thank you for your work. I want to run Selenium4.8. and Request I have a urllib3 library conflict. what could be the problem?
Can you send screenshot (send url of screenshot) of the error?
Hi, I have this error: "errorMessage": "2024-03-12T04:26:45.989Z 394e1df8-6683-4e3b-aa9c-da1864b9bba9 Task timed out after 20.01 seconds". I followed all steps but this error appears.
Maybe increase timeout? Probably doesnt have enough time to complete the script
Hello, thanks for sharing. I tested it and it worked perfectly as in the video, but I have a question. I couldn't get it to work with newer versions of the libraries. Why does this happen? Today I have several codes implemented with the latest versions and it would be a lot of work to adapt the old versions. Tks
Many requested this, I've added it to the to-do list, will try to make a video asap
Thanks for sharing this, incredibly helpful!
This is amazing, thank you!
Hello, thanks for the video, but but when i am adding the layer of chormdrive there, i got a error 'Layers consume more than the available size of 262144000 bytes'
I have the same issue!! The chromedriver ZIP is a larger file than the ones from the other layers.
@@alexkeila I have managed to solve that by using the exact version of selenium , however , the one run on lambda does not have the same result as I run on my local machine
You need to use the exact version of selenium as @user-zy7qo3ch1s mentioned.
aws lambda gonna end support for Python 3.7. Also it seems like the current setup we got cannot be migrated to python 3.11. I tried to migrate but chromedriver couldnt start. How should we go from here, Michael?
@MichaelKitas
Let me know if you find a way to work the code in python 3.11
Many requested this, I've added it to the to-do list, will try to make a video asap
@@MichaelKitas Still waiting, I guess docker container is the only option
Just uploaded a new video which supports any python version: th-cam.com/video/j_qOVqQDBWw/w-d-xo.html
Hi I want to use undetected-chromedriver in my AWS lambda, which version is compliant with selenium 3.8.0?
I think latest version works: pypi.org/project/undetected-chromedriver/3.4.7/
It says Python3.6 and newer, so latest works
Did you get it to run with Undetected-chromedriver? I'm also trying to run in Lambda but I can't make it work at all
@@pedrodellazzari528 unfortunately no.
Hello Michael ,
How to Making Captcha Breaking Script Work in AWS Lambda
Same way you do it locally, I have made a video for bypassing captcha, check it out here: th-cam.com/video/5puz9Mb2d_c/w-d-xo.html
i've been working with python 3.9 and i am struggle with this error : "Message: Service /opt/chromedriver unexpectedly exited. Status code was: 127"
Does anyone have any idea how to fix it? I have followed step by step very carefully
same issue with me as well. Strugling with this from past few days and tried almost all the options available on internet. But still can't figure out. Did you got the solution??
AWS gives us the option to set up specific runtime for each lambda function. Use the same version as I said, and there shouldn't be an issue.
@@khushindarshan156 It is an version issue, it will only run on 3.7, any higher version will give 127 error
@@VIKASSAINI007 there's a way to make it work with 3.8 or later?, aws ends support for 3.7, so it can't be created anyways
Thank you very much! It works!
You're welcome!
using the commands provided, it downloads the chromedriver and the headless chromium but it doesnt open, it always says that the file format is not supported or is broken
Going to make an updated video today, stay tuned.
@@MichaelKitas Hey Michael, when can we expect the updated video?
Thanks so much, unfortunately python 3.7 isn't supported any longer. An updated video would be awesome!
Very soon, working on it, found a solution 👍
@@MichaelKitas How did you get past the error - Error setting up WebDriver: Message: disconnected: unable to connect to renderer? I am using a Docker image.
Just uploaded a new video which supports any python version: th-cam.com/video/j_qOVqQDBWw/w-d-xo.html
@@MichaelKitas damn, thanks for following it up!
Thank you very much!
Hello Michael and everyone, does anyone have where I can find compatible versions for headless chrome and selenium that supports Python3.8?
Just uploaded a new video which supports any python version: th-cam.com/video/j_qOVqQDBWw/w-d-xo.html
Very informative.
Do you know if there is a way to configure your AWS lambda account, to tell Amazon a max dollar amount you want to spend, and then if you accidentally exceed that amount with your resources, they either immediately notify you, or just simply shut down your site? I'd prefer it works that way.
I'd hate to get a big surprise bill from amazon, because I didn't know my code was very inefficient, and the code was spending too much time during a function invocation.
Yes, check how to do that here: aws.amazon.com/getting-started/hands-on/control-your-costs-free-tier-budgets/
@@MichaelKitas Thank you
Hey Mike! This can work with 'playwright'?
Not sure, will add a series for playwright in the future, and try it
I get the message below when putting it to test:
{
"errorMessage": "'module' object is not callable",
"errorType": "TypeError",
"stackTrace": [
" File \"/var/runtime/bootstrap.py\", line 127, in handle_event_request
response = request_handler(event, lambda_context)
"
]
}
can you help me?
Can you send full code?
Ok, so how can I know which version of Python, Selenium and Chromedriver works with Lambda?
I showed on the video the versions I used
@@MichaelKitas Yes, and only that version works for me. But how do I know when AWS supports newer versions and which combination of python, selenium and chromedriver? I don't want to operate with 3.7 forever.
@@ghoulflier1016 Not sure, I don't think there is a specific combination each time
@@MichaelKitas Thank you anyways for your great videos. They are such a big help for me to get started with Selenium!
Do you know about this error "errorMessage": "Message: Service /opt/chromedriver unexpectedly exited. Status code was: 127
",?
Python version must be 3.7 and selenium 3.8.0
Yes, first of all did you add chromedriver as a layer?
I was also getting the same error when I was using latest selenium and chromedriver version.
Hello. Thank you for so necessary video. How could you upload the file more than 10Mb in the layer? (chromedriver.zip 44.6Mb) System should had to make you to upload through S3. Is not it?
Yes, you can use S3, max is 50mb if not using s3
Hi, I keep getting the error "no chrome binary at /var/taskheadless-chromium
". I added the selenium layer, connected it to the lambda function and installed the specific versions you had. Nothing is working to fix it.
Did you add correct executable path? It’s searching in the wrong place
It works for me.
does anyone knows if there is a way to do it in python3.9 or python3.8? i did not found a solution without the use of a 5xxmb approx docker image
My concern is because python3.7 is "deprecated" since june 2023, and if aws lambda does not support it anymore, we are done...
Looking for a solution and will make a video for newer version
@@CristóbalAbarca-j4b So if i want to do it now, Can't i do it with python 3.7 ? Please elaborate more . And if I have to use Python3.9/3.10 then What versions of chromedriver and headless chromium should i use?
Just uploaded a new video which supports any python version: th-cam.com/video/j_qOVqQDBWw/w-d-xo.html
Does downloading the chromedriver on windows follows the same command?
You can download chromedriver from the official website: chromedriver.chromium.org/downloads
@@MichaelKitas for windows, do we also need to download the headless chromium like you did on the video or just a chrome driver downloaded from the site you mentioned above is okay?if yes, where do i download the headless chromium?
Is it possible to teach how to create a website or blog traffic bot that uses VPS and sends requests from Virtual Private server 🙏🙏
Very specific, but I might do that in the future, stay tuned!
can we run in aws glue
Message: Service /opt/chromedriver unexpectedly exited. Status code was: 127
:(
I had the same issue, and did not find anything that could solve it. Apparently a new tutorial is on the way!
@@zozoland9057 Hey bro, did you find the new tutorial video?
I have done exactly the same as shown in the video . But when i run I get this response .-> {
"errorMessage": "Syntax error in module 'main': invalid syntax (selenium_manager.py, line 46)",
"errorType": "Runtime.UserCodeSyntaxError",
"stackTrace": [
" File \"/opt/python/lib/python3.7/site-packages/selenium/webdriver/common/selenium_manager.py\" Line 46
if (path := os.getenv(\"SE_MANAGER_PATH\")) is not None:
"
]
}
Going to make an updated video today, stay tuned.