Create AWS EMR Cluster Using AWS CLI and Submit job

Configuring EMR
Creating Cluster using CLI
You can refer below GitHub link
github.com/sau...
In this Video , create and run an EMR cluster using AWS CLI.
I have covered below topics.
Create an IAM user
Configure and install AWS CLI
Create an EMR cluster using AWS CLI
Create Pyspark code or Job
Submit job using AWS CLI
Submit job using Primary node
Check output in s3 bucket
I have also created videos related to AWS EMR.
Create AWS EMR cluster using AWS management console and submit job
• AWS EMR Cluster Create...
AWS EMR theory
• AWS EMR Tutorial | Ama...
DIFF AWS GLUE VS EMR
• DIFF AWS GLUE VS EMR| ...
Command for creating cluster using AWS CLI
You can also refer doc for commands
github.com/sau...
aws emr create-cluster --name MyEMRCluster --use-default-roles --release-label emr-6.11.0 --instance-count 3 --instance-type m5.xlarge --applications Name=Spark Name=Hadoop --ec2-attributes SubnetIds=subnet-03xxxx64,KeyName=monu --log-uri s3://myemrproject/logs/
Execute job using awl cli step
aws emr add-steps --cluster-id j-GAVB3ZN07CUB --steps Type=Spark,Name="MySparkJob",ActionOnFailure=CONTINUE,Args=[--deploy-mode,cluster,--master,yarn,--conf,spark.yarn.submit.waitAppCompletion=true,s3://myemrproject/scripts/mypysparkscript_1.py]

[@electricalsir]

thanks

[@sakshiwakode]

🙏🙏👌tysm sir

[@kashifsadiq4145]

Make a playlist about dataops in hindi thanks

[@waseemswork]

Hi Saurabh,
Thanks for the this tutorial and other tutorials which are very helpful.
If time permit, could you please explain, how can we automate creating EMR cluster creation for spark jobs through Cloud Formation Templates ?

[@isharkpraveen]

unable to install pip install awscli

[@rahulpamnani6574]

An error occurred (ValidationException) when calling the RunJobFlow operation: Invalid InstanceProfile: EMR_EC2_DefaultRole.

[@TechnoDevs]

Thanks for watching video. it's looked your EMR_EC2_DefaultRole" is invalid.
I think you tried to modified this policies.
please review the changes that you made.
or what you can do delete existing one and try to recreate it.

[@ajinkyadeshmukh2343]

i'm also struggling for this issue
in aws cli use this coomand( aws emr create-default-roles) and then again create emr cluster it can run