Techno Devs with Saurabh
Techno Devs with Saurabh
  • 65
  • 188 336
Running EMR jobs with Airflow| Create EMR cluster and Submit a job on EMR using AWS MWAA (Part3)
create emr cluster and submit pyspark job with Amazon Managed Workflows for Apache Airflow.
GitHub link for code
github.com/saurabhgarg013/Airflowcode.git
มุมมอง: 1 086

วีดีโอ

AWS Airflow Demo| Python code |AWS MWAA SETUP (part2)
มุมมอง 1.6K4 หลายเดือนก่อน
In this video, we have covered below topics. Create Amazon Managed Workflows for Apache Airflow. Create DAG python code Run python code by scheduling job using Airflow understand airflow environment GitHub link for code github.com/saurabhgarg013/Airflowcode.git
Apache Airflow Tutorial in Hindi | AWS Airflow (part1)
มุมมอง 4.3K4 หลายเดือนก่อน
Amazon Managed Workflows for Apache Airflow (MWAA). In this video , you will learn below topics. I will create another video for demo. About Apache airflow and AWS managed airflow . Airflow use DAG Operators dag and operator example AIRFLOW COMPONENT Amazon MWAA environment Amazon MWAA execution role EMR job by Airflow Airflow cost Below video link for demo lab for airflow th-cam.com/video/6apd...
AWS RDS PostgreSQL Database Tutorial with Demo
มุมมอง 5525 หลายเดือนก่อน
In this video , I have covered below topics. About PostgreSQL and its Features PostgreSQL vs MySQL AWS RDS Aurora PostgreSQL vs AWS RDS PostgreSQL Create AWS PostgreSQL database Connect with PostgreSQL using cmd and pgAdmin Create Database, User , Schema and Tables Used Datatype like json, uuid and array. insert and create table with json and uuid data type Execute create table and insert scrip...
Create EMR cluster and add steps | Submit job
มุมมอง 1.5K5 หลายเดือนก่อน
In this video covered below topics: 1.How to submit EMR job in cluster mode. 2.Add a Spark step - Amazon EMR 3.Run PySpark code in EMR master node terminal Learn how to adding steps to a cluster with the help of master node and cli
Amazon EMR Studio | Launch Jupyter notebooks
มุมมอง 8435 หลายเดือนก่อน
Amazon EMR Studio is a web-based integrated development environment (IDE) for fully managed Jupyter notebooks that run on Amazon EMR clusters. In this video I have cover below topics 1.create EMR cluster 2.Create EMR Studio 3.Create a workspace 4.Run Pyspark code Use Pyspark with a Jupyter Notebook in an AWS EMR. Amazon EMR Studio - Creating a new Studio Workspace
AWS Glue Catalog With AWS EMR | Spark SQL in EMR
มุมมอง 1.6K7 หลายเดือนก่อน
Use the AWS Glue Data Catalog as the metastore for Spark SQL Amazon EMR access to AWS Glue Data Catalog
Launch Jupyter Notebook With AWS EMR| Run Pyspark Code
มุมมอง 1.7K7 หลายเดือนก่อน
In this video, You get to know how to setup Jupyter Notebook and run Pyspark code Jupyter Notebook (formerly IPython) is one of the most popular user interfaces for running Python Use JupyterHub on Amazon EMR to host multiple instances of a single-user Jupyter notebook server for multiple users.
AWS Glue Studio Join and Custom transformation | Joining Datasets in AWS Glue ETL Job
มุมมอง 1.8K11 หลายเดือนก่อน
In this video , I explain how o join two datasets and also apply custom transformation in AWS Glue. AWS Glue Studio which joins two datasets, transforms the joined dataset and than apply custom transformation and then put down data into destination bucket. AWS Glue Studio is a graphical interface that makes it easy to create, run, and monitor extract, transform, and load (ETL) jobs in AWS Glue....
AWS Kinesis theory with demo| ETL Hands on demo using lambda and Kinesis
มุมมอง 3.5Kปีที่แล้ว
AWS - Kinesis Data Stream and data firehose practical lab with theory. Topics that we will cover in this video are as follows: what is Amazon Kinesis Data Streams? what is Amazon kinesis Data firehose? diff between Data Streams vs Data firehose Shards and partition key concept Demo ETL using Aws lambda with Kinesis data stream Demo ETL using Kinesis data stream and data firehose · Request you t...
AWS Code Pipeline with Elastic Beanstalk |CICD AWS Code pipeline
มุมมอง 630ปีที่แล้ว
we commit the code to an Git repository. Using Git you can then automatically push changes using AWS code pipeline. we deploy python web app into Elastic Beanstalk
AWS Elastic Beanstalk Tutorial with Demo
มุมมอง 4.2Kปีที่แล้ว
Elastic Beanstalk is a service for deploying and scaling web applications and services. it is platform as a service (PaaS) takes your application code and deploys it while provisioning the supporting architecture and compute resources ec2 required for your code to run.it also manages the patching and security updates. Below topics are explained in this AWS Elastic Beanstalk tutorial: 1.Why AWS ...
AWS CodePipeline |Deploy Glue ETL script using CI/CD Pipeline on AWS
มุมมอง 4.4Kปีที่แล้ว
CodePipeline automates the steps required to release your software changes continuously. A pipeline is a workflow construct that describes how software changes go through a release process. Each pipeline is made up of a series of stages. CloudFormation is a service that helps you model and set up your AWS resources so that you can spend less time managing those resources and more time focusing ...
Create Redshift Cluster and Load Data using Python| Connect redshift using Python
มุมมอง 2.8Kปีที่แล้ว
AWS Redshift connects with Python and Create Cluster using Python jupyter notebook. Load data from s3 into redshift using python This video really important for who want to connect with redshift using python boto3 and applied various operation like create cluster, create redshift table and upload data from s3 bucket. you should have only AWS access key and secrets key and VPC ID and IAM ROLE. r...
Building ETL Pipeline using AWS Glue and Step Functions
มุมมอง 4.7Kปีที่แล้ว
Manage AWS Glue Jobs with Step Functions Create an ETL solution using AWS Step Functions also send notification when job succeeded by event bridge. In this video , we create workflow using AWS Glue, AWS Step Function and AWS Event Bridge and SNS service. we follows below steps. 1.create glue crawler and Glue ETL job 2.Create step function to execute Crawler and glue ETL job 3.Create SNS topic w...
Create Amazon Elastic Container Service (ECS) with a Load Balancer Demo | Step by Step
มุมมอง 1.4Kปีที่แล้ว
Create Amazon Elastic Container Service (ECS) with a Load Balancer Demo | Step by Step
AWS ECR| Create Docker Image and AWS push docker image to ECR|AWS ECS
มุมมอง 1.3Kปีที่แล้ว
AWS ECR| Create Docker Image and AWS push docker image to ECR|AWS ECS
AWS ECS Fargate Demo for Beginner | Deploying Containers on ECS Fargate
มุมมอง 3.2Kปีที่แล้ว
AWS ECS Fargate Demo for Beginner | Deploying Containers on ECS Fargate
Complete Docker Tutorial with Demo |Docker installation| Deploy image
มุมมอง 1.2Kปีที่แล้ว
Complete Docker Tutorial with Demo |Docker installation| Deploy image
Create AWS EMR Cluster Using AWS CLI and Submit job
มุมมอง 1.5Kปีที่แล้ว
Create AWS EMR Cluster Using AWS CLI and Submit job
DIFF AWS GLUE VS EMR| AWS EMR VS AWS Glue
มุมมอง 1.4Kปีที่แล้ว
DIFF AWS GLUE VS EMR| AWS EMR VS AWS Glue
AWS EMR Tutorial | Amazon EMR Architecture
มุมมอง 4.9Kปีที่แล้ว
AWS EMR Tutorial | Amazon EMR Architecture
AWS EMR Cluster Create using AWS Console | Submitting Spark Jobs in AWS EMR Cluster
มุมมอง 14Kปีที่แล้ว
AWS EMR Cluster Create using AWS Console | Submitting Spark Jobs in AWS EMR Cluster
AWS Ec2 VS Amazon Lambda|Whats the difference in 5 minute
มุมมอง 667ปีที่แล้ว
AWS Ec2 VS Amazon Lambda|Whats the difference in 5 minute
What is AWS Lambda and AWS Glue? |Difference between AWS Glue and AWS lambda explained in 5 mins
มุมมอง 2.1Kปีที่แล้ว
What is AWS Lambda and AWS Glue? |Difference between AWS Glue and AWS lambda explained in 5 mins
AWS Glue ETL with Python shell |Read data from S3 and insert Redshift
มุมมอง 5Kปีที่แล้ว
AWS Glue ETL with Python shell |Read data from S3 and insert Redshift
AWS GLUE Complete ETL Project Demo| Load Data from AWS S3 to Amazon RedShift(Data engineer Project)
มุมมอง 36Kปีที่แล้ว
AWS GLUE Complete ETL Project Demo| Load Data from AWS S3 to Amazon RedShift(Data engineer Project)
Redshift using Python| Load and insert and copy data into redshift using psycopg2
มุมมอง 4.9Kปีที่แล้ว
Redshift using Python| Load and insert and copy data into redshift using psycopg2
How to Create Redshift Cluster, Load Data, Unload Data| Redshift spectrum
มุมมอง 11Kปีที่แล้ว
How to Create Redshift Cluster, Load Data, Unload Data| Redshift spectrum
How to Create AWS Account Free | Create a Free AWS Account
มุมมอง 140ปีที่แล้ว
How to Create AWS Account Free | Create a Free AWS Account

ความคิดเห็น

  • @Sunnykhatnani
    @Sunnykhatnani 22 ชั่วโมงที่ผ่านมา

    Why are we crawling redshift when it is empty?

  • @nadianizam6101
    @nadianizam6101 วันที่ผ่านมา

    Excellent explanation

  • @paurashdewangan7349
    @paurashdewangan7349 2 วันที่ผ่านมา

    🔥🔥🔥thank you so much. can you please provide the ppt🙂

  • @ssalgia
    @ssalgia 2 วันที่ผ่านมา

    lambda is a big pain

  • @prabhatgupta6415
    @prabhatgupta6415 3 วันที่ผ่านมา

    can we practice glue in free tier or it will be chargable?

  • @ssrakeshsharma
    @ssrakeshsharma 7 วันที่ผ่านมา

    Bro great Job. BC just 131 likes on this video, Yaha kisi ne apni ek baar G##D hila di hoti to 1 million like aa jate. In Bharat there is no value of science, hardwork and technical work.

  • @deepakkaushik4192
    @deepakkaushik4192 9 วันที่ผ่านมา

    Do we need to create 10 code pipeline for 10 glue jobs..can we use same pipeline for 10 glue jobs deployed on different times

  • @deepakkaushik4192
    @deepakkaushik4192 9 วันที่ผ่านมา

    Great work

  • @hafizadeelarif3415
    @hafizadeelarif3415 15 วันที่ผ่านมา

    In AWS Redshift cluster, what is zero ETL and how does it work, sir?

  • @hafizadeelarif3415
    @hafizadeelarif3415 15 วันที่ผ่านมา

    In AWS Redshift cluster, what is zero ETL and how does it work, sir?

  • @thelifehackerpro9943
    @thelifehackerpro9943 15 วันที่ผ่านมา

    Why need two crawler, can't we directly read parquet and load to Redshift

  • @420nyk
    @420nyk 16 วันที่ผ่านมา

    Bro love you videos.. especially long form videos where you explain each and every option or a feature of a service

  • @apurbarana8731
    @apurbarana8731 19 วันที่ผ่านมา

    Hello sir, video is too good. I have cleared lots of things from this video. 🥳🎉👌👌

  • @vandung6025
    @vandung6025 22 วันที่ผ่านมา

    Thank you so much

  • @thapasujan07
    @thapasujan07 23 วันที่ผ่านมา

    Nicely done Sir. Thank you.

  • @thapasujan07
    @thapasujan07 23 วันที่ผ่านมา

    Nice one Sir. Thank you.

  • @hafizadeelarif3415
    @hafizadeelarif3415 24 วันที่ผ่านมา

    Hi sir, could you please share the slides that you used in this video?

  • @venkataramanamurthypasumar4542
    @venkataramanamurthypasumar4542 25 วันที่ผ่านมา

    Very Good

  • @DiptiprakashChhanda
    @DiptiprakashChhanda 25 วันที่ผ่านมา

    This is the best video which cleared my doubts and i completely understood the concepts.

  • @nadianizam6101
    @nadianizam6101 27 วันที่ผ่านมา

    excellent

  • @venkataramanamurthypasumar4542
    @venkataramanamurthypasumar4542 28 วันที่ผ่านมา

    Very good explanation

  • @vinodvashistha4260
    @vinodvashistha4260 หลายเดือนก่อน

    Great work, Saurabh!

  • @krishnakantsharma8021
    @krishnakantsharma8021 หลายเดือนก่อน

    Sir bohut badiya smjhaate ho par pls gala thoda saaf krke videos banaye kabhi kabhi irritating ho jata hai ye

  • @devprasad609
    @devprasad609 หลายเดือนก่อน

    Well streamlined video and content.!

  • @AnkitaKulshreshtha
    @AnkitaKulshreshtha หลายเดือนก่อน

    Very informative 🎉🎉 Thank you

  • @user-uo8jg2qm8q
    @user-uo8jg2qm8q หลายเดือนก่อน

    Thankyou so much....with your help I got role in jpmc... however I am scared as I don't belongs to coding background ...how I will survive as data engineer

  • @mashalkhan7699
    @mashalkhan7699 หลายเดือนก่อน

    Sir you are great and No. 1

  • @hitesh7719
    @hitesh7719 หลายเดือนก่อน

    Excellent video . Can not thank you enough for giving me the confidence of mastering redshift foundation from both your redshift theory and this practical tutorial.

  • @ravithejaburugu8926
    @ravithejaburugu8926 หลายเดือนก่อน

    Hi Saurabh. If I have to create output data into different partition folders based on year/month/day/ should I have to create all the folders for each day manually??

  • @electricalsir
    @electricalsir หลายเดือนก่อน

    thanks

  • @prathapn01
    @prathapn01 หลายเดือนก่อน

    Is it possible to include SSH -i command using ec2 keypair -- to connect to the EMR cluster using bootstrap script ?

  • @tejaspise4638
    @tejaspise4638 หลายเดือนก่อน

    Really great work

  • @SonuKumar-fn1gn
    @SonuKumar-fn1gn หลายเดือนก่อน

    Please create a video helm airflow and helm spark chart in kubernetes and submit a job in pyspark. Thank you sir ji so much for the video ❤❤❤

  • @SonuKumar-fn1gn
    @SonuKumar-fn1gn หลายเดือนก่อน

    Please create a video helm airflow and helm spark chart in kubernetes and submit a job in pyspark. Thank you sir ji so much for the video ❤❤❤

  • @SonuKumar-fn1gn
    @SonuKumar-fn1gn หลายเดือนก่อน

    Very nice video ❤ thank u so so much 😊

  • @basavapn6487
    @basavapn6487 หลายเดือนก่อน

    One more thing to confirm about depends_on_past is not correctlty explained,For example if we have two tasks task1 and task2 and suppose if dag starts to run and when task1 completes and if task2 fails ,and When next day dag runs and task1 runs and completes but it will not start task2 because the previous day task2 is in failed state ,so depends_on_past works in this way i,e it depends on the same task for the previous instances to see if it got completed.

  • @basavapn6487
    @basavapn6487 หลายเดือนก่อน

    Why simple_3.py file ran for 19times,I couldn't get why it ran for so many time ,Did you get to know whats the reson behind the run?

  • @basavapn6487
    @basavapn6487 หลายเดือนก่อน

    Very well structured,Thanks for the detailed explanation

  • @abhishekmcajec
    @abhishekmcajec หลายเดือนก่อน

    Very good explanation

  • @sampathpatil2439
    @sampathpatil2439 2 หลายเดือนก่อน

    Bro while explaining just be relaxed, sometimes you only got stuck while explaining. in some places it really got so much confused. while explaining the lambda code you need to be little slow bcoz if you explain in this way a fresher will not be able to understand, learn to break the code while explaining. Be relaxed and focus on what you are explaining.

  • @faisalshaikh4271
    @faisalshaikh4271 2 หลายเดือนก่อน

    Very detailed explanation. Thank you...

  • @rohitanand7901
    @rohitanand7901 2 หลายเดือนก่อน

    Very well explained, great for interviews preparation

  • @ideaJoBadalDeDuniya
    @ideaJoBadalDeDuniya 2 หลายเดือนก่อน

    Very nice explanation Saurabh

  • @FelixTrk
    @FelixTrk 2 หลายเดือนก่อน

    ELB in Tamil th-cam.com/video/rhEZG48Ih3M/w-d-xo.htmlsi=mUdsb2IjWoXqU1jX

  • @deepanshuaggarwal7042
    @deepanshuaggarwal7042 2 หลายเดือนก่อน

    Does AWS Athena takes extra storage to show data into table? If yes, how does it cost us ?

    • @TechnoDevs
      @TechnoDevs 2 หลายเดือนก่อน

      AWS Athena does not require additional storage to show data in a table because it queries data directly from Amazon S3. and Athena itself doesn't store data, the data you query must be stored in Amazon S3. You will incur standard S3 storage costs for the data stored there.

  • @harshitkumar5147
    @harshitkumar5147 2 หลายเดือนก่อน

    When I create Interactive workloads with IAM non-root user I get the error: This studio is not enabled for interactive workspaces. Edit studio settings to configure network settings, studio service role and workspace storage S3 bucket to enable Create workspace. Refresh this page after editing the studio. To learn more, see Enabling Interactive Workloads .

  • @narendrakashyap7730
    @narendrakashyap7730 2 หลายเดือนก่อน

    Salute you for your efforts and courage. Thank you Saurabh.

  • @narendrakashyap7730
    @narendrakashyap7730 2 หลายเดือนก่อน

    Your videos make sense. Thank you Saurabh

  • @narendrakashyap7730
    @narendrakashyap7730 2 หลายเดือนก่อน

    Thank you so much…❤

  • @ankushchavhan_
    @ankushchavhan_ 2 หลายเดือนก่อน

    Sir , Can I get the ppt file ?