New setup? 👀 I am working on next GCP project, I will upload it this month and launch SQL for Data Engineering course too :D Don't forget to like this video :)
TL;DR 0:50 AWS S3 (Simple Storage Service ) To store any type of data 1:51 AWS Glue used for transformation scripts 2:29 AWS EMR compute over a distribued cluster 2:55 AWS Redshift data warehouse where extracted datas are loaded for further process 3:30 AWS Athena to directly querry data from S3 without storing it in warehouse 4:39 AWS Lambda compute service 5:26 AWS Kinesis to process data in real time 6:06 AWS DMS (Data Migration Service)
After completing your python course this particular video made lot of sense, still practicing and learning these cloud services thanks man! Btw setup seems nice🔥👏🏻
Great video Darshil, comprehensive explaination and good scope of DE processes. After I've completed the 2 main courses I'm currently studying, I'll switch focus to some AWS projects.
Hi Darshil your videos on Data Engineer are very interesting and to the point. I would like to make a request can you please make videos on Azure. For example showing, completea pipeline for any OLTP or OLAP project? Thanking you in advance
I really loved your classes but I have one concern if you make any projects related to streaming data using technologies like Hadoop, Kafka, spark, hive, pig, and Flik related to hospital data it would be very helpful for me as a learner of your videos
Hey Darshil, I was waiting for this video to understand the integration of cloud services with ETL process and fundamentals of these services. Thanks for the explanation !!
Need your advice brother Between data analyst vs data engineering vs data science 1)Which role requires less coding ? I am a civil engineer who can do decent coding but don’t really enjoyed it like coding geeks do. So need your advice. People suggested me for data analysis profile as it has less doing and needs knowledge of data visualisation but would like to know your inputs 2)which role has more jobs in India ? Data scientist has jobs abroad but in India it’s still in less no and limited to folks from top colleges or brilliant coders, data engineering has decent jobs, no clue about data analyst job scenario Please help
start with data analysis>after gaining some experience move to DE> then you will get a clear understanding of how an end2end business problem is solved. By that time you will know should I go with data science or stay in Data Engineering (which includes skills of data analyst too). hope this helps, happy learning : )
I used aws glue crawler. I am taking my mssql table to s3 and then through glue to athena so that i can query. In s3 i just checked the columns using objects action button available on bucket and then run a query. So there data is same as mssql. Data in columns where it should be. In athena some columns got mismatched and i don't know how to fix glue crawler now. Kindly help me out Basically crawler got confused because some cols has json type data in rows
Darshil what do you think in future slowly slowly in data engineering cloud services will be used more and decrease open source tools like hive hbase etc.. means data engineering tech stack will go as much as possible on the cloud ?
Open Source will never go away, most of the cloud services use Open Source projects and provide their own feature on top of it, like Managed Airflow, EMR (Hadoop MapReduce) Cloud is future but open is a base for everything
Because you can't do everything on snowflake, theoretically you can say (for marketing purpose) but real-world problems are not that straight forward you need custom solution based on requirements
Bro bura mat maano but , I am following you since 1 year.. And aap ne last 1 saal main 10 baar roadmap pe video banayi aur wahi baatein repeat ki har video main.. wahi baatein pipeline creation main batate ho... Aap isse accha AWS ki pipelines wali ek ek service in depth padhao
![[LIVE] : ONE ลุมพินี 82 วันนี้!! คู่เอก | ยอดไอคิว vs อับดัลลาห์](http://i.ytimg.com/vi/hSsyTl_9bqs/mqdefault.jpg)
New setup? 👀 I am working on next GCP project, I will upload it this month and launch SQL for Data Engineering course too :D
Don't forget to like this video :)
hi i am not able to get any internship in data field i have a ba hons degree i have made portfolio also what should i do
Love the new setup! and very very thanks for this amazing info. Excited for SQL course 🤩🤩
TL;DR
0:50 AWS S3 (Simple Storage Service ) To store any type of data
1:51 AWS Glue used for transformation scripts
2:29 AWS EMR compute over a distribued cluster
2:55 AWS Redshift data warehouse where extracted datas are loaded for further process
3:30 AWS Athena to directly querry data from S3 without storing it in warehouse
4:39 AWS Lambda compute service
5:26 AWS Kinesis to process data in real time
6:06 AWS DMS (Data Migration Service)
7:09 AWS Lake Formation
7:13 Amazon Managed Workflow for Airflow
7:25 Amazon QuickSight
7:27 Amazon RDS
7:48 Amazon VPC
7:50 Amazon Ec2
7:56 Amazon SNS
7:58 Amazon CloudWatch
8:00 Amazon IAM
whats the difference between EMR and Lambda as both are for compute purpose?
😊
After completing your python course this particular video made lot of sense, still practicing and learning these cloud services thanks man! Btw setup seems nice🔥👏🏻
Awesome Hritika, keep practising and you’ll understand it better
Great video Darshil, comprehensive explaination and good scope of DE processes. After I've completed the 2 main courses I'm currently studying, I'll switch focus to some AWS projects.
Thanks Gavin, all the best 👍🏼
Please also make video on GCP Data Engineer Roadmap.
Hi Darshil your videos on Data Engineer are very interesting and to the point.
I would like to make a request can you please make videos on Azure.
For example showing, completea pipeline for any OLTP or OLAP project?
Thanking you in advance
I will put this in my list
I really loved your classes but I have one concern if you make any projects related to streaming data using technologies like Hadoop, Kafka, spark, hive, pig, and Flik related to hospital data it would be very helpful for me as a learner of your videos
Thank you for this informative video
Book My Show has a crazy ETL pipeline. Thanks for showing us.
Bro, What do you think what's the future of data engineering after this recent AI boom ? I joined an MNC as a fresher in for data integration role.
wel that was very useull now i know everything!
Great video brother ur all videos are very helpful for me..
Data engineer with AWS Book is best to study and understand AWS
Hi, if u have this book please send me
I don't have the pdf but i can tell u that orielly's website's providing 10 days free subscription u can read there
We can even modify data through Athena by using ICEBERG type table.
Yes
Darshil in data engineering
We should know a lot other than practice
Or practice will make us stronger in data engineering
Theory + Practice = Win
Keep it up! It helps a lot for students like us who want to get into data engineering
Thank you Muskan
please make video and courses on azure Data engineering road map, thanks.
Hey Darshil, I was waiting for this video to understand the integration of cloud services with ETL process and fundamentals of these services.
Thanks for the explanation !!
Thank you Anurag
Make video of dms
Thanks man
Hey Pls tell whether it is mandatory to give AWS certification exam to become cloud data engineer ?
Well explained Great stuff buddy ✌️
Great Video man!
If I start learning cloud, which cloud is good and which one can get a job quickly
very informative video and new setup is looking great
Thank you Bilal
helpful and providing your insights.
Glad it was helpful!
Need your advice brother
Between data analyst vs data engineering vs data science
1)Which role requires less coding ? I am a civil engineer who can do decent coding but don’t really enjoyed it like coding geeks do. So need your advice. People suggested me for data analysis profile as it has less doing and needs knowledge of data visualisation but would like to know your inputs
2)which role has more jobs in India ?
Data scientist has jobs abroad but in India it’s still in less no and limited to folks from top colleges or brilliant coders, data engineering has decent jobs, no clue about data analyst job scenario
Please help
start with data analysis>after gaining some experience move to DE> then you will get a clear understanding of how an end2end business problem is solved. By that time you will know should I go with data science or stay in Data Engineering (which includes skills of data analyst too).
hope this helps, happy learning : )
Is aws glue and airflow is same?
Hello Darshil, Can you please bring some more real time projects? On Azure as well
I am working on it, the thing is creating end-to-end projects takes around 3-4 weeks of efforts and requires lot of planning
Hi Darshil, Need similar one on GCP
i will add one more service, aws cloudformation for automating all these
Hii darshil pls do the project on azure
I will put this on my list
Great video bro❤
Man you are a great deal !
Hey thanks man
I used aws glue crawler.
I am taking my mssql table to s3 and then through glue to athena so that i can query.
In s3 i just checked the columns using objects action button available on bucket and then run a query. So there data is same as mssql. Data in columns where it should be.
In athena some columns got mismatched and i don't know how to fix glue crawler now.
Kindly help me out
Basically crawler got confused because some cols has json type data in rows
Darshil what do you think in future slowly slowly in data engineering cloud services will be used more and decrease open source tools like hive hbase etc.. means data engineering tech stack will go as much as possible on the cloud ?
Open Source will never go away, most of the cloud services use Open Source projects and provide their own feature on top of it, like Managed Airflow, EMR (Hadoop MapReduce)
Cloud is future but open is a base for everything
Can I put on my resume the real time projects after completing your course ?
Yes you can
What about gcp project darshil when he coming
Working on it, will update once it is done
Can't we do all these in Snowflake ? Why would anyone use these services?
Because you can't do everything on snowflake, theoretically you can say (for marketing purpose) but real-world problems are not that straight forward you need custom solution based on requirements
Bro bura mat maano but , I am following you since 1 year.. And aap ne last 1 saal main 10 baar roadmap pe video banayi aur wahi baatein repeat ki har video main.. wahi baatein pipeline creation main batate ho... Aap isse accha AWS ki pipelines wali ek ek service in depth padhao
Last 1 year me 5 end-to-end projects bhi banai hai jisme sab service ka indepth explanation and hands-on bataya hai vo bhi dekhlo ;)
@@DarshilParmar ok bro , thanks
bro is this sufficient any free resources and shall i do cloud practisner certification im an aspiring data engineer , can u guide please
This is enough but you can do AWS Data Analytics Certification
@@DarshilParmar thanks bro
@@DarshilParmarfrom where we can do it ???
You got 6 more hours to make a aprils fools community post 😂
😂😂
helpful and informative
Thank you Revanth