Worked really hard for this :) Don't forget to hit LIKEEEE and Subscribe! There are lot of "Okays" in earlier part of video, then it reduces are you watch, I didn't even realize, will keep this in my mind for next time! Also for part to suggest topics 👇🏻
I am a mid senior level DE from Canada and I would like to mention that the work you are doing towards the Data Engineering community is a great help! keep up the good work
Summary : The video is a comprehensive Data Engineering Master Class that explains the fundamentals of data engineering, including data generation, storage, processing systems, architectures, and tools like ETL (Extract, Transform, Load). It covers various topics such as SQL versus NoSQL, OLTP and OLAP systems, data warehousing, dimensional modeling, and cloud data services on platforms like AWS, GCP, and Azure. The instructor emphasizes hands-on learning through projects and insights on key technologies necessary for aspiring data engineers. Key Points Fundamentals of Data Engineering 00:00 The master class introduces the concept of data engineering and outlines the topics that will be covered in the video. 04:30 The significance of data in understanding customer behavior and improving business processes is discussed, underscoring the importance of data-driven decision-making. Data Engineering Lifecycle 09:00 The data engineering lifecycle begins with data generation, which happens through applications and interactions, capturing valuable customer data. 12:40 Various database management systems (DBMS) are introduced, explaining the CRUD (Create, Read, Update, Delete) operations crucial for data handling. SQL vs. NoSQL and Data Modeling 18:00 Differences between SQL (relational) and NoSQL (non-relational) databases are highlighted, detailing their specific use cases in data storage. 23:15 The process of data modeling is explained, showing how to create a structured representation of data through tables and relationships. Data Warehousing and OLTP vs. OLAP 30:00 The purpose and design of data warehouses are elaborated, including the importance of dimensional modeling. 35:40 The functionalities and use cases of OLTP (Online Transaction Processing) versus OLAP (Online Analytical Processing) systems are clarified, focusing on their roles in transactional and analytical tasks. ETL Processes and Modern Data Architecture 45:00 The ETL (Extract, Transform, Load) process is discussed in detail, emphasizing the transformation of raw data into meaningful insights. 51:15 The importance of modern data architecture is introduced, encouraging cloud-based solutions for scalability and flexibility. Tools and Technologies 58:00 Key tools for data engineering such as Apache Spark, Kafka, and Airflow are recommended, illustrating their applications in processing and orchestration. 01:10:00 The viability of different cloud platforms (AWS, GCP, Azure) is assessed, along with their data services and the emerging modern data stack. Real-World Applications and Case Studies 01:30:00 Real-world architectures, such as those used by companies like Dream11, are showcased, demonstrating how data engineering principles are applied in practice. 01:45:00 The use of data lakes versus data warehouses is compared, explaining how to handle unstructured and structured data effectively. Summary and Resources 02:30:00 The instructor summarizes the course, emphasizing the importance of learning through projects and offers resources for further learning in the field of data engineering. 02:50:00 The video concludes with encouragement to explore various tools and to engage in practical projects, reinforcing the skill set necessary for future data engineers.
I had been searching for a data engineering video for a long time and then I found this one.Great course to understand the basic concepts of data engineering and how it really works in the real world.Keep up the good work and please make a 2nd part of this video where you can build a real world project using the components you just explained.It can be a very small project but should shows all the stages from beginning to end. Thanks for the awesome video.
being a data analyst ,, never kept much interest in DE but your content is compelling to learn one, u explained in simple layman language,,, appreciate that , really helpful, hoping for more such content for DS , DE , DA , ML field , kudos to you...
"Wow, this was such an amazing effort! 👏🔥 Loved your energy and the way you improved as the video went on. The 'Okays' actually added a personal touch in the beginning! 💬 Keep up the great work, and can't wait to see what you bring next! 💪🎥✨ #HardWorkPaysOff #Subscribed"
I found out I have an interest in computers and started majoring in computer science and looking at jobs like software engineer, data engineer and more. This video, and all your videos are incredibly valuable to me to learn what fields I'm most interested in. I wanted to say an extra thank you for helping me.
Hi Darshil, I am usually into Data Analyst background, but I loved the way you made us understand the fundamentals of Data Engineering and now I want to shift my career into DE. It's like knowing the birth of data.
Hi Darshil, I kindly request you to please make detailed long videos on every single topic you covered in this video as an intro.. please please think of it and please do upload such long and detailed videos.. Thankyou so much for this video
Oh my goodness, it's crazy how you could re-record and do the whole thing once again! Kudos mahn! I stumbled across this video and then watched so many more videos from your channel. Keep doing what you do, you have been really helpful and inspiring.
Thank you for putting this effort into making this content available to newbies. You make the concepts so easy to understand better than most of my lecturers.
@DarshilParmar, This is Hands Down THE BEST Data Engineering Introduction I have seen ever. ❤ Kudos and Thanks a Ton for making it which will help many beginners like me. Your Efforts are tremendous. I'm saving this Video Permanently in my List. I have seen it Video till end and I can feel your Throat Pain due to long hours of talking 😢. Pls take care man, you need to pause and rest Ample time before Recording video again.
very good and informative video, just finished watching after a few days. And this is so true, after learning one tool or cloud provider, it becomes way easier to understand the other, but we should not forget the basics.
Great video ,thanks for the time and effort you put in this channel and thanful you are on youtube ,I'm going to continue watching your awesome videos!
Thank you Darshil. It was an amazing walkthrough of the whole functionalities. I am preparing for Data Analytics role and often I have been told to understand the flow of Data in order to have a solid understanding of the background of data and to excel in job. This is exactly what I needed.
I think this is one of the best structured presentation on the Data Engineering course, very insightful. Your energy and hardwork will definitely benefit the working and the job seekers in the industry.
Thanks for your efforts. I am learning SQL right now. Going to buy your courses too. I am slowly understanding how SQL is a backbome of data engineering instead of using fancy tools.
I am a RPA Developer but want to switch my career as a Data Engineer and I was looking for an excellent course that can help me to create some solid base and then I came across with this video, Thanks Man! keep doing the good work.
This was a tremendous effort. Hats off for taking the time to explain everything, simplify big concepts and get into the details. The 3 hours flew by...
This is really a great guidance and information for the person to know what is data engineering all about in the real-world scenario. You covered most of the topics and fundamentals. Really great man. I am happy that i subscribed to your channel. Looking forward for the more informative content in future.
Thanks for all the topics covered. It surely takes a lot of Hard work behind the scene to create such content. Keep doing, you are helping lots of students.
Before this video I thought I have to visit 3,4 videos to understand each concept which are coverd in this video but now I have a fundamental idea on all of the concepts in data engineering. Tqq
@DarshilParmar - you are amazing. Your method of teaching is quite simple. Particularly about this code - I was looking for something like this to understand the entire data engineering in a couple of hours and it helped. Thanks! Keep Going!
Hello @darshilparmar. I just finished watching this 3 hour video. I like complete roadmap and being early data engineer. It surely helpful for me and I want to join the course. Thank you s0 so much for the info, I Have been following you since quite long time via linkedIn and youtube. 😃👍. Literally from 2:10:06 is next level! 🤩feeling very good for being data engineer😅
Hey Darshil, I was tasked to do ETL using Apache Nifi, snowflake and KNIME for analytics. As a noobie Data Engineer this video is really informative and gave me the knowledge I needed. Thank you!
Worked really hard for this :) Don't forget to hit LIKEEEE and Subscribe!
There are lot of "Okays" in earlier part of video, then it reduces are you watch, I didn't even realize, will keep this in my mind for next time!
Also for part to suggest topics 👇🏻
Have you taken inspiration from joe reis book for this video?
Yes, some concepts are from that book, also added credits for those section
Thank you 💜
Brilliant! Delivery and content. Thank you very much
Like is trivial you deserve more. Loved the way you structured this course and the way you taught. Hats off ♥
I am a mid senior level DE from Canada and I would like to mention that the work you are doing towards the Data Engineering community is a great help! keep up the good work
Summary :
The video is a comprehensive Data Engineering Master Class that explains the fundamentals of data engineering, including data generation, storage, processing systems, architectures, and tools like ETL (Extract, Transform, Load). It covers various topics such as SQL versus NoSQL, OLTP and OLAP systems, data warehousing, dimensional modeling, and cloud data services on platforms like AWS, GCP, and Azure. The instructor emphasizes hands-on learning through projects and insights on key technologies necessary for aspiring data engineers.
Key Points
Fundamentals of Data Engineering
00:00
The master class introduces the concept of data engineering and outlines the topics that will be covered in the video.
04:30
The significance of data in understanding customer behavior and improving business processes is discussed, underscoring the importance of data-driven decision-making.
Data Engineering Lifecycle
09:00
The data engineering lifecycle begins with data generation, which happens through applications and interactions, capturing valuable customer data.
12:40
Various database management systems (DBMS) are introduced, explaining the CRUD (Create, Read, Update, Delete) operations crucial for data handling.
SQL vs. NoSQL and Data Modeling
18:00
Differences between SQL (relational) and NoSQL (non-relational) databases are highlighted, detailing their specific use cases in data storage.
23:15
The process of data modeling is explained, showing how to create a structured representation of data through tables and relationships.
Data Warehousing and OLTP vs. OLAP
30:00
The purpose and design of data warehouses are elaborated, including the importance of dimensional modeling.
35:40
The functionalities and use cases of OLTP (Online Transaction Processing) versus OLAP (Online Analytical Processing) systems are clarified, focusing on their roles in transactional and analytical tasks.
ETL Processes and Modern Data Architecture
45:00
The ETL (Extract, Transform, Load) process is discussed in detail, emphasizing the transformation of raw data into meaningful insights.
51:15
The importance of modern data architecture is introduced, encouraging cloud-based solutions for scalability and flexibility.
Tools and Technologies
58:00
Key tools for data engineering such as Apache Spark, Kafka, and Airflow are recommended, illustrating their applications in processing and orchestration.
01:10:00
The viability of different cloud platforms (AWS, GCP, Azure) is assessed, along with their data services and the emerging modern data stack.
Real-World Applications and Case Studies
01:30:00
Real-world architectures, such as those used by companies like Dream11, are showcased, demonstrating how data engineering principles are applied in practice.
01:45:00
The use of data lakes versus data warehouses is compared, explaining how to handle unstructured and structured data effectively.
Summary and Resources
02:30:00
The instructor summarizes the course, emphasizing the importance of learning through projects and offers resources for further learning in the field of data engineering.
02:50:00
The video concludes with encouragement to explore various tools and to engage in practical projects, reinforcing the skill set necessary for future data engineers.
Doing god’s work. Hope your pillow stays cool.
Brushing up my basics before my DE interview. Thank you for existing
Bro can u help me ? I want to learn DE from scratch
@@jk_roh3t hey brother can we connect , i also want to learn DE from scratch
Hey brother. Even I want to brush up my DE Skills. Can we connect?@@prathamsharma7325
Best DATA Engineering fundamental Video I Ever Seen.LOVE U AND BLESS U.
I had been searching for a data engineering video for a long time and then I found this one.Great course to understand the basic concepts of data engineering and how it really works in the real world.Keep up the good work and please make a 2nd part of this video where you can build a real world project using the components you just explained.It can be a very small project but should shows all the stages from beginning to end. Thanks for the awesome video.
this is the best data engineering course i have seen on youtube
Great video to understand the fundamentals of Data engineering . Keep up the good work. Thankyou for sharing your knowledge among us.
being a data analyst ,, never kept much interest in DE but your content is compelling to learn one, u explained in simple layman language,,, appreciate that , really helpful, hoping for more such content for DS , DE , DA , ML field , kudos to you...
I watched this video till this time 70% done.......best video so far😄
Writing this comment to push this video in algorythms of recomendations. Very good job, thank you for the effort.
Congrats Darshil. I'm new to Data Engineering. Your explanation is obvious and systematic. Learned lot of things. Thank you for your efforts.
"Wow, this was such an amazing effort! 👏🔥 Loved your energy and the way you improved as the video went on. The 'Okays' actually added a personal touch in the beginning! 💬 Keep up the great work, and can't wait to see what you bring next! 💪🎥✨ #HardWorkPaysOff #Subscribed"
All i can say is You are doing an Awesome job!! Learning a lot from your teachings. Thanks a lot!!
I found out I have an interest in computers and started majoring in computer science and looking at jobs like software engineer, data engineer and more. This video, and all your videos are incredibly valuable to me to learn what fields I'm most interested in. I wanted to say an extra thank you for helping me.
All the best
So much covered in this video. Thanks for going into a depth. It's definitely appreciated for the data engineering community!
Your videos are amazing!!! Thank you Darshil :)
No more searching in youtube for DE fundamentals after watching this video, Thanks man for you constant support for DE community.💌
100% rare and best video ever for data engineer fundamentals in the youtube history🔥, hatsoff bro🫡
Best TH-cam Video seen this year. I'll definitely be enrolling for the full combo courses when you are done creating them. Thanks a lot for this
Hi Darshil, I am usually into Data Analyst background, but I loved the way you made us understand the fundamentals of Data Engineering and now I want to shift my career into DE. It's like knowing the birth of data.
Watched entire video in 1 go. Best data engineering channel
Started watching this today❤ I'm new to Data Engineering.
Hands off to you bro🙌❤️
Hi Darshil, I kindly request you to please make detailed long videos on every single topic you covered in this video as an intro.. please please think of it and please do upload such long and detailed videos.. Thankyou so much for this video
Wow such a tremendous job Darshil ! You explained it so clearly that every basic person to data field, can understand it. Thank u so much man.
Oh my goodness, it's crazy how you could re-record and do the whole thing once again! Kudos mahn!
I stumbled across this video and then watched so many more videos from your channel. Keep doing what you do, you have been really helpful and inspiring.
Thank you for putting this effort into making this content available to newbies. You make the concepts so easy to understand better than most of my lecturers.
You are amazing, how much effort you put in to make this video and. the explanation was awesome - thanks a lot
Hello from Brazil, you are the best teacher.
Great Video. I learned alot. Thanks
Thanks Darshil for all your efforts to create the video and sharing it with us. This video gives a holistic view of Data Engineering.
The effort and dedication to teach and put together this is insane! Hands down bro, you truly gave a master DE class!
It would really help me a lot. thank you☺.
The Best video ever for stating career as Data Engineer. Thank you Darshil for everything.
Thank you for the basic info.... just completed... will be starting for further preparation now....was really helpful
Awesome content for getting started in the field of Data Engineering
I can only say one thing
Thank you Thank you Thank you form my heart
This was a joy to watch
Thank you so much for the efforts and teaching the fundamentals. It will help a lot of people who aspire to become Data Engineer.
Hey! How are you? I watched the entire video, and it gave me a broader understanding of data engineering. Thanks for this masterclass.
One hour in. Great content !🔥🔥🔥🔥✨✨✨✨✨✨
Great video! Your videos have tremendously helped me in pursuing DE as career.
Thanks so much!!!
You are so welcome!
hey ! can we connect , i am also thinking about DE as a career
@@prathamsharma7325 this is not linkedin, this is youtube
This was VERY informative. As someone that is wanting to persue DE, this is the foundation I was looking for. :) Thanks mate.
Appreciate the effort in putting this together. Really helping me solidfy those foundations as someone who is pursuing a career in Data Engineering.
That's really informative! Thank you Darshil!!
Awesome explanation, Loved the way you structured this course and the way you taught. Hats off ♥
amount of understanding i got through these video cleared all real world work of data engineering.Thank You Darshil.
@DarshilParmar,
This is Hands Down THE BEST Data Engineering Introduction I have seen ever. ❤
Kudos and Thanks a Ton for making it which will help many beginners like me. Your Efforts are tremendous.
I'm saving this Video Permanently in my List.
I have seen it Video till end and I can feel your Throat Pain due to long hours of talking 😢.
Pls take care man, you need to pause and rest Ample time before Recording video again.
very good and informative video, just finished watching after a few days. And this is so true, after learning one tool or cloud provider, it becomes way easier to understand the other, but we should not forget the basics.
Loved the video. Great work!!!
Everything in one place. Appreciate you efforts
Great video ,thanks for the time and effort you put in this channel and thanful you are on youtube ,I'm going to continue watching your awesome videos!
Awesome! Thank you!
Thank you Darshil. It was an amazing walkthrough of the whole functionalities. I am preparing for Data Analytics role and often I have been told to understand the flow of Data in order to have a solid understanding of the background of data and to excel in job. This is exactly what I needed.
Good explanation, easy to understand
Excellent video for fundamentals
Great Video to understand Data Engineering!!
The best video I have ever come across regarding data engineering basics. ❤❤
Glad you liked it!
You have done a great job, thank you very much!
Amazing class for all aspiring DE even for software engineers. Really appreciate you efforts @DarshilParmar for making this.
Amazing bro, helping a lot of people. Keep it going 👍
Great video for leaning data engineering. can you talk more about data orchestration concept in next video?
Thankyou so much. It really helped clear the basics and put the pieces together to understand the whole picture of data engineering.
This video is great. Your mode of teaching is top-notch.
Thank you
okay should a total beginner can take this
I think this is one of the best structured presentation on the Data Engineering course, very insightful. Your energy and hardwork will definitely benefit the working and the job seekers in the industry.
Great Video
Thanks for your efforts. I am learning SQL right now. Going to buy your courses too. I am slowly understanding how SQL is a backbome of data engineering instead of using fancy tools.
Amazing content and the way you explain things.. Watched fully..!
@Darshil Parmar Thanks for your lesson for around 3 hours, please make more related to DE or DS or DA.
What a wonderful video , thoroughly enjoyed watching it , waiting for your detailed course in AWS and Azure
Thanks Darshil for creating such needed introductory video. It helps a lot!!
Damn I loved your video this is very helpful covered maximum important topics .👍🏻
thanks great job keep up the good work
I am a RPA Developer but want to switch my career as a Data Engineer and I was looking for an excellent course that can help me to create some solid base and then I came across with this video, Thanks Man!
keep doing the good work.
Salute to your efforts man! Really helpful video.
Glad it was helpful!
This was a tremendous effort. Hats off for taking the time to explain everything, simplify big concepts and get into the details.
The 3 hours flew by...
You are welcome
I learned a lot from your video Darshil, expecting a lot more from you. Thank you so much for yours efforts. I appreciate it.
Great content as always!
Great video @DarshilParmar.This video is a one stop shop for gaining knowledge about DE fundamentals.
Everything in one place. Great DE roadmap. Appreciate you efforts. Thank you!
Awesome explained
Great video man, I really appreciate your efforts
beautifully covered all the basics. kudos
Great Video!! Thanks for this in-depth introduction.
Glad it was helpful!
This is really a great guidance and information for the person to know what is data engineering all about in the real-world scenario. You covered most of the topics and fundamentals. Really great man. I am happy that i subscribed to your channel. Looking forward for the more informative content in future.
Thank you for helping me refresh my fundamentals before the interview. I really enjoyed learning from your videos.
Thanks for all the topics covered. It surely takes a lot of Hard work behind the scene to create such content. Keep doing, you are helping lots of students.
Thanks a ton
This is incredible, I understood so much of the concepts from this video. The application or use cases to explain the concepts were mind blowing
Glad it was helpful!
Great video and good job Sir. ❤
Before this video I thought I have to visit 3,4 videos to understand each concept which are coverd in this video but now I have a fundamental idea on all of the concepts in data engineering.
Tqq
Making data easier is my goal
thank you brother, this intro course very well explained
very informative data in one single video thankyou for making
Great Work Man!!! Appreciate your efforts.
@DarshilParmar - you are amazing. Your method of teaching is quite simple. Particularly about this code - I was looking for something like this to understand the entire data engineering in a couple of hours and it helped. Thanks! Keep Going!
You are most welcome
2:01:11 yep still watching bro I really like the way you teach 😊
Thanks Darshil for your hard-work & easy explanation of such a tough topic.
Hello @darshilparmar. I just finished watching this 3 hour video. I like complete roadmap and being early data engineer. It surely helpful for me and I want to join the course. Thank you s0 so much for the info, I Have been following you since quite long time via linkedIn and youtube. 😃👍. Literally from 2:10:06 is next level! 🤩feeling very good for being data engineer😅
Amazing
Seriously Man, you made this video. I just started reading this book after you recommended.
Hats off to you man.
You are welcome
What's that book ?
@@chandangadamsetty4633Fundamentals of Data Engineering
What's the book name
Hey Darshil, I was tasked to do ETL using Apache Nifi, snowflake and KNIME for analytics. As a noobie Data Engineer this video is really informative and gave me the knowledge I needed. Thank you!
Glad it was helpful!