What do DATA ENGINEERS do? Is data engineering a good career choice in 2020?

แชร์
ฝัง
  • เผยแพร่เมื่อ 21 ก.ค. 2024
  • ***
    My FREE guide on how to become a data engineer:
    karolinasowinska.notion.site/...
    ***
    I currently work as a data engineer and I'd like to tell you what data engineering is all about! It's commonly confused with machine learning, data science etc., when in fact data engineering is quite a distinct area of technology.
    So, what do big data engineers do?
    The job revolves around bringing new data to the company (ETL process if you like) and then maintaining the data flow!
    Is data engineering a good career choice in 2020? Long story short - yes. As the amount of data we produce is growing rapidly, so is the demand for data professionals. However, my prediction is that many of the current data engineering tasks might get standardised, and eventually automated in the future. But hey, this is the XXI century! We'll have to learn new skills and adapt all the time!
    -------
    If you'd like to learn data engineering, I recommend following the 4 simple steps below to land you the first job interview:
    In this long-awaited Airflow for Beginners video I'm showing you how to install Airflow from scratch, and how to schedule your first ETL job in Airflow!
    We will run the Spotify ETL job that we created in my Data Engineering Course for Beginners. But if you have your own code to schedule in Airflow - feel free to use it instead!
    In this Airflow tutorial I just wanted to get you started with Airflow, because I know that configuring Airflow can be non-trivial. I won't be going in-depth into all Airflow's features, so hopefully this tutorial will be as beginners friendly as it can get!
    I'll show you how to fix some installation errors, how to set up your project folders, how to edit Airflow config file and where it sits, and how to write your first DAG file!
    Get ready for a ride!
    Oh, but grab some useful links first:
    -github code repo: github.com/karolina-sowinska/...
    -Airflow quick start guide: airflow.apache.org/docs/stabl...
    -Spotify access token: developer.spotify.com/console...
    Don't forget to subscribe if you're new here, more content is coming!
    :)
    ---------
    If you'd like to learn data engineering, I recommend following the 4 simple steps below to land you the first job interview:
    1. Learn Python
    I recommend following the Python for Everybody specialization course on Coursera, which is one of the most popular courses there:
    imp.i384100.net/x9gVO3
    2. Learn SQL
    SQL is still the lingua franca of data. I recommend going with Learn SQL Basics for Data Science course, because it contains some chapters which are very releavant to data engineering in partcular, e.g. distributed computing with Spark
    imp.i384100.net/QOMZ09
    3. Learn Bash scripting/Linux
    I wouldn't take a full course on it, but at least read a good article.
    if you do prefer to take a course/guided project, I think this one is short and good:
    www.coursera.org/projects/com...
    4. Learn how to develop on the cloud, e.g. on AWS
    There are a few good courses around there, but I think the Coursera one is the most comprehensive
    imp.i384100.net/P0MJBM
    Discloure: The above contain affiliate links, meaning when you click the links and make a purchase, I receive a commission!
    -------
    Find me on Instagram: @karo_sowinska
    And if you want to make my day with a cup of coffee... :)
    ko-fi.com/karolina_sowinska
  • วิทยาศาสตร์และเทคโนโลยี

ความคิดเห็น • 366

  • @karolinasowinska
    @karolinasowinska  3 ปีที่แล้ว +57

    I've just released my Free Data Engineering Course for Beginners on TH-cam! Make sure to check it out if you're interested in a career in data engineering! :)
    th-cam.com/video/dvviIUKwH7o/w-d-xo.html

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      @Profound Programing I don't, but you can also find me on Instagram: @karo_sowinska :)

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      @Profound Programing I'll consider setting it up at one point! :)

    • @annmaryjo3803
      @annmaryjo3803 3 ปีที่แล้ว +1

      U r really amazing to starters. NB: I am just studying in 9th standard 😅 just checking how it works so that I don't have to look for this in more big way😁

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      @@annmaryjo3803 Thanks, and good luck on your journey! :)

    • @giljav
      @giljav 3 ปีที่แล้ว

      What is your background and how you got into data engineering?

  • @patrickd4468
    @patrickd4468 3 ปีที่แล้ว +105

    I am Data Engineer and I describe myself as follows: "I am like a plumber in the data world and ensure data flows from one abstract system to another in a scalable, consistent and standardized manner to allow the business users to consume the data in a usable way" . My only advise is to ensure all your pipelines are robust and fault tolerance/notifications are built into your data layer.

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว +14

      Absolutely, a data engineer should always consider as many edge cases as possible, because they happen more often than not! I'm glad to read a comment from a fellow data engineer! ;)

    • @mauricewalshe8339
      @mauricewalshe8339 3 ปีที่แล้ว +1

      And in finance make sure the pennies add up :-) I used to work on billing systems for telecoms multiple types of records multiple types of data sources - we used Map Reduce (back in the early 1980's ) with F77 and Pl/1G

    • @sumittamang3548
      @sumittamang3548 2 ปีที่แล้ว

      If you had to suggest an undergrad student between data science and data engineering, what would you suggest?

  • @peterndiforchu2622
    @peterndiforchu2622 3 ปีที่แล้ว +2

    Love it! How comes I'm just discovering you. Love your sense of humor. I'm working on the path of becoming a data engineer. This is encouraging as all the 5 points are things that excite me. Keep sharing great content. We'll keep supporting.

  • @vsevapopov2
    @vsevapopov2 3 ปีที่แล้ว +3

    So great to have such video lessons on youtube! I had an internship 2 and a half years ago where I did the data engineering task and now I want to continue my journey!! Now I study at university and I have much time to do that! Thank you!!!!!

  • @victorborges5291
    @victorborges5291 3 ปีที่แล้ว +4

    Knowing full well each job title is like a passing show these days, without doubt, I'm inspired! 👍

  • @anyasyht
    @anyasyht 3 ปีที่แล้ว +21

    As a data engineer I couldn’t agree more with you!

  • @MisterNorthernCanuck
    @MisterNorthernCanuck 2 ปีที่แล้ว +4

    I just got myself a job as a data engineer as my new career challenge. I appreciate your content, it essentially confirms what I'm getting into :)

  • @hypebeastuchiha9229
    @hypebeastuchiha9229 3 ปีที่แล้ว

    Thanks for this video I’m happy to see an actual data engineer give her opinion unlike bigger cs career channels that just read articles

  • @chrisv8341
    @chrisv8341 3 ปีที่แล้ว +3

    This is extremely clear and concise! Thank you for sharing Karolina :)

  • @samguy7209
    @samguy7209 3 ปีที่แล้ว +5

    Thank you for sharing your experience and the great description. I love your accent. You have helped me decide my future career. ❣️

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      That's wonderful, I'm glad my video was so useful! ;))

  • @md.shahriarrahmanrana6692
    @md.shahriarrahmanrana6692 4 ปีที่แล้ว +1

    You helped me a lot Karolina. Continue to share your experience, so that we can learn from you.

  • @jmrodrig00
    @jmrodrig00 3 ปีที่แล้ว +2

    Hi Karolina, thanks for sharing your thoughts about the subject, I starting a career as a data engineer and you video clarifies me and motivate me to do it as quickle as possible.
    Best regards.

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      Good luck on your journey and I hope to see you around here! ;)

  • @yooalejandro386
    @yooalejandro386 3 ปีที่แล้ว +11

    That's amazing! Your speaking is very logic and concise, covering a lot of the most important concepts or sessions a data engineer does everyday. A very talented and graceful lady. Thanks for sharing and god bless you.

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      Thank you so much! I'm glad you like my delivery! :)

  • @caassimbah2485
    @caassimbah2485 2 ปีที่แล้ว +1

    This was so helpful and informative. Thank you!

  • @bhuvaneshrao6322
    @bhuvaneshrao6322 4 ปีที่แล้ว +5

    This is one of the very good videos i have seen on data engineering, thanks.

    • @karolinasowinska
      @karolinasowinska  4 ปีที่แล้ว +1

      Thank you, that means a lot. The saga is to be continued very soon!

  • @ryunosukefuriya3748
    @ryunosukefuriya3748 3 ปีที่แล้ว +3

    Super glad I came across your channel, Karolina! I'm currently a DA working to make some long-term transitional goals come true! -> data engineering -> data science. Thanks for your awesome insights! Your delivery is really engaging and easy to understand! Best wishes, Karolina!

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว +2

      Thanks, I'm glad you enjoy the videos! :) And your career transition plan sounds great, good luck on your journey!

  • @lfrqz
    @lfrqz 3 ปีที่แล้ว +7

    Your vlog feels real, honest and accurate. I am learning about data science, sql, python, etc. and I find your content very inspiring. Thanks for sharing your knowledge and experience. I am looking forward your future content. Greetings from Mexico.

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว +1

      Thanks so much Luis, I'm glad that you're enjoying my videos! :) Good luck on your data science journey!

  • @mokshgrover9517
    @mokshgrover9517 3 ปีที่แล้ว +3

    I wanted clarity on this profession for a while now. Nobody could explain this in an easy way. You did. Thank you! Very informative.

  • @Robi2009
    @Robi2009 4 ปีที่แล้ว +2

    Komentarz dla zasięgu ;)
    Powodzenia w rozwoju kanału!

  • @juanluisduquelafuentes1823
    @juanluisduquelafuentes1823 4 ปีที่แล้ว +4

    Thanks for another interesting video Karolina! You know...as I was watching the video and hearing you talk about the different tasks data engineers have to perform, I kept thinking about the inevitability of all of those processes getting automated eventually. And of course you concluded with that interesting tidbit about having to learn new skills because you anticipated automation. Just goes to show, not only do you have to identify new opportunities and learn new skills in order to keep thriving, but you also have to develop a strategy to become as financially independent as possible so that you’re not caught all of a sudden in a situation where your skills just cannot support a decent wage anymore. If that doesn’t motivate someone to get off the daily grind for a few hours a night and just sit down, make new plans, pursue other interests, and just think about the meaning of it all, I don’t know what will!

    • @karolinasowinska
      @karolinasowinska  4 ปีที่แล้ว

      My thoughts exactly, Juan! What you said about financial independence in particular. Sadly, in the face of automation, it is those who are not financially independent, or whose skills are not sufficiently transferable, that will be hit the most.

  • @dianerocher6705
    @dianerocher6705 3 ปีที่แล้ว +12

    As a recruiter trying to understand job queries, this is gold ! Many thanks

  • @splashoui3760
    @splashoui3760 2 ปีที่แล้ว

    Was a good video. Thank you for this one!

  • @kevinle5772
    @kevinle5772 3 ปีที่แล้ว +3

    I like your clip, very well articulated. It's like you are reading my minds.

  • @vineetjainorg
    @vineetjainorg 4 ปีที่แล้ว

    Really liked your video! I work as a Data Scientist and can totally relate to the hard work our data engineering team puts in.
    Thanks, keep making more videos!

    • @norpriest521
      @norpriest521 4 ปีที่แล้ว +1

      Can you let me know what is the career path for data engineer?
      Say I don't want to work as data scientist or machine learning engineer, I only like to do data engineering my my entire professional career.
      Can one go far just by working as data engineer only and doesn't move to other areas like ML or DS on the later stage?

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      Hi, absolutely! Data Engineering career path, like all career paths in technology, can take many shapes. But in majority of cases, you'll start out as a analyst/developer (building, fixing data pipelines), and then you will move on to assuming more responsibility, e.g. designing the entire data architecture of a company or managing teams.

  • @sajarnitsa
    @sajarnitsa 3 ปีที่แล้ว +1

    Thank you very much for this video. You could believe you are terrible helping other people installing software and things like that, but your videos are the most honest and down to earth description of the job of Data Engineering You are amazing.

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      I'm so happy that I could help some people with the videos! Thanks!

  • @exxodduss2008
    @exxodduss2008 3 ปีที่แล้ว +2

    Perfect description

  • @lovemaakhe
    @lovemaakhe 2 ปีที่แล้ว

    lol the shade in calling data scientists 'not-technical folks', great video!

  • @Techkhamun
    @Techkhamun 4 ปีที่แล้ว +1

    Very interesting! Keep up the great work!

  • @jaimecortesguevara5908
    @jaimecortesguevara5908 3 ปีที่แล้ว +10

    I really felt that part when you mention to "adapt" for the next 5-10 years, because that's when you need to keep investing in your self. Thanks for sharing your opinion

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว +3

      Absolutely, we will change jobs more frequently than people did in the past, so one needs to expect that and learn to be more adaptable! :)

  • @longinteger
    @longinteger 4 ปีที่แล้ว +3

    Just started my career as a data engineer in a professional environment as well and I can align with everything you say, Karolina!
    It's also very interesting because many people think Data Science is just that one single field but it can be divided in so many sub ones. I think you make the role of a DE clearer in this video!
    Great work. I am very excited to get to the same point and share the same experience with you.

    • @karolinasowinska
      @karolinasowinska  4 ปีที่แล้ว

      Hey, thanks so much! I'm glad to hear that you've embarked on your DE journey, I'll be curious to hear how you're finding it! :)

    • @norpriest521
      @norpriest521 4 ปีที่แล้ว

      How many tech stacks you need to have to become junior DE?
      Like you need to know Python, SQL or smth else?
      How many prerequisites should we have before go on to become junior DE?

  • @ngee4925
    @ngee4925 2 ปีที่แล้ว +1

    This was super helpful, thank you so much!

  • @adriang.4839
    @adriang.4839 3 ปีที่แล้ว +3

    Thank you!! I just entered university in the career of data engineer, your video helped me clarify certain doubts

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว +1

      I'm glad you found the video useful! Good luck at university! ;)

    • @divyanshshrivastav2469
      @divyanshshrivastav2469 3 ปีที่แล้ว

      Hi, i also want to study data engg in an university but didn't knw there is any good universities and procedures for it.. If u can send me an email regarding that knwldge will be helpfull..... Divyyansh05@gmail.com

  • @CarlaAnneQ
    @CarlaAnneQ 3 ปีที่แล้ว +4

    thank you for the insights! = )

  • @claudiusandika5366
    @claudiusandika5366 3 ปีที่แล้ว +2

    wonderful insights!! keep creating :)

  • @matheusmeirelles4189
    @matheusmeirelles4189 3 ปีที่แล้ว +2

    Thx! That's worked very well to me, cause I'm already work with data but I didn't know what I was lol.

  • @geoffreyreceveur4605
    @geoffreyreceveur4605 3 ปีที่แล้ว +1

    Thank you for the video. Very clear

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      My pleasure, I'm glad you found it useful! :)

  • @weirdinventor
    @weirdinventor 3 ปีที่แล้ว +3

    thank you so much , keep up !

  • @JorgeEstradaMty
    @JorgeEstradaMty 3 ปีที่แล้ว +1

    Hello, I come from the @CodinEric channel. Your content is very interesting too, thanks for sharing

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว +1

      Thanks and welcome! ;) Sorry for the delay in responding, I've just noticed your comment!

  • @kubasochacki9599
    @kubasochacki9599 3 ปีที่แล้ว +2

    Przydatny film dla studenta I roku Data Engineering!

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว +1

      O, super! Fajnie, że znalazłeś taki konkretny kierunek, na większości uniwersytetów jest tylko Computer Science. Powodzenia! :)

  • @fbalseiro
    @fbalseiro 2 ปีที่แล้ว

    Great content! It's funny you said that the thing that you like the least is to help others and here you're creating videos to help others on how to become a data engineer. Plot twist. 😅 Nice job Karolina 👏

  • @roopaljain1390
    @roopaljain1390 3 ปีที่แล้ว +3

    From a women in tech to a women in tech, great vdo. I can see the spark in your eyes mentioning your fav part.

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      Thanks! Great to hear from a fellow woman in tech! :)

  • @spideken123
    @spideken123 3 ปีที่แล้ว +3

    gorgeous as always miss karolina. very imformative.. thanks dear :)

  • @luisalamo2658
    @luisalamo2658 2 ปีที่แล้ว +1

    Thanks a lot from these kind of contents about Data Engineering, I'm a Java developer and I'm looking for conquer a new boundary because I'm a little bored doing micro-services and I want to try some related new topic. Greetings from Chile.

    • @karolinasowinska
      @karolinasowinska  2 ปีที่แล้ว

      I'm glad this was helpful, and best of luck! ;)

  • @DataAnalyticsIreland
    @DataAnalyticsIreland 3 ปีที่แล้ว

    Great explanation of this role, thanks for sharing!

  • @skiddi_botti
    @skiddi_botti 3 ปีที่แล้ว +1

    Welcome to my world! Best of luck 👍

  • @shaileshxjain
    @shaileshxjain 3 ปีที่แล้ว +3

    Karolina, You are a pure heart 💓.
    (Evidence at 11:18)
    Keep uploading videos and share us your experience about Data engineering and it's prospects.
    May you achieve a Million Subscribers soon.
    Kudos, from India ✌️✌️

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว +1

      Aw, I was just honest! Thanks so much! :)

  • @codingevelynmiami4230
    @codingevelynmiami4230 3 ปีที่แล้ว +1

    Appreciate your explanation.

  • @qasimzaheer1140
    @qasimzaheer1140 3 ปีที่แล้ว +1

    Very very good video! Awesome.

  • @yash1152
    @yash1152 2 ปีที่แล้ว

    11:46 - 12:19 - 12:37 - 12:57
    this part of the vid is the one i liked the most. not that it's something i already didnt had the idea of, but it felt nice to get reminded of the upcoming highs.

  • @newaccbub
    @newaccbub 3 ปีที่แล้ว +2

    Super helpful!

  • @vitelinha123
    @vitelinha123 3 ปีที่แล้ว

    Apache Airflow looks like a good fit for this sort of pipeline, really powerfull

  • @farwanqv
    @farwanqv 3 ปีที่แล้ว +1

    A good one to explain what data engineers are supposed to do

  • @freemind.d2714
    @freemind.d2714 3 ปีที่แล้ว +2

    Always pleased to see experienced people talk about they work and life.
    According to this video, I guess your work is much more focus on the pre-process the data rather than implementing and training the model right?

  • @ssbrunocode
    @ssbrunocode 2 ปีที่แล้ว

    Thank you for the good content! You are awesome o/

  • @ahsanmohammed1
    @ahsanmohammed1 3 ปีที่แล้ว +2

    Great video! Good info! Sincerely done! Appreciated! Thank you! Liked! Subscribed!

  • @mytechchannel2519
    @mytechchannel2519 3 ปีที่แล้ว +1

    Useful information, thanks🌹

  • @epicwhat001
    @epicwhat001 3 ปีที่แล้ว +3

    1. Bring Data in from different vendors. Check
    2. Data pipeline , data warehousing , and aws. Check
    3. Table-valued function , Procs and Views. Check
    4. Analytics, debug and Monitoring. Check
    5. I will Help but Not going to be nice about it. Check
    Thank you this gives me confidence to change my career to data engineering . I Am not a Data Engineer but a BI developer. my company is kind of like a start up so we don't have a data engineer so I ended up doing all the pipeline to create my SSRS and Tableau report.

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      Sounds like you're already doing data engineering! Awesome stuff ;)

  • @markslima1557
    @markslima1557 ปีที่แล้ว

    very cool thanks for posting - i'm learning about data analytics/python/sql wondering if data engineering is a better fit with my IT background

  • @seb6226
    @seb6226 3 ปีที่แล้ว +1

    Is it possible to create a video(s) about data scientist vs data analyst vs data engineer?
    Or data jobs vs machine/deep learning/ai vs computer science jobs maybe a series of videos?
    Could u do a video about how much ai/machine/deep learning could impact jobs and which jobs will likely become automated or disappear vs which jobs will remain strong in the next 5-10 years?
    Also based on your last name, its great seeing someone else from a Polish or European background!!

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      Awesome suggestions - noted, they land on my list! And yes, you're right, I'm Polish! :)

  • @muhammadsaad907
    @muhammadsaad907 2 ปีที่แล้ว

    Thank you so much :)

  • @ReenanOFC
    @ReenanOFC 4 ปีที่แล้ว +3

    Awesome video.

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว +1

      Thank you! More videos on data engineering are coming soon :)

  • @yorkstbd
    @yorkstbd 3 ปีที่แล้ว +2

    I'm as a junior BI developer, studying and working with tableau, and I'm missing the programming. I'm going to start working on some projects on my own and see if that's really what I want to do, because of everything you said I liked it so much! The bad part is that where I live there is only Senior vacancy for data engineering

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว +1

      Maybe some junior positions will pop up soon! Definitely go for what you enjoy!

    • @yorkstbd
      @yorkstbd 3 ปีที่แล้ว +1

      @@karolinasowinska

  • @buithanhlam3726
    @buithanhlam3726 4 ปีที่แล้ว +1

    This is very helpful!!!

  • @statisticianwordnpl4071
    @statisticianwordnpl4071 3 ปีที่แล้ว +2

    Really interesting video!
    I Don't Know why in my country there aren't girl that talk about it carrers. 😅
    Cya from Italy!

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว +2

      Thank you! Tech careers seem to attract guys a bit more than girls, but I think it's slowly changing now, and many girls do coding! :) Ciao!

  • @TheCaesius29
    @TheCaesius29 3 ปีที่แล้ว +12

    i'm thinking of a career shift to a data engineer. thank you for this :)

  • @bryanortiz9753
    @bryanortiz9753 2 ปีที่แล้ว

    Beautiful, funny, smart and data engineer like me! IM IN LOVE!!🥰

  • @ihgnmah
    @ihgnmah 3 ปีที่แล้ว +1

    Hi Karolina, your content is amazing and informative. I just curious that what to learn to get started with data engineering. Thank you

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว +2

      Thank you so much! I'm soon going to release a series of videos on data engineering, so stay tuned! :)

    • @ihgnmah
      @ihgnmah 3 ปีที่แล้ว +1

      Karolina Sowinska That sounds interesting, Thank you

  • @irishuang5294
    @irishuang5294 3 ปีที่แล้ว +1

    Thanks for the video and it's rly nice to see another female data engineer voicing their thoughts. I've been a data engineer for officially, I'd say 1.5 years and I really enjoyed the net result of ups and downs. Data engineers understand business better than software engineers, and understand tech better than data analysts/scientists. So to me, when I look at it as a role in translating data for business, I find it quite impossible to be obsolete in the next 5-10 years UNLESS the average data literacy within a company has increased (making better and more careful choices on data inputs) or people turn to machine-generated data more than human-generated data (or when NLP and Deep Learning married for good to successfully "guess" the right data). Otherwise the business would always need to pull in a data engineer to scale their data while preserving its correctness.

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      Hi there, nice to meet a fellow female data engineer! :) That's a very interesting and convincing perspective - perhaps you're right (and I definitely hope so!)

    • @davidthewise6617
      @davidthewise6617 2 ปีที่แล้ว

      @@karolinasowinska
      You look beautiful 😍😍😍

    • @sligon00
      @sligon00 ปีที่แล้ว

      @irishuang A very insightful comment , thanks for being so clear and concise about what goes on behind the scenes in the world of data.

  • @blah-blah-blah4715
    @blah-blah-blah4715 2 ปีที่แล้ว

    To your last point about standardization and automation, I'd honestly love if Data Engineering and ML Engineering came closer once this happened. DE's already have to deal with all the requests from MLE's, might as well just put them in the same room.
    I see both getting closer to being abstracted away to standardized tools within the broader software engineering ecosystem. More successful and matured design patterns just need matured first. But once this happens, I very well could just see software engineering jobs focused on ML and/or Data experience being the new norm.

  • @victorpinasarnault9135
    @victorpinasarnault9135 3 ปีที่แล้ว +1

    2021 and still worthy.

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      Not much has changed since I recorded this! :) Perhaps the accelerated move to the cloud!

  • @ErickHernandez-ri2ft
    @ErickHernandez-ri2ft 2 ปีที่แล้ว

    It's great to know more about my degree (I am data enginerring student, I am in sixth quarter) and I think, oh my gosh, I need to learn a lot of topics because my teachers don't teach me but I am learning on my own. So, the question is: What are the main topics as a data engineer needs to know? and can you recommend me any page to learn about data engineering? More specific topics such as: Big data, distributed systems and data mining. Greetings from Mexico!

  • @badass_omelette5166
    @badass_omelette5166 3 ปีที่แล้ว +1

    I have been enjoying data engineering and cloud engineering over data science recently

  • @daringvishalverma7983
    @daringvishalverma7983 4 ปีที่แล้ว +1

    Thanks a lot. You explained everything very well.I'm starting my career as a Data Engineer, so wish me luck.

  • @medisettycharan9152
    @medisettycharan9152 3 ปีที่แล้ว +3

    Many thanks for the information, it’s really helpful. Can I please ask you with whom will the data engineers will be interacting most of the time. And what kind of information will be exchanged.

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      My pleasure! Mostly with data scientists and data vendors:)

  • @troyfrericks-wti2510
    @troyfrericks-wti2510 3 ปีที่แล้ว +2

    Would love to hear your opinion about what "Data Architects" do.

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      Very similar job to what Data Engineers do, but I guess Data Architects are typically higher up in the hierarchy/they're more experienced, so they can decide on the architecture, and not just build pipelines

  • @gracechan3039
    @gracechan3039 3 ปีที่แล้ว +5

    I work for an IT consultancy that primarily migrates businesses databases to the cloud. The quantity of work seems endless.

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      100%, I can imagine that, it's a very trendy thing to do at the moment!

    • @akeemadedigba8279
      @akeemadedigba8279 3 ปีที่แล้ว

      working on cloud computing makes life easy and flow

  • @ofekezra7567
    @ofekezra7567 3 ปีที่แล้ว +1

    Amazing video. simple and straight to the point. Karolina can you please explain if degree is needed to became a data engineer ? and if yes - what can i learn in home before (which skills)

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      Hi, thanks so much! A degree in a technical subject is a plus, but not a strict prerequisite. I've just released a free data engineering course here on youtube, so if you want to learn some relevant skills, check it out :) th-cam.com/video/dvviIUKwH7o/w-d-xo.html

    • @ofekezra7567
      @ofekezra7567 3 ปีที่แล้ว +1

      @@karolinasowinska thank you so much

  • @jae11011
    @jae11011 3 ปีที่แล้ว +1

    Thanks a lot for the intro! It helps a lot for me as I am thinking of making data engineering as my career goal.
    For the last part, if data engineer becomes less demanding in the future, may I know personally which way would you recommend a data engineer to go, software engineer or data science?

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      I believe you should go for what's most suited to your personality and interest :) Engineering vs science/research is the key question here.

    • @sumittamang3548
      @sumittamang3548 2 ปีที่แล้ว

      If you had to suggest an undergrad student between data science and data engineering, what would you suggest?

  • @denismerigold486
    @denismerigold486 4 ปีที่แล้ว +6

    being a data engineer is cool.
    But deep learning is much more interesting.
    Thank you!

    • @karolinasowinska
      @karolinasowinska  4 ปีที่แล้ว +5

      I agree that deep learning is more intelectually challenging!

    • @norpriest521
      @norpriest521 4 ปีที่แล้ว +2

      Can one work as DE the entire career without having to move onto other relevant areas like MLE or DS later on?
      Say I love DE job so much, can I stick to it forever?
      What will be career path for DE?
      I don't like machine learning and data scientist role 😂

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว +2

      @@norpriest521 Hi, yes, you can! Typically the career progression is such that you eventually become the person responsible for the overall data flow in an organisation, eg. the Head of Data Engineering.

  • @DAJakaRedAries
    @DAJakaRedAries 3 ปีที่แล้ว +3

    Thank you for th enice video. Subscribed :)
    I'm wondering if data engineering job is more demanded comparing to data analyst jobs?
    Do you need to learn Scala in order to manage ETL pipelines? (I know Python/SQL already)

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว +1

      I would say that data engineering is in more demand. You don't necessarily need Scala - the particular technologies depend on the company that you'd be working for. Python + SQL is definitely a base. I'd recommend learning about data modelling rather than Scala, because it's more universal :)

  • @Fruitslycer
    @Fruitslycer 3 ปีที่แล้ว +1

    Probably gonna have to merge Data Analysis, Machine Learning theory wise like MSc paper reading, implementation wise, like X amount of frameworks, C++ production code, and Data Engineering Pipeline/MLOps, into a single title, and know all those stuff, to have a job in the future. I feel like the market has been a lot softer on older guys, and us young new ones are gonna need to raise 3x of their for same salary (or less). Also older guys are far away from being nice to young ones imho, and the technical debt will become greater and greater too!

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      Perhaps that's true. One benefit that the younger generation of developers have is knowledge/information availability!

  • @Anandhusreekumar
    @Anandhusreekumar 3 ปีที่แล้ว +1

    Thanks for this video.entire TH-cam is full of data science videos.im looking for data engineering channels

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว +1

      Here is one! ;) Thanks for stepping by, and if you've got any video suggestions, please do let me know!

  • @datawolf2318
    @datawolf2318 3 ปีที่แล้ว +1

    I'm an unspoken data engineer lol. I didn't know what ETL was...read about it and realized I have experience in it lol.

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      Haha I can completely relate - I wrote my Master's thesis on this topic, and only later did I realise that was ETL!

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      @Jared Dudley or data engineering specifically! ;)

  • @harshavardhan4523
    @harshavardhan4523 3 ปีที่แล้ว +1

    Wow what a great video.....It was funny even thank you

  • @FrancoisLeCoguiec
    @FrancoisLeCoguiec 3 ปีที่แล้ว +1

    Karolina: Subscribed! Great video, really nice and simple down-to-earth explanations! One point though: I think at one time you talk of HADOOP and you seem to imply that HADOOP has to be on premise? Technically I think that is not true. So, your company might have their own on-premise HADOOP cluster(s), but it is also possible to get "HADOOP as a service" in the cloud, on AWS or other cloud providers. Still, that makes no big difference is what you explained, since the activities you described apply no matter if on premise or in the cloud, but I thought I would point it out.

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      Yes, you're right, I only later realised that Hadoop might as well be done in the cloud! Thanks! :)

    • @irishuang5294
      @irishuang5294 3 ปีที่แล้ว

      Is there an increasing or existing trend of hosting Hadoop on cloud? I also feel Hadoop tends to be an on-premise thing. My company actually has been trying to migrate from on-premise Hadoop to Azure

  • @omkarbhiwande7723
    @omkarbhiwande7723 3 ปีที่แล้ว +2

    Can you make a video on how should one go about learning data engineering?

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว +2

      That's a highly requested video - I think I'm going to make it very soon! Stay tuned! :)

  • @alejandrocarvajal9068
    @alejandrocarvajal9068 2 ปีที่แล้ว +1

    Another great video, You seem to have a lot of experience (or at least that's what it looks like) on scalability systems. I have one question for you, how much do you charge for a tutoring session or a workshop meeting ?

    • @karolinasowinska
      @karolinasowinska  2 ปีที่แล้ว +1

      I don't have much experience with them, I'm afraid:(

    • @alejandrocarvajal9068
      @alejandrocarvajal9068 2 ปีที่แล้ว

      @@karolinasowinska Well, don't worry about it. I'll just hope success on your journey.

  • @candyfloss184
    @candyfloss184 4 ปีที่แล้ว +2

    Suppose a company wants to have an erp from oracle to sap, now past data is in oracle db, future will be in sap db, now CEO of the company would like to have all data in same format report to make a year on year comparison, then you would need data engineers.

    • @karolinasowinska
      @karolinasowinska  4 ปีที่แล้ว

      Definitely! Data engineers are useful people to have on board! :)

    • @candyfloss184
      @candyfloss184 4 ปีที่แล้ว

      Try to learn sap BW if you are so much interested to work in data engineering. But you could gain lot more insight if you would understand how business works. But then you donot appreciate economics much. So I think technical role is better for you.

    • @karolinasowinska
      @karolinasowinska  4 ปีที่แล้ว

      @@candyfloss184 Economics has little bearing on reality. Thanks for your suggestions though.

    • @candyfloss184
      @candyfloss184 4 ปีที่แล้ว +1

      Karolina Sowinska I have always like macro economics. Microeconomics sounded a bit technical and it was difficult. Development economics and international economics too were interesting to me.

  • @cosmosbrasil5403
    @cosmosbrasil5403 4 ปีที่แล้ว +4

    Half a year? You know so much apparently for that period hahah. Anyways, I recently got my Physics degree and worked with Python in a few coursework and final year project. Next, I'll do a Masters in London but intend to start my career in the 'big data world' and I definitely consider data engineering (but mostly data science :D ). That video was quite helpful (although I don't know 10% of all programs you've mentioned lol ). Anyways, thumbs up! Thanks Karol. Ah, I hope I never need your help to install a driver or smth ahhaha

    • @karolinasowinska
      @karolinasowinska  4 ปีที่แล้ว +2

      Physics, nice! Awesome foundation for data science! Oh, I also did a Masters in London - have you started looking at which unis you'd want to apply to? P.S. Data Science is probably more interesting (if you like statistics more than technology/coding) P.S.2 I also hope you won't need me to install drivers 😂

    • @cosmosbrasil5403
      @cosmosbrasil5403 4 ปีที่แล้ว

      @@karolinasowinska I already applied to 3 unis, I was lucky enough to have 3 offers from Queen Mary, Imperial and UCL. I'll think I'll stick with UCL, although there's still a small chance to change my mind. I'm glad to know that Physics is a good foundation. I love almost everything related to mathematics, coding, statistics, etc (a true geek, ahah), but have no experience in data industry : / a bit scared of not finding any job. About P.S.2: if I do, I prefer to ask you to make a video about it ahahah. Thanks Karol.

    • @karolinasowinska
      @karolinasowinska  4 ปีที่แล้ว +1

      Wow, congratulations, that's a lovely set of offers! I'm an Imperial graduate myself - if I can give you my biased opinion, it was a very good course (but also very, very tough). Don't worry, the industry loves geeks haha, you'll surely find a great job! A video about drivers you're saying... I'd give this one a long think! 😅

    • @cosmosbrasil5403
      @cosmosbrasil5403 4 ปีที่แล้ว +1

      @@karolinasowinska ahahaha is better doing vids about your relationship with Apple ahhah. Imperial graduate, uow. Congrats!! I hope to go well through this toughness and start working soon (eager to learn :D ). Have an amazing evening. :D :D

  • @muratisik6956
    @muratisik6956 4 ปีที่แล้ว +1

    You make a boring subject sound interesting, to me, so my compliments! 👍🏼
    Did you receive the MacBook Pro 16 inch yet? Like it? I’m waiting for a MacBook Pro with WiFi 6 and a much better camera.

    • @karolinasowinska
      @karolinasowinska  4 ปีที่แล้ว +1

      Thank you so much :) Yes, I've received it. So far, no issues, I'm liking it!

  • @captainphysix
    @captainphysix 3 ปีที่แล้ว +1

    Hello, just subscribed, AMAZING content! Very helpful! May I ask which country you are from, because sometimes you sound british or french or even russian!

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว +1

      Wow, I sound like I'm a citizen of the world! Which is what I like to think of myself! But I'm originally from Poland :)

  • @robinsoncastano1594
    @robinsoncastano1594 3 ปีที่แล้ว

    Hi! Could you please talk about Scala programming language.

  • @pouinichan
    @pouinichan 3 ปีที่แล้ว +1

    I'm a BI analyst who got approached for a data engineer job. I'm really hesitant about doing the move. I'm interested about learning more about the architecture and big data field but I'm afraid that I won't like it and won't be able to go back in an analyst position 🥺🤯

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      If you enjoy being an analyst, don't necessarily jump at the opportunity!

  • @derekx3699
    @derekx3699 3 ปีที่แล้ว +2

    Which country are you working at? Your video is the clearest I've ever watched on DE role. I am a full stack developer(junior, have just 1 year experience), so can I ask how do you see the career path between DE and full stack developer? Thank you

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      Thanks so much! I live in the UK. I'd say DE is always a backend role, whereas full stack can involve some backend, and some frontend (like building GUIs). One more difference is that typically DE is responsible for bringing the data to a company for researchers to analyse. On the other hand, if SE does some back end work, this is typically to deal with data that their application generates. Of course everything will depend on a specific job. :)

  • @esm2000
    @esm2000 3 ปีที่แล้ว +1

    Thanks for the video and the insight. Do you think becoming a data engineer is feasible for someone coming from undergrad who's done two data science-oriented internships. I want to make a transition from data science to data engineering and the upcoming school year is my last before I earn my degree in CS.

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว +1

      Yes, absolutely, you can! That's exactly what I did. I had an internship in data science first, and after that I transitioned to my field :)

  • @RobertoMartinez-sx6ih
    @RobertoMartinez-sx6ih 3 ปีที่แล้ว +1

    You look a lot like Jodie Foster. thanks for the video, it was great.

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว

      Haha I've heard that before (many times!) Thanks :)

  • @Yok685
    @Yok685 3 ปีที่แล้ว +7

    hahah "each data vendor, they are not nice people, they want data in certain format" hahahah true! I can relate so much

  • @logic9948
    @logic9948 ปีที่แล้ว +1

    9:40 - that smile hit me, you are so beautiful ❤

  • @mojloginjuzzajety4071
    @mojloginjuzzajety4071 2 ปีที่แล้ว

    It is really nice to listen to You, such a distance and so nice reflections. More over, You are just beautiful.

  • @milantripathi720
    @milantripathi720 3 ปีที่แล้ว +2

    well in my country every single employee in most AI company is ML Engineer/data analyst/data engineer/Data scientist/dev ops....full stack data scientist

    • @karolinasowinska
      @karolinasowinska  3 ปีที่แล้ว +1

      I can imagine that! Where is that, by the way? :)