The best alternative to Airlfow!? Mage.ai
ฝัง
- เผยแพร่เมื่อ 8 ก.ค. 2024
- The guys from Mage.ai asked me to check out their way of building ETL, ELT and Streaming pipelines.
So, here's what I found out. Spoiler: It's really fun!
▶︎ Checkout Mage.ai: www.mage.ai
▶︎ Learn Data Engineering Academy: learndataengineering.com/p/ac...
Jump marks:
0:00 Introduction
1:00 Setting up Mage
1:42 Scaling
02:23 The user interface
03:01 ETL pipeline Data Loader
03:36 Data Transformer
04:00 Data Exporter
04:51 Data exploration with charts
05:50 Arranging pipelines
07:28 Running tests
08:10 Scheduling pipelines
09:13 dbt and Apache Spark
10:16 Summary
#dataengineering #datascience #bigdata
![[안방1열 풀캠4K] 베이비몬스터 'FOREVER' (BABYMONSTER FullCam)│@SBS Inkigayo 240707](http://i.ytimg.com/vi/_2HL-ogzzBU/mqdefault.jpg)
I've been using mage for about a week now. I'm blown away. It is so well thought out. Love how easy it is to get setup. Also the community has been great. Tommy is very responsive and helpful!
wow, truly an eye-opener! It seems like a powerful alternative to Apache Airflow. Thanks for all you do, and keep the insights coming. Cheers! 🚀🙌
Thanks! It's a really interesting tool
This is such an awesome overview!!! Thank you so much for sharing!
Amazing video! Thank you so much for this overview!!
Keep doing what you do!
Hi, Thanks for this great video. Are we bound to code in their default UI?
Really cool video Andreas!
DAM ! it's...; wow it's a great vidéo !
I'll try it !
Bravo for being so clear
Thanks! Let me know if you know something you want me to review
If I have multiple projects, how do I go to a specific project I want to resume working on?
That’s awesome! Can you talk about Kubernetes and scalability in detailed session?
Thank god for an airflow replacement. Even, data factory is better than airflow, we needed an open source alternative
My feeling is that we will see a lot of movement on the tools front over the next years. Demand is constantly growing
Author! Let me know plz...
Can I add "Data Exporter" block after "Data Transformer" using trial account. I am not been able to find any way to add that after transformer.
However If I wish to add only "Data Exporter" block with out any upstream then I can do that without any problem.
Thank you for the video Mr Andreas, please can we do spark jobs in the transformation connector?
Thanks.
I'd like to see and better understand more on what the potential of this tool are if you want to use CDC or manage streaming pipelines
I'll ask Tommy the CEO. I'll have him on my live show tomorrow
can i deploy it on a dedicated server?
Great video. Thank you for sharing. Overall do you prefer Mage or Airflow?
Depends on what you prefer. Mage’s top 3 design principles are: Easy developer experience, Engineering best practices built-in, Data is a first-class citizen.
So in Mage, it works only with data and so it comes with a lot of native features that support data operations and passing data between blocks.
hmmm... but how to setup a visualization of multiple pipelines... because 1 is dependant on the other.... still trying to find out
Not sure. Think you can't
Do you have rough estimates on how much a single pipeline costs to run on AWS, say?
I did add a comment, but it was auto-deleted by TH-cam Andreas 🙁
I actually saw that there were 4 comments, but I couldn't find it. Now I know why. Super annoying!
@@andreaskayy detected as potential spam I think! Oh well.
is there any data catalog that integrate with Mage?
Not yet, but that’s on the roadmap to integrate with DataHub.
My Transformer takes eternity to run! Really would appreciate some advice!
Get in touch with the guys at Mage.ai. They have a discord server where they 100% can help you :)
Hi All, what do you think about the adoption of Mage in the professional environment so far? I have been following for a while and to my feeling the real growth in user bases seems a bit lacking.
How do you measure that?
Stats as of March 2023:
6.9k+ GitHub stars
16.5k+ LinkedIn followers
5.2k+ community Slack members
1.5M+ Docker downloads
33k+ developers using Mage
17.2M+ runs
1k+ companies using Mage in production
Doesn't it support node, bun
Not yet, but TypeScript blocks will be supported soon.
influencer should clearly mentioned if the streams are sponsored or not
Did, there is a clear notification in the beginning. I marked that on TH-cam. I also only do partnerships with companies who create products I like. This here is a prime example for that
@@andreaskayy can you please clarify if this is sponsored? Being asked isn’t the same as being sponsored. Mage looks great, regardless of sponsorship or not.
Ok, I see the “Includes paid…” banner popping up at beginning now, thanks.
Bruh why'd you steal my thumbnail?