Mage AI Pipeline - API to BigQuery
ฝัง
- เผยแพร่เมื่อ 5 ต.ค. 2024
- This video is a walkthrough for making a simple Mage AI Pipeline that pulls from an API and loads the data into BigQuery. It is very cost efficient using Instance Schedules.
Github repo for Docker and Docker Compose Install:
github.com/Mic...
Partitioning and Clustering Google Documentation:
cloud.google.c...
Install Docker and Docker Compose on GCP VM Manually:
• Installing Docker and ...
Chicago Data Portal Sign Up Walkthrough:
• How to Retrieve Data f...
Videos Explaining Partitioning and Clustering MUCH better than I can:
• DE Zoomcamp 3.1.2 - Pa...
Big thanks to Xiaoyou Wang in the Mage Slack for helping me through the timestamp conversion. :-)
Thank you, Michael!
Indeed, makes for a very good starter template for projects.
Thanks for this Michael!
Thanks, I'll use this as a template!
Very useful, thanks for sharing. Did you have to install python and other packages including Bigquery separately? or using the docker installer?
Just had to install docker and then use the mage docker image
Hello, Michael!
Thanks for the amazing content! I have a few questions:
1. The Mage Ai docs recommend deploying the application using their official terraform scripts and cloud run when working with GCP. Any particular reason you used a VM? Which approach should be more realiable / cost effective (really hard to be cheaper than in your video though).
2. Does the mage built in data exporter to bigquery automatically enforces schema, according to the pandas dataframe dtypes? Or is there a way to create a schema dictionary like one can do when using the regular BigQuery Python API?
Thanks a lot!!
When I made this video there wasn't really much content around Mage. That why I made it. I think that the way they use cloud run with the setup now (not super familiar with it) would come in at about the same price as just using the VM. Probably a little higher. Using just a VM is a great way to get familiar with a tool and for personal projects. Using their Terraform Setup is taking a big step towards "Production Ready", but not quite there. Also, it's awesome to read through their setup to learn about Terraform and Mage and how the different components interact.
As for the Big Query question I honestly don't know. I'd jump into their Slack Channel and ask there. Tommy and team were always ninja quick with responses when I ran into issues. Hope that helps and Thanks for watching. 🙂
@@dataslinger6379 Thanks a lot for the quick response! Keep up the amazing work.
@@dataslinger6379 Thanks a lot for the response, Michael! Keep up the great work
Thanks for this