Can you please prepare a video vor documentation which has step by step procedure to setup dbt on composer. I am trying with pypi but facing dependency issue
Hey Kumar, can you explain what issues you were facing? The pip install is done outside of composer on a separate docker build, which is independent of composer. So it’s no different to if you just do it on a blank Python virtual env, if you can explain which step you got stuck I can help
Hi there, git isn’t directed integrated with composer. The way it works is you deploy DBT by packaging it in a docker container and push it to Artifact Registry (in the past GCR) then you pull the image when the dag runs (image is cached between runs if you don’t use :latest but a specific version of the tagged image). Composer will automatically get access to the image sorted in Artifact Registry if it’s in the same GCP project.
It's quite beneficial and mirrors real-life scenarios closely.
This is gold! Thanks for sharing ⭐
Glad you found it useful Rahul 👍
This is Godly, thank you so much
Can you please prepare a video vor documentation which has step by step procedure to setup dbt on composer. I am trying with pypi but facing dependency issue
Hey Kumar, can you explain what issues you were facing? The pip install is done outside of composer on a separate docker build, which is independent of composer. So it’s no different to if you just do it on a blank Python virtual env, if you can explain which step you got stuck I can help
How is the authentication between git and the airflow composer?
Hi there, git isn’t directed integrated with composer. The way it works is you deploy DBT by packaging it in a docker container and push it to Artifact Registry (in the past GCR) then you pull the image when the dag runs (image is cached between runs if you don’t use :latest but a specific version of the tagged image). Composer will automatically get access to the image sorted in Artifact Registry if it’s in the same GCP project.