Bro, your channel is such a well-rounded one on data engineering, and I've really learned a lot, especially from the videos where you walk through the coding process. It would be awesome to see a video on building a good resume for an entry-level data engineering job. I'm sure it would help a lot of us in the audience get better and more confident in our job search.
Thanks for the good overview with demo. Quick question. How to handle IF THEN logic within Dagster Graph? Say, based on move type route to different downstream data assets. Ex. If it is Spanish movie, route to a particular asset. These downstream assets generate completely different data schema.
Hi Jay, Im from Latin America, im going to start this month a career related to Web developing and design, I speak english really well and got XP as a field engineering in many places, how can i apply to a job in US/EU from SouthAmerica? Any tip will be considered, Ty!
In the video at this timestamp th-cam.com/video/Xe8wYYC2gWQ/w-d-xo.htmlfeature=shared&t=675, I noticed you're using `setuptools` in `setup.py` instead of relying on a `requirements.txt` file. I'm curious-what are the advantages of using `setuptools` over the more common `pip` or `pipenv` approaches? Many of the packages you listed seem to be available via `pip`, so it seems to add a bit of complexity. Could you explain the reasoning behind this choice? By the way, I’m not criticizing the approach, just genuinely interested in understanding the benefits.
Hey this is a really interesting question `requirements.txt` lists your packages you want to install using pip, but it doesn't describe how the package is installed. The filename is arbitrary, and u can even call it `another_requirements.txt` and run pip install -r `setup.py` and `setuptools` is an alternative approach that installs pip dependencies + how you define a python package (name, metadata, packages, etc). It's more narrow in the sense that you're building for a single project only, and it's meant for redistributing your software on other machines, whereas `requirements.txt` is more suited for development environments.
Dagster is amazing but rather complex. This course is amazing!!
Pretty excited to dive into this one!! Had been waiting a video from you 😃
Thanks for the wait man! Hope u'll learn something. I'm pretty excited about Dagster
Hi, thanks for the top view of the Dagster, do you plan to do the next one, but how to test the whole pipeline or their elements?
Honestly still thinking! If there's enough interests on more Dagster videos. Hard to make videos when I'm working full time 😅
Bro, your channel is such a well-rounded one on data engineering, and I've really learned a lot, especially from the videos where you walk through the coding process. It would be awesome to see a video on building a good resume for an entry-level data engineering job. I'm sure it would help a lot of us in the audience get better and more confident in our job search.
Hmm job hunt is rough now. I’m not the best in resume building so wasn’t sure it will be helpful to others. Glad you brought it up.
Thanks for the good overview with demo.
Quick question. How to handle IF THEN logic within Dagster Graph?
Say, based on move type route to different downstream data assets.
Ex. If it is Spanish movie, route to a particular asset. These downstream assets generate completely different data schema.
You can use multi-asset with is_required field equal to false docs.dagster.io/concepts/assets/multi-assets#conditional-materialization
What are the versions of pandas and matplotlib used in your project?
Hi Jay, Im from Latin America, im going to start this month a career related to Web developing and design, I speak english really well and got XP as a field engineering in many places, how can i apply to a job in US/EU from SouthAmerica? Any tip will be considered, Ty!
Hey man, can you make a video on using python loggers in dagster instead of having to use context loggers?
Hmm good suggestion, will think about Python loggers in dagster for future videos
Could you show us your terminal setup? Really like the layout of it
Yea for sure. I'm using pure zsh terminal by sindresorhus. For vscode, I use Dracula theme which is my fav
🙏
In the video at this timestamp th-cam.com/video/Xe8wYYC2gWQ/w-d-xo.htmlfeature=shared&t=675, I noticed you're using `setuptools` in `setup.py` instead of relying on a `requirements.txt` file. I'm curious-what are the advantages of using `setuptools` over the more common `pip` or `pipenv` approaches? Many of the packages you listed seem to be available via `pip`, so it seems to add a bit of complexity. Could you explain the reasoning behind this choice?
By the way, I’m not criticizing the approach, just genuinely interested in understanding the benefits.
Hey this is a really interesting question
`requirements.txt` lists your packages you want to install using pip, but it doesn't describe how the package is installed. The filename is arbitrary, and u can even call it `another_requirements.txt` and run pip install -r
`setup.py` and `setuptools` is an alternative approach that installs pip dependencies + how you define a python package (name, metadata, packages, etc). It's more narrow in the sense that you're building for a single project only, and it's meant for redistributing your software on other machines, whereas `requirements.txt` is more suited for development environments.