This looks amazing! We're are running in to so many issues and headaches trying to use Airflow the last year or so. Hopefully this can solve most (if not all) of our problems
Can Dagster be used to orchestrate a spark streaming YARN job that pulls data from Kafka and writes to HDFS?.. the idea is if the spark streaming job queues and it can be monitored/alerted/detected and restarted automatically by Dagster?
Thanks for asking. TH-cam is a great sharing platform, but sadly it monetizes all posted videos. We at Dagster have looked to prevent Ads from showing on our videos, but from what we can see, this is no longer an option. If you know otherwise, please let us know how to do that.
Yep, the framework has been evolving rapidly. Luckily we just released a set of courses: Dagster University - check it out. courses.dagster.io/courses/dagster-essentials
![What is Dagster? Asset Based Orchestration [2hr full course]](http://i.ytimg.com/vi/Xe8wYYC2gWQ/mqdefault.jpg)
![What is Dagster? Asset Based Orchestration [2hr full course]](/img/tr.png)
This looks amazing! We're are running in to so many issues and headaches trying to use Airflow the last year or so. Hopefully this can solve most (if not all) of our problems
This looks awesome. Seems like it does for data engineers Like what next/vercel did for web developers.
Yes I’m excited. Thanks
Is the dbt, ml, pandas, jupyter all working on different containers? how are dependencies being managed?
Can Dagster be used to orchestrate a spark streaming YARN job that pulls data from Kafka and writes to HDFS?.. the idea is if the spark streaming job queues and it can be monitored/alerted/detected and restarted automatically by Dagster?
hey Dagster, how to add these flags on each asset? Like API, ML Tool, pandas...
Is FreshnessPolicy being deprecated?
Why is this video monetized?
Thanks for asking. TH-cam is a great sharing platform, but sadly it monetizes all posted videos. We at Dagster have looked to prevent Ads from showing on our videos, but from what we can see, this is no longer an option. If you know otherwise, please let us know how to do that.
Dagster documentation is poor, as a developer the learning curve with Dagster is much longer than Prefect.
Yep, the framework has been evolving rapidly. Luckily we just released a set of courses: Dagster University - check it out. courses.dagster.io/courses/dagster-essentials
True