วีดีโอ

tinyurl.com/30daysdagster Day 17 looks at how to implement conditional pipelines in Dagster.

tinyurl.com/30daysdagster Day 16 talks about multi-assets, the case where one function call should create more than one asset; or in other words, when running one thing should lead to more record keeping / metadata / lineage updates on the dagster side.

tinyurl.com/30daysdagster Day 14 is all about parallelization and concurrency. What should you do in an orchestrator vs in Python code? How can you dynamically create a bunch of (parallel) assets using an asset factory? This video covers these topics and more!

tinyurl.com/30daysdagster Day 13 discusses how to move towards incremental data processing, with an example using daily partitions in Dagster. We also discuss how partitions allow you to easily backfill historic data.

tinyurl.com/30daysdagster This video looks at how Dagster can be used to invoke pre-existing data processing code using Dagster pipes, similar to Airflow Operators.

tinyurl.com/30daysdagster This video looks at adding data quality checks to data pipelines.

tinyurl.com/30daysdagster Today's challenge discusses how to make pipelines more "DRY" (less repetitive, easier to refactor) by encapsulating system interactions with resources.

tinyurl.com/30daysdagster This video shows how to parameterize pipelines with asset config.

tinyurl.com/30daysdagster This video shows one way to implement an event driven pipeline in Dagster using a sensor. SPOILER: I ended this video and the code had a bug! I fixed the bug in the solution, but watch the video and see if you can guess what the error was.

tinyurl.com/30daysdagster This video discusses various ways to pass data between Dagster assets.

tinyurl.com/30daysdagster This video shows how to add metadata and logs to Dagster. Along the way, enjoy some very hilarious real-time Python debugging.

tinyurl.com/30daysdagster This video shows how to update a Dagster asset to load a csv file into a data frame. As part of the process, LLM code generation is attempted, and the VS Code Python debugger/refactor tools are briefly used.

tinyurl.com/30daysdagster This video shows how to make a pipeline more robust to transient errors by introducing automatic retry conditions.

tinyurl.com/30daysdagster This video shows how to approach declarative automation for the case where you have 3 steps in a pipeline with complex SLAs for each. (Also, a sneak peak of Holiday Daggy!)