Is there native support for mapping time based partitions to static partitions defined like "today", "rest of month", "rest of year", "rest of history"? This is a common setup for power bi datasets, which can be represented as assets in dagster. Would be nice to take advantage of auto materialize policies.
Dagster does not natively support mapping time-based partitions to static partitions like "today," "rest of month," "rest of year," and "rest of history" directly out of the box. However, you can achieve similar functionality by defining custom partitioning schemes and using the appropriate partition mappings. You can define custom partitions using StaticPartitionsDefinition for static and TimeWindowPartitionsDefinition for time-based partitions.
All the code for the demos from the deep dives are in this repository ( github.com/dagster-io/devrel-project-demos )! This one in particular is in the partitions directory.
I like what you folks have done with this product.
Thanks - there is a lot more in store coming next month!
+1 I am rooting for you guys. Thank you for all of your hard work
We appreciate it - thanks @@quinnherden !
@@dagsterio Sure, thanks for making these sessions, these are really helpful.
Is there native support for mapping time based partitions to static partitions defined like "today", "rest of month", "rest of year", "rest of history"? This is a common setup for power bi datasets, which can be represented as assets in dagster. Would be nice to take advantage of auto materialize policies.
Dagster does not natively support mapping time-based partitions to static partitions like "today," "rest of month," "rest of year," and "rest of history" directly out of the box. However, you can achieve similar functionality by defining custom partitioning schemes and using the appropriate partition mappings. You can define custom partitions using StaticPartitionsDefinition for static and TimeWindowPartitionsDefinition for time-based partitions.
Integrate or replace slurm?
@dagsterio Do you have the source of the demo avaialble somewhere?
All the code for the demos from the deep dives are in this repository ( github.com/dagster-io/devrel-project-demos )! This one in particular is in the partitions directory.
@@dagsterio unfortunately it is private/ link is broken.
@@Jahaniam Sorry, the final parenthesis got included by TH-cam in the URL - try this:
github.com/dagster-io/devrel-project-demos