dlt is better. light years beyond sling imo, and they have a better and more commited and active team. dlt is like the dagster of the lightweight integration libraries.
Does Dagster work on premises as well as the cloud? Are any features on cloud not available on premises? My company is moving back to on premises and doesn’t want to use SSIS.
Hi Joshi. Dagster is an open-source project and all the capabilities that Pedram describes in this video are available in the open-source solution, which you can self-host locally or on-prem. Dagster Cloud offers additional capabilities, which are detailed at dagster.io/cloud or on the pricing page here: dagster.io/pricing
If the requirement is to get the data from S3 files into a BQ table but perform some validations on those files before inserting into the table, how would we do it with Embedded ELT? We are using Dagster OSS heavily and looking to use embedded-elt for getting data from files, tables and APIs..
Hey Abishek! In your case, would you be able to represent the S3 files as source assets first, adding asset checks onto those, and running Embedded ELT only if those asset checks pass? Sling currently (afaik) is heavily focused on doing ingestion well, so you can defer to the rest of the Dagster ecosystem (such as asset checks) for validations.
@@AbhishekAgrawal-dv1id we've found that dlt is a powerful framework for ingesting from APIs and it's definitely mature enough for production settings. I'll also say that neither Sling's or dlt's integration currently allow for creating asset checks in-flight during ingestion. Instead, have you thought about ingesting the files into a quarantined dataset first using whichever tool you'd like, applying asset checks to that, and then moving that data to your real "analytics-ready" BQ datasets once you've vetted the data? This way, you can do ad hoc analysis to understand why the data failed data quality tests easily, but also keep it isolated from your production analytics.
Yeah, I am also leaning towards doing something like this. Thanks for this, Tim. Would you suggest using a similar approach to pull data from a different database? We'd still need to run minor validations on the incoming data, though. Would dlt help here at all?
TIL of Sling - thanks Pedram!
dlt is better. light years beyond sling imo, and they have a better and more commited and active team. dlt is like the dagster of the lightweight integration libraries.
Is dlt embedded in Dagster now?
Does Dagster work on premises as well as the cloud?
Are any features on cloud not available on premises?
My company is moving back to on premises and doesn’t want to use SSIS.
Hi Joshi. Dagster is an open-source project and all the capabilities that Pedram describes in this video are available in the open-source solution, which you can self-host locally or on-prem. Dagster Cloud offers additional capabilities, which are detailed at dagster.io/cloud or on the pricing page here: dagster.io/pricing
If the requirement is to get the data from S3 files into a BQ table but perform some validations on those files before inserting into the table, how would we do it with Embedded ELT? We are using Dagster OSS heavily and looking to use embedded-elt for getting data from files, tables and APIs..
Hey Abishek! In your case, would you be able to represent the S3 files as source assets first, adding asset checks onto those, and running Embedded ELT only if those asset checks pass? Sling currently (afaik) is heavily focused on doing ingestion well, so you can defer to the rest of the Dagster ecosystem (such as asset checks) for validations.
@@tim-at-elementl Thanks, Tim. How would you rate dlt for my use-case? I see dlt is far more mature..
@@AbhishekAgrawal-dv1id we've found that dlt is a powerful framework for ingesting from APIs and it's definitely mature enough for production settings. I'll also say that neither Sling's or dlt's integration currently allow for creating asset checks in-flight during ingestion.
Instead, have you thought about ingesting the files into a quarantined dataset first using whichever tool you'd like, applying asset checks to that, and then moving that data to your real "analytics-ready" BQ datasets once you've vetted the data? This way, you can do ad hoc analysis to understand why the data failed data quality tests easily, but also keep it isolated from your production analytics.
Yeah, I am also leaning towards doing something like this. Thanks for this, Tim.
Would you suggest using a similar approach to pull data from a different database? We'd still need to run minor validations on the incoming data, though. Would dlt help here at all?