I would configure the expectation results storage location to be a bucket and then have a pipeline that takes the expectation results and stores them in a database
May I know if, we can create our own expectations. If I have a expectations who is not in the script that is on the documentation Could I create my own one?For Example, if one column is created from a formula and used a diferent database Could I create a expectation of who makes the math? Hi, from Colombia :)
I've only recently started using Soda so I'm not sure if I have enough experience to form a definitive opinion, but I have definitely enjoyed the UX much more so far, SCL is a lot more human readable than great expectation "expectations" imo
Very simple video with excellent explanation and not overcomplicating things. Thanks for sharing it!
Thanks for watching!
Great explaination!Any thoughts on how we can save the great expectation results in the Database?
I would configure the expectation results storage location to be a bucket and then have a pipeline that takes the expectation results and stores them in a database
Also how can we automate the threshold changes with the changing business needs?
You'd want to have another helper pipeline that checks for changing business requirements and then either alerts you or makes adjustments
Also can we include more than one expectation in the expectation file?
Yes!
On failure of great expectation validation, would this raise alerts?
Yes as long as you have Alerts configured for your Airflow DAG
May I know if, we can create our own expectations. If I have a expectations who is not in the script that is on the documentation Could I create my own one?For Example, if one column is created from a formula and used a diferent database Could I create a expectation of who makes the math?
Hi, from Colombia :)
Definitely can create your own expectations, honestly one of the best features of great expectations!
@@thedataguygeorge Do you have a video-tutorial where you’re teaching how to connect GX with Databricks? 😊
Very useful bud. Thank you.
No problem, do it all for you!
@@thedataguygeorge will reach out on the side to talk shop if that's cool, loving your work.
Always cool!
On minute 3:57 how could create the default file of great_expectations? I created the json but I got a blank one.
You then just fill out that json with all the expectation info you want!
Hi, do you prefer soda or great expectations?
I've only recently started using Soda so I'm not sure if I have enough experience to form a definitive opinion, but I have definitely enjoyed the UX much more so far, SCL is a lot more human readable than great expectation "expectations" imo
Thank You
No worries, let me know if there's any other videos you'd like to see!