Learn to Efficiently Test ETL Pipelines

This talk is a story, using examples in Python and pySpark, about testing ETL pipelines efficiently. I won’t try to convince you that you need unit tests or automated tests - that’s up to you. If you do have unit tests for your ETL pipelines, or if you want them, it can be useful to make sure you aren’t testing more than you need.
I’ll be describing how a practical (non-pyramid shaped) heuristic helps me efficiently cover edge cases and unexpected bugs in my code by ensuring I test only the code needed for the feature I’m building.
Connect with us:
Website: databricks.com
Facebook: / databricksinc
Twitter: / databricks
LinkedIn: / data. .
Instagram: / databricksinc

[@felixa4705]

Great talk! It's hard to be vulnerable about mistakes you make, let alone in front of a crowd of strangers. I am definitely going to look into the Saff Squeeze now! Thanks for the explanation!

[@lucasdepetris]

Found this video while looking for solutions for a problem like the one you had. I hope it works for me as it did for you. Congrats for the amazing talk!

[@kaname223]

Great presentation, thanks lot , Automation testing is stressful and painful should be better alternatives.

[@allthingsdata]

awesome talk, not only did I learn sth, I also had fun.

[@netanelmalka6191]

Great talk :)

[@qa_career]

Amazing!!!