Unit testing with Databricks | Jonathan Neo | November 2021
ฝัง
- เผยแพร่เมื่อ 5 ก.ย. 2024
- Just like eating vegetables, no one likes writing tests. However, writing unit tests is good for your programming diet. It helps ensure that data flows from one end of the pipeline to the other without any hitches.
In this talk, Jonathan Neo, Senior Data Engineer at Cuusoo, will explain why and how you can write unit tests, and where does unit testing fit in the bigger picture.
Jonathan will demo how you can write your own unit tests in Databricks using Databricks Connect and PyTest (a popular Python testing library), and also automate the execution of unit tests using CI/CD pipelines.
--
This was the talk from the November 2021 event of the Melbourne Databricks User Group.
Unit testing info starts at 10:32
Great session very much thankful to you.
Hi Jonathan, thanks for this demo, this is fantastic. few things have changed in this 2 years but the basics are same. When are you going to show us the remaining two parts like integration testing and data quality testing? or if you already have those videos can you please upload them to your channel. Thanks.
Good stuff! Liked and subscribed ! :)
Thanks a ton. Loved it
Great video! Since databricks-connect is now deprecated, how should we set up unit testing?
Hi Bjarne, have you managed to get an answer on this somewhere else?
@@yoshitjuh, actually there's an update to the databricks-connect package so that it now supports runtime 10.4 LTS, but databricks apparently recommends not using it and rather use dbx by databrickslab to setup a project, run all unit testing locally and supply convenient command line functions for deployment and running jobs. Still not sure about testing during pull requests and stuff like that.
informative
Why 45 minutes needed for a demo example?
nothing is visible on your code screenshots. Very bad presentation.