Advancing Spark - Databricks SQL Variables & Dynamic WHERE

Whenever someone comes from a heavy SQL background, they find some of their favourite toys missing from the Spark SQL world - but there have been huge investments from Databricks to catch up here over the past few months. We thought it was time to talk SQL Variables, Parameterised Tables and even the WHERE (*) functionality!
So if you've been sitting in the pyspark wastelands dreaming of the good old days of SQL Variables, this is the video you've been waiting for! In this video, Simon runs through how Spark SQL parameters work, some of the neat tricks you can do using them, and has finally thought of a use case for "any column filtering"!
For more info on Variables in Spark SQL, check out the docs here: learn.microsof...

[@mkrichey1]

Awesome presentation as always Simon! Great features which will be very useful!

[@Juan-PaulHynek]

Brilliantly put together, thanks Simon!

[@rommelbojorgee.8902]

Nice video Simon! thanks 👊

[@selimberntsen7868]

Great video, will really come in handy! Can these features only be used on SQL pools or also on Spark clusters?

[@TheDataArchitect]

Great job bro.

[@johnnywinter861]

oomph... throwing shade at the SQL editor 😵

[@pini22ki]

Is this working in databrick SQL mode?

[@alexischicoine2072]

Where would you use a variable instead of a widget?

[@AdvancingAnalytics]

I realised as I finished filming I should have thrown SQL Widgets in as well. The main difference is Variables can be derived from the data (ie: select max(id) from mytable), but widgets can be passed in externally. Both are super useful!

[@NeumsFor9]

​​​@@AdvancingAnalytics Data driven parameters vs Externally passed in params? Looks a lot like Snowflake (named_struct vs object_construct)....Big vendors always going at each other.....trying to steal market share

[@alexischicoine2072]

Nice name no servers here. Can’t wait to try the serverless for jobs.

[@StartDataLate]

Could you point out what exactly the "where 'text' in (T.*) "is called? at 10:55 timeframe. I am running issue with searching in a table that has struct type. and it is saying that the data type mismatch. ... and this search does not work