17. Databricks & Pyspark: Azure Data Lake Storage Integration with Databricks

How to integrate Azure Data Lake Storage with Databricks?
There are several ways to integrate ADLS with Databricks such as using service principal, Azure Active Directory credentials etc. In this demo, two methods are demonstrated i.e directly accessing using access key and creating mount point.
What is mount point?
The mount point is a pointer to azure data lake storage. Once mount point is created databricks can access the files in ADLS as if local file system
This video covers end to end process to integrate ADLS with databricks. This demo exercise covers these three areas
1. Create Azure Data Lake Storage in Azure Portal
2. Create Mount point using ADLS Access Key
3. Read files in ADLS through Databricks using mount point
#DatabricksIntegrateADLS #SparkADLSIntegration #DatabricksMount #ADLSMount #SparkMount #DatabricksReadADLSFiles #DatabricksReadADLS #DatabricksReadCSVfromADLS #Unmount #DatabriksUnmount #Pyspark-unmount #DatabricksDButility #DbutilsMount #DbutilsList #Sparkmountunmount #DatabricksRealTimeproject #DatabricksRealTimeExercise #Sparkrealtimeproject #Pysparkrealtimeproject #DatabricksTutorial, #AzureDatabricks #Databricks #Pyspark #Spark #AzureDatabricks #AzureADF #Databricks #LearnPyspark #LearnDataBRicks #DataBricksTutorial databricks spark tutorial databricks tutorial databricks azure databricks notebook tutorial databricks delta lake databricks azure tutorial, Databricks Tutorial for beginners, azure Databricks tutorial databricks tutorial, databricks community edition, databricks community edition cluster creation, databricks community edition tutorial databricks community edition pyspark databricks community edition cluster databricks pyspark tutorial databricks community edition tutorial databricks spark certification databricks cli databricks tutorial for beginners databricks interview questions databricks azure

[@JalindarVarpe-h4o]

Enjoying the PySpark tutorials! Can you make a video on setting up Azure and navigating the portal? It would be super helpful. Thanks for the great content!

[@a2zhi976]

you are my guru from onwards ..

[@rajasdataengineering7585]

Thank you

[@shivanisaini2076]

this video is worth watching, my concepts related to access the file in databricks are clear now thank you sir

[@rajasdataengineering7585]

Thanks Shivani!

[@ndbweurt34485]

very clear explaination. god bless u.

[@rajasdataengineering7585]

Thank you

[@naveenkumarsingh3829]

hey you are using location as wasbs:// which is nothing but azure blob storage location , and sometimes you are taking abfss:// which is path to azure data lake gen2 location.. Since I am still learning , I am getting really confused now.. And your video says adls connection with databricks..then it should be abfss:// right for a file path?

[@felipedonosotapia]

Thanks so much!!! nice tutorial

[@rajasdataengineering7585]

Glad it was helpful!

[@HariprasanthSenthilkumar]

Can you please make a video to connect to ADLS by service principal

[@rajasdataengineering7585]

Sure I will do

[@dhivakarb-ds9mi]

I am getting this error
Operation failed: "This request is not authorized to perform this operation using this permission."

[@jagadeeswaran330]

Nice explanation!

[@rajasdataengineering7585]

Glad it was helpful! Thanks

[@sujitunim]

Really very helpful... could you please create video for on premise Kafka integration with databricks

[@rajasdataengineering7585]

Sure Sujit, will do one video on this requirement

[@anoopkumar-f1r]

Great Raja!

[@rajasdataengineering7585]

Thank you

[@Ramakrishna410]

Great knowledge. How can we apply access polices on mounted containers?
For ex , 50 users have acess for databricks so , 50 users can see the all files under mounted container but i want to give read acess for few users only? How can we?

[@rajasdataengineering7585]

Hi Alavala, Good question.
Mount points can be accessed from darabricks through service principal or Azure Active Directory.
If we use service principal (SP) to create a mount point, all users/groups under the databricks workspace can access all files/folders in mount point.
So if you want restrict access for set of people, there are many ways. One common approach is use AAD to create mount point so that user access ca be controlled using IAM within Azure portal.
Another approach could be creating 2 different databricks workspaces and accessing mount point through 2 different service principals one with read access, another with write access.
Hope it helps

[@DivyenduJ]

Hello All, I am new to this and getting below error , many thanks if anyone could help for step 1:
Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key

[@rajasdataengineering7585]

Hi, seems the access key is invalid. Could you check it once again from storage account

[@DivyenduJ]

@@rajasdataengineering7585 Thanks a lot sir for the guidance it worked , mistakenly set on rotate key. May be that's the reason.

[@rajasdataengineering7585]

Glad to know it worked!

[@kartechindustries3069]

Sir does azure data lake comes under community groups or free services

[@rajasdataengineering7585]

No, azure data lake is paid services but Microsoft provides one month free subscription with some free credit. You can take advantage of it for your learning purpose

[@lucaslira5]

with this option, is possible writing in data lake? Or only read?

[@rajasdataengineering7585]

We can write as well

[@dataengineerazure2983]

@@rajasdataengineering7585 How get the dataset source(csv files)? thanks

[@sravankumar1767]

Nice explanation bro 👍

[@rajasdataengineering7585]

Thanks bro

[@alexfernandodossantossilva4785]

If we have a Vnet on the storage account? How can we access?

[@rambevara5702]

Don't we need to app registration for data lake?

[@rajasdataengineering7585]

That is another way of integration through service principal

[@rambevara5702]

@@rajasdataengineering7585 whatever it is fine right..brother where can I get this databricks notebook..do you have any GitHub

[@lucaslira5]

How would I do if the container had more files instead of just 1?

[@rajasdataengineering7585]

We can use wildcard to select multiple files

[@lucaslira5]

@@rajasdataengineering7585 what would this wildcard be like? I have two files in the container (city.csv and people.csv) but it's only bringing people.csv

[@rajasdataengineering7585]

You can give *.csv so that it can pick all CSV files

[@lucaslira5]

@@rajasdataengineering7585 But I would like to bring a specific file, for example my blob has 50 .csv files but I only want to bring people.csv to perform an ETL

[@lucaslira5]

Would it be here for example to put .option("name","people.csv)?
df = spark.read.format("csv").option("inferSchema","true").option("header", "true").option("delimiter",";").option("encoding","UTF-8").load(file_location)

[@rajivkashyap2816]

Hi sir,
Any git link is dere so that we can copy and paste the code

[@subbareddybhavanam5829]

Hi Raj, Can you please add data files too. like CSV and Json ...

[@MohanGonnabathula]

Yes

[@natarajbeelagi569]

How to hide access keys?

[@rajasdataengineering7585]

We can use databricks scoped credentials

[@AkashVerma-o7o]

is it free to use azure data lake?

[@rajasdataengineering7585]

No, it's not free

[@lovepeace2112]

good