Hi Piotr, i came across your channel and i have been buying resources esp books on Azure data engineering but trust me your videos are so simple, eloquent and easy to understand. I am grateful for your effort. keep going and keep your smile.
This is a really good and helpful video. Couldn't thank you much. I am a newbie trying to get into data engineering. I will try my best to make out of this playlist. Thanks and please keep up the good work. :)
Hi Piotr, I understand creating a dataset is a good practice but I have seen production level pipelines with copy activity directly connected to SQL on prem databases. When is that a dataset is a must? Thanks, P
@@TybulOnAzure Sorry my bad, I was looking for dataset as an activity (similar to copy activity, etc) which you place somewhere on the pipeline but now understand that dataset is like a configuration.
Hi Tybul, I got to know that the Microsoft role based and specialty exams are now open book exam since August update. Any tips for how to use Microsoft Learning efficiently while facing the exam? Thank you!
I would advise to use MS Learn for exam preparation so during the exam you will be familiar with it and you would know what to look for, where and how. The great thing about this change is that no longer you will have to memorize everything - it will be enough to know that some feature exists and you will just find details in MS Learn.
Hey Piotr, Thanks for the great explanation, In the video, you mentioned that we use ADF for data ingestion and orchestration of activities. Could you clarify why we don’t use it for data transformation as well, given that ADF includes several components designed for transformation?
@@TybulOnAzure Thanks for the detailed answer! I have a another question: You previously mentioned that we use a staging (landing) area in ADLSg2 to create a copy of the data from the source. Could you clarify what an ODS is and why it differs from the staging area?
Thank you very much. I have wasted quite a some time because I chose the Ms Entra ID authentication and went digging into parameters because I could not switch the created server back to SQL auth. Seems possible only by destroy/create. Beware!
Thanks! Yes, at some point I'll cover Fabric but I don't know yet when it would be. Maybe I'll make a separate playlist about Fabric that will be a kind of appendix to DP-203. Time will tell.
Hi Piotr, I wanted to know exactly what is a dataset and what is its role? I have watched carefully in the video and it is still not clear to me..... Also Your content is extremely Good, thanks for it
Dataset represents the layout of your data and its properties. Let's assume that you want to load customers from: a) SQL database, b) CSV file. In both cases you would have a dataset that represents your customers so it would have the schema (columns and data types). However, depending on source of your data (db or file), you would have additional properties set in the dataset: a) For SQL DB - name of source table, b) For CSV file - path to the file, column delimiter, row delimiter, encoding, quote character, etc.
Not only prepping a person for the cert but asserting best practices and big picture thinking. PHENOMENAL AND SENSATIONAL
Thanks! I'm glad you like it!
Hi Piotr, i came across your channel and i have been buying resources esp books on Azure data engineering but trust me your videos are so simple, eloquent and easy to understand. I am grateful for your effort. keep going and keep your smile.
Thanks!
Piotr, you're the man! Thanks for your great content. Very structured and easy to understand
Thanks, I'm glad you enjoyed it.
This is a really good and helpful video. Couldn't thank you much.
I am a newbie trying to get into data engineering. I will try my best to make out of this playlist.
Thanks and please keep up the good work. :)
Glad it was helpful!
Good overview to ADF !! Thank you!!
Super Introduction do Azure Data Factory & Data Ingestion;
Glad it was useful!
the hierarchy is different from the one you brilliant teached in the last video "Data Lake Structure". First domain, then source system. Thank you!
Hi Piotr,
I understand creating a dataset is a good practice but I have seen production level pipelines with copy activity directly connected to SQL on prem databases. When is that a dataset is a must?
Thanks,
P
Always. How was this pipeline configured that it wasn't using any dataset?
@@TybulOnAzure Sorry my bad, I was looking for dataset as an activity (similar to copy activity, etc) which you place somewhere on the pipeline but now understand that dataset is like a configuration.
Hi Tybul, I got to know that the Microsoft role based and specialty exams are now open book exam since August update. Any tips for how to use Microsoft Learning efficiently while facing the exam? Thank you!
I would advise to use MS Learn for exam preparation so during the exam you will be familiar with it and you would know what to look for, where and how.
The great thing about this change is that no longer you will have to memorize everything - it will be enough to know that some feature exists and you will just find details in MS Learn.
Hey Piotr,
Thanks for the great explanation,
In the video, you mentioned that we use ADF for data ingestion and orchestration of activities. Could you clarify why we don’t use it for data transformation as well, given that ADF includes several components designed for transformation?
We can use it for transformations (it is covered later), but there are other tools that simply do it better.
@@TybulOnAzure Thanks for the detailed answer!
I have a another question: You previously mentioned that we use a staging (landing) area in ADLSg2 to create a copy of the data from the source. Could you clarify what an ODS is and why it differs from the staging area?
U r the best.
i have a question : do you have loading adventureworks db before the episode or it is a db provided by azure like a demo???
It is a sample database available from Azure - that's why I used "Sample" option when provisioning an Azure SQL DB.
@@TybulOnAzure thank u
Thank you very much. I have wasted quite a some time because I chose the Ms Entra ID authentication and went digging into parameters because I could not switch the created server back to SQL auth. Seems possible only by destroy/create. Beware!
So, Synapse Studio in Azure Synapse Analytics and ADF are same? they look very similar.
Synapse pipelines is pretty much the same thing as ADF.
For ZoomIt, you might be able to click the right mouse key twice instead of using the Escape key. Ctrl-1 might also work.
Thanks! I'll definitely try that.
Great tutorial Piotr! Do you have any plans of doing videos on Microsoft Fabric?
Thanks! Yes, at some point I'll cover Fabric but I don't know yet when it would be. Maybe I'll make a separate playlist about Fabric that will be a kind of appendix to DP-203. Time will tell.
Hi Piotr,
I wanted to know exactly what is a dataset and what is its role?
I have watched carefully in the video and it is still not clear to me.....
Also Your content is extremely Good, thanks for it
Dataset represents the layout of your data and its properties. Let's assume that you want to load customers from:
a) SQL database,
b) CSV file.
In both cases you would have a dataset that represents your customers so it would have the schema (columns and data types). However, depending on source of your data (db or file), you would have additional properties set in the dataset:
a) For SQL DB - name of source table,
b) For CSV file - path to the file, column delimiter, row delimiter, encoding, quote character, etc.
@@TybulOnAzure got it! Thank you 😊
♥♥♥♥♥
Topman:)
what a co-incidence the practicle shows in the video same date n month as 20 th Oct except year (2023) doing in 2024. 😀
Yup, it's been already a year since I recorded it. Time flies.
Same pinch 42:37😂
thanks.