- 84
- 173 368
Pytalista
เข้าร่วมเมื่อ 11 ต.ค. 2022
Azure DevOps Tutorial for Beginners Python Example CICD
In this quick tutorial I will teach you how to create your first CICD pipeline in Azure DevOps services.
GitHub Code: github.com/microsoft/python-sample-vscode-flask-tutorial
Tutorial Link: learn.microsoft.com/en-us/azure/devops/pipelines/ecosystems/python?view=azure-devops
Create Azure DevOps Organization: learn.microsoft.com/en-us/azure/devops/pipelines/get-started/pipelines-sign-up?view=azure-devops
Parallel Job Free Tier Request: forms.office.com/pages/responsepage.aspx?id=v4j5cvGGr0GRqy180BHbR5zsR558741CrNi6q8iTpANURUhKMVA3WE4wMFhHRExTVlpET1BEMlZSTCQlQCN0PWcu&route=shorturl
GitHub Code: github.com/microsoft/python-sample-vscode-flask-tutorial
Tutorial Link: learn.microsoft.com/en-us/azure/devops/pipelines/ecosystems/python?view=azure-devops
Create Azure DevOps Organization: learn.microsoft.com/en-us/azure/devops/pipelines/get-started/pipelines-sign-up?view=azure-devops
Parallel Job Free Tier Request: forms.office.com/pages/responsepage.aspx?id=v4j5cvGGr0GRqy180BHbR5zsR558741CrNi6q8iTpANURUhKMVA3WE4wMFhHRExTVlpET1BEMlZSTCQlQCN0PWcu&route=shorturl
มุมมอง: 30
วีดีโอ
How to create a custom GitHub Action with Docker [Python]
มุมมอง 6714 วันที่ผ่านมา
In this video I will teach you step by step how to create a custom GitHub Action with Docker to run a Python script that calls the Databricks API. Code: github.com/pedrojunqueira/github-custom-docker GitHub Actions Documentation: docs.github.com/en/actions/sharing-automations/creating-actions/creating-a-docker-container-action
How to use tags to create a release in GitHub with Actions
มุมมอง 11528 วันที่ผ่านมา
In this video I am going to explain the difference between a tag and a commit and also a branch and also create a release using GitHub Actions. Code in GitHub: github.com/pedrojunqueira/git-tag-demo
Setting Permissions in Databricks Asset Bundle - Advanced Example
มุมมอง 107หลายเดือนก่อน
In this example I will take you through an example of how to implement permissions in Databricks Asset Bundle Documentation: learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/permissions Code Example: github.com/pedrojunqueira/PytalistaYT/tree/master/Python/databricks-asset-bundle-permission/test_permissions
How to Enable and Use Databricks Serverless Compute
มุมมอง 303หลายเดือนก่อน
I am going to teach you how to start with Databricks Serverless Compute by enabling and examples in notebooks and workflows. Getting Started with Databricks Serverless Compute #databricks #code #serverlesscomputing #spark #python Documentation: learn.microsoft.com/en-us/azure/databricks/compute/serverless/ Limitations: learn.microsoft.com/en-us/azure/databricks/compute/serverless/limitations?so...
Unable to access Account console under Azure Databricks
มุมมอง 270หลายเดือนก่อน
Unable to access Account console under Azure Databricks because Selected user account does not exist in tenant 'Microsoft Services' and cannot access the application '2ff814a6-3304-4ab8-85cb-cd0e6f879c1d' in that tenant. The account needs to be added as an external user in the tenant first. Please use a different account. Solution from this video: community.databricks.com/t5/administration-arch...
How to ssh from a linux client into a Azure Linux VM
มุมมอง 99หลายเดือนก่อน
In this video I will do a quick demo on how to create a Linux vm in Azure and ssh into it using your local Linux client. Azure Docs: Connect using public key: learn.microsoft.com/en-us/azure/virtual-machines/linux/create-ssh-keys-detailed Connect using private key: learn.microsoft.com/en-us/azure/virtual-machines/linux-vm-connect?tabs=Linux
Using Variables in Databricks Asset Bundle - Advanced Example
มุมมอง 3962 หลายเดือนก่อน
In this video I will show for to use variables in Databricks Asset Bundle effectively and having variables in different environments and also how to overwrite target resources and lookup resources in your Databricks workspaces. Code: github.com/pedrojunqueira/PytalistaYT/tree/master/Python/databricks-asset-bundle-variable/demo Documentation: learn.microsoft.com/en-us/azure/databricks/dev-tools/...
Finally Monetized after 1000 subscribers
มุมมอง 942 หลายเดือนก่อน
Not so quick as some youtubers claim after 20 months and 70 videos and hours of work and money invested. Is it worth?
Deploy Terraform with GitHub Actions [Terraform Cloud API]
มุมมอง 1922 หลายเดือนก่อน
In this video I go step by step on how to deploy a Databricks cluster with terraforming using GitHub Actions and Terraform Cloud Api Code: github.com/pedrojunqueira/terraform-cloud-databricks Tutorial Inspiration: developer.hashicorp.com/terraform/tutorials/automation/github-actions Databricks Terraform Provider: registry.terraform.io/providers/databricks/databricks/latest/docs/resources/cluster
Deploy Databricks Workspace with Terraform
มุมมอง 5123 หลายเดือนก่อน
How to deploy Azure Databricks Workspace with Terraform. GitHub code : github.com/pedrojunqueira/PytalistaYT/tree/master/Python/terraform_adb_ws
Deploy a Unity Catalog Cluster in Azure Databricks using Terraform
มุมมอง 2323 หลายเดือนก่อน
Steps on how to deploy a Cluster in Azure Databricks using Terraform GitHub: github.com/pedrojunqueira/PytalistaYT/tree/master/Python/terraform-databricks-cluster Terraform Databricks Provider: registry.terraform.io/providers/databricks/databricks/latest/docs
Deploy Resources in Azure with Terraform
มุมมอง 3894 หลายเดือนก่อน
This video I am going to go step by step how to set up your environment, service principle and deploy to azure a resource group and a storage account using terraform. Terraform documentation: registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_account Install Terraform CLI in linux: developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli Install Azure...
Deploy Databricks Asset Bundles using GitHub Actions [DevOps]
มุมมอง 1.6K4 หลายเดือนก่อน
Deploy Databricks Asset Bundles using GitHub Actions [DevOps]
Build Python Packages in a Databricks Asset Bundle
มุมมอง 1.1K4 หลายเดือนก่อน
Build Python Packages in a Databricks Asset Bundle
How to create and deploy Azure Function Using VS Code NO MUSIC
มุมมอง 1.1K4 หลายเดือนก่อน
How to create and deploy Azure Function Using VS Code NO MUSIC
How to deploy Databricks Asset Bundle Projects
มุมมอง 2.4K4 หลายเดือนก่อน
How to deploy Databricks Asset Bundle Projects
Deploy Storage Account with Bicep and GitHub Actions CI/CD [VS code]
มุมมอง 2075 หลายเดือนก่อน
Deploy Storage Account with Bicep and GitHub Actions CI/CD [VS code]
How to build your ChatGPT clone with Django and HTMX
มุมมอง 3095 หลายเดือนก่อน
How to build your ChatGPT clone with Django and HTMX
Demo how to CDC with Debezium into Kafka in the cloud
มุมมอง 4485 หลายเดือนก่อน
Demo how to CDC with Debezium into Kafka in the cloud
How to read Kafka Topic with Databricks [Confluent]
มุมมอง 9655 หลายเดือนก่อน
How to read Kafka Topic with Databricks [Confluent]
How to run multiple notebooks in a thread in Databricks [Python]
มุมมอง 5846 หลายเดือนก่อน
How to run multiple notebooks in a thread in Databricks [Python]
How to send Python logs to Applications Insights (Azure Monitor)
มุมมอง 2.4K6 หลายเดือนก่อน
How to send Python logs to Applications Insights (Azure Monitor)
Using Change Data Feed and Structured Streaming in Fabric [PySpark]
มุมมอง 3406 หลายเดือนก่อน
Using Change Data Feed and Structured Streaming in Fabric [PySpark]
How to do Slow Changing Dimension in Delta Tables [Python]
มุมมอง 2137 หลายเดือนก่อน
How to do Slow Changing Dimension in Delta Tables [Python]
Handling secrets in Fabric Notebooks - Azure Key Vault
มุมมอง 1.4K7 หลายเดือนก่อน
Handling secrets in Fabric Notebooks - Azure Key Vault
Doing XMLA endpoint in Fabric with a F2 Capacity
มุมมอง 3307 หลายเดือนก่อน
Doing XMLA endpoint in Fabric with a F2 Capacity
Work with Azure Data Lake as though it was your local File System [Python adlfs]
มุมมอง 4827 หลายเดือนก่อน
Work with Azure Data Lake as though it was your local File System [Python adlfs]
How to upload fake data to azure storage
มุมมอง 1689 หลายเดือนก่อน
How to upload fake data to azure storage
When people say simple many times during a demonstration, you know its exactly the opposite.
😀
let me know if you are stuck and where you did not understand.
Thank you! This helped me move on to the actual coding :)
Glad I could help!
Thanks for this videos. If we’re using storage accounts like ADLS how we can use this variables?
Thanks. In this case variables are to be applied across your bundle files in many cases.
Thank you very much. It was really helpful!
You're welcome!
Do you have an example using telemetry?
No sorry. I will think about it.
In local setup it is working but when i deploy its not working
Check if any environment variables in your code. Look at logs do debug
Your videos are so helpful! Please keep up the great work. More Databricks videos on how to build robust, scalable enterprise pipelines for medallion architecture would be awesome to see.
Noted.
Hi, great video. Is this possible with an on-prem sql source? Thanks
No the source needs to be in the data lake.
Can we use Timer Trigger with durable functions ?
Technically yes.
What extension for VSC terminal you are you using?
I am using Linux WSL
Excellent video! Liked and subbed!
Thanks 🙏🏻
I am getting error while accessing the mount point via cluster . In my case its just standard storage account non lake storage and I am using SAS token for accessing , I noticed you are using Lake storage and using wasbs is it depricated . Can you tell me how to fix this , I tried cluster config but still its facing error
Hi this method is no longer recommended. Use now unit catalog meta store or external storage.
Where does it save the files to? On windows10
It will save to the path you specify. In the code example because no path were specified then it will save in the current directory where the .py file is executed.
It was awesome, Can you share the code or github link? Thanks in advance
thanks. github.com/pedrojunqueira/PytalistaYT/tree/master/Python/function-queue-trigger
the above fails for me when running the test for postman in that it wont work with 3.11.9 and only func only works with 3.6-3.9 - advise please and function apps in azure only offering 3.10 or 3.11 so do you need to shift to v2 functions code or ?
This example is V1. V1 should work ok for all python available. I will update this eventually for V2.
Hi Bro , i got this , thanks for this video and have one doubt like how to run the particular file with showing coverage percentage? as of now if i have 10 files and if i ran single file that coverage is showing for all 10 files , Can you fix this how ?
You can do a coverage report then it will give you the line and file that does not have coverage. install the coverage report and run -> coverage report -m. More info here. -> coverage.readthedocs.io/en/7.6.1/
What should be the password which we need to enter for new emailid?
You can either get a generated one but create a password that is safe :)
thank you bro , I strugled to run pytest for whole week
👍🏻
It is not creating funtion.js file for me
This example is with the Python language. The .js is JavaScript. 👍🏻
Thanks Man !! This helped a lot :D
Welcome. 🙏🏻
How to remove job name [dev My_User_Name] please suggest
If your target is in development mode it adds automatically. To remove just change to production mode.
@@pytalista Thank you Sir 🙏
@praveenreddy177 thanks. All good. 👍🏻
Thanks
Np 😉
Thanks for the reference article. Prior to this I managed to set everything up manually with cert available on the host machine. But I can now automate the rotation with the dedicated certbot container! Seems obvious in retrospect to use dedicated conatiner to avoid all the snap/setup issues with an alpine base image...
Thanks for the comment. I need to do a better version of this video. The quality is really bad. Glad it helped.
Hi Sir , Thank you for this detailed video. I was very helpful. I think you can avoid the background music, it is overpowering.
Thanks. There is another version without the music if you do not like it. th-cam.com/video/wUqohFhYHl0/w-d-xo.htmlsi=VqhLyUMYZMXo5YQ2
good day is there away to send opentelmtry to eventhub including logs
This was such a great help. I know how easy it is now, but trying to figure this out on my own took an embarrassing amount of time, and I wasn’t even able to deploy it 😅 Edit: wanted to add that I’m on the newest version of Python Azure tools and the video was still a great guide. As far as the main steps to setup/deploy they’re practically the same which is great for a noobie like me.
Glad it helped. Hope this unlocks in you more confidence to do even more complex stuff. All the best.
Thank you so much! Great tutorial! However, I encountered a bug at the end: 76689cbf-b257-4d6a-902e-86a3778c4ed3 does not have View permissions on 136172929345010. Please contact the owner or an administrator for access. Can you please kindly shed some lights?
it seems like the Service Principal does not have VIEW permission on 136...010. However, the Service Princepal has the contributor role
Glad you liked it. reply below
@@Traveling_with_Tyler does this service principal is a workspace administrator? If so it have all the permissions. I have not done a video about permissions. Will do next. But you can read the documentation here. learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/permissions
@@pytalista Thanks for your reply. The service principal is a contributor. Should I change it to the owner?
@Traveling_with_Tyler is it a contributor to what resource ?
Is it a public databricks workpsace? where are the networking and encryption modules?
Hi. What do you mean by a public Databricks Workspace ?
Great video, how to do that in databricks jobs ? How to fail job, when one run fail ?
If one run fail the job will fail.
what does the name "this" mean?
It is just a label for the object. You can call anything you want. Usually if you only have one instance of the object in the module you just call it this.
Thank you so much for uploading a version without music. This was very helpful for me.
Thanks 🙏🏻. Please subscribe and smash 💥 the like button 😃
It is a very greate video. I was looking for very long time for a video like that. Thank you! My code works perfectly when i run it on Visual Studio. But it is not triggered on Azure. Is there any special thing I need to do more? Code is deployed on Azure. I can see the updated code on Azure. Connection strings in Environment Variables are correct. When I run code in Azure I get 202 response. But I get no logs. There are no invocations on Azure.
There is no question :). I have figured it out. My connection was connection="BlobStorageConnectionString" and i did not add it to Environments variables. Stupid error.
Awesome. Great it worked
Great to hear! Below you sorted out. Cheers
command "databricks bundle init" is getting failed with exit code 1. I have installed databricks cli version 0.223.0 and authentication has been done properly. Can you please suggest what could be the issue?
Have you done a databricks auth profiles ? Do you get a green tick ✅?
@@pytalista Yes, I have used debug and I found the issue. I have used the command databricks --debug bundle init to find the issue. Then it has thrown an error to choose the profile among default and new created by providing --profile flag. Then I used the databricks --debug bundle init --profile "new profile".
You mean that now is working ?
@@pytalista Yes
🙌🏻👏🏻
Most helpful video
This is very helpful. Thank You Very much Have you extended this exercise to create partition-based parquet file inside table folder.?
Hi, do not intend to go on this topic. This is a way to bring data to the lake form transactional systems. then from there I would ingest in delta tables.
Thanks for the vedio .......this is extremely useful vedio .Can you please include the vedio on deployment pyspark codes in on prem or cloud
Thanks 🙏🏻. I mostly use spark on the cloud which is already implemented. You focus on running code.
Very useful video particularly around Databricks Rest Api. Thanks
Thanks 🙏🏻
Thanks for the video. Would delta live tables , with streaming table , be better for this?
Hi good point. Certainly delta live table simplify the code and the developer experience in a more declarative way. This is a decision engineer need to make. DLT cost is a bit higher and some prefer to code more declarative. I would say that the simpler the transformation and more "cookie cutter" I would go for DLT otherwise better to have CDF.
Thank you for the content. However, databricks bundle init does not work. Error: No such command 'bundle'. Do you know why?
Thanks 🙏🏻. Make sure the Databricks cli version is 0.218.0 or higher. docs.databricks.com/en/dev-tools/bundles/index.html
if the function doesnt work and it keeps getting nothing, the solution might be going to your funtionApp on Azure, and go to environment variables , and then try to add the connection name as a key and the connection string of your storage account as a value , and it will work.
Yes
Thank you this is exactly what i concluded after A/B testing creating the app from the portal or with vs code
Great video. I have a question though. You said once you start using DAB, everything should be done without the UI. What if I need to write a Notebook, where do I do this? When writing this in VSCode I think I always need to submit it as a job to databricks for testing and this takes so much time. In the UI it is very fast and easy to execute cell by cell.
Yes in this case for speed you can still develop using the UI and when done export the notebook and deploy in a DAB. Also you can run the notebook in VS code by attaching cluster in VS code. But still UI offers a better experience while developing.
Oh I literally check each time of my teams when it is beeping on yours🤣
🤣
I have tried this approach, but facing unauthorized error. I have all required access/ roles on Key vault, still unable to figure out the issue
Is the same user in Fabric been granted the permission in lay vault ?
Very good, straight to the point. You could make a video running pytest on databricks. Do you think that is a valid approach, since many tests depends on reading data from catalog?
Hi I think is a valid approach. Test in data engineering is a bit different. I would separate the test into 2 categories. Unit Test which would be test of the functions that is used in your code. Then use pytest for that. Then tests of data that you can use Delta Live tables expectations or libraries like great expectations. This because in data engineer data is stateful.
@@pytalista Good point. So basicaly, you're suggesting that queries are basically tested by the data quality checks you apply in the result, and not testing the query itself using mock data and expected dataframes?
@SergioPolimante on data quality you can test things like. Not nulls, unique, data ranges, valid keys etc. that is the part of your data is statefull. About testing the logic of your query with mock data is also a valid approach maybe it is a lower return on time investment than quality expectations checks. Then there is the unit test of your functions you may have in your pipeline in this case you would do classic unit tests.
Thank you for the nice tutorial brother
Glad it helped
Hi, in the palette i do not see the option for durable function, but just for trigger function. Any advice?
follow this then it will make sense learn.microsoft.com/en-us/azure/azure-functions/durable/quickstart-python-vscode?tabs=linux%2Cazure-cli-set-indexing-flag&pivots=python-mode-decorators
I ran into the same issue, I just did it via the command line. Use 'func templates list' to get the list of templates you want to use.
@carlosr204 good one thanks 🙏🏻
Thanks I hope you bring some more such content involving pyspark and databricks
th-cam.com/video/lYYIFRaY8Tk/w-d-xo.htmlsi=RiYQqRffGBTVkYZR
Is there any specific reason why the blob size comes out as None everytime? I have a use case which extends blob trigger to call a search service on newly added/modified files, and I'm getting blob not found 404 error 😑 despite the blob being there in my container. Thank you for the video though, it confirmed I'm not doing something astronomically wrong 😂!
No worries Thanks. Hard to tell without more context. 404 is a not found error. If you follow the tutorial should work exact the same.
@@pytalista Oh yes, I had to implement custom logic for my use case, so the code was quite different. The storage account I was trying to access did not allow anonymous access and an incorrect request URL was being generated. All in all, Blob was not found. I was able to debug it. Your video set a great precedent, thank you for replying.
@diyakhajuria2478 great work. Happy it helped.
Great Video! I was stuck on the how to get the Path variable and you demonstrated it perfectly. The documentation still doesn't have it.
Thanks
Awesome video! Great explanations! Thanks à lot
Thanks 🙏🏻