- 17
- 141 435
javiBI
Spain
เข้าร่วมเมื่อ 12 ธ.ค. 2020
Welcome to JaviBI!
I'm Javier Merino, a data enthusiast with over 14 years of experience. On this channel, I share my passion for the world of data through bite-sized, easy-to-understand videos on a wide range of data-related topics. Whether you're a seasoned data professional or just starting your journey, you'll find valuable insights and practical tips to help you navigate the data landscape.
Join our community and let’s play with data together, exploring its endless possibilities and unlocking the power of information.
I'm Javier Merino, a data enthusiast with over 14 years of experience. On this channel, I share my passion for the world of data through bite-sized, easy-to-understand videos on a wide range of data-related topics. Whether you're a seasoned data professional or just starting your journey, you'll find valuable insights and practical tips to help you navigate the data landscape.
Join our community and let’s play with data together, exploring its endless possibilities and unlocking the power of information.
Databricks Unity Catalog - Basic concepts and Configuration
In this video, you will find a detailed explanation of Unity Catalog in Databricks and you will understand how metastores works and how to create new ones.
มุมมอง: 69
วีดีโอ
Databricks Unity Catalog Configuring Delta Sharing
มุมมอง 77หลายเดือนก่อน
In this video, you will find a detailed explanation and a step-by-step process of creating a Delta Sharing in Databricks with Unity Catalog
Databricks Unity Catalog Granting permissions
มุมมอง 91หลายเดือนก่อน
In this video, you will find a detailed explanation and a step-by-step process of how to grant access at different levels in Databricks with Unity Catalog
Databricks Unity Catalog Creating external locations
มุมมอง 79หลายเดือนก่อน
"In this video, you will find a detailed explanation and step-by-step process of creating an External location in Databricks with Unity Catalog
DataFactory: Extract data from SharePoint Online
มุมมอง 79K2 ปีที่แล้ว
docs.microsoft.com/en-us/azure/data-factory/connector-sharepoint-online-list?tabs=data-factory#prerequisites docs.microsoft.com/en-us/azure/data-factory/connector-sharepoint-online-list?tabs=data-factory#copy-file-from-sharepoint-online
Data Factory: How to split data in rows and columns
มุมมอง 1.4K3 ปีที่แล้ว
In this video we are going work with a sample data file that requires the data to be splitted in to rows and columns.
Data Factory: Accesing Key Vault secrets in your Pipelines
มุมมอง 1.6K3 ปีที่แล้ว
In this video you will learn how to extract secrets stored on an Azure Key Vault resource from your data factory pipelines, how to use them on the activities and how to secure them throw the process.
Data Factory: Google Analytics data extraction step by step
มุมมอง 16K3 ปีที่แล้ว
console.cloud.google.com/apis/dashboard analytics.google.com/analytics/web/ ga-dev-tools.web.app/dimensions-metrics-explorer/ richardswinbank.net/adf/access_google_analytics_with_azure_data_factory richardswinbank.net/adf/extract_data_from_google_analytics_with_azure_data_factory
Databricks: Configure a Blob storage as a MountPoint
มุมมอง 1603 ปีที่แล้ว
Databricks: Configure a Blob storage as a MountPoint
DataFactory: Creating an IBM Informix linked service
มุมมอง 9523 ปีที่แล้ว
This video will explain the steps required to be able to create a linked service for IBM Informix
Data Factory: using Key Vault on Linked Services
มุมมอง 3K3 ปีที่แล้ว
In this video you can see how to create a Linked Service using a Key Vault, step by step
Data Factory: Foreach loop from SQL db to Blob container
มุมมอง 4794 ปีที่แล้ว
In this video, I'm going to show you how to load data coming from an Azure SQL database into multiple files in a blob storage container, using a Foreach loop activity
Power BI: Creating and consuming a Streaming dataset
มุมมอง 9664 ปีที่แล้ว
In this video, I'm going to show the required steps to create and consume a Power Streaming API, sending data to it with PowerShell.
Azure: Deploying resources with Powershell
มุมมอง 2524 ปีที่แล้ว
Deploying and creating Azure resources with PowerShell from your local machine
Power BI: working with JSON file stored in a Blob Storage
มุมมอง 8K4 ปีที่แล้ว
This video shows you how to connect from Power BI to Azure blob storage getting the data contained in a JSON file. Also, the process of transformation in this JSON file to be readable. Enjoy it!!
Power BI: Azure table storage to Power BI
มุมมอง 5K4 ปีที่แล้ว
Power BI: Azure table storage to Power BI
Data Factory: API data to Azure Table storage
มุมมอง 25K4 ปีที่แล้ว
Data Factory: API data to Azure Table storage
Can this work the other way round? Output/Sink to SharePoint online and replace excel file if exists.
Never tried that. Maybe there is an API you can try for this.
You are amazing!!! If you are doing an xlsx file, instead of Binary use xlsx file type and you can copy right into SQL rather than having to drop into data lake or blob storage first.
I don't think you can do that, but give it a try and let us know if it's possible. Anyway it's a good practice to keep an archive of the data you are processing, therefore, keeping a copy should be the best approach in most of the cases.
This is a useful tips but I have question, if you have multiple files in SharePoint document folder how do you read the folder metadata to get ChildItem to iterate in the ForEach activity since metadata activity has no connector for SharePoint?
Sorry, but that's not possible on SharePoint. I couldn't find a way to do that.
Extremely useful and explained much better than any other article that I've come across on getting this setup - thanks for saving me hair loss!
Thanks for the feedback, happy to help
Hi Javibi,I'm getting an error after 4:34 when I click on "Trust," which says, "Sorry, you don't have access." Could this be because I don’t have access to the SharePoint site, or should I inform the Azure admin to grant the necessary API permissions for the app? I see that the permissions are currently "Not granted." Please advise.
Yes, seems so. Usually this part of the task is responsibility of IT o SharePoint admins, so maybe they need to do that for you
Thanks for the great video. Can I also copy excel stored in sharepoint library into SQL database using same method?
Sure, in two steps atl least. SharePoint to data lake, data lake to database
Can I also upload the data to Sharepoint?
Didn't find an API to do so, but maybe there's a way
I want pull the secrent form Keyvault how can I do that
Look I at my videos, I have another one exactly for that
How can you copy multiple files in a particualr sharepoint folder
I don't think you can push them with ADF
How to do it for multiple files like entire sp library
I don't think that is feasible in a dynamic way. Files need to be hard coded 1 by 1
@javibi, What an excellent video!!!. This process also works for MS Fabric? In my case, I will copy txt data from SharePoint to the bronze layer in MS Fabric.
I guess it works, but I would say that One Lake at some point should replace SharePoint as a file storage
Error during server relative url, says specified value not supported
Probably you have something wrong somewhere
is anonymous access mandatory for this process to work? our SP replies with "... Type=System.Net.WebException,Message=The remote server returned an error: (403) Forbidden" i can run the link moving the file into $value both from my personal browser aswell as for a technical user i gave access to the folder where the file is ... but running it from ADF results in the (shortened) error message from above
So useful! Excuse me, i have a question, can redirect data after use ADF to Azure datalake gen 2. Thanks!
Yes, absolutely
Hi how to get multiple files from sharepoint site folder
Using multiple activities for each of them, but you can't dynamically get the names of the files. They should be hardcoded
Thank You for the Video, I've followed all the steps, but I still get the following error. "ErrorCode=HttpRequestFailedWithClientError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Http request failed with client error, status code 403 Forbidden, please check your activity settings. If you configured a baseUrl that includes path, please make sure it ends with '/'
Hi Did you get the resolution for the above error??
Maybe permissions were not properly granted?
HI, Thank You for the Video. I followed all the steps but still I am getting below Error. "Operation on target Copy from Share Point failed: ErrorCode=HttpRequestFailedWithClientError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Http request failed with client error, status code 403 Forbidden, please check your activity settings. If you configured a baseUrl that includes path, please make sure it ends with '/'." Please help on this.
Hi Did you get the resolution for the above error??
Maybe permissions were not properly granted?
How to extract items from sharepoint to ADF?
Follow the steps
I followed all the steps and i have the following error : ErrorCode=HttpRequestFailedWithUnauthorizedError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Http request failed with status code 401 Unauthorized, usually this is caused by invalid credentials, please check your activity settings. Request URL: help my plis
@javibi
Were you able to make it run @jeancid7702?
Hi, It was a great video. I wanted to know if I am copying data from sharepoint to ADLS and if I delete one file from sharepoint then I want to delete the same file from ADLS too. How can I implement this?
I don't think that there's a way to do that.
I keep getting this error; "Input Payload is invalid, validation result - '[\"Connector or activity name: Datalake_LSb00ee7af200d441da59965050eb96f7d, connector or activity type: AzureBlobFS, error: The domain of this endpoint is not in allow list. Original endpoint: does any one faced it?
It's not clear to me what this issue is
Great tutorial! Wanted to know if there is a way to use this to pull down the entire documents library?
No, I don't think so. File by file
simple and straight , nice video. can you please make video like vararible value in dev and variable value in prd is different . like variable in container name ( var1 = dev_calls) when migrated to prod same var1=prod_call
I would think about it. Thanks for the idea.
Is it possible to give the app permission to all sharepoint sites in your tenant? Instead of 1 by 1
I'm not expert in SharePoint, but I would say it's not possible.
Hi, why it don't have a HTTP source??
Probably you are looking at the wrong place. Check it again
Exactly what I needed and nothing more. This was perfect. Tnx a lot!
Thanks for the feedback!
Thanks for sharing, I have tried the same steps to get data from a SharePoint list and it keeps saying Unauthorized, I already checked the URL passed to the copy activity returns values. I did not see access to SharePoint was given when the app was registered in Azure App Registration. Can this be the reason?
This method is to export files from a SharePoint site. For lists there is a connector that you can use
@@javibi thanks for responding, I am having a really hard time setting up the linked service, I registered the app in Azure, gave full permissions, and I am using the client id and secret but no success. Have you done this before? Any ideas? Thanks!
Yes I did it a couple of times. I'll suggest you to test the process on Postman with hardcoded values to check if it works. This way you confirm that the client Is and setting are working fine. Probably it is just a small thing you missed
I was finally able to connect using Graph API. Thank you!
Very good explaination!!!Thank you!
Thanks!!
Great tutorial! Is there a way to make the filename dynamic for xlsx? So if we have filename.xlsx, how do we do it for *.xlsx or %file%.xlsx to capture anything named file or all files in the directory?
You can build the url dynamically, but I don't think you can't get dynamically the list of files in a folder.
supper clear! Thanks for your time and effort!
Thank you!
Hello ! thank youu for the video.. I suppose we can do reverse the same way? by switching sink and source in copy data? from datalake to sharepoint? if not, why? thank you again 👍
I don't think that there's a way to do that. The idea of data platforms is to integrate the data in them, not to move it away.
Hii i hv built the pipeline like you said and it was working very well, but suddenly it is throwing error. I also checked the linked service which is also testing connection successfully. But i'm getting the error ErrorCode=HttpFileFailedToRead,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to read data from http server. Check the error from http server:The underlying connection was closed: An unexpected error occurred on a send.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.WebException,Message=The underlying connection was closed: An unexpected error occurred on a send.,Source=System,''Type=System.IO.IOException,Message=Authentication failed because the remote party has closed the transport stream.,Source=System,' can you let me know what may be the issue ??
As per the Lastest update by Microsoft. This process of http is no more acceptable
Maybe, with the Fabric approach is SharePoint who will be replaced with One Lake
Hi javibi , after giving appid while lookuping app name not coming, iam using different adf account and different SharePoint
I think both should be in the same tenant
Straight to the point, no bs. Thanks for the tutorial.
Thanks!
Outstanding!!! Thank you!!!
Thanks!
can we try this with .xlsx file. I done with .xlsx file with utf-8 encoding, but still i'm finding � like characters when i open in notepad. Is there any way to this ?
With Binary option for the destination this shouldn't be an issue. Try to then transform the file into parquet with the right encoding
@@javibi when i open this $value file after downloading for the .xlsx file i'm seeing like PK ! ýÉ7¯ ™ ×[Content_Types].xml but when i try with .csv file, i can see the content in the $value file. Why is it happening, i hv also checked the encoding
If it's an xlsx file. You need to open it with excel, not as XML or CSV. The only way to see it properly is with excel, if not you are going to see weird characters
Is it requires Microsoft Grap API Permission?
No, nothing to do with that.
So well explained. Well done man. Thank you.
Thank you!!
hello there, the video is such usefull, have you posted any other video copying data from an excel file?
It doesn't matter if you want to extract a CSV, and Excel file or whatever other type. The process is exactly the same
thanks, very informative.
Thanks
Hi Sir, Thanks for the video. How to capture the info of modified by and Last updated time from share point to maintain a audit log.
I don't think SharePoint allows that. There's an endpoint that returns metadata from a specific folder, but I don't think that will solve your issue
I already have the data in json format in the web activity, it is according to your video but how I took this data to azure blob storage or table storage you did not show that part, thank you very much if you can help me pleaseeee
I have another tutorial to split the data in columns. Is not completely what you want but you can have some good ideas from it.
I have created a project in Visual Studio and created the function GetOAuthToken but when I publish it, I am not able to see the function under my GetGoogleAnalyticsToken Function App in Azure. I am not sure what to do at this point with the code. Is there anyway I can get a copy of the program from someone who has made it work?
can we connect ibm i series ?
I don't now, but I'm sure not with this method
it works, thanks!
Nice, thank you!
If I have more than one property in analytics, should I give access to the API email for each of them?
I don't think so, you just need to enable the properties
Excellent!
Thanks!
Graciela!
A ti!
Get this error on last step: ErrorCode=HttpRequestFailedWithClientError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Http request failed with client error, status code 411 LengthRequired, please check your activity settings.
Are you adding the header? 411 usually raise when you are missing it
@@javibi Found the error . Thanks!