Hi Will, thank you for your tutorials! I have a question regarding Fabric. Can we set limits in Fabric to automatically shut down any operation that reaches the cost limit? I recently heard about a situation in my company where a process was left running over the weekend, resulting in costs of approximately 70,000 Euros on Monday. While there might have been an error, I wonder if implementing a kill-cost limit could have prevented this situation. Any insights on this? Thank you! P.S.: At the end no more Fabric in the company 😞
Hi, I'm running into a problem with loading multiple CSV files into a table from a lakehouse. I have 12 CSV files, each with common columns like 'date' and 'employee.' However, some files also have extra columns related to payroll items. My goal is to append all these files into a single table but only keep the common columns like 'employee' and 'date.' The issue I'm facing is that, when the files are appended, the data doesn't align correctly under the corresponding columns. Any help on how to resolve this would be greatly appreciated!
Yes you can do this by writing sql statements directly on the file, pulling in only the columns you want spark.apache.org/docs/latest/sql-data-sources-load-save-functions.html#run-sql-on-files-directly
Hey all! I hope you’re enjoying Spark September! We’re coming to the end of the month and soon I’ll be recording a video to answer any questions you have about Spark in Fabric. Feel free to leave your questions below and I’ll answer them in a video in the next few days ☺💪👇👇
hi Will, thanks for the educational videos - a really good introduction! Went through all of them so far. I am trying to figure out how to connect a Cognitive search from Azure (and/or Azure OpenAI) with a Fabric LH. Do you know if there is a tutorial/video on this? Thanks again for your great work.
Hey thanks for watching! Yes I've seen a few things on this already - it's something I plan to create some videos on very shortly. Here's some of the things I've seen so far: learn.microsoft.com/en-us/fabric/data-science/open-ai erwindekreuk.com/2023/09/azure-open-ai-and-microsoft-fabric/
In step 3 they refer to: service_name = "synapseml-openai" deployment_name = "gpt-35-turbo" deployment_name_embeddings = "text-embedding-ada-002" I am not sure where to find service_name and deployment_name_embeddings. In the Azure OpenAI deployment, I can find endpoint and keys. This sounds like a very basic question, but I would appreciate a video or a tip on how to establish this connection. @@LearnMicrosoftFabric
the service_name and deployment_name_embeddings are exactly as you’ve written above, you don’t need to change them. they are string values that define the settings you pass into the API; completion = ( OpenAICompletion() .setSubscriptionKey(key) .setDeploymentName(deployment_name) .setCustomServiceName(service_name) .setMaxTokens(200) .setPromptCol("prompt") .setErrorCol("error") .setOutputCol("completions") )
Hi Will, thank you for your tutorials! I have a question regarding Fabric. Can we set limits in Fabric to automatically shut down any operation that reaches the cost limit? I recently heard about a situation in my company where a process was left running over the weekend, resulting in costs of approximately 70,000 Euros on Monday. While there might have been an error, I wonder if implementing a kill-cost limit could have prevented this situation. Any insights on this? Thank you!
P.S.: At the end no more Fabric in the company 😞
Hi, I'm running into a problem with loading multiple CSV files into a table from a lakehouse. I have 12 CSV files, each with common columns like 'date' and 'employee.' However, some files also have extra columns related to payroll items.
My goal is to append all these files into a single table but only keep the common columns like 'employee' and 'date.' The issue I'm facing is that, when the files are appended, the data doesn't align correctly under the corresponding columns.
Any help on how to resolve this would be greatly appreciated!
Yes you can do this by writing sql statements directly on the file, pulling in only the columns you want spark.apache.org/docs/latest/sql-data-sources-load-save-functions.html#run-sql-on-files-directly
Hey all! I hope you’re enjoying Spark September! We’re coming to the end of the month and soon I’ll be recording a video to answer any questions you have about Spark in Fabric. Feel free to leave your questions below and I’ll answer them in a video in the next few days ☺💪👇👇
hi Will, thanks for the educational videos - a really good introduction! Went through all of them so far. I am trying to figure out how to connect a Cognitive search from Azure (and/or Azure OpenAI) with a Fabric LH. Do you know if there is a tutorial/video on this? Thanks again for your great work.
Hey thanks for watching! Yes I've seen a few things on this already - it's something I plan to create some videos on very shortly.
Here's some of the things I've seen so far:
learn.microsoft.com/en-us/fabric/data-science/open-ai
erwindekreuk.com/2023/09/azure-open-ai-and-microsoft-fabric/
In step 3 they refer to:
service_name = "synapseml-openai"
deployment_name = "gpt-35-turbo"
deployment_name_embeddings = "text-embedding-ada-002"
I am not sure where to find service_name and deployment_name_embeddings.
In the Azure OpenAI deployment, I can find endpoint and keys.
This sounds like a very basic question, but I would appreciate a video or a tip on how to establish this connection.
@@LearnMicrosoftFabric
the service_name and deployment_name_embeddings are exactly as you’ve written above, you don’t need to change them. they are string values that define the settings you pass into the API;
completion = (
OpenAICompletion()
.setSubscriptionKey(key)
.setDeploymentName(deployment_name)
.setCustomServiceName(service_name)
.setMaxTokens(200)
.setPromptCol("prompt")
.setErrorCol("error")
.setOutputCol("completions")
)