Great video. Straight to the point and informative. I really appreciate that you do not just go through the setup steps that you can find 1000 other places but you also answer how, when and why the agents scale, a lot of docs leave that info out.
Thanks! In my video I usually try to get beyond the docs that are readily available adding information that I have gained through experience or experimentation. I want to give the highest value I can to my viewers. And I’m very happy to know this came through in this video ☺️
hey. So the price of VMSS is the same as for normal Azure VMs... so when you create a VMSS you can select the size and therefore you know they price for a single agent. Things get a little more complex if you wanna know the overall cost per month... that would depend on the number of max agents you can scale to, and how much utilization you would have on the platform
Could you please explain how I can change the basic image, that I use for VMSS? For example, if I need use the Windows-based agent and I need to install Azure CLI or .Net6 for my pipelines. I want to create an image with the pre-installed necessary tools and then use it for my VMSS.
Hi, you can create a custom Scale Set Template, that uses a custom image. And in the custom image you can install anything you want/need. You can read more about it here: docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-mvss-custom-image
Hi, can you please create a video on how to automatically scale our own hosted agents (fargate agents). The scheduling probably in the form of the lambda. Thank you so much, i need it asap.
Hi Friend! Graet video! I need to create it from a script. The only way I founded was creating with Rest API, but I can't connect the Instances with a Agent into the Agent Pool. Do you ever have a similar issue?
@@CoderDave Hi! Exact! But Terraform has no module for that. We already tried with REST API and although the scale set is generated and scaling is achieved, but on the AzDevops side no agent appears within the Agent Pool.
Did u have to install first the deveops agent in the image and after that create the vms? or no need to manually install the agent? I didnt install it, and agents dosent show up in the pool...
No, I didn't have to install anything manually. AzDO does that for you. Just create the VMSS in Azure, and then create a new Agent Pool in AzDO selecting the VMSS as target
What are the conditions to get the jobs running in parallel? I have 2 parallel jobs allowed by Azure DevOps (private project, 1 default + 1 for Visual Studio subscription). I have a stage with ~20 jobs with no interdependencies. I have a VMSS-based agent pool with max pool size set to 4 machines. And still only one job is running at a time - even though I see 2 or even 4 VMs in the VMSS. Although it must be noted that I only see one entry in the agent pool with the name of the VMSS itself, not the machines it spawned.
uhm, that is interesting... there may be other factors in play here on why you can't get multiple parallel jobs, but it is hard to say without seeing the environment 🤔 Is you case maybe in one of these? learn.microsoft.com/en-us/azure/devops/pipelines/licensing/concurrent-jobs?view=azure-devops&tabs=self-hosted#how-is-a-parallel-job-consumed-in-devops-services
It is running jobs randomly in diff. machines, what if we have a pipeline that has some dependencies for Example if we have added a key vault task and we need key vault secret values for whole the pipeline then how we can manage it?
hey there. In case of key vault secrets, or other job-level environment variables, those are kept even between different machines. It's part of the featureset of Pipelines. If instead you need to use the output of one job into another one, then you have to use the upload artifact in job A, and use download artifact in job B
So we switched to vmss agents using the same specs hosted agents use docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#capabilities-and-limitations but they are EXTREMELY slower (2-3x slower) than the hosted agents. Any ideas on what might be wrong or what we could do to improve them without breaking the bank?
Hey, thanks for reaching out. Not really sure why your agents are that slow... that's quite strange. One thing I can think of is disks... perhaps the hosted agents use SSDs (don't quote me on this :) ) or it could just be the Azure region... sometimes you land on a older cluster, but that shouldn't make the agents 2-3x slower anyway For me when I went with VMSS I;ve actually noticed the opposite, my agents were slightly faster due to no noisy neighbors...
If you want to do it in AWS (or anything else that is not Azure) then you'd need to do it "manually". You could for example use the containerized agents in K8S. But the scaling mechanisms may be on the actual utilization rather than on the build/release queue... so not that efficient. Or you can use something like KEDA which takes in consideration the queue: keda.sh/blog/2021-05-27-azure-pipelines-scaler/ Maybe I can make a video on this :)
That could have an impact on the operations that require connection to the instance; like for checkout, upload/download artifacts, etc… but I don’t expect it to be huge since the traffic between azure regions still travels on the Azure Backbone
![Azure Artifacts vs Build Artifacts vs Pipeline Artifacts: Difference EXPLAINED! [Azure DevOps]](http://i.ytimg.com/vi/WWCmEUCt3Cc/mqdefault.jpg)
Great video.
Straight to the point and informative.
I really appreciate that you do not just go through the setup steps that you can find 1000 other places but you also answer how, when and why the agents scale, a lot of docs leave that info out.
Thanks! In my video I usually try to get beyond the docs that are readily available adding information that I have gained through experience or experimentation. I want to give the highest value I can to my viewers. And I’m very happy to know this came through in this video ☺️
Thank you so much!
Welcome ☺️
Thanks for your explanation.
How can I calculate how many I will pay per agent?
hey. So the price of VMSS is the same as for normal Azure VMs... so when you create a VMSS you can select the size and therefore you know they price for a single agent.
Things get a little more complex if you wanna know the overall cost per month... that would depend on the number of max agents you can scale to, and how much utilization you would have on the platform
Could you please explain how I can change the basic image, that I use for VMSS? For example, if I need use the Windows-based agent and I need to install Azure CLI or .Net6 for my pipelines. I want to create an image with the pre-installed necessary tools and then use it for my VMSS.
Hi, you can create a custom Scale Set Template, that uses a custom image. And in the custom image you can install anything you want/need. You can read more about it here: docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-mvss-custom-image
I am looking for the same. Are you able to achieve this?
Hi, can you please create a video on how to automatically scale our own hosted agents (fargate agents). The scheduling probably in the form of the lambda. Thank you so much, i need it asap.
Sorry but I can’t do that shortly , I’ve never used them in AWS so it’d take me a while to understand how they work.
Hi Friend! Graet video! I need to create it from a script. The only way I founded was creating with Rest API, but I can't connect the Instances with a Agent into the Agent Pool. Do you ever have a similar issue?
Hey, do you mean you need to create the ScaleSet with a script? Never done that, but you could use ARM or Terraform for that instead
@@CoderDave Hi! Exact! But Terraform has no module for that. We already tried with REST API and although the scale set is generated and scaling is achieved, but on the AzDevops side no agent appears within the Agent Pool.
Did u have to install first the deveops agent in the image and after that create the vms? or no need to manually install the agent? I didnt install it, and agents dosent show up in the pool...
No, I didn't have to install anything manually. AzDO does that for you.
Just create the VMSS in Azure, and then create a new Agent Pool in AzDO selecting the VMSS as target
What are the conditions to get the jobs running in parallel? I have 2 parallel jobs allowed by Azure DevOps (private project, 1 default + 1 for Visual Studio subscription). I have a stage with ~20 jobs with no interdependencies. I have a VMSS-based agent pool with max pool size set to 4 machines. And still only one job is running at a time - even though I see 2 or even 4 VMs in the VMSS. Although it must be noted that I only see one entry in the agent pool with the name of the VMSS itself, not the machines it spawned.
uhm, that is interesting... there may be other factors in play here on why you can't get multiple parallel jobs, but it is hard to say without seeing the environment 🤔
Is you case maybe in one of these? learn.microsoft.com/en-us/azure/devops/pipelines/licensing/concurrent-jobs?view=azure-devops&tabs=self-hosted#how-is-a-parallel-job-consumed-in-devops-services
Always on point.
Thanks!
How do I switch from the current set up (single self hosted agents) to VMSSA? Is there any documentation with steps to follow?
What do you mean by "switch"? you have to create a new VMSS with the agent installed and add it to AzDO
It is running jobs randomly in diff. machines, what if we have a pipeline that has some dependencies for Example if we have added a key vault task and we need key vault secret values for whole the pipeline then how we can manage it?
hey there. In case of key vault secrets, or other job-level environment variables, those are kept even between different machines. It's part of the featureset of Pipelines.
If instead you need to use the output of one job into another one, then you have to use the upload artifact in job A, and use download artifact in job B
@@CoderDaveHi, thank you for your reply. Can I run all the jobs/stages of a pipeline on a single machine of the VMSS?
AFAIK there is no way to control that, it’s up to the azure pipelines engine
thank you
You are very welcome ☺️
So we switched to vmss agents using the same specs hosted agents use docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#capabilities-and-limitations but they are EXTREMELY slower (2-3x slower) than the hosted agents. Any ideas on what might be wrong or what we could do to improve them without breaking the bank?
Also how can you do the auto scaling in AWS instead of a VMSS?
Hey, thanks for reaching out. Not really sure why your agents are that slow... that's quite strange. One thing I can think of is disks... perhaps the hosted agents use SSDs (don't quote me on this :) ) or it could just be the Azure region... sometimes you land on a older cluster, but that shouldn't make the agents 2-3x slower anyway
For me when I went with VMSS I;ve actually noticed the opposite, my agents were slightly faster due to no noisy neighbors...
If you want to do it in AWS (or anything else that is not Azure) then you'd need to do it "manually". You could for example use the containerized agents in K8S. But the scaling mechanisms may be on the actual utilization rather than on the build/release queue... so not that efficient.
Or you can use something like KEDA which takes in consideration the queue: keda.sh/blog/2021-05-27-azure-pipelines-scaler/
Maybe I can make a video on this :)
@@CoderDave My agents aren't in the same region as our org, do you think that would factor in at all?
That could have an impact on the operations that require connection to the instance; like for checkout, upload/download artifacts, etc… but I don’t expect it to be huge since the traffic between azure regions still travels on the Azure Backbone