How to deploy Azure Automation Runbooks with source control through Azure DevOps

I use Azure Automation for orchestrating alot of my Powershell automation. I use it for both my Azure environment but also my on-premises environment. I find Azure automation to be a great solution for managing and running your automation, instead of having them run on a server with scheduled tasks.

One of the best things about running the scripts in Azure Automations is that you have the possibility to upload your scripts to Azure through Source Control, such as Github or Azure DevOps.

With the possibility to use Source Control you also get the possibility to set up a pipeline for automating the deployment of Runbooks to Azure. Here you could set up a pipeline that used PSScryptAnalyzer and Pester, for analyzing and testing your script before it gets uploaded to Azure. You could also just set up a pipeline which when you push into the main branch, automatically uploads the script into Azure.

in this blog post, I am going to demonstrate how you very easily can set up a pipeline on Azure DevOps, which uploads your Powershell scripts into Azure Automation whenever to push into the main branch.

The environment and prerequisites

I have created a repository in Azure DevOps, this could just as well be on Github, the procedure is more or less the same.

In my repository i have create a folder structure like so:

Root
|__runbooks\
|		|__TestRunbook1\
|		|		|__TestRunbook1.ps1
|		|__TestRunnbook2\
|				|__TestRunbook2.ps2
|
|__scripts\
|		|__run.ps1
|
|__azure-pipelines.yml

I have created a folder called runbooks. This folder will contain a folder with the exact name of the runbook script, and inside this folder, I have the script. The way I have created this pipeline the runbook folder and script must have the same name.

Inside the scripts folder, I have created a single script called run.ps1. This script is used by the pipeline, so this will handle the functionality of actually connecting to Azure and deploying the runbooks.

The last file in the repository is the azure-piipelines.yml file. This file is the pipeline created in Azure DevOps.

To create a new pipeline in Azure DevOps you can go to “Pipelines” –> “New Pipeline”

Then select which repository you want to use your pipeline with. In my case, I have my repository stored in Azure DevOps

Then you will need to select your repository and select “Starter pipeline”

And then you should be shown a yml file looking like this:

What is usually do is just, delete everything. Save the pipeline, without running and then in my visual studio code editor I will pull the latest changes, so I can edit the pipeline in vscode.

Creating the run.ps1 script to use with the pipeline

The run.ps1 script is the main functionality of running the pipeline. The script needs to be able to do two things. The first thing is that it needs to connect to Azure, and the second thing is that it needs to deploy the runbooks to an Azure Automation account.

Connecting to Azure

First, I will need the script to connect to Azure so that it can deploy the runbooks to my Automation Account. To do this I will need the Powershell modules Az.Resources, Az.Accounts and Az.Automation. Az.Accounts, I will need for connecting to Azure, and Az.Automation I will need for deploying the runbooks. Az.Resources are used for searching in the resource group for an Automation Accountname.

Another thing I will need is a Service Principal which has permission to connect to my Azure Automation Account. If you don’t know how to create a Service Principal you can read my blog post HERE.

Since I will need the two Powershell modules, I will need my pipeline runner(the cloud instance which runs the code in the pipeline) to make sure it has the modules installed.

if (-not(Get-Module -Name Az.Accounts -ListAvailable)){
	Write-Warning "Module 'Az.Accounts' is missing or out of date. Installing module now."
	Install-Module -Name Az.Accounts, Az.Automation, Az.Resources -Scope CurrentUser -Force -AllowClobber
}

The second thing i need to do, is to connect to Azure with the Service Principal:

$ServicePrincipalPassword = ConvertTo-SecureString -AsPlainText -Force -String $ServicePrincipalPass
$azureAppCred = New-Object System.Management.Automation.PSCredential ($ServicePrincipalName,$ServicePrincipalPassword)
Connect-AzAccount -ServicePrincipal -Credential $azureAppCred -Tenant $tenantId -Subscription $SubscriptionId

Deploying the Runbooks

The next part of the pipeline is to deploy the runbooks. To do this I will use two Powershell cmdlets found in the module: Az.Automation

– Start-AzAutomationSourceControlSyncJob

– New-AzAutomationSourceControl

Start-AzAutomationSourceControlSyncJob will start a synchronization between your repository and the script located in Azure Automation, so if your runbooks already have been deployed, and you just created some changes which need to be updated, you would run this command.

New-AzAutomationSourceControl is used to deploy new runbooks into your Azure Automation Account. A quick note about this command is that it will need to take a “-FolderPah” parameter. This is the path in your repository to where the runbook is located. This is the reason I have placed the runbooks inside a folder, named exactly like the Powershell script.

Finding the Automations Account name and already existing runbooks

The first thing I need to do is to find the automation account name and to do this since in my environment I only have a single automation account per resource group, I can just search the resource group for a specific resource and grab the name.

$AutomationAccountName = (Get-AzResource -ResourceGroupName $ResourceGroupName | Where-Object ResourceType -eq "Microsoft.Automation/automationAccounts").name

You could also just hardcode the automation account name into the variable or pass it as a parameter to your script, instead.

The next thing I need is to find all the runbooks located in my repository and the runbooks located in Azure Automation. I can compare and check if any local runbooks haven’t been uploaded yet.

$Runbooks = (Get-ChildItem -Path "./runbooks").Name
$AutomationSourceControl = Get-AzAutomationSourceControl -ResourceGroupName $ResourceGroupName -AutomationAccountName $AutomationAccountName
$AutomationSourceControlList = New-Object -TypeName System.Collections.ArrayList
foreach($sourceControl in $AutomationSourceControl){
	$AutomationSourceControlList.Add($sourceControl.Name)
}

Here I will grab all the names of the folders inside the runbooks folder in my repository. Then I will use the cmdlet Get-AzAutomationSourceControl to find all the runbooks located in the Automation Account. Then I will create an array containing all the names of the runbooks located in the Automation Account.

Choosing to either synching the existing runbooks or deploying the new runbooks

In this section, I will create a foreach loop that will go through all of the runbooks located in my repository. If the runbook already exists in the Automation Account it will start a Sync Job with the command: Start-AzAutomationSourceControlSyncJob. And if it doesn’t exist in the Automation Account it will create a new AutomationSourceControl with the command: New-AzAutomationSourceControl and upload the runbook to the Automation Account. Once the new runbook is uploaded I will start synchronization of the runbook.

# Switch parameter to activate this section of the script
if($DeployRunbooks.IsPresent) {
    $Runbooks = (Get-ChildItem -Path "./runbooks").Name
    $AutomationAccountName = (Get-AzResource -ResourceGroupName $ResourceGroupName | Where-Object ResourceType -eq "Microsoft.Automation/automationAccounts").name
    $AutomationSourceControl = Get-AzAutomationSourceControl -ResourceGroupName $ResourceGroupName -AutomationAccountName $AutomationAccountName
    # Adding the names of existing runbooks into an Array
    $AutomationSourceControlList = New-Object -TypeName System.Collections.ArrayList
    foreach($sourceControl in $AutomationSourceControl){ 
        $AutomationSourceControlList.Add($sourceControl.Name)
    }
    
    # foreach loop running through all the runbooks located in your repository
    foreach($Runbook in $Runbooks) {
        if($AutomationSourceControlList -contains $Runbook) {
            # If the runbook exists in Azure, then just run a sync on it
            try {
                Write-Verbose -Message "Runbook: $($Runbook) was found in Automation Source Control list. Updating source code now"
                Start-AzAutomationSourceControlSyncJob -SourceControlName $Runbook -ResourceGroupName $ResourceGroupName -AutomationAccountName $AutomationAccountName
            }
            catch {
                Write-Error -Message "$($_)"
            }
        }
        else {
            # If the runbooks doesn't exist, the create a new source control job, and sync it to Azure
            Write-Verbose -Message "Runbook hasn't been connected with Azure Automation. Uploading source code for runbook"
            $FolderPath = "/runbooks/" + $Runbook + "/"
            try {
                New-AzAutomationSourceControl -ResourceGroupName $ResourceGroupName `
                    -AutomationAccountName $AutomationAccountName `
                    -Name $Runbook `
                    -RepoUrl $RepoURL `
                    -SourceType $SourceControlType `
                    -Branch $SourceControlBranch `
                    -FolderPath $FolderPath `
                    -AccessToken (ConvertTo-SecureString $RepoAccessToken -AsPlainText -Force)
                
                Start-AzAutomationSourceControlSyncJob -SourceControlName $Runbook -ResourceGroupName $ResourceGroupName -AutomationAccountName $AutomationAccountName
    
            }
            catch {
                Write-Error -Message "$($_)"
            }
        }
    }
}

In the command: New-AzAutomationSourceControl you will need to specify the branch and the SourceType. In my case, since I am using Azure DevOps the SourceType is “VsoGit” if you were using Github, the SourceType would be: “Github”. And I have chosen the branch main. This means that the runbooks in Azure Automation will synchronize with your main branch.

The Access token is a personal access token either created in Github or Azure DevOps. You can read here how to create a PAT: How to create a PAT in Azure DevOps and How to create a PAT in Github

The parameter “RepoUrl” is the actual repository URL, the same one you use when you want to clone your repository to your local machine.

To split up the two tasks of the Powershell script: Connecting to Azure, and Deploying the Runbooks. I will use Switch parameters to activate the parts of the script when I need to call it.

So the first section will have a switch parameter called $ConnectAzure and when I run the script and call that parameter, only then will that code run. In the second part, the parameter is called: $DeployRunbooks

The complete script will look like so:

param (
    [Parameter(Mandatory=$false)][String]$ResourceGroupName,
    [Parameter(Mandatory=$false)][String]$ServicePrincipalName,
    [Parameter(Mandatory=$false)][String]$ServicePrincipalPass,
    [Parameter(Mandatory=$false)][String]$SubscriptionId,
    [Parameter(Mandatory=$false)][String]$TenantId,
    [Parameter(Mandatory=$false)][String]$RepoURL,
    [Parameter(Mandatory=$false)][String]$RepoAccessToken,
    [Parameter(Mandatory=$false)][String]$SourceControlType = "VsoGit",
    [Parameter(Mandatory=$false)][String]$SourceControlBranch = "main",
    [Parameter(Mandatory=$false)][Switch]$ConnectAzure,
    [Parameter(Mandatory=$false)][Switch]$DeployRunbooks
)


#Region - Connecting to Azure
if($ConnectAzure.IsPresent) {
    Write-Verbose -Message "Checking and Installing Azure Powershell Module"
    if (-not(Get-Module -Name Az.Accounts -ListAvailable)){
        Write-Warning "Module 'Az.Accounts' is missing or out of date. Installing module now."
        Install-Module -Name Az.Accounts, Az.Resources, Az.Automation -Scope CurrentUser -Force -AllowClobber
    }

    Write-Verbose -Message "Connecting to Azure"
    $ServicePrincipalPassword = ConvertTo-SecureString -AsPlainText -Force -String $ServicePrincipalPass
    $azureAppCred = New-Object System.Management.Automation.PSCredential ($ServicePrincipalName,$ServicePrincipalPassword)
    Connect-AzAccount -ServicePrincipal -Credential $azureAppCred -Tenant $tenantId -Subscription $SubscriptionId
}
#EndRegion


#Region - Deploying the Azure Runbooks
if($DeployRunbooks.IsPresent) {
    $Runbooks = (Get-ChildItem -Path "./runbooks").Name
    $AutomationAccountName = (Get-AzResource -ResourceGroupName $ResourceGroupName | Where-Object ResourceType -eq "Microsoft.Automation/automationAccounts").name
    $AutomationSourceControl = Get-AzAutomationSourceControl -ResourceGroupName $ResourceGroupName -AutomationAccountName $AutomationAccountName
    # Adding the names of existing runbooks into an Array
    $AutomationSourceControlList = New-Object -TypeName System.Collections.ArrayList
    foreach($sourceControl in $AutomationSourceControl){ 
        $AutomationSourceControlList.Add($sourceControl.Name)
    }
    
    # foreach loop running through all the runbooks located in your repository
    foreach($Runbook in $Runbooks) {
        if($AutomationSourceControlList -contains $Runbook) {
            # If the runbook exists in Azure, then just run a sync on it
            try {
                Write-Verbose -Message "Runbook: $($Runbook) was found in Automation Source Control list. Updating source code now"
                Start-AzAutomationSourceControlSyncJob -SourceControlName $Runbook -ResourceGroupName $ResourceGroupName -AutomationAccountName $AutomationAccountName
            }
            catch {
                Write-Error -Message "$($_)"
            }
        }
        else {
            # If the runbooks doesn't exist, the create a new source control job, and sync it to Azure
            Write-Verbose -Message "Runbook hasn't been connected with Azure Automation. Uploading source code for runbook"
            $FolderPath = "/runbooks/" + $Runbook + "/"
            try {
                New-AzAutomationSourceControl -ResourceGroupName $ResourceGroupName `
                    -AutomationAccountName $AutomationAccountName `
                    -Name $Runbook `
                    -RepoUrl $RepoURL `
                    -SourceType $SourceControlType `
                    -Branch $SourceControlBranch `
                    -FolderPath $FolderPath `
                    -AccessToken (ConvertTo-SecureString $RepoAccessToken -AsPlainText -Force)
                
                Start-AzAutomationSourceControlSyncJob -SourceControlName $Runbook -ResourceGroupName $ResourceGroupName -AutomationAccountName $AutomationAccountName
    
            }
            catch {
                Write-Error -Message "$($_)"
            }
        }
    }
}
#EndRegion

Creating the pipeline yml file

The pipeline for deploying the runbooks is fairly simple. All it needs to do in this case is create a new runner instance and then run the Powershell script run.ps1. I have chosen to split the script into two parts. This means that the pipeline will create two tasks. The first task will connect to Azure and make sure the needed Powershell module is installed. The second task will deploy or synchronize the runbooks with Azure Automation.

I will set the trigger to be my “main” branch so that when a push or pull request has been made into the branch the pipeline will be activated. Then I create a stage and create a runner, running the vmImage ‘ubuntu-latest’

trigger:
- main

stages:
- stage: DeployRunbook
  pool:
    vmImage: 'ubuntu-latest'

Then I create a new job called: deploy_runbooks and inside this job I create two tasks. The task is using the action: PowerShell@2, running a Powershell script inline. The task is just calling the run.ps1 script with the switch parameter depending on what part of the script is called.

jobs:
- job: deploy_runbooks
displayName: 'Deploy Runbooks'
steps:
  - task: PowerShell@2
	displayName: 'Connecting to Azure'
	inputs:
	  targetType: 'inline'
	  script: ./scripts/run.ps1 -ConnectAzure -ResourceGroupName "sc-dev-aa-test-rg" -ServicePrincipalName $(ServicePrincipalName) -ServicePrincipalPass $(ServicePrincipalPass) -SubscriptionId $(SubscriptionId) -TenantId $(TenantId) -Verbose
  - task: PowerShell@2
	displayName: 'Deploying Runbooks'
	inputs:
	  targetType: 'inline'
	  script: ./scripts/run.ps1 -DeployRunbooks -RepoURL $(RepoURL) -RepoAccessToken $(RepoAccessToken) -ResourceGroupName "sc-dev-aa-test-rg" -Verbose

In the parameters for calling the script, you will see variables looking like this: $(RepoURL). These are the environment variables I have created for the pipeline. These a just a way to store information, even secrets in a variable to pass into your pipeline.

To create an Environment variable in Azure DevOps you can go into your pipeline and press “Edit”

Then click on Variables and then the “+” sign to create a new variable

The complete pipeline yaml file will look like this:

trigger:
- main

stages:
- stage: DeployRunbook
  pool:
    vmImage: 'ubuntu-latest'
  jobs:
  - job: deploy_runbooks
    displayName: 'Deploy Runbooks'
    steps:
      - task: PowerShell@2
        displayName: 'Connecting to Azure'
        inputs:
          targetType: 'inline'
          script: ./scripts/run.ps1 -ConnectAzure -ResourceGroupName "sc-dev-aa-test-rg" -ServicePrincipalName $(ServicePrincipalName) -ServicePrincipalPass $(ServicePrincipalPass) -SubscriptionId $(SubscriptionId) -TenantId $(TenantId) -Verbose
      - task: PowerShell@2
        displayName: 'Deploying Runbooks'
        inputs:
          targetType: 'inline'
          script: ./scripts/run.ps1 -DeployRunbooks -RepoURL $(RepoURL) -RepoAccessToken $(RepoAccessToken) -ResourceGroupName "sc-dev-aa-test-rg" -Verbose

Testing out the pipeline

If you go to your Azure Automation account you can scroll down to “Account Settings” and you should see “Source Control”. Inside this setting you should at this point not see any Source Control jobs.

So let’s start the pipeline by pushing the code into the main branch

If you go into your pipeline you should now see that it has started and is running:

If you go into the jobs, you can see that it has successfully created the Source Control Jobs and is synching the runbooks with Azure

And now if I go into my Automation Account I should see the Source Control Sync Jobs has been created.

And if I go into my Runbooks I should now see my two runbooks which have been uploaded.

From here I can either run the Runbooks, Create webhooks for them you just schedule them to run at certain times.

The awesome thing about this is that whenever you want to create a new runbook, you just create a new folder inside the folder \\runbooks and then your runbooks script inside that folder. Once the script is done you can just push your code into your main branch and it will automatically deploy the new runbook into Azure Automations.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

This website uses cookies. By continuing to use this site, you accept our use of cookies.