Testing Azure Pipelines YAML templates

At ClearBank Azure Pipelines is used extensively for builds and releases. We maintain a large library of internal YAML templates which developers can extend from to provide safe, repeatable releases of code to production. Using templates for our releases has several advantages:

  • Developers can focus on writing and releasing code without having to know the detailed mechanism of how it is released
  • We can include security and quality checks in the pipelines
  • We ensure all release follow the same process and have the correct approvals

For the team that maintains these templates it is vital that they can make updates and add features without breaking the templates our developers rely on. To help with this we have begun adding unit and integration tests to our template files.

Aren't templates just YAML files, how do you test those?

The templates may just be YAML files, but they implement some complex logic which we need to ensure is correct before someone runs a pipeline. Our pipelines are made up of multiple templates linked with references which are merged together on the server to produce the final YAML document. The templates also contain expressions which can alter the behaviour of the pipeline based on parameters.

There are two sets of tests we run; unit tests against individual files and integration tests against the final rendered document.

Tooling

We use the Pester framework for our tests.

Install-Module Pester

We also need a way to deserialize our YAML files so we can inspect them using PowerShell. We are using an internally developed function to do this, but there are some publicly available ones such as powershell-yaml.

Unit Tests

We have multiple child templates for the different technologies that we release. In order for them to be called correctly from other templates they need to have the following four parameters.

parameters:
  - name: environment
    type: string
  - name: depends_on
    type: object
    default: []
  - name: condition
    type: string
  - name: user_parameters
    type: object

Here's an example of a test that checks the required parameters are present in each file:

#Requires -Modules @{ ModuleName="Pester"; ModuleVersion="5.1.0" }

Describe "Technologies" {
  Context "<technology.name>" {
    BeforeEach {
      # Deserialize the template file for each technology
      $template = ConvertFrom-Yaml -InputObject "$PSScriptRoot/../technologies/$($technology.name)/applyjob.yaml"
    }
    It "Has a required parameter <parameter.name> with type <parameter.type>" {
      $template["parameters"].Where( { $_["name"] -eq $parameter.name } ) | Should -HaveCount 1
      $template["parameters"].Where( { $_["name"] -eq $parameter.name } )[0]["type"] | Should -BeExactly $parameter.type
    } -ForEach @(
      @{ parameter = @{ name = "environment"; type = "string" } },
      @{ parameter = @{ name = "depends_on"; type = "object" } },
      @{ parameter = @{ name = "condition"; type = "string" } },
      @{ parameter = @{ name = "user_parameters"; type = "object" } }
    )
  } -ForEach $technologies
}

I can run the tests to check the files are correct:

Invoke-Pester .\technologies.tests.ps1 -Output Detailed

Describing Technologies
 Context azure_function
    [+] Has a required parameter environment with type string 10ms (3ms|6ms)
    [+] Has a required parameter depends_on with type object 9ms (8ms|1ms)
    [+] Has a required parameter condition with type string 5ms (4ms|1ms)
    [+] Has a required parameter user_parameters with type object 13ms (12ms|1ms)
 Context dacpac
    [+] Has a required parameter environment with type string 9ms (7ms|2ms)
    [+] Has a required parameter depends_on with type object 10ms (9ms|1ms)
    [+] Has a required parameter condition with type string 16ms (14ms|1ms)
    [+] Has a required parameter user_parameters with type object 7ms (6ms|1ms)
 Context terraform
  Context Apply Job
    [+] Has a required parameter environment with type string 13ms (10ms|3ms)
    [+] Has a required parameter depends_on with type object 12ms (10ms|2ms)
    [+] Has a required parameter condition with type string 6ms (5ms|1ms)
    [+] Has a required parameter user_parameters with type object 7ms (5ms|3ms)
Tests completed in 4.32s
Tests Passed: 12, Failed: 0, Skipped: 0 NotRun: 0

Say I add a new technology called web_app but I forget to include the condition parameter. Now when I run the tests I get a failure.

Invoke-Pester .\technologies.tests.ps1 -Output Detailed

Describing Technologies
 Context web_app
    [+] Has a required parameter environment with type string 10ms (3ms|6ms)
    [+] Has a required parameter depends_on with type object 9ms (8ms|1ms)
    [-] Has a required parameter condition with type string 31ms (28ms|2ms)
     Expected a collection with size 1, but got an empty collection.
     at $template["parameters"].Where( { $_["name"] -eq $parameter.name } ) | Should -HaveCount 1, C:\Source\cbi-yaml-templates\v3-beta\tests\technologies_v2.tests.ps1:71
     at <ScriptBlock>, C:\Source\cbi-yaml-templates\v3-beta\tests\technologies_v2.tests.ps1:71
    [+] Has a required parameter user_parameters with type object 13ms (12ms|1ms)
Tests completed in 4.32s
Tests Passed: 12, Failed: 0, Skipped: 0 NotRun: 0

Integration Tests

The unit tests are a great start, but what if we want to go further and test how our pipeline behaves after it has been rendered by the server? Normally you would have to trigger a pipeline run to see this, which is not a great way of working. Fortunately we can use the Azure Devops API to preview the final YAML file without triggering a run. Get-YamlPreview is a custom function that submits a dummy template to the preview API and retrieves the rendered template.

Here's an example of a test that verifies the correct stages are present in the correct order by checking the dependsOn setting contains the previous stage.

template.yaml:

resources:
  repositories:
    - repository: templates
      type: git
      name: cbi/cbi-yaml-templates
      ref: refs/heads/master

extends:
  template: software_release.yaml@templates
  parameters:
    environment: none
#Requires -Modules @{ ModuleName="Pester"; ModuleVersion="5.1.0" }

Describe "Software Release" {
  BeforeEach {
    $template = ConvertFrom-Yaml "$PSScriptRoot/template.yaml"
  }
  Context "Production Environments" {
    It "Deploys to staging" {
      $template["extends"]["parameters"]["environment"] = "staging"
      $rendered = Get-YamlPreview -Template $template
      $rendered["stages"] | Where-Object { $_["stage"] -eq "staging" } | Should -HaveCount 1
    }
    It "Deploys to simulation after staging" {
      $template["extends"]["parameters"]["environment"] = "simulation"
      $rendered = Get-YamlPreview -Template $template
      $rendered["stages"] | Where-Object { $_["stage"] -eq "simulation" } | Should -HaveCount 1
      $simulation = $rendered["stages"] | Where-Object { $_["stage"] -eq "simulation" } | Select-Object -First 1
      $simulation["dependsOn"] | Should -Contain "staging"
    }
    It "Deploys to production after simulation" {
      $template["extends"]["parameters"]["environment"] = "production"
      $rendered = Get-YamlPreview -Template $single_technology
      $rendered["stages"] | Where-Object { $_["stage"] -eq "production" } | Should -HaveCount 1
      $production = $rendered["stages"] | Where-Object { $_["stage"] -eq "production" } | Select-Object -First 1
      $production["dependsOn"] | Should -Contain "simulation"
    }
  }
}

Notice how in each It block we set the value of the environment parameter before submitting the template to the preview API. This allows us to test all of the logic in our templates by setting different parameters.

Final Words

With these two approaches we can validate many aspects of our templates' behaviour, giving us confidence to make change without breaking our release process.

Henry Buckle

Principal DevOps Engineer, ClearBank