This post was created for theAzure Back to School 2023event.There is a lot of great content published every day, I would encourage you to check it out.

Testing is hard. Testing your Bicep code, doubly so. Until now. At least that's what BenchPress, a new Azure testing framework promises.

💡

This article was originally published on my Wordpress blog: BenchPress: a must-have tool to test your Bicep muscles – David Pazdera (pazdedav.blog )

Let's explore together this interesting open-source project to understand:

how it works
how you can test locally
how it can be integrated in our CI/CD pipeline
and how it can complement existing validation options like linting, pre-flight and what-if deployment for Bicep templates and modules

How it works

The central idea behind BenchPress is about adding a verification after you deploy your Bicep template that would confirm that the resources you declared exist (were deployed successfully) and assert if the actual values match with expected ones.

Compared to other tools and linters that would inspect your code and run some checks before you hit deploy, BenchPress needs some Azure environment to run its validation!

Apart from adding such verification to your regular flow when working with various environments, you could use it with ephemeral environments, where you could deploy, validate, and destroy in a single flow. This could work great when building a library of Bicep modules, you want to test and validate before publishing to your private registry.

BenchPress is technically an extension ofPester, an exceedingly popular testing framework for PowerShell, and it has several prerequisites to function properly:

PowerShell 7+
Az PowerShell module
Pester module
Bicep CLI

Since it uses Pester as an underlying engine, it won't surprise you that BenchPress itself is published as a module in the PowerShell Gallery and it can be easily downloaded and enabled in your local machine using the following commands:

Install-Module -Name Az.InfrastructureTesting
Import-Module -Name Az.InfrastructureTesting

Apart from the obvious need to have an Azure subscription (account), you will also need to create a Service Principal and inject its key properties to your environment variables. In other words, BenchPress doesn't support the signed-in user credentials yet (only Service Principal and Managed Identities).

What BenchPress adds on top of Pester is a collection of cmdlets that simplify your assertions, for example:

Confirm-AzBPResourceGroup -ResourceGroupName $rgName | Should -BeSuccessful
Confirm-AzBPResourceGroup -ResourceGroupName $rgName | Should -BeInLocation $location

Of course, you could craft your own tests with "pure" Pester syntax and Azure PowerShell modules instead, but it is easier with BenchPress.

A small hiccup

If you search for "benchpress" in the PowerShell Gallery, you will get the following result:

Why am I pointing this out? When I was experimenting with BenchPress and the library of examples a couple of months ago, it worked without problems. When I was re-running the same tests while authoring this article, it was throwing this error:

ParameterBindingException: Parameter set cannot be resolved using the specified named parameters. One or more parameters issued cannot be used together or an insufficient number of parameters were provided.

I tried to figure out what was causing it but what helped me as a workaround was to import "the other BenchPress" module using the Import-Module BenchPress.Azure command, rather than Az.InfrastructureTesting.

How to test locally

Install tools

Before you can test anything, you need to make sure your box has all the dependencies installed. Your setup will vary depending on what OS you're using but for Windows, you could use winget or Chocolatey:

winget install --id Microsoft.Powershell --source winget
winget install --id Microsoft.Bicep --source winget
winget install --id Microsoft.VisualStudioCode --source winget

Then you would install the following PowerShell modules and VS Code extensions:

Install-Module -Name Az
Install-Module -Name Pester
Install-Module -Name Az.InfrastructureTesting

code --install-extension ms-vscode.powershell
code --install-extension ms-azuretools.vscode-bicep
code --install-extension pspester.pester-test

\> TIP: It would make sense to create a Dev Container definition for BenchPress, so you don’t need to install any of these tools directly.

Workload identity and environment

Remember that you need to create a new Service Principal or re-use an existing one for BenchPress to work. This script will create one for you and populate environment variables (Check the Getting started guide for more details):

Connect-AzAccount
$sp = New-AzADServicePrincipal -DisplayName BenchPressSPN
New-AzRoleAssignment -ApplicationId $sp.Id -RoleDefinitionName 'Contributor'

$encryptedPass = $sp.PasswordCredentials.SecretText | ConvertTo-SecureString -AsPlainText -Force | ConvertFrom-SecureString

$Env:AZ_APPLICATION_ID= $sp.Id
$Env:AZ_TENANT_ID= (Get-AzSubscription).TenantId
$Env:AZ_SUBSCRIPTION_ID= (Get-AzSubscription).id
$Env:AZ_ENCRYPTED_PASSWORD= $encryptedPass

\> Note the special form in which the SPN password needs to be stored ($encryptedPass variable).

First local test

Now when your dev box has everything it needs, you could begin with the simplest available example: Resource Group. You can either clone the entire BenchPress repo or copy specific examples you'd like to test.

The instructions for the Resource Group can be found in ResourceGroup_Example.md:

Run the deployment with this resourceGroup.bicep template:

targetScope = 'subscription'

param name string = 'rg${take(uniqueString(subscription().id), 5)}'
param location string = deployment().location

// https://docs.microsoft.com/en-us/azure/templates/microsoft.resources/resourcegroups?tabs=bicep
resource resourceGroup 'Microsoft.Resources/resourceGroups@2022-09-01' = {
  name: name
  location: location
}

output name string = resourceGroup.name
output id string = resourceGroup.id

Update the ResourceGroup.Tests.ps1 file with correct values (name and location):

BeforeAll {
  Import-Module Az.InfrastructureTesting

  $Script:rgName = 'rg-test'
  $Script:noRgName = 'notestrg'
  $Script:location = 'westus3'
}

Describe 'Verify Resource Group Exists' {
  It "Should contain a Resource Group named $rgName - Confirm-AzBPResource" {
    # arrange
    $params = @{
      ResourceType      = "ResourceGroup"
      ResourceName      = $rgName
    }

    # act and assert
    Confirm-AzBPResource @params | Should -BeSuccessful
  }


  It "Should contain a Resource Group named $rgName - Confirm-AzBPResource" {
    # arrange
    $params = @{
      ResourceType      = "ResourceGroup"
      ResourceName      = $rgName
      PropertyKey       = 'ResourceGroupName'
      PropertyValue     = $rgName
    }

    # act and assert
    Confirm-AzBPResource @params | Should -BeSuccessful
  }

  It "Should contain a Resource Group named $rgName" {
    Confirm-AzBPResourceGroup -ResourceGroupName $rgName | Should -BeSuccessful
  }

  It "Should not contain a Resource Group named $noRgName" {
    # The '-ErrorAction SilentlyContinue' command suppresses all errors.
    # In this test, it will suppress the error message when a resource cannot be found.
    # Remove this field to see all errors.
    Confirm-AzBPResourceGroup -ResourceGroupName $noRgName -ErrorAction SilentlyContinue | Should -Not -BeSuccessful
  }

  It "Should contain a Resource Group named $rgName in $location" {
    Confirm-AzBPResourceGroup -ResourceGroupName $rgName | Should -BeInLocation $location
  }
}

AfterAll {
  Get-Module Az.InfrastructureTesting | Remove-Module
  Get-Module BenchPress.Azure | Remove-Module
}

Run Invoke-Pester -Path .\ResourceGroup.Tests.ps1 command and ideally get the following output:

Tests completed in 952ms
Tests Passed: 2, Failed: 0, Skipped: 0 NotRun: 0

This was quite a simple test, I must admit. You should try several examples or even write your own tests from scratch to assess the true potential this tool can provide you.

VS Code extension

If you are a Visual Studio Code user, there is a special treat available. Simply install the Pester Tests extension with e.g., code --install-extension pspester.pester-test

You will get a nice side panel that will show all *.Tests.ps1 files in your workspace and allow you to run and re-run all tests or just individual ones or debug them if your tests don't do what you expect.

\> Note: The extension is in preview, but it worked great for me, and I haven’t encountered any issues.

Integration with GitHub Actions workflow

Although the BenchPress repo doesn't contain any guidance on how to integrate it with various CI/CD tools, since it is powered by Pester, I used the 'Test Results' article from Pester documentation as a starting point and added what BenchPress required on top.

If you want to check my GitHub workflow definition, you can find it here: https://github.com/pazdedav/bicep-pester-validation

My demo setup is using several key "ingredients":

a workflow with workflow_dispatch: trigger, so I could run it manually and inject a few inputs
a set of variables - AZ_SUBSCRIPTION_ID, AZ_TENANT_ID, AZ_APPLICATION_ID - and a secret (AZ_ENCRYPTED_PASSWORD) exactly like BenchPress requires. Those are then made available as environment variables

The key step in the workflow looks like this:

- name: BenchPress tests
  id: pester
  uses: azure/powershell@v1
  with:
    inlineScript: |
      Set-PSRepository psgallery -InstallationPolicy trusted
      Install-Module -Name Pester -RequiredVersion 5.5.0 -Confirm:$false -Force -SkipPublisherCheck
      Install-Module -Name BenchPress.Azure -RequiredVersion 0.2.1 -Confirm:$false -Force -SkipPublisherCheck
      Import-Module Pester -Force
      Import-Module BenchPress.Azure -Force
      $configuration = [PesterConfiguration]::Default
      $configuration.TestResult.Enabled = $true
      $configuration.Output.Verbosity = 'Detailed'
      $configuration.TestResult.OutputFormat = 'NUnitXml'
      $configuration.TestResult.OutputPath = 'Test.xml'
      $configuration.Output.CIFormat = 'GithubActions'
      $container = New-PesterContainer -Path "./infrastructure/resourceGroup/resourceGroup.Tests.ps1" -Data @{ ResourceGroupName = "${{ env.RESOURCE_GROUP }}"; location = "${{ env.LOCATION }}" }
      $configuration.Run.Container = $container
      $configuration.Run.PassThru = $true
      $result = Invoke-Pester -Configuration $configuration
      exit $result.FailedCount
    azPSVersion: "latest"
  env:
    RESOURCE_GROUP: ${{ inputs.resourceGroupName }}
    LOCATION: ${{ inputs.location }}

After importing those two PowerShell modules, the script builds a Pester configuration object, where you can specify many properties based on your preference. It also creates a $container object, where you provide path to test files and inject variables those test use.

After a few trial-and-error attempts, I finally got this result:

How it fits into existing tools

I believe that BenchPress tests can nicely complement other tests and validations that have been around for Bicep / JSON ARM for a while:

Bicep linter - az bicep build --file main.bicep
ARM pre-flight validation - azure/arm-deploy@v1 action with deploymentMode: Validate
what-if deployment - running azure/arm-deploy@v1 action with additionalArguments: --what-if
PSRules for Azure - docs

The significant difference though is the fact that all the other tools listed above are checking your Bicep templates and modules before you deploy them. BenchPress tests are asserting against a "real" Azure environment, so the resources must exist. You could think of it as a post-deployment smoke test.

Going forward

One slightly concerning thing is the frequency of commits to the BenchPress repo and the fact that some issues that are still open were created eight months ago. I couldn't find any roadmap (e.g., a GitHub Project or a similar publicly available view), so the future of this project is uncertain. It currently has fourteen maintainers.

The list of cmdlets that are available in the Az.InfrastructureTesting module is also quite limited:

Native testing framework in Bicep?

In addition to this project, the Bicep core team announced a new experimental testing framework during their Community call in July 2023 that would bring native support directly in the language.