Using Pester to validate deployment readiness for a large number of machines

Recently, I had to roll out an upgrade of our software for a customer. The upgrade failed for about 80 client machines (out of around 400). There was a lot of head-scratching and quite a few “It was working in the Test environment !”. Because we couldn’t afford much more downtime for end-users, I had to develop an ugly workaround to allow machines to upgrade. But even so, this upgrade rollout went well over the planned maintenance window.

In short, it was a pain. And you know what ?
Pains are great learning opportunities and powerful incentives to take action. The first lesson was that there was a multitude of different causes which boiled down to misconfiguration, due to inconsistently managed client machines.

The second lesson was that we needed some kind of tool to validate the upgrade (or deployment) readiness of a bunch of machines to prevent this kind of mess in the future. This tool would allow to check whether all the machines meet the prerequisites for a new deployment or upgrade before rolling it out. This should also provide a nice, visual report so that non-technical stakeholders can see :

  • The overall number and percentage of machines not ready
  • Which machines are ready
  • Which ones are not ready
  • Which prerequisites (and prerequisite categories) are met
  • Which prerequisites (and prerequisite categories) are not met

The report should also allow technical stakeholders to drill down to see for a specific machine which prerequisite(s) were not met and why.

Knowing that Pester can be used to validate the operation of a system, I figured I could build a tool leveraging Pester tests to validate prerequisites. So I wrote the DeploymentReadinessChecker PowerShell module and made it available on the PowerShell Gallery. Yes, anyone can use it because it is designed as BYOPS (Bring Your Own Pester Script).

Regarding the HTML report, I didn’t reinvent the wheel, this is using a great utility named ReportUnit.

Basic usage :

First, we need :

  • PowerShell 4.0 (or later).
  • The Pester module should be installed on the machine from which we run DeploymentReadinessChecker.
  • A Pester script containing tests for the prerequisites we want to validate and optionally, “Describe” blocks to group tests in “prerequisite categories”.
  • If the above Pester script (validation script) takes parameters, the values for the parameters we need to pass to it.
  • A list of computer names for the machines we want to check the prerequisites against.
  • Credentials to connect to all the target machines

Now that we have everything we need, let’s get to it.

The module comes with an example validation script : Example.Tests.ps1 and that is what we are going to use here. For your own deployments or upgrades, you will need a validation script containing tests for your own prerequisites : hardware prerequisites, OS requirements, runtime or other software dependencies, network connectivity prerequisites… whatever you need.

Here are a few examples from the first 2 “Describe” blocks of Example.Tests.ps1 :

Describe 'Hardware prerequisites' -Tag 'Hardware' {
    
    It 'Has at least 4096 MB of total RAM' {

        Invoke-Command -Session $RemoteSession {
        (Get-CimInstance -ClassName Win32_PhysicalMemory).Capacity / 1MB } |
        Should Not BeLessThan 4096
    }
}
Describe 'Networking prerequisites' -Tag 'Networking' {

    It 'Can ping the Management server by name' {

        Invoke-Command -Session $RemoteSession { param($ManagementServerName)
        Test-Connection -ComputerName $ManagementServerName -Quiet } -ArgumentList $ManagementServerName |
        Should Be $True
    }
    It 'Can ping the Deployment server by name' {

        Invoke-Command -Session $RemoteSession { param($DeploymentServerName)
        Test-Connection -ComputerName $DeploymentServerName -Quiet } -ArgumentList $DeploymentServerName |
        Should Be $True
    }
    It 'Has connectivity to the Management server on TCP port 80' {

        Invoke-Command -Session $RemoteSession { param($ManagementServerName)
        (Test-NetConnection -ComputerName $ManagementServerName -CommonTCPPort HTTP).TcpTestSucceeded } -ArgumentList $ManagementServerName |
        Should Be $True
    }
    It 'Has the firewall profile set to "Domain" or "Private"' {

        Invoke-Command -Session $RemoteSession {
        $FirewallProfile = (Get-NetConnectionProfile)[0].NetworkCategory.ToString();
        $FirewallProfile -eq 'Domain' -or $FirewallProfile -eq 'Private' } |
        Should Be $True
    }
}

As we can see, it is the validation script’s responsibility to handle the remoting to the target machines.
The validation script should be located in $Module_Folder\ReadinessValidationScript\, for example : C:\Program Files\WindowsPowerShell\Modules\DeploymentReadinessChecker\1.0.0\ReadinessValidationScript\Example.Tests.ps1.

Also, its extension should be “.Tests.ps1” because that’s what Invoke-Pester looks for.

There is no support for multiple validation scripts, so before adding your own validation script in there, rename Example.Tests.ps1 by changing its extension to something else than “.Tests.ps1“. This is to ensure that the example script is ignored by Invoke-Pester.

UPDATE :
I added support for multiple validation scripts being present in $Module_Folder\ReadinessValidationScript\.
Test-DeploymentReadiness can only invoke one validation script at a time, but if there is more than one validation script present, a dynamic parameter named ValidationScript is made available (mandatory, even) to specify the name of the validation script.

It is highly recommended to group related tests into distinct and meaningful “Describe” blocks because, as we’ll see later on, some items in the report are displayed on a per-Describe block basis.

Optionally, “Describe” blocks can have tags and the tool can use these tags to include or exclude some tests, just like Invoke-Pester does.

The module contains a single cmdlet : Test-DeploymentReadiness.

Our computer names list can be fed to the -ComputerName parameter at the command line, from a file, or via pipeline input. For example, for a single computer, this could look like :

C:\> Test-DeploymentReadiness -ComputerName Computer1 -Credential $Cred -OutputPath $env:USERPROFILE\Desktop\Readiness\ |
Invoke-Item

Here is the console output :

Simple example with Invoke-Item

So we get the normal output from Invoke-Pester for each target machine specified via the -ComputerName parameter and a little bit more text at the end. All of this is just written to the console (using Write-Host) but it outputs a single object to the pipeline : a FileInfo object for the Index.html of the report. That way, if we want instant gratification, we can directly open the report in our default browser by piping the output of Test-DeploymentReadiness to Invoke-Item, as seen above.

Off course, it generates a bunch of files, as well. These are generated in the current directory by default, or in the directory specified via the -OutputPath parameter. Invoke-Pester generates one test result file (.xml) per target machine and ReportUnit.exe generates one HTML report per target machine and the overall report Index.html. To view the report, we only need to open the Index.html because it has the links to machine-specific files if we want to drill down to the per-machine reports.

Filtering the tests using tags :

As said earlier, all the Pester tests representing the prerequisites should be in a single validation script, so we can potentially end up with a script containing a very large number of tests. To make this more modular and flexible, we can group tests related to the same topic, purpose, or component into distinct “Describe” blocks and give these “Describe” blocks some tags.

Then, Test-DeploymentReadiness can include only the tests contained in the “Describe” blocks which have the tag(s) specified via the -Tag parameter. Let’s see what it looks like :

Simple example with tag

Similarly, we can exclude the tests contained in the “Describe” blocks which have the tag(s) specified via the -ExcludeTag parameter.

Passing parameters to the validation script :

It is more than likely that the Pester-based validation script takes parameters, especially since it remotes into the target machines, so it may need a -ComputerName and a -Credential parameter. If your validation script has parameter names matching “ComputerName” or “Credential“, then Test-DeploymentReadiness does a bit of work for you.

If the validation script has a ComputerName parameter, Test-DeploymentReadiness passes one computer at a time to its ComputerName parameter, via the Script parameter of Invoke-Pester.

If the validation script has a Credential parameter, the Test-DeploymentReadiness passes the value of its own Credential parameter to the validation script, via the Script parameter of Invoke-Pester.

Cool, but what about any other parameters ?
That’s where the -TestParameters parameter comes in. The parameter names and values can be passed as a hashtable to the -TestParameters parameter of Test-DeploymentReadiness. Then, Test-DeploymentReadiness passes these into the Script parameter of Invoke-Pester, when calling the validation script.

The example validation script Example.Tests.ps1 takes quite a few parameters, among them are DeploymentServerName and ManagementServerName . We can pass values to these 2 parameters, like so :

C:\> $TestParameters= @{ DeploymentServerName = 'DeplServer1'
                      ManagementServerName = 'Mgmtserver1'
                   }
C:\>
C:\> 'Computer1','Computer2','Computer3','Computer4','Computer5' |
Test-DeploymentReadiness -Credential $Cred -OutputPath $env:USERPROFILE\Desktop\Readiness\ -TestParameters $TestParameters
   

 

The Reports :

As mentioned earlier, we only need to open the generated Index.html and this opens the overview report. After running the above command, here is what this looks like :

Overview Report

Fixture summary” gives us the number of ready machines and not-so-ready machines whereas the “Pass percentage” gives us the percentage of machines which are ready.

We can see that Computer4 is the only machine which failed more than 1 prerequisite. We can see what’s going on with it in more detail by clicking on the link named “Computer4” :

Computer4 Report

We can clearly see 4 distinct prerequisite categories, which corresponds with “Describe” blocks in our validation script. Here, “Fixture summary” tells us which prerequisite categories contained at least one failed prerequisite(s). In this case, there were 2.

Let’s check which Networking prerequisite(s) were not met by clicking on “Networking prerequisites” :

Network Prerequisites Details

So now, we have can a good idea of what the issue is (the actual usefulness of the Pester error message will depend on how the test was written).

Pretty neat, huh ? I can see this saving me hours and hours of work, and considerably reduce the maintenance windows in future deployments and upgrades.

If this tool doesn’t exactly fit your needs or if you think of an enhancement, the code is on GitHub, feel free to submit an issue, or even better, to fork it and improve it.

5 thoughts on “Using Pester to validate deployment readiness for a large number of machines”

Leave a Reply

Your email address will not be published. Required fields are marked *