Making PSScriptAnalyzer a first-class citizen in a PowerShell CI pipeline

As you already know if you have read this or this, I’m a big fan of PSScriptAnalyzer to maintain a certain standard of coding style and quality. Where this is especially powerful is inside a continuous integration pipeline because this allows us to enforce that coding standard.

In our CI pipeline, we can easily make the build fail if our code violates one or more PSScriptAnalyzer rule(s). That’s great, but the main point of continuous integration is to give quick feedback to developers about their code change(s). Continuous integration is about catching problems early to fix them early. So, Green/Red or Pass/Fail is OK, but providing meaningful information about a problem to help remediate it is better. And pretty darn important.

So now, the question is :

How can we make our CI tool publish PSScriptAnalyzer results with the information we need to remediate any violation ?

All CI tools have ways to publish test results to make them highly visible, to drill down into a test failure, and even do some reporting on these test results. Since we are talking about a PowerShell pipeline, we are most likely already using Pester to test our PowerShell code. Pester can spit out results in the same XML format as NUnit and these NUnit XML files can be consumed and published by most CI tools.

It makes a lot of sense to leverage this Pester integration as a universal CI glue and run our PSScriptAnalyzer checks as Pester tests. Let’s look at possible ways to do that.

One Pester test checking if the PSScriptAnalyzer result is null :

This is probably the simplest way to invoke PSScriptAnalyzer from Pester :

Describe 'PSScriptAnalyzer analysis' {
    
    $ScriptAnalyzerResults = Invoke-ScriptAnalyzer -Path ".\ExampleScript.ps1" -Severity Warning
    
    It 'Should not return any violation' {
        $ScriptAnalyzerResults | Should BeNullOrEmpty
    }
}
  

 
Here, we are checking all the rules which have a “Warning” severity within one single test. Then, we rely on the fact that if PSScriptAnalyzer returns something, it means that they were at least one violation and if PSScriptAnalyzer returns nothing, it’s all good.

There are 2 problems here :

  • We are evaluating a whole bunch of rules in a single test, so the test name cannot tell us which rule was violated
  • As soon as there are more than one violation, the Pester message gives us useless information

How useless ? Well, let’s see :

useless-pester-stacktrace
 
The Pester failure message gives us the object type of the PSScriptAnalyzer results, instead of their contents. This does not provide what we need to locate and remediate the problem, like the name of the file which violated the rule and the line number in that file where the violation is located.

One Pester test per PSScriptAnalyzer rule :

This is a pretty typical (and better) way of running PSScriptAnalyzer checks via Pester.

Describe 'PSScriptAnalyzer analysis' {
    
    $ScriptAnalyzerRules = Get-ScriptAnalyzerRule -Name "PSAvoid*"

    Foreach ( $Rule in $ScriptAnalyzerRules ) {

        It "Should not return any violation for the rule : $($Rule.RuleName)" {
            Invoke-ScriptAnalyzer -Path ".\ExampleScript.ps1" -IncludeRule $Rule.RuleName |
            Should BeNullOrEmpty
        }
    }
}
  

 
In this case, the first step is to get a list of the rules that we want to evaluate. Here, I changed the list of rules to : all rules which have a name starting with “PSAvoid”.
This is just to show that we can filter the rules by name, as well as by severity.

Then, we loop through this list of rules and have a Pester test evaluating each rule, one by one. As we can see below, the output is much more useful :

psscriptanalyzer-by-rule
 
This is definitely better but we still encounter the same issue as before because there were more than one violation for that “PSAvoidUsingWMICmdlet” rule. So we still don’t get the file name and the line number.

We could use a nested loop : for each rule, we would loop through each file and evaluate that rule against each file one-by-one. That would be more granular and reduce the risk of this particular issue. But if a single file violated the same rule more than once, we would still have the same problem.

So, I decided to go take a different direction to address this problem : taking the output from PSScriptAnalyzer and converting it to a test result file, using the same XML schema as Pester and NUnit.

Converting PSScriptAnalyzer output to a test result file :

For that purpose, I wrote a function named Export-NUnitXml, which is available on GitHub in this module.

Here are the high-level steps of what Export-NUnitXml does :

  • Take the output of PSScriptAnalyzer as its input (zero or more objects of the type [Microsoft.Windows.PowerShell.ScriptAnalyzer.Generic.DiagnosticRecord]
  • Create an XML document containing a “test-case” node for each input object(s).
  • Write this XML document to the file specified via the Path parameter.

Here is an example of how we can use this within a build script (in Appveyor.com as the CI tool, in this case) :

$ScriptAnalyzerRules = Get-ScriptAnalyzerRule -Severity Warning
$ScriptAnalyzerResult = Invoke-ScriptAnalyzer -Path ".\CustomPSScriptAnalyzerRules\ExampleScript.ps1" -IncludeRule $ScriptAnalyzerRules
If ( $ScriptAnalyzerResult ) {
  
    $ScriptAnalyzerResultString = $ScriptAnalyzerResult | Out-String
    Write-Warning $ScriptAnalyzerResultString
}
Import-Module ".\Export-NUnitXml\Export-NUnitXml.psm1" -Force
Export-NUnitXml -ScriptAnalyzerResult $ScriptAnalyzerResult -Path ".\ScriptAnalyzerResult.xml"

(New-Object 'System.Net.WebClient').UploadFile("https://ci.appveyor.com/api/testresults/nunit/$($env:APPVEYOR_JOB_ID)", (Resolve-Path .\ScriptAnalyzerResult.xml))

If ( $ScriptAnalyzerResult ) {        
    # Failing the build
    Throw "Build failed because there was one or more PSScriptAnalyzer violation. See test results for more information."
}
   

 
And here is the result in Appveyor :

appveyor-overview
 
Just by reading the name of the test case, we get the essential information : the rule name, the file name and even the line number. Pretty nice, huh ?

Also, we can expand any failed test (by clicking on it) to get additional information. For example, the last 2 tests are expanded below :

appveyor-test-details
 
The “Stacktrace” section provides additional details, like the rule severity and the actual offending code. Another nice touch is that the “Message” section gives us the rule message, which normally provides an actionable recommendation to remediate the problem.

But, what if PSScriptAnalyzer returns nothing ?
Export-NUnitXml does handle this scenario gracefully because its ScriptAnalyzerResult parameter accepts $Null.
In this case, the test result file will contain only one test case and this test passes.

Let’s test this :

Import-Module -Name 'PsScriptAnalyzer' -Force
Import-Module ".\Export-NUnitXml\Export-NUnitXml.psm1" -Force
Export-NUnitXml -ScriptAnalyzerResult $Null -Path ".\ScriptAnalyzerResult.xml"

(New-Object 'System.Net.WebClient').UploadFile("https://ci.appveyor.com/api/testresults/nunit/$($env:APPVEYOR_JOB_ID)", (Resolve-Path .\ScriptAnalyzerResult.xml))
  

 
Here is what it looks like in Appveyor:

appveyor-passed-psscriptanalyzer-tests
 
There’s nothing more beautiful than a green test…

So now, as developers, we not only have quick feedback on our adherence to coding standards, but we also get actionable guidance on how to improve.
And remember, this NUnit XML format is widely supported in the CI/CD world, so even though I only showed Appveyor, this would work similarly in TeamCity, Microsoft VSTS, and others…

2 thoughts on “Making PSScriptAnalyzer a first-class citizen in a PowerShell CI pipeline”

Leave a Reply

Your email address will not be published. Required fields are marked *