Integrating PSScriptAnalyzer in an Appveyor Continuous Integration pipeline

Many of us who are writing PowerShell code are using the free (and awesome) Appveyor service for Continuous Integration (especially for personal projects). And most of us use this to run Pester tests. Automated testing is great, it allows to set a certain standard of code quality without slowing down code delivery. But, this is just checking that the code behaves as we intended to.

What about code consistency, style, readability and following best practices ?

This is where a static code analysis tool like PSScriptAnalyzer comes in. Even though PSScriptAnalyzer is a perfect fit in a PowerShell “build” process, searching the web for “integrating PSScriptAnalyzer and Appveyor” doesn’t yield very helpful results. So here is the solution I came up with :

version: 1.0.{build}

os: WMF 5

# Skip on updates to the readme
skip_commits:
  message: /readme*/
  
install:
  - ps: Install-PackageProvider -Name NuGet -Force
  - ps: Install-Module PsScriptAnalyzer -Force
  
build: false

test_script:
  - ps: |
      Add-AppveyorTest -Name "PsScriptAnalyzer" -Outcome Running
      $Results = Invoke-ScriptAnalyzer -Path $pwd -Recurse -Severity Error -ErrorAction SilentlyContinue
      If ($Results) {
        $ResultString = $Results | Out-String
        Write-Warning $ResultString
        Add-AppveyorMessage -Message "PSScriptAnalyzer output contained one or more result(s) with 'Error' severity.`
        Check the 'Tests' tab of this build for more details." -Category Error
        Update-AppveyorTest -Name "PsScriptAnalyzer" -Outcome Failed -ErrorMessage $ResultString
        
        # Failing the build
        Throw "Build failed"
      }
      Else {
        Update-AppveyorTest -Name "PsScriptAnalyzer" -Outcome Passed
      }

This the content of my appveyor.yml file, which is the file from which Appveyor gets the build configuration.

Line 3 : This indicates from which VM template the build agent will be deployed. As its name indicates, this allows to have a build agent running in a VM with PowerShell version 5. If you believe only what you can see, add $PSVersionTable in the appveyor.yml and check the result in the build console. PowerShell 5 means we can easily add PowerShell scripts, modules and DSC resources to our build agent from the PowerShell Gallery using PackageManagement.

Line 10-11 : This is exactly what we do here. But first, because the PowerShell Gallery relies on NuGet, we need to install the NuGet provider. Then, we can install any PowerShell module we want from the PowerShell Gallery, PsScriptAnalyzer in this case. We didn’t specify the repository because the PowerShell Gallery is the default one.

Line 13 : This refers specifically to MSBuild and we don’t need or want MSBuild for a PowerShell project.

Line 15-End : This is where all the PSScriptAnalyzer stuff goes. So from an Appveyor point of view, this will be a test. Even though static code analysis is not testing, it kinda makes sense : we are assessing the code against a set of rules which represent a certain standard and we want a “Pass” or a “Fail” depending on whether the code meets the standard or not.

Line 16 : In YAML, the pipe character “|” allows values to span multiple lines. This is very convenient for code blocks, like here. That way, we don’t need to add “- ps:” at the beginning of each line.

Line 17 : Appveyor doesn’t have a direct integration with PSScriptAnalyzer like it has for some testing frameworks (NUnit, MSTest, etc…) but it’s OK. The Appveyor worker (the actual build agent) provides a REST API and even a few PowerShell cmdlets leveraging this API. One of these cmdlets is Add-AppveyorTest. Using this cmdlet, we are adding a new test, giving it a name and telling the build agent that the test is in the “Running” state.

Line 18 : We run PSScriptAnalyzer against all the files in the current directory, recursively. We specify the “Error” severity to output only the violations of level “Error“, because we don’t want a violation of severity “Information” or even “Warning” to make the test fail. We store the result in a variable for later use.

Line 20 : If there are any “errors” from PSScriptAnalyzer perspective, we want to display them as a message in the build console and in the error message of the “test”. That’s why we need to convert the output object(s) from PSScriptAnalyzer to a string.

Line 21 : Writing the violation(s) to the build console. We could use Write-Host or Write-Output as well but as we’ll see in a later screenshot, the warning stream makes it stand out more visibly.

Line 22 : This Appveyor-specific cmdlet adds a message to the build’s “Messages” tab. Specifying “Error” for the category just displays the message with a touch of red on its left :

Appveyor Message fail
 
Line 24 : Update-AppveyorTest is another cmdlet leveraging the Appveyor build worker API. Here, we are using it to update the status of our existing test and add an error message to it. This message is PSScriptAnalyzer output converted to a string, so we can check the test message to see exactly what the problem is :

Appveyor Test fail
 

Line 27 : We need to use “Throw” to explicitly fail the build. Otherwise, the build is considered as succeeded, even if the “test” fails.

Line 30 : If PSScriptAnalyzer didn’t output anything, meaning if there were no violation of the “Error” severity in any file scanned by PSScriptAnalyzer, we considered that our project passes the test. Again, we use Update-AppveyorTest but this time, we tell it that the outcome of the “test” is a pass.

Now, let’s see how this looks like when we run a build :

Appveyor Build success
 
Not much output, because all is well. Also, the test is green :

Appveyor Test Success
 
Do you like watching “Fail” videos on Youtube ? If yes, you are probably dying to see my build fail, right ? So, here we go :

Appveyor Build fail
 
Wow, the yellow background of the warning stream is not elegant but it sure stands out !

Also, if you want to see the “Passing” Appveyor badge on the GitHub repository, head over THERE.

This is it.
PSScriptAnalyzer is an important tool that any PowerShell scripter should use. Appveyor is awesome, so combining both of these tools is pretty powerful.

Documentation as Code : Exporting the contents of DSC MOF files to Excel

One of the greatest benefits of PowerShell DSC (and other Configuration Management tools/platforms) is the declarative syntax (as opposed to imperative scripting). Sure, a DSC configuration can contain some logic, using loops and conditional statements, but we don’t need to care about handling errors or checking if something is already present. All this (and the large majority of the logic) is handled within the resource, so we just need to describe the end result, the “Desired State”.

So all the settings and information that a configuration is made of are stored in a very simple (and pretty much human-readable) syntax, like :

Node $AllNodes.NodeName
    {
        cWindowsErrorReporting Disabled
        {
            State = "Disabled"
        }
    }

 
This allows us to use this “code” (for lack of a better word) as documentation in a way that wouldn’t be possible or practical with imperative code. For this purpose, we could use DSC configurations, or DSC configuration data files if all the configuration data is stored separately. But the best files for that would probably be the MOF files for 2 reasons :

  • Even if some settings are in different files, we can be sure that all the settings for a given node is in a single MOF file (the exception being partial configurations)
  • Even if the DSC configuration contains complex logic, there is no need to understand or parse this logic to get the end result. All this has been done for us when the MOF file has been generated

Now, imagine you have all your MOF files stored in a directory structure like this :

PS C:\> tree C:\DSCConfigs /F
Folder PATH listing for volume OS
C:\DSCCONFIGS
├───Customer A
│   ├───Dev
│   │       Server1.mof
│   │       Server2.mof
│   │
│   ├───Prod
│   │       Server1.mof
│   │       Server2.mof
│   │
│   └───QA
│           Server1.mof
│           Server2.mof
│
├───Customer B
│   ├───Dev
│   │       Server1.mof
│   │       Server2.mof
│   │
│   ├───Prod
│   │       Server1.mof
│   │       Server2.mof
│   │
│   └───QA
│           Server1.mof
│           Server2.mof
│
└───Customer C
    ├───Dev
    │       Server1.mof
    │       Server2.mof
    │
    ├───Prod
    │       Server1.mof
    │       Server2.mof
    │
    └───QA
            Server1.mof
            Server2.mof

You most likely have much more than 2 servers per environment, so there can easily be a large number a MOF files.
Then, imagine your boss tells you : “I need all the configuration settings, for all customers, all environments and all servers in an Excel spreadsheet to sort and group the data easily and to find out the differences between Dev and QA, and between QA and Prod”.

If you are like me, you may not quite understand bosses’ uncanny obsession with Excel but this definitely sounds like something useful and an interesting challenge. So, let’s do it.

We’ll divide this in 3 broad steps :

  • Converting the contents of MOF files to PowerShell objects
  • Exporting the resulting PowerShell objects to a CSV file
  • Processing the data using PowerShell and/or Excel

Converting the contents of MOF files to PowerShell objects

This is by far the most tricky part.
Fortunately, I wrote a function, called ConvertFrom-DscMof, which does exactly that. We won’t go into much details about how it works, but you can have a look at the code here.

Basically, it parses one or more MOF files and it outputs an object for each resource instance contained in the MOF file(s). All the properties of a given resource instance become properties of the corresponding object, plus a few properties related to the MOF file.

Here is an example with a very simple MOF file :

ConvertFrom-DscMofExample
 
And here is an example with all the properties of a single resource instance :

ConvertFrom-DscMofSingle
 

Exporting the resulting PowerShell objects to a CSV file

As we have the ability to get DSC configuration information in the form of PowerShell objects, it is now very easy to export all this information as CSV. But there’s a catch : different resources have different parameters, for example the Registry resource has the ValueName and ValueData parameters and the xTimeZone resource has a TimeZone parameter.

So the resulting resource instances objects will have ValueName and ValueData properties if they are an instance of the Registry resource and a TimeZone property if they are an instance of the xTimeZone resource. Even for a given resource, some parameters are optional and they will end up in the properties of the resulting PowerShell object only if they were explicitly specified in the configuration.

The problem is that Export-Csv doesn’t handle intelligently objects with different properties, it will just create the columns from the properties of the first object in the collection and apply that to all objects, even for objects which have different properties.

But, we can rely on the “ResourceID” property of each resource instance because it uniquely identify the resource instance. Also, it contains the name we gave to the resource block in the DSC configuration, which should be a nice meaningful name, right ?
Yeah, this is where “Documentation as code” meets “self-documenting code” : they are both important and very much complementary. To get an idea of what the values of ResourceID look like, refer back to the first screenshot.

Below, we can see how to export only the properties we need, and only the properties that we know will be present for all resource instances :


Get-ChildItem C:\MOFs\ -File -Filter "*.mof" -Recurse |
ConvertFrom-DscMof |
Select-Object -Property "MOF file Path","MOF Generation Date","Target Node","Resource ID","DSC Configuration Info","DSC Resource Module" |
Export-Csv -Path 'C:\DSCConfig Data\AllDSCConfigs.csv' -NoTypeInformation

 

Processing the data using PowerShell and/or Excel

The resulting CSV file can be readily opened and processed by Excel (or equivalent applications) :

CSVFileInExcel
 
Now, we have all the power of Excel at our fingertips, we can sort, filter, group all this data however we want.

Now, here is a very typical scenario : the Dev guys have tested their new build and it worked smoothly in their environment. However, the QA guys say that the same build is failing miserably in their environment. The first question which should come to mind is : “What is the difference between the Dev and QA environments ?

If all the configuration of these environments is done with PowerShell DSC, the ConvertFrom-DscMof function can be a great help to answer that very question :

C:\> $CustomerCDev = Get-ChildItem -File -Filter '*.mof' -Recurse 'C:\MOFs\Customer C\Dev\' |
ConvertFrom-DscMof
C:\> $CustomerCQA = Get-ChildItem -File -Filter '*.mof' -Recurse 'C:\MOFs\Customer C\QA\' |
ConvertFrom-DscMof
C:\> Compare-Object -ReferenceObject $CustomerCDev -DifferenceObject $CustomerCQA -Property 'Target Node','Resource ID'

Target Node Resource ID                    SideIndicator
----------- -----------                    -------------
Server1     [xRemoteFile]RabbitMQInstaller <=
Server1     [Package]RabbitMQ              <=

 
Oops, we forgot to install RabbitMQ on Server1 ! No wonder it’s not working in QA.
But now, there is hope. We, forgetful and naturally flawed human beings, can rely on this documentation automation to tell us how things really are.

So, as we have seen, Infrastructure-as-code (PowerShell DSC in this case) can be a nice stepping-stone for an infrastructure documentation.
What is the number 1 problem for any infrastructure/configuration documentation ?
Keeping it up-to-date. This can help generate dynamically the documentation, which means this documentation can be kept up-to-date pretty easily without any human intervention.

My favorite PowerShell interview questions

As you may already know, the only PowerShell certification program is being abandoned. Some people in the PowerShell community are trying to justify this by saying “There is no need for PowerShell cert” or “it’s too difficult to test PowerShell knowledge”. This makes me sad and even, a little bit angry. The reason I feel so strongly about this is not because I am one of the few who passed this exam, it is because I completely, totally, utterly disagree with these arguments.

First, as the DevOps culture and practices pervade more and more Windows shops, professional PowerShell development skills (and the assessment of these skills) are becoming more and more critical for both employers and IT professionals.
Second, assessing PowerShell knowledge is not more difficult than assessing Java, C# or T-SQL skills, though there are certifications on all of these.

I could rant about this for ages, but instead, I’m going to show that evaluating someone’s PowerShell skills is not that difficult.
How ?
By sharing some of the PowerShell questions I ask to potential future colleagues during technical job interviews.

First, what is your scripting experience ?

This serves as an ice-breaker. Hopefully, the candidate will be happy and passionate to talk about the scripts he/she has written and his/her cool projects.
This is great to get an idea of the candidate’s experience (or his perception of his experience).
Notice I said “scripting experience” not “PowerShell scripting experience”. This is very much on purpose.

If the candidate has scripting experience in at least one other language, I jump with 2 feet in the next question :

Can you tell me two major differences between <$Other_Language> and PowerShell ?

This question is an fantastic opportunity get an insight into the candidate understanding of PowerShell core concepts and his/her favourite PowerShell features.
For example, one of the most fundamental difference between bash and PowerShell is that bash is text-oriented whereas PowerShell is object-oriented. A good candidate should be able to tell that. A great candidate, would most likely go on and on about the benefits of the object-oriented nature of PowerShell.

Also, if a candidate cites 2 valid differences between the two languages, but not a major one, the candidate might have a partial knowledge of PowerShell.

How would you set a registry value with PowerShell ?

Yeah, we start easy, but it is not that intuitive. This requires to know that there is no registry-specific cmdlets, so we have to use the registry provider and Set-ItemProperty. The registry has been an integral part of Windows for… ever, so this is hardly area-specific knowledge. It is very likely that any PowerShell scripter has already performed this task at least once, maybe even on a almost-daily basis.

How would you ping a remote computer with 5 packets using PowerShell ?

Again, this is basic stuff. I wouldn’t get caught up in ideology, the good old ping.exe is perfectly valid, as long as the candidate knows the option to specify 5 packets.
If you really want the more “PowerShelly” Test-Connection, then ask : I just want the command to return $True if the ping is successful and $False if it is not.

If I run : Get-Service -Name “bits” | Start-Process ; what is going to happen ?

Now, we are finally in the thick of it ! The pipeline is such an important underlying concept of PowerShell. This is a very concrete, practical way of asking :

How does the PowerShell pipeline work ?

This variant is a bit vague, but open questions have their place : this allows to verify how articulate the candidate is when explaining complex technical concepts. If the interview is for a senior position, the ability to explain complex mechanisms to less technical people (junior team members, managers, etc…) is very valuable.

Yet another variant on the same topic would be :

What are the 2 ways for a PowerShell cmdlet to accept input from the pipeline ?

I really love this one. A good candidate should understand that objects in the pipeline are not miraculously bound to the next cmdlet, they are bound to a specific parameter of the next cmdlet. A great candidate will tell the 2 ways in which a parameter can take pipeline input quickly and effortlessly.

If he/she gives the 2 ways but struggles to find this information in his/her memory, that’s kinda suspicious. To me, that would mean the candidate has read about it when preparing for the interview but doesn’t master the concept and has never implemented pipeline input in his/her functions.

Also, the 2 ways have a specific order, a great candidate should know which way is tried first.

You have a script which uses Read-Host to prompt the user for an IP address. You need to make sure the user inputs a valid IP address. For example, if the user inputs “297.0.126.274”, the script should prompt the user again. How would you do that ?

A good candidate would probably use one of 2 ways :
Splitting the address in 4 elements and try to convert each element to the type [byte].
Or
Complicated regular expressions.

A great candidate would use the much simpler and much more elegant method : let the .NET Framework do the hard work and just try to cast the input string to the [System.Net.IPAddress] class.
By the way, the candidate should know that what we get from Read-Host is a [string].

What is the difference, in PowerShell, between a function and an “advanced function” ?

This could be a trick question for an average candidate, but normally, good candidates should know that the only thing that makes a function “advanced” and unlocks the wonderful tooling which comes with it is : [cmdletbinding()].
A great candidate would be pretty enthusiastic about all the powerful tools enabled by advanced functions. Any PowerShell scripter who strives to build professional-grade tools should be thankful for all the work PowerShell is doing for us.

To display a text message to the user, when would you rather use Write-Host instead of Write-Output ?

This is a very popular topic in the PowerShell community and a little controversial. As Don Jones famously said : “a puppy dies every time you use Write-Host“.
An experienced PowerShell professional would say something like : “The general best practice is to NOT use Write-Host, unless …”

Off the top of my head, I see 3 cases where Write-Host makes more sense than Write-Output : purely interactive script where we don’t care about outputting to the pipeline, the end-users want pretty colors, and Visual Studio Team Services build scripts … I’m sure there are other cases.

Inside of a Try block, you have this code : Format-Volume -DriveLetter “C” -ErrorAction SilentlyContinue. What is the problem with that ?

Proper and deliberate error handling makes the difference between a script that any amateur can hack together for his/her own use and a professional-grade tool which can be used in production. This particular question highlights two fundamental aspects of error handling in PowerShell. In fact, these are so important that I wrote a whole blog post on this very subject.

This question should be pretty obvious to a good candidate. If the candidate goes in the wrong direction, repeat that part of the question : “Inside of a Try block”. If he/she is even more confused by this hint, then he/she doesn’t understand PowerShell error handling.

In PowerShell DSC, what is the file extension of the configuration document that is applied by the DSC agent on the target node ?

Answering this one doesn’t even require to have real-world experience with DSC. It is my way of checking if the candidate has ever heard about PowerShell DSC, and if he/she was curious enough to spend about 30 minutes to an hour reading up on it. Being curious and genuinely interested in learning new things is becoming crucial in this rapidly changing IT world.

To ask only if the candidate has indicated PowerShell DSC as a skill/strength :

 

You write a DSC resource and you are working on the function Test-TargetResource. What should be the output of this function ?

Anyone who has ever written a MOF-based DSC resource should know that, especially if using the xDSCResourceDesigner module to create the resource scaffolding. I would tend to think that anything which is included in the template generated that xDSCResourceDesigner is fair game.

Also, PowerShell practitioners should be able to guess the answer, even without any DSC resource authoring experience.
How ? Simply by remembering that cmdlets with a common verb tend to have consistent behaviours. So, what is a common output for Test-* cmdlets, Test-Path for example ?

You write a DSC resource and you are working on the function Get-TargetResource. What should be the output of this function ?

Again, the answer is explicitly written in the template generated by the xDSCResourceDesigner module and its New-xDscResource cmdlet.

And what should be the output of Set-TargetResource ?

This might be a slightly trick question : there is no return type requirement for this function. But I think asking about the output of each function in a DSC resource is an effective way of probing the candidate’s resource authoring experience and understanding of each function’s purpose.

Also, if the candidate has more experience in authoring class-based DSC resources, it is easy to switch to the class-based variants : what is the return type of the Get method ? Of the Set method ? And the Test method ?

Again, this is explicitly written in the PowerShell ISE (and likely other PowerShell editors) built-in snippets, so this can be considered essential knowledge for any DSC resource author.

So, assessing someone’s PowerShell skills/knowledge is really not that difficult, as long as the interviewer ask the right questions and knows his stuff.
By “asking the right questions”, I mean 2 things :

  • Steer away from area-specific or product-specific knowledge (like Exchange, SQL Server or VMware…) unless it is relevant for the job in question.
  • Focus on probing the candidate’s mastery of PowerShell fundamental concepts and the ability to apply them in the real world.