Checking vCenter Server certificate requirements with PowerShell

Given the number and the complexity of certificate-related issues, I wanted an automated way to check whether a certificate file meets the vCenter Server certificate requirements.

There are 2 ways to extract the necessary information from a certificate file : openssl.exe and the cmdlet Get-PfxCertificate, which was introduced with PowerShell 3.0.

Here is the openSSL command and its output :

OpenSSL command
 
Here is the Get-PfxCertificate command and its output :

Get-PfxCertificate
 

I wrote a PowerShell function named Test-VpxdCertificate, which checks if a certificate file (.crt or .cer) meets all the requirements. I chose to build this tool upon the Get-PfxCertificate cmdlet for 2 main reasons :

  • openssl.exe returns plain text and you know what I think about text-parsing in PowerShell.
  •  

  • You don’t really need to have PowerShell 3.0 on the vCenter Server itself.
    You can copy the certificate file to another machine (which has PowerShell 3.0 or later) and run the cmdlet from there.
    Given the huge boost in functionality in 3.0 compared with 2.0 and the fact that 3.0 can be installed on down-level OSes, if you still don’t have any machine with 3.0 or later, you probably don’t need, or want, or deserve PowerShell automation.

What does it check ?

 
The vCenter Server certificate requirements are not clearly and exhaustively documented. There is something in the vSphere 5.0 documentation, but this is old and not exhaustive.
Fortunately, we can deduce most of the requirements from the KB article explaining how to create certificate requests for custom certificates.

So here are the certificate requirements for vCenter Server 5.x :

  • Certificate must be X.509 v3.
  • Certificate should begin with : “—–BEGIN CERTIFICATE—–“.
  • Certificate should end with : “—–END CERTIFICATE—–“.
  • The subject Alternative Name must contain the FQDN of the vCenter server.
  • The certificate must be valid : the current date must be between the “Valid from” date and the “Valid to” date.
  • The Key usage must contain the following usages : Digital Signature, Key Encipherment, Data Encipherment
  • The Enhanced key usage must contain : “Server Authentication” and “Client Authentication”.
  • The public key algorithm must be : RSA (2048 Bits).
  • The certificate must NOT be a wildcard certificate.
  • The signature hash algorithm must be SHA256, SHA384, or SHA512.

And here are the requirements for vCenter Server 6.0 :

  • Certificate must be X.509 v3.
  • Certificate should begin with : “—–BEGIN CERTIFICATE—–“.
  • Certificate should end with : “—–END CERTIFICATE—–“.
  • The subject Alternative Name must contain the FQDN of the vCenter server.
  • The certificate must be valid : the current date must be between the “Valid from” date and the “Valid to” date.
  • The Key usage must contain the following usages : Digital Signature, Key Encipherment.
  • The public key algorithm must be : RSA (2048 Bits).
  • The certificate must NOT be a wildcard certificate.
  • The signature hash algorithm must be SHA256, SHA384, or SHA512.

How to use it :

 
CertFilePath2
 
Here, we specified the certificate file path and the vCenter FQDN because this was not run from the vCenter Server itself.
If we run this command from the vCenter Server and if it is able to resolve its own FQDN, then there is no need to use the -vCenterFQDN parameter.
If we run this command from the vCenter Server and if the certificate is at its default location, then there is no need to use the -CertFilePath parameter.

The function performs a test for each of the requirements listed above and outputs an object with a property corresponding to each of these tests. The value of each property is either True or False. True means that the certificate passed the corresponding test and False means that it failed the test.

As we have seen above, the vCenter Server certificate requirements are different between vCenter 5.x and 6.0.
So, the -VpxdVersion parameter is used to specify the version of vCenter Server and the value of this parameter determines which tests are performed.
If the function is run from the vCenter Server itself, it will detect the vCenter version by itself and use this value, unless it is specified via the -VpxdVersion parameter.

By the way, the parameters -CertFilePath, -vCenterServerFQDN and -VpxdVersion are positional, so we can save some typing, like so :

PS C:\> Test-VpxdCertificate "$env:USERPROFILE\Desktop\rui.crt" "VC1.vcloud.local" "5.x"


Certificate ends with "-----END CERTIFICATE-----"             : True
Signature hash algorithm is SHA256 or higher                  : True
Certificate is NOT a wildcard certificate                     : True
Current date is between the "Valid from" and "Valid to" dates : True
Certificate has the required key usages                       : True
Certificate begins with "-----BEGIN CERTIFICATE-----"         : True
Certificate has the required enhanced key usages              : False
Subject alternative names contain the vCenter FQDN            : True
Public key algorithm is RSA 2048 or higher                    : True
Certificate is X.509 v3                                       : True

 
Here, we see that the vCenter certificate doesn’t meet the requirement regarding the enhanced key usages. To have better insight into why a certificate failed a test, we can use the -Verbose parameter. This allows us to see the certificate properties which are checked and their values :

Verbose2
 
Now, we can see why it failed the test for enhanced key usage : it is missing the “Client Authentication” enhanced key usage.

But there is no requirement for enhanced key usage in vCenter 6.0, so this certificate passes all the checks for 6.0 :

vpxd60
 
On the other side of the verbosity spectrum, we can just output a boolean value : True or False using the -Quiet parameter. True means that the specified certificate meets all the requirements, False means that it doesn’t meet all the requirements :

Quiet2
 
This behaviour is conform to the -Quiet parameter of Test-Connection and other Test-* cmdlets. This is useful mostly when we call the cmdlet from another automation tool and we do something, depending on the result (True or False).

As usual, I packaged the function in a nice little module, which is available here.

Auditing events and changes in a vCenter environment

When we get a customer opening a support case telling : “… was working before and now it is not working anymore”.

The first question which pops up is :

“What changed ? What happened before it stopped working ?”

Quite often, the customer is not sure. Or, some customers have a selective memory and tend to “forget” configuration changes for various reasons.

Also, there are cases where we know something happened but we want to know if it was a user-initiated action. Or, the customer might even want us to retrieve by who a specific action was initiated (probably to engage in some finger-pointing/blame-assigning fun).

The vSphere client being limited to display a maximum of 1000 events, this corresponds generally to only a few days worth of events. Besides, the “Export Events” functionality of the C# client and the Web client doesn’t work well (http://kb.vmware.com/kb/2071612).
As a result, we cannot rely on the vSphere client/web client for auditing purposes.

You might think : “all this information is in the tables VPX_EVENT and VPX_EVENT_ARG in the vCenter database, I can just query it”. I would answer with 3 points :

  • This data is not user-friendly and requires further processing to become consumable information
  • A kitten dies every time you query the VCDB
  • If you want to keep 2, 3, or 5 years of events for compliance purposes, the vCenter database is a typical OLTP system and should be used as such , not as a Data Warehouse !

To make querying this kind of information easier, I wrote a function Audit-vCenterOperation .

This is essentially a wrapper around the PowerCLI cmdlet Get-VIEvent with a bit of intelligence to filter out irrelevant events and 2 additional parameters : one to select actions initiated by a specific user and one to select actions of a specific type.

Let’s see the stuff we can do with it.

To track when the configuration of the cluster “Test Cluster” was changed and by who :

vCenterOperation Cluster Reconfigure
 
Notice that here, we sent the cluster object from the pipeline. In fact, we can throw any vCenter inventory object at it, through the pipeline, and the object will be bound to the -Entity parameter of the function.

The parameter -OperationType allows to filter the operations down to a specific type or topic.
The possible values are : Create, Reconfigure, Remove, DRS, HA, Password, Migration, Snapshot, Power, Alarm, Datastore, Permission.
Don’t worry, we don’t have to know all the possible values, we can just make use of tab completion.

To track all the migrations (vMotion and Storage vMotion) for all VMs over the last 6 hours :

C:\> Audit-vCenterOperation -OperationType Migration -Start (Get-Date).AddHours(-6)


Key               : 10624
CreatedTime       : 11/25/2015 9:57:58 AM
UserName          : User
Datacenter        : Test-DC
ComputeResource   : Test Cluster
Host              : 192.168.1.187
Vm                : Test-VM
Message           : Migration of virtual machine Test-VM from 192.168.1.187, iscsi-lefthand-1 to 192.168.1.188, iscsi-lefthand-1 completed
NewConfigSettings :

Key               : 10632
CreatedTime       : 11/25/2015 10:01:16 AM
UserName          : VSPHERE.LOCAL\Administrator
Datacenter        : Test-DC
ComputeResource   : Test Cluster
Host              : 192.168.1.187
Vm                : New-VM
Message           : Task: Migrate virtual machine
NewConfigSettings :

Key               : 10633
CreatedTime       : 11/25/2015 10:01:17 AM
UserName          : VSPHERE.LOCAL\Administrator
Datacenter        : Test-DC
ComputeResource   : Test Cluster
Host              : 192.168.1.187
Vm                : New-VM
Message           : Migrating New-VM from 192.168.1.187, ISCSI-2 to 192.168.1.188, ISCSI-2 in
                    Test-DC
NewConfigSettings :

 
We can easily see the VM name, the source host/datastore and the destination host/datastore.

What is this empty property named NewConfigSettings ?
It is populated only for “reconfigure” operations on VMs. It contains the VM setting(s) which have been changed and their new value :

PowerCLI C:\> Get-VM "New-VM" | Audit-vCenterOperation -OperationType Reconfigure -Start (Get-Date).
AddDays(-7)


Key               : 10376
CreatedTime       : 11/24/2015 7:59:08 AM
UserName          : VSPHERE.LOCAL\Administrator
Datacenter        : Test-DC
ComputeResource   : Test Cluster
Host              : 192.168.1.188
Vm                : New-VM
Message           : Reconfigured New-VM on 192.168.1.188 in Test-DC
NewConfigSettings : @{ChangeVersion=2015-11-13T16:22:54.801558Z;
                    Files=VMware.Vim.VirtualMachineFileInfo; MemoryMB=3072; NumCPUs=2;
                    VmProfile=VMware.Vim.VirtualMachineProfileSpec[]}



PowerCLI C:\> Get-VM "New-VM" | Audit-vCenterOperation -OperationType Reconfigure -Start (Get-Date).
AddDays(-7) | Select-Object -ExpandProperty NewConfigSettings | Format-List *


ChangeVersion : 2015-11-13T16:22:54.801558Z
Files         : VMware.Vim.VirtualMachineFileInfo
MemoryMB      : 3072
NumCPUs       : 2
VmProfile     : {VMware.Vim.VirtualMachineEmptyProfileSpec}

 
In this case, the number of vCPUs and the memory amount were changed. Unfortunately, there is no easy way to get the original values of the changed settings, so it gets only the new values.

If we have an ESXi host which have been rebooted and we want to know if it was a user-initiated action (and if yes, by who) :

vCenterOperation power
 
Aha ! It was the (suspiciously named) user “RogueAdmin”.

To track down snapshot operations which happened today on a VM named “New-VM”, here is how to do it :

vCenterOperation Snapshot
 
This “RogueAdmin” user is definitely doing weird things. Let’s track down everything he has done (output cut for brevity) :

PowerCLI C:\> Audit-vCenterOperation -Username "VCLOUD\RogueAdmin"


Key               : 13157
CreatedTime       : 11/26/2015 10:03:30 AM
UserName          : VCLOUD\RogueAdmin
Datacenter        :
ComputeResource   :
Host              :
Vm                :
Message           : User VCLOUD\RogueAdmin@192.168.1.10 logged in as VMware vim-java 1.0
NewConfigSettings :

Key               : 13162
CreatedTime       : 11/26/2015 10:04:58 AM
UserName          : VCLOUD\RogueAdmin
Datacenter        : Test-DC
ComputeResource   : Test Cluster
Host              : 192.168.1.188
Vm                :
Message           : User logged event: Because I can
NewConfigSettings :

 
That’s it.

This Audit-vCenterOperation is in a module available here. It requires Powershell 3.0 or later and PowerCLI 5.5 or later.