Auditing events and changes in a vCenter environment

When we get a customer opening a support case telling : “… was working before and now it is not working anymore”.

The first question which pops up is :

“What changed ? What happened before it stopped working ?”

Quite often, the customer is not sure. Or, some customers have a selective memory and tend to “forget” configuration changes for various reasons.

Also, there are cases where we know something happened but we want to know if it was a user-initiated action. Or, the customer might even want us to retrieve by who a specific action was initiated (probably to engage in some finger-pointing/blame-assigning fun).

The vSphere client being limited to display a maximum of 1000 events, this corresponds generally to only a few days worth of events. Besides, the “Export Events” functionality of the C# client and the Web client doesn’t work well (http://kb.vmware.com/kb/2071612).
As a result, we cannot rely on the vSphere client/web client for auditing purposes.

You might think : “all this information is in the tables VPX_EVENT and VPX_EVENT_ARG in the vCenter database, I can just query it”. I would answer with 3 points :

  • This data is not user-friendly and requires further processing to become consumable information
  • A kitten dies every time you query the VCDB
  • If you want to keep 2, 3, or 5 years of events for compliance purposes, the vCenter database is a typical OLTP system and should be used as such , not as a Data Warehouse !

To make querying this kind of information easier, I wrote a function Audit-vCenterOperation .

This is essentially a wrapper around the PowerCLI cmdlet Get-VIEvent with a bit of intelligence to filter out irrelevant events and 2 additional parameters : one to select actions initiated by a specific user and one to select actions of a specific type.

Let’s see the stuff we can do with it.

To track when the configuration of the cluster “Test Cluster” was changed and by who :

vCenterOperation Cluster Reconfigure
 
Notice that here, we sent the cluster object from the pipeline. In fact, we can throw any vCenter inventory object at it, through the pipeline, and the object will be bound to the -Entity parameter of the function.

The parameter -OperationType allows to filter the operations down to a specific type or topic.
The possible values are : Create, Reconfigure, Remove, DRS, HA, Password, Migration, Snapshot, Power, Alarm, Datastore, Permission.
Don’t worry, we don’t have to know all the possible values, we can just make use of tab completion.

To track all the migrations (vMotion and Storage vMotion) for all VMs over the last 6 hours :

C:\> Audit-vCenterOperation -OperationType Migration -Start (Get-Date).AddHours(-6)


Key               : 10624
CreatedTime       : 11/25/2015 9:57:58 AM
UserName          : User
Datacenter        : Test-DC
ComputeResource   : Test Cluster
Host              : 192.168.1.187
Vm                : Test-VM
Message           : Migration of virtual machine Test-VM from 192.168.1.187, iscsi-lefthand-1 to 192.168.1.188, iscsi-lefthand-1 completed
NewConfigSettings :

Key               : 10632
CreatedTime       : 11/25/2015 10:01:16 AM
UserName          : VSPHERE.LOCAL\Administrator
Datacenter        : Test-DC
ComputeResource   : Test Cluster
Host              : 192.168.1.187
Vm                : New-VM
Message           : Task: Migrate virtual machine
NewConfigSettings :

Key               : 10633
CreatedTime       : 11/25/2015 10:01:17 AM
UserName          : VSPHERE.LOCAL\Administrator
Datacenter        : Test-DC
ComputeResource   : Test Cluster
Host              : 192.168.1.187
Vm                : New-VM
Message           : Migrating New-VM from 192.168.1.187, ISCSI-2 to 192.168.1.188, ISCSI-2 in
                    Test-DC
NewConfigSettings :

 
We can easily see the VM name, the source host/datastore and the destination host/datastore.

What is this empty property named NewConfigSettings ?
It is populated only for “reconfigure” operations on VMs. It contains the VM setting(s) which have been changed and their new value :

PowerCLI C:\> Get-VM "New-VM" | Audit-vCenterOperation -OperationType Reconfigure -Start (Get-Date).
AddDays(-7)


Key               : 10376
CreatedTime       : 11/24/2015 7:59:08 AM
UserName          : VSPHERE.LOCAL\Administrator
Datacenter        : Test-DC
ComputeResource   : Test Cluster
Host              : 192.168.1.188
Vm                : New-VM
Message           : Reconfigured New-VM on 192.168.1.188 in Test-DC
NewConfigSettings : @{ChangeVersion=2015-11-13T16:22:54.801558Z;
                    Files=VMware.Vim.VirtualMachineFileInfo; MemoryMB=3072; NumCPUs=2;
                    VmProfile=VMware.Vim.VirtualMachineProfileSpec[]}



PowerCLI C:\> Get-VM "New-VM" | Audit-vCenterOperation -OperationType Reconfigure -Start (Get-Date).
AddDays(-7) | Select-Object -ExpandProperty NewConfigSettings | Format-List *


ChangeVersion : 2015-11-13T16:22:54.801558Z
Files         : VMware.Vim.VirtualMachineFileInfo
MemoryMB      : 3072
NumCPUs       : 2
VmProfile     : {VMware.Vim.VirtualMachineEmptyProfileSpec}

 
In this case, the number of vCPUs and the memory amount were changed. Unfortunately, there is no easy way to get the original values of the changed settings, so it gets only the new values.

If we have an ESXi host which have been rebooted and we want to know if it was a user-initiated action (and if yes, by who) :

vCenterOperation power
 
Aha ! It was the (suspiciously named) user “RogueAdmin”.

To track down snapshot operations which happened today on a VM named “New-VM”, here is how to do it :

vCenterOperation Snapshot
 
This “RogueAdmin” user is definitely doing weird things. Let’s track down everything he has done (output cut for brevity) :

PowerCLI C:\> Audit-vCenterOperation -Username "VCLOUD\RogueAdmin"


Key               : 13157
CreatedTime       : 11/26/2015 10:03:30 AM
UserName          : VCLOUD\RogueAdmin
Datacenter        :
ComputeResource   :
Host              :
Vm                :
Message           : User VCLOUD\RogueAdmin@192.168.1.10 logged in as VMware vim-java 1.0
NewConfigSettings :

Key               : 13162
CreatedTime       : 11/26/2015 10:04:58 AM
UserName          : VCLOUD\RogueAdmin
Datacenter        : Test-DC
ComputeResource   : Test Cluster
Host              : 192.168.1.188
Vm                :
Message           : User logged event: Because I can
NewConfigSettings :

 
That’s it.

This Audit-vCenterOperation is in a module available here. It requires Powershell 3.0 or later and PowerCLI 5.5 or later.

Leave a Reply

Your email address will not be published. Required fields are marked *