April 21, 2014

Tempdb database continues to grow after upgrading vCenter

After upgrading a small customer from 5.1 (Simple Install) to 5.5 we discovered that the system disk was filling up. This system was installed using the MSSQL Express version that ships with vCenter that by default will put databases on the system disk. We soon figured out that tempdb was growing quickly and we were worried that we would soon run out of disk space. When the SQL Server service (+vcenter) is restarted tempdb is cleared, but it would start growing quickly again if the services were started.

In this process we saw the file C:\Program Files\Microsoft SQL Server\MSSQL10_50.VIM_SQLEXP\MSSQL\DATA\tempdb.mdf  growing to 40 Gigabytes before we had to restart services to avoid running out of disk space.

We first suspected that the read_committed_snapshot option was enabled, but it turned out it wasn't:

We also changed the tempdb file location so it wouldn't fill up the data disk, but we were still worried about it's growth rate.

In the end we set a growth limit on the database from SQL Server Configuration.

We defined a fixed growth limit of 2000MB.

After doing this the server was running happily ever after and the disks wouldn't fill up. Setting a growth limit on tempdb may affect database performance negatively, but we found it more important that we wouldn't run out of disk space.

I later also discussed this issues with a database admin and he had also seen tempdb temporarily growing quickly after upgrades to different systems, and he said it would normally stop growing at some point and it's size would eventually go back to normal again.

March 27, 2014

Sphere 5.5 client cannot be installed on a domain controller

When trying to install the fat native vsphere 5.5 windows client  you get the message:
vSphere Client requires Windows XP SP2 or later. 
vSphere Client cannot be installed on a Domain Controller.
The vSphere client has always been possible to install on a domain controller. vCenter also used to be able to coexist on a domain controller, but after 4.0 was introduced it was no longer possible as itincluded a separate ADAM database.

Even though the fat native windows client is being phased out, you still need to use both the web client and the fat native client in order to access all the functionality. For standalone hosts, the native client is the only choice. 

Microsoft in general recommend that you avoid installing third party software on Domain Controllers and that seems to be the reason why VMware has included this check. According to this posting VMware's reason for doing this was:
We did this deliberately to enforce a Microsoft standard that our guys agree with - don't install software on a DC, but they made that decision in isolation. Nothing more than that.  So use the workaround safely and hopefully we can undo this in the future.
The workaround is to install the vsphere client from the command line using the parameter /VSKIP_OS_CHECKS="1".

Another workaround would be to use a thinapped vsphere client package instead. Hopefully we will see a 5.5 client someday at the Thinapped vSphere Client Fling page.

March 11, 2014

Default gateway lost after reboot when using Host Profiles

When using Auto Deploy and Host Profiles with static vmkernel ip configuration, default gateway is not getting set as it should.

It is possible to enter the gateway address manually and it will work nicely until the host is rebooted. But when it boots up again it needs to get the gateway set manually again. VMware has a knowledge base article on this topic: Host Profiles do not save default gateway configuration in vSphere 5.1

A solution that doesn't require manual steps after reboot is normally preferred. A solution solution that works is to set the MAC address of the vmkernel nic to the same address as the physical PXE nic and to set the ip address assignment to DHCP. This can be configured in the Host Profiles:
It is because of this also a good idea to reserve these ip addresses to mac addresses on the DHCP server and reserve dns names accordingly.

We have also observed that when applying this profile for the first time the host will loose networking permanently (after 22% of profile remediation) when using Nexus 1000v switches.  The vmkernel log will have repeated messages like these:
Dropped : DP IP address (0x0) not set
The solution is to reboot the server from DRAC/iLO/RSA and it will boot up just fine with the new Host Profile attached.

Products used:
vCenter 5.5 build 1466327
ESXi build 1474528 and 1331820

March 7, 2014

vMotion not getting enabled on vmkernel interface when using Host Profiles

In an environment where we are using Host Profiles and Auto Deploy we experienced a problem with vMotion not getting enabled on the given vmkernel nic even though it was specified in the Host Profiles.

The error message we had was "Given Services are not enabled on the group". As shown in the picture above we also had an error message regarding Network coredump.

We could easily verify that vMotion had not been enabled by Host Profiles and if we enabled it manually it would work as it should.

After talking to VMware Support we were told that this was a known bug and that a patch will be released in the future.

The workaround to get  vMotion working automatically again was to disable the Network Coredump settings that was configured in Host Profiles.

If you're booting ESXi statelessly or from sdcard you will now get a warning "No coredump target has been configured. Host core dumps cannot be saved."

This warning is completely true and in case of a PSOD no data will get dumped anywhere now that this setting is disabled. If you want to get rid of this warning message you can set UserVars.SuppressCoreDumpWarning to 1. In order to include this setting in your Host Profile you will first need to set it on one of your hosts and then use that host as a reference for your profile (Copy settings from Host).

Products used:
vCenter 5.5 build 1466327
ESXi build 1474528 and 1331820

October 7, 2013

Splunk: Horizon View clients coming up with the same id

Splunk is an excellent tool for collecting log data and generating statistics and graphs based on logs. In a VMware Horizon View environment you by default have no control of the user experience, but there are logs locally on each VDI desktop that provides information on packet loss, latency, and other important things. It is possible to use the Splunk app for VMware PCoIP logs to gather logs from all your VMware View clients into a single console. Even though this app is from 2010 is works fine with Horizon View 5.2.

When using the universal forwarder from the parent VM, all the VMs created from it will use the name and id of the parent VM.

Removing the file inputs.conf from the local directory of the parent VM before creating the pool solves the problem.

Splunk gives a very quick overview of all view clients. For a better view of a single client it is best to use the PCoIP Log Viewer.


July 10, 2013

Nvidia Quadro card and server cabling

In a recent VMware View PoC we were using a Nvidia Quadro card together with APEX 2800 in an IBM x3650M4 Server. One problem we had with this configuration that we had no power cable for the Quadro card. Old Quadro cards have been known to work at degraded performance without such a cable, but this new one would not work at all.


We researched our options and found it surprisingly hard to order such a cable from our official channels. We did however find some cables on eBay & Amazon. The cable we were looking for was a PCI Express 6 pin to 8 pin Power Adapter Cable.

Luckily I discovered that I had such an unused cable in my home lab. In one of my whitebox servers I'm using a modular PSU named Coolermaster SilentPro M600 that came with such a 8-6 cable.

I also have another similar PSU, Coolermaster M2 SilentPro M620, that has a compatible cable which is a 6+2-6 cable.

Note that the card we were testing was a Quadro K4000 card which is not officially supported by VMware (or Nvidia) and the Nvidia driver for ESXi does not support it. It does however still work in a vDGA setup.

June 17, 2013

Error installing VMware Blast: Failed to communicate with LDAP server

VMware Blast installs on the Connection server, not the Security server, but Blast gets available from the Security server after installing it on the connection server. Blast provides clientless access to VMware View desktops through HTML5 instead of a PCoIP/RDP/RGS client.

After trying to install VMware Blast using a local administrator account on the connection server we had the following error message: "1: Failed to communicate with LDAP server. Please check if the installer is running on a Connection Server machine. Contact your administrator for the support."

The user that is used during installation of VMware Blast needs admin privileges in VMware View Manager. Either grant the user access or use another user.