At work, I'm currently deploying a Windows 2008 R2 on a VMware ESXi 4.0, and noticed an odd behaviour. At irregular intervals, the console of the guest OS simply locks up and doesn't do anything anymore. Networking and remote access seem to work fine, it's just the console that is freezed. Restarting the vSphere client doesn't help, and the only way to get rid of the problem is by hard-resetting the VM. The issue only occurs on Windows 2008 R2, the other VMs run fine.

Cause

The problem is caused by the VMware SVGA II driver, and has two underlying issues: the driver itself, and an apparent lack of video RAM (due to the new HW-accelerated GDI of 2008R2/Win7).

Solutions

The most quickest way is to get rid of the VMware SVGA driver but this makes console responsiveness incredibly slow. Therefore, it's not the best solution.

There are several things to do: increasing the video RAM, and upgrade/replace the driver with a newer (and alternate) version. Increasing the RAM stabilized the problem in my case, although I did not wait long enough to see whether the problem was permanently fixed. Either way, I recommend updating the driver as well as the new VMware SVGA 3D driver seriously increases performance over the old one.

Increase video RAM

In a quiet moment, turn off the VM and change its video adapter settings. For some reason, they were set to a fixed of 4MB, which is too low for proper operating 2008R2.

Set the memory size to 32MB, then start the VM again.

Install/update to the 3D driver

Additionally, try to install the new VMware SVGA 3D driver to resolve and increase your performance in the process.

Notice

If you have VMware ESXi 4.0 (no update), the 3D driver is not yet present in the supplied VMware Tools. In this case, navigate to http://packages.vmware.com/tools/esx/index.html and download the latest VMware Tools available.
You can choose to download the ISO of VMware Tools newer than the ESXi version you're running: e.g. if you're running ESXi 4.0, it's no problem to download the 4.1 tools. Therefore, it is recommended to simply download and install the latest VMware Tools, even if you are already running 4.0 update 1 or update 2.

To upgrade/replace the VMware SVGA II driver with the new VMware SVGA 3D driver, follow instructions:

1. First, make sure the right version of VMware Tools are installed (see above notice).
2. Navigate to Device Manager (via Server Manager), and look for the VMware SVGA II driver. Right-click and select Update Driver Software; choose to browse for the driver yourself.

3. Next, navigate to the folder C:\Program Files\Common Files\VMware\Drivers\wddm_video, and continue with the installation.
   If for some reason, you cannot select the driver, choose Let me pick from a list of device drivers on my computer and try to install the driver manually.
4. When finished, reboot the VM. The Device Manager should now show VMware SVGA 3D driver.