Vmware Vcenter 4.1 Upgrade and Search Bug

We have upgraded our virtual center from 4.0 update2 to 4.1 on the 4th of August. After a period of time, we have realized that the search function did not find new vms that we have created since the upgrade.
Made a little bit troubleshooting and diagnosed the problem's cause. It was Tomcat's VcCache-default-0.XhiveDatabase.DB cache database, not working properly. We decided to open a call to vmware and learned that this is accepted as a bug waiting to be fixed soon. And desperately began to wait for the solution.
Today by chance, I made a search in google about the problem again and found the solution!
It was solved on the 25th of August in another case declared to vmware again.
The bug is solved by raising the memory dedicated to tomcat java application. In our case it is solved by raising up to 2 GB.
Here is the link that I found.
I suppose also an official article would be available soon about the bug.

http://communities.vmware.com/thread/282117

Vmware Tools Automatic Upgrade

We have been facing a problem in our environment to upgrade the vmware tools of our guests.
The problem is that some of the guests never automatically upgrades vmware tools even if you have enabled "Check and upgrade tools during power cycling" or selected "Upgrade Tools" within vmware tools tray.
Basically the guest does nothing after you trigger upgrade or power cycle.
After searching for a while we have found an article which has no relation with our case but somehow true and solved the problem.
The problem was caused by a prior vmware tools installation file "vmwaretoolsupgrader.exe" in C:\windows\temp directory to be deleted before triggering the upgrade. After deleting the file, everything works seamless. here is i link to the kb;

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1023459

Tips for a Successful Vmware Converter Operation

For the ones making P2V operations using Vmware Converter plug-in, sometimes it is a nightmare to troubleshoot why the agent fails to proceed. I've tried to compile my experiences on the Vmware Converter plug-in for a successful import without facing any errors.

1. First of all make sure you read and open the ports necessary if there is firewall(s) between vc, hosts and guests. you can find the official link to the article below;
   http://kb.vmware.com/kb/1010056/
2. Before you launch the plug-in, make sure the guest resolves both the vc and host with their fqdn. If necessary, enter the fqdns in the host file.
3. Always reboot the guest after the agent is installed (even the vmware says a reboot is not necessary)
4. Make sure the admin$ of the guest is reachable from vc (in order to install agent).
5. And finally if all of the above are true and you still see face an error while importing, you can find a verbose log named vmware-converter-agent-0.log (increments if a file size threshold exceeds) under the client's below directory;
   
   %allusersprofile%\Application Data\VMware\VMware Converter Enterprise\Logs

Terminal Server 2008 - Issues and Solutions - Part 1

I have been working on a project where we setup a TS 2008 farm (made up of three virtual servers) on a three node VMware Infrastructure 3 environment. During the implementation of the systems, we had to tackle a number of problems and discrepancies. Some of the problems were related to Windows 2008 compatibility, and some of them were direct results of feature set changes in the new OS.

1. Windows 2008 on VMWare ESX and Painfully Slow Performance

Our Session Broker Load Balanced farm included three virtual Windows 2008 Terminal Services servers. Along with them we had two seperate file servers. One of them was designated for roaming terminal services profiles, the second one was for shared MS Office files and users' home folders. All systems were running on Windows 2008, except one single Windows 2003 domain controller for another which was used for domain migration. ESX hosts were running on 16GB of RAM and Quad Core 2.5GHz CPUs, while guest TS servers were configured with 4GB of RAM and 4 CPUs.

Right after the installation and data migration, we invited some pilot users to test the environment. They were supposed to test standard Microsoft Office Applications, along with their third party Business Management Software. All pilot users agreed on one point: systems were "painfully slow". Logons were taking extremely long, sometimes up to 5-6 minutes. Opening a 40KB Excel file was taking a minute on average and sometimes users had to wait forever for a small file to open. Furthermore, working in their Business Software was almost impossible.

The environment described in that article was very similar to ours with one exception: while we were running Symantec Corporate Edition 10.2, their problems were caused by Symantec EndPoint Protection Server 11.0. Knowing that both security solutions share the same roots and just to see the effects, we removed SAV Corporate Edition from the file servers. After the reboot, performance returned back to normal.

Removing the SAV Corporate Edition from the Terminal Servers helped reaching the normal perfomance values. We were able to open any Excel file with no latency. As described in the article above, instead of Symantec Corporate Edition 10.2, we upgraded the systems to Symantec EndPoint Protection 11 MR3. Installing Symantec EndPoint 11 MR3 did not affect the performance, so I hope the problem has been resolved with this release. We have also installed a number of OS patches to the systems after this, so far the systems are still running properly, I assume Symantec had adressed and resolved the problem with MR3. At the time of this writing, I can see that the latest release for Symantec Endpoint Protection is MR4. We are planning to implement this soon, I will also update the post if it affects the performance.

2. Terminal Services 2008 and Adobe Reader Problem

Some users were not able to run Adobe Reader on the Terminal Servers. After every failure, they were getting "Adobe Reader has stopped working" error message. With each failure, the following message was logged in the Application log:

Log Name: Application

Source: Application Error

Date: 12/16/2008 1:53:03 PM

Event ID: 1000

Task Category: (100)

Level: Error

Keywords: Classic

User: N/A

Computer: TS01.domain.local

Description:

Faulting application AcroRd32.exe, version 9.0.0.332, time stamp 0x4850f0a3, faulting module Annots.api, version 9.0.0.332, time stamp 0x4850e57f, exception code 0xc0000005, fault offset 0x001bd9e0, process id 0x2e1c, application start time 0x01c95fc0491c84c1.

Since the error points out to "Annots.api" file, my initial response was to locate the file in Plug_ins (C:\Program Files\Adobe\Reader 9.0\Reader\plug_ins) folder and rename it to "annots1.api". However this did not help. The user received the exact same ""Adobe Reader has stopped working" error, and in the application log complained about "annots1.api" this time.

My second action was to move the file out of plug_ins folder. After the move the problem disappeared. However, I was sure that this is not a wise solution. So as a final test, I moved the file back to the plug_ins folder then enabled Compatibility mode and set it to "Windows XP SP2" on "Acrord32.exe" file. I was able to verify that this action resolved the problem for good. No problems were encountered after this change. I also had to repeat this action two other Terminal Servers.

3. Terminal Services 2008 and Session Broker Problem

To be able to distribute the load among three Terminal Servers, we have configured Terminal Services Session Broker service and Session Broker Load Balancing. I will not go through all the configuration steps, you can easily find detailed configuration information from many different resources in the web. However, I would like to mention about a discrepancy in Session Broker configuration. One of the steps during configuration is to supply each Terminal Server with the name of the Session Broker Server. This can be achieved by two different methods:

i. You can specify the name manually on each server's Terminal Services Configuration

ii. You can specify the name in a GPO and assign this GPO to all Terminal Servers

Regardless of the configuration method, as per Microsoft, the name you need to specify can be either in host name, IP address of FQDN form. In GPO Editor, if you examine the "Explain" section of the group policy setting (Computer Configuration\Policies\Administrative Templates\Windows Components\Terminal Services\Terminal Server\TS Session Broker\Configure TS Session Broker server name), it reads:

"If you enable this policy setting, you must specify the TS Session Broker server, using either its host name, IP address, or fully qualified domain name. If you specify a name or IP address for the TS Session Broker server that is not valid, an error message is logged in Event Viewer on the terminal server."

Unfortunately, I found that above statement is only partially correct. If you select to use FQDN, you will eventually see the following errors on each Terminal Server's event log:

Log Name: System
Source: Microsoft-Windows-TerminalServices-SessionBroker-Client
Date: 12/6/2008 9:36:19 PM
Event ID: 1014
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: TS01.domain.local
Description:
The server failed to retrieve the security identifier (SID) of the TS Session Broker server.
Win32 error code: 0x6FC.

You will find that Session Broker functionality intermittently stops (the service stays running but terminal servers fail to join) when you use FQDN in this configuration. To get rid of this warning and intermittent failures, you need to specify host name (i.e. NetBIOS name) of the Session Broker server. After changing it to the host name, the warning does not show up anymore. Somehow, FQDN does not work properly. I did not get a chance to test out the effect of using straight IP address for this setting, if you have similar experience and if you are using the IP address, please let me know if it works.

Install XP Guest on ESX Systems

As ESX does not support IDE disk controllers, it will be tricky to install an XP guest on the ESX host. You will either need LSI or Bus Logic SCSI drivers in FLP image format.
So it is a question where to find or how to create these floppy images.
Here is a link to download official BusLogic floppy image (also can be found in VmWare site but I could not find it easily in my second attempt...)

Download Floppy Image

VMware Converter Fails at 99%

Last week, I was migrating some physical servers to VMware Server images with VMware Converter tool. I was using the latest and improved version of VMware Converter (v3.0.3 - Build 89816). I had two servers to migrate: one SBS 2003 server and one Windows 2003 based Terminal Server. I had to run the converter on a third piece of hardware while the servers were up and running, installing the converter agent service on them (AKA "hot clone"). SBS server had two volumes, and the total used disk space was around 40GB. Terminal Server had two volumes and used disk space was around 25GB. 

First P2V migration session ran approximately for 6 hours on the SBS server and completed without any problems. I started the second session on the Terminal Server and checked the status every hour. 

If you are familiar with the P2V process, you probably know that the tool initially creates the vmdk files, then fills them in with the files from the source machine (Cloning Phase). When all files are transferred to the virtual disks (vmdk files), then final step is called "conversion". At this stage, the storage, HAL and network specific driver files and their configuration gets modified. This step is necessary, otherwise the virtual disk will not be able to boot. 

Anyhow, after running for 3 hours and 30 minutes, my Terminal Server P2V migration session failed at 99% with "Unknown Error"! A quick Google search revealed that the 97% and 99% range belongs to Customization/Reconfig phase, meaning that it failed at the "conversion" step. VMware converter has the ability to manually run the "conversion". I selected the newly created virtual machine as the source, and ran the "Conversion Wizard", but the process failed again. This time, I decided to examine the log files to see what's wrong. Converter creates log files in two locations: Converter Console log files reside in user's profile folder (in my Vista laptop, they are located at: "%userprofile%\AppData\Local\Temp"). Converter Agent log files reside in "Windows\Temp" folder. After a quick review, I realized that the log files in the profile folder did not provide the detail level I needed. Then after going through the log files in "Windows\Temp" (vmware-converter-n.log) I found what I was looking for:

[#2] [2008-08-23 07:33:53.707 'App' 3932 verbose] [registryUpdate,73] Applying registry update S_SYMMPI
[#2] [2008-08-23 07:33:53.708 'App' 3932 verbose] [registryHiveProxy,176] Saving registry key ControlSet001\Services\symmpi to \\.\vstor2-p2v30-B4BDB4BD007E00000000000003000000\WINDOWS\$Reconfig$\ControlSet001-Services-symmpi-reg
[#2] [2008-08-23 07:33:54.174 'App' 3932 verbose] [registryHiveProxy,152] Restoring registry key ControlSet001\Services\symmpi from C:\Program Files\VMware\VMware Converter\data\52-S_SYMMPI
[#2] [2008-08-23 07:33:54.202 'App' 3932 verbose] [fileUpdate,69] Update File system32\drivers\symmpi.sys
[#2] [2008-08-23 07:33:54.202 'App' 3932 verbose] [volumeProxy,95] Checking if WINDOWS\system32\drivers\symmpi.sys exists...
[#2] [2008-08-23 07:33:54.205 'App' 3932 verbose] [volumeProxy,102] WINDOWS\system32\drivers\symmpi.sys does not exist!
[#2] [2008-08-23 07:33:54.223 'App' 3932 error] [fileUpdate,120] Unable to find symmpi.sys in the specified CAB files
[#2] [2008-08-23 07:33:54.250 'App' 3932 info] [imageProcessingTaskStep,194] UfaSysReconfig::task{1}::task{2} step "applying reconfigurations" destroyed

The Converter was failing at a specific file, symmpi.sys! Because it was looking for the file in "WINDOWS\system32\drivers" folder, failing to find it there, then aborting the process. Another Google search revealed that this was the LSI Logic Miniport Driver file. Luckily, I found the same file in one of my other virtual machines, in the "WINDOWS\system32\drivers" folder. To copy the file to my failing virtual Terminal Server, I connected its C:\ drive as an additional disk to my old virtual machine. After this, I could easily add a drive letter to the drive and was able copy the file.

I shutdown my old virtual machine, removed the drive connection. Then I ran the "conversion Wizard" one more time selecting the new virtual Terminal Server as the source. The "Conversion Wizard" only ran for a couple of minutes, then completed successfully. After this step, I was able to verify that I could successfully boot my new virtual Terminal Server.

My Google searches during this troubleshooting period showed me that this is a very common error. I spent about 2 hours and 30 minutes to find the solution and I hope this post can speed up your troubleshooting process. Please use the comments link below if you have any questions.

MS Virtual Server to VMware ESX Migration Issue

Last week, I was trying to migrate some virtual guest servers from MS Virtual Server 2005 environment to VMware ESX. I had to run the "import" task three times to achieve a successful migration, and had to spend 4-5 hours for a simple migration operation. Hope the tips here will save you some time.
At my first attempt, the process seemed to halt at 22%. There was no activity in the Virtual Center and there were no log file entries at the VMware side. I examined the event viewer logs of the Virtual Server 2005 guest machine and found that the following error was logged:
Log Name: System
Source: msvmscsi
Date: 5/15/2008 5:14:45 PM
Event ID: 9
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: xxx-xxx.xxx.xxx
Description:The device, \Device\Scsi\msvmscsi1, did not respond within the timeout period.

A quick Google search directed me to this link:
http://www.eventid.net/display.asp?eventid=9&eventno=9355&source=vmscsi&phase=1
Before my second attempt I created the following value as described in the above link, and rebooted the guest server:
Key: HKEY_LOCAL_MACHINE/System/CurrentControlSet/Services/Disk
Sub Key: TimeOutValue
Data Type: REG_DWORD
Value: 03c hex (or 60 decimal)

The import process ran up to 97% in my second attempt, then halted there. Same error messages were displayed in the event log of the guest machine.
Before the third attempt, I decided to increase the same registry key value as follows and rebooted the guest server:
Key: HKEY_LOCAL_MACHINE/System/CurrentControlSet/Services/Disk
Sub Key: TimeOutValue
Data Type: REG_DWORD
Value: 05a hex (or 90 decimal)
After this final change and reboot, third import attempt ran successfully. As a final step, I removed the registry setting at the ESX guest machine since it is no longer needed.