VMware Converter Fails at 99%

Last week, I was migrating some physical servers to VMware Server images with VMware Converter tool. I was using the latest and improved version of VMware Converter (v3.0.3 - Build 89816). I had two servers to migrate: one SBS 2003 server and one Windows 2003 based Terminal Server. I had to run the converter on a third piece of hardware while the servers were up and running, installing the converter agent service on them (AKA "hot clone"). SBS server had two volumes, and the total used disk space was around 40GB. Terminal Server had two volumes and used disk space was around 25GB. 

First P2V migration session ran approximately for 6 hours on the SBS server and completed without any problems. I started the second session on the Terminal Server and checked the status every hour. 

If you are familiar with the P2V process, you probably know that the tool initially creates the vmdk files, then fills them in with the files from the source machine (Cloning Phase). When all files are transferred to the virtual disks (vmdk files), then final step is called "conversion". At this stage, the storage, HAL and network specific driver files and their configuration gets modified. This step is necessary, otherwise the virtual disk will not be able to boot. 

Anyhow, after running for 3 hours and 30 minutes, my Terminal Server P2V migration session failed at 99% with "Unknown Error"! A quick Google search revealed that the 97% and 99% range belongs to Customization/Reconfig phase, meaning that it failed at the "conversion" step. VMware converter has the ability to manually run the "conversion". I selected the newly created virtual machine as the source, and ran the "Conversion Wizard", but the process failed again. This time, I decided to examine the log files to see what's wrong. Converter creates log files in two locations: Converter Console log files reside in user's profile folder (in my Vista laptop, they are located at: "%userprofile%\AppData\Local\Temp"). Converter Agent log files reside in "Windows\Temp" folder. After a quick review, I realized that the log files in the profile folder did not provide the detail level I needed. Then after going through the log files in "Windows\Temp" (vmware-converter-n.log) I found what I was looking for:

[#2] [2008-08-23 07:33:53.707 'App' 3932 verbose] [registryUpdate,73] Applying registry update S_SYMMPI
[#2] [2008-08-23 07:33:53.708 'App' 3932 verbose] [registryHiveProxy,176] Saving registry key ControlSet001\Services\symmpi to \\.\vstor2-p2v30-B4BDB4BD007E00000000000003000000\WINDOWS\$Reconfig$\ControlSet001-Services-symmpi-reg
[#2] [2008-08-23 07:33:54.174 'App' 3932 verbose] [registryHiveProxy,152] Restoring registry key ControlSet001\Services\symmpi from C:\Program Files\VMware\VMware Converter\data\52-S_SYMMPI
[#2] [2008-08-23 07:33:54.202 'App' 3932 verbose] [fileUpdate,69] Update File system32\drivers\symmpi.sys
[#2] [2008-08-23 07:33:54.202 'App' 3932 verbose] [volumeProxy,95] Checking if WINDOWS\system32\drivers\symmpi.sys exists...
[#2] [2008-08-23 07:33:54.205 'App' 3932 verbose] [volumeProxy,102] WINDOWS\system32\drivers\symmpi.sys does not exist!
[#2] [2008-08-23 07:33:54.223 'App' 3932 error] [fileUpdate,120] Unable to find symmpi.sys in the specified CAB files
[#2] [2008-08-23 07:33:54.250 'App' 3932 info] [imageProcessingTaskStep,194] UfaSysReconfig::task{1}::task{2} step "applying reconfigurations" destroyed

The Converter was failing at a specific file, symmpi.sys! Because it was looking for the file in "WINDOWS\system32\drivers" folder, failing to find it there, then aborting the process. Another Google search revealed that this was the LSI Logic Miniport Driver file. Luckily, I found the same file in one of my other virtual machines, in the "WINDOWS\system32\drivers" folder. To copy the file to my failing virtual Terminal Server, I connected its C:\ drive as an additional disk to my old virtual machine. After this, I could easily add a drive letter to the drive and was able copy the file.

I shutdown my old virtual machine, removed the drive connection. Then I ran the "conversion Wizard" one more time selecting the new virtual Terminal Server as the source. The "Conversion Wizard" only ran for a couple of minutes, then completed successfully. After this step, I was able to verify that I could successfully boot my new virtual Terminal Server.

My Google searches during this troubleshooting period showed me that this is a very common error. I spent about 2 hours and 30 minutes to find the solution and I hope this post can speed up your troubleshooting process. Please use the comments link below if you have any questions.

Backup Exec 11d and Windows 2008 Compatibility

From the referrals of this blog site, I can see that many users are trying to find out if Backup Exec 11d supports Windows Server 2008. Unfortunately, Backup Exec 11d does not officially support Windows Server 2008. If you want to backup Windows Server 2008, you will need Backup Exec 12. If you purchased 11d recently (during the past year), you may also be eligible for free version 12 upgrade. Hope this post provides the clarification you have been looking for.

MS Virtual Server to VMware ESX Migration Issue

Last week, I was trying to migrate some virtual guest servers from MS Virtual Server 2005 environment to VMware ESX. I had to run the "import" task three times to achieve a successful migration, and had to spend 4-5 hours for a simple migration operation. Hope the tips here will save you some time.
At my first attempt, the process seemed to halt at 22%. There was no activity in the Virtual Center and there were no log file entries at the VMware side. I examined the event viewer logs of the Virtual Server 2005 guest machine and found that the following error was logged:
Log Name: System
Source: msvmscsi
Date: 5/15/2008 5:14:45 PM
Event ID: 9
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: xxx-xxx.xxx.xxx
Description:The device, \Device\Scsi\msvmscsi1, did not respond within the timeout period.

A quick Google search directed me to this link:
http://www.eventid.net/display.asp?eventid=9&eventno=9355&source=vmscsi&phase=1
Before my second attempt I created the following value as described in the above link, and rebooted the guest server:
Key: HKEY_LOCAL_MACHINE/System/CurrentControlSet/Services/Disk
Sub Key: TimeOutValue
Data Type: REG_DWORD
Value: 03c hex (or 60 decimal)

The import process ran up to 97% in my second attempt, then halted there. Same error messages were displayed in the event log of the guest machine.
Before the third attempt, I decided to increase the same registry key value as follows and rebooted the guest server:
Key: HKEY_LOCAL_MACHINE/System/CurrentControlSet/Services/Disk
Sub Key: TimeOutValue
Data Type: REG_DWORD
Value: 05a hex (or 90 decimal)
After this final change and reboot, third import attempt ran successfully. As a final step, I removed the registry setting at the ESX guest machine since it is no longer needed.

Windows 2008 - Detailed Technical Articles

Windows Server Performance Team Blog, has been publushing very nice articles about new features of Windows 2008 for more than a month. Most recent ones are about the new Terminal Server features, and they provide very detailed "behind the scenes" information which is not available in step-by-step guides. If you want to discover the technology details and get some fine tuning tips directly from Microsoft, check out this link:

http://blogs.technet.com/askperf/default.aspx

Differences between Windows 2003 and Windows 2008

Microsoft published an excellent document which highlights the major differences between the new and old version of Windows. It's quite a long read, but it's definitely valuable.

http://www.microsoft.com/downloads/details.aspx?familyid=173E6E9B-4D3E-4FD4-A2CF-73684FA46B60&displaylang=en

Windows Server 2008 and Vista SP1 RTM Today

Microsoft announced that Windows Server 2008 development is complete and it has been released to manufacturing. For those who are participants of the TAP or RDP programs, the code can be downloaded from Microsoft Connect Web site. The others have to wait a little bit more.

At the same time, Vista SP1 completed the RC phase and has been RTM'd. Unfortunately, there's no download link available yet. I will try to update this post when more information comes in.

For some major highlights about the releases you can check out this link:

http://blogs.zdnet.com/Ou/?p=988

Script to Audit Service Accounts

Although it is not a good practice to use domain Administrator as a service account, it is still very likely to find such implementations in many organizations. In addition, even if administrator is not used, the same service account can be in use on multiple servers and for multiple services. If there's no document or database available in your environment to keep track of these service accounts, you are in trouble. And if for some reason you need to change the password of these service accounts, you have to know which servers and which services will be affected with the password change.

The script addresses this need. It scans the given list of servers and tries to find a match for the provided keyword in the service accounts. Any matching service and computer name is logged and displayed on the screen. You can use the output file as a reference for updating the passwords. Script does not provide any password change functionality.

Usage:

cscript q_svc.vbs filename keyword

For example:

cscript q_svc.vbs serverlist.txt mydomain\adm

or

cscript q_svc.vbs serverlist.txt svcadmin@mydomain.local


Since the script uses string functions, for example, "mydomain\adm" search will catch both "mydomain\admin" and "mydomain\administrator" accounts if they exist.

You have to be careful and remember that service accounts sometimes are stored in UPN (username@domain) format. If you do a search for "mydomain\adm" only, you will miss the entries like administrator@mydomain.local. On the other hand, a search with the keyword "admin" will catch both type of entries.

To download the script and sample serverlist.txt file, click on "I agree", you will be forwarded to the download site. Modify and populate the serverlist.txt file with your server names, decide on the keyword to search for, then start the script. Test it and if you have any ideas or questions regarding this script, feel free to contact me using the comment link.

Update to the Post:
Somehow, the old free file hosting company decided that these utility scripts cause "abuse/copyright violations" and decided to remove them. Instead of trying to convince them that these scripts belong to me and has nothing to do with abuse, I simply moved them over to another (and better) file hosting service. Sorry for the inconvenience, and thanks for warning me about the dead links.

Another Update to the Post:

Thanks to Mark Vaughn, I realized that the script was not working as intended. I had to change the search logic a little bit. Instead of using queries with "like" on Windows 2003 systems, it checks each service for the given keyword now. I apologize for any inconvenience I might have caused with the previous version.

 

Windows Server 2008 RC1 is Available for Download

Check out this link:

http://www.microsoft.com/windowsserver2008/audsel.mspx

Exchange 2007 SP1 is ready...

At last, Exchange 2007 SP1 is available for download. Check out the following link:

http://www.microsoft.com/downloads/details.aspx?FamilyId=44C66AD6-F185-4A1D-A9AB-473C1188954C&displaylang=en

Interesting Behavior in Backup Exec 11d

If you have a combination of Microsoft Exchange Server and Symantec Backup Exec in your environment, you are probably aware of the new (approximately one year old) feature in Backup Exec 11d which eliminates the need for a seperate Mailbox-Level (or Brick Level) backups. With 11d, you can perform mail-level restores from your Information Store level backups. Very neat, isn't it? Well, if you can configure it properly, yes.

According to administrator's guide and web documentation, all you need is a dedicated mailbox for the Backup Exec Service Account if you are using tape devices as backup targets.
When you define your backup job, you should also select the "Enable the restore of individual mail messages and folders from Information Store backups" check box intuitively.

Interestingly, this configuration only seems to work for Exchange 2007. If you have Exchange 2003 in your environment and try to perform mail message restores from the Information Store backups, you will notice that you cannot even browse the list of mailboxes in the Restore Wizard screen. Solution to this problem lies in the general options screen. Go to Tools menu, select Options, then select Exchange under Job Defaults. You will see "Enable the restore of individual mail messages and folders from Information Store backups" option already selected. Additionally, don't be fooled by the misleading "not recommended" section, and select "Enable legacy mailbox support (not recommended - use information store instead)" check box.














Remember, you still do not want to do mailbox level backups. After the legacy mailbox support is enabled, test these new settings with a new Exchange Information Store backup. You will see that the individual mailboxes can be browsed and individual messages can be restored after these changes. As a final reminder, I have to add that this information applies to Backup Exec 11d Build 7170. Maybe Symantec will decide to change at least the "not recommended" section in the following releases, who knows?

Update:
My recent experiences with Backup Exec showed me that this behavior is not consistent among all Exchange 2003 and Backup Exec 11d deployments. So far I haven't been able to define the exact conditions to reproduce the problem. I have done two implementations with Exchange 2003 and Backup Exec 11d which required the settings above for mail level restores, and done one recent implementation which did not require any modifications and worked as described in the Admin Guide. I will try to update this post if I can find more information on this topic.