Exchange 2007 Multi Node Single Copy Cluster and Additional MTA Issue

Based on my experience , there is no need for addinational MTA on Exchange 2003 Multi Node Cluster. If you have an MTA resource on first Exchange 2003 node, you don’t need additional MTA resource at other Exchange Nodes.

But If you install multi-node single copy Exchange 2007 Cluster, you have to create additional MTA in Active Directory (by using AdsiEdit) for every additional Exchange 2007 nodes. If you don’t do that you cannot create a mailbox on these Exchange nodes.

You receive the following error in Exchange Management Console: “A proxy generator DLL on server FQDN.serverName could not be found or failed to initialize. Proxy addresses for the current recipient cannot be calculated. Please ensure that all proxy address generator DLLs have been installed on the target server”

Additionally, the following event may be logged:
Event Type: Error Event Source: MSExchangeIS Event Category: General Event ID: 1201 Description: Error -2147221233 reading/decrypting the msExchResponsibleMTAServer attribute on the Server object in the Active Directory. For more information, see Help and Support Center at Data: 0000: 5b 44 49 41 47 5f 43 54 [DIAG_CT 0008: 58 5d 00 00 26 00 00 00 X]..&... 0010: ff f0 0d 00 00 00 00 00 ÿð...... 0018: 00 02 18 00 00 00 cd 4a ......ÍJ 0020: 20 10 00 00 00 00 bd 5f .....½_ 0028: 20 10 00 00 00 00 bd 5f .....½_ 0030: 20 10 00 00 00 00 .....

Move Mailbox operations doesn’t care about additional MTA on every Exchange2007 Server on cluster. You can successfully move mailboxes to these Exchange servers. But these clients cannot logon via Outlook Web Access or MAPI (Outlook).

You can find additional information at http://support.microsoft.com/kb/928811 Microsoft support article.

802.1X NAP and Certificate Issue

At this posting , I want to talk about 802.1x NAP and Certificate related problems. According to my experience , you have to use a certificate from Enterprise CA at Network Policy Server’s certificate. StandAlone CA’s certificate is not working properly at NPS. Another issue about certificate is Client Machine certificate. According to my experience also, you have to use computer certificate and machine authentication at 802.1x NAP. If you don’t use computer certificate and also computer authentication on 802.1x NAP clients, there would be authentication failure at Pre-Logon stage. This problem causes cached logons on client computers, clients wouldn’t get either group policy or password expiration notifications.

Microsoft Office Communications Server 2007 using Direct SIP to Cisco Unified Communications Manager Simultaneous Ringer 7.0(1) Feature

Hi Everyone
This time , I want to talk about OCS 2007 and Cisco Call Manager Integration. Upon my OCS 2007 telephony experience , Cisco Call Manager has only direct integration with OCS . İf you want to remote call control , you need to Cisco Unified Presence Server. But Cisco Unified Communications Manager‘s mobility feature allows you simultaneous ringing (not Dual Forking) , cisco ip phones or Cisco IP Communicators and OCS MOC clients or Microsoft Phone Editions at the same time. You can find related howto solution at http://www.cisco.com/en/US/solutions/collateral/ns340/ns414/ns728/ns784/716742.pdf. Also you can test this feature at virtual test environment. Because Unified Call Manager 6.x or above can be intalled at WmwareWorkstation or ESX for testing purposes. If you use Dual forking and Remote Call Control you will have "in a call" status even when you are using the phone. It's only Nortel that supports Dual forking and Remote Call Control right now. Here is the Supported IP-PBXs for Microsoft Office Communications Server 2007 link :http://technet.microsoft.com/en-us/office/bb735838.aspx#ippbx2

Vista SP2 Beta and Windows 2008 SP2 Beta Available

Microsoft has released Windows 2008 SP2 Beta with some interesting changes. For me, most important news is that same installer can be used for both Windows 2008 SP2 and Vista SP2. As many of you already know, Windows 2008 was released with SP1 code included. Essentially Windows 2008 RTM and SP1 are the same code. That's why the first downloadable service pack for Windows 2008 is SP2.

If you want to test out the new SP2 Beta, check out the following link:

http://technet.microsoft.com/en-us/windowsserver/dd262148.aspx

You can also find the detailed information here:

http://technet.microsoft.com/en-us/library/dd351466.aspx

Useless hiberfil.sys file on Windows 2008

Hibernation is disabled by default when you first install Windows 2008. However, a hidden file called hiberfil.sys is created as big as the amount of physical memory on the system partition of the disk (not more than 4 GB). As using hibernation is not reasonable on production servers, you can delete this file and gain space by simply running "powercfg.exe /h off" command without quotations.
There is also a support article http://support.microsoft.com/kb/920730/en-us

Check State & Start Specific Service Script

The below script checks the state of the DHCP Server service and tries to start it if stopped.

Download Script

A User Deleted Emails by Mistake? No Need to Restore from Backup!

Before I start, I have to admit that the title of this post can be found little bit misleading. This statement is only true under certain circumstances. However, to my experience most Outlook users usually are quick to realize their own mistakes and they usually inform IT Department about accidental deletions within a week. So if your "Deleted Item Retention Period" value is set to a number larger than 7 (or still at default value of 14) days, and if the user comes with a restore request within this period, there really is no need for a restore from backup. Although this is feature has been around for a long time, I recently realized that it's not well known among most IT Generalists. 

Last week, for two occasions I was asked for assistance in restoring lost emails from backups. In the first case, one of my clients was trying to restore emails with Symantec Backup Exec 12. In the second case, another client needed step by step guidance with MS Data Protection Manager (DPM) 2007 to restore an accidentally deleted folder within "Inbox". Surprisingly, they both had users who accidentally deleted a bunch of emails and email folders from their Exchange Server mailboxes. For both cases, it took a little investigation to find out that the deletions took place within the last 7 days. So when I recommended them to use "Recover Deleted Items" feature in Outlook instead of tape restores, both of my clients were surprised. 

Both of these clients are sharp and intelligent IT Generalists. They manage mid-sized networks with a bunch of business applications along with their Exchange Servers. So Exchange Server management is not their only (or primary) responsibility. I think that most of the IT Generalists might be missing the fact that you can recover any deleted mail item within the limits of "Deleted Item Retention Period". So this post is for you, the IT Generalist who missed this powerful Exchange Server and Outlook feature. If you're an Exchange Server Admin (I mean, if Exchange Server is your primary responsibility), I assume you're already aware of this really neat feature. 

You can find plenty of online blog posts and articles on this topic. Microsoft KB Article 246153 explains the required registry key change for Outlook 2003 and older versions. MSExchange.org has an excellent article which describes all the basic information in detail along with OWA restore method. However, because these resources are relatively older, they fail to mention that Outlook 2007 does not require a registry trick to enable this feature. In other words, with Outlook 2007, you don't need to add the "DumpsterAlwaysOn" registry value to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Exchange\Client\Options key, because it's already there by default. For the Outlook versions prior to 2007, you have to add the registry value manually as described in MS KB Article, then restart Outlook. Alternatively, if you are still on Exchange 2003, you can use the OWA method described at MSExchange.org. However, Exchange 2007 OWA does not provide a similar functionality. You can only recover deleted items at the "Deleted Items" folder in OWA 2007 SP1.

With Recover Deleted Items feature, you can recover individual mail items, folders along with their contents, even if they were deleted using "Shift + Delete". If the deleted folder is one of the top level folders (like Inbox, Sent Items, Junk Email, etc.), you need to select the "Mailbox" at the Navigation Pane, then select Tools and Recover Deleted Items to reach the list of deleted folder(s). With this method you can also recover items that are deleted with "Mailbox Management" process. 

Finally, I want to mention about a great MSDN blog post by Doug Gowan, which discusses the necessity of backups in Exchange 2007 environments and outlines the scenarios when "Recover Deleted Items" feature can be used.

Limitations of Branch Distribution Points in SCCM

If you are going to implement the branch office scenario in SCCM and deploy branch distribution points on workstations, be sure to read the article "Clarifying: Retry Behavior for Distribution Points" which is not officially documented in TechNet and posted by SMS Writing Team.
It is about the 10 concurrent TCP connections limitation in client operating systems.

Install XP Guest on ESX Systems

As ESX does not support IDE disk controllers, it will be tricky to install an XP guest on the ESX host. You will either need LSI or Bus Logic SCSI drivers in FLP image format.
So it is a question where to find or how to create these floppy images.
Here is a link to download official BusLogic floppy image (also can be found in VmWare site but I could not find it easily in my second attempt...)

Download Floppy Image

Windows Firewall and SCOM Agent

When windows firewall is enabled by group policy on clients and you want to install SCOM agent using discovery in SCOM console, you get a message saying "RPC server is unavailable" at the end of discovery and agent install fails.

The below powershell script reads computer accounts from AgentList.txt and uses .net to stop windows firewall service on the remote host (as powershell does not support remote shell at the moment) and then uses discovery to install SCOM agent. After finishing, it starts the Windows Firewall service again.

Download PowerShell Script

Desired Configuration Management in Configuration Manager

Desired Configuration Management is a a new and useful fetaure in Configuration Manager. I tried to explain the basic steps to configure and interpret the reports of DCM.

Download Article

Script to Audit Disk Usage

This is a very basic script, which connects each server in a given list, then collects and reports the disk usage information for the fixed disks on each system. Script displays the drive letter, disk size, used and available disk space information on the screen and writes to a tab separated text file. I selected the tab separated file format for portability: You can easily copy and paste the contents to an Excel sheet. Script uses WMI to connect to the servers and collect the data, so please be aware that you will either need to shutdown Windows Firewall or add a "file and printer sharing exception" on the target systems.

I have tested the script on Windows XP, Windows Vista, Windows 2003 and Windows 2008. I hope the script will be very handy for easily creating disk usage reports on multiple servers and/or workstations. This script uses a very similar logic to my service account audit script. Only difference is in the WMI query section.

Usage:

cscript q_FixedDrv.vbs filename

For example:

cscript q_FixedDrv.vbs serverlist.txt

To download the script and sample serverlist.txt file, click on "I agree", you will be forwarded to the download site. Modify the serverlist.txt file to fit your test environment, then start the script. If the result you get is satisfactory, then create another list file which includes your production servers/workstations. If you have any feedback regarding this script, feel free to contact me using the comment link. I really want to know if you like it or hate it.

Exchange 2007 Migration - Common Issues

During my recent Exchange 2007 Migration Projects, I had to work on and produce solutions for a number of interesting problems.

To ease your pain and shorten the troubleshooting process associated with such issues, I will try to list some of the most common issues and their solutions in this post. All of the problems and solutions listed here can also be found in different discussion forums, blog sites or KB articles, but so far I haven't seen any consolidated document for common issues in Exchange 2007 migrations. I hope this post will fill in this gap.

1. Problem with Autodiscover (Outlook Web Services) on Windows 2008
2. Outlook Anywhere (RPC over HTTP) Does not Work on Windows 2008
3. ActiveSync (EAS) Issue with dedicated CAS and Exchange 2003 Back-end
4. Unable to Uninstall Legacy Exchange Server (2003)


1. Problem with Autodiscover (Outlook Web Services) on Windows 2008
Background:
When Outlook 2007 is used to connect to an Exchange 2007 server, by default Offline Address Book, Out of Office Assistant and Scheduling Assistant features try to connect to the Web based services. All these features actually depend on Autodiscover feature of the Exchange 2007 Client Access server (CAS).

Problem:
Outlook 2007 users detect problems with Offline Address Book, Out of Office Assistant and Scheduling Assistant. Especially, when trying to "Add Attendees" using Scheduling Assistant, Outlook crashes and restarts.

Solution:
The problem is caused by .NET 2.0 SP2. Uninstalling .NET 2.0 SP2 resolves the issue, all Autodiscover related features start to function properly.The details of the problem and recommended solution can be found at my previous post:

http://adventuresofanitpro.blogspot.com/2008/10/problem-with-autodiscover-on-exchange.html

2. Outlook Anywhere (RPC over HTTP) Fails on Windows 2008
Background:
For Outlook Anywhere feature to work properly, Client Access servers (CAS) need to access three TCP ports on the Mailbox Server: 6001, 6002 and 6004. Even in single server Exchange 2007 implementations, these three ports are needed.

Problem:
When you try to access these three ports through telnet tests, you will notice that 6001 and 6002 are accessible. However, you cannot access port 6004. If you check the ports on the Mailbox Server using netstat, you can verify that the server is listening on the all three ports. For some reason, because when IPv6 is enabled (and it's enabled by default on Windows 2008), it prevents access to TCP port 6004. There is one workaround and one permanent solution.

Solution:
For the quick workaround you can add the server's IP address and name in the local HOSTS file. Surprisingly this simple action re-enables access to port 6004.

For the permanent solution, you need to disable IPv6 for good. To disable IPv6, you need to perform the following steps:

In the Network Connections folder, obtain properties on all of your connections and adapters and clear the check box next to the Internet Protocol version 6 (TCP/IPv6) component in the list under This connection uses the following items. This method disables IPv6 on your LAN interfaces and connections, but does not disable IPv6 on tunnel interfaces or the IPv6 loopback interface.

Add the following registry value (DWORD type) set to 0xFFFFFFFF:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters\DisabledComponents.

This method disables IPv6 on all your LAN interfaces, connections, and tunnel interfaces but does not disable the IPv6 loopback interface. You must restart the computer for this registry value to take effect.

You will also need to replace the following line in HOSTS file

::1 localhost

with this line:

#::1 localhost

and add the following lines
IPv4 address hostname of the computer
IPv4 address FQDN of the computer

(Please replace IPv4 address with the IP address value of your server and names with the actual name of your server for the entries given above)
After these changes, you need to restart the server. Following the restart you will notice that you can connect via RPC/HTTP(s) with no problems.

MS Exchange Team Blog has an excellent post about this topic at the following link:

http://msexchangeteam.com/archive/2008/06/20/449053.aspx

Also Aaron Marks describes the problem and the solution in detail at his blog site:

http://blog.aaronmarks.com/?p=65


3. ActiveSync (EAS) Issue with dedicated CAS and Exchange 2003 Back-end
Background:
During a well planned transition to Exchange 2007 from legacy Exchange environment, one of the recommended actions is to install a CAS server (on a seperate box other than Mailbox server) and redirect all OWA, RPC/HTTP and ActiveSync requests to this new CAS server. Since CAS uses a process called proxying, and has the ability to proxy for both Exchange 2003 and Exchange 2007 mailboxes, all requests will be proxied to proper back-end (or mailbox) server. For This setup provides the best co-existence support with the legacy Exchange mailboxes and lets you seamlessly transition to Exchange 2007. You can find more information on this topic at the following MS Exchange Team Blog post:

http://msexchangeteam.com/archive/2007/09/04/446918.aspx

Problem:
For mailboxes located on Exchange 2003 back-end server, you have to enable Integrated Windows Authentication on /Microsoft-Server-ActiveSync virtual folder in the Exchange System Manager (ESM) interface. Otherwise, OWA and Activesync access fails. Now comes the tricky part: When you try to enable "Windows Integrated Authentication" for /Microsoft-Server-ActiveSync virtual directory, you will notice that you cannot click on "Authentication" button since it's dimmed by default. If you enable "Integrated Windows Authentication" on the virtual directory using IIS Manager interface, Activesync users will receive multiple username/password prompts and will fail to access their mailboxes. This is behavior is the result of DS to Metabase (DS2MB) process which turns off Integrated Windows Authentication on the Windows 2003 back-end server.

Solution:
Download and install the hotfix described in the following MS KB Article:

http://support.microsoft.com/kb/937031

As described in the KB article, this hotfix enables the dimmed button, and lets you select "Integrated Windows Authentication" in ESM. Modifying the setting in ESM makes it permanent, and Activesync users can seamlessly access their mailboxes regardless of their Exchange server version.

4. Unable to Uninstall Legacy Exchange Server (2003)
Background:
At the final stages of an Exchange 2003 to Exchange 2007 Migration, you need to uninstall Exchange 2003 server(s) as a cleanup process.

Problem:
When you try to uninstall the old Exchange server, you cannot select "Remove" option and receive the following message:

The component "Microsoft Exchange Messaging and Collaboration Services" cannot be assigned the action "Remove" because:

-One or more users currently use a mailbox store on this server. These users must be moved to a mailbox store on a different server or be mail disabled before uninstalling this server.

You double check the old Mailbox Store and make sure that no mailboxes are listed, the mailboxes were either moved to Exchange 2007 stores or removed from the organization. However Exchange Uninstall still creates the warning message and does not let you proceed with the "remove" operation.

The problem is actually not directly related to Exchange mailboxes. The uninstall program queries AD for the msExchHomeServer attribute and tries to find the name of the old server. If a match is found, simply produces the above error message and refuses to proceed.

Solution:
To resolve the problem, you need to perform the following steps:

1. Right-click the domain container, and then click Find
2. Select the Advanced tab
3. Select User from the Field button
4. From the list of attributes displayed, choose Exchange Home Server
5. Set the Condition field to Ends With
6. Enter the Exchange server name into the Value field (Exchange 2003 server name)
7. Click Add
8. Click the Find
9. In the results Window, select the listed user account
10. Right-click on the user account name, then select "Exchange Tasks"
11. Click Next on the first screen
12. Select "Remove Exchange Attributes" on the second screen click Next and continue with the wizard till the end
13. Repeat the steps 9 to 12 for each user in the results list

After this cleanup operation, you will be able to select "Remove" during the Exchange Setup, and will be able to proceed with the uninstall with no problems.

Following Microsoft KB Article discusses the same topic with a slightly different resolution:

http://support.microsoft.com/kb/924170

Oz Ozugurlu also describes the problem and resolution at his blog site:

http://smtp25.blogspot.com/2007/04/having-trouble-to-uninstall-exchange.html

Problem with Autodiscover on Exchange 2007 SP1

I have run into this problem with an Exchange Server 2007 SP1 implementation running on Windows Server 2008 x64. I also noticed some forum posts where people had the exact same issue on Windows Server 2003 x64, however I cannot verify the solution method described here would help if you're on Windows Server 2003 x64. 

Although the main problem is with the Autodiscover web service, you may experience crashes in Outlook 2007 with Scheduling Assistant or errors with Out of Office Asisstant (OOF). Also Offline Addressbook (OAB) downloads may not work properly. When used with Outlook 2007, all these features depend on Autodiscover, so they fail when Autodiscover has a problem. In my case, users noticed that they were not able to "Add Attendees" when trying to create a meeting request with Scheduling Assistant. Outlook 2007 was crashing when they selected an attendee from the addressbook. After a brief research, I also noticed that Out of Office Assitant was also failing with "Your Out of Office settings cannot be displayed, because the server is currently unavailable" message. My attempts to download Offline Address Book were also unsuccessful. Download attempts did not succeed, but they ran forever and did not produce an error message either.

I will not list all the boring troubleshooting details here, but I want to mention that I deleted and recreated Autodiscover virtual directory with "Remove-AutodiscoverVirtualDirectory" and "New-AutodiscoverVirtualDirectory" commandlets in Powershell. These commands ran without any issues, but they did not help. 

Also I ran "Test-OutlookWebServices" commandlet on Exchange Server and received the following error at the fourth step:

PS] C:\Windows\System32>Test-OutlookWebServices |fl

Id      : 1003

Type    : Information

Message : About to test AutoDiscover with the e-mail address Administrator@xyz.com.

Id      : 1007

Type    : Information

Message : Testing server EX01.xyz.com with the published name https://EX01.xyz.com/EWS/Exchange.asmx & https://mail.xyz.com/EWS/Exchange.asmx.

Id      : 1019

Type    : Information

Message : Found a valid AutoDiscover service connection point. The AutoDiscover URL on this object is https://EX01.xyz.com/autodiscover/autodiscover.xml.

Id      : 1006

Type    : Information

Message : The Autodiscover service was contacted at https://EX01.xyz.com/autodiscover/autodiscover.xml.

WARNING: An unexpected error has occurred and debug information is being generated: Object reference not set to an instance of an object.

Test-OutlookWebServices : Object reference not set to an instance of an object.

At line:1 char:24 

+ Test-OutlookWebServices  <<<< |fl

Because Outlook "Test E-mail AutoConfiguration" ended with a "certificate error" I also re-created the SSL certificate and enabled it for all services. No need to say, none of these actions resolved the problem. 

I was able to find the actual problem source and solution at Technet Forums. (At http://forums.microsoft.com/TechNet/ShowPost.aspx?PageIndex=0&SiteID=17&PageID=0&PostID=3760137, to be exact.) Apparently, the problem was caused by a bug in ".NET Framework 2.0 SP2". It simply was not playing nicely with Exchange Server 2007 SP1. Even though you don't explicitly select to install this .NET 2.0 Service Pack 2, it is installed as a part of ".NET Framework 3.5 SP1" setup. With this information, I searched for .NET Framework 2.0 SP2 in Programs and Features, but failed to find it. So I uninstalled .NET Framework 3.5 SP1 instead and restarted the server. However this did not help. Finally, I found that the .NET 2.0 SP2 was displayed as "Update for Microsoft Windows (KB948609)" in Windows 2008 Programs and Features. After I had uninstalled this update, all of my problems disapperared. I was able to verify that Out of Office Assistant, Scheduling Assistant and Offline Address Book downloads were fully functional. In addition, all error messages disappeared from "Test-OutlookWebServices" test in Powershell and "Test E-mail AutoConfiguration" test in Outlook. 

I had to spend long hours searching the web and testing every possible solution to find the actual one. Hope this post will reduce your troubleshooting period. Please use the Comments link below if want to say something about this post.

Service Fails to Start After Reboot - Port Conflicts

For the last couple of months, I have experienced high number of unexpected service failures. Randomly, one of the services fails to start successfully after a reboot. Usually, the failed service belongs to a recently installed application. 100% of these incidents were caused by conflicting TCP or UDP ports. Because the port number is already in use, the service fails to start.

In one of the past incidents it was Blackberry Router service, today it was Internet Authentication Server (IAS) service. All of the port conflicts I have seen were on Windows 2003 Domain Controller servers or Windows 2003 SBS. Since the troubleshooting methodology is the same, I will try to describe how to determine the application causing the port conflict and how to resolve the problem.

First of all, you have to know the port numbers and port type of the failed service. For example, Blackberry Router service works on TCP 3101, IAS works on UDP 1645-1646 and UDP 1812-1813. To find the specific port information for a specific service, easiest way is to perform a Google search. If we want to find the port information about Blackberry Router Service, typing “blackberry router service port” in the Google search bar will easily yield to port 3101.

As a second step, you need to verify the port conflict (i.e. verify that the port is already in use). You can achieve this by simply running the following command:

Netstat –an |findstr :portnumber

You need to replace the portnumber with the port number of the failed service. If the failed service uses multiple ports, you have to repeat the command for each port. For example, for Blackberry Router Service, use the following command:

Netstat –an |findstr :3101

If the port is not in use, above command will not return any results. This also means that your root cause is not a port conflict. Unfortunately, this post will not provide any assistance for your problem.

If the port is already in use, you should see a result similar to the line below.

TCP 0.0.0.0:3101 0.0.0.0:0 LISTENING

To verify the port conflict, you can alternatively search Windows Application Log for errors, or look at text based log files for your failed service.

After verifying that the port is in use, the next step would be to determine the application/process using this port. For this purpose, the tool we need to use is TCPView from Sysinternals. You can download the tool at http://technet.microsoft.com/en-us/sysinternals/bb897437.aspx.

The compressed download includes two executables: TCPView.exe – the GUI version, TCPvcon.exe – command line version. I strongly recommend using the command line version because it’s faster. Simply extract the file TCPvcon.exe to a folder, open command prompt, navigate to the folder, and run the command with the following syntax:

TCPvcon –a >tcpview.txt

Now, you can open tcpview.txt with your favorite text editor. Then, search the file contents for the conflicting port number(s). When you find the matching port number, you will also see the application using that port.

In my experience, most common applications causing the conflict are LSASS.EXE and DNS.EXE.

LSASS.EXE
Local Security Authority Subsystem Service (LSASS), is a process in Microsoft Windows operating systems that is responsible for enforcing the security policy on the system. In addition, it is responsible for the following components:
• Local Security Authority
• Net Logon service
• Security Accounts Manager service
• LSA Server service
• Secure Sockets Layer (SSL)
• Kerberos v5 authentication protocol
• NTLM authentication protocol

By default, LSASS.EXE uses random TCP ports ranging from 1024 to 65535. To restrict the port usage of this process, you need to follow the instructions in the MS KB article below:

http://support.microsoft.com/kb/224196

After selecting a dedicated non-conflicting port for LSASS and applying the registry changes in the article, you need to restart the server.

DNS.EXE
In the past, DNS server was designed to listen on TCP port 53 and UDP port 53. However, this behavior has been changed by a recent security patch. If you applied security update 953230 on your DNS server, you will find that it allocates 2500 UDP ports by default. This is the root cause of the most port conflicts. For general information about this security patch and the side effects, you need to read the following article:

http://support.microsoft.com/kb/953230

Although the problem has multiple workarounds, if you want to restore the service functionality and resolve the port conflict while avoiding opening a gap in the security, you need to define reserved ports in the registry. For SBS 2003 version of the detailed resolution you need to check out the following article:

http://support.microsoft.com/kb/956189

For a general Windows 2000/2003 servers, the resolution information can be found at:

http://support.microsoft.com/kb/812873

As described in the articles, after the registry changes, a reboot will be required.

The Official SBS Blog also has a great entry where they explain some additional problems caused by DNS and their recommended solutions in their site. For me, most important one is experienced at IPSEC level, where IPSEC service fails and IPSEC driver enters "block mode". This is a very common issue and can only be avoided by adding "ReservedPorts" entry for IPSEC. Please read the following post for details.

http://blogs.technet.com/sbs/archive/2008/07/17/some-services-may-fail-to-start-or-may-not-work-properly-after-installing-ms08-037-951746-and-951748.aspx

I tried to explain some common root causes of the IP port conflicts and the methodology to resolve these problems. I hope this post will ease the pain of troubleshooting similar problems. I also would like to hear about other conflicting applications and your resolutions. Please use the comments link if you want to share your story about a conflicting port.

VMware Converter Fails at 99%

Last week, I was migrating some physical servers to VMware Server images with VMware Converter tool. I was using the latest and improved version of VMware Converter (v3.0.3 - Build 89816). I had two servers to migrate: one SBS 2003 server and one Windows 2003 based Terminal Server. I had to run the converter on a third piece of hardware while the servers were up and running, installing the converter agent service on them (AKA "hot clone"). SBS server had two volumes, and the total used disk space was around 40GB. Terminal Server had two volumes and used disk space was around 25GB. 

First P2V migration session ran approximately for 6 hours on the SBS server and completed without any problems. I started the second session on the Terminal Server and checked the status every hour. 

If you are familiar with the P2V process, you probably know that the tool initially creates the vmdk files, then fills them in with the files from the source machine (Cloning Phase). When all files are transferred to the virtual disks (vmdk files), then final step is called "conversion". At this stage, the storage, HAL and network specific driver files and their configuration gets modified. This step is necessary, otherwise the virtual disk will not be able to boot. 

Anyhow, after running for 3 hours and 30 minutes, my Terminal Server P2V migration session failed at 99% with "Unknown Error"! A quick Google search revealed that the 97% and 99% range belongs to Customization/Reconfig phase, meaning that it failed at the "conversion" step. VMware converter has the ability to manually run the "conversion". I selected the newly created virtual machine as the source, and ran the "Conversion Wizard", but the process failed again. This time, I decided to examine the log files to see what's wrong. Converter creates log files in two locations: Converter Console log files reside in user's profile folder (in my Vista laptop, they are located at: "%userprofile%\AppData\Local\Temp"). Converter Agent log files reside in "Windows\Temp" folder. After a quick review, I realized that the log files in the profile folder did not provide the detail level I needed. Then after going through the log files in "Windows\Temp" (vmware-converter-n.log) I found what I was looking for:

[#2] [2008-08-23 07:33:53.707 'App' 3932 verbose] [registryUpdate,73] Applying registry update S_SYMMPI
[#2] [2008-08-23 07:33:53.708 'App' 3932 verbose] [registryHiveProxy,176] Saving registry key ControlSet001\Services\symmpi to \\.\vstor2-p2v30-B4BDB4BD007E00000000000003000000\WINDOWS\$Reconfig$\ControlSet001-Services-symmpi-reg
[#2] [2008-08-23 07:33:54.174 'App' 3932 verbose] [registryHiveProxy,152] Restoring registry key ControlSet001\Services\symmpi from C:\Program Files\VMware\VMware Converter\data\52-S_SYMMPI
[#2] [2008-08-23 07:33:54.202 'App' 3932 verbose] [fileUpdate,69] Update File system32\drivers\symmpi.sys
[#2] [2008-08-23 07:33:54.202 'App' 3932 verbose] [volumeProxy,95] Checking if WINDOWS\system32\drivers\symmpi.sys exists...
[#2] [2008-08-23 07:33:54.205 'App' 3932 verbose] [volumeProxy,102] WINDOWS\system32\drivers\symmpi.sys does not exist!
[#2] [2008-08-23 07:33:54.223 'App' 3932 error] [fileUpdate,120] Unable to find symmpi.sys in the specified CAB files
[#2] [2008-08-23 07:33:54.250 'App' 3932 info] [imageProcessingTaskStep,194] UfaSysReconfig::task{1}::task{2} step "applying reconfigurations" destroyed

The Converter was failing at a specific file, symmpi.sys! Because it was looking for the file in "WINDOWS\system32\drivers" folder, failing to find it there, then aborting the process. Another Google search revealed that this was the LSI Logic Miniport Driver file. Luckily, I found the same file in one of my other virtual machines, in the "WINDOWS\system32\drivers" folder. To copy the file to my failing virtual Terminal Server, I connected its C:\ drive as an additional disk to my old virtual machine. After this, I could easily add a drive letter to the drive and was able copy the file.

I shutdown my old virtual machine, removed the drive connection. Then I ran the "conversion Wizard" one more time selecting the new virtual Terminal Server as the source. The "Conversion Wizard" only ran for a couple of minutes, then completed successfully. After this step, I was able to verify that I could successfully boot my new virtual Terminal Server.

My Google searches during this troubleshooting period showed me that this is a very common error. I spent about 2 hours and 30 minutes to find the solution and I hope this post can speed up your troubleshooting process. Please use the comments link below if you have any questions.

Backup Exec 11d and Windows 2008 Compatibility

From the referrals of this blog site, I can see that many users are trying to find out if Backup Exec 11d supports Windows Server 2008. Unfortunately, Backup Exec 11d does not officially support Windows Server 2008. If you want to backup Windows Server 2008, you will need Backup Exec 12. If you purchased 11d recently (during the past year), you may also be eligible for free version 12 upgrade. Hope this post provides the clarification you have been looking for.

MS Virtual Server to VMware ESX Migration Issue

Last week, I was trying to migrate some virtual guest servers from MS Virtual Server 2005 environment to VMware ESX. I had to run the "import" task three times to achieve a successful migration, and had to spend 4-5 hours for a simple migration operation. Hope the tips here will save you some time.
At my first attempt, the process seemed to halt at 22%. There was no activity in the Virtual Center and there were no log file entries at the VMware side. I examined the event viewer logs of the Virtual Server 2005 guest machine and found that the following error was logged:
Log Name: System
Source: msvmscsi
Date: 5/15/2008 5:14:45 PM
Event ID: 9
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: xxx-xxx.xxx.xxx
Description:The device, \Device\Scsi\msvmscsi1, did not respond within the timeout period.

A quick Google search directed me to this link:
http://www.eventid.net/display.asp?eventid=9&eventno=9355&source=vmscsi&phase=1
Before my second attempt I created the following value as described in the above link, and rebooted the guest server:
Key: HKEY_LOCAL_MACHINE/System/CurrentControlSet/Services/Disk
Sub Key: TimeOutValue
Data Type: REG_DWORD
Value: 03c hex (or 60 decimal)

The import process ran up to 97% in my second attempt, then halted there. Same error messages were displayed in the event log of the guest machine.
Before the third attempt, I decided to increase the same registry key value as follows and rebooted the guest server:
Key: HKEY_LOCAL_MACHINE/System/CurrentControlSet/Services/Disk
Sub Key: TimeOutValue
Data Type: REG_DWORD
Value: 05a hex (or 90 decimal)
After this final change and reboot, third import attempt ran successfully. As a final step, I removed the registry setting at the ESX guest machine since it is no longer needed.

Windows 2008 - Detailed Technical Articles

Windows Server Performance Team Blog, has been publushing very nice articles about new features of Windows 2008 for more than a month. Most recent ones are about the new Terminal Server features, and they provide very detailed "behind the scenes" information which is not available in step-by-step guides. If you want to discover the technology details and get some fine tuning tips directly from Microsoft, check out this link:

http://blogs.technet.com/askperf/default.aspx

Differences between Windows 2003 and Windows 2008

Microsoft published an excellent document which highlights the major differences between the new and old version of Windows. It's quite a long read, but it's definitely valuable.

http://www.microsoft.com/downloads/details.aspx?familyid=173E6E9B-4D3E-4FD4-A2CF-73684FA46B60&displaylang=en

Windows Server 2008 and Vista SP1 RTM Today

Microsoft announced that Windows Server 2008 development is complete and it has been released to manufacturing. For those who are participants of the TAP or RDP programs, the code can be downloaded from Microsoft Connect Web site. The others have to wait a little bit more.

At the same time, Vista SP1 completed the RC phase and has been RTM'd. Unfortunately, there's no download link available yet. I will try to update this post when more information comes in.

For some major highlights about the releases you can check out this link:

http://blogs.zdnet.com/Ou/?p=988