The below script checks the state of the DHCP Server service and tries to start it if stopped.
Monday, December 1, 2008
Sunday, November 30, 2008
A User Deleted Emails by Mistake? No Need to Restore from Backup!
Before I start, I have to admit that the title of this post can be found little bit misleading. This statement is only true under certain circumstances. However, to my experience most Outlook users usually are quick to realize their own mistakes and they usually inform IT Department about accidental deletions within a week. So if your "Deleted Item Retention Period" value is set to a number larger than 7 (or still at default value of 14) days, and if the user comes with a restore request within this period, there really is no need for a restore from backup. Although this is feature has been around for a long time, I recently realized that it's not well known among most IT Generalists.
Last week, for two occasions I was asked for assistance in restoring lost emails from backups. In the first case, one of my clients was trying to restore emails with Symantec Backup Exec 12. In the second case, another client needed step by step guidance with MS Data Protection Manager (DPM) 2007 to restore an accidentally deleted folder within "Inbox". Surprisingly, they both had users who accidentally deleted a bunch of emails and email folders from their Exchange Server mailboxes. For both cases, it took a little investigation to find out that the deletions took place within the last 7 days. So when I recommended them to use "Recover Deleted Items" feature in Outlook instead of tape restores, both of my clients were surprised.
Both of these clients are sharp and intelligent IT Generalists. They manage mid-sized networks with a bunch of business applications along with their Exchange Servers. So Exchange Server management is not their only (or primary) responsibility. I think that most of the IT Generalists might be missing the fact that you can recover any deleted mail item within the limits of "Deleted Item Retention Period". So this post is for you, the IT Generalist who missed this powerful Exchange Server and Outlook feature. If you're an Exchange Server Admin (I mean, if Exchange Server is your primary responsibility), I assume you're already aware of this really neat feature.
You can find plenty of online blog posts and articles on this topic. Microsoft KB Article 246153 explains the required registry key change for Outlook 2003 and older versions. MSExchange.org has an excellent article which describes all the basic information in detail along with OWA restore method. However, because these resources are relatively older, they fail to mention that Outlook 2007 does not require a registry trick to enable this feature. In other words, with Outlook 2007, you don't need to add the "DumpsterAlwaysOn" registry value to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Exchange\Client\Options key, because it's already there by default. For the Outlook versions prior to 2007, you have to add the registry value manually as described in MS KB Article, then restart Outlook. Alternatively, if you are still on Exchange 2003, you can use the OWA method described at MSExchange.org. However, Exchange 2007 OWA does not provide a similar functionality. You can only recover deleted items at the "Deleted Items" folder in OWA 2007 SP1.
With Recover Deleted Items feature, you can recover individual mail items, folders along with their contents, even if they were deleted using "Shift + Delete". If the deleted folder is one of the top level folders (like Inbox, Sent Items, Junk Email, etc.), you need to select the "Mailbox" at the Navigation Pane, then select Tools and Recover Deleted Items to reach the list of deleted folder(s). With this method you can also recover items that are deleted with "Mailbox Management" process.
Finally, I want to mention about a great MSDN blog post by Doug Gowan, which discusses the necessity of backups in Exchange 2007 environments and outlines the scenarios when "Recover Deleted Items" feature can be used.
Friday, November 28, 2008
Limitations of Branch Distribution Points in SCCM
If you are going to implement the branch office scenario in SCCM and deploy branch distribution points on workstations, be sure to read the article "Clarifying: Retry Behavior for Distribution Points" which is not officially documented in TechNet and posted by SMS Writing Team.
It is about the 10 concurrent TCP connections limitation in client operating systems.
It is about the 10 concurrent TCP connections limitation in client operating systems.
Install XP Guest on ESX Systems
As ESX does not support IDE disk controllers, it will be tricky to install an XP guest on the ESX host. You will either need LSI or Bus Logic SCSI drivers in FLP image format.
So it is a question where to find or how to create these floppy images.
Here is a link to download official BusLogic floppy image (also can be found in VmWare site but I could not find it easily in my second attempt...)
So it is a question where to find or how to create these floppy images.
Here is a link to download official BusLogic floppy image (also can be found in VmWare site but I could not find it easily in my second attempt...)
Thursday, November 27, 2008
Windows Firewall and SCOM Agent
When windows firewall is enabled by group policy on clients and you want to install SCOM agent using discovery in SCOM console, you get a message saying "RPC server is unavailable" at the end of discovery and agent install fails.
The below powershell script reads computer accounts from AgentList.txt and uses .net to stop windows firewall service on the remote host (as powershell does not support remote shell at the moment) and then uses discovery to install SCOM agent. After finishing, it starts the Windows Firewall service again.
The below powershell script reads computer accounts from AgentList.txt and uses .net to stop windows firewall service on the remote host (as powershell does not support remote shell at the moment) and then uses discovery to install SCOM agent. After finishing, it starts the Windows Firewall service again.
Desired Configuration Management in Configuration Manager
Desired Configuration Management is a a new and useful fetaure in Configuration Manager. I tried to explain the basic steps to configure and interpret the reports of DCM.
Friday, November 21, 2008
Script to Audit Disk Usage
This is a very basic script, which connects each server in a given list, then collects and reports the disk usage information for the fixed disks on each system. Script displays the drive letter, disk size, used and available disk space information on the screen and writes to a tab separated text file. I selected the tab separated file format for portability: You can easily copy and paste the contents to an Excel sheet. Script uses WMI to connect to the servers and collect the data, so please be aware that you will either need to shutdown Windows Firewall or add a "file and printer sharing exception" on the target systems.
I have tested the script on Windows XP, Windows Vista, Windows 2003 and Windows 2008. I hope the script will be very handy for easily creating disk usage reports on multiple servers and/or workstations. This script uses a very similar logic to my service account audit script. Only difference is in the WMI query section.
Usage:
cscript q_FixedDrv.vbs filename
For example:
cscript q_FixedDrv.vbs serverlist.txt
cscript q_FixedDrv.vbs filename
For example:
cscript q_FixedDrv.vbs serverlist.txt
To download the script and sample serverlist.txt file, click on "I agree", you will be forwarded to the download site. Modify the serverlist.txt file to fit your test environment, then start the script. If the result you get is satisfactory, then create another list file which includes your production servers/workstations. If you have any feedback regarding this script, feel free to contact me using the comment link. I really want to know if you like it or hate it.
Saturday, November 1, 2008
Exchange 2007 Migration - Common Issues
During my recent Exchange 2007 Migration Projects, I had to work on and produce solutions for a number of interesting problems.
To ease your pain and shorten the troubleshooting process associated with such issues, I will try to list some of the most common issues and their solutions in this post. All of the problems and solutions listed here can also be found in different discussion forums, blog sites or KB articles, but so far I haven't seen any consolidated document for common issues in Exchange 2007 migrations. I hope this post will fill in this gap.
1. Problem with Autodiscover (Outlook Web Services) on Windows 2008
2. Outlook Anywhere (RPC over HTTP) Does not Work on Windows 2008
3. ActiveSync (EAS) Issue with dedicated CAS and Exchange 2003 Back-end
4. Unable to Uninstall Legacy Exchange Server (2003)
1. Problem with Autodiscover (Outlook Web Services) on Windows 2008
Background:
When Outlook 2007 is used to connect to an Exchange 2007 server, by default Offline Address Book, Out of Office Assistant and Scheduling Assistant features try to connect to the Web based services. All these features actually depend on Autodiscover feature of the Exchange 2007 Client Access server (CAS).
Problem:
Outlook 2007 users detect problems with Offline Address Book, Out of Office Assistant and Scheduling Assistant. Especially, when trying to "Add Attendees" using Scheduling Assistant, Outlook crashes and restarts.
Solution:
The problem is caused by .NET 2.0 SP2. Uninstalling .NET 2.0 SP2 resolves the issue, all Autodiscover related features start to function properly.The details of the problem and recommended solution can be found at my previous post:
http://adventuresofanitpro.blogspot.com/2008/10/problem-with-autodiscover-on-exchange.html
2. Outlook Anywhere (RPC over HTTP) Fails on Windows 2008
Background:
For Outlook Anywhere feature to work properly, Client Access servers (CAS) need to access three TCP ports on the Mailbox Server: 6001, 6002 and 6004. Even in single server Exchange 2007 implementations, these three ports are needed.
Problem:
When you try to access these three ports through telnet tests, you will notice that 6001 and 6002 are accessible. However, you cannot access port 6004. If you check the ports on the Mailbox Server using netstat, you can verify that the server is listening on the all three ports. For some reason, because when IPv6 is enabled (and it's enabled by default on Windows 2008), it prevents access to TCP port 6004. There is one workaround and one permanent solution.
Solution:
For the quick workaround you can add the server's IP address and name in the local HOSTS file. Surprisingly this simple action re-enables access to port 6004.
For the permanent solution, you need to disable IPv6 for good. To disable IPv6, you need to perform the following steps:
In the Network Connections folder, obtain properties on all of your connections and adapters and clear the check box next to the Internet Protocol version 6 (TCP/IPv6) component in the list under This connection uses the following items. This method disables IPv6 on your LAN interfaces and connections, but does not disable IPv6 on tunnel interfaces or the IPv6 loopback interface.
Add the following registry value (DWORD type) set to 0xFFFFFFFF:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters\DisabledComponents.
This method disables IPv6 on all your LAN interfaces, connections, and tunnel interfaces but does not disable the IPv6 loopback interface. You must restart the computer for this registry value to take effect.
You will also need to replace the following line in HOSTS file
::1 localhost
with this line:
#::1 localhost
and add the following lines
IPv4 address hostname of the computer
IPv4 address FQDN of the computer
(Please replace IPv4 address with the IP address value of your server and names with the actual name of your server for the entries given above)
After these changes, you need to restart the server. Following the restart you will notice that you can connect via RPC/HTTP(s) with no problems.
MS Exchange Team Blog has an excellent post about this topic at the following link:
http://msexchangeteam.com/archive/2008/06/20/449053.aspx
Also Aaron Marks describes the problem and the solution in detail at his blog site:
http://blog.aaronmarks.com/?p=65
3. ActiveSync (EAS) Issue with dedicated CAS and Exchange 2003 Back-end
Background:
During a well planned transition to Exchange 2007 from legacy Exchange environment, one of the recommended actions is to install a CAS server (on a seperate box other than Mailbox server) and redirect all OWA, RPC/HTTP and ActiveSync requests to this new CAS server. Since CAS uses a process called proxying, and has the ability to proxy for both Exchange 2003 and Exchange 2007 mailboxes, all requests will be proxied to proper back-end (or mailbox) server. For This setup provides the best co-existence support with the legacy Exchange mailboxes and lets you seamlessly transition to Exchange 2007. You can find more information on this topic at the following MS Exchange Team Blog post:
http://msexchangeteam.com/archive/2007/09/04/446918.aspx
Problem:
For mailboxes located on Exchange 2003 back-end server, you have to enable Integrated Windows Authentication on /Microsoft-Server-ActiveSync virtual folder in the Exchange System Manager (ESM) interface. Otherwise, OWA and Activesync access fails. Now comes the tricky part: When you try to enable "Windows Integrated Authentication" for /Microsoft-Server-ActiveSync virtual directory, you will notice that you cannot click on "Authentication" button since it's dimmed by default. If you enable "Integrated Windows Authentication" on the virtual directory using IIS Manager interface, Activesync users will receive multiple username/password prompts and will fail to access their mailboxes. This is behavior is the result of DS to Metabase (DS2MB) process which turns off Integrated Windows Authentication on the Windows 2003 back-end server.
Solution:
Download and install the hotfix described in the following MS KB Article:
http://support.microsoft.com/kb/937031
As described in the KB article, this hotfix enables the dimmed button, and lets you select "Integrated Windows Authentication" in ESM. Modifying the setting in ESM makes it permanent, and Activesync users can seamlessly access their mailboxes regardless of their Exchange server version.
4. Unable to Uninstall Legacy Exchange Server (2003)
Background:
At the final stages of an Exchange 2003 to Exchange 2007 Migration, you need to uninstall Exchange 2003 server(s) as a cleanup process.
Problem:
When you try to uninstall the old Exchange server, you cannot select "Remove" option and receive the following message:
The component "Microsoft Exchange Messaging and Collaboration Services" cannot be assigned the action "Remove" because:
-One or more users currently use a mailbox store on this server. These users must be moved to a mailbox store on a different server or be mail disabled before uninstalling this server.
You double check the old Mailbox Store and make sure that no mailboxes are listed, the mailboxes were either moved to Exchange 2007 stores or removed from the organization. However Exchange Uninstall still creates the warning message and does not let you proceed with the "remove" operation.
The problem is actually not directly related to Exchange mailboxes. The uninstall program queries AD for the msExchHomeServer attribute and tries to find the name of the old server. If a match is found, simply produces the above error message and refuses to proceed.
Solution:
To resolve the problem, you need to perform the following steps:
Following Microsoft KB Article discusses the same topic with a slightly different resolution:
http://support.microsoft.com/kb/924170
Oz Ozugurlu also describes the problem and resolution at his blog site:
http://smtp25.blogspot.com/2007/04/having-trouble-to-uninstall-exchange.html
To ease your pain and shorten the troubleshooting process associated with such issues, I will try to list some of the most common issues and their solutions in this post. All of the problems and solutions listed here can also be found in different discussion forums, blog sites or KB articles, but so far I haven't seen any consolidated document for common issues in Exchange 2007 migrations. I hope this post will fill in this gap.
1. Problem with Autodiscover (Outlook Web Services) on Windows 2008
2. Outlook Anywhere (RPC over HTTP) Does not Work on Windows 2008
3. ActiveSync (EAS) Issue with dedicated CAS and Exchange 2003 Back-end
4. Unable to Uninstall Legacy Exchange Server (2003)
1. Problem with Autodiscover (Outlook Web Services) on Windows 2008
Background:
When Outlook 2007 is used to connect to an Exchange 2007 server, by default Offline Address Book, Out of Office Assistant and Scheduling Assistant features try to connect to the Web based services. All these features actually depend on Autodiscover feature of the Exchange 2007 Client Access server (CAS).
Problem:
Outlook 2007 users detect problems with Offline Address Book, Out of Office Assistant and Scheduling Assistant. Especially, when trying to "Add Attendees" using Scheduling Assistant, Outlook crashes and restarts.
Solution:
The problem is caused by .NET 2.0 SP2. Uninstalling .NET 2.0 SP2 resolves the issue, all Autodiscover related features start to function properly.The details of the problem and recommended solution can be found at my previous post:
http://adventuresofanitpro.blogspot.com/2008/10/problem-with-autodiscover-on-exchange.html
2. Outlook Anywhere (RPC over HTTP) Fails on Windows 2008
Background:
For Outlook Anywhere feature to work properly, Client Access servers (CAS) need to access three TCP ports on the Mailbox Server: 6001, 6002 and 6004. Even in single server Exchange 2007 implementations, these three ports are needed.
Problem:
When you try to access these three ports through telnet tests, you will notice that 6001 and 6002 are accessible. However, you cannot access port 6004. If you check the ports on the Mailbox Server using netstat, you can verify that the server is listening on the all three ports. For some reason, because when IPv6 is enabled (and it's enabled by default on Windows 2008), it prevents access to TCP port 6004. There is one workaround and one permanent solution.
Solution:
For the quick workaround you can add the server's IP address and name in the local HOSTS file. Surprisingly this simple action re-enables access to port 6004.
For the permanent solution, you need to disable IPv6 for good. To disable IPv6, you need to perform the following steps:
In the Network Connections folder, obtain properties on all of your connections and adapters and clear the check box next to the Internet Protocol version 6 (TCP/IPv6) component in the list under This connection uses the following items. This method disables IPv6 on your LAN interfaces and connections, but does not disable IPv6 on tunnel interfaces or the IPv6 loopback interface.
Add the following registry value (DWORD type) set to 0xFFFFFFFF:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters\DisabledComponents.
This method disables IPv6 on all your LAN interfaces, connections, and tunnel interfaces but does not disable the IPv6 loopback interface. You must restart the computer for this registry value to take effect.
You will also need to replace the following line in HOSTS file
::1 localhost
with this line:
#::1 localhost
and add the following lines
IPv4 address hostname of the computer
IPv4 address FQDN of the computer
(Please replace IPv4 address with the IP address value of your server and names with the actual name of your server for the entries given above)
After these changes, you need to restart the server. Following the restart you will notice that you can connect via RPC/HTTP(s) with no problems.
MS Exchange Team Blog has an excellent post about this topic at the following link:
http://msexchangeteam.com/archive/2008/06/20/449053.aspx
Also Aaron Marks describes the problem and the solution in detail at his blog site:
http://blog.aaronmarks.com/?p=65
3. ActiveSync (EAS) Issue with dedicated CAS and Exchange 2003 Back-end
Background:
During a well planned transition to Exchange 2007 from legacy Exchange environment, one of the recommended actions is to install a CAS server (on a seperate box other than Mailbox server) and redirect all OWA, RPC/HTTP and ActiveSync requests to this new CAS server. Since CAS uses a process called proxying, and has the ability to proxy for both Exchange 2003 and Exchange 2007 mailboxes, all requests will be proxied to proper back-end (or mailbox) server. For This setup provides the best co-existence support with the legacy Exchange mailboxes and lets you seamlessly transition to Exchange 2007. You can find more information on this topic at the following MS Exchange Team Blog post:
http://msexchangeteam.com/archive/2007/09/04/446918.aspx
Problem:
For mailboxes located on Exchange 2003 back-end server, you have to enable Integrated Windows Authentication on /Microsoft-Server-ActiveSync virtual folder in the Exchange System Manager (ESM) interface. Otherwise, OWA and Activesync access fails. Now comes the tricky part: When you try to enable "Windows Integrated Authentication" for /Microsoft-Server-ActiveSync virtual directory, you will notice that you cannot click on "Authentication" button since it's dimmed by default. If you enable "Integrated Windows Authentication" on the virtual directory using IIS Manager interface, Activesync users will receive multiple username/password prompts and will fail to access their mailboxes. This is behavior is the result of DS to Metabase (DS2MB) process which turns off Integrated Windows Authentication on the Windows 2003 back-end server.
Solution:
Download and install the hotfix described in the following MS KB Article:
http://support.microsoft.com/kb/937031
As described in the KB article, this hotfix enables the dimmed button, and lets you select "Integrated Windows Authentication" in ESM. Modifying the setting in ESM makes it permanent, and Activesync users can seamlessly access their mailboxes regardless of their Exchange server version.
4. Unable to Uninstall Legacy Exchange Server (2003)
Background:
At the final stages of an Exchange 2003 to Exchange 2007 Migration, you need to uninstall Exchange 2003 server(s) as a cleanup process.
Problem:
When you try to uninstall the old Exchange server, you cannot select "Remove" option and receive the following message:
The component "Microsoft Exchange Messaging and Collaboration Services" cannot be assigned the action "Remove" because:
-One or more users currently use a mailbox store on this server. These users must be moved to a mailbox store on a different server or be mail disabled before uninstalling this server.
You double check the old Mailbox Store and make sure that no mailboxes are listed, the mailboxes were either moved to Exchange 2007 stores or removed from the organization. However Exchange Uninstall still creates the warning message and does not let you proceed with the "remove" operation.
The problem is actually not directly related to Exchange mailboxes. The uninstall program queries AD for the msExchHomeServer attribute and tries to find the name of the old server. If a match is found, simply produces the above error message and refuses to proceed.
Solution:
To resolve the problem, you need to perform the following steps:
- Right-click the domain container, and then click Find
- Select the Advanced tab
- Select User from the Field button
- From the list of attributes displayed, choose Exchange Home Server
- Set the Condition field to Ends With
- Enter the Exchange server name into the Value field (Exchange 2003 server name)
- Click Add
- Click the Find
- In the results Window, select the listed user account
- Right-click on the user account name, then select "Exchange Tasks"
- Click Next on the first screen
- Select "Remove Exchange Attributes" on the second screen click Next and continue with the wizard till the end
- Repeat the steps 9 to 12 for each user in the results list
Following Microsoft KB Article discusses the same topic with a slightly different resolution:
http://support.microsoft.com/kb/924170
Oz Ozugurlu also describes the problem and resolution at his blog site:
http://smtp25.blogspot.com/2007/04/having-trouble-to-uninstall-exchange.html
Sunday, October 19, 2008
Problem with Autodiscover on Exchange 2007 SP1
I have run into this problem with an Exchange Server 2007 SP1 implementation running on Windows Server 2008 x64. I also noticed some forum posts where people had the exact same issue on Windows Server 2003 x64, however I cannot verify the solution method described here would help if you're on Windows Server 2003 x64.
Although the main problem is with the Autodiscover web service, you may experience crashes in Outlook 2007 with Scheduling Assistant or errors with Out of Office Asisstant (OOF). Also Offline Addressbook (OAB) downloads may not work properly. When used with Outlook 2007, all these features depend on Autodiscover, so they fail when Autodiscover has a problem. In my case, users noticed that they were not able to "Add Attendees" when trying to create a meeting request with Scheduling Assistant. Outlook 2007 was crashing when they selected an attendee from the addressbook. After a brief research, I also noticed that Out of Office Assitant was also failing with "Your Out of Office settings cannot be displayed, because the server is currently unavailable" message. My attempts to download Offline Address Book were also unsuccessful. Download attempts did not succeed, but they ran forever and did not produce an error message either.
I will not list all the boring troubleshooting details here, but I want to mention that I deleted and recreated Autodiscover virtual directory with "Remove-AutodiscoverVirtualDirectory" and "New-AutodiscoverVirtualDirectory" commandlets in Powershell. These commands ran without any issues, but they did not help.
Also I ran "Test-OutlookWebServices" commandlet on Exchange Server and received the following error at the fourth step:
PS] C:\Windows\System32>Test-OutlookWebServices |fl
Id : 1003
Type : Information
Message : About to test AutoDiscover with the e-mail address Administrator@xyz.com.
Id : 1007
Type : Information
Message : Testing server EX01.xyz.com with the published name https://EX01.xyz.com/EWS/Exchange.asmx & https://mail.xyz.com/EWS/Exchange.asmx.
Id : 1019
Type : Information
Message : Found a valid AutoDiscover service connection point. The AutoDiscover URL on this object is https://EX01.xyz.com/autodiscover/autodiscover.xml.
Id : 1006
Type : Information
Message : The Autodiscover service was contacted at https://EX01.xyz.com/autodiscover/autodiscover.xml.
WARNING: An unexpected error has occurred and debug information is being generated: Object reference not set to an instance of an object.
Test-OutlookWebServices : Object reference not set to an instance of an object.
At line:1 char:24
+ Test-OutlookWebServices <<<< |fl
Because Outlook "Test E-mail AutoConfiguration" ended with a "certificate error" I also re-created the SSL certificate and enabled it for all services. No need to say, none of these actions resolved the problem.
I was able to find the actual problem source and solution at Technet Forums. (At http://forums.microsoft.com/TechNet/ShowPost.aspx?PageIndex=0&SiteID=17&PageID=0&PostID=3760137, to be exact.) Apparently, the problem was caused by a bug in ".NET Framework 2.0 SP2". It simply was not playing nicely with Exchange Server 2007 SP1. Even though you don't explicitly select to install this .NET 2.0 Service Pack 2, it is installed as a part of ".NET Framework 3.5 SP1" setup. With this information, I searched for .NET Framework 2.0 SP2 in Programs and Features, but failed to find it. So I uninstalled .NET Framework 3.5 SP1 instead and restarted the server. However this did not help. Finally, I found that the .NET 2.0 SP2 was displayed as "Update for Microsoft Windows (KB948609)" in Windows 2008 Programs and Features. After I had uninstalled this update, all of my problems disapperared. I was able to verify that Out of Office Assistant, Scheduling Assistant and Offline Address Book downloads were fully functional. In addition, all error messages disappeared from "Test-OutlookWebServices" test in Powershell and "Test E-mail AutoConfiguration" test in Outlook.
I had to spend long hours searching the web and testing every possible solution to find the actual one. Hope this post will reduce your troubleshooting period. Please use the Comments link below if want to say something about this post.
Thursday, October 9, 2008
Service Fails to Start After Reboot - Port Conflicts
For the last couple of months, I have experienced high number of unexpected service failures. Randomly, one of the services fails to start successfully after a reboot. Usually, the failed service belongs to a recently installed application. 100% of these incidents were caused by conflicting TCP or UDP ports. Because the port number is already in use, the service fails to start.
In one of the past incidents it was Blackberry Router service, today it was Internet Authentication Server (IAS) service. All of the port conflicts I have seen were on Windows 2003 Domain Controller servers or Windows 2003 SBS. Since the troubleshooting methodology is the same, I will try to describe how to determine the application causing the port conflict and how to resolve the problem.
First of all, you have to know the port numbers and port type of the failed service. For example, Blackberry Router service works on TCP 3101, IAS works on UDP 1645-1646 and UDP 1812-1813. To find the specific port information for a specific service, easiest way is to perform a Google search. If we want to find the port information about Blackberry Router Service, typing “blackberry router service port” in the Google search bar will easily yield to port 3101.
As a second step, you need to verify the port conflict (i.e. verify that the port is already in use). You can achieve this by simply running the following command:
Netstat –an |findstr :portnumber
You need to replace the portnumber with the port number of the failed service. If the failed service uses multiple ports, you have to repeat the command for each port. For example, for Blackberry Router Service, use the following command:
Netstat –an |findstr :3101
If the port is not in use, above command will not return any results. This also means that your root cause is not a port conflict. Unfortunately, this post will not provide any assistance for your problem.
If the port is already in use, you should see a result similar to the line below.
TCP 0.0.0.0:3101 0.0.0.0:0 LISTENING
To verify the port conflict, you can alternatively search Windows Application Log for errors, or look at text based log files for your failed service.
After verifying that the port is in use, the next step would be to determine the application/process using this port. For this purpose, the tool we need to use is TCPView from Sysinternals. You can download the tool at http://technet.microsoft.com/en-us/sysinternals/bb897437.aspx.
The compressed download includes two executables: TCPView.exe – the GUI version, TCPvcon.exe – command line version. I strongly recommend using the command line version because it’s faster. Simply extract the file TCPvcon.exe to a folder, open command prompt, navigate to the folder, and run the command with the following syntax:
TCPvcon –a >tcpview.txt
Now, you can open tcpview.txt with your favorite text editor. Then, search the file contents for the conflicting port number(s). When you find the matching port number, you will also see the application using that port.
In my experience, most common applications causing the conflict are LSASS.EXE and DNS.EXE.
LSASS.EXE
Local Security Authority Subsystem Service (LSASS), is a process in Microsoft Windows operating systems that is responsible for enforcing the security policy on the system. In addition, it is responsible for the following components:
• Local Security Authority
• Net Logon service
• Security Accounts Manager service
• LSA Server service
• Secure Sockets Layer (SSL)
• Kerberos v5 authentication protocol
• NTLM authentication protocol
By default, LSASS.EXE uses random TCP ports ranging from 1024 to 65535. To restrict the port usage of this process, you need to follow the instructions in the MS KB article below:
http://support.microsoft.com/kb/224196
After selecting a dedicated non-conflicting port for LSASS and applying the registry changes in the article, you need to restart the server.
DNS.EXE
In the past, DNS server was designed to listen on TCP port 53 and UDP port 53. However, this behavior has been changed by a recent security patch. If you applied security update 953230 on your DNS server, you will find that it allocates 2500 UDP ports by default. This is the root cause of the most port conflicts. For general information about this security patch and the side effects, you need to read the following article:
http://support.microsoft.com/kb/953230
Although the problem has multiple workarounds, if you want to restore the service functionality and resolve the port conflict while avoiding opening a gap in the security, you need to define reserved ports in the registry. For SBS 2003 version of the detailed resolution you need to check out the following article:
http://support.microsoft.com/kb/956189
For a general Windows 2000/2003 servers, the resolution information can be found at:
http://support.microsoft.com/kb/812873
As described in the articles, after the registry changes, a reboot will be required.
In one of the past incidents it was Blackberry Router service, today it was Internet Authentication Server (IAS) service. All of the port conflicts I have seen were on Windows 2003 Domain Controller servers or Windows 2003 SBS. Since the troubleshooting methodology is the same, I will try to describe how to determine the application causing the port conflict and how to resolve the problem.
First of all, you have to know the port numbers and port type of the failed service. For example, Blackberry Router service works on TCP 3101, IAS works on UDP 1645-1646 and UDP 1812-1813. To find the specific port information for a specific service, easiest way is to perform a Google search. If we want to find the port information about Blackberry Router Service, typing “blackberry router service port” in the Google search bar will easily yield to port 3101.
As a second step, you need to verify the port conflict (i.e. verify that the port is already in use). You can achieve this by simply running the following command:
Netstat –an |findstr :portnumber
You need to replace the portnumber with the port number of the failed service. If the failed service uses multiple ports, you have to repeat the command for each port. For example, for Blackberry Router Service, use the following command:
Netstat –an |findstr :3101
If the port is not in use, above command will not return any results. This also means that your root cause is not a port conflict. Unfortunately, this post will not provide any assistance for your problem.
If the port is already in use, you should see a result similar to the line below.
TCP 0.0.0.0:3101 0.0.0.0:0 LISTENING
To verify the port conflict, you can alternatively search Windows Application Log for errors, or look at text based log files for your failed service.
After verifying that the port is in use, the next step would be to determine the application/process using this port. For this purpose, the tool we need to use is TCPView from Sysinternals. You can download the tool at http://technet.microsoft.com/en-us/sysinternals/bb897437.aspx.
The compressed download includes two executables: TCPView.exe – the GUI version, TCPvcon.exe – command line version. I strongly recommend using the command line version because it’s faster. Simply extract the file TCPvcon.exe to a folder, open command prompt, navigate to the folder, and run the command with the following syntax:
TCPvcon –a >tcpview.txt
Now, you can open tcpview.txt with your favorite text editor. Then, search the file contents for the conflicting port number(s). When you find the matching port number, you will also see the application using that port.
In my experience, most common applications causing the conflict are LSASS.EXE and DNS.EXE.
LSASS.EXE
Local Security Authority Subsystem Service (LSASS), is a process in Microsoft Windows operating systems that is responsible for enforcing the security policy on the system. In addition, it is responsible for the following components:
• Local Security Authority
• Net Logon service
• Security Accounts Manager service
• LSA Server service
• Secure Sockets Layer (SSL)
• Kerberos v5 authentication protocol
• NTLM authentication protocol
By default, LSASS.EXE uses random TCP ports ranging from 1024 to 65535. To restrict the port usage of this process, you need to follow the instructions in the MS KB article below:
http://support.microsoft.com/kb/224196
After selecting a dedicated non-conflicting port for LSASS and applying the registry changes in the article, you need to restart the server.
DNS.EXE
In the past, DNS server was designed to listen on TCP port 53 and UDP port 53. However, this behavior has been changed by a recent security patch. If you applied security update 953230 on your DNS server, you will find that it allocates 2500 UDP ports by default. This is the root cause of the most port conflicts. For general information about this security patch and the side effects, you need to read the following article:
http://support.microsoft.com/kb/953230
Although the problem has multiple workarounds, if you want to restore the service functionality and resolve the port conflict while avoiding opening a gap in the security, you need to define reserved ports in the registry. For SBS 2003 version of the detailed resolution you need to check out the following article:
http://support.microsoft.com/kb/956189
For a general Windows 2000/2003 servers, the resolution information can be found at:
http://support.microsoft.com/kb/812873
As described in the articles, after the registry changes, a reboot will be required.
The Official SBS Blog also has a great entry where they explain some additional problems caused by DNS and their recommended solutions in their site. For me, most important one is experienced at IPSEC level, where IPSEC service fails and IPSEC driver enters "block mode". This is a very common issue and can only be avoided by adding "ReservedPorts" entry for IPSEC. Please read the following post for details.
I tried to explain some common root causes of the IP port conflicts and the methodology to resolve these problems. I hope this post will ease the pain of troubleshooting similar problems. I also would like to hear about other conflicting applications and your resolutions. Please use the comments link if you want to share your story about a conflicting port.
Subscribe to:
Posts (Atom)
Posts
