Service Fails to Start After Reboot - Port Conflicts

For the last couple of months, I have experienced high number of unexpected service failures. Randomly, one of the services fails to start successfully after a reboot. Usually, the failed service belongs to a recently installed application. 100% of these incidents were caused by conflicting TCP or UDP ports. Because the port number is already in use, the service fails to start.

In one of the past incidents it was Blackberry Router service, today it was Internet Authentication Server (IAS) service. All of the port conflicts I have seen were on Windows 2003 Domain Controller servers or Windows 2003 SBS. Since the troubleshooting methodology is the same, I will try to describe how to determine the application causing the port conflict and how to resolve the problem.

First of all, you have to know the port numbers and port type of the failed service. For example, Blackberry Router service works on TCP 3101, IAS works on UDP 1645-1646 and UDP 1812-1813. To find the specific port information for a specific service, easiest way is to perform a Google search. If we want to find the port information about Blackberry Router Service, typing “blackberry router service port” in the Google search bar will easily yield to port 3101.

As a second step, you need to verify the port conflict (i.e. verify that the port is already in use). You can achieve this by simply running the following command:

Netstat –an |findstr :portnumber

You need to replace the portnumber with the port number of the failed service. If the failed service uses multiple ports, you have to repeat the command for each port. For example, for Blackberry Router Service, use the following command:

Netstat –an |findstr :3101

If the port is not in use, above command will not return any results. This also means that your root cause is not a port conflict. Unfortunately, this post will not provide any assistance for your problem.

If the port is already in use, you should see a result similar to the line below.

TCP 0.0.0.0:3101 0.0.0.0:0 LISTENING

To verify the port conflict, you can alternatively search Windows Application Log for errors, or look at text based log files for your failed service.

After verifying that the port is in use, the next step would be to determine the application/process using this port. For this purpose, the tool we need to use is TCPView from Sysinternals. You can download the tool at http://technet.microsoft.com/en-us/sysinternals/bb897437.aspx.

The compressed download includes two executables: TCPView.exe – the GUI version, TCPvcon.exe – command line version. I strongly recommend using the command line version because it’s faster. Simply extract the file TCPvcon.exe to a folder, open command prompt, navigate to the folder, and run the command with the following syntax:

TCPvcon –a >tcpview.txt

Now, you can open tcpview.txt with your favorite text editor. Then, search the file contents for the conflicting port number(s). When you find the matching port number, you will also see the application using that port.

In my experience, most common applications causing the conflict are LSASS.EXE and DNS.EXE.

LSASS.EXE
Local Security Authority Subsystem Service (LSASS), is a process in Microsoft Windows operating systems that is responsible for enforcing the security policy on the system. In addition, it is responsible for the following components:
• Local Security Authority
• Net Logon service
• Security Accounts Manager service
• LSA Server service
• Secure Sockets Layer (SSL)
• Kerberos v5 authentication protocol
• NTLM authentication protocol

By default, LSASS.EXE uses random TCP ports ranging from 1024 to 65535. To restrict the port usage of this process, you need to follow the instructions in the MS KB article below:

http://support.microsoft.com/kb/224196

After selecting a dedicated non-conflicting port for LSASS and applying the registry changes in the article, you need to restart the server.

DNS.EXE
In the past, DNS server was designed to listen on TCP port 53 and UDP port 53. However, this behavior has been changed by a recent security patch. If you applied security update 953230 on your DNS server, you will find that it allocates 2500 UDP ports by default. This is the root cause of the most port conflicts. For general information about this security patch and the side effects, you need to read the following article:

http://support.microsoft.com/kb/953230

Although the problem has multiple workarounds, if you want to restore the service functionality and resolve the port conflict while avoiding opening a gap in the security, you need to define reserved ports in the registry. For SBS 2003 version of the detailed resolution you need to check out the following article:

http://support.microsoft.com/kb/956189

For a general Windows 2000/2003 servers, the resolution information can be found at:

http://support.microsoft.com/kb/812873

As described in the articles, after the registry changes, a reboot will be required.

The Official SBS Blog also has a great entry where they explain some additional problems caused by DNS and their recommended solutions in their site. For me, most important one is experienced at IPSEC level, where IPSEC service fails and IPSEC driver enters "block mode". This is a very common issue and can only be avoided by adding "ReservedPorts" entry for IPSEC. Please read the following post for details.

http://blogs.technet.com/sbs/archive/2008/07/17/some-services-may-fail-to-start-or-may-not-work-properly-after-installing-ms08-037-951746-and-951748.aspx

I tried to explain some common root causes of the IP port conflicts and the methodology to resolve these problems. I hope this post will ease the pain of troubleshooting similar problems. I also would like to hear about other conflicting applications and your resolutions. Please use the comments link if you want to share your story about a conflicting port.