Terminal Server 2008 - Issues and Solutions - Part 1

I have been working on a project where we setup a TS 2008 farm (made up of three virtual servers) on a three node VMware Infrastructure 3 environment. During the implementation of the systems, we had to tackle a number of problems and discrepancies. Some of the problems were related to Windows 2008 compatibility, and some of them were direct results of feature set changes in the new OS.

1. Windows 2008 on VMWare ESX and Painfully Slow Performance

Our Session Broker Load Balanced farm included three virtual Windows 2008 Terminal Services servers. Along with them we had two seperate file servers. One of them was designated for roaming terminal services profiles, the second one was for shared MS Office files and users' home folders. All systems were running on Windows 2008, except one single Windows 2003 domain controller for another which was used for domain migration. ESX hosts were running on 16GB of RAM and Quad Core 2.5GHz CPUs, while guest TS servers were configured with 4GB of RAM and 4 CPUs.

Right after the installation and data migration, we invited some pilot users to test the environment. They were supposed to test standard Microsoft Office Applications, along with their third party Business Management Software. All pilot users agreed on one point: systems were "painfully slow". Logons were taking extremely long, sometimes up to 5-6 minutes. Opening a 40KB Excel file was taking a minute on average and sometimes users had to wait forever for a small file to open. Furthermore, working in their Business Software was almost impossible.

The environment described in that article was very similar to ours with one exception: while we were running Symantec Corporate Edition 10.2, their problems were caused by Symantec EndPoint Protection Server 11.0. Knowing that both security solutions share the same roots and just to see the effects, we removed SAV Corporate Edition from the file servers. After the reboot, performance returned back to normal.

Removing the SAV Corporate Edition from the Terminal Servers helped reaching the normal perfomance values. We were able to open any Excel file with no latency. As described in the article above, instead of Symantec Corporate Edition 10.2, we upgraded the systems to Symantec EndPoint Protection 11 MR3. Installing Symantec EndPoint 11 MR3 did not affect the performance, so I hope the problem has been resolved with this release. We have also installed a number of OS patches to the systems after this, so far the systems are still running properly, I assume Symantec had adressed and resolved the problem with MR3. At the time of this writing, I can see that the latest release for Symantec Endpoint Protection is MR4. We are planning to implement this soon, I will also update the post if it affects the performance.

2. Terminal Services 2008 and Adobe Reader Problem

Some users were not able to run Adobe Reader on the Terminal Servers. After every failure, they were getting "Adobe Reader has stopped working" error message. With each failure, the following message was logged in the Application log:

Log Name: Application

Source: Application Error

Date: 12/16/2008 1:53:03 PM

Event ID: 1000

Task Category: (100)

Level: Error

Keywords: Classic

User: N/A

Computer: TS01.domain.local

Description:

Faulting application AcroRd32.exe, version 9.0.0.332, time stamp 0x4850f0a3, faulting module Annots.api, version 9.0.0.332, time stamp 0x4850e57f, exception code 0xc0000005, fault offset 0x001bd9e0, process id 0x2e1c, application start time 0x01c95fc0491c84c1.

Since the error points out to "Annots.api" file, my initial response was to locate the file in Plug_ins (C:\Program Files\Adobe\Reader 9.0\Reader\plug_ins) folder and rename it to "annots1.api". However this did not help. The user received the exact same ""Adobe Reader has stopped working" error, and in the application log complained about "annots1.api" this time.

My second action was to move the file out of plug_ins folder. After the move the problem disappeared. However, I was sure that this is not a wise solution. So as a final test, I moved the file back to the plug_ins folder then enabled Compatibility mode and set it to "Windows XP SP2" on "Acrord32.exe" file. I was able to verify that this action resolved the problem for good. No problems were encountered after this change. I also had to repeat this action two other Terminal Servers.

3. Terminal Services 2008 and Session Broker Problem

To be able to distribute the load among three Terminal Servers, we have configured Terminal Services Session Broker service and Session Broker Load Balancing. I will not go through all the configuration steps, you can easily find detailed configuration information from many different resources in the web. However, I would like to mention about a discrepancy in Session Broker configuration. One of the steps during configuration is to supply each Terminal Server with the name of the Session Broker Server. This can be achieved by two different methods:

i. You can specify the name manually on each server's Terminal Services Configuration

ii. You can specify the name in a GPO and assign this GPO to all Terminal Servers

Regardless of the configuration method, as per Microsoft, the name you need to specify can be either in host name, IP address of FQDN form. In GPO Editor, if you examine the "Explain" section of the group policy setting (Computer Configuration\Policies\Administrative Templates\Windows Components\Terminal Services\Terminal Server\TS Session Broker\Configure TS Session Broker server name), it reads:

"If you enable this policy setting, you must specify the TS Session Broker server, using either its host name, IP address, or fully qualified domain name. If you specify a name or IP address for the TS Session Broker server that is not valid, an error message is logged in Event Viewer on the terminal server."

Unfortunately, I found that above statement is only partially correct. If you select to use FQDN, you will eventually see the following errors on each Terminal Server's event log:

Log Name: System
Source: Microsoft-Windows-TerminalServices-SessionBroker-Client
Date: 12/6/2008 9:36:19 PM
Event ID: 1014
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: TS01.domain.local
Description:
The server failed to retrieve the security identifier (SID) of the TS Session Broker server.
Win32 error code: 0x6FC.

You will find that Session Broker functionality intermittently stops (the service stays running but terminal servers fail to join) when you use FQDN in this configuration. To get rid of this warning and intermittent failures, you need to specify host name (i.e. NetBIOS name) of the Session Broker server. After changing it to the host name, the warning does not show up anymore. Somehow, FQDN does not work properly. I did not get a chance to test out the effect of using straight IP address for this setting, if you have similar experience and if you are using the IP address, please let me know if it works.