Update Rollup 7 for Exchange Server 2007 Service Pack 1 Can Kill OWA and Web Based Services

No matter how careful you are with your network, you cannot avoid getting bitten by patches or fixes occasionally. I have seen this exact same behavior with three different networks, so I think there's some sort of pattern going on. 

Here's what's been happening: Right after you install "Update Rollup 7 for Exchange Server 2007 Service Pack 1" successfully in your environment, OWA and ActiveSync may not work at all. I want to repeat this one more time: the installation might run without any errors or warnings, but you may still end up with a non-functional OWA. 

Uninstalling Rollup 7 might solve the problem, but I have found a better solution: Simply repeat the installation of the "Update Rollup 7 for Exchange Server 2007 Service Pack 1" package one more time. This second install will take care of OWA and Web Based Services problems for good. Only in one instance, the installation process failed to re-enable the Exchange services at the end of the installation. Simply re-enabling Exchange Services and restarting the server solved the problem.

I am not sure under what circumstances this issue occurs, but I hope this simple yet effective tip can save you from frustration.

How to Install Operations Manager 2007 R2 on Windows 2008

It is a little tricky path (espacially reporting) to install OpsMgr 2007 wSP1 or OpsMgr 2007 R2 on a Windows 2008 system.
I tried to explain the steps to take before every component installation for a successful OpsMgr installation.

Download Document

Terminal Server 2008 - Issues and Solutions - Part 1

I have been working on a project where we setup a TS 2008 farm (made up of three virtual servers) on a three node VMware Infrastructure 3 environment. During the implementation of the systems, we had to tackle a number of problems and discrepancies. Some of the problems were related to Windows 2008 compatibility, and some of them were direct results of feature set changes in the new OS.

1. Windows 2008 on VMWare ESX and Painfully Slow Performance

Our Session Broker Load Balanced farm included three virtual Windows 2008 Terminal Services servers. Along with them we had two seperate file servers. One of them was designated for roaming terminal services profiles, the second one was for shared MS Office files and users' home folders. All systems were running on Windows 2008, except one single Windows 2003 domain controller for another which was used for domain migration. ESX hosts were running on 16GB of RAM and Quad Core 2.5GHz CPUs, while guest TS servers were configured with 4GB of RAM and 4 CPUs.

Right after the installation and data migration, we invited some pilot users to test the environment. They were supposed to test standard Microsoft Office Applications, along with their third party Business Management Software. All pilot users agreed on one point: systems were "painfully slow". Logons were taking extremely long, sometimes up to 5-6 minutes. Opening a 40KB Excel file was taking a minute on average and sometimes users had to wait forever for a small file to open. Furthermore, working in their Business Software was almost impossible.

The environment described in that article was very similar to ours with one exception: while we were running Symantec Corporate Edition 10.2, their problems were caused by Symantec EndPoint Protection Server 11.0. Knowing that both security solutions share the same roots and just to see the effects, we removed SAV Corporate Edition from the file servers. After the reboot, performance returned back to normal.

Removing the SAV Corporate Edition from the Terminal Servers helped reaching the normal perfomance values. We were able to open any Excel file with no latency. As described in the article above, instead of Symantec Corporate Edition 10.2, we upgraded the systems to Symantec EndPoint Protection 11 MR3. Installing Symantec EndPoint 11 MR3 did not affect the performance, so I hope the problem has been resolved with this release. We have also installed a number of OS patches to the systems after this, so far the systems are still running properly, I assume Symantec had adressed and resolved the problem with MR3. At the time of this writing, I can see that the latest release for Symantec Endpoint Protection is MR4. We are planning to implement this soon, I will also update the post if it affects the performance.

2. Terminal Services 2008 and Adobe Reader Problem

Some users were not able to run Adobe Reader on the Terminal Servers. After every failure, they were getting "Adobe Reader has stopped working" error message. With each failure, the following message was logged in the Application log:

Log Name: Application

Source: Application Error

Date: 12/16/2008 1:53:03 PM

Event ID: 1000

Task Category: (100)

Level: Error

Keywords: Classic

User: N/A

Computer: TS01.domain.local

Description:

Faulting application AcroRd32.exe, version 9.0.0.332, time stamp 0x4850f0a3, faulting module Annots.api, version 9.0.0.332, time stamp 0x4850e57f, exception code 0xc0000005, fault offset 0x001bd9e0, process id 0x2e1c, application start time 0x01c95fc0491c84c1.

Since the error points out to "Annots.api" file, my initial response was to locate the file in Plug_ins (C:\Program Files\Adobe\Reader 9.0\Reader\plug_ins) folder and rename it to "annots1.api". However this did not help. The user received the exact same ""Adobe Reader has stopped working" error, and in the application log complained about "annots1.api" this time.

My second action was to move the file out of plug_ins folder. After the move the problem disappeared. However, I was sure that this is not a wise solution. So as a final test, I moved the file back to the plug_ins folder then enabled Compatibility mode and set it to "Windows XP SP2" on "Acrord32.exe" file. I was able to verify that this action resolved the problem for good. No problems were encountered after this change. I also had to repeat this action two other Terminal Servers.

3. Terminal Services 2008 and Session Broker Problem

To be able to distribute the load among three Terminal Servers, we have configured Terminal Services Session Broker service and Session Broker Load Balancing. I will not go through all the configuration steps, you can easily find detailed configuration information from many different resources in the web. However, I would like to mention about a discrepancy in Session Broker configuration. One of the steps during configuration is to supply each Terminal Server with the name of the Session Broker Server. This can be achieved by two different methods:

i. You can specify the name manually on each server's Terminal Services Configuration

ii. You can specify the name in a GPO and assign this GPO to all Terminal Servers

Regardless of the configuration method, as per Microsoft, the name you need to specify can be either in host name, IP address of FQDN form. In GPO Editor, if you examine the "Explain" section of the group policy setting (Computer Configuration\Policies\Administrative Templates\Windows Components\Terminal Services\Terminal Server\TS Session Broker\Configure TS Session Broker server name), it reads:

"If you enable this policy setting, you must specify the TS Session Broker server, using either its host name, IP address, or fully qualified domain name. If you specify a name or IP address for the TS Session Broker server that is not valid, an error message is logged in Event Viewer on the terminal server."

Unfortunately, I found that above statement is only partially correct. If you select to use FQDN, you will eventually see the following errors on each Terminal Server's event log:

Log Name: System
Source: Microsoft-Windows-TerminalServices-SessionBroker-Client
Date: 12/6/2008 9:36:19 PM
Event ID: 1014
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: TS01.domain.local
Description:
The server failed to retrieve the security identifier (SID) of the TS Session Broker server.
Win32 error code: 0x6FC.

You will find that Session Broker functionality intermittently stops (the service stays running but terminal servers fail to join) when you use FQDN in this configuration. To get rid of this warning and intermittent failures, you need to specify host name (i.e. NetBIOS name) of the Session Broker server. After changing it to the host name, the warning does not show up anymore. Somehow, FQDN does not work properly. I did not get a chance to test out the effect of using straight IP address for this setting, if you have similar experience and if you are using the IP address, please let me know if it works.

Exchange 2007 Multi Node Single Copy Cluster and Additional MTA Issue

Based on my experience , there is no need for addinational MTA on Exchange 2003 Multi Node Cluster. If you have an MTA resource on first Exchange 2003 node, you don’t need additional MTA resource at other Exchange Nodes.

But If you install multi-node single copy Exchange 2007 Cluster, you have to create additional MTA in Active Directory (by using AdsiEdit) for every additional Exchange 2007 nodes. If you don’t do that you cannot create a mailbox on these Exchange nodes.

You receive the following error in Exchange Management Console: “A proxy generator DLL on server FQDN.serverName could not be found or failed to initialize. Proxy addresses for the current recipient cannot be calculated. Please ensure that all proxy address generator DLLs have been installed on the target server”

Additionally, the following event may be logged:
Event Type: Error Event Source: MSExchangeIS Event Category: General Event ID: 1201 Description: Error -2147221233 reading/decrypting the msExchResponsibleMTAServer attribute on the Server object in the Active Directory. For more information, see Help and Support Center at Data: 0000: 5b 44 49 41 47 5f 43 54 [DIAG_CT 0008: 58 5d 00 00 26 00 00 00 X]..&... 0010: ff f0 0d 00 00 00 00 00 ÿð...... 0018: 00 02 18 00 00 00 cd 4a ......ÍJ 0020: 20 10 00 00 00 00 bd 5f .....½_ 0028: 20 10 00 00 00 00 bd 5f .....½_ 0030: 20 10 00 00 00 00 .....

Move Mailbox operations doesn’t care about additional MTA on every Exchange2007 Server on cluster. You can successfully move mailboxes to these Exchange servers. But these clients cannot logon via Outlook Web Access or MAPI (Outlook).

You can find additional information at http://support.microsoft.com/kb/928811 Microsoft support article.

802.1X NAP and Certificate Issue

At this posting , I want to talk about 802.1x NAP and Certificate related problems. According to my experience , you have to use a certificate from Enterprise CA at Network Policy Server’s certificate. StandAlone CA’s certificate is not working properly at NPS. Another issue about certificate is Client Machine certificate. According to my experience also, you have to use computer certificate and machine authentication at 802.1x NAP. If you don’t use computer certificate and also computer authentication on 802.1x NAP clients, there would be authentication failure at Pre-Logon stage. This problem causes cached logons on client computers, clients wouldn’t get either group policy or password expiration notifications.

Microsoft Office Communications Server 2007 using Direct SIP to Cisco Unified Communications Manager Simultaneous Ringer 7.0(1) Feature

Hi Everyone
This time , I want to talk about OCS 2007 and Cisco Call Manager Integration. Upon my OCS 2007 telephony experience , Cisco Call Manager has only direct integration with OCS . İf you want to remote call control , you need to Cisco Unified Presence Server. But Cisco Unified Communications Manager‘s mobility feature allows you simultaneous ringing (not Dual Forking) , cisco ip phones or Cisco IP Communicators and OCS MOC clients or Microsoft Phone Editions at the same time. You can find related howto solution at http://www.cisco.com/en/US/solutions/collateral/ns340/ns414/ns728/ns784/716742.pdf. Also you can test this feature at virtual test environment. Because Unified Call Manager 6.x or above can be intalled at WmwareWorkstation or ESX for testing purposes. If you use Dual forking and Remote Call Control you will have "in a call" status even when you are using the phone. It's only Nortel that supports Dual forking and Remote Call Control right now. Here is the Supported IP-PBXs for Microsoft Office Communications Server 2007 link :http://technet.microsoft.com/en-us/office/bb735838.aspx#ippbx2

Vista SP2 Beta and Windows 2008 SP2 Beta Available

Microsoft has released Windows 2008 SP2 Beta with some interesting changes. For me, most important news is that same installer can be used for both Windows 2008 SP2 and Vista SP2. As many of you already know, Windows 2008 was released with SP1 code included. Essentially Windows 2008 RTM and SP1 are the same code. That's why the first downloadable service pack for Windows 2008 is SP2.

If you want to test out the new SP2 Beta, check out the following link:

http://technet.microsoft.com/en-us/windowsserver/dd262148.aspx

You can also find the detailed information here:

http://technet.microsoft.com/en-us/library/dd351466.aspx

Useless hiberfil.sys file on Windows 2008

Hibernation is disabled by default when you first install Windows 2008. However, a hidden file called hiberfil.sys is created as big as the amount of physical memory on the system partition of the disk (not more than 4 GB). As using hibernation is not reasonable on production servers, you can delete this file and gain space by simply running "powercfg.exe /h off" command without quotations.
There is also a support article http://support.microsoft.com/kb/920730/en-us

Check State & Start Specific Service Script

The below script checks the state of the DHCP Server service and tries to start it if stopped.

Download Script

A User Deleted Emails by Mistake? No Need to Restore from Backup!

Before I start, I have to admit that the title of this post can be found little bit misleading. This statement is only true under certain circumstances. However, to my experience most Outlook users usually are quick to realize their own mistakes and they usually inform IT Department about accidental deletions within a week. So if your "Deleted Item Retention Period" value is set to a number larger than 7 (or still at default value of 14) days, and if the user comes with a restore request within this period, there really is no need for a restore from backup. Although this is feature has been around for a long time, I recently realized that it's not well known among most IT Generalists. 

Last week, for two occasions I was asked for assistance in restoring lost emails from backups. In the first case, one of my clients was trying to restore emails with Symantec Backup Exec 12. In the second case, another client needed step by step guidance with MS Data Protection Manager (DPM) 2007 to restore an accidentally deleted folder within "Inbox". Surprisingly, they both had users who accidentally deleted a bunch of emails and email folders from their Exchange Server mailboxes. For both cases, it took a little investigation to find out that the deletions took place within the last 7 days. So when I recommended them to use "Recover Deleted Items" feature in Outlook instead of tape restores, both of my clients were surprised. 

Both of these clients are sharp and intelligent IT Generalists. They manage mid-sized networks with a bunch of business applications along with their Exchange Servers. So Exchange Server management is not their only (or primary) responsibility. I think that most of the IT Generalists might be missing the fact that you can recover any deleted mail item within the limits of "Deleted Item Retention Period". So this post is for you, the IT Generalist who missed this powerful Exchange Server and Outlook feature. If you're an Exchange Server Admin (I mean, if Exchange Server is your primary responsibility), I assume you're already aware of this really neat feature. 

You can find plenty of online blog posts and articles on this topic. Microsoft KB Article 246153 explains the required registry key change for Outlook 2003 and older versions. MSExchange.org has an excellent article which describes all the basic information in detail along with OWA restore method. However, because these resources are relatively older, they fail to mention that Outlook 2007 does not require a registry trick to enable this feature. In other words, with Outlook 2007, you don't need to add the "DumpsterAlwaysOn" registry value to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Exchange\Client\Options key, because it's already there by default. For the Outlook versions prior to 2007, you have to add the registry value manually as described in MS KB Article, then restart Outlook. Alternatively, if you are still on Exchange 2003, you can use the OWA method described at MSExchange.org. However, Exchange 2007 OWA does not provide a similar functionality. You can only recover deleted items at the "Deleted Items" folder in OWA 2007 SP1.

With Recover Deleted Items feature, you can recover individual mail items, folders along with their contents, even if they were deleted using "Shift + Delete". If the deleted folder is one of the top level folders (like Inbox, Sent Items, Junk Email, etc.), you need to select the "Mailbox" at the Navigation Pane, then select Tools and Recover Deleted Items to reach the list of deleted folder(s). With this method you can also recover items that are deleted with "Mailbox Management" process. 

Finally, I want to mention about a great MSDN blog post by Doug Gowan, which discusses the necessity of backups in Exchange 2007 environments and outlines the scenarios when "Recover Deleted Items" feature can be used.