A User Deleted Emails by Mistake? No Need to Restore from Backup!

Before I start, I have to admit that the title of this post can be found little bit misleading. This statement is only true under certain circumstances. However, to my experience most Outlook users usually are quick to realize their own mistakes and they usually inform IT Department about accidental deletions within a week. So if your "Deleted Item Retention Period" value is set to a number larger than 7 (or still at default value of 14) days, and if the user comes with a restore request within this period, there really is no need for a restore from backup. Although this is feature has been around for a long time, I recently realized that it's not well known among most IT Generalists. 

Last week, for two occasions I was asked for assistance in restoring lost emails from backups. In the first case, one of my clients was trying to restore emails with Symantec Backup Exec 12. In the second case, another client needed step by step guidance with MS Data Protection Manager (DPM) 2007 to restore an accidentally deleted folder within "Inbox". Surprisingly, they both had users who accidentally deleted a bunch of emails and email folders from their Exchange Server mailboxes. For both cases, it took a little investigation to find out that the deletions took place within the last 7 days. So when I recommended them to use "Recover Deleted Items" feature in Outlook instead of tape restores, both of my clients were surprised. 

Both of these clients are sharp and intelligent IT Generalists. They manage mid-sized networks with a bunch of business applications along with their Exchange Servers. So Exchange Server management is not their only (or primary) responsibility. I think that most of the IT Generalists might be missing the fact that you can recover any deleted mail item within the limits of "Deleted Item Retention Period". So this post is for you, the IT Generalist who missed this powerful Exchange Server and Outlook feature. If you're an Exchange Server Admin (I mean, if Exchange Server is your primary responsibility), I assume you're already aware of this really neat feature. 

You can find plenty of online blog posts and articles on this topic. Microsoft KB Article 246153 explains the required registry key change for Outlook 2003 and older versions. MSExchange.org has an excellent article which describes all the basic information in detail along with OWA restore method. However, because these resources are relatively older, they fail to mention that Outlook 2007 does not require a registry trick to enable this feature. In other words, with Outlook 2007, you don't need to add the "DumpsterAlwaysOn" registry value to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Exchange\Client\Options key, because it's already there by default. For the Outlook versions prior to 2007, you have to add the registry value manually as described in MS KB Article, then restart Outlook. Alternatively, if you are still on Exchange 2003, you can use the OWA method described at MSExchange.org. However, Exchange 2007 OWA does not provide a similar functionality. You can only recover deleted items at the "Deleted Items" folder in OWA 2007 SP1.

With Recover Deleted Items feature, you can recover individual mail items, folders along with their contents, even if they were deleted using "Shift + Delete". If the deleted folder is one of the top level folders (like Inbox, Sent Items, Junk Email, etc.), you need to select the "Mailbox" at the Navigation Pane, then select Tools and Recover Deleted Items to reach the list of deleted folder(s). With this method you can also recover items that are deleted with "Mailbox Management" process. 

Finally, I want to mention about a great MSDN blog post by Doug Gowan, which discusses the necessity of backups in Exchange 2007 environments and outlines the scenarios when "Recover Deleted Items" feature can be used.

Limitations of Branch Distribution Points in SCCM

If you are going to implement the branch office scenario in SCCM and deploy branch distribution points on workstations, be sure to read the article "Clarifying: Retry Behavior for Distribution Points" which is not officially documented in TechNet and posted by SMS Writing Team.
It is about the 10 concurrent TCP connections limitation in client operating systems.

Install XP Guest on ESX Systems

As ESX does not support IDE disk controllers, it will be tricky to install an XP guest on the ESX host. You will either need LSI or Bus Logic SCSI drivers in FLP image format.
So it is a question where to find or how to create these floppy images.
Here is a link to download official BusLogic floppy image (also can be found in VmWare site but I could not find it easily in my second attempt...)

Download Floppy Image

Windows Firewall and SCOM Agent

When windows firewall is enabled by group policy on clients and you want to install SCOM agent using discovery in SCOM console, you get a message saying "RPC server is unavailable" at the end of discovery and agent install fails.

The below powershell script reads computer accounts from AgentList.txt and uses .net to stop windows firewall service on the remote host (as powershell does not support remote shell at the moment) and then uses discovery to install SCOM agent. After finishing, it starts the Windows Firewall service again.

Download PowerShell Script

Desired Configuration Management in Configuration Manager

Desired Configuration Management is a a new and useful fetaure in Configuration Manager. I tried to explain the basic steps to configure and interpret the reports of DCM.

Download Article

Script to Audit Disk Usage

This is a very basic script, which connects each server in a given list, then collects and reports the disk usage information for the fixed disks on each system. Script displays the drive letter, disk size, used and available disk space information on the screen and writes to a tab separated text file. I selected the tab separated file format for portability: You can easily copy and paste the contents to an Excel sheet. Script uses WMI to connect to the servers and collect the data, so please be aware that you will either need to shutdown Windows Firewall or add a "file and printer sharing exception" on the target systems.

I have tested the script on Windows XP, Windows Vista, Windows 2003 and Windows 2008. I hope the script will be very handy for easily creating disk usage reports on multiple servers and/or workstations. This script uses a very similar logic to my service account audit script. Only difference is in the WMI query section.

Usage:

cscript q_FixedDrv.vbs filename

For example:

cscript q_FixedDrv.vbs serverlist.txt

To download the script and sample serverlist.txt file, click on "I agree", you will be forwarded to the download site. Modify the serverlist.txt file to fit your test environment, then start the script. If the result you get is satisfactory, then create another list file which includes your production servers/workstations. If you have any feedback regarding this script, feel free to contact me using the comment link. I really want to know if you like it or hate it.

Exchange 2007 Migration - Common Issues

During my recent Exchange 2007 Migration Projects, I had to work on and produce solutions for a number of interesting problems.

To ease your pain and shorten the troubleshooting process associated with such issues, I will try to list some of the most common issues and their solutions in this post. All of the problems and solutions listed here can also be found in different discussion forums, blog sites or KB articles, but so far I haven't seen any consolidated document for common issues in Exchange 2007 migrations. I hope this post will fill in this gap.

1. Problem with Autodiscover (Outlook Web Services) on Windows 2008
2. Outlook Anywhere (RPC over HTTP) Does not Work on Windows 2008
3. ActiveSync (EAS) Issue with dedicated CAS and Exchange 2003 Back-end
4. Unable to Uninstall Legacy Exchange Server (2003)


1. Problem with Autodiscover (Outlook Web Services) on Windows 2008
Background:
When Outlook 2007 is used to connect to an Exchange 2007 server, by default Offline Address Book, Out of Office Assistant and Scheduling Assistant features try to connect to the Web based services. All these features actually depend on Autodiscover feature of the Exchange 2007 Client Access server (CAS).

Problem:
Outlook 2007 users detect problems with Offline Address Book, Out of Office Assistant and Scheduling Assistant. Especially, when trying to "Add Attendees" using Scheduling Assistant, Outlook crashes and restarts.

Solution:
The problem is caused by .NET 2.0 SP2. Uninstalling .NET 2.0 SP2 resolves the issue, all Autodiscover related features start to function properly.The details of the problem and recommended solution can be found at my previous post:

http://adventuresofanitpro.blogspot.com/2008/10/problem-with-autodiscover-on-exchange.html

2. Outlook Anywhere (RPC over HTTP) Fails on Windows 2008
Background:
For Outlook Anywhere feature to work properly, Client Access servers (CAS) need to access three TCP ports on the Mailbox Server: 6001, 6002 and 6004. Even in single server Exchange 2007 implementations, these three ports are needed.

Problem:
When you try to access these three ports through telnet tests, you will notice that 6001 and 6002 are accessible. However, you cannot access port 6004. If you check the ports on the Mailbox Server using netstat, you can verify that the server is listening on the all three ports. For some reason, because when IPv6 is enabled (and it's enabled by default on Windows 2008), it prevents access to TCP port 6004. There is one workaround and one permanent solution.

Solution:
For the quick workaround you can add the server's IP address and name in the local HOSTS file. Surprisingly this simple action re-enables access to port 6004.

For the permanent solution, you need to disable IPv6 for good. To disable IPv6, you need to perform the following steps:

In the Network Connections folder, obtain properties on all of your connections and adapters and clear the check box next to the Internet Protocol version 6 (TCP/IPv6) component in the list under This connection uses the following items. This method disables IPv6 on your LAN interfaces and connections, but does not disable IPv6 on tunnel interfaces or the IPv6 loopback interface.

Add the following registry value (DWORD type) set to 0xFFFFFFFF:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters\DisabledComponents.

This method disables IPv6 on all your LAN interfaces, connections, and tunnel interfaces but does not disable the IPv6 loopback interface. You must restart the computer for this registry value to take effect.

You will also need to replace the following line in HOSTS file

::1 localhost

with this line:

#::1 localhost

and add the following lines
IPv4 address hostname of the computer
IPv4 address FQDN of the computer

(Please replace IPv4 address with the IP address value of your server and names with the actual name of your server for the entries given above)
After these changes, you need to restart the server. Following the restart you will notice that you can connect via RPC/HTTP(s) with no problems.

MS Exchange Team Blog has an excellent post about this topic at the following link:

http://msexchangeteam.com/archive/2008/06/20/449053.aspx

Also Aaron Marks describes the problem and the solution in detail at his blog site:

http://blog.aaronmarks.com/?p=65


3. ActiveSync (EAS) Issue with dedicated CAS and Exchange 2003 Back-end
Background:
During a well planned transition to Exchange 2007 from legacy Exchange environment, one of the recommended actions is to install a CAS server (on a seperate box other than Mailbox server) and redirect all OWA, RPC/HTTP and ActiveSync requests to this new CAS server. Since CAS uses a process called proxying, and has the ability to proxy for both Exchange 2003 and Exchange 2007 mailboxes, all requests will be proxied to proper back-end (or mailbox) server. For This setup provides the best co-existence support with the legacy Exchange mailboxes and lets you seamlessly transition to Exchange 2007. You can find more information on this topic at the following MS Exchange Team Blog post:

http://msexchangeteam.com/archive/2007/09/04/446918.aspx

Problem:
For mailboxes located on Exchange 2003 back-end server, you have to enable Integrated Windows Authentication on /Microsoft-Server-ActiveSync virtual folder in the Exchange System Manager (ESM) interface. Otherwise, OWA and Activesync access fails. Now comes the tricky part: When you try to enable "Windows Integrated Authentication" for /Microsoft-Server-ActiveSync virtual directory, you will notice that you cannot click on "Authentication" button since it's dimmed by default. If you enable "Integrated Windows Authentication" on the virtual directory using IIS Manager interface, Activesync users will receive multiple username/password prompts and will fail to access their mailboxes. This is behavior is the result of DS to Metabase (DS2MB) process which turns off Integrated Windows Authentication on the Windows 2003 back-end server.

Solution:
Download and install the hotfix described in the following MS KB Article:

http://support.microsoft.com/kb/937031

As described in the KB article, this hotfix enables the dimmed button, and lets you select "Integrated Windows Authentication" in ESM. Modifying the setting in ESM makes it permanent, and Activesync users can seamlessly access their mailboxes regardless of their Exchange server version.

4. Unable to Uninstall Legacy Exchange Server (2003)
Background:
At the final stages of an Exchange 2003 to Exchange 2007 Migration, you need to uninstall Exchange 2003 server(s) as a cleanup process.

Problem:
When you try to uninstall the old Exchange server, you cannot select "Remove" option and receive the following message:

The component "Microsoft Exchange Messaging and Collaboration Services" cannot be assigned the action "Remove" because:

-One or more users currently use a mailbox store on this server. These users must be moved to a mailbox store on a different server or be mail disabled before uninstalling this server.

You double check the old Mailbox Store and make sure that no mailboxes are listed, the mailboxes were either moved to Exchange 2007 stores or removed from the organization. However Exchange Uninstall still creates the warning message and does not let you proceed with the "remove" operation.

The problem is actually not directly related to Exchange mailboxes. The uninstall program queries AD for the msExchHomeServer attribute and tries to find the name of the old server. If a match is found, simply produces the above error message and refuses to proceed.

Solution:
To resolve the problem, you need to perform the following steps:

1. Right-click the domain container, and then click Find
2. Select the Advanced tab
3. Select User from the Field button
4. From the list of attributes displayed, choose Exchange Home Server
5. Set the Condition field to Ends With
6. Enter the Exchange server name into the Value field (Exchange 2003 server name)
7. Click Add
8. Click the Find
9. In the results Window, select the listed user account
10. Right-click on the user account name, then select "Exchange Tasks"
11. Click Next on the first screen
12. Select "Remove Exchange Attributes" on the second screen click Next and continue with the wizard till the end
13. Repeat the steps 9 to 12 for each user in the results list

After this cleanup operation, you will be able to select "Remove" during the Exchange Setup, and will be able to proceed with the uninstall with no problems.

Following Microsoft KB Article discusses the same topic with a slightly different resolution:

http://support.microsoft.com/kb/924170

Oz Ozugurlu also describes the problem and resolution at his blog site:

http://smtp25.blogspot.com/2007/04/having-trouble-to-uninstall-exchange.html