Web Application Proxy Event 13007

Posted on by rkmigblog
Written by Robert Kettel

When you start to use Web Application Proxy Server (WAP) as a replacement for ISA, TMG or UAG and publish Active Sync through it, you might face a lot of Event 13007 warnings in the Microsoft-Windows-Web Application Proxy/Admin event log (I mean, really “a lot”). These are paired with various Event IDs 13006 (“Connection to the backend server failed. Error: (0x80072efe)”).

On the other hand, you don’t get any complaints from the user community. There does not seem to be an impact at all.

Where do these warnings come from? Do they impact our service and can we prevent them from showing up over and over again?

The main cause for the warning 13007 and 13006 is how devices with Exchange Active Sync (EAS) and Direct Push technology connect to Exchange.

Following this Microsoft TechNet article, “a mobile device that’s configured to synchronize with an Exchange 2013 server issues an HTTPS request to the server. This request is known as a PING. The request tells the server to notify the device if any items change in the next 15 minutes in any folder that’s configured to synchronize. Otherwise, the server should return an HTTP 200 OK message. The mobile device then stands by. The 15-minute time span is known as a heartbeat interval.”

With other words, there is a steady HTTPS session for 15 minutes between the EAS device and the Exchange backend which must be supported by all components taking part in the HTTPS session build, usually firewalls, load balancers and (in our case) the WAP servers which proxy the HTTPS session coming from the LBs to the Exchange backend. The long lasting session request is finally terminated by the Exchange server by posting an HTTP 200 message.

Now, looking at the default settings of our WAP servers we find a parameter which can have an influence on that behavior.
The default value for the InactiveTransactionTimeoutSec parameter is 300 (= 5 minutes). That means that if the accessing party does retrieve new responses from the backend service defined in the application settings for more than 5 minutes, the connection is identified as “timed out” and dropped by the WAP server.

From that perspective a connection to the Exchange Backend Service is timed out for the Web Application Proxy (causing a warning event 13007), when
a) the heartbeat interval of Direct Push is longer than the InactiveTransactionTimeoutSec
AND
b) the session was not renewed by the device
AND
c) there was nothing to synchronize in the first 300 seconds of the connection

However, if the device’s HTTPS session is dropped by the WAP server, it will automatically re-initiate a new session (the same what the device would do when getting an HTTP 200 OK message from the backend Exchange server). Therefore this is not a critical behavior at all.

The difference can be found how the device reacts on the dropped connection in comparison with how it handles the HTTP 200 OK post.
In the latter case, the device starts a new HTTPS session with the same heartbeat interval again.
In the first case, the device “assumes” that 15-minute HTTPS requests are blocked and re-initiates a new session with only 8 minutes (480 seconds) – which is still beyond the 300 second default setting of the WAP servers.

To avoid the Event 13007, the InactiveTransactionTimeOutSec parameter needs to be set to a value greater than the defined Active Sync max period. By default, the Active Sync device starts with a 15-minute interval, which would fit into a value of 910 for the InactiveTransactionTimeoutSec.

Since the parameter can be set for each published application individually, you luckily do not need to touch other published applications than Exchange Active Sync.

Example command:

Get-WebApplicationProxyApplication ExchangeActiveSync | Set-WebApplicationProxyApplication –InactiveTransactionsTimeoutSec

Note:

A support engineer with Microsoft Premier stated that there is currently no way to suppress the Event 13007 from appearing in the log files completely.

Exchange Migration first & Mailbox Folder Permissions

Posted on by slshareadex

In many inter forest migration projects, the mailbox migration to the new Exchange Organization into the new forest is performed first and the user migration is performed in a separate step at a later time. After the mailbox migration and before migrating the user objects, you have a classic Resource Forest scenario. Users are created as disabled user accounts in the target forest, receiving a linked mailbox, connected to the source Active Directory user. Important AD attribute in this scenario is msExchMasterAccountSID. This attribute of the disabled target user object holds the objectSID of the source user account. This allows to connect to the own mailbox and shared mailbox resources (Delegate permissions etc.) with the active source user object.

ExMigirst01

Did you ever thought about mailbox folder permissions in this scenario? 

For every migrated folder permission (e.g. with QMM for Exchange) and also every time a user manually adds mailbox folder permissions for another (not Active Directory migrated) user in the target mailbox, the SID of the source user object is added to the mailbox folder permissions. In this example, we’ve selected the not migrated user UserA from the Global Address list and added him as delegate for the Inbox and the calendar for the Info MBX:  ExMigirst02

At some point, the Active Directory migration will start. During this process, the user account in the target domain will be activated and the Linked Mailbox is converted to a User Mailbox. This action will clear the attribute msExchMasterAccountSID. This is necessary because the target account will now be used to access the own mailbox and resources of other mailboxes. If a migrated user is now added to the mailbox folder permissions, the target SID will be added and no longer the source SID. Let’s use the mailbox example above and add the migrated user UserB as additional delegate for the Calendar of the Info MBX:ExMigirst03

In this example, UserB will of course not have any problems accessing the Info MBX. But what happens if UserA will be migrated and the user starts accessing the Info MBX with TARGETDOMAIN\UserA? The SID of the target account has no permissions on the Inbox and the Calendar folder. Will UserA loose access to these folders now? Generally, the answer is YES, UserA will lose access! But…

In Active Directory Migration projects, it is best practice to migrate the SIDHistory to the target user account. In this case, the objectSID of the source user is copied to the attribute SIDHistory of the target account. For our example, it means that UserA will not lose access to the Info MBX because his Access Token contains the Source SID which has permissions on the Inbox and Calendar folder in the Info MBX.

SIDHistory CleanUp & mailbox folder permissions?

Clearing SIDHistory is part of most of the migration projects. Before clearing the SIDHistory attribute of the target accounts, it is required to replace the source SIDs with the corresponding target SIDs inside the mailbox folder permissions. This process is called ReACLn. Without this action, many users will lose access to shared mailbox resources when the SIDHistory attribute is cleared.

Exchange Processing Wizard (Part of Dell Migration Manager for Active Directory)

Dell migration Manager for Active Directory contains the Exchange Processing Wizard. This wizard is able to replace existing source SIDs with the matching target SIDs for permissions inside the exchange environment. The wizard is using the matching information in the QMM AD LDS database, created during the directory synchronization.

To ReACL permissions inside the mailboxes, we have to select the option “Update client permissions”:

ExMigirst04

Now we can choose to process all Public Folders and Mailboxes or select individual Mailboxes or Public Folder or even skip Public Folders or Mailboxes completely:

ExMigirst05

The wizard provides the possibility to only process one server or process multiple servers in parallel.

Known limitation of the Exchange Processing Wizard:

The wizard is unable to set the Free/Busy permissions Free/Busy time, subject, location. After processing, the permission is changed to Free/Busy time only:

ExMigirst06

Good to know – Check real SID behind folder permissions

Get-MailboxfolderPermission: Unfortunately, as long as the SIDHistory is set for a user, Exchange will always resolve the permissions to the target account. So Exchange will always show the TARGETDOMAIN\User although in fact the source SID has permissions on the mailbox folder. You will also see the same result if you query folder permissions via EWS (e.g. with EWSEditor).

MFCMAPI:

To check which SID is really behind the permission, you can use MFCMAPI to access the mailbox.

  1. Create a new profile for mailbox and disable Exchange Cached Mode.
  2. Start MFCMAPI
  3. Click Session->Logon and choose the profile you’ve created in step 1.
  4. Double Click the Mailbox entry and now navigate to the folder for which you want to display the permissions.
  5. On the right side, now double click PR_NT_SECURITY_DESCRIPTOR

In the Smart View, you can see which SID is really behind the Access Control Entry.

Exchange 2010: Supposed Misbehavior of Outlook 2010 when proposing “New Time” in meeting requests

Posted on by rkmigblog

Created by Ingo G. from Germany:

The other day some colleagues complained about a misbehaviour of Outlook. It was reported if someone propose a new time for a meeting request, the meeting organizer is not able to accept the proposal. It looks like this:

image-a

This looked quite interesting. It took me a while to find the root cause for this:

As this feature is completely handled by Outlook, first I thought the default configuration of Outlook was causing the issue. So I had a look:

File->Options->Calendar

image-b

But it was set like you can see here in screenshot. As I couldn’t reproduce this behavior, I decided to have a closer look what’s going on. So I stand behind the users to see each step:

• the organizer sent a normal meeting request

• the attendee proposed the new time as follows:

He opened the meeting request and clicked on “Propose New Time”

image-c

When you do so, you will get a menu and can choose. Of course it was choosen “Decline and Propose New Time”

Note: When you choose “Decline and Propose New Time” Outlook will first decline the meeting request and then create a new one, which will be send back to the organizer. This is by design! 

Quest Migration Manager for Exchange® and Exchange 2010 Calendar Repair Assistant (CRA) – there can be only one!

Posted on by rkmigblog

The CRA

Microsoft Exchange 2010 ships with a very helpful utility that helps to check the consistent state of meeting invitations and responses and can repair broken relationships – the Exchange 2010 Calendar Repair Assistant (CRA). For example: John organizes a meeting and sends meeting invitations by Outlook. Jack accepts the meeting, which is registered in John’s mailbox calendar (as meeting acceptance) and in Jack’s mailbox calendar as appointment. Maybe somehow Jack did a misconfiguration with his handheld and lots of appointments are missing. The CRA would detect, that Jack is listed as participant of John’s meeting and would create the missing item appointment in Jack’s mailbox calendar to bring back the proper meeting organizer – participant relationship.

You can find a deep-dive explanation of Exchange 2010 CRA here:

http://blogs.technet.com/b/exchange/archive/2013/01/17/exchange-2010-calendar-repair-assistant.aspx

The Problem

Although the CRA is a pretty helpful tool and saves many people from missing meetings because of missing reminders of calendar items, we cannot recommend to use it during an Inter Org Exchange Migration with a running Quest Calendar Synchronization (either uni-directional or bi-directional). We have seen various cases where calendar items in the target mailbox have been duplicated while the duplicates did not exist in the source mailbox. Checking the items’ creator, one can see that one item was created by the calendar sync agent, the other one by the service account of the RCA.

How come? Let’s think of the following example:

Exchange Inter-Org migration from Exchange 2007 to Exchange 2010 SP2 by using Quest Migration Manager for Exchange. For all user to migrate, we have a source mailbox in 2007 and a target mailbox in 2010. The Quest mail and calendar synchronization synchronizes mail content and calendar data between the corresponding mailbox pairs.

John is a user who was already migrated and switched to Exchange 2010 environment. Jack is a user who is still active on his Exchange 2007 mailbox – nevertheless his Exchange 2010 mailbox is already present and filled by the Quest mail and calendar sync.

John creates a meeting in Outlook, invites Jack to the meeting and sends a meeting request. The meeting request is send as an Outlook message into the Exchange 2010 system. Since the Exchange 2010 mailbox of Jack is present but not active, the targetaddress redirection in the Active Directory object of Jack in the target domain will redirect the message to the Exchange 2007 mailbox of Jack. Jack recognizes the meeting invitation mail in his Outlook Inbox and accepts. A system message is now sent back to John who will see now Jack as confirmed meeting participant. In the same step a new appointment is created in Jack’s calendar to reflect the scheduled meeting. So far business as usual. Please note, that at this time, the calendar of John in Exchange 2010 shows that Jack has accepted, the calendar of Jack in 2007 shows the meeting as appointment, but the calendar of Jack in Exchange 2010 is “empty” due to the “nature” of calendar synchronization latency.

Taking a look onto the Exchange 2010 system only at this moment – we will find an inconsistent state between meeting organizer and participant’s calendar data.

The Race CSA vs. CRA

And now it’s like 2 fire brigades trying to fight the fire at same time – the race begins. Quest calendar synchronization Agent (CSA) detected a change in the source Exchange 2007 mailbox of Jack, picks up the change and transfers to the new mailbox which takes time depending on number of mailboxes, synchronization interval etc. At the same time the CRA running in Exchange 2010 environment for good reasons detects the described inconsistency and tries to correct it by creating the “missing” item from scratch. Whenever the CRA is faster we will see most probably duplicates. The CRA is always running per mailbox server while the Quest Calendar Sync can be tuned to run with multiple instances per mailbox server.

Recommendation

RECOMMENDATION: Disable the MS Exchange CRA for the duration of migration.

Dell/Quest support is reporting the issue in their Knowledge Base in brief:

https://support.quest.com/SolutionDetail.aspx?id=SOL102260&pr=Migration Manager for Exchange

They also list some possibilities to fix the duplicates:

Workaround 1 (for switched mailboxes)

 1. Export the problematic calendar to a PST file with the “Do not export duplicate items” option.

 2. Delete all calendar items.

 3. Re-import the calendar items from the PST.

 Workaround 2 (for non-switched mailboxes)

 1. Disconnect the problematic target mailbox.

 2. Purge it from the mailbox database (to prevent tombstone issues).

 3. Run a full AD re-sync to recreate the target mailbox. Alternatively, perform a “Proxy Toggle” (no need for a full re-sync):

  – Add a dummy address to the source mailbox (ex. <alias>@delete.me)

  – Let DSA sync the change during a normal delta sync cycle, which should recreate the mailbox

  – Remove the dummy address from the source mailbox

 4. Run a calendar sync