Quest Migration Manager for Exchange® – Pimp your QMM calendar sync power

In one of our previous posts [http://wp.me/p300CJ-15] we showed how important the calendar synchonization is in your Inter-Org Exchange migration project when using Quest Migration Manager for Exchange. The Quest approach runs 2 mailboxes for every user (1 in source, 1 in target organization) which enables a long coexistence but also requires a fast running calendar synchronization between those mailboxes for actual Free/Busy information. As less mailboxes per calendar sync agent instance as faster your calendar sync will run. That is a simple truth. But it is not the only one.

When we started to design our Quest Mail and Calendar Sync Agent setup in our large scale project, the customer was very generous and spent us 25 virtual machines with 2 processor cores each. We installed 9 calendar sync agent instances (QMM 8.8) on each of the servers. The customer still had some resources left and spent us another 2 cores for each virtual machines. Running this “QMM server farm” we expected to meet and even exceed the customer’s expectation of 20 minutes calendar sync latencies when bringing over a new calendar entry. However, the performance was not like expected.
Investigation on the servers gave us the following insight on processor usage:

cal_sync_instances

You can see that the first 2 cores are heavily used while the other 2 cores are in “idle mode”. Some experts started to create a powershell script to set processor affinity to the QMM calendar sync agent (CSA) processes which you can find here. set_proc_affinity

But sometimes things are simpler than they look at first sight. In every Quest mail or calendar sync agent folder you can find an RCmd.ini file. The RCmd.ini file contains the string “ProcessorIndex” which defines the processor instance to be used by the agent. When we parsed the files we recognized the distribution as shown in red column. Only the cores 0 and 1 were defined in the RCmd.ini, but not 2 and 3.

instances

We modified the RCmd.ini files of all calendar sync agent instances like shown in the green column. A new investigation on the server presented a different picture of a well-balanced processor usage:

cal_sync_instances_d

With this insight it was a straight-forward task to create a powershell script that parsed through all agent instances on all QMM agent hosts to adjust the ProcessorIndex settings in the RCmd.ini files.

Good to know how things work.

Quest Migration Manager for Exchange® and Exchange 2010 Calendar Repair Assistant (CRA) – there can be only one!

The CRA

Microsoft Exchange 2010 ships with a very helpful utility that helps to check the consistent state of meeting invitations and responses and can repair broken relationships – the Exchange 2010 Calendar Repair Assistant (CRA). For example: John organizes a meeting and sends meeting invitations by Outlook. Jack accepts the meeting, which is registered in John’s mailbox calendar (as meeting acceptance) and in Jack’s mailbox calendar as appointment. Maybe somehow Jack did a misconfiguration with his handheld and lots of appointments are missing. The CRA would detect, that Jack is listed as participant of John’s meeting and would create the missing item appointment in Jack’s mailbox calendar to bring back the proper meeting organizer – participant relationship.

You can find a deep-dive explanation of Exchange 2010 CRA here:

http://blogs.technet.com/b/exchange/archive/2013/01/17/exchange-2010-calendar-repair-assistant.aspx

The Problem

Although the CRA is a pretty helpful tool and saves many people from missing meetings because of missing reminders of calendar items, we cannot recommend to use it during an Inter Org Exchange Migration with a running Quest Calendar Synchronization (either uni-directional or bi-directional). We have seen various cases where calendar items in the target mailbox have been duplicated while the duplicates did not exist in the source mailbox. Checking the items’ creator, one can see that one item was created by the calendar sync agent, the other one by the service account of the RCA.

How come? Let’s think of the following example:

Exchange Inter-Org migration from Exchange 2007 to Exchange 2010 SP2 by using Quest Migration Manager for Exchange. For all user to migrate, we have a source mailbox in 2007 and a target mailbox in 2010. The Quest mail and calendar synchronization synchronizes mail content and calendar data between the corresponding mailbox pairs.

John is a user who was already migrated and switched to Exchange 2010 environment. Jack is a user who is still active on his Exchange 2007 mailbox – nevertheless his Exchange 2010 mailbox is already present and filled by the Quest mail and calendar sync.

John creates a meeting in Outlook, invites Jack to the meeting and sends a meeting request. The meeting request is send as an Outlook message into the Exchange 2010 system. Since the Exchange 2010 mailbox of Jack is present but not active, the targetaddress redirection in the Active Directory object of Jack in the target domain will redirect the message to the Exchange 2007 mailbox of Jack. Jack recognizes the meeting invitation mail in his Outlook Inbox and accepts. A system message is now sent back to John who will see now Jack as confirmed meeting participant. In the same step a new appointment is created in Jack’s calendar to reflect the scheduled meeting. So far business as usual. Please note, that at this time, the calendar of John in Exchange 2010 shows that Jack has accepted, the calendar of Jack in 2007 shows the meeting as appointment, but the calendar of Jack in Exchange 2010 is “empty” due to the “nature” of calendar synchronization latency.

Taking a look onto the Exchange 2010 system only at this moment – we will find an inconsistent state between meeting organizer and participant’s calendar data.

The Race CSA vs. CRA

And now it’s like 2 fire brigades trying to fight the fire at same time – the race begins. Quest calendar synchronization Agent (CSA) detected a change in the source Exchange 2007 mailbox of Jack, picks up the change and transfers to the new mailbox which takes time depending on number of mailboxes, synchronization interval etc. At the same time the CRA running in Exchange 2010 environment for good reasons detects the described inconsistency and tries to correct it by creating the “missing” item from scratch. Whenever the CRA is faster we will see most probably duplicates. The CRA is always running per mailbox server while the Quest Calendar Sync can be tuned to run with multiple instances per mailbox server.

Recommendation

RECOMMENDATION: Disable the MS Exchange CRA for the duration of migration.

Dell/Quest support is reporting the issue in their Knowledge Base in brief:

https://support.quest.com/SolutionDetail.aspx?id=SOL102260&pr=Migration Manager for Exchange

They also list some possibilities to fix the duplicates:

Workaround 1 (for switched mailboxes)

 1. Export the problematic calendar to a PST file with the “Do not export duplicate items” option.

 2. Delete all calendar items.

 3. Re-import the calendar items from the PST.

 Workaround 2 (for non-switched mailboxes)

 1. Disconnect the problematic target mailbox.

 2. Purge it from the mailbox database (to prevent tombstone issues).

 3. Run a full AD re-sync to recreate the target mailbox. Alternatively, perform a “Proxy Toggle” (no need for a full re-sync):

  – Add a dummy address to the source mailbox (ex. <alias>@delete.me)

  – Let DSA sync the change during a normal delta sync cycle, which should recreate the mailbox

  – Remove the dummy address from the source mailbox

 4. Run a calendar sync