Quest Migration Manager for Exchange®: Issues with emwmta.dat file growing to the limit of 2 GB – “cannot open database” error

In some of the large scale migrations (or even in smaller ones when running few mail target agents) we recognize the following issue: Mail Target Agent slows down dramatically and does not seem to process *.pst files completely any further.
The characteristic message in the MTA log file is a frequent logging of the error 3049. You see a normal logon to target mailbox, then the synchronization of a message deletion and then when MTA tries to write to the tracking database it fails.

19.02.2013 21:23:55       CaeMAPISession::OpenPST            TraceMsg             0             Trying to open store.
19.02.2013 21:23:55       CStore::LogInfo   Informational      4905      InfoStore is ‘Mailbox – <DISPLAYNAME>’, locale ‘1033’, codepage ‘1252’, EntryId ‘0000000
[…]
19.02.2013 21:23:55       SyncMachine::Init             TraceMsg             2203      Locking tracking database ‘d:\emwmta.dat’.
[…]
19.02.2013 21:24:00       MailImport::ImportDeletedMessages         TraceMsg             1406               Synchronizing deleted messages: 1 message(s) will be deleted.
19.02.2013 21:24:00       SyncMachine::CreateMessage       Error      3049     
Cannot open database ”.  It may not be a database that your application recognizes, or the file may be corrupt.. ExchangeId:

We investigated the so called tracking database, an MS Access style agent database of the Quest agent, which can be found in the root drive of the agent installation location. There is a *.dat file for every agent. In our scenario, only the *.dat files of the Mail Target Agents gave us headaches.
emwmta

We could identify that the error is shown in the log files whenever the tracking database has reached the size of 2.097.152 KB, which exactly gives 2 GB. And 2 GB is the limitation of Access databases, see discussions here.Now the important question was how to fix this. Support came up with the idea of re-installing agents, but did not want us to delete the *.dat file, which is not helpful when re-installing the agents. Apart from this, repairing the agent deletes your files in the queue.In the end we ran through a row of lab tests and decided to take the risk of deleting the file and starting with a new one for the sake of moving on with the sync. At least for our mail target agents we can confirm that this did the trick without impacting the sync quality.Please contact Dell Support before moving any steps in this direction.

Quest Migration Manager for Exchange® – Pimp your QMM calendar sync power

In one of our previous posts [http://wp.me/p300CJ-15] we showed how important the calendar synchonization is in your Inter-Org Exchange migration project when using Quest Migration Manager for Exchange. The Quest approach runs 2 mailboxes for every user (1 in source, 1 in target organization) which enables a long coexistence but also requires a fast running calendar synchronization between those mailboxes for actual Free/Busy information. As less mailboxes per calendar sync agent instance as faster your calendar sync will run. That is a simple truth. But it is not the only one.

When we started to design our Quest Mail and Calendar Sync Agent setup in our large scale project, the customer was very generous and spent us 25 virtual machines with 2 processor cores each. We installed 9 calendar sync agent instances (QMM 8.8) on each of the servers. The customer still had some resources left and spent us another 2 cores for each virtual machines. Running this “QMM server farm” we expected to meet and even exceed the customer’s expectation of 20 minutes calendar sync latencies when bringing over a new calendar entry. However, the performance was not like expected.
Investigation on the servers gave us the following insight on processor usage:

cal_sync_instances

You can see that the first 2 cores are heavily used while the other 2 cores are in “idle mode”. Some experts started to create a powershell script to set processor affinity to the QMM calendar sync agent (CSA) processes which you can find here. set_proc_affinity

But sometimes things are simpler than they look at first sight. In every Quest mail or calendar sync agent folder you can find an RCmd.ini file. The RCmd.ini file contains the string “ProcessorIndex” which defines the processor instance to be used by the agent. When we parsed the files we recognized the distribution as shown in red column. Only the cores 0 and 1 were defined in the RCmd.ini, but not 2 and 3.

instances

We modified the RCmd.ini files of all calendar sync agent instances like shown in the green column. A new investigation on the server presented a different picture of a well-balanced processor usage:

cal_sync_instances_d

With this insight it was a straight-forward task to create a powershell script that parsed through all agent instances on all QMM agent hosts to adjust the ProcessorIndex settings in the RCmd.ini files.

Good to know how things work.

Exchange 2010: Supposed Misbehavior of Outlook 2010 when proposing “New Time” in meeting requests

Created by Ingo G. from Germany:

The other day some colleagues complained about a misbehaviour of Outlook. It was reported if someone propose a new time for a meeting request, the meeting organizer is not able to accept the proposal. It looks like this:

image-a

This looked quite interesting. It took me a while to find the root cause for this:

As this feature is completely handled by Outlook, first I thought the default configuration of Outlook was causing the issue. So I had a look:

File->Options->Calendar

image-b

But it was set like you can see here in screenshot. As I couldn’t reproduce this behavior, I decided to have a closer look what’s going on. So I stand behind the users to see each step:

• the organizer sent a normal meeting request

• the attendee proposed the new time as follows:

He opened the meeting request and clicked on “Propose New Time”

image-c

When you do so, you will get a menu and can choose. Of course it was choosen “Decline and Propose New Time”

Note: When you choose “Decline and Propose New Time” Outlook will first decline the meeting request and then create a new one, which will be send back to the organizer. This is by design! 

Coexistence Manager for Notes – new release 3.3

Dell/Quest Software released a new version of Coexistence Manager for Notes 3.3. Coexistence Manager for Notes provides coexistence between Exchange and Notes E-Mail Systems and eases migration activities in large scale infrastructures.

The release 3.3 is compatible with Exchange 2013, Exchange 2010 SP3 and Outlook 2013 and provides enhanced iOS Support.

Quest Migration Manager for Exchange® and Exchange 2010 Calendar Repair Assistant (CRA) – there can be only one!

The CRA

Microsoft Exchange 2010 ships with a very helpful utility that helps to check the consistent state of meeting invitations and responses and can repair broken relationships – the Exchange 2010 Calendar Repair Assistant (CRA). For example: John organizes a meeting and sends meeting invitations by Outlook. Jack accepts the meeting, which is registered in John’s mailbox calendar (as meeting acceptance) and in Jack’s mailbox calendar as appointment. Maybe somehow Jack did a misconfiguration with his handheld and lots of appointments are missing. The CRA would detect, that Jack is listed as participant of John’s meeting and would create the missing item appointment in Jack’s mailbox calendar to bring back the proper meeting organizer – participant relationship.

You can find a deep-dive explanation of Exchange 2010 CRA here:

http://blogs.technet.com/b/exchange/archive/2013/01/17/exchange-2010-calendar-repair-assistant.aspx

The Problem

Although the CRA is a pretty helpful tool and saves many people from missing meetings because of missing reminders of calendar items, we cannot recommend to use it during an Inter Org Exchange Migration with a running Quest Calendar Synchronization (either uni-directional or bi-directional). We have seen various cases where calendar items in the target mailbox have been duplicated while the duplicates did not exist in the source mailbox. Checking the items’ creator, one can see that one item was created by the calendar sync agent, the other one by the service account of the RCA.

How come? Let’s think of the following example:

Exchange Inter-Org migration from Exchange 2007 to Exchange 2010 SP2 by using Quest Migration Manager for Exchange. For all user to migrate, we have a source mailbox in 2007 and a target mailbox in 2010. The Quest mail and calendar synchronization synchronizes mail content and calendar data between the corresponding mailbox pairs.

John is a user who was already migrated and switched to Exchange 2010 environment. Jack is a user who is still active on his Exchange 2007 mailbox – nevertheless his Exchange 2010 mailbox is already present and filled by the Quest mail and calendar sync.

John creates a meeting in Outlook, invites Jack to the meeting and sends a meeting request. The meeting request is send as an Outlook message into the Exchange 2010 system. Since the Exchange 2010 mailbox of Jack is present but not active, the targetaddress redirection in the Active Directory object of Jack in the target domain will redirect the message to the Exchange 2007 mailbox of Jack. Jack recognizes the meeting invitation mail in his Outlook Inbox and accepts. A system message is now sent back to John who will see now Jack as confirmed meeting participant. In the same step a new appointment is created in Jack’s calendar to reflect the scheduled meeting. So far business as usual. Please note, that at this time, the calendar of John in Exchange 2010 shows that Jack has accepted, the calendar of Jack in 2007 shows the meeting as appointment, but the calendar of Jack in Exchange 2010 is “empty” due to the “nature” of calendar synchronization latency.

Taking a look onto the Exchange 2010 system only at this moment – we will find an inconsistent state between meeting organizer and participant’s calendar data.

The Race CSA vs. CRA

And now it’s like 2 fire brigades trying to fight the fire at same time – the race begins. Quest calendar synchronization Agent (CSA) detected a change in the source Exchange 2007 mailbox of Jack, picks up the change and transfers to the new mailbox which takes time depending on number of mailboxes, synchronization interval etc. At the same time the CRA running in Exchange 2010 environment for good reasons detects the described inconsistency and tries to correct it by creating the “missing” item from scratch. Whenever the CRA is faster we will see most probably duplicates. The CRA is always running per mailbox server while the Quest Calendar Sync can be tuned to run with multiple instances per mailbox server.

Recommendation

RECOMMENDATION: Disable the MS Exchange CRA for the duration of migration.

Dell/Quest support is reporting the issue in their Knowledge Base in brief:

https://support.quest.com/SolutionDetail.aspx?id=SOL102260&pr=Migration Manager for Exchange

They also list some possibilities to fix the duplicates:

Workaround 1 (for switched mailboxes)

 1. Export the problematic calendar to a PST file with the “Do not export duplicate items” option.

 2. Delete all calendar items.

 3. Re-import the calendar items from the PST.

 Workaround 2 (for non-switched mailboxes)

 1. Disconnect the problematic target mailbox.

 2. Purge it from the mailbox database (to prevent tombstone issues).

 3. Run a full AD re-sync to recreate the target mailbox. Alternatively, perform a “Proxy Toggle” (no need for a full re-sync):

  – Add a dummy address to the source mailbox (ex. <alias>@delete.me)

  – Let DSA sync the change during a normal delta sync cycle, which should recreate the mailbox

  – Remove the dummy address from the source mailbox

 4. Run a calendar sync