Exchange Migration to Office 365 can change behind the scenes – wave 15 rolls in

In these days Microsoft is moving from wave 14 to wave 15 for Office 365 cloud installations. This means a service transition from Exchange 2010 Server backend to Exchange 2013 Server backend, Lync 2010 to Lync 2013 and other cool updates.

However, if you have travelled a long way to get a smooth migration setup (with all the back and forth of finding the right strategy and technical conditions) Microsoft can make you a big surprise by changing your Exchange target infrastructure. You started your migration project with Exchange 2010 in the cloud as target system and you end up with Exchange 2013.

This is “by design” when moving services to the cloud – as MVP Sean McNeill stated in his post [http://office365evangelist.com/?p=938]:
This is an important questions because with a move to the cloud, the company give up some control on when, and even if, you will go through an upgrade of the service. The company now relies on the Service Provider, Microsoft in this case, to handle the upgrade and the cadence of the upgrades. This needs to be fully understood and accepted by a company moving to the Cloud.”

To mitigate the risk of forcing the customers to update in times where it is just neither “comfortable” nor “amusable” – as it might be in the middle of an Exchange migration project – Microsoft offers to postpone the update one time. The Office 365 admins receive a notification e-mail which announces the update schedule. From that information the customer has 3 weeks to decide that he better postpone or let Microsoft execute. When he decides to postpone, Microsoft will not start the update for the next 2 months. The timespan to complete the wave 15 upgrade is end of 2013 latest, which means your upgrade cannot be later than this deadline.
http://community.office365.com/en-us/wikis/upgrade/what-to-expect-during-the-service-upgrade.aspx

For more information check the Office 365 Upgrade Center: http://community.office365.com/en-us/wikis/upgrade/office-365-service-upgrade-center-for-enterprise.aspx

Dell/Quest Software seems to recognize first problems in running migration projects and recommends to postpone the Office 365 tenant upgrade by contacting Microsoft.

https://support.quest.com/SolutionDetail.aspx?id=SOL105116&pr=Notes+Migrator+for+Exchange

Quest Migration Manager for Exchange®: 10 commendments for EMWProf usage

Updating Outlook profiles after completing the back end mailbox switch is required to connect the user to the target mailbox.
Quest Migration Manager for Exchange ships with the CPUU (Client Profile Updating Utility), aka EMWProf, to streamline and automate this task. Although the configuration of a starter script is easy, we usually end up with an educated script that does a lot around the EMWProf procedure to avoid issues and to prepare for other services like Lync and Exchange UM.
Based on the experience of several long time projects we recommend to have a look into the following aspects.

10 things to know about Outlook profile updating with EMWProf (CPUU)

 

  1. Check if Outlook 2010 is configured with multiple accounts. Since there is a restriction with CPUU, users with multiple Exchange accounts in a single Outlook 2010 profile have to reconfigure Outlook or need to be supported after migration.
  2. Running EMWProf with administrative credentials can save a lot of access problems when switching profiles. Passwords are encrypted as a feature of the CPUU and this works fine.
  3. Running into the EMWProf script each time when a user logs on, creates too much impact. You can create a group, where membership is given after backend mailbox switch. You can create a check in the EMWProf script that checks whether EMWProf did run before on the client and prevents additional starting of the EMWProf procedure.
  4. Disable Windows Search. In many cases, Windows Search locked the ost file because of indexing and EMWProf could not rename it, what EMWProf caused to fail. You cannot stop the Windows Search service because it restarts by itself. Disable and enable afterwards. With Lync client installed and integrated with Exchange/Outlook, several Lync processes are active and block EMWProf from working properly.
  5. If the user has multiple profiles configured in Outlook 2010, EMWProf tries to process all of them and will create a return code that is not unique. You should first scan the registry for Outlook profiles in CPUU script and then run dedicated EMWProf for each profile, to get the return code. EMWProf will also send a message for each profile separately, which makes it easy to differ between good and bad, important and unimportant.
  6. Deployment of EMWProf via logon script is not enough in nowadays. Many laptops stay in Hibernate only and are unlocked w/o domain logon. As a fallback, a link in the Goodbye message should point to the EMWProf script and can be executed by the user. More advanced solutions distribute the EMWProf binaries via software distribution and the EMWProf script checks first, whether the binaries exist locally and if not, it pulls them from a remote share before executing. This helps a lot in small bandwidth scenarios. Even more educated solutions use a migration database where the EMWProf script can upload the results of the client side part of the mailbox migration.
  7. For terminal server use you should configure a specific EMWProf script. Processing of offline profiles is not necessary there as an example. Make the script as slim as possible and it will work fast and with less issues.
  8. Sometimes localized Goodbye/Welcome messages sent by QMM switch procedure are important. You can change the message text per Mail Source Agent. If you have regional based setup, you can send localized language mails easily by feeding the MSAs with a specific message text.
  9. Do you need a Welcome message sent by QMM? It has advantages and disadvantages. If mailbox switch and Outlook switch was successful, why confusing the user with more technical explanations and notifications.
  10. Heaven and hell of transferring read/unread status of items. QMM mail agents sync read/unread status and CPUU does as well to fill the last gap. This is very helpful to make the target mailbox experience for the user close to the “zero impact” (Quest language!) idea. However, this feature can turn into black, when the user starts to work for weeks with his Laptop and his new mailbox, then goes back to his Desktop Computer, is still connected to his old mailbox and executes EMWProf again. It does what it should and will make the items in target look like in the source mailbox. Be careful, we have seen assistants being very set up when realizing that they have 120 unread messages in the inbox from past weeks [again]. For the very same reason we do not use the SwitchRESMBX utility in situations where people work with passive (not yet switched) mailboxes in target.

Dell Quest Migration Tools: Readiness for Windows Server 2012 and Exchange 2013

Actually we can see a good market response to the new Microsoft Server flagships, Windows Server 2012 and Exchange 2013.
Migration projects are still ahead and probably will not die in 2013.

The following table shows the readiness of the Quest migration tool suite  and answers the question whether the tool can be installed on a Windows Server 2012, whether it can migrate Active Directory to Windows 2012 and mail systems to Exchange 2013.

Microsoft did not release a new version of ADMT yet (the most actual version is still 3.2), that is fully compliant with Windows 2012 functional mode domains, nor can you install ADMT 3.2 on a Windows Server 2012 member server. Actually, a migration from a Windows 2008 R2 domain to a Windows 2012 domain with 2012 functional level can neither be achieved by using the native tool (ADMT) nor Quest Migration Manager for Active Directory.

Active Directory Product Version Tool installation on Windows Server 2012? Can backup/restore AD data on Windows Server 2012 DCs? OR Can migrate data to Windows 2012 DCs?
Recovery Manager for Active Directory Forest Edition 8.2.1 yes yes
Recovery Manager for Active Directory 8.2.1 yes yes
Migration Manager for Active Directory 8.9 No no
       
Migration Product Version Tool installation on Windows Server 2012? Can migrate data to Exchange 2013? Office 365?
Migration Manager for Exchange 8.9 no no/yes
Migration Manager for Exchange IntraOrg Edition 1.0.1 no yes
Notes Migrator for Exchange 4.6.1 no (no Windows 8 admin workstation) yes/yes
Coexistence Manager for Notes 3.4 no yes/yes
Groupwise Migrator for Exchange 4.2 no (no Windows 8 admin workstation) yes/yes

 

Quest Migration Manager for Exchange®: Issues with emwmta.dat file growing to the limit of 2 GB – “cannot open database” error

In some of the large scale migrations (or even in smaller ones when running few mail target agents) we recognize the following issue: Mail Target Agent slows down dramatically and does not seem to process *.pst files completely any further.
The characteristic message in the MTA log file is a frequent logging of the error 3049. You see a normal logon to target mailbox, then the synchronization of a message deletion and then when MTA tries to write to the tracking database it fails.

19.02.2013 21:23:55       CaeMAPISession::OpenPST            TraceMsg             0             Trying to open store.
19.02.2013 21:23:55       CStore::LogInfo   Informational      4905      InfoStore is ‘Mailbox – <DISPLAYNAME>’, locale ‘1033’, codepage ‘1252’, EntryId ‘0000000
[…]
19.02.2013 21:23:55       SyncMachine::Init             TraceMsg             2203      Locking tracking database ‘d:\emwmta.dat’.
[…]
19.02.2013 21:24:00       MailImport::ImportDeletedMessages         TraceMsg             1406               Synchronizing deleted messages: 1 message(s) will be deleted.
19.02.2013 21:24:00       SyncMachine::CreateMessage       Error      3049     
Cannot open database ”.  It may not be a database that your application recognizes, or the file may be corrupt.. ExchangeId:

We investigated the so called tracking database, an MS Access style agent database of the Quest agent, which can be found in the root drive of the agent installation location. There is a *.dat file for every agent. In our scenario, only the *.dat files of the Mail Target Agents gave us headaches.
emwmta

We could identify that the error is shown in the log files whenever the tracking database has reached the size of 2.097.152 KB, which exactly gives 2 GB. And 2 GB is the limitation of Access databases, see discussions here.Now the important question was how to fix this. Support came up with the idea of re-installing agents, but did not want us to delete the *.dat file, which is not helpful when re-installing the agents. Apart from this, repairing the agent deletes your files in the queue.In the end we ran through a row of lab tests and decided to take the risk of deleting the file and starting with a new one for the sake of moving on with the sync. At least for our mail target agents we can confirm that this did the trick without impacting the sync quality.Please contact Dell Support before moving any steps in this direction.

Quest Migration Manager for Exchange® – Pimp your QMM calendar sync power

In one of our previous posts [http://wp.me/p300CJ-15] we showed how important the calendar synchonization is in your Inter-Org Exchange migration project when using Quest Migration Manager for Exchange. The Quest approach runs 2 mailboxes for every user (1 in source, 1 in target organization) which enables a long coexistence but also requires a fast running calendar synchronization between those mailboxes for actual Free/Busy information. As less mailboxes per calendar sync agent instance as faster your calendar sync will run. That is a simple truth. But it is not the only one.

When we started to design our Quest Mail and Calendar Sync Agent setup in our large scale project, the customer was very generous and spent us 25 virtual machines with 2 processor cores each. We installed 9 calendar sync agent instances (QMM 8.8) on each of the servers. The customer still had some resources left and spent us another 2 cores for each virtual machines. Running this “QMM server farm” we expected to meet and even exceed the customer’s expectation of 20 minutes calendar sync latencies when bringing over a new calendar entry. However, the performance was not like expected.
Investigation on the servers gave us the following insight on processor usage:

cal_sync_instances

You can see that the first 2 cores are heavily used while the other 2 cores are in “idle mode”. Some experts started to create a powershell script to set processor affinity to the QMM calendar sync agent (CSA) processes which you can find here. set_proc_affinity

But sometimes things are simpler than they look at first sight. In every Quest mail or calendar sync agent folder you can find an RCmd.ini file. The RCmd.ini file contains the string “ProcessorIndex” which defines the processor instance to be used by the agent. When we parsed the files we recognized the distribution as shown in red column. Only the cores 0 and 1 were defined in the RCmd.ini, but not 2 and 3.

instances

We modified the RCmd.ini files of all calendar sync agent instances like shown in the green column. A new investigation on the server presented a different picture of a well-balanced processor usage:

cal_sync_instances_d

With this insight it was a straight-forward task to create a powershell script that parsed through all agent instances on all QMM agent hosts to adjust the ProcessorIndex settings in the RCmd.ini files.

Good to know how things work.