QMM 8.10 error: Agent is not ready to start – SCP not found

We used Quest Migration Manager 8.10 recently in a project at a customer for a combined Active Directory and Exchange migration. Overall target was to integrate a Windows 2003 domain cross forest and cross org into the central AD Forest with several child domains. Since from mail perspective our migration source was Exchange 2007 and our migration target Exchange 2013, we decided to use the Native Move Job option along with the Migration Manager for Exchange Agent (MAgE) services.

Situation:

The customer environment look like the following:
Source Domain in Single Domain Forest with Domain Controllers on Windows 2003 and Exchange 2007 as mail system.
Target Domain was one of several child domains in the central Forest. All domain controllers running Windows 2012 R2 and mail system was Exchange 2013 SP1.
All Exchange 2013 servers had been deployed to root domain which also kept all important system and admin accounts.
To limit complexity in the setup of Quest Migration Manager 8.10, we decided to use a single administrative account from target Forest’s root domain and granted all necessary permissions in the domains to run both, Active Directory and Exchange migration. Only for access to source Exchange 2007 when running the move request, we used an account from source domain with Org Admin permissions.

Native Move Job
Setup for Native Move Job

Installation of Migration Manager 8.10. on a member server in target domain (best practice recommendation) including all cumulative hotfixes went smoothly. After successful Directory Synchronization, we connected to the Exchange source and target Organization and finally deployed 2 Instances of the MAgE agent for native mailbox move jobs on our agent host and console server. Note: For agent hosts Windows 2012 R2 is currently (May 2014) not supported. You have to stay with Windows 2008 R2 here.

Problem:

However, after starting the agent services running with our administrative account , we recognized, that we could not open the log file of the agent in the Log Panel inside the Migration Manager for Exchange GUI. We searched for the log file and found it in “c:\progamdata\quest software\Migration Agent for Exchange\NativeMove directory”.

scp not found
Log snippet from MAgE agent

The log file showed that the agent was not starting to process the migration collection due to missing settings and then went to sleep. The lines of error:

 

Waiting for agent settings: Not found: (&(objectClass=serviceConnectionPoint) …..

Agent is not ready to start. Agent going to sleep at 1 minute.

repeated over and over.

Obviously the agent tried to execute an LDAP query to find a connection point in Active Directory.
Note: Currently QMM 8.10 uses 3 different systems to store configuration data: An ADLDS server, a SQL Server Instance and the Active Directory (ADDS).

Service Connection Point (SCP):

We ran the query which was shown in the log file against the target domain and we could find the Service Connection Point (SCP) immediately in the System container of the domain naming context.

QMM_8.10_SCP

The Service Connection Point consists primarily of the keywords array attribute and the serviceBindingInformation attribute. The QMM MAgE looks for the serviceBindingInformation attribute to get its SQL connection properties. In SQL it will finally find all information to process the collection.
QMM_8.10_SCP_3
We do not know why Developers at Dell Software made this process so complex. However, in our setup the agent could not find the Service Connection Point, because the agent was looking in the domain, where its service account was located and this was the root domain of the forest while the agent host had installed the SCP during installation in the child domain where the computer account was member of.

Solution:

Switching the agent host and agent service account to an account from child domain would have been a solution, but was not in compliance with customer policy to host all system accounts in root domain.
Moving agent host and console to root domain would not have meet best practices and would have interfered running directory synchronization.

So we ended up in giving the agent just what it requested:
We manually created a Service Connection Point in the root domain and copied all serviceBindingInformation values over.

The agent started immediately and worked without errors.

For future design we can only recommend to store Service Connection Point in the Configuration Partition as Exchange and lots of other software. Using the domain naming context will always lead to problems in a big Enterprise environment with Active Directory consisting of multiple domains in a  forest.

 

Powershell 5 in Windows Management Framework V5 Preview

Microsoft released a Preview of the Windows Management Framework V5. As in the past, this package ships with the according version of Powershell. Powershell V5 will bring interesting new Features.

Among those are:

  1. OneGet module with a set of comandlets to manage Software packages
  2. Commandlets to manage L2 Layer Network Switches

Find the introduction article for Windows Managagement Framework V5 by Jeffrey Snover here on TechNet.

You can dowload the Preview here.

However, the mixture of Powershell Versions we find at customer Environments will get wider and wider. Same is valid for modules like ActiveDirectory or SnapIns for Exchange. One will need to start with a lot of checks in the beginning of the code when a script is planned to be used universally.

How to write or migrate sidHistory with Powershell (3)

In our large scale Active Directory Cross Forest migration project, we now have migrated already 40.000 user accounts globally. Our self made scripting routine to migrate/write sidHistory into the target accounts turned out to be a robust, reliable part of the process and I feel safe now to share some experiences. We are running it on multiple migration servers around the globe as scheduled task – which you can easily call a “service” as it is running every 5 minutes.
I will write about the whole mechanism of how we automated our large scale Active Directory migration in another blog post, but will concentrate here to share our way of managing the sidHistory part.
As you know already from part 2 of this blog post, we were buidling our code on the examples that MSFT Jiri Formacek published here.

However, 2 main restrictions prevented us from using this code as is:

  1. We wanted to make sure that we really used the Domain Controller with the PDC Emulator role from source domain. Our source environment has 100+ domain controllers and the PDC role is siwtched from one DC to another DC under certain conditions. Therefore to use a fixed name for the PDC role Domain Controller was not acceptable.
  2. Our Active Directory account migration process was fully automated and it was the user who starts his/her migration not us. Therefore the requirement was given, that we only can run sidHistory migration (together with the account activation in target domain) as a continuous background service. Every session based approach would not have helped like we can find it in ADMT or Dell Migration Manager for Active Directory.
    Prepopulating sidHistory on the previously created disabled accounts in target domain was not an option, since Exchange 2010 was giving errors for disabled users with sidHistory of source active users under certain circumstances.

Solutions:
1) This was not a big thing. A small function could do the trick.

function getPDCE($domain) {
$context = new-object System.DirectoryServices.ActiveDirectory.DirectoryContext("domain",$domain)
$PDC=[System.DirectoryServices.ActiveDirectory.Domain]::GetDomain($context).PdcRoleOwner.Name
return $PDC
} 

2) This was not that easy (for us). Running our account migration script as usual – means as scheduled task with admin credentials – did not work for the sidHistory part in it since the credentials of the logged user account were not handed over to the SIDCloner routine.
All the code we could find on Jiri’s page asked for credential information interactively or would need explicit credentials in the script in another way.
Although we are packaging our Powershell Scripts into an .exe file by using Sapien Powershell Studio and could hide the password from simple file editing, putting user name and password into the script was not an acceptable way for us to go.
After testing back and forth, someone cam up with the idea of using the Windows credential manager to work around our deadlock situation.
The script would access the credential manager interface, get the credential information from there and would then pass them to the DsAddSidHistory function.
We created a function to retrieve credentials from Credential Manager store based on a very good script example to be found on Technet here.
While this seemed to be a clever way of achieving our target of having a scheduled user account activation script with sidHistory functionality, we ran into errors again. Retrieving credentials from Credential Manager by script obviously fails, when the script runs with exactly the credentials that you want to retrieve. This was true in our case, because the user account migration script was scheduled with that “big admin” account.

The solution finally was:
The user account migration script was running as a scheduled task with full admin credentials. When it came to migrate (in our project setup: activate) a user account in the target domain, it did not (could not) write sidHistory, but created an input file with username and target DC (the DC closest to the site where the user was had logged in from – remember that the user triggers his/her migration in our project).
On the same migration server a second script was scheduled with a server-local admin account. This script consists of 3 parts. First part is to check if there are new input files. Second part is to retrieve the full admin credentials from Credential manager and passing them to second part. Third part is to migrate sidHistory which succeeds because you have put all parts together for the SIDCloner routine:
PDC Emulator DC for source you have found by query.
Target DC was in file (but you can take every writable you want if replication delay does not matter).
Explicit credentials you get from Credential Manager.
Nowhere in both scripts password information is saved in clear text.

Additional Information

Recovery Manager for Active Directory Forest Edition – 8.5.1 release is out

Dell Software released Version 8.5.1 of Recovery Manager for Active Directory Forest Edition.

The new release ships with multiple new features which were requested by many customers for a Long time:

Building virtual test environments
In the past the clone wizard provided only limited capabilities to build a test lab on base of our production Active Directory. Now the built-in logic of the Forest Recovery is used to create virtual lab to mirror an existing Active Directory.
The program component Active Directory Virtualization Lab works with MS System Center Virtual Machine Manager, VMWare ESX and VMWare vCenter.

Active Directory management capabilities
The tool now helps to manage Global catalog functionality and FSMO roles throughout your Forest’s domain controller infrastructure

Web Interface

The new Web Interface for Recovery Manager for Active Directory allows you to distribute the tool’s frontend much more easily

Support for BitLocker Drive Encryption when recovering Domain Controllers
If BitLocker is enabled on the domain controllers to be recovered, the tool will take care to disable BitLocker before starting recovery and to enable afterwards.

Enhanced Management Shell
Release 8.5.1 also ships with new Powershell commandlets to manage RMAD activities by script:
Expand-RMADBackup extracts the content of a backup file to a location you can select (same functionality as Extract Wizard from GUI)
Remove-RMADUnpackedComponent cleans up unpacked data from former restore operations
Remove-RMADCollectionItem removes items from a specific computer collection.

With Recovery Manager for Active Directory Forest Edition (RMAD FE) 8.5.1 you can restore Domain Controllers running the following OS Versions:

  • Microsoft Windows Server 2012 (including Server Core installation)
  • Microsoft Windows Server 2008 R2 without Service Pack or with Service Pack 1 (including Server Core installation)
  • Microsoft Windows Server 2008 with Service Pack 1 or Service Pack 2 (including Server Core installation)
  • Microsoft Windows Server 2003 R2 without Service Pack or with Service Pack 2
  • Microsoft Windows Server 2003 without Service Pack or with Service Pack 1 or Service Pack 2

You can install Recovery Manager for Active Directory Forest Edition 8.5.1 on the following platforms:

  • Microsoft Windows Server 2012
  • Microsoft Windows 8
  • Microsoft Windows Server 2008 R2 without Service Pack or with Service Pack 1
  • Microsoft Windows Server 2008 with Service Pack 1 or Service Pack 2
  • Microsoft Windows 7 without Service Pack or with Service Pack 1
  • possible but not recommended:
    Microsoft Windows Vista with Service Pack 2
    Microsoft Windows Server 2003 R2 with Service Pack 2
    Microsoft Windows Server 2003 with Service Pack 2
    Microsoft Windows XP with Service Pack 3

You can view all additional Information on the DELL Website here.

If you need assistance in deploying the Software and set up an Active Directory Forest Recovery plan with this tool, please get in contact here.

How to write or migrate sidHistory with Powershell (2)

SIDCloner.dll
When you look in WWW to find a way to write sidHistory attribute, you probably will stop at the marvelous SID Cloner website created by Jiri Formacek, MSFT (http://code.msdn.microsoft.com/windowsdesktop/SIDCloner-add-sIDHistory-831ae24b). Jiri provides a managed class library that implements the SIDCloner class, which we can use in Powershell and any .NET programming code.
The SID Cloner class is built upon native API to migrate sidHistory and therefore uses the DsAddSidHistory function under the hood (http://msdn.microsoft.com/en-us/library/windows/desktop/ms675918(v=vs.85).aspx).
Although we do not need to install ADMT on any machine to run the SID Cloner code, we still have to consider to meet the same requirements for the migration setup as ADMT does. SID Cloner and ADMT come from the same “mothership” DsAddSidHistory.

SidHistory requirements
In brief we need the following prerequisites to be in place before we can start writing sidHistory (http://msdn.microsoft.com/en-us/library/windows/desktop/ms677982(v=vs.85).aspx):

+ a trust relationship must exist between source and target domain
+ source and target domain must not be in the same Active Directory forest
+ for source domain all actions will have the PDC Emulator as target; you cannot bind to another DC than the one with the PDC Emulator role
+ Auditing must be enabled in source domain
+ a domain-local group “domain$$$” must be created in source domain
+ a special registry key must be created on PDC Emulator DC in source domain: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Lsa\TcpipClientSupport
+ Audit Mode must be turned on in each domain and also Account Management auditing of Success/Failure events must turned to on.
+ the Security Identifier we want to transfer must not already exist in any sidHstory attributes of objects in target domain

Permissions:
For running Powershell code based on SID Cloner you do not necessarily need domain admin credentials in target domain. While read permissions on objects in source domain are sufficient (you are reading the “standard” attribute objectSID there), the permissions to modify the object in target domain by writing the sidHistory value requires more:
+ full access permissions to the object (better OU)
+ special permission “migrateSIDHistory” on the Active Directory domain object in target domain
migrateSidHistoryPermission

With all the requirements settled, you are able to migrate sidHistory by using the sample script, that Jiri published on the SID Cloner Website.

However, the most easy way to use the 4 fold overload with SIDCloner did never work in our tests.
Overload with 4 arguments means, you simply define source and target domain, source account from which you want to take the objectSID and target account where you want to write sidHistory.
This approach will probably never succeed because
a) the strict PDC Emulator access is not guaranteed when using a domain call
b) the credentials of the interactively logged user are obviously not passed to the DsAddSidHistory function inside the SID Cloner dll

Let’s have more findings around this Topics in Part 3 of this blog post.

 

How to write (migrate) sidHistory with Powershell (1)

The adventure of sidHistory
I spent quite some hours during the last weeks to create a Powershell script routine that is able to “migrate sidHistory”. Migrate sidHistory in this context means to read the objectSID of a given user or group source object in Active Directory Forest A and write this value into the sidHistory attribute of a selected user or group object in Forest B.
Assuming there is a trust relationship between Forest A and Forest B and sidFiltering is disabled, user B from Forest B who has the sidHistory attribute filled with the SID of the user A from Forest A will have access to the same resources in Forest A like user A himself. The reason for this behavioris found in the fact that the security token of User B after successful logon will contain the SID of user A. From Windows’ token based access strategy, user B is now a user AB as long as we are talking about SID releated resource access.
From those few lines everyone agrees on the statememt that sidHistory functionality can be abused to get access to resources which are restricted for a user by default. In principle, you can add the SID of a given user from Forest A to any user in Forest B. There does not have to be a dedicated relationship like identical samaccountname of the two users (groups) etc. While exactly his functionality helps to ease Inter-Forest Active Directory migrations (and Intra Forest Migrations as well when you take care), it can also be a dangerous thread against your Active Directory security.
However, this not a new finding and Microsoft did well in treating sidHistory as a special attribute. It needs special treatment when you try to clear the values and it needs special treatment when you want to write values into it. I already published 2 posts about deleting sidHistory, see [Link], so we will concentrate here on writing sidHistory.

Writing sidHistory
The most common way in today’s Active Directory migration scenarios is writing sidHistory by using a migration software. Microsoft ships its own migration software called Active Directory Migration Tool (ADMT) which is capable of writing sidHistory. Other vendors like Dell Software’s Migration Manager for Active Directory (formerly known as Quest Migration Manager for Active Directory) provide a similiar functionality with a lot more options and the possibility to write sidHistory in an ongoing Active Directory Synchronization. Up to now Microsoft Forefront Identity Manager cannot help us here out of the box to fill this attribute as part of an Active Directory synchronization.
When you try to put a SID into the sidHistory attribute by using the standard Microsoft administrative tools like the attribute editor from ADUC, you will fail for sure.

errorsidhistory

You will also fail by using Powershell integrated LDAP based write operations for this attribute like set-aduser or set-qaduser.

errorsidHistory_a

We have to dig deeper here to reach our goal of Powershell based writing of sidHistory, which we will do in Part 2 of this blog post.

Coexistence Manager for Notes – new release 3.5.1

Dell Software released a new version of Coexistence Manager for Notes(R). Coexistence Manager for Notes version 3.5.1 provides coexistence between Exchange and Notes E-Mail Systems and eases migration activities in large scale infrastructures.
New features include:
• Expanded Free/Busy support to include Lotus Domino 8.5.3 FP2
• Improved message conversion provides more robust coexistence with Exchange 2003
• More granular control of object names when Directory Connector performs an update
• Improved HTML fidelity for Outlook 2003 users

News for the Exchange Professional (2): High Level Exchange events – Autumn Ignite Summits

Exchange Product Group announced and recommended the Ignite Summit events, which are held in 4 location around the globe:
Hong Kong, 7-10, 2013
Prague, Oct 21-24, 2013
Washington DC, Nov 4-7, 2013
Dubai, Nov 18-21, 2013

The 3 days track of session is designed and delivered by product experts from across Microsoft (Engineers, Technical Writers, Product Managers & Consultants). The over 70 session cover topics from Office, Office 365, SharePoint, Exchange, Project, Yammer and developer content.

The Ingnite Exchange Track comes with this agenda actual:
Day 1 Ignite Keynotes
Exchange Architecture
Exchange Deployment & Coexistence
Storage, High Availability & Site Resilience
Day 2 Exchange Managed Availability
Exchange Server Sizing & Performance
Exchange Server 2013 Virtualization Best Practices
Collaboration with Exchange
Exchange Online Hybrid Migration
Day 3 Archiving, eDiscovery & DLP
Exchange Online Protection Overview
Implementing Exchange Online Protection
Exchange Tips, Tricks and Troubleshooting

Microsoft Partners register here.
Not Microsoft Partners register here.

ignite