QMM 8.10 error: Agent is not ready to start – SCP not found

We used Quest Migration Manager 8.10 recently in a project at a customer for a combined Active Directory and Exchange migration. Overall target was to integrate a Windows 2003 domain cross forest and cross org into the central AD Forest with several child domains. Since from mail perspective our migration source was Exchange 2007 and our migration target Exchange 2013, we decided to use the Native Move Job option along with the Migration Manager for Exchange Agent (MAgE) services.

Situation:

The customer environment look like the following:
Source Domain in Single Domain Forest with Domain Controllers on Windows 2003 and Exchange 2007 as mail system.
Target Domain was one of several child domains in the central Forest. All domain controllers running Windows 2012 R2 and mail system was Exchange 2013 SP1.
All Exchange 2013 servers had been deployed to root domain which also kept all important system and admin accounts.
To limit complexity in the setup of Quest Migration Manager 8.10, we decided to use a single administrative account from target Forest’s root domain and granted all necessary permissions in the domains to run both, Active Directory and Exchange migration. Only for access to source Exchange 2007 when running the move request, we used an account from source domain with Org Admin permissions.

Native Move Job
Setup for Native Move Job

Installation of Migration Manager 8.10. on a member server in target domain (best practice recommendation) including all cumulative hotfixes went smoothly. After successful Directory Synchronization, we connected to the Exchange source and target Organization and finally deployed 2 Instances of the MAgE agent for native mailbox move jobs on our agent host and console server. Note: For agent hosts Windows 2012 R2 is currently (May 2014) not supported. You have to stay with Windows 2008 R2 here.

Problem:

However, after starting the agent services running with our administrative account , we recognized, that we could not open the log file of the agent in the Log Panel inside the Migration Manager for Exchange GUI. We searched for the log file and found it in “c:\progamdata\quest software\Migration Agent for Exchange\NativeMove directory”.

scp not found
Log snippet from MAgE agent

The log file showed that the agent was not starting to process the migration collection due to missing settings and then went to sleep. The lines of error:

 

Waiting for agent settings: Not found: (&(objectClass=serviceConnectionPoint) …..

Agent is not ready to start. Agent going to sleep at 1 minute.

repeated over and over.

Obviously the agent tried to execute an LDAP query to find a connection point in Active Directory.
Note: Currently QMM 8.10 uses 3 different systems to store configuration data: An ADLDS server, a SQL Server Instance and the Active Directory (ADDS).

Service Connection Point (SCP):

We ran the query which was shown in the log file against the target domain and we could find the Service Connection Point (SCP) immediately in the System container of the domain naming context.

QMM_8.10_SCP

The Service Connection Point consists primarily of the keywords array attribute and the serviceBindingInformation attribute. The QMM MAgE looks for the serviceBindingInformation attribute to get its SQL connection properties. In SQL it will finally find all information to process the collection.
QMM_8.10_SCP_3
We do not know why Developers at Dell Software made this process so complex. However, in our setup the agent could not find the Service Connection Point, because the agent was looking in the domain, where its service account was located and this was the root domain of the forest while the agent host had installed the SCP during installation in the child domain where the computer account was member of.

Solution:

Switching the agent host and agent service account to an account from child domain would have been a solution, but was not in compliance with customer policy to host all system accounts in root domain.
Moving agent host and console to root domain would not have meet best practices and would have interfered running directory synchronization.

So we ended up in giving the agent just what it requested:
We manually created a Service Connection Point in the root domain and copied all serviceBindingInformation values over.

The agent started immediately and worked without errors.

For future design we can only recommend to store Service Connection Point in the Configuration Partition as Exchange and lots of other software. Using the domain naming context will always lead to problems in a big Enterprise environment with Active Directory consisting of multiple domains in a  forest.

 

Moving servers between domains with Powershell v3 add-computer commandlet

Background
Migrating computers as part of an Active Directory migration has 2 aspects. There is an Active Directory object migration as it is with user and group objects. And in addition you have to disjoin the Windows computer from the old domain and join it to the new domain which requires to modify the workstation or server OS.
The most simple way to move a computer between domains is of course to use the GUI and change the domain or workgroup association of the computer in the system property settings manually.
However, this is not a solution for migration projects where you want to move many computers at the same time remotely logging on to the machines interactively.

Requirements
In our actual large scale migration project we have to deal with multiple thousands servers that have to be migrated to the new domain. While the client computers run through an SCCM controlled imaging process, the server domain move is the task of the server admins. Some of the server admins are responsible for large scale test and development environments with hundreds of servers.
To ease the one time task of domain migration for those administrators, our idea was to implement a remote service utility which can migrate servers to the new domain in bulk mode.
While we would operate and maintain the service centrally, the server admins should decide which servers to migrate and when to migrate them. Another requirement was to leave the servers as is without installing QMM agents that could interfere with running applications.

server move automat


Solution

To be as most flexible as can be, we chose the add-computer Powershell commandlet for our scripting solution. (We ended up with a 320 line script to combine multiple modifications during server move). Server owners would place a config file on a share. The script server periodically scans the share for new config files and processes the server names.
While the final script contains multiple functions, the core function with the add-computer commandlet to disjoin/join the Computer can be found here:

function domain_move($compacc,$fqdn) {
$username_joinTarget=”TARGETDOMAIN\SERVICEACCOUNT”
$password_joinTarget=cat“d:\scripts\server_move\JoinTarget.txt”|convertto-securestring
$cred_JoinTarget=new-object -typename System.Management.Automation.PSCredential –argumentlist $username_joinTarget,$password_joinTarget
$username_unjoinSource=”SOURCEDOMAIN\SERVICEACCOUNT”
$password_unjoinSource=cat“d:\scripts\server_move\UnjoinSource.txt”|convertto-securestring
$cred_UnjoinSource=new-object -typename System.Management.Automation.PSCredential -argumentlist $username_unjoinSource,$password_unjoinSource
$Error.clear
Try {Add-Computer -ComputerName $compacc -DomainName $TARGETDOMAIN -Credential $cred_JoinTarget -UnjoinDomainCredential $cred_UnJoinSource -Server $TargetDC -PassThru -Verbose}
Catch {return $false}
Start-Sleep -Seconds 10
Restart-Computer -ComputerName $fqdn
return $true}

The variables $compacc and $fqdn come from the main part of the script as parameters when calling the function.
$compacc=”samaccountname of computer to migrate”
$fqdn=”full qualified domain name of computer to migrate”
The text files with the encrypted passwords are located in the same directory as the executable or ps1 script.

Discussion
The add-computer commandlet was introduced with Powershell 2, but had the restriction that you could only migrate the local computer and needed to run Powershell Remoting to make it useful for other computers.
With Powershell version 3 the parameter –computer was added which allows you to address remote computers for domain move.
Note: This parameter does not rely on Powershell Remoting
Another important parameter for us is –server which defines the target DC that will control the domain join operation. Since we are creating the computer object in the target domain in a specific OU in advance in the same script, it is important not to run into replication delays (trying to join while the computer object was not replicated). The –server parameter which never worked properly in Powershell version 2, did its job for us as long as we used the FQDN of the Domain Controller as syntax.
Note: If you cannot succeed with the domain\DC value for the –server parameter as listed in the Tech Net article, try out the FQDN instead.
A remarkable caveat of the add-computer commandlet for server domain move is the explicit input of domain credentials for disjoin and join actions. Even when the account that is running the script or scheduled task keeps all necessary permissions, you still have to pass account and credentials to make the domain join working. We suppose that this is a WMI restriction and that WMI is underlying code here. Check out the WMI commands below. To overcome this limitation we captured the encrypted passwords in 2 separate text files and only listed the service account in the script code. In the final version the script code was transformed into an exe file by using Powershell Admin Studio by Sapien Technlogies.
The add-computer commandlet also provides a parameter –restart. We cannot recommend to use this parameter, because it might trigger the reboot too fast, which can lead to RPC connection errors after reboot. We recommend to set a sleep time of multiple seconds and trigger a separate restart-computer commandlet which provides you with multiple options and restart dependencies.
We do not use the –path parameter but create the computer account in a separate function.
For a full amount of Parameters please check Tech Net.

Alternatives to the Add-Computer commandlet

Quest Resource Updating Manager
If you have deployed Migration Manager for Active Directory in your migration project, you can create collections for computers that should undergo a domain move. The collections can be filled by import scripts, so that you can achieve a semi-automatic solution.
QMM Resource Updating Manager
While the main purpose of QMM Resource Updating Manager is to prepare the resources (file shares, local groups, registry etc) for the domain move (which either requires to install agents or to deploy vmover scripts), it also has an option to move computers remotely without installing agents.
QMM Resource Updating Manager


NETDOM

Another option is the NETDOM JOIN legacy command which is around since Windows NT 4.
http://technet.microsoft.com/de-de/library/cc772217(v=ws.10).aspx
(To use netdom, you must run the netdom command from an elevated command prompt.)

WMI
Another way is to go WMI native and use the commands that might be underlying of the add-computer commandlet. However, we find WMI a bit “clumsy” for this purpose (we like it easy).
Example:
$currentserver= (gwmi -computername $Computer -class “Win32_ComputerSystem” -Authentication 6)
$currentserver.JoinDomainorWorkGroup($newdomain,$password,$username,$Null,33)

Quest Migration Manager for Active Directory®: QMM vmover’s registry access blocked by security software

In our Exchange and Active Directory migration project we recently deployed a vmover package on a large number of client computers where QMM vmover.exe performs all resource updating locally without stressing the network. The results were quite positive, but after a while the Client protection team of the customer, who is actually running McAffee® security software on all client computers, was complaining about vmover activities. The security software identified vmover as intruder and blocked actions.
They said, that vmover.exe would try to add new keys in the McAffee® agent part of the registry. We could not believe that, but the AccessProtectionLog.txt of McAffee® exactly provided evidence:

14.01.2013 14:49:37
Blocked by Access Protection rule          NT AUTHORITY\SYSTEM
C:\Program Files\Quest\vmover\Vmover.exe
\REGISTRY\MACHINE\SYSTEM\CurrentControlSet\services\McAfeeFramework\Security
Common Standard Protection:
Prevent modification of McAfee Common Management Agent files and settings
Action blocked :
Create

Our settings in vmover.ini did allow vmover to update registry keys, which means Re-ACLing of permissions on registry keys, but we did not have an explanation why vmover would create something outside the user hive when updating user profiles.
Using the ProcessMonitor it was even more obvious that vmover.exe tries to create keys in the registry.

vmover_createregkey

The response from Quest Development came after short time:

What we saw in Process Monitor did not necessarily mean that vmover actually tries to create anything in there, but it’s rather the fact that RegCreateKeyEx function is used to enumerate the registry. There are two functions, one is RegOpenKeyEx and second is RegCreateKeyEx, both can be used to read information from a key, but the later will create a key if it does not exist depending on parameters passed. RegCreateKeyEx is used by Vmover for performance reasons. Also the entire registry is processed when process registry option is selected and all services are enumerated this way in registry when service processing is enabled.

With those arguments we got back to the Client protection team and after spending a beer or two, they agreed to put McAffee® on the Whitelist which solved the access block problem.
Good to know how things work.