How to write or migrate sidHistory with Powershell (3)

In our large scale Active Directory Cross Forest migration project, we now have migrated already 40.000 user accounts globally. Our self made scripting routine to migrate/write sidHistory into the target accounts turned out to be a robust, reliable part of the process and I feel safe now to share some experiences. We are running it on multiple migration servers around the globe as scheduled task – which you can easily call a “service” as it is running every 5 minutes.
I will write about the whole mechanism of how we automated our large scale Active Directory migration in another blog post, but will concentrate here to share our way of managing the sidHistory part.
As you know already from part 2 of this blog post, we were buidling our code on the examples that MSFT Jiri Formacek published here.

However, 2 main restrictions prevented us from using this code as is:

  1. We wanted to make sure that we really used the Domain Controller with the PDC Emulator role from source domain. Our source environment has 100+ domain controllers and the PDC role is siwtched from one DC to another DC under certain conditions. Therefore to use a fixed name for the PDC role Domain Controller was not acceptable.
  2. Our Active Directory account migration process was fully automated and it was the user who starts his/her migration not us. Therefore the requirement was given, that we only can run sidHistory migration (together with the account activation in target domain) as a continuous background service. Every session based approach would not have helped like we can find it in ADMT or Dell Migration Manager for Active Directory.
    Prepopulating sidHistory on the previously created disabled accounts in target domain was not an option, since Exchange 2010 was giving errors for disabled users with sidHistory of source active users under certain circumstances.

Solutions:
1) This was not a big thing. A small function could do the trick.

function getPDCE($domain) {
$context = new-object System.DirectoryServices.ActiveDirectory.DirectoryContext("domain",$domain)
$PDC=[System.DirectoryServices.ActiveDirectory.Domain]::GetDomain($context).PdcRoleOwner.Name
return $PDC
} 

2) This was not that easy (for us). Running our account migration script as usual – means as scheduled task with admin credentials – did not work for the sidHistory part in it since the credentials of the logged user account were not handed over to the SIDCloner routine.
All the code we could find on Jiri’s page asked for credential information interactively or would need explicit credentials in the script in another way.
Although we are packaging our Powershell Scripts into an .exe file by using Sapien Powershell Studio and could hide the password from simple file editing, putting user name and password into the script was not an acceptable way for us to go.
After testing back and forth, someone cam up with the idea of using the Windows credential manager to work around our deadlock situation.
The script would access the credential manager interface, get the credential information from there and would then pass them to the DsAddSidHistory function.
We created a function to retrieve credentials from Credential Manager store based on a very good script example to be found on Technet here.
While this seemed to be a clever way of achieving our target of having a scheduled user account activation script with sidHistory functionality, we ran into errors again. Retrieving credentials from Credential Manager by script obviously fails, when the script runs with exactly the credentials that you want to retrieve. This was true in our case, because the user account migration script was scheduled with that “big admin” account.

The solution finally was:
The user account migration script was running as a scheduled task with full admin credentials. When it came to migrate (in our project setup: activate) a user account in the target domain, it did not (could not) write sidHistory, but created an input file with username and target DC (the DC closest to the site where the user was had logged in from – remember that the user triggers his/her migration in our project).
On the same migration server a second script was scheduled with a server-local admin account. This script consists of 3 parts. First part is to check if there are new input files. Second part is to retrieve the full admin credentials from Credential manager and passing them to second part. Third part is to migrate sidHistory which succeeds because you have put all parts together for the SIDCloner routine:
PDC Emulator DC for source you have found by query.
Target DC was in file (but you can take every writable you want if replication delay does not matter).
Explicit credentials you get from Credential Manager.
Nowhere in both scripts password information is saved in clear text.

Additional Information

Quest Migration Manager for Active Directory – password error when synchronizing user objects – part 2

In part 1 of this post we explained why QMM Directory Sync Agent (DSA) might run into problems when sychronizing user passwords that have been resetted by using administrative credentials to a value which is present in the password history. In this post we will show how we can identify affected user accounts and how we can work around the issue.
As we have learned in the first part, there are 3 good methods to identify the password synchronization errors:

  • QMM AD GUI – failed objects link in the Status page of the Active Directory synchronization
  • QMM Error Reporter Utility – Quest Utility you can download from support site
  • DSA Log File Parsing – you can parse the log files with any good Parser/Scripting engine

Methods of resolving the password synchronization Problem:

1. User changes password
The simplest approach to solve the problem is the user himself – maybe after contacting the user. When the user changes the password of his Acive Directory account the default way (e.g. via CTR+ALT+DEL). Changing the password this way will ensure that the password policy of the domain is enforced (instead of bypassed via admin reset). Assumed that password policies between source and target domain are aligned, Quest Active Directory Synchronization Agent (DSA) will successfully set the new password on the target user account.

2. User is forced to change password
Another method similiar to 1. is to force the user to change the password by setting the “User must change password at next logon” flag. This can be achieved by using ADUC for single users.

user_must_change_password

However, when it comes to mass operations, you can achieve the same goal by setting the attribute “pwdLastSet” to “0” programmatically by using Powershell, VB etc.
Approach 1. and 2. have in common that you have to make sure that users do not call the help line and ask for an admin reset to their “usual” password again.

3. Temporary Fine Grained Password Policy controlled by DSA Parser script
Our customers often complain that they do not like to inform users to change their passwords with messages like “your actual password is not compliant with corporate policies – please change”. Educated users will ask: “How come that you know my password. We have been told, admins do not know users’ passwords …
Well, to workaround this situation, a new approach is possible if your target Active Directory domain is Windows 2008 or higher.
The plan:

  • Increase DSA log file size to make sure you have a full DSA cycle in the log (optional). A full cycle will always work once through the failed objects queue and list the password sync Errors.
  • Create a group in target domain that will contain user objects with password sync error.
  • Create a Fine Grained Password Policy (FGPP) in target domain that contains the same password settings as the default domain policy with the exception of password history which is set to Zero
  • Assign the FGPP to the domain group
  • Create a script that parses the DSA log and fills the group. Empty the group before filling to remove already processed accounts

As you can see, the idea is to allow DSA temporarily and only once for the users with password sync problems to bypass the password history setting. This way the password transfer is possible and a further user migration will not end up in a logon error for these users.

From a security standpoint one can argue that bypassing the password history setting is not advisable. We share this opinion, but we have to recognize that the bypassing already started in the source domain. We neither improve the situation during migration, nor do we make it more worse. But we will prevent user logon errors to target domain later.

A scripting example (example, not more 😉 ) can be found here:

Powershell Script INPUT PWDUSER

Quest Migration Manager for Active Directory – password error when synchronizing user objects – part 1

One of the most useful features of QMM Active Directory synchronization is the ability to synchronize the password of user objects between Active Directory Domains. While Microsoft’s Forefront Identity Manager (FIM) first needs to capture the user password on the Domain Controller when the user actual changes the password, QMM can transport the password hash directly at any time. While FIM needs to install an agent on every Domain Controller to capture the password changes, QMM places an agent “on the fly” on only one dedicated Domain Controller. This can make a big difference in large Active Directory infrastructures.
However, running a long term “ongoing continous” Active Directory synchronization often shows one or many errors like this (snippet from Migration Manager GUI) and fails to update the password to the actual value:

pwdsync_error

The error is a  bit misleading here. QMM is purely transporting the password hash and therefore cannot measure the length of the user password nor can QMM prove the complexity. That means, we have to deal with a password history problem. Assuming we have the same password policies in source domain and target domain and an ongoing password synchronization, this error may never come up, because the password history policy of the source domain would prevent the user to change the password to a value that is still in the password history store.
But there is a second method of changing passwords: The admin reset of passwords. When an admimistrator changes (resets) the password on behalf of a user, he can set the password to a value that is in the password history store. An administrative reset can bypass the password policy. Our investigations showed that several users bypassed the password history policy by calling the help line …
After the administrative reset of the password in source domain, QMM directory synchronization agent (DSA) recognizes a change of the password of the user object and tries to replicate the password hash to the target domain user object. But the DSA has to go “through” the password policy check like a standard user password change which finally results in the password error message above.

You also can find specific error codes in the DSA log file:
05/07/13 08:32:45 (GMT+01:00)     Common AcAdSwitches Error 0xe100004f. Cannot synchronize passwords, source user: “<user name>”, target user: “<user name>” Error 0x8007052d. Unable to update the password. The value provided for the new password does not meet the length, complexity, or history requirements of the domain.

In part 2 of this post, we will show ways to work around the password sync error.

QMM AD – Incorrect Directory Synchronization Agent Matching, Caching and Repairing

QMM AD stores matching data in ADLDS and in Cache DB

From our experience, the Directory Service Agent component (DSA) from Quest Migration Manager for Active Directory is a reliant and powerful way to synchronize Active Directory objects and attribute data from one domain to another (and vice versa). It also has the ability to synchronize user passwords by installing a single agent on one domain controller. More than this, DSA is also responsible for mailbox creation in the Quest setup and synchronizes mailbox and Active Directory permissons.

The speed of delta synchronization (synchronizing changes of object attributes) is a combination of matching and caching. Quest DSA uses an AD LDS database to create matching objects that describe the synchronization relationship between an object in source domain and its peer in the target domain.

However, the ADLDS matching objects are most important when starting the synchronization and performing a Full Resync. In the ongoing synchronization, DSA takes the matching information from its cache which is a JET database located in “…\DSA\CONFIG\Cache” directory. The cache database file can grow up to a size which exceeds the size of the AD LDS by far. If disk space matters on your DSA machine, have an eye on the cache file size first.

DSA cache files

Solving incorrect matching

By default, Quest Active Directory sync knows 3 criteria for object matching (the way, how DSA identifies, whether it has to merge an existing account in target or create a new one) – mail address, sid-sidHistory, samAccountName. Both decisions (merge or create) have consequences since DSA will create or modify the matching object and bind objects together, that should form a unity (or not).

matching_criteria

However, we do not live in a perfect world and situations occur where the matching went wrong.

Real word scenario:

  1. Group A is created in source domain, mail-enabled and filled with 10 members. It is part of DSA migration scope.
  2. DSA picks up the group and looks up the matching criteria. All 3 criteria are activated and mail has highest precedence. DSA does not find a peer and creates a new group A in target domain with e-mail and the link resolver fills the group membership with the target user objects. DSA also creates a matching object and updates the cache file. So good so far.
  3. Now somebody decides to create a new group B in source domain and shifts the mail address from group A to group B while the mail address on group A is renamed in the same step.
  4. DSA will recognize that group B is existing and looks up for matching criteria. It will find a match for the mail address in group A of the target domain and will set up a matching of group source B to target A. It also will replicate the membership from source B to target A.
  5. We have now a lot of uncomfortable issues. Membership in the DLs looks different for users in source and users in target domain. Group A in target has 2 entries in sidHistory, one for source group A and one for source group B. The matching attribute in group A from target domain is now filled with the object GUID from group B in source domain and the proxyAddresses including X.500 are mixed as well. Other attributes depend on your skip list settings
  6.  And we still have group A in source. Since the matching criteria of sid-sidHistory is still valid, you can end up with a flip/flop, so that DSA runs over the two accounts and whenever there is a new attribute change on one of the source groups, it can either be group A or group B which is merged to group A in target.

OK, we should try to clean up the confusion.

  1. We better remove mail address matching in our setup since it has problems with the domestic way of changing groups in this customer environment. We clean up all wrong values of group A in target. We run a full resync (which is restricted to once per quarter)
  2. Same thing again, because the matching attribute was filled with the wrong value and the matching sid-sidHistory was still in place.
  3. We clean up again and delete the matching attribute. We modify group B in source to trigger DSA and expect that a new group B in target is created.
  4. Do we succeed? No. Of course not. There is a wrong matching object for group B (and group A) in ADLDS. OK. We clean up again and we delete the matching objects in ADLDS.
  5. No way. The same thing happens again. No group B in target, but a matching of group A and group B to group A in target.
  6. This time we stop DSA, clean up group A in target domain including wrong entries in proxyAddresses, sidHistory and delete the matching attributes. We delete the cache file and start with Full Resync – and we succeed

It’s all about cache. All the cleanup and repair actions can fail as long as the cache file still contains the wrong linking. Since a selective cleaning of the wrong object matching of the cache is not possible (anyone to try?), we always will need a full resync (of thousand objects) to repair a single object pair with wrong matching.

An alternative would have been to delete all 3 groups and create fresh objects. I would call it the “brute force method”. Not acceptable in many cases though.

Quest Migration Manager for Active Directory®: QMM vmover’s registry access blocked by security software

In our Exchange and Active Directory migration project we recently deployed a vmover package on a large number of client computers where QMM vmover.exe performs all resource updating locally without stressing the network. The results were quite positive, but after a while the Client protection team of the customer, who is actually running McAffee® security software on all client computers, was complaining about vmover activities. The security software identified vmover as intruder and blocked actions.
They said, that vmover.exe would try to add new keys in the McAffee® agent part of the registry. We could not believe that, but the AccessProtectionLog.txt of McAffee® exactly provided evidence:

14.01.2013 14:49:37
Blocked by Access Protection rule          NT AUTHORITY\SYSTEM
C:\Program Files\Quest\vmover\Vmover.exe
\REGISTRY\MACHINE\SYSTEM\CurrentControlSet\services\McAfeeFramework\Security
Common Standard Protection:
Prevent modification of McAfee Common Management Agent files and settings
Action blocked :
Create

Our settings in vmover.ini did allow vmover to update registry keys, which means Re-ACLing of permissions on registry keys, but we did not have an explanation why vmover would create something outside the user hive when updating user profiles.
Using the ProcessMonitor it was even more obvious that vmover.exe tries to create keys in the registry.

vmover_createregkey

The response from Quest Development came after short time:

What we saw in Process Monitor did not necessarily mean that vmover actually tries to create anything in there, but it’s rather the fact that RegCreateKeyEx function is used to enumerate the registry. There are two functions, one is RegOpenKeyEx and second is RegCreateKeyEx, both can be used to read information from a key, but the later will create a key if it does not exist depending on parameters passed. RegCreateKeyEx is used by Vmover for performance reasons. Also the entire registry is processed when process registry option is selected and all services are enumerated this way in registry when service processing is enabled.

With those arguments we got back to the Client protection team and after spending a beer or two, they agreed to put McAffee® on the Whitelist which solved the access block problem.
Good to know how things work.