How to address Federation Trust issues in Hybrid Configuration Wizard (HCW)

During my time as a PFE for Microsoft, I have encounted many issues with Federation in a Hybrid Exchange Deployment. Recently, the following support announcement came out and I wanted to share as I hope this can help others that may be having issues out there.

One of the more common causes of HCW failures is the Federation Trust step for the Exchange on-premises organizations in Full hybrid configurations (Classic or Modern topologies).

Federation trust is a mandatory step in the on-premises Exchange organizations when configuring Full hybrid deployments, as this allows us to create organization relationships (for features like hybrid free/busy or OWA/EAS redirection) and sharing policies (1:1 hybrid calendar sharing). In Exchange Online multi-tenant organizations, federation trust is already in place.

Below is an illustration of an Exchange hybrid deployment where both the Exchange on-premises organization and the Exchange Online organization have a trust with Azure Authentication System (formerly called Microsoft Federation Gateway):

Before getting to our subject, let’s quickly go over different hybrid configurations and Hybrid Configuration Wizard (HCW) – as this is the supported tool to configure hybrid deployments.
There are 2 flavors of hybrid configurations:
• Classic hybrid
• Modern hybrid

At this time, each of those supports the following hybrid modes:

• Full
• Minimal (which further breaks down into…)
  • Express (a one-time sync)
  • “Actual minimal”

A quick overview of Full / Minimal / Express options, can be found here. More info on HCW is here.

As mentioned earlier, a federation trust is created by HCW only in Full Hybrid.

HCW logs are located at %appdata%\Microsoft\Exchange Hybrid Configuration on the machine from where HCW was ran. The easiest way to get to them is to press F12 in the HCW window to open the Diagnostic tools and from there you can Open Folder Logging or Open Log File directly.

When you have issues with federation trust, the log will usually show errors when one of the following cmdlets are executed:

Set-FederationOrganizationIdentifier
or
Add-FederatedDomain (but can be other cmdlets as well).

Once you identified the exact cmdlet failing and where (Session=OnPremises – means Exchange Management Shell and Session=Tenant means Exchange Online PowerShell), you should copy-paste the failing command and try to execute it manually and see if that is failing as well (most likely it will). You can also open the shells from F12 Diagnostic tools windows in HCW.

In order to get more details on the error and to rule out this is not an issue with HCW itself, you will need to separately run the same command that threw exception in HCW log and add Verbose switch to get verbose details of the error and the serialized remote exception.

For example, if the Exchange server version is Exchange 2010, you will run the failing command with Verbose switch in Exchange Management Shell (EMS), see if that fails and then get the serialized remote exception.

Example:

If the Exchange Server version is Exchange 2013/2016 and the above commands didn’t show more details on the error, we can also try the following:

• Open regular Windows PowerShell (blue background) on the Exchange Server 2013/2016
• Run command: add-pssnapin exchange
• Run command that gave error in HCW and add a Verbose switch

Example:

Once you’ve gathered the verbose error / serialized exception, try to understand where it is failing (or provide it to Microsoft Support together with the HCW log).

Common Errors with Remediation Steps

• Federation trust fails with “Object reference not set to an instance of an object”

This is a known old issue on Exchange 2016 CU7 servers, make sure your Exchange servers are updated to the latest CU.

Full error in the HCW log:

Resolution: Install the latest CU for Exchange 2016

• Federation fails with “Proof of domain ownership has failed”

Full error in the HCW log:

Resolution:

• Check the TXT record for your domain(s) in HCW log or in Exchange Management Shell with command Get-FederatedDomainProof -DomainName
• See if it matches your published TXT record with either nslookup utility or by checking internet websites like https://www.whatsmydns.net/ put your domain in hostnames, type=txt, Nameservers – Authoritative

You would look for errors, missing records or unusual formatting (characters, spaces, quotes, TXT record split in half).

• Federation fails with “An unexpected error occurred on a receive” or “An unexpected error occurred on a send.”

Full error in the HCW log:

Resolution:

Check outbound access from all your Exchange Servers to Microsoft Federation Gateway by browsing using Internet Explorer with PSEXEC tool (with -s and -i switches) from the Exchange Server (this will use Internet Explorer under System Account / Exchange Server Account).

In this example, “Windows Live” is actually this exact URL: https://domains.live.com/service/managedelegation2.asmx

From on-premises Exchange to Office 365, the Exchange 2010 MBX & CAS or 2013 MBX (backend) or 2016 / 2019 would need outbound Internet access to the Microsoft Federation Gateway in addition to https://outlook.office365.com/ews/exchange.asmx

Verify the machine/system account can access these Microsoft Federation Gateway URLs:

• https://nexus.microsoftonline-p.com/federationmetadata/2006-12/federationmetadata.xml [<– You should see an xml page.]
• https://login.microsoftonline.com/extSTS.srf [<– You should see “Sorry, but we’re having trouble signing you in”.]
• https://domains.live.com/service/managedelegation2.asmx [<– You should see the operations supported by ManageDelegation2.]

For a complete list of O365 URL & IP addresses, see these articles:

• Office 365 URLs and IP address ranges – Exchange Online
• Office 365 URLs and IP address ranges – Common
• Additional endpoints

Note: If the Exchange requires a proxy server to access the Internet, specify the proxy server using “Set-ExchangeServer myExchange01 -InternetWebProxy http://myproxy:80”. Notice such proxy can’t require any user authentication for outbound Internet access, and the proxy must start with HTTP: and not HTTPS: (secure SSL).

You can also set the proxy using netsh as well.

set proxy proxy-server=”http=myproxy;https=sproxy:88″ bypass-list=”*.contoso.com” 

In rare instances, you can use the machine/system account to access the URLs from the browser, but Exchange cmdlets still failed with “Could not establish trust relationship for the SSL/TLS secure channel.” If that happens, make sure the certificate authorities for the urls are installed at the Third-Party Root Certification Authorities of the machine local certificate location.

REFERENCE:
Netsh Commands for Windows Hypertext Transfer Protocol (WINHTTP)
Firewall Considerations for Federated Delegation 

Federated delegation features require that the Mailbox and Client Access servers in your organization have outbound access to the Internet by using HTTPS. You must allow outbound HTTPS access (port 443 for TCP) from all Exchange 2010 Mailbox and Client Access servers in the organization.

• There is no specific error / exception, in HCW log you would see it stops without any specific error.

Full error in the HCW log:

Resolution:

Look for orphaned federation trust:

Get-FederatedOrganizationIdentifier | FL

or

in HCW log if you see something with “DEL“: “contoso.com/Configuration/Deleted Objects/Microsoft Federation Gateway/DEL: <xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx>”.

Solution is to remove the orphaned federation trust and re-run HCW.

Reference here.

NOTE: as a first step, you can try to run the command remove-federateddomain with the switch -Force. Also, you don’t need to recreate federation trust manually, just re-run HCW (this will recreate federation trust for us)

• Federation Trust fails with “InternalError InternalError: Internal error.”.”.””

Full error in the HCW log:

Resolution:

Open request with Microsoft Support or check if any Service Incident is published. Please see this for more information.

• Federation trust fails with “1007 Access Denied”

Full error in the HCW log:

Resolution:

“1007 Access Denied” error is usually when we have issues with:

1. Windows Time on the Exchange Server. See this article or this article.
2. Outdated federation trust (for example, federation trust certificate expired) and in this case you would remove federation trust by following these steps.

If the federation trust certificate is not found on any of the servers, then proceed with resolution from the next error.

As an example, from one HCW log, there seems to be this federation trust certificate expired on 05/13/2019:

• Federation trust fails with “Federation Certificate cannot be found”

Full error in the HCW log:

Resolution:

Follow the procedure here to manually cleanup the federation trust from AD. Once this is done, re-run the HCW to re-create it automatically.

KEEP TROUBLESHOOTING!
REMAIN VIGILANT USING FOCUSED INTENT, NEVER EMOTIONALIZING (FINE)!

REFERENCE:
How To Address Federation Trust Issues using the Hybrid Configuration Wizard