How to address Federation Trust issues in Hybrid Configuration Wizard (HCW)

During my time as a PFE for Microsoft, I have encounted many issues with Federation in a Hybrid Exchange Deployment. Recently, the following support announcement came out and I wanted to share as I hope this can help others that may be having issues out there.

One of the more common causes of HCW failures is the Federation Trust step for the Exchange on-premises organizations in Full hybrid configurations (Classic or Modern topologies).

Federation trust is a mandatory step in the on-premises Exchange organizations when configuring Full hybrid deployments, as this allows us to create organization relationships (for features like hybrid free/busy or OWA/EAS redirection) and sharing policies (1:1 hybrid calendar sharing). In Exchange Online multi-tenant organizations, federation trust is already in place.

Below is an illustration of an Exchange hybrid deployment where both the Exchange on-premises organization and the Exchange Online organization have a trust with Azure Authentication System (formerly called Microsoft Federation Gateway):

Example of Hybrid Federation

Before getting to our subject, let’s quickly go over different hybrid configurations and Hybrid Configuration Wizard (HCW) – as this is the supported tool to configure hybrid deployments.
There are 2 flavors of hybrid configurations:
Classic hybrid
Modern hybrid

At this time, each of those supports the following hybrid modes:

  • Full
  • Minimal (which further breaks down into…)
    • Express (a one-time sync)
    • “Actual minimal”

A quick overview of Full / Minimal / Express options, can be found here. More info on HCW is here.

As mentioned earlier, a federation trust is created by HCW only in Full Hybrid.

HCW logs are located at %appdata%\Microsoft\Exchange Hybrid Configuration on the machine from where HCW was ran. The easiest way to get to them is to press F12 in the HCW window to open the Diagnostic tools and from there you can Open Folder Logging or Open Log File directly.

When you have issues with federation trust, the log will usually show errors when one of the following cmdlets are executed:

Set-FederationOrganizationIdentifier
or
Add-FederatedDomain (but can be other cmdlets as well).

Once you identified the exact cmdlet failing and where (Session=OnPremises – means Exchange Management Shell and Session=Tenant means Exchange Online PowerShell), you should copy-paste the failing command and try to execute it manually and see if that is failing as well (most likely it will). You can also open the shells from F12 Diagnostic tools windows in HCW.

In order to get more details on the error and to rule out this is not an issue with HCW itself, you will need to separately run the same command that threw exception in HCW log and add Verbose switch to get verbose details of the error and the serialized remote exception.

For example, if the Exchange server version is Exchange 2010, you will run the failing command with Verbose switch in Exchange Management Shell (EMS), see if that fails and then get the serialized remote exception.

Example:

If the Exchange Server version is Exchange 2013/2016 and the above commands didn’t show more details on the error, we can also try the following:

  • Open regular Windows PowerShell (blue background) on the Exchange Server 2013/2016
  • Run command: add-pssnapin exchange
  • Run command that gave error in HCW and add a Verbose switch

Example:

Once you’ve gathered the verbose error / serialized exception, try to understand where it is failing (or provide it to Microsoft Support together with the HCW log).

Common Errors with Remediation Steps

  • Federation trust fails with “Object reference not set to an instance of an object”

This is a known old issue on Exchange 2016 CU7 servers, make sure your Exchange servers are updated to the latest CU.

Full error in the HCW log:

Resolution: Install the latest CU for Exchange 2016


  • Federation fails with “Proof of domain ownership has failed”

Full error in the HCW log:

Resolution:

• Check the TXT record for your domain(s) in HCW log or in Exchange Management Shell with command Get-FederatedDomainProof -DomainName
• See if it matches your published TXT record with either nslookup utility or by checking internet websites like https://www.whatsmydns.net/ put your domain in hostnames, type=txt, Nameservers – Authoritative

You would look for errors, missing records or unusual formatting (characters, spaces, quotes, TXT record split in half).


  • Federation fails with “An unexpected error occurred on a receive” or “An unexpected error occurred on a send.”

Full error in the HCW log:

Resolution:

Check outbound access from all your Exchange Servers to Microsoft Federation Gateway by browsing using Internet Explorer with PSEXEC tool (with -s and -i switches) from the Exchange Server (this will use Internet Explorer under System Account / Exchange Server Account).

In this example, “Windows Live” is actually this exact URL: https://domains.live.com/service/managedelegation2.asmx

From on-premises Exchange to Office 365, the Exchange 2010 MBX & CAS or 2013 MBX (backend) or 2016 / 2019 would need outbound Internet access to the Microsoft Federation Gateway in addition to https://outlook.office365.com/ews/exchange.asmx

Verify the machine/system account can access these Microsoft Federation Gateway URLs:

For a complete list of O365 URL & IP addresses, see these articles:

Note: If the Exchange requires a proxy server to access the Internet, specify the proxy server using “Set-ExchangeServer myExchange01 -InternetWebProxy http://myproxy:80”. Notice such proxy can’t require any user authentication for outbound Internet access, and the proxy must start with HTTP: and not HTTPS: (secure SSL).

You can also set the proxy using netsh as well.

set proxy proxy-server=”http=myproxy;https=sproxy:88″ bypass-list=”*.contoso.com” 

In rare instances, you can use the machine/system account to access the URLs from the browser, but Exchange cmdlets still failed with “Could not establish trust relationship for the SSL/TLS secure channel.” If that happens, make sure the certificate authorities for the urls are installed at the Third-Party Root Certification Authorities of the machine local certificate location.

REFERENCE:
Netsh Commands for Windows Hypertext Transfer Protocol (WINHTTP)
Firewall Considerations for Federated Delegation 

Federated delegation features require that the Mailbox and Client Access servers in your organization have outbound access to the Internet by using HTTPS. You must allow outbound HTTPS access (port 443 for TCP) from all Exchange 2010 Mailbox and Client Access servers in the organization.


  • There is no specific error / exception, in HCW log you would see it stops without any specific error.

Full error in the HCW log:

Resolution:

Look for orphaned federation trust:

Get-FederatedOrganizationIdentifier | FL

or

in HCW log if you see something with “DEL“: “contoso.com/Configuration/Deleted Objects/Microsoft Federation Gateway/DEL: <xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx>”.

Solution is to remove the orphaned federation trust and re-run HCW.

Reference here.

NOTE: as a first step, you can try to run the command remove-federateddomain with the switch -Force. Also, you don’t need to recreate federation trust manually, just re-run HCW (this will recreate federation trust for us)


  • Federation Trust fails with “InternalError InternalError: Internal error.”.”.””

Full error in the HCW log:

Resolution:

Open request with Microsoft Support or check if any Service Incident is published. Please see this for more information.


  • Federation trust fails with “1007 Access Denied”

Full error in the HCW log:

Resolution:

“1007 Access Denied” error is usually when we have issues with:

  1. Windows Time on the Exchange Server. See this article or this article.
  2. Outdated federation trust (for example, federation trust certificate expired) and in this case you would remove federation trust by following these steps.

If the federation trust certificate is not found on any of the servers, then proceed with resolution from the next error.

As an example, from one HCW log, there seems to be this federation trust certificate expired on 05/13/2019:


  • Federation trust fails with “Federation Certificate cannot be found”

Full error in the HCW log:

Resolution:

Follow the procedure here to manually cleanup the federation trust from AD. Once this is done, re-run the HCW to re-create it automatically.

KEEP TROUBLESHOOTING!
REMAIN VIGILANT USING FOCUSED INTENT, NEVER EMOTIONALIZING (FINE)!

REFERENCE:
How To Address Federation Trust Issues using the Hybrid Configuration Wizard

Exchange Hybrid Configuration Wizard Link

Wanted to do a quick post as I was working on my Hybrid Exchange Environment. I was unable to get the HCW to download and start from the Exchange Control Panel with the link provided on the page. This has happened to me for a while, so I went online and found a link that would work that could be downloaded and reused to open the HCW:

Hybrid Configuration Wizard Link

HOPE THIS HELPS!
LET ME HAVE KNOWLEDGE SHOULD THE LINK CHANGE!

References:
HYBRID CONFIGURATION WIZARD WON’T START ON WINDOWS 2016

What the Hybrid Configuration Wizard Performs in the background and configuring Hybrid Co-Existence with Exchange Online

****UPDATE 3/23/2020****

Changes have been made to the HCW and the installation since this original post. Please read the following to gain knowledge of the updates to the tool and the installation.


March 2020 significant update to Hybrid Configuration Wizard

We wanted to let you know that we are releasing what we consider a significant update to Exchange Hybrid Configuration Wizard (HCW). Along with a handful of small bug fixes, there are four major changes coming that we wanted to share with you:

  1. HCW will no longer enable Federation Trust by default for all installations. Instead, it will only enable Federation Trust if there are Exchange 2010 servers on premises. HCW will call Get-ExchangeServer and if no Exchange 2010 servers are reported, the workflow to enable Federation Trust and subsequently require domain proof will not execute. Note that organization relationships are still created.
  2. When uninstalling the hybrid agent and switching to Classic in the HCW, this action would sometimes fail with a “null reference” error. We have fixed this!
  3. How many of you have hit the HCW 8064 error – unable to configure OAuth, and subsequently had no idea why OAuth failed to configure? Yes, we heard you loud and clear! In this release, we have completely changed the way we enable and configure OAuth. Instead of enabling OAuth at the service layer, we now enable OAuth via a Graph API under the context of the Tenant Admin. This in turn removes the error obfuscation we had with the service layer enablement and allows us to include a detailed error entry in the HCW log. So while you still see the HCW 8064 error in the HCW UI, you can now review the log for the specific error detail which will make it easier to troubleshoot and resolve.
  4. When verifying DNS, we had a fallback mechanism that would reach out to an external site to verify domains. While this fallback mechanism was rarely hit, we received overwhelming feedback to not use this mechanism/site as it was not listed in our IPs & URLs web page. We have removed that fallback and now only use the endpoint “mshybridservice.trafficmanager.net”, which is listed in our endpoints documentation.

Because this is a major version update, the build begins with 17.x vs 16.x. The build number can be found in the top right corner once you download and open the HCW.

Because of the web-based distribution nature HCW uses and this version is a brand new package, you will get all this goodness simply by installing the new HCW from here. The current builds of HCW (16.x) will not automatically update to 17.x build, in fact – you could run the two side-by-side. Once you are on 17.x build – the HCW will then auto-update as usual.

A few additional notes: At this time, we do not anticipate new HCW 16.x builds. Therefore, to continue getting new HCW builds in the future, uninstall the current version of HCW (16.x) and then install the new version (17.x). The new version of HCW has a new dependency, .NET 4.7.2. The installer should take care of this for you, but just so you are aware.


ORIGINAL POST

I’m working on getting certified in Exchange Hybrid Scenarios and Exchange Online configuration as part of my skill set for Exchange. In doing so, I had successfully implemented a complete Full Hybrid Exchange Environment between my Exchange Online Tenant and my On Premises Exchange 2019 Environment last evening.

I wanted to give an update that was posted to my LinkedIn Posting on this. Thank you Brian Day for the vote of confidence and caution that running these cmdlets manually is not supported by Microsoft and that the HCW, like all the Online Microsoft Products, is constantly changing and being updated.

Important Note

As preparation, I bought some Exchange Online Plan 1 licenses which give me a 50 GB mailbox limit and basic mailbox functionality. It does not include the more advanced features such as ATP, or DLP. I am running most of those features through my On Premises Environment. I mainly wanted to be able to place mailboxes in the cloud and have a hybrid setup. My plan was to have mail flow continue through my On Premises environment so that my Exchange Server features would be used and I would not have to change any MX or SPF records. I also had my certificates in place for SSL and OWA so I would want keep mail flow routed that way, through on premises. I do want to be able to have Free/Busy lookups cross-premise so federation would have to be enabled as well. I would also have to enable the MRS proxy on my Exchange Server so that mailbox migration could be implemented cross-premise. I also have previously configured Azure AD Sync along with ADFS for Single Sign On. In my case, another server was not needed as I didn’t have enough mailboxes or real need to split my frontend and backend deployment. Running the Hybrid Configuration Wizard would not open any new ports or change any existing port traffic that was already configured on my firewall. These are just a few of the considerations that need to be looked at when considering a hybrid integration.

Here is a great article to read for the prerequisites
Exchange Hybrid Deployment Pre-requisites

So, once I had all those considerations handled in my design, I ran the Hybrid Configuration Wizard. What I want to do in this blog post is to go through the steps that the wizard does in the background to setup the Hybrid Environment as you go through the Wizard.

I mainly used the following blog post as a reference, but have approached it differently by diving into the cmdlets that are run during the process:

https://www.codetwo.com/admins-blog/office-365-hybrid-configuration-wizard-step-by-step/#validating-connection

1. The HCW validates the On-premises and Online Exchange Connection.

The Hybrid Configuration Wizard checks if it is possible to connect to both servers with PowerShell. It runs the Get-ExchangeServer cmdlet on premises after resolving the server in DNS. It then connects to Exchange Online, authorizing the connection:

Authority=https://login.windows.net/common Resource=https://outlook.office365.com ClientId=abcdefgh-a123-4566-9abc-2bdflancelin

2. The HCW collects data about Exchange configuration from the on-premises Active Directory

The Wizard gathers information about the local domain. In order to do that, the HCW executes a series of cmdlets.

These include, in order:

3. The HCW collects information on the Exchange online (Office 365) configuration

This task repeats what has been done in the previous step, only for the Exchange online, instead of the on-premises one.

The cmdlets include, in order:

4. Federation Trust is determined. If not present, a new Federation Trust and the required certificate will be created on the local Exchange Server

You will be prompted in the Wizard to create a Federation Trust if not present. The following articles explain Federation and its requirements:

Understanding Federation – Link Here
Understanding Federated Delegation – 
Link Here
Create a Federation Trust – 
Link Here

If the activity is finished successfully, a new certificate should appear on the on-premises Exchange Certificates list. The new certificate includes “Federation” in its Subject field. To make sure the certificate is there, you can run a cmdlet: Get-ExchangeCertificate | ft -a -wr


The results will look like this

5. The HCW creates a new Hybrid Configuration Object in the local Active Directory

The HCW will run cmdlets based on the information you provide in the HCW for the certificate, the on premises Exchange Server, the domain(s), and what features you want turned on:

It then checks the settings through the following cmdlets:

It then enables Organization Customization for both environments through this cmdlet:

6. Configuration is then completed to modify the settings on the on premises Exchange environment 

EmailAddressPolicy – HCW adds address @tenant.mail.onmicrosoft.com
The HCW configures remote domains – adds tenant.mail.onmicrosoft.com and tenant.onmicrosoft.com
The HCW adds a new accepted domain – adds tenant.mail.onmicrosoft.com

Some of the cmdlets run:

7. The HCW Configures the Organization Relationship between the local server and the cloud.

This configuration is not necessary in minimal hybrid deployment. Since I have a full hybrid deployment configured, the cmdlets were run as needed to configure it. Thanks to the correct configuration, it is possible to synchronize free/busy status of mailboxes and their elements between the on-premises Exchange Environment and Exchange online. 

Some of the cmdlets run in the process:

8. The HCW and setting connectors on both Exchange servers

The HCW checks to see if the connectors are there, if not, it sets them up. During this workflow, four connectors are set – one receive and one send connector for each server. Those connectors guarantee the mail flow between the on-premises and Exchange Online.

Some of the cmdlets run in the process:

The Intra-Organization is set as well:

9. The HCW configures OAuth Authentication across the Hybrid

This LINK explains how OAuth is configured between Exchange On Premises and Exchange Online. It’s a very good article to read as it shows how to get the Modern Authentication style working. Now the HCW does this for you and at the end of the article, you can run cmdlets to test the validity of the configuration.

If you want to go into a deep dive about how the Hybrid Authentication works, see the following:
Deep Dive Into Hybrid Authentication – from the MS Exchange Team Blog

Here are some of cmdlets run during this process workflow:

Again, look at both of those links to get a little more detail as to what each cmdlet does and how it sets up OAuth. Here are the two cmdlets used to test OAuth:

10. Enable MRS Proxy for Migration

In order to be able to move mailboxes between Exchange On Premises and Exchange Online, you have to enable the Exchange Web Services Virtual Directory to use the MRSProxy (Microsoft Replication Service proxy). You also have to set your EWS Virtual Directory to use Basic Authentication. You’ll want to do this before running the HCW or else you will receive the following error when the HCW validates the Migration setup and configuration:

Microsoft.Exchange.Migration.MigrationServerConnectionFailedException: The connection to the server ‘mail.ldlnet.net’ could not be completed. —> Microsoft.Exchange.MailboxReplicationService.RemoteTransientException: The call to ‘https://mail.ldlnet.net/EWS/mrsproxy.svc’ failed. Error details: The HTTP request was forbidden with client authentication scheme ‘Negotiate’. –> The remote server returned an error: (403) Forbidden.. —> Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The HTTP request was forbidden with client authentication scheme ‘Negotiate’. —> Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The remote server returned an error: (403) Forbidden.

Some of the cmdlets run to test Migration and MRS Proxy Settings are as follows:

11. Final HCW Configuration and cleanup.

The HCW runs from final cmdlets to finish up the installation of the Hybrid environment. Here are the cmdlets run:

All this information was found in the setup logs that are in the following directory
C:\Users\%username%\AppData\Roaming\Microsoft\Exchange Hybrid Configuration

REFERENCES
Understanding Federation
Understanding Federated Delegation
Create a Federation Trust
Hybrid deployment prerequisites
Exchange Specific OAuth 2.0 Protocol Specification
Understanding WS-Security
JSON Web Tokens
Using OAuth2 to access Calendar, Contact and Mail API in Office 365 Exchange Online
Configurable token lifetimes in Azure Active Directory (Public Preview)
OAuth Troubleshooting
Principles of Token Validation
Troubleshooting free/busy issues in Exchange hybrid environment
How to configure Exchange Server on-premises to use Hybrid Modern Authentication
Microsoft 365 Messaging Administrator Certification Transition (beta)
Microsoft 365 certification exams
Exchange Server build numbers and release dates
March 2020 Updates to the HCW

PLEASE LEAVE QUESTIONS, COMMENTS, UPDATES! I WOULD LOVE TO HEAR FROM YOU!