Federation trust is a mandatory step in the on-premises Exchange organizations when configuring Full hybrid deployments, as this allows us to create organization relationships (for features like hybrid free\/busy or OWA\/EAS redirection) and sharing policies (1:1 hybrid calendar sharing). In Exchange Online multi-tenant organizations, federation trust is already in place.
Below is an illustration of an Exchange hybrid deployment where both the Exchange on-premises organization and the Exchange Online organization have a trust with Azure Authentication System (formerly called Microsoft Federation Gateway):<\/p>\n\n\n\n
![\"\"](\"https:\/\/itblog.ldlnet.net\/wp-content\/uploads\/2020\/02\/HybridFed.jpg\")
Before getting to our subject, let\u2019s quickly go over different hybrid configurations and Hybrid Configuration Wizard (HCW) – as this is the supported tool to configure hybrid deployments. At this time, each of those supports the following hybrid modes:<\/p>\n\n\n\n A quick overview of Full \/ Minimal \/ Express options, can be found here<\/a>. More info on HCW is here<\/a>. If the Exchange Server version is Exchange 2013\/2016 and the above commands didn\u2019t show more details on the error, we can also try the following:<\/p>\n\n\n\n Example:<\/p>\n\n\n Once you’ve gathered the verbose error \/ serialized exception, try to understand where it is failing (or provide it to Microsoft Support together with the HCW log).<\/p>\n\n\n\n This is a known old issue on Exchange 2016 CU7 servers, make sure your Exchange servers are updated to the latest CU.<\/em><\/p>\n\n\n\n Full error in the HCW log:<\/p>\n\n\n Resolution:<\/strong> Install the latest CU for Exchange 2016<\/a><\/p>\n\n\n\n Full error in the HCW log:<\/p>\n\n\n Resolution:<\/strong><\/p>\n\n\n\n \u2022 Check the TXT record for your domain(s) in HCW log or in Exchange Management Shell with command Get-FederatedDomainProof -DomainName Full error in the HCW log:<\/p>\n\n\n Resolution:<\/strong><\/p>\n\n\n\n Check outbound access from all your Exchange Servers to Microsoft Federation Gateway by browsing using Internet Explorer with PSEXEC tool<\/a> (with -s and -i switches) from the Exchange Server (this will use Internet Explorer under System Account \/ Exchange Server Account).<\/p>\n\n\n\n In this example, \u201cWindows Live\u201d is actually this exact URL: https:\/\/domains.live.com\/service\/managedelegation2.asmx<\/a><\/p>\n\n\n\n From on-premises Exchange to Office 365, the Exchange 2010 MBX & CAS or 2013 MBX (backend) or 2016 \/ 2019 would need outbound Internet access to the Microsoft Federation Gateway in addition to https:\/\/outlook.office365.com\/ews\/exchange.asmx<\/a><\/p>\n\n\n\n Verify the machine\/system account can access these Microsoft Federation Gateway <\/strong>URLs:<\/p>\n\n\n\n For a complete list of O365 URL & IP addresses, see these articles:<\/p>\n\n\n\n Note:<\/strong> If the Exchange requires a proxy server to access the Internet, specify the proxy server using “Set-ExchangeServer myExchange01 -InternetWebProxy http:\/\/myproxy:80”. Notice such proxy can’t require any user authentication for outbound Internet access, and the proxy must start with HTTP: and not HTTPS: (secure SSL).<\/p>\n\n\n\n You can also set the proxy using netsh as well.<\/p>\n\n\n\n set proxy proxy-server=”http=myproxy;https=sproxy:88″ bypass-list=”*.contoso.com” <\/p>\n\n\n\n In rare instances, you can use the machine\/system account to access the URLs from the browser, but Exchange cmdlets still failed with “Could not establish trust relationship for the SSL\/TLS secure channel.” If that happens, make sure the certificate authorities for the urls are installed at the Third-Party Root Certification Authorities<\/strong> of the machine local certificate location.<\/p>\n\n\n\n REFERENCE:<\/strong> Full error in the HCW log:<\/p>\n\n\n Resolution:<\/strong><\/p>\n\n\n\n Look for orphaned federation trust:<\/p>\n\n\n\n Get-FederatedOrganizationIdentifier | FL Reference\u00a0here<\/a>.<\/em><\/strong><\/p>\n\n\n\n NOTE: as a first step, you can try to run the command\u00a0remove-federateddomain<\/em><\/strong>\u00a0with the switch\u00a0-Force<\/strong>. Also, you don’t need to recreate federation trust manually, just re-run HCW (this will recreate federation trust for us)<\/p>\n\n\n\n Full error in the HCW log:<\/p>\n\n\n Resolution:<\/strong><\/p>\n\n\n\n Open request with Microsoft Support or check if any Service Incident is published. Please\u00a0see this<\/a> for more information.<\/em><\/strong><\/p>\n\n\n\n Full error in the HCW log:<\/p>\n\n\n Resolution:<\/strong><\/p>\n\n\n\n “1007 Access Denied” error is usually when we have issues with:<\/p>\n\n\n\n If the federation trust certificate is not found on any of the servers, then proceed with resolution from the next error. Full error in the HCW log:<\/p>\n\n\n Resolution:<\/strong><\/p>\n\n\n\n Follow the procedure\u00a0her<\/em><\/strong>e<\/a>\u00a0to manually cleanup the federation trust from AD. Once this is done, re-run the HCW to re-create it automatically. <\/p>\n\n\n\n
There are 2 flavors of hybrid configurations:
\u2022 Classic hybrid<\/a>
\u2022 Modern hybrid<\/a><\/p>\n\n\n\n
As mentioned earlier, a federation trust is created by HCW only in Full Hybrid<\/em><\/strong>.
HCW logs are located at %appdata%\\Microsoft\\Exchange Hybrid Configuration<\/em><\/strong> on the machine from where HCW was ran. The easiest way to get to them is to press F12 in the HCW window to open the Diagnostic tools and from there you can Open Folder Logging or Open Log File directly.
When you have issues with federation trust, the log will usually show errors when one of the following cmdlets are executed:
Set-FederationOrganizationIdentifier
or
Add-FederatedDomain (but can be other cmdlets as well).
Once you identified the exact cmdlet failing and where (Session=OnPremises \u2013 means Exchange Management Shell and Session=Tenant means Exchange Online PowerShell), you should copy-paste the failing command and try to execute it manually and see if that is failing as well (most likely it will). You can also open the shells from F12 Diagnostic tools windows in HCW.
In order to get more details on the error and to rule out this is not an issue with HCW itself, you will need to separately run the same command that threw exception in HCW log and add Verbose switch to get verbose details of the error and the serialized remote exception.
For example, if the Exchange server version is Exchange 2010, you will run the failing command with Verbose switch in Exchange Management Shell (EMS), see if that fails and then get the serialized remote exception.
Example: <\/p>\n\n\n\nstart-transcript\n\nSet-FederatedOrganizationIdentifier -AccountNamespace
\nstart-transcript\n\nSet-FederatedOrganizationIdentifier -AccountNamespace
Common Errors with Remediation Steps<\/h3>\n\n\n\n
\n2017.10.06 01:45:56.562 *ERROR* 10277 [Client=UX, Activity=Domain Ownership, Session=OnPremises, Cmdlet=Set-FederatedOrganizationIdentifier, Thread=21] FINISH Time=398.4ms Results=PowerShell failed to invoke 'Set-FederatedOrganizationIdentifier': Object reference not set to an instance of an object. An unexpected error has occurred and a Watson dump is being generated: Object reference not set to an instance of an object.\n\n2017.10.06 01:45:56.563 *ERROR* 10224 [Client=UX, Page=DomainProof, Thread=21] Microsoft.Online.CSE.Hybrid.PowerShell.PowerShellInvokeException: PowerShell failed to invoke 'Set-FederatedOrganizationIdentifier': Object reference not set to an instance of an object. An unexpected error has occurred and a Watson dump is being generated: Object reference not set to an instance of an object. ---> System.Management.Automation.RemoteException: Object reference not set to an instance of an object.\n<\/pre>\n\n\n\n
\n\n\n\n\n2019.07.16 17:53:14.750 10276 [Client=UX, Activity=Domain Ownership, Session=OnPremises, Cmdlet=Add-FederatedDomain, Thread=19] START Add-FederatedDomain -DomainName 'contoso.com'\n\n2019.07.06 17:53:15.375 10177 [Client=UX, Activity=Domain Ownership, Provider=OnPremises, Thread=19] PowerShell Error Record: {CategoryInfo={Activity=Add-FederatedDomain,Category=InvalidResult,Reason=DomainProofOwnershipException,TargetName=,TargetType=},ErrorDetails=,Exception=Proof of domain ownership has failed. Make sure that the TXT record for the specified domain is available in DNS. The format of the TXT record should be \"example.com IN TXT hash-value\" where \"example.com\" is the domain you want to configure for Federation and \"hash-value\" is the proof value generated with \"Get-FederatedDomainProof -DomainName example.com\".,FullyQualifiedErrorId=367408EF,Microsoft.Exchange.Management.SystemConfigurationTasks.AddFederatedDomain}\n<\/pre>\n\n\n\n
\u2022 See if it matches your published TXT record with either nslookup utility or by checking internet websites like https:\/\/www.whatsmydns.net\/<\/a> put your domain in hostnames, type=txt, Nameservers – Authoritative
You would look for errors, missing records or unusual formatting (characters, spaces, quotes, TXT record split in half).<\/p>\n\n\n\n
\n\n\n\n\n2018.10.10 17:03:31.277 *ERROR* [Activity=Domain Ownership, Session=OnPremises, Cmdlet=Set-FederatedOrganizationIdentifier] FINISH Time=64.3s Results=PowerShell failed to invoke 'Set-FederatedOrganizationIdentifier': An error occurred while attempting to provision Exchange to the Partner STS. Detailed Information \"An error occurred accessing Windows Live. Detailed information: \"The underlying connection was closed: An unexpected error occurred on a receive.\".\".\n<\/pre>\n\n\n\n
Netsh Commands for Windows Hypertext Transfer Protocol (WINHTTP)<\/a>
Firewall Considerations for Federated Delegation<\/a>\u00a0
Federated delegation features require that the Mailbox and Client Access servers in your organization have outbound access to the Internet by using HTTPS. You must allow outbound HTTPS access (port 443 for TCP) from all Exchange 2010 Mailbox and Client Access servers in the organization.<\/p>\n\n\n\n
\n\n\n\n\n2019.02.14 12:56:21.658 [Activity=Domain Ownership, Session=OnPremises, Cmdlet=Get-FederatedOrganizationIdentifier] FINISH Time=133.0ms Results=1 FederatedOrganizationIdWithDomainStatus {AccountNamespace='FYDIBOHF25SPDLT.contoso.com' DelegationTrustLink='contoso.local\/Configuration\/Deleted Objects\/Microsoft Federation Gateway DEL:8e834abf-5154-4540-a3c6-5a5c614c6a06'Enabled=1 ExchangeVersion='0.10 (14.0.100.0)' Guid=2e1da884-9686-4221-8098-d34ced5a2f85 Id='Federation' Identity='Federation' IsValid=1 Name='Federation' ObjectState='Unchanged' WhenChanged='8\/11\/2015 5:35:58 PM' WhenChangedUTC='8\/11\/2015 2:35:58 PM' WhenCreated='10\/18\/2009 10:30:09 AM' WhenCreatedUTC='10\/18\/2009 6:30:09 AM'}\n2019.02.14 12:56:21.677 [Client=UX, Page=DomainProof] Unproven Domains: 1\n<\/pre>\n\n\n\n
or
in HCW log if you see something with “DEL<\/strong>“: “contoso.com\/Configuration\/Deleted Objects\/Microsoft Federation Gateway\/DEL: <xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx>”.
Solution is to remove the orphaned federation trust and re-run HCW.<\/p>\n\n\n\n
\n\n\n\n\n2019.08.23 07:45:22.914 10276 [Client=UX, Activity=Domain Ownership, Session=OnPremises, Cmdlet=Set-FederatedOrganizationIdentifier, Thread=20] START Set-FederatedOrganizationIdentifier -AccountNamespace 'contoso.com' -DelegationFederationTrust 'Microsoft Federation Gateway' -Enabled: $true -DefaultDomain $null\n\n2019.08.23 07:45:23.239 *ERROR* 10277 [Client=UX, Activity=Domain Ownership, Session=OnPremises, Cmdlet=Set-FederatedOrganizationIdentifier, Thread=20] FINISH Time=325.0ms Results=PowerShell failed to invoke 'Set-FederatedOrganizationIdentifier': An error occurred while attempting to provision Exchange to the Partner STS. Detailed Information \"An unexpected result was received from Windows Live. Detailed information: \"InternalError InternalError: Internal error.\".\". {CategoryInfo={Activity=[System.String] Set-FederatedOrganizationIdentifier,Category=[System.Management.Automation.ErrorCategory] InvalidResult,Reason=[System.String]\n<\/pre>\n\n\n\n
\n\n\n\n\nSet-FederatedOrganizationIdentifier,Category=[System.Management.Automation.ErrorCategory] InvalidResult,Reason=[System.String] ProvisioningFederatedExchangeException,TargetName=[System.String] ,TargetType=[System.String] },ErrorDetails=,Exception=[System.Management.Automation.RemoteException] An error occurred while attempting to provision Exchange to the Partner STS. Detailed Information \"An unexpected result was received from Windows Live. Detailed information: \"1007 AccessDenied: Access Denied.\".\"\n<\/pre>\n\n\n\n
As an example, from one HCW log, there seems to be this federation trust certificate expired on 05\/13\/2019: <\/p>\n\n\n\nOrgCertificate=[Subject] CN=Federation [Issuer] CN=Federation [Serial Number] 4E91XXXXXXXXXXXXXXXXXXXXXXXXXXXX [Not Before] 5\/13\/2014 11:21:36 AM [Not After] 5\/13\/2019 11:21:36 AM\n<\/pre>\n\n\n\n
\n\n\n\n\n2019.11.22 14:40:32.569 *ERROR* 10277 [Client=UX, Activity=Domain Ownership, Session=OnPremises, Cmdlet=Get-FederatedDomainProof, Thread=6] FINISH Time=125.1ms Results=PowerShell failed to invoke Get-FederatedDomainProof: Federation certificate with the thumbprint \u201803650FFAF05E83E3B007DF3473CB5753F5C4459\u2019cannot be found.\n<\/pre>\n\n\n\n
KEEP TROUBLESHOOTING!
REMAIN VIGILANT USING FOCUSED INTENT, NEVER EMOTIONALIZING<\/em> (FINE)! <\/h2>\n\n\n\n