****UPDATE 3/23/2020****
Changes have been made to the HCW and the installation since this original post. Please read the following to gain knowledge of the updates to the tool and the installation.
March 2020 significant update to Hybrid Configuration Wizard
We wanted to let you know that we are releasing what we consider a significant update to Exchange Hybrid Configuration Wizard (HCW). Along with a handful of small bug fixes, there are four major changes coming that we wanted to share with you:
- HCW will no longer enable Federation Trust by default for all installations. Instead, it will only enable Federation Trust if there are Exchange 2010 servers on premises. HCW will call Get-ExchangeServer and if no Exchange 2010 servers are reported, the workflow to enable Federation Trust and subsequently require domain proof will not execute. Note that organization relationships are still created.
- When uninstalling the hybrid agent and switching to Classic in the HCW, this action would sometimes fail with a “null reference” error. We have fixed this!
- How many of you have hit the HCW 8064 error – unable to configure OAuth, and subsequently had no idea why OAuth failed to configure? Yes, we heard you loud and clear! In this release, we have completely changed the way we enable and configure OAuth. Instead of enabling OAuth at the service layer, we now enable OAuth via a Graph API under the context of the Tenant Admin. This in turn removes the error obfuscation we had with the service layer enablement and allows us to include a detailed error entry in the HCW log. So while you still see the HCW 8064 error in the HCW UI, you can now review the log for the specific error detail which will make it easier to troubleshoot and resolve.
- When verifying DNS, we had a fallback mechanism that would reach out to an external site to verify domains. While this fallback mechanism was rarely hit, we received overwhelming feedback to not use this mechanism/site as it was not listed in our IPs & URLs web page. We have removed that fallback and now only use the endpoint “mshybridservice.trafficmanager.net”, which is listed in our endpoints documentation.
Because this is a major version update, the build begins with 17.x vs 16.x. The build number can be found in the top right corner once you download and open the HCW.
Because of the web-based distribution nature HCW uses and this version is a brand new package, you will get all this goodness simply by installing the new HCW from here. The current builds of HCW (16.x) will not automatically update to 17.x build, in fact – you could run the two side-by-side. Once you are on 17.x build – the HCW will then auto-update as usual.
A few additional notes: At this time, we do not anticipate new HCW 16.x builds. Therefore, to continue getting new HCW builds in the future, uninstall the current version of HCW (16.x) and then install the new version (17.x). The new version of HCW has a new dependency, .NET 4.7.2. The installer should take care of this for you, but just so you are aware.
ORIGINAL POST
I’m working on getting certified in Exchange Hybrid Scenarios and Exchange Online configuration as part of my skill set for Exchange. In doing so, I had successfully implemented a complete Full Hybrid Exchange Environment between my Exchange Online Tenant and my On Premises Exchange 2019 Environment last evening.
I wanted to give an update that was posted to my LinkedIn Posting on this. Thank you Brian Day for the vote of confidence and caution that running these cmdlets manually is not supported by Microsoft and that the HCW, like all the Online Microsoft Products, is constantly changing and being updated.
As preparation, I bought some Exchange Online Plan 1 licenses which give me a 50 GB mailbox limit and basic mailbox functionality. It does not include the more advanced features such as ATP, or DLP. I am running most of those features through my On Premises Environment. I mainly wanted to be able to place mailboxes in the cloud and have a hybrid setup. My plan was to have mail flow continue through my On Premises environment so that my Exchange Server features would be used and I would not have to change any MX or SPF records. I also had my certificates in place for SSL and OWA so I would want keep mail flow routed that way, through on premises. I do want to be able to have Free/Busy lookups cross-premise so federation would have to be enabled as well. I would also have to enable the MRS proxy on my Exchange Server so that mailbox migration could be implemented cross-premise. I also have previously configured Azure AD Sync along with ADFS for Single Sign On. In my case, another server was not needed as I didn’t have enough mailboxes or real need to split my frontend and backend deployment. Running the Hybrid Configuration Wizard would not open any new ports or change any existing port traffic that was already configured on my firewall. These are just a few of the considerations that need to be looked at when considering a hybrid integration.
Here is a great article to read for the prerequisites
Exchange Hybrid Deployment Pre-requisites
So, once I had all those considerations handled in my design, I ran the Hybrid Configuration Wizard. What I want to do in this blog post is to go through the steps that the wizard does in the background to setup the Hybrid Environment as you go through the Wizard.
I mainly used the following blog post as a reference, but have approached it differently by diving into the cmdlets that are run during the process:
1. The HCW validates the On-premises and Online Exchange Connection.
The Hybrid Configuration Wizard checks if it is possible to connect to both servers with PowerShell. It runs the Get-ExchangeServer cmdlet on premises after resolving the server in DNS. It then connects to Exchange Online, authorizing the connection:
Authority=https://login.windows.net/common Resource=https://outlook.office365.com ClientId=abcdefgh-a123-4566-9abc-2bdflancelin
2. The HCW collects data about Exchange configuration from the on-premises Active Directory
The Wizard gathers information about the local domain. In order to do that, the HCW executes a series of cmdlets.
These include, in order:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | Get-MailboxDatabase -IncludePreExchange2013: $true Get-OrganizationConfig Get-HybridConfiguration Get-AcceptedDomain Get-FederatedOrganizationIdentifier -IncludeExtendedDomainInfo: $false Get-FederationTrust Get-WebServicesVirtualDirectory -ADPropertiesOnly: $true Get-RemoteDomain |
3. The HCW collects information on the Exchange online (Office 365) configuration
This task repeats what has been done in the previous step, only for the Exchange online, instead of the on-premises one.
The cmdlets include, in order:
1 2 3 4 5 6 7 | Get-OrganizationConfig Get-OnPremisesOrganization Get-AcceptedDomain Get-MigrationEndpoint |
4. Federation Trust is determined. If not present, a new Federation Trust and the required certificate will be created on the local Exchange Server
You will be prompted in the Wizard to create a Federation Trust if not present. The following articles explain Federation and its requirements:
Understanding Federation – Link Here
Understanding Federated Delegation – Link Here
Create a Federation Trust – Link Here
If the activity is finished successfully, a new certificate should appear on the on-premises Exchange Certificates list. The new certificate includes “Federation” in its Subject field. To make sure the certificate is there, you can run a cmdlet: Get-ExchangeCertificate | ft -a -wr
The results will look like this
5. The HCW creates a new Hybrid Configuration Object in the local Active Directory
The HCW will run cmdlets based on the information you provide in the HCW for the certificate, the on premises Exchange Server, the domain(s), and what features you want turned on:
1 2 3 | New-HybridConfiguration -Name 'Hybrid Configuration' Set-HybridConfiguration -ClientAccessServers $null -ExternalIPAddresses $null -Domains 'ldlnet.net' -OnPremisesSmartHost 'mail.ldlnet.net' -TLSCertificateName 'CN=Go Daddy Secure Certificate Authority - G2, OU=http://certs.godaddy.com/repository/, O="GoDaddy.com, Inc.", L=Scottsdale, S=Arizona, C=USCN=mail.ldlnet.net, OU=Domain Control Validated' -SendingTransportServers EX01 -ReceivingTransportServers EX01 -EdgeTransportServers $null -Features FreeBusy,MoveMailbox,Mailtips,MessageTracking,OwaRedirection,OnlineArchive,SecureMail,CentralizedTransport,Photos |
It then checks the settings through the following cmdlets:
1 2 3 4 5 | Get-OrganizationConfig Get-AcceptedDomain Get-RemoteDomain |
It then enables Organization Customization for both environments through this cmdlet:
1 | Enable-OrganizationCustomization |
6. Configuration is then completed to modify the settings on the on premises Exchange environment
EmailAddressPolicy – HCW adds address @tenant.mail.onmicrosoft.com
The HCW configures remote domains – adds tenant.mail.onmicrosoft.com and tenant.onmicrosoft.com
The HCW adds a new accepted domain – adds tenant.mail.onmicrosoft.com
Some of the cmdlets run:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | Set-EmailAddressPolicy -Identity 'ldlnet.net' -ForceUpgrade: $true -EnabledEmailAddressTemplates 'smtp:%1g%s@ldlnet.net','SMTP:%g.%s@ldlnet.net','smtp:%m@LDLNET.mail.onmicrosoft.com' Update-EmailAddressPolicy -Identity 'ldlnet.net' -UpdateSecondaryAddressesOnly: $true New-RemoteDomain -Name 'Hybrid Domain - LDLNET.mail.onmicrosoft.com' -DomainName 'LDLNET.mail.onmicrosoft.com' Set-RemoteDomain -TargetDeliveryDomain: $true -Identity 'Hybrid Domain - LDLNET.mail.onmicrosoft.com' New-RemoteDomain -Name 'Hybrid Domain - LDLNET.onmicrosoft.com' -DomainName 'LDLNET.onmicrosoft.com' Set-RemoteDomain -TrustedMailInboundEnabled: $true -Identity 'Hybrid Domain - LDLNET.onmicrosoft.com' New-AcceptedDomain -DomainName 'LDLNET.mail.onmicrosoft.com' -Name 'LDLNET.mail.onmicrosoft.com' |
7. The HCW Configures the Organization Relationship between the local server and the cloud.
This configuration is not necessary in minimal hybrid deployment. Since I have a full hybrid deployment configured, the cmdlets were run as needed to configure it. Thanks to the correct configuration, it is possible to synchronize free/busy status of mailboxes and their elements between the on-premises Exchange Environment and Exchange online.
Some of the cmdlets run in the process:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | Get-FederationTrust Get-FederatedOrganizationIdentifier -IncludeExtendedDomainInfo: $false Get-FederationInformation -DomainName 'LDLNET.mail.onmicrosoft.com' -BypassAdditionalDomainValidation: $true Get-OrganizationConfig Get-OrganizationRelationship Set-FederatedOrganizationIdentifier -DefaultDomain 'LDLNET.mail.onmicrosoft.com' -Enabled: $true New-OrganizationRelationship -Name 'On-premises to O365 - 48e7bec9-404c-4d24-b59e-4b46b64d7e03' -TargetApplicationUri 'outlook.com' -TargetAutodiscoverEpr 'https://autodiscover-s.outlook.com/autodiscover/autodiscover.svc/WSSecurity' -Enabled: $true -DomainNames 'LDLNET.mail.onmicrosoft.com' New-OrganizationRelationship -Name 'O365 to On-premises - 48e7bec9-404c-4d24-b59e-4b46b64d7e03' -TargetApplicationUri 'FYDIBOHF25SPDLT.ldlnet.net' -TargetAutodiscoverEpr 'https://autodiscover.ldlnet.net/autodiscover/autodiscover.svc/WSSecurity' -Enabled: $true -DomainNames 'ldlnet.net' Set-OrganizationRelationship -MailboxMoveEnabled: $true -FreeBusyAccessEnabled: $true -FreeBusyAccessLevel LimitedDetails -ArchiveAccessEnabled: $true -MailTipsAccessEnabled: $true -MailTipsAccessLevel All -DeliveryReportEnabled: $true -PhotosEnabled: $true -TargetOwaURL 'http://outlook.com/owa/LDLNET.onmicrosoft.com' -Identity 'On-premises to O365 - 48e7bec9-404c-4d24-b59e-4b46b64d7e03' Set-OrganizationRelationship -FreeBusyAccessEnabled: $true -FreeBusyAccessLevel LimitedDetails -MailTipsAccessEnabled: $true -MailTipsAccessLevel All -DeliveryReportEnabled: $true -PhotosEnabled: $true -TargetOwaURL 'https://mail.ldlnet.net/owa' -Identity 'O365 to On-premises - 48e7bec9-404c-4d24-b59e-4b46b64d7e03' Add-AvailabilityAddressSpace -ForestName 'LDLNET.mail.onmicrosoft.com' -AccessMethod InternalProxy -UseServiceAccount: $true -ProxyUrl 'https://mail.ldlnet.net/EWS/Exchange.asmx' |
8. The HCW and setting connectors on both Exchange servers
The HCW checks to see if the connectors are there, if not, it sets them up. During this workflow, four connectors are set – one receive and one send connector for each server. Those connectors guarantee the mail flow between the on-premises and Exchange Online.
Some of the cmdlets run in the process:
1 2 3 4 5 6 7 8 9 | Set-HybridConfiguration -ClientAccessServers $null -ExternalIPAddresses $null New-SendConnector -Name 'Outbound to Office 365' -AddressSpaces 'smtp:LDLNET.mail.onmicrosoft.com;1' -DNSRoutingEnabled: $true -ErrorPolicies Default -Fqdn 'mail.ldlnet.net' -RequireTLS: $true -IgnoreSTARTTLS: $false -SourceTransportServers EX2019 -SmartHosts $null -TLSAuthLevel DomainValidation -DomainSecureEnabled: $false -TLSDomain 'mail.protection.outlook.com' -CloudServicesMailEnabled: $true -TLSCertificateName 'CN=Go Daddy Secure Certificate Authority - G2, OU=http://certs.godaddy.com/repository/, O="GoDaddy.com, Inc.", L=Scottsdale, S=Arizona, C=USCN=mail.ldlnet.net, OU=Domain Control Validated' Set-ReceiveConnector -AuthMechanism 'Tls, Integrated, BasicAuth, BasicAuthRequireTLS, ExchangeServer' -Bindings '[::]:25','0.0.0.0:25' -Fqdn 'EX2019.ldlnet.local' -PermissionGroups 'AnonymousUsers, ExchangeServers, ExchangeLegacyServers' -RemoteIPRanges '::-ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff','0.0.0.0-255.255.255.255' -RequireTLS: $false -TLSDomainCapabilities 'mail.protection.outlook.com:AcceptCloudServicesMail' -TLSCertificateName 'CN=Go Daddy Secure Certificate Authority - G2, OU=http://certs.godaddy.com/repository/, O="GoDaddy.com, Inc.", L=Scottsdale, S=Arizona, C=USCN=mail.ldlnet.net, OU=Domain Control Validated' -TransportRole FrontendTransport -Identity 'EX2019\Default Frontend EX2019' New-InboundConnector -Name 'Inbound from 48e7bec9-404c-4d24-b59e-4b46b64d7e03' -CloudServicesMailEnabled: $true -ConnectorSource HybridWizard -ConnectorType OnPremises -RequireTLS: $true -SenderDomains '' -SenderIPAddresses $null -RestrictDomainsToIPAddresses: $false -TLSSenderCertificateName '.ldlnet.net' -AssociatedAcceptedDomains $null New-OutboundConnector -Name 'Outbound to 48e7bec9-404c-4d24-b59e-4b46b64d7e03' -RecipientDomains '*' -SmartHosts 'mail.ldlnet.net' -ConnectorSource HybridWizard -ConnectorType OnPremises -TLSSettings DomainValidation -TLSDomain 'mail.ldlnet.net' -CloudServicesMailEnabled: $true -RouteAllMessagesViaOnPremises: $true -UseMxRecord: $false -IsTransportRuleScoped: $false |
The Intra-Organization is set as well:
1 2 3 4 5 | Get-IntraOrganizationConfiguration -OrganizationGuid '48e7bec9-404c-4d24-b59e-4b46b64d7e03' New-IntraOrganizationConnector -Name 'HybridIOC - 48e7bec9-404c-4d24-b59e-4b46b64d7e03' -DiscoveryEndpoint 'https://autodiscover-s.outlook.com/autodiscover/autodiscover.svc' -TargetAddressDomains 'LDLNET.mail.onmicrosoft.com' -Enabled: $true New-IntraOrganizationConnector -Name 'HybridIOC - 48e7bec9-404c-4d24-b59e-4b46b64d7e03' -DiscoveryEndpoint 'https://mail.ldlnet.net/autodiscover/autodiscover.svc' -TargetAddressDomains 'ldlnet.net' -Enabled: $true |
9. The HCW configures OAuth Authentication across the Hybrid
This LINK explains how OAuth is configured between Exchange On Premises and Exchange Online. It’s a very good article to read as it shows how to get the Modern Authentication style working. Now the HCW does this for you and at the end of the article, you can run cmdlets to test the validity of the configuration.
If you want to go into a deep dive about how the Hybrid Authentication works, see the following:
Deep Dive Into Hybrid Authentication – from the MS Exchange Team Blog
Here are some of cmdlets run during this process workflow:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | Get-AuthConfig Get-ExchangeCertificate -Thumbprint F6C56C57EA7C86C3963FXXXXXXXXXXXXXXXXXXX Get-ActiveSyncVirtualDirectory -ADPropertiesOnly: $true Get-PartnerApplication -Identity 'Exchange Online' Set-PartnerApplication -Identity 'Exchange Online' -Enabled: $true New-AuthServer -Name ACS -AuthMetadataUrl 'https://accounts.accesscontrol.windows.net/663cad0b-70f7-42a5-b411-9d63337f0130/metadata/json/1' Get-AuthServer New-AuthServer -Name EvoSts -AuthMetadataUrl 'https://login.windows.net/LDLNET.onmicrosoft.com/federationmetadata/2007-06/federationmetadata.xml' -Type AzureAD |
Again, look at both of those links to get a little more detail as to what each cmdlet does and how it sets up OAuth. Here are the two cmdlets used to test OAuth:
1 | Test-OAuthConnectivity -Service EWS -TargetUri https://outlook.office365.com/ews/exchange.asmx -Mailbox onprem@ldlnet.net -Verbose | fl |
1 | Test-OAuthConnectivity -Service EWS -TargetUri https://mail.ldlnet.net/metadata/json/1 -Mailbox ExOnlineMbx@ldlnet.net -Verbose | fl |
10. Enable MRS Proxy for Migration
In order to be able to move mailboxes between Exchange On Premises and Exchange Online, you have to enable the Exchange Web Services Virtual Directory to use the MRSProxy (Microsoft Replication Service proxy). You also have to set your EWS Virtual Directory to use Basic Authentication. You’ll want to do this before running the HCW or else you will receive the following error when the HCW validates the Migration setup and configuration:
Microsoft.Exchange.Migration.MigrationServerConnectionFailedException: The connection to the server ‘mail.ldlnet.net’ could not be completed. —> Microsoft.Exchange.MailboxReplicationService.RemoteTransientException: The call to ‘https://mail.ldlnet.net/EWS/mrsproxy.svc’ failed. Error details: The HTTP request was forbidden with client authentication scheme ‘Negotiate’. –> The remote server returned an error: (403) Forbidden.. —> Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The HTTP request was forbidden with client authentication scheme ‘Negotiate’. —> Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The remote server returned an error: (403) Forbidden.
Some of the cmdlets run to test Migration and MRS Proxy Settings are as follows:
1 2 3 4 5 6 7 8 | Test-MigrationServerAvailability -ExchangeRemoteMove: $true -RemoteServer 'mail.ldlnet.net' -Credentials (Get-Credential -UserName DOMAIN\MigrationAdminAccount) Get-MigrationEndpoint New-MigrationEndpoint -Name 'Hybrid Migration Endpoint - EWS (Default Web Site)' -ExchangeRemoteMove: $true -RemoteServer 'mail.ldlnet.net' -Credentials (Get-Credential -UserName DOMAIN\MigrationAdminAccount) Get-WebServicesVirtualDirectory -ADPropertiesOnly: $true |
11. Final HCW Configuration and cleanup.
The HCW runs from final cmdlets to finish up the installation of the Hybrid environment. Here are the cmdlets run:
1 2 3 | Get-OnPremisesOrganization Set-OnPremisesOrganization -Identity '48e7bec9-404c-4d24-b59e-4b46b64d7e03' -Comment 'Bunch of letters and numbers' |
All this information was found in the setup logs that are in the following directory
C:\Users\%username%\AppData\Roaming\Microsoft\Exchange Hybrid Configuration
REFERENCES
Understanding Federation
Understanding Federated Delegation
Create a Federation Trust
Hybrid deployment prerequisites
Exchange Specific OAuth 2.0 Protocol Specification
Understanding WS-Security
JSON Web Tokens
Using OAuth2 to access Calendar, Contact and Mail API in Office 365 Exchange Online
Configurable token lifetimes in Azure Active Directory (Public Preview)
OAuth Troubleshooting
Principles of Token Validation
Troubleshooting free/busy issues in Exchange hybrid environment
How to configure Exchange Server on-premises to use Hybrid Modern Authentication
Microsoft 365 Messaging Administrator Certification Transition (beta)
Microsoft 365 certification exams
Exchange Server build numbers and release dates
March 2020 Updates to the HCW
PLEASE LEAVE QUESTIONS, COMMENTS, UPDATES! I WOULD LOVE TO HEAR FROM YOU!
This an EXCELLENT (likely the best) account of the actions taken behind the scene by the HCW! Thank you very much for taking the time to share and update your work! MM_MikeM