Lifting EWS Throttling for Exchange Online Mailbox Migrations now in the M365 Support Assistant

In my experience with doing Exchange migrations to the cloud, there always seemed to be an issue with EWS throttling causing very slow mailbox moves. The remediation to this was to always contact Microsoft EXO Support and open a ticket to request that your EWS throttling for your tenant be lifted so you could move your mailboxes more quickly.

The reason for the EWS Throttling was to keep large amounts of data from flooding the front end server farm possibly causing a temporary outage or corruption of data going to the cloud since there are literally thousands of customers going through the same server farm and possibly migrations. Throttling was a way to keep everything in check.

Microsoft recently made an interesting change to the automated support handling capabilities of the Microsoft 365 admin center to handle requests for Exchange Web Services (EWS) throttling to be lifted for up to 90 days without human intervention.

Here is how to request the throttling be lifted via the M365 Support Assistant:

  • Go to the Help (?) section of the Microsoft 365 admin center.
  • Click the Need Help icon.
  • Enter “EWS throttling” as the search phrase.
  • Click Run tests when asked to check your environment. Essentially, the tests check what EWS throttling applies to the tenant.
Getting EWS Throttling Support
Running the Tests from the Admin Center
  • The support assistant checks the tenant settings and concludes that EWS is throttled (the normal situation). You’ll then be offered the chance to update the settings to the tenant EWS policy to lift throttling for 30, 60, or 90 days.
  • Select the number of days you’d like to adjust the policy for and then Update Settings.
  • After a short delay, the support assistant should confirm that the settings have been changed.
Select the number of days to lift throttling and click Update Settings

Once the setting is changed in the tenant, it will be effective after about 15 minutes for replication to the server farm. You should then be able to run your migrations at full speed.

Settings Changed Successfully

NOTE: Changing this setting is only effective for EWS migrations and NOT IMAP Migrations (such as from G-Mail).

Having this option online in the portal saves a tremendous amount of time when working to get your Exchange migrations to the cloud completed. Hopefully there will be more to come. I just hope it doesn’t keep me out of a job!

I AM CURRENTLY LOOKING FOR NEW OPPORTUNITES!
POSITIVE MINDSET, POSITIVE ATTITUDE!

REFERENCES:
Microsoft Automates EWS Throttling

Troubleshooting OOF/OOO (Out of Office) Replies in Exchange and Exchange Online

Since the COVID-19 virus has managed to get me laid-off and not working, I have not had too much to post in the past months. And although it has wavered greatly, I will remain positive and hope that someone will see the value that I can bring to their organization in the near future.

With that said, I wanted to repost this article that was sent to me as this issue arises quite often in the workplace when someone has left for vacation and is not in the office. So, here it is:

Understanding and troubleshooting Out of Office (OOF) replies

Out of Office replies can sometimes be a bit of a mystery to people; how do they work? What do you do if they don’t work? In this blog post, we will discuss the bits and pieces of Out of Office and some of the main reasons why an Out of Office (aka. OOF) reply might not get delivered to users. Note that while we are writing this from the viewpoint of an Exchange Online configuration, many of same things can be applied to on-premises configuration also. By the way – did you ever wonder why “OOF” is used instead of “OOO”? If you did, see this!

What is an Out of Office reply?

Out of Office replies are also known as OOF (or OOO) replies or automatic replies. They are Inbox rules that are set in the user’s mailbox by the client. OOF rules are server-side rules, so the response is sent regardless of whether a client is running, or not.

There are several ways of setting up automatic replies:

First, it can be set up as an automatic reply feature from Outlook, like this. It can also be configured using other clients, such as Outlook on the web (OWA), PowerShell command (Set-MailboxAutoReplyConfiguration). Admins can set up OOF replies on behalf of (forgetful) users from the M365 Admin Portal.

Other than using built-in OOF functionality, another thing people sometimes do is use rules to create an Out of Office message while they are away.

By design, Exchange Online Protection uses the high risk delivery pool (HRDP) to send the out of office replies, because they are lower priority messages.

Types of OOF rules

There are three types of OOF rules: Internal, External and Known Senders (Contact list). They are stored in the mailbox with the names in the following table. If a mailbox has set only Internal OOF, there will be no external rule created and the mailbox will have one OOF rule.

TypeMessage ClassPR_RULE_MSG_NAME
InternalIPM.Rule.Version2.MessageMicrosoft.Exchange.OOF.KnownExternalSenders.Global
ExternalIPM.Rule.Version2.MessageMicrosoft.Exchange.OOF.AllExternalSenders.Global
Known ExternalIPM.ExtendedRule.MessageMicrosoft.Exchange.OOF.KnownExternalSenders.Global

Note: Apart from OOF rule, other rules like the Junk Email rule will also have the “IPM.ExtendedRule.Message”  message class; the MSG_NAME will determine what the rule is for.

OOF rule details

All Inbox rules can be viewed using MFCMapi tool:

Logon > profile that you are accessing > Top of information store > Inbox > right click ‘Open associated contents table’. They are listed under the Message Class column. All Inbox rules will have the same message class IPM.Rule.Version2.Message and there is one message class and name for each Inbox rule.

For all rules, the name of the rule is in stored in the PR_RULE_MSG_NAME property. So, if there are 4 Inbox rules, there will be 4 IPM.Rule.Version2.Message one for each rule, and the name of the rule is stored in PR_RULE_MSG_NAME.

OOF rules in MFCMapi:

OOF01.jpg

And OOF rule templates in MFCMapi:

OOF02.jpg

History of OOF replies

OOF response is sent once per recipient. Recipients to whom the OOF was sent are stored in the OOF history and are cleared out when the OOF state changes (enabled/disabled) or the OOF rule is modified. OOF history is stored in the user’s mailbox and can be seen using MFCMapi tool at: Freebusy Data > PR_DELEGATED_BY_RULE.

OOF03.jpg

Note: If you want to send response to the sender every time instead of just once, you can apply mailbox server side rule “have server reply using a specific message” to send automatic reply instead of using the OOF rule. This server-side rule will send reply to the sender every time a message is received.

Now that we know what OOF replies are and how they are stored on the server, we can move on to address some of the scenarios where OOF is not sent to the sender. We will also discuss possible fixes and some more frequently seen issues you may have with OOF configuration.

The first category of issues we will talk about are issues related to OOF replies not being received by the sender of the original message.

OOF issues related to transport rules

When OOF doesn’t seem to be sent for all users in the tenant, usually there is a transport rule causing the issue. Check all the transport rules that may apply to the affected mailbox using step two of this article.

If you suspect a delivery problem, run a message trace from the Office 365 tenant.  We know that OOF response is sent back to the original sender of the message, so for OOF messages, the sender of the original message becomes the recipient when tracking. We should then be able to tell if OOF reply has been triggered and sent to external or internal recipient. If a Transport rule is blocking the OOF response, the message trace will clearly show you that.

There is one scenario I would like to highlight when it comes to transport rules blocking OOF replies. Let’s assume that you moved the MX record to a 3rd party anti-spam solution; you have created a transport rule to reject any email coming from any other IP address than the 3rd party anti-spam.

The transport rule will look something like this:

Description:
If the message: Is received from ‘Outside the organization’ Take the following actions: reject the message and include the explanation ‘You are not permitted to bypass the MX record!’ with the status code: ‘5.7.1’ Except if the message: sender ip addresses belong to one of these ranges: ‘1xx.1xx.7x.3x’

ManuallyModified: False

SenderAddressLocation: Envelope

As OOF replies have a blank (<>) Return-Path, you will see that the rule is unexpectedly matching the transport rule and the OOF responses are getting blocked.

In order to fix this, you can change the transport rule property of ‘Match sender address in message’ to ‘Header or envelope’, so the checks will also be done against ‘From’ (also known as the ‘Header From’ address), ‘Sender’ or ‘reply-to’ fields. More information about the mail flow rule conditions is here.

OOF04.jpg

JournalingReportNdrTo mailbox setting

If the affected mailbox is the one which is configured under JournalingReportNdrTo, OOF replies will not be sent for that mailbox. Moreover, journaling emails may be affected as well. It is recommended to create a dedicated mailbox for the JournalingReportNdrTo setting. Alternatively, you can set it to an external address.

For more details on how to solve this, please see this KB Article.

Forwarding SMTP address is enabled on the mailbox

If the affected user mailbox has SMTP forwarding enabled, OOF replies won’t be generated. This can be checked in user mailbox settings (OWA):

OOF05.jpg

In PowerShell:

OOF06.jpg

Or, in the M365 Portal, user properties:

OOF07.jpg

Please follow the action on step one of this article for more information.

The type of the OOF reply set on remote domains

Remote domains offer you (among other settings) the opportunity to set the type of OOF reply that can be sent to users.

These types are the following:

  • External
  • ExternalLegacy
  • InternalLegacy
  • None

For more information about each of these OOF types, please refer to AllowedOOFType parameter in our Set-Remotedomain document.

The Out of Office type can be checked from Exchange Admin Center > Mail flow > Remote domains

OOF08.jpg

Or, with PowerShell:

OOF12.jpg

You need pay attention to what OOF type you have set up, as this will impact the OOF response and OOF may not be generated at all if the configuration is incorrect. Let’s assume you have a hybrid organization with mailboxes hosted both in Exchange on-premises and Exchange Online. In this scenario, by design only external messages will be sent to on-premises while AllowedOOFType is set to External. To be able to send internal OOF messages to on-premises in hybrid environment, you need to set the AllowedOOFType to InternalLegacy.

You also have option to send external Out of Office replies only for contacts at the mailbox configuration level (ExternalAudience: Known). This can make automatic replies not being sent to anyone external but contacts. The command to check the configuration is:

OOF10.jpg

Remote domain blocking OOF replies

Another setting on remote domains is one which lets you dictate whether you allow or prevent messages that are automatic replies from client email programs in your organization.

This can be found in Exchange Admin Center > Mail flow > Remote domains

OOF11.jpg

Or by running this PowerShell cmdlet:

OOF12.jpg

Note: If this option is set to false, no automatic replies will be sent to users for that domain. This setting takes precedence over the automatic replies set up at the mailbox level or over the OOF type (discussed above).

Please, keep in mind that $false is the default value for new remote domains that you create as well as the built-in remote domain named Default in Exchange Online.

If the email was marked as spam and sent to junk, an automatic reply will not be generated at all.

Pretty self-explanatory, that one!

Message trace shows delivery failure

If you investigate an OOF reply issue and in the message trace you find the following error message:

“550 5.7.750 Service unavailable. Client blocked from sending from unregistered domains.”

You should reach out to Support to find out why the unregistered domain block was enforced.

There are some other scenarios that might come up when working with OOF replies, let’s cover those next!

An old or duplicate OOF message is sent

This is likely due to a duplicate Inbox rule or the OOF history limit. The OOF history has a limit of 10,000 entries, if this threshold is hit, OOF will continue to be sent to recipients that are not already in the list as any new users can’t be added to the list. All users already in the list will not receive duplicate OOF replies. For more information you may want to check this article or follow the action plan below.

  • Remove the OOF rules and the OOF rule templates from the mailbox. To locate the rules and delete them go back to > “Inbox OOF rules”
  • Disable and then re-enable the OOF feature for the mailbox

Now, you can check again whether the OOF feature works as expected and symptoms do not occur.

Automatic replies cannot be enabled; an error message is received

While attempting to access automatic replies from the Outlook client, an error message is received saying that “Your automatic reply settings cannot be displayed because the server is currently unavailable. Try again later.”

To narrow down this issue, you should perform the following steps:

  • Confirm EWS protocol is enabled on the mailbox as OOF replies rely on it to be enabled (note that re-enabling this might take several hours to take effect)
  • Enable the OOF feature by using the following command:Set-MailboxAutoReplyConfiguration <identity> -AutoReplyState Enabled
  • Check whether the OOF feature works as expected.
  • If the issue is still there, review the rules quota on the mailbox: Get-mailbox -identity <mailbox> | fl RulesQuota
OOF13.jpg

By default, the RulesQuota has a maximum quota which is calculated by the size of the rules (not the number of rules). Maximum is 256 KB (262,144 bytes).

  • Remove the OOF rules and the OOF rule templates from the mailbox. To locate the rules and delete them go back to > “Inbox OOF rules”
    After you remove them, you can re-enable the OOF feature and then test again.

An automatic reply is still sent despite OOF being disabled

We have encountered scenarios in which OOF messages are still being sent, although it is disabled. Most of the time, we found that the rule is created manually by the end users using the out-of-office template.

So, as you can see, there is quite a bit that goes into troubleshooting OOF Replies and it is not all straight forward.

THANKS FOR READING!
I’M AVAILABLE FOR WORK!
PLEASE CONTACT ME FOR AN APPOINTMENT!

REFERENCES:
Troubleshooting OOF Replies (Exchange Team Blog)

How to address Federation Trust issues in Hybrid Configuration Wizard (HCW)

During my time as a PFE for Microsoft, I have encounted many issues with Federation in a Hybrid Exchange Deployment. Recently, the following support announcement came out and I wanted to share as I hope this can help others that may be having issues out there.

One of the more common causes of HCW failures is the Federation Trust step for the Exchange on-premises organizations in Full hybrid configurations (Classic or Modern topologies).

Federation trust is a mandatory step in the on-premises Exchange organizations when configuring Full hybrid deployments, as this allows us to create organization relationships (for features like hybrid free/busy or OWA/EAS redirection) and sharing policies (1:1 hybrid calendar sharing). In Exchange Online multi-tenant organizations, federation trust is already in place.

Below is an illustration of an Exchange hybrid deployment where both the Exchange on-premises organization and the Exchange Online organization have a trust with Azure Authentication System (formerly called Microsoft Federation Gateway):

Example of Hybrid Federation

Before getting to our subject, let’s quickly go over different hybrid configurations and Hybrid Configuration Wizard (HCW) – as this is the supported tool to configure hybrid deployments.
There are 2 flavors of hybrid configurations:
Classic hybrid
Modern hybrid

At this time, each of those supports the following hybrid modes:

  • Full
  • Minimal (which further breaks down into…)
    • Express (a one-time sync)
    • “Actual minimal”

A quick overview of Full / Minimal / Express options, can be found here. More info on HCW is here.

As mentioned earlier, a federation trust is created by HCW only in Full Hybrid.

HCW logs are located at %appdata%\Microsoft\Exchange Hybrid Configuration on the machine from where HCW was ran. The easiest way to get to them is to press F12 in the HCW window to open the Diagnostic tools and from there you can Open Folder Logging or Open Log File directly.

When you have issues with federation trust, the log will usually show errors when one of the following cmdlets are executed:

Set-FederationOrganizationIdentifier
or
Add-FederatedDomain (but can be other cmdlets as well).

Once you identified the exact cmdlet failing and where (Session=OnPremises – means Exchange Management Shell and Session=Tenant means Exchange Online PowerShell), you should copy-paste the failing command and try to execute it manually and see if that is failing as well (most likely it will). You can also open the shells from F12 Diagnostic tools windows in HCW.

In order to get more details on the error and to rule out this is not an issue with HCW itself, you will need to separately run the same command that threw exception in HCW log and add Verbose switch to get verbose details of the error and the serialized remote exception.

For example, if the Exchange server version is Exchange 2010, you will run the failing command with Verbose switch in Exchange Management Shell (EMS), see if that fails and then get the serialized remote exception.

Example:

If the Exchange Server version is Exchange 2013/2016 and the above commands didn’t show more details on the error, we can also try the following:

  • Open regular Windows PowerShell (blue background) on the Exchange Server 2013/2016
  • Run command: add-pssnapin exchange
  • Run command that gave error in HCW and add a Verbose switch

Example:

Once you’ve gathered the verbose error / serialized exception, try to understand where it is failing (or provide it to Microsoft Support together with the HCW log).

Common Errors with Remediation Steps

  • Federation trust fails with “Object reference not set to an instance of an object”

This is a known old issue on Exchange 2016 CU7 servers, make sure your Exchange servers are updated to the latest CU.

Full error in the HCW log:

Resolution: Install the latest CU for Exchange 2016


  • Federation fails with “Proof of domain ownership has failed”

Full error in the HCW log:

Resolution:

• Check the TXT record for your domain(s) in HCW log or in Exchange Management Shell with command Get-FederatedDomainProof -DomainName
• See if it matches your published TXT record with either nslookup utility or by checking internet websites like https://www.whatsmydns.net/ put your domain in hostnames, type=txt, Nameservers – Authoritative

You would look for errors, missing records or unusual formatting (characters, spaces, quotes, TXT record split in half).


  • Federation fails with “An unexpected error occurred on a receive” or “An unexpected error occurred on a send.”

Full error in the HCW log:

Resolution:

Check outbound access from all your Exchange Servers to Microsoft Federation Gateway by browsing using Internet Explorer with PSEXEC tool (with -s and -i switches) from the Exchange Server (this will use Internet Explorer under System Account / Exchange Server Account).

In this example, “Windows Live” is actually this exact URL: https://domains.live.com/service/managedelegation2.asmx

From on-premises Exchange to Office 365, the Exchange 2010 MBX & CAS or 2013 MBX (backend) or 2016 / 2019 would need outbound Internet access to the Microsoft Federation Gateway in addition to https://outlook.office365.com/ews/exchange.asmx

Verify the machine/system account can access these Microsoft Federation Gateway URLs:

For a complete list of O365 URL & IP addresses, see these articles:

Note: If the Exchange requires a proxy server to access the Internet, specify the proxy server using “Set-ExchangeServer myExchange01 -InternetWebProxy http://myproxy:80”. Notice such proxy can’t require any user authentication for outbound Internet access, and the proxy must start with HTTP: and not HTTPS: (secure SSL).

You can also set the proxy using netsh as well.

set proxy proxy-server=”http=myproxy;https=sproxy:88″ bypass-list=”*.contoso.com” 

In rare instances, you can use the machine/system account to access the URLs from the browser, but Exchange cmdlets still failed with “Could not establish trust relationship for the SSL/TLS secure channel.” If that happens, make sure the certificate authorities for the urls are installed at the Third-Party Root Certification Authorities of the machine local certificate location.

REFERENCE:
Netsh Commands for Windows Hypertext Transfer Protocol (WINHTTP)
Firewall Considerations for Federated Delegation 

Federated delegation features require that the Mailbox and Client Access servers in your organization have outbound access to the Internet by using HTTPS. You must allow outbound HTTPS access (port 443 for TCP) from all Exchange 2010 Mailbox and Client Access servers in the organization.


  • There is no specific error / exception, in HCW log you would see it stops without any specific error.

Full error in the HCW log:

Resolution:

Look for orphaned federation trust:

Get-FederatedOrganizationIdentifier | FL

or

in HCW log if you see something with “DEL“: “contoso.com/Configuration/Deleted Objects/Microsoft Federation Gateway/DEL: <xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx>”.

Solution is to remove the orphaned federation trust and re-run HCW.

Reference here.

NOTE: as a first step, you can try to run the command remove-federateddomain with the switch -Force. Also, you don’t need to recreate federation trust manually, just re-run HCW (this will recreate federation trust for us)


  • Federation Trust fails with “InternalError InternalError: Internal error.”.”.””

Full error in the HCW log:

Resolution:

Open request with Microsoft Support or check if any Service Incident is published. Please see this for more information.


  • Federation trust fails with “1007 Access Denied”

Full error in the HCW log:

Resolution:

“1007 Access Denied” error is usually when we have issues with:

  1. Windows Time on the Exchange Server. See this article or this article.
  2. Outdated federation trust (for example, federation trust certificate expired) and in this case you would remove federation trust by following these steps.

If the federation trust certificate is not found on any of the servers, then proceed with resolution from the next error.

As an example, from one HCW log, there seems to be this federation trust certificate expired on 05/13/2019:


  • Federation trust fails with “Federation Certificate cannot be found”

Full error in the HCW log:

Resolution:

Follow the procedure here to manually cleanup the federation trust from AD. Once this is done, re-run the HCW to re-create it automatically.

KEEP TROUBLESHOOTING!
REMAIN VIGILANT USING FOCUSED INTENT, NEVER EMOTIONALIZING (FINE)!

REFERENCE:
How To Address Federation Trust Issues using the Hybrid Configuration Wizard

Importing User Photos to Office 365 in bulk for your company.

In a previous post, I showed how you could update one user’s photo for their Outlook and AD profiles via PowerShell. In this post, we will explore how to do this for your entire organization via PowerShell to Office365.

NOTE: I have not tested the scripts as I do not have enough mailboxes in my O365 tenant along with not using a ‘.’ in my alias. If the scripts are incorrect, please inform me with the correction and I will update accordingly.

Please make sure that your photos are reviewed before posting, and try to keep the file size of the photos to a minimum. In Office 365, there exists a limitation for the user photo not to be more than 10 KB in size, but I will show you how to get around that limitation.

Having a user photo for each of your users is very beneficial as it personalizes each account to a face in the company. The user photos can be viewed in below locations:

  • Outlook Web Access
  • Contact Card
  • Thumbnail in emails
  • Outlook Client
  • Yammer
  • Lync Client
  • SharePoint (People Search / Newsfeed)

Steps to take:

  1. Remove the 10KB photo size limitation in Exchange Online
  2. Prepare a folder with all users photos
  3. Update the profile photos via a PowerShell cmdlet.

Connect to Exchange Online with the RPS Proxy Method to remove the 10K size limitation

NOTE: In the PowerShell cmdlet above, we connected using a different proxy method. This was to overwrite the limitation of uploading the images with size more than 10KB. Using the different proxy method (/?proxyMethod=RPS ) to connect to Office 365 in the above cmdlet accomplishes this.

Prepare a folder locally and place all the photos in that folder

Create a folder named C:/UserPics and make the filename of each photo be the username of that particular user. (i.e. llingerfelt.png)
The below script should be able account for aliases that have a ‘.’ in the id as well. (i.e. lance.lingerfelt)

NOTE: From my research, there is no set photo type that is required for the photo. My suggestion would be to keep the photos .png for size constraints while maintaining picture clarity.

Update the profile pictures via PowerShell

Create the following script and name it Photos-Update.ps1

Run Photos-Update.ps1 and the script should upload the photos to Office 365 and apply each photo to the corresponding user.

NOTE: If you’re still having some issues with the alias having a ‘.’ in the name, you can also configure the Photos-Update.ps1 script in this manner to get that working properly:

HAPPY SCRIPTING!
PLEASE COMMENT!

store.ldlnet.net
LDLNET LLC! Your Source for Pofessional IT Services!
www.servermonkey.com
Contact ServerMonkey.com for your IT Hardware Needs!

REFERENCES:
How to import Office365 User photos over 10KB & without CSV in bulk

Set the profile pic for a single Exchange user via PowerShell

I wanted to update my picture within my Outlook profile and AD account really quickly without having to go through OWA to do so. I found this cmdlet that will allow for that picture to be changed very quickly via Exchange PowerShell.

NOTE: This can be done with On-Premises Exchange and Exchange Online PowerShell

Old picture within my account

First, download the picture you want to use to the computer that you want to run the cmdlet from. Also, make sure the picture is cropped and centered prior to running the cmdlet. I saved the pic to C:\temp for my scenario. The best format to use would be jpg. I named the file User1_Profile.jpg

Next, open Exchange PowerShell on the computer you saved the pic to and run the following cmdlet to change the photo:

Once completed, the Outlook client should be closed and reopen so that the new picture is visible in the profile.

Picture change completed

I will post how to perform this for multiple users for Exchange and Office365 in a later post.

REFERENCES:
Set User Photo with Exchange PowerShell

Exchange Hybrid Configuration Wizard Link

Wanted to do a quick post as I was working on my Hybrid Exchange Environment. I was unable to get the HCW to download and start from the Exchange Control Panel with the link provided on the page. This has happened to me for a while, so I went online and found a link that would work that could be downloaded and reused to open the HCW:

Hybrid Configuration Wizard Link

HOPE THIS HELPS!
LET ME HAVE KNOWLEDGE SHOULD THE LINK CHANGE!

References:
HYBRID CONFIGURATION WIZARD WON’T START ON WINDOWS 2016

Reconnecting Shared Mailboxes after an O365 Migration

I get a lot of these incidents in my queue after a user has been migrated to O365. For whatever reason, most likely due to the mailbox being moved itself, whether it is the user’s mailbox, the shared mailbox, or both, the connections to the shared mailboxes stop working in Outlook and the user cannot connect to the shared mailbox.

Here is a quick and easy solution to use to disconnect and reconnect the shared mailbox(es) that you lose connectivity to when migrated. This is usually performed on Outlook 2016 and above as most users upgrade their client software when moved to O365.

First, we remove the existing shared mailbox connection:

  • Click the File > Account Settings > Account Settings.
  • Select your company email address in the account list.
  • Click Change > More Settings > Advanced tab > Select the Shared Mailbox > Remove
  • Click Apply > OK > Next > Finish.
  • The shared mailbox will now automatically be removed in your Folder pane in Outlook.

Second, we re-add the shared mailbox connection to Outlook:

  • Click the File > Account Settings > Account Settings.
  • Select your company email address in the account list.
  • Click Change > More Settings > Advanced tab > Add
  • Type the name of the shared mailbox in the window and click OK.
  • Click Apply > OK > Next > Finish.
  • The shared mailbox will now automatically be added to your Folder List pane within Outlook.

Note: The above procedure must be followed in order to properly reconnect the shared mailbox. You cannot remove and re-add the mailbox in the same process as that will not reset the connection properly. You must save the settings when disconnecting.

I hope that this will assist everyone when troubleshooting Outlook connectivity issues to shared mailboxes after a migration.

HAPPY TROUBLESHOOTING!
PLEASE COMMENT!

Moving mailboxes to O365 via PowerShell in Hybrid Configuration

As many of you have knowledge, I am studying for my MS-202 Exam. And, part of the knowledge needed is to be able to migrate mailboxes between on premises and Exchange Online through PowerShell. Here are the steps for the scenario to move a mailbox from on premises to O365:

1. Connect to Exchange Online via PowerShell

If you have read my previous post: Connect to All PowerShell Modules in O365 with one script
You should have all the settings needed to connect your PowerShell to O365. Note in this scenario, that all these cmdlets will be run from O365 PowerShell and will be monitored from O365 by either PowerShell or the Exchange Admin Center. You will not be able to monitor the moves from On-Premises.

2. Provide your on premises Migration Administrator credentials as a variable for your cmdlet.

3. Move a single mailbox.

In your hybrid configuration you should be doing directory sync with O365/Azure and the accounts should be available in the cloud showing that they are synced with AD. This also assumes that you have your MRS Proxy endpoint enabled, which can be done by the HCW. Also, make sure you have your licensing available for your mailboxes. From my knowledge, you can assign your license to the account in the cloud before moving, especially if you have a particular license that you need to assign the account. Other than that, moving the mailbox will assign an existing license that is available that includes an Exchange Online mailbox feature when the mailbox is moved.
Now we initiate the move with the cmdlet. Similar to what you would do in the GUI, this simple mailbox move cmdlet initiates the move request. It has most of the same parameters as a local move request including BadItemLimit, LargeItemLimit, AcceptLargeDataLoss, etc…
 
Use the following LINK for documentation on the New-MoveRequest cmdlet.

Now with all migration projects, we expect to have to move multiple mailboxes in a single batch. The following will show the process for moving mailboxes in bulk from on premises to O365:

1. Connect to Exchange Online via PowerShell

If you have read my previous post: Connect to All PowerShell Modules in O365 with one script
You should have all the settings needed to connect your PowerShell to O365. Note in this scenario, that all these cmdlets will be run from O365 PowerShell and will be monitored from O365 by either PowerShell or the Exchange Admin Center. You will not be able to monitor the moves from On-Premises.

2. Provide your on premises Migration Administrator credentials as a variable for your cmdlet.

3. Move multiple mailboxes in a single batch.

In your hybrid configuration you should be doing directory sync with O365/Azure and the accounts should be available in the cloud showing that they are synced with AD. This also assumes that you have your MRS Proxy endpoint enabled, which can be done by the HCW. Also, make sure you have your licensing available for your mailboxes. From my knowledge, you can assign your license to the account in the cloud before moving, especially if you have a particular license that you need to assign the account. Other than that, moving the mailbox will assign an existing license that is available that includes an Exchange Online mailbox feature when the mailbox is moved.

This time you want to create a CSV file using the alias or emailaddress as your header and then list the appropriate value for all the users in your batch group. Save the file locally as MigrationBatch01.csv or a name of your choice.

Use EMailAddress
 OR
 Alias as the header

Next you initiate the mailbox moves. When specifying the mailbox identity in the cmdlet, use the respective header in your variable declaration (either $user.EMailAddress OR $user.Alias)

Use the following LINK for documentation on the New-MoveRequest cmdlet.

GOOD LUCK WITH YOUR MIGRATIONS!
HAPPY TROUBLESHOOTING!

References:
Moving Individual Mailboxes to O365
Move Mailboxes in Bulk to O365
PowerShell Mailbox Migration to O365
Connect to all PowerShell Modules in O365 with one script
New-MoveRequest Microsoft Document

Connect to all PowerShell Modules in O365 with one script

Let’s say you’re an admin that needs to connect to Office365 via PowerShell often. Now, there are many different websites or blogs that will show you how to connect to each session via PowerShell. That can cause a headache since you can end up having five different PowerShell sessions running in five different windows. You end up having to enter a username and password all those times, which can become time consuming.

I want to show you here how to combine all those sessions into one script where, if you’re security is tight enough on your computer, you don’t even have to enter credentials. This way, you can click on one icon and pull up all the O365 PowerShell commands that you’ll need to manage your organization.

First you need to download the following PowerShell Module Installation Files so that your PowerShell Database will have the correct modules installed:

Microsoft Online Service Sign-in Assistant for IT Professionals RTW
Windows Azure Active Directory Module for Windows PowerShell v2
SharePoint Online Management Shell
Skype for Business Online, Windows PowerShell Module

Next, we want to setup the CLI (Command Line Interface) to be too cool for school. I have learned it helps to have knowledge of how to customize the CLI window. You can do all of this in PowerShell ISE or Notepad, which ever you prefer. Here are the commands for the script that I use to setup the CLI:

Next, you want to set your Execution Policy and put in your credentials so that you won’t be prompted to enter the user credentials when you run the script.

NOTE: MAKE SURE YOU KEEP YOUR SCRIPT SAFE AS THE CREDENTIALS ARE VISIBLE WITHIN THE SCRIPT IN PLAIN TEXT!

You can, alternatively, set your script to prompt for credentials every time by using the following:

$LiveCred = Get-Credential

Here is that part of the script:

Now we get into the importing of the modules for each O365 service:

Get the MSOnline Module:

Connect to the MSOnline Service:

Connect to Azure AD PowerShell:

Connect to SharePoint Online PowerShell:
NOTE – MAKE SURE YOU CHANGE TO YOUR COMPANY NAME IN THE URL!!

Connect to Exchange Online PowerShell:

Connect to Skype For Business Online PowerShell:

Connect to the Security & Compliance PowerShell:
NOTE – This one I still get “Access Denied” when trying to connect. I have looked for an answer to that issue, but have not found one. Please comment with a link if you have an answer so that I can update this script!

Lastly, put in a note to show that the PS load is completed:

So Here is the final script in its entirety:

Now you can create your icon for your desktop so that you can easily access the script. I would save the script to your Scripts directory.

That will usually be C:\Users\’username’\Documents\WindowsPowerShell\Scripts or wherever directory you choose.

To start, right click the desktop and choose New > Shortcut
In the Target Field, enter the following for your PowerShell Shortcut, pointing to the path of your script:

C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -noexit -ExecutionPolicy Unrestricted -File “C:\Users\username\Documents\WindowsPowerShell\Scripts\ConnectO365All.ps1”

Click on the Advanced button and check the box: Run As Administrator
Under the General Tab, name your shortcut: (CompanyName) O365 All PowerShell
Click OK to save the shortcut to your desktop.

LAST BUT NOT LEAST, RUN THE FOLLOWING COMMAND BEFORE EXITING OR CLOSING YOUR POWERSHELL WINDOW. THIS WILL REMOVE ALL THE SESSIONS YOU’VE CONNECTED TO:

Get-PSSession | Remove-PSSession

HAPPY SCRIPTING!
LEARN, DO, LIVE!

References:
Connect to all O365 Services in one PowerShell Window
How to connect to all O365 Services through PowerShell
Connecting to Office 365 “Everything” via PowerShell

What the Hybrid Configuration Wizard Performs in the background and configuring Hybrid Co-Existence with Exchange Online

****UPDATE 3/23/2020****

Changes have been made to the HCW and the installation since this original post. Please read the following to gain knowledge of the updates to the tool and the installation.


March 2020 significant update to Hybrid Configuration Wizard

We wanted to let you know that we are releasing what we consider a significant update to Exchange Hybrid Configuration Wizard (HCW). Along with a handful of small bug fixes, there are four major changes coming that we wanted to share with you:

  1. HCW will no longer enable Federation Trust by default for all installations. Instead, it will only enable Federation Trust if there are Exchange 2010 servers on premises. HCW will call Get-ExchangeServer and if no Exchange 2010 servers are reported, the workflow to enable Federation Trust and subsequently require domain proof will not execute. Note that organization relationships are still created.
  2. When uninstalling the hybrid agent and switching to Classic in the HCW, this action would sometimes fail with a “null reference” error. We have fixed this!
  3. How many of you have hit the HCW 8064 error – unable to configure OAuth, and subsequently had no idea why OAuth failed to configure? Yes, we heard you loud and clear! In this release, we have completely changed the way we enable and configure OAuth. Instead of enabling OAuth at the service layer, we now enable OAuth via a Graph API under the context of the Tenant Admin. This in turn removes the error obfuscation we had with the service layer enablement and allows us to include a detailed error entry in the HCW log. So while you still see the HCW 8064 error in the HCW UI, you can now review the log for the specific error detail which will make it easier to troubleshoot and resolve.
  4. When verifying DNS, we had a fallback mechanism that would reach out to an external site to verify domains. While this fallback mechanism was rarely hit, we received overwhelming feedback to not use this mechanism/site as it was not listed in our IPs & URLs web page. We have removed that fallback and now only use the endpoint “mshybridservice.trafficmanager.net”, which is listed in our endpoints documentation.

Because this is a major version update, the build begins with 17.x vs 16.x. The build number can be found in the top right corner once you download and open the HCW.

Because of the web-based distribution nature HCW uses and this version is a brand new package, you will get all this goodness simply by installing the new HCW from here. The current builds of HCW (16.x) will not automatically update to 17.x build, in fact – you could run the two side-by-side. Once you are on 17.x build – the HCW will then auto-update as usual.

A few additional notes: At this time, we do not anticipate new HCW 16.x builds. Therefore, to continue getting new HCW builds in the future, uninstall the current version of HCW (16.x) and then install the new version (17.x). The new version of HCW has a new dependency, .NET 4.7.2. The installer should take care of this for you, but just so you are aware.


ORIGINAL POST

I’m working on getting certified in Exchange Hybrid Scenarios and Exchange Online configuration as part of my skill set for Exchange. In doing so, I had successfully implemented a complete Full Hybrid Exchange Environment between my Exchange Online Tenant and my On Premises Exchange 2019 Environment last evening.

I wanted to give an update that was posted to my LinkedIn Posting on this. Thank you Brian Day for the vote of confidence and caution that running these cmdlets manually is not supported by Microsoft and that the HCW, like all the Online Microsoft Products, is constantly changing and being updated.

Important Note

As preparation, I bought some Exchange Online Plan 1 licenses which give me a 50 GB mailbox limit and basic mailbox functionality. It does not include the more advanced features such as ATP, or DLP. I am running most of those features through my On Premises Environment. I mainly wanted to be able to place mailboxes in the cloud and have a hybrid setup. My plan was to have mail flow continue through my On Premises environment so that my Exchange Server features would be used and I would not have to change any MX or SPF records. I also had my certificates in place for SSL and OWA so I would want keep mail flow routed that way, through on premises. I do want to be able to have Free/Busy lookups cross-premise so federation would have to be enabled as well. I would also have to enable the MRS proxy on my Exchange Server so that mailbox migration could be implemented cross-premise. I also have previously configured Azure AD Sync along with ADFS for Single Sign On. In my case, another server was not needed as I didn’t have enough mailboxes or real need to split my frontend and backend deployment. Running the Hybrid Configuration Wizard would not open any new ports or change any existing port traffic that was already configured on my firewall. These are just a few of the considerations that need to be looked at when considering a hybrid integration.

Here is a great article to read for the prerequisites
Exchange Hybrid Deployment Pre-requisites

So, once I had all those considerations handled in my design, I ran the Hybrid Configuration Wizard. What I want to do in this blog post is to go through the steps that the wizard does in the background to setup the Hybrid Environment as you go through the Wizard.

I mainly used the following blog post as a reference, but have approached it differently by diving into the cmdlets that are run during the process:

https://www.codetwo.com/admins-blog/office-365-hybrid-configuration-wizard-step-by-step/#validating-connection

1. The HCW validates the On-premises and Online Exchange Connection.

The Hybrid Configuration Wizard checks if it is possible to connect to both servers with PowerShell. It runs the Get-ExchangeServer cmdlet on premises after resolving the server in DNS. It then connects to Exchange Online, authorizing the connection:

Authority=https://login.windows.net/common Resource=https://outlook.office365.com ClientId=abcdefgh-a123-4566-9abc-2bdflancelin

2. The HCW collects data about Exchange configuration from the on-premises Active Directory

The Wizard gathers information about the local domain. In order to do that, the HCW executes a series of cmdlets.

These include, in order:

3. The HCW collects information on the Exchange online (Office 365) configuration

This task repeats what has been done in the previous step, only for the Exchange online, instead of the on-premises one.

The cmdlets include, in order:

4. Federation Trust is determined. If not present, a new Federation Trust and the required certificate will be created on the local Exchange Server

You will be prompted in the Wizard to create a Federation Trust if not present. The following articles explain Federation and its requirements:

Understanding Federation – Link Here
Understanding Federated Delegation – 
Link Here
Create a Federation Trust – 
Link Here

If the activity is finished successfully, a new certificate should appear on the on-premises Exchange Certificates list. The new certificate includes “Federation” in its Subject field. To make sure the certificate is there, you can run a cmdlet: Get-ExchangeCertificate | ft -a -wr


The results will look like this

5. The HCW creates a new Hybrid Configuration Object in the local Active Directory

The HCW will run cmdlets based on the information you provide in the HCW for the certificate, the on premises Exchange Server, the domain(s), and what features you want turned on:

It then checks the settings through the following cmdlets:

It then enables Organization Customization for both environments through this cmdlet:

6. Configuration is then completed to modify the settings on the on premises Exchange environment 

EmailAddressPolicy – HCW adds address @tenant.mail.onmicrosoft.com
The HCW configures remote domains – adds tenant.mail.onmicrosoft.com and tenant.onmicrosoft.com
The HCW adds a new accepted domain – adds tenant.mail.onmicrosoft.com

Some of the cmdlets run:

7. The HCW Configures the Organization Relationship between the local server and the cloud.

This configuration is not necessary in minimal hybrid deployment. Since I have a full hybrid deployment configured, the cmdlets were run as needed to configure it. Thanks to the correct configuration, it is possible to synchronize free/busy status of mailboxes and their elements between the on-premises Exchange Environment and Exchange online. 

Some of the cmdlets run in the process:

8. The HCW and setting connectors on both Exchange servers

The HCW checks to see if the connectors are there, if not, it sets them up. During this workflow, four connectors are set – one receive and one send connector for each server. Those connectors guarantee the mail flow between the on-premises and Exchange Online.

Some of the cmdlets run in the process:

The Intra-Organization is set as well:

9. The HCW configures OAuth Authentication across the Hybrid

This LINK explains how OAuth is configured between Exchange On Premises and Exchange Online. It’s a very good article to read as it shows how to get the Modern Authentication style working. Now the HCW does this for you and at the end of the article, you can run cmdlets to test the validity of the configuration.

If you want to go into a deep dive about how the Hybrid Authentication works, see the following:
Deep Dive Into Hybrid Authentication – from the MS Exchange Team Blog

Here are some of cmdlets run during this process workflow:

Again, look at both of those links to get a little more detail as to what each cmdlet does and how it sets up OAuth. Here are the two cmdlets used to test OAuth:

10. Enable MRS Proxy for Migration

In order to be able to move mailboxes between Exchange On Premises and Exchange Online, you have to enable the Exchange Web Services Virtual Directory to use the MRSProxy (Microsoft Replication Service proxy). You also have to set your EWS Virtual Directory to use Basic Authentication. You’ll want to do this before running the HCW or else you will receive the following error when the HCW validates the Migration setup and configuration:

Microsoft.Exchange.Migration.MigrationServerConnectionFailedException: The connection to the server ‘mail.ldlnet.net’ could not be completed. —> Microsoft.Exchange.MailboxReplicationService.RemoteTransientException: The call to ‘https://mail.ldlnet.net/EWS/mrsproxy.svc’ failed. Error details: The HTTP request was forbidden with client authentication scheme ‘Negotiate’. –> The remote server returned an error: (403) Forbidden.. —> Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The HTTP request was forbidden with client authentication scheme ‘Negotiate’. —> Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The remote server returned an error: (403) Forbidden.

Some of the cmdlets run to test Migration and MRS Proxy Settings are as follows:

11. Final HCW Configuration and cleanup.

The HCW runs from final cmdlets to finish up the installation of the Hybrid environment. Here are the cmdlets run:

All this information was found in the setup logs that are in the following directory
C:\Users\%username%\AppData\Roaming\Microsoft\Exchange Hybrid Configuration

REFERENCES
Understanding Federation
Understanding Federated Delegation
Create a Federation Trust
Hybrid deployment prerequisites
Exchange Specific OAuth 2.0 Protocol Specification
Understanding WS-Security
JSON Web Tokens
Using OAuth2 to access Calendar, Contact and Mail API in Office 365 Exchange Online
Configurable token lifetimes in Azure Active Directory (Public Preview)
OAuth Troubleshooting
Principles of Token Validation
Troubleshooting free/busy issues in Exchange hybrid environment
How to configure Exchange Server on-premises to use Hybrid Modern Authentication
Microsoft 365 Messaging Administrator Certification Transition (beta)
Microsoft 365 certification exams
Exchange Server build numbers and release dates
March 2020 Updates to the HCW

PLEASE LEAVE QUESTIONS, COMMENTS, UPDATES! I WOULD LOVE TO HEAR FROM YOU!