Vulnerabilities addressed in the April 2021 security updates were responsibly reported to Microsoft by a security partner. Although we are not aware of any active exploits in the wild, our recommendation is to install these updates immediately to protect your environment.
These vulnerabilities affect Microsoft Exchange Server. Exchange Online customers are already protected and do not need to take any action.
Use the Exchange Server Health Checker script, which can be downloaded from GitHub (use the latest release), to inventory your servers. Running this script will tell you if any of your Exchange Servers are behind on updates (CUs and SUs).
Update to the latest Cumulative Update
Go to https://aka.ms/ExchangeUpdateWizard and choose your currently running CU and your target CU. Then click the “Tell me the steps” button, to get directions for your environment.
If you encounter errors during or after installation of Exchange Server updates
My organization is in Hybrid mode with Exchange Online. Do I need to do anything? While Exchange Online customers are already protected, the April 2021 security updates do need to be applied to your on-premises Exchange Server, even if it is used only for management purposes. You do not need to re-run the Hybrid Configuration Wizard (HCW) after applying updates.
Do the April 2021 security updates contain the March 2021 security updates for Exchange Server? Yes, our security updates are cumulative. Customers who installed the March 2021 security updates for supported CUs can install the April 2021 security updates and be protected against the vulnerabilities that were disclosed during both months. If you are installing an update manually, do not double-click on the .msp file, but instead run the install from an elevated CMD prompt.
Is Microsoft planning to release April 2021 security updates for older (unsupported) versions of Exchange CUs? No, we have no plans to release the April 2021 security updates for older or unsupported CUs. In March, we took unprecedented steps and released SUs for unsupported CUs because there were active exploits in the wild. You should update your Exchange Servers to supported CUs and then install the SUs. There are 47 unsupported CUs for the affected versions of Exchange Server, and it is not sustainable to release updates for all of them. We strongly recommend that you keep your environments current.
Can we use March 2021 mitigation scripts (like EOMT) as a temporary solution? The vulnerabilities fixed in the April 2021 updates are different from those we fixed before. Therefore, running March 2021 security tools and scripts will not mitigate the vulnerabilities fixed in April 2021. You should update your servers as soon as possible.
Do I need to install the updates on ‘Exchange Management Tools only’ workstations? Servers or workstations running only Microsoft Exchange Management Tools (no Exchange services) do not need to apply these updates.
Why are there security updates two months in a row? Microsoft regularly releases Exchange Server security updates on ‘patch Tuesday’. We are always looking for ways to make Exchange Server more secure. You should expect us to continue releasing updates for Exchange Server in the future. The best way to be prepared for new updates is to keep your environment current.
Is there no update for Exchange Server 2010? No, Exchange 2010 is not affected by the vulnerabilities fixed in the April 2021 security updates.
Is there a specific order of installation for the April 2021 security updates? We recommend that you update all on-premises Exchange Servers with the April 2021 security updates using your usual update process.
I always have issues getting my certificate renewed using OpenSSL and the certificates that GoDaddy lets you download. I chose IIS and I chose Exchange Server in the GoDaddy download section of the site to get the CRT file. The issue I always have is converting it to PFX so that I can install it with a private key on my IIS Server. This is also relevant if you are using Azure to host your certificates as Microsoft requires PFX certificates in that realm.
So finally after I get it working today, I wanted to write this blog post to make sure I at least have a sure method to get the certificate converted with the private key. NOTE that this is for a certificate that has NOT expired.
First, download your certificate from GoDaddy to the server you have OpenSSL installed on.
Download the Certificate
Next, extract the cert to your directory and note the path. You will use the path in your OpenSSL cmdlet.
You may be seeing other files in there. Well the issue was that I couldn’t generate the proper private key and the PEM file given by GoDaddy did not work in the conversion. So, here is what I had to do on the Web Server to export the proper private key:
In the MMC Certificate Utility, export the current certificate with the private key:
Choose to Export the Key and Extended Properties
Choose a password and set the encryption to SHA256
Name the File and Export it to the directory you’re working from
Next, run the following cmd in OpenSSL to extract the private key from the exported certificate. Enter the password you created during the export when prompted:
You should now have the proper NEW PFX file to import into IIS or Azure or where ever you need to the certificate installed with the private key! DON’T forget your password!
THANKS FOR READING!! KEEP LEARNING AND REMEBER TO DOCUMENT SO YOU DON’T HAVE TO REMEMBER ALL THE TIME!
I like to post the Exchange Server CU Updates when they are released by the Exchange Team. It is important to keep up to date with the Exchange Updates especially after the Zero Day threat that came out the early part of March. Thank you for your continued reading and patronage!
Summary
Today we are announcing the availability of quarterly servicing cumulative updates (CUs) for Exchange Server 2016 and Exchange Server 2019. These CUs include fixes for customer reported issues as well as all previously released security updates. Although we mentioned in a previous announcement that this release would be the final cumulative update for Exchange 2016, we do expect to release one more CU for Exchange 2016 next quarter which will includes all fixes made for customer reported and accepted issues received before the end of mainstream support.
A full list of fixes is contained in the KB article for each CU, but we wanted to highlight that these latest CUs contain the fixes that were previously released as Exchange Server Security Updates on March 2, 2021. This means you don’t have to install the March 2021 Security Updates after installing the March 2021 CUs.
The KB articles that describe the fixes in each release and product downloads are as follows:
Microsoft recommends all customers test the deployment of any update in their lab environment to determine the proper installation process for your production environment. These updates contain schema and directory changes and so require you prepare Active Directory (AD) and all domains. You can find more information on that process here.
Also, to prevent installation issues you should ensure that the Windows PowerShell Script Execution Policy is set to Unrestricted on the server being upgraded or installed. To verify the policy settings, run the Get-ExecutionPolicy cmdlet from PowerShell on the machine being upgraded. If the policies are NOT set to Unrestricted you should use these resolution steps to adjust the settings.
Additionally, a reminder that if you plan to install any Cumulative Update using the unattended option with either PowerShell or Command Prompt, make sure you specify either the full path to the setup.exe file or use a “.” in front of the command if you are running it directly from directory containing the update. If you do not the Exchange Server Setup program may indicate that it completed successfully when it did not. Read more here.
Reminder: Customers in hybrid deployments where Exchange is deployed on-premises and in the cloud, and those using Exchange Online Archiving with their on-premises Exchange deployment are required to deploy the currently supported CU for the product version in use.
This article came out in February and I have been behind on my blog updates due to my current project, but I feel this post is important and am going to relay the message that I received here for your review. Thanks again for your support of this blog and its continued longevity.
Microsoft periodically refreshes certificates in Office 365 as part of our effort to maintain a highly available and secure environment. From Jan 23rd, 2021, we are making a certificate change on our Microsoft Federation Gateway every six weeks that could affect some customers as detailed in this knowledge base article. Please note that longer term, this “six week” rhythm to renew the certificate will be further shortened to daily renewals which will further enhance security of the environment. The good news is you can easily avoid any disruption.
Who is affected?
This certificate change can affect any customer that is using the Microsoft Federation Gateway (MFG). If you are in a hybrid configuration that relies on a Federation Trust established with MFG in the Exchange on-premises organization or if you are sharing free/busy information between two different on-premises organizations using the Microsoft Federation Gateway as a trust broker, you need to take action.
When will the change occur?
The change is scheduled to occur every six weeks to begin with, with this frequency further increasing. You must take action to avoid any disruptions.
What type of issues will you face if no action is taken?
If you don’t take action, you won’t be able to use services that rely on the Microsoft Federation Gateway. For example:
A cloud user might not be able to see free/busy information for an on-premises user and vice versa.
MailTips might not work in a Hybrid configuration.
Cross-premises free/busy might stop working between organizations that have organization relationships in place.
Additionally, if you run the Test-FederationTrust cmdlet, you might receive an error message that indicates that the Delegation token has validation issues. For example, you receive an error message that resembles the following:
Id : TokenValidation Type : Error Message : Failed to validate delegation token.
And, you might receive one of the following error messages in the Exchange Web Services (EWS) responses:
An error occurred when processing the security tokens in the message Autodiscover failed for email address User@contoso.com with error System.Web.Services.Protocols.SoapHeaderException: An error occurred when verifying security for the message
What action should you take?
You can use the following command on your Exchange Server to create a scheduled task to run the update process daily. This is how we recommend you keep your Federation Trust constantly updated. This will prevent you from being negatively affected by future metadata changes.
If you prefer to not use a scheduled task, you can manually run the command at any time to refresh the metadata. This is not recommended as the frequency to refresh certificate will increase from 6 week period to daily, and manually updating this would be quite cumbersome.
There was a zero day threat in Exchange recently and I wanted to put out this update that I received from the Microsoft team so that it would be available to my followers and readers. I will try to keep this updated as much as I can as I am not updating my blog as much with my current projects taking up most of my time. Thanks to everyone and keep in touch!
Summary
To help customers more quickly protect their environments in light of the March 2021 Exchange Server Security Updates, Microsoft is producing an additional series of security updates (SUs) that can be applied to some older (and unsupported) Cumulative Updates (CUs). The availability of these updates does not mean that you don’t have to keep your environment current. This is intended only as a temporary measure to help you protect vulnerable machines right now. You still need to update to the latest supported CU and then apply the applicable SUs. If you are already mid-update to a later CU, you should continue with that update.
With these new updates, you will have a new path you can take:
What are these updates?
These update packages contain only fixes for March 2021 CVEs (CVE-2021-26855, CVE-2021-26857, CVE-2021-26858, CVE-2021-27065); no other product updates or security fixes are included. Installing these updates does not mean an unsupported CU is now supported.
Updates are available only through the Microsoft Download Center (not on Microsoft Update).
We are producing updates only for some older CUs for Exchange 2016 and 2019.
If you are running a version of Exchange not covered by these updates, consider either rolling forward to a CU package that has an applicable SU, or rolling forward to a supported CU (preferred option). In case you need to go forward with CUs, please see: best practices for installation of Exchange updates (applies to all versions of Exchange).
About installation of these updates
These updates must be installed from an elevated command prompt:
Download the update but do not run it immediately.
Select Start, and type CMD.
In the results, right-click Command Prompt, and then select Run as administrator.
If the User Account Control dialog box appears, choose Yes, and then select Continue.
Type the full path of the .msp file, and then press Enter.
Installing the SUs mentioned here and then installing a later CU will make the server vulnerable to exploits again until the CU you install contains the March 2021 security fixes (Exchange 2016 CU 20 and Exchange 2019 CU 9 – and newer – will include March 2021 security updates).
Installing updates requires a reboot (even if not prompted). The server will not be protected until after the reboot.
After installing one of these updates, you might see older Exchange security updates for your older CU available for download from Microsoft Update. Install the older security update from Microsoft Update and your servers will stay protected (for 4 CVEs mentioned before).
If you run into issues after installation, please see https://aka.ms/exupdatefaq first. You can also uninstall these updates (using Add/Remove Programs) if needed.
These additional updates are about to be to available in KB5000871.
IMPORTANT: You must install .msp updates from elevated command prompt (see Known Issues in the update KB article)
If you install these additional updates, please ensure that you continue to bring your Exchange environment to supported state as soon as possible. Our original announcement Released: March 2021 Exchange Server Security Updates contains information and resources that can help you plan your updates, troubleshoot problems, and help you with mitigations, investigation, and remediation of the vulnerabilities.
Additional news about investigations
To aid defenders in investigating these attacks where Microsoft security products and tooling may not be deployed, we are releasing a feed of observed indicators of compromise (IOCs). The feed of malware hashes and known malicious file paths observed in related attacks is available in both JSON and CSV formats at the below GitHub links. This information is being shared as TLP:WHITE.
I have renewed my Microsoft Certified Trainer certification for 2021-2022. The big requirement was to teach a MOC class during the calendar year and get the MTM surveys in the system to be able to renew. Most Microsoft employees renew internally, but I am currently NOT working for Microsoft. I was able to get the classes taught so that I was able to renew for the new year.
I am still available for music performance through my company but the IT Side of the business has been closed as I am on a full time project and cannot devote any outside time to IT consulting. Thank everyone for supporting me over the years. My blog will still remain current so please check here often for the latest updates for Exchange, M365, Security, Compliance, and Windows!
I have been working on updating my skillset to M365 and passed the final exam today to achieve it. I am hoping that the certification will assist me with attaining work moving forward.
In my environment, I have a Windows Server (2019) Core edition server installed with Exchange 2019. Most of the time, I have to get on the server to run PowerShell commands for maintenance purposes, etc…
Well, by default, Windows Server Core opens the command prompt when you logon and then I have to manually open PowerShell from there to run cmdlets, etc…
However, if you would like to change the default cmd to PowerShell, you can change it by changing the Registry value.
The Registry that I’m talking is located under the following location:
Once completed, you will need to reboot the computer from PowerShell:
Reboot Computer From PowerShell
1
Restart-Computer-force
When the computer has rebooted and you have logged on, PowerShell should load by default instead of Command Prompt.
EVEN MORE INFORMATION
Now, since I have an Exchange Server installed on this server, there is a Command in the $bin directory called LaunchEMS.cmd that will load the Exchange Management Shell for you. So instead of loading just PowerShell, I tell WinLogon to load Exchange Management Shell so that I do not have to do any additional typing or searching for EMS on the box. Remember, Server Core has no GUI!
I run the same commands as above, but just change the value to LaunchEMS.cmd
Once Rebooted, you can logon and EMS will be the only window prompt that loads in the shell!
Exchange Management Shell loads when you logon
NOTE: You can always run cmd from the prompt to open Command Prompt and also run PowerShell.exe to open regular PowerShell from the EMS Session Window.
As many of you that follow Exchange have knowledge of, Microsoft releases their updates for Exchange Server every 3 months. Below is the latest update for Exchange Server 2016. I do not run 2016 any longer in my lab, but please post if you have issues with the installation and I can investigate!
Cumulative Update 17 for Microsoft Exchange Server 2016 was released on June 16, 2020. This cumulative update includes fixes for nonsecurity issues and all previously released fixes for security and nonsecurity issues. These fixes will also be included in later cumulative updates for Exchange Server 2016.
In multidomain Active Directory forests in which Exchange is installed or has been prepared previously by using the /PrepareDomain option in Setup, this action must be completed after the /PrepareAD command for this cumulative update has been completed and the changes are replicated to all domains. Setup will try to run the /PrepareAD command during the first server installation. Installation will finish only if the user who initiated Setup has the appropriate permissions.
If you have ever skipped a Cumulative Update (for example, you are upgrading from an earlier version before Cumulative Update 13 for Exchange Server 2016), or this is a first Exchange Server installation in the AD, then this Known Issue section should be taken care of.
About the /PrepareDomain operation in multidomain:
The /PrepareDomain operation automatically runs in the Active Directory domain in which the /PrepareAD commandis run. However, it may be unable to update other domains in the forest. Therefore, a domain administrator should run the /PrepareDomain in other domains in the forest.
About the permission question:
As the /PrepareAD is triggered in Setup, if the user who initiates Setup isn’t a member of Schema Admins and Enterprise Admins, the readiness check will fail and you receive the following error messages.
To avoid the errors, either the user should join Schema Admins and Enterprise Admins groups or another user in Schema Admins and Enterprise Admins groups manually runs the /PrepareAD for this Cumulative Update first. Then the Exchange admin user can start Setup.
Autodiscover Event ID 1 occurs after you install Cumulative Update 14 for Exchange Server 2016. For more information, see KB 4532190.
Issues that this cumulative update fixes
This cumulative update fixes the issues that are described in the following Microsoft Knowledge Base articles:
4559444 Conversion from HTML to RTF removes non-breaking space in Exchange Server 2016
4559435 Introduce an OrganizationConfig flag to enable or disable recipient read session in Exchange Server 2016
4547707 Enable piping for Restore-RecoverableItems in Exchange Server 2019 and 2016
4559436 Attachments with properties (like Azure Information Protection labels) don’t always match in Exchange Server 2016
4559437 PR_RECIPIENT_ENTRYID is computed if no email address or type in Exchange Server 2016
4559438 Edge Transport server hangs in Exchange Server 2016
4559439 EAS creates failure report if a message with unknown recipients is in Drafts in Exchange Server 2016
4559440 Export to a PST for an eDiscovery search fails in Exchange Server 2016
4559441 Foreign language characters set in RejectMessageReasonText of a transport rule aren’t shown correctly in Exchange Server 2016
4559442 2080 Events caused by empty values in HKLM\SYSTEM\CurrentControlSet\Services\MSExchange ADAccess\Instance0 in Exchange Server 2016
4549689 HMA EvoSTS certificate rollover causes authentication prompts due to stalled key on worker process spawn (warmup phase) in Exchange Server 2016
4559443 Managed Folder Assistant fails with Event ID 9004 NotInBagPropertyErrorException in Exchange Server 2016
4559446 Changes to Outlook on the web blocked file extensions and MIME types in Exchange Server 2016
The Cumulative Update 17 package can be used to run a new installation of Exchange Server 2016 or to upgrade an existing Exchange Server 2016 installation to Cumulative Update 17.
You don’t have to install any previously released Exchange Server 2016 cumulative updates or service packs before you install Cumulative Update 17.
Cumulative update information
Prerequisites
This cumulative update requires Microsoft .NET Framework 4.8.
For more information about the prerequisites to set up Exchange Server 2016, see Exchange 2016 prerequisites.
Restart requirement
You may have to restart the computer after you apply this cumulative update package.
Registry information
You don’t have to make any changes to the registry after you apply this cumulative update package.
Removal information
After you install this cumulative update package, you can’t uninstall the package to revert to an earlier version of Exchange Server 2016. If you uninstall this cumulative update package, Exchange Server 2016 is removed from the server.
CHECK FOR UPDATES REGULARLY! POSITIVE ATTITUDE EQUALS POSITIVE MINDSET!
As many of you that follow Exchange have knowledge of, Microsoft releases their updates for Exchange Server every 3 months. Below is the latest update for Exchange Server 2019. I will be installing it very soon and will post any issues I may have with Server Core and GUI versions of Windows 2019.
Cumulative Update 6 for Microsoft Exchange Server 2019 was released on June 16, 2020. This cumulative update is a security update. It includes fixes for nonsecurity issues and all previously released fixes for security and nonsecurity issues. These fixes will also be included in later cumulative updates for Exchange Server 2019.
In multidomain Active Directory forests in which Exchange is installed or has been prepared previously by using the /PrepareDomain option in Setup, this action must be completed after the /PrepareAD command for this cumulative update has been completed and the changes are replicated to all domains. Setup will try to run the /PrepareAD command during the first server installation. Installation will finish only if the user who initiated Setup has the appropriate permissions.
If you have ever skipped a Cumulative Update (for example, you are upgrading from an earlier version before Cumulative Update 2 for Exchange Server 2019), or this is a first Exchange Server installation in the AD, then this Known Issue section should be taken care of.
About the /PrepareDomain operation in multidomain:
The /PrepareDomain operation automatically runs in the Active Directory domain in which the /PrepareAD commandis run. However, it may be unable to update other domains in the forest. Therefore, a domain administrator should run the /PrepareDomain in other domains in the forest.
About the permission question:
As the /PrepareAD is triggered in Setup, if the user who initiates Setup isn’t a member of Schema Admins and Enterprise Admins, the readiness check will fail and you receive the following error messages.
To avoid the errors, either the user should join Schema Admins and Enterprise Admins groups or another user in Schema Admins and Enterprise Admins groups manually runs the /PrepareAD for this Cumulative Update first. Then, the Exchange admin user can start Setup.
Autodiscover Event ID 1 occurs after you install Cumulative Update 3 for Exchange Server 2019. For more information, see KB 4532190.
Issues that this cumulative update fixes
This cumulative update also fixes the issues that are described in the following Microsoft Knowledge Base articles:
4559441 Foreign language characters set in RejectMessageReasonText of a transport rule aren’t shown correctly in Exchange Server 2019
4547707 Enable piping for Restore-RecoverableItems in Exchange Server 2019
4549689 HMA EvoSTS certificate rollover causes authentication prompts due to stalled key on worker process spawn (warmup phase) in Exchange Server 2019
4559446 Changes to Outlook on the web blocked file extensions and MIME types in Exchange Server 2019
4559440 Export to a PST for an eDiscovery search fails Exchange Server 2019
4559439 EAS creates failure report if a message with unknown recipients is in Drafts in Exchange Server 2019
4559442 2080 Events caused by empty values in HKLM\SYSTEM\CurrentControlSet\Services\MSExchange ADAccess\Instance0 in Exchange Server 2019
4559438 Edge Transport server hangs in Exchange Server 2019
4559443 Managed Folder Assistant fails with Event ID 9004 NotInBagPropertyErrorException in Exchange Server 2019
4559437 PR_RECIPIENT_ENTRYID is computed if no email address or type in Exchange Server 2019
4559444 Conversion from HTML to RTF removes non-breaking space in Exchange Server 2019
4559436 Attachments with properties (like Azure Information Protection labels) not always matching in Exchange Server 2019
4559435 Introduce an OrganizationConfig flag to enable or disable recipient read session in Exchange Server 2019
Note The Cumulative Update 6 package can be used to run a new installation of Exchange Server 2019 or to upgrade an existing Exchange Server 2019 installation to Cumulative Update 6.
Cumulative update information
Prerequisites
This cumulative update requires Microsoft .NET Framework 4.8.
For more information about the prerequisites to set up Exchange Server 2019, see Exchange 2019 prerequisites.
Restart requirement
You may have to restart the computer after you apply this cumulative update package.
Registry information
You don’t have to make any changes to the registry after you apply this cumulative update package.
Removal information
After you install this cumulative update package, you can’t uninstall the package to revert to an earlier version of Exchange Server 2019. If you uninstall this cumulative update package, Exchange Server 2019 is removed from the server.
CHECK FOR UPDATES REGULARLY! POSITIVE ATTITUDE EQUALS POSITIVE MINDSET!
In my experience with doing Exchange migrations to the cloud, there always seemed to be an issue with EWS throttling causing very slow mailbox moves. The remediation to this was to always contact Microsoft EXO Support and open a ticket to request that your EWS throttling for your tenant be lifted so you could move your mailboxes more quickly.
The reason for the EWS Throttling was to keep large amounts of data from flooding the front end server farm possibly causing a temporary outage or corruption of data going to the cloud since there are literally thousands of customers going through the same server farm and possibly migrations. Throttling was a way to keep everything in check.
Microsoft recently made an interesting change to the automated support handling capabilities of the Microsoft 365 admin center to handle requests for Exchange Web Services (EWS) throttling to be lifted for up to 90 days without human intervention.
Here is how to request the throttling be lifted via the M365 Support Assistant:
Go to the Help (?) section of the Microsoft 365 admin center.
Click the Need Help icon.
Enter “EWS throttling” as the search phrase.
Click Run tests when asked to check your environment. Essentially, the tests check what EWS throttling applies to the tenant.
Getting EWS Throttling Support
Running the Tests from the Admin Center
The support assistant checks the tenant settings and concludes that EWS is throttled (the normal situation). You’ll then be offered the chance to update the settings to the tenant EWS policy to lift throttling for 30, 60, or 90 days.
Select the number of days you’d like to adjust the policy for and then Update Settings.
After a short delay, the support assistant should confirm that the settings have been changed.
Select the number of days to lift throttling and click Update Settings
Once the setting is changed in the tenant, it will be effective after about 15 minutes for replication to the server farm. You should then be able to run your migrations at full speed.
Settings Changed Successfully
NOTE: Changing this setting is only effective for EWS migrations and NOT IMAP Migrations (such as from G-Mail).
Having this option online in the portal saves a tremendous amount of time when working to get your Exchange migrations to the cloud completed. Hopefully there will be more to come. I just hope it doesn’t keep me out of a job!
I AM CURRENTLY LOOKING FOR NEW OPPORTUNITES! POSITIVE MINDSET, POSITIVE ATTITUDE!
Since the COVID-19 virus has managed to get me laid-off and not working, I have not had too much to post in the past months. And although it has wavered greatly, I will remain positive and hope that someone will see the value that I can bring to their organization in the near future.
With that said, I wanted to repost this article that was sent to me as this issue arises quite often in the workplace when someone has left for vacation and is not in the office. So, here it is:
Out of Office replies can sometimes be a bit of a mystery to people; how do they work? What do you do if they don’t work? In this blog post, we will discuss the bits and pieces of Out of Office and some of the main reasons why an Out of Office (aka. OOF) reply might not get delivered to users. Note that while we are writing this from the viewpoint of an Exchange Online configuration, many of same things can be applied to on-premises configuration also. By the way – did you ever wonder why “OOF” is used instead of “OOO”? If you did, see this!
What is an Out of Office reply?
Out of Office replies are also known as OOF (or OOO) replies or automatic replies. They are Inbox rules that are set in the user’s mailbox by the client. OOF rules are server-side rules, so the response is sent regardless of whether a client is running, or not.
There are several ways of setting up automatic replies:
First, it can be set up as an automatic reply feature from Outlook, like this. It can also be configured using other clients, such as Outlook on the web (OWA), PowerShell command (Set-MailboxAutoReplyConfiguration). Admins can set up OOF replies on behalf of (forgetful) users from the M365 Admin Portal.
By design, Exchange Online Protection uses the high risk delivery pool (HRDP) to send the out of office replies, because they are lower priority messages.
Types of OOF rules
There are three types of OOF rules: Internal, External and Known Senders (Contact list). They are stored in the mailbox with the names in the following table. If a mailbox has set only Internal OOF, there will be no external rule created and the mailbox will have one OOF rule.
Note: Apart from OOF rule, other rules like the Junk Email rule will also have the “IPM.ExtendedRule.Message” message class; the MSG_NAME will determine what the rule is for.
Logon > profile that you are accessing > Top of information store > Inbox > right click ‘Open associated contents table’. They are listed under the Message Class column. All Inbox rules will have the same message class IPM.Rule.Version2.Message and there is one message class and name for each Inbox rule.
For all rules, the name of the rule is in stored in the PR_RULE_MSG_NAME property. So, if there are 4 Inbox rules, there will be 4 IPM.Rule.Version2.Message one for each rule, and the name of the rule is stored in PR_RULE_MSG_NAME.
OOF rules in MFCMapi:
And OOF rule templates in MFCMapi:
History of OOF replies
OOF response is sent once per recipient. Recipients to whom the OOF was sent are stored in the OOF history and are cleared out when the OOF state changes (enabled/disabled) or the OOF rule is modified. OOF history is stored in the user’s mailbox and can be seen using MFCMapi tool at: Freebusy Data > PR_DELEGATED_BY_RULE.
Note: If you want to send response to the sender every time instead of just once, you can apply mailbox server side rule “have server reply using a specific message” to send automatic reply instead of using the OOF rule. This server-side rule will send reply to the sender every time a message is received.
Now that we know what OOF replies are and how they are stored on the server, we can move on to address some of the scenarios where OOF is not sent to the sender. We will also discuss possible fixes and some more frequently seen issues you may have with OOF configuration.
The first category of issues we will talk about are issues related to OOF replies not being received by the sender of the original message.
OOF issues related to transport rules
When OOF doesn’t seem to be sent for all users in the tenant, usually there is a transport rule causing the issue. Check all the transport rules that may apply to the affected mailbox using step two of this article.
If you suspect a delivery problem, run a message trace from the Office 365 tenant. We know that OOF response is sent back to the original sender of the message, so for OOF messages, the sender of the original message becomes the recipient when tracking. We should then be able to tell if OOF reply has been triggered and sent to external or internal recipient. If a Transport rule is blocking the OOF response, the message trace will clearly show you that.
There is one scenario I would like to highlight when it comes to transport rules blocking OOF replies. Let’s assume that you moved the MX record to a 3rd party anti-spam solution; you have created a transport rule to reject any email coming from any other IP address than the 3rd party anti-spam.
The transport rule will look something like this:
Description: If the message: Is received from ‘Outside the organization’ Take the following actions: reject the message and include the explanation ‘You are not permitted to bypass the MX record!’ with the status code: ‘5.7.1’ Except if the message: sender ip addresses belong to one of these ranges: ‘1xx.1xx.7x.3x’
ManuallyModified: False
SenderAddressLocation: Envelope
As OOF replies have a blank (<>) Return-Path, you will see that the rule is unexpectedly matching the transport rule and the OOF responses are getting blocked.
In order to fix this, you can change the transport rule property of ‘Match sender address in message’ to ‘Header or envelope’, so the checks will also be done against ‘From’ (also known as the ‘Header From’ address), ‘Sender’ or ‘reply-to’ fields. More information about the mail flow rule conditions is here.
JournalingReportNdrTo mailbox setting
If the affected mailbox is the one which is configured under JournalingReportNdrTo, OOF replies will not be sent for that mailbox. Moreover, journaling emails may be affected as well. It is recommended to create a dedicated mailbox for the JournalingReportNdrTo setting. Alternatively, you can set it to an external address.
For more details on how to solve this, please see this KB Article.
Forwarding SMTP address is enabled on the mailbox
If the affected user mailbox has SMTP forwarding enabled, OOF replies won’t be generated. This can be checked in user mailbox settings (OWA):
You need pay attention to what OOF type you have set up, as this will impact the OOF response and OOF may not be generated at all if the configuration is incorrect. Let’s assume you have a hybrid organization with mailboxes hosted both in Exchange on-premises and Exchange Online. In this scenario, by design only external messages will be sent to on-premises while AllowedOOFType is set to External. To be able to send internal OOF messages to on-premises in hybrid environment, you need to set the AllowedOOFType to InternalLegacy.
You also have option to send external Out of Office replies only for contacts at the mailbox configuration level (ExternalAudience: Known). This can make automatic replies not being sent to anyone external but contacts. The command to check the configuration is:
Another setting on remote domains is one which lets you dictate whether you allow or prevent messages that are automatic replies from client email programs in your organization.
This can be found in Exchange Admin Center > Mail flow > Remote domains
Note: If this option is set to false, no automatic replies will be sent to users for that domain. This setting takes precedence over the automatic replies set up at the mailbox level or over the OOF type (discussed above).
Please, keep in mind that $false is the default value for new remote domains that you create as well as the built-in remote domain named Default in Exchange Online.
If the email was marked as spam and sent to junk, an automatic reply will not be generated at all.
Pretty self-explanatory, that one!
Message trace shows delivery failure
If you investigate an OOF reply issue and in the message trace you find the following error message:
“550 5.7.750 Service unavailable. Client blocked from sending from unregistered domains.”
You should reach out to Support to find out why the unregistered domain block was enforced.
There are some other scenarios that might come up when working with OOF replies, let’s cover those next!
An old or duplicate OOF message is sent
This is likely due to a duplicate Inbox rule or the OOF history limit. The OOF history has a limit of 10,000 entries, if this threshold is hit, OOF will continue to be sent to recipients that are not already in the list as any new users can’t be added to the list. All users already in the list will not receive duplicate OOF replies. For more information you may want to check this article or follow the action plan below.
Remove the OOF rules and the OOF rule templates from the mailbox. To locate the rules and delete them go back to > “Inbox OOF rules”
Disable and then re-enable the OOF feature for the mailbox
Now, you can check again whether the OOF feature works as expected and symptoms do not occur.
Automatic replies cannot be enabled; an error message is received
While attempting to access automatic replies from the Outlook client, an error message is received saying that “Your automatic reply settings cannot be displayed because the server is currently unavailable. Try again later.”
To narrow down this issue, you should perform the following steps:
Confirm EWS protocol is enabled on the mailbox as OOF replies rely on it to be enabled (note that re-enabling this might take several hours to take effect)
Enable the OOF feature by using the following command:Set-MailboxAutoReplyConfiguration <identity> -AutoReplyState Enabled
Check whether the OOF feature works as expected.
If the issue is still there, review the rules quota on the mailbox: Get-mailbox -identity <mailbox> | fl RulesQuota
By default, the RulesQuota has a maximum quota which is calculated by the size of the rules (not the number of rules). Maximum is 256 KB (262,144 bytes).
Remove the OOF rules and the OOF rule templates from the mailbox. To locate the rules and delete them go back to > “Inbox OOF rules” After you remove them, you can re-enable the OOF feature and then test again.
An automatic reply is still sent despite OOF being disabled
We have encountered scenarios in which OOF messages are still being sent, although it is disabled. Most of the time, we found that the rule is created manually by the end users using the out-of-office template.
So, as you can see, there is quite a bit that goes into troubleshooting OOF Replies and it is not all straight forward.
THANKS FOR READING! I’M AVAILABLE FOR WORK! PLEASE CONTACT ME FOR AN APPOINTMENT!
This was a great article released by the Exchange Team Blog today, and as I have been dealing with MANY customers still having Exchange 2010, I wanted to have this available for quick review! It has great links and steps to consider when finally getting off Exchange 2010.
As many of you know from the previous post regarding Exchange On-Premises Best Practices for Migrations from 2010 to 2016 the end of support for Exchange 2010 is quickly approaching. We’ve created this post to cover the best practices for decommissioning an Exchange 2010 environment after the migration has completed.
Uninstalling Exchange 2010 is as easy as running Setup and selecting to remove the server roles, but there are prerequisites to removing the roles and legacy items left over, which should be removed.
This post is intended to provide best practices to plan for and complete the Exchange 2010 decommission. Please note that since there are many different types of deployments and configurations it is difficult to cover all scenarios, but many of the common steps are included here. Please plan the decommission process carefully.
As a general statement, here are some things that we want to caution against:
Do not reuse Exchange 2010 server names (until they have been fully decommissioned).
Do not reuse Exchange 2010 server IP addresses (until they have been fully decommissioned).
This post assumes that your organization is maintaining some Exchange presence on-premises, whether Exchange 2013 or 2016 (we do not mention Exchange 2019 in this post because it cannot coexist with Exchange 2010). If your organization has moved all mailboxes to Office 365 and is in a Hybrid environment, we are assuming you will maintain an Exchange footprint per Scenario 2 in How and when to decommission your on-premises Exchange servers in a hybrid deployment.
Preparing for Soft Shut Down
Once you’ve completed the migration from Exchange 2010 to, let’s say, Exchange 2016, you should prepare the 2010 environment prior to decommissioning the servers. The following steps to consider are separated into server roles when preparing for a soft shut down and preparing for the removal of server roles.
Client Access (CAS) Role
Check Server FQDNs
Review all namespaces (e.g. DNS records and load balanced virtual IP addresses) used for client connectivity and ensure they are routing to the 2016 environment. These are all the names that are published for Outlook Anywhere, AutoDiscover, and all Exchange Virtual Directories.
Tip: Verify that all clients such as ActiveSync, Outlook, EWS, OWA, OAB, POP3/IMAP, and Autodiscover are no longer connecting to the legacy Exchange servers. Verification of this can be done by reviewing the servers’ IIS Logs with Log Parser Studio (LPS). LPS is a GUI for Log Parser 2.2 and it greatly reduces the complexity of parsing logs. LPS can parse large sets of logs concurrently (we have tested with total log sizes of >60GB). Please refer to the following blog post with tips and information on using LPS.
If present, ensure that if the AutoDiscoverServiceInternalURI routes to an Exchange 2016 endpoint. You can also remove this value by setting the AutoDiscoverServiceInternalURI to $Null.
Follow the items below to review all mail flow connectors. We will not be removing connectors themselves, simply auditing to ensure that the server is ready to be decommissioned.
Review the Send Connectors
Review the send connectors and ensure that the legacy servers have been removed and Exchange 2016 servers have been added. Most organizations only permit outbound network traffic on port 25 to a small number of IP addresses, so you may also need to review the outbound network configuration.
Review the receive connectors on legacy servers and ensure they are recreated on your Exchange 2016 servers (e.g. SMTP relay; anonymous relay; partner, etc.). Review all namespaces (e.g. DNS records and load balanced virtual IP addresses) used for inbound mail routing and ensure they are terminating against the Exchange 2016 environment. If your legacy Exchange servers have any custom, third-party, or foreign connectors installed (for example, with fax services), ensure that they can be reinstalled on 2016 Exchange servers.
1
Get-ReceiveConnector-Server<ServerToDecommission>
Tip: Check the SMTP logs to see if any outside systems are still sending SMTP traffic to the servers via hard coded names or IP addresses. To enable logging, review Configure Protocol Logging. Also, ensure we have “time coverage” for any apps relaying weekly/monthly emails that may not be caught in a small sample size of SMTP Protocol logs. There is a great script available here that can help find any applications that may be relaying off your legacy environment.
In general, the decommissioning process is a great time to audit your mail flow configuration to ensure that all the connectors are properly configured and secured. Maybe it’s time to get rid of any of those Anonymous Relay connectors that may be in use in your environment. Or, if Hybrid, possibly relay against Office 365.
Transport Rules
Exchange 2010 base transport rules are held in a different AD container than Exchange 2013 and newer rules. When installing Exchange 2016 in your environment it will import those Exchange 2010 based rules. However, any changes to Exchange 2010 rules after a later version of Exchange is installed must also be applied to your Exchange 2016 rules. This is further explained here under section Coexistence with Exchange 2010.
Run the following command to get all your Exchange Transport Rules. Must be run on Exchange 2016 to see all rules.
1
Get-TransportRule|Select Name,RuleVersion
Compare the rules with RuleVersion of 14.X.X.X to those with 15.1.X.X. If any Exchange 2010 rules don’t exist on Exchange 2016, they must be created. Also review all settings of each Exchange 2010 rule and replicate them to Exchange 2016.
Mailbox Role
Identity and move all Exchange 2010 mailboxes to Exchange 2016
Decommissioning Exchange 2010 cannot be initiated until all mailboxes have been moved to Exchange 2016. As an example, we cannot decommission Exchange 2010 Hub Transport servers completely until all of the mailboxes are moved off the legacy platform, this is due to how Delivery Groups are handled.
We encourage using the newest Exchange platform to process any move requests. If moving to Exchange 2016, move all mailboxes via Exchange 2016. Also, ensure that once all moves are completed, and that all associated Move Requests are removed as well. Any lingering move requests or mailboxes will prevent uninstallation of Exchange 2010.
To move all user mailboxes, run the following command to identify the mailboxes, and then plan to move them to the new platform.
Tip: Ensure that Archives are included with “Get-Mailbox -Archive” if you used Exchange Archives in 2010. Also, do not forget about your Discovery Search mailboxes – these can be found with: Get-Mailbox -Filter { RecipientTypeDetails -eq “DiscoveryMailbox”}. These will need to be moved (if they haven’t yet already), to Exchange 2016 as well.
Identify and Move Arbitration Mailboxes to Exchange 2016
It’s necessary to move the arbitration mailboxes from Exchange 2010 to 2016 for many Exchange Services to work properly, including the Exchange Admin Center (EAC). This is typically executed when Exchange 2016 is first installed, however, if that was missed, we will ensure that is handled now. The process to move is defined at: Move the Exchange 2010 system mailbox to Exchange 2013+. To verify which system mailboxes are located on 2010, use PowerShell on your Exchange 2010 server with the following example:
Note: If any mailboxes are present, move them to an Exchange 2016 database.
OAB Generation
Installing first Exchange Server 2013+ into Exchange 2010 organization creates a new OAB. It also marks the new OAB as default. The Exchange 2010 OAB is not used by Exchange 2013+ servers so moving the OAB is not necessary. Move the OAB to another Exchange 2010 server, if you are removing an Exchange 2010 server that’s currently hosting the OAB, and there are other Exchange 2010 servers in the org. If you are removing the last Exchange 2010 server in the org, remove the OAB.
Exchange Server 2010 public folders are migrated to Exchange Online
Exchange Server 2013/2016 was introduced on-premises
MEPF’s are still used on-premises to send emails to Exchange Online
In that case, you may need to run the SetMailPublicFolderExternalAddress.ps1 script to ensure Exchange 2013+ servers can continue sending emails to Exchange Online MEPFs.
Decommission the Database Availability Group (DAG)
Assuming best practices were followed for the Exchange 2010 environment, we will have a DAG for HA/DR capabilities. Now that all mailboxes have been removed from the 2010 environment, we are ready to tear down this DAG to move forward with decommissioning Exchange 2010.
Remove Database Availability Group (DAG) Copies
First, we start with the copies. For every mailbox database copy in the environment hosted on Exchange 2010, we will need to remove the Mailbox Database Copy. This can be done via the UI, or via PowerShell:
NOTE: Removing the copy will not remove the actual .edb database file from the Server.
Remove All Nodes from Database Availability Group(s) (DAG)
For each Exchange 2010 server in the environment, we will need to remove the individual server from the DAG. This is evicting the server from the cluster. This can be done via the UI, or through PowerShell.
Lastly, once the Database copies are removed, and the servers are evicted from the cluster, the last thing is to finally remove the DAG from the environment. This can be done with the following PowerShell command:
Tip: If you have an even-membered DAG, and leveraged a File Share Witness, don’t forget to decommission the file share witness that was used for the Exchange 2010 DAG.
Unified Messaging Role
Configuration steps are required to move Exchange 2010 UM to Exchange 2016 servers. The following link can be used to guide through removal of UM from Exchange 2010. If moving to a third-party UM solution, remove the UM components to allow un-installation of the UM role.
Edge Role
If you have an Edge server, you will need to install Exchange 2016 Edge and recreate the Edge Subscription on the E2016 server. This is further documented here.
Other
As mentioned in the beginning of the document, due to so many different types of deployments and configurations, it’s difficult to cover all scenarios however it’s recommended to check any other possible scenarios that apply to your environment.
Third Party Applications
Make a list of applications that may be using Exchange 2010 (e.g. EWS, mail transport, database-aware) and make sure to configure these applications to start using Exchange 2016 infrastructure.
Shut-Down Exchange 2010 Servers
Test shutting down the Exchange servers for a few days to a few weeks to see if there are any issues. You are auditing for any applications that are trying to connect to the Exchange 2010 servers or trying to send email through the Exchange 2010 servers. Enabling protocol logging on the Hub Transport roles prior to shutting down the servers is an option. That way if any mail is processing through these servers, upon restart, the logging will begin immediately. If applications or servers are trying to connect you can remediate those or power on the Exchange 2010 servers until remediation can happen.
Tip: Check Active Directory DNS Zone settings to see if DNS Scavenging is enabled. If this is enabled, the DNS record could become stale during the shutdown time frame and cause DNS issues for the Exchange 2010 server.
Preparing for Removal of Server Roles
As you begin the process of removing servers, you should go through the list below and ensure you have everything tested and ready to go.
CAS
Remove CAS Arrays
Remove Any Exchange 2010 Client Access Arrays from Active Directory and DNS. Refer to the following document to remove the Client Access Array object with Shell using the following example:
Be sure to also remove any references in DNS to the CAS Array Name.
Remove Unused 2010 ASAs
If you followed either the Best practices for Migrations blog or the Coexistence with Kerberos blog, we recommend that any old alternate service accounts (ASAs) used for E2010 be removed. If you are using a different namespace than Exchange 2016, please verify old SPNs are also removed.
Remove Exchange 2010 OAB
Use the following command to remove Exchange 2010 OAB:
Now that all mailboxes are migrated from the Exchange 2010 platform, and the DAG is properly removed, we will want to decommission any leftover databases from the Exchange 2010 environment. To remove all Exchange 2010 databases, review the output of the following, and remove individually:
1
Get-MailboxDatabase-Server<ServerToDecommission>
And then remove the database with:
1
Remove-MailboxDatabase-Identity DB1
NOTE: If there are any mailboxes currently residing on the database, we will not let you remove the database, it will fail with the following error:
Remove Legacy Public Folders
If you chose not to migrate public folders, refer to the following document to remove public folders with either EMC or Shell using the following example:
Remove Legacy Public Folder Databases
Refer to the following document to remove the public folder databases with PowerShell using the following example:
1
Remove-PublicFolderDatabase-Identity"PFDB01"
Tip: Remember the .edb files linger after the above is done. Feel free to delete, backup, or do with these as you please.
Uninstall Exchange 2010
It’s recommended to uninstall in the following order: CAS, Hub, UM (if any), then Mailbox.
Starting the Uninstall Process
When you begin the uninstall process, close EMC, EMS, and any additional programs that could delay uninstall process (i.e. programs using .NET assemblies; antivirus and backup agents are examples). You can either run Exchange 2010 Setup.exe or navigate to Control Panel to modify or remove Exchange 2010 (either server roles or the entire installation). Specific steps are discussed in Modify or Remove Exchange 2010.
Tip: Exchange will protect itself! If you properly uninstall via Add/Remove Programs, it will ensure that it is ready to be uninstalled via Readiness Checks! If all the above prep work is completed before hand, it should uninstall just fine.
After Uninstall of Exchange 2010
After uninstalling Exchange there will be some general “housekeeping” tasks. These may vary depending on the steps taken during your upgrade and depending on your organization’s operational requirements.
Examples include:
Removing the legacy Exchange computer accounts from AD (including the DAG’s Cluster Name Object and any Kerberos ASA object).
Removing the legacy Exchange name records from DNS (including the DAG’s Cluster Name Object and any Kerberos ASA object).
Ensure the folder on the DAG file share witness (FSW) servers were successfully removed, possibly removing Exchange’s rights on the server if it isn’t serving double duty for Exchange 2016.
Removing old load balanced IP addresses and routes from your network load balancer.
Remove old firewall rules that open ports to Exchange 2010 environment.
Removing and disposing of the legacy Exchange environment’s physical equipment.
Deleting of the legacy Exchange environment’s virtual machines.
Conclusion
With the uninstall of the last server, hopefully Exchange 2010 treated your organization well. The Exchange product team takes great pride of the success of the platform and hope that you see the same success with Exchange 2016 (or Exchange Online!). Messaging sure has come a long way since it was released way back in 2009.
Today Microsoft is announcing the availability of quarterly servicing cumulative updates for Exchange Server 2016 and 2019. These updates include fixes for customer reported issues as well as all previously released security updates.
Personal Note: I was recently involved in a layoff at Microsoft in the Vendor PFE world. I am currently looking for new engagements.
Calculator Updates
This quarterly Exchange release includes an important update to the Exchange 2019 Sizing Calculator. We’ve made improvements to the logic to detect whether a design is bound by mailbox size (capacity) or throughput (IOPs) which affects the maximum number of mailboxes a database will support. Previous versions of the calculator produced incorrect results in some situations.
The Exchange team highly recommends using calculator version 10.4, included with the March 2020 quarterly CU release, to size Exchange Server 2019 deployments.
MCDB Configuration Issues
Cumulative Update 5 for Exchange Server 2019 also fixes an issue that can happen when you use the Manage-MetaCacheDatabase.ps1 script to enable MetaCacheDatabase (MCDB).
This issue occurred because of a change in behavior in Windows Server 2019 that caused Get-Disk to return all uninitialized discs within the Database Availability Group (DAG) or cluster. The script then incorrectly tried to format an SSD on another DAG member. We documented a workaround for CU4 here, but we’ve fixed it in CU5.
Online Mode Search Issues
Cumulative Update 5 for Exchange Server 2019 is also required to fix a known issue with partial word searches when the client is using Outlook in online mode.
Release Details
The KB articles that describe the fixes in each release and product downloads are available as follows:
Microsoft recommends all customers test the deployment of any update in their lab environment to determine the proper installation process for your production environment. For information on extending the schema and configuring Active Directory, please review the appropriate documentation.
Also, to prevent installation issues you should ensure that the Windows PowerShell Script Execution Policy is set to “Unrestricted” on the server being upgraded or installed. To verify the policy settings, run the Get-ExecutionPolicy cmdlet from PowerShell on the machine being upgraded. If the policies are NOT set to Unrestricted you should use the resolution steps in KB981474 to adjust the settings.
Reminder: Customers in hybrid deployments where Exchange is deployed on-premises and in the cloud, or who are using Exchange Online Archiving (EOA) with their on-premises Exchange deployment are required to deploy the currently supported cumulative update for the product version in use, e.g.,
2013 Cumulative Update 23 2016 Cumulative Update 16 or 15 2019 Cumulative Update 5 or 4.
I had a failure on one of my two Exchange 2019 CU4 servers when installing the Security Update for them:
I could not restart the install as it would fail when getting to the services stoppage part of the installation. I saw this error in the ServiceControl.log file in the C:\ExchangeSetupLogs Directory
I downloaded the .msp file to install manually and read the answers on the web-page. It was said that there was an issue with the ServiceControl.ps1 file that is in the Exchange Server BIN directory when running Patch with the Verbose logging enabled:
3. Next, Change all the Stop-SetupService Entries in the script to Stop-Service
4. Next, Change all the Start-SetupService to Start-Service
5. Next, Add the following to StopServices function to bypass the attempt that was trying to stop services, because they have already been stopped ...
Start at line 300
300: Status "Stopping services for '$Roles'..."
301:$services=Get-ServiceToControl$Roles-Active
302:##Change Here
303:if($services-eq$null){
304:return$true
305:}
306:##End Change Here
307:[array]::Reverse($services)
6.Inoticed that these2files ServiceStartupMode.xml andServiceState.xml may have permission issue since the last failure intheC:\ExchangeSetupLogs directory.So,Ialso renamed the files to.old
Once I made those changes. I renamed the original ServiceContol.ps1 to a .old file and saved this modified file to the BIN directory. I was then able to successfully run the Security Update.
BUT WHY DID THIS SERVER FAIL AND NOT THE OTHER?!?
Both are CU4, but the server that failed had been upgraded from CU1 where the one that did NOT fail was a clean installation of CU4. So, I checked the ServiceConrol.ps1 file on both servers:
Older Exchange Install Date of ServiceControl.ps1 is 1/1/2020CU4 Clean Installation Date of ServiceControl.ps1 is 2/3/2020
I have not tested this, but maybe if I had copied the ServiceControl.ps1 file from the CU4 clean installation to the original install, the script might have worked since the creation dates are different and I have no other version information to go on. I will verify this though. For now, the changes to the script allowed me to successfully install the Security Update Successfully.
NOTE: All the Exchange and IIS Services were in a Startup Mode: Disabled state and I had to reset them ALL to Automatic. Once that was completed and the server rebooted, Exchange was returned to normal state. ****Also, just to be safe, I ran a great script that assures the server is out of maintenance mode. You can get the script HERE. You can also get the script to put the server into maintenance mode. You can get the script HERE. These scripts will work on Exchange 2013 and above servers.
UPDATE / 3/12/2020 2:39 AM EST
Something was still wrong with the server. ActiveSync and PAM started breaking. I started having all sorts of authentication problems. I decided to restore the server from backup from Tuesday. I forgot to backup the ServiceControl.ps1 file that modified, but I moved the one from the successful installation to the restored server and am currently running the update. I will see if it works and send the information to you.
UPDATE / 3/12/2020 3:45 AM EST
The restore worked great and placing the ServiceControl.ps1 file in the BIN directory on the prior failed Exchange Server did allow for the installation to complete successfully. I have tested ActiveSync and Authentication which is now functioning properly. Hooray!
SEND ME YOUR IDEAS/ FOR POSTS! HAPPY TROUBLESHOOTING!
In my role as a PFE for Microsoft, I have been going through many deployments of Exchange 2016 from Exchange 2010 due to the end of life deadline for Exchange 2010. I am actually in the middle of three enterprise level deployments this month.
Because of this, I wanted to provide some MUST READ links with consideration to the Exchange 2016 Deployment process. Please click on the links below, and they will take you to the documentation needed when preparing for a Exchange 2016 migration and deployment from Exchange 2010.
I will add more articles as they become relevant in my experiences with my customers and feel they could be relevant here as well. If you have a suggestion for a link that should be considered added to this post, feel free to leave a comment!
THANKS AGAIN FOR READING! SUGGEST A LINK! GOOD LUCK ON YOUR DEPLOYMENT!
During my time as a PFE for Microsoft, I have encounted many issues with Federation in a Hybrid Exchange Deployment. Recently, the following support announcement came out and I wanted to share as I hope this can help others that may be having issues out there.
One of the more common causes of HCW failures is the Federation Trust step for the Exchange on-premises organizations in Full hybrid configurations (Classic or Modern topologies).
Federation trust is a mandatory step in the on-premises Exchange organizations when configuring Full hybrid deployments, as this allows us to create organization relationships (for features like hybrid free/busy or OWA/EAS redirection) and sharing policies (1:1 hybrid calendar sharing). In Exchange Online multi-tenant organizations, federation trust is already in place.
Below is an illustration of an Exchange hybrid deployment where both the Exchange on-premises organization and the Exchange Online organization have a trust with Azure Authentication System (formerly called Microsoft Federation Gateway):
Example of Hybrid Federation
Before getting to our subject, let’s quickly go over different hybrid configurations and Hybrid Configuration Wizard (HCW) – as this is the supported tool to configure hybrid deployments. There are 2 flavors of hybrid configurations: • Classic hybrid • Modern hybrid
At this time, each of those supports the following hybrid modes:
Full
Minimal (which further breaks down into…)
Express (a one-time sync)
“Actual minimal”
A quick overview of Full / Minimal / Express options, can be found here. More info on HCW is here.
As mentioned earlier, a federation trust is created by HCW only in Full Hybrid.
HCW logs are located at %appdata%\Microsoft\Exchange Hybrid Configuration on the machine from where HCW was ran. The easiest way to get to them is to press F12 in the HCW window to open the Diagnostic tools and from there you can Open Folder Logging or Open Log File directly.
When you have issues with federation trust, the log will usually show errors when one of the following cmdlets are executed:
Set-FederationOrganizationIdentifier or Add-FederatedDomain (but can be other cmdlets as well).
Once you identified the exact cmdlet failing and where (Session=OnPremises – means Exchange Management Shell and Session=Tenant means Exchange Online PowerShell), you should copy-paste the failing command and try to execute it manually and see if that is failing as well (most likely it will). You can also open the shells from F12 Diagnostic tools windows in HCW.
In order to get more details on the error and to rule out this is not an issue with HCW itself, you will need to separately run the same command that threw exception in HCW log and add Verbose switch to get verbose details of the error and the serialized remote exception.
For example, if the Exchange server version is Exchange 2010, you will run the failing command with Verbose switch in Exchange Management Shell (EMS), see if that fails and then get the serialized remote exception.
Once you’ve gathered the verbose error / serialized exception, try to understand where it is failing (or provide it to Microsoft Support together with the HCW log).
Common Errors with Remediation Steps
Federation trust fails with “Object reference not set to an instance of an object”
This is a known old issue on Exchange 2016 CU7 servers, make sure your Exchange servers are updated to the latest CU.
Full error in the HCW log:
Log Entries Extended
1
2
3
2017.10.0601:45:56.562*ERROR*10277[Client=UX,Activity=Domain Ownership,Session=OnPremises,Cmdlet=Set-FederatedOrganizationIdentifier,Thread=21]FINISH Time=398.4msResults=PowerShell failed toinvoke'Set-FederatedOrganizationIdentifier':Objectreference notset toan instance of an object.An unexpected error has occurred andaWatson dump isbeing generated:Objectreference notset toan instance of an object.
2017.10.0601:45:56.563*ERROR*10224[Client=UX,Page=DomainProof,Thread=21]Microsoft.Online.CSE.Hybrid.PowerShell.PowerShellInvokeException:PowerShell failed toinvoke'Set-FederatedOrganizationIdentifier':Objectreference notset toan instance of an object.An unexpected error has occurred andaWatson dump isbeing generated:Objectreference notset toan instance of an object.--->System.Management.Automation.RemoteException:Objectreference notset toan instance of an object.
2019.07.0617:53:15.37510177[Client=UX,Activity=Domain Ownership,Provider=OnPremises,Thread=19]PowerShell Error Record:{CategoryInfo={Activity=Add-FederatedDomain,Category=InvalidResult,Reason=DomainProofOwnershipException,TargetName=,TargetType=},ErrorDetails=,Exception=Proof of domain ownership has failed.Make sure that the TXT record forthe specified domain isavailable inDNS.The format of the TXT record should be"example.com IN TXT hash-value"where"example.com"isthe domain you want toconfigure forFederation and"hash-value"isthe proof value generated with"Get-FederatedDomainProof -DomainName example.com".,FullyQualifiedErrorId=367408EF,Microsoft.Exchange.Management.SystemConfigurationTasks.AddFederatedDomain}
Resolution:
• Check the TXT record for your domain(s) in HCW log or in Exchange Management Shell with command Get-FederatedDomainProof -DomainName • See if it matches your published TXT record with either nslookup utility or by checking internet websites like https://www.whatsmydns.net/ put your domain in hostnames, type=txt, Nameservers – Authoritative
You would look for errors, missing records or unusual formatting (characters, spaces, quotes, TXT record split in half).
Federation fails with “An unexpected error occurred on a receive” or “An unexpected error occurred on a send.”
Full error in the HCW log:
Log Entries Extended
1
2018.10.1017:03:31.277*ERROR*[Activity=Domain Ownership,Session=OnPremises,Cmdlet=Set-FederatedOrganizationIdentifier]FINISH Time=64.3sResults=PowerShell failed toinvoke'Set-FederatedOrganizationIdentifier':An error occurred whileattempting toprovision Exchange tothe Partner STS.Detailed Information"An error occurred accessing Windows Live. Detailed information: "The underlying connection was closed:An unexpected error occurred onareceive.".".
Resolution:
Check outbound access from all your Exchange Servers to Microsoft Federation Gateway by browsing using Internet Explorer with PSEXEC tool (with -s and -i switches) from the Exchange Server (this will use Internet Explorer under System Account / Exchange Server Account).
From on-premises Exchange to Office 365, the Exchange 2010 MBX & CAS or 2013 MBX (backend) or 2016 / 2019 would need outbound Internet access to the Microsoft Federation Gateway in addition to https://outlook.office365.com/ews/exchange.asmx
Verify the machine/system account can access these Microsoft Federation Gateway URLs:
Note: If the Exchange requires a proxy server to access the Internet, specify the proxy server using “Set-ExchangeServer myExchange01 -InternetWebProxy http://myproxy:80”. Notice such proxy can’t require any user authentication for outbound Internet access, and the proxy must start with HTTP: and not HTTPS: (secure SSL).
You can also set the proxy using netsh as well.
set proxy proxy-server=”http=myproxy;https=sproxy:88″ bypass-list=”*.contoso.com”
In rare instances, you can use the machine/system account to access the URLs from the browser, but Exchange cmdlets still failed with “Could not establish trust relationship for the SSL/TLS secure channel.” If that happens, make sure the certificate authorities for the urls are installed at the Third-Party Root Certification Authorities of the machine local certificate location.
Federated delegation features require that the Mailbox and Client Access servers in your organization have outbound access to the Internet by using HTTPS. You must allow outbound HTTPS access (port 443 for TCP) from all Exchange 2010 Mailbox and Client Access servers in the organization.
There is no specific error / exception, in HCW log you would see it stops without any specific error.
in HCW log if you see something with “DEL“: “contoso.com/Configuration/Deleted Objects/Microsoft Federation Gateway/DEL: <xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx>”.
Solution is to remove the orphaned federation trust and re-run HCW.
NOTE: as a first step, you can try to run the commandremove-federateddomain with the switch -Force. Also, you don’t need to recreate federation trust manually, just re-run HCW (this will recreate federation trust for us)
Federation Trust fails with “InternalError InternalError: Internal error.”.”.””
2019.08.2307:45:23.239*ERROR*10277[Client=UX,Activity=Domain Ownership,Session=OnPremises,Cmdlet=Set-FederatedOrganizationIdentifier,Thread=20]FINISH Time=325.0msResults=PowerShell failed toinvoke'Set-FederatedOrganizationIdentifier':An error occurred whileattempting toprovision Exchange tothe Partner STS.Detailed Information"An unexpected result was received from Windows Live. Detailed information: "InternalError InternalError:Internal error.".".{CategoryInfo={Activity=[System.String]Set-FederatedOrganizationIdentifier,Category=[System.Management.Automation.ErrorCategory]InvalidResult,Reason=[System.String]
Resolution:
Open request with Microsoft Support or check if any Service Incident is published. Please see this for more information.
Federation trust fails with “1007 Access Denied”
Full error in the HCW log:
Log Entries Extended
1
Set-FederatedOrganizationIdentifier,Category=[System.Management.Automation.ErrorCategory]InvalidResult,Reason=[System.String]ProvisioningFederatedExchangeException,TargetName=[System.String],TargetType=[System.String]},ErrorDetails=,Exception=[System.Management.Automation.RemoteException]An error occurred whileattempting toprovision Exchange tothe Partner STS.Detailed Information"An unexpected result was received from Windows Live. Detailed information: "1007AccessDenied:Access Denied."."
Resolution:
“1007 Access Denied” error is usually when we have issues with:
Outdated federation trust (for example, federation trust certificate expired) and in this case you would remove federation trust by following these steps.
If the federation trust certificate is not found on any of the servers, then proceed with resolution from the next error.
As an example, from one HCW log, there seems to be this federation trust certificate expired on 05/13/2019:
Federation trust fails with “Federation Certificate cannot be found”
Full error in the HCW log:
Log Entries Extended
1
2019.11.2214:40:32.569*ERROR*10277[Client=UX,Activity=Domain Ownership,Session=OnPremises,Cmdlet=Get-FederatedDomainProof,Thread=6]FINISH Time=125.1msResults=PowerShell failed toinvoke Get-FederatedDomainProof:Federation certificate with the thumbprint‘03650FFAF05E83E3B007DF3473CB5753F5C4459’cannot be found.
Resolution:
Follow the procedure here to manually cleanup the federation trust from AD. Once this is done, re-run the HCW to re-create it automatically.
KEEP TROUBLESHOOTING! REMAIN VIGILANT USING FOCUSED INTENT, NEVER EMOTIONALIZING (FINE)!
When you
deploy or upgrade to Microsoft Exchange 2019 Cumulative Update 4 (CU4) on
Microsoft Windows Server 2019 or Windows 10 (Management Tools only) builds
1909 or 1903, the system prerequisites check fails, and you receive the
following error message:
By default, Windows builds 1909 and 1903 already have .NET Framework 4.8 installed. When you try to reinstall the software, the installation fails.
Cause
This
problem is caused by a prerequisite check that was introduced in Cumulative
Update 4. This process checks incorrectly for .NET Framework 4.8. Because the
prerequisite check doesn’t recognize that .NET Framework 4.8 is already
installed.
Status
Microsoft is researching this problem and will post more
information in this article when it becomes available.
Workaround
Important
Follow the steps in this section carefully. Serious problems might occur if you modify the registry incorrectly. Before you modify it, back up the registry for restoration in case problems occur.
Right-click the subkey, and select Permissions from the shortcut menu.
Select Advanced.
In the Advanced Security Settings window, locate the Owner attribute at the top of the window.
Select Change next to the listed owner.
In the Enter the object name to select field, enter the name of the local administrator group. For example, enter <computername>\Administrator. Then, select OK.
In the Advanced Security Settings window, select the local administrator group that you changed ownership to, and then select Edit.
Change the basic permissions to Full Control.
Select OK three times to save the changes and return to the main Registry Editor window.
Locate the following key in the path:
Name: Release Type: REG_DWORD Data: 528040 (decimal) Change the Data value to 528049 (decimal)
Rerun the Exchange system prerequisites check, and deploy or update Exchange Server 2019.
Start Registry Editor, locate the subkey that’s mentioned in step 2, repeat the necessary steps to locate the Release key, and then revert the Data value to 528040 (decimal).
This should allow everything to run correctly after the installation.
POSITIVE DAY TO YOU! FINE = Focused Intent Not Emotionalizing
I have run into this issue at a number of my customers that utilize an Exchange Edge Server in their Hybrid Deployment. They’ll need to modify their send connectors for their forced TLS communication with their partners or own mailboxes in Office365. Whenever they want to modify the send connector and save the changes, they get the following error messages:
Symptoms
“PowerShell failed to invoke ‘Set-SendConnector’: Error 0x5 (Access is denied) from cli_GetCertificate”
or
“Error 0x6ba (the RPC server is unavailable) from cli_GetCertificate”
This issue occurs because the TLS certificate check (in case the TlsCertificateName attribute is populated on the send connector) doesn’t work against the Edge servers as the RPC communication is blocked against the Edge servers.
Workaround
Now the current workaround for this has been to delete the Edge Send Connector and recreate the connector from scratch via PowerShell with all the settings and changes entered. This is not a viable solution especially if your communications with your partners change constantly and changes are made to the secure communications channel between you and them.
Resolution
To fix this issue, install one of the following updates:
This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. AcceptRead More
Privacy & Cookies Policy
Privacy Overview
This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
